2024-05-27T06:42:43.256 INFO:root:teuthology version: 0.0.1.dev291+ge7ae758 2024-05-27T06:42:43.257 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/dbench}} --owner scheduled_vshankar@teuthology --name vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi --archive /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630 -- /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630/orig.config.yaml 2024-05-27T06:42:43.312 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T06:42:43.410 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630 branch: wip-vshankar-testing-20240521.063558-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/dbench}} email: vshankar@redhat.com first_in_suite: false job_id: '7727630' kernel: flavor: default kdb: 1 sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240521.063558-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240521.063558-debug sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 owner: scheduled_vshankar@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2499 sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 sleep_before_teardown: 0 subset: 573/1024 suite: fs suite_branch: wip-vshankar-testing-20240521.063558-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPWhEbpGKQm9aH4rpfMoTu635NSpqwWrrlTpiW4dom3BIxjmiSQMBZqSGIXXm2FF/KrTt3gOU+yQDIeheJK+lZI= smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAGyNAqormBFeOf2C6t0LYyZ2ASucC9yAlRdusgUQY5bglbAHnn9btJVlAmMLHgNjPIMFmc1JKYzcisHzoU49lk= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKgkb25LHDnA/b9NkEk66tk/dfI+JJwNiCHTN3gjfOM0cwU9PMHo8bvaWx9Knw2G0MMbuZgtFoyROp+KVxz84JQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e7ae7580c417d6ab27fda98fecadca5d9f85d63e timestamp: 2024-05-27_05:52:49 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-27T06:42:43.411 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa; will attempt to use it 2024-05-27T06:42:43.411 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks 2024-05-27T06:42:43.411 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-27T06:42:43.413 INFO:teuthology.task.internal:Checking packages... 2024-05-27T06:42:43.431 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '38f9d900c83c63799fbdbe61acc9a11b0d3554a6' 2024-05-27T06:42:43.432 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-27T06:42:43.432 INFO:teuthology.packaging:ref: None 2024-05-27T06:42:43.432 INFO:teuthology.packaging:tag: None 2024-05-27T06:42:43.432 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240521.063558-debug 2024-05-27T06:42:43.432 INFO:teuthology.packaging:sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:42:43.432 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240521.063558-debug 2024-05-27T06:42:43.652 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3872.g38f9d900 2024-05-27T06:42:43.654 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-27T06:42:43.660 INFO:teuthology.task.internal:no buildpackages task found 2024-05-27T06:42:43.660 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-27T06:42:43.713 INFO:teuthology.task.internal:Saving configuration 2024-05-27T06:42:43.733 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-27T06:42:43.739 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-27T06:42:43.763 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:38:01.071919', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPWhEbpGKQm9aH4rpfMoTu635NSpqwWrrlTpiW4dom3BIxjmiSQMBZqSGIXXm2FF/KrTt3gOU+yQDIeheJK+lZI='} 2024-05-27T06:42:43.788 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:38:01.075262', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAGyNAqormBFeOf2C6t0LYyZ2ASucC9yAlRdusgUQY5bglbAHnn9btJVlAmMLHgNjPIMFmc1JKYzcisHzoU49lk='} 2024-05-27T06:42:43.810 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:38:01.073784', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKgkb25LHDnA/b9NkEk66tk/dfI+JJwNiCHTN3gjfOM0cwU9PMHo8bvaWx9Knw2G0MMbuZgtFoyROp+KVxz84JQ='} 2024-05-27T06:42:43.810 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-27T06:42:43.816 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-27T06:42:43.816 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-27T06:42:43.816 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-27T06:42:43.816 INFO:teuthology.run_tasks:Running task console_log... 2024-05-27T06:42:43.913 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2c72e83ac0>, signals=[15]) 2024-05-27T06:42:43.914 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-27T06:42:43.920 INFO:teuthology.task.internal:Opening connections... 2024-05-27T06:42:43.920 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2024-05-27T06:42:43.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:42:44.000 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-05-27T06:42:44.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:42:44.074 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2024-05-27T06:42:44.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:42:44.149 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-27T06:42:44.156 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2024-05-27T06:42:44.174 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2024-05-27T06:42:44.175 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2024-05-27T06:42:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="CentOS Stream" 2024-05-27T06:42:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="9" 2024-05-27T06:42:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:ID="centos" 2024-05-27T06:42:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="rhel fedora" 2024-05-27T06:42:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="9" 2024-05-27T06:42:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el9" 2024-05-27T06:42:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T06:42:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2024-05-27T06:42:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:LOGO="fedora-logo-icon" 2024-05-27T06:42:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T06:42:44.234 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://centos.org/" 2024-05-27T06:42:44.234 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T06:42:44.234 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T06:42:44.234 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T06:42:44.234 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-05-27T06:42:44.258 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-05-27T06:42:44.276 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-05-27T06:42:44.276 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-05-27T06:42:44.333 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2024-05-27T06:42:44.333 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="9" 2024-05-27T06:42:44.333 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2024-05-27T06:42:44.333 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2024-05-27T06:42:44.333 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="9" 2024-05-27T06:42:44.333 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el9" 2024-05-27T06:42:44.333 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T06:42:44.334 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-05-27T06:42:44.334 INFO:teuthology.orchestra.run.smithi132.stdout:LOGO="fedora-logo-icon" 2024-05-27T06:42:44.334 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T06:42:44.334 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2024-05-27T06:42:44.334 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T06:42:44.334 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T06:42:44.334 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T06:42:44.334 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-05-27T06:42:44.355 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2024-05-27T06:42:44.373 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2024-05-27T06:42:44.373 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2024-05-27T06:42:44.430 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2024-05-27T06:42:44.430 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="9" 2024-05-27T06:42:44.430 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2024-05-27T06:42:44.430 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2024-05-27T06:42:44.430 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="9" 2024-05-27T06:42:44.430 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el9" 2024-05-27T06:42:44.431 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-27T06:42:44.431 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2024-05-27T06:42:44.431 INFO:teuthology.orchestra.run.smithi179.stdout:LOGO="fedora-logo-icon" 2024-05-27T06:42:44.431 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-27T06:42:44.431 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2024-05-27T06:42:44.431 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-27T06:42:44.431 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-27T06:42:44.431 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-27T06:42:44.431 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2024-05-27T06:42:44.455 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-27T06:42:44.466 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-27T06:42:44.502 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-27T06:42:44.502 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2024-05-27T06:42:44.504 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-05-27T06:42:44.507 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2024-05-27T06:42:44.523 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-27T06:42:44.529 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-27T06:42:44.529 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2024-05-27T06:42:44.562 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-05-27T06:42:44.565 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2024-05-27T06:42:44.643 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-27T06:42:44.720 INFO:teuthology.run_tasks:Running task kernel... 2024-05-27T06:42:44.734 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'} 2024-05-27T06:42:44.734 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}, 'mon.b': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}, 'mon.c': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}}, timeout 300 2024-05-27T06:42:44.734 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T06:42:44.735 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T06:42:44.736 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T06:42:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:42:44.752 DEBUG:teuthology.task.kernel:sha1 for mon.b is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:42:44.753 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-27T06:42:44.753 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-05-27T06:42:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:42:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:42:44.754 DEBUG:teuthology.task.kernel:sha1 for mon.a is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:42:44.754 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-27T06:42:44.754 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-05-27T06:42:44.755 DEBUG:teuthology.task.kernel:sha1 for mon.c is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:42:44.755 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-27T06:42:44.755 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-05-27T06:42:44.810 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-437.el9.x86_64 2024-05-27T06:42:44.810 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:42:44.811 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-27T06:42:44.811 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:42:44.815 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-437.el9.x86_64 2024-05-27T06:42:44.815 INFO:teuthology.orchestra.run.smithi043.stdout:5.14.0-437.el9.x86_64 2024-05-27T06:42:44.815 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:42:44.815 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-27T06:42:44.815 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:42:44.816 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:42:44.816 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-27T06:42:44.816 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:42:44.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum upgrade -y linux-firmware 2024-05-27T06:42:44.966 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum upgrade -y linux-firmware 2024-05-27T06:42:45.033 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum upgrade -y linux-firmware 2024-05-27T06:42:47.092 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-05-27T06:42:47.101 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.1 MB 00:00 2024-05-27T06:42:48.247 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.1 MB 00:01 2024-05-27T06:42:50.115 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-05-27T06:42:50.543 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 19 MB 00:02 2024-05-27T06:42:51.060 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 19 MB 00:02 2024-05-27T06:42:54.565 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.2 MB 00:00 2024-05-27T06:42:54.752 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-05-27T06:42:55.727 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.2 MB 00:00 2024-05-27T06:42:56.169 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 17 kB 00:00 2024-05-27T06:42:56.233 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 17 kB 00:00 2024-05-27T06:42:57.127 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-05-27T06:42:57.221 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-05-27T06:42:57.294 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 17 kB 00:00 2024-05-27T06:43:02.344 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-05-27T06:43:02.596 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-05-27T06:43:04.290 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-05-27T06:43:04.291 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-05-27T06:43:04.291 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-05-27T06:43:04.402 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.c... 2024-05-27T06:43:04.402 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:43:04.548 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ 2024-05-27T06:43:04.548 DEBUG:teuthology.orchestra.run.smithi179:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_gb9459a26b088-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-27T06:43:04.550 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi179.front.sepia.ceph.com to complete... 2024-05-27T06:43:04.658 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-05-27T06:43:04.659 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-05-27T06:43:04.660 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-05-27T06:43:04.761 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.a... 2024-05-27T06:43:04.762 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:43:04.988 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ 2024-05-27T06:43:04.989 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_gb9459a26b088-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-27T06:43:04.991 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi043.front.sepia.ceph.com to complete... 2024-05-27T06:43:12.552 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 1.4 MB/s | 21 MB 00:15 2024-05-27T06:43:17.710 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-05-27T06:43:19.683 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-05-27T06:43:19.684 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-05-27T06:43:19.684 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-05-27T06:43:19.801 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.b... 2024-05-27T06:43:19.801 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:43:19.953 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ 2024-05-27T06:43:19.954 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_gb9459a26b088-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-27T06:43:19.956 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi132.front.sepia.ceph.com to complete... 2024-05-27T06:43:34.639 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27 06:43:34 URL:https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/kernel-6.9.0_gb9459a26b088-1.x86_64.rpm [356863406/356863406] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-27T06:43:34.639 INFO:teuthology.orchestra.run.smithi179.stderr:FINISHED --2024-05-27 06:43:34-- 2024-05-27T06:43:34.640 INFO:teuthology.orchestra.run.smithi179.stderr:Total wall clock time: 30s 2024-05-27T06:43:34.640 INFO:teuthology.orchestra.run.smithi179.stderr:Downloaded: 1 files, 340M in 30s (11.5 MB/s) 2024-05-27T06:43:34.642 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.c... 2024-05-27T06:43:34.642 DEBUG:teuthology.orchestra.run.smithi179:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-27T06:43:36.781 INFO:teuthology.orchestra.run.smithi179.stdout:Verifying... ######################################## 2024-05-27T06:43:37.508 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing... ######################################## 2024-05-27T06:43:38.457 INFO:teuthology.orchestra.run.smithi179.stdout:Updating / installing... 2024-05-27T06:43:46.412 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-6.9.0_gb9459a26b088-1 ######################################## 2024-05-27T06:44:05.557 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27 06:44:05 URL:https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/kernel-6.9.0_gb9459a26b088-1.x86_64.rpm [356863406/356863406] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-27T06:44:05.558 INFO:teuthology.orchestra.run.smithi132.stderr:FINISHED --2024-05-27 06:44:05-- 2024-05-27T06:44:05.558 INFO:teuthology.orchestra.run.smithi132.stderr:Total wall clock time: 46s 2024-05-27T06:44:05.558 INFO:teuthology.orchestra.run.smithi132.stderr:Downloaded: 1 files, 340M in 45s (7.56 MB/s) 2024-05-27T06:44:05.561 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.b... 2024-05-27T06:44:05.561 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-27T06:44:07.647 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying... ######################################## 2024-05-27T06:44:07.870 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing... ######################################## 2024-05-27T06:44:07.984 INFO:teuthology.orchestra.run.smithi132.stdout:Updating / installing... 2024-05-27T06:44:12.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27 06:44:12 URL:https://3.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/centos/9/flavors/default/x86_64/kernel-6.9.0_gb9459a26b088-1.x86_64.rpm [356863406/356863406] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-27T06:44:12.468 INFO:teuthology.orchestra.run.smithi043.stderr:FINISHED --2024-05-27 06:44:12-- 2024-05-27T06:44:12.468 INFO:teuthology.orchestra.run.smithi043.stderr:Total wall clock time: 1m 7s 2024-05-27T06:44:12.468 INFO:teuthology.orchestra.run.smithi043.stderr:Downloaded: 1 files, 340M in 1m 7s (5.08 MB/s) 2024-05-27T06:44:12.470 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.a... 2024-05-27T06:44:12.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-27T06:44:14.659 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying... ######################################## 2024-05-27T06:44:14.797 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing... ######################################## 2024-05-27T06:44:14.913 INFO:teuthology.orchestra.run.smithi043.stdout:Updating / installing... 2024-05-27T06:44:15.912 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-6.9.0_gb9459a26b088-1 ######################################## 2024-05-27T06:44:23.187 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-6.9.0_gb9459a26b088-1 ######################################## 2024-05-27T06:44:24.041 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-27T06:44:24.041 DEBUG:teuthology.orchestra.run.smithi179:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-27T06:44:24.125 INFO:teuthology.orchestra.run.smithi179.stdout:/boot/System.map-6.9.0-gb9459a26b088 2024-05-27T06:44:24.125 INFO:teuthology.orchestra.run.smithi179.stdout:/boot/config-6.9.0-gb9459a26b088 2024-05-27T06:44:24.125 INFO:teuthology.orchestra.run.smithi179.stdout:/boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:44:24.125 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/System.map 2024-05-27T06:44:24.125 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/config 2024-05-27T06:44:24.125 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel 2024-05-27T06:44:24.125 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch 2024-05-27T06:44:24.125 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86 2024-05-27T06:44:24.125 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto 2024-05-27T06:44:24.125 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-27T06:44:24.126 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-27T06:44:24.127 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-27T06:44:24.128 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-27T06:44:24.129 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-27T06:44:24.130 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers 2024-05-27T06:44:24.131 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ata_generic.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-27T06:44:24.132 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-27T06:44:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-27T06:44:24.134 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sis.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-27T06:44:24.135 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay 2024-05-27T06:44:24.136 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/brd.ko 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-27T06:44:24.137 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/loop.ko 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/virtio_blk.ko 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-27T06:44:24.138 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkfront.ko 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi 2024-05-27T06:44:24.139 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-27T06:44:24.140 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm.ko 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/uv_mmtimer.ko 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/virtio_console.ko 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-27T06:44:24.141 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-27T06:44:24.142 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat 2024-05-27T06:44:24.143 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-27T06:44:24.144 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/edd.ko 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-27T06:44:24.145 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-27T06:44:24.146 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-27T06:44:24.147 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-27T06:44:24.148 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-27T06:44:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-27T06:44:24.150 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-27T06:44:24.151 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-27T06:44:24.152 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-27T06:44:24.153 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-27T06:44:24.154 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-27T06:44:24.154 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-27T06:44:24.154 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-27T06:44:24.154 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-27T06:44:24.154 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-27T06:44:24.154 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-27T06:44:24.154 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-27T06:44:24.154 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-27T06:44:24.154 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-27T06:44:24.154 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-27T06:44:24.155 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-27T06:44:24.156 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-27T06:44:24.157 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-27T06:44:24.158 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-27T06:44:24.159 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-27T06:44:24.160 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-27T06:44:24.161 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-27T06:44:24.162 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-27T06:44:24.163 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-dev.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-27T06:44:24.164 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca 2024-05-27T06:44:24.165 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input 2024-05-27T06:44:24.166 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-27T06:44:24.167 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-27T06:44:24.168 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md 2024-05-27T06:44:24.169 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-27T06:44:24.170 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-27T06:44:24.171 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mfd-core.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-27T06:44:24.172 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-27T06:44:24.173 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru 2024-05-27T06:44:24.174 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net 2024-05-27T06:44:24.175 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet 2024-05-27T06:44:24.176 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere 2024-05-27T06:44:24.177 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-27T06:44:24.178 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet 2024-05-27T06:44:24.179 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-27T06:44:24.180 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-27T06:44:24.181 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-27T06:44:24.182 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-27T06:44:24.183 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-27T06:44:24.184 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia 2024-05-27T06:44:24.185 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-27T06:44:24.186 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-27T06:44:24.187 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-27T06:44:24.188 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-27T06:44:24.189 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-27T06:44:24.190 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/net_failover.ko 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy 2024-05-27T06:44:24.191 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/dp83640.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-27T06:44:24.192 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-27T06:44:24.193 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slhc.ko 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-27T06:44:24.194 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tun.ko 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/virtio_net.ko 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-27T06:44:24.195 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netfront.ko 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-27T06:44:24.196 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl 2024-05-27T06:44:24.197 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-27T06:44:24.198 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-27T06:44:24.199 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-27T06:44:24.200 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-27T06:44:24.201 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-27T06:44:24.202 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-27T06:44:24.203 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-27T06:44:24.204 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-27T06:44:24.205 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-27T06:44:24.206 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-27T06:44:24.207 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas 2024-05-27T06:44:24.208 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-27T06:44:24.209 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-27T06:44:24.210 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-27T06:44:24.211 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal 2024-05-27T06:44:24.212 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-27T06:44:24.213 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial 2024-05-27T06:44:24.214 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-27T06:44:24.215 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-27T06:44:24.216 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-27T06:44:24.217 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-27T06:44:24.218 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-27T06:44:24.219 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-27T06:44:24.220 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-27T06:44:24.221 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-27T06:44:24.222 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci 2024-05-27T06:44:24.223 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio 2024-05-27T06:44:24.224 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio.ko 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_balloon.ko 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_mmio.ko 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci.ko 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_ring.ko 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves 2024-05-27T06:44:24.225 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-27T06:44:24.226 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-27T06:44:24.227 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-27T06:44:24.228 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p 2024-05-27T06:44:24.229 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-27T06:44:24.230 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/fat.ko 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/vfat.ko 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse 2024-05-27T06:44:24.231 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-27T06:44:24.232 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/fuse.ko 2024-05-27T06:44:24.232 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2 2024-05-27T06:44:24.232 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-27T06:44:24.232 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs 2024-05-27T06:44:24.232 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-27T06:44:24.232 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus 2024-05-27T06:44:24.232 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-27T06:44:24.232 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs 2024-05-27T06:44:24.232 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-27T06:44:24.232 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-27T06:44:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-27T06:44:24.234 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-27T06:44:24.235 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-27T06:44:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-27T06:44:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs 2024-05-27T06:44:24.238 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb 2024-05-27T06:44:24.239 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs 2024-05-27T06:44:24.240 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu/rcutorture.ko 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/asn1_encoder.ko 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-27T06:44:24.241 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-27T06:44:24.242 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd/zstd_compress.ko 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-27T06:44:24.243 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p 2024-05-27T06:44:24.244 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-27T06:44:24.245 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-27T06:44:24.246 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/failover.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-27T06:44:24.247 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver/dns_resolver.ko 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-27T06:44:24.248 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-27T06:44:24.249 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-27T06:44:24.250 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-27T06:44:24.251 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-27T06:44:24.252 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-27T06:44:24.253 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-27T06:44:24.254 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-27T06:44:24.255 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-27T06:44:24.256 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-27T06:44:24.257 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-27T06:44:24.258 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-27T06:44:24.259 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-27T06:44:24.260 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-27T06:44:24.261 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_quota.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-27T06:44:24.262 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tunnel.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-27T06:44:24.263 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-27T06:44:24.264 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-27T06:44:24.265 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-27T06:44:24.266 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-27T06:44:24.267 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-27T06:44:24.268 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-27T06:44:24.269 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-27T06:44:24.270 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-27T06:44:24.271 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-27T06:44:24.272 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma 2024-05-27T06:44:24.273 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-27T06:44:24.274 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys/trusted.ko 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-27T06:44:24.275 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-27T06:44:24.276 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-27T06:44:24.276 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-27T06:44:24.276 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-27T06:44:24.276 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-27T06:44:24.276 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-27T06:44:24.276 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-27T06:44:24.276 INFO:teuthology.orchestra.run.smithi179.stdout:/lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:44:24.280 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-27T06:44:24.280 DEBUG:teuthology.orchestra.run.smithi179:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-27T06:44:24.303 INFO:teuthology.orchestra.run.smithi179.stdout:postinstall scriptlet (using /bin/sh): 2024-05-27T06:44:24.303 INFO:teuthology.orchestra.run.smithi179.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:44:24.303 INFO:teuthology.orchestra.run.smithi179.stdout: /usr/bin/kernel-install add 6.9.0-gb9459a26b088 /lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout:fi 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout:for file in vmlinuz System.map config; do 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout: if ! cmp --silent "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088"; then 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout: cp "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088" 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout: fi 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout:done 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout:if [ -d "/lib/modules/6.9.0-gb9459a26b088/dtb" ] && \ 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout: ! diff -rq "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" >/dev/null 2>&1; then 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout: rm -rf "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout: cp -r "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout:fi 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout:if [ ! -e "/lib/modules/6.9.0-gb9459a26b088/modules.dep" ]; then 2024-05-27T06:44:24.304 INFO:teuthology.orchestra.run.smithi179.stdout: /usr/sbin/depmod 6.9.0-gb9459a26b088 2024-05-27T06:44:24.305 INFO:teuthology.orchestra.run.smithi179.stdout:fi 2024-05-27T06:44:24.305 INFO:teuthology.orchestra.run.smithi179.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-27T06:44:24.305 INFO:teuthology.orchestra.run.smithi179.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:44:24.305 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-install remove 6.9.0-gb9459a26b088 2024-05-27T06:44:24.305 INFO:teuthology.orchestra.run.smithi179.stdout:fi 2024-05-27T06:44:24.306 DEBUG:teuthology.orchestra.run.smithi179:> sudo rpm -qi grub2-tools 2024-05-27T06:44:24.380 INFO:teuthology.orchestra.run.smithi179.stdout:Name : grub2-tools 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Epoch : 1 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Version : 2.06 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Release : 68.el9 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: x86_64 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Group : Unspecified 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Size : 8264254 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:License : GPLv3+ 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-27T06:44:24.381 INFO:teuthology.orchestra.run.smithi179.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout:Packager : builder@centos.org 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout:Vendor : CentOS 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout:Summary : Support tools for GRUB. 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout:Description : 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout:hardware devices. 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:44:24.382 INFO:teuthology.orchestra.run.smithi179.stdout:This subpackage provides tools for support of all platforms. 2024-05-27T06:44:24.383 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-27T06:44:24.383 INFO:teuthology.task.kernel:Updating grub on smithi179 to boot 6.9.0-gb9459a26b088 2024-05-27T06:44:24.383 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-27T06:44:24.989 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2024-05-27T06:44:26.354 INFO:teuthology.orchestra.run.smithi179.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-27T06:44:26.398 INFO:teuthology.orchestra.run.smithi179.stderr:done 2024-05-27T06:44:26.400 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-05-27T06:44:26.415 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.5awten8eUC 2024-05-27T06:44:26.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5awten8eUC 2024-05-27T06:44:26.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.5awten8eUC 2024-05-27T06:44:26.596 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.5awten8eUC is 6KB 2024-05-27T06:44:26.645 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.5awten8eUC 2024-05-27T06:44:26.659 DEBUG:teuthology.orchestra.run.smithi179:> sudo /bin/ls /boot/loader/entries || true 2024-05-27T06:44:26.721 INFO:teuthology.orchestra.run.smithi179.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-27T06:44:26.721 INFO:teuthology.orchestra.run.smithi179.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-27T06:44:26.721 INFO:teuthology.orchestra.run.smithi179.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-27T06:44:26.721 INFO:teuthology.orchestra.run.smithi179.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088.conf 2024-05-27T06:44:26.723 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088 2024-05-27T06:44:26.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2024-05-27T06:44:53.393 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi132.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-27T06:44:53.394 DEBUG:teuthology.orchestra.run.smithi132:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/boot/System.map-6.9.0-gb9459a26b088 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/boot/config-6.9.0-gb9459a26b088 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/System.map 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/config 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-27T06:44:53.470 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-27T06:44:53.471 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-27T06:44:53.472 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-27T06:44:53.473 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-27T06:44:53.474 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-27T06:44:53.475 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-27T06:44:53.476 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ata_generic.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-27T06:44:53.477 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-27T06:44:53.478 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-27T06:44:53.479 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sis.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-27T06:44:53.480 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block 2024-05-27T06:44:53.481 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/brd.ko 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/loop.ko 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-27T06:44:53.482 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/virtio_blk.ko 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkfront.ko 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-27T06:44:53.483 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-27T06:44:53.484 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/uv_mmtimer.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/virtio_console.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-27T06:44:53.485 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-27T06:44:53.486 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac 2024-05-27T06:44:53.487 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/edd.ko 2024-05-27T06:44:53.488 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-27T06:44:53.489 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-27T06:44:53.490 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-27T06:44:53.491 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-27T06:44:53.492 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-27T06:44:53.493 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-27T06:44:53.494 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-27T06:44:53.495 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-27T06:44:53.496 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-27T06:44:53.497 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-27T06:44:53.498 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-27T06:44:53.499 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-27T06:44:53.500 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-27T06:44:53.501 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-27T06:44:53.502 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-27T06:44:53.503 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-27T06:44:53.504 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-27T06:44:53.505 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-dev.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-27T06:44:53.506 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-27T06:44:53.507 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-27T06:44:53.508 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-27T06:44:53.509 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-27T06:44:53.510 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-27T06:44:53.511 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-27T06:44:53.512 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd 2024-05-27T06:44:53.513 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mfd-core.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710 2024-05-27T06:44:53.514 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-27T06:44:53.515 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net 2024-05-27T06:44:53.516 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-27T06:44:53.517 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera 2024-05-27T06:44:53.518 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-27T06:44:53.519 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-27T06:44:53.520 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip 2024-05-27T06:44:53.521 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-27T06:44:53.522 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb 2024-05-27T06:44:53.523 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-27T06:44:53.524 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-27T06:44:53.525 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-27T06:44:53.526 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-27T06:44:53.527 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-27T06:44:53.528 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-27T06:44:53.529 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-27T06:44:53.530 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/net_failover.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-27T06:44:53.531 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/dp83640.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-27T06:44:53.532 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-27T06:44:53.533 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slhc.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tun.ko 2024-05-27T06:44:53.534 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/virtio_net.ko 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netfront.ko 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-27T06:44:53.535 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-27T06:44:53.536 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-27T06:44:53.537 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-27T06:44:53.538 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-27T06:44:53.539 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-27T06:44:53.540 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-27T06:44:53.541 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-27T06:44:53.542 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-27T06:44:53.543 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-27T06:44:53.544 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-27T06:44:53.545 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-27T06:44:53.546 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid 2024-05-27T06:44:53.547 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx 2024-05-27T06:44:53.548 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-27T06:44:53.549 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-27T06:44:53.550 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-27T06:44:53.551 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-27T06:44:53.552 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core 2024-05-27T06:44:53.553 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-27T06:44:53.554 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-27T06:44:53.555 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-27T06:44:53.556 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-27T06:44:53.557 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-27T06:44:53.558 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-27T06:44:53.559 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-27T06:44:53.560 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-27T06:44:53.561 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio.ko 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_balloon.ko 2024-05-27T06:44:53.562 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_mmio.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_ring.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-27T06:44:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-27T06:44:53.564 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-27T06:44:53.565 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-27T06:44:53.566 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-27T06:44:53.567 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/fat.ko 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-27T06:44:53.568 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/vfat.ko 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/fuse.ko 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd 2024-05-27T06:44:53.569 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-27T06:44:53.570 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-27T06:44:53.571 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-27T06:44:53.572 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-27T06:44:53.573 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-27T06:44:53.574 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-27T06:44:53.575 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-27T06:44:53.576 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu 2024-05-27T06:44:53.577 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu/rcutorture.ko 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/asn1_encoder.ko 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4 2024-05-27T06:44:53.578 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd/zstd_compress.ko 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-27T06:44:53.579 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-27T06:44:53.580 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter 2024-05-27T06:44:53.581 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-27T06:44:53.582 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core 2024-05-27T06:44:53.583 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/failover.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver/dns_resolver.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-27T06:44:53.584 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-27T06:44:53.585 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-27T06:44:53.586 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-27T06:44:53.587 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-27T06:44:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-27T06:44:53.589 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-27T06:44:53.590 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-27T06:44:53.591 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-27T06:44:53.592 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-27T06:44:53.593 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-27T06:44:53.594 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-27T06:44:53.595 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-27T06:44:53.596 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-27T06:44:53.597 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_quota.ko 2024-05-27T06:44:53.598 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tunnel.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-27T06:44:53.599 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-27T06:44:53.600 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-27T06:44:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-27T06:44:53.602 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-27T06:44:53.603 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch 2024-05-27T06:44:53.604 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-27T06:44:53.605 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-27T06:44:53.606 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-27T06:44:53.607 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp 2024-05-27T06:44:53.608 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock 2024-05-27T06:44:53.609 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys/trusted.ko 2024-05-27T06:44:53.610 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-27T06:44:53.611 INFO:teuthology.orchestra.run.smithi132.stdout:/lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:44:53.615 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-27T06:44:53.615 DEBUG:teuthology.orchestra.run.smithi132:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-27T06:44:53.636 INFO:teuthology.orchestra.run.smithi132.stdout:postinstall scriptlet (using /bin/sh): 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: /usr/bin/kernel-install add 6.9.0-gb9459a26b088 /lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout:fi 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout:for file in vmlinuz System.map config; do 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: if ! cmp --silent "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088"; then 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: cp "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088" 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: fi 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout:done 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout:if [ -d "/lib/modules/6.9.0-gb9459a26b088/dtb" ] && \ 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: ! diff -rq "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" >/dev/null 2>&1; then 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: rm -rf "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:44:53.637 INFO:teuthology.orchestra.run.smithi132.stdout: cp -r "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:44:53.638 INFO:teuthology.orchestra.run.smithi132.stdout:fi 2024-05-27T06:44:53.638 INFO:teuthology.orchestra.run.smithi132.stdout:if [ ! -e "/lib/modules/6.9.0-gb9459a26b088/modules.dep" ]; then 2024-05-27T06:44:53.638 INFO:teuthology.orchestra.run.smithi132.stdout: /usr/sbin/depmod 6.9.0-gb9459a26b088 2024-05-27T06:44:53.638 INFO:teuthology.orchestra.run.smithi132.stdout:fi 2024-05-27T06:44:53.638 INFO:teuthology.orchestra.run.smithi132.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-27T06:44:53.638 INFO:teuthology.orchestra.run.smithi132.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:44:53.638 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-install remove 6.9.0-gb9459a26b088 2024-05-27T06:44:53.638 INFO:teuthology.orchestra.run.smithi132.stdout:fi 2024-05-27T06:44:53.639 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -qi grub2-tools 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:Name : grub2-tools 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:Epoch : 1 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:Version : 2.06 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:Release : 68.el9 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: x86_64 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:Group : Unspecified 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:Size : 8264254 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:License : GPLv3+ 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-27T06:44:53.709 INFO:teuthology.orchestra.run.smithi132.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:Packager : builder@centos.org 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:Vendor : CentOS 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:Summary : Support tools for GRUB. 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:Description : 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:hardware devices. 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:44:53.710 INFO:teuthology.orchestra.run.smithi132.stdout:This subpackage provides tools for support of all platforms. 2024-05-27T06:44:53.711 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-27T06:44:53.711 INFO:teuthology.task.kernel:Updating grub on smithi132 to boot 6.9.0-gb9459a26b088 2024-05-27T06:44:53.712 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-27T06:44:54.251 INFO:teuthology.orchestra.run.smithi132.stderr:Generating grub configuration file ... 2024-05-27T06:44:55.631 INFO:teuthology.orchestra.run.smithi132.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-27T06:44:55.656 INFO:teuthology.orchestra.run.smithi132.stderr:done 2024-05-27T06:44:55.658 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-05-27T06:44:55.673 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.aHs40FIuYM 2024-05-27T06:44:55.673 DEBUG:teuthology.orchestra.run.smithi132:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aHs40FIuYM 2024-05-27T06:44:55.748 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.aHs40FIuYM 2024-05-27T06:44:55.891 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.aHs40FIuYM is 6KB 2024-05-27T06:44:55.938 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.aHs40FIuYM 2024-05-27T06:44:55.952 DEBUG:teuthology.orchestra.run.smithi132:> sudo /bin/ls /boot/loader/entries || true 2024-05-27T06:44:56.022 INFO:teuthology.orchestra.run.smithi132.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-27T06:44:56.022 INFO:teuthology.orchestra.run.smithi132.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-27T06:44:56.022 INFO:teuthology.orchestra.run.smithi132.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-27T06:44:56.022 INFO:teuthology.orchestra.run.smithi132.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088.conf 2024-05-27T06:44:56.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088 2024-05-27T06:44:56.177 DEBUG:teuthology.orchestra.run.smithi132:> sudo shutdown -r now 2024-05-27T06:44:56.869 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-05-27T06:44:56.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-05-27T06:44:56.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:45:01.512 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi043.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-27T06:45:01.512 DEBUG:teuthology.orchestra.run.smithi043:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-27T06:45:01.597 INFO:teuthology.orchestra.run.smithi043.stdout:/boot/System.map-6.9.0-gb9459a26b088 2024-05-27T06:45:01.597 INFO:teuthology.orchestra.run.smithi043.stdout:/boot/config-6.9.0-gb9459a26b088 2024-05-27T06:45:01.597 INFO:teuthology.orchestra.run.smithi043.stdout:/boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:45:01.597 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/System.map 2024-05-27T06:45:01.597 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/config 2024-05-27T06:45:01.597 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel 2024-05-27T06:45:01.597 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch 2024-05-27T06:45:01.597 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-27T06:45:01.598 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-27T06:45:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-27T06:45:01.602 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-27T06:45:01.603 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata 2024-05-27T06:45:01.604 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ata_generic.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-27T06:45:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-27T06:45:01.606 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sis.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-27T06:45:01.607 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-27T06:45:01.608 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/brd.ko 2024-05-27T06:45:01.609 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/loop.ko 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/virtio_blk.ko 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback 2024-05-27T06:45:01.610 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkfront.ko 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-27T06:45:01.611 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm 2024-05-27T06:45:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/uv_mmtimer.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/virtio_console.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-27T06:45:01.613 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-27T06:45:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-27T06:45:01.615 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-27T06:45:01.616 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-27T06:45:01.616 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-27T06:45:01.616 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-27T06:45:01.616 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-27T06:45:01.616 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-27T06:45:01.616 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-27T06:45:01.616 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-27T06:45:01.616 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-27T06:45:01.616 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-27T06:45:01.616 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/edd.ko 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-27T06:45:01.617 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-27T06:45:01.618 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-27T06:45:01.619 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-27T06:45:01.619 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-27T06:45:01.619 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-27T06:45:01.619 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-27T06:45:01.619 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-27T06:45:01.619 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-27T06:45:01.619 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-27T06:45:01.619 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-27T06:45:01.619 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-27T06:45:01.620 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-27T06:45:01.621 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv 2024-05-27T06:45:01.622 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-27T06:45:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-27T06:45:01.624 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-27T06:45:01.625 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-27T06:45:01.626 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-27T06:45:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-27T06:45:01.628 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-27T06:45:01.629 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-27T06:45:01.630 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-27T06:45:01.631 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c 2024-05-27T06:45:01.632 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos 2024-05-27T06:45:01.633 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-27T06:45:01.633 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-27T06:45:01.633 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses 2024-05-27T06:45:01.633 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-27T06:45:01.633 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-27T06:45:01.633 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-27T06:45:01.633 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-27T06:45:01.634 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-27T06:45:01.635 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-27T06:45:01.635 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-27T06:45:01.635 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-27T06:45:01.635 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-27T06:45:01.635 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-27T06:45:01.635 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-27T06:45:01.635 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-dev.ko 2024-05-27T06:45:01.635 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-27T06:45:01.635 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-27T06:45:01.635 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-27T06:45:01.636 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw 2024-05-27T06:45:01.637 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4 2024-05-27T06:45:01.637 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-27T06:45:01.637 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4 2024-05-27T06:45:01.637 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-27T06:45:01.637 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5 2024-05-27T06:45:01.637 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-27T06:45:01.637 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca 2024-05-27T06:45:01.637 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-27T06:45:01.637 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-27T06:45:01.638 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-27T06:45:01.639 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-27T06:45:01.640 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-27T06:45:01.641 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-27T06:45:01.642 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion 2024-05-27T06:45:01.643 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mfd-core.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-27T06:45:01.644 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-27T06:45:01.645 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp 2024-05-27T06:45:01.646 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53 2024-05-27T06:45:01.647 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-27T06:45:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd 2024-05-27T06:45:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-27T06:45:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-27T06:45:01.651 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip 2024-05-27T06:45:01.652 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-27T06:45:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf 2024-05-27T06:45:01.654 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-27T06:45:01.655 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-27T06:45:01.656 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-27T06:45:01.657 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-27T06:45:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun 2024-05-27T06:45:01.659 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-27T06:45:01.660 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-27T06:45:01.661 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/net_failover.ko 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-27T06:45:01.662 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/dp83640.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-27T06:45:01.663 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-27T06:45:01.664 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slhc.ko 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-27T06:45:01.665 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tun.ko 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/virtio_net.ko 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-27T06:45:01.666 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netfront.ko 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci 2024-05-27T06:45:01.667 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86 2024-05-27T06:45:01.668 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-27T06:45:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-27T06:45:01.670 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl 2024-05-27T06:45:01.671 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-27T06:45:01.672 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-27T06:45:01.673 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-27T06:45:01.674 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa 2024-05-27T06:45:01.675 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-27T06:45:01.676 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-27T06:45:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-27T06:45:01.678 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-27T06:45:01.679 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-27T06:45:01.680 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target 2024-05-27T06:45:01.681 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal 2024-05-27T06:45:01.682 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-27T06:45:01.683 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250 2024-05-27T06:45:01.684 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-27T06:45:01.685 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc 2024-05-27T06:45:01.686 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-27T06:45:01.687 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-27T06:45:01.688 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-27T06:45:01.689 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-27T06:45:01.690 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage 2024-05-27T06:45:01.691 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-27T06:45:01.692 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-27T06:45:01.693 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_balloon.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_mmio.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-27T06:45:01.694 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/virtio/virtio_ring.ko 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-27T06:45:01.695 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-27T06:45:01.696 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-27T06:45:01.697 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs 2024-05-27T06:45:01.698 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph 2024-05-27T06:45:01.699 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/fat.ko 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-27T06:45:01.700 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/vfat.ko 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/fuse.ko 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs 2024-05-27T06:45:01.701 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout 2024-05-27T06:45:01.702 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-27T06:45:01.703 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-27T06:45:01.704 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-27T06:45:01.705 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-27T06:45:01.706 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-27T06:45:01.707 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client 2024-05-27T06:45:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-27T06:45:01.709 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common 2024-05-27T06:45:01.709 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-27T06:45:01.709 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-27T06:45:01.709 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs 2024-05-27T06:45:01.709 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-27T06:45:01.709 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv 2024-05-27T06:45:01.709 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-27T06:45:01.709 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf 2024-05-27T06:45:01.709 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-27T06:45:01.709 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/rcu/rcutorture.ko 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/asn1_encoder.ko 2024-05-27T06:45:01.710 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6 2024-05-27T06:45:01.711 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/lib/zstd/zstd_compress.ko 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan 2024-05-27T06:45:01.712 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-27T06:45:01.713 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-27T06:45:01.713 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-27T06:45:01.713 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-27T06:45:01.713 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-27T06:45:01.713 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-27T06:45:01.713 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-27T06:45:01.713 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-27T06:45:01.713 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802 2024-05-27T06:45:01.713 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-27T06:45:01.714 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-27T06:45:01.714 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-27T06:45:01.714 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-27T06:45:01.714 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-27T06:45:01.714 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q 2024-05-27T06:45:01.714 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-27T06:45:01.714 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p 2024-05-27T06:45:01.714 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-27T06:45:01.714 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-27T06:45:01.715 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-27T06:45:01.716 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/failover.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-27T06:45:01.717 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dns_resolver/dns_resolver.ko 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan 2024-05-27T06:45:01.718 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-27T06:45:01.719 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-27T06:45:01.720 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-27T06:45:01.721 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6 2024-05-27T06:45:01.722 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-27T06:45:01.723 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-27T06:45:01.724 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-27T06:45:01.725 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-27T06:45:01.726 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-27T06:45:01.727 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-27T06:45:01.728 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-27T06:45:01.729 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-27T06:45:01.730 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-27T06:45:01.731 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_quota.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-27T06:45:01.732 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tunnel.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-27T06:45:01.733 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-27T06:45:01.734 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-27T06:45:01.735 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-27T06:45:01.736 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-27T06:45:01.737 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-27T06:45:01.738 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-27T06:45:01.739 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-27T06:45:01.740 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-27T06:45:01.741 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc 2024-05-27T06:45:01.742 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-27T06:45:01.743 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/kernel/security/keys/trusted-keys/trusted.ko 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-27T06:45:01.744 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-27T06:45:01.745 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:45:01.749 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-27T06:45:01.749 DEBUG:teuthology.orchestra.run.smithi043:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-27T06:45:01.775 INFO:teuthology.orchestra.run.smithi043.stdout:postinstall scriptlet (using /bin/sh): 2024-05-27T06:45:01.775 INFO:teuthology.orchestra.run.smithi043.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:45:01.775 INFO:teuthology.orchestra.run.smithi043.stdout: /usr/bin/kernel-install add 6.9.0-gb9459a26b088 /lib/modules/6.9.0-gb9459a26b088/vmlinuz 2024-05-27T06:45:01.775 INFO:teuthology.orchestra.run.smithi043.stdout:fi 2024-05-27T06:45:01.775 INFO:teuthology.orchestra.run.smithi043.stdout:for file in vmlinuz System.map config; do 2024-05-27T06:45:01.775 INFO:teuthology.orchestra.run.smithi043.stdout: if ! cmp --silent "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088"; then 2024-05-27T06:45:01.775 INFO:teuthology.orchestra.run.smithi043.stdout: cp "/lib/modules/6.9.0-gb9459a26b088/${file}" "/boot/${file}-6.9.0-gb9459a26b088" 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout: fi 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout:done 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout:if [ -d "/lib/modules/6.9.0-gb9459a26b088/dtb" ] && \ 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout: ! diff -rq "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" >/dev/null 2>&1; then 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout: rm -rf "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout: cp -r "/lib/modules/6.9.0-gb9459a26b088/dtb" "/boot/dtb-6.9.0-gb9459a26b088" 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout:fi 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout:if [ ! -e "/lib/modules/6.9.0-gb9459a26b088/modules.dep" ]; then 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout: /usr/sbin/depmod 6.9.0-gb9459a26b088 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout:fi 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-27T06:45:01.776 INFO:teuthology.orchestra.run.smithi043.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-27T06:45:01.777 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-install remove 6.9.0-gb9459a26b088 2024-05-27T06:45:01.777 INFO:teuthology.orchestra.run.smithi043.stdout:fi 2024-05-27T06:45:01.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo rpm -qi grub2-tools 2024-05-27T06:45:01.856 INFO:teuthology.orchestra.run.smithi043.stdout:Name : grub2-tools 2024-05-27T06:45:01.856 INFO:teuthology.orchestra.run.smithi043.stdout:Epoch : 1 2024-05-27T06:45:01.856 INFO:teuthology.orchestra.run.smithi043.stdout:Version : 2.06 2024-05-27T06:45:01.856 INFO:teuthology.orchestra.run.smithi043.stdout:Release : 68.el9 2024-05-27T06:45:01.856 INFO:teuthology.orchestra.run.smithi043.stdout:Architecture: x86_64 2024-05-27T06:45:01.856 INFO:teuthology.orchestra.run.smithi043.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-27T06:45:01.856 INFO:teuthology.orchestra.run.smithi043.stdout:Group : Unspecified 2024-05-27T06:45:01.856 INFO:teuthology.orchestra.run.smithi043.stdout:Size : 8264254 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:License : GPLv3+ 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:Packager : builder@centos.org 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:Vendor : CentOS 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:Summary : Support tools for GRUB. 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:Description : 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:45:01.857 INFO:teuthology.orchestra.run.smithi043.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-27T06:45:01.858 INFO:teuthology.orchestra.run.smithi043.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-27T06:45:01.858 INFO:teuthology.orchestra.run.smithi043.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-27T06:45:01.858 INFO:teuthology.orchestra.run.smithi043.stdout:hardware devices. 2024-05-27T06:45:01.858 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:45:01.858 INFO:teuthology.orchestra.run.smithi043.stdout:This subpackage provides tools for support of all platforms. 2024-05-27T06:45:01.859 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-27T06:45:01.859 INFO:teuthology.task.kernel:Updating grub on smithi043 to boot 6.9.0-gb9459a26b088 2024-05-27T06:45:01.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-27T06:45:02.396 INFO:teuthology.orchestra.run.smithi043.stderr:Generating grub configuration file ... 2024-05-27T06:45:03.770 INFO:teuthology.orchestra.run.smithi043.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-27T06:45:03.791 INFO:teuthology.orchestra.run.smithi043.stderr:done 2024-05-27T06:45:03.793 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-05-27T06:45:03.810 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.5CKS6yhK5W 2024-05-27T06:45:03.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5CKS6yhK5W 2024-05-27T06:45:03.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.5CKS6yhK5W 2024-05-27T06:45:04.037 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.5CKS6yhK5W is 6KB 2024-05-27T06:45:04.085 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.5CKS6yhK5W 2024-05-27T06:45:04.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo /bin/ls /boot/loader/entries || true 2024-05-27T06:45:04.163 INFO:teuthology.orchestra.run.smithi043.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-27T06:45:04.163 INFO:teuthology.orchestra.run.smithi043.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-27T06:45:04.163 INFO:teuthology.orchestra.run.smithi043.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-27T06:45:04.163 INFO:teuthology.orchestra.run.smithi043.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088.conf 2024-05-27T06:45:04.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-gb9459a26b088 2024-05-27T06:45:04.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo shutdown -r now 2024-05-27T06:45:26.182 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-27T06:45:26.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-05-27T06:45:26.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:45:34.322 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-27T06:45:34.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-05-27T06:45:34.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:45:56.895 DEBUG:teuthology.orchestra.remote:timed out 2024-05-27T06:46:05.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-05-27T06:46:05.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:46:13.411 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-05-27T06:46:13.998 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-05-27T06:46:13.998 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "6.9.0-gb9459a26b088"... 2024-05-27T06:46:13.998 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-05-27T06:46:14.039 INFO:teuthology.orchestra.run.smithi179.stdout:6.9.0-gb9459a26b088 2024-05-27T06:46:14.039 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-27T06:46:14.039 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T06:46:14.039 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-27T06:46:15.041 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-05-27T06:46:15.041 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-27T06:46:15.175 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2024-05-27T06:46:15.226 DEBUG:teuthology.parallel:result is None 2024-05-27T06:46:26.185 DEBUG:teuthology.orchestra.remote:timed out 2024-05-27T06:46:34.524 DEBUG:teuthology.orchestra.remote:timed out 2024-05-27T06:46:35.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-05-27T06:46:35.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:46:42.856 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-05-27T06:46:43.478 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-05-27T06:46:43.478 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.9.0-gb9459a26b088"... 2024-05-27T06:46:43.479 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-05-27T06:46:43.518 INFO:teuthology.orchestra.run.smithi132.stdout:6.9.0-gb9459a26b088 2024-05-27T06:46:43.518 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-27T06:46:43.518 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T06:46:43.518 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-27T06:46:43.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-05-27T06:46:43.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:46:43.941 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-05-27T06:46:44.519 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-27T06:46:44.519 DEBUG:teuthology.orchestra.run.smithi132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-27T06:46:44.546 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-05-27T06:46:44.546 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.9.0-gb9459a26b088"... 2024-05-27T06:46:44.546 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-05-27T06:46:44.583 INFO:teuthology.orchestra.run.smithi043.stdout:6.9.0-gb9459a26b088 2024-05-27T06:46:44.583 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-27T06:46:44.583 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T06:46:44.583 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-27T06:46:44.634 INFO:teuthology.orchestra.run.smithi132.stdout:ttyS1 2024-05-27T06:46:44.678 DEBUG:teuthology.parallel:result is None 2024-05-27T06:46:45.584 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-27T06:46:45.584 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-27T06:46:45.729 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2024-05-27T06:46:45.787 DEBUG:teuthology.parallel:result is None 2024-05-27T06:46:45.787 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-27T06:46:45.797 INFO:teuthology.task.internal:Creating test directory... 2024-05-27T06:46:45.797 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T06:46:45.802 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T06:46:45.808 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T06:46:45.868 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-27T06:46:45.880 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-27T06:46:45.928 INFO:teuthology.task.internal:Creating archive directory... 2024-05-27T06:46:45.929 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T06:46:45.936 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T06:46:45.941 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T06:46:46.010 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-27T06:46:46.019 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-27T06:46:46.019 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T06:46:46.025 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T06:46:46.032 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T06:46:46.095 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:46:46.120 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:46:46.125 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:46:46.127 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:46:46.147 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:46:46.159 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:46:46.163 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-27T06:46:46.173 INFO:teuthology.task.internal:Configuring sudo... 2024-05-27T06:46:46.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T06:46:46.177 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T06:46:46.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T06:46:46.266 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-27T06:46:46.279 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-27T06:46:46.280 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T06:46:46.285 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T06:46:46.289 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T06:46:46.341 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:46:46.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:46:46.539 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:46:46.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:46:46.705 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:46:46.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T06:46:46.768 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:46:46.853 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:46:46.934 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:46:47.005 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:46:47.076 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:46:47.076 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T06:46:47.154 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:46:47.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:46:47.339 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:46:47.421 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:46:47.504 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:46:47.504 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T06:46:47.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2024-05-27T06:46:47.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-05-27T06:46:47.601 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2024-05-27T06:46:47.688 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T06:46:47.703 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T06:46:47.744 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T06:46:48.217 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-27T06:46:48.228 INFO:teuthology.task.internal:Starting timer... 2024-05-27T06:46:48.228 INFO:teuthology.run_tasks:Running task pcp... 2024-05-27T06:46:48.281 INFO:teuthology.run_tasks:Running task selinux... 2024-05-27T06:46:48.329 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-27T06:46:48.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2024-05-27T06:46:48.430 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2024-05-27T06:46:48.434 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-05-27T06:46:48.543 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-05-27T06:46:48.547 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2024-05-27T06:46:48.674 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2024-05-27T06:46:48.679 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-27T06:46:48.679 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2024-05-27T06:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2024-05-27T06:46:48.724 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-05-27T06:46:48.776 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-05-27T06:46:48.776 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2024-05-27T06:46:48.834 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2024-05-27T06:46:48.834 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi043.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi132.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2024-05-27T06:46:48.834 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T06:46:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:46:48.916 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T06:46:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:46:48.986 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T06:46:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:46:49.053 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-27T06:46:49.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2024-05-27T06:46:49.143 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-05-27T06:46:49.208 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2024-05-27T06:46:49.287 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-27T06:46:49.298 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-27T06:46:49.309 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-27T06:46:49.310 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi043.front.sepia.ceph.com,smithi132.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-27T06:53:11.543 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi132.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2024-05-27T06:53:11.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-05-27T06:53:11.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:53:11.663 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-05-27T06:53:11.814 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-05-27T06:53:11.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-05-27T06:53:11.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:53:11.922 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-05-27T06:53:12.078 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-05-27T06:53:12.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-05-27T06:53:12.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:53:12.189 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-05-27T06:53:12.342 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-05-27T06:53:12.344 INFO:teuthology.run_tasks:Running task clock... 2024-05-27T06:53:12.357 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-27T06:53:12.357 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T06:53:12.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T06:53:12.365 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T06:53:12.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T06:53:12.373 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T06:53:12.373 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T06:53:12.458 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T06:53:12.459 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T06:53:12.464 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-27T06:53:12.496 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T06:53:12.503 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T06:53:12.504 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-27T06:53:12.568 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-05-27T06:53:12.578 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2024-05-27T06:53:12.587 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2024-05-27T06:53:12.599 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-05-27T06:53:12.612 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2024-05-27T06:53:12.625 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2024-05-27T06:53:12.639 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T06:53:12.655 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T06:53:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-27T06:53:12.677 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T06:53:12.696 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T06:53:12.716 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-27T06:53:12.799 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-05-27T06:53:12.807 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T06:53:12.807 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-05-27T06:53:12.807 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.808 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.808 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.808 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.835 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-05-27T06:53:12.844 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T06:53:12.844 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-05-27T06:53:12.844 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.844 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.844 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.844 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.872 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-05-27T06:53:12.880 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T06:53:12.880 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-05-27T06:53:12.880 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.880 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.880 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.880 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T06:53:12.881 INFO:teuthology.run_tasks:Running task install... 2024-05-27T06:53:12.897 DEBUG:teuthology.task.install:project ceph 2024-05-27T06:53:12.897 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'}} 2024-05-27T06:53:12.897 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'} 2024-05-27T06:53:12.897 INFO:teuthology.task.install:Using flavor: default 2024-05-27T06:53:12.903 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-27T06:53:12.903 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-27T06:53:12.903 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T06:53:12.904 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:53:12.906 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T06:53:12.906 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:53:12.907 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'tag': None, 'wait_for_package': False} 2024-05-27T06:53:12.907 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:53:13.055 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T06:53:13.056 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T06:53:13.126 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T06:53:13.126 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T06:53:13.144 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/ 2024-05-27T06:53:13.144 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3872.g38f9d900 2024-05-27T06:53:13.166 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T06:53:13.166 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:53:13.166 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T06:53:13.235 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T06:53:13.235 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:53:13.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T06:53:13.256 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T06:53:13.256 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T06:53:13.312 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-27T06:53:13.313 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:53:13.313 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-27T06:53:13.326 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T06:53:13.326 DEBUG:teuthology.orchestra.run.smithi043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T06:53:13.386 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T06:53:13.403 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-27T06:53:13.403 DEBUG:teuthology.orchestra.run.smithi179:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-27T06:53:13.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T06:53:13.495 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T06:53:13.533 DEBUG:teuthology.orchestra.run.smithi179:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-27T06:53:13.571 DEBUG:teuthology.orchestra.run.smithi043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T06:53:13.574 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-05-27T06:53:13.578 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-05-27T06:53:13.641 DEBUG:teuthology.orchestra.run.smithi179:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-27T06:53:13.654 INFO:teuthology.orchestra.run.smithi043.stdout:check_obsoletes = 1 2024-05-27T06:53:13.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum clean all 2024-05-27T06:53:13.723 INFO:teuthology.orchestra.run.smithi179.stdout:check_obsoletes = 1 2024-05-27T06:53:13.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum clean all 2024-05-27T06:53:13.912 INFO:teuthology.orchestra.run.smithi132.stdout:45 files removed 2024-05-27T06:53:13.950 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T06:53:13.981 INFO:teuthology.orchestra.run.smithi043.stdout:45 files removed 2024-05-27T06:53:14.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T06:53:14.057 INFO:teuthology.orchestra.run.smithi179.stdout:45 files removed 2024-05-27T06:53:14.096 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-27T06:53:14.841 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 181 kB/s | 84 kB 00:00 2024-05-27T06:53:14.935 INFO:teuthology.orchestra.run.smithi043.stdout:ceph packages for x86_64 177 kB/s | 84 kB 00:00 2024-05-27T06:53:15.090 INFO:teuthology.orchestra.run.smithi179.stdout:ceph packages for x86_64 152 kB/s | 84 kB 00:00 2024-05-27T06:53:15.408 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 25 kB/s | 12 kB 00:00 2024-05-27T06:53:15.518 INFO:teuthology.orchestra.run.smithi043.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-05-27T06:53:15.548 INFO:teuthology.orchestra.run.smithi179.stdout:ceph noarch packages 33 kB/s | 12 kB 00:00 2024-05-27T06:53:15.876 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-05-27T06:53:16.033 INFO:teuthology.orchestra.run.smithi179.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-05-27T06:53:16.102 INFO:teuthology.orchestra.run.smithi043.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-05-27T06:53:16.656 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.1 MB 00:00 2024-05-27T06:53:17.533 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.1 MB 00:01 2024-05-27T06:53:18.024 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.1 MB 00:02 2024-05-27T06:53:18.591 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-05-27T06:53:21.093 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 19 MB 00:02 2024-05-27T06:53:23.017 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-05-27T06:53:24.643 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 17 kB 00:00 2024-05-27T06:53:25.862 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-05-27T06:53:26.294 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.2 MB 00:01 2024-05-27T06:53:28.086 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 17 kB 00:00 2024-05-27T06:53:30.288 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:02 2024-05-27T06:53:32.012 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-05-27T06:53:34.496 INFO:teuthology.orchestra.run.smithi179.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T06:53:34.498 INFO:teuthology.orchestra.run.smithi179.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T06:53:34.518 INFO:teuthology.orchestra.run.smithi179.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T06:53:34.519 INFO:teuthology.orchestra.run.smithi179.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T06:53:34.520 INFO:teuthology.orchestra.run.smithi179.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T06:53:34.522 INFO:teuthology.orchestra.run.smithi179.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T06:53:34.524 INFO:teuthology.orchestra.run.smithi179.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T06:53:34.527 INFO:teuthology.orchestra.run.smithi179.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T06:53:34.529 INFO:teuthology.orchestra.run.smithi179.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T06:53:34.600 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================= 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================= 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T06:53:34.621 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T06:53:34.622 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T06:53:34.622 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T06:53:34.622 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T06:53:34.622 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T06:53:34.622 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T06:53:34.622 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T06:53:34.622 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T06:53:34.622 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T06:53:34.622 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T06:53:34.622 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T06:53:34.623 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrading: 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T06:53:34.624 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T06:53:34.625 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T06:53:34.626 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T06:53:34.627 INFO:teuthology.orchestra.run.smithi179.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T06:53:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T06:53:34.629 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T06:53:34.630 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T06:53:34.631 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T06:53:34.632 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T06:53:34.633 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T06:53:34.634 INFO:teuthology.orchestra.run.smithi179.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T06:53:34.634 INFO:teuthology.orchestra.run.smithi179.stdout:Installing weak dependencies: 2024-05-27T06:53:34.634 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T06:53:34.634 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:53:34.634 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-05-27T06:53:34.634 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================= 2024-05-27T06:53:34.634 INFO:teuthology.orchestra.run.smithi179.stdout:Install 148 Packages 2024-05-27T06:53:34.634 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrade 11 Packages 2024-05-27T06:53:34.634 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:53:34.635 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 219 M 2024-05-27T06:53:34.635 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-05-27T06:53:35.257 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-05-27T06:53:35.464 INFO:teuthology.orchestra.run.smithi179.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-05-27T06:53:35.698 INFO:teuthology.orchestra.run.smithi179.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 3.6 MB/s | 863 kB 00:00 2024-05-27T06:53:35.767 INFO:teuthology.orchestra.run.smithi179.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 2.2 MB/s | 147 kB 00:00 2024-05-27T06:53:36.115 INFO:teuthology.orchestra.run.smithi179.stdout:(4/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 5.9 MB/s | 5.4 MB 00:00 2024-05-27T06:53:36.182 INFO:teuthology.orchestra.run.smithi179.stdout:(5/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 5.4 MB/s | 2.3 MB 00:00 2024-05-27T06:53:36.316 INFO:teuthology.orchestra.run.smithi179.stdout:(6/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 7.5 MB/s | 1.5 MB 00:00 2024-05-27T06:53:36.696 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-05-27T06:53:36.891 INFO:teuthology.orchestra.run.smithi179.stdout:(7/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 6.9 MB/s | 4.9 MB 00:00 2024-05-27T06:53:37.609 INFO:teuthology.orchestra.run.smithi179.stdout:(8/159): ceph-common-19.0.0-3872.g38f9d900.el9. 8.2 MB/s | 20 MB 00:02 2024-05-27T06:53:37.667 INFO:teuthology.orchestra.run.smithi179.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 448 kB/s | 25 kB 00:00 2024-05-27T06:53:38.122 INFO:teuthology.orchestra.run.smithi179.stdout:(10/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 9.8 MB/s | 18 MB 00:01 2024-05-27T06:53:38.184 INFO:teuthology.orchestra.run.smithi179.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 3.6 MB/s | 224 kB 00:00 2024-05-27T06:53:38.234 INFO:teuthology.orchestra.run.smithi179.stdout:(12/159): cephfs-shell-19.0.0-3872.g38f9d900.el 551 kB/s | 27 kB 00:00 2024-05-27T06:53:38.402 INFO:teuthology.orchestra.run.smithi179.stdout:(13/159): ceph-radosgw-19.0.0-3872.g38f9d900.el 6.7 MB/s | 10 MB 00:01 2024-05-27T06:53:38.427 INFO:teuthology.orchestra.run.smithi179.stdout:(14/159): libcephfs-devel-19.0.0-3872.g38f9d900 166 kB/s | 32 kB 00:00 2024-05-27T06:53:38.485 INFO:teuthology.orchestra.run.smithi179.stdout:(15/159): libcephsqlite-19.0.0-3872.g38f9d900.e 2.7 MB/s | 157 kB 00:00 2024-05-27T06:53:38.527 INFO:teuthology.orchestra.run.smithi179.stdout:(16/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 5.9 MB/s | 755 kB 00:00 2024-05-27T06:53:38.561 INFO:teuthology.orchestra.run.smithi179.stdout:(17/159): librados-devel-19.0.0-3872.g38f9d900. 1.7 MB/s | 127 kB 00:00 2024-05-27T06:53:38.634 INFO:teuthology.orchestra.run.smithi179.stdout:(18/159): libradosstriper1-19.0.0-3872.g38f9d90 4.6 MB/s | 498 kB 00:00 2024-05-27T06:53:38.704 INFO:teuthology.orchestra.run.smithi179.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 661 kB/s | 45 kB 00:00 2024-05-27T06:53:38.779 INFO:teuthology.orchestra.run.smithi179.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 1.8 MB/s | 136 kB 00:00 2024-05-27T06:53:38.863 INFO:teuthology.orchestra.run.smithi179.stdout:(21/159): python3-cephfs-19.0.0-3872.g38f9d900. 1.9 MB/s | 161 kB 00:00 2024-05-27T06:53:38.955 INFO:teuthology.orchestra.run.smithi179.stdout:(22/159): python3-rados-19.0.0-3872.g38f9d900.e 3.4 MB/s | 320 kB 00:00 2024-05-27T06:53:39.055 INFO:teuthology.orchestra.run.smithi179.stdout:(23/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 10 MB/s | 5.2 MB 00:00 2024-05-27T06:53:39.089 INFO:teuthology.orchestra.run.smithi179.stdout:(24/159): python3-rbd-19.0.0-3872.g38f9d900.el9 2.2 MB/s | 300 kB 00:00 2024-05-27T06:53:39.147 INFO:teuthology.orchestra.run.smithi179.stdout:(25/159): python3-rgw-19.0.0-3872.g38f9d900.el9 1.1 MB/s | 99 kB 00:00 2024-05-27T06:53:39.166 INFO:teuthology.orchestra.run.smithi043.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T06:53:39.167 INFO:teuthology.orchestra.run.smithi043.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T06:53:39.181 INFO:teuthology.orchestra.run.smithi179.stdout:(26/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 949 kB/s | 86 kB 00:00 2024-05-27T06:53:39.189 INFO:teuthology.orchestra.run.smithi043.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T06:53:39.190 INFO:teuthology.orchestra.run.smithi043.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T06:53:39.192 INFO:teuthology.orchestra.run.smithi043.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T06:53:39.193 INFO:teuthology.orchestra.run.smithi043.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T06:53:39.196 INFO:teuthology.orchestra.run.smithi043.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T06:53:39.199 INFO:teuthology.orchestra.run.smithi043.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T06:53:39.200 INFO:teuthology.orchestra.run.smithi043.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T06:53:39.248 INFO:teuthology.orchestra.run.smithi179.stdout:(27/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 2.5 MB/s | 169 kB 00:00 2024-05-27T06:53:39.279 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-05-27T06:53:39.294 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-05-27T06:53:39.294 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T06:53:39.295 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T06:53:39.296 INFO:teuthology.orchestra.run.smithi043.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T06:53:39.297 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T06:53:39.298 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T06:53:39.299 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T06:53:39.300 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T06:53:39.301 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T06:53:39.302 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T06:53:39.303 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T06:53:39.304 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T06:53:39.305 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T06:53:39.306 INFO:teuthology.orchestra.run.smithi043.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout:Installing weak dependencies: 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout:Install 148 Packages 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 11 Packages 2024-05-27T06:53:39.307 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:53:39.308 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 219 M 2024-05-27T06:53:39.308 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-05-27T06:53:39.315 INFO:teuthology.orchestra.run.smithi179.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3872.g 518 kB/s | 34 kB 00:00 2024-05-27T06:53:39.407 INFO:teuthology.orchestra.run.smithi179.stdout:(29/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 12 MB/s | 3.0 MB 00:00 2024-05-27T06:53:39.441 INFO:teuthology.orchestra.run.smithi179.stdout:(30/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 1.1 MB/s | 143 kB 00:00 2024-05-27T06:53:39.825 INFO:teuthology.orchestra.run.smithi179.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 8.5 MB/s | 3.5 MB 00:00 2024-05-27T06:53:39.908 INFO:teuthology.orchestra.run.smithi179.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 3.3 MB/s | 274 kB 00:00 2024-05-27T06:53:39.984 INFO:teuthology.orchestra.run.smithi179.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 667 kB/s | 49 kB 00:00 2024-05-27T06:53:40.059 INFO:teuthology.orchestra.run.smithi179.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3872.g3 222 kB/s | 16 kB 00:00 2024-05-27T06:53:40.172 INFO:teuthology.orchestra.run.smithi179.stdout:(35/159): ceph-volume-19.0.0-3872.g38f9d900.el9 2.4 MB/s | 273 kB 00:00 2024-05-27T06:53:40.293 INFO:teuthology.orchestra.run.smithi179.stdout:(36/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 6.2 MB/s | 764 kB 00:00 2024-05-27T06:53:40.391 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.2 MB 00:01 2024-05-27T06:53:40.414 INFO:teuthology.orchestra.run.smithi043.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-05-27T06:53:40.460 INFO:teuthology.orchestra.run.smithi179.stdout:(37/159): ceph-mgr-diskprediction-local-19.0.0- 7.3 MB/s | 7.4 MB 00:01 2024-05-27T06:53:40.486 INFO:teuthology.orchestra.run.smithi179.stdout:(38/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 96 kB/s | 18 kB 00:00 2024-05-27T06:53:40.823 INFO:teuthology.orchestra.run.smithi043.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 2.1 MB/s | 863 kB 00:00 2024-05-27T06:53:40.898 INFO:teuthology.orchestra.run.smithi043.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 1.9 MB/s | 147 kB 00:00 2024-05-27T06:53:41.122 INFO:teuthology.orchestra.run.smithi179.stdout:(39/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 13 MB/s | 43 MB 00:03 2024-05-27T06:53:41.145 INFO:teuthology.orchestra.run.smithi179.stdout:(40/159): fuse-2.9.9-15.el9.x86_64.rpm 118 kB/s | 80 kB 00:00 2024-05-27T06:53:41.253 INFO:teuthology.orchestra.run.smithi179.stdout:(41/159): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-05-27T06:53:41.287 INFO:teuthology.orchestra.run.smithi179.stdout:(42/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-05-27T06:53:41.313 INFO:teuthology.orchestra.run.smithi179.stdout:(43/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-05-27T06:53:41.346 INFO:teuthology.orchestra.run.smithi179.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-05-27T06:53:41.380 INFO:teuthology.orchestra.run.smithi179.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-27T06:53:41.413 INFO:teuthology.orchestra.run.smithi179.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-05-27T06:53:41.447 INFO:teuthology.orchestra.run.smithi179.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-05-27T06:53:41.490 INFO:teuthology.orchestra.run.smithi043.stdout:(4/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 3.8 MB/s | 2.3 MB 00:00 2024-05-27T06:53:41.498 INFO:teuthology.orchestra.run.smithi179.stdout:(48/159): python3-cryptography-36.0.1-4.el9.x86 5.0 MB/s | 1.2 MB 00:00 2024-05-27T06:53:41.791 INFO:teuthology.orchestra.run.smithi043.stdout:(5/159): ceph-common-19.0.0-3872.g38f9d900.el9. 11 MB/s | 20 MB 00:01 2024-05-27T06:53:41.891 INFO:teuthology.orchestra.run.smithi043.stdout:(6/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 3.8 MB/s | 1.5 MB 00:00 2024-05-27T06:53:42.008 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 17 kB 00:00 2024-05-27T06:53:42.158 INFO:teuthology.orchestra.run.smithi043.stdout:(7/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 13 MB/s | 4.9 MB 00:00 2024-05-27T06:53:42.290 INFO:teuthology.orchestra.run.smithi179.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 135 kB/s | 107 kB 00:00 2024-05-27T06:53:42.358 INFO:teuthology.orchestra.run.smithi043.stdout:(8/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 2.3 MB/s | 5.4 MB 00:02 2024-05-27T06:53:42.448 INFO:teuthology.orchestra.run.smithi179.stdout:(50/159): automake-1.16.2-8.el9.noarch.rpm 667 kB/s | 689 kB 00:01 2024-05-27T06:53:42.458 INFO:teuthology.orchestra.run.smithi043.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 255 kB/s | 25 kB 00:00 2024-05-27T06:53:42.507 INFO:teuthology.orchestra.run.smithi179.stdout:(51/159): bison-3.7.4-5.el9.x86_64.rpm 891 kB/s | 943 kB 00:01 2024-05-27T06:53:42.883 INFO:teuthology.orchestra.run.smithi043.stdout:(10/159): ceph-radosgw-19.0.0-3872.g38f9d900.el 14 MB/s | 10 MB 00:00 2024-05-27T06:53:42.942 INFO:teuthology.orchestra.run.smithi043.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 3.8 MB/s | 224 kB 00:00 2024-05-27T06:53:43.009 INFO:teuthology.orchestra.run.smithi043.stdout:(12/159): cephfs-shell-19.0.0-3872.g38f9d900.el 410 kB/s | 27 kB 00:00 2024-05-27T06:53:43.032 INFO:teuthology.orchestra.run.smithi179.stdout:(52/159): elfutils-libelf-devel-0.191-4.el9.x86 29 kB/s | 22 kB 00:00 2024-05-27T06:53:43.076 INFO:teuthology.orchestra.run.smithi043.stdout:(13/159): libcephfs-devel-19.0.0-3872.g38f9d900 482 kB/s | 32 kB 00:00 2024-05-27T06:53:43.174 INFO:teuthology.orchestra.run.smithi179.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 49 kB/s | 33 kB 00:00 2024-05-27T06:53:43.218 INFO:teuthology.orchestra.run.smithi043.stdout:(14/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 5.2 MB/s | 755 kB 00:00 2024-05-27T06:53:43.285 INFO:teuthology.orchestra.run.smithi043.stdout:(15/159): libcephsqlite-19.0.0-3872.g38f9d900.e 2.3 MB/s | 157 kB 00:00 2024-05-27T06:53:43.343 INFO:teuthology.orchestra.run.smithi043.stdout:(16/159): librados-devel-19.0.0-3872.g38f9d900. 2.2 MB/s | 127 kB 00:00 2024-05-27T06:53:43.366 INFO:teuthology.orchestra.run.smithi179.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 342 kB/s | 313 kB 00:00 2024-05-27T06:53:43.435 INFO:teuthology.orchestra.run.smithi043.stdout:(17/159): libradosstriper1-19.0.0-3872.g38f9d90 5.4 MB/s | 498 kB 00:00 2024-05-27T06:53:43.858 INFO:teuthology.orchestra.run.smithi179.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 27 kB/s | 18 kB 00:00 2024-05-27T06:53:44.009 INFO:teuthology.orchestra.run.smithi179.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 17 kB/s | 11 kB 00:00 2024-05-27T06:53:44.026 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-27T06:53:44.035 INFO:teuthology.orchestra.run.smithi043.stdout:(18/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 8.6 MB/s | 5.2 MB 00:00 2024-05-27T06:53:44.278 INFO:teuthology.orchestra.run.smithi043.stdout:(19/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 7.4 MB/s | 18 MB 00:02 2024-05-27T06:53:44.303 INFO:teuthology.orchestra.run.smithi043.stdout:(20/159): python3-ceph-argparse-19.0.0-3872.g38 170 kB/s | 45 kB 00:00 2024-05-27T06:53:44.336 INFO:teuthology.orchestra.run.smithi043.stdout:(21/159): python3-ceph-common-19.0.0-3872.g38f9 2.3 MB/s | 136 kB 00:00 2024-05-27T06:53:44.370 INFO:teuthology.orchestra.run.smithi043.stdout:(22/159): python3-cephfs-19.0.0-3872.g38f9d900. 2.4 MB/s | 161 kB 00:00 2024-05-27T06:53:44.412 INFO:teuthology.orchestra.run.smithi043.stdout:(23/159): python3-rados-19.0.0-3872.g38f9d900.e 4.2 MB/s | 320 kB 00:00 2024-05-27T06:53:44.454 INFO:teuthology.orchestra.run.smithi043.stdout:(24/159): python3-rbd-19.0.0-3872.g38f9d900.el9 3.5 MB/s | 300 kB 00:00 2024-05-27T06:53:44.459 INFO:teuthology.orchestra.run.smithi179.stdout:(57/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.1 MB/s | 3.0 MB 00:01 2024-05-27T06:53:44.487 INFO:teuthology.orchestra.run.smithi043.stdout:(25/159): python3-rgw-19.0.0-3872.g38f9d900.el9 1.3 MB/s | 99 kB 00:00 2024-05-27T06:53:44.513 INFO:teuthology.orchestra.run.smithi043.stdout:(26/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 1.5 MB/s | 86 kB 00:00 2024-05-27T06:53:44.605 INFO:teuthology.orchestra.run.smithi043.stdout:(27/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 1.8 MB/s | 169 kB 00:00 2024-05-27T06:53:44.655 INFO:teuthology.orchestra.run.smithi043.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3872.g 700 kB/s | 34 kB 00:00 2024-05-27T06:53:44.714 INFO:teuthology.orchestra.run.smithi043.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 2.4 MB/s | 143 kB 00:00 2024-05-27T06:53:44.806 INFO:teuthology.orchestra.run.smithi043.stdout:(30/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 9.6 MB/s | 3.0 MB 00:00 2024-05-27T06:53:44.859 INFO:teuthology.orchestra.run.smithi179.stdout:(58/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 160 kB/s | 160 kB 00:00 2024-05-27T06:53:44.910 INFO:teuthology.orchestra.run.smithi179.stdout:(59/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 50 kB/s | 45 kB 00:00 2024-05-27T06:53:45.098 INFO:teuthology.orchestra.run.smithi043.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 9.3 MB/s | 3.5 MB 00:00 2024-05-27T06:53:45.182 INFO:teuthology.orchestra.run.smithi043.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 3.3 MB/s | 274 kB 00:00 2024-05-27T06:53:45.240 INFO:teuthology.orchestra.run.smithi043.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 862 kB/s | 49 kB 00:00 2024-05-27T06:53:45.299 INFO:teuthology.orchestra.run.smithi043.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3872.g3 286 kB/s | 16 kB 00:00 2024-05-27T06:53:45.374 INFO:teuthology.orchestra.run.smithi043.stdout:(35/159): ceph-volume-19.0.0-3872.g38f9d900.el9 3.6 MB/s | 273 kB 00:00 2024-05-27T06:53:45.524 INFO:teuthology.orchestra.run.smithi043.stdout:(36/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 5.0 MB/s | 764 kB 00:00 2024-05-27T06:53:45.583 INFO:teuthology.orchestra.run.smithi043.stdout:(37/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 317 kB/s | 18 kB 00:00 2024-05-27T06:53:45.602 INFO:teuthology.orchestra.run.smithi179.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 37 kB/s | 25 kB 00:00 2024-05-27T06:53:45.685 INFO:teuthology.orchestra.run.smithi179.stdout:(61/159): librdkafka-1.6.1-102.el9.x86_64.rpm 540 kB/s | 662 kB 00:01 2024-05-27T06:53:45.742 INFO:teuthology.orchestra.run.smithi043.stdout:(38/159): ceph-mgr-diskprediction-local-19.0.0- 7.9 MB/s | 7.4 MB 00:00 2024-05-27T06:53:45.969 INFO:teuthology.orchestra.run.smithi179.stdout:(62/159): libtool-2.4.6-45.el9.x86_64.rpm 527 kB/s | 582 kB 00:01 2024-05-27T06:53:46.419 INFO:teuthology.orchestra.run.smithi179.stdout:(63/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 58 kB/s | 47 kB 00:00 2024-05-27T06:53:46.584 INFO:teuthology.orchestra.run.smithi043.stdout:(39/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 10 MB/s | 43 MB 00:04 2024-05-27T06:53:46.745 INFO:teuthology.orchestra.run.smithi179.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 276 kB/s | 292 kB 00:01 2024-05-27T06:53:46.937 INFO:teuthology.orchestra.run.smithi179.stdout:(65/159): lua-5.4.4-4.el9.x86_64.rpm 207 kB/s | 188 kB 00:00 2024-05-27T06:53:46.951 INFO:teuthology.orchestra.run.smithi043.stdout:(40/159): fuse-2.9.9-15.el9.x86_64.rpm 59 kB/s | 80 kB 00:01 2024-05-27T06:53:47.079 INFO:teuthology.orchestra.run.smithi179.stdout:(66/159): openblas-0.3.21-2.el9.x86_64.rpm 53 kB/s | 35 kB 00:00 2024-05-27T06:53:47.293 INFO:teuthology.orchestra.run.smithi043.stdout:(41/159): libgfortran-11.4.1-3.el9.x86_64.rpm 518 kB/s | 803 kB 00:01 2024-05-27T06:53:47.459 INFO:teuthology.orchestra.run.smithi043.stdout:(42/159): libquadmath-11.4.1-3.el9.x86_64.rpm 224 kB/s | 195 kB 00:00 2024-05-27T06:53:47.893 INFO:teuthology.orchestra.run.smithi043.stdout:(43/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 269 kB/s | 253 kB 00:00 2024-05-27T06:53:48.087 INFO:teuthology.orchestra.run.smithi179.stdout:(67/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.5 MB/s | 4.7 MB 00:01 2024-05-27T06:53:48.287 INFO:teuthology.orchestra.run.smithi179.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 1.3 MB/s | 1.6 MB 00:01 2024-05-27T06:53:48.368 INFO:teuthology.orchestra.run.smithi043.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 117 kB/s | 106 kB 00:00 2024-05-27T06:53:48.435 INFO:teuthology.orchestra.run.smithi043.stdout:(45/159): python3-cryptography-36.0.1-4.el9.x86 1.1 MB/s | 1.2 MB 00:01 2024-05-27T06:53:48.471 INFO:teuthology.orchestra.run.smithi179.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2024-05-27T06:53:48.769 INFO:teuthology.orchestra.run.smithi043.stdout:(46/159): python3-pycparser-2.20-6.el9.noarch.r 155 kB/s | 135 kB 00:00 2024-05-27T06:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 689 kB 00:00 2024-05-27T06:53:49.053 INFO:teuthology.orchestra.run.smithi043.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 11 MB/s | 943 kB 00:00 2024-05-27T06:53:49.103 INFO:teuthology.orchestra.run.smithi043.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-05-27T06:53:49.121 INFO:teuthology.orchestra.run.smithi179.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 160 kB/s | 133 kB 00:00 2024-05-27T06:53:49.153 INFO:teuthology.orchestra.run.smithi043.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 441 kB/s | 22 kB 00:00 2024-05-27T06:53:49.238 INFO:teuthology.orchestra.run.smithi179.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 793 kB/s | 911 kB 00:01 2024-05-27T06:53:49.279 INFO:teuthology.orchestra.run.smithi043.stdout:(51/159): python3-requests-2.25.1-8.el9.noarch. 137 kB/s | 125 kB 00:00 2024-05-27T06:53:49.313 INFO:teuthology.orchestra.run.smithi043.stdout:(52/159): python3-urllib3-1.26.5-5.el9.noarch.r 245 kB/s | 215 kB 00:00 2024-05-27T06:53:49.346 INFO:teuthology.orchestra.run.smithi043.stdout:(53/159): flex-2.6.4-9.el9.x86_64.rpm 1.6 MB/s | 313 kB 00:00 2024-05-27T06:53:49.380 INFO:teuthology.orchestra.run.smithi043.stdout:(54/159): flexiblas-3.0.4-8.el9.x86_64.rpm 328 kB/s | 33 kB 00:00 2024-05-27T06:53:49.405 INFO:teuthology.orchestra.run.smithi043.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 315 kB/s | 18 kB 00:00 2024-05-27T06:53:49.430 INFO:teuthology.orchestra.run.smithi043.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 226 kB/s | 11 kB 00:00 2024-05-27T06:53:49.464 INFO:teuthology.orchestra.run.smithi043.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-05-27T06:53:49.489 INFO:teuthology.orchestra.run.smithi043.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 785 kB/s | 45 kB 00:00 2024-05-27T06:53:49.540 INFO:teuthology.orchestra.run.smithi043.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 662 kB 00:00 2024-05-27T06:53:49.590 INFO:teuthology.orchestra.run.smithi043.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 519 kB/s | 25 kB 00:00 2024-05-27T06:53:49.632 INFO:teuthology.orchestra.run.smithi043.stdout:(61/159): libtool-2.4.6-45.el9.x86_64.rpm 4.0 MB/s | 582 kB 00:00 2024-05-27T06:53:49.657 INFO:teuthology.orchestra.run.smithi043.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 719 kB/s | 47 kB 00:00 2024-05-27T06:53:49.691 INFO:teuthology.orchestra.run.smithi043.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-05-27T06:53:49.725 INFO:teuthology.orchestra.run.smithi043.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-05-27T06:53:49.742 INFO:teuthology.orchestra.run.smithi043.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 715 kB/s | 35 kB 00:00 2024-05-27T06:53:49.825 INFO:teuthology.orchestra.run.smithi043.stdout:(66/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.9 MB/s | 3.0 MB 00:00 2024-05-27T06:53:49.916 INFO:teuthology.orchestra.run.smithi179.stdout:(72/159): postgresql-server-13.14-1.el9.x86_64. 4.0 MB/s | 5.8 MB 00:01 2024-05-27T06:53:49.926 INFO:teuthology.orchestra.run.smithi043.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-05-27T06:53:50.031 INFO:teuthology.orchestra.run.smithi179.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 111 kB/s | 87 kB 00:00 2024-05-27T06:53:50.059 INFO:teuthology.orchestra.run.smithi043.stdout:(68/159): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-05-27T06:53:50.152 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-05-27T06:53:50.159 INFO:teuthology.orchestra.run.smithi043.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-05-27T06:53:50.201 INFO:teuthology.orchestra.run.smithi043.stdout:(70/159): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-05-27T06:53:50.231 INFO:teuthology.orchestra.run.smithi179.stdout:(74/159): protobuf-3.14.0-13.el9.x86_64.rpm 933 kB/s | 1.0 MB 00:01 2024-05-27T06:53:50.235 INFO:teuthology.orchestra.run.smithi043.stdout:(71/159): postgresql-private-libs-13.14-1.el9.x 762 kB/s | 133 kB 00:00 2024-05-27T06:53:50.285 INFO:teuthology.orchestra.run.smithi043.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-05-27T06:53:50.336 INFO:teuthology.orchestra.run.smithi043.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-05-27T06:53:50.386 INFO:teuthology.orchestra.run.smithi043.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-05-27T06:53:50.444 INFO:teuthology.orchestra.run.smithi043.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 4.2 MB/s | 244 kB 00:00 2024-05-27T06:53:50.503 INFO:teuthology.orchestra.run.smithi043.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-05-27T06:53:50.553 INFO:teuthology.orchestra.run.smithi043.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 524 kB/s | 26 kB 00:00 2024-05-27T06:53:50.679 INFO:teuthology.orchestra.run.smithi043.stdout:(78/159): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-05-27T06:53:50.704 INFO:teuthology.orchestra.run.smithi043.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 128 kB/s | 19 kB 00:00 2024-05-27T06:53:50.746 INFO:teuthology.orchestra.run.smithi043.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-05-27T06:53:50.771 INFO:teuthology.orchestra.run.smithi043.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 524 kB/s | 35 kB 00:00 2024-05-27T06:53:50.824 INFO:teuthology.orchestra.run.smithi179.stdout:(75/159): python3-colorama-0.4.6-3.el9.noarch.r 77 kB/s | 61 kB 00:00 2024-05-27T06:53:50.905 INFO:teuthology.orchestra.run.smithi043.stdout:(82/159): python3-babel-2.9.1-2.el9.noarch.rpm 9.6 MB/s | 6.0 MB 00:00 2024-05-27T06:53:50.930 INFO:teuthology.orchestra.run.smithi043.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-05-27T06:53:50.981 INFO:teuthology.orchestra.run.smithi043.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 77 kB 00:00 2024-05-27T06:53:51.014 INFO:teuthology.orchestra.run.smithi043.stdout:(85/159): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-05-27T06:53:51.039 INFO:teuthology.orchestra.run.smithi043.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 728 kB/s | 42 kB 00:00 2024-05-27T06:53:51.073 INFO:teuthology.orchestra.run.smithi043.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-05-27T06:53:51.131 INFO:teuthology.orchestra.run.smithi043.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-05-27T06:53:51.140 INFO:teuthology.orchestra.run.smithi179.stdout:(76/159): python3-devel-3.9.18-3.el9.x86_64.rpm 269 kB/s | 244 kB 00:00 2024-05-27T06:53:51.165 INFO:teuthology.orchestra.run.smithi043.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 279 kB 00:00 2024-05-27T06:53:51.191 INFO:teuthology.orchestra.run.smithi043.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 864 kB/s | 51 kB 00:00 2024-05-27T06:53:51.216 INFO:teuthology.orchestra.run.smithi043.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-05-27T06:53:51.249 INFO:teuthology.orchestra.run.smithi179.stdout:(77/159): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 6.0 MB 00:01 2024-05-27T06:53:51.266 INFO:teuthology.orchestra.run.smithi043.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 848 kB/s | 42 kB 00:00 2024-05-27T06:53:51.392 INFO:teuthology.orchestra.run.smithi043.stdout:(93/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.9 MB/s | 5.1 MB 00:00 2024-05-27T06:53:51.417 INFO:teuthology.orchestra.run.smithi043.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 295 kB/s | 44 kB 00:00 2024-05-27T06:53:51.451 INFO:teuthology.orchestra.run.smithi043.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-05-27T06:53:51.476 INFO:teuthology.orchestra.run.smithi043.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 622 kB/s | 36 kB 00:00 2024-05-27T06:53:51.562 INFO:teuthology.orchestra.run.smithi043.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 440 kB/s | 47 kB 00:00 2024-05-27T06:53:51.593 INFO:teuthology.orchestra.run.smithi043.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 2.6 MB/s | 305 kB 00:00 2024-05-27T06:53:51.618 INFO:teuthology.orchestra.run.smithi043.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-05-27T06:53:51.643 INFO:teuthology.orchestra.run.smithi043.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.9 MB/s | 94 kB 00:00 2024-05-27T06:53:51.669 INFO:teuthology.orchestra.run.smithi043.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-05-27T06:53:51.694 INFO:teuthology.orchestra.run.smithi043.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-05-27T06:53:51.841 INFO:teuthology.orchestra.run.smithi179.stdout:(78/159): python3-jinja2-2.11.3-6.el9.noarch.rp 244 kB/s | 248 kB 00:01 2024-05-27T06:53:51.908 INFO:teuthology.orchestra.run.smithi179.stdout:(79/159): python3-jsonpatch-1.21-16.el9.noarch. 33 kB/s | 26 kB 00:00 2024-05-27T06:53:51.967 INFO:teuthology.orchestra.run.smithi179.stdout:(80/159): python3-jsonpointer-2.0-4.el9.noarch. 27 kB/s | 19 kB 00:00 2024-05-27T06:53:51.970 INFO:teuthology.orchestra.run.smithi043.stdout:(103/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-05-27T06:53:51.995 INFO:teuthology.orchestra.run.smithi043.stdout:(104/159): lua-devel-5.4.4-4.el9.x86_64.rpm 69 kB/s | 22 kB 00:00 2024-05-27T06:53:52.487 INFO:teuthology.orchestra.run.smithi043.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-05-27T06:53:52.577 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T06:53:52.579 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-27T06:53:52.599 INFO:teuthology.orchestra.run.smithi132.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-27T06:53:52.600 INFO:teuthology.orchestra.run.smithi132.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-27T06:53:52.602 INFO:teuthology.orchestra.run.smithi132.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-27T06:53:52.603 INFO:teuthology.orchestra.run.smithi132.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-27T06:53:52.606 INFO:teuthology.orchestra.run.smithi132.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-27T06:53:52.609 INFO:teuthology.orchestra.run.smithi132.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-27T06:53:52.610 INFO:teuthology.orchestra.run.smithi132.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-27T06:53:52.642 INFO:teuthology.orchestra.run.smithi179.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 47 kB/s | 35 kB 00:00 2024-05-27T06:53:52.681 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 6.6 k 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.4 M 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 863 k 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 147 k 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 1.5 M 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 143 k 2024-05-27T06:53:52.706 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 3.5 M 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 7.4 M 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 49 k 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 10 M 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 43 M 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 273 k 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 764 k 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 224 k 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-shell x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 27 k 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-top noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 18 k 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-27T06:53:52.707 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 32 k 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 755 k 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 127 k 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 161 k 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 320 k 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 300 k 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 99 k 2024-05-27T06:53:52.708 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 86 k 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.0 M 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 169 k 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-27T06:53:52.709 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.3 M 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 3.1 M 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 20 M 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 34 k 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 2.3 M 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 274 k 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 4.9 M 2024-05-27T06:53:52.710 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 18 M 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3872.g38f9d900.el9 ceph-noarch 16 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 25 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 157 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-27T06:53:52.711 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 498 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 5.2 M 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-27T06:53:52.712 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 45 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:19.0.0-3872.g38f9d900.el9 ceph 136 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-27T06:53:52.713 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-27T06:53:52.714 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-27T06:53:52.715 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-27T06:53:52.716 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-27T06:53:52.717 INFO:teuthology.orchestra.run.smithi132.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout:Install 148 Packages 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 11 Packages 2024-05-27T06:53:52.718 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:53:52.719 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 219 M 2024-05-27T06:53:52.719 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-05-27T06:53:52.767 INFO:teuthology.orchestra.run.smithi179.stdout:(82/159): python3-mako-1.1.4-6.el9.noarch.rpm 186 kB/s | 172 kB 00:00 2024-05-27T06:53:52.779 INFO:teuthology.orchestra.run.smithi043.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 67 kB/s | 19 kB 00:00 2024-05-27T06:53:52.954 INFO:teuthology.orchestra.run.smithi043.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 322 kB/s | 309 kB 00:00 2024-05-27T06:53:53.071 INFO:teuthology.orchestra.run.smithi043.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-05-27T06:53:53.188 INFO:teuthology.orchestra.run.smithi043.stdout:(109/159): liboath-2.6.7-2.el9.x86_64.rpm 419 kB/s | 49 kB 00:00 2024-05-27T06:53:53.222 INFO:teuthology.orchestra.run.smithi043.stdout:(110/159): abseil-cpp-20211102.0-3.el9.x86_64.r 438 kB/s | 548 kB 00:01 2024-05-27T06:53:53.397 INFO:teuthology.orchestra.run.smithi043.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 324 kB/s | 67 kB 00:00 2024-05-27T06:53:53.468 INFO:teuthology.orchestra.run.smithi179.stdout:(83/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.4 MB/s | 5.1 MB 00:01 2024-05-27T06:53:53.514 INFO:teuthology.orchestra.run.smithi043.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 519 kB/s | 151 kB 00:00 2024-05-27T06:53:53.593 INFO:teuthology.orchestra.run.smithi179.stdout:(84/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 264 kB/s | 250 kB 00:00 2024-05-27T06:53:53.693 INFO:teuthology.orchestra.run.smithi179.stdout:(85/159): python3-oauthlib-3.1.1-5.el9.noarch.r 240 kB/s | 222 kB 00:00 2024-05-27T06:53:53.853 INFO:teuthology.orchestra.run.smithi132.stdout:(1/159): ceph-19.0.0-3872.g38f9d900.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-05-27T06:53:54.302 INFO:teuthology.orchestra.run.smithi179.stdout:(86/159): python3-packaging-20.9-5.el9.noarch.r 93 kB/s | 77 kB 00:00 2024-05-27T06:53:54.356 INFO:teuthology.orchestra.run.smithi043.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 652 kB/s | 548 kB 00:00 2024-05-27T06:53:54.379 INFO:teuthology.orchestra.run.smithi132.stdout:(2/159): ceph-fuse-19.0.0-3872.g38f9d900.el9.x8 1.6 MB/s | 863 kB 00:00 2024-05-27T06:53:54.446 INFO:teuthology.orchestra.run.smithi132.stdout:(3/159): ceph-immutable-object-cache-19.0.0-387 2.2 MB/s | 147 kB 00:00 2024-05-27T06:53:54.473 INFO:teuthology.orchestra.run.smithi043.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 374 kB/s | 43 kB 00:00 2024-05-27T06:53:54.538 INFO:teuthology.orchestra.run.smithi179.stdout:(87/159): python3-prettytable-0.7.2-27.el9.noar 45 kB/s | 42 kB 00:00 2024-05-27T06:53:54.590 INFO:teuthology.orchestra.run.smithi043.stdout:(115/159): python3-cachetools-4.2.4-1.el9.noarc 277 kB/s | 32 kB 00:00 2024-05-27T06:53:54.640 INFO:teuthology.orchestra.run.smithi043.stdout:(116/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 675 kB/s | 839 kB 00:01 2024-05-27T06:53:54.707 INFO:teuthology.orchestra.run.smithi043.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 129 kB/s | 15 kB 00:00 2024-05-27T06:53:54.949 INFO:teuthology.orchestra.run.smithi043.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 558 kB/s | 172 kB 00:00 2024-05-27T06:53:54.969 INFO:teuthology.orchestra.run.smithi179.stdout:(88/159): python3-protobuf-3.14.0-13.el9.noarch 223 kB/s | 269 kB 00:01 2024-05-27T06:53:55.054 INFO:teuthology.orchestra.run.smithi132.stdout:(4/159): ceph-base-19.0.0-3872.g38f9d900.el9.x8 3.5 MB/s | 5.4 MB 00:01 2024-05-27T06:53:55.166 INFO:teuthology.orchestra.run.smithi132.stdout:(5/159): ceph-mds-19.0.0-3872.g38f9d900.el9.x86 3.2 MB/s | 2.3 MB 00:00 2024-05-27T06:53:55.274 INFO:teuthology.orchestra.run.smithi043.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 633 kB/s | 358 kB 00:00 2024-05-27T06:53:55.294 INFO:teuthology.orchestra.run.smithi179.stdout:(89/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 160 kB/s | 159 kB 00:00 2024-05-27T06:53:55.305 INFO:teuthology.orchestra.run.smithi132.stdout:(6/159): ceph-mgr-19.0.0-3872.g38f9d900.el9.x86 6.0 MB/s | 1.5 MB 00:00 2024-05-27T06:53:55.433 INFO:teuthology.orchestra.run.smithi043.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 628 kB/s | 303 kB 00:00 2024-05-27T06:53:55.511 INFO:teuthology.orchestra.run.smithi179.stdout:(90/159): python3-pyasn1-modules-0.4.8-6.el9.no 287 kB/s | 279 kB 00:00 2024-05-27T06:53:55.675 INFO:teuthology.orchestra.run.smithi043.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 514 kB/s | 206 kB 00:00 2024-05-27T06:53:55.678 INFO:teuthology.orchestra.run.smithi179.stdout:(91/159): python3-pytz-2021.1-5.el9.noarch.rpm 72 kB/s | 51 kB 00:00 2024-05-27T06:53:55.955 INFO:teuthology.orchestra.run.smithi132.stdout:(7/159): ceph-mon-19.0.0-3872.g38f9d900.el9.x86 6.2 MB/s | 4.9 MB 00:00 2024-05-27T06:53:55.991 INFO:teuthology.orchestra.run.smithi043.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 457 kB/s | 144 kB 00:00 2024-05-27T06:53:56.095 INFO:teuthology.orchestra.run.smithi179.stdout:(92/159): python3-requests-oauthlib-1.3.0-12.el 67 kB/s | 54 kB 00:00 2024-05-27T06:53:56.108 INFO:teuthology.orchestra.run.smithi043.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-05-27T06:53:56.226 INFO:teuthology.orchestra.run.smithi043.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2024-05-27T06:53:56.359 INFO:teuthology.orchestra.run.smithi043.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 175 kB/s | 23 kB 00:00 2024-05-27T06:53:56.463 INFO:teuthology.orchestra.run.smithi179.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 53 kB/s | 42 kB 00:00 2024-05-27T06:53:56.485 INFO:teuthology.orchestra.run.smithi043.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 156 kB/s | 19 kB 00:00 2024-05-27T06:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout:(127/159): python3-jaraco-text-3.2.0-6.el9.noar 158 kB/s | 20 kB 00:00 2024-05-27T06:53:56.772 INFO:teuthology.orchestra.run.smithi132.stdout:(8/159): ceph-common-19.0.0-3872.g38f9d900.el9. 6.0 MB/s | 20 MB 00:03 2024-05-27T06:53:56.838 INFO:teuthology.orchestra.run.smithi179.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 60 kB/s | 44 kB 00:00 2024-05-27T06:53:56.847 INFO:teuthology.orchestra.run.smithi132.stdout:(9/159): ceph-selinux-19.0.0-3872.g38f9d900.el9 341 kB/s | 25 kB 00:00 2024-05-27T06:53:57.097 INFO:teuthology.orchestra.run.smithi132.stdout:(10/159): ceph-osd-19.0.0-3872.g38f9d900.el9.x8 9.9 MB/s | 18 MB 00:01 2024-05-27T06:53:57.207 INFO:teuthology.orchestra.run.smithi132.stdout:(11/159): cephfs-mirror-19.0.0-3872.g38f9d900.e 2.0 MB/s | 224 kB 00:00 2024-05-27T06:53:57.290 INFO:teuthology.orchestra.run.smithi132.stdout:(12/159): cephfs-shell-19.0.0-3872.g38f9d900.el 331 kB/s | 27 kB 00:00 2024-05-27T06:53:57.355 INFO:teuthology.orchestra.run.smithi179.stdout:(95/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.8 MB/s | 16 MB 00:01 2024-05-27T06:53:57.365 INFO:teuthology.orchestra.run.smithi132.stdout:(13/159): libcephfs-devel-19.0.0-3872.g38f9d900 428 kB/s | 32 kB 00:00 2024-05-27T06:53:57.388 INFO:teuthology.orchestra.run.smithi179.stdout:(96/159): qatlib-24.02.0-1.el9.x86_64.rpm 239 kB/s | 221 kB 00:00 2024-05-27T06:53:57.524 INFO:teuthology.orchestra.run.smithi132.stdout:(14/159): ceph-radosgw-19.0.0-3872.g38f9d900.el 6.5 MB/s | 10 MB 00:01 2024-05-27T06:53:57.582 INFO:teuthology.orchestra.run.smithi132.stdout:(15/159): libcephfs2-19.0.0-3872.g38f9d900.el9. 3.4 MB/s | 755 kB 00:00 2024-05-27T06:53:57.649 INFO:teuthology.orchestra.run.smithi132.stdout:(16/159): libcephsqlite-19.0.0-3872.g38f9d900.e 1.2 MB/s | 157 kB 00:00 2024-05-27T06:53:57.691 INFO:teuthology.orchestra.run.smithi132.stdout:(17/159): librados-devel-19.0.0-3872.g38f9d900. 1.1 MB/s | 127 kB 00:00 2024-05-27T06:53:57.747 INFO:teuthology.orchestra.run.smithi179.stdout:(97/159): qatlib-service-24.02.0-1.el9.x86_64.r 40 kB/s | 36 kB 00:00 2024-05-27T06:53:58.044 INFO:teuthology.orchestra.run.smithi043.stdout:(128/159): python3-kubernetes-26.1.0-2.el9.noar 730 kB/s | 1.0 MB 00:01 2024-05-27T06:53:58.090 INFO:teuthology.orchestra.run.smithi179.stdout:(98/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 65 kB/s | 47 kB 00:00 2024-05-27T06:53:58.109 INFO:teuthology.orchestra.run.smithi132.stdout:(18/159): libradosstriper1-19.0.0-3872.g38f9d90 1.1 MB/s | 498 kB 00:00 2024-05-27T06:53:58.160 INFO:teuthology.orchestra.run.smithi043.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 400 kB/s | 46 kB 00:00 2024-05-27T06:53:58.217 INFO:teuthology.orchestra.run.smithi132.stdout:(19/159): python3-ceph-argparse-19.0.0-3872.g38 420 kB/s | 45 kB 00:00 2024-05-27T06:53:58.319 INFO:teuthology.orchestra.run.smithi043.stdout:(130/159): python3-grpcio-1.46.7-10.el9.x86_64. 723 kB/s | 2.0 MB 00:02 2024-05-27T06:53:58.322 INFO:teuthology.orchestra.run.smithi179.stdout:(99/159): socat-1.7.4.1-5.el9.x86_64.rpm 327 kB/s | 305 kB 00:00 2024-05-27T06:53:58.326 INFO:teuthology.orchestra.run.smithi132.stdout:(20/159): python3-ceph-common-19.0.0-3872.g38f9 1.2 MB/s | 136 kB 00:00 2024-05-27T06:53:58.369 INFO:teuthology.orchestra.run.smithi043.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 379 kB/s | 79 kB 00:00 2024-05-27T06:53:58.436 INFO:teuthology.orchestra.run.smithi043.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 496 kB/s | 58 kB 00:00 2024-05-27T06:53:58.459 INFO:teuthology.orchestra.run.smithi132.stdout:(21/159): python3-cephfs-19.0.0-3872.g38f9d900. 1.2 MB/s | 161 kB 00:00 2024-05-27T06:53:58.473 INFO:teuthology.orchestra.run.smithi179.stdout:(100/159): uuid-1.6.2-55.el9.x86_64.rpm 80 kB/s | 58 kB 00:00 2024-05-27T06:53:58.553 INFO:teuthology.orchestra.run.smithi043.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 141 kB/s | 16 kB 00:00 2024-05-27T06:53:58.643 INFO:teuthology.orchestra.run.smithi132.stdout:(22/159): librgw2-19.0.0-3872.g38f9d900.el9.x86 5.4 MB/s | 5.2 MB 00:00 2024-05-27T06:53:58.677 INFO:teuthology.orchestra.run.smithi132.stdout:(23/159): python3-rados-19.0.0-3872.g38f9d900.e 1.4 MB/s | 320 kB 00:00 2024-05-27T06:53:58.727 INFO:teuthology.orchestra.run.smithi132.stdout:(24/159): python3-rbd-19.0.0-3872.g38f9d900.el9 3.6 MB/s | 300 kB 00:00 2024-05-27T06:53:58.761 INFO:teuthology.orchestra.run.smithi132.stdout:(25/159): python3-rgw-19.0.0-3872.g38f9d900.el9 1.2 MB/s | 99 kB 00:00 2024-05-27T06:53:58.762 INFO:teuthology.orchestra.run.smithi043.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 695 kB/s | 272 kB 00:00 2024-05-27T06:53:58.787 INFO:teuthology.orchestra.run.smithi043.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 386 kB/s | 90 kB 00:00 2024-05-27T06:53:58.803 INFO:teuthology.orchestra.run.smithi132.stdout:(26/159): rbd-fuse-19.0.0-3872.g38f9d900.el9.x8 1.1 MB/s | 86 kB 00:00 2024-05-27T06:53:58.886 INFO:teuthology.orchestra.run.smithi132.stdout:(27/159): rbd-nbd-19.0.0-3872.g38f9d900.el9.x86 2.0 MB/s | 169 kB 00:00 2024-05-27T06:53:58.887 INFO:teuthology.orchestra.run.smithi043.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 227 kB/s | 28 kB 00:00 2024-05-27T06:53:58.915 INFO:teuthology.orchestra.run.smithi179.stdout:(101/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 113 kB/s | 94 kB 00:00 2024-05-27T06:53:58.936 INFO:teuthology.orchestra.run.smithi132.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3872.g 696 kB/s | 34 kB 00:00 2024-05-27T06:53:58.979 INFO:teuthology.orchestra.run.smithi043.stdout:(137/159): libarrow-9.0.0-11.el9.x86_64.rpm 729 kB/s | 4.4 MB 00:06 2024-05-27T06:53:59.004 INFO:teuthology.orchestra.run.smithi132.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3872.g38f9d90 2.1 MB/s | 143 kB 00:00 2024-05-27T06:53:59.005 INFO:teuthology.orchestra.run.smithi043.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 142 kB/s | 31 kB 00:00 2024-05-27T06:53:59.007 INFO:teuthology.orchestra.run.smithi179.stdout:(102/159): lua-devel-5.4.4-4.el9.x86_64.rpm 256 kB/s | 22 kB 00:00 2024-05-27T06:53:59.097 INFO:teuthology.orchestra.run.smithi043.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 507 kB/s | 59 kB 00:00 2024-05-27T06:53:59.122 INFO:teuthology.orchestra.run.smithi043.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 308 kB/s | 36 kB 00:00 2024-05-27T06:53:59.133 INFO:teuthology.orchestra.run.smithi179.stdout:(103/159): protobuf-compiler-3.14.0-13.el9.x86_ 6.8 MB/s | 863 kB 00:00 2024-05-27T06:53:59.189 INFO:teuthology.orchestra.run.smithi043.stdout:(141/159): python3-routes-2.5.1-5.el9.noarch.rp 628 kB/s | 188 kB 00:00 2024-05-27T06:53:59.214 INFO:teuthology.orchestra.run.smithi043.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 438 kB/s | 51 kB 00:00 2024-05-27T06:53:59.241 INFO:teuthology.orchestra.run.smithi179.stdout:(104/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 75 kB/s | 64 kB 00:00 2024-05-27T06:53:59.283 INFO:teuthology.orchestra.run.smithi179.stdout:(105/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 56 kB/s | 45 kB 00:00 2024-05-27T06:53:59.337 INFO:teuthology.orchestra.run.smithi132.stdout:(30/159): rbd-mirror-19.0.0-3872.g38f9d900.el9. 5.3 MB/s | 3.0 MB 00:00 2024-05-27T06:53:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 431 kB/s | 90 kB 00:00 2024-05-27T06:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout:(144/159): python3-webob-1.8.7-6.el9.noarch.rpm 665 kB/s | 230 kB 00:00 2024-05-27T06:53:59.542 INFO:teuthology.orchestra.run.smithi179.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-05-27T06:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 83 kB/s | 20 kB 00:00 2024-05-27T06:53:59.713 INFO:teuthology.orchestra.run.smithi132.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3872.g38f9d 5.0 MB/s | 3.5 MB 00:00 2024-05-27T06:53:59.792 INFO:teuthology.orchestra.run.smithi179.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 563 kB/s | 309 kB 00:00 2024-05-27T06:53:59.813 INFO:teuthology.orchestra.run.smithi132.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3872.g38 2.7 MB/s | 274 kB 00:00 2024-05-27T06:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 527 kB/s | 192 kB 00:00 2024-05-27T06:53:59.834 INFO:teuthology.orchestra.run.smithi179.stdout:(108/159): abseil-cpp-20211102.0-3.el9.x86_64.r 782 kB/s | 548 kB 00:00 2024-05-27T06:53:59.866 INFO:teuthology.orchestra.run.smithi043.stdout:(147/159): python3-werkzeug-2.0.3-3.el9.1.noarc 657 kB/s | 427 kB 00:00 2024-05-27T06:53:59.897 INFO:teuthology.orchestra.run.smithi132.stdout:(33/159): ceph-mgr-rook-19.0.0-3872.g38f9d900.e 597 kB/s | 49 kB 00:00 2024-05-27T06:53:59.902 INFO:teuthology.orchestra.run.smithi179.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-05-27T06:53:59.952 INFO:teuthology.orchestra.run.smithi179.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 417 kB/s | 49 kB 00:00 2024-05-27T06:53:59.972 INFO:teuthology.orchestra.run.smithi132.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3872.g3 222 kB/s | 16 kB 00:00 2024-05-27T06:53:59.994 INFO:teuthology.orchestra.run.smithi179.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 737 kB/s | 67 kB 00:00 2024-05-27T06:54:00.044 INFO:teuthology.orchestra.run.smithi179.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-05-27T06:54:00.097 INFO:teuthology.orchestra.run.smithi132.stdout:(35/159): ceph-volume-19.0.0-3872.g38f9d900.el9 2.1 MB/s | 273 kB 00:00 2024-05-27T06:54:00.169 INFO:teuthology.orchestra.run.smithi179.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-05-27T06:54:00.228 INFO:teuthology.orchestra.run.smithi179.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.5 MB/s | 839 kB 00:00 2024-05-27T06:54:00.319 INFO:teuthology.orchestra.run.smithi179.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 292 kB/s | 43 kB 00:00 2024-05-27T06:54:00.356 INFO:teuthology.orchestra.run.smithi132.stdout:(36/159): cephadm-19.0.0-3872.g38f9d900.el9.noa 2.9 MB/s | 764 kB 00:00 2024-05-27T06:54:00.379 INFO:teuthology.orchestra.run.smithi179.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 215 kB/s | 32 kB 00:00 2024-05-27T06:54:00.488 INFO:teuthology.orchestra.run.smithi179.stdout:(117/159): libarrow-9.0.0-11.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-05-27T06:54:00.538 INFO:teuthology.orchestra.run.smithi179.stdout:(118/159): python3-certifi-2021.10.8-2.el9.noar 69 kB/s | 15 kB 00:00 2024-05-27T06:54:00.571 INFO:teuthology.orchestra.run.smithi179.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 897 kB/s | 172 kB 00:00 2024-05-27T06:54:00.605 INFO:teuthology.orchestra.run.smithi179.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-05-27T06:54:00.639 INFO:teuthology.orchestra.run.smithi179.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-05-27T06:54:00.672 INFO:teuthology.orchestra.run.smithi179.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-05-27T06:54:00.683 INFO:teuthology.orchestra.run.smithi043.stdout:(148/159): librados2-19.0.0-3872.g38f9d900.el9. 3.9 MB/s | 3.3 MB 00:00 2024-05-27T06:54:00.731 INFO:teuthology.orchestra.run.smithi179.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-05-27T06:54:00.765 INFO:teuthology.orchestra.run.smithi132.stdout:(37/159): cephfs-top-19.0.0-3872.g38f9d900.el9. 45 kB/s | 18 kB 00:00 2024-05-27T06:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout:(149/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 3.4 MB/s | 3.1 MB 00:00 2024-05-27T06:54:00.781 INFO:teuthology.orchestra.run.smithi179.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-05-27T06:54:00.874 INFO:teuthology.orchestra.run.smithi179.stdout:(125/159): python3-grpcio-1.46.7-10.el9.x86_64. 7.6 MB/s | 2.0 MB 00:00 2024-05-27T06:54:00.916 INFO:teuthology.orchestra.run.smithi179.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 96 kB/s | 18 kB 00:00 2024-05-27T06:54:00.948 INFO:teuthology.orchestra.run.smithi132.stdout:(38/159): ceph-mgr-diskprediction-local-19.0.0- 4.6 MB/s | 7.4 MB 00:01 2024-05-27T06:54:00.966 INFO:teuthology.orchestra.run.smithi179.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 126 kB/s | 23 kB 00:00 2024-05-27T06:54:01.008 INFO:teuthology.orchestra.run.smithi179.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 146 kB/s | 19 kB 00:00 2024-05-27T06:54:01.075 INFO:teuthology.orchestra.run.smithi179.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 183 kB/s | 20 kB 00:00 2024-05-27T06:54:01.125 INFO:teuthology.orchestra.run.smithi179.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 397 kB/s | 46 kB 00:00 2024-05-27T06:54:01.175 INFO:teuthology.orchestra.run.smithi179.stdout:(131/159): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-05-27T06:54:01.193 INFO:teuthology.orchestra.run.smithi179.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 676 kB/s | 79 kB 00:00 2024-05-27T06:54:01.243 INFO:teuthology.orchestra.run.smithi179.stdout:(133/159): python3-natsort-7.1.1-5.el9.noarch.r 494 kB/s | 58 kB 00:00 2024-05-27T06:54:01.277 INFO:teuthology.orchestra.run.smithi179.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-05-27T06:54:01.343 INFO:teuthology.orchestra.run.smithi179.stdout:(135/159): python3-portend-3.1.0-2.el9.noarch.r 109 kB/s | 16 kB 00:00 2024-05-27T06:54:01.377 INFO:teuthology.orchestra.run.smithi179.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 673 kB/s | 90 kB 00:00 2024-05-27T06:54:01.419 INFO:teuthology.orchestra.run.smithi179.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 199 kB/s | 28 kB 00:00 2024-05-27T06:54:01.461 INFO:teuthology.orchestra.run.smithi179.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 262 kB/s | 31 kB 00:00 2024-05-27T06:54:01.490 INFO:teuthology.orchestra.run.smithi132.stdout:(39/159): ceph-test-19.0.0-3872.g38f9d900.el9.x 9.3 MB/s | 43 MB 00:04 2024-05-27T06:54:01.495 INFO:teuthology.orchestra.run.smithi179.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-05-27T06:54:01.515 INFO:teuthology.orchestra.run.smithi132.stdout:(40/159): fuse-2.9.9-15.el9.x86_64.rpm 107 kB/s | 80 kB 00:00 2024-05-27T06:54:01.537 INFO:teuthology.orchestra.run.smithi179.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 507 kB/s | 59 kB 00:00 2024-05-27T06:54:01.579 INFO:teuthology.orchestra.run.smithi179.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 306 kB/s | 36 kB 00:00 2024-05-27T06:54:01.658 INFO:teuthology.orchestra.run.smithi043.stdout:(150/159): thrift-0.15.0-2.el9.x86_64.rpm 803 kB/s | 1.6 MB 00:02 2024-05-27T06:54:01.680 INFO:teuthology.orchestra.run.smithi179.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 277 kB/s | 51 kB 00:00 2024-05-27T06:54:01.722 INFO:teuthology.orchestra.run.smithi179.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 634 kB/s | 90 kB 00:00 2024-05-27T06:54:01.755 INFO:teuthology.orchestra.run.smithi179.stdout:(144/159): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-05-27T06:54:01.789 INFO:teuthology.orchestra.run.smithi179.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.9 MB/s | 427 kB 00:00 2024-05-27T06:54:01.831 INFO:teuthology.orchestra.run.smithi179.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-05-27T06:54:01.890 INFO:teuthology.orchestra.run.smithi179.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-05-27T06:54:01.958 INFO:teuthology.orchestra.run.smithi132.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 574 kB/s | 253 kB 00:00 2024-05-27T06:54:02.000 INFO:teuthology.orchestra.run.smithi132.stdout:(42/159): libquadmath-11.4.1-3.el9.x86_64.rpm 385 kB/s | 195 kB 00:00 2024-05-27T06:54:02.012 INFO:teuthology.orchestra.run.smithi043.stdout:(151/159): NetworkManager-1.47.90-1.el9.x86_64. 1.7 MB/s | 2.3 MB 00:01 2024-05-27T06:54:02.084 INFO:teuthology.orchestra.run.smithi043.stdout:(152/159): NetworkManager-libnm-1.47.90-1.el9.x 1.4 MB/s | 1.8 MB 00:01 2024-05-27T06:54:02.117 INFO:teuthology.orchestra.run.smithi132.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 916 kB/s | 106 kB 00:00 2024-05-27T06:54:02.159 INFO:teuthology.orchestra.run.smithi132.stdout:(44/159): libgfortran-11.4.1-3.el9.x86_64.rpm 664 kB/s | 803 kB 00:01 2024-05-27T06:54:02.251 INFO:teuthology.orchestra.run.smithi132.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 1.0 MB/s | 135 kB 00:00 2024-05-27T06:54:02.284 INFO:teuthology.orchestra.run.smithi132.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-05-27T06:54:02.315 INFO:teuthology.orchestra.run.smithi179.stdout:(148/159): librados2-19.0.0-3872.g38f9d900.el9. 6.9 MB/s | 3.3 MB 00:00 2024-05-27T06:54:02.335 INFO:teuthology.orchestra.run.smithi132.stdout:(47/159): python3-cryptography-36.0.1-4.el9.x86 3.3 MB/s | 1.2 MB 00:00 2024-05-27T06:54:02.342 INFO:teuthology.orchestra.run.smithi043.stdout:(153/159): NetworkManager-team-1.47.90-1.el9.x8 61 kB/s | 41 kB 00:00 2024-05-27T06:54:02.368 INFO:teuthology.orchestra.run.smithi132.stdout:(48/159): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-05-27T06:54:02.415 INFO:teuthology.orchestra.run.smithi179.stdout:(149/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 5.9 MB/s | 3.1 MB 00:00 2024-05-27T06:54:02.516 INFO:teuthology.orchestra.run.smithi179.stdout:(150/159): thrift-0.15.0-2.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2024-05-27T06:54:02.527 INFO:teuthology.orchestra.run.smithi132.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 679 kB/s | 107 kB 00:00 2024-05-27T06:54:02.632 INFO:teuthology.orchestra.run.smithi132.stdout:(50/159): automake-1.16.2-8.el9.noarch.rpm 1.9 MB/s | 689 kB 00:00 2024-05-27T06:54:02.649 INFO:teuthology.orchestra.run.smithi179.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 312 kB/s | 41 kB 00:00 2024-05-27T06:54:02.661 INFO:teuthology.orchestra.run.smithi132.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 163 kB/s | 22 kB 00:00 2024-05-27T06:54:02.708 INFO:teuthology.orchestra.run.smithi179.stdout:(152/159): NetworkManager-1.47.90-1.el9.x86_64. 5.9 MB/s | 2.3 MB 00:00 2024-05-27T06:54:02.728 INFO:teuthology.orchestra.run.smithi132.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 495 kB/s | 33 kB 00:00 2024-05-27T06:54:02.770 INFO:teuthology.orchestra.run.smithi132.stdout:(53/159): flex-2.6.4-9.el9.x86_64.rpm 2.2 MB/s | 313 kB 00:00 2024-05-27T06:54:02.775 INFO:teuthology.orchestra.run.smithi179.stdout:(153/159): NetworkManager-libnm-1.47.90-1.el9.x 5.0 MB/s | 1.8 MB 00:00 2024-05-27T06:54:02.812 INFO:teuthology.orchestra.run.smithi132.stdout:(54/159): bison-3.7.4-5.el9.x86_64.rpm 1.9 MB/s | 943 kB 00:00 2024-05-27T06:54:02.825 INFO:teuthology.orchestra.run.smithi179.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 316 kB/s | 37 kB 00:00 2024-05-27T06:54:02.837 INFO:teuthology.orchestra.run.smithi132.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 274 kB/s | 18 kB 00:00 2024-05-27T06:54:02.877 INFO:teuthology.orchestra.run.smithi179.stdout:(155/159): NetworkManager-tui-1.47.90-1.el9.x86 1.1 MB/s | 248 kB 00:00 2024-05-27T06:54:02.901 INFO:teuthology.orchestra.run.smithi043.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 45 kB/s | 37 kB 00:00 2024-05-27T06:54:02.910 INFO:teuthology.orchestra.run.smithi179.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.5 MB/s | 207 kB 00:00 2024-05-27T06:54:02.921 INFO:teuthology.orchestra.run.smithi132.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-05-27T06:54:02.944 INFO:teuthology.orchestra.run.smithi179.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 2.2 MB/s | 260 kB 00:00 2024-05-27T06:54:02.946 INFO:teuthology.orchestra.run.smithi132.stdout:(57/159): libaio-devel-0.3.111-13.el9.x86_64.rp 83 kB/s | 11 kB 00:00 2024-05-27T06:54:02.993 INFO:teuthology.orchestra.run.smithi043.stdout:(155/159): NetworkManager-tui-1.47.90-1.el9.x86 253 kB/s | 248 kB 00:00 2024-05-27T06:54:02.996 INFO:teuthology.orchestra.run.smithi132.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 607 kB/s | 45 kB 00:00 2024-05-27T06:54:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 9.6 MB/s | 1.3 MB 00:00 2024-05-27T06:54:03.135 INFO:teuthology.orchestra.run.smithi179.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-05-27T06:54:03.142 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-05-27T06:54:03.143 INFO:teuthology.orchestra.run.smithi179.stdout:Total 7.7 MB/s | 219 MB 00:28 2024-05-27T06:54:03.172 INFO:teuthology.orchestra.run.smithi132.stdout:(59/159): libtool-2.4.6-45.el9.x86_64.rpm 3.3 MB/s | 582 kB 00:00 2024-05-27T06:54:03.243 INFO:teuthology.orchestra.run.smithi043.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 230 kB/s | 207 kB 00:00 2024-05-27T06:54:03.247 INFO:teuthology.orchestra.run.smithi132.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 342 kB/s | 25 kB 00:00 2024-05-27T06:54:03.281 INFO:teuthology.orchestra.run.smithi132.stdout:(61/159): librdkafka-1.6.1-102.el9.x86_64.rpm 1.9 MB/s | 662 kB 00:00 2024-05-27T06:54:03.323 INFO:teuthology.orchestra.run.smithi132.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 636 kB/s | 47 kB 00:00 2024-05-27T06:54:03.373 INFO:teuthology.orchestra.run.smithi132.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-05-27T06:54:03.407 INFO:teuthology.orchestra.run.smithi132.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-05-27T06:54:03.466 INFO:teuthology.orchestra.run.smithi132.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 386 kB/s | 35 kB 00:00 2024-05-27T06:54:03.649 INFO:teuthology.orchestra.run.smithi132.stdout:(66/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.2 MB/s | 3.0 MB 00:00 2024-05-27T06:54:03.902 INFO:teuthology.orchestra.run.smithi043.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 260 kB/s | 260 kB 00:00 2024-05-27T06:54:04.116 INFO:teuthology.orchestra.run.smithi132.stdout:(67/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 6.8 MB/s | 4.4 MB 00:00 2024-05-27T06:54:04.152 INFO:teuthology.orchestra.run.smithi043.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 1.2 MB/s | 1.3 MB 00:01 2024-05-27T06:54:04.166 INFO:teuthology.orchestra.run.smithi132.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-05-27T06:54:04.292 INFO:teuthology.orchestra.run.smithi132.stdout:(69/159): openblas-openmp-0.3.21-2.el9.x86_64.r 5.4 MB/s | 4.7 MB 00:00 2024-05-27T06:54:04.334 INFO:teuthology.orchestra.run.smithi132.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 801 kB/s | 133 kB 00:00 2024-05-27T06:54:04.393 INFO:teuthology.orchestra.run.smithi132.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-05-27T06:54:04.460 INFO:teuthology.orchestra.run.smithi043.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.0 MB/s | 2.4 MB 00:01 2024-05-27T06:54:04.468 INFO:teuthology.orchestra.run.smithi132.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 87 kB 00:00 2024-05-27T06:54:04.469 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-05-27T06:54:04.469 INFO:teuthology.orchestra.run.smithi043.stdout:Total 8.7 MB/s | 219 MB 00:25 2024-05-27T06:54:04.627 INFO:teuthology.orchestra.run.smithi132.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-05-27T06:54:04.702 INFO:teuthology.orchestra.run.smithi132.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 825 kB/s | 61 kB 00:00 2024-05-27T06:54:04.802 INFO:teuthology.orchestra.run.smithi132.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-05-27T06:54:04.903 INFO:teuthology.orchestra.run.smithi132.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-05-27T06:54:04.961 INFO:teuthology.orchestra.run.smithi132.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 446 kB/s | 26 kB 00:00 2024-05-27T06:54:05.020 INFO:teuthology.orchestra.run.smithi132.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 333 kB/s | 19 kB 00:00 2024-05-27T06:54:05.195 INFO:teuthology.orchestra.run.smithi132.stdout:(79/159): postgresql-server-13.14-1.el9.x86_64. 6.4 MB/s | 5.8 MB 00:00 2024-05-27T06:54:05.229 INFO:teuthology.orchestra.run.smithi132.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 828 kB/s | 172 kB 00:00 2024-05-27T06:54:05.254 INFO:teuthology.orchestra.run.smithi132.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 602 kB/s | 35 kB 00:00 2024-05-27T06:54:05.371 INFO:teuthology.orchestra.run.smithi132.stdout:(82/159): python3-babel-2.9.1-2.el9.noarch.rpm 6.6 MB/s | 6.0 MB 00:00 2024-05-27T06:54:05.405 INFO:teuthology.orchestra.run.smithi132.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-05-27T06:54:05.464 INFO:teuthology.orchestra.run.smithi132.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-05-27T06:54:05.514 INFO:teuthology.orchestra.run.smithi132.stdout:(85/159): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 222 kB 00:00 2024-05-27T06:54:05.539 INFO:teuthology.orchestra.run.smithi132.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 568 kB/s | 42 kB 00:00 2024-05-27T06:54:05.606 INFO:teuthology.orchestra.run.smithi132.stdout:(87/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-05-27T06:54:05.682 INFO:teuthology.orchestra.run.smithi132.stdout:(88/159): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 279 kB 00:00 2024-05-27T06:54:05.715 INFO:teuthology.orchestra.run.smithi132.stdout:(89/159): python3-protobuf-3.14.0-13.el9.noarch 1.3 MB/s | 269 kB 00:00 2024-05-27T06:54:05.740 INFO:teuthology.orchestra.run.smithi132.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 881 kB/s | 51 kB 00:00 2024-05-27T06:54:05.774 INFO:teuthology.orchestra.run.smithi132.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 928 kB/s | 54 kB 00:00 2024-05-27T06:54:05.841 INFO:teuthology.orchestra.run.smithi132.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 632 kB/s | 42 kB 00:00 2024-05-27T06:54:05.933 INFO:teuthology.orchestra.run.smithi132.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 486 kB/s | 44 kB 00:00 2024-05-27T06:54:06.050 INFO:teuthology.orchestra.run.smithi132.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 1.9 MB/s | 221 kB 00:00 2024-05-27T06:54:06.109 INFO:teuthology.orchestra.run.smithi132.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 625 kB/s | 36 kB 00:00 2024-05-27T06:54:06.168 INFO:teuthology.orchestra.run.smithi132.stdout:(96/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 823 kB/s | 47 kB 00:00 2024-05-27T06:54:06.327 INFO:teuthology.orchestra.run.smithi132.stdout:(97/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.7 MB/s | 5.1 MB 00:01 2024-05-27T06:54:06.368 INFO:teuthology.orchestra.run.smithi132.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 305 kB 00:00 2024-05-27T06:54:06.371 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-05-27T06:54:06.394 INFO:teuthology.orchestra.run.smithi132.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 876 kB/s | 58 kB 00:00 2024-05-27T06:54:06.427 INFO:teuthology.orchestra.run.smithi132.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-05-27T06:54:06.452 INFO:teuthology.orchestra.run.smithi132.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-27T06:54:06.486 INFO:teuthology.orchestra.run.smithi132.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 772 kB/s | 45 kB 00:00 2024-05-27T06:54:06.537 INFO:teuthology.orchestra.run.smithi132.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 269 kB/s | 22 kB 00:00 2024-05-27T06:54:06.622 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-05-27T06:54:06.622 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-05-27T06:54:06.768 INFO:teuthology.orchestra.run.smithi132.stdout:(104/159): protobuf-compiler-3.14.0-13.el9.x86_ 3.0 MB/s | 863 kB 00:00 2024-05-27T06:54:07.471 INFO:teuthology.orchestra.run.smithi132.stdout:(105/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.4 MB/s | 16 MB 00:01 2024-05-27T06:54:07.513 INFO:teuthology.orchestra.run.smithi132.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 415 kB/s | 309 kB 00:00 2024-05-27T06:54:07.546 INFO:teuthology.orchestra.run.smithi132.stdout:(107/159): abseil-cpp-20211102.0-3.el9.x86_64.r 544 kB/s | 548 kB 00:01 2024-05-27T06:54:07.680 INFO:teuthology.orchestra.run.smithi132.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 196 kB/s | 26 kB 00:00 2024-05-27T06:54:07.744 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-05-27T06:54:07.763 INFO:teuthology.orchestra.run.smithi132.stdout:(109/159): grpc-data-1.46.7-10.el9.noarch.rpm 67 kB/s | 19 kB 00:00 2024-05-27T06:54:07.797 INFO:teuthology.orchestra.run.smithi132.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 418 kB/s | 49 kB 00:00 2024-05-27T06:54:07.939 INFO:teuthology.orchestra.run.smithi132.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-05-27T06:54:08.006 INFO:teuthology.orchestra.run.smithi132.stdout:(112/159): libunwind-1.6.2-1.el9.x86_64.rpm 279 kB/s | 67 kB 00:00 2024-05-27T06:54:08.018 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-05-27T06:54:08.018 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-05-27T06:54:08.331 INFO:teuthology.orchestra.run.smithi132.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 1.7 MB/s | 548 kB 00:00 2024-05-27T06:54:08.431 INFO:teuthology.orchestra.run.smithi132.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.7 MB/s | 839 kB 00:00 2024-05-27T06:54:08.448 INFO:teuthology.orchestra.run.smithi132.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 374 kB/s | 43 kB 00:00 2024-05-27T06:54:08.549 INFO:teuthology.orchestra.run.smithi132.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 277 kB/s | 32 kB 00:00 2024-05-27T06:54:08.566 INFO:teuthology.orchestra.run.smithi132.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 129 kB/s | 15 kB 00:00 2024-05-27T06:54:08.683 INFO:teuthology.orchestra.run.smithi132.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-05-27T06:54:08.733 INFO:teuthology.orchestra.run.smithi132.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 2.1 MB/s | 358 kB 00:00 2024-05-27T06:54:08.875 INFO:teuthology.orchestra.run.smithi132.stdout:(120/159): python3-google-auth-2.29.0-1.el9.noa 1.4 MB/s | 206 kB 00:00 2024-05-27T06:54:08.909 INFO:teuthology.orchestra.run.smithi132.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.3 MB/s | 303 kB 00:00 2024-05-27T06:54:09.043 INFO:teuthology.orchestra.run.smithi132.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-05-27T06:54:09.160 INFO:teuthology.orchestra.run.smithi132.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-05-27T06:54:09.277 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-05-27T06:54:09.277 INFO:teuthology.orchestra.run.smithi132.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2024-05-27T06:54:09.277 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-05-27T06:54:09.402 INFO:teuthology.orchestra.run.smithi132.stdout:(125/159): python3-grpcio-1.46.7-10.el9.x86_64. 3.9 MB/s | 2.0 MB 00:00 2024-05-27T06:54:09.419 INFO:teuthology.orchestra.run.smithi132.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 164 kB/s | 23 kB 00:00 2024-05-27T06:54:09.511 INFO:teuthology.orchestra.run.smithi132.stdout:(127/159): python3-jaraco-functools-3.5.0-2.el9 180 kB/s | 19 kB 00:00 2024-05-27T06:54:09.536 INFO:teuthology.orchestra.run.smithi132.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 169 kB/s | 20 kB 00:00 2024-05-27T06:54:09.653 INFO:teuthology.orchestra.run.smithi132.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 399 kB/s | 46 kB 00:00 2024-05-27T06:54:09.745 INFO:teuthology.orchestra.run.smithi132.stdout:(130/159): python3-kubernetes-26.1.0-2.el9.noar 4.4 MB/s | 1.0 MB 00:00 2024-05-27T06:54:09.937 INFO:teuthology.orchestra.run.smithi132.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 278 kB/s | 79 kB 00:00 2024-05-27T06:54:09.962 INFO:teuthology.orchestra.run.smithi132.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 266 kB/s | 58 kB 00:00 2024-05-27T06:54:10.071 INFO:teuthology.orchestra.run.smithi132.stdout:(133/159): libarrow-9.0.0-11.el9.x86_64.rpm 1.7 MB/s | 4.4 MB 00:02 2024-05-27T06:54:10.096 INFO:teuthology.orchestra.run.smithi132.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 123 kB/s | 16 kB 00:00 2024-05-27T06:54:10.214 INFO:teuthology.orchestra.run.smithi132.stdout:(135/159): python3-pecan-1.4.2-3.el9.noarch.rpm 990 kB/s | 272 kB 00:00 2024-05-27T06:54:10.239 INFO:teuthology.orchestra.run.smithi132.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 200 kB/s | 28 kB 00:00 2024-05-27T06:54:10.264 INFO:teuthology.orchestra.run.smithi132.stdout:(137/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 469 kB/s | 90 kB 00:00 2024-05-27T06:54:10.331 INFO:teuthology.orchestra.run.smithi132.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 264 kB/s | 31 kB 00:00 2024-05-27T06:54:10.365 INFO:teuthology.orchestra.run.smithi132.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-05-27T06:54:10.382 INFO:teuthology.orchestra.run.smithi132.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2024-05-27T06:54:10.457 INFO:teuthology.orchestra.run.smithi132.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 287 kB/s | 36 kB 00:00 2024-05-27T06:54:10.482 INFO:teuthology.orchestra.run.smithi132.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 439 kB/s | 51 kB 00:00 2024-05-27T06:54:10.516 INFO:teuthology.orchestra.run.smithi132.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-05-27T06:54:10.591 INFO:teuthology.orchestra.run.smithi132.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 672 kB/s | 90 kB 00:00 2024-05-27T06:54:10.625 INFO:teuthology.orchestra.run.smithi132.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-05-27T06:54:10.642 INFO:teuthology.orchestra.run.smithi132.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 159 kB/s | 20 kB 00:00 2024-05-27T06:54:10.708 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-05-27T06:54:10.709 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-05-27T06:54:10.725 INFO:teuthology.orchestra.run.smithi132.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-05-27T06:54:10.976 INFO:teuthology.orchestra.run.smithi132.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 4.5 MB/s | 1.6 MB 00:00 2024-05-27T06:54:11.418 INFO:teuthology.orchestra.run.smithi132.stdout:(149/159): librados2-19.0.0-3872.g38f9d900.el9. 4.3 MB/s | 3.3 MB 00:00 2024-05-27T06:54:11.493 INFO:teuthology.orchestra.run.smithi132.stdout:(150/159): librbd1-19.0.0-3872.g38f9d900.el9.x8 4.0 MB/s | 3.1 MB 00:00 2024-05-27T06:54:11.877 INFO:teuthology.orchestra.run.smithi132.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 108 kB/s | 41 kB 00:00 2024-05-27T06:54:11.969 INFO:teuthology.orchestra.run.smithi132.stdout:(152/159): NetworkManager-1.47.90-1.el9.x86_64. 2.3 MB/s | 2.3 MB 00:00 2024-05-27T06:54:12.087 INFO:teuthology.orchestra.run.smithi132.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 315 kB/s | 37 kB 00:00 2024-05-27T06:54:12.220 INFO:teuthology.orchestra.run.smithi132.stdout:(154/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.5 MB/s | 207 kB 00:00 2024-05-27T06:54:12.254 INFO:teuthology.orchestra.run.smithi132.stdout:(155/159): NetworkManager-tui-1.47.90-1.el9.x86 661 kB/s | 248 kB 00:00 2024-05-27T06:54:12.354 INFO:teuthology.orchestra.run.smithi132.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 1.9 MB/s | 260 kB 00:00 2024-05-27T06:54:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:(157/159): NetworkManager-libnm-1.47.90-1.el9.x 1.6 MB/s | 1.8 MB 00:01 2024-05-27T06:54:12.755 INFO:teuthology.orchestra.run.smithi132.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 2.7 MB/s | 1.3 MB 00:00 2024-05-27T06:54:13.438 INFO:teuthology.orchestra.run.smithi132.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.2 MB/s | 2.4 MB 00:01 2024-05-27T06:54:13.444 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-05-27T06:54:13.445 INFO:teuthology.orchestra.run.smithi132.stdout:Total 11 MB/s | 219 MB 00:20 2024-05-27T06:54:15.889 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-05-27T06:54:16.018 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T06:54:16.110 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T06:54:16.219 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T06:54:16.278 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T06:54:16.312 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T06:54:16.405 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T06:54:16.675 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-05-27T06:54:16.846 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T06:54:16.931 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-05-27T06:54:16.931 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-05-27T06:54:16.952 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T06:54:17.118 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T06:54:17.188 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T06:54:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T06:54:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-05-27T06:54:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T06:54:17.461 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T06:54:17.494 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T06:54:17.576 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T06:54:17.650 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T06:54:17.699 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T06:54:17.735 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T06:54:17.824 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T06:54:17.832 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T06:54:17.911 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T06:54:17.935 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T06:54:18.123 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T06:54:18.153 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T06:54:18.264 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T06:54:18.286 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T06:54:18.296 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T06:54:18.370 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T06:54:18.417 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T06:54:18.614 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T06:54:18.643 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T06:54:18.774 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T06:54:18.801 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T06:54:18.818 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T06:54:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T06:54:19.116 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T06:54:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T06:54:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T06:54:19.340 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T06:54:19.368 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T06:54:19.450 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T06:54:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T06:54:19.567 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T06:54:19.618 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T06:54:19.663 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T06:54:19.681 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-05-27T06:54:19.682 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-05-27T06:54:19.693 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T06:54:19.743 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T06:54:19.774 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T06:54:20.090 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T06:54:20.092 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T06:54:20.289 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T06:54:20.323 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T06:54:20.360 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T06:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T06:54:20.480 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T06:54:20.545 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T06:54:20.610 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T06:54:20.626 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T06:54:20.657 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T06:54:20.697 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T06:54:20.721 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T06:54:20.814 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T06:54:20.884 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T06:54:20.916 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T06:54:20.923 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T06:54:20.971 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T06:54:21.011 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T06:54:21.077 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T06:54:21.119 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T06:54:21.172 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T06:54:21.210 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T06:54:21.329 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T06:54:21.349 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T06:54:21.389 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T06:54:21.497 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T06:54:21.556 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T06:54:21.592 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T06:54:21.649 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T06:54:21.670 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T06:54:21.731 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T06:54:21.771 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T06:54:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T06:54:21.917 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T06:54:21.970 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T06:54:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T06:54:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T06:54:22.205 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T06:54:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T06:54:22.300 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T06:54:22.417 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T06:54:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T06:54:22.463 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T06:54:22.513 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T06:54:22.517 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T06:54:22.549 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T06:54:22.553 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T06:54:22.584 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T06:54:22.642 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T06:54:22.677 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T06:54:22.689 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T06:54:22.743 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T06:54:22.759 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T06:54:22.857 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T06:54:22.959 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T06:54:23.026 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T06:54:23.040 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T06:54:23.101 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T06:54:23.153 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T06:54:23.221 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T06:54:23.305 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T06:54:23.344 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T06:54:23.380 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T06:54:23.445 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T06:54:23.491 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T06:54:23.552 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T06:54:23.610 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T06:54:23.806 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T06:54:23.841 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T06:54:23.851 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T06:54:23.904 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T06:54:23.954 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T06:54:24.001 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T06:54:24.055 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T06:54:24.062 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T06:54:24.101 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T06:54:24.121 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T06:54:24.147 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T06:54:24.178 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T06:54:24.183 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T06:54:24.216 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T06:54:24.254 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T06:54:24.406 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T06:54:24.472 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T06:54:24.554 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T06:54:24.625 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T06:54:24.664 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T06:54:24.684 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T06:54:24.695 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T06:54:24.750 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T06:54:24.754 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T06:54:24.816 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T06:54:24.852 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T06:54:24.869 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T06:54:24.891 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T06:54:24.918 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T06:54:24.954 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T06:54:24.964 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T06:54:25.021 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T06:54:25.023 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T06:54:25.071 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T06:54:25.091 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T06:54:25.135 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T06:54:25.136 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T06:54:25.203 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T06:54:25.289 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T06:54:25.351 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T06:54:25.395 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T06:54:25.404 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T06:54:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T06:54:25.465 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T06:54:25.509 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T06:54:25.629 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T06:54:25.636 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T06:54:25.680 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T06:54:25.690 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T06:54:25.725 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T06:54:25.761 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T06:54:25.793 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T06:54:25.965 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T06:54:26.177 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T06:54:26.218 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T06:54:26.247 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T06:54:26.296 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T06:54:26.362 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T06:54:26.417 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T06:54:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T06:54:26.510 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T06:54:26.526 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-05-27T06:54:26.559 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T06:54:26.621 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-27T06:54:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T06:54:26.651 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T06:54:26.683 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T06:54:26.694 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-27T06:54:26.706 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T06:54:26.740 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T06:54:26.794 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-27T06:54:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T06:54:26.855 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T06:54:26.864 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-27T06:54:26.887 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T06:54:26.897 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-27T06:54:26.914 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T06:54:26.931 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T06:54:26.967 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T06:54:26.999 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-27T06:54:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T06:54:27.095 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T06:54:27.134 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T06:54:27.167 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T06:54:27.228 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T06:54:27.358 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T06:54:27.381 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T06:54:27.448 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-27T06:54:27.487 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T06:54:27.579 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T06:54:27.797 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T06:54:27.821 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T06:54:27.945 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T06:54:27.979 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T06:54:28.061 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-27T06:54:28.234 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T06:54:28.293 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T06:54:28.327 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T06:54:28.333 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T06:54:28.432 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-27T06:54:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T06:54:28.476 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T06:54:28.503 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-27T06:54:28.519 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T06:54:28.527 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T06:54:28.574 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T06:54:28.679 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T06:54:28.686 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T06:54:28.712 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T06:54:28.728 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T06:54:28.797 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 15/170 2024-05-27T06:54:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T06:54:28.903 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-27T06:54:28.978 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T06:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T06:54:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T06:54:29.196 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T06:54:29.512 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-27T06:54:29.759 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T06:54:29.850 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-27T06:54:30.013 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T06:54:30.052 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T06:54:30.115 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T06:54:30.143 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T06:54:30.203 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T06:54:30.245 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T06:54:30.281 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T06:54:30.316 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T06:54:30.424 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-27T06:54:30.612 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-27T06:54:30.645 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-27T06:54:30.677 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T06:54:30.707 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-27T06:54:30.724 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T06:54:30.824 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-27T06:54:30.876 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-27T06:54:30.878 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T06:54:30.906 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T06:54:30.940 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-27T06:54:30.962 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T06:54:31.021 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-27T06:54:31.063 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-27T06:54:31.183 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-27T06:54:31.253 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-27T06:54:31.296 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-27T06:54:31.362 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-27T06:54:31.491 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-27T06:54:31.517 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T06:54:31.533 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 33/170 2024-05-27T06:54:31.560 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T06:54:31.589 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T06:54:31.624 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 35/170 2024-05-27T06:54:31.671 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T06:54:31.704 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T06:54:31.743 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-27T06:54:31.822 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-27T06:54:31.907 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T06:54:31.946 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-27T06:54:32.006 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T06:54:32.042 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T06:54:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T06:54:32.129 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-27T06:54:32.202 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-27T06:54:32.477 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T06:54:32.538 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T06:54:32.671 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T06:54:32.704 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T06:54:32.759 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T06:54:32.851 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-27T06:54:32.960 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-27T06:54:33.008 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-27T06:54:33.080 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-27T06:54:33.144 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-27T06:54:33.205 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 46/170 2024-05-27T06:54:33.289 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-27T06:54:33.326 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T06:54:33.356 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T06:54:33.438 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T06:54:33.465 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T06:54:33.503 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-27T06:54:33.569 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-27T06:54:33.692 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-27T06:54:33.765 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-27T06:54:33.862 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-27T06:54:33.911 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-27T06:54:33.950 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-27T06:54:34.017 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-27T06:54:34.059 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-27T06:54:34.120 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-27T06:54:34.177 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-27T06:54:34.331 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T06:54:34.372 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-27T06:54:34.375 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T06:54:34.427 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-27T06:54:34.481 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-27T06:54:34.608 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-27T06:54:34.679 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-27T06:54:34.758 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-27T06:54:34.801 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-27T06:54:34.834 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-27T06:54:35.039 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-27T06:54:35.057 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T06:54:35.112 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T06:54:35.244 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-27T06:54:35.289 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-27T06:54:35.301 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T06:54:35.301 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T06:54:35.301 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T06:54:35.302 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:35.337 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-27T06:54:35.338 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T06:54:35.393 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-27T06:54:35.460 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-27T06:54:35.537 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-27T06:54:35.585 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-27T06:54:35.631 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-27T06:54:35.681 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-27T06:54:35.755 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-27T06:54:35.803 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-27T06:54:35.945 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-27T06:54:36.018 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-27T06:54:36.061 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-27T06:54:36.132 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-27T06:54:36.241 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T06:54:36.277 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-27T06:54:36.286 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T06:54:36.367 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-27T06:54:36.979 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T06:54:37.022 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T06:54:37.171 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T06:54:37.171 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T06:54:37.171 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T06:54:37.171 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:54:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T06:54:37.332 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-27T06:54:37.389 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-27T06:54:37.423 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-27T06:54:37.543 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-27T06:54:37.606 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-27T06:54:37.661 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-27T06:54:37.770 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-27T06:54:37.817 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-27T06:54:38.051 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-27T06:54:38.340 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-27T06:54:38.512 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-27T06:54:39.493 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-27T06:54:39.645 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-27T06:54:41.260 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-27T06:54:41.370 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-27T06:54:41.439 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-27T06:54:41.484 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-27T06:54:41.888 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-27T06:54:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T06:54:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 103/170 2024-05-27T06:54:42.115 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 104/170 2024-05-27T06:54:42.176 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-27T06:54:42.770 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-27T06:54:42.802 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T06:54:42.887 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 107/170 2024-05-27T06:54:42.913 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 108/170 2024-05-27T06:54:45.619 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T06:54:45.668 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T06:54:46.347 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 109/170 2024-05-27T06:54:46.397 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T06:54:46.550 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 110/170 2024-05-27T06:54:46.550 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-27T06:54:46.550 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-27T06:54:46.550 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:54:46.589 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T06:54:48.445 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T06:54:48.445 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /sys 2024-05-27T06:54:48.445 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /proc 2024-05-27T06:54:48.445 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /mnt 2024-05-27T06:54:48.445 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /var/tmp 2024-05-27T06:54:48.445 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /home 2024-05-27T06:54:48.445 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /root 2024-05-27T06:54:48.445 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /tmp 2024-05-27T06:54:48.445 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:48.692 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T06:54:48.741 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T06:54:48.741 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:48.741 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T06:54:48.741 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:50.473 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T06:54:50.561 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T06:54:50.561 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:50.562 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T06:54:50.562 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T06:54:50.562 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T06:54:50.562 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:50.815 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T06:54:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T06:54:50.893 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:50.893 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T06:54:50.893 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T06:54:50.893 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T06:54:50.893 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T06:54:51.115 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /sys 2024-05-27T06:54:51.115 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /proc 2024-05-27T06:54:51.115 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /mnt 2024-05-27T06:54:51.115 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /var/tmp 2024-05-27T06:54:51.115 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /home 2024-05-27T06:54:51.115 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /root 2024-05-27T06:54:51.115 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /tmp 2024-05-27T06:54:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:54:51.356 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T06:54:51.399 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T06:54:51.404 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T06:54:51.404 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:51.404 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T06:54:51.404 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:54:51.477 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T06:54:51.477 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:51.477 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T06:54:51.477 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T06:54:51.477 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T06:54:51.477 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:51.608 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T06:54:51.670 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T06:54:51.746 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T06:54:51.982 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T06:54:52.036 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T06:54:52.121 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T06:54:52.231 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T06:54:52.416 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T06:54:52.469 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T06:54:52.518 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T06:54:52.552 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T06:54:53.160 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T06:54:53.198 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T06:54:53.215 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T06:54:53.253 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T06:54:53.317 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T06:54:53.317 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:53.318 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T06:54:53.318 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T06:54:53.318 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T06:54:53.318 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:54:53.378 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T06:54:53.424 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T06:54:53.591 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T06:54:53.672 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T06:54:53.672 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:53.672 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T06:54:53.672 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T06:54:53.672 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T06:54:53.672 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:54:54.208 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T06:54:54.290 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T06:54:54.290 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:54.290 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T06:54:54.290 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T06:54:54.290 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T06:54:54.290 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:54:54.426 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T06:54:54.489 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T06:54:54.566 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T06:54:54.743 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T06:54:54.802 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T06:54:54.814 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T06:54:54.871 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T06:54:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T06:54:55.074 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T06:54:55.250 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T06:54:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T06:54:55.353 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T06:54:55.386 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T06:54:55.899 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T06:54:55.926 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T06:54:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T06:54:56.080 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T06:54:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T06:54:56.172 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T06:54:56.272 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T06:54:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T06:54:56.304 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T06:54:56.315 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T06:54:56.371 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T06:54:56.371 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:56.371 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T06:54:56.372 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T06:54:56.372 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T06:54:56.372 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:56.429 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T06:54:56.467 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T06:54:56.581 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T06:54:56.649 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T06:54:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T06:54:57.678 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T06:54:57.715 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T06:54:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T06:54:57.784 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T06:54:57.784 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:57.784 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T06:54:57.785 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T06:54:57.785 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T06:54:57.785 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:57.881 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T06:54:57.953 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T06:54:57.953 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:57.953 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T06:54:57.953 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:58.033 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T06:54:58.111 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T06:54:58.112 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:58.112 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T06:54:58.112 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T06:54:58.112 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T06:54:58.112 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:58.492 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T06:54:58.557 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T06:54:58.557 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:58.557 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T06:54:58.557 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T06:54:58.557 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T06:54:58.557 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:54:58.905 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T06:54:58.932 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T06:54:59.099 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T06:54:59.313 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T06:54:59.342 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T06:54:59.428 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T06:54:59.428 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T06:54:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T06:54:59.429 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T06:54:59.429 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:54:59.493 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T06:54:59.511 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 111/170 2024-05-27T06:54:59.511 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-05-27T06:54:59.511 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-05-27T06:54:59.511 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-05-27T06:54:59.511 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-05-27T06:54:59.511 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-05-27T06:54:59.511 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /root 2024-05-27T06:54:59.511 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-05-27T06:54:59.511 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:54:59.530 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T06:54:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T06:54:59.702 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T06:54:59.755 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T06:54:59.804 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 112/170 2024-05-27T06:54:59.804 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:54:59.804 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-27T06:54:59.804 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:00.796 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T06:55:00.829 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T06:55:00.911 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T06:55:00.911 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:00.911 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T06:55:00.911 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T06:55:00.912 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T06:55:00.912 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:55:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T06:55:01.058 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T06:55:01.059 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:01.059 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T06:55:01.059 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:55:01.146 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T06:55:01.258 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T06:55:01.259 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:01.259 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T06:55:01.259 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T06:55:01.259 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T06:55:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:55:01.557 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T06:55:01.635 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 113/170 2024-05-27T06:55:01.635 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:01.636 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-27T06:55:01.636 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T06:55:01.636 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-27T06:55:01.636 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:01.664 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T06:55:01.748 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T06:55:01.748 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:01.748 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T06:55:01.748 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T06:55:01.748 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T06:55:01.748 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:55:01.899 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T06:55:01.967 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 114/170 2024-05-27T06:55:01.967 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:01.967 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-27T06:55:01.967 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T06:55:01.967 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-27T06:55:01.967 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:02.474 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T06:55:02.544 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 115/170 2024-05-27T06:55:02.544 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:02.544 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-27T06:55:02.544 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T06:55:02.544 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-27T06:55:02.544 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:02.683 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-27T06:55:02.744 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-27T06:55:02.820 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-27T06:55:03.049 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-27T06:55:03.111 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-27T06:55:03.214 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T06:55:03.219 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-27T06:55:03.266 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T06:55:03.320 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T06:55:03.332 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-27T06:55:03.354 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T06:55:03.392 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T06:55:03.497 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-27T06:55:03.554 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-27T06:55:03.621 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-27T06:55:03.653 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T06:55:03.670 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-27T06:55:03.711 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T06:55:03.747 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T06:55:03.784 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T06:55:03.842 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T06:55:03.893 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T06:55:04.271 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T06:55:04.318 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 127/170 2024-05-27T06:55:04.381 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 128/170 2024-05-27T06:55:04.503 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 129/170 2024-05-27T06:55:04.537 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T06:55:04.876 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T06:55:04.922 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T06:55:04.977 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T06:55:05.864 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 130/170 2024-05-27T06:55:05.906 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T06:55:06.356 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T06:55:06.546 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T06:55:06.635 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T06:55:06.678 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T06:55:06.730 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T06:55:06.744 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T06:55:06.789 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T06:55:06.802 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T06:55:06.830 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T06:55:06.844 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T06:55:06.865 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T06:55:06.877 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T06:55:06.905 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T06:55:06.905 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T06:55:07.007 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 131/170 2024-05-27T06:55:07.039 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T06:55:07.048 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T06:55:07.127 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T06:55:07.127 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T06:55:07.176 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T06:55:07.193 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 132/170 2024-05-27T06:55:07.232 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T06:55:07.256 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T06:55:07.265 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T06:55:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T06:55:07.303 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T06:55:07.343 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T06:55:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T06:55:07.344 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T06:55:07.389 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T06:55:07.399 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 133/170 2024-05-27T06:55:07.425 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T06:55:07.493 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 134/170 2024-05-27T06:55:07.493 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:07.493 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-27T06:55:07.493 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T06:55:07.494 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-27T06:55:07.494 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:07.559 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T06:55:07.596 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 135/170 2024-05-27T06:55:07.691 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T06:55:07.691 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T06:55:07.716 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 136/170 2024-05-27T06:55:07.784 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 137/170 2024-05-27T06:55:07.845 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T06:55:07.898 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T06:55:08.018 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T06:55:08.307 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T06:55:08.307 INFO:teuthology.orchestra.run.smithi179.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-27T06:55:08.307 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:55:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T06:55:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T06:55:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T06:55:08.804 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-27T06:55:08.836 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T06:55:08.933 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 139/170 2024-05-27T06:55:08.933 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:08.933 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-27T06:55:08.933 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T06:55:08.933 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-27T06:55:08.933 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:08.987 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T06:55:09.082 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 140/170 2024-05-27T06:55:09.083 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:09.083 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-27T06:55:09.083 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:09.171 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T06:55:09.276 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 141/170 2024-05-27T06:55:09.276 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:09.276 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-27T06:55:09.276 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T06:55:09.276 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-27T06:55:09.276 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:09.719 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T06:55:09.819 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 142/170 2024-05-27T06:55:09.819 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-27T06:55:09.819 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-27T06:55:09.819 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T06:55:09.820 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-27T06:55:09.820 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T06:55:10.108 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T06:55:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T06:55:10.329 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T06:55:10.377 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T06:55:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T06:55:10.443 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T06:55:10.487 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T06:55:10.487 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T06:55:10.619 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T06:55:10.665 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T06:55:10.665 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T06:55:10.802 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T06:55:10.840 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T06:55:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T06:55:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T06:55:11.236 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T06:55:11.236 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T06:55:11.380 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T06:55:11.426 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T06:55:11.548 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T06:55:11.816 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T06:55:11.816 INFO:teuthology.orchestra.run.smithi043.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-27T06:55:11.816 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:55:14.210 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T06:55:14.210 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T06:55:14.210 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T06:55:14.210 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T06:55:14.210 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T06:55:14.210 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T06:55:14.210 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T06:55:14.211 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T06:55:14.212 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T06:55:14.212 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T06:55:14.212 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T06:55:14.212 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T06:55:14.212 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T06:55:14.212 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T06:55:14.212 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T06:55:14.212 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T06:55:14.212 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T06:55:14.223 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T06:55:14.223 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T06:55:14.224 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T06:55:14.225 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T06:55:14.226 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T06:55:14.227 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T06:55:14.228 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T06:55:14.229 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T06:55:14.230 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T06:55:14.231 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T06:55:14.232 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T06:55:14.233 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T06:55:14.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T06:55:14.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T06:55:14.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T06:55:14.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T06:55:14.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T06:55:14.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T06:55:14.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T06:55:14.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T06:55:14.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T06:55:14.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T06:55:14.519 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 143/170 2024-05-27T06:55:14.580 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 144/170 2024-05-27T06:55:14.669 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 145/170 2024-05-27T06:55:14.710 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-27T06:55:14.750 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-27T06:55:15.046 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 148/170 2024-05-27T06:55:15.098 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-27T06:55:15.133 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 150/170 2024-05-27T06:55:15.172 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-27T06:55:15.222 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-27T06:55:15.274 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T06:55:16.183 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T06:55:16.270 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T06:55:16.326 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-27T06:55:16.522 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T06:55:16.522 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:55:16.522 INFO:teuthology.orchestra.run.smithi179.stdout:Upgraded: 2024-05-27T06:55:16.522 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-05-27T06:55:16.523 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:16.524 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.525 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T06:55:16.526 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.527 INFO:teuthology.orchestra.run.smithi179.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T06:55:16.528 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.529 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T06:55:16.530 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T06:55:16.531 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T06:55:16.532 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T06:55:16.533 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T06:55:16.534 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.535 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.535 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:16.535 INFO:teuthology.orchestra.run.smithi179.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T06:55:16.535 INFO:teuthology.orchestra.run.smithi179.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T06:55:16.535 INFO:teuthology.orchestra.run.smithi179.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T06:55:16.535 INFO:teuthology.orchestra.run.smithi179.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T06:55:16.535 INFO:teuthology.orchestra.run.smithi179.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T06:55:16.535 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T06:55:16.535 INFO:teuthology.orchestra.run.smithi179.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T06:55:16.536 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:55:16.536 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-05-27T06:55:17.173 DEBUG:teuthology.parallel:result is None 2024-05-27T06:55:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T06:55:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T06:55:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T06:55:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T06:55:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T06:55:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T06:55:17.542 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T06:55:17.543 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T06:55:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T06:55:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T06:55:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T06:55:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T06:55:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T06:55:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T06:55:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T06:55:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T06:55:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T06:55:17.544 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T06:55:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T06:55:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T06:55:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T06:55:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T06:55:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T06:55:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T06:55:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T06:55:17.556 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T06:55:17.557 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T06:55:17.558 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T06:55:17.559 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T06:55:17.560 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T06:55:17.561 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T06:55:17.562 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T06:55:17.563 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T06:55:17.564 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T06:55:17.565 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T06:55:17.566 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T06:55:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T06:55:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T06:55:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T06:55:17.567 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T06:55:17.638 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 155/170 2024-05-27T06:55:17.743 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-27T06:55:17.801 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-27T06:55:17.939 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-27T06:55:17.979 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-27T06:55:18.012 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T06:55:18.046 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-27T06:55:18.087 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-27T06:55:18.087 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T06:55:18.218 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-27T06:55:18.266 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-27T06:55:18.267 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T06:55:18.409 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-27T06:55:18.441 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-27T06:55:18.468 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T06:55:18.469 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T06:55:18.790 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-27T06:55:18.790 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T06:55:18.926 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-27T06:55:18.971 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-27T06:55:19.099 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T06:55:19.358 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-27T06:55:19.358 INFO:teuthology.orchestra.run.smithi132.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-27T06:55:19.358 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:19.780 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T06:55:19.780 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:55:19.780 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2024-05-27T06:55:19.780 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:19.780 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:19.780 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:19.780 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T06:55:19.781 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:19.782 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.783 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T06:55:19.784 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.785 INFO:teuthology.orchestra.run.smithi043.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T06:55:19.786 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.787 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T06:55:19.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T06:55:19.789 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T06:55:19.790 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T06:55:19.791 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T06:55:19.792 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T06:55:19.793 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.793 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.793 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:19.793 INFO:teuthology.orchestra.run.smithi043.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T06:55:19.793 INFO:teuthology.orchestra.run.smithi043.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T06:55:19.793 INFO:teuthology.orchestra.run.smithi043.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T06:55:19.793 INFO:teuthology.orchestra.run.smithi043.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T06:55:19.793 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T06:55:19.793 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T06:55:19.794 INFO:teuthology.orchestra.run.smithi043.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T06:55:19.794 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:55:19.794 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-05-27T06:55:20.445 DEBUG:teuthology.parallel:result is None 2024-05-27T06:55:25.404 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 1/170 2024-05-27T06:55:25.404 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2/170 2024-05-27T06:55:25.404 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 3/170 2024-05-27T06:55:25.404 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 4/170 2024-05-27T06:55:25.404 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3872.g38f9d 5/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 6/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 7/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 8/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 9/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 10/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 11/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 12/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 13/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 14/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_ 15/170 2024-05-27T06:55:25.405 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 16/170 2024-05-27T06:55:25.406 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 17/170 2024-05-27T06:55:25.406 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_6 18/170 2024-05-27T06:55:25.406 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86 19/170 2024-05-27T06:55:25.406 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 20/170 2024-05-27T06:55:25.406 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el 21/170 2024-05-27T06:55:25.406 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:19.0.0-3872.g38f9d900.el9. 22/170 2024-05-27T06:55:25.406 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_6 23/170 2024-05-27T06:55:25.406 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 24/170 2024-05-27T06:55:25.406 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 25/170 2024-05-27T06:55:25.406 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 26/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 27/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 28/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 29/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900. 30/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noa 31/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.n 32/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f 33/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el 34/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 35/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.e 36/170 2024-05-27T06:55:25.407 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 37/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 38/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 39/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-27T06:55:25.408 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-27T06:55:25.409 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-27T06:55:25.410 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-27T06:55:25.411 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-27T06:55:25.412 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-27T06:55:25.412 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-27T06:55:25.412 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-27T06:55:25.412 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-27T06:55:25.412 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-27T06:55:25.412 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-27T06:55:25.412 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-27T06:55:25.412 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-27T06:55:25.412 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-27T06:55:25.412 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-27T06:55:25.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-27T06:55:25.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-27T06:55:25.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-27T06:55:25.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-27T06:55:25.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-27T06:55:25.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-27T06:55:25.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-27T06:55:25.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-27T06:55:25.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-27T06:55:25.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-27T06:55:25.414 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-27T06:55:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-27T06:55:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-27T06:55:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-27T06:55:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-27T06:55:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-27T06:55:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-27T06:55:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-27T06:55:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-27T06:55:25.418 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-27T06:55:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-27T06:55:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-27T06:55:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-27T06:55:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-27T06:55:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-27T06:55:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-27T06:55:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-27T06:55:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-27T06:55:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-27T06:55:25.419 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-27T06:55:25.420 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 149/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 151/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-27T06:55:25.421 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-27T06:55:25.422 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-27T06:55:25.423 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-27T06:55:25.423 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-27T06:55:25.423 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-27T06:55:27.744 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.745 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: automake-1.16.2-8.el9.noarch 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: bison-3.7.4-5.el9.x86_64 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.746 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:27.747 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-shell-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-top-2:19.0.0-3872.g38f9d900.el9.noarch 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: flex-2.6.4-9.el9.x86_64 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-27T06:55:27.748 INFO:teuthology.orchestra.run.smithi132.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-27T06:55:27.749 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: lua-5.4.4-4.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-27T06:55:27.750 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-13.14-1.el9.x86_64 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-27T06:55:27.751 INFO:teuthology.orchestra.run.smithi132.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-27T06:55:27.752 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-27T06:55:27.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply-3.11-14.el9.noarch 2024-05-27T06:55:27.754 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.755 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-27T06:55:27.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:19.0.0-3872.g38f9d900.el9.x86_64 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: re2-1:20211101-3.el9.x86_64 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-27T06:55:27.757 INFO:teuthology.orchestra.run.smithi132.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-27T06:55:27.758 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-27T06:55:27.758 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-27T06:55:27.758 INFO:teuthology.orchestra.run.smithi132.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-27T06:55:27.758 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-05-27T06:55:27.758 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-05-27T06:55:28.415 DEBUG:teuthology.parallel:result is None 2024-05-27T06:55:28.416 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:55:28.644 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T06:55:28.722 INFO:teuthology.orchestra.run.smithi043.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T06:55:28.723 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T06:55:28.723 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T06:55:28.725 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:55:28.866 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T06:55:28.927 INFO:teuthology.orchestra.run.smithi132.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T06:55:28.927 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T06:55:28.927 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T06:55:28.929 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:55:29.078 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-27T06:55:29.151 INFO:teuthology.orchestra.run.smithi179.stdout:19.0.0-3872.g38f9d900.el9 2024-05-27T06:55:29.152 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872.g38f9d900.el9 2024-05-27T06:55:29.152 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872.g38f9d900 is installed. 2024-05-27T06:55:29.154 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-27T06:55:29.154 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:55:29.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T06:55:29.256 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:55:29.256 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T06:55:29.325 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:55:29.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T06:55:29.419 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-27T06:55:29.419 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:55:29.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T06:55:29.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T06:55:29.646 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:55:29.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T06:55:29.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T06:55:29.810 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:55:29.810 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T06:55:29.899 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T06:55:30.020 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-27T06:55:30.021 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:55:30.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T06:55:30.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T06:55:30.248 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:55:30.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T06:55:30.327 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T06:55:30.430 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:55:30.430 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T06:55:30.523 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T06:55:30.650 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-27T06:55:30.651 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:55:30.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T06:55:30.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T06:55:30.834 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:55:30.834 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T06:55:30.903 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T06:55:31.001 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:55:31.001 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T06:55:31.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T06:55:31.169 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-27T06:55:31.260 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'subvols': {'create': 2}} 2024-05-27T06:55:31.260 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:55:31.261 INFO:tasks.cephadm:Cluster fsid is 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:55:31.261 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-27T06:55:31.261 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.43', 'mon.b': '172.21.15.132', 'mon.c': '172.21.15.179'} 2024-05-27T06:55:31.261 INFO:tasks.cephadm:First mon is mon.a on smithi043 2024-05-27T06:55:31.261 INFO:tasks.cephadm:First mgr is x 2024-05-27T06:55:31.261 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-27T06:55:31.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo hostname $(hostname -s) 2024-05-27T06:55:31.352 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname $(hostname -s) 2024-05-27T06:55:31.434 DEBUG:teuthology.orchestra.run.smithi179:> sudo hostname $(hostname -s) 2024-05-27T06:55:31.523 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-27T06:55:31.523 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:55:31.671 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79587/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3872-g38f9d900', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3872.g38f9d900'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-21 08:08:15.155937', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240521.063558-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79583/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3872-g38f9d900', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3872.g38f9d900'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-21 07:54:08.730060', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240521.063558-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-27T06:55:31.814 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240521.063558-debug, sha1 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:55:31.816 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm 2024-05-27T06:55:31.816 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm 2024-05-27T06:55:31.816 DEBUG:teuthology.orchestra.run.smithi043:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T06:55:32.553 INFO:teuthology.orchestra.run.smithi043.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 06:55 /home/ubuntu/cephtest/cephadm 2024-05-27T06:55:32.554 DEBUG:teuthology.orchestra.run.smithi132:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T06:55:33.256 INFO:teuthology.orchestra.run.smithi132.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 06:55 /home/ubuntu/cephtest/cephadm 2024-05-27T06:55:33.257 DEBUG:teuthology.orchestra.run.smithi179:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-27T06:55:33.802 INFO:teuthology.orchestra.run.smithi179.stdout:-rw-r--r--. 1 ubuntu ubuntu 782786 May 27 06:55 /home/ubuntu/cephtest/cephadm 2024-05-27T06:55:33.802 DEBUG:teuthology.orchestra.run.smithi043:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T06:55:33.866 DEBUG:teuthology.orchestra.run.smithi132:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T06:55:33.926 DEBUG:teuthology.orchestra.run.smithi179:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-27T06:55:34.008 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 on all hosts... 2024-05-27T06:55:34.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T06:55:34.017 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T06:55:34.024 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 pull 2024-05-27T06:55:35.151 INFO:teuthology.orchestra.run.smithi179.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T06:55:35.193 INFO:teuthology.orchestra.run.smithi043.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T06:55:35.338 INFO:teuthology.orchestra.run.smithi132.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T06:56:16.206 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T06:56:16.206 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T06:56:16.206 INFO:teuthology.orchestra.run.smithi043.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T06:56:16.206 INFO:teuthology.orchestra.run.smithi043.stdout: "repo_digests": [ 2024-05-27T06:56:16.206 INFO:teuthology.orchestra.run.smithi043.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T06:56:16.206 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T06:56:16.206 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T06:56:20.895 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-05-27T06:56:20.895 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T06:56:20.895 INFO:teuthology.orchestra.run.smithi132.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T06:56:20.895 INFO:teuthology.orchestra.run.smithi132.stdout: "repo_digests": [ 2024-05-27T06:56:20.895 INFO:teuthology.orchestra.run.smithi132.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T06:56:20.895 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-05-27T06:56:20.895 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-05-27T06:56:30.111 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-05-27T06:56:30.112 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T06:56:30.112 INFO:teuthology.orchestra.run.smithi179.stdout: "image_id": "e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799", 2024-05-27T06:56:30.112 INFO:teuthology.orchestra.run.smithi179.stdout: "repo_digests": [ 2024-05-27T06:56:30.112 INFO:teuthology.orchestra.run.smithi179.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6" 2024-05-27T06:56:30.112 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-05-27T06:56:30.112 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-05-27T06:56:30.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph 2024-05-27T06:56:30.272 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph 2024-05-27T06:56:30.367 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph 2024-05-27T06:56:30.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /etc/ceph 2024-05-27T06:56:30.556 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /etc/ceph 2024-05-27T06:56:30.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /etc/ceph 2024-05-27T06:56:30.733 INFO:tasks.cephadm:Writing seed config... 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-27T06:56:30.735 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-27T06:56:30.736 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-27T06:56:30.737 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-27T06:56:30.738 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-27T06:56:30.738 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-27T06:56:30.738 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-27T06:56:30.738 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-27T06:56:30.738 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-05-27T06:56:30.739 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:56:30.739 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-27T06:56:30.800 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1bb64d42-1bf6-11ef-bc9b-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-27T06:56:30.800 DEBUG:teuthology.orchestra.run.smithi043:mon.a> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.a.service 2024-05-27T06:56:30.847 DEBUG:teuthology.orchestra.run.smithi043:mgr.x> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.x.service 2024-05-27T06:56:30.894 INFO:tasks.cephadm:Bootstrapping... 2024-05-27T06:56:30.894 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 -v bootstrap --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.43 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-27T06:56:31.172 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-05-27T06:56:31.172 INFO:teuthology.orchestra.run.smithi043.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6', '-v', 'bootstrap', '--fsid', '1bb64d42-1bf6-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.43', '--skip-admin-label'] 2024-05-27T06:56:31.172 INFO:teuthology.orchestra.run.smithi043.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-27T06:56:31.173 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying podman|docker is present... 2024-05-27T06:56:31.236 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout 5.0.2 2024-05-27T06:56:31.236 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying lvm2 is present... 2024-05-27T06:56:31.236 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying time synchronization is in place... 2024-05-27T06:56:31.253 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-27T06:56:31.253 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-27T06:56:31.269 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-27T06:56:31.269 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-05-27T06:56:31.294 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout enabled 2024-05-27T06:56:31.316 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout active 2024-05-27T06:56:31.317 INFO:teuthology.orchestra.run.smithi043.stdout:Unit chronyd.service is enabled and running 2024-05-27T06:56:31.317 INFO:teuthology.orchestra.run.smithi043.stdout:Repeating the final host check... 2024-05-27T06:56:31.379 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout 5.0.2 2024-05-27T06:56:31.379 INFO:teuthology.orchestra.run.smithi043.stdout:podman (/bin/podman) version 5.0.2 is present 2024-05-27T06:56:31.379 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl is present 2024-05-27T06:56:31.379 INFO:teuthology.orchestra.run.smithi043.stdout:lvcreate is present 2024-05-27T06:56:31.394 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-27T06:56:31.394 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-27T06:56:31.413 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-27T06:56:31.413 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-05-27T06:56:31.436 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout enabled 2024-05-27T06:56:31.459 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout active 2024-05-27T06:56:31.459 INFO:teuthology.orchestra.run.smithi043.stdout:Unit chronyd.service is enabled and running 2024-05-27T06:56:31.460 INFO:teuthology.orchestra.run.smithi043.stdout:Host looks OK 2024-05-27T06:56:31.460 INFO:teuthology.orchestra.run.smithi043.stdout:Cluster fsid: 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:56:31.461 INFO:teuthology.orchestra.run.smithi043.stdout:Acquiring lock 139821391490592 on /run/cephadm/1bb64d42-1bf6-11ef-bc9b-c7b262605968.lock 2024-05-27T06:56:31.461 INFO:teuthology.orchestra.run.smithi043.stdout:Lock 139821391490592 acquired on /run/cephadm/1bb64d42-1bf6-11ef-bc9b-c7b262605968.lock 2024-05-27T06:56:31.462 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying IP 172.21.15.43 port 3300 ... 2024-05-27T06:56:31.463 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying IP 172.21.15.43 port 6789 ... 2024-05-27T06:56:31.464 INFO:teuthology.orchestra.run.smithi043.stdout:Base mon IP(s) is [172.21.15.43:3300, 172.21.15.43:6789], mon addrv is [v2:172.21.15.43:3300,v1:172.21.15.43:6789] 2024-05-27T06:56:31.479 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.43 metric 100 2024-05-27T06:56:31.480 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.43 metric 100 2024-05-27T06:56:31.493 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-27T06:56:31.493 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-05-27T06:56:31.493 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-05-27T06:56:31.506 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-27T06:56:31.506 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-27T06:56:31.506 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-27T06:56:31.506 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-05-27T06:56:31.506 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-05-27T06:56:31.506 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-27T06:56:31.508 INFO:teuthology.orchestra.run.smithi043.stdout:Mon IP `172.21.15.43` is in CIDR network `172.21.0.0/20` 2024-05-27T06:56:31.508 INFO:teuthology.orchestra.run.smithi043.stdout:Mon IP `172.21.15.43` is in CIDR network `172.21.0.0/20` 2024-05-27T06:56:31.509 INFO:teuthology.orchestra.run.smithi043.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-27T06:56:31.511 INFO:teuthology.orchestra.run.smithi043.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-27T06:56:31.514 INFO:teuthology.orchestra.run.smithi043.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T06:56:32.093 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 2024-05-27T06:56:32.093 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6... 2024-05-27T06:56:32.093 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Getting image source signatures 2024-05-27T06:56:32.094 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying blob sha256:cf46015854c6cdc81d3273f73f8f001fe4a1edc6ed77e464da8c9d5e01e9b108 2024-05-27T06:56:32.094 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying blob sha256:6ac36e7c57f227110df5cd65cfdca1cc05e0b17df6748637a6aa04c5c80f89c6 2024-05-27T06:56:32.094 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying config sha256:e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 2024-05-27T06:56:32.094 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-27T06:56:33.654 INFO:teuthology.orchestra.run.smithi043.stdout:ceph: stdout ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T06:56:33.654 INFO:teuthology.orchestra.run.smithi043.stdout:Ceph version: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T06:56:33.654 INFO:teuthology.orchestra.run.smithi043.stdout:Extracting ceph user uid/gid from container image... 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi043.stdout:stat: stdout 167 167 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial keys... 2024-05-27T06:56:35.966 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQCjLlRm9vJqFBAAEH7l+8rvKRk/xS4sPHRdUg== 2024-05-27T06:56:37.288 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQCkLlRm/s3vIRAA3GxhzUQs/w/74TXzdyZ/vQ== 2024-05-27T06:56:38.606 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQClLlRmh6qdMRAAh9/0g7WfdQ3jYfMiBK+2AA== 2024-05-27T06:56:38.610 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial monmap... 2024-05-27T06:56:39.771 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-27T06:56:39.772 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-27T06:56:39.772 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:56:39.772 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-27T06:56:39.773 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool for a [v2:172.21.15.43:3300,v1:172.21.15.43:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-27T06:56:39.773 INFO:teuthology.orchestra.run.smithi043.stdout:setting min_mon_release = quincy 2024-05-27T06:56:39.773 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: set fsid to 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:56:39.773 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-27T06:56:39.773 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:56:39.774 INFO:teuthology.orchestra.run.smithi043.stdout:Creating mon... 2024-05-27T06:56:41.014 INFO:teuthology.orchestra.run.smithi043.stdout:create mon.a on 2024-05-27T06:56:41.443 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-27T06:56:41.704 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-27T06:56:41.995 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968.target. 2024-05-27T06:56:41.995 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968.target. 2024-05-27T06:56:42.398 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.a 2024-05-27T06:56:42.399 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.a.service: Unit ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.a.service not loaded. 2024-05-27T06:56:42.713 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968.target.wants/ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.a.service → /etc/systemd/system/ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@.service. 2024-05-27T06:56:43.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:43 smithi043 podman[31133]: 2024-05-27 06:56:43.386671117 +0000 UTC m=+0.378028496 container start afcf945d10bb46acecad8b057499b6ceb1aafe847c960ef4ea546cc1dc20a3e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T06:56:43.508 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T06:56:43.508 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-05-27T06:56:43.525 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T06:56:43.525 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-05-27T06:56:43.525 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-05-27T06:56:43.525 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to enable service . firewalld.service is not available 2024-05-27T06:56:43.526 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mon to start... 2024-05-27T06:56:43.526 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mon... 2024-05-27T06:56:43.868 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:43 smithi043 bash[31133]: afcf945d10bb46acecad8b057499b6ceb1aafe847c960ef4ea546cc1dc20a3e5 2024-05-27T06:56:43.868 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:43 smithi043 systemd[1]: Started Ceph mon.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T06:56:43.868 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:43 smithi043 ceph-mon[31152]: mkfs 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:56:43.868 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:43 smithi043 ceph-mon[31152]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T06:56:44.821 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T06:56:44.821 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: monmap epoch 1 2024-05-27T06:56:44.821 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:56:44.821 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: last_changed 2024-05-27T06:56:39.161167+0000 2024-05-27T06:56:44.821 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: created 2024-05-27T06:56:39.161167+0000 2024-05-27T06:56:44.821 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: min_mon_release 19 (squid) 2024-05-27T06:56:44.821 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: election_strategy: 1 2024-05-27T06:56:44.821 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-05-27T06:56:44.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: fsmap 2024-05-27T06:56:44.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: osdmap e1: 0 total, 0 up, 0 in 2024-05-27T06:56:44.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: mgrmap e1: no daemons active 2024-05-27T06:56:44.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:44 smithi043 ceph-mon[31152]: from='client.? 172.21.15.43:0/1899662880' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-27T06:56:45.014 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout cluster: 2024-05-27T06:56:45.015 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout id: 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:56:45.015 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-27T06:56:45.015 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:56:45.015 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout services: 2024-05-27T06:56:45.015 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.812826s) 2024-05-27T06:56:45.016 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-27T06:56:45.016 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-27T06:56:45.016 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:56:45.022 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout data: 2024-05-27T06:56:45.022 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-27T06:56:45.022 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-27T06:56:45.022 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-27T06:56:45.022 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout pgs: 2024-05-27T06:56:45.022 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:56:45.022 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.266+0000 7fe727c00700 1 Processor -- start 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.266+0000 7fe727c00700 1 -- start start 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.266+0000 7fe727c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72007d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.266+0000 7fe727c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe72007d720 con 0x7fe72007ed10 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.268+0000 7fe725200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72007d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.268+0000 7fe725200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72007d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54814/0 (socket says 172.21.15.43:54814) 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.268+0000 7fe725200700 1 -- 172.21.15.43:0/1578017755 learned_addr learned my addr 172.21.15.43:0/1578017755 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.268+0000 7fe725200700 1 -- 172.21.15.43:0/1578017755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72007d8a0 con 0x7fe72007ed10 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.269+0000 7fe725200700 1 --2- 172.21.15.43:0/1578017755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72007d1e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe7140098a0 tx=0x7fe714009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb8c7864585fe1e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.269+0000 7fe71f400700 1 -- 172.21.15.43:0/1578017755 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe714004160 con 0x7fe72007ed10 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.269+0000 7fe71f400700 1 -- 172.21.15.43:0/1578017755 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe714004300 con 0x7fe72007ed10 2024-05-27T06:56:45.023 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.269+0000 7fe71f400700 1 -- 172.21.15.43:0/1578017755 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7140036a0 con 0x7fe72007ed10 2024-05-27T06:56:45.024 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.270+0000 7fe727c00700 1 -- 172.21.15.43:0/1578017755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 msgr2=0x7fe72007d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:45.024 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.270+0000 7fe727c00700 1 --2- 172.21.15.43:0/1578017755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72007d1e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe7140098a0 tx=0x7fe714009bb0 comp rx=0 tx=0).stop 2024-05-27T06:56:45.024 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.270+0000 7fe727c00700 1 -- 172.21.15.43:0/1578017755 shutdown_connections 2024-05-27T06:56:45.024 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.270+0000 7fe727c00700 1 --2- 172.21.15.43:0/1578017755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72007d1e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:45.024 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.270+0000 7fe727c00700 1 -- 172.21.15.43:0/1578017755 >> 172.21.15.43:0/1578017755 conn(0x7fe720105100 msgr2=0x7fe720107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:45.024 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.270+0000 7fe727c00700 1 -- 172.21.15.43:0/1578017755 shutdown_connections 2024-05-27T06:56:45.024 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.270+0000 7fe727c00700 1 -- 172.21.15.43:0/1578017755 wait complete. 2024-05-27T06:56:45.024 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.272+0000 7fe727c00700 1 Processor -- start 2024-05-27T06:56:45.024 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.272+0000 7fe727c00700 1 -- start start 2024-05-27T06:56:45.024 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.273+0000 7fe727c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72019f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.273+0000 7fe727c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe72007ddd0 con 0x7fe72007ed10 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.273+0000 7fe725200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72019f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.273+0000 7fe725200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72019f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54830/0 (socket says 172.21.15.43:54830) 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.273+0000 7fe725200700 1 -- 172.21.15.43:0/1899662880 learned_addr learned my addr 172.21.15.43:0/1899662880 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.274+0000 7fe725200700 1 -- 172.21.15.43:0/1899662880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72019fb80 con 0x7fe72007ed10 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.274+0000 7fe725200700 1 --2- 172.21.15.43:0/1899662880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72019f640 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe714000c00 tx=0x7fe714004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.274+0000 7fe71d600700 1 -- 172.21.15.43:0/1899662880 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe714003db0 con 0x7fe72007ed10 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.274+0000 7fe71d600700 1 -- 172.21.15.43:0/1899662880 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe714011630 con 0x7fe72007ed10 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.274+0000 7fe71d600700 1 -- 172.21.15.43:0/1899662880 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe714011950 con 0x7fe72007ed10 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.274+0000 7fe727c00700 1 -- 172.21.15.43:0/1899662880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe72019fe10 con 0x7fe72007ed10 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.274+0000 7fe727c00700 1 -- 172.21.15.43:0/1899662880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7201a2a20 con 0x7fe72007ed10 2024-05-27T06:56:45.025 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.275+0000 7fe71d600700 1 -- 172.21.15.43:0/1899662880 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe714011af0 con 0x7fe72007ed10 2024-05-27T06:56:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.275+0000 7fe71d600700 1 -- 172.21.15.43:0/1899662880 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe71401c450 con 0x7fe72007ed10 2024-05-27T06:56:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.276+0000 7fe727c00700 1 -- 172.21.15.43:0/1899662880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe720064e10 con 0x7fe72007ed10 2024-05-27T06:56:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.279+0000 7fe71d600700 1 -- 172.21.15.43:0/1899662880 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fe714032020 con 0x7fe72007ed10 2024-05-27T06:56:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.361+0000 7fe727c00700 1 -- 172.21.15.43:0/1899662880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fe72002e8e0 con 0x7fe72007ed10 2024-05-27T06:56:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.362+0000 7fe71d600700 1 -- 172.21.15.43:0/1899662880 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fe714004260 con 0x7fe72007ed10 2024-05-27T06:56:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.363+0000 7fe727c00700 1 -- 172.21.15.43:0/1899662880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 msgr2=0x7fe72019f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.363+0000 7fe727c00700 1 --2- 172.21.15.43:0/1899662880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72019f640 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe714000c00 tx=0x7fe714004750 comp rx=0 tx=0).stop 2024-05-27T06:56:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.364+0000 7fe727c00700 1 -- 172.21.15.43:0/1899662880 shutdown_connections 2024-05-27T06:56:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.364+0000 7fe727c00700 1 --2- 172.21.15.43:0/1899662880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72007ed10 0x7fe72019f640 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:45.026 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.364+0000 7fe727c00700 1 -- 172.21.15.43:0/1899662880 >> 172.21.15.43:0/1899662880 conn(0x7fe720105100 msgr2=0x7fe720105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:45.027 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.364+0000 7fe727c00700 1 -- 172.21.15.43:0/1899662880 shutdown_connections 2024-05-27T06:56:45.027 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:44.364+0000 7fe727c00700 1 -- 172.21.15.43:0/1899662880 wait complete. 2024-05-27T06:56:45.027 INFO:teuthology.orchestra.run.smithi043.stdout:mon is available 2024-05-27T06:56:45.027 INFO:teuthology.orchestra.run.smithi043.stdout:Assimilating anything we can from ceph.conf... 2024-05-27T06:56:46.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:45 smithi043 ceph-mon[31152]: from='client.? 172.21.15.43:0/3868331711' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-27T06:56:46.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:45 smithi043 ceph-mon[31152]: from='client.? 172.21.15.43:0/3868331711' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [global] 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout fsid = 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.43:3300,v1:172.21.15.43:6789] 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:56:46.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [osd] 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.654+0000 7f0b31600700 1 Processor -- start 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.654+0000 7f0b31600700 1 -- start start 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.654+0000 7f0b31600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c07d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:46.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.654+0000 7f0b31600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0b2c07d700 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.655+0000 7f0b2aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c07d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.655+0000 7f0b2aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c07d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47730/0 (socket says 172.21.15.43:47730) 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.655+0000 7f0b2aa00700 1 -- 172.21.15.43:0/1763985368 learned_addr learned my addr 172.21.15.43:0/1763985368 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.656+0000 7f0b2aa00700 1 -- 172.21.15.43:0/1763985368 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b2c07d880 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.656+0000 7f0b2aa00700 1 --2- 172.21.15.43:0/1763985368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c07d1c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c0098a0 tx=0x7f0b1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14aa8b613ea2996 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.656+0000 7f0b29600700 1 -- 172.21.15.43:0/1763985368 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b1c004160 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.656+0000 7f0b29600700 1 -- 172.21.15.43:0/1763985368 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b1c004300 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.656+0000 7f0b29600700 1 -- 172.21.15.43:0/1763985368 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b1c0036a0 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.657+0000 7f0b31600700 1 -- 172.21.15.43:0/1763985368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 msgr2=0x7f0b2c07d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.657+0000 7f0b31600700 1 --2- 172.21.15.43:0/1763985368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c07d1c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c0098a0 tx=0x7f0b1c009bb0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.657+0000 7f0b31600700 1 -- 172.21.15.43:0/1763985368 shutdown_connections 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.657+0000 7f0b31600700 1 --2- 172.21.15.43:0/1763985368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c07d1c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.657+0000 7f0b31600700 1 -- 172.21.15.43:0/1763985368 >> 172.21.15.43:0/1763985368 conn(0x7f0b2c105100 msgr2=0x7f0b2c107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.657+0000 7f0b31600700 1 -- 172.21.15.43:0/1763985368 shutdown_connections 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.657+0000 7f0b31600700 1 -- 172.21.15.43:0/1763985368 wait complete. 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.659+0000 7f0b31600700 1 Processor -- start 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.659+0000 7f0b31600700 1 -- start start 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.660+0000 7f0b31600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.660+0000 7f0b31600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0b2c07ddb0 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.660+0000 7f0b2aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.660+0000 7f0b2aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c19f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47732/0 (socket says 172.21.15.43:47732) 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.660+0000 7f0b2aa00700 1 -- 172.21.15.43:0/3868331711 learned_addr learned my addr 172.21.15.43:0/3868331711 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.660+0000 7f0b2aa00700 1 -- 172.21.15.43:0/3868331711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b2c19fc10 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.661+0000 7f0b2aa00700 1 --2- 172.21.15.43:0/3868331711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c19f6d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c000c00 tx=0x7f0b1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.661+0000 7f0b23400700 1 -- 172.21.15.43:0/3868331711 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b1c003db0 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.661+0000 7f0b23400700 1 -- 172.21.15.43:0/3868331711 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b1c011630 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.661+0000 7f0b23400700 1 -- 172.21.15.43:0/3868331711 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b1c011950 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.661+0000 7f0b31600700 1 -- 172.21.15.43:0/3868331711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b2c19fea0 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.661+0000 7f0b31600700 1 -- 172.21.15.43:0/3868331711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b2c1a2ab0 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.662+0000 7f0b23400700 1 -- 172.21.15.43:0/3868331711 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0b1c011af0 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.662+0000 7f0b23400700 1 -- 172.21.15.43:0/3868331711 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0b1c01c450 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.663+0000 7f0b31600700 1 -- 172.21.15.43:0/3868331711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b2c064e10 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.666+0000 7f0b23400700 1 -- 172.21.15.43:0/3868331711 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f0b1c032020 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.757+0000 7f0b31600700 1 -- 172.21.15.43:0/3868331711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f0b2c02e8e0 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.766+0000 7f0b23400700 1 -- 172.21.15.43:0/3868331711 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f0b1c004260 con 0x7f0b2c07ecf0 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.767+0000 7f0b31600700 1 -- 172.21.15.43:0/3868331711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 msgr2=0x7f0b2c19f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.767+0000 7f0b31600700 1 --2- 172.21.15.43:0/3868331711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c19f6d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c000c00 tx=0x7f0b1c004750 comp rx=0 tx=0).stop 2024-05-27T06:56:46.420 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.768+0000 7f0b31600700 1 -- 172.21.15.43:0/3868331711 shutdown_connections 2024-05-27T06:56:46.421 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.768+0000 7f0b31600700 1 --2- 172.21.15.43:0/3868331711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b2c07ecf0 0x7f0b2c19f6d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.421 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.768+0000 7f0b31600700 1 -- 172.21.15.43:0/3868331711 >> 172.21.15.43:0/3868331711 conn(0x7f0b2c105100 msgr2=0x7f0b2c105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:46.421 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.768+0000 7f0b31600700 1 -- 172.21.15.43:0/3868331711 shutdown_connections 2024-05-27T06:56:46.421 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:45.768+0000 7f0b31600700 1 -- 172.21.15.43:0/3868331711 wait complete. 2024-05-27T06:56:46.421 INFO:teuthology.orchestra.run.smithi043.stdout:Generating new minimal ceph.conf... 2024-05-27T06:56:47.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:47 smithi043 ceph-mon[31152]: from='client.? 172.21.15.43:0/3299444577' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:56:47.860 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.075+0000 7fd794600700 1 Processor -- start 2024-05-27T06:56:47.861 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.075+0000 7fd794600700 1 -- start start 2024-05-27T06:56:47.861 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.076+0000 7fd794600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c07d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:47.862 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.076+0000 7fd794600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c07d740 con 0x7fd78c07ed30 2024-05-27T06:56:47.862 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.076+0000 7fd791c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c07d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:47.863 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.076+0000 7fd791c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c07d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47734/0 (socket says 172.21.15.43:47734) 2024-05-27T06:56:47.863 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.076+0000 7fd791c00700 1 -- 172.21.15.43:0/1327766858 learned_addr learned my addr 172.21.15.43:0/1327766858 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:47.863 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.077+0000 7fd791c00700 1 -- 172.21.15.43:0/1327766858 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c07d8c0 con 0x7fd78c07ed30 2024-05-27T06:56:47.863 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.077+0000 7fd791c00700 1 --2- 172.21.15.43:0/1327766858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c07d200 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd7800098a0 tx=0x7fd780009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b3ed3da069ba5a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:47.864 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.077+0000 7fd78be00700 1 -- 172.21.15.43:0/1327766858 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd780004160 con 0x7fd78c07ed30 2024-05-27T06:56:47.864 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.077+0000 7fd78be00700 1 -- 172.21.15.43:0/1327766858 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fd780004300 con 0x7fd78c07ed30 2024-05-27T06:56:47.865 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.077+0000 7fd78be00700 1 -- 172.21.15.43:0/1327766858 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7800119e0 con 0x7fd78c07ed30 2024-05-27T06:56:47.865 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.078+0000 7fd794600700 1 -- 172.21.15.43:0/1327766858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 msgr2=0x7fd78c07d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:47.865 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.078+0000 7fd794600700 1 --2- 172.21.15.43:0/1327766858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c07d200 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd7800098a0 tx=0x7fd780009bb0 comp rx=0 tx=0).stop 2024-05-27T06:56:47.865 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.078+0000 7fd794600700 1 -- 172.21.15.43:0/1327766858 shutdown_connections 2024-05-27T06:56:47.865 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.078+0000 7fd794600700 1 --2- 172.21.15.43:0/1327766858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c07d200 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:47.865 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.078+0000 7fd794600700 1 -- 172.21.15.43:0/1327766858 >> 172.21.15.43:0/1327766858 conn(0x7fd78c105100 msgr2=0x7fd78c107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:47.865 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.079+0000 7fd794600700 1 -- 172.21.15.43:0/1327766858 shutdown_connections 2024-05-27T06:56:47.865 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.079+0000 7fd794600700 1 -- 172.21.15.43:0/1327766858 wait complete. 2024-05-27T06:56:47.865 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.080+0000 7fd794600700 1 Processor -- start 2024-05-27T06:56:47.867 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.080+0000 7fd794600700 1 -- start start 2024-05-27T06:56:47.867 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.080+0000 7fd794600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c19fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:47.867 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.080+0000 7fd794600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c10fe80 con 0x7fd78c07ed30 2024-05-27T06:56:47.867 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.081+0000 7fd791c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c19fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:47.867 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.081+0000 7fd791c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c19fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47748/0 (socket says 172.21.15.43:47748) 2024-05-27T06:56:47.867 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.081+0000 7fd791c00700 1 -- 172.21.15.43:0/3299444577 learned_addr learned my addr 172.21.15.43:0/3299444577 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:47.867 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.081+0000 7fd791c00700 1 -- 172.21.15.43:0/3299444577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c1a0360 con 0x7fd78c07ed30 2024-05-27T06:56:47.868 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.081+0000 7fd791c00700 1 --2- 172.21.15.43:0/3299444577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c19fe20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd7800116a0 tx=0x7fd780011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:47.868 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.082+0000 7fd78a000700 1 -- 172.21.15.43:0/3299444577 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd780003970 con 0x7fd78c07ed30 2024-05-27T06:56:47.868 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.082+0000 7fd78a000700 1 -- 172.21.15.43:0/3299444577 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fd780003b10 con 0x7fd78c07ed30 2024-05-27T06:56:47.868 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.082+0000 7fd794600700 1 -- 172.21.15.43:0/3299444577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78c1a05f0 con 0x7fd78c07ed30 2024-05-27T06:56:47.868 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.082+0000 7fd78a000700 1 -- 172.21.15.43:0/3299444577 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd780019550 con 0x7fd78c07ed30 2024-05-27T06:56:47.868 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.082+0000 7fd794600700 1 -- 172.21.15.43:0/3299444577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78c1a3200 con 0x7fd78c07ed30 2024-05-27T06:56:47.868 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.082+0000 7fd78a000700 1 -- 172.21.15.43:0/3299444577 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd780019ba0 con 0x7fd78c07ed30 2024-05-27T06:56:47.868 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.082+0000 7fd78a000700 1 -- 172.21.15.43:0/3299444577 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd78000b720 con 0x7fd78c07ed30 2024-05-27T06:56:47.869 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.083+0000 7fd794600700 1 -- 172.21.15.43:0/3299444577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78c064e10 con 0x7fd78c07ed30 2024-05-27T06:56:47.870 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.086+0000 7fd78a000700 1 -- 172.21.15.43:0/3299444577 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fd78001aa50 con 0x7fd78c07ed30 2024-05-27T06:56:47.870 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.165+0000 7fd794600700 1 -- 172.21.15.43:0/3299444577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fd78c1a34f0 con 0x7fd78c07ed30 2024-05-27T06:56:47.870 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.166+0000 7fd78a000700 1 -- 172.21.15.43:0/3299444577 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fd78001a430 con 0x7fd78c07ed30 2024-05-27T06:56:47.870 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.167+0000 7fd794600700 1 -- 172.21.15.43:0/3299444577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 msgr2=0x7fd78c19fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:47.870 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.167+0000 7fd794600700 1 --2- 172.21.15.43:0/3299444577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c19fe20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd7800116a0 tx=0x7fd780011a60 comp rx=0 tx=0).stop 2024-05-27T06:56:47.870 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.167+0000 7fd794600700 1 -- 172.21.15.43:0/3299444577 shutdown_connections 2024-05-27T06:56:47.870 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.167+0000 7fd794600700 1 --2- 172.21.15.43:0/3299444577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd78c07ed30 0x7fd78c19fe20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:47.870 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.167+0000 7fd794600700 1 -- 172.21.15.43:0/3299444577 >> 172.21.15.43:0/3299444577 conn(0x7fd78c105100 msgr2=0x7fd78c105980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:47.870 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.168+0000 7fd794600700 1 -- 172.21.15.43:0/3299444577 shutdown_connections 2024-05-27T06:56:47.870 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:47.168+0000 7fd794600700 1 -- 172.21.15.43:0/3299444577 wait complete. 2024-05-27T06:56:47.871 INFO:teuthology.orchestra.run.smithi043.stdout:Restarting the monitor... 2024-05-27T06:56:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:47 smithi043 systemd[1]: Stopping Ceph mon.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T06:56:48.618 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a[31148]: 2024-05-27T06:56:48.336+0000 7f5064e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T06:56:48.618 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a[31148]: 2024-05-27T06:56:48.336+0000 7f5064e00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-27T06:56:48.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:48 smithi043 podman[31408]: 2024-05-27 06:56:48.625951889 +0000 UTC m=+0.398002824 container died afcf945d10bb46acecad8b057499b6ceb1aafe847c960ef4ea546cc1dc20a3e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T06:56:49.186 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:48 smithi043 podman[31408]: 2024-05-27 06:56:48.987273534 +0000 UTC m=+0.759324470 container cleanup afcf945d10bb46acecad8b057499b6ceb1aafe847c960ef4ea546cc1dc20a3e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T06:56:49.186 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:48 smithi043 bash[31408]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a 2024-05-27T06:56:49.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:49 smithi043 podman[31422]: 2024-05-27 06:56:49.194106342 +0000 UTC m=+0.551522861 container remove afcf945d10bb46acecad8b057499b6ceb1aafe847c960ef4ea546cc1dc20a3e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-27T06:56:49.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:49 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-27T06:56:49.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:49 smithi043 systemd[1]: Stopped Ceph mon.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T06:56:49.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:49 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.a.service: Consumed 2.046s CPU time. 2024-05-27T06:56:49.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:49 smithi043 systemd[1]: Starting Ceph mon.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T06:56:50.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 podman[31513]: 2024-05-27 06:56:50.011384025 +0000 UTC m=+0.042887484 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:56:50.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 podman[31513]: 2024-05-27 06:56:50.105097404 +0000 UTC m=+0.136600863 container create 83c5e162e73ac83426edc994a78b2a05b76872cd934b4bdcf1fd06e3b26c3ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T06:56:50.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 podman[31513]: 2024-05-27 06:56:50.350610797 +0000 UTC m=+0.382114290 container init 83c5e162e73ac83426edc994a78b2a05b76872cd934b4bdcf1fd06e3b26c3ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-27T06:56:50.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 podman[31513]: 2024-05-27 06:56:50.36193244 +0000 UTC m=+0.393435932 container start 83c5e162e73ac83426edc994a78b2a05b76872cd934b4bdcf1fd06e3b26c3ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T06:56:50.464 INFO:teuthology.orchestra.run.smithi043.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-27T06:56:50.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 bash[31513]: 83c5e162e73ac83426edc994a78b2a05b76872cd934b4bdcf1fd06e3b26c3ea8 2024-05-27T06:56:50.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 systemd[1]: Started Ceph mon.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T06:56:50.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T06:56:50.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T06:56:50.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: pidfile_write: ignore empty --pid-file 2024-05-27T06:56:50.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: load: jerasure load: lrc 2024-05-27T06:56:50.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: RocksDB version: 7.9.2 2024-05-27T06:56:50.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Git sha 0 2024-05-27T06:56:50.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T06:56:50.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: DB SUMMARY 2024-05-27T06:56:50.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: DB Session ID: YPE45W6AWF8O2QIWIL81 2024-05-27T06:56:50.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: CURRENT file: CURRENT 2024-05-27T06:56:50.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T06:56:50.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-27T06:56:50.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-27T06:56:50.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103424 ; 2024-05-27T06:56:50.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.error_if_exists: 0 2024-05-27T06:56:50.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.create_if_missing: 0 2024-05-27T06:56:50.629 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.paranoid_checks: 1 2024-05-27T06:56:50.629 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T06:56:50.629 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T06:56:50.629 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T06:56:50.629 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.env: 0x55f89ebd6500 2024-05-27T06:56:50.629 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T06:56:50.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.info_log: 0x55f8d4a86040 2024-05-27T06:56:50.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T06:56:50.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.statistics: (nil) 2024-05-27T06:56:50.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.use_fsync: 0 2024-05-27T06:56:50.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_log_file_size: 0 2024-05-27T06:56:50.630 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T06:56:50.631 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T06:56:50.631 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T06:56:50.631 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T06:56:50.631 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.allow_fallocate: 1 2024-05-27T06:56:50.631 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T06:56:50.631 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T06:56:50.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.use_direct_reads: 0 2024-05-27T06:56:50.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T06:56:50.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T06:56:50.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.db_log_dir: 2024-05-27T06:56:50.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.wal_dir: 2024-05-27T06:56:50.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T06:56:50.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T06:56:50.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T06:56:50.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T06:56:50.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T06:56:50.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T06:56:50.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T06:56:50.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T06:56:50.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.write_buffer_manager: 0x55f8d49508c0 2024-05-27T06:56:50.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T06:56:50.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T06:56:50.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T06:56:50.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T06:56:50.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T06:56:50.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T06:56:50.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T06:56:50.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T06:56:50.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.unordered_write: 0 2024-05-27T06:56:50.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T06:56:50.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T06:56:50.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T06:56:50.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T06:56:50.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.row_cache: None 2024-05-27T06:56:50.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.wal_filter: None 2024-05-27T06:56:50.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T06:56:50.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T06:56:50.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.two_write_queues: 0 2024-05-27T06:56:50.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T06:56:50.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.wal_compression: 0 2024-05-27T06:56:50.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.atomic_flush: 0 2024-05-27T06:56:50.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T06:56:50.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T06:56:50.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T06:56:50.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.log_readahead_size: 0 2024-05-27T06:56:50.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T06:56:50.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T06:56:50.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T06:56:50.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T06:56:50.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T06:56:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T06:56:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T06:56:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_background_jobs: 2 2024-05-27T06:56:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_background_compactions: -1 2024-05-27T06:56:50.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_subcompactions: 1 2024-05-27T06:56:50.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T06:56:50.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T06:56:50.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T06:56:50.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T06:56:50.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T06:56:50.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T06:56:50.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T06:56:50.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T06:56:50.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_open_files: -1 2024-05-27T06:56:50.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T06:56:50.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T06:56:50.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T06:56:50.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T06:56:50.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_background_flushes: -1 2024-05-27T06:56:50.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Compression algorithms supported: 2024-05-27T06:56:50.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T06:56:50.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: kZSTD supported: 0 2024-05-27T06:56:50.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: kXpressCompression supported: 0 2024-05-27T06:56:50.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T06:56:50.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: kZlibCompression supported: 1 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: kSnappyCompression supported: 1 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: kLZ4Compression supported: 1 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: kBZip2Compression supported: 0 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.merge_operator: 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_filter: None 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T06:56:50.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f8d4a86200) 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout: cache_index_and_filter_blocks: 1 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout: pin_top_level_index_and_filter: 1 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_type: 0 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout: data_block_index_type: 0 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_shortening: 1 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T06:56:50.646 INFO:journalctl@ceph.mon.a.smithi043.stdout: checksum: 4 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: no_block_cache: 0 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache: 0x55f8d49831f0 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_name: BinnedLRUCache 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_options: 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: capacity : 536870912 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: num_shard_bits : 4 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: strict_capacity_limit : 0 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: high_pri_pool_ratio: 0.000 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_compressed: (nil) 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: persistent_cache: (nil) 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_size: 4096 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_size_deviation: 10 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_restart_interval: 16 2024-05-27T06:56:50.647 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_block_restart_interval: 1 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: metadata_block_size: 4096 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: partition_filters: 0 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: use_delta_encoding: 1 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: filter_policy: bloomfilter 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: whole_key_filtering: 1 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: verify_compression: 0 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: read_amp_bytes_per_bit: 0 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: format_version: 5 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: enable_index_compression: 1 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_align: 0 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: max_auto_readahead_size: 262144 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: prepopulate_block_cache: 0 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: initial_auto_readahead_size: 8192 2024-05-27T06:56:50.648 INFO:journalctl@ceph.mon.a.smithi043.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compression: NoCompression 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.num_levels: 7 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T06:56:50.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T06:56:50.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T06:56:50.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T06:56:50.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.inplace_update_support: 0 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.bloom_locality: 0 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.max_successive_merges: 0 2024-05-27T06:56:50.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T06:56:50.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T06:56:50.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T06:56:50.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T06:56:50.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.ttl: 2592000 2024-05-27T06:56:50.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T06:56:50.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T06:56:50.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T06:56:50.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.enable_blob_files: false 2024-05-27T06:56:50.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.min_blob_size: 0 2024-05-27T06:56:50.654 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 79b222cf-27f3-4632-a939-a410031d5175 2024-05-27T06:56:50.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793010495622, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793010499023, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 98715, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 96668, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13762, "raw_average_key_size": 49, "raw_value_size": 89137, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716793010, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "79b222cf-27f3-4632-a939-a410031d5175", "db_session_id": "YPE45W6AWF8O2QIWIL81", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793010499258, "job": 1, "event": "recovery_finished"} 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f8d4967c00 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: DB pointer 0x55f8d975a000 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** DB Stats ** 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T06:56:50.656 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: L0 2/0 98.26 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 54.3 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: Sum 2/0 98.26 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 54.3 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 54.3 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-05-27T06:56:50.657 INFO:journalctl@ceph.mon.a.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 54.3 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative compaction: 0.00 GB write, 8.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval compaction: 0.00 GB write, 8.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T06:56:50.658 INFO:journalctl@ceph.mon.a.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout: Block cache BinnedLRUCache@0x55f8d49831f0#2 capacity: 512.00 MB usage: 1.41 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.91 KB,0.000172853%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: starting mon.a rank 0 at public addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] at bind addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: mon.a@-1(???) e1 preinit fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: mon.a@-1(???).mds e1 new map 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: mon.a@-1(???).mds e1 print_map 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout: e1 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout: legacy client fscid: -1 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-27T06:56:50.659 INFO:journalctl@ceph.mon.a.smithi043.stdout: No filesystems configured 2024-05-27T06:56:50.660 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T06:56:50.660 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T06:56:50.660 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T06:56:50.660 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T06:56:50.660 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-27T06:56:50.660 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-27T06:56:50.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: monmap epoch 1 2024-05-27T06:56:50.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:56:50.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: last_changed 2024-05-27T06:56:39.161167+0000 2024-05-27T06:56:50.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: created 2024-05-27T06:56:39.161167+0000 2024-05-27T06:56:50.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: min_mon_release 19 (squid) 2024-05-27T06:56:50.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: election_strategy: 1 2024-05-27T06:56:50.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-05-27T06:56:50.662 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: fsmap 2024-05-27T06:56:50.662 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: osdmap e1: 0 total, 0 up, 0 in 2024-05-27T06:56:50.662 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:50 smithi043 ceph-mon[31531]: mgrmap e1: no daemons active 2024-05-27T06:56:51.986 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.287+0000 7ffb2c600700 1 Processor -- start 2024-05-27T06:56:51.987 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.287+0000 7ffb2c600700 1 -- start start 2024-05-27T06:56:51.987 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.288+0000 7ffb2c600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2407d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:51.987 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.288+0000 7ffb2c600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2407d700 con 0x7ffb2407ecf0 2024-05-27T06:56:51.987 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.288+0000 7ffb29c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2407d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:51.988 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.288+0000 7ffb29c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2407d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47762/0 (socket says 172.21.15.43:47762) 2024-05-27T06:56:51.988 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.288+0000 7ffb29c00700 1 -- 172.21.15.43:0/1507527467 learned_addr learned my addr 172.21.15.43:0/1507527467 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:51.988 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.289+0000 7ffb29c00700 1 -- 172.21.15.43:0/1507527467 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2407d880 con 0x7ffb2407ecf0 2024-05-27T06:56:51.988 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.289+0000 7ffb29c00700 1 --2- 172.21.15.43:0/1507527467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2407d1c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009e80 tx=0x7ffb1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d01bd40468de93 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:51.988 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.290+0000 7ffb23e00700 1 -- 172.21.15.43:0/1507527467 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb18017070 con 0x7ffb2407ecf0 2024-05-27T06:56:51.988 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.290+0000 7ffb23e00700 1 -- 172.21.15.43:0/1507527467 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7ffb1800b8c0 con 0x7ffb2407ecf0 2024-05-27T06:56:51.988 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.290+0000 7ffb23e00700 1 -- 172.21.15.43:0/1507527467 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb18004420 con 0x7ffb2407ecf0 2024-05-27T06:56:51.990 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.290+0000 7ffb2c600700 1 -- 172.21.15.43:0/1507527467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 msgr2=0x7ffb2407d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:51.990 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.290+0000 7ffb2c600700 1 --2- 172.21.15.43:0/1507527467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2407d1c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009e80 tx=0x7ffb1800b330 comp rx=0 tx=0).stop 2024-05-27T06:56:51.990 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.291+0000 7ffb2c600700 1 -- 172.21.15.43:0/1507527467 shutdown_connections 2024-05-27T06:56:51.990 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.291+0000 7ffb2c600700 1 --2- 172.21.15.43:0/1507527467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2407d1c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.990 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.291+0000 7ffb2c600700 1 -- 172.21.15.43:0/1507527467 >> 172.21.15.43:0/1507527467 conn(0x7ffb24105100 msgr2=0x7ffb24107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:51.990 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.291+0000 7ffb2c600700 1 -- 172.21.15.43:0/1507527467 shutdown_connections 2024-05-27T06:56:51.990 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.291+0000 7ffb2c600700 1 -- 172.21.15.43:0/1507527467 wait complete. 2024-05-27T06:56:51.990 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.292+0000 7ffb2c600700 1 Processor -- start 2024-05-27T06:56:51.990 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.292+0000 7ffb2c600700 1 -- start start 2024-05-27T06:56:51.991 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.292+0000 7ffb2c600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2419fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:51.991 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.292+0000 7ffb2c600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2410fec0 con 0x7ffb2407ecf0 2024-05-27T06:56:51.991 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.293+0000 7ffb29c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2419fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:51.991 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.293+0000 7ffb29c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2419fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47776/0 (socket says 172.21.15.43:47776) 2024-05-27T06:56:51.991 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.293+0000 7ffb29c00700 1 -- 172.21.15.43:0/1808015473 learned_addr learned my addr 172.21.15.43:0/1808015473 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:51.991 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.293+0000 7ffb29c00700 1 -- 172.21.15.43:0/1808015473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb241a0380 con 0x7ffb2407ecf0 2024-05-27T06:56:51.991 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.294+0000 7ffb29c00700 1 --2- 172.21.15.43:0/1808015473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2419fe40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb18004000 tx=0x7ffb18003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:51.991 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.294+0000 7ffb22000700 1 -- 172.21.15.43:0/1808015473 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb18020020 con 0x7ffb2407ecf0 2024-05-27T06:56:51.991 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.294+0000 7ffb22000700 1 -- 172.21.15.43:0/1808015473 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7ffb1800be50 con 0x7ffb2407ecf0 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.294+0000 7ffb22000700 1 -- 172.21.15.43:0/1808015473 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb18017040 con 0x7ffb2407ecf0 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.294+0000 7ffb2c600700 1 -- 172.21.15.43:0/1808015473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb241a0610 con 0x7ffb2407ecf0 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.294+0000 7ffb2c600700 1 -- 172.21.15.43:0/1808015473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb241a3220 con 0x7ffb2407ecf0 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.294+0000 7ffb22000700 1 -- 172.21.15.43:0/1808015473 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffb18023020 con 0x7ffb2407ecf0 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.295+0000 7ffb22000700 1 -- 172.21.15.43:0/1808015473 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ffb1800f050 con 0x7ffb2407ecf0 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.295+0000 7ffb2c600700 1 -- 172.21.15.43:0/1808015473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb24064e10 con 0x7ffb2407ecf0 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.299+0000 7ffb22000700 1 -- 172.21.15.43:0/1808015473 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7ffb1802a020 con 0x7ffb2407ecf0 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.376+0000 7ffb2c600700 1 -- 172.21.15.43:0/1808015473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7ffb241a33f0 con 0x7ffb2407ecf0 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.381+0000 7ffb22000700 1 -- 172.21.15.43:0/1808015473 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7ffb1800b860 con 0x7ffb2407ecf0 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.382+0000 7ffb2c600700 1 -- 172.21.15.43:0/1808015473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 msgr2=0x7ffb2419fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.382+0000 7ffb2c600700 1 --2- 172.21.15.43:0/1808015473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2419fe40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb18004000 tx=0x7ffb18003d00 comp rx=0 tx=0).stop 2024-05-27T06:56:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.383+0000 7ffb2c600700 1 -- 172.21.15.43:0/1808015473 shutdown_connections 2024-05-27T06:56:51.994 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.383+0000 7ffb2c600700 1 --2- 172.21.15.43:0/1808015473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb2407ecf0 0x7ffb2419fe40 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.994 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.383+0000 7ffb2c600700 1 -- 172.21.15.43:0/1808015473 >> 172.21.15.43:0/1808015473 conn(0x7ffb24105100 msgr2=0x7ffb241059a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:51.994 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.383+0000 7ffb2c600700 1 -- 172.21.15.43:0/1808015473 shutdown_connections 2024-05-27T06:56:51.994 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:51.383+0000 7ffb2c600700 1 -- 172.21.15.43:0/1808015473 wait complete. 2024-05-27T06:56:51.994 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-27T06:56:52.022 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-27T06:56:52.022 INFO:teuthology.orchestra.run.smithi043.stdout:Creating mgr... 2024-05-27T06:56:52.024 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-27T06:56:52.025 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-27T06:56:52.462 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.x 2024-05-27T06:56:52.463 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.x.service: Unit ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.x.service not loaded. 2024-05-27T06:56:52.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:52 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1808015473' entity='client.admin' 2024-05-27T06:56:52.737 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968.target.wants/ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@.service. 2024-05-27T06:56:53.148 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:52 smithi043 systemd[1]: Starting Ceph mgr.x for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T06:56:53.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:53 smithi043 podman[31745]: 2024-05-27 06:56:53.059265471 +0000 UTC m=+0.033580183 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:56:53.455 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:53 smithi043 podman[31745]: 2024-05-27 06:56:53.332877128 +0000 UTC m=+0.307191876 container create 0ae7c08d9f686e3ad97fb8ca31c8075df36a3ffb0a09dba42c7136bc0405afbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T06:56:53.718 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T06:56:53.718 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-05-27T06:56:53.740 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T06:56:53.740 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-05-27T06:56:53.740 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-05-27T06:56:53.740 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to enable service . firewalld.service is not available 2024-05-27T06:56:53.759 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T06:56:53.759 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-05-27T06:56:53.775 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T06:56:53.775 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-05-27T06:56:53.775 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-05-27T06:56:53.775 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-27T06:56:53.775 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr to start... 2024-05-27T06:56:53.775 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr... 2024-05-27T06:56:53.911 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:53 smithi043 podman[31745]: 2024-05-27 06:56:53.603185255 +0000 UTC m=+0.577499983 container init 0ae7c08d9f686e3ad97fb8ca31c8075df36a3ffb0a09dba42c7136bc0405afbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T06:56:53.911 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:53 smithi043 podman[31745]: 2024-05-27 06:56:53.615339468 +0000 UTC m=+0.589654246 container start 0ae7c08d9f686e3ad97fb8ca31c8075df36a3ffb0a09dba42c7136bc0405afbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-27T06:56:53.911 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:53 smithi043 bash[31745]: 0ae7c08d9f686e3ad97fb8ca31c8075df36a3ffb0a09dba42c7136bc0405afbe 2024-05-27T06:56:53.912 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:53 smithi043 systemd[1]: Started Ceph mgr.x for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T06:56:54.190 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:53 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:53.938+0000 7f6a4afb0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T06:56:54.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:54.209+0000 7f6a4afb0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:56:54.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:54.350+0000 7f6a4afb0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T06:56:54.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:54 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1550237213' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T06:56:55.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:54.818+0000 7f6a4afb0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T06:56:55.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:54.940+0000 7f6a4afb0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T06:56:55.251 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:56:55.253 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-05-27T06:56:55.253 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T06:56:55.253 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T06:56:55.253 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T06:56:55.253 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T06:56:55.253 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T06:56:55.253 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:55.253 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T06:56:55.253 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:56:55.254 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T06:56:55.255 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T06:56:55.255 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T06:56:55.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T06:56:55.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T06:56:55.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T06:56:55.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:55.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T06:56:55.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T06:56:55.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T06:56:55.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T06:56:55.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T06:56:43.552988+0000", 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:55.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.432+0000 7fc5ccc00700 1 Processor -- start 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.432+0000 7fc5ccc00700 1 -- start start 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.433+0000 7fc5ccc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c8109e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.433+0000 7fc5ccc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c810a480 con 0x7fc5c8109a80 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.433+0000 7fc5c5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c8109e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.433+0000 7fc5c5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c8109e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37450/0 (socket says 172.21.15.43:37450) 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.433+0000 7fc5c5e00700 1 -- 172.21.15.43:0/2826888576 learned_addr learned my addr 172.21.15.43:0/2826888576 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.434+0000 7fc5c5e00700 1 -- 172.21.15.43:0/2826888576 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c810abb0 con 0x7fc5c8109a80 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.434+0000 7fc5c5e00700 1 --2- 172.21.15.43:0/2826888576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c8109e90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80098a0 tx=0x7fc5b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2072ff0239e6f2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.435+0000 7fc5c4a00700 1 -- 172.21.15.43:0/2826888576 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5b8004160 con 0x7fc5c8109a80 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.435+0000 7fc5c4a00700 1 -- 172.21.15.43:0/2826888576 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc5b8004300 con 0x7fc5c8109a80 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.435+0000 7fc5c4a00700 1 -- 172.21.15.43:0/2826888576 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5b80119e0 con 0x7fc5c8109a80 2024-05-27T06:56:55.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.436+0000 7fc5ccc00700 1 -- 172.21.15.43:0/2826888576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 msgr2=0x7fc5c8109e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.436+0000 7fc5ccc00700 1 --2- 172.21.15.43:0/2826888576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c8109e90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80098a0 tx=0x7fc5b8009bb0 comp rx=0 tx=0).stop 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.436+0000 7fc5ccc00700 1 -- 172.21.15.43:0/2826888576 shutdown_connections 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.436+0000 7fc5ccc00700 1 --2- 172.21.15.43:0/2826888576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c8109e90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.436+0000 7fc5ccc00700 1 -- 172.21.15.43:0/2826888576 >> 172.21.15.43:0/2826888576 conn(0x7fc5c8105100 msgr2=0x7fc5c8107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.436+0000 7fc5ccc00700 1 -- 172.21.15.43:0/2826888576 shutdown_connections 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.436+0000 7fc5ccc00700 1 -- 172.21.15.43:0/2826888576 wait complete. 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.437+0000 7fc5ccc00700 1 Processor -- start 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.438+0000 7fc5ccc00700 1 -- start start 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.438+0000 7fc5ccc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c819dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.438+0000 7fc5ccc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c810dde0 con 0x7fc5c8109a80 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.438+0000 7fc5c5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c819dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.438+0000 7fc5c5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c819dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37452/0 (socket says 172.21.15.43:37452) 2024-05-27T06:56:55.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.438+0000 7fc5c5e00700 1 -- 172.21.15.43:0/1550237213 learned_addr learned my addr 172.21.15.43:0/1550237213 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.439+0000 7fc5c5e00700 1 -- 172.21.15.43:0/1550237213 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c819e160 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.439+0000 7fc5c5e00700 1 --2- 172.21.15.43:0/1550237213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c819dc20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80046e0 tx=0x7fc5b8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.439+0000 7fc5bea00700 1 -- 172.21.15.43:0/1550237213 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5b80039e0 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.439+0000 7fc5bea00700 1 -- 172.21.15.43:0/1550237213 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc5b8003b80 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.439+0000 7fc5ccc00700 1 -- 172.21.15.43:0/1550237213 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c819e3f0 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.440+0000 7fc5bea00700 1 -- 172.21.15.43:0/1550237213 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5b8024700 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.440+0000 7fc5ccc00700 1 -- 172.21.15.43:0/1550237213 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c81a1000 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.440+0000 7fc5bea00700 1 -- 172.21.15.43:0/1550237213 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc5b8022020 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.440+0000 7fc5bea00700 1 -- 172.21.15.43:0/1550237213 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc5b8019530 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.441+0000 7fc5ccc00700 1 -- 172.21.15.43:0/1550237213 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c8064e10 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.444+0000 7fc5bea00700 1 -- 172.21.15.43:0/1550237213 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fc5b801ab30 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.550+0000 7fc5ccc00700 1 -- 172.21.15.43:0/1550237213 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc5c81a11d0 con 0x7fc5c8109a80 2024-05-27T06:56:55.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.551+0000 7fc5bea00700 1 -- 172.21.15.43:0/1550237213 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc5b801a430 con 0x7fc5c8109a80 2024-05-27T06:56:55.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.552+0000 7fc5ccc00700 1 -- 172.21.15.43:0/1550237213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 msgr2=0x7fc5c819dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:55.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.552+0000 7fc5ccc00700 1 --2- 172.21.15.43:0/1550237213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c819dc20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80046e0 tx=0x7fc5b8011a60 comp rx=0 tx=0).stop 2024-05-27T06:56:55.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.553+0000 7fc5ccc00700 1 -- 172.21.15.43:0/1550237213 shutdown_connections 2024-05-27T06:56:55.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.553+0000 7fc5ccc00700 1 --2- 172.21.15.43:0/1550237213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c8109a80 0x7fc5c819dc20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:55.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.553+0000 7fc5ccc00700 1 -- 172.21.15.43:0/1550237213 >> 172.21.15.43:0/1550237213 conn(0x7fc5c8105100 msgr2=0x7fc5c8105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:55.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.553+0000 7fc5ccc00700 1 -- 172.21.15.43:0/1550237213 shutdown_connections 2024-05-27T06:56:55.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:54.553+0000 7fc5ccc00700 1 -- 172.21.15.43:0/1550237213 wait complete. 2024-05-27T06:56:55.268 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (1/15)... 2024-05-27T06:56:56.887 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:56.607+0000 7f6a4afb0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T06:56:56.887 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:56.850+0000 7f6a4afb0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T06:56:57.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:57.075+0000 7f6a4afb0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T06:56:57.688 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:57.485+0000 7f6a4afb0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T06:56:57.688 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:57.617+0000 7f6a4afb0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T06:56:57.954 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:57.886+0000 7f6a4afb0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T06:56:58.409 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:56:58 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2043953573' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T06:56:58.409 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:58 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:58.168+0000 7f6a4afb0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:56:58.717 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:56:58.717 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-05-27T06:56:58.717 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T06:56:58.717 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T06:56:58.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T06:56:43.552988+0000", 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.928+0000 7f420c000700 1 Processor -- start 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.928+0000 7f420c000700 1 -- start start 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.929+0000 7f420c000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420407d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:58.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.929+0000 7f420c000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f420407d720 con 0x7f420407ed10 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.929+0000 7f4209600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420407d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.930+0000 7f4209600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420407d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37454/0 (socket says 172.21.15.43:37454) 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.930+0000 7f4209600700 1 -- 172.21.15.43:0/252062590 learned_addr learned my addr 172.21.15.43:0/252062590 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.930+0000 7f4209600700 1 -- 172.21.15.43:0/252062590 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420407d8a0 con 0x7f420407ed10 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.930+0000 7f4209600700 1 --2- 172.21.15.43:0/252062590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420407d1e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f41f80098a0 tx=0x7f41f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8006d7e282ad01f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.931+0000 7f4203e00700 1 -- 172.21.15.43:0/252062590 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41f8004160 con 0x7f420407ed10 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.931+0000 7f4203e00700 1 -- 172.21.15.43:0/252062590 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f41f8004300 con 0x7f420407ed10 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.931+0000 7f4203e00700 1 -- 172.21.15.43:0/252062590 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41f80119e0 con 0x7f420407ed10 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.932+0000 7f420c000700 1 -- 172.21.15.43:0/252062590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 msgr2=0x7f420407d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.932+0000 7f420c000700 1 --2- 172.21.15.43:0/252062590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420407d1e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f41f80098a0 tx=0x7f41f8009bb0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.932+0000 7f420c000700 1 -- 172.21.15.43:0/252062590 shutdown_connections 2024-05-27T06:56:58.723 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.932+0000 7f420c000700 1 --2- 172.21.15.43:0/252062590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420407d1e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.932+0000 7f420c000700 1 -- 172.21.15.43:0/252062590 >> 172.21.15.43:0/252062590 conn(0x7f4204105100 msgr2=0x7f4204107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.932+0000 7f420c000700 1 -- 172.21.15.43:0/252062590 shutdown_connections 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.932+0000 7f420c000700 1 -- 172.21.15.43:0/252062590 wait complete. 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.933+0000 7f420c000700 1 Processor -- start 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.933+0000 7f420c000700 1 -- start start 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.934+0000 7f420c000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420419fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.934+0000 7f420c000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f420410fea0 con 0x7f420407ed10 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.934+0000 7f4209600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420419fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.934+0000 7f4209600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420419fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37456/0 (socket says 172.21.15.43:37456) 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.934+0000 7f4209600700 1 -- 172.21.15.43:0/2043953573 learned_addr learned my addr 172.21.15.43:0/2043953573 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.934+0000 7f4209600700 1 -- 172.21.15.43:0/2043953573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42041a0320 con 0x7f420407ed10 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.935+0000 7f4209600700 1 --2- 172.21.15.43:0/2043953573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420419fde0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41f80116a0 tx=0x7f41f8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.935+0000 7f4202000700 1 -- 172.21.15.43:0/2043953573 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41f8003970 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.935+0000 7f4202000700 1 -- 172.21.15.43:0/2043953573 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f41f8003b10 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.935+0000 7f420c000700 1 -- 172.21.15.43:0/2043953573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42041a05b0 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.935+0000 7f4202000700 1 -- 172.21.15.43:0/2043953573 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41f8019550 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.935+0000 7f420c000700 1 -- 172.21.15.43:0/2043953573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42041a3180 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.935+0000 7f4202000700 1 -- 172.21.15.43:0/2043953573 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f41f8019b70 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.936+0000 7f4202000700 1 -- 172.21.15.43:0/2043953573 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f41f8024730 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.937+0000 7f420c000700 1 -- 172.21.15.43:0/2043953573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4204064e10 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:57.939+0000 7f4202000700 1 -- 172.21.15.43:0/2043953573 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f41f8024930 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:58.038+0000 7f420c000700 1 -- 172.21.15.43:0/2043953573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f42041a3350 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:58.039+0000 7f4202000700 1 -- 172.21.15.43:0/2043953573 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f41f801aa50 con 0x7f420407ed10 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:58.040+0000 7f420c000700 1 -- 172.21.15.43:0/2043953573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 msgr2=0x7f420419fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:58.040+0000 7f420c000700 1 --2- 172.21.15.43:0/2043953573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420419fde0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41f80116a0 tx=0x7f41f8011a60 comp rx=0 tx=0).stop 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:58.040+0000 7f420c000700 1 -- 172.21.15.43:0/2043953573 shutdown_connections 2024-05-27T06:56:58.725 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:58.040+0000 7f420c000700 1 --2- 172.21.15.43:0/2043953573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f420407ed10 0x7f420419fde0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:58.040+0000 7f420c000700 1 -- 172.21.15.43:0/2043953573 >> 172.21.15.43:0/2043953573 conn(0x7f4204105100 msgr2=0x7f42041058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:58.041+0000 7f420c000700 1 -- 172.21.15.43:0/2043953573 shutdown_connections 2024-05-27T06:56:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:56:58.041+0000 7f420c000700 1 -- 172.21.15.43:0/2043953573 wait complete. 2024-05-27T06:56:58.726 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (2/15)... 2024-05-27T06:56:59.301 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:59.008+0000 7f6a4afb0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T06:56:59.704 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:59.308+0000 7f6a4afb0200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T06:57:00.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:56:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:56:59.885+0000 7f6a4afb0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T06:57:00.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:00.017+0000 7f6a4afb0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T06:57:00.867 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:00.499+0000 7f6a4afb0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T06:57:01.119 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:00.978+0000 7f6a4afb0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T06:57:01.120 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:01.111+0000 7f6a4afb0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T06:57:01.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:01 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2401790005' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T06:57:01.704 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:01.460+0000 7f6a4afb0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T06:57:01.704 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:01.652+0000 7f6a4afb0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T06:57:02.196 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:01.786+0000 7f6a4afb0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T06:57:02.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:57:02.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-05-27T06:57:02.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T06:57:02.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-05-27T06:57:02.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:02.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T06:57:02.270 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T06:57:02.271 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T06:56:43.552988+0000", 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.356+0000 7f08ef000700 1 Processor -- start 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.357+0000 7f08ef000700 1 -- start start 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.357+0000 7f08ef000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e807d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.357+0000 7f08ef000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f08e807d700 con 0x7f08e807ecf0 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.357+0000 7f08e7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e807d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.357+0000 7f08e7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e807d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37464/0 (socket says 172.21.15.43:37464) 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.357+0000 7f08e7e00700 1 -- 172.21.15.43:0/546294149 learned_addr learned my addr 172.21.15.43:0/546294149 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:02.272 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.358+0000 7f08e7e00700 1 -- 172.21.15.43:0/546294149 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e807d880 con 0x7f08e807ecf0 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.358+0000 7f08e7e00700 1 --2- 172.21.15.43:0/546294149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e807d1c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f08d80098a0 tx=0x7f08d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f528afecb28a4c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.358+0000 7f08e6a00700 1 -- 172.21.15.43:0/546294149 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08d8004160 con 0x7f08e807ecf0 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.359+0000 7f08e6a00700 1 -- 172.21.15.43:0/546294149 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f08d8004300 con 0x7f08e807ecf0 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.359+0000 7f08e6a00700 1 -- 172.21.15.43:0/546294149 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08d80119e0 con 0x7f08e807ecf0 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.359+0000 7f08ef000700 1 -- 172.21.15.43:0/546294149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 msgr2=0x7f08e807d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.359+0000 7f08ef000700 1 --2- 172.21.15.43:0/546294149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e807d1c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f08d80098a0 tx=0x7f08d8009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.359+0000 7f08ef000700 1 -- 172.21.15.43:0/546294149 shutdown_connections 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.359+0000 7f08ef000700 1 --2- 172.21.15.43:0/546294149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e807d1c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.359+0000 7f08ef000700 1 -- 172.21.15.43:0/546294149 >> 172.21.15.43:0/546294149 conn(0x7f08e8105100 msgr2=0x7f08e8107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.360+0000 7f08ef000700 1 -- 172.21.15.43:0/546294149 shutdown_connections 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.360+0000 7f08ef000700 1 -- 172.21.15.43:0/546294149 wait complete. 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.361+0000 7f08ef000700 1 Processor -- start 2024-05-27T06:57:02.273 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.361+0000 7f08ef000700 1 -- start start 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.361+0000 7f08ef000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e819fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.361+0000 7f08ef000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f08e810fec0 con 0x7f08e807ecf0 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.362+0000 7f08e7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e819fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.362+0000 7f08e7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e819fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37470/0 (socket says 172.21.15.43:37470) 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.362+0000 7f08e7e00700 1 -- 172.21.15.43:0/2401790005 learned_addr learned my addr 172.21.15.43:0/2401790005 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.362+0000 7f08e7e00700 1 -- 172.21.15.43:0/2401790005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e81a0320 con 0x7f08e807ecf0 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.362+0000 7f08e7e00700 1 --2- 172.21.15.43:0/2401790005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e819fde0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f08d8003f20 tx=0x7f08d8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.363+0000 7f08e4c00700 1 -- 172.21.15.43:0/2401790005 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08d80038b0 con 0x7f08e807ecf0 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.363+0000 7f08e4c00700 1 -- 172.21.15.43:0/2401790005 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f08d8003a50 con 0x7f08e807ecf0 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.363+0000 7f08ef000700 1 -- 172.21.15.43:0/2401790005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e81a05b0 con 0x7f08e807ecf0 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.363+0000 7f08e4c00700 1 -- 172.21.15.43:0/2401790005 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08d8019550 con 0x7f08e807ecf0 2024-05-27T06:57:02.274 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.363+0000 7f08ef000700 1 -- 172.21.15.43:0/2401790005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e81a3180 con 0x7f08e807ecf0 2024-05-27T06:57:02.275 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.363+0000 7f08e4c00700 1 -- 172.21.15.43:0/2401790005 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f08d8019be0 con 0x7f08e807ecf0 2024-05-27T06:57:02.275 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.363+0000 7f08e4c00700 1 -- 172.21.15.43:0/2401790005 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f08d8024730 con 0x7f08e807ecf0 2024-05-27T06:57:02.275 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.364+0000 7f08ef000700 1 -- 172.21.15.43:0/2401790005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08e8064e10 con 0x7f08e807ecf0 2024-05-27T06:57:02.275 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.367+0000 7f08e4c00700 1 -- 172.21.15.43:0/2401790005 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f08d8043ec0 con 0x7f08e807ecf0 2024-05-27T06:57:02.275 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.447+0000 7f08ef000700 1 -- 172.21.15.43:0/2401790005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f08e81a3350 con 0x7f08e807ecf0 2024-05-27T06:57:02.275 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.448+0000 7f08e4c00700 1 -- 172.21.15.43:0/2401790005 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f08d8032030 con 0x7f08e807ecf0 2024-05-27T06:57:02.275 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.449+0000 7f08ef000700 1 -- 172.21.15.43:0/2401790005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 msgr2=0x7f08e819fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:02.275 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.450+0000 7f08ef000700 1 --2- 172.21.15.43:0/2401790005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e819fde0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f08d8003f20 tx=0x7f08d8011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:02.276 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.450+0000 7f08ef000700 1 -- 172.21.15.43:0/2401790005 shutdown_connections 2024-05-27T06:57:02.276 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.450+0000 7f08ef000700 1 --2- 172.21.15.43:0/2401790005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08e807ecf0 0x7f08e819fde0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:02.276 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.450+0000 7f08ef000700 1 -- 172.21.15.43:0/2401790005 >> 172.21.15.43:0/2401790005 conn(0x7f08e8105100 msgr2=0x7f08e8105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:02.276 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.450+0000 7f08ef000700 1 -- 172.21.15.43:0/2401790005 shutdown_connections 2024-05-27T06:57:02.276 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:01.450+0000 7f08ef000700 1 -- 172.21.15.43:0/2401790005 wait complete. 2024-05-27T06:57:02.276 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (3/15)... 2024-05-27T06:57:02.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:02.202+0000 7f6a4afb0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T06:57:02.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:02.338+0000 7f6a4afb0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T06:57:03.426 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:03.134+0000 7f6a4afb0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T06:57:03.704 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:03.432+0000 7f6a4afb0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T06:57:03.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: Activating manager daemon x 2024-05-27T06:57:03.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: mgrmap e2: x(active, starting, since 0.00608666s) 2024-05-27T06:57:03.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: from='mgr.14100 172.21.15.43:0/2058419450' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T06:57:03.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: from='mgr.14100 172.21.15.43:0/2058419450' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T06:57:03.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: from='mgr.14100 172.21.15.43:0/2058419450' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T06:57:03.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: from='mgr.14100 172.21.15.43:0/2058419450' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T06:57:03.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: from='mgr.14100 172.21.15.43:0/2058419450' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T06:57:03.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: Manager daemon x is now available 2024-05-27T06:57:03.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: from='mgr.14100 172.21.15.43:0/2058419450' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T06:57:03.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: from='mgr.14100 172.21.15.43:0/2058419450' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T06:57:03.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:03 smithi043 ceph-mon[31531]: from='mgr.14100 172.21.15.43:0/2058419450' entity='mgr.x' 2024-05-27T06:57:04.692 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:04 smithi043 ceph-mon[31531]: from='mgr.14100 172.21.15.43:0/2058419450' entity='mgr.x' 2024-05-27T06:57:04.693 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:04 smithi043 ceph-mon[31531]: from='mgr.14100 172.21.15.43:0/2058419450' entity='mgr.x' 2024-05-27T06:57:04.693 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:04 smithi043 ceph-mon[31531]: mgrmap e3: x(active, since 1.02225s) 2024-05-27T06:57:05.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:05 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3742994780' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-27T06:57:06.056 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:57:06.056 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-05-27T06:57:06.056 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T06:57:06.056 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-05-27T06:57:06.057 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-27T06:57:06.057 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-27T06:57:06.057 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-27T06:57:06.057 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:06.057 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-27T06:57:06.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:06.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-27T06:57:06.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:57:06.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-27T06:57:06.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-27T06:57:06.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-27T06:57:06.068 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-27T06:57:06.068 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-27T06:57:06.068 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-27T06:57:06.068 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:06.068 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-27T06:57:06.068 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-27T06:57:06.068 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-27T06:57:06.068 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-27T06:57:06.068 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-27T06:57:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-05-27T06:56:43.552988+0000", 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-27T06:57:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.941+0000 7fcc96800700 1 Processor -- start 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.941+0000 7fcc96800700 1 -- start start 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.942+0000 7fcc96800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9010c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.942+0000 7fcc96800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9010c9d0 con 0x7fcc9007cbb0 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.942+0000 7fcc95400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9010c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.942+0000 7fcc95400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9010c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50824/0 (socket says 172.21.15.43:50824) 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.942+0000 7fcc95400700 1 -- 172.21.15.43:0/280586712 learned_addr learned my addr 172.21.15.43:0/280586712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.943+0000 7fcc95400700 1 -- 172.21.15.43:0/280586712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9010cb50 con 0x7fcc9007cbb0 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.943+0000 7fcc95400700 1 --2- 172.21.15.43:0/280586712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9010c400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcc840098a0 tx=0x7fcc84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3fd14f5c8e55b0c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.943+0000 7fcc8fe00700 1 -- 172.21.15.43:0/280586712 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc84004160 con 0x7fcc9007cbb0 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.943+0000 7fcc8fe00700 1 -- 172.21.15.43:0/280586712 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fcc84004300 con 0x7fcc9007cbb0 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.943+0000 7fcc8fe00700 1 -- 172.21.15.43:0/280586712 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc840119e0 con 0x7fcc9007cbb0 2024-05-27T06:57:06.071 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.944+0000 7fcc96800700 1 -- 172.21.15.43:0/280586712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 msgr2=0x7fcc9010c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.944+0000 7fcc96800700 1 --2- 172.21.15.43:0/280586712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9010c400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcc840098a0 tx=0x7fcc84009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.944+0000 7fcc96800700 1 -- 172.21.15.43:0/280586712 shutdown_connections 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.944+0000 7fcc96800700 1 --2- 172.21.15.43:0/280586712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9010c400 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.944+0000 7fcc96800700 1 -- 172.21.15.43:0/280586712 >> 172.21.15.43:0/280586712 conn(0x7fcc90103db0 msgr2=0x7fcc90106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.944+0000 7fcc96800700 1 -- 172.21.15.43:0/280586712 shutdown_connections 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.945+0000 7fcc96800700 1 -- 172.21.15.43:0/280586712 wait complete. 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.946+0000 7fcc96800700 1 Processor -- start 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.946+0000 7fcc96800700 1 -- start start 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.946+0000 7fcc96800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.946+0000 7fcc96800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9010daa0 con 0x7fcc9007cbb0 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.946+0000 7fcc95400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.946+0000 7fcc95400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9019fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50838/0 (socket says 172.21.15.43:50838) 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.946+0000 7fcc95400700 1 -- 172.21.15.43:0/3742994780 learned_addr learned my addr 172.21.15.43:0/3742994780 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:06.072 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.947+0000 7fcc95400700 1 -- 172.21.15.43:0/3742994780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc901a01c0 con 0x7fcc9007cbb0 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.947+0000 7fcc95400700 1 --2- 172.21.15.43:0/3742994780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9019fc80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcc840116a0 tx=0x7fcc84011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.947+0000 7fcc8e000700 1 -- 172.21.15.43:0/3742994780 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc84003970 con 0x7fcc9007cbb0 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.947+0000 7fcc8e000700 1 -- 172.21.15.43:0/3742994780 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fcc84003b10 con 0x7fcc9007cbb0 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.947+0000 7fcc8e000700 1 -- 172.21.15.43:0/3742994780 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc84019550 con 0x7fcc9007cbb0 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.947+0000 7fcc96800700 1 -- 172.21.15.43:0/3742994780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc901a0450 con 0x7fcc9007cbb0 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.948+0000 7fcc96800700 1 -- 172.21.15.43:0/3742994780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc901a3060 con 0x7fcc9007cbb0 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.948+0000 7fcc96800700 1 -- 172.21.15.43:0/3742994780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc90064e10 con 0x7fcc9007cbb0 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.949+0000 7fcc8e000700 1 -- 172.21.15.43:0/3742994780 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 3) ==== 50105+0+0 (secure 0 0 0) 0x7fcc84019810 con 0x7fcc9007cbb0 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.949+0000 7fcc8e000700 1 --2- 172.21.15.43:0/3742994780 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fcc7803e300 0x7fcc78040810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.949+0000 7fcc8e000700 1 -- 172.21.15.43:0/3742994780 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcc84053730 con 0x7fcc9007cbb0 2024-05-27T06:57:06.073 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.950+0000 7fcc94a00700 1 --2- 172.21.15.43:0/3742994780 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fcc7803e300 0x7fcc78040810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.951+0000 7fcc94a00700 1 --2- 172.21.15.43:0/3742994780 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fcc7803e300 0x7fcc78040810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c006fd0 tx=0x7fcc7c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:04.956+0000 7fcc8e000700 1 -- 172.21.15.43:0/3742994780 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcc84019b00 con 0x7fcc9007cbb0 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.287+0000 7fcc96800700 1 -- 172.21.15.43:0/3742994780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcc901a36e0 con 0x7fcc9007cbb0 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.288+0000 7fcc8e000700 1 -- 172.21.15.43:0/3742994780 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fcc84010380 con 0x7fcc9007cbb0 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.291+0000 7fcc96800700 1 -- 172.21.15.43:0/3742994780 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fcc7803e300 msgr2=0x7fcc78040810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.291+0000 7fcc96800700 1 --2- 172.21.15.43:0/3742994780 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fcc7803e300 0x7fcc78040810 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c006fd0 tx=0x7fcc7c006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.291+0000 7fcc96800700 1 -- 172.21.15.43:0/3742994780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 msgr2=0x7fcc9019fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.291+0000 7fcc96800700 1 --2- 172.21.15.43:0/3742994780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9019fc80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcc840116a0 tx=0x7fcc84011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.291+0000 7fcc96800700 1 -- 172.21.15.43:0/3742994780 shutdown_connections 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.291+0000 7fcc96800700 1 --2- 172.21.15.43:0/3742994780 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fcc7803e300 0x7fcc78040810 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:06.074 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.291+0000 7fcc96800700 1 --2- 172.21.15.43:0/3742994780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc9007cbb0 0x7fcc9019fc80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:06.075 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.291+0000 7fcc96800700 1 -- 172.21.15.43:0/3742994780 >> 172.21.15.43:0/3742994780 conn(0x7fcc90103db0 msgr2=0x7fcc90108310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:06.075 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.291+0000 7fcc96800700 1 -- 172.21.15.43:0/3742994780 shutdown_connections 2024-05-27T06:57:06.075 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:05.291+0000 7fcc96800700 1 -- 172.21.15.43:0/3742994780 wait complete. 2024-05-27T06:57:06.075 INFO:teuthology.orchestra.run.smithi043.stdout:mgr is available 2024-05-27T06:57:06.647 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:06 smithi043 ceph-mon[31531]: mgrmap e4: x(active, since 2s) 2024-05-27T06:57:07.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:07 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3586458297' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-27T06:57:07.612 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:57:07.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [global] 2024-05-27T06:57:07.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout fsid = 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:57:07.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-27T06:57:07.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-27T06:57:07.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-27T06:57:07.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-27T06:57:07.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-27T06:57:07.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:57:07.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-27T06:57:07.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-27T06:57:07.619 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-05-27T06:57:07.619 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [osd] 2024-05-27T06:57:07.619 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-27T06:57:07.619 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-27T06:57:07.619 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-27T06:57:07.619 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-27T06:57:07.619 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-27T06:57:07.619 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.761+0000 7fbf0a000700 1 Processor -- start 2024-05-27T06:57:07.619 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.762+0000 7fbf0a000700 1 -- start start 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.762+0000 7fbf0a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0410c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.762+0000 7fbf0a000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0410c610 con 0x7fbf0410bc10 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.762+0000 7fbf03400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0410c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.762+0000 7fbf03400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0410c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50852/0 (socket says 172.21.15.43:50852) 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.762+0000 7fbf03400700 1 -- 172.21.15.43:0/2731687207 learned_addr learned my addr 172.21.15.43:0/2731687207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.763+0000 7fbf03400700 1 -- 172.21.15.43:0/2731687207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0410cd40 con 0x7fbf0410bc10 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.763+0000 7fbf03400700 1 --2- 172.21.15.43:0/2731687207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0410c020 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbef40098a0 tx=0x7fbef4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f42984b9d2f4bb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.763+0000 7fbf02000700 1 -- 172.21.15.43:0/2731687207 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbef4004160 con 0x7fbf0410bc10 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.763+0000 7fbf02000700 1 -- 172.21.15.43:0/2731687207 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fbef4004300 con 0x7fbf0410bc10 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.763+0000 7fbf02000700 1 -- 172.21.15.43:0/2731687207 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbef40119e0 con 0x7fbf0410bc10 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.764+0000 7fbf0a000700 1 -- 172.21.15.43:0/2731687207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 msgr2=0x7fbf0410c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.764+0000 7fbf0a000700 1 --2- 172.21.15.43:0/2731687207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0410c020 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbef40098a0 tx=0x7fbef4009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.764+0000 7fbf0a000700 1 -- 172.21.15.43:0/2731687207 shutdown_connections 2024-05-27T06:57:07.620 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.764+0000 7fbf0a000700 1 --2- 172.21.15.43:0/2731687207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0410c020 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.764+0000 7fbf0a000700 1 -- 172.21.15.43:0/2731687207 >> 172.21.15.43:0/2731687207 conn(0x7fbf0407f130 msgr2=0x7fbf04109ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.764+0000 7fbf0a000700 1 -- 172.21.15.43:0/2731687207 shutdown_connections 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.765+0000 7fbf0a000700 1 -- 172.21.15.43:0/2731687207 wait complete. 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.765+0000 7fbf0a000700 1 Processor -- start 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.766+0000 7fbf0a000700 1 -- start start 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.766+0000 7fbf0a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0419fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.766+0000 7fbf0a000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0410ff70 con 0x7fbf0410bc10 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.766+0000 7fbf03400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0419fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.766+0000 7fbf03400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0419fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50862/0 (socket says 172.21.15.43:50862) 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.766+0000 7fbf03400700 1 -- 172.21.15.43:0/3586458297 learned_addr learned my addr 172.21.15.43:0/3586458297 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.767+0000 7fbf03400700 1 -- 172.21.15.43:0/3586458297 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf041a03a0 con 0x7fbf0410bc10 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.767+0000 7fbf03400700 1 --2- 172.21.15.43:0/3586458297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0419fe60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009fb0 tx=0x7fbef4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:07.621 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.767+0000 7fbefbe00700 1 -- 172.21.15.43:0/3586458297 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbef40038b0 con 0x7fbf0410bc10 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.767+0000 7fbefbe00700 1 -- 172.21.15.43:0/3586458297 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fbef4003a50 con 0x7fbf0410bc10 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.767+0000 7fbf0a000700 1 -- 172.21.15.43:0/3586458297 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf041a0630 con 0x7fbf0410bc10 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.767+0000 7fbefbe00700 1 -- 172.21.15.43:0/3586458297 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbef401a5c0 con 0x7fbf0410bc10 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.767+0000 7fbf0a000700 1 -- 172.21.15.43:0/3586458297 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf041a3240 con 0x7fbf0410bc10 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.769+0000 7fbf0a000700 1 -- 172.21.15.43:0/3586458297 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf04064e10 con 0x7fbf0410bc10 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.769+0000 7fbefbe00700 1 -- 172.21.15.43:0/3586458297 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7fbef401a8c0 con 0x7fbf0410bc10 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.769+0000 7fbefbe00700 1 --2- 172.21.15.43:0/3586458297 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fbee803e3d0 0x7fbee80408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.769+0000 7fbefbe00700 1 -- 172.21.15.43:0/3586458297 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbef4053b90 con 0x7fbf0410bc10 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.770+0000 7fbf02a00700 1 --2- 172.21.15.43:0/3586458297 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fbee803e3d0 0x7fbee80408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.770+0000 7fbf02a00700 1 --2- 172.21.15.43:0/3586458297 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fbee803e3d0 0x7fbee80408e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbef0006fd0 tx=0x7fbef0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.775+0000 7fbefbe00700 1 -- 172.21.15.43:0/3586458297 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbef4017e90 con 0x7fbf0410bc10 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:06.998+0000 7fbf0a000700 1 -- 172.21.15.43:0/3586458297 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fbf041a4a40 con 0x7fbf0410bc10 2024-05-27T06:57:07.622 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.000+0000 7fbefbe00700 1 -- 172.21.15.43:0/3586458297 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7fbef402c350 con 0x7fbf0410bc10 2024-05-27T06:57:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.003+0000 7fbf0a000700 1 -- 172.21.15.43:0/3586458297 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fbee803e3d0 msgr2=0x7fbee80408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.003+0000 7fbf0a000700 1 --2- 172.21.15.43:0/3586458297 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fbee803e3d0 0x7fbee80408e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbef0006fd0 tx=0x7fbef0006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.003+0000 7fbf0a000700 1 -- 172.21.15.43:0/3586458297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 msgr2=0x7fbf0419fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.003+0000 7fbf0a000700 1 --2- 172.21.15.43:0/3586458297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0419fe60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009fb0 tx=0x7fbef4011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.003+0000 7fbf0a000700 1 -- 172.21.15.43:0/3586458297 shutdown_connections 2024-05-27T06:57:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.003+0000 7fbf0a000700 1 --2- 172.21.15.43:0/3586458297 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fbee803e3d0 0x7fbee80408e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.003+0000 7fbf0a000700 1 --2- 172.21.15.43:0/3586458297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0410bc10 0x7fbf0419fe60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.003+0000 7fbf0a000700 1 -- 172.21.15.43:0/3586458297 >> 172.21.15.43:0/3586458297 conn(0x7fbf0407f130 msgr2=0x7fbf04109040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.004+0000 7fbf0a000700 1 -- 172.21.15.43:0/3586458297 shutdown_connections 2024-05-27T06:57:07.623 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:07.004+0000 7fbf0a000700 1 -- 172.21.15.43:0/3586458297 wait complete. 2024-05-27T06:57:07.624 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling cephadm module... 2024-05-27T06:57:08.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:08 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/118283944' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-27T06:57:08.955 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:08 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: ignoring --setuser ceph since I am not root 2024-05-27T06:57:08.955 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:08 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: ignoring --setgroup ceph since I am not root 2024-05-27T06:57:09.274 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:09 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:09.027+0000 7fd49426d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T06:57:09.274 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:09 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:09.280+0000 7fd49426d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:57:09.376 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.285+0000 7f01d7e00700 1 Processor -- start 2024-05-27T06:57:09.376 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.285+0000 7f01d7e00700 1 -- start start 2024-05-27T06:57:09.376 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.285+0000 7f01d7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d810be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.285+0000 7f01d7e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f01d810c430 con 0x7f01d810ba30 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.286+0000 7f01d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d810be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.286+0000 7f01d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d810be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50868/0 (socket says 172.21.15.43:50868) 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.286+0000 7f01d6a00700 1 -- 172.21.15.43:0/276290540 learned_addr learned my addr 172.21.15.43:0/276290540 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.287+0000 7f01d6a00700 1 -- 172.21.15.43:0/276290540 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d810cb60 con 0x7f01d810ba30 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.287+0000 7f01d6a00700 1 --2- 172.21.15.43:0/276290540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d810be40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01c80098a0 tx=0x7f01c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04bfe5a59239e48 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.287+0000 7f01d5600700 1 -- 172.21.15.43:0/276290540 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01c8004160 con 0x7f01d810ba30 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.287+0000 7f01d5600700 1 -- 172.21.15.43:0/276290540 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f01c8004300 con 0x7f01d810ba30 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.287+0000 7f01d5600700 1 -- 172.21.15.43:0/276290540 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01c80119e0 con 0x7f01d810ba30 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.288+0000 7f01d7e00700 1 -- 172.21.15.43:0/276290540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 msgr2=0x7f01d810be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.288+0000 7f01d7e00700 1 --2- 172.21.15.43:0/276290540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d810be40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01c80098a0 tx=0x7f01c8009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.288+0000 7f01d7e00700 1 -- 172.21.15.43:0/276290540 shutdown_connections 2024-05-27T06:57:09.377 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.288+0000 7f01d7e00700 1 --2- 172.21.15.43:0/276290540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d810be40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.288+0000 7f01d7e00700 1 -- 172.21.15.43:0/276290540 >> 172.21.15.43:0/276290540 conn(0x7f01d807eed0 msgr2=0x7f01d807f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.288+0000 7f01d7e00700 1 -- 172.21.15.43:0/276290540 shutdown_connections 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.289+0000 7f01d7e00700 1 -- 172.21.15.43:0/276290540 wait complete. 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.290+0000 7f01d7e00700 1 Processor -- start 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.290+0000 7f01d7e00700 1 -- start start 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.290+0000 7f01d7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.290+0000 7f01d7e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f01d810fd90 con 0x7f01d810ba30 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.290+0000 7f01d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.290+0000 7f01d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d819fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50882/0 (socket says 172.21.15.43:50882) 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.290+0000 7f01d6a00700 1 -- 172.21.15.43:0/118283944 learned_addr learned my addr 172.21.15.43:0/118283944 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.291+0000 7f01d6a00700 1 -- 172.21.15.43:0/118283944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d81a01f0 con 0x7f01d810ba30 2024-05-27T06:57:09.378 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.291+0000 7f01d6a00700 1 --2- 172.21.15.43:0/118283944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d819fcb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01c80046e0 tx=0x7f01c8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.291+0000 7f01cf400700 1 -- 172.21.15.43:0/118283944 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01c80039e0 con 0x7f01d810ba30 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.292+0000 7f01cf400700 1 -- 172.21.15.43:0/118283944 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f01c8003b80 con 0x7f01d810ba30 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.292+0000 7f01d7e00700 1 -- 172.21.15.43:0/118283944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d81a0480 con 0x7f01d810ba30 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.292+0000 7f01cf400700 1 -- 172.21.15.43:0/118283944 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01c8024700 con 0x7f01d810ba30 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.292+0000 7f01d7e00700 1 -- 172.21.15.43:0/118283944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d81a3090 con 0x7f01d810ba30 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.293+0000 7f01d7e00700 1 -- 172.21.15.43:0/118283944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d8064e10 con 0x7f01d810ba30 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.293+0000 7f01cf400700 1 -- 172.21.15.43:0/118283944 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f01c8022020 con 0x7f01d810ba30 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.294+0000 7f01cf400700 1 --2- 172.21.15.43:0/118283944 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f01bc03e470 0x7f01bc040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.294+0000 7f01cf400700 1 -- 172.21.15.43:0/118283944 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f01c80535a0 con 0x7f01d810ba30 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.294+0000 7f01d6000700 1 --2- 172.21.15.43:0/118283944 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f01bc03e470 0x7f01bc040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.295+0000 7f01d6000700 1 --2- 172.21.15.43:0/118283944 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f01bc03e470 0x7f01bc040980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01c4006fd0 tx=0x7f01c4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.300+0000 7f01cf400700 1 -- 172.21.15.43:0/118283944 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f01c800fdd0 con 0x7f01d810ba30 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.599+0000 7f01d7e00700 1 -- 172.21.15.43:0/118283944 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f01d81a3710 con 0x7f01d810ba30 2024-05-27T06:57:09.379 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.653+0000 7f01cf400700 1 -- 172.21.15.43:0/118283944 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f01c8053300 con 0x7f01d810ba30 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.657+0000 7f01cf400700 1 -- 172.21.15.43:0/118283944 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f01c801ab30 con 0x7f01d810ba30 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.657+0000 7f01d7e00700 1 -- 172.21.15.43:0/118283944 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f01bc03e470 msgr2=0x7f01bc040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.657+0000 7f01d7e00700 1 --2- 172.21.15.43:0/118283944 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f01bc03e470 0x7f01bc040980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01c4006fd0 tx=0x7f01c4006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.657+0000 7f01d7e00700 1 -- 172.21.15.43:0/118283944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 msgr2=0x7f01d819fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.657+0000 7f01d7e00700 1 --2- 172.21.15.43:0/118283944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d819fcb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01c80046e0 tx=0x7f01c8011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.658+0000 7f01d7e00700 1 -- 172.21.15.43:0/118283944 shutdown_connections 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.658+0000 7f01d7e00700 1 --2- 172.21.15.43:0/118283944 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f01bc03e470 0x7f01bc040980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.658+0000 7f01d7e00700 1 --2- 172.21.15.43:0/118283944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d810ba30 0x7f01d819fcb0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.658+0000 7f01d7e00700 1 -- 172.21.15.43:0/118283944 >> 172.21.15.43:0/118283944 conn(0x7f01d807eed0 msgr2=0x7f01d8108a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.658+0000 7f01d7e00700 1 -- 172.21.15.43:0/118283944 shutdown_connections 2024-05-27T06:57:09.380 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:08.658+0000 7f01d7e00700 1 -- 172.21.15.43:0/118283944 wait complete. 2024-05-27T06:57:09.526 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:09 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:09.422+0000 7fd49426d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T06:57:09.810 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:09 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/118283944' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-27T06:57:09.810 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:09 smithi043 ceph-mon[31531]: mgrmap e5: x(active, since 5s) 2024-05-27T06:57:10.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:09 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:09.853+0000 7fd49426d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T06:57:10.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:09 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:09.975+0000 7fd49426d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T06:57:10.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:10 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1110113764' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-27T06:57:11.004 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-05-27T06:57:11.004 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-27T06:57:11.004 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T06:57:11.004 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-27T06:57:11.004 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-27T06:57:11.004 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-05-27T06:57:11.004 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.046+0000 7fc6eb200700 1 Processor -- start 2024-05-27T06:57:11.004 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.046+0000 7fc6eb200700 1 -- start start 2024-05-27T06:57:11.004 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.046+0000 7fc6eb200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec07d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:11.004 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.046+0000 7fc6eb200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec07d700 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.046+0000 7fc6e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec07d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.047+0000 7fc6e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec07d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50920/0 (socket says 172.21.15.43:50920) 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.047+0000 7fc6e9e00700 1 -- 172.21.15.43:0/4217610828 learned_addr learned my addr 172.21.15.43:0/4217610828 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.047+0000 7fc6e9e00700 1 -- 172.21.15.43:0/4217610828 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec07d880 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.047+0000 7fc6e9e00700 1 --2- 172.21.15.43:0/4217610828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec07d1c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009e80 tx=0x7fc6dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9556cde696ab2d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.047+0000 7fc6e8a00700 1 -- 172.21.15.43:0/4217610828 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6dc017070 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.048+0000 7fc6e8a00700 1 -- 172.21.15.43:0/4217610828 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc6dc00b8c0 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.048+0000 7fc6e8a00700 1 -- 172.21.15.43:0/4217610828 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6dc004420 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.048+0000 7fc6eb200700 1 -- 172.21.15.43:0/4217610828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 msgr2=0x7fc6ec07d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.048+0000 7fc6eb200700 1 --2- 172.21.15.43:0/4217610828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec07d1c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009e80 tx=0x7fc6dc00b330 comp rx=0 tx=0).stop 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.049+0000 7fc6eb200700 1 -- 172.21.15.43:0/4217610828 shutdown_connections 2024-05-27T06:57:11.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.049+0000 7fc6eb200700 1 --2- 172.21.15.43:0/4217610828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec07d1c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.049+0000 7fc6eb200700 1 -- 172.21.15.43:0/4217610828 >> 172.21.15.43:0/4217610828 conn(0x7fc6ec104f00 msgr2=0x7fc6ec107390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.049+0000 7fc6eb200700 1 -- 172.21.15.43:0/4217610828 shutdown_connections 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.049+0000 7fc6eb200700 1 -- 172.21.15.43:0/4217610828 wait complete. 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.050+0000 7fc6eb200700 1 Processor -- start 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.050+0000 7fc6eb200700 1 -- start start 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.050+0000 7fc6eb200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.050+0000 7fc6eb200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec10fce0 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.051+0000 7fc6e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.051+0000 7fc6e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec19fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50936/0 (socket says 172.21.15.43:50936) 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.051+0000 7fc6e9e00700 1 -- 172.21.15.43:0/1110113764 learned_addr learned my addr 172.21.15.43:0/1110113764 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:11.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.051+0000 7fc6e9e00700 1 -- 172.21.15.43:0/1110113764 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec1a0180 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.051+0000 7fc6e9e00700 1 --2- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec19fc40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc004000 tx=0x7fc6dc003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.052+0000 7fc6e2a00700 1 -- 172.21.15.43:0/1110113764 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6dc020020 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.052+0000 7fc6e2a00700 1 -- 172.21.15.43:0/1110113764 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc6dc00be50 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.052+0000 7fc6e2a00700 1 -- 172.21.15.43:0/1110113764 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6dc017040 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.052+0000 7fc6eb200700 1 -- 172.21.15.43:0/1110113764 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ec1a0410 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.052+0000 7fc6eb200700 1 -- 172.21.15.43:0/1110113764 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ec1a3020 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.053+0000 7fc6eb200700 1 -- 172.21.15.43:0/1110113764 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ec064e10 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.053+0000 7fc6e2a00700 1 -- 172.21.15.43:0/1110113764 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7fc6dc023020 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.054+0000 7fc6e2a00700 1 --2- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fc6d003e4c0 0x7fc6d00409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.054+0000 7fc6e2a00700 1 -- 172.21.15.43:0/1110113764 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc6dc00f050 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.054+0000 7fc6e9400700 1 -- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fc6d003e4c0 msgr2=0x7fc6d00409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3521718752 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.054+0000 7fc6e9400700 1 --2- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fc6d003e4c0 0x7fc6d00409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.059+0000 7fc6e2a00700 1 -- 172.21.15.43:0/1110113764 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc6dc02a020 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.255+0000 7fc6e9400700 1 -- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fc6d003e4c0 msgr2=0x7fc6d00409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3521718752 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.255+0000 7fc6e9400700 1 --2- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fc6d003e4c0 0x7fc6d00409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.349+0000 7fc6eb200700 1 -- 172.21.15.43:0/1110113764 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc6ec1a36f0 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.350+0000 7fc6e2a00700 1 -- 172.21.15.43:0/1110113764 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fc6dc01c890 con 0x7fc6ec07ecf0 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.353+0000 7fc6eb200700 1 -- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fc6d003e4c0 msgr2=0x7fc6d00409d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.353+0000 7fc6eb200700 1 --2- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fc6d003e4c0 0x7fc6d00409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.353+0000 7fc6eb200700 1 -- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 msgr2=0x7fc6ec19fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.353+0000 7fc6eb200700 1 --2- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec19fc40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc004000 tx=0x7fc6dc003d00 comp rx=0 tx=0).stop 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.353+0000 7fc6eb200700 1 -- 172.21.15.43:0/1110113764 shutdown_connections 2024-05-27T06:57:11.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.353+0000 7fc6eb200700 1 --2- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7fc6d003e4c0 0x7fc6d00409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:11.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.353+0000 7fc6eb200700 1 --2- 172.21.15.43:0/1110113764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6ec07ecf0 0x7fc6ec19fc40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:11.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.353+0000 7fc6eb200700 1 -- 172.21.15.43:0/1110113764 >> 172.21.15.43:0/1110113764 conn(0x7fc6ec104f00 msgr2=0x7fc6ec105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:11.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.353+0000 7fc6eb200700 1 -- 172.21.15.43:0/1110113764 shutdown_connections 2024-05-27T06:57:11.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:10.354+0000 7fc6eb200700 1 -- 172.21.15.43:0/1110113764 wait complete. 2024-05-27T06:57:11.009 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for the mgr to restart... 2024-05-27T06:57:11.009 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr epoch 5... 2024-05-27T06:57:11.954 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:11 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:11.609+0000 7fd49426d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T06:57:11.954 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:11 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:11.848+0000 7fd49426d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T06:57:12.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:12.071+0000 7fd49426d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T06:57:12.852 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:12.470+0000 7fd49426d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T06:57:12.853 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:12.599+0000 7fd49426d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T06:57:13.120 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:12.860+0000 7fd49426d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T06:57:13.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:13 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:13.127+0000 7fd49426d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:57:14.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:13 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:13.955+0000 7fd49426d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T06:57:14.585 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:14 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:14.258+0000 7fd49426d200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T06:57:15.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:14 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:14.851+0000 7fd49426d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T06:57:15.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:14 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:14.987+0000 7fd49426d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T06:57:15.862 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:15 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:15.471+0000 7fd49426d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T06:57:16.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:15 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:15.949+0000 7fd49426d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T06:57:16.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:16 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:16.080+0000 7fd49426d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T06:57:16.704 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:16 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:16.416+0000 7fd49426d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T06:57:16.704 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:16 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:16.604+0000 7fd49426d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T06:57:17.144 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:16 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:16.738+0000 7fd49426d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T06:57:17.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:17.151+0000 7fd49426d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T06:57:17.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:17.280+0000 7fd49426d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T06:57:18.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:18 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:18.077+0000 7fd49426d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T06:57:18.704 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:18 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:18.376+0000 7fd49426d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T06:57:18.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: Active manager daemon x restarted 2024-05-27T06:57:18.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: Activating manager daemon x 2024-05-27T06:57:18.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: osdmap e2: 0 total, 0 up, 0 in 2024-05-27T06:57:18.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: mgrmap e6: x(active, starting, since 0.00442465s) 2024-05-27T06:57:18.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T06:57:18.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T06:57:18.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T06:57:18.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T06:57:18.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T06:57:18.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: Manager daemon x is now available 2024-05-27T06:57:18.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:18 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:19 smithi043 ceph-mon[31531]: Found migration_current of "None". Setting to last migration. 2024-05-27T06:57:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:19 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:19 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:57:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:19 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:57:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:19 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T06:57:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:19 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T06:57:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:19 smithi043 ceph-mon[31531]: mgrmap e7: x(active, since 1.0178s) 2024-05-27T06:57:20.030 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-05-27T06:57:20.037 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-27T06:57:20.037 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-27T06:57:20.037 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-05-27T06:57:20.037 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.631+0000 7f8b65c00700 1 Processor -- start 2024-05-27T06:57:20.037 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.631+0000 7f8b65c00700 1 -- start start 2024-05-27T06:57:20.037 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.632+0000 7f8b65c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6010e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:20.037 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.632+0000 7f8b65c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6010eeb0 con 0x7f8b6007cbf0 2024-05-27T06:57:20.037 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.632+0000 7f8b5ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6010e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:20.037 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.632+0000 7f8b5ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6010e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50944/0 (socket says 172.21.15.43:50944) 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.632+0000 7f8b5ea00700 1 -- 172.21.15.43:0/2329964216 learned_addr learned my addr 172.21.15.43:0/2329964216 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.633+0000 7f8b5ea00700 1 -- 172.21.15.43:0/2329964216 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6010f030 con 0x7f8b6007cbf0 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.633+0000 7f8b5ea00700 1 --2- 172.21.15.43:0/2329964216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6010e970 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8b500098a0 tx=0x7f8b50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d9a891bc2e2bc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.633+0000 7f8b5d600700 1 -- 172.21.15.43:0/2329964216 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b50004160 con 0x7f8b6007cbf0 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.633+0000 7f8b5d600700 1 -- 172.21.15.43:0/2329964216 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f8b50004300 con 0x7f8b6007cbf0 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.633+0000 7f8b5d600700 1 -- 172.21.15.43:0/2329964216 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b500119e0 con 0x7f8b6007cbf0 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.634+0000 7f8b65c00700 1 -- 172.21.15.43:0/2329964216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 msgr2=0x7f8b6010e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.634+0000 7f8b65c00700 1 --2- 172.21.15.43:0/2329964216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6010e970 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8b500098a0 tx=0x7f8b50009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.635+0000 7f8b65c00700 1 -- 172.21.15.43:0/2329964216 shutdown_connections 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.635+0000 7f8b65c00700 1 --2- 172.21.15.43:0/2329964216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6010e970 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.038 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.635+0000 7f8b65c00700 1 -- 172.21.15.43:0/2329964216 >> 172.21.15.43:0/2329964216 conn(0x7f8b60103f20 msgr2=0x7f8b601063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.635+0000 7f8b65c00700 1 -- 172.21.15.43:0/2329964216 shutdown_connections 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.635+0000 7f8b65c00700 1 -- 172.21.15.43:0/2329964216 wait complete. 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.636+0000 7f8b65c00700 1 Processor -- start 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.636+0000 7f8b65c00700 1 -- start start 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.636+0000 7f8b65c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6019fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.636+0000 7f8b65c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6010fe70 con 0x7f8b6007cbf0 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.637+0000 7f8b5ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6019fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.637+0000 7f8b5ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6019fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50960/0 (socket says 172.21.15.43:50960) 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.637+0000 7f8b5ea00700 1 -- 172.21.15.43:0/1813207931 learned_addr learned my addr 172.21.15.43:0/1813207931 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.637+0000 7f8b5ea00700 1 -- 172.21.15.43:0/1813207931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b601a0320 con 0x7f8b6007cbf0 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.637+0000 7f8b5ea00700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6019fde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8b500116a0 tx=0x7f8b50011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.638+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b50003970 con 0x7f8b6007cbf0 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.638+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f8b50003b10 con 0x7f8b6007cbf0 2024-05-27T06:57:20.039 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.638+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b50019550 con 0x7f8b6007cbf0 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.638+0000 7f8b65c00700 1 -- 172.21.15.43:0/1813207931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b601a05b0 con 0x7f8b6007cbf0 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.638+0000 7f8b65c00700 1 -- 172.21.15.43:0/1813207931 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b601a3180 con 0x7f8b6007cbf0 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.639+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f8b5001aa50 con 0x7f8b6007cbf0 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.640+0000 7f8b57400700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 0x7f8b440409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.640+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 --> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8b440410e0 con 0x7f8b4403e4c0 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.640+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8b50053830 con 0x7f8b6007cbf0 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.640+0000 7f8b5e000700 1 -- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 msgr2=0x7f8b440409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3521718752 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.640+0000 7f8b5e000700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 0x7f8b440409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.841+0000 7f8b5e000700 1 -- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 msgr2=0x7f8b440409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3521718752 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:11.841+0000 7f8b5e000700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 0x7f8b440409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:12.242+0000 7f8b5e000700 1 -- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 msgr2=0x7f8b440409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3521718752 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:12.242+0000 7f8b5e000700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 0x7f8b440409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:13.043+0000 7f8b5e000700 1 -- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 msgr2=0x7f8b440409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3521718752 2024-05-27T06:57:20.040 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:13.043+0000 7f8b5e000700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 0x7f8b440409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:14.645+0000 7f8b5e000700 1 -- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 msgr2=0x7f8b440409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3521718752 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:14.645+0000 7f8b5e000700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 0x7f8b440409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:17.848+0000 7f8b5e000700 1 -- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 msgr2=0x7f8b440409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3521718752 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:17.848+0000 7f8b5e000700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 0x7f8b440409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:18.385+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mgrmap(e 6) ==== 49989+0+0 (secure 0 0 0) 0x7f8b5001a430 con 0x7f8b6007cbf0 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:18.385+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 msgr2=0x7f8b440409d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:18.385+0000 7f8b57400700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 0x7f8b440409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.391+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7f8b50055420 con 0x7f8b6007cbf0 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.391+0000 7f8b57400700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f8b44041e00 0x7f8b44044220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.391+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 --> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8b440410e0 con 0x7f8b44041e00 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.393+0000 7f8b5e000700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f8b44041e00 0x7f8b44044220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.396+0000 7f8b5e000700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f8b44041e00 0x7f8b44044220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c003a10 tx=0x7f8b4c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.398+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 <== mgr.14120 v2:172.21.15.43:6800/1651567761 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f8b440410e0 con 0x7f8b44041e00 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.407+0000 7f8b65c00700 1 -- 172.21.15.43:0/1813207931 --> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f8b60064e10 con 0x7f8b44041e00 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.408+0000 7f8b57400700 1 -- 172.21.15.43:0/1813207931 <== mgr.14120 v2:172.21.15.43:6800/1651567761 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f8b60064e10 con 0x7f8b44041e00 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.408+0000 7f8b65c00700 1 -- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f8b44041e00 msgr2=0x7f8b44044220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.408+0000 7f8b65c00700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f8b44041e00 0x7f8b44044220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c003a10 tx=0x7f8b4c0092b0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.408+0000 7f8b65c00700 1 -- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 msgr2=0x7f8b6019fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.408+0000 7f8b65c00700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6019fde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8b500116a0 tx=0x7f8b50011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.409+0000 7f8b65c00700 1 -- 172.21.15.43:0/1813207931 shutdown_connections 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.409+0000 7f8b65c00700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f8b44041e00 0x7f8b44044220 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.409+0000 7f8b65c00700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b6007cbf0 0x7f8b6019fde0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.409+0000 7f8b65c00700 1 --2- 172.21.15.43:0/1813207931 >> [v2:172.21.15.43:6800/3521718752,v1:172.21.15.43:6801/3521718752] conn(0x7f8b4403e4c0 0x7f8b440409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.042 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.409+0000 7f8b65c00700 1 -- 172.21.15.43:0/1813207931 >> 172.21.15.43:0/1813207931 conn(0x7f8b60103f20 msgr2=0x7f8b60104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:20.043 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.409+0000 7f8b65c00700 1 -- 172.21.15.43:0/1813207931 shutdown_connections 2024-05-27T06:57:20.043 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:19.409+0000 7f8b65c00700 1 -- 172.21.15.43:0/1813207931 wait complete. 2024-05-27T06:57:20.043 INFO:teuthology.orchestra.run.smithi043.stdout:mgr epoch 5 is available 2024-05-27T06:57:20.043 INFO:teuthology.orchestra.run.smithi043.stdout:Setting orchestrator backend to cephadm... 2024-05-27T06:57:20.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:20 smithi043 ceph-mon[31531]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-27T06:57:20.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:20 smithi043 ceph-mon[31531]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-27T06:57:20.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:20 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:20.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:20 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:20.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:20 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:57:21.659 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.744+0000 7f6bf2400700 1 Processor -- start 2024-05-27T06:57:21.659 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.744+0000 7f6bf2400700 1 -- start start 2024-05-27T06:57:21.659 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.745+0000 7f6bf2400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec07dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:21.659 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.745+0000 7f6bf2400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec07e190 con 0x7f6bec07f780 2024-05-27T06:57:21.661 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.745+0000 7f6bf1000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec07dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:21.661 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.745+0000 7f6bf1000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec07dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59722/0 (socket says 172.21.15.43:59722) 2024-05-27T06:57:21.661 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.745+0000 7f6bf1000700 1 -- 172.21.15.43:0/3105117689 learned_addr learned my addr 172.21.15.43:0/3105117689 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:21.661 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.746+0000 7f6bf1000700 1 -- 172.21.15.43:0/3105117689 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec07e310 con 0x7f6bec07f780 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.746+0000 7f6bf1000700 1 --2- 172.21.15.43:0/3105117689 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec07dc50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6be00098a0 tx=0x7f6be0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152b63c6d29a03e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.746+0000 7f6beb400700 1 -- 172.21.15.43:0/3105117689 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6be0004160 con 0x7f6bec07f780 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.747+0000 7f6beb400700 1 -- 172.21.15.43:0/3105117689 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f6be0004300 con 0x7f6bec07f780 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.747+0000 7f6beb400700 1 -- 172.21.15.43:0/3105117689 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6be00119e0 con 0x7f6bec07f780 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.747+0000 7f6bf2400700 1 -- 172.21.15.43:0/3105117689 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 msgr2=0x7f6bec07dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.747+0000 7f6bf2400700 1 --2- 172.21.15.43:0/3105117689 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec07dc50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6be00098a0 tx=0x7f6be0009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.747+0000 7f6bf2400700 1 -- 172.21.15.43:0/3105117689 shutdown_connections 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.747+0000 7f6bf2400700 1 --2- 172.21.15.43:0/3105117689 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec07dc50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.747+0000 7f6bf2400700 1 -- 172.21.15.43:0/3105117689 >> 172.21.15.43:0/3105117689 conn(0x7f6bec104e30 msgr2=0x7f6bec107300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.748+0000 7f6bf2400700 1 -- 172.21.15.43:0/3105117689 shutdown_connections 2024-05-27T06:57:21.662 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.748+0000 7f6bf2400700 1 -- 172.21.15.43:0/3105117689 wait complete. 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.749+0000 7f6bf2400700 1 Processor -- start 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.749+0000 7f6bf2400700 1 -- start start 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.749+0000 7f6bf2400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec1a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.749+0000 7f6bf2400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10b8b0 con 0x7f6bec07f780 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.749+0000 7f6bf1000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec1a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.749+0000 7f6bf1000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec1a4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59732/0 (socket says 172.21.15.43:59732) 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.750+0000 7f6bf1000700 1 -- 172.21.15.43:0/1681647449 learned_addr learned my addr 172.21.15.43:0/1681647449 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.750+0000 7f6bf1000700 1 -- 172.21.15.43:0/1681647449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec1a4580 con 0x7f6bec07f780 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.750+0000 7f6bf1000700 1 --2- 172.21.15.43:0/1681647449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec1a4040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6be0003f20 tx=0x7f6be0011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.750+0000 7f6be9600700 1 -- 172.21.15.43:0/1681647449 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6be00038b0 con 0x7f6bec07f780 2024-05-27T06:57:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.751+0000 7f6bf2400700 1 -- 172.21.15.43:0/1681647449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bec1a4810 con 0x7f6bec07f780 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.751+0000 7f6be9600700 1 -- 172.21.15.43:0/1681647449 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f6be0003a50 con 0x7f6bec07f780 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.751+0000 7f6be9600700 1 -- 172.21.15.43:0/1681647449 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6be0019550 con 0x7f6bec07f780 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.751+0000 7f6bf2400700 1 -- 172.21.15.43:0/1681647449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bec1a7420 con 0x7f6bec07f780 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.752+0000 7f6bf2400700 1 -- 172.21.15.43:0/1681647449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bec064e10 con 0x7f6bec07f780 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.752+0000 7f6be9600700 1 -- 172.21.15.43:0/1681647449 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7f6be0019850 con 0x7f6bec07f780 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.753+0000 7f6be9600700 1 --2- 172.21.15.43:0/1681647449 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f6bd403e2f0 0x7f6bd4040800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.753+0000 7f6be9600700 1 -- 172.21.15.43:0/1681647449 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6be0053b40 con 0x7f6bec07f780 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.753+0000 7f6bebe00700 1 --2- 172.21.15.43:0/1681647449 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f6bd403e2f0 0x7f6bd4040800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.753+0000 7f6bebe00700 1 --2- 172.21.15.43:0/1681647449 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f6bd403e2f0 0x7f6bd4040800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8006fd0 tx=0x7f6bd8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.760+0000 7f6be9600700 1 -- 172.21.15.43:0/1681647449 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6be001a4e0 con 0x7f6bec07f780 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:20.794+0000 7f6be9600700 1 -- 172.21.15.43:0/1681647449 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f6be0018a40 con 0x7f6bec07f780 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.018+0000 7f6bf2400700 1 -- 172.21.15.43:0/1681647449 --> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f6bec02e950 con 0x7f6bd403e2f0 2024-05-27T06:57:21.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.029+0000 7f6be9600700 1 -- 172.21.15.43:0/1681647449 <== mgr.14120 v2:172.21.15.43:6800/1651567761 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6bec02e950 con 0x7f6bd403e2f0 2024-05-27T06:57:21.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.032+0000 7f6bf2400700 1 -- 172.21.15.43:0/1681647449 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f6bd403e2f0 msgr2=0x7f6bd4040800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:21.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.032+0000 7f6bf2400700 1 --2- 172.21.15.43:0/1681647449 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f6bd403e2f0 0x7f6bd4040800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8006fd0 tx=0x7f6bd8006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:21.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.032+0000 7f6bf2400700 1 -- 172.21.15.43:0/1681647449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 msgr2=0x7f6bec1a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:21.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.032+0000 7f6bf2400700 1 --2- 172.21.15.43:0/1681647449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec1a4040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6be0003f20 tx=0x7f6be0011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:21.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.032+0000 7f6bf2400700 1 -- 172.21.15.43:0/1681647449 shutdown_connections 2024-05-27T06:57:21.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.032+0000 7f6bf2400700 1 --2- 172.21.15.43:0/1681647449 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f6bd403e2f0 0x7f6bd4040800 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:21.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.032+0000 7f6bf2400700 1 --2- 172.21.15.43:0/1681647449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bec07f780 0x7f6bec1a4040 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:21.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.032+0000 7f6bf2400700 1 -- 172.21.15.43:0/1681647449 >> 172.21.15.43:0/1681647449 conn(0x7f6bec104e30 msgr2=0x7f6bec19ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:21.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.033+0000 7f6bf2400700 1 -- 172.21.15.43:0/1681647449 shutdown_connections 2024-05-27T06:57:21.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:21.034+0000 7f6bf2400700 1 -- 172.21.15.43:0/1681647449 wait complete. 2024-05-27T06:57:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:21 smithi043 ceph-mon[31531]: [27/May/2024:06:57:20] ENGINE Bus STARTING 2024-05-27T06:57:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:21 smithi043 ceph-mon[31531]: [27/May/2024:06:57:20] ENGINE Serving on http://172.21.15.43:8765 2024-05-27T06:57:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:21 smithi043 ceph-mon[31531]: [27/May/2024:06:57:20] ENGINE Serving on https://172.21.15.43:7150 2024-05-27T06:57:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:21 smithi043 ceph-mon[31531]: [27/May/2024:06:57:20] ENGINE Bus STARTED 2024-05-27T06:57:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:21 smithi043 ceph-mon[31531]: mgrmap e8: x(active, since 2s) 2024-05-27T06:57:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:21 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:21 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:57:22.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:22 smithi043 ceph-mon[31531]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:57:23.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-27T06:57:23.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.367+0000 7fc83de00700 1 Processor -- start 2024-05-27T06:57:23.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.367+0000 7fc83de00700 1 -- start start 2024-05-27T06:57:23.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.368+0000 7fc83de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83810bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:23.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.368+0000 7fc83de00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc83810c5e0 con 0x7fc83810bbe0 2024-05-27T06:57:23.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.368+0000 7fc836a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83810bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:23.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.368+0000 7fc836a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83810bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59738/0 (socket says 172.21.15.43:59738) 2024-05-27T06:57:23.264 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.368+0000 7fc836a00700 1 -- 172.21.15.43:0/1840057912 learned_addr learned my addr 172.21.15.43:0/1840057912 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.369+0000 7fc836a00700 1 -- 172.21.15.43:0/1840057912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83810cd10 con 0x7fc83810bbe0 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.369+0000 7fc836a00700 1 --2- 172.21.15.43:0/1840057912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83810bff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc8280098a0 tx=0x7fc828009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1555ef0b242a11b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.370+0000 7fc836000700 1 -- 172.21.15.43:0/1840057912 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc828004160 con 0x7fc83810bbe0 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.370+0000 7fc836000700 1 -- 172.21.15.43:0/1840057912 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc828004300 con 0x7fc83810bbe0 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.370+0000 7fc836000700 1 -- 172.21.15.43:0/1840057912 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8280119e0 con 0x7fc83810bbe0 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.370+0000 7fc83de00700 1 -- 172.21.15.43:0/1840057912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 msgr2=0x7fc83810bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.370+0000 7fc83de00700 1 --2- 172.21.15.43:0/1840057912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83810bff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc8280098a0 tx=0x7fc828009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.371+0000 7fc83de00700 1 -- 172.21.15.43:0/1840057912 shutdown_connections 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.371+0000 7fc83de00700 1 --2- 172.21.15.43:0/1840057912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83810bff0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.371+0000 7fc83de00700 1 -- 172.21.15.43:0/1840057912 >> 172.21.15.43:0/1840057912 conn(0x7fc838107440 msgr2=0x7fc838109860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:23.265 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.371+0000 7fc83de00700 1 -- 172.21.15.43:0/1840057912 shutdown_connections 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.371+0000 7fc83de00700 1 -- 172.21.15.43:0/1840057912 wait complete. 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.372+0000 7fc83de00700 1 Processor -- start 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.372+0000 7fc83de00700 1 -- start start 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.373+0000 7fc83de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83819fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.373+0000 7fc83de00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc83810ff40 con 0x7fc83810bbe0 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.373+0000 7fc836a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83819fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.373+0000 7fc836a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83819fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59754/0 (socket says 172.21.15.43:59754) 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.373+0000 7fc836a00700 1 -- 172.21.15.43:0/2405611759 learned_addr learned my addr 172.21.15.43:0/2405611759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.373+0000 7fc836a00700 1 -- 172.21.15.43:0/2405611759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8381a0370 con 0x7fc83810bbe0 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.374+0000 7fc836a00700 1 --2- 172.21.15.43:0/2405611759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83819fe30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8280046e0 tx=0x7fc828011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.374+0000 7fc83ca00700 1 -- 172.21.15.43:0/2405611759 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8280039e0 con 0x7fc83810bbe0 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.374+0000 7fc83ca00700 1 -- 172.21.15.43:0/2405611759 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc828003b80 con 0x7fc83810bbe0 2024-05-27T06:57:23.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.374+0000 7fc83de00700 1 -- 172.21.15.43:0/2405611759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8381a0600 con 0x7fc83810bbe0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.374+0000 7fc83ca00700 1 -- 172.21.15.43:0/2405611759 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc828024700 con 0x7fc83810bbe0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.374+0000 7fc83de00700 1 -- 172.21.15.43:0/2405611759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8381a3210 con 0x7fc83810bbe0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.376+0000 7fc83de00700 1 -- 172.21.15.43:0/2405611759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc838064e10 con 0x7fc83810bbe0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.376+0000 7fc83ca00700 1 -- 172.21.15.43:0/2405611759 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fc828022020 con 0x7fc83810bbe0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.376+0000 7fc83ca00700 1 --2- 172.21.15.43:0/2405611759 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc81c03e4c0 0x7fc81c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.376+0000 7fc83ca00700 1 -- 172.21.15.43:0/2405611759 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc828053a70 con 0x7fc83810bbe0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.377+0000 7fc82e000700 1 --2- 172.21.15.43:0/2405611759 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc81c03e4c0 0x7fc81c0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.377+0000 7fc82e000700 1 --2- 172.21.15.43:0/2405611759 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc81c03e4c0 0x7fc81c0409d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc820006fd0 tx=0x7fc820006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.382+0000 7fc83ca00700 1 -- 172.21.15.43:0/2405611759 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc82800b3e0 con 0x7fc83810bbe0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.618+0000 7fc83de00700 1 -- 172.21.15.43:0/2405611759 --> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fc8381092d0 con 0x7fc81c03e4c0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.620+0000 7fc83ca00700 1 -- 172.21.15.43:0/2405611759 <== mgr.14120 v2:172.21.15.43:6800/1651567761 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fc8381092d0 con 0x7fc81c03e4c0 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.623+0000 7fc83de00700 1 -- 172.21.15.43:0/2405611759 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc81c03e4c0 msgr2=0x7fc81c0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.623+0000 7fc83de00700 1 --2- 172.21.15.43:0/2405611759 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc81c03e4c0 0x7fc81c0409d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc820006fd0 tx=0x7fc820006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:23.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.624+0000 7fc83de00700 1 -- 172.21.15.43:0/2405611759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 msgr2=0x7fc83819fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:23.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.624+0000 7fc83de00700 1 --2- 172.21.15.43:0/2405611759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83819fe30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8280046e0 tx=0x7fc828011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:23.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.624+0000 7fc83de00700 1 -- 172.21.15.43:0/2405611759 shutdown_connections 2024-05-27T06:57:23.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.624+0000 7fc83de00700 1 --2- 172.21.15.43:0/2405611759 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc81c03e4c0 0x7fc81c0409d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:23.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.624+0000 7fc83de00700 1 --2- 172.21.15.43:0/2405611759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83810bbe0 0x7fc83819fe30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:23.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.624+0000 7fc83de00700 1 -- 172.21.15.43:0/2405611759 >> 172.21.15.43:0/2405611759 conn(0x7fc838107440 msgr2=0x7fc838108bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:23.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.624+0000 7fc83de00700 1 -- 172.21.15.43:0/2405611759 shutdown_connections 2024-05-27T06:57:23.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:22.624+0000 7fc83de00700 1 -- 172.21.15.43:0/2405611759 wait complete. 2024-05-27T06:57:23.268 INFO:teuthology.orchestra.run.smithi043.stdout:Generating ssh key... 2024-05-27T06:57:23.946 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:23 smithi043 ceph-mon[31531]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: Generating public/private rsa key pair. 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: Your identification has been saved in /tmp/tmpep_lk5zb/key. 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: Your public key has been saved in /tmp/tmpep_lk5zb/key.pub. 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: The key fingerprint is: 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: SHA256:1uBZlBUKVW9vuEU1vI3qIf7FcrSu8eZidtf3k63QlT0 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: The key's randomart image is: 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: +---[RSA 3072]----+ 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: | ..o++....| 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: | o.. . .o| 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: | . o o.+| 2024-05-27T06:57:24.706 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: | . = ..=+| 2024-05-27T06:57:24.707 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: | S . .oE*| 2024-05-27T06:57:24.707 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: | . . o+ *.| 2024-05-27T06:57:24.707 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: | . o+.B +| 2024-05-27T06:57:24.707 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: | . =Ooo=| 2024-05-27T06:57:24.707 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: | +o*=o=| 2024-05-27T06:57:24.707 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: +----[SHA256]-----+ 2024-05-27T06:57:25.000 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.953+0000 7f7e42200700 1 Processor -- start 2024-05-27T06:57:25.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.954+0000 7f7e42200700 1 -- start start 2024-05-27T06:57:25.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.954+0000 7f7e42200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c108cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:25.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.954+0000 7f7e42200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c1092e0 con 0x7f7e3c1088e0 2024-05-27T06:57:25.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.955+0000 7f7e3b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c108cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:25.005 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.955+0000 7f7e3b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c108cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59770/0 (socket says 172.21.15.43:59770) 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.955+0000 7f7e3b400700 1 -- 172.21.15.43:0/2925091701 learned_addr learned my addr 172.21.15.43:0/2925091701 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.955+0000 7f7e3b400700 1 -- 172.21.15.43:0/2925091701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3c109a10 con 0x7f7e3c1088e0 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.956+0000 7f7e3b400700 1 --2- 172.21.15.43:0/2925091701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c108cf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0098a0 tx=0x7f7e2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140580949a65089 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.956+0000 7f7e3a000700 1 -- 172.21.15.43:0/2925091701 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e2c004160 con 0x7f7e3c1088e0 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.956+0000 7f7e3a000700 1 -- 172.21.15.43:0/2925091701 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f7e2c004300 con 0x7f7e3c1088e0 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.956+0000 7f7e3a000700 1 -- 172.21.15.43:0/2925091701 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e2c0119e0 con 0x7f7e3c1088e0 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.957+0000 7f7e42200700 1 -- 172.21.15.43:0/2925091701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 msgr2=0x7f7e3c108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.957+0000 7f7e42200700 1 --2- 172.21.15.43:0/2925091701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c108cf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0098a0 tx=0x7f7e2c009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.957+0000 7f7e42200700 1 -- 172.21.15.43:0/2925091701 shutdown_connections 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.957+0000 7f7e42200700 1 --2- 172.21.15.43:0/2925091701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c108cf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.957+0000 7f7e42200700 1 -- 172.21.15.43:0/2925091701 >> 172.21.15.43:0/2925091701 conn(0x7f7e3c103f20 msgr2=0x7f7e3c1063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.957+0000 7f7e42200700 1 -- 172.21.15.43:0/2925091701 shutdown_connections 2024-05-27T06:57:25.006 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.958+0000 7f7e42200700 1 -- 172.21.15.43:0/2925091701 wait complete. 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.959+0000 7f7e42200700 1 Processor -- start 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.959+0000 7f7e42200700 1 -- start start 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.959+0000 7f7e42200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c19fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.959+0000 7f7e42200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c10cc40 con 0x7f7e3c1088e0 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.960+0000 7f7e3b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c19fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.960+0000 7f7e3b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c19fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59786/0 (socket says 172.21.15.43:59786) 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.960+0000 7f7e3b400700 1 -- 172.21.15.43:0/4069235727 learned_addr learned my addr 172.21.15.43:0/4069235727 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.960+0000 7f7e3b400700 1 -- 172.21.15.43:0/4069235727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3c1a0370 con 0x7f7e3c1088e0 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.960+0000 7f7e3b400700 1 --2- 172.21.15.43:0/4069235727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c19fe30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0046e0 tx=0x7f7e2c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.961+0000 7f7e33e00700 1 -- 172.21.15.43:0/4069235727 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e2c0039e0 con 0x7f7e3c1088e0 2024-05-27T06:57:25.007 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.961+0000 7f7e33e00700 1 -- 172.21.15.43:0/4069235727 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f7e2c003b80 con 0x7f7e3c1088e0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.961+0000 7f7e42200700 1 -- 172.21.15.43:0/4069235727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e3c1a0600 con 0x7f7e3c1088e0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.961+0000 7f7e33e00700 1 -- 172.21.15.43:0/4069235727 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e2c024700 con 0x7f7e3c1088e0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.961+0000 7f7e42200700 1 -- 172.21.15.43:0/4069235727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e3c1a3210 con 0x7f7e3c1088e0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.962+0000 7f7e33e00700 1 -- 172.21.15.43:0/4069235727 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f7e2c022020 con 0x7f7e3c1088e0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.962+0000 7f7e42200700 1 -- 172.21.15.43:0/4069235727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e3c064e10 con 0x7f7e3c1088e0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.963+0000 7f7e33e00700 1 --2- 172.21.15.43:0/4069235727 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f7e2003e4c0 0x7f7e200409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.963+0000 7f7e33e00700 1 -- 172.21.15.43:0/4069235727 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7e2c053920 con 0x7f7e3c1088e0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.963+0000 7f7e3aa00700 1 --2- 172.21.15.43:0/4069235727 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f7e2003e4c0 0x7f7e200409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.964+0000 7f7e3aa00700 1 --2- 172.21.15.43:0/4069235727 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f7e2003e4c0 0x7f7e200409d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7e28006fd0 tx=0x7f7e28006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:23.968+0000 7f7e33e00700 1 -- 172.21.15.43:0/4069235727 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7e2c0172b0 con 0x7f7e3c1088e0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.203+0000 7f7e42200700 1 -- 172.21.15.43:0/4069235727 --> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f7e3c02e950 con 0x7f7e2003e4c0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.401+0000 7f7e33e00700 1 -- 172.21.15.43:0/4069235727 <== mgr.14120 v2:172.21.15.43:6800/1651567761 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7e3c02e950 con 0x7f7e2003e4c0 2024-05-27T06:57:25.008 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.405+0000 7f7e42200700 1 -- 172.21.15.43:0/4069235727 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f7e2003e4c0 msgr2=0x7f7e200409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:25.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.405+0000 7f7e42200700 1 --2- 172.21.15.43:0/4069235727 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f7e2003e4c0 0x7f7e200409d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7e28006fd0 tx=0x7f7e28006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:25.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.405+0000 7f7e42200700 1 -- 172.21.15.43:0/4069235727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 msgr2=0x7f7e3c19fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:25.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.405+0000 7f7e42200700 1 --2- 172.21.15.43:0/4069235727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c19fe30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0046e0 tx=0x7f7e2c011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:25.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.405+0000 7f7e42200700 1 -- 172.21.15.43:0/4069235727 shutdown_connections 2024-05-27T06:57:25.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.405+0000 7f7e42200700 1 --2- 172.21.15.43:0/4069235727 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f7e2003e4c0 0x7f7e200409d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.405+0000 7f7e42200700 1 --2- 172.21.15.43:0/4069235727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e3c1088e0 0x7f7e3c19fe30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.405+0000 7f7e42200700 1 -- 172.21.15.43:0/4069235727 >> 172.21.15.43:0/4069235727 conn(0x7f7e3c103f20 msgr2=0x7f7e3c1965e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:25.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.405+0000 7f7e42200700 1 -- 172.21.15.43:0/4069235727 shutdown_connections 2024-05-27T06:57:25.009 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:24.405+0000 7f7e42200700 1 -- 172.21.15.43:0/4069235727 wait complete. 2024-05-27T06:57:25.658 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:25 smithi043 ceph-mon[31531]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:57:25.658 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:25 smithi043 ceph-mon[31531]: Generating ssh key... 2024-05-27T06:57:25.658 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:25 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:25.658 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:25 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:26.556 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6co39Flbni/XB+4t2Vw9ErJkROs61wosglo8SXB82bm1IQtovkwzMPRpgy1qyo/1bhus1HZDoLXALGUlXDjHp3WcyDaBq/yPPJBS7xr0r6UkvciT7N6DDA5h3flwQ+QhDyT+Na8wN2BOMrsQddk2go95N0z0fu7IA0xE5CUApwlFX3Z8jKNxW41t0YzMsvf2qNh3BDsC9yIecaFCgYYrbwu+owPT7YdeSE3072O2S3ff2DhAw5BVFoD55TIAkR02sByFGkh0cBcb3URagF89c3q9ot8Cw1iJ0IaXIZ9bbNBzDE4fczMfpZoMiy9lKCltCY7H916/CtmgTI7UP0D+ws6vTm7B0KMHI5RpdiZwZB+BgV8eVmzmxt6pbRG6ddMtekKPFf4O1rd6CcJPGjkm8VpeiKVy15xjn7csfnjqmyNDOq1ZNRDmEMEHJ6dom050EGPEHwDJiLebnLEJAeWxgHwaFps4i+SZD0lxFwYEpjL3Yy/Xs5w3/CB7u49EALIc= ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:57:26.556 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.665+0000 7f8700000700 1 Processor -- start 2024-05-27T06:57:26.556 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.665+0000 7f8700000700 1 -- start start 2024-05-27T06:57:26.556 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.666+0000 7f8700000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f807da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:26.556 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.666+0000 7f8700000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86f807df90 con 0x7f86f807f580 2024-05-27T06:57:26.556 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.666+0000 7f86fd600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f807da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:26.556 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.667+0000 7f86fd600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f807da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60462/0 (socket says 172.21.15.43:60462) 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.667+0000 7f86fd600700 1 -- 172.21.15.43:0/1600113240 learned_addr learned my addr 172.21.15.43:0/1600113240 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.667+0000 7f86fd600700 1 -- 172.21.15.43:0/1600113240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f807e110 con 0x7f86f807f580 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.667+0000 7f86fd600700 1 --2- 172.21.15.43:0/1600113240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f807da50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0098a0 tx=0x7f86ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7694689fb871e021 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.668+0000 7f86f7e00700 1 -- 172.21.15.43:0/1600113240 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86ec004160 con 0x7f86f807f580 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.668+0000 7f86f7e00700 1 -- 172.21.15.43:0/1600113240 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f86ec004300 con 0x7f86f807f580 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.668+0000 7f86f7e00700 1 -- 172.21.15.43:0/1600113240 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86ec0119e0 con 0x7f86f807f580 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.668+0000 7f8700000700 1 -- 172.21.15.43:0/1600113240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 msgr2=0x7f86f807da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.668+0000 7f8700000700 1 --2- 172.21.15.43:0/1600113240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f807da50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0098a0 tx=0x7f86ec009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.669+0000 7f8700000700 1 -- 172.21.15.43:0/1600113240 shutdown_connections 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.669+0000 7f8700000700 1 --2- 172.21.15.43:0/1600113240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f807da50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.669+0000 7f8700000700 1 -- 172.21.15.43:0/1600113240 >> 172.21.15.43:0/1600113240 conn(0x7f86f81050b0 msgr2=0x7f86f8107500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:26.557 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.669+0000 7f8700000700 1 -- 172.21.15.43:0/1600113240 shutdown_connections 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.669+0000 7f8700000700 1 -- 172.21.15.43:0/1600113240 wait complete. 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.670+0000 7f8700000700 1 Processor -- start 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.670+0000 7f8700000700 1 -- start start 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.671+0000 7f8700000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f819fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.671+0000 7f8700000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86f810fec0 con 0x7f86f807f580 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.671+0000 7f86fd600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f819fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.671+0000 7f86fd600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f819fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60474/0 (socket says 172.21.15.43:60474) 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.671+0000 7f86fd600700 1 -- 172.21.15.43:0/2638126363 learned_addr learned my addr 172.21.15.43:0/2638126363 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.671+0000 7f86fd600700 1 -- 172.21.15.43:0/2638126363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f81a0360 con 0x7f86f807f580 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.672+0000 7f86fd600700 1 --2- 172.21.15.43:0/2638126363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f819fe20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0116a0 tx=0x7f86ec011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.672+0000 7f86f6000700 1 -- 172.21.15.43:0/2638126363 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86ec003970 con 0x7f86f807f580 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.672+0000 7f8700000700 1 -- 172.21.15.43:0/2638126363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f81a05f0 con 0x7f86f807f580 2024-05-27T06:57:26.558 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.672+0000 7f86f6000700 1 -- 172.21.15.43:0/2638126363 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f86ec003b10 con 0x7f86f807f580 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.672+0000 7f86f6000700 1 -- 172.21.15.43:0/2638126363 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86ec019550 con 0x7f86f807f580 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.672+0000 7f8700000700 1 -- 172.21.15.43:0/2638126363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f81a3200 con 0x7f86f807f580 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.673+0000 7f86f6000700 1 -- 172.21.15.43:0/2638126363 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f86ec019810 con 0x7f86f807f580 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.674+0000 7f8700000700 1 -- 172.21.15.43:0/2638126363 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f8064e10 con 0x7f86f807f580 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.674+0000 7f86f6000700 1 --2- 172.21.15.43:0/2638126363 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f86e003e470 0x7f86e0040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.674+0000 7f86f6000700 1 -- 172.21.15.43:0/2638126363 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f86ec0538e0 con 0x7f86f807f580 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.674+0000 7f86fcc00700 1 --2- 172.21.15.43:0/2638126363 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f86e003e470 0x7f86e0040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.675+0000 7f86fcc00700 1 --2- 172.21.15.43:0/2638126363 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f86e003e470 0x7f86e0040980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86e4006fd0 tx=0x7f86e4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.680+0000 7f86f6000700 1 -- 172.21.15.43:0/2638126363 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f86ec016260 con 0x7f86f807f580 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.918+0000 7f8700000700 1 -- 172.21.15.43:0/2638126363 --> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f86f8106020 con 0x7f86e003e470 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.919+0000 7f86f6000700 1 -- 172.21.15.43:0/2638126363 <== mgr.14120 v2:172.21.15.43:6800/1651567761 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f86f8106020 con 0x7f86e003e470 2024-05-27T06:57:26.559 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.923+0000 7f8700000700 1 -- 172.21.15.43:0/2638126363 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f86e003e470 msgr2=0x7f86e0040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.923+0000 7f8700000700 1 --2- 172.21.15.43:0/2638126363 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f86e003e470 0x7f86e0040980 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86e4006fd0 tx=0x7f86e4006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.923+0000 7f8700000700 1 -- 172.21.15.43:0/2638126363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 msgr2=0x7f86f819fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.923+0000 7f8700000700 1 --2- 172.21.15.43:0/2638126363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f819fe20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0116a0 tx=0x7f86ec011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.923+0000 7f8700000700 1 -- 172.21.15.43:0/2638126363 shutdown_connections 2024-05-27T06:57:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.923+0000 7f8700000700 1 --2- 172.21.15.43:0/2638126363 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f86e003e470 0x7f86e0040980 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.923+0000 7f8700000700 1 --2- 172.21.15.43:0/2638126363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86f807f580 0x7f86f819fe20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.923+0000 7f8700000700 1 -- 172.21.15.43:0/2638126363 >> 172.21.15.43:0/2638126363 conn(0x7f86f81050b0 msgr2=0x7f86f8105910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.923+0000 7f8700000700 1 -- 172.21.15.43:0/2638126363 shutdown_connections 2024-05-27T06:57:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:25.923+0000 7f8700000700 1 -- 172.21.15.43:0/2638126363 wait complete. 2024-05-27T06:57:26.560 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-27T06:57:26.561 INFO:teuthology.orchestra.run.smithi043.stdout:Adding key to root@localhost authorized_keys... 2024-05-27T06:57:26.561 INFO:teuthology.orchestra.run.smithi043.stdout:Adding host smithi043... 2024-05-27T06:57:27.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:26 smithi043 ceph-mon[31531]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:57:28.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:27 smithi043 ceph-mon[31531]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi043", "addr": "172.21.15.43", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:57:29.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:28 smithi043 ceph-mon[31531]: Deploying cephadm binary to smithi043 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Added host 'smithi043' with addr '172.21.15.43' 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.244+0000 7fc8eb200700 1 Processor -- start 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.244+0000 7fc8eb200700 1 -- start start 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.244+0000 7fc8eb200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec10be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.244+0000 7fc8eb200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec10c430 con 0x7fc8ec10ba30 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.245+0000 7fc8e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec10be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.245+0000 7fc8e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec10be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60490/0 (socket says 172.21.15.43:60490) 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.245+0000 7fc8e9e00700 1 -- 172.21.15.43:0/592776825 learned_addr learned my addr 172.21.15.43:0/592776825 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.245+0000 7fc8e9e00700 1 -- 172.21.15.43:0/592776825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ec10cb60 con 0x7fc8ec10ba30 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.246+0000 7fc8e9e00700 1 --2- 172.21.15.43:0/592776825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec10be40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc009e80 tx=0x7fc8dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc3b70e4140bc78 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.246+0000 7fc8e8a00700 1 -- 172.21.15.43:0/592776825 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8dc017070 con 0x7fc8ec10ba30 2024-05-27T06:57:34.479 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.246+0000 7fc8e8a00700 1 -- 172.21.15.43:0/592776825 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc8dc00b8c0 con 0x7fc8ec10ba30 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.246+0000 7fc8e8a00700 1 -- 172.21.15.43:0/592776825 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8dc004420 con 0x7fc8ec10ba30 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.247+0000 7fc8eb200700 1 -- 172.21.15.43:0/592776825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 msgr2=0x7fc8ec10be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.247+0000 7fc8eb200700 1 --2- 172.21.15.43:0/592776825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec10be40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc009e80 tx=0x7fc8dc00b330 comp rx=0 tx=0).stop 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.247+0000 7fc8eb200700 1 -- 172.21.15.43:0/592776825 shutdown_connections 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.247+0000 7fc8eb200700 1 --2- 172.21.15.43:0/592776825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec10be40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.247+0000 7fc8eb200700 1 -- 172.21.15.43:0/592776825 >> 172.21.15.43:0/592776825 conn(0x7fc8ec07eed0 msgr2=0x7fc8ec07f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.247+0000 7fc8eb200700 1 -- 172.21.15.43:0/592776825 shutdown_connections 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.247+0000 7fc8eb200700 1 -- 172.21.15.43:0/592776825 wait complete. 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.248+0000 7fc8eb200700 1 Processor -- start 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.249+0000 7fc8eb200700 1 -- start start 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.249+0000 7fc8eb200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.249+0000 7fc8eb200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec10fd90 con 0x7fc8ec10ba30 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.249+0000 7fc8e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.249+0000 7fc8e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec19fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60496/0 (socket says 172.21.15.43:60496) 2024-05-27T06:57:34.480 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.249+0000 7fc8e9e00700 1 -- 172.21.15.43:0/1142923631 learned_addr learned my addr 172.21.15.43:0/1142923631 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.250+0000 7fc8e9e00700 1 -- 172.21.15.43:0/1142923631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ec1a0220 con 0x7fc8ec10ba30 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.250+0000 7fc8e9e00700 1 --2- 172.21.15.43:0/1142923631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec19fce0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc004000 tx=0x7fc8dc0093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.250+0000 7fc8e2a00700 1 -- 172.21.15.43:0/1142923631 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8dc020020 con 0x7fc8ec10ba30 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.250+0000 7fc8e2a00700 1 -- 172.21.15.43:0/1142923631 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fc8dc003d00 con 0x7fc8ec10ba30 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.250+0000 7fc8eb200700 1 -- 172.21.15.43:0/1142923631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8ec1a04b0 con 0x7fc8ec10ba30 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.251+0000 7fc8e2a00700 1 -- 172.21.15.43:0/1142923631 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8dc017040 con 0x7fc8ec10ba30 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.251+0000 7fc8eb200700 1 -- 172.21.15.43:0/1142923631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8ec1a30c0 con 0x7fc8ec10ba30 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.252+0000 7fc8eb200700 1 -- 172.21.15.43:0/1142923631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8ec064e10 con 0x7fc8ec10ba30 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.252+0000 7fc8e2a00700 1 -- 172.21.15.43:0/1142923631 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fc8dc023020 con 0x7fc8ec10ba30 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.253+0000 7fc8e2a00700 1 --2- 172.21.15.43:0/1142923631 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc8d003e470 0x7fc8d0040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.253+0000 7fc8e2a00700 1 -- 172.21.15.43:0/1142923631 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc8dc00f050 con 0x7fc8ec10ba30 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.253+0000 7fc8e9400700 1 --2- 172.21.15.43:0/1142923631 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc8d003e470 0x7fc8d0040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.254+0000 7fc8e9400700 1 --2- 172.21.15.43:0/1142923631 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc8d003e470 0x7fc8d0040980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8006fd0 tx=0x7fc8d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.258+0000 7fc8e2a00700 1 -- 172.21.15.43:0/1142923631 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc8dc02a020 con 0x7fc8ec10ba30 2024-05-27T06:57:34.481 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:27.499+0000 7fc8eb200700 1 -- 172.21.15.43:0/1142923631 --> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi043", "addr": "172.21.15.43", "target": ["mon-mgr", ""]}) -- 0x7fc8ec109110 con 0x7fc8d003e470 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.868+0000 7fc8e2a00700 1 -- 172.21.15.43:0/1142923631 <== mgr.14120 v2:172.21.15.43:6800/1651567761 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fc8ec109110 con 0x7fc8d003e470 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.872+0000 7fc8eb200700 1 -- 172.21.15.43:0/1142923631 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc8d003e470 msgr2=0x7fc8d0040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.872+0000 7fc8eb200700 1 --2- 172.21.15.43:0/1142923631 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc8d003e470 0x7fc8d0040980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8006fd0 tx=0x7fc8d8006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.872+0000 7fc8eb200700 1 -- 172.21.15.43:0/1142923631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 msgr2=0x7fc8ec19fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.872+0000 7fc8eb200700 1 --2- 172.21.15.43:0/1142923631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec19fce0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc004000 tx=0x7fc8dc0093b0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.872+0000 7fc8eb200700 1 -- 172.21.15.43:0/1142923631 shutdown_connections 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.872+0000 7fc8eb200700 1 --2- 172.21.15.43:0/1142923631 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fc8d003e470 0x7fc8d0040980 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.872+0000 7fc8eb200700 1 --2- 172.21.15.43:0/1142923631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8ec10ba30 0x7fc8ec19fce0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.872+0000 7fc8eb200700 1 -- 172.21.15.43:0/1142923631 >> 172.21.15.43:0/1142923631 conn(0x7fc8ec07eed0 msgr2=0x7fc8ec108a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.873+0000 7fc8eb200700 1 -- 172.21.15.43:0/1142923631 shutdown_connections 2024-05-27T06:57:34.482 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:33.873+0000 7fc8eb200700 1 -- 172.21.15.43:0/1142923631 wait complete. 2024-05-27T06:57:34.483 INFO:teuthology.orchestra.run.smithi043.stdout:Deploying unmanaged mon service... 2024-05-27T06:57:35.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:34 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:35.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:34 smithi043 ceph-mon[31531]: Added host smithi043 2024-05-27T06:57:35.153 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:34 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:57:36.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-27T06:57:36.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.160+0000 7f94a0400700 1 Processor -- start 2024-05-27T06:57:36.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.161+0000 7f94a0400700 1 -- start start 2024-05-27T06:57:36.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.161+0000 7f94a0400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f9498109d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:36.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.161+0000 7f94a0400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f949810a340 con 0x7f9498109940 2024-05-27T06:57:36.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.162+0000 7f949da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f9498109d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:36.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.162+0000 7f949da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f9498109d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45742/0 (socket says 172.21.15.43:45742) 2024-05-27T06:57:36.195 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.162+0000 7f949da00700 1 -- 172.21.15.43:0/747399421 learned_addr learned my addr 172.21.15.43:0/747399421 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:36.195 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.162+0000 7f949da00700 1 -- 172.21.15.43:0/747399421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949810ab10 con 0x7f9498109940 2024-05-27T06:57:36.195 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.163+0000 7f949da00700 1 --2- 172.21.15.43:0/747399421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f9498109d50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f948c0098a0 tx=0x7f948c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a134420656e6da47 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:36.195 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.163+0000 7f9497e00700 1 -- 172.21.15.43:0/747399421 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f948c004160 con 0x7f9498109940 2024-05-27T06:57:36.195 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.163+0000 7f9497e00700 1 -- 172.21.15.43:0/747399421 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f948c004300 con 0x7f9498109940 2024-05-27T06:57:36.199 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.163+0000 7f9497e00700 1 -- 172.21.15.43:0/747399421 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f948c0119e0 con 0x7f9498109940 2024-05-27T06:57:36.199 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.164+0000 7f94a0400700 1 -- 172.21.15.43:0/747399421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 msgr2=0x7f9498109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:36.199 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.164+0000 7f94a0400700 1 --2- 172.21.15.43:0/747399421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f9498109d50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f948c0098a0 tx=0x7f948c009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:36.199 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.164+0000 7f94a0400700 1 -- 172.21.15.43:0/747399421 shutdown_connections 2024-05-27T06:57:36.199 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.164+0000 7f94a0400700 1 --2- 172.21.15.43:0/747399421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f9498109d50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:36.199 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.164+0000 7f94a0400700 1 -- 172.21.15.43:0/747399421 >> 172.21.15.43:0/747399421 conn(0x7f9498105000 msgr2=0x7f9498107450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:36.199 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.164+0000 7f94a0400700 1 -- 172.21.15.43:0/747399421 shutdown_connections 2024-05-27T06:57:36.199 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.164+0000 7f94a0400700 1 -- 172.21.15.43:0/747399421 wait complete. 2024-05-27T06:57:36.199 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.166+0000 7f94a0400700 1 Processor -- start 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.166+0000 7f94a0400700 1 -- start start 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.166+0000 7f94a0400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f949819fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.166+0000 7f94a0400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f949807da90 con 0x7f9498109940 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.166+0000 7f949da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f949819fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.167+0000 7f949da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f949819fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45754/0 (socket says 172.21.15.43:45754) 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.167+0000 7f949da00700 1 -- 172.21.15.43:0/2198699047 learned_addr learned my addr 172.21.15.43:0/2198699047 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.167+0000 7f949da00700 1 -- 172.21.15.43:0/2198699047 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94981a0360 con 0x7f9498109940 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.167+0000 7f949da00700 1 --2- 172.21.15.43:0/2198699047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f949819fe20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f948c006ba0 tx=0x7f948c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.168+0000 7f9496000700 1 -- 172.21.15.43:0/2198699047 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f948c003970 con 0x7f9498109940 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.168+0000 7f9496000700 1 -- 172.21.15.43:0/2198699047 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f948c003b10 con 0x7f9498109940 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.168+0000 7f94a0400700 1 -- 172.21.15.43:0/2198699047 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94981a05f0 con 0x7f9498109940 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.168+0000 7f9496000700 1 -- 172.21.15.43:0/2198699047 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f948c019550 con 0x7f9498109940 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.168+0000 7f94a0400700 1 -- 172.21.15.43:0/2198699047 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94981a3200 con 0x7f9498109940 2024-05-27T06:57:36.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.169+0000 7f94a0400700 1 -- 172.21.15.43:0/2198699047 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9498064e10 con 0x7f9498109940 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.170+0000 7f9496000700 1 -- 172.21.15.43:0/2198699047 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f948c01a430 con 0x7f9498109940 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.170+0000 7f9496000700 1 --2- 172.21.15.43:0/2198699047 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f948003e420 0x7f9480040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.170+0000 7f9496000700 1 -- 172.21.15.43:0/2198699047 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f948c053c80 con 0x7f9498109940 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.171+0000 7f949d000700 1 --2- 172.21.15.43:0/2198699047 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f948003e420 0x7f9480040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.172+0000 7f949d000700 1 --2- 172.21.15.43:0/2198699047 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f948003e420 0x7f9480040930 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9484006fd0 tx=0x7f9484006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.177+0000 7f9496000700 1 -- 172.21.15.43:0/2198699047 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f948c05a080 con 0x7f9498109940 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.435+0000 7f94a0400700 1 -- 172.21.15.43:0/2198699047 --> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f949802e950 con 0x7f948003e420 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.443+0000 7f9496000700 1 -- 172.21.15.43:0/2198699047 <== mgr.14120 v2:172.21.15.43:6800/1651567761 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f949802e950 con 0x7f948003e420 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.447+0000 7f94a0400700 1 -- 172.21.15.43:0/2198699047 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f948003e420 msgr2=0x7f9480040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.447+0000 7f94a0400700 1 --2- 172.21.15.43:0/2198699047 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f948003e420 0x7f9480040930 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9484006fd0 tx=0x7f9484006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.447+0000 7f94a0400700 1 -- 172.21.15.43:0/2198699047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 msgr2=0x7f949819fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.447+0000 7f94a0400700 1 --2- 172.21.15.43:0/2198699047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f949819fe20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f948c006ba0 tx=0x7f948c011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.447+0000 7f94a0400700 1 -- 172.21.15.43:0/2198699047 shutdown_connections 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.447+0000 7f94a0400700 1 --2- 172.21.15.43:0/2198699047 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f948003e420 0x7f9480040930 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:36.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.447+0000 7f94a0400700 1 --2- 172.21.15.43:0/2198699047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9498109940 0x7f949819fe20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:36.202 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.447+0000 7f94a0400700 1 -- 172.21.15.43:0/2198699047 >> 172.21.15.43:0/2198699047 conn(0x7f9498105000 msgr2=0x7f9498105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:36.202 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.448+0000 7f94a0400700 1 -- 172.21.15.43:0/2198699047 shutdown_connections 2024-05-27T06:57:36.202 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:35.448+0000 7f94a0400700 1 -- 172.21.15.43:0/2198699047 wait complete. 2024-05-27T06:57:36.202 INFO:teuthology.orchestra.run.smithi043.stdout:Deploying unmanaged mgr service... 2024-05-27T06:57:36.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:36 smithi043 ceph-mon[31531]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:57:36.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:36 smithi043 ceph-mon[31531]: Saving service mon spec with placement count:5 2024-05-27T06:57:36.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:36 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-27T06:57:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.154+0000 7f0830a00700 1 Processor -- start 2024-05-27T06:57:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.154+0000 7f0830a00700 1 -- start start 2024-05-27T06:57:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.155+0000 7f0830a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c07dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.155+0000 7f0830a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f082c07dff0 con 0x7f082c07f5e0 2024-05-27T06:57:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.155+0000 7f0829e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c07dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:38.172 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.155+0000 7f0829e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c07dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45768/0 (socket says 172.21.15.43:45768) 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.155+0000 7f0829e00700 1 -- 172.21.15.43:0/2371252527 learned_addr learned my addr 172.21.15.43:0/2371252527 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.156+0000 7f0829e00700 1 -- 172.21.15.43:0/2371252527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082c07e170 con 0x7f082c07f5e0 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.156+0000 7f0829e00700 1 --2- 172.21.15.43:0/2371252527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c07dab0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f081c0098a0 tx=0x7f081c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873ee814c8f3e9b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.156+0000 7f0828a00700 1 -- 172.21.15.43:0/2371252527 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f081c004160 con 0x7f082c07f5e0 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.156+0000 7f0828a00700 1 -- 172.21.15.43:0/2371252527 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f081c004300 con 0x7f082c07f5e0 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.157+0000 7f0828a00700 1 -- 172.21.15.43:0/2371252527 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f081c0119e0 con 0x7f082c07f5e0 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.157+0000 7f0830a00700 1 -- 172.21.15.43:0/2371252527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 msgr2=0x7f082c07dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.157+0000 7f0830a00700 1 --2- 172.21.15.43:0/2371252527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c07dab0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f081c0098a0 tx=0x7f081c009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.157+0000 7f0830a00700 1 -- 172.21.15.43:0/2371252527 shutdown_connections 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.157+0000 7f0830a00700 1 --2- 172.21.15.43:0/2371252527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c07dab0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.157+0000 7f0830a00700 1 -- 172.21.15.43:0/2371252527 >> 172.21.15.43:0/2371252527 conn(0x7f082c1050a0 msgr2=0x7f082c1074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.158+0000 7f0830a00700 1 -- 172.21.15.43:0/2371252527 shutdown_connections 2024-05-27T06:57:38.173 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.158+0000 7f0830a00700 1 -- 172.21.15.43:0/2371252527 wait complete. 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.159+0000 7f0830a00700 1 Processor -- start 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.159+0000 7f0830a00700 1 -- start start 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.159+0000 7f0830a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c19fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.159+0000 7f0830a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f082c10feb0 con 0x7f082c07f5e0 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.160+0000 7f0829e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c19fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.160+0000 7f0829e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c19fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45780/0 (socket says 172.21.15.43:45780) 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.160+0000 7f0829e00700 1 -- 172.21.15.43:0/768720095 learned_addr learned my addr 172.21.15.43:0/768720095 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.160+0000 7f0829e00700 1 -- 172.21.15.43:0/768720095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082c1a0370 con 0x7f082c07f5e0 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.160+0000 7f0829e00700 1 --2- 172.21.15.43:0/768720095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c19fe30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f081c0116a0 tx=0x7f081c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.161+0000 7f0822a00700 1 -- 172.21.15.43:0/768720095 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f081c003970 con 0x7f082c07f5e0 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.161+0000 7f0822a00700 1 -- 172.21.15.43:0/768720095 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f081c003b10 con 0x7f082c07f5e0 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.161+0000 7f0822a00700 1 -- 172.21.15.43:0/768720095 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f081c019550 con 0x7f082c07f5e0 2024-05-27T06:57:38.174 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.161+0000 7f0830a00700 1 -- 172.21.15.43:0/768720095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f082c1a0600 con 0x7f082c07f5e0 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.161+0000 7f0830a00700 1 -- 172.21.15.43:0/768720095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f082c1a3210 con 0x7f082c07f5e0 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.162+0000 7f0830a00700 1 -- 172.21.15.43:0/768720095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f082c064e10 con 0x7f082c07f5e0 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.162+0000 7f0822a00700 1 -- 172.21.15.43:0/768720095 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f081c01a430 con 0x7f082c07f5e0 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.163+0000 7f0822a00700 1 --2- 172.21.15.43:0/768720095 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f081003e4c0 0x7f08100409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.163+0000 7f0822a00700 1 -- 172.21.15.43:0/768720095 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f081c0538c0 con 0x7f082c07f5e0 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.163+0000 7f0829400700 1 --2- 172.21.15.43:0/768720095 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f081003e4c0 0x7f08100409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.164+0000 7f0829400700 1 --2- 172.21.15.43:0/768720095 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f081003e4c0 0x7f08100409d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0818006fd0 tx=0x7f0818006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.169+0000 7f0822a00700 1 -- 172.21.15.43:0/768720095 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f081c02a260 con 0x7f082c07f5e0 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.425+0000 7f0830a00700 1 -- 172.21.15.43:0/768720095 --> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f082c106060 con 0x7f081003e4c0 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.437+0000 7f0822a00700 1 -- 172.21.15.43:0/768720095 <== mgr.14120 v2:172.21.15.43:6800/1651567761 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f082c106060 con 0x7f081003e4c0 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.440+0000 7f0830a00700 1 -- 172.21.15.43:0/768720095 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f081003e4c0 msgr2=0x7f08100409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.440+0000 7f0830a00700 1 --2- 172.21.15.43:0/768720095 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f081003e4c0 0x7f08100409d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0818006fd0 tx=0x7f0818006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:38.175 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.440+0000 7f0830a00700 1 -- 172.21.15.43:0/768720095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 msgr2=0x7f082c19fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.440+0000 7f0830a00700 1 --2- 172.21.15.43:0/768720095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c19fe30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f081c0116a0 tx=0x7f081c011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.440+0000 7f0830a00700 1 -- 172.21.15.43:0/768720095 shutdown_connections 2024-05-27T06:57:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.440+0000 7f0830a00700 1 --2- 172.21.15.43:0/768720095 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f081003e4c0 0x7f08100409d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.440+0000 7f0830a00700 1 --2- 172.21.15.43:0/768720095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f082c07f5e0 0x7f082c19fe30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.440+0000 7f0830a00700 1 -- 172.21.15.43:0/768720095 >> 172.21.15.43:0/768720095 conn(0x7f082c1050a0 msgr2=0x7f082c105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.440+0000 7f0830a00700 1 -- 172.21.15.43:0/768720095 shutdown_connections 2024-05-27T06:57:38.176 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:37.440+0000 7f0830a00700 1 -- 172.21.15.43:0/768720095 wait complete. 2024-05-27T06:57:38.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:38 smithi043 ceph-mon[31531]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:57:38.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:38 smithi043 ceph-mon[31531]: Saving service mgr spec with placement count:2 2024-05-27T06:57:38.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:38 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:38.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:38 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:39.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:39 smithi043 ceph-mon[31531]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:57:39.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:39 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:39.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:39 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2004091421' entity='client.admin' 2024-05-27T06:57:39.748 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.809+0000 7f645ce00700 1 Processor -- start 2024-05-27T06:57:39.748 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.809+0000 7f645ce00700 1 -- start start 2024-05-27T06:57:39.748 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.809+0000 7f645ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645807d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.748 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.809+0000 7f645ce00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f645807d700 con 0x7f645807ecf0 2024-05-27T06:57:39.748 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.810+0000 7f6455e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645807d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.748 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.810+0000 7f6455e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645807d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45792/0 (socket says 172.21.15.43:45792) 2024-05-27T06:57:39.748 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.810+0000 7f6455e00700 1 -- 172.21.15.43:0/819085519 learned_addr learned my addr 172.21.15.43:0/819085519 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:39.748 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.810+0000 7f6455e00700 1 -- 172.21.15.43:0/819085519 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645807d880 con 0x7f645807ecf0 2024-05-27T06:57:39.748 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.811+0000 7f6455e00700 1 --2- 172.21.15.43:0/819085519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645807d1c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f64480098a0 tx=0x7f6448009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee1143a7a6c2683 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:39.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.811+0000 7f6454a00700 1 -- 172.21.15.43:0/819085519 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6448004160 con 0x7f645807ecf0 2024-05-27T06:57:39.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.811+0000 7f6454a00700 1 -- 172.21.15.43:0/819085519 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6448004300 con 0x7f645807ecf0 2024-05-27T06:57:39.752 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.811+0000 7f6454a00700 1 -- 172.21.15.43:0/819085519 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6448011960 con 0x7f645807ecf0 2024-05-27T06:57:39.752 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.812+0000 7f645ce00700 1 -- 172.21.15.43:0/819085519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 msgr2=0x7f645807d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:39.752 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.812+0000 7f645ce00700 1 --2- 172.21.15.43:0/819085519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645807d1c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f64480098a0 tx=0x7f6448009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.812+0000 7f645ce00700 1 -- 172.21.15.43:0/819085519 shutdown_connections 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.812+0000 7f645ce00700 1 --2- 172.21.15.43:0/819085519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645807d1c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.812+0000 7f645ce00700 1 -- 172.21.15.43:0/819085519 >> 172.21.15.43:0/819085519 conn(0x7f6458105100 msgr2=0x7f6458107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.812+0000 7f645ce00700 1 -- 172.21.15.43:0/819085519 shutdown_connections 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.812+0000 7f645ce00700 1 -- 172.21.15.43:0/819085519 wait complete. 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.813+0000 7f645ce00700 1 Processor -- start 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.814+0000 7f645ce00700 1 -- start start 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.814+0000 7f645ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645819fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.814+0000 7f645ce00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f645810ffe0 con 0x7f645807ecf0 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.814+0000 7f6455e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645819fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.814+0000 7f6455e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645819fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45806/0 (socket says 172.21.15.43:45806) 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.814+0000 7f6455e00700 1 -- 172.21.15.43:0/2004091421 learned_addr learned my addr 172.21.15.43:0/2004091421 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.815+0000 7f6455e00700 1 -- 172.21.15.43:0/2004091421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64581a0320 con 0x7f645807ecf0 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.815+0000 7f6455e00700 1 --2- 172.21.15.43:0/2004091421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645819fde0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6448011670 tx=0x7f6448011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.815+0000 7f644ea00700 1 -- 172.21.15.43:0/2004091421 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f644801f070 con 0x7f645807ecf0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.815+0000 7f644ea00700 1 -- 172.21.15.43:0/2004091421 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6448003b20 con 0x7f645807ecf0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.815+0000 7f644ea00700 1 -- 172.21.15.43:0/2004091421 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6448019580 con 0x7f645807ecf0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.815+0000 7f645ce00700 1 -- 172.21.15.43:0/2004091421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64581a05b0 con 0x7f645807ecf0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.816+0000 7f645ce00700 1 -- 172.21.15.43:0/2004091421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64581a31c0 con 0x7f645807ecf0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.817+0000 7f645ce00700 1 -- 172.21.15.43:0/2004091421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6458064e10 con 0x7f645807ecf0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.817+0000 7f644ea00700 1 -- 172.21.15.43:0/2004091421 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f6448022070 con 0x7f645807ecf0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.818+0000 7f644ea00700 1 --2- 172.21.15.43:0/2004091421 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f643c03e470 0x7f643c040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.818+0000 7f644ea00700 1 -- 172.21.15.43:0/2004091421 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6448053a70 con 0x7f645807ecf0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.818+0000 7f6455400700 1 --2- 172.21.15.43:0/2004091421 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f643c03e470 0x7f643c040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.819+0000 7f6455400700 1 --2- 172.21.15.43:0/2004091421 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f643c03e470 0x7f643c040980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6444006fd0 tx=0x7f6444006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:38.823+0000 7f644ea00700 1 -- 172.21.15.43:0/2004091421 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f644801ba80 con 0x7f645807ecf0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.059+0000 7f645ce00700 1 -- 172.21.15.43:0/2004091421 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f64581a34d0 con 0x7f645807ecf0 2024-05-27T06:57:39.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.062+0000 7f644ea00700 1 -- 172.21.15.43:0/2004091421 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f644802ca40 con 0x7f645807ecf0 2024-05-27T06:57:39.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.066+0000 7f645ce00700 1 -- 172.21.15.43:0/2004091421 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f643c03e470 msgr2=0x7f643c040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:39.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.066+0000 7f645ce00700 1 --2- 172.21.15.43:0/2004091421 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f643c03e470 0x7f643c040980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6444006fd0 tx=0x7f6444006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:39.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.066+0000 7f645ce00700 1 -- 172.21.15.43:0/2004091421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 msgr2=0x7f645819fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:39.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.066+0000 7f645ce00700 1 --2- 172.21.15.43:0/2004091421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645819fde0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6448011670 tx=0x7f6448011a30 comp rx=0 tx=0).stop 2024-05-27T06:57:39.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.066+0000 7f645ce00700 1 -- 172.21.15.43:0/2004091421 shutdown_connections 2024-05-27T06:57:39.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.066+0000 7f645ce00700 1 --2- 172.21.15.43:0/2004091421 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f643c03e470 0x7f643c040980 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.066+0000 7f645ce00700 1 --2- 172.21.15.43:0/2004091421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f645807ecf0 0x7f645819fde0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.066+0000 7f645ce00700 1 -- 172.21.15.43:0/2004091421 >> 172.21.15.43:0/2004091421 conn(0x7f6458105100 msgr2=0x7f64581058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:39.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.067+0000 7f645ce00700 1 -- 172.21.15.43:0/2004091421 shutdown_connections 2024-05-27T06:57:39.755 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:39.067+0000 7f645ce00700 1 -- 172.21.15.43:0/2004091421 wait complete. 2024-05-27T06:57:41.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.413+0000 7fe30d000700 1 Processor -- start 2024-05-27T06:57:41.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.414+0000 7fe30d000700 1 -- start start 2024-05-27T06:57:41.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.414+0000 7fe30d000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30810bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:41.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.414+0000 7fe30d000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe30810c5e0 con 0x7fe30810bbe0 2024-05-27T06:57:41.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.414+0000 7fe305e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30810bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:41.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.415+0000 7fe305e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30810bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45822/0 (socket says 172.21.15.43:45822) 2024-05-27T06:57:41.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.415+0000 7fe305e00700 1 -- 172.21.15.43:0/218835954 learned_addr learned my addr 172.21.15.43:0/218835954 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:41.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.415+0000 7fe305e00700 1 -- 172.21.15.43:0/218835954 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30810cd10 con 0x7fe30810bbe0 2024-05-27T06:57:41.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.415+0000 7fe305e00700 1 --2- 172.21.15.43:0/218835954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30810bff0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80098a0 tx=0x7fe2f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547413434f6a793d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:41.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.416+0000 7fe304a00700 1 -- 172.21.15.43:0/218835954 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2f8004160 con 0x7fe30810bbe0 2024-05-27T06:57:41.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.416+0000 7fe304a00700 1 -- 172.21.15.43:0/218835954 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2f8004300 con 0x7fe30810bbe0 2024-05-27T06:57:41.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.416+0000 7fe304a00700 1 -- 172.21.15.43:0/218835954 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2f8011960 con 0x7fe30810bbe0 2024-05-27T06:57:41.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.416+0000 7fe30d000700 1 -- 172.21.15.43:0/218835954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 msgr2=0x7fe30810bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:41.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.416+0000 7fe30d000700 1 --2- 172.21.15.43:0/218835954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30810bff0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80098a0 tx=0x7fe2f8009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:41.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.417+0000 7fe30d000700 1 -- 172.21.15.43:0/218835954 shutdown_connections 2024-05-27T06:57:41.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.417+0000 7fe30d000700 1 --2- 172.21.15.43:0/218835954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30810bff0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:41.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.417+0000 7fe30d000700 1 -- 172.21.15.43:0/218835954 >> 172.21.15.43:0/218835954 conn(0x7fe308107440 msgr2=0x7fe308109860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:41.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.417+0000 7fe30d000700 1 -- 172.21.15.43:0/218835954 shutdown_connections 2024-05-27T06:57:41.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.417+0000 7fe30d000700 1 -- 172.21.15.43:0/218835954 wait complete. 2024-05-27T06:57:41.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.418+0000 7fe30d000700 1 Processor -- start 2024-05-27T06:57:41.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.418+0000 7fe30d000700 1 -- start start 2024-05-27T06:57:41.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.418+0000 7fe30d000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30819fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:41.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.418+0000 7fe30d000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe3081100c0 con 0x7fe30810bbe0 2024-05-27T06:57:41.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.419+0000 7fe305e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30819fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:41.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.419+0000 7fe305e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30819fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45826/0 (socket says 172.21.15.43:45826) 2024-05-27T06:57:41.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.419+0000 7fe305e00700 1 -- 172.21.15.43:0/1614909551 learned_addr learned my addr 172.21.15.43:0/1614909551 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:41.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.419+0000 7fe305e00700 1 -- 172.21.15.43:0/1614909551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3081a03c0 con 0x7fe30810bbe0 2024-05-27T06:57:41.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.419+0000 7fe305e00700 1 --2- 172.21.15.43:0/1614909551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30819fe80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8011980 tx=0x7fe2f8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:41.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.420+0000 7fe2fea00700 1 -- 172.21.15.43:0/1614909551 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2f801f070 con 0x7fe30810bbe0 2024-05-27T06:57:41.697 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.420+0000 7fe2fea00700 1 -- 172.21.15.43:0/1614909551 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2f8003aa0 con 0x7fe30810bbe0 2024-05-27T06:57:41.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.420+0000 7fe2fea00700 1 -- 172.21.15.43:0/1614909551 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2f8019560 con 0x7fe30810bbe0 2024-05-27T06:57:41.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.420+0000 7fe30d000700 1 -- 172.21.15.43:0/1614909551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3081a0650 con 0x7fe30810bbe0 2024-05-27T06:57:41.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.420+0000 7fe30d000700 1 -- 172.21.15.43:0/1614909551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3081a3260 con 0x7fe30810bbe0 2024-05-27T06:57:41.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.421+0000 7fe30d000700 1 -- 172.21.15.43:0/1614909551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe308064e10 con 0x7fe30810bbe0 2024-05-27T06:57:41.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.421+0000 7fe2fea00700 1 -- 172.21.15.43:0/1614909551 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fe2f8022070 con 0x7fe30810bbe0 2024-05-27T06:57:41.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.422+0000 7fe2fea00700 1 --2- 172.21.15.43:0/1614909551 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fe2ec03e4c0 0x7fe2ec0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:41.698 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.422+0000 7fe2fea00700 1 -- 172.21.15.43:0/1614909551 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe2f8053920 con 0x7fe30810bbe0 2024-05-27T06:57:41.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.422+0000 7fe305400700 1 --2- 172.21.15.43:0/1614909551 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fe2ec03e4c0 0x7fe2ec0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:41.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.423+0000 7fe305400700 1 --2- 172.21.15.43:0/1614909551 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fe2ec03e4c0 0x7fe2ec0409d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4006fd0 tx=0x7fe2f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:41.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.428+0000 7fe2fea00700 1 -- 172.21.15.43:0/1614909551 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe2f80172b0 con 0x7fe30810bbe0 2024-05-27T06:57:41.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.671+0000 7fe30d000700 1 -- 172.21.15.43:0/1614909551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fe3081a3430 con 0x7fe30810bbe0 2024-05-27T06:57:41.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.676+0000 7fe2fea00700 1 -- 172.21.15.43:0/1614909551 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fe2f8023aa0 con 0x7fe30810bbe0 2024-05-27T06:57:41.699 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.679+0000 7fe30d000700 1 -- 172.21.15.43:0/1614909551 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fe2ec03e4c0 msgr2=0x7fe2ec0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:41.700 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.679+0000 7fe30d000700 1 --2- 172.21.15.43:0/1614909551 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fe2ec03e4c0 0x7fe2ec0409d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4006fd0 tx=0x7fe2f4006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:41.700 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.679+0000 7fe30d000700 1 -- 172.21.15.43:0/1614909551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 msgr2=0x7fe30819fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:41.700 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.679+0000 7fe30d000700 1 --2- 172.21.15.43:0/1614909551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30819fe80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8011980 tx=0x7fe2f8011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:41.700 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.680+0000 7fe30d000700 1 -- 172.21.15.43:0/1614909551 shutdown_connections 2024-05-27T06:57:41.700 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.680+0000 7fe30d000700 1 --2- 172.21.15.43:0/1614909551 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fe2ec03e4c0 0x7fe2ec0409d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:41.700 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.680+0000 7fe30d000700 1 --2- 172.21.15.43:0/1614909551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe30810bbe0 0x7fe30819fe80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:41.701 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.680+0000 7fe30d000700 1 -- 172.21.15.43:0/1614909551 >> 172.21.15.43:0/1614909551 conn(0x7fe308107440 msgr2=0x7fe308108c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:41.701 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.680+0000 7fe30d000700 1 -- 172.21.15.43:0/1614909551 shutdown_connections 2024-05-27T06:57:41.701 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:40.680+0000 7fe30d000700 1 -- 172.21.15.43:0/1614909551 wait complete. 2024-05-27T06:57:41.701 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling the dashboard module... 2024-05-27T06:57:41.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:41 smithi043 ceph-mon[31531]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:57:41.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:41 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1614909551' entity='client.admin' 2024-05-27T06:57:41.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:41 smithi043 ceph-mon[31531]: from='mgr.14120 172.21.15.43:0/3437687080' entity='mgr.x' 2024-05-27T06:57:42.913 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:42 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2435841057' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-27T06:57:43.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: ignoring --setuser ceph since I am not root 2024-05-27T06:57:43.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: ignoring --setgroup ceph since I am not root 2024-05-27T06:57:43.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:43 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:43.100+0000 7fc3b5233200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T06:57:43.401 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.345+0000 7fd91ba00700 1 Processor -- start 2024-05-27T06:57:43.401 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.345+0000 7fd91ba00700 1 -- start start 2024-05-27T06:57:43.401 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.346+0000 7fd91ba00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd914109e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:43.401 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.346+0000 7fd91ba00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd91410a400 con 0x7fd914109a00 2024-05-27T06:57:43.401 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.346+0000 7fd919000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd914109e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:43.401 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.346+0000 7fd919000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd914109e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45838/0 (socket says 172.21.15.43:45838) 2024-05-27T06:57:43.401 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.346+0000 7fd919000700 1 -- 172.21.15.43:0/10346098 learned_addr learned my addr 172.21.15.43:0/10346098 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.347+0000 7fd919000700 1 -- 172.21.15.43:0/10346098 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91410ab30 con 0x7fd914109a00 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.347+0000 7fd919000700 1 --2- 172.21.15.43:0/10346098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd914109e10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd9080098a0 tx=0x7fd908009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae27bc3facd3a8af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.347+0000 7fd913400700 1 -- 172.21.15.43:0/10346098 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd908004160 con 0x7fd914109a00 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.347+0000 7fd913400700 1 -- 172.21.15.43:0/10346098 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd908004300 con 0x7fd914109a00 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.347+0000 7fd913400700 1 -- 172.21.15.43:0/10346098 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd908011960 con 0x7fd914109a00 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.348+0000 7fd91ba00700 1 -- 172.21.15.43:0/10346098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 msgr2=0x7fd914109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.348+0000 7fd91ba00700 1 --2- 172.21.15.43:0/10346098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd914109e10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd9080098a0 tx=0x7fd908009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.348+0000 7fd91ba00700 1 -- 172.21.15.43:0/10346098 shutdown_connections 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.348+0000 7fd91ba00700 1 --2- 172.21.15.43:0/10346098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd914109e10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.348+0000 7fd91ba00700 1 -- 172.21.15.43:0/10346098 >> 172.21.15.43:0/10346098 conn(0x7fd9141050a0 msgr2=0x7fd9141074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.348+0000 7fd91ba00700 1 -- 172.21.15.43:0/10346098 shutdown_connections 2024-05-27T06:57:43.402 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.349+0000 7fd91ba00700 1 -- 172.21.15.43:0/10346098 wait complete. 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.350+0000 7fd91ba00700 1 Processor -- start 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.350+0000 7fd91ba00700 1 -- start start 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.350+0000 7fd91ba00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd91419fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.350+0000 7fd91ba00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd91410dee0 con 0x7fd914109a00 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.350+0000 7fd919000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd91419fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.350+0000 7fd919000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd91419fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45842/0 (socket says 172.21.15.43:45842) 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.350+0000 7fd919000700 1 -- 172.21.15.43:0/2435841057 learned_addr learned my addr 172.21.15.43:0/2435841057 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.351+0000 7fd919000700 1 -- 172.21.15.43:0/2435841057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9141a03c0 con 0x7fd914109a00 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.351+0000 7fd919000700 1 --2- 172.21.15.43:0/2435841057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd91419fe80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd9080119b0 tx=0x7fd908011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.352+0000 7fd911600700 1 -- 172.21.15.43:0/2435841057 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd908003aa0 con 0x7fd914109a00 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.352+0000 7fd911600700 1 -- 172.21.15.43:0/2435841057 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd90801b8d0 con 0x7fd914109a00 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.352+0000 7fd911600700 1 -- 172.21.15.43:0/2435841057 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9080237d0 con 0x7fd914109a00 2024-05-27T06:57:43.403 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.352+0000 7fd91ba00700 1 -- 172.21.15.43:0/2435841057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9141a0650 con 0x7fd914109a00 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.352+0000 7fd91ba00700 1 -- 172.21.15.43:0/2435841057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9141a3260 con 0x7fd914109a00 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.353+0000 7fd911600700 1 -- 172.21.15.43:0/2435841057 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fd90801b440 con 0x7fd914109a00 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.353+0000 7fd91ba00700 1 -- 172.21.15.43:0/2435841057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd914064e10 con 0x7fd914109a00 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.354+0000 7fd911600700 1 --2- 172.21.15.43:0/2435841057 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fd8fc03e420 0x7fd8fc040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.354+0000 7fd911600700 1 -- 172.21.15.43:0/2435841057 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd908053ec0 con 0x7fd914109a00 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.354+0000 7fd913e00700 1 --2- 172.21.15.43:0/2435841057 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fd8fc03e420 0x7fd8fc040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.355+0000 7fd913e00700 1 --2- 172.21.15.43:0/2435841057 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fd8fc03e420 0x7fd8fc040930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd900006fd0 tx=0x7fd900006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.361+0000 7fd911600700 1 -- 172.21.15.43:0/2435841057 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd90801f030 con 0x7fd914109a00 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.662+0000 7fd91ba00700 1 -- 172.21.15.43:0/2435841057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fd9140643c0 con 0x7fd914109a00 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.679+0000 7fd911600700 1 -- 172.21.15.43:0/2435841057 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fd908017e90 con 0x7fd914109a00 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.682+0000 7fd911600700 1 -- 172.21.15.43:0/2435841057 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fd90801acb0 con 0x7fd914109a00 2024-05-27T06:57:43.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.682+0000 7fd91ba00700 1 -- 172.21.15.43:0/2435841057 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fd8fc03e420 msgr2=0x7fd8fc040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:43.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.683+0000 7fd91ba00700 1 --2- 172.21.15.43:0/2435841057 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fd8fc03e420 0x7fd8fc040930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd900006fd0 tx=0x7fd900006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:43.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.683+0000 7fd91ba00700 1 -- 172.21.15.43:0/2435841057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 msgr2=0x7fd91419fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:43.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.683+0000 7fd91ba00700 1 --2- 172.21.15.43:0/2435841057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd91419fe80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd9080119b0 tx=0x7fd908011a90 comp rx=0 tx=0).stop 2024-05-27T06:57:43.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.683+0000 7fd91ba00700 1 -- 172.21.15.43:0/2435841057 shutdown_connections 2024-05-27T06:57:43.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.683+0000 7fd91ba00700 1 --2- 172.21.15.43:0/2435841057 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fd8fc03e420 0x7fd8fc040930 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:43.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.683+0000 7fd91ba00700 1 --2- 172.21.15.43:0/2435841057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd914109a00 0x7fd91419fe80 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd9080119b0 tx=0x7fd908011a90 comp rx=0 tx=0).stop 2024-05-27T06:57:43.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.683+0000 7fd91ba00700 1 -- 172.21.15.43:0/2435841057 >> 172.21.15.43:0/2435841057 conn(0x7fd9141050a0 msgr2=0x7fd91407db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:43.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.683+0000 7fd91ba00700 1 -- 172.21.15.43:0/2435841057 shutdown_connections 2024-05-27T06:57:43.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:42.684+0000 7fd91ba00700 1 -- 172.21.15.43:0/2435841057 wait complete. 2024-05-27T06:57:43.671 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:43 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:43.351+0000 7fc3b5233200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:57:43.672 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:43 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:43.492+0000 7fc3b5233200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T06:57:43.954 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:43 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:43.911+0000 7fc3b5233200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T06:57:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:43 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2435841057' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-27T06:57:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:43 smithi043 ceph-mon[31531]: mgrmap e9: x(active, since 24s) 2024-05-27T06:57:44.284 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:44.035+0000 7fc3b5233200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T06:57:44.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:44 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3200959361' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-27T06:57:45.084 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-05-27T06:57:45.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-27T06:57:45.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-27T06:57:45.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-27T06:57:45.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-27T06:57:45.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-05-27T06:57:45.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.082+0000 7fa8d5e00700 1 Processor -- start 2024-05-27T06:57:45.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.082+0000 7fa8d5e00700 1 -- start start 2024-05-27T06:57:45.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.083+0000 7fa8d5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d0109cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:45.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.083+0000 7fa8d5e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d010a2a0 con 0x7fa8d01098a0 2024-05-27T06:57:45.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.083+0000 7fa8d4a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d0109cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:45.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.084+0000 7fa8d4a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d0109cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45872/0 (socket says 172.21.15.43:45872) 2024-05-27T06:57:45.089 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.084+0000 7fa8d4a00700 1 -- 172.21.15.43:0/652606336 learned_addr learned my addr 172.21.15.43:0/652606336 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:45.089 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.084+0000 7fa8d4a00700 1 -- 172.21.15.43:0/652606336 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d010a9d0 con 0x7fa8d01098a0 2024-05-27T06:57:45.089 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.084+0000 7fa8d4a00700 1 --2- 172.21.15.43:0/652606336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d0109cb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40098a0 tx=0x7fa8c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db643350313c784d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:45.089 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.085+0000 7fa8cf400700 1 -- 172.21.15.43:0/652606336 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8c4004160 con 0x7fa8d01098a0 2024-05-27T06:57:45.089 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.085+0000 7fa8cf400700 1 -- 172.21.15.43:0/652606336 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8c4004300 con 0x7fa8d01098a0 2024-05-27T06:57:45.089 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.085+0000 7fa8cf400700 1 -- 172.21.15.43:0/652606336 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8c4011960 con 0x7fa8d01098a0 2024-05-27T06:57:45.089 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.086+0000 7fa8d5e00700 1 -- 172.21.15.43:0/652606336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 msgr2=0x7fa8d0109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.086+0000 7fa8d5e00700 1 --2- 172.21.15.43:0/652606336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d0109cb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40098a0 tx=0x7fa8c4009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.086+0000 7fa8d5e00700 1 -- 172.21.15.43:0/652606336 shutdown_connections 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.086+0000 7fa8d5e00700 1 --2- 172.21.15.43:0/652606336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d0109cb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.086+0000 7fa8d5e00700 1 -- 172.21.15.43:0/652606336 >> 172.21.15.43:0/652606336 conn(0x7fa8d0104f00 msgr2=0x7fa8d0107390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.086+0000 7fa8d5e00700 1 -- 172.21.15.43:0/652606336 shutdown_connections 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.086+0000 7fa8d5e00700 1 -- 172.21.15.43:0/652606336 wait complete. 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.087+0000 7fa8d5e00700 1 Processor -- start 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.087+0000 7fa8d5e00700 1 -- start start 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.088+0000 7fa8d5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.088+0000 7fa8d5e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d010dd80 con 0x7fa8d01098a0 2024-05-27T06:57:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.088+0000 7fa8d4a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:45.093 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.088+0000 7fa8d4a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d019fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45888/0 (socket says 172.21.15.43:45888) 2024-05-27T06:57:45.093 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.088+0000 7fa8d4a00700 1 -- 172.21.15.43:0/3200959361 learned_addr learned my addr 172.21.15.43:0/3200959361 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.088+0000 7fa8d4a00700 1 -- 172.21.15.43:0/3200959361 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d01a0240 con 0x7fa8d01098a0 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.089+0000 7fa8d4a00700 1 --2- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d019fd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4011980 tx=0x7fa8c4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.089+0000 7fa8cd600700 1 -- 172.21.15.43:0/3200959361 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8c401f070 con 0x7fa8d01098a0 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.089+0000 7fa8cd600700 1 -- 172.21.15.43:0/3200959361 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8c4003aa0 con 0x7fa8d01098a0 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.089+0000 7fa8d5e00700 1 -- 172.21.15.43:0/3200959361 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8d01a04d0 con 0x7fa8d01098a0 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.089+0000 7fa8cd600700 1 -- 172.21.15.43:0/3200959361 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8c4019560 con 0x7fa8d01098a0 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.089+0000 7fa8d5e00700 1 -- 172.21.15.43:0/3200959361 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8d01a30e0 con 0x7fa8d01098a0 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.091+0000 7fa8cd600700 1 -- 172.21.15.43:0/3200959361 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fa8c4022070 con 0x7fa8d01098a0 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.091+0000 7fa8d5e00700 1 -- 172.21.15.43:0/3200959361 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8d0064e10 con 0x7fa8d01098a0 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.091+0000 7fa8cd600700 1 --2- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fa8b803e4c0 0x7fa8b80409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.091+0000 7fa8cd600700 1 -- 172.21.15.43:0/3200959361 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa8c4053970 con 0x7fa8d01098a0 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.091+0000 7fa8cfe00700 1 -- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fa8b803e4c0 msgr2=0x7fa8b80409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1651567761 2024-05-27T06:57:45.094 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.091+0000 7fa8cfe00700 1 --2- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fa8b803e4c0 0x7fa8b80409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.097+0000 7fa8cd600700 1 -- 172.21.15.43:0/3200959361 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa8c4023d60 con 0x7fa8d01098a0 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.292+0000 7fa8cfe00700 1 -- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fa8b803e4c0 msgr2=0x7fa8b80409d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1651567761 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.292+0000 7fa8cfe00700 1 --2- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fa8b803e4c0 0x7fa8b80409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.413+0000 7fa8d5e00700 1 -- 172.21.15.43:0/3200959361 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa8d01a36e0 con 0x7fa8d01098a0 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.414+0000 7fa8cd600700 1 -- 172.21.15.43:0/3200959361 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fa8c401ba80 con 0x7fa8d01098a0 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.417+0000 7fa8d5e00700 1 -- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fa8b803e4c0 msgr2=0x7fa8b80409d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.418+0000 7fa8d5e00700 1 --2- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fa8b803e4c0 0x7fa8b80409d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.418+0000 7fa8d5e00700 1 -- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 msgr2=0x7fa8d019fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.418+0000 7fa8d5e00700 1 --2- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d019fd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4011980 tx=0x7fa8c4011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.418+0000 7fa8d5e00700 1 -- 172.21.15.43:0/3200959361 shutdown_connections 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.418+0000 7fa8d5e00700 1 --2- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7fa8b803e4c0 0x7fa8b80409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.418+0000 7fa8d5e00700 1 --2- 172.21.15.43:0/3200959361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8d01098a0 0x7fa8d019fd00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.418+0000 7fa8d5e00700 1 -- 172.21.15.43:0/3200959361 >> 172.21.15.43:0/3200959361 conn(0x7fa8d0104f00 msgr2=0x7fa8d007d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.418+0000 7fa8d5e00700 1 -- 172.21.15.43:0/3200959361 shutdown_connections 2024-05-27T06:57:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:44.418+0000 7fa8d5e00700 1 -- 172.21.15.43:0/3200959361 wait complete. 2024-05-27T06:57:45.096 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for the mgr to restart... 2024-05-27T06:57:45.096 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr epoch 9... 2024-05-27T06:57:45.954 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:45 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:45.736+0000 7fc3b5233200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T06:57:46.373 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:45 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:45.979+0000 7fc3b5233200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T06:57:46.373 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:46 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:46.202+0000 7fc3b5233200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T06:57:46.704 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:46 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:46.601+0000 7fc3b5233200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T06:57:46.984 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:46 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:46.729+0000 7fc3b5233200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T06:57:47.252 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:46 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:46.991+0000 7fc3b5233200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T06:57:47.252 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:47 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:47.259+0000 7fc3b5233200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:57:48.379 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:48.086+0000 7fc3b5233200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T06:57:48.380 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:48.387+0000 7fc3b5233200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T06:57:49.093 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:48.967+0000 7fc3b5233200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T06:57:49.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:49 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:49.101+0000 7fc3b5233200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T06:57:49.954 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:49 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:49.582+0000 7fc3b5233200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T06:57:50.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:50.059+0000 7fc3b5233200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T06:57:50.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:50.189+0000 7fc3b5233200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T06:57:50.846 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:50.529+0000 7fc3b5233200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T06:57:50.847 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:50.717+0000 7fc3b5233200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T06:57:50.847 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:50.854+0000 7fc3b5233200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T06:57:51.704 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:51.267+0000 7fc3b5233200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T06:57:51.704 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:51.397+0000 7fc3b5233200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T06:57:52.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:52.196+0000 7fc3b5233200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T06:57:52.954 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:57:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:57:52.502+0000 7fc3b5233200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T06:57:52.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:52 smithi043 ceph-mon[31531]: Active manager daemon x restarted 2024-05-27T06:57:52.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:52 smithi043 ceph-mon[31531]: Activating manager daemon x 2024-05-27T06:57:52.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:52 smithi043 ceph-mon[31531]: osdmap e3: 0 total, 0 up, 0 in 2024-05-27T06:57:52.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:52 smithi043 ceph-mon[31531]: mgrmap e10: x(active, starting, since 0.00455563s) 2024-05-27T06:57:52.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T06:57:52.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T06:57:52.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T06:57:52.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T06:57:52.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T06:57:52.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:52 smithi043 ceph-mon[31531]: Manager daemon x is now available 2024-05-27T06:57:53.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:57:53.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T06:57:53.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T06:57:53.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:53 smithi043 ceph-mon[31531]: mgrmap e11: x(active, since 1.01846s) 2024-05-27T06:57:54.256 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-05-27T06:57:54.256 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-27T06:57:54.256 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-27T06:57:54.256 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-05-27T06:57:54.256 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.768+0000 7f0e9a800700 1 Processor -- start 2024-05-27T06:57:54.256 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.768+0000 7f0e9a800700 1 -- start start 2024-05-27T06:57:54.256 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.769+0000 7f0e9a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e941086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:54.256 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.769+0000 7f0e9a800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9407d5f0 con 0x7f0e941082e0 2024-05-27T06:57:54.258 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.769+0000 7f0e93400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e941086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:54.258 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.769+0000 7f0e93400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e941086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57810/0 (socket says 172.21.15.43:57810) 2024-05-27T06:57:54.258 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.769+0000 7f0e93400700 1 -- 172.21.15.43:0/3639500206 learned_addr learned my addr 172.21.15.43:0/3639500206 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:54.258 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.770+0000 7f0e93400700 1 -- 172.21.15.43:0/3639500206 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e94108c30 con 0x7f0e941082e0 2024-05-27T06:57:54.258 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.770+0000 7f0e93400700 1 --2- 172.21.15.43:0/3639500206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e941086f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e840098a0 tx=0x7f0e84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4becfe8838f6d02 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:54.258 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.770+0000 7f0e92000700 1 -- 172.21.15.43:0/3639500206 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e84004160 con 0x7f0e941082e0 2024-05-27T06:57:54.258 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.770+0000 7f0e92000700 1 -- 172.21.15.43:0/3639500206 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e84004300 con 0x7f0e941082e0 2024-05-27T06:57:54.258 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.770+0000 7f0e92000700 1 -- 172.21.15.43:0/3639500206 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e84011960 con 0x7f0e941082e0 2024-05-27T06:57:54.258 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.771+0000 7f0e9a800700 1 -- 172.21.15.43:0/3639500206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 msgr2=0x7f0e941086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:54.259 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.771+0000 7f0e9a800700 1 --2- 172.21.15.43:0/3639500206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e941086f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e840098a0 tx=0x7f0e84009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:54.259 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.772+0000 7f0e9a800700 1 -- 172.21.15.43:0/3639500206 shutdown_connections 2024-05-27T06:57:54.259 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.772+0000 7f0e9a800700 1 --2- 172.21.15.43:0/3639500206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e941086f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:54.259 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.772+0000 7f0e9a800700 1 -- 172.21.15.43:0/3639500206 >> 172.21.15.43:0/3639500206 conn(0x7f0e94103f90 msgr2=0x7f0e94106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:54.259 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.772+0000 7f0e9a800700 1 -- 172.21.15.43:0/3639500206 shutdown_connections 2024-05-27T06:57:54.261 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.772+0000 7f0e9a800700 1 -- 172.21.15.43:0/3639500206 wait complete. 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.773+0000 7f0e9a800700 1 Processor -- start 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.773+0000 7f0e9a800700 1 -- start start 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.774+0000 7f0e9a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e9419fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.774+0000 7f0e9a800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0e94109b10 con 0x7f0e941082e0 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.774+0000 7f0e93400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e9419fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.774+0000 7f0e93400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e9419fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57812/0 (socket says 172.21.15.43:57812) 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.774+0000 7f0e93400700 1 -- 172.21.15.43:0/1061463701 learned_addr learned my addr 172.21.15.43:0/1061463701 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.774+0000 7f0e93400700 1 -- 172.21.15.43:0/1061463701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e941a0380 con 0x7f0e941082e0 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.775+0000 7f0e93400700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e9419fe40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0e84011670 tx=0x7f0e84011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.775+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e8401f070 con 0x7f0e941082e0 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.775+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0e84003b20 con 0x7f0e941082e0 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.775+0000 7f0e9a800700 1 -- 172.21.15.43:0/1061463701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e941a0610 con 0x7f0e941082e0 2024-05-27T06:57:54.262 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.775+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e84019580 con 0x7f0e941082e0 2024-05-27T06:57:54.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.775+0000 7f0e9a800700 1 -- 172.21.15.43:0/1061463701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e941a3220 con 0x7f0e941082e0 2024-05-27T06:57:54.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.777+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f0e84022070 con 0x7f0e941082e0 2024-05-27T06:57:54.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.778+0000 7f0e8be00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 0x7f0e78040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:54.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.778+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 --> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0e78041140 con 0x7f0e7803e520 2024-05-27T06:57:54.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.778+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0e84053b60 con 0x7f0e941082e0 2024-05-27T06:57:54.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.778+0000 7f0e92a00700 1 -- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 msgr2=0x7f0e78040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1651567761 2024-05-27T06:57:54.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.778+0000 7f0e92a00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 0x7f0e78040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T06:57:54.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.979+0000 7f0e92a00700 1 -- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 msgr2=0x7f0e78040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1651567761 2024-05-27T06:57:54.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:45.979+0000 7f0e92a00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 0x7f0e78040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T06:57:54.263 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:46.380+0000 7f0e92a00700 1 -- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 msgr2=0x7f0e78040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1651567761 2024-05-27T06:57:54.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:46.380+0000 7f0e92a00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 0x7f0e78040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T06:57:54.266 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:47.181+0000 7f0e92a00700 1 -- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 msgr2=0x7f0e78040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1651567761 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:47.181+0000 7f0e92a00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 0x7f0e78040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:48.783+0000 7f0e92a00700 1 -- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 msgr2=0x7f0e78040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1651567761 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:48.783+0000 7f0e92a00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 0x7f0e78040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:51.986+0000 7f0e92a00700 1 -- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 msgr2=0x7f0e78040a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1651567761 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:51.986+0000 7f0e92a00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 0x7f0e78040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:52.510+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mgrmap(e 10) ==== 50002+0+0 (secure 0 0 0) 0x7f0e8401ba80 con 0x7f0e941082e0 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:52.510+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 msgr2=0x7f0e78040a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:52.510+0000 7f0e8be00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 0x7f0e78040a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.515+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7f0e84055520 con 0x7f0e941082e0 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.515+0000 7f0e8be00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0e840536f0 0x7f0e78044000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.515+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0e84052020 con 0x7f0e840536f0 2024-05-27T06:57:54.267 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.517+0000 7f0e92a00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0e840536f0 0x7f0e78044000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.520+0000 7f0e92a00700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0e840536f0 0x7f0e78044000 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0e80003a10 tx=0x7f0e80009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.521+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f0e84052020 con 0x7f0e840536f0 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.532+0000 7f0e9a800700 1 -- 172.21.15.43:0/1061463701 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f0e94064e10 con 0x7f0e840536f0 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.533+0000 7f0e8be00700 1 -- 172.21.15.43:0/1061463701 <== mgr.14152 v2:172.21.15.43:6800/1809594794 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f0e94064e10 con 0x7f0e840536f0 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.533+0000 7f0e9a800700 1 -- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0e840536f0 msgr2=0x7f0e78044000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.533+0000 7f0e9a800700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0e840536f0 0x7f0e78044000 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0e80003a10 tx=0x7f0e80009270 comp rx=0 tx=0).stop 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.533+0000 7f0e9a800700 1 -- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 msgr2=0x7f0e9419fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.533+0000 7f0e9a800700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e9419fe40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0e84011670 tx=0x7f0e84011a30 comp rx=0 tx=0).stop 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.534+0000 7f0e9a800700 1 -- 172.21.15.43:0/1061463701 shutdown_connections 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.534+0000 7f0e9a800700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0e840536f0 0x7f0e78044000 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.534+0000 7f0e9a800700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0e941082e0 0x7f0e9419fe40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.534+0000 7f0e9a800700 1 --2- 172.21.15.43:0/1061463701 >> [v2:172.21.15.43:6800/1651567761,v1:172.21.15.43:6801/1651567761] conn(0x7f0e7803e520 0x7f0e78040a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:54.268 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.534+0000 7f0e9a800700 1 -- 172.21.15.43:0/1061463701 >> 172.21.15.43:0/1061463701 conn(0x7f0e94103f90 msgr2=0x7f0e94193df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:54.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.534+0000 7f0e9a800700 1 -- 172.21.15.43:0/1061463701 shutdown_connections 2024-05-27T06:57:54.269 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:53.534+0000 7f0e9a800700 1 -- 172.21.15.43:0/1061463701 wait complete. 2024-05-27T06:57:54.269 INFO:teuthology.orchestra.run.smithi043.stdout:mgr epoch 9 is available 2024-05-27T06:57:54.269 INFO:teuthology.orchestra.run.smithi043.stdout:Generating a dashboard self-signed certificate... 2024-05-27T06:57:54.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:54 smithi043 ceph-mon[31531]: [27/May/2024:06:57:53] ENGINE Bus STARTING 2024-05-27T06:57:54.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:54 smithi043 ceph-mon[31531]: [27/May/2024:06:57:53] ENGINE Serving on https://172.21.15.43:7150 2024-05-27T06:57:54.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:54 smithi043 ceph-mon[31531]: [27/May/2024:06:57:53] ENGINE Serving on http://172.21.15.43:8765 2024-05-27T06:57:54.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:54 smithi043 ceph-mon[31531]: [27/May/2024:06:57:53] ENGINE Bus STARTED 2024-05-27T06:57:54.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:54 smithi043 ceph-mon[31531]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-27T06:57:54.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:54 smithi043 ceph-mon[31531]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-27T06:57:55.750 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:55 smithi043 ceph-mon[31531]: mgrmap e12: x(active, since 2s) 2024-05-27T06:57:55.750 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:55 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:57:56.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:56 smithi043 ceph-mon[31531]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:57:56.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:56 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:57:56.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:56 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:57:56.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-27T06:57:56.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.494+0000 7f3110400700 1 Processor -- start 2024-05-27T06:57:56.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.494+0000 7f3110400700 1 -- start start 2024-05-27T06:57:56.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.495+0000 7f3110400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f3108109e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.495+0000 7f3110400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f310810a480 con 0x7f3108109a80 2024-05-27T06:57:56.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.495+0000 7f310da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f3108109e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.495+0000 7f310da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f3108109e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33298/0 (socket says 172.21.15.43:33298) 2024-05-27T06:57:56.720 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.495+0000 7f310da00700 1 -- 172.21.15.43:0/2425703935 learned_addr learned my addr 172.21.15.43:0/2425703935 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:56.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.496+0000 7f310da00700 1 -- 172.21.15.43:0/2425703935 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310810abb0 con 0x7f3108109a80 2024-05-27T06:57:56.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.496+0000 7f310da00700 1 --2- 172.21.15.43:0/2425703935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f3108109e90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f30fc0098a0 tx=0x7f30fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f6bc5b85b7bbd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:56.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.496+0000 7f3107e00700 1 -- 172.21.15.43:0/2425703935 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30fc004160 con 0x7f3108109a80 2024-05-27T06:57:56.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.497+0000 7f3107e00700 1 -- 172.21.15.43:0/2425703935 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30fc004300 con 0x7f3108109a80 2024-05-27T06:57:56.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.497+0000 7f3107e00700 1 -- 172.21.15.43:0/2425703935 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30fc011960 con 0x7f3108109a80 2024-05-27T06:57:56.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.498+0000 7f3110400700 1 -- 172.21.15.43:0/2425703935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 msgr2=0x7f3108109e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:56.721 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.498+0000 7f3110400700 1 --2- 172.21.15.43:0/2425703935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f3108109e90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f30fc0098a0 tx=0x7f30fc009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.498+0000 7f3110400700 1 -- 172.21.15.43:0/2425703935 shutdown_connections 2024-05-27T06:57:56.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.498+0000 7f3110400700 1 --2- 172.21.15.43:0/2425703935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f3108109e90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.722 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.498+0000 7f3110400700 1 -- 172.21.15.43:0/2425703935 >> 172.21.15.43:0/2425703935 conn(0x7f3108105100 msgr2=0x7f3108107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:56.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.498+0000 7f3110400700 1 -- 172.21.15.43:0/2425703935 shutdown_connections 2024-05-27T06:57:56.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.498+0000 7f3110400700 1 -- 172.21.15.43:0/2425703935 wait complete. 2024-05-27T06:57:56.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.499+0000 7f3110400700 1 Processor -- start 2024-05-27T06:57:56.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.499+0000 7f3110400700 1 -- start start 2024-05-27T06:57:56.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.500+0000 7f3110400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f310819dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.500+0000 7f3110400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f310810df60 con 0x7f3108109a80 2024-05-27T06:57:56.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.500+0000 7f310da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f310819dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.726 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.500+0000 7f310da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f310819dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33312/0 (socket says 172.21.15.43:33312) 2024-05-27T06:57:56.727 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.500+0000 7f310da00700 1 -- 172.21.15.43:0/1812957873 learned_addr learned my addr 172.21.15.43:0/1812957873 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:56.727 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.500+0000 7f310da00700 1 -- 172.21.15.43:0/1812957873 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310819e1c0 con 0x7f3108109a80 2024-05-27T06:57:56.727 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.501+0000 7f310da00700 1 --2- 172.21.15.43:0/1812957873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f310819dc80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f30fc011980 tx=0x7f30fc011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:56.727 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.501+0000 7f3106000700 1 -- 172.21.15.43:0/1812957873 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30fc01f070 con 0x7f3108109a80 2024-05-27T06:57:56.727 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.501+0000 7f3106000700 1 -- 172.21.15.43:0/1812957873 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30fc003aa0 con 0x7f3108109a80 2024-05-27T06:57:56.727 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.501+0000 7f3110400700 1 -- 172.21.15.43:0/1812957873 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f310819e450 con 0x7f3108109a80 2024-05-27T06:57:56.727 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.501+0000 7f3106000700 1 -- 172.21.15.43:0/1812957873 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30fc019560 con 0x7f3108109a80 2024-05-27T06:57:56.727 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.501+0000 7f3110400700 1 -- 172.21.15.43:0/1812957873 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31081a1060 con 0x7f3108109a80 2024-05-27T06:57:56.727 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.503+0000 7f3110400700 1 -- 172.21.15.43:0/1812957873 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f310810e0e0 con 0x7f3108109a80 2024-05-27T06:57:56.727 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.503+0000 7f3106000700 1 -- 172.21.15.43:0/1812957873 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f30fc022070 con 0x7f3108109a80 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.504+0000 7f3106000700 1 --2- 172.21.15.43:0/1812957873 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f30f003e4c0 0x7f30f00409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.504+0000 7f3106000700 1 -- 172.21.15.43:0/1812957873 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f30fc053d20 con 0x7f3108109a80 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.504+0000 7f310d000700 1 --2- 172.21.15.43:0/1812957873 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f30f003e4c0 0x7f30f00409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.505+0000 7f310d000700 1 --2- 172.21.15.43:0/1812957873 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f30f003e4c0 0x7f30f00409d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f30f4006fd0 tx=0x7f30f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.510+0000 7f3106000700 1 -- 172.21.15.43:0/1812957873 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f30fc019d30 con 0x7f3108109a80 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:55.757+0000 7f3110400700 1 -- 172.21.15.43:0/1812957873 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f3108106080 con 0x7f30f003e4c0 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.025+0000 7f3106000700 1 -- 172.21.15.43:0/1812957873 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f3108106080 con 0x7f30f003e4c0 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.028+0000 7f3110400700 1 -- 172.21.15.43:0/1812957873 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f30f003e4c0 msgr2=0x7f30f00409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.028+0000 7f3110400700 1 --2- 172.21.15.43:0/1812957873 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f30f003e4c0 0x7f30f00409d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f30f4006fd0 tx=0x7f30f4006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.028+0000 7f3110400700 1 -- 172.21.15.43:0/1812957873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 msgr2=0x7f310819dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.028+0000 7f3110400700 1 --2- 172.21.15.43:0/1812957873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f310819dc80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f30fc011980 tx=0x7f30fc011a60 comp rx=0 tx=0).stop 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.028+0000 7f3110400700 1 -- 172.21.15.43:0/1812957873 shutdown_connections 2024-05-27T06:57:56.728 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.028+0000 7f3110400700 1 --2- 172.21.15.43:0/1812957873 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f30f003e4c0 0x7f30f00409d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.729 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.028+0000 7f3110400700 1 --2- 172.21.15.43:0/1812957873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3108109a80 0x7f310819dc80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.729 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.028+0000 7f3110400700 1 -- 172.21.15.43:0/1812957873 >> 172.21.15.43:0/1812957873 conn(0x7f3108105100 msgr2=0x7f3108105970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:56.729 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.029+0000 7f3110400700 1 -- 172.21.15.43:0/1812957873 shutdown_connections 2024-05-27T06:57:56.729 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:56.029+0000 7f3110400700 1 -- 172.21.15.43:0/1812957873 wait complete. 2024-05-27T06:57:56.729 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial admin user... 2024-05-27T06:57:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:57:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:58 smithi043 ceph-mon[31531]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:57:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:57:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:57:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$zoeEuYMoOxzFCPOzpDwKJu6G0yMNxaAAaBu15CmH.8c3FwaoaZlAK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1716793077, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-27T06:57:58.607 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.384+0000 7f5a4c400700 1 Processor -- start 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.384+0000 7f5a4c400700 1 -- start start 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.385+0000 7f5a4c400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a441081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.385+0000 7f5a4c400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a441087a0 con 0x7f5a44107da0 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.385+0000 7f5a49a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a441081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.385+0000 7f5a49a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a441081b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33320/0 (socket says 172.21.15.43:33320) 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.385+0000 7f5a49a00700 1 -- 172.21.15.43:0/3424826868 learned_addr learned my addr 172.21.15.43:0/3424826868 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.386+0000 7f5a49a00700 1 -- 172.21.15.43:0/3424826868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a44108920 con 0x7f5a44107da0 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.386+0000 7f5a49a00700 1 --2- 172.21.15.43:0/3424826868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a441081b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a380098a0 tx=0x7f5a38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8d7e7de8a1cbd5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.386+0000 7f5a43e00700 1 -- 172.21.15.43:0/3424826868 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a38004160 con 0x7f5a44107da0 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.386+0000 7f5a43e00700 1 -- 172.21.15.43:0/3424826868 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a38004300 con 0x7f5a44107da0 2024-05-27T06:57:58.608 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.386+0000 7f5a43e00700 1 -- 172.21.15.43:0/3424826868 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a38011960 con 0x7f5a44107da0 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.387+0000 7f5a4c400700 1 -- 172.21.15.43:0/3424826868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 msgr2=0x7f5a441081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.387+0000 7f5a4c400700 1 --2- 172.21.15.43:0/3424826868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a441081b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a380098a0 tx=0x7f5a38009bb0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.387+0000 7f5a4c400700 1 -- 172.21.15.43:0/3424826868 shutdown_connections 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.388+0000 7f5a4c400700 1 --2- 172.21.15.43:0/3424826868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a441081b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.388+0000 7f5a4c400700 1 -- 172.21.15.43:0/3424826868 >> 172.21.15.43:0/3424826868 conn(0x7f5a44103720 msgr2=0x7f5a44105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.388+0000 7f5a4c400700 1 -- 172.21.15.43:0/3424826868 shutdown_connections 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.388+0000 7f5a4c400700 1 -- 172.21.15.43:0/3424826868 wait complete. 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.389+0000 7f5a4c400700 1 Processor -- start 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.389+0000 7f5a4c400700 1 -- start start 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.389+0000 7f5a4c400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a4419fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:58.613 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.389+0000 7f5a4c400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4410dd60 con 0x7f5a44107da0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.390+0000 7f5a49a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a4419fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.390+0000 7f5a49a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a4419fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33332/0 (socket says 172.21.15.43:33332) 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.390+0000 7f5a49a00700 1 -- 172.21.15.43:0/3180488194 learned_addr learned my addr 172.21.15.43:0/3180488194 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.390+0000 7f5a49a00700 1 -- 172.21.15.43:0/3180488194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a441a0410 con 0x7f5a44107da0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.391+0000 7f5a49a00700 1 --2- 172.21.15.43:0/3180488194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a4419fed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5a38003f50 tx=0x7f5a38011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.391+0000 7f5a42000700 1 -- 172.21.15.43:0/3180488194 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a3801f070 con 0x7f5a44107da0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.391+0000 7f5a42000700 1 -- 172.21.15.43:0/3180488194 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a38003aa0 con 0x7f5a44107da0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.391+0000 7f5a42000700 1 -- 172.21.15.43:0/3180488194 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a38019580 con 0x7f5a44107da0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.391+0000 7f5a4c400700 1 -- 172.21.15.43:0/3180488194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a441a06a0 con 0x7f5a44107da0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.391+0000 7f5a4c400700 1 -- 172.21.15.43:0/3180488194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a441a32b0 con 0x7f5a44107da0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.392+0000 7f5a4c400700 1 -- 172.21.15.43:0/3180488194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a44064e10 con 0x7f5a44107da0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.393+0000 7f5a42000700 1 -- 172.21.15.43:0/3180488194 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f5a38022070 con 0x7f5a44107da0 2024-05-27T06:57:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.393+0000 7f5a42000700 1 --2- 172.21.15.43:0/3180488194 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a2c03e510 0x7f5a2c040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.393+0000 7f5a42000700 1 -- 172.21.15.43:0/3180488194 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5a38053b50 con 0x7f5a44107da0 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.394+0000 7f5a49000700 1 --2- 172.21.15.43:0/3180488194 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a2c03e510 0x7f5a2c040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.395+0000 7f5a49000700 1 --2- 172.21.15.43:0/3180488194 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a2c03e510 0x7f5a2c040a20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a30006fd0 tx=0x7f5a30006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.400+0000 7f5a42000700 1 -- 172.21.15.43:0/3180488194 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5a3801ba80 con 0x7f5a44107da0 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.658+0000 7f5a4c400700 1 -- 172.21.15.43:0/3180488194 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f5a441a3480 con 0x7f5a2c03e510 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.933+0000 7f5a42000700 1 -- 172.21.15.43:0/3180488194 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f5a441a3480 con 0x7f5a2c03e510 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.936+0000 7f5a4c400700 1 -- 172.21.15.43:0/3180488194 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a2c03e510 msgr2=0x7f5a2c040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.936+0000 7f5a4c400700 1 --2- 172.21.15.43:0/3180488194 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a2c03e510 0x7f5a2c040a20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a30006fd0 tx=0x7f5a30006e40 comp rx=0 tx=0).stop 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.936+0000 7f5a4c400700 1 -- 172.21.15.43:0/3180488194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 msgr2=0x7f5a4419fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.936+0000 7f5a4c400700 1 --2- 172.21.15.43:0/3180488194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a4419fed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5a38003f50 tx=0x7f5a38011a30 comp rx=0 tx=0).stop 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.937+0000 7f5a4c400700 1 -- 172.21.15.43:0/3180488194 shutdown_connections 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.937+0000 7f5a4c400700 1 --2- 172.21.15.43:0/3180488194 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a2c03e510 0x7f5a2c040a20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.937+0000 7f5a4c400700 1 --2- 172.21.15.43:0/3180488194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a44107da0 0x7f5a4419fed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.937+0000 7f5a4c400700 1 -- 172.21.15.43:0/3180488194 >> 172.21.15.43:0/3180488194 conn(0x7f5a44103720 msgr2=0x7f5a44103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.937+0000 7f5a4c400700 1 -- 172.21.15.43:0/3180488194 shutdown_connections 2024-05-27T06:57:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:57.937+0000 7f5a4c400700 1 -- 172.21.15.43:0/3180488194 wait complete. 2024-05-27T06:57:58.616 INFO:teuthology.orchestra.run.smithi043.stdout:Fetching dashboard port number... 2024-05-27T06:58:00.303 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:00 smithi043 ceph-mon[31531]: mgrmap e13: x(active, since 6s) 2024-05-27T06:58:00.304 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:00 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/507373847' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-27T06:58:00.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 8443 2024-05-27T06:58:00.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.548+0000 7f6442000700 1 Processor -- start 2024-05-27T06:58:00.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.549+0000 7f6442000700 1 -- start start 2024-05-27T06:58:00.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.549+0000 7f6442000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c07d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:00.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.549+0000 7f6442000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f643c07d700 con 0x7f643c07ecf0 2024-05-27T06:58:00.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.550+0000 7f6440c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c07d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:00.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.550+0000 7f6440c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c07d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33334/0 (socket says 172.21.15.43:33334) 2024-05-27T06:58:00.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.550+0000 7f6440c00700 1 -- 172.21.15.43:0/1671896759 learned_addr learned my addr 172.21.15.43:0/1671896759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:00.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.550+0000 7f6440c00700 1 -- 172.21.15.43:0/1671896759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643c07d880 con 0x7f643c07ecf0 2024-05-27T06:58:00.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.550+0000 7f6440c00700 1 --2- 172.21.15.43:0/1671896759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c07d1c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f642c0098a0 tx=0x7f642c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc56f5a870dc112e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:00.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.551+0000 7f643be00700 1 -- 172.21.15.43:0/1671896759 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f642c004160 con 0x7f643c07ecf0 2024-05-27T06:58:00.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.551+0000 7f643be00700 1 -- 172.21.15.43:0/1671896759 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f642c004300 con 0x7f643c07ecf0 2024-05-27T06:58:00.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.551+0000 7f643be00700 1 -- 172.21.15.43:0/1671896759 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f642c011960 con 0x7f643c07ecf0 2024-05-27T06:58:00.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.552+0000 7f6442000700 1 -- 172.21.15.43:0/1671896759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 msgr2=0x7f643c07d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:00.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.552+0000 7f6442000700 1 --2- 172.21.15.43:0/1671896759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c07d1c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f642c0098a0 tx=0x7f642c009bb0 comp rx=0 tx=0).stop 2024-05-27T06:58:00.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.552+0000 7f6442000700 1 -- 172.21.15.43:0/1671896759 shutdown_connections 2024-05-27T06:58:00.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.552+0000 7f6442000700 1 --2- 172.21.15.43:0/1671896759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c07d1c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:00.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.552+0000 7f6442000700 1 -- 172.21.15.43:0/1671896759 >> 172.21.15.43:0/1671896759 conn(0x7f643c104f00 msgr2=0x7f643c107390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:00.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.552+0000 7f6442000700 1 -- 172.21.15.43:0/1671896759 shutdown_connections 2024-05-27T06:58:00.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.552+0000 7f6442000700 1 -- 172.21.15.43:0/1671896759 wait complete. 2024-05-27T06:58:00.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.553+0000 7f6442000700 1 Processor -- start 2024-05-27T06:58:00.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.553+0000 7f6442000700 1 -- start start 2024-05-27T06:58:00.674 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.554+0000 7f6442000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:00.674 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.554+0000 7f6442000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f643c10fe00 con 0x7f643c07ecf0 2024-05-27T06:58:00.674 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.554+0000 7f6440c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:00.674 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.554+0000 7f6440c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c19fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33340/0 (socket says 172.21.15.43:33340) 2024-05-27T06:58:00.674 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.554+0000 7f6440c00700 1 -- 172.21.15.43:0/507373847 learned_addr learned my addr 172.21.15.43:0/507373847 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:00.674 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.554+0000 7f6440c00700 1 -- 172.21.15.43:0/507373847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643c1a01c0 con 0x7f643c07ecf0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.555+0000 7f6440c00700 1 --2- 172.21.15.43:0/507373847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c19fc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f642c011670 tx=0x7f642c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.555+0000 7f643a000700 1 -- 172.21.15.43:0/507373847 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f642c01f070 con 0x7f643c07ecf0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.555+0000 7f643a000700 1 -- 172.21.15.43:0/507373847 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f642c003b20 con 0x7f643c07ecf0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.555+0000 7f643a000700 1 -- 172.21.15.43:0/507373847 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f642c019580 con 0x7f643c07ecf0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.555+0000 7f6442000700 1 -- 172.21.15.43:0/507373847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f643c1a0450 con 0x7f643c07ecf0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.555+0000 7f6442000700 1 -- 172.21.15.43:0/507373847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f643c1a3060 con 0x7f643c07ecf0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.556+0000 7f6442000700 1 -- 172.21.15.43:0/507373847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f643c064e10 con 0x7f643c07ecf0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.557+0000 7f643a000700 1 -- 172.21.15.43:0/507373847 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f642c022070 con 0x7f643c07ecf0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.557+0000 7f643a000700 1 --2- 172.21.15.43:0/507373847 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f642003e5c0 0x7f6420040ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.557+0000 7f643a000700 1 -- 172.21.15.43:0/507373847 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f642c053ce0 con 0x7f643c07ecf0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.558+0000 7f6433e00700 1 --2- 172.21.15.43:0/507373847 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f642003e5c0 0x7f6420040ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.558+0000 7f6433e00700 1 --2- 172.21.15.43:0/507373847 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f642003e5c0 0x7f6420040ad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6424006fd0 tx=0x7f6424006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:00.675 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.563+0000 7f643a000700 1 -- 172.21.15.43:0/507373847 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f642c01ba80 con 0x7f643c07ecf0 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.803+0000 7f6442000700 1 -- 172.21.15.43:0/507373847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f643c1a3350 con 0x7f643c07ecf0 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.804+0000 7f643a000700 1 -- 172.21.15.43:0/507373847 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f642c02ca40 con 0x7f643c07ecf0 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.807+0000 7f6442000700 1 -- 172.21.15.43:0/507373847 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f642003e5c0 msgr2=0x7f6420040ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.807+0000 7f6442000700 1 --2- 172.21.15.43:0/507373847 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f642003e5c0 0x7f6420040ad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6424006fd0 tx=0x7f6424006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.807+0000 7f6442000700 1 -- 172.21.15.43:0/507373847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 msgr2=0x7f643c19fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.807+0000 7f6442000700 1 --2- 172.21.15.43:0/507373847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c19fc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f642c011670 tx=0x7f642c011a30 comp rx=0 tx=0).stop 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.807+0000 7f6442000700 1 -- 172.21.15.43:0/507373847 shutdown_connections 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.807+0000 7f6442000700 1 --2- 172.21.15.43:0/507373847 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f642003e5c0 0x7f6420040ad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.807+0000 7f6442000700 1 --2- 172.21.15.43:0/507373847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f643c07ecf0 0x7f643c19fc80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.807+0000 7f6442000700 1 -- 172.21.15.43:0/507373847 >> 172.21.15.43:0/507373847 conn(0x7f643c104f00 msgr2=0x7f643c105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.807+0000 7f6442000700 1 -- 172.21.15.43:0/507373847 shutdown_connections 2024-05-27T06:58:00.676 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:57:59.808+0000 7f6442000700 1 -- 172.21.15.43:0/507373847 wait complete. 2024-05-27T06:58:00.713 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-27T06:58:00.713 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-05-27T06:58:00.740 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-27T06:58:00.740 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-05-27T06:58:00.740 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-05-27T06:58:00.740 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-27T06:58:00.748 INFO:teuthology.orchestra.run.smithi043.stdout:Ceph Dashboard is now available at: 2024-05-27T06:58:00.748 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:00.748 INFO:teuthology.orchestra.run.smithi043.stdout: URL: https://smithi043.front.sepia.ceph.com:8443/ 2024-05-27T06:58:00.748 INFO:teuthology.orchestra.run.smithi043.stdout: User: admin 2024-05-27T06:58:00.748 INFO:teuthology.orchestra.run.smithi043.stdout: Password: 2jlrzt7h1a 2024-05-27T06:58:00.748 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:00.749 INFO:teuthology.orchestra.run.smithi043.stdout:Saving cluster configuration to /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config directory 2024-05-27T06:58:00.751 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling autotune for osd_memory_target 2024-05-27T06:58:02.344 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.486+0000 7fd312e00700 1 Processor -- start 2024-05-27T06:58:02.344 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.486+0000 7fd312e00700 1 -- start start 2024-05-27T06:58:02.345 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.486+0000 7fd312e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c10c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:02.345 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.486+0000 7fd312e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd30c10c610 con 0x7fd30c10bc10 2024-05-27T06:58:02.345 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.487+0000 7fd30be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c10c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:02.345 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.487+0000 7fd30be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c10c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33356/0 (socket says 172.21.15.43:33356) 2024-05-27T06:58:02.345 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.487+0000 7fd30be00700 1 -- 172.21.15.43:0/60528882 learned_addr learned my addr 172.21.15.43:0/60528882 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:02.346 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.487+0000 7fd30be00700 1 -- 172.21.15.43:0/60528882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd30c10cd40 con 0x7fd30c10bc10 2024-05-27T06:58:02.346 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.488+0000 7fd30be00700 1 --2- 172.21.15.43:0/60528882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c10c020 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0098a0 tx=0x7fd2fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efc3c50a7caa9c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:02.346 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.488+0000 7fd30aa00700 1 -- 172.21.15.43:0/60528882 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2fc004160 con 0x7fd30c10bc10 2024-05-27T06:58:02.346 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.488+0000 7fd30aa00700 1 -- 172.21.15.43:0/60528882 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2fc004300 con 0x7fd30c10bc10 2024-05-27T06:58:02.346 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.488+0000 7fd30aa00700 1 -- 172.21.15.43:0/60528882 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2fc011960 con 0x7fd30c10bc10 2024-05-27T06:58:02.348 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.489+0000 7fd312e00700 1 -- 172.21.15.43:0/60528882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 msgr2=0x7fd30c10c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:02.348 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.489+0000 7fd312e00700 1 --2- 172.21.15.43:0/60528882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c10c020 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0098a0 tx=0x7fd2fc009bb0 comp rx=0 tx=0).stop 2024-05-27T06:58:02.348 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.489+0000 7fd312e00700 1 -- 172.21.15.43:0/60528882 shutdown_connections 2024-05-27T06:58:02.348 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.489+0000 7fd312e00700 1 --2- 172.21.15.43:0/60528882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c10c020 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:02.348 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.489+0000 7fd312e00700 1 -- 172.21.15.43:0/60528882 >> 172.21.15.43:0/60528882 conn(0x7fd30c07f130 msgr2=0x7fd30c109ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:02.349 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.489+0000 7fd312e00700 1 -- 172.21.15.43:0/60528882 shutdown_connections 2024-05-27T06:58:02.349 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.489+0000 7fd312e00700 1 -- 172.21.15.43:0/60528882 wait complete. 2024-05-27T06:58:02.349 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.490+0000 7fd312e00700 1 Processor -- start 2024-05-27T06:58:02.349 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.490+0000 7fd312e00700 1 -- start start 2024-05-27T06:58:02.349 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.491+0000 7fd312e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c19feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:02.349 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.491+0000 7fd312e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd30c1100f0 con 0x7fd30c10bc10 2024-05-27T06:58:02.349 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.491+0000 7fd30be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c19feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:02.349 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.491+0000 7fd30be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c19feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33358/0 (socket says 172.21.15.43:33358) 2024-05-27T06:58:02.351 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.491+0000 7fd30be00700 1 -- 172.21.15.43:0/4025467769 learned_addr learned my addr 172.21.15.43:0/4025467769 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:02.351 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.491+0000 7fd30be00700 1 -- 172.21.15.43:0/4025467769 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd30c1a03f0 con 0x7fd30c10bc10 2024-05-27T06:58:02.351 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.492+0000 7fd30be00700 1 --2- 172.21.15.43:0/4025467769 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c19feb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc011980 tx=0x7fd2fc011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:02.351 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.492+0000 7fd308c00700 1 -- 172.21.15.43:0/4025467769 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2fc01f070 con 0x7fd30c10bc10 2024-05-27T06:58:02.351 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.492+0000 7fd308c00700 1 -- 172.21.15.43:0/4025467769 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2fc003aa0 con 0x7fd30c10bc10 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.492+0000 7fd312e00700 1 -- 172.21.15.43:0/4025467769 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd30c1a0680 con 0x7fd30c10bc10 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.492+0000 7fd308c00700 1 -- 172.21.15.43:0/4025467769 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2fc019560 con 0x7fd30c10bc10 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.492+0000 7fd312e00700 1 -- 172.21.15.43:0/4025467769 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd30c1a3290 con 0x7fd30c10bc10 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.493+0000 7fd312e00700 1 -- 172.21.15.43:0/4025467769 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd30c064e10 con 0x7fd30c10bc10 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.494+0000 7fd308c00700 1 -- 172.21.15.43:0/4025467769 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fd2fc022070 con 0x7fd30c10bc10 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.494+0000 7fd308c00700 1 --2- 172.21.15.43:0/4025467769 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd2f003e5b0 0x7fd2f0040ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.495+0000 7fd308c00700 1 -- 172.21.15.43:0/4025467769 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd2fc053d50 con 0x7fd30c10bc10 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.495+0000 7fd30b400700 1 --2- 172.21.15.43:0/4025467769 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd2f003e5b0 0x7fd2f0040ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.496+0000 7fd30b400700 1 --2- 172.21.15.43:0/4025467769 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd2f003e5b0 0x7fd2f0040ac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8006fd0 tx=0x7fd2f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.501+0000 7fd308c00700 1 -- 172.21.15.43:0/4025467769 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd2fc019810 con 0x7fd30c10bc10 2024-05-27T06:58:02.352 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.740+0000 7fd312e00700 1 -- 172.21.15.43:0/4025467769 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fd30c1a3460 con 0x7fd30c10bc10 2024-05-27T06:58:02.353 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.742+0000 7fd308c00700 1 -- 172.21.15.43:0/4025467769 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fd2fc00bd00 con 0x7fd30c10bc10 2024-05-27T06:58:02.354 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.745+0000 7fd312e00700 1 -- 172.21.15.43:0/4025467769 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd2f003e5b0 msgr2=0x7fd2f0040ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:02.354 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.745+0000 7fd312e00700 1 --2- 172.21.15.43:0/4025467769 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd2f003e5b0 0x7fd2f0040ac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8006fd0 tx=0x7fd2f8006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:02.354 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.745+0000 7fd312e00700 1 -- 172.21.15.43:0/4025467769 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 msgr2=0x7fd30c19feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:02.354 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.745+0000 7fd312e00700 1 --2- 172.21.15.43:0/4025467769 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c19feb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc011980 tx=0x7fd2fc011a60 comp rx=0 tx=0).stop 2024-05-27T06:58:02.354 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.746+0000 7fd312e00700 1 -- 172.21.15.43:0/4025467769 shutdown_connections 2024-05-27T06:58:02.354 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.746+0000 7fd312e00700 1 --2- 172.21.15.43:0/4025467769 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd2f003e5b0 0x7fd2f0040ac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:02.354 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.746+0000 7fd312e00700 1 --2- 172.21.15.43:0/4025467769 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd30c10bc10 0x7fd30c19feb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:02.354 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.746+0000 7fd312e00700 1 -- 172.21.15.43:0/4025467769 >> 172.21.15.43:0/4025467769 conn(0x7fd30c07f130 msgr2=0x7fd30c109130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:02.354 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.746+0000 7fd312e00700 1 -- 172.21.15.43:0/4025467769 shutdown_connections 2024-05-27T06:58:02.354 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:01.746+0000 7fd312e00700 1 -- 172.21.15.43:0/4025467769 wait complete. 2024-05-27T06:58:04.055 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.025+0000 7fc883a00700 1 Processor -- start 2024-05-27T06:58:04.055 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.025+0000 7fc883a00700 1 -- start start 2024-05-27T06:58:04.055 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.026+0000 7fc883a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c10bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:04.056 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.026+0000 7fc883a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c10c5e0 con 0x7fc87c10bbe0 2024-05-27T06:58:04.056 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.026+0000 7fc881000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c10bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:04.056 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.026+0000 7fc881000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c10bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33368/0 (socket says 172.21.15.43:33368) 2024-05-27T06:58:04.056 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.026+0000 7fc881000700 1 -- 172.21.15.43:0/541176851 learned_addr learned my addr 172.21.15.43:0/541176851 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:04.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.027+0000 7fc881000700 1 -- 172.21.15.43:0/541176851 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87c10cd10 con 0x7fc87c10bbe0 2024-05-27T06:58:04.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.027+0000 7fc881000700 1 --2- 172.21.15.43:0/541176851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c10bff0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc8700098a0 tx=0x7fc870009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbeb138e1d32d88 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:04.059 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.028+0000 7fc87b400700 1 -- 172.21.15.43:0/541176851 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc870004160 con 0x7fc87c10bbe0 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.028+0000 7fc87b400700 1 -- 172.21.15.43:0/541176851 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc870004300 con 0x7fc87c10bbe0 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.028+0000 7fc87b400700 1 -- 172.21.15.43:0/541176851 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc870011960 con 0x7fc87c10bbe0 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.029+0000 7fc883a00700 1 -- 172.21.15.43:0/541176851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 msgr2=0x7fc87c10bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.029+0000 7fc883a00700 1 --2- 172.21.15.43:0/541176851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c10bff0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc8700098a0 tx=0x7fc870009bb0 comp rx=0 tx=0).stop 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.029+0000 7fc883a00700 1 -- 172.21.15.43:0/541176851 shutdown_connections 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.029+0000 7fc883a00700 1 --2- 172.21.15.43:0/541176851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c10bff0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.029+0000 7fc883a00700 1 -- 172.21.15.43:0/541176851 >> 172.21.15.43:0/541176851 conn(0x7fc87c107440 msgr2=0x7fc87c109860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.029+0000 7fc883a00700 1 -- 172.21.15.43:0/541176851 shutdown_connections 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.029+0000 7fc883a00700 1 -- 172.21.15.43:0/541176851 wait complete. 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.030+0000 7fc883a00700 1 Processor -- start 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.030+0000 7fc883a00700 1 -- start start 2024-05-27T06:58:04.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.031+0000 7fc883a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c19fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.031+0000 7fc883a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c1100c0 con 0x7fc87c10bbe0 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.031+0000 7fc881000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c19fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.031+0000 7fc881000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c19fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33372/0 (socket says 172.21.15.43:33372) 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.031+0000 7fc881000700 1 -- 172.21.15.43:0/2284709126 learned_addr learned my addr 172.21.15.43:0/2284709126 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.032+0000 7fc881000700 1 -- 172.21.15.43:0/2284709126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87c1a03c0 con 0x7fc87c10bbe0 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.032+0000 7fc881000700 1 --2- 172.21.15.43:0/2284709126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c19fe80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc870011980 tx=0x7fc870011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.032+0000 7fc879600700 1 -- 172.21.15.43:0/2284709126 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc87001f070 con 0x7fc87c10bbe0 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.032+0000 7fc879600700 1 -- 172.21.15.43:0/2284709126 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc870003aa0 con 0x7fc87c10bbe0 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.032+0000 7fc879600700 1 -- 172.21.15.43:0/2284709126 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc870019560 con 0x7fc87c10bbe0 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.032+0000 7fc883a00700 1 -- 172.21.15.43:0/2284709126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87c1a0650 con 0x7fc87c10bbe0 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.033+0000 7fc883a00700 1 -- 172.21.15.43:0/2284709126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc87c1a3260 con 0x7fc87c10bbe0 2024-05-27T06:58:04.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.034+0000 7fc879600700 1 -- 172.21.15.43:0/2284709126 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fc870022070 con 0x7fc87c10bbe0 2024-05-27T06:58:04.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.034+0000 7fc883a00700 1 -- 172.21.15.43:0/2284709126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc87c064e10 con 0x7fc87c10bbe0 2024-05-27T06:58:04.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.034+0000 7fc879600700 1 --2- 172.21.15.43:0/2284709126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc86403e5b0 0x7fc864040ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:04.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.034+0000 7fc879600700 1 -- 172.21.15.43:0/2284709126 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc870053b30 con 0x7fc87c10bbe0 2024-05-27T06:58:04.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.035+0000 7fc87be00700 1 --2- 172.21.15.43:0/2284709126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc86403e5b0 0x7fc864040ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:04.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.035+0000 7fc87be00700 1 --2- 172.21.15.43:0/2284709126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc86403e5b0 0x7fc864040ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc868006fd0 tx=0x7fc868006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:04.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.040+0000 7fc879600700 1 -- 172.21.15.43:0/2284709126 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc87000bcc0 con 0x7fc87c10bbe0 2024-05-27T06:58:04.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.404+0000 7fc883a00700 1 -- 172.21.15.43:0/2284709126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fc87c1a3430 con 0x7fc87c10bbe0 2024-05-27T06:58:04.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.410+0000 7fc879600700 1 -- 172.21.15.43:0/2284709126 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fc87002a260 con 0x7fc87c10bbe0 2024-05-27T06:58:04.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.414+0000 7fc883a00700 1 -- 172.21.15.43:0/2284709126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc86403e5b0 msgr2=0x7fc864040ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:04.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.414+0000 7fc883a00700 1 --2- 172.21.15.43:0/2284709126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc86403e5b0 0x7fc864040ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc868006fd0 tx=0x7fc868006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:04.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.414+0000 7fc883a00700 1 -- 172.21.15.43:0/2284709126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 msgr2=0x7fc87c19fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:04.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.414+0000 7fc883a00700 1 --2- 172.21.15.43:0/2284709126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c19fe80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc870011980 tx=0x7fc870011a60 comp rx=0 tx=0).stop 2024-05-27T06:58:04.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.414+0000 7fc883a00700 1 -- 172.21.15.43:0/2284709126 shutdown_connections 2024-05-27T06:58:04.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.414+0000 7fc883a00700 1 --2- 172.21.15.43:0/2284709126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc86403e5b0 0x7fc864040ac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.414+0000 7fc883a00700 1 --2- 172.21.15.43:0/2284709126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc87c10bbe0 0x7fc87c19fe80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.414+0000 7fc883a00700 1 -- 172.21.15.43:0/2284709126 >> 172.21.15.43:0/2284709126 conn(0x7fc87c107440 msgr2=0x7fc87c108c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.414+0000 7fc883a00700 1 -- 172.21.15.43:0/2284709126 shutdown_connections 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-05-27T06:58:03.414+0000 7fc883a00700 1 -- 172.21.15.43:0/2284709126 wait complete. 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout:Or, if you are only running a single cluster on this host: 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:04.065 INFO:teuthology.orchestra.run.smithi043.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-27T06:58:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: ceph telemetry on 2024-05-27T06:58:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:04.066 INFO:teuthology.orchestra.run.smithi043.stdout:For more information see: 2024-05-27T06:58:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-27T06:58:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:04.066 INFO:teuthology.orchestra.run.smithi043.stdout:Bootstrap complete. 2024-05-27T06:58:04.174 INFO:tasks.cephadm:Fetching config... 2024-05-27T06:58:04.175 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:58:04.175 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-27T06:58:04.229 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-27T06:58:04.229 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:58:04.229 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-27T06:58:04.272 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-27T06:58:04.272 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:58:04.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-27T06:58:04.390 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-27T06:58:04.390 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:58:04.390 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-27T06:58:04.488 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-27T06:58:04.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6co39Flbni/XB+4t2Vw9ErJkROs61wosglo8SXB82bm1IQtovkwzMPRpgy1qyo/1bhus1HZDoLXALGUlXDjHp3WcyDaBq/yPPJBS7xr0r6UkvciT7N6DDA5h3flwQ+QhDyT+Na8wN2BOMrsQddk2go95N0z0fu7IA0xE5CUApwlFX3Z8jKNxW41t0YzMsvf2qNh3BDsC9yIecaFCgYYrbwu+owPT7YdeSE3072O2S3ff2DhAw5BVFoD55TIAkR02sByFGkh0cBcb3URagF89c3q9ot8Cw1iJ0IaXIZ9bbNBzDE4fczMfpZoMiy9lKCltCY7H916/CtmgTI7UP0D+ws6vTm7B0KMHI5RpdiZwZB+BgV8eVmzmxt6pbRG6ddMtekKPFf4O1rd6CcJPGjkm8VpeiKVy15xjn7csfnjqmyNDOq1ZNRDmEMEHJ6dom050EGPEHwDJiLebnLEJAeWxgHwaFps4i+SZD0lxFwYEpjL3Yy/Xs5w3/CB7u49EALIc= ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T06:58:04.617 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:04 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2284709126' entity='client.admin' 2024-05-27T06:58:04.622 INFO:teuthology.orchestra.run.smithi043.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6co39Flbni/XB+4t2Vw9ErJkROs61wosglo8SXB82bm1IQtovkwzMPRpgy1qyo/1bhus1HZDoLXALGUlXDjHp3WcyDaBq/yPPJBS7xr0r6UkvciT7N6DDA5h3flwQ+QhDyT+Na8wN2BOMrsQddk2go95N0z0fu7IA0xE5CUApwlFX3Z8jKNxW41t0YzMsvf2qNh3BDsC9yIecaFCgYYrbwu+owPT7YdeSE3072O2S3ff2DhAw5BVFoD55TIAkR02sByFGkh0cBcb3URagF89c3q9ot8Cw1iJ0IaXIZ9bbNBzDE4fczMfpZoMiy9lKCltCY7H916/CtmgTI7UP0D+ws6vTm7B0KMHI5RpdiZwZB+BgV8eVmzmxt6pbRG6ddMtekKPFf4O1rd6CcJPGjkm8VpeiKVy15xjn7csfnjqmyNDOq1ZNRDmEMEHJ6dom050EGPEHwDJiLebnLEJAeWxgHwaFps4i+SZD0lxFwYEpjL3Yy/Xs5w3/CB7u49EALIc= ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:58:04.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6co39Flbni/XB+4t2Vw9ErJkROs61wosglo8SXB82bm1IQtovkwzMPRpgy1qyo/1bhus1HZDoLXALGUlXDjHp3WcyDaBq/yPPJBS7xr0r6UkvciT7N6DDA5h3flwQ+QhDyT+Na8wN2BOMrsQddk2go95N0z0fu7IA0xE5CUApwlFX3Z8jKNxW41t0YzMsvf2qNh3BDsC9yIecaFCgYYrbwu+owPT7YdeSE3072O2S3ff2DhAw5BVFoD55TIAkR02sByFGkh0cBcb3URagF89c3q9ot8Cw1iJ0IaXIZ9bbNBzDE4fczMfpZoMiy9lKCltCY7H916/CtmgTI7UP0D+ws6vTm7B0KMHI5RpdiZwZB+BgV8eVmzmxt6pbRG6ddMtekKPFf4O1rd6CcJPGjkm8VpeiKVy15xjn7csfnjqmyNDOq1ZNRDmEMEHJ6dom050EGPEHwDJiLebnLEJAeWxgHwaFps4i+SZD0lxFwYEpjL3Yy/Xs5w3/CB7u49EALIc= ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T06:58:04.773 INFO:teuthology.orchestra.run.smithi132.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6co39Flbni/XB+4t2Vw9ErJkROs61wosglo8SXB82bm1IQtovkwzMPRpgy1qyo/1bhus1HZDoLXALGUlXDjHp3WcyDaBq/yPPJBS7xr0r6UkvciT7N6DDA5h3flwQ+QhDyT+Na8wN2BOMrsQddk2go95N0z0fu7IA0xE5CUApwlFX3Z8jKNxW41t0YzMsvf2qNh3BDsC9yIecaFCgYYrbwu+owPT7YdeSE3072O2S3ff2DhAw5BVFoD55TIAkR02sByFGkh0cBcb3URagF89c3q9ot8Cw1iJ0IaXIZ9bbNBzDE4fczMfpZoMiy9lKCltCY7H916/CtmgTI7UP0D+ws6vTm7B0KMHI5RpdiZwZB+BgV8eVmzmxt6pbRG6ddMtekKPFf4O1rd6CcJPGjkm8VpeiKVy15xjn7csfnjqmyNDOq1ZNRDmEMEHJ6dom050EGPEHwDJiLebnLEJAeWxgHwaFps4i+SZD0lxFwYEpjL3Yy/Xs5w3/CB7u49EALIc= ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:58:04.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6co39Flbni/XB+4t2Vw9ErJkROs61wosglo8SXB82bm1IQtovkwzMPRpgy1qyo/1bhus1HZDoLXALGUlXDjHp3WcyDaBq/yPPJBS7xr0r6UkvciT7N6DDA5h3flwQ+QhDyT+Na8wN2BOMrsQddk2go95N0z0fu7IA0xE5CUApwlFX3Z8jKNxW41t0YzMsvf2qNh3BDsC9yIecaFCgYYrbwu+owPT7YdeSE3072O2S3ff2DhAw5BVFoD55TIAkR02sByFGkh0cBcb3URagF89c3q9ot8Cw1iJ0IaXIZ9bbNBzDE4fczMfpZoMiy9lKCltCY7H916/CtmgTI7UP0D+ws6vTm7B0KMHI5RpdiZwZB+BgV8eVmzmxt6pbRG6ddMtekKPFf4O1rd6CcJPGjkm8VpeiKVy15xjn7csfnjqmyNDOq1ZNRDmEMEHJ6dom050EGPEHwDJiLebnLEJAeWxgHwaFps4i+SZD0lxFwYEpjL3Yy/Xs5w3/CB7u49EALIc= ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-27T06:58:04.951 INFO:teuthology.orchestra.run.smithi179.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6co39Flbni/XB+4t2Vw9ErJkROs61wosglo8SXB82bm1IQtovkwzMPRpgy1qyo/1bhus1HZDoLXALGUlXDjHp3WcyDaBq/yPPJBS7xr0r6UkvciT7N6DDA5h3flwQ+QhDyT+Na8wN2BOMrsQddk2go95N0z0fu7IA0xE5CUApwlFX3Z8jKNxW41t0YzMsvf2qNh3BDsC9yIecaFCgYYrbwu+owPT7YdeSE3072O2S3ff2DhAw5BVFoD55TIAkR02sByFGkh0cBcb3URagF89c3q9ot8Cw1iJ0IaXIZ9bbNBzDE4fczMfpZoMiy9lKCltCY7H916/CtmgTI7UP0D+ws6vTm7B0KMHI5RpdiZwZB+BgV8eVmzmxt6pbRG6ddMtekKPFf4O1rd6CcJPGjkm8VpeiKVy15xjn7csfnjqmyNDOq1ZNRDmEMEHJ6dom050EGPEHwDJiLebnLEJAeWxgHwaFps4i+SZD0lxFwYEpjL3Yy/Xs5w3/CB7u49EALIc= ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:58:04.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-27T06:58:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T06:58:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.902+0000 7fdf06400700 1 -- 172.21.15.43:0/3957800893 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf00103bd0 msgr2=0x7fdf00103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.902+0000 7fdf06400700 1 --2- 172.21.15.43:0/3957800893 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf00103bd0 0x7fdf00103fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdef000b0a0 tx=0x7fdef000b3b0 comp rx=0 tx=0).stop 2024-05-27T06:58:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.902+0000 7fdf06400700 1 -- 172.21.15.43:0/3957800893 shutdown_connections 2024-05-27T06:58:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.902+0000 7fdf06400700 1 --2- 172.21.15.43:0/3957800893 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf00103bd0 0x7fdf00103fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.902+0000 7fdf06400700 1 -- 172.21.15.43:0/3957800893 >> 172.21.15.43:0/3957800893 conn(0x7fdf000ff210 msgr2=0x7fdf001016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.902+0000 7fdf06400700 1 -- 172.21.15.43:0/3957800893 shutdown_connections 2024-05-27T06:58:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.902+0000 7fdf06400700 1 -- 172.21.15.43:0/3957800893 wait complete. 2024-05-27T06:58:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.904+0000 7fdf06400700 1 Processor -- start 2024-05-27T06:58:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.904+0000 7fdf06400700 1 -- start start 2024-05-27T06:58:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.905+0000 7fdf06400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf00103bd0 0x7fdf00077370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.905+0000 7fdf06400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdf00109a40 con 0x7fdf00103bd0 2024-05-27T06:58:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.905+0000 7fdeff400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf00103bd0 0x7fdf00077370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.905+0000 7fdeff400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf00103bd0 0x7fdf00077370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50932/0 (socket says 172.21.15.43:50932) 2024-05-27T06:58:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.905+0000 7fdeff400700 1 -- 172.21.15.43:0/1872157704 learned_addr learned my addr 172.21.15.43:0/1872157704 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.906+0000 7fdeff400700 1 -- 172.21.15.43:0/1872157704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf000778b0 con 0x7fdf00103bd0 2024-05-27T06:58:07.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.906+0000 7fdeff400700 1 --2- 172.21.15.43:0/1872157704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf00103bd0 0x7fdf00077370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdef0015040 tx=0x7fdef0003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:07.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.906+0000 7fdefcc00700 1 -- 172.21.15.43:0/1872157704 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdef0017040 con 0x7fdf00103bd0 2024-05-27T06:58:07.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.906+0000 7fdefcc00700 1 -- 172.21.15.43:0/1872157704 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdef0022070 con 0x7fdf00103bd0 2024-05-27T06:58:07.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.906+0000 7fdefcc00700 1 -- 172.21.15.43:0/1872157704 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdef001ddb0 con 0x7fdf00103bd0 2024-05-27T06:58:07.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.906+0000 7fdf06400700 1 -- 172.21.15.43:0/1872157704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf00075b00 con 0x7fdf00103bd0 2024-05-27T06:58:07.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.907+0000 7fdf06400700 1 -- 172.21.15.43:0/1872157704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf00075fe0 con 0x7fdf00103bd0 2024-05-27T06:58:07.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.908+0000 7fdefcc00700 1 -- 172.21.15.43:0/1872157704 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fdef000bc90 con 0x7fdf00103bd0 2024-05-27T06:58:07.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.908+0000 7fdf06400700 1 -- 172.21.15.43:0/1872157704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf0010a1c0 con 0x7fdf00103bd0 2024-05-27T06:58:07.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.909+0000 7fdefcc00700 1 --2- 172.21.15.43:0/1872157704 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdee403e5c0 0x7fdee4040ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:07.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.909+0000 7fdefcc00700 1 -- 172.21.15.43:0/1872157704 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdef000f050 con 0x7fdf00103bd0 2024-05-27T06:58:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.909+0000 7fdef7e00700 1 --2- 172.21.15.43:0/1872157704 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdee403e5c0 0x7fdee4040ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.910+0000 7fdef7e00700 1 --2- 172.21.15.43:0/1872157704 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdee403e5c0 0x7fdee4040ad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdee8006fd0 tx=0x7fdee8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:07.915+0000 7fdefcc00700 1 -- 172.21.15.43:0/1872157704 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdef0027460 con 0x7fdf00103bd0 2024-05-27T06:58:08.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.148+0000 7fdf06400700 1 -- 172.21.15.43:0/1872157704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fdf001923b0 con 0x7fdf00103bd0 2024-05-27T06:58:08.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.155+0000 7fdefcc00700 1 -- 172.21.15.43:0/1872157704 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fdef00263c0 con 0x7fdf00103bd0 2024-05-27T06:58:08.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.159+0000 7fdf06400700 1 -- 172.21.15.43:0/1872157704 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdee403e5c0 msgr2=0x7fdee4040ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:08.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.159+0000 7fdf06400700 1 --2- 172.21.15.43:0/1872157704 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdee403e5c0 0x7fdee4040ad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdee8006fd0 tx=0x7fdee8006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:08.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.159+0000 7fdf06400700 1 -- 172.21.15.43:0/1872157704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf00103bd0 msgr2=0x7fdf00077370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:08.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.159+0000 7fdf06400700 1 --2- 172.21.15.43:0/1872157704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf00103bd0 0x7fdf00077370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdef0015040 tx=0x7fdef0003ce0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.159+0000 7fdf06400700 1 -- 172.21.15.43:0/1872157704 shutdown_connections 2024-05-27T06:58:08.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.159+0000 7fdf06400700 1 --2- 172.21.15.43:0/1872157704 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdee403e5c0 0x7fdee4040ad0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.159+0000 7fdf06400700 1 --2- 172.21.15.43:0/1872157704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf00103bd0 0x7fdf00077370 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.159+0000 7fdf06400700 1 -- 172.21.15.43:0/1872157704 >> 172.21.15.43:0/1872157704 conn(0x7fdf000ff210 msgr2=0x7fdf000fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:08.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.159+0000 7fdf06400700 1 -- 172.21.15.43:0/1872157704 shutdown_connections 2024-05-27T06:58:08.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:08.159+0000 7fdf06400700 1 -- 172.21.15.43:0/1872157704 wait complete. 2024-05-27T06:58:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]: dispatch 2024-05-27T06:58:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:58:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:08 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1872157704' entity='client.admin' 2024-05-27T06:58:08.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:58:08.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:08.911 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-27T06:58:08.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-27T06:58:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T06:58:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.330+0000 7f52f7200700 1 -- 172.21.15.43:0/1034535741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52f8102900 msgr2=0x7f52f8102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.330+0000 7f52f7200700 1 --2- 172.21.15.43:0/1034535741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52f8102900 0x7f52f8102d10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009910 tx=0x7f52e8009c20 comp rx=0 tx=0).stop 2024-05-27T06:58:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.330+0000 7f52f7200700 1 -- 172.21.15.43:0/1034535741 shutdown_connections 2024-05-27T06:58:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.330+0000 7f52f7200700 1 --2- 172.21.15.43:0/1034535741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52f8102900 0x7f52f8102d10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.330+0000 7f52f7200700 1 -- 172.21.15.43:0/1034535741 >> 172.21.15.43:0/1034535741 conn(0x7f52f80fdf60 msgr2=0x7f52f81003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.330+0000 7f52f7200700 1 -- 172.21.15.43:0/1034535741 shutdown_connections 2024-05-27T06:58:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.330+0000 7f52f7200700 1 -- 172.21.15.43:0/1034535741 wait complete. 2024-05-27T06:58:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.331+0000 7f52f7200700 1 Processor -- start 2024-05-27T06:58:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.332+0000 7f52f7200700 1 -- start start 2024-05-27T06:58:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.332+0000 7f52f7200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52f8102900 0x7f52f81994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.332+0000 7f52f7200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f52f8108770 con 0x7f52f8102900 2024-05-27T06:58:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.332+0000 7f52f5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52f8102900 0x7f52f81994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.332+0000 7f52f5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52f8102900 0x7f52f81994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50958/0 (socket says 172.21.15.43:50958) 2024-05-27T06:58:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.332+0000 7f52f5e00700 1 -- 172.21.15.43:0/2744441346 learned_addr learned my addr 172.21.15.43:0/2744441346 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.333+0000 7f52f5e00700 1 -- 172.21.15.43:0/2744441346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f8199a00 con 0x7f52f8102900 2024-05-27T06:58:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.333+0000 7f52f5e00700 1 --2- 172.21.15.43:0/2744441346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52f8102900 0x7f52f81994c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f52e8005950 tx=0x7f52e8005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.333+0000 7f52eea00700 1 -- 172.21.15.43:0/2744441346 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52e8027070 con 0x7f52f8102900 2024-05-27T06:58:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.334+0000 7f52eea00700 1 -- 172.21.15.43:0/2744441346 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52e800bdf0 con 0x7f52f8102900 2024-05-27T06:58:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.334+0000 7f52f7200700 1 -- 172.21.15.43:0/2744441346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f8199c90 con 0x7f52f8102900 2024-05-27T06:58:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.334+0000 7f52eea00700 1 -- 172.21.15.43:0/2744441346 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52e800fdb0 con 0x7f52f8102900 2024-05-27T06:58:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.334+0000 7f52f7200700 1 -- 172.21.15.43:0/2744441346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f819c8a0 con 0x7f52f8102900 2024-05-27T06:58:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.335+0000 7f52f7200700 1 -- 172.21.15.43:0/2744441346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f8108ef0 con 0x7f52f8102900 2024-05-27T06:58:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.335+0000 7f52eea00700 1 -- 172.21.15.43:0/2744441346 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f52e802a070 con 0x7f52f8102900 2024-05-27T06:58:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.336+0000 7f52eea00700 1 --2- 172.21.15.43:0/2744441346 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f52dc03e570 0x7f52dc040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.336+0000 7f52eea00700 1 -- 172.21.15.43:0/2744441346 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f52e8053a30 con 0x7f52f8102900 2024-05-27T06:58:11.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.336+0000 7f52f5400700 1 --2- 172.21.15.43:0/2744441346 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f52dc03e570 0x7f52dc040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:11.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.337+0000 7f52f5400700 1 --2- 172.21.15.43:0/2744441346 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f52dc03e570 0x7f52dc040a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f52e4006fd0 tx=0x7f52e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:11.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.342+0000 7f52eea00700 1 -- 172.21.15.43:0/2744441346 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f52e8013070 con 0x7f52f8102900 2024-05-27T06:58:11.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.583+0000 7f52f7200700 1 -- 172.21.15.43:0/2744441346 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f52f802e6f0 con 0x7f52dc03e570 2024-05-27T06:58:11.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.592+0000 7f52eea00700 1 -- 172.21.15.43:0/2744441346 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f52f802e6f0 con 0x7f52dc03e570 2024-05-27T06:58:11.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.596+0000 7f52f7200700 1 -- 172.21.15.43:0/2744441346 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f52dc03e570 msgr2=0x7f52dc040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:11.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.596+0000 7f52f7200700 1 --2- 172.21.15.43:0/2744441346 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f52dc03e570 0x7f52dc040a80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f52e4006fd0 tx=0x7f52e4006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:11.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.596+0000 7f52f7200700 1 -- 172.21.15.43:0/2744441346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52f8102900 msgr2=0x7f52f81994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:11.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.596+0000 7f52f7200700 1 --2- 172.21.15.43:0/2744441346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52f8102900 0x7f52f81994c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f52e8005950 tx=0x7f52e8005810 comp rx=0 tx=0).stop 2024-05-27T06:58:11.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.596+0000 7f52f7200700 1 -- 172.21.15.43:0/2744441346 shutdown_connections 2024-05-27T06:58:11.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.596+0000 7f52f7200700 1 --2- 172.21.15.43:0/2744441346 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f52dc03e570 0x7f52dc040a80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:11.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.596+0000 7f52f7200700 1 --2- 172.21.15.43:0/2744441346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52f8102900 0x7f52f81994c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:11.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.596+0000 7f52f7200700 1 -- 172.21.15.43:0/2744441346 >> 172.21.15.43:0/2744441346 conn(0x7f52f80fdf60 msgr2=0x7f52f806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:11.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.597+0000 7f52f7200700 1 -- 172.21.15.43:0/2744441346 shutdown_connections 2024-05-27T06:58:11.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:11.597+0000 7f52f7200700 1 -- 172.21.15.43:0/2744441346 wait complete. 2024-05-27T06:58:12.337 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi132 2024-05-27T06:58:12.338 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:58:12.338 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.conf 2024-05-27T06:58:12.396 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:58:12.396 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:58:12.493 INFO:tasks.cephadm:Adding host smithi132 to orchestrator... 2024-05-27T06:58:12.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch host add smithi132 2024-05-27T06:58:12.913 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T06:58:12.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:12 smithi043 ceph-mon[31531]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:58:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:58:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:58:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:58:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:12 smithi043 ceph-mon[31531]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-27T06:58:13.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:13 smithi043 ceph-mon[31531]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:58:13.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:13 smithi043 ceph-mon[31531]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:14.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:14 smithi043 ceph-mon[31531]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:58:14.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:14 smithi043 ceph-mon[31531]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T06:58:14.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:14 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:14.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:14 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:14.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:14 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:14.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.994+0000 7f23c6600700 1 -- 172.21.15.43:0/3286295123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23c010a3c0 msgr2=0x7f23c010a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:14.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.994+0000 7f23c6600700 1 --2- 172.21.15.43:0/3286295123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23c010a3c0 0x7f23c010a7d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009910 tx=0x7f23b0009c20 comp rx=0 tx=0).stop 2024-05-27T06:58:14.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.994+0000 7f23c6600700 1 -- 172.21.15.43:0/3286295123 shutdown_connections 2024-05-27T06:58:14.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.994+0000 7f23c6600700 1 --2- 172.21.15.43:0/3286295123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23c010a3c0 0x7f23c010a7d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:14.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.994+0000 7f23c6600700 1 -- 172.21.15.43:0/3286295123 >> 172.21.15.43:0/3286295123 conn(0x7f23c00799e0 msgr2=0x7f23c01085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:14.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.994+0000 7f23c6600700 1 -- 172.21.15.43:0/3286295123 shutdown_connections 2024-05-27T06:58:14.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.994+0000 7f23c6600700 1 -- 172.21.15.43:0/3286295123 wait complete. 2024-05-27T06:58:14.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.996+0000 7f23c6600700 1 Processor -- start 2024-05-27T06:58:14.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.996+0000 7f23c6600700 1 -- start start 2024-05-27T06:58:14.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.997+0000 7f23c6600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23c010a3c0 0x7f23c019fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:14.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.997+0000 7f23c6600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f23c0114560 con 0x7f23c010a3c0 2024-05-27T06:58:14.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.997+0000 7f23bf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23c010a3c0 0x7f23c019fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:14.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.997+0000 7f23bf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23c010a3c0 0x7f23c019fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44464/0 (socket says 172.21.15.43:44464) 2024-05-27T06:58:14.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.997+0000 7f23bf400700 1 -- 172.21.15.43:0/376061945 learned_addr learned my addr 172.21.15.43:0/376061945 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:14.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.998+0000 7f23bf400700 1 -- 172.21.15.43:0/376061945 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c01a03c0 con 0x7f23c010a3c0 2024-05-27T06:58:14.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.998+0000 7f23bf400700 1 --2- 172.21.15.43:0/376061945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23c010a3c0 0x7f23c019fe80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f23b0006010 tx=0x7f23b0005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:14.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.998+0000 7f23b7e00700 1 -- 172.21.15.43:0/376061945 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f23b0026070 con 0x7f23c010a3c0 2024-05-27T06:58:14.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.999+0000 7f23b7e00700 1 -- 172.21.15.43:0/376061945 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23b000b6f0 con 0x7f23c010a3c0 2024-05-27T06:58:14.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.999+0000 7f23c6600700 1 -- 172.21.15.43:0/376061945 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23c01a06b0 con 0x7f23c010a3c0 2024-05-27T06:58:14.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.999+0000 7f23b7e00700 1 -- 172.21.15.43:0/376061945 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f23b0003810 con 0x7f23c010a3c0 2024-05-27T06:58:14.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:14.999+0000 7f23c6600700 1 -- 172.21.15.43:0/376061945 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23c01a4660 con 0x7f23c010a3c0 2024-05-27T06:58:14.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:15.000+0000 7f23c6600700 1 -- 172.21.15.43:0/376061945 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23c010b870 con 0x7f23c010a3c0 2024-05-27T06:58:14.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:15.000+0000 7f23b7e00700 1 -- 172.21.15.43:0/376061945 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f23b0029070 con 0x7f23c010a3c0 2024-05-27T06:58:14.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:15.001+0000 7f23b7e00700 1 --2- 172.21.15.43:0/376061945 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f23a403e570 0x7f23a4040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:14.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:15.001+0000 7f23b7e00700 1 -- 172.21.15.43:0/376061945 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f23b001e070 con 0x7f23c010a3c0 2024-05-27T06:58:14.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:15.001+0000 7f23bea00700 1 --2- 172.21.15.43:0/376061945 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f23a403e570 0x7f23a4040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:14.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:15.002+0000 7f23bea00700 1 --2- 172.21.15.43:0/376061945 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f23a403e570 0x7f23a4040a80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f23ac006fd0 tx=0x7f23ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:15.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:15.009+0000 7f23b7e00700 1 -- 172.21.15.43:0/376061945 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f23b0013070 con 0x7f23c010a3c0 2024-05-27T06:58:15.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:15.251+0000 7f23c6600700 1 -- 172.21.15.43:0/376061945 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}) -- 0x7f23c01a1190 con 0x7f23a403e570 2024-05-27T06:58:15.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:15 smithi043 ceph-mon[31531]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:16.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:16 smithi043 ceph-mon[31531]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:58:17.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:17 smithi043 ceph-mon[31531]: Deploying cephadm binary to smithi132 2024-05-27T06:58:17.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:17 smithi043 ceph-mon[31531]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:19 smithi043 ceph-mon[31531]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:21.433 INFO:teuthology.orchestra.run.smithi043.stdout:Added host 'smithi132' with addr '172.21.15.132' 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.436+0000 7f23b7e00700 1 -- 172.21.15.43:0/376061945 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f23c01a1190 con 0x7f23a403e570 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.440+0000 7f23c6600700 1 -- 172.21.15.43:0/376061945 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f23a403e570 msgr2=0x7f23a4040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.440+0000 7f23c6600700 1 --2- 172.21.15.43:0/376061945 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f23a403e570 0x7f23a4040a80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f23ac006fd0 tx=0x7f23ac006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.440+0000 7f23c6600700 1 -- 172.21.15.43:0/376061945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23c010a3c0 msgr2=0x7f23c019fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.440+0000 7f23c6600700 1 --2- 172.21.15.43:0/376061945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23c010a3c0 0x7f23c019fe80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f23b0006010 tx=0x7f23b0005710 comp rx=0 tx=0).stop 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.440+0000 7f23c6600700 1 -- 172.21.15.43:0/376061945 shutdown_connections 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.440+0000 7f23c6600700 1 --2- 172.21.15.43:0/376061945 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f23a403e570 0x7f23a4040a80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.440+0000 7f23c6600700 1 --2- 172.21.15.43:0/376061945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23c010a3c0 0x7f23c019fe80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.440+0000 7f23c6600700 1 -- 172.21.15.43:0/376061945 >> 172.21.15.43:0/376061945 conn(0x7f23c00799e0 msgr2=0x7f23c010bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.440+0000 7f23c6600700 1 -- 172.21.15.43:0/376061945 shutdown_connections 2024-05-27T06:58:21.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:21.440+0000 7f23c6600700 1 -- 172.21.15.43:0/376061945 wait complete. 2024-05-27T06:58:21.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:21 smithi043 ceph-mon[31531]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:21.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:21.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:58:22.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-27T06:58:22.647 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T06:58:22.917 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:22 smithi043 ceph-mon[31531]: Added host smithi132 2024-05-27T06:58:22.918 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:22 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:23.781 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:23 smithi043 ceph-mon[31531]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:23.781 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:24.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.640+0000 7fcc7d000700 1 -- 172.21.15.43:0/1526256536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc78108a80 msgr2=0x7fcc78108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.640+0000 7fcc7d000700 1 --2- 172.21.15.43:0/1526256536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc78108a80 0x7fcc78108e90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009910 tx=0x7fcc68009c20 comp rx=0 tx=0).stop 2024-05-27T06:58:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.640+0000 7fcc7d000700 1 -- 172.21.15.43:0/1526256536 shutdown_connections 2024-05-27T06:58:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.640+0000 7fcc7d000700 1 --2- 172.21.15.43:0/1526256536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc78108a80 0x7fcc78108e90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.640+0000 7fcc7d000700 1 -- 172.21.15.43:0/1526256536 >> 172.21.15.43:0/1526256536 conn(0x7fcc780785a0 msgr2=0x7fcc78078a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.640+0000 7fcc7d000700 1 -- 172.21.15.43:0/1526256536 shutdown_connections 2024-05-27T06:58:24.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.641+0000 7fcc7d000700 1 -- 172.21.15.43:0/1526256536 wait complete. 2024-05-27T06:58:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.642+0000 7fcc7d000700 1 Processor -- start 2024-05-27T06:58:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.642+0000 7fcc7d000700 1 -- start start 2024-05-27T06:58:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.643+0000 7fcc7d000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc78108a80 0x7fcc781a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:24.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.643+0000 7fcc7d000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7810ce80 con 0x7fcc78108a80 2024-05-27T06:58:24.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.643+0000 7fcc75e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc78108a80 0x7fcc781a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:24.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.643+0000 7fcc75e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc78108a80 0x7fcc781a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37824/0 (socket says 172.21.15.43:37824) 2024-05-27T06:58:24.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.643+0000 7fcc75e00700 1 -- 172.21.15.43:0/2893343539 learned_addr learned my addr 172.21.15.43:0/2893343539 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:24.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.643+0000 7fcc75e00700 1 -- 172.21.15.43:0/2893343539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc781a1bc0 con 0x7fcc78108a80 2024-05-27T06:58:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.644+0000 7fcc75e00700 1 --2- 172.21.15.43:0/2893343539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc78108a80 0x7fcc781a1680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcc680092e0 tx=0x7fcc68005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.644+0000 7fcc6f400700 1 -- 172.21.15.43:0/2893343539 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc68026070 con 0x7fcc78108a80 2024-05-27T06:58:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.644+0000 7fcc6f400700 1 -- 172.21.15.43:0/2893343539 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc6800b6f0 con 0x7fcc78108a80 2024-05-27T06:58:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.644+0000 7fcc7d000700 1 -- 172.21.15.43:0/2893343539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc781a1eb0 con 0x7fcc78108a80 2024-05-27T06:58:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.644+0000 7fcc6f400700 1 -- 172.21.15.43:0/2893343539 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc680036c0 con 0x7fcc78108a80 2024-05-27T06:58:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.644+0000 7fcc7d000700 1 -- 172.21.15.43:0/2893343539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7819b8a0 con 0x7fcc78108a80 2024-05-27T06:58:24.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.646+0000 7fcc6f400700 1 -- 172.21.15.43:0/2893343539 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fcc68029070 con 0x7fcc78108a80 2024-05-27T06:58:24.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.646+0000 7fcc7d000700 1 -- 172.21.15.43:0/2893343539 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc78104190 con 0x7fcc78108a80 2024-05-27T06:58:24.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.646+0000 7fcc6f400700 1 --2- 172.21.15.43:0/2893343539 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcc5c03e570 0x7fcc5c040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:24.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.646+0000 7fcc6f400700 1 -- 172.21.15.43:0/2893343539 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcc6801e070 con 0x7fcc78108a80 2024-05-27T06:58:24.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.647+0000 7fcc6d400700 1 --2- 172.21.15.43:0/2893343539 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcc5c03e570 0x7fcc5c040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:24.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.647+0000 7fcc6d400700 1 --2- 172.21.15.43:0/2893343539 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcc5c03e570 0x7fcc5c040a80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcc60006fd0 tx=0x7fcc60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:24.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.653+0000 7fcc6f400700 1 -- 172.21.15.43:0/2893343539 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcc68013070 con 0x7fcc78108a80 2024-05-27T06:58:24.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.896+0000 7fcc7d000700 1 -- 172.21.15.43:0/2893343539 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc7819cdd0 con 0x7fcc5c03e570 2024-05-27T06:58:24.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.898+0000 7fcc6f400700 1 -- 172.21.15.43:0/2893343539 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fcc7819cdd0 con 0x7fcc5c03e570 2024-05-27T06:58:24.891 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:24.891 INFO:teuthology.orchestra.run.smithi043.stdout:[{"addr": "172.21.15.43", "hostname": "smithi043", "labels": [], "status": ""}, {"addr": "172.21.15.132", "hostname": "smithi132", "labels": [], "status": ""}] 2024-05-27T06:58:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.901+0000 7fcc7d000700 1 -- 172.21.15.43:0/2893343539 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcc5c03e570 msgr2=0x7fcc5c040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.901+0000 7fcc7d000700 1 --2- 172.21.15.43:0/2893343539 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcc5c03e570 0x7fcc5c040a80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcc60006fd0 tx=0x7fcc60006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.901+0000 7fcc7d000700 1 -- 172.21.15.43:0/2893343539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc78108a80 msgr2=0x7fcc781a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.901+0000 7fcc7d000700 1 --2- 172.21.15.43:0/2893343539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc78108a80 0x7fcc781a1680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcc680092e0 tx=0x7fcc68005710 comp rx=0 tx=0).stop 2024-05-27T06:58:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.902+0000 7fcc7d000700 1 -- 172.21.15.43:0/2893343539 shutdown_connections 2024-05-27T06:58:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.902+0000 7fcc7d000700 1 --2- 172.21.15.43:0/2893343539 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcc5c03e570 0x7fcc5c040a80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.902+0000 7fcc7d000700 1 --2- 172.21.15.43:0/2893343539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc78108a80 0x7fcc781a1680 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.902+0000 7fcc7d000700 1 -- 172.21.15.43:0/2893343539 >> 172.21.15.43:0/2893343539 conn(0x7fcc780785a0 msgr2=0x7fcc78104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:24.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.902+0000 7fcc7d000700 1 -- 172.21.15.43:0/2893343539 shutdown_connections 2024-05-27T06:58:24.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:24.902+0000 7fcc7d000700 1 -- 172.21.15.43:0/2893343539 wait complete. 2024-05-27T06:58:25.565 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi179 2024-05-27T06:58:25.565 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:58:25.565 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/etc/ceph/ceph.conf 2024-05-27T06:58:25.623 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:58:25.624 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:58:25.722 INFO:tasks.cephadm:Adding host smithi179 to orchestrator... 2024-05-27T06:58:25.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch host add smithi179 2024-05-27T06:58:25.805 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:25 smithi043 ceph-mon[31531]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:25.805 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:25 smithi043 ceph-mon[31531]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T06:58:25.806 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:25 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T06:58:27.866 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:27 smithi043 ceph-mon[31531]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:28.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.225+0000 7ff84f200700 1 -- 172.21.15.43:0/1473579484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff850105e20 msgr2=0x7ff850106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:28.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.225+0000 7ff84f200700 1 --2- 172.21.15.43:0/1473579484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff850105e20 0x7ff850106230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff840009910 tx=0x7ff840009c20 comp rx=0 tx=0).stop 2024-05-27T06:58:28.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.225+0000 7ff84f200700 1 -- 172.21.15.43:0/1473579484 shutdown_connections 2024-05-27T06:58:28.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.225+0000 7ff84f200700 1 --2- 172.21.15.43:0/1473579484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff850105e20 0x7ff850106230 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:28.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.225+0000 7ff84f200700 1 -- 172.21.15.43:0/1473579484 >> 172.21.15.43:0/1473579484 conn(0x7ff850101670 msgr2=0x7ff850103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:28.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.225+0000 7ff84f200700 1 -- 172.21.15.43:0/1473579484 shutdown_connections 2024-05-27T06:58:28.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.225+0000 7ff84f200700 1 -- 172.21.15.43:0/1473579484 wait complete. 2024-05-27T06:58:28.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.227+0000 7ff84f200700 1 Processor -- start 2024-05-27T06:58:28.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.227+0000 7ff84f200700 1 -- start start 2024-05-27T06:58:28.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.227+0000 7ff84f200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff850105e20 0x7ff8501a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:28.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.227+0000 7ff84f200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff85010ffc0 con 0x7ff850105e20 2024-05-27T06:58:28.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.227+0000 7ff84de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff850105e20 0x7ff8501a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:28.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.228+0000 7ff84de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff850105e20 0x7ff8501a1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37854/0 (socket says 172.21.15.43:37854) 2024-05-27T06:58:28.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.228+0000 7ff84de00700 1 -- 172.21.15.43:0/2693328636 learned_addr learned my addr 172.21.15.43:0/2693328636 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:28.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.228+0000 7ff84de00700 1 -- 172.21.15.43:0/2693328636 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8501a1ab0 con 0x7ff850105e20 2024-05-27T06:58:28.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.228+0000 7ff84de00700 1 --2- 172.21.15.43:0/2693328636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff850105e20 0x7ff8501a1570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff840005660 tx=0x7ff840005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:28.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.229+0000 7ff846a00700 1 -- 172.21.15.43:0/2693328636 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff840026070 con 0x7ff850105e20 2024-05-27T06:58:28.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.229+0000 7ff846a00700 1 -- 172.21.15.43:0/2693328636 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff84000b6f0 con 0x7ff850105e20 2024-05-27T06:58:28.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.229+0000 7ff84f200700 1 -- 172.21.15.43:0/2693328636 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8501a1da0 con 0x7ff850105e20 2024-05-27T06:58:28.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.229+0000 7ff846a00700 1 -- 172.21.15.43:0/2693328636 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8400037b0 con 0x7ff850105e20 2024-05-27T06:58:28.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.229+0000 7ff84f200700 1 -- 172.21.15.43:0/2693328636 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff85019b8d0 con 0x7ff850105e20 2024-05-27T06:58:28.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.230+0000 7ff84f200700 1 -- 172.21.15.43:0/2693328636 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8501072d0 con 0x7ff850105e20 2024-05-27T06:58:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.230+0000 7ff846a00700 1 -- 172.21.15.43:0/2693328636 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7ff840029070 con 0x7ff850105e20 2024-05-27T06:58:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.231+0000 7ff846a00700 1 --2- 172.21.15.43:0/2693328636 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff83403e510 0x7ff834040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:28.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.231+0000 7ff846a00700 1 -- 172.21.15.43:0/2693328636 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff84001e070 con 0x7ff850105e20 2024-05-27T06:58:28.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.231+0000 7ff84d400700 1 --2- 172.21.15.43:0/2693328636 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff83403e510 0x7ff834040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:28.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.232+0000 7ff84d400700 1 --2- 172.21.15.43:0/2693328636 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff83403e510 0x7ff834040a20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff83c009940 tx=0x7ff83c006e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:28.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.237+0000 7ff846a00700 1 -- 172.21.15.43:0/2693328636 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff840013070 con 0x7ff850105e20 2024-05-27T06:58:28.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:28.485+0000 7ff84f200700 1 -- 172.21.15.43:0/2693328636 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}) -- 0x7ff85019cc10 con 0x7ff83403e510 2024-05-27T06:58:29.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:29 smithi043 ceph-mon[31531]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:58:29.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:29 smithi043 ceph-mon[31531]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:30.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:30 smithi043 ceph-mon[31531]: Deploying cephadm binary to smithi179 2024-05-27T06:58:30.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:30.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:30.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:30.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:30.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-05-27T06:58:30.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:58:30.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:58:31.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:31 smithi043 ceph-mon[31531]: Updating smithi132:/etc/ceph/ceph.conf 2024-05-27T06:58:31.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:31 smithi043 ceph-mon[31531]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:33.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:33 smithi043 ceph-mon[31531]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:34.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.807+0000 7ff846a00700 1 -- 172.21.15.43:0/2693328636 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7ff85019cc10 con 0x7ff83403e510 2024-05-27T06:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout:Added host 'smithi179' with addr '172.21.15.179' 2024-05-27T06:58:34.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.812+0000 7ff84f200700 1 -- 172.21.15.43:0/2693328636 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff83403e510 msgr2=0x7ff834040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:34.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.812+0000 7ff84f200700 1 --2- 172.21.15.43:0/2693328636 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff83403e510 0x7ff834040a20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff83c009940 tx=0x7ff83c006e30 comp rx=0 tx=0).stop 2024-05-27T06:58:34.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.812+0000 7ff84f200700 1 -- 172.21.15.43:0/2693328636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff850105e20 msgr2=0x7ff8501a1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:34.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.812+0000 7ff84f200700 1 --2- 172.21.15.43:0/2693328636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff850105e20 0x7ff8501a1570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff840005660 tx=0x7ff840005740 comp rx=0 tx=0).stop 2024-05-27T06:58:34.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.813+0000 7ff84f200700 1 -- 172.21.15.43:0/2693328636 shutdown_connections 2024-05-27T06:58:34.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.813+0000 7ff84f200700 1 --2- 172.21.15.43:0/2693328636 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff83403e510 0x7ff834040a20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:34.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.813+0000 7ff84f200700 1 --2- 172.21.15.43:0/2693328636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff850105e20 0x7ff8501a1570 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:34.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.813+0000 7ff84f200700 1 -- 172.21.15.43:0/2693328636 >> 172.21.15.43:0/2693328636 conn(0x7ff850101670 msgr2=0x7ff8501077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:34.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.813+0000 7ff84f200700 1 -- 172.21.15.43:0/2693328636 shutdown_connections 2024-05-27T06:58:34.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:34.813+0000 7ff84f200700 1 -- 172.21.15.43:0/2693328636 wait complete. 2024-05-27T06:58:34.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:34 smithi043 ceph-mon[31531]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:58:35.461 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-27T06:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T06:58:36.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:35 smithi043 ceph-mon[31531]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:58:36.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:35 smithi043 ceph-mon[31531]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:36.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:36.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:35 smithi043 ceph-mon[31531]: Added host smithi179 2024-05-27T06:58:36.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:36 smithi043 ceph-mon[31531]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T06:58:36.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:36 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:36.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:36 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:36.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:36 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:36.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:36 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:58:37.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:37 smithi043 ceph-mon[31531]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:37.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:37 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:38.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.155+0000 7f4d19000700 1 -- 172.21.15.43:0/1761980372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d1410a220 msgr2=0x7f4d1410a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:38.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.155+0000 7f4d19000700 1 --2- 172.21.15.43:0/1761980372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d1410a220 0x7f4d1410a630 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4d04009910 tx=0x7f4d04009c20 comp rx=0 tx=0).stop 2024-05-27T06:58:38.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.155+0000 7f4d19000700 1 -- 172.21.15.43:0/1761980372 shutdown_connections 2024-05-27T06:58:38.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.155+0000 7f4d19000700 1 --2- 172.21.15.43:0/1761980372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d1410a220 0x7f4d1410a630 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:38.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.155+0000 7f4d19000700 1 -- 172.21.15.43:0/1761980372 >> 172.21.15.43:0/1761980372 conn(0x7f4d14079720 msgr2=0x7f4d141085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:38.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.155+0000 7f4d19000700 1 -- 172.21.15.43:0/1761980372 shutdown_connections 2024-05-27T06:58:38.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.155+0000 7f4d19000700 1 -- 172.21.15.43:0/1761980372 wait complete. 2024-05-27T06:58:38.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.157+0000 7f4d19000700 1 Processor -- start 2024-05-27T06:58:38.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.157+0000 7f4d19000700 1 -- start start 2024-05-27T06:58:38.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.157+0000 7f4d19000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d1410a220 0x7f4d141a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:38.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.157+0000 7f4d19000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4d141143c0 con 0x7f4d1410a220 2024-05-27T06:58:38.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.158+0000 7f4d13400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d1410a220 0x7f4d141a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:38.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.158+0000 7f4d13400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d1410a220 0x7f4d141a58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52082/0 (socket says 172.21.15.43:52082) 2024-05-27T06:58:38.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.158+0000 7f4d13400700 1 -- 172.21.15.43:0/3048500219 learned_addr learned my addr 172.21.15.43:0/3048500219 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:38.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.158+0000 7f4d13400700 1 -- 172.21.15.43:0/3048500219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d141a5de0 con 0x7f4d1410a220 2024-05-27T06:58:38.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.159+0000 7f4d13400700 1 --2- 172.21.15.43:0/3048500219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d1410a220 0x7f4d141a58a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d04006010 tx=0x7f4d04005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.159+0000 7f4d0be00700 1 -- 172.21.15.43:0/3048500219 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d04026070 con 0x7f4d1410a220 2024-05-27T06:58:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.159+0000 7f4d0be00700 1 -- 172.21.15.43:0/3048500219 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4d0400b6f0 con 0x7f4d1410a220 2024-05-27T06:58:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.159+0000 7f4d19000700 1 -- 172.21.15.43:0/3048500219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d141a60d0 con 0x7f4d1410a220 2024-05-27T06:58:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.159+0000 7f4d0be00700 1 -- 172.21.15.43:0/3048500219 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d04003810 con 0x7f4d1410a220 2024-05-27T06:58:38.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.159+0000 7f4d19000700 1 -- 172.21.15.43:0/3048500219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d1419fb10 con 0x7f4d1410a220 2024-05-27T06:58:38.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.160+0000 7f4d19000700 1 -- 172.21.15.43:0/3048500219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d1410b6d0 con 0x7f4d1410a220 2024-05-27T06:58:38.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.161+0000 7f4d0be00700 1 -- 172.21.15.43:0/3048500219 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f4d04029070 con 0x7f4d1410a220 2024-05-27T06:58:38.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.161+0000 7f4d0be00700 1 --2- 172.21.15.43:0/3048500219 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4cf803e510 0x7f4cf8040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:38.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.161+0000 7f4d0be00700 1 -- 172.21.15.43:0/3048500219 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4d0401e070 con 0x7f4d1410a220 2024-05-27T06:58:38.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.162+0000 7f4d12a00700 1 --2- 172.21.15.43:0/3048500219 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4cf803e510 0x7f4cf8040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:38.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.162+0000 7f4d12a00700 1 --2- 172.21.15.43:0/3048500219 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4cf803e510 0x7f4cf8040a20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4d00006fd0 tx=0x7f4d00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:38.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.167+0000 7f4d0be00700 1 -- 172.21.15.43:0/3048500219 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4d04013070 con 0x7f4d1410a220 2024-05-27T06:58:38.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.411+0000 7f4d19000700 1 -- 172.21.15.43:0/3048500219 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d141a1060 con 0x7f4cf803e510 2024-05-27T06:58:38.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.414+0000 7f4d0be00700 1 -- 172.21.15.43:0/3048500219 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f4d141a1060 con 0x7f4cf803e510 2024-05-27T06:58:38.406 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T06:58:38.406 INFO:teuthology.orchestra.run.smithi043.stdout:[{"addr": "172.21.15.43", "hostname": "smithi043", "labels": [], "status": ""}, {"addr": "172.21.15.132", "hostname": "smithi132", "labels": [], "status": ""}, {"addr": "172.21.15.179", "hostname": "smithi179", "labels": [], "status": ""}] 2024-05-27T06:58:38.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.418+0000 7f4d19000700 1 -- 172.21.15.43:0/3048500219 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4cf803e510 msgr2=0x7f4cf8040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:38.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.418+0000 7f4d19000700 1 --2- 172.21.15.43:0/3048500219 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4cf803e510 0x7f4cf8040a20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4d00006fd0 tx=0x7f4d00006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:38.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.418+0000 7f4d19000700 1 -- 172.21.15.43:0/3048500219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d1410a220 msgr2=0x7f4d141a58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:38.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.418+0000 7f4d19000700 1 --2- 172.21.15.43:0/3048500219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d1410a220 0x7f4d141a58a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d04006010 tx=0x7f4d04005710 comp rx=0 tx=0).stop 2024-05-27T06:58:38.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.418+0000 7f4d19000700 1 -- 172.21.15.43:0/3048500219 shutdown_connections 2024-05-27T06:58:38.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.418+0000 7f4d19000700 1 --2- 172.21.15.43:0/3048500219 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4cf803e510 0x7f4cf8040a20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:38.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.418+0000 7f4d19000700 1 --2- 172.21.15.43:0/3048500219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4d1410a220 0x7f4d141a58a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:38.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.418+0000 7f4d19000700 1 -- 172.21.15.43:0/3048500219 >> 172.21.15.43:0/3048500219 conn(0x7f4d14079720 msgr2=0x7f4d1410bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:38.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.418+0000 7f4d19000700 1 -- 172.21.15.43:0/3048500219 shutdown_connections 2024-05-27T06:58:38.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:38.418+0000 7f4d19000700 1 -- 172.21.15.43:0/3048500219 wait complete. 2024-05-27T06:58:39.157 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-27T06:58:39.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-05-27T06:58:39.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:39.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:39 smithi043 ceph-mon[31531]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T06:58:39.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:39 smithi043 ceph-mon[31531]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:39.613 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T06:58:41.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.591+0000 7f251ca00700 1 -- 172.21.15.43:0/8526042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2518103de0 msgr2=0x7f25181041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:41.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.591+0000 7f251ca00700 1 --2- 172.21.15.43:0/8526042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2518103de0 0x7f25181041f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2508009910 tx=0x7f2508009c20 comp rx=0 tx=0).stop 2024-05-27T06:58:41.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.591+0000 7f251ca00700 1 -- 172.21.15.43:0/8526042 shutdown_connections 2024-05-27T06:58:41.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.591+0000 7f251ca00700 1 --2- 172.21.15.43:0/8526042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2518103de0 0x7f25181041f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.591+0000 7f251ca00700 1 -- 172.21.15.43:0/8526042 >> 172.21.15.43:0/8526042 conn(0x7f25180ff690 msgr2=0x7f2518101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:41.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.592+0000 7f251ca00700 1 -- 172.21.15.43:0/8526042 shutdown_connections 2024-05-27T06:58:41.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.592+0000 7f251ca00700 1 -- 172.21.15.43:0/8526042 wait complete. 2024-05-27T06:58:41.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:41.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:41 smithi043 ceph-mon[31531]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:41.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.593+0000 7f251ca00700 1 Processor -- start 2024-05-27T06:58:41.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.593+0000 7f251ca00700 1 -- start start 2024-05-27T06:58:41.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.593+0000 7f251ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2518103de0 0x7f25181a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:41.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.593+0000 7f251ca00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f251810df80 con 0x7f2518103de0 2024-05-27T06:58:41.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.594+0000 7f2515e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2518103de0 0x7f25181a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:41.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.594+0000 7f2515e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2518103de0 0x7f25181a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52114/0 (socket says 172.21.15.43:52114) 2024-05-27T06:58:41.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.594+0000 7f2515e00700 1 -- 172.21.15.43:0/4057861902 learned_addr learned my addr 172.21.15.43:0/4057861902 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:58:41.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.594+0000 7f2515e00700 1 -- 172.21.15.43:0/4057861902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25181a1bc0 con 0x7f2518103de0 2024-05-27T06:58:41.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.595+0000 7f2515e00700 1 --2- 172.21.15.43:0/4057861902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2518103de0 0x7f25181a1680 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2508005660 tx=0x7f2508005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:41.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.595+0000 7f250ea00700 1 -- 172.21.15.43:0/4057861902 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2508026070 con 0x7f2518103de0 2024-05-27T06:58:41.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.595+0000 7f250ea00700 1 -- 172.21.15.43:0/4057861902 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f250800b6f0 con 0x7f2518103de0 2024-05-27T06:58:41.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.595+0000 7f250ea00700 1 -- 172.21.15.43:0/4057861902 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25080037b0 con 0x7f2518103de0 2024-05-27T06:58:41.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.595+0000 7f251ca00700 1 -- 172.21.15.43:0/4057861902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25181a1eb0 con 0x7f2518103de0 2024-05-27T06:58:41.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.595+0000 7f251ca00700 1 -- 172.21.15.43:0/4057861902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f251819b8a0 con 0x7f2518103de0 2024-05-27T06:58:41.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.597+0000 7f250ea00700 1 -- 172.21.15.43:0/4057861902 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f2508029070 con 0x7f2518103de0 2024-05-27T06:58:41.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.597+0000 7f251ca00700 1 -- 172.21.15.43:0/4057861902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2518105290 con 0x7f2518103de0 2024-05-27T06:58:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.597+0000 7f250ea00700 1 --2- 172.21.15.43:0/4057861902 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f24fc03e570 0x7f24fc040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.597+0000 7f250ea00700 1 -- 172.21.15.43:0/4057861902 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f250801e070 con 0x7f2518103de0 2024-05-27T06:58:41.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.597+0000 7f250fe00700 1 --2- 172.21.15.43:0/4057861902 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f24fc03e570 0x7f24fc040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:41.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.598+0000 7f250fe00700 1 --2- 172.21.15.43:0/4057861902 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f24fc03e570 0x7f24fc040a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2500006fd0 tx=0x7f2500006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:41.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.603+0000 7f250ea00700 1 -- 172.21.15.43:0/4057861902 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2508013070 con 0x7f2518103de0 2024-05-27T06:58:41.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:41.854+0000 7f251ca00700 1 -- 172.21.15.43:0/4057861902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f251819cdd0 con 0x7f2518103de0 2024-05-27T06:58:42.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.318+0000 7f250ea00700 1 -- 172.21.15.43:0/4057861902 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f250800e3e0 con 0x7f2518103de0 2024-05-27T06:58:42.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.322+0000 7f251ca00700 1 -- 172.21.15.43:0/4057861902 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f24fc03e570 msgr2=0x7f24fc040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.322+0000 7f251ca00700 1 --2- 172.21.15.43:0/4057861902 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f24fc03e570 0x7f24fc040a80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2500006fd0 tx=0x7f2500006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.322+0000 7f251ca00700 1 -- 172.21.15.43:0/4057861902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2518103de0 msgr2=0x7f25181a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.322+0000 7f251ca00700 1 --2- 172.21.15.43:0/4057861902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2518103de0 0x7f25181a1680 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2508005660 tx=0x7f2508005740 comp rx=0 tx=0).stop 2024-05-27T06:58:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.323+0000 7f251ca00700 1 -- 172.21.15.43:0/4057861902 shutdown_connections 2024-05-27T06:58:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.323+0000 7f251ca00700 1 --2- 172.21.15.43:0/4057861902 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f24fc03e570 0x7f24fc040a80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.323+0000 7f251ca00700 1 --2- 172.21.15.43:0/4057861902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2518103de0 0x7f25181a1680 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.323+0000 7f251ca00700 1 -- 172.21.15.43:0/4057861902 >> 172.21.15.43:0/4057861902 conn(0x7f25180ff690 msgr2=0x7f2518109f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.323+0000 7f251ca00700 1 -- 172.21.15.43:0/4057861902 shutdown_connections 2024-05-27T06:58:42.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:58:42.323+0000 7f251ca00700 1 -- 172.21.15.43:0/4057861902 wait complete. 2024-05-27T06:58:42.316 INFO:teuthology.orchestra.run.smithi043.stderr:adjusted tunables profile to default 2024-05-27T06:58:42.678 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:42 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4057861902' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T06:58:43.034 INFO:tasks.cephadm:Adding mon.a on smithi043 2024-05-27T06:58:43.034 INFO:tasks.cephadm:Adding mon.b on smithi132 2024-05-27T06:58:43.034 INFO:tasks.cephadm:Adding mon.c on smithi179 2024-05-27T06:58:43.035 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch apply mon '3;smithi043:172.21.15.43=a;smithi132:172.21.15.132=b;smithi179:172.21.15.179=c' 2024-05-27T06:58:43.435 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-27T06:58:43.557 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /etc/ceph/ceph.conf 2024-05-27T06:58:43.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:43 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4057861902' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T06:58:43.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:43 smithi043 ceph-mon[31531]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T06:58:43.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:43 smithi043 ceph-mon[31531]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:45.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:45 smithi043 ceph-mon[31531]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:46.281 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.288+0000 7fedef600700 1 -- 172.21.15.179:0/2850622812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fede8077770 msgr2=0x7fede8075ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:46.281 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.288+0000 7fedef600700 1 --2- 172.21.15.179:0/2850622812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fede8077770 0x7fede8075ba0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feddc009910 tx=0x7feddc009c20 comp rx=0 tx=0).stop 2024-05-27T06:58:46.281 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.288+0000 7fedef600700 1 -- 172.21.15.179:0/2850622812 shutdown_connections 2024-05-27T06:58:46.281 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.288+0000 7fedef600700 1 --2- 172.21.15.179:0/2850622812 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fede8077770 0x7fede8075ba0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.281 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.288+0000 7fedef600700 1 -- 172.21.15.179:0/2850622812 >> 172.21.15.179:0/2850622812 conn(0x7fede80ff250 msgr2=0x7fede8101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:46.281 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.289+0000 7fedef600700 1 -- 172.21.15.179:0/2850622812 shutdown_connections 2024-05-27T06:58:46.281 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.289+0000 7fedef600700 1 -- 172.21.15.179:0/2850622812 wait complete. 2024-05-27T06:58:46.282 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.290+0000 7fedef600700 1 Processor -- start 2024-05-27T06:58:46.282 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.290+0000 7fedef600700 1 -- start start 2024-05-27T06:58:46.282 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.290+0000 7fedef600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fede8077770 0x7fede819b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:46.282 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.290+0000 7fedef600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fede810bc70 con 0x7fede8077770 2024-05-27T06:58:46.283 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.291+0000 7fedecc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fede8077770 0x7fede819b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:46.283 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.291+0000 7fedecc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fede8077770 0x7fede819b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.179:43052/0 (socket says 172.21.15.179:43052) 2024-05-27T06:58:46.283 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.291+0000 7fedecc00700 1 -- 172.21.15.179:0/1635777828 learned_addr learned my addr 172.21.15.179:0/1635777828 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-05-27T06:58:46.283 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.292+0000 7fedecc00700 1 -- 172.21.15.179:0/1635777828 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede819bd50 con 0x7fede8077770 2024-05-27T06:58:46.283 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.292+0000 7fedecc00700 1 --2- 172.21.15.179:0/1635777828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fede8077770 0x7fede819b810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feddc005120 tx=0x7feddc005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:46.284 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.293+0000 7fede5600700 1 -- 172.21.15.179:0/1635777828 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feddc027070 con 0x7fede8077770 2024-05-27T06:58:46.284 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.293+0000 7fedef600700 1 -- 172.21.15.179:0/1635777828 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede819bfe0 con 0x7fede8077770 2024-05-27T06:58:46.284 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.293+0000 7fede5600700 1 -- 172.21.15.179:0/1635777828 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feddc00bdf0 con 0x7fede8077770 2024-05-27T06:58:46.284 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.293+0000 7fede5600700 1 -- 172.21.15.179:0/1635777828 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feddc00fd70 con 0x7fede8077770 2024-05-27T06:58:46.284 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.293+0000 7fedef600700 1 -- 172.21.15.179:0/1635777828 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede819ebf0 con 0x7fede8077770 2024-05-27T06:58:46.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.294+0000 7fedef600700 1 -- 172.21.15.179:0/1635777828 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede810c3f0 con 0x7fede8077770 2024-05-27T06:58:46.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.295+0000 7fede5600700 1 -- 172.21.15.179:0/1635777828 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7feddc02a070 con 0x7fede8077770 2024-05-27T06:58:46.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.295+0000 7fede5600700 1 --2- 172.21.15.179:0/1635777828 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fedd003e570 0x7fedd0040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:46.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.295+0000 7fede5600700 1 -- 172.21.15.179:0/1635777828 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7feddc053ad0 con 0x7fede8077770 2024-05-27T06:58:46.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.296+0000 7fede7e00700 1 --2- 172.21.15.179:0/1635777828 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fedd003e570 0x7fedd0040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:46.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.297+0000 7fede7e00700 1 --2- 172.21.15.179:0/1635777828 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fedd003e570 0x7fedd0040a80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fedd4006fd0 tx=0x7fedd4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:46.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.313+0000 7fede5600700 1 -- 172.21.15.179:0/1635777828 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7feddc013070 con 0x7fede8077770 2024-05-27T06:58:46.548 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.556+0000 7fedef600700 1 -- 172.21.15.179:0/1635777828 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi132:172.21.15.132=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}) -- 0x7fede802e870 con 0x7fedd003e570 2024-05-27T06:58:46.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.572+0000 7fede5600700 1 -- 172.21.15.179:0/1635777828 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fede802e870 con 0x7fedd003e570 2024-05-27T06:58:46.565 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mon update... 2024-05-27T06:58:46.567 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.575+0000 7fedef600700 1 -- 172.21.15.179:0/1635777828 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fedd003e570 msgr2=0x7fedd0040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:46.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.575+0000 7fedef600700 1 --2- 172.21.15.179:0/1635777828 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fedd003e570 0x7fedd0040a80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fedd4006fd0 tx=0x7fedd4006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:46.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.575+0000 7fedef600700 1 -- 172.21.15.179:0/1635777828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fede8077770 msgr2=0x7fede819b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:46.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.575+0000 7fedef600700 1 --2- 172.21.15.179:0/1635777828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fede8077770 0x7fede819b810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feddc005120 tx=0x7feddc005810 comp rx=0 tx=0).stop 2024-05-27T06:58:46.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.576+0000 7fedef600700 1 -- 172.21.15.179:0/1635777828 shutdown_connections 2024-05-27T06:58:46.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.576+0000 7fedef600700 1 --2- 172.21.15.179:0/1635777828 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fedd003e570 0x7fedd0040a80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.576+0000 7fedef600700 1 --2- 172.21.15.179:0/1635777828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fede8077770 0x7fede819b810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.576+0000 7fedef600700 1 -- 172.21.15.179:0/1635777828 >> 172.21.15.179:0/1635777828 conn(0x7fede80ff250 msgr2=0x7fede80fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:46.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.576+0000 7fedef600700 1 -- 172.21.15.179:0/1635777828 shutdown_connections 2024-05-27T06:58:46.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:46.576+0000 7fedef600700 1 -- 172.21.15.179:0/1635777828 wait complete. 2024-05-27T06:58:46.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:46 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:46.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:46 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:46.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:46 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:46.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:46 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:46.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:46 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-05-27T06:58:46.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:46 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:58:46.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:46 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:58:46.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:46 smithi043 ceph-mon[31531]: Updating smithi179:/etc/ceph/ceph.conf 2024-05-27T06:58:46.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:46 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:47.599 DEBUG:teuthology.orchestra.run.smithi132:mon.b> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.b.service 2024-05-27T06:58:47.607 DEBUG:teuthology.orchestra.run.smithi179:mon.c> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.c.service 2024-05-27T06:58:47.611 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T06:58:47.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T06:58:47.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:47 smithi043 ceph-mon[31531]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:58:47.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:47 smithi043 ceph-mon[31531]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:47.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:47 smithi043 ceph-mon[31531]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi132:172.21.15.132=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:58:47.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:47 smithi043 ceph-mon[31531]: Saving service mon spec with placement smithi043:172.21.15.43=a;smithi132:172.21.15.132=b;smithi179:172.21.15.179=c;count:3 2024-05-27T06:58:48.017 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:58:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:48 smithi043 ceph-mon[31531]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:58:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:48 smithi043 ceph-mon[31531]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T06:58:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:58:48.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:58:49.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:49 smithi043 ceph-mon[31531]: Deploying daemon mon.c on smithi179 2024-05-27T06:58:49.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:49 smithi043 ceph-mon[31531]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:50.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.825+0000 7f3928000700 1 -- 172.21.15.179:0/325081186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3920105fd0 msgr2=0x7f39201063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:50.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.825+0000 7f3928000700 1 --2- 172.21.15.179:0/325081186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3920105fd0 0x7f39201063e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3914009910 tx=0x7f3914009c20 comp rx=0 tx=0).stop 2024-05-27T06:58:50.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.825+0000 7f3928000700 1 -- 172.21.15.179:0/325081186 shutdown_connections 2024-05-27T06:58:50.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.825+0000 7f3928000700 1 --2- 172.21.15.179:0/325081186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3920105fd0 0x7f39201063e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:50.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.825+0000 7f3928000700 1 -- 172.21.15.179:0/325081186 >> 172.21.15.179:0/325081186 conn(0x7f3920101880 msgr2=0x7f3920103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:50.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.825+0000 7f3928000700 1 -- 172.21.15.179:0/325081186 shutdown_connections 2024-05-27T06:58:50.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.825+0000 7f3928000700 1 -- 172.21.15.179:0/325081186 wait complete. 2024-05-27T06:58:50.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.826+0000 7f3928000700 1 Processor -- start 2024-05-27T06:58:50.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.826+0000 7f3928000700 1 -- start start 2024-05-27T06:58:50.819 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.827+0000 7f3928000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3920105fd0 0x7f39201a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:50.819 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.827+0000 7f3928000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3920110170 con 0x7f3920105fd0 2024-05-27T06:58:50.819 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.828+0000 7f3925600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3920105fd0 0x7f39201a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:50.819 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.828+0000 7f3925600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3920105fd0 0x7f39201a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.179:43066/0 (socket says 172.21.15.179:43066) 2024-05-27T06:58:50.819 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.828+0000 7f3925600700 1 -- 172.21.15.179:0/3240653581 learned_addr learned my addr 172.21.15.179:0/3240653581 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-05-27T06:58:50.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.828+0000 7f3925600700 1 -- 172.21.15.179:0/3240653581 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39201a1c40 con 0x7f3920105fd0 2024-05-27T06:58:50.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.829+0000 7f3925600700 1 --2- 172.21.15.179:0/3240653581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3920105fd0 0x7f39201a1700 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3914005660 tx=0x7f3914005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:50.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.829+0000 7f391e000700 1 -- 172.21.15.179:0/3240653581 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3914026070 con 0x7f3920105fd0 2024-05-27T06:58:50.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.829+0000 7f3928000700 1 -- 172.21.15.179:0/3240653581 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39201a1f30 con 0x7f3920105fd0 2024-05-27T06:58:50.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.829+0000 7f3928000700 1 -- 172.21.15.179:0/3240653581 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f392019b920 con 0x7f3920105fd0 2024-05-27T06:58:50.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.829+0000 7f391e000700 1 -- 172.21.15.179:0/3240653581 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f391400b6f0 con 0x7f3920105fd0 2024-05-27T06:58:50.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.829+0000 7f391e000700 1 -- 172.21.15.179:0/3240653581 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3914003680 con 0x7f3920105fd0 2024-05-27T06:58:50.822 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.831+0000 7f3928000700 1 -- 172.21.15.179:0/3240653581 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3900005330 con 0x7f3920105fd0 2024-05-27T06:58:50.823 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.831+0000 7f391e000700 1 -- 172.21.15.179:0/3240653581 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f3914029070 con 0x7f3920105fd0 2024-05-27T06:58:50.823 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.832+0000 7f391e000700 1 --2- 172.21.15.179:0/3240653581 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f390803e510 0x7f3908040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:50.823 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.832+0000 7f391e000700 1 -- 172.21.15.179:0/3240653581 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f391401e070 con 0x7f3920105fd0 2024-05-27T06:58:50.824 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.833+0000 7f3924c00700 1 --2- 172.21.15.179:0/3240653581 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f390803e510 0x7f3908040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:50.825 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.834+0000 7f3924c00700 1 --2- 172.21.15.179:0/3240653581 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f390803e510 0x7f3908040a20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f390c006fd0 tx=0x7f390c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:50.841 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:50.849+0000 7f391e000700 1 -- 172.21.15.179:0/3240653581 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3914013070 con 0x7f3920105fd0 2024-05-27T06:58:51.175 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.183+0000 7f3928000700 1 -- 172.21.15.179:0/3240653581 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3900005620 con 0x7f3920105fd0 2024-05-27T06:58:51.176 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.185+0000 7f391e000700 1 -- 172.21.15.179:0/3240653581 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f3914030090 con 0x7f3920105fd0 2024-05-27T06:58:51.176 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:58:51.177 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":1,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","modified":"2024-05-27T06:56:39.161167Z","created":"2024-05-27T06:56:39.161167Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-27T06:58:51.180 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.188+0000 7f3928000700 1 -- 172.21.15.179:0/3240653581 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f390803e510 msgr2=0x7f3908040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:51.180 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.188+0000 7f3928000700 1 --2- 172.21.15.179:0/3240653581 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f390803e510 0x7f3908040a20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f390c006fd0 tx=0x7f390c006e40 comp rx=0 tx=0).stop 2024-05-27T06:58:51.180 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.188+0000 7f3928000700 1 -- 172.21.15.179:0/3240653581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3920105fd0 msgr2=0x7f39201a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:51.180 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.188+0000 7f3928000700 1 --2- 172.21.15.179:0/3240653581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3920105fd0 0x7f39201a1700 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3914005660 tx=0x7f3914005740 comp rx=0 tx=0).stop 2024-05-27T06:58:51.180 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.188+0000 7f3928000700 1 -- 172.21.15.179:0/3240653581 shutdown_connections 2024-05-27T06:58:51.180 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.188+0000 7f3928000700 1 --2- 172.21.15.179:0/3240653581 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f390803e510 0x7f3908040a20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.188+0000 7f3928000700 1 --2- 172.21.15.179:0/3240653581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3920105fd0 0x7f39201a1700 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.188+0000 7f3928000700 1 -- 172.21.15.179:0/3240653581 >> 172.21.15.179:0/3240653581 conn(0x7f3920101880 msgr2=0x7f392010bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:51.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.188+0000 7f3928000700 1 -- 172.21.15.179:0/3240653581 shutdown_connections 2024-05-27T06:58:51.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:58:51.189+0000 7f3928000700 1 -- 172.21.15.179:0/3240653581 wait complete. 2024-05-27T06:58:51.182 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 1 2024-05-27T06:58:51.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:51 smithi043 ceph-mon[31531]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:51.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:51 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/3240653581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T06:58:53.013 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T06:58:53.013 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T06:58:53.663 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T06:58:53.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:53 smithi043 ceph-mon[31531]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 podman[35846]: 2024-05-27 06:58:55.186770792 +0000 UTC m=+0.133375020 container create 554dd2bd2723005388a7e57562bae43876e1674f6257eb9150c3b07af13a7952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T06:58:55.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:58:55 smithi043 ceph-mon[31531]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.044 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 podman[35846]: 2024-05-27 06:58:55.72705729 +0000 UTC m=+0.673661585 container init 554dd2bd2723005388a7e57562bae43876e1674f6257eb9150c3b07af13a7952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T06:58:56.044 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 podman[35846]: 2024-05-27 06:58:55.740095071 +0000 UTC m=+0.686699270 container start 554dd2bd2723005388a7e57562bae43876e1674f6257eb9150c3b07af13a7952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T06:58:56.044 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pidfile_write: ignore empty --pid-file 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: load: jerasure load: lrc 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: RocksDB version: 7.9.2 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Git sha 0 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: DB SUMMARY 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: DB Session ID: R3AL2ZEQ78LFD5QGQ38I 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: CURRENT file: CURRENT 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-27T06:58:56.045 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-27T06:58:56.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.error_if_exists: 0 2024-05-27T06:58:56.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.create_if_missing: 0 2024-05-27T06:58:56.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.paranoid_checks: 1 2024-05-27T06:58:56.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T06:58:56.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T06:58:56.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T06:58:56.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.env: 0x557d6bdac500 2024-05-27T06:58:56.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T06:58:56.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.info_log: 0x557d9b094040 2024-05-27T06:58:56.046 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T06:58:56.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.statistics: (nil) 2024-05-27T06:58:56.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.use_fsync: 0 2024-05-27T06:58:56.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_log_file_size: 0 2024-05-27T06:58:56.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T06:58:56.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T06:58:56.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T06:58:56.047 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T06:58:56.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.allow_fallocate: 1 2024-05-27T06:58:56.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T06:58:56.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T06:58:56.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.use_direct_reads: 0 2024-05-27T06:58:56.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T06:58:56.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T06:58:56.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.db_log_dir: 2024-05-27T06:58:56.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.wal_dir: 2024-05-27T06:58:56.048 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.write_buffer_manager: 0x557d9af5e8c0 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T06:58:56.049 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.unordered_write: 0 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.row_cache: None 2024-05-27T06:58:56.050 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.wal_filter: None 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.two_write_queues: 0 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.wal_compression: 0 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.atomic_flush: 0 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.log_readahead_size: 0 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T06:58:56.051 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_background_jobs: 2 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_background_compactions: -1 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_subcompactions: 1 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T06:58:56.052 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T06:58:56.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T06:58:56.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T06:58:56.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T06:58:56.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T06:58:56.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_open_files: -1 2024-05-27T06:58:56.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T06:58:56.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T06:58:56.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T06:58:56.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T06:58:56.053 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_background_flushes: -1 2024-05-27T06:58:56.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Compression algorithms supported: 2024-05-27T06:58:56.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T06:58:56.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: kZSTD supported: 0 2024-05-27T06:58:56.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: kXpressCompression supported: 0 2024-05-27T06:58:56.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T06:58:56.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: kZlibCompression supported: 1 2024-05-27T06:58:56.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: kSnappyCompression supported: 1 2024-05-27T06:58:56.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: kLZ4Compression supported: 1 2024-05-27T06:58:56.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: kBZip2Compression supported: 0 2024-05-27T06:58:56.054 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.merge_operator: 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_filter: None 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557d9b094200) 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout: cache_index_and_filter_blocks: 1 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T06:58:56.055 INFO:journalctl@ceph.mon.c.smithi179.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: pin_top_level_index_and_filter: 1 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_type: 0 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: data_block_index_type: 0 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_shortening: 1 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: checksum: 4 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: no_block_cache: 0 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache: 0x557d9af911f0 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_name: BinnedLRUCache 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_options: 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: capacity : 536870912 2024-05-27T06:58:56.056 INFO:journalctl@ceph.mon.c.smithi179.stdout: num_shard_bits : 4 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: strict_capacity_limit : 0 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: high_pri_pool_ratio: 0.000 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_compressed: (nil) 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: persistent_cache: (nil) 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_size: 4096 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_size_deviation: 10 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_restart_interval: 16 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_block_restart_interval: 1 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: metadata_block_size: 4096 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: partition_filters: 0 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: use_delta_encoding: 1 2024-05-27T06:58:56.057 INFO:journalctl@ceph.mon.c.smithi179.stdout: filter_policy: bloomfilter 2024-05-27T06:58:56.058 INFO:journalctl@ceph.mon.c.smithi179.stdout: whole_key_filtering: 1 2024-05-27T06:58:56.058 INFO:journalctl@ceph.mon.c.smithi179.stdout: verify_compression: 0 2024-05-27T06:58:56.058 INFO:journalctl@ceph.mon.c.smithi179.stdout: read_amp_bytes_per_bit: 0 2024-05-27T06:58:56.058 INFO:journalctl@ceph.mon.c.smithi179.stdout: format_version: 5 2024-05-27T06:58:56.058 INFO:journalctl@ceph.mon.c.smithi179.stdout: enable_index_compression: 1 2024-05-27T06:58:56.058 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_align: 0 2024-05-27T06:58:56.058 INFO:journalctl@ceph.mon.c.smithi179.stdout: max_auto_readahead_size: 262144 2024-05-27T06:58:56.058 INFO:journalctl@ceph.mon.c.smithi179.stdout: prepopulate_block_cache: 0 2024-05-27T06:58:56.058 INFO:journalctl@ceph.mon.c.smithi179.stdout: initial_auto_readahead_size: 8192 2024-05-27T06:58:56.060 INFO:journalctl@ceph.mon.c.smithi179.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compression: NoCompression 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.num_levels: 7 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T06:58:56.061 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T06:58:56.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T06:58:56.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T06:58:56.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T06:58:56.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T06:58:56.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T06:58:56.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T06:58:56.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T06:58:56.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T06:58:56.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T06:58:56.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T06:58:56.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T06:58:56.064 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T06:58:56.065 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.inplace_update_support: 0 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.bloom_locality: 0 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.max_successive_merges: 0 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.ttl: 2592000 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T06:58:56.066 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.enable_blob_files: false 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.min_blob_size: 0 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-27T06:58:56.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b482204b-89e8-4a7e-b329-40f430e864dd 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793135873342, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793135874818, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716793135, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b482204b-89e8-4a7e-b329-40f430e864dd", "db_session_id": "R3AL2ZEQ78LFD5QGQ38I", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793135875161, "job": 1, "event": "recovery_finished"} 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557d9af75c00 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: DB pointer 0x557d9fd68000 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** DB Stats ** 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T06:58:56.068 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** Compaction Stats [default] ** 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T06:58:56.069 INFO:journalctl@ceph.mon.c.smithi179.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** Compaction Stats [default] ** 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T06:58:56.070 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T06:58:56.071 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T06:58:56.071 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative compaction: 0.00 GB write, 0.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T06:58:56.071 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval compaction: 0.00 GB write, 0.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T06:58:56.071 INFO:journalctl@ceph.mon.c.smithi179.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T06:58:56.071 INFO:journalctl@ceph.mon.c.smithi179.stdout: Block cache BinnedLRUCache@0x557d9af911f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-05-27T06:58:56.071 INFO:journalctl@ceph.mon.c.smithi179.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-27T06:58:56.071 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-05-27T06:58:56.071 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T06:58:56.071 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: using public_addr v2:172.21.15.179:0/0 -> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: starting mon.c rank -1 at public addrs [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] at bind addrs [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(???) e0 preinit fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).mds e1 new map 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout: e1 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout: legacy client fscid: -1 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout: No filesystems configured 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-27T06:58:56.072 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Added host smithi132 2024-05-27T06:58:56.073 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Deploying cephadm binary to smithi179 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.074 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Updating smithi132:/etc/ceph/ceph.conf 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Added host smithi179 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.075 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4057861902' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4057861902' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.076 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Updating smithi179:/etc/ceph/ceph.conf 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi132:172.21.15.132=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Saving service mon spec with placement smithi043:172.21.15.43=a;smithi132:172.21.15.132=b;smithi179:172.21.15.179=c;count:3 2024-05-27T06:58:56.077 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: Deploying daemon mon.c on smithi179 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/3240653581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:58:56.078 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 ceph-mon[35888]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-27T06:58:56.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 bash[35846]: 554dd2bd2723005388a7e57562bae43876e1674f6257eb9150c3b07af13a7952 2024-05-27T06:58:56.079 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:58:55 smithi179 systemd[1]: Started Ceph mon.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T06:59:01.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T06:59:01.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: mon.a calling monitor election 2024-05-27T06:59:01.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:01.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: mon.c calling monitor election 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: monmap epoch 2 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: last_changed 2024-05-27T06:58:55.931615+0000 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: created 2024-05-27T06:56:39.161167+0000 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: min_mon_release 19 (squid) 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: election_strategy: 1 2024-05-27T06:59:01.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-05-27T06:59:01.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-05-27T06:59:01.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: fsmap 2024-05-27T06:59:01.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T06:59:01.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: mgrmap e13: x(active, since 68s) 2024-05-27T06:59:01.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: overall HEALTH_OK 2024-05-27T06:59:01.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:01.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:01.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:01.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:01.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: mon.a calling monitor election 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: mon.c calling monitor election 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:01.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: monmap epoch 2 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: last_changed 2024-05-27T06:58:55.931615+0000 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: created 2024-05-27T06:56:39.161167+0000 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: min_mon_release 19 (squid) 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: election_strategy: 1 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: fsmap 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: mgrmap e13: x(active, since 68s) 2024-05-27T06:59:01.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: overall HEALTH_OK 2024-05-27T06:59:01.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:01.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:01.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:01.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:01.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:02.089 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.094+0000 7fc0f1a00700 1 -- 172.21.15.179:0/60551391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d0006280 msgr2=0x7fc0d0008740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:02.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.094+0000 7fc0f1a00700 1 --2- 172.21.15.179:0/60551391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d0006280 0x7fc0d0008740 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec06bf30 tx=0x7fc0dc00fe90 comp rx=0 tx=0).stop 2024-05-27T06:59:02.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.094+0000 7fc0f1a00700 1 -- 172.21.15.179:0/60551391 shutdown_connections 2024-05-27T06:59:02.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.094+0000 7fc0f1a00700 1 --2- 172.21.15.179:0/60551391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d0006280 0x7fc0d0008740 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:02.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.094+0000 7fc0f1a00700 1 -- 172.21.15.179:0/60551391 >> 172.21.15.179:0/60551391 conn(0x7fc0ec079a70 msgr2=0x7fc0ec079e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:02.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.094+0000 7fc0f1a00700 1 -- 172.21.15.179:0/60551391 shutdown_connections 2024-05-27T06:59:02.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.095+0000 7fc0f1a00700 1 -- 172.21.15.179:0/60551391 wait complete. 2024-05-27T06:59:02.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.097+0000 7fc0f1a00700 1 Processor -- start 2024-05-27T06:59:02.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.098+0000 7fc0f1a00700 1 -- start start 2024-05-27T06:59:02.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.098+0000 7fc0f1a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d0006280 0x7fc0ec19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:02.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.098+0000 7fc0f1a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0ec19c260 0x7fc0ec1a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:02.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.098+0000 7fc0f1a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec10ba90 con 0x7fc0d0006280 2024-05-27T06:59:02.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.098+0000 7fc0f1a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec10b910 con 0x7fc0ec19c260 2024-05-27T06:59:02.092 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.099+0000 7fc0eaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d0006280 0x7fc0ec19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:02.092 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.099+0000 7fc0eaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d0006280 0x7fc0ec19bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.179:54778/0 (socket says 172.21.15.179:54778) 2024-05-27T06:59:02.092 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.099+0000 7fc0eaa00700 1 -- 172.21.15.179:0/1562161054 learned_addr learned my addr 172.21.15.179:0/1562161054 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-05-27T06:59:02.092 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.100+0000 7fc0ea000700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0ec19c260 0x7fc0ec1a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:02.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.100+0000 7fc0ea000700 1 -- 172.21.15.179:0/1562161054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0ec19c260 msgr2=0x7fc0ec1a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-27T06:59:02.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.100+0000 7fc0ea000700 1 -- 172.21.15.179:0/1562161054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0ec19c260 msgr2=0x7fc0ec1a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-27T06:59:02.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.100+0000 7fc0ea000700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0ec19c260 0x7fc0ec1a0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T06:59:02.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.100+0000 7fc0ea000700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0ec19c260 0x7fc0ec1a0550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T06:59:02.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.100+0000 7fc0eaa00700 1 -- 172.21.15.179:0/1562161054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0ec19c260 msgr2=0x7fc0ec1a0550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T06:59:02.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.100+0000 7fc0eaa00700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0ec19c260 0x7fc0ec1a0550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:02.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.101+0000 7fc0eaa00700 1 -- 172.21.15.179:0/1562161054 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ec19c860 con 0x7fc0d0006280 2024-05-27T06:59:02.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.102+0000 7fc0eaa00700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d0006280 0x7fc0ec19bd20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc005920 tx=0x7fc0dc01da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:02.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.103+0000 7fc0e3400700 1 -- 172.21.15.179:0/1562161054 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc0dc005400 con 0x7fc0d0006280 2024-05-27T06:59:02.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.103+0000 7fc0f1a00700 1 -- 172.21.15.179:0/1562161054 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0ec1a0a90 con 0x7fc0d0006280 2024-05-27T06:59:02.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.103+0000 7fc0e3400700 1 -- 172.21.15.179:0/1562161054 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0dc0055a0 con 0x7fc0d0006280 2024-05-27T06:59:02.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.103+0000 7fc0e3400700 1 -- 172.21.15.179:0/1562161054 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc0dc011730 con 0x7fc0d0006280 2024-05-27T06:59:02.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.103+0000 7fc0f1a00700 1 -- 172.21.15.179:0/1562161054 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0ec1a1020 con 0x7fc0d0006280 2024-05-27T06:59:02.097 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.105+0000 7fc0f1a00700 1 -- 172.21.15.179:0/1562161054 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0ec1a1390 con 0x7fc0d0006280 2024-05-27T06:59:02.099 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.106+0000 7fc0e3400700 1 -- 172.21.15.179:0/1562161054 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fc0dc0119e0 con 0x7fc0d0006280 2024-05-27T06:59:02.100 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.107+0000 7fc0e3400700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0cc03e300 0x7fc0cc040810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:02.100 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.107+0000 7fc0e3400700 1 -- 172.21.15.179:0/1562161054 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc0dc055690 con 0x7fc0d0006280 2024-05-27T06:59:02.100 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.108+0000 7fc0ea000700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0cc03e300 0x7fc0cc040810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:02.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.109+0000 7fc0ea000700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0cc03e300 0x7fc0cc040810 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec104a80 tx=0x7fc0d8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:02.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.123+0000 7fc0e3400700 1 -- 172.21.15.179:0/1562161054 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc0dc003ba0 con 0x7fc0d0006280 2024-05-27T06:59:02.451 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.457+0000 7fc0f1a00700 1 -- 172.21.15.179:0/1562161054 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc0ec19d080 con 0x7fc0d0006280 2024-05-27T06:59:02.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:02 smithi179 ceph-mon[35888]: Deploying daemon mon.b on smithi132 2024-05-27T06:59:02.451 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:02 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:02.451 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.459+0000 7fc0e3400700 1 -- 172.21.15.179:0/1562161054 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fc0dc024ce0 con 0x7fc0d0006280 2024-05-27T06:59:02.452 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:59:02.452 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":2,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","modified":"2024-05-27T06:58:55.931615Z","created":"2024-05-27T06:56:39.161167Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-27T06:59:02.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:02 smithi043 ceph-mon[31531]: Deploying daemon mon.b on smithi132 2024-05-27T06:59:02.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:02 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:02.455 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.462+0000 7fc0f1a00700 1 -- 172.21.15.179:0/1562161054 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0cc03e300 msgr2=0x7fc0cc040810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:02.455 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.462+0000 7fc0f1a00700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0cc03e300 0x7fc0cc040810 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec104a80 tx=0x7fc0d8008040 comp rx=0 tx=0).stop 2024-05-27T06:59:02.455 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.462+0000 7fc0f1a00700 1 -- 172.21.15.179:0/1562161054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d0006280 msgr2=0x7fc0ec19bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:02.455 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.462+0000 7fc0f1a00700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d0006280 0x7fc0ec19bd20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc005920 tx=0x7fc0dc01da60 comp rx=0 tx=0).stop 2024-05-27T06:59:02.455 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.462+0000 7fc0f1a00700 1 -- 172.21.15.179:0/1562161054 shutdown_connections 2024-05-27T06:59:02.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.462+0000 7fc0f1a00700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0cc03e300 0x7fc0cc040810 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:02.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.462+0000 7fc0f1a00700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d0006280 0x7fc0ec19bd20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:02.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.462+0000 7fc0f1a00700 1 --2- 172.21.15.179:0/1562161054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0ec19c260 0x7fc0ec1a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:02.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.462+0000 7fc0f1a00700 1 -- 172.21.15.179:0/1562161054 >> 172.21.15.179:0/1562161054 conn(0x7fc0ec079a70 msgr2=0x7fc0ec109300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:02.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.463+0000 7fc0f1a00700 1 -- 172.21.15.179:0/1562161054 shutdown_connections 2024-05-27T06:59:02.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:02.463+0000 7fc0f1a00700 1 -- 172.21.15.179:0/1562161054 wait complete. 2024-05-27T06:59:02.456 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 2 2024-05-27T06:59:03.129 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:03 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/1562161054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T06:59:03.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:03 smithi179 ceph-mon[35888]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:03.204 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 06:59:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T06:59:02.933+0000 7fc37a000700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-27T06:59:03.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:03 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/1562161054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T06:59:03.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:03 smithi043 ceph-mon[31531]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:04.182 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-27T06:59:04.183 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-27T06:59:04.625 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T06:59:05.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:05 smithi179 ceph-mon[35888]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:05.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:05 smithi043 ceph-mon[31531]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:06.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 podman[35264]: 2024-05-27 06:59:06.254679232 +0000 UTC m=+0.033747372 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T06:59:06.671 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.677+0000 7fafd7e00700 1 -- 172.21.15.179:0/430120156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafd8109aa0 msgr2=0x7fafd8109eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:06.671 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.677+0000 7fafd7e00700 1 --2- 172.21.15.179:0/430120156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafd8109aa0 0x7fafd8109eb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009910 tx=0x7fafc8009c20 comp rx=0 tx=0).stop 2024-05-27T06:59:06.671 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.677+0000 7fafd7e00700 1 -- 172.21.15.179:0/430120156 shutdown_connections 2024-05-27T06:59:06.671 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.677+0000 7fafd7e00700 1 --2- 172.21.15.179:0/430120156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafd8109aa0 0x7fafd8109eb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:06.671 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.677+0000 7fafd7e00700 1 -- 172.21.15.179:0/430120156 >> 172.21.15.179:0/430120156 conn(0x7fafd80ff5b0 msgr2=0x7fafd8101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:06.671 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.677+0000 7fafd7e00700 1 -- 172.21.15.179:0/430120156 shutdown_connections 2024-05-27T06:59:06.671 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.677+0000 7fafd7e00700 1 -- 172.21.15.179:0/430120156 wait complete. 2024-05-27T06:59:06.672 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.679+0000 7fafd7e00700 1 Processor -- start 2024-05-27T06:59:06.672 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.679+0000 7fafd7e00700 1 -- start start 2024-05-27T06:59:06.672 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.679+0000 7fafd7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafd8109aa0 0x7fafd8075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:06.672 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.679+0000 7fafd7e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafd8075ff0 0x7fafd8076400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:06.672 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.679+0000 7fafd7e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fafd810dea0 con 0x7fafd8109aa0 2024-05-27T06:59:06.672 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.679+0000 7fafd7e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fafd810dd20 con 0x7fafd8075ff0 2024-05-27T06:59:06.673 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.680+0000 7fafd6000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafd8075ff0 0x7fafd8076400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:06.673 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.680+0000 7fafd6000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafd8075ff0 0x7fafd8076400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:56176/0 (socket says 172.21.15.179:56176) 2024-05-27T06:59:06.674 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.680+0000 7fafd6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafd8109aa0 0x7fafd8075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:06.674 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.680+0000 7fafd6000700 1 -- 172.21.15.179:0/2650185450 learned_addr learned my addr 172.21.15.179:0/2650185450 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-05-27T06:59:06.674 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.681+0000 7fafd6000700 1 -- 172.21.15.179:0/2650185450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafd8109aa0 msgr2=0x7fafd8075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:06.674 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.681+0000 7fafd6000700 1 --2- 172.21.15.179:0/2650185450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafd8109aa0 0x7fafd8075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:06.674 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.681+0000 7fafd6000700 1 -- 172.21.15.179:0/2650185450 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd80779e0 con 0x7fafd8075ff0 2024-05-27T06:59:06.674 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.681+0000 7fafd6000700 1 --2- 172.21.15.179:0/2650185450 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafd8075ff0 0x7fafd8076400 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fafc400b700 tx=0x7fafc400ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:06.675 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.681+0000 7fafcf400700 1 -- 172.21.15.179:0/2650185450 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fafc4011630 con 0x7fafd8075ff0 2024-05-27T06:59:06.675 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.681+0000 7fafcf400700 1 -- 172.21.15.179:0/2650185450 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fafc4011c60 con 0x7fafd8075ff0 2024-05-27T06:59:06.675 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.681+0000 7fafd7e00700 1 -- 172.21.15.179:0/2650185450 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd81a5ca0 con 0x7fafd8075ff0 2024-05-27T06:59:06.675 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.681+0000 7fafcf400700 1 -- 172.21.15.179:0/2650185450 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fafc4010b80 con 0x7fafd8075ff0 2024-05-27T06:59:06.675 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.681+0000 7fafd7e00700 1 -- 172.21.15.179:0/2650185450 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd81a6200 con 0x7fafd8075ff0 2024-05-27T06:59:06.675 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.682+0000 7fafd7e00700 1 -- 172.21.15.179:0/2650185450 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd81051b0 con 0x7fafd8075ff0 2024-05-27T06:59:06.676 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.683+0000 7fafcf400700 1 -- 172.21.15.179:0/2650185450 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fafc4018020 con 0x7fafd8075ff0 2024-05-27T06:59:06.676 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.683+0000 7fafcf400700 1 --2- 172.21.15.179:0/2650185450 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fafbc03e2b0 0x7fafbc0407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:06.676 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.684+0000 7fafcf400700 1 -- 172.21.15.179:0/2650185450 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fafc4051bb0 con 0x7fafd8075ff0 2024-05-27T06:59:06.677 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.684+0000 7fafd6a00700 1 --2- 172.21.15.179:0/2650185450 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fafbc03e2b0 0x7fafbc0407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:06.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.685+0000 7fafd6a00700 1 --2- 172.21.15.179:0/2650185450 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fafbc03e2b0 0x7fafbc0407c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009750 tx=0x7fafc801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:06.683 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.690+0000 7fafcf400700 1 -- 172.21.15.179:0/2650185450 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fafc401f070 con 0x7fafd8075ff0 2024-05-27T06:59:06.848 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.855+0000 7fafcf400700 1 -- 172.21.15.179:0/2650185450 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc40184f0 con 0x7fafd8075ff0 2024-05-27T06:59:06.990 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:06.997+0000 7fafd7e00700 1 -- 172.21.15.179:0/2650185450 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fafd81a6510 con 0x7fafd8075ff0 2024-05-27T06:59:07.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 podman[35264]: 2024-05-27 06:59:06.647055349 +0000 UTC m=+0.426123563 container init 0f4e4ee5f492472259d70a517cde08a69176b19804c7a4972bfbf58e81ba1e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-b, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-05-27T06:59:07.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 podman[35264]: 2024-05-27 06:59:06.657773229 +0000 UTC m=+0.436841403 container start 0f4e4ee5f492472259d70a517cde08a69176b19804c7a4972bfbf58e81ba1e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-b, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T06:59:07.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 bash[35264]: 0f4e4ee5f492472259d70a517cde08a69176b19804c7a4972bfbf58e81ba1e48 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 systemd[1]: Started Ceph mon.b for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: set uid:gid to 167:167 (ceph:ceph) 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev), process ceph-mon, pid 2 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pidfile_write: ignore empty --pid-file 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: load: jerasure load: lrc 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: RocksDB version: 7.9.2 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Git sha 0 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Compile date 2024-05-21 07:00:12 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: DB SUMMARY 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: DB Session ID: 282MEYW7DGJF9RRCJFJQ 2024-05-27T06:59:07.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: CURRENT file: CURRENT 2024-05-27T06:59:07.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: IDENTITY file: IDENTITY 2024-05-27T06:59:07.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-27T06:59:07.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-05-27T06:59:07.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-05-27T06:59:07.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.error_if_exists: 0 2024-05-27T06:59:07.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.create_if_missing: 0 2024-05-27T06:59:07.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.paranoid_checks: 1 2024-05-27T06:59:07.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-27T06:59:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-27T06:59:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-27T06:59:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.env: 0x55cb973d2500 2024-05-27T06:59:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.fs: PosixFileSystem 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.info_log: 0x55cbc0a06040 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_file_opening_threads: 16 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.statistics: (nil) 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.use_fsync: 0 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_log_file_size: 0 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.keep_log_file_num: 1000 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.recycle_log_file_num: 0 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.allow_fallocate: 1 2024-05-27T06:59:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.allow_mmap_reads: 0 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.allow_mmap_writes: 0 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.use_direct_reads: 0 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.create_missing_column_families: 0 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.db_log_dir: 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.wal_dir: 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-27T06:59:07.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.advise_random_on_open: 1 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.db_write_buffer_size: 0 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.write_buffer_manager: 0x55cbc08d08c0 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.rate_limiter: (nil) 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.wal_recovery_mode: 2 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.enable_thread_tracking: 0 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.enable_pipelined_write: 0 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.unordered_write: 0 2024-05-27T06:59:07.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.row_cache: None 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.wal_filter: None 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.allow_ingest_behind: 0 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.two_write_queues: 0 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.manual_wal_flush: 0 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.wal_compression: 0 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.atomic_flush: 0 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-27T06:59:07.100 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.log_readahead_size: 0 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.best_efforts_recovery: 0 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.allow_data_in_errors: 0 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.db_host_id: __hostname__ 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_background_jobs: 2 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_background_compactions: -1 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_subcompactions: 1 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-27T06:59:07.101 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_total_wal_size: 0 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_open_files: -1 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bytes_per_sync: 0 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_readahead_size: 0 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_background_flushes: -1 2024-05-27T06:59:07.102 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Compression algorithms supported: 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: kZSTD supported: 0 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: kXpressCompression supported: 0 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: kLZ4HCCompression supported: 1 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: kZlibCompression supported: 1 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: kSnappyCompression supported: 1 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: kLZ4Compression supported: 1 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: kBZip2Compression supported: 0 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-27T06:59:07.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.merge_operator: 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_filter: None 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_filter_factory: None 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.sst_partitioner_factory: None 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cbc0a06200) 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout: cache_index_and_filter_blocks: 1 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout: pin_top_level_index_and_filter: 1 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout: index_type: 0 2024-05-27T06:59:07.104 INFO:journalctl@ceph.mon.b.smithi132.stdout: data_block_index_type: 0 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: index_shortening: 1 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: checksum: 4 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: no_block_cache: 0 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache: 0x55cbc09031f0 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache_name: BinnedLRUCache 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache_options: 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: capacity : 536870912 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: num_shard_bits : 4 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: strict_capacity_limit : 0 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: high_pri_pool_ratio: 0.000 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache_compressed: (nil) 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: persistent_cache: (nil) 2024-05-27T06:59:07.105 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_size: 4096 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_size_deviation: 10 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_restart_interval: 16 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: index_block_restart_interval: 1 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: metadata_block_size: 4096 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: partition_filters: 0 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: use_delta_encoding: 1 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: filter_policy: bloomfilter 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: whole_key_filtering: 1 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: verify_compression: 0 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: read_amp_bytes_per_bit: 0 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: format_version: 5 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: enable_index_compression: 1 2024-05-27T06:59:07.106 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_align: 0 2024-05-27T06:59:07.108 INFO:journalctl@ceph.mon.b.smithi132.stdout: max_auto_readahead_size: 262144 2024-05-27T06:59:07.108 INFO:journalctl@ceph.mon.b.smithi132.stdout: prepopulate_block_cache: 0 2024-05-27T06:59:07.108 INFO:journalctl@ceph.mon.b.smithi132.stdout: initial_auto_readahead_size: 8192 2024-05-27T06:59:07.108 INFO:journalctl@ceph.mon.b.smithi132.stdout: num_file_reads_for_auto_readahead: 2 2024-05-27T06:59:07.108 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.write_buffer_size: 33554432 2024-05-27T06:59:07.108 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_write_buffer_number: 2 2024-05-27T06:59:07.108 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compression: NoCompression 2024-05-27T06:59:07.108 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bottommost_compression: Disabled 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.prefix_extractor: nullptr 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.num_levels: 7 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-27T06:59:07.109 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compression_opts.level: 32767 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compression_opts.strategy: 0 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compression_opts.enabled: false 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-27T06:59:07.110 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.target_file_size_base: 67108864 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-27T06:59:07.111 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.arena_block_size: 1048576 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.disable_auto_compactions: 0 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-27T06:59:07.112 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.inplace_update_support: 0 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.bloom_locality: 0 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.max_successive_merges: 0 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.paranoid_file_checks: 0 2024-05-27T06:59:07.113 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.force_consistency_checks: 1 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.report_bg_io_stats: 0 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.ttl: 2592000 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.enable_blob_files: false 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.min_blob_size: 0 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.blob_file_size: 268435456 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-27T06:59:07.114 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.blob_file_starting_level: 0 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ec7956dd-d703-4eee-a00b-0e339b1b1db9 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793146789794, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793146790701, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716793146, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ec7956dd-d703-4eee-a00b-0e339b1b1db9", "db_session_id": "282MEYW7DGJF9RRCJFJQ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716793146790953, "job": 1, "event": "recovery_finished"} 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cbc08e7c00 2024-05-27T06:59:07.115 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: DB pointer 0x55cbc56da000 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** DB Stats ** 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** Compaction Stats [default] ** 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T06:59:07.116 INFO:journalctl@ceph.mon.b.smithi132.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** Compaction Stats [default] ** 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-27T06:59:07.117 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative compaction: 0.00 GB write, 0.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval compaction: 0.00 GB write, 0.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout: Block cache BinnedLRUCache@0x55cbc09031f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-05-27T06:59:07.118 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: using public_addr v2:172.21.15.132:0/0 -> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: starting mon.b rank -1 at public addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] at bind addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(???) e0 preinit fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).mds e1 new map 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).mds e1 print_map 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout: e1 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout: legacy client fscid: -1 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout: No filesystems configured 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-27T06:59:07.119 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Added host smithi132 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.120 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Deploying cephadm binary to smithi179 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:07.121 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Updating smithi132:/etc/ceph/ceph.conf 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Added host smithi179 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.122 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4057861902' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4057861902' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.123 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Updating smithi179:/etc/ceph/ceph.conf 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi132:172.21.15.132=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Saving service mon spec with placement smithi043:172.21.15.43=a;smithi132:172.21.15.132=b;smithi179:172.21.15.179=c;count:3 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T06:59:07.124 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Deploying daemon mon.c on smithi179 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/3240653581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.a calling monitor election 2024-05-27T06:59:07.125 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.c calling monitor election 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: monmap epoch 2 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: last_changed 2024-05-27T06:58:55.931615+0000 2024-05-27T06:59:07.126 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: created 2024-05-27T06:56:39.161167+0000 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: min_mon_release 19 (squid) 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: election_strategy: 1 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: fsmap 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mgrmap e13: x(active, since 68s) 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: overall HEALTH_OK 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:07.127 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: Deploying daemon mon.b on smithi132 2024-05-27T06:59:07.128 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:07.128 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/1562161054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T06:59:07.128 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.128 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:07.128 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:06 smithi132 ceph-mon[35283]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-05-27T06:59:11.875 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.881+0000 7fafcf400700 1 -- 172.21.15.179:0/2650185450 <== mon.1 v2:172.21.15.179:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7fafc4041520 con 0x7fafd8075ff0 2024-05-27T06:59:11.875 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-05-27T06:59:11.875 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":3,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","modified":"2024-05-27T06:59:06.842962Z","created":"2024-05-27T06:56:39.161167Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-27T06:59:11.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 -- 172.21.15.179:0/2650185450 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fafbc03e2b0 msgr2=0x7fafbc0407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:11.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 --2- 172.21.15.179:0/2650185450 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fafbc03e2b0 0x7fafbc0407c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009750 tx=0x7fafc801a040 comp rx=0 tx=0).stop 2024-05-27T06:59:11.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 -- 172.21.15.179:0/2650185450 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafd8075ff0 msgr2=0x7fafd8076400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:11.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 --2- 172.21.15.179:0/2650185450 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafd8075ff0 0x7fafd8076400 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fafc400b700 tx=0x7fafc400ba10 comp rx=0 tx=0).stop 2024-05-27T06:59:11.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 -- 172.21.15.179:0/2650185450 shutdown_connections 2024-05-27T06:59:11.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 --2- 172.21.15.179:0/2650185450 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fafbc03e2b0 0x7fafbc0407c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:11.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 --2- 172.21.15.179:0/2650185450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fafd8109aa0 0x7fafd8075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:11.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 --2- 172.21.15.179:0/2650185450 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafd8075ff0 0x7fafd8076400 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:11.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 -- 172.21.15.179:0/2650185450 >> 172.21.15.179:0/2650185450 conn(0x7fafd80ff5b0 msgr2=0x7fafd8102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:11.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 -- 172.21.15.179:0/2650185450 shutdown_connections 2024-05-27T06:59:11.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:11.887+0000 7fafd7e00700 1 -- 172.21.15.179:0/2650185450 wait complete. 2024-05-27T06:59:11.880 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 3 2024-05-27T06:59:12.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:12.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T06:59:12.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: mon.a calling monitor election 2024-05-27T06:59:12.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: mon.c calling monitor election 2024-05-27T06:59:12.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/2650185450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: mon.b calling monitor election 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: monmap epoch 3 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:59:12.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: last_changed 2024-05-27T06:59:06.842962+0000 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: created 2024-05-27T06:56:39.161167+0000 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: min_mon_release 19 (squid) 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: election_strategy: 1 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: 2: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.b 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: fsmap 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: mgrmap e13: x(active, since 79s) 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: overall HEALTH_OK 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:12.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T06:59:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: mon.a calling monitor election 2024-05-27T06:59:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: mon.c calling monitor election 2024-05-27T06:59:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/2650185450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: mon.b calling monitor election 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: monmap epoch 3 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: last_changed 2024-05-27T06:59:06.842962+0000 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: created 2024-05-27T06:56:39.161167+0000 2024-05-27T06:59:12.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: min_mon_release 19 (squid) 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: election_strategy: 1 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: 2: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.b 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: fsmap 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: mgrmap e13: x(active, since 79s) 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: overall HEALTH_OK 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:12.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:12.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: mon.a calling monitor election 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: mon.c calling monitor election 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/2650185450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: mon.b calling monitor election 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:12.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: monmap epoch 3 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: last_changed 2024-05-27T06:59:06.842962+0000 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: created 2024-05-27T06:56:39.161167+0000 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: min_mon_release 19 (squid) 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: election_strategy: 1 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: 2: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.b 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: fsmap 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: osdmap e4: 0 total, 0 up, 0 in 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: mgrmap e13: x(active, since 79s) 2024-05-27T06:59:12.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: overall HEALTH_OK 2024-05-27T06:59:12.346 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:12.346 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:12.346 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:12.634 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-27T06:59:12.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-05-27T06:59:13.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:13.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:12 smithi132 ceph-mon[35283]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:13.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:13.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:13.100 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T06:59:13.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:13.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:12 smithi043 ceph-mon[31531]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:13.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:13.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:12 smithi179 ceph-mon[35888]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T06:59:15.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.147+0000 7f73e1c00700 1 -- 172.21.15.43:0/2205537469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73dc103de0 msgr2=0x7f73dc1041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.147+0000 7f73e1c00700 1 --2- 172.21.15.43:0/2205537469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73dc103de0 0x7f73dc1041f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009910 tx=0x7f73cc009c20 comp rx=0 tx=0).stop 2024-05-27T06:59:15.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.147+0000 7f73e1c00700 1 -- 172.21.15.43:0/2205537469 shutdown_connections 2024-05-27T06:59:15.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.147+0000 7f73e1c00700 1 --2- 172.21.15.43:0/2205537469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73dc103de0 0x7f73dc1041f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.147+0000 7f73e1c00700 1 -- 172.21.15.43:0/2205537469 >> 172.21.15.43:0/2205537469 conn(0x7f73dc0ff690 msgr2=0x7f73dc101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:15.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.147+0000 7f73e1c00700 1 -- 172.21.15.43:0/2205537469 shutdown_connections 2024-05-27T06:59:15.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.147+0000 7f73e1c00700 1 -- 172.21.15.43:0/2205537469 wait complete. 2024-05-27T06:59:15.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.149+0000 7f73e1c00700 1 Processor -- start 2024-05-27T06:59:15.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.149+0000 7f73e1c00700 1 -- start start 2024-05-27T06:59:15.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.149+0000 7f73e1c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73dc103de0 0x7f73dc075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.149+0000 7f73e1c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73dc075ff0 0x7f73dc076400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.149+0000 7f73e1c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73dc1a5d30 0x7f73dc1a8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.149+0000 7f73e1c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc10df80 con 0x7f73dc103de0 2024-05-27T06:59:15.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.149+0000 7f73e1c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc10de00 con 0x7f73dc075ff0 2024-05-27T06:59:15.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.149+0000 7f73e1c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc10e100 con 0x7f73dc1a5d30 2024-05-27T06:59:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.150+0000 7f73daa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73dc103de0 0x7f73dc075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.150+0000 7f73daa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73dc103de0 0x7f73dc075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55974/0 (socket says 172.21.15.43:55974) 2024-05-27T06:59:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.150+0000 7f73daa00700 1 -- 172.21.15.43:0/3313316611 learned_addr learned my addr 172.21.15.43:0/3313316611 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:59:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.150+0000 7f73da000700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73dc075ff0 0x7f73dc076400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.150+0000 7f73db400700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73dc1a5d30 0x7f73dc1a8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.150+0000 7f73daa00700 1 -- 172.21.15.43:0/3313316611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73dc1a5d30 msgr2=0x7f73dc1a8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.150+0000 7f73daa00700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73dc1a5d30 0x7f73dc1a8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.150+0000 7f73daa00700 1 -- 172.21.15.43:0/3313316611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73dc075ff0 msgr2=0x7f73dc076400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.150+0000 7f73daa00700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73dc075ff0 0x7f73dc076400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.150+0000 7f73daa00700 1 -- 172.21.15.43:0/3313316611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc1a87d0 con 0x7f73dc103de0 2024-05-27T06:59:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.151+0000 7f73daa00700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73dc103de0 0x7f73dc075ab0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00f690 tx=0x7f73cc00f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.151+0000 7f73d3400700 1 -- 172.21.15.43:0/3313316611 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc026070 con 0x7f73dc103de0 2024-05-27T06:59:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.151+0000 7f73e1c00700 1 -- 172.21.15.43:0/3313316611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73dc1a8ac0 con 0x7f73dc103de0 2024-05-27T06:59:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.151+0000 7f73d3400700 1 -- 172.21.15.43:0/3313316611 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f73cc0044e0 con 0x7f73dc103de0 2024-05-27T06:59:15.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.151+0000 7f73d3400700 1 -- 172.21.15.43:0/3313316611 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc003ba0 con 0x7f73dc103de0 2024-05-27T06:59:15.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.151+0000 7f73e1c00700 1 -- 172.21.15.43:0/3313316611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73dc1a9050 con 0x7f73dc103de0 2024-05-27T06:59:15.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.153+0000 7f73e1c00700 1 -- 172.21.15.43:0/3313316611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73dc064e10 con 0x7f73dc103de0 2024-05-27T06:59:15.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.154+0000 7f73d3400700 1 -- 172.21.15.43:0/3313316611 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f73cc029070 con 0x7f73dc103de0 2024-05-27T06:59:15.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.154+0000 7f73d3400700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f73c003e9a0 0x7f73c0040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.154+0000 7f73d3400700 1 -- 172.21.15.43:0/3313316611 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f73cc0533c0 con 0x7f73dc103de0 2024-05-27T06:59:15.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.155+0000 7f73da000700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f73c003e9a0 0x7f73c0040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.155+0000 7f73da000700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f73c003e9a0 0x7f73c0040eb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f73dc10c1f0 tx=0x7f73c8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.165+0000 7f73d3400700 1 -- 172.21.15.43:0/3313316611 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f73cc013070 con 0x7f73dc103de0 2024-05-27T06:59:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:15 smithi043 ceph-mon[31531]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-27T06:59:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:15 smithi043 ceph-mon[31531]: Updating smithi132:/etc/ceph/ceph.conf 2024-05-27T06:59:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:15 smithi043 ceph-mon[31531]: Updating smithi179:/etc/ceph/ceph.conf 2024-05-27T06:59:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:15 smithi043 ceph-mon[31531]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:15 smithi179 ceph-mon[35888]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-27T06:59:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:15 smithi179 ceph-mon[35888]: Updating smithi132:/etc/ceph/ceph.conf 2024-05-27T06:59:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:15 smithi179 ceph-mon[35888]: Updating smithi179:/etc/ceph/ceph.conf 2024-05-27T06:59:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:15 smithi179 ceph-mon[35888]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:15.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.469+0000 7f73e1c00700 1 -- 172.21.15.43:0/3313316611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f73dc1a92e0 con 0x7f73dc103de0 2024-05-27T06:59:15.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.470+0000 7f73d3400700 1 -- 172.21.15.43:0/3313316611 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f73cc0142b0 con 0x7f73dc103de0 2024-05-27T06:59:15.462 INFO:teuthology.orchestra.run.smithi043.stdout:# minimal ceph.conf for 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:59:15.462 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-05-27T06:59:15.462 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T06:59:15.462 INFO:teuthology.orchestra.run.smithi043.stdout: mon_host = [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] 2024-05-27T06:59:15.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.474+0000 7f73e1c00700 1 -- 172.21.15.43:0/3313316611 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f73c003e9a0 msgr2=0x7f73c0040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.474+0000 7f73e1c00700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f73c003e9a0 0x7f73c0040eb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f73dc10c1f0 tx=0x7f73c8008040 comp rx=0 tx=0).stop 2024-05-27T06:59:15.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.474+0000 7f73e1c00700 1 -- 172.21.15.43:0/3313316611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73dc103de0 msgr2=0x7f73dc075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.474+0000 7f73e1c00700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73dc103de0 0x7f73dc075ab0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00f690 tx=0x7f73cc00f770 comp rx=0 tx=0).stop 2024-05-27T06:59:15.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.475+0000 7f73e1c00700 1 -- 172.21.15.43:0/3313316611 shutdown_connections 2024-05-27T06:59:15.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.475+0000 7f73e1c00700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f73c003e9a0 0x7f73c0040eb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.475+0000 7f73e1c00700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73dc1a5d30 0x7f73dc1a8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.475+0000 7f73e1c00700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73dc103de0 0x7f73dc075ab0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.475+0000 7f73e1c00700 1 --2- 172.21.15.43:0/3313316611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73dc075ff0 0x7f73dc076400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.475+0000 7f73e1c00700 1 -- 172.21.15.43:0/3313316611 >> 172.21.15.43:0/3313316611 conn(0x7f73dc0ff690 msgr2=0x7f73dc109910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:15.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.475+0000 7f73e1c00700 1 -- 172.21.15.43:0/3313316611 shutdown_connections 2024-05-27T06:59:15.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:15.475+0000 7f73e1c00700 1 -- 172.21.15.43:0/3313316611 wait complete. 2024-05-27T06:59:15.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:15.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:15.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:15.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:15 smithi132 ceph-mon[35283]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-27T06:59:15.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:15 smithi132 ceph-mon[35283]: Updating smithi132:/etc/ceph/ceph.conf 2024-05-27T06:59:15.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:15 smithi132 ceph-mon[35283]: Updating smithi179:/etc/ceph/ceph.conf 2024-05-27T06:59:15.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:15 smithi132 ceph-mon[35283]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:16.228 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-27T06:59:16.228 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:59:16.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T06:59:16.299 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:59:16.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:59:16.412 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:59:16.412 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T06:59:16.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3313316611' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.507 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:59:16.507 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:59:16.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:16.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:16.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:16.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3313316611' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:16.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.637 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:59:16.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.conf 2024-05-27T06:59:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T06:59:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3313316611' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:16.731 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:59:16.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-27T06:59:16.859 INFO:tasks.cephadm:Adding mgr.x on smithi043 2024-05-27T06:59:16.859 INFO:tasks.cephadm:Adding mgr.y on smithi132 2024-05-27T06:59:16.859 INFO:tasks.cephadm:Adding mgr.z on smithi179 2024-05-27T06:59:16.860 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch apply mgr '3;smithi043=x;smithi132=y;smithi179=z' 2024-05-27T06:59:17.327 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T06:59:17.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:17 smithi043 ceph-mon[31531]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T06:59:17.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:17.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T06:59:17.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:17.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:17 smithi043 ceph-mon[31531]: Reconfiguring daemon mon.a on smithi043 2024-05-27T06:59:17.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:17 smithi043 ceph-mon[31531]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:17.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:17 smithi179 ceph-mon[35888]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T06:59:17.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:17.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T06:59:17.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:17.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:17 smithi179 ceph-mon[35888]: Reconfiguring daemon mon.a on smithi043 2024-05-27T06:59:17.604 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:17 smithi179 ceph-mon[35888]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:17.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:17.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:17 smithi132 ceph-mon[35283]: Reconfiguring mon.a (unknown last config time)... 2024-05-27T06:59:17.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:17.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T06:59:17.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:17.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:17 smithi132 ceph-mon[35283]: Reconfiguring daemon mon.a on smithi043 2024-05-27T06:59:17.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:17 smithi132 ceph-mon[35283]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:19 smithi179 ceph-mon[35888]: Reconfiguring mon.b (monmap changed)... 2024-05-27T06:59:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T06:59:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:19 smithi179 ceph-mon[35888]: Reconfiguring daemon mon.b on smithi132 2024-05-27T06:59:19.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:19 smithi179 ceph-mon[35888]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:19.571 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.579+0000 7f5ce7a00700 1 -- 172.21.15.179:0/2040264952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce0103d20 msgr2=0x7f5ce0104130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:19.571 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.579+0000 7f5ce7a00700 1 --2- 172.21.15.179:0/2040264952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce0103d20 0x7f5ce0104130 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009910 tx=0x7f5cd4009c20 comp rx=0 tx=0).stop 2024-05-27T06:59:19.571 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.579+0000 7f5ce7a00700 1 -- 172.21.15.179:0/2040264952 shutdown_connections 2024-05-27T06:59:19.571 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.579+0000 7f5ce7a00700 1 --2- 172.21.15.179:0/2040264952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce0103d20 0x7f5ce0104130 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:19.571 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.579+0000 7f5ce7a00700 1 -- 172.21.15.179:0/2040264952 >> 172.21.15.179:0/2040264952 conn(0x7f5ce00ff360 msgr2=0x7f5ce0101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:19.572 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.579+0000 7f5ce7a00700 1 -- 172.21.15.179:0/2040264952 shutdown_connections 2024-05-27T06:59:19.572 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.579+0000 7f5ce7a00700 1 -- 172.21.15.179:0/2040264952 wait complete. 2024-05-27T06:59:19.572 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.580+0000 7f5ce7a00700 1 Processor -- start 2024-05-27T06:59:19.572 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.581+0000 7f5ce7a00700 1 -- start start 2024-05-27T06:59:19.573 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.581+0000 7f5ce7a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ce0103d20 0x7f5ce0077450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:19.573 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.581+0000 7f5ce7a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce0075b00 0x7f5ce0075f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:19.573 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.581+0000 7f5ce7a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ce0076450 0x7f5ce01a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:19.573 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.581+0000 7f5ce7a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce0109b90 con 0x7f5ce0075b00 2024-05-27T06:59:19.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.581+0000 7f5ce7a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce0109a10 con 0x7f5ce0103d20 2024-05-27T06:59:19.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.581+0000 7f5ce7a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce0109d10 con 0x7f5ce0076450 2024-05-27T06:59:19.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.581+0000 7f5ce5a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ce0076450 0x7f5ce01a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:19.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.581+0000 7f5ce5a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ce0076450 0x7f5ce01a3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:52904/0 (socket says 172.21.15.179:52904) 2024-05-27T06:59:19.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.581+0000 7f5ce5a00700 1 -- 172.21.15.179:0/182781295 learned_addr learned my addr 172.21.15.179:0/182781295 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-05-27T06:59:19.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.582+0000 7f5cdfe00700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce0075b00 0x7f5ce0075f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:19.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.582+0000 7f5ce5000700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ce0103d20 0x7f5ce0077450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:19.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.582+0000 7f5ce5a00700 1 -- 172.21.15.179:0/182781295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ce0103d20 msgr2=0x7f5ce0077450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:19.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.582+0000 7f5ce5a00700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ce0103d20 0x7f5ce0077450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:19.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.582+0000 7f5ce5a00700 1 -- 172.21.15.179:0/182781295 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce0075b00 msgr2=0x7f5ce0075f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:19.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.582+0000 7f5ce5a00700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce0075b00 0x7f5ce0075f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:19.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.582+0000 7f5ce5a00700 1 -- 172.21.15.179:0/182781295 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce01a3ed0 con 0x7f5ce0076450 2024-05-27T06:59:19.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.582+0000 7f5ce5000700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ce0103d20 0x7f5ce0077450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:19.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.582+0000 7f5ce5a00700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ce0076450 0x7f5ce01a3990 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5cd800ba50 tx=0x7f5cd800bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:19.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.583+0000 7f5cdd600700 1 -- 172.21.15.179:0/182781295 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8019070 con 0x7f5ce0076450 2024-05-27T06:59:19.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.583+0000 7f5cdd600700 1 -- 172.21.15.179:0/182781295 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5cd800ce40 con 0x7f5ce0076450 2024-05-27T06:59:19.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.583+0000 7f5ce7a00700 1 -- 172.21.15.179:0/182781295 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce01a4160 con 0x7f5ce0076450 2024-05-27T06:59:19.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.583+0000 7f5cdd600700 1 -- 172.21.15.179:0/182781295 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8007350 con 0x7f5ce0076450 2024-05-27T06:59:19.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.583+0000 7f5ce7a00700 1 -- 172.21.15.179:0/182781295 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce01a46a0 con 0x7f5ce0076450 2024-05-27T06:59:19.577 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.584+0000 7f5ce7a00700 1 -- 172.21.15.179:0/182781295 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce0064e10 con 0x7f5ce0076450 2024-05-27T06:59:19.577 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.584+0000 7f5cdd600700 1 -- 172.21.15.179:0/182781295 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f5cd801c070 con 0x7f5ce0076450 2024-05-27T06:59:19.577 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.585+0000 7f5cdd600700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5cc803e9a0 0x7f5cc8040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:19.577 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.585+0000 7f5cdd600700 1 -- 172.21.15.179:0/182781295 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5cd8051da0 con 0x7f5ce0076450 2024-05-27T06:59:19.577 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.585+0000 7f5ce5000700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5cc803e9a0 0x7f5cc8040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:19.578 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.586+0000 7f5ce5000700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5cc803e9a0 0x7f5cc8040eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4004690 tx=0x7f5cd401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:19.583 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.591+0000 7f5cdd600700 1 -- 172.21.15.179:0/182781295 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5cd8014680 con 0x7f5ce0076450 2024-05-27T06:59:19.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:19.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:19.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:19 smithi132 ceph-mon[35283]: Reconfiguring mon.b (monmap changed)... 2024-05-27T06:59:19.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:19.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T06:59:19.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:19.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:19 smithi132 ceph-mon[35283]: Reconfiguring daemon mon.b on smithi132 2024-05-27T06:59:19.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:19 smithi132 ceph-mon[35283]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:19 smithi043 ceph-mon[31531]: Reconfiguring mon.b (monmap changed)... 2024-05-27T06:59:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T06:59:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:19.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:19 smithi043 ceph-mon[31531]: Reconfiguring daemon mon.b on smithi132 2024-05-27T06:59:19.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:19 smithi043 ceph-mon[31531]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:19.823 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.831+0000 7f5ce7a00700 1 -- 172.21.15.179:0/182781295 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi043=x;smithi132=y;smithi179=z", "target": ["mon-mgr", ""]}) -- 0x7f5ce0101220 con 0x7f5cc803e9a0 2024-05-27T06:59:19.840 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.848+0000 7f5cdd600700 1 -- 172.21.15.179:0/182781295 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5ce0101220 con 0x7f5cc803e9a0 2024-05-27T06:59:19.840 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mgr update... 2024-05-27T06:59:19.843 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 -- 172.21.15.179:0/182781295 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5cc803e9a0 msgr2=0x7f5cc8040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:19.843 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5cc803e9a0 0x7f5cc8040eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4004690 tx=0x7f5cd401a040 comp rx=0 tx=0).stop 2024-05-27T06:59:19.843 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 -- 172.21.15.179:0/182781295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ce0076450 msgr2=0x7f5ce01a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:19.843 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ce0076450 0x7f5ce01a3990 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5cd800ba50 tx=0x7f5cd800bd60 comp rx=0 tx=0).stop 2024-05-27T06:59:19.843 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 -- 172.21.15.179:0/182781295 shutdown_connections 2024-05-27T06:59:19.843 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5cc803e9a0 0x7f5cc8040eb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:19.843 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ce0076450 0x7f5ce01a3990 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:19.843 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ce0103d20 0x7f5ce0077450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:19.843 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 --2- 172.21.15.179:0/182781295 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce0075b00 0x7f5ce0075f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:19.843 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 -- 172.21.15.179:0/182781295 >> 172.21.15.179:0/182781295 conn(0x7f5ce00ff360 msgr2=0x7f5ce0100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:19.844 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.851+0000 7f5ce7a00700 1 -- 172.21.15.179:0/182781295 shutdown_connections 2024-05-27T06:59:19.844 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T06:59:19.852+0000 7f5ce7a00700 1 -- 172.21.15.179:0/182781295 wait complete. 2024-05-27T06:59:20.587 DEBUG:teuthology.orchestra.run.smithi132:mgr.y> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.y.service 2024-05-27T06:59:20.595 DEBUG:teuthology.orchestra.run.smithi179:mgr.z> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.z.service 2024-05-27T06:59:20.600 INFO:tasks.cephadm:Deploying OSDs... 2024-05-27T06:59:20.600 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T06:59:20.600 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T06:59:20.661 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T06:59:20.661 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2024-05-27T06:59:20.713 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T06:59:20.713 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:20.713 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-05-27T06:59:20.713 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:20.713 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:20.713 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-05-27 06:58:05.755032212 +0000 2024-05-27T06:59:20.713 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-05-27 06:52:15.541626366 +0000 2024-05-27T06:59:20.714 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-05-27 06:52:15.541626366 +0000 2024-05-27T06:59:20.714 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-05-27 06:52:15.541626366 +0000 2024-05-27T06:59:20.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T06:59:20.790 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-05-27T06:59:20.790 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-05-27T06:59:20.790 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000284319 s, 1.8 MB/s 2024-05-27T06:59:20.793 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T06:59:20.852 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2024-05-27T06:59:20.905 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T06:59:20.906 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:20.906 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-05-27T06:59:20.906 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:20.906 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:20.906 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-05-27 06:58:05.859030978 +0000 2024-05-27T06:59:20.906 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-05-27 06:52:16.285617525 +0000 2024-05-27T06:59:20.906 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-05-27 06:52:16.285617525 +0000 2024-05-27T06:59:20.906 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-05-27 06:52:16.285617525 +0000 2024-05-27T06:59:20.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T06:59:20.988 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-05-27T06:59:20.988 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-05-27T06:59:20.988 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000277208 s, 1.8 MB/s 2024-05-27T06:59:20.992 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T06:59:21.055 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2024-05-27T06:59:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:20 smithi132 ceph-mon[35283]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi043=x;smithi132=y;smithi179=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:59:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:20 smithi132 ceph-mon[35283]: Saving service mgr spec with placement smithi043=x;smithi132=y;smithi179=z;count:3 2024-05-27T06:59:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:21.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:21.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:21.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:21.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T06:59:21.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:21.108 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T06:59:21.109 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:21.109 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-05-27T06:59:21.109 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:21.109 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:21.109 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-05-27 06:58:05.931030124 +0000 2024-05-27T06:59:21.109 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-05-27 06:52:16.872610549 +0000 2024-05-27T06:59:21.109 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-05-27 06:52:16.872610549 +0000 2024-05-27T06:59:21.109 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-05-27 06:52:16.872610549 +0000 2024-05-27T06:59:21.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T06:59:21.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:20 smithi043 ceph-mon[31531]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi043=x;smithi132=y;smithi179=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:59:21.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:20 smithi043 ceph-mon[31531]: Saving service mgr spec with placement smithi043=x;smithi132=y;smithi179=z;count:3 2024-05-27T06:59:21.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:21.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:21.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:21.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:21.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T06:59:21.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:21.196 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-05-27T06:59:21.197 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-05-27T06:59:21.197 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000316646 s, 1.6 MB/s 2024-05-27T06:59:21.200 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T06:59:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:20 smithi179 ceph-mon[35888]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi043=x;smithi132=y;smithi179=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:59:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:20 smithi179 ceph-mon[35888]: Saving service mgr spec with placement smithi043=x;smithi132=y;smithi179=z;count:3 2024-05-27T06:59:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-27T06:59:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-27T06:59:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:21.258 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2024-05-27T06:59:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T06:59:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:21.312 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 691 Links: 1 2024-05-27T06:59:21.312 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:21.312 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:21.312 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-05-27 06:58:06.006029235 +0000 2024-05-27T06:59:21.312 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-05-27 06:52:17.450603681 +0000 2024-05-27T06:59:21.312 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-05-27 06:52:17.450603681 +0000 2024-05-27T06:59:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-05-27 06:52:17.450603681 +0000 2024-05-27T06:59:21.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T06:59:21.399 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-05-27T06:59:21.399 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-05-27T06:59:21.399 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000294063 s, 1.7 MB/s 2024-05-27T06:59:21.403 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T06:59:21.460 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-05-27T06:59:21.461 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T06:59:21.518 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T06:59:21.518 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-05-27T06:59:21.570 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T06:59:21.570 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:21.570 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-05-27T06:59:21.571 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:21.571 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:21.571 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-05-27 06:59:05.800705030 +0000 2024-05-27T06:59:21.571 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-05-27 06:52:19.006241000 +0000 2024-05-27T06:59:21.571 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-05-27 06:52:19.006241000 +0000 2024-05-27T06:59:21.571 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-05-27 06:52:19.006241000 +0000 2024-05-27T06:59:21.571 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T06:59:21.650 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-05-27T06:59:21.650 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-05-27T06:59:21.650 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000390648 s, 1.3 MB/s 2024-05-27T06:59:21.654 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T06:59:21.710 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-05-27T06:59:21.759 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T06:59:21.760 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:21.760 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-05-27T06:59:21.760 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:21.760 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:21.760 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-05-27 06:59:05.801705005 +0000 2024-05-27T06:59:21.760 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-05-27 06:52:19.589226091 +0000 2024-05-27T06:59:21.760 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-05-27 06:52:19.589226091 +0000 2024-05-27T06:59:21.760 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-05-27 06:52:19.589226091 +0000 2024-05-27T06:59:21.760 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T06:59:21.836 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-05-27T06:59:21.837 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-05-27T06:59:21.837 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000365679 s, 1.4 MB/s 2024-05-27T06:59:21.841 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T06:59:21.897 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-05-27T06:59:21.946 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T06:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-05-27T06:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-05-27 06:59:05.801705005 +0000 2024-05-27T06:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-05-27 06:52:20.134212153 +0000 2024-05-27T06:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-05-27 06:52:20.134212153 +0000 2024-05-27T06:59:21.947 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-05-27 06:52:20.134212153 +0000 2024-05-27T06:59:21.948 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T06:59:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:21 smithi179 ceph-mon[35888]: Reconfiguring mon.c (monmap changed)... 2024-05-27T06:59:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:21 smithi179 ceph-mon[35888]: Reconfiguring daemon mon.c on smithi179 2024-05-27T06:59:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:21 smithi179 ceph-mon[35888]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:22.015 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:21 smithi132 ceph-mon[35283]: Reconfiguring mon.c (monmap changed)... 2024-05-27T06:59:22.015 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:21 smithi132 ceph-mon[35283]: Reconfiguring daemon mon.c on smithi179 2024-05-27T06:59:22.015 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:21 smithi132 ceph-mon[35283]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-05-27T06:59:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-05-27T06:59:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000406107 s, 1.3 MB/s 2024-05-27T06:59:22.027 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T06:59:22.081 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-05-27T06:59:22.130 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T06:59:22.130 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:22.130 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-05-27T06:59:22.130 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:22.130 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:22.130 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-05-27 06:59:05.802704979 +0000 2024-05-27T06:59:22.130 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-05-27 06:52:20.683198113 +0000 2024-05-27T06:59:22.130 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-05-27 06:52:20.683198113 +0000 2024-05-27T06:59:22.131 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-05-27 06:52:20.683198113 +0000 2024-05-27T06:59:22.131 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T06:59:22.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:21 smithi043 ceph-mon[31531]: Reconfiguring mon.c (monmap changed)... 2024-05-27T06:59:22.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:21 smithi043 ceph-mon[31531]: Reconfiguring daemon mon.c on smithi179 2024-05-27T06:59:22.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:21 smithi043 ceph-mon[31531]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:22.206 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-05-27T06:59:22.206 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-05-27T06:59:22.206 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000440012 s, 1.2 MB/s 2024-05-27T06:59:22.210 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T06:59:22.263 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-05-27T06:59:22.264 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T06:59:22.321 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T06:59:22.321 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2024-05-27T06:59:22.365 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T06:59:22.365 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:22.365 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 634 Links: 1 2024-05-27T06:59:22.365 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:22.365 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:22.365 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-05-27 06:58:54.228038078 +0000 2024-05-27T06:59:22.365 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-05-27 06:52:20.598352864 +0000 2024-05-27T06:59:22.365 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-05-27 06:52:20.598352864 +0000 2024-05-27T06:59:22.366 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-05-27 06:52:20.598352864 +0000 2024-05-27T06:59:22.366 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T06:59:22.474 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-05-27T06:59:22.475 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-05-27T06:59:22.475 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000322321 s, 1.6 MB/s 2024-05-27T06:59:22.478 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T06:59:22.524 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2024-05-27T06:59:22.571 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T06:59:22.572 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:22.572 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-05-27T06:59:22.572 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:22.572 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:22.572 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-05-27 06:58:54.229038056 +0000 2024-05-27T06:59:22.572 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-05-27 06:52:21.183339889 +0000 2024-05-27T06:59:22.572 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-05-27 06:52:21.183339889 +0000 2024-05-27T06:59:22.572 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-05-27 06:52:21.183339889 +0000 2024-05-27T06:59:22.572 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T06:59:22.638 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-05-27T06:59:22.638 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-05-27T06:59:22.638 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000305669 s, 1.7 MB/s 2024-05-27T06:59:22.640 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T06:59:22.681 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2024-05-27T06:59:22.768 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T06:59:22.768 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:22.768 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-05-27T06:59:22.768 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:22.768 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:22.768 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-05-27 06:58:54.230038035 +0000 2024-05-27T06:59:22.768 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-05-27 06:52:21.755327203 +0000 2024-05-27T06:59:22.769 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-05-27 06:52:21.755327203 +0000 2024-05-27T06:59:22.769 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-05-27 06:52:21.755327203 +0000 2024-05-27T06:59:22.769 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T06:59:22.876 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-05-27T06:59:22.877 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-05-27T06:59:22.877 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000396348 s, 1.3 MB/s 2024-05-27T06:59:22.880 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T06:59:22.928 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2024-05-27T06:59:22.973 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T06:59:22.973 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:59:22.973 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-05-27T06:59:22.973 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:59:22.974 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-05-27T06:59:22.974 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-05-27 06:58:54.230038035 +0000 2024-05-27T06:59:22.974 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-05-27 06:52:22.306314983 +0000 2024-05-27T06:59:22.974 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-05-27 06:52:22.306314983 +0000 2024-05-27T06:59:22.974 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-05-27 06:52:22.306314983 +0000 2024-05-27T06:59:22.974 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T06:59:23.093 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-05-27T06:59:23.093 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-05-27T06:59:23.093 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000391975 s, 1.3 MB/s 2024-05-27T06:59:23.097 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T06:59:23.148 INFO:tasks.cephadm:Deploying osd.0 on smithi043 with /dev/vg_nvme/lv_4... 2024-05-27T06:59:23.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T06:59:23.624 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T06:59:23.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:23.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:23.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:23.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:23.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:23.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:23.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:23.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T06:59:23.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T06:59:23.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T06:59:23.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:23.779 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:23 smithi043 ceph-mon[31531]: Deploying daemon mgr.z on smithi179 2024-05-27T06:59:23.784 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:23.784 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:23.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:23.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:23.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:23.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:23.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:23.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T06:59:23.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T06:59:23.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T06:59:23.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:23.785 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:23 smithi179 ceph-mon[35888]: Deploying daemon mgr.z on smithi179 2024-05-27T06:59:23.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:23.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:23.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:23.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:23.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:23.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:23.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:23.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T06:59:23.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T06:59:23.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T06:59:23.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:23.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:23 smithi132 ceph-mon[35283]: Deploying daemon mgr.z on smithi179 2024-05-27T06:59:25.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:25 smithi043 ceph-mon[31531]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:25.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:25 smithi179 ceph-mon[35888]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:25.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:25 smithi132 ceph-mon[35283]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:26.060 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:25 smithi179 podman[37603]: 2024-05-27 06:59:25.822951378 +0000 UTC m=+0.132876617 container create dc3f23bc322893644abf48800075e652f9a41551ce69bde6d125d086c7c1a3c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-05-27T06:59:26.060 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:26 smithi179 podman[37603]: 2024-05-27 06:59:26.068106072 +0000 UTC m=+0.378031359 container init dc3f23bc322893644abf48800075e652f9a41551ce69bde6d125d086c7c1a3c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T06:59:26.331 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T06:59:26.331 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T06:59:26.332 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-05-27T06:59:26.332 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-05-27T06:59:26.332 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0378307 s, 277 MB/s 2024-05-27T06:59:26.332 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-05-27T06:59:26.390 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:26 smithi179 podman[37603]: 2024-05-27 06:59:26.079095813 +0000 UTC m=+0.389021027 container start dc3f23bc322893644abf48800075e652f9a41551ce69bde6d125d086c7c1a3c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T06:59:26.391 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:26 smithi179 ceph-mgr[37622]: -- 172.21.15.179:0/4137429216 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556109ffc5a0 con 0x55610ed86480 2024-05-27T06:59:26.391 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:26 smithi179 bash[37603]: dc3f23bc322893644abf48800075e652f9a41551ce69bde6d125d086c7c1a3c0 2024-05-27T06:59:26.391 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:26 smithi179 systemd[1]: Started Ceph mgr.z for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T06:59:26.646 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:26.398+0000 7f4bd9b55200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T06:59:26.646 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:26.654+0000 7f4bd9b55200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:59:26.960 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:26.788+0000 7f4bd9b55200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T06:59:27.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_4 2024-05-27T06:59:27.460 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:27.190+0000 7f4bd9b55200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T06:59:27.460 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:27.304+0000 7f4bd9b55200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T06:59:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T06:59:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T06:59:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T06:59:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:27 smithi179 ceph-mon[35888]: Deploying daemon mgr.y on smithi132 2024-05-27T06:59:27.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:27 smithi179 ceph-mon[35888]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:27.480 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T06:59:27.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:27.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:27.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:27.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T06:59:27.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T06:59:27.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T06:59:27.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:27.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:27 smithi132 ceph-mon[35283]: Deploying daemon mgr.y on smithi132 2024-05-27T06:59:27.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:27 smithi132 ceph-mon[35283]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:27.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:27.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:27.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:27.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T06:59:27.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-27T06:59:27.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T06:59:27.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:27.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:27 smithi043 ceph-mon[31531]: Deploying daemon mgr.y on smithi132 2024-05-27T06:59:27.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:27 smithi043 ceph-mon[31531]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:29.210 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:28 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:28.882+0000 7f4bd9b55200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T06:59:29.210 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:29.110+0000 7f4bd9b55200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T06:59:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.528+0000 7ff105a00700 1 -- 172.21.15.43:0/2878205078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10010ab00 msgr2=0x7ff10010af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.528+0000 7ff105a00700 1 --2- 172.21.15.43:0/2878205078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10010ab00 0x7ff10010af10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009990 tx=0x7ff0f0009ca0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.528+0000 7ff105a00700 1 -- 172.21.15.43:0/2878205078 shutdown_connections 2024-05-27T06:59:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.528+0000 7ff105a00700 1 --2- 172.21.15.43:0/2878205078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10010c410 0x7ff100112cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.528+0000 7ff105a00700 1 --2- 172.21.15.43:0/2878205078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10010ab00 0x7ff10010af10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.528+0000 7ff105a00700 1 --2- 172.21.15.43:0/2878205078 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10010ba70 0x7ff10010be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.528+0000 7ff105a00700 1 -- 172.21.15.43:0/2878205078 >> 172.21.15.43:0/2878205078 conn(0x7ff1000799b0 msgr2=0x7ff10007be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.528+0000 7ff105a00700 1 -- 172.21.15.43:0/2878205078 shutdown_connections 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.528+0000 7ff105a00700 1 -- 172.21.15.43:0/2878205078 wait complete. 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.529+0000 7ff105a00700 1 Processor -- start 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.530+0000 7ff105a00700 1 -- start start 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.530+0000 7ff105a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10010ab00 0x7ff1001a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.530+0000 7ff105a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10010ba70 0x7ff1001a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.530+0000 7ff105a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10010c410 0x7ff1001a8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.530+0000 7ff105a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1001162a0 con 0x7ff10010ab00 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.530+0000 7ff105a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff100116120 con 0x7ff10010c410 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.530+0000 7ff105a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff100116420 con 0x7ff10010ba70 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.530+0000 7ff0fea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10010ab00 0x7ff1001a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.531+0000 7ff0fea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10010ab00 0x7ff1001a0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42838/0 (socket says 172.21.15.43:42838) 2024-05-27T06:59:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.531+0000 7ff0fea00700 1 -- 172.21.15.43:0/2086260852 learned_addr learned my addr 172.21.15.43:0/2086260852 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T06:59:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.531+0000 7ff0fe000700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10010ba70 0x7ff1001a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.531+0000 7ff0ff400700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10010c410 0x7ff1001a8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.531+0000 7ff0fea00700 1 -- 172.21.15.43:0/2086260852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10010ba70 msgr2=0x7ff1001a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.531+0000 7ff0fea00700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10010ba70 0x7ff1001a13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.531+0000 7ff0fea00700 1 -- 172.21.15.43:0/2086260852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10010c410 msgr2=0x7ff1001a8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.531+0000 7ff0fea00700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10010c410 0x7ff1001a8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.531+0000 7ff0fea00700 1 -- 172.21.15.43:0/2086260852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1001a8a00 con 0x7ff10010ab00 2024-05-27T06:59:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.532+0000 7ff0fea00700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10010ab00 0x7ff1001a0ea0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0006010 tx=0x7ff0f00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.532+0000 7ff0f7400700 1 -- 172.21.15.43:0/2086260852 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0026070 con 0x7ff10010ab00 2024-05-27T06:59:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.532+0000 7ff0f7400700 1 -- 172.21.15.43:0/2086260852 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff0f0004020 con 0x7ff10010ab00 2024-05-27T06:59:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.532+0000 7ff105a00700 1 -- 172.21.15.43:0/2086260852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1001a8c90 con 0x7ff10010ab00 2024-05-27T06:59:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.532+0000 7ff0f7400700 1 -- 172.21.15.43:0/2086260852 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f000fc00 con 0x7ff10010ab00 2024-05-27T06:59:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.532+0000 7ff105a00700 1 -- 172.21.15.43:0/2086260852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1001a90e0 con 0x7ff10010ab00 2024-05-27T06:59:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.533+0000 7ff105a00700 1 -- 172.21.15.43:0/2086260852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1000512b0 con 0x7ff10010ab00 2024-05-27T06:59:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.534+0000 7ff0f7400700 1 -- 172.21.15.43:0/2086260852 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7ff0f0029070 con 0x7ff10010ab00 2024-05-27T06:59:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.534+0000 7ff0f7400700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff0e403e750 0x7ff0e4040c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.534+0000 7ff0f7400700 1 -- 172.21.15.43:0/2086260852 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff0f00535d0 con 0x7ff10010ab00 2024-05-27T06:59:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.535+0000 7ff0fe000700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff0e403e750 0x7ff0e4040c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.535+0000 7ff0fe000700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff0e403e750 0x7ff0e4040c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff1001a1f50 tx=0x7ff0ec00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.541+0000 7ff0f7400700 1 -- 172.21.15.43:0/2086260852 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff0f0013070 con 0x7ff10010ab00 2024-05-27T06:59:29.539 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:29 smithi132 systemd[1]: Starting Ceph mgr.y for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T06:59:29.590 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:29.336+0000 7f4bd9b55200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T06:59:29.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:29.790+0000 7ff105a00700 1 -- 172.21.15.43:0/2086260852 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff100063bf0 con 0x7ff0e403e750 2024-05-27T06:59:29.788 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:29 smithi043 ceph-mon[31531]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:29.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:29 smithi132 ceph-mon[35283]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:29.846 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:29 smithi132 podman[36648]: 2024-05-27 06:59:29.453861649 +0000 UTC m=+0.040229535 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T06:59:29.846 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:29 smithi132 podman[36648]: 2024-05-27 06:59:29.573704007 +0000 UTC m=+0.160071926 container create a66e6a3505753eaea9c0f4e3c7a712297c94290187fbfc82a87f469078423a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T06:59:29.846 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:29 smithi132 podman[36648]: 2024-05-27 06:59:29.844005662 +0000 UTC m=+0.430373641 container init a66e6a3505753eaea9c0f4e3c7a712297c94290187fbfc82a87f469078423a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T06:59:29.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:29 smithi179 ceph-mon[35888]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:29.852 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:29.723+0000 7f4bd9b55200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T06:59:29.852 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:29.848+0000 7f4bd9b55200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T06:59:30.092 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:29 smithi132 podman[36648]: 2024-05-27 06:59:29.854073781 +0000 UTC m=+0.440441727 container start a66e6a3505753eaea9c0f4e3c7a712297c94290187fbfc82a87f469078423a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T06:59:30.093 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:29 smithi132 bash[36648]: a66e6a3505753eaea9c0f4e3c7a712297c94290187fbfc82a87f469078423a86 2024-05-27T06:59:30.093 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:29 smithi132 systemd[1]: Started Ceph mgr.y for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T06:59:30.093 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:29 smithi132 ceph-mgr[36667]: -- 172.21.15.132:0/398335880 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55672d2b25a0 con 0x55673203c900 2024-05-27T06:59:30.354 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:30 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:30.106+0000 7f4bd9b55200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T06:59:30.354 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:30 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:30.362+0000 7f4bd9b55200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:59:30.425 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:30.168+0000 7f86f30c5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T06:59:30.425 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:30.421+0000 7f86f30c5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:59:30.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:30 smithi179 ceph-mon[35888]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:59:30.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T06:59:30.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T06:59:30.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:30.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:30.842 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:30.564+0000 7f86f30c5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T06:59:30.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:30 smithi132 ceph-mon[35283]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:59:30.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T06:59:30.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T06:59:30.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:30.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:30.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:30 smithi043 ceph-mon[31531]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T06:59:30.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T06:59:30.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T06:59:30.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:30.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.934 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.934 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:30.934 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:31.341 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:30.988+0000 7f86f30c5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T06:59:31.342 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:31 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:31.107+0000 7f86f30c5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T06:59:31.459 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:31 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:31.198+0000 7f4bd9b55200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T06:59:31.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:31 smithi043 ceph-mon[31531]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:31.933 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:31 smithi132 ceph-mon[35283]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:31.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:31 smithi179 ceph-mon[35888]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:31.960 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:31 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:31.544+0000 7f4bd9b55200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T06:59:32.244 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:32 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:32.122+0000 7f4bd9b55200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T06:59:32.709 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:32 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:32.251+0000 7f4bd9b55200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T06:59:33.092 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:32 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:32.842+0000 7f86f30c5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T06:59:33.092 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:33.081+0000 7f86f30c5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T06:59:33.197 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:32 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:32.752+0000 7f4bd9b55200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T06:59:33.197 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:33.205+0000 7f4bd9b55200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T06:59:33.460 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:33.331+0000 7f4bd9b55200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T06:59:33.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:33 smithi179 ceph-mon[35888]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:33.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:33 smithi132 ceph-mon[35283]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:33.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.593 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:33.302+0000 7f86f30c5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T06:59:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:33 smithi043 ceph-mon[31531]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:33.960 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:33.653+0000 7f4bd9b55200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T06:59:33.960 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:33.836+0000 7f4bd9b55200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T06:59:34.092 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:33.705+0000 7f86f30c5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T06:59:34.092 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:33.842+0000 7f86f30c5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T06:59:34.352 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:34 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:34.101+0000 7f86f30c5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T06:59:34.353 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:34 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:34.359+0000 7f86f30c5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:59:34.368 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:33.969+0000 7f4bd9b55200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T06:59:34.710 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:34 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:34.375+0000 7f4bd9b55200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T06:59:34.710 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:34 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:34.508+0000 7f4bd9b55200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T06:59:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T06:59:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T06:59:34.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T06:59:34.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:34.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: Reconfiguring daemon mgr.x on smithi043 2024-05-27T06:59:34.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2840325200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92381bc8-d8af-4c8d-b55c-2311e7de0211"}]: dispatch 2024-05-27T06:59:34.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92381bc8-d8af-4c8d-b55c-2311e7de0211"}]: dispatch 2024-05-27T06:59:34.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92381bc8-d8af-4c8d-b55c-2311e7de0211"}]': finished 2024-05-27T06:59:34.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T06:59:34.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T06:59:34.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:34.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:34.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:34.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T06:59:34.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T06:59:34.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T06:59:34.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:34.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: Reconfiguring daemon mgr.x on smithi043 2024-05-27T06:59:34.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2840325200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92381bc8-d8af-4c8d-b55c-2311e7de0211"}]: dispatch 2024-05-27T06:59:34.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92381bc8-d8af-4c8d-b55c-2311e7de0211"}]: dispatch 2024-05-27T06:59:34.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92381bc8-d8af-4c8d-b55c-2311e7de0211"}]': finished 2024-05-27T06:59:34.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T06:59:34.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T06:59:34.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:34.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:34.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: Reconfiguring mgr.x (unknown last config time)... 2024-05-27T06:59:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-27T06:59:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-27T06:59:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: Reconfiguring daemon mgr.x on smithi043 2024-05-27T06:59:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2840325200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92381bc8-d8af-4c8d-b55c-2311e7de0211"}]: dispatch 2024-05-27T06:59:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92381bc8-d8af-4c8d-b55c-2311e7de0211"}]: dispatch 2024-05-27T06:59:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92381bc8-d8af-4c8d-b55c-2311e7de0211"}]': finished 2024-05-27T06:59:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: osdmap e5: 1 total, 0 up, 1 in 2024-05-27T06:59:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T06:59:35.444 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:35 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:35.158+0000 7f86f30c5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T06:59:35.444 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:35 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:35.451+0000 7f86f30c5200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T06:59:35.598 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:35 smithi179 ceph-mon[35888]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:35.598 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:35 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3576664915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T06:59:35.599 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:35 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:35.307+0000 7f4bd9b55200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T06:59:35.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:35 smithi132 ceph-mon[35283]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:35.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:35 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3576664915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T06:59:35.939 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 06:59:35 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T06:59:35.606+0000 7f4bd9b55200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T06:59:35.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:35 smithi043 ceph-mon[31531]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:35.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:35 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3576664915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T06:59:36.342 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:36 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:36.013+0000 7f86f30c5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T06:59:36.342 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:36 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:36.141+0000 7f86f30c5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T06:59:36.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:36.530+0000 7ff0f7400700 1 -- 172.21.15.43:0/2086260852 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 14) ==== 99937+0+0 (secure 0 0 0) 0x7ff0f002bce0 con 0x7ff10010ab00 2024-05-27T06:59:36.842 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:36 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:36.619+0000 7f86f30c5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T06:59:36.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:36 smithi132 ceph-mon[35283]: Standby manager daemon z started 2024-05-27T06:59:36.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:36 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T06:59:36.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:36 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T06:59:36.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:36 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T06:59:36.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:36 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T06:59:36.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:36 smithi043 ceph-mon[31531]: Standby manager daemon z started 2024-05-27T06:59:36.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:36 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T06:59:36.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:36 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T06:59:36.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:36 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T06:59:36.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:36 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T06:59:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:36 smithi179 ceph-mon[35888]: Standby manager daemon z started 2024-05-27T06:59:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:36 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T06:59:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:36 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T06:59:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:36 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T06:59:36.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:36 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.179:0/1557223226' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T06:59:37.342 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:37.071+0000 7f86f30c5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T06:59:37.342 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:37.196+0000 7f86f30c5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T06:59:37.826 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:37 smithi132 ceph-mon[35283]: mgrmap e14: x(active, since 104s), standbys: z 2024-05-27T06:59:37.826 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:37 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T06:59:37.826 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:37 smithi132 ceph-mon[35283]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:37.826 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:37.517+0000 7f86f30c5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T06:59:37.826 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:37.703+0000 7f86f30c5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T06:59:37.827 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:37.833+0000 7f86f30c5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T06:59:37.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:37 smithi043 ceph-mon[31531]: mgrmap e14: x(active, since 104s), standbys: z 2024-05-27T06:59:37.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:37 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T06:59:37.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:37 smithi043 ceph-mon[31531]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:37 smithi179 ceph-mon[35888]: mgrmap e14: x(active, since 104s), standbys: z 2024-05-27T06:59:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:37 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T06:59:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:37 smithi179 ceph-mon[35888]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:38.592 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:38 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:38.239+0000 7f86f30c5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T06:59:38.592 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:38 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:38.363+0000 7f86f30c5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T06:59:39.411 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:39.129+0000 7f86f30c5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T06:59:39.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:39.621+0000 7ff0f7400700 1 -- 172.21.15.43:0/2086260852 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7ff0f0003b90 con 0x7ff10010ab00 2024-05-27T06:59:39.819 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:39 smithi043 ceph-mon[31531]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:39.820 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:39 smithi043 ceph-mon[31531]: Standby manager daemon y started 2024-05-27T06:59:39.820 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:39 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T06:59:39.820 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:39 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T06:59:39.820 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:39 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T06:59:39.820 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:39 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T06:59:39.842 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 06:59:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T06:59:39.417+0000 7f86f30c5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T06:59:39.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:39 smithi132 ceph-mon[35283]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:39.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:39 smithi132 ceph-mon[35283]: Standby manager daemon y started 2024-05-27T06:59:39.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:39 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T06:59:39.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:39 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T06:59:39.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:39 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T06:59:39.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:39 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T06:59:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:39 smithi179 ceph-mon[35888]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:39 smithi179 ceph-mon[35888]: Standby manager daemon y started 2024-05-27T06:59:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:39 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T06:59:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:39 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T06:59:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:39 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T06:59:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:39 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.132:0/367682447' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T06:59:40.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:40 smithi043 ceph-mon[31531]: mgrmap e15: x(active, since 107s), standbys: z, y 2024-05-27T06:59:40.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:40 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T06:59:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:40 smithi179 ceph-mon[35888]: mgrmap e15: x(active, since 107s), standbys: z, y 2024-05-27T06:59:40.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:40 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T06:59:41.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:40 smithi132 ceph-mon[35283]: mgrmap e15: x(active, since 107s), standbys: z, y 2024-05-27T06:59:41.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:40 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T06:59:41.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:41 smithi043 ceph-mon[31531]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:41.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:41.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:41.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:41 smithi179 ceph-mon[35888]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:41.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:41.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:41.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:41.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:41 smithi132 ceph-mon[35283]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:41.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:41.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:41.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:42.755 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:42 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:42.756 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:42 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:42.756 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:42 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:42.802 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:42 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:42.802 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:42 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:42.802 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:42 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:42.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:42 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:42.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:42 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:42.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:42 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:43.928 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:43 smithi132 ceph-mon[35283]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:43.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:43 smithi043 ceph-mon[31531]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:43.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:43 smithi179 ceph-mon[35888]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:44 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T06:59:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:44 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:44.962 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:44 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T06:59:44.963 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:44 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:45.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:44 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-27T06:59:45.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:44 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:45.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:45 smithi043 ceph-mon[31531]: Deploying daemon osd.0 on smithi043 2024-05-27T06:59:45.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:45 smithi043 ceph-mon[31531]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:45 smithi179 ceph-mon[35888]: Deploying daemon osd.0 on smithi043 2024-05-27T06:59:45.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:45 smithi179 ceph-mon[35888]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:46.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:45 smithi132 ceph-mon[35283]: Deploying daemon osd.0 on smithi043 2024-05-27T06:59:46.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:45 smithi132 ceph-mon[35283]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:47.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:47 smithi043 ceph-mon[31531]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:47.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:47 smithi179 ceph-mon[35888]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:47 smithi132 ceph-mon[35283]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:49.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:49 smithi043 ceph-mon[31531]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:49 smithi179 ceph-mon[35888]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:50.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:49 smithi132 ceph-mon[35283]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:51.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:51 smithi043 ceph-mon[31531]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:51.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:51 smithi179 ceph-mon[35888]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:52.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:51 smithi132 ceph-mon[35283]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:53.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:53 smithi043 ceph-mon[31531]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:53.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:53.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:53.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:53.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:53 smithi179 ceph-mon[35888]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:53.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:53.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:53.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:54.014 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:53 smithi132 ceph-mon[35283]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:54.014 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:54.014 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:54.014 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:55.922 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:55 smithi043 ceph-mon[31531]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:55.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:55 smithi179 ceph-mon[35888]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:56.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:55 smithi132 ceph-mon[35283]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:57.404 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:57.404 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:57.404 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:57.404 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:57.404 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:57.404 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:57 smithi043 ceph-mon[31531]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:57.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:57.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:57.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:57.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:57.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:57.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:57 smithi179 ceph-mon[35888]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:57.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:57.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:57.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T06:59:57.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T06:59:57.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:57.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:57 smithi132 ceph-mon[35283]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:58 smithi043 ceph-mon[31531]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T06:59:58.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:58 smithi179 ceph-mon[35888]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T06:59:58.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:58 smithi132 ceph-mon[35283]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-27T06:59:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.849+0000 7ff0f7400700 1 -- 172.21.15.43:0/2086260852 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff100063bf0 con 0x7ff0e403e750 2024-05-27T06:59:58.841 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 0 on host 'smithi043' 2024-05-27T06:59:58.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.852+0000 7ff105a00700 1 -- 172.21.15.43:0/2086260852 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff0e403e750 msgr2=0x7ff0e4040c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:58.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.852+0000 7ff105a00700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff0e403e750 0x7ff0e4040c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff1001a1f50 tx=0x7ff0ec00b3c0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.852+0000 7ff105a00700 1 -- 172.21.15.43:0/2086260852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10010ab00 msgr2=0x7ff1001a0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:58.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.852+0000 7ff105a00700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10010ab00 0x7ff1001a0ea0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0006010 tx=0x7ff0f00057c0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.852+0000 7ff105a00700 1 -- 172.21.15.43:0/2086260852 shutdown_connections 2024-05-27T06:59:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.852+0000 7ff105a00700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff0e403e750 0x7ff0e4040c60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.852+0000 7ff105a00700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10010c410 0x7ff1001a8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.852+0000 7ff105a00700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10010ab00 0x7ff1001a0ea0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.852+0000 7ff105a00700 1 --2- 172.21.15.43:0/2086260852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10010ba70 0x7ff1001a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.852+0000 7ff105a00700 1 -- 172.21.15.43:0/2086260852 >> 172.21.15.43:0/2086260852 conn(0x7ff1000799b0 msgr2=0x7ff10010eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.853+0000 7ff105a00700 1 -- 172.21.15.43:0/2086260852 shutdown_connections 2024-05-27T06:59:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T06:59:58.853+0000 7ff105a00700 1 -- 172.21.15.43:0/2086260852 wait complete. 2024-05-27T06:59:59.550 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:59 smithi043 ceph-mon[31531]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T06:59:59.550 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:59 smithi043 ceph-mon[31531]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T06:59:59.550 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:59 smithi043 ceph-mon[31531]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T06:59:59.550 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:59 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T06:59:59.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:59 smithi043 ceph-mon[31531]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:59.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:59 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:59.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:59 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:59.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 06:59:59 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:59.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:59 smithi132 ceph-mon[35283]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T06:59:59.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:59 smithi132 ceph-mon[35283]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T06:59:59.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:59 smithi132 ceph-mon[35283]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T06:59:59.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:59 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T06:59:59.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:59 smithi132 ceph-mon[35283]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:59.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:59 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:59.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:59 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:59.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 06:59:59 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:59.604 DEBUG:teuthology.orchestra.run.smithi043:osd.0> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.0.service 2024-05-27T06:59:59.609 INFO:tasks.cephadm:Deploying osd.1 on smithi132 with /dev/vg_nvme/lv_4... 2024-05-27T06:59:59.609 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T06:59:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:59 smithi179 ceph-mon[35888]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-27T06:59:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:59 smithi179 ceph-mon[35888]: osdmap e6: 1 total, 0 up, 1 in 2024-05-27T06:59:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:59 smithi179 ceph-mon[35888]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T06:59:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:59 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T06:59:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:59 smithi179 ceph-mon[35888]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T06:59:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:59 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T06:59:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:59 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T06:59:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 06:59:59 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:00.063 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:00:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:00 smithi043 ceph-mon[31531]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:00:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:00 smithi043 ceph-mon[31531]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:00:00.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:00 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:00.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:00 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:00.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:00 smithi043 ceph-mon[31531]: overall HEALTH_OK 2024-05-27T07:00:00.532 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:00 smithi132 ceph-mon[35283]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:00:00.532 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:00 smithi132 ceph-mon[35283]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:00:00.533 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:00 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:00.533 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:00 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:00.533 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:00 smithi132 ceph-mon[35283]: overall HEALTH_OK 2024-05-27T07:00:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:00 smithi179 ceph-mon[35888]: from='osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:00:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:00 smithi179 ceph-mon[35888]: osdmap e7: 1 total, 0 up, 1 in 2024-05-27T07:00:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:00 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:00 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:00 smithi179 ceph-mon[35888]: overall HEALTH_OK 2024-05-27T07:00:01.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:01 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:00:01.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:01 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:00:01.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:01.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:01 smithi043 ceph-mon[31531]: osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] boot 2024-05-27T07:00:01.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:01 smithi043 ceph-mon[31531]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:00:01.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:01.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:01 smithi043 ceph-mon[31531]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:00:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:01 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:00:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:01 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:00:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:01 smithi132 ceph-mon[35283]: osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] boot 2024-05-27T07:00:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:01 smithi132 ceph-mon[35283]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:00:01.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:01.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:01 smithi132 ceph-mon[35283]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:00:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:01 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:00:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:01 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:00:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:01 smithi179 ceph-mon[35888]: osd.0 [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] boot 2024-05-27T07:00:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:01 smithi179 ceph-mon[35888]: osdmap e8: 1 total, 1 up, 1 in 2024-05-27T07:00:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:00:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:01 smithi179 ceph-mon[35888]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-27T07:00:02.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:02 smithi132 ceph-mon[35283]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:00:02.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:02 smithi043 ceph-mon[31531]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:00:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:02 smithi179 ceph-mon[35888]: osdmap e9: 1 total, 1 up, 1 in 2024-05-27T07:00:02.866 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:00:02.866 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:00:02.866 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-05-27T07:00:02.866 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-05-27T07:00:02.866 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0426287 s, 246 MB/s 2024-05-27T07:00:02.866 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-05-27T07:00:03.550 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:03 smithi132 ceph-mon[35283]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:03.550 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:03 smithi043 ceph-mon[31531]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:03.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:03 smithi179 ceph-mon[35888]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:03.861 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_4 2024-05-27T07:00:04.267 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:00:05.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:05 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi043 2024-05-27T07:00:05.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:00:05.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:05 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi043 to 17094M 2024-05-27T07:00:05.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:05.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:00:05.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:05 smithi132 ceph-mon[35283]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:05.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:05 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi043 2024-05-27T07:00:05.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:00:05.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:05 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi043 to 17094M 2024-05-27T07:00:05.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:05.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:00:05.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:05 smithi043 ceph-mon[31531]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:05 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi043 2024-05-27T07:00:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:00:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:05 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi043 to 17094M 2024-05-27T07:00:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:05.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:00:05.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:05.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:05 smithi179 ceph-mon[35888]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:06.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.543+0000 7fe5b4200700 1 -- 172.21.15.132:0/1093760432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5ac106540 msgr2=0x7fe5ac106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:06.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.543+0000 7fe5b4200700 1 --2- 172.21.15.132:0/1093760432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5ac106540 0x7fe5ac106950 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009990 tx=0x7fe5a0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:00:06.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.543+0000 7fe5b4200700 1 -- 172.21.15.132:0/1093760432 shutdown_connections 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.543+0000 7fe5b4200700 1 --2- 172.21.15.132:0/1093760432 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5ac108010 0x7fe5ac10e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.543+0000 7fe5b4200700 1 --2- 172.21.15.132:0/1093760432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5ac106540 0x7fe5ac106950 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.543+0000 7fe5b4200700 1 --2- 172.21.15.132:0/1093760432 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5ac107670 0x7fe5ac107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.543+0000 7fe5b4200700 1 -- 172.21.15.132:0/1093760432 >> 172.21.15.132:0/1093760432 conn(0x7fe5ac101b80 msgr2=0x7fe5ac104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.543+0000 7fe5b4200700 1 -- 172.21.15.132:0/1093760432 shutdown_connections 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.543+0000 7fe5b4200700 1 -- 172.21.15.132:0/1093760432 wait complete. 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.544+0000 7fe5b4200700 1 Processor -- start 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.544+0000 7fe5b4200700 1 -- start start 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.545+0000 7fe5b4200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5ac106540 0x7fe5ac19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.545+0000 7fe5b4200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5ac107670 0x7fe5ac19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:06.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.545+0000 7fe5b4200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5ac108010 0x7fe5ac1a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.545+0000 7fe5b4200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac111e80 con 0x7fe5ac108010 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.545+0000 7fe5b4200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac111d00 con 0x7fe5ac106540 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.545+0000 7fe5b4200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac112000 con 0x7fe5ac107670 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.545+0000 7fe5b0e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5ac107670 0x7fe5ac19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.545+0000 7fe5b0e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5ac107670 0x7fe5ac19cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.132:40684/0 (socket says 172.21.15.132:40684) 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.545+0000 7fe5b0e00700 1 -- 172.21.15.132:0/2344852002 learned_addr learned my addr 172.21.15.132:0/2344852002 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.546+0000 7fe5b2200700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5ac108010 0x7fe5ac1a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.546+0000 7fe5b1800700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5ac106540 0x7fe5ac19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.546+0000 7fe5b0e00700 1 -- 172.21.15.132:0/2344852002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5ac106540 msgr2=0x7fe5ac19ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.546+0000 7fe5b0e00700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5ac106540 0x7fe5ac19ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:06.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.546+0000 7fe5b0e00700 1 -- 172.21.15.132:0/2344852002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5ac108010 msgr2=0x7fe5ac1a3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:06.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.546+0000 7fe5b0e00700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5ac108010 0x7fe5ac1a3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:06.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.546+0000 7fe5b0e00700 1 -- 172.21.15.132:0/2344852002 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ac1a45b0 con 0x7fe5ac107670 2024-05-27T07:00:06.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.546+0000 7fe5b1800700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5ac106540 0x7fe5ac19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:00:06.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.547+0000 7fe5b0e00700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5ac107670 0x7fe5ac19cf90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe59800d950 tx=0x7fe59800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:06.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.547+0000 7fe5aa000700 1 -- 172.21.15.132:0/2344852002 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe598016070 con 0x7fe5ac107670 2024-05-27T07:00:06.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.547+0000 7fe5aa000700 1 -- 172.21.15.132:0/2344852002 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe59800fd20 con 0x7fe5ac107670 2024-05-27T07:00:06.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.547+0000 7fe5b4200700 1 -- 172.21.15.132:0/2344852002 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ac1a48a0 con 0x7fe5ac107670 2024-05-27T07:00:06.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.547+0000 7fe5aa000700 1 -- 172.21.15.132:0/2344852002 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59800e3b0 con 0x7fe5ac107670 2024-05-27T07:00:06.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.547+0000 7fe5b4200700 1 -- 172.21.15.132:0/2344852002 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ac1a4de0 con 0x7fe5ac107670 2024-05-27T07:00:06.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.548+0000 7fe5b4200700 1 -- 172.21.15.132:0/2344852002 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ac0512b0 con 0x7fe5ac107670 2024-05-27T07:00:06.545 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.553+0000 7fe5aa000700 1 -- 172.21.15.132:0/2344852002 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7fe598019070 con 0x7fe5ac107670 2024-05-27T07:00:06.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.554+0000 7fe5aa000700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5940b2870 0x7fe5940b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:06.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.555+0000 7fe5aa000700 1 -- 172.21.15.132:0/2344852002 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fe5980e02d0 con 0x7fe5ac107670 2024-05-27T07:00:06.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.555+0000 7fe5b1800700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5940b2870 0x7fe5940b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:06.548 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.556+0000 7fe5b1800700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5940b2870 0x7fe5940b4d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0003820 tx=0x7fe5a001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:06.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.566+0000 7fe5aa000700 1 -- 172.21.15.132:0/2344852002 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe5980a99c0 con 0x7fe5ac107670 2024-05-27T07:00:06.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:06.810+0000 7fe5b4200700 1 -- 172.21.15.132:0/2344852002 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe5ac063bf0 con 0x7fe5940b2870 2024-05-27T07:00:07.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:07 smithi043 ceph-mon[31531]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:07.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:07 smithi043 ceph-mon[31531]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:00:07.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:00:07.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:00:07.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:07 smithi179 ceph-mon[35888]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:07 smithi179 ceph-mon[35888]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:00:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:00:07.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:00:07.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:07.966 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:07 smithi132 ceph-mon[35283]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:07.966 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:07 smithi132 ceph-mon[35283]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:00:07.966 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:00:07.966 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:00:07.966 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:09.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:09 smithi043 ceph-mon[31531]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:09.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:09 smithi179 ceph-mon[35888]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:10.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:09 smithi132 ceph-mon[35283]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:11 smithi043 ceph-mon[31531]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:11 smithi043 ceph-mon[31531]: from='client.? 172.21.15.132:0/3643504766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8749b216-d75b-4f50-be38-9d9c7c67fe57"}]: dispatch 2024-05-27T07:00:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:11 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8749b216-d75b-4f50-be38-9d9c7c67fe57"}]: dispatch 2024-05-27T07:00:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:11 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8749b216-d75b-4f50-be38-9d9c7c67fe57"}]': finished 2024-05-27T07:00:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:11 smithi043 ceph-mon[31531]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:00:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:11 smithi179 ceph-mon[35888]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:11 smithi179 ceph-mon[35888]: from='client.? 172.21.15.132:0/3643504766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8749b216-d75b-4f50-be38-9d9c7c67fe57"}]: dispatch 2024-05-27T07:00:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:11 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8749b216-d75b-4f50-be38-9d9c7c67fe57"}]: dispatch 2024-05-27T07:00:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:11 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8749b216-d75b-4f50-be38-9d9c7c67fe57"}]': finished 2024-05-27T07:00:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:11 smithi179 ceph-mon[35888]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:00:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:11 smithi132 ceph-mon[35283]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:11 smithi132 ceph-mon[35283]: from='client.? 172.21.15.132:0/3643504766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8749b216-d75b-4f50-be38-9d9c7c67fe57"}]: dispatch 2024-05-27T07:00:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:11 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8749b216-d75b-4f50-be38-9d9c7c67fe57"}]: dispatch 2024-05-27T07:00:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:11 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8749b216-d75b-4f50-be38-9d9c7c67fe57"}]': finished 2024-05-27T07:00:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:11 smithi132 ceph-mon[35283]: osdmap e10: 2 total, 1 up, 2 in 2024-05-27T07:00:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:12.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:12 smithi043 ceph-mon[31531]: from='client.? 172.21.15.132:0/4087246473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:00:12.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:12 smithi179 ceph-mon[35888]: from='client.? 172.21.15.132:0/4087246473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:00:12.994 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:12 smithi132 ceph-mon[35283]: from='client.? 172.21.15.132:0/4087246473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:00:13.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:13 smithi043 ceph-mon[31531]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:13.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:13 smithi179 ceph-mon[35888]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:14.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:13 smithi132 ceph-mon[35283]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:15.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:15 smithi132 ceph-mon[35283]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:15.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:15 smithi043 ceph-mon[31531]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:15 smithi179 ceph-mon[35888]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:17.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:17 smithi043 ceph-mon[31531]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:17 smithi179 ceph-mon[35888]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:18.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:17 smithi132 ceph-mon[35283]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:19.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:19 smithi043 ceph-mon[31531]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:19.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:19 smithi179 ceph-mon[35888]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:20.088 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:19 smithi132 ceph-mon[35283]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:20.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:00:20.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:20.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:00:20.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:20.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-27T07:00:20.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:21 smithi132 ceph-mon[35283]: Deploying daemon osd.1 on smithi132 2024-05-27T07:00:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:21 smithi132 ceph-mon[35283]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:21 smithi043 ceph-mon[31531]: Deploying daemon osd.1 on smithi132 2024-05-27T07:00:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:21 smithi043 ceph-mon[31531]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:21 smithi179 ceph-mon[35888]: Deploying daemon osd.1 on smithi132 2024-05-27T07:00:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:21 smithi179 ceph-mon[35888]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:23.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:23 smithi043 ceph-mon[31531]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:23 smithi179 ceph-mon[35888]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:24.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:23 smithi132 ceph-mon[35283]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:25.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:25 smithi132 ceph-mon[35283]: pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:25.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:25 smithi043 ceph-mon[31531]: pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:25.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:25 smithi179 ceph-mon[35888]: pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:27.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:27 smithi043 ceph-mon[31531]: pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:27.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:27 smithi179 ceph-mon[35888]: pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:28.024 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:27 smithi132 ceph-mon[35283]: pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:29.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:29 smithi132 ceph-mon[35283]: pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:29.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:29 smithi043 ceph-mon[31531]: pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:29 smithi179 ceph-mon[35888]: pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:31.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:31.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:31.009 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:00:31.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:31.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:31.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:00:31.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:31.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:31.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:00:32.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:31 smithi132 ceph-mon[35283]: pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:32.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:31 smithi043 ceph-mon[31531]: pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:31 smithi179 ceph-mon[35888]: pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:33 smithi132 ceph-mon[35283]: pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:00:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:33 smithi043 ceph-mon[31531]: pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:00:34.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:33 smithi179 ceph-mon[35888]: pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:00:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:35.085 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:34 smithi132 ceph-mon[35283]: from='osd.1 [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:00:35.085 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:34 smithi132 ceph-mon[35283]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:00:35.111 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:34 smithi179 ceph-mon[35888]: from='osd.1 [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:00:35.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:34 smithi179 ceph-mon[35888]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:00:35.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:34 smithi043 ceph-mon[31531]: from='osd.1 [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:00:35.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:34 smithi043 ceph-mon[31531]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-27T07:00:35.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.535+0000 7fe5aa000700 1 -- 172.21.15.132:0/2344852002 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe5ac063bf0 con 0x7fe5940b2870 2024-05-27T07:00:35.529 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 1 on host 'smithi132' 2024-05-27T07:00:35.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.539+0000 7fe5b4200700 1 -- 172.21.15.132:0/2344852002 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5940b2870 msgr2=0x7fe5940b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:35.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.539+0000 7fe5b4200700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5940b2870 0x7fe5940b4d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0003820 tx=0x7fe5a001a040 comp rx=0 tx=0).stop 2024-05-27T07:00:35.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.539+0000 7fe5b4200700 1 -- 172.21.15.132:0/2344852002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5ac107670 msgr2=0x7fe5ac19cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:35.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.539+0000 7fe5b4200700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5ac107670 0x7fe5ac19cf90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe59800d950 tx=0x7fe59800dc60 comp rx=0 tx=0).stop 2024-05-27T07:00:35.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.540+0000 7fe5b4200700 1 -- 172.21.15.132:0/2344852002 shutdown_connections 2024-05-27T07:00:35.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.540+0000 7fe5b4200700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5940b2870 0x7fe5940b4d80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:35.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.540+0000 7fe5b4200700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5ac108010 0x7fe5ac1a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:35.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.540+0000 7fe5b4200700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5ac106540 0x7fe5ac19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:35.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.540+0000 7fe5b4200700 1 --2- 172.21.15.132:0/2344852002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5ac107670 0x7fe5ac19cf90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:35.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.540+0000 7fe5b4200700 1 -- 172.21.15.132:0/2344852002 >> 172.21.15.132:0/2344852002 conn(0x7fe5ac101b80 msgr2=0x7fe5ac10a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:35.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.540+0000 7fe5b4200700 1 -- 172.21.15.132:0/2344852002 shutdown_connections 2024-05-27T07:00:35.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:00:35.540+0000 7fe5b4200700 1 -- 172.21.15.132:0/2344852002 wait complete. 2024-05-27T07:00:36.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:35 smithi043 ceph-mon[31531]: pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:36.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:35 smithi043 ceph-mon[31531]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:00:36.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:35 smithi043 ceph-mon[31531]: from='osd.1 [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:00:36.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:35 smithi043 ceph-mon[31531]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:00:36.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:36.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:35 smithi043 ceph-mon[31531]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:00:36.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:36.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:36.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:00:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:35 smithi179 ceph-mon[35888]: pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:35 smithi179 ceph-mon[35888]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:00:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:35 smithi179 ceph-mon[35888]: from='osd.1 [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:00:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:35 smithi179 ceph-mon[35888]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:00:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:35 smithi179 ceph-mon[35888]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:00:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:36.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:00:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:35 smithi132 ceph-mon[35283]: pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:35 smithi132 ceph-mon[35283]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-27T07:00:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:35 smithi132 ceph-mon[35283]: from='osd.1 [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:00:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:35 smithi132 ceph-mon[35283]: osdmap e11: 2 total, 1 up, 2 in 2024-05-27T07:00:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:36.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:35 smithi132 ceph-mon[35283]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:00:36.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:36.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:36.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:00:36.434 DEBUG:teuthology.orchestra.run.smithi132:osd.1> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.1.service 2024-05-27T07:00:36.439 INFO:tasks.cephadm:Deploying osd.2 on smithi179 with /dev/vg_nvme/lv_4... 2024-05-27T07:00:36.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-27T07:00:36.872 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:00:37.182 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:36 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:00:37.182 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:36 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:00:37.182 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:36 smithi179 ceph-mon[35888]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:00:37.182 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:36 smithi179 ceph-mon[35888]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:00:37.182 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:36 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:37.182 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:36 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:37.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:36 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:00:37.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:36 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:00:37.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:36 smithi043 ceph-mon[31531]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:00:37.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:36 smithi043 ceph-mon[31531]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:00:37.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:36 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:37.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:36 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:37.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:36 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:00:37.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:36 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:00:37.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:36 smithi132 ceph-mon[35283]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:00:37.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:36 smithi132 ceph-mon[35283]: osdmap e12: 2 total, 1 up, 2 in 2024-05-27T07:00:37.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:36 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:37.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:36 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:38.062 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:37 smithi179 ceph-mon[35888]: pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:38.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:37 smithi179 ceph-mon[35888]: osd.1 [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] boot 2024-05-27T07:00:38.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:37 smithi179 ceph-mon[35888]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:00:38.063 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:37 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:38.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:37 smithi132 ceph-mon[35283]: pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:38.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:37 smithi132 ceph-mon[35283]: osd.1 [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] boot 2024-05-27T07:00:38.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:37 smithi132 ceph-mon[35283]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:00:38.145 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:37 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:38.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:37 smithi043 ceph-mon[31531]: pgmap v84: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-27T07:00:38.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:37 smithi043 ceph-mon[31531]: osd.1 [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] boot 2024-05-27T07:00:38.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:37 smithi043 ceph-mon[31531]: osdmap e13: 2 total, 2 up, 2 in 2024-05-27T07:00:38.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:37 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:00:38.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:38 smithi179 ceph-mon[35888]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:00:39.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:38 smithi132 ceph-mon[35283]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:00:39.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:38 smithi043 ceph-mon[31531]: osdmap e14: 2 total, 2 up, 2 in 2024-05-27T07:00:39.553 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-27T07:00:39.553 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-27T07:00:39.554 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-05-27T07:00:39.554 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-05-27T07:00:39.554 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0332113 s, 316 MB/s 2024-05-27T07:00:39.554 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-05-27T07:00:40.017 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:39 smithi132 ceph-mon[35283]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:39 smithi043 ceph-mon[31531]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:39 smithi179 ceph-mon[35888]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:40.289 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_4 2024-05-27T07:00:40.711 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:00:41.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:41 smithi179 ceph-mon[35888]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:42.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:41 smithi132 ceph-mon[35283]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:42.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:41 smithi043 ceph-mon[31531]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:42.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.820+0000 7ffaf0a00700 1 -- 172.21.15.179:0/3547058292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaec104160 msgr2=0x7ffaec104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:42.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.820+0000 7ffaf0a00700 1 --2- 172.21.15.179:0/3547058292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaec104160 0x7ffaec104570 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009990 tx=0x7ffadc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:00:42.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.820+0000 7ffaf0a00700 1 -- 172.21.15.179:0/3547058292 shutdown_connections 2024-05-27T07:00:42.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.820+0000 7ffaf0a00700 1 --2- 172.21.15.179:0/3547058292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffaec105c30 0x7ffaec10c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:42.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.820+0000 7ffaf0a00700 1 --2- 172.21.15.179:0/3547058292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaec104160 0x7ffaec104570 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:42.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.820+0000 7ffaf0a00700 1 --2- 172.21.15.179:0/3547058292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaec105290 0x7ffaec1056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:42.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.820+0000 7ffaf0a00700 1 -- 172.21.15.179:0/3547058292 >> 172.21.15.179:0/3547058292 conn(0x7ffaec0ff7a0 msgr2=0x7ffaec101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:42.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.820+0000 7ffaf0a00700 1 -- 172.21.15.179:0/3547058292 shutdown_connections 2024-05-27T07:00:42.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.820+0000 7ffaf0a00700 1 -- 172.21.15.179:0/3547058292 wait complete. 2024-05-27T07:00:42.814 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.821+0000 7ffaf0a00700 1 Processor -- start 2024-05-27T07:00:42.814 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffaf0a00700 1 -- start start 2024-05-27T07:00:42.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffaf0a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaec104160 0x7ffaec077450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:42.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffaf0a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffaec105290 0x7ffaec075b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:42.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffaf0a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaec105c30 0x7ffaec076040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:42.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffaf0a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec10fa60 con 0x7ffaec105290 2024-05-27T07:00:42.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffaf0a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec10f8e0 con 0x7ffaec104160 2024-05-27T07:00:42.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffaf0a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec10fbe0 con 0x7ffaec105c30 2024-05-27T07:00:42.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffaeb200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaec105c30 0x7ffaec076040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:42.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffaeb200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaec105c30 0x7ffaec076040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:33786/0 (socket says 172.21.15.179:33786) 2024-05-27T07:00:42.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffaeb200700 1 -- 172.21.15.179:0/2228212538 learned_addr learned my addr 172.21.15.179:0/2228212538 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-05-27T07:00:42.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.822+0000 7ffae9e00700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffaec105290 0x7ffaec075b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:42.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.823+0000 7ffaea800700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaec104160 0x7ffaec077450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:42.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.823+0000 7ffaeb200700 1 -- 172.21.15.179:0/2228212538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaec104160 msgr2=0x7ffaec077450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:42.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.823+0000 7ffaeb200700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaec104160 0x7ffaec077450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:42.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.823+0000 7ffaeb200700 1 -- 172.21.15.179:0/2228212538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffaec105290 msgr2=0x7ffaec075b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:42.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.823+0000 7ffaeb200700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffaec105290 0x7ffaec075b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:42.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.823+0000 7ffaeb200700 1 -- 172.21.15.179:0/2228212538 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaec0767a0 con 0x7ffaec105c30 2024-05-27T07:00:42.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.823+0000 7ffaeb200700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaec105c30 0x7ffaec076040 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffae400c370 tx=0x7ffae400c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:42.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.824+0000 7ffae3400700 1 -- 172.21.15.179:0/2228212538 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae400ce20 con 0x7ffaec105c30 2024-05-27T07:00:42.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.824+0000 7ffae3400700 1 -- 172.21.15.179:0/2228212538 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffae40090d0 con 0x7ffaec105c30 2024-05-27T07:00:42.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.824+0000 7ffae3400700 1 -- 172.21.15.179:0/2228212538 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4004700 con 0x7ffaec105c30 2024-05-27T07:00:42.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.824+0000 7ffaf0a00700 1 -- 172.21.15.179:0/2228212538 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaec1a4790 con 0x7ffaec105c30 2024-05-27T07:00:42.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.824+0000 7ffaf0a00700 1 -- 172.21.15.179:0/2228212538 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaec1a4bf0 con 0x7ffaec105c30 2024-05-27T07:00:42.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.825+0000 7ffaf0a00700 1 -- 172.21.15.179:0/2228212538 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaec0512b0 con 0x7ffaec105c30 2024-05-27T07:00:42.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.828+0000 7ffae3400700 1 -- 172.21.15.179:0/2228212538 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7ffae401a020 con 0x7ffaec105c30 2024-05-27T07:00:42.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.829+0000 7ffae3400700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ffad00b2870 0x7ffad00b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:42.822 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.829+0000 7ffae3400700 1 -- 172.21.15.179:0/2228212538 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7ffae40e0a00 con 0x7ffaec105c30 2024-05-27T07:00:42.822 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.830+0000 7ffaea800700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ffad00b2870 0x7ffad00b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:42.823 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.830+0000 7ffaea800700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ffad00b2870 0x7ffad00b4d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00b650 tx=0x7ffadc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:42.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:42.834+0000 7ffae3400700 1 -- 172.21.15.179:0/2228212538 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ffae4010040 con 0x7ffaec105c30 2024-05-27T07:00:43.069 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:00:43.076+0000 7ffaf0a00700 1 -- 172.21.15.179:0/2228212538 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ffaec063bf0 con 0x7ffad00b2870 2024-05-27T07:00:43.949 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:43.949 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi132 2024-05-27T07:00:43.949 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:43.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:43.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:00:43.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi132 to 17089M 2024-05-27T07:00:43.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:43.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:43.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:00:43.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:43.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:00:43.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:00:43.950 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi132 2024-05-27T07:00:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:00:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 17089M 2024-05-27T07:00:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:43.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:43.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:00:43.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:43.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:00:43.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:00:43.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi132 2024-05-27T07:00:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:00:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi132 to 17089M 2024-05-27T07:00:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:00:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:00:44.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:00:44.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:00:44.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:44.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:44 smithi043 ceph-mon[31531]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:00:44.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:44 smithi179 ceph-mon[35888]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:00:45.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:44 smithi132 ceph-mon[35283]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:00:45.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:45 smithi043 ceph-mon[31531]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:45.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:45 smithi179 ceph-mon[35888]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:46.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:45 smithi132 ceph-mon[35283]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:47.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:47 smithi043 ceph-mon[31531]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:47.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:47 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/796679699' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97582b6e-2b42-48b6-bf39-511a1121e069"}]: dispatch 2024-05-27T07:00:47.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:47 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97582b6e-2b42-48b6-bf39-511a1121e069"}]: dispatch 2024-05-27T07:00:47.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:47 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97582b6e-2b42-48b6-bf39-511a1121e069"}]': finished 2024-05-27T07:00:47.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:47 smithi043 ceph-mon[31531]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:00:47.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:00:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:47 smithi179 ceph-mon[35888]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:47 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/796679699' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97582b6e-2b42-48b6-bf39-511a1121e069"}]: dispatch 2024-05-27T07:00:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:47 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97582b6e-2b42-48b6-bf39-511a1121e069"}]: dispatch 2024-05-27T07:00:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:47 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97582b6e-2b42-48b6-bf39-511a1121e069"}]': finished 2024-05-27T07:00:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:47 smithi179 ceph-mon[35888]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:00:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:00:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:47 smithi132 ceph-mon[35283]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:47 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/796679699' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97582b6e-2b42-48b6-bf39-511a1121e069"}]: dispatch 2024-05-27T07:00:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:47 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97582b6e-2b42-48b6-bf39-511a1121e069"}]: dispatch 2024-05-27T07:00:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:47 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97582b6e-2b42-48b6-bf39-511a1121e069"}]': finished 2024-05-27T07:00:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:47 smithi132 ceph-mon[35283]: osdmap e15: 3 total, 2 up, 3 in 2024-05-27T07:00:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:00:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:48 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/298440927' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:00:49.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:48 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/298440927' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:00:49.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:48 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/298440927' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:00:50.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:49 smithi132 ceph-mon[35283]: pgmap v93: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:50.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:49 smithi043 ceph-mon[31531]: pgmap v93: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:50.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:49 smithi179 ceph-mon[35888]: pgmap v93: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:51.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:51 smithi179 ceph-mon[35888]: pgmap v94: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:52.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:51 smithi132 ceph-mon[35283]: pgmap v94: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:52.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:51 smithi043 ceph-mon[31531]: pgmap v94: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:53.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:53 smithi179 ceph-mon[35888]: pgmap v95: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:54.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:53 smithi132 ceph-mon[35283]: pgmap v95: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:54.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:53 smithi043 ceph-mon[31531]: pgmap v95: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:55.936 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:55 smithi179 ceph-mon[35888]: pgmap v96: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:56.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:55 smithi132 ceph-mon[35283]: pgmap v96: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:55 smithi043 ceph-mon[31531]: pgmap v96: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:57.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:56 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:00:57.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:56 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:57.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:56 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:00:57.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:56 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:57.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:56 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-27T07:00:57.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:56 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:00:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:57 smithi179 ceph-mon[35888]: Deploying daemon osd.2 on smithi179 2024-05-27T07:00:57.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:57 smithi179 ceph-mon[35888]: pgmap v97: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:57 smithi132 ceph-mon[35283]: Deploying daemon osd.2 on smithi179 2024-05-27T07:00:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:57 smithi132 ceph-mon[35283]: pgmap v97: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:57 smithi043 ceph-mon[31531]: Deploying daemon osd.2 on smithi179 2024-05-27T07:00:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:57 smithi043 ceph-mon[31531]: pgmap v97: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:00:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:00:59 smithi179 ceph-mon[35888]: pgmap v98: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:00.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:00:59 smithi132 ceph-mon[35283]: pgmap v98: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:00.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:00:59 smithi043 ceph-mon[31531]: pgmap v98: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:01.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:01 smithi132 ceph-mon[35283]: pgmap v99: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:01.988 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:01 smithi179 ceph-mon[35888]: pgmap v99: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:02.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:01 smithi043 ceph-mon[31531]: pgmap v99: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:04.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:03 smithi132 ceph-mon[35283]: pgmap v100: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:04.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:03 smithi043 ceph-mon[31531]: pgmap v100: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:04.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:03 smithi179 ceph-mon[35888]: pgmap v100: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:05.994 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:05 smithi179 ceph-mon[35888]: pgmap v101: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:05.994 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:05.994 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:05.994 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:06.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:05 smithi132 ceph-mon[35283]: pgmap v101: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:06.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:06.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:06.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:06.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:05 smithi043 ceph-mon[31531]: pgmap v101: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:06.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:06.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:06.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:08.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:07 smithi132 ceph-mon[35283]: pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:08.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:07 smithi043 ceph-mon[31531]: pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:07 smithi179 ceph-mon[35888]: pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:09 smithi179 ceph-mon[35888]: pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:09 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:09 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:09 smithi179 ceph-mon[35888]: from='osd.2 [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:01:09.921 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:09 smithi179 ceph-mon[35888]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:01:10.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:09 smithi043 ceph-mon[31531]: pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:10.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:09 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:10.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:09 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:10.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:09 smithi043 ceph-mon[31531]: from='osd.2 [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:01:10.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:09 smithi043 ceph-mon[31531]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:01:10.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:09 smithi132 ceph-mon[35283]: pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:10.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:09 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:10.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:09 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:10.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:09 smithi132 ceph-mon[35283]: from='osd.2 [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:01:10.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:09 smithi132 ceph-mon[35283]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-27T07:01:10.795 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.802+0000 7ffae3400700 1 -- 172.21.15.179:0/2228212538 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffaec063bf0 con 0x7ffad00b2870 2024-05-27T07:01:10.795 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 2 on host 'smithi179' 2024-05-27T07:01:10.798 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.806+0000 7ffaf0a00700 1 -- 172.21.15.179:0/2228212538 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ffad00b2870 msgr2=0x7ffad00b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:10.798 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.806+0000 7ffaf0a00700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ffad00b2870 0x7ffad00b4d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00b650 tx=0x7ffadc005fb0 comp rx=0 tx=0).stop 2024-05-27T07:01:10.798 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.806+0000 7ffaf0a00700 1 -- 172.21.15.179:0/2228212538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaec105c30 msgr2=0x7ffaec076040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:10.798 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.806+0000 7ffaf0a00700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaec105c30 0x7ffaec076040 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffae400c370 tx=0x7ffae400c680 comp rx=0 tx=0).stop 2024-05-27T07:01:10.799 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.807+0000 7ffaf0a00700 1 -- 172.21.15.179:0/2228212538 shutdown_connections 2024-05-27T07:01:10.799 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.807+0000 7ffaf0a00700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ffad00b2870 0x7ffad00b4d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:10.799 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.807+0000 7ffaf0a00700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaec105c30 0x7ffaec076040 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:10.799 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.807+0000 7ffaf0a00700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaec104160 0x7ffaec077450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:10.799 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.807+0000 7ffaf0a00700 1 --2- 172.21.15.179:0/2228212538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffaec105290 0x7ffaec075b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:10.799 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.807+0000 7ffaf0a00700 1 -- 172.21.15.179:0/2228212538 >> 172.21.15.179:0/2228212538 conn(0x7ffaec0ff7a0 msgr2=0x7ffaec1084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:10.799 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.807+0000 7ffaf0a00700 1 -- 172.21.15.179:0/2228212538 shutdown_connections 2024-05-27T07:01:10.799 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:01:10.807+0000 7ffaf0a00700 1 -- 172.21.15.179:0/2228212538 wait complete. 2024-05-27T07:01:10.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:01:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:01:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: from='osd.2 [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:01:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:01:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:11.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:01:11.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:01:11.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:11.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: from='osd.2 [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:01:11.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:01:11.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:11.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:11.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:11.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:11.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:11.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:11.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-27T07:01:11.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: osdmap e16: 3 total, 2 up, 3 in 2024-05-27T07:01:11.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:11.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: from='osd.2 [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:01:11.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:01:11.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:11.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:11.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:11.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:11.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:11.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:11.613 DEBUG:teuthology.orchestra.run.smithi179:osd.2> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.2.service 2024-05-27T07:01:11.618 INFO:tasks.cephadm:Deploying osd.3 on smithi043 with /dev/vg_nvme/lv_3... 2024-05-27T07:01:11.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:01:11.888 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:01:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2[40928]: 2024-05-27T07:01:11.762+0000 7fceb5400700 -1 osd.2 0 waiting for initial osdmap 2024-05-27T07:01:11.888 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:01:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2[40928]: 2024-05-27T07:01:11.784+0000 7fceab200700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:01:11.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:11 smithi043 ceph-mon[31531]: pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:11 smithi043 ceph-mon[31531]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:01:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:11 smithi043 ceph-mon[31531]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:01:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:11 smithi043 ceph-mon[31531]: from='osd.2 ' entity='osd.2' 2024-05-27T07:01:12.096 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:01:12.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:11 smithi179 ceph-mon[35888]: pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:11 smithi179 ceph-mon[35888]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:01:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:11 smithi179 ceph-mon[35888]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:01:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:12.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:11 smithi179 ceph-mon[35888]: from='osd.2 ' entity='osd.2' 2024-05-27T07:01:12.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:11 smithi132 ceph-mon[35283]: pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:12.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:11 smithi132 ceph-mon[35283]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:01:12.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:11 smithi132 ceph-mon[35283]: osdmap e17: 3 total, 2 up, 3 in 2024-05-27T07:01:12.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:12.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:12.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:11 smithi132 ceph-mon[35283]: from='osd.2 ' entity='osd.2' 2024-05-27T07:01:13.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:12 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:01:13.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:12 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:01:13.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:13.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:12 smithi043 ceph-mon[31531]: osd.2 [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] boot 2024-05-27T07:01:13.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:12 smithi043 ceph-mon[31531]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:01:13.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:12 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:01:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:12 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:01:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:12 smithi179 ceph-mon[35888]: osd.2 [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] boot 2024-05-27T07:01:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:12 smithi179 ceph-mon[35888]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:01:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:13.244 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:12 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:01:13.244 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:12 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:01:13.244 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:13.244 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:12 smithi132 ceph-mon[35283]: osd.2 [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] boot 2024-05-27T07:01:13.244 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:12 smithi132 ceph-mon[35283]: osdmap e18: 3 total, 3 up, 3 in 2024-05-27T07:01:13.244 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:01:14.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:13 smithi043 ceph-mon[31531]: pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:14.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:13 smithi179 ceph-mon[35888]: pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:14.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:13 smithi132 ceph-mon[35283]: pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-27T07:01:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:01:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:01:14.847 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-05-27T07:01:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-05-27T07:01:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0391557 s, 268 MB/s 2024-05-27T07:01:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-05-27T07:01:15.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:14 smithi043 ceph-mon[31531]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:01:15.105 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:14 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:01:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:14 smithi179 ceph-mon[35888]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:01:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:14 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:01:15.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:14 smithi132 ceph-mon[35283]: osdmap e19: 3 total, 3 up, 3 in 2024-05-27T07:01:15.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:14 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:01:15.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_3 2024-05-27T07:01:16.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:15 smithi043 ceph-mon[31531]: pgmap v110: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:16.100 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:01:16.100 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:15 smithi043 ceph-mon[31531]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:01:16.100 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:01:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:01:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:15 smithi179 ceph-mon[35888]: pgmap v110: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:01:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:15 smithi179 ceph-mon[35888]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:01:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:01:16.275 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:15 smithi132 ceph-mon[35283]: pgmap v110: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:16.276 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-27T07:01:16.276 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:15 smithi132 ceph-mon[35283]: osdmap e20: 3 total, 3 up, 3 in 2024-05-27T07:01:16.276 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-27T07:01:16.443 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48182]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:01:16.443 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48182]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:16.443 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48182]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:16.443 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48182]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:16.443 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48198]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:01:16.443 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48198]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:16.443 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48198]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:16.443 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48198]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:16.538 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44576]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:01:16.538 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44576]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:16.539 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44576]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:16.539 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44576]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:16.592 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44026]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:01:16.592 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44026]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:16.592 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44026]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:16.592 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44026]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:16.592 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44029]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:01:16.592 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44029]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:16.592 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44029]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:16.593 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44029]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:16.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48260]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:01:16.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48260]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:16.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48260]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:16.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48260]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:16.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48269]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:01:16.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48269]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:16.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48269]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:16.792 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44608]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:01:16.792 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44608]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:16.792 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44608]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:16.792 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44608]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:16.792 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44579]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:01:16.793 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44579]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:16.793 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44579]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:16.793 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44579]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:16.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 sudo[48269]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44617]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:01:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44617]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44617]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:17.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 sudo[44617]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:17.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:01:17.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:01:17.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:01:17.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:17.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:17.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:01:17.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:01:17.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:01:17.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:01:17.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:01:17.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:01:17.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:01:17.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:01:17.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:01:17.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:01:17.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:01:17.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:01:17.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:01:17.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:01:17.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:16 smithi179 ceph-mon[35888]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:01:17.333 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:01:17.333 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:01:17.333 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:01:17.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:01:17.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:16 smithi043 ceph-mon[31531]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:01:17.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44064]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-27T07:01:17.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44064]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:17.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44064]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:17.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44064]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:17.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44067]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-27T07:01:17.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44067]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-27T07:01:17.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44067]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 sudo[44067]: pam_unix(sudo:session): session closed for user root 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: osdmap e21: 3 total, 3 up, 3 in 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:01:17.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:01:17.346 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:16 smithi132 ceph-mon[35283]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-27T07:01:18.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.247+0000 7fba7b000700 1 -- 172.21.15.43:0/3467490345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7410c420 msgr2=0x7fba74112cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:18.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.247+0000 7fba7b000700 1 --2- 172.21.15.43:0/3467490345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7410c420 0x7fba74112cd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00b0a0 tx=0x7fba6c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:01:18.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.247+0000 7fba7b000700 1 -- 172.21.15.43:0/3467490345 shutdown_connections 2024-05-27T07:01:18.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.247+0000 7fba7b000700 1 --2- 172.21.15.43:0/3467490345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7410c420 0x7fba74112cd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:18.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.247+0000 7fba7b000700 1 --2- 172.21.15.43:0/3467490345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba7410a950 0x7fba7410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:18.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.247+0000 7fba7b000700 1 --2- 172.21.15.43:0/3467490345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba7410ba80 0x7fba7410be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:18.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.247+0000 7fba7b000700 1 -- 172.21.15.43:0/3467490345 >> 172.21.15.43:0/3467490345 conn(0x7fba740799c0 msgr2=0x7fba741085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:18.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.247+0000 7fba7b000700 1 -- 172.21.15.43:0/3467490345 shutdown_connections 2024-05-27T07:01:18.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.247+0000 7fba7b000700 1 -- 172.21.15.43:0/3467490345 wait complete. 2024-05-27T07:01:18.242 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:17 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi179 2024-05-27T07:01:18.242 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:17 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi179 to 17089M 2024-05-27T07:01:18.242 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:17 smithi043 ceph-mon[31531]: pgmap v113: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:18.242 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:17 smithi043 ceph-mon[31531]: osdmap e22: 3 total, 3 up, 3 in 2024-05-27T07:01:18.242 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:17 smithi043 ceph-mon[31531]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-05-27T07:01:18.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.249+0000 7fba7b000700 1 Processor -- start 2024-05-27T07:01:18.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.249+0000 7fba7b000700 1 -- start start 2024-05-27T07:01:18.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba7b000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba7410a950 0x7fba741a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:18.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba7b000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7410ba80 0x7fba741a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:18.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba7b000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba7410c420 0x7fba741a82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:18.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba7b000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fba741162e0 con 0x7fba7410ba80 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba7b000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fba74116160 con 0x7fba7410c420 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba7b000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fba74116460 con 0x7fba7410a950 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7410ba80 0x7fba741a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7410ba80 0x7fba741a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50176/0 (socket says 172.21.15.43:50176) 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba73400700 1 -- 172.21.15.43:0/2689093562 learned_addr learned my addr 172.21.15.43:0/2689093562 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba73e00700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba7410a950 0x7fba741a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.250+0000 7fba79000700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba7410c420 0x7fba741a82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.251+0000 7fba73400700 1 -- 172.21.15.43:0/2689093562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba7410a950 msgr2=0x7fba741a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.251+0000 7fba73400700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba7410a950 0x7fba741a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.251+0000 7fba73400700 1 -- 172.21.15.43:0/2689093562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba7410c420 msgr2=0x7fba741a82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.251+0000 7fba73400700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba7410c420 0x7fba741a82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:18.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.251+0000 7fba73400700 1 -- 172.21.15.43:0/2689093562 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba741a8950 con 0x7fba7410ba80 2024-05-27T07:01:18.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.251+0000 7fba73400700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7410ba80 0x7fba741a1330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fba6000d950 tx=0x7fba6000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:18.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.251+0000 7fba70c00700 1 -- 172.21.15.43:0/2689093562 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60016070 con 0x7fba7410ba80 2024-05-27T07:01:18.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.251+0000 7fba70c00700 1 -- 172.21.15.43:0/2689093562 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba60010460 con 0x7fba7410ba80 2024-05-27T07:01:18.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.251+0000 7fba7b000700 1 -- 172.21.15.43:0/2689093562 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba741a8c40 con 0x7fba7410ba80 2024-05-27T07:01:18.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.251+0000 7fba70c00700 1 -- 172.21.15.43:0/2689093562 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6000f620 con 0x7fba7410ba80 2024-05-27T07:01:18.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.252+0000 7fba7b000700 1 -- 172.21.15.43:0/2689093562 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba741a9180 con 0x7fba7410ba80 2024-05-27T07:01:18.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.253+0000 7fba7b000700 1 -- 172.21.15.43:0/2689093562 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba740512b0 con 0x7fba7410ba80 2024-05-27T07:01:18.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.255+0000 7fba70c00700 1 -- 172.21.15.43:0/2689093562 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fba60019070 con 0x7fba7410ba80 2024-05-27T07:01:18.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.257+0000 7fba70c00700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fba580b2900 0x7fba580b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:18.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.257+0000 7fba70c00700 1 -- 172.21.15.43:0/2689093562 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 3237+0+0 (secure 0 0 0) 0x7fba600e0be0 con 0x7fba7410ba80 2024-05-27T07:01:18.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.257+0000 7fba73e00700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fba580b2900 0x7fba580b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:18.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.258+0000 7fba73e00700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fba580b2900 0x7fba580b4e10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fba64005950 tx=0x7fba640058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:18.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.260+0000 7fba70c00700 1 -- 172.21.15.43:0/2689093562 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fba600a9c30 con 0x7fba7410ba80 2024-05-27T07:01:18.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:17 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi179 2024-05-27T07:01:18.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:17 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi179 to 17089M 2024-05-27T07:01:18.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:17 smithi132 ceph-mon[35283]: pgmap v113: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:18.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:17 smithi132 ceph-mon[35283]: osdmap e22: 3 total, 3 up, 3 in 2024-05-27T07:01:18.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:17 smithi132 ceph-mon[35283]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-05-27T07:01:18.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:17 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi179 2024-05-27T07:01:18.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:17 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi179 to 17089M 2024-05-27T07:01:18.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:17 smithi179 ceph-mon[35888]: pgmap v113: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:18.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:17 smithi179 ceph-mon[35888]: osdmap e22: 3 total, 3 up, 3 in 2024-05-27T07:01:18.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:17 smithi179 ceph-mon[35888]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-05-27T07:01:18.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:18.510+0000 7fba7b000700 1 -- 172.21.15.43:0/2689093562 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fba74063cb0 con 0x7fba580b2900 2024-05-27T07:01:19.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:18 smithi132 ceph-mon[35283]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:01:19.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:18 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:01:19.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:18 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:01:19.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:18 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:19.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:18 smithi132 ceph-mon[35283]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:19.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:18 smithi043 ceph-mon[31531]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:01:19.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:18 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:01:19.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:18 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:01:19.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:18 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:19.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:18 smithi043 ceph-mon[31531]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:18 smithi179 ceph-mon[35888]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:01:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:18 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:01:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:18 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:01:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:18 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:19.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:18 smithi179 ceph-mon[35888]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:21 smithi043 ceph-mon[31531]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:21 smithi179 ceph-mon[35888]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:21 smithi132 ceph-mon[35283]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:23.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:23 smithi043 ceph-mon[31531]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:23.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:23 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3784910676' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8247fd20-bc21-430a-a3b9-bd5221468995"}]: dispatch 2024-05-27T07:01:23.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:23 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3784910676' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8247fd20-bc21-430a-a3b9-bd5221468995"}]': finished 2024-05-27T07:01:23.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:23 smithi043 ceph-mon[31531]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:01:23.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:23 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:23 smithi179 ceph-mon[35888]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:23 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3784910676' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8247fd20-bc21-430a-a3b9-bd5221468995"}]: dispatch 2024-05-27T07:01:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:23 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3784910676' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8247fd20-bc21-430a-a3b9-bd5221468995"}]': finished 2024-05-27T07:01:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:23 smithi179 ceph-mon[35888]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:01:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:23 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:24.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:23 smithi132 ceph-mon[35283]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:24.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:23 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3784910676' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8247fd20-bc21-430a-a3b9-bd5221468995"}]: dispatch 2024-05-27T07:01:24.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:23 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3784910676' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8247fd20-bc21-430a-a3b9-bd5221468995"}]': finished 2024-05-27T07:01:24.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:23 smithi132 ceph-mon[35283]: osdmap e23: 4 total, 3 up, 4 in 2024-05-27T07:01:24.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:23 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:24.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:24 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3786024871' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:01:24.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:24 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3786024871' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:01:25.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:24 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3786024871' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:01:25.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:25 smithi043 ceph-mon[31531]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:25.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:25 smithi179 ceph-mon[35888]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:26.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:25 smithi132 ceph-mon[35283]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:27.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:27 smithi043 ceph-mon[31531]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:27 smithi179 ceph-mon[35888]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:28.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:27 smithi132 ceph-mon[35283]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:29.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:29 smithi043 ceph-mon[31531]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:29 smithi179 ceph-mon[35888]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:30.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:29 smithi132 ceph-mon[35283]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:31.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:31 smithi043 ceph-mon[31531]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:31.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:31 smithi179 ceph-mon[35888]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:32.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:31 smithi132 ceph-mon[35283]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:32.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:32 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:01:32.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:32 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:32.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:32 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:01:32.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:32 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:33.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:32 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-27T07:01:33.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:32 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:33.769 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:33 smithi043 ceph-mon[31531]: Deploying daemon osd.3 on smithi043 2024-05-27T07:01:33.769 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:33 smithi043 ceph-mon[31531]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:33 smithi179 ceph-mon[35888]: Deploying daemon osd.3 on smithi043 2024-05-27T07:01:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:33 smithi179 ceph-mon[35888]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:33 smithi132 ceph-mon[35283]: Deploying daemon osd.3 on smithi043 2024-05-27T07:01:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:33 smithi132 ceph-mon[35283]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:35.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:35 smithi043 ceph-mon[31531]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:35 smithi179 ceph-mon[35888]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:35 smithi132 ceph-mon[35283]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:37.797 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:37 smithi043 ceph-mon[31531]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:37 smithi179 ceph-mon[35888]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:37 smithi132 ceph-mon[35283]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:39 smithi043 ceph-mon[31531]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:39.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:39 smithi179 ceph-mon[35888]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:39 smithi132 ceph-mon[35283]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:41.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:41 smithi043 ceph-mon[31531]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:41 smithi179 ceph-mon[35888]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:42.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:41 smithi132 ceph-mon[35283]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:43 smithi043 ceph-mon[31531]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:43.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:43.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:43.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:43.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:43 smithi179 ceph-mon[35888]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:43.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:43.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:43.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:43.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:43 smithi132 ceph-mon[35283]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:45.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:45 smithi043 ceph-mon[31531]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:45.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:45 smithi179 ceph-mon[35888]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:46.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:45 smithi132 ceph-mon[35283]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:47.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:46 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:47.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:46 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:47.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:46 smithi132 ceph-mon[35283]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:01:47.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:46 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:47.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:46 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:47.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:46 smithi043 ceph-mon[31531]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:01:47.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:46 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:47.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:46 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:47.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:46 smithi179 ceph-mon[35888]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-27T07:01:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.918+0000 7fba70c00700 1 -- 172.21.15.43:0/2689093562 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fba74063cb0 con 0x7fba580b2900 2024-05-27T07:01:47.910 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 3 on host 'smithi043' 2024-05-27T07:01:47.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.922+0000 7fba7b000700 1 -- 172.21.15.43:0/2689093562 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fba580b2900 msgr2=0x7fba580b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:47.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.922+0000 7fba7b000700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fba580b2900 0x7fba580b4e10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fba64005950 tx=0x7fba640058e0 comp rx=0 tx=0).stop 2024-05-27T07:01:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.922+0000 7fba7b000700 1 -- 172.21.15.43:0/2689093562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7410ba80 msgr2=0x7fba741a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.922+0000 7fba7b000700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7410ba80 0x7fba741a1330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fba6000d950 tx=0x7fba6000dd10 comp rx=0 tx=0).stop 2024-05-27T07:01:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.922+0000 7fba7b000700 1 -- 172.21.15.43:0/2689093562 shutdown_connections 2024-05-27T07:01:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.922+0000 7fba7b000700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fba580b2900 0x7fba580b4e10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.922+0000 7fba7b000700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba7410c420 0x7fba741a82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.922+0000 7fba7b000700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba7410a950 0x7fba741a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.922+0000 7fba7b000700 1 --2- 172.21.15.43:0/2689093562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba7410ba80 0x7fba741a1330 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.923+0000 7fba7b000700 1 -- 172.21.15.43:0/2689093562 >> 172.21.15.43:0/2689093562 conn(0x7fba740799c0 msgr2=0x7fba7410ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.923+0000 7fba7b000700 1 -- 172.21.15.43:0/2689093562 shutdown_connections 2024-05-27T07:01:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:01:47.923+0000 7fba7b000700 1 -- 172.21.15.43:0/2689093562 wait complete. 2024-05-27T07:01:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:47 smithi132 ceph-mon[35283]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:47 smithi132 ceph-mon[35283]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:01:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:47 smithi132 ceph-mon[35283]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:01:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:47 smithi132 ceph-mon[35283]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T07:01:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:48.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:48.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:47 smithi043 ceph-mon[31531]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:47 smithi043 ceph-mon[31531]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:01:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:47 smithi043 ceph-mon[31531]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:01:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:47 smithi043 ceph-mon[31531]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T07:01:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:47 smithi179 ceph-mon[35888]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:47 smithi179 ceph-mon[35888]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-27T07:01:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:47 smithi179 ceph-mon[35888]: osdmap e24: 4 total, 3 up, 4 in 2024-05-27T07:01:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:47 smithi179 ceph-mon[35888]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T07:01:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:48.931 DEBUG:teuthology.orchestra.run.smithi043:osd.3> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.3.service 2024-05-27T07:01:48.937 INFO:tasks.cephadm:Deploying osd.4 on smithi132 with /dev/vg_nvme/lv_3... 2024-05-27T07:01:48.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:01:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:48 smithi132 ceph-mon[35283]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:01:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:48 smithi132 ceph-mon[35283]: osdmap e25: 4 total, 3 up, 4 in 2024-05-27T07:01:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:49.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:49.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:48 smithi043 ceph-mon[31531]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:01:49.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:48 smithi043 ceph-mon[31531]: osdmap e25: 4 total, 3 up, 4 in 2024-05-27T07:01:49.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:49.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:49.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:49.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:49.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:48 smithi179 ceph-mon[35888]: from='osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:01:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:48 smithi179 ceph-mon[35888]: osdmap e25: 4 total, 3 up, 4 in 2024-05-27T07:01:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:01:49.376 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:01:50.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:49 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:01:50.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:49 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:01:50.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:49 smithi132 ceph-mon[35283]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:50.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:49 smithi132 ceph-mon[35283]: osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] boot 2024-05-27T07:01:50.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:49 smithi132 ceph-mon[35283]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:01:50.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:49 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:50.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:49 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:01:50.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:49 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:01:50.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:49 smithi043 ceph-mon[31531]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:50.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:49 smithi043 ceph-mon[31531]: osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] boot 2024-05-27T07:01:50.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:49 smithi043 ceph-mon[31531]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:01:50.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:49 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:49 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:01:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:49 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:01:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:49 smithi179 ceph-mon[35888]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-27T07:01:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:49 smithi179 ceph-mon[35888]: osd.3 [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] boot 2024-05-27T07:01:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:49 smithi179 ceph-mon[35888]: osdmap e26: 4 total, 4 up, 4 in 2024-05-27T07:01:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:49 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:01:51.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:50 smithi132 ceph-mon[35283]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:01:51.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:50 smithi043 ceph-mon[31531]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:01:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:50 smithi179 ceph-mon[35888]: osdmap e27: 4 total, 4 up, 4 in 2024-05-27T07:01:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:51 smithi043 ceph-mon[31531]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:01:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:51 smithi043 ceph-mon[31531]: osdmap e28: 4 total, 4 up, 4 in 2024-05-27T07:01:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:51 smithi179 ceph-mon[35888]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:01:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:51 smithi179 ceph-mon[35888]: osdmap e28: 4 total, 4 up, 4 in 2024-05-27T07:01:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:51 smithi132 ceph-mon[35283]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:01:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:51 smithi132 ceph-mon[35283]: osdmap e28: 4 total, 4 up, 4 in 2024-05-27T07:01:52.362 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:01:52.362 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:01:52.362 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-05-27T07:01:52.362 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-05-27T07:01:52.362 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0318607 s, 329 MB/s 2024-05-27T07:01:52.362 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-05-27T07:01:53.406 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_3 2024-05-27T07:01:53.831 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:01:54.159 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:53 smithi132 ceph-mon[35283]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-05-27T07:01:54.159 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.159 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.159 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:01:54.159 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:01:54.159 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.159 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:54.159 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:54.159 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:53 smithi043 ceph-mon[31531]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-05-27T07:01:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:01:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:01:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:54.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:54.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:53 smithi179 ceph-mon[35888]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-05-27T07:01:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:01:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:01:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:01:54.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:01:55.162 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:54 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi043 2024-05-27T07:01:55.162 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:54 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi043 to 8547M 2024-05-27T07:01:55.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:54 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi043 2024-05-27T07:01:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:54 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi043 to 8547M 2024-05-27T07:01:55.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:54 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi043 2024-05-27T07:01:55.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:54 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi043 to 8547M 2024-05-27T07:01:56.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:55 smithi132 ceph-mon[35283]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:01:56.103 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.112+0000 7f5549000700 1 -- 172.21.15.132:0/1637240415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f554410a760 msgr2=0x7f554410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:56.103 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.112+0000 7f5549000700 1 --2- 172.21.15.132:0/1637240415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f554410a760 0x7f554410ab70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5534009990 tx=0x7f5534009ca0 comp rx=0 tx=0).stop 2024-05-27T07:01:56.103 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.112+0000 7f5549000700 1 -- 172.21.15.132:0/1637240415 shutdown_connections 2024-05-27T07:01:56.103 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.112+0000 7f5549000700 1 --2- 172.21.15.132:0/1637240415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f554410c230 0x7f5544112ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:56.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.112+0000 7f5549000700 1 --2- 172.21.15.132:0/1637240415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f554410a760 0x7f554410ab70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:56.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.112+0000 7f5549000700 1 --2- 172.21.15.132:0/1637240415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f554410b890 0x7f554410bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:56.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.112+0000 7f5549000700 1 -- 172.21.15.132:0/1637240415 >> 172.21.15.132:0/1637240415 conn(0x7f55440797d0 msgr2=0x7f55441085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:56.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.112+0000 7f5549000700 1 -- 172.21.15.132:0/1637240415 shutdown_connections 2024-05-27T07:01:56.104 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.112+0000 7f5549000700 1 -- 172.21.15.132:0/1637240415 wait complete. 2024-05-27T07:01:56.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.113+0000 7f5549000700 1 Processor -- start 2024-05-27T07:01:56.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.113+0000 7f5549000700 1 -- start start 2024-05-27T07:01:56.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5549000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f554410a760 0x7f55441a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:56.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5549000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f554410b890 0x7f55441a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:56.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5549000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f554410c230 0x7f55441a8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:56.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5549000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5544116050 con 0x7f554410a760 2024-05-27T07:01:56.106 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5549000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5544115ed0 con 0x7f554410c230 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5549000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f55441161d0 con 0x7f554410b890 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5543e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f554410c230 0x7f55441a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5543e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f554410c230 0x7f55441a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:38300/0 (socket says 172.21.15.132:38300) 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5543e00700 1 -- 172.21.15.132:0/1569277315 learned_addr learned my addr 172.21.15.132:0/1569277315 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5543400700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f554410a760 0x7f55441a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5542a00700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f554410b890 0x7f55441a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5543e00700 1 -- 172.21.15.132:0/1569277315 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f554410b890 msgr2=0x7f55441a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5543e00700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f554410b890 0x7f55441a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5543e00700 1 -- 172.21.15.132:0/1569277315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f554410a760 msgr2=0x7f55441a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5543e00700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f554410a760 0x7f55441a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.114+0000 7f5543e00700 1 -- 172.21.15.132:0/1569277315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55441a87f0 con 0x7f554410c230 2024-05-27T07:01:56.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.115+0000 7f5543e00700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f554410c230 0x7f55441a8120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f553c007ee0 tx=0x7f553c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:56.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.115+0000 7f553be00700 1 -- 172.21.15.132:0/1569277315 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c00dbd0 con 0x7f554410c230 2024-05-27T07:01:56.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.115+0000 7f553be00700 1 -- 172.21.15.132:0/1569277315 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f553c00dd70 con 0x7f554410c230 2024-05-27T07:01:56.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.115+0000 7f5549000700 1 -- 172.21.15.132:0/1569277315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55441a8ae0 con 0x7f554410c230 2024-05-27T07:01:56.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.115+0000 7f553be00700 1 -- 172.21.15.132:0/1569277315 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c0058d0 con 0x7f554410c230 2024-05-27T07:01:56.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.115+0000 7f5549000700 1 -- 172.21.15.132:0/1569277315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55441a9020 con 0x7f554410c230 2024-05-27T07:01:56.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.116+0000 7f5549000700 1 -- 172.21.15.132:0/1569277315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55440512b0 con 0x7f554410c230 2024-05-27T07:01:56.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.119+0000 7f553be00700 1 -- 172.21.15.132:0/1569277315 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f553c01a020 con 0x7f554410c230 2024-05-27T07:01:56.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.120+0000 7f553be00700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55280b2900 0x7f55280b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:56.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.121+0000 7f553be00700 1 -- 172.21.15.132:0/1569277315 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f553c0e0ed0 con 0x7f554410c230 2024-05-27T07:01:56.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.121+0000 7f5543400700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55280b2900 0x7f55280b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:56.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.122+0000 7f5543400700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55280b2900 0x7f55280b4e10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f55441a2080 tx=0x7f553401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:56.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.124+0000 7f553be00700 1 -- 172.21.15.132:0/1569277315 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f553c010070 con 0x7f554410c230 2024-05-27T07:01:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:55 smithi043 ceph-mon[31531]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:01:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:55 smithi179 ceph-mon[35888]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-27T07:01:56.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:01:56.364+0000 7f5549000700 1 -- 172.21.15.132:0/1569277315 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f5544063bf0 con 0x7f55280b2900 2024-05-27T07:01:57.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:56 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:01:57.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:56 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:01:57.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:56 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:57.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:56 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:01:57.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:56 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:01:57.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:56 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:57.330 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:56 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:01:57.330 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:56 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:01:57.330 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:56 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:01:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:57 smithi132 ceph-mon[35283]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:01:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:57 smithi132 ceph-mon[35283]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:01:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:57 smithi043 ceph-mon[31531]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:01:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:57 smithi043 ceph-mon[31531]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:01:58.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:57 smithi179 ceph-mon[35888]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:01:58.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:57 smithi179 ceph-mon[35888]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-27T07:02:00.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:01:59 smithi043 ceph-mon[31531]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:02:00.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:01:59 smithi179 ceph-mon[35888]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:02:00.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:01:59 smithi132 ceph-mon[35283]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-27T07:02:01.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:00 smithi043 ceph-mon[31531]: from='client.? 172.21.15.132:0/1135712844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6a847d-4753-4061-bf71-27d740189512"}]: dispatch 2024-05-27T07:02:01.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:00 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6a847d-4753-4061-bf71-27d740189512"}]: dispatch 2024-05-27T07:02:01.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:00 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f6a847d-4753-4061-bf71-27d740189512"}]': finished 2024-05-27T07:02:01.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:00 smithi043 ceph-mon[31531]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:02:01.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:00 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:01.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:00 smithi179 ceph-mon[35888]: from='client.? 172.21.15.132:0/1135712844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6a847d-4753-4061-bf71-27d740189512"}]: dispatch 2024-05-27T07:02:01.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:00 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6a847d-4753-4061-bf71-27d740189512"}]: dispatch 2024-05-27T07:02:01.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:00 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f6a847d-4753-4061-bf71-27d740189512"}]': finished 2024-05-27T07:02:01.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:00 smithi179 ceph-mon[35888]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:02:01.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:00 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:01.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:00 smithi132 ceph-mon[35283]: from='client.? 172.21.15.132:0/1135712844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6a847d-4753-4061-bf71-27d740189512"}]: dispatch 2024-05-27T07:02:01.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:00 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6a847d-4753-4061-bf71-27d740189512"}]: dispatch 2024-05-27T07:02:01.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:00 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f6a847d-4753-4061-bf71-27d740189512"}]': finished 2024-05-27T07:02:01.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:00 smithi132 ceph-mon[35283]: osdmap e29: 5 total, 4 up, 5 in 2024-05-27T07:02:01.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:00 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:02.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:01 smithi043 ceph-mon[31531]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:02:02.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:01 smithi043 ceph-mon[31531]: from='client.? 172.21.15.132:0/1991712225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:02:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:01 smithi179 ceph-mon[35888]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:02:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:01 smithi179 ceph-mon[35888]: from='client.? 172.21.15.132:0/1991712225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:02:02.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:01 smithi132 ceph-mon[35283]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:02:02.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:01 smithi132 ceph-mon[35283]: from='client.? 172.21.15.132:0/1991712225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:02:04.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:03 smithi043 ceph-mon[31531]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:02:04.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:03 smithi179 ceph-mon[35888]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:02:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:03 smithi132 ceph-mon[35283]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:02:06.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:05 smithi132 ceph-mon[35283]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:06.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:05 smithi043 ceph-mon[31531]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:05 smithi179 ceph-mon[35888]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:08.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:07 smithi043 ceph-mon[31531]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:07 smithi179 ceph-mon[35888]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:08.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:07 smithi132 ceph-mon[35283]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:10.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:09 smithi043 ceph-mon[31531]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:10.208 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:09 smithi132 ceph-mon[35283]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:10.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:09 smithi179 ceph-mon[35888]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:11.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:02:11.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:11.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:02:11.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:11.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-27T07:02:11.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:12.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:11 smithi043 ceph-mon[31531]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:12.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:11 smithi043 ceph-mon[31531]: Deploying daemon osd.4 on smithi132 2024-05-27T07:02:12.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:11 smithi179 ceph-mon[35888]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:12.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:11 smithi179 ceph-mon[35888]: Deploying daemon osd.4 on smithi132 2024-05-27T07:02:12.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:11 smithi132 ceph-mon[35283]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:12.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:11 smithi132 ceph-mon[35283]: Deploying daemon osd.4 on smithi132 2024-05-27T07:02:14.139 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:13 smithi132 ceph-mon[35283]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:14.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:13 smithi043 ceph-mon[31531]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:14.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:13 smithi179 ceph-mon[35888]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:16.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:15 smithi179 ceph-mon[35888]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:16.318 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:15 smithi132 ceph-mon[35283]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:15 smithi043 ceph-mon[31531]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:18.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:17 smithi043 ceph-mon[31531]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:18.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:17 smithi132 ceph-mon[35283]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:17 smithi179 ceph-mon[35888]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:20.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:19 smithi132 ceph-mon[35283]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:20.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:19 smithi043 ceph-mon[31531]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:20.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:19 smithi179 ceph-mon[35888]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:21.441 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:21.441 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:21.441 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:02:21.441 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:21 smithi132 ceph-mon[35283]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:21.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:21.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:21.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:02:21.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:21 smithi043 ceph-mon[31531]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:21.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:21.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:02:21.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:21 smithi179 ceph-mon[35888]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:23.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:23 smithi043 ceph-mon[31531]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:23 smithi179 ceph-mon[35888]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:23.968 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:23 smithi132 ceph-mon[35283]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:25.211 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:24 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:25.211 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:24 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:25.211 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:24 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:25.211 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:24 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:02:25.211 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:24 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:25.211 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:24 smithi043 ceph-mon[31531]: from='osd.4 [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:02:25.211 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:24 smithi043 ceph-mon[31531]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:02:25.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:24 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:25.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:24 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:25.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:24 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:25.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:24 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:02:25.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:24 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:25.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:24 smithi179 ceph-mon[35888]: from='osd.4 [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:02:25.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:24 smithi179 ceph-mon[35888]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:02:25.336 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:24 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:25.337 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:24 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:25.337 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:24 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:25.337 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:24 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:02:25.337 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:24 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:25.337 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:24 smithi132 ceph-mon[35283]: from='osd.4 [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:02:25.337 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:24 smithi132 ceph-mon[35283]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-27T07:02:26.012 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:25 smithi132 ceph-mon[35283]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:26.012 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:25 smithi132 ceph-mon[35283]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:02:26.012 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:25 smithi132 ceph-mon[35283]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:02:26.012 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:25 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:26.012 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:25 smithi132 ceph-mon[35283]: from='osd.4 [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:02:26.012 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:25 smithi132 ceph-mon[35283]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:02:26.286 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 4 on host 'smithi132' 2024-05-27T07:02:26.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.293+0000 7f553be00700 1 -- 172.21.15.132:0/1569277315 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5544063bf0 con 0x7f55280b2900 2024-05-27T07:02:26.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 -- 172.21.15.132:0/1569277315 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55280b2900 msgr2=0x7f55280b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:02:26.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55280b2900 0x7f55280b4e10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f55441a2080 tx=0x7f553401a040 comp rx=0 tx=0).stop 2024-05-27T07:02:26.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 -- 172.21.15.132:0/1569277315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f554410c230 msgr2=0x7f55441a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:02:26.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f554410c230 0x7f55441a8120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f553c007ee0 tx=0x7f553c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:02:26.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 -- 172.21.15.132:0/1569277315 shutdown_connections 2024-05-27T07:02:26.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55280b2900 0x7f55280b4e10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:26.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f554410c230 0x7f55441a8120 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:26.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f554410a760 0x7f55441a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:26.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 --2- 172.21.15.132:0/1569277315 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f554410b890 0x7f55441a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:26.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 -- 172.21.15.132:0/1569277315 >> 172.21.15.132:0/1569277315 conn(0x7f55440797d0 msgr2=0x7f554410eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:02:26.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 -- 172.21.15.132:0/1569277315 shutdown_connections 2024-05-27T07:02:26.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:02:26.298+0000 7f5549000700 1 -- 172.21.15.132:0/1569277315 wait complete. 2024-05-27T07:02:26.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:25 smithi043 ceph-mon[31531]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:26.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:25 smithi043 ceph-mon[31531]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:02:26.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:25 smithi043 ceph-mon[31531]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:02:26.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:25 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:26.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:25 smithi043 ceph-mon[31531]: from='osd.4 [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:02:26.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:25 smithi043 ceph-mon[31531]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:02:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:25 smithi179 ceph-mon[35888]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:25 smithi179 ceph-mon[35888]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-27T07:02:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:25 smithi179 ceph-mon[35888]: osdmap e30: 5 total, 4 up, 5 in 2024-05-27T07:02:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:25 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:26.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:25 smithi179 ceph-mon[35888]: from='osd.4 [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:02:26.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:25 smithi179 ceph-mon[35888]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:02:27.202 DEBUG:teuthology.orchestra.run.smithi132:osd.4> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.4.service 2024-05-27T07:02:27.208 INFO:tasks.cephadm:Deploying osd.5 on smithi179 with /dev/vg_nvme/lv_3... 2024-05-27T07:02:27.208 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-27T07:02:27.261 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:27 smithi132 ceph-mon[35283]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:02:27.261 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:27 smithi132 ceph-mon[35283]: osdmap e31: 5 total, 4 up, 5 in 2024-05-27T07:02:27.261 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:02:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:27.262 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:27 smithi132 ceph-mon[35283]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:27.285 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:27 smithi179 ceph-mon[35888]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:02:27.285 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:27 smithi179 ceph-mon[35888]: osdmap e31: 5 total, 4 up, 5 in 2024-05-27T07:02:27.285 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:27.286 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:27.286 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:27.286 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:02:27.286 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:27.286 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:27 smithi179 ceph-mon[35888]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:27 smithi043 ceph-mon[31531]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:02:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:27 smithi043 ceph-mon[31531]: osdmap e31: 5 total, 4 up, 5 in 2024-05-27T07:02:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:02:27.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:27.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:27 smithi043 ceph-mon[31531]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-27T07:02:27.634 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:02:28.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:28 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:02:28.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:28 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:02:28.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:28 smithi179 ceph-mon[35888]: osd.4 [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] boot 2024-05-27T07:02:28.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:28 smithi179 ceph-mon[35888]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:02:28.059 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:28 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:28.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:28 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:02:28.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:28 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:02:28.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:28 smithi132 ceph-mon[35283]: osd.4 [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] boot 2024-05-27T07:02:28.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:28 smithi132 ceph-mon[35283]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:02:28.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:28 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:28.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:28 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:02:28.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:28 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:02:28.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:28 smithi043 ceph-mon[31531]: osd.4 [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] boot 2024-05-27T07:02:28.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:28 smithi043 ceph-mon[31531]: osdmap e32: 5 total, 5 up, 5 in 2024-05-27T07:02:28.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:28 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:02:29.571 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:29 smithi132 ceph-mon[35283]: osdmap e33: 5 total, 5 up, 5 in 2024-05-27T07:02:29.571 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:29 smithi132 ceph-mon[35283]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:29.653 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:29 smithi179 ceph-mon[35888]: osdmap e33: 5 total, 5 up, 5 in 2024-05-27T07:02:29.653 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:29 smithi179 ceph-mon[35888]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:29.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:29 smithi043 ceph-mon[31531]: osdmap e33: 5 total, 5 up, 5 in 2024-05-27T07:02:29.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:29 smithi043 ceph-mon[31531]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:30.728 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-27T07:02:30.728 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-27T07:02:30.728 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-05-27T07:02:30.728 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-05-27T07:02:30.728 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0334603 s, 313 MB/s 2024-05-27T07:02:30.729 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-05-27T07:02:31.587 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_3 2024-05-27T07:02:31.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:31 smithi043 ceph-mon[31531]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:31.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:31 smithi179 ceph-mon[35888]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:32.010 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:02:32.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:31 smithi132 ceph-mon[35283]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:33.650 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi132 2024-05-27T07:02:33.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:02:33.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:02:33.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi132 to 8544M 2024-05-27T07:02:33.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:33.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:02:33.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.651 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:33 smithi179 ceph-mon[35888]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi132 2024-05-27T07:02:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:02:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:02:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 8544M 2024-05-27T07:02:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:33.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:02:33.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:33 smithi043 ceph-mon[31531]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:33.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi132 2024-05-27T07:02:33.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:02:33.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:02:33.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi132 to 8544M 2024-05-27T07:02:33.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:33.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:02:33.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:33.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:33 smithi132 ceph-mon[35283]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:34.272 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.280+0000 7f2a0f600700 1 -- 172.21.15.179:0/3078963123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a0810ba70 msgr2=0x7f2a0810be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:02:34.272 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.280+0000 7f2a0f600700 1 --2- 172.21.15.179:0/3078963123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a0810ba70 0x7f2a0810be80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009990 tx=0x7f29f4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:02:34.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.281+0000 7f2a0f600700 1 -- 172.21.15.179:0/3078963123 shutdown_connections 2024-05-27T07:02:34.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.281+0000 7f2a0f600700 1 --2- 172.21.15.179:0/3078963123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a0810c410 0x7f2a08112cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:34.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.281+0000 7f2a0f600700 1 --2- 172.21.15.179:0/3078963123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a0810ab00 0x7f2a0810af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:34.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.281+0000 7f2a0f600700 1 --2- 172.21.15.179:0/3078963123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a0810ba70 0x7f2a0810be80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:34.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.281+0000 7f2a0f600700 1 -- 172.21.15.179:0/3078963123 >> 172.21.15.179:0/3078963123 conn(0x7f2a080799b0 msgr2=0x7f2a0807be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:02:34.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.281+0000 7f2a0f600700 1 -- 172.21.15.179:0/3078963123 shutdown_connections 2024-05-27T07:02:34.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.281+0000 7f2a0f600700 1 -- 172.21.15.179:0/3078963123 wait complete. 2024-05-27T07:02:34.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.282+0000 7f2a0f600700 1 Processor -- start 2024-05-27T07:02:34.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.282+0000 7f2a0f600700 1 -- start start 2024-05-27T07:02:34.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.283+0000 7f2a0f600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a0810ab00 0x7f2a081a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:34.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.283+0000 7f2a0f600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a0810ba70 0x7f2a081a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:34.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.283+0000 7f2a0f600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a0810c410 0x7f2a081a83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:34.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.283+0000 7f2a0f600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2a081162b0 con 0x7f2a0810ba70 2024-05-27T07:02:34.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.283+0000 7f2a0f600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a08116130 con 0x7f2a0810ab00 2024-05-27T07:02:34.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.283+0000 7f2a0f600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2a08116430 con 0x7f2a0810c410 2024-05-27T07:02:34.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.283+0000 7f2a0d600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a0810c410 0x7f2a081a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:02:34.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.283+0000 7f2a0d600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a0810c410 0x7f2a081a83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:39532/0 (socket says 172.21.15.179:39532) 2024-05-27T07:02:34.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.283+0000 7f2a0d600700 1 -- 172.21.15.179:0/1103327407 learned_addr learned my addr 172.21.15.179:0/1103327407 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-05-27T07:02:34.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.283+0000 7f2a07e00700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a0810ba70 0x7f2a081a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:02:34.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a0cc00700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a0810ab00 0x7f2a081a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:02:34.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a0d600700 1 -- 172.21.15.179:0/1103327407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a0810ab00 msgr2=0x7f2a081a0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:02:34.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a0d600700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a0810ab00 0x7f2a081a0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:34.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a0d600700 1 -- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a0810ba70 msgr2=0x7f2a081a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:02:34.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a0d600700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a0810ba70 0x7f2a081a1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:34.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a0d600700 1 -- 172.21.15.179:0/1103327407 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a081a8a40 con 0x7f2a0810c410 2024-05-27T07:02:34.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a07e00700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a0810ba70 0x7f2a081a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:02:34.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a0d600700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a0810c410 0x7f2a081a83c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000ef30 tx=0x7f2a0000c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:02:34.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a05600700 1 -- 172.21.15.179:0/1103327407 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a000092e0 con 0x7f2a0810c410 2024-05-27T07:02:34.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a05600700 1 -- 172.21.15.179:0/1103327407 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a00004bb0 con 0x7f2a0810c410 2024-05-27T07:02:34.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a05600700 1 -- 172.21.15.179:0/1103327407 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00004080 con 0x7f2a0810c410 2024-05-27T07:02:34.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.284+0000 7f2a0f600700 1 -- 172.21.15.179:0/1103327407 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a081a8cd0 con 0x7f2a0810c410 2024-05-27T07:02:34.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.285+0000 7f2a0f600700 1 -- 172.21.15.179:0/1103327407 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a081a91c0 con 0x7f2a0810c410 2024-05-27T07:02:34.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.286+0000 7f2a0f600700 1 -- 172.21.15.179:0/1103327407 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a080512b0 con 0x7f2a0810c410 2024-05-27T07:02:34.280 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.289+0000 7f2a05600700 1 -- 172.21.15.179:0/1103327407 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f2a0001b030 con 0x7f2a0810c410 2024-05-27T07:02:34.281 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.290+0000 7f2a05600700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f29f00b2900 0x7f29f00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:34.281 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.290+0000 7f2a05600700 1 -- 172.21.15.179:0/1103327407 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7f2a000e16f0 con 0x7f2a0810c410 2024-05-27T07:02:34.282 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.291+0000 7f2a0cc00700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f29f00b2900 0x7f29f00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:02:34.283 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.292+0000 7f2a0cc00700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f29f00b2900 0x7f29f00b4e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2a0810b9a0 tx=0x7f29fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:02:34.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.294+0000 7f2a05600700 1 -- 172.21.15.179:0/1103327407 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2a00022020 con 0x7f2a0810c410 2024-05-27T07:02:34.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:02:34.534+0000 7f2a0f600700 1 -- 172.21.15.179:0/1103327407 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2a08063cb0 con 0x7f29f00b2900 2024-05-27T07:02:34.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:02:34.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:02:34.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:34.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:02:34.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:02:34.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:34.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:02:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:02:34.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:35.812 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:35 smithi179 ceph-mon[35888]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:02:35.812 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:35 smithi179 ceph-mon[35888]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:35.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:35 smithi043 ceph-mon[31531]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:02:35.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:35 smithi043 ceph-mon[31531]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:35 smithi132 ceph-mon[35283]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:02:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:35 smithi132 ceph-mon[35283]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:37.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:37 smithi043 ceph-mon[31531]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:37 smithi179 ceph-mon[35888]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:37 smithi132 ceph-mon[35283]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:39 smithi043 ceph-mon[31531]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:39 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/207954422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12"}]: dispatch 2024-05-27T07:02:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:39 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12"}]: dispatch 2024-05-27T07:02:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:39 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12"}]': finished 2024-05-27T07:02:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:39 smithi043 ceph-mon[31531]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:02:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:02:39.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:39 smithi179 ceph-mon[35888]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:39 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/207954422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12"}]: dispatch 2024-05-27T07:02:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:39 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12"}]: dispatch 2024-05-27T07:02:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:39 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12"}]': finished 2024-05-27T07:02:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:39 smithi179 ceph-mon[35888]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:02:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:02:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:39 smithi132 ceph-mon[35283]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:39 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/207954422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12"}]: dispatch 2024-05-27T07:02:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:39 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12"}]: dispatch 2024-05-27T07:02:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:39 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12"}]': finished 2024-05-27T07:02:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:39 smithi132 ceph-mon[35283]: osdmap e34: 6 total, 5 up, 6 in 2024-05-27T07:02:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:02:40.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:40 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/2736244039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:02:40.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:40 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/2736244039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:02:41.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:40 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/2736244039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:02:41.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:41 smithi043 ceph-mon[31531]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:41 smithi179 ceph-mon[35888]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:42.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:41 smithi132 ceph-mon[35283]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:43.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:43 smithi043 ceph-mon[31531]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:43 smithi179 ceph-mon[35888]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:44.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:43 smithi132 ceph-mon[35283]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:45.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:45 smithi132 ceph-mon[35283]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:45.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:45 smithi043 ceph-mon[31531]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:45.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:45 smithi179 ceph-mon[35888]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:47.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:47 smithi043 ceph-mon[31531]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:47.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:47 smithi179 ceph-mon[35888]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:47 smithi132 ceph-mon[35283]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:48.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:02:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:48.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:02:48.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:49.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-27T07:02:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:02:49.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:49 smithi043 ceph-mon[31531]: Deploying daemon osd.5 on smithi179 2024-05-27T07:02:49.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:49 smithi043 ceph-mon[31531]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:49.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:49 smithi179 ceph-mon[35888]: Deploying daemon osd.5 on smithi179 2024-05-27T07:02:49.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:49 smithi179 ceph-mon[35888]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:50.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:49 smithi132 ceph-mon[35283]: Deploying daemon osd.5 on smithi179 2024-05-27T07:02:50.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:49 smithi132 ceph-mon[35283]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:52.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:51 smithi132 ceph-mon[35283]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:52.106 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:51 smithi179 ceph-mon[35888]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:51 smithi043 ceph-mon[31531]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:53.909 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:53 smithi179 ceph-mon[35888]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:54.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:53 smithi132 ceph-mon[35283]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:54.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:53 smithi043 ceph-mon[31531]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:55.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:55 smithi132 ceph-mon[35283]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:55 smithi043 ceph-mon[31531]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:55 smithi179 ceph-mon[35888]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:58.035 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:57 smithi179 ceph-mon[35888]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:58.035 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:58.035 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:58.035 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:02:58.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:57 smithi132 ceph-mon[35283]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:02:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:57 smithi043 ceph-mon[31531]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:02:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:02:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:02:59.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:02:59 smithi179 ceph-mon[35888]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:00.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:02:59 smithi132 ceph-mon[35283]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:00.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:02:59 smithi043 ceph-mon[31531]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:01 smithi179 ceph-mon[35888]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:02.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:01 smithi179 ceph-mon[35888]: from='osd.5 [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:03:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:01 smithi179 ceph-mon[35888]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:03:02.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:01 smithi132 ceph-mon[35283]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:02.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:02.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:02.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:02.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:02.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:02.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:01 smithi132 ceph-mon[35283]: from='osd.5 [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:03:02.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:01 smithi132 ceph-mon[35283]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:03:02.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:01 smithi043 ceph-mon[31531]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:02.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:02.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:02.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:02.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:02.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:02.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:01 smithi043 ceph-mon[31531]: from='osd.5 [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:03:02.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:01 smithi043 ceph-mon[31531]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-27T07:03:03.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:03 smithi179 ceph-mon[35888]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:03:03.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:03 smithi179 ceph-mon[35888]: from='osd.5 [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:03:03.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:03 smithi179 ceph-mon[35888]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:03:03.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:03 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:03.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:03 smithi179 ceph-mon[35888]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:03:03.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:03 smithi179 ceph-mon[35888]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:03.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:03 smithi132 ceph-mon[35283]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:03:03.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:03 smithi132 ceph-mon[35283]: from='osd.5 [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:03:03.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:03 smithi132 ceph-mon[35283]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:03:03.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:03 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:03.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:03 smithi132 ceph-mon[35283]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:03:03.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:03 smithi132 ceph-mon[35283]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:03.358 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 5 on host 'smithi179' 2024-05-27T07:03:03.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.366+0000 7f2a05600700 1 -- 172.21.15.179:0/1103327407 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2a08063cb0 con 0x7f29f00b2900 2024-05-27T07:03:03.361 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 -- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f29f00b2900 msgr2=0x7f29f00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:03:03.361 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f29f00b2900 0x7f29f00b4e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2a0810b9a0 tx=0x7f29fc009260 comp rx=0 tx=0).stop 2024-05-27T07:03:03.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 -- 172.21.15.179:0/1103327407 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a0810c410 msgr2=0x7f2a081a83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:03:03.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a0810c410 0x7f2a081a83c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000ef30 tx=0x7f2a0000c450 comp rx=0 tx=0).stop 2024-05-27T07:03:03.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 -- 172.21.15.179:0/1103327407 shutdown_connections 2024-05-27T07:03:03.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f29f00b2900 0x7f29f00b4e10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:03.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a0810c410 0x7f2a081a83c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:03.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a0810ab00 0x7f2a081a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:03.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 --2- 172.21.15.179:0/1103327407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a0810ba70 0x7f2a081a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:03.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 -- 172.21.15.179:0/1103327407 >> 172.21.15.179:0/1103327407 conn(0x7f2a080799b0 msgr2=0x7f2a0810eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:03:03.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 -- 172.21.15.179:0/1103327407 shutdown_connections 2024-05-27T07:03:03.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:03:03.370+0000 7f2a0f600700 1 -- 172.21.15.179:0/1103327407 wait complete. 2024-05-27T07:03:03.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:03 smithi043 ceph-mon[31531]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-27T07:03:03.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:03 smithi043 ceph-mon[31531]: from='osd.5 [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:03:03.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:03 smithi043 ceph-mon[31531]: osdmap e35: 6 total, 5 up, 6 in 2024-05-27T07:03:03.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:03 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:03.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:03 smithi043 ceph-mon[31531]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:03:03.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:03 smithi043 ceph-mon[31531]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:04.279 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:04 smithi179 ceph-mon[35888]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:03:04.279 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:04 smithi179 ceph-mon[35888]: osdmap e36: 6 total, 5 up, 6 in 2024-05-27T07:03:04.279 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:04.279 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:04.280 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:04.280 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:04.280 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:03:04.283 DEBUG:teuthology.orchestra.run.smithi179:osd.5> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.5.service 2024-05-27T07:03:04.291 INFO:tasks.cephadm:Deploying osd.6 on smithi043 with /dev/vg_nvme/lv_2... 2024-05-27T07:03:04.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:03:04.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:04 smithi132 ceph-mon[35283]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:03:04.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:04 smithi132 ceph-mon[35283]: osdmap e36: 6 total, 5 up, 6 in 2024-05-27T07:03:04.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:04.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:04.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:04.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:04.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:03:04.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:04 smithi043 ceph-mon[31531]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:03:04.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:04 smithi043 ceph-mon[31531]: osdmap e36: 6 total, 5 up, 6 in 2024-05-27T07:03:04.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:04.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:04.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:04.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:04.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:03:04.764 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:03:05.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:05 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:03:05.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:05 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:03:05.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:05.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:05 smithi043 ceph-mon[31531]: osd.5 [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] boot 2024-05-27T07:03:05.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:05 smithi043 ceph-mon[31531]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:03:05.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:05.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:05 smithi043 ceph-mon[31531]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:05 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:03:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:05 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:03:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:05 smithi179 ceph-mon[35888]: osd.5 [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] boot 2024-05-27T07:03:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:05 smithi179 ceph-mon[35888]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:03:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:05.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:05 smithi179 ceph-mon[35888]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:05.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:05 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:03:05.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:05 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:03:05.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:05.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:05 smithi132 ceph-mon[35283]: osd.5 [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] boot 2024-05-27T07:03:05.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:05 smithi132 ceph-mon[35283]: osdmap e37: 6 total, 6 up, 6 in 2024-05-27T07:03:05.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:03:05.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:05 smithi132 ceph-mon[35283]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-27T07:03:06.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:06 smithi043 ceph-mon[31531]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:03:06.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:06 smithi179 ceph-mon[35888]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:03:06.354 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:06 smithi132 ceph-mon[35283]: osdmap e38: 6 total, 6 up, 6 in 2024-05-27T07:03:07.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:07 smithi043 ceph-mon[31531]: osdmap e39: 6 total, 6 up, 6 in 2024-05-27T07:03:07.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:07 smithi043 ceph-mon[31531]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:07.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:07 smithi132 ceph-mon[35283]: osdmap e39: 6 total, 6 up, 6 in 2024-05-27T07:03:07.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:07 smithi132 ceph-mon[35283]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:07.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:07 smithi179 ceph-mon[35888]: osdmap e39: 6 total, 6 up, 6 in 2024-05-27T07:03:07.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:07 smithi179 ceph-mon[35888]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-05-27T07:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-05-27T07:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0369854 s, 284 MB/s 2024-05-27T07:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-05-27T07:03:08.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_2 2024-05-27T07:03:09.150 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:03:09.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:09 smithi043 ceph-mon[31531]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-05-27T07:03:09.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:09 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:09.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:09 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:09.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:09 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:09.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:09 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:09.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:09 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:09.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:09 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:09.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:09 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:09.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:09 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:09 smithi179 ceph-mon[35888]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-05-27T07:03:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:09 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:09 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:09 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:09 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:09 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:09 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:09 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:09 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:10.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:09 smithi132 ceph-mon[35283]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-05-27T07:03:10.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:09 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:10.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:09 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:10.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:09 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:10.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:09 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:10.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:09 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:10.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:09 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:10.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:09 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:10.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:09 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:10.949 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:10 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi179 2024-05-27T07:03:10.949 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:10 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi179 to 8544M 2024-05-27T07:03:10.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:10 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi179 2024-05-27T07:03:10.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:10 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi179 to 8544M 2024-05-27T07:03:11.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:10 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi179 2024-05-27T07:03:11.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:10 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi179 to 8544M 2024-05-27T07:03:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.332+0000 7f5bbea00700 1 -- 172.21.15.43:0/1411522725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bb80776d0 msgr2=0x7f5bb8075ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:03:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.332+0000 7f5bbea00700 1 --2- 172.21.15.43:0/1411522725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bb80776d0 0x7f5bb8075ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8009990 tx=0x7f5ba8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:03:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.332+0000 7f5bbea00700 1 -- 172.21.15.43:0/1411522725 shutdown_connections 2024-05-27T07:03:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.332+0000 7f5bbea00700 1 --2- 172.21.15.43:0/1411522725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb810c1e0 0x7f5bb810e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.332+0000 7f5bbea00700 1 --2- 172.21.15.43:0/1411522725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bb80776d0 0x7f5bb8075ba0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.332+0000 7f5bbea00700 1 --2- 172.21.15.43:0/1411522725 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb80760e0 0x7f5bb8076570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.332+0000 7f5bbea00700 1 -- 172.21.15.43:0/1411522725 >> 172.21.15.43:0/1411522725 conn(0x7f5bb80ff970 msgr2=0x7f5bb8101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:03:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.332+0000 7f5bbea00700 1 -- 172.21.15.43:0/1411522725 shutdown_connections 2024-05-27T07:03:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.332+0000 7f5bbea00700 1 -- 172.21.15.43:0/1411522725 wait complete. 2024-05-27T07:03:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.333+0000 7f5bbea00700 1 Processor -- start 2024-05-27T07:03:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.333+0000 7f5bbea00700 1 -- start start 2024-05-27T07:03:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.333+0000 7f5bbea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bb80760e0 0x7f5bb819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.334+0000 7f5bbea00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb80776d0 0x7f5bb819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.334+0000 7f5bbea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb810c1e0 0x7f5bb81a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.334+0000 7f5bbea00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb8111e20 con 0x7f5bb80760e0 2024-05-27T07:03:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.334+0000 7f5bbea00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb8111ca0 con 0x7f5bb810c1e0 2024-05-27T07:03:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.334+0000 7f5bbea00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb8111fa0 con 0x7f5bb80776d0 2024-05-27T07:03:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.334+0000 7f5bb7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bb80760e0 0x7f5bb819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:03:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.334+0000 7f5bb7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bb80760e0 0x7f5bb819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52620/0 (socket says 172.21.15.43:52620) 2024-05-27T07:03:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.334+0000 7f5bb7e00700 1 -- 172.21.15.43:0/3038849721 learned_addr learned my addr 172.21.15.43:0/3038849721 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:03:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.334+0000 7f5bb7400700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb80776d0 0x7f5bb819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:03:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.334+0000 7f5bbca00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb810c1e0 0x7f5bb81a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:03:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bb7e00700 1 -- 172.21.15.43:0/3038849721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb80776d0 msgr2=0x7f5bb819cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:03:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bb7e00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb80776d0 0x7f5bb819cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bb7e00700 1 -- 172.21.15.43:0/3038849721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb810c1e0 msgr2=0x7f5bb81a3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:03:11.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bb7e00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb810c1e0 0x7f5bb81a3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bb7e00700 1 -- 172.21.15.43:0/3038849721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb81a45b0 con 0x7f5bb80760e0 2024-05-27T07:03:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bb7e00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bb80760e0 0x7f5bb819ca50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80098e0 tx=0x7f5ba800ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:03:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bb4c00700 1 -- 172.21.15.43:0/3038849721 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8026070 con 0x7f5bb80760e0 2024-05-27T07:03:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bb4c00700 1 -- 172.21.15.43:0/3038849721 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5ba800f460 con 0x7f5bb80760e0 2024-05-27T07:03:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bbea00700 1 -- 172.21.15.43:0/3038849721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb81a4840 con 0x7f5bb80760e0 2024-05-27T07:03:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bb4c00700 1 -- 172.21.15.43:0/3038849721 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba800fe60 con 0x7f5bb80760e0 2024-05-27T07:03:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.335+0000 7f5bbea00700 1 -- 172.21.15.43:0/3038849721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb81a4cf0 con 0x7f5bb80760e0 2024-05-27T07:03:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.337+0000 7f5bbea00700 1 -- 172.21.15.43:0/3038849721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb80512b0 con 0x7f5bb80760e0 2024-05-27T07:03:11.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.341+0000 7f5bb4c00700 1 -- 172.21.15.43:0/3038849721 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5ba8029070 con 0x7f5bb80760e0 2024-05-27T07:03:11.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.343+0000 7f5bb4c00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5b9c0b2900 0x7f5b9c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:11.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.343+0000 7f5bb4c00700 1 -- 172.21.15.43:0/3038849721 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4533+0+0 (secure 0 0 0) 0x7f5ba80e2550 con 0x7f5bb80760e0 2024-05-27T07:03:11.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.343+0000 7f5bb7400700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5b9c0b2900 0x7f5b9c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:03:11.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.344+0000 7f5bb7400700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5b9c0b2900 0x7f5b9c0b4e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bb819db00 tx=0x7f5ba4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:03:11.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.345+0000 7f5bb4c00700 1 -- 172.21.15.43:0/3038849721 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5ba8013070 con 0x7f5bb80760e0 2024-05-27T07:03:11.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:11.593+0000 7f5bbea00700 1 -- 172.21.15.43:0/3038849721 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5bb8063bf0 con 0x7f5b9c0b2900 2024-05-27T07:03:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:11 smithi043 ceph-mon[31531]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-05-27T07:03:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:03:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:03:11.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:11 smithi179 ceph-mon[35888]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-05-27T07:03:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:03:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:03:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:11 smithi132 ceph-mon[35283]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-05-27T07:03:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:03:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:03:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:12.885 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:12 smithi043 ceph-mon[31531]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:03:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:12 smithi179 ceph-mon[35888]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:03:12.993 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:12 smithi132 ceph-mon[35283]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:03:13.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:13 smithi043 ceph-mon[31531]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:03:13.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:13 smithi179 ceph-mon[35888]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:03:14.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:13 smithi132 ceph-mon[35283]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-27T07:03:15.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:15 smithi043 ceph-mon[31531]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:03:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:15 smithi179 ceph-mon[35888]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:03:16.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:15 smithi132 ceph-mon[35283]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:03:16.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:16 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3281211631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a722fe71-7939-43d7-9c9e-72f2c7e559be"}]: dispatch 2024-05-27T07:03:16.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:16 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3281211631' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a722fe71-7939-43d7-9c9e-72f2c7e559be"}]': finished 2024-05-27T07:03:16.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:16 smithi043 ceph-mon[31531]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:03:16.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:16.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:16 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3281211631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a722fe71-7939-43d7-9c9e-72f2c7e559be"}]: dispatch 2024-05-27T07:03:16.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:16 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3281211631' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a722fe71-7939-43d7-9c9e-72f2c7e559be"}]': finished 2024-05-27T07:03:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:16 smithi179 ceph-mon[35888]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:03:16.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:17.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:16 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3281211631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a722fe71-7939-43d7-9c9e-72f2c7e559be"}]: dispatch 2024-05-27T07:03:17.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:16 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3281211631' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a722fe71-7939-43d7-9c9e-72f2c7e559be"}]': finished 2024-05-27T07:03:17.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:16 smithi132 ceph-mon[35283]: osdmap e40: 7 total, 6 up, 7 in 2024-05-27T07:03:17.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:17.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:17 smithi043 ceph-mon[31531]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:03:17.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:17 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1956052246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:03:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:17 smithi179 ceph-mon[35888]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:03:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:17 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1956052246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:03:18.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:17 smithi132 ceph-mon[35283]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:03:18.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:17 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1956052246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:03:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:19 smithi043 ceph-mon[31531]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:19.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:19 smithi179 ceph-mon[35888]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:20.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:19 smithi132 ceph-mon[35283]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:21.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:21 smithi043 ceph-mon[31531]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:21 smithi132 ceph-mon[35283]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:21 smithi179 ceph-mon[35888]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:23.796 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:23 smithi043 ceph-mon[31531]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:24.037 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:23 smithi132 ceph-mon[35283]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:24.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:23 smithi179 ceph-mon[35888]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:25.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:25 smithi043 ceph-mon[31531]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:26.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:25 smithi132 ceph-mon[35283]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:26.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:25 smithi179 ceph-mon[35888]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:03:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:27.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:03:27.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:27.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-27T07:03:27.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:28.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:27 smithi132 ceph-mon[35283]: Deploying daemon osd.6 on smithi043 2024-05-27T07:03:28.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:27 smithi132 ceph-mon[35283]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:28.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:27 smithi043 ceph-mon[31531]: Deploying daemon osd.6 on smithi043 2024-05-27T07:03:28.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:27 smithi043 ceph-mon[31531]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:28.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:27 smithi179 ceph-mon[35888]: Deploying daemon osd.6 on smithi043 2024-05-27T07:03:28.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:27 smithi179 ceph-mon[35888]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:30.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:29 smithi132 ceph-mon[35283]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:30.167 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:29 smithi043 ceph-mon[31531]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:30.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:29 smithi179 ceph-mon[35888]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:31.947 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:31 smithi043 ceph-mon[31531]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:31 smithi132 ceph-mon[35283]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:31 smithi179 ceph-mon[35888]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:33 smithi132 ceph-mon[35283]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:34.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:33 smithi043 ceph-mon[31531]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:34.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:33 smithi179 ceph-mon[35888]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:36.012 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:35 smithi179 ceph-mon[35888]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:36.012 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:36.012 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:36.012 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:03:36.022 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:35 smithi043 ceph-mon[31531]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:36.022 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:36.022 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:36.022 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:03:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:35 smithi132 ceph-mon[35283]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:03:37.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:37 smithi043 ceph-mon[31531]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:37 smithi132 ceph-mon[35283]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:37 smithi179 ceph-mon[35888]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:39.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:39 smithi043 ceph-mon[31531]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:39.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:39 smithi043 ceph-mon[31531]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:03:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:39 smithi132 ceph-mon[35283]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:40.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:39 smithi132 ceph-mon[35283]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:03:40.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:39 smithi179 ceph-mon[35888]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:40.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:40.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:40.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:40.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:40.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:40.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:39 smithi179 ceph-mon[35888]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-27T07:03:41.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.299+0000 7f5bb4c00700 1 -- 172.21.15.43:0/3038849721 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5bb8063bf0 con 0x7f5b9c0b2900 2024-05-27T07:03:41.293 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 6 on host 'smithi043' 2024-05-27T07:03:41.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.304+0000 7f5bbea00700 1 -- 172.21.15.43:0/3038849721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5b9c0b2900 msgr2=0x7f5b9c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:03:41.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.304+0000 7f5bbea00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5b9c0b2900 0x7f5b9c0b4e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bb819db00 tx=0x7f5ba4009260 comp rx=0 tx=0).stop 2024-05-27T07:03:41.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.304+0000 7f5bbea00700 1 -- 172.21.15.43:0/3038849721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bb80760e0 msgr2=0x7f5bb819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:03:41.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.304+0000 7f5bbea00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bb80760e0 0x7f5bb819ca50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80098e0 tx=0x7f5ba800ba90 comp rx=0 tx=0).stop 2024-05-27T07:03:41.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.305+0000 7f5bbea00700 1 -- 172.21.15.43:0/3038849721 shutdown_connections 2024-05-27T07:03:41.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.305+0000 7f5bbea00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5b9c0b2900 0x7f5b9c0b4e10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:41.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.305+0000 7f5bbea00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb810c1e0 0x7f5bb81a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:41.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.305+0000 7f5bbea00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bb80760e0 0x7f5bb819ca50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:41.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.305+0000 7f5bbea00700 1 --2- 172.21.15.43:0/3038849721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb80776d0 0x7f5bb819cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:41.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.305+0000 7f5bbea00700 1 -- 172.21.15.43:0/3038849721 >> 172.21.15.43:0/3038849721 conn(0x7f5bb80ff970 msgr2=0x7f5bb8108170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:03:41.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.305+0000 7f5bbea00700 1 -- 172.21.15.43:0/3038849721 shutdown_connections 2024-05-27T07:03:41.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:03:41.305+0000 7f5bbea00700 1 -- 172.21.15.43:0/3038849721 wait complete. 2024-05-27T07:03:41.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:41 smithi043 ceph-mon[31531]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:03:41.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:41 smithi043 ceph-mon[31531]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:03:41.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:41 smithi043 ceph-mon[31531]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T07:03:41.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:41.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:41 smithi043 ceph-mon[31531]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:41.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:41.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:41.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:03:41.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:41 smithi179 ceph-mon[35888]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:03:41.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:41 smithi179 ceph-mon[35888]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:03:41.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:41 smithi179 ceph-mon[35888]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T07:03:41.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:41.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:41 smithi179 ceph-mon[35888]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:41.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:41.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:41.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:03:41.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:41 smithi132 ceph-mon[35283]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-27T07:03:41.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:41 smithi132 ceph-mon[35283]: osdmap e41: 7 total, 6 up, 7 in 2024-05-27T07:03:41.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:41 smithi132 ceph-mon[35283]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T07:03:41.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:41.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:41 smithi132 ceph-mon[35283]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:41.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:41.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:41.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:03:42.083 DEBUG:teuthology.orchestra.run.smithi043:osd.6> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.6.service 2024-05-27T07:03:42.087 INFO:tasks.cephadm:Deploying osd.7 on smithi132 with /dev/vg_nvme/lv_2... 2024-05-27T07:03:42.087 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:03:42.521 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:03:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:42 smithi132 ceph-mon[35283]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:03:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:42 smithi132 ceph-mon[35283]: osdmap e42: 7 total, 6 up, 7 in 2024-05-27T07:03:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:42 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:42 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:42 smithi132 ceph-mon[35283]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' 2024-05-27T07:03:42.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:42 smithi043 ceph-mon[31531]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:03:42.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:42 smithi043 ceph-mon[31531]: osdmap e42: 7 total, 6 up, 7 in 2024-05-27T07:03:42.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:42 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:42.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:42 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:42.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:42 smithi043 ceph-mon[31531]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' 2024-05-27T07:03:42.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:42 smithi179 ceph-mon[35888]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:03:42.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:42 smithi179 ceph-mon[35888]: osdmap e42: 7 total, 6 up, 7 in 2024-05-27T07:03:42.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:42 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:42.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:42 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:42.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:42 smithi179 ceph-mon[35888]: from='osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127]' entity='osd.6' 2024-05-27T07:03:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:43 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:03:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:43 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:03:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:43 smithi043 ceph-mon[31531]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:43 smithi043 ceph-mon[31531]: osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] boot 2024-05-27T07:03:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:43 smithi043 ceph-mon[31531]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:03:43.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:43 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:03:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:43 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:03:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:43 smithi179 ceph-mon[35888]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:43 smithi179 ceph-mon[35888]: osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] boot 2024-05-27T07:03:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:43 smithi179 ceph-mon[35888]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:03:43.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:43.711 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:43 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:03:43.711 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:43 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:03:43.711 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:43.711 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:43 smithi132 ceph-mon[35283]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-27T07:03:43.711 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:43 smithi132 ceph-mon[35283]: osd.6 [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] boot 2024-05-27T07:03:43.711 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:43 smithi132 ceph-mon[35283]: osdmap e43: 7 total, 7 up, 7 in 2024-05-27T07:03:43.711 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:03:45.614 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:03:45.615 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:03:45.615 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-05-27T07:03:45.615 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-05-27T07:03:45.615 INFO:teuthology.orchestra.run.smithi132.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0356803 s, 294 MB/s 2024-05-27T07:03:45.615 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 2024-05-27T07:03:45.615 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-05-27T07:03:45.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:45 smithi043 ceph-mon[31531]: osdmap e44: 7 total, 7 up, 7 in 2024-05-27T07:03:45.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:45 smithi043 ceph-mon[31531]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:45.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:45 smithi179 ceph-mon[35888]: osdmap e44: 7 total, 7 up, 7 in 2024-05-27T07:03:45.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:45 smithi179 ceph-mon[35888]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:45.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:45 smithi132 ceph-mon[35283]: osdmap e44: 7 total, 7 up, 7 in 2024-05-27T07:03:45.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:45 smithi132 ceph-mon[35283]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:46.531 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_2 2024-05-27T07:03:46.941 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:03:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi043 2024-05-27T07:03:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi043 to 5698M 2024-05-27T07:03:48.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:48.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:48.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi043 2024-05-27T07:03:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi043 to 5698M 2024-05-27T07:03:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:48.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:48.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:48.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi043 2024-05-27T07:03:48.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:48.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:48.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:03:48.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi043 to 5698M 2024-05-27T07:03:48.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:48.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:48.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:03:48.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:03:49.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.331+0000 7f9d54400700 1 -- 172.21.15.132:0/1376249944 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c06c2c0 msgr2=0x7f9d4c106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:03:49.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.331+0000 7f9d54400700 1 --2- 172.21.15.132:0/1376249944 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c06c2c0 0x7f9d4c106ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009990 tx=0x7f9d40009ca0 comp rx=0 tx=0).stop 2024-05-27T07:03:49.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.331+0000 7f9d54400700 1 -- 172.21.15.132:0/1376249944 shutdown_connections 2024-05-27T07:03:49.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.331+0000 7f9d54400700 1 --2- 172.21.15.132:0/1376249944 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d4c109fe0 0x7f9d4c10c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:49.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.331+0000 7f9d54400700 1 --2- 172.21.15.132:0/1376249944 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c06c2c0 0x7f9d4c106ea0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:49.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.331+0000 7f9d54400700 1 --2- 172.21.15.132:0/1376249944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d4c107470 0x7f9d4c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:49.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.331+0000 7f9d54400700 1 -- 172.21.15.132:0/1376249944 >> 172.21.15.132:0/1376249944 conn(0x7f9d4c0fe870 msgr2=0x7f9d4c100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:03:49.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.332+0000 7f9d54400700 1 -- 172.21.15.132:0/1376249944 shutdown_connections 2024-05-27T07:03:49.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.332+0000 7f9d54400700 1 -- 172.21.15.132:0/1376249944 wait complete. 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.333+0000 7f9d54400700 1 Processor -- start 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.333+0000 7f9d54400700 1 -- start start 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.333+0000 7f9d54400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d4c06c2c0 0x7f9d4c19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.333+0000 7f9d54400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d4c107470 0x7f9d4c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.333+0000 7f9d54400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c109fe0 0x7f9d4c1a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.333+0000 7f9d54400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c10fc70 con 0x7f9d4c107470 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.333+0000 7f9d54400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c10faf0 con 0x7f9d4c109fe0 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.333+0000 7f9d54400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c10fdf0 con 0x7f9d4c06c2c0 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.334+0000 7f9d52400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c109fe0 0x7f9d4c1a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.334+0000 7f9d52400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c109fe0 0x7f9d4c1a1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:35776/0 (socket says 172.21.15.132:35776) 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.334+0000 7f9d52400700 1 -- 172.21.15.132:0/3915688416 learned_addr learned my addr 172.21.15.132:0/3915688416 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-05-27T07:03:49.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.334+0000 7f9d51a00700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d4c06c2c0 0x7f9d4c19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:03:49.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.334+0000 7f9d51000700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d4c107470 0x7f9d4c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:03:49.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.334+0000 7f9d52400700 1 -- 172.21.15.132:0/3915688416 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d4c06c2c0 msgr2=0x7f9d4c19a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:03:49.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.334+0000 7f9d52400700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d4c06c2c0 0x7f9d4c19a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:49.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.334+0000 7f9d52400700 1 -- 172.21.15.132:0/3915688416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d4c107470 msgr2=0x7f9d4c19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:03:49.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.334+0000 7f9d52400700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d4c107470 0x7f9d4c19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:49.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.335+0000 7f9d52400700 1 -- 172.21.15.132:0/3915688416 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c1a23e0 con 0x7f9d4c109fe0 2024-05-27T07:03:49.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.335+0000 7f9d52400700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c109fe0 0x7f9d4c1a1d60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9d44007ee0 tx=0x7f9d4400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:03:49.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.335+0000 7f9d4a000700 1 -- 172.21.15.132:0/3915688416 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4400ddd0 con 0x7f9d4c109fe0 2024-05-27T07:03:49.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.335+0000 7f9d4a000700 1 -- 172.21.15.132:0/3915688416 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d44004020 con 0x7f9d4c109fe0 2024-05-27T07:03:49.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.335+0000 7f9d54400700 1 -- 172.21.15.132:0/3915688416 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4c1a26d0 con 0x7f9d4c109fe0 2024-05-27T07:03:49.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.335+0000 7f9d4a000700 1 -- 172.21.15.132:0/3915688416 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44015660 con 0x7f9d4c109fe0 2024-05-27T07:03:49.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.335+0000 7f9d54400700 1 -- 172.21.15.132:0/3915688416 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4c1a2c10 con 0x7f9d4c109fe0 2024-05-27T07:03:49.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.337+0000 7f9d54400700 1 -- 172.21.15.132:0/3915688416 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d4c0512b0 con 0x7f9d4c109fe0 2024-05-27T07:03:49.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.339+0000 7f9d4a000700 1 -- 172.21.15.132:0/3915688416 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9d440041d0 con 0x7f9d4c109fe0 2024-05-27T07:03:49.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.341+0000 7f9d4a000700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9d340b2900 0x7f9d340b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:49.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.341+0000 7f9d4a000700 1 -- 172.21.15.132:0/3915688416 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4965+0+0 (secure 0 0 0) 0x7f9d440e1770 con 0x7f9d4c109fe0 2024-05-27T07:03:49.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.341+0000 7f9d51a00700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9d340b2900 0x7f9d340b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:03:49.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.343+0000 7f9d51a00700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9d340b2900 0x7f9d340b4e10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000b710 tx=0x7f9d4001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:03:49.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.345+0000 7f9d4a000700 1 -- 172.21.15.132:0/3915688416 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9d44010070 con 0x7f9d4c109fe0 2024-05-27T07:03:49.580 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:03:49.588+0000 7f9d54400700 1 -- 172.21.15.132:0/3915688416 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f9d4c063bf0 con 0x7f9d340b2900 2024-05-27T07:03:50.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:49 smithi043 ceph-mon[31531]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:50.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:49 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:03:50.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:49 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:03:50.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:49 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:50.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:49 smithi179 ceph-mon[35888]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:50.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:49 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:03:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:49 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:03:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:49 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:50.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:49 smithi132 ceph-mon[35283]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:50.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:49 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:03:50.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:49 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:03:50.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:49 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:03:51.077 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:50 smithi132 ceph-mon[35283]: from='client.34277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:03:51.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:50 smithi043 ceph-mon[31531]: from='client.34277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:03:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:50 smithi179 ceph-mon[35888]: from='client.34277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:03:52.186 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:51 smithi132 ceph-mon[35283]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:51 smithi043 ceph-mon[31531]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:51 smithi179 ceph-mon[35888]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:54.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:53 smithi043 ceph-mon[31531]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:54.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:53 smithi179 ceph-mon[35888]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:53 smithi132 ceph-mon[35283]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:55.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:54 smithi043 ceph-mon[31531]: from='client.? 172.21.15.132:0/367056871' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b6b6895-34fe-4afc-90c1-a0d30b365713"}]: dispatch 2024-05-27T07:03:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:54 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b6b6895-34fe-4afc-90c1-a0d30b365713"}]: dispatch 2024-05-27T07:03:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:54 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b6b6895-34fe-4afc-90c1-a0d30b365713"}]': finished 2024-05-27T07:03:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:54 smithi043 ceph-mon[31531]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:03:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:54 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:03:55.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:54 smithi179 ceph-mon[35888]: from='client.? 172.21.15.132:0/367056871' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b6b6895-34fe-4afc-90c1-a0d30b365713"}]: dispatch 2024-05-27T07:03:55.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:54 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b6b6895-34fe-4afc-90c1-a0d30b365713"}]: dispatch 2024-05-27T07:03:55.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:54 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b6b6895-34fe-4afc-90c1-a0d30b365713"}]': finished 2024-05-27T07:03:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:54 smithi179 ceph-mon[35888]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:03:55.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:54 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:03:55.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:54 smithi132 ceph-mon[35283]: from='client.? 172.21.15.132:0/367056871' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b6b6895-34fe-4afc-90c1-a0d30b365713"}]: dispatch 2024-05-27T07:03:55.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:54 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b6b6895-34fe-4afc-90c1-a0d30b365713"}]: dispatch 2024-05-27T07:03:55.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:54 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b6b6895-34fe-4afc-90c1-a0d30b365713"}]': finished 2024-05-27T07:03:55.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:54 smithi132 ceph-mon[35283]: osdmap e45: 8 total, 7 up, 8 in 2024-05-27T07:03:55.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:54 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:03:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:55 smithi043 ceph-mon[31531]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:55 smithi043 ceph-mon[31531]: from='client.? 172.21.15.132:0/1077207156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:03:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:55 smithi179 ceph-mon[35888]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:55 smithi179 ceph-mon[35888]: from='client.? 172.21.15.132:0/1077207156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:03:56.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:55 smithi132 ceph-mon[35283]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:56.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:55 smithi132 ceph-mon[35283]: from='client.? 172.21.15.132:0/1077207156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:03:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:57 smithi043 ceph-mon[31531]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:58.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:57 smithi179 ceph-mon[35888]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:03:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:57 smithi132 ceph-mon[35283]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:00.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:03:59 smithi043 ceph-mon[31531]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:00.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:03:59 smithi179 ceph-mon[35888]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:00.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:03:59 smithi132 ceph-mon[35283]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:02.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:01 smithi132 ceph-mon[35283]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:02.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:01 smithi043 ceph-mon[31531]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:01 smithi179 ceph-mon[35888]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:04.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:03 smithi179 ceph-mon[35888]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:03 smithi132 ceph-mon[35283]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:04.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:03 smithi043 ceph-mon[31531]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:05.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:04:05.281 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:05.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:04:05.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-27T07:04:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:06.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:05 smithi132 ceph-mon[35283]: Deploying daemon osd.7 on smithi132 2024-05-27T07:04:06.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:05 smithi132 ceph-mon[35283]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:06.361 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:05 smithi179 ceph-mon[35888]: Deploying daemon osd.7 on smithi132 2024-05-27T07:04:06.361 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:05 smithi179 ceph-mon[35888]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:06.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:05 smithi043 ceph-mon[31531]: Deploying daemon osd.7 on smithi132 2024-05-27T07:04:06.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:05 smithi043 ceph-mon[31531]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:08.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:07 smithi132 ceph-mon[35283]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:07 smithi043 ceph-mon[31531]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:08.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:07 smithi179 ceph-mon[35888]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:10.236 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:09 smithi132 ceph-mon[35283]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:10.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:09 smithi043 ceph-mon[31531]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:09 smithi179 ceph-mon[35888]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:12.248 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:11 smithi132 ceph-mon[35283]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:12.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:11 smithi043 ceph-mon[31531]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:11 smithi179 ceph-mon[35888]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:14.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:14 smithi132 ceph-mon[35283]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:14.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:14 smithi043 ceph-mon[31531]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:14.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:14 smithi179 ceph-mon[35888]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:15.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:15.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:15.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:15.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:15 smithi132 ceph-mon[35283]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:15 smithi043 ceph-mon[31531]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:15.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:15.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:15.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:15 smithi179 ceph-mon[35888]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:17.696 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:17 smithi132 ceph-mon[35283]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:17.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:17 smithi043 ceph-mon[31531]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:17.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:17 smithi179 ceph-mon[35888]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:18.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:18 smithi043 ceph-mon[31531]: from='osd.7 [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:04:18.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:18 smithi043 ceph-mon[31531]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:04:18.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:18 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:18.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:18 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:18.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:18 smithi179 ceph-mon[35888]: from='osd.7 [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:04:18.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:18 smithi179 ceph-mon[35888]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:04:18.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:18 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:18.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:18 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:19.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:18 smithi132 ceph-mon[35283]: from='osd.7 [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:04:19.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:18 smithi132 ceph-mon[35283]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-27T07:04:19.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:18 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:19.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:18 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:19 smithi043 ceph-mon[31531]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:19 smithi043 ceph-mon[31531]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:04:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:19 smithi043 ceph-mon[31531]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:04:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:19 smithi043 ceph-mon[31531]: from='osd.7 [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:04:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:19 smithi043 ceph-mon[31531]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:04:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:04:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:19 smithi179 ceph-mon[35888]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:19 smithi179 ceph-mon[35888]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:04:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:19 smithi179 ceph-mon[35888]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:04:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:19 smithi179 ceph-mon[35888]: from='osd.7 [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:04:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:19 smithi179 ceph-mon[35888]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:04:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:04:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:20.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:19 smithi132 ceph-mon[35283]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:20.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:19 smithi132 ceph-mon[35283]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-27T07:04:20.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:19 smithi132 ceph-mon[35283]: osdmap e46: 8 total, 7 up, 8 in 2024-05-27T07:04:20.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:19 smithi132 ceph-mon[35283]: from='osd.7 [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:04:20.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:20.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:19 smithi132 ceph-mon[35283]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:04:20.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:20.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:04:20.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:20.219 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 7 on host 'smithi132' 2024-05-27T07:04:20.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.226+0000 7f9d4a000700 1 -- 172.21.15.132:0/3915688416 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9d4c063bf0 con 0x7f9d340b2900 2024-05-27T07:04:20.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.231+0000 7f9d54400700 1 -- 172.21.15.132:0/3915688416 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9d340b2900 msgr2=0x7f9d340b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:04:20.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.231+0000 7f9d54400700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9d340b2900 0x7f9d340b4e10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000b710 tx=0x7f9d4001a040 comp rx=0 tx=0).stop 2024-05-27T07:04:20.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.231+0000 7f9d54400700 1 -- 172.21.15.132:0/3915688416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c109fe0 msgr2=0x7f9d4c1a1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:04:20.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.231+0000 7f9d54400700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c109fe0 0x7f9d4c1a1d60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9d44007ee0 tx=0x7f9d4400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:04:20.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.232+0000 7f9d54400700 1 -- 172.21.15.132:0/3915688416 shutdown_connections 2024-05-27T07:04:20.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.232+0000 7f9d54400700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9d340b2900 0x7f9d340b4e10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:20.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.232+0000 7f9d54400700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d4c109fe0 0x7f9d4c1a1d60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:20.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.232+0000 7f9d54400700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d4c06c2c0 0x7f9d4c19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:20.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.232+0000 7f9d54400700 1 --2- 172.21.15.132:0/3915688416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d4c107470 0x7f9d4c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:20.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.232+0000 7f9d54400700 1 -- 172.21.15.132:0/3915688416 >> 172.21.15.132:0/3915688416 conn(0x7f9d4c0fe870 msgr2=0x7f9d4c105940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:04:20.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.232+0000 7f9d54400700 1 -- 172.21.15.132:0/3915688416 shutdown_connections 2024-05-27T07:04:20.224 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:04:20.232+0000 7f9d54400700 1 -- 172.21.15.132:0/3915688416 wait complete. 2024-05-27T07:04:20.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:04:20.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: osdmap e47: 8 total, 7 up, 8 in 2024-05-27T07:04:20.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:20.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:20.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:20.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:20.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:20.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: from='osd.7 ' entity='osd.7' 2024-05-27T07:04:20.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: osd.7 [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] boot 2024-05-27T07:04:20.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:04:20.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: osdmap e47: 8 total, 7 up, 8 in 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: from='osd.7 ' entity='osd.7' 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: osd.7 [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] boot 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:04:21.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:04:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: osdmap e47: 8 total, 7 up, 8 in 2024-05-27T07:04:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: from='osd.7 ' entity='osd.7' 2024-05-27T07:04:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: osd.7 [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] boot 2024-05-27T07:04:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: osdmap e48: 8 total, 8 up, 8 in 2024-05-27T07:04:21.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:04:21.252 DEBUG:teuthology.orchestra.run.smithi132:osd.7> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.7.service 2024-05-27T07:04:21.257 INFO:tasks.cephadm:Deploying osd.8 on smithi179 with /dev/vg_nvme/lv_2... 2024-05-27T07:04:21.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-27T07:04:21.698 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:04:21.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:21 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:04:21.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:21 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:04:21.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:21 smithi132 ceph-mon[35283]: pgmap v231: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:21 smithi132 ceph-mon[35283]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-05-27T07:04:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:21 smithi132 ceph-mon[35283]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:04:22.007 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:21 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:04:22.007 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:21 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:04:22.007 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:21 smithi179 ceph-mon[35888]: pgmap v231: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:22.007 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:21 smithi179 ceph-mon[35888]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-05-27T07:04:22.008 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:21 smithi179 ceph-mon[35888]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:04:22.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:21 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:04:22.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:21 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:04:22.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:21 smithi043 ceph-mon[31531]: pgmap v231: 1 pgs: 1 unknown; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-05-27T07:04:22.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:21 smithi043 ceph-mon[31531]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-05-27T07:04:22.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:21 smithi043 ceph-mon[31531]: osdmap e49: 8 total, 8 up, 8 in 2024-05-27T07:04:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:23 smithi179 ceph-mon[35888]: pgmap v234: 1 pgs: 1 unknown; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:23.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:23 smithi179 ceph-mon[35888]: osdmap e50: 8 total, 8 up, 8 in 2024-05-27T07:04:23.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:23 smithi132 ceph-mon[35283]: pgmap v234: 1 pgs: 1 unknown; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:23.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:23 smithi132 ceph-mon[35283]: osdmap e50: 8 total, 8 up, 8 in 2024-05-27T07:04:23.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:23 smithi043 ceph-mon[31531]: pgmap v234: 1 pgs: 1 unknown; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:23.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:23 smithi043 ceph-mon[31531]: osdmap e50: 8 total, 8 up, 8 in 2024-05-27T07:04:24.726 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-27T07:04:24.727 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-27T07:04:24.727 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-05-27T07:04:24.727 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-05-27T07:04:24.727 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0348929 s, 301 MB/s 2024-05-27T07:04:24.727 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-05-27T07:04:25.627 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_2 2024-05-27T07:04:25.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:25 smithi043 ceph-mon[31531]: pgmap v236: 1 pgs: 1 unknown; 577 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:25.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:25 smithi179 ceph-mon[35888]: pgmap v236: 1 pgs: 1 unknown; 577 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:26.046 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:04:26.062 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:25 smithi132 ceph-mon[35283]: pgmap v236: 1 pgs: 1 unknown; 577 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi132 2024-05-27T07:04:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:04:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:04:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:04:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 5696M 2024-05-27T07:04:27.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:27.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:04:27.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:27 smithi043 ceph-mon[31531]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s, 0 objects/s recovering 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi132 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi132 to 5696M 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:27 smithi179 ceph-mon[35888]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s, 0 objects/s recovering 2024-05-27T07:04:27.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi132 2024-05-27T07:04:27.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:04:27.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:04:27.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:04:27.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi132 to 5696M 2024-05-27T07:04:27.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:27.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:04:27.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:27.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:27 smithi132 ceph-mon[35283]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s, 0 objects/s recovering 2024-05-27T07:04:28.402 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.409+0000 7f5dc1c00700 1 -- 172.21.15.179:0/3741390089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc106540 msgr2=0x7f5dbc106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:04:28.402 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.409+0000 7f5dc1c00700 1 --2- 172.21.15.179:0/3741390089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc106540 0x7f5dbc106950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009990 tx=0x7f5dac009ca0 comp rx=0 tx=0).stop 2024-05-27T07:04:28.402 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.409+0000 7f5dc1c00700 1 -- 172.21.15.179:0/3741390089 shutdown_connections 2024-05-27T07:04:28.402 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.409+0000 7f5dc1c00700 1 --2- 172.21.15.179:0/3741390089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dbc108010 0x7f5dbc10e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:28.402 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.409+0000 7f5dc1c00700 1 --2- 172.21.15.179:0/3741390089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc106540 0x7f5dbc106950 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:28.402 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.409+0000 7f5dc1c00700 1 --2- 172.21.15.179:0/3741390089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dbc107670 0x7f5dbc107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:28.403 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.409+0000 7f5dc1c00700 1 -- 172.21.15.179:0/3741390089 >> 172.21.15.179:0/3741390089 conn(0x7f5dbc101bc0 msgr2=0x7f5dbc104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:04:28.403 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.409+0000 7f5dc1c00700 1 -- 172.21.15.179:0/3741390089 shutdown_connections 2024-05-27T07:04:28.403 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.409+0000 7f5dc1c00700 1 -- 172.21.15.179:0/3741390089 wait complete. 2024-05-27T07:04:28.404 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.410+0000 7f5dc1c00700 1 Processor -- start 2024-05-27T07:04:28.404 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.411+0000 7f5dc1c00700 1 -- start start 2024-05-27T07:04:28.404 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.411+0000 7f5dc1c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dbc106540 0x7f5dbc19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:28.404 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.411+0000 7f5dc1c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dbc107670 0x7f5dbc19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:28.404 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.411+0000 7f5dc1c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc108010 0x7f5dbc1a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:28.404 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.411+0000 7f5dc1c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5dbc111e40 con 0x7f5dbc107670 2024-05-27T07:04:28.404 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.411+0000 7f5dc1c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5dbc111cc0 con 0x7f5dbc106540 2024-05-27T07:04:28.404 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.411+0000 7f5dc1c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5dbc111fc0 con 0x7f5dbc108010 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.411+0000 7f5dbb400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc108010 0x7f5dbc1a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.411+0000 7f5dbb400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc108010 0x7f5dbc1a3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:35810/0 (socket says 172.21.15.179:35810) 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.411+0000 7f5dbb400700 1 -- 172.21.15.179:0/2410587140 learned_addr learned my addr 172.21.15.179:0/2410587140 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.412+0000 7f5dba000700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dbc107670 0x7f5dbc19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.412+0000 7f5dbaa00700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dbc106540 0x7f5dbc19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.412+0000 7f5dbb400700 1 -- 172.21.15.179:0/2410587140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dbc106540 msgr2=0x7f5dbc19ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.412+0000 7f5dbb400700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dbc106540 0x7f5dbc19ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.412+0000 7f5dbb400700 1 -- 172.21.15.179:0/2410587140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dbc107670 msgr2=0x7f5dbc19cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.412+0000 7f5dbb400700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dbc107670 0x7f5dbc19cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.412+0000 7f5dbb400700 1 -- 172.21.15.179:0/2410587140 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dbc1a45b0 con 0x7f5dbc108010 2024-05-27T07:04:28.405 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.412+0000 7f5dbaa00700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dbc106540 0x7f5dbc19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:04:28.406 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.412+0000 7f5dbb400700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc108010 0x7f5dbc1a3f30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5db4007ee0 tx=0x7f5db400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:04:28.406 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.413+0000 7f5db3400700 1 -- 172.21.15.179:0/2410587140 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db400dbd0 con 0x7f5dbc108010 2024-05-27T07:04:28.406 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.413+0000 7f5db3400700 1 -- 172.21.15.179:0/2410587140 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5db400dd70 con 0x7f5dbc108010 2024-05-27T07:04:28.406 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.413+0000 7f5dc1c00700 1 -- 172.21.15.179:0/2410587140 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dbc1a48a0 con 0x7f5dbc108010 2024-05-27T07:04:28.406 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.413+0000 7f5db3400700 1 -- 172.21.15.179:0/2410587140 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db40058d0 con 0x7f5dbc108010 2024-05-27T07:04:28.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.413+0000 7f5dc1c00700 1 -- 172.21.15.179:0/2410587140 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dbc1a4de0 con 0x7f5dbc108010 2024-05-27T07:04:28.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.414+0000 7f5dc1c00700 1 -- 172.21.15.179:0/2410587140 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dbc0512b0 con 0x7f5dbc108010 2024-05-27T07:04:28.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.416+0000 7f5db3400700 1 -- 172.21.15.179:0/2410587140 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5db401a020 con 0x7f5dbc108010 2024-05-27T07:04:28.409 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:28 smithi179 ceph-mon[35888]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-05-27T07:04:28.409 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:28 smithi179 ceph-mon[35888]: Cluster is now healthy 2024-05-27T07:04:28.410 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.418+0000 7f5db3400700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5da00b2900 0x7f5da00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:28.410 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.418+0000 7f5db3400700 1 -- 172.21.15.179:0/2410587140 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5397+0+0 (secure 0 0 0) 0x7f5db40e0f30 con 0x7f5dbc108010 2024-05-27T07:04:28.410 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.418+0000 7f5dbaa00700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5da00b2900 0x7f5da00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:04:28.412 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.420+0000 7f5dbaa00700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5da00b2900 0x7f5da00b4e10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5dac003820 tx=0x7f5dac01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:04:28.415 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.423+0000 7f5db3400700 1 -- 172.21.15.179:0/2410587140 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5db4010070 con 0x7f5dbc108010 2024-05-27T07:04:28.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:28 smithi043 ceph-mon[31531]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-05-27T07:04:28.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:28 smithi043 ceph-mon[31531]: Cluster is now healthy 2024-05-27T07:04:28.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:28 smithi132 ceph-mon[35283]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-05-27T07:04:28.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:28 smithi132 ceph-mon[35283]: Cluster is now healthy 2024-05-27T07:04:28.656 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:28.663+0000 7f5dc1c00700 1 -- 172.21.15.179:0/2410587140 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5dbc063bf0 con 0x7f5da00b2900 2024-05-27T07:04:29.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:29 smithi043 ceph-mon[31531]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-05-27T07:04:29.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:29 smithi043 ceph-mon[31531]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:04:29.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:29 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:04:29.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:29 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:04:29.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:29 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:29.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:29 smithi179 ceph-mon[35888]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-05-27T07:04:29.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:29 smithi179 ceph-mon[35888]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:04:29.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:29 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:04:29.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:29 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:04:29.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:29 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:29.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:29 smithi132 ceph-mon[35283]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-05-27T07:04:29.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:29 smithi132 ceph-mon[35283]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:04:29.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:29 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:04:29.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:29 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:04:29.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:29 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:31.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:31 smithi043 ceph-mon[31531]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-27T07:04:31.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:31 smithi179 ceph-mon[35888]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-27T07:04:32.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:31 smithi132 ceph-mon[35283]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-05-27T07:04:33.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:33 smithi043 ceph-mon[31531]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-27T07:04:33.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:33 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/1692587796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10a2bd5e-5ac8-441f-b2e5-deb1f9239b17"}]: dispatch 2024-05-27T07:04:33.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:33 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10a2bd5e-5ac8-441f-b2e5-deb1f9239b17"}]: dispatch 2024-05-27T07:04:33.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:33 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10a2bd5e-5ac8-441f-b2e5-deb1f9239b17"}]': finished 2024-05-27T07:04:33.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:33 smithi043 ceph-mon[31531]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:04:33.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:33.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:33 smithi179 ceph-mon[35888]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-27T07:04:33.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:33 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/1692587796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10a2bd5e-5ac8-441f-b2e5-deb1f9239b17"}]: dispatch 2024-05-27T07:04:33.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:33 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10a2bd5e-5ac8-441f-b2e5-deb1f9239b17"}]: dispatch 2024-05-27T07:04:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:33 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10a2bd5e-5ac8-441f-b2e5-deb1f9239b17"}]': finished 2024-05-27T07:04:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:33 smithi179 ceph-mon[35888]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:04:33.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:33 smithi132 ceph-mon[35283]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-27T07:04:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:33 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/1692587796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10a2bd5e-5ac8-441f-b2e5-deb1f9239b17"}]: dispatch 2024-05-27T07:04:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:33 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10a2bd5e-5ac8-441f-b2e5-deb1f9239b17"}]: dispatch 2024-05-27T07:04:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:33 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10a2bd5e-5ac8-441f-b2e5-deb1f9239b17"}]': finished 2024-05-27T07:04:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:33 smithi132 ceph-mon[35283]: osdmap e51: 9 total, 8 up, 9 in 2024-05-27T07:04:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:34.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:34 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/1638631534' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:04:34.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:34 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/1638631534' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:04:35.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:34 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/1638631534' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:04:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:35 smithi132 ceph-mon[35283]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-27T07:04:36.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:35 smithi043 ceph-mon[31531]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-27T07:04:36.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:35 smithi179 ceph-mon[35888]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-27T07:04:38.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:37 smithi132 ceph-mon[35283]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:38.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:37 smithi043 ceph-mon[31531]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:37 smithi179 ceph-mon[35888]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:39.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:39 smithi179 ceph-mon[35888]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:39 smithi132 ceph-mon[35283]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:39 smithi043 ceph-mon[31531]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:41.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:41 smithi179 ceph-mon[35888]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:42.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:41 smithi132 ceph-mon[35283]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:42.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:41 smithi043 ceph-mon[31531]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:43.014 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:42 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:04:43.014 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:42 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:43.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:42 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:04:43.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:42 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:43.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:42 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-27T07:04:43.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:42 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:43.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:43 smithi179 ceph-mon[35888]: Deploying daemon osd.8 on smithi179 2024-05-27T07:04:43.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:43 smithi179 ceph-mon[35888]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:44.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:43 smithi132 ceph-mon[35283]: Deploying daemon osd.8 on smithi179 2024-05-27T07:04:44.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:43 smithi132 ceph-mon[35283]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:44.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:43 smithi043 ceph-mon[31531]: Deploying daemon osd.8 on smithi179 2024-05-27T07:04:44.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:43 smithi043 ceph-mon[31531]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:46.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:45 smithi132 ceph-mon[35283]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:46.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:45 smithi043 ceph-mon[31531]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:46.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:45 smithi179 ceph-mon[35888]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:48.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:47 smithi132 ceph-mon[35283]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:48.120 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:47 smithi179 ceph-mon[35888]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:47 smithi043 ceph-mon[31531]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:50.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:49 smithi132 ceph-mon[35283]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:50.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:49 smithi043 ceph-mon[31531]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:50.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:49 smithi179 ceph-mon[35888]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:52.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:51 smithi132 ceph-mon[35283]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:51 smithi043 ceph-mon[31531]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:51 smithi179 ceph-mon[35888]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:53.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:53.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:53.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:54.181 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:54 smithi179 ceph-mon[35888]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:54 smithi132 ceph-mon[35283]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:54.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:53 smithi043 ceph-mon[31531]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:56.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:56 smithi132 ceph-mon[35283]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:56.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:56 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:56.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:56 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:56.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:56 smithi043 ceph-mon[31531]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:56.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:56 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:56.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:56 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:56.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:56 smithi179 ceph-mon[35888]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:56.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:56 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:56.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:56 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:57.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:57 smithi132 ceph-mon[35283]: from='osd.8 [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:04:57.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:57 smithi132 ceph-mon[35283]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:04:57.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:57 smithi179 ceph-mon[35888]: from='osd.8 [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:04:57.429 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:57 smithi179 ceph-mon[35888]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:04:57.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:57 smithi043 ceph-mon[31531]: from='osd.8 [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:04:57.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:57 smithi043 ceph-mon[31531]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-27T07:04:57.933 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.941+0000 7f5db3400700 1 -- 172.21.15.179:0/2410587140 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5dbc063bf0 con 0x7f5da00b2900 2024-05-27T07:04:57.934 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 8 on host 'smithi179' 2024-05-27T07:04:57.937 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 -- 172.21.15.179:0/2410587140 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5da00b2900 msgr2=0x7f5da00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5da00b2900 0x7f5da00b4e10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5dac003820 tx=0x7f5dac01a040 comp rx=0 tx=0).stop 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 -- 172.21.15.179:0/2410587140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc108010 msgr2=0x7f5dbc1a3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc108010 0x7f5dbc1a3f30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5db4007ee0 tx=0x7f5db400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 -- 172.21.15.179:0/2410587140 shutdown_connections 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5da00b2900 0x7f5da00b4e10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc108010 0x7f5dbc1a3f30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5dbc106540 0x7f5dbc19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 --2- 172.21.15.179:0/2410587140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dbc107670 0x7f5dbc19cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 -- 172.21.15.179:0/2410587140 >> 172.21.15.179:0/2410587140 conn(0x7f5dbc101bc0 msgr2=0x7f5dbc10a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 -- 172.21.15.179:0/2410587140 shutdown_connections 2024-05-27T07:04:57.938 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:04:57.946+0000 7f5dc1c00700 1 -- 172.21.15.179:0/2410587140 wait complete. 2024-05-27T07:04:58.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: from='osd.8 [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:58.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:58.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:04:58.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: from='osd.8 [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:04:58.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:04:58.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:58.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:04:58.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:58.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:04:58.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:58.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:58.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:58.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:58.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:58.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-27T07:04:58.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: from='osd.8 [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:04:58.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: osdmap e52: 9 total, 8 up, 9 in 2024-05-27T07:04:58.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:58.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:04:58.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:04:58.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:04:58.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:58.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:58.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:04:58.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:04:58.803 DEBUG:teuthology.orchestra.run.smithi179:osd.8> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.8.service 2024-05-27T07:04:58.808 INFO:tasks.cephadm:Deploying osd.9 on smithi043 with /dev/vg_nvme/lv_1... 2024-05-27T07:04:58.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:04:59.008 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:04:58 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:04:58.901+0000 7f7201e00700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:04:59.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:59 smithi043 ceph-mon[31531]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:04:59.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:59 smithi043 ceph-mon[31531]: osdmap e53: 9 total, 8 up, 9 in 2024-05-27T07:04:59.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:59 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:59.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:59 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:59.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:59 smithi043 ceph-mon[31531]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:59.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:04:59 smithi043 ceph-mon[31531]: from='osd.8 ' entity='osd.8' 2024-05-27T07:04:59.264 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:59 smithi179 ceph-mon[35888]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:04:59.264 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:59 smithi179 ceph-mon[35888]: osdmap e53: 9 total, 8 up, 9 in 2024-05-27T07:04:59.264 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:59 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:59.264 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:59 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:59.265 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:59 smithi179 ceph-mon[35888]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:59.265 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:04:59 smithi179 ceph-mon[35888]: from='osd.8 ' entity='osd.8' 2024-05-27T07:04:59.290 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:04:59.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:59 smithi132 ceph-mon[35283]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:04:59.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:59 smithi132 ceph-mon[35283]: osdmap e53: 9 total, 8 up, 9 in 2024-05-27T07:04:59.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:59 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:59.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:59 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:04:59.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:59 smithi132 ceph-mon[35283]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-05-27T07:04:59.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:04:59 smithi132 ceph-mon[35283]: from='osd.8 ' entity='osd.8' 2024-05-27T07:05:00.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:00 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:05:00.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:00 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:05:00.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:00 smithi179 ceph-mon[35888]: osd.8 [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] boot 2024-05-27T07:05:00.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:00 smithi179 ceph-mon[35888]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:05:00.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:00 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:05:00.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:00 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:05:00.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:00 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:05:00.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:00 smithi132 ceph-mon[35283]: osd.8 [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] boot 2024-05-27T07:05:00.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:00 smithi132 ceph-mon[35283]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:05:00.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:00 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:05:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:00 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:05:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:00 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:05:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:00 smithi043 ceph-mon[31531]: osd.8 [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] boot 2024-05-27T07:05:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:00 smithi043 ceph-mon[31531]: osdmap e54: 9 total, 9 up, 9 in 2024-05-27T07:05:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:00 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:05:01.179 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:01 smithi043 ceph-mon[31531]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:05:01.179 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:01 smithi043 ceph-mon[31531]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:01.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:01 smithi179 ceph-mon[35888]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:05:01.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:01 smithi179 ceph-mon[35888]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:01.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:01 smithi132 ceph-mon[35283]: osdmap e55: 9 total, 9 up, 9 in 2024-05-27T07:05:01.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:01 smithi132 ceph-mon[35283]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:02.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:02 smithi132 ceph-mon[35283]: osdmap e56: 9 total, 9 up, 9 in 2024-05-27T07:05:02.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:02 smithi043 ceph-mon[31531]: osdmap e56: 9 total, 9 up, 9 in 2024-05-27T07:05:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:02 smithi179 ceph-mon[35888]: osdmap e56: 9 total, 9 up, 9 in 2024-05-27T07:05:02.526 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:05:02.526 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:05:02.526 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-05-27T07:05:02.526 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-05-27T07:05:02.527 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0373907 s, 280 MB/s 2024-05-27T07:05:02.527 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-05-27T07:05:03.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:03 smithi043 ceph-mon[31531]: pgmap v261: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:03.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:03 smithi132 ceph-mon[35283]: pgmap v261: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:03.432 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:03 smithi179 ceph-mon[35888]: pgmap v261: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:03.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_1 2024-05-27T07:05:03.942 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:05:04.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi179 2024-05-27T07:05:04.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:04.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:04.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:04.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi179 to 5696M 2024-05-27T07:05:04.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:04.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:05:04.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:04 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi179 2024-05-27T07:05:04.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:04.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:04.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:04.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi179 to 5696M 2024-05-27T07:05:04.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:04.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:05:04.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:04 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi179 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi179 to 5696M 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:05:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:04 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:05.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:05 smithi043 ceph-mon[31531]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:05.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:05 smithi132 ceph-mon[35283]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:05.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:05 smithi179 ceph-mon[35888]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:06.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.057+0000 7fc1f9c00700 1 -- 172.21.15.43:0/2667011845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1f410c410 msgr2=0x7fc1f4112cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:06.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.057+0000 7fc1f9c00700 1 --2- 172.21.15.43:0/2667011845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1f410c410 0x7fc1f4112cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00b0a0 tx=0x7fc1ec00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:05:06.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.057+0000 7fc1f9c00700 1 -- 172.21.15.43:0/2667011845 shutdown_connections 2024-05-27T07:05:06.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.057+0000 7fc1f9c00700 1 --2- 172.21.15.43:0/2667011845 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1f410c410 0x7fc1f4112cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:06.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.057+0000 7fc1f9c00700 1 --2- 172.21.15.43:0/2667011845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1f410ab00 0x7fc1f410af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:06.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.057+0000 7fc1f9c00700 1 --2- 172.21.15.43:0/2667011845 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc1f410ba70 0x7fc1f410be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:06.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.057+0000 7fc1f9c00700 1 -- 172.21.15.43:0/2667011845 >> 172.21.15.43:0/2667011845 conn(0x7fc1f40799b0 msgr2=0x7fc1f407be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:06.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.057+0000 7fc1f9c00700 1 -- 172.21.15.43:0/2667011845 shutdown_connections 2024-05-27T07:05:06.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.057+0000 7fc1f9c00700 1 -- 172.21.15.43:0/2667011845 wait complete. 2024-05-27T07:05:06.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.058+0000 7fc1f9c00700 1 Processor -- start 2024-05-27T07:05:06.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.059+0000 7fc1f9c00700 1 -- start start 2024-05-27T07:05:06.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.059+0000 7fc1f9c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc1f410ab00 0x7fc1f41a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:06.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.059+0000 7fc1f9c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1f410ba70 0x7fc1f41a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:06.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.059+0000 7fc1f9c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1f410c410 0x7fc1f41a8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:06.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.059+0000 7fc1f9c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f4116260 con 0x7fc1f410c410 2024-05-27T07:05:06.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.059+0000 7fc1f9c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f41160e0 con 0x7fc1f410ba70 2024-05-27T07:05:06.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.059+0000 7fc1f9c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f41163e0 con 0x7fc1f410ab00 2024-05-27T07:05:06.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.060+0000 7fc1f3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1f410c410 0x7fc1f41a8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:06.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.060+0000 7fc1f3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1f410c410 0x7fc1f41a8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39284/0 (socket says 172.21.15.43:39284) 2024-05-27T07:05:06.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.060+0000 7fc1f3400700 1 -- 172.21.15.43:0/4140314927 learned_addr learned my addr 172.21.15.43:0/4140314927 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:05:06.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.060+0000 7fc1f2000700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1f410ba70 0x7fc1f41a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:06.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.060+0000 7fc1f2a00700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc1f410ab00 0x7fc1f41a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.060+0000 7fc1f3400700 1 -- 172.21.15.43:0/4140314927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc1f410ab00 msgr2=0x7fc1f41a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.060+0000 7fc1f3400700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc1f410ab00 0x7fc1f41a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.060+0000 7fc1f3400700 1 -- 172.21.15.43:0/4140314927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1f410ba70 msgr2=0x7fc1f41a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.060+0000 7fc1f3400700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1f410ba70 0x7fc1f41a12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.060+0000 7fc1f3400700 1 -- 172.21.15.43:0/4140314927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f41a8900 con 0x7fc1f410c410 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.061+0000 7fc1f3400700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1f410c410 0x7fc1f41a8280 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec009030 tx=0x7fc1ec0097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.061+0000 7fc1eb400700 1 -- 172.21.15.43:0/4140314927 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec01f030 con 0x7fc1f410c410 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.061+0000 7fc1eb400700 1 -- 172.21.15.43:0/4140314927 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1ec005040 con 0x7fc1f410c410 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.061+0000 7fc1eb400700 1 -- 172.21.15.43:0/4140314927 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec01b690 con 0x7fc1f410c410 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.061+0000 7fc1f9c00700 1 -- 172.21.15.43:0/4140314927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f41a8b90 con 0x7fc1f410c410 2024-05-27T07:05:06.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.061+0000 7fc1f9c00700 1 -- 172.21.15.43:0/4140314927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1f41a8ff0 con 0x7fc1f410c410 2024-05-27T07:05:06.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.063+0000 7fc1f9c00700 1 -- 172.21.15.43:0/4140314927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1f40512b0 con 0x7fc1f410c410 2024-05-27T07:05:06.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.065+0000 7fc1eb400700 1 -- 172.21.15.43:0/4140314927 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc1ec022030 con 0x7fc1f410c410 2024-05-27T07:05:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.067+0000 7fc1eb400700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1d80b2900 0x7fc1d80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.067+0000 7fc1eb400700 1 -- 172.21.15.43:0/4140314927 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5829+0+0 (secure 0 0 0) 0x7fc1ec00f020 con 0x7fc1f410c410 2024-05-27T07:05:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.067+0000 7fc1f2a00700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1d80b2900 0x7fc1d80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:06.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.068+0000 7fc1f2a00700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1d80b2900 0x7fc1d80b4e10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc1f410b9a0 tx=0x7fc1e400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:06.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.070+0000 7fc1eb400700 1 -- 172.21.15.43:0/4140314927 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc1ec029020 con 0x7fc1f410c410 2024-05-27T07:05:06.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:06.319+0000 7fc1f9c00700 1 -- 172.21.15.43:0/4140314927 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc1f4063cb0 con 0x7fc1d80b2900 2024-05-27T07:05:06.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:05:06.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:05:06.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:06 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:06.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:06 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:05:06.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:06 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:05:06.861 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:06 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:06.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:06 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:05:06.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:06 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:05:06.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:06 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:07.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:07 smithi132 ceph-mon[35283]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:05:07.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:07 smithi132 ceph-mon[35283]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:05:07.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:07 smithi043 ceph-mon[31531]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:05:07.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:07 smithi043 ceph-mon[31531]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:05:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:07 smithi179 ceph-mon[35888]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:05:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:07 smithi179 ceph-mon[35888]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:05:09.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:09 smithi043 ceph-mon[31531]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-27T07:05:09.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:09 smithi179 ceph-mon[35888]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-27T07:05:10.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:09 smithi132 ceph-mon[35283]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-05-27T07:05:10.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:10 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4282842775' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19e77076-5052-4d5f-a35b-00bb595cf435"}]: dispatch 2024-05-27T07:05:10.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:10 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4282842775' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19e77076-5052-4d5f-a35b-00bb595cf435"}]': finished 2024-05-27T07:05:10.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:10 smithi043 ceph-mon[31531]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:05:10.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:10.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:10 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4282842775' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19e77076-5052-4d5f-a35b-00bb595cf435"}]: dispatch 2024-05-27T07:05:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:10 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4282842775' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19e77076-5052-4d5f-a35b-00bb595cf435"}]': finished 2024-05-27T07:05:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:10 smithi179 ceph-mon[35888]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:05:10.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:11.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:10 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4282842775' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19e77076-5052-4d5f-a35b-00bb595cf435"}]: dispatch 2024-05-27T07:05:11.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:10 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4282842775' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19e77076-5052-4d5f-a35b-00bb595cf435"}]': finished 2024-05-27T07:05:11.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:10 smithi132 ceph-mon[35283]: osdmap e57: 10 total, 9 up, 10 in 2024-05-27T07:05:11.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:11.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:11 smithi043 ceph-mon[31531]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:05:11.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:11 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3124001934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:05:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:11 smithi179 ceph-mon[35888]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:05:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:11 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3124001934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:05:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:11 smithi132 ceph-mon[35283]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-27T07:05:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:11 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3124001934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:05:13.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:13 smithi179 ceph-mon[35888]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:05:14.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:13 smithi132 ceph-mon[35283]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:05:14.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:13 smithi043 ceph-mon[31531]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:05:15.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:15 smithi043 ceph-mon[31531]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:05:16.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:15 smithi132 ceph-mon[35283]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:05:16.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:15 smithi179 ceph-mon[35888]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:05:18.063 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:17 smithi043 ceph-mon[31531]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:18.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:17 smithi132 ceph-mon[35283]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:18.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:17 smithi179 ceph-mon[35888]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:20.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:19 smithi132 ceph-mon[35283]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:20.147 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:19 smithi043 ceph-mon[31531]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:20.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:19 smithi179 ceph-mon[35888]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:21.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:05:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:21.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:05:21.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-27T07:05:21.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:22.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:21 smithi132 ceph-mon[35283]: Deploying daemon osd.9 on smithi043 2024-05-27T07:05:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:21 smithi132 ceph-mon[35283]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:22.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:21 smithi043 ceph-mon[31531]: Deploying daemon osd.9 on smithi043 2024-05-27T07:05:22.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:21 smithi043 ceph-mon[31531]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:21 smithi179 ceph-mon[35888]: Deploying daemon osd.9 on smithi043 2024-05-27T07:05:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:21 smithi179 ceph-mon[35888]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:23.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:23 smithi043 ceph-mon[31531]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:24.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:23 smithi132 ceph-mon[35283]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:24.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:23 smithi179 ceph-mon[35888]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:25.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:25 smithi132 ceph-mon[35283]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:25.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:25 smithi043 ceph-mon[31531]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:26.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:25 smithi179 ceph-mon[35888]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:28.037 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:27 smithi043 ceph-mon[31531]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:28.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:27 smithi132 ceph-mon[35283]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:28.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:27 smithi179 ceph-mon[35888]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:30.087 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:29 smithi043 ceph-mon[31531]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:30.087 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:29 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:30.087 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:29 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:30.087 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:29 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:05:30.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:29 smithi132 ceph-mon[35283]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:30.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:29 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:30.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:29 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:30.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:29 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:05:30.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:29 smithi179 ceph-mon[35888]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:30.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:29 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:30.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:29 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:30.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:29 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:05:31.887 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:31 smithi043 ceph-mon[31531]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:31 smithi132 ceph-mon[35283]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:31 smithi179 ceph-mon[35888]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:33.947 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:33 smithi043 ceph-mon[31531]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:33.947 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:33.947 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:33 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:34.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:33 smithi132 ceph-mon[35283]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:34.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:33 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:34.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:33 smithi179 ceph-mon[35888]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:34.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:33 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:35.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:34 smithi132 ceph-mon[35283]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:05:35.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:35.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:05:35.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:35.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:34 smithi043 ceph-mon[31531]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:05:35.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:35.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:05:35.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:35.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:34 smithi179 ceph-mon[35888]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-27T07:05:35.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:35.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:05:35.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.426+0000 7fc1eb400700 1 -- 172.21.15.43:0/4140314927 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc1f4063cb0 con 0x7fc1d80b2900 2024-05-27T07:05:35.420 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 9 on host 'smithi043' 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.431+0000 7fc1f9c00700 1 -- 172.21.15.43:0/4140314927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1d80b2900 msgr2=0x7fc1d80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.431+0000 7fc1f9c00700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1d80b2900 0x7fc1d80b4e10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc1f410b9a0 tx=0x7fc1e400b410 comp rx=0 tx=0).stop 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.431+0000 7fc1f9c00700 1 -- 172.21.15.43:0/4140314927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1f410c410 msgr2=0x7fc1f41a8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.431+0000 7fc1f9c00700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1f410c410 0x7fc1f41a8280 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec009030 tx=0x7fc1ec0097f0 comp rx=0 tx=0).stop 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.431+0000 7fc1f9c00700 1 -- 172.21.15.43:0/4140314927 shutdown_connections 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.431+0000 7fc1f9c00700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1d80b2900 0x7fc1d80b4e10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.431+0000 7fc1f9c00700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1f410c410 0x7fc1f41a8280 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.431+0000 7fc1f9c00700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc1f410ab00 0x7fc1f41a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.431+0000 7fc1f9c00700 1 --2- 172.21.15.43:0/4140314927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1f410ba70 0x7fc1f41a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.431+0000 7fc1f9c00700 1 -- 172.21.15.43:0/4140314927 >> 172.21.15.43:0/4140314927 conn(0x7fc1f40799b0 msgr2=0x7fc1f410eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.432+0000 7fc1f9c00700 1 -- 172.21.15.43:0/4140314927 shutdown_connections 2024-05-27T07:05:35.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:05:35.432+0000 7fc1f9c00700 1 -- 172.21.15.43:0/4140314927 wait complete. 2024-05-27T07:05:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:35 smithi132 ceph-mon[35283]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:35 smithi132 ceph-mon[35283]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:05:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:35 smithi132 ceph-mon[35283]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:05:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:35 smithi132 ceph-mon[35283]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T07:05:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:05:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:36.111 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:35 smithi179 ceph-mon[35888]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:36.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:35 smithi179 ceph-mon[35888]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:05:36.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:35 smithi179 ceph-mon[35888]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:05:36.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:35 smithi179 ceph-mon[35888]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T07:05:36.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:36.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:05:36.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:36.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:36.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:35 smithi043 ceph-mon[31531]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:36.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:35 smithi043 ceph-mon[31531]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-27T07:05:36.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:35 smithi043 ceph-mon[31531]: osdmap e58: 10 total, 9 up, 10 in 2024-05-27T07:05:36.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:35 smithi043 ceph-mon[31531]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-27T07:05:36.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:36.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:05:36.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:36.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:36.175 DEBUG:teuthology.orchestra.run.smithi043:osd.9> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.9.service 2024-05-27T07:05:36.179 INFO:tasks.cephadm:Deploying osd.10 on smithi132 with /dev/vg_nvme/lv_1... 2024-05-27T07:05:36.180 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:05:36.615 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:05:36.736 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:05:36 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:05:36.531+0000 7f2386c00700 -1 osd.9 0 waiting for initial osdmap 2024-05-27T07:05:36.736 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:05:36 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:05:36.573+0000 7f237ca00700 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T07:05:37.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:36 smithi132 ceph-mon[35283]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:05:37.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:36 smithi132 ceph-mon[35283]: osdmap e59: 10 total, 9 up, 10 in 2024-05-27T07:05:37.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:36 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:37.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:36 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:37.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:36 smithi132 ceph-mon[35283]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' 2024-05-27T07:05:37.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:36 smithi043 ceph-mon[31531]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:05:37.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:36 smithi043 ceph-mon[31531]: osdmap e59: 10 total, 9 up, 10 in 2024-05-27T07:05:37.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:36 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:37.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:36 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:37.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:36 smithi043 ceph-mon[31531]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' 2024-05-27T07:05:37.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:36 smithi179 ceph-mon[35888]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-27T07:05:37.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:36 smithi179 ceph-mon[35888]: osdmap e59: 10 total, 9 up, 10 in 2024-05-27T07:05:37.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:36 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:37.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:36 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:37.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:36 smithi179 ceph-mon[35888]: from='osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073]' entity='osd.9' 2024-05-27T07:05:38.085 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:37 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:05:38.085 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:37 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:05:38.085 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:37 smithi043 ceph-mon[31531]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:38.085 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:37 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:38.085 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:37 smithi043 ceph-mon[31531]: osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] boot 2024-05-27T07:05:38.085 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:37 smithi043 ceph-mon[31531]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:05:38.085 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:37 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:37 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:05:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:37 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:05:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:37 smithi132 ceph-mon[35283]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:37 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:37 smithi132 ceph-mon[35283]: osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] boot 2024-05-27T07:05:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:37 smithi132 ceph-mon[35283]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:05:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:37 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:37 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:05:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:37 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:05:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:37 smithi179 ceph-mon[35888]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-27T07:05:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:37 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:37 smithi179 ceph-mon[35888]: osd.9 [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] boot 2024-05-27T07:05:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:37 smithi179 ceph-mon[35888]: osdmap e60: 10 total, 10 up, 10 in 2024-05-27T07:05:38.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:37 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:05:39.823 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:05:39.823 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:05:39.823 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-05-27T07:05:39.823 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-05-27T07:05:39.823 INFO:teuthology.orchestra.run.smithi132.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0341727 s, 307 MB/s 2024-05-27T07:05:39.823 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-05-27T07:05:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:39 smithi132 ceph-mon[35283]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:39 smithi132 ceph-mon[35283]: osdmap e61: 10 total, 10 up, 10 in 2024-05-27T07:05:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:39 smithi043 ceph-mon[31531]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:39 smithi043 ceph-mon[31531]: osdmap e61: 10 total, 10 up, 10 in 2024-05-27T07:05:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:39 smithi179 ceph-mon[35888]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:39 smithi179 ceph-mon[35888]: osdmap e61: 10 total, 10 up, 10 in 2024-05-27T07:05:40.803 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_1 2024-05-27T07:05:41.242 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.b/config 2024-05-27T07:05:42.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi043 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi043 to 4273M 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:05:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:41 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:42.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi043 2024-05-27T07:05:42.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi043 to 4273M 2024-05-27T07:05:42.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:42.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:05:42.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:41 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi043 2024-05-27T07:05:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:05:42.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi043 to 4273M 2024-05-27T07:05:42.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:42.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:42.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:05:42.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:41 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:05:43.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.798+0000 7f60d9e00700 1 -- 172.21.15.132:0/192274735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60d4105c10 msgr2=0x7f60d410c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:43.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.798+0000 7f60d9e00700 1 --2- 172.21.15.132:0/192274735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60d4105c10 0x7f60d410c4c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00b0a0 tx=0x7f60cc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:05:43.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.798+0000 7f60d9e00700 1 -- 172.21.15.132:0/192274735 shutdown_connections 2024-05-27T07:05:43.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.798+0000 7f60d9e00700 1 --2- 172.21.15.132:0/192274735 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60d4105c10 0x7f60d410c4c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:43.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.798+0000 7f60d9e00700 1 --2- 172.21.15.132:0/192274735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d4104140 0x7f60d4104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:43.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.798+0000 7f60d9e00700 1 --2- 172.21.15.132:0/192274735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60d4105270 0x7f60d4105680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:43.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.798+0000 7f60d9e00700 1 -- 172.21.15.132:0/192274735 >> 172.21.15.132:0/192274735 conn(0x7f60d40ff7a0 msgr2=0x7f60d4101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:05:43.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.798+0000 7f60d9e00700 1 -- 172.21.15.132:0/192274735 shutdown_connections 2024-05-27T07:05:43.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.798+0000 7f60d9e00700 1 -- 172.21.15.132:0/192274735 wait complete. 2024-05-27T07:05:43.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.800+0000 7f60d9e00700 1 Processor -- start 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.800+0000 7f60d9e00700 1 -- start start 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.800+0000 7f60d9e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60d4104140 0x7f60d41a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.800+0000 7f60d9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d4105270 0x7f60d41a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.800+0000 7f60d9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60d4105c10 0x7f60d41a8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.800+0000 7f60d9e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f60d410fac0 con 0x7f60d4105c10 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.800+0000 7f60d9e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f60d410f940 con 0x7f60d4104140 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.800+0000 7f60d9e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f60d410fc40 con 0x7f60d4105270 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.800+0000 7f60d8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60d4104140 0x7f60d41a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.801+0000 7f60d8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60d4104140 0x7f60d41a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59254/0 (socket says 172.21.15.132:59254) 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.801+0000 7f60d8a00700 1 -- 172.21.15.132:0/1145868101 learned_addr learned my addr 172.21.15.132:0/1145868101 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.801+0000 7f60cbe00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d4105270 0x7f60d41a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:43.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.801+0000 7f60d9400700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60d4105c10 0x7f60d41a8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:43.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.801+0000 7f60d8a00700 1 -- 172.21.15.132:0/1145868101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d4105270 msgr2=0x7f60d41a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:43.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.801+0000 7f60d8a00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d4105270 0x7f60d41a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:43.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.801+0000 7f60d8a00700 1 -- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60d4105c10 msgr2=0x7f60d41a8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:05:43.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.801+0000 7f60d8a00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60d4105c10 0x7f60d41a8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:43.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.801+0000 7f60d8a00700 1 -- 172.21.15.132:0/1145868101 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d41a8740 con 0x7f60d4104140 2024-05-27T07:05:43.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.801+0000 7f60d9400700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60d4105c10 0x7f60d41a8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:05:43.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.802+0000 7f60d8a00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60d4104140 0x7f60d41a0c60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c970 tx=0x7f60c400cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:43.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.802+0000 7f60d2000700 1 -- 172.21.15.132:0/1145868101 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4007b00 con 0x7f60d4104140 2024-05-27T07:05:43.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.802+0000 7f60d2000700 1 -- 172.21.15.132:0/1145868101 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f60c4007ca0 con 0x7f60d4104140 2024-05-27T07:05:43.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.802+0000 7f60d2000700 1 -- 172.21.15.132:0/1145868101 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4018660 con 0x7f60d4104140 2024-05-27T07:05:43.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.802+0000 7f60d9e00700 1 -- 172.21.15.132:0/1145868101 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d41a8a30 con 0x7f60d4104140 2024-05-27T07:05:43.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.802+0000 7f60d9e00700 1 -- 172.21.15.132:0/1145868101 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d41a8f20 con 0x7f60d4104140 2024-05-27T07:05:43.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.804+0000 7f60d9e00700 1 -- 172.21.15.132:0/1145868101 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d40512b0 con 0x7f60d4104140 2024-05-27T07:05:43.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.808+0000 7f60d2000700 1 -- 172.21.15.132:0/1145868101 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f60c4004020 con 0x7f60d4104140 2024-05-27T07:05:43.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.810+0000 7f60d2000700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60b80b2880 0x7f60b80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:43.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.810+0000 7f60d2000700 1 -- 172.21.15.132:0/1145868101 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6261+0+0 (secure 0 0 0) 0x7f60c40e1c80 con 0x7f60d4104140 2024-05-27T07:05:43.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.810+0000 7f60cbe00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60b80b2880 0x7f60b80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:05:43.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.811+0000 7f60d2000700 1 -- 172.21.15.132:0/1145868101 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f60c4014030 con 0x7f60d4104140 2024-05-27T07:05:43.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:43.812+0000 7f60cbe00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60b80b2880 0x7f60b80b4d90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f60bc005950 tx=0x7f60bc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:05:44.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:05:44.052+0000 7f60d9e00700 1 -- 172.21.15.132:0/1145868101 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f60d4063bf0 con 0x7f60b80b2880 2024-05-27T07:05:44.045 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:43 smithi132 ceph-mon[35283]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:44.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:43 smithi043 ceph-mon[31531]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:44.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:43 smithi179 ceph-mon[35888]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:44.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:44 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:05:44.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:44 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:05:44.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:44 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:45.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:44 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:05:45.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:44 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:05:45.008 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:44 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:45.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:44 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:05:45.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:44 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:05:45.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:44 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:46.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:45 smithi132 ceph-mon[35283]: from='client.34337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:05:46.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:45 smithi132 ceph-mon[35283]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:46.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:45 smithi043 ceph-mon[31531]: from='client.34337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:05:46.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:45 smithi043 ceph-mon[31531]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:46.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:45 smithi179 ceph-mon[35888]: from='client.34337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:05:46.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:45 smithi179 ceph-mon[35888]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:48.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:47 smithi132 ceph-mon[35283]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:48.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:47 smithi043 ceph-mon[31531]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:48.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:47 smithi179 ceph-mon[35888]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:49.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:48 smithi132 ceph-mon[35283]: from='client.? 172.21.15.132:0/386327508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78b7db29-142a-4f61-a360-8139d14f676b"}]: dispatch 2024-05-27T07:05:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:48 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78b7db29-142a-4f61-a360-8139d14f676b"}]: dispatch 2024-05-27T07:05:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:48 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78b7db29-142a-4f61-a360-8139d14f676b"}]': finished 2024-05-27T07:05:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:48 smithi132 ceph-mon[35283]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:05:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:05:49.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:48 smithi043 ceph-mon[31531]: from='client.? 172.21.15.132:0/386327508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78b7db29-142a-4f61-a360-8139d14f676b"}]: dispatch 2024-05-27T07:05:49.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:48 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78b7db29-142a-4f61-a360-8139d14f676b"}]: dispatch 2024-05-27T07:05:49.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:48 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78b7db29-142a-4f61-a360-8139d14f676b"}]': finished 2024-05-27T07:05:49.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:48 smithi043 ceph-mon[31531]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:05:49.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:05:49.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:48 smithi179 ceph-mon[35888]: from='client.? 172.21.15.132:0/386327508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78b7db29-142a-4f61-a360-8139d14f676b"}]: dispatch 2024-05-27T07:05:49.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:48 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78b7db29-142a-4f61-a360-8139d14f676b"}]: dispatch 2024-05-27T07:05:49.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:48 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78b7db29-142a-4f61-a360-8139d14f676b"}]': finished 2024-05-27T07:05:49.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:48 smithi179 ceph-mon[35888]: osdmap e62: 11 total, 10 up, 11 in 2024-05-27T07:05:49.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:05:50.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:49 smithi132 ceph-mon[35283]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:50.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:49 smithi132 ceph-mon[35283]: from='client.? 172.21.15.132:0/3639384717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:05:50.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:49 smithi043 ceph-mon[31531]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:51.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:49 smithi043 ceph-mon[31531]: from='client.? 172.21.15.132:0/3639384717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:05:51.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:49 smithi179 ceph-mon[35888]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:51.348 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:49 smithi179 ceph-mon[35888]: from='client.? 172.21.15.132:0/3639384717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:05:52.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:51 smithi132 ceph-mon[35283]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:51 smithi043 ceph-mon[31531]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:51 smithi179 ceph-mon[35888]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:54.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:53 smithi132 ceph-mon[35283]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:53 smithi043 ceph-mon[31531]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:54.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:53 smithi179 ceph-mon[35888]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:56.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:55 smithi132 ceph-mon[35283]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:55 smithi043 ceph-mon[31531]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:55 smithi179 ceph-mon[35888]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:58.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:57 smithi132 ceph-mon[35283]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:57 smithi043 ceph-mon[31531]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:58.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:57 smithi179 ceph-mon[35888]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:05:59.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:05:59.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:59.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:05:59.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:05:59.303 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-27T07:05:59.303 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:00.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:59 smithi132 ceph-mon[35283]: Deploying daemon osd.10 on smithi132 2024-05-27T07:06:00.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:05:59 smithi132 ceph-mon[35283]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:00.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:59 smithi043 ceph-mon[31531]: Deploying daemon osd.10 on smithi132 2024-05-27T07:06:00.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:05:59 smithi043 ceph-mon[31531]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:00.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:59 smithi179 ceph-mon[35888]: Deploying daemon osd.10 on smithi132 2024-05-27T07:06:00.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:05:59 smithi179 ceph-mon[35888]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:02.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:01 smithi132 ceph-mon[35283]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:02.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:01 smithi043 ceph-mon[31531]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:01 smithi179 ceph-mon[35888]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:03.984 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:03 smithi132 ceph-mon[35283]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:04.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:03 smithi043 ceph-mon[31531]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:04.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:03 smithi179 ceph-mon[35888]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:06.148 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:05 smithi132 ceph-mon[35283]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:06.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:05 smithi043 ceph-mon[31531]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:05 smithi179 ceph-mon[35888]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:08.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:07 smithi132 ceph-mon[35283]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:08.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:07 smithi043 ceph-mon[31531]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:07 smithi179 ceph-mon[35888]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:09.202 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:08 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:09.202 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:08 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:09.202 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:08 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:09.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:09.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:09.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:08 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:09.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:08 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:09.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:08 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:09.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:08 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:10.238 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:10 smithi132 ceph-mon[35283]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:10.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:10 smithi043 ceph-mon[31531]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:10 smithi179 ceph-mon[35888]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:12 smithi132 ceph-mon[35283]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:12.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:12 smithi043 ceph-mon[31531]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:12 smithi179 ceph-mon[35888]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:13.243 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:13 smithi132 ceph-mon[35283]: from='osd.10 [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:06:13.243 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:13 smithi132 ceph-mon[35283]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:06:13.243 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:13 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:13.243 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:13 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:13.244 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:13 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:13.244 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:13 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:13.244 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:13 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:13.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:13 smithi043 ceph-mon[31531]: from='osd.10 [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:06:13.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:13 smithi043 ceph-mon[31531]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:06:13.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:13 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:13.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:13 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:13.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:13 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:13.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:13 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:13.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:13 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:13 smithi179 ceph-mon[35888]: from='osd.10 [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:06:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:13 smithi179 ceph-mon[35888]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-27T07:06:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:13 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:13 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:13 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:13 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:13.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:13 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:13.881 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 10 on host 'smithi132' 2024-05-27T07:06:13.882 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.888+0000 7f60d2000700 1 -- 172.21.15.132:0/1145868101 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f60d4063bf0 con 0x7f60b80b2880 2024-05-27T07:06:13.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.892+0000 7f60d9e00700 1 -- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60b80b2880 msgr2=0x7f60b80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:13.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.892+0000 7f60d9e00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60b80b2880 0x7f60b80b4d90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f60bc005950 tx=0x7f60bc0058e0 comp rx=0 tx=0).stop 2024-05-27T07:06:13.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.892+0000 7f60d9e00700 1 -- 172.21.15.132:0/1145868101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60d4104140 msgr2=0x7f60d41a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:13.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.892+0000 7f60d9e00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60d4104140 0x7f60d41a0c60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c970 tx=0x7f60c400cd30 comp rx=0 tx=0).stop 2024-05-27T07:06:13.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.893+0000 7f60d9e00700 1 -- 172.21.15.132:0/1145868101 shutdown_connections 2024-05-27T07:06:13.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.893+0000 7f60d9e00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60b80b2880 0x7f60b80b4d90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:13.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.893+0000 7f60d9e00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60d4105c10 0x7f60d41a8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:13.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.893+0000 7f60d9e00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60d4104140 0x7f60d41a0c60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:13.885 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.893+0000 7f60d9e00700 1 --2- 172.21.15.132:0/1145868101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d4105270 0x7f60d41a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:13.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.893+0000 7f60d9e00700 1 -- 172.21.15.132:0/1145868101 >> 172.21.15.132:0/1145868101 conn(0x7f60d40ff7a0 msgr2=0x7f60d4106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:13.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.893+0000 7f60d9e00700 1 -- 172.21.15.132:0/1145868101 shutdown_connections 2024-05-27T07:06:13.886 INFO:teuthology.orchestra.run.smithi132.stderr:2024-05-27T07:06:13.893+0000 7f60d9e00700 1 -- 172.21.15.132:0/1145868101 wait complete. 2024-05-27T07:06:14.319 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:14 smithi132 ceph-mon[35283]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:14.319 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:14 smithi132 ceph-mon[35283]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:06:14.319 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:14 smithi132 ceph-mon[35283]: from='osd.10 [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:06:14.319 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:14 smithi132 ceph-mon[35283]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:06:14.320 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:14 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:14.320 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:14 smithi132 ceph-mon[35283]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:06:14.320 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:14 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:14.320 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:14 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:14.320 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:14 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:14.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:14 smithi043 ceph-mon[31531]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:14.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:14 smithi043 ceph-mon[31531]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:06:14.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:14 smithi043 ceph-mon[31531]: from='osd.10 [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:06:14.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:14 smithi043 ceph-mon[31531]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:06:14.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:14 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:14.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:14 smithi043 ceph-mon[31531]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:06:14.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:14 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:14.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:14 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:14.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:14 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:14.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:14 smithi179 ceph-mon[35888]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:14.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:14 smithi179 ceph-mon[35888]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-27T07:06:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:14 smithi179 ceph-mon[35888]: from='osd.10 [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:06:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:14 smithi179 ceph-mon[35888]: osdmap e63: 11 total, 10 up, 11 in 2024-05-27T07:06:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:14 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:14 smithi179 ceph-mon[35888]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-05-27T07:06:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:14 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:14 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:14 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:14.779 DEBUG:teuthology.orchestra.run.smithi132:osd.10> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.10.service 2024-05-27T07:06:14.784 INFO:tasks.cephadm:Deploying osd.11 on smithi179 with /dev/vg_nvme/lv_1... 2024-05-27T07:06:14.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-27T07:06:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:15 smithi179 ceph-mon[35888]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:06:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:15 smithi179 ceph-mon[35888]: osdmap e64: 11 total, 10 up, 11 in 2024-05-27T07:06:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:15 smithi179 ceph-mon[35888]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:15.231 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:06:15.319 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:15 smithi132 ceph-mon[35283]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:06:15.319 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:15 smithi132 ceph-mon[35283]: osdmap e64: 11 total, 10 up, 11 in 2024-05-27T07:06:15.319 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:15.319 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:15.320 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:15 smithi132 ceph-mon[35283]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:15 smithi043 ceph-mon[31531]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-05-27T07:06:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:15 smithi043 ceph-mon[31531]: osdmap e64: 11 total, 10 up, 11 in 2024-05-27T07:06:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:15.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:15 smithi043 ceph-mon[31531]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-05-27T07:06:16.257 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:16 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:06:16.257 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:16 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:06:16.257 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:16 smithi132 ceph-mon[35283]: osd.10 [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] boot 2024-05-27T07:06:16.257 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:16 smithi132 ceph-mon[35283]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:06:16.257 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:16.257 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:16 smithi132 ceph-mon[35283]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:06:16.404 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:16 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:06:16.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:16 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:06:16.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:16 smithi179 ceph-mon[35888]: osd.10 [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] boot 2024-05-27T07:06:16.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:16 smithi179 ceph-mon[35888]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:06:16.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:16.405 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:16 smithi179 ceph-mon[35888]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:06:16.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:16 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:06:16.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:16 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:06:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:16 smithi043 ceph-mon[31531]: osd.10 [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] boot 2024-05-27T07:06:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:16 smithi043 ceph-mon[31531]: osdmap e65: 11 total, 11 up, 11 in 2024-05-27T07:06:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:06:16.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:16 smithi043 ceph-mon[31531]: osdmap e66: 11 total, 11 up, 11 in 2024-05-27T07:06:17.855 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:17 smithi179 ceph-mon[35888]: pgmap v308: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:17.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:17 smithi179 ceph-mon[35888]: osdmap e67: 11 total, 11 up, 11 in 2024-05-27T07:06:17.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:17 smithi043 ceph-mon[31531]: pgmap v308: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:17.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:17 smithi043 ceph-mon[31531]: osdmap e67: 11 total, 11 up, 11 in 2024-05-27T07:06:18.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:17 smithi132 ceph-mon[35283]: pgmap v308: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:18.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:17 smithi132 ceph-mon[35283]: osdmap e67: 11 total, 11 up, 11 in 2024-05-27T07:06:18.146 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-27T07:06:18.147 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-27T07:06:18.147 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-05-27T07:06:18.147 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-05-27T07:06:18.147 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0337446 s, 311 MB/s 2024-05-27T07:06:18.147 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-05-27T07:06:19.080 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_1 2024-05-27T07:06:19.528 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:06:19.928 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: pgmap v310: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:19.929 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:19.929 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:19.929 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:19.929 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:19.929 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:19.929 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:19.929 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:19.929 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:19.929 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:19.929 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:20.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: pgmap v310: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:20.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:20.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:20.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:20.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:20.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:20.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:20.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:20.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:20.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:20.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:20.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: pgmap v310: 1 pgs: 1 peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:20.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:20.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:20.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:20.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:20.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:20.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:20.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:20.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:20.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:20.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:20 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi132 2024-05-27T07:06:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:20 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi132 to 4272M 2024-05-27T07:06:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:20 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi132 2024-05-27T07:06:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:20 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi132 to 4272M 2024-05-27T07:06:21.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:20 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi132 2024-05-27T07:06:21.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:20 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 4272M 2024-05-27T07:06:21.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.899+0000 7f5a79e00700 1 -- 172.21.15.179:0/1911333798 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a74107280 msgr2=0x7f5a74109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:21.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.899+0000 7f5a79e00700 1 --2- 172.21.15.179:0/1911333798 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a74107280 0x7f5a74109760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009990 tx=0x7f5a60009ca0 comp rx=0 tx=0).stop 2024-05-27T07:06:21.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.900+0000 7f5a79e00700 1 -- 172.21.15.179:0/1911333798 shutdown_connections 2024-05-27T07:06:21.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.900+0000 7f5a79e00700 1 --2- 172.21.15.179:0/1911333798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a74109df0 0x7f5a7410c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:21.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.900+0000 7f5a79e00700 1 --2- 172.21.15.179:0/1911333798 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a7406c2c0 0x7f5a74106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:21.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.900+0000 7f5a79e00700 1 --2- 172.21.15.179:0/1911333798 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a74107280 0x7f5a74109760 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:21.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.900+0000 7f5a79e00700 1 -- 172.21.15.179:0/1911333798 >> 172.21.15.179:0/1911333798 conn(0x7f5a740fe680 msgr2=0x7f5a74100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:21.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.900+0000 7f5a79e00700 1 -- 172.21.15.179:0/1911333798 shutdown_connections 2024-05-27T07:06:21.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.900+0000 7f5a79e00700 1 -- 172.21.15.179:0/1911333798 wait complete. 2024-05-27T07:06:21.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:21 smithi179 ceph-mon[35888]: pgmap v311: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.901+0000 7f5a79e00700 1 Processor -- start 2024-05-27T07:06:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.901+0000 7f5a79e00700 1 -- start start 2024-05-27T07:06:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a79e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7406c2c0 0x7f5a7419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a79e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a74107280 0x7f5a7419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a79e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a74109df0 0x7f5a741a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a79e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7410fa90 con 0x7f5a74107280 2024-05-27T07:06:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a79e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7410f910 con 0x7f5a74109df0 2024-05-27T07:06:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a79e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7410fc10 con 0x7f5a7406c2c0 2024-05-27T07:06:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a78a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7406c2c0 0x7f5a7419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:21.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a78a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7406c2c0 0x7f5a7419c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:35218/0 (socket says 172.21.15.179:35218) 2024-05-27T07:06:21.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a78a00700 1 -- 172.21.15.179:0/1988498403 learned_addr learned my addr 172.21.15.179:0/1988498403 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-05-27T07:06:21.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a73e00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a74107280 0x7f5a7419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:21.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a79400700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a74109df0 0x7f5a741a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:21.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a78a00700 1 -- 172.21.15.179:0/1988498403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a74109df0 msgr2=0x7f5a741a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:21.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a78a00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a74109df0 0x7f5a741a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:21.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a78a00700 1 -- 172.21.15.179:0/1988498403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a74107280 msgr2=0x7f5a7419ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:21.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.902+0000 7f5a78a00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a74107280 0x7f5a7419ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:21.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.903+0000 7f5a78a00700 1 -- 172.21.15.179:0/1988498403 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a741a43c0 con 0x7f5a7406c2c0 2024-05-27T07:06:21.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.903+0000 7f5a78a00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7406c2c0 0x7f5a7419c8f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800c9b0 tx=0x7f5a6800ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:21.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.903+0000 7f5a71600700 1 -- 172.21.15.179:0/1988498403 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68007b00 con 0x7f5a7406c2c0 2024-05-27T07:06:21.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.903+0000 7f5a79e00700 1 -- 172.21.15.179:0/1988498403 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a741a46b0 con 0x7f5a7406c2c0 2024-05-27T07:06:21.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.903+0000 7f5a71600700 1 -- 172.21.15.179:0/1988498403 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a68007ca0 con 0x7f5a7406c2c0 2024-05-27T07:06:21.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.903+0000 7f5a71600700 1 -- 172.21.15.179:0/1988498403 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68018660 con 0x7f5a7406c2c0 2024-05-27T07:06:21.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.903+0000 7f5a79e00700 1 -- 172.21.15.179:0/1988498403 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a741a4bf0 con 0x7f5a7406c2c0 2024-05-27T07:06:21.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.904+0000 7f5a79e00700 1 -- 172.21.15.179:0/1988498403 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a740512b0 con 0x7f5a7406c2c0 2024-05-27T07:06:21.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.907+0000 7f5a71600700 1 -- 172.21.15.179:0/1988498403 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5a6800f450 con 0x7f5a7406c2c0 2024-05-27T07:06:21.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.909+0000 7f5a71600700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a5c0b2880 0x7f5a5c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:21.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.909+0000 7f5a71600700 1 -- 172.21.15.179:0/1988498403 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6694+0+0 (secure 0 0 0) 0x7f5a680e1e80 con 0x7f5a7406c2c0 2024-05-27T07:06:21.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.910+0000 7f5a73e00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a5c0b2880 0x7f5a5c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:21.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.911+0000 7f5a73e00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a5c0b2880 0x7f5a5c0b4d90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5a7419d9a0 tx=0x7f5a6001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:21.904 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:21.912+0000 7f5a71600700 1 -- 172.21.15.179:0/1988498403 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5a68014030 con 0x7f5a7406c2c0 2024-05-27T07:06:21.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:21 smithi043 ceph-mon[31531]: pgmap v311: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:21 smithi132 ceph-mon[35283]: pgmap v311: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:22.147 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:22.156+0000 7f5a79e00700 1 -- 172.21.15.179:0/1988498403 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5a741a4eb0 con 0x7f5a5c0b2880 2024-05-27T07:06:23.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:22 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:06:23.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:22 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:06:23.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:22 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:23.130 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:22 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:06:23.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:22 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:06:23.131 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:22 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:23.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:22 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-27T07:06:23.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:22 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-27T07:06:23.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:22 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:24.007 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:23 smithi179 ceph-mon[35888]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:24.008 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:23 smithi179 ceph-mon[35888]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:06:24.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:23 smithi132 ceph-mon[35283]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:24.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:23 smithi132 ceph-mon[35283]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:06:24.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:23 smithi043 ceph-mon[31531]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:06:24.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:23 smithi043 ceph-mon[31531]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-05-27T07:06:26.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:25 smithi132 ceph-mon[35283]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-27T07:06:26.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:25 smithi043 ceph-mon[31531]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-27T07:06:26.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:25 smithi179 ceph-mon[35888]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-27T07:06:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:26 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/186783255' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfe7c464-567c-4836-8042-4bb9daedb76b"}]: dispatch 2024-05-27T07:06:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:26 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfe7c464-567c-4836-8042-4bb9daedb76b"}]: dispatch 2024-05-27T07:06:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:26 smithi132 ceph-mon[35283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfe7c464-567c-4836-8042-4bb9daedb76b"}]': finished 2024-05-27T07:06:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:26 smithi132 ceph-mon[35283]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:06:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:27.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:26 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/186783255' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfe7c464-567c-4836-8042-4bb9daedb76b"}]: dispatch 2024-05-27T07:06:27.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:26 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfe7c464-567c-4836-8042-4bb9daedb76b"}]: dispatch 2024-05-27T07:06:27.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:26 smithi179 ceph-mon[35888]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfe7c464-567c-4836-8042-4bb9daedb76b"}]': finished 2024-05-27T07:06:27.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:26 smithi179 ceph-mon[35888]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:06:27.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:27.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:26 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/186783255' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfe7c464-567c-4836-8042-4bb9daedb76b"}]: dispatch 2024-05-27T07:06:27.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:26 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfe7c464-567c-4836-8042-4bb9daedb76b"}]: dispatch 2024-05-27T07:06:27.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:26 smithi043 ceph-mon[31531]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfe7c464-567c-4836-8042-4bb9daedb76b"}]': finished 2024-05-27T07:06:27.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:26 smithi043 ceph-mon[31531]: osdmap e68: 12 total, 11 up, 12 in 2024-05-27T07:06:27.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:28.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:27 smithi132 ceph-mon[35283]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:06:28.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:27 smithi132 ceph-mon[35283]: from='client.? 172.21.15.179:0/1743068112' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:06:28.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:27 smithi043 ceph-mon[31531]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:06:28.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:27 smithi043 ceph-mon[31531]: from='client.? 172.21.15.179:0/1743068112' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:06:28.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:27 smithi179 ceph-mon[35888]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:06:28.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:27 smithi179 ceph-mon[35888]: from='client.? 172.21.15.179:0/1743068112' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-27T07:06:30.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:29 smithi132 ceph-mon[35283]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:06:30.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:29 smithi043 ceph-mon[31531]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:06:30.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:29 smithi179 ceph-mon[35888]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:06:31.907 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:31 smithi179 ceph-mon[35888]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:06:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:31 smithi132 ceph-mon[35283]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:06:32.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:31 smithi043 ceph-mon[31531]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-27T07:06:34.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:33 smithi132 ceph-mon[35283]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:34.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:33 smithi043 ceph-mon[31531]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:34.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:33 smithi179 ceph-mon[35888]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:36.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:35 smithi132 ceph-mon[35283]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:36.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:35 smithi043 ceph-mon[31531]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:36.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:35 smithi179 ceph-mon[35888]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:37.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:36 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:06:37.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:36 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:37.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:36 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:06:37.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:36 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:37.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:36 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-27T07:06:37.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:36 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:38.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:37 smithi132 ceph-mon[35283]: Deploying daemon osd.11 on smithi179 2024-05-27T07:06:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:37 smithi132 ceph-mon[35283]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:38.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:37 smithi043 ceph-mon[31531]: Deploying daemon osd.11 on smithi179 2024-05-27T07:06:38.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:37 smithi043 ceph-mon[31531]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:37 smithi179 ceph-mon[35888]: Deploying daemon osd.11 on smithi179 2024-05-27T07:06:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:37 smithi179 ceph-mon[35888]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:39 smithi132 ceph-mon[35283]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:39 smithi043 ceph-mon[31531]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:39 smithi179 ceph-mon[35888]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:42.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:41 smithi132 ceph-mon[35283]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:42.108 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:41 smithi179 ceph-mon[35888]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:42.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:41 smithi043 ceph-mon[31531]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:44.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:43 smithi132 ceph-mon[35283]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:44.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:43 smithi043 ceph-mon[31531]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:44.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:43 smithi179 ceph-mon[35888]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:46.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:45 smithi132 ceph-mon[35283]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:46.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:45 smithi043 ceph-mon[31531]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:46.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:45 smithi179 ceph-mon[35888]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:47.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:47.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:47.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:47 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:47.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:47 smithi132 ceph-mon[35283]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:47.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:47.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:47.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:47 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:47.406 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:47 smithi179 ceph-mon[35888]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:47.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:47.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:47.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:47 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:47.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:47 smithi043 ceph-mon[31531]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:49.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:49 smithi179 ceph-mon[35888]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:50.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:49 smithi132 ceph-mon[35283]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:50.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:49 smithi043 ceph-mon[31531]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:51.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:50 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:51.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:50 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:51.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:50 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:51.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:50 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:51.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:50 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:51.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:50 smithi043 ceph-mon[31531]: from='osd.11 [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:06:51.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:50 smithi043 ceph-mon[31531]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:06:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:50 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:50 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:50 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:50 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:50 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:50 smithi179 ceph-mon[35888]: from='osd.11 [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:06:51.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:50 smithi179 ceph-mon[35888]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:06:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:50 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:50 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:50 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:50 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:50 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:50 smithi132 ceph-mon[35283]: from='osd.11 [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:06:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:50 smithi132 ceph-mon[35283]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-27T07:06:51.925 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 11 on host 'smithi179' 2024-05-27T07:06:51.925 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.933+0000 7f5a71600700 1 -- 172.21.15.179:0/1988498403 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f5a741a4eb0 con 0x7f5a5c0b2880 2024-05-27T07:06:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 -- 172.21.15.179:0/1988498403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a5c0b2880 msgr2=0x7f5a5c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a5c0b2880 0x7f5a5c0b4d90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5a7419d9a0 tx=0x7f5a6001a040 comp rx=0 tx=0).stop 2024-05-27T07:06:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 -- 172.21.15.179:0/1988498403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7406c2c0 msgr2=0x7f5a7419c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7406c2c0 0x7f5a7419c8f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800c9b0 tx=0x7f5a6800ccc0 comp rx=0 tx=0).stop 2024-05-27T07:06:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 -- 172.21.15.179:0/1988498403 shutdown_connections 2024-05-27T07:06:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a5c0b2880 0x7f5a5c0b4d90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a74109df0 0x7f5a741a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:51.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7406c2c0 0x7f5a7419c8f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:51.932 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 --2- 172.21.15.179:0/1988498403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a74107280 0x7f5a7419ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:51.932 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 -- 172.21.15.179:0/1988498403 >> 172.21.15.179:0/1988498403 conn(0x7f5a740fe680 msgr2=0x7f5a74100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:51.932 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 -- 172.21.15.179:0/1988498403 shutdown_connections 2024-05-27T07:06:51.932 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:06:51.937+0000 7f5a79e00700 1 -- 172.21.15.179:0/1988498403 wait complete. 2024-05-27T07:06:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:52 smithi179 ceph-mon[35888]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:52 smithi179 ceph-mon[35888]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:06:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:52 smithi179 ceph-mon[35888]: from='osd.11 [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:06:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:52 smithi179 ceph-mon[35888]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:06:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:52.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:52 smithi179 ceph-mon[35888]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:06:52.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:52.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:52.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:52 smithi132 ceph-mon[35283]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:52 smithi132 ceph-mon[35283]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:06:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:52 smithi132 ceph-mon[35283]: from='osd.11 [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:06:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:52 smithi132 ceph-mon[35283]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:06:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:52 smithi132 ceph-mon[35283]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:06:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:52.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:52.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:52 smithi043 ceph-mon[31531]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:52.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:52 smithi043 ceph-mon[31531]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-27T07:06:52.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:52 smithi043 ceph-mon[31531]: from='osd.11 [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:06:52.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:52 smithi043 ceph-mon[31531]: osdmap e69: 12 total, 11 up, 12 in 2024-05-27T07:06:52.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:52.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:52 smithi043 ceph-mon[31531]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-05-27T07:06:52.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:52.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:52.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:06:52.794 DEBUG:teuthology.orchestra.run.smithi179:osd.11> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.11.service 2024-05-27T07:06:52.799 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-27T07:06:52.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-05-27T07:06:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:53 smithi043 ceph-mon[31531]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:06:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:53 smithi043 ceph-mon[31531]: osdmap e70: 12 total, 11 up, 12 in 2024-05-27T07:06:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:53 smithi043 ceph-mon[31531]: from='osd.11 ' entity='osd.11' 2024-05-27T07:06:53.271 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:06:53.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:53 smithi179 ceph-mon[35888]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:06:53.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:53 smithi179 ceph-mon[35888]: osdmap e70: 12 total, 11 up, 12 in 2024-05-27T07:06:53.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:53.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:53.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:53 smithi179 ceph-mon[35888]: from='osd.11 ' entity='osd.11' 2024-05-27T07:06:53.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:53 smithi132 ceph-mon[35283]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-05-27T07:06:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:53 smithi132 ceph-mon[35283]: osdmap e70: 12 total, 11 up, 12 in 2024-05-27T07:06:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:53 smithi132 ceph-mon[35283]: from='osd.11 ' entity='osd.11' 2024-05-27T07:06:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:54 smithi132 ceph-mon[35283]: purged_snaps scrub starts 2024-05-27T07:06:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:54 smithi132 ceph-mon[35283]: purged_snaps scrub ok 2024-05-27T07:06:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:54 smithi132 ceph-mon[35283]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:54 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:54 smithi132 ceph-mon[35283]: osd.11 [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] boot 2024-05-27T07:06:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:54 smithi132 ceph-mon[35283]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:06:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:54 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:54.387 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:54 smithi179 ceph-mon[35888]: purged_snaps scrub starts 2024-05-27T07:06:54.388 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:54 smithi179 ceph-mon[35888]: purged_snaps scrub ok 2024-05-27T07:06:54.388 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:54 smithi179 ceph-mon[35888]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:54.388 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:54 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:54.388 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:54 smithi179 ceph-mon[35888]: osd.11 [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] boot 2024-05-27T07:06:54.388 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:54 smithi179 ceph-mon[35888]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:06:54.388 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:54 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:54.407 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:54 smithi043 ceph-mon[31531]: purged_snaps scrub starts 2024-05-27T07:06:54.407 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:54 smithi043 ceph-mon[31531]: purged_snaps scrub ok 2024-05-27T07:06:54.407 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:54 smithi043 ceph-mon[31531]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-05-27T07:06:54.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:54 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:54.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:54 smithi043 ceph-mon[31531]: osd.11 [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] boot 2024-05-27T07:06:54.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:54 smithi043 ceph-mon[31531]: osdmap e71: 12 total, 12 up, 12 in 2024-05-27T07:06:54.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:54 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:06:55.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.957+0000 7f7d72800700 1 -- 172.21.15.43:0/2116583472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d6c10c1f0 msgr2=0x7f7d6c10e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:55.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.957+0000 7f7d72800700 1 --2- 172.21.15.43:0/2116583472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d6c10c1f0 0x7f7d6c10e700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400b0a0 tx=0x7f7d6400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:06:55.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.958+0000 7f7d72800700 1 -- 172.21.15.43:0/2116583472 shutdown_connections 2024-05-27T07:06:55.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.958+0000 7f7d72800700 1 --2- 172.21.15.43:0/2116583472 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d6c10c1f0 0x7f7d6c10e700 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:55.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.958+0000 7f7d72800700 1 --2- 172.21.15.43:0/2116583472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d6c077670 0x7f7d6c075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:55.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.958+0000 7f7d72800700 1 --2- 172.21.15.43:0/2116583472 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d6c076080 0x7f7d6c0764b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:55.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.958+0000 7f7d72800700 1 -- 172.21.15.43:0/2116583472 >> 172.21.15.43:0/2116583472 conn(0x7f7d6c0ff9b0 msgr2=0x7f7d6c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:55.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.958+0000 7f7d72800700 1 -- 172.21.15.43:0/2116583472 shutdown_connections 2024-05-27T07:06:55.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.958+0000 7f7d72800700 1 -- 172.21.15.43:0/2116583472 wait complete. 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.959+0000 7f7d72800700 1 Processor -- start 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.959+0000 7f7d72800700 1 -- start start 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.959+0000 7f7d72800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d6c076080 0x7f7d6c19c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.959+0000 7f7d72800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d6c077670 0x7f7d6c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d72800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d6c10c1f0 0x7f7d6c1a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d72800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c111e90 con 0x7f7d6c077670 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d72800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c111d10 con 0x7f7d6c10c1f0 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d72800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c112010 con 0x7f7d6c076080 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d6c077670 0x7f7d6c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d6c077670 0x7f7d6c19cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43476/0 (socket says 172.21.15.43:43476) 2024-05-27T07:06:55.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6aa00700 1 -- 172.21.15.43:0/3725385020 learned_addr learned my addr 172.21.15.43:0/3725385020 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:06:55.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6b400700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d6c076080 0x7f7d6c19c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:55.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6be00700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d6c10c1f0 0x7f7d6c1a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:55.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6aa00700 1 -- 172.21.15.43:0/3725385020 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d6c076080 msgr2=0x7f7d6c19c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:55.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6aa00700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d6c076080 0x7f7d6c19c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:55.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6aa00700 1 -- 172.21.15.43:0/3725385020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d6c10c1f0 msgr2=0x7f7d6c1a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:55.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6aa00700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d6c10c1f0 0x7f7d6c1a3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:55.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6aa00700 1 -- 172.21.15.43:0/3725385020 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c1a4510 con 0x7f7d6c077670 2024-05-27T07:06:55.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6b400700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d6c076080 0x7f7d6c19c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:06:55.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.960+0000 7f7d6be00700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d6c10c1f0 0x7f7d6c1a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:06:55.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.961+0000 7f7d6aa00700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d6c077670 0x7f7d6c19cef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7d5800d950 tx=0x7f7d5800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:55.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.961+0000 7f7d63e00700 1 -- 172.21.15.43:0/3725385020 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58016070 con 0x7f7d6c077670 2024-05-27T07:06:55.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.961+0000 7f7d63e00700 1 -- 172.21.15.43:0/3725385020 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d58010460 con 0x7f7d6c077670 2024-05-27T07:06:55.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.961+0000 7f7d63e00700 1 -- 172.21.15.43:0/3725385020 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5800f620 con 0x7f7d6c077670 2024-05-27T07:06:55.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.961+0000 7f7d72800700 1 -- 172.21.15.43:0/3725385020 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6c1a4800 con 0x7f7d6c077670 2024-05-27T07:06:55.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.961+0000 7f7d72800700 1 -- 172.21.15.43:0/3725385020 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6c1a4cf0 con 0x7f7d6c077670 2024-05-27T07:06:55.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.963+0000 7f7d72800700 1 -- 172.21.15.43:0/3725385020 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d6c0512b0 con 0x7f7d6c077670 2024-05-27T07:06:55.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.966+0000 7f7d63e00700 1 -- 172.21.15.43:0/3725385020 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f7d58019070 con 0x7f7d6c077670 2024-05-27T07:06:55.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.968+0000 7f7d63e00700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7d500b2900 0x7f7d500b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:55.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.968+0000 7f7d63e00700 1 -- 172.21.15.43:0/3725385020 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f7d580e19a0 con 0x7f7d6c077670 2024-05-27T07:06:55.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.968+0000 7f7d6b400700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7d500b2900 0x7f7d500b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:55.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.969+0000 7f7d6b400700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7d500b2900 0x7f7d500b4e10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c005950 tx=0x7f7d5c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:55.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:55.970+0000 7f7d63e00700 1 -- 172.21.15.43:0/3725385020 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7d580a9ac0 con 0x7f7d6c077670 2024-05-27T07:06:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:56 smithi043 ceph-mon[31531]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:06:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:56 smithi043 ceph-mon[31531]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:06:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.211+0000 7f7d72800700 1 -- 172.21.15.43:0/3725385020 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f7d6c02e5e0 con 0x7f7d6c077670 2024-05-27T07:06:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.212+0000 7f7d63e00700 1 -- 172.21.15.43:0/3725385020 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f7d580a93c0 con 0x7f7d6c077670 2024-05-27T07:06:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:06:56.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.215+0000 7f7d72800700 1 -- 172.21.15.43:0/3725385020 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7d500b2900 msgr2=0x7f7d500b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:56.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.215+0000 7f7d72800700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7d500b2900 0x7f7d500b4e10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c005950 tx=0x7f7d5c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:06:56.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.215+0000 7f7d72800700 1 -- 172.21.15.43:0/3725385020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d6c077670 msgr2=0x7f7d6c19cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:56.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.215+0000 7f7d72800700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d6c077670 0x7f7d6c19cef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7d5800d950 tx=0x7f7d5800dd10 comp rx=0 tx=0).stop 2024-05-27T07:06:56.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.216+0000 7f7d72800700 1 -- 172.21.15.43:0/3725385020 shutdown_connections 2024-05-27T07:06:56.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.216+0000 7f7d72800700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7d500b2900 0x7f7d500b4e10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:56.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.216+0000 7f7d72800700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d6c10c1f0 0x7f7d6c1a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:56.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.216+0000 7f7d72800700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d6c076080 0x7f7d6c19c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:56.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.216+0000 7f7d72800700 1 --2- 172.21.15.43:0/3725385020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d6c077670 0x7f7d6c19cef0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:56.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.216+0000 7f7d72800700 1 -- 172.21.15.43:0/3725385020 >> 172.21.15.43:0/3725385020 conn(0x7f7d6c0ff9b0 msgr2=0x7f7d6c1081d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:56.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.216+0000 7f7d72800700 1 -- 172.21.15.43:0/3725385020 shutdown_connections 2024-05-27T07:06:56.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:56.216+0000 7f7d72800700 1 -- 172.21.15.43:0/3725385020 wait complete. 2024-05-27T07:06:56.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:56 smithi132 ceph-mon[35283]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:06:56.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:56 smithi132 ceph-mon[35283]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:06:56.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:56 smithi179 ceph-mon[35888]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:06:56.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:56 smithi179 ceph-mon[35888]: osdmap e72: 12 total, 12 up, 12 in 2024-05-27T07:06:56.974 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1716793613,"num_in_osds":12,"osd_in_since":1716793586,"num_remapped_pgs":0} 2024-05-27T07:06:56.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:06:57.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:57 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3725385020' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:06:57.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:57 smithi043 ceph-mon[31531]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:06:57.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:57 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3725385020' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:06:57.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:57 smithi132 ceph-mon[35283]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:06:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:06:57.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:57 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3725385020' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-27T07:06:57.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:57 smithi179 ceph-mon[35888]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:06:58.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: Detected new or changed devices on smithi179 2024-05-27T07:06:58.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi179 to 4272M 2024-05-27T07:06:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: Detected new or changed devices on smithi179 2024-05-27T07:06:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi179 to 4272M 2024-05-27T07:06:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:58.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:58.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:58.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:59.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: Detected new or changed devices on smithi179 2024-05-27T07:06:59.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:59.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:59.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:59.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:59.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:59.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:06:59.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi179 to 4272M 2024-05-27T07:06:59.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:59.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:06:59.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:06:59.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:06:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.686+0000 7fc165600700 1 -- 172.21.15.43:0/1868118955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1601030f0 msgr2=0x7fc160103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.686+0000 7fc165600700 1 --2- 172.21.15.43:0/1868118955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1601030f0 0x7fc160103500 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc150009990 tx=0x7fc150009ca0 comp rx=0 tx=0).stop 2024-05-27T07:06:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.686+0000 7fc165600700 1 -- 172.21.15.43:0/1868118955 shutdown_connections 2024-05-27T07:06:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.687+0000 7fc165600700 1 --2- 172.21.15.43:0/1868118955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc160110ca0 0x7fc1601130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.687+0000 7fc165600700 1 --2- 172.21.15.43:0/1868118955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc1601030f0 0x7fc160103500 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.687+0000 7fc165600700 1 --2- 172.21.15.43:0/1868118955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc160103a40 0x7fc160110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.687+0000 7fc165600700 1 -- 172.21.15.43:0/1868118955 >> 172.21.15.43:0/1868118955 conn(0x7fc1600fedb0 msgr2=0x7fc1601011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.687+0000 7fc165600700 1 -- 172.21.15.43:0/1868118955 shutdown_connections 2024-05-27T07:06:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.687+0000 7fc165600700 1 -- 172.21.15.43:0/1868118955 wait complete. 2024-05-27T07:06:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.688+0000 7fc165600700 1 Processor -- start 2024-05-27T07:06:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.688+0000 7fc165600700 1 -- start start 2024-05-27T07:06:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.688+0000 7fc165600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1601030f0 0x7fc16019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.688+0000 7fc165600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc160103a40 0x7fc16019d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.688+0000 7fc165600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc160110ca0 0x7fc1601a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc165600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc160116550 con 0x7fc160103a40 2024-05-27T07:06:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc165600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc1601163d0 con 0x7fc1601030f0 2024-05-27T07:06:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc165600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc1601166d0 con 0x7fc160110ca0 2024-05-27T07:06:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc160103a40 0x7fc16019d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:59.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:06:59 smithi043 ceph-mon[31531]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:06:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc160103a40 0x7fc16019d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43504/0 (socket says 172.21.15.43:43504) 2024-05-27T07:06:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15e000700 1 -- 172.21.15.43:0/3488818719 learned_addr learned my addr 172.21.15.43:0/3488818719 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:06:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15ea00700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1601030f0 0x7fc16019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15f400700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc160110ca0 0x7fc1601a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15e000700 1 -- 172.21.15.43:0/3488818719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc160110ca0 msgr2=0x7fc1601a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15e000700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc160110ca0 0x7fc1601a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15e000700 1 -- 172.21.15.43:0/3488818719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1601030f0 msgr2=0x7fc16019cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15e000700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1601030f0 0x7fc16019cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15e000700 1 -- 172.21.15.43:0/3488818719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1601a1e10 con 0x7fc160103a40 2024-05-27T07:06:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.689+0000 7fc15ea00700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1601030f0 0x7fc16019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:06:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.690+0000 7fc15e000700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc160103a40 0x7fc16019d4e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00c970 tx=0x7fc14c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.691+0000 7fc157400700 1 -- 172.21.15.43:0/3488818719 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c004220 con 0x7fc160103a40 2024-05-27T07:06:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.691+0000 7fc157400700 1 -- 172.21.15.43:0/3488818719 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc14c007d30 con 0x7fc160103a40 2024-05-27T07:06:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.691+0000 7fc157400700 1 -- 172.21.15.43:0/3488818719 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c003c90 con 0x7fc160103a40 2024-05-27T07:06:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.691+0000 7fc165600700 1 -- 172.21.15.43:0/3488818719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1601a2100 con 0x7fc160103a40 2024-05-27T07:06:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.691+0000 7fc165600700 1 -- 172.21.15.43:0/3488818719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1601a9a20 con 0x7fc160103a40 2024-05-27T07:06:59.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.693+0000 7fc165600700 1 -- 172.21.15.43:0/3488818719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1600512b0 con 0x7fc160103a40 2024-05-27T07:06:59.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.697+0000 7fc157400700 1 -- 172.21.15.43:0/3488818719 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc14c0078a0 con 0x7fc160103a40 2024-05-27T07:06:59.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.698+0000 7fc157400700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1440b2900 0x7fc1440b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:06:59.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.698+0000 7fc157400700 1 -- 172.21.15.43:0/3488818719 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fc14c0e1da0 con 0x7fc160103a40 2024-05-27T07:06:59.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.699+0000 7fc15ea00700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1440b2900 0x7fc1440b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:06:59.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.699+0000 7fc15ea00700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1440b2900 0x7fc1440b4e10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc150006010 tx=0x7fc15000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:06:59.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.699+0000 7fc157400700 1 -- 172.21.15.43:0/3488818719 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc14c014030 con 0x7fc160103a40 2024-05-27T07:06:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.934+0000 7fc165600700 1 -- 172.21.15.43:0/3488818719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc16002e5e0 con 0x7fc160103a40 2024-05-27T07:06:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.938+0000 7fc157400700 1 -- 172.21.15.43:0/3488818719 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18456 (secure 0 0 0) 0x7fc14c0a9f40 con 0x7fc160103a40 2024-05-27T07:06:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:06:59.930 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":72,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:06:55.033706+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:06:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 -- 172.21.15.43:0/3488818719 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1440b2900 msgr2=0x7fc1440b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1440b2900 0x7fc1440b4e10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc150006010 tx=0x7fc15000b540 comp rx=0 tx=0).stop 2024-05-27T07:06:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 -- 172.21.15.43:0/3488818719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc160103a40 msgr2=0x7fc16019d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:06:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc160103a40 0x7fc16019d4e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00c970 tx=0x7fc14c00cd30 comp rx=0 tx=0).stop 2024-05-27T07:06:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 -- 172.21.15.43:0/3488818719 shutdown_connections 2024-05-27T07:06:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1440b2900 0x7fc1440b4e10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc160110ca0 0x7fc1601a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1601030f0 0x7fc16019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 --2- 172.21.15.43:0/3488818719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc160103a40 0x7fc16019d4e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:06:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 -- 172.21.15.43:0/3488818719 >> 172.21.15.43:0/3488818719 conn(0x7fc1600fedb0 msgr2=0x7fc160101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:06:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 -- 172.21.15.43:0/3488818719 shutdown_connections 2024-05-27T07:06:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:06:59.941+0000 7fc165600700 1 -- 172.21.15.43:0/3488818719 wait complete. 2024-05-27T07:06:59.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:06:59 smithi179 ceph-mon[35888]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:00.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:06:59 smithi132 ceph-mon[35283]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:00.707 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:01:14.696118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T07:07:00.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-27T07:07:00.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:00 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3488818719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:07:00.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:00 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3488818719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:07:01.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:00 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3488818719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:07:01.180 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:01.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:01 smithi043 ceph-mon[31531]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:01.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:01 smithi179 ceph-mon[35888]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:02.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:01 smithi132 ceph-mon[35283]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:03.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.365+0000 7f70e0c00700 1 -- 172.21.15.43:0/47902315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70dc10b890 msgr2=0x7f70dc1131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.365+0000 7f70e0c00700 1 --2- 172.21.15.43:0/47902315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70dc10b890 0x7f70dc1131f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f70d400b0d0 tx=0x7f70d400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:07:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.365+0000 7f70e0c00700 1 -- 172.21.15.43:0/47902315 shutdown_connections 2024-05-27T07:07:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.365+0000 7f70e0c00700 1 --2- 172.21.15.43:0/47902315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70dc10b890 0x7f70dc1131f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.365+0000 7f70e0c00700 1 --2- 172.21.15.43:0/47902315 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70dc106610 0x7f70dc106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.365+0000 7f70e0c00700 1 --2- 172.21.15.43:0/47902315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70dc10af40 0x7f70dc10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.366+0000 7f70e0c00700 1 -- 172.21.15.43:0/47902315 >> 172.21.15.43:0/47902315 conn(0x7f70dc102020 msgr2=0x7f70dc104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.366+0000 7f70e0c00700 1 -- 172.21.15.43:0/47902315 shutdown_connections 2024-05-27T07:07:03.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.366+0000 7f70e0c00700 1 -- 172.21.15.43:0/47902315 wait complete. 2024-05-27T07:07:03.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.367+0000 7f70e0c00700 1 Processor -- start 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.367+0000 7f70e0c00700 1 -- start start 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70e0c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70dc106610 0x7f70dc1a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70e0c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70dc10af40 0x7f70dc1a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70e0c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70dc10b890 0x7f70dc19cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70e0c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc116370 con 0x7f70dc106610 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70e0c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc1161f0 con 0x7f70dc10b890 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70e0c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc1164f0 con 0x7f70dc10af40 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70db400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70dc106610 0x7f70dc1a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70db400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70dc106610 0x7f70dc1a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43522/0 (socket says 172.21.15.43:43522) 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70db400700 1 -- 172.21.15.43:0/1343961925 learned_addr learned my addr 172.21.15.43:0/1343961925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70daa00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70dc10af40 0x7f70dc1a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:03.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70dbe00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70dc10b890 0x7f70dc19cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70db400700 1 -- 172.21.15.43:0/1343961925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70dc10af40 msgr2=0x7f70dc1a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70db400700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70dc10af40 0x7f70dc1a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.368+0000 7f70db400700 1 -- 172.21.15.43:0/1343961925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70dc10b890 msgr2=0x7f70dc19cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.369+0000 7f70db400700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70dc10b890 0x7f70dc19cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.369+0000 7f70db400700 1 -- 172.21.15.43:0/1343961925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70dc19d260 con 0x7f70dc106610 2024-05-27T07:07:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.369+0000 7f70daa00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70dc10af40 0x7f70dc1a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:07:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.369+0000 7f70dbe00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70dc10b890 0x7f70dc19cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:07:03.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.369+0000 7f70db400700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70dc106610 0x7f70dc1a29a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00d950 tx=0x7f70cc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:03.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.369+0000 7f70d3e00700 1 -- 172.21.15.43:0/1343961925 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc016070 con 0x7f70dc106610 2024-05-27T07:07:03.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.369+0000 7f70d3e00700 1 -- 172.21.15.43:0/1343961925 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70cc010460 con 0x7f70dc106610 2024-05-27T07:07:03.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.369+0000 7f70e0c00700 1 -- 172.21.15.43:0/1343961925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70dc19d550 con 0x7f70dc106610 2024-05-27T07:07:03.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.369+0000 7f70d3e00700 1 -- 172.21.15.43:0/1343961925 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc00f620 con 0x7f70dc106610 2024-05-27T07:07:03.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.370+0000 7f70e0c00700 1 -- 172.21.15.43:0/1343961925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70dc1a9750 con 0x7f70dc106610 2024-05-27T07:07:03.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.371+0000 7f70e0c00700 1 -- 172.21.15.43:0/1343961925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70dc107ac0 con 0x7f70dc106610 2024-05-27T07:07:03.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.373+0000 7f70d3e00700 1 -- 172.21.15.43:0/1343961925 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f70cc019070 con 0x7f70dc106610 2024-05-27T07:07:03.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.375+0000 7f70d3e00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f70c00b2900 0x7f70c00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:03.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.375+0000 7f70d3e00700 1 -- 172.21.15.43:0/1343961925 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f70cc0e1d20 con 0x7f70dc106610 2024-05-27T07:07:03.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.375+0000 7f70daa00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f70c00b2900 0x7f70c00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:03.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.376+0000 7f70daa00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f70c00b2900 0x7f70c00b4e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f70c8005e90 tx=0x7f70c8005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.378+0000 7f70d3e00700 1 -- 172.21.15.43:0/1343961925 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f70cc0a9e40 con 0x7f70dc106610 2024-05-27T07:07:03.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.616+0000 7f70e0c00700 1 -- 172.21.15.43:0/1343961925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f70dc02e5e0 con 0x7f70dc106610 2024-05-27T07:07:03.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.617+0000 7f70d3e00700 1 -- 172.21.15.43:0/1343961925 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f70cc0a9740 con 0x7f70dc106610 2024-05-27T07:07:03.609 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-05-27T07:07:03.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.620+0000 7f70e0c00700 1 -- 172.21.15.43:0/1343961925 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f70c00b2900 msgr2=0x7f70c00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:03.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.620+0000 7f70e0c00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f70c00b2900 0x7f70c00b4e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f70c8005e90 tx=0x7f70c8005d40 comp rx=0 tx=0).stop 2024-05-27T07:07:03.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.620+0000 7f70e0c00700 1 -- 172.21.15.43:0/1343961925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70dc106610 msgr2=0x7f70dc1a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:03.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.620+0000 7f70e0c00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70dc106610 0x7f70dc1a29a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00d950 tx=0x7f70cc00dd10 comp rx=0 tx=0).stop 2024-05-27T07:07:03.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.621+0000 7f70e0c00700 1 -- 172.21.15.43:0/1343961925 shutdown_connections 2024-05-27T07:07:03.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.621+0000 7f70e0c00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f70c00b2900 0x7f70c00b4e10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:03.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.621+0000 7f70e0c00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70dc10b890 0x7f70dc19cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:03.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.621+0000 7f70e0c00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70dc106610 0x7f70dc1a29a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:03.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.621+0000 7f70e0c00700 1 --2- 172.21.15.43:0/1343961925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70dc10af40 0x7f70dc1a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:03.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.621+0000 7f70e0c00700 1 -- 172.21.15.43:0/1343961925 >> 172.21.15.43:0/1343961925 conn(0x7f70dc102020 msgr2=0x7f70dc10a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:03.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.621+0000 7f70e0c00700 1 -- 172.21.15.43:0/1343961925 shutdown_connections 2024-05-27T07:07:03.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:03.621+0000 7f70e0c00700 1 -- 172.21.15.43:0/1343961925 wait complete. 2024-05-27T07:07:03.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:03 smithi043 ceph-mon[31531]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:03.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:03 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1343961925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:07:03.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:03 smithi179 ceph-mon[35888]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:03.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:03 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1343961925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:07:04.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:03 smithi132 ceph-mon[35283]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:04.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:03 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1343961925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:07:04.447 INFO:tasks.cephadm:Adding mds.a on smithi043 2024-05-27T07:07:04.448 INFO:tasks.cephadm:Adding mds.d on smithi043 2024-05-27T07:07:04.448 INFO:tasks.cephadm:Adding mds.g on smithi043 2024-05-27T07:07:04.448 INFO:tasks.cephadm:Adding mds.j on smithi043 2024-05-27T07:07:04.448 INFO:tasks.cephadm:Adding mds.b on smithi132 2024-05-27T07:07:04.448 INFO:tasks.cephadm:Adding mds.e on smithi132 2024-05-27T07:07:04.448 INFO:tasks.cephadm:Adding mds.h on smithi132 2024-05-27T07:07:04.448 INFO:tasks.cephadm:Adding mds.k on smithi132 2024-05-27T07:07:04.448 INFO:tasks.cephadm:Adding mds.c on smithi179 2024-05-27T07:07:04.448 INFO:tasks.cephadm:Adding mds.f on smithi179 2024-05-27T07:07:04.448 INFO:tasks.cephadm:Adding mds.i on smithi179 2024-05-27T07:07:04.449 INFO:tasks.cephadm:Adding mds.l on smithi179 2024-05-27T07:07:04.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph orch apply mds all '12;smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l' 2024-05-27T07:07:04.898 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.c/config 2024-05-27T07:07:05.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:05 smithi043 ceph-mon[31531]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:05.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:05 smithi179 ceph-mon[35888]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:06.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:05 smithi132 ceph-mon[35283]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:07.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.329+0000 7f7ff0e00700 1 -- 172.21.15.179:0/4165588188 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fec110980 msgr2=0x7f7fec112da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:07.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.329+0000 7f7ff0e00700 1 --2- 172.21.15.179:0/4165588188 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fec110980 0x7f7fec112da0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400b0d0 tx=0x7f7fe400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:07:07.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.329+0000 7f7ff0e00700 1 -- 172.21.15.179:0/4165588188 shutdown_connections 2024-05-27T07:07:07.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.329+0000 7f7ff0e00700 1 --2- 172.21.15.179:0/4165588188 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fec110980 0x7f7fec112da0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:07.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.329+0000 7f7ff0e00700 1 --2- 172.21.15.179:0/4165588188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fec0776d0 0x7f7fec075ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:07.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.329+0000 7f7ff0e00700 1 --2- 172.21.15.179:0/4165588188 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fec0760e0 0x7f7fec076570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:07.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.329+0000 7f7ff0e00700 1 -- 172.21.15.179:0/4165588188 >> 172.21.15.179:0/4165588188 conn(0x7f7fec0ff970 msgr2=0x7f7fec101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:07.322 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.329+0000 7f7ff0e00700 1 -- 172.21.15.179:0/4165588188 shutdown_connections 2024-05-27T07:07:07.322 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.329+0000 7f7ff0e00700 1 -- 172.21.15.179:0/4165588188 wait complete. 2024-05-27T07:07:07.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.330+0000 7f7ff0e00700 1 Processor -- start 2024-05-27T07:07:07.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.331+0000 7f7ff0e00700 1 -- start start 2024-05-27T07:07:07.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.331+0000 7f7ff0e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fec0760e0 0x7f7fec1a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:07.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.331+0000 7f7ff0e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fec0776d0 0x7f7fec1a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:07.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.331+0000 7f7ff0e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fec110980 0x7f7fec1a82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.331+0000 7f7ff0e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7fec1162c0 con 0x7f7fec0760e0 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.331+0000 7f7ff0e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7fec116140 con 0x7f7fec110980 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.331+0000 7f7ff0e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7fec116440 con 0x7f7fec0776d0 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.331+0000 7f7fe9400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fec0776d0 0x7f7fec1a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.331+0000 7f7fe9400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fec0776d0 0x7f7fec1a1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:49878/0 (socket says 172.21.15.179:49878) 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.331+0000 7f7fe9400700 1 -- 172.21.15.179:0/2981036162 learned_addr learned my addr 172.21.15.179:0/2981036162 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.332+0000 7f7fe9e00700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fec0760e0 0x7f7fec1a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.332+0000 7f7fea800700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fec110980 0x7f7fec1a82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.332+0000 7f7fe9400700 1 -- 172.21.15.179:0/2981036162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fec110980 msgr2=0x7f7fec1a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.332+0000 7f7fe9400700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fec110980 0x7f7fec1a82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:07.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.332+0000 7f7fe9400700 1 -- 172.21.15.179:0/2981036162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fec0760e0 msgr2=0x7f7fec1a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:07.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.332+0000 7f7fe9400700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fec0760e0 0x7f7fec1a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:07.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.332+0000 7f7fe9400700 1 -- 172.21.15.179:0/2981036162 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fec1a8970 con 0x7f7fec0776d0 2024-05-27T07:07:07.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.332+0000 7f7fe9400700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fec0776d0 0x7f7fec1a1350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800d950 tx=0x7f7fd800dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:07.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.333+0000 7f7fe2a00700 1 -- 172.21.15.179:0/2981036162 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8016070 con 0x7f7fec0776d0 2024-05-27T07:07:07.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.333+0000 7f7fe2a00700 1 -- 172.21.15.179:0/2981036162 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7fd8010460 con 0x7f7fec0776d0 2024-05-27T07:07:07.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.333+0000 7f7ff0e00700 1 -- 172.21.15.179:0/2981036162 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fec1a8c60 con 0x7f7fec0776d0 2024-05-27T07:07:07.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.333+0000 7f7fe2a00700 1 -- 172.21.15.179:0/2981036162 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd800f620 con 0x7f7fec0776d0 2024-05-27T07:07:07.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.333+0000 7f7ff0e00700 1 -- 172.21.15.179:0/2981036162 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fec1a91a0 con 0x7f7fec0776d0 2024-05-27T07:07:07.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.334+0000 7f7ff0e00700 1 -- 172.21.15.179:0/2981036162 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fec0512b0 con 0x7f7fec0776d0 2024-05-27T07:07:07.328 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.337+0000 7f7fe2a00700 1 -- 172.21.15.179:0/2981036162 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f7fd8019070 con 0x7f7fec0776d0 2024-05-27T07:07:07.330 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.339+0000 7f7fe2a00700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7fd00b2880 0x7f7fd00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:07.330 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.339+0000 7f7fe2a00700 1 -- 172.21.15.179:0/2981036162 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f7fd80e1a20 con 0x7f7fec0776d0 2024-05-27T07:07:07.331 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.340+0000 7f7fe9e00700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7fd00b2880 0x7f7fd00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:07.332 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.341+0000 7f7fe2a00700 1 -- 172.21.15.179:0/2981036162 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7fd80a9b40 con 0x7f7fec0776d0 2024-05-27T07:07:07.332 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.341+0000 7f7fe9e00700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7fd00b2880 0x7f7fd00b4d90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc006fd0 tx=0x7f7fdc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:07.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.583+0000 7f7ff0e00700 1 -- 172.21.15.179:0/2981036162 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}) -- 0x7f7fec063eb0 con 0x7f7fd00b2880 2024-05-27T07:07:07.594 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.603+0000 7f7fe2a00700 1 -- 172.21.15.179:0/2981036162 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f7fec063eb0 con 0x7f7fd00b2880 2024-05-27T07:07:07.594 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mds.all update... 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 -- 172.21.15.179:0/2981036162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7fd00b2880 msgr2=0x7f7fd00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7fd00b2880 0x7f7fd00b4d90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc006fd0 tx=0x7f7fdc008040 comp rx=0 tx=0).stop 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 -- 172.21.15.179:0/2981036162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fec0776d0 msgr2=0x7f7fec1a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fec0776d0 0x7f7fec1a1350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800d950 tx=0x7f7fd800dd10 comp rx=0 tx=0).stop 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 -- 172.21.15.179:0/2981036162 shutdown_connections 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7fd00b2880 0x7f7fd00b4d90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fec110980 0x7f7fec1a82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fec0760e0 0x7f7fec1a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 --2- 172.21.15.179:0/2981036162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fec0776d0 0x7f7fec1a1350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 -- 172.21.15.179:0/2981036162 >> 172.21.15.179:0/2981036162 conn(0x7f7fec0ff970 msgr2=0x7f7fec103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:07.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.606+0000 7f7ff0e00700 1 -- 172.21.15.179:0/2981036162 shutdown_connections 2024-05-27T07:07:07.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-05-27T07:07:07.607+0000 7f7ff0e00700 1 -- 172.21.15.179:0/2981036162 wait complete. 2024-05-27T07:07:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:07 smithi179 ceph-mon[35888]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:07:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:07:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:07.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:07.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:07.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:07 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:08.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:07 smithi132 ceph-mon[35283]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:08.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:08.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:07:08.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:08.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:07:08.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:08.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:08.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:08.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:07 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:08.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:07 smithi043 ceph-mon[31531]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:08.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:08.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:07:08.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:08.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:07:08.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:08.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:08.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:08.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:08.427 DEBUG:teuthology.orchestra.run.smithi043:mds.a> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service 2024-05-27T07:07:08.435 DEBUG:teuthology.orchestra.run.smithi043:mds.d> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service 2024-05-27T07:07:08.481 DEBUG:teuthology.orchestra.run.smithi043:mds.g> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service 2024-05-27T07:07:08.532 DEBUG:teuthology.orchestra.run.smithi043:mds.j> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service 2024-05-27T07:07:08.577 DEBUG:teuthology.orchestra.run.smithi132:mds.b> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.b.service 2024-05-27T07:07:08.585 DEBUG:teuthology.orchestra.run.smithi132:mds.e> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.e.service 2024-05-27T07:07:08.632 DEBUG:teuthology.orchestra.run.smithi132:mds.h> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.h.service 2024-05-27T07:07:08.678 DEBUG:teuthology.orchestra.run.smithi132:mds.k> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.k.service 2024-05-27T07:07:08.725 DEBUG:teuthology.orchestra.run.smithi179:mds.c> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service 2024-05-27T07:07:08.735 DEBUG:teuthology.orchestra.run.smithi179:mds.f> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service 2024-05-27T07:07:08.780 DEBUG:teuthology.orchestra.run.smithi179:mds.i> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.i.service 2024-05-27T07:07:08.826 DEBUG:teuthology.orchestra.run.smithi179:mds.l> sudo journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service 2024-05-27T07:07:08.870 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-27T07:07:08.871 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:07:08.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:08 smithi043 ceph-mon[31531]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:07:08.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:08 smithi043 ceph-mon[31531]: Saving service mds.all spec with placement smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-05-27T07:07:08.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:08 smithi043 ceph-mon[31531]: Deploying daemon mds.l on smithi179 2024-05-27T07:07:08.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:08 smithi179 ceph-mon[35888]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:07:08.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:08 smithi179 ceph-mon[35888]: Saving service mds.all spec with placement smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-05-27T07:07:08.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:08 smithi179 ceph-mon[35888]: Deploying daemon mds.l on smithi179 2024-05-27T07:07:09.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:08 smithi132 ceph-mon[35283]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:07:09.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:08 smithi132 ceph-mon[35283]: Saving service mds.all spec with placement smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-05-27T07:07:09.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:08 smithi132 ceph-mon[35283]: Deploying daemon mds.l on smithi179 2024-05-27T07:07:09.334 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:09.798 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:09 smithi043 ceph-mon[31531]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:09.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:09 smithi179 ceph-mon[35888]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:10.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:09 smithi132 ceph-mon[35283]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:07:10 smithi179 systemd[1]: Starting Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:07:11.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.514+0000 7efe79200700 1 -- 172.21.15.43:0/3727931443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe74106ec0 msgr2=0x7efe74113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:11.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.514+0000 7efe79200700 1 --2- 172.21.15.43:0/3727931443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe74106ec0 0x7efe74113350 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00b0a0 tx=0x7efe6c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.514+0000 7efe79200700 1 -- 172.21.15.43:0/3727931443 shutdown_connections 2024-05-27T07:07:11.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.514+0000 7efe79200700 1 --2- 172.21.15.43:0/3727931443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe74106ec0 0x7efe74113350 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.514+0000 7efe79200700 1 --2- 172.21.15.43:0/3727931443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe7410c5b0 0x7efe7410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.514+0000 7efe79200700 1 --2- 172.21.15.43:0/3727931443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe74106570 0x7efe74106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.514+0000 7efe79200700 1 -- 172.21.15.43:0/3727931443 >> 172.21.15.43:0/3727931443 conn(0x7efe74102070 msgr2=0x7efe741044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:11.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.515+0000 7efe79200700 1 -- 172.21.15.43:0/3727931443 shutdown_connections 2024-05-27T07:07:11.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.515+0000 7efe79200700 1 -- 172.21.15.43:0/3727931443 wait complete. 2024-05-27T07:07:11.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.516+0000 7efe79200700 1 Processor -- start 2024-05-27T07:07:11.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.516+0000 7efe79200700 1 -- start start 2024-05-27T07:07:11.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe79200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe74106570 0x7efe7419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe79200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe74106ec0 0x7efe7419d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe79200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe7410c5b0 0x7efe741a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe79200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efe74116590 con 0x7efe74106ec0 2024-05-27T07:07:11.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe79200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efe74116410 con 0x7efe7410c5b0 2024-05-27T07:07:11.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe79200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efe74116710 con 0x7efe74106570 2024-05-27T07:07:11.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe71400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe74106ec0 0x7efe7419d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:11.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe71400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe74106ec0 0x7efe7419d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56696/0 (socket says 172.21.15.43:56696) 2024-05-27T07:07:11.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe71400700 1 -- 172.21.15.43:0/3770690821 learned_addr learned my addr 172.21.15.43:0/3770690821 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:07:11.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe71e00700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe74106570 0x7efe7419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:11.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.517+0000 7efe72800700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe7410c5b0 0x7efe741a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:11.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.518+0000 7efe71400700 1 -- 172.21.15.43:0/3770690821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe74106570 msgr2=0x7efe7419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:11.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.518+0000 7efe71400700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe74106570 0x7efe7419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.518+0000 7efe71400700 1 -- 172.21.15.43:0/3770690821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe7410c5b0 msgr2=0x7efe741a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:11.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.518+0000 7efe71400700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe7410c5b0 0x7efe741a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.518+0000 7efe71400700 1 -- 172.21.15.43:0/3770690821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe741a1c40 con 0x7efe74106ec0 2024-05-27T07:07:11.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.518+0000 7efe71400700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe74106ec0 0x7efe7419d3c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efe6000c9b0 tx=0x7efe6000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:11.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.519+0000 7efe6aa00700 1 -- 172.21.15.43:0/3770690821 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe60007aa0 con 0x7efe74106ec0 2024-05-27T07:07:11.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.519+0000 7efe6aa00700 1 -- 172.21.15.43:0/3770690821 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efe6000ce50 con 0x7efe74106ec0 2024-05-27T07:07:11.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.519+0000 7efe6aa00700 1 -- 172.21.15.43:0/3770690821 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe60018400 con 0x7efe74106ec0 2024-05-27T07:07:11.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.519+0000 7efe79200700 1 -- 172.21.15.43:0/3770690821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe741a1ed0 con 0x7efe74106ec0 2024-05-27T07:07:11.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.519+0000 7efe79200700 1 -- 172.21.15.43:0/3770690821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe741a2410 con 0x7efe74106ec0 2024-05-27T07:07:11.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.520+0000 7efe79200700 1 -- 172.21.15.43:0/3770690821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe740512b0 con 0x7efe74106ec0 2024-05-27T07:07:11.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.523+0000 7efe6aa00700 1 -- 172.21.15.43:0/3770690821 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7efe6000f450 con 0x7efe74106ec0 2024-05-27T07:07:11.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.525+0000 7efe6aa00700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efe580b2880 0x7efe580b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:11.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.525+0000 7efe6aa00700 1 -- 172.21.15.43:0/3770690821 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7efe600e2080 con 0x7efe74106ec0 2024-05-27T07:07:11.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.525+0000 7efe71e00700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efe580b2880 0x7efe580b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:11.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.526+0000 7efe71e00700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efe580b2880 0x7efe580b4d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efe741079e0 tx=0x7efe6400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:11.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.527+0000 7efe6aa00700 1 -- 172.21.15.43:0/3770690821 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efe60014030 con 0x7efe74106ec0 2024-05-27T07:07:11.526 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:07:11 smithi179 podman[65508]: 2024-05-27 07:07:11.124798035 +0000 UTC m=+0.033978464 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:07:11.526 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:07:11 smithi179 podman[65508]: 2024-05-27 07:07:11.254462519 +0000 UTC m=+0.163643011 container create 83caa31734dd4fa0f625f2deea24cdd45f088f96815618ce547f710a5e63a7ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:07:11.526 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:07:11 smithi179 podman[65508]: 2024-05-27 07:07:11.533698642 +0000 UTC m=+0.442879144 container init 83caa31734dd4fa0f625f2deea24cdd45f088f96815618ce547f710a5e63a7ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:07:11.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.819+0000 7efe79200700 1 -- 172.21.15.43:0/3770690821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe7402e5e0 con 0x7efe74106ec0 2024-05-27T07:07:11.811 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:11 smithi043 ceph-mon[31531]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:11.811 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:11 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/4280267074,v1:172.21.15.179:6833/4280267074] up:boot 2024-05-27T07:07:11.811 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:11 smithi043 ceph-mon[31531]: fsmap 1 up:standby 2024-05-27T07:07:11.811 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:11 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:07:11.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.820+0000 7efe6aa00700 1 -- 172.21.15.43:0/3770690821 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7efe600aa1a0 con 0x7efe74106ec0 2024-05-27T07:07:11.812 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:07:11.812 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 -- 172.21.15.43:0/3770690821 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efe580b2880 msgr2=0x7efe580b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efe580b2880 0x7efe580b4d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efe741079e0 tx=0x7efe6400a380 comp rx=0 tx=0).stop 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 -- 172.21.15.43:0/3770690821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe74106ec0 msgr2=0x7efe7419d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe74106ec0 0x7efe7419d3c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efe6000c9b0 tx=0x7efe6000ccc0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 -- 172.21.15.43:0/3770690821 shutdown_connections 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efe580b2880 0x7efe580b4d90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe7410c5b0 0x7efe741a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe74106570 0x7efe7419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 --2- 172.21.15.43:0/3770690821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efe74106ec0 0x7efe7419d3c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 -- 172.21.15.43:0/3770690821 >> 172.21.15.43:0/3770690821 conn(0x7efe74102070 msgr2=0x7efe74104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.823+0000 7efe79200700 1 -- 172.21.15.43:0/3770690821 shutdown_connections 2024-05-27T07:07:11.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:11.824+0000 7efe79200700 1 -- 172.21.15.43:0/3770690821 wait complete. 2024-05-27T07:07:11.816 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 2 2024-05-27T07:07:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:11 smithi179 ceph-mon[35888]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:11 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/4280267074,v1:172.21.15.179:6833/4280267074] up:boot 2024-05-27T07:07:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:11 smithi179 ceph-mon[35888]: fsmap 1 up:standby 2024-05-27T07:07:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:11 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:07:11.960 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:07:11 smithi179 podman[65508]: 2024-05-27 07:07:11.544682254 +0000 UTC m=+0.453862695 container start 83caa31734dd4fa0f625f2deea24cdd45f088f96815618ce547f710a5e63a7ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-27T07:07:11.960 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:07:11 smithi179 ceph-mds[65527]: -- 172.21.15.179:0/2350215487 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ebb37c3a40 con 0x55ebb380b600 2024-05-27T07:07:11.960 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:07:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: starting mds.l at 2024-05-27T07:07:11.961 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:07:11 smithi179 bash[65508]: 83caa31734dd4fa0f625f2deea24cdd45f088f96815618ce547f710a5e63a7ae 2024-05-27T07:07:11.961 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:07:11 smithi179 systemd[1]: Started Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:07:12.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:11 smithi132 ceph-mon[35283]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:11 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/4280267074,v1:172.21.15.179:6833/4280267074] up:boot 2024-05-27T07:07:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:11 smithi132 ceph-mon[35283]: fsmap 1 up:standby 2024-05-27T07:07:12.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:11 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:07:12.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:12 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:12 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3770690821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:12 smithi043 ceph-mon[31531]: Deploying daemon mds.k on smithi132 2024-05-27T07:07:13.042 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:12 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:12 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3770690821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:13.043 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:12 smithi132 ceph-mon[35283]: Deploying daemon mds.k on smithi132 2024-05-27T07:07:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:12 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:12 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3770690821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:13.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:12 smithi179 ceph-mon[35888]: Deploying daemon mds.k on smithi132 2024-05-27T07:07:14.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:13 smithi132 ceph-mon[35283]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:14.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:13 smithi043 ceph-mon[31531]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:14.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:13 smithi179 ceph-mon[35888]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:14.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:07:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:15.341 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:07:15 smithi132 systemd[1]: Starting Ceph mds.k for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:07:15.813 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:07:15 smithi132 podman[65209]: 2024-05-27 07:07:15.387035993 +0000 UTC m=+0.037587983 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:07:15.813 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:07:15 smithi132 podman[65209]: 2024-05-27 07:07:15.508553776 +0000 UTC m=+0.159105769 container create 5e67a7a8e86d8a49b6d55b8c401658f3f2fab82f8af9f06cf2a9277e9da3055d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-k, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-27T07:07:15.813 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:07:15 smithi132 podman[65209]: 2024-05-27 07:07:15.821065959 +0000 UTC m=+0.471617992 container init 5e67a7a8e86d8a49b6d55b8c401658f3f2fab82f8af9f06cf2a9277e9da3055d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-k, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-27T07:07:16.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:15 smithi132 ceph-mon[35283]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:16.092 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:07:15 smithi132 podman[65209]: 2024-05-27 07:07:15.831824994 +0000 UTC m=+0.482376962 container start 5e67a7a8e86d8a49b6d55b8c401658f3f2fab82f8af9f06cf2a9277e9da3055d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-k, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:07:16.092 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:07:15 smithi132 bash[65209]: 5e67a7a8e86d8a49b6d55b8c401658f3f2fab82f8af9f06cf2a9277e9da3055d 2024-05-27T07:07:16.092 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:07:15 smithi132 systemd[1]: Started Ceph mds.k for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:07:16.092 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:07:15 smithi132 ceph-mds[65230]: -- 172.21.15.132:0/2424945627 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560c146a1a40 con 0x560c146e9180 2024-05-27T07:07:16.092 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:07:15 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-k[65226]: starting mds.k at 2024-05-27T07:07:16.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:15 smithi043 ceph-mon[31531]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:16.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:15 smithi179 ceph-mon[35888]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:17.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.205+0000 7f53ea800700 1 -- 172.21.15.43:0/775959233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53e41045e0 msgr2=0x7f53e41049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:17.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.205+0000 7f53ea800700 1 --2- 172.21.15.43:0/775959233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53e41045e0 0x7f53e41049f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f53d40099c0 tx=0x7f53d4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.205+0000 7f53ea800700 1 -- 172.21.15.43:0/775959233 shutdown_connections 2024-05-27T07:07:17.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.205+0000 7f53ea800700 1 --2- 172.21.15.43:0/775959233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f53e4109860 0x7f53e41111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.205+0000 7f53ea800700 1 --2- 172.21.15.43:0/775959233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53e41045e0 0x7f53e41049f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.205+0000 7f53ea800700 1 --2- 172.21.15.43:0/775959233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53e4108f10 0x7f53e4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.205+0000 7f53ea800700 1 -- 172.21.15.43:0/775959233 >> 172.21.15.43:0/775959233 conn(0x7f53e4100030 msgr2=0x7f53e4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.205+0000 7f53ea800700 1 -- 172.21.15.43:0/775959233 shutdown_connections 2024-05-27T07:07:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.205+0000 7f53ea800700 1 -- 172.21.15.43:0/775959233 wait complete. 2024-05-27T07:07:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.207+0000 7f53ea800700 1 Processor -- start 2024-05-27T07:07:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.207+0000 7f53ea800700 1 -- start start 2024-05-27T07:07:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.207+0000 7f53ea800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f53e41045e0 0x7f53e419ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.207+0000 7f53ea800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53e4108f10 0x7f53e419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.207+0000 7f53ea800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53e4109860 0x7f53e419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.207+0000 7f53ea800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f53e4114390 con 0x7f53e4109860 2024-05-27T07:07:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.207+0000 7f53ea800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f53e4114210 con 0x7f53e4108f10 2024-05-27T07:07:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.207+0000 7f53ea800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f53e4114510 con 0x7f53e41045e0 2024-05-27T07:07:17.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53e4109860 0x7f53e419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:17.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53e4109860 0x7f53e419f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36392/0 (socket says 172.21.15.43:36392) 2024-05-27T07:07:17.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e3e00700 1 -- 172.21.15.43:0/3613573637 learned_addr learned my addr 172.21.15.43:0/3613573637 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:07:17.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e2a00700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53e4108f10 0x7f53e419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:17.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e3400700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f53e41045e0 0x7f53e419ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:17.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e3e00700 1 -- 172.21.15.43:0/3613573637 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f53e41045e0 msgr2=0x7f53e419ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:17.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e3e00700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f53e41045e0 0x7f53e419ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e3e00700 1 -- 172.21.15.43:0/3613573637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53e4108f10 msgr2=0x7f53e419b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:17.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e3e00700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53e4108f10 0x7f53e419b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e3e00700 1 -- 172.21.15.43:0/3613573637 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e419fc70 con 0x7f53e4109860 2024-05-27T07:07:17.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.208+0000 7f53e3400700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f53e41045e0 0x7f53e419ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:07:17.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.209+0000 7f53e3e00700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53e4109860 0x7f53e419f5d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f53dc007ee0 tx=0x7f53dc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:17.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.209+0000 7f53dbe00700 1 -- 172.21.15.43:0/3613573637 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc00dbd0 con 0x7f53e4109860 2024-05-27T07:07:17.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.209+0000 7f53dbe00700 1 -- 172.21.15.43:0/3613573637 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53dc00dd70 con 0x7f53e4109860 2024-05-27T07:07:17.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.209+0000 7f53ea800700 1 -- 172.21.15.43:0/3613573637 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e419ff60 con 0x7f53e4109860 2024-05-27T07:07:17.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.209+0000 7f53dbe00700 1 -- 172.21.15.43:0/3613573637 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc0058d0 con 0x7f53e4109860 2024-05-27T07:07:17.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.209+0000 7f53ea800700 1 -- 172.21.15.43:0/3613573637 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e41a7880 con 0x7f53e4109860 2024-05-27T07:07:17.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.211+0000 7f53ea800700 1 -- 172.21.15.43:0/3613573637 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53e40512b0 con 0x7f53e4109860 2024-05-27T07:07:17.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.215+0000 7f53dbe00700 1 -- 172.21.15.43:0/3613573637 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f53dc01a020 con 0x7f53e4109860 2024-05-27T07:07:17.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:17.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:17.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:17.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:17.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:17.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:17.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:17 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] up:boot 2024-05-27T07:07:17.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:17 smithi043 ceph-mon[31531]: fsmap 2 up:standby 2024-05-27T07:07:17.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:07:17.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.217+0000 7f53dbe00700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f53c80b2880 0x7f53c80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:17.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.217+0000 7f53dbe00700 1 -- 172.21.15.43:0/3613573637 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f53dc0e1bc0 con 0x7f53e4109860 2024-05-27T07:07:17.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.217+0000 7f53e3400700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f53c80b2880 0x7f53c80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:17.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.218+0000 7f53e3400700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f53c80b2880 0x7f53c80b4d90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f53d4000c00 tx=0x7f53d4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:17.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.221+0000 7f53dbe00700 1 -- 172.21.15.43:0/3613573637 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f53dc010070 con 0x7f53e4109860 2024-05-27T07:07:17.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:17.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:17.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:17.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:17.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:17.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:17.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:17 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] up:boot 2024-05-27T07:07:17.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:17 smithi132 ceph-mon[35283]: fsmap 2 up:standby 2024-05-27T07:07:17.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:07:17.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:17.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:17.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:17 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] up:boot 2024-05-27T07:07:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:17 smithi179 ceph-mon[35888]: fsmap 2 up:standby 2024-05-27T07:07:17.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:07:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.518+0000 7f53ea800700 1 -- 172.21.15.43:0/3613573637 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53e402e5e0 con 0x7f53e4109860 2024-05-27T07:07:17.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.519+0000 7f53dbe00700 1 -- 172.21.15.43:0/3613573637 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f53dc0a9ce0 con 0x7f53e4109860 2024-05-27T07:07:17.510 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:07:17.510 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-27T07:07:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.522+0000 7f53ea800700 1 -- 172.21.15.43:0/3613573637 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f53c80b2880 msgr2=0x7f53c80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.522+0000 7f53ea800700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f53c80b2880 0x7f53c80b4d90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f53d4000c00 tx=0x7f53d4005fb0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.522+0000 7f53ea800700 1 -- 172.21.15.43:0/3613573637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53e4109860 msgr2=0x7f53e419f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.522+0000 7f53ea800700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53e4109860 0x7f53e419f5d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f53dc007ee0 tx=0x7f53dc00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.522+0000 7f53ea800700 1 -- 172.21.15.43:0/3613573637 shutdown_connections 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.522+0000 7f53ea800700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f53c80b2880 0x7f53c80b4d90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.522+0000 7f53ea800700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53e4109860 0x7f53e419f5d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.522+0000 7f53ea800700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f53e41045e0 0x7f53e419ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.522+0000 7f53ea800700 1 --2- 172.21.15.43:0/3613573637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53e4108f10 0x7f53e419b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.522+0000 7f53ea800700 1 -- 172.21.15.43:0/3613573637 >> 172.21.15.43:0/3613573637 conn(0x7f53e4100030 msgr2=0x7f53e4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.523+0000 7f53ea800700 1 -- 172.21.15.43:0/3613573637 shutdown_connections 2024-05-27T07:07:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:17.523+0000 7f53ea800700 1 -- 172.21.15.43:0/3613573637 wait complete. 2024-05-27T07:07:17.515 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 3 2024-05-27T07:07:18.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:18 smithi043 ceph-mon[31531]: Deploying daemon mds.j on smithi043 2024-05-27T07:07:18.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:18 smithi043 ceph-mon[31531]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:18.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:18 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3613573637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:18.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:18 smithi132 ceph-mon[35283]: Deploying daemon mds.j on smithi043 2024-05-27T07:07:18.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:18 smithi132 ceph-mon[35283]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:18.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:18 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3613573637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:18 smithi179 ceph-mon[35888]: Deploying daemon mds.j on smithi043 2024-05-27T07:07:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:18 smithi179 ceph-mon[35888]: pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:18 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3613573637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:19.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:07:19 smithi043 systemd[1]: Starting Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:07:20.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:20 smithi132 ceph-mon[35283]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:20.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:20 smithi043 ceph-mon[31531]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:20.353 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:07:20 smithi043 podman[68971]: 2024-05-27 07:07:19.922587306 +0000 UTC m=+0.034663399 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:07:20.353 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:07:20 smithi043 podman[68971]: 2024-05-27 07:07:20.038012037 +0000 UTC m=+0.150088188 container create 6726de02f1655b0d4317dda56f7105b6e72f405b0d7bc928610f0e5deae3c591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-27T07:07:20.353 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:07:20 smithi043 podman[68971]: 2024-05-27 07:07:20.360847862 +0000 UTC m=+0.472923999 container init 6726de02f1655b0d4317dda56f7105b6e72f405b0d7bc928610f0e5deae3c591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:07:20.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:20 smithi179 ceph-mon[35888]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:20.704 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:07:20 smithi043 podman[68971]: 2024-05-27 07:07:20.371775781 +0000 UTC m=+0.483851894 container start 6726de02f1655b0d4317dda56f7105b6e72f405b0d7bc928610f0e5deae3c591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:07:20.704 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:07:20 smithi043 bash[68971]: 6726de02f1655b0d4317dda56f7105b6e72f405b0d7bc928610f0e5deae3c591 2024-05-27T07:07:20.704 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:07:20 smithi043 systemd[1]: Started Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:07:20.704 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:07:20 smithi043 ceph-mds[68989]: -- 172.21.15.43:0/1458229337 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555e0f235a40 con 0x555e0f27d600 2024-05-27T07:07:20.704 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:07:20 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: starting mds.j at 2024-05-27T07:07:21.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:07:21.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:21.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:21.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:21.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:21.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:21.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:21.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: Deploying daemon mds.d on smithi043 2024-05-27T07:07:21.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:21.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/3664629623,v1:172.21.15.43:6835/3664629623] up:boot 2024-05-27T07:07:21.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: fsmap 3 up:standby 2024-05-27T07:07:21.839 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: Deploying daemon mds.d on smithi043 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/3664629623,v1:172.21.15.43:6835/3664629623] up:boot 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: fsmap 3 up:standby 2024-05-27T07:07:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:07:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: Deploying daemon mds.d on smithi043 2024-05-27T07:07:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/3664629623,v1:172.21.15.43:6835/3664629623] up:boot 2024-05-27T07:07:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: fsmap 3 up:standby 2024-05-27T07:07:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:07:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:23.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:23 smithi043 ceph-mon[31531]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:24.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:23 smithi132 ceph-mon[35283]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:24.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:23 smithi179 ceph-mon[35888]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.010+0000 7f3492000700 1 -- 172.21.15.43:0/1700324178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f348c105e40 msgr2=0x7f348c10e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:25.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.010+0000 7f3492000700 1 --2- 172.21.15.43:0/1700324178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f348c105e40 0x7f348c10e390 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3474009990 tx=0x7f3474009ca0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.010+0000 7f3492000700 1 -- 172.21.15.43:0/1700324178 shutdown_connections 2024-05-27T07:07:25.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.010+0000 7f3492000700 1 --2- 172.21.15.43:0/1700324178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f348c10e8d0 0x7f348c110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.010+0000 7f3492000700 1 --2- 172.21.15.43:0/1700324178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f348c1054f0 0x7f348c105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.010+0000 7f3492000700 1 --2- 172.21.15.43:0/1700324178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f348c105e40 0x7f348c10e390 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.010+0000 7f3492000700 1 -- 172.21.15.43:0/1700324178 >> 172.21.15.43:0/1700324178 conn(0x7f348c0fedb0 msgr2=0x7f348c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:25.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.010+0000 7f3492000700 1 -- 172.21.15.43:0/1700324178 shutdown_connections 2024-05-27T07:07:25.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.010+0000 7f3492000700 1 -- 172.21.15.43:0/1700324178 wait complete. 2024-05-27T07:07:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.012+0000 7f3492000700 1 Processor -- start 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.012+0000 7f3492000700 1 -- start start 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.012+0000 7f3492000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f348c1054f0 0x7f348c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.012+0000 7f3492000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f348c105e40 0x7f348c19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.013+0000 7f3492000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f348c10e8d0 0x7f348c1a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.013+0000 7f3492000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f348c114330 con 0x7f348c105e40 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.013+0000 7f3492000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f348c1141b0 con 0x7f348c10e8d0 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.013+0000 7f3492000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f348c1144b0 con 0x7f348c1054f0 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.013+0000 7f3483e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f348c105e40 0x7f348c19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.013+0000 7f3483e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f348c105e40 0x7f348c19d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51362/0 (socket says 172.21.15.43:51362) 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.013+0000 7f3483e00700 1 -- 172.21.15.43:0/3812257032 learned_addr learned my addr 172.21.15.43:0/3812257032 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.013+0000 7f348b400700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f348c1054f0 0x7f348c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.013+0000 7f348be00700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f348c10e8d0 0x7f348c1a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.014+0000 7f3483e00700 1 -- 172.21.15.43:0/3812257032 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f348c1054f0 msgr2=0x7f348c19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.014+0000 7f3483e00700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f348c1054f0 0x7f348c19cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.014+0000 7f3483e00700 1 -- 172.21.15.43:0/3812257032 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f348c10e8d0 msgr2=0x7f348c1a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.014+0000 7f3483e00700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f348c10e8d0 0x7f348c1a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.014+0000 7f3483e00700 1 -- 172.21.15.43:0/3812257032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f348c1a1e70 con 0x7f348c105e40 2024-05-27T07:07:25.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.014+0000 7f348be00700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f348c10e8d0 0x7f348c1a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:07:25.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.014+0000 7f3483e00700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f348c105e40 0x7f348c19d540 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3474005910 tx=0x7f34740047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:25.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.015+0000 7f3488c00700 1 -- 172.21.15.43:0/3812257032 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474004830 con 0x7f348c105e40 2024-05-27T07:07:25.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.015+0000 7f3488c00700 1 -- 172.21.15.43:0/3812257032 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3474005020 con 0x7f348c105e40 2024-05-27T07:07:25.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.015+0000 7f3488c00700 1 -- 172.21.15.43:0/3812257032 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347400f960 con 0x7f348c105e40 2024-05-27T07:07:25.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.015+0000 7f3492000700 1 -- 172.21.15.43:0/3812257032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f348c1a2100 con 0x7f348c105e40 2024-05-27T07:07:25.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.015+0000 7f3492000700 1 -- 172.21.15.43:0/3812257032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f348c1a2530 con 0x7f348c105e40 2024-05-27T07:07:25.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.016+0000 7f3492000700 1 -- 172.21.15.43:0/3812257032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f348c0512b0 con 0x7f348c105e40 2024-05-27T07:07:25.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.019+0000 7f3488c00700 1 -- 172.21.15.43:0/3812257032 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f34740054d0 con 0x7f348c105e40 2024-05-27T07:07:25.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.021+0000 7f3488c00700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34700b2900 0x7f34700b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:25.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.021+0000 7f3488c00700 1 -- 172.21.15.43:0/3812257032 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f34740e3060 con 0x7f348c105e40 2024-05-27T07:07:25.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.021+0000 7f348b400700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34700b2900 0x7f34700b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:25.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.022+0000 7f348b400700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34700b2900 0x7f34700b4e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f347c005950 tx=0x7f347c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:25.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.024+0000 7f3488c00700 1 -- 172.21.15.43:0/3812257032 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3474013070 con 0x7f348c105e40 2024-05-27T07:07:25.296 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:07:25 smithi043 podman[69399]: 2024-05-27 07:07:25.130597584 +0000 UTC m=+0.144218694 container create d390a0d3996b1f15a11e739cd567d555832d7b4091e91ce8b6f996835013e269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:07:25.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.322+0000 7f3492000700 1 -- 172.21.15.43:0/3812257032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f348c02e5e0 con 0x7f348c105e40 2024-05-27T07:07:25.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.323+0000 7f3488c00700 1 -- 172.21.15.43:0/3812257032 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f34740b3030 con 0x7f348c105e40 2024-05-27T07:07:25.314 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:07:25.314 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-27T07:07:25.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 -- 172.21.15.43:0/3812257032 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34700b2900 msgr2=0x7f34700b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:25.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34700b2900 0x7f34700b4e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f347c005950 tx=0x7f347c00b410 comp rx=0 tx=0).stop 2024-05-27T07:07:25.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 -- 172.21.15.43:0/3812257032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f348c105e40 msgr2=0x7f348c19d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:25.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f348c105e40 0x7f348c19d540 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3474005910 tx=0x7f34740047b0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 -- 172.21.15.43:0/3812257032 shutdown_connections 2024-05-27T07:07:25.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34700b2900 0x7f34700b4e10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f348c10e8d0 0x7f348c1a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f348c1054f0 0x7f348c19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 --2- 172.21.15.43:0/3812257032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f348c105e40 0x7f348c19d540 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:25.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 -- 172.21.15.43:0/3812257032 >> 172.21.15.43:0/3812257032 conn(0x7f348c0fedb0 msgr2=0x7f348c103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:25.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 -- 172.21.15.43:0/3812257032 shutdown_connections 2024-05-27T07:07:25.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:25.326+0000 7f3492000700 1 -- 172.21.15.43:0/3812257032 wait complete. 2024-05-27T07:07:25.319 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 4 2024-05-27T07:07:25.704 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:07:25 smithi043 podman[69399]: 2024-05-27 07:07:25.426342293 +0000 UTC m=+0.439963404 container init d390a0d3996b1f15a11e739cd567d555832d7b4091e91ce8b6f996835013e269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-27T07:07:25.704 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:07:25 smithi043 podman[69399]: 2024-05-27 07:07:25.434852837 +0000 UTC m=+0.448473926 container start d390a0d3996b1f15a11e739cd567d555832d7b4091e91ce8b6f996835013e269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:07:25.704 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:07:25 smithi043 ceph-mds[69453]: -- 172.21.15.43:0/819267627 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5634c9221a40 con 0x5634c9269600 2024-05-27T07:07:25.704 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:07:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: starting mds.d at 2024-05-27T07:07:26.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:25 smithi043 ceph-mon[31531]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:26.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:25 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3812257032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:26.011 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:07:26 smithi043 bash[69399]: d390a0d3996b1f15a11e739cd567d555832d7b4091e91ce8b6f996835013e269 2024-05-27T07:07:26.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:25 smithi132 ceph-mon[35283]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:26.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:25 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3812257032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:26.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:25 smithi179 ceph-mon[35888]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:26.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:25 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3812257032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:26.336 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:07:26 smithi043 systemd[1]: Started Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:07:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:26 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6836/3666274467,v1:172.21.15.43:6837/3666274467] up:boot 2024-05-27T07:07:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:26 smithi132 ceph-mon[35283]: fsmap 4 up:standby 2024-05-27T07:07:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:07:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:27.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:27.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:26 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6836/3666274467,v1:172.21.15.43:6837/3666274467] up:boot 2024-05-27T07:07:27.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:26 smithi043 ceph-mon[31531]: fsmap 4 up:standby 2024-05-27T07:07:27.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:07:27.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:27.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:27.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:27.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:27.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:27.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:27.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:26 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6836/3666274467,v1:172.21.15.43:6837/3666274467] up:boot 2024-05-27T07:07:27.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:26 smithi179 ceph-mon[35888]: fsmap 4 up:standby 2024-05-27T07:07:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:07:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:27.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:28.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:27 smithi132 ceph-mon[35283]: Deploying daemon mds.e on smithi132 2024-05-27T07:07:28.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:27 smithi132 ceph-mon[35283]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:28.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:27 smithi043 ceph-mon[31531]: Deploying daemon mds.e on smithi132 2024-05-27T07:07:28.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:27 smithi043 ceph-mon[31531]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:28.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:27 smithi179 ceph-mon[35888]: Deploying daemon mds.e on smithi132 2024-05-27T07:07:28.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:27 smithi179 ceph-mon[35888]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:28.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:07:28.887 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:29.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:29 smithi132 ceph-mon[35283]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:30.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:29 smithi043 ceph-mon[31531]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:30.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:29 smithi179 ceph-mon[35888]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:30.219 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:07:29 smithi132 podman[65511]: 2024-05-27 07:07:29.914889806 +0000 UTC m=+0.155809361 container create 983414c01141f7f9ff13d595c1054c4bfa6ac2de60cb2fb0f9f9a86c0afad972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-e, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-27T07:07:30.219 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:07:30 smithi132 podman[65511]: 2024-05-27 07:07:30.22723145 +0000 UTC m=+0.468151064 container init 983414c01141f7f9ff13d595c1054c4bfa6ac2de60cb2fb0f9f9a86c0afad972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-e, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-27T07:07:30.592 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:07:30 smithi132 podman[65511]: 2024-05-27 07:07:30.237082684 +0000 UTC m=+0.478002271 container start 983414c01141f7f9ff13d595c1054c4bfa6ac2de60cb2fb0f9f9a86c0afad972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:07:30.592 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:07:30 smithi132 bash[65511]: 983414c01141f7f9ff13d595c1054c4bfa6ac2de60cb2fb0f9f9a86c0afad972 2024-05-27T07:07:30.592 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:07:30 smithi132 systemd[1]: Started Ceph mds.e for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:07:30.592 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:07:30 smithi132 ceph-mds[65529]: -- 172.21.15.132:0/2580121767 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f92fe43a40 con 0x55f92fe8b600 2024-05-27T07:07:30.592 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:07:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-e[65525]: starting mds.e at 2024-05-27T07:07:31.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.109+0000 7f2176e00700 1 -- 172.21.15.43:0/3470506088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f217010e8c0 msgr2=0x7f2170110da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:31.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.109+0000 7f2176e00700 1 --2- 172.21.15.43:0/3470506088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f217010e8c0 0x7f2170110da0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2168009990 tx=0x7f2168009ca0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.109+0000 7f2176e00700 1 -- 172.21.15.43:0/3470506088 shutdown_connections 2024-05-27T07:07:31.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.109+0000 7f2176e00700 1 --2- 172.21.15.43:0/3470506088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f217010e8c0 0x7f2170110da0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.109+0000 7f2176e00700 1 --2- 172.21.15.43:0/3470506088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2170105450 0x7f2170105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.109+0000 7f2176e00700 1 --2- 172.21.15.43:0/3470506088 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2170105da0 0x7f217010e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.109+0000 7f2176e00700 1 -- 172.21.15.43:0/3470506088 >> 172.21.15.43:0/3470506088 conn(0x7f21700fedb0 msgr2=0x7f21701011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:31.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.109+0000 7f2176e00700 1 -- 172.21.15.43:0/3470506088 shutdown_connections 2024-05-27T07:07:31.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.110+0000 7f2176e00700 1 -- 172.21.15.43:0/3470506088 wait complete. 2024-05-27T07:07:31.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.111+0000 7f2176e00700 1 Processor -- start 2024-05-27T07:07:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.111+0000 7f2176e00700 1 -- start start 2024-05-27T07:07:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.111+0000 7f2176e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2170105450 0x7f2170103140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.111+0000 7f2176e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2170105da0 0x7f2170103680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.112+0000 7f2176e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f217010e8c0 0x7f2170103bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.112+0000 7f2176e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2170114320 con 0x7f217010e8c0 2024-05-27T07:07:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.112+0000 7f2176e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f21701141a0 con 0x7f2170105450 2024-05-27T07:07:31.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.112+0000 7f2176e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f21701144a0 con 0x7f2170105da0 2024-05-27T07:07:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.112+0000 7f2174e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f217010e8c0 0x7f2170103bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.112+0000 7f2174e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f217010e8c0 0x7f2170103bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51402/0 (socket says 172.21.15.43:51402) 2024-05-27T07:07:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.112+0000 7f2174e00700 1 -- 172.21.15.43:0/1676418198 learned_addr learned my addr 172.21.15.43:0/1676418198 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:07:31.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.112+0000 7f2167400700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2170105da0 0x7f2170103680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.112+0000 7f216fe00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2170105450 0x7f2170103140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.112+0000 7f2174e00700 1 -- 172.21.15.43:0/1676418198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2170105da0 msgr2=0x7f2170103680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.113+0000 7f2174e00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2170105da0 0x7f2170103680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.113+0000 7f2174e00700 1 -- 172.21.15.43:0/1676418198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2170105450 msgr2=0x7f2170103140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.113+0000 7f2174e00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2170105450 0x7f2170103140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.113+0000 7f2174e00700 1 -- 172.21.15.43:0/1676418198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21701a9050 con 0x7f217010e8c0 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.113+0000 7f2174e00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f217010e8c0 0x7f2170103bc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f21680047b0 tx=0x7f2168005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.113+0000 7f216d600700 1 -- 172.21.15.43:0/1676418198 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2168004830 con 0x7f217010e8c0 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.113+0000 7f216d600700 1 -- 172.21.15.43:0/1676418198 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2168005020 con 0x7f217010e8c0 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.113+0000 7f2176e00700 1 -- 172.21.15.43:0/1676418198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21701a92e0 con 0x7f217010e8c0 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.113+0000 7f216d600700 1 -- 172.21.15.43:0/1676418198 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216800f8d0 con 0x7f217010e8c0 2024-05-27T07:07:31.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.113+0000 7f2176e00700 1 -- 172.21.15.43:0/1676418198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21701a96d0 con 0x7f217010e8c0 2024-05-27T07:07:31.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.115+0000 7f2176e00700 1 -- 172.21.15.43:0/1676418198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21700512b0 con 0x7f217010e8c0 2024-05-27T07:07:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.117+0000 7f216d600700 1 -- 172.21.15.43:0/1676418198 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f21680054d0 con 0x7f217010e8c0 2024-05-27T07:07:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.119+0000 7f216d600700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f21540b2880 0x7f21540b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.119+0000 7f216d600700 1 -- 172.21.15.43:0/1676418198 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f21680e2cb0 con 0x7f217010e8c0 2024-05-27T07:07:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.119+0000 7f216fe00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f21540b2880 0x7f21540b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.120+0000 7f216fe00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f21540b2880 0x7f21540b4d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f21600060b0 tx=0x7f216000c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.122+0000 7f216d600700 1 -- 172.21.15.43:0/1676418198 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2168013070 con 0x7f217010e8c0 2024-05-27T07:07:31.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.408+0000 7f2176e00700 1 -- 172.21.15.43:0/1676418198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f217002e7a0 con 0x7f217010e8c0 2024-05-27T07:07:31.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.410+0000 7f216d600700 1 -- 172.21.15.43:0/1676418198 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7f21680aae60 con 0x7f217010e8c0 2024-05-27T07:07:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:07:31.401 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34424,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-27T07:07:31.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.414+0000 7f2176e00700 1 -- 172.21.15.43:0/1676418198 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f21540b2880 msgr2=0x7f21540b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:31.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.414+0000 7f2176e00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f21540b2880 0x7f21540b4d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f21600060b0 tx=0x7f216000c040 comp rx=0 tx=0).stop 2024-05-27T07:07:31.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.414+0000 7f2176e00700 1 -- 172.21.15.43:0/1676418198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f217010e8c0 msgr2=0x7f2170103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:31.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.414+0000 7f2176e00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f217010e8c0 0x7f2170103bc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f21680047b0 tx=0x7f2168005ef0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.415+0000 7f2176e00700 1 -- 172.21.15.43:0/1676418198 shutdown_connections 2024-05-27T07:07:31.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.415+0000 7f2176e00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f21540b2880 0x7f21540b4d90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.415+0000 7f2176e00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f217010e8c0 0x7f2170103bc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.415+0000 7f2176e00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2170105450 0x7f2170103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.415+0000 7f2176e00700 1 --2- 172.21.15.43:0/1676418198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2170105da0 0x7f2170103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:31.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.415+0000 7f2176e00700 1 -- 172.21.15.43:0/1676418198 >> 172.21.15.43:0/1676418198 conn(0x7f21700fedb0 msgr2=0x7f21700ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:31.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.415+0000 7f2176e00700 1 -- 172.21.15.43:0/1676418198 shutdown_connections 2024-05-27T07:07:31.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:31.415+0000 7f2176e00700 1 -- 172.21.15.43:0/1676418198 wait complete. 2024-05-27T07:07:31.408 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 6 2024-05-27T07:07:31.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:31.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:31.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:31.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:31.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:31.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:31.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: Deploying daemon mds.f on smithi179 2024-05-27T07:07:31.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:31.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] up:boot 2024-05-27T07:07:31.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: fsmap 5 up:standby 2024-05-27T07:07:31.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:07:31.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:31 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1676418198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:31.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: Deploying daemon mds.f on smithi179 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] up:boot 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: fsmap 5 up:standby 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:07:31.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:31 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1676418198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:31.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:31.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:31.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:31.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:31.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:31.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:31.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: Deploying daemon mds.f on smithi179 2024-05-27T07:07:31.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:31.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] up:boot 2024-05-27T07:07:31.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: fsmap 5 up:standby 2024-05-27T07:07:31.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:07:31.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:31 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1676418198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:34.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:33 smithi132 ceph-mon[35283]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:34.117 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:33 smithi179 ceph-mon[35888]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:34.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:07:34.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:33 smithi043 ceph-mon[31531]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:34.413 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:07:34 smithi179 podman[65813]: 2024-05-27 07:07:34.247171424 +0000 UTC m=+0.180906743 container create 374a158fddff45b23791ef94bb5347cf67b5e12456c9a1154f2986f68f5a70cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T07:07:34.597 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:34.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:07:34 smithi179 podman[65813]: 2024-05-27 07:07:34.542853583 +0000 UTC m=+0.476588929 container init 374a158fddff45b23791ef94bb5347cf67b5e12456c9a1154f2986f68f5a70cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:07:34.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:07:34 smithi179 podman[65813]: 2024-05-27 07:07:34.551605285 +0000 UTC m=+0.485340567 container start 374a158fddff45b23791ef94bb5347cf67b5e12456c9a1154f2986f68f5a70cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T07:07:34.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:07:34 smithi179 bash[65813]: 374a158fddff45b23791ef94bb5347cf67b5e12456c9a1154f2986f68f5a70cd 2024-05-27T07:07:34.960 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:07:34 smithi179 systemd[1]: Started Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:07:34.960 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:07:34 smithi179 ceph-mds[65838]: -- 172.21.15.179:0/2063594966 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564820d6fa40 con 0x564820db7a80 2024-05-27T07:07:34.960 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:07:34 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: starting mds.f at 2024-05-27T07:07:36.026 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:36.026 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:36.026 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:36.026 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6834/2538262781,v1:172.21.15.179:6835/2538262781] up:boot 2024-05-27T07:07:36.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: fsmap 6 up:standby 2024-05-27T07:07:36.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:07:36.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:36.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:36.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:36.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:36.027 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:35 smithi179 ceph-mon[35888]: Deploying daemon mds.c on smithi179 2024-05-27T07:07:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6834/2538262781,v1:172.21.15.179:6835/2538262781] up:boot 2024-05-27T07:07:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: fsmap 6 up:standby 2024-05-27T07:07:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:07:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:36.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:36.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:36.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:35 smithi132 ceph-mon[35283]: Deploying daemon mds.c on smithi179 2024-05-27T07:07:36.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:36.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:36.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:36.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6834/2538262781,v1:172.21.15.179:6835/2538262781] up:boot 2024-05-27T07:07:36.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: fsmap 6 up:standby 2024-05-27T07:07:36.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:07:36.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:36.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:36.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:36.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:36.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:35 smithi043 ceph-mon[31531]: Deploying daemon mds.c on smithi179 2024-05-27T07:07:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.725+0000 7ff9a4c00700 1 -- 172.21.15.43:0/238541703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a010b2d0 msgr2=0x7ff9a0117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.725+0000 7ff9a4c00700 1 --2- 172.21.15.43:0/238541703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a010b2d0 0x7ff9a0117800 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff99800b0a0 tx=0x7ff99800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:07:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.725+0000 7ff9a4c00700 1 -- 172.21.15.43:0/238541703 shutdown_connections 2024-05-27T07:07:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.725+0000 7ff9a4c00700 1 --2- 172.21.15.43:0/238541703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a010b2d0 0x7ff9a0117800 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.725+0000 7ff9a4c00700 1 --2- 172.21.15.43:0/238541703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9a0110960 0x7ff9a0110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.725+0000 7ff9a4c00700 1 --2- 172.21.15.43:0/238541703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff9a010a980 0x7ff9a010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.725+0000 7ff9a4c00700 1 -- 172.21.15.43:0/238541703 >> 172.21.15.43:0/238541703 conn(0x7ff9a0077c70 msgr2=0x7ff9a00780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.725+0000 7ff9a4c00700 1 -- 172.21.15.43:0/238541703 shutdown_connections 2024-05-27T07:07:36.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.725+0000 7ff9a4c00700 1 -- 172.21.15.43:0/238541703 wait complete. 2024-05-27T07:07:36.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.726+0000 7ff9a4c00700 1 Processor -- start 2024-05-27T07:07:36.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.727+0000 7ff9a4c00700 1 -- start start 2024-05-27T07:07:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.727+0000 7ff9a4c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a010a980 0x7ff9a01a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.727+0000 7ff9a4c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff9a010b2d0 0x7ff9a01a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.727+0000 7ff9a4c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9a0110960 0x7ff9a01a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.727+0000 7ff9a4c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a011a980 con 0x7ff9a010a980 2024-05-27T07:07:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.727+0000 7ff9a4c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a011a800 con 0x7ff9a010b2d0 2024-05-27T07:07:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.727+0000 7ff9a4c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a011ab00 con 0x7ff9a0110960 2024-05-27T07:07:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a010a980 0x7ff9a01a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a010a980 0x7ff9a01a1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60090/0 (socket says 172.21.15.43:60090) 2024-05-27T07:07:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99de00700 1 -- 172.21.15.43:0/3953638258 learned_addr learned my addr 172.21.15.43:0/3953638258 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:07:36.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99d400700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff9a010b2d0 0x7ff9a01a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:36.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99e800700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9a0110960 0x7ff9a01a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:36.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99de00700 1 -- 172.21.15.43:0/3953638258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9a0110960 msgr2=0x7ff9a01a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99de00700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9a0110960 0x7ff9a01a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99de00700 1 -- 172.21.15.43:0/3953638258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff9a010b2d0 msgr2=0x7ff9a01a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99de00700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff9a010b2d0 0x7ff9a01a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99de00700 1 -- 172.21.15.43:0/3953638258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a01a6170 con 0x7ff9a010a980 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.728+0000 7ff99d400700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff9a010b2d0 0x7ff9a01a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.729+0000 7ff99de00700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a010a980 0x7ff9a01a1350 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff99000d980 tx=0x7ff99000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.729+0000 7ff996a00700 1 -- 172.21.15.43:0/3953638258 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff990017070 con 0x7ff9a010a980 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.729+0000 7ff996a00700 1 -- 172.21.15.43:0/3953638258 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff990010460 con 0x7ff9a010a980 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.729+0000 7ff996a00700 1 -- 172.21.15.43:0/3953638258 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99000de60 con 0x7ff9a010a980 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.729+0000 7ff9a4c00700 1 -- 172.21.15.43:0/3953638258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a01a6460 con 0x7ff9a010a980 2024-05-27T07:07:36.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.729+0000 7ff9a4c00700 1 -- 172.21.15.43:0/3953638258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a01a6950 con 0x7ff9a010a980 2024-05-27T07:07:36.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.730+0000 7ff9a4c00700 1 -- 172.21.15.43:0/3953638258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a00512b0 con 0x7ff9a010a980 2024-05-27T07:07:36.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.733+0000 7ff996a00700 1 -- 172.21.15.43:0/3953638258 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff990010610 con 0x7ff9a010a980 2024-05-27T07:07:36.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.734+0000 7ff996a00700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff9840b2900 0x7ff9840b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.735+0000 7ff996a00700 1 -- 172.21.15.43:0/3953638258 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ff9900e1820 con 0x7ff9a010a980 2024-05-27T07:07:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.735+0000 7ff99d400700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff9840b2900 0x7ff9840b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.736+0000 7ff99d400700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff9840b2900 0x7ff9840b4e10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff98c005fd0 tx=0x7ff98c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:36.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:36.737+0000 7ff996a00700 1 -- 172.21.15.43:0/3953638258 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff9900a99c0 con 0x7ff9a010a980 2024-05-27T07:07:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.026+0000 7ff9a4c00700 1 -- 172.21.15.43:0/3953638258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9a002e5e0 con 0x7ff9a010a980 2024-05-27T07:07:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.027+0000 7ff996a00700 1 -- 172.21.15.43:0/3953638258 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7ff9900a92c0 con 0x7ff9a010a980 2024-05-27T07:07:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:07:37.019 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34424,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-27T07:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.030+0000 7ff9a4c00700 1 -- 172.21.15.43:0/3953638258 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff9840b2900 msgr2=0x7ff9840b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.030+0000 7ff9a4c00700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff9840b2900 0x7ff9840b4e10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff98c005fd0 tx=0x7ff98c005e30 comp rx=0 tx=0).stop 2024-05-27T07:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.030+0000 7ff9a4c00700 1 -- 172.21.15.43:0/3953638258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a010a980 msgr2=0x7ff9a01a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.030+0000 7ff9a4c00700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a010a980 0x7ff9a01a1350 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff99000d980 tx=0x7ff99000dd40 comp rx=0 tx=0).stop 2024-05-27T07:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.030+0000 7ff9a4c00700 1 -- 172.21.15.43:0/3953638258 shutdown_connections 2024-05-27T07:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.030+0000 7ff9a4c00700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff9840b2900 0x7ff9840b4e10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.030+0000 7ff9a4c00700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9a0110960 0x7ff9a01a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.030+0000 7ff9a4c00700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a010a980 0x7ff9a01a1350 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.030+0000 7ff9a4c00700 1 --2- 172.21.15.43:0/3953638258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff9a010b2d0 0x7ff9a01a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.030+0000 7ff9a4c00700 1 -- 172.21.15.43:0/3953638258 >> 172.21.15.43:0/3953638258 conn(0x7ff9a0077c70 msgr2=0x7ff9a007b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.031+0000 7ff9a4c00700 1 -- 172.21.15.43:0/3953638258 shutdown_connections 2024-05-27T07:07:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:37.031+0000 7ff9a4c00700 1 -- 172.21.15.43:0/3953638258 wait complete. 2024-05-27T07:07:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 7 2024-05-27T07:07:38.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:37 smithi132 ceph-mon[35283]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:37 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3953638258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:38.147 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:37 smithi179 ceph-mon[35888]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:38.147 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:37 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3953638258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:38.147 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:07:38 smithi179 systemd[1]: Starting Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:07:38.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:37 smithi043 ceph-mon[31531]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:38.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:37 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3953638258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:38.850 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:07:38 smithi179 podman[66124]: 2024-05-27 07:07:38.414004774 +0000 UTC m=+0.032788927 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:07:38.851 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:07:38 smithi179 podman[66124]: 2024-05-27 07:07:38.537101931 +0000 UTC m=+0.155886147 container create f3c9408632d818fef921c5d9ffdea7d82874431699e8705a9d69f1339ec1dcb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-27T07:07:38.851 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:07:38 smithi179 podman[66124]: 2024-05-27 07:07:38.858135702 +0000 UTC m=+0.476919895 container init f3c9408632d818fef921c5d9ffdea7d82874431699e8705a9d69f1339ec1dcb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T07:07:39.210 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:07:38 smithi179 podman[66124]: 2024-05-27 07:07:38.868888505 +0000 UTC m=+0.487672674 container start f3c9408632d818fef921c5d9ffdea7d82874431699e8705a9d69f1339ec1dcb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:07:39.210 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:07:38 smithi179 bash[66124]: f3c9408632d818fef921c5d9ffdea7d82874431699e8705a9d69f1339ec1dcb2 2024-05-27T07:07:39.210 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:07:38 smithi179 systemd[1]: Started Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:07:39.211 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:07:38 smithi179 ceph-mds[66142]: -- 172.21.15.179:0/1659043611 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ad3251a40 con 0x562ad3299600 2024-05-27T07:07:39.211 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:07:38 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: starting mds.c at 2024-05-27T07:07:39.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:07:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:39 smithi132 ceph-mon[35283]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:40.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:39 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:39 smithi043 ceph-mon[31531]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:39 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:39 smithi179 ceph-mon[35888]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:40.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:40.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:40.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:39 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:40.322 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:41.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:40 smithi132 ceph-mon[35283]: Deploying daemon mds.h on smithi132 2024-05-27T07:07:41.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:40 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6836/4061292168,v1:172.21.15.179:6837/4061292168] up:boot 2024-05-27T07:07:41.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:40 smithi132 ceph-mon[35283]: fsmap 7 up:standby 2024-05-27T07:07:41.063 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:40 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:07:41.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:40 smithi043 ceph-mon[31531]: Deploying daemon mds.h on smithi132 2024-05-27T07:07:41.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:40 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6836/4061292168,v1:172.21.15.179:6837/4061292168] up:boot 2024-05-27T07:07:41.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:40 smithi043 ceph-mon[31531]: fsmap 7 up:standby 2024-05-27T07:07:41.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:40 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:07:41.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:40 smithi179 ceph-mon[35888]: Deploying daemon mds.h on smithi132 2024-05-27T07:07:41.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:40 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6836/4061292168,v1:172.21.15.179:6837/4061292168] up:boot 2024-05-27T07:07:41.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:40 smithi179 ceph-mon[35888]: fsmap 7 up:standby 2024-05-27T07:07:41.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:40 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:07:41.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:41 smithi043 ceph-mon[31531]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:42.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:41 smithi132 ceph-mon[35283]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:42.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:41 smithi179 ceph-mon[35888]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:42.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.585+0000 7f3005a00700 1 -- 172.21.15.43:0/669710618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f300010b890 msgr2=0x7f30001131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:42.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.585+0000 7f3005a00700 1 --2- 172.21.15.43:0/669710618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f300010b890 0x7f30001131f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8009fc0 tx=0x7f2ff800b330 comp rx=0 tx=0).stop 2024-05-27T07:07:42.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.586+0000 7f3005a00700 1 -- 172.21.15.43:0/669710618 shutdown_connections 2024-05-27T07:07:42.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.586+0000 7f3005a00700 1 --2- 172.21.15.43:0/669710618 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f300010b890 0x7f30001131f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:42.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.586+0000 7f3005a00700 1 --2- 172.21.15.43:0/669710618 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3000106610 0x7f3000106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:42.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.586+0000 7f3005a00700 1 --2- 172.21.15.43:0/669710618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f300010af40 0x7f300010b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:42.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.586+0000 7f3005a00700 1 -- 172.21.15.43:0/669710618 >> 172.21.15.43:0/669710618 conn(0x7f3000102020 msgr2=0x7f3000104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:42.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.586+0000 7f3005a00700 1 -- 172.21.15.43:0/669710618 shutdown_connections 2024-05-27T07:07:42.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.586+0000 7f3005a00700 1 -- 172.21.15.43:0/669710618 wait complete. 2024-05-27T07:07:42.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.587+0000 7f3005a00700 1 Processor -- start 2024-05-27T07:07:42.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.587+0000 7f3005a00700 1 -- start start 2024-05-27T07:07:42.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f3005a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3000106610 0x7f300019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:42.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f3005a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f300010af40 0x7f300019d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:42.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f3005a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f300010b890 0x7f30001a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:42.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f3005a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f30001163c0 con 0x7f300010b890 2024-05-27T07:07:42.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f3005a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3000116240 con 0x7f300010af40 2024-05-27T07:07:42.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f3005a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3000116540 con 0x7f3000106610 2024-05-27T07:07:42.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f3005000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f300010b890 0x7f30001a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:42.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f2fff400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f300010af40 0x7f300019d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:42.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f3005000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f300010b890 0x7f30001a1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60100/0 (socket says 172.21.15.43:60100) 2024-05-27T07:07:42.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f3005000700 1 -- 172.21.15.43:0/2917818778 learned_addr learned my addr 172.21.15.43:0/2917818778 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:07:42.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.588+0000 7f2fffe00700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3000106610 0x7f300019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:42.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f3005000700 1 -- 172.21.15.43:0/2917818778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3000106610 msgr2=0x7f300019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:42.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f3005000700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3000106610 0x7f300019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f3005000700 1 -- 172.21.15.43:0/2917818778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f300010af40 msgr2=0x7f300019d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f3005000700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f300010af40 0x7f300019d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f3005000700 1 -- 172.21.15.43:0/2917818778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30001a1c00 con 0x7f300010b890 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f2fff400700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f300010af40 0x7f300019d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f3005000700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f300010b890 0x7f30001a1560 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8009790 tx=0x7f2ff800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f2ffcc00700 1 -- 172.21.15.43:0/2917818778 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8021070 con 0x7f300010b890 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f2ffcc00700 1 -- 172.21.15.43:0/2917818778 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ff8007ac0 con 0x7f300010b890 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f3005a00700 1 -- 172.21.15.43:0/2917818778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30001a1e90 con 0x7f300010b890 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.589+0000 7f2ffcc00700 1 -- 172.21.15.43:0/2917818778 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8005040 con 0x7f300010b890 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.590+0000 7f3005a00700 1 -- 172.21.15.43:0/2917818778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30001a9810 con 0x7f300010b890 2024-05-27T07:07:42.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.591+0000 7f3005a00700 1 -- 172.21.15.43:0/2917818778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3000107ac0 con 0x7f300010b890 2024-05-27T07:07:42.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.593+0000 7f2ffcc00700 1 -- 172.21.15.43:0/2917818778 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f2ff8024070 con 0x7f300010b890 2024-05-27T07:07:42.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.595+0000 7f2ffcc00700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2fe40b2880 0x7f2fe40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:42.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.595+0000 7f2ffcc00700 1 -- 172.21.15.43:0/2917818778 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f2ff800f020 con 0x7f300010b890 2024-05-27T07:07:42.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.595+0000 7f2fffe00700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2fe40b2880 0x7f2fe40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:42.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.596+0000 7f2fffe00700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2fe40b2880 0x7f2fe40b4d90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009bf0 tx=0x7f2ff0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:42.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.598+0000 7f2ffcc00700 1 -- 172.21.15.43:0/2917818778 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2ff802b020 con 0x7f300010b890 2024-05-27T07:07:42.818 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:07:42 smithi132 systemd[1]: Starting Ceph mds.h for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:07:42.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.891+0000 7f3005a00700 1 -- 172.21.15.43:0/2917818778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f300019e090 con 0x7f300010b890 2024-05-27T07:07:42.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.893+0000 7f2ffcc00700 1 -- 172.21.15.43:0/2917818778 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7f2ff80ab8e0 con 0x7f300010b890 2024-05-27T07:07:42.885 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:07:42.885 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34424,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-27T07:07:42.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.896+0000 7f3005a00700 1 -- 172.21.15.43:0/2917818778 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2fe40b2880 msgr2=0x7f2fe40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:42.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.896+0000 7f3005a00700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2fe40b2880 0x7f2fe40b4d90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009bf0 tx=0x7f2ff0009260 comp rx=0 tx=0).stop 2024-05-27T07:07:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.896+0000 7f3005a00700 1 -- 172.21.15.43:0/2917818778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f300010b890 msgr2=0x7f30001a1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.896+0000 7f3005a00700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f300010b890 0x7f30001a1560 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8009790 tx=0x7f2ff800cce0 comp rx=0 tx=0).stop 2024-05-27T07:07:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.896+0000 7f3005a00700 1 -- 172.21.15.43:0/2917818778 shutdown_connections 2024-05-27T07:07:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.896+0000 7f3005a00700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2fe40b2880 0x7f2fe40b4d90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.896+0000 7f3005a00700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f300010b890 0x7f30001a1560 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.896+0000 7f3005a00700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3000106610 0x7f300019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.896+0000 7f3005a00700 1 --2- 172.21.15.43:0/2917818778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f300010af40 0x7f300019d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.896+0000 7f3005a00700 1 -- 172.21.15.43:0/2917818778 >> 172.21.15.43:0/2917818778 conn(0x7f3000102020 msgr2=0x7f300010a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.897+0000 7f3005a00700 1 -- 172.21.15.43:0/2917818778 shutdown_connections 2024-05-27T07:07:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:42.897+0000 7f3005a00700 1 -- 172.21.15.43:0/2917818778 wait complete. 2024-05-27T07:07:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 8 2024-05-27T07:07:43.078 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:07:42 smithi132 podman[65812]: 2024-05-27 07:07:42.729932721 +0000 UTC m=+0.035114288 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:07:43.078 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:07:42 smithi132 podman[65812]: 2024-05-27 07:07:42.904495648 +0000 UTC m=+0.209677237 container create b73a5f07d84bc1c662be33babdcbff5f74212cf9f6362f9f4a7fa31374662ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-h, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T07:07:43.342 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:07:43 smithi132 podman[65812]: 2024-05-27 07:07:43.267736836 +0000 UTC m=+0.572918505 container init b73a5f07d84bc1c662be33babdcbff5f74212cf9f6362f9f4a7fa31374662ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-27T07:07:43.342 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:07:43 smithi132 podman[65812]: 2024-05-27 07:07:43.279206123 +0000 UTC m=+0.584387705 container start b73a5f07d84bc1c662be33babdcbff5f74212cf9f6362f9f4a7fa31374662ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-h, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:07:43.800 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:07:43 smithi132 bash[65812]: b73a5f07d84bc1c662be33babdcbff5f74212cf9f6362f9f4a7fa31374662ff7 2024-05-27T07:07:43.800 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:07:43 smithi132 systemd[1]: Started Ceph mds.h for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:07:43.800 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:07:43 smithi132 ceph-mds[65830]: -- 172.21.15.132:0/50539533 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5595f8a95a40 con 0x5595f8add180 2024-05-27T07:07:43.800 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:07:43 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-h[65826]: starting mds.h at 2024-05-27T07:07:43.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:43 smithi043 ceph-mon[31531]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:43 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2917818778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:43.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:43 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:44.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:43 smithi132 ceph-mon[35283]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:44.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:43 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2917818778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:44.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:44.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:44.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:44.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:44.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:44.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:43 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:43 smithi179 ceph-mon[35888]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:43 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2917818778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:44.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:43 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:44.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:44 smithi179 ceph-mon[35888]: Deploying daemon mds.i on smithi179 2024-05-27T07:07:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:44 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] up:boot 2024-05-27T07:07:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:44 smithi179 ceph-mon[35888]: fsmap 8 up:standby 2024-05-27T07:07:44.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:44 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:44 smithi132 ceph-mon[35283]: Deploying daemon mds.i on smithi179 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:44 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] up:boot 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:44 smithi132 ceph-mon[35283]: fsmap 8 up:standby 2024-05-27T07:07:45.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:44 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:07:45.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:44 smithi043 ceph-mon[31531]: Deploying daemon mds.i on smithi179 2024-05-27T07:07:45.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:44 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] up:boot 2024-05-27T07:07:45.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:44 smithi043 ceph-mon[31531]: fsmap 8 up:standby 2024-05-27T07:07:45.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:44 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:07:45.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:07:45.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:45 smithi179 ceph-mon[35888]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:46.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:45 smithi132 ceph-mon[35283]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:46.180 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:46.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:45 smithi043 ceph-mon[31531]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:46.928 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:07:46 smithi179 systemd[1]: Starting Ceph mds.i for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:07:47.607 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:07:47 smithi179 podman[66424]: 2024-05-27 07:07:47.20063573 +0000 UTC m=+0.034728470 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:07:47.607 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:07:47 smithi179 podman[66424]: 2024-05-27 07:07:47.319201807 +0000 UTC m=+0.153294580 container create 0c675283f0dcfa73cd017db1ece474a9c5e42cdfe35d1bb574feadc8a0d2635a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-i, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T07:07:47.607 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:07:47 smithi179 podman[66424]: 2024-05-27 07:07:47.615209587 +0000 UTC m=+0.449302340 container init 0c675283f0dcfa73cd017db1ece474a9c5e42cdfe35d1bb574feadc8a0d2635a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-i, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-27T07:07:47.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:47 smithi179 ceph-mon[35888]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:47.960 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:07:47 smithi179 podman[66424]: 2024-05-27 07:07:47.626576776 +0000 UTC m=+0.460669570 container start 0c675283f0dcfa73cd017db1ece474a9c5e42cdfe35d1bb574feadc8a0d2635a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:07:47.960 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:07:47 smithi179 bash[66424]: 0c675283f0dcfa73cd017db1ece474a9c5e42cdfe35d1bb574feadc8a0d2635a 2024-05-27T07:07:47.960 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:07:47 smithi179 ceph-mds[66443]: -- 172.21.15.179:0/3831959481 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563b0db01a40 con 0x563b0db49600 2024-05-27T07:07:47.960 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:07:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-i[66439]: starting mds.i at 2024-05-27T07:07:47.960 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:07:47 smithi179 systemd[1]: Started Ceph mds.i for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:07:48.066 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:47 smithi043 ceph-mon[31531]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:48.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:47 smithi132 ceph-mon[35283]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:48.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.318+0000 7f5af5e00700 1 -- 172.21.15.43:0/953413750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5af0108730 msgr2=0x7f5af0110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:48.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.318+0000 7f5af5e00700 1 --2- 172.21.15.43:0/953413750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5af0108730 0x7f5af0110090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800b0d0 tx=0x7f5ae800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:07:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.318+0000 7f5af5e00700 1 -- 172.21.15.43:0/953413750 shutdown_connections 2024-05-27T07:07:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.318+0000 7f5af5e00700 1 --2- 172.21.15.43:0/953413750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5af0108730 0x7f5af0110090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.318+0000 7f5af5e00700 1 --2- 172.21.15.43:0/953413750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5af01034b0 0x7f5af01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.318+0000 7f5af5e00700 1 --2- 172.21.15.43:0/953413750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af0107de0 0x7f5af01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.318+0000 7f5af5e00700 1 -- 172.21.15.43:0/953413750 >> 172.21.15.43:0/953413750 conn(0x7f5af00fedb0 msgr2=0x7f5af01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.318+0000 7f5af5e00700 1 -- 172.21.15.43:0/953413750 shutdown_connections 2024-05-27T07:07:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.318+0000 7f5af5e00700 1 -- 172.21.15.43:0/953413750 wait complete. 2024-05-27T07:07:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.319+0000 7f5af5e00700 1 Processor -- start 2024-05-27T07:07:48.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.320+0000 7f5af5e00700 1 -- start start 2024-05-27T07:07:48.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.320+0000 7f5af5e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af01034b0 0x7f5af00ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:48.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.320+0000 7f5af5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5af0107de0 0x7f5af00ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:48.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.320+0000 7f5af5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5af0108730 0x7f5af00ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:48.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.320+0000 7f5af5e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5af0113260 con 0x7f5af0107de0 2024-05-27T07:07:48.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.320+0000 7f5af5e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5af01130e0 con 0x7f5af0108730 2024-05-27T07:07:48.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.320+0000 7f5af5e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5af01133e0 con 0x7f5af01034b0 2024-05-27T07:07:48.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.321+0000 7f5aee000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5af0107de0 0x7f5af00ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:48.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.321+0000 7f5aee000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5af0107de0 0x7f5af00ff7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53910/0 (socket says 172.21.15.43:53910) 2024-05-27T07:07:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.321+0000 7f5aee000700 1 -- 172.21.15.43:0/4291358537 learned_addr learned my addr 172.21.15.43:0/4291358537 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:07:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.321+0000 7f5aeea00700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af01034b0 0x7f5af00ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.321+0000 7f5aef400700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5af0108730 0x7f5af00ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.321+0000 7f5aee000700 1 -- 172.21.15.43:0/4291358537 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af01034b0 msgr2=0x7f5af00ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.321+0000 7f5aee000700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af01034b0 0x7f5af00ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.321+0000 7f5aee000700 1 -- 172.21.15.43:0/4291358537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5af0108730 msgr2=0x7f5af00ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.321+0000 7f5aee000700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5af0108730 0x7f5af00ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.321+0000 7f5aee000700 1 -- 172.21.15.43:0/4291358537 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af01a9100 con 0x7f5af0107de0 2024-05-27T07:07:48.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.322+0000 7f5aee000700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5af0107de0 0x7f5af00ff7d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00d980 tx=0x7f5adc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:48.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.322+0000 7f5af4a00700 1 -- 172.21.15.43:0/4291358537 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc017070 con 0x7f5af0107de0 2024-05-27T07:07:48.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.322+0000 7f5af5e00700 1 -- 172.21.15.43:0/4291358537 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af01a93f0 con 0x7f5af0107de0 2024-05-27T07:07:48.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.322+0000 7f5af5e00700 1 -- 172.21.15.43:0/4291358537 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af01a9980 con 0x7f5af0107de0 2024-05-27T07:07:48.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.322+0000 7f5af4a00700 1 -- 172.21.15.43:0/4291358537 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5adc009dc0 con 0x7f5af0107de0 2024-05-27T07:07:48.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.322+0000 7f5af4a00700 1 -- 172.21.15.43:0/4291358537 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc00f3c0 con 0x7f5af0107de0 2024-05-27T07:07:48.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.324+0000 7f5af5e00700 1 -- 172.21.15.43:0/4291358537 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af00512b0 con 0x7f5af0107de0 2024-05-27T07:07:48.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.326+0000 7f5af4a00700 1 -- 172.21.15.43:0/4291358537 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5adc009930 con 0x7f5af0107de0 2024-05-27T07:07:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.328+0000 7f5af4a00700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5ad40b2900 0x7f5ad40b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.328+0000 7f5af4a00700 1 -- 172.21.15.43:0/4291358537 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f5adc0e1820 con 0x7f5af0107de0 2024-05-27T07:07:48.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.328+0000 7f5aeea00700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5ad40b2900 0x7f5ad40b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:48.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.329+0000 7f5aeea00700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5ad40b2900 0x7f5ad40b4e10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005950 tx=0x7f5ae000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:48.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.335+0000 7f5af4a00700 1 -- 172.21.15.43:0/4291358537 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5adc0a99c0 con 0x7f5af0107de0 2024-05-27T07:07:48.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.640+0000 7f5af5e00700 1 -- 172.21.15.43:0/4291358537 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5af002e5e0 con 0x7f5af0107de0 2024-05-27T07:07:48.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.641+0000 7f5af4a00700 1 -- 172.21.15.43:0/4291358537 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f5adc0a92c0 con 0x7f5af0107de0 2024-05-27T07:07:48.632 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:07:48.632 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34424,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34448,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-27T07:07:48.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.643+0000 7f5af5e00700 1 -- 172.21.15.43:0/4291358537 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5ad40b2900 msgr2=0x7f5ad40b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:48.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.643+0000 7f5af5e00700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5ad40b2900 0x7f5ad40b4e10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005950 tx=0x7f5ae000b410 comp rx=0 tx=0).stop 2024-05-27T07:07:48.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.643+0000 7f5af5e00700 1 -- 172.21.15.43:0/4291358537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5af0107de0 msgr2=0x7f5af00ff7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:48.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.643+0000 7f5af5e00700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5af0107de0 0x7f5af00ff7d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00d980 tx=0x7f5adc00dc90 comp rx=0 tx=0).stop 2024-05-27T07:07:48.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.644+0000 7f5af5e00700 1 -- 172.21.15.43:0/4291358537 shutdown_connections 2024-05-27T07:07:48.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.644+0000 7f5af5e00700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5ad40b2900 0x7f5ad40b4e10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:48.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.644+0000 7f5af5e00700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5af0108730 0x7f5af00ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:48.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.644+0000 7f5af5e00700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af01034b0 0x7f5af00ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:48.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.644+0000 7f5af5e00700 1 --2- 172.21.15.43:0/4291358537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5af0107de0 0x7f5af00ff7d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:48.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.644+0000 7f5af5e00700 1 -- 172.21.15.43:0/4291358537 >> 172.21.15.43:0/4291358537 conn(0x7f5af00fedb0 msgr2=0x7f5af006b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:48.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.644+0000 7f5af5e00700 1 -- 172.21.15.43:0/4291358537 shutdown_connections 2024-05-27T07:07:48.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:48.644+0000 7f5af5e00700 1 -- 172.21.15.43:0/4291358537 wait complete. 2024-05-27T07:07:48.637 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 10 2024-05-27T07:07:48.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] up:boot 2024-05-27T07:07:48.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: fsmap 9 up:standby 2024-05-27T07:07:48.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:07:48.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:48.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: Deploying daemon mds.g on smithi043 2024-05-27T07:07:48.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:48 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4291358537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] up:boot 2024-05-27T07:07:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: fsmap 9 up:standby 2024-05-27T07:07:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:07:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:49.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: Deploying daemon mds.g on smithi043 2024-05-27T07:07:49.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:48 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4291358537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] up:boot 2024-05-27T07:07:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: fsmap 9 up:standby 2024-05-27T07:07:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:07:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:49.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: Deploying daemon mds.g on smithi043 2024-05-27T07:07:49.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:48 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4291358537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:50.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:49 smithi043 ceph-mon[31531]: pgmap v360: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:50.304 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:49 smithi179 ceph-mon[35888]: pgmap v360: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:50.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:49 smithi132 ceph-mon[35283]: pgmap v360: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:51.426 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:07:52.022 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:51 smithi043 ceph-mon[31531]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:51 smithi179 ceph-mon[35888]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:52.325 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:52.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:51 smithi132 ceph-mon[35283]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:52.453 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:07:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: starting mds.g at 2024-05-27T07:07:53.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:53.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:53.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:53.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:53.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:53.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:53.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: Deploying daemon mds.a on smithi043 2024-05-27T07:07:53.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:53.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6838/807126288,v1:172.21.15.43:6839/807126288] up:boot 2024-05-27T07:07:53.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: fsmap 10 up:standby 2024-05-27T07:07:53.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:07:53.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:53.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:53.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:53.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:53.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:53.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:53.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: Deploying daemon mds.a on smithi043 2024-05-27T07:07:53.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:53.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6838/807126288,v1:172.21.15.43:6839/807126288] up:boot 2024-05-27T07:07:53.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: fsmap 10 up:standby 2024-05-27T07:07:53.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:07:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: Deploying daemon mds.a on smithi043 2024-05-27T07:07:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6838/807126288,v1:172.21.15.43:6839/807126288] up:boot 2024-05-27T07:07:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: fsmap 10 up:standby 2024-05-27T07:07:53.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:07:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.084+0000 7f38db800700 1 -- 172.21.15.43:0/758698060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38d4107e00 msgr2=0x7f38d4108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.084+0000 7f38db800700 1 --2- 172.21.15.43:0/758698060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38d4107e00 0x7f38d4108210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f38c00099c0 tx=0x7f38c0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.084+0000 7f38db800700 1 -- 172.21.15.43:0/758698060 shutdown_connections 2024-05-27T07:07:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.084+0000 7f38db800700 1 --2- 172.21.15.43:0/758698060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38d4108750 0x7f38d41100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.084+0000 7f38db800700 1 --2- 172.21.15.43:0/758698060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f38d41034d0 0x7f38d41038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.084+0000 7f38db800700 1 --2- 172.21.15.43:0/758698060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38d4107e00 0x7f38d4108210 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.084+0000 7f38db800700 1 -- 172.21.15.43:0/758698060 >> 172.21.15.43:0/758698060 conn(0x7f38d40fef20 msgr2=0x7f38d4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.084+0000 7f38db800700 1 -- 172.21.15.43:0/758698060 shutdown_connections 2024-05-27T07:07:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.084+0000 7f38db800700 1 -- 172.21.15.43:0/758698060 wait complete. 2024-05-27T07:07:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.085+0000 7f38db800700 1 Processor -- start 2024-05-27T07:07:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.086+0000 7f38db800700 1 -- start start 2024-05-27T07:07:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.086+0000 7f38db800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38d41034d0 0x7f38d40ff400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.086+0000 7f38db800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f38d4107e00 0x7f38d40ff940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.086+0000 7f38db800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38d4108750 0x7f38d40ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.086+0000 7f38db800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f38d4113260 con 0x7f38d4108750 2024-05-27T07:07:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.086+0000 7f38db800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f38d41130e0 con 0x7f38d41034d0 2024-05-27T07:07:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.086+0000 7f38db800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f38d41133e0 con 0x7f38d4107e00 2024-05-27T07:07:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.087+0000 7f38d9800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38d4108750 0x7f38d40ffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.087+0000 7f38d9800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38d4108750 0x7f38d40ffe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35266/0 (socket says 172.21.15.43:35266) 2024-05-27T07:07:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.087+0000 7f38d9800700 1 -- 172.21.15.43:0/3358312262 learned_addr learned my addr 172.21.15.43:0/3358312262 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:07:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.087+0000 7f38d3e00700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f38d4107e00 0x7f38d40ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.087+0000 7f38d8e00700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38d41034d0 0x7f38d40ff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.087+0000 7f38d9800700 1 -- 172.21.15.43:0/3358312262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f38d4107e00 msgr2=0x7f38d40ff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.087+0000 7f38d9800700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f38d4107e00 0x7f38d40ff940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.087+0000 7f38d9800700 1 -- 172.21.15.43:0/3358312262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38d41034d0 msgr2=0x7f38d40ff400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.087+0000 7f38d9800700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38d41034d0 0x7f38d40ff400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.087+0000 7f38d9800700 1 -- 172.21.15.43:0/3358312262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d41a9120 con 0x7f38d4108750 2024-05-27T07:07:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.088+0000 7f38d9800700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38d4108750 0x7f38d40ffe80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f38cc007ee0 tx=0x7f38cc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.088+0000 7f38d1600700 1 -- 172.21.15.43:0/3358312262 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38cc004bb0 con 0x7f38d4108750 2024-05-27T07:07:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.088+0000 7f38d1600700 1 -- 172.21.15.43:0/3358312262 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f38cc004d50 con 0x7f38d4108750 2024-05-27T07:07:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.088+0000 7f38d1600700 1 -- 172.21.15.43:0/3358312262 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38cc0058b0 con 0x7f38d4108750 2024-05-27T07:07:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.088+0000 7f38db800700 1 -- 172.21.15.43:0/3358312262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d41a9410 con 0x7f38d4108750 2024-05-27T07:07:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.088+0000 7f38db800700 1 -- 172.21.15.43:0/3358312262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d41a9920 con 0x7f38d4108750 2024-05-27T07:07:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.089+0000 7f38db800700 1 -- 172.21.15.43:0/3358312262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d4104980 con 0x7f38d4108750 2024-05-27T07:07:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.092+0000 7f38d1600700 1 -- 172.21.15.43:0/3358312262 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f38cc01a020 con 0x7f38d4108750 2024-05-27T07:07:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.094+0000 7f38d1600700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f38bc0b2900 0x7f38bc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:07:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.094+0000 7f38d1600700 1 -- 172.21.15.43:0/3358312262 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f38cc0e1a30 con 0x7f38d4108750 2024-05-27T07:07:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.094+0000 7f38d8e00700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f38bc0b2900 0x7f38bc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:07:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.095+0000 7f38d8e00700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f38bc0b2900 0x7f38bc0b4e10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009c60 tx=0x7f38c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:07:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.096+0000 7f38d1600700 1 -- 172.21.15.43:0/3358312262 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f38cc010070 con 0x7f38d4108750 2024-05-27T07:07:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.392+0000 7f38db800700 1 -- 172.21.15.43:0/3358312262 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38d40512b0 con 0x7f38d4108750 2024-05-27T07:07:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.393+0000 7f38d1600700 1 -- 172.21.15.43:0/3358312262 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f38cc0a9b50 con 0x7f38d4108750 2024-05-27T07:07:55.385 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:07:55.385 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34424,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34448,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-27T07:07:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.397+0000 7f38db800700 1 -- 172.21.15.43:0/3358312262 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f38bc0b2900 msgr2=0x7f38bc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.397+0000 7f38db800700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f38bc0b2900 0x7f38bc0b4e10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009c60 tx=0x7f38c8009260 comp rx=0 tx=0).stop 2024-05-27T07:07:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.397+0000 7f38db800700 1 -- 172.21.15.43:0/3358312262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38d4108750 msgr2=0x7f38d40ffe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:07:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.397+0000 7f38db800700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38d4108750 0x7f38d40ffe80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f38cc007ee0 tx=0x7f38cc00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.397+0000 7f38db800700 1 -- 172.21.15.43:0/3358312262 shutdown_connections 2024-05-27T07:07:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.398+0000 7f38db800700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f38bc0b2900 0x7f38bc0b4e10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.398+0000 7f38db800700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38d4108750 0x7f38d40ffe80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.398+0000 7f38db800700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38d41034d0 0x7f38d40ff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.398+0000 7f38db800700 1 --2- 172.21.15.43:0/3358312262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f38d4107e00 0x7f38d40ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:07:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.398+0000 7f38db800700 1 -- 172.21.15.43:0/3358312262 >> 172.21.15.43:0/3358312262 conn(0x7f38d40fef20 msgr2=0x7f38d41096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:07:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.398+0000 7f38db800700 1 -- 172.21.15.43:0/3358312262 shutdown_connections 2024-05-27T07:07:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:07:55.398+0000 7f38db800700 1 -- 172.21.15.43:0/3358312262 wait complete. 2024-05-27T07:07:55.391 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 11 2024-05-27T07:07:55.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:55 smithi132 ceph-mon[35283]: pgmap v363: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:55.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:55 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3358312262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:55.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:55 smithi043 ceph-mon[31531]: pgmap v363: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:55.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:55 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3358312262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:55 smithi179 ceph-mon[35888]: pgmap v363: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:55 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3358312262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:07:57.111 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:07:56 smithi043 podman[70824]: 2024-05-27 07:07:56.830859235 +0000 UTC m=+0.163457408 container create 840760a352fe09043f334b550867b03c9a81526b36d2918c680af6bd450ed588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T07:07:57.111 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:07:57 smithi043 podman[70824]: 2024-05-27 07:07:57.118676507 +0000 UTC m=+0.451274664 container init 840760a352fe09043f334b550867b03c9a81526b36d2918c680af6bd450ed588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T07:07:57.453 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:07:57 smithi043 podman[70824]: 2024-05-27 07:07:57.129299084 +0000 UTC m=+0.461897226 container start 840760a352fe09043f334b550867b03c9a81526b36d2918c680af6bd450ed588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:07:57.454 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:07:57 smithi043 bash[70824]: 840760a352fe09043f334b550867b03c9a81526b36d2918c680af6bd450ed588 2024-05-27T07:07:57.454 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:07:57 smithi043 systemd[1]: Started Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:07:57.454 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:07:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: starting mds.a at 2024-05-27T07:07:58.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:57 smithi132 ceph-mon[35283]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:58.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:57 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:57 smithi043 ceph-mon[31531]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:57 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:58.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:57 smithi179 ceph-mon[35888]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:07:58.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:07:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-27T07:07:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-27T07:07:58.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:57 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:07:58.499 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:07:58.938 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:58 smithi132 ceph-mon[35283]: Deploying daemon mds.b on smithi132 2024-05-27T07:07:58.938 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:58 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6840/2697504369,v1:172.21.15.43:6841/2697504369] up:boot 2024-05-27T07:07:58.938 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:58 smithi132 ceph-mon[35283]: fsmap 11 up:standby 2024-05-27T07:07:58.938 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:58 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:07:58.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:58 smithi043 ceph-mon[31531]: Deploying daemon mds.b on smithi132 2024-05-27T07:07:58.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:58 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6840/2697504369,v1:172.21.15.43:6841/2697504369] up:boot 2024-05-27T07:07:58.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:58 smithi043 ceph-mon[31531]: fsmap 11 up:standby 2024-05-27T07:07:58.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:58 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:07:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:07:59.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:58 smithi179 ceph-mon[35888]: Deploying daemon mds.b on smithi132 2024-05-27T07:07:59.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:58 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6840/2697504369,v1:172.21.15.43:6841/2697504369] up:boot 2024-05-27T07:07:59.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:58 smithi179 ceph-mon[35888]: fsmap 11 up:standby 2024-05-27T07:07:59.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:58 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:08:00.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:07:59 smithi043 ceph-mon[31531]: pgmap v365: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:00.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:07:59 smithi132 ceph-mon[35283]: pgmap v365: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:00.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:07:59 smithi179 ceph-mon[35888]: pgmap v365: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:00.927 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:08:00 smithi132 systemd[1]: Starting Ceph mds.b for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:08:01.220 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:08:01 smithi132 podman[66129]: 2024-05-27 07:08:00.938623591 +0000 UTC m=+0.031766546 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:08:01.221 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:08:01 smithi132 podman[66129]: 2024-05-27 07:08:01.054189992 +0000 UTC m=+0.147332984 container create 22b3908559b9711ad263d0860ccaa9332a5dcaf78db8d4e109de55e2ead12ab7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T07:08:01.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.288+0000 7fc60ec00700 1 -- 172.21.15.43:0/3898555682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc608105c50 msgr2=0x7fc60810e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.288+0000 7fc60ec00700 1 --2- 172.21.15.43:0/3898555682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc608105c50 0x7fc60810e1a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009990 tx=0x7fc5f4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.288+0000 7fc60ec00700 1 -- 172.21.15.43:0/3898555682 shutdown_connections 2024-05-27T07:08:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.288+0000 7fc60ec00700 1 --2- 172.21.15.43:0/3898555682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc60810e730 0x7fc608110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.288+0000 7fc60ec00700 1 --2- 172.21.15.43:0/3898555682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc608105300 0x7fc608105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.288+0000 7fc60ec00700 1 --2- 172.21.15.43:0/3898555682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc608105c50 0x7fc60810e1a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.288+0000 7fc60ec00700 1 -- 172.21.15.43:0/3898555682 >> 172.21.15.43:0/3898555682 conn(0x7fc608078ab0 msgr2=0x7fc608078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.288+0000 7fc60ec00700 1 -- 172.21.15.43:0/3898555682 shutdown_connections 2024-05-27T07:08:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.288+0000 7fc60ec00700 1 -- 172.21.15.43:0/3898555682 wait complete. 2024-05-27T07:08:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.290+0000 7fc60ec00700 1 Processor -- start 2024-05-27T07:08:01.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.290+0000 7fc60ec00700 1 -- start start 2024-05-27T07:08:01.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.290+0000 7fc60ec00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc608105300 0x7fc608106210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:01.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.291+0000 7fc60ec00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc608105c50 0x7fc608106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:01.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.291+0000 7fc60ec00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc60810e730 0x7fc608106c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:01.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.291+0000 7fc60ec00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc608114150 con 0x7fc608105300 2024-05-27T07:08:01.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.291+0000 7fc60ec00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc608113fd0 con 0x7fc608105c50 2024-05-27T07:08:01.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.291+0000 7fc60ec00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc6081142d0 con 0x7fc60810e730 2024-05-27T07:08:01.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.291+0000 7fc60d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc608105300 0x7fc608106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:01.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.291+0000 7fc60d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc608105300 0x7fc608106210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35292/0 (socket says 172.21.15.43:35292) 2024-05-27T07:08:01.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.291+0000 7fc60d800700 1 -- 172.21.15.43:0/366073403 learned_addr learned my addr 172.21.15.43:0/366073403 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:01.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.291+0000 7fc60ce00700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc608105c50 0x7fc608106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:01.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.291+0000 7fc60e200700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc60810e730 0x7fc608106c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.292+0000 7fc60d800700 1 -- 172.21.15.43:0/366073403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc60810e730 msgr2=0x7fc608106c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.292+0000 7fc60d800700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc60810e730 0x7fc608106c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.292+0000 7fc60d800700 1 -- 172.21.15.43:0/366073403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc608105c50 msgr2=0x7fc608106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.292+0000 7fc60d800700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc608105c50 0x7fc608106750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.292+0000 7fc60d800700 1 -- 172.21.15.43:0/366073403 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6081ad360 con 0x7fc608105300 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.292+0000 7fc60d800700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc608105300 0x7fc608106210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00d950 tx=0x7fc5fc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.292+0000 7fc606000700 1 -- 172.21.15.43:0/366073403 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc016070 con 0x7fc608105300 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.292+0000 7fc606000700 1 -- 172.21.15.43:0/366073403 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc5fc009dd0 con 0x7fc608105300 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.292+0000 7fc60ec00700 1 -- 172.21.15.43:0/366073403 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6081ad5f0 con 0x7fc608105300 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.292+0000 7fc606000700 1 -- 172.21.15.43:0/366073403 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc00f3c0 con 0x7fc608105300 2024-05-27T07:08:01.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.293+0000 7fc60ec00700 1 -- 172.21.15.43:0/366073403 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6081adb30 con 0x7fc608105300 2024-05-27T07:08:01.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.294+0000 7fc60ec00700 1 -- 172.21.15.43:0/366073403 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60806be50 con 0x7fc608105300 2024-05-27T07:08:01.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.297+0000 7fc606000700 1 -- 172.21.15.43:0/366073403 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc5fc019070 con 0x7fc608105300 2024-05-27T07:08:01.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.298+0000 7fc606000700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc5f00b2880 0x7fc5f00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:01.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.298+0000 7fc606000700 1 -- 172.21.15.43:0/366073403 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fc5fc0e1a40 con 0x7fc608105300 2024-05-27T07:08:01.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.299+0000 7fc60ce00700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc5f00b2880 0x7fc5f00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:01.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.299+0000 7fc60ce00700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc5f00b2880 0x7fc5f00b4d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc608101d60 tx=0x7fc5f4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:01.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.301+0000 7fc606000700 1 -- 172.21.15.43:0/366073403 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc5fc0a9b60 con 0x7fc608105300 2024-05-27T07:08:01.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.598+0000 7fc60ec00700 1 -- 172.21.15.43:0/366073403 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6081add90 con 0x7fc608105300 2024-05-27T07:08:01.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.600+0000 7fc606000700 1 -- 172.21.15.43:0/366073403 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7fc5fc0a9460 con 0x7fc608105300 2024-05-27T07:08:01.592 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:08:01.592 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34424,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34448,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-27T07:08:01.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 -- 172.21.15.43:0/366073403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc5f00b2880 msgr2=0x7fc5f00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:01.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc5f00b2880 0x7fc5f00b4d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc608101d60 tx=0x7fc5f4005fd0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 -- 172.21.15.43:0/366073403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc608105300 msgr2=0x7fc608106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:01.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc608105300 0x7fc608106210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00d950 tx=0x7fc5fc00dc60 comp rx=0 tx=0).stop 2024-05-27T07:08:01.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 -- 172.21.15.43:0/366073403 shutdown_connections 2024-05-27T07:08:01.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc5f00b2880 0x7fc5f00b4d90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc60810e730 0x7fc608106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc608105300 0x7fc608106210 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 --2- 172.21.15.43:0/366073403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc608105c50 0x7fc608106750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:01.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 -- 172.21.15.43:0/366073403 >> 172.21.15.43:0/366073403 conn(0x7fc608078ab0 msgr2=0x7fc608103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:01.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 -- 172.21.15.43:0/366073403 shutdown_connections 2024-05-27T07:08:01.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:01.603+0000 7fc60ec00700 1 -- 172.21.15.43:0/366073403 wait complete. 2024-05-27T07:08:01.597 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 12 2024-05-27T07:08:01.797 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:08:01 smithi132 podman[66129]: 2024-05-27 07:08:01.366713635 +0000 UTC m=+0.459856681 container init 22b3908559b9711ad263d0860ccaa9332a5dcaf78db8d4e109de55e2ead12ab7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True) 2024-05-27T07:08:01.797 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:08:01 smithi132 podman[66129]: 2024-05-27 07:08:01.375866893 +0000 UTC m=+0.469009868 container start 22b3908559b9711ad263d0860ccaa9332a5dcaf78db8d4e109de55e2ead12ab7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T07:08:01.797 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:08:01 smithi132 bash[66129]: 22b3908559b9711ad263d0860ccaa9332a5dcaf78db8d4e109de55e2ead12ab7 2024-05-27T07:08:01.797 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:08:01 smithi132 systemd[1]: Started Ceph mds.b for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:08:01.797 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:08:01 smithi132 ceph-mds[66154]: -- 172.21.15.132:0/2835616213 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5586c1621a40 con 0x5586c1669600 2024-05-27T07:08:01.797 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:08:01 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b[66148]: starting mds.b at 2024-05-27T07:08:01.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:01 smithi043 ceph-mon[31531]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:01.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:01.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:01.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:01 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/366073403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:08:01.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:01.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:01.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:01 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:02.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:01 smithi132 ceph-mon[35283]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:02.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:02.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:02.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:01 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/366073403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:08:02.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:02.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:02.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:01 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:01 smithi179 ceph-mon[35888]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:01 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/366073403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:08:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:01 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:03.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:02 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] up:boot 2024-05-27T07:08:03.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:02 smithi132 ceph-mon[35283]: fsmap 12 up:standby 2024-05-27T07:08:03.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:02 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:08:03.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:02 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:03.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:02 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:03.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:02 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] up:boot 2024-05-27T07:08:03.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:02 smithi043 ceph-mon[31531]: fsmap 12 up:standby 2024-05-27T07:08:03.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:02 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:08:03.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:02 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:03.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:02 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:03.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:02 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] up:boot 2024-05-27T07:08:03.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:02 smithi179 ceph-mon[35888]: fsmap 12 up:standby 2024-05-27T07:08:03.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:02 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:08:03.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:02 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:03.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:02 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:04.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:03 smithi043 ceph-mon[31531]: pgmap v367: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:04.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:03 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:04.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:03 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:03 smithi132 ceph-mon[35283]: pgmap v367: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:03 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:03 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:04.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:08:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:03 smithi179 ceph-mon[35888]: pgmap v367: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:03 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:03 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:05.015 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:06.262 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:05 smithi179 ceph-mon[35888]: pgmap v368: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:06.262 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.262 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.262 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.263 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:05 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:05 smithi132 ceph-mon[35283]: pgmap v368: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:06.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:05 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:05 smithi043 ceph-mon[31531]: pgmap v368: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:06.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:06.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:05 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.268+0000 7f261b600700 1 -- 172.21.15.43:0/3521152922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f261410c5a0 msgr2=0x7f261410c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.268+0000 7f261b600700 1 --2- 172.21.15.43:0/3521152922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f261410c5a0 0x7f261410c9b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f260800b0a0 tx=0x7f260800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.269+0000 7f261b600700 1 -- 172.21.15.43:0/3521152922 shutdown_connections 2024-05-27T07:08:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.269+0000 7f261b600700 1 --2- 172.21.15.43:0/3521152922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2614106eb0 0x7f2614113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.269+0000 7f261b600700 1 --2- 172.21.15.43:0/3521152922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f261410c5a0 0x7f261410c9b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.269+0000 7f261b600700 1 --2- 172.21.15.43:0/3521152922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2614106560 0x7f2614106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.269+0000 7f261b600700 1 -- 172.21.15.43:0/3521152922 >> 172.21.15.43:0/3521152922 conn(0x7f2614102080 msgr2=0x7f26141044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.269+0000 7f261b600700 1 -- 172.21.15.43:0/3521152922 shutdown_connections 2024-05-27T07:08:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.269+0000 7f261b600700 1 -- 172.21.15.43:0/3521152922 wait complete. 2024-05-27T07:08:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.270+0000 7f261b600700 1 Processor -- start 2024-05-27T07:08:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.270+0000 7f261b600700 1 -- start start 2024-05-27T07:08:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.271+0000 7f261b600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2614106560 0x7f261419cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.271+0000 7f261b600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2614106eb0 0x7f261419d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.271+0000 7f261b600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f261410c5a0 0x7f26141a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.271+0000 7f261b600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f26141165b0 con 0x7f261410c5a0 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.271+0000 7f261b600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2614116430 con 0x7f2614106560 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.271+0000 7f261b600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2614116730 con 0x7f2614106eb0 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.271+0000 7f2619600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f261410c5a0 0x7f26141a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.271+0000 7f2619600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f261410c5a0 0x7f26141a17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60884/0 (socket says 172.21.15.43:60884) 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.271+0000 7f2619600700 1 -- 172.21.15.43:0/1796489155 learned_addr learned my addr 172.21.15.43:0/1796489155 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.271+0000 7f2613e00700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2614106eb0 0x7f261419d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.272+0000 7f2618c00700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2614106560 0x7f261419cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.272+0000 7f2619600700 1 -- 172.21.15.43:0/1796489155 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2614106eb0 msgr2=0x7f261419d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.272+0000 7f2619600700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2614106eb0 0x7f261419d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.272+0000 7f2619600700 1 -- 172.21.15.43:0/1796489155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2614106560 msgr2=0x7f261419cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.272+0000 7f2619600700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2614106560 0x7f261419cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.272+0000 7f2619600700 1 -- 172.21.15.43:0/1796489155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26141a1e90 con 0x7f261410c5a0 2024-05-27T07:08:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.272+0000 7f2619600700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f261410c5a0 0x7f26141a17f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f260c00d980 tx=0x7f260c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.272+0000 7f2611600700 1 -- 172.21.15.43:0/1796489155 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260c017070 con 0x7f261410c5a0 2024-05-27T07:08:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.273+0000 7f261b600700 1 -- 172.21.15.43:0/1796489155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26141a2180 con 0x7f261410c5a0 2024-05-27T07:08:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.273+0000 7f261b600700 1 -- 172.21.15.43:0/1796489155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26141a9aa0 con 0x7f261410c5a0 2024-05-27T07:08:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.273+0000 7f2611600700 1 -- 172.21.15.43:0/1796489155 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f260c009dd0 con 0x7f261410c5a0 2024-05-27T07:08:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.273+0000 7f2611600700 1 -- 172.21.15.43:0/1796489155 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260c005260 con 0x7f261410c5a0 2024-05-27T07:08:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.274+0000 7f261b600700 1 -- 172.21.15.43:0/1796489155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25f4005330 con 0x7f261410c5a0 2024-05-27T07:08:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.277+0000 7f2611600700 1 -- 172.21.15.43:0/1796489155 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f260c009940 con 0x7f261410c5a0 2024-05-27T07:08:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.279+0000 7f2611600700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f25fc0b2900 0x7f25fc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.279+0000 7f2611600700 1 -- 172.21.15.43:0/1796489155 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f260c0e20e0 con 0x7f261410c5a0 2024-05-27T07:08:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.279+0000 7f2618c00700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f25fc0b2900 0x7f25fc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.280+0000 7f2618c00700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f25fc0b2900 0x7f25fc0b4e10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2608009fd0 tx=0x7f260800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.282+0000 7f2611600700 1 -- 172.21.15.43:0/1796489155 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f260c020020 con 0x7f261410c5a0 2024-05-27T07:08:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.577+0000 7f261b600700 1 -- 172.21.15.43:0/1796489155 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25f4005620 con 0x7f261410c5a0 2024-05-27T07:08:07.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.578+0000 7f2611600700 1 -- 172.21.15.43:0/1796489155 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10485 (secure 0 0 0) 0x7f260c0aa280 con 0x7f261410c5a0 2024-05-27T07:08:07.571 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:08:07.571 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34424,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34448,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":34484,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-27T07:08:07.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.581+0000 7f261b600700 1 -- 172.21.15.43:0/1796489155 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f25fc0b2900 msgr2=0x7f25fc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:07.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.581+0000 7f261b600700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f25fc0b2900 0x7f25fc0b4e10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2608009fd0 tx=0x7f260800b7b0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.581+0000 7f261b600700 1 -- 172.21.15.43:0/1796489155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f261410c5a0 msgr2=0x7f26141a17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:07.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.581+0000 7f261b600700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f261410c5a0 0x7f26141a17f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f260c00d980 tx=0x7f260c00dd40 comp rx=0 tx=0).stop 2024-05-27T07:08:07.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.582+0000 7f261b600700 1 -- 172.21.15.43:0/1796489155 shutdown_connections 2024-05-27T07:08:07.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.582+0000 7f261b600700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f25fc0b2900 0x7f25fc0b4e10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.582+0000 7f261b600700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f261410c5a0 0x7f26141a17f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.582+0000 7f261b600700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2614106560 0x7f261419cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.582+0000 7f261b600700 1 --2- 172.21.15.43:0/1796489155 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2614106eb0 0x7f261419d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.582+0000 7f261b600700 1 -- 172.21.15.43:0/1796489155 >> 172.21.15.43:0/1796489155 conn(0x7f2614102080 msgr2=0x7f26141048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.582+0000 7f261b600700 1 -- 172.21.15.43:0/1796489155 shutdown_connections 2024-05-27T07:08:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:07.582+0000 7f261b600700 1 -- 172.21.15.43:0/1796489155 wait complete. 2024-05-27T07:08:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 13 2024-05-27T07:08:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:08 smithi179 ceph-mon[35888]: pgmap v369: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:08 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:08.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:08 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:08.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:08 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1796489155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:08:08.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:08 smithi132 ceph-mon[35283]: pgmap v369: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:08.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:08 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:08.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:08 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:08.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:08 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1796489155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:08:08.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:07 smithi043 ceph-mon[31531]: pgmap v369: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:08.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:08.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:07 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:08.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:07 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1796489155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:08:08.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-05-27T07:08:08.979 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:10.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:10 smithi132 ceph-mon[35283]: pgmap v370: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:10.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:10.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:10.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:10.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:10 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:10.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:10 smithi043 ceph-mon[31531]: pgmap v370: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:10.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:10.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:10.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:10.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:10 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:10 smithi179 ceph-mon[35888]: pgmap v370: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:10 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:08:11.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.571+0000 7f50a4200700 1 -- 172.21.15.43:0/169093807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f509c108f10 msgr2=0x7f509c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:11.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.571+0000 7f50a4200700 1 --2- 172.21.15.43:0/169093807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f509c108f10 0x7f509c109320 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f50880099c0 tx=0x7f5088009cd0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.571+0000 7f50a4200700 1 -- 172.21.15.43:0/169093807 shutdown_connections 2024-05-27T07:08:11.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.571+0000 7f50a4200700 1 --2- 172.21.15.43:0/169093807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f509c109860 0x7f509c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.571+0000 7f50a4200700 1 --2- 172.21.15.43:0/169093807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509c1045e0 0x7f509c1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.571+0000 7f50a4200700 1 --2- 172.21.15.43:0/169093807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f509c108f10 0x7f509c109320 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.571+0000 7f50a4200700 1 -- 172.21.15.43:0/169093807 >> 172.21.15.43:0/169093807 conn(0x7f509c100030 msgr2=0x7f509c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:11.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.571+0000 7f50a4200700 1 -- 172.21.15.43:0/169093807 shutdown_connections 2024-05-27T07:08:11.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.571+0000 7f50a4200700 1 -- 172.21.15.43:0/169093807 wait complete. 2024-05-27T07:08:11.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.572+0000 7f50a4200700 1 Processor -- start 2024-05-27T07:08:11.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.573+0000 7f50a4200700 1 -- start start 2024-05-27T07:08:11.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.573+0000 7f50a4200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f509c1045e0 0x7f509c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:11.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.573+0000 7f50a4200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f509c108f10 0x7f509c19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:11.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.573+0000 7f50a4200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509c109860 0x7f509c19f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:11.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.573+0000 7f50a4200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f509c114370 con 0x7f509c108f10 2024-05-27T07:08:11.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.573+0000 7f50a4200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f509c1141f0 con 0x7f509c1045e0 2024-05-27T07:08:11.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.573+0000 7f50a4200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f509c1144f0 con 0x7f509c109860 2024-05-27T07:08:11.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.574+0000 7f50a1800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f509c1045e0 0x7f509c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.574+0000 7f50a1800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f509c1045e0 0x7f509c19adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:35416/0 (socket says 172.21.15.43:35416) 2024-05-27T07:08:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.574+0000 7f50a1800700 1 -- 172.21.15.43:0/2189012351 learned_addr learned my addr 172.21.15.43:0/2189012351 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.574+0000 7f50a0e00700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f509c108f10 0x7f509c19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.574+0000 7f50a2200700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509c109860 0x7f509c19f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.574+0000 7f50a1800700 1 -- 172.21.15.43:0/2189012351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509c109860 msgr2=0x7f509c19f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.574+0000 7f50a1800700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509c109860 0x7f509c19f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.574+0000 7f50a1800700 1 -- 172.21.15.43:0/2189012351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f509c108f10 msgr2=0x7f509c19b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.574+0000 7f50a1800700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f509c108f10 0x7f509c19b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.574+0000 7f50a1800700 1 -- 172.21.15.43:0/2189012351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509c19fc90 con 0x7f509c1045e0 2024-05-27T07:08:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.575+0000 7f50a1800700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f509c1045e0 0x7f509c19adc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f509000e9a0 tx=0x7f509000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:11.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.575+0000 7f509a000700 1 -- 172.21.15.43:0/2189012351 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509000cbd0 con 0x7f509c1045e0 2024-05-27T07:08:11.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.575+0000 7f50a4200700 1 -- 172.21.15.43:0/2189012351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509c19ff80 con 0x7f509c1045e0 2024-05-27T07:08:11.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.575+0000 7f509a000700 1 -- 172.21.15.43:0/2189012351 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f509000cd70 con 0x7f509c1045e0 2024-05-27T07:08:11.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.575+0000 7f509a000700 1 -- 172.21.15.43:0/2189012351 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090018680 con 0x7f509c1045e0 2024-05-27T07:08:11.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.575+0000 7f50a4200700 1 -- 172.21.15.43:0/2189012351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f509c0759e0 con 0x7f509c1045e0 2024-05-27T07:08:11.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.577+0000 7f50a4200700 1 -- 172.21.15.43:0/2189012351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509c105a90 con 0x7f509c1045e0 2024-05-27T07:08:11.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.580+0000 7f509a000700 1 -- 172.21.15.43:0/2189012351 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5090010430 con 0x7f509c1045e0 2024-05-27T07:08:11.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.581+0000 7f509a000700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50840b2880 0x7f50840b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:11.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.582+0000 7f509a000700 1 -- 172.21.15.43:0/2189012351 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f50900e23d0 con 0x7f509c1045e0 2024-05-27T07:08:11.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.582+0000 7f50a0e00700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50840b2880 0x7f50840b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:11.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.582+0000 7f50a0e00700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50840b2880 0x7f50840b4d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5088000c00 tx=0x7f508801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:11.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.591+0000 7f509a000700 1 -- 172.21.15.43:0/2189012351 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f50900aa4f0 con 0x7f509c1045e0 2024-05-27T07:08:11.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.834+0000 7f50a4200700 1 -- 172.21.15.43:0/2189012351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f509c19c120 con 0x7f509c1045e0 2024-05-27T07:08:11.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.835+0000 7f509a000700 1 -- 172.21.15.43:0/2189012351 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f50900a9df0 con 0x7f509c1045e0 2024-05-27T07:08:11.827 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:08:11.827 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-05-27T07:08:11.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:08:11.828 INFO:teuthology.orchestra.run.smithi043.stdout: "poolnum": 1, 2024-05-27T07:08:11.828 INFO:teuthology.orchestra.run.smithi043.stdout: "poolname": ".mgr" 2024-05-27T07:08:11.828 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:08:11.828 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-05-27T07:08:11.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 -- 172.21.15.43:0/2189012351 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50840b2880 msgr2=0x7f50840b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:11.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50840b2880 0x7f50840b4d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5088000c00 tx=0x7f508801a040 comp rx=0 tx=0).stop 2024-05-27T07:08:11.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 -- 172.21.15.43:0/2189012351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f509c1045e0 msgr2=0x7f509c19adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:11.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f509c1045e0 0x7f509c19adc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f509000e9a0 tx=0x7f509000ecb0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 -- 172.21.15.43:0/2189012351 shutdown_connections 2024-05-27T07:08:11.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50840b2880 0x7f50840b4d90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509c109860 0x7f509c19f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f509c1045e0 0x7f509c19adc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 --2- 172.21.15.43:0/2189012351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f509c108f10 0x7f509c19b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:11.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 -- 172.21.15.43:0/2189012351 >> 172.21.15.43:0/2189012351 conn(0x7f509c100030 msgr2=0x7f509c10a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:11.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 -- 172.21.15.43:0/2189012351 shutdown_connections 2024-05-27T07:08:11.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:11.839+0000 7f50a4200700 1 -- 172.21.15.43:0/2189012351 wait complete. 2024-05-27T07:08:12.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:12 smithi043 ceph-mon[31531]: pgmap v371: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:12.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:12 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2189012351' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:08:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:12 smithi132 ceph-mon[35283]: pgmap v371: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:12 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2189012351' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:08:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:12 smithi179 ceph-mon[35888]: pgmap v371: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:12 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2189012351' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-27T07:08:12.881 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-27T07:08:12.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-27T07:08:13.343 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:13.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:13 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:13 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:13 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:13 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:13 smithi043 ceph-mon[31531]: pgmap v372: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:13 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:13 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:13 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:13 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:13 smithi179 ceph-mon[35888]: pgmap v372: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:13.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:13 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:13 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:13 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:13 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:13 smithi132 ceph-mon[35283]: pgmap v372: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.750+0000 7f4e2f200700 1 -- 172.21.15.43:0/2014928313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e30105bd0 msgr2=0x7f4e3010e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.750+0000 7f4e2f200700 1 --2- 172.21.15.43:0/2014928313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e30105bd0 0x7f4e3010e120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4e18009990 tx=0x7f4e18009ca0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.750+0000 7f4e2f200700 1 -- 172.21.15.43:0/2014928313 shutdown_connections 2024-05-27T07:08:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.750+0000 7f4e2f200700 1 --2- 172.21.15.43:0/2014928313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e3010e6f0 0x7f4e30110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.750+0000 7f4e2f200700 1 --2- 172.21.15.43:0/2014928313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e30105280 0x7f4e30105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.750+0000 7f4e2f200700 1 --2- 172.21.15.43:0/2014928313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e30105bd0 0x7f4e3010e120 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.750+0000 7f4e2f200700 1 -- 172.21.15.43:0/2014928313 >> 172.21.15.43:0/2014928313 conn(0x7f4e30078ab0 msgr2=0x7f4e30078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:15.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.750+0000 7f4e2f200700 1 -- 172.21.15.43:0/2014928313 shutdown_connections 2024-05-27T07:08:15.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.750+0000 7f4e2f200700 1 -- 172.21.15.43:0/2014928313 wait complete. 2024-05-27T07:08:15.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.752+0000 7f4e2f200700 1 Processor -- start 2024-05-27T07:08:15.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.752+0000 7f4e2f200700 1 -- start start 2024-05-27T07:08:15.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.752+0000 7f4e2f200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e30105280 0x7f4e3019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:15.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.752+0000 7f4e2f200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e30105bd0 0x7f4e3019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:15.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.752+0000 7f4e2f200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e3010e6f0 0x7f4e301a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:15.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.752+0000 7f4e2f200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4e30114100 con 0x7f4e3010e6f0 2024-05-27T07:08:15.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.752+0000 7f4e2f200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4e30113f80 con 0x7f4e30105bd0 2024-05-27T07:08:15.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.752+0000 7f4e2f200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4e30114280 con 0x7f4e30105280 2024-05-27T07:08:15.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.753+0000 7f4e2e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e3010e6f0 0x7f4e301a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:15.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.753+0000 7f4e2e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e3010e6f0 0x7f4e301a16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49206/0 (socket says 172.21.15.43:49206) 2024-05-27T07:08:15.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.753+0000 7f4e2e800700 1 -- 172.21.15.43:0/3410647566 learned_addr learned my addr 172.21.15.43:0/3410647566 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:15.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.753+0000 7f4e2de00700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e30105280 0x7f4e3019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:15.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.753+0000 7f4e25400700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e30105bd0 0x7f4e3019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:15.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.753+0000 7f4e2e800700 1 -- 172.21.15.43:0/3410647566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e30105280 msgr2=0x7f4e3019ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:15.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.753+0000 7f4e2e800700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e30105280 0x7f4e3019ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.753+0000 7f4e2e800700 1 -- 172.21.15.43:0/3410647566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e30105bd0 msgr2=0x7f4e3019d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:15.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.753+0000 7f4e2e800700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e30105bd0 0x7f4e3019d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:15.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.753+0000 7f4e2e800700 1 -- 172.21.15.43:0/3410647566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e301a1c90 con 0x7f4e3010e6f0 2024-05-27T07:08:15.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.754+0000 7f4e2e800700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e3010e6f0 0x7f4e301a16a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800d980 tx=0x7f4e2800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:15.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.757+0000 7f4e27400700 1 -- 172.21.15.43:0/3410647566 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e28017070 con 0x7f4e3010e6f0 2024-05-27T07:08:15.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.757+0000 7f4e27400700 1 -- 172.21.15.43:0/3410647566 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e28005020 con 0x7f4e3010e6f0 2024-05-27T07:08:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.757+0000 7f4e27400700 1 -- 172.21.15.43:0/3410647566 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e28009990 con 0x7f4e3010e6f0 2024-05-27T07:08:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.757+0000 7f4e2f200700 1 -- 172.21.15.43:0/3410647566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e301a1f80 con 0x7f4e3010e6f0 2024-05-27T07:08:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.758+0000 7f4e2f200700 1 -- 172.21.15.43:0/3410647566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e301a2470 con 0x7f4e3010e6f0 2024-05-27T07:08:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.759+0000 7f4e2f200700 1 -- 172.21.15.43:0/3410647566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e300512b0 con 0x7f4e3010e6f0 2024-05-27T07:08:15.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.762+0000 7f4e27400700 1 -- 172.21.15.43:0/3410647566 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f4e28004a30 con 0x7f4e3010e6f0 2024-05-27T07:08:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.763+0000 7f4e27400700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4e140b2900 0x7f4e140b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.764+0000 7f4e27400700 1 -- 172.21.15.43:0/3410647566 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f4e280e1f30 con 0x7f4e3010e6f0 2024-05-27T07:08:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.764+0000 7f4e2de00700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4e140b2900 0x7f4e140b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:15.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.765+0000 7f4e2de00700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4e140b2900 0x7f4e140b4e10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000f440 tx=0x7f4e200079e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:15.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:15.770+0000 7f4e27400700 1 -- 172.21.15.43:0/3410647566 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4e28111df0 con 0x7f4e3010e6f0 2024-05-27T07:08:16.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.009+0000 7f4e2f200700 1 -- 172.21.15.43:0/3410647566 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4e3019e1d0 con 0x7f4e3010e6f0 2024-05-27T07:08:16.001 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:15 smithi043 ceph-mon[31531]: pgmap v373: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:16.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:16.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:16.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:16.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:16.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:15 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:15 smithi179 ceph-mon[35888]: pgmap v373: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:16.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:15 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:16.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:15 smithi132 ceph-mon[35283]: pgmap v373: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:16.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:16.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:16.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:08:16.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:08:16.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:15 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:08:16.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.967+0000 7f4e27400700 1 -- 172.21.15.43:0/3410647566 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7f4e28020020 con 0x7f4e3010e6f0 2024-05-27T07:08:16.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 -- 172.21.15.43:0/3410647566 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4e140b2900 msgr2=0x7f4e140b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:16.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4e140b2900 0x7f4e140b4e10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000f440 tx=0x7f4e200079e0 comp rx=0 tx=0).stop 2024-05-27T07:08:16.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 -- 172.21.15.43:0/3410647566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e3010e6f0 msgr2=0x7f4e301a16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:16.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e3010e6f0 0x7f4e301a16a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800d980 tx=0x7f4e2800dc90 comp rx=0 tx=0).stop 2024-05-27T07:08:16.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 -- 172.21.15.43:0/3410647566 shutdown_connections 2024-05-27T07:08:16.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f4e140b2900 0x7f4e140b4e10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:16.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e3010e6f0 0x7f4e301a16a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:16.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e30105280 0x7f4e3019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:16.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 --2- 172.21.15.43:0/3410647566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e30105bd0 0x7f4e3019d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:16.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 -- 172.21.15.43:0/3410647566 >> 172.21.15.43:0/3410647566 conn(0x7f4e30078ab0 msgr2=0x7f4e30104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:16.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 -- 172.21.15.43:0/3410647566 shutdown_connections 2024-05-27T07:08:16.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:16.974+0000 7f4e2f200700 1 -- 172.21.15.43:0/3410647566 wait complete. 2024-05-27T07:08:16.969 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' created 2024-05-27T07:08:17.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:16 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3410647566' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:08:17.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:16 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3410647566' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:08:17.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:16 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3410647566' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-27T07:08:17.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-27T07:08:18.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:17 smithi043 ceph-mon[31531]: pgmap v374: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:18.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:17 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3410647566' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:08:18.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:17 smithi043 ceph-mon[31531]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:08:18.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:17 smithi132 ceph-mon[35283]: pgmap v374: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:18.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:17 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3410647566' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:08:18.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:17 smithi132 ceph-mon[35283]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:08:18.346 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:17 smithi179 ceph-mon[35888]: pgmap v374: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:17 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3410647566' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-27T07:08:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:17 smithi179 ceph-mon[35888]: osdmap e73: 12 total, 12 up, 12 in 2024-05-27T07:08:19.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:18 smithi043 ceph-mon[31531]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:08:19.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:18 smithi132 ceph-mon[35283]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:08:19.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:18 smithi179 ceph-mon[35888]: osdmap e74: 12 total, 12 up, 12 in 2024-05-27T07:08:20.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:19 smithi043 ceph-mon[31531]: pgmap v377: 33 pgs: 10 creating+peering, 4 active+clean, 19 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:20.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:19 smithi043 ceph-mon[31531]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:08:20.253 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:19 smithi043 ceph-mon[31531]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:08:20.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:19 smithi132 ceph-mon[35283]: pgmap v377: 33 pgs: 10 creating+peering, 4 active+clean, 19 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:20.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:19 smithi132 ceph-mon[35283]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:08:20.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:19 smithi132 ceph-mon[35283]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:08:20.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:19 smithi179 ceph-mon[35888]: pgmap v377: 33 pgs: 10 creating+peering, 4 active+clean, 19 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:19 smithi179 ceph-mon[35888]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:08:20.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:19 smithi179 ceph-mon[35888]: osdmap e75: 12 total, 12 up, 12 in 2024-05-27T07:08:20.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.644+0000 7fb48a400700 1 -- 172.21.15.43:0/3898287787 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb484103a30 msgr2=0x7fb484110750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:20.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.644+0000 7fb48a400700 1 --2- 172.21.15.43:0/3898287787 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb484103a30 0x7fb484110750 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb470009990 tx=0x7fb470009ca0 comp rx=0 tx=0).stop 2024-05-27T07:08:20.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.644+0000 7fb48a400700 1 -- 172.21.15.43:0/3898287787 shutdown_connections 2024-05-27T07:08:20.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.644+0000 7fb48a400700 1 --2- 172.21.15.43:0/3898287787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb484110c90 0x7fb4841130b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:20.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.644+0000 7fb48a400700 1 --2- 172.21.15.43:0/3898287787 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4841030e0 0x7fb4841034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:20.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.644+0000 7fb48a400700 1 --2- 172.21.15.43:0/3898287787 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb484103a30 0x7fb484110750 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:20.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.644+0000 7fb48a400700 1 -- 172.21.15.43:0/3898287787 >> 172.21.15.43:0/3898287787 conn(0x7fb4840fef10 msgr2=0x7fb484101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:20.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.644+0000 7fb48a400700 1 -- 172.21.15.43:0/3898287787 shutdown_connections 2024-05-27T07:08:20.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.644+0000 7fb48a400700 1 -- 172.21.15.43:0/3898287787 wait complete. 2024-05-27T07:08:20.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.645+0000 7fb48a400700 1 Processor -- start 2024-05-27T07:08:20.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.645+0000 7fb48a400700 1 -- start start 2024-05-27T07:08:20.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.646+0000 7fb48a400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4841030e0 0x7fb48419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:20.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.646+0000 7fb48a400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb484103a30 0x7fb48419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:20.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.646+0000 7fb48a400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb484110c90 0x7fb4841a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:20.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.646+0000 7fb48a400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb484116550 con 0x7fb4841030e0 2024-05-27T07:08:20.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.646+0000 7fb48a400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb4841163d0 con 0x7fb484110c90 2024-05-27T07:08:20.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.646+0000 7fb48a400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb4841166d0 con 0x7fb484103a30 2024-05-27T07:08:20.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.647+0000 7fb483e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb484110c90 0x7fb4841a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:20.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.647+0000 7fb482a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb484103a30 0x7fb48419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:20.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.647+0000 7fb482a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb484103a30 0x7fb48419d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:35050/0 (socket says 172.21.15.43:35050) 2024-05-27T07:08:20.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.647+0000 7fb483e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb484110c90 0x7fb4841a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41610/0 (socket says 172.21.15.43:41610) 2024-05-27T07:08:20.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.647+0000 7fb482a00700 1 -- 172.21.15.43:0/510745047 learned_addr learned my addr 172.21.15.43:0/510745047 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:20.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.647+0000 7fb483400700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4841030e0 0x7fb48419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:20.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.648+0000 7fb482a00700 1 -- 172.21.15.43:0/510745047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb484110c90 msgr2=0x7fb4841a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:20.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.648+0000 7fb482a00700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb484110c90 0x7fb4841a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:20.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.648+0000 7fb482a00700 1 -- 172.21.15.43:0/510745047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4841030e0 msgr2=0x7fb48419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:20.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.648+0000 7fb482a00700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4841030e0 0x7fb48419cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:20.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.648+0000 7fb482a00700 1 -- 172.21.15.43:0/510745047 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4841a1d20 con 0x7fb484103a30 2024-05-27T07:08:20.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.648+0000 7fb483e00700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb484110c90 0x7fb4841a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:08:20.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.648+0000 7fb482a00700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb484103a30 0x7fb48419d4a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb470000c00 tx=0x7fb470003710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:20.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.649+0000 7fb47be00700 1 -- 172.21.15.43:0/510745047 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470004830 con 0x7fb484103a30 2024-05-27T07:08:20.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.649+0000 7fb48a400700 1 -- 172.21.15.43:0/510745047 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4841a1fb0 con 0x7fb484103a30 2024-05-27T07:08:20.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.649+0000 7fb48a400700 1 -- 172.21.15.43:0/510745047 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4841a2480 con 0x7fb484103a30 2024-05-27T07:08:20.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.649+0000 7fb47be00700 1 -- 172.21.15.43:0/510745047 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb470005020 con 0x7fb484103a30 2024-05-27T07:08:20.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.649+0000 7fb47be00700 1 -- 172.21.15.43:0/510745047 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47000f990 con 0x7fb484103a30 2024-05-27T07:08:20.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.651+0000 7fb48a400700 1 -- 172.21.15.43:0/510745047 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4840512b0 con 0x7fb484103a30 2024-05-27T07:08:20.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.654+0000 7fb47be00700 1 -- 172.21.15.43:0/510745047 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb47000fc30 con 0x7fb484103a30 2024-05-27T07:08:20.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.656+0000 7fb47be00700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb4680b2900 0x7fb4680b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:20.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.656+0000 7fb47be00700 1 -- 172.21.15.43:0/510745047 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7497+0+0 (secure 0 0 0) 0x7fb4700e2f90 con 0x7fb484103a30 2024-05-27T07:08:20.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.656+0000 7fb483400700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb4680b2900 0x7fb4680b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:20.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.657+0000 7fb483400700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb4680b2900 0x7fb4680b4e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb47400ac50 tx=0x7fb47400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:20.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.669+0000 7fb47be00700 1 -- 172.21.15.43:0/510745047 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb470013070 con 0x7fb484103a30 2024-05-27T07:08:20.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:20.910+0000 7fb48a400700 1 -- 172.21.15.43:0/510745047 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fb48402e5e0 con 0x7fb484103a30 2024-05-27T07:08:21.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.021+0000 7fb47be00700 1 -- 172.21.15.43:0/510745047 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v76) ==== 171+0+0 (secure 0 0 0) 0x7fb4700aafd0 con 0x7fb484103a30 2024-05-27T07:08:21.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.028+0000 7fb48a400700 1 -- 172.21.15.43:0/510745047 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb4680b2900 msgr2=0x7fb4680b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:21.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.028+0000 7fb48a400700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb4680b2900 0x7fb4680b4e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb47400ac50 tx=0x7fb47400a380 comp rx=0 tx=0).stop 2024-05-27T07:08:21.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.028+0000 7fb48a400700 1 -- 172.21.15.43:0/510745047 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb484103a30 msgr2=0x7fb48419d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:21.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.028+0000 7fb48a400700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb484103a30 0x7fb48419d4a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb470000c00 tx=0x7fb470003710 comp rx=0 tx=0).stop 2024-05-27T07:08:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.029+0000 7fb48a400700 1 -- 172.21.15.43:0/510745047 shutdown_connections 2024-05-27T07:08:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.029+0000 7fb48a400700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb4680b2900 0x7fb4680b4e10 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb47400ac50 tx=0x7fb47400a380 comp rx=0 tx=0).stop 2024-05-27T07:08:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.029+0000 7fb48a400700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb484110c90 0x7fb4841a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.029+0000 7fb48a400700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4841030e0 0x7fb48419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.029+0000 7fb48a400700 1 --2- 172.21.15.43:0/510745047 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb484103a30 0x7fb48419d4a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.029+0000 7fb48a400700 1 -- 172.21.15.43:0/510745047 >> 172.21.15.43:0/510745047 conn(0x7fb4840fef10 msgr2=0x7fb484112790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.031+0000 7fb48a400700 1 -- 172.21.15.43:0/510745047 shutdown_connections 2024-05-27T07:08:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:21.031+0000 7fb48a400700 1 -- 172.21.15.43:0/510745047 wait complete. 2024-05-27T07:08:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' created 2024-05-27T07:08:21.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:21 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/510745047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:08:21.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:21 smithi043 ceph-mon[31531]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:08:21.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:21 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/510745047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:08:21.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:21 smithi132 ceph-mon[35283]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:08:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:21 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/510745047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:08:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:21 smithi179 ceph-mon[35888]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-27T07:08:21.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-27T07:08:22.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:22 smithi043 ceph-mon[31531]: pgmap v379: 33 pgs: 10 creating+peering, 20 active+clean, 3 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:22.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:22 smithi043 ceph-mon[31531]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:08:22.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:22 smithi043 ceph-mon[31531]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:08:22.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:22 smithi132 ceph-mon[35283]: pgmap v379: 33 pgs: 10 creating+peering, 20 active+clean, 3 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:22.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:22 smithi132 ceph-mon[35283]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:08:22.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:22 smithi132 ceph-mon[35283]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:08:22.367 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:22 smithi179 ceph-mon[35888]: pgmap v379: 33 pgs: 10 creating+peering, 20 active+clean, 3 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:22 smithi179 ceph-mon[35888]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-27T07:08:22.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:22 smithi179 ceph-mon[35888]: osdmap e76: 12 total, 12 up, 12 in 2024-05-27T07:08:23.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:23 smithi043 ceph-mon[31531]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:08:23.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:23 smithi132 ceph-mon[35283]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:08:23.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:23 smithi179 ceph-mon[35888]: osdmap e77: 12 total, 12 up, 12 in 2024-05-27T07:08:24.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:24 smithi043 ceph-mon[31531]: pgmap v382: 97 pgs: 54 unknown, 15 creating+peering, 28 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:24.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:24 smithi043 ceph-mon[31531]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:08:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:24 smithi132 ceph-mon[35283]: pgmap v382: 97 pgs: 54 unknown, 15 creating+peering, 28 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:24.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:24 smithi132 ceph-mon[35283]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:08:24.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:24 smithi179 ceph-mon[35888]: pgmap v382: 97 pgs: 54 unknown, 15 creating+peering, 28 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:24.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:24 smithi179 ceph-mon[35888]: osdmap e78: 12 total, 12 up, 12 in 2024-05-27T07:08:24.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.819+0000 7fef8a600700 1 -- 172.21.15.43:0/3643088761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fef84076670 msgr2=0x7fef84110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:24.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.819+0000 7fef8a600700 1 --2- 172.21.15.43:0/3643088761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fef84076670 0x7fef84110460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fef74009990 tx=0x7fef74009ca0 comp rx=0 tx=0).stop 2024-05-27T07:08:24.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.819+0000 7fef8a600700 1 -- 172.21.15.43:0/3643088761 shutdown_connections 2024-05-27T07:08:24.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.819+0000 7fef8a600700 1 --2- 172.21.15.43:0/3643088761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef841109a0 0x7fef84112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:24.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.819+0000 7fef8a600700 1 --2- 172.21.15.43:0/3643088761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef84075d20 0x7fef84076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:24.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.819+0000 7fef8a600700 1 --2- 172.21.15.43:0/3643088761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fef84076670 0x7fef84110460 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:24.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.819+0000 7fef8a600700 1 -- 172.21.15.43:0/3643088761 >> 172.21.15.43:0/3643088761 conn(0x7fef84079d00 msgr2=0x7fef8407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:24.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.819+0000 7fef8a600700 1 -- 172.21.15.43:0/3643088761 shutdown_connections 2024-05-27T07:08:24.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.819+0000 7fef8a600700 1 -- 172.21.15.43:0/3643088761 wait complete. 2024-05-27T07:08:24.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.822+0000 7fef8a600700 1 Processor -- start 2024-05-27T07:08:24.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.822+0000 7fef8a600700 1 -- start start 2024-05-27T07:08:24.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.822+0000 7fef8a600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fef84075d20 0x7fef8419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:24.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.822+0000 7fef8a600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef84076670 0x7fef8419d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:24.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.822+0000 7fef8a600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef841109a0 0x7fef841a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:24.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.822+0000 7fef8a600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fef84116350 con 0x7fef84075d20 2024-05-27T07:08:24.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef8a600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fef841161d0 con 0x7fef841109a0 2024-05-27T07:08:24.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef8a600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fef841164d0 con 0x7fef84076670 2024-05-27T07:08:24.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef89200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fef84075d20 0x7fef8419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:24.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef89200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fef84075d20 0x7fef8419cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56046/0 (socket says 172.21.15.43:56046) 2024-05-27T07:08:24.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef89200700 1 -- 172.21.15.43:0/3760828303 learned_addr learned my addr 172.21.15.43:0/3760828303 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:24.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef83e00700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef84076670 0x7fef8419d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef89c00700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef841109a0 0x7fef841a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef89200700 1 -- 172.21.15.43:0/3760828303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef84076670 msgr2=0x7fef8419d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef89200700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef84076670 0x7fef8419d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef89200700 1 -- 172.21.15.43:0/3760828303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef841109a0 msgr2=0x7fef841a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef89200700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef841109a0 0x7fef841a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef89200700 1 -- 172.21.15.43:0/3760828303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef841a1c10 con 0x7fef84075d20 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef83e00700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef84076670 0x7fef8419d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.823+0000 7fef89c00700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef841109a0 0x7fef841a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.824+0000 7fef89200700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fef84075d20 0x7fef8419cd40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fef7800d950 tx=0x7fef7800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.824+0000 7fef81600700 1 -- 172.21.15.43:0/3760828303 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78016070 con 0x7fef84075d20 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.824+0000 7fef8a600700 1 -- 172.21.15.43:0/3760828303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef841a1f00 con 0x7fef84075d20 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.824+0000 7fef81600700 1 -- 172.21.15.43:0/3760828303 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef78010460 con 0x7fef84075d20 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.824+0000 7fef81600700 1 -- 172.21.15.43:0/3760828303 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7800f620 con 0x7fef84075d20 2024-05-27T07:08:24.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.824+0000 7fef8a600700 1 -- 172.21.15.43:0/3760828303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef841a9820 con 0x7fef84075d20 2024-05-27T07:08:24.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.825+0000 7fef8a600700 1 -- 172.21.15.43:0/3760828303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef840512b0 con 0x7fef84075d20 2024-05-27T07:08:24.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.828+0000 7fef81600700 1 -- 172.21.15.43:0/3760828303 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fef78019070 con 0x7fef84075d20 2024-05-27T07:08:24.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.830+0000 7fef81600700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fef6c0b2880 0x7fef6c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:24.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.830+0000 7fef81600700 1 -- 172.21.15.43:0/3760828303 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7879+0+0 (secure 0 0 0) 0x7fef780e1d10 con 0x7fef84075d20 2024-05-27T07:08:24.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.830+0000 7fef83e00700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fef6c0b2880 0x7fef6c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:24.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.831+0000 7fef83e00700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fef6c0b2880 0x7fef6c0b4d90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fef7400b6f0 tx=0x7fef7400b2a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:24.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:24.833+0000 7fef81600700 1 -- 172.21.15.43:0/3760828303 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fef780a9b40 con 0x7fef84075d20 2024-05-27T07:08:25.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:25.126+0000 7fef8a600700 1 -- 172.21.15.43:0/3760828303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fef8402e5e0 con 0x7fef84075d20 2024-05-27T07:08:26.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.081+0000 7fef81600700 1 -- 172.21.15.43:0/3760828303 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fef780a9440 con 0x7fef84075d20 2024-05-27T07:08:26.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.086+0000 7fef8a600700 1 -- 172.21.15.43:0/3760828303 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fef6c0b2880 msgr2=0x7fef6c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:26.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.086+0000 7fef8a600700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fef6c0b2880 0x7fef6c0b4d90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fef7400b6f0 tx=0x7fef7400b2a0 comp rx=0 tx=0).stop 2024-05-27T07:08:26.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.087+0000 7fef8a600700 1 -- 172.21.15.43:0/3760828303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fef84075d20 msgr2=0x7fef8419cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:26.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.087+0000 7fef8a600700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fef84075d20 0x7fef8419cd40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fef7800d950 tx=0x7fef7800dd10 comp rx=0 tx=0).stop 2024-05-27T07:08:26.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.087+0000 7fef8a600700 1 -- 172.21.15.43:0/3760828303 shutdown_connections 2024-05-27T07:08:26.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.087+0000 7fef8a600700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fef6c0b2880 0x7fef6c0b4d90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:26.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.087+0000 7fef8a600700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef841109a0 0x7fef841a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:26.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.087+0000 7fef8a600700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fef84075d20 0x7fef8419cd40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:26.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.087+0000 7fef8a600700 1 --2- 172.21.15.43:0/3760828303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef84076670 0x7fef8419d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:26.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.087+0000 7fef8a600700 1 -- 172.21.15.43:0/3760828303 >> 172.21.15.43:0/3760828303 conn(0x7fef84079d00 msgr2=0x7fef84100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:26.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.087+0000 7fef8a600700 1 -- 172.21.15.43:0/3760828303 shutdown_connections 2024-05-27T07:08:26.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:26.087+0000 7fef8a600700 1 -- 172.21.15.43:0/3760828303 wait complete. 2024-05-27T07:08:26.080 INFO:teuthology.orchestra.run.smithi043.stderr:new fs with metadata pool 2 and data pool 3 2024-05-27T07:08:26.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:26 smithi132 ceph-mon[35283]: pgmap v384: 97 pgs: 25 unknown, 8 creating+peering, 64 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:26.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:26 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3760828303' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:08:26.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:26 smithi132 ceph-mon[35283]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T07:08:26.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:26 smithi132 ceph-mon[35283]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:08:26.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:26 smithi043 ceph-mon[31531]: pgmap v384: 97 pgs: 25 unknown, 8 creating+peering, 64 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:26.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:26 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3760828303' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:08:26.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:26 smithi043 ceph-mon[31531]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T07:08:26.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:26 smithi043 ceph-mon[31531]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:08:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:26 smithi179 ceph-mon[35888]: pgmap v384: 97 pgs: 25 unknown, 8 creating+peering, 64 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:26 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3760828303' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-27T07:08:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:26 smithi179 ceph-mon[35888]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-27T07:08:26.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:26 smithi179 ceph-mon[35888]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:08:27.007 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-27T07:08:27.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-27T07:08:27.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:27 smithi043 ceph-mon[31531]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:08:27.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:27 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3760828303' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:08:27.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:27 smithi043 ceph-mon[31531]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T07:08:27.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:27 smithi043 ceph-mon[31531]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T07:08:27.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:27 smithi132 ceph-mon[35283]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:08:27.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:27 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3760828303' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:08:27.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:27 smithi132 ceph-mon[35283]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T07:08:27.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:27 smithi132 ceph-mon[35283]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T07:08:27.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:27 smithi179 ceph-mon[35888]: osdmap e79: 12 total, 12 up, 12 in 2024-05-27T07:08:27.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:27 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3760828303' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-27T07:08:27.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:27 smithi179 ceph-mon[35888]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-27T07:08:27.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:27 smithi179 ceph-mon[35888]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-27T07:08:27.486 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:28.247 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:28 smithi043 ceph-mon[31531]: pgmap v386: 97 pgs: 97 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:28.248 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:28 smithi043 ceph-mon[31531]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:08:28.248 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:28 smithi043 ceph-mon[31531]: Cluster is now healthy 2024-05-27T07:08:28.248 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:28 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] up:active 2024-05-27T07:08:28.248 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:28 smithi043 ceph-mon[31531]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:08:28.248 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:28 smithi043 ceph-mon[31531]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:08:28.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:28 smithi132 ceph-mon[35283]: pgmap v386: 97 pgs: 97 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:28.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:28 smithi132 ceph-mon[35283]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:08:28.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:28 smithi132 ceph-mon[35283]: Cluster is now healthy 2024-05-27T07:08:28.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:28 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] up:active 2024-05-27T07:08:28.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:28 smithi132 ceph-mon[35283]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:08:28.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:28 smithi132 ceph-mon[35283]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:08:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:28 smithi179 ceph-mon[35888]: pgmap v386: 97 pgs: 97 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:28 smithi179 ceph-mon[35888]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-27T07:08:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:28 smithi179 ceph-mon[35888]: Cluster is now healthy 2024-05-27T07:08:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:28 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] up:active 2024-05-27T07:08:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:28 smithi179 ceph-mon[35888]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:08:28.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:28 smithi179 ceph-mon[35888]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:08:29.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:29 smithi132 ceph-mon[35283]: pgmap v387: 97 pgs: 97 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-05-27T07:08:29.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:29 smithi043 ceph-mon[31531]: pgmap v387: 97 pgs: 97 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-05-27T07:08:29.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:29 smithi179 ceph-mon[35888]: pgmap v387: 97 pgs: 97 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-05-27T07:08:30.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.129+0000 7fc056600700 1 -- 172.21.15.43:0/4006460123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc050108f10 msgr2=0x7fc050109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:30.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.129+0000 7fc056600700 1 --2- 172.21.15.43:0/4006460123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc050108f10 0x7fc050109320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0099c0 tx=0x7fc03c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:08:30.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.130+0000 7fc056600700 1 -- 172.21.15.43:0/4006460123 shutdown_connections 2024-05-27T07:08:30.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.130+0000 7fc056600700 1 --2- 172.21.15.43:0/4006460123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc050109860 0x7fc0501111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.130+0000 7fc056600700 1 --2- 172.21.15.43:0/4006460123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0501045e0 0x7fc0501049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.130+0000 7fc056600700 1 --2- 172.21.15.43:0/4006460123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc050108f10 0x7fc050109320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.130+0000 7fc056600700 1 -- 172.21.15.43:0/4006460123 >> 172.21.15.43:0/4006460123 conn(0x7fc050100030 msgr2=0x7fc050102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.130+0000 7fc056600700 1 -- 172.21.15.43:0/4006460123 shutdown_connections 2024-05-27T07:08:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.130+0000 7fc056600700 1 -- 172.21.15.43:0/4006460123 wait complete. 2024-05-27T07:08:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.131+0000 7fc056600700 1 Processor -- start 2024-05-27T07:08:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.132+0000 7fc056600700 1 -- start start 2024-05-27T07:08:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.132+0000 7fc056600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0501045e0 0x7fc05019ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.132+0000 7fc056600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc050108f10 0x7fc05019b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.132+0000 7fc056600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc050109860 0x7fc05019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.132+0000 7fc056600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc050114340 con 0x7fc050108f10 2024-05-27T07:08:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.132+0000 7fc056600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc0501141c0 con 0x7fc050109860 2024-05-27T07:08:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.132+0000 7fc056600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc0501144c0 con 0x7fc0501045e0 2024-05-27T07:08:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.133+0000 7fc04ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc050108f10 0x7fc05019b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.133+0000 7fc04ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc050108f10 0x7fc05019b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56058/0 (socket says 172.21.15.43:56058) 2024-05-27T07:08:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.133+0000 7fc04ea00700 1 -- 172.21.15.43:0/72377911 learned_addr learned my addr 172.21.15.43:0/72377911 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.133+0000 7fc04f400700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0501045e0 0x7fc05019ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.133+0000 7fc04fe00700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc050109860 0x7fc05019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:30.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.133+0000 7fc04ea00700 1 -- 172.21.15.43:0/72377911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0501045e0 msgr2=0x7fc05019ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:30.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.133+0000 7fc04ea00700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0501045e0 0x7fc05019ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:30.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.133+0000 7fc04ea00700 1 -- 172.21.15.43:0/72377911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc050109860 msgr2=0x7fc05019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:30.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.133+0000 7fc04ea00700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc050109860 0x7fc05019f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:30.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.133+0000 7fc04ea00700 1 -- 172.21.15.43:0/72377911 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05019fc90 con 0x7fc050108f10 2024-05-27T07:08:30.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.134+0000 7fc04ea00700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc050108f10 0x7fc05019b3b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00b580 tx=0x7fc03c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:30.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.134+0000 7fc047e00700 1 -- 172.21.15.43:0/72377911 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c027070 con 0x7fc050108f10 2024-05-27T07:08:30.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.134+0000 7fc047e00700 1 -- 172.21.15.43:0/72377911 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc03c00fc00 con 0x7fc050108f10 2024-05-27T07:08:30.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.134+0000 7fc056600700 1 -- 172.21.15.43:0/72377911 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc05019ff20 con 0x7fc050108f10 2024-05-27T07:08:30.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.134+0000 7fc047e00700 1 -- 172.21.15.43:0/72377911 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c022cb0 con 0x7fc050108f10 2024-05-27T07:08:30.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.134+0000 7fc056600700 1 -- 172.21.15.43:0/72377911 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0501a78a0 con 0x7fc050108f10 2024-05-27T07:08:30.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.135+0000 7fc056600700 1 -- 172.21.15.43:0/72377911 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0500512b0 con 0x7fc050108f10 2024-05-27T07:08:30.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.138+0000 7fc047e00700 1 -- 172.21.15.43:0/72377911 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fc03c02a070 con 0x7fc050108f10 2024-05-27T07:08:30.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.140+0000 7fc047e00700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0340b2900 0x7fc0340b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:30.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.140+0000 7fc047e00700 1 -- 172.21.15.43:0/72377911 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7fc03c0e3530 con 0x7fc050108f10 2024-05-27T07:08:30.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.140+0000 7fc04f400700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0340b2900 0x7fc0340b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:30.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.141+0000 7fc04f400700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0340b2900 0x7fc0340b4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc040005950 tx=0x7fc04000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.143+0000 7fc047e00700 1 -- 172.21.15.43:0/72377911 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc03c031020 con 0x7fc050108f10 2024-05-27T07:08:30.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.395+0000 7fc056600700 1 -- 172.21.15.43:0/72377911 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fc05019c120 con 0x7fc050108f10 2024-05-27T07:08:30.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.450+0000 7fc047e00700 1 -- 172.21.15.43:0/72377911 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v80) ==== 157+0+0 (secure 0 0 0) 0x7fc03c013070 con 0x7fc050108f10 2024-05-27T07:08:30.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.455+0000 7fc056600700 1 -- 172.21.15.43:0/72377911 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0340b2900 msgr2=0x7fc0340b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:30.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.455+0000 7fc056600700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0340b2900 0x7fc0340b4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc040005950 tx=0x7fc04000b410 comp rx=0 tx=0).stop 2024-05-27T07:08:30.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.455+0000 7fc056600700 1 -- 172.21.15.43:0/72377911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc050108f10 msgr2=0x7fc05019b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.455+0000 7fc056600700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc050108f10 0x7fc05019b3b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00b580 tx=0x7fc03c004790 comp rx=0 tx=0).stop 2024-05-27T07:08:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.455+0000 7fc056600700 1 -- 172.21.15.43:0/72377911 shutdown_connections 2024-05-27T07:08:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.455+0000 7fc056600700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0340b2900 0x7fc0340b4e10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.455+0000 7fc056600700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc050109860 0x7fc05019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.455+0000 7fc056600700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0501045e0 0x7fc05019ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.455+0000 7fc056600700 1 --2- 172.21.15.43:0/72377911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc050108f10 0x7fc05019b3b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.456+0000 7fc056600700 1 -- 172.21.15.43:0/72377911 >> 172.21.15.43:0/72377911 conn(0x7fc050100030 msgr2=0x7fc050102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.456+0000 7fc056600700 1 -- 172.21.15.43:0/72377911 shutdown_connections 2024-05-27T07:08:30.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:30.456+0000 7fc056600700 1 -- 172.21.15.43:0/72377911 wait complete. 2024-05-27T07:08:30.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:30 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/72377911' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:08:30.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:30 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/72377911' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:08:30.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:30 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/72377911' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-27T07:08:31.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-27T07:08:31.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:31 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/72377911' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:08:31.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:31 smithi043 ceph-mon[31531]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:08:31.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:31 smithi043 ceph-mon[31531]: pgmap v389: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T07:08:31.716 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:31 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/72377911' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:08:31.717 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:31 smithi179 ceph-mon[35888]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:08:31.717 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:31 smithi179 ceph-mon[35888]: pgmap v389: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T07:08:31.830 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:31 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/72377911' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-27T07:08:31.831 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:31 smithi132 ceph-mon[35283]: osdmap e80: 12 total, 12 up, 12 in 2024-05-27T07:08:31.831 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:31 smithi132 ceph-mon[35283]: pgmap v389: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-05-27T07:08:31.912 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:32.091 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:08:31 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b[66148]: 2024-05-27T07:08:31.832+0000 7f0941000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T07:08:34.026 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:33 smithi043 ceph-mon[31531]: pgmap v390: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:08:34.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:33 smithi132 ceph-mon[35283]: pgmap v390: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:08:34.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:33 smithi179 ceph-mon[35888]: pgmap v390: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-27T07:08:34.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.293+0000 7f67d3600700 1 -- 172.21.15.43:0/1102919270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67cc109860 msgr2=0x7f67cc1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.293+0000 7f67d3600700 1 --2- 172.21.15.43:0/1102919270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67cc109860 0x7f67cc1111c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f67c40099c0 tx=0x7f67c4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:08:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.293+0000 7f67d3600700 1 -- 172.21.15.43:0/1102919270 shutdown_connections 2024-05-27T07:08:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.293+0000 7f67d3600700 1 --2- 172.21.15.43:0/1102919270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67cc109860 0x7f67cc1111c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.293+0000 7f67d3600700 1 --2- 172.21.15.43:0/1102919270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67cc1045e0 0x7f67cc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.293+0000 7f67d3600700 1 --2- 172.21.15.43:0/1102919270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67cc108f10 0x7f67cc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.293+0000 7f67d3600700 1 -- 172.21.15.43:0/1102919270 >> 172.21.15.43:0/1102919270 conn(0x7f67cc100030 msgr2=0x7f67cc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.293+0000 7f67d3600700 1 -- 172.21.15.43:0/1102919270 shutdown_connections 2024-05-27T07:08:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.294+0000 7f67d3600700 1 -- 172.21.15.43:0/1102919270 wait complete. 2024-05-27T07:08:34.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.295+0000 7f67d3600700 1 Processor -- start 2024-05-27T07:08:34.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.295+0000 7f67d3600700 1 -- start start 2024-05-27T07:08:34.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.295+0000 7f67d3600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67cc1045e0 0x7f67cc19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:34.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.295+0000 7f67d3600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67cc108f10 0x7f67cc19b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:34.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.295+0000 7f67d3600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67cc109860 0x7f67cc19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:34.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d3600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc114340 con 0x7f67cc1045e0 2024-05-27T07:08:34.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d3600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc1141c0 con 0x7f67cc109860 2024-05-27T07:08:34.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d3600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc1144c0 con 0x7f67cc108f10 2024-05-27T07:08:34.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d0c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67cc1045e0 0x7f67cc19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:34.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d0c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67cc1045e0 0x7f67cc19ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56082/0 (socket says 172.21.15.43:56082) 2024-05-27T07:08:34.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d0c00700 1 -- 172.21.15.43:0/3881378328 learned_addr learned my addr 172.21.15.43:0/3881378328 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67cbe00700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67cc108f10 0x7f67cc19b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d1600700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67cc109860 0x7f67cc19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d0c00700 1 -- 172.21.15.43:0/3881378328 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67cc108f10 msgr2=0x7f67cc19b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d0c00700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67cc108f10 0x7f67cc19b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d0c00700 1 -- 172.21.15.43:0/3881378328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67cc109860 msgr2=0x7f67cc19f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d0c00700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67cc109860 0x7f67cc19f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67d0c00700 1 -- 172.21.15.43:0/3881378328 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67cc19fb30 con 0x7f67cc1045e0 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.296+0000 7f67cbe00700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67cc108f10 0x7f67cc19b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.297+0000 7f67d0c00700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67cc1045e0 0x7f67cc19ad10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f67c0007e20 tx=0x7f67c000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.297+0000 7f67c9600700 1 -- 172.21.15.43:0/3881378328 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0016020 con 0x7f67cc1045e0 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.297+0000 7f67c9600700 1 -- 172.21.15.43:0/3881378328 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f67c000ae00 con 0x7f67cc1045e0 2024-05-27T07:08:34.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.297+0000 7f67d3600700 1 -- 172.21.15.43:0/3881378328 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67cc19fe20 con 0x7f67cc1045e0 2024-05-27T07:08:34.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.297+0000 7f67c9600700 1 -- 172.21.15.43:0/3881378328 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0011670 con 0x7f67cc1045e0 2024-05-27T07:08:34.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.298+0000 7f67d3600700 1 -- 172.21.15.43:0/3881378328 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67cc1a0360 con 0x7f67cc1045e0 2024-05-27T07:08:34.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.299+0000 7f67d3600700 1 -- 172.21.15.43:0/3881378328 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67cc0512b0 con 0x7f67cc1045e0 2024-05-27T07:08:34.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.303+0000 7f67c9600700 1 -- 172.21.15.43:0/3881378328 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f67c0019020 con 0x7f67cc1045e0 2024-05-27T07:08:34.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.305+0000 7f67c9600700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f67b40b2880 0x7f67b40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:34.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.305+0000 7f67c9600700 1 -- 172.21.15.43:0/3881378328 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8277+0+0 (secure 0 0 0) 0x7f67c00e2440 con 0x7f67cc1045e0 2024-05-27T07:08:34.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.305+0000 7f67cbe00700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f67b40b2880 0x7f67b40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:34.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.306+0000 7f67cbe00700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f67b40b2880 0x7f67b40b4d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f67b8005e10 tx=0x7f67b8005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:34.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.308+0000 7f67c9600700 1 -- 172.21.15.43:0/3881378328 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f67c00aa0d0 con 0x7f67cc1045e0 2024-05-27T07:08:34.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:34.548+0000 7f67d3600700 1 -- 172.21.15.43:0/3881378328 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f67cc02e5e0 con 0x7f67cc1045e0 2024-05-27T07:08:34.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:34 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3881378328' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:08:35.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:34 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3881378328' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:08:35.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:34 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3881378328' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-27T07:08:35.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.811+0000 7f67c9600700 1 -- 172.21.15.43:0/3881378328 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v82) ==== 228+0+0 (secure 0 0 0) 0x7f67c00b2150 con 0x7f67cc1045e0 2024-05-27T07:08:35.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 -- 172.21.15.43:0/3881378328 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f67b40b2880 msgr2=0x7f67b40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:35.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f67b40b2880 0x7f67b40b4d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f67b8005e10 tx=0x7f67b8005d60 comp rx=0 tx=0).stop 2024-05-27T07:08:35.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 -- 172.21.15.43:0/3881378328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67cc1045e0 msgr2=0x7f67cc19ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:35.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67cc1045e0 0x7f67cc19ad10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f67c0007e20 tx=0x7f67c000a340 comp rx=0 tx=0).stop 2024-05-27T07:08:35.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 -- 172.21.15.43:0/3881378328 shutdown_connections 2024-05-27T07:08:35.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f67b40b2880 0x7f67b40b4d90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:35.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67cc109860 0x7f67cc19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:35.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f67cc1045e0 0x7f67cc19ad10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:35.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 --2- 172.21.15.43:0/3881378328 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67cc108f10 0x7f67cc19b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:35.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 -- 172.21.15.43:0/3881378328 >> 172.21.15.43:0/3881378328 conn(0x7f67cc100030 msgr2=0x7f67cc108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:35.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.816+0000 7f67d3600700 1 -- 172.21.15.43:0/3881378328 shutdown_connections 2024-05-27T07:08:35.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:35.817+0000 7f67d3600700 1 -- 172.21.15.43:0/3881378328 wait complete. 2024-05-27T07:08:35.810 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data_ec' created 2024-05-27T07:08:36.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:35 smithi132 ceph-mon[35283]: pgmap v391: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T07:08:36.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:35 smithi132 ceph-mon[35283]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:08:36.110 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:35 smithi179 ceph-mon[35888]: pgmap v391: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T07:08:36.111 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:35 smithi179 ceph-mon[35888]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:08:36.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:35 smithi043 ceph-mon[31531]: pgmap v391: 97 pgs: 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-05-27T07:08:36.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:35 smithi043 ceph-mon[31531]: osdmap e81: 12 total, 12 up, 12 in 2024-05-27T07:08:36.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-27T07:08:36.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:36 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3881378328' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:08:36.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:36 smithi043 ceph-mon[31531]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:08:37.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:36 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3881378328' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:08:37.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:36 smithi132 ceph-mon[35283]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:08:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:37.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:36 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3881378328' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-27T07:08:37.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:36 smithi179 ceph-mon[35888]: osdmap e82: 12 total, 12 up, 12 in 2024-05-27T07:08:37.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:37 smithi043 ceph-mon[31531]: pgmap v394: 129 pgs: 2 creating+peering, 30 unknown, 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:37.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:37 smithi043 ceph-mon[31531]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:08:37.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:37 smithi043 ceph-mon[31531]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:08:38.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:37 smithi132 ceph-mon[35283]: pgmap v394: 129 pgs: 2 creating+peering, 30 unknown, 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:38.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:37 smithi132 ceph-mon[35283]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:08:38.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:37 smithi132 ceph-mon[35283]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:08:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:37 smithi179 ceph-mon[35888]: pgmap v394: 129 pgs: 2 creating+peering, 30 unknown, 97 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:37 smithi179 ceph-mon[35888]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-27T07:08:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:37 smithi179 ceph-mon[35888]: osdmap e83: 12 total, 12 up, 12 in 2024-05-27T07:08:38.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:38 smithi043 ceph-mon[31531]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:08:39.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:38 smithi179 ceph-mon[35888]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:08:39.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:38 smithi132 ceph-mon[35283]: osdmap e84: 12 total, 12 up, 12 in 2024-05-27T07:08:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.716+0000 7f812ac00700 1 -- 172.21.15.43:0/1925655371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81241045e0 msgr2=0x7f81241049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.716+0000 7f812ac00700 1 --2- 172.21.15.43:0/1925655371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81241045e0 0x7f81241049f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f81140099c0 tx=0x7f8114009cd0 comp rx=0 tx=0).stop 2024-05-27T07:08:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.717+0000 7f812ac00700 1 -- 172.21.15.43:0/1925655371 shutdown_connections 2024-05-27T07:08:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.717+0000 7f812ac00700 1 --2- 172.21.15.43:0/1925655371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8124109860 0x7f81241111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.717+0000 7f812ac00700 1 --2- 172.21.15.43:0/1925655371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81241045e0 0x7f81241049f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.717+0000 7f812ac00700 1 --2- 172.21.15.43:0/1925655371 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8124108f10 0x7f8124109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.717+0000 7f812ac00700 1 -- 172.21.15.43:0/1925655371 >> 172.21.15.43:0/1925655371 conn(0x7f81240ffeb0 msgr2=0x7f8124102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:39.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.717+0000 7f812ac00700 1 -- 172.21.15.43:0/1925655371 shutdown_connections 2024-05-27T07:08:39.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.717+0000 7f812ac00700 1 -- 172.21.15.43:0/1925655371 wait complete. 2024-05-27T07:08:39.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.718+0000 7f812ac00700 1 Processor -- start 2024-05-27T07:08:39.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.718+0000 7f812ac00700 1 -- start start 2024-05-27T07:08:39.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f812ac00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81241045e0 0x7f812419d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:39.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f812ac00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8124108f10 0x7f812419d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:39.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f812ac00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8124109860 0x7f81241a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:39.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f812ac00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8124114480 con 0x7f8124109860 2024-05-27T07:08:39.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f812ac00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8124114300 con 0x7f8124108f10 2024-05-27T07:08:39.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f812ac00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8124114600 con 0x7f81241045e0 2024-05-27T07:08:39.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f8128c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8124109860 0x7f81241a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f8128c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8124109860 0x7f81241a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58906/0 (socket says 172.21.15.43:58906) 2024-05-27T07:08:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f8128c00700 1 -- 172.21.15.43:0/3473069661 learned_addr learned my addr 172.21.15.43:0/3473069661 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f8123400700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8124108f10 0x7f812419d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f8123e00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81241045e0 0x7f812419d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f8128c00700 1 -- 172.21.15.43:0/3473069661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81241045e0 msgr2=0x7f812419d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.719+0000 7f8128c00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81241045e0 0x7f812419d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.720+0000 7f8128c00700 1 -- 172.21.15.43:0/3473069661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8124108f10 msgr2=0x7f812419d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.720+0000 7f8128c00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8124108f10 0x7f812419d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.720+0000 7f8128c00700 1 -- 172.21.15.43:0/3473069661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81241a1fa0 con 0x7f8124109860 2024-05-27T07:08:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.720+0000 7f8128c00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8124109860 0x7f81241a1900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f811c00b970 tx=0x7f811c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.720+0000 7f8120c00700 1 -- 172.21.15.43:0/3473069661 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c019070 con 0x7f8124109860 2024-05-27T07:08:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.720+0000 7f8120c00700 1 -- 172.21.15.43:0/3473069661 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f811c00ce40 con 0x7f8124109860 2024-05-27T07:08:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.720+0000 7f8120c00700 1 -- 172.21.15.43:0/3473069661 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c007370 con 0x7f8124109860 2024-05-27T07:08:39.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.720+0000 7f812ac00700 1 -- 172.21.15.43:0/3473069661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81241a2290 con 0x7f8124109860 2024-05-27T07:08:39.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.721+0000 7f812ac00700 1 -- 172.21.15.43:0/3473069661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81241a9bb0 con 0x7f8124109860 2024-05-27T07:08:39.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.722+0000 7f812ac00700 1 -- 172.21.15.43:0/3473069661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81240512b0 con 0x7f8124109860 2024-05-27T07:08:39.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.726+0000 7f8120c00700 1 -- 172.21.15.43:0/3473069661 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f811c01c070 con 0x7f8124109860 2024-05-27T07:08:39.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.728+0000 7f8120c00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f81080b2880 0x7f81080b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.728+0000 7f8120c00700 1 -- 172.21.15.43:0/3473069661 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8770+0+0 (secure 0 0 0) 0x7f811c0e2170 con 0x7f8124109860 2024-05-27T07:08:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.728+0000 7f8123e00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f81080b2880 0x7f81080b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.729+0000 7f8123e00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f81080b2880 0x7f81080b4d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8114000c00 tx=0x7f8114005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:39.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.731+0000 7f8120c00700 1 -- 172.21.15.43:0/3473069661 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f811c023070 con 0x7f8124109860 2024-05-27T07:08:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:39.971+0000 7f812ac00700 1 -- 172.21.15.43:0/3473069661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f812419e430 con 0x7f8124109860 2024-05-27T07:08:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:39 smithi043 ceph-mon[31531]: pgmap v397: 129 pgs: 2 creating+peering, 11 unknown, 116 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:39 smithi179 ceph-mon[35888]: pgmap v397: 129 pgs: 2 creating+peering, 11 unknown, 116 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:40.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:39 smithi132 ceph-mon[35283]: pgmap v397: 129 pgs: 2 creating+peering, 11 unknown, 116 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:40.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.874+0000 7f8120c00700 1 -- 172.21.15.43:0/3473069661 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) ==== 169+0+0 (secure 0 0 0) 0x7f812419e430 con 0x7f8124109860 2024-05-27T07:08:40.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.878+0000 7f812ac00700 1 -- 172.21.15.43:0/3473069661 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f81080b2880 msgr2=0x7f81080b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:40.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.878+0000 7f812ac00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f81080b2880 0x7f81080b4d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8114000c00 tx=0x7f8114005fb0 comp rx=0 tx=0).stop 2024-05-27T07:08:40.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.878+0000 7f812ac00700 1 -- 172.21.15.43:0/3473069661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8124109860 msgr2=0x7f81241a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:40.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.878+0000 7f812ac00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8124109860 0x7f81241a1900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f811c00b970 tx=0x7f811c00bd30 comp rx=0 tx=0).stop 2024-05-27T07:08:40.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.879+0000 7f812ac00700 1 -- 172.21.15.43:0/3473069661 shutdown_connections 2024-05-27T07:08:40.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.879+0000 7f812ac00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f81080b2880 0x7f81080b4d90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:40.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.879+0000 7f812ac00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8124109860 0x7f81241a1900 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:40.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.879+0000 7f812ac00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81241045e0 0x7f812419d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:40.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.879+0000 7f812ac00700 1 --2- 172.21.15.43:0/3473069661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8124108f10 0x7f812419d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:40.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.879+0000 7f812ac00700 1 -- 172.21.15.43:0/3473069661 >> 172.21.15.43:0/3473069661 conn(0x7f81240ffeb0 msgr2=0x7f812410a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:40.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.879+0000 7f812ac00700 1 -- 172.21.15.43:0/3473069661 shutdown_connections 2024-05-27T07:08:40.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:40.880+0000 7f812ac00700 1 -- 172.21.15.43:0/3473069661 wait complete. 2024-05-27T07:08:40.872 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 4 allow_ec_overwrites to true 2024-05-27T07:08:41.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:40 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3473069661' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:08:41.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:40 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3473069661' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:08:41.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:40 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3473069661' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-27T07:08:41.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-27T07:08:41.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:41 smithi043 ceph-mon[31531]: pgmap v398: 129 pgs: 2 creating+peering, 1 unknown, 126 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:41.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:41 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3473069661' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:08:41.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:41 smithi043 ceph-mon[31531]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:08:42.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:41 smithi179 ceph-mon[35888]: pgmap v398: 129 pgs: 2 creating+peering, 1 unknown, 126 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:41 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3473069661' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:08:42.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:41 smithi179 ceph-mon[35888]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:08:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:42.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:41 smithi132 ceph-mon[35283]: pgmap v398: 129 pgs: 2 creating+peering, 1 unknown, 126 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:42.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:41 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3473069661' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-27T07:08:42.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:41 smithi132 ceph-mon[35283]: osdmap e85: 12 total, 12 up, 12 in 2024-05-27T07:08:43.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:43 smithi043 ceph-mon[31531]: pgmap v400: 129 pgs: 129 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:44.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:43 smithi132 ceph-mon[35283]: pgmap v400: 129 pgs: 129 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:44.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:43 smithi179 ceph-mon[35888]: pgmap v400: 129 pgs: 129 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:44.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.655+0000 7f1c84c00700 1 -- 172.21.15.43:0/263195497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c80110960 msgr2=0x7f1c80110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:44.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.655+0000 7f1c84c00700 1 --2- 172.21.15.43:0/263195497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c80110960 0x7f1c80110d70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009990 tx=0x7f1c70009ca0 comp rx=0 tx=0).stop 2024-05-27T07:08:44.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.655+0000 7f1c84c00700 1 -- 172.21.15.43:0/263195497 shutdown_connections 2024-05-27T07:08:44.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.655+0000 7f1c84c00700 1 --2- 172.21.15.43:0/263195497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c8010b2d0 0x7f1c80117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:44.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.655+0000 7f1c84c00700 1 --2- 172.21.15.43:0/263195497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c80110960 0x7f1c80110d70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:44.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.655+0000 7f1c84c00700 1 --2- 172.21.15.43:0/263195497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8010a980 0x7f1c8010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:44.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.655+0000 7f1c84c00700 1 -- 172.21.15.43:0/263195497 >> 172.21.15.43:0/263195497 conn(0x7f1c80077c70 msgr2=0x7f1c800780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:44.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.656+0000 7f1c84c00700 1 -- 172.21.15.43:0/263195497 shutdown_connections 2024-05-27T07:08:44.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.656+0000 7f1c84c00700 1 -- 172.21.15.43:0/263195497 wait complete. 2024-05-27T07:08:44.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.657+0000 7f1c84c00700 1 Processor -- start 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.657+0000 7f1c84c00700 1 -- start start 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.657+0000 7f1c84c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c8010a980 0x7f1c801a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c84c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8010b2d0 0x7f1c801a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c84c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c80110960 0x7f1c801a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c84c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8011a9e0 con 0x7f1c8010b2d0 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c84c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8011a860 con 0x7f1c80110960 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c84c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8011ab60 con 0x7f1c8010a980 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c7de00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c8010a980 0x7f1c801a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c7d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8010b2d0 0x7f1c801a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c7de00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c8010a980 0x7f1c801a1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:52946/0 (socket says 172.21.15.43:52946) 2024-05-27T07:08:44.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c7d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8010b2d0 0x7f1c801a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38306/0 (socket says 172.21.15.43:38306) 2024-05-27T07:08:44.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c7de00700 1 -- 172.21.15.43:0/1096331222 learned_addr learned my addr 172.21.15.43:0/1096331222 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:44.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.658+0000 7f1c7e800700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c80110960 0x7f1c801a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:44.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.659+0000 7f1c7d400700 1 -- 172.21.15.43:0/1096331222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c8010a980 msgr2=0x7f1c801a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:44.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.659+0000 7f1c7d400700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c8010a980 0x7f1c801a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:44.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.659+0000 7f1c7d400700 1 -- 172.21.15.43:0/1096331222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c80110960 msgr2=0x7f1c801a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:44.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.659+0000 7f1c7d400700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c80110960 0x7f1c801a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:44.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.659+0000 7f1c7d400700 1 -- 172.21.15.43:0/1096331222 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c801a6340 con 0x7f1c8010b2d0 2024-05-27T07:08:44.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.659+0000 7f1c7e800700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c80110960 0x7f1c801a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:08:44.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.659+0000 7f1c7d400700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8010b2d0 0x7f1c801a1a10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00c940 tx=0x7f1c6c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:44.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.659+0000 7f1c76a00700 1 -- 172.21.15.43:0/1096331222 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c007b00 con 0x7f1c8010b2d0 2024-05-27T07:08:44.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.660+0000 7f1c76a00700 1 -- 172.21.15.43:0/1096331222 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1c6c00ce30 con 0x7f1c8010b2d0 2024-05-27T07:08:44.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.660+0000 7f1c76a00700 1 -- 172.21.15.43:0/1096331222 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c0053d0 con 0x7f1c8010b2d0 2024-05-27T07:08:44.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.660+0000 7f1c84c00700 1 -- 172.21.15.43:0/1096331222 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c801a6630 con 0x7f1c8010b2d0 2024-05-27T07:08:44.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.660+0000 7f1c84c00700 1 -- 172.21.15.43:0/1096331222 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c801adf50 con 0x7f1c8010b2d0 2024-05-27T07:08:44.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.661+0000 7f1c84c00700 1 -- 172.21.15.43:0/1096331222 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c800512b0 con 0x7f1c8010b2d0 2024-05-27T07:08:44.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.664+0000 7f1c76a00700 1 -- 172.21.15.43:0/1096331222 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f1c6c005570 con 0x7f1c8010b2d0 2024-05-27T07:08:44.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.666+0000 7f1c76a00700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1c640b2900 0x7f1c640b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:44.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.666+0000 7f1c76a00700 1 -- 172.21.15.43:0/1096331222 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8770+0+0 (secure 0 0 0) 0x7f1c6c0e1ec0 con 0x7f1c8010b2d0 2024-05-27T07:08:44.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.666+0000 7f1c7de00700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1c640b2900 0x7f1c640b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:44.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.667+0000 7f1c7de00700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1c640b2900 0x7f1c640b4e10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1c70003760 tx=0x7f1c7000b560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:44.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.671+0000 7f1c76a00700 1 -- 172.21.15.43:0/1096331222 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1c6c014030 con 0x7f1c8010b2d0 2024-05-27T07:08:44.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:44.971+0000 7f1c84c00700 1 -- 172.21.15.43:0/1096331222 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f1c8002e5e0 con 0x7f1c8010b2d0 2024-05-27T07:08:45.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.902+0000 7f1c76a00700 1 -- 172.21.15.43:0/1096331222 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f1c6c0a9960 con 0x7f1c8010b2d0 2024-05-27T07:08:45.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.906+0000 7f1c84c00700 1 -- 172.21.15.43:0/1096331222 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1c640b2900 msgr2=0x7f1c640b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:45.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.906+0000 7f1c84c00700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1c640b2900 0x7f1c640b4e10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1c70003760 tx=0x7f1c7000b560 comp rx=0 tx=0).stop 2024-05-27T07:08:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.906+0000 7f1c84c00700 1 -- 172.21.15.43:0/1096331222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8010b2d0 msgr2=0x7f1c801a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.906+0000 7f1c84c00700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8010b2d0 0x7f1c801a1a10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00c940 tx=0x7f1c6c00cc50 comp rx=0 tx=0).stop 2024-05-27T07:08:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.907+0000 7f1c84c00700 1 -- 172.21.15.43:0/1096331222 shutdown_connections 2024-05-27T07:08:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.907+0000 7f1c84c00700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1c640b2900 0x7f1c640b4e10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.907+0000 7f1c84c00700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c80110960 0x7f1c801a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.907+0000 7f1c84c00700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c8010a980 0x7f1c801a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:45.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.907+0000 7f1c84c00700 1 --2- 172.21.15.43:0/1096331222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c8010b2d0 0x7f1c801a1a10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:45.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.907+0000 7f1c84c00700 1 -- 172.21.15.43:0/1096331222 >> 172.21.15.43:0/1096331222 conn(0x7f1c80077c70 msgr2=0x7f1c8007bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:45.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.907+0000 7f1c84c00700 1 -- 172.21.15.43:0/1096331222 shutdown_connections 2024-05-27T07:08:45.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:45.907+0000 7f1c84c00700 1 -- 172.21.15.43:0/1096331222 wait complete. 2024-05-27T07:08:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:added data pool 4 to fsmap 2024-05-27T07:08:46.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:45 smithi043 ceph-mon[31531]: pgmap v401: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:46.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:45 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1096331222' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:08:46.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:45 smithi043 ceph-mon[31531]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:08:46.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:45 smithi179 ceph-mon[35888]: pgmap v401: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:46.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:45 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1096331222' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:08:46.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:45 smithi179 ceph-mon[35888]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:08:46.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:45 smithi132 ceph-mon[35283]: pgmap v401: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:46.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:45 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1096331222' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-27T07:08:46.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:45 smithi132 ceph-mon[35283]: osdmap e86: 12 total, 12 up, 12 in 2024-05-27T07:08:46.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:08:47.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:46 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1096331222' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:08:47.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:46 smithi043 ceph-mon[31531]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:08:47.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:46 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1096331222' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:08:47.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:46 smithi179 ceph-mon[35888]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:08:47.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:46 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1096331222' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-27T07:08:47.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:46 smithi132 ceph-mon[35283]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:08:47.439 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:48.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:47 smithi043 ceph-mon[31531]: pgmap v403: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:47 smithi043 ceph-mon[31531]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:08:48.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:47 smithi043 ceph-mon[31531]: Cluster is now healthy 2024-05-27T07:08:48.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:47 smithi179 ceph-mon[35888]: pgmap v403: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:48.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:47 smithi179 ceph-mon[35888]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:08:48.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:47 smithi179 ceph-mon[35888]: Cluster is now healthy 2024-05-27T07:08:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:47 smithi132 ceph-mon[35283]: pgmap v403: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:47 smithi132 ceph-mon[35283]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-27T07:08:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:47 smithi132 ceph-mon[35283]: Cluster is now healthy 2024-05-27T07:08:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.726+0000 7f955f800700 1 -- 172.21.15.43:0/2538393374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9558103240 msgr2=0x7f9558103650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:49.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.727+0000 7f955f800700 1 --2- 172.21.15.43:0/2538393374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9558103240 0x7f9558103650 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f954c00b0a0 tx=0x7f954c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:08:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.727+0000 7f955f800700 1 -- 172.21.15.43:0/2538393374 shutdown_connections 2024-05-27T07:08:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.727+0000 7f955f800700 1 --2- 172.21.15.43:0/2538393374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9558108690 0x7f9558108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.727+0000 7f955f800700 1 --2- 172.21.15.43:0/2538393374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9558103240 0x7f9558103650 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.727+0000 7f955f800700 1 --2- 172.21.15.43:0/2538393374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9558103b90 0x7f95581080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.727+0000 7f955f800700 1 -- 172.21.15.43:0/2538393374 >> 172.21.15.43:0/2538393374 conn(0x7f95580fedb0 msgr2=0x7f95581011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.727+0000 7f955f800700 1 -- 172.21.15.43:0/2538393374 shutdown_connections 2024-05-27T07:08:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.727+0000 7f955f800700 1 -- 172.21.15.43:0/2538393374 wait complete. 2024-05-27T07:08:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.728+0000 7f955f800700 1 Processor -- start 2024-05-27T07:08:49.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.729+0000 7f955f800700 1 -- start start 2024-05-27T07:08:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.729+0000 7f955f800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9558103240 0x7f95581a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.729+0000 7f955f800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9558103b90 0x7f95581a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.729+0000 7f955f800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9558108690 0x7f95581a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.729+0000 7f955f800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f955811a910 con 0x7f9558108690 2024-05-27T07:08:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.729+0000 7f955f800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f955811a790 con 0x7f9558103240 2024-05-27T07:08:49.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.729+0000 7f955f800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f955811aa90 con 0x7f9558103b90 2024-05-27T07:08:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.730+0000 7f955d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9558108690 0x7f95581a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.730+0000 7f955d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9558108690 0x7f95581a5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38328/0 (socket says 172.21.15.43:38328) 2024-05-27T07:08:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.730+0000 7f955d800700 1 -- 172.21.15.43:0/4144150010 learned_addr learned my addr 172.21.15.43:0/4144150010 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.730+0000 7f955ce00700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9558103240 0x7f95581a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:49.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.730+0000 7f9557e00700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9558103b90 0x7f95581a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.730+0000 7f955d800700 1 -- 172.21.15.43:0/4144150010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9558103b90 msgr2=0x7f95581a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.730+0000 7f955d800700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9558103b90 0x7f95581a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.730+0000 7f955d800700 1 -- 172.21.15.43:0/4144150010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9558103240 msgr2=0x7f95581a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.730+0000 7f955d800700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9558103240 0x7f95581a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:49.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.730+0000 7f955d800700 1 -- 172.21.15.43:0/4144150010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95581a6180 con 0x7f9558108690 2024-05-27T07:08:49.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.731+0000 7f955d800700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9558108690 0x7f95581a5b90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f955000c900 tx=0x7f955000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:49.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.731+0000 7f9555600700 1 -- 172.21.15.43:0/4144150010 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95500078d0 con 0x7f9558108690 2024-05-27T07:08:49.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.731+0000 7f9555600700 1 -- 172.21.15.43:0/4144150010 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f955000f450 con 0x7f9558108690 2024-05-27T07:08:49.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.731+0000 7f955f800700 1 -- 172.21.15.43:0/4144150010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95581a6410 con 0x7f9558108690 2024-05-27T07:08:49.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.731+0000 7f9555600700 1 -- 172.21.15.43:0/4144150010 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9550018660 con 0x7f9558108690 2024-05-27T07:08:49.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.731+0000 7f955f800700 1 -- 172.21.15.43:0/4144150010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95581a6950 con 0x7f9558108690 2024-05-27T07:08:49.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.733+0000 7f955f800700 1 -- 172.21.15.43:0/4144150010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95580512b0 con 0x7f9558108690 2024-05-27T07:08:49.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.735+0000 7f9555600700 1 -- 172.21.15.43:0/4144150010 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9550007a70 con 0x7f9558108690 2024-05-27T07:08:49.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.737+0000 7f9555600700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f95400b2900 0x7f95400b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:49.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.737+0000 7f9555600700 1 -- 172.21.15.43:0/4144150010 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f95500e22f0 con 0x7f9558108690 2024-05-27T07:08:49.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.737+0000 7f955ce00700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f95400b2900 0x7f95400b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:49.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.738+0000 7f955ce00700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f95400b2900 0x7f95400b4e10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f954c009fd0 tx=0x7f954c006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:49.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:49.739+0000 7f9555600700 1 -- 172.21.15.43:0/4144150010 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9550014030 con 0x7f9558108690 2024-05-27T07:08:50.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.032+0000 7f955f800700 1 -- 172.21.15.43:0/4144150010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95581adf00 con 0x7f9558108690 2024-05-27T07:08:50.024 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:49 smithi043 ceph-mon[31531]: pgmap v404: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:50.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.035+0000 7f9555600700 1 -- 172.21.15.43:0/4144150010 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11803 (secure 0 0 0) 0x7f95500b0e20 con 0x7f9558108690 2024-05-27T07:08:50.026 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:08:50.026 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34424,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34448,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:08:44.973134+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34484,"qdb_cluster":[34484]},"id":1}]} 2024-05-27T07:08:50.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 -- 172.21.15.43:0/4144150010 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f95400b2900 msgr2=0x7f95400b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:50.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f95400b2900 0x7f95400b4e10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f954c009fd0 tx=0x7f954c006210 comp rx=0 tx=0).stop 2024-05-27T07:08:50.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 -- 172.21.15.43:0/4144150010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9558108690 msgr2=0x7f95581a5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:50.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9558108690 0x7f95581a5b90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f955000c900 tx=0x7f955000cc10 comp rx=0 tx=0).stop 2024-05-27T07:08:50.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 -- 172.21.15.43:0/4144150010 shutdown_connections 2024-05-27T07:08:50.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f95400b2900 0x7f95400b4e10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:50.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9558108690 0x7f95581a5b90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:50.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9558103240 0x7f95581a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:50.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 --2- 172.21.15.43:0/4144150010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9558103b90 0x7f95581a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:50.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 -- 172.21.15.43:0/4144150010 >> 172.21.15.43:0/4144150010 conn(0x7f95580fedb0 msgr2=0x7f9558106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:50.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 -- 172.21.15.43:0/4144150010 shutdown_connections 2024-05-27T07:08:50.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:50.038+0000 7f955f800700 1 -- 172.21.15.43:0/4144150010 wait complete. 2024-05-27T07:08:50.031 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 17 2024-05-27T07:08:50.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:49 smithi179 ceph-mon[35888]: pgmap v404: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:50.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:49 smithi132 ceph-mon[35283]: pgmap v404: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:50.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:08:51.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:50 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4144150010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:08:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:50 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4144150010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:08:51.298 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:51.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:50 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4144150010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:08:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:51 smithi043 ceph-mon[31531]: pgmap v405: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:51 smithi179 ceph-mon[35888]: pgmap v405: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:52.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:51 smithi132 ceph-mon[35283]: pgmap v405: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:53.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-05-27T07:08:53.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-05-27T07:08:53.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-27T07:08:53.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-05-27T07:08:53.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-27T07:08:53.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-05-27T07:08:53.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-05-27T07:08:53.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-05-27T07:08:53.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-05-27T07:08:53.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-05-27T07:08:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-05-27T07:08:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-05-27T07:08:53.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-27T07:08:53.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-05-27T07:08:53.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-27T07:08:53.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-05-27T07:08:53.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-05-27T07:08:53.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-05-27T07:08:53.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-05-27T07:08:53.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-05-27T07:08:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-05-27T07:08:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-05-27T07:08:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-27T07:08:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-05-27T07:08:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-27T07:08:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-05-27T07:08:53.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-05-27T07:08:53.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-05-27T07:08:53.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-05-27T07:08:53.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-05-27T07:08:53.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.512+0000 7f7e49200700 1 -- 172.21.15.43:0/4233763202 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e44106eb0 msgr2=0x7f7e44113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.512+0000 7f7e49200700 1 --2- 172.21.15.43:0/4233763202 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e44106eb0 0x7f7e44113340 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00b0a0 tx=0x7f7e3c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:08:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.512+0000 7f7e49200700 1 -- 172.21.15.43:0/4233763202 shutdown_connections 2024-05-27T07:08:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.512+0000 7f7e49200700 1 --2- 172.21.15.43:0/4233763202 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e44106eb0 0x7f7e44113340 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.512+0000 7f7e49200700 1 --2- 172.21.15.43:0/4233763202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e4410c5a0 0x7f7e4410c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.512+0000 7f7e49200700 1 --2- 172.21.15.43:0/4233763202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e44106560 0x7f7e44106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.512+0000 7f7e49200700 1 -- 172.21.15.43:0/4233763202 >> 172.21.15.43:0/4233763202 conn(0x7f7e44102040 msgr2=0x7f7e44104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.512+0000 7f7e49200700 1 -- 172.21.15.43:0/4233763202 shutdown_connections 2024-05-27T07:08:53.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.512+0000 7f7e49200700 1 -- 172.21.15.43:0/4233763202 wait complete. 2024-05-27T07:08:53.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.513+0000 7f7e49200700 1 Processor -- start 2024-05-27T07:08:53.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.514+0000 7f7e49200700 1 -- start start 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.514+0000 7f7e49200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e44106560 0x7f7e4419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.514+0000 7f7e49200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e44106eb0 0x7f7e4419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.514+0000 7f7e49200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e4410c5a0 0x7f7e441a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.514+0000 7f7e49200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e44116580 con 0x7f7e44106560 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.514+0000 7f7e49200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7e44116400 con 0x7f7e4410c5a0 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.514+0000 7f7e49200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7e44116700 con 0x7f7e44106eb0 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.514+0000 7f7e41e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e44106560 0x7f7e4419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.514+0000 7f7e41e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e44106560 0x7f7e4419ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38358/0 (socket says 172.21.15.43:38358) 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.514+0000 7f7e41e00700 1 -- 172.21.15.43:0/2695092820 learned_addr learned my addr 172.21.15.43:0/2695092820 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.515+0000 7f7e41400700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e44106eb0 0x7f7e4419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:53.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.515+0000 7f7e42800700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e4410c5a0 0x7f7e441a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.515+0000 7f7e41e00700 1 -- 172.21.15.43:0/2695092820 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e44106eb0 msgr2=0x7f7e4419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.515+0000 7f7e41e00700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e44106eb0 0x7f7e4419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.515+0000 7f7e41e00700 1 -- 172.21.15.43:0/2695092820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e4410c5a0 msgr2=0x7f7e441a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.515+0000 7f7e41e00700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e4410c5a0 0x7f7e441a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.515+0000 7f7e41e00700 1 -- 172.21.15.43:0/2695092820 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e441a1d60 con 0x7f7e44106560 2024-05-27T07:08:53.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.515+0000 7f7e42800700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e4410c5a0 0x7f7e441a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:08:53.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.515+0000 7f7e41e00700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e44106560 0x7f7e4419ce90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400d980 tx=0x7f7e3400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:53.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.516+0000 7f7e3aa00700 1 -- 172.21.15.43:0/2695092820 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34017070 con 0x7f7e44106560 2024-05-27T07:08:53.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.516+0000 7f7e3aa00700 1 -- 172.21.15.43:0/2695092820 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e34009dc0 con 0x7f7e44106560 2024-05-27T07:08:53.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.516+0000 7f7e49200700 1 -- 172.21.15.43:0/2695092820 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e441a2050 con 0x7f7e44106560 2024-05-27T07:08:53.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.516+0000 7f7e3aa00700 1 -- 172.21.15.43:0/2695092820 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3400f3c0 con 0x7f7e44106560 2024-05-27T07:08:53.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.516+0000 7f7e49200700 1 -- 172.21.15.43:0/2695092820 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e441a9970 con 0x7f7e44106560 2024-05-27T07:08:53.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.517+0000 7f7e49200700 1 -- 172.21.15.43:0/2695092820 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e440512b0 con 0x7f7e44106560 2024-05-27T07:08:53.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.520+0000 7f7e3aa00700 1 -- 172.21.15.43:0/2695092820 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f7e34009930 con 0x7f7e44106560 2024-05-27T07:08:53.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.521+0000 7f7e3aa00700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7e280b2880 0x7f7e280b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:53.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.522+0000 7f7e3aa00700 1 -- 172.21.15.43:0/2695092820 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f7e340e20b0 con 0x7f7e44106560 2024-05-27T07:08:53.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.522+0000 7f7e41400700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7e280b2880 0x7f7e280b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:53.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.523+0000 7f7e41400700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7e280b2880 0x7f7e280b4d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7e44105650 tx=0x7f7e3000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:53.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.524+0000 7f7e3aa00700 1 -- 172.21.15.43:0/2695092820 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7e340a9a10 con 0x7f7e44106560 2024-05-27T07:08:53.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.760+0000 7f7e49200700 1 -- 172.21.15.43:0/2695092820 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e4402e5e0 con 0x7f7e44106560 2024-05-27T07:08:53.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.765+0000 7f7e3aa00700 1 -- 172.21.15.43:0/2695092820 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+24656 (secure 0 0 0) 0x7f7e340a9310 con 0x7f7e44106560 2024-05-27T07:08:53.757 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:08:53.758 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":87,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:08:52.930837+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:08:53.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.769+0000 7f7e49200700 1 -- 172.21.15.43:0/2695092820 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7e280b2880 msgr2=0x7f7e280b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:53.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.769+0000 7f7e49200700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7e280b2880 0x7f7e280b4d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7e44105650 tx=0x7f7e3000b410 comp rx=0 tx=0).stop 2024-05-27T07:08:53.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.769+0000 7f7e49200700 1 -- 172.21.15.43:0/2695092820 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e44106560 msgr2=0x7f7e4419ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:53.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.769+0000 7f7e49200700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e44106560 0x7f7e4419ce90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400d980 tx=0x7f7e3400dc90 comp rx=0 tx=0).stop 2024-05-27T07:08:53.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.769+0000 7f7e49200700 1 -- 172.21.15.43:0/2695092820 shutdown_connections 2024-05-27T07:08:53.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.769+0000 7f7e49200700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7e280b2880 0x7f7e280b4d90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:53.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.769+0000 7f7e49200700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e4410c5a0 0x7f7e441a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:53.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.769+0000 7f7e49200700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e44106560 0x7f7e4419ce90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:53.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.769+0000 7f7e49200700 1 --2- 172.21.15.43:0/2695092820 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e44106eb0 0x7f7e4419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:53.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.770+0000 7f7e49200700 1 -- 172.21.15.43:0/2695092820 >> 172.21.15.43:0/2695092820 conn(0x7f7e44102040 msgr2=0x7f7e441037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:53.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.770+0000 7f7e49200700 1 -- 172.21.15.43:0/2695092820 shutdown_connections 2024-05-27T07:08:53.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:53.770+0000 7f7e49200700 1 -- 172.21.15.43:0/2695092820 wait complete. 2024-05-27T07:08:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: pgmap v406: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-05-27T07:08:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-05-27T07:08:54.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-27T07:08:54.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-05-27T07:08:54.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-27T07:08:54.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-05-27T07:08:54.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-05-27T07:08:54.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-05-27T07:08:54.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-05-27T07:08:54.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-05-27T07:08:54.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:08:54.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:53 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2695092820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:08:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: pgmap v406: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-05-27T07:08:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-05-27T07:08:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-27T07:08:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-05-27T07:08:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-27T07:08:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-05-27T07:08:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-05-27T07:08:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-05-27T07:08:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-05-27T07:08:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-05-27T07:08:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:08:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:53 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2695092820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:08:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: pgmap v406: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:08:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-05-27T07:08:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-05-27T07:08:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-27T07:08:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-05-27T07:08:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-27T07:08:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-05-27T07:08:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-05-27T07:08:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-05-27T07:08:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-05-27T07:08:54.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-05-27T07:08:54.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: osdmap e87: 12 total, 12 up, 12 in 2024-05-27T07:08:54.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:53 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2695092820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:08:54.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:08:55.222 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:08:55.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:54 smithi132 ceph-mon[35283]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:08:55.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:54 smithi043 ceph-mon[31531]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:08:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:54 smithi179 ceph-mon[35888]: osdmap e88: 12 total, 12 up, 12 in 2024-05-27T07:08:56.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:55 smithi043 ceph-mon[31531]: pgmap v409: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-05-27T07:08:56.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:55 smithi132 ceph-mon[35283]: pgmap v409: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-05-27T07:08:56.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:55 smithi179 ceph-mon[35888]: pgmap v409: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-05-27T07:08:57.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.562+0000 7fcb69c00700 1 -- 172.21.15.43:0/4226129360 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb64075d20 msgr2=0x7fcb64076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:57.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.562+0000 7fcb69c00700 1 --2- 172.21.15.43:0/4226129360 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb64075d20 0x7fcb64076130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcb54009990 tx=0x7fcb54009ca0 comp rx=0 tx=0).stop 2024-05-27T07:08:57.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.562+0000 7fcb69c00700 1 -- 172.21.15.43:0/4226129360 shutdown_connections 2024-05-27T07:08:57.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.562+0000 7fcb69c00700 1 --2- 172.21.15.43:0/4226129360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb64110b90 0x7fcb64112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:57.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.562+0000 7fcb69c00700 1 --2- 172.21.15.43:0/4226129360 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb64075d20 0x7fcb64076130 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:57.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.562+0000 7fcb69c00700 1 --2- 172.21.15.43:0/4226129360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb64076670 0x7fcb64110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:57.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.562+0000 7fcb69c00700 1 -- 172.21.15.43:0/4226129360 >> 172.21.15.43:0/4226129360 conn(0x7fcb64100030 msgr2=0x7fcb64102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:57.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.562+0000 7fcb69c00700 1 -- 172.21.15.43:0/4226129360 shutdown_connections 2024-05-27T07:08:57.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.562+0000 7fcb69c00700 1 -- 172.21.15.43:0/4226129360 wait complete. 2024-05-27T07:08:57.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.563+0000 7fcb69c00700 1 Processor -- start 2024-05-27T07:08:57.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.564+0000 7fcb69c00700 1 -- start start 2024-05-27T07:08:57.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.564+0000 7fcb69c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb64075d20 0x7fcb6419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:57.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.564+0000 7fcb69c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb64076670 0x7fcb6419d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:57.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.564+0000 7fcb69c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb64110b90 0x7fcb641a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:57.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.564+0000 7fcb69c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcb64116580 con 0x7fcb64076670 2024-05-27T07:08:57.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.564+0000 7fcb69c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcb64116400 con 0x7fcb64075d20 2024-05-27T07:08:57.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.564+0000 7fcb69c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fcb64116700 con 0x7fcb64110b90 2024-05-27T07:08:57.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.564+0000 7fcb5a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb64076670 0x7fcb6419d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:57.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.564+0000 7fcb5a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb64076670 0x7fcb6419d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35972/0 (socket says 172.21.15.43:35972) 2024-05-27T07:08:57.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.564+0000 7fcb5a000700 1 -- 172.21.15.43:0/835437770 learned_addr learned my addr 172.21.15.43:0/835437770 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:08:57.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.565+0000 7fcb62a00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb64075d20 0x7fcb6419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:57.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.565+0000 7fcb63400700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb64110b90 0x7fcb641a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:57.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.565+0000 7fcb5a000700 1 -- 172.21.15.43:0/835437770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb64110b90 msgr2=0x7fcb641a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:57.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.565+0000 7fcb5a000700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb64110b90 0x7fcb641a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:57.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.565+0000 7fcb5a000700 1 -- 172.21.15.43:0/835437770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb64075d20 msgr2=0x7fcb6419cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:57.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.565+0000 7fcb5a000700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb64075d20 0x7fcb6419cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:57.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.565+0000 7fcb5a000700 1 -- 172.21.15.43:0/835437770 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb641a1d90 con 0x7fcb64076670 2024-05-27T07:08:57.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.565+0000 7fcb62a00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb64075d20 0x7fcb6419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:08:57.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.565+0000 7fcb5a000700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb64076670 0x7fcb6419d510 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00d950 tx=0x7fcb4c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.566+0000 7fcb5be00700 1 -- 172.21.15.43:0/835437770 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c016070 con 0x7fcb64076670 2024-05-27T07:08:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.566+0000 7fcb5be00700 1 -- 172.21.15.43:0/835437770 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb4c010460 con 0x7fcb64076670 2024-05-27T07:08:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.566+0000 7fcb5be00700 1 -- 172.21.15.43:0/835437770 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c00f620 con 0x7fcb64076670 2024-05-27T07:08:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.566+0000 7fcb69c00700 1 -- 172.21.15.43:0/835437770 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb641a2080 con 0x7fcb64076670 2024-05-27T07:08:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.566+0000 7fcb69c00700 1 -- 172.21.15.43:0/835437770 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb641a2570 con 0x7fcb64076670 2024-05-27T07:08:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.567+0000 7fcb69c00700 1 -- 172.21.15.43:0/835437770 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb640512b0 con 0x7fcb64076670 2024-05-27T07:08:57.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.570+0000 7fcb5be00700 1 -- 172.21.15.43:0/835437770 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fcb4c019070 con 0x7fcb64076670 2024-05-27T07:08:57.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.571+0000 7fcb5be00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcb480b2900 0x7fcb480b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:08:57.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.572+0000 7fcb5be00700 1 -- 172.21.15.43:0/835437770 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fcb4c0e2160 con 0x7fcb64076670 2024-05-27T07:08:57.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.572+0000 7fcb62a00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcb480b2900 0x7fcb480b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:08:57.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.572+0000 7fcb62a00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcb480b2900 0x7fcb480b4e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcb54006010 tx=0x7fcb5400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:08:57.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.575+0000 7fcb5be00700 1 -- 172.21.15.43:0/835437770 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcb4c0a9ac0 con 0x7fcb64076670 2024-05-27T07:08:57.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.816+0000 7fcb69c00700 1 -- 172.21.15.43:0/835437770 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb6402e5e0 con 0x7fcb64076670 2024-05-27T07:08:57.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.821+0000 7fcb5be00700 1 -- 172.21.15.43:0/835437770 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24656 (secure 0 0 0) 0x7fcb4c0a93c0 con 0x7fcb64076670 2024-05-27T07:08:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:08:57.812 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":88,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:08:53.944807+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:08:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.824+0000 7fcb69c00700 1 -- 172.21.15.43:0/835437770 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcb480b2900 msgr2=0x7fcb480b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.824+0000 7fcb69c00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcb480b2900 0x7fcb480b4e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcb54006010 tx=0x7fcb5400b540 comp rx=0 tx=0).stop 2024-05-27T07:08:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.824+0000 7fcb69c00700 1 -- 172.21.15.43:0/835437770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb64076670 msgr2=0x7fcb6419d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:08:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.824+0000 7fcb69c00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb64076670 0x7fcb6419d510 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00d950 tx=0x7fcb4c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:08:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.824+0000 7fcb69c00700 1 -- 172.21.15.43:0/835437770 shutdown_connections 2024-05-27T07:08:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.824+0000 7fcb69c00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcb480b2900 0x7fcb480b4e10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.824+0000 7fcb69c00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb64110b90 0x7fcb641a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.824+0000 7fcb69c00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb64075d20 0x7fcb6419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.824+0000 7fcb69c00700 1 --2- 172.21.15.43:0/835437770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb64076670 0x7fcb6419d510 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:08:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.824+0000 7fcb69c00700 1 -- 172.21.15.43:0/835437770 >> 172.21.15.43:0/835437770 conn(0x7fcb64100030 msgr2=0x7fcb641006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:08:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.825+0000 7fcb69c00700 1 -- 172.21.15.43:0/835437770 shutdown_connections 2024-05-27T07:08:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:08:57.825+0000 7fcb69c00700 1 -- 172.21.15.43:0/835437770 wait complete. 2024-05-27T07:08:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:57 smithi043 ceph-mon[31531]: pgmap v410: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 1 objects/s recovering 2024-05-27T07:08:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:57 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/835437770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:08:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:57 smithi132 ceph-mon[35283]: pgmap v410: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 1 objects/s recovering 2024-05-27T07:08:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:57 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/835437770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:08:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:57 smithi179 ceph-mon[35888]: pgmap v410: 129 pgs: 1 peering, 128 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 1 objects/s recovering 2024-05-27T07:08:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:57 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/835437770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:08:58.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:08:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:00.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:08:59 smithi132 ceph-mon[35283]: pgmap v411: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 1 objects/s recovering 2024-05-27T07:09:00.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:08:59 smithi043 ceph-mon[31531]: pgmap v411: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 1 objects/s recovering 2024-05-27T07:09:00.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:08:59 smithi179 ceph-mon[35888]: pgmap v411: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 1 objects/s recovering 2024-05-27T07:09:01.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.447+0000 7f111b000700 1 -- 172.21.15.43:0/1530618841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1114106570 msgr2=0x7f1114106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:01.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.447+0000 7f111b000700 1 --2- 172.21.15.43:0/1530618841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1114106570 0x7f1114106980 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009990 tx=0x7f10fc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.447+0000 7f111b000700 1 -- 172.21.15.43:0/1530618841 shutdown_connections 2024-05-27T07:09:01.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.447+0000 7f111b000700 1 --2- 172.21.15.43:0/1530618841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1114106ec0 0x7f1114113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.447+0000 7f111b000700 1 --2- 172.21.15.43:0/1530618841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f111410c5b0 0x7f111410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.447+0000 7f111b000700 1 --2- 172.21.15.43:0/1530618841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1114106570 0x7f1114106980 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.447+0000 7f111b000700 1 -- 172.21.15.43:0/1530618841 >> 172.21.15.43:0/1530618841 conn(0x7f11141020b0 msgr2=0x7f1114104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:01.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.447+0000 7f111b000700 1 -- 172.21.15.43:0/1530618841 shutdown_connections 2024-05-27T07:09:01.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.448+0000 7f111b000700 1 -- 172.21.15.43:0/1530618841 wait complete. 2024-05-27T07:09:01.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.449+0000 7f111b000700 1 Processor -- start 2024-05-27T07:09:01.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.449+0000 7f111b000700 1 -- start start 2024-05-27T07:09:01.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f111b000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1114106570 0x7f111419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:01.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f111b000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1114106ec0 0x7f111419d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:01.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f111b000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f111410c5b0 0x7f11141a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:01.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f111b000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1114116590 con 0x7f1114106ec0 2024-05-27T07:09:01.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f111b000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1114116410 con 0x7f1114106570 2024-05-27T07:09:01.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f111b000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1114116710 con 0x7f111410c5b0 2024-05-27T07:09:01.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f110b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1114106ec0 0x7f111419d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:01.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f110b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1114106ec0 0x7f111419d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35986/0 (socket says 172.21.15.43:35986) 2024-05-27T07:09:01.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f110b400700 1 -- 172.21.15.43:0/1623471894 learned_addr learned my addr 172.21.15.43:0/1623471894 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:01.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f1113e00700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1114106570 0x7f111419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:01.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f110b400700 1 -- 172.21.15.43:0/1623471894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f111410c5b0 msgr2=0x7f11141a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:01.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.450+0000 7f1119000700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f111410c5b0 0x7f11141a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:01.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.451+0000 7f110b400700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f111410c5b0 0x7f11141a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.451+0000 7f110b400700 1 -- 172.21.15.43:0/1623471894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1114106570 msgr2=0x7f111419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:01.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.451+0000 7f110b400700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1114106570 0x7f111419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.451+0000 7f110b400700 1 -- 172.21.15.43:0/1623471894 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11141a1dc0 con 0x7f1114106ec0 2024-05-27T07:09:01.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.451+0000 7f110b400700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1114106ec0 0x7f111419d540 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0098e0 tx=0x7f10fc00bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:01.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.451+0000 7f1111600700 1 -- 172.21.15.43:0/1623471894 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc027070 con 0x7f1114106ec0 2024-05-27T07:09:01.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.451+0000 7f111b000700 1 -- 172.21.15.43:0/1623471894 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11141a2050 con 0x7f1114106ec0 2024-05-27T07:09:01.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.451+0000 7f1111600700 1 -- 172.21.15.43:0/1623471894 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f10fc00fb80 con 0x7f1114106ec0 2024-05-27T07:09:01.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.451+0000 7f111b000700 1 -- 172.21.15.43:0/1623471894 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11141a2530 con 0x7f1114106ec0 2024-05-27T07:09:01.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.452+0000 7f1111600700 1 -- 172.21.15.43:0/1623471894 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc031060 con 0x7f1114106ec0 2024-05-27T07:09:01.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.453+0000 7f111b000700 1 -- 172.21.15.43:0/1623471894 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1114107c90 con 0x7f1114106ec0 2024-05-27T07:09:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.455+0000 7f1111600700 1 -- 172.21.15.43:0/1623471894 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f10fc02a070 con 0x7f1114106ec0 2024-05-27T07:09:01.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.457+0000 7f1111600700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f10f80b2880 0x7f10f80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:01.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.457+0000 7f1111600700 1 -- 172.21.15.43:0/1623471894 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f10fc0e3590 con 0x7f1114106ec0 2024-05-27T07:09:01.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.457+0000 7f1113e00700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f10f80b2880 0x7f10f80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:01.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.458+0000 7f1113e00700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f10f80b2880 0x7f10f80b4d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1104005950 tx=0x7f11040058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:01.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.460+0000 7f1111600700 1 -- 172.21.15.43:0/1623471894 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f10fc013070 con 0x7f1114106ec0 2024-05-27T07:09:01.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.754+0000 7f111b000700 1 -- 172.21.15.43:0/1623471894 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f111419e300 con 0x7f1114106ec0 2024-05-27T07:09:01.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.756+0000 7f1111600700 1 -- 172.21.15.43:0/1623471894 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11803 (secure 0 0 0) 0x7f10fc0aaf80 con 0x7f1114106ec0 2024-05-27T07:09:01.747 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:09:01.747 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":34403,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34424,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":34448,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:08:44.973134+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34484,"qdb_cluster":[34484]},"id":1}]} 2024-05-27T07:09:01.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.759+0000 7f111b000700 1 -- 172.21.15.43:0/1623471894 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f10f80b2880 msgr2=0x7f10f80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:01.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.759+0000 7f111b000700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f10f80b2880 0x7f10f80b4d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1104005950 tx=0x7f11040058e0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.759+0000 7f111b000700 1 -- 172.21.15.43:0/1623471894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1114106ec0 msgr2=0x7f111419d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.759+0000 7f111b000700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1114106ec0 0x7f111419d540 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0098e0 tx=0x7f10fc00bfa0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.759+0000 7f111b000700 1 -- 172.21.15.43:0/1623471894 shutdown_connections 2024-05-27T07:09:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.759+0000 7f111b000700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f10f80b2880 0x7f10f80b4d90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.759+0000 7f111b000700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f111410c5b0 0x7f11141a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.759+0000 7f111b000700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1114106570 0x7f111419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.759+0000 7f111b000700 1 --2- 172.21.15.43:0/1623471894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1114106ec0 0x7f111419d540 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.759+0000 7f111b000700 1 -- 172.21.15.43:0/1623471894 >> 172.21.15.43:0/1623471894 conn(0x7f11141020b0 msgr2=0x7f1114104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.760+0000 7f111b000700 1 -- 172.21.15.43:0/1623471894 shutdown_connections 2024-05-27T07:09:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:01.760+0000 7f111b000700 1 -- 172.21.15.43:0/1623471894 wait complete. 2024-05-27T07:09:01.752 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 17 2024-05-27T07:09:02.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:02 smithi043 ceph-mon[31531]: pgmap v412: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 1 objects/s recovering 2024-05-27T07:09:02.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:02 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1623471894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:09:02.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:02 smithi132 ceph-mon[35283]: pgmap v412: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 1 objects/s recovering 2024-05-27T07:09:02.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:02 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1623471894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:09:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:02 smithi179 ceph-mon[35888]: pgmap v412: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 201 B/s, 1 objects/s recovering 2024-05-27T07:09:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:02 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1623471894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:09:02.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:09:03.094 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:04.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:04 smithi043 ceph-mon[31531]: pgmap v413: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 164 B/s, 1 objects/s recovering 2024-05-27T07:09:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:04 smithi132 ceph-mon[35283]: pgmap v413: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 164 B/s, 1 objects/s recovering 2024-05-27T07:09:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:04 smithi179 ceph-mon[35888]: pgmap v413: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 164 B/s, 1 objects/s recovering 2024-05-27T07:09:05.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.427+0000 7f892c800700 1 -- 172.21.15.43:0/4089906712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f892410c5a0 msgr2=0x7f892410c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:05.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.427+0000 7f892c800700 1 --2- 172.21.15.43:0/4089906712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f892410c5a0 0x7f892410c9b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8918009990 tx=0x7f8918009ca0 comp rx=0 tx=0).stop 2024-05-27T07:09:05.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.427+0000 7f892c800700 1 -- 172.21.15.43:0/4089906712 shutdown_connections 2024-05-27T07:09:05.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.427+0000 7f892c800700 1 --2- 172.21.15.43:0/4089906712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8924106eb0 0x7f8924113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:05.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.427+0000 7f892c800700 1 --2- 172.21.15.43:0/4089906712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f892410c5a0 0x7f892410c9b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:05.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.427+0000 7f892c800700 1 --2- 172.21.15.43:0/4089906712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8924106560 0x7f8924106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:05.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.427+0000 7f892c800700 1 -- 172.21.15.43:0/4089906712 >> 172.21.15.43:0/4089906712 conn(0x7f8924102040 msgr2=0x7f8924104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:05.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.427+0000 7f892c800700 1 -- 172.21.15.43:0/4089906712 shutdown_connections 2024-05-27T07:09:05.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.427+0000 7f892c800700 1 -- 172.21.15.43:0/4089906712 wait complete. 2024-05-27T07:09:05.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.429+0000 7f892c800700 1 Processor -- start 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.429+0000 7f892c800700 1 -- start start 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.429+0000 7f892c800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8924106560 0x7f892419cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.429+0000 7f892c800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8924106eb0 0x7f892419d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.429+0000 7f892c800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f892410c5a0 0x7f89241a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.429+0000 7f892c800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8924116560 con 0x7f8924106eb0 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.429+0000 7f892c800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89241163e0 con 0x7f8924106560 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.429+0000 7f892c800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f89241166e0 con 0x7f892410c5a0 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f8929400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8924106eb0 0x7f892419d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f8929400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8924106eb0 0x7f892419d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60610/0 (socket says 172.21.15.43:60610) 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f8929400700 1 -- 172.21.15.43:0/1672253721 learned_addr learned my addr 172.21.15.43:0/1672253721 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f8929e00700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8924106560 0x7f892419cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:05.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f892a800700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f892410c5a0 0x7f89241a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:05.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f8929400700 1 -- 172.21.15.43:0/1672253721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f892410c5a0 msgr2=0x7f89241a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f8929400700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f892410c5a0 0x7f89241a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f8929400700 1 -- 172.21.15.43:0/1672253721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8924106560 msgr2=0x7f892419cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f8929400700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8924106560 0x7f892419cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f8929400700 1 -- 172.21.15.43:0/1672253721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89241a1e20 con 0x7f8924106eb0 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.430+0000 7f8929e00700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8924106560 0x7f892419cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.431+0000 7f8929400700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8924106eb0 0x7f892419d4f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f891400c8c0 tx=0x7f891400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.431+0000 7f891ea00700 1 -- 172.21.15.43:0/1672253721 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8914004220 con 0x7f8924106eb0 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.431+0000 7f891ea00700 1 -- 172.21.15.43:0/1672253721 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8914007d90 con 0x7f8924106eb0 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.431+0000 7f892c800700 1 -- 172.21.15.43:0/1672253721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89241a2110 con 0x7f8924106eb0 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.431+0000 7f891ea00700 1 -- 172.21.15.43:0/1672253721 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8914004e30 con 0x7f8924106eb0 2024-05-27T07:09:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.431+0000 7f892c800700 1 -- 172.21.15.43:0/1672253721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89241a9a30 con 0x7f8924106eb0 2024-05-27T07:09:05.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.433+0000 7f892c800700 1 -- 172.21.15.43:0/1672253721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f892410e960 con 0x7f8924106eb0 2024-05-27T07:09:05.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.436+0000 7f891ea00700 1 -- 172.21.15.43:0/1672253721 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f89140043c0 con 0x7f8924106eb0 2024-05-27T07:09:05.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.437+0000 7f891ea00700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f890c0b2880 0x7f890c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.438+0000 7f891ea00700 1 -- 172.21.15.43:0/1672253721 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f89140e2510 con 0x7f8924106eb0 2024-05-27T07:09:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.438+0000 7f8929e00700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f890c0b2880 0x7f890c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.439+0000 7f8929e00700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f890c0b2880 0x7f890c0b4d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8918003760 tx=0x7f891800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:05.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.439+0000 7f891ea00700 1 -- 172.21.15.43:0/1672253721 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8914014030 con 0x7f8924106eb0 2024-05-27T07:09:05.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.686+0000 7f892c800700 1 -- 172.21.15.43:0/1672253721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f892419e2b0 con 0x7f8924106eb0 2024-05-27T07:09:05.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.689+0000 7f891ea00700 1 -- 172.21.15.43:0/1672253721 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24656 (secure 0 0 0) 0x7f89140a9e70 con 0x7f8924106eb0 2024-05-27T07:09:05.681 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:09:05.681 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":88,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:08:53.944807+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:09:05.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.692+0000 7f892c800700 1 -- 172.21.15.43:0/1672253721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f890c0b2880 msgr2=0x7f890c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:05.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.692+0000 7f892c800700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f890c0b2880 0x7f890c0b4d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8918003760 tx=0x7f891800b540 comp rx=0 tx=0).stop 2024-05-27T07:09:05.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.692+0000 7f892c800700 1 -- 172.21.15.43:0/1672253721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8924106eb0 msgr2=0x7f892419d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:05.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.692+0000 7f892c800700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8924106eb0 0x7f892419d4f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f891400c8c0 tx=0x7f891400cc80 comp rx=0 tx=0).stop 2024-05-27T07:09:05.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.693+0000 7f892c800700 1 -- 172.21.15.43:0/1672253721 shutdown_connections 2024-05-27T07:09:05.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.693+0000 7f892c800700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f890c0b2880 0x7f890c0b4d90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:05.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.693+0000 7f892c800700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f892410c5a0 0x7f89241a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:05.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.693+0000 7f892c800700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8924106560 0x7f892419cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:05.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.693+0000 7f892c800700 1 --2- 172.21.15.43:0/1672253721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8924106eb0 0x7f892419d4f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:05.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.693+0000 7f892c800700 1 -- 172.21.15.43:0/1672253721 >> 172.21.15.43:0/1672253721 conn(0x7f8924102040 msgr2=0x7f8924103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:05.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.693+0000 7f892c800700 1 -- 172.21.15.43:0/1672253721 shutdown_connections 2024-05-27T07:09:05.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:05.693+0000 7f892c800700 1 -- 172.21.15.43:0/1672253721 wait complete. 2024-05-27T07:09:06.262 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:06 smithi179 ceph-mon[35888]: pgmap v414: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 149 B/s, 0 objects/s recovering 2024-05-27T07:09:06.262 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:06 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1672253721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:09:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:06 smithi132 ceph-mon[35283]: pgmap v414: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 149 B/s, 0 objects/s recovering 2024-05-27T07:09:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:06 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1672253721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:09:06.451 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:06 smithi043 ceph-mon[31531]: pgmap v414: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 149 B/s, 0 objects/s recovering 2024-05-27T07:09:06.451 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:06 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1672253721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:09:06.524 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-27T07:08:25.128097+0000', 'modified': '2024-05-27T07:08:44.973134+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34484}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34484': {'gid': 34484, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.132:6839/4104367678', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 4104367678}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 4104367678}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 34484, 'qdb_cluster': [34484]} 2024-05-27T07:09:06.524 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-27T07:09:06.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T07:09:06.987 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:08.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:08 smithi132 ceph-mon[35283]: pgmap v415: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 134 B/s, 0 objects/s recovering 2024-05-27T07:09:08.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:08 smithi179 ceph-mon[35888]: pgmap v415: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 134 B/s, 0 objects/s recovering 2024-05-27T07:09:08.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:08 smithi043 ceph-mon[31531]: pgmap v415: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 134 B/s, 0 objects/s recovering 2024-05-27T07:09:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.361+0000 7f29bcc00700 1 -- 172.21.15.43:0/3931763722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b8076e30 msgr2=0x7f29b8077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.361+0000 7f29bcc00700 1 --2- 172.21.15.43:0/3931763722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b8076e30 0x7f29b8077240 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009990 tx=0x7f29a8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.362+0000 7f29bcc00700 1 -- 172.21.15.43:0/3931763722 shutdown_connections 2024-05-27T07:09:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.362+0000 7f29bcc00700 1 --2- 172.21.15.43:0/3931763722 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29b8110b70 0x7f29b8112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.362+0000 7f29bcc00700 1 --2- 172.21.15.43:0/3931763722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b8076e30 0x7f29b8077240 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.362+0000 7f29bcc00700 1 --2- 172.21.15.43:0/3931763722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29b8077780 0x7f29b8110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.362+0000 7f29bcc00700 1 -- 172.21.15.43:0/3931763722 >> 172.21.15.43:0/3931763722 conn(0x7f29b80ffe90 msgr2=0x7f29b81022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.362+0000 7f29bcc00700 1 -- 172.21.15.43:0/3931763722 shutdown_connections 2024-05-27T07:09:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.362+0000 7f29bcc00700 1 -- 172.21.15.43:0/3931763722 wait complete. 2024-05-27T07:09:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.363+0000 7f29bcc00700 1 Processor -- start 2024-05-27T07:09:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.363+0000 7f29bcc00700 1 -- start start 2024-05-27T07:09:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29bcc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29b8076e30 0x7f29b819cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29bcc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b8077780 0x7f29b819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29bcc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29b8110b70 0x7f29b81a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29bcc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f29b8116510 con 0x7f29b8077780 2024-05-27T07:09:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29bcc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f29b8116390 con 0x7f29b8110b70 2024-05-27T07:09:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29bcc00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f29b8116690 con 0x7f29b8076e30 2024-05-27T07:09:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29b5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b8077780 0x7f29b819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29b5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b8077780 0x7f29b819d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60626/0 (socket says 172.21.15.43:60626) 2024-05-27T07:09:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29b5400700 1 -- 172.21.15.43:0/4009068294 learned_addr learned my addr 172.21.15.43:0/4009068294 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29b5e00700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29b8076e30 0x7f29b819cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.364+0000 7f29b6800700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29b8110b70 0x7f29b81a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29b5400700 1 -- 172.21.15.43:0/4009068294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29b8076e30 msgr2=0x7f29b819cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29b5400700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29b8076e30 0x7f29b819cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29b5400700 1 -- 172.21.15.43:0/4009068294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29b8110b70 msgr2=0x7f29b81a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29b5400700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29b8110b70 0x7f29b81a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29b5400700 1 -- 172.21.15.43:0/4009068294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b81a1d20 con 0x7f29b8077780 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29b5e00700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29b8076e30 0x7f29b819cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29b5400700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b8077780 0x7f29b819d4a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f29a400c9b0 tx=0x7f29a400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29aea00700 1 -- 172.21.15.43:0/4009068294 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4007b00 con 0x7f29b8077780 2024-05-27T07:09:09.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29aea00700 1 -- 172.21.15.43:0/4009068294 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29a4007ca0 con 0x7f29b8077780 2024-05-27T07:09:09.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29bcc00700 1 -- 172.21.15.43:0/4009068294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b81a2010 con 0x7f29b8077780 2024-05-27T07:09:09.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.365+0000 7f29aea00700 1 -- 172.21.15.43:0/4009068294 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4018660 con 0x7f29b8077780 2024-05-27T07:09:09.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.366+0000 7f29bcc00700 1 -- 172.21.15.43:0/4009068294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b81a99e0 con 0x7f29b8077780 2024-05-27T07:09:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.370+0000 7f29aea00700 1 -- 172.21.15.43:0/4009068294 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f29a400f450 con 0x7f29b8077780 2024-05-27T07:09:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.371+0000 7f29cb7a4b80 1 Processor -- start 2024-05-27T07:09:09.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.371+0000 7f29cb7a4b80 1 -- start start 2024-05-27T07:09:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.371+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b481d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.371+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b481d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.371+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b481de20 0x55c5b482a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.371+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c5b474bdf0 con 0x55c5b478a9a0 2024-05-27T07:09:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.371+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c5b473b990 con 0x55c5b481de20 2024-05-27T07:09:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.371+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c5b473f9d0 con 0x55c5b469edf0 2024-05-27T07:09:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29b6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b481d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29b6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b481d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60630/0 (socket says 172.21.15.43:60630) 2024-05-27T07:09:09.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29b6800700 1 -- 172.21.15.43:0/1803411695 learned_addr learned my addr 172.21.15.43:0/1803411695 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:09.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29b5400700 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b481de20 0x55c5b482a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29aea00700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f299c0b2900 0x7f299c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29b5e00700 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b481d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29b6800700 1 -- 172.21.15.43:0/1803411695 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 msgr2=0x55c5b481d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:09:09.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29b6800700 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b481d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29b6800700 1 -- 172.21.15.43:0/1803411695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b481de20 msgr2=0x55c5b482a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:09:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29b6800700 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b481de20 0x55c5b482a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29b6800700 1 -- 172.21.15.43:0/1803411695 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5b47043d0 con 0x55c5b478a9a0 2024-05-27T07:09:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.372+0000 7f29aea00700 1 -- 172.21.15.43:0/4009068294 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f29a40e2610 con 0x7f29b8077780 2024-05-27T07:09:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.373+0000 7f29b5e00700 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b481d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:09:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.373+0000 7f29b5400700 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b481de20 0x55c5b482a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:09:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.373+0000 7f29b6800700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f299c0b2900 0x7f299c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.373+0000 7f29b6800700 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b481d2e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f29b00084c0 tx=0x7f29b00087d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50389c5e2899039 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.373+0000 7f29a3400700 1 -- 172.21.15.43:0/1803411695 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0016a20 con 0x55c5b478a9a0 2024-05-27T07:09:09.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.374+0000 7f29a3400700 1 -- 172.21.15.43:0/1803411695 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29b00109d0 con 0x55c5b478a9a0 2024-05-27T07:09:09.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.374+0000 7f29a3400700 1 -- 172.21.15.43:0/1803411695 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0020780 con 0x55c5b478a9a0 2024-05-27T07:09:09.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.375+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/1803411695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 msgr2=0x55c5b481d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:09.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.375+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b481d2e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f29b00084c0 tx=0x7f29b00087d0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.375+0000 7f29b6800700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f299c0b2900 0x7f299c0b4e10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f29b001d000 tx=0x7f29b001a000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:09.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.375+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/1803411695 shutdown_connections 2024-05-27T07:09:09.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.375+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b481de20 0x55c5b482a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.375+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b481d2e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.375+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/1803411695 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b481d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.375+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/1803411695 >> 172.21.15.43:0/1803411695 conn(0x55c5b4738340 msgr2=0x55c5b4819470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:09.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.375+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/1803411695 shutdown_connections 2024-05-27T07:09:09.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.375+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/1803411695 wait complete. 2024-05-27T07:09:09.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.376+0000 7f29cb7a4b80 1 -- start start 2024-05-27T07:09:09.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.376+0000 7f29cb7a4b80 1 Processor -- start 2024-05-27T07:09:09.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.376+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b469edf0 0x55c5b4827600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.376+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b4827b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.376+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b481de20 0x55c5b4828080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.376+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c5b4641f50 con 0x55c5b478a9a0 2024-05-27T07:09:09.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.376+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c5b46dc710 con 0x55c5b469edf0 2024-05-27T07:09:09.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.376+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c5b482ab30 con 0x55c5b481de20 2024-05-27T07:09:09.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29cb7a4b80 10 client.0 _ll_register_callbacks cb 0x55c5b46e5c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:09:09.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b4827b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b4827b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60634/0 (socket says 172.21.15.43:60634) 2024-05-27T07:09:09.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b5400700 1 -- 172.21.15.43:0/3868708751 learned_addr learned my addr 172.21.15.43:0/3868708751 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:09.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b5e00700 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b469edf0 0x55c5b4827600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b6800700 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b481de20 0x55c5b4828080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b5400700 1 -- 172.21.15.43:0/3868708751 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b481de20 msgr2=0x55c5b4828080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:09:09.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b5400700 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b481de20 0x55c5b4828080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b5400700 1 -- 172.21.15.43:0/3868708751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b469edf0 msgr2=0x55c5b4827600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:09:09.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b5400700 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b469edf0 0x55c5b4827600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b5400700 1 -- 172.21.15.43:0/3868708751 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5b468b880 con 0x55c5b478a9a0 2024-05-27T07:09:09.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b6800700 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b481de20 0x55c5b4828080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:09:09.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.377+0000 7f29b5e00700 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b469edf0 0x55c5b4827600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:09:09.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.378+0000 7f29b5400700 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b4827b40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29a40dfd50 tx=0x7f29a40a9770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5dd89f82fdb029 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:09.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.378+0000 7f2997e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:09:09.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.378+0000 7f2997e00700 1 -- 172.21.15.43:0/3868708751 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a40a9380 con 0x55c5b478a9a0 2024-05-27T07:09:09.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.378+0000 7f2997e00700 1 -- 172.21.15.43:0/3868708751 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f29a40e2340 con 0x55c5b478a9a0 2024-05-27T07:09:09.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.378+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/3868708751 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29b00109d0 con 0x55c5b478a9a0 2024-05-27T07:09:09.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.378+0000 7f2997e00700 1 -- 172.21.15.43:0/3868708751 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f29a40a9f70 con 0x55c5b478a9a0 2024-05-27T07:09:09.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.378+0000 7f29cb7a4b80 10 client.14721 fetch_fsmap learned FSMap version 17 2024-05-27T07:09:09.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.378+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/3868708751 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55c5b470c010 con 0x55c5b478a9a0 2024-05-27T07:09:09.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.379+0000 7f2997e00700 1 -- 172.21.15.43:0/3868708751 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29b00109d0 con 0x55c5b478a9a0 2024-05-27T07:09:09.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.379+0000 7f2997e00700 1 -- 172.21.15.43:0/3868708751 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f29a40b2240 con 0x55c5b478a9a0 2024-05-27T07:09:09.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.379+0000 7f29cb7a4b80 10 client.14721 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T07:09:09.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.379+0000 7f29cb7a4b80 10 client.14721 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:09:09.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.379+0000 7f29cb7a4b80 20 client.14721 populate_metadata read hostname 'smithi043' 2024-05-27T07:09:09.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.379+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c5b48379b0 0x55c5b4838e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.379+0000 7f29cb7a4b80 4 client.14721 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:09:09.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.379+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/3868708751 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5b474dd40 con 0x55c5b48379b0 2024-05-27T07:09:09.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.380+0000 7f29b5e00700 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c5b48379b0 0x55c5b4838e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.381+0000 7f29b5e00700 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c5b48379b0 0x55c5b4838e20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0faf6bc14d04912 server_cookie=9b1b962df3d55df1 in_seq=0 out_seq=0 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.381+0000 7f2997e00700 10 client.14721 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.383+0000 7f2997e00700 1 -- 172.21.15.43:0/3868708751 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c5b474dd40 con 0x55c5b48379b0 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.383+0000 7f2997e00700 10 client.14721 handle_command_reply: tid=0 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 client.14721 shutdown 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/3868708751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 msgr2=0x55c5b4827b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b4827b40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29a40dfd50 tx=0x7f29a40a9770 comp rx=0 tx=0).stop 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/3868708751 shutdown_connections 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b481de20 0x55c5b4828080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b469edf0 0x55c5b4827600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c5b48379b0 0x55c5b4838e20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/3868708751 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b478a9a0 0x55c5b4827b40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/3868708751 >> 172.21.15.43:0/3868708751 conn(0x55c5b4738340 msgr2=0x55c5b481d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:09.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f2997e00700 0 client.14721 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:09:09.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/3868708751 shutdown_connections 2024-05-27T07:09:09.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.384+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/3868708751 wait complete. 2024-05-27T07:09:09.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.385+0000 7f29cb7a4b80 20 client.14721 trim_cache size 0 max 16384 2024-05-27T07:09:09.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.408+0000 7f29cb7a4b80 1 Processor -- start 2024-05-27T07:09:09.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.409+0000 7f29cb7a4b80 1 -- start start 2024-05-27T07:09:09.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.409+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b48372e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.409+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b478a9a0 0x55c5b4819610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.409+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4833e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.409+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c5b4837160 con 0x55c5b481de20 2024-05-27T07:09:09.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.409+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c5b4837830 con 0x55c5b478a9a0 2024-05-27T07:09:09.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.409+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c5b4837f00 con 0x55c5b469edf0 2024-05-27T07:09:09.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4833e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4833e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60640/0 (socket says 172.21.15.43:60640) 2024-05-27T07:09:09.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b6800700 1 -- 172.21.15.43:0/549064943 learned_addr learned my addr 172.21.15.43:0/549064943 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:09.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b5e00700 1 --2- 172.21.15.43:0/549064943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b48372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b5400700 1 --2- 172.21.15.43:0/549064943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b478a9a0 0x55c5b4819610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b6800700 1 -- 172.21.15.43:0/549064943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 msgr2=0x55c5b48372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:09:09.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b6800700 1 --2- 172.21.15.43:0/549064943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b48372e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b6800700 1 -- 172.21.15.43:0/549064943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b478a9a0 msgr2=0x55c5b4819610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:09:09.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b6800700 1 --2- 172.21.15.43:0/549064943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b478a9a0 0x55c5b4819610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b6800700 1 -- 172.21.15.43:0/549064943 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5b482a5e0 con 0x55c5b481de20 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b5e00700 1 --2- 172.21.15.43:0/549064943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b48372e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.410+0000 7f29b6800700 1 --2- 172.21.15.43:0/549064943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4833e30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f29b0008e10 tx=0x7f29b0018040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb80b2f7aecbc346 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.411+0000 7f29a2a00700 1 -- 172.21.15.43:0/549064943 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0032070 con 0x55c5b481de20 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.411+0000 7f29a2a00700 1 -- 172.21.15.43:0/549064943 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29b0020d20 con 0x55c5b481de20 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.411+0000 7f29a2a00700 1 -- 172.21.15.43:0/549064943 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b00107a0 con 0x55c5b481de20 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.412+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/549064943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 msgr2=0x55c5b4833e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.412+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/549064943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4833e30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f29b0008e10 tx=0x7f29b0018040 comp rx=0 tx=0).stop 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.412+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/549064943 shutdown_connections 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.412+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/549064943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4833e30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.412+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/549064943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b48372e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.412+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/549064943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b478a9a0 0x55c5b4819610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.412+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/549064943 >> 172.21.15.43:0/549064943 conn(0x55c5b4738340 msgr2=0x55c5b4818f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.412+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/549064943 shutdown_connections 2024-05-27T07:09:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.412+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/549064943 wait complete. 2024-05-27T07:09:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.413+0000 7f29cb7a4b80 1 -- start start 2024-05-27T07:09:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.413+0000 7f29cb7a4b80 1 Processor -- start 2024-05-27T07:09:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.413+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b4842810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.413+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b478a9a0 0x55c5b48218d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29cb7a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4821e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c5b4838ca0 con 0x55c5b481de20 2024-05-27T07:09:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c5b48385d0 con 0x55c5b478a9a0 2024-05-27T07:09:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29cb7a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c5b4839370 con 0x55c5b469edf0 2024-05-27T07:09:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29cb7a4b80 10 client.0 _ll_register_callbacks cb 0x55c5b46e5c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:09:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29b6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4821e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29b6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4821e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60644/0 (socket says 172.21.15.43:60644) 2024-05-27T07:09:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29b6800700 1 -- 172.21.15.43:0/823529805 learned_addr learned my addr 172.21.15.43:0/823529805 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29b5e00700 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b4842810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29b5400700 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b478a9a0 0x55c5b48218d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29b6800700 1 -- 172.21.15.43:0/823529805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 msgr2=0x55c5b4842810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:09:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29b6800700 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b4842810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29b6800700 1 -- 172.21.15.43:0/823529805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b478a9a0 msgr2=0x55c5b48218d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:09:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29b6800700 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b478a9a0 0x55c5b48218d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.414+0000 7f29b6800700 1 -- 172.21.15.43:0/823529805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5b48283f0 con 0x55c5b481de20 2024-05-27T07:09:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.415+0000 7f29b6800700 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4821e10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f29b0008c50 tx=0x7f29b0020780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2901936ba8e54cef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.415+0000 7f2997e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:09:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.415+0000 7f2997e00700 1 -- 172.21.15.43:0/823529805 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0032040 con 0x55c5b481de20 2024-05-27T07:09:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.415+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/823529805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29b0020d20 con 0x55c5b481de20 2024-05-27T07:09:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.415+0000 7f2997e00700 1 -- 172.21.15.43:0/823529805 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f29b0010540 con 0x55c5b481de20 2024-05-27T07:09:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.415+0000 7f2997e00700 1 -- 172.21.15.43:0/823529805 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f29b0046050 con 0x55c5b481de20 2024-05-27T07:09:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.415+0000 7f29cb7a4b80 10 client.14727 fetch_fsmap learned FSMap version 17 2024-05-27T07:09:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.415+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/823529805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55c5b483ee30 con 0x55c5b481de20 2024-05-27T07:09:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.416+0000 7f2997e00700 1 -- 172.21.15.43:0/823529805 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29b0020d20 con 0x55c5b481de20 2024-05-27T07:09:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.416+0000 7f2997e00700 1 -- 172.21.15.43:0/823529805 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f29b0013070 con 0x55c5b481de20 2024-05-27T07:09:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.416+0000 7f29cb7a4b80 10 client.14727 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T07:09:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.416+0000 7f29cb7a4b80 10 client.14727 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:09:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.416+0000 7f29cb7a4b80 20 client.14727 populate_metadata read hostname 'smithi043' 2024-05-27T07:09:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.416+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c5b48a49a0 0x55c5b48a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.416+0000 7f29cb7a4b80 4 client.14727 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:09:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.416+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/823529805 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "status"}) -- 0x55c5b4747520 con 0x55c5b48a49a0 2024-05-27T07:09:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.417+0000 7f29b5e00700 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c5b48a49a0 0x55c5b48a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.418+0000 7f29b5e00700 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c5b48a49a0 0x55c5b48a5e10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b7e57987959af9e server_cookie=f47c0e552735707c in_seq=0 out_seq=0 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.418+0000 7f2997e00700 10 client.14727 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.418+0000 7f2997e00700 1 -- 172.21.15.43:0/823529805 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c5b4747520 con 0x55c5b48a49a0 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.418+0000 7f2997e00700 10 client.14727 handle_command_reply: tid=0 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 client.14727 shutdown 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/823529805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 msgr2=0x55c5b4821e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4821e10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f29b0008c50 tx=0x7f29b0020780 comp rx=0 tx=0).stop 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/823529805 shutdown_connections 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c5b48a49a0 0x55c5b48a5e10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5b481de20 0x55c5b4821e10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c5b469edf0 0x55c5b4842810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 --2- 172.21.15.43:0/823529805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5b478a9a0 0x55c5b48218d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/823529805 >> 172.21.15.43:0/823529805 conn(0x55c5b4738340 msgr2=0x55c5b48273e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f2997e00700 0 client.14727 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:09:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/823529805 shutdown_connections 2024-05-27T07:09:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 1 -- 172.21.15.43:0/823529805 wait complete. 2024-05-27T07:09:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.419+0000 7f29cb7a4b80 20 client.14727 trim_cache size 0 max 16384 2024-05-27T07:09:09.415 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:09:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 34484, 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 43.333128070000001, 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 17, 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 88, 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 86, 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 67.920219336000002 2024-05-27T07:09:09.416 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:09:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 -- 172.21.15.43:0/4009068294 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f299c0b2900 msgr2=0x7f299c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f299c0b2900 0x7f299c0b4e10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f29b001d000 tx=0x7f29b001a000 comp rx=0 tx=0).stop 2024-05-27T07:09:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 -- 172.21.15.43:0/4009068294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b8077780 msgr2=0x7f29b819d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b8077780 0x7f29b819d4a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f29a400c9b0 tx=0x7f29a400cd70 comp rx=0 tx=0).stop 2024-05-27T07:09:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 -- 172.21.15.43:0/4009068294 shutdown_connections 2024-05-27T07:09:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f299c0b2900 0x7f299c0b4e10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29b8110b70 0x7f29b81a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29b8076e30 0x7f29b819cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 --2- 172.21.15.43:0/4009068294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29b8077780 0x7f29b819d4a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 -- 172.21.15.43:0/4009068294 >> 172.21.15.43:0/4009068294 conn(0x7f29b80ffe90 msgr2=0x7f29b8100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 -- 172.21.15.43:0/4009068294 shutdown_connections 2024-05-27T07:09:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:09.420+0000 7f29bcc00700 1 -- 172.21.15.43:0/4009068294 wait complete. 2024-05-27T07:09:10.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:10 smithi043 ceph-mon[31531]: pgmap v416: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-05-27T07:09:10.263 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T07:09:10.264 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:09:10.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-05-27T07:09:10.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:10 smithi132 ceph-mon[35283]: pgmap v416: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-05-27T07:09:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:10 smithi179 ceph-mon[35888]: pgmap v416: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 0 objects/s recovering 2024-05-27T07:09:10.723 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:12 smithi132 ceph-mon[35283]: pgmap v417: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:12.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:12 smithi043 ceph-mon[31531]: pgmap v417: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:12 smithi179 ceph-mon[35888]: pgmap v417: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:12.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.969+0000 7fe51a600700 1 -- 172.21.15.43:0/3916232396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe51410c5a0 msgr2=0x7fe51410c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:12.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.969+0000 7fe51a600700 1 --2- 172.21.15.43:0/3916232396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe51410c5a0 0x7fe51410c9b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe504009990 tx=0x7fe504009ca0 comp rx=0 tx=0).stop 2024-05-27T07:09:12.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.969+0000 7fe51a600700 1 -- 172.21.15.43:0/3916232396 shutdown_connections 2024-05-27T07:09:12.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.969+0000 7fe51a600700 1 --2- 172.21.15.43:0/3916232396 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514106eb0 0x7fe514113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:12.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.969+0000 7fe51a600700 1 --2- 172.21.15.43:0/3916232396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe51410c5a0 0x7fe51410c9b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:12.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.969+0000 7fe51a600700 1 --2- 172.21.15.43:0/3916232396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe514106560 0x7fe514106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:12.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.969+0000 7fe51a600700 1 -- 172.21.15.43:0/3916232396 >> 172.21.15.43:0/3916232396 conn(0x7fe514102040 msgr2=0x7fe514104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:12.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.969+0000 7fe51a600700 1 -- 172.21.15.43:0/3916232396 shutdown_connections 2024-05-27T07:09:12.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.969+0000 7fe51a600700 1 -- 172.21.15.43:0/3916232396 wait complete. 2024-05-27T07:09:12.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.970+0000 7fe51a600700 1 Processor -- start 2024-05-27T07:09:12.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.971+0000 7fe51a600700 1 -- start start 2024-05-27T07:09:12.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.971+0000 7fe51a600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe514106560 0x7fe51419d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.971+0000 7fe51a600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514106eb0 0x7fe51419d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.971+0000 7fe51a600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe51410c5a0 0x7fe5141a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.971+0000 7fe51a600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe514116630 con 0x7fe514106560 2024-05-27T07:09:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.971+0000 7fe51a600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe5141164b0 con 0x7fe51410c5a0 2024-05-27T07:09:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.971+0000 7fe51a600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe5141167b0 con 0x7fe514106eb0 2024-05-27T07:09:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe513400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe514106560 0x7fe51419d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe513400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe514106560 0x7fe51419d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60664/0 (socket says 172.21.15.43:60664) 2024-05-27T07:09:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe513400700 1 -- 172.21.15.43:0/222030462 learned_addr learned my addr 172.21.15.43:0/222030462 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe512a00700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514106eb0 0x7fe51419d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe513e00700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe51410c5a0 0x7fe5141a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe513400700 1 -- 172.21.15.43:0/222030462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514106eb0 msgr2=0x7fe51419d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe513400700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514106eb0 0x7fe51419d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe513400700 1 -- 172.21.15.43:0/222030462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe51410c5a0 msgr2=0x7fe5141a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe513400700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe51410c5a0 0x7fe5141a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe513400700 1 -- 172.21.15.43:0/222030462 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5141a1ff0 con 0x7fe514106560 2024-05-27T07:09:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.972+0000 7fe513400700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe514106560 0x7fe51419d120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe5040098e0 tx=0x7fe504005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.973+0000 7fe50be00700 1 -- 172.21.15.43:0/222030462 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe504004830 con 0x7fe514106560 2024-05-27T07:09:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.973+0000 7fe50be00700 1 -- 172.21.15.43:0/222030462 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe504005020 con 0x7fe514106560 2024-05-27T07:09:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.973+0000 7fe50be00700 1 -- 172.21.15.43:0/222030462 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50400fb20 con 0x7fe514106560 2024-05-27T07:09:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.973+0000 7fe51a600700 1 -- 172.21.15.43:0/222030462 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5141a2280 con 0x7fe514106560 2024-05-27T07:09:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.973+0000 7fe51a600700 1 -- 172.21.15.43:0/222030462 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5141a2750 con 0x7fe514106560 2024-05-27T07:09:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.975+0000 7fe51a600700 1 -- 172.21.15.43:0/222030462 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe51410e960 con 0x7fe514106560 2024-05-27T07:09:12.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.977+0000 7fe50be00700 1 -- 172.21.15.43:0/222030462 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe50400fcc0 con 0x7fe514106560 2024-05-27T07:09:12.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.978+0000 7fe50be00700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4f80b2900 0x7fe4f80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.979+0000 7fe50be00700 1 -- 172.21.15.43:0/222030462 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fe5040e3470 con 0x7fe514106560 2024-05-27T07:09:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.979+0000 7fe512a00700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4f80b2900 0x7fe4f80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.980+0000 7fe512a00700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4f80b2900 0x7fe4f80b4e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe500006fd0 tx=0x7fe500008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:12.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:12.984+0000 7fe50be00700 1 -- 172.21.15.43:0/222030462 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe504013070 con 0x7fe514106560 2024-05-27T07:09:13.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.275+0000 7fe51a600700 1 -- 172.21.15.43:0/222030462 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe5140512b0 con 0x7fe514106560 2024-05-27T07:09:13.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.276+0000 7fe50be00700 1 -- 172.21.15.43:0/222030462 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fe5040aae60 con 0x7fe514106560 2024-05-27T07:09:13.267 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-27T07:09:13.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.279+0000 7fe51a600700 1 -- 172.21.15.43:0/222030462 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4f80b2900 msgr2=0x7fe4f80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:13.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.279+0000 7fe51a600700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4f80b2900 0x7fe4f80b4e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe500006fd0 tx=0x7fe500008040 comp rx=0 tx=0).stop 2024-05-27T07:09:13.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.280+0000 7fe51a600700 1 -- 172.21.15.43:0/222030462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe514106560 msgr2=0x7fe51419d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:13.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.280+0000 7fe51a600700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe514106560 0x7fe51419d120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe5040098e0 tx=0x7fe504005e90 comp rx=0 tx=0).stop 2024-05-27T07:09:13.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.280+0000 7fe51a600700 1 -- 172.21.15.43:0/222030462 shutdown_connections 2024-05-27T07:09:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.280+0000 7fe51a600700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4f80b2900 0x7fe4f80b4e10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.280+0000 7fe51a600700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe51410c5a0 0x7fe5141a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.280+0000 7fe51a600700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe514106560 0x7fe51419d120 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.280+0000 7fe51a600700 1 --2- 172.21.15.43:0/222030462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514106eb0 0x7fe51419d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.280+0000 7fe51a600700 1 -- 172.21.15.43:0/222030462 >> 172.21.15.43:0/222030462 conn(0x7fe514102040 msgr2=0x7fe514103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.280+0000 7fe51a600700 1 -- 172.21.15.43:0/222030462 shutdown_connections 2024-05-27T07:09:13.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:13.280+0000 7fe51a600700 1 -- 172.21.15.43:0/222030462 wait complete. 2024-05-27T07:09:14.111 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-27T07:09:14.112 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T07:09:14.112 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-05-27T07:09:14.112 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-27T07:09:14.112 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:09:14.112 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-27T07:09:14.112 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T07:09:14.112 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T07:09:14.112 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-27T07:09:14.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:14.112 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:09:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 41847sec preferred_lft 41847sec 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:09:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-05-27T07:09:14.175 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T07:09:14.175 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:09:14.175 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-05-27T07:09:14.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add name ceph-brx type bridge 2024-05-27T07:09:14.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip addr flush dev ceph-brx 2024-05-27T07:09:14.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set ceph-brx up 2024-05-27T07:09:14.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T07:09:14.175 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-05-27T07:09:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:09:14.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:14 smithi043 ceph-mon[31531]: pgmap v418: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:14.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:14 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/222030462' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:09:14.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:14 smithi132 ceph-mon[35283]: pgmap v418: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:14.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:14 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/222030462' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:09:14.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:14 smithi179 ceph-mon[35888]: pgmap v418: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:14.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:14 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/222030462' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:09:14.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:09:14.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:14.672 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T07:09:14.742 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-05-27T07:09:14.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:14.747 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-05-27T07:09:14.802 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-05-27T07:09:14.802 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T07:09:14.803 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-27T07:09:14.803 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-27T07:09:14.803 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T07:09:14.803 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:09:14.803 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-05-27T07:09:14.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-27T07:09:14.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-27T07:09:14.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-27T07:09:14.804 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-05-27T07:09:14.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:09:15.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:09:15.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:15.590 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-05-27T07:09:15.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:15.639 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list-id 2024-05-27T07:09:15.686 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:09:15.686 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-05-27T07:09:15.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:09:15.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-27T07:09:15.686 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-05-27T07:09:15.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:09:15.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:09:15.862 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-27T07:09:15.863 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:09:15.863 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-05-27T07:09:15.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-27T07:09:15.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T07:09:15.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-27T07:09:15.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-27T07:09:15.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-27T07:09:15.864 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-05-27T07:09:15.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:09:16.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:16 smithi043 ceph-mon[31531]: pgmap v419: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:16.101 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:16 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:09:16.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:09:16.158 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:09:16.158 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-05-27T07:09:16.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-05-27T07:09:16.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T07:09:16.158 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-05-27T07:09:16.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:09:16.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:16 smithi132 ceph-mon[35283]: pgmap v419: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:16.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:16 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:09:16.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:09:16.366 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-27T07:09:16.367 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:09:16.367 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:09:16.409 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-27T07:09:16.409 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:09:16.409 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:09:16.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:16 smithi179 ceph-mon[35888]: pgmap v419: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:16.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:16 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:09:16.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-05-27T07:09:16.614 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:09:16.662 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-05-27T07:09:16.662 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-05-27T07:09:16.662 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-05-27T07:09:16.662 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-05-27T07:09:16.662 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-05-27T07:09:16.662 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-05-27T07:09:16.662 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-05-27T07:09:16.662 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/bpf 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:09:16.663 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fb962e0896f306e6bdf309817399bbb3496e7c31069501c545db8fe3e7bba85d/merged 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/6741f2331dca4059307dfc7f032fc0462f11aef53f9941537451b7c8713b2447/merged 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/0 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d94143aab52c0cf4e74c4ce713e61b04a988bdf6896e24080ad3b89fe8435b77/merged 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fe62c8ab8b1aeaa91d01ab9fe0bf51ede8e7f0357669aaf589eab6a2ce1bac66/merged 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/202cde062bc89da2b696fbfe1f7746610c02b5423d4469ab514a3cbb53acd07f/merged 2024-05-27T07:09:16.664 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/df5388b1b92ef39ed24085545b944bc3ed739d064b74cb685878895bf4818382/merged 2024-05-27T07:09:16.665 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/5fbcb271e6c21124319502ad3025fc6461d8a6854221c4825ab7f11c56ec4a21/merged 2024-05-27T07:09:16.665 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/484787d0f6d97e5d65195de880768fe8e92235f422c129fa06f3d45a43eee6fb/merged 2024-05-27T07:09:16.665 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/1b86423c0412d7d490dab40bc424d2df4478d6849737c098ddd5feb887413f9b/merged 2024-05-27T07:09:16.665 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d011ecf4d242761a3d35b1dc879cbeda1e6484d35d2d8ca10cd410c5f937d874/merged 2024-05-27T07:09:16.665 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-05-27T07:09:16.665 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:09:16.665 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:09:16.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:16.666 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-05-27T07:09:16.710 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-27T07:09:16.711 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-27T07:09:16.756 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-05-27T07:09:16.823 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:09:16.867 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-05-27T07:09:16.867 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-05-27T07:09:16.867 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-05-27T07:09:16.867 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/bpf 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-05-27T07:09:16.868 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fb962e0896f306e6bdf309817399bbb3496e7c31069501c545db8fe3e7bba85d/merged 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/6741f2331dca4059307dfc7f032fc0462f11aef53f9941537451b7c8713b2447/merged 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/0 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:09:16.869 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d94143aab52c0cf4e74c4ce713e61b04a988bdf6896e24080ad3b89fe8435b77/merged 2024-05-27T07:09:16.870 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fe62c8ab8b1aeaa91d01ab9fe0bf51ede8e7f0357669aaf589eab6a2ce1bac66/merged 2024-05-27T07:09:16.870 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/202cde062bc89da2b696fbfe1f7746610c02b5423d4469ab514a3cbb53acd07f/merged 2024-05-27T07:09:16.870 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/df5388b1b92ef39ed24085545b944bc3ed739d064b74cb685878895bf4818382/merged 2024-05-27T07:09:16.870 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/5fbcb271e6c21124319502ad3025fc6461d8a6854221c4825ab7f11c56ec4a21/merged 2024-05-27T07:09:16.870 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/484787d0f6d97e5d65195de880768fe8e92235f422c129fa06f3d45a43eee6fb/merged 2024-05-27T07:09:16.870 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/1b86423c0412d7d490dab40bc424d2df4478d6849737c098ddd5feb887413f9b/merged 2024-05-27T07:09:16.870 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d011ecf4d242761a3d35b1dc879cbeda1e6484d35d2d8ca10cd410c5f937d874/merged 2024-05-27T07:09:16.870 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-05-27T07:09:16.870 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:09:16.870 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:09:16.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:16.871 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-05-27T07:09:17.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi043.stderr:2024-05-27T07:09:17.063+0000 7f7093b21480 -1 init, newargv = 0x7f7074004ba0 newargc=15 2024-05-27T07:09:17.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi043.stderr:ceph-fuse[75082]: starting ceph client 2024-05-27T07:09:17.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi043.stderr:ceph-fuse[75082]: starting fuse 2024-05-27T07:09:17.576 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.576 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.576 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.576 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:17 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.576 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:17 smithi132 ceph-mon[35283]: pgmap v420: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:17.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:17 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:17 smithi179 ceph-mon[35888]: pgmap v420: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:17.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:17 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:17.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:17 smithi043 ceph-mon[31531]: pgmap v420: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:18.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-05-27T07:09:18.139 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:09:18.182 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-05-27T07:09:18.182 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-05-27T07:09:18.182 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-05-27T07:09:18.182 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-05-27T07:09:18.182 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-05-27T07:09:18.182 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/bpf 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-05-27T07:09:18.183 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fb962e0896f306e6bdf309817399bbb3496e7c31069501c545db8fe3e7bba85d/merged 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/6741f2331dca4059307dfc7f032fc0462f11aef53f9941537451b7c8713b2447/merged 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/0 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d94143aab52c0cf4e74c4ce713e61b04a988bdf6896e24080ad3b89fe8435b77/merged 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fe62c8ab8b1aeaa91d01ab9fe0bf51ede8e7f0357669aaf589eab6a2ce1bac66/merged 2024-05-27T07:09:18.184 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/202cde062bc89da2b696fbfe1f7746610c02b5423d4469ab514a3cbb53acd07f/merged 2024-05-27T07:09:18.185 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/df5388b1b92ef39ed24085545b944bc3ed739d064b74cb685878895bf4818382/merged 2024-05-27T07:09:18.185 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/5fbcb271e6c21124319502ad3025fc6461d8a6854221c4825ab7f11c56ec4a21/merged 2024-05-27T07:09:18.185 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/484787d0f6d97e5d65195de880768fe8e92235f422c129fa06f3d45a43eee6fb/merged 2024-05-27T07:09:18.185 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/1b86423c0412d7d490dab40bc424d2df4478d6849737c098ddd5feb887413f9b/merged 2024-05-27T07:09:18.185 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d011ecf4d242761a3d35b1dc879cbeda1e6484d35d2d8ca10cd410c5f937d874/merged 2024-05-27T07:09:18.185 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-05-27T07:09:18.185 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:09:18.185 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:09:18.185 INFO:teuthology.orchestra.run.smithi043.stdout:/home/ubuntu/cephtest/mnt.admin 2024-05-27T07:09:18.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:18.186 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-05-27T07:09:18.273 INFO:teuthology.orchestra.run.smithi043.stdout:119 2024-05-27T07:09:18.273 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-05-27T07:09:18.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-05-27T07:09:18.273 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-05-27T07:09:18.273 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-05-27T07:09:18.273 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-05-27T07:09:18.273 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-27T07:09:18.274 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-27T07:09:18.275 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-27T07:09:18.275 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-05-27T07:09:18.275 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-05-27T07:09:18.275 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-05-27T07:09:18.275 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-27T07:09:18.276 DEBUG:teuthology.orchestra.run.smithi043:> 2024-05-27T07:09:18.276 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.admin")) 2024-05-27T07:09:18.276 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-05-27T07:09:18.470 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.admin.75082.asok 2024-05-27T07:09:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:09:18.487 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75082.asok 2024-05-27T07:09:18.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:18.487 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75082.asok status 2024-05-27T07:09:18.778 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:09:18.779 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-05-27T07:09:18.779 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "38f9d900c83c63799fbdbe61acc9a11b0d3554a6", 2024-05-27T07:09:18.779 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:09:18.779 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "admin", 2024-05-27T07:09:18.779 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-05-27T07:09:18.779 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-27T07:09:18.779 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "75082", 2024-05-27T07:09:18.779 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-05-27T07:09:18.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:09:18.779 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 14739, 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 14739 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 462402102 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:09:18.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "192.168.0.1:0", 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 462402102 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.14739 192.168.0.1:0/462402102", 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "192.168.0.1:0/462402102", 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 17, 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 88, 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-05-27T07:09:18.781 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-05-27T07:09:18.782 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-05-27T07:09:18.782 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:09:18.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:18.802 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-05-27T07:09:18.802 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:09:18.856 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-05-27T07:09:18.856 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:09:18.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:18.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:09:18.939 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-27T07:09:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T07:09:19.083 INFO:teuthology.orchestra.run.smithi043.stdout:# file: . 2024-05-27T07:09:19.083 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-27T07:09:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:09:19.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:09:19.097 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T07:09:19.098 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:09:19.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:09:19.632 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:19.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:19 smithi043 ceph-mon[31531]: pgmap v421: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:19.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:19 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:19 smithi179 ceph-mon[35888]: pgmap v421: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:19.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:19 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:19 smithi132 ceph-mon[35283]: pgmap v421: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:19.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:19.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:19 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:20.823 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:20.824 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:20.824 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:20 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:20.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:20.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:20.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:20 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:21.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:09:21.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:09:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:20 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:09:21.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.829+0000 7ff1ab200700 1 -- 172.21.15.43:0/2201060984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a4107c00 msgr2=0x7ff1a4108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:21.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.829+0000 7ff1ab200700 1 --2- 172.21.15.43:0/2201060984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a4107c00 0x7ff1a4108010 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff1940099c0 tx=0x7ff194009cd0 comp rx=0 tx=0).stop 2024-05-27T07:09:21.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.829+0000 7ff1ab200700 1 -- 172.21.15.43:0/2201060984 shutdown_connections 2024-05-27T07:09:21.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.829+0000 7ff1ab200700 1 --2- 172.21.15.43:0/2201060984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4108550 0x7ff1a410feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:21.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.829+0000 7ff1ab200700 1 --2- 172.21.15.43:0/2201060984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a41032d0 0x7ff1a41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:21.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.829+0000 7ff1ab200700 1 --2- 172.21.15.43:0/2201060984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a4107c00 0x7ff1a4108010 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:21.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.829+0000 7ff1ab200700 1 -- 172.21.15.43:0/2201060984 >> 172.21.15.43:0/2201060984 conn(0x7ff1a4078ab0 msgr2=0x7ff1a4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:21.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.829+0000 7ff1ab200700 1 -- 172.21.15.43:0/2201060984 shutdown_connections 2024-05-27T07:09:21.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.829+0000 7ff1ab200700 1 -- 172.21.15.43:0/2201060984 wait complete. 2024-05-27T07:09:21.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:21 smithi043 ceph-mon[31531]: pgmap v422: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:21.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.830+0000 7ff1ab200700 1 Processor -- start 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.830+0000 7ff1ab200700 1 -- start start 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.831+0000 7ff1ab200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a41032d0 0x7ff1a419cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.831+0000 7ff1ab200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a4107c00 0x7ff1a419d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.831+0000 7ff1ab200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4108550 0x7ff1a41a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.831+0000 7ff1ab200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a4113060 con 0x7ff1a41032d0 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.831+0000 7ff1ab200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a4112ee0 con 0x7ff1a4107c00 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.831+0000 7ff1ab200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a41131e0 con 0x7ff1a4108550 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.831+0000 7ff1a9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a41032d0 0x7ff1a419cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.831+0000 7ff1a9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a41032d0 0x7ff1a419cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36534/0 (socket says 172.21.15.43:36534) 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.831+0000 7ff1a9e00700 1 -- 172.21.15.43:0/1020111076 learned_addr learned my addr 172.21.15.43:0/1020111076 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.832+0000 7ff1a9400700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a4107c00 0x7ff1a419d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.832+0000 7ff1aa800700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4108550 0x7ff1a41a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.832+0000 7ff1a9e00700 1 -- 172.21.15.43:0/1020111076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4108550 msgr2=0x7ff1a41a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:21.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.832+0000 7ff1a9e00700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4108550 0x7ff1a41a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:21.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.832+0000 7ff1a9e00700 1 -- 172.21.15.43:0/1020111076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a4107c00 msgr2=0x7ff1a419d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:21.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.832+0000 7ff1a9e00700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a4107c00 0x7ff1a419d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:21.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.832+0000 7ff1a9e00700 1 -- 172.21.15.43:0/1020111076 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a41a1c00 con 0x7ff1a41032d0 2024-05-27T07:09:21.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.832+0000 7ff1a9e00700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a41032d0 0x7ff1a419cde0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff19800d950 tx=0x7ff19800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:21.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.833+0000 7ff19ea00700 1 -- 172.21.15.43:0/1020111076 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198016070 con 0x7ff1a41032d0 2024-05-27T07:09:21.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.833+0000 7ff19ea00700 1 -- 172.21.15.43:0/1020111076 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff198010460 con 0x7ff1a41032d0 2024-05-27T07:09:21.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.833+0000 7ff1ab200700 1 -- 172.21.15.43:0/1020111076 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a41a1ef0 con 0x7ff1a41032d0 2024-05-27T07:09:21.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.833+0000 7ff19ea00700 1 -- 172.21.15.43:0/1020111076 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19800f620 con 0x7ff1a41032d0 2024-05-27T07:09:21.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.833+0000 7ff1ab200700 1 -- 172.21.15.43:0/1020111076 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a41a2400 con 0x7ff1a41032d0 2024-05-27T07:09:21.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.834+0000 7ff1ab200700 1 -- 172.21.15.43:0/1020111076 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a40512b0 con 0x7ff1a41032d0 2024-05-27T07:09:21.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.838+0000 7ff19ea00700 1 -- 172.21.15.43:0/1020111076 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff198019070 con 0x7ff1a41032d0 2024-05-27T07:09:21.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.840+0000 7ff19ea00700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff18c0b2880 0x7ff18c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:21.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.840+0000 7ff19ea00700 1 -- 172.21.15.43:0/1020111076 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7ff1980e21c0 con 0x7ff1a41032d0 2024-05-27T07:09:21.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.840+0000 7ff1a9400700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff18c0b2880 0x7ff18c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:21.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.841+0000 7ff1a9400700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff18c0b2880 0x7ff18c0b4d90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff194009990 tx=0x7ff194005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:21.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:21.843+0000 7ff19ea00700 1 -- 172.21.15.43:0/1020111076 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff1980a9b20 con 0x7ff1a41032d0 2024-05-27T07:09:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:21 smithi179 ceph-mon[35888]: pgmap v422: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:22.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.096+0000 7ff1ab200700 1 -- 172.21.15.43:0/1020111076 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff1a402e5e0 con 0x7ff1a41032d0 2024-05-27T07:09:22.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.098+0000 7ff19ea00700 1 -- 172.21.15.43:0/1020111076 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+590 (secure 0 0 0) 0x7ff1980a9420 con 0x7ff1a41032d0 2024-05-27T07:09:22.090 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/1651567761 2024-05-28T06:57:52.506320+0000 2024-05-27T07:09:22.090 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/1651567761 2024-05-28T06:57:52.506320+0000 2024-05-27T07:09:22.090 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1332970913 2024-05-28T06:57:18.380825+0000 2024-05-27T07:09:22.090 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1752945868 2024-05-28T06:57:18.380825+0000 2024-05-27T07:09:22.090 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2426463283 2024-05-28T06:57:18.380825+0000 2024-05-27T07:09:22.090 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2081880966 2024-05-28T06:57:52.506320+0000 2024-05-27T07:09:22.090 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/3521718752 2024-05-28T06:57:18.380825+0000 2024-05-27T07:09:22.090 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/830904719 2024-05-28T06:57:52.506320+0000 2024-05-27T07:09:22.090 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/3521718752 2024-05-28T06:57:18.380825+0000 2024-05-27T07:09:22.091 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/338672925 2024-05-28T06:57:52.506320+0000 2024-05-27T07:09:22.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:21 smithi132 ceph-mon[35283]: pgmap v422: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:22.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 -- 172.21.15.43:0/1020111076 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff18c0b2880 msgr2=0x7ff18c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:22.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff18c0b2880 0x7ff18c0b4d90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff194009990 tx=0x7ff194005fb0 comp rx=0 tx=0).stop 2024-05-27T07:09:22.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 -- 172.21.15.43:0/1020111076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a41032d0 msgr2=0x7ff1a419cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:22.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a41032d0 0x7ff1a419cde0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff19800d950 tx=0x7ff19800dd10 comp rx=0 tx=0).stop 2024-05-27T07:09:22.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 -- 172.21.15.43:0/1020111076 shutdown_connections 2024-05-27T07:09:22.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff18c0b2880 0x7ff18c0b4d90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:22.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4108550 0x7ff1a41a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:22.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1a41032d0 0x7ff1a419cde0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:22.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 --2- 172.21.15.43:0/1020111076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a4107c00 0x7ff1a419d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:22.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 -- 172.21.15.43:0/1020111076 >> 172.21.15.43:0/1020111076 conn(0x7ff1a4078ab0 msgr2=0x7ff1a410eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:22.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 -- 172.21.15.43:0/1020111076 shutdown_connections 2024-05-27T07:09:22.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:22.101+0000 7ff1ab200700 1 -- 172.21.15.43:0/1020111076 wait complete. 2024-05-27T07:09:22.094 INFO:teuthology.orchestra.run.smithi043.stderr:listed 10 entries 2024-05-27T07:09:22.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:22 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1020111076' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:09:22.932 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T07:09:22.932 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:09:22.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:22 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1020111076' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:09:22.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:09:23.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:22 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1020111076' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:09:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:23.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:23 smithi043 ceph-mon[31531]: pgmap v423: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:23 smithi179 ceph-mon[35888]: pgmap v423: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:24.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:23 smithi132 ceph-mon[35283]: pgmap v423: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:25.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.796+0000 7f5a68a00700 1 -- 172.21.15.43:0/2645376521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a64115360 msgr2=0x7f5a64117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:25.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.796+0000 7f5a68a00700 1 --2- 172.21.15.43:0/2645376521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a64115360 0x7f5a64117780 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00b0a0 tx=0x7f5a5c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:09:25.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.797+0000 7f5a68a00700 1 -- 172.21.15.43:0/2645376521 shutdown_connections 2024-05-27T07:09:25.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.797+0000 7f5a68a00700 1 --2- 172.21.15.43:0/2645376521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a64115360 0x7f5a64117780 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:25.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.797+0000 7f5a68a00700 1 --2- 172.21.15.43:0/2645376521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a64075b50 0x7f5a64075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:25.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.797+0000 7f5a68a00700 1 --2- 172.21.15.43:0/2645376521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a640764a0 0x7f5a64114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:25.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.797+0000 7f5a68a00700 1 -- 172.21.15.43:0/2645376521 >> 172.21.15.43:0/2645376521 conn(0x7f5a64100030 msgr2=0x7f5a64102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:25.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.797+0000 7f5a68a00700 1 -- 172.21.15.43:0/2645376521 shutdown_connections 2024-05-27T07:09:25.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.797+0000 7f5a68a00700 1 -- 172.21.15.43:0/2645376521 wait complete. 2024-05-27T07:09:25.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.798+0000 7f5a68a00700 1 Processor -- start 2024-05-27T07:09:25.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.799+0000 7f5a68a00700 1 -- start start 2024-05-27T07:09:25.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.799+0000 7f5a68a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a64075b50 0x7f5a641a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.799+0000 7f5a68a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a640764a0 0x7f5a641a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.799+0000 7f5a68a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a64115360 0x7f5a641a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.799+0000 7f5a68a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6411a910 con 0x7f5a64075b50 2024-05-27T07:09:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.799+0000 7f5a68a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6411a790 con 0x7f5a64115360 2024-05-27T07:09:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.799+0000 7f5a68a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6411aa90 con 0x7f5a640764a0 2024-05-27T07:09:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.799+0000 7f5a61e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a64075b50 0x7f5a641a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.800+0000 7f5a61e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a64075b50 0x7f5a641a12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47224/0 (socket says 172.21.15.43:47224) 2024-05-27T07:09:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.800+0000 7f5a61e00700 1 -- 172.21.15.43:0/355886454 learned_addr learned my addr 172.21.15.43:0/355886454 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.800+0000 7f5a5be00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a640764a0 0x7f5a641a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:25.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.800+0000 7f5a62800700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a64115360 0x7f5a641a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.800+0000 7f5a61e00700 1 -- 172.21.15.43:0/355886454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a640764a0 msgr2=0x7f5a641a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.800+0000 7f5a61e00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a640764a0 0x7f5a641a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.800+0000 7f5a61e00700 1 -- 172.21.15.43:0/355886454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a64115360 msgr2=0x7f5a641a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.800+0000 7f5a61e00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a64115360 0x7f5a641a5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.800+0000 7f5a61e00700 1 -- 172.21.15.43:0/355886454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a641a60e0 con 0x7f5a64075b50 2024-05-27T07:09:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.800+0000 7f5a62800700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a64115360 0x7f5a641a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:09:25.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.801+0000 7f5a61e00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a64075b50 0x7f5a641a12c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400d950 tx=0x7f5a5400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.805+0000 7f5a5aa00700 1 -- 172.21.15.43:0/355886454 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a54016070 con 0x7f5a64075b50 2024-05-27T07:09:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.805+0000 7f5a5aa00700 1 -- 172.21.15.43:0/355886454 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5a54010460 con 0x7f5a64075b50 2024-05-27T07:09:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.805+0000 7f5a5aa00700 1 -- 172.21.15.43:0/355886454 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5400f620 con 0x7f5a64075b50 2024-05-27T07:09:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.805+0000 7f5a68a00700 1 -- 172.21.15.43:0/355886454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a641a63d0 con 0x7f5a64075b50 2024-05-27T07:09:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.805+0000 7f5a68a00700 1 -- 172.21.15.43:0/355886454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a641adda0 con 0x7f5a64075b50 2024-05-27T07:09:25.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.806+0000 7f5a68a00700 1 -- 172.21.15.43:0/355886454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a640512b0 con 0x7f5a64075b50 2024-05-27T07:09:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.809+0000 7f5a5aa00700 1 -- 172.21.15.43:0/355886454 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5a54019070 con 0x7f5a64075b50 2024-05-27T07:09:25.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.810+0000 7f5a5aa00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a480b2900 0x7f5a480b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:25.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.810+0000 7f5a5aa00700 1 -- 172.21.15.43:0/355886454 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f5a540e21a0 con 0x7f5a64075b50 2024-05-27T07:09:25.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.811+0000 7f5a5be00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a480b2900 0x7f5a480b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:25.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.811+0000 7f5a5be00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a480b2900 0x7f5a480b4e10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c005fd0 tx=0x7f5a4c00af20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:25.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:25.815+0000 7f5a5aa00700 1 -- 172.21.15.43:0/355886454 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5a540a9b00 con 0x7f5a64075b50 2024-05-27T07:09:26.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.081+0000 7f5a68a00700 1 -- 172.21.15.43:0/355886454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5a6402e5e0 con 0x7f5a64075b50 2024-05-27T07:09:26.074 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:25 smithi043 ceph-mon[31531]: pgmap v424: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:26.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.083+0000 7f5a5aa00700 1 -- 172.21.15.43:0/355886454 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+590 (secure 0 0 0) 0x7f5a540a9400 con 0x7f5a64075b50 2024-05-27T07:09:26.080 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/1651567761 2024-05-28T06:57:52.506320+0000 2024-05-27T07:09:26.080 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/1651567761 2024-05-28T06:57:52.506320+0000 2024-05-27T07:09:26.080 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1332970913 2024-05-28T06:57:18.380825+0000 2024-05-27T07:09:26.080 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1752945868 2024-05-28T06:57:18.380825+0000 2024-05-27T07:09:26.080 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2426463283 2024-05-28T06:57:18.380825+0000 2024-05-27T07:09:26.081 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2081880966 2024-05-28T06:57:52.506320+0000 2024-05-27T07:09:26.081 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/3521718752 2024-05-28T06:57:18.380825+0000 2024-05-27T07:09:26.081 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/830904719 2024-05-28T06:57:52.506320+0000 2024-05-27T07:09:26.081 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/3521718752 2024-05-28T06:57:18.380825+0000 2024-05-27T07:09:26.081 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/338672925 2024-05-28T06:57:52.506320+0000 2024-05-27T07:09:26.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 -- 172.21.15.43:0/355886454 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a480b2900 msgr2=0x7f5a480b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a480b2900 0x7f5a480b4e10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c005fd0 tx=0x7f5a4c00af20 comp rx=0 tx=0).stop 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 -- 172.21.15.43:0/355886454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a64075b50 msgr2=0x7f5a641a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a64075b50 0x7f5a641a12c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400d950 tx=0x7f5a5400dd10 comp rx=0 tx=0).stop 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 -- 172.21.15.43:0/355886454 shutdown_connections 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f5a480b2900 0x7f5a480b4e10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a64115360 0x7f5a641a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a64075b50 0x7f5a641a12c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 --2- 172.21.15.43:0/355886454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a640764a0 0x7f5a641a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 -- 172.21.15.43:0/355886454 >> 172.21.15.43:0/355886454 conn(0x7f5a64100030 msgr2=0x7f5a64100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 -- 172.21.15.43:0/355886454 shutdown_connections 2024-05-27T07:09:26.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:26.087+0000 7f5a68a00700 1 -- 172.21.15.43:0/355886454 wait complete. 2024-05-27T07:09:26.083 INFO:teuthology.orchestra.run.smithi043.stderr:listed 10 entries 2024-05-27T07:09:26.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:25 smithi132 ceph-mon[35283]: pgmap v424: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:26.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:25 smithi179 ceph-mon[35888]: pgmap v424: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:26.947 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-05-27T07:09:26.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:26.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-27T07:09:27.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi043.stderr:ceph-fuse[75082]: fuse finished with error 0 and tester_r 0 2024-05-27T07:09:27.045 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:09:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:26 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/355886454' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:09:27.169 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:26 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/355886454' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:09:27.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:26 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/355886454' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:09:28.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:27 smithi132 ceph-mon[35283]: pgmap v425: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:28.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:27 smithi043 ceph-mon[31531]: pgmap v425: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:28.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:27 smithi179 ceph-mon[35888]: pgmap v425: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:30.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:29 smithi132 ceph-mon[35283]: pgmap v426: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:30.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:29 smithi043 ceph-mon[31531]: pgmap v426: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:30.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:29 smithi179 ceph-mon[35888]: pgmap v426: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-27T07:09:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:31 smithi132 ceph-mon[35283]: pgmap v427: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-27T07:09:32.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:31 smithi043 ceph-mon[31531]: pgmap v427: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-27T07:09:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:31 smithi179 ceph-mon[35888]: pgmap v427: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-27T07:09:33.146 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-05-27T07:09:33.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:09:33.146 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-27T07:09:33.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:09:33.693 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:34.090 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:33 smithi043 ceph-mon[31531]: pgmap v428: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:34.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:33 smithi132 ceph-mon[35283]: pgmap v428: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:34.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:33 smithi179 ceph-mon[35888]: pgmap v428: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:36.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.043+0000 7f2d92000700 1 -- 172.21.15.43:0/85710431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d8c10a980 msgr2=0x7f2d8c10ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:36.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.043+0000 7f2d92000700 1 --2- 172.21.15.43:0/85710431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d8c10a980 0x7f2d8c10ad90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2d74009990 tx=0x7f2d74009ca0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.044+0000 7f2d92000700 1 -- 172.21.15.43:0/85710431 shutdown_connections 2024-05-27T07:09:36.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.044+0000 7f2d92000700 1 --2- 172.21.15.43:0/85710431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d8c10b2d0 0x7f2d8c117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.044+0000 7f2d92000700 1 --2- 172.21.15.43:0/85710431 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d8c110960 0x7f2d8c110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.044+0000 7f2d92000700 1 --2- 172.21.15.43:0/85710431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d8c10a980 0x7f2d8c10ad90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.044+0000 7f2d92000700 1 -- 172.21.15.43:0/85710431 >> 172.21.15.43:0/85710431 conn(0x7f2d8c077c70 msgr2=0x7f2d8c0780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:36.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.044+0000 7f2d92000700 1 -- 172.21.15.43:0/85710431 shutdown_connections 2024-05-27T07:09:36.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.044+0000 7f2d92000700 1 -- 172.21.15.43:0/85710431 wait complete. 2024-05-27T07:09:36.037 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:35 smithi043 ceph-mon[31531]: pgmap v429: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.045+0000 7f2d92000700 1 Processor -- start 2024-05-27T07:09:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.045+0000 7f2d92000700 1 -- start start 2024-05-27T07:09:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d92000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d8c10a980 0x7f2d8c1a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d92000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d8c10b2d0 0x7f2d8c1a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d92000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d8c110960 0x7f2d8c1a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d92000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c11a960 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d92000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c11a7e0 con 0x7f2d8c110960 2024-05-27T07:09:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d92000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c11aae0 con 0x7f2d8c10a980 2024-05-27T07:09:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d82a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d8c10b2d0 0x7f2d8c1a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:36.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d82a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d8c10b2d0 0x7f2d8c1a1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53572/0 (socket says 172.21.15.43:53572) 2024-05-27T07:09:36.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d82a00700 1 -- 172.21.15.43:0/2261737409 learned_addr learned my addr 172.21.15.43:0/2261737409 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:36.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d8b400700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d8c10a980 0x7f2d8c1a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.046+0000 7f2d8be00700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d8c110960 0x7f2d8c1a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.047+0000 7f2d82a00700 1 -- 172.21.15.43:0/2261737409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d8c10a980 msgr2=0x7f2d8c1a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.047+0000 7f2d82a00700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d8c10a980 0x7f2d8c1a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.047+0000 7f2d82a00700 1 -- 172.21.15.43:0/2261737409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d8c110960 msgr2=0x7f2d8c1a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.047+0000 7f2d82a00700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d8c110960 0x7f2d8c1a5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.047+0000 7f2d82a00700 1 -- 172.21.15.43:0/2261737409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8c1a62c0 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.047+0000 7f2d82a00700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d8c10b2d0 0x7f2d8c1a1990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2d74000c00 tx=0x7f2d740057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.048+0000 7f2d88c00700 1 -- 172.21.15.43:0/2261737409 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d74027070 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.048+0000 7f2d92000700 1 -- 172.21.15.43:0/2261737409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8c1a6550 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.048+0000 7f2d88c00700 1 -- 172.21.15.43:0/2261737409 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2d7400fb00 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.048+0000 7f2d88c00700 1 -- 172.21.15.43:0/2261737409 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d74022a50 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.048+0000 7f2d92000700 1 -- 172.21.15.43:0/2261737409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d8c1a6a30 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.049+0000 7f2d92000700 1 -- 172.21.15.43:0/2261737409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d8c0512b0 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.052+0000 7f2d88c00700 1 -- 172.21.15.43:0/2261737409 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f2d7402a070 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.053+0000 7f2d88c00700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2d700b2880 0x7f2d700b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:36.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.053+0000 7f2d88c00700 1 -- 172.21.15.43:0/2261737409 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f2d740e38b0 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.054+0000 7f2d8b400700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2d700b2880 0x7f2d700b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:36.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.054+0000 7f2d8b400700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2d700b2880 0x7f2d700b4d90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009550 tx=0x7f2d7c0091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:36.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.056+0000 7f2d88c00700 1 -- 172.21.15.43:0/2261737409 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2d7405fdf0 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:35 smithi132 ceph-mon[35283]: pgmap v429: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:36.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:35 smithi179 ceph-mon[35888]: pgmap v429: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:36.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.298+0000 7f2d92000700 1 -- 172.21.15.43:0/2261737409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d8c02e5e0 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.303+0000 7f2d88c00700 1 -- 172.21.15.43:0/2261737409 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24656 (secure 0 0 0) 0x7f2d74013070 con 0x7f2d8c10b2d0 2024-05-27T07:09:36.296 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:09:36.296 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":88,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:08:53.944807+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 -- 172.21.15.43:0/2261737409 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2d700b2880 msgr2=0x7f2d700b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2d700b2880 0x7f2d700b4d90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009550 tx=0x7f2d7c0091b0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 -- 172.21.15.43:0/2261737409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d8c10b2d0 msgr2=0x7f2d8c1a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d8c10b2d0 0x7f2d8c1a1990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2d74000c00 tx=0x7f2d740057a0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 -- 172.21.15.43:0/2261737409 shutdown_connections 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2d700b2880 0x7f2d700b4d90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2d8c110960 0x7f2d8c1a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d8c10a980 0x7f2d8c1a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 --2- 172.21.15.43:0/2261737409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d8c10b2d0 0x7f2d8c1a1990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 -- 172.21.15.43:0/2261737409 >> 172.21.15.43:0/2261737409 conn(0x7f2d8c077c70 msgr2=0x7f2d8c07b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:36.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 -- 172.21.15.43:0/2261737409 shutdown_connections 2024-05-27T07:09:36.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:36.307+0000 7f2d92000700 1 -- 172.21.15.43:0/2261737409 wait complete. 2024-05-27T07:09:36.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:36 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2261737409' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:09:37.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:09:37.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:36 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2261737409' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:09:37.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:36 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2261737409' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:09:37.516 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:37.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:37 smithi043 ceph-mon[31531]: pgmap v430: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:37 smithi179 ceph-mon[35888]: pgmap v430: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:38.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:37 smithi132 ceph-mon[35283]: pgmap v430: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.828+0000 7fdc4a800700 1 -- 172.21.15.43:0/2274120540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc44105dc0 msgr2=0x7fdc4410e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.828+0000 7fdc4a800700 1 --2- 172.21.15.43:0/2274120540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc44105dc0 0x7fdc4410e310 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009990 tx=0x7fdc2c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:09:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.829+0000 7fdc4a800700 1 -- 172.21.15.43:0/2274120540 shutdown_connections 2024-05-27T07:09:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.829+0000 7fdc4a800700 1 --2- 172.21.15.43:0/2274120540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc4410e8e0 0x7fdc44110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.829+0000 7fdc4a800700 1 --2- 172.21.15.43:0/2274120540 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc44105470 0x7fdc44105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.829+0000 7fdc4a800700 1 --2- 172.21.15.43:0/2274120540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc44105dc0 0x7fdc4410e310 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.829+0000 7fdc4a800700 1 -- 172.21.15.43:0/2274120540 >> 172.21.15.43:0/2274120540 conn(0x7fdc440fef20 msgr2=0x7fdc44101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:39.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.829+0000 7fdc4a800700 1 -- 172.21.15.43:0/2274120540 shutdown_connections 2024-05-27T07:09:39.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.829+0000 7fdc4a800700 1 -- 172.21.15.43:0/2274120540 wait complete. 2024-05-27T07:09:39.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.830+0000 7fdc4a800700 1 Processor -- start 2024-05-27T07:09:39.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.830+0000 7fdc4a800700 1 -- start start 2024-05-27T07:09:39.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc4a800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc44105470 0x7fdc4419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc4a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc44105dc0 0x7fdc4419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc4a800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc4410e8e0 0x7fdc4419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc4a800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441142f0 con 0x7fdc44105dc0 2024-05-27T07:09:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc4a800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdc44114170 con 0x7fdc44105470 2024-05-27T07:09:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc4a800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdc44114470 con 0x7fdc4410e8e0 2024-05-27T07:09:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc3aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc44105dc0 0x7fdc4419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc3aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc44105dc0 0x7fdc4419b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53594/0 (socket says 172.21.15.43:53594) 2024-05-27T07:09:39.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc3aa00700 1 -- 172.21.15.43:0/3824646319 learned_addr learned my addr 172.21.15.43:0/3824646319 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:39.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc43e00700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc4410e8e0 0x7fdc4419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:39.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.831+0000 7fdc43400700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc44105470 0x7fdc4419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.832+0000 7fdc3aa00700 1 -- 172.21.15.43:0/3824646319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc4410e8e0 msgr2=0x7fdc4419f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.832+0000 7fdc3aa00700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc4410e8e0 0x7fdc4419f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.832+0000 7fdc3aa00700 1 -- 172.21.15.43:0/3824646319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc44105470 msgr2=0x7fdc4419ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.832+0000 7fdc3aa00700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc44105470 0x7fdc4419ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.832+0000 7fdc3aa00700 1 -- 172.21.15.43:0/3824646319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4419fb80 con 0x7fdc44105dc0 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.832+0000 7fdc3aa00700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc44105dc0 0x7fdc4419b300 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0057f0 tx=0x7fdc2c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.832+0000 7fdc40c00700 1 -- 172.21.15.43:0/3824646319 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c027070 con 0x7fdc44105dc0 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.832+0000 7fdc40c00700 1 -- 172.21.15.43:0/3824646319 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc2c00fd60 con 0x7fdc44105dc0 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.832+0000 7fdc40c00700 1 -- 172.21.15.43:0/3824646319 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c022d70 con 0x7fdc44105dc0 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.832+0000 7fdc4a800700 1 -- 172.21.15.43:0/3824646319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc4419fe10 con 0x7fdc44105dc0 2024-05-27T07:09:39.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.833+0000 7fdc4a800700 1 -- 172.21.15.43:0/3824646319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc441a02a0 con 0x7fdc44105dc0 2024-05-27T07:09:39.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.834+0000 7fdc4a800700 1 -- 172.21.15.43:0/3824646319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc440512b0 con 0x7fdc44105dc0 2024-05-27T07:09:39.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.838+0000 7fdc40c00700 1 -- 172.21.15.43:0/3824646319 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fdc2c02a070 con 0x7fdc44105dc0 2024-05-27T07:09:39.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.840+0000 7fdc40c00700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc280b2880 0x7fdc280b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:39.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.840+0000 7fdc40c00700 1 -- 172.21.15.43:0/3824646319 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fdc2c0e38b0 con 0x7fdc44105dc0 2024-05-27T07:09:39.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.840+0000 7fdc43400700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc280b2880 0x7fdc280b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:39.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.841+0000 7fdc43400700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc280b2880 0x7fdc280b4d90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdc34005fd0 tx=0x7fdc3400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:39.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:39.842+0000 7fdc40c00700 1 -- 172.21.15.43:0/3824646319 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdc2c013070 con 0x7fdc44105dc0 2024-05-27T07:09:40.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.087+0000 7fdc4a800700 1 -- 172.21.15.43:0/3824646319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc4402e5e0 con 0x7fdc44105dc0 2024-05-27T07:09:40.080 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:39 smithi043 ceph-mon[31531]: pgmap v431: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.092+0000 7fdc40c00700 1 -- 172.21.15.43:0/3824646319 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24656 (secure 0 0 0) 0x7fdc2c0ab330 con 0x7fdc44105dc0 2024-05-27T07:09:40.085 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:09:40.087 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":88,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:08:53.944807+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:09:40.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.095+0000 7fdc4a800700 1 -- 172.21.15.43:0/3824646319 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc280b2880 msgr2=0x7fdc280b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:40.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.095+0000 7fdc4a800700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc280b2880 0x7fdc280b4d90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdc34005fd0 tx=0x7fdc3400a560 comp rx=0 tx=0).stop 2024-05-27T07:09:40.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.095+0000 7fdc4a800700 1 -- 172.21.15.43:0/3824646319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc44105dc0 msgr2=0x7fdc4419b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:40.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.095+0000 7fdc4a800700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc44105dc0 0x7fdc4419b300 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0057f0 tx=0x7fdc2c009f90 comp rx=0 tx=0).stop 2024-05-27T07:09:40.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.096+0000 7fdc4a800700 1 -- 172.21.15.43:0/3824646319 shutdown_connections 2024-05-27T07:09:40.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.096+0000 7fdc4a800700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc280b2880 0x7fdc280b4d90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:40.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.096+0000 7fdc4a800700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc4410e8e0 0x7fdc4419f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:40.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.096+0000 7fdc4a800700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc44105470 0x7fdc4419ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:40.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.096+0000 7fdc4a800700 1 --2- 172.21.15.43:0/3824646319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc44105dc0 0x7fdc4419b300 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:40.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.096+0000 7fdc4a800700 1 -- 172.21.15.43:0/3824646319 >> 172.21.15.43:0/3824646319 conn(0x7fdc440fef20 msgr2=0x7fdc44100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:40.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.096+0000 7fdc4a800700 1 -- 172.21.15.43:0/3824646319 shutdown_connections 2024-05-27T07:09:40.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:40.096+0000 7fdc4a800700 1 -- 172.21.15.43:0/3824646319 wait complete. 2024-05-27T07:09:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:39 smithi179 ceph-mon[35888]: pgmap v431: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:40.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:39 smithi132 ceph-mon[35283]: pgmap v431: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:40.886 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-05-27T07:09:41.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:40 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3824646319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:09:41.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:40 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3824646319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:09:41.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:40 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3824646319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:09:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:42.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:41 smithi043 ceph-mon[31531]: pgmap v432: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:42.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:41 smithi179 ceph-mon[35888]: pgmap v432: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:42.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:41 smithi132 ceph-mon[35283]: pgmap v432: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-05-27T07:09:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.653+0000 7f97b3e00700 1 -- 172.21.15.43:0/771697818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97b4109670 msgr2=0x7f97b4110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.653+0000 7f97b3e00700 1 --2- 172.21.15.43:0/771697818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97b4109670 0x7f97b4110fd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00b0d0 tx=0x7f97ac00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:09:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.653+0000 7f97b3e00700 1 -- 172.21.15.43:0/771697818 shutdown_connections 2024-05-27T07:09:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.653+0000 7f97b3e00700 1 --2- 172.21.15.43:0/771697818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97b4109670 0x7f97b4110fd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.653+0000 7f97b3e00700 1 --2- 172.21.15.43:0/771697818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f97b41043f0 0x7f97b4104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.653+0000 7f97b3e00700 1 --2- 172.21.15.43:0/771697818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b4108d20 0x7f97b4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.653+0000 7f97b3e00700 1 -- 172.21.15.43:0/771697818 >> 172.21.15.43:0/771697818 conn(0x7f97b4079d00 msgr2=0x7f97b407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.653+0000 7f97b3e00700 1 -- 172.21.15.43:0/771697818 shutdown_connections 2024-05-27T07:09:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.653+0000 7f97b3e00700 1 -- 172.21.15.43:0/771697818 wait complete. 2024-05-27T07:09:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.655+0000 7f97b3e00700 1 Processor -- start 2024-05-27T07:09:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.655+0000 7f97b3e00700 1 -- start start 2024-05-27T07:09:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.655+0000 7f97b3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97b41043f0 0x7f97b4075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.656+0000 7f97b3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f97b4108d20 0x7f97b4075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.656+0000 7f97b3e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b4109670 0x7f97b4076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.656+0000 7f97b3e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4114150 con 0x7f97b41043f0 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.656+0000 7f97b3e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4113fd0 con 0x7f97b4108d20 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.656+0000 7f97b3e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f97b41142d0 con 0x7f97b4109670 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.656+0000 7f97b2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97b41043f0 0x7f97b4075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.656+0000 7f97b2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97b41043f0 0x7f97b4075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53616/0 (socket says 172.21.15.43:53616) 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.656+0000 7f97b2a00700 1 -- 172.21.15.43:0/1399061533 learned_addr learned my addr 172.21.15.43:0/1399061533 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.656+0000 7f97b2000700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f97b4108d20 0x7f97b4075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.656+0000 7f97b3400700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b4109670 0x7f97b4076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.657+0000 7f97b2a00700 1 -- 172.21.15.43:0/1399061533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b4109670 msgr2=0x7f97b4076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.657+0000 7f97b2a00700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b4109670 0x7f97b4076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.657+0000 7f97b2a00700 1 -- 172.21.15.43:0/1399061533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f97b4108d20 msgr2=0x7f97b4075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.657+0000 7f97b2a00700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f97b4108d20 0x7f97b4075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.657+0000 7f97b2a00700 1 -- 172.21.15.43:0/1399061533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b41a8e30 con 0x7f97b41043f0 2024-05-27T07:09:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.657+0000 7f97b2a00700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97b41043f0 0x7f97b4075ab0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f97a400c9b0 tx=0x7f97a400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.657+0000 7f97ab400700 1 -- 172.21.15.43:0/1399061533 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a4007b00 con 0x7f97b41043f0 2024-05-27T07:09:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.657+0000 7f97ab400700 1 -- 172.21.15.43:0/1399061533 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97a4007ca0 con 0x7f97b41043f0 2024-05-27T07:09:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.657+0000 7f97b3e00700 1 -- 172.21.15.43:0/1399061533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b41a9120 con 0x7f97b41043f0 2024-05-27T07:09:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.658+0000 7f97ab400700 1 -- 172.21.15.43:0/1399061533 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a4018660 con 0x7f97b41043f0 2024-05-27T07:09:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.658+0000 7f97b3e00700 1 -- 172.21.15.43:0/1399061533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b41a96b0 con 0x7f97b41043f0 2024-05-27T07:09:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.659+0000 7f97b3e00700 1 -- 172.21.15.43:0/1399061533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b41058a0 con 0x7f97b41043f0 2024-05-27T07:09:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.661+0000 7f97ab400700 1 -- 172.21.15.43:0/1399061533 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f97a400f450 con 0x7f97b41043f0 2024-05-27T07:09:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.664+0000 7f97ab400700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f97980b2900 0x7f97980b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.664+0000 7f97ab400700 1 -- 172.21.15.43:0/1399061533 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f97a40e28a0 con 0x7f97b41043f0 2024-05-27T07:09:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.664+0000 7f97b2000700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f97980b2900 0x7f97980b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.665+0000 7f97b2000700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f97980b2900 0x7f97980b4e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f97a0005950 tx=0x7f97a00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:43.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.667+0000 7f97ab400700 1 -- 172.21.15.43:0/1399061533 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f97a4014030 con 0x7f97b41043f0 2024-05-27T07:09:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:43.959+0000 7f97b3e00700 1 -- 172.21.15.43:0/1399061533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f97b402e5e0 con 0x7f97b41043f0 2024-05-27T07:09:43.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:43 smithi043 ceph-mon[31531]: pgmap v433: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:44.144 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:43 smithi132 ceph-mon[35283]: pgmap v433: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:44.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:43 smithi179 ceph-mon[35888]: pgmap v433: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:44.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.908+0000 7f97ab400700 1 -- 172.21.15.43:0/1399061533 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f97a40aa310 con 0x7f97b41043f0 2024-05-27T07:09:44.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.912+0000 7f97b3e00700 1 -- 172.21.15.43:0/1399061533 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f97980b2900 msgr2=0x7f97980b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:44.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.912+0000 7f97b3e00700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f97980b2900 0x7f97980b4e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f97a0005950 tx=0x7f97a00058e0 comp rx=0 tx=0).stop 2024-05-27T07:09:44.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.912+0000 7f97b3e00700 1 -- 172.21.15.43:0/1399061533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97b41043f0 msgr2=0x7f97b4075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:44.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.912+0000 7f97b3e00700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97b41043f0 0x7f97b4075ab0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f97a400c9b0 tx=0x7f97a400cd70 comp rx=0 tx=0).stop 2024-05-27T07:09:44.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.913+0000 7f97b3e00700 1 -- 172.21.15.43:0/1399061533 shutdown_connections 2024-05-27T07:09:44.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.913+0000 7f97b3e00700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f97980b2900 0x7f97980b4e10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:44.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.913+0000 7f97b3e00700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b4109670 0x7f97b4076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:44.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.913+0000 7f97b3e00700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97b41043f0 0x7f97b4075ab0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:44.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.913+0000 7f97b3e00700 1 --2- 172.21.15.43:0/1399061533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f97b4108d20 0x7f97b4075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:44.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.913+0000 7f97b3e00700 1 -- 172.21.15.43:0/1399061533 >> 172.21.15.43:0/1399061533 conn(0x7f97b4079d00 msgr2=0x7f97b4101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:44.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.913+0000 7f97b3e00700 1 -- 172.21.15.43:0/1399061533 shutdown_connections 2024-05-27T07:09:44.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:44.913+0000 7f97b3e00700 1 -- 172.21.15.43:0/1399061533 wait complete. 2024-05-27T07:09:45.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:44 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1399061533' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:09:45.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:44 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1399061533' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:09:45.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:44 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1399061533' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-27T07:09:45.749 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-05-27T07:09:45.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-05-27T07:09:45.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:45 smithi043 ceph-mon[31531]: pgmap v434: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:45.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:45 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1399061533' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:09:45.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:45 smithi043 ceph-mon[31531]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:09:46.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:45 smithi179 ceph-mon[35888]: pgmap v434: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:46.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:45 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1399061533' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:09:46.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:45 smithi179 ceph-mon[35888]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:09:46.213 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:46.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:45 smithi132 ceph-mon[35283]: pgmap v434: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:46.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:45 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1399061533' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-27T07:09:46.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:45 smithi132 ceph-mon[35283]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:09:48.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:47 smithi043 ceph-mon[31531]: pgmap v435: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:48.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:47 smithi179 ceph-mon[35888]: pgmap v435: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:47 smithi132 ceph-mon[35283]: pgmap v435: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:48.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.563+0000 7fe57cc00700 1 -- 172.21.15.43:0/2001985187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5781043f0 msgr2=0x7fe578104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:48.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.563+0000 7fe57cc00700 1 --2- 172.21.15.43:0/2001985187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5781043f0 0x7fe578104800 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe5680099c0 tx=0x7fe568009cd0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.563+0000 7fe57cc00700 1 -- 172.21.15.43:0/2001985187 shutdown_connections 2024-05-27T07:09:48.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.563+0000 7fe57cc00700 1 --2- 172.21.15.43:0/2001985187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe578109670 0x7fe578110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.563+0000 7fe57cc00700 1 --2- 172.21.15.43:0/2001985187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5781043f0 0x7fe578104800 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.563+0000 7fe57cc00700 1 --2- 172.21.15.43:0/2001985187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe578108d20 0x7fe578109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.563+0000 7fe57cc00700 1 -- 172.21.15.43:0/2001985187 >> 172.21.15.43:0/2001985187 conn(0x7fe578079d00 msgr2=0x7fe57807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:48.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.563+0000 7fe57cc00700 1 -- 172.21.15.43:0/2001985187 shutdown_connections 2024-05-27T07:09:48.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.563+0000 7fe57cc00700 1 -- 172.21.15.43:0/2001985187 wait complete. 2024-05-27T07:09:48.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.565+0000 7fe57cc00700 1 Processor -- start 2024-05-27T07:09:48.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.565+0000 7fe57cc00700 1 -- start start 2024-05-27T07:09:48.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.565+0000 7fe57cc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5781043f0 0x7fe578075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:48.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.565+0000 7fe57cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe578108d20 0x7fe578075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:48.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.565+0000 7fe57cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe578109670 0x7fe578076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:48.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.566+0000 7fe57cc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe578114150 con 0x7fe578108d20 2024-05-27T07:09:48.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.566+0000 7fe57cc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe578113fd0 con 0x7fe578109670 2024-05-27T07:09:48.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.566+0000 7fe57cc00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe5781142d0 con 0x7fe5781043f0 2024-05-27T07:09:48.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.566+0000 7fe576a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe578108d20 0x7fe578075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:48.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.566+0000 7fe576a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe578108d20 0x7fe578075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57746/0 (socket says 172.21.15.43:57746) 2024-05-27T07:09:48.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.566+0000 7fe576a00700 1 -- 172.21.15.43:0/194138156 learned_addr learned my addr 172.21.15.43:0/194138156 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:48.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.566+0000 7fe577400700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5781043f0 0x7fe578075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:48.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.566+0000 7fe577e00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe578109670 0x7fe578076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:48.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.567+0000 7fe576a00700 1 -- 172.21.15.43:0/194138156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5781043f0 msgr2=0x7fe578075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:48.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.567+0000 7fe576a00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5781043f0 0x7fe578075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.567+0000 7fe576a00700 1 -- 172.21.15.43:0/194138156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe578109670 msgr2=0x7fe578076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:48.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.567+0000 7fe576a00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe578109670 0x7fe578076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.567+0000 7fe576a00700 1 -- 172.21.15.43:0/194138156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5781a8ee0 con 0x7fe578108d20 2024-05-27T07:09:48.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.567+0000 7fe576a00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe578108d20 0x7fe578075ff0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe56400d980 tx=0x7fe56400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:48.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.567+0000 7fe56fe00700 1 -- 172.21.15.43:0/194138156 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564017070 con 0x7fe578108d20 2024-05-27T07:09:48.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.567+0000 7fe56fe00700 1 -- 172.21.15.43:0/194138156 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe564009dc0 con 0x7fe578108d20 2024-05-27T07:09:48.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.567+0000 7fe57cc00700 1 -- 172.21.15.43:0/194138156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5781a91d0 con 0x7fe578108d20 2024-05-27T07:09:48.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.567+0000 7fe56fe00700 1 -- 172.21.15.43:0/194138156 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56400f3c0 con 0x7fe578108d20 2024-05-27T07:09:48.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.568+0000 7fe57cc00700 1 -- 172.21.15.43:0/194138156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5781a9760 con 0x7fe578108d20 2024-05-27T07:09:48.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.569+0000 7fe57cc00700 1 -- 172.21.15.43:0/194138156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5780512b0 con 0x7fe578108d20 2024-05-27T07:09:48.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.572+0000 7fe56fe00700 1 -- 172.21.15.43:0/194138156 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe564009930 con 0x7fe578108d20 2024-05-27T07:09:48.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.574+0000 7fe56fe00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe55c0b2900 0x7fe55c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:48.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.574+0000 7fe56fe00700 1 -- 172.21.15.43:0/194138156 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fe5640e1ff0 con 0x7fe578108d20 2024-05-27T07:09:48.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.574+0000 7fe577400700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe55c0b2900 0x7fe55c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:48.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.575+0000 7fe577400700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe55c0b2900 0x7fe55c0b4e10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe568000c00 tx=0x7fe568005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:48.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.576+0000 7fe56fe00700 1 -- 172.21.15.43:0/194138156 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe5640a9a10 con 0x7fe578108d20 2024-05-27T07:09:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.869+0000 7fe57cc00700 1 -- 172.21.15.43:0/194138156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fe57802e5e0 con 0x7fe578108d20 2024-05-27T07:09:48.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.932+0000 7fe56fe00700 1 -- 172.21.15.43:0/194138156 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fe5640a9310 con 0x7fe578108d20 2024-05-27T07:09:48.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.936+0000 7fe57cc00700 1 -- 172.21.15.43:0/194138156 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe55c0b2900 msgr2=0x7fe55c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:48.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.936+0000 7fe57cc00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe55c0b2900 0x7fe55c0b4e10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe568000c00 tx=0x7fe568005fb0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.936+0000 7fe57cc00700 1 -- 172.21.15.43:0/194138156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe578108d20 msgr2=0x7fe578075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:48.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.936+0000 7fe57cc00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe578108d20 0x7fe578075ff0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe56400d980 tx=0x7fe56400dc90 comp rx=0 tx=0).stop 2024-05-27T07:09:48.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.937+0000 7fe57cc00700 1 -- 172.21.15.43:0/194138156 shutdown_connections 2024-05-27T07:09:48.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.937+0000 7fe57cc00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe55c0b2900 0x7fe55c0b4e10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.937+0000 7fe57cc00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe578109670 0x7fe578076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.937+0000 7fe57cc00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5781043f0 0x7fe578075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.937+0000 7fe57cc00700 1 --2- 172.21.15.43:0/194138156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe578108d20 0x7fe578075ff0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:48.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.937+0000 7fe57cc00700 1 -- 172.21.15.43:0/194138156 >> 172.21.15.43:0/194138156 conn(0x7fe578079d00 msgr2=0x7fe578101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:48.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.937+0000 7fe57cc00700 1 -- 172.21.15.43:0/194138156 shutdown_connections 2024-05-27T07:09:48.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:48.937+0000 7fe57cc00700 1 -- 172.21.15.43:0/194138156 wait complete. 2024-05-27T07:09:49.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:48 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/194138156' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-27T07:09:49.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:48 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/194138156' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-27T07:09:49.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:48 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/194138156' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-27T07:09:49.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-05-27T07:09:50.195 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:50.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:49 smithi043 ceph-mon[31531]: pgmap v436: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:50.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:49 smithi043 ceph-mon[31531]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T07:09:50.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:49 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/194138156' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-27T07:09:50.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:49 smithi043 ceph-mon[31531]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T07:09:50.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:49 smithi043 ceph-mon[31531]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:09:50.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:49 smithi043 ceph-mon[31531]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-27T07:09:50.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:49 smithi043 ceph-mon[31531]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T07:09:50.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:49 smithi179 ceph-mon[35888]: pgmap v436: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:50.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:49 smithi179 ceph-mon[35888]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T07:09:50.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:49 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/194138156' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-27T07:09:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:49 smithi179 ceph-mon[35888]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T07:09:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:49 smithi179 ceph-mon[35888]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:09:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:49 smithi179 ceph-mon[35888]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-27T07:09:50.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:49 smithi179 ceph-mon[35888]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T07:09:50.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:49 smithi132 ceph-mon[35283]: pgmap v436: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:09:50.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:49 smithi132 ceph-mon[35283]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-27T07:09:50.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:49 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/194138156' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-27T07:09:50.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:49 smithi132 ceph-mon[35283]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-27T07:09:50.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:49 smithi132 ceph-mon[35283]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-27T07:09:50.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:49 smithi132 ceph-mon[35283]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-27T07:09:50.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:49 smithi132 ceph-mon[35283]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-27T07:09:51.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:50 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] up:active 2024-05-27T07:09:51.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:50 smithi043 ceph-mon[31531]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T07:09:51.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:50 smithi043 ceph-mon[31531]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-27T07:09:51.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:50 smithi043 ceph-mon[31531]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-27T07:09:51.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:50 smithi043 ceph-mon[31531]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-27T07:09:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:50 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] up:active 2024-05-27T07:09:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:50 smithi179 ceph-mon[35888]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T07:09:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:50 smithi179 ceph-mon[35888]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-27T07:09:51.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:50 smithi179 ceph-mon[35888]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-27T07:09:51.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:50 smithi179 ceph-mon[35888]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-27T07:09:51.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:50 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] up:active 2024-05-27T07:09:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:50 smithi132 ceph-mon[35283]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-27T07:09:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:50 smithi132 ceph-mon[35283]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-27T07:09:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:50 smithi132 ceph-mon[35283]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-27T07:09:51.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:50 smithi132 ceph-mon[35283]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-27T07:09:52.091 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:09:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b[66148]: 2024-05-27T07:09:51.842+0000 7f0941000700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-05-27T07:09:52.092 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:09:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b[66148]: 2024-05-27T07:09:51.842+0000 7f0941000700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-05-27T07:09:52.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:51 smithi132 ceph-mon[35283]: pgmap v437: 129 pgs: 129 active+clean; 582 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 2 op/s 2024-05-27T07:09:52.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:51 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] up:active 2024-05-27T07:09:52.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:51 smithi132 ceph-mon[35283]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-27T07:09:52.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:51 smithi132 ceph-mon[35283]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-27T07:09:52.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:51 smithi132 ceph-mon[35283]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:creating} 8 up:standby 2024-05-27T07:09:52.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:51 smithi132 ceph-mon[35283]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-05-27T07:09:52.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:51 smithi043 ceph-mon[31531]: pgmap v437: 129 pgs: 129 active+clean; 582 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 2 op/s 2024-05-27T07:09:52.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:51 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] up:active 2024-05-27T07:09:52.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:51 smithi043 ceph-mon[31531]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-27T07:09:52.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:51 smithi043 ceph-mon[31531]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-27T07:09:52.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:51 smithi043 ceph-mon[31531]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:creating} 8 up:standby 2024-05-27T07:09:52.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:51 smithi043 ceph-mon[31531]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-05-27T07:09:52.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.368+0000 7fb71a800700 1 -- 172.21.15.43:0/1903510931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb714103540 msgr2=0x7fb714103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:52.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.368+0000 7fb71a800700 1 --2- 172.21.15.43:0/1903510931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb714103540 0x7fb714103950 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb70400b0d0 tx=0x7fb70400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:09:52.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.368+0000 7fb71a800700 1 -- 172.21.15.43:0/1903510931 shutdown_connections 2024-05-27T07:09:52.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.368+0000 7fb71a800700 1 --2- 172.21.15.43:0/1903510931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb714108770 0x7fb7141100d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:52.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.368+0000 7fb71a800700 1 --2- 172.21.15.43:0/1903510931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb714103540 0x7fb714103950 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:52.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.368+0000 7fb71a800700 1 --2- 172.21.15.43:0/1903510931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb714107e20 0x7fb714108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:52.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.368+0000 7fb71a800700 1 -- 172.21.15.43:0/1903510931 >> 172.21.15.43:0/1903510931 conn(0x7fb714078df0 msgr2=0x7fb714079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:52.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.368+0000 7fb71a800700 1 -- 172.21.15.43:0/1903510931 shutdown_connections 2024-05-27T07:09:52.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.368+0000 7fb71a800700 1 -- 172.21.15.43:0/1903510931 wait complete. 2024-05-27T07:09:52.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.370+0000 7fb71a800700 1 Processor -- start 2024-05-27T07:09:52.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.370+0000 7fb71a800700 1 -- start start 2024-05-27T07:09:52.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.370+0000 7fb71a800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb714103540 0x7fb71419ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:52.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.370+0000 7fb71a800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb714107e20 0x7fb71419b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:52.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.370+0000 7fb71a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb714108770 0x7fb71419f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:52.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.370+0000 7fb71a800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb714113250 con 0x7fb714108770 2024-05-27T07:09:52.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.370+0000 7fb71a800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb7141130d0 con 0x7fb714107e20 2024-05-27T07:09:52.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.371+0000 7fb71a800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb7141133d0 con 0x7fb714103540 2024-05-27T07:09:52.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.371+0000 7fb713e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb714108770 0x7fb71419f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:52.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.371+0000 7fb713e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb714108770 0x7fb71419f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57758/0 (socket says 172.21.15.43:57758) 2024-05-27T07:09:52.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.371+0000 7fb713e00700 1 -- 172.21.15.43:0/2213594808 learned_addr learned my addr 172.21.15.43:0/2213594808 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:52.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.371+0000 7fb713400700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb714103540 0x7fb71419ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:52.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.371+0000 7fb712a00700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb714107e20 0x7fb71419b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:52.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.372+0000 7fb713e00700 1 -- 172.21.15.43:0/2213594808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb714103540 msgr2=0x7fb71419ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:52.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.372+0000 7fb713e00700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb714103540 0x7fb71419ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:52.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.372+0000 7fb713e00700 1 -- 172.21.15.43:0/2213594808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb714107e20 msgr2=0x7fb71419b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:52.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.372+0000 7fb713e00700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb714107e20 0x7fb71419b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:52.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.372+0000 7fb713e00700 1 -- 172.21.15.43:0/2213594808 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71419fc00 con 0x7fb714108770 2024-05-27T07:09:52.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.372+0000 7fb712a00700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb714107e20 0x7fb71419b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:09:52.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.372+0000 7fb713e00700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb714108770 0x7fb71419f560 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00d980 tx=0x7fb70c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:52.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.372+0000 7fb70be00700 1 -- 172.21.15.43:0/2213594808 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70c017070 con 0x7fb714108770 2024-05-27T07:09:52.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.372+0000 7fb70be00700 1 -- 172.21.15.43:0/2213594808 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb70c005020 con 0x7fb714108770 2024-05-27T07:09:52.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.372+0000 7fb71a800700 1 -- 172.21.15.43:0/2213594808 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71419fef0 con 0x7fb714108770 2024-05-27T07:09:52.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.373+0000 7fb71a800700 1 -- 172.21.15.43:0/2213594808 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb714104df0 con 0x7fb714108770 2024-05-27T07:09:52.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.373+0000 7fb70be00700 1 -- 172.21.15.43:0/2213594808 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70c009990 con 0x7fb714108770 2024-05-27T07:09:52.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.374+0000 7fb71a800700 1 -- 172.21.15.43:0/2213594808 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7141049a0 con 0x7fb714108770 2024-05-27T07:09:52.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.378+0000 7fb70be00700 1 -- 172.21.15.43:0/2213594808 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb70c004a30 con 0x7fb714108770 2024-05-27T07:09:52.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.379+0000 7fb70be00700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb6f80b2880 0x7fb6f80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:52.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.380+0000 7fb70be00700 1 -- 172.21.15.43:0/2213594808 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fb70c0e26f0 con 0x7fb714108770 2024-05-27T07:09:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.380+0000 7fb713400700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb6f80b2880 0x7fb6f80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.381+0000 7fb713400700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb6f80b2880 0x7fb6f80b4d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb704009fd0 tx=0x7fb70400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:52.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.382+0000 7fb70be00700 1 -- 172.21.15.43:0/2213594808 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb70c020020 con 0x7fb714108770 2024-05-27T07:09:52.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:51 smithi179 ceph-mon[35888]: pgmap v437: 129 pgs: 129 active+clean; 582 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 2 op/s 2024-05-27T07:09:52.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:51 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] up:active 2024-05-27T07:09:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:51 smithi179 ceph-mon[35888]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-27T07:09:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:51 smithi179 ceph-mon[35888]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-27T07:09:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:51 smithi179 ceph-mon[35888]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:creating} 8 up:standby 2024-05-27T07:09:52.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:51 smithi179 ceph-mon[35888]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-05-27T07:09:52.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:52.667+0000 7fb71a800700 1 -- 172.21.15.43:0/2213594808 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fb71419c090 con 0x7fb714108770 2024-05-27T07:09:53.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.018+0000 7fb70be00700 1 -- 172.21.15.43:0/2213594808 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v27) ==== 121+0+0 (secure 0 0 0) 0x7fb70c0aa050 con 0x7fb714108770 2024-05-27T07:09:53.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.024+0000 7fb71a800700 1 -- 172.21.15.43:0/2213594808 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb6f80b2880 msgr2=0x7fb6f80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:53.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.024+0000 7fb71a800700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb6f80b2880 0x7fb6f80b4d90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb704009fd0 tx=0x7fb70400b040 comp rx=0 tx=0).stop 2024-05-27T07:09:53.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.024+0000 7fb71a800700 1 -- 172.21.15.43:0/2213594808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb714108770 msgr2=0x7fb71419f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:53.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.024+0000 7fb71a800700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb714108770 0x7fb71419f560 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00d980 tx=0x7fb70c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:09:53.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.025+0000 7fb71a800700 1 -- 172.21.15.43:0/2213594808 shutdown_connections 2024-05-27T07:09:53.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.025+0000 7fb71a800700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb6f80b2880 0x7fb6f80b4d90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:53.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.025+0000 7fb71a800700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb714108770 0x7fb71419f560 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:53.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.025+0000 7fb71a800700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb714103540 0x7fb71419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:53.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.025+0000 7fb71a800700 1 --2- 172.21.15.43:0/2213594808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb714107e20 0x7fb71419b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:53.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.025+0000 7fb71a800700 1 -- 172.21.15.43:0/2213594808 >> 172.21.15.43:0/2213594808 conn(0x7fb714078df0 msgr2=0x7fb714075e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:53.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.026+0000 7fb71a800700 1 -- 172.21.15.43:0/2213594808 shutdown_connections 2024-05-27T07:09:53.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:53.026+0000 7fb71a800700 1 -- 172.21.15.43:0/2213594808 wait complete. 2024-05-27T07:09:53.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:52 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] up:active 2024-05-27T07:09:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:52 smithi132 ceph-mon[35283]: daemon mds.i assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-27T07:09:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:52 smithi132 ceph-mon[35283]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T07:09:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:52 smithi132 ceph-mon[35283]: Cluster is now healthy 2024-05-27T07:09:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:52 smithi132 ceph-mon[35283]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:active} 8 up:standby 2024-05-27T07:09:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:52 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-27T07:09:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:52 smithi132 ceph-mon[35283]: daemon mds.i is now active in filesystem cephfs as rank 4 2024-05-27T07:09:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:52 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2213594808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:09:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-05-27T07:09:53.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:52 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-05-27T07:09:53.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:52 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] up:active 2024-05-27T07:09:53.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:52 smithi043 ceph-mon[31531]: daemon mds.i assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-27T07:09:53.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:52 smithi043 ceph-mon[31531]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T07:09:53.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:52 smithi043 ceph-mon[31531]: Cluster is now healthy 2024-05-27T07:09:53.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:52 smithi043 ceph-mon[31531]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:active} 8 up:standby 2024-05-27T07:09:53.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:52 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-27T07:09:53.408 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:52 smithi043 ceph-mon[31531]: daemon mds.i is now active in filesystem cephfs as rank 4 2024-05-27T07:09:53.409 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:52 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2213594808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:09:53.409 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-05-27T07:09:53.409 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:52 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-05-27T07:09:53.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:52 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] up:active 2024-05-27T07:09:53.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:52 smithi179 ceph-mon[35888]: daemon mds.i assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-27T07:09:53.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:52 smithi179 ceph-mon[35888]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-27T07:09:53.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:52 smithi179 ceph-mon[35888]: Cluster is now healthy 2024-05-27T07:09:53.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:52 smithi179 ceph-mon[35888]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=k=up:active} 8 up:standby 2024-05-27T07:09:53.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:52 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-27T07:09:53.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:52 smithi179 ceph-mon[35888]: daemon mds.i is now active in filesystem cephfs as rank 4 2024-05-27T07:09:53.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:52 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2213594808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-27T07:09:53.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-05-27T07:09:53.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:52 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-05-27T07:09:53.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-05-27T07:09:54.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:54 smithi043 ceph-mon[31531]: pgmap v438: 129 pgs: 129 active+clean; 583 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-27T07:09:54.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:54 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2213594808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:09:54.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:54 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-05-27T07:09:54.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:54 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-05-27T07:09:54.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:54 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] up:active 2024-05-27T07:09:54.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:54 smithi043 ceph-mon[31531]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-27T07:09:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:54 smithi043 ceph-mon[31531]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:09:54.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:54 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:09:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:54 smithi132 ceph-mon[35283]: pgmap v438: 129 pgs: 129 active+clean; 583 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-27T07:09:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:54 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2213594808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:09:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:54 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-05-27T07:09:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:54 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-05-27T07:09:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:54 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] up:active 2024-05-27T07:09:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:54 smithi132 ceph-mon[35283]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-27T07:09:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:54 smithi132 ceph-mon[35283]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:09:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:54 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:09:54.349 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:54 smithi179 ceph-mon[35888]: pgmap v438: 129 pgs: 129 active+clean; 583 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-05-27T07:09:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:54 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2213594808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-27T07:09:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:54 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-05-27T07:09:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:54 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-05-27T07:09:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:54 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] up:active 2024-05-27T07:09:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:54 smithi179 ceph-mon[35888]: fsmap cephfs:5 7 up:standby 5 up:active 2024-05-27T07:09:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:54 smithi179 ceph-mon[35888]: osdmap e89: 12 total, 12 up, 12 in 2024-05-27T07:09:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:54 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:09:55.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:55 smithi043 ceph-mon[31531]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:09:55.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:55 smithi132 ceph-mon[35283]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:09:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:55 smithi179 ceph-mon[35888]: osdmap e90: 12 total, 12 up, 12 in 2024-05-27T07:09:56.180 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:56 smithi043 ceph-mon[31531]: pgmap v441: 129 pgs: 129 active+clean; 585 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 17 op/s 2024-05-27T07:09:56.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:56 smithi132 ceph-mon[35283]: pgmap v441: 129 pgs: 129 active+clean; 585 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 17 op/s 2024-05-27T07:09:56.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:56 smithi179 ceph-mon[35888]: pgmap v441: 129 pgs: 129 active+clean; 585 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 17 op/s 2024-05-27T07:09:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.784+0000 7f8806400700 1 -- 172.21.15.43:0/1432997692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8800077780 msgr2=0x7f8800110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.784+0000 7f8806400700 1 --2- 172.21.15.43:0/1432997692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8800077780 0x7f8800110650 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f87e8009990 tx=0x7f87e8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:09:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.784+0000 7f8806400700 1 -- 172.21.15.43:0/1432997692 shutdown_connections 2024-05-27T07:09:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.784+0000 7f8806400700 1 --2- 172.21.15.43:0/1432997692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8800110b90 0x7f8800112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.784+0000 7f8806400700 1 --2- 172.21.15.43:0/1432997692 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8800076e30 0x7f8800077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.784+0000 7f8806400700 1 --2- 172.21.15.43:0/1432997692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8800077780 0x7f8800110650 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.784+0000 7f8806400700 1 -- 172.21.15.43:0/1432997692 >> 172.21.15.43:0/1432997692 conn(0x7f88000ffeb0 msgr2=0x7f8800102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.785+0000 7f8806400700 1 -- 172.21.15.43:0/1432997692 shutdown_connections 2024-05-27T07:09:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.785+0000 7f8806400700 1 -- 172.21.15.43:0/1432997692 wait complete. 2024-05-27T07:09:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.787+0000 7f8806400700 1 Processor -- start 2024-05-27T07:09:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.787+0000 7f8806400700 1 -- start start 2024-05-27T07:09:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.787+0000 7f8806400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8800076e30 0x7f880019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.787+0000 7f8806400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8800077780 0x7f880019d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.788+0000 7f8806400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8800110b90 0x7f88001a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.788+0000 7f8806400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8800116590 con 0x7f8800077780 2024-05-27T07:09:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.788+0000 7f8806400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8800116410 con 0x7f8800076e30 2024-05-27T07:09:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.788+0000 7f8806400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8800116710 con 0x7f8800110b90 2024-05-27T07:09:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.788+0000 7f87f6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8800077780 0x7f880019d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.788+0000 7f87f6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8800077780 0x7f880019d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53098/0 (socket says 172.21.15.43:53098) 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.788+0000 7f87f6a00700 1 -- 172.21.15.43:0/504158418 learned_addr learned my addr 172.21.15.43:0/504158418 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.788+0000 7f87ff400700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8800076e30 0x7f880019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.788+0000 7f87ffe00700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8800110b90 0x7f88001a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.788+0000 7f87f6a00700 1 -- 172.21.15.43:0/504158418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8800110b90 msgr2=0x7f88001a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.789+0000 7f87f6a00700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8800110b90 0x7f88001a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.789+0000 7f87f6a00700 1 -- 172.21.15.43:0/504158418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8800076e30 msgr2=0x7f880019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.789+0000 7f87f6a00700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8800076e30 0x7f880019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.789+0000 7f87f6a00700 1 -- 172.21.15.43:0/504158418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88001a1e50 con 0x7f8800077780 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.789+0000 7f87ff400700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8800076e30 0x7f880019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:09:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.789+0000 7f87f6a00700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8800077780 0x7f880019d520 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f87e80098e0 tx=0x7f87e8009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.789+0000 7f87fcc00700 1 -- 172.21.15.43:0/504158418 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e8027070 con 0x7f8800077780 2024-05-27T07:09:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.789+0000 7f87fcc00700 1 -- 172.21.15.43:0/504158418 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f87e8022470 con 0x7f8800077780 2024-05-27T07:09:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.789+0000 7f8806400700 1 -- 172.21.15.43:0/504158418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88001a20e0 con 0x7f8800077780 2024-05-27T07:09:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.789+0000 7f87fcc00700 1 -- 172.21.15.43:0/504158418 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e8017650 con 0x7f8800077780 2024-05-27T07:09:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.790+0000 7f8806400700 1 -- 172.21.15.43:0/504158418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88001a25c0 con 0x7f8800077780 2024-05-27T07:09:56.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.791+0000 7f8806400700 1 -- 172.21.15.43:0/504158418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88000512b0 con 0x7f8800077780 2024-05-27T07:09:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.793+0000 7f87fcc00700 1 -- 172.21.15.43:0/504158418 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f87e802a070 con 0x7f8800077780 2024-05-27T07:09:56.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.795+0000 7f87fcc00700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f87e00b2880 0x7f87e00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:09:56.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.795+0000 7f87fcc00700 1 -- 172.21.15.43:0/504158418 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f87e80e39c0 con 0x7f8800077780 2024-05-27T07:09:56.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.795+0000 7f87ff400700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f87e00b2880 0x7f87e00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:09:56.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.796+0000 7f87ff400700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f87e00b2880 0x7f87e00b4d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009cb0 tx=0x7f87f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:09:56.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:56.798+0000 7f87fcc00700 1 -- 172.21.15.43:0/504158418 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f87e8013070 con 0x7f8800077780 2024-05-27T07:09:57.091 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:09:56 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b[66148]: 2024-05-27T07:09:56.841+0000 7f0941000700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-05-27T07:09:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:57.100+0000 7f8806400700 1 -- 172.21.15.43:0/504158418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f880002e5e0 con 0x7f8800077780 2024-05-27T07:09:58.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.064+0000 7f87fcc00700 1 -- 172.21.15.43:0/504158418 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v29) ==== 115+0+0 (secure 0 0 0) 0x7f87e80ab380 con 0x7f8800077780 2024-05-27T07:09:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.069+0000 7f8806400700 1 -- 172.21.15.43:0/504158418 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f87e00b2880 msgr2=0x7f87e00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.069+0000 7f8806400700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f87e00b2880 0x7f87e00b4d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009cb0 tx=0x7f87f0009260 comp rx=0 tx=0).stop 2024-05-27T07:09:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.069+0000 7f8806400700 1 -- 172.21.15.43:0/504158418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8800077780 msgr2=0x7f880019d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:09:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.069+0000 7f8806400700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8800077780 0x7f880019d520 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f87e80098e0 tx=0x7f87e8009550 comp rx=0 tx=0).stop 2024-05-27T07:09:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.070+0000 7f8806400700 1 -- 172.21.15.43:0/504158418 shutdown_connections 2024-05-27T07:09:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.070+0000 7f8806400700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f87e00b2880 0x7f87e00b4d90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.070+0000 7f8806400700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8800110b90 0x7f88001a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.070+0000 7f8806400700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8800076e30 0x7f880019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.070+0000 7f8806400700 1 --2- 172.21.15.43:0/504158418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8800077780 0x7f880019d520 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:09:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.070+0000 7f8806400700 1 -- 172.21.15.43:0/504158418 >> 172.21.15.43:0/504158418 conn(0x7f88000ffeb0 msgr2=0x7f88001005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:09:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.070+0000 7f8806400700 1 -- 172.21.15.43:0/504158418 shutdown_connections 2024-05-27T07:09:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:09:58.071+0000 7f8806400700 1 -- 172.21.15.43:0/504158418 wait complete. 2024-05-27T07:09:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:58 smithi132 ceph-mon[35283]: pgmap v442: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6.0 KiB/s wr, 24 op/s 2024-05-27T07:09:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:58 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/504158418' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:09:58.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:58 smithi043 ceph-mon[31531]: pgmap v442: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6.0 KiB/s wr, 24 op/s 2024-05-27T07:09:58.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:58 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/504158418' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:09:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:58 smithi179 ceph-mon[35888]: pgmap v442: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6.0 KiB/s wr, 24 op/s 2024-05-27T07:09:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:58 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/504158418' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-27T07:09:58.853 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-05-27T07:09:58.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-05-27T07:09:59.330 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:09:59.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:59 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/504158418' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:09:59.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:09:59 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:09:59.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:59 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/504158418' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:09:59.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:09:59 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:09:59.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:59 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/504158418' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-27T07:09:59.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:09:59 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:10:00.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:00 smithi043 ceph-mon[31531]: pgmap v443: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.2 KiB/s wr, 25 op/s 2024-05-27T07:10:00.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:00 smithi043 ceph-mon[31531]: overall HEALTH_OK 2024-05-27T07:10:00.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:00 smithi132 ceph-mon[35283]: pgmap v443: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.2 KiB/s wr, 25 op/s 2024-05-27T07:10:00.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:00 smithi132 ceph-mon[35283]: overall HEALTH_OK 2024-05-27T07:10:00.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:00 smithi179 ceph-mon[35888]: pgmap v443: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.2 KiB/s wr, 25 op/s 2024-05-27T07:10:00.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:00 smithi179 ceph-mon[35888]: overall HEALTH_OK 2024-05-27T07:10:01.414 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:01 smithi043 ceph-mon[31531]: pgmap v444: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 KiB/s wr, 25 op/s 2024-05-27T07:10:01.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:01 smithi179 ceph-mon[35888]: pgmap v444: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 KiB/s wr, 25 op/s 2024-05-27T07:10:01.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:01 smithi132 ceph-mon[35283]: pgmap v444: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 KiB/s wr, 25 op/s 2024-05-27T07:10:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.855+0000 7f9bdbe00700 1 -- 172.21.15.43:0/2646465814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bd4106570 msgr2=0x7f9bd4106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.855+0000 7f9bdbe00700 1 --2- 172.21.15.43:0/2646465814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bd4106570 0x7f9bd4106980 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009990 tx=0x7f9bc0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:10:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.856+0000 7f9bdbe00700 1 -- 172.21.15.43:0/2646465814 shutdown_connections 2024-05-27T07:10:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.856+0000 7f9bdbe00700 1 --2- 172.21.15.43:0/2646465814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9bd4106ec0 0x7f9bd4113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.856+0000 7f9bdbe00700 1 --2- 172.21.15.43:0/2646465814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd410c5b0 0x7f9bd410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.856+0000 7f9bdbe00700 1 --2- 172.21.15.43:0/2646465814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bd4106570 0x7f9bd4106980 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.856+0000 7f9bdbe00700 1 -- 172.21.15.43:0/2646465814 >> 172.21.15.43:0/2646465814 conn(0x7f9bd4102070 msgr2=0x7f9bd41044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.856+0000 7f9bdbe00700 1 -- 172.21.15.43:0/2646465814 shutdown_connections 2024-05-27T07:10:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.856+0000 7f9bdbe00700 1 -- 172.21.15.43:0/2646465814 wait complete. 2024-05-27T07:10:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.857+0000 7f9bdbe00700 1 Processor -- start 2024-05-27T07:10:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.857+0000 7f9bdbe00700 1 -- start start 2024-05-27T07:10:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.857+0000 7f9bdbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9bd4106570 0x7f9bd419cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.858+0000 7f9bdbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bd4106ec0 0x7f9bd419d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.858+0000 7f9bdbe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd410c5b0 0x7f9bd41a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.858+0000 7f9bdbe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd4116540 con 0x7f9bd4106ec0 2024-05-27T07:10:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.858+0000 7f9bdbe00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd41163c0 con 0x7f9bd4106570 2024-05-27T07:10:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.858+0000 7f9bdbe00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd41166c0 con 0x7f9bd410c5b0 2024-05-27T07:10:01.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.858+0000 7f9bd8a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bd4106ec0 0x7f9bd419d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:01.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.858+0000 7f9bd8a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bd4106ec0 0x7f9bd419d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53110/0 (socket says 172.21.15.43:53110) 2024-05-27T07:10:01.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.858+0000 7f9bd8a00700 1 -- 172.21.15.43:0/4082703601 learned_addr learned my addr 172.21.15.43:0/4082703601 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:01.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.859+0000 7f9bd9400700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9bd4106570 0x7f9bd419cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:01.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.859+0000 7f9bd8a00700 1 -- 172.21.15.43:0/4082703601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd410c5b0 msgr2=0x7f9bd41a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.859+0000 7f9bd9e00700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd410c5b0 0x7f9bd41a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.859+0000 7f9bd8a00700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd410c5b0 0x7f9bd41a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.859+0000 7f9bd8a00700 1 -- 172.21.15.43:0/4082703601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9bd4106570 msgr2=0x7f9bd419cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.859+0000 7f9bd8a00700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9bd4106570 0x7f9bd419cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.859+0000 7f9bd8a00700 1 -- 172.21.15.43:0/4082703601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd41a1dc0 con 0x7f9bd4106ec0 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.859+0000 7f9bd9400700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9bd4106570 0x7f9bd419cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.859+0000 7f9bd8a00700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bd4106ec0 0x7f9bd419d490 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0006010 tx=0x7f9bc00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.859+0000 7f9bd2000700 1 -- 172.21.15.43:0/4082703601 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0027070 con 0x7f9bd4106ec0 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.860+0000 7f9bdbe00700 1 -- 172.21.15.43:0/4082703601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd41a2050 con 0x7f9bd4106ec0 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.860+0000 7f9bd2000700 1 -- 172.21.15.43:0/4082703601 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9bc000fcc0 con 0x7f9bd4106ec0 2024-05-27T07:10:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.860+0000 7f9bd2000700 1 -- 172.21.15.43:0/4082703601 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0022b80 con 0x7f9bd4106ec0 2024-05-27T07:10:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.860+0000 7f9bdbe00700 1 -- 172.21.15.43:0/4082703601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd41a99d0 con 0x7f9bd4106ec0 2024-05-27T07:10:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.861+0000 7f9bdbe00700 1 -- 172.21.15.43:0/4082703601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd40512b0 con 0x7f9bd4106ec0 2024-05-27T07:10:01.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.864+0000 7f9bd2000700 1 -- 172.21.15.43:0/4082703601 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9bc002a070 con 0x7f9bd4106ec0 2024-05-27T07:10:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.866+0000 7f9bd2000700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9bbc0b2900 0x7f9bbc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.867+0000 7f9bd2000700 1 -- 172.21.15.43:0/4082703601 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f9bc00e3b00 con 0x7f9bd4106ec0 2024-05-27T07:10:01.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.867+0000 7f9bd9400700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9bbc0b2900 0x7f9bbc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:01.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.868+0000 7f9bd9400700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9bbc0b2900 0x7f9bbc0b4e10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8005fd0 tx=0x7f9bc8005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:01.869+0000 7f9bd2000700 1 -- 172.21.15.43:0/4082703601 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9bc0031080 con 0x7f9bd4106ec0 2024-05-27T07:10:02.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.151+0000 7f9bdbe00700 1 -- 172.21.15.43:0/4082703601 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9bd419e250 con 0x7f9bbc0b2900 2024-05-27T07:10:02.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.228+0000 7f9bd2000700 1 -- 172.21.15.43:0/4082703601 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9bd419e250 con 0x7f9bbc0b2900 2024-05-27T07:10:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 -- 172.21.15.43:0/4082703601 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9bbc0b2900 msgr2=0x7f9bbc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9bbc0b2900 0x7f9bbc0b4e10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8005fd0 tx=0x7f9bc8005ee0 comp rx=0 tx=0).stop 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 -- 172.21.15.43:0/4082703601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bd4106ec0 msgr2=0x7f9bd419d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bd4106ec0 0x7f9bd419d490 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0006010 tx=0x7f9bc00047b0 comp rx=0 tx=0).stop 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 -- 172.21.15.43:0/4082703601 shutdown_connections 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9bbc0b2900 0x7f9bbc0b4e10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd410c5b0 0x7f9bd41a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9bd4106570 0x7f9bd419cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 --2- 172.21.15.43:0/4082703601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bd4106ec0 0x7f9bd419d490 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 -- 172.21.15.43:0/4082703601 >> 172.21.15.43:0/4082703601 conn(0x7f9bd4102070 msgr2=0x7f9bd4104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 -- 172.21.15.43:0/4082703601 shutdown_connections 2024-05-27T07:10:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:02.232+0000 7f9bdbe00700 1 -- 172.21.15.43:0/4082703601 wait complete. 2024-05-27T07:10:02.453 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:10:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:10:02.161+0000 7fc309600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:10:02.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:10:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:10:02.161+0000 7fc309600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:10:02.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:10:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:10:02.161+0000 7fc309600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:10:02.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:10:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:10:02.161+0000 7fc309600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:10:02.454 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:10:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:10:02.161+0000 7fc309600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:10:03.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-27T07:10:03.766 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:04.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:03 smithi043 ceph-mon[31531]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:04.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:03 smithi043 ceph-mon[31531]: pgmap v445: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.9 KiB/s wr, 17 op/s 2024-05-27T07:10:04.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:03 smithi179 ceph-mon[35888]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:04.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:03 smithi179 ceph-mon[35888]: pgmap v445: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.9 KiB/s wr, 17 op/s 2024-05-27T07:10:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:03 smithi132 ceph-mon[35283]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:04.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:03 smithi132 ceph-mon[35283]: pgmap v445: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.9 KiB/s wr, 17 op/s 2024-05-27T07:10:05.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:04 smithi043 ceph-mon[31531]: mgrmap e17: x(active, since 12m), standbys: z, y 2024-05-27T07:10:05.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:04 smithi179 ceph-mon[35888]: mgrmap e17: x(active, since 12m), standbys: z, y 2024-05-27T07:10:05.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:04 smithi132 ceph-mon[35283]: mgrmap e17: x(active, since 12m), standbys: z, y 2024-05-27T07:10:06.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:05 smithi043 ceph-mon[31531]: pgmap v446: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.0 KiB/s wr, 16 op/s 2024-05-27T07:10:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:05 smithi179 ceph-mon[35888]: pgmap v446: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.0 KiB/s wr, 16 op/s 2024-05-27T07:10:06.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.275+0000 7f4660200700 1 -- 172.21.15.43:0/2109680903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46581031c0 msgr2=0x7f4658111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:06.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.275+0000 7f4660200700 1 --2- 172.21.15.43:0/2109680903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46581031c0 0x7f4658111160 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f465000b0a0 tx=0x7f465000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:10:06.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.275+0000 7f4660200700 1 -- 172.21.15.43:0/2109680903 shutdown_connections 2024-05-27T07:10:06.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.275+0000 7f4660200700 1 --2- 172.21.15.43:0/2109680903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46581031c0 0x7f4658111160 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:06.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.275+0000 7f4660200700 1 --2- 172.21.15.43:0/2109680903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46581092f0 0x7f4658109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:06.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.275+0000 7f4660200700 1 --2- 172.21.15.43:0/2109680903 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4658102870 0x7f4658102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:06.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.275+0000 7f4660200700 1 -- 172.21.15.43:0/2109680903 >> 172.21.15.43:0/2109680903 conn(0x7f46580fe6e0 msgr2=0x7f4658100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:06.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.275+0000 7f4660200700 1 -- 172.21.15.43:0/2109680903 shutdown_connections 2024-05-27T07:10:06.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.275+0000 7f4660200700 1 -- 172.21.15.43:0/2109680903 wait complete. 2024-05-27T07:10:06.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.277+0000 7f4660200700 1 Processor -- start 2024-05-27T07:10:06.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.277+0000 7f4660200700 1 -- start start 2024-05-27T07:10:06.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.277+0000 7f4660200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658102870 0x7f465819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:06.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.277+0000 7f4660200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46581031c0 0x7f465819d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:06.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.277+0000 7f4660200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581092f0 0x7f46581a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:06.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.277+0000 7f4660200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4658114440 con 0x7f4658102870 2024-05-27T07:10:06.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.277+0000 7f4660200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46581142c0 con 0x7f46581031c0 2024-05-27T07:10:06.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.277+0000 7f4660200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f46581145c0 con 0x7f46581092f0 2024-05-27T07:10:06.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.278+0000 7f465d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658102870 0x7f465819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:06.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.278+0000 7f465d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658102870 0x7f465819cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54112/0 (socket says 172.21.15.43:54112) 2024-05-27T07:10:06.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.278+0000 7f465d800700 1 -- 172.21.15.43:0/2825910498 learned_addr learned my addr 172.21.15.43:0/2825910498 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:06.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.278+0000 7f465e200700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581092f0 0x7f46581a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:06.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.278+0000 7f465ce00700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46581031c0 0x7f465819d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:06.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.278+0000 7f465d800700 1 -- 172.21.15.43:0/2825910498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581092f0 msgr2=0x7f46581a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.278+0000 7f465d800700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581092f0 0x7f46581a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.278+0000 7f465d800700 1 -- 172.21.15.43:0/2825910498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46581031c0 msgr2=0x7f465819d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.278+0000 7f465d800700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46581031c0 0x7f465819d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.279+0000 7f465d800700 1 -- 172.21.15.43:0/2825910498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46581a1e40 con 0x7f4658102870 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.279+0000 7f465e200700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581092f0 0x7f46581a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.279+0000 7f465d800700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658102870 0x7f465819cf70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f464c00d950 tx=0x7f464c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.279+0000 7f4656000700 1 -- 172.21.15.43:0/2825910498 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c017070 con 0x7f4658102870 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.279+0000 7f4656000700 1 -- 172.21.15.43:0/2825910498 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f464c009730 con 0x7f4658102870 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.279+0000 7f4660200700 1 -- 172.21.15.43:0/2825910498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46581a2130 con 0x7f4658102870 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.279+0000 7f4656000700 1 -- 172.21.15.43:0/2825910498 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c003a00 con 0x7f4658102870 2024-05-27T07:10:06.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.279+0000 7f4660200700 1 -- 172.21.15.43:0/2825910498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46581a9a50 con 0x7f4658102870 2024-05-27T07:10:06.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.281+0000 7f4660200700 1 -- 172.21.15.43:0/2825910498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4638005360 con 0x7f4658102870 2024-05-27T07:10:06.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.283+0000 7f4656000700 1 -- 172.21.15.43:0/2825910498 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f464c003ba0 con 0x7f4658102870 2024-05-27T07:10:06.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.285+0000 7f4656000700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f46400b2910 0x7f46400b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:06.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.285+0000 7f4656000700 1 -- 172.21.15.43:0/2825910498 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f464c0e2a00 con 0x7f4658102870 2024-05-27T07:10:06.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.285+0000 7f465ce00700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f46400b2910 0x7f46400b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:06.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.286+0000 7f465ce00700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f46400b2910 0x7f46400b4e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4658110ee0 tx=0x7f464400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:06.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.288+0000 7f4656000700 1 -- 172.21.15.43:0/2825910498 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f464c020030 con 0x7f4658102870 2024-05-27T07:10:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:05 smithi132 ceph-mon[35283]: pgmap v446: 129 pgs: 129 active+clean; 587 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.0 KiB/s wr, 16 op/s 2024-05-27T07:10:06.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.567+0000 7f4660200700 1 -- 172.21.15.43:0/2825910498 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4638000bf0 con 0x7f46400b2910 2024-05-27T07:10:06.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.633+0000 7f4656000700 1 -- 172.21.15.43:0/2825910498 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4638000bf0 con 0x7f46400b2910 2024-05-27T07:10:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.636+0000 7f4660200700 1 -- 172.21.15.43:0/2825910498 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f46400b2910 msgr2=0x7f46400b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.636+0000 7f4660200700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f46400b2910 0x7f46400b4e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4658110ee0 tx=0x7f464400b410 comp rx=0 tx=0).stop 2024-05-27T07:10:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.637+0000 7f4660200700 1 -- 172.21.15.43:0/2825910498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658102870 msgr2=0x7f465819cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.637+0000 7f4660200700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658102870 0x7f465819cf70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f464c00d950 tx=0x7f464c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:10:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.637+0000 7f4660200700 1 -- 172.21.15.43:0/2825910498 shutdown_connections 2024-05-27T07:10:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.637+0000 7f4660200700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f46400b2910 0x7f46400b4e20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.637+0000 7f4660200700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581092f0 0x7f46581a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:06.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.637+0000 7f4660200700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658102870 0x7f465819cf70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:06.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.637+0000 7f4660200700 1 --2- 172.21.15.43:0/2825910498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46581031c0 0x7f465819d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:06.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.637+0000 7f4660200700 1 -- 172.21.15.43:0/2825910498 >> 172.21.15.43:0/2825910498 conn(0x7f46580fe6e0 msgr2=0x7f465810a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:06.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.637+0000 7f4660200700 1 -- 172.21.15.43:0/2825910498 shutdown_connections 2024-05-27T07:10:06.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:06.637+0000 7f4660200700 1 -- 172.21.15.43:0/2825910498 wait complete. 2024-05-27T07:10:07.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-27T07:10:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:08.165 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:07 smithi043 ceph-mon[31531]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:08.165 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:07 smithi043 ceph-mon[31531]: pgmap v447: 129 pgs: 129 active+clean; 593 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-05-27T07:10:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:07 smithi179 ceph-mon[35888]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:07 smithi179 ceph-mon[35888]: pgmap v447: 129 pgs: 129 active+clean; 593 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-05-27T07:10:08.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:07 smithi132 ceph-mon[35283]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:08.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:07 smithi132 ceph-mon[35283]: pgmap v447: 129 pgs: 129 active+clean; 593 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-05-27T07:10:09.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:09 smithi043 ceph-mon[31531]: pgmap v448: 129 pgs: 129 active+clean; 594 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-05-27T07:10:10.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:09 smithi179 ceph-mon[35888]: pgmap v448: 129 pgs: 129 active+clean; 594 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-05-27T07:10:10.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.309+0000 7f078dc00700 1 -- 172.21.15.43:0/2888989420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0788106ec0 msgr2=0x7f0788113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:10.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.309+0000 7f078dc00700 1 --2- 172.21.15.43:0/2888989420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0788106ec0 0x7f0788113350 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f078000b0a0 tx=0x7f078000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.310+0000 7f078dc00700 1 -- 172.21.15.43:0/2888989420 shutdown_connections 2024-05-27T07:10:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.310+0000 7f078dc00700 1 --2- 172.21.15.43:0/2888989420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0788106ec0 0x7f0788113350 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.310+0000 7f078dc00700 1 --2- 172.21.15.43:0/2888989420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f078810c5b0 0x7f078810c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.310+0000 7f078dc00700 1 --2- 172.21.15.43:0/2888989420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0788106570 0x7f0788106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.310+0000 7f078dc00700 1 -- 172.21.15.43:0/2888989420 >> 172.21.15.43:0/2888989420 conn(0x7f0788102070 msgr2=0x7f07881044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:10.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.310+0000 7f078dc00700 1 -- 172.21.15.43:0/2888989420 shutdown_connections 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.310+0000 7f078dc00700 1 -- 172.21.15.43:0/2888989420 wait complete. 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.312+0000 7f078dc00700 1 Processor -- start 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.312+0000 7f078dc00700 1 -- start start 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.312+0000 7f078dc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0788106570 0x7f078819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.312+0000 7f078dc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0788106ec0 0x7f078819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.312+0000 7f078dc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f078810c5b0 0x7f07881a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.312+0000 7f078dc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0788116590 con 0x7f0788106570 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.312+0000 7f078dc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0788116410 con 0x7f0788106ec0 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.312+0000 7f078dc00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0788116710 con 0x7f078810c5b0 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.313+0000 7f0786a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0788106570 0x7f078819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.313+0000 7f0786a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0788106570 0x7f078819cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54144/0 (socket says 172.21.15.43:54144) 2024-05-27T07:10:10.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.313+0000 7f0786a00700 1 -- 172.21.15.43:0/2588756385 learned_addr learned my addr 172.21.15.43:0/2588756385 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:10.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.313+0000 7f0786000700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0788106ec0 0x7f078819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:10.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.313+0000 7f0786a00700 1 -- 172.21.15.43:0/2588756385 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f078810c5b0 msgr2=0x7f07881a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:10.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.313+0000 7f0786a00700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f078810c5b0 0x7f07881a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.313+0000 7f0786a00700 1 -- 172.21.15.43:0/2588756385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0788106ec0 msgr2=0x7f078819d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:10.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.313+0000 7f0786a00700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0788106ec0 0x7f078819d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.314+0000 7f0786a00700 1 -- 172.21.15.43:0/2588756385 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07881a1df0 con 0x7f0788106570 2024-05-27T07:10:10.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.314+0000 7f0786a00700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0788106570 0x7f078819cf20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f077800e970 tx=0x7f077800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:10.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.314+0000 7f077f400700 1 -- 172.21.15.43:0/2588756385 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077800cbd0 con 0x7f0788106570 2024-05-27T07:10:10.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.314+0000 7f077f400700 1 -- 172.21.15.43:0/2588756385 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f077800cd70 con 0x7f0788106570 2024-05-27T07:10:10.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.314+0000 7f077f400700 1 -- 172.21.15.43:0/2588756385 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778005610 con 0x7f0788106570 2024-05-27T07:10:10.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.314+0000 7f078dc00700 1 -- 172.21.15.43:0/2588756385 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07881a20e0 con 0x7f0788106570 2024-05-27T07:10:10.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.315+0000 7f078dc00700 1 -- 172.21.15.43:0/2588756385 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07881a9a00 con 0x7f0788106570 2024-05-27T07:10:10.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.316+0000 7f078dc00700 1 -- 172.21.15.43:0/2588756385 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07880512b0 con 0x7f0788106570 2024-05-27T07:10:10.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.318+0000 7f077f400700 1 -- 172.21.15.43:0/2588756385 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0778004750 con 0x7f0788106570 2024-05-27T07:10:10.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.321+0000 7f077f400700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f076c0b2990 0x7f076c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:10.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.321+0000 7f077f400700 1 -- 172.21.15.43:0/2588756385 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f07780e2a10 con 0x7f0788106570 2024-05-27T07:10:10.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.321+0000 7f0786000700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f076c0b2990 0x7f076c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:10.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.323+0000 7f0786000700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f076c0b2990 0x7f076c0b4ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0774005950 tx=0x7f07740058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:10.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.326+0000 7f077f400700 1 -- 172.21.15.43:0/2588756385 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f07780aa1e0 con 0x7f0788106570 2024-05-27T07:10:10.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:09 smithi132 ceph-mon[35283]: pgmap v448: 129 pgs: 129 active+clean; 594 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-05-27T07:10:10.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.618+0000 7f078dc00700 1 -- 172.21.15.43:0/2588756385 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f078802e650 con 0x7f076c0b2990 2024-05-27T07:10:10.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.630+0000 7f077f400700 1 -- 172.21.15.43:0/2588756385 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f078802e650 con 0x7f076c0b2990 2024-05-27T07:10:10.629 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/qa/sv_0/40de48eb-1580-4e20-8968-b75b2a70113a 2024-05-27T07:10:10.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 -- 172.21.15.43:0/2588756385 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f076c0b2990 msgr2=0x7f076c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:10.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f076c0b2990 0x7f076c0b4ea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0774005950 tx=0x7f07740058e0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 -- 172.21.15.43:0/2588756385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0788106570 msgr2=0x7f078819cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:10.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0788106570 0x7f078819cf20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f077800e970 tx=0x7f077800ed30 comp rx=0 tx=0).stop 2024-05-27T07:10:10.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 -- 172.21.15.43:0/2588756385 shutdown_connections 2024-05-27T07:10:10.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f076c0b2990 0x7f076c0b4ea0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f078810c5b0 0x7f07881a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0788106570 0x7f078819cf20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 --2- 172.21.15.43:0/2588756385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0788106ec0 0x7f078819d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:10.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 -- 172.21.15.43:0/2588756385 >> 172.21.15.43:0/2588756385 conn(0x7f0788102070 msgr2=0x7f07881047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:10.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 -- 172.21.15.43:0/2588756385 shutdown_connections 2024-05-27T07:10:10.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:10.634+0000 7f078dc00700 1 -- 172.21.15.43:0/2588756385 wait complete. 2024-05-27T07:10:11.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-27T07:10:11.973 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:12.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:11 smithi043 ceph-mon[31531]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:12.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:11 smithi043 ceph-mon[31531]: pgmap v449: 129 pgs: 129 active+clean; 608 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-05-27T07:10:12.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:11 smithi179 ceph-mon[35888]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:12.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:11 smithi179 ceph-mon[35888]: pgmap v449: 129 pgs: 129 active+clean; 608 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-05-27T07:10:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:11 smithi132 ceph-mon[35283]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:11 smithi132 ceph-mon[35283]: pgmap v449: 129 pgs: 129 active+clean; 608 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-05-27T07:10:13.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:13 smithi043 ceph-mon[31531]: pgmap v450: 129 pgs: 129 active+clean; 608 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-05-27T07:10:14.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:13 smithi179 ceph-mon[35888]: pgmap v450: 129 pgs: 129 active+clean; 608 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-05-27T07:10:14.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:13 smithi132 ceph-mon[35283]: pgmap v450: 129 pgs: 129 active+clean; 608 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-05-27T07:10:14.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.357+0000 7fb56ca00700 1 -- 172.21.15.43:0/515292658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb568106610 msgr2=0x7fb568106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:14.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.357+0000 7fb56ca00700 1 --2- 172.21.15.43:0/515292658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb568106610 0x7fb568106a20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb5580099c0 tx=0x7fb558009cd0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.358+0000 7fb56ca00700 1 -- 172.21.15.43:0/515292658 shutdown_connections 2024-05-27T07:10:14.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.358+0000 7fb56ca00700 1 --2- 172.21.15.43:0/515292658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb56810b890 0x7fb5681131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.358+0000 7fb56ca00700 1 --2- 172.21.15.43:0/515292658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb568106610 0x7fb568106a20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.358+0000 7fb56ca00700 1 --2- 172.21.15.43:0/515292658 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56810af40 0x7fb56810b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.358+0000 7fb56ca00700 1 -- 172.21.15.43:0/515292658 >> 172.21.15.43:0/515292658 conn(0x7fb568102020 msgr2=0x7fb568104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:14.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.358+0000 7fb56ca00700 1 -- 172.21.15.43:0/515292658 shutdown_connections 2024-05-27T07:10:14.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.358+0000 7fb56ca00700 1 -- 172.21.15.43:0/515292658 wait complete. 2024-05-27T07:10:14.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.359+0000 7fb56ca00700 1 Processor -- start 2024-05-27T07:10:14.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.359+0000 7fb56ca00700 1 -- start start 2024-05-27T07:10:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.360+0000 7fb56ca00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb568106610 0x7fb568110610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.360+0000 7fb56ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb56810af40 0x7fb568110b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.360+0000 7fb56ca00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56810b890 0x7fb568111090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.360+0000 7fb56ca00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb568116370 con 0x7fb56810af40 2024-05-27T07:10:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.360+0000 7fb56ca00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb5681161f0 con 0x7fb568106610 2024-05-27T07:10:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.360+0000 7fb56ca00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb5681164f0 con 0x7fb56810b890 2024-05-27T07:10:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.361+0000 7fb567400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb568106610 0x7fb568110610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.361+0000 7fb55ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb56810af40 0x7fb568110b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.361+0000 7fb55ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb56810af40 0x7fb568110b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54174/0 (socket says 172.21.15.43:54174) 2024-05-27T07:10:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.361+0000 7fb55ea00700 1 -- 172.21.15.43:0/3856664263 learned_addr learned my addr 172.21.15.43:0/3856664263 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.361+0000 7fb567e00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56810b890 0x7fb568111090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.361+0000 7fb55ea00700 1 -- 172.21.15.43:0/3856664263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56810b890 msgr2=0x7fb568111090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.361+0000 7fb55ea00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56810b890 0x7fb568111090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.361+0000 7fb55ea00700 1 -- 172.21.15.43:0/3856664263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb568106610 msgr2=0x7fb568110610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.361+0000 7fb55ea00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb568106610 0x7fb568110610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.361+0000 7fb55ea00700 1 -- 172.21.15.43:0/3856664263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5681ad240 con 0x7fb56810af40 2024-05-27T07:10:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.362+0000 7fb55ea00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb56810af40 0x7fb568110b50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb55000d980 tx=0x7fb55000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.362+0000 7fb564c00700 1 -- 172.21.15.43:0/3856664263 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb550017070 con 0x7fb56810af40 2024-05-27T07:10:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.363+0000 7fb564c00700 1 -- 172.21.15.43:0/3856664263 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb55000fd20 con 0x7fb56810af40 2024-05-27T07:10:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.363+0000 7fb56ca00700 1 -- 172.21.15.43:0/3856664263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5681ad530 con 0x7fb56810af40 2024-05-27T07:10:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.363+0000 7fb564c00700 1 -- 172.21.15.43:0/3856664263 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55000e3b0 con 0x7fb56810af40 2024-05-27T07:10:14.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.363+0000 7fb56ca00700 1 -- 172.21.15.43:0/3856664263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5681adac0 con 0x7fb56810af40 2024-05-27T07:10:14.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.365+0000 7fb56ca00700 1 -- 172.21.15.43:0/3856664263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb568107ac0 con 0x7fb56810af40 2024-05-27T07:10:14.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.367+0000 7fb564c00700 1 -- 172.21.15.43:0/3856664263 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fb55000f890 con 0x7fb56810af40 2024-05-27T07:10:14.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.368+0000 7fb564c00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb54c0b2990 0x7fb54c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:14.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.369+0000 7fb564c00700 1 -- 172.21.15.43:0/3856664263 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fb5500e21a0 con 0x7fb56810af40 2024-05-27T07:10:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.369+0000 7fb567400700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb54c0b2990 0x7fb54c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.370+0000 7fb567400700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb54c0b2990 0x7fb54c0b4ea0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb558000c00 tx=0x7fb558005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:14.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.374+0000 7fb564c00700 1 -- 172.21.15.43:0/3856664263 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb5500a9970 con 0x7fb56810af40 2024-05-27T07:10:14.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.648+0000 7fb56ca00700 1 -- 172.21.15.43:0/3856664263 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb568063bf0 con 0x7fb54c0b2990 2024-05-27T07:10:14.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.725+0000 7fb564c00700 1 -- 172.21.15.43:0/3856664263 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb568063bf0 con 0x7fb54c0b2990 2024-05-27T07:10:14.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 -- 172.21.15.43:0/3856664263 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb54c0b2990 msgr2=0x7fb54c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:14.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb54c0b2990 0x7fb54c0b4ea0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb558000c00 tx=0x7fb558005fb0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 -- 172.21.15.43:0/3856664263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb56810af40 msgr2=0x7fb568110b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:14.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb56810af40 0x7fb568110b50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb55000d980 tx=0x7fb55000dc90 comp rx=0 tx=0).stop 2024-05-27T07:10:14.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 -- 172.21.15.43:0/3856664263 shutdown_connections 2024-05-27T07:10:14.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb54c0b2990 0x7fb54c0b4ea0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56810b890 0x7fb568111090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb568106610 0x7fb568110610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 --2- 172.21.15.43:0/3856664263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb56810af40 0x7fb568110b50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:14.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 -- 172.21.15.43:0/3856664263 >> 172.21.15.43:0/3856664263 conn(0x7fb568102020 msgr2=0x7fb56810a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:14.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 -- 172.21.15.43:0/3856664263 shutdown_connections 2024-05-27T07:10:14.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:14.729+0000 7fb56ca00700 1 -- 172.21.15.43:0/3856664263 wait complete. 2024-05-27T07:10:15.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-27T07:10:16.185 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:16.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:15 smithi043 ceph-mon[31531]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:16.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:15 smithi043 ceph-mon[31531]: pgmap v451: 129 pgs: 129 active+clean; 608 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-05-27T07:10:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:15 smithi179 ceph-mon[35888]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:16.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:15 smithi179 ceph-mon[35888]: pgmap v451: 129 pgs: 129 active+clean; 608 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-05-27T07:10:16.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:15 smithi132 ceph-mon[35283]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:16.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:15 smithi132 ceph-mon[35283]: pgmap v451: 129 pgs: 129 active+clean; 608 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-05-27T07:10:17.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:17 smithi043 ceph-mon[31531]: pgmap v452: 129 pgs: 129 active+clean; 627 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-05-27T07:10:18.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:17 smithi179 ceph-mon[35888]: pgmap v452: 129 pgs: 129 active+clean; 627 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-05-27T07:10:18.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:17 smithi132 ceph-mon[35283]: pgmap v452: 129 pgs: 129 active+clean; 627 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-05-27T07:10:18.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.635+0000 7f0423000700 1 -- 172.21.15.43:0/4035348272 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f041c106610 msgr2=0x7f041c106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:18.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.635+0000 7f0423000700 1 --2- 172.21.15.43:0/4035348272 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f041c106610 0x7f041c106a20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f04100099c0 tx=0x7f0410009cd0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.636+0000 7f0423000700 1 -- 172.21.15.43:0/4035348272 shutdown_connections 2024-05-27T07:10:18.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.636+0000 7f0423000700 1 --2- 172.21.15.43:0/4035348272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f041c10b890 0x7f041c1131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.636+0000 7f0423000700 1 --2- 172.21.15.43:0/4035348272 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f041c106610 0x7f041c106a20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.636+0000 7f0423000700 1 --2- 172.21.15.43:0/4035348272 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10af40 0x7f041c10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.636+0000 7f0423000700 1 -- 172.21.15.43:0/4035348272 >> 172.21.15.43:0/4035348272 conn(0x7f041c102020 msgr2=0x7f041c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:18.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.636+0000 7f0423000700 1 -- 172.21.15.43:0/4035348272 shutdown_connections 2024-05-27T07:10:18.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.636+0000 7f0423000700 1 -- 172.21.15.43:0/4035348272 wait complete. 2024-05-27T07:10:18.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.638+0000 7f0423000700 1 Processor -- start 2024-05-27T07:10:18.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.638+0000 7f0423000700 1 -- start start 2024-05-27T07:10:18.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.639+0000 7f0423000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f041c106610 0x7f041c19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:18.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.639+0000 7f0423000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f041c10af40 0x7f041c19d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:18.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.639+0000 7f0423000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10b890 0x7f041c1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:18.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.640+0000 7f0423000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f041c1163e0 con 0x7f041c106610 2024-05-27T07:10:18.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.640+0000 7f0423000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f041c116260 con 0x7f041c10b890 2024-05-27T07:10:18.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.640+0000 7f0423000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f041c116560 con 0x7f041c10af40 2024-05-27T07:10:18.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.640+0000 7f0421c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f041c106610 0x7f041c19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:18.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.640+0000 7f0421c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f041c106610 0x7f041c19cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32852/0 (socket says 172.21.15.43:32852) 2024-05-27T07:10:18.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.640+0000 7f0421c00700 1 -- 172.21.15.43:0/1980134625 learned_addr learned my addr 172.21.15.43:0/1980134625 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:18.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.640+0000 7f0421200700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f041c10af40 0x7f041c19d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:18.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.640+0000 7f0422600700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10b890 0x7f041c1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:18.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.641+0000 7f0421c00700 1 -- 172.21.15.43:0/1980134625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f041c10af40 msgr2=0x7f041c19d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:18.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.641+0000 7f0421c00700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f041c10af40 0x7f041c19d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.641+0000 7f0421c00700 1 -- 172.21.15.43:0/1980134625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10b890 msgr2=0x7f041c1a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:18.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.641+0000 7f0421c00700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10b890 0x7f041c1a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.641+0000 7f0421c00700 1 -- 172.21.15.43:0/1980134625 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f041c1a1c60 con 0x7f041c106610 2024-05-27T07:10:18.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.641+0000 7f0421c00700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f041c106610 0x7f041c19cd90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0410000c00 tx=0x7f041000f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:18.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.641+0000 7f041a000700 1 -- 172.21.15.43:0/1980134625 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410027070 con 0x7f041c106610 2024-05-27T07:10:18.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.642+0000 7f041a000700 1 -- 172.21.15.43:0/1980134625 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f041000fda0 con 0x7f041c106610 2024-05-27T07:10:18.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.642+0000 7f0423000700 1 -- 172.21.15.43:0/1980134625 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f041c1a1ef0 con 0x7f041c106610 2024-05-27T07:10:18.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.642+0000 7f041a000700 1 -- 172.21.15.43:0/1980134625 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410022e50 con 0x7f041c106610 2024-05-27T07:10:18.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.642+0000 7f0423000700 1 -- 172.21.15.43:0/1980134625 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f041c1a23d0 con 0x7f041c106610 2024-05-27T07:10:18.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.643+0000 7f0423000700 1 -- 172.21.15.43:0/1980134625 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f041c0512b0 con 0x7f041c106610 2024-05-27T07:10:18.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.647+0000 7f041a000700 1 -- 172.21.15.43:0/1980134625 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f041002a070 con 0x7f041c106610 2024-05-27T07:10:18.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.649+0000 7f041a000700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f04040b2ab0 0x7f04040b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:18.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.649+0000 7f041a000700 1 -- 172.21.15.43:0/1980134625 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f04100e3600 con 0x7f041c106610 2024-05-27T07:10:18.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.650+0000 7f0421200700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f04040b2ab0 0x7f04040b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:18.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.650+0000 7f0421200700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f04040b2ab0 0x7f04040b4fc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f041c105b30 tx=0x7f04080073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:18.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.652+0000 7f041a000700 1 -- 172.21.15.43:0/1980134625 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0410013070 con 0x7f041c106610 2024-05-27T07:10:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.929+0000 7f0423000700 1 -- 172.21.15.43:0/1980134625 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f041c063bf0 con 0x7f04040b2ab0 2024-05-27T07:10:18.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.937+0000 7f041a000700 1 -- 172.21.15.43:0/1980134625 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f041c063bf0 con 0x7f04040b2ab0 2024-05-27T07:10:18.929 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/qa/sv_1/43ec6463-e0ec-4886-a34c-90626615d7b7 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.940+0000 7f0423000700 1 -- 172.21.15.43:0/1980134625 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f04040b2ab0 msgr2=0x7f04040b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.940+0000 7f0423000700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f04040b2ab0 0x7f04040b4fc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f041c105b30 tx=0x7f04080073d0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.940+0000 7f0423000700 1 -- 172.21.15.43:0/1980134625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f041c106610 msgr2=0x7f041c19cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.940+0000 7f0423000700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f041c106610 0x7f041c19cd90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0410000c00 tx=0x7f041000f460 comp rx=0 tx=0).stop 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.941+0000 7f0423000700 1 -- 172.21.15.43:0/1980134625 shutdown_connections 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.941+0000 7f0423000700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f04040b2ab0 0x7f04040b4fc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.941+0000 7f0423000700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f041c10b890 0x7f041c1a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.941+0000 7f0423000700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f041c106610 0x7f041c19cd90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.941+0000 7f0423000700 1 --2- 172.21.15.43:0/1980134625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f041c10af40 0x7f041c19d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.941+0000 7f0423000700 1 -- 172.21.15.43:0/1980134625 >> 172.21.15.43:0/1980134625 conn(0x7f041c102020 msgr2=0x7f041c10a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:18.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.941+0000 7f0423000700 1 -- 172.21.15.43:0/1980134625 shutdown_connections 2024-05-27T07:10:18.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:18.941+0000 7f0423000700 1 -- 172.21.15.43:0/1980134625 wait complete. 2024-05-27T07:10:19.889 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:10:20.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:19 smithi043 ceph-mon[31531]: pgmap v453: 129 pgs: 129 active+clean; 627 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-05-27T07:10:20.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:19 smithi043 ceph-mon[31531]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:20.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:19 smithi179 ceph-mon[35888]: pgmap v453: 129 pgs: 129 active+clean; 627 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-05-27T07:10:20.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:19 smithi179 ceph-mon[35888]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:20.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:19 smithi132 ceph-mon[35283]: pgmap v453: 129 pgs: 129 active+clean; 627 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-05-27T07:10:20.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:19 smithi132 ceph-mon[35283]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:10:20.345 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:22.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:21 smithi043 ceph-mon[31531]: pgmap v454: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-05-27T07:10:22.152 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:21 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:21 smithi179 ceph-mon[35888]: pgmap v454: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-05-27T07:10:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:21 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:22.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:21 smithi132 ceph-mon[35283]: pgmap v454: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-05-27T07:10:22.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:21 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:10:22.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.735+0000 7f791e600700 1 -- 172.21.15.43:0/378838879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7918105e20 msgr2=0x7f791810e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:22.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.735+0000 7f791e600700 1 --2- 172.21.15.43:0/378838879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7918105e20 0x7f791810e370 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7900009990 tx=0x7f7900009ca0 comp rx=0 tx=0).stop 2024-05-27T07:10:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.735+0000 7f791e600700 1 -- 172.21.15.43:0/378838879 shutdown_connections 2024-05-27T07:10:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.735+0000 7f791e600700 1 --2- 172.21.15.43:0/378838879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f791810e900 0x7f7918110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.735+0000 7f791e600700 1 --2- 172.21.15.43:0/378838879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f79181054d0 0x7f79181058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.735+0000 7f791e600700 1 --2- 172.21.15.43:0/378838879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7918105e20 0x7f791810e370 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.735+0000 7f791e600700 1 -- 172.21.15.43:0/378838879 >> 172.21.15.43:0/378838879 conn(0x7f79180fedb0 msgr2=0x7f79181011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.735+0000 7f791e600700 1 -- 172.21.15.43:0/378838879 shutdown_connections 2024-05-27T07:10:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.735+0000 7f791e600700 1 -- 172.21.15.43:0/378838879 wait complete. 2024-05-27T07:10:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.737+0000 7f791e600700 1 Processor -- start 2024-05-27T07:10:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.737+0000 7f791e600700 1 -- start start 2024-05-27T07:10:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.737+0000 7f791e600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79181054d0 0x7f79181031c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.737+0000 7f791e600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7918105e20 0x7f7918103700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.737+0000 7f791e600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f791810e900 0x7f7918103c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.737+0000 7f791e600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f79181142e0 con 0x7f79181054d0 2024-05-27T07:10:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.737+0000 7f791e600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7918114160 con 0x7f7918105e20 2024-05-27T07:10:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.737+0000 7f791e600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7918114460 con 0x7f791810e900 2024-05-27T07:10:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.738+0000 7f7917400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79181054d0 0x7f79181031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.738+0000 7f7917400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79181054d0 0x7f79181031c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32860/0 (socket says 172.21.15.43:32860) 2024-05-27T07:10:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.738+0000 7f7917400700 1 -- 172.21.15.43:0/3539752494 learned_addr learned my addr 172.21.15.43:0/3539752494 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.738+0000 7f7917e00700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f791810e900 0x7f7918103c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.738+0000 7f790ea00700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7918105e20 0x7f7918103700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.738+0000 7f7917400700 1 -- 172.21.15.43:0/3539752494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f791810e900 msgr2=0x7f7918103c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.738+0000 7f7917400700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f791810e900 0x7f7918103c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.738+0000 7f7917400700 1 -- 172.21.15.43:0/3539752494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7918105e20 msgr2=0x7f7918103700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.738+0000 7f7917400700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7918105e20 0x7f7918103700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.738+0000 7f7917400700 1 -- 172.21.15.43:0/3539752494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79181a9090 con 0x7f79181054d0 2024-05-27T07:10:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.739+0000 7f7917400700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79181054d0 0x7f79181031c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f790800d950 tx=0x7f790800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.739+0000 7f7914c00700 1 -- 172.21.15.43:0/3539752494 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7908016070 con 0x7f79181054d0 2024-05-27T07:10:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.739+0000 7f7914c00700 1 -- 172.21.15.43:0/3539752494 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7908010460 con 0x7f79181054d0 2024-05-27T07:10:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.739+0000 7f791e600700 1 -- 172.21.15.43:0/3539752494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79181a9380 con 0x7f79181054d0 2024-05-27T07:10:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.739+0000 7f7914c00700 1 -- 172.21.15.43:0/3539752494 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790800f620 con 0x7f79181054d0 2024-05-27T07:10:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.739+0000 7f791e600700 1 -- 172.21.15.43:0/3539752494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79181a98c0 con 0x7f79181054d0 2024-05-27T07:10:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.740+0000 7f791e600700 1 -- 172.21.15.43:0/3539752494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79180512b0 con 0x7f79181054d0 2024-05-27T07:10:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.743+0000 7f7914c00700 1 -- 172.21.15.43:0/3539752494 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7908019070 con 0x7f79181054d0 2024-05-27T07:10:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.744+0000 7f7914c00700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f78fc0b2910 0x7f78fc0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.744+0000 7f7914c00700 1 -- 172.21.15.43:0/3539752494 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f79080e23a0 con 0x7f79181054d0 2024-05-27T07:10:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.745+0000 7f790ea00700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f78fc0b2910 0x7f78fc0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.745+0000 7f790ea00700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f78fc0b2910 0x7f78fc0b4e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7900006010 tx=0x7f790000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:22.747+0000 7f7914c00700 1 -- 172.21.15.43:0/3539752494 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f79080a9b70 con 0x7f79181054d0 2024-05-27T07:10:23.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.035+0000 7f791e600700 1 -- 172.21.15.43:0/3539752494 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f791802e5e0 con 0x7f79181054d0 2024-05-27T07:10:23.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.036+0000 7f7914c00700 1 -- 172.21.15.43:0/3539752494 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7f79080a9470 con 0x7f79181054d0 2024-05-27T07:10:23.028 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:10:23.028 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:09:57.102577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[24484,34403,34424,34448,34484]},"id":1}]} 2024-05-27T07:10:23.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.039+0000 7f791e600700 1 -- 172.21.15.43:0/3539752494 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f78fc0b2910 msgr2=0x7f78fc0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:23.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.039+0000 7f791e600700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f78fc0b2910 0x7f78fc0b4e20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7900006010 tx=0x7f790000b540 comp rx=0 tx=0).stop 2024-05-27T07:10:23.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.039+0000 7f791e600700 1 -- 172.21.15.43:0/3539752494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79181054d0 msgr2=0x7f79181031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:23.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.039+0000 7f791e600700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79181054d0 0x7f79181031c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f790800d950 tx=0x7f790800dd10 comp rx=0 tx=0).stop 2024-05-27T07:10:23.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.039+0000 7f791e600700 1 -- 172.21.15.43:0/3539752494 shutdown_connections 2024-05-27T07:10:23.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.040+0000 7f791e600700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f78fc0b2910 0x7f78fc0b4e20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:23.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.040+0000 7f791e600700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f791810e900 0x7f7918103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:23.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.040+0000 7f791e600700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79181054d0 0x7f79181031c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:23.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.040+0000 7f791e600700 1 --2- 172.21.15.43:0/3539752494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7918105e20 0x7f7918103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:23.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.040+0000 7f791e600700 1 -- 172.21.15.43:0/3539752494 >> 172.21.15.43:0/3539752494 conn(0x7f79180fedb0 msgr2=0x7f79180ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:23.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.040+0000 7f791e600700 1 -- 172.21.15.43:0/3539752494 shutdown_connections 2024-05-27T07:10:23.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:23.040+0000 7f791e600700 1 -- 172.21.15.43:0/3539752494 wait complete. 2024-05-27T07:10:23.032 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-05-27T07:10:24.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:23 smithi043 ceph-mon[31531]: pgmap v455: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-05-27T07:10:24.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:23 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3539752494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:10:24.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:23 smithi179 ceph-mon[35888]: pgmap v455: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-05-27T07:10:24.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:23 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3539752494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:10:24.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:10:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:23 smithi132 ceph-mon[35283]: pgmap v455: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-05-27T07:10:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:23 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3539752494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:10:24.734 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:25.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:24 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:24 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:24 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:24 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:24 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:24 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:24 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:24 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:24 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:24 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:24 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:25.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:24 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:26.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:26 smithi043 ceph-mon[31531]: pgmap v456: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-05-27T07:10:26.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:26.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:26 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:26.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:26 smithi132 ceph-mon[35283]: pgmap v456: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-05-27T07:10:26.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:26.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:26 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:26 smithi179 ceph-mon[35888]: pgmap v456: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-05-27T07:10:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:26 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:27.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.033+0000 7f9e53c00700 1 -- 172.21.15.43:0/3192561817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e4c109840 msgr2=0x7f9e4c1111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:27.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.033+0000 7f9e53c00700 1 --2- 172.21.15.43:0/3192561817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e4c109840 0x7f9e4c1111a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400b0d0 tx=0x7f9e4400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.033+0000 7f9e53c00700 1 -- 172.21.15.43:0/3192561817 shutdown_connections 2024-05-27T07:10:27.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.033+0000 7f9e53c00700 1 --2- 172.21.15.43:0/3192561817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e4c109840 0x7f9e4c1111a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.033+0000 7f9e53c00700 1 --2- 172.21.15.43:0/3192561817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e4c1045c0 0x7f9e4c1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.033+0000 7f9e53c00700 1 --2- 172.21.15.43:0/3192561817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c108ef0 0x7f9e4c109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.033+0000 7f9e53c00700 1 -- 172.21.15.43:0/3192561817 >> 172.21.15.43:0/3192561817 conn(0x7f9e4c0ffe90 msgr2=0x7f9e4c1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:27.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.033+0000 7f9e53c00700 1 -- 172.21.15.43:0/3192561817 shutdown_connections 2024-05-27T07:10:27.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.033+0000 7f9e53c00700 1 -- 172.21.15.43:0/3192561817 wait complete. 2024-05-27T07:10:27.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.035+0000 7f9e53c00700 1 Processor -- start 2024-05-27T07:10:27.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.035+0000 7f9e53c00700 1 -- start start 2024-05-27T07:10:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.035+0000 7f9e53c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c1045c0 0x7f9e4c076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.035+0000 7f9e53c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e4c108ef0 0x7f9e4c077100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.035+0000 7f9e53c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e4c109840 0x7f9e4c077640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e53c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c114370 con 0x7f9e4c108ef0 2024-05-27T07:10:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e53c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c1141f0 con 0x7f9e4c109840 2024-05-27T07:10:27.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e53c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4c1144f0 con 0x7f9e4c1045c0 2024-05-27T07:10:27.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e4be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e4c108ef0 0x7f9e4c077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:27.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e4be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e4c108ef0 0x7f9e4c077100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38402/0 (socket says 172.21.15.43:38402) 2024-05-27T07:10:27.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e4be00700 1 -- 172.21.15.43:0/2303409126 learned_addr learned my addr 172.21.15.43:0/2303409126 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:27.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e51200700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c1045c0 0x7f9e4c076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:27.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e51c00700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e4c109840 0x7f9e4c077640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e4be00700 1 -- 172.21.15.43:0/2303409126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c1045c0 msgr2=0x7f9e4c076bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e4be00700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c1045c0 0x7f9e4c076bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e4be00700 1 -- 172.21.15.43:0/2303409126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e4c109840 msgr2=0x7f9e4c077640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e4be00700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e4c109840 0x7f9e4c077640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.036+0000 7f9e4be00700 1 -- 172.21.15.43:0/2303409126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4c1a8f90 con 0x7f9e4c108ef0 2024-05-27T07:10:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.037+0000 7f9e4be00700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e4c108ef0 0x7f9e4c077100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800c9e0 tx=0x7f9e3800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.037+0000 7f9e49600700 1 -- 172.21.15.43:0/2303409126 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e38007aa0 con 0x7f9e4c108ef0 2024-05-27T07:10:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.037+0000 7f9e49600700 1 -- 172.21.15.43:0/2303409126 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9e38007c40 con 0x7f9e4c108ef0 2024-05-27T07:10:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.037+0000 7f9e53c00700 1 -- 172.21.15.43:0/2303409126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e4c1a9280 con 0x7f9e4c108ef0 2024-05-27T07:10:27.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.037+0000 7f9e49600700 1 -- 172.21.15.43:0/2303409126 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e38018660 con 0x7f9e4c108ef0 2024-05-27T07:10:27.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.038+0000 7f9e53c00700 1 -- 172.21.15.43:0/2303409126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e4c1a97c0 con 0x7f9e4c108ef0 2024-05-27T07:10:27.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.039+0000 7f9e53c00700 1 -- 172.21.15.43:0/2303409126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e4c0512b0 con 0x7f9e4c108ef0 2024-05-27T07:10:27.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.042+0000 7f9e49600700 1 -- 172.21.15.43:0/2303409126 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9e3800f450 con 0x7f9e4c108ef0 2024-05-27T07:10:27.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.043+0000 7f9e49600700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9e340b2910 0x7f9e340b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:27.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.043+0000 7f9e49600700 1 -- 172.21.15.43:0/2303409126 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f9e380e26b0 con 0x7f9e4c108ef0 2024-05-27T07:10:27.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.044+0000 7f9e51200700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9e340b2910 0x7f9e340b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:27.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.044+0000 7f9e51200700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9e340b2910 0x7f9e340b4e20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9e4000ac50 tx=0x7f9e4000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:27.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.048+0000 7f9e49600700 1 -- 172.21.15.43:0/2303409126 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9e38014030 con 0x7f9e4c108ef0 2024-05-27T07:10:27.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.309+0000 7f9e53c00700 1 -- 172.21.15.43:0/2303409126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e4c02e5e0 con 0x7f9e4c108ef0 2024-05-27T07:10:27.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.313+0000 7f9e49600700 1 -- 172.21.15.43:0/2303409126 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24748 (secure 0 0 0) 0x7f9e380a9e80 con 0x7f9e4c108ef0 2024-05-27T07:10:27.304 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:10:27.305 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":90,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:09:54.019617+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:10:27.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.316+0000 7f9e53c00700 1 -- 172.21.15.43:0/2303409126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9e340b2910 msgr2=0x7f9e340b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.316+0000 7f9e53c00700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9e340b2910 0x7f9e340b4e20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9e4000ac50 tx=0x7f9e4000a380 comp rx=0 tx=0).stop 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.316+0000 7f9e53c00700 1 -- 172.21.15.43:0/2303409126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e4c108ef0 msgr2=0x7f9e4c077100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.316+0000 7f9e53c00700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e4c108ef0 0x7f9e4c077100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800c9e0 tx=0x7f9e3800ccf0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.316+0000 7f9e53c00700 1 -- 172.21.15.43:0/2303409126 shutdown_connections 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.316+0000 7f9e53c00700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f9e340b2910 0x7f9e340b4e20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.316+0000 7f9e53c00700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e4c109840 0x7f9e4c077640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.316+0000 7f9e53c00700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c1045c0 0x7f9e4c076bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.316+0000 7f9e53c00700 1 --2- 172.21.15.43:0/2303409126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e4c108ef0 0x7f9e4c077100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.316+0000 7f9e53c00700 1 -- 172.21.15.43:0/2303409126 >> 172.21.15.43:0/2303409126 conn(0x7f9e4c0ffe90 msgr2=0x7f9e4c108630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.317+0000 7f9e53c00700 1 -- 172.21.15.43:0/2303409126 shutdown_connections 2024-05-27T07:10:27.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:27.317+0000 7f9e53c00700 1 -- 172.21.15.43:0/2303409126 wait complete. 2024-05-27T07:10:28.126 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T07:10:28.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:10:28.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:28 smithi132 ceph-mon[35283]: pgmap v457: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.4 KiB/s wr, 12 op/s 2024-05-27T07:10:28.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:28 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:28.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:28 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:28.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:28 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:28.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:28 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2303409126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:10:28.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:28 smithi043 ceph-mon[31531]: pgmap v457: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.4 KiB/s wr, 12 op/s 2024-05-27T07:10:28.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:28 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:28.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:28 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:28.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:28 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:28.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:28 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2303409126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:10:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:28 smithi179 ceph-mon[35888]: pgmap v457: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.4 KiB/s wr, 12 op/s 2024-05-27T07:10:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:28 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:10:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:28 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:10:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:28 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:10:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:28 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2303409126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:10:28.625 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:30.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:30 smithi132 ceph-mon[35283]: pgmap v458: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-05-27T07:10:30.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:30 smithi043 ceph-mon[31531]: pgmap v458: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-05-27T07:10:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:30 smithi179 ceph-mon[35888]: pgmap v458: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-05-27T07:10:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.092+0000 7feea0000700 1 -- 172.21.15.43:0/2392662667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9810a950 msgr2=0x7fee9810ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.092+0000 7feea0000700 1 --2- 172.21.15.43:0/2392662667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9810a950 0x7fee9810ad60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fee84009990 tx=0x7fee84009ca0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.092+0000 7feea0000700 1 -- 172.21.15.43:0/2392662667 shutdown_connections 2024-05-27T07:10:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.092+0000 7feea0000700 1 --2- 172.21.15.43:0/2392662667 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9810b2a0 0x7fee98117820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.092+0000 7feea0000700 1 --2- 172.21.15.43:0/2392662667 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee98110980 0x7fee98110d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.092+0000 7feea0000700 1 --2- 172.21.15.43:0/2392662667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9810a950 0x7fee9810ad60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.092+0000 7feea0000700 1 -- 172.21.15.43:0/2392662667 >> 172.21.15.43:0/2392662667 conn(0x7fee98076d20 msgr2=0x7fee98077130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.092+0000 7feea0000700 1 -- 172.21.15.43:0/2392662667 shutdown_connections 2024-05-27T07:10:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.092+0000 7feea0000700 1 -- 172.21.15.43:0/2392662667 wait complete. 2024-05-27T07:10:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.094+0000 7feea0000700 1 Processor -- start 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.094+0000 7feea0000700 1 -- start start 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.094+0000 7feea0000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9810a950 0x7fee981a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.094+0000 7feea0000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9810b2a0 0x7fee981a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.094+0000 7feea0000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee98110980 0x7fee981a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.094+0000 7feea0000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fee9811a9a0 con 0x7fee9810a950 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.094+0000 7feea0000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fee9811a820 con 0x7fee98110980 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.094+0000 7feea0000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fee9811ab20 con 0x7fee9810b2a0 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.095+0000 7fee9d600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9810a950 0x7fee981a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.095+0000 7fee9d600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9810a950 0x7fee981a13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38408/0 (socket says 172.21.15.43:38408) 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.095+0000 7fee9d600700 1 -- 172.21.15.43:0/291513456 learned_addr learned my addr 172.21.15.43:0/291513456 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.095+0000 7fee9e000700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee98110980 0x7fee981a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.095+0000 7fee9cc00700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9810b2a0 0x7fee981a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.096+0000 7fee9d600700 1 -- 172.21.15.43:0/291513456 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9810b2a0 msgr2=0x7fee981a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.096+0000 7fee9d600700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9810b2a0 0x7fee981a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.096+0000 7fee9d600700 1 -- 172.21.15.43:0/291513456 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee98110980 msgr2=0x7fee981a5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.096+0000 7fee9d600700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee98110980 0x7fee981a5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.096+0000 7fee9d600700 1 -- 172.21.15.43:0/291513456 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee981a6280 con 0x7fee9810a950 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.096+0000 7fee9d600700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9810a950 0x7fee981a13b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00e970 tx=0x7fee8c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.096+0000 7fee96000700 1 -- 172.21.15.43:0/291513456 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c004a30 con 0x7fee9810a950 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.097+0000 7feea0000700 1 -- 172.21.15.43:0/291513456 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee981a6570 con 0x7fee9810a950 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.097+0000 7fee96000700 1 -- 172.21.15.43:0/291513456 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee8c00cc70 con 0x7fee9810a950 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.097+0000 7fee96000700 1 -- 172.21.15.43:0/291513456 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c005520 con 0x7fee9810a950 2024-05-27T07:10:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.097+0000 7feea0000700 1 -- 172.21.15.43:0/291513456 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee981ade90 con 0x7fee9810a950 2024-05-27T07:10:31.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.098+0000 7feea0000700 1 -- 172.21.15.43:0/291513456 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee980512b0 con 0x7fee9810a950 2024-05-27T07:10:31.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.100+0000 7fee96000700 1 -- 172.21.15.43:0/291513456 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fee8c00c7e0 con 0x7fee9810a950 2024-05-27T07:10:31.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.102+0000 7fee96000700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fee800b2910 0x7fee800b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:31.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.102+0000 7fee96000700 1 -- 172.21.15.43:0/291513456 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fee8c0e2890 con 0x7fee9810a950 2024-05-27T07:10:31.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.102+0000 7fee9cc00700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fee800b2910 0x7fee800b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:31.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.104+0000 7fee9cc00700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fee800b2910 0x7fee800b4e20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee8400b2e0 tx=0x7fee84005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:31.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.107+0000 7fee96000700 1 -- 172.21.15.43:0/291513456 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fee8c00ce20 con 0x7fee9810a950 2024-05-27T07:10:31.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.338+0000 7feea0000700 1 -- 172.21.15.43:0/291513456 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fee981a2710 con 0x7fee800b2910 2024-05-27T07:10:31.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.352+0000 7fee96000700 1 -- 172.21.15.43:0/291513456 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340440 (secure 0 0 0) 0x7fee8407a1f0 con 0x7fee800b2910 2024-05-27T07:10:31.351 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:10:31.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 -- 172.21.15.43:0/291513456 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fee800b2910 msgr2=0x7fee800b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fee800b2910 0x7fee800b4e20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee8400b2e0 tx=0x7fee84005fb0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 -- 172.21.15.43:0/291513456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9810a950 msgr2=0x7fee981a13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9810a950 0x7fee981a13b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00e970 tx=0x7fee8c00ec80 comp rx=0 tx=0).stop 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 -- 172.21.15.43:0/291513456 shutdown_connections 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fee800b2910 0x7fee800b4e20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fee98110980 0x7fee981a5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee9810a950 0x7fee981a13b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 --2- 172.21.15.43:0/291513456 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9810b2a0 0x7fee981a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 -- 172.21.15.43:0/291513456 >> 172.21.15.43:0/291513456 conn(0x7fee98076d20 msgr2=0x7fee98108930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 -- 172.21.15.43:0/291513456 shutdown_connections 2024-05-27T07:10:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:31.361+0000 7feea0000700 1 -- 172.21.15.43:0/291513456 wait complete. 2024-05-27T07:10:31.355 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-05-27T07:10:32.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:32 smithi043 ceph-mon[31531]: pgmap v459: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-05-27T07:10:32.322 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-05-27T07:10:30.782427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661942,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":480,"num_read_kb":350,"num_write":374,"num_write_kb":2295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953452,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":308,"ondisk_log_size":308,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367672,"kb_used_data":14336,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574152,"statfs":{"total":1151940427776,"available":1151563931648,"internally_reserved":0,"allocated":14680064,"data_stored":8178440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19657,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":129,"num_read_kb":100,"num_write":11,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006934"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188611+0000","last_change":"2024-05-27T07:08:22.053540+0000","last_active":"2024-05-27T07:09:54.188611+0000","last_peered":"2024-05-27T07:09:54.188611+0000","last_clean":"2024-05-27T07:09:54.188611+0000","last_became_active":"2024-05-27T07:08:22.051872+0000","last_became_peered":"2024-05-27T07:08:22.051872+0000","last_unstale":"2024-05-27T07:09:54.188611+0000","last_undegraded":"2024-05-27T07:09:54.188611+0000","last_fullsized":"2024-05-27T07:09:54.188611+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:02.787957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185542+0000","last_change":"2024-05-27T07:08:22.049341+0000","last_active":"2024-05-27T07:09:54.185542+0000","last_peered":"2024-05-27T07:09:54.185542+0000","last_clean":"2024-05-27T07:09:54.185542+0000","last_became_active":"2024-05-27T07:08:22.048905+0000","last_became_peered":"2024-05-27T07:08:22.048905+0000","last_unstale":"2024-05-27T07:09:54.185542+0000","last_undegraded":"2024-05-27T07:09:54.185542+0000","last_fullsized":"2024-05-27T07:09:54.185542+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:27.271336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.184788+0000","last_change":"2024-05-27T07:08:22.053925+0000","last_active":"2024-05-27T07:09:54.184788+0000","last_peered":"2024-05-27T07:09:54.184788+0000","last_clean":"2024-05-27T07:09:54.184788+0000","last_became_active":"2024-05-27T07:08:22.053409+0000","last_became_peered":"2024-05-27T07:08:22.053409+0000","last_unstale":"2024-05-27T07:09:54.184788+0000","last_undegraded":"2024-05-27T07:09:54.184788+0000","last_fullsized":"2024-05-27T07:09:54.184788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:07:55.439966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079038+0000","last_change":"2024-05-27T07:08:22.048214+0000","last_active":"2024-05-27T07:09:54.079038+0000","last_peered":"2024-05-27T07:09:54.079038+0000","last_clean":"2024-05-27T07:09:54.079038+0000","last_became_active":"2024-05-27T07:08:22.047791+0000","last_became_peered":"2024-05-27T07:08:22.047791+0000","last_unstale":"2024-05-27T07:09:54.079038+0000","last_undegraded":"2024-05-27T07:09:54.079038+0000","last_fullsized":"2024-05-27T07:09:54.079038+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:23:56.381008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189219+0000","last_change":"2024-05-27T07:08:22.037883+0000","last_active":"2024-05-27T07:09:54.189219+0000","last_peered":"2024-05-27T07:09:54.189219+0000","last_clean":"2024-05-27T07:09:54.189219+0000","last_became_active":"2024-05-27T07:08:22.037089+0000","last_became_peered":"2024-05-27T07:08:22.037089+0000","last_unstale":"2024-05-27T07:09:54.189219+0000","last_undegraded":"2024-05-27T07:09:54.189219+0000","last_fullsized":"2024-05-27T07:09:54.189219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:21.929086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062958+0000","last_change":"2024-05-27T07:08:22.061833+0000","last_active":"2024-05-27T07:09:54.062958+0000","last_peered":"2024-05-27T07:09:54.062958+0000","last_clean":"2024-05-27T07:09:54.062958+0000","last_became_active":"2024-05-27T07:08:22.057692+0000","last_became_peered":"2024-05-27T07:08:22.057692+0000","last_unstale":"2024-05-27T07:09:54.062958+0000","last_undegraded":"2024-05-27T07:09:54.062958+0000","last_fullsized":"2024-05-27T07:09:54.062958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:38:33.656872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044241+0000","last_change":"2024-05-27T07:08:22.047880+0000","last_active":"2024-05-27T07:09:54.044241+0000","last_peered":"2024-05-27T07:09:54.044241+0000","last_clean":"2024-05-27T07:09:54.044241+0000","last_became_active":"2024-05-27T07:08:22.047482+0000","last_became_peered":"2024-05-27T07:08:22.047482+0000","last_unstale":"2024-05-27T07:09:54.044241+0000","last_undegraded":"2024-05-27T07:09:54.044241+0000","last_fullsized":"2024-05-27T07:09:54.044241+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:07.751588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060141+0000","last_change":"2024-05-27T07:08:22.055970+0000","last_active":"2024-05-27T07:09:54.060141+0000","last_peered":"2024-05-27T07:09:54.060141+0000","last_clean":"2024-05-27T07:09:54.060141+0000","last_became_active":"2024-05-27T07:08:22.055591+0000","last_became_peered":"2024-05-27T07:08:22.055591+0000","last_unstale":"2024-05-27T07:09:54.060141+0000","last_undegraded":"2024-05-27T07:09:54.060141+0000","last_fullsized":"2024-05-27T07:09:54.060141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:58:46.571473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188998+0000","last_change":"2024-05-27T07:08:22.059929+0000","last_active":"2024-05-27T07:09:54.188998+0000","last_peered":"2024-05-27T07:09:54.188998+0000","last_clean":"2024-05-27T07:09:54.188998+0000","last_became_active":"2024-05-27T07:08:22.059118+0000","last_became_peered":"2024-05-27T07:08:22.059118+0000","last_unstale":"2024-05-27T07:09:54.188998+0000","last_undegraded":"2024-05-27T07:09:54.188998+0000","last_fullsized":"2024-05-27T07:09:54.188998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:29.231220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063952+0000","last_change":"2024-05-27T07:08:53.979269+0000","last_active":"2024-05-27T07:09:54.063952+0000","last_peered":"2024-05-27T07:09:54.063952+0000","last_clean":"2024-05-27T07:09:54.063952+0000","last_became_active":"2024-05-27T07:08:53.978433+0000","last_became_peered":"2024-05-27T07:08:53.978433+0000","last_unstale":"2024-05-27T07:09:54.063952+0000","last_undegraded":"2024-05-27T07:09:54.063952+0000","last_fullsized":"2024-05-27T07:09:54.063952+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:57:37.600911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.046188+0000","last_change":"2024-05-27T07:08:22.053499+0000","last_active":"2024-05-27T07:09:54.046188+0000","last_peered":"2024-05-27T07:09:54.046188+0000","last_clean":"2024-05-27T07:09:54.046188+0000","last_became_active":"2024-05-27T07:08:22.052428+0000","last_became_peered":"2024-05-27T07:08:22.052428+0000","last_unstale":"2024-05-27T07:09:54.046188+0000","last_undegraded":"2024-05-27T07:09:54.046188+0000","last_fullsized":"2024-05-27T07:09:54.046188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:04.352262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079297+0000","last_change":"2024-05-27T07:08:22.058882+0000","last_active":"2024-05-27T07:09:54.079297+0000","last_peered":"2024-05-27T07:09:54.079297+0000","last_clean":"2024-05-27T07:09:54.079297+0000","last_became_active":"2024-05-27T07:08:22.057874+0000","last_became_peered":"2024-05-27T07:08:22.057874+0000","last_unstale":"2024-05-27T07:09:54.079297+0000","last_undegraded":"2024-05-27T07:09:54.079297+0000","last_fullsized":"2024-05-27T07:09:54.079297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:52:11.289488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041675+0000","last_change":"2024-05-27T07:08:22.054315+0000","last_active":"2024-05-27T07:09:54.041675+0000","last_peered":"2024-05-27T07:09:54.041675+0000","last_clean":"2024-05-27T07:09:54.041675+0000","last_became_active":"2024-05-27T07:08:22.053944+0000","last_became_peered":"2024-05-27T07:08:22.053944+0000","last_unstale":"2024-05-27T07:09:54.041675+0000","last_undegraded":"2024-05-27T07:09:54.041675+0000","last_fullsized":"2024-05-27T07:09:54.041675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:43:06.720147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.184713+0000","last_change":"2024-05-27T07:08:22.054649+0000","last_active":"2024-05-27T07:09:54.184713+0000","last_peered":"2024-05-27T07:09:54.184713+0000","last_clean":"2024-05-27T07:09:54.184713+0000","last_became_active":"2024-05-27T07:08:22.051834+0000","last_became_peered":"2024-05-27T07:08:22.051834+0000","last_unstale":"2024-05-27T07:09:54.184713+0000","last_undegraded":"2024-05-27T07:09:54.184713+0000","last_fullsized":"2024-05-27T07:09:54.184713+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:16:56.893458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.184870+0000","last_change":"2024-05-27T07:08:22.052231+0000","last_active":"2024-05-27T07:09:54.184870+0000","last_peered":"2024-05-27T07:09:54.184870+0000","last_clean":"2024-05-27T07:09:54.184870+0000","last_became_active":"2024-05-27T07:08:22.051495+0000","last_became_peered":"2024-05-27T07:08:22.051495+0000","last_unstale":"2024-05-27T07:09:54.184870+0000","last_undegraded":"2024-05-27T07:09:54.184870+0000","last_fullsized":"2024-05-27T07:09:54.184870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:30:21.036874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187151+0000","last_change":"2024-05-27T07:08:22.056847+0000","last_active":"2024-05-27T07:09:54.187151+0000","last_peered":"2024-05-27T07:09:54.187151+0000","last_clean":"2024-05-27T07:09:54.187151+0000","last_became_active":"2024-05-27T07:08:22.056127+0000","last_became_peered":"2024-05-27T07:08:22.056127+0000","last_unstale":"2024-05-27T07:09:54.187151+0000","last_undegraded":"2024-05-27T07:09:54.187151+0000","last_fullsized":"2024-05-27T07:09:54.187151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:04.757452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188092+0000","last_change":"2024-05-27T07:08:22.058078+0000","last_active":"2024-05-27T07:09:54.188092+0000","last_peered":"2024-05-27T07:09:54.188092+0000","last_clean":"2024-05-27T07:09:54.188092+0000","last_became_active":"2024-05-27T07:08:22.056886+0000","last_became_peered":"2024-05-27T07:08:22.056886+0000","last_unstale":"2024-05-27T07:09:54.188092+0000","last_undegraded":"2024-05-27T07:09:54.188092+0000","last_fullsized":"2024-05-27T07:09:54.188092+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.104572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.048468+0000","last_change":"2024-05-27T07:08:22.051492+0000","last_active":"2024-05-27T07:09:54.048468+0000","last_peered":"2024-05-27T07:09:54.048468+0000","last_clean":"2024-05-27T07:09:54.048468+0000","last_became_active":"2024-05-27T07:08:22.051091+0000","last_became_peered":"2024-05-27T07:08:22.051091+0000","last_unstale":"2024-05-27T07:09:54.048468+0000","last_undegraded":"2024-05-27T07:09:54.048468+0000","last_fullsized":"2024-05-27T07:09:54.048468+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:47.816312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.071314+0000","last_change":"2024-05-27T07:08:22.050714+0000","last_active":"2024-05-27T07:09:54.071314+0000","last_peered":"2024-05-27T07:09:54.071314+0000","last_clean":"2024-05-27T07:09:54.071314+0000","last_became_active":"2024-05-27T07:08:22.050322+0000","last_became_peered":"2024-05-27T07:08:22.050322+0000","last_unstale":"2024-05-27T07:09:54.071314+0000","last_undegraded":"2024-05-27T07:09:54.071314+0000","last_fullsized":"2024-05-27T07:09:54.071314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:01.867467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070275+0000","last_change":"2024-05-27T07:08:36.859916+0000","last_active":"2024-05-27T07:09:54.070275+0000","last_peered":"2024-05-27T07:09:54.070275+0000","last_clean":"2024-05-27T07:09:54.070275+0000","last_became_active":"2024-05-27T07:08:36.859429+0000","last_became_peered":"2024-05-27T07:08:36.859429+0000","last_unstale":"2024-05-27T07:09:54.070275+0000","last_undegraded":"2024-05-27T07:09:54.070275+0000","last_fullsized":"2024-05-27T07:09:54.070275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:14.026604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":94,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:26.151120+0000","last_change":"2024-05-27T07:08:18.008912+0000","last_active":"2024-05-27T07:10:26.151120+0000","last_peered":"2024-05-27T07:10:26.151120+0000","last_clean":"2024-05-27T07:10:26.151120+0000","last_became_active":"2024-05-27T07:08:18.007603+0000","last_became_peered":"2024-05-27T07:08:18.007603+0000","last_unstale":"2024-05-27T07:10:26.151120+0000","last_undegraded":"2024-05-27T07:10:26.151120+0000","last_fullsized":"2024-05-27T07:10:26.151120+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:17:40.681754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062991+0000","last_change":"2024-05-27T07:08:22.055426+0000","last_active":"2024-05-27T07:09:54.062991+0000","last_peered":"2024-05-27T07:09:54.062991+0000","last_clean":"2024-05-27T07:09:54.062991+0000","last_became_active":"2024-05-27T07:08:22.055063+0000","last_became_peered":"2024-05-27T07:08:22.055063+0000","last_unstale":"2024-05-27T07:09:54.062991+0000","last_undegraded":"2024-05-27T07:09:54.062991+0000","last_fullsized":"2024-05-27T07:09:54.062991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:06:49.923192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079424+0000","last_change":"2024-05-27T07:08:53.982756+0000","last_active":"2024-05-27T07:09:54.079424+0000","last_peered":"2024-05-27T07:09:54.079424+0000","last_clean":"2024-05-27T07:09:54.079424+0000","last_became_active":"2024-05-27T07:08:53.981816+0000","last_became_peered":"2024-05-27T07:08:53.981816+0000","last_unstale":"2024-05-27T07:09:54.079424+0000","last_undegraded":"2024-05-27T07:09:54.079424+0000","last_fullsized":"2024-05-27T07:09:54.079424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:09.984488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063371+0000","last_change":"2024-05-27T07:08:18.307659+0000","last_active":"2024-05-27T07:09:54.063371+0000","last_peered":"2024-05-27T07:09:54.063371+0000","last_clean":"2024-05-27T07:09:54.063371+0000","last_became_active":"2024-05-27T07:08:18.307239+0000","last_became_peered":"2024-05-27T07:08:18.307239+0000","last_unstale":"2024-05-27T07:09:54.063371+0000","last_undegraded":"2024-05-27T07:09:54.063371+0000","last_fullsized":"2024-05-27T07:09:54.063371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:09:02.816813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185149+0000","last_change":"2024-05-27T07:08:22.050403+0000","last_active":"2024-05-27T07:09:54.185149+0000","last_peered":"2024-05-27T07:09:54.185149+0000","last_clean":"2024-05-27T07:09:54.185149+0000","last_became_active":"2024-05-27T07:08:22.049939+0000","last_became_peered":"2024-05-27T07:08:22.049939+0000","last_unstale":"2024-05-27T07:09:54.185149+0000","last_undegraded":"2024-05-27T07:09:54.185149+0000","last_fullsized":"2024-05-27T07:09:54.185149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:45.857508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.078740+0000","last_change":"2024-05-27T07:08:36.864458+0000","last_active":"2024-05-27T07:09:54.078740+0000","last_peered":"2024-05-27T07:09:54.078740+0000","last_clean":"2024-05-27T07:09:54.078740+0000","last_became_active":"2024-05-27T07:08:36.863894+0000","last_became_peered":"2024-05-27T07:08:36.863894+0000","last_unstale":"2024-05-27T07:09:54.078740+0000","last_undegraded":"2024-05-27T07:09:54.078740+0000","last_fullsized":"2024-05-27T07:09:54.078740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:49.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190260+0000","last_change":"2024-05-27T07:08:18.016051+0000","last_active":"2024-05-27T07:09:54.190260+0000","last_peered":"2024-05-27T07:09:54.190260+0000","last_clean":"2024-05-27T07:09:54.190260+0000","last_became_active":"2024-05-27T07:08:18.015590+0000","last_became_peered":"2024-05-27T07:08:18.015590+0000","last_unstale":"2024-05-27T07:09:54.190260+0000","last_undegraded":"2024-05-27T07:09:54.190260+0000","last_fullsized":"2024-05-27T07:09:54.190260+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:33.396027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188148+0000","last_change":"2024-05-27T07:08:22.055603+0000","last_active":"2024-05-27T07:09:54.188148+0000","last_peered":"2024-05-27T07:09:54.188148+0000","last_clean":"2024-05-27T07:09:54.188148+0000","last_became_active":"2024-05-27T07:08:22.054543+0000","last_became_peered":"2024-05-27T07:08:22.054543+0000","last_unstale":"2024-05-27T07:09:54.188148+0000","last_undegraded":"2024-05-27T07:09:54.188148+0000","last_fullsized":"2024-05-27T07:09:54.188148+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:29:01.738598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.078315+0000","last_change":"2024-05-27T07:08:36.870358+0000","last_active":"2024-05-27T07:09:54.078315+0000","last_peered":"2024-05-27T07:09:54.078315+0000","last_clean":"2024-05-27T07:09:54.078315+0000","last_became_active":"2024-05-27T07:08:36.869457+0000","last_became_peered":"2024-05-27T07:08:36.869457+0000","last_unstale":"2024-05-27T07:09:54.078315+0000","last_undegraded":"2024-05-27T07:09:54.078315+0000","last_fullsized":"2024-05-27T07:09:54.078315+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:50.510098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187456+0000","last_change":"2024-05-27T07:08:53.985562+0000","last_active":"2024-05-27T07:09:54.187456+0000","last_peered":"2024-05-27T07:09:54.187456+0000","last_clean":"2024-05-27T07:09:54.187456+0000","last_became_active":"2024-05-27T07:08:53.977778+0000","last_became_peered":"2024-05-27T07:08:53.977778+0000","last_unstale":"2024-05-27T07:09:54.187456+0000","last_undegraded":"2024-05-27T07:09:54.187456+0000","last_fullsized":"2024-05-27T07:09:54.187456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:44.455508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044017+0000","last_change":"2024-05-27T07:08:22.057622+0000","last_active":"2024-05-27T07:09:54.044017+0000","last_peered":"2024-05-27T07:09:54.044017+0000","last_clean":"2024-05-27T07:09:54.044017+0000","last_became_active":"2024-05-27T07:08:22.056544+0000","last_became_peered":"2024-05-27T07:08:22.056544+0000","last_unstale":"2024-05-27T07:09:54.044017+0000","last_undegraded":"2024-05-27T07:09:54.044017+0000","last_fullsized":"2024-05-27T07:09:54.044017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:38.484560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070946+0000","last_change":"2024-05-27T07:08:36.864984+0000","last_active":"2024-05-27T07:09:54.070946+0000","last_peered":"2024-05-27T07:09:54.070946+0000","last_clean":"2024-05-27T07:09:54.070946+0000","last_became_active":"2024-05-27T07:08:36.864302+0000","last_became_peered":"2024-05-27T07:08:36.864302+0000","last_unstale":"2024-05-27T07:09:54.070946+0000","last_undegraded":"2024-05-27T07:09:54.070946+0000","last_fullsized":"2024-05-27T07:09:54.070946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:44.183562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:28.163382+0000","last_change":"2024-05-27T07:08:18.017918+0000","last_active":"2024-05-27T07:10:28.163382+0000","last_peered":"2024-05-27T07:10:28.163382+0000","last_clean":"2024-05-27T07:10:28.163382+0000","last_became_active":"2024-05-27T07:08:18.016174+0000","last_became_peered":"2024-05-27T07:08:18.016174+0000","last_unstale":"2024-05-27T07:10:28.163382+0000","last_undegraded":"2024-05-27T07:10:28.163382+0000","last_fullsized":"2024-05-27T07:10:28.163382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:21.906187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189895+0000","last_change":"2024-05-27T07:08:22.043172+0000","last_active":"2024-05-27T07:09:54.189895+0000","last_peered":"2024-05-27T07:09:54.189895+0000","last_clean":"2024-05-27T07:09:54.189895+0000","last_became_active":"2024-05-27T07:08:22.041399+0000","last_became_peered":"2024-05-27T07:08:22.041399+0000","last_unstale":"2024-05-27T07:09:54.189895+0000","last_undegraded":"2024-05-27T07:09:54.189895+0000","last_fullsized":"2024-05-27T07:09:54.189895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:19.651330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070122+0000","last_change":"2024-05-27T07:08:36.866352+0000","last_active":"2024-05-27T07:09:54.070122+0000","last_peered":"2024-05-27T07:09:54.070122+0000","last_clean":"2024-05-27T07:09:54.070122+0000","last_became_active":"2024-05-27T07:08:36.865447+0000","last_became_peered":"2024-05-27T07:08:36.865447+0000","last_unstale":"2024-05-27T07:09:54.070122+0000","last_undegraded":"2024-05-27T07:09:54.070122+0000","last_fullsized":"2024-05-27T07:09:54.070122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:55:42.634374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062875+0000","last_change":"2024-05-27T07:08:18.022142+0000","last_active":"2024-05-27T07:09:54.062875+0000","last_peered":"2024-05-27T07:09:54.062875+0000","last_clean":"2024-05-27T07:09:54.062875+0000","last_became_active":"2024-05-27T07:08:18.021453+0000","last_became_peered":"2024-05-27T07:08:18.021453+0000","last_unstale":"2024-05-27T07:09:54.062875+0000","last_undegraded":"2024-05-27T07:09:54.062875+0000","last_fullsized":"2024-05-27T07:09:54.062875+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:01:09.049440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043456+0000","last_change":"2024-05-27T07:08:22.055742+0000","last_active":"2024-05-27T07:09:54.043456+0000","last_peered":"2024-05-27T07:09:54.043456+0000","last_clean":"2024-05-27T07:09:54.043456+0000","last_became_active":"2024-05-27T07:08:22.052077+0000","last_became_peered":"2024-05-27T07:08:22.052077+0000","last_unstale":"2024-05-27T07:09:54.043456+0000","last_undegraded":"2024-05-27T07:09:54.043456+0000","last_fullsized":"2024-05-27T07:09:54.043456+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:09:13.418027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187972+0000","last_change":"2024-05-27T07:08:36.874225+0000","last_active":"2024-05-27T07:09:54.187972+0000","last_peered":"2024-05-27T07:09:54.187972+0000","last_clean":"2024-05-27T07:09:54.187972+0000","last_became_active":"2024-05-27T07:08:36.873745+0000","last_became_peered":"2024-05-27T07:08:36.873745+0000","last_unstale":"2024-05-27T07:09:54.187972+0000","last_undegraded":"2024-05-27T07:09:54.187972+0000","last_fullsized":"2024-05-27T07:09:54.187972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:40.633667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.040832+0000","last_change":"2024-05-27T07:08:18.009678+0000","last_active":"2024-05-27T07:09:54.040832+0000","last_peered":"2024-05-27T07:09:54.040832+0000","last_clean":"2024-05-27T07:09:54.040832+0000","last_became_active":"2024-05-27T07:08:18.006502+0000","last_became_peered":"2024-05-27T07:08:18.006502+0000","last_unstale":"2024-05-27T07:09:54.040832+0000","last_undegraded":"2024-05-27T07:09:54.040832+0000","last_fullsized":"2024-05-27T07:09:54.040832+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:37.457044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060622+0000","last_change":"2024-05-27T07:08:22.047302+0000","last_active":"2024-05-27T07:09:54.060622+0000","last_peered":"2024-05-27T07:09:54.060622+0000","last_clean":"2024-05-27T07:09:54.060622+0000","last_became_active":"2024-05-27T07:08:22.045932+0000","last_became_peered":"2024-05-27T07:08:22.045932+0000","last_unstale":"2024-05-27T07:09:54.060622+0000","last_undegraded":"2024-05-27T07:09:54.060622+0000","last_fullsized":"2024-05-27T07:09:54.060622+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:27.129539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185351+0000","last_change":"2024-05-27T07:08:36.870512+0000","last_active":"2024-05-27T07:09:54.185351+0000","last_peered":"2024-05-27T07:09:54.185351+0000","last_clean":"2024-05-27T07:09:54.185351+0000","last_became_active":"2024-05-27T07:08:36.870061+0000","last_became_peered":"2024-05-27T07:08:36.870061+0000","last_unstale":"2024-05-27T07:09:54.185351+0000","last_undegraded":"2024-05-27T07:09:54.185351+0000","last_fullsized":"2024-05-27T07:09:54.185351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:03.608758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042027+0000","last_change":"2024-05-27T07:08:18.018420+0000","last_active":"2024-05-27T07:09:54.042027+0000","last_peered":"2024-05-27T07:09:54.042027+0000","last_clean":"2024-05-27T07:09:54.042027+0000","last_became_active":"2024-05-27T07:08:18.017567+0000","last_became_peered":"2024-05-27T07:08:18.017567+0000","last_unstale":"2024-05-27T07:09:54.042027+0000","last_undegraded":"2024-05-27T07:09:54.042027+0000","last_fullsized":"2024-05-27T07:09:54.042027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:37.214408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.047241+0000","last_change":"2024-05-27T07:08:22.050943+0000","last_active":"2024-05-27T07:09:54.047241+0000","last_peered":"2024-05-27T07:09:54.047241+0000","last_clean":"2024-05-27T07:09:54.047241+0000","last_became_active":"2024-05-27T07:08:22.047706+0000","last_became_peered":"2024-05-27T07:08:22.047706+0000","last_unstale":"2024-05-27T07:09:54.047241+0000","last_undegraded":"2024-05-27T07:09:54.047241+0000","last_fullsized":"2024-05-27T07:09:54.047241+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:52:09.739528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060515+0000","last_change":"2024-05-27T07:08:36.867487+0000","last_active":"2024-05-27T07:09:54.060515+0000","last_peered":"2024-05-27T07:09:54.060515+0000","last_clean":"2024-05-27T07:09:54.060515+0000","last_became_active":"2024-05-27T07:08:36.867032+0000","last_became_peered":"2024-05-27T07:08:36.867032+0000","last_unstale":"2024-05-27T07:09:54.060515+0000","last_undegraded":"2024-05-27T07:09:54.060515+0000","last_fullsized":"2024-05-27T07:09:54.060515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:26:24.912382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:29.165194+0000","last_change":"2024-05-27T07:08:18.020168+0000","last_active":"2024-05-27T07:10:29.165194+0000","last_peered":"2024-05-27T07:10:29.165194+0000","last_clean":"2024-05-27T07:10:29.165194+0000","last_became_active":"2024-05-27T07:08:18.018309+0000","last_became_peered":"2024-05-27T07:08:18.018309+0000","last_unstale":"2024-05-27T07:10:29.165194+0000","last_undegraded":"2024-05-27T07:10:29.165194+0000","last_fullsized":"2024-05-27T07:10:29.165194+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:48:16.839510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188891+0000","last_change":"2024-05-27T07:08:22.054888+0000","last_active":"2024-05-27T07:09:54.188891+0000","last_peered":"2024-05-27T07:09:54.188891+0000","last_clean":"2024-05-27T07:09:54.188891+0000","last_became_active":"2024-05-27T07:08:22.054417+0000","last_became_peered":"2024-05-27T07:08:22.054417+0000","last_unstale":"2024-05-27T07:09:54.188891+0000","last_undegraded":"2024-05-27T07:09:54.188891+0000","last_fullsized":"2024-05-27T07:09:54.188891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:43:00.866755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.047828+0000","last_change":"2024-05-27T07:08:36.854516+0000","last_active":"2024-05-27T07:09:54.047828+0000","last_peered":"2024-05-27T07:09:54.047828+0000","last_clean":"2024-05-27T07:09:54.047828+0000","last_became_active":"2024-05-27T07:08:36.853978+0000","last_became_peered":"2024-05-27T07:08:36.853978+0000","last_unstale":"2024-05-27T07:09:54.047828+0000","last_undegraded":"2024-05-27T07:09:54.047828+0000","last_fullsized":"2024-05-27T07:09:54.047828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:45.299100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045324+0000","last_change":"2024-05-27T07:08:18.010539+0000","last_active":"2024-05-27T07:09:54.045324+0000","last_peered":"2024-05-27T07:09:54.045324+0000","last_clean":"2024-05-27T07:09:54.045324+0000","last_became_active":"2024-05-27T07:08:18.009643+0000","last_became_peered":"2024-05-27T07:08:18.009643+0000","last_unstale":"2024-05-27T07:09:54.045324+0000","last_undegraded":"2024-05-27T07:09:54.045324+0000","last_fullsized":"2024-05-27T07:09:54.045324+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:48:40.335403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185027+0000","last_change":"2024-05-27T07:08:53.980165+0000","last_active":"2024-05-27T07:09:54.185027+0000","last_peered":"2024-05-27T07:09:54.185027+0000","last_clean":"2024-05-27T07:09:54.185027+0000","last_became_active":"2024-05-27T07:08:53.978565+0000","last_became_peered":"2024-05-27T07:08:53.978565+0000","last_unstale":"2024-05-27T07:09:54.185027+0000","last_undegraded":"2024-05-27T07:09:54.185027+0000","last_fullsized":"2024-05-27T07:09:54.185027+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:56:33.513961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079469+0000","last_change":"2024-05-27T07:08:36.857814+0000","last_active":"2024-05-27T07:09:54.079469+0000","last_peered":"2024-05-27T07:09:54.079469+0000","last_clean":"2024-05-27T07:09:54.079469+0000","last_became_active":"2024-05-27T07:08:36.856790+0000","last_became_peered":"2024-05-27T07:08:36.856790+0000","last_unstale":"2024-05-27T07:09:54.079469+0000","last_undegraded":"2024-05-27T07:09:54.079469+0000","last_fullsized":"2024-05-27T07:09:54.079469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:39:10.322261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043460+0000","last_change":"2024-05-27T07:08:18.308032+0000","last_active":"2024-05-27T07:09:54.043460+0000","last_peered":"2024-05-27T07:09:54.043460+0000","last_clean":"2024-05-27T07:09:54.043460+0000","last_became_active":"2024-05-27T07:08:18.306867+0000","last_became_peered":"2024-05-27T07:08:18.306867+0000","last_unstale":"2024-05-27T07:09:54.043460+0000","last_undegraded":"2024-05-27T07:09:54.043460+0000","last_fullsized":"2024-05-27T07:09:54.043460+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:37.816237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187935+0000","last_change":"2024-05-27T07:08:22.057940+0000","last_active":"2024-05-27T07:09:54.187935+0000","last_peered":"2024-05-27T07:09:54.187935+0000","last_clean":"2024-05-27T07:09:54.187935+0000","last_became_active":"2024-05-27T07:08:22.055109+0000","last_became_peered":"2024-05-27T07:08:22.055109+0000","last_unstale":"2024-05-27T07:09:54.187935+0000","last_undegraded":"2024-05-27T07:09:54.187935+0000","last_fullsized":"2024-05-27T07:09:54.187935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:46.905064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062562+0000","last_change":"2024-05-27T07:08:53.980209+0000","last_active":"2024-05-27T07:09:54.062562+0000","last_peered":"2024-05-27T07:09:54.062562+0000","last_clean":"2024-05-27T07:09:54.062562+0000","last_became_active":"2024-05-27T07:08:53.979353+0000","last_became_peered":"2024-05-27T07:08:53.979353+0000","last_unstale":"2024-05-27T07:09:54.062562+0000","last_undegraded":"2024-05-27T07:09:54.062562+0000","last_fullsized":"2024-05-27T07:09:54.062562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:06.163563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:29.184680+0000","last_change":"2024-05-27T07:08:53.996491+0000","last_active":"2024-05-27T07:10:29.184680+0000","last_peered":"2024-05-27T07:10:29.184680+0000","last_clean":"2024-05-27T07:10:29.184680+0000","last_became_active":"2024-05-27T07:08:53.976013+0000","last_became_peered":"2024-05-27T07:08:53.976013+0000","last_unstale":"2024-05-27T07:10:29.184680+0000","last_undegraded":"2024-05-27T07:10:29.184680+0000","last_fullsized":"2024-05-27T07:10:29.184680+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:44:16.592270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64136,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":83,"num_read_kb":104,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189909+0000","last_change":"2024-05-27T07:08:22.057231+0000","last_active":"2024-05-27T07:09:54.189909+0000","last_peered":"2024-05-27T07:09:54.189909+0000","last_clean":"2024-05-27T07:09:54.189909+0000","last_became_active":"2024-05-27T07:08:22.056837+0000","last_became_peered":"2024-05-27T07:08:22.056837+0000","last_unstale":"2024-05-27T07:09:54.189909+0000","last_undegraded":"2024-05-27T07:09:54.189909+0000","last_fullsized":"2024-05-27T07:09:54.189909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:10:28.645103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044380+0000","last_change":"2024-05-27T07:08:36.870307+0000","last_active":"2024-05-27T07:09:54.044380+0000","last_peered":"2024-05-27T07:09:54.044380+0000","last_clean":"2024-05-27T07:09:54.044380+0000","last_became_active":"2024-05-27T07:08:36.869390+0000","last_became_peered":"2024-05-27T07:08:36.869390+0000","last_unstale":"2024-05-27T07:09:54.044380+0000","last_undegraded":"2024-05-27T07:09:54.044380+0000","last_fullsized":"2024-05-27T07:09:54.044380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:43:45.228885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070573+0000","last_change":"2024-05-27T07:08:18.014458+0000","last_active":"2024-05-27T07:09:54.070573+0000","last_peered":"2024-05-27T07:09:54.070573+0000","last_clean":"2024-05-27T07:09:54.070573+0000","last_became_active":"2024-05-27T07:08:18.013365+0000","last_became_peered":"2024-05-27T07:08:18.013365+0000","last_unstale":"2024-05-27T07:09:54.070573+0000","last_undegraded":"2024-05-27T07:09:54.070573+0000","last_fullsized":"2024-05-27T07:09:54.070573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:41.352618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041318+0000","last_change":"2024-05-27T07:08:22.043354+0000","last_active":"2024-05-27T07:09:54.041318+0000","last_peered":"2024-05-27T07:09:54.041318+0000","last_clean":"2024-05-27T07:09:54.041318+0000","last_became_active":"2024-05-27T07:08:22.042876+0000","last_became_peered":"2024-05-27T07:08:22.042876+0000","last_unstale":"2024-05-27T07:09:54.041318+0000","last_undegraded":"2024-05-27T07:09:54.041318+0000","last_fullsized":"2024-05-27T07:09:54.041318+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:52.575568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187862+0000","last_change":"2024-05-27T07:08:36.869579+0000","last_active":"2024-05-27T07:09:54.187862+0000","last_peered":"2024-05-27T07:09:54.187862+0000","last_clean":"2024-05-27T07:09:54.187862+0000","last_became_active":"2024-05-27T07:08:36.869055+0000","last_became_peered":"2024-05-27T07:08:36.869055+0000","last_unstale":"2024-05-27T07:09:54.187862+0000","last_undegraded":"2024-05-27T07:09:54.187862+0000","last_fullsized":"2024-05-27T07:09:54.187862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.870079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043959+0000","last_change":"2024-05-27T07:08:18.015322+0000","last_active":"2024-05-27T07:09:54.043959+0000","last_peered":"2024-05-27T07:09:54.043959+0000","last_clean":"2024-05-27T07:09:54.043959+0000","last_became_active":"2024-05-27T07:08:18.014535+0000","last_became_peered":"2024-05-27T07:08:18.014535+0000","last_unstale":"2024-05-27T07:09:54.043959+0000","last_undegraded":"2024-05-27T07:09:54.043959+0000","last_fullsized":"2024-05-27T07:09:54.043959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:35.116140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045527+0000","last_change":"2024-05-27T07:08:22.056181+0000","last_active":"2024-05-27T07:09:54.045527+0000","last_peered":"2024-05-27T07:09:54.045527+0000","last_clean":"2024-05-27T07:09:54.045527+0000","last_became_active":"2024-05-27T07:08:22.055755+0000","last_became_peered":"2024-05-27T07:08:22.055755+0000","last_unstale":"2024-05-27T07:09:54.045527+0000","last_undegraded":"2024-05-27T07:09:54.045527+0000","last_fullsized":"2024-05-27T07:09:54.045527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:38:35.634116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.048385+0000","last_change":"2024-05-27T07:08:36.863184+0000","last_active":"2024-05-27T07:09:54.048385+0000","last_peered":"2024-05-27T07:09:54.048385+0000","last_clean":"2024-05-27T07:09:54.048385+0000","last_became_active":"2024-05-27T07:08:36.862760+0000","last_became_peered":"2024-05-27T07:08:36.862760+0000","last_unstale":"2024-05-27T07:09:54.048385+0000","last_undegraded":"2024-05-27T07:09:54.048385+0000","last_fullsized":"2024-05-27T07:09:54.048385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:48.379157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:25.176630+0000","last_change":"2024-05-27T07:08:18.019119+0000","last_active":"2024-05-27T07:10:25.176630+0000","last_peered":"2024-05-27T07:10:25.176630+0000","last_clean":"2024-05-27T07:10:25.176630+0000","last_became_active":"2024-05-27T07:08:18.018408+0000","last_became_peered":"2024-05-27T07:08:18.018408+0000","last_unstale":"2024-05-27T07:10:25.176630+0000","last_undegraded":"2024-05-27T07:10:25.176630+0000","last_fullsized":"2024-05-27T07:10:25.176630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:19.978328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045094+0000","last_change":"2024-05-27T07:08:22.059464+0000","last_active":"2024-05-27T07:09:54.045094+0000","last_peered":"2024-05-27T07:09:54.045094+0000","last_clean":"2024-05-27T07:09:54.045094+0000","last_became_active":"2024-05-27T07:08:22.058890+0000","last_became_peered":"2024-05-27T07:08:22.058890+0000","last_unstale":"2024-05-27T07:09:54.045094+0000","last_undegraded":"2024-05-27T07:09:54.045094+0000","last_fullsized":"2024-05-27T07:09:54.045094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:30.122552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041831+0000","last_change":"2024-05-27T07:08:36.865884+0000","last_active":"2024-05-27T07:09:54.041831+0000","last_peered":"2024-05-27T07:09:54.041831+0000","last_clean":"2024-05-27T07:09:54.041831+0000","last_became_active":"2024-05-27T07:08:36.865443+0000","last_became_peered":"2024-05-27T07:08:36.865443+0000","last_unstale":"2024-05-27T07:09:54.041831+0000","last_undegraded":"2024-05-27T07:09:54.041831+0000","last_fullsized":"2024-05-27T07:09:54.041831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:47:15.026632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.186967+0000","last_change":"2024-05-27T07:08:53.994809+0000","last_active":"2024-05-27T07:09:54.186967+0000","last_peered":"2024-05-27T07:09:54.186967+0000","last_clean":"2024-05-27T07:09:54.186967+0000","last_became_active":"2024-05-27T07:08:53.977628+0000","last_became_peered":"2024-05-27T07:08:53.977628+0000","last_unstale":"2024-05-27T07:09:54.186967+0000","last_undegraded":"2024-05-27T07:09:54.186967+0000","last_fullsized":"2024-05-27T07:09:54.186967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:13:37.176319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042726+0000","last_change":"2024-05-27T07:08:22.050733+0000","last_active":"2024-05-27T07:09:54.042726+0000","last_peered":"2024-05-27T07:09:54.042726+0000","last_clean":"2024-05-27T07:09:54.042726+0000","last_became_active":"2024-05-27T07:08:22.050321+0000","last_became_peered":"2024-05-27T07:08:22.050321+0000","last_unstale":"2024-05-27T07:09:54.042726+0000","last_undegraded":"2024-05-27T07:09:54.042726+0000","last_fullsized":"2024-05-27T07:09:54.042726+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:44.590595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042726+0000","last_change":"2024-05-27T07:08:36.867784+0000","last_active":"2024-05-27T07:09:54.042726+0000","last_peered":"2024-05-27T07:09:54.042726+0000","last_clean":"2024-05-27T07:09:54.042726+0000","last_became_active":"2024-05-27T07:08:36.867290+0000","last_became_peered":"2024-05-27T07:08:36.867290+0000","last_unstale":"2024-05-27T07:09:54.042726+0000","last_undegraded":"2024-05-27T07:09:54.042726+0000","last_fullsized":"2024-05-27T07:09:54.042726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:44:07.649640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190683+0000","last_change":"2024-05-27T07:08:18.013193+0000","last_active":"2024-05-27T07:09:54.190683+0000","last_peered":"2024-05-27T07:09:54.190683+0000","last_clean":"2024-05-27T07:09:54.190683+0000","last_became_active":"2024-05-27T07:08:18.011767+0000","last_became_peered":"2024-05-27T07:08:18.011767+0000","last_unstale":"2024-05-27T07:09:54.190683+0000","last_undegraded":"2024-05-27T07:09:54.190683+0000","last_fullsized":"2024-05-27T07:09:54.190683+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:15:03.679719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187747+0000","last_change":"2024-05-27T07:08:22.067145+0000","last_active":"2024-05-27T07:09:54.187747+0000","last_peered":"2024-05-27T07:09:54.187747+0000","last_clean":"2024-05-27T07:09:54.187747+0000","last_became_active":"2024-05-27T07:08:22.066750+0000","last_became_peered":"2024-05-27T07:08:22.066750+0000","last_unstale":"2024-05-27T07:09:54.187747+0000","last_undegraded":"2024-05-27T07:09:54.187747+0000","last_fullsized":"2024-05-27T07:09:54.187747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:41:35.333311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189061+0000","last_change":"2024-05-27T07:08:53.982497+0000","last_active":"2024-05-27T07:09:54.189061+0000","last_peered":"2024-05-27T07:09:54.189061+0000","last_clean":"2024-05-27T07:09:54.189061+0000","last_became_active":"2024-05-27T07:08:53.981920+0000","last_became_peered":"2024-05-27T07:08:53.981920+0000","last_unstale":"2024-05-27T07:09:54.189061+0000","last_undegraded":"2024-05-27T07:09:54.189061+0000","last_fullsized":"2024-05-27T07:09:54.189061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:43:40.095357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.048681+0000","last_change":"2024-05-27T07:08:18.308330+0000","last_active":"2024-05-27T07:09:54.048681+0000","last_peered":"2024-05-27T07:09:54.048681+0000","last_clean":"2024-05-27T07:09:54.048681+0000","last_became_active":"2024-05-27T07:08:18.306961+0000","last_became_peered":"2024-05-27T07:08:18.306961+0000","last_unstale":"2024-05-27T07:09:54.048681+0000","last_undegraded":"2024-05-27T07:09:54.048681+0000","last_fullsized":"2024-05-27T07:09:54.048681+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:27.875936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063450+0000","last_change":"2024-05-27T07:08:22.047562+0000","last_active":"2024-05-27T07:09:54.063450+0000","last_peered":"2024-05-27T07:09:54.063450+0000","last_clean":"2024-05-27T07:09:54.063450+0000","last_became_active":"2024-05-27T07:08:22.046977+0000","last_became_peered":"2024-05-27T07:08:22.046977+0000","last_unstale":"2024-05-27T07:09:54.063450+0000","last_undegraded":"2024-05-27T07:09:54.063450+0000","last_fullsized":"2024-05-27T07:09:54.063450+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:49:43.376617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041023+0000","last_change":"2024-05-27T07:08:36.867382+0000","last_active":"2024-05-27T07:09:54.041023+0000","last_peered":"2024-05-27T07:09:54.041023+0000","last_clean":"2024-05-27T07:09:54.041023+0000","last_became_active":"2024-05-27T07:08:36.866296+0000","last_became_peered":"2024-05-27T07:08:36.866296+0000","last_unstale":"2024-05-27T07:09:54.041023+0000","last_undegraded":"2024-05-27T07:09:54.041023+0000","last_fullsized":"2024-05-27T07:09:54.041023+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:14.305169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:25.149868+0000","last_change":"2024-05-27T07:08:18.012031+0000","last_active":"2024-05-27T07:10:25.149868+0000","last_peered":"2024-05-27T07:10:25.149868+0000","last_clean":"2024-05-27T07:10:25.149868+0000","last_became_active":"2024-05-27T07:08:18.009593+0000","last_became_peered":"2024-05-27T07:08:18.009593+0000","last_unstale":"2024-05-27T07:10:25.149868+0000","last_undegraded":"2024-05-27T07:10:25.149868+0000","last_fullsized":"2024-05-27T07:10:25.149868+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:25:22.316575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063203+0000","last_change":"2024-05-27T07:08:22.061136+0000","last_active":"2024-05-27T07:09:54.063203+0000","last_peered":"2024-05-27T07:09:54.063203+0000","last_clean":"2024-05-27T07:09:54.063203+0000","last_became_active":"2024-05-27T07:08:22.060735+0000","last_became_peered":"2024-05-27T07:08:22.060735+0000","last_unstale":"2024-05-27T07:09:54.063203+0000","last_undegraded":"2024-05-27T07:09:54.063203+0000","last_fullsized":"2024-05-27T07:09:54.063203+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:50.392130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041651+0000","last_change":"2024-05-27T07:08:36.932187+0000","last_active":"2024-05-27T07:09:54.041651+0000","last_peered":"2024-05-27T07:09:54.041651+0000","last_clean":"2024-05-27T07:09:54.041651+0000","last_became_active":"2024-05-27T07:08:36.931214+0000","last_became_peered":"2024-05-27T07:08:36.931214+0000","last_unstale":"2024-05-27T07:09:54.041651+0000","last_undegraded":"2024-05-27T07:09:54.041651+0000","last_fullsized":"2024-05-27T07:09:54.041651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:20:59.645352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.083887+0000","last_change":"2024-05-27T07:09:54.082674+0000","last_active":"2024-05-27T07:09:54.083887+0000","last_peered":"2024-05-27T07:09:54.083887+0000","last_clean":"2024-05-27T07:09:54.083887+0000","last_became_active":"2024-05-27T07:09:54.081616+0000","last_became_peered":"2024-05-27T07:09:54.081616+0000","last_unstale":"2024-05-27T07:09:54.083887+0000","last_undegraded":"2024-05-27T07:09:54.083887+0000","last_fullsized":"2024-05-27T07:09:54.083887+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:38:11.230026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:27.180963+0000","last_change":"2024-05-27T07:08:18.306439+0000","last_active":"2024-05-27T07:10:27.180963+0000","last_peered":"2024-05-27T07:10:27.180963+0000","last_clean":"2024-05-27T07:10:27.180963+0000","last_became_active":"2024-05-27T07:08:18.305978+0000","last_became_peered":"2024-05-27T07:08:18.305978+0000","last_unstale":"2024-05-27T07:10:27.180963+0000","last_undegraded":"2024-05-27T07:10:27.180963+0000","last_fullsized":"2024-05-27T07:10:27.180963+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:20:22.648922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.046025+0000","last_change":"2024-05-27T07:08:36.866247+0000","last_active":"2024-05-27T07:09:54.046025+0000","last_peered":"2024-05-27T07:09:54.046025+0000","last_clean":"2024-05-27T07:09:54.046025+0000","last_became_active":"2024-05-27T07:08:36.865516+0000","last_became_peered":"2024-05-27T07:08:36.865516+0000","last_unstale":"2024-05-27T07:09:54.046025+0000","last_undegraded":"2024-05-27T07:09:54.046025+0000","last_fullsized":"2024-05-27T07:09:54.046025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:39:41.235352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.061812+0000","last_change":"2024-05-27T07:08:22.045312+0000","last_active":"2024-05-27T07:09:54.061812+0000","last_peered":"2024-05-27T07:09:54.061812+0000","last_clean":"2024-05-27T07:09:54.061812+0000","last_became_active":"2024-05-27T07:08:22.044724+0000","last_became_peered":"2024-05-27T07:08:22.044724+0000","last_unstale":"2024-05-27T07:09:54.061812+0000","last_undegraded":"2024-05-27T07:09:54.061812+0000","last_fullsized":"2024-05-27T07:09:54.061812+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:05.487794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079152+0000","last_change":"2024-05-27T07:08:18.019333+0000","last_active":"2024-05-27T07:09:54.079152+0000","last_peered":"2024-05-27T07:09:54.079152+0000","last_clean":"2024-05-27T07:09:54.079152+0000","last_became_active":"2024-05-27T07:08:18.018816+0000","last_became_peered":"2024-05-27T07:08:18.018816+0000","last_unstale":"2024-05-27T07:09:54.079152+0000","last_undegraded":"2024-05-27T07:09:54.079152+0000","last_fullsized":"2024-05-27T07:09:54.079152+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:09.421346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044348+0000","last_change":"2024-05-27T07:06:16.865587+0000","last_active":"2024-05-27T07:09:54.044348+0000","last_peered":"2024-05-27T07:09:54.044348+0000","last_clean":"2024-05-27T07:09:54.044348+0000","last_became_active":"2024-05-27T07:06:16.850646+0000","last_became_peered":"2024-05-27T07:06:16.850646+0000","last_unstale":"2024-05-27T07:09:54.044348+0000","last_undegraded":"2024-05-27T07:09:54.044348+0000","last_fullsized":"2024-05-27T07:09:54.044348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_clean_scrub_stamp":"2024-05-27T07:01:14.931400+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:37.272619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043593+0000","last_change":"2024-05-27T07:08:36.934696+0000","last_active":"2024-05-27T07:09:54.043593+0000","last_peered":"2024-05-27T07:09:54.043593+0000","last_clean":"2024-05-27T07:09:54.043593+0000","last_became_active":"2024-05-27T07:08:36.934008+0000","last_became_peered":"2024-05-27T07:08:36.934008+0000","last_unstale":"2024-05-27T07:09:54.043593+0000","last_undegraded":"2024-05-27T07:09:54.043593+0000","last_fullsized":"2024-05-27T07:09:54.043593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:39.099180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.040590+0000","last_change":"2024-05-27T07:08:22.054479+0000","last_active":"2024-05-27T07:09:54.040590+0000","last_peered":"2024-05-27T07:09:54.040590+0000","last_clean":"2024-05-27T07:09:54.040590+0000","last_became_active":"2024-05-27T07:08:22.054127+0000","last_became_peered":"2024-05-27T07:08:22.054127+0000","last_unstale":"2024-05-27T07:09:54.040590+0000","last_undegraded":"2024-05-27T07:09:54.040590+0000","last_fullsized":"2024-05-27T07:09:54.040590+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:37:25.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045671+0000","last_change":"2024-05-27T07:08:18.010944+0000","last_active":"2024-05-27T07:09:54.045671+0000","last_peered":"2024-05-27T07:09:54.045671+0000","last_clean":"2024-05-27T07:09:54.045671+0000","last_became_active":"2024-05-27T07:08:18.009816+0000","last_became_peered":"2024-05-27T07:08:18.009816+0000","last_unstale":"2024-05-27T07:09:54.045671+0000","last_undegraded":"2024-05-27T07:09:54.045671+0000","last_fullsized":"2024-05-27T07:09:54.045671+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:34:26.061787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187383+0000","last_change":"2024-05-27T07:08:36.870618+0000","last_active":"2024-05-27T07:09:54.187383+0000","last_peered":"2024-05-27T07:09:54.187383+0000","last_clean":"2024-05-27T07:09:54.187383+0000","last_became_active":"2024-05-27T07:08:36.869799+0000","last_became_peered":"2024-05-27T07:08:36.869799+0000","last_unstale":"2024-05-27T07:09:54.187383+0000","last_undegraded":"2024-05-27T07:09:54.187383+0000","last_fullsized":"2024-05-27T07:09:54.187383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:53.094337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190088+0000","last_change":"2024-05-27T07:08:18.308834+0000","last_active":"2024-05-27T07:09:54.190088+0000","last_peered":"2024-05-27T07:09:54.190088+0000","last_clean":"2024-05-27T07:09:54.190088+0000","last_became_active":"2024-05-27T07:08:18.306953+0000","last_became_peered":"2024-05-27T07:08:18.306953+0000","last_unstale":"2024-05-27T07:09:54.190088+0000","last_undegraded":"2024-05-27T07:09:54.190088+0000","last_fullsized":"2024-05-27T07:09:54.190088+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:26:14.747631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041301+0000","last_change":"2024-05-27T07:08:22.048219+0000","last_active":"2024-05-27T07:09:54.041301+0000","last_peered":"2024-05-27T07:09:54.041301+0000","last_clean":"2024-05-27T07:09:54.041301+0000","last_became_active":"2024-05-27T07:08:22.047848+0000","last_became_peered":"2024-05-27T07:08:22.047848+0000","last_unstale":"2024-05-27T07:09:54.041301+0000","last_undegraded":"2024-05-27T07:09:54.041301+0000","last_fullsized":"2024-05-27T07:09:54.041301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:51.707603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060218+0000","last_change":"2024-05-27T07:08:36.932747+0000","last_active":"2024-05-27T07:09:54.060218+0000","last_peered":"2024-05-27T07:09:54.060218+0000","last_clean":"2024-05-27T07:09:54.060218+0000","last_became_active":"2024-05-27T07:08:36.932050+0000","last_became_peered":"2024-05-27T07:08:36.932050+0000","last_unstale":"2024-05-27T07:09:54.060218+0000","last_undegraded":"2024-05-27T07:09:54.060218+0000","last_fullsized":"2024-05-27T07:09:54.060218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:26:07.015651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'15","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:25.122124+0000","last_change":"2024-05-27T07:08:18.008844+0000","last_active":"2024-05-27T07:10:25.122124+0000","last_peered":"2024-05-27T07:10:25.122124+0000","last_clean":"2024-05-27T07:10:25.122124+0000","last_became_active":"2024-05-27T07:08:18.008140+0000","last_became_peered":"2024-05-27T07:08:18.008140+0000","last_unstale":"2024-05-27T07:10:25.122124+0000","last_undegraded":"2024-05-27T07:10:25.122124+0000","last_fullsized":"2024-05-27T07:10:25.122124+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:22:22.075341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.061571+0000","last_change":"2024-05-27T07:08:22.050855+0000","last_active":"2024-05-27T07:09:54.061571+0000","last_peered":"2024-05-27T07:09:54.061571+0000","last_clean":"2024-05-27T07:09:54.061571+0000","last_became_active":"2024-05-27T07:08:22.048482+0000","last_became_peered":"2024-05-27T07:08:22.048482+0000","last_unstale":"2024-05-27T07:09:54.061571+0000","last_undegraded":"2024-05-27T07:09:54.061571+0000","last_fullsized":"2024-05-27T07:09:54.061571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:04.319884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.069514+0000","last_change":"2024-05-27T07:08:36.862470+0000","last_active":"2024-05-27T07:09:54.069514+0000","last_peered":"2024-05-27T07:09:54.069514+0000","last_clean":"2024-05-27T07:09:54.069514+0000","last_became_active":"2024-05-27T07:08:36.861951+0000","last_became_peered":"2024-05-27T07:08:36.861951+0000","last_unstale":"2024-05-27T07:09:54.069514+0000","last_undegraded":"2024-05-27T07:09:54.069514+0000","last_fullsized":"2024-05-27T07:09:54.069514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:56:31.659951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.080450+0000","last_change":"2024-05-27T07:08:18.438518+0000","last_active":"2024-05-27T07:09:54.080450+0000","last_peered":"2024-05-27T07:09:54.080450+0000","last_clean":"2024-05-27T07:09:54.080450+0000","last_became_active":"2024-05-27T07:08:18.437588+0000","last_became_peered":"2024-05-27T07:08:18.437588+0000","last_unstale":"2024-05-27T07:09:54.080450+0000","last_undegraded":"2024-05-27T07:09:54.080450+0000","last_fullsized":"2024-05-27T07:09:54.080450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:52:12.895009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185484+0000","last_change":"2024-05-27T07:08:22.064188+0000","last_active":"2024-05-27T07:09:54.185484+0000","last_peered":"2024-05-27T07:09:54.185484+0000","last_clean":"2024-05-27T07:09:54.185484+0000","last_became_active":"2024-05-27T07:08:22.063868+0000","last_became_peered":"2024-05-27T07:08:22.063868+0000","last_unstale":"2024-05-27T07:09:54.185484+0000","last_undegraded":"2024-05-27T07:09:54.185484+0000","last_fullsized":"2024-05-27T07:09:54.185484+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:43:17.227079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079129+0000","last_change":"2024-05-27T07:08:36.934922+0000","last_active":"2024-05-27T07:09:54.079129+0000","last_peered":"2024-05-27T07:09:54.079129+0000","last_clean":"2024-05-27T07:09:54.079129+0000","last_became_active":"2024-05-27T07:08:36.934000+0000","last_became_peered":"2024-05-27T07:08:36.934000+0000","last_unstale":"2024-05-27T07:09:54.079129+0000","last_undegraded":"2024-05-27T07:09:54.079129+0000","last_fullsized":"2024-05-27T07:09:54.079129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:19:00.570557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187967+0000","last_change":"2024-05-27T07:08:18.308743+0000","last_active":"2024-05-27T07:09:54.187967+0000","last_peered":"2024-05-27T07:09:54.187967+0000","last_clean":"2024-05-27T07:09:54.187967+0000","last_became_active":"2024-05-27T07:08:18.307597+0000","last_became_peered":"2024-05-27T07:08:18.307597+0000","last_unstale":"2024-05-27T07:09:54.187967+0000","last_undegraded":"2024-05-27T07:09:54.187967+0000","last_fullsized":"2024-05-27T07:09:54.187967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:39.914998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188931+0000","last_change":"2024-05-27T07:08:22.049132+0000","last_active":"2024-05-27T07:09:54.188931+0000","last_peered":"2024-05-27T07:09:54.188931+0000","last_clean":"2024-05-27T07:09:54.188931+0000","last_became_active":"2024-05-27T07:08:22.048606+0000","last_became_peered":"2024-05-27T07:08:22.048606+0000","last_unstale":"2024-05-27T07:09:54.188931+0000","last_undegraded":"2024-05-27T07:09:54.188931+0000","last_fullsized":"2024-05-27T07:09:54.188931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:58.253461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.049190+0000","last_change":"2024-05-27T07:08:36.867286+0000","last_active":"2024-05-27T07:09:54.049190+0000","last_peered":"2024-05-27T07:09:54.049190+0000","last_clean":"2024-05-27T07:09:54.049190+0000","last_became_active":"2024-05-27T07:08:36.866732+0000","last_became_peered":"2024-05-27T07:08:36.866732+0000","last_unstale":"2024-05-27T07:09:54.049190+0000","last_undegraded":"2024-05-27T07:09:54.049190+0000","last_fullsized":"2024-05-27T07:09:54.049190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:14:18.386441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060788+0000","last_change":"2024-05-27T07:08:18.010026+0000","last_active":"2024-05-27T07:09:54.060788+0000","last_peered":"2024-05-27T07:09:54.060788+0000","last_clean":"2024-05-27T07:09:54.060788+0000","last_became_active":"2024-05-27T07:08:18.009502+0000","last_became_peered":"2024-05-27T07:08:18.009502+0000","last_unstale":"2024-05-27T07:09:54.060788+0000","last_undegraded":"2024-05-27T07:09:54.060788+0000","last_fullsized":"2024-05-27T07:09:54.060788+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:28:27.500933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187046+0000","last_change":"2024-05-27T07:08:22.054536+0000","last_active":"2024-05-27T07:09:54.187046+0000","last_peered":"2024-05-27T07:09:54.187046+0000","last_clean":"2024-05-27T07:09:54.187046+0000","last_became_active":"2024-05-27T07:08:22.054141+0000","last_became_peered":"2024-05-27T07:08:22.054141+0000","last_unstale":"2024-05-27T07:09:54.187046+0000","last_undegraded":"2024-05-27T07:09:54.187046+0000","last_fullsized":"2024-05-27T07:09:54.187046+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:53.413683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.080431+0000","last_change":"2024-05-27T07:08:36.859721+0000","last_active":"2024-05-27T07:09:54.080431+0000","last_peered":"2024-05-27T07:09:54.080431+0000","last_clean":"2024-05-27T07:09:54.080431+0000","last_became_active":"2024-05-27T07:08:36.858770+0000","last_became_peered":"2024-05-27T07:08:36.858770+0000","last_unstale":"2024-05-27T07:09:54.080431+0000","last_undegraded":"2024-05-27T07:09:54.080431+0000","last_fullsized":"2024-05-27T07:09:54.080431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:32.583869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070392+0000","last_change":"2024-05-27T07:08:18.008762+0000","last_active":"2024-05-27T07:09:54.070392+0000","last_peered":"2024-05-27T07:09:54.070392+0000","last_clean":"2024-05-27T07:09:54.070392+0000","last_became_active":"2024-05-27T07:08:18.007680+0000","last_became_peered":"2024-05-27T07:08:18.007680+0000","last_unstale":"2024-05-27T07:09:54.070392+0000","last_undegraded":"2024-05-27T07:09:54.070392+0000","last_fullsized":"2024-05-27T07:09:54.070392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:22.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062292+0000","last_change":"2024-05-27T07:08:22.051955+0000","last_active":"2024-05-27T07:09:54.062292+0000","last_peered":"2024-05-27T07:09:54.062292+0000","last_clean":"2024-05-27T07:09:54.062292+0000","last_became_active":"2024-05-27T07:08:22.051618+0000","last_became_peered":"2024-05-27T07:08:22.051618+0000","last_unstale":"2024-05-27T07:09:54.062292+0000","last_undegraded":"2024-05-27T07:09:54.062292+0000","last_fullsized":"2024-05-27T07:09:54.062292+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:13.219186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.078961+0000","last_change":"2024-05-27T07:08:36.932048+0000","last_active":"2024-05-27T07:09:54.078961+0000","last_peered":"2024-05-27T07:09:54.078961+0000","last_clean":"2024-05-27T07:09:54.078961+0000","last_became_active":"2024-05-27T07:08:36.930777+0000","last_became_peered":"2024-05-27T07:08:36.930777+0000","last_unstale":"2024-05-27T07:09:54.078961+0000","last_undegraded":"2024-05-27T07:09:54.078961+0000","last_fullsized":"2024-05-27T07:09:54.078961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:32.284487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189264+0000","last_change":"2024-05-27T07:08:18.011343+0000","last_active":"2024-05-27T07:09:54.189264+0000","last_peered":"2024-05-27T07:09:54.189264+0000","last_clean":"2024-05-27T07:09:54.189264+0000","last_became_active":"2024-05-27T07:08:18.009988+0000","last_became_peered":"2024-05-27T07:08:18.009988+0000","last_unstale":"2024-05-27T07:09:54.189264+0000","last_undegraded":"2024-05-27T07:09:54.189264+0000","last_fullsized":"2024-05-27T07:09:54.189264+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:39.620568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045033+0000","last_change":"2024-05-27T07:08:22.051630+0000","last_active":"2024-05-27T07:09:54.045033+0000","last_peered":"2024-05-27T07:09:54.045033+0000","last_clean":"2024-05-27T07:09:54.045033+0000","last_became_active":"2024-05-27T07:08:22.049140+0000","last_became_peered":"2024-05-27T07:08:22.049140+0000","last_unstale":"2024-05-27T07:09:54.045033+0000","last_undegraded":"2024-05-27T07:09:54.045033+0000","last_fullsized":"2024-05-27T07:09:54.045033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:51.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187236+0000","last_change":"2024-05-27T07:08:36.870668+0000","last_active":"2024-05-27T07:09:54.187236+0000","last_peered":"2024-05-27T07:09:54.187236+0000","last_clean":"2024-05-27T07:09:54.187236+0000","last_became_active":"2024-05-27T07:08:36.869986+0000","last_became_peered":"2024-05-27T07:08:36.869986+0000","last_unstale":"2024-05-27T07:09:54.187236+0000","last_undegraded":"2024-05-27T07:09:54.187236+0000","last_fullsized":"2024-05-27T07:09:54.187236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:59.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'6","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:26.166984+0000","last_change":"2024-05-27T07:08:18.439082+0000","last_active":"2024-05-27T07:10:26.166984+0000","last_peered":"2024-05-27T07:10:26.166984+0000","last_clean":"2024-05-27T07:10:26.166984+0000","last_became_active":"2024-05-27T07:08:18.437844+0000","last_became_peered":"2024-05-27T07:08:18.437844+0000","last_unstale":"2024-05-27T07:10:26.166984+0000","last_undegraded":"2024-05-27T07:10:26.166984+0000","last_fullsized":"2024-05-27T07:10:26.166984+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:38:30.276747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.046436+0000","last_change":"2024-05-27T07:08:22.052308+0000","last_active":"2024-05-27T07:09:54.046436+0000","last_peered":"2024-05-27T07:09:54.046436+0000","last_clean":"2024-05-27T07:09:54.046436+0000","last_became_active":"2024-05-27T07:08:22.051615+0000","last_became_peered":"2024-05-27T07:08:22.051615+0000","last_unstale":"2024-05-27T07:09:54.046436+0000","last_undegraded":"2024-05-27T07:09:54.046436+0000","last_fullsized":"2024-05-27T07:09:54.046436+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:14.241454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.184726+0000","last_change":"2024-05-27T07:08:36.861082+0000","last_active":"2024-05-27T07:09:54.184726+0000","last_peered":"2024-05-27T07:09:54.184726+0000","last_clean":"2024-05-27T07:09:54.184726+0000","last_became_active":"2024-05-27T07:08:36.860548+0000","last_became_peered":"2024-05-27T07:08:36.860548+0000","last_unstale":"2024-05-27T07:09:54.184726+0000","last_undegraded":"2024-05-27T07:09:54.184726+0000","last_fullsized":"2024-05-27T07:09:54.184726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:46:32.446464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043709+0000","last_change":"2024-05-27T07:08:18.018218+0000","last_active":"2024-05-27T07:09:54.043709+0000","last_peered":"2024-05-27T07:09:54.043709+0000","last_clean":"2024-05-27T07:09:54.043709+0000","last_became_active":"2024-05-27T07:08:18.017200+0000","last_became_peered":"2024-05-27T07:08:18.017200+0000","last_unstale":"2024-05-27T07:09:54.043709+0000","last_undegraded":"2024-05-27T07:09:54.043709+0000","last_fullsized":"2024-05-27T07:09:54.043709+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:22.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.052709+0000","last_change":"2024-05-27T07:09:54.052709+0000","last_active":"2024-05-27T07:09:54.052709+0000","last_peered":"2024-05-27T07:09:54.052709+0000","last_clean":"2024-05-27T07:09:54.052709+0000","last_became_active":"2024-05-27T07:09:54.051425+0000","last_became_peered":"2024-05-27T07:09:54.051425+0000","last_unstale":"2024-05-27T07:09:54.052709+0000","last_undegraded":"2024-05-27T07:09:54.052709+0000","last_fullsized":"2024-05-27T07:09:54.052709+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:51:47.166228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188615+0000","last_change":"2024-05-27T07:08:53.979067+0000","last_active":"2024-05-27T07:09:54.188615+0000","last_peered":"2024-05-27T07:09:54.188615+0000","last_clean":"2024-05-27T07:09:54.188615+0000","last_became_active":"2024-05-27T07:08:53.977620+0000","last_became_peered":"2024-05-27T07:08:53.977620+0000","last_unstale":"2024-05-27T07:09:54.188615+0000","last_undegraded":"2024-05-27T07:09:54.188615+0000","last_fullsized":"2024-05-27T07:09:54.188615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:31.619861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041714+0000","last_change":"2024-05-27T07:08:54.001216+0000","last_active":"2024-05-27T07:09:54.041714+0000","last_peered":"2024-05-27T07:09:54.041714+0000","last_clean":"2024-05-27T07:09:54.041714+0000","last_became_active":"2024-05-27T07:08:53.982121+0000","last_became_peered":"2024-05-27T07:08:53.982121+0000","last_unstale":"2024-05-27T07:09:54.041714+0000","last_undegraded":"2024-05-27T07:09:54.041714+0000","last_fullsized":"2024-05-27T07:09:54.041714+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:01.474383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.046191+0000","last_change":"2024-05-27T07:08:22.053334+0000","last_active":"2024-05-27T07:09:54.046191+0000","last_peered":"2024-05-27T07:09:54.046191+0000","last_clean":"2024-05-27T07:09:54.046191+0000","last_became_active":"2024-05-27T07:08:22.051660+0000","last_became_peered":"2024-05-27T07:08:22.051660+0000","last_unstale":"2024-05-27T07:09:54.046191+0000","last_undegraded":"2024-05-27T07:09:54.046191+0000","last_fullsized":"2024-05-27T07:09:54.046191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:07:19.554620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187439+0000","last_change":"2024-05-27T07:08:22.064044+0000","last_active":"2024-05-27T07:09:54.187439+0000","last_peered":"2024-05-27T07:09:54.187439+0000","last_clean":"2024-05-27T07:09:54.187439+0000","last_became_active":"2024-05-27T07:08:22.063539+0000","last_became_peered":"2024-05-27T07:08:22.063539+0000","last_unstale":"2024-05-27T07:09:54.187439+0000","last_undegraded":"2024-05-27T07:09:54.187439+0000","last_fullsized":"2024-05-27T07:09:54.187439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:07.291250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070471+0000","last_change":"2024-05-27T07:08:22.059727+0000","last_active":"2024-05-27T07:09:54.070471+0000","last_peered":"2024-05-27T07:09:54.070471+0000","last_clean":"2024-05-27T07:09:54.070471+0000","last_became_active":"2024-05-27T07:08:22.058933+0000","last_became_peered":"2024-05-27T07:08:22.058933+0000","last_unstale":"2024-05-27T07:09:54.070471+0000","last_undegraded":"2024-05-27T07:09:54.070471+0000","last_fullsized":"2024-05-27T07:09:54.070471+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:14.828031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.069950+0000","last_change":"2024-05-27T07:08:22.044871+0000","last_active":"2024-05-27T07:09:54.069950+0000","last_peered":"2024-05-27T07:09:54.069950+0000","last_clean":"2024-05-27T07:09:54.069950+0000","last_became_active":"2024-05-27T07:08:22.044394+0000","last_became_peered":"2024-05-27T07:08:22.044394+0000","last_unstale":"2024-05-27T07:09:54.069950+0000","last_undegraded":"2024-05-27T07:09:54.069950+0000","last_fullsized":"2024-05-27T07:09:54.069950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:11.151064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185139+0000","last_change":"2024-05-27T07:08:22.048318+0000","last_active":"2024-05-27T07:09:54.185139+0000","last_peered":"2024-05-27T07:09:54.185139+0000","last_clean":"2024-05-27T07:09:54.185139+0000","last_became_active":"2024-05-27T07:08:22.047983+0000","last_became_peered":"2024-05-27T07:08:22.047983+0000","last_unstale":"2024-05-27T07:09:54.185139+0000","last_undegraded":"2024-05-27T07:09:54.185139+0000","last_fullsized":"2024-05-27T07:09:54.185139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:11.479943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041376+0000","last_change":"2024-05-27T07:08:22.056951+0000","last_active":"2024-05-27T07:09:54.041376+0000","last_peered":"2024-05-27T07:09:54.041376+0000","last_clean":"2024-05-27T07:09:54.041376+0000","last_became_active":"2024-05-27T07:08:22.053701+0000","last_became_peered":"2024-05-27T07:08:22.053701+0000","last_unstale":"2024-05-27T07:09:54.041376+0000","last_undegraded":"2024-05-27T07:09:54.041376+0000","last_fullsized":"2024-05-27T07:09:54.041376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:22:31.041730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188030+0000","last_change":"2024-05-27T07:08:22.062745+0000","last_active":"2024-05-27T07:09:54.188030+0000","last_peered":"2024-05-27T07:09:54.188030+0000","last_clean":"2024-05-27T07:09:54.188030+0000","last_became_active":"2024-05-27T07:08:22.062339+0000","last_became_peered":"2024-05-27T07:08:22.062339+0000","last_unstale":"2024-05-27T07:09:54.188030+0000","last_undegraded":"2024-05-27T07:09:54.188030+0000","last_fullsized":"2024-05-27T07:09:54.188030+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:58:02.934021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.071386+0000","last_change":"2024-05-27T07:08:22.052621+0000","last_active":"2024-05-27T07:09:54.071386+0000","last_peered":"2024-05-27T07:09:54.071386+0000","last_clean":"2024-05-27T07:09:54.071386+0000","last_became_active":"2024-05-27T07:08:22.052070+0000","last_became_peered":"2024-05-27T07:08:22.052070+0000","last_unstale":"2024-05-27T07:09:54.071386+0000","last_undegraded":"2024-05-27T07:09:54.071386+0000","last_fullsized":"2024-05-27T07:09:54.071386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:34.491068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062117+0000","last_change":"2024-05-27T07:08:22.055704+0000","last_active":"2024-05-27T07:09:54.062117+0000","last_peered":"2024-05-27T07:09:54.062117+0000","last_clean":"2024-05-27T07:09:54.062117+0000","last_became_active":"2024-05-27T07:08:22.055251+0000","last_became_peered":"2024-05-27T07:08:22.055251+0000","last_unstale":"2024-05-27T07:09:54.062117+0000","last_undegraded":"2024-05-27T07:09:54.062117+0000","last_fullsized":"2024-05-27T07:09:54.062117+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:28.074825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042253+0000","last_change":"2024-05-27T07:08:22.052160+0000","last_active":"2024-05-27T07:09:54.042253+0000","last_peered":"2024-05-27T07:09:54.042253+0000","last_clean":"2024-05-27T07:09:54.042253+0000","last_became_active":"2024-05-27T07:08:22.049495+0000","last_became_peered":"2024-05-27T07:08:22.049495+0000","last_unstale":"2024-05-27T07:09:54.042253+0000","last_undegraded":"2024-05-27T07:09:54.042253+0000","last_fullsized":"2024-05-27T07:09:54.042253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:54:59.595031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060755+0000","last_change":"2024-05-27T07:08:22.053828+0000","last_active":"2024-05-27T07:09:54.060755+0000","last_peered":"2024-05-27T07:09:54.060755+0000","last_clean":"2024-05-27T07:09:54.060755+0000","last_became_active":"2024-05-27T07:08:22.053491+0000","last_became_peered":"2024-05-27T07:08:22.053491+0000","last_unstale":"2024-05-27T07:09:54.060755+0000","last_undegraded":"2024-05-27T07:09:54.060755+0000","last_fullsized":"2024-05-27T07:09:54.060755+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:20.568772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187914+0000","last_change":"2024-05-27T07:08:22.050791+0000","last_active":"2024-05-27T07:09:54.187914+0000","last_peered":"2024-05-27T07:09:54.187914+0000","last_clean":"2024-05-27T07:09:54.187914+0000","last_became_active":"2024-05-27T07:08:22.050323+0000","last_became_peered":"2024-05-27T07:08:22.050323+0000","last_unstale":"2024-05-27T07:09:54.187914+0000","last_undegraded":"2024-05-27T07:09:54.187914+0000","last_fullsized":"2024-05-27T07:09:54.187914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:32.400758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041173+0000","last_change":"2024-05-27T07:08:22.041286+0000","last_active":"2024-05-27T07:09:54.041173+0000","last_peered":"2024-05-27T07:09:54.041173+0000","last_clean":"2024-05-27T07:09:54.041173+0000","last_became_active":"2024-05-27T07:08:22.040811+0000","last_became_peered":"2024-05-27T07:08:22.040811+0000","last_unstale":"2024-05-27T07:09:54.041173+0000","last_undegraded":"2024-05-27T07:09:54.041173+0000","last_fullsized":"2024-05-27T07:09:54.041173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:40:09.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188402+0000","last_change":"2024-05-27T07:08:22.066490+0000","last_active":"2024-05-27T07:09:54.188402+0000","last_peered":"2024-05-27T07:09:54.188402+0000","last_clean":"2024-05-27T07:09:54.188402+0000","last_became_active":"2024-05-27T07:08:22.064779+0000","last_became_peered":"2024-05-27T07:08:22.064779+0000","last_unstale":"2024-05-27T07:09:54.188402+0000","last_undegraded":"2024-05-27T07:09:54.188402+0000","last_fullsized":"2024-05-27T07:09:54.188402+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:23.967856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71340,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":399,"num_read_kb":281,"num_write":249,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678062,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":519752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874293,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1736704,"data_stored":1170010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037819,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1069056,"data_stored":522693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234052,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":518281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430283,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32796,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712356,"statfs":{"total":95995035648,"available":95961452544,"internally_reserved":0,"allocated":1658880,"data_stored":1115472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593812,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":514101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":579267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":515948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1056768,"data_stored":515742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":513809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953569,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":582702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790043,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1110663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:10:32.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:10:32.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:32 smithi132 ceph-mon[35283]: pgmap v459: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-05-27T07:10:32.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:32 smithi179 ceph-mon[35888]: pgmap v459: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-05-27T07:10:32.847 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:33.333 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:33 smithi043 ceph-mon[31531]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:10:33.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:33 smithi132 ceph-mon[35283]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:10:33.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:33 smithi179 ceph-mon[35888]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:10:34.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:34 smithi043 ceph-mon[31531]: pgmap v460: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 852 B/s wr, 9 op/s 2024-05-27T07:10:34.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:34 smithi132 ceph-mon[35283]: pgmap v460: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 852 B/s wr, 9 op/s 2024-05-27T07:10:34.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:34 smithi179 ceph-mon[35888]: pgmap v460: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 852 B/s wr, 9 op/s 2024-05-27T07:10:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.263+0000 7f117da00700 1 -- 172.21.15.43:0/3245134657 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11781054f0 msgr2=0x7f1178105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.263+0000 7f117da00700 1 --2- 172.21.15.43:0/3245134657 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11781054f0 0x7f1178105900 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1168009990 tx=0x7f1168009ca0 comp rx=0 tx=0).stop 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.263+0000 7f117da00700 1 -- 172.21.15.43:0/3245134657 shutdown_connections 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.263+0000 7f117da00700 1 --2- 172.21.15.43:0/3245134657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f117810e8d0 0x7f1178110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.263+0000 7f117da00700 1 --2- 172.21.15.43:0/3245134657 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11781054f0 0x7f1178105900 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.263+0000 7f117da00700 1 --2- 172.21.15.43:0/3245134657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1178105e40 0x7f117810e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.263+0000 7f117da00700 1 -- 172.21.15.43:0/3245134657 >> 172.21.15.43:0/3245134657 conn(0x7f11780fedb0 msgr2=0x7f11781011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.263+0000 7f117da00700 1 -- 172.21.15.43:0/3245134657 shutdown_connections 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.263+0000 7f117da00700 1 -- 172.21.15.43:0/3245134657 wait complete. 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.264+0000 7f117da00700 1 Processor -- start 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.265+0000 7f117da00700 1 -- start start 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.265+0000 7f117da00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11781054f0 0x7f117819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.265+0000 7f117da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1178105e40 0x7f117819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.265+0000 7f117da00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f117810e8d0 0x7f11781a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.265+0000 7f117da00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1178114320 con 0x7f1178105e40 2024-05-27T07:10:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.265+0000 7f117da00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f11781141a0 con 0x7f117810e8d0 2024-05-27T07:10:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.265+0000 7f117da00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f11781144a0 con 0x7f11781054f0 2024-05-27T07:10:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.265+0000 7f1176000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1178105e40 0x7f117819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.266+0000 7f1176000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1178105e40 0x7f117819d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53774/0 (socket says 172.21.15.43:53774) 2024-05-27T07:10:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.266+0000 7f1176000700 1 -- 172.21.15.43:0/2102533564 learned_addr learned my addr 172.21.15.43:0/2102533564 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.266+0000 7f1176a00700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11781054f0 0x7f117819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.266+0000 7f1177400700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f117810e8d0 0x7f11781a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.266+0000 7f1176000700 1 -- 172.21.15.43:0/2102533564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11781054f0 msgr2=0x7f117819cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.266+0000 7f1176000700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11781054f0 0x7f117819cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.266+0000 7f1176000700 1 -- 172.21.15.43:0/2102533564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f117810e8d0 msgr2=0x7f11781a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.266+0000 7f1176000700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f117810e8d0 0x7f11781a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.266+0000 7f1176000700 1 -- 172.21.15.43:0/2102533564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11781a1ea0 con 0x7f1178105e40 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.266+0000 7f1177400700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f117810e8d0 0x7f11781a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.267+0000 7f1176000700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1178105e40 0x7f117819d570 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f116400d950 tx=0x7f116400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.267+0000 7f116f400700 1 -- 172.21.15.43:0/2102533564 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164016070 con 0x7f1178105e40 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.267+0000 7f116f400700 1 -- 172.21.15.43:0/2102533564 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1164009dd0 con 0x7f1178105e40 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.267+0000 7f117da00700 1 -- 172.21.15.43:0/2102533564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11781a2190 con 0x7f1178105e40 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.267+0000 7f116f400700 1 -- 172.21.15.43:0/2102533564 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116400f3c0 con 0x7f1178105e40 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.267+0000 7f117da00700 1 -- 172.21.15.43:0/2102533564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11781a9ab0 con 0x7f1178105e40 2024-05-27T07:10:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.268+0000 7f117da00700 1 -- 172.21.15.43:0/2102533564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11780512b0 con 0x7f1178105e40 2024-05-27T07:10:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.272+0000 7f116f400700 1 -- 172.21.15.43:0/2102533564 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1164019070 con 0x7f1178105e40 2024-05-27T07:10:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.273+0000 7f116f400700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f115c0b2990 0x7f115c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.273+0000 7f116f400700 1 -- 172.21.15.43:0/2102533564 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f11640e2370 con 0x7f1178105e40 2024-05-27T07:10:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.274+0000 7f1176a00700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f115c0b2990 0x7f115c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.274+0000 7f1176a00700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f115c0b2990 0x7f115c0b4ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1168006010 tx=0x7f116800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:35.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.278+0000 7f116f400700 1 -- 172.21.15.43:0/2102533564 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f11640a9b40 con 0x7f1178105e40 2024-05-27T07:10:35.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.513+0000 7f117da00700 1 -- 172.21.15.43:0/2102533564 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f117819e330 con 0x7f115c0b2990 2024-05-27T07:10:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.532+0000 7f116f400700 1 -- 172.21.15.43:0/2102533564 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340413 (secure 0 0 0) 0x7f116807b1d0 con 0x7f115c0b2990 2024-05-27T07:10:35.529 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:10:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 -- 172.21.15.43:0/2102533564 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f115c0b2990 msgr2=0x7f115c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f115c0b2990 0x7f115c0b4ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1168006010 tx=0x7f116800b540 comp rx=0 tx=0).stop 2024-05-27T07:10:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 -- 172.21.15.43:0/2102533564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1178105e40 msgr2=0x7f117819d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1178105e40 0x7f117819d570 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f116400d950 tx=0x7f116400dc60 comp rx=0 tx=0).stop 2024-05-27T07:10:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 -- 172.21.15.43:0/2102533564 shutdown_connections 2024-05-27T07:10:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f115c0b2990 0x7f115c0b4ea0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f117810e8d0 0x7f11781a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11781054f0 0x7f117819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 --2- 172.21.15.43:0/2102533564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1178105e40 0x7f117819d570 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 -- 172.21.15.43:0/2102533564 >> 172.21.15.43:0/2102533564 conn(0x7f11780fedb0 msgr2=0x7f11781032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.541+0000 7f117da00700 1 -- 172.21.15.43:0/2102533564 shutdown_connections 2024-05-27T07:10:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:35.542+0000 7f117da00700 1 -- 172.21.15.43:0/2102533564 wait complete. 2024-05-27T07:10:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-05-27T07:10:36.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:36 smithi043 ceph-mon[31531]: pgmap v461: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 10 op/s 2024-05-27T07:10:36.298 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:36 smithi179 ceph-mon[35888]: pgmap v461: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 10 op/s 2024-05-27T07:10:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:36 smithi132 ceph-mon[35283]: pgmap v461: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 10 op/s 2024-05-27T07:10:36.465 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-05-27T07:10:34.784597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661942,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":505,"num_read_kb":360,"num_write":374,"num_write_kb":2295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953452,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":308,"ondisk_log_size":308,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367672,"kb_used_data":14336,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574152,"statfs":{"total":1151940427776,"available":1151563931648,"internally_reserved":0,"allocated":14680064,"data_stored":8178440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7391,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":69,"num_write":8,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006532"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188611+0000","last_change":"2024-05-27T07:08:22.053540+0000","last_active":"2024-05-27T07:09:54.188611+0000","last_peered":"2024-05-27T07:09:54.188611+0000","last_clean":"2024-05-27T07:09:54.188611+0000","last_became_active":"2024-05-27T07:08:22.051872+0000","last_became_peered":"2024-05-27T07:08:22.051872+0000","last_unstale":"2024-05-27T07:09:54.188611+0000","last_undegraded":"2024-05-27T07:09:54.188611+0000","last_fullsized":"2024-05-27T07:09:54.188611+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:02.787957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185542+0000","last_change":"2024-05-27T07:08:22.049341+0000","last_active":"2024-05-27T07:09:54.185542+0000","last_peered":"2024-05-27T07:09:54.185542+0000","last_clean":"2024-05-27T07:09:54.185542+0000","last_became_active":"2024-05-27T07:08:22.048905+0000","last_became_peered":"2024-05-27T07:08:22.048905+0000","last_unstale":"2024-05-27T07:09:54.185542+0000","last_undegraded":"2024-05-27T07:09:54.185542+0000","last_fullsized":"2024-05-27T07:09:54.185542+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:27.271336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.184788+0000","last_change":"2024-05-27T07:08:22.053925+0000","last_active":"2024-05-27T07:09:54.184788+0000","last_peered":"2024-05-27T07:09:54.184788+0000","last_clean":"2024-05-27T07:09:54.184788+0000","last_became_active":"2024-05-27T07:08:22.053409+0000","last_became_peered":"2024-05-27T07:08:22.053409+0000","last_unstale":"2024-05-27T07:09:54.184788+0000","last_undegraded":"2024-05-27T07:09:54.184788+0000","last_fullsized":"2024-05-27T07:09:54.184788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:07:55.439966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079038+0000","last_change":"2024-05-27T07:08:22.048214+0000","last_active":"2024-05-27T07:09:54.079038+0000","last_peered":"2024-05-27T07:09:54.079038+0000","last_clean":"2024-05-27T07:09:54.079038+0000","last_became_active":"2024-05-27T07:08:22.047791+0000","last_became_peered":"2024-05-27T07:08:22.047791+0000","last_unstale":"2024-05-27T07:09:54.079038+0000","last_undegraded":"2024-05-27T07:09:54.079038+0000","last_fullsized":"2024-05-27T07:09:54.079038+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:23:56.381008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189219+0000","last_change":"2024-05-27T07:08:22.037883+0000","last_active":"2024-05-27T07:09:54.189219+0000","last_peered":"2024-05-27T07:09:54.189219+0000","last_clean":"2024-05-27T07:09:54.189219+0000","last_became_active":"2024-05-27T07:08:22.037089+0000","last_became_peered":"2024-05-27T07:08:22.037089+0000","last_unstale":"2024-05-27T07:09:54.189219+0000","last_undegraded":"2024-05-27T07:09:54.189219+0000","last_fullsized":"2024-05-27T07:09:54.189219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:21.929086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062958+0000","last_change":"2024-05-27T07:08:22.061833+0000","last_active":"2024-05-27T07:09:54.062958+0000","last_peered":"2024-05-27T07:09:54.062958+0000","last_clean":"2024-05-27T07:09:54.062958+0000","last_became_active":"2024-05-27T07:08:22.057692+0000","last_became_peered":"2024-05-27T07:08:22.057692+0000","last_unstale":"2024-05-27T07:09:54.062958+0000","last_undegraded":"2024-05-27T07:09:54.062958+0000","last_fullsized":"2024-05-27T07:09:54.062958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:38:33.656872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044241+0000","last_change":"2024-05-27T07:08:22.047880+0000","last_active":"2024-05-27T07:09:54.044241+0000","last_peered":"2024-05-27T07:09:54.044241+0000","last_clean":"2024-05-27T07:09:54.044241+0000","last_became_active":"2024-05-27T07:08:22.047482+0000","last_became_peered":"2024-05-27T07:08:22.047482+0000","last_unstale":"2024-05-27T07:09:54.044241+0000","last_undegraded":"2024-05-27T07:09:54.044241+0000","last_fullsized":"2024-05-27T07:09:54.044241+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:07.751588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060141+0000","last_change":"2024-05-27T07:08:22.055970+0000","last_active":"2024-05-27T07:09:54.060141+0000","last_peered":"2024-05-27T07:09:54.060141+0000","last_clean":"2024-05-27T07:09:54.060141+0000","last_became_active":"2024-05-27T07:08:22.055591+0000","last_became_peered":"2024-05-27T07:08:22.055591+0000","last_unstale":"2024-05-27T07:09:54.060141+0000","last_undegraded":"2024-05-27T07:09:54.060141+0000","last_fullsized":"2024-05-27T07:09:54.060141+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:58:46.571473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188998+0000","last_change":"2024-05-27T07:08:22.059929+0000","last_active":"2024-05-27T07:09:54.188998+0000","last_peered":"2024-05-27T07:09:54.188998+0000","last_clean":"2024-05-27T07:09:54.188998+0000","last_became_active":"2024-05-27T07:08:22.059118+0000","last_became_peered":"2024-05-27T07:08:22.059118+0000","last_unstale":"2024-05-27T07:09:54.188998+0000","last_undegraded":"2024-05-27T07:09:54.188998+0000","last_fullsized":"2024-05-27T07:09:54.188998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:29.231220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063952+0000","last_change":"2024-05-27T07:08:53.979269+0000","last_active":"2024-05-27T07:09:54.063952+0000","last_peered":"2024-05-27T07:09:54.063952+0000","last_clean":"2024-05-27T07:09:54.063952+0000","last_became_active":"2024-05-27T07:08:53.978433+0000","last_became_peered":"2024-05-27T07:08:53.978433+0000","last_unstale":"2024-05-27T07:09:54.063952+0000","last_undegraded":"2024-05-27T07:09:54.063952+0000","last_fullsized":"2024-05-27T07:09:54.063952+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:57:37.600911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.046188+0000","last_change":"2024-05-27T07:08:22.053499+0000","last_active":"2024-05-27T07:09:54.046188+0000","last_peered":"2024-05-27T07:09:54.046188+0000","last_clean":"2024-05-27T07:09:54.046188+0000","last_became_active":"2024-05-27T07:08:22.052428+0000","last_became_peered":"2024-05-27T07:08:22.052428+0000","last_unstale":"2024-05-27T07:09:54.046188+0000","last_undegraded":"2024-05-27T07:09:54.046188+0000","last_fullsized":"2024-05-27T07:09:54.046188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:04.352262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079297+0000","last_change":"2024-05-27T07:08:22.058882+0000","last_active":"2024-05-27T07:09:54.079297+0000","last_peered":"2024-05-27T07:09:54.079297+0000","last_clean":"2024-05-27T07:09:54.079297+0000","last_became_active":"2024-05-27T07:08:22.057874+0000","last_became_peered":"2024-05-27T07:08:22.057874+0000","last_unstale":"2024-05-27T07:09:54.079297+0000","last_undegraded":"2024-05-27T07:09:54.079297+0000","last_fullsized":"2024-05-27T07:09:54.079297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:52:11.289488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041675+0000","last_change":"2024-05-27T07:08:22.054315+0000","last_active":"2024-05-27T07:09:54.041675+0000","last_peered":"2024-05-27T07:09:54.041675+0000","last_clean":"2024-05-27T07:09:54.041675+0000","last_became_active":"2024-05-27T07:08:22.053944+0000","last_became_peered":"2024-05-27T07:08:22.053944+0000","last_unstale":"2024-05-27T07:09:54.041675+0000","last_undegraded":"2024-05-27T07:09:54.041675+0000","last_fullsized":"2024-05-27T07:09:54.041675+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:43:06.720147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.184713+0000","last_change":"2024-05-27T07:08:22.054649+0000","last_active":"2024-05-27T07:09:54.184713+0000","last_peered":"2024-05-27T07:09:54.184713+0000","last_clean":"2024-05-27T07:09:54.184713+0000","last_became_active":"2024-05-27T07:08:22.051834+0000","last_became_peered":"2024-05-27T07:08:22.051834+0000","last_unstale":"2024-05-27T07:09:54.184713+0000","last_undegraded":"2024-05-27T07:09:54.184713+0000","last_fullsized":"2024-05-27T07:09:54.184713+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:16:56.893458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.184870+0000","last_change":"2024-05-27T07:08:22.052231+0000","last_active":"2024-05-27T07:09:54.184870+0000","last_peered":"2024-05-27T07:09:54.184870+0000","last_clean":"2024-05-27T07:09:54.184870+0000","last_became_active":"2024-05-27T07:08:22.051495+0000","last_became_peered":"2024-05-27T07:08:22.051495+0000","last_unstale":"2024-05-27T07:09:54.184870+0000","last_undegraded":"2024-05-27T07:09:54.184870+0000","last_fullsized":"2024-05-27T07:09:54.184870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:30:21.036874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187151+0000","last_change":"2024-05-27T07:08:22.056847+0000","last_active":"2024-05-27T07:09:54.187151+0000","last_peered":"2024-05-27T07:09:54.187151+0000","last_clean":"2024-05-27T07:09:54.187151+0000","last_became_active":"2024-05-27T07:08:22.056127+0000","last_became_peered":"2024-05-27T07:08:22.056127+0000","last_unstale":"2024-05-27T07:09:54.187151+0000","last_undegraded":"2024-05-27T07:09:54.187151+0000","last_fullsized":"2024-05-27T07:09:54.187151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:04.757452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188092+0000","last_change":"2024-05-27T07:08:22.058078+0000","last_active":"2024-05-27T07:09:54.188092+0000","last_peered":"2024-05-27T07:09:54.188092+0000","last_clean":"2024-05-27T07:09:54.188092+0000","last_became_active":"2024-05-27T07:08:22.056886+0000","last_became_peered":"2024-05-27T07:08:22.056886+0000","last_unstale":"2024-05-27T07:09:54.188092+0000","last_undegraded":"2024-05-27T07:09:54.188092+0000","last_fullsized":"2024-05-27T07:09:54.188092+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.104572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.048468+0000","last_change":"2024-05-27T07:08:22.051492+0000","last_active":"2024-05-27T07:09:54.048468+0000","last_peered":"2024-05-27T07:09:54.048468+0000","last_clean":"2024-05-27T07:09:54.048468+0000","last_became_active":"2024-05-27T07:08:22.051091+0000","last_became_peered":"2024-05-27T07:08:22.051091+0000","last_unstale":"2024-05-27T07:09:54.048468+0000","last_undegraded":"2024-05-27T07:09:54.048468+0000","last_fullsized":"2024-05-27T07:09:54.048468+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:47.816312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.071314+0000","last_change":"2024-05-27T07:08:22.050714+0000","last_active":"2024-05-27T07:09:54.071314+0000","last_peered":"2024-05-27T07:09:54.071314+0000","last_clean":"2024-05-27T07:09:54.071314+0000","last_became_active":"2024-05-27T07:08:22.050322+0000","last_became_peered":"2024-05-27T07:08:22.050322+0000","last_unstale":"2024-05-27T07:09:54.071314+0000","last_undegraded":"2024-05-27T07:09:54.071314+0000","last_fullsized":"2024-05-27T07:09:54.071314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:01.867467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070275+0000","last_change":"2024-05-27T07:08:36.859916+0000","last_active":"2024-05-27T07:09:54.070275+0000","last_peered":"2024-05-27T07:09:54.070275+0000","last_clean":"2024-05-27T07:09:54.070275+0000","last_became_active":"2024-05-27T07:08:36.859429+0000","last_became_peered":"2024-05-27T07:08:36.859429+0000","last_unstale":"2024-05-27T07:09:54.070275+0000","last_undegraded":"2024-05-27T07:09:54.070275+0000","last_fullsized":"2024-05-27T07:09:54.070275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:14.026604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":99,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:31.164194+0000","last_change":"2024-05-27T07:08:18.008912+0000","last_active":"2024-05-27T07:10:31.164194+0000","last_peered":"2024-05-27T07:10:31.164194+0000","last_clean":"2024-05-27T07:10:31.164194+0000","last_became_active":"2024-05-27T07:08:18.007603+0000","last_became_peered":"2024-05-27T07:08:18.007603+0000","last_unstale":"2024-05-27T07:10:31.164194+0000","last_undegraded":"2024-05-27T07:10:31.164194+0000","last_fullsized":"2024-05-27T07:10:31.164194+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:17:40.681754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062991+0000","last_change":"2024-05-27T07:08:22.055426+0000","last_active":"2024-05-27T07:09:54.062991+0000","last_peered":"2024-05-27T07:09:54.062991+0000","last_clean":"2024-05-27T07:09:54.062991+0000","last_became_active":"2024-05-27T07:08:22.055063+0000","last_became_peered":"2024-05-27T07:08:22.055063+0000","last_unstale":"2024-05-27T07:09:54.062991+0000","last_undegraded":"2024-05-27T07:09:54.062991+0000","last_fullsized":"2024-05-27T07:09:54.062991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:06:49.923192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079424+0000","last_change":"2024-05-27T07:08:53.982756+0000","last_active":"2024-05-27T07:09:54.079424+0000","last_peered":"2024-05-27T07:09:54.079424+0000","last_clean":"2024-05-27T07:09:54.079424+0000","last_became_active":"2024-05-27T07:08:53.981816+0000","last_became_peered":"2024-05-27T07:08:53.981816+0000","last_unstale":"2024-05-27T07:09:54.079424+0000","last_undegraded":"2024-05-27T07:09:54.079424+0000","last_fullsized":"2024-05-27T07:09:54.079424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:09.984488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063371+0000","last_change":"2024-05-27T07:08:18.307659+0000","last_active":"2024-05-27T07:09:54.063371+0000","last_peered":"2024-05-27T07:09:54.063371+0000","last_clean":"2024-05-27T07:09:54.063371+0000","last_became_active":"2024-05-27T07:08:18.307239+0000","last_became_peered":"2024-05-27T07:08:18.307239+0000","last_unstale":"2024-05-27T07:09:54.063371+0000","last_undegraded":"2024-05-27T07:09:54.063371+0000","last_fullsized":"2024-05-27T07:09:54.063371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:09:02.816813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185149+0000","last_change":"2024-05-27T07:08:22.050403+0000","last_active":"2024-05-27T07:09:54.185149+0000","last_peered":"2024-05-27T07:09:54.185149+0000","last_clean":"2024-05-27T07:09:54.185149+0000","last_became_active":"2024-05-27T07:08:22.049939+0000","last_became_peered":"2024-05-27T07:08:22.049939+0000","last_unstale":"2024-05-27T07:09:54.185149+0000","last_undegraded":"2024-05-27T07:09:54.185149+0000","last_fullsized":"2024-05-27T07:09:54.185149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:45.857508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.078740+0000","last_change":"2024-05-27T07:08:36.864458+0000","last_active":"2024-05-27T07:09:54.078740+0000","last_peered":"2024-05-27T07:09:54.078740+0000","last_clean":"2024-05-27T07:09:54.078740+0000","last_became_active":"2024-05-27T07:08:36.863894+0000","last_became_peered":"2024-05-27T07:08:36.863894+0000","last_unstale":"2024-05-27T07:09:54.078740+0000","last_undegraded":"2024-05-27T07:09:54.078740+0000","last_fullsized":"2024-05-27T07:09:54.078740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:49.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190260+0000","last_change":"2024-05-27T07:08:18.016051+0000","last_active":"2024-05-27T07:09:54.190260+0000","last_peered":"2024-05-27T07:09:54.190260+0000","last_clean":"2024-05-27T07:09:54.190260+0000","last_became_active":"2024-05-27T07:08:18.015590+0000","last_became_peered":"2024-05-27T07:08:18.015590+0000","last_unstale":"2024-05-27T07:09:54.190260+0000","last_undegraded":"2024-05-27T07:09:54.190260+0000","last_fullsized":"2024-05-27T07:09:54.190260+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:33.396027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188148+0000","last_change":"2024-05-27T07:08:22.055603+0000","last_active":"2024-05-27T07:09:54.188148+0000","last_peered":"2024-05-27T07:09:54.188148+0000","last_clean":"2024-05-27T07:09:54.188148+0000","last_became_active":"2024-05-27T07:08:22.054543+0000","last_became_peered":"2024-05-27T07:08:22.054543+0000","last_unstale":"2024-05-27T07:09:54.188148+0000","last_undegraded":"2024-05-27T07:09:54.188148+0000","last_fullsized":"2024-05-27T07:09:54.188148+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:29:01.738598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.078315+0000","last_change":"2024-05-27T07:08:36.870358+0000","last_active":"2024-05-27T07:09:54.078315+0000","last_peered":"2024-05-27T07:09:54.078315+0000","last_clean":"2024-05-27T07:09:54.078315+0000","last_became_active":"2024-05-27T07:08:36.869457+0000","last_became_peered":"2024-05-27T07:08:36.869457+0000","last_unstale":"2024-05-27T07:09:54.078315+0000","last_undegraded":"2024-05-27T07:09:54.078315+0000","last_fullsized":"2024-05-27T07:09:54.078315+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:50.510098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187456+0000","last_change":"2024-05-27T07:08:53.985562+0000","last_active":"2024-05-27T07:09:54.187456+0000","last_peered":"2024-05-27T07:09:54.187456+0000","last_clean":"2024-05-27T07:09:54.187456+0000","last_became_active":"2024-05-27T07:08:53.977778+0000","last_became_peered":"2024-05-27T07:08:53.977778+0000","last_unstale":"2024-05-27T07:09:54.187456+0000","last_undegraded":"2024-05-27T07:09:54.187456+0000","last_fullsized":"2024-05-27T07:09:54.187456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:44.455508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044017+0000","last_change":"2024-05-27T07:08:22.057622+0000","last_active":"2024-05-27T07:09:54.044017+0000","last_peered":"2024-05-27T07:09:54.044017+0000","last_clean":"2024-05-27T07:09:54.044017+0000","last_became_active":"2024-05-27T07:08:22.056544+0000","last_became_peered":"2024-05-27T07:08:22.056544+0000","last_unstale":"2024-05-27T07:09:54.044017+0000","last_undegraded":"2024-05-27T07:09:54.044017+0000","last_fullsized":"2024-05-27T07:09:54.044017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:38.484560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070946+0000","last_change":"2024-05-27T07:08:36.864984+0000","last_active":"2024-05-27T07:09:54.070946+0000","last_peered":"2024-05-27T07:09:54.070946+0000","last_clean":"2024-05-27T07:09:54.070946+0000","last_became_active":"2024-05-27T07:08:36.864302+0000","last_became_peered":"2024-05-27T07:08:36.864302+0000","last_unstale":"2024-05-27T07:09:54.070946+0000","last_undegraded":"2024-05-27T07:09:54.070946+0000","last_fullsized":"2024-05-27T07:09:54.070946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:44.183562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:28.163382+0000","last_change":"2024-05-27T07:08:18.017918+0000","last_active":"2024-05-27T07:10:28.163382+0000","last_peered":"2024-05-27T07:10:28.163382+0000","last_clean":"2024-05-27T07:10:28.163382+0000","last_became_active":"2024-05-27T07:08:18.016174+0000","last_became_peered":"2024-05-27T07:08:18.016174+0000","last_unstale":"2024-05-27T07:10:28.163382+0000","last_undegraded":"2024-05-27T07:10:28.163382+0000","last_fullsized":"2024-05-27T07:10:28.163382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:21.906187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189895+0000","last_change":"2024-05-27T07:08:22.043172+0000","last_active":"2024-05-27T07:09:54.189895+0000","last_peered":"2024-05-27T07:09:54.189895+0000","last_clean":"2024-05-27T07:09:54.189895+0000","last_became_active":"2024-05-27T07:08:22.041399+0000","last_became_peered":"2024-05-27T07:08:22.041399+0000","last_unstale":"2024-05-27T07:09:54.189895+0000","last_undegraded":"2024-05-27T07:09:54.189895+0000","last_fullsized":"2024-05-27T07:09:54.189895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:19.651330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070122+0000","last_change":"2024-05-27T07:08:36.866352+0000","last_active":"2024-05-27T07:09:54.070122+0000","last_peered":"2024-05-27T07:09:54.070122+0000","last_clean":"2024-05-27T07:09:54.070122+0000","last_became_active":"2024-05-27T07:08:36.865447+0000","last_became_peered":"2024-05-27T07:08:36.865447+0000","last_unstale":"2024-05-27T07:09:54.070122+0000","last_undegraded":"2024-05-27T07:09:54.070122+0000","last_fullsized":"2024-05-27T07:09:54.070122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:55:42.634374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062875+0000","last_change":"2024-05-27T07:08:18.022142+0000","last_active":"2024-05-27T07:09:54.062875+0000","last_peered":"2024-05-27T07:09:54.062875+0000","last_clean":"2024-05-27T07:09:54.062875+0000","last_became_active":"2024-05-27T07:08:18.021453+0000","last_became_peered":"2024-05-27T07:08:18.021453+0000","last_unstale":"2024-05-27T07:09:54.062875+0000","last_undegraded":"2024-05-27T07:09:54.062875+0000","last_fullsized":"2024-05-27T07:09:54.062875+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:01:09.049440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043456+0000","last_change":"2024-05-27T07:08:22.055742+0000","last_active":"2024-05-27T07:09:54.043456+0000","last_peered":"2024-05-27T07:09:54.043456+0000","last_clean":"2024-05-27T07:09:54.043456+0000","last_became_active":"2024-05-27T07:08:22.052077+0000","last_became_peered":"2024-05-27T07:08:22.052077+0000","last_unstale":"2024-05-27T07:09:54.043456+0000","last_undegraded":"2024-05-27T07:09:54.043456+0000","last_fullsized":"2024-05-27T07:09:54.043456+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:09:13.418027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187972+0000","last_change":"2024-05-27T07:08:36.874225+0000","last_active":"2024-05-27T07:09:54.187972+0000","last_peered":"2024-05-27T07:09:54.187972+0000","last_clean":"2024-05-27T07:09:54.187972+0000","last_became_active":"2024-05-27T07:08:36.873745+0000","last_became_peered":"2024-05-27T07:08:36.873745+0000","last_unstale":"2024-05-27T07:09:54.187972+0000","last_undegraded":"2024-05-27T07:09:54.187972+0000","last_fullsized":"2024-05-27T07:09:54.187972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:40.633667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.040832+0000","last_change":"2024-05-27T07:08:18.009678+0000","last_active":"2024-05-27T07:09:54.040832+0000","last_peered":"2024-05-27T07:09:54.040832+0000","last_clean":"2024-05-27T07:09:54.040832+0000","last_became_active":"2024-05-27T07:08:18.006502+0000","last_became_peered":"2024-05-27T07:08:18.006502+0000","last_unstale":"2024-05-27T07:09:54.040832+0000","last_undegraded":"2024-05-27T07:09:54.040832+0000","last_fullsized":"2024-05-27T07:09:54.040832+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:37.457044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060622+0000","last_change":"2024-05-27T07:08:22.047302+0000","last_active":"2024-05-27T07:09:54.060622+0000","last_peered":"2024-05-27T07:09:54.060622+0000","last_clean":"2024-05-27T07:09:54.060622+0000","last_became_active":"2024-05-27T07:08:22.045932+0000","last_became_peered":"2024-05-27T07:08:22.045932+0000","last_unstale":"2024-05-27T07:09:54.060622+0000","last_undegraded":"2024-05-27T07:09:54.060622+0000","last_fullsized":"2024-05-27T07:09:54.060622+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:27.129539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185351+0000","last_change":"2024-05-27T07:08:36.870512+0000","last_active":"2024-05-27T07:09:54.185351+0000","last_peered":"2024-05-27T07:09:54.185351+0000","last_clean":"2024-05-27T07:09:54.185351+0000","last_became_active":"2024-05-27T07:08:36.870061+0000","last_became_peered":"2024-05-27T07:08:36.870061+0000","last_unstale":"2024-05-27T07:09:54.185351+0000","last_undegraded":"2024-05-27T07:09:54.185351+0000","last_fullsized":"2024-05-27T07:09:54.185351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:03.608758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042027+0000","last_change":"2024-05-27T07:08:18.018420+0000","last_active":"2024-05-27T07:09:54.042027+0000","last_peered":"2024-05-27T07:09:54.042027+0000","last_clean":"2024-05-27T07:09:54.042027+0000","last_became_active":"2024-05-27T07:08:18.017567+0000","last_became_peered":"2024-05-27T07:08:18.017567+0000","last_unstale":"2024-05-27T07:09:54.042027+0000","last_undegraded":"2024-05-27T07:09:54.042027+0000","last_fullsized":"2024-05-27T07:09:54.042027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:37.214408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.047241+0000","last_change":"2024-05-27T07:08:22.050943+0000","last_active":"2024-05-27T07:09:54.047241+0000","last_peered":"2024-05-27T07:09:54.047241+0000","last_clean":"2024-05-27T07:09:54.047241+0000","last_became_active":"2024-05-27T07:08:22.047706+0000","last_became_peered":"2024-05-27T07:08:22.047706+0000","last_unstale":"2024-05-27T07:09:54.047241+0000","last_undegraded":"2024-05-27T07:09:54.047241+0000","last_fullsized":"2024-05-27T07:09:54.047241+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:52:09.739528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060515+0000","last_change":"2024-05-27T07:08:36.867487+0000","last_active":"2024-05-27T07:09:54.060515+0000","last_peered":"2024-05-27T07:09:54.060515+0000","last_clean":"2024-05-27T07:09:54.060515+0000","last_became_active":"2024-05-27T07:08:36.867032+0000","last_became_peered":"2024-05-27T07:08:36.867032+0000","last_unstale":"2024-05-27T07:09:54.060515+0000","last_undegraded":"2024-05-27T07:09:54.060515+0000","last_fullsized":"2024-05-27T07:09:54.060515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:26:24.912382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:29.165194+0000","last_change":"2024-05-27T07:08:18.020168+0000","last_active":"2024-05-27T07:10:29.165194+0000","last_peered":"2024-05-27T07:10:29.165194+0000","last_clean":"2024-05-27T07:10:29.165194+0000","last_became_active":"2024-05-27T07:08:18.018309+0000","last_became_peered":"2024-05-27T07:08:18.018309+0000","last_unstale":"2024-05-27T07:10:29.165194+0000","last_undegraded":"2024-05-27T07:10:29.165194+0000","last_fullsized":"2024-05-27T07:10:29.165194+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:48:16.839510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188891+0000","last_change":"2024-05-27T07:08:22.054888+0000","last_active":"2024-05-27T07:09:54.188891+0000","last_peered":"2024-05-27T07:09:54.188891+0000","last_clean":"2024-05-27T07:09:54.188891+0000","last_became_active":"2024-05-27T07:08:22.054417+0000","last_became_peered":"2024-05-27T07:08:22.054417+0000","last_unstale":"2024-05-27T07:09:54.188891+0000","last_undegraded":"2024-05-27T07:09:54.188891+0000","last_fullsized":"2024-05-27T07:09:54.188891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:43:00.866755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.047828+0000","last_change":"2024-05-27T07:08:36.854516+0000","last_active":"2024-05-27T07:09:54.047828+0000","last_peered":"2024-05-27T07:09:54.047828+0000","last_clean":"2024-05-27T07:09:54.047828+0000","last_became_active":"2024-05-27T07:08:36.853978+0000","last_became_peered":"2024-05-27T07:08:36.853978+0000","last_unstale":"2024-05-27T07:09:54.047828+0000","last_undegraded":"2024-05-27T07:09:54.047828+0000","last_fullsized":"2024-05-27T07:09:54.047828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:45.299100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045324+0000","last_change":"2024-05-27T07:08:18.010539+0000","last_active":"2024-05-27T07:09:54.045324+0000","last_peered":"2024-05-27T07:09:54.045324+0000","last_clean":"2024-05-27T07:09:54.045324+0000","last_became_active":"2024-05-27T07:08:18.009643+0000","last_became_peered":"2024-05-27T07:08:18.009643+0000","last_unstale":"2024-05-27T07:09:54.045324+0000","last_undegraded":"2024-05-27T07:09:54.045324+0000","last_fullsized":"2024-05-27T07:09:54.045324+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:48:40.335403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185027+0000","last_change":"2024-05-27T07:08:53.980165+0000","last_active":"2024-05-27T07:09:54.185027+0000","last_peered":"2024-05-27T07:09:54.185027+0000","last_clean":"2024-05-27T07:09:54.185027+0000","last_became_active":"2024-05-27T07:08:53.978565+0000","last_became_peered":"2024-05-27T07:08:53.978565+0000","last_unstale":"2024-05-27T07:09:54.185027+0000","last_undegraded":"2024-05-27T07:09:54.185027+0000","last_fullsized":"2024-05-27T07:09:54.185027+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:56:33.513961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079469+0000","last_change":"2024-05-27T07:08:36.857814+0000","last_active":"2024-05-27T07:09:54.079469+0000","last_peered":"2024-05-27T07:09:54.079469+0000","last_clean":"2024-05-27T07:09:54.079469+0000","last_became_active":"2024-05-27T07:08:36.856790+0000","last_became_peered":"2024-05-27T07:08:36.856790+0000","last_unstale":"2024-05-27T07:09:54.079469+0000","last_undegraded":"2024-05-27T07:09:54.079469+0000","last_fullsized":"2024-05-27T07:09:54.079469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:39:10.322261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043460+0000","last_change":"2024-05-27T07:08:18.308032+0000","last_active":"2024-05-27T07:09:54.043460+0000","last_peered":"2024-05-27T07:09:54.043460+0000","last_clean":"2024-05-27T07:09:54.043460+0000","last_became_active":"2024-05-27T07:08:18.306867+0000","last_became_peered":"2024-05-27T07:08:18.306867+0000","last_unstale":"2024-05-27T07:09:54.043460+0000","last_undegraded":"2024-05-27T07:09:54.043460+0000","last_fullsized":"2024-05-27T07:09:54.043460+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:37.816237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187935+0000","last_change":"2024-05-27T07:08:22.057940+0000","last_active":"2024-05-27T07:09:54.187935+0000","last_peered":"2024-05-27T07:09:54.187935+0000","last_clean":"2024-05-27T07:09:54.187935+0000","last_became_active":"2024-05-27T07:08:22.055109+0000","last_became_peered":"2024-05-27T07:08:22.055109+0000","last_unstale":"2024-05-27T07:09:54.187935+0000","last_undegraded":"2024-05-27T07:09:54.187935+0000","last_fullsized":"2024-05-27T07:09:54.187935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:46.905064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062562+0000","last_change":"2024-05-27T07:08:53.980209+0000","last_active":"2024-05-27T07:09:54.062562+0000","last_peered":"2024-05-27T07:09:54.062562+0000","last_clean":"2024-05-27T07:09:54.062562+0000","last_became_active":"2024-05-27T07:08:53.979353+0000","last_became_peered":"2024-05-27T07:08:53.979353+0000","last_unstale":"2024-05-27T07:09:54.062562+0000","last_undegraded":"2024-05-27T07:09:54.062562+0000","last_fullsized":"2024-05-27T07:09:54.062562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:06.163563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:29.184680+0000","last_change":"2024-05-27T07:08:53.996491+0000","last_active":"2024-05-27T07:10:29.184680+0000","last_peered":"2024-05-27T07:10:29.184680+0000","last_clean":"2024-05-27T07:10:29.184680+0000","last_became_active":"2024-05-27T07:08:53.976013+0000","last_became_peered":"2024-05-27T07:08:53.976013+0000","last_unstale":"2024-05-27T07:10:29.184680+0000","last_undegraded":"2024-05-27T07:10:29.184680+0000","last_fullsized":"2024-05-27T07:10:29.184680+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:44:16.592270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64136,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":83,"num_read_kb":104,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189909+0000","last_change":"2024-05-27T07:08:22.057231+0000","last_active":"2024-05-27T07:09:54.189909+0000","last_peered":"2024-05-27T07:09:54.189909+0000","last_clean":"2024-05-27T07:09:54.189909+0000","last_became_active":"2024-05-27T07:08:22.056837+0000","last_became_peered":"2024-05-27T07:08:22.056837+0000","last_unstale":"2024-05-27T07:09:54.189909+0000","last_undegraded":"2024-05-27T07:09:54.189909+0000","last_fullsized":"2024-05-27T07:09:54.189909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:10:28.645103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044380+0000","last_change":"2024-05-27T07:08:36.870307+0000","last_active":"2024-05-27T07:09:54.044380+0000","last_peered":"2024-05-27T07:09:54.044380+0000","last_clean":"2024-05-27T07:09:54.044380+0000","last_became_active":"2024-05-27T07:08:36.869390+0000","last_became_peered":"2024-05-27T07:08:36.869390+0000","last_unstale":"2024-05-27T07:09:54.044380+0000","last_undegraded":"2024-05-27T07:09:54.044380+0000","last_fullsized":"2024-05-27T07:09:54.044380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:43:45.228885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070573+0000","last_change":"2024-05-27T07:08:18.014458+0000","last_active":"2024-05-27T07:09:54.070573+0000","last_peered":"2024-05-27T07:09:54.070573+0000","last_clean":"2024-05-27T07:09:54.070573+0000","last_became_active":"2024-05-27T07:08:18.013365+0000","last_became_peered":"2024-05-27T07:08:18.013365+0000","last_unstale":"2024-05-27T07:09:54.070573+0000","last_undegraded":"2024-05-27T07:09:54.070573+0000","last_fullsized":"2024-05-27T07:09:54.070573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:41.352618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041318+0000","last_change":"2024-05-27T07:08:22.043354+0000","last_active":"2024-05-27T07:09:54.041318+0000","last_peered":"2024-05-27T07:09:54.041318+0000","last_clean":"2024-05-27T07:09:54.041318+0000","last_became_active":"2024-05-27T07:08:22.042876+0000","last_became_peered":"2024-05-27T07:08:22.042876+0000","last_unstale":"2024-05-27T07:09:54.041318+0000","last_undegraded":"2024-05-27T07:09:54.041318+0000","last_fullsized":"2024-05-27T07:09:54.041318+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:52.575568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187862+0000","last_change":"2024-05-27T07:08:36.869579+0000","last_active":"2024-05-27T07:09:54.187862+0000","last_peered":"2024-05-27T07:09:54.187862+0000","last_clean":"2024-05-27T07:09:54.187862+0000","last_became_active":"2024-05-27T07:08:36.869055+0000","last_became_peered":"2024-05-27T07:08:36.869055+0000","last_unstale":"2024-05-27T07:09:54.187862+0000","last_undegraded":"2024-05-27T07:09:54.187862+0000","last_fullsized":"2024-05-27T07:09:54.187862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.870079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043959+0000","last_change":"2024-05-27T07:08:18.015322+0000","last_active":"2024-05-27T07:09:54.043959+0000","last_peered":"2024-05-27T07:09:54.043959+0000","last_clean":"2024-05-27T07:09:54.043959+0000","last_became_active":"2024-05-27T07:08:18.014535+0000","last_became_peered":"2024-05-27T07:08:18.014535+0000","last_unstale":"2024-05-27T07:09:54.043959+0000","last_undegraded":"2024-05-27T07:09:54.043959+0000","last_fullsized":"2024-05-27T07:09:54.043959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:35.116140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045527+0000","last_change":"2024-05-27T07:08:22.056181+0000","last_active":"2024-05-27T07:09:54.045527+0000","last_peered":"2024-05-27T07:09:54.045527+0000","last_clean":"2024-05-27T07:09:54.045527+0000","last_became_active":"2024-05-27T07:08:22.055755+0000","last_became_peered":"2024-05-27T07:08:22.055755+0000","last_unstale":"2024-05-27T07:09:54.045527+0000","last_undegraded":"2024-05-27T07:09:54.045527+0000","last_fullsized":"2024-05-27T07:09:54.045527+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:38:35.634116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.048385+0000","last_change":"2024-05-27T07:08:36.863184+0000","last_active":"2024-05-27T07:09:54.048385+0000","last_peered":"2024-05-27T07:09:54.048385+0000","last_clean":"2024-05-27T07:09:54.048385+0000","last_became_active":"2024-05-27T07:08:36.862760+0000","last_became_peered":"2024-05-27T07:08:36.862760+0000","last_unstale":"2024-05-27T07:09:54.048385+0000","last_undegraded":"2024-05-27T07:09:54.048385+0000","last_fullsized":"2024-05-27T07:09:54.048385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:48.379157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:30.190806+0000","last_change":"2024-05-27T07:08:18.019119+0000","last_active":"2024-05-27T07:10:30.190806+0000","last_peered":"2024-05-27T07:10:30.190806+0000","last_clean":"2024-05-27T07:10:30.190806+0000","last_became_active":"2024-05-27T07:08:18.018408+0000","last_became_peered":"2024-05-27T07:08:18.018408+0000","last_unstale":"2024-05-27T07:10:30.190806+0000","last_undegraded":"2024-05-27T07:10:30.190806+0000","last_fullsized":"2024-05-27T07:10:30.190806+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:19.978328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045094+0000","last_change":"2024-05-27T07:08:22.059464+0000","last_active":"2024-05-27T07:09:54.045094+0000","last_peered":"2024-05-27T07:09:54.045094+0000","last_clean":"2024-05-27T07:09:54.045094+0000","last_became_active":"2024-05-27T07:08:22.058890+0000","last_became_peered":"2024-05-27T07:08:22.058890+0000","last_unstale":"2024-05-27T07:09:54.045094+0000","last_undegraded":"2024-05-27T07:09:54.045094+0000","last_fullsized":"2024-05-27T07:09:54.045094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:30.122552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041831+0000","last_change":"2024-05-27T07:08:36.865884+0000","last_active":"2024-05-27T07:09:54.041831+0000","last_peered":"2024-05-27T07:09:54.041831+0000","last_clean":"2024-05-27T07:09:54.041831+0000","last_became_active":"2024-05-27T07:08:36.865443+0000","last_became_peered":"2024-05-27T07:08:36.865443+0000","last_unstale":"2024-05-27T07:09:54.041831+0000","last_undegraded":"2024-05-27T07:09:54.041831+0000","last_fullsized":"2024-05-27T07:09:54.041831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:47:15.026632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.186967+0000","last_change":"2024-05-27T07:08:53.994809+0000","last_active":"2024-05-27T07:09:54.186967+0000","last_peered":"2024-05-27T07:09:54.186967+0000","last_clean":"2024-05-27T07:09:54.186967+0000","last_became_active":"2024-05-27T07:08:53.977628+0000","last_became_peered":"2024-05-27T07:08:53.977628+0000","last_unstale":"2024-05-27T07:09:54.186967+0000","last_undegraded":"2024-05-27T07:09:54.186967+0000","last_fullsized":"2024-05-27T07:09:54.186967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:13:37.176319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042726+0000","last_change":"2024-05-27T07:08:22.050733+0000","last_active":"2024-05-27T07:09:54.042726+0000","last_peered":"2024-05-27T07:09:54.042726+0000","last_clean":"2024-05-27T07:09:54.042726+0000","last_became_active":"2024-05-27T07:08:22.050321+0000","last_became_peered":"2024-05-27T07:08:22.050321+0000","last_unstale":"2024-05-27T07:09:54.042726+0000","last_undegraded":"2024-05-27T07:09:54.042726+0000","last_fullsized":"2024-05-27T07:09:54.042726+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:44.590595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042726+0000","last_change":"2024-05-27T07:08:36.867784+0000","last_active":"2024-05-27T07:09:54.042726+0000","last_peered":"2024-05-27T07:09:54.042726+0000","last_clean":"2024-05-27T07:09:54.042726+0000","last_became_active":"2024-05-27T07:08:36.867290+0000","last_became_peered":"2024-05-27T07:08:36.867290+0000","last_unstale":"2024-05-27T07:09:54.042726+0000","last_undegraded":"2024-05-27T07:09:54.042726+0000","last_fullsized":"2024-05-27T07:09:54.042726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:44:07.649640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190834+0000","last_change":"2024-05-27T07:08:18.013193+0000","last_active":"2024-05-27T07:09:54.190834+0000","last_peered":"2024-05-27T07:09:54.190834+0000","last_clean":"2024-05-27T07:09:54.190834+0000","last_became_active":"2024-05-27T07:08:18.011767+0000","last_became_peered":"2024-05-27T07:08:18.011767+0000","last_unstale":"2024-05-27T07:09:54.190834+0000","last_undegraded":"2024-05-27T07:09:54.190834+0000","last_fullsized":"2024-05-27T07:09:54.190834+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:15:03.679719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187747+0000","last_change":"2024-05-27T07:08:22.067145+0000","last_active":"2024-05-27T07:09:54.187747+0000","last_peered":"2024-05-27T07:09:54.187747+0000","last_clean":"2024-05-27T07:09:54.187747+0000","last_became_active":"2024-05-27T07:08:22.066750+0000","last_became_peered":"2024-05-27T07:08:22.066750+0000","last_unstale":"2024-05-27T07:09:54.187747+0000","last_undegraded":"2024-05-27T07:09:54.187747+0000","last_fullsized":"2024-05-27T07:09:54.187747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:41:35.333311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189061+0000","last_change":"2024-05-27T07:08:53.982497+0000","last_active":"2024-05-27T07:09:54.189061+0000","last_peered":"2024-05-27T07:09:54.189061+0000","last_clean":"2024-05-27T07:09:54.189061+0000","last_became_active":"2024-05-27T07:08:53.981920+0000","last_became_peered":"2024-05-27T07:08:53.981920+0000","last_unstale":"2024-05-27T07:09:54.189061+0000","last_undegraded":"2024-05-27T07:09:54.189061+0000","last_fullsized":"2024-05-27T07:09:54.189061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:43:40.095357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.048681+0000","last_change":"2024-05-27T07:08:18.308330+0000","last_active":"2024-05-27T07:09:54.048681+0000","last_peered":"2024-05-27T07:09:54.048681+0000","last_clean":"2024-05-27T07:09:54.048681+0000","last_became_active":"2024-05-27T07:08:18.306961+0000","last_became_peered":"2024-05-27T07:08:18.306961+0000","last_unstale":"2024-05-27T07:09:54.048681+0000","last_undegraded":"2024-05-27T07:09:54.048681+0000","last_fullsized":"2024-05-27T07:09:54.048681+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:27.875936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063450+0000","last_change":"2024-05-27T07:08:22.047562+0000","last_active":"2024-05-27T07:09:54.063450+0000","last_peered":"2024-05-27T07:09:54.063450+0000","last_clean":"2024-05-27T07:09:54.063450+0000","last_became_active":"2024-05-27T07:08:22.046977+0000","last_became_peered":"2024-05-27T07:08:22.046977+0000","last_unstale":"2024-05-27T07:09:54.063450+0000","last_undegraded":"2024-05-27T07:09:54.063450+0000","last_fullsized":"2024-05-27T07:09:54.063450+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:49:43.376617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041023+0000","last_change":"2024-05-27T07:08:36.867382+0000","last_active":"2024-05-27T07:09:54.041023+0000","last_peered":"2024-05-27T07:09:54.041023+0000","last_clean":"2024-05-27T07:09:54.041023+0000","last_became_active":"2024-05-27T07:08:36.866296+0000","last_became_peered":"2024-05-27T07:08:36.866296+0000","last_unstale":"2024-05-27T07:09:54.041023+0000","last_undegraded":"2024-05-27T07:09:54.041023+0000","last_fullsized":"2024-05-27T07:09:54.041023+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:14.305169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:29.159813+0000","last_change":"2024-05-27T07:08:18.012031+0000","last_active":"2024-05-27T07:10:29.159813+0000","last_peered":"2024-05-27T07:10:29.159813+0000","last_clean":"2024-05-27T07:10:29.159813+0000","last_became_active":"2024-05-27T07:08:18.009593+0000","last_became_peered":"2024-05-27T07:08:18.009593+0000","last_unstale":"2024-05-27T07:10:29.159813+0000","last_undegraded":"2024-05-27T07:10:29.159813+0000","last_fullsized":"2024-05-27T07:10:29.159813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:25:22.316575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063203+0000","last_change":"2024-05-27T07:08:22.061136+0000","last_active":"2024-05-27T07:09:54.063203+0000","last_peered":"2024-05-27T07:09:54.063203+0000","last_clean":"2024-05-27T07:09:54.063203+0000","last_became_active":"2024-05-27T07:08:22.060735+0000","last_became_peered":"2024-05-27T07:08:22.060735+0000","last_unstale":"2024-05-27T07:09:54.063203+0000","last_undegraded":"2024-05-27T07:09:54.063203+0000","last_fullsized":"2024-05-27T07:09:54.063203+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:50.392130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041651+0000","last_change":"2024-05-27T07:08:36.932187+0000","last_active":"2024-05-27T07:09:54.041651+0000","last_peered":"2024-05-27T07:09:54.041651+0000","last_clean":"2024-05-27T07:09:54.041651+0000","last_became_active":"2024-05-27T07:08:36.931214+0000","last_became_peered":"2024-05-27T07:08:36.931214+0000","last_unstale":"2024-05-27T07:09:54.041651+0000","last_undegraded":"2024-05-27T07:09:54.041651+0000","last_fullsized":"2024-05-27T07:09:54.041651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:20:59.645352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.083887+0000","last_change":"2024-05-27T07:09:54.082674+0000","last_active":"2024-05-27T07:09:54.083887+0000","last_peered":"2024-05-27T07:09:54.083887+0000","last_clean":"2024-05-27T07:09:54.083887+0000","last_became_active":"2024-05-27T07:09:54.081616+0000","last_became_peered":"2024-05-27T07:09:54.081616+0000","last_unstale":"2024-05-27T07:09:54.083887+0000","last_undegraded":"2024-05-27T07:09:54.083887+0000","last_fullsized":"2024-05-27T07:09:54.083887+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:38:11.230026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:32.196133+0000","last_change":"2024-05-27T07:08:18.306439+0000","last_active":"2024-05-27T07:10:32.196133+0000","last_peered":"2024-05-27T07:10:32.196133+0000","last_clean":"2024-05-27T07:10:32.196133+0000","last_became_active":"2024-05-27T07:08:18.305978+0000","last_became_peered":"2024-05-27T07:08:18.305978+0000","last_unstale":"2024-05-27T07:10:32.196133+0000","last_undegraded":"2024-05-27T07:10:32.196133+0000","last_fullsized":"2024-05-27T07:10:32.196133+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:20:22.648922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.046025+0000","last_change":"2024-05-27T07:08:36.866247+0000","last_active":"2024-05-27T07:09:54.046025+0000","last_peered":"2024-05-27T07:09:54.046025+0000","last_clean":"2024-05-27T07:09:54.046025+0000","last_became_active":"2024-05-27T07:08:36.865516+0000","last_became_peered":"2024-05-27T07:08:36.865516+0000","last_unstale":"2024-05-27T07:09:54.046025+0000","last_undegraded":"2024-05-27T07:09:54.046025+0000","last_fullsized":"2024-05-27T07:09:54.046025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:39:41.235352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.061812+0000","last_change":"2024-05-27T07:08:22.045312+0000","last_active":"2024-05-27T07:09:54.061812+0000","last_peered":"2024-05-27T07:09:54.061812+0000","last_clean":"2024-05-27T07:09:54.061812+0000","last_became_active":"2024-05-27T07:08:22.044724+0000","last_became_peered":"2024-05-27T07:08:22.044724+0000","last_unstale":"2024-05-27T07:09:54.061812+0000","last_undegraded":"2024-05-27T07:09:54.061812+0000","last_fullsized":"2024-05-27T07:09:54.061812+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:05.487794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079152+0000","last_change":"2024-05-27T07:08:18.019333+0000","last_active":"2024-05-27T07:09:54.079152+0000","last_peered":"2024-05-27T07:09:54.079152+0000","last_clean":"2024-05-27T07:09:54.079152+0000","last_became_active":"2024-05-27T07:08:18.018816+0000","last_became_peered":"2024-05-27T07:08:18.018816+0000","last_unstale":"2024-05-27T07:09:54.079152+0000","last_undegraded":"2024-05-27T07:09:54.079152+0000","last_fullsized":"2024-05-27T07:09:54.079152+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:09.421346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044348+0000","last_change":"2024-05-27T07:06:16.865587+0000","last_active":"2024-05-27T07:09:54.044348+0000","last_peered":"2024-05-27T07:09:54.044348+0000","last_clean":"2024-05-27T07:09:54.044348+0000","last_became_active":"2024-05-27T07:06:16.850646+0000","last_became_peered":"2024-05-27T07:06:16.850646+0000","last_unstale":"2024-05-27T07:09:54.044348+0000","last_undegraded":"2024-05-27T07:09:54.044348+0000","last_fullsized":"2024-05-27T07:09:54.044348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_clean_scrub_stamp":"2024-05-27T07:01:14.931400+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:37.272619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043593+0000","last_change":"2024-05-27T07:08:36.934696+0000","last_active":"2024-05-27T07:09:54.043593+0000","last_peered":"2024-05-27T07:09:54.043593+0000","last_clean":"2024-05-27T07:09:54.043593+0000","last_became_active":"2024-05-27T07:08:36.934008+0000","last_became_peered":"2024-05-27T07:08:36.934008+0000","last_unstale":"2024-05-27T07:09:54.043593+0000","last_undegraded":"2024-05-27T07:09:54.043593+0000","last_fullsized":"2024-05-27T07:09:54.043593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:39.099180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.040590+0000","last_change":"2024-05-27T07:08:22.054479+0000","last_active":"2024-05-27T07:09:54.040590+0000","last_peered":"2024-05-27T07:09:54.040590+0000","last_clean":"2024-05-27T07:09:54.040590+0000","last_became_active":"2024-05-27T07:08:22.054127+0000","last_became_peered":"2024-05-27T07:08:22.054127+0000","last_unstale":"2024-05-27T07:09:54.040590+0000","last_undegraded":"2024-05-27T07:09:54.040590+0000","last_fullsized":"2024-05-27T07:09:54.040590+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:37:25.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045671+0000","last_change":"2024-05-27T07:08:18.010944+0000","last_active":"2024-05-27T07:09:54.045671+0000","last_peered":"2024-05-27T07:09:54.045671+0000","last_clean":"2024-05-27T07:09:54.045671+0000","last_became_active":"2024-05-27T07:08:18.009816+0000","last_became_peered":"2024-05-27T07:08:18.009816+0000","last_unstale":"2024-05-27T07:09:54.045671+0000","last_undegraded":"2024-05-27T07:09:54.045671+0000","last_fullsized":"2024-05-27T07:09:54.045671+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:34:26.061787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187383+0000","last_change":"2024-05-27T07:08:36.870618+0000","last_active":"2024-05-27T07:09:54.187383+0000","last_peered":"2024-05-27T07:09:54.187383+0000","last_clean":"2024-05-27T07:09:54.187383+0000","last_became_active":"2024-05-27T07:08:36.869799+0000","last_became_peered":"2024-05-27T07:08:36.869799+0000","last_unstale":"2024-05-27T07:09:54.187383+0000","last_undegraded":"2024-05-27T07:09:54.187383+0000","last_fullsized":"2024-05-27T07:09:54.187383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:53.094337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190088+0000","last_change":"2024-05-27T07:08:18.308834+0000","last_active":"2024-05-27T07:09:54.190088+0000","last_peered":"2024-05-27T07:09:54.190088+0000","last_clean":"2024-05-27T07:09:54.190088+0000","last_became_active":"2024-05-27T07:08:18.306953+0000","last_became_peered":"2024-05-27T07:08:18.306953+0000","last_unstale":"2024-05-27T07:09:54.190088+0000","last_undegraded":"2024-05-27T07:09:54.190088+0000","last_fullsized":"2024-05-27T07:09:54.190088+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:26:14.747631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041301+0000","last_change":"2024-05-27T07:08:22.048219+0000","last_active":"2024-05-27T07:09:54.041301+0000","last_peered":"2024-05-27T07:09:54.041301+0000","last_clean":"2024-05-27T07:09:54.041301+0000","last_became_active":"2024-05-27T07:08:22.047848+0000","last_became_peered":"2024-05-27T07:08:22.047848+0000","last_unstale":"2024-05-27T07:09:54.041301+0000","last_undegraded":"2024-05-27T07:09:54.041301+0000","last_fullsized":"2024-05-27T07:09:54.041301+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:51.707603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060218+0000","last_change":"2024-05-27T07:08:36.932747+0000","last_active":"2024-05-27T07:09:54.060218+0000","last_peered":"2024-05-27T07:09:54.060218+0000","last_clean":"2024-05-27T07:09:54.060218+0000","last_became_active":"2024-05-27T07:08:36.932050+0000","last_became_peered":"2024-05-27T07:08:36.932050+0000","last_unstale":"2024-05-27T07:09:54.060218+0000","last_undegraded":"2024-05-27T07:09:54.060218+0000","last_fullsized":"2024-05-27T07:09:54.060218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:26:07.015651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'15","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:25.122124+0000","last_change":"2024-05-27T07:08:18.008844+0000","last_active":"2024-05-27T07:10:25.122124+0000","last_peered":"2024-05-27T07:10:25.122124+0000","last_clean":"2024-05-27T07:10:25.122124+0000","last_became_active":"2024-05-27T07:08:18.008140+0000","last_became_peered":"2024-05-27T07:08:18.008140+0000","last_unstale":"2024-05-27T07:10:25.122124+0000","last_undegraded":"2024-05-27T07:10:25.122124+0000","last_fullsized":"2024-05-27T07:10:25.122124+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:22:22.075341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.061571+0000","last_change":"2024-05-27T07:08:22.050855+0000","last_active":"2024-05-27T07:09:54.061571+0000","last_peered":"2024-05-27T07:09:54.061571+0000","last_clean":"2024-05-27T07:09:54.061571+0000","last_became_active":"2024-05-27T07:08:22.048482+0000","last_became_peered":"2024-05-27T07:08:22.048482+0000","last_unstale":"2024-05-27T07:09:54.061571+0000","last_undegraded":"2024-05-27T07:09:54.061571+0000","last_fullsized":"2024-05-27T07:09:54.061571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:04.319884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.069514+0000","last_change":"2024-05-27T07:08:36.862470+0000","last_active":"2024-05-27T07:09:54.069514+0000","last_peered":"2024-05-27T07:09:54.069514+0000","last_clean":"2024-05-27T07:09:54.069514+0000","last_became_active":"2024-05-27T07:08:36.861951+0000","last_became_peered":"2024-05-27T07:08:36.861951+0000","last_unstale":"2024-05-27T07:09:54.069514+0000","last_undegraded":"2024-05-27T07:09:54.069514+0000","last_fullsized":"2024-05-27T07:09:54.069514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:56:31.659951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.080450+0000","last_change":"2024-05-27T07:08:18.438518+0000","last_active":"2024-05-27T07:09:54.080450+0000","last_peered":"2024-05-27T07:09:54.080450+0000","last_clean":"2024-05-27T07:09:54.080450+0000","last_became_active":"2024-05-27T07:08:18.437588+0000","last_became_peered":"2024-05-27T07:08:18.437588+0000","last_unstale":"2024-05-27T07:09:54.080450+0000","last_undegraded":"2024-05-27T07:09:54.080450+0000","last_fullsized":"2024-05-27T07:09:54.080450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:52:12.895009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185539+0000","last_change":"2024-05-27T07:08:22.064188+0000","last_active":"2024-05-27T07:09:54.185539+0000","last_peered":"2024-05-27T07:09:54.185539+0000","last_clean":"2024-05-27T07:09:54.185539+0000","last_became_active":"2024-05-27T07:08:22.063868+0000","last_became_peered":"2024-05-27T07:08:22.063868+0000","last_unstale":"2024-05-27T07:09:54.185539+0000","last_undegraded":"2024-05-27T07:09:54.185539+0000","last_fullsized":"2024-05-27T07:09:54.185539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:43:17.227079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079129+0000","last_change":"2024-05-27T07:08:36.934922+0000","last_active":"2024-05-27T07:09:54.079129+0000","last_peered":"2024-05-27T07:09:54.079129+0000","last_clean":"2024-05-27T07:09:54.079129+0000","last_became_active":"2024-05-27T07:08:36.934000+0000","last_became_peered":"2024-05-27T07:08:36.934000+0000","last_unstale":"2024-05-27T07:09:54.079129+0000","last_undegraded":"2024-05-27T07:09:54.079129+0000","last_fullsized":"2024-05-27T07:09:54.079129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:19:00.570557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187967+0000","last_change":"2024-05-27T07:08:18.308743+0000","last_active":"2024-05-27T07:09:54.187967+0000","last_peered":"2024-05-27T07:09:54.187967+0000","last_clean":"2024-05-27T07:09:54.187967+0000","last_became_active":"2024-05-27T07:08:18.307597+0000","last_became_peered":"2024-05-27T07:08:18.307597+0000","last_unstale":"2024-05-27T07:09:54.187967+0000","last_undegraded":"2024-05-27T07:09:54.187967+0000","last_fullsized":"2024-05-27T07:09:54.187967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:39.914998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189080+0000","last_change":"2024-05-27T07:08:22.049132+0000","last_active":"2024-05-27T07:09:54.189080+0000","last_peered":"2024-05-27T07:09:54.189080+0000","last_clean":"2024-05-27T07:09:54.189080+0000","last_became_active":"2024-05-27T07:08:22.048606+0000","last_became_peered":"2024-05-27T07:08:22.048606+0000","last_unstale":"2024-05-27T07:09:54.189080+0000","last_undegraded":"2024-05-27T07:09:54.189080+0000","last_fullsized":"2024-05-27T07:09:54.189080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:58.253461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.049190+0000","last_change":"2024-05-27T07:08:36.867286+0000","last_active":"2024-05-27T07:09:54.049190+0000","last_peered":"2024-05-27T07:09:54.049190+0000","last_clean":"2024-05-27T07:09:54.049190+0000","last_became_active":"2024-05-27T07:08:36.866732+0000","last_became_peered":"2024-05-27T07:08:36.866732+0000","last_unstale":"2024-05-27T07:09:54.049190+0000","last_undegraded":"2024-05-27T07:09:54.049190+0000","last_fullsized":"2024-05-27T07:09:54.049190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:14:18.386441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060788+0000","last_change":"2024-05-27T07:08:18.010026+0000","last_active":"2024-05-27T07:09:54.060788+0000","last_peered":"2024-05-27T07:09:54.060788+0000","last_clean":"2024-05-27T07:09:54.060788+0000","last_became_active":"2024-05-27T07:08:18.009502+0000","last_became_peered":"2024-05-27T07:08:18.009502+0000","last_unstale":"2024-05-27T07:09:54.060788+0000","last_undegraded":"2024-05-27T07:09:54.060788+0000","last_fullsized":"2024-05-27T07:09:54.060788+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:28:27.500933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187190+0000","last_change":"2024-05-27T07:08:22.054536+0000","last_active":"2024-05-27T07:09:54.187190+0000","last_peered":"2024-05-27T07:09:54.187190+0000","last_clean":"2024-05-27T07:09:54.187190+0000","last_became_active":"2024-05-27T07:08:22.054141+0000","last_became_peered":"2024-05-27T07:08:22.054141+0000","last_unstale":"2024-05-27T07:09:54.187190+0000","last_undegraded":"2024-05-27T07:09:54.187190+0000","last_fullsized":"2024-05-27T07:09:54.187190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:53.413683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.080431+0000","last_change":"2024-05-27T07:08:36.859721+0000","last_active":"2024-05-27T07:09:54.080431+0000","last_peered":"2024-05-27T07:09:54.080431+0000","last_clean":"2024-05-27T07:09:54.080431+0000","last_became_active":"2024-05-27T07:08:36.858770+0000","last_became_peered":"2024-05-27T07:08:36.858770+0000","last_unstale":"2024-05-27T07:09:54.080431+0000","last_undegraded":"2024-05-27T07:09:54.080431+0000","last_fullsized":"2024-05-27T07:09:54.080431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:32.583869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070392+0000","last_change":"2024-05-27T07:08:18.008762+0000","last_active":"2024-05-27T07:09:54.070392+0000","last_peered":"2024-05-27T07:09:54.070392+0000","last_clean":"2024-05-27T07:09:54.070392+0000","last_became_active":"2024-05-27T07:08:18.007680+0000","last_became_peered":"2024-05-27T07:08:18.007680+0000","last_unstale":"2024-05-27T07:09:54.070392+0000","last_undegraded":"2024-05-27T07:09:54.070392+0000","last_fullsized":"2024-05-27T07:09:54.070392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:22.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062292+0000","last_change":"2024-05-27T07:08:22.051955+0000","last_active":"2024-05-27T07:09:54.062292+0000","last_peered":"2024-05-27T07:09:54.062292+0000","last_clean":"2024-05-27T07:09:54.062292+0000","last_became_active":"2024-05-27T07:08:22.051618+0000","last_became_peered":"2024-05-27T07:08:22.051618+0000","last_unstale":"2024-05-27T07:09:54.062292+0000","last_undegraded":"2024-05-27T07:09:54.062292+0000","last_fullsized":"2024-05-27T07:09:54.062292+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:13.219186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.078961+0000","last_change":"2024-05-27T07:08:36.932048+0000","last_active":"2024-05-27T07:09:54.078961+0000","last_peered":"2024-05-27T07:09:54.078961+0000","last_clean":"2024-05-27T07:09:54.078961+0000","last_became_active":"2024-05-27T07:08:36.930777+0000","last_became_peered":"2024-05-27T07:08:36.930777+0000","last_unstale":"2024-05-27T07:09:54.078961+0000","last_undegraded":"2024-05-27T07:09:54.078961+0000","last_fullsized":"2024-05-27T07:09:54.078961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:32.284487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189428+0000","last_change":"2024-05-27T07:08:18.011343+0000","last_active":"2024-05-27T07:09:54.189428+0000","last_peered":"2024-05-27T07:09:54.189428+0000","last_clean":"2024-05-27T07:09:54.189428+0000","last_became_active":"2024-05-27T07:08:18.009988+0000","last_became_peered":"2024-05-27T07:08:18.009988+0000","last_unstale":"2024-05-27T07:09:54.189428+0000","last_undegraded":"2024-05-27T07:09:54.189428+0000","last_fullsized":"2024-05-27T07:09:54.189428+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:39.620568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045033+0000","last_change":"2024-05-27T07:08:22.051630+0000","last_active":"2024-05-27T07:09:54.045033+0000","last_peered":"2024-05-27T07:09:54.045033+0000","last_clean":"2024-05-27T07:09:54.045033+0000","last_became_active":"2024-05-27T07:08:22.049140+0000","last_became_peered":"2024-05-27T07:08:22.049140+0000","last_unstale":"2024-05-27T07:09:54.045033+0000","last_undegraded":"2024-05-27T07:09:54.045033+0000","last_fullsized":"2024-05-27T07:09:54.045033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:51.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187236+0000","last_change":"2024-05-27T07:08:36.870668+0000","last_active":"2024-05-27T07:09:54.187236+0000","last_peered":"2024-05-27T07:09:54.187236+0000","last_clean":"2024-05-27T07:09:54.187236+0000","last_became_active":"2024-05-27T07:08:36.869986+0000","last_became_peered":"2024-05-27T07:08:36.869986+0000","last_unstale":"2024-05-27T07:09:54.187236+0000","last_undegraded":"2024-05-27T07:09:54.187236+0000","last_fullsized":"2024-05-27T07:09:54.187236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:59.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'6","reported_seq":94,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:32.185712+0000","last_change":"2024-05-27T07:08:18.439082+0000","last_active":"2024-05-27T07:10:32.185712+0000","last_peered":"2024-05-27T07:10:32.185712+0000","last_clean":"2024-05-27T07:10:32.185712+0000","last_became_active":"2024-05-27T07:08:18.437844+0000","last_became_peered":"2024-05-27T07:08:18.437844+0000","last_unstale":"2024-05-27T07:10:32.185712+0000","last_undegraded":"2024-05-27T07:10:32.185712+0000","last_fullsized":"2024-05-27T07:10:32.185712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:38:30.276747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.046436+0000","last_change":"2024-05-27T07:08:22.052308+0000","last_active":"2024-05-27T07:09:54.046436+0000","last_peered":"2024-05-27T07:09:54.046436+0000","last_clean":"2024-05-27T07:09:54.046436+0000","last_became_active":"2024-05-27T07:08:22.051615+0000","last_became_peered":"2024-05-27T07:08:22.051615+0000","last_unstale":"2024-05-27T07:09:54.046436+0000","last_undegraded":"2024-05-27T07:09:54.046436+0000","last_fullsized":"2024-05-27T07:09:54.046436+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:14.241454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.184726+0000","last_change":"2024-05-27T07:08:36.861082+0000","last_active":"2024-05-27T07:09:54.184726+0000","last_peered":"2024-05-27T07:09:54.184726+0000","last_clean":"2024-05-27T07:09:54.184726+0000","last_became_active":"2024-05-27T07:08:36.860548+0000","last_became_peered":"2024-05-27T07:08:36.860548+0000","last_unstale":"2024-05-27T07:09:54.184726+0000","last_undegraded":"2024-05-27T07:09:54.184726+0000","last_fullsized":"2024-05-27T07:09:54.184726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:46:32.446464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043709+0000","last_change":"2024-05-27T07:08:18.018218+0000","last_active":"2024-05-27T07:09:54.043709+0000","last_peered":"2024-05-27T07:09:54.043709+0000","last_clean":"2024-05-27T07:09:54.043709+0000","last_became_active":"2024-05-27T07:08:18.017200+0000","last_became_peered":"2024-05-27T07:08:18.017200+0000","last_unstale":"2024-05-27T07:09:54.043709+0000","last_undegraded":"2024-05-27T07:09:54.043709+0000","last_fullsized":"2024-05-27T07:09:54.043709+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:22.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.052709+0000","last_change":"2024-05-27T07:09:54.052709+0000","last_active":"2024-05-27T07:09:54.052709+0000","last_peered":"2024-05-27T07:09:54.052709+0000","last_clean":"2024-05-27T07:09:54.052709+0000","last_became_active":"2024-05-27T07:09:54.051425+0000","last_became_peered":"2024-05-27T07:09:54.051425+0000","last_unstale":"2024-05-27T07:09:54.052709+0000","last_undegraded":"2024-05-27T07:09:54.052709+0000","last_fullsized":"2024-05-27T07:09:54.052709+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:51:47.166228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188615+0000","last_change":"2024-05-27T07:08:53.979067+0000","last_active":"2024-05-27T07:09:54.188615+0000","last_peered":"2024-05-27T07:09:54.188615+0000","last_clean":"2024-05-27T07:09:54.188615+0000","last_became_active":"2024-05-27T07:08:53.977620+0000","last_became_peered":"2024-05-27T07:08:53.977620+0000","last_unstale":"2024-05-27T07:09:54.188615+0000","last_undegraded":"2024-05-27T07:09:54.188615+0000","last_fullsized":"2024-05-27T07:09:54.188615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:31.619861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042083+0000","last_change":"2024-05-27T07:08:54.001216+0000","last_active":"2024-05-27T07:09:54.042083+0000","last_peered":"2024-05-27T07:09:54.042083+0000","last_clean":"2024-05-27T07:09:54.042083+0000","last_became_active":"2024-05-27T07:08:53.982121+0000","last_became_peered":"2024-05-27T07:08:53.982121+0000","last_unstale":"2024-05-27T07:09:54.042083+0000","last_undegraded":"2024-05-27T07:09:54.042083+0000","last_fullsized":"2024-05-27T07:09:54.042083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:01.474383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.046191+0000","last_change":"2024-05-27T07:08:22.053334+0000","last_active":"2024-05-27T07:09:54.046191+0000","last_peered":"2024-05-27T07:09:54.046191+0000","last_clean":"2024-05-27T07:09:54.046191+0000","last_became_active":"2024-05-27T07:08:22.051660+0000","last_became_peered":"2024-05-27T07:08:22.051660+0000","last_unstale":"2024-05-27T07:09:54.046191+0000","last_undegraded":"2024-05-27T07:09:54.046191+0000","last_fullsized":"2024-05-27T07:09:54.046191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:07:19.554620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187439+0000","last_change":"2024-05-27T07:08:22.064044+0000","last_active":"2024-05-27T07:09:54.187439+0000","last_peered":"2024-05-27T07:09:54.187439+0000","last_clean":"2024-05-27T07:09:54.187439+0000","last_became_active":"2024-05-27T07:08:22.063539+0000","last_became_peered":"2024-05-27T07:08:22.063539+0000","last_unstale":"2024-05-27T07:09:54.187439+0000","last_undegraded":"2024-05-27T07:09:54.187439+0000","last_fullsized":"2024-05-27T07:09:54.187439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:07.291250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070471+0000","last_change":"2024-05-27T07:08:22.059727+0000","last_active":"2024-05-27T07:09:54.070471+0000","last_peered":"2024-05-27T07:09:54.070471+0000","last_clean":"2024-05-27T07:09:54.070471+0000","last_became_active":"2024-05-27T07:08:22.058933+0000","last_became_peered":"2024-05-27T07:08:22.058933+0000","last_unstale":"2024-05-27T07:09:54.070471+0000","last_undegraded":"2024-05-27T07:09:54.070471+0000","last_fullsized":"2024-05-27T07:09:54.070471+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:14.828031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.069950+0000","last_change":"2024-05-27T07:08:22.044871+0000","last_active":"2024-05-27T07:09:54.069950+0000","last_peered":"2024-05-27T07:09:54.069950+0000","last_clean":"2024-05-27T07:09:54.069950+0000","last_became_active":"2024-05-27T07:08:22.044394+0000","last_became_peered":"2024-05-27T07:08:22.044394+0000","last_unstale":"2024-05-27T07:09:54.069950+0000","last_undegraded":"2024-05-27T07:09:54.069950+0000","last_fullsized":"2024-05-27T07:09:54.069950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:11.151064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185139+0000","last_change":"2024-05-27T07:08:22.048318+0000","last_active":"2024-05-27T07:09:54.185139+0000","last_peered":"2024-05-27T07:09:54.185139+0000","last_clean":"2024-05-27T07:09:54.185139+0000","last_became_active":"2024-05-27T07:08:22.047983+0000","last_became_peered":"2024-05-27T07:08:22.047983+0000","last_unstale":"2024-05-27T07:09:54.185139+0000","last_undegraded":"2024-05-27T07:09:54.185139+0000","last_fullsized":"2024-05-27T07:09:54.185139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:11.479943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041437+0000","last_change":"2024-05-27T07:08:22.056951+0000","last_active":"2024-05-27T07:09:54.041437+0000","last_peered":"2024-05-27T07:09:54.041437+0000","last_clean":"2024-05-27T07:09:54.041437+0000","last_became_active":"2024-05-27T07:08:22.053701+0000","last_became_peered":"2024-05-27T07:08:22.053701+0000","last_unstale":"2024-05-27T07:09:54.041437+0000","last_undegraded":"2024-05-27T07:09:54.041437+0000","last_fullsized":"2024-05-27T07:09:54.041437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:22:31.041730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188161+0000","last_change":"2024-05-27T07:08:22.062745+0000","last_active":"2024-05-27T07:09:54.188161+0000","last_peered":"2024-05-27T07:09:54.188161+0000","last_clean":"2024-05-27T07:09:54.188161+0000","last_became_active":"2024-05-27T07:08:22.062339+0000","last_became_peered":"2024-05-27T07:08:22.062339+0000","last_unstale":"2024-05-27T07:09:54.188161+0000","last_undegraded":"2024-05-27T07:09:54.188161+0000","last_fullsized":"2024-05-27T07:09:54.188161+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:58:02.934021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.071509+0000","last_change":"2024-05-27T07:08:22.052621+0000","last_active":"2024-05-27T07:09:54.071509+0000","last_peered":"2024-05-27T07:09:54.071509+0000","last_clean":"2024-05-27T07:09:54.071509+0000","last_became_active":"2024-05-27T07:08:22.052070+0000","last_became_peered":"2024-05-27T07:08:22.052070+0000","last_unstale":"2024-05-27T07:09:54.071509+0000","last_undegraded":"2024-05-27T07:09:54.071509+0000","last_fullsized":"2024-05-27T07:09:54.071509+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:34.491068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062117+0000","last_change":"2024-05-27T07:08:22.055704+0000","last_active":"2024-05-27T07:09:54.062117+0000","last_peered":"2024-05-27T07:09:54.062117+0000","last_clean":"2024-05-27T07:09:54.062117+0000","last_became_active":"2024-05-27T07:08:22.055251+0000","last_became_peered":"2024-05-27T07:08:22.055251+0000","last_unstale":"2024-05-27T07:09:54.062117+0000","last_undegraded":"2024-05-27T07:09:54.062117+0000","last_fullsized":"2024-05-27T07:09:54.062117+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:28.074825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042253+0000","last_change":"2024-05-27T07:08:22.052160+0000","last_active":"2024-05-27T07:09:54.042253+0000","last_peered":"2024-05-27T07:09:54.042253+0000","last_clean":"2024-05-27T07:09:54.042253+0000","last_became_active":"2024-05-27T07:08:22.049495+0000","last_became_peered":"2024-05-27T07:08:22.049495+0000","last_unstale":"2024-05-27T07:09:54.042253+0000","last_undegraded":"2024-05-27T07:09:54.042253+0000","last_fullsized":"2024-05-27T07:09:54.042253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:54:59.595031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060755+0000","last_change":"2024-05-27T07:08:22.053828+0000","last_active":"2024-05-27T07:09:54.060755+0000","last_peered":"2024-05-27T07:09:54.060755+0000","last_clean":"2024-05-27T07:09:54.060755+0000","last_became_active":"2024-05-27T07:08:22.053491+0000","last_became_peered":"2024-05-27T07:08:22.053491+0000","last_unstale":"2024-05-27T07:09:54.060755+0000","last_undegraded":"2024-05-27T07:09:54.060755+0000","last_fullsized":"2024-05-27T07:09:54.060755+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:20.568772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187914+0000","last_change":"2024-05-27T07:08:22.050791+0000","last_active":"2024-05-27T07:09:54.187914+0000","last_peered":"2024-05-27T07:09:54.187914+0000","last_clean":"2024-05-27T07:09:54.187914+0000","last_became_active":"2024-05-27T07:08:22.050323+0000","last_became_peered":"2024-05-27T07:08:22.050323+0000","last_unstale":"2024-05-27T07:09:54.187914+0000","last_undegraded":"2024-05-27T07:09:54.187914+0000","last_fullsized":"2024-05-27T07:09:54.187914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:32.400758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041173+0000","last_change":"2024-05-27T07:08:22.041286+0000","last_active":"2024-05-27T07:09:54.041173+0000","last_peered":"2024-05-27T07:09:54.041173+0000","last_clean":"2024-05-27T07:09:54.041173+0000","last_became_active":"2024-05-27T07:08:22.040811+0000","last_became_peered":"2024-05-27T07:08:22.040811+0000","last_unstale":"2024-05-27T07:09:54.041173+0000","last_undegraded":"2024-05-27T07:09:54.041173+0000","last_fullsized":"2024-05-27T07:09:54.041173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:40:09.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188402+0000","last_change":"2024-05-27T07:08:22.066490+0000","last_active":"2024-05-27T07:09:54.188402+0000","last_peered":"2024-05-27T07:09:54.188402+0000","last_clean":"2024-05-27T07:09:54.188402+0000","last_became_active":"2024-05-27T07:08:22.064779+0000","last_became_peered":"2024-05-27T07:08:22.064779+0000","last_unstale":"2024-05-27T07:09:54.188402+0000","last_undegraded":"2024-05-27T07:09:54.188402+0000","last_fullsized":"2024-05-27T07:09:54.188402+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:23.967856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71340,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":424,"num_read_kb":291,"num_write":249,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678063,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":519752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874293,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1736704,"data_stored":1170010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037820,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1069056,"data_stored":522693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234053,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":518281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430284,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32796,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712356,"statfs":{"total":95995035648,"available":95961452544,"internally_reserved":0,"allocated":1658880,"data_stored":1115472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593813,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":514101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":579267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738497,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":515948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1056768,"data_stored":515742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":513809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953570,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":582702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790043,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1110663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:10:36.472 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T07:10:36.473 INFO:tasks.cephadm:Setting up client nodes... 2024-05-27T07:10:36.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-27T07:10:36.975 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:37.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:37 smithi132 ceph-mon[35283]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:10:37.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:37 smithi043 ceph-mon[31531]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:10:37.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:37 smithi179 ceph-mon[35888]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:10:38.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:38 smithi179 ceph-mon[35888]: pgmap v462: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-05-27T07:10:38.304 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:38 smithi043 ceph-mon[31531]: pgmap v462: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-05-27T07:10:38.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:38 smithi132 ceph-mon[35283]: pgmap v462: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-05-27T07:10:39.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.274+0000 7f3e2a200700 1 -- 172.21.15.43:0/3997721333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e24104e30 msgr2=0x7f3e24107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:39.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.274+0000 7f3e2a200700 1 --2- 172.21.15.43:0/3997721333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e24104e30 0x7f3e24107250 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009990 tx=0x7f3e14009ca0 comp rx=0 tx=0).stop 2024-05-27T07:10:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.275+0000 7f3e2a200700 1 -- 172.21.15.43:0/3997721333 shutdown_connections 2024-05-27T07:10:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.275+0000 7f3e2a200700 1 --2- 172.21.15.43:0/3997721333 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e2410a0f0 0x7f3e2410c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.275+0000 7f3e2a200700 1 --2- 172.21.15.43:0/3997721333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e24104e30 0x7f3e24107250 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.275+0000 7f3e2a200700 1 --2- 172.21.15.43:0/3997721333 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e24107790 0x7f3e24109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.275+0000 7f3e2a200700 1 -- 172.21.15.43:0/3997721333 >> 172.21.15.43:0/3997721333 conn(0x7f3e240fe870 msgr2=0x7f3e24100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.275+0000 7f3e2a200700 1 -- 172.21.15.43:0/3997721333 shutdown_connections 2024-05-27T07:10:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.275+0000 7f3e2a200700 1 -- 172.21.15.43:0/3997721333 wait complete. 2024-05-27T07:10:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.276+0000 7f3e2a200700 1 Processor -- start 2024-05-27T07:10:39.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.277+0000 7f3e2a200700 1 -- start start 2024-05-27T07:10:39.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.277+0000 7f3e2a200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e24104e30 0x7f3e2419a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:39.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.277+0000 7f3e2a200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e24107790 0x7f3e2419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:39.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.277+0000 7f3e2a200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e2410a0f0 0x7f3e241a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.277+0000 7f3e2a200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2410fc70 con 0x7f3e24107790 2024-05-27T07:10:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.277+0000 7f3e2a200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2410faf0 con 0x7f3e24104e30 2024-05-27T07:10:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.277+0000 7f3e2a200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2410fdf0 con 0x7f3e2410a0f0 2024-05-27T07:10:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.278+0000 7f3e22a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e24107790 0x7f3e2419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.278+0000 7f3e22a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e24107790 0x7f3e2419adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53794/0 (socket says 172.21.15.43:53794) 2024-05-27T07:10:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.278+0000 7f3e22a00700 1 -- 172.21.15.43:0/1814626768 learned_addr learned my addr 172.21.15.43:0/1814626768 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.278+0000 7f3e23400700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e24104e30 0x7f3e2419a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.278+0000 7f3e23e00700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e2410a0f0 0x7f3e241a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.278+0000 7f3e22a00700 1 -- 172.21.15.43:0/1814626768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e2410a0f0 msgr2=0x7f3e241a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.278+0000 7f3e22a00700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e2410a0f0 0x7f3e241a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.279+0000 7f3e22a00700 1 -- 172.21.15.43:0/1814626768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e24104e30 msgr2=0x7f3e2419a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.279+0000 7f3e22a00700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e24104e30 0x7f3e2419a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.279+0000 7f3e22a00700 1 -- 172.21.15.43:0/1814626768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e241a23e0 con 0x7f3e24107790 2024-05-27T07:10:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.279+0000 7f3e22a00700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e24107790 0x7f3e2419adc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000d950 tx=0x7f3e1000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.279+0000 7f3e1be00700 1 -- 172.21.15.43:0/1814626768 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10016070 con 0x7f3e24107790 2024-05-27T07:10:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.279+0000 7f3e1be00700 1 -- 172.21.15.43:0/1814626768 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e1000fd20 con 0x7f3e24107790 2024-05-27T07:10:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.279+0000 7f3e2a200700 1 -- 172.21.15.43:0/1814626768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e241a26d0 con 0x7f3e24107790 2024-05-27T07:10:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.279+0000 7f3e1be00700 1 -- 172.21.15.43:0/1814626768 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1000e3b0 con 0x7f3e24107790 2024-05-27T07:10:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.280+0000 7f3e2a200700 1 -- 172.21.15.43:0/1814626768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e241a2c10 con 0x7f3e24107790 2024-05-27T07:10:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.281+0000 7f3e2a200700 1 -- 172.21.15.43:0/1814626768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e240512b0 con 0x7f3e24107790 2024-05-27T07:10:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.288+0000 7f3e1be00700 1 -- 172.21.15.43:0/1814626768 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f3e10019070 con 0x7f3e24107790 2024-05-27T07:10:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.289+0000 7f3e1be00700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f3e080b2910 0x7f3e080b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.290+0000 7f3e1be00700 1 -- 172.21.15.43:0/1814626768 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f3e100e2230 con 0x7f3e24107790 2024-05-27T07:10:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.290+0000 7f3e23400700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f3e080b2910 0x7f3e080b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.290+0000 7f3e1be00700 1 -- 172.21.15.43:0/1814626768 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3e100e26b0 con 0x7f3e24107790 2024-05-27T07:10:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.291+0000 7f3e23400700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f3e080b2910 0x7f3e080b4e20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400b5c0 tx=0x7f3e1401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:39.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.626+0000 7f3e2a200700 1 -- 172.21.15.43:0/1814626768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f3e2402e5e0 con 0x7f3e24107790 2024-05-27T07:10:39.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.632+0000 7f3e1be00700 1 -- 172.21.15.43:0/1814626768 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v34) ==== 170+0+59 (secure 0 0 0) 0x7f3e100a9a00 con 0x7f3e24107790 2024-05-27T07:10:39.624 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-05-27T07:10:39.624 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDvMVRmRjduJRAAWGeWs6hWsk0G3wNSQ7alZw== 2024-05-27T07:10:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 -- 172.21.15.43:0/1814626768 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f3e080b2910 msgr2=0x7f3e080b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f3e080b2910 0x7f3e080b4e20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400b5c0 tx=0x7f3e1401a040 comp rx=0 tx=0).stop 2024-05-27T07:10:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 -- 172.21.15.43:0/1814626768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e24107790 msgr2=0x7f3e2419adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e24107790 0x7f3e2419adc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000d950 tx=0x7f3e1000dc60 comp rx=0 tx=0).stop 2024-05-27T07:10:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 -- 172.21.15.43:0/1814626768 shutdown_connections 2024-05-27T07:10:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f3e080b2910 0x7f3e080b4e20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e2410a0f0 0x7f3e241a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e24104e30 0x7f3e2419a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 --2- 172.21.15.43:0/1814626768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3e24107790 0x7f3e2419adc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 -- 172.21.15.43:0/1814626768 >> 172.21.15.43:0/1814626768 conn(0x7f3e240fe870 msgr2=0x7f3e24106e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 -- 172.21.15.43:0/1814626768 shutdown_connections 2024-05-27T07:10:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:39.636+0000 7f3e2a200700 1 -- 172.21.15.43:0/1814626768 wait complete. 2024-05-27T07:10:40.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:40 smithi132 ceph-mon[35283]: pgmap v463: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 596 B/s wr, 10 op/s 2024-05-27T07:10:40.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:40 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1814626768' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:10:40.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:40 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1814626768' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:10:40.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:40 smithi043 ceph-mon[31531]: pgmap v463: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 596 B/s wr, 10 op/s 2024-05-27T07:10:40.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:40 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1814626768' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:10:40.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:40 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1814626768' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:10:40.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:40 smithi179 ceph-mon[35888]: pgmap v463: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 596 B/s wr, 10 op/s 2024-05-27T07:10:40.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:40 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1814626768' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-27T07:10:40.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:40 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1814626768' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-27T07:10:40.562 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T07:10:40.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-27T07:10:40.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-27T07:10:40.708 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-27T07:10:40.708 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-27T07:10:40.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-05-27T07:10:41.231 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:42.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:42 smithi132 ceph-mon[35283]: pgmap v464: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 596 B/s wr, 10 op/s 2024-05-27T07:10:42.413 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:42 smithi043 ceph-mon[31531]: pgmap v464: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 596 B/s wr, 10 op/s 2024-05-27T07:10:42.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:42 smithi179 ceph-mon[35888]: pgmap v464: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 596 B/s wr, 10 op/s 2024-05-27T07:10:43.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.441+0000 7f091a400700 1 -- 172.21.15.43:0/2186960954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f091410af40 msgr2=0x7f091410b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:43.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.441+0000 7f091a400700 1 --2- 172.21.15.43:0/2186960954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f091410af40 0x7f091410b350 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f09000099c0 tx=0x7f0900009cd0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.441+0000 7f091a400700 1 -- 172.21.15.43:0/2186960954 shutdown_connections 2024-05-27T07:10:43.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.441+0000 7f091a400700 1 --2- 172.21.15.43:0/2186960954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091410b890 0x7f09141131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.441+0000 7f091a400700 1 --2- 172.21.15.43:0/2186960954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0914106610 0x7f0914106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.441+0000 7f091a400700 1 --2- 172.21.15.43:0/2186960954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f091410af40 0x7f091410b350 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.441+0000 7f091a400700 1 -- 172.21.15.43:0/2186960954 >> 172.21.15.43:0/2186960954 conn(0x7f0914102020 msgr2=0x7f0914104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:43.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.441+0000 7f091a400700 1 -- 172.21.15.43:0/2186960954 shutdown_connections 2024-05-27T07:10:43.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.442+0000 7f091a400700 1 -- 172.21.15.43:0/2186960954 wait complete. 2024-05-27T07:10:43.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.443+0000 7f091a400700 1 Processor -- start 2024-05-27T07:10:43.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.443+0000 7f091a400700 1 -- start start 2024-05-27T07:10:43.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.444+0000 7f091a400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0914106610 0x7f091419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:43.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.444+0000 7f091a400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091410af40 0x7f091419d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:43.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.444+0000 7f091a400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f091410b890 0x7f09141a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:43.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.444+0000 7f091a400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0914116350 con 0x7f0914106610 2024-05-27T07:10:43.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.444+0000 7f091a400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f09141161d0 con 0x7f091410b890 2024-05-27T07:10:43.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.444+0000 7f091a400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f09141164d0 con 0x7f091410af40 2024-05-27T07:10:43.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.444+0000 7f0919000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0914106610 0x7f091419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:43.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.444+0000 7f0919000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0914106610 0x7f091419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53810/0 (socket says 172.21.15.43:53810) 2024-05-27T07:10:43.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.444+0000 7f0919000700 1 -- 172.21.15.43:0/1544199706 learned_addr learned my addr 172.21.15.43:0/1544199706 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:43.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.445+0000 7f0913e00700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091410af40 0x7f091419d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:43.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.445+0000 7f0919a00700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f091410b890 0x7f09141a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:43.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.445+0000 7f0919000700 1 -- 172.21.15.43:0/1544199706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091410af40 msgr2=0x7f091419d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:43.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.445+0000 7f0919000700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091410af40 0x7f091419d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.445+0000 7f0919000700 1 -- 172.21.15.43:0/1544199706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f091410b890 msgr2=0x7f09141a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:43.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.445+0000 7f0919000700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f091410b890 0x7f09141a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.445+0000 7f0919000700 1 -- 172.21.15.43:0/1544199706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09141a1ba0 con 0x7f0914106610 2024-05-27T07:10:43.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.445+0000 7f0913e00700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091410af40 0x7f091419d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:10:43.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.445+0000 7f0919000700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0914106610 0x7f091419cd80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f090800d950 tx=0x7f090800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:43.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.446+0000 7f0911600700 1 -- 172.21.15.43:0/1544199706 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0908016070 con 0x7f0914106610 2024-05-27T07:10:43.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.446+0000 7f0911600700 1 -- 172.21.15.43:0/1544199706 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0908009dd0 con 0x7f0914106610 2024-05-27T07:10:43.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.446+0000 7f091a400700 1 -- 172.21.15.43:0/1544199706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09141a1e90 con 0x7f0914106610 2024-05-27T07:10:43.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.446+0000 7f0911600700 1 -- 172.21.15.43:0/1544199706 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090800f3c0 con 0x7f0914106610 2024-05-27T07:10:43.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.446+0000 7f091a400700 1 -- 172.21.15.43:0/1544199706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09141a23a0 con 0x7f0914106610 2024-05-27T07:10:43.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.448+0000 7f091a400700 1 -- 172.21.15.43:0/1544199706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09140512b0 con 0x7f0914106610 2024-05-27T07:10:43.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.450+0000 7f0911600700 1 -- 172.21.15.43:0/1544199706 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0908019070 con 0x7f0914106610 2024-05-27T07:10:43.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.452+0000 7f0911600700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f08fc0b2910 0x7f08fc0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:43.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.452+0000 7f0911600700 1 -- 172.21.15.43:0/1544199706 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f09080e23b0 con 0x7f0914106610 2024-05-27T07:10:43.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.452+0000 7f0913e00700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f08fc0b2910 0x7f08fc0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:43.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.453+0000 7f0913e00700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f08fc0b2910 0x7f08fc0b4e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0900009990 tx=0x7f0900005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.454+0000 7f0911600700 1 -- 172.21.15.43:0/1544199706 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f09080a9b80 con 0x7f0914106610 2024-05-27T07:10:43.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.753+0000 7f091a400700 1 -- 172.21.15.43:0/1544199706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f091402e5e0 con 0x7f0914106610 2024-05-27T07:10:43.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.773+0000 7f0911600700 1 -- 172.21.15.43:0/1544199706 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+288647 (secure 0 0 0) 0x7f09080a9480 con 0x7f0914106610 2024-05-27T07:10:43.769 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 -- 172.21.15.43:0/1544199706 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f08fc0b2910 msgr2=0x7f08fc0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f08fc0b2910 0x7f08fc0b4e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0900009990 tx=0x7f0900005fd0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 -- 172.21.15.43:0/1544199706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0914106610 msgr2=0x7f091419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0914106610 0x7f091419cd80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f090800d950 tx=0x7f090800dc60 comp rx=0 tx=0).stop 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 -- 172.21.15.43:0/1544199706 shutdown_connections 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f08fc0b2910 0x7f08fc0b4e20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f091410b890 0x7f09141a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0914106610 0x7f091419cd80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 --2- 172.21.15.43:0/1544199706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091410af40 0x7f091419d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 -- 172.21.15.43:0/1544199706 >> 172.21.15.43:0/1544199706 conn(0x7f0914102020 msgr2=0x7f091410a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:43.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 -- 172.21.15.43:0/1544199706 shutdown_connections 2024-05-27T07:10:43.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:43.779+0000 7f091a400700 1 -- 172.21.15.43:0/1544199706 wait complete. 2024-05-27T07:10:44.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:44 smithi132 ceph-mon[35283]: pgmap v465: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:10:44.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:44 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1544199706' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:10:44.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:44 smithi043 ceph-mon[31531]: pgmap v465: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:10:44.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:44 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1544199706' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:10:44.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:44 smithi179 ceph-mon[35888]: pgmap v465: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:10:44.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:44 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1544199706' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-27T07:10:44.633 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1809594794},{"type":"v1","addr":"172.21.15.43:6801","nonce":1809594794}]},"active_addr":"172.21.15.43:6801/1809594794","active_change":"2024-05-27T06:57:52.506502+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":34118,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.43:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3331041602}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3399450175}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":2266081183}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3253329315}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":2134954289}]}]} 2024-05-27T07:10:44.641 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-27T07:10:44.641 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-27T07:10:44.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:10:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:46.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:46 smithi132 ceph-mon[35283]: pgmap v466: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:10:46.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:46 smithi043 ceph-mon[31531]: pgmap v466: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:10:46.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:46 smithi179 ceph-mon[35888]: pgmap v466: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:10:47.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:47 smithi043 ceph-mon[31531]: pgmap v467: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:10:47.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:47 smithi179 ceph-mon[35888]: pgmap v467: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:10:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.485+0000 7fdc89a00700 1 -- 172.21.15.43:0/803290106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc841045d0 msgr2=0x7fdc841049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.485+0000 7fdc89a00700 1 --2- 172.21.15.43:0/803290106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc841045d0 0x7fdc841049e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdc740099c0 tx=0x7fdc74009cd0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.486+0000 7fdc89a00700 1 -- 172.21.15.43:0/803290106 shutdown_connections 2024-05-27T07:10:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.486+0000 7fdc89a00700 1 --2- 172.21.15.43:0/803290106 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc84109850 0x7fdc841111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.486+0000 7fdc89a00700 1 --2- 172.21.15.43:0/803290106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc841045d0 0x7fdc841049e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.486+0000 7fdc89a00700 1 --2- 172.21.15.43:0/803290106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc84108f00 0x7fdc84109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.486+0000 7fdc89a00700 1 -- 172.21.15.43:0/803290106 >> 172.21.15.43:0/803290106 conn(0x7fdc840ffec0 msgr2=0x7fdc84102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.486+0000 7fdc89a00700 1 -- 172.21.15.43:0/803290106 shutdown_connections 2024-05-27T07:10:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.486+0000 7fdc89a00700 1 -- 172.21.15.43:0/803290106 wait complete. 2024-05-27T07:10:47.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.487+0000 7fdc89a00700 1 Processor -- start 2024-05-27T07:10:47.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.487+0000 7fdc89a00700 1 -- start start 2024-05-27T07:10:47.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc89a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc841045d0 0x7fdc8419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:47.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc89a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc84108f00 0x7fdc8419d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:47.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc89a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc84109850 0x7fdc841a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:47.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc89a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdc84114380 con 0x7fdc84108f00 2024-05-27T07:10:47.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc89a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdc84114200 con 0x7fdc84109850 2024-05-27T07:10:47.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc89a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdc84114500 con 0x7fdc841045d0 2024-05-27T07:10:47.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc82000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc84108f00 0x7fdc8419d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:47.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc82000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc84108f00 0x7fdc8419d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41614/0 (socket says 172.21.15.43:41614) 2024-05-27T07:10:47.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc82000700 1 -- 172.21.15.43:0/3874402329 learned_addr learned my addr 172.21.15.43:0/3874402329 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:47.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc82a00700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc841045d0 0x7fdc8419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:47.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.488+0000 7fdc83400700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc84109850 0x7fdc841a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:47.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc82000700 1 -- 172.21.15.43:0/3874402329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc841045d0 msgr2=0x7fdc8419cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:47.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc82000700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc841045d0 0x7fdc8419cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc82000700 1 -- 172.21.15.43:0/3874402329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc84109850 msgr2=0x7fdc841a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:47.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc82000700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc84109850 0x7fdc841a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc82000700 1 -- 172.21.15.43:0/3874402329 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc841a1e40 con 0x7fdc84108f00 2024-05-27T07:10:47.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc82000700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc84108f00 0x7fdc8419d510 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000d980 tx=0x7fdc7000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:47.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc7b400700 1 -- 172.21.15.43:0/3874402329 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70017070 con 0x7fdc84108f00 2024-05-27T07:10:47.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc7b400700 1 -- 172.21.15.43:0/3874402329 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc70010460 con 0x7fdc84108f00 2024-05-27T07:10:47.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc89a00700 1 -- 172.21.15.43:0/3874402329 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc841a2130 con 0x7fdc84108f00 2024-05-27T07:10:47.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc7b400700 1 -- 172.21.15.43:0/3874402329 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7000de60 con 0x7fdc84108f00 2024-05-27T07:10:47.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.489+0000 7fdc89a00700 1 -- 172.21.15.43:0/3874402329 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc841a9a50 con 0x7fdc84108f00 2024-05-27T07:10:47.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.491+0000 7fdc89a00700 1 -- 172.21.15.43:0/3874402329 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc840512b0 con 0x7fdc84108f00 2024-05-27T07:10:47.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.494+0000 7fdc7b400700 1 -- 172.21.15.43:0/3874402329 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fdc70010610 con 0x7fdc84108f00 2024-05-27T07:10:47.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.495+0000 7fdc7b400700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc680b2910 0x7fdc680b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:47.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.496+0000 7fdc7b400700 1 -- 172.21.15.43:0/3874402329 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fdc700e22b0 con 0x7fdc84108f00 2024-05-27T07:10:47.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.496+0000 7fdc82a00700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc680b2910 0x7fdc680b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:47.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.496+0000 7fdc82a00700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc680b2910 0x7fdc680b4e20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdc74000c00 tx=0x7fdc74005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:47.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.498+0000 7fdc7b400700 1 -- 172.21.15.43:0/3874402329 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdc700a9a80 con 0x7fdc84108f00 2024-05-27T07:10:47.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:47 smithi132 ceph-mon[35283]: pgmap v467: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:10:47.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.734+0000 7fdc89a00700 1 -- 172.21.15.43:0/3874402329 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc841a9c20 con 0x7fdc84108f00 2024-05-27T07:10:47.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.739+0000 7fdc7b400700 1 -- 172.21.15.43:0/3874402329 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24748 (secure 0 0 0) 0x7fdc700a9380 con 0x7fdc84108f00 2024-05-27T07:10:47.735 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:10:47.735 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":90,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:09:54.019617+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:10:47.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.742+0000 7fdc89a00700 1 -- 172.21.15.43:0/3874402329 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc680b2910 msgr2=0x7fdc680b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:47.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.742+0000 7fdc89a00700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc680b2910 0x7fdc680b4e20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdc74000c00 tx=0x7fdc74005fb0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.742+0000 7fdc89a00700 1 -- 172.21.15.43:0/3874402329 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc84108f00 msgr2=0x7fdc8419d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:47.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.742+0000 7fdc89a00700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc84108f00 0x7fdc8419d510 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000d980 tx=0x7fdc7000dd40 comp rx=0 tx=0).stop 2024-05-27T07:10:47.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.743+0000 7fdc89a00700 1 -- 172.21.15.43:0/3874402329 shutdown_connections 2024-05-27T07:10:47.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.743+0000 7fdc89a00700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdc680b2910 0x7fdc680b4e20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.743+0000 7fdc89a00700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc84109850 0x7fdc841a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.743+0000 7fdc89a00700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc841045d0 0x7fdc8419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.743+0000 7fdc89a00700 1 --2- 172.21.15.43:0/3874402329 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc84108f00 0x7fdc8419d510 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:47.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.743+0000 7fdc89a00700 1 -- 172.21.15.43:0/3874402329 >> 172.21.15.43:0/3874402329 conn(0x7fdc840ffec0 msgr2=0x7fdc84108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:47.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.743+0000 7fdc89a00700 1 -- 172.21.15.43:0/3874402329 shutdown_connections 2024-05-27T07:10:47.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:47.743+0000 7fdc89a00700 1 -- 172.21.15.43:0/3874402329 wait complete. 2024-05-27T07:10:48.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:48 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3874402329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:10:48.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:48 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3874402329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:10:48.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:48 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3874402329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:10:48.629 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-27T07:10:48.630 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:10:49.125 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:49.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:49 smithi043 ceph-mon[31531]: pgmap v468: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:10:49.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:49 smithi179 ceph-mon[35888]: pgmap v468: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:10:49.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:49 smithi132 ceph-mon[35283]: pgmap v468: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:10:51.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.544+0000 7f15ba400700 1 -- 172.21.15.43:0/3222528638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15b4076670 msgr2=0x7f15b4110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:51.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.544+0000 7f15ba400700 1 --2- 172.21.15.43:0/3222528638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15b4076670 0x7f15b4110650 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009990 tx=0x7f15a0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.545+0000 7f15ba400700 1 -- 172.21.15.43:0/3222528638 shutdown_connections 2024-05-27T07:10:51.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.545+0000 7f15ba400700 1 --2- 172.21.15.43:0/3222528638 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15b4110b90 0x7f15b4112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.545+0000 7f15ba400700 1 --2- 172.21.15.43:0/3222528638 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15b4075d20 0x7f15b4076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.545+0000 7f15ba400700 1 --2- 172.21.15.43:0/3222528638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15b4076670 0x7f15b4110650 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.545+0000 7f15ba400700 1 -- 172.21.15.43:0/3222528638 >> 172.21.15.43:0/3222528638 conn(0x7f15b4100030 msgr2=0x7f15b4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:51.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.545+0000 7f15ba400700 1 -- 172.21.15.43:0/3222528638 shutdown_connections 2024-05-27T07:10:51.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.545+0000 7f15ba400700 1 -- 172.21.15.43:0/3222528638 wait complete. 2024-05-27T07:10:51.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.547+0000 7f15ba400700 1 Processor -- start 2024-05-27T07:10:51.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.547+0000 7f15ba400700 1 -- start start 2024-05-27T07:10:51.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.547+0000 7f15ba400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15b4075d20 0x7f15b419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:51.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.547+0000 7f15ba400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15b4076670 0x7f15b419d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:51.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.547+0000 7f15ba400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15b4110b90 0x7f15b41a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:51.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.548+0000 7f15ba400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f15b4116560 con 0x7f15b4110b90 2024-05-27T07:10:51.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.548+0000 7f15ba400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41163e0 con 0x7f15b4075d20 2024-05-27T07:10:51.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.548+0000 7f15ba400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41166e0 con 0x7f15b4076670 2024-05-27T07:10:51.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.548+0000 7f15b3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15b4110b90 0x7f15b41a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:51.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.548+0000 7f15b3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15b4075d20 0x7f15b419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:51.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.548+0000 7f15b3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15b4110b90 0x7f15b41a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41638/0 (socket says 172.21.15.43:41638) 2024-05-27T07:10:51.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.548+0000 7f15b3e00700 1 -- 172.21.15.43:0/1775372465 learned_addr learned my addr 172.21.15.43:0/1775372465 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:10:51.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.548+0000 7f15b3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15b4075d20 0x7f15b419cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:49004/0 (socket says 172.21.15.43:49004) 2024-05-27T07:10:51.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.548+0000 7f15b2a00700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15b4076670 0x7f15b419d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:51.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.549+0000 7f15b3e00700 1 -- 172.21.15.43:0/1775372465 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15b4076670 msgr2=0x7f15b419d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.549+0000 7f15b3e00700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15b4076670 0x7f15b419d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.549+0000 7f15b3e00700 1 -- 172.21.15.43:0/1775372465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15b4075d20 msgr2=0x7f15b419cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.549+0000 7f15b3e00700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15b4075d20 0x7f15b419cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.549+0000 7f15b3e00700 1 -- 172.21.15.43:0/1775372465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b41a1e40 con 0x7f15b4110b90 2024-05-27T07:10:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.549+0000 7f15b3e00700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15b4110b90 0x7f15b41a17a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0094f0 tx=0x7f15ac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.549+0000 7f15abe00700 1 -- 172.21.15.43:0/1775372465 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac004db0 con 0x7f15b4110b90 2024-05-27T07:10:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.550+0000 7f15ba400700 1 -- 172.21.15.43:0/1775372465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b41a2130 con 0x7f15b4110b90 2024-05-27T07:10:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.550+0000 7f15abe00700 1 -- 172.21.15.43:0/1775372465 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f15ac009e20 con 0x7f15b4110b90 2024-05-27T07:10:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.550+0000 7f15abe00700 1 -- 172.21.15.43:0/1775372465 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac015400 con 0x7f15b4110b90 2024-05-27T07:10:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.550+0000 7f15ba400700 1 -- 172.21.15.43:0/1775372465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b41a9a50 con 0x7f15b4110b90 2024-05-27T07:10:51.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.551+0000 7f15ba400700 1 -- 172.21.15.43:0/1775372465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b4077290 con 0x7f15b4110b90 2024-05-27T07:10:51.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.556+0000 7f15abe00700 1 -- 172.21.15.43:0/1775372465 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f15ac005260 con 0x7f15b4110b90 2024-05-27T07:10:51.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.557+0000 7f15abe00700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f15980b2910 0x7f15980b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:10:51.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.557+0000 7f15abe00700 1 -- 172.21.15.43:0/1775372465 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f15ac0e2940 con 0x7f15b4110b90 2024-05-27T07:10:51.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.558+0000 7f15b3400700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f15980b2910 0x7f15980b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:10:51.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.558+0000 7f15b3400700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f15980b2910 0x7f15980b4e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f15a4004630 tx=0x7f15a400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:10:51.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.560+0000 7f15abe00700 1 -- 172.21.15.43:0/1775372465 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f15ac010070 con 0x7f15b4110b90 2024-05-27T07:10:51.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.800+0000 7f15ba400700 1 -- 172.21.15.43:0/1775372465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15b402e5e0 con 0x7f15b4110b90 2024-05-27T07:10:51.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.803+0000 7f15abe00700 1 -- 172.21.15.43:0/1775372465 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24748 (secure 0 0 0) 0x7f15ac0aa110 con 0x7f15b4110b90 2024-05-27T07:10:51.795 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:10:51.795 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":90,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:09:54.019617+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:10:51.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.806+0000 7f15ba400700 1 -- 172.21.15.43:0/1775372465 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f15980b2910 msgr2=0x7f15980b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:51.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.806+0000 7f15ba400700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f15980b2910 0x7f15980b4e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f15a4004630 tx=0x7f15a400b3c0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.806+0000 7f15ba400700 1 -- 172.21.15.43:0/1775372465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15b4110b90 msgr2=0x7f15b41a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:10:51.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.806+0000 7f15ba400700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15b4110b90 0x7f15b41a17a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0094f0 tx=0x7f15ac00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.806+0000 7f15ba400700 1 -- 172.21.15.43:0/1775372465 shutdown_connections 2024-05-27T07:10:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.806+0000 7f15ba400700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f15980b2910 0x7f15980b4e20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.806+0000 7f15ba400700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15b4110b90 0x7f15b41a17a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.806+0000 7f15ba400700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15b4075d20 0x7f15b419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.806+0000 7f15ba400700 1 --2- 172.21.15.43:0/1775372465 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15b4076670 0x7f15b419d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:10:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.806+0000 7f15ba400700 1 -- 172.21.15.43:0/1775372465 >> 172.21.15.43:0/1775372465 conn(0x7f15b4100030 msgr2=0x7f15b4100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:10:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.807+0000 7f15ba400700 1 -- 172.21.15.43:0/1775372465 shutdown_connections 2024-05-27T07:10:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:10:51.807+0000 7f15ba400700 1 -- 172.21.15.43:0/1775372465 wait complete. 2024-05-27T07:10:52.136 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:51 smithi043 ceph-mon[31531]: pgmap v469: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:10:52.136 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:51 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1775372465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:10:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:51 smithi179 ceph-mon[35888]: pgmap v469: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:10:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:51 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1775372465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:10:52.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:51 smithi132 ceph-mon[35283]: pgmap v469: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:10:52.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:51 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1775372465' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:10:52.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-27T07:10:52.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-27T07:10:52.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-27T07:10:52.564 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-27T07:10:52.564 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-27T07:10:52.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-27T07:10:52.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-27T07:10:52.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-27T07:10:52.566 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-27T07:10:52.566 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-27T07:10:52.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-27T07:10:52.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-27T07:10:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:53.757 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:53.763 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:53.764 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:53.774 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:53.774 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:53.784 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:53.787 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:53.791 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:53.793 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:53.880 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:54.192 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:53 smithi043 ceph-mon[31531]: pgmap v470: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:10:54.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:53 smithi179 ceph-mon[35888]: pgmap v470: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:10:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:53 smithi132 ceph-mon[35283]: pgmap v470: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:10:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:10:56.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:55 smithi043 ceph-mon[31531]: pgmap v471: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:10:56.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:55 smithi179 ceph-mon[35888]: pgmap v471: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:10:56.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:55 smithi132 ceph-mon[35283]: pgmap v471: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:10:57.927 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:57 smithi043 ceph-mon[31531]: pgmap v472: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:10:58.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:57 smithi179 ceph-mon[35888]: pgmap v472: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:10:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:57 smithi132 ceph-mon[35283]: pgmap v472: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:00.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:10:59 smithi043 ceph-mon[31531]: pgmap v473: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:00.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:10:59 smithi179 ceph-mon[35888]: pgmap v473: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:00.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:10:59 smithi132 ceph-mon[35283]: pgmap v473: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:01.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:01 smithi043 ceph-mon[31531]: pgmap v474: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:01 smithi179 ceph-mon[35888]: pgmap v474: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:02.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:01 smithi132 ceph-mon[35283]: pgmap v474: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:02.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.979+0000 7f0216a00700 1 -- 172.21.15.43:0/2134577188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f021010b2d0 msgr2=0x7f0210117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:02.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.979+0000 7f0216a00700 1 --2- 172.21.15.43:0/2134577188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f021010b2d0 0x7f0210117800 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f020800b0a0 tx=0x7f020800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:11:02.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.979+0000 7f0216a00700 1 -- 172.21.15.43:0/2134577188 shutdown_connections 2024-05-27T07:11:02.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.979+0000 7f0216a00700 1 --2- 172.21.15.43:0/2134577188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f021010b2d0 0x7f0210117800 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:02.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.979+0000 7f0216a00700 1 --2- 172.21.15.43:0/2134577188 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0210110960 0x7f0210110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:02.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.979+0000 7f0216a00700 1 --2- 172.21.15.43:0/2134577188 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f021010a980 0x7f021010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:02.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.979+0000 7f0216a00700 1 -- 172.21.15.43:0/2134577188 >> 172.21.15.43:0/2134577188 conn(0x7f0210077c70 msgr2=0x7f02100780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:02.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.979+0000 7f0216a00700 1 -- 172.21.15.43:0/2134577188 shutdown_connections 2024-05-27T07:11:02.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.979+0000 7f0216a00700 1 -- 172.21.15.43:0/2134577188 wait complete. 2024-05-27T07:11:02.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.981+0000 7f0216a00700 1 Processor -- start 2024-05-27T07:11:02.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.981+0000 7f0216a00700 1 -- start start 2024-05-27T07:11:02.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.981+0000 7f0216a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f021010a980 0x7f02101a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:02.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.981+0000 7f0216a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f021010b2d0 0x7f02101a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:02.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.981+0000 7f0216a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0210110960 0x7f02101a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:02.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.981+0000 7f0216a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f021011a980 con 0x7f021010a980 2024-05-27T07:11:02.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.981+0000 7f0216a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f021011a800 con 0x7f0210110960 2024-05-27T07:11:02.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.981+0000 7f0216a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f021011ab00 con 0x7f021010b2d0 2024-05-27T07:11:02.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.982+0000 7f0214a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0210110960 0x7f02101a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:02.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.982+0000 7f020fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f021010a980 0x7f02101a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:02.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.982+0000 7f0207e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f021010b2d0 0x7f02101a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:02.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.982+0000 7f020fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f021010a980 0x7f02101a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51546/0 (socket says 172.21.15.43:51546) 2024-05-27T07:11:02.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.982+0000 7f0214a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0210110960 0x7f02101a5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48546/0 (socket says 172.21.15.43:48546) 2024-05-27T07:11:02.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.982+0000 7f020fe00700 1 -- 172.21.15.43:0/811098927 learned_addr learned my addr 172.21.15.43:0/811098927 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:02.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.982+0000 7f0207e00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f021010b2d0 0x7f02101a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:38422/0 (socket says 172.21.15.43:38422) 2024-05-27T07:11:02.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.983+0000 7f020fe00700 1 -- 172.21.15.43:0/811098927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f021010b2d0 msgr2=0x7f02101a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:02.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.983+0000 7f020fe00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f021010b2d0 0x7f02101a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:02.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.983+0000 7f020fe00700 1 -- 172.21.15.43:0/811098927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0210110960 msgr2=0x7f02101a5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:02.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.983+0000 7f020fe00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0210110960 0x7f02101a5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:02.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.983+0000 7f020fe00700 1 -- 172.21.15.43:0/811098927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02101a6200 con 0x7f021010a980 2024-05-27T07:11:02.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.983+0000 7f020fe00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f021010a980 0x7f02101a1330 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f020000d980 tx=0x7f020000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:02.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.983+0000 7f020d600700 1 -- 172.21.15.43:0/811098927 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200017070 con 0x7f021010a980 2024-05-27T07:11:02.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.983+0000 7f020d600700 1 -- 172.21.15.43:0/811098927 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f020000fd20 con 0x7f021010a980 2024-05-27T07:11:02.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.983+0000 7f020d600700 1 -- 172.21.15.43:0/811098927 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020000e3b0 con 0x7f021010a980 2024-05-27T07:11:02.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.983+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02101a6490 con 0x7f021010a980 2024-05-27T07:11:02.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.984+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02101a6980 con 0x7f021010a980 2024-05-27T07:11:02.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.985+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0210063f40 con 0x7f021010a980 2024-05-27T07:11:02.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.988+0000 7f020d600700 1 -- 172.21.15.43:0/811098927 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f020000f890 con 0x7f021010a980 2024-05-27T07:11:02.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.989+0000 7f020d600700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f01f40b2910 0x7f01f40b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:02.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.989+0000 7f020d600700 1 -- 172.21.15.43:0/811098927 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f02000e2590 con 0x7f021010a980 2024-05-27T07:11:02.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.989+0000 7f0207e00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f01f40b2910 0x7f01f40b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:02.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.990+0000 7f020d600700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] conn(0x7f01f40bdbc0 0x7f01f40c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:02.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.990+0000 7f020d600700 1 -- 172.21.15.43:0/811098927 --> [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f020000d130 con 0x7f01f40bdbc0 2024-05-27T07:11:02.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.990+0000 7f020d600700 1 -- 172.21.15.43:0/811098927 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f02000e2940 con 0x7f021010a980 2024-05-27T07:11:02.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.990+0000 7f0214a00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] conn(0x7f01f40bdbc0 0x7f01f40c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:02.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.990+0000 7f0207e00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f01f40b2910 0x7f01f40b4e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0210109800 tx=0x7f01f800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:02.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.991+0000 7f0214a00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] conn(0x7f01f40bdbc0 0x7f01f40c00b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:02.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:02.992+0000 7f020d600700 1 -- 172.21.15.43:0/811098927 <== osd.0 v2:172.21.15.43:6802/3368788577 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f020000d130 con 0x7f01f40bdbc0 2024-05-27T07:11:03.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.011+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 --> [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f02100643c0 con 0x7f01f40bdbc0 2024-05-27T07:11:03.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.011+0000 7f020d600700 1 -- 172.21.15.43:0/811098927 <== osd.0 v2:172.21.15.43:6802/3368788577 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f02100643c0 con 0x7f01f40bdbc0 2024-05-27T07:11:03.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.012+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] conn(0x7f01f40bdbc0 msgr2=0x7f01f40c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:03.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.012+0000 7f0216a00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] conn(0x7f01f40bdbc0 0x7f01f40c00b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:03.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.012+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f01f40b2910 msgr2=0x7f01f40b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:03.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.012+0000 7f0216a00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f01f40b2910 0x7f01f40b4e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0210109800 tx=0x7f01f800b410 comp rx=0 tx=0).stop 2024-05-27T07:11:03.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.012+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f021010a980 msgr2=0x7f02101a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:03.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.012+0000 7f0216a00700 1 --2- 172.21.15.43:0/811098927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f021010a980 0x7f02101a1330 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f020000d980 tx=0x7f020000dc90 comp rx=0 tx=0).stop 2024-05-27T07:11:03.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.012+0000 7f0214a00700 1 -- 172.21.15.43:0/811098927 reap_dead start 2024-05-27T07:11:03.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.013+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 shutdown_connections 2024-05-27T07:11:03.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.013+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 >> 172.21.15.43:0/811098927 conn(0x7f0210077c70 msgr2=0x7f021007b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:03.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.013+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 shutdown_connections 2024-05-27T07:11:03.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:03.013+0000 7f0216a00700 1 -- 172.21.15.43:0/811098927 wait complete. 2024-05-27T07:11:03.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:03 smithi043 ceph-mon[31531]: pgmap v475: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:11:04.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.095+0000 7fdcfa400700 1 -- 172.21.15.43:0/3022954113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdcf410e910 msgr2=0x7fdcf4110df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:04.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.095+0000 7fdcfa400700 1 --2- 172.21.15.43:0/3022954113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdcf410e910 0x7fdcf4110df0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdcec00b0a0 tx=0x7fdcec00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:11:04.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.095+0000 7fdcfa400700 1 -- 172.21.15.43:0/3022954113 shutdown_connections 2024-05-27T07:11:04.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.095+0000 7fdcfa400700 1 --2- 172.21.15.43:0/3022954113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdcf410e910 0x7fdcf4110df0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:04.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.095+0000 7fdcfa400700 1 --2- 172.21.15.43:0/3022954113 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdcf41054e0 0x7fdcf41058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:04.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.095+0000 7fdcfa400700 1 --2- 172.21.15.43:0/3022954113 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcf4105e30 0x7fdcf410e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:04.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.095+0000 7fdcfa400700 1 -- 172.21.15.43:0/3022954113 >> 172.21.15.43:0/3022954113 conn(0x7fdcf4078df0 msgr2=0x7fdcf4079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:04.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.095+0000 7fdcfa400700 1 -- 172.21.15.43:0/3022954113 shutdown_connections 2024-05-27T07:11:04.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.095+0000 7fdcfa400700 1 -- 172.21.15.43:0/3022954113 wait complete. 2024-05-27T07:11:04.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.097+0000 7fdcfa400700 1 Processor -- start 2024-05-27T07:11:04.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.097+0000 7fdcfa400700 1 -- start start 2024-05-27T07:11:04.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.097+0000 7fdcfa400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdcf41054e0 0x7fdcf419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:04.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.097+0000 7fdcfa400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcf4105e30 0x7fdcf419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:04.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.097+0000 7fdcfa400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdcf410e910 0x7fdcf41a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:04.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.097+0000 7fdcfa400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf4114320 con 0x7fdcf410e910 2024-05-27T07:11:04.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.097+0000 7fdcfa400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf41141a0 con 0x7fdcf41054e0 2024-05-27T07:11:04.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.097+0000 7fdcfa400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf41144a0 con 0x7fdcf4105e30 2024-05-27T07:11:04.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdcf410e910 0x7fdcf41a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:04.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdcf410e910 0x7fdcf41a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51570/0 (socket says 172.21.15.43:51570) 2024-05-27T07:11:04.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf3e00700 1 -- 172.21.15.43:0/2673463882 learned_addr learned my addr 172.21.15.43:0/2673463882 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:04.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf3400700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdcf41054e0 0x7fdcf419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:04.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf2a00700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcf4105e30 0x7fdcf419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:04.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf3e00700 1 -- 172.21.15.43:0/2673463882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcf4105e30 msgr2=0x7fdcf419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:04.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf3e00700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcf4105e30 0x7fdcf419d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:04.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf3e00700 1 -- 172.21.15.43:0/2673463882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdcf41054e0 msgr2=0x7fdcf419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:04.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf3e00700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdcf41054e0 0x7fdcf419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:04.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf3e00700 1 -- 172.21.15.43:0/2673463882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf41a1da0 con 0x7fdcf410e910 2024-05-27T07:11:04.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.098+0000 7fdcf2a00700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcf4105e30 0x7fdcf419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:11:04.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.099+0000 7fdcf3e00700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdcf410e910 0x7fdcf41a1700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdcec00b830 tx=0x7fdcec004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:04.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.099+0000 7fdcebe00700 1 -- 172.21.15.43:0/2673463882 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcec021070 con 0x7fdcf410e910 2024-05-27T07:11:04.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.099+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf41a2030 con 0x7fdcf410e910 2024-05-27T07:11:04.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.099+0000 7fdcebe00700 1 -- 172.21.15.43:0/2673463882 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdcec007b80 con 0x7fdcf410e910 2024-05-27T07:11:04.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.099+0000 7fdcebe00700 1 -- 172.21.15.43:0/2673463882 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcec0052a0 con 0x7fdcf410e910 2024-05-27T07:11:04.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.099+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf41a99b0 con 0x7fdcf410e910 2024-05-27T07:11:04.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.100+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdcf4103be0 con 0x7fdcf410e910 2024-05-27T07:11:04.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.104+0000 7fdcebe00700 1 -- 172.21.15.43:0/2673463882 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fdcec024070 con 0x7fdcf410e910 2024-05-27T07:11:04.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.105+0000 7fdcebe00700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdcd80b2990 0x7fdcd80b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:04.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.105+0000 7fdcebe00700 1 -- 172.21.15.43:0/2673463882 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fdcec00f020 con 0x7fdcf410e910 2024-05-27T07:11:04.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.105+0000 7fdcf3400700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdcd80b2990 0x7fdcd80b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:04.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.105+0000 7fdcebe00700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] conn(0x7fdcd80bdc40 0x7fdcd80c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:04.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.106+0000 7fdcebe00700 1 -- 172.21.15.43:0/2673463882 --> [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdcec02ce50 con 0x7fdcd80bdc40 2024-05-27T07:11:04.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.106+0000 7fdcebe00700 1 -- 172.21.15.43:0/2673463882 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdcec0e4350 con 0x7fdcf410e910 2024-05-27T07:11:04.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.106+0000 7fdcf3400700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdcd80b2990 0x7fdcd80b4ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdce4005950 tx=0x7fdce400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:04.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.106+0000 7fdcf2a00700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] conn(0x7fdcd80bdc40 0x7fdcd80c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:04.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.107+0000 7fdcf2a00700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] conn(0x7fdcd80bdc40 0x7fdcd80c0130 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.110+0000 7fdcebe00700 1 -- 172.21.15.43:0/2673463882 <== osd.8 v2:172.21.15.179:6816/2617404629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdcec02ce50 con 0x7fdcd80bdc40 2024-05-27T07:11:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.129+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 --> [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdcf4076270 con 0x7fdcd80bdc40 2024-05-27T07:11:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.130+0000 7fdcebe00700 1 -- 172.21.15.43:0/2673463882 <== osd.8 v2:172.21.15.179:6816/2617404629 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdcf4076270 con 0x7fdcd80bdc40 2024-05-27T07:11:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.131+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 >> [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] conn(0x7fdcd80bdc40 msgr2=0x7fdcd80c0130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.131+0000 7fdcfa400700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] conn(0x7fdcd80bdc40 0x7fdcd80c0130 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.131+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdcd80b2990 msgr2=0x7fdcd80b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.131+0000 7fdcfa400700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fdcd80b2990 0x7fdcd80b4ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdce4005950 tx=0x7fdce400b410 comp rx=0 tx=0).stop 2024-05-27T07:11:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.131+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdcf410e910 msgr2=0x7fdcf41a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.131+0000 7fdcfa400700 1 --2- 172.21.15.43:0/2673463882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdcf410e910 0x7fdcf41a1700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdcec00b830 tx=0x7fdcec004d00 comp rx=0 tx=0).stop 2024-05-27T07:11:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.131+0000 7fdcf3e00700 1 -- 172.21.15.43:0/2673463882 reap_dead start 2024-05-27T07:11:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.131+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 shutdown_connections 2024-05-27T07:11:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.131+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 >> 172.21.15.43:0/2673463882 conn(0x7fdcf4078df0 msgr2=0x7fdcf4103180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.131+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 shutdown_connections 2024-05-27T07:11:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:04.132+0000 7fdcfa400700 1 -- 172.21.15.43:0/2673463882 wait complete. 2024-05-27T07:11:04.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:03 smithi179 ceph-mon[35888]: pgmap v475: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:11:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:03 smithi132 ceph-mon[35283]: pgmap v475: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:11:06.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:05 smithi043 ceph-mon[31531]: pgmap v476: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:05 smithi179 ceph-mon[35888]: pgmap v476: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:05 smithi132 ceph-mon[35283]: pgmap v476: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:08.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:07 smithi043 ceph-mon[31531]: pgmap v477: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:07 smithi179 ceph-mon[35888]: pgmap v477: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:08.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:07 smithi132 ceph-mon[35283]: pgmap v477: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:09.376 INFO:teuthology.orchestra.run.smithi043.stdout:231928234060 2024-05-27T07:11:09.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-27T07:11:10.158 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:09 smithi043 ceph-mon[31531]: pgmap v478: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:10.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:09 smithi179 ceph-mon[35888]: pgmap v478: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:10.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:09 smithi132 ceph-mon[35283]: pgmap v478: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:10.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.593+0000 7fa65b600700 1 -- 172.21.15.43:0/4257378375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6541045e0 msgr2=0x7fa6541049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:10.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.593+0000 7fa65b600700 1 --2- 172.21.15.43:0/4257378375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6541045e0 0x7fa6541049f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa6480099c0 tx=0x7fa648009cd0 comp rx=0 tx=0).stop 2024-05-27T07:11:10.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.593+0000 7fa65b600700 1 -- 172.21.15.43:0/4257378375 shutdown_connections 2024-05-27T07:11:10.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.593+0000 7fa65b600700 1 --2- 172.21.15.43:0/4257378375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa654109860 0x7fa6541111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:10.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.593+0000 7fa65b600700 1 --2- 172.21.15.43:0/4257378375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6541045e0 0x7fa6541049f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:10.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.593+0000 7fa65b600700 1 --2- 172.21.15.43:0/4257378375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa654108f10 0x7fa654109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:10.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.593+0000 7fa65b600700 1 -- 172.21.15.43:0/4257378375 >> 172.21.15.43:0/4257378375 conn(0x7fa654100030 msgr2=0x7fa654102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:10.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.593+0000 7fa65b600700 1 -- 172.21.15.43:0/4257378375 shutdown_connections 2024-05-27T07:11:10.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.593+0000 7fa65b600700 1 -- 172.21.15.43:0/4257378375 wait complete. 2024-05-27T07:11:10.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.595+0000 7fa65b600700 1 Processor -- start 2024-05-27T07:11:10.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.595+0000 7fa65b600700 1 -- start start 2024-05-27T07:11:10.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.595+0000 7fa65b600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6541045e0 0x7fa65419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:10.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.595+0000 7fa65b600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa654108f10 0x7fa65419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:10.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.595+0000 7fa65b600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa654109860 0x7fa6541a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:10.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.595+0000 7fa65b600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa654114390 con 0x7fa654109860 2024-05-27T07:11:10.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.595+0000 7fa65b600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa654114210 con 0x7fa6541045e0 2024-05-27T07:11:10.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.595+0000 7fa65b600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa654114510 con 0x7fa654108f10 2024-05-27T07:11:10.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.596+0000 7fa659600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa654109860 0x7fa6541a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:10.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.596+0000 7fa659600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa654109860 0x7fa6541a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32910/0 (socket says 172.21.15.43:32910) 2024-05-27T07:11:10.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.596+0000 7fa659600700 1 -- 172.21.15.43:0/3168723029 learned_addr learned my addr 172.21.15.43:0/3168723029 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:10.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.596+0000 7fa658c00700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6541045e0 0x7fa65419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:10.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.596+0000 7fa653e00700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa654108f10 0x7fa65419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:10.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.596+0000 7fa659600700 1 -- 172.21.15.43:0/3168723029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa654108f10 msgr2=0x7fa65419d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:10.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.597+0000 7fa659600700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa654108f10 0x7fa65419d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:10.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.597+0000 7fa659600700 1 -- 172.21.15.43:0/3168723029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6541045e0 msgr2=0x7fa65419d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:10.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.597+0000 7fa659600700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6541045e0 0x7fa65419d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:10.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.597+0000 7fa659600700 1 -- 172.21.15.43:0/3168723029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6541a1ed0 con 0x7fa654109860 2024-05-27T07:11:10.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.597+0000 7fa659600700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa654109860 0x7fa6541a1830 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00ef30 tx=0x7fa64c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:10.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.597+0000 7fa651600700 1 -- 172.21.15.43:0/3168723029 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c0092e0 con 0x7fa654109860 2024-05-27T07:11:10.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.597+0000 7fa651600700 1 -- 172.21.15.43:0/3168723029 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa64c004bb0 con 0x7fa654109860 2024-05-27T07:11:10.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.597+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6541a2160 con 0x7fa654109860 2024-05-27T07:11:10.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.597+0000 7fa651600700 1 -- 172.21.15.43:0/3168723029 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c004020 con 0x7fa654109860 2024-05-27T07:11:10.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.598+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6541a2650 con 0x7fa654109860 2024-05-27T07:11:10.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.599+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa654076cc0 con 0x7fa654109860 2024-05-27T07:11:10.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.601+0000 7fa651600700 1 -- 172.21.15.43:0/3168723029 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa64c01b030 con 0x7fa654109860 2024-05-27T07:11:10.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.603+0000 7fa651600700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fa63c0b2910 0x7fa63c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:10.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.603+0000 7fa651600700 1 -- 172.21.15.43:0/3168723029 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fa64c0e2bc0 con 0x7fa654109860 2024-05-27T07:11:10.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.603+0000 7fa658c00700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fa63c0b2910 0x7fa63c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:10.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.604+0000 7fa651600700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] conn(0x7fa63c0bdb40 0x7fa63c0c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:10.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.604+0000 7fa651600700 1 -- 172.21.15.43:0/3168723029 --> [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa63c0c0700 con 0x7fa63c0bdb40 2024-05-27T07:11:10.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.604+0000 7fa651600700 1 -- 172.21.15.43:0/3168723029 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fa64c0e2f70 con 0x7fa654109860 2024-05-27T07:11:10.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.604+0000 7fa653e00700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] conn(0x7fa63c0bdb40 0x7fa63c0c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:10.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.604+0000 7fa658c00700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fa63c0b2910 0x7fa63c0b4e20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa648000c00 tx=0x7fa64801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:10.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.605+0000 7fa653e00700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] conn(0x7fa63c0bdb40 0x7fa63c0c0030 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:10.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.609+0000 7fa651600700 1 -- 172.21.15.43:0/3168723029 <== osd.5 v2:172.21.15.179:6808/1714122228 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa63c0c0700 con 0x7fa63c0bdb40 2024-05-27T07:11:10.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.626+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 --> [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa654105a90 con 0x7fa63c0bdb40 2024-05-27T07:11:10.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.627+0000 7fa651600700 1 -- 172.21.15.43:0/3168723029 <== osd.5 v2:172.21.15.179:6808/1714122228 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa654105a90 con 0x7fa63c0bdb40 2024-05-27T07:11:10.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 >> [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] conn(0x7fa63c0bdb40 msgr2=0x7fa63c0c0030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:10.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa65b600700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] conn(0x7fa63c0bdb40 0x7fa63c0c0030 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:10.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fa63c0b2910 msgr2=0x7fa63c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:10.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa65b600700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fa63c0b2910 0x7fa63c0b4e20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa648000c00 tx=0x7fa64801a040 comp rx=0 tx=0).stop 2024-05-27T07:11:10.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa654109860 msgr2=0x7fa6541a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:10.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa65b600700 1 --2- 172.21.15.43:0/3168723029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa654109860 0x7fa6541a1830 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa64c00ef30 tx=0x7fa64c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:11:10.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa659600700 1 -- 172.21.15.43:0/3168723029 reap_dead start 2024-05-27T07:11:10.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 shutdown_connections 2024-05-27T07:11:10.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 >> 172.21.15.43:0/3168723029 conn(0x7fa654100030 msgr2=0x7fa65410a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:10.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 shutdown_connections 2024-05-27T07:11:10.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:10.628+0000 7fa65b600700 1 -- 172.21.15.43:0/3168723029 wait complete. 2024-05-27T07:11:11.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.020+0000 7f5030400700 1 -- 172.21.15.43:0/115150684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5028106560 msgr2=0x7f5028106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:11.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.020+0000 7f5030400700 1 --2- 172.21.15.43:0/115150684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5028106560 0x7f5028106970 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5014009990 tx=0x7f5014009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:11.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.020+0000 7f5030400700 1 -- 172.21.15.43:0/115150684 shutdown_connections 2024-05-27T07:11:11.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.020+0000 7f5030400700 1 --2- 172.21.15.43:0/115150684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5028106eb0 0x7f5028113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:11.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.020+0000 7f5030400700 1 --2- 172.21.15.43:0/115150684 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f502810c5a0 0x7f502810c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.020+0000 7f5030400700 1 --2- 172.21.15.43:0/115150684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5028106560 0x7f5028106970 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.020+0000 7f5030400700 1 -- 172.21.15.43:0/115150684 >> 172.21.15.43:0/115150684 conn(0x7f5028102040 msgr2=0x7f5028104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.020+0000 7f5030400700 1 -- 172.21.15.43:0/115150684 shutdown_connections 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.020+0000 7f5030400700 1 -- 172.21.15.43:0/115150684 wait complete. 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.021+0000 7f5030400700 1 Processor -- start 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.022+0000 7f5030400700 1 -- start start 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.022+0000 7f5030400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5028106560 0x7f5028077cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.022+0000 7f5030400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5028106eb0 0x7f5028078200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.022+0000 7f5030400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502810c5a0 0x7f5028078740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.022+0000 7f5030400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5028116530 con 0x7f5028106560 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f5030400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f50281163b0 con 0x7f502810c5a0 2024-05-27T07:11:11.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f5030400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f50281166b0 con 0x7f5028106eb0 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5028106560 0x7f5028077cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5028106560 0x7f5028077cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32916/0 (socket says 172.21.15.43:32916) 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502da00700 1 -- 172.21.15.43:0/324627649 learned_addr learned my addr 172.21.15.43:0/324627649 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502e400700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502810c5a0 0x7f5028078740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502d000700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5028106eb0 0x7f5028078200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502da00700 1 -- 172.21.15.43:0/324627649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5028106eb0 msgr2=0x7f5028078200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502da00700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5028106eb0 0x7f5028078200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502da00700 1 -- 172.21.15.43:0/324627649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502810c5a0 msgr2=0x7f5028078740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502da00700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502810c5a0 0x7f5028078740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502da00700 1 -- 172.21.15.43:0/324627649 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502819cc60 con 0x7f5028106560 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.023+0000 7f502e400700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502810c5a0 0x7f5028078740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.024+0000 7f502da00700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5028106560 0x7f5028077cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f501c007f10 tx=0x7f501c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:11.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.024+0000 7f5026000700 1 -- 172.21.15.43:0/324627649 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c0133f0 con 0x7f5028106560 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.024+0000 7f5026000700 1 -- 172.21.15.43:0/324627649 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f501c018070 con 0x7f5028106560 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.024+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f502819cf50 con 0x7f5028106560 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.024+0000 7f5026000700 1 -- 172.21.15.43:0/324627649 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c012640 con 0x7f5028106560 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.024+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f502819d490 con 0x7f5028106560 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.025+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5028063ec0 con 0x7f5028106560 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.028+0000 7f5026000700 1 -- 172.21.15.43:0/324627649 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f501c013ab0 con 0x7f5028106560 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.030+0000 7f5026000700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50100b2910 0x7f50100b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.030+0000 7f5026000700 1 -- 172.21.15.43:0/324627649 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f501c0e2950 con 0x7f5028106560 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.030+0000 7f502d000700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50100b2910 0x7f50100b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.030+0000 7f5026000700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] conn(0x7f50100bdbc0 0x7f50100c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.030+0000 7f5026000700 1 -- 172.21.15.43:0/324627649 --> [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f501c023e50 con 0x7f50100bdbc0 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.030+0000 7f5026000700 1 -- 172.21.15.43:0/324627649 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f501c0e2d00 con 0x7f5028106560 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.031+0000 7f502d000700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50100b2910 0x7f50100b4e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f50281057d0 tx=0x7f501400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:11.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.031+0000 7f502e400700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] conn(0x7f50100bdbc0 0x7f50100c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.032+0000 7f502e400700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] conn(0x7f50100bdbc0 0x7f50100c00b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.035+0000 7f5026000700 1 -- 172.21.15.43:0/324627649 <== osd.10 v2:172.21.15.132:6824/3139130496 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f501c023e50 con 0x7f50100bdbc0 2024-05-27T07:11:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.053+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 --> [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5028064340 con 0x7f50100bdbc0 2024-05-27T07:11:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.054+0000 7f5026000700 1 -- 172.21.15.43:0/324627649 <== osd.10 v2:172.21.15.132:6824/3139130496 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5028064340 con 0x7f50100bdbc0 2024-05-27T07:11:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 >> [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] conn(0x7f50100bdbc0 msgr2=0x7f50100c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f5030400700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] conn(0x7f50100bdbc0 0x7f50100c00b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50100b2910 msgr2=0x7f50100b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:11.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f5030400700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f50100b2910 0x7f50100b4e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f50281057d0 tx=0x7f501400b540 comp rx=0 tx=0).stop 2024-05-27T07:11:11.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5028106560 msgr2=0x7f5028077cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:11.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f5030400700 1 --2- 172.21.15.43:0/324627649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5028106560 0x7f5028077cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f501c007f10 tx=0x7f501c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:11:11.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f502e400700 1 -- 172.21.15.43:0/324627649 reap_dead start 2024-05-27T07:11:11.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 shutdown_connections 2024-05-27T07:11:11.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 >> 172.21.15.43:0/324627649 conn(0x7f5028102040 msgr2=0x7f50281038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:11.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 shutdown_connections 2024-05-27T07:11:11.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:11.055+0000 7f5030400700 1 -- 172.21.15.43:0/324627649 wait complete. 2024-05-27T07:11:11.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:11 smithi043 ceph-mon[31531]: pgmap v479: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:12.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:11 smithi179 ceph-mon[35888]: pgmap v479: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:11 smithi132 ceph-mon[35283]: pgmap v479: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.200+0000 7ff165800700 1 -- 172.21.15.43:0/3452227258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff160110960 msgr2=0x7ff160110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.200+0000 7ff165800700 1 --2- 172.21.15.43:0/3452227258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff160110960 0x7ff160110d70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff150009990 tx=0x7ff150009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.200+0000 7ff165800700 1 -- 172.21.15.43:0/3452227258 shutdown_connections 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.200+0000 7ff165800700 1 --2- 172.21.15.43:0/3452227258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff16010b2d0 0x7ff160117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.200+0000 7ff165800700 1 --2- 172.21.15.43:0/3452227258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff160110960 0x7ff160110d70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.200+0000 7ff165800700 1 --2- 172.21.15.43:0/3452227258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff16010a980 0x7ff16010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.200+0000 7ff165800700 1 -- 172.21.15.43:0/3452227258 >> 172.21.15.43:0/3452227258 conn(0x7ff160077c70 msgr2=0x7ff1600780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.201+0000 7ff165800700 1 -- 172.21.15.43:0/3452227258 shutdown_connections 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.201+0000 7ff165800700 1 -- 172.21.15.43:0/3452227258 wait complete. 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.202+0000 7ff165800700 1 Processor -- start 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.202+0000 7ff165800700 1 -- start start 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.202+0000 7ff165800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff16010a980 0x7ff1601a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.202+0000 7ff165800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff16010b2d0 0x7ff1601a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff165800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff160110960 0x7ff1601a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff165800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff16011a960 con 0x7ff160110960 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff165800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff16011a7e0 con 0x7ff16010a980 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff165800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff16011aae0 con 0x7ff16010b2d0 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff15f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff160110960 0x7ff1601a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff15f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff160110960 0x7ff1601a5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32936/0 (socket says 172.21.15.43:32936) 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff15f400700 1 -- 172.21.15.43:0/1082980807 learned_addr learned my addr 172.21.15.43:0/1082980807 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff15ea00700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff16010a980 0x7ff1601a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff156000700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff16010b2d0 0x7ff1601a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff15f400700 1 -- 172.21.15.43:0/1082980807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff16010b2d0 msgr2=0x7ff1601a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.203+0000 7ff15f400700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff16010b2d0 0x7ff1601a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.204+0000 7ff15f400700 1 -- 172.21.15.43:0/1082980807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff16010a980 msgr2=0x7ff1601a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.204+0000 7ff15f400700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff16010a980 0x7ff1601a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.204+0000 7ff15f400700 1 -- 172.21.15.43:0/1082980807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1601a61d0 con 0x7ff160110960 2024-05-27T07:11:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.204+0000 7ff156000700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff16010b2d0 0x7ff1601a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.204+0000 7ff15f400700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff160110960 0x7ff1601a5be0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff158007d70 tx=0x7ff15800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.204+0000 7ff157e00700 1 -- 172.21.15.43:0/1082980807 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff158016020 con 0x7ff160110960 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.204+0000 7ff157e00700 1 -- 172.21.15.43:0/1082980807 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff15800ad40 con 0x7ff160110960 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.204+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1601a64c0 con 0x7ff160110960 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.204+0000 7ff157e00700 1 -- 172.21.15.43:0/1082980807 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1580115e0 con 0x7ff160110960 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.204+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1601a69b0 con 0x7ff160110960 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.206+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff160063f00 con 0x7ff160110960 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.208+0000 7ff157e00700 1 -- 172.21.15.43:0/1082980807 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff158019020 con 0x7ff160110960 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.210+0000 7ff157e00700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff1440b2910 0x7ff1440b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.210+0000 7ff157e00700 1 -- 172.21.15.43:0/1082980807 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7ff1580e2a40 con 0x7ff160110960 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.210+0000 7ff15ea00700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff1440b2910 0x7ff1440b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.210+0000 7ff157e00700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] conn(0x7ff1440bdbc0 0x7ff1440c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.210+0000 7ff157e00700 1 -- 172.21.15.43:0/1082980807 --> [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff158024e50 con 0x7ff1440bdbc0 2024-05-27T07:11:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.210+0000 7ff157e00700 1 -- 172.21.15.43:0/1082980807 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7ff1580e2e90 con 0x7ff160110960 2024-05-27T07:11:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.211+0000 7ff15ea00700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff1440b2910 0x7ff1440b4e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff150006010 tx=0x7ff15000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.211+0000 7ff156000700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] conn(0x7ff1440bdbc0 0x7ff1440c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.212+0000 7ff156000700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] conn(0x7ff1440bdbc0 0x7ff1440c00b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.215+0000 7ff157e00700 1 -- 172.21.15.43:0/1082980807 <== osd.1 v2:172.21.15.132:6800/3434174667 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff158024e50 con 0x7ff1440bdbc0 2024-05-27T07:11:13.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.233+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 --> [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff160064380 con 0x7ff1440bdbc0 2024-05-27T07:11:13.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.235+0000 7ff157e00700 1 -- 172.21.15.43:0/1082980807 <== osd.1 v2:172.21.15.132:6800/3434174667 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff160064380 con 0x7ff1440bdbc0 2024-05-27T07:11:13.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.235+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 >> [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] conn(0x7ff1440bdbc0 msgr2=0x7ff1440c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:13.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.235+0000 7ff165800700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] conn(0x7ff1440bdbc0 0x7ff1440c00b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.236+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff1440b2910 msgr2=0x7ff1440b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.236+0000 7ff165800700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff1440b2910 0x7ff1440b4e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff150006010 tx=0x7ff15000b540 comp rx=0 tx=0).stop 2024-05-27T07:11:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.236+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff160110960 msgr2=0x7ff1601a5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.236+0000 7ff165800700 1 --2- 172.21.15.43:0/1082980807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff160110960 0x7ff1601a5be0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff158007d70 tx=0x7ff15800a340 comp rx=0 tx=0).stop 2024-05-27T07:11:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.236+0000 7ff15f400700 1 -- 172.21.15.43:0/1082980807 reap_dead start 2024-05-27T07:11:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.236+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 shutdown_connections 2024-05-27T07:11:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.236+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 >> 172.21.15.43:0/1082980807 conn(0x7ff160077c70 msgr2=0x7ff16007bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.236+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 shutdown_connections 2024-05-27T07:11:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:13.236+0000 7ff165800700 1 -- 172.21.15.43:0/1082980807 wait complete. 2024-05-27T07:11:13.802 INFO:teuthology.orchestra.run.smithi043.stdout:34359738504 2024-05-27T07:11:13.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-27T07:11:13.982 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:13 smithi043 ceph-mon[31531]: pgmap v480: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-27T07:11:14.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:13 smithi179 ceph-mon[35888]: pgmap v480: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-27T07:11:14.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:13 smithi132 ceph-mon[35283]: pgmap v480: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-27T07:11:14.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.354+0000 7fd742c00700 1 -- 172.21.15.43:0/3136194532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c109860 msgr2=0x7fd73c1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:14.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.354+0000 7fd742c00700 1 --2- 172.21.15.43:0/3136194532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c109860 0x7fd73c1111c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd73400b0d0 tx=0x7fd73400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:11:14.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.354+0000 7fd742c00700 1 -- 172.21.15.43:0/3136194532 shutdown_connections 2024-05-27T07:11:14.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.354+0000 7fd742c00700 1 --2- 172.21.15.43:0/3136194532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c109860 0x7fd73c1111c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:14.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.354+0000 7fd742c00700 1 --2- 172.21.15.43:0/3136194532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd73c1045e0 0x7fd73c1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:14.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.354+0000 7fd742c00700 1 --2- 172.21.15.43:0/3136194532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd73c108f10 0x7fd73c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:14.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.354+0000 7fd742c00700 1 -- 172.21.15.43:0/3136194532 >> 172.21.15.43:0/3136194532 conn(0x7fd73c100030 msgr2=0x7fd73c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:14.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.354+0000 7fd742c00700 1 -- 172.21.15.43:0/3136194532 shutdown_connections 2024-05-27T07:11:14.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.354+0000 7fd742c00700 1 -- 172.21.15.43:0/3136194532 wait complete. 2024-05-27T07:11:14.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.356+0000 7fd742c00700 1 Processor -- start 2024-05-27T07:11:14.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.356+0000 7fd742c00700 1 -- start start 2024-05-27T07:11:14.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.357+0000 7fd742c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c1045e0 0x7fd73c075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:14.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.357+0000 7fd742c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd73c108f10 0x7fd73c075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:14.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.357+0000 7fd742c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd73c109860 0x7fd73c076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:14.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.357+0000 7fd742c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c114340 con 0x7fd73c1045e0 2024-05-27T07:11:14.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.357+0000 7fd742c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c1141c0 con 0x7fd73c109860 2024-05-27T07:11:14.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.357+0000 7fd742c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c1144c0 con 0x7fd73c108f10 2024-05-27T07:11:14.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.357+0000 7fd73be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c1045e0 0x7fd73c075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:14.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.357+0000 7fd73be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c1045e0 0x7fd73c075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32944/0 (socket says 172.21.15.43:32944) 2024-05-27T07:11:14.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.357+0000 7fd73be00700 1 -- 172.21.15.43:0/562873127 learned_addr learned my addr 172.21.15.43:0/562873127 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.358+0000 7fd740c00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd73c109860 0x7fd73c076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.358+0000 7fd73b400700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd73c108f10 0x7fd73c075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.358+0000 7fd73be00700 1 -- 172.21.15.43:0/562873127 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd73c108f10 msgr2=0x7fd73c075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.358+0000 7fd73be00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd73c108f10 0x7fd73c075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.358+0000 7fd73be00700 1 -- 172.21.15.43:0/562873127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd73c109860 msgr2=0x7fd73c076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.358+0000 7fd73be00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd73c109860 0x7fd73c076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.358+0000 7fd73be00700 1 -- 172.21.15.43:0/562873127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c1a9110 con 0x7fd73c1045e0 2024-05-27T07:11:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.358+0000 7fd73be00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c1045e0 0x7fd73c075ab0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00d950 tx=0x7fd72c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.359+0000 7fd738c00700 1 -- 172.21.15.43:0/562873127 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c016070 con 0x7fd73c1045e0 2024-05-27T07:11:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.359+0000 7fd738c00700 1 -- 172.21.15.43:0/562873127 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd72c010460 con 0x7fd73c1045e0 2024-05-27T07:11:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.359+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73c1a9400 con 0x7fd73c1045e0 2024-05-27T07:11:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.359+0000 7fd738c00700 1 -- 172.21.15.43:0/562873127 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c00f620 con 0x7fd73c1045e0 2024-05-27T07:11:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.359+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd73c1a9940 con 0x7fd73c1045e0 2024-05-27T07:11:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.360+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd73c063ec0 con 0x7fd73c1045e0 2024-05-27T07:11:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.363+0000 7fd738c00700 1 -- 172.21.15.43:0/562873127 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd72c019070 con 0x7fd73c1045e0 2024-05-27T07:11:14.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.364+0000 7fd738c00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd7200b2910 0x7fd7200b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:14.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.365+0000 7fd738c00700 1 -- 172.21.15.43:0/562873127 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fd72c0e28e0 con 0x7fd73c1045e0 2024-05-27T07:11:14.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.365+0000 7fd73b400700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd7200b2910 0x7fd7200b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:14.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.365+0000 7fd738c00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] conn(0x7fd7200bdbc0 0x7fd7200c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:14.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.365+0000 7fd738c00700 1 -- 172.21.15.43:0/562873127 --> [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7200c0780 con 0x7fd7200bdbc0 2024-05-27T07:11:14.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.365+0000 7fd738c00700 1 -- 172.21.15.43:0/562873127 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd72c0e2c90 con 0x7fd73c1045e0 2024-05-27T07:11:14.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.366+0000 7fd73b400700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd7200b2910 0x7fd7200b4e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd728005950 tx=0x7fd7280058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:14.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.366+0000 7fd740c00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] conn(0x7fd7200bdbc0 0x7fd7200c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:14.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.367+0000 7fd740c00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] conn(0x7fd7200bdbc0 0x7fd7200c00b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.370+0000 7fd738c00700 1 -- 172.21.15.43:0/562873127 <== osd.11 v2:172.21.15.179:6824/55249493 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd7200c0780 con 0x7fd7200bdbc0 2024-05-27T07:11:14.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.389+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 --> [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd73c064340 con 0x7fd7200bdbc0 2024-05-27T07:11:14.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.390+0000 7fd738c00700 1 -- 172.21.15.43:0/562873127 <== osd.11 v2:172.21.15.179:6824/55249493 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd73c064340 con 0x7fd7200bdbc0 2024-05-27T07:11:14.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.391+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 >> [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] conn(0x7fd7200bdbc0 msgr2=0x7fd7200c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:14.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.391+0000 7fd742c00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] conn(0x7fd7200bdbc0 0x7fd7200c00b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:14.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.391+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd7200b2910 msgr2=0x7fd7200b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:14.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.391+0000 7fd742c00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fd7200b2910 0x7fd7200b4e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd728005950 tx=0x7fd7280058e0 comp rx=0 tx=0).stop 2024-05-27T07:11:14.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.391+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c1045e0 msgr2=0x7fd73c075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:14.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.391+0000 7fd742c00700 1 --2- 172.21.15.43:0/562873127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c1045e0 0x7fd73c075ab0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00d950 tx=0x7fd72c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:11:14.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.391+0000 7fd740c00700 1 -- 172.21.15.43:0/562873127 reap_dead start 2024-05-27T07:11:14.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.392+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 shutdown_connections 2024-05-27T07:11:14.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.392+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 >> 172.21.15.43:0/562873127 conn(0x7fd73c100030 msgr2=0x7fd73c108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:14.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.393+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 shutdown_connections 2024-05-27T07:11:14.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.393+0000 7fd742c00700 1 -- 172.21.15.43:0/562873127 wait complete. 2024-05-27T07:11:16.038 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:15 smithi043 ceph-mon[31531]: pgmap v481: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:11:16.042 INFO:teuthology.orchestra.run.smithi043.stdout:158913790052 2024-05-27T07:11:16.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-27T07:11:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.512+0000 7fbf91e00700 1 -- 172.21.15.43:0/1860550695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf8c1045e0 msgr2=0x7fbf8c1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.512+0000 7fbf91e00700 1 --2- 172.21.15.43:0/1860550695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf8c1045e0 0x7fbf8c1049f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0099c0 tx=0x7fbf7c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.512+0000 7fbf91e00700 1 -- 172.21.15.43:0/1860550695 shutdown_connections 2024-05-27T07:11:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.512+0000 7fbf91e00700 1 --2- 172.21.15.43:0/1860550695 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf8c109860 0x7fbf8c1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.512+0000 7fbf91e00700 1 --2- 172.21.15.43:0/1860550695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf8c1045e0 0x7fbf8c1049f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.512+0000 7fbf91e00700 1 --2- 172.21.15.43:0/1860550695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf8c108f10 0x7fbf8c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.512+0000 7fbf91e00700 1 -- 172.21.15.43:0/1860550695 >> 172.21.15.43:0/1860550695 conn(0x7fbf8c100030 msgr2=0x7fbf8c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.512+0000 7fbf91e00700 1 -- 172.21.15.43:0/1860550695 shutdown_connections 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.513+0000 7fbf91e00700 1 -- 172.21.15.43:0/1860550695 wait complete. 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.514+0000 7fbf91e00700 1 Processor -- start 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.514+0000 7fbf91e00700 1 -- start start 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf91e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf8c108f10 0x7fbf8c075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf91e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf8c109860 0x7fbf8c075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf91e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf8c076530 0x7fbf8c1a9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf91e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c1143b0 con 0x7fbf8c109860 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf91e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c114230 con 0x7fbf8c108f10 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf91e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c114530 con 0x7fbf8c076530 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf8a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf8c109860 0x7fbf8c075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf8a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf8c109860 0x7fbf8c075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34656/0 (socket says 172.21.15.43:34656) 2024-05-27T07:11:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf8a000700 1 -- 172.21.15.43:0/1240475434 learned_addr learned my addr 172.21.15.43:0/1240475434 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf8b400700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf8c076530 0x7fbf8c1a9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.515+0000 7fbf8aa00700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf8c108f10 0x7fbf8c075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.516+0000 7fbf8a000700 1 -- 172.21.15.43:0/1240475434 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf8c076530 msgr2=0x7fbf8c1a9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.516+0000 7fbf8a000700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf8c076530 0x7fbf8c1a9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.516+0000 7fbf8a000700 1 -- 172.21.15.43:0/1240475434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf8c108f10 msgr2=0x7fbf8c075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.516+0000 7fbf8a000700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf8c108f10 0x7fbf8c075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.516+0000 7fbf8a000700 1 -- 172.21.15.43:0/1240475434 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c1a97a0 con 0x7fbf8c109860 2024-05-27T07:11:16.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.516+0000 7fbf8a000700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf8c109860 0x7fbf8c075ff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800d9b0 tx=0x7fbf7800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.517+0000 7fbf90a00700 1 -- 172.21.15.43:0/1240475434 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78017070 con 0x7fbf8c109860 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.517+0000 7fbf90a00700 1 -- 172.21.15.43:0/1240475434 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf78009dc0 con 0x7fbf8c109860 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.517+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8c1a9a90 con 0x7fbf8c109860 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.517+0000 7fbf90a00700 1 -- 172.21.15.43:0/1240475434 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78010e10 con 0x7fbf8c109860 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.517+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8c1a9fd0 con 0x7fbf8c109860 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.518+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbf8c063ec0 con 0x7fbf8c109860 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.522+0000 7fbf90a00700 1 -- 172.21.15.43:0/1240475434 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fbf78009930 con 0x7fbf8c109860 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.524+0000 7fbf90a00700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbf700b2a60 0x7fbf700b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.524+0000 7fbf90a00700 1 -- 172.21.15.43:0/1240475434 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fbf780e26a0 con 0x7fbf8c109860 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.524+0000 7fbf8aa00700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbf700b2a60 0x7fbf700b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.524+0000 7fbf90a00700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] conn(0x7fbf700bdd10 0x7fbf700c0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.524+0000 7fbf90a00700 1 -- 172.21.15.43:0/1240475434 --> [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf78023e50 con 0x7fbf700bdd10 2024-05-27T07:11:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.524+0000 7fbf90a00700 1 -- 172.21.15.43:0/1240475434 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fbf780e2a50 con 0x7fbf8c109860 2024-05-27T07:11:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.525+0000 7fbf8b400700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] conn(0x7fbf700bdd10 0x7fbf700c0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.525+0000 7fbf8aa00700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbf700b2a60 0x7fbf700b4f70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009990 tx=0x7fbf7c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.526+0000 7fbf8b400700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] conn(0x7fbf700bdd10 0x7fbf700c0200 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.528+0000 7fbf90a00700 1 -- 172.21.15.43:0/1240475434 <== osd.2 v2:172.21.15.179:6800/2313942243 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf78023e50 con 0x7fbf700bdd10 2024-05-27T07:11:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.548+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 --> [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbf8c0642c0 con 0x7fbf700bdd10 2024-05-27T07:11:16.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.549+0000 7fbf90a00700 1 -- 172.21.15.43:0/1240475434 <== osd.2 v2:172.21.15.179:6800/2313942243 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbf8c0642c0 con 0x7fbf700bdd10 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 >> [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] conn(0x7fbf700bdd10 msgr2=0x7fbf700c0200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf91e00700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] conn(0x7fbf700bdd10 0x7fbf700c0200 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbf700b2a60 msgr2=0x7fbf700b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf91e00700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbf700b2a60 0x7fbf700b4f70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009990 tx=0x7fbf7c01a040 comp rx=0 tx=0).stop 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf8c109860 msgr2=0x7fbf8c075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf91e00700 1 --2- 172.21.15.43:0/1240475434 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf8c109860 0x7fbf8c075ff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800d9b0 tx=0x7fbf7800dcc0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf8b400700 1 -- 172.21.15.43:0/1240475434 reap_dead start 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 shutdown_connections 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 >> 172.21.15.43:0/1240475434 conn(0x7fbf8c100030 msgr2=0x7fbf8c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 shutdown_connections 2024-05-27T07:11:16.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.550+0000 7fbf91e00700 1 -- 172.21.15.43:0/1240475434 wait complete. 2024-05-27T07:11:16.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:15 smithi179 ceph-mon[35888]: pgmap v481: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:11:16.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:15 smithi132 ceph-mon[35283]: pgmap v481: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-05-27T07:11:16.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.639+0000 7fb66dc00700 1 -- 172.21.15.43:0/2781914521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb668109670 msgr2=0x7fb668110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.639+0000 7fb66dc00700 1 --2- 172.21.15.43:0/2781914521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb668109670 0x7fb668110fd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb66000b0d0 tx=0x7fb66000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.640+0000 7fb66dc00700 1 -- 172.21.15.43:0/2781914521 shutdown_connections 2024-05-27T07:11:16.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.640+0000 7fb66dc00700 1 --2- 172.21.15.43:0/2781914521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb668109670 0x7fb668110fd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.640+0000 7fb66dc00700 1 --2- 172.21.15.43:0/2781914521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb6681043f0 0x7fb668104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.640+0000 7fb66dc00700 1 --2- 172.21.15.43:0/2781914521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb668108d20 0x7fb668109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.640+0000 7fb66dc00700 1 -- 172.21.15.43:0/2781914521 >> 172.21.15.43:0/2781914521 conn(0x7fb668079d00 msgr2=0x7fb66807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.640+0000 7fb66dc00700 1 -- 172.21.15.43:0/2781914521 shutdown_connections 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.640+0000 7fb66dc00700 1 -- 172.21.15.43:0/2781914521 wait complete. 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.641+0000 7fb66dc00700 1 Processor -- start 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.641+0000 7fb66dc00700 1 -- start start 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.641+0000 7fb66dc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6681043f0 0x7fb66819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.641+0000 7fb66dc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb668108d20 0x7fb66819d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.641+0000 7fb66dc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb668109670 0x7fb6681a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb66dc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb668114150 con 0x7fb6681043f0 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb66dc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb668113fd0 con 0x7fb668108d20 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb66dc00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb6681142d0 con 0x7fb668109670 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb667e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6681043f0 0x7fb66819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:16.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb667e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6681043f0 0x7fb66819cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34674/0 (socket says 172.21.15.43:34674) 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb667e00700 1 -- 172.21.15.43:0/1342535267 learned_addr learned my addr 172.21.15.43:0/1342535267 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb667400700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb668108d20 0x7fb66819d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb66d200700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb668109670 0x7fb6681a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb667e00700 1 -- 172.21.15.43:0/1342535267 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb668109670 msgr2=0x7fb6681a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb667e00700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb668109670 0x7fb6681a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb667e00700 1 -- 172.21.15.43:0/1342535267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb668108d20 msgr2=0x7fb66819d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb667e00700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb668108d20 0x7fb66819d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb667e00700 1 -- 172.21.15.43:0/1342535267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6681a1bf0 con 0x7fb6681043f0 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.642+0000 7fb66d200700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb668109670 0x7fb6681a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.643+0000 7fb667e00700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6681043f0 0x7fb66819cd20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb65800d950 tx=0x7fb65800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.643+0000 7fb664c00700 1 -- 172.21.15.43:0/1342535267 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658016070 con 0x7fb6681043f0 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.643+0000 7fb664c00700 1 -- 172.21.15.43:0/1342535267 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb658010460 con 0x7fb6681043f0 2024-05-27T07:11:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.643+0000 7fb664c00700 1 -- 172.21.15.43:0/1342535267 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65800f620 con 0x7fb6681043f0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.643+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6681a1ee0 con 0x7fb6681043f0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.644+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6681a9800 con 0x7fb6681043f0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.644+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb66810fdf0 con 0x7fb6681043f0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.647+0000 7fb664c00700 1 -- 172.21.15.43:0/1342535267 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fb658019070 con 0x7fb6681043f0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.649+0000 7fb664c00700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb64c0b2910 0x7fb64c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.649+0000 7fb664c00700 1 -- 172.21.15.43:0/1342535267 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fb6580e2870 con 0x7fb6681043f0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.649+0000 7fb667400700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb64c0b2910 0x7fb64c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.649+0000 7fb664c00700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] conn(0x7fb64c0bdbc0 0x7fb64c0c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.649+0000 7fb664c00700 1 -- 172.21.15.43:0/1342535267 --> [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb64c0c0780 con 0x7fb64c0bdbc0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.649+0000 7fb664c00700 1 -- 172.21.15.43:0/1342535267 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb6580e2cc0 con 0x7fb6681043f0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.649+0000 7fb66d200700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] conn(0x7fb64c0bdbc0 0x7fb64c0c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.650+0000 7fb667400700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb64c0b2910 0x7fb64c0b4e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb6681111f0 tx=0x7fb654009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:16.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.650+0000 7fb66d200700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] conn(0x7fb64c0bdbc0 0x7fb64c0c00b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:16.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.651+0000 7fb664c00700 1 -- 172.21.15.43:0/1342535267 <== osd.9 v2:172.21.15.43:6826/3342932073 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb64c0c0780 con 0x7fb64c0bdbc0 2024-05-27T07:11:16.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.670+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 --> [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb6681058a0 con 0x7fb64c0bdbc0 2024-05-27T07:11:16.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.670+0000 7fb664c00700 1 -- 172.21.15.43:0/1342535267 <== osd.9 v2:172.21.15.43:6826/3342932073 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb6681058a0 con 0x7fb64c0bdbc0 2024-05-27T07:11:16.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] conn(0x7fb64c0bdbc0 msgr2=0x7fb64c0c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66dc00700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] conn(0x7fb64c0bdbc0 0x7fb64c0c00b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:16.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb64c0b2910 msgr2=0x7fb64c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66dc00700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb64c0b2910 0x7fb64c0b4e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb6681111f0 tx=0x7fb654009260 comp rx=0 tx=0).stop 2024-05-27T07:11:16.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6681043f0 msgr2=0x7fb66819cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:16.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66dc00700 1 --2- 172.21.15.43:0/1342535267 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb6681043f0 0x7fb66819cd20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb65800d950 tx=0x7fb65800dd10 comp rx=0 tx=0).stop 2024-05-27T07:11:16.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66d200700 1 -- 172.21.15.43:0/1342535267 reap_dead start 2024-05-27T07:11:16.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 shutdown_connections 2024-05-27T07:11:16.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 >> 172.21.15.43:0/1342535267 conn(0x7fb668079d00 msgr2=0x7fb66810f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:16.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 shutdown_connections 2024-05-27T07:11:16.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.671+0000 7fb66dc00700 1 -- 172.21.15.43:0/1342535267 wait complete. 2024-05-27T07:11:17.455 INFO:teuthology.orchestra.run.smithi043.stdout:77309411451 2024-05-27T07:11:17.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-27T07:11:17.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.632+0000 7fe08ea00700 1 -- 172.21.15.43:0/1899092645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe088106590 msgr2=0x7fe0881069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:17.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.632+0000 7fe08ea00700 1 --2- 172.21.15.43:0/1899092645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe088106590 0x7fe0881069a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe074009990 tx=0x7fe074009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:17.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.632+0000 7fe08ea00700 1 -- 172.21.15.43:0/1899092645 shutdown_connections 2024-05-27T07:11:17.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.632+0000 7fe08ea00700 1 --2- 172.21.15.43:0/1899092645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe088106ee0 0x7fe088113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:17.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.632+0000 7fe08ea00700 1 --2- 172.21.15.43:0/1899092645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe08810c5d0 0x7fe08810c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:17.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.632+0000 7fe08ea00700 1 --2- 172.21.15.43:0/1899092645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe088106590 0x7fe0881069a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:17.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.632+0000 7fe08ea00700 1 -- 172.21.15.43:0/1899092645 >> 172.21.15.43:0/1899092645 conn(0x7fe088102070 msgr2=0x7fe0881044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:17.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.632+0000 7fe08ea00700 1 -- 172.21.15.43:0/1899092645 shutdown_connections 2024-05-27T07:11:17.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.633+0000 7fe08ea00700 1 -- 172.21.15.43:0/1899092645 wait complete. 2024-05-27T07:11:17.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.634+0000 7fe08ea00700 1 Processor -- start 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.634+0000 7fe08ea00700 1 -- start start 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.635+0000 7fe08ea00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe088106590 0x7fe08819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.635+0000 7fe08ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe088106ee0 0x7fe08819d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.635+0000 7fe08ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe08810c5d0 0x7fe0881a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.635+0000 7fe08ea00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe0881165b0 con 0x7fe08810c5d0 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.635+0000 7fe08ea00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe088116430 con 0x7fe088106ee0 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.635+0000 7fe08ea00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe088116730 con 0x7fe088106590 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.635+0000 7fe08ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe08810c5d0 0x7fe0881a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.635+0000 7fe08ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe08810c5d0 0x7fe0881a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34610/0 (socket says 172.21.15.43:34610) 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.635+0000 7fe08ca00700 1 -- 172.21.15.43:0/697864217 learned_addr learned my addr 172.21.15.43:0/697864217 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.635+0000 7fe087e00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe088106590 0x7fe08819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:17.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe087400700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe088106ee0 0x7fe08819d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:17.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe08ca00700 1 -- 172.21.15.43:0/697864217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe088106590 msgr2=0x7fe08819cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:17.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe08ca00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe088106590 0x7fe08819cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:17.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe08ca00700 1 -- 172.21.15.43:0/697864217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe088106ee0 msgr2=0x7fe08819d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:17.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe08ca00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe088106ee0 0x7fe08819d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:17.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe08ca00700 1 -- 172.21.15.43:0/697864217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0881a1e70 con 0x7fe08810c5d0 2024-05-27T07:11:17.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe087e00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe088106590 0x7fe08819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe08ca00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe08810c5d0 0x7fe0881a17d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe08000ef30 tx=0x7fe08000c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe084c00700 1 -- 172.21.15.43:0/697864217 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0800092e0 con 0x7fe08810c5d0 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe084c00700 1 -- 172.21.15.43:0/697864217 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe080004bb0 con 0x7fe08810c5d0 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.636+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0881a2100 con 0x7fe08810c5d0 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.637+0000 7fe084c00700 1 -- 172.21.15.43:0/697864217 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe080004080 con 0x7fe08810c5d0 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.637+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0881a2610 con 0x7fe08810c5d0 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.638+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe088063f40 con 0x7fe08810c5d0 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.641+0000 7fe084c00700 1 -- 172.21.15.43:0/697864217 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fe08001b030 con 0x7fe08810c5d0 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.642+0000 7fe084c00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe06c0b2990 0x7fe06c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.642+0000 7fe084c00700 1 -- 172.21.15.43:0/697864217 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fe0800e2c50 con 0x7fe08810c5d0 2024-05-27T07:11:17.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.643+0000 7fe087e00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe06c0b2990 0x7fe06c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:17.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.643+0000 7fe084c00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] conn(0x7fe06c0bdc40 0x7fe06c0c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:17.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.643+0000 7fe084c00700 1 -- 172.21.15.43:0/697864217 --> [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe080008ec0 con 0x7fe06c0bdc40 2024-05-27T07:11:17.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.643+0000 7fe084c00700 1 -- 172.21.15.43:0/697864217 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe0800e30a0 con 0x7fe08810c5d0 2024-05-27T07:11:17.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.643+0000 7fe087400700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] conn(0x7fe06c0bdc40 0x7fe06c0c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:17.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.644+0000 7fe087e00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe06c0b2990 0x7fe06c0b4ea0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe088107a00 tx=0x7fe078009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:17.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.644+0000 7fe087400700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] conn(0x7fe06c0bdc40 0x7fe06c0c0130 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:17.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.645+0000 7fe084c00700 1 -- 172.21.15.43:0/697864217 <== osd.3 v2:172.21.15.43:6810/902648995 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe080008ec0 con 0x7fe06c0bdc40 2024-05-27T07:11:17.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.664+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 --> [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe0880643c0 con 0x7fe06c0bdc40 2024-05-27T07:11:17.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.665+0000 7fe084c00700 1 -- 172.21.15.43:0/697864217 <== osd.3 v2:172.21.15.43:6810/902648995 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe0880643c0 con 0x7fe06c0bdc40 2024-05-27T07:11:17.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] conn(0x7fe06c0bdc40 msgr2=0x7fe06c0c0130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:17.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ea00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] conn(0x7fe06c0bdc40 0x7fe06c0c0130 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:17.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe06c0b2990 msgr2=0x7fe06c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:17.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ea00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe06c0b2990 0x7fe06c0b4ea0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe088107a00 tx=0x7fe078009260 comp rx=0 tx=0).stop 2024-05-27T07:11:17.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe08810c5d0 msgr2=0x7fe0881a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:17.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ea00700 1 --2- 172.21.15.43:0/697864217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe08810c5d0 0x7fe0881a17d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe08000ef30 tx=0x7fe08000c450 comp rx=0 tx=0).stop 2024-05-27T07:11:17.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ca00700 1 -- 172.21.15.43:0/697864217 reap_dead start 2024-05-27T07:11:17.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 shutdown_connections 2024-05-27T07:11:17.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 >> 172.21.15.43:0/697864217 conn(0x7fe088102070 msgr2=0x7fe0881038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:17.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 shutdown_connections 2024-05-27T07:11:17.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:14.666+0000 7fe08ea00700 1 -- 172.21.15.43:0/697864217 wait complete. 2024-05-27T07:11:18.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.391+0000 7f34bae00700 1 -- 172.21.15.43:0/521494229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b4110b90 msgr2=0x7f34b4112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.391+0000 7f34bae00700 1 --2- 172.21.15.43:0/521494229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b4110b90 0x7f34b4112ff0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00b0a0 tx=0x7f34ac00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.391+0000 7f34bae00700 1 -- 172.21.15.43:0/521494229 shutdown_connections 2024-05-27T07:11:18.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.391+0000 7f34bae00700 1 --2- 172.21.15.43:0/521494229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b4110b90 0x7f34b4112ff0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.391+0000 7f34bae00700 1 --2- 172.21.15.43:0/521494229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34b4075d20 0x7f34b4076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.391+0000 7f34bae00700 1 --2- 172.21.15.43:0/521494229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34b4076670 0x7f34b4110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.391+0000 7f34bae00700 1 -- 172.21.15.43:0/521494229 >> 172.21.15.43:0/521494229 conn(0x7f34b4100030 msgr2=0x7f34b4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:18.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.391+0000 7f34bae00700 1 -- 172.21.15.43:0/521494229 shutdown_connections 2024-05-27T07:11:18.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.391+0000 7f34bae00700 1 -- 172.21.15.43:0/521494229 wait complete. 2024-05-27T07:11:18.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.392+0000 7f34bae00700 1 Processor -- start 2024-05-27T07:11:18.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.393+0000 7f34bae00700 1 -- start start 2024-05-27T07:11:18.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.393+0000 7f34bae00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34b4075d20 0x7f34b419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:18.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.393+0000 7f34bae00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34b4076670 0x7f34b419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:18.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.393+0000 7f34bae00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b4110b90 0x7f34b41a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:18.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.393+0000 7f34bae00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f34b4116520 con 0x7f34b4110b90 2024-05-27T07:11:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.393+0000 7f34bae00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f34b41163a0 con 0x7f34b4075d20 2024-05-27T07:11:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.393+0000 7f34bae00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f34b41166a0 con 0x7f34b4076670 2024-05-27T07:11:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.394+0000 7f34b8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b4110b90 0x7f34b41a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.394+0000 7f34b8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b4110b90 0x7f34b41a16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34630/0 (socket says 172.21.15.43:34630) 2024-05-27T07:11:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.394+0000 7f34b3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34b4075d20 0x7f34b419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:18.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.394+0000 7f34b8e00700 1 -- 172.21.15.43:0/100680386 learned_addr learned my addr 172.21.15.43:0/100680386 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:18.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.394+0000 7f34b3400700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34b4076670 0x7f34b419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:18.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.394+0000 7f34b8e00700 1 -- 172.21.15.43:0/100680386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34b4076670 msgr2=0x7f34b419d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.394+0000 7f34b8e00700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34b4076670 0x7f34b419d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.394+0000 7f34b8e00700 1 -- 172.21.15.43:0/100680386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34b4075d20 msgr2=0x7f34b419ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.394+0000 7f34b8e00700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34b4075d20 0x7f34b419ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.394+0000 7f34b8e00700 1 -- 172.21.15.43:0/100680386 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b41a1d80 con 0x7f34b4110b90 2024-05-27T07:11:18.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.395+0000 7f34b8e00700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b4110b90 0x7f34b41a16e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00b830 tx=0x7f34ac004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:18.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.395+0000 7f34b0c00700 1 -- 172.21.15.43:0/100680386 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac021070 con 0x7f34b4110b90 2024-05-27T07:11:18.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.395+0000 7f34b0c00700 1 -- 172.21.15.43:0/100680386 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34ac007b80 con 0x7f34b4110b90 2024-05-27T07:11:18.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.395+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b41a2010 con 0x7f34b4110b90 2024-05-27T07:11:18.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.395+0000 7f34b0c00700 1 -- 172.21.15.43:0/100680386 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac0052a0 con 0x7f34b4110b90 2024-05-27T07:11:18.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.395+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b41a9990 con 0x7f34b4110b90 2024-05-27T07:11:18.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.396+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f34b4063f40 con 0x7f34b4110b90 2024-05-27T07:11:18.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.401+0000 7f34b0c00700 1 -- 172.21.15.43:0/100680386 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f34ac024070 con 0x7f34b4110b90 2024-05-27T07:11:18.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.403+0000 7f34b0c00700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34980b2990 0x7f34980b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:18.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.403+0000 7f34b0c00700 1 -- 172.21.15.43:0/100680386 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f34ac00f020 con 0x7f34b4110b90 2024-05-27T07:11:18.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.403+0000 7f34b3e00700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34980b2990 0x7f34980b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:18.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.403+0000 7f34b0c00700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] conn(0x7f34980bdc40 0x7f34980c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:18.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.403+0000 7f34b0c00700 1 -- 172.21.15.43:0/100680386 --> [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34ac031e50 con 0x7f34980bdc40 2024-05-27T07:11:18.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.403+0000 7f34b0c00700 1 -- 172.21.15.43:0/100680386 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f34ac0e4480 con 0x7f34b4110b90 2024-05-27T07:11:18.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.404+0000 7f34b3e00700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34980b2990 0x7f34980b4ea0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34a4005950 tx=0x7f34a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:18.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.415+0000 7f34b3400700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] conn(0x7f34980bdc40 0x7f34980c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:18.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.416+0000 7f34b3400700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] conn(0x7f34980bdc40 0x7f34980c0130 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:18.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.418+0000 7f34b0c00700 1 -- 172.21.15.43:0/100680386 <== osd.4 v2:172.21.15.132:6808/4132674108 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f34ac031e50 con 0x7f34980bdc40 2024-05-27T07:11:18.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.437+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 --> [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f34b4064340 con 0x7f34980bdc40 2024-05-27T07:11:18.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.438+0000 7f34b0c00700 1 -- 172.21.15.43:0/100680386 <== osd.4 v2:172.21.15.132:6808/4132674108 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f34b4064340 con 0x7f34980bdc40 2024-05-27T07:11:18.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 >> [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] conn(0x7f34980bdc40 msgr2=0x7f34980c0130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34bae00700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] conn(0x7f34980bdc40 0x7f34980c0130 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34980b2990 msgr2=0x7f34980b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34bae00700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f34980b2990 0x7f34980b4ea0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34a4005950 tx=0x7f34a400b410 comp rx=0 tx=0).stop 2024-05-27T07:11:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b4110b90 msgr2=0x7f34b41a16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34bae00700 1 --2- 172.21.15.43:0/100680386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34b4110b90 0x7f34b41a16e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00b830 tx=0x7f34ac004d00 comp rx=0 tx=0).stop 2024-05-27T07:11:18.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34b8e00700 1 -- 172.21.15.43:0/100680386 reap_dead start 2024-05-27T07:11:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 shutdown_connections 2024-05-27T07:11:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 >> 172.21.15.43:0/100680386 conn(0x7f34b4100030 msgr2=0x7f34b4100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 shutdown_connections 2024-05-27T07:11:18.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:15.439+0000 7f34bae00700 1 -- 172.21.15.43:0/100680386 wait complete. 2024-05-27T07:11:18.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:17 smithi043 ceph-mon[31531]: pgmap v482: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-27T07:11:18.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:17 smithi179 ceph-mon[35888]: pgmap v482: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-27T07:11:18.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:17 smithi132 ceph-mon[35283]: pgmap v482: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-27T07:11:18.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.779+0000 7ff4be400700 1 -- 172.21.15.43:0/216570148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4b810c5b0 msgr2=0x7ff4b810c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.779+0000 7ff4be400700 1 --2- 172.21.15.43:0/216570148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4b810c5b0 0x7ff4b810c9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009990 tx=0x7ff4a8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.779+0000 7ff4be400700 1 -- 172.21.15.43:0/216570148 shutdown_connections 2024-05-27T07:11:18.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.779+0000 7ff4be400700 1 --2- 172.21.15.43:0/216570148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4b8106ec0 0x7ff4b8113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.779+0000 7ff4be400700 1 --2- 172.21.15.43:0/216570148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4b810c5b0 0x7ff4b810c9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.779+0000 7ff4be400700 1 --2- 172.21.15.43:0/216570148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4b8106570 0x7ff4b8106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.779+0000 7ff4be400700 1 -- 172.21.15.43:0/216570148 >> 172.21.15.43:0/216570148 conn(0x7ff4b8102070 msgr2=0x7ff4b81044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:18.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.779+0000 7ff4be400700 1 -- 172.21.15.43:0/216570148 shutdown_connections 2024-05-27T07:11:18.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.779+0000 7ff4be400700 1 -- 172.21.15.43:0/216570148 wait complete. 2024-05-27T07:11:18.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.780+0000 7ff4be400700 1 Processor -- start 2024-05-27T07:11:18.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.780+0000 7ff4be400700 1 -- start start 2024-05-27T07:11:18.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.781+0000 7ff4be400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4b8106570 0x7ff4b8078dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:18.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.781+0000 7ff4be400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4b8106ec0 0x7ff4b8079310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:18.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.781+0000 7ff4be400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4b810c5b0 0x7ff4b8079850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:18.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.781+0000 7ff4be400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b81165c0 con 0x7ff4b8106570 2024-05-27T07:11:18.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.781+0000 7ff4be400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b8116440 con 0x7ff4b810c5b0 2024-05-27T07:11:18.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.781+0000 7ff4be400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b8116740 con 0x7ff4b8106ec0 2024-05-27T07:11:18.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.781+0000 7ff4b7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4b8106570 0x7ff4b8078dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:18.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.781+0000 7ff4b7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4b8106570 0x7ff4b8078dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34692/0 (socket says 172.21.15.43:34692) 2024-05-27T07:11:18.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.781+0000 7ff4b7400700 1 -- 172.21.15.43:0/1700954029 learned_addr learned my addr 172.21.15.43:0/1700954029 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:18.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.782+0000 7ff4b7e00700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4b810c5b0 0x7ff4b8079850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.782+0000 7ff4b6a00700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4b8106ec0 0x7ff4b8079310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.782+0000 7ff4b7400700 1 -- 172.21.15.43:0/1700954029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4b8106ec0 msgr2=0x7ff4b8079310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.782+0000 7ff4b7400700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4b8106ec0 0x7ff4b8079310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.782+0000 7ff4b7400700 1 -- 172.21.15.43:0/1700954029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4b810c5b0 msgr2=0x7ff4b8079850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.782+0000 7ff4b7400700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4b810c5b0 0x7ff4b8079850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.782+0000 7ff4b7400700 1 -- 172.21.15.43:0/1700954029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b81ad550 con 0x7ff4b8106570 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.782+0000 7ff4b6a00700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4b8106ec0 0x7ff4b8079310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.782+0000 7ff4b7400700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4b8106570 0x7ff4b8078dd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8000c00 tx=0x7ff4a8005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.783+0000 7ff4afe00700 1 -- 172.21.15.43:0/1700954029 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8027070 con 0x7ff4b8106570 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.783+0000 7ff4afe00700 1 -- 172.21.15.43:0/1700954029 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4a800fb80 con 0x7ff4b8106570 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.783+0000 7ff4afe00700 1 -- 172.21.15.43:0/1700954029 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8022c00 con 0x7ff4b8106570 2024-05-27T07:11:18.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.783+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b81ad780 con 0x7ff4b8106570 2024-05-27T07:11:18.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.783+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b81adbd0 con 0x7ff4b8106570 2024-05-27T07:11:18.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.784+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff4b8063f00 con 0x7ff4b8106570 2024-05-27T07:11:18.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.787+0000 7ff4afe00700 1 -- 172.21.15.43:0/1700954029 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff4a802a070 con 0x7ff4b8106570 2024-05-27T07:11:18.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.788+0000 7ff4afe00700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff49c0b2910 0x7ff49c0b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:18.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.789+0000 7ff4afe00700 1 -- 172.21.15.43:0/1700954029 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7ff4a80e3b50 con 0x7ff4b8106570 2024-05-27T07:11:18.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.789+0000 7ff4b6a00700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff49c0b2910 0x7ff49c0b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:18.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.789+0000 7ff4afe00700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] conn(0x7ff49c0bdbc0 0x7ff49c0c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:18.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.789+0000 7ff4afe00700 1 -- 172.21.15.43:0/1700954029 --> [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4a800b0b0 con 0x7ff49c0bdbc0 2024-05-27T07:11:18.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.789+0000 7ff4afe00700 1 -- 172.21.15.43:0/1700954029 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7ff4a80e3f00 con 0x7ff4b8106570 2024-05-27T07:11:18.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.789+0000 7ff4b7e00700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] conn(0x7ff49c0bdbc0 0x7ff49c0c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:18.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.790+0000 7ff4b6a00700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff49c0b2910 0x7ff49c0b4e20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4005fd0 tx=0x7ff4a4005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:18.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.790+0000 7ff4b7e00700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] conn(0x7ff49c0bdbc0 0x7ff49c0c00b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:18.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.791+0000 7ff4afe00700 1 -- 172.21.15.43:0/1700954029 <== osd.6 v2:172.21.15.43:6818/2712807127 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4a800b0b0 con 0x7ff49c0bdbc0 2024-05-27T07:11:18.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.812+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 --> [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff4b8064300 con 0x7ff49c0bdbc0 2024-05-27T07:11:18.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.812+0000 7ff4afe00700 1 -- 172.21.15.43:0/1700954029 <== osd.6 v2:172.21.15.43:6818/2712807127 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff4b8064300 con 0x7ff49c0bdbc0 2024-05-27T07:11:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.813+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] conn(0x7ff49c0bdbc0 msgr2=0x7ff49c0c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.813+0000 7ff4be400700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] conn(0x7ff49c0bdbc0 0x7ff49c0c00b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.813+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff49c0b2910 msgr2=0x7ff49c0b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.813+0000 7ff4be400700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff49c0b2910 0x7ff49c0b4e20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4005fd0 tx=0x7ff4a4005ee0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.813+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4b8106570 msgr2=0x7ff4b8078dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.813+0000 7ff4be400700 1 --2- 172.21.15.43:0/1700954029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4b8106570 0x7ff4b8078dd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8000c00 tx=0x7ff4a8005ef0 comp rx=0 tx=0).stop 2024-05-27T07:11:18.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.813+0000 7ff4b7e00700 1 -- 172.21.15.43:0/1700954029 reap_dead start 2024-05-27T07:11:18.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.813+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 shutdown_connections 2024-05-27T07:11:18.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.813+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 >> 172.21.15.43:0/1700954029 conn(0x7ff4b8102070 msgr2=0x7ff4b8104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:18.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.813+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 shutdown_connections 2024-05-27T07:11:18.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:18.814+0000 7ff4be400700 1 -- 172.21.15.43:0/1700954029 wait complete. 2024-05-27T07:11:19.639 INFO:teuthology.orchestra.run.smithi043.stdout:111669149811 2024-05-27T07:11:19.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-27T07:11:19.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:19 smithi043 ceph-mon[31531]: pgmap v483: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:20.045 INFO:teuthology.orchestra.run.smithi043.stdout:55834574978 2024-05-27T07:11:20.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-27T07:11:20.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:19 smithi179 ceph-mon[35888]: pgmap v483: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:20.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:19 smithi132 ceph-mon[35283]: pgmap v483: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:20.910 INFO:teuthology.orchestra.run.smithi043.stdout:137438953579 2024-05-27T07:11:20.910 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-27T07:11:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.469+0000 7f0961400700 1 -- 172.21.15.43:0/4076383793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f095c109860 msgr2=0x7f095c1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.469+0000 7f0961400700 1 --2- 172.21.15.43:0/4076383793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f095c109860 0x7f095c1111c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f095400b0d0 tx=0x7f095400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.469+0000 7f0961400700 1 -- 172.21.15.43:0/4076383793 shutdown_connections 2024-05-27T07:11:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.469+0000 7f0961400700 1 --2- 172.21.15.43:0/4076383793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f095c109860 0x7f095c1111c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.469+0000 7f0961400700 1 --2- 172.21.15.43:0/4076383793 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f095c1045e0 0x7f095c1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.469+0000 7f0961400700 1 --2- 172.21.15.43:0/4076383793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f095c108f10 0x7f095c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.469+0000 7f0961400700 1 -- 172.21.15.43:0/4076383793 >> 172.21.15.43:0/4076383793 conn(0x7f095c100030 msgr2=0x7f095c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.469+0000 7f0961400700 1 -- 172.21.15.43:0/4076383793 shutdown_connections 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.469+0000 7f0961400700 1 -- 172.21.15.43:0/4076383793 wait complete. 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.470+0000 7f0961400700 1 Processor -- start 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.470+0000 7f0961400700 1 -- start start 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.471+0000 7f0961400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f095c1045e0 0x7f095c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.471+0000 7f0961400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f095c108f10 0x7f095c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.471+0000 7f0961400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f095c109860 0x7f095c1a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.471+0000 7f0961400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f095c114390 con 0x7f095c108f10 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.471+0000 7f0961400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f095c114210 con 0x7f095c109860 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.471+0000 7f0961400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f095c114510 con 0x7f095c1045e0 2024-05-27T07:11:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.471+0000 7f0959e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f095c108f10 0x7f095c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.471+0000 7f0959e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f095c108f10 0x7f095c19d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34716/0 (socket says 172.21.15.43:34716) 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.471+0000 7f0959e00700 1 -- 172.21.15.43:0/1645427692 learned_addr learned my addr 172.21.15.43:0/1645427692 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.472+0000 7f095a800700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f095c1045e0 0x7f095c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.472+0000 7f095b200700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f095c109860 0x7f095c1a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.472+0000 7f0959e00700 1 -- 172.21.15.43:0/1645427692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f095c1045e0 msgr2=0x7f095c19cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.472+0000 7f0959e00700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f095c1045e0 0x7f095c19cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.472+0000 7f0959e00700 1 -- 172.21.15.43:0/1645427692 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f095c109860 msgr2=0x7f095c1a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.472+0000 7f0959e00700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f095c109860 0x7f095c1a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.472+0000 7f0959e00700 1 -- 172.21.15.43:0/1645427692 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095c1a1df0 con 0x7f095c108f10 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.472+0000 7f095b200700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f095c109860 0x7f095c1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.472+0000 7f0959e00700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f095c108f10 0x7f095c19d4c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f094800d980 tx=0x7f094800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.473+0000 7f0953400700 1 -- 172.21.15.43:0/1645427692 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0948017070 con 0x7f095c108f10 2024-05-27T07:11:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.473+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f095c1a20e0 con 0x7f095c108f10 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.473+0000 7f0953400700 1 -- 172.21.15.43:0/1645427692 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0948010460 con 0x7f095c108f10 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.473+0000 7f0953400700 1 -- 172.21.15.43:0/1645427692 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094800de60 con 0x7f095c108f10 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.473+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f095c1a9a00 con 0x7f095c108f10 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.474+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f095c076d40 con 0x7f095c108f10 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.479+0000 7f0953400700 1 -- 172.21.15.43:0/1645427692 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0948010610 con 0x7f095c108f10 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.480+0000 7f0953400700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f09400b2990 0x7f09400b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.480+0000 7f0953400700 1 -- 172.21.15.43:0/1645427692 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f09480e2610 con 0x7f095c108f10 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.481+0000 7f095a800700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f09400b2990 0x7f09400b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.481+0000 7f0953400700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] conn(0x7f09400bdc40 0x7f09400c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.481+0000 7f0953400700 1 -- 172.21.15.43:0/1645427692 --> [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0948016ec0 con 0x7f09400bdc40 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.481+0000 7f0953400700 1 -- 172.21.15.43:0/1645427692 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f09480e29c0 con 0x7f095c108f10 2024-05-27T07:11:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.481+0000 7f095a800700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f09400b2990 0x7f09400b4ea0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f094c005950 tx=0x7f094c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:21.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.481+0000 7f095b200700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] conn(0x7f09400bdc40 0x7f09400c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:21.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.483+0000 7f095b200700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] conn(0x7f09400bdc40 0x7f09400c0130 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:21.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.485+0000 7f0953400700 1 -- 172.21.15.43:0/1645427692 <== osd.7 v2:172.21.15.132:6816/2427045470 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0948016ec0 con 0x7f09400bdc40 2024-05-27T07:11:21.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.504+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 --> [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f095c105a90 con 0x7f09400bdc40 2024-05-27T07:11:21.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.505+0000 7f0953400700 1 -- 172.21.15.43:0/1645427692 <== osd.7 v2:172.21.15.132:6816/2427045470 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f095c105a90 con 0x7f09400bdc40 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 >> [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] conn(0x7f09400bdc40 msgr2=0x7f09400c0130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f0961400700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] conn(0x7f09400bdc40 0x7f09400c0130 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f09400b2990 msgr2=0x7f09400b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f0961400700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f09400b2990 0x7f09400b4ea0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f094c005950 tx=0x7f094c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f095c108f10 msgr2=0x7f095c19d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f0961400700 1 --2- 172.21.15.43:0/1645427692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f095c108f10 0x7f095c19d4c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f094800d980 tx=0x7f094800dd40 comp rx=0 tx=0).stop 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f095b200700 1 -- 172.21.15.43:0/1645427692 reap_dead start 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 shutdown_connections 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 >> 172.21.15.43:0/1645427692 conn(0x7f095c100030 msgr2=0x7f095c110e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 shutdown_connections 2024-05-27T07:11:21.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:20.506+0000 7f0961400700 1 -- 172.21.15.43:0/1645427692 wait complete. 2024-05-27T07:11:21.666 INFO:teuthology.orchestra.run.smithi043.stdout:279172874302 2024-05-27T07:11:21.666 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-27T07:11:22.060 INFO:teuthology.orchestra.run.smithi043.stdout:257698037829 2024-05-27T07:11:22.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-27T07:11:22.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:21 smithi043 ceph-mon[31531]: pgmap v484: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:22.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:21 smithi179 ceph-mon[35888]: pgmap v484: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:22.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:21 smithi132 ceph-mon[35283]: pgmap v484: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:22.855 INFO:teuthology.orchestra.run.smithi043.stdout:304942678072 2024-05-27T07:11:22.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-27T07:11:23.183 INFO:teuthology.orchestra.run.smithi043.stdout:184683593823 2024-05-27T07:11:23.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-27T07:11:23.579 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:23.585 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:23.631 INFO:teuthology.orchestra.run.smithi043.stdout:206158430294 2024-05-27T07:11:23.631 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-27T07:11:24.174 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:24.178 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:24.182 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:24.185 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:23 smithi043 ceph-mon[31531]: pgmap v485: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:11:24.189 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:24.198 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:24.205 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:24.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:23 smithi179 ceph-mon[35888]: pgmap v485: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:11:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:23 smithi132 ceph-mon[35283]: pgmap v485: 129 pgs: 129 active+clean; 646 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-05-27T07:11:25.181 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:24 smithi043 ceph-mon[31531]: mgrmap e18: x(active, since 13m), standbys: z, y 2024-05-27T07:11:25.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:24 smithi132 ceph-mon[35283]: mgrmap e18: x(active, since 13m), standbys: z, y 2024-05-27T07:11:25.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:24 smithi179 ceph-mon[35888]: mgrmap e18: x(active, since 13m), standbys: z, y 2024-05-27T07:11:25.945 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:26.216 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:25 smithi043 ceph-mon[31531]: pgmap v486: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-27T07:11:26.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:25 smithi132 ceph-mon[35283]: pgmap v486: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-27T07:11:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:25 smithi179 ceph-mon[35888]: pgmap v486: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-05-27T07:11:28.259 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:27 smithi043 ceph-mon[31531]: pgmap v487: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:11:28.259 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:27 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:28.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:27 smithi132 ceph-mon[35283]: pgmap v487: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:11:28.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:27 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:27 smithi179 ceph-mon[35888]: pgmap v487: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-05-27T07:11:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:27 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:11:28.910 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:28.912 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:29.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.368+0000 7f7248600700 1 -- 172.21.15.43:0/277906740 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7240075d20 msgr2=0x7f7240076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:29.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.368+0000 7f7248600700 1 --2- 172.21.15.43:0/277906740 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7240075d20 0x7f7240076130 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7234009990 tx=0x7f7234009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.369+0000 7f7248600700 1 -- 172.21.15.43:0/277906740 shutdown_connections 2024-05-27T07:11:29.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.369+0000 7f7248600700 1 --2- 172.21.15.43:0/277906740 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7240110b90 0x7f7240112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.369+0000 7f7248600700 1 --2- 172.21.15.43:0/277906740 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7240075d20 0x7f7240076130 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.369+0000 7f7248600700 1 --2- 172.21.15.43:0/277906740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7240076670 0x7f7240110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.369+0000 7f7248600700 1 -- 172.21.15.43:0/277906740 >> 172.21.15.43:0/277906740 conn(0x7f7240100030 msgr2=0x7f7240102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:29.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.369+0000 7f7248600700 1 -- 172.21.15.43:0/277906740 shutdown_connections 2024-05-27T07:11:29.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.369+0000 7f7248600700 1 -- 172.21.15.43:0/277906740 wait complete. 2024-05-27T07:11:29.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.371+0000 7f7248600700 1 Processor -- start 2024-05-27T07:11:29.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.371+0000 7f7248600700 1 -- start start 2024-05-27T07:11:29.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.371+0000 7f7248600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7240075d20 0x7f724019d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:29.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.371+0000 7f7248600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7240076670 0x7f724019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:29.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.371+0000 7f7248600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7240110b90 0x7f72401a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:29.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.371+0000 7f7248600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7240116680 con 0x7f7240075d20 2024-05-27T07:11:29.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.371+0000 7f7248600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7240116500 con 0x7f7240110b90 2024-05-27T07:11:29.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.371+0000 7f7248600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7240116800 con 0x7f7240076670 2024-05-27T07:11:29.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7245c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7240075d20 0x7f724019d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7245c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7240075d20 0x7f724019d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42146/0 (socket says 172.21.15.43:42146) 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7245c00700 1 -- 172.21.15.43:0/1087083683 learned_addr learned my addr 172.21.15.43:0/1087083683 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7245200700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7240076670 0x7f724019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7246600700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7240110b90 0x7f72401a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7245c00700 1 -- 172.21.15.43:0/1087083683 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7240076670 msgr2=0x7f724019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7245c00700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7240076670 0x7f724019d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7245c00700 1 -- 172.21.15.43:0/1087083683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7240110b90 msgr2=0x7f72401a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7245c00700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7240110b90 0x7f72401a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7245c00700 1 -- 172.21.15.43:0/1087083683 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72401a1f80 con 0x7f7240075d20 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.372+0000 7f7246600700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7240110b90 0x7f72401a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:11:29.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.373+0000 7f7245c00700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7240075d20 0x7f724019d0b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f72340057a0 tx=0x7f72340036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.373+0000 7f723e000700 1 -- 172.21.15.43:0/1087083683 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234027070 con 0x7f7240075d20 2024-05-27T07:11:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.373+0000 7f723e000700 1 -- 172.21.15.43:0/1087083683 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f723400fc80 con 0x7f7240075d20 2024-05-27T07:11:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.373+0000 7f7248600700 1 -- 172.21.15.43:0/1087083683 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72401a2210 con 0x7f7240075d20 2024-05-27T07:11:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.373+0000 7f723e000700 1 -- 172.21.15.43:0/1087083683 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234022ca0 con 0x7f7240075d20 2024-05-27T07:11:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.374+0000 7f7248600700 1 -- 172.21.15.43:0/1087083683 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72401a26f0 con 0x7f7240075d20 2024-05-27T07:11:29.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.375+0000 7f7248600700 1 -- 172.21.15.43:0/1087083683 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72400512b0 con 0x7f7240075d20 2024-05-27T07:11:29.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.377+0000 7f723e000700 1 -- 172.21.15.43:0/1087083683 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f723402a070 con 0x7f7240075d20 2024-05-27T07:11:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.380+0000 7f723e000700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f72280b2880 0x7f72280b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.381+0000 7f723e000700 1 -- 172.21.15.43:0/1087083683 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f72340e3640 con 0x7f7240075d20 2024-05-27T07:11:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.381+0000 7f7245200700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f72280b2880 0x7f72280b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:29.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.382+0000 7f7245200700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f72280b2880 0x7f72280b4d90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f722c006fd0 tx=0x7f722c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:29.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.385+0000 7f723e000700 1 -- 172.21.15.43:0/1087083683 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7234013070 con 0x7f7240075d20 2024-05-27T07:11:29.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.631+0000 7f7248600700 1 -- 172.21.15.43:0/1087083683 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f724002e5e0 con 0x7f7240075d20 2024-05-27T07:11:29.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.632+0000 7f723e000700 1 -- 172.21.15.43:0/1087083683 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f72340ab000 con 0x7f7240075d20 2024-05-27T07:11:29.623 INFO:teuthology.orchestra.run.smithi043.stdout:34359738509 2024-05-27T07:11:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 -- 172.21.15.43:0/1087083683 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f72280b2880 msgr2=0x7f72280b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:29.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f72280b2880 0x7f72280b4d90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f722c006fd0 tx=0x7f722c008040 comp rx=0 tx=0).stop 2024-05-27T07:11:29.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 -- 172.21.15.43:0/1087083683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7240075d20 msgr2=0x7f724019d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:29.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7240075d20 0x7f724019d0b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f72340057a0 tx=0x7f72340036f0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 -- 172.21.15.43:0/1087083683 shutdown_connections 2024-05-27T07:11:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f72280b2880 0x7f72280b4d90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7240110b90 0x7f72401a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7240075d20 0x7f724019d0b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 --2- 172.21.15.43:0/1087083683 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7240076670 0x7f724019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 -- 172.21.15.43:0/1087083683 >> 172.21.15.43:0/1087083683 conn(0x7f7240100030 msgr2=0x7f7240100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 -- 172.21.15.43:0/1087083683 shutdown_connections 2024-05-27T07:11:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:29.635+0000 7f7248600700 1 -- 172.21.15.43:0/1087083683 wait complete. 2024-05-27T07:11:30.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:30 smithi043 ceph-mon[31531]: pgmap v488: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-27T07:11:30.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:30.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:30.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:11:30.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:30 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1087083683' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:11:30.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:30 smithi132 ceph-mon[35283]: pgmap v488: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-27T07:11:30.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:30.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:30.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:11:30.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:30 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1087083683' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:11:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:30 smithi179 ceph-mon[35888]: pgmap v488: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-05-27T07:11:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:11:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:11:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:11:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:30 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1087083683' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-27T07:11:32.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:32 smithi043 ceph-mon[31531]: pgmap v489: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-05-27T07:11:32.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:32 smithi132 ceph-mon[35283]: pgmap v489: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-05-27T07:11:32.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:32 smithi179 ceph-mon[35888]: pgmap v489: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-05-27T07:11:34.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:34 smithi043 ceph-mon[31531]: pgmap v490: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 596 B/s wr, 9 op/s 2024-05-27T07:11:34.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:34 smithi132 ceph-mon[35283]: pgmap v490: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 596 B/s wr, 9 op/s 2024-05-27T07:11:34.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:34 smithi179 ceph-mon[35888]: pgmap v490: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 596 B/s wr, 9 op/s 2024-05-27T07:11:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.066+0000 7f705b800700 1 -- 172.21.15.43:0/3019333085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7054075b50 msgr2=0x7f7054075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.066+0000 7f705b800700 1 --2- 172.21.15.43:0/3019333085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7054075b50 0x7f7054075f60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7048009990 tx=0x7f7048009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.066+0000 7f705b800700 1 -- 172.21.15.43:0/3019333085 shutdown_connections 2024-05-27T07:11:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.066+0000 7f705b800700 1 --2- 172.21.15.43:0/3019333085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7054115360 0x7f7054117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.066+0000 7f705b800700 1 --2- 172.21.15.43:0/3019333085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7054075b50 0x7f7054075f60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.066+0000 7f705b800700 1 --2- 172.21.15.43:0/3019333085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70540764a0 0x7f7054114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.066+0000 7f705b800700 1 -- 172.21.15.43:0/3019333085 >> 172.21.15.43:0/3019333085 conn(0x7f7054100030 msgr2=0x7f7054102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.066+0000 7f705b800700 1 -- 172.21.15.43:0/3019333085 shutdown_connections 2024-05-27T07:11:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.066+0000 7f705b800700 1 -- 172.21.15.43:0/3019333085 wait complete. 2024-05-27T07:11:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.068+0000 7f705b800700 1 Processor -- start 2024-05-27T07:11:35.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.068+0000 7f705b800700 1 -- start start 2024-05-27T07:11:35.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f705b800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7054075b50 0x7f70541a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f705b800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70540764a0 0x7f70541a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f705b800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7054115360 0x7f70541a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f705b800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f705411a920 con 0x7f70540764a0 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f705b800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f705411a7a0 con 0x7f7054075b50 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f705b800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f705411aaa0 con 0x7f7054115360 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f7053e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70540764a0 0x7f70541a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f7053e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70540764a0 0x7f70541a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52058/0 (socket says 172.21.15.43:52058) 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f7053e00700 1 -- 172.21.15.43:0/3184991760 learned_addr learned my addr 172.21.15.43:0/3184991760 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f7058e00700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7054075b50 0x7f70541a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.069+0000 7f7059800700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7054115360 0x7f70541a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.070+0000 7f7053e00700 1 -- 172.21.15.43:0/3184991760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7054115360 msgr2=0x7f70541a5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.070+0000 7f7053e00700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7054115360 0x7f70541a5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.070+0000 7f7053e00700 1 -- 172.21.15.43:0/3184991760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7054075b50 msgr2=0x7f70541a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.070+0000 7f7053e00700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7054075b50 0x7f70541a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:35.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.070+0000 7f7053e00700 1 -- 172.21.15.43:0/3184991760 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70541a6150 con 0x7f70540764a0 2024-05-27T07:11:35.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.070+0000 7f7058e00700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7054075b50 0x7f70541a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:11:35.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.070+0000 7f7053e00700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70540764a0 0x7f70541a18d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f704000d950 tx=0x7f704000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:35.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.071+0000 7f7051600700 1 -- 172.21.15.43:0/3184991760 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040016070 con 0x7f70540764a0 2024-05-27T07:11:35.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.071+0000 7f705b800700 1 -- 172.21.15.43:0/3184991760 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70541a6440 con 0x7f70540764a0 2024-05-27T07:11:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.071+0000 7f7051600700 1 -- 172.21.15.43:0/3184991760 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7040010460 con 0x7f70540764a0 2024-05-27T07:11:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.071+0000 7f7051600700 1 -- 172.21.15.43:0/3184991760 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704000f620 con 0x7f70540764a0 2024-05-27T07:11:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.071+0000 7f705b800700 1 -- 172.21.15.43:0/3184991760 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70541ade10 con 0x7f70540764a0 2024-05-27T07:11:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.072+0000 7f705b800700 1 -- 172.21.15.43:0/3184991760 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70540512b0 con 0x7f70540764a0 2024-05-27T07:11:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.075+0000 7f7051600700 1 -- 172.21.15.43:0/3184991760 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f7040019070 con 0x7f70540764a0 2024-05-27T07:11:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.077+0000 7f7051600700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f703c0b2880 0x7f703c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.078+0000 7f7051600700 1 -- 172.21.15.43:0/3184991760 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f70400e2210 con 0x7f70540764a0 2024-05-27T07:11:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.078+0000 7f7058e00700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f703c0b2880 0x7f703c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.079+0000 7f7058e00700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f703c0b2880 0x7f703c0b4d90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7048003760 tx=0x7f704800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.081+0000 7f7051600700 1 -- 172.21.15.43:0/3184991760 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f70400a9b40 con 0x7f70540764a0 2024-05-27T07:11:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.330+0000 7f705b800700 1 -- 172.21.15.43:0/3184991760 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f705402e5e0 con 0x7f70540764a0 2024-05-27T07:11:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.330+0000 7f7051600700 1 -- 172.21.15.43:0/3184991760 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f70400a9440 con 0x7f70540764a0 2024-05-27T07:11:35.322 INFO:teuthology.orchestra.run.smithi043.stdout:304942678076 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 -- 172.21.15.43:0/3184991760 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f703c0b2880 msgr2=0x7f703c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f703c0b2880 0x7f703c0b4d90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7048003760 tx=0x7f704800b540 comp rx=0 tx=0).stop 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 -- 172.21.15.43:0/3184991760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70540764a0 msgr2=0x7f70541a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70540764a0 0x7f70541a18d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f704000d950 tx=0x7f704000dd10 comp rx=0 tx=0).stop 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 -- 172.21.15.43:0/3184991760 shutdown_connections 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f703c0b2880 0x7f703c0b4d90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7054115360 0x7f70541a5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7054075b50 0x7f70541a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 --2- 172.21.15.43:0/3184991760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70540764a0 0x7f70541a18d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 -- 172.21.15.43:0/3184991760 >> 172.21.15.43:0/3184991760 conn(0x7f7054100030 msgr2=0x7f7054100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 -- 172.21.15.43:0/3184991760 shutdown_connections 2024-05-27T07:11:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.334+0000 7f705b800700 1 -- 172.21.15.43:0/3184991760 wait complete. 2024-05-27T07:11:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:36 smithi043 ceph-mon[31531]: pgmap v491: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 596 B/s wr, 10 op/s 2024-05-27T07:11:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:36 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3184991760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:11:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:36 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4068980420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:11:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:36 smithi132 ceph-mon[35283]: pgmap v491: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 596 B/s wr, 10 op/s 2024-05-27T07:11:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:36 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3184991760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:11:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:36 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4068980420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:11:36.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:36 smithi179 ceph-mon[35888]: pgmap v491: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 596 B/s wr, 10 op/s 2024-05-27T07:11:36.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:36 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3184991760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-27T07:11:36.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:36 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4068980420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-27T07:11:37.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.301+0000 7fea67e00700 1 -- 172.21.15.43:0/98624867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea681043f0 msgr2=0x7fea68104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:37.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.301+0000 7fea67e00700 1 --2- 172.21.15.43:0/98624867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea681043f0 0x7fea68104800 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fea580099c0 tx=0x7fea58009cd0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.301+0000 7fea67e00700 1 -- 172.21.15.43:0/98624867 shutdown_connections 2024-05-27T07:11:37.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.301+0000 7fea67e00700 1 --2- 172.21.15.43:0/98624867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea68109670 0x7fea68110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.301+0000 7fea67e00700 1 --2- 172.21.15.43:0/98624867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea681043f0 0x7fea68104800 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.301+0000 7fea67e00700 1 --2- 172.21.15.43:0/98624867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea68108d20 0x7fea68109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.301+0000 7fea67e00700 1 -- 172.21.15.43:0/98624867 >> 172.21.15.43:0/98624867 conn(0x7fea68079d00 msgr2=0x7fea6807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.301+0000 7fea67e00700 1 -- 172.21.15.43:0/98624867 shutdown_connections 2024-05-27T07:11:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.301+0000 7fea67e00700 1 -- 172.21.15.43:0/98624867 wait complete. 2024-05-27T07:11:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.303+0000 7fea67e00700 1 Processor -- start 2024-05-27T07:11:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.303+0000 7fea67e00700 1 -- start start 2024-05-27T07:11:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea67e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea681043f0 0x7fea68075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea67e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea68108d20 0x7fea68075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea67e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea68109670 0x7fea68076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea67e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fea681141c0 con 0x7fea68109670 2024-05-27T07:11:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea67e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fea68114040 con 0x7fea68108d20 2024-05-27T07:11:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea67e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fea68114340 con 0x7fea681043f0 2024-05-27T07:11:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea67400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea68109670 0x7fea68076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea67400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea68109670 0x7fea68076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52068/0 (socket says 172.21.15.43:52068) 2024-05-27T07:11:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea67400700 1 -- 172.21.15.43:0/4068980420 learned_addr learned my addr 172.21.15.43:0/4068980420 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea66a00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea681043f0 0x7fea68075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.304+0000 7fea5e000700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea68108d20 0x7fea68075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.305+0000 7fea67400700 1 -- 172.21.15.43:0/4068980420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea681043f0 msgr2=0x7fea68075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.305+0000 7fea67400700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea681043f0 0x7fea68075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.305+0000 7fea67400700 1 -- 172.21.15.43:0/4068980420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea68108d20 msgr2=0x7fea68075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.305+0000 7fea67400700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea68108d20 0x7fea68075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.305+0000 7fea67400700 1 -- 172.21.15.43:0/4068980420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea681a8f70 con 0x7fea68109670 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.305+0000 7fea66a00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea681043f0 0x7fea68075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.305+0000 7fea67400700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea68109670 0x7fea68076530 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fea60007ee0 tx=0x7fea6000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.305+0000 7fea5fe00700 1 -- 172.21.15.43:0/4068980420 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6000dbd0 con 0x7fea68109670 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.305+0000 7fea67e00700 1 -- 172.21.15.43:0/4068980420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea681a9260 con 0x7fea68109670 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.305+0000 7fea5fe00700 1 -- 172.21.15.43:0/4068980420 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fea6000dd70 con 0x7fea68109670 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.306+0000 7fea5fe00700 1 -- 172.21.15.43:0/4068980420 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea600058d0 con 0x7fea68109670 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.306+0000 7fea67e00700 1 -- 172.21.15.43:0/4068980420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea681a97f0 con 0x7fea68109670 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.307+0000 7fea67e00700 1 -- 172.21.15.43:0/4068980420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea680512b0 con 0x7fea68109670 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.310+0000 7fea5fe00700 1 -- 172.21.15.43:0/4068980420 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fea6001a020 con 0x7fea68109670 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.311+0000 7fea5fe00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fea4c0b2880 0x7fea4c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.311+0000 7fea5fe00700 1 -- 172.21.15.43:0/4068980420 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fea600e23f0 con 0x7fea68109670 2024-05-27T07:11:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.312+0000 7fea66a00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fea4c0b2880 0x7fea4c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:37.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.313+0000 7fea66a00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fea4c0b2880 0x7fea4c0b4d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fea58000c00 tx=0x7fea58005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:37.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.314+0000 7fea5fe00700 1 -- 172.21.15.43:0/4068980420 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fea60010070 con 0x7fea68109670 2024-05-27T07:11:37.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.564+0000 7fea67e00700 1 -- 172.21.15.43:0/4068980420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fea6802e5e0 con 0x7fea68109670 2024-05-27T07:11:37.294 INFO:teuthology.orchestra.run.smithi043.stdout:257698037833 2024-05-27T07:11:37.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.565+0000 7fea5fe00700 1 -- 172.21.15.43:0/4068980420 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fea600a9d80 con 0x7fea68109670 2024-05-27T07:11:37.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.569+0000 7fea67e00700 1 -- 172.21.15.43:0/4068980420 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fea4c0b2880 msgr2=0x7fea4c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.569+0000 7fea67e00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fea4c0b2880 0x7fea4c0b4d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fea58000c00 tx=0x7fea58005fd0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.569+0000 7fea67e00700 1 -- 172.21.15.43:0/4068980420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea68109670 msgr2=0x7fea68076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.569+0000 7fea67e00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea68109670 0x7fea68076530 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fea60007ee0 tx=0x7fea6000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.570+0000 7fea67e00700 1 -- 172.21.15.43:0/4068980420 shutdown_connections 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.570+0000 7fea67e00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fea4c0b2880 0x7fea4c0b4d90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.570+0000 7fea67e00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea68109670 0x7fea68076530 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.570+0000 7fea67e00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea681043f0 0x7fea68075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.570+0000 7fea67e00700 1 --2- 172.21.15.43:0/4068980420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea68108d20 0x7fea68075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.570+0000 7fea67e00700 1 -- 172.21.15.43:0/4068980420 >> 172.21.15.43:0/4068980420 conn(0x7fea68079d00 msgr2=0x7fea68101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.570+0000 7fea67e00700 1 -- 172.21.15.43:0/4068980420 shutdown_connections 2024-05-27T07:11:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:35.570+0000 7fea67e00700 1 -- 172.21.15.43:0/4068980420 wait complete. 2024-05-27T07:11:37.644 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738504 got 34359738509 for osd.0 2024-05-27T07:11:37.645 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:38.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:38 smithi043 ceph-mon[31531]: pgmap v492: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-27T07:11:38.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:38 smithi132 ceph-mon[35283]: pgmap v492: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-27T07:11:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:38 smithi179 ceph-mon[35888]: pgmap v492: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-27T07:11:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.651+0000 7ff58e000700 1 -- 172.21.15.43:0/2503846130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff588076670 msgr2=0x7ff588110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.651+0000 7ff58e000700 1 --2- 172.21.15.43:0/2503846130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff588076670 0x7ff588110650 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff570009990 tx=0x7ff570009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.652+0000 7ff58e000700 1 -- 172.21.15.43:0/2503846130 shutdown_connections 2024-05-27T07:11:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.652+0000 7ff58e000700 1 --2- 172.21.15.43:0/2503846130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff588110b90 0x7ff588112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.652+0000 7ff58e000700 1 --2- 172.21.15.43:0/2503846130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff588075d20 0x7ff588076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.652+0000 7ff58e000700 1 --2- 172.21.15.43:0/2503846130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff588076670 0x7ff588110650 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.652+0000 7ff58e000700 1 -- 172.21.15.43:0/2503846130 >> 172.21.15.43:0/2503846130 conn(0x7ff588100030 msgr2=0x7ff588102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.652+0000 7ff58e000700 1 -- 172.21.15.43:0/2503846130 shutdown_connections 2024-05-27T07:11:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.652+0000 7ff58e000700 1 -- 172.21.15.43:0/2503846130 wait complete. 2024-05-27T07:11:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.653+0000 7ff58e000700 1 Processor -- start 2024-05-27T07:11:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.653+0000 7ff58e000700 1 -- start start 2024-05-27T07:11:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.654+0000 7ff58e000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff588075d20 0x7ff58819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.654+0000 7ff58e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff588076670 0x7ff58819d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.654+0000 7ff58e000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff588110b90 0x7ff5881a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.654+0000 7ff58e000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff588116590 con 0x7ff588076670 2024-05-27T07:11:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.654+0000 7ff58e000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff588116410 con 0x7ff588110b90 2024-05-27T07:11:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.654+0000 7ff58e000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff588116710 con 0x7ff588075d20 2024-05-27T07:11:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.654+0000 7ff57fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff588076670 0x7ff58819d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.654+0000 7ff57fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff588076670 0x7ff58819d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52074/0 (socket says 172.21.15.43:52074) 2024-05-27T07:11:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.654+0000 7ff57fe00700 1 -- 172.21.15.43:0/3263566653 learned_addr learned my addr 172.21.15.43:0/3263566653 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.655+0000 7ff587400700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff588075d20 0x7ff58819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.655+0000 7ff587e00700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff588110b90 0x7ff5881a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.655+0000 7ff57fe00700 1 -- 172.21.15.43:0/3263566653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff588075d20 msgr2=0x7ff58819cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.655+0000 7ff57fe00700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff588075d20 0x7ff58819cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.655+0000 7ff57fe00700 1 -- 172.21.15.43:0/3263566653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff588110b90 msgr2=0x7ff5881a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.655+0000 7ff57fe00700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff588110b90 0x7ff5881a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.655+0000 7ff57fe00700 1 -- 172.21.15.43:0/3263566653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5881a1e90 con 0x7ff588076670 2024-05-27T07:11:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.655+0000 7ff57fe00700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff588076670 0x7ff58819d560 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff570000c00 tx=0x7ff570005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.656+0000 7ff584c00700 1 -- 172.21.15.43:0/3263566653 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff570027070 con 0x7ff588076670 2024-05-27T07:11:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.656+0000 7ff58e000700 1 -- 172.21.15.43:0/3263566653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5881a2120 con 0x7ff588076670 2024-05-27T07:11:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.656+0000 7ff584c00700 1 -- 172.21.15.43:0/3263566653 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff57000fb80 con 0x7ff588076670 2024-05-27T07:11:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.656+0000 7ff584c00700 1 -- 172.21.15.43:0/3263566653 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff570022c30 con 0x7ff588076670 2024-05-27T07:11:38.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.656+0000 7ff58e000700 1 -- 172.21.15.43:0/3263566653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5881a2600 con 0x7ff588076670 2024-05-27T07:11:38.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.657+0000 7ff58e000700 1 -- 172.21.15.43:0/3263566653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5880512b0 con 0x7ff588076670 2024-05-27T07:11:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.662+0000 7ff584c00700 1 -- 172.21.15.43:0/3263566653 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7ff57002a070 con 0x7ff588076670 2024-05-27T07:11:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.663+0000 7ff584c00700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff56c0b2880 0x7ff56c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.663+0000 7ff584c00700 1 -- 172.21.15.43:0/3263566653 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7ff5700e3860 con 0x7ff588076670 2024-05-27T07:11:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.664+0000 7ff587400700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff56c0b2880 0x7ff56c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.664+0000 7ff587400700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff56c0b2880 0x7ff56c0b4d90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff578005fd0 tx=0x7ff57800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:38.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.666+0000 7ff584c00700 1 -- 172.21.15.43:0/3263566653 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff57005fdf0 con 0x7ff588076670 2024-05-27T07:11:38.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.910+0000 7ff58e000700 1 -- 172.21.15.43:0/3263566653 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff58802e5e0 con 0x7ff588076670 2024-05-27T07:11:38.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.911+0000 7ff584c00700 1 -- 172.21.15.43:0/3263566653 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff570013070 con 0x7ff588076670 2024-05-27T07:11:38.902 INFO:teuthology.orchestra.run.smithi043.stdout:77309411455 2024-05-27T07:11:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.914+0000 7ff58e000700 1 -- 172.21.15.43:0/3263566653 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff56c0b2880 msgr2=0x7ff56c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.914+0000 7ff58e000700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff56c0b2880 0x7ff56c0b4d90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff578005fd0 tx=0x7ff57800a560 comp rx=0 tx=0).stop 2024-05-27T07:11:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.914+0000 7ff58e000700 1 -- 172.21.15.43:0/3263566653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff588076670 msgr2=0x7ff58819d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.914+0000 7ff58e000700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff588076670 0x7ff58819d560 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff570000c00 tx=0x7ff570005ef0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.914+0000 7ff58e000700 1 -- 172.21.15.43:0/3263566653 shutdown_connections 2024-05-27T07:11:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.914+0000 7ff58e000700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff56c0b2880 0x7ff56c0b4d90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.914+0000 7ff58e000700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff588110b90 0x7ff5881a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.914+0000 7ff58e000700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff588075d20 0x7ff58819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.914+0000 7ff58e000700 1 --2- 172.21.15.43:0/3263566653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff588076670 0x7ff58819d560 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.914+0000 7ff58e000700 1 -- 172.21.15.43:0/3263566653 >> 172.21.15.43:0/3263566653 conn(0x7ff588100030 msgr2=0x7ff5881006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.915+0000 7ff58e000700 1 -- 172.21.15.43:0/3263566653 shutdown_connections 2024-05-27T07:11:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:38.915+0000 7ff58e000700 1 -- 172.21.15.43:0/3263566653 wait complete. 2024-05-27T07:11:39.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:39 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3263566653' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:11:39.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:39 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3263566653' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:11:39.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:39 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3263566653' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-27T07:11:40.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:40 smithi043 ceph-mon[31531]: pgmap v493: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 10 op/s 2024-05-27T07:11:40.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:40 smithi132 ceph-mon[35283]: pgmap v493: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 10 op/s 2024-05-27T07:11:40.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:40 smithi179 ceph-mon[35888]: pgmap v493: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 10 op/s 2024-05-27T07:11:40.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.796+0000 7f0ce6800700 1 -- 172.21.15.43:0/3216637420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce010a980 msgr2=0x7f0ce010ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:40.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.796+0000 7f0ce6800700 1 --2- 172.21.15.43:0/3216637420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce010a980 0x7f0ce010ad90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009990 tx=0x7f0ccc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:40.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.797+0000 7f0ce6800700 1 -- 172.21.15.43:0/3216637420 shutdown_connections 2024-05-27T07:11:40.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.797+0000 7f0ce6800700 1 --2- 172.21.15.43:0/3216637420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ce010b2d0 0x7f0ce0117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:40.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.797+0000 7f0ce6800700 1 --2- 172.21.15.43:0/3216637420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce0110960 0x7f0ce0110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:40.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.797+0000 7f0ce6800700 1 --2- 172.21.15.43:0/3216637420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce010a980 0x7f0ce010ad90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:40.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.797+0000 7f0ce6800700 1 -- 172.21.15.43:0/3216637420 >> 172.21.15.43:0/3216637420 conn(0x7f0ce0077c70 msgr2=0x7f0ce00780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:40.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.797+0000 7f0ce6800700 1 -- 172.21.15.43:0/3216637420 shutdown_connections 2024-05-27T07:11:40.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.797+0000 7f0ce6800700 1 -- 172.21.15.43:0/3216637420 wait complete. 2024-05-27T07:11:40.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.798+0000 7f0ce6800700 1 Processor -- start 2024-05-27T07:11:40.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.798+0000 7f0ce6800700 1 -- start start 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0ce6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ce010a980 0x7f0ce01a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0ce6800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce010b2d0 0x7f0ce01a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0ce6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce0110960 0x7f0ce01a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0ce6800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce011a980 con 0x7f0ce0110960 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0ce6800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce011a800 con 0x7f0ce010a980 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0ce6800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce011ab00 con 0x7f0ce010b2d0 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0cdfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce0110960 0x7f0ce01a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0cdfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce0110960 0x7f0ce01a5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52082/0 (socket says 172.21.15.43:52082) 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0cdfe00700 1 -- 172.21.15.43:0/2220530868 learned_addr learned my addr 172.21.15.43:0/2220530868 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0cdf400700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ce010a980 0x7f0ce01a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.799+0000 7f0cdea00700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce010b2d0 0x7f0ce01a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:40.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.800+0000 7f0cdfe00700 1 -- 172.21.15.43:0/2220530868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce010b2d0 msgr2=0x7f0ce01a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.800+0000 7f0cdfe00700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce010b2d0 0x7f0ce01a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.800+0000 7f0cdfe00700 1 -- 172.21.15.43:0/2220530868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ce010a980 msgr2=0x7f0ce01a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.800+0000 7f0cdfe00700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ce010a980 0x7f0ce01a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.800+0000 7f0cdfe00700 1 -- 172.21.15.43:0/2220530868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce01a6280 con 0x7f0ce0110960 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.800+0000 7f0cdea00700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce010b2d0 0x7f0ce01a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.800+0000 7f0cdfe00700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce0110960 0x7f0ce01a5be0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8007ee0 tx=0x7f0cd800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.800+0000 7f0cd7e00700 1 -- 172.21.15.43:0/2220530868 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd800dbd0 con 0x7f0ce0110960 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.800+0000 7f0ce6800700 1 -- 172.21.15.43:0/2220530868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce01a6570 con 0x7f0ce0110960 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.801+0000 7f0cd7e00700 1 -- 172.21.15.43:0/2220530868 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0cd800dd70 con 0x7f0ce0110960 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.801+0000 7f0cd7e00700 1 -- 172.21.15.43:0/2220530868 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd80058d0 con 0x7f0ce0110960 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.801+0000 7f0ce6800700 1 -- 172.21.15.43:0/2220530868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce01ade90 con 0x7f0ce0110960 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.802+0000 7f0ce6800700 1 -- 172.21.15.43:0/2220530868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ce00512b0 con 0x7f0ce0110960 2024-05-27T07:11:40.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.806+0000 7f0cd7e00700 1 -- 172.21.15.43:0/2220530868 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0cd801a020 con 0x7f0ce0110960 2024-05-27T07:11:40.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.808+0000 7f0cd7e00700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0cc40b2880 0x7f0cc40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:40.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.808+0000 7f0cd7e00700 1 -- 172.21.15.43:0/2220530868 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f0cd80e2400 con 0x7f0ce0110960 2024-05-27T07:11:40.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.808+0000 7f0cdf400700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0cc40b2880 0x7f0cc40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:40.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.809+0000 7f0cdf400700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0cc40b2880 0x7f0cc40b4d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0005950 tx=0x7f0cd000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:40.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:40.811+0000 7f0cd7e00700 1 -- 172.21.15.43:0/2220530868 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0cd8010070 con 0x7f0ce0110960 2024-05-27T07:11:41.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.064+0000 7f0ce6800700 1 -- 172.21.15.43:0/2220530868 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0ce002e5e0 con 0x7f0ce0110960 2024-05-27T07:11:41.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.064+0000 7f0cd7e00700 1 -- 172.21.15.43:0/2220530868 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0cd80a9d30 con 0x7f0ce0110960 2024-05-27T07:11:41.055 INFO:teuthology.orchestra.run.smithi043.stdout:158913790058 2024-05-27T07:11:41.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.067+0000 7f0ce6800700 1 -- 172.21.15.43:0/2220530868 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0cc40b2880 msgr2=0x7f0cc40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:41.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.067+0000 7f0ce6800700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0cc40b2880 0x7f0cc40b4d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0005950 tx=0x7f0cd000b410 comp rx=0 tx=0).stop 2024-05-27T07:11:41.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.067+0000 7f0ce6800700 1 -- 172.21.15.43:0/2220530868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce0110960 msgr2=0x7f0ce01a5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:41.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.067+0000 7f0ce6800700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce0110960 0x7f0ce01a5be0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8007ee0 tx=0x7f0cd800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:11:41.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.068+0000 7f0ce6800700 1 -- 172.21.15.43:0/2220530868 shutdown_connections 2024-05-27T07:11:41.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.068+0000 7f0ce6800700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0cc40b2880 0x7f0cc40b4d90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:41.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.068+0000 7f0ce6800700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ce0110960 0x7f0ce01a5be0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:41.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.068+0000 7f0ce6800700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ce010a980 0x7f0ce01a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:41.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.068+0000 7f0ce6800700 1 --2- 172.21.15.43:0/2220530868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce010b2d0 0x7f0ce01a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:41.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.068+0000 7f0ce6800700 1 -- 172.21.15.43:0/2220530868 >> 172.21.15.43:0/2220530868 conn(0x7f0ce0077c70 msgr2=0x7f0ce007b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:41.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.068+0000 7f0ce6800700 1 -- 172.21.15.43:0/2220530868 shutdown_connections 2024-05-27T07:11:41.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:41.068+0000 7f0ce6800700 1 -- 172.21.15.43:0/2220530868 wait complete. 2024-05-27T07:11:41.286 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037829 got 257698037833 for osd.9 2024-05-27T07:11:41.287 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:41.649 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:42.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:42 smithi043 ceph-mon[31531]: pgmap v494: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:42.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:42 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2220530868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:11:42.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:42 smithi132 ceph-mon[35283]: pgmap v494: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:42.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:42 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2220530868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:11:42.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:42 smithi179 ceph-mon[35888]: pgmap v494: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:42.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:42 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2220530868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-27T07:11:43.070 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411451 got 77309411455 for osd.2 2024-05-27T07:11:43.071 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:43.525 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678072 got 304942678076 for osd.11 2024-05-27T07:11:43.525 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.872+0000 7fcecb200700 1 -- 172.21.15.43:0/2849616433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcec4105e00 msgr2=0x7fcec410e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.873+0000 7fcecb200700 1 --2- 172.21.15.43:0/2849616433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcec4105e00 0x7fcec410e350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009990 tx=0x7fceb0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.873+0000 7fcecb200700 1 -- 172.21.15.43:0/2849616433 shutdown_connections 2024-05-27T07:11:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.873+0000 7fcecb200700 1 --2- 172.21.15.43:0/2849616433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcec410e8e0 0x7fcec4110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.873+0000 7fcecb200700 1 --2- 172.21.15.43:0/2849616433 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec41054b0 0x7fcec41058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.873+0000 7fcecb200700 1 --2- 172.21.15.43:0/2849616433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcec4105e00 0x7fcec410e350 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.873+0000 7fcecb200700 1 -- 172.21.15.43:0/2849616433 >> 172.21.15.43:0/2849616433 conn(0x7fcec40fedb0 msgr2=0x7fcec41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.873+0000 7fcecb200700 1 -- 172.21.15.43:0/2849616433 shutdown_connections 2024-05-27T07:11:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.873+0000 7fcecb200700 1 -- 172.21.15.43:0/2849616433 wait complete. 2024-05-27T07:11:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.874+0000 7fcecb200700 1 Processor -- start 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.874+0000 7fcecb200700 1 -- start start 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.875+0000 7fcecb200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec41054b0 0x7fcec41a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.875+0000 7fcecb200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcec4105e00 0x7fcec41a2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.875+0000 7fcecb200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcec410e8e0 0x7fcec41ad070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.875+0000 7fcecb200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4114310 con 0x7fcec4105e00 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.875+0000 7fcecb200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4114190 con 0x7fcec410e8e0 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.875+0000 7fcecb200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4114490 con 0x7fcec41054b0 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.875+0000 7fcec3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcec4105e00 0x7fcec41a2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.875+0000 7fcec3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcec4105e00 0x7fcec41a2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52108/0 (socket says 172.21.15.43:52108) 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.876+0000 7fcec3400700 1 -- 172.21.15.43:0/4175532284 learned_addr learned my addr 172.21.15.43:0/4175532284 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.876+0000 7fcec3e00700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec41054b0 0x7fcec41a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.876+0000 7fcec9200700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcec410e8e0 0x7fcec41ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.876+0000 7fcec3400700 1 -- 172.21.15.43:0/4175532284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec41054b0 msgr2=0x7fcec41a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.876+0000 7fcec3400700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec41054b0 0x7fcec41a21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.876+0000 7fcec3400700 1 -- 172.21.15.43:0/4175532284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcec410e8e0 msgr2=0x7fcec41ad070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.876+0000 7fcec3400700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcec410e8e0 0x7fcec41ad070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.876+0000 7fcec3400700 1 -- 172.21.15.43:0/4175532284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec41ad5b0 con 0x7fcec4105e00 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.877+0000 7fcec3400700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcec4105e00 0x7fcec41a2700 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fceb0000c00 tx=0x7fceb00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.877+0000 7fcec0c00700 1 -- 172.21.15.43:0/4175532284 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0027070 con 0x7fcec4105e00 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.877+0000 7fcec0c00700 1 -- 172.21.15.43:0/4175532284 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fceb000fb80 con 0x7fcec4105e00 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.877+0000 7fcec0c00700 1 -- 172.21.15.43:0/4175532284 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0022d00 con 0x7fcec4105e00 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.877+0000 7fcecb200700 1 -- 172.21.15.43:0/4175532284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec41ad780 con 0x7fcec4105e00 2024-05-27T07:11:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.877+0000 7fcecb200700 1 -- 172.21.15.43:0/4175532284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec41adbe0 con 0x7fcec4105e00 2024-05-27T07:11:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.878+0000 7fcecb200700 1 -- 172.21.15.43:0/4175532284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec4107a70 con 0x7fcec4105e00 2024-05-27T07:11:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.881+0000 7fcec0c00700 1 -- 172.21.15.43:0/4175532284 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fceb002a070 con 0x7fcec4105e00 2024-05-27T07:11:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.884+0000 7fcec0c00700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcea80b2900 0x7fcea80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.884+0000 7fcec0c00700 1 -- 172.21.15.43:0/4175532284 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fceb00e37f0 con 0x7fcec4105e00 2024-05-27T07:11:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.884+0000 7fcec3e00700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcea80b2900 0x7fcea80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.885+0000 7fcec3e00700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcea80b2900 0x7fcea80b4e10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fceb4004630 tx=0x7fceb400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:43.886+0000 7fcec0c00700 1 -- 172.21.15.43:0/4175532284 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fceb0031080 con 0x7fcec4105e00 2024-05-27T07:11:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.137+0000 7fcecb200700 1 -- 172.21.15.43:0/4175532284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fcec4107450 con 0x7fcec4105e00 2024-05-27T07:11:44.175 INFO:teuthology.orchestra.run.smithi043.stdout:231928234067 2024-05-27T07:11:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.138+0000 7fcec0c00700 1 -- 172.21.15.43:0/4175532284 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fceb00b3030 con 0x7fcec4105e00 2024-05-27T07:11:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 -- 172.21.15.43:0/4175532284 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcea80b2900 msgr2=0x7fcea80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcea80b2900 0x7fcea80b4e10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fceb4004630 tx=0x7fceb400b3c0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 -- 172.21.15.43:0/4175532284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcec4105e00 msgr2=0x7fcec41a2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcec4105e00 0x7fcec41a2700 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fceb0000c00 tx=0x7fceb00047b0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 -- 172.21.15.43:0/4175532284 shutdown_connections 2024-05-27T07:11:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcea80b2900 0x7fcea80b4e10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcec410e8e0 0x7fcec41ad070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec41054b0 0x7fcec41a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 --2- 172.21.15.43:0/4175532284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcec4105e00 0x7fcec41a2700 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 -- 172.21.15.43:0/4175532284 >> 172.21.15.43:0/4175532284 conn(0x7fcec40fedb0 msgr2=0x7fcec41032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 -- 172.21.15.43:0/4175532284 shutdown_connections 2024-05-27T07:11:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.141+0000 7fcecb200700 1 -- 172.21.15.43:0/4175532284 wait complete. 2024-05-27T07:11:44.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:44 smithi132 ceph-mon[35283]: pgmap v495: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:11:44.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:44 smithi043 ceph-mon[31531]: pgmap v495: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:11:44.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:44 smithi179 ceph-mon[35888]: pgmap v495: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:11:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.871+0000 7fb1c0c00700 1 -- 172.21.15.43:0/128800051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1bc1045e0 msgr2=0x7fb1bc1049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.871+0000 7fb1c0c00700 1 --2- 172.21.15.43:0/128800051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1bc1045e0 0x7fb1bc1049f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0099c0 tx=0x7fb1ac009cd0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.871+0000 7fb1c0c00700 1 -- 172.21.15.43:0/128800051 shutdown_connections 2024-05-27T07:11:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.871+0000 7fb1c0c00700 1 --2- 172.21.15.43:0/128800051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1bc109860 0x7fb1bc1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.871+0000 7fb1c0c00700 1 --2- 172.21.15.43:0/128800051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1bc1045e0 0x7fb1bc1049f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.871+0000 7fb1c0c00700 1 --2- 172.21.15.43:0/128800051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1bc108f10 0x7fb1bc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.871+0000 7fb1c0c00700 1 -- 172.21.15.43:0/128800051 >> 172.21.15.43:0/128800051 conn(0x7fb1bc100030 msgr2=0x7fb1bc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.871+0000 7fb1c0c00700 1 -- 172.21.15.43:0/128800051 shutdown_connections 2024-05-27T07:11:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.871+0000 7fb1c0c00700 1 -- 172.21.15.43:0/128800051 wait complete. 2024-05-27T07:11:44.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.873+0000 7fb1c0c00700 1 Processor -- start 2024-05-27T07:11:44.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.873+0000 7fb1c0c00700 1 -- start start 2024-05-27T07:11:44.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1c0c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1bc1045e0 0x7fb1bc19ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:44.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1c0c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1bc108f10 0x7fb1bc19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:44.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1c0c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1bc109860 0x7fb1bc19f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:44.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1c0c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc114390 con 0x7fb1bc1045e0 2024-05-27T07:11:44.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1c0c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc114210 con 0x7fb1bc108f10 2024-05-27T07:11:44.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1c0c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc114510 con 0x7fb1bc109860 2024-05-27T07:11:44.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1b9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1bc1045e0 0x7fb1bc19ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:44.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1b9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1bc1045e0 0x7fb1bc19ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44114/0 (socket says 172.21.15.43:44114) 2024-05-27T07:11:44.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1b9e00700 1 -- 172.21.15.43:0/1123952288 learned_addr learned my addr 172.21.15.43:0/1123952288 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1b9400700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1bc108f10 0x7fb1bc19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.874+0000 7fb1ba800700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1bc109860 0x7fb1bc19f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.875+0000 7fb1b9e00700 1 -- 172.21.15.43:0/1123952288 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1bc109860 msgr2=0x7fb1bc19f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.875+0000 7fb1b9e00700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1bc109860 0x7fb1bc19f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.875+0000 7fb1b9e00700 1 -- 172.21.15.43:0/1123952288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1bc108f10 msgr2=0x7fb1bc19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.875+0000 7fb1b9e00700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1bc108f10 0x7fb1bc19b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:44.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.875+0000 7fb1b9e00700 1 -- 172.21.15.43:0/1123952288 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1bc19fb80 con 0x7fb1bc1045e0 2024-05-27T07:11:44.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.875+0000 7fb1b9e00700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1bc1045e0 0x7fb1bc19ad60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac000c00 tx=0x7fb1ac00f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:44.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.875+0000 7fb1b2a00700 1 -- 172.21.15.43:0/1123952288 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac027070 con 0x7fb1bc1045e0 2024-05-27T07:11:44.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.875+0000 7fb1b2a00700 1 -- 172.21.15.43:0/1123952288 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb1ac00fe50 con 0x7fb1bc1045e0 2024-05-27T07:11:44.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.875+0000 7fb1b2a00700 1 -- 172.21.15.43:0/1123952288 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac0173f0 con 0x7fb1bc1045e0 2024-05-27T07:11:44.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.876+0000 7fb1c0c00700 1 -- 172.21.15.43:0/1123952288 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1bc19fe10 con 0x7fb1bc1045e0 2024-05-27T07:11:44.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.876+0000 7fb1c0c00700 1 -- 172.21.15.43:0/1123952288 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1bc1a02a0 con 0x7fb1bc1045e0 2024-05-27T07:11:44.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.877+0000 7fb1c0c00700 1 -- 172.21.15.43:0/1123952288 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1bc0512b0 con 0x7fb1bc1045e0 2024-05-27T07:11:44.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.880+0000 7fb1b2a00700 1 -- 172.21.15.43:0/1123952288 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fb1ac02a070 con 0x7fb1bc1045e0 2024-05-27T07:11:44.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.881+0000 7fb1b2a00700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb1a00b2880 0x7fb1a00b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.881+0000 7fb1b2a00700 1 -- 172.21.15.43:0/1123952288 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fb1ac0e37c0 con 0x7fb1bc1045e0 2024-05-27T07:11:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.881+0000 7fb1b9400700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb1a00b2880 0x7fb1a00b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.882+0000 7fb1b9400700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb1a00b2880 0x7fb1a00b4d90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8005950 tx=0x7fb1a80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:44.883+0000 7fb1b2a00700 1 -- 172.21.15.43:0/1123952288 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb1ac05fdf0 con 0x7fb1bc1045e0 2024-05-27T07:11:45.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.136+0000 7fb1c0c00700 1 -- 172.21.15.43:0/1123952288 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb1bc02e5e0 con 0x7fb1bc1045e0 2024-05-27T07:11:45.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:45 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4175532284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:11:45.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.137+0000 7fb1b2a00700 1 -- 172.21.15.43:0/1123952288 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb1ac013070 con 0x7fb1bc1045e0 2024-05-27T07:11:45.128 INFO:teuthology.orchestra.run.smithi043.stdout:111669149817 2024-05-27T07:11:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.141+0000 7fb1c0c00700 1 -- 172.21.15.43:0/1123952288 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb1a00b2880 msgr2=0x7fb1a00b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.141+0000 7fb1c0c00700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb1a00b2880 0x7fb1a00b4d90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8005950 tx=0x7fb1a80058e0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.141+0000 7fb1c0c00700 1 -- 172.21.15.43:0/1123952288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1bc1045e0 msgr2=0x7fb1bc19ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.141+0000 7fb1c0c00700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1bc1045e0 0x7fb1bc19ad60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac000c00 tx=0x7fb1ac00f510 comp rx=0 tx=0).stop 2024-05-27T07:11:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.141+0000 7fb1c0c00700 1 -- 172.21.15.43:0/1123952288 shutdown_connections 2024-05-27T07:11:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.141+0000 7fb1c0c00700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb1a00b2880 0x7fb1a00b4d90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.141+0000 7fb1c0c00700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1bc109860 0x7fb1bc19f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.141+0000 7fb1c0c00700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1bc1045e0 0x7fb1bc19ad60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.141+0000 7fb1c0c00700 1 --2- 172.21.15.43:0/1123952288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1bc108f10 0x7fb1bc19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.141+0000 7fb1c0c00700 1 -- 172.21.15.43:0/1123952288 >> 172.21.15.43:0/1123952288 conn(0x7fb1bc100030 msgr2=0x7fb1bc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.142+0000 7fb1c0c00700 1 -- 172.21.15.43:0/1123952288 shutdown_connections 2024-05-27T07:11:45.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.142+0000 7fb1c0c00700 1 -- 172.21.15.43:0/1123952288 wait complete. 2024-05-27T07:11:45.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:45 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4175532284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:11:45.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:45 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4175532284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-27T07:11:45.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.846+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2675587416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d8108f10 msgr2=0x7fc0d8109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:45.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.846+0000 7fc0dfa00700 1 --2- 172.21.15.43:0/2675587416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d8108f10 0x7fc0d8109320 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc0c40099c0 tx=0x7fc0c4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.846+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2675587416 shutdown_connections 2024-05-27T07:11:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.846+0000 7fc0dfa00700 1 --2- 172.21.15.43:0/2675587416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0d8109860 0x7fc0d81111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.846+0000 7fc0dfa00700 1 --2- 172.21.15.43:0/2675587416 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d81045e0 0x7fc0d81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.846+0000 7fc0dfa00700 1 --2- 172.21.15.43:0/2675587416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d8108f10 0x7fc0d8109320 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.846+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2675587416 >> 172.21.15.43:0/2675587416 conn(0x7fc0d8100030 msgr2=0x7fc0d8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.846+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2675587416 shutdown_connections 2024-05-27T07:11:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.846+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2675587416 wait complete. 2024-05-27T07:11:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.848+0000 7fc0dfa00700 1 Processor -- start 2024-05-27T07:11:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.848+0000 7fc0dfa00700 1 -- start start 2024-05-27T07:11:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.849+0000 7fc0dfa00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d81045e0 0x7fc0d8075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.849+0000 7fc0dfa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0d8108f10 0x7fc0d8075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.849+0000 7fc0dfa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d8109860 0x7fc0d8076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.849+0000 7fc0dfa00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d8114370 con 0x7fc0d8109860 2024-05-27T07:11:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.849+0000 7fc0dfa00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d81141f0 con 0x7fc0d8108f10 2024-05-27T07:11:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.849+0000 7fc0dfa00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d81144f0 con 0x7fc0d81045e0 2024-05-27T07:11:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.849+0000 7fc0dda00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d8109860 0x7fc0d8076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.849+0000 7fc0dda00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d8109860 0x7fc0d8076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44140/0 (socket says 172.21.15.43:44140) 2024-05-27T07:11:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.849+0000 7fc0dda00700 1 -- 172.21.15.43:0/2563186162 learned_addr learned my addr 172.21.15.43:0/2563186162 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.850+0000 7fc0d7e00700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0d8108f10 0x7fc0d8075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.850+0000 7fc0dd000700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d81045e0 0x7fc0d8075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.850+0000 7fc0dda00700 1 -- 172.21.15.43:0/2563186162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d81045e0 msgr2=0x7fc0d8075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.850+0000 7fc0dda00700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d81045e0 0x7fc0d8075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.850+0000 7fc0dda00700 1 -- 172.21.15.43:0/2563186162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0d8108f10 msgr2=0x7fc0d8075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.850+0000 7fc0dda00700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0d8108f10 0x7fc0d8075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.850+0000 7fc0dda00700 1 -- 172.21.15.43:0/2563186162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d81a9070 con 0x7fc0d8109860 2024-05-27T07:11:45.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.850+0000 7fc0dda00700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d8109860 0x7fc0d8076530 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0007d70 tx=0x7fc0d000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:45.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.851+0000 7fc0d5600700 1 -- 172.21.15.43:0/2563186162 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0016020 con 0x7fc0d8109860 2024-05-27T07:11:45.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.851+0000 7fc0d5600700 1 -- 172.21.15.43:0/2563186162 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0d000ac90 con 0x7fc0d8109860 2024-05-27T07:11:45.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.851+0000 7fc0d5600700 1 -- 172.21.15.43:0/2563186162 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d00115e0 con 0x7fc0d8109860 2024-05-27T07:11:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.851+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2563186162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0d81a9360 con 0x7fc0d8109860 2024-05-27T07:11:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.851+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2563186162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0d81a9820 con 0x7fc0d8109860 2024-05-27T07:11:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.852+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2563186162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0d80512b0 con 0x7fc0d8109860 2024-05-27T07:11:45.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.857+0000 7fc0d5600700 1 -- 172.21.15.43:0/2563186162 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fc0d0019020 con 0x7fc0d8109860 2024-05-27T07:11:45.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.858+0000 7fc0d5600700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0c00b2900 0x7fc0c00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.859+0000 7fc0d5600700 1 -- 172.21.15.43:0/2563186162 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fc0d00e2990 con 0x7fc0d8109860 2024-05-27T07:11:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.859+0000 7fc0dd000700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0c00b2900 0x7fc0c00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.860+0000 7fc0dd000700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0c00b2900 0x7fc0c00b4e10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc00ac50 tx=0x7fc0cc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:45.862+0000 7fc0d5600700 1 -- 172.21.15.43:0/2563186162 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc0d0020020 con 0x7fc0d8109860 2024-05-27T07:11:46.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.112+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2563186162 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc0d802e5e0 con 0x7fc0d8109860 2024-05-27T07:11:46.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:46 smithi043 ceph-mon[31531]: pgmap v496: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:46.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:46 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1123952288' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:11:46.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.113+0000 7fc0d5600700 1 -- 172.21.15.43:0/2563186162 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc0d00aa320 con 0x7fc0d8109860 2024-05-27T07:11:46.105 INFO:teuthology.orchestra.run.smithi043.stdout:55834574984 2024-05-27T07:11:46.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.116+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2563186162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0c00b2900 msgr2=0x7fc0c00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:46.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.116+0000 7fc0dfa00700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0c00b2900 0x7fc0c00b4e10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc00ac50 tx=0x7fc0cc00a380 comp rx=0 tx=0).stop 2024-05-27T07:11:46.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.116+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2563186162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d8109860 msgr2=0x7fc0d8076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:46.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.116+0000 7fc0dfa00700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d8109860 0x7fc0d8076530 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0007d70 tx=0x7fc0d000a340 comp rx=0 tx=0).stop 2024-05-27T07:11:46.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.117+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2563186162 shutdown_connections 2024-05-27T07:11:46.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.117+0000 7fc0dfa00700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc0c00b2900 0x7fc0c00b4e10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:46.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.117+0000 7fc0dfa00700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0d8109860 0x7fc0d8076530 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:46.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.117+0000 7fc0dfa00700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d81045e0 0x7fc0d8075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:46.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.117+0000 7fc0dfa00700 1 --2- 172.21.15.43:0/2563186162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0d8108f10 0x7fc0d8075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:46.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.117+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2563186162 >> 172.21.15.43:0/2563186162 conn(0x7fc0d8100030 msgr2=0x7fc0d810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:46.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.117+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2563186162 shutdown_connections 2024-05-27T07:11:46.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.117+0000 7fc0dfa00700 1 -- 172.21.15.43:0/2563186162 wait complete. 2024-05-27T07:11:46.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:46 smithi132 ceph-mon[35283]: pgmap v496: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:46.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:46 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1123952288' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:11:46.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:46 smithi179 ceph-mon[35888]: pgmap v496: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:46.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:46 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1123952288' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-27T07:11:46.645 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149811 got 111669149817 for osd.3 2024-05-27T07:11:46.645 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:46.749 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790052 got 158913790058 for osd.5 2024-05-27T07:11:46.750 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:47.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:47 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2563186162' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:11:47.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:47 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/288437106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:11:47.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:47 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2563186162' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:11:47.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:47 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/288437106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:11:47.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:47 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2563186162' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-27T07:11:47.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:47 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/288437106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-27T07:11:47.644 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574978 got 55834574984 for osd.1 2024-05-27T07:11:47.645 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:47.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.130+0000 7fc2f7200700 1 -- 172.21.15.43:0/152551908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f810e730 msgr2=0x7fc2f8110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:47.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.130+0000 7fc2f7200700 1 --2- 172.21.15.43:0/152551908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f810e730 0x7fc2f8110bd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000b0a0 tx=0x7fc2f000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.130+0000 7fc2f7200700 1 -- 172.21.15.43:0/152551908 shutdown_connections 2024-05-27T07:11:47.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.130+0000 7fc2f7200700 1 --2- 172.21.15.43:0/152551908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f810e730 0x7fc2f8110bd0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.130+0000 7fc2f7200700 1 --2- 172.21.15.43:0/152551908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2f8105300 0x7fc2f8105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.130+0000 7fc2f7200700 1 --2- 172.21.15.43:0/152551908 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2f8105c50 0x7fc2f810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.130+0000 7fc2f7200700 1 -- 172.21.15.43:0/152551908 >> 172.21.15.43:0/152551908 conn(0x7fc2f8078ab0 msgr2=0x7fc2f8078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:47.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.130+0000 7fc2f7200700 1 -- 172.21.15.43:0/152551908 shutdown_connections 2024-05-27T07:11:47.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.130+0000 7fc2f7200700 1 -- 172.21.15.43:0/152551908 wait complete. 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.131+0000 7fc2f7200700 1 Processor -- start 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.131+0000 7fc2f7200700 1 -- start start 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2f7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2f8105300 0x7fc2f81a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2f7200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2f8105c50 0x7fc2f81a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2f7200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f810e730 0x7fc2f819a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2f7200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f8114150 con 0x7fc2f810e730 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2f7200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f8113fd0 con 0x7fc2f8105300 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2f7200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81142d0 con 0x7fc2f8105c50 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2f5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2f8105300 0x7fc2f81a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2efe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2f8105c50 0x7fc2f81a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:47.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2efe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2f8105c50 0x7fc2f81a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:55688/0 (socket says 172.21.15.43:55688) 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2efe00700 1 -- 172.21.15.43:0/37221383 learned_addr learned my addr 172.21.15.43:0/37221383 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2f5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2f8105300 0x7fc2f81a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:32784/0 (socket says 172.21.15.43:32784) 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.132+0000 7fc2f6800700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f810e730 0x7fc2f819a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.133+0000 7fc2f6800700 1 -- 172.21.15.43:0/37221383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2f8105c50 msgr2=0x7fc2f81a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.133+0000 7fc2f6800700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2f8105c50 0x7fc2f81a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.133+0000 7fc2f6800700 1 -- 172.21.15.43:0/37221383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2f8105300 msgr2=0x7fc2f81a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.133+0000 7fc2f6800700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2f8105300 0x7fc2f81a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.133+0000 7fc2f6800700 1 -- 172.21.15.43:0/37221383 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f819afe0 con 0x7fc2f810e730 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.133+0000 7fc2f6800700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f810e730 0x7fc2f819a930 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000b070 tx=0x7fc2f0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.134+0000 7fc2eea00700 1 -- 172.21.15.43:0/37221383 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0021070 con 0x7fc2f810e730 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.134+0000 7fc2eea00700 1 -- 172.21.15.43:0/37221383 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2f0007b80 con 0x7fc2f810e730 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.134+0000 7fc2f7200700 1 -- 172.21.15.43:0/37221383 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f819b270 con 0x7fc2f810e730 2024-05-27T07:11:47.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.134+0000 7fc2eea00700 1 -- 172.21.15.43:0/37221383 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00052a0 con 0x7fc2f810e730 2024-05-27T07:11:47.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.134+0000 7fc2f7200700 1 -- 172.21.15.43:0/37221383 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f819b750 con 0x7fc2f810e730 2024-05-27T07:11:47.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.135+0000 7fc2f7200700 1 -- 172.21.15.43:0/37221383 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f80512b0 con 0x7fc2f810e730 2024-05-27T07:11:47.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.138+0000 7fc2eea00700 1 -- 172.21.15.43:0/37221383 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fc2f0024070 con 0x7fc2f810e730 2024-05-27T07:11:47.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.140+0000 7fc2eea00700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc2dc0b2880 0x7fc2dc0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:47.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.141+0000 7fc2eea00700 1 -- 172.21.15.43:0/37221383 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fc2f000f020 con 0x7fc2f810e730 2024-05-27T07:11:47.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.141+0000 7fc2f5e00700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc2dc0b2880 0x7fc2dc0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:47.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.141+0000 7fc2f5e00700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc2dc0b2880 0x7fc2dc0b4d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc2f819ba70 tx=0x7fc2e80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:47.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.145+0000 7fc2eea00700 1 -- 172.21.15.43:0/37221383 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc2f002b020 con 0x7fc2f810e730 2024-05-27T07:11:47.796 INFO:teuthology.orchestra.run.smithi043.stdout:279172874309 2024-05-27T07:11:47.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.388+0000 7fc2f7200700 1 -- 172.21.15.43:0/37221383 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fc2f802e5e0 con 0x7fc2f810e730 2024-05-27T07:11:47.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.390+0000 7fc2eea00700 1 -- 172.21.15.43:0/37221383 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fc2f00ab480 con 0x7fc2f810e730 2024-05-27T07:11:47.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.393+0000 7fc2f7200700 1 -- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc2dc0b2880 msgr2=0x7fc2dc0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:47.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.393+0000 7fc2f7200700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc2dc0b2880 0x7fc2dc0b4d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc2f819ba70 tx=0x7fc2e80073d0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.393+0000 7fc2f7200700 1 -- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f810e730 msgr2=0x7fc2f819a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:47.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.393+0000 7fc2f7200700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f810e730 0x7fc2f819a930 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000b070 tx=0x7fc2f0004c80 comp rx=0 tx=0).stop 2024-05-27T07:11:47.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.393+0000 7fc2f7200700 1 -- 172.21.15.43:0/37221383 shutdown_connections 2024-05-27T07:11:47.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.393+0000 7fc2f7200700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc2dc0b2880 0x7fc2dc0b4d90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.393+0000 7fc2f7200700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc2f810e730 0x7fc2f819a930 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.393+0000 7fc2f7200700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2f8105300 0x7fc2f81a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.393+0000 7fc2f7200700 1 --2- 172.21.15.43:0/37221383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2f8105c50 0x7fc2f81a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:47.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.393+0000 7fc2f7200700 1 -- 172.21.15.43:0/37221383 >> 172.21.15.43:0/37221383 conn(0x7fc2f8078ab0 msgr2=0x7fc2f80fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:47.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.394+0000 7fc2f7200700 1 -- 172.21.15.43:0/37221383 shutdown_connections 2024-05-27T07:11:47.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:47.394+0000 7fc2f7200700 1 -- 172.21.15.43:0/37221383 wait complete. 2024-05-27T07:11:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:48 smithi132 ceph-mon[35283]: pgmap v497: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:48 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/37221383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:11:48.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:48 smithi043 ceph-mon[31531]: pgmap v497: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:48.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:48 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/37221383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:11:48.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:48 smithi179 ceph-mon[35888]: pgmap v497: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:48.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:48 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/37221383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-27T07:11:49.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.191+0000 7f3998600700 1 -- 172.21.15.43:0/3268736794 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3990108730 msgr2=0x7f3990110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:49.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.191+0000 7f3998600700 1 --2- 172.21.15.43:0/3268736794 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3990108730 0x7f3990110090 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f398800b0d0 tx=0x7f398800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.191+0000 7f3998600700 1 -- 172.21.15.43:0/3268736794 shutdown_connections 2024-05-27T07:11:49.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.191+0000 7f3998600700 1 --2- 172.21.15.43:0/3268736794 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3990108730 0x7f3990110090 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.191+0000 7f3998600700 1 --2- 172.21.15.43:0/3268736794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f39901034b0 0x7f39901038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.191+0000 7f3998600700 1 --2- 172.21.15.43:0/3268736794 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3990107de0 0x7f39901081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.191+0000 7f3998600700 1 -- 172.21.15.43:0/3268736794 >> 172.21.15.43:0/3268736794 conn(0x7f39900fedb0 msgr2=0x7f39901011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.191+0000 7f3998600700 1 -- 172.21.15.43:0/3268736794 shutdown_connections 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.191+0000 7f3998600700 1 -- 172.21.15.43:0/3268736794 wait complete. 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.192+0000 7f3998600700 1 Processor -- start 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.193+0000 7f3998600700 1 -- start start 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.193+0000 7f3998600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f39901034b0 0x7f399019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.193+0000 7f3998600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3990107de0 0x7f399019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.193+0000 7f3998600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3990108730 0x7f39901a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.193+0000 7f3998600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3990113260 con 0x7f3990108730 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.193+0000 7f3998600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f39901130e0 con 0x7f39901034b0 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.193+0000 7f3998600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f39901133e0 con 0x7f3990107de0 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.193+0000 7f3996600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3990108730 0x7f39901a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:49.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.193+0000 7f3996600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3990108730 0x7f39901a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44150/0 (socket says 172.21.15.43:44150) 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.193+0000 7f3996600700 1 -- 172.21.15.43:0/288437106 learned_addr learned my addr 172.21.15.43:0/288437106 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f3995c00700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f39901034b0 0x7f399019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f3996600700 1 -- 172.21.15.43:0/288437106 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3990107de0 msgr2=0x7f399019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f3996600700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3990107de0 0x7f399019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f3996600700 1 -- 172.21.15.43:0/288437106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f39901034b0 msgr2=0x7f399019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f3996600700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f39901034b0 0x7f399019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f3996600700 1 -- 172.21.15.43:0/288437106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39901a1d10 con 0x7f3990108730 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f3995c00700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f39901034b0 0x7f399019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f3996600700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3990108730 0x7f39901a1720 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3988007d10 tx=0x7f3988007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f398e000700 1 -- 172.21.15.43:0/288437106 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398801f020 con 0x7f3990108730 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f398e000700 1 -- 172.21.15.43:0/288437106 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3988003e10 con 0x7f3990108730 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f3998600700 1 -- 172.21.15.43:0/288437106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39901a1fa0 con 0x7f3990108730 2024-05-27T07:11:49.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.194+0000 7f398e000700 1 -- 172.21.15.43:0/288437106 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398801bab0 con 0x7f3990108730 2024-05-27T07:11:49.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.195+0000 7f3998600700 1 -- 172.21.15.43:0/288437106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39901a24e0 con 0x7f3990108730 2024-05-27T07:11:49.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.196+0000 7f3998600700 1 -- 172.21.15.43:0/288437106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39900512b0 con 0x7f3990108730 2024-05-27T07:11:49.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.198+0000 7f398e000700 1 -- 172.21.15.43:0/288437106 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f3988022020 con 0x7f3990108730 2024-05-27T07:11:49.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.199+0000 7f398e000700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f39780b2880 0x7f39780b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:49.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.200+0000 7f398e000700 1 -- 172.21.15.43:0/288437106 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f398800f020 con 0x7f3990108730 2024-05-27T07:11:49.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.200+0000 7f3995c00700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f39780b2880 0x7f39780b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:49.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.200+0000 7f3995c00700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f39780b2880 0x7f39780b4d90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3984005fd0 tx=0x7f3984005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:49.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.204+0000 7f398e000700 1 -- 172.21.15.43:0/288437106 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f39880ab230 con 0x7f3990108730 2024-05-27T07:11:49.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.447+0000 7f3998600700 1 -- 172.21.15.43:0/288437106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f399002e5e0 con 0x7f3990108730 2024-05-27T07:11:49.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.448+0000 7f398e000700 1 -- 172.21.15.43:0/288437106 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f39880aab30 con 0x7f3990108730 2024-05-27T07:11:49.123 INFO:teuthology.orchestra.run.smithi043.stdout:137438953585 2024-05-27T07:11:49.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.451+0000 7f3998600700 1 -- 172.21.15.43:0/288437106 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f39780b2880 msgr2=0x7f39780b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:49.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.451+0000 7f3998600700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f39780b2880 0x7f39780b4d90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3984005fd0 tx=0x7f3984005dc0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.452+0000 7f3998600700 1 -- 172.21.15.43:0/288437106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3990108730 msgr2=0x7f39901a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:49.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.452+0000 7f3998600700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3990108730 0x7f39901a1720 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3988007d10 tx=0x7f3988007df0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.452+0000 7f3998600700 1 -- 172.21.15.43:0/288437106 shutdown_connections 2024-05-27T07:11:49.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.452+0000 7f3998600700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f39780b2880 0x7f39780b4d90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.452+0000 7f3998600700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3990108730 0x7f39901a1720 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.452+0000 7f3998600700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f39901034b0 0x7f399019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.452+0000 7f3998600700 1 --2- 172.21.15.43:0/288437106 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3990107de0 0x7f399019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:49.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.452+0000 7f3998600700 1 -- 172.21.15.43:0/288437106 >> 172.21.15.43:0/288437106 conn(0x7f39900fedb0 msgr2=0x7f399010ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:49.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.452+0000 7f3998600700 1 -- 172.21.15.43:0/288437106 shutdown_connections 2024-05-27T07:11:49.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:46.452+0000 7f3998600700 1 -- 172.21.15.43:0/288437106 wait complete. 2024-05-27T07:11:49.926 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234060 got 231928234067 for osd.8 2024-05-27T07:11:49.926 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:50.117 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:50 smithi043 ceph-mon[31531]: pgmap v498: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:50.173 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874302 got 279172874309 for osd.10 2024-05-27T07:11:50.173 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:50.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:50 smithi132 ceph-mon[35283]: pgmap v498: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:50 smithi179 ceph-mon[35888]: pgmap v498: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:50.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.897+0000 7f691be00700 1 -- 172.21.15.43:0/2021714353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f691410e8e0 msgr2=0x7f6914110dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:50.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.897+0000 7f691be00700 1 --2- 172.21.15.43:0/2021714353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f691410e8e0 0x7f6914110dc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f690c009990 tx=0x7f690c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:50.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.897+0000 7f691be00700 1 -- 172.21.15.43:0/2021714353 shutdown_connections 2024-05-27T07:11:50.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.897+0000 7f691be00700 1 --2- 172.21.15.43:0/2021714353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f691410e8e0 0x7f6914110dc0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:50.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.897+0000 7f691be00700 1 --2- 172.21.15.43:0/2021714353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914105470 0x7f6914105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:50.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.897+0000 7f691be00700 1 --2- 172.21.15.43:0/2021714353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6914105dc0 0x7f691410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:50.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.897+0000 7f691be00700 1 -- 172.21.15.43:0/2021714353 >> 172.21.15.43:0/2021714353 conn(0x7f69140fef20 msgr2=0x7f6914101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:50.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.897+0000 7f691be00700 1 -- 172.21.15.43:0/2021714353 shutdown_connections 2024-05-27T07:11:50.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.897+0000 7f691be00700 1 -- 172.21.15.43:0/2021714353 wait complete. 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.899+0000 7f691be00700 1 Processor -- start 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.899+0000 7f691be00700 1 -- start start 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.899+0000 7f691be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105470 0x7f69141a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.899+0000 7f691be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914105dc0 0x7f69141a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.899+0000 7f691be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410e8e0 0x7f691419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.900+0000 7f691be00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6914114310 con 0x7f6914105470 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.900+0000 7f691be00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6914114190 con 0x7f691410e8e0 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.900+0000 7f691be00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6914114490 con 0x7f6914105dc0 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.900+0000 7f6919400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105470 0x7f69141a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.900+0000 7f6919400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105470 0x7f69141a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44194/0 (socket says 172.21.15.43:44194) 2024-05-27T07:11:50.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.900+0000 7f6919400700 1 -- 172.21.15.43:0/3234998652 learned_addr learned my addr 172.21.15.43:0/3234998652 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:50.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.900+0000 7f6918a00700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914105dc0 0x7f69141a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:50.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.900+0000 7f6919e00700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410e8e0 0x7f691419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.901+0000 7f6919400700 1 -- 172.21.15.43:0/3234998652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914105dc0 msgr2=0x7f69141a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.901+0000 7f6919400700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914105dc0 0x7f69141a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.901+0000 7f6919400700 1 -- 172.21.15.43:0/3234998652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410e8e0 msgr2=0x7f691419aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.901+0000 7f6919400700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410e8e0 0x7f691419aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.901+0000 7f6919400700 1 -- 172.21.15.43:0/3234998652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f691419b1d0 con 0x7f6914105470 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.901+0000 7f6919400700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105470 0x7f69141a08b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6908007f10 tx=0x7f690800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.901+0000 7f6912000700 1 -- 172.21.15.43:0/3234998652 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690800dbd0 con 0x7f6914105470 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.901+0000 7f691be00700 1 -- 172.21.15.43:0/3234998652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f691419b4c0 con 0x7f6914105470 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.902+0000 7f691be00700 1 -- 172.21.15.43:0/3234998652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69141a7760 con 0x7f6914105470 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.902+0000 7f6912000700 1 -- 172.21.15.43:0/3234998652 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f690800dd70 con 0x7f6914105470 2024-05-27T07:11:50.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.902+0000 7f6912000700 1 -- 172.21.15.43:0/3234998652 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6908005950 con 0x7f6914105470 2024-05-27T07:11:50.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.903+0000 7f691be00700 1 -- 172.21.15.43:0/3234998652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f4005330 con 0x7f6914105470 2024-05-27T07:11:50.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.906+0000 7f6912000700 1 -- 172.21.15.43:0/3234998652 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f690801a020 con 0x7f6914105470 2024-05-27T07:11:50.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.907+0000 7f6912000700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f68fc0b2900 0x7f68fc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:50.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.908+0000 7f6912000700 1 -- 172.21.15.43:0/3234998652 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f69080e1c30 con 0x7f6914105470 2024-05-27T07:11:50.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.908+0000 7f6918a00700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f68fc0b2900 0x7f68fc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:50.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.909+0000 7f6918a00700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f68fc0b2900 0x7f68fc0b4e10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6914102420 tx=0x7f690000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:50.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:50.913+0000 7f6912000700 1 -- 172.21.15.43:0/3234998652 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6908010070 con 0x7f6914105470 2024-05-27T07:11:51.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.156+0000 7f691be00700 1 -- 172.21.15.43:0/3234998652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f68f4005620 con 0x7f6914105470 2024-05-27T07:11:51.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.156+0000 7f6912000700 1 -- 172.21.15.43:0/3234998652 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f69080a9560 con 0x7f6914105470 2024-05-27T07:11:51.147 INFO:teuthology.orchestra.run.smithi043.stdout:206158430301 2024-05-27T07:11:51.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.159+0000 7f691be00700 1 -- 172.21.15.43:0/3234998652 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f68fc0b2900 msgr2=0x7f68fc0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:51.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.159+0000 7f691be00700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f68fc0b2900 0x7f68fc0b4e10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6914102420 tx=0x7f690000b410 comp rx=0 tx=0).stop 2024-05-27T07:11:51.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.159+0000 7f691be00700 1 -- 172.21.15.43:0/3234998652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105470 msgr2=0x7f69141a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:51.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.159+0000 7f691be00700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105470 0x7f69141a08b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6908007f10 tx=0x7f690800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:11:51.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.160+0000 7f691be00700 1 -- 172.21.15.43:0/3234998652 shutdown_connections 2024-05-27T07:11:51.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.160+0000 7f691be00700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f68fc0b2900 0x7f68fc0b4e10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:51.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.160+0000 7f691be00700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410e8e0 0x7f691419aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:51.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.160+0000 7f691be00700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105470 0x7f69141a08b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:51.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.160+0000 7f691be00700 1 --2- 172.21.15.43:0/3234998652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914105dc0 0x7f69141a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:51.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.160+0000 7f691be00700 1 -- 172.21.15.43:0/3234998652 >> 172.21.15.43:0/3234998652 conn(0x7f69140fef20 msgr2=0x7f69140ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:51.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.161+0000 7f691be00700 1 -- 172.21.15.43:0/3234998652 shutdown_connections 2024-05-27T07:11:51.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:51.161+0000 7f691be00700 1 -- 172.21.15.43:0/3234998652 wait complete. 2024-05-27T07:11:51.196 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953579 got 137438953585 for osd.4 2024-05-27T07:11:51.197 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:51.993 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430294 got 206158430301 for osd.7 2024-05-27T07:11:51.993 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:52.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:52 smithi043 ceph-mon[31531]: pgmap v499: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:52.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:52 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3234998652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:11:52.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:52 smithi179 ceph-mon[35888]: pgmap v499: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:52.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:52 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3234998652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:11:52.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.571+0000 7fb317400700 1 -- 172.21.15.43:0/559420374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb310103bb0 msgr2=0x7fb3101080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:52.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.571+0000 7fb317400700 1 --2- 172.21.15.43:0/559420374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb310103bb0 0x7fb3101080e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009990 tx=0x7fb2fc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:52.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.571+0000 7fb317400700 1 -- 172.21.15.43:0/559420374 shutdown_connections 2024-05-27T07:11:52.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.571+0000 7fb317400700 1 --2- 172.21.15.43:0/559420374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3101086b0 0x7fb310108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:52.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.571+0000 7fb317400700 1 --2- 172.21.15.43:0/559420374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb310103260 0x7fb310103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:52.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.571+0000 7fb317400700 1 --2- 172.21.15.43:0/559420374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb310103bb0 0x7fb3101080e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:52.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.571+0000 7fb317400700 1 -- 172.21.15.43:0/559420374 >> 172.21.15.43:0/559420374 conn(0x7fb3100fef20 msgr2=0x7fb310101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:52.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.571+0000 7fb317400700 1 -- 172.21.15.43:0/559420374 shutdown_connections 2024-05-27T07:11:52.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.571+0000 7fb317400700 1 -- 172.21.15.43:0/559420374 wait complete. 2024-05-27T07:11:52.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.573+0000 7fb317400700 1 Processor -- start 2024-05-27T07:11:52.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.573+0000 7fb317400700 1 -- start start 2024-05-27T07:11:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.573+0000 7fb317400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb310103260 0x7fb3101a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.573+0000 7fb317400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb310103bb0 0x7fb3101a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.573+0000 7fb317400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3101086b0 0x7fb3101a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.573+0000 7fb317400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb31011a930 con 0x7fb3101086b0 2024-05-27T07:11:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.573+0000 7fb317400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb31011a7b0 con 0x7fb310103bb0 2024-05-27T07:11:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.573+0000 7fb317400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb31011aab0 con 0x7fb310103260 2024-05-27T07:11:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb315400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3101086b0 0x7fb3101a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb315400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3101086b0 0x7fb3101a5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44214/0 (socket says 172.21.15.43:44214) 2024-05-27T07:11:52.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb315400700 1 -- 172.21.15.43:0/869954341 learned_addr learned my addr 172.21.15.43:0/869954341 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:52.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb314a00700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb310103260 0x7fb3101a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:52.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb30fe00700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb310103bb0 0x7fb3101a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:52.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb315400700 1 -- 172.21.15.43:0/869954341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb310103260 msgr2=0x7fb3101a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:52.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb315400700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb310103260 0x7fb3101a1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:52.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb315400700 1 -- 172.21.15.43:0/869954341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb310103bb0 msgr2=0x7fb3101a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:52.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb315400700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb310103bb0 0x7fb3101a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:52.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb315400700 1 -- 172.21.15.43:0/869954341 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3101a6250 con 0x7fb3101086b0 2024-05-27T07:11:52.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.574+0000 7fb315400700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3101086b0 0x7fb3101a5bb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb30800ada0 tx=0x7fb30800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:52.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.575+0000 7fb30d600700 1 -- 172.21.15.43:0/869954341 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308017020 con 0x7fb3101086b0 2024-05-27T07:11:52.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.575+0000 7fb30d600700 1 -- 172.21.15.43:0/869954341 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb30800ce10 con 0x7fb3101086b0 2024-05-27T07:11:52.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.575+0000 7fb30d600700 1 -- 172.21.15.43:0/869954341 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308005420 con 0x7fb3101086b0 2024-05-27T07:11:52.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.575+0000 7fb317400700 1 -- 172.21.15.43:0/869954341 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3101a6540 con 0x7fb3101086b0 2024-05-27T07:11:52.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.575+0000 7fb317400700 1 -- 172.21.15.43:0/869954341 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3101ade60 con 0x7fb3101086b0 2024-05-27T07:11:52.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.576+0000 7fb317400700 1 -- 172.21.15.43:0/869954341 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb31006c910 con 0x7fb3101086b0 2024-05-27T07:11:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.580+0000 7fb30d600700 1 -- 172.21.15.43:0/869954341 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fb308004ad0 con 0x7fb3101086b0 2024-05-27T07:11:52.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.582+0000 7fb30d600700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb2f80b2880 0x7fb2f80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:52.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.582+0000 7fb30d600700 1 -- 172.21.15.43:0/869954341 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fb3080e2340 con 0x7fb3101086b0 2024-05-27T07:11:52.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.582+0000 7fb314a00700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb2f80b2880 0x7fb2f80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:52.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.583+0000 7fb314a00700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb2f80b2880 0x7fb2f80b4d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb304005fd0 tx=0x7fb30400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:52.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.584+0000 7fb30d600700 1 -- 172.21.15.43:0/869954341 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb308021020 con 0x7fb3101086b0 2024-05-27T07:11:52.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:52 smithi132 ceph-mon[35283]: pgmap v499: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:52.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:52 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3234998652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-27T07:11:52.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.828+0000 7fb317400700 1 -- 172.21.15.43:0/869954341 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb3101ae030 con 0x7fb3101086b0 2024-05-27T07:11:52.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.829+0000 7fb30d600700 1 -- 172.21.15.43:0/869954341 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb3080a9d60 con 0x7fb3101086b0 2024-05-27T07:11:52.820 INFO:teuthology.orchestra.run.smithi043.stdout:184683593829 2024-05-27T07:11:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.831+0000 7fb317400700 1 -- 172.21.15.43:0/869954341 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb2f80b2880 msgr2=0x7fb2f80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.831+0000 7fb317400700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb2f80b2880 0x7fb2f80b4d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb304005fd0 tx=0x7fb30400a560 comp rx=0 tx=0).stop 2024-05-27T07:11:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.832+0000 7fb317400700 1 -- 172.21.15.43:0/869954341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3101086b0 msgr2=0x7fb3101a5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.832+0000 7fb317400700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3101086b0 0x7fb3101a5bb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb30800ada0 tx=0x7fb30800c330 comp rx=0 tx=0).stop 2024-05-27T07:11:52.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.832+0000 7fb317400700 1 -- 172.21.15.43:0/869954341 shutdown_connections 2024-05-27T07:11:52.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.832+0000 7fb317400700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb2f80b2880 0x7fb2f80b4d90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:52.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.832+0000 7fb317400700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3101086b0 0x7fb3101a5bb0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:52.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.832+0000 7fb317400700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb310103260 0x7fb3101a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:52.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.832+0000 7fb317400700 1 --2- 172.21.15.43:0/869954341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb310103bb0 0x7fb3101a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.832+0000 7fb317400700 1 -- 172.21.15.43:0/869954341 >> 172.21.15.43:0/869954341 conn(0x7fb3100fef20 msgr2=0x7fb3101060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.832+0000 7fb317400700 1 -- 172.21.15.43:0/869954341 shutdown_connections 2024-05-27T07:11:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:52.832+0000 7fb317400700 1 -- 172.21.15.43:0/869954341 wait complete. 2024-05-27T07:11:53.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:53 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/869954341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:11:53.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:53 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/869954341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:11:53.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:53 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/869954341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-27T07:11:53.781 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593823 got 184683593829 for osd.6 2024-05-27T07:11:53.781 DEBUG:teuthology.parallel:result is None 2024-05-27T07:11:53.781 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-27T07:11:53.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:11:54.268 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:11:54.421 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:54 smithi043 ceph-mon[31531]: pgmap v500: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:11:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:54 smithi179 ceph-mon[35888]: pgmap v500: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:11:54.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:54 smithi132 ceph-mon[35283]: pgmap v500: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:11:55.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:55 smithi043 ceph-mon[31531]: pgmap v501: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:55 smithi179 ceph-mon[35888]: pgmap v501: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:55.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:55 smithi132 ceph-mon[35283]: pgmap v501: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:11:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.761+0000 7fbae1800700 1 -- 172.21.15.43:0/815223970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbadc10f6b0 msgr2=0x7fbadc10fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.761+0000 7fbae1800700 1 --2- 172.21.15.43:0/815223970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbadc10f6b0 0x7fbadc10fac0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009990 tx=0x7fbacc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:11:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.761+0000 7fbae1800700 1 -- 172.21.15.43:0/815223970 shutdown_connections 2024-05-27T07:11:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.761+0000 7fbae1800700 1 --2- 172.21.15.43:0/815223970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbadc10b100 0x7fbadc117610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.761+0000 7fbae1800700 1 --2- 172.21.15.43:0/815223970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbadc10f6b0 0x7fbadc10fac0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.761+0000 7fbae1800700 1 --2- 172.21.15.43:0/815223970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbadc10a7b0 0x7fbadc10abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.761+0000 7fbae1800700 1 -- 172.21.15.43:0/815223970 >> 172.21.15.43:0/815223970 conn(0x7fbadc076af0 msgr2=0x7fbadc076f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.761+0000 7fbae1800700 1 -- 172.21.15.43:0/815223970 shutdown_connections 2024-05-27T07:11:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.761+0000 7fbae1800700 1 -- 172.21.15.43:0/815223970 wait complete. 2024-05-27T07:11:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.763+0000 7fbae1800700 1 Processor -- start 2024-05-27T07:11:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.763+0000 7fbae1800700 1 -- start start 2024-05-27T07:11:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.763+0000 7fbae1800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbadc10a7b0 0x7fbadc1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.763+0000 7fbae1800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbadc10b100 0x7fbadc1a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.763+0000 7fbae1800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbadc10f6b0 0x7fbadc1a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.763+0000 7fbae1800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc11a750 con 0x7fbadc10f6b0 2024-05-27T07:11:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.763+0000 7fbae1800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc11a5d0 con 0x7fbadc10a7b0 2024-05-27T07:11:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.763+0000 7fbae1800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc11a8d0 con 0x7fbadc10b100 2024-05-27T07:11:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.764+0000 7fbae0e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbadc10f6b0 0x7fbadc1a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.764+0000 7fbae0e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbadc10f6b0 0x7fbadc1a5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54298/0 (socket says 172.21.15.43:54298) 2024-05-27T07:11:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.764+0000 7fbae0e00700 1 -- 172.21.15.43:0/4250583467 learned_addr learned my addr 172.21.15.43:0/4250583467 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:11:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.764+0000 7fbadbe00700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbadc10a7b0 0x7fbadc1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.764+0000 7fbad3400700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbadc10b100 0x7fbadc1a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.764+0000 7fbae0e00700 1 -- 172.21.15.43:0/4250583467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbadc10b100 msgr2=0x7fbadc1a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.764+0000 7fbae0e00700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbadc10b100 0x7fbadc1a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.764+0000 7fbae0e00700 1 -- 172.21.15.43:0/4250583467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbadc10a7b0 msgr2=0x7fbadc1a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.764+0000 7fbae0e00700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbadc10a7b0 0x7fbadc1a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.764+0000 7fbae0e00700 1 -- 172.21.15.43:0/4250583467 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc1a6010 con 0x7fbadc10f6b0 2024-05-27T07:11:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.765+0000 7fbae0e00700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbadc10f6b0 0x7fbadc1a5970 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbad4007ee0 tx=0x7fbad400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.765+0000 7fbad9600700 1 -- 172.21.15.43:0/4250583467 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad4004bb0 con 0x7fbadc10f6b0 2024-05-27T07:11:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.765+0000 7fbae1800700 1 -- 172.21.15.43:0/4250583467 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbadc1a6300 con 0x7fbadc10f6b0 2024-05-27T07:11:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.765+0000 7fbad9600700 1 -- 172.21.15.43:0/4250583467 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbad4004d50 con 0x7fbadc10f6b0 2024-05-27T07:11:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.765+0000 7fbad9600700 1 -- 172.21.15.43:0/4250583467 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad40058b0 con 0x7fbadc10f6b0 2024-05-27T07:11:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.765+0000 7fbae1800700 1 -- 172.21.15.43:0/4250583467 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbadc1adc20 con 0x7fbadc10f6b0 2024-05-27T07:11:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.767+0000 7fbae1800700 1 -- 172.21.15.43:0/4250583467 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbadc0512b0 con 0x7fbadc10f6b0 2024-05-27T07:11:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.771+0000 7fbad9600700 1 -- 172.21.15.43:0/4250583467 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fbad401a020 con 0x7fbadc10f6b0 2024-05-27T07:11:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.772+0000 7fbad9600700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbac00b2900 0x7fbac00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:11:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.773+0000 7fbad9600700 1 -- 172.21.15.43:0/4250583467 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fbad40e21e0 con 0x7fbadc10f6b0 2024-05-27T07:11:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.773+0000 7fbadbe00700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbac00b2900 0x7fbac00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:11:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.773+0000 7fbadbe00700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbac00b2900 0x7fbac00b4e10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00b5c0 tx=0x7fbacc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:11:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:56.775+0000 7fbad9600700 1 -- 172.21.15.43:0/4250583467 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbad4010070 con 0x7fbadc10f6b0 2024-05-27T07:11:57.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.020+0000 7fbae1800700 1 -- 172.21.15.43:0/4250583467 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbadc063bf0 con 0x7fbac00b2900 2024-05-27T07:11:57.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.033+0000 7fbad9600700 1 -- 172.21.15.43:0/4250583467 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340512 (secure 0 0 0) 0x7fbacc07b230 con 0x7fbac00b2900 2024-05-27T07:11:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:11:57.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 -- 172.21.15.43:0/4250583467 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbac00b2900 msgr2=0x7fbac00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:57.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbac00b2900 0x7fbac00b4e10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00b5c0 tx=0x7fbacc005960 comp rx=0 tx=0).stop 2024-05-27T07:11:57.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 -- 172.21.15.43:0/4250583467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbadc10f6b0 msgr2=0x7fbadc1a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:11:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbadc10f6b0 0x7fbadc1a5970 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbad4007ee0 tx=0x7fbad400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:11:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 -- 172.21.15.43:0/4250583467 shutdown_connections 2024-05-27T07:11:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fbac00b2900 0x7fbac00b4e10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbadc10f6b0 0x7fbadc1a5970 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbadc10a7b0 0x7fbadc1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 --2- 172.21.15.43:0/4250583467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbadc10b100 0x7fbadc1a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:11:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 -- 172.21.15.43:0/4250583467 >> 172.21.15.43:0/4250583467 conn(0x7fbadc076af0 msgr2=0x7fbadc108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:11:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.040+0000 7fbae1800700 1 -- 172.21.15.43:0/4250583467 shutdown_connections 2024-05-27T07:11:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:11:57.041+0000 7fbae1800700 1 -- 172.21.15.43:0/4250583467 wait complete. 2024-05-27T07:11:57.035 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-05-27T07:11:58.004 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-05-27T07:11:56.833556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662473,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1338,"num_read_kb":778,"num_write":383,"num_write_kb":2302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953452,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":816,"ondisk_log_size":816,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367648,"kb_used_data":14336,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574176,"statfs":{"total":1151940427776,"available":1151563956224,"internally_reserved":0,"allocated":14680064,"data_stored":8180033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":130,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006982"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714202+0000","last_change":"2024-05-27T07:08:22.053540+0000","last_active":"2024-05-27T07:11:22.714202+0000","last_peered":"2024-05-27T07:11:22.714202+0000","last_clean":"2024-05-27T07:11:22.714202+0000","last_became_active":"2024-05-27T07:08:22.051872+0000","last_became_peered":"2024-05-27T07:08:22.051872+0000","last_unstale":"2024-05-27T07:11:22.714202+0000","last_undegraded":"2024-05-27T07:11:22.714202+0000","last_fullsized":"2024-05-27T07:11:22.714202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:02.787957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.700741+0000","last_change":"2024-05-27T07:08:22.049341+0000","last_active":"2024-05-27T07:11:22.700741+0000","last_peered":"2024-05-27T07:11:22.700741+0000","last_clean":"2024-05-27T07:11:22.700741+0000","last_became_active":"2024-05-27T07:08:22.048905+0000","last_became_peered":"2024-05-27T07:08:22.048905+0000","last_unstale":"2024-05-27T07:11:22.700741+0000","last_undegraded":"2024-05-27T07:11:22.700741+0000","last_fullsized":"2024-05-27T07:11:22.700741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:27.271336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.713108+0000","last_change":"2024-05-27T07:08:22.053925+0000","last_active":"2024-05-27T07:11:22.713108+0000","last_peered":"2024-05-27T07:11:22.713108+0000","last_clean":"2024-05-27T07:11:22.713108+0000","last_became_active":"2024-05-27T07:08:22.053409+0000","last_became_peered":"2024-05-27T07:08:22.053409+0000","last_unstale":"2024-05-27T07:11:22.713108+0000","last_undegraded":"2024-05-27T07:11:22.713108+0000","last_fullsized":"2024-05-27T07:11:22.713108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:07:55.439966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.696388+0000","last_change":"2024-05-27T07:08:22.048214+0000","last_active":"2024-05-27T07:11:22.696388+0000","last_peered":"2024-05-27T07:11:22.696388+0000","last_clean":"2024-05-27T07:11:22.696388+0000","last_became_active":"2024-05-27T07:08:22.047791+0000","last_became_peered":"2024-05-27T07:08:22.047791+0000","last_unstale":"2024-05-27T07:11:22.696388+0000","last_undegraded":"2024-05-27T07:11:22.696388+0000","last_fullsized":"2024-05-27T07:11:22.696388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:23:56.381008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.712561+0000","last_change":"2024-05-27T07:08:22.037883+0000","last_active":"2024-05-27T07:11:22.712561+0000","last_peered":"2024-05-27T07:11:22.712561+0000","last_clean":"2024-05-27T07:11:22.712561+0000","last_became_active":"2024-05-27T07:08:22.037089+0000","last_became_peered":"2024-05-27T07:08:22.037089+0000","last_unstale":"2024-05-27T07:11:22.712561+0000","last_undegraded":"2024-05-27T07:11:22.712561+0000","last_fullsized":"2024-05-27T07:11:22.712561+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:21.929086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.693343+0000","last_change":"2024-05-27T07:08:22.061833+0000","last_active":"2024-05-27T07:11:22.693343+0000","last_peered":"2024-05-27T07:11:22.693343+0000","last_clean":"2024-05-27T07:11:22.693343+0000","last_became_active":"2024-05-27T07:08:22.057692+0000","last_became_peered":"2024-05-27T07:08:22.057692+0000","last_unstale":"2024-05-27T07:11:22.693343+0000","last_undegraded":"2024-05-27T07:11:22.693343+0000","last_fullsized":"2024-05-27T07:11:22.693343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:38:33.656872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.690332+0000","last_change":"2024-05-27T07:08:22.047880+0000","last_active":"2024-05-27T07:11:22.690332+0000","last_peered":"2024-05-27T07:11:22.690332+0000","last_clean":"2024-05-27T07:11:22.690332+0000","last_became_active":"2024-05-27T07:08:22.047482+0000","last_became_peered":"2024-05-27T07:08:22.047482+0000","last_unstale":"2024-05-27T07:11:22.690332+0000","last_undegraded":"2024-05-27T07:11:22.690332+0000","last_fullsized":"2024-05-27T07:11:22.690332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:07.751588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.711214+0000","last_change":"2024-05-27T07:08:22.055970+0000","last_active":"2024-05-27T07:11:22.711214+0000","last_peered":"2024-05-27T07:11:22.711214+0000","last_clean":"2024-05-27T07:11:22.711214+0000","last_became_active":"2024-05-27T07:08:22.055591+0000","last_became_peered":"2024-05-27T07:08:22.055591+0000","last_unstale":"2024-05-27T07:11:22.711214+0000","last_undegraded":"2024-05-27T07:11:22.711214+0000","last_fullsized":"2024-05-27T07:11:22.711214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:58:46.571473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.715285+0000","last_change":"2024-05-27T07:08:22.059929+0000","last_active":"2024-05-27T07:11:22.715285+0000","last_peered":"2024-05-27T07:11:22.715285+0000","last_clean":"2024-05-27T07:11:22.715285+0000","last_became_active":"2024-05-27T07:08:22.059118+0000","last_became_peered":"2024-05-27T07:08:22.059118+0000","last_unstale":"2024-05-27T07:11:22.715285+0000","last_undegraded":"2024-05-27T07:11:22.715285+0000","last_fullsized":"2024-05-27T07:11:22.715285+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:29.231220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.700504+0000","last_change":"2024-05-27T07:08:53.979269+0000","last_active":"2024-05-27T07:11:22.700504+0000","last_peered":"2024-05-27T07:11:22.700504+0000","last_clean":"2024-05-27T07:11:22.700504+0000","last_became_active":"2024-05-27T07:08:53.978433+0000","last_became_peered":"2024-05-27T07:08:53.978433+0000","last_unstale":"2024-05-27T07:11:22.700504+0000","last_undegraded":"2024-05-27T07:11:22.700504+0000","last_fullsized":"2024-05-27T07:11:22.700504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:57:37.600911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.713893+0000","last_change":"2024-05-27T07:08:22.053499+0000","last_active":"2024-05-27T07:11:22.713893+0000","last_peered":"2024-05-27T07:11:22.713893+0000","last_clean":"2024-05-27T07:11:22.713893+0000","last_became_active":"2024-05-27T07:08:22.052428+0000","last_became_peered":"2024-05-27T07:08:22.052428+0000","last_unstale":"2024-05-27T07:11:22.713893+0000","last_undegraded":"2024-05-27T07:11:22.713893+0000","last_fullsized":"2024-05-27T07:11:22.713893+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:04.352262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.699790+0000","last_change":"2024-05-27T07:08:22.058882+0000","last_active":"2024-05-27T07:11:22.699790+0000","last_peered":"2024-05-27T07:11:22.699790+0000","last_clean":"2024-05-27T07:11:22.699790+0000","last_became_active":"2024-05-27T07:08:22.057874+0000","last_became_peered":"2024-05-27T07:08:22.057874+0000","last_unstale":"2024-05-27T07:11:22.699790+0000","last_undegraded":"2024-05-27T07:11:22.699790+0000","last_fullsized":"2024-05-27T07:11:22.699790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:52:11.289488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706679+0000","last_change":"2024-05-27T07:08:22.054315+0000","last_active":"2024-05-27T07:11:22.706679+0000","last_peered":"2024-05-27T07:11:22.706679+0000","last_clean":"2024-05-27T07:11:22.706679+0000","last_became_active":"2024-05-27T07:08:22.053944+0000","last_became_peered":"2024-05-27T07:08:22.053944+0000","last_unstale":"2024-05-27T07:11:22.706679+0000","last_undegraded":"2024-05-27T07:11:22.706679+0000","last_fullsized":"2024-05-27T07:11:22.706679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:43:06.720147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706279+0000","last_change":"2024-05-27T07:08:22.054649+0000","last_active":"2024-05-27T07:11:22.706279+0000","last_peered":"2024-05-27T07:11:22.706279+0000","last_clean":"2024-05-27T07:11:22.706279+0000","last_became_active":"2024-05-27T07:08:22.051834+0000","last_became_peered":"2024-05-27T07:08:22.051834+0000","last_unstale":"2024-05-27T07:11:22.706279+0000","last_undegraded":"2024-05-27T07:11:22.706279+0000","last_fullsized":"2024-05-27T07:11:22.706279+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:16:56.893458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.705614+0000","last_change":"2024-05-27T07:08:22.052231+0000","last_active":"2024-05-27T07:11:22.705614+0000","last_peered":"2024-05-27T07:11:22.705614+0000","last_clean":"2024-05-27T07:11:22.705614+0000","last_became_active":"2024-05-27T07:08:22.051495+0000","last_became_peered":"2024-05-27T07:08:22.051495+0000","last_unstale":"2024-05-27T07:11:22.705614+0000","last_undegraded":"2024-05-27T07:11:22.705614+0000","last_fullsized":"2024-05-27T07:11:22.705614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:30:21.036874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714436+0000","last_change":"2024-05-27T07:08:22.056847+0000","last_active":"2024-05-27T07:11:22.714436+0000","last_peered":"2024-05-27T07:11:22.714436+0000","last_clean":"2024-05-27T07:11:22.714436+0000","last_became_active":"2024-05-27T07:08:22.056127+0000","last_became_peered":"2024-05-27T07:08:22.056127+0000","last_unstale":"2024-05-27T07:11:22.714436+0000","last_undegraded":"2024-05-27T07:11:22.714436+0000","last_fullsized":"2024-05-27T07:11:22.714436+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:04.757452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.716087+0000","last_change":"2024-05-27T07:08:22.058078+0000","last_active":"2024-05-27T07:11:22.716087+0000","last_peered":"2024-05-27T07:11:22.716087+0000","last_clean":"2024-05-27T07:11:22.716087+0000","last_became_active":"2024-05-27T07:08:22.056886+0000","last_became_peered":"2024-05-27T07:08:22.056886+0000","last_unstale":"2024-05-27T07:11:22.716087+0000","last_undegraded":"2024-05-27T07:11:22.716087+0000","last_fullsized":"2024-05-27T07:11:22.716087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.104572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.705110+0000","last_change":"2024-05-27T07:08:22.051492+0000","last_active":"2024-05-27T07:11:22.705110+0000","last_peered":"2024-05-27T07:11:22.705110+0000","last_clean":"2024-05-27T07:11:22.705110+0000","last_became_active":"2024-05-27T07:08:22.051091+0000","last_became_peered":"2024-05-27T07:08:22.051091+0000","last_unstale":"2024-05-27T07:11:22.705110+0000","last_undegraded":"2024-05-27T07:11:22.705110+0000","last_fullsized":"2024-05-27T07:11:22.705110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:47.816312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.712229+0000","last_change":"2024-05-27T07:08:22.050714+0000","last_active":"2024-05-27T07:11:22.712229+0000","last_peered":"2024-05-27T07:11:22.712229+0000","last_clean":"2024-05-27T07:11:22.712229+0000","last_became_active":"2024-05-27T07:08:22.050322+0000","last_became_peered":"2024-05-27T07:08:22.050322+0000","last_unstale":"2024-05-27T07:11:22.712229+0000","last_undegraded":"2024-05-27T07:11:22.712229+0000","last_fullsized":"2024-05-27T07:11:22.712229+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:01.867467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070420+0000","last_change":"2024-05-27T07:08:36.859916+0000","last_active":"2024-05-27T07:09:54.070420+0000","last_peered":"2024-05-27T07:09:54.070420+0000","last_clean":"2024-05-27T07:09:54.070420+0000","last_became_active":"2024-05-27T07:08:36.859429+0000","last_became_peered":"2024-05-27T07:08:36.859429+0000","last_unstale":"2024-05-27T07:09:54.070420+0000","last_undegraded":"2024-05-27T07:09:54.070420+0000","last_fullsized":"2024-05-27T07:09:54.070420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:14.026604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":179,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:51.406880+0000","last_change":"2024-05-27T07:08:18.008912+0000","last_active":"2024-05-27T07:11:51.406880+0000","last_peered":"2024-05-27T07:11:51.406880+0000","last_clean":"2024-05-27T07:11:51.406880+0000","last_became_active":"2024-05-27T07:08:18.007603+0000","last_became_peered":"2024-05-27T07:08:18.007603+0000","last_unstale":"2024-05-27T07:11:51.406880+0000","last_undegraded":"2024-05-27T07:11:51.406880+0000","last_fullsized":"2024-05-27T07:11:51.406880+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:17:40.681754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.711671+0000","last_change":"2024-05-27T07:08:22.055426+0000","last_active":"2024-05-27T07:11:22.711671+0000","last_peered":"2024-05-27T07:11:22.711671+0000","last_clean":"2024-05-27T07:11:22.711671+0000","last_became_active":"2024-05-27T07:08:22.055063+0000","last_became_peered":"2024-05-27T07:08:22.055063+0000","last_unstale":"2024-05-27T07:11:22.711671+0000","last_undegraded":"2024-05-27T07:11:22.711671+0000","last_fullsized":"2024-05-27T07:11:22.711671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:06:49.923192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079599+0000","last_change":"2024-05-27T07:08:53.982756+0000","last_active":"2024-05-27T07:09:54.079599+0000","last_peered":"2024-05-27T07:09:54.079599+0000","last_clean":"2024-05-27T07:09:54.079599+0000","last_became_active":"2024-05-27T07:08:53.981816+0000","last_became_peered":"2024-05-27T07:08:53.981816+0000","last_unstale":"2024-05-27T07:09:54.079599+0000","last_undegraded":"2024-05-27T07:09:54.079599+0000","last_fullsized":"2024-05-27T07:09:54.079599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:09.984488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063371+0000","last_change":"2024-05-27T07:08:18.307659+0000","last_active":"2024-05-27T07:09:54.063371+0000","last_peered":"2024-05-27T07:09:54.063371+0000","last_clean":"2024-05-27T07:09:54.063371+0000","last_became_active":"2024-05-27T07:08:18.307239+0000","last_became_peered":"2024-05-27T07:08:18.307239+0000","last_unstale":"2024-05-27T07:09:54.063371+0000","last_undegraded":"2024-05-27T07:09:54.063371+0000","last_fullsized":"2024-05-27T07:09:54.063371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:09:02.816813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.713140+0000","last_change":"2024-05-27T07:08:22.050403+0000","last_active":"2024-05-27T07:11:22.713140+0000","last_peered":"2024-05-27T07:11:22.713140+0000","last_clean":"2024-05-27T07:11:22.713140+0000","last_became_active":"2024-05-27T07:08:22.049939+0000","last_became_peered":"2024-05-27T07:08:22.049939+0000","last_unstale":"2024-05-27T07:11:22.713140+0000","last_undegraded":"2024-05-27T07:11:22.713140+0000","last_fullsized":"2024-05-27T07:11:22.713140+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:45.857508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.078966+0000","last_change":"2024-05-27T07:08:36.864458+0000","last_active":"2024-05-27T07:09:54.078966+0000","last_peered":"2024-05-27T07:09:54.078966+0000","last_clean":"2024-05-27T07:09:54.078966+0000","last_became_active":"2024-05-27T07:08:36.863894+0000","last_became_peered":"2024-05-27T07:08:36.863894+0000","last_unstale":"2024-05-27T07:09:54.078966+0000","last_undegraded":"2024-05-27T07:09:54.078966+0000","last_fullsized":"2024-05-27T07:09:54.078966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:49.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190406+0000","last_change":"2024-05-27T07:08:18.016051+0000","last_active":"2024-05-27T07:09:54.190406+0000","last_peered":"2024-05-27T07:09:54.190406+0000","last_clean":"2024-05-27T07:09:54.190406+0000","last_became_active":"2024-05-27T07:08:18.015590+0000","last_became_peered":"2024-05-27T07:08:18.015590+0000","last_unstale":"2024-05-27T07:09:54.190406+0000","last_undegraded":"2024-05-27T07:09:54.190406+0000","last_fullsized":"2024-05-27T07:09:54.190406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:33.396027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.715250+0000","last_change":"2024-05-27T07:08:22.055603+0000","last_active":"2024-05-27T07:11:22.715250+0000","last_peered":"2024-05-27T07:11:22.715250+0000","last_clean":"2024-05-27T07:11:22.715250+0000","last_became_active":"2024-05-27T07:08:22.054543+0000","last_became_peered":"2024-05-27T07:08:22.054543+0000","last_unstale":"2024-05-27T07:11:22.715250+0000","last_undegraded":"2024-05-27T07:11:22.715250+0000","last_fullsized":"2024-05-27T07:11:22.715250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:29:01.738598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.078533+0000","last_change":"2024-05-27T07:08:36.870358+0000","last_active":"2024-05-27T07:09:54.078533+0000","last_peered":"2024-05-27T07:09:54.078533+0000","last_clean":"2024-05-27T07:09:54.078533+0000","last_became_active":"2024-05-27T07:08:36.869457+0000","last_became_peered":"2024-05-27T07:08:36.869457+0000","last_unstale":"2024-05-27T07:09:54.078533+0000","last_undegraded":"2024-05-27T07:09:54.078533+0000","last_fullsized":"2024-05-27T07:09:54.078533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:50.510098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187633+0000","last_change":"2024-05-27T07:08:53.985562+0000","last_active":"2024-05-27T07:09:54.187633+0000","last_peered":"2024-05-27T07:09:54.187633+0000","last_clean":"2024-05-27T07:09:54.187633+0000","last_became_active":"2024-05-27T07:08:53.977778+0000","last_became_peered":"2024-05-27T07:08:53.977778+0000","last_unstale":"2024-05-27T07:09:54.187633+0000","last_undegraded":"2024-05-27T07:09:54.187633+0000","last_fullsized":"2024-05-27T07:09:54.187633+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:44.455508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706977+0000","last_change":"2024-05-27T07:08:22.057622+0000","last_active":"2024-05-27T07:11:22.706977+0000","last_peered":"2024-05-27T07:11:22.706977+0000","last_clean":"2024-05-27T07:11:22.706977+0000","last_became_active":"2024-05-27T07:08:22.056544+0000","last_became_peered":"2024-05-27T07:08:22.056544+0000","last_unstale":"2024-05-27T07:11:22.706977+0000","last_undegraded":"2024-05-27T07:11:22.706977+0000","last_fullsized":"2024-05-27T07:11:22.706977+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:38.484560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.071085+0000","last_change":"2024-05-27T07:08:36.864984+0000","last_active":"2024-05-27T07:09:54.071085+0000","last_peered":"2024-05-27T07:09:54.071085+0000","last_clean":"2024-05-27T07:09:54.071085+0000","last_became_active":"2024-05-27T07:08:36.864302+0000","last_became_peered":"2024-05-27T07:08:36.864302+0000","last_unstale":"2024-05-27T07:09:54.071085+0000","last_undegraded":"2024-05-27T07:09:54.071085+0000","last_fullsized":"2024-05-27T07:09:54.071085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:44.183562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":174,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:53.423807+0000","last_change":"2024-05-27T07:08:18.017918+0000","last_active":"2024-05-27T07:11:53.423807+0000","last_peered":"2024-05-27T07:11:53.423807+0000","last_clean":"2024-05-27T07:11:53.423807+0000","last_became_active":"2024-05-27T07:08:18.016174+0000","last_became_peered":"2024-05-27T07:08:18.016174+0000","last_unstale":"2024-05-27T07:11:53.423807+0000","last_undegraded":"2024-05-27T07:11:53.423807+0000","last_fullsized":"2024-05-27T07:11:53.423807+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:21.906187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.697882+0000","last_change":"2024-05-27T07:08:22.043172+0000","last_active":"2024-05-27T07:11:22.697882+0000","last_peered":"2024-05-27T07:11:22.697882+0000","last_clean":"2024-05-27T07:11:22.697882+0000","last_became_active":"2024-05-27T07:08:22.041399+0000","last_became_peered":"2024-05-27T07:08:22.041399+0000","last_unstale":"2024-05-27T07:11:22.697882+0000","last_undegraded":"2024-05-27T07:11:22.697882+0000","last_fullsized":"2024-05-27T07:11:22.697882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:19.651330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070394+0000","last_change":"2024-05-27T07:08:36.866352+0000","last_active":"2024-05-27T07:09:54.070394+0000","last_peered":"2024-05-27T07:09:54.070394+0000","last_clean":"2024-05-27T07:09:54.070394+0000","last_became_active":"2024-05-27T07:08:36.865447+0000","last_became_peered":"2024-05-27T07:08:36.865447+0000","last_unstale":"2024-05-27T07:09:54.070394+0000","last_undegraded":"2024-05-27T07:09:54.070394+0000","last_fullsized":"2024-05-27T07:09:54.070394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:55:42.634374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063044+0000","last_change":"2024-05-27T07:08:18.022142+0000","last_active":"2024-05-27T07:09:54.063044+0000","last_peered":"2024-05-27T07:09:54.063044+0000","last_clean":"2024-05-27T07:09:54.063044+0000","last_became_active":"2024-05-27T07:08:18.021453+0000","last_became_peered":"2024-05-27T07:08:18.021453+0000","last_unstale":"2024-05-27T07:09:54.063044+0000","last_undegraded":"2024-05-27T07:09:54.063044+0000","last_fullsized":"2024-05-27T07:09:54.063044+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:01:09.049440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714060+0000","last_change":"2024-05-27T07:08:22.055742+0000","last_active":"2024-05-27T07:11:22.714060+0000","last_peered":"2024-05-27T07:11:22.714060+0000","last_clean":"2024-05-27T07:11:22.714060+0000","last_became_active":"2024-05-27T07:08:22.052077+0000","last_became_peered":"2024-05-27T07:08:22.052077+0000","last_unstale":"2024-05-27T07:11:22.714060+0000","last_undegraded":"2024-05-27T07:11:22.714060+0000","last_fullsized":"2024-05-27T07:11:22.714060+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:09:13.418027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188119+0000","last_change":"2024-05-27T07:08:36.874225+0000","last_active":"2024-05-27T07:09:54.188119+0000","last_peered":"2024-05-27T07:09:54.188119+0000","last_clean":"2024-05-27T07:09:54.188119+0000","last_became_active":"2024-05-27T07:08:36.873745+0000","last_became_peered":"2024-05-27T07:08:36.873745+0000","last_unstale":"2024-05-27T07:09:54.188119+0000","last_undegraded":"2024-05-27T07:09:54.188119+0000","last_fullsized":"2024-05-27T07:09:54.188119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:40.633667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041025+0000","last_change":"2024-05-27T07:08:18.009678+0000","last_active":"2024-05-27T07:09:54.041025+0000","last_peered":"2024-05-27T07:09:54.041025+0000","last_clean":"2024-05-27T07:09:54.041025+0000","last_became_active":"2024-05-27T07:08:18.006502+0000","last_became_peered":"2024-05-27T07:08:18.006502+0000","last_unstale":"2024-05-27T07:09:54.041025+0000","last_undegraded":"2024-05-27T07:09:54.041025+0000","last_fullsized":"2024-05-27T07:09:54.041025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:37.457044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.713616+0000","last_change":"2024-05-27T07:08:22.047302+0000","last_active":"2024-05-27T07:11:22.713616+0000","last_peered":"2024-05-27T07:11:22.713616+0000","last_clean":"2024-05-27T07:11:22.713616+0000","last_became_active":"2024-05-27T07:08:22.045932+0000","last_became_peered":"2024-05-27T07:08:22.045932+0000","last_unstale":"2024-05-27T07:11:22.713616+0000","last_undegraded":"2024-05-27T07:11:22.713616+0000","last_fullsized":"2024-05-27T07:11:22.713616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:27.129539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185409+0000","last_change":"2024-05-27T07:08:36.870512+0000","last_active":"2024-05-27T07:09:54.185409+0000","last_peered":"2024-05-27T07:09:54.185409+0000","last_clean":"2024-05-27T07:09:54.185409+0000","last_became_active":"2024-05-27T07:08:36.870061+0000","last_became_peered":"2024-05-27T07:08:36.870061+0000","last_unstale":"2024-05-27T07:09:54.185409+0000","last_undegraded":"2024-05-27T07:09:54.185409+0000","last_fullsized":"2024-05-27T07:09:54.185409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:03.608758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042222+0000","last_change":"2024-05-27T07:08:18.018420+0000","last_active":"2024-05-27T07:09:54.042222+0000","last_peered":"2024-05-27T07:09:54.042222+0000","last_clean":"2024-05-27T07:09:54.042222+0000","last_became_active":"2024-05-27T07:08:18.017567+0000","last_became_peered":"2024-05-27T07:08:18.017567+0000","last_unstale":"2024-05-27T07:09:54.042222+0000","last_undegraded":"2024-05-27T07:09:54.042222+0000","last_fullsized":"2024-05-27T07:09:54.042222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:37.214408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.713166+0000","last_change":"2024-05-27T07:08:22.050943+0000","last_active":"2024-05-27T07:11:22.713166+0000","last_peered":"2024-05-27T07:11:22.713166+0000","last_clean":"2024-05-27T07:11:22.713166+0000","last_became_active":"2024-05-27T07:08:22.047706+0000","last_became_peered":"2024-05-27T07:08:22.047706+0000","last_unstale":"2024-05-27T07:11:22.713166+0000","last_undegraded":"2024-05-27T07:11:22.713166+0000","last_fullsized":"2024-05-27T07:11:22.713166+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:52:09.739528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060574+0000","last_change":"2024-05-27T07:08:36.867487+0000","last_active":"2024-05-27T07:09:54.060574+0000","last_peered":"2024-05-27T07:09:54.060574+0000","last_clean":"2024-05-27T07:09:54.060574+0000","last_became_active":"2024-05-27T07:08:36.867032+0000","last_became_peered":"2024-05-27T07:08:36.867032+0000","last_unstale":"2024-05-27T07:09:54.060574+0000","last_undegraded":"2024-05-27T07:09:54.060574+0000","last_fullsized":"2024-05-27T07:09:54.060574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:26:24.912382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":294,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:54.425519+0000","last_change":"2024-05-27T07:08:18.020168+0000","last_active":"2024-05-27T07:11:54.425519+0000","last_peered":"2024-05-27T07:11:54.425519+0000","last_clean":"2024-05-27T07:11:54.425519+0000","last_became_active":"2024-05-27T07:08:18.018309+0000","last_became_peered":"2024-05-27T07:08:18.018309+0000","last_unstale":"2024-05-27T07:11:54.425519+0000","last_undegraded":"2024-05-27T07:11:54.425519+0000","last_fullsized":"2024-05-27T07:11:54.425519+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:48:16.839510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.715433+0000","last_change":"2024-05-27T07:08:22.054888+0000","last_active":"2024-05-27T07:11:22.715433+0000","last_peered":"2024-05-27T07:11:22.715433+0000","last_clean":"2024-05-27T07:11:22.715433+0000","last_became_active":"2024-05-27T07:08:22.054417+0000","last_became_peered":"2024-05-27T07:08:22.054417+0000","last_unstale":"2024-05-27T07:11:22.715433+0000","last_undegraded":"2024-05-27T07:11:22.715433+0000","last_fullsized":"2024-05-27T07:11:22.715433+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:43:00.866755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.047889+0000","last_change":"2024-05-27T07:08:36.854516+0000","last_active":"2024-05-27T07:09:54.047889+0000","last_peered":"2024-05-27T07:09:54.047889+0000","last_clean":"2024-05-27T07:09:54.047889+0000","last_became_active":"2024-05-27T07:08:36.853978+0000","last_became_peered":"2024-05-27T07:08:36.853978+0000","last_unstale":"2024-05-27T07:09:54.047889+0000","last_undegraded":"2024-05-27T07:09:54.047889+0000","last_fullsized":"2024-05-27T07:09:54.047889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:45.299100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045529+0000","last_change":"2024-05-27T07:08:18.010539+0000","last_active":"2024-05-27T07:09:54.045529+0000","last_peered":"2024-05-27T07:09:54.045529+0000","last_clean":"2024-05-27T07:09:54.045529+0000","last_became_active":"2024-05-27T07:08:18.009643+0000","last_became_peered":"2024-05-27T07:08:18.009643+0000","last_unstale":"2024-05-27T07:09:54.045529+0000","last_undegraded":"2024-05-27T07:09:54.045529+0000","last_fullsized":"2024-05-27T07:09:54.045529+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:48:40.335403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706673+0000","last_change":"2024-05-27T07:08:53.980165+0000","last_active":"2024-05-27T07:11:22.706673+0000","last_peered":"2024-05-27T07:11:22.706673+0000","last_clean":"2024-05-27T07:11:22.706673+0000","last_became_active":"2024-05-27T07:08:53.978565+0000","last_became_peered":"2024-05-27T07:08:53.978565+0000","last_unstale":"2024-05-27T07:11:22.706673+0000","last_undegraded":"2024-05-27T07:11:22.706673+0000","last_fullsized":"2024-05-27T07:11:22.706673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:56:33.513961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079609+0000","last_change":"2024-05-27T07:08:36.857814+0000","last_active":"2024-05-27T07:09:54.079609+0000","last_peered":"2024-05-27T07:09:54.079609+0000","last_clean":"2024-05-27T07:09:54.079609+0000","last_became_active":"2024-05-27T07:08:36.856790+0000","last_became_peered":"2024-05-27T07:08:36.856790+0000","last_unstale":"2024-05-27T07:09:54.079609+0000","last_undegraded":"2024-05-27T07:09:54.079609+0000","last_fullsized":"2024-05-27T07:09:54.079609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:39:10.322261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043549+0000","last_change":"2024-05-27T07:08:18.308032+0000","last_active":"2024-05-27T07:09:54.043549+0000","last_peered":"2024-05-27T07:09:54.043549+0000","last_clean":"2024-05-27T07:09:54.043549+0000","last_became_active":"2024-05-27T07:08:18.306867+0000","last_became_peered":"2024-05-27T07:08:18.306867+0000","last_unstale":"2024-05-27T07:09:54.043549+0000","last_undegraded":"2024-05-27T07:09:54.043549+0000","last_fullsized":"2024-05-27T07:09:54.043549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:37.816237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714964+0000","last_change":"2024-05-27T07:08:22.057940+0000","last_active":"2024-05-27T07:11:22.714964+0000","last_peered":"2024-05-27T07:11:22.714964+0000","last_clean":"2024-05-27T07:11:22.714964+0000","last_became_active":"2024-05-27T07:08:22.055109+0000","last_became_peered":"2024-05-27T07:08:22.055109+0000","last_unstale":"2024-05-27T07:11:22.714964+0000","last_undegraded":"2024-05-27T07:11:22.714964+0000","last_fullsized":"2024-05-27T07:11:22.714964+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:46.905064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062722+0000","last_change":"2024-05-27T07:08:53.980209+0000","last_active":"2024-05-27T07:09:54.062722+0000","last_peered":"2024-05-27T07:09:54.062722+0000","last_clean":"2024-05-27T07:09:54.062722+0000","last_became_active":"2024-05-27T07:08:53.979353+0000","last_became_peered":"2024-05-27T07:08:53.979353+0000","last_unstale":"2024-05-27T07:09:54.062722+0000","last_undegraded":"2024-05-27T07:09:54.062722+0000","last_fullsized":"2024-05-27T07:09:54.062722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:06.163563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"90'38","reported_seq":353,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:54.444743+0000","last_change":"2024-05-27T07:08:53.996491+0000","last_active":"2024-05-27T07:11:54.444743+0000","last_peered":"2024-05-27T07:11:54.444743+0000","last_clean":"2024-05-27T07:11:54.444743+0000","last_became_active":"2024-05-27T07:08:53.976013+0000","last_became_peered":"2024-05-27T07:08:53.976013+0000","last_unstale":"2024-05-27T07:11:54.444743+0000","last_undegraded":"2024-05-27T07:11:54.444743+0000","last_fullsized":"2024-05-27T07:11:54.444743+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:44:16.592270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64385,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":254,"num_read_kb":190,"num_write":40,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.691765+0000","last_change":"2024-05-27T07:08:22.057231+0000","last_active":"2024-05-27T07:11:22.691765+0000","last_peered":"2024-05-27T07:11:22.691765+0000","last_clean":"2024-05-27T07:11:22.691765+0000","last_became_active":"2024-05-27T07:08:22.056837+0000","last_became_peered":"2024-05-27T07:08:22.056837+0000","last_unstale":"2024-05-27T07:11:22.691765+0000","last_undegraded":"2024-05-27T07:11:22.691765+0000","last_fullsized":"2024-05-27T07:11:22.691765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:10:28.645103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044424+0000","last_change":"2024-05-27T07:08:36.870307+0000","last_active":"2024-05-27T07:09:54.044424+0000","last_peered":"2024-05-27T07:09:54.044424+0000","last_clean":"2024-05-27T07:09:54.044424+0000","last_became_active":"2024-05-27T07:08:36.869390+0000","last_became_peered":"2024-05-27T07:08:36.869390+0000","last_unstale":"2024-05-27T07:09:54.044424+0000","last_undegraded":"2024-05-27T07:09:54.044424+0000","last_fullsized":"2024-05-27T07:09:54.044424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:43:45.228885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070799+0000","last_change":"2024-05-27T07:08:18.014458+0000","last_active":"2024-05-27T07:09:54.070799+0000","last_peered":"2024-05-27T07:09:54.070799+0000","last_clean":"2024-05-27T07:09:54.070799+0000","last_became_active":"2024-05-27T07:08:18.013365+0000","last_became_peered":"2024-05-27T07:08:18.013365+0000","last_unstale":"2024-05-27T07:09:54.070799+0000","last_undegraded":"2024-05-27T07:09:54.070799+0000","last_fullsized":"2024-05-27T07:09:54.070799+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:41.352618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.705768+0000","last_change":"2024-05-27T07:08:22.043354+0000","last_active":"2024-05-27T07:11:22.705768+0000","last_peered":"2024-05-27T07:11:22.705768+0000","last_clean":"2024-05-27T07:11:22.705768+0000","last_became_active":"2024-05-27T07:08:22.042876+0000","last_became_peered":"2024-05-27T07:08:22.042876+0000","last_unstale":"2024-05-27T07:11:22.705768+0000","last_undegraded":"2024-05-27T07:11:22.705768+0000","last_fullsized":"2024-05-27T07:11:22.705768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:52.575568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188013+0000","last_change":"2024-05-27T07:08:36.869579+0000","last_active":"2024-05-27T07:09:54.188013+0000","last_peered":"2024-05-27T07:09:54.188013+0000","last_clean":"2024-05-27T07:09:54.188013+0000","last_became_active":"2024-05-27T07:08:36.869055+0000","last_became_peered":"2024-05-27T07:08:36.869055+0000","last_unstale":"2024-05-27T07:09:54.188013+0000","last_undegraded":"2024-05-27T07:09:54.188013+0000","last_fullsized":"2024-05-27T07:09:54.188013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.870079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044142+0000","last_change":"2024-05-27T07:08:18.015322+0000","last_active":"2024-05-27T07:09:54.044142+0000","last_peered":"2024-05-27T07:09:54.044142+0000","last_clean":"2024-05-27T07:09:54.044142+0000","last_became_active":"2024-05-27T07:08:18.014535+0000","last_became_peered":"2024-05-27T07:08:18.014535+0000","last_unstale":"2024-05-27T07:09:54.044142+0000","last_undegraded":"2024-05-27T07:09:54.044142+0000","last_fullsized":"2024-05-27T07:09:54.044142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:35.116140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.703585+0000","last_change":"2024-05-27T07:08:22.056181+0000","last_active":"2024-05-27T07:11:22.703585+0000","last_peered":"2024-05-27T07:11:22.703585+0000","last_clean":"2024-05-27T07:11:22.703585+0000","last_became_active":"2024-05-27T07:08:22.055755+0000","last_became_peered":"2024-05-27T07:08:22.055755+0000","last_unstale":"2024-05-27T07:11:22.703585+0000","last_undegraded":"2024-05-27T07:11:22.703585+0000","last_fullsized":"2024-05-27T07:11:22.703585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:38:35.634116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.048450+0000","last_change":"2024-05-27T07:08:36.863184+0000","last_active":"2024-05-27T07:09:54.048450+0000","last_peered":"2024-05-27T07:09:54.048450+0000","last_clean":"2024-05-27T07:09:54.048450+0000","last_became_active":"2024-05-27T07:08:36.862760+0000","last_became_peered":"2024-05-27T07:08:36.862760+0000","last_unstale":"2024-05-27T07:09:54.048450+0000","last_undegraded":"2024-05-27T07:09:54.048450+0000","last_fullsized":"2024-05-27T07:09:54.048450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:48.379157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":174,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:50.443882+0000","last_change":"2024-05-27T07:08:18.019119+0000","last_active":"2024-05-27T07:11:50.443882+0000","last_peered":"2024-05-27T07:11:50.443882+0000","last_clean":"2024-05-27T07:11:50.443882+0000","last_became_active":"2024-05-27T07:08:18.018408+0000","last_became_peered":"2024-05-27T07:08:18.018408+0000","last_unstale":"2024-05-27T07:11:50.443882+0000","last_undegraded":"2024-05-27T07:11:50.443882+0000","last_fullsized":"2024-05-27T07:11:50.443882+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:19.978328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.698925+0000","last_change":"2024-05-27T07:08:22.059464+0000","last_active":"2024-05-27T07:11:22.698925+0000","last_peered":"2024-05-27T07:11:22.698925+0000","last_clean":"2024-05-27T07:11:22.698925+0000","last_became_active":"2024-05-27T07:08:22.058890+0000","last_became_peered":"2024-05-27T07:08:22.058890+0000","last_unstale":"2024-05-27T07:11:22.698925+0000","last_undegraded":"2024-05-27T07:11:22.698925+0000","last_fullsized":"2024-05-27T07:11:22.698925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:30.122552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041973+0000","last_change":"2024-05-27T07:08:36.865884+0000","last_active":"2024-05-27T07:09:54.041973+0000","last_peered":"2024-05-27T07:09:54.041973+0000","last_clean":"2024-05-27T07:09:54.041973+0000","last_became_active":"2024-05-27T07:08:36.865443+0000","last_became_peered":"2024-05-27T07:08:36.865443+0000","last_unstale":"2024-05-27T07:09:54.041973+0000","last_undegraded":"2024-05-27T07:09:54.041973+0000","last_fullsized":"2024-05-27T07:09:54.041973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:47:15.026632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187097+0000","last_change":"2024-05-27T07:08:53.994809+0000","last_active":"2024-05-27T07:09:54.187097+0000","last_peered":"2024-05-27T07:09:54.187097+0000","last_clean":"2024-05-27T07:09:54.187097+0000","last_became_active":"2024-05-27T07:08:53.977628+0000","last_became_peered":"2024-05-27T07:08:53.977628+0000","last_unstale":"2024-05-27T07:09:54.187097+0000","last_undegraded":"2024-05-27T07:09:54.187097+0000","last_fullsized":"2024-05-27T07:09:54.187097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:13:37.176319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706591+0000","last_change":"2024-05-27T07:08:22.050733+0000","last_active":"2024-05-27T07:11:22.706591+0000","last_peered":"2024-05-27T07:11:22.706591+0000","last_clean":"2024-05-27T07:11:22.706591+0000","last_became_active":"2024-05-27T07:08:22.050321+0000","last_became_peered":"2024-05-27T07:08:22.050321+0000","last_unstale":"2024-05-27T07:11:22.706591+0000","last_undegraded":"2024-05-27T07:11:22.706591+0000","last_fullsized":"2024-05-27T07:11:22.706591+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:44.590595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043754+0000","last_change":"2024-05-27T07:08:36.867784+0000","last_active":"2024-05-27T07:09:54.043754+0000","last_peered":"2024-05-27T07:09:54.043754+0000","last_clean":"2024-05-27T07:09:54.043754+0000","last_became_active":"2024-05-27T07:08:36.867290+0000","last_became_peered":"2024-05-27T07:08:36.867290+0000","last_unstale":"2024-05-27T07:09:54.043754+0000","last_undegraded":"2024-05-27T07:09:54.043754+0000","last_fullsized":"2024-05-27T07:09:54.043754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:44:07.649640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190834+0000","last_change":"2024-05-27T07:08:18.013193+0000","last_active":"2024-05-27T07:09:54.190834+0000","last_peered":"2024-05-27T07:09:54.190834+0000","last_clean":"2024-05-27T07:09:54.190834+0000","last_became_active":"2024-05-27T07:08:18.011767+0000","last_became_peered":"2024-05-27T07:08:18.011767+0000","last_unstale":"2024-05-27T07:09:54.190834+0000","last_undegraded":"2024-05-27T07:09:54.190834+0000","last_fullsized":"2024-05-27T07:09:54.190834+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:15:03.679719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.691219+0000","last_change":"2024-05-27T07:08:22.067145+0000","last_active":"2024-05-27T07:11:22.691219+0000","last_peered":"2024-05-27T07:11:22.691219+0000","last_clean":"2024-05-27T07:11:22.691219+0000","last_became_active":"2024-05-27T07:08:22.066750+0000","last_became_peered":"2024-05-27T07:08:22.066750+0000","last_unstale":"2024-05-27T07:11:22.691219+0000","last_undegraded":"2024-05-27T07:11:22.691219+0000","last_fullsized":"2024-05-27T07:11:22.691219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:41:35.333311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189235+0000","last_change":"2024-05-27T07:08:53.982497+0000","last_active":"2024-05-27T07:09:54.189235+0000","last_peered":"2024-05-27T07:09:54.189235+0000","last_clean":"2024-05-27T07:09:54.189235+0000","last_became_active":"2024-05-27T07:08:53.981920+0000","last_became_peered":"2024-05-27T07:08:53.981920+0000","last_unstale":"2024-05-27T07:09:54.189235+0000","last_undegraded":"2024-05-27T07:09:54.189235+0000","last_fullsized":"2024-05-27T07:09:54.189235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:43:40.095357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.048681+0000","last_change":"2024-05-27T07:08:18.308330+0000","last_active":"2024-05-27T07:09:54.048681+0000","last_peered":"2024-05-27T07:09:54.048681+0000","last_clean":"2024-05-27T07:09:54.048681+0000","last_became_active":"2024-05-27T07:08:18.306961+0000","last_became_peered":"2024-05-27T07:08:18.306961+0000","last_unstale":"2024-05-27T07:09:54.048681+0000","last_undegraded":"2024-05-27T07:09:54.048681+0000","last_fullsized":"2024-05-27T07:09:54.048681+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:27.875936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.690952+0000","last_change":"2024-05-27T07:08:22.047562+0000","last_active":"2024-05-27T07:11:22.690952+0000","last_peered":"2024-05-27T07:11:22.690952+0000","last_clean":"2024-05-27T07:11:22.690952+0000","last_became_active":"2024-05-27T07:08:22.046977+0000","last_became_peered":"2024-05-27T07:08:22.046977+0000","last_unstale":"2024-05-27T07:11:22.690952+0000","last_undegraded":"2024-05-27T07:11:22.690952+0000","last_fullsized":"2024-05-27T07:11:22.690952+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:49:43.376617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041089+0000","last_change":"2024-05-27T07:08:36.867382+0000","last_active":"2024-05-27T07:09:54.041089+0000","last_peered":"2024-05-27T07:09:54.041089+0000","last_clean":"2024-05-27T07:09:54.041089+0000","last_became_active":"2024-05-27T07:08:36.866296+0000","last_became_peered":"2024-05-27T07:08:36.866296+0000","last_unstale":"2024-05-27T07:09:54.041089+0000","last_undegraded":"2024-05-27T07:09:54.041089+0000","last_fullsized":"2024-05-27T07:09:54.041089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:14.305169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":173,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:54.418036+0000","last_change":"2024-05-27T07:08:18.012031+0000","last_active":"2024-05-27T07:11:54.418036+0000","last_peered":"2024-05-27T07:11:54.418036+0000","last_clean":"2024-05-27T07:11:54.418036+0000","last_became_active":"2024-05-27T07:08:18.009593+0000","last_became_peered":"2024-05-27T07:08:18.009593+0000","last_unstale":"2024-05-27T07:11:54.418036+0000","last_undegraded":"2024-05-27T07:11:54.418036+0000","last_fullsized":"2024-05-27T07:11:54.418036+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:25:22.316575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.693495+0000","last_change":"2024-05-27T07:08:22.061136+0000","last_active":"2024-05-27T07:11:22.693495+0000","last_peered":"2024-05-27T07:11:22.693495+0000","last_clean":"2024-05-27T07:11:22.693495+0000","last_became_active":"2024-05-27T07:08:22.060735+0000","last_became_peered":"2024-05-27T07:08:22.060735+0000","last_unstale":"2024-05-27T07:11:22.693495+0000","last_undegraded":"2024-05-27T07:11:22.693495+0000","last_fullsized":"2024-05-27T07:11:22.693495+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:50.392130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041856+0000","last_change":"2024-05-27T07:08:36.932187+0000","last_active":"2024-05-27T07:09:54.041856+0000","last_peered":"2024-05-27T07:09:54.041856+0000","last_clean":"2024-05-27T07:09:54.041856+0000","last_became_active":"2024-05-27T07:08:36.931214+0000","last_became_peered":"2024-05-27T07:08:36.931214+0000","last_unstale":"2024-05-27T07:09:54.041856+0000","last_undegraded":"2024-05-27T07:09:54.041856+0000","last_fullsized":"2024-05-27T07:09:54.041856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:20:59.645352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.689976+0000","last_change":"2024-05-27T07:09:54.082674+0000","last_active":"2024-05-27T07:11:22.689976+0000","last_peered":"2024-05-27T07:11:22.689976+0000","last_clean":"2024-05-27T07:11:22.689976+0000","last_became_active":"2024-05-27T07:09:54.081616+0000","last_became_peered":"2024-05-27T07:09:54.081616+0000","last_unstale":"2024-05-27T07:11:22.689976+0000","last_undegraded":"2024-05-27T07:11:22.689976+0000","last_fullsized":"2024-05-27T07:11:22.689976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:38:11.230026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":173,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:52.448719+0000","last_change":"2024-05-27T07:08:18.306439+0000","last_active":"2024-05-27T07:11:52.448719+0000","last_peered":"2024-05-27T07:11:52.448719+0000","last_clean":"2024-05-27T07:11:52.448719+0000","last_became_active":"2024-05-27T07:08:18.305978+0000","last_became_peered":"2024-05-27T07:08:18.305978+0000","last_unstale":"2024-05-27T07:11:52.448719+0000","last_undegraded":"2024-05-27T07:11:52.448719+0000","last_fullsized":"2024-05-27T07:11:52.448719+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:20:22.648922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:06.623537+0000","last_change":"2024-05-27T07:08:36.866247+0000","last_active":"2024-05-27T07:10:06.623537+0000","last_peered":"2024-05-27T07:10:06.623537+0000","last_clean":"2024-05-27T07:10:06.623537+0000","last_became_active":"2024-05-27T07:08:36.865516+0000","last_became_peered":"2024-05-27T07:08:36.865516+0000","last_unstale":"2024-05-27T07:10:06.623537+0000","last_undegraded":"2024-05-27T07:10:06.623537+0000","last_fullsized":"2024-05-27T07:10:06.623537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:39:41.235352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.716330+0000","last_change":"2024-05-27T07:08:22.045312+0000","last_active":"2024-05-27T07:11:22.716330+0000","last_peered":"2024-05-27T07:11:22.716330+0000","last_clean":"2024-05-27T07:11:22.716330+0000","last_became_active":"2024-05-27T07:08:22.044724+0000","last_became_peered":"2024-05-27T07:08:22.044724+0000","last_unstale":"2024-05-27T07:11:22.716330+0000","last_undegraded":"2024-05-27T07:11:22.716330+0000","last_fullsized":"2024-05-27T07:11:22.716330+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:05.487794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079152+0000","last_change":"2024-05-27T07:08:18.019333+0000","last_active":"2024-05-27T07:09:54.079152+0000","last_peered":"2024-05-27T07:09:54.079152+0000","last_clean":"2024-05-27T07:09:54.079152+0000","last_became_active":"2024-05-27T07:08:18.018816+0000","last_became_peered":"2024-05-27T07:08:18.018816+0000","last_unstale":"2024-05-27T07:09:54.079152+0000","last_undegraded":"2024-05-27T07:09:54.079152+0000","last_fullsized":"2024-05-27T07:09:54.079152+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:09.421346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044348+0000","last_change":"2024-05-27T07:06:16.865587+0000","last_active":"2024-05-27T07:09:54.044348+0000","last_peered":"2024-05-27T07:09:54.044348+0000","last_clean":"2024-05-27T07:09:54.044348+0000","last_became_active":"2024-05-27T07:06:16.850646+0000","last_became_peered":"2024-05-27T07:06:16.850646+0000","last_unstale":"2024-05-27T07:09:54.044348+0000","last_undegraded":"2024-05-27T07:09:54.044348+0000","last_fullsized":"2024-05-27T07:09:54.044348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_clean_scrub_stamp":"2024-05-27T07:01:14.931400+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:37.272619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043690+0000","last_change":"2024-05-27T07:08:36.934696+0000","last_active":"2024-05-27T07:09:54.043690+0000","last_peered":"2024-05-27T07:09:54.043690+0000","last_clean":"2024-05-27T07:09:54.043690+0000","last_became_active":"2024-05-27T07:08:36.934008+0000","last_became_peered":"2024-05-27T07:08:36.934008+0000","last_unstale":"2024-05-27T07:09:54.043690+0000","last_undegraded":"2024-05-27T07:09:54.043690+0000","last_fullsized":"2024-05-27T07:09:54.043690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:39.099180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.701045+0000","last_change":"2024-05-27T07:08:22.054479+0000","last_active":"2024-05-27T07:11:22.701045+0000","last_peered":"2024-05-27T07:11:22.701045+0000","last_clean":"2024-05-27T07:11:22.701045+0000","last_became_active":"2024-05-27T07:08:22.054127+0000","last_became_peered":"2024-05-27T07:08:22.054127+0000","last_unstale":"2024-05-27T07:11:22.701045+0000","last_undegraded":"2024-05-27T07:11:22.701045+0000","last_fullsized":"2024-05-27T07:11:22.701045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:37:25.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045671+0000","last_change":"2024-05-27T07:08:18.010944+0000","last_active":"2024-05-27T07:09:54.045671+0000","last_peered":"2024-05-27T07:09:54.045671+0000","last_clean":"2024-05-27T07:09:54.045671+0000","last_became_active":"2024-05-27T07:08:18.009816+0000","last_became_peered":"2024-05-27T07:08:18.009816+0000","last_unstale":"2024-05-27T07:09:54.045671+0000","last_undegraded":"2024-05-27T07:09:54.045671+0000","last_fullsized":"2024-05-27T07:09:54.045671+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:34:26.061787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187569+0000","last_change":"2024-05-27T07:08:36.870618+0000","last_active":"2024-05-27T07:09:54.187569+0000","last_peered":"2024-05-27T07:09:54.187569+0000","last_clean":"2024-05-27T07:09:54.187569+0000","last_became_active":"2024-05-27T07:08:36.869799+0000","last_became_peered":"2024-05-27T07:08:36.869799+0000","last_unstale":"2024-05-27T07:09:54.187569+0000","last_undegraded":"2024-05-27T07:09:54.187569+0000","last_fullsized":"2024-05-27T07:09:54.187569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:53.094337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190088+0000","last_change":"2024-05-27T07:08:18.308834+0000","last_active":"2024-05-27T07:09:54.190088+0000","last_peered":"2024-05-27T07:09:54.190088+0000","last_clean":"2024-05-27T07:09:54.190088+0000","last_became_active":"2024-05-27T07:08:18.306953+0000","last_became_peered":"2024-05-27T07:08:18.306953+0000","last_unstale":"2024-05-27T07:09:54.190088+0000","last_undegraded":"2024-05-27T07:09:54.190088+0000","last_fullsized":"2024-05-27T07:09:54.190088+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:26:14.747631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.701750+0000","last_change":"2024-05-27T07:08:22.048219+0000","last_active":"2024-05-27T07:11:22.701750+0000","last_peered":"2024-05-27T07:11:22.701750+0000","last_clean":"2024-05-27T07:11:22.701750+0000","last_became_active":"2024-05-27T07:08:22.047848+0000","last_became_peered":"2024-05-27T07:08:22.047848+0000","last_unstale":"2024-05-27T07:11:22.701750+0000","last_undegraded":"2024-05-27T07:11:22.701750+0000","last_fullsized":"2024-05-27T07:11:22.701750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:51.707603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060451+0000","last_change":"2024-05-27T07:08:36.932747+0000","last_active":"2024-05-27T07:09:54.060451+0000","last_peered":"2024-05-27T07:09:54.060451+0000","last_clean":"2024-05-27T07:09:54.060451+0000","last_became_active":"2024-05-27T07:08:36.932050+0000","last_became_peered":"2024-05-27T07:08:36.932050+0000","last_unstale":"2024-05-27T07:09:54.060451+0000","last_undegraded":"2024-05-27T07:09:54.060451+0000","last_fullsized":"2024-05-27T07:09:54.060451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:26:07.015651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'16","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:27.880470+0000","last_change":"2024-05-27T07:08:18.008844+0000","last_active":"2024-05-27T07:10:27.880470+0000","last_peered":"2024-05-27T07:10:27.880470+0000","last_clean":"2024-05-27T07:10:27.880470+0000","last_became_active":"2024-05-27T07:08:18.008140+0000","last_became_peered":"2024-05-27T07:08:18.008140+0000","last_unstale":"2024-05-27T07:10:27.880470+0000","last_undegraded":"2024-05-27T07:10:27.880470+0000","last_fullsized":"2024-05-27T07:10:27.880470+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:22:22.075341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714727+0000","last_change":"2024-05-27T07:08:22.050855+0000","last_active":"2024-05-27T07:11:22.714727+0000","last_peered":"2024-05-27T07:11:22.714727+0000","last_clean":"2024-05-27T07:11:22.714727+0000","last_became_active":"2024-05-27T07:08:22.048482+0000","last_became_peered":"2024-05-27T07:08:22.048482+0000","last_unstale":"2024-05-27T07:11:22.714727+0000","last_undegraded":"2024-05-27T07:11:22.714727+0000","last_fullsized":"2024-05-27T07:11:22.714727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:04.319884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.069731+0000","last_change":"2024-05-27T07:08:36.862470+0000","last_active":"2024-05-27T07:09:54.069731+0000","last_peered":"2024-05-27T07:09:54.069731+0000","last_clean":"2024-05-27T07:09:54.069731+0000","last_became_active":"2024-05-27T07:08:36.861951+0000","last_became_peered":"2024-05-27T07:08:36.861951+0000","last_unstale":"2024-05-27T07:09:54.069731+0000","last_undegraded":"2024-05-27T07:09:54.069731+0000","last_fullsized":"2024-05-27T07:09:54.069731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:56:31.659951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.080450+0000","last_change":"2024-05-27T07:08:18.438518+0000","last_active":"2024-05-27T07:09:54.080450+0000","last_peered":"2024-05-27T07:09:54.080450+0000","last_clean":"2024-05-27T07:09:54.080450+0000","last_became_active":"2024-05-27T07:08:18.437588+0000","last_became_peered":"2024-05-27T07:08:18.437588+0000","last_unstale":"2024-05-27T07:09:54.080450+0000","last_undegraded":"2024-05-27T07:09:54.080450+0000","last_fullsized":"2024-05-27T07:09:54.080450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:52:12.895009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706881+0000","last_change":"2024-05-27T07:08:22.064188+0000","last_active":"2024-05-27T07:11:22.706881+0000","last_peered":"2024-05-27T07:11:22.706881+0000","last_clean":"2024-05-27T07:11:22.706881+0000","last_became_active":"2024-05-27T07:08:22.063868+0000","last_became_peered":"2024-05-27T07:08:22.063868+0000","last_unstale":"2024-05-27T07:11:22.706881+0000","last_undegraded":"2024-05-27T07:11:22.706881+0000","last_fullsized":"2024-05-27T07:11:22.706881+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:43:17.227079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079354+0000","last_change":"2024-05-27T07:08:36.934922+0000","last_active":"2024-05-27T07:09:54.079354+0000","last_peered":"2024-05-27T07:09:54.079354+0000","last_clean":"2024-05-27T07:09:54.079354+0000","last_became_active":"2024-05-27T07:08:36.934000+0000","last_became_peered":"2024-05-27T07:08:36.934000+0000","last_unstale":"2024-05-27T07:09:54.079354+0000","last_undegraded":"2024-05-27T07:09:54.079354+0000","last_fullsized":"2024-05-27T07:09:54.079354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:19:00.570557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187967+0000","last_change":"2024-05-27T07:08:18.308743+0000","last_active":"2024-05-27T07:09:54.187967+0000","last_peered":"2024-05-27T07:09:54.187967+0000","last_clean":"2024-05-27T07:09:54.187967+0000","last_became_active":"2024-05-27T07:08:18.307597+0000","last_became_peered":"2024-05-27T07:08:18.307597+0000","last_unstale":"2024-05-27T07:09:54.187967+0000","last_undegraded":"2024-05-27T07:09:54.187967+0000","last_fullsized":"2024-05-27T07:09:54.187967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:39.914998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.645960+0000","last_change":"2024-05-27T07:08:22.049132+0000","last_active":"2024-05-27T07:11:22.645960+0000","last_peered":"2024-05-27T07:11:22.645960+0000","last_clean":"2024-05-27T07:11:22.645960+0000","last_became_active":"2024-05-27T07:08:22.048606+0000","last_became_peered":"2024-05-27T07:08:22.048606+0000","last_unstale":"2024-05-27T07:11:22.645960+0000","last_undegraded":"2024-05-27T07:11:22.645960+0000","last_fullsized":"2024-05-27T07:11:22.645960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:58.253461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.049257+0000","last_change":"2024-05-27T07:08:36.867286+0000","last_active":"2024-05-27T07:09:54.049257+0000","last_peered":"2024-05-27T07:09:54.049257+0000","last_clean":"2024-05-27T07:09:54.049257+0000","last_became_active":"2024-05-27T07:08:36.866732+0000","last_became_peered":"2024-05-27T07:08:36.866732+0000","last_unstale":"2024-05-27T07:09:54.049257+0000","last_undegraded":"2024-05-27T07:09:54.049257+0000","last_fullsized":"2024-05-27T07:09:54.049257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:14:18.386441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060788+0000","last_change":"2024-05-27T07:08:18.010026+0000","last_active":"2024-05-27T07:09:54.060788+0000","last_peered":"2024-05-27T07:09:54.060788+0000","last_clean":"2024-05-27T07:09:54.060788+0000","last_became_active":"2024-05-27T07:08:18.009502+0000","last_became_peered":"2024-05-27T07:08:18.009502+0000","last_unstale":"2024-05-27T07:09:54.060788+0000","last_undegraded":"2024-05-27T07:09:54.060788+0000","last_fullsized":"2024-05-27T07:09:54.060788+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:28:27.500933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.699081+0000","last_change":"2024-05-27T07:08:22.054536+0000","last_active":"2024-05-27T07:11:22.699081+0000","last_peered":"2024-05-27T07:11:22.699081+0000","last_clean":"2024-05-27T07:11:22.699081+0000","last_became_active":"2024-05-27T07:08:22.054141+0000","last_became_peered":"2024-05-27T07:08:22.054141+0000","last_unstale":"2024-05-27T07:11:22.699081+0000","last_undegraded":"2024-05-27T07:11:22.699081+0000","last_fullsized":"2024-05-27T07:11:22.699081+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:53.413683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:14.712891+0000","last_change":"2024-05-27T07:08:36.859721+0000","last_active":"2024-05-27T07:10:14.712891+0000","last_peered":"2024-05-27T07:10:14.712891+0000","last_clean":"2024-05-27T07:10:14.712891+0000","last_became_active":"2024-05-27T07:08:36.858770+0000","last_became_peered":"2024-05-27T07:08:36.858770+0000","last_unstale":"2024-05-27T07:10:14.712891+0000","last_undegraded":"2024-05-27T07:10:14.712891+0000","last_fullsized":"2024-05-27T07:10:14.712891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:32.583869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070392+0000","last_change":"2024-05-27T07:08:18.008762+0000","last_active":"2024-05-27T07:09:54.070392+0000","last_peered":"2024-05-27T07:09:54.070392+0000","last_clean":"2024-05-27T07:09:54.070392+0000","last_became_active":"2024-05-27T07:08:18.007680+0000","last_became_peered":"2024-05-27T07:08:18.007680+0000","last_unstale":"2024-05-27T07:09:54.070392+0000","last_undegraded":"2024-05-27T07:09:54.070392+0000","last_fullsized":"2024-05-27T07:09:54.070392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:22.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.707253+0000","last_change":"2024-05-27T07:08:22.051955+0000","last_active":"2024-05-27T07:11:22.707253+0000","last_peered":"2024-05-27T07:11:22.707253+0000","last_clean":"2024-05-27T07:11:22.707253+0000","last_became_active":"2024-05-27T07:08:22.051618+0000","last_became_peered":"2024-05-27T07:08:22.051618+0000","last_unstale":"2024-05-27T07:11:22.707253+0000","last_undegraded":"2024-05-27T07:11:22.707253+0000","last_fullsized":"2024-05-27T07:11:22.707253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:13.219186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079150+0000","last_change":"2024-05-27T07:08:36.932048+0000","last_active":"2024-05-27T07:09:54.079150+0000","last_peered":"2024-05-27T07:09:54.079150+0000","last_clean":"2024-05-27T07:09:54.079150+0000","last_became_active":"2024-05-27T07:08:36.930777+0000","last_became_peered":"2024-05-27T07:08:36.930777+0000","last_unstale":"2024-05-27T07:09:54.079150+0000","last_undegraded":"2024-05-27T07:09:54.079150+0000","last_fullsized":"2024-05-27T07:09:54.079150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:32.284487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189428+0000","last_change":"2024-05-27T07:08:18.011343+0000","last_active":"2024-05-27T07:09:54.189428+0000","last_peered":"2024-05-27T07:09:54.189428+0000","last_clean":"2024-05-27T07:09:54.189428+0000","last_became_active":"2024-05-27T07:08:18.009988+0000","last_became_peered":"2024-05-27T07:08:18.009988+0000","last_unstale":"2024-05-27T07:09:54.189428+0000","last_undegraded":"2024-05-27T07:09:54.189428+0000","last_fullsized":"2024-05-27T07:09:54.189428+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:39.620568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.712978+0000","last_change":"2024-05-27T07:08:22.051630+0000","last_active":"2024-05-27T07:11:22.712978+0000","last_peered":"2024-05-27T07:11:22.712978+0000","last_clean":"2024-05-27T07:11:22.712978+0000","last_became_active":"2024-05-27T07:08:22.049140+0000","last_became_peered":"2024-05-27T07:08:22.049140+0000","last_unstale":"2024-05-27T07:11:22.712978+0000","last_undegraded":"2024-05-27T07:11:22.712978+0000","last_fullsized":"2024-05-27T07:11:22.712978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:51.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187451+0000","last_change":"2024-05-27T07:08:36.870668+0000","last_active":"2024-05-27T07:09:54.187451+0000","last_peered":"2024-05-27T07:09:54.187451+0000","last_clean":"2024-05-27T07:09:54.187451+0000","last_became_active":"2024-05-27T07:08:36.869986+0000","last_became_peered":"2024-05-27T07:08:36.869986+0000","last_unstale":"2024-05-27T07:09:54.187451+0000","last_undegraded":"2024-05-27T07:09:54.187451+0000","last_fullsized":"2024-05-27T07:09:54.187451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:59.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'7","reported_seq":176,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:51.442363+0000","last_change":"2024-05-27T07:08:18.439082+0000","last_active":"2024-05-27T07:11:51.442363+0000","last_peered":"2024-05-27T07:11:51.442363+0000","last_clean":"2024-05-27T07:11:51.442363+0000","last_became_active":"2024-05-27T07:08:18.437844+0000","last_became_peered":"2024-05-27T07:08:18.437844+0000","last_unstale":"2024-05-27T07:11:51.442363+0000","last_undegraded":"2024-05-27T07:11:51.442363+0000","last_fullsized":"2024-05-27T07:11:51.442363+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:38:30.276747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714007+0000","last_change":"2024-05-27T07:08:22.052308+0000","last_active":"2024-05-27T07:11:22.714007+0000","last_peered":"2024-05-27T07:11:22.714007+0000","last_clean":"2024-05-27T07:11:22.714007+0000","last_became_active":"2024-05-27T07:08:22.051615+0000","last_became_peered":"2024-05-27T07:08:22.051615+0000","last_unstale":"2024-05-27T07:11:22.714007+0000","last_undegraded":"2024-05-27T07:11:22.714007+0000","last_fullsized":"2024-05-27T07:11:22.714007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:14.241454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.184787+0000","last_change":"2024-05-27T07:08:36.861082+0000","last_active":"2024-05-27T07:09:54.184787+0000","last_peered":"2024-05-27T07:09:54.184787+0000","last_clean":"2024-05-27T07:09:54.184787+0000","last_became_active":"2024-05-27T07:08:36.860548+0000","last_became_peered":"2024-05-27T07:08:36.860548+0000","last_unstale":"2024-05-27T07:09:54.184787+0000","last_undegraded":"2024-05-27T07:09:54.184787+0000","last_fullsized":"2024-05-27T07:09:54.184787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:46:32.446464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043794+0000","last_change":"2024-05-27T07:08:18.018218+0000","last_active":"2024-05-27T07:09:54.043794+0000","last_peered":"2024-05-27T07:09:54.043794+0000","last_clean":"2024-05-27T07:09:54.043794+0000","last_became_active":"2024-05-27T07:08:18.017200+0000","last_became_peered":"2024-05-27T07:08:18.017200+0000","last_unstale":"2024-05-27T07:09:54.043794+0000","last_undegraded":"2024-05-27T07:09:54.043794+0000","last_fullsized":"2024-05-27T07:09:54.043794+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:22.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.698746+0000","last_change":"2024-05-27T07:09:54.052709+0000","last_active":"2024-05-27T07:11:22.698746+0000","last_peered":"2024-05-27T07:11:22.698746+0000","last_clean":"2024-05-27T07:11:22.698746+0000","last_became_active":"2024-05-27T07:09:54.051425+0000","last_became_peered":"2024-05-27T07:09:54.051425+0000","last_unstale":"2024-05-27T07:11:22.698746+0000","last_undegraded":"2024-05-27T07:11:22.698746+0000","last_fullsized":"2024-05-27T07:11:22.698746+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:51:47.166228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188807+0000","last_change":"2024-05-27T07:08:53.979067+0000","last_active":"2024-05-27T07:09:54.188807+0000","last_peered":"2024-05-27T07:09:54.188807+0000","last_clean":"2024-05-27T07:09:54.188807+0000","last_became_active":"2024-05-27T07:08:53.977620+0000","last_became_peered":"2024-05-27T07:08:53.977620+0000","last_unstale":"2024-05-27T07:09:54.188807+0000","last_undegraded":"2024-05-27T07:09:54.188807+0000","last_fullsized":"2024-05-27T07:09:54.188807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:31.619861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042083+0000","last_change":"2024-05-27T07:08:54.001216+0000","last_active":"2024-05-27T07:09:54.042083+0000","last_peered":"2024-05-27T07:09:54.042083+0000","last_clean":"2024-05-27T07:09:54.042083+0000","last_became_active":"2024-05-27T07:08:53.982121+0000","last_became_peered":"2024-05-27T07:08:53.982121+0000","last_unstale":"2024-05-27T07:09:54.042083+0000","last_undegraded":"2024-05-27T07:09:54.042083+0000","last_fullsized":"2024-05-27T07:09:54.042083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:01.474383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.687889+0000","last_change":"2024-05-27T07:08:22.053334+0000","last_active":"2024-05-27T07:11:22.687889+0000","last_peered":"2024-05-27T07:11:22.687889+0000","last_clean":"2024-05-27T07:11:22.687889+0000","last_became_active":"2024-05-27T07:08:22.051660+0000","last_became_peered":"2024-05-27T07:08:22.051660+0000","last_unstale":"2024-05-27T07:11:22.687889+0000","last_undegraded":"2024-05-27T07:11:22.687889+0000","last_fullsized":"2024-05-27T07:11:22.687889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:07:19.554620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.697780+0000","last_change":"2024-05-27T07:08:22.064044+0000","last_active":"2024-05-27T07:11:22.697780+0000","last_peered":"2024-05-27T07:11:22.697780+0000","last_clean":"2024-05-27T07:11:22.697780+0000","last_became_active":"2024-05-27T07:08:22.063539+0000","last_became_peered":"2024-05-27T07:08:22.063539+0000","last_unstale":"2024-05-27T07:11:22.697780+0000","last_undegraded":"2024-05-27T07:11:22.697780+0000","last_fullsized":"2024-05-27T07:11:22.697780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:07.291250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.695212+0000","last_change":"2024-05-27T07:08:22.059727+0000","last_active":"2024-05-27T07:11:22.695212+0000","last_peered":"2024-05-27T07:11:22.695212+0000","last_clean":"2024-05-27T07:11:22.695212+0000","last_became_active":"2024-05-27T07:08:22.058933+0000","last_became_peered":"2024-05-27T07:08:22.058933+0000","last_unstale":"2024-05-27T07:11:22.695212+0000","last_undegraded":"2024-05-27T07:11:22.695212+0000","last_fullsized":"2024-05-27T07:11:22.695212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:14.828031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.711535+0000","last_change":"2024-05-27T07:08:22.044871+0000","last_active":"2024-05-27T07:11:22.711535+0000","last_peered":"2024-05-27T07:11:22.711535+0000","last_clean":"2024-05-27T07:11:22.711535+0000","last_became_active":"2024-05-27T07:08:22.044394+0000","last_became_peered":"2024-05-27T07:08:22.044394+0000","last_unstale":"2024-05-27T07:11:22.711535+0000","last_undegraded":"2024-05-27T07:11:22.711535+0000","last_fullsized":"2024-05-27T07:11:22.711535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:11.151064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706637+0000","last_change":"2024-05-27T07:08:22.048318+0000","last_active":"2024-05-27T07:11:22.706637+0000","last_peered":"2024-05-27T07:11:22.706637+0000","last_clean":"2024-05-27T07:11:22.706637+0000","last_became_active":"2024-05-27T07:08:22.047983+0000","last_became_peered":"2024-05-27T07:08:22.047983+0000","last_unstale":"2024-05-27T07:11:22.706637+0000","last_undegraded":"2024-05-27T07:11:22.706637+0000","last_fullsized":"2024-05-27T07:11:22.706637+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:11.479943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.708261+0000","last_change":"2024-05-27T07:08:22.056951+0000","last_active":"2024-05-27T07:11:22.708261+0000","last_peered":"2024-05-27T07:11:22.708261+0000","last_clean":"2024-05-27T07:11:22.708261+0000","last_became_active":"2024-05-27T07:08:22.053701+0000","last_became_peered":"2024-05-27T07:08:22.053701+0000","last_unstale":"2024-05-27T07:11:22.708261+0000","last_undegraded":"2024-05-27T07:11:22.708261+0000","last_fullsized":"2024-05-27T07:11:22.708261+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:22:31.041730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.697404+0000","last_change":"2024-05-27T07:08:22.062745+0000","last_active":"2024-05-27T07:11:22.697404+0000","last_peered":"2024-05-27T07:11:22.697404+0000","last_clean":"2024-05-27T07:11:22.697404+0000","last_became_active":"2024-05-27T07:08:22.062339+0000","last_became_peered":"2024-05-27T07:08:22.062339+0000","last_unstale":"2024-05-27T07:11:22.697404+0000","last_undegraded":"2024-05-27T07:11:22.697404+0000","last_fullsized":"2024-05-27T07:11:22.697404+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:58:02.934021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.703314+0000","last_change":"2024-05-27T07:08:22.052621+0000","last_active":"2024-05-27T07:11:22.703314+0000","last_peered":"2024-05-27T07:11:22.703314+0000","last_clean":"2024-05-27T07:11:22.703314+0000","last_became_active":"2024-05-27T07:08:22.052070+0000","last_became_peered":"2024-05-27T07:08:22.052070+0000","last_unstale":"2024-05-27T07:11:22.703314+0000","last_undegraded":"2024-05-27T07:11:22.703314+0000","last_fullsized":"2024-05-27T07:11:22.703314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:34.491068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.707876+0000","last_change":"2024-05-27T07:08:22.055704+0000","last_active":"2024-05-27T07:11:22.707876+0000","last_peered":"2024-05-27T07:11:22.707876+0000","last_clean":"2024-05-27T07:11:22.707876+0000","last_became_active":"2024-05-27T07:08:22.055251+0000","last_became_peered":"2024-05-27T07:08:22.055251+0000","last_unstale":"2024-05-27T07:11:22.707876+0000","last_undegraded":"2024-05-27T07:11:22.707876+0000","last_fullsized":"2024-05-27T07:11:22.707876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:28.074825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.712118+0000","last_change":"2024-05-27T07:08:22.052160+0000","last_active":"2024-05-27T07:11:22.712118+0000","last_peered":"2024-05-27T07:11:22.712118+0000","last_clean":"2024-05-27T07:11:22.712118+0000","last_became_active":"2024-05-27T07:08:22.049495+0000","last_became_peered":"2024-05-27T07:08:22.049495+0000","last_unstale":"2024-05-27T07:11:22.712118+0000","last_undegraded":"2024-05-27T07:11:22.712118+0000","last_fullsized":"2024-05-27T07:11:22.712118+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:54:59.595031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.711205+0000","last_change":"2024-05-27T07:08:22.053828+0000","last_active":"2024-05-27T07:11:22.711205+0000","last_peered":"2024-05-27T07:11:22.711205+0000","last_clean":"2024-05-27T07:11:22.711205+0000","last_became_active":"2024-05-27T07:08:22.053491+0000","last_became_peered":"2024-05-27T07:08:22.053491+0000","last_unstale":"2024-05-27T07:11:22.711205+0000","last_undegraded":"2024-05-27T07:11:22.711205+0000","last_fullsized":"2024-05-27T07:11:22.711205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:20.568772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.715919+0000","last_change":"2024-05-27T07:08:22.050791+0000","last_active":"2024-05-27T07:11:22.715919+0000","last_peered":"2024-05-27T07:11:22.715919+0000","last_clean":"2024-05-27T07:11:22.715919+0000","last_became_active":"2024-05-27T07:08:22.050323+0000","last_became_peered":"2024-05-27T07:08:22.050323+0000","last_unstale":"2024-05-27T07:11:22.715919+0000","last_undegraded":"2024-05-27T07:11:22.715919+0000","last_fullsized":"2024-05-27T07:11:22.715919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:32.400758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.710462+0000","last_change":"2024-05-27T07:08:22.041286+0000","last_active":"2024-05-27T07:11:22.710462+0000","last_peered":"2024-05-27T07:11:22.710462+0000","last_clean":"2024-05-27T07:11:22.710462+0000","last_became_active":"2024-05-27T07:08:22.040811+0000","last_became_peered":"2024-05-27T07:08:22.040811+0000","last_unstale":"2024-05-27T07:11:22.710462+0000","last_undegraded":"2024-05-27T07:11:22.710462+0000","last_fullsized":"2024-05-27T07:11:22.710462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:40:09.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.690607+0000","last_change":"2024-05-27T07:08:22.066490+0000","last_active":"2024-05-27T07:11:22.690607+0000","last_peered":"2024-05-27T07:11:22.690607+0000","last_clean":"2024-05-27T07:11:22.690607+0000","last_became_active":"2024-05-27T07:08:22.064779+0000","last_became_peered":"2024-05-27T07:08:22.064779+0000","last_unstale":"2024-05-27T07:11:22.690607+0000","last_undegraded":"2024-05-27T07:11:22.690607+0000","last_fullsized":"2024-05-27T07:11:22.690607+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:23.967856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71871,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1257,"num_read_kb":709,"num_write":258,"num_write_kb":150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":215613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678080,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":519893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874311,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1736704,"data_stored":1170400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037837,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1069056,"data_stored":522693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234070,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":518281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430302,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32796,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712356,"statfs":{"total":95995035648,"available":95961452544,"internally_reserved":0,"allocated":1658880,"data_stored":1115613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593830,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":514101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":579516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738515,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":516089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411459,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":515883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":513809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953587,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":583092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790061,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1110663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:11:58.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-27T07:11:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:57 smithi043 ceph-mon[31531]: pgmap v502: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:57 smithi043 ceph-mon[31531]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:11:58.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:57 smithi179 ceph-mon[35888]: pgmap v502: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:58.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:57 smithi179 ceph-mon[35888]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:11:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:57 smithi132 ceph-mon[35283]: pgmap v502: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:11:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:57 smithi132 ceph-mon[35283]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:11:58.502 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:00.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:11:59 smithi043 ceph-mon[31531]: pgmap v503: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:00.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:11:59 smithi179 ceph-mon[35888]: pgmap v503: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:00.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:11:59 smithi132 ceph-mon[35283]: pgmap v503: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.940+0000 7f343ce00700 1 -- 172.21.15.43:0/2909229055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3438075b50 msgr2=0x7f3438075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.940+0000 7f343ce00700 1 --2- 172.21.15.43:0/2909229055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3438075b50 0x7f3438075f60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f342800b0a0 tx=0x7f342800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:12:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.941+0000 7f343ce00700 1 -- 172.21.15.43:0/2909229055 shutdown_connections 2024-05-27T07:12:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.941+0000 7f343ce00700 1 --2- 172.21.15.43:0/2909229055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3438115360 0x7f3438117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.941+0000 7f343ce00700 1 --2- 172.21.15.43:0/2909229055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3438075b50 0x7f3438075f60 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.941+0000 7f343ce00700 1 --2- 172.21.15.43:0/2909229055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34380764a0 0x7f3438114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.941+0000 7f343ce00700 1 -- 172.21.15.43:0/2909229055 >> 172.21.15.43:0/2909229055 conn(0x7f3438100030 msgr2=0x7f3438102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.941+0000 7f343ce00700 1 -- 172.21.15.43:0/2909229055 shutdown_connections 2024-05-27T07:12:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.941+0000 7f343ce00700 1 -- 172.21.15.43:0/2909229055 wait complete. 2024-05-27T07:12:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.942+0000 7f343ce00700 1 Processor -- start 2024-05-27T07:12:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.942+0000 7f343ce00700 1 -- start start 2024-05-27T07:12:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.943+0000 7f343ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3438075b50 0x7f34381a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.943+0000 7f343ce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34380764a0 0x7f34381a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.943+0000 7f343ce00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3438115360 0x7f34381a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.943+0000 7f343ce00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f343811a960 con 0x7f3438075b50 2024-05-27T07:12:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.943+0000 7f343ce00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f343811a7e0 con 0x7f34380764a0 2024-05-27T07:12:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.943+0000 7f343ce00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f343811aae0 con 0x7f3438115360 2024-05-27T07:12:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.943+0000 7f3435e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3438075b50 0x7f34381a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.943+0000 7f3435e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3438075b50 0x7f34381a1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54314/0 (socket says 172.21.15.43:54314) 2024-05-27T07:12:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.943+0000 7f3435e00700 1 -- 172.21.15.43:0/2380157512 learned_addr learned my addr 172.21.15.43:0/2380157512 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.944+0000 7f342d400700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34380764a0 0x7f34381a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.944+0000 7f3436800700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3438115360 0x7f34381a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.944+0000 7f3435e00700 1 -- 172.21.15.43:0/2380157512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3438115360 msgr2=0x7f34381a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.944+0000 7f3435e00700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3438115360 0x7f34381a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.944+0000 7f3435e00700 1 -- 172.21.15.43:0/2380157512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34380764a0 msgr2=0x7f34381a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.944+0000 7f3435e00700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34380764a0 0x7f34381a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.944+0000 7f3435e00700 1 -- 172.21.15.43:0/2380157512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34381a6220 con 0x7f3438075b50 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.944+0000 7f3436800700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3438115360 0x7f34381a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.944+0000 7f3435e00700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3438075b50 0x7f34381a1350 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f342800b070 tx=0x7f3428004e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.944+0000 7f342f400700 1 -- 172.21.15.43:0/2380157512 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428021070 con 0x7f3438075b50 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.945+0000 7f342f400700 1 -- 172.21.15.43:0/2380157512 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3428007c80 con 0x7f3438075b50 2024-05-27T07:12:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.945+0000 7f342f400700 1 -- 172.21.15.43:0/2380157512 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428004860 con 0x7f3438075b50 2024-05-27T07:12:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.945+0000 7f343ce00700 1 -- 172.21.15.43:0/2380157512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34381a64b0 con 0x7f3438075b50 2024-05-27T07:12:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.945+0000 7f343ce00700 1 -- 172.21.15.43:0/2380157512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34381ade30 con 0x7f3438075b50 2024-05-27T07:12:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.946+0000 7f343ce00700 1 -- 172.21.15.43:0/2380157512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3438077290 con 0x7f3438075b50 2024-05-27T07:12:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.948+0000 7f342f400700 1 -- 172.21.15.43:0/2380157512 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f3428004020 con 0x7f3438075b50 2024-05-27T07:12:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.950+0000 7f342f400700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f341c0b2900 0x7f341c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.950+0000 7f342f400700 1 -- 172.21.15.43:0/2380157512 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f342800f020 con 0x7f3438075b50 2024-05-27T07:12:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.950+0000 7f342d400700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f341c0b2900 0x7f341c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.951+0000 7f342d400700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f341c0b2900 0x7f341c0b4e10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3420005950 tx=0x7f34200058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:00.953+0000 7f342f400700 1 -- 172.21.15.43:0/2380157512 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f342802a020 con 0x7f3438075b50 2024-05-27T07:12:01.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.191+0000 7f343ce00700 1 -- 172.21.15.43:0/2380157512 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f34381a26b0 con 0x7f341c0b2900 2024-05-27T07:12:01.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.221+0000 7f342f400700 1 -- 172.21.15.43:0/2380157512 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340512 (secure 0 0 0) 0x7f34381a26b0 con 0x7f341c0b2900 2024-05-27T07:12:01.219 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:12:01.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.229+0000 7f343ce00700 1 -- 172.21.15.43:0/2380157512 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f341c0b2900 msgr2=0x7f341c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:01.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f341c0b2900 0x7f341c0b4e10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3420005950 tx=0x7f34200058e0 comp rx=0 tx=0).stop 2024-05-27T07:12:01.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 -- 172.21.15.43:0/2380157512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3438075b50 msgr2=0x7f34381a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:01.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3438075b50 0x7f34381a1350 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f342800b070 tx=0x7f3428004e00 comp rx=0 tx=0).stop 2024-05-27T07:12:01.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 -- 172.21.15.43:0/2380157512 shutdown_connections 2024-05-27T07:12:01.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f341c0b2900 0x7f341c0b4e10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:01.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3438115360 0x7f34381a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:01.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3438075b50 0x7f34381a1350 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:01.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 --2- 172.21.15.43:0/2380157512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34380764a0 0x7f34381a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:01.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 -- 172.21.15.43:0/2380157512 >> 172.21.15.43:0/2380157512 conn(0x7f3438100030 msgr2=0x7f3438100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:01.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 -- 172.21.15.43:0/2380157512 shutdown_connections 2024-05-27T07:12:01.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:01.230+0000 7f343ce00700 1 -- 172.21.15.43:0/2380157512 wait complete. 2024-05-27T07:12:01.223 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-05-27T07:12:02.178 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:01 smithi043 ceph-mon[31531]: pgmap v504: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:02.182 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-05-27T07:12:00.835906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662473,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1388,"num_read_kb":803,"num_write":383,"num_write_kb":2302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953452,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":816,"ondisk_log_size":816,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367648,"kb_used_data":14336,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574176,"statfs":{"total":1151940427776,"available":1151563956224,"internally_reserved":0,"allocated":14680064,"data_stored":8180033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":130,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006845"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714202+0000","last_change":"2024-05-27T07:08:22.053540+0000","last_active":"2024-05-27T07:11:22.714202+0000","last_peered":"2024-05-27T07:11:22.714202+0000","last_clean":"2024-05-27T07:11:22.714202+0000","last_became_active":"2024-05-27T07:08:22.051872+0000","last_became_peered":"2024-05-27T07:08:22.051872+0000","last_unstale":"2024-05-27T07:11:22.714202+0000","last_undegraded":"2024-05-27T07:11:22.714202+0000","last_fullsized":"2024-05-27T07:11:22.714202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:02.787957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.700741+0000","last_change":"2024-05-27T07:08:22.049341+0000","last_active":"2024-05-27T07:11:22.700741+0000","last_peered":"2024-05-27T07:11:22.700741+0000","last_clean":"2024-05-27T07:11:22.700741+0000","last_became_active":"2024-05-27T07:08:22.048905+0000","last_became_peered":"2024-05-27T07:08:22.048905+0000","last_unstale":"2024-05-27T07:11:22.700741+0000","last_undegraded":"2024-05-27T07:11:22.700741+0000","last_fullsized":"2024-05-27T07:11:22.700741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:27.271336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.713108+0000","last_change":"2024-05-27T07:08:22.053925+0000","last_active":"2024-05-27T07:11:22.713108+0000","last_peered":"2024-05-27T07:11:22.713108+0000","last_clean":"2024-05-27T07:11:22.713108+0000","last_became_active":"2024-05-27T07:08:22.053409+0000","last_became_peered":"2024-05-27T07:08:22.053409+0000","last_unstale":"2024-05-27T07:11:22.713108+0000","last_undegraded":"2024-05-27T07:11:22.713108+0000","last_fullsized":"2024-05-27T07:11:22.713108+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:07:55.439966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.696388+0000","last_change":"2024-05-27T07:08:22.048214+0000","last_active":"2024-05-27T07:11:22.696388+0000","last_peered":"2024-05-27T07:11:22.696388+0000","last_clean":"2024-05-27T07:11:22.696388+0000","last_became_active":"2024-05-27T07:08:22.047791+0000","last_became_peered":"2024-05-27T07:08:22.047791+0000","last_unstale":"2024-05-27T07:11:22.696388+0000","last_undegraded":"2024-05-27T07:11:22.696388+0000","last_fullsized":"2024-05-27T07:11:22.696388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:23:56.381008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.712561+0000","last_change":"2024-05-27T07:08:22.037883+0000","last_active":"2024-05-27T07:11:22.712561+0000","last_peered":"2024-05-27T07:11:22.712561+0000","last_clean":"2024-05-27T07:11:22.712561+0000","last_became_active":"2024-05-27T07:08:22.037089+0000","last_became_peered":"2024-05-27T07:08:22.037089+0000","last_unstale":"2024-05-27T07:11:22.712561+0000","last_undegraded":"2024-05-27T07:11:22.712561+0000","last_fullsized":"2024-05-27T07:11:22.712561+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:21.929086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.693343+0000","last_change":"2024-05-27T07:08:22.061833+0000","last_active":"2024-05-27T07:11:22.693343+0000","last_peered":"2024-05-27T07:11:22.693343+0000","last_clean":"2024-05-27T07:11:22.693343+0000","last_became_active":"2024-05-27T07:08:22.057692+0000","last_became_peered":"2024-05-27T07:08:22.057692+0000","last_unstale":"2024-05-27T07:11:22.693343+0000","last_undegraded":"2024-05-27T07:11:22.693343+0000","last_fullsized":"2024-05-27T07:11:22.693343+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:38:33.656872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.690332+0000","last_change":"2024-05-27T07:08:22.047880+0000","last_active":"2024-05-27T07:11:22.690332+0000","last_peered":"2024-05-27T07:11:22.690332+0000","last_clean":"2024-05-27T07:11:22.690332+0000","last_became_active":"2024-05-27T07:08:22.047482+0000","last_became_peered":"2024-05-27T07:08:22.047482+0000","last_unstale":"2024-05-27T07:11:22.690332+0000","last_undegraded":"2024-05-27T07:11:22.690332+0000","last_fullsized":"2024-05-27T07:11:22.690332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:07.751588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.711214+0000","last_change":"2024-05-27T07:08:22.055970+0000","last_active":"2024-05-27T07:11:22.711214+0000","last_peered":"2024-05-27T07:11:22.711214+0000","last_clean":"2024-05-27T07:11:22.711214+0000","last_became_active":"2024-05-27T07:08:22.055591+0000","last_became_peered":"2024-05-27T07:08:22.055591+0000","last_unstale":"2024-05-27T07:11:22.711214+0000","last_undegraded":"2024-05-27T07:11:22.711214+0000","last_fullsized":"2024-05-27T07:11:22.711214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:58:46.571473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.715285+0000","last_change":"2024-05-27T07:08:22.059929+0000","last_active":"2024-05-27T07:11:22.715285+0000","last_peered":"2024-05-27T07:11:22.715285+0000","last_clean":"2024-05-27T07:11:22.715285+0000","last_became_active":"2024-05-27T07:08:22.059118+0000","last_became_peered":"2024-05-27T07:08:22.059118+0000","last_unstale":"2024-05-27T07:11:22.715285+0000","last_undegraded":"2024-05-27T07:11:22.715285+0000","last_fullsized":"2024-05-27T07:11:22.715285+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:29.231220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.700504+0000","last_change":"2024-05-27T07:08:53.979269+0000","last_active":"2024-05-27T07:11:22.700504+0000","last_peered":"2024-05-27T07:11:22.700504+0000","last_clean":"2024-05-27T07:11:22.700504+0000","last_became_active":"2024-05-27T07:08:53.978433+0000","last_became_peered":"2024-05-27T07:08:53.978433+0000","last_unstale":"2024-05-27T07:11:22.700504+0000","last_undegraded":"2024-05-27T07:11:22.700504+0000","last_fullsized":"2024-05-27T07:11:22.700504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:57:37.600911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.713893+0000","last_change":"2024-05-27T07:08:22.053499+0000","last_active":"2024-05-27T07:11:22.713893+0000","last_peered":"2024-05-27T07:11:22.713893+0000","last_clean":"2024-05-27T07:11:22.713893+0000","last_became_active":"2024-05-27T07:08:22.052428+0000","last_became_peered":"2024-05-27T07:08:22.052428+0000","last_unstale":"2024-05-27T07:11:22.713893+0000","last_undegraded":"2024-05-27T07:11:22.713893+0000","last_fullsized":"2024-05-27T07:11:22.713893+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:04.352262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.699790+0000","last_change":"2024-05-27T07:08:22.058882+0000","last_active":"2024-05-27T07:11:22.699790+0000","last_peered":"2024-05-27T07:11:22.699790+0000","last_clean":"2024-05-27T07:11:22.699790+0000","last_became_active":"2024-05-27T07:08:22.057874+0000","last_became_peered":"2024-05-27T07:08:22.057874+0000","last_unstale":"2024-05-27T07:11:22.699790+0000","last_undegraded":"2024-05-27T07:11:22.699790+0000","last_fullsized":"2024-05-27T07:11:22.699790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:52:11.289488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706679+0000","last_change":"2024-05-27T07:08:22.054315+0000","last_active":"2024-05-27T07:11:22.706679+0000","last_peered":"2024-05-27T07:11:22.706679+0000","last_clean":"2024-05-27T07:11:22.706679+0000","last_became_active":"2024-05-27T07:08:22.053944+0000","last_became_peered":"2024-05-27T07:08:22.053944+0000","last_unstale":"2024-05-27T07:11:22.706679+0000","last_undegraded":"2024-05-27T07:11:22.706679+0000","last_fullsized":"2024-05-27T07:11:22.706679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:43:06.720147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706279+0000","last_change":"2024-05-27T07:08:22.054649+0000","last_active":"2024-05-27T07:11:22.706279+0000","last_peered":"2024-05-27T07:11:22.706279+0000","last_clean":"2024-05-27T07:11:22.706279+0000","last_became_active":"2024-05-27T07:08:22.051834+0000","last_became_peered":"2024-05-27T07:08:22.051834+0000","last_unstale":"2024-05-27T07:11:22.706279+0000","last_undegraded":"2024-05-27T07:11:22.706279+0000","last_fullsized":"2024-05-27T07:11:22.706279+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:16:56.893458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.705614+0000","last_change":"2024-05-27T07:08:22.052231+0000","last_active":"2024-05-27T07:11:22.705614+0000","last_peered":"2024-05-27T07:11:22.705614+0000","last_clean":"2024-05-27T07:11:22.705614+0000","last_became_active":"2024-05-27T07:08:22.051495+0000","last_became_peered":"2024-05-27T07:08:22.051495+0000","last_unstale":"2024-05-27T07:11:22.705614+0000","last_undegraded":"2024-05-27T07:11:22.705614+0000","last_fullsized":"2024-05-27T07:11:22.705614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:30:21.036874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714436+0000","last_change":"2024-05-27T07:08:22.056847+0000","last_active":"2024-05-27T07:11:22.714436+0000","last_peered":"2024-05-27T07:11:22.714436+0000","last_clean":"2024-05-27T07:11:22.714436+0000","last_became_active":"2024-05-27T07:08:22.056127+0000","last_became_peered":"2024-05-27T07:08:22.056127+0000","last_unstale":"2024-05-27T07:11:22.714436+0000","last_undegraded":"2024-05-27T07:11:22.714436+0000","last_fullsized":"2024-05-27T07:11:22.714436+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:04.757452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.716087+0000","last_change":"2024-05-27T07:08:22.058078+0000","last_active":"2024-05-27T07:11:22.716087+0000","last_peered":"2024-05-27T07:11:22.716087+0000","last_clean":"2024-05-27T07:11:22.716087+0000","last_became_active":"2024-05-27T07:08:22.056886+0000","last_became_peered":"2024-05-27T07:08:22.056886+0000","last_unstale":"2024-05-27T07:11:22.716087+0000","last_undegraded":"2024-05-27T07:11:22.716087+0000","last_fullsized":"2024-05-27T07:11:22.716087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.104572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.705110+0000","last_change":"2024-05-27T07:08:22.051492+0000","last_active":"2024-05-27T07:11:22.705110+0000","last_peered":"2024-05-27T07:11:22.705110+0000","last_clean":"2024-05-27T07:11:22.705110+0000","last_became_active":"2024-05-27T07:08:22.051091+0000","last_became_peered":"2024-05-27T07:08:22.051091+0000","last_unstale":"2024-05-27T07:11:22.705110+0000","last_undegraded":"2024-05-27T07:11:22.705110+0000","last_fullsized":"2024-05-27T07:11:22.705110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:47.816312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.712229+0000","last_change":"2024-05-27T07:08:22.050714+0000","last_active":"2024-05-27T07:11:22.712229+0000","last_peered":"2024-05-27T07:11:22.712229+0000","last_clean":"2024-05-27T07:11:22.712229+0000","last_became_active":"2024-05-27T07:08:22.050322+0000","last_became_peered":"2024-05-27T07:08:22.050322+0000","last_unstale":"2024-05-27T07:11:22.712229+0000","last_undegraded":"2024-05-27T07:11:22.712229+0000","last_fullsized":"2024-05-27T07:11:22.712229+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:01.867467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070420+0000","last_change":"2024-05-27T07:08:36.859916+0000","last_active":"2024-05-27T07:09:54.070420+0000","last_peered":"2024-05-27T07:09:54.070420+0000","last_clean":"2024-05-27T07:09:54.070420+0000","last_became_active":"2024-05-27T07:08:36.859429+0000","last_became_peered":"2024-05-27T07:08:36.859429+0000","last_unstale":"2024-05-27T07:09:54.070420+0000","last_undegraded":"2024-05-27T07:09:54.070420+0000","last_fullsized":"2024-05-27T07:09:54.070420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:14.026604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:56.422739+0000","last_change":"2024-05-27T07:08:18.008912+0000","last_active":"2024-05-27T07:11:56.422739+0000","last_peered":"2024-05-27T07:11:56.422739+0000","last_clean":"2024-05-27T07:11:56.422739+0000","last_became_active":"2024-05-27T07:08:18.007603+0000","last_became_peered":"2024-05-27T07:08:18.007603+0000","last_unstale":"2024-05-27T07:11:56.422739+0000","last_undegraded":"2024-05-27T07:11:56.422739+0000","last_fullsized":"2024-05-27T07:11:56.422739+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:17:40.681754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.711671+0000","last_change":"2024-05-27T07:08:22.055426+0000","last_active":"2024-05-27T07:11:22.711671+0000","last_peered":"2024-05-27T07:11:22.711671+0000","last_clean":"2024-05-27T07:11:22.711671+0000","last_became_active":"2024-05-27T07:08:22.055063+0000","last_became_peered":"2024-05-27T07:08:22.055063+0000","last_unstale":"2024-05-27T07:11:22.711671+0000","last_undegraded":"2024-05-27T07:11:22.711671+0000","last_fullsized":"2024-05-27T07:11:22.711671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:06:49.923192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079599+0000","last_change":"2024-05-27T07:08:53.982756+0000","last_active":"2024-05-27T07:09:54.079599+0000","last_peered":"2024-05-27T07:09:54.079599+0000","last_clean":"2024-05-27T07:09:54.079599+0000","last_became_active":"2024-05-27T07:08:53.981816+0000","last_became_peered":"2024-05-27T07:08:53.981816+0000","last_unstale":"2024-05-27T07:09:54.079599+0000","last_undegraded":"2024-05-27T07:09:54.079599+0000","last_fullsized":"2024-05-27T07:09:54.079599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:09.984488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063371+0000","last_change":"2024-05-27T07:08:18.307659+0000","last_active":"2024-05-27T07:09:54.063371+0000","last_peered":"2024-05-27T07:09:54.063371+0000","last_clean":"2024-05-27T07:09:54.063371+0000","last_became_active":"2024-05-27T07:08:18.307239+0000","last_became_peered":"2024-05-27T07:08:18.307239+0000","last_unstale":"2024-05-27T07:09:54.063371+0000","last_undegraded":"2024-05-27T07:09:54.063371+0000","last_fullsized":"2024-05-27T07:09:54.063371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:09:02.816813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.713140+0000","last_change":"2024-05-27T07:08:22.050403+0000","last_active":"2024-05-27T07:11:22.713140+0000","last_peered":"2024-05-27T07:11:22.713140+0000","last_clean":"2024-05-27T07:11:22.713140+0000","last_became_active":"2024-05-27T07:08:22.049939+0000","last_became_peered":"2024-05-27T07:08:22.049939+0000","last_unstale":"2024-05-27T07:11:22.713140+0000","last_undegraded":"2024-05-27T07:11:22.713140+0000","last_fullsized":"2024-05-27T07:11:22.713140+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:45.857508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.078966+0000","last_change":"2024-05-27T07:08:36.864458+0000","last_active":"2024-05-27T07:09:54.078966+0000","last_peered":"2024-05-27T07:09:54.078966+0000","last_clean":"2024-05-27T07:09:54.078966+0000","last_became_active":"2024-05-27T07:08:36.863894+0000","last_became_peered":"2024-05-27T07:08:36.863894+0000","last_unstale":"2024-05-27T07:09:54.078966+0000","last_undegraded":"2024-05-27T07:09:54.078966+0000","last_fullsized":"2024-05-27T07:09:54.078966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:49.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190406+0000","last_change":"2024-05-27T07:08:18.016051+0000","last_active":"2024-05-27T07:09:54.190406+0000","last_peered":"2024-05-27T07:09:54.190406+0000","last_clean":"2024-05-27T07:09:54.190406+0000","last_became_active":"2024-05-27T07:08:18.015590+0000","last_became_peered":"2024-05-27T07:08:18.015590+0000","last_unstale":"2024-05-27T07:09:54.190406+0000","last_undegraded":"2024-05-27T07:09:54.190406+0000","last_fullsized":"2024-05-27T07:09:54.190406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:33.396027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.715250+0000","last_change":"2024-05-27T07:08:22.055603+0000","last_active":"2024-05-27T07:11:22.715250+0000","last_peered":"2024-05-27T07:11:22.715250+0000","last_clean":"2024-05-27T07:11:22.715250+0000","last_became_active":"2024-05-27T07:08:22.054543+0000","last_became_peered":"2024-05-27T07:08:22.054543+0000","last_unstale":"2024-05-27T07:11:22.715250+0000","last_undegraded":"2024-05-27T07:11:22.715250+0000","last_fullsized":"2024-05-27T07:11:22.715250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:29:01.738598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.078533+0000","last_change":"2024-05-27T07:08:36.870358+0000","last_active":"2024-05-27T07:09:54.078533+0000","last_peered":"2024-05-27T07:09:54.078533+0000","last_clean":"2024-05-27T07:09:54.078533+0000","last_became_active":"2024-05-27T07:08:36.869457+0000","last_became_peered":"2024-05-27T07:08:36.869457+0000","last_unstale":"2024-05-27T07:09:54.078533+0000","last_undegraded":"2024-05-27T07:09:54.078533+0000","last_fullsized":"2024-05-27T07:09:54.078533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:50.510098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187633+0000","last_change":"2024-05-27T07:08:53.985562+0000","last_active":"2024-05-27T07:09:54.187633+0000","last_peered":"2024-05-27T07:09:54.187633+0000","last_clean":"2024-05-27T07:09:54.187633+0000","last_became_active":"2024-05-27T07:08:53.977778+0000","last_became_peered":"2024-05-27T07:08:53.977778+0000","last_unstale":"2024-05-27T07:09:54.187633+0000","last_undegraded":"2024-05-27T07:09:54.187633+0000","last_fullsized":"2024-05-27T07:09:54.187633+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:44.455508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706977+0000","last_change":"2024-05-27T07:08:22.057622+0000","last_active":"2024-05-27T07:11:22.706977+0000","last_peered":"2024-05-27T07:11:22.706977+0000","last_clean":"2024-05-27T07:11:22.706977+0000","last_became_active":"2024-05-27T07:08:22.056544+0000","last_became_peered":"2024-05-27T07:08:22.056544+0000","last_unstale":"2024-05-27T07:11:22.706977+0000","last_undegraded":"2024-05-27T07:11:22.706977+0000","last_fullsized":"2024-05-27T07:11:22.706977+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:38.484560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.071085+0000","last_change":"2024-05-27T07:08:36.864984+0000","last_active":"2024-05-27T07:09:54.071085+0000","last_peered":"2024-05-27T07:09:54.071085+0000","last_clean":"2024-05-27T07:09:54.071085+0000","last_became_active":"2024-05-27T07:08:36.864302+0000","last_became_peered":"2024-05-27T07:08:36.864302+0000","last_unstale":"2024-05-27T07:09:54.071085+0000","last_undegraded":"2024-05-27T07:09:54.071085+0000","last_fullsized":"2024-05-27T07:09:54.071085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:44.183562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":179,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:58.439077+0000","last_change":"2024-05-27T07:08:18.017918+0000","last_active":"2024-05-27T07:11:58.439077+0000","last_peered":"2024-05-27T07:11:58.439077+0000","last_clean":"2024-05-27T07:11:58.439077+0000","last_became_active":"2024-05-27T07:08:18.016174+0000","last_became_peered":"2024-05-27T07:08:18.016174+0000","last_unstale":"2024-05-27T07:11:58.439077+0000","last_undegraded":"2024-05-27T07:11:58.439077+0000","last_fullsized":"2024-05-27T07:11:58.439077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:21.906187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.697882+0000","last_change":"2024-05-27T07:08:22.043172+0000","last_active":"2024-05-27T07:11:22.697882+0000","last_peered":"2024-05-27T07:11:22.697882+0000","last_clean":"2024-05-27T07:11:22.697882+0000","last_became_active":"2024-05-27T07:08:22.041399+0000","last_became_peered":"2024-05-27T07:08:22.041399+0000","last_unstale":"2024-05-27T07:11:22.697882+0000","last_undegraded":"2024-05-27T07:11:22.697882+0000","last_fullsized":"2024-05-27T07:11:22.697882+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:19.651330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070394+0000","last_change":"2024-05-27T07:08:36.866352+0000","last_active":"2024-05-27T07:09:54.070394+0000","last_peered":"2024-05-27T07:09:54.070394+0000","last_clean":"2024-05-27T07:09:54.070394+0000","last_became_active":"2024-05-27T07:08:36.865447+0000","last_became_peered":"2024-05-27T07:08:36.865447+0000","last_unstale":"2024-05-27T07:09:54.070394+0000","last_undegraded":"2024-05-27T07:09:54.070394+0000","last_fullsized":"2024-05-27T07:09:54.070394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:55:42.634374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.063044+0000","last_change":"2024-05-27T07:08:18.022142+0000","last_active":"2024-05-27T07:09:54.063044+0000","last_peered":"2024-05-27T07:09:54.063044+0000","last_clean":"2024-05-27T07:09:54.063044+0000","last_became_active":"2024-05-27T07:08:18.021453+0000","last_became_peered":"2024-05-27T07:08:18.021453+0000","last_unstale":"2024-05-27T07:09:54.063044+0000","last_undegraded":"2024-05-27T07:09:54.063044+0000","last_fullsized":"2024-05-27T07:09:54.063044+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:01:09.049440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714060+0000","last_change":"2024-05-27T07:08:22.055742+0000","last_active":"2024-05-27T07:11:22.714060+0000","last_peered":"2024-05-27T07:11:22.714060+0000","last_clean":"2024-05-27T07:11:22.714060+0000","last_became_active":"2024-05-27T07:08:22.052077+0000","last_became_peered":"2024-05-27T07:08:22.052077+0000","last_unstale":"2024-05-27T07:11:22.714060+0000","last_undegraded":"2024-05-27T07:11:22.714060+0000","last_fullsized":"2024-05-27T07:11:22.714060+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:09:13.418027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188119+0000","last_change":"2024-05-27T07:08:36.874225+0000","last_active":"2024-05-27T07:09:54.188119+0000","last_peered":"2024-05-27T07:09:54.188119+0000","last_clean":"2024-05-27T07:09:54.188119+0000","last_became_active":"2024-05-27T07:08:36.873745+0000","last_became_peered":"2024-05-27T07:08:36.873745+0000","last_unstale":"2024-05-27T07:09:54.188119+0000","last_undegraded":"2024-05-27T07:09:54.188119+0000","last_fullsized":"2024-05-27T07:09:54.188119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:40.633667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041025+0000","last_change":"2024-05-27T07:08:18.009678+0000","last_active":"2024-05-27T07:09:54.041025+0000","last_peered":"2024-05-27T07:09:54.041025+0000","last_clean":"2024-05-27T07:09:54.041025+0000","last_became_active":"2024-05-27T07:08:18.006502+0000","last_became_peered":"2024-05-27T07:08:18.006502+0000","last_unstale":"2024-05-27T07:09:54.041025+0000","last_undegraded":"2024-05-27T07:09:54.041025+0000","last_fullsized":"2024-05-27T07:09:54.041025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:37.457044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.713616+0000","last_change":"2024-05-27T07:08:22.047302+0000","last_active":"2024-05-27T07:11:22.713616+0000","last_peered":"2024-05-27T07:11:22.713616+0000","last_clean":"2024-05-27T07:11:22.713616+0000","last_became_active":"2024-05-27T07:08:22.045932+0000","last_became_peered":"2024-05-27T07:08:22.045932+0000","last_unstale":"2024-05-27T07:11:22.713616+0000","last_undegraded":"2024-05-27T07:11:22.713616+0000","last_fullsized":"2024-05-27T07:11:22.713616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:27.129539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.185409+0000","last_change":"2024-05-27T07:08:36.870512+0000","last_active":"2024-05-27T07:09:54.185409+0000","last_peered":"2024-05-27T07:09:54.185409+0000","last_clean":"2024-05-27T07:09:54.185409+0000","last_became_active":"2024-05-27T07:08:36.870061+0000","last_became_peered":"2024-05-27T07:08:36.870061+0000","last_unstale":"2024-05-27T07:09:54.185409+0000","last_undegraded":"2024-05-27T07:09:54.185409+0000","last_fullsized":"2024-05-27T07:09:54.185409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:03.608758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042222+0000","last_change":"2024-05-27T07:08:18.018420+0000","last_active":"2024-05-27T07:09:54.042222+0000","last_peered":"2024-05-27T07:09:54.042222+0000","last_clean":"2024-05-27T07:09:54.042222+0000","last_became_active":"2024-05-27T07:08:18.017567+0000","last_became_peered":"2024-05-27T07:08:18.017567+0000","last_unstale":"2024-05-27T07:09:54.042222+0000","last_undegraded":"2024-05-27T07:09:54.042222+0000","last_fullsized":"2024-05-27T07:09:54.042222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:37.214408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.713166+0000","last_change":"2024-05-27T07:08:22.050943+0000","last_active":"2024-05-27T07:11:22.713166+0000","last_peered":"2024-05-27T07:11:22.713166+0000","last_clean":"2024-05-27T07:11:22.713166+0000","last_became_active":"2024-05-27T07:08:22.047706+0000","last_became_peered":"2024-05-27T07:08:22.047706+0000","last_unstale":"2024-05-27T07:11:22.713166+0000","last_undegraded":"2024-05-27T07:11:22.713166+0000","last_fullsized":"2024-05-27T07:11:22.713166+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:52:09.739528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060574+0000","last_change":"2024-05-27T07:08:36.867487+0000","last_active":"2024-05-27T07:09:54.060574+0000","last_peered":"2024-05-27T07:09:54.060574+0000","last_clean":"2024-05-27T07:09:54.060574+0000","last_became_active":"2024-05-27T07:08:36.867032+0000","last_became_peered":"2024-05-27T07:08:36.867032+0000","last_unstale":"2024-05-27T07:09:54.060574+0000","last_undegraded":"2024-05-27T07:09:54.060574+0000","last_fullsized":"2024-05-27T07:09:54.060574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:26:24.912382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":304,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:59.440759+0000","last_change":"2024-05-27T07:08:18.020168+0000","last_active":"2024-05-27T07:11:59.440759+0000","last_peered":"2024-05-27T07:11:59.440759+0000","last_clean":"2024-05-27T07:11:59.440759+0000","last_became_active":"2024-05-27T07:08:18.018309+0000","last_became_peered":"2024-05-27T07:08:18.018309+0000","last_unstale":"2024-05-27T07:11:59.440759+0000","last_undegraded":"2024-05-27T07:11:59.440759+0000","last_fullsized":"2024-05-27T07:11:59.440759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:48:16.839510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":254,"num_read_kb":254,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.715433+0000","last_change":"2024-05-27T07:08:22.054888+0000","last_active":"2024-05-27T07:11:22.715433+0000","last_peered":"2024-05-27T07:11:22.715433+0000","last_clean":"2024-05-27T07:11:22.715433+0000","last_became_active":"2024-05-27T07:08:22.054417+0000","last_became_peered":"2024-05-27T07:08:22.054417+0000","last_unstale":"2024-05-27T07:11:22.715433+0000","last_undegraded":"2024-05-27T07:11:22.715433+0000","last_fullsized":"2024-05-27T07:11:22.715433+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:43:00.866755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.047889+0000","last_change":"2024-05-27T07:08:36.854516+0000","last_active":"2024-05-27T07:09:54.047889+0000","last_peered":"2024-05-27T07:09:54.047889+0000","last_clean":"2024-05-27T07:09:54.047889+0000","last_became_active":"2024-05-27T07:08:36.853978+0000","last_became_peered":"2024-05-27T07:08:36.853978+0000","last_unstale":"2024-05-27T07:09:54.047889+0000","last_undegraded":"2024-05-27T07:09:54.047889+0000","last_fullsized":"2024-05-27T07:09:54.047889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:45.299100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045529+0000","last_change":"2024-05-27T07:08:18.010539+0000","last_active":"2024-05-27T07:09:54.045529+0000","last_peered":"2024-05-27T07:09:54.045529+0000","last_clean":"2024-05-27T07:09:54.045529+0000","last_became_active":"2024-05-27T07:08:18.009643+0000","last_became_peered":"2024-05-27T07:08:18.009643+0000","last_unstale":"2024-05-27T07:09:54.045529+0000","last_undegraded":"2024-05-27T07:09:54.045529+0000","last_fullsized":"2024-05-27T07:09:54.045529+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:48:40.335403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706673+0000","last_change":"2024-05-27T07:08:53.980165+0000","last_active":"2024-05-27T07:11:22.706673+0000","last_peered":"2024-05-27T07:11:22.706673+0000","last_clean":"2024-05-27T07:11:22.706673+0000","last_became_active":"2024-05-27T07:08:53.978565+0000","last_became_peered":"2024-05-27T07:08:53.978565+0000","last_unstale":"2024-05-27T07:11:22.706673+0000","last_undegraded":"2024-05-27T07:11:22.706673+0000","last_fullsized":"2024-05-27T07:11:22.706673+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:56:33.513961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079609+0000","last_change":"2024-05-27T07:08:36.857814+0000","last_active":"2024-05-27T07:09:54.079609+0000","last_peered":"2024-05-27T07:09:54.079609+0000","last_clean":"2024-05-27T07:09:54.079609+0000","last_became_active":"2024-05-27T07:08:36.856790+0000","last_became_peered":"2024-05-27T07:08:36.856790+0000","last_unstale":"2024-05-27T07:09:54.079609+0000","last_undegraded":"2024-05-27T07:09:54.079609+0000","last_fullsized":"2024-05-27T07:09:54.079609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:39:10.322261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043549+0000","last_change":"2024-05-27T07:08:18.308032+0000","last_active":"2024-05-27T07:09:54.043549+0000","last_peered":"2024-05-27T07:09:54.043549+0000","last_clean":"2024-05-27T07:09:54.043549+0000","last_became_active":"2024-05-27T07:08:18.306867+0000","last_became_peered":"2024-05-27T07:08:18.306867+0000","last_unstale":"2024-05-27T07:09:54.043549+0000","last_undegraded":"2024-05-27T07:09:54.043549+0000","last_fullsized":"2024-05-27T07:09:54.043549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:37.816237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714964+0000","last_change":"2024-05-27T07:08:22.057940+0000","last_active":"2024-05-27T07:11:22.714964+0000","last_peered":"2024-05-27T07:11:22.714964+0000","last_clean":"2024-05-27T07:11:22.714964+0000","last_became_active":"2024-05-27T07:08:22.055109+0000","last_became_peered":"2024-05-27T07:08:22.055109+0000","last_unstale":"2024-05-27T07:11:22.714964+0000","last_undegraded":"2024-05-27T07:11:22.714964+0000","last_fullsized":"2024-05-27T07:11:22.714964+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:46.905064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.062722+0000","last_change":"2024-05-27T07:08:53.980209+0000","last_active":"2024-05-27T07:09:54.062722+0000","last_peered":"2024-05-27T07:09:54.062722+0000","last_clean":"2024-05-27T07:09:54.062722+0000","last_became_active":"2024-05-27T07:08:53.979353+0000","last_became_peered":"2024-05-27T07:08:53.979353+0000","last_unstale":"2024-05-27T07:09:54.062722+0000","last_undegraded":"2024-05-27T07:09:54.062722+0000","last_fullsized":"2024-05-27T07:09:54.062722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:06.163563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"90'38","reported_seq":363,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:59.460014+0000","last_change":"2024-05-27T07:08:53.996491+0000","last_active":"2024-05-27T07:11:59.460014+0000","last_peered":"2024-05-27T07:11:59.460014+0000","last_clean":"2024-05-27T07:11:59.460014+0000","last_became_active":"2024-05-27T07:08:53.976013+0000","last_became_peered":"2024-05-27T07:08:53.976013+0000","last_unstale":"2024-05-27T07:11:59.460014+0000","last_undegraded":"2024-05-27T07:11:59.460014+0000","last_fullsized":"2024-05-27T07:11:59.460014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:44:16.592270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64385,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":264,"num_read_kb":195,"num_write":40,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.691765+0000","last_change":"2024-05-27T07:08:22.057231+0000","last_active":"2024-05-27T07:11:22.691765+0000","last_peered":"2024-05-27T07:11:22.691765+0000","last_clean":"2024-05-27T07:11:22.691765+0000","last_became_active":"2024-05-27T07:08:22.056837+0000","last_became_peered":"2024-05-27T07:08:22.056837+0000","last_unstale":"2024-05-27T07:11:22.691765+0000","last_undegraded":"2024-05-27T07:11:22.691765+0000","last_fullsized":"2024-05-27T07:11:22.691765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:10:28.645103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044424+0000","last_change":"2024-05-27T07:08:36.870307+0000","last_active":"2024-05-27T07:09:54.044424+0000","last_peered":"2024-05-27T07:09:54.044424+0000","last_clean":"2024-05-27T07:09:54.044424+0000","last_became_active":"2024-05-27T07:08:36.869390+0000","last_became_peered":"2024-05-27T07:08:36.869390+0000","last_unstale":"2024-05-27T07:09:54.044424+0000","last_undegraded":"2024-05-27T07:09:54.044424+0000","last_fullsized":"2024-05-27T07:09:54.044424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:43:45.228885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070799+0000","last_change":"2024-05-27T07:08:18.014458+0000","last_active":"2024-05-27T07:09:54.070799+0000","last_peered":"2024-05-27T07:09:54.070799+0000","last_clean":"2024-05-27T07:09:54.070799+0000","last_became_active":"2024-05-27T07:08:18.013365+0000","last_became_peered":"2024-05-27T07:08:18.013365+0000","last_unstale":"2024-05-27T07:09:54.070799+0000","last_undegraded":"2024-05-27T07:09:54.070799+0000","last_fullsized":"2024-05-27T07:09:54.070799+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:41.352618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.705768+0000","last_change":"2024-05-27T07:08:22.043354+0000","last_active":"2024-05-27T07:11:22.705768+0000","last_peered":"2024-05-27T07:11:22.705768+0000","last_clean":"2024-05-27T07:11:22.705768+0000","last_became_active":"2024-05-27T07:08:22.042876+0000","last_became_peered":"2024-05-27T07:08:22.042876+0000","last_unstale":"2024-05-27T07:11:22.705768+0000","last_undegraded":"2024-05-27T07:11:22.705768+0000","last_fullsized":"2024-05-27T07:11:22.705768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:52.575568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188013+0000","last_change":"2024-05-27T07:08:36.869579+0000","last_active":"2024-05-27T07:09:54.188013+0000","last_peered":"2024-05-27T07:09:54.188013+0000","last_clean":"2024-05-27T07:09:54.188013+0000","last_became_active":"2024-05-27T07:08:36.869055+0000","last_became_peered":"2024-05-27T07:08:36.869055+0000","last_unstale":"2024-05-27T07:09:54.188013+0000","last_undegraded":"2024-05-27T07:09:54.188013+0000","last_fullsized":"2024-05-27T07:09:54.188013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.870079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044142+0000","last_change":"2024-05-27T07:08:18.015322+0000","last_active":"2024-05-27T07:09:54.044142+0000","last_peered":"2024-05-27T07:09:54.044142+0000","last_clean":"2024-05-27T07:09:54.044142+0000","last_became_active":"2024-05-27T07:08:18.014535+0000","last_became_peered":"2024-05-27T07:08:18.014535+0000","last_unstale":"2024-05-27T07:09:54.044142+0000","last_undegraded":"2024-05-27T07:09:54.044142+0000","last_fullsized":"2024-05-27T07:09:54.044142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:35.116140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.703585+0000","last_change":"2024-05-27T07:08:22.056181+0000","last_active":"2024-05-27T07:11:22.703585+0000","last_peered":"2024-05-27T07:11:22.703585+0000","last_clean":"2024-05-27T07:11:22.703585+0000","last_became_active":"2024-05-27T07:08:22.055755+0000","last_became_peered":"2024-05-27T07:08:22.055755+0000","last_unstale":"2024-05-27T07:11:22.703585+0000","last_undegraded":"2024-05-27T07:11:22.703585+0000","last_fullsized":"2024-05-27T07:11:22.703585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:38:35.634116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.048450+0000","last_change":"2024-05-27T07:08:36.863184+0000","last_active":"2024-05-27T07:09:54.048450+0000","last_peered":"2024-05-27T07:09:54.048450+0000","last_clean":"2024-05-27T07:09:54.048450+0000","last_became_active":"2024-05-27T07:08:36.862760+0000","last_became_peered":"2024-05-27T07:08:36.862760+0000","last_unstale":"2024-05-27T07:09:54.048450+0000","last_undegraded":"2024-05-27T07:09:54.048450+0000","last_fullsized":"2024-05-27T07:09:54.048450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:48.379157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":179,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:55.459781+0000","last_change":"2024-05-27T07:08:18.019119+0000","last_active":"2024-05-27T07:11:55.459781+0000","last_peered":"2024-05-27T07:11:55.459781+0000","last_clean":"2024-05-27T07:11:55.459781+0000","last_became_active":"2024-05-27T07:08:18.018408+0000","last_became_peered":"2024-05-27T07:08:18.018408+0000","last_unstale":"2024-05-27T07:11:55.459781+0000","last_undegraded":"2024-05-27T07:11:55.459781+0000","last_fullsized":"2024-05-27T07:11:55.459781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:19.978328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":129,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.698925+0000","last_change":"2024-05-27T07:08:22.059464+0000","last_active":"2024-05-27T07:11:22.698925+0000","last_peered":"2024-05-27T07:11:22.698925+0000","last_clean":"2024-05-27T07:11:22.698925+0000","last_became_active":"2024-05-27T07:08:22.058890+0000","last_became_peered":"2024-05-27T07:08:22.058890+0000","last_unstale":"2024-05-27T07:11:22.698925+0000","last_undegraded":"2024-05-27T07:11:22.698925+0000","last_fullsized":"2024-05-27T07:11:22.698925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:30.122552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041973+0000","last_change":"2024-05-27T07:08:36.865884+0000","last_active":"2024-05-27T07:09:54.041973+0000","last_peered":"2024-05-27T07:09:54.041973+0000","last_clean":"2024-05-27T07:09:54.041973+0000","last_became_active":"2024-05-27T07:08:36.865443+0000","last_became_peered":"2024-05-27T07:08:36.865443+0000","last_unstale":"2024-05-27T07:09:54.041973+0000","last_undegraded":"2024-05-27T07:09:54.041973+0000","last_fullsized":"2024-05-27T07:09:54.041973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:47:15.026632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187097+0000","last_change":"2024-05-27T07:08:53.994809+0000","last_active":"2024-05-27T07:09:54.187097+0000","last_peered":"2024-05-27T07:09:54.187097+0000","last_clean":"2024-05-27T07:09:54.187097+0000","last_became_active":"2024-05-27T07:08:53.977628+0000","last_became_peered":"2024-05-27T07:08:53.977628+0000","last_unstale":"2024-05-27T07:09:54.187097+0000","last_undegraded":"2024-05-27T07:09:54.187097+0000","last_fullsized":"2024-05-27T07:09:54.187097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:13:37.176319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706591+0000","last_change":"2024-05-27T07:08:22.050733+0000","last_active":"2024-05-27T07:11:22.706591+0000","last_peered":"2024-05-27T07:11:22.706591+0000","last_clean":"2024-05-27T07:11:22.706591+0000","last_became_active":"2024-05-27T07:08:22.050321+0000","last_became_peered":"2024-05-27T07:08:22.050321+0000","last_unstale":"2024-05-27T07:11:22.706591+0000","last_undegraded":"2024-05-27T07:11:22.706591+0000","last_fullsized":"2024-05-27T07:11:22.706591+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:44.590595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043754+0000","last_change":"2024-05-27T07:08:36.867784+0000","last_active":"2024-05-27T07:09:54.043754+0000","last_peered":"2024-05-27T07:09:54.043754+0000","last_clean":"2024-05-27T07:09:54.043754+0000","last_became_active":"2024-05-27T07:08:36.867290+0000","last_became_peered":"2024-05-27T07:08:36.867290+0000","last_unstale":"2024-05-27T07:09:54.043754+0000","last_undegraded":"2024-05-27T07:09:54.043754+0000","last_fullsized":"2024-05-27T07:09:54.043754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:44:07.649640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190834+0000","last_change":"2024-05-27T07:08:18.013193+0000","last_active":"2024-05-27T07:09:54.190834+0000","last_peered":"2024-05-27T07:09:54.190834+0000","last_clean":"2024-05-27T07:09:54.190834+0000","last_became_active":"2024-05-27T07:08:18.011767+0000","last_became_peered":"2024-05-27T07:08:18.011767+0000","last_unstale":"2024-05-27T07:09:54.190834+0000","last_undegraded":"2024-05-27T07:09:54.190834+0000","last_fullsized":"2024-05-27T07:09:54.190834+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:15:03.679719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.691219+0000","last_change":"2024-05-27T07:08:22.067145+0000","last_active":"2024-05-27T07:11:22.691219+0000","last_peered":"2024-05-27T07:11:22.691219+0000","last_clean":"2024-05-27T07:11:22.691219+0000","last_became_active":"2024-05-27T07:08:22.066750+0000","last_became_peered":"2024-05-27T07:08:22.066750+0000","last_unstale":"2024-05-27T07:11:22.691219+0000","last_undegraded":"2024-05-27T07:11:22.691219+0000","last_fullsized":"2024-05-27T07:11:22.691219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:41:35.333311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189235+0000","last_change":"2024-05-27T07:08:53.982497+0000","last_active":"2024-05-27T07:09:54.189235+0000","last_peered":"2024-05-27T07:09:54.189235+0000","last_clean":"2024-05-27T07:09:54.189235+0000","last_became_active":"2024-05-27T07:08:53.981920+0000","last_became_peered":"2024-05-27T07:08:53.981920+0000","last_unstale":"2024-05-27T07:09:54.189235+0000","last_undegraded":"2024-05-27T07:09:54.189235+0000","last_fullsized":"2024-05-27T07:09:54.189235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:43:40.095357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.048681+0000","last_change":"2024-05-27T07:08:18.308330+0000","last_active":"2024-05-27T07:09:54.048681+0000","last_peered":"2024-05-27T07:09:54.048681+0000","last_clean":"2024-05-27T07:09:54.048681+0000","last_became_active":"2024-05-27T07:08:18.306961+0000","last_became_peered":"2024-05-27T07:08:18.306961+0000","last_unstale":"2024-05-27T07:09:54.048681+0000","last_undegraded":"2024-05-27T07:09:54.048681+0000","last_fullsized":"2024-05-27T07:09:54.048681+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:27.875936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.690952+0000","last_change":"2024-05-27T07:08:22.047562+0000","last_active":"2024-05-27T07:11:22.690952+0000","last_peered":"2024-05-27T07:11:22.690952+0000","last_clean":"2024-05-27T07:11:22.690952+0000","last_became_active":"2024-05-27T07:08:22.046977+0000","last_became_peered":"2024-05-27T07:08:22.046977+0000","last_unstale":"2024-05-27T07:11:22.690952+0000","last_undegraded":"2024-05-27T07:11:22.690952+0000","last_fullsized":"2024-05-27T07:11:22.690952+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:49:43.376617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041089+0000","last_change":"2024-05-27T07:08:36.867382+0000","last_active":"2024-05-27T07:09:54.041089+0000","last_peered":"2024-05-27T07:09:54.041089+0000","last_clean":"2024-05-27T07:09:54.041089+0000","last_became_active":"2024-05-27T07:08:36.866296+0000","last_became_peered":"2024-05-27T07:08:36.866296+0000","last_unstale":"2024-05-27T07:09:54.041089+0000","last_undegraded":"2024-05-27T07:09:54.041089+0000","last_fullsized":"2024-05-27T07:09:54.041089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:14.305169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:59.432937+0000","last_change":"2024-05-27T07:08:18.012031+0000","last_active":"2024-05-27T07:11:59.432937+0000","last_peered":"2024-05-27T07:11:59.432937+0000","last_clean":"2024-05-27T07:11:59.432937+0000","last_became_active":"2024-05-27T07:08:18.009593+0000","last_became_peered":"2024-05-27T07:08:18.009593+0000","last_unstale":"2024-05-27T07:11:59.432937+0000","last_undegraded":"2024-05-27T07:11:59.432937+0000","last_fullsized":"2024-05-27T07:11:59.432937+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:25:22.316575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.693495+0000","last_change":"2024-05-27T07:08:22.061136+0000","last_active":"2024-05-27T07:11:22.693495+0000","last_peered":"2024-05-27T07:11:22.693495+0000","last_clean":"2024-05-27T07:11:22.693495+0000","last_became_active":"2024-05-27T07:08:22.060735+0000","last_became_peered":"2024-05-27T07:08:22.060735+0000","last_unstale":"2024-05-27T07:11:22.693495+0000","last_undegraded":"2024-05-27T07:11:22.693495+0000","last_fullsized":"2024-05-27T07:11:22.693495+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:50.392130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.041856+0000","last_change":"2024-05-27T07:08:36.932187+0000","last_active":"2024-05-27T07:09:54.041856+0000","last_peered":"2024-05-27T07:09:54.041856+0000","last_clean":"2024-05-27T07:09:54.041856+0000","last_became_active":"2024-05-27T07:08:36.931214+0000","last_became_peered":"2024-05-27T07:08:36.931214+0000","last_unstale":"2024-05-27T07:09:54.041856+0000","last_undegraded":"2024-05-27T07:09:54.041856+0000","last_fullsized":"2024-05-27T07:09:54.041856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:20:59.645352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.690342+0000","last_change":"2024-05-27T07:09:54.082674+0000","last_active":"2024-05-27T07:11:22.690342+0000","last_peered":"2024-05-27T07:11:22.690342+0000","last_clean":"2024-05-27T07:11:22.690342+0000","last_became_active":"2024-05-27T07:09:54.081616+0000","last_became_peered":"2024-05-27T07:09:54.081616+0000","last_unstale":"2024-05-27T07:11:22.690342+0000","last_undegraded":"2024-05-27T07:11:22.690342+0000","last_fullsized":"2024-05-27T07:11:22.690342+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:38:11.230026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:57.464446+0000","last_change":"2024-05-27T07:08:18.306439+0000","last_active":"2024-05-27T07:11:57.464446+0000","last_peered":"2024-05-27T07:11:57.464446+0000","last_clean":"2024-05-27T07:11:57.464446+0000","last_became_active":"2024-05-27T07:08:18.305978+0000","last_became_peered":"2024-05-27T07:08:18.305978+0000","last_unstale":"2024-05-27T07:11:57.464446+0000","last_undegraded":"2024-05-27T07:11:57.464446+0000","last_fullsized":"2024-05-27T07:11:57.464446+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:20:22.648922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":127,"num_read_kb":127,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:06.623537+0000","last_change":"2024-05-27T07:08:36.866247+0000","last_active":"2024-05-27T07:10:06.623537+0000","last_peered":"2024-05-27T07:10:06.623537+0000","last_clean":"2024-05-27T07:10:06.623537+0000","last_became_active":"2024-05-27T07:08:36.865516+0000","last_became_peered":"2024-05-27T07:08:36.865516+0000","last_unstale":"2024-05-27T07:10:06.623537+0000","last_undegraded":"2024-05-27T07:10:06.623537+0000","last_fullsized":"2024-05-27T07:10:06.623537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:39:41.235352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.716330+0000","last_change":"2024-05-27T07:08:22.045312+0000","last_active":"2024-05-27T07:11:22.716330+0000","last_peered":"2024-05-27T07:11:22.716330+0000","last_clean":"2024-05-27T07:11:22.716330+0000","last_became_active":"2024-05-27T07:08:22.044724+0000","last_became_peered":"2024-05-27T07:08:22.044724+0000","last_unstale":"2024-05-27T07:11:22.716330+0000","last_undegraded":"2024-05-27T07:11:22.716330+0000","last_fullsized":"2024-05-27T07:11:22.716330+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:05.487794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079152+0000","last_change":"2024-05-27T07:08:18.019333+0000","last_active":"2024-05-27T07:09:54.079152+0000","last_peered":"2024-05-27T07:09:54.079152+0000","last_clean":"2024-05-27T07:09:54.079152+0000","last_became_active":"2024-05-27T07:08:18.018816+0000","last_became_peered":"2024-05-27T07:08:18.018816+0000","last_unstale":"2024-05-27T07:09:54.079152+0000","last_undegraded":"2024-05-27T07:09:54.079152+0000","last_fullsized":"2024-05-27T07:09:54.079152+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:09.421346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.044348+0000","last_change":"2024-05-27T07:06:16.865587+0000","last_active":"2024-05-27T07:09:54.044348+0000","last_peered":"2024-05-27T07:09:54.044348+0000","last_clean":"2024-05-27T07:09:54.044348+0000","last_became_active":"2024-05-27T07:06:16.850646+0000","last_became_peered":"2024-05-27T07:06:16.850646+0000","last_unstale":"2024-05-27T07:09:54.044348+0000","last_undegraded":"2024-05-27T07:09:54.044348+0000","last_fullsized":"2024-05-27T07:09:54.044348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_clean_scrub_stamp":"2024-05-27T07:01:14.931400+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:37.272619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043690+0000","last_change":"2024-05-27T07:08:36.934696+0000","last_active":"2024-05-27T07:09:54.043690+0000","last_peered":"2024-05-27T07:09:54.043690+0000","last_clean":"2024-05-27T07:09:54.043690+0000","last_became_active":"2024-05-27T07:08:36.934008+0000","last_became_peered":"2024-05-27T07:08:36.934008+0000","last_unstale":"2024-05-27T07:09:54.043690+0000","last_undegraded":"2024-05-27T07:09:54.043690+0000","last_fullsized":"2024-05-27T07:09:54.043690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:39.099180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.701045+0000","last_change":"2024-05-27T07:08:22.054479+0000","last_active":"2024-05-27T07:11:22.701045+0000","last_peered":"2024-05-27T07:11:22.701045+0000","last_clean":"2024-05-27T07:11:22.701045+0000","last_became_active":"2024-05-27T07:08:22.054127+0000","last_became_peered":"2024-05-27T07:08:22.054127+0000","last_unstale":"2024-05-27T07:11:22.701045+0000","last_undegraded":"2024-05-27T07:11:22.701045+0000","last_fullsized":"2024-05-27T07:11:22.701045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:37:25.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.045671+0000","last_change":"2024-05-27T07:08:18.010944+0000","last_active":"2024-05-27T07:09:54.045671+0000","last_peered":"2024-05-27T07:09:54.045671+0000","last_clean":"2024-05-27T07:09:54.045671+0000","last_became_active":"2024-05-27T07:08:18.009816+0000","last_became_peered":"2024-05-27T07:08:18.009816+0000","last_unstale":"2024-05-27T07:09:54.045671+0000","last_undegraded":"2024-05-27T07:09:54.045671+0000","last_fullsized":"2024-05-27T07:09:54.045671+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:34:26.061787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187569+0000","last_change":"2024-05-27T07:08:36.870618+0000","last_active":"2024-05-27T07:09:54.187569+0000","last_peered":"2024-05-27T07:09:54.187569+0000","last_clean":"2024-05-27T07:09:54.187569+0000","last_became_active":"2024-05-27T07:08:36.869799+0000","last_became_peered":"2024-05-27T07:08:36.869799+0000","last_unstale":"2024-05-27T07:09:54.187569+0000","last_undegraded":"2024-05-27T07:09:54.187569+0000","last_fullsized":"2024-05-27T07:09:54.187569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:53.094337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.190088+0000","last_change":"2024-05-27T07:08:18.308834+0000","last_active":"2024-05-27T07:09:54.190088+0000","last_peered":"2024-05-27T07:09:54.190088+0000","last_clean":"2024-05-27T07:09:54.190088+0000","last_became_active":"2024-05-27T07:08:18.306953+0000","last_became_peered":"2024-05-27T07:08:18.306953+0000","last_unstale":"2024-05-27T07:09:54.190088+0000","last_undegraded":"2024-05-27T07:09:54.190088+0000","last_fullsized":"2024-05-27T07:09:54.190088+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:26:14.747631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.701750+0000","last_change":"2024-05-27T07:08:22.048219+0000","last_active":"2024-05-27T07:11:22.701750+0000","last_peered":"2024-05-27T07:11:22.701750+0000","last_clean":"2024-05-27T07:11:22.701750+0000","last_became_active":"2024-05-27T07:08:22.047848+0000","last_became_peered":"2024-05-27T07:08:22.047848+0000","last_unstale":"2024-05-27T07:11:22.701750+0000","last_undegraded":"2024-05-27T07:11:22.701750+0000","last_fullsized":"2024-05-27T07:11:22.701750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:51.707603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060451+0000","last_change":"2024-05-27T07:08:36.932747+0000","last_active":"2024-05-27T07:09:54.060451+0000","last_peered":"2024-05-27T07:09:54.060451+0000","last_clean":"2024-05-27T07:09:54.060451+0000","last_became_active":"2024-05-27T07:08:36.932050+0000","last_became_peered":"2024-05-27T07:08:36.932050+0000","last_unstale":"2024-05-27T07:09:54.060451+0000","last_undegraded":"2024-05-27T07:09:54.060451+0000","last_fullsized":"2024-05-27T07:09:54.060451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:26:07.015651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'16","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:27.880470+0000","last_change":"2024-05-27T07:08:18.008844+0000","last_active":"2024-05-27T07:10:27.880470+0000","last_peered":"2024-05-27T07:10:27.880470+0000","last_clean":"2024-05-27T07:10:27.880470+0000","last_became_active":"2024-05-27T07:08:18.008140+0000","last_became_peered":"2024-05-27T07:08:18.008140+0000","last_unstale":"2024-05-27T07:10:27.880470+0000","last_undegraded":"2024-05-27T07:10:27.880470+0000","last_fullsized":"2024-05-27T07:10:27.880470+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:22:22.075341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714727+0000","last_change":"2024-05-27T07:08:22.050855+0000","last_active":"2024-05-27T07:11:22.714727+0000","last_peered":"2024-05-27T07:11:22.714727+0000","last_clean":"2024-05-27T07:11:22.714727+0000","last_became_active":"2024-05-27T07:08:22.048482+0000","last_became_peered":"2024-05-27T07:08:22.048482+0000","last_unstale":"2024-05-27T07:11:22.714727+0000","last_undegraded":"2024-05-27T07:11:22.714727+0000","last_fullsized":"2024-05-27T07:11:22.714727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:04.319884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.069731+0000","last_change":"2024-05-27T07:08:36.862470+0000","last_active":"2024-05-27T07:09:54.069731+0000","last_peered":"2024-05-27T07:09:54.069731+0000","last_clean":"2024-05-27T07:09:54.069731+0000","last_became_active":"2024-05-27T07:08:36.861951+0000","last_became_peered":"2024-05-27T07:08:36.861951+0000","last_unstale":"2024-05-27T07:09:54.069731+0000","last_undegraded":"2024-05-27T07:09:54.069731+0000","last_fullsized":"2024-05-27T07:09:54.069731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:56:31.659951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.080450+0000","last_change":"2024-05-27T07:08:18.438518+0000","last_active":"2024-05-27T07:09:54.080450+0000","last_peered":"2024-05-27T07:09:54.080450+0000","last_clean":"2024-05-27T07:09:54.080450+0000","last_became_active":"2024-05-27T07:08:18.437588+0000","last_became_peered":"2024-05-27T07:08:18.437588+0000","last_unstale":"2024-05-27T07:09:54.080450+0000","last_undegraded":"2024-05-27T07:09:54.080450+0000","last_fullsized":"2024-05-27T07:09:54.080450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:52:12.895009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706881+0000","last_change":"2024-05-27T07:08:22.064188+0000","last_active":"2024-05-27T07:11:22.706881+0000","last_peered":"2024-05-27T07:11:22.706881+0000","last_clean":"2024-05-27T07:11:22.706881+0000","last_became_active":"2024-05-27T07:08:22.063868+0000","last_became_peered":"2024-05-27T07:08:22.063868+0000","last_unstale":"2024-05-27T07:11:22.706881+0000","last_undegraded":"2024-05-27T07:11:22.706881+0000","last_fullsized":"2024-05-27T07:11:22.706881+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:43:17.227079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079354+0000","last_change":"2024-05-27T07:08:36.934922+0000","last_active":"2024-05-27T07:09:54.079354+0000","last_peered":"2024-05-27T07:09:54.079354+0000","last_clean":"2024-05-27T07:09:54.079354+0000","last_became_active":"2024-05-27T07:08:36.934000+0000","last_became_peered":"2024-05-27T07:08:36.934000+0000","last_unstale":"2024-05-27T07:09:54.079354+0000","last_undegraded":"2024-05-27T07:09:54.079354+0000","last_fullsized":"2024-05-27T07:09:54.079354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:19:00.570557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187967+0000","last_change":"2024-05-27T07:08:18.308743+0000","last_active":"2024-05-27T07:09:54.187967+0000","last_peered":"2024-05-27T07:09:54.187967+0000","last_clean":"2024-05-27T07:09:54.187967+0000","last_became_active":"2024-05-27T07:08:18.307597+0000","last_became_peered":"2024-05-27T07:08:18.307597+0000","last_unstale":"2024-05-27T07:09:54.187967+0000","last_undegraded":"2024-05-27T07:09:54.187967+0000","last_fullsized":"2024-05-27T07:09:54.187967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:39.914998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.645960+0000","last_change":"2024-05-27T07:08:22.049132+0000","last_active":"2024-05-27T07:11:22.645960+0000","last_peered":"2024-05-27T07:11:22.645960+0000","last_clean":"2024-05-27T07:11:22.645960+0000","last_became_active":"2024-05-27T07:08:22.048606+0000","last_became_peered":"2024-05-27T07:08:22.048606+0000","last_unstale":"2024-05-27T07:11:22.645960+0000","last_undegraded":"2024-05-27T07:11:22.645960+0000","last_fullsized":"2024-05-27T07:11:22.645960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:58.253461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.049257+0000","last_change":"2024-05-27T07:08:36.867286+0000","last_active":"2024-05-27T07:09:54.049257+0000","last_peered":"2024-05-27T07:09:54.049257+0000","last_clean":"2024-05-27T07:09:54.049257+0000","last_became_active":"2024-05-27T07:08:36.866732+0000","last_became_peered":"2024-05-27T07:08:36.866732+0000","last_unstale":"2024-05-27T07:09:54.049257+0000","last_undegraded":"2024-05-27T07:09:54.049257+0000","last_fullsized":"2024-05-27T07:09:54.049257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:14:18.386441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.060788+0000","last_change":"2024-05-27T07:08:18.010026+0000","last_active":"2024-05-27T07:09:54.060788+0000","last_peered":"2024-05-27T07:09:54.060788+0000","last_clean":"2024-05-27T07:09:54.060788+0000","last_became_active":"2024-05-27T07:08:18.009502+0000","last_became_peered":"2024-05-27T07:08:18.009502+0000","last_unstale":"2024-05-27T07:09:54.060788+0000","last_undegraded":"2024-05-27T07:09:54.060788+0000","last_fullsized":"2024-05-27T07:09:54.060788+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:28:27.500933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.699081+0000","last_change":"2024-05-27T07:08:22.054536+0000","last_active":"2024-05-27T07:11:22.699081+0000","last_peered":"2024-05-27T07:11:22.699081+0000","last_clean":"2024-05-27T07:11:22.699081+0000","last_became_active":"2024-05-27T07:08:22.054141+0000","last_became_peered":"2024-05-27T07:08:22.054141+0000","last_unstale":"2024-05-27T07:11:22.699081+0000","last_undegraded":"2024-05-27T07:11:22.699081+0000","last_fullsized":"2024-05-27T07:11:22.699081+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:53.413683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:10:14.712891+0000","last_change":"2024-05-27T07:08:36.859721+0000","last_active":"2024-05-27T07:10:14.712891+0000","last_peered":"2024-05-27T07:10:14.712891+0000","last_clean":"2024-05-27T07:10:14.712891+0000","last_became_active":"2024-05-27T07:08:36.858770+0000","last_became_peered":"2024-05-27T07:08:36.858770+0000","last_unstale":"2024-05-27T07:10:14.712891+0000","last_undegraded":"2024-05-27T07:10:14.712891+0000","last_fullsized":"2024-05-27T07:10:14.712891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:32.583869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.070392+0000","last_change":"2024-05-27T07:08:18.008762+0000","last_active":"2024-05-27T07:09:54.070392+0000","last_peered":"2024-05-27T07:09:54.070392+0000","last_clean":"2024-05-27T07:09:54.070392+0000","last_became_active":"2024-05-27T07:08:18.007680+0000","last_became_peered":"2024-05-27T07:08:18.007680+0000","last_unstale":"2024-05-27T07:09:54.070392+0000","last_undegraded":"2024-05-27T07:09:54.070392+0000","last_fullsized":"2024-05-27T07:09:54.070392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:22.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.707253+0000","last_change":"2024-05-27T07:08:22.051955+0000","last_active":"2024-05-27T07:11:22.707253+0000","last_peered":"2024-05-27T07:11:22.707253+0000","last_clean":"2024-05-27T07:11:22.707253+0000","last_became_active":"2024-05-27T07:08:22.051618+0000","last_became_peered":"2024-05-27T07:08:22.051618+0000","last_unstale":"2024-05-27T07:11:22.707253+0000","last_undegraded":"2024-05-27T07:11:22.707253+0000","last_fullsized":"2024-05-27T07:11:22.707253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:13.219186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.079150+0000","last_change":"2024-05-27T07:08:36.932048+0000","last_active":"2024-05-27T07:09:54.079150+0000","last_peered":"2024-05-27T07:09:54.079150+0000","last_clean":"2024-05-27T07:09:54.079150+0000","last_became_active":"2024-05-27T07:08:36.930777+0000","last_became_peered":"2024-05-27T07:08:36.930777+0000","last_unstale":"2024-05-27T07:09:54.079150+0000","last_undegraded":"2024-05-27T07:09:54.079150+0000","last_fullsized":"2024-05-27T07:09:54.079150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:32.284487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.189428+0000","last_change":"2024-05-27T07:08:18.011343+0000","last_active":"2024-05-27T07:09:54.189428+0000","last_peered":"2024-05-27T07:09:54.189428+0000","last_clean":"2024-05-27T07:09:54.189428+0000","last_became_active":"2024-05-27T07:08:18.009988+0000","last_became_peered":"2024-05-27T07:08:18.009988+0000","last_unstale":"2024-05-27T07:09:54.189428+0000","last_undegraded":"2024-05-27T07:09:54.189428+0000","last_fullsized":"2024-05-27T07:09:54.189428+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:39.620568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.712978+0000","last_change":"2024-05-27T07:08:22.051630+0000","last_active":"2024-05-27T07:11:22.712978+0000","last_peered":"2024-05-27T07:11:22.712978+0000","last_clean":"2024-05-27T07:11:22.712978+0000","last_became_active":"2024-05-27T07:08:22.049140+0000","last_became_peered":"2024-05-27T07:08:22.049140+0000","last_unstale":"2024-05-27T07:11:22.712978+0000","last_undegraded":"2024-05-27T07:11:22.712978+0000","last_fullsized":"2024-05-27T07:11:22.712978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:51.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.187451+0000","last_change":"2024-05-27T07:08:36.870668+0000","last_active":"2024-05-27T07:09:54.187451+0000","last_peered":"2024-05-27T07:09:54.187451+0000","last_clean":"2024-05-27T07:09:54.187451+0000","last_became_active":"2024-05-27T07:08:36.869986+0000","last_became_peered":"2024-05-27T07:08:36.869986+0000","last_unstale":"2024-05-27T07:09:54.187451+0000","last_undegraded":"2024-05-27T07:09:54.187451+0000","last_fullsized":"2024-05-27T07:09:54.187451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:59.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'7","reported_seq":181,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:56.458560+0000","last_change":"2024-05-27T07:08:18.439082+0000","last_active":"2024-05-27T07:11:56.458560+0000","last_peered":"2024-05-27T07:11:56.458560+0000","last_clean":"2024-05-27T07:11:56.458560+0000","last_became_active":"2024-05-27T07:08:18.437844+0000","last_became_peered":"2024-05-27T07:08:18.437844+0000","last_unstale":"2024-05-27T07:11:56.458560+0000","last_undegraded":"2024-05-27T07:11:56.458560+0000","last_fullsized":"2024-05-27T07:11:56.458560+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:38:30.276747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":129,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.714007+0000","last_change":"2024-05-27T07:08:22.052308+0000","last_active":"2024-05-27T07:11:22.714007+0000","last_peered":"2024-05-27T07:11:22.714007+0000","last_clean":"2024-05-27T07:11:22.714007+0000","last_became_active":"2024-05-27T07:08:22.051615+0000","last_became_peered":"2024-05-27T07:08:22.051615+0000","last_unstale":"2024-05-27T07:11:22.714007+0000","last_undegraded":"2024-05-27T07:11:22.714007+0000","last_fullsized":"2024-05-27T07:11:22.714007+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:14.241454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.184787+0000","last_change":"2024-05-27T07:08:36.861082+0000","last_active":"2024-05-27T07:09:54.184787+0000","last_peered":"2024-05-27T07:09:54.184787+0000","last_clean":"2024-05-27T07:09:54.184787+0000","last_became_active":"2024-05-27T07:08:36.860548+0000","last_became_peered":"2024-05-27T07:08:36.860548+0000","last_unstale":"2024-05-27T07:09:54.184787+0000","last_undegraded":"2024-05-27T07:09:54.184787+0000","last_fullsized":"2024-05-27T07:09:54.184787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:46:32.446464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.043794+0000","last_change":"2024-05-27T07:08:18.018218+0000","last_active":"2024-05-27T07:09:54.043794+0000","last_peered":"2024-05-27T07:09:54.043794+0000","last_clean":"2024-05-27T07:09:54.043794+0000","last_became_active":"2024-05-27T07:08:18.017200+0000","last_became_peered":"2024-05-27T07:08:18.017200+0000","last_unstale":"2024-05-27T07:09:54.043794+0000","last_undegraded":"2024-05-27T07:09:54.043794+0000","last_fullsized":"2024-05-27T07:09:54.043794+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:22.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.698746+0000","last_change":"2024-05-27T07:09:54.052709+0000","last_active":"2024-05-27T07:11:22.698746+0000","last_peered":"2024-05-27T07:11:22.698746+0000","last_clean":"2024-05-27T07:11:22.698746+0000","last_became_active":"2024-05-27T07:09:54.051425+0000","last_became_peered":"2024-05-27T07:09:54.051425+0000","last_unstale":"2024-05-27T07:11:22.698746+0000","last_undegraded":"2024-05-27T07:11:22.698746+0000","last_fullsized":"2024-05-27T07:11:22.698746+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:51:47.166228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.188807+0000","last_change":"2024-05-27T07:08:53.979067+0000","last_active":"2024-05-27T07:09:54.188807+0000","last_peered":"2024-05-27T07:09:54.188807+0000","last_clean":"2024-05-27T07:09:54.188807+0000","last_became_active":"2024-05-27T07:08:53.977620+0000","last_became_peered":"2024-05-27T07:08:53.977620+0000","last_unstale":"2024-05-27T07:09:54.188807+0000","last_undegraded":"2024-05-27T07:09:54.188807+0000","last_fullsized":"2024-05-27T07:09:54.188807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:31.619861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:09:54.042083+0000","last_change":"2024-05-27T07:08:54.001216+0000","last_active":"2024-05-27T07:09:54.042083+0000","last_peered":"2024-05-27T07:09:54.042083+0000","last_clean":"2024-05-27T07:09:54.042083+0000","last_became_active":"2024-05-27T07:08:53.982121+0000","last_became_peered":"2024-05-27T07:08:53.982121+0000","last_unstale":"2024-05-27T07:09:54.042083+0000","last_undegraded":"2024-05-27T07:09:54.042083+0000","last_fullsized":"2024-05-27T07:09:54.042083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:01.474383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.687889+0000","last_change":"2024-05-27T07:08:22.053334+0000","last_active":"2024-05-27T07:11:22.687889+0000","last_peered":"2024-05-27T07:11:22.687889+0000","last_clean":"2024-05-27T07:11:22.687889+0000","last_became_active":"2024-05-27T07:08:22.051660+0000","last_became_peered":"2024-05-27T07:08:22.051660+0000","last_unstale":"2024-05-27T07:11:22.687889+0000","last_undegraded":"2024-05-27T07:11:22.687889+0000","last_fullsized":"2024-05-27T07:11:22.687889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:07:19.554620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.697780+0000","last_change":"2024-05-27T07:08:22.064044+0000","last_active":"2024-05-27T07:11:22.697780+0000","last_peered":"2024-05-27T07:11:22.697780+0000","last_clean":"2024-05-27T07:11:22.697780+0000","last_became_active":"2024-05-27T07:08:22.063539+0000","last_became_peered":"2024-05-27T07:08:22.063539+0000","last_unstale":"2024-05-27T07:11:22.697780+0000","last_undegraded":"2024-05-27T07:11:22.697780+0000","last_fullsized":"2024-05-27T07:11:22.697780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:07.291250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.695212+0000","last_change":"2024-05-27T07:08:22.059727+0000","last_active":"2024-05-27T07:11:22.695212+0000","last_peered":"2024-05-27T07:11:22.695212+0000","last_clean":"2024-05-27T07:11:22.695212+0000","last_became_active":"2024-05-27T07:08:22.058933+0000","last_became_peered":"2024-05-27T07:08:22.058933+0000","last_unstale":"2024-05-27T07:11:22.695212+0000","last_undegraded":"2024-05-27T07:11:22.695212+0000","last_fullsized":"2024-05-27T07:11:22.695212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:14.828031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.711535+0000","last_change":"2024-05-27T07:08:22.044871+0000","last_active":"2024-05-27T07:11:22.711535+0000","last_peered":"2024-05-27T07:11:22.711535+0000","last_clean":"2024-05-27T07:11:22.711535+0000","last_became_active":"2024-05-27T07:08:22.044394+0000","last_became_peered":"2024-05-27T07:08:22.044394+0000","last_unstale":"2024-05-27T07:11:22.711535+0000","last_undegraded":"2024-05-27T07:11:22.711535+0000","last_fullsized":"2024-05-27T07:11:22.711535+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:11.151064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.706637+0000","last_change":"2024-05-27T07:08:22.048318+0000","last_active":"2024-05-27T07:11:22.706637+0000","last_peered":"2024-05-27T07:11:22.706637+0000","last_clean":"2024-05-27T07:11:22.706637+0000","last_became_active":"2024-05-27T07:08:22.047983+0000","last_became_peered":"2024-05-27T07:08:22.047983+0000","last_unstale":"2024-05-27T07:11:22.706637+0000","last_undegraded":"2024-05-27T07:11:22.706637+0000","last_fullsized":"2024-05-27T07:11:22.706637+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:11.479943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.708261+0000","last_change":"2024-05-27T07:08:22.056951+0000","last_active":"2024-05-27T07:11:22.708261+0000","last_peered":"2024-05-27T07:11:22.708261+0000","last_clean":"2024-05-27T07:11:22.708261+0000","last_became_active":"2024-05-27T07:08:22.053701+0000","last_became_peered":"2024-05-27T07:08:22.053701+0000","last_unstale":"2024-05-27T07:11:22.708261+0000","last_undegraded":"2024-05-27T07:11:22.708261+0000","last_fullsized":"2024-05-27T07:11:22.708261+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:22:31.041730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.697404+0000","last_change":"2024-05-27T07:08:22.062745+0000","last_active":"2024-05-27T07:11:22.697404+0000","last_peered":"2024-05-27T07:11:22.697404+0000","last_clean":"2024-05-27T07:11:22.697404+0000","last_became_active":"2024-05-27T07:08:22.062339+0000","last_became_peered":"2024-05-27T07:08:22.062339+0000","last_unstale":"2024-05-27T07:11:22.697404+0000","last_undegraded":"2024-05-27T07:11:22.697404+0000","last_fullsized":"2024-05-27T07:11:22.697404+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:58:02.934021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.703314+0000","last_change":"2024-05-27T07:08:22.052621+0000","last_active":"2024-05-27T07:11:22.703314+0000","last_peered":"2024-05-27T07:11:22.703314+0000","last_clean":"2024-05-27T07:11:22.703314+0000","last_became_active":"2024-05-27T07:08:22.052070+0000","last_became_peered":"2024-05-27T07:08:22.052070+0000","last_unstale":"2024-05-27T07:11:22.703314+0000","last_undegraded":"2024-05-27T07:11:22.703314+0000","last_fullsized":"2024-05-27T07:11:22.703314+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:34.491068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.707876+0000","last_change":"2024-05-27T07:08:22.055704+0000","last_active":"2024-05-27T07:11:22.707876+0000","last_peered":"2024-05-27T07:11:22.707876+0000","last_clean":"2024-05-27T07:11:22.707876+0000","last_became_active":"2024-05-27T07:08:22.055251+0000","last_became_peered":"2024-05-27T07:08:22.055251+0000","last_unstale":"2024-05-27T07:11:22.707876+0000","last_undegraded":"2024-05-27T07:11:22.707876+0000","last_fullsized":"2024-05-27T07:11:22.707876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:28.074825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.712118+0000","last_change":"2024-05-27T07:08:22.052160+0000","last_active":"2024-05-27T07:11:22.712118+0000","last_peered":"2024-05-27T07:11:22.712118+0000","last_clean":"2024-05-27T07:11:22.712118+0000","last_became_active":"2024-05-27T07:08:22.049495+0000","last_became_peered":"2024-05-27T07:08:22.049495+0000","last_unstale":"2024-05-27T07:11:22.712118+0000","last_undegraded":"2024-05-27T07:11:22.712118+0000","last_fullsized":"2024-05-27T07:11:22.712118+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:54:59.595031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.711205+0000","last_change":"2024-05-27T07:08:22.053828+0000","last_active":"2024-05-27T07:11:22.711205+0000","last_peered":"2024-05-27T07:11:22.711205+0000","last_clean":"2024-05-27T07:11:22.711205+0000","last_became_active":"2024-05-27T07:08:22.053491+0000","last_became_peered":"2024-05-27T07:08:22.053491+0000","last_unstale":"2024-05-27T07:11:22.711205+0000","last_undegraded":"2024-05-27T07:11:22.711205+0000","last_fullsized":"2024-05-27T07:11:22.711205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:20.568772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.715919+0000","last_change":"2024-05-27T07:08:22.050791+0000","last_active":"2024-05-27T07:11:22.715919+0000","last_peered":"2024-05-27T07:11:22.715919+0000","last_clean":"2024-05-27T07:11:22.715919+0000","last_became_active":"2024-05-27T07:08:22.050323+0000","last_became_peered":"2024-05-27T07:08:22.050323+0000","last_unstale":"2024-05-27T07:11:22.715919+0000","last_undegraded":"2024-05-27T07:11:22.715919+0000","last_fullsized":"2024-05-27T07:11:22.715919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:32.400758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.710462+0000","last_change":"2024-05-27T07:08:22.041286+0000","last_active":"2024-05-27T07:11:22.710462+0000","last_peered":"2024-05-27T07:11:22.710462+0000","last_clean":"2024-05-27T07:11:22.710462+0000","last_became_active":"2024-05-27T07:08:22.040811+0000","last_became_peered":"2024-05-27T07:08:22.040811+0000","last_unstale":"2024-05-27T07:11:22.710462+0000","last_undegraded":"2024-05-27T07:11:22.710462+0000","last_fullsized":"2024-05-27T07:11:22.710462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:40:09.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-05-27T07:11:22.690607+0000","last_change":"2024-05-27T07:08:22.066490+0000","last_active":"2024-05-27T07:11:22.690607+0000","last_peered":"2024-05-27T07:11:22.690607+0000","last_clean":"2024-05-27T07:11:22.690607+0000","last_became_active":"2024-05-27T07:08:22.064779+0000","last_became_peered":"2024-05-27T07:08:22.064779+0000","last_unstale":"2024-05-27T07:11:22.690607+0000","last_undegraded":"2024-05-27T07:11:22.690607+0000","last_fullsized":"2024-05-27T07:11:22.690607+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:23.967856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71871,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1307,"num_read_kb":734,"num_write":258,"num_write_kb":150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":215613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678081,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":519893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874312,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1736704,"data_stored":1170400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037838,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1069056,"data_stored":522693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234071,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":518281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430303,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32796,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712356,"statfs":{"total":95995035648,"available":95961452544,"internally_reserved":0,"allocated":1658880,"data_stored":1115613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593831,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":514101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":579516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738515,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":516089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411460,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":515883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":513809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953588,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":583092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790062,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1110663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:12:02.188 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-27T07:12:02.188 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-27T07:12:02.188 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-27T07:12:02.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-05-27T07:12:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:01 smithi179 ceph-mon[35888]: pgmap v504: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:02.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:01 smithi132 ceph-mon[35283]: pgmap v504: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:02.709 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:03.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:02 smithi043 ceph-mon[31531]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:12:03.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:02 smithi179 ceph-mon[35888]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:12:03.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:02 smithi132 ceph-mon[35283]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:12:04.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:03 smithi043 ceph-mon[31531]: pgmap v505: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:04.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:03 smithi179 ceph-mon[35888]: pgmap v505: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:03 smithi132 ceph-mon[35283]: pgmap v505: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.157+0000 7f03a9c00700 1 -- 172.21.15.43:0/3769214063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03a4075ad0 msgr2=0x7f03a4075ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.157+0000 7f03a9c00700 1 --2- 172.21.15.43:0/3769214063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03a4075ad0 0x7f03a4075ee0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0394009990 tx=0x7f0394009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.157+0000 7f03a9c00700 1 -- 172.21.15.43:0/3769214063 shutdown_connections 2024-05-27T07:12:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.157+0000 7f03a9c00700 1 --2- 172.21.15.43:0/3769214063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03a4110c30 0x7f03a4113050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.157+0000 7f03a9c00700 1 --2- 172.21.15.43:0/3769214063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03a4075ad0 0x7f03a4075ee0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.157+0000 7f03a9c00700 1 --2- 172.21.15.43:0/3769214063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03a4076420 0x7f03a41106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.157+0000 7f03a9c00700 1 -- 172.21.15.43:0/3769214063 >> 172.21.15.43:0/3769214063 conn(0x7f03a407af60 msgr2=0x7f03a407b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.157+0000 7f03a9c00700 1 -- 172.21.15.43:0/3769214063 shutdown_connections 2024-05-27T07:12:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.157+0000 7f03a9c00700 1 -- 172.21.15.43:0/3769214063 wait complete. 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.158+0000 7f03a9c00700 1 Processor -- start 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.158+0000 7f03a9c00700 1 -- start start 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a9c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03a4075ad0 0x7f03a419d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a9c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03a4076420 0x7f03a419d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a9c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03a4110c30 0x7f03a41a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a9c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03a4116670 con 0x7f03a4110c30 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a9c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f03a41164f0 con 0x7f03a4076420 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a9c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f03a41167f0 con 0x7f03a4075ad0 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03a4110c30 0x7f03a41a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03a4110c30 0x7f03a41a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38174/0 (socket says 172.21.15.43:38174) 2024-05-27T07:12:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a3400700 1 -- 172.21.15.43:0/3162693623 learned_addr learned my addr 172.21.15.43:0/3162693623 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:05.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a2a00700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03a4075ad0 0x7f03a419d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:05.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.159+0000 7f03a2000700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03a4076420 0x7f03a419d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:05.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f03a3400700 1 -- 172.21.15.43:0/3162693623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03a4075ad0 msgr2=0x7f03a419d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f03a3400700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03a4075ad0 0x7f03a419d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f03a3400700 1 -- 172.21.15.43:0/3162693623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03a4076420 msgr2=0x7f03a419d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f03a3400700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03a4076420 0x7f03a419d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f03a3400700 1 -- 172.21.15.43:0/3162693623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a41a1f90 con 0x7f03a4110c30 2024-05-27T07:12:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f03a3400700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03a4110c30 0x7f03a41a18f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b970 tx=0x7f039c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f039b400700 1 -- 172.21.15.43:0/3162693623 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c019070 con 0x7f03a4110c30 2024-05-27T07:12:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f039b400700 1 -- 172.21.15.43:0/3162693623 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f039c00ce40 con 0x7f03a4110c30 2024-05-27T07:12:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f039b400700 1 -- 172.21.15.43:0/3162693623 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c007370 con 0x7f03a4110c30 2024-05-27T07:12:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f03a9c00700 1 -- 172.21.15.43:0/3162693623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a41a2280 con 0x7f03a4110c30 2024-05-27T07:12:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.160+0000 7f03a9c00700 1 -- 172.21.15.43:0/3162693623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a41a9ba0 con 0x7f03a4110c30 2024-05-27T07:12:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.161+0000 7f03a9c00700 1 -- 172.21.15.43:0/3162693623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03a40512b0 con 0x7f03a4110c30 2024-05-27T07:12:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.164+0000 7f039b400700 1 -- 172.21.15.43:0/3162693623 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f039c01c070 con 0x7f03a4110c30 2024-05-27T07:12:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.166+0000 7f039b400700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f03880b2980 0x7f03880b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.166+0000 7f039b400700 1 -- 172.21.15.43:0/3162693623 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f039c0e2260 con 0x7f03a4110c30 2024-05-27T07:12:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.166+0000 7f03a2a00700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f03880b2980 0x7f03880b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.167+0000 7f03a2a00700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f03880b2980 0x7f03880b4e90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f039400b5c0 tx=0x7f0394005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.171+0000 7f039b400700 1 -- 172.21.15.43:0/3162693623 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f039c023070 con 0x7f03a4110c30 2024-05-27T07:12:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.461+0000 7f03a9c00700 1 -- 172.21.15.43:0/3162693623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f03a419e420 con 0x7f03a4110c30 2024-05-27T07:12:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.462+0000 7f039b400700 1 -- 172.21.15.43:0/3162693623 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f039c0a9c20 con 0x7f03a4110c30 2024-05-27T07:12:05.455 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:12:05.455 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-27T07:12:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 -- 172.21.15.43:0/3162693623 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f03880b2980 msgr2=0x7f03880b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f03880b2980 0x7f03880b4e90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f039400b5c0 tx=0x7f0394005fb0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 -- 172.21.15.43:0/3162693623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03a4110c30 msgr2=0x7f03a41a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03a4110c30 0x7f03a41a18f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b970 tx=0x7f039c00bd30 comp rx=0 tx=0).stop 2024-05-27T07:12:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 -- 172.21.15.43:0/3162693623 shutdown_connections 2024-05-27T07:12:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f03880b2980 0x7f03880b4e90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03a4110c30 0x7f03a41a18f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03a4075ad0 0x7f03a419d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 --2- 172.21.15.43:0/3162693623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f03a4076420 0x7f03a419d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 -- 172.21.15.43:0/3162693623 >> 172.21.15.43:0/3162693623 conn(0x7f03a407af60 msgr2=0x7f03a41011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.465+0000 7f03a9c00700 1 -- 172.21.15.43:0/3162693623 shutdown_connections 2024-05-27T07:12:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:05.466+0000 7f03a9c00700 1 -- 172.21.15.43:0/3162693623 wait complete. 2024-05-27T07:12:06.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:05 smithi043 ceph-mon[31531]: pgmap v506: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:06.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:05 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3162693623' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:12:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:05 smithi179 ceph-mon[35888]: pgmap v506: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:05 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3162693623' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:12:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:05 smithi132 ceph-mon[35283]: pgmap v506: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:05 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3162693623' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-27T07:12:06.401 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-27T07:12:06.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:12:06.875 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:08.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:07 smithi043 ceph-mon[31531]: pgmap v507: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:07 smithi179 ceph-mon[35888]: pgmap v507: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:08.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:07 smithi132 ceph-mon[35283]: pgmap v507: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:09.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.180+0000 7f5856a00700 1 -- 172.21.15.43:0/3528482095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f585010b2d0 msgr2=0x7f5850117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:09.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.180+0000 7f5856a00700 1 --2- 172.21.15.43:0/3528482095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f585010b2d0 0x7f5850117800 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f584800b0a0 tx=0x7f584800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:12:09.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.180+0000 7f5856a00700 1 -- 172.21.15.43:0/3528482095 shutdown_connections 2024-05-27T07:12:09.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.180+0000 7f5856a00700 1 --2- 172.21.15.43:0/3528482095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f585010b2d0 0x7f5850117800 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:09.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.180+0000 7f5856a00700 1 --2- 172.21.15.43:0/3528482095 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5850110960 0x7f5850110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:09.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.180+0000 7f5856a00700 1 --2- 172.21.15.43:0/3528482095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f585010a980 0x7f585010ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:09.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.180+0000 7f5856a00700 1 -- 172.21.15.43:0/3528482095 >> 172.21.15.43:0/3528482095 conn(0x7f5850077c70 msgr2=0x7f58500780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:09.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.180+0000 7f5856a00700 1 -- 172.21.15.43:0/3528482095 shutdown_connections 2024-05-27T07:12:09.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.180+0000 7f5856a00700 1 -- 172.21.15.43:0/3528482095 wait complete. 2024-05-27T07:12:09.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.182+0000 7f5856a00700 1 Processor -- start 2024-05-27T07:12:09.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.182+0000 7f5856a00700 1 -- start start 2024-05-27T07:12:09.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.182+0000 7f5856a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f585010a980 0x7f58501a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:09.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.182+0000 7f5856a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f585010b2d0 0x7f58501a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:09.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.182+0000 7f5856a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5850110960 0x7f58501a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:09.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.182+0000 7f5856a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f585011a930 con 0x7f5850110960 2024-05-27T07:12:09.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.182+0000 7f5856a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f585011a7b0 con 0x7f585010b2d0 2024-05-27T07:12:09.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.182+0000 7f5856a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f585011aab0 con 0x7f585010a980 2024-05-27T07:12:09.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f5854a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5850110960 0x7f58501a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:09.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f5854a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5850110960 0x7f58501a5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38192/0 (socket says 172.21.15.43:38192) 2024-05-27T07:12:09.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f5854a00700 1 -- 172.21.15.43:0/1970962669 learned_addr learned my addr 172.21.15.43:0/1970962669 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:09.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f584fe00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f585010a980 0x7f58501a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:09.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f584f400700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f585010b2d0 0x7f58501a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:09.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f5854a00700 1 -- 172.21.15.43:0/1970962669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f585010a980 msgr2=0x7f58501a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:09.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f5854a00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f585010a980 0x7f58501a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:09.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f5854a00700 1 -- 172.21.15.43:0/1970962669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f585010b2d0 msgr2=0x7f58501a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:09.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f5854a00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f585010b2d0 0x7f58501a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f5854a00700 1 -- 172.21.15.43:0/1970962669 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58501a61f0 con 0x7f5850110960 2024-05-27T07:12:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.183+0000 7f584f400700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f585010b2d0 0x7f58501a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.184+0000 7f5854a00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5850110960 0x7f58501a5b50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f584800b830 tx=0x7f5848004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.184+0000 7f584cc00700 1 -- 172.21.15.43:0/1970962669 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848021070 con 0x7f5850110960 2024-05-27T07:12:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.184+0000 7f584cc00700 1 -- 172.21.15.43:0/1970962669 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5848007b80 con 0x7f5850110960 2024-05-27T07:12:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.184+0000 7f584cc00700 1 -- 172.21.15.43:0/1970962669 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58480052a0 con 0x7f5850110960 2024-05-27T07:12:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.184+0000 7f5856a00700 1 -- 172.21.15.43:0/1970962669 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58501a6480 con 0x7f5850110960 2024-05-27T07:12:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.185+0000 7f5856a00700 1 -- 172.21.15.43:0/1970962669 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58501a6960 con 0x7f5850110960 2024-05-27T07:12:09.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.186+0000 7f5856a00700 1 -- 172.21.15.43:0/1970962669 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58500512b0 con 0x7f5850110960 2024-05-27T07:12:09.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.188+0000 7f584cc00700 1 -- 172.21.15.43:0/1970962669 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f5848024070 con 0x7f5850110960 2024-05-27T07:12:09.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.190+0000 7f584cc00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f58340b2880 0x7f58340b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:09.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.190+0000 7f584cc00700 1 -- 172.21.15.43:0/1970962669 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f584800f020 con 0x7f5850110960 2024-05-27T07:12:09.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.190+0000 7f584fe00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f58340b2880 0x7f58340b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:09.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.191+0000 7f584fe00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f58340b2880 0x7f58340b4d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5840005950 tx=0x7f584000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:09.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.195+0000 7f584cc00700 1 -- 172.21.15.43:0/1970962669 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f584802b020 con 0x7f5850110960 2024-05-27T07:12:09.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.484+0000 7f5856a00700 1 -- 172.21.15.43:0/1970962669 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f585002e5e0 con 0x7f5850110960 2024-05-27T07:12:09.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.485+0000 7f584cc00700 1 -- 172.21.15.43:0/1970962669 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7f58480ab610 con 0x7f5850110960 2024-05-27T07:12:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:12:09.478 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:09:57.102577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[24484,34403,34424,34448,34484]},"id":1}]} 2024-05-27T07:12:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.489+0000 7f5856a00700 1 -- 172.21.15.43:0/1970962669 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f58340b2880 msgr2=0x7f58340b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.489+0000 7f5856a00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f58340b2880 0x7f58340b4d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5840005950 tx=0x7f584000b410 comp rx=0 tx=0).stop 2024-05-27T07:12:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.489+0000 7f5856a00700 1 -- 172.21.15.43:0/1970962669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5850110960 msgr2=0x7f58501a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.489+0000 7f5856a00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5850110960 0x7f58501a5b50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f584800b830 tx=0x7f5848004d00 comp rx=0 tx=0).stop 2024-05-27T07:12:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.489+0000 7f5856a00700 1 -- 172.21.15.43:0/1970962669 shutdown_connections 2024-05-27T07:12:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.489+0000 7f5856a00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f58340b2880 0x7f58340b4d90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.489+0000 7f5856a00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5850110960 0x7f58501a5b50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.489+0000 7f5856a00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f585010a980 0x7f58501a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.489+0000 7f5856a00700 1 --2- 172.21.15.43:0/1970962669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f585010b2d0 0x7f58501a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.489+0000 7f5856a00700 1 -- 172.21.15.43:0/1970962669 >> 172.21.15.43:0/1970962669 conn(0x7f5850077c70 msgr2=0x7f585007b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.490+0000 7f5856a00700 1 -- 172.21.15.43:0/1970962669 shutdown_connections 2024-05-27T07:12:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:09.490+0000 7f5856a00700 1 -- 172.21.15.43:0/1970962669 wait complete. 2024-05-27T07:12:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-05-27T07:12:10.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:09 smithi043 ceph-mon[31531]: pgmap v508: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:10.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:09 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1970962669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:12:10.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:09 smithi179 ceph-mon[35888]: pgmap v508: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:10.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:09 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1970962669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:12:10.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:09 smithi132 ceph-mon[35283]: pgmap v508: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:10.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:09 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1970962669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:12:10.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:12:10.916 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:12.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:11 smithi043 ceph-mon[31531]: pgmap v509: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:12.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:11 smithi179 ceph-mon[35888]: pgmap v509: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:11 smithi132 ceph-mon[35283]: pgmap v509: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.289+0000 7f60f4400700 1 -- 172.21.15.43:0/445759398 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60ec108730 msgr2=0x7f60ec110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:13.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.289+0000 7f60f4400700 1 --2- 172.21.15.43:0/445759398 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60ec108730 0x7f60ec110090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f60e40099c0 tx=0x7f60e4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:12:13.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.289+0000 7f60f4400700 1 -- 172.21.15.43:0/445759398 shutdown_connections 2024-05-27T07:12:13.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.289+0000 7f60f4400700 1 --2- 172.21.15.43:0/445759398 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60ec108730 0x7f60ec110090 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:13.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.289+0000 7f60f4400700 1 --2- 172.21.15.43:0/445759398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60ec1034b0 0x7f60ec1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:13.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.289+0000 7f60f4400700 1 --2- 172.21.15.43:0/445759398 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec107de0 0x7f60ec1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:13.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.289+0000 7f60f4400700 1 -- 172.21.15.43:0/445759398 >> 172.21.15.43:0/445759398 conn(0x7f60ec0fedb0 msgr2=0x7f60ec1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:13.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.289+0000 7f60f4400700 1 -- 172.21.15.43:0/445759398 shutdown_connections 2024-05-27T07:12:13.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.289+0000 7f60f4400700 1 -- 172.21.15.43:0/445759398 wait complete. 2024-05-27T07:12:13.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.291+0000 7f60f4400700 1 Processor -- start 2024-05-27T07:12:13.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.291+0000 7f60f4400700 1 -- start start 2024-05-27T07:12:13.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f4400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec1034b0 0x7f60ec19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:13.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f4400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60ec107de0 0x7f60ec19b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:13.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f4400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60ec108730 0x7f60ec19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:13.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f4400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec113230 con 0x7f60ec108730 2024-05-27T07:12:13.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f4400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec1130b0 con 0x7f60ec107de0 2024-05-27T07:12:13.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f4400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec1133b0 con 0x7f60ec1034b0 2024-05-27T07:12:13.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f2400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60ec108730 0x7f60ec19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f2400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60ec108730 0x7f60ec19f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38204/0 (socket says 172.21.15.43:38204) 2024-05-27T07:12:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f2400700 1 -- 172.21.15.43:0/4276071224 learned_addr learned my addr 172.21.15.43:0/4276071224 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f1000700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60ec107de0 0x7f60ec19b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.292+0000 7f60f1a00700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec1034b0 0x7f60ec19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.293+0000 7f60f2400700 1 -- 172.21.15.43:0/4276071224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec1034b0 msgr2=0x7f60ec19ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.293+0000 7f60f2400700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec1034b0 0x7f60ec19ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.293+0000 7f60f2400700 1 -- 172.21.15.43:0/4276071224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60ec107de0 msgr2=0x7f60ec19b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.293+0000 7f60f2400700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60ec107de0 0x7f60ec19b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.293+0000 7f60f2400700 1 -- 172.21.15.43:0/4276071224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ec19fc00 con 0x7f60ec108730 2024-05-27T07:12:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.293+0000 7f60f2400700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60ec108730 0x7f60ec19f560 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009990 tx=0x7f60e400f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.293+0000 7f60ea000700 1 -- 172.21.15.43:0/4276071224 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e400f940 con 0x7f60ec108730 2024-05-27T07:12:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.293+0000 7f60ea000700 1 -- 172.21.15.43:0/4276071224 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f60e4004830 con 0x7f60ec108730 2024-05-27T07:12:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.293+0000 7f60f4400700 1 -- 172.21.15.43:0/4276071224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ec19fe90 con 0x7f60ec108730 2024-05-27T07:12:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.293+0000 7f60ea000700 1 -- 172.21.15.43:0/4276071224 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e40053a0 con 0x7f60ec108730 2024-05-27T07:12:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.294+0000 7f60f4400700 1 -- 172.21.15.43:0/4276071224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ec1a0310 con 0x7f60ec108730 2024-05-27T07:12:13.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.295+0000 7f60f4400700 1 -- 172.21.15.43:0/4276071224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60ec0512b0 con 0x7f60ec108730 2024-05-27T07:12:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.297+0000 7f60ea000700 1 -- 172.21.15.43:0/4276071224 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f60e4005560 con 0x7f60ec108730 2024-05-27T07:12:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.299+0000 7f60ea000700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60d40b2880 0x7f60d40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.299+0000 7f60ea000700 1 -- 172.21.15.43:0/4276071224 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f60e40e3fc0 con 0x7f60ec108730 2024-05-27T07:12:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.299+0000 7f60f1a00700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60d40b2880 0x7f60d40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.300+0000 7f60f1a00700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60d40b2880 0x7f60d40b4d90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f60e00060b0 tx=0x7f60e0007b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:13.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.302+0000 7f60ea000700 1 -- 172.21.15.43:0/4276071224 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f60e4030070 con 0x7f60ec108730 2024-05-27T07:12:13.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.592+0000 7f60f4400700 1 -- 172.21.15.43:0/4276071224 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60ec02e5e0 con 0x7f60ec108730 2024-05-27T07:12:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.594+0000 7f60ea000700 1 -- 172.21.15.43:0/4276071224 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7f60e4013070 con 0x7f60ec108730 2024-05-27T07:12:13.586 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:12:13.586 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:09:57.102577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[24484,34403,34424,34448,34484]},"id":1}]} 2024-05-27T07:12:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 -- 172.21.15.43:0/4276071224 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60d40b2880 msgr2=0x7f60d40b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60d40b2880 0x7f60d40b4d90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f60e00060b0 tx=0x7f60e0007b80 comp rx=0 tx=0).stop 2024-05-27T07:12:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 -- 172.21.15.43:0/4276071224 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60ec108730 msgr2=0x7f60ec19f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60ec108730 0x7f60ec19f560 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f60e4009990 tx=0x7f60e400f510 comp rx=0 tx=0).stop 2024-05-27T07:12:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 -- 172.21.15.43:0/4276071224 shutdown_connections 2024-05-27T07:12:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f60d40b2880 0x7f60d40b4d90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:13.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60ec108730 0x7f60ec19f560 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:13.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec1034b0 0x7f60ec19ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:13.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 --2- 172.21.15.43:0/4276071224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60ec107de0 0x7f60ec19b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:13.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 -- 172.21.15.43:0/4276071224 >> 172.21.15.43:0/4276071224 conn(0x7f60ec0fedb0 msgr2=0x7f60ec06b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:13.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 -- 172.21.15.43:0/4276071224 shutdown_connections 2024-05-27T07:12:13.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:13.597+0000 7f60f4400700 1 -- 172.21.15.43:0/4276071224 wait complete. 2024-05-27T07:12:13.590 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-05-27T07:12:14.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:13 smithi043 ceph-mon[31531]: pgmap v510: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:14.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:13 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4276071224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:12:14.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:13 smithi179 ceph-mon[35888]: pgmap v510: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:14.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:13 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4276071224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:12:14.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:13 smithi132 ceph-mon[35283]: pgmap v510: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:14.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:13 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4276071224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:12:14.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:12:14.915 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:16.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:15 smithi043 ceph-mon[31531]: pgmap v511: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:12:16.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:15 smithi179 ceph-mon[35888]: pgmap v511: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:12:16.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:15 smithi132 ceph-mon[35283]: pgmap v511: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:12:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.313+0000 7f1839000700 1 -- 172.21.15.43:0/3313849208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18341043e0 msgr2=0x7f18341047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.313+0000 7f1839000700 1 --2- 172.21.15.43:0/3313849208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18341043e0 0x7f18341047f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f18240099c0 tx=0x7f1824009cd0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.313+0000 7f1839000700 1 -- 172.21.15.43:0/3313849208 shutdown_connections 2024-05-27T07:12:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.313+0000 7f1839000700 1 --2- 172.21.15.43:0/3313849208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1834109660 0x7f1834110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.313+0000 7f1839000700 1 --2- 172.21.15.43:0/3313849208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f18341043e0 0x7f18341047f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.313+0000 7f1839000700 1 --2- 172.21.15.43:0/3313849208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1834108d10 0x7f1834109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.313+0000 7f1839000700 1 -- 172.21.15.43:0/3313849208 >> 172.21.15.43:0/3313849208 conn(0x7f1834079c00 msgr2=0x7f183407a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.313+0000 7f1839000700 1 -- 172.21.15.43:0/3313849208 shutdown_connections 2024-05-27T07:12:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.313+0000 7f1839000700 1 -- 172.21.15.43:0/3313849208 wait complete. 2024-05-27T07:12:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.315+0000 7f1839000700 1 Processor -- start 2024-05-27T07:12:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.315+0000 7f1839000700 1 -- start start 2024-05-27T07:12:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.315+0000 7f1839000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18341043e0 0x7f183419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.315+0000 7f1839000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1834108d10 0x7f183419d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.315+0000 7f1839000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1834109660 0x7f18341a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.315+0000 7f1839000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1834114190 con 0x7f1834108d10 2024-05-27T07:12:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.315+0000 7f1839000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1834114010 con 0x7f1834109660 2024-05-27T07:12:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.315+0000 7f1839000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1834114310 con 0x7f18341043e0 2024-05-27T07:12:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f182aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1834108d10 0x7f183419d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f182aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1834108d10 0x7f183419d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38370/0 (socket says 172.21.15.43:38370) 2024-05-27T07:12:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f182aa00700 1 -- 172.21.15.43:0/2159047940 learned_addr learned my addr 172.21.15.43:0/2159047940 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f1833400700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18341043e0 0x7f183419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f1833e00700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1834109660 0x7f18341a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f182aa00700 1 -- 172.21.15.43:0/2159047940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18341043e0 msgr2=0x7f183419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f182aa00700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18341043e0 0x7f183419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f182aa00700 1 -- 172.21.15.43:0/2159047940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1834109660 msgr2=0x7f18341a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f182aa00700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1834109660 0x7f18341a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f182aa00700 1 -- 172.21.15.43:0/2159047940 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18341a1c40 con 0x7f1834108d10 2024-05-27T07:12:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.316+0000 7f1833e00700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1834109660 0x7f18341a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.317+0000 7f182aa00700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1834108d10 0x7f183419d3c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f181c00c9e0 tx=0x7f181c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.317+0000 7f1830c00700 1 -- 172.21.15.43:0/2159047940 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c007aa0 con 0x7f1834108d10 2024-05-27T07:12:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.317+0000 7f1830c00700 1 -- 172.21.15.43:0/2159047940 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f181c007c40 con 0x7f1834108d10 2024-05-27T07:12:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.317+0000 7f1839000700 1 -- 172.21.15.43:0/2159047940 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18341a1f30 con 0x7f1834108d10 2024-05-27T07:12:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.317+0000 7f1830c00700 1 -- 172.21.15.43:0/2159047940 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c018660 con 0x7f1834108d10 2024-05-27T07:12:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.317+0000 7f1839000700 1 -- 172.21.15.43:0/2159047940 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18341a2470 con 0x7f1834108d10 2024-05-27T07:12:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.318+0000 7f1839000700 1 -- 172.21.15.43:0/2159047940 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18340512b0 con 0x7f1834108d10 2024-05-27T07:12:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.321+0000 7f1830c00700 1 -- 172.21.15.43:0/2159047940 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f181c00f450 con 0x7f1834108d10 2024-05-27T07:12:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.324+0000 7f1830c00700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f18180b2880 0x7f18180b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.324+0000 7f1830c00700 1 -- 172.21.15.43:0/2159047940 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f181c0e28c0 con 0x7f1834108d10 2024-05-27T07:12:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.324+0000 7f1833400700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f18180b2880 0x7f18180b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.325+0000 7f1833400700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f18180b2880 0x7f18180b4d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1824000c00 tx=0x7f1824005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.327+0000 7f1830c00700 1 -- 172.21.15.43:0/2159047940 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f181c014030 con 0x7f1834108d10 2024-05-27T07:12:17.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.568+0000 7f1839000700 1 -- 172.21.15.43:0/2159047940 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18341a99f0 con 0x7f1834108d10 2024-05-27T07:12:17.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.575+0000 7f1830c00700 1 -- 172.21.15.43:0/2159047940 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24748 (secure 0 0 0) 0x7f181c0aa280 con 0x7f1834108d10 2024-05-27T07:12:17.568 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:12:17.568 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":90,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:09:54.019617+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:12:17.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.579+0000 7f1839000700 1 -- 172.21.15.43:0/2159047940 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f18180b2880 msgr2=0x7f18180b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:17.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.579+0000 7f1839000700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f18180b2880 0x7f18180b4d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1824000c00 tx=0x7f1824005fb0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.579+0000 7f1839000700 1 -- 172.21.15.43:0/2159047940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1834108d10 msgr2=0x7f183419d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:17.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.579+0000 7f1839000700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1834108d10 0x7f183419d3c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f181c00c9e0 tx=0x7f181c00ccf0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.579+0000 7f1839000700 1 -- 172.21.15.43:0/2159047940 shutdown_connections 2024-05-27T07:12:17.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.579+0000 7f1839000700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f18180b2880 0x7f18180b4d90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.579+0000 7f1839000700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1834109660 0x7f18341a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.579+0000 7f1839000700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18341043e0 0x7f183419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.579+0000 7f1839000700 1 --2- 172.21.15.43:0/2159047940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1834108d10 0x7f183419d3c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:17.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.579+0000 7f1839000700 1 -- 172.21.15.43:0/2159047940 >> 172.21.15.43:0/2159047940 conn(0x7f1834079c00 msgr2=0x7f1834101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:17.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.580+0000 7f1839000700 1 -- 172.21.15.43:0/2159047940 shutdown_connections 2024-05-27T07:12:17.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:17.580+0000 7f1839000700 1 -- 172.21.15.43:0/2159047940 wait complete. 2024-05-27T07:12:18.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:17 smithi179 ceph-mon[35888]: pgmap v512: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-27T07:12:18.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:17 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2159047940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:12:18.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:17 smithi132 ceph-mon[35283]: pgmap v512: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-27T07:12:18.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:17 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2159047940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:12:18.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:17 smithi043 ceph-mon[31531]: pgmap v512: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-27T07:12:18.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:17 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2159047940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:12:18.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:12:19.010 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:20.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:19 smithi043 ceph-mon[31531]: pgmap v513: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:12:20.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:19 smithi132 ceph-mon[35283]: pgmap v513: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:12:20.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:19 smithi179 ceph-mon[35888]: pgmap v513: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:12:21.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.280+0000 7f384be00700 1 -- 172.21.15.43:0/3722922316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3844076be0 msgr2=0x7f3844076ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:21.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.280+0000 7f384be00700 1 --2- 172.21.15.43:0/3722922316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3844076be0 0x7f3844076ff0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3838009990 tx=0x7f3838009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.281+0000 7f384be00700 1 -- 172.21.15.43:0/3722922316 shutdown_connections 2024-05-27T07:12:21.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.281+0000 7f384be00700 1 --2- 172.21.15.43:0/3722922316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3844110c00 0x7f3844113020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.281+0000 7f384be00700 1 --2- 172.21.15.43:0/3722922316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3844076be0 0x7f3844076ff0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.281+0000 7f384be00700 1 --2- 172.21.15.43:0/3722922316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3844077530 0x7f38441106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.281+0000 7f384be00700 1 -- 172.21.15.43:0/3722922316 >> 172.21.15.43:0/3722922316 conn(0x7f38440ffee0 msgr2=0x7f3844102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:21.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.281+0000 7f384be00700 1 -- 172.21.15.43:0/3722922316 shutdown_connections 2024-05-27T07:12:21.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.281+0000 7f384be00700 1 -- 172.21.15.43:0/3722922316 wait complete. 2024-05-27T07:12:21.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.282+0000 7f384be00700 1 Processor -- start 2024-05-27T07:12:21.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.282+0000 7f384be00700 1 -- start start 2024-05-27T07:12:21.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.283+0000 7f384be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3844076be0 0x7f384419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:21.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.283+0000 7f384be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3844077530 0x7f384419d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:21.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.283+0000 7f384be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3844110c00 0x7f38441a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:21.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.283+0000 7f384be00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3844116580 con 0x7f3844076be0 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.283+0000 7f384be00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3844116400 con 0x7f3844077530 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.283+0000 7f384be00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3844116700 con 0x7f3844110c00 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.283+0000 7f3849400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3844076be0 0x7f384419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.283+0000 7f3849400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3844076be0 0x7f384419cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38386/0 (socket says 172.21.15.43:38386) 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.283+0000 7f3849400700 1 -- 172.21.15.43:0/1142903872 learned_addr learned my addr 172.21.15.43:0/1142903872 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.284+0000 7f3848a00700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3844077530 0x7f384419d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.284+0000 7f3849e00700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3844110c00 0x7f38441a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.284+0000 7f3849400700 1 -- 172.21.15.43:0/1142903872 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3844110c00 msgr2=0x7f38441a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.284+0000 7f3849400700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3844110c00 0x7f38441a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.284+0000 7f3849400700 1 -- 172.21.15.43:0/1142903872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3844077530 msgr2=0x7f384419d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.284+0000 7f3849400700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3844077530 0x7f384419d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.284+0000 7f3849400700 1 -- 172.21.15.43:0/1142903872 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38441a1e60 con 0x7f3844076be0 2024-05-27T07:12:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.284+0000 7f3849400700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3844076be0 0x7f384419cf90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3838000c00 tx=0x7f3838005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.285+0000 7f3842000700 1 -- 172.21.15.43:0/1142903872 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3838027070 con 0x7f3844076be0 2024-05-27T07:12:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.285+0000 7f3842000700 1 -- 172.21.15.43:0/1142903872 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f383800fb80 con 0x7f3844076be0 2024-05-27T07:12:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.285+0000 7f384be00700 1 -- 172.21.15.43:0/1142903872 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38441a20f0 con 0x7f3844076be0 2024-05-27T07:12:21.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.285+0000 7f3842000700 1 -- 172.21.15.43:0/1142903872 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3838022c30 con 0x7f3844076be0 2024-05-27T07:12:21.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.285+0000 7f384be00700 1 -- 172.21.15.43:0/1142903872 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38441a9a70 con 0x7f3844076be0 2024-05-27T07:12:21.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.286+0000 7f384be00700 1 -- 172.21.15.43:0/1142903872 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3844078320 con 0x7f3844076be0 2024-05-27T07:12:21.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.289+0000 7f3842000700 1 -- 172.21.15.43:0/1142903872 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f383802a070 con 0x7f3844076be0 2024-05-27T07:12:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.290+0000 7f3842000700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f382c0b2900 0x7f382c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.290+0000 7f3842000700 1 -- 172.21.15.43:0/1142903872 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f38380e3800 con 0x7f3844076be0 2024-05-27T07:12:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.290+0000 7f3848a00700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f382c0b2900 0x7f382c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.291+0000 7f3848a00700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f382c0b2900 0x7f382c0b4e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3830005950 tx=0x7f38300058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:21.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.293+0000 7f3842000700 1 -- 172.21.15.43:0/1142903872 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f383805fdf0 con 0x7f3844076be0 2024-05-27T07:12:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.592+0000 7f384be00700 1 -- 172.21.15.43:0/1142903872 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f384419e2f0 con 0x7f3844076be0 2024-05-27T07:12:21.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.594+0000 7f3842000700 1 -- 172.21.15.43:0/1142903872 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7f3838013070 con 0x7f3844076be0 2024-05-27T07:12:21.586 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:12:21.587 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:09:57.102577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[24484,34403,34424,34448,34484]},"id":1}]} 2024-05-27T07:12:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 -- 172.21.15.43:0/1142903872 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f382c0b2900 msgr2=0x7f382c0b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f382c0b2900 0x7f382c0b4e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3830005950 tx=0x7f38300058e0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 -- 172.21.15.43:0/1142903872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3844076be0 msgr2=0x7f384419cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:21.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3844076be0 0x7f384419cf90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3838000c00 tx=0x7f3838005ef0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 -- 172.21.15.43:0/1142903872 shutdown_connections 2024-05-27T07:12:21.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f382c0b2900 0x7f382c0b4e10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3844110c00 0x7f38441a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3844076be0 0x7f384419cf90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 --2- 172.21.15.43:0/1142903872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3844077530 0x7f384419d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:21.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 -- 172.21.15.43:0/1142903872 >> 172.21.15.43:0/1142903872 conn(0x7f38440ffee0 msgr2=0x7f38441005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:21.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 -- 172.21.15.43:0/1142903872 shutdown_connections 2024-05-27T07:12:21.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:21.597+0000 7f384be00700 1 -- 172.21.15.43:0/1142903872 wait complete. 2024-05-27T07:12:21.592 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-05-27T07:12:22.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:21 smithi132 ceph-mon[35283]: pgmap v514: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-27T07:12:22.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:21 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1142903872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:12:22.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:21 smithi043 ceph-mon[31531]: pgmap v514: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-27T07:12:22.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:21 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1142903872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:12:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:21 smithi179 ceph-mon[35888]: pgmap v514: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-05-27T07:12:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:21 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1142903872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:12:22.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T07:08:25.128097+0000', 'modified': '2024-05-27T07:09:57.102577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 34484, 'mds_1': 34448, 'mds_2': 34424, 'mds_3': 34403, 'mds_4': 24484}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14601': {'gid': 14601, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.43:6839/807126288', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6838', 'nonce': 807126288}, {'type': 'v1', 'addr': '172.21.15.43:6839', 'nonce': 807126288}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14613': {'gid': 14613, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.43:6841/2697504369', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6840', 'nonce': 2697504369}, {'type': 'v1', 'addr': '172.21.15.43:6841', 'nonce': 2697504369}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24412': {'gid': 24412, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6833/4280267074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6832', 'nonce': 4280267074}, {'type': 'v1', 'addr': '172.21.15.179:6833', 'nonce': 4280267074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6835/2538262781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6834', 'nonce': 2538262781}, {'type': 'v1', 'addr': '172.21.15.179:6835', 'nonce': 2538262781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6837/4061292168', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6836', 'nonce': 4061292168}, {'type': 'v1', 'addr': '172.21.15.179:6837', 'nonce': 4061292168}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 33, 'addr': '172.21.15.179:6839/93338133', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6838', 'nonce': 93338133}, {'type': 'v1', 'addr': '172.21.15.179:6839', 'nonce': 93338133}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34403': {'gid': 34403, 'name': 'k', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.132:6833/512644396', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6832', 'nonce': 512644396}, {'type': 'v1', 'addr': '172.21.15.132:6833', 'nonce': 512644396}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34424': {'gid': 34424, 'name': 'e', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 36, 'addr': '172.21.15.132:6835/1074601310', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6834', 'nonce': 1074601310}, {'type': 'v1', 'addr': '172.21.15.132:6835', 'nonce': 1074601310}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34448': {'gid': 34448, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 33, 'addr': '172.21.15.132:6837/2777087908', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6836', 'nonce': 2777087908}, {'type': 'v1', 'addr': '172.21.15.132:6837', 'nonce': 2777087908}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34484': {'gid': 34484, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.132:6839/4104367678', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 4104367678}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 4104367678}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34484, 'qdb_cluster': [24484, 34403, 34424, 34448, 34484]} 2024-05-27T07:12:22.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-27T07:12:22.627 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-27T07:12:23.078 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:24.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:23 smithi043 ceph-mon[31531]: pgmap v515: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-05-27T07:12:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:23 smithi132 ceph-mon[35283]: pgmap v515: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-05-27T07:12:24.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:23 smithi179 ceph-mon[35888]: pgmap v515: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-05-27T07:12:25.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.421+0000 7fe72f800700 1 -- 172.21.15.43:0/3318754838 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72810b2d0 msgr2=0x7fe728117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:25.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.421+0000 7fe72f800700 1 --2- 172.21.15.43:0/3318754838 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72810b2d0 0x7fe728117800 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe720009990 tx=0x7fe720009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.421+0000 7fe72f800700 1 -- 172.21.15.43:0/3318754838 shutdown_connections 2024-05-27T07:12:25.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.421+0000 7fe72f800700 1 --2- 172.21.15.43:0/3318754838 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72810b2d0 0x7fe728117800 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.421+0000 7fe72f800700 1 --2- 172.21.15.43:0/3318754838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe728110960 0x7fe728110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.421+0000 7fe72f800700 1 --2- 172.21.15.43:0/3318754838 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe72810a980 0x7fe72810ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.421+0000 7fe72f800700 1 -- 172.21.15.43:0/3318754838 >> 172.21.15.43:0/3318754838 conn(0x7fe728077c70 msgr2=0x7fe7280780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:25.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.422+0000 7fe72f800700 1 -- 172.21.15.43:0/3318754838 shutdown_connections 2024-05-27T07:12:25.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.422+0000 7fe72f800700 1 -- 172.21.15.43:0/3318754838 wait complete. 2024-05-27T07:12:25.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.423+0000 7fe72f800700 1 Processor -- start 2024-05-27T07:12:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.423+0000 7fe72f800700 1 -- start start 2024-05-27T07:12:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe72f800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72810a980 0x7fe7281a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe72f800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe72810b2d0 0x7fe7281a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe72f800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe728110960 0x7fe7281a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe72f800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe72811a980 con 0x7fe72810a980 2024-05-27T07:12:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe72f800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe72811a800 con 0x7fe728110960 2024-05-27T07:12:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe72f800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe72811ab00 con 0x7fe72810b2d0 2024-05-27T07:12:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe72ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72810a980 0x7fe7281a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe72ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72810a980 0x7fe7281a12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56638/0 (socket says 172.21.15.43:56638) 2024-05-27T07:12:25.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe72ce00700 1 -- 172.21.15.43:0/1199288567 learned_addr learned my addr 172.21.15.43:0/1199288567 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe727e00700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe72810b2d0 0x7fe7281a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.424+0000 7fe72d800700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe728110960 0x7fe7281a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.425+0000 7fe72ce00700 1 -- 172.21.15.43:0/1199288567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe72810b2d0 msgr2=0x7fe7281a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.425+0000 7fe72ce00700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe72810b2d0 0x7fe7281a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.425+0000 7fe72ce00700 1 -- 172.21.15.43:0/1199288567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe728110960 msgr2=0x7fe7281a5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.425+0000 7fe72ce00700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe728110960 0x7fe7281a5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.425+0000 7fe72ce00700 1 -- 172.21.15.43:0/1199288567 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7281a60f0 con 0x7fe72810a980 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.425+0000 7fe72ce00700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72810a980 0x7fe7281a12d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe71c007df0 tx=0x7fe71c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.425+0000 7fe725600700 1 -- 172.21.15.43:0/1199288567 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c016020 con 0x7fe72810a980 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.425+0000 7fe725600700 1 -- 172.21.15.43:0/1199288567 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe71c00ad40 con 0x7fe72810a980 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.425+0000 7fe725600700 1 -- 172.21.15.43:0/1199288567 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c011740 con 0x7fe72810a980 2024-05-27T07:12:25.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.425+0000 7fe72f800700 1 -- 172.21.15.43:0/1199288567 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7281a63e0 con 0x7fe72810a980 2024-05-27T07:12:25.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.426+0000 7fe72f800700 1 -- 172.21.15.43:0/1199288567 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7281addb0 con 0x7fe72810a980 2024-05-27T07:12:25.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.429+0000 7fe725600700 1 -- 172.21.15.43:0/1199288567 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fe71c019020 con 0x7fe72810a980 2024-05-27T07:12:25.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.431+0000 7fe73e27fb80 1 Processor -- start 2024-05-27T07:12:25.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.431+0000 7fe73e27fb80 1 -- start start 2024-05-27T07:12:25.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eae820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ebb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4ddcdf0 con 0x55b5a4eaee20 2024-05-27T07:12:25.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4dcc990 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4dd09d0 con 0x55b5a4e1b9a0 2024-05-27T07:12:25.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe725600700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe7100b2900 0x7fe7100b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe727e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ebb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe727e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ebb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56646/0 (socket says 172.21.15.43:56646) 2024-05-27T07:12:25.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe727e00700 1 -- 172.21.15.43:0/465719883 learned_addr learned my addr 172.21.15.43:0/465719883 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:25.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe72d800700 1 --2- 172.21.15.43:0/465719883 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.432+0000 7fe725600700 1 -- 172.21.15.43:0/1199288567 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fe71c0e2300 con 0x7fe72810a980 2024-05-27T07:12:25.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.433+0000 7fe72d800700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe7100b2900 0x7fe7100b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.433+0000 7fe72ce00700 1 --2- 172.21.15.43:0/465719883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.433+0000 7fe727e00700 1 -- 172.21.15.43:0/465719883 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4e1b9a0 msgr2=0x55b5a4eae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:25.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.433+0000 7fe727e00700 1 --2- 172.21.15.43:0/465719883 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eae2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.433+0000 7fe727e00700 1 -- 172.21.15.43:0/465719883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4d2fdf0 msgr2=0x55b5a4eae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:25.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.433+0000 7fe727e00700 1 --2- 172.21.15.43:0/465719883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.433+0000 7fe727e00700 1 -- 172.21.15.43:0/465719883 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a4d953d0 con 0x55b5a4eaee20 2024-05-27T07:12:25.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.434+0000 7fe727e00700 1 --2- 172.21.15.43:0/465719883 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ebb0c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe714008c00 tx=0x7fe714008f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e23125908a192b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:25.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.434+0000 7fe72d800700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe7100b2900 0x7fe7100b4e10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x55b5a4eb0fa0 tx=0x7fe72000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:25.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.434+0000 7fe71a000700 1 -- 172.21.15.43:0/465719883 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe714014070 con 0x55b5a4eaee20 2024-05-27T07:12:25.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.434+0000 7fe71a000700 1 -- 172.21.15.43:0/465719883 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe714004cc0 con 0x55b5a4eaee20 2024-05-27T07:12:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.434+0000 7fe71a000700 1 -- 172.21.15.43:0/465719883 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe714005610 con 0x55b5a4eaee20 2024-05-27T07:12:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.435+0000 7fe73e27fb80 1 -- 172.21.15.43:0/465719883 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 msgr2=0x55b5a4ebb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.435+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/465719883 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ebb0c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe714008c00 tx=0x7fe714008f10 comp rx=0 tx=0).stop 2024-05-27T07:12:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.435+0000 7fe73e27fb80 1 -- 172.21.15.43:0/465719883 shutdown_connections 2024-05-27T07:12:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.435+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/465719883 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ebb0c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.435+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/465719883 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.435+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/465719883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.435+0000 7fe73e27fb80 1 -- 172.21.15.43:0/465719883 >> 172.21.15.43:0/465719883 conn(0x55b5a4dc9340 msgr2=0x55b5a4eaa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.435+0000 7fe73e27fb80 1 -- 172.21.15.43:0/465719883 shutdown_connections 2024-05-27T07:12:25.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.435+0000 7fe73e27fb80 1 -- 172.21.15.43:0/465719883 wait complete. 2024-05-27T07:12:25.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.436+0000 7fe73e27fb80 1 -- start start 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.436+0000 7fe73e27fb80 1 Processor -- start 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.437+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eb8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.437+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.437+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4eaee20 0x55b5a4eb9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.437+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4cd2f50 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.437+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4d6d710 con 0x55b5a4eaee20 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.437+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4ebbb30 con 0x55b5a4e1b9a0 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.437+0000 7fe73e27fb80 10 client.0 _ll_register_callbacks cb 0x55b5a4d76c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.437+0000 7fe72ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eb8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.437+0000 7fe72ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eb8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56652/0 (socket says 172.21.15.43:56652) 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.437+0000 7fe72ce00700 1 -- 172.21.15.43:0/4203937447 learned_addr learned my addr 172.21.15.43:0/4203937447 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:25.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.438+0000 7fe727e00700 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.438+0000 7fe72d800700 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4eaee20 0x55b5a4eb9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.438+0000 7fe72ce00700 1 -- 172.21.15.43:0/4203937447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4e1b9a0 msgr2=0x55b5a4eb8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.438+0000 7fe72ce00700 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.438+0000 7fe72ce00700 1 -- 172.21.15.43:0/4203937447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4eaee20 msgr2=0x55b5a4eb9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.438+0000 7fe72ce00700 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4eaee20 0x55b5a4eb9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.438+0000 7fe72ce00700 1 -- 172.21.15.43:0/4203937447 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a4d1c880 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.438+0000 7fe72ce00700 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eb8590 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe71c007dc0 tx=0x7fe71c0a91e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786e5fc6caec484d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.438+0000 7fe70ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.438+0000 7fe70ea00700 1 -- 172.21.15.43:0/4203937447 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c019520 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.439+0000 7fe70ea00700 1 -- 172.21.15.43:0/4203937447 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fe71c0a97f0 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.439+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4203937447 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe714004cc0 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.439+0000 7fe70ea00700 1 -- 172.21.15.43:0/4203937447 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe71c0ad940 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.439+0000 7fe73e27fb80 10 client.15069 fetch_fsmap learned FSMap version 29 2024-05-27T07:12:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.439+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4203937447 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5a4d9d010 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.439+0000 7fe70ea00700 1 -- 172.21.15.43:0/4203937447 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe714004cc0 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.439+0000 7fe70ea00700 1 -- 172.21.15.43:0/4203937447 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe71c023060 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.439+0000 7fe73e27fb80 10 client.15069 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:12:25.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.439+0000 7fe73e27fb80 10 client.15069 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:12:25.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.440+0000 7fe73e27fb80 20 client.15069 populate_metadata read hostname 'smithi043' 2024-05-27T07:12:25.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.440+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b5a4ec8a60 0x55b5a4ec9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.440+0000 7fe73e27fb80 4 client.15069 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:12:25.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.440+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4203937447 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5a4dded40 con 0x55b5a4ec8a60 2024-05-27T07:12:25.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.440+0000 7fe727e00700 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b5a4ec8a60 0x55b5a4ec9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.441+0000 7fe727e00700 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b5a4ec8a60 0x55b5a4ec9ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=939ab0c5d32ece2f server_cookie=e59626ae333844d in_seq=0 out_seq=0 2024-05-27T07:12:25.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.441+0000 7fe70ea00700 10 client.15069 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:12:25.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe70ea00700 1 -- 172.21.15.43:0/4203937447 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b5a4dded40 con 0x55b5a4ec8a60 2024-05-27T07:12:25.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe70ea00700 10 client.15069 handle_command_reply: tid=0 2024-05-27T07:12:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 client.15069 shutdown 2024-05-27T07:12:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4203937447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 msgr2=0x55b5a4eb8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eb8590 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe71c007dc0 tx=0x7fe71c0a91e0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4203937447 shutdown_connections 2024-05-27T07:12:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b5a4ec8a60 0x55b5a4ec9ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4eaee20 0x55b5a4eb9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eb8590 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4203937447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4203937447 >> 172.21.15.43:0/4203937447 conn(0x55b5a4dc9340 msgr2=0x55b5a4eae2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe70ea00700 0 client.15069 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:12:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4203937447 shutdown_connections 2024-05-27T07:12:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.444+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4203937447 wait complete. 2024-05-27T07:12:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.445+0000 7fe73e27fb80 20 client.15069 trim_cache size 0 max 16384 2024-05-27T07:12:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.468+0000 7fe73e27fb80 1 Processor -- start 2024-05-27T07:12:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.468+0000 7fe73e27fb80 1 -- start start 2024-05-27T07:12:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.468+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eaa610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.468+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.468+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ec4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.468+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4ec8210 con 0x55b5a4eaee20 2024-05-27T07:12:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.468+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4ec88e0 con 0x55b5a4e1b9a0 2024-05-27T07:12:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.468+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4ec8fb0 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe72d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ec4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe72d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ec4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56654/0 (socket says 172.21.15.43:56654) 2024-05-27T07:12:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe72d800700 1 -- 172.21.15.43:0/227158967 learned_addr learned my addr 172.21.15.43:0/227158967 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe72ce00700 1 --2- 172.21.15.43:0/227158967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eaa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe727e00700 1 --2- 172.21.15.43:0/227158967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe72d800700 1 -- 172.21.15.43:0/227158967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4d2fdf0 msgr2=0x55b5a4eaa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe72d800700 1 --2- 172.21.15.43:0/227158967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eaa610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe72d800700 1 -- 172.21.15.43:0/227158967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 msgr2=0x55b5a4eb3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe72d800700 1 --2- 172.21.15.43:0/227158967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe72d800700 1 -- 172.21.15.43:0/227158967 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a4ec5310 con 0x55b5a4eaee20 2024-05-27T07:12:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.469+0000 7fe727e00700 1 --2- 172.21.15.43:0/227158967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.470+0000 7fe72d800700 1 --2- 172.21.15.43:0/227158967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ec4dd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe720015000 tx=0x7fe720005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2849ff6d8351b3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.470+0000 7fe719600700 1 -- 172.21.15.43:0/227158967 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe720034070 con 0x55b5a4eaee20 2024-05-27T07:12:25.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.470+0000 7fe719600700 1 -- 172.21.15.43:0/227158967 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe72000bb80 con 0x55b5a4eaee20 2024-05-27T07:12:25.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.470+0000 7fe719600700 1 -- 172.21.15.43:0/227158967 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72000fb00 con 0x55b5a4eaee20 2024-05-27T07:12:25.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.471+0000 7fe73e27fb80 1 -- 172.21.15.43:0/227158967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 msgr2=0x55b5a4ec4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.471+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/227158967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ec4dd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe720015000 tx=0x7fe720005ee0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.471+0000 7fe73e27fb80 1 -- 172.21.15.43:0/227158967 shutdown_connections 2024-05-27T07:12:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.471+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/227158967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4eaee20 0x55b5a4ec4dd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.471+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/227158967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4eaa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.471+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/227158967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.471+0000 7fe73e27fb80 1 -- 172.21.15.43:0/227158967 >> 172.21.15.43:0/227158967 conn(0x55b5a4dc9340 msgr2=0x55b5a4ea9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.471+0000 7fe73e27fb80 1 -- 172.21.15.43:0/227158967 shutdown_connections 2024-05-27T07:12:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.471+0000 7fe73e27fb80 1 -- 172.21.15.43:0/227158967 wait complete. 2024-05-27T07:12:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.472+0000 7fe73e27fb80 1 -- start start 2024-05-27T07:12:25.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.472+0000 7fe73e27fb80 1 Processor -- start 2024-05-27T07:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4ed3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe73e27fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4eaee20 0x55b5a4eb3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4ec9d50 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4ec9680 con 0x55b5a4e1b9a0 2024-05-27T07:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe73e27fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b5a4eca420 con 0x55b5a4eaee20 2024-05-27T07:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe73e27fb80 10 client.0 _ll_register_callbacks cb 0x55b5a4d76c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe72ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4ed3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe72ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4ed3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56664/0 (socket says 172.21.15.43:56664) 2024-05-27T07:12:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe72ce00700 1 -- 172.21.15.43:0/4066201668 learned_addr learned my addr 172.21.15.43:0/4066201668 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe727e00700 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe72d800700 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4eaee20 0x55b5a4eb3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe72ce00700 1 -- 172.21.15.43:0/4066201668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4eaee20 msgr2=0x55b5a4eb3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe72ce00700 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4eaee20 0x55b5a4eb3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.473+0000 7fe72ce00700 1 -- 172.21.15.43:0/4066201668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 msgr2=0x55b5a4eb2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.474+0000 7fe72ce00700 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.474+0000 7fe72ce00700 1 -- 172.21.15.43:0/4066201668 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a4eb3780 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.474+0000 7fe72d800700 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4eaee20 0x55b5a4eb3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.474+0000 7fe72ce00700 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4ed3990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe71c0a9290 tx=0x7fe71c0a8aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e4045d8c28eae9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.474+0000 7fe70ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.474+0000 7fe70ea00700 1 -- 172.21.15.43:0/4066201668 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c019520 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.474+0000 7fe70ea00700 1 -- 172.21.15.43:0/4066201668 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fe71c08fd10 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.474+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4066201668 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe72000bb80 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.474+0000 7fe70ea00700 1 -- 172.21.15.43:0/4066201668 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe71c0b23f0 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.474+0000 7fe73e27fb80 10 client.15075 fetch_fsmap learned FSMap version 29 2024-05-27T07:12:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.475+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4066201668 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5a4ed01c0 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.475+0000 7fe70ea00700 1 -- 172.21.15.43:0/4066201668 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe72000bb80 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.475+0000 7fe70ea00700 1 -- 172.21.15.43:0/4066201668 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe71c08b030 con 0x55b5a4d2fdf0 2024-05-27T07:12:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.475+0000 7fe73e27fb80 10 client.15075 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:12:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.475+0000 7fe73e27fb80 10 client.15075 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:12:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.475+0000 7fe73e27fb80 20 client.15075 populate_metadata read hostname 'smithi043' 2024-05-27T07:12:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.475+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b5a4f36250 0x55b5a4f376c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:25.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.475+0000 7fe73e27fb80 4 client.15075 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.475+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4066201668 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "status"}) -- 0x55b5a4dd8520 con 0x55b5a4f36250 2024-05-27T07:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.476+0000 7fe727e00700 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b5a4f36250 0x55b5a4f376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.477+0000 7fe727e00700 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b5a4f36250 0x55b5a4f376c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b422d38de4559703 server_cookie=2d94c819cf67d792 in_seq=0 out_seq=0 2024-05-27T07:12:25.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.477+0000 7fe70ea00700 10 client.15075 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.477+0000 7fe70ea00700 1 -- 172.21.15.43:0/4066201668 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b5a4dd8520 con 0x55b5a4f36250 2024-05-27T07:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe70ea00700 10 client.15075 handle_command_reply: tid=0 2024-05-27T07:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 client.15075 shutdown 2024-05-27T07:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4066201668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 msgr2=0x55b5a4ed3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4ed3990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe71c0a9290 tx=0x7fe71c0a8aa0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4066201668 shutdown_connections 2024-05-27T07:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b5a4f36250 0x55b5a4f376c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b5a4eaee20 0x55b5a4eb3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b5a4d2fdf0 0x55b5a4ed3990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 --2- 172.21.15.43:0/4066201668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5a4e1b9a0 0x55b5a4eb2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4066201668 >> 172.21.15.43:0/4066201668 conn(0x55b5a4dc9340 msgr2=0x55b5a4eae060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:25.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe70ea00700 0 client.15075 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:12:25.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4066201668 shutdown_connections 2024-05-27T07:12:25.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 1 -- 172.21.15.43:0/4066201668 wait complete. 2024-05-27T07:12:25.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.478+0000 7fe73e27fb80 20 client.15075 trim_cache size 0 max 16384 2024-05-27T07:12:25.472 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:12:25.472 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:12:25.472 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 4, 2024-05-27T07:12:25.472 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24484, 2024-05-27T07:12:25.472 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:12:25.473 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:12:25.473 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:12:25.473 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 153.47217106400001, 2024-05-27T07:12:25.473 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 29, 2024-05-27T07:12:25.473 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-05-27T07:12:25.473 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 90, 2024-05-27T07:12:25.473 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 277.72526720500002 2024-05-27T07:12:25.473 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 -- 172.21.15.43:0/1199288567 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe7100b2900 msgr2=0x7fe7100b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe7100b2900 0x7fe7100b4e10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x55b5a4eb0fa0 tx=0x7fe72000b540 comp rx=0 tx=0).stop 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 -- 172.21.15.43:0/1199288567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72810a980 msgr2=0x7fe7281a12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72810a980 0x7fe7281a12d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe71c007df0 tx=0x7fe71c00a340 comp rx=0 tx=0).stop 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 -- 172.21.15.43:0/1199288567 shutdown_connections 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe7100b2900 0x7fe7100b4e10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe728110960 0x7fe7281a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe72810a980 0x7fe7281a12d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 --2- 172.21.15.43:0/1199288567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe72810b2d0 0x7fe7281a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 -- 172.21.15.43:0/1199288567 >> 172.21.15.43:0/1199288567 conn(0x7fe728077c70 msgr2=0x7fe72807b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:25.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 -- 172.21.15.43:0/1199288567 shutdown_connections 2024-05-27T07:12:25.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:25.479+0000 7fe72f800700 1 -- 172.21.15.43:0/1199288567 wait complete. 2024-05-27T07:12:26.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:25 smithi132 ceph-mon[35283]: pgmap v516: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:12:26.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:25 smithi043 ceph-mon[31531]: pgmap v516: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:12:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:25 smithi179 ceph-mon[35888]: pgmap v516: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-27T07:12:26.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.k status 2024-05-27T07:12:26.949 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:28.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:28 smithi132 ceph-mon[35283]: pgmap v517: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:28.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:28 smithi043 ceph-mon[31531]: pgmap v517: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:28 smithi179 ceph-mon[35888]: pgmap v517: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:29.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.376+0000 7f0c1ba00700 1 -- 172.21.15.43:0/3499986748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c14075b50 msgr2=0x7f0c14075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:29.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.376+0000 7f0c1ba00700 1 --2- 172.21.15.43:0/3499986748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c14075b50 0x7f0c14075f60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009990 tx=0x7f0c04009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.376+0000 7f0c1ba00700 1 -- 172.21.15.43:0/3499986748 shutdown_connections 2024-05-27T07:12:29.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.376+0000 7f0c1ba00700 1 --2- 172.21.15.43:0/3499986748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c14115360 0x7f0c14117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.376+0000 7f0c1ba00700 1 --2- 172.21.15.43:0/3499986748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c14075b50 0x7f0c14075f60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.376+0000 7f0c1ba00700 1 --2- 172.21.15.43:0/3499986748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c140764a0 0x7f0c14114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.376+0000 7f0c1ba00700 1 -- 172.21.15.43:0/3499986748 >> 172.21.15.43:0/3499986748 conn(0x7f0c14100030 msgr2=0x7f0c14102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:29.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.376+0000 7f0c1ba00700 1 -- 172.21.15.43:0/3499986748 shutdown_connections 2024-05-27T07:12:29.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.377+0000 7f0c1ba00700 1 -- 172.21.15.43:0/3499986748 wait complete. 2024-05-27T07:12:29.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.378+0000 7f0c1ba00700 1 Processor -- start 2024-05-27T07:12:29.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.378+0000 7f0c1ba00700 1 -- start start 2024-05-27T07:12:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.379+0000 7f0c1ba00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c14075b50 0x7f0c141a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.379+0000 7f0c1ba00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c140764a0 0x7f0c141a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.379+0000 7f0c1ba00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c14115360 0x7f0c141a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.379+0000 7f0c1ba00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1411a970 con 0x7f0c140764a0 2024-05-27T07:12:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.379+0000 7f0c1ba00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1411a7f0 con 0x7f0c14115360 2024-05-27T07:12:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.379+0000 7f0c1ba00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1411aaf0 con 0x7f0c14075b50 2024-05-27T07:12:29.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.379+0000 7f0c0be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c140764a0 0x7f0c141a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.379+0000 7f0c0be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c140764a0 0x7f0c141a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56694/0 (socket says 172.21.15.43:56694) 2024-05-27T07:12:29.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.379+0000 7f0c0be00700 1 -- 172.21.15.43:0/668299949 learned_addr learned my addr 172.21.15.43:0/668299949 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:29.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.380+0000 7f0c19000700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c14075b50 0x7f0c141a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.380+0000 7f0c19a00700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c14115360 0x7f0c141a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.380+0000 7f0c0be00700 1 -- 172.21.15.43:0/668299949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c14075b50 msgr2=0x7f0c141a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:29.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.380+0000 7f0c0be00700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c14075b50 0x7f0c141a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.380+0000 7f0c0be00700 1 -- 172.21.15.43:0/668299949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c14115360 msgr2=0x7f0c141a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:29.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.380+0000 7f0c0be00700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c14115360 0x7f0c141a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.380+0000 7f0c0be00700 1 -- 172.21.15.43:0/668299949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c141a6180 con 0x7f0c140764a0 2024-05-27T07:12:29.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.380+0000 7f0c19000700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c14075b50 0x7f0c141a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:29.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.380+0000 7f0c0be00700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c140764a0 0x7f0c141a1900 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00c9b0 tx=0x7f0bfc00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:29.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.381+0000 7f0c12000700 1 -- 172.21.15.43:0/668299949 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc007b00 con 0x7f0c140764a0 2024-05-27T07:12:29.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.381+0000 7f0c1ba00700 1 -- 172.21.15.43:0/668299949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c141a6470 con 0x7f0c140764a0 2024-05-27T07:12:29.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.381+0000 7f0c12000700 1 -- 172.21.15.43:0/668299949 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0bfc007ca0 con 0x7f0c140764a0 2024-05-27T07:12:29.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.381+0000 7f0c12000700 1 -- 172.21.15.43:0/668299949 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc018660 con 0x7f0c140764a0 2024-05-27T07:12:29.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.381+0000 7f0c1ba00700 1 -- 172.21.15.43:0/668299949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c141ade40 con 0x7f0c140764a0 2024-05-27T07:12:29.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.385+0000 7f0c12000700 1 -- 172.21.15.43:0/668299949 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f0bfc00f450 con 0x7f0c140764a0 2024-05-27T07:12:29.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.386+0000 7f0c2a55db80 1 Processor -- start 2024-05-27T07:12:29.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.386+0000 7f0c2a55db80 1 -- start start 2024-05-27T07:12:29.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.386+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0fb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.386+0000 7f0c12000700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0bf80b2900 0x7f0bf80b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.386+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555ddf7cdf0 0x5555de0fb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.386+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de1080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.386+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5555de029df0 con 0x5555de0689a0 2024-05-27T07:12:29.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.386+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5555de019990 con 0x5555ddf7cdf0 2024-05-27T07:12:29.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.386+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5555de01d9d0 con 0x5555de0fbe20 2024-05-27T07:12:29.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c12000700 1 -- 172.21.15.43:0/668299949 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f0bfc0e2600 con 0x7f0c140764a0 2024-05-27T07:12:29.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c19a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0fb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c19a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0fb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56700/0 (socket says 172.21.15.43:56700) 2024-05-27T07:12:29.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c19a00700 1 -- 172.21.15.43:0/1654924893 learned_addr learned my addr 172.21.15.43:0/1654924893 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:29.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c19000700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0bf80b2900 0x7f0bf80b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c0be00700 1 --2- 172.21.15.43:0/1654924893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555ddf7cdf0 0x5555de0fb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c19a00700 1 --2- 172.21.15.43:0/1654924893 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de1080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c19a00700 1 -- 172.21.15.43:0/1654924893 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 msgr2=0x5555de1080c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:29.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c19a00700 1 --2- 172.21.15.43:0/1654924893 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de1080c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c19a00700 1 -- 172.21.15.43:0/1654924893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555ddf7cdf0 msgr2=0x5555de0fb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:29.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.387+0000 7f0c19a00700 1 --2- 172.21.15.43:0/1654924893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555ddf7cdf0 0x5555de0fb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.388+0000 7f0c19a00700 1 -- 172.21.15.43:0/1654924893 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555ddfe23d0 con 0x5555de0689a0 2024-05-27T07:12:29.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.388+0000 7f0c19a00700 1 --2- 172.21.15.43:0/1654924893 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0fb2e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c012840 tx=0x7f0c0c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b0d3ccff827e9b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:29.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.388+0000 7f0c19000700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0bf80b2900 0x7f0bf80b4e10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0c04003760 tx=0x7f0c04005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:29.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.388+0000 7f0c0a000700 1 -- 172.21.15.43:0/1654924893 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c004db0 con 0x5555de0689a0 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.388+0000 7f0c0a000700 1 -- 172.21.15.43:0/1654924893 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c0c011040 con 0x5555de0689a0 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.388+0000 7f0c0a000700 1 -- 172.21.15.43:0/1654924893 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c005340 con 0x5555de0689a0 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.389+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1654924893 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 msgr2=0x5555de0fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.389+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/1654924893 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0fb2e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c012840 tx=0x7f0c0c007f90 comp rx=0 tx=0).stop 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.389+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1654924893 shutdown_connections 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.389+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/1654924893 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de1080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.389+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/1654924893 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0fb2e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.389+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/1654924893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555ddf7cdf0 0x5555de0fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.389+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1654924893 >> 172.21.15.43:0/1654924893 conn(0x5555de016340 msgr2=0x5555de0f7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.390+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1654924893 shutdown_connections 2024-05-27T07:12:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.390+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1654924893 wait complete. 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.390+0000 7f0c2a55db80 1 -- start start 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.390+0000 7f0c2a55db80 1 Processor -- start 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.391+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555ddf7cdf0 0x5555de105630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.391+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555de0689a0 0x5555de105b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.391+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0fbe20 0x5555de1060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.391+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5555ddf1ff50 con 0x5555de0fbe20 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.391+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5555ddfba710 con 0x5555de0689a0 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.391+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5555de108b30 con 0x5555ddf7cdf0 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.391+0000 7f0c2a55db80 10 client.0 _ll_register_callbacks cb 0x5555ddfc3c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.391+0000 7f0c19a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0fbe20 0x5555de1060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.391+0000 7f0c19a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0fbe20 0x5555de1060b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56710/0 (socket says 172.21.15.43:56710) 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.391+0000 7f0c19a00700 1 -- 172.21.15.43:0/650054630 learned_addr learned my addr 172.21.15.43:0/650054630 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:29.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.392+0000 7f0c0be00700 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555de0689a0 0x5555de105b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.392+0000 7f0c19000700 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555ddf7cdf0 0x5555de105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.392+0000 7f0c19a00700 1 -- 172.21.15.43:0/650054630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555ddf7cdf0 msgr2=0x5555de105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.392+0000 7f0c19a00700 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555ddf7cdf0 0x5555de105630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.392+0000 7f0c19a00700 1 -- 172.21.15.43:0/650054630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555de0689a0 msgr2=0x5555de105b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.392+0000 7f0c19a00700 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555de0689a0 0x5555de105b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.392+0000 7f0c19a00700 1 -- 172.21.15.43:0/650054630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555ddf69880 con 0x5555de0fbe20 2024-05-27T07:12:29.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.392+0000 7f0c19a00700 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0fbe20 0x5555de1060b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c01a040 tx=0x7f0c0c007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7963bdb320e97526 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:29.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.392+0000 7f0c02a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:12:29.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.393+0000 7f0c02a00700 1 -- 172.21.15.43:0/650054630 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c028020 con 0x5555de0fbe20 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.393+0000 7f0c02a00700 1 -- 172.21.15.43:0/650054630 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f0c0c015660 con 0x5555de0fbe20 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.393+0000 7f0c2a55db80 1 -- 172.21.15.43:0/650054630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c0c011040 con 0x5555de0fbe20 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.393+0000 7f0c02a00700 1 -- 172.21.15.43:0/650054630 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0c0c033050 con 0x5555de0fbe20 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.393+0000 7f0c2a55db80 10 client.15087 fetch_fsmap learned FSMap version 29 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.393+0000 7f0c2a55db80 1 -- 172.21.15.43:0/650054630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5555ddfea010 con 0x5555de0fbe20 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.394+0000 7f0c02a00700 1 -- 172.21.15.43:0/650054630 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0c0c011040 con 0x5555de0fbe20 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.394+0000 7f0c02a00700 1 -- 172.21.15.43:0/650054630 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0c0c0044d0 con 0x5555de0fbe20 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.394+0000 7f0c2a55db80 10 client.15087 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.394+0000 7f0c2a55db80 10 client.15087 resolve_mds: resolved name 'k' to daemon mds.k 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.394+0000 7f0c2a55db80 20 client.15087 populate_metadata read hostname 'smithi043' 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.394+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5555de115860 0x5555de116cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.394+0000 7f0c2a55db80 4 client.15087 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.394+0000 7f0c2a55db80 1 -- 172.21.15.43:0/650054630 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555de02bd40 con 0x5555de115860 2024-05-27T07:12:29.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.395+0000 7f0c19000700 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5555de115860 0x5555de116cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.396+0000 7f0c19000700 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5555de115860 0x5555de116cd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eb8d00003dc5c3c0 server_cookie=34e97d0046761d38 in_seq=0 out_seq=0 2024-05-27T07:12:29.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.396+0000 7f0c02a00700 10 client.15087 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:12:29.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.398+0000 7f0c02a00700 1 -- 172.21.15.43:0/650054630 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5555de02bd40 con 0x5555de115860 2024-05-27T07:12:29.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.398+0000 7f0c02a00700 10 client.15087 handle_command_reply: tid=0 2024-05-27T07:12:29.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.398+0000 7f0c2a55db80 1 client.15087 shutdown 2024-05-27T07:12:29.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 1 -- 172.21.15.43:0/650054630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0fbe20 msgr2=0x5555de1060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:29.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0fbe20 0x5555de1060b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c01a040 tx=0x7f0c0c007830 comp rx=0 tx=0).stop 2024-05-27T07:12:29.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 1 -- 172.21.15.43:0/650054630 shutdown_connections 2024-05-27T07:12:29.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0fbe20 0x5555de1060b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555ddf7cdf0 0x5555de105630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5555de115860 0x5555de116cd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/650054630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555de0689a0 0x5555de105b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c02a00700 0 client.15087 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:12:29.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 1 -- 172.21.15.43:0/650054630 >> 172.21.15.43:0/650054630 conn(0x5555de016340 msgr2=0x5555de0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:29.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 1 -- 172.21.15.43:0/650054630 shutdown_connections 2024-05-27T07:12:29.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 1 -- 172.21.15.43:0/650054630 wait complete. 2024-05-27T07:12:29.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.399+0000 7f0c2a55db80 20 client.15087 trim_cache size 0 max 16384 2024-05-27T07:12:29.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.424+0000 7f0c2a55db80 1 Processor -- start 2024-05-27T07:12:29.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.424+0000 7f0c2a55db80 1 -- start start 2024-05-27T07:12:29.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.424+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555ddf7cdf0 0x5555de100e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.424+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.424+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de0f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.425+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5555de115010 con 0x5555de0689a0 2024-05-27T07:12:29.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.425+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5555de1156e0 con 0x5555ddf7cdf0 2024-05-27T07:12:29.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.425+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5555de115db0 con 0x5555de0fbe20 2024-05-27T07:12:29.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.425+0000 7f0c0be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.425+0000 7f0c0be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0f6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56716/0 (socket says 172.21.15.43:56716) 2024-05-27T07:12:29.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.425+0000 7f0c0be00700 1 -- 172.21.15.43:0/3365211257 learned_addr learned my addr 172.21.15.43:0/3365211257 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:29.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.425+0000 7f0c19000700 1 --2- 172.21.15.43:0/3365211257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555ddf7cdf0 0x5555de100e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.425+0000 7f0c19a00700 1 --2- 172.21.15.43:0/3365211257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de0f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.425+0000 7f0c0be00700 1 -- 172.21.15.43:0/3365211257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 msgr2=0x5555de0f73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.425+0000 7f0c0be00700 1 --2- 172.21.15.43:0/3365211257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de0f73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.426+0000 7f0c0be00700 1 -- 172.21.15.43:0/3365211257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555ddf7cdf0 msgr2=0x5555de100e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.426+0000 7f0c0be00700 1 --2- 172.21.15.43:0/3365211257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555ddf7cdf0 0x5555de100e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.426+0000 7f0c0be00700 1 -- 172.21.15.43:0/3365211257 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555de0f7fb0 con 0x5555de0689a0 2024-05-27T07:12:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.426+0000 7f0c0be00700 1 --2- 172.21.15.43:0/3365211257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0f6e90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00afd0 tx=0x7f0bfc0a5050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29055d7b9a63e593 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.426+0000 7f0c09600700 1 -- 172.21.15.43:0/3365211257 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc0a9f30 con 0x5555de0689a0 2024-05-27T07:12:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.426+0000 7f0c09600700 1 -- 172.21.15.43:0/3365211257 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0bfc0bcc90 con 0x5555de0689a0 2024-05-27T07:12:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.426+0000 7f0c09600700 1 -- 172.21.15.43:0/3365211257 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc0cc110 con 0x5555de0689a0 2024-05-27T07:12:29.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.427+0000 7f0c2a55db80 1 -- 172.21.15.43:0/3365211257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 msgr2=0x5555de0f6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:29.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.427+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/3365211257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0f6e90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00afd0 tx=0x7f0bfc0a5050 comp rx=0 tx=0).stop 2024-05-27T07:12:29.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.427+0000 7f0c2a55db80 1 -- 172.21.15.43:0/3365211257 shutdown_connections 2024-05-27T07:12:29.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.427+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/3365211257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de0f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.427+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/3365211257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555ddf7cdf0 0x5555de100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.427+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/3365211257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555de0689a0 0x5555de0f6e90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.427+0000 7f0c2a55db80 1 -- 172.21.15.43:0/3365211257 >> 172.21.15.43:0/3365211257 conn(0x5555de016340 msgr2=0x5555de103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.427+0000 7f0c2a55db80 1 -- 172.21.15.43:0/3365211257 shutdown_connections 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.428+0000 7f0c2a55db80 1 -- 172.21.15.43:0/3365211257 wait complete. 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.428+0000 7f0c2a55db80 1 -- start start 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.428+0000 7f0c2a55db80 1 Processor -- start 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.429+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555ddf7cdf0 0x5555de0ffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.429+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555de0689a0 0x5555de100200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.429+0000 7f0c2a55db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de11f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.429+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5555de116b50 con 0x5555ddf7cdf0 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.429+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5555de116480 con 0x5555de0689a0 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.429+0000 7f0c2a55db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5555de117220 con 0x5555de0fbe20 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.429+0000 7f0c2a55db80 10 client.0 _ll_register_callbacks cb 0x5555ddfc3c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:12:29.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.429+0000 7f0c19000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555ddf7cdf0 0x5555de0ffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.429+0000 7f0c19000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555ddf7cdf0 0x5555de0ffcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56720/0 (socket says 172.21.15.43:56720) 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.429+0000 7f0c19000700 1 -- 172.21.15.43:0/1692601991 learned_addr learned my addr 172.21.15.43:0/1692601991 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.430+0000 7f0c0be00700 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555de0689a0 0x5555de100200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.430+0000 7f0c19a00700 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de11f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.430+0000 7f0c19000700 1 -- 172.21.15.43:0/1692601991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 msgr2=0x5555de11f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.430+0000 7f0c19000700 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de11f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.430+0000 7f0c19000700 1 -- 172.21.15.43:0/1692601991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555de0689a0 msgr2=0x5555de100200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.430+0000 7f0c19000700 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555de0689a0 0x5555de100200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.430+0000 7f0c19000700 1 -- 172.21.15.43:0/1692601991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555de11f680 con 0x5555ddf7cdf0 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.430+0000 7f0c19000700 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555ddf7cdf0 0x5555de0ffcc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009990 tx=0x7f0c04009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e12880aad22dfc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.430+0000 7f0c02a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.431+0000 7f0c02a00700 1 -- 172.21.15.43:0/1692601991 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04034070 con 0x5555ddf7cdf0 2024-05-27T07:12:29.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.431+0000 7f0c02a00700 1 -- 172.21.15.43:0/1692601991 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f0c0400f8f0 con 0x5555ddf7cdf0 2024-05-27T07:12:29.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.431+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1692601991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bfc0bcc90 con 0x5555ddf7cdf0 2024-05-27T07:12:29.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.431+0000 7f0c02a00700 1 -- 172.21.15.43:0/1692601991 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0c04040050 con 0x5555ddf7cdf0 2024-05-27T07:12:29.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.431+0000 7f0c2a55db80 10 client.15093 fetch_fsmap learned FSMap version 29 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.431+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1692601991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5555de120870 con 0x5555ddf7cdf0 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.431+0000 7f0c02a00700 1 -- 172.21.15.43:0/1692601991 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0bfc0bcc90 con 0x5555ddf7cdf0 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.432+0000 7f0c02a00700 1 -- 172.21.15.43:0/1692601991 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0c04023410 con 0x5555ddf7cdf0 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.432+0000 7f0c2a55db80 10 client.15093 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.432+0000 7f0c2a55db80 10 client.15093 resolve_mds: resolved name 'k' to daemon mds.k 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.432+0000 7f0c2a55db80 20 client.15093 populate_metadata read hostname 'smithi043' 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.432+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5555de182e00 0x5555de184270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.432+0000 7f0c2a55db80 4 client.15093 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.432+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1692601991 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "status"}) -- 0x5555de025520 con 0x5555de182e00 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.432+0000 7f0c0be00700 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5555de182e00 0x5555de184270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.433+0000 7f0c0be00700 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5555de182e00 0x5555de184270 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=35a47b62acb21b3 server_cookie=4df58348ab276305 in_seq=0 out_seq=0 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.433+0000 7f0c02a00700 10 client.15093 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:12:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c02a00700 1 -- 172.21.15.43:0/1692601991 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5555de025520 con 0x5555de182e00 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c02a00700 10 client.15093 handle_command_reply: tid=0 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c2a55db80 1 client.15093 shutdown 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1692601991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555ddf7cdf0 msgr2=0x5555de0ffcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555ddf7cdf0 0x5555de0ffcc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009990 tx=0x7f0c04009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1692601991 shutdown_connections 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5555de0fbe20 0x5555de11f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5555ddf7cdf0 0x5555de0ffcc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5555de182e00 0x5555de184270 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c2a55db80 1 --2- 172.21.15.43:0/1692601991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5555de0689a0 0x5555de100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1692601991 >> 172.21.15.43:0/1692601991 conn(0x5555de016340 msgr2=0x5555de0fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c02a00700 0 client.15093 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.434+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1692601991 shutdown_connections 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.435+0000 7f0c2a55db80 1 -- 172.21.15.43:0/1692601991 wait complete. 2024-05-27T07:12:29.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.435+0000 7f0c2a55db80 20 client.15093 trim_cache size 0 max 16384 2024-05-27T07:12:29.429 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:12:29.429 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:12:29.429 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 3, 2024-05-27T07:12:29.429 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 34403, 2024-05-27T07:12:29.429 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:12:29.429 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:12:29.429 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:12:29.429 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 158.44598618800001, 2024-05-27T07:12:29.429 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 29, 2024-05-27T07:12:29.429 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-05-27T07:12:29.430 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 90, 2024-05-27T07:12:29.430 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 313.47707726499999 2024-05-27T07:12:29.430 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:12:29.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.435+0000 7f0c1ba00700 1 -- 172.21.15.43:0/668299949 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0bf80b2900 msgr2=0x7f0bf80b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:29.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.435+0000 7f0c1ba00700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0bf80b2900 0x7f0bf80b4e10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0c04003760 tx=0x7f0c04005fd0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.435+0000 7f0c1ba00700 1 -- 172.21.15.43:0/668299949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c140764a0 msgr2=0x7f0c141a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:29.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.435+0000 7f0c1ba00700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c140764a0 0x7f0c141a1900 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00c9b0 tx=0x7f0bfc00cd70 comp rx=0 tx=0).stop 2024-05-27T07:12:29.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.436+0000 7f0c1ba00700 1 -- 172.21.15.43:0/668299949 shutdown_connections 2024-05-27T07:12:29.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.436+0000 7f0c1ba00700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f0bf80b2900 0x7f0bf80b4e10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.436+0000 7f0c1ba00700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c14115360 0x7f0c141a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.436+0000 7f0c1ba00700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c14075b50 0x7f0c141a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.436+0000 7f0c1ba00700 1 --2- 172.21.15.43:0/668299949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0c140764a0 0x7f0c141a1900 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:29.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.436+0000 7f0c1ba00700 1 -- 172.21.15.43:0/668299949 >> 172.21.15.43:0/668299949 conn(0x7f0c14100030 msgr2=0x7f0c141006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:29.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.436+0000 7f0c1ba00700 1 -- 172.21.15.43:0/668299949 shutdown_connections 2024-05-27T07:12:29.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:29.436+0000 7f0c1ba00700 1 -- 172.21.15.43:0/668299949 wait complete. 2024-05-27T07:12:30.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:30 smithi043 ceph-mon[31531]: pgmap v518: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:30.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:30 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:12:30.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:30 smithi132 ceph-mon[35283]: pgmap v518: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:30.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:30 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:12:30.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.e status 2024-05-27T07:12:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:30 smithi179 ceph-mon[35888]: pgmap v518: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:30 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:12:30.854 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:32.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:32 smithi043 ceph-mon[31531]: pgmap v519: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:32.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:32 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:32.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:32 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:32.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:32 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:12:32.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:32 smithi132 ceph-mon[35283]: pgmap v519: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:32.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:32 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:32.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:32 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:32.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:32 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:12:32.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:32 smithi179 ceph-mon[35888]: pgmap v519: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:32.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:32 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:12:32.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:32 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:12:32.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:32 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:12:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.357+0000 7fc9afe00700 1 -- 172.21.15.43:0/1689086868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9a81034b0 msgr2=0x7fc9a81038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.357+0000 7fc9afe00700 1 --2- 172.21.15.43:0/1689086868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9a81034b0 0x7fc9a81038c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0099c0 tx=0x7fc99c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.358+0000 7fc9afe00700 1 -- 172.21.15.43:0/1689086868 shutdown_connections 2024-05-27T07:12:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.358+0000 7fc9afe00700 1 --2- 172.21.15.43:0/1689086868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a8108730 0x7fc9a8110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.358+0000 7fc9afe00700 1 --2- 172.21.15.43:0/1689086868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9a81034b0 0x7fc9a81038c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.358+0000 7fc9afe00700 1 --2- 172.21.15.43:0/1689086868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9a8107de0 0x7fc9a81081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.358+0000 7fc9afe00700 1 -- 172.21.15.43:0/1689086868 >> 172.21.15.43:0/1689086868 conn(0x7fc9a80fedb0 msgr2=0x7fc9a81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:33.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.358+0000 7fc9afe00700 1 -- 172.21.15.43:0/1689086868 shutdown_connections 2024-05-27T07:12:33.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.358+0000 7fc9afe00700 1 -- 172.21.15.43:0/1689086868 wait complete. 2024-05-27T07:12:33.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.359+0000 7fc9afe00700 1 Processor -- start 2024-05-27T07:12:33.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.360+0000 7fc9afe00700 1 -- start start 2024-05-27T07:12:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.360+0000 7fc9afe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9a81034b0 0x7fc9a819ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.360+0000 7fc9afe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9a8107de0 0x7fc9a819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.360+0000 7fc9afe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a8108730 0x7fc9a819f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.360+0000 7fc9afe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a8113210 con 0x7fc9a8107de0 2024-05-27T07:12:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.360+0000 7fc9afe00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a8113090 con 0x7fc9a81034b0 2024-05-27T07:12:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.360+0000 7fc9afe00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a8113390 con 0x7fc9a8108730 2024-05-27T07:12:33.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9aca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9a8107de0 0x7fc9a819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9aca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9a8107de0 0x7fc9a819b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56740/0 (socket says 172.21.15.43:56740) 2024-05-27T07:12:33.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9aca00700 1 -- 172.21.15.43:0/3338234004 learned_addr learned my addr 172.21.15.43:0/3338234004 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:33.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9ad400700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9a81034b0 0x7fc9a819ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9ade00700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a8108730 0x7fc9a819f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9aca00700 1 -- 172.21.15.43:0/3338234004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a8108730 msgr2=0x7fc9a819f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:33.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9aca00700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a8108730 0x7fc9a819f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9aca00700 1 -- 172.21.15.43:0/3338234004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9a81034b0 msgr2=0x7fc9a819ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:33.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9aca00700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9a81034b0 0x7fc9a819ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9aca00700 1 -- 172.21.15.43:0/3338234004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a819fbf0 con 0x7fc9a8107de0 2024-05-27T07:12:33.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.361+0000 7fc9ad400700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9a81034b0 0x7fc9a819ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:33.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.362+0000 7fc9aca00700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9a8107de0 0x7fc9a819b310 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc99400e970 tx=0x7fc99400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:33.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.362+0000 7fc9a6000700 1 -- 172.21.15.43:0/3338234004 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99400cbd0 con 0x7fc9a8107de0 2024-05-27T07:12:33.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.362+0000 7fc9a6000700 1 -- 172.21.15.43:0/3338234004 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc99400cd70 con 0x7fc9a8107de0 2024-05-27T07:12:33.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.362+0000 7fc9afe00700 1 -- 172.21.15.43:0/3338234004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a819fee0 con 0x7fc9a8107de0 2024-05-27T07:12:33.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.362+0000 7fc9a6000700 1 -- 172.21.15.43:0/3338234004 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc994005610 con 0x7fc9a8107de0 2024-05-27T07:12:33.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.362+0000 7fc9afe00700 1 -- 172.21.15.43:0/3338234004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a81a7800 con 0x7fc9a8107de0 2024-05-27T07:12:33.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.367+0000 7fc9a6000700 1 -- 172.21.15.43:0/3338234004 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fc994004750 con 0x7fc9a8107de0 2024-05-27T07:12:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.368+0000 7fc9a6000700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc9900b2880 0x7fc9900b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.369+0000 7fc9be86eb80 1 Processor -- start 2024-05-27T07:12:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.369+0000 7fc9a6000700 1 -- 172.21.15.43:0/3338234004 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fc9940e2920 con 0x7fc9a8107de0 2024-05-27T07:12:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.369+0000 7fc9be86eb80 1 -- start start 2024-05-27T07:12:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.369+0000 7fc9ad400700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc9900b2880 0x7fc9900b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e025f0bdf0 0x55e026094b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e026095090 0x55e02608ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e02608f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e025fb8df0 con 0x55e02608f170 2024-05-27T07:12:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e025fa8990 con 0x55e026095090 2024-05-27T07:12:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e025fac9d0 con 0x55e025f0bdf0 2024-05-27T07:12:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9ad400700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc9900b2880 0x7fc9900b4d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc99c003760 tx=0x7fc99c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9ade00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e02608f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9ade00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e02608f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56754/0 (socket says 172.21.15.43:56754) 2024-05-27T07:12:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9ade00700 1 -- 172.21.15.43:0/3774716496 learned_addr learned my addr 172.21.15.43:0/3774716496 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.370+0000 7fc9aca00700 1 --2- 172.21.15.43:0/3774716496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e026095090 0x55e02608ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.371+0000 7fc9ad400700 1 --2- 172.21.15.43:0/3774716496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e025f0bdf0 0x55e026094b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.371+0000 7fc9ade00700 1 -- 172.21.15.43:0/3774716496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e025f0bdf0 msgr2=0x55e026094b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:33.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.371+0000 7fc9ade00700 1 --2- 172.21.15.43:0/3774716496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e025f0bdf0 0x55e026094b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.371+0000 7fc9ade00700 1 -- 172.21.15.43:0/3774716496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e026095090 msgr2=0x55e02608ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:33.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.371+0000 7fc9ade00700 1 --2- 172.21.15.43:0/3774716496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e026095090 0x55e02608ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.371+0000 7fc9ade00700 1 -- 172.21.15.43:0/3774716496 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e025f713d0 con 0x55e02608f170 2024-05-27T07:12:33.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.371+0000 7fc9ad400700 1 --2- 172.21.15.43:0/3774716496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e025f0bdf0 0x55e026094b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:33.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.371+0000 7fc9ade00700 1 --2- 172.21.15.43:0/3774716496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e02608f580 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009440 tx=0x7fc9a0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314c85ca689601e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:33.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.372+0000 7fc99aa00700 1 -- 172.21.15.43:0/3774716496 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0013750 con 0x55e02608f170 2024-05-27T07:12:33.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.372+0000 7fc99aa00700 1 -- 172.21.15.43:0/3774716496 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9a0013d80 con 0x55e02608f170 2024-05-27T07:12:33.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.372+0000 7fc99aa00700 1 -- 172.21.15.43:0/3774716496 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0015d90 con 0x55e02608f170 2024-05-27T07:12:33.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.373+0000 7fc9be86eb80 1 -- 172.21.15.43:0/3774716496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 msgr2=0x55e02608f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:33.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.373+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/3774716496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e02608f580 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009440 tx=0x7fc9a0009920 comp rx=0 tx=0).stop 2024-05-27T07:12:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.373+0000 7fc9be86eb80 1 -- 172.21.15.43:0/3774716496 shutdown_connections 2024-05-27T07:12:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.373+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/3774716496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e02608f580 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.373+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/3774716496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e025f0bdf0 0x55e026094b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.373+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/3774716496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e026095090 0x55e02608ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.373+0000 7fc9be86eb80 1 -- 172.21.15.43:0/3774716496 >> 172.21.15.43:0/3774716496 conn(0x55e025f6bf70 msgr2=0x55e026085880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.373+0000 7fc9be86eb80 1 -- 172.21.15.43:0/3774716496 shutdown_connections 2024-05-27T07:12:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.373+0000 7fc9be86eb80 1 -- 172.21.15.43:0/3774716496 wait complete. 2024-05-27T07:12:33.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.374+0000 7fc9be86eb80 1 -- start start 2024-05-27T07:12:33.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.374+0000 7fc9be86eb80 1 Processor -- start 2024-05-27T07:12:33.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.375+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 0x55e026098920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.375+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026098e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.375+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e0260993a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.375+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e025eaef50 con 0x55e02608f170 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.375+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e025f49710 con 0x55e025f0bdf0 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.375+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e02609bb70 con 0x55e026095090 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.375+0000 7fc9be86eb80 10 client.0 _ll_register_callbacks cb 0x55e025f52c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.376+0000 7fc9aca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026098e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.376+0000 7fc9aca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026098e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56766/0 (socket says 172.21.15.43:56766) 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.376+0000 7fc9aca00700 1 -- 172.21.15.43:0/136983077 learned_addr learned my addr 172.21.15.43:0/136983077 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.376+0000 7fc9ad400700 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 0x55e026098920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.376+0000 7fc9ade00700 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e0260993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.376+0000 7fc9aca00700 1 -- 172.21.15.43:0/136983077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 msgr2=0x55e0260993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.376+0000 7fc9aca00700 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e0260993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.376+0000 7fc9aca00700 1 -- 172.21.15.43:0/136983077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 msgr2=0x55e026098920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.376+0000 7fc9aca00700 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 0x55e026098920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.376+0000 7fc9aca00700 1 -- 172.21.15.43:0/136983077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e025ef8880 con 0x55e02608f170 2024-05-27T07:12:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.377+0000 7fc9aca00700 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026098e60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc9940a5bd0 tx=0x7fc9940a5360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4dd8e20e7d5b28 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.377+0000 7fc98f400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:12:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.377+0000 7fc98f400700 1 -- 172.21.15.43:0/136983077 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc994016540 con 0x55e02608f170 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.377+0000 7fc98f400700 1 -- 172.21.15.43:0/136983077 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fc9940a6140 con 0x55e02608f170 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.377+0000 7fc9be86eb80 1 -- 172.21.15.43:0/136983077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9a0013d80 con 0x55e02608f170 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.377+0000 7fc98f400700 1 -- 172.21.15.43:0/136983077 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc9940aa250 con 0x55e02608f170 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.377+0000 7fc9be86eb80 10 client.15105 fetch_fsmap learned FSMap version 29 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.377+0000 7fc9be86eb80 1 -- 172.21.15.43:0/136983077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e025f79010 con 0x55e02608f170 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.378+0000 7fc98f400700 1 -- 172.21.15.43:0/136983077 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9a0013d80 con 0x55e02608f170 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.378+0000 7fc98f400700 1 -- 172.21.15.43:0/136983077 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc9940a9e40 con 0x55e02608f170 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.378+0000 7fc9be86eb80 10 client.15105 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.378+0000 7fc9be86eb80 10 client.15105 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.378+0000 7fc9be86eb80 20 client.15105 populate_metadata read hostname 'smithi043' 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.378+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55e0260a8ae0 0x55e0260a9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.378+0000 7fc9be86eb80 4 client.15105 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:12:33.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.378+0000 7fc9be86eb80 1 -- 172.21.15.43:0/136983077 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e025fbad40 con 0x55e0260a8ae0 2024-05-27T07:12:33.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.379+0000 7fc9ad400700 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55e0260a8ae0 0x55e0260a9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.380+0000 7fc9ad400700 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55e0260a8ae0 0x55e0260a9f50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=484e8223934b1987 server_cookie=9922d186154a8b5b in_seq=0 out_seq=0 2024-05-27T07:12:33.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.380+0000 7fc98f400700 10 client.15105 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:12:33.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.382+0000 7fc98f400700 1 -- 172.21.15.43:0/136983077 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e025fbad40 con 0x55e0260a8ae0 2024-05-27T07:12:33.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.382+0000 7fc98f400700 10 client.15105 handle_command_reply: tid=0 2024-05-27T07:12:33.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.382+0000 7fc9be86eb80 1 client.15105 shutdown 2024-05-27T07:12:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc9be86eb80 1 -- 172.21.15.43:0/136983077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 msgr2=0x55e026098e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026098e60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc9940a5bd0 tx=0x7fc9940a5360 comp rx=0 tx=0).stop 2024-05-27T07:12:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc9be86eb80 1 -- 172.21.15.43:0/136983077 shutdown_connections 2024-05-27T07:12:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55e0260a8ae0 0x55e0260a9f50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e0260993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 0x55e026098920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/136983077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026098e60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc9be86eb80 1 -- 172.21.15.43:0/136983077 >> 172.21.15.43:0/136983077 conn(0x55e025f6bf70 msgr2=0x55e026085530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:33.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc98f400700 0 client.15105 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:12:33.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc9be86eb80 1 -- 172.21.15.43:0/136983077 shutdown_connections 2024-05-27T07:12:33.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.383+0000 7fc9be86eb80 1 -- 172.21.15.43:0/136983077 wait complete. 2024-05-27T07:12:33.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.384+0000 7fc9be86eb80 20 client.15105 trim_cache size 0 max 16384 2024-05-27T07:12:33.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.407+0000 7fc9be86eb80 1 Processor -- start 2024-05-27T07:12:33.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.408+0000 7fc9be86eb80 1 -- start start 2024-05-27T07:12:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.408+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e025f0bdf0 0x55e026093a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.408+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e02608f170 0x55e026093fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.408+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e026089810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.408+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e0260a8290 con 0x55e025f0bdf0 2024-05-27T07:12:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.408+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e0260a8960 con 0x55e02608f170 2024-05-27T07:12:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.408+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e0260a9030 con 0x55e026095090 2024-05-27T07:12:33.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.408+0000 7fc9ad400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e025f0bdf0 0x55e026093a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.409+0000 7fc9ad400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e025f0bdf0 0x55e026093a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56782/0 (socket says 172.21.15.43:56782) 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.409+0000 7fc9ad400700 1 -- 172.21.15.43:0/3121104497 learned_addr learned my addr 172.21.15.43:0/3121104497 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.409+0000 7fc9aca00700 1 --2- 172.21.15.43:0/3121104497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e02608f170 0x55e026093fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.409+0000 7fc9ade00700 1 --2- 172.21.15.43:0/3121104497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e026089810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.409+0000 7fc9ad400700 1 -- 172.21.15.43:0/3121104497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 msgr2=0x55e026089810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.409+0000 7fc9ad400700 1 --2- 172.21.15.43:0/3121104497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e026089810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.409+0000 7fc9ad400700 1 -- 172.21.15.43:0/3121104497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e02608f170 msgr2=0x55e026093fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.409+0000 7fc9ad400700 1 --2- 172.21.15.43:0/3121104497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e02608f170 0x55e026093fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.409+0000 7fc9ad400700 1 -- 172.21.15.43:0/3121104497 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e026087150 con 0x55e025f0bdf0 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.409+0000 7fc9ade00700 1 --2- 172.21.15.43:0/3121104497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e026089810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:12:33.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.410+0000 7fc9ad400700 1 --2- 172.21.15.43:0/3121104497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e025f0bdf0 0x55e026093a80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0098e0 tx=0x7fc99c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edfa804bfa33ce5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:33.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.410+0000 7fc99a000700 1 -- 172.21.15.43:0/3121104497 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00ba50 con 0x55e025f0bdf0 2024-05-27T07:12:33.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.410+0000 7fc99a000700 1 -- 172.21.15.43:0/3121104497 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc99c00bbf0 con 0x55e025f0bdf0 2024-05-27T07:12:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.410+0000 7fc99a000700 1 -- 172.21.15.43:0/3121104497 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c025420 con 0x55e025f0bdf0 2024-05-27T07:12:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.411+0000 7fc9be86eb80 1 -- 172.21.15.43:0/3121104497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e025f0bdf0 msgr2=0x55e026093a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.411+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/3121104497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e025f0bdf0 0x55e026093a80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0098e0 tx=0x7fc99c01e000 comp rx=0 tx=0).stop 2024-05-27T07:12:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.411+0000 7fc9be86eb80 1 -- 172.21.15.43:0/3121104497 shutdown_connections 2024-05-27T07:12:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.411+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/3121104497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e026089810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.411+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/3121104497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e025f0bdf0 0x55e026093a80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.411+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/3121104497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e02608f170 0x55e026093fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.411+0000 7fc9be86eb80 1 -- 172.21.15.43:0/3121104497 >> 172.21.15.43:0/3121104497 conn(0x55e025f6bf70 msgr2=0x55e026097210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.411+0000 7fc9be86eb80 1 -- 172.21.15.43:0/3121104497 shutdown_connections 2024-05-27T07:12:33.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.411+0000 7fc9be86eb80 1 -- 172.21.15.43:0/3121104497 wait complete. 2024-05-27T07:12:33.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.412+0000 7fc9be86eb80 1 -- start start 2024-05-27T07:12:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.412+0000 7fc9be86eb80 1 Processor -- start 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.413+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 0x55e0260b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.413+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026099ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.413+0000 7fc9be86eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e02609a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.413+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e0260a9dd0 con 0x55e02608f170 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.413+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e0260a9700 con 0x55e025f0bdf0 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.413+0000 7fc9be86eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e0260aa4a0 con 0x55e026095090 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.413+0000 7fc9be86eb80 10 client.0 _ll_register_callbacks cb 0x55e025f52c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.413+0000 7fc9aca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026099ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.413+0000 7fc9aca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026099ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56798/0 (socket says 172.21.15.43:56798) 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.413+0000 7fc9aca00700 1 -- 172.21.15.43:0/32662242 learned_addr learned my addr 172.21.15.43:0/32662242 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc9ad400700 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 0x55e0260b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc9ade00700 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e02609a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc9aca00700 1 -- 172.21.15.43:0/32662242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 msgr2=0x55e02609a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc9aca00700 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e02609a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc9aca00700 1 -- 172.21.15.43:0/32662242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 msgr2=0x55e0260b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc9aca00700 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 0x55e0260b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc9aca00700 1 -- 172.21.15.43:0/32662242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e02609ab00 con 0x55e02608f170 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc9ade00700 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e02609a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc9ad400700 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 0x55e0260b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc9aca00700 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026099ec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc994009230 tx=0x7fc9940a5820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4447c9727900ebf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.414+0000 7fc98f400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.415+0000 7fc98f400700 1 -- 172.21.15.43:0/32662242 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc994016540 con 0x55e02608f170 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.415+0000 7fc98f400700 1 -- 172.21.15.43:0/32662242 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fc9940a6140 con 0x55e02608f170 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.415+0000 7fc9be86eb80 1 -- 172.21.15.43:0/32662242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc99c00bbf0 con 0x55e02608f170 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.415+0000 7fc98f400700 1 -- 172.21.15.43:0/32662242 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc99408f050 con 0x55e02608f170 2024-05-27T07:12:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.415+0000 7fc9be86eb80 10 client.15111 fetch_fsmap learned FSMap version 29 2024-05-27T07:12:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.415+0000 7fc9be86eb80 1 -- 172.21.15.43:0/32662242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e0260b2270 con 0x55e02608f170 2024-05-27T07:12:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.415+0000 7fc98f400700 1 -- 172.21.15.43:0/32662242 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc99c00bbf0 con 0x55e02608f170 2024-05-27T07:12:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.416+0000 7fc98f400700 1 -- 172.21.15.43:0/32662242 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc9940a8a30 con 0x55e02608f170 2024-05-27T07:12:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.416+0000 7fc9be86eb80 10 client.15111 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:12:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.416+0000 7fc9be86eb80 10 client.15111 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:12:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.416+0000 7fc9be86eb80 20 client.15111 populate_metadata read hostname 'smithi043' 2024-05-27T07:12:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.416+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55e0261164b0 0x55e026117920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.416+0000 7fc9be86eb80 4 client.15111 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:12:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.416+0000 7fc9be86eb80 1 -- 172.21.15.43:0/32662242 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "status"}) -- 0x55e025fb4520 con 0x55e0261164b0 2024-05-27T07:12:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.416+0000 7fc9ad400700 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55e0261164b0 0x55e026117920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.418+0000 7fc9ad400700 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55e0261164b0 0x55e026117920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4575bc77537e446 server_cookie=804c5b4328c7f983 in_seq=0 out_seq=0 2024-05-27T07:12:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.418+0000 7fc98f400700 10 client.15111 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:12:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.418+0000 7fc98f400700 1 -- 172.21.15.43:0/32662242 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55e025fb4520 con 0x55e0261164b0 2024-05-27T07:12:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.418+0000 7fc98f400700 10 client.15111 handle_command_reply: tid=0 2024-05-27T07:12:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.418+0000 7fc9be86eb80 1 client.15111 shutdown 2024-05-27T07:12:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.418+0000 7fc9be86eb80 1 -- 172.21.15.43:0/32662242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 msgr2=0x55e026099ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.418+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026099ec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc994009230 tx=0x7fc9940a5820 comp rx=0 tx=0).stop 2024-05-27T07:12:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.419+0000 7fc9be86eb80 1 -- 172.21.15.43:0/32662242 shutdown_connections 2024-05-27T07:12:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.419+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55e0261164b0 0x55e026117920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.419+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e026095090 0x55e02609a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.419+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e025f0bdf0 0x55e0260b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.419+0000 7fc98f400700 0 client.15111 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:12:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.419+0000 7fc9be86eb80 1 --2- 172.21.15.43:0/32662242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e02608f170 0x55e026099ec0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.419+0000 7fc9be86eb80 1 -- 172.21.15.43:0/32662242 >> 172.21.15.43:0/32662242 conn(0x55e025f6bf70 msgr2=0x55e026096e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.419+0000 7fc9be86eb80 1 -- 172.21.15.43:0/32662242 shutdown_connections 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.419+0000 7fc9be86eb80 1 -- 172.21.15.43:0/32662242 wait complete. 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.419+0000 7fc9be86eb80 20 client.15111 trim_cache size 0 max 16384 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 2, 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 34424, 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 163.45498836499999, 2024-05-27T07:12:33.413 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 29, 2024-05-27T07:12:33.414 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-05-27T07:12:33.414 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 90, 2024-05-27T07:12:33.414 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 303.05176596500002 2024-05-27T07:12:33.414 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:12:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 -- 172.21.15.43:0/3338234004 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc9900b2880 msgr2=0x7fc9900b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc9900b2880 0x7fc9900b4d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc99c003760 tx=0x7fc99c01a040 comp rx=0 tx=0).stop 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 -- 172.21.15.43:0/3338234004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9a8107de0 msgr2=0x7fc9a819b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9a8107de0 0x7fc9a819b310 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc99400e970 tx=0x7fc99400ed30 comp rx=0 tx=0).stop 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 -- 172.21.15.43:0/3338234004 shutdown_connections 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc9900b2880 0x7fc9900b4d90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a8108730 0x7fc9a819f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9a81034b0 0x7fc9a819ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 --2- 172.21.15.43:0/3338234004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc9a8107de0 0x7fc9a819b310 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 -- 172.21.15.43:0/3338234004 >> 172.21.15.43:0/3338234004 conn(0x7fc9a80fedb0 msgr2=0x7fc9a806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 -- 172.21.15.43:0/3338234004 shutdown_connections 2024-05-27T07:12:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:33.420+0000 7fc9afe00700 1 -- 172.21.15.43:0/3338234004 wait complete. 2024-05-27T07:12:34.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:34 smithi043 ceph-mon[31531]: pgmap v520: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:34.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-05-27T07:12:34.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:34 smithi132 ceph-mon[35283]: pgmap v520: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:34.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:34 smithi179 ceph-mon[35888]: pgmap v520: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:34.754 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:36 smithi132 ceph-mon[35283]: pgmap v521: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:36.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:36 smithi043 ceph-mon[31531]: pgmap v521: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:36.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:36 smithi179 ceph-mon[35888]: pgmap v521: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:37.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.090+0000 7fc155000700 1 -- 172.21.15.43:0/1493252861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc15010af40 msgr2=0x7fc15010b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:37.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.090+0000 7fc155000700 1 --2- 172.21.15.43:0/1493252861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc15010af40 0x7fc15010b350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc1380099c0 tx=0x7fc138009cd0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.090+0000 7fc155000700 1 -- 172.21.15.43:0/1493252861 shutdown_connections 2024-05-27T07:12:37.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.090+0000 7fc155000700 1 --2- 172.21.15.43:0/1493252861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc15010b890 0x7fc1501131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.090+0000 7fc155000700 1 --2- 172.21.15.43:0/1493252861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc150106610 0x7fc150106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.090+0000 7fc155000700 1 --2- 172.21.15.43:0/1493252861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc15010af40 0x7fc15010b350 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.090+0000 7fc155000700 1 -- 172.21.15.43:0/1493252861 >> 172.21.15.43:0/1493252861 conn(0x7fc150102020 msgr2=0x7fc150104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:37.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.090+0000 7fc155000700 1 -- 172.21.15.43:0/1493252861 shutdown_connections 2024-05-27T07:12:37.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.091+0000 7fc155000700 1 -- 172.21.15.43:0/1493252861 wait complete. 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.092+0000 7fc155000700 1 Processor -- start 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.092+0000 7fc155000700 1 -- start start 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.092+0000 7fc155000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc150106610 0x7fc15019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.092+0000 7fc155000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc15010af40 0x7fc15019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.092+0000 7fc155000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc15010b890 0x7fc1501a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.092+0000 7fc155000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc1501163a0 con 0x7fc15010af40 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.092+0000 7fc155000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc150116220 con 0x7fc15010b890 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.092+0000 7fc155000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc150116520 con 0x7fc150106610 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc146a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc15010af40 0x7fc15019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc146a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc15010af40 0x7fc15019d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40876/0 (socket says 172.21.15.43:40876) 2024-05-27T07:12:37.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc146a00700 1 -- 172.21.15.43:0/3384618178 learned_addr learned my addr 172.21.15.43:0/3384618178 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:37.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc14f400700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc150106610 0x7fc15019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc14fe00700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc15010b890 0x7fc1501a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc146a00700 1 -- 172.21.15.43:0/3384618178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc150106610 msgr2=0x7fc15019ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:37.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc146a00700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc150106610 0x7fc15019ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc146a00700 1 -- 172.21.15.43:0/3384618178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc15010b890 msgr2=0x7fc1501a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:37.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc146a00700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc15010b890 0x7fc1501a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc146a00700 1 -- 172.21.15.43:0/3384618178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1501a1d80 con 0x7fc15010af40 2024-05-27T07:12:37.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.093+0000 7fc14f400700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc150106610 0x7fc15019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:37.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.094+0000 7fc146a00700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc15010af40 0x7fc15019d450 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc13800b580 tx=0x7fc13800f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:37.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.094+0000 7fc14cc00700 1 -- 172.21.15.43:0/3384618178 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138027070 con 0x7fc15010af40 2024-05-27T07:12:37.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.094+0000 7fc14cc00700 1 -- 172.21.15.43:0/3384618178 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc13800fda0 con 0x7fc15010af40 2024-05-27T07:12:37.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.094+0000 7fc155000700 1 -- 172.21.15.43:0/3384618178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1501a2010 con 0x7fc15010af40 2024-05-27T07:12:37.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.094+0000 7fc14cc00700 1 -- 172.21.15.43:0/3384618178 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1380173f0 con 0x7fc15010af40 2024-05-27T07:12:37.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.094+0000 7fc155000700 1 -- 172.21.15.43:0/3384618178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1501a24f0 con 0x7fc15010af40 2024-05-27T07:12:37.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.098+0000 7fc14cc00700 1 -- 172.21.15.43:0/3384618178 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fc13802a070 con 0x7fc15010af40 2024-05-27T07:12:37.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.099+0000 7fc165178b80 1 Processor -- start 2024-05-27T07:12:37.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.099+0000 7fc165178b80 1 -- start start 2024-05-27T07:12:37.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc14cc00700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1340b2880 0x7fc1340b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc165178b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe845700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc165178b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe6c5df0 0x55a8fe845c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc165178b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe851690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc165178b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe772df0 con 0x55a8fe7b19a0 2024-05-27T07:12:37.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc165178b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe762990 con 0x55a8fe84f270 2024-05-27T07:12:37.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc165178b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe7669d0 con 0x55a8fe6c5df0 2024-05-27T07:12:37.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc14cc00700 1 -- 172.21.15.43:0/3384618178 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fc1380e3560 con 0x7fc15010af40 2024-05-27T07:12:37.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc14fe00700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1340b2880 0x7fc1340b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc14f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe845700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc14f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe845700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40878/0 (socket says 172.21.15.43:40878) 2024-05-27T07:12:37.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.100+0000 7fc14f400700 1 -- 172.21.15.43:0/228270450 learned_addr learned my addr 172.21.15.43:0/228270450 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:37.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.101+0000 7fc14fe00700 1 --2- 172.21.15.43:0/228270450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe851690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.101+0000 7fc146a00700 1 --2- 172.21.15.43:0/228270450 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe6c5df0 0x55a8fe845c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.101+0000 7fc14f400700 1 -- 172.21.15.43:0/228270450 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe6c5df0 msgr2=0x55a8fe845c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.101+0000 7fc14f400700 1 --2- 172.21.15.43:0/228270450 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe6c5df0 0x55a8fe845c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.101+0000 7fc14f400700 1 -- 172.21.15.43:0/228270450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 msgr2=0x55a8fe851690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.101+0000 7fc14f400700 1 --2- 172.21.15.43:0/228270450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe851690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.101+0000 7fc14f400700 1 -- 172.21.15.43:0/228270450 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8fe72b3d0 con 0x55a8fe7b19a0 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.101+0000 7fc14fe00700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1340b2880 0x7fc1340b4d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc148005e10 tx=0x7fc1480137e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.101+0000 7fc14f400700 1 --2- 172.21.15.43:0/228270450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe845700 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc140009ee0 tx=0x7fc1400077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38eaf2c4ae4dbad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.102+0000 7fc144c00700 1 -- 172.21.15.43:0/228270450 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140013780 con 0x55a8fe7b19a0 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.102+0000 7fc144c00700 1 -- 172.21.15.43:0/228270450 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc140019070 con 0x55a8fe7b19a0 2024-05-27T07:12:37.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.102+0000 7fc144c00700 1 -- 172.21.15.43:0/228270450 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140014a30 con 0x55a8fe7b19a0 2024-05-27T07:12:37.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.103+0000 7fc165178b80 1 -- 172.21.15.43:0/228270450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 msgr2=0x55a8fe845700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:37.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.103+0000 7fc165178b80 1 --2- 172.21.15.43:0/228270450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe845700 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc140009ee0 tx=0x7fc1400077e0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.103+0000 7fc165178b80 1 -- 172.21.15.43:0/228270450 shutdown_connections 2024-05-27T07:12:37.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.103+0000 7fc165178b80 1 --2- 172.21.15.43:0/228270450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe851690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.103+0000 7fc165178b80 1 --2- 172.21.15.43:0/228270450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe845700 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.103+0000 7fc165178b80 1 --2- 172.21.15.43:0/228270450 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe6c5df0 0x55a8fe845c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.103+0000 7fc165178b80 1 -- 172.21.15.43:0/228270450 >> 172.21.15.43:0/228270450 conn(0x55a8fe75f340 msgr2=0x55a8fe8406c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:37.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.103+0000 7fc165178b80 1 -- 172.21.15.43:0/228270450 shutdown_connections 2024-05-27T07:12:37.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.103+0000 7fc165178b80 1 -- 172.21.15.43:0/228270450 wait complete. 2024-05-27T07:12:37.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.104+0000 7fc165178b80 1 -- start start 2024-05-27T07:12:37.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.104+0000 7fc165178b80 1 Processor -- start 2024-05-27T07:12:37.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.105+0000 7fc165178b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe6c5df0 0x55a8fe84c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.105+0000 7fc165178b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe7b19a0 0x55a8fe84c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.105+0000 7fc165178b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe84f270 0x55a8fe84cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.105+0000 7fc165178b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe668f50 con 0x55a8fe84f270 2024-05-27T07:12:37.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.105+0000 7fc165178b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe703710 con 0x55a8fe6c5df0 2024-05-27T07:12:37.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.105+0000 7fc165178b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe851db0 con 0x55a8fe7b19a0 2024-05-27T07:12:37.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.105+0000 7fc165178b80 10 client.0 _ll_register_callbacks cb 0x55a8fe70cc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:12:37.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.105+0000 7fc14fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe84f270 0x55a8fe84cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.105+0000 7fc14fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe84f270 0x55a8fe84cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40884/0 (socket says 172.21.15.43:40884) 2024-05-27T07:12:37.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.105+0000 7fc14fe00700 1 -- 172.21.15.43:0/2355759575 learned_addr learned my addr 172.21.15.43:0/2355759575 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:37.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.106+0000 7fc14f400700 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe6c5df0 0x55a8fe84c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.106+0000 7fc146a00700 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe7b19a0 0x55a8fe84c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.106+0000 7fc14fe00700 1 -- 172.21.15.43:0/2355759575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe7b19a0 msgr2=0x55a8fe84c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.106+0000 7fc14fe00700 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe7b19a0 0x55a8fe84c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.106+0000 7fc14fe00700 1 -- 172.21.15.43:0/2355759575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe6c5df0 msgr2=0x55a8fe84c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.106+0000 7fc14fe00700 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe6c5df0 0x55a8fe84c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.106+0000 7fc14fe00700 1 -- 172.21.15.43:0/2355759575 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8fe6b2880 con 0x55a8fe84f270 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.106+0000 7fc14fe00700 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe84f270 0x55a8fe84cc90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc14800e5b0 tx=0x7fc1480047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea5a70a205fdbd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.106+0000 7fc13d600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.107+0000 7fc13d600700 1 -- 172.21.15.43:0/2355759575 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc148024030 con 0x55a8fe84f270 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.107+0000 7fc13d600700 1 -- 172.21.15.43:0/2355759575 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fc148020630 con 0x55a8fe84f270 2024-05-27T07:12:37.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.107+0000 7fc165178b80 1 -- 172.21.15.43:0/2355759575 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc140019070 con 0x55a8fe84f270 2024-05-27T07:12:37.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.107+0000 7fc13d600700 1 -- 172.21.15.43:0/2355759575 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc148031050 con 0x55a8fe84f270 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.107+0000 7fc165178b80 10 client.15123 fetch_fsmap learned FSMap version 29 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.107+0000 7fc165178b80 1 -- 172.21.15.43:0/2355759575 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8fe733010 con 0x55a8fe84f270 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.107+0000 7fc13d600700 1 -- 172.21.15.43:0/2355759575 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc140019070 con 0x55a8fe84f270 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.108+0000 7fc13d600700 1 -- 172.21.15.43:0/2355759575 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc1480286c0 con 0x55a8fe84f270 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.108+0000 7fc165178b80 10 client.15123 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.108+0000 7fc165178b80 10 client.15123 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.108+0000 7fc165178b80 20 client.15123 populate_metadata read hostname 'smithi043' 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.108+0000 7fc165178b80 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a8fe85ec20 0x55a8fe860090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.108+0000 7fc165178b80 4 client.15123 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.108+0000 7fc165178b80 1 -- 172.21.15.43:0/2355759575 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8fe774d40 con 0x55a8fe85ec20 2024-05-27T07:12:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.109+0000 7fc14f400700 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a8fe85ec20 0x55a8fe860090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.110+0000 7fc14f400700 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a8fe85ec20 0x55a8fe860090 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=788c75ecb8f42168 server_cookie=12b2edc0f762d19f in_seq=0 out_seq=0 2024-05-27T07:12:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.110+0000 7fc13d600700 10 client.15123 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc13d600700 1 -- 172.21.15.43:0/2355759575 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a8fe774d40 con 0x55a8fe85ec20 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc13d600700 10 client.15123 handle_command_reply: tid=0 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc165178b80 1 client.15123 shutdown 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc165178b80 1 -- 172.21.15.43:0/2355759575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe84f270 msgr2=0x55a8fe84cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc165178b80 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe84f270 0x55a8fe84cc90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc14800e5b0 tx=0x7fc1480047c0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc165178b80 1 -- 172.21.15.43:0/2355759575 shutdown_connections 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc165178b80 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a8fe85ec20 0x55a8fe860090 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc165178b80 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe84f270 0x55a8fe84cc90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc13d600700 0 client.15123 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc165178b80 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe6c5df0 0x55a8fe84c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc165178b80 1 --2- 172.21.15.43:0/2355759575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe7b19a0 0x55a8fe84c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc165178b80 1 -- 172.21.15.43:0/2355759575 >> 172.21.15.43:0/2355759575 conn(0x55a8fe75f340 msgr2=0x55a8fe845700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:37.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.112+0000 7fc165178b80 1 -- 172.21.15.43:0/2355759575 shutdown_connections 2024-05-27T07:12:37.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.113+0000 7fc165178b80 1 -- 172.21.15.43:0/2355759575 wait complete. 2024-05-27T07:12:37.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.113+0000 7fc165178b80 20 client.15123 trim_cache size 0 max 16384 2024-05-27T07:12:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.138+0000 7fc165178b80 1 Processor -- start 2024-05-27T07:12:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.138+0000 7fc165178b80 1 -- start start 2024-05-27T07:12:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc165178b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe6c5df0 0x55a8fe860090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc165178b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe8406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc165178b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe847140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc165178b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe85e3d0 con 0x55a8fe7b19a0 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc165178b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe85eaa0 con 0x55a8fe84f270 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc165178b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe85f170 con 0x55a8fe6c5df0 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc146a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe8406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc146a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe8406f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40890/0 (socket says 172.21.15.43:40890) 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc146a00700 1 -- 172.21.15.43:0/318423750 learned_addr learned my addr 172.21.15.43:0/318423750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc14f400700 1 --2- 172.21.15.43:0/318423750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe6c5df0 0x55a8fe860090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.139+0000 7fc14fe00700 1 --2- 172.21.15.43:0/318423750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe847140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.140+0000 7fc146a00700 1 -- 172.21.15.43:0/318423750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe6c5df0 msgr2=0x55a8fe860090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.140+0000 7fc146a00700 1 --2- 172.21.15.43:0/318423750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe6c5df0 0x55a8fe860090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.140+0000 7fc146a00700 1 -- 172.21.15.43:0/318423750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 msgr2=0x55a8fe847140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.140+0000 7fc146a00700 1 --2- 172.21.15.43:0/318423750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe847140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.140+0000 7fc146a00700 1 -- 172.21.15.43:0/318423750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8fe84d000 con 0x55a8fe7b19a0 2024-05-27T07:12:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.140+0000 7fc146a00700 1 --2- 172.21.15.43:0/318423750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe8406f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc1380a68a0 tx=0x7fc1380b3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a96c6de976b05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.140+0000 7fc13fe00700 1 -- 172.21.15.43:0/318423750 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1380cd0d0 con 0x55a8fe7b19a0 2024-05-27T07:12:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.140+0000 7fc13fe00700 1 -- 172.21.15.43:0/318423750 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1380aaf20 con 0x55a8fe7b19a0 2024-05-27T07:12:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.140+0000 7fc13fe00700 1 -- 172.21.15.43:0/318423750 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1380b10b0 con 0x55a8fe7b19a0 2024-05-27T07:12:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.141+0000 7fc165178b80 1 -- 172.21.15.43:0/318423750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 msgr2=0x55a8fe8406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.141+0000 7fc165178b80 1 --2- 172.21.15.43:0/318423750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe8406f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc1380a68a0 tx=0x7fc1380b3040 comp rx=0 tx=0).stop 2024-05-27T07:12:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.141+0000 7fc165178b80 1 -- 172.21.15.43:0/318423750 shutdown_connections 2024-05-27T07:12:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.141+0000 7fc165178b80 1 --2- 172.21.15.43:0/318423750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe847140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.141+0000 7fc165178b80 1 --2- 172.21.15.43:0/318423750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe6c5df0 0x55a8fe860090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.141+0000 7fc165178b80 1 --2- 172.21.15.43:0/318423750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe7b19a0 0x55a8fe8406f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.141+0000 7fc165178b80 1 -- 172.21.15.43:0/318423750 >> 172.21.15.43:0/318423750 conn(0x55a8fe75f340 msgr2=0x55a8fe84ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.142+0000 7fc165178b80 1 -- 172.21.15.43:0/318423750 shutdown_connections 2024-05-27T07:12:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.142+0000 7fc165178b80 1 -- 172.21.15.43:0/318423750 wait complete. 2024-05-27T07:12:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.142+0000 7fc165178b80 1 -- start start 2024-05-27T07:12:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.142+0000 7fc165178b80 1 Processor -- start 2024-05-27T07:12:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.143+0000 7fc165178b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe6c5df0 0x55a8fe869850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.143+0000 7fc165178b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe7b19a0 0x55a8fe862b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.143+0000 7fc165178b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe8630d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.143+0000 7fc165178b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe85ff10 con 0x55a8fe6c5df0 2024-05-27T07:12:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.143+0000 7fc165178b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe85f840 con 0x55a8fe84f270 2024-05-27T07:12:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.143+0000 7fc165178b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe8605e0 con 0x55a8fe7b19a0 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.143+0000 7fc165178b80 10 client.0 _ll_register_callbacks cb 0x55a8fe70cc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.143+0000 7fc14f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe6c5df0 0x55a8fe869850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.143+0000 7fc14f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe6c5df0 0x55a8fe869850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40900/0 (socket says 172.21.15.43:40900) 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.143+0000 7fc14f400700 1 -- 172.21.15.43:0/3034078325 learned_addr learned my addr 172.21.15.43:0/3034078325 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc14fe00700 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe8630d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc14f400700 1 -- 172.21.15.43:0/3034078325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe7b19a0 msgr2=0x55a8fe862b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc14f400700 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe7b19a0 0x55a8fe862b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc14f400700 1 -- 172.21.15.43:0/3034078325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 msgr2=0x55a8fe8630d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc14f400700 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe8630d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc14f400700 1 -- 172.21.15.43:0/3034078325 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8fe863610 con 0x55a8fe6c5df0 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc14fe00700 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe8630d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:12:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc14f400700 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe6c5df0 0x55a8fe869850 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55a8fe841a40 tx=0x7fc140009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2c4486b7de471e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc13d600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc13d600700 1 -- 172.21.15.43:0/3034078325 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140003a00 con 0x55a8fe6c5df0 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc165178b80 1 -- 172.21.15.43:0/3034078325 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1380aaf20 con 0x55a8fe6c5df0 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.144+0000 7fc13d600700 1 -- 172.21.15.43:0/3034078325 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fc140003ba0 con 0x55a8fe6c5df0 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc13d600700 1 -- 172.21.15.43:0/3034078325 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc140003de0 con 0x55a8fe6c5df0 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc165178b80 10 client.15129 fetch_fsmap learned FSMap version 29 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc165178b80 1 -- 172.21.15.43:0/3034078325 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8fe867f30 con 0x55a8fe6c5df0 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc13d600700 1 -- 172.21.15.43:0/3034078325 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc1380aaf20 con 0x55a8fe6c5df0 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc13d600700 1 -- 172.21.15.43:0/3034078325 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc14002e030 con 0x55a8fe6c5df0 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc165178b80 10 client.15129 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc165178b80 10 client.15129 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:12:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc165178b80 20 client.15129 populate_metadata read hostname 'smithi043' 2024-05-27T07:12:37.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc165178b80 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a8fe8cc190 0x55a8fe8cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:37.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc165178b80 4 client.15129 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:12:37.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.145+0000 7fc165178b80 1 -- 172.21.15.43:0/3034078325 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "status"}) -- 0x55a8fe76e520 con 0x55a8fe8cc190 2024-05-27T07:12:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.146+0000 7fc146a00700 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a8fe8cc190 0x55a8fe8cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.147+0000 7fc146a00700 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a8fe8cc190 0x55a8fe8cd600 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=186121c4405baaf6 server_cookie=d8d1e059127bd39e in_seq=0 out_seq=0 2024-05-27T07:12:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.147+0000 7fc13d600700 10 client.15129 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:12:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc13d600700 1 -- 172.21.15.43:0/3034078325 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a8fe76e520 con 0x55a8fe8cc190 2024-05-27T07:12:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc13d600700 10 client.15129 handle_command_reply: tid=0 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 client.15129 shutdown 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 -- 172.21.15.43:0/3034078325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe6c5df0 msgr2=0x55a8fe869850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe6c5df0 0x55a8fe869850 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55a8fe841a40 tx=0x7fc140009aa0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 -- 172.21.15.43:0/3034078325 shutdown_connections 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a8fe8cc190 0x55a8fe8cd600 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8fe84f270 0x55a8fe8630d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8fe6c5df0 0x55a8fe869850 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 --2- 172.21.15.43:0/3034078325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8fe7b19a0 0x55a8fe862b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc13d600700 0 client.15129 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 -- 172.21.15.43:0/3034078325 >> 172.21.15.43:0/3034078325 conn(0x55a8fe75f340 msgr2=0x55a8fe847020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 -- 172.21.15.43:0/3034078325 shutdown_connections 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.148+0000 7fc165178b80 1 -- 172.21.15.43:0/3034078325 wait complete. 2024-05-27T07:12:37.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.149+0000 7fc165178b80 20 client.15129 trim_cache size 0 max 16384 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 1, 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 34448, 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 168.197712107, 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 29, 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 90, 2024-05-27T07:12:37.142 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 293.74364416200001 2024-05-27T07:12:37.143 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:12:37.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.149+0000 7fc155000700 1 -- 172.21.15.43:0/3384618178 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1340b2880 msgr2=0x7fc1340b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:37.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.149+0000 7fc155000700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1340b2880 0x7fc1340b4d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc148005e10 tx=0x7fc1480137e0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.149+0000 7fc155000700 1 -- 172.21.15.43:0/3384618178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc15010af40 msgr2=0x7fc15019d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:37.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.149+0000 7fc155000700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc15010af40 0x7fc15019d450 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc13800b580 tx=0x7fc13800f460 comp rx=0 tx=0).stop 2024-05-27T07:12:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.150+0000 7fc155000700 1 -- 172.21.15.43:0/3384618178 shutdown_connections 2024-05-27T07:12:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.150+0000 7fc155000700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fc1340b2880 0x7fc1340b4d90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.150+0000 7fc155000700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc15010b890 0x7fc1501a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.150+0000 7fc155000700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc150106610 0x7fc15019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.150+0000 7fc155000700 1 --2- 172.21.15.43:0/3384618178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc15010af40 0x7fc15019d450 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.150+0000 7fc155000700 1 -- 172.21.15.43:0/3384618178 >> 172.21.15.43:0/3384618178 conn(0x7fc150102020 msgr2=0x7fc15010a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.150+0000 7fc155000700 1 -- 172.21.15.43:0/3384618178 shutdown_connections 2024-05-27T07:12:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:37.150+0000 7fc155000700 1 -- 172.21.15.43:0/3384618178 wait complete. 2024-05-27T07:12:38.141 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T07:12:38.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:38 smithi132 ceph-mon[35283]: pgmap v522: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:38.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:38 smithi043 ceph-mon[31531]: pgmap v522: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:38 smithi179 ceph-mon[35888]: pgmap v522: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:38.635 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:40 smithi043 ceph-mon[31531]: pgmap v523: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:40.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:40 smithi132 ceph-mon[35283]: pgmap v523: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:40.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:40 smithi179 ceph-mon[35888]: pgmap v523: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.870+0000 7f432e200700 1 -- 172.21.15.43:0/3115348933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4328102ea0 msgr2=0x7f43281032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.870+0000 7f432e200700 1 --2- 172.21.15.43:0/3115348933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4328102ea0 0x7f43281032b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4318009990 tx=0x7f4318009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.870+0000 7f432e200700 1 -- 172.21.15.43:0/3115348933 shutdown_connections 2024-05-27T07:12:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.870+0000 7f432e200700 1 --2- 172.21.15.43:0/3115348933 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4328110a50 0x7f4328112e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.870+0000 7f432e200700 1 --2- 172.21.15.43:0/3115348933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4328102ea0 0x7f43281032b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.870+0000 7f432e200700 1 --2- 172.21.15.43:0/3115348933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4328103880 0x7f4328110510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.870+0000 7f432e200700 1 -- 172.21.15.43:0/3115348933 >> 172.21.15.43:0/3115348933 conn(0x7f4328078b40 msgr2=0x7f4328078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.870+0000 7f432e200700 1 -- 172.21.15.43:0/3115348933 shutdown_connections 2024-05-27T07:12:40.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.870+0000 7f432e200700 1 -- 172.21.15.43:0/3115348933 wait complete. 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.872+0000 7f432e200700 1 Processor -- start 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.872+0000 7f432e200700 1 -- start start 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.872+0000 7f432e200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4328102ea0 0x7f432819cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.872+0000 7f432e200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4328103880 0x7f432819d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.872+0000 7f432e200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4328110a50 0x7f43281a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.873+0000 7f432e200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f43281163a0 con 0x7f4328103880 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.873+0000 7f432e200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4328116220 con 0x7f4328102ea0 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.873+0000 7f432e200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4328116520 con 0x7f4328110a50 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.873+0000 7f431fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4328103880 0x7f432819d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.873+0000 7f431fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4328103880 0x7f432819d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40916/0 (socket says 172.21.15.43:40916) 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.873+0000 7f431fe00700 1 -- 172.21.15.43:0/530421365 learned_addr learned my addr 172.21.15.43:0/530421365 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:40.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.873+0000 7f432ce00700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4328102ea0 0x7f432819cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.873+0000 7f432d800700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4328110a50 0x7f43281a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.874+0000 7f431fe00700 1 -- 172.21.15.43:0/530421365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4328110a50 msgr2=0x7f43281a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:40.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.874+0000 7f431fe00700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4328110a50 0x7f43281a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.874+0000 7f431fe00700 1 -- 172.21.15.43:0/530421365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4328102ea0 msgr2=0x7f432819cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:40.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.874+0000 7f431fe00700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4328102ea0 0x7f432819cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.874+0000 7f431fe00700 1 -- 172.21.15.43:0/530421365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43281a1c60 con 0x7f4328103880 2024-05-27T07:12:40.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.874+0000 7f431fe00700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4328103880 0x7f432819d330 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f431000d950 tx=0x7f431000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:40.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.875+0000 7f4326000700 1 -- 172.21.15.43:0/530421365 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310017070 con 0x7f4328103880 2024-05-27T07:12:40.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.875+0000 7f4326000700 1 -- 172.21.15.43:0/530421365 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f431000f690 con 0x7f4328103880 2024-05-27T07:12:40.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.875+0000 7f432e200700 1 -- 172.21.15.43:0/530421365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43281a1f50 con 0x7f4328103880 2024-05-27T07:12:40.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.875+0000 7f4326000700 1 -- 172.21.15.43:0/530421365 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43100039e0 con 0x7f4328103880 2024-05-27T07:12:40.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.875+0000 7f432e200700 1 -- 172.21.15.43:0/530421365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43281a9870 con 0x7f4328103880 2024-05-27T07:12:40.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.879+0000 7f4326000700 1 -- 172.21.15.43:0/530421365 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f4310003b80 con 0x7f4328103880 2024-05-27T07:12:40.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.880+0000 7f433e323b80 1 Processor -- start 2024-05-27T07:12:40.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.880+0000 7f433e323b80 1 -- start start 2024-05-27T07:12:40.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f433e323b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3f3b9a0 0x55ece3fcf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f433e323b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3e4fdf0 0x55ece3fcfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f433e323b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 0x55ece3fdb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f4326000700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f430c0b2880 0x7f430c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f433e323b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ece3efcdf0 con 0x55ece3e4fdf0 2024-05-27T07:12:40.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f433e323b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ece3eec990 con 0x55ece3fd9270 2024-05-27T07:12:40.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f433e323b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ece3ef09d0 con 0x55ece3f3b9a0 2024-05-27T07:12:40.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f4326000700 1 -- 172.21.15.43:0/530421365 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f43100e2090 con 0x7f4328103880 2024-05-27T07:12:40.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f432ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3e4fdf0 0x55ece3fcfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f432ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3e4fdf0 0x55ece3fcfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40932/0 (socket says 172.21.15.43:40932) 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f432ce00700 1 -- 172.21.15.43:0/3548202041 learned_addr learned my addr 172.21.15.43:0/3548202041 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.881+0000 7f432d800700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f430c0b2880 0x7f430c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.882+0000 7f431fe00700 1 --2- 172.21.15.43:0/3548202041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 0x55ece3fdb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.882+0000 7f432d800700 1 --2- 172.21.15.43:0/3548202041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3f3b9a0 0x55ece3fcf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.882+0000 7f432ce00700 1 -- 172.21.15.43:0/3548202041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3f3b9a0 msgr2=0x55ece3fcf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.882+0000 7f432ce00700 1 --2- 172.21.15.43:0/3548202041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3f3b9a0 0x55ece3fcf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.882+0000 7f432ce00700 1 -- 172.21.15.43:0/3548202041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 msgr2=0x55ece3fdb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.882+0000 7f432ce00700 1 --2- 172.21.15.43:0/3548202041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 0x55ece3fdb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.882+0000 7f432ce00700 1 -- 172.21.15.43:0/3548202041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ece3eb53d0 con 0x55ece3e4fdf0 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.882+0000 7f432ce00700 1 --2- 172.21.15.43:0/3548202041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3e4fdf0 0x55ece3fcfc40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4318009750 tx=0x7f4318003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45122b56436ed64e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.883+0000 7f432d800700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f430c0b2880 0x7f430c0b4d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55ece3fcba40 tx=0x7f432000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.883+0000 7f431e000700 1 -- 172.21.15.43:0/3548202041 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4318027070 con 0x55ece3e4fdf0 2024-05-27T07:12:40.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.883+0000 7f431e000700 1 -- 172.21.15.43:0/3548202041 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43180037c0 con 0x55ece3e4fdf0 2024-05-27T07:12:40.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.883+0000 7f431e000700 1 -- 172.21.15.43:0/3548202041 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431800b9d0 con 0x55ece3e4fdf0 2024-05-27T07:12:40.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.884+0000 7f433e323b80 1 -- 172.21.15.43:0/3548202041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3e4fdf0 msgr2=0x55ece3fcfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:40.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.884+0000 7f433e323b80 1 --2- 172.21.15.43:0/3548202041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3e4fdf0 0x55ece3fcfc40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4318009750 tx=0x7f4318003680 comp rx=0 tx=0).stop 2024-05-27T07:12:40.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.884+0000 7f433e323b80 1 -- 172.21.15.43:0/3548202041 shutdown_connections 2024-05-27T07:12:40.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.884+0000 7f433e323b80 1 --2- 172.21.15.43:0/3548202041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 0x55ece3fdb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.884+0000 7f433e323b80 1 --2- 172.21.15.43:0/3548202041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3f3b9a0 0x55ece3fcf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.884+0000 7f433e323b80 1 --2- 172.21.15.43:0/3548202041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3e4fdf0 0x55ece3fcfc40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.884+0000 7f433e323b80 1 -- 172.21.15.43:0/3548202041 >> 172.21.15.43:0/3548202041 conn(0x55ece3ee9340 msgr2=0x55ece3fca6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:40.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.884+0000 7f433e323b80 1 -- 172.21.15.43:0/3548202041 shutdown_connections 2024-05-27T07:12:40.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.884+0000 7f433e323b80 1 -- 172.21.15.43:0/3548202041 wait complete. 2024-05-27T07:12:40.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.885+0000 7f433e323b80 1 -- start start 2024-05-27T07:12:40.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.885+0000 7f433e323b80 1 Processor -- start 2024-05-27T07:12:40.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.886+0000 7f433e323b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3e4fdf0 0x55ece3fd6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.886+0000 7f433e323b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.886+0000 7f433e323b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3fd9270 0x55ece3fd6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.886+0000 7f433e323b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ece3df2f50 con 0x55ece3f3b9a0 2024-05-27T07:12:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.886+0000 7f433e323b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ece3e8d710 con 0x55ece3e4fdf0 2024-05-27T07:12:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.886+0000 7f433e323b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ece3fdbd50 con 0x55ece3fd9270 2024-05-27T07:12:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.886+0000 7f433e323b80 10 client.0 _ll_register_callbacks cb 0x55ece3e96c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:12:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.886+0000 7f431fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.886+0000 7f431fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40934/0 (socket says 172.21.15.43:40934) 2024-05-27T07:12:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.886+0000 7f431fe00700 1 -- 172.21.15.43:0/1654815561 learned_addr learned my addr 172.21.15.43:0/1654815561 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:40.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.887+0000 7f432ce00700 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3e4fdf0 0x55ece3fd6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.887+0000 7f432d800700 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3fd9270 0x55ece3fd6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.887+0000 7f431fe00700 1 -- 172.21.15.43:0/1654815561 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3fd9270 msgr2=0x55ece3fd6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.887+0000 7f431fe00700 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3fd9270 0x55ece3fd6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.887+0000 7f431fe00700 1 -- 172.21.15.43:0/1654815561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3e4fdf0 msgr2=0x55ece3fd6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.887+0000 7f431fe00700 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3e4fdf0 0x55ece3fd6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.887+0000 7f431fe00700 1 -- 172.21.15.43:0/1654815561 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ece3e3c880 con 0x55ece3f3b9a0 2024-05-27T07:12:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.887+0000 7f432ce00700 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3e4fdf0 0x55ece3fd6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:40.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.887+0000 7f431fe00700 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd66d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f43100a8fc0 tx=0x7f43100a99c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6831c916984c5720 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.887+0000 7f4316a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.888+0000 7f4316a00700 1 -- 172.21.15.43:0/1654815561 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43100b2120 con 0x55ece3f3b9a0 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.888+0000 7f4316a00700 1 -- 172.21.15.43:0/1654815561 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f43100a5550 con 0x55ece3f3b9a0 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.888+0000 7f433e323b80 1 -- 172.21.15.43:0/1654815561 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43180037c0 con 0x55ece3f3b9a0 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.888+0000 7f4316a00700 1 -- 172.21.15.43:0/1654815561 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f43100911a0 con 0x55ece3f3b9a0 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.888+0000 7f433e323b80 10 client.15141 fetch_fsmap learned FSMap version 29 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.888+0000 7f433e323b80 1 -- 172.21.15.43:0/1654815561 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ece3ebd010 con 0x55ece3f3b9a0 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.888+0000 7f4316a00700 1 -- 172.21.15.43:0/1654815561 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43180037c0 con 0x55ece3f3b9a0 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.888+0000 7f4316a00700 1 -- 172.21.15.43:0/1654815561 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f43100a9620 con 0x55ece3f3b9a0 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.889+0000 7f433e323b80 10 client.15141 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.889+0000 7f433e323b80 10 client.15141 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:12:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.889+0000 7f433e323b80 20 client.15141 populate_metadata read hostname 'smithi043' 2024-05-27T07:12:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.889+0000 7f433e323b80 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ece3fe8ca0 0x55ece3fea110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.889+0000 7f433e323b80 4 client.15141 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:12:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.889+0000 7f433e323b80 1 -- 172.21.15.43:0/1654815561 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ece3efed40 con 0x55ece3fe8ca0 2024-05-27T07:12:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.890+0000 7f432ce00700 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ece3fe8ca0 0x55ece3fea110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.891+0000 7f432ce00700 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ece3fe8ca0 0x55ece3fea110 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f19e9b528157e753 server_cookie=c0f48805684d36db in_seq=0 out_seq=0 2024-05-27T07:12:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.891+0000 7f4316a00700 10 client.15141 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f4316a00700 1 -- 172.21.15.43:0/1654815561 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ece3efed40 con 0x55ece3fe8ca0 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f4316a00700 10 client.15141 handle_command_reply: tid=0 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f433e323b80 1 client.15141 shutdown 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f433e323b80 1 -- 172.21.15.43:0/1654815561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 msgr2=0x55ece3fd66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f433e323b80 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd66d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f43100a8fc0 tx=0x7f43100a99c0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f433e323b80 1 -- 172.21.15.43:0/1654815561 shutdown_connections 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f433e323b80 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3fd9270 0x55ece3fd6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f433e323b80 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3e4fdf0 0x55ece3fd6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f433e323b80 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ece3fe8ca0 0x55ece3fea110 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f433e323b80 1 --2- 172.21.15.43:0/1654815561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd66d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f433e323b80 1 -- 172.21.15.43:0/1654815561 >> 172.21.15.43:0/1654815561 conn(0x55ece3ee9340 msgr2=0x55ece3fca2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:40.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f4316a00700 0 client.15141 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:12:40.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.893+0000 7f433e323b80 1 -- 172.21.15.43:0/1654815561 shutdown_connections 2024-05-27T07:12:40.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.894+0000 7f433e323b80 1 -- 172.21.15.43:0/1654815561 wait complete. 2024-05-27T07:12:40.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.894+0000 7f433e323b80 20 client.15141 trim_cache size 0 max 16384 2024-05-27T07:12:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.917+0000 7f433e323b80 1 Processor -- start 2024-05-27T07:12:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.918+0000 7f433e323b80 1 -- start start 2024-05-27T07:12:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.918+0000 7f433e323b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 0x55ece3fe5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.918+0000 7f433e323b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.918+0000 7f433e323b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 0x55ece3fd33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.918+0000 7f433e323b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ece3fe8450 con 0x55ece3f3b9a0 2024-05-27T07:12:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.918+0000 7f433e323b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ece3fe8b20 con 0x55ece3fd9270 2024-05-27T07:12:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.918+0000 7f433e323b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ece3fe91f0 con 0x55ece3e4fdf0 2024-05-27T07:12:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.918+0000 7f431fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f432d800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 0x55ece3fd33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f432ce00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 0x55ece3fe5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f431fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40938/0 (socket says 172.21.15.43:40938) 2024-05-27T07:12:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f432d800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 0x55ece3fd33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:42318/0 (socket says 172.21.15.43:42318) 2024-05-27T07:12:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f431fe00700 1 -- 172.21.15.43:0/1736626722 learned_addr learned my addr 172.21.15.43:0/1736626722 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f432ce00700 1 -- 172.21.15.43:0/1736626722 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 msgr2=0x55ece3fd33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f432ce00700 1 --2- 172.21.15.43:0/1736626722 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 0x55ece3fd33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f432ce00700 1 -- 172.21.15.43:0/1736626722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 msgr2=0x55ece3fd1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f432ce00700 1 --2- 172.21.15.43:0/1736626722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f432ce00700 1 -- 172.21.15.43:0/1736626722 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ece3fd6f80 con 0x55ece3e4fdf0 2024-05-27T07:12:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.919+0000 7f431fe00700 1 --2- 172.21.15.43:0/1736626722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:12:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.920+0000 7f432ce00700 1 --2- 172.21.15.43:0/1736626722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 0x55ece3fe5020 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4318025040 tx=0x7f4318005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd29ef20437fe22f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.920+0000 7f431d600700 1 -- 172.21.15.43:0/1736626722 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4318027040 con 0x55ece3e4fdf0 2024-05-27T07:12:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.920+0000 7f431d600700 1 -- 172.21.15.43:0/1736626722 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4318032030 con 0x55ece3e4fdf0 2024-05-27T07:12:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.920+0000 7f431d600700 1 -- 172.21.15.43:0/1736626722 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431800bd30 con 0x55ece3e4fdf0 2024-05-27T07:12:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.921+0000 7f433e323b80 1 -- 172.21.15.43:0/1736626722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 msgr2=0x55ece3fe5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.921+0000 7f433e323b80 1 --2- 172.21.15.43:0/1736626722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 0x55ece3fe5020 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4318025040 tx=0x7f4318005fd0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.921+0000 7f433e323b80 1 -- 172.21.15.43:0/1736626722 shutdown_connections 2024-05-27T07:12:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.921+0000 7f433e323b80 1 --2- 172.21.15.43:0/1736626722 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3fd9270 0x55ece3fd33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.921+0000 7f433e323b80 1 --2- 172.21.15.43:0/1736626722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 0x55ece3fe5020 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.921+0000 7f433e323b80 1 --2- 172.21.15.43:0/1736626722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3f3b9a0 0x55ece3fd1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.921+0000 7f433e323b80 1 -- 172.21.15.43:0/1736626722 >> 172.21.15.43:0/1736626722 conn(0x55ece3ee9340 msgr2=0x55ece3fd1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.922+0000 7f433e323b80 1 -- 172.21.15.43:0/1736626722 shutdown_connections 2024-05-27T07:12:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.922+0000 7f433e323b80 1 -- 172.21.15.43:0/1736626722 wait complete. 2024-05-27T07:12:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.922+0000 7f433e323b80 1 -- start start 2024-05-27T07:12:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.923+0000 7f433e323b80 1 Processor -- start 2024-05-27T07:12:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.923+0000 7f433e323b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 0x55ece3ff3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.923+0000 7f433e323b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3f3b9a0 0x55ece3fed1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.923+0000 7f433e323b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3fd9270 0x55ece3fed8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.923+0000 7f433e323b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ece3fe9f90 con 0x55ece3fd9270 2024-05-27T07:12:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.923+0000 7f433e323b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ece3fe98c0 con 0x55ece3f3b9a0 2024-05-27T07:12:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.923+0000 7f433e323b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ece3fea660 con 0x55ece3e4fdf0 2024-05-27T07:12:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f433e323b80 10 client.0 _ll_register_callbacks cb 0x55ece3e96c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:12:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f432d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3fd9270 0x55ece3fed8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f432d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3fd9270 0x55ece3fed8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40948/0 (socket says 172.21.15.43:40948) 2024-05-27T07:12:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f432d800700 1 -- 172.21.15.43:0/1020558458 learned_addr learned my addr 172.21.15.43:0/1020558458 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f432ce00700 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 0x55ece3ff3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f431fe00700 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3f3b9a0 0x55ece3fed1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f432d800700 1 -- 172.21.15.43:0/1020558458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 msgr2=0x55ece3ff3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f432d800700 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 0x55ece3ff3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f432d800700 1 -- 172.21.15.43:0/1020558458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3f3b9a0 msgr2=0x55ece3fed1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:12:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f432d800700 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3f3b9a0 0x55ece3fed1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.924+0000 7f432d800700 1 -- 172.21.15.43:0/1020558458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ece3ff1900 con 0x55ece3fd9270 2024-05-27T07:12:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.925+0000 7f432d800700 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3fd9270 0x55ece3fed8b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4320019ce0 tx=0x7f4320004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e34cae5ff18222f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.925+0000 7f4316a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:12:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.925+0000 7f4316a00700 1 -- 172.21.15.43:0/1020558458 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320004bb0 con 0x55ece3fd9270 2024-05-27T07:12:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.925+0000 7f4316a00700 1 -- 172.21.15.43:0/1020558458 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f4320012040 con 0x55ece3fd9270 2024-05-27T07:12:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.925+0000 7f433e323b80 1 -- 172.21.15.43:0/1020558458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4318032030 con 0x55ece3fd9270 2024-05-27T07:12:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f4316a00700 1 -- 172.21.15.43:0/1020558458 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4320033050 con 0x55ece3fd9270 2024-05-27T07:12:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f433e323b80 10 client.15147 fetch_fsmap learned FSMap version 29 2024-05-27T07:12:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f433e323b80 1 -- 172.21.15.43:0/1020558458 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ece40545a0 con 0x55ece3fd9270 2024-05-27T07:12:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f4316a00700 1 -- 172.21.15.43:0/1020558458 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4318032030 con 0x55ece3fd9270 2024-05-27T07:12:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f4316a00700 1 -- 172.21.15.43:0/1020558458 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4320015050 con 0x55ece3fd9270 2024-05-27T07:12:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f433e323b80 10 client.15147 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:12:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f433e323b80 10 client.15147 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:12:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f433e323b80 20 client.15147 populate_metadata read hostname 'smithi043' 2024-05-27T07:12:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f433e323b80 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ece4056b30 0x55ece4057fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f433e323b80 4 client.15147 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:12:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.926+0000 7f433e323b80 1 -- 172.21.15.43:0/1020558458 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "status"}) -- 0x55ece3ef8520 con 0x55ece4056b30 2024-05-27T07:12:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.927+0000 7f432ce00700 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ece4056b30 0x55ece4057fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.928+0000 7f432ce00700 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ece4056b30 0x55ece4057fa0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83d16311c03aea50 server_cookie=e2231590d230e75d in_seq=0 out_seq=0 2024-05-27T07:12:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.928+0000 7f4316a00700 10 client.15147 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:12:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f4316a00700 1 -- 172.21.15.43:0/1020558458 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ece3ef8520 con 0x55ece4056b30 2024-05-27T07:12:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f4316a00700 10 client.15147 handle_command_reply: tid=0 2024-05-27T07:12:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 client.15147 shutdown 2024-05-27T07:12:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 -- 172.21.15.43:0/1020558458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3fd9270 msgr2=0x55ece3fed8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3fd9270 0x55ece3fed8b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4320019ce0 tx=0x7f4320004b40 comp rx=0 tx=0).stop 2024-05-27T07:12:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 -- 172.21.15.43:0/1020558458 shutdown_connections 2024-05-27T07:12:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ece4056b30 0x55ece4057fa0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ece3fd9270 0x55ece3fed8b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ece3e4fdf0 0x55ece3ff3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 --2- 172.21.15.43:0/1020558458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ece3f3b9a0 0x55ece3fed1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f4316a00700 0 client.15147 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:12:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 -- 172.21.15.43:0/1020558458 >> 172.21.15.43:0/1020558458 conn(0x55ece3ee9340 msgr2=0x55ece3fce0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 -- 172.21.15.43:0/1020558458 shutdown_connections 2024-05-27T07:12:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 1 -- 172.21.15.43:0/1020558458 wait complete. 2024-05-27T07:12:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.929+0000 7f433e323b80 20 client.15147 trim_cache size 0 max 16384 2024-05-27T07:12:40.921 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 34484, 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 254.84343296599999, 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 29, 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 90, 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 279.43052541899999 2024-05-27T07:12:40.922 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:12:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.930+0000 7f432e200700 1 -- 172.21.15.43:0/530421365 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f430c0b2880 msgr2=0x7f430c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.930+0000 7f432e200700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f430c0b2880 0x7f430c0b4d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55ece3fcba40 tx=0x7f432000b040 comp rx=0 tx=0).stop 2024-05-27T07:12:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.930+0000 7f432e200700 1 -- 172.21.15.43:0/530421365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4328103880 msgr2=0x7f432819d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.930+0000 7f432e200700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4328103880 0x7f432819d330 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f431000d950 tx=0x7f431000dc60 comp rx=0 tx=0).stop 2024-05-27T07:12:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.930+0000 7f432e200700 1 -- 172.21.15.43:0/530421365 shutdown_connections 2024-05-27T07:12:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.930+0000 7f432e200700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f430c0b2880 0x7f430c0b4d90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.930+0000 7f432e200700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4328110a50 0x7f43281a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.930+0000 7f432e200700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4328102ea0 0x7f432819cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.930+0000 7f432e200700 1 --2- 172.21.15.43:0/530421365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4328103880 0x7f432819d330 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.930+0000 7f432e200700 1 -- 172.21.15.43:0/530421365 >> 172.21.15.43:0/530421365 conn(0x7f4328078b40 msgr2=0x7f4328111ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.931+0000 7f432e200700 1 -- 172.21.15.43:0/530421365 shutdown_connections 2024-05-27T07:12:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:40.931+0000 7f432e200700 1 -- 172.21.15.43:0/530421365 wait complete. 2024-05-27T07:12:41.874 INFO:tasks.cephadm:Setup complete, yielding 2024-05-27T07:12:41.875 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-27T07:12:41.884 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-05-27T07:12:41.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch status' 2024-05-27T07:12:42.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:42 smithi043 ceph-mon[31531]: pgmap v524: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:42.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:42 smithi132 ceph-mon[35283]: pgmap v524: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:42.391 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:42.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:42 smithi179 ceph-mon[35888]: pgmap v524: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:44.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:44 smithi043 ceph-mon[31531]: pgmap v525: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:44.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:44 smithi132 ceph-mon[35283]: pgmap v525: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:44.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:44 smithi179 ceph-mon[35888]: pgmap v525: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.866+0000 7fb0a2600700 1 -- 172.21.15.43:0/2485267882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb09c105e00 msgr2=0x7fb09c10c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.866+0000 7fb0a2600700 1 --2- 172.21.15.43:0/2485267882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb09c105e00 0x7fb09c10c6b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb094009f90 tx=0x7fb09400b330 comp rx=0 tx=0).stop 2024-05-27T07:12:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.866+0000 7fb0a2600700 1 -- 172.21.15.43:0/2485267882 shutdown_connections 2024-05-27T07:12:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.866+0000 7fb0a2600700 1 --2- 172.21.15.43:0/2485267882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb09c105e00 0x7fb09c10c6b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.866+0000 7fb0a2600700 1 --2- 172.21.15.43:0/2485267882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb09c104330 0x7fb09c104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.866+0000 7fb0a2600700 1 --2- 172.21.15.43:0/2485267882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c105460 0x7fb09c105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.866+0000 7fb0a2600700 1 -- 172.21.15.43:0/2485267882 >> 172.21.15.43:0/2485267882 conn(0x7fb09c0ff970 msgr2=0x7fb09c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.867+0000 7fb0a2600700 1 -- 172.21.15.43:0/2485267882 shutdown_connections 2024-05-27T07:12:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.867+0000 7fb0a2600700 1 -- 172.21.15.43:0/2485267882 wait complete. 2024-05-27T07:12:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.868+0000 7fb0a2600700 1 Processor -- start 2024-05-27T07:12:44.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.869+0000 7fb0a2600700 1 -- start start 2024-05-27T07:12:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.869+0000 7fb0a2600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c104330 0x7fb09c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.869+0000 7fb0a2600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb09c105460 0x7fb09c19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.869+0000 7fb0a2600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb09c105e00 0x7fb09c1a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.869+0000 7fb0a2600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c10fc90 con 0x7fb09c104330 2024-05-27T07:12:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.869+0000 7fb0a2600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c10fb10 con 0x7fb09c105460 2024-05-27T07:12:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.869+0000 7fb0a2600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c10fe10 con 0x7fb09c105e00 2024-05-27T07:12:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c104330 0x7fb09c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c104330 0x7fb09c19ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44196/0 (socket says 172.21.15.43:44196) 2024-05-27T07:12:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09b400700 1 -- 172.21.15.43:0/972298677 learned_addr learned my addr 172.21.15.43:0/972298677 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09aa00700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb09c105460 0x7fb09c19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09be00700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb09c105e00 0x7fb09c1a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09b400700 1 -- 172.21.15.43:0/972298677 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb09c105e00 msgr2=0x7fb09c1a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09b400700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb09c105e00 0x7fb09c1a3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09b400700 1 -- 172.21.15.43:0/972298677 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb09c105460 msgr2=0x7fb09c19cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09b400700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb09c105460 0x7fb09c19cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09b400700 1 -- 172.21.15.43:0/972298677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09c1a45a0 con 0x7fb09c104330 2024-05-27T07:12:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.870+0000 7fb09aa00700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb09c105460 0x7fb09c19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.871+0000 7fb09b400700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c104330 0x7fb09c19ca40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00e940 tx=0x7fb08c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.871+0000 7fb093e00700 1 -- 172.21.15.43:0/972298677 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c00cbd0 con 0x7fb09c104330 2024-05-27T07:12:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.871+0000 7fb093e00700 1 -- 172.21.15.43:0/972298677 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb08c0044b0 con 0x7fb09c104330 2024-05-27T07:12:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.871+0000 7fb0a2600700 1 -- 172.21.15.43:0/972298677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09c1a4890 con 0x7fb09c104330 2024-05-27T07:12:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.871+0000 7fb093e00700 1 -- 172.21.15.43:0/972298677 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c003d50 con 0x7fb09c104330 2024-05-27T07:12:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.871+0000 7fb0a2600700 1 -- 172.21.15.43:0/972298677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb09c1a4e20 con 0x7fb09c104330 2024-05-27T07:12:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.873+0000 7fb0a2600700 1 -- 172.21.15.43:0/972298677 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb09c0512b0 con 0x7fb09c104330 2024-05-27T07:12:44.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.876+0000 7fb093e00700 1 -- 172.21.15.43:0/972298677 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fb08c004000 con 0x7fb09c104330 2024-05-27T07:12:44.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.877+0000 7fb093e00700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0800b2900 0x7fb0800b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:44.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.878+0000 7fb093e00700 1 -- 172.21.15.43:0/972298677 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fb08c0e2810 con 0x7fb09c104330 2024-05-27T07:12:44.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.878+0000 7fb09aa00700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0800b2900 0x7fb0800b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:44.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.879+0000 7fb09aa00700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0800b2900 0x7fb0800b4e10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb09c19daf0 tx=0x7fb088009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:44.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:44.883+0000 7fb093e00700 1 -- 172.21.15.43:0/972298677 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb08c0aa140 con 0x7fb09c104330 2024-05-27T07:12:45.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.146+0000 7fb0a2600700 1 -- 172.21.15.43:0/972298677 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fb09c077100 con 0x7fb0800b2900 2024-05-27T07:12:45.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.150+0000 7fb093e00700 1 -- 172.21.15.43:0/972298677 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fb09c077100 con 0x7fb0800b2900 2024-05-27T07:12:45.141 INFO:teuthology.orchestra.run.smithi043.stdout:Backend: cephadm 2024-05-27T07:12:45.141 INFO:teuthology.orchestra.run.smithi043.stdout:Available: Yes 2024-05-27T07:12:45.141 INFO:teuthology.orchestra.run.smithi043.stdout:Paused: No 2024-05-27T07:12:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.153+0000 7fb0a2600700 1 -- 172.21.15.43:0/972298677 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0800b2900 msgr2=0x7fb0800b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.153+0000 7fb0a2600700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0800b2900 0x7fb0800b4e10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb09c19daf0 tx=0x7fb088009310 comp rx=0 tx=0).stop 2024-05-27T07:12:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.153+0000 7fb0a2600700 1 -- 172.21.15.43:0/972298677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c104330 msgr2=0x7fb09c19ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.153+0000 7fb0a2600700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c104330 0x7fb09c19ca40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00e940 tx=0x7fb08c00ed00 comp rx=0 tx=0).stop 2024-05-27T07:12:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.153+0000 7fb0a2600700 1 -- 172.21.15.43:0/972298677 shutdown_connections 2024-05-27T07:12:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.153+0000 7fb0a2600700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0800b2900 0x7fb0800b4e10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.154+0000 7fb0a2600700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb09c105e00 0x7fb09c1a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.154+0000 7fb0a2600700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb09c104330 0x7fb09c19ca40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.154+0000 7fb0a2600700 1 --2- 172.21.15.43:0/972298677 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb09c105460 0x7fb09c19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.154+0000 7fb0a2600700 1 -- 172.21.15.43:0/972298677 >> 172.21.15.43:0/972298677 conn(0x7fb09c0ff970 msgr2=0x7fb09c0759e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.154+0000 7fb0a2600700 1 -- 172.21.15.43:0/972298677 shutdown_connections 2024-05-27T07:12:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:45.154+0000 7fb0a2600700 1 -- 172.21.15.43:0/972298677 wait complete. 2024-05-27T07:12:45.970 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-27T07:12:46.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:46 smithi043 ceph-mon[31531]: pgmap v526: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:46.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:46 smithi132 ceph-mon[35283]: pgmap v526: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:46.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:46 smithi179 ceph-mon[35888]: pgmap v526: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:46.476 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:47.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:47 smithi043 ceph-mon[31531]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:47.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:47 smithi132 ceph-mon[35283]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:47.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:47 smithi179 ceph-mon[35888]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:48 smithi132 ceph-mon[35283]: pgmap v527: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:48.442 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:48 smithi043 ceph-mon[31531]: pgmap v527: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:48.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:48 smithi179 ceph-mon[35888]: pgmap v527: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:49.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.072+0000 7f558e000700 1 -- 172.21.15.43:0/3894728065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5588105270 msgr2=0x7f5588105680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.072+0000 7f558e000700 1 --2- 172.21.15.43:0/3894728065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5588105270 0x7f5588105680 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5578009990 tx=0x7f5578009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.072+0000 7f558e000700 1 -- 172.21.15.43:0/3894728065 shutdown_connections 2024-05-27T07:12:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.072+0000 7f558e000700 1 --2- 172.21.15.43:0/3894728065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5588105c10 0x7f558810c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.072+0000 7f558e000700 1 --2- 172.21.15.43:0/3894728065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5588104140 0x7f5588104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.072+0000 7f558e000700 1 --2- 172.21.15.43:0/3894728065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5588105270 0x7f5588105680 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.072+0000 7f558e000700 1 -- 172.21.15.43:0/3894728065 >> 172.21.15.43:0/3894728065 conn(0x7f55880ff7a0 msgr2=0x7f5588101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.072+0000 7f558e000700 1 -- 172.21.15.43:0/3894728065 shutdown_connections 2024-05-27T07:12:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.072+0000 7f558e000700 1 -- 172.21.15.43:0/3894728065 wait complete. 2024-05-27T07:12:49.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.074+0000 7f558e000700 1 Processor -- start 2024-05-27T07:12:49.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.074+0000 7f558e000700 1 -- start start 2024-05-27T07:12:49.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.074+0000 7f558e000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5588104140 0x7f558819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:49.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.074+0000 7f558e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5588105c10 0x7f558819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.074+0000 7f558e000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f558819fc60 0x7f55881a2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.074+0000 7f558e000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f558810fac0 con 0x7f5588105c10 2024-05-27T07:12:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.074+0000 7f558e000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f558810f940 con 0x7f5588104140 2024-05-27T07:12:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.074+0000 7f558e000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f558810fc40 con 0x7f558819fc60 2024-05-27T07:12:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.075+0000 7f557fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5588105c10 0x7f558819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.075+0000 7f557fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5588105c10 0x7f558819ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44212/0 (socket says 172.21.15.43:44212) 2024-05-27T07:12:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.075+0000 7f557fe00700 1 -- 172.21.15.43:0/176702430 learned_addr learned my addr 172.21.15.43:0/176702430 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.075+0000 7f558cc00700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5588104140 0x7f558819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:49.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.075+0000 7f558d600700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f558819fc60 0x7f55881a2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:49.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.075+0000 7f557fe00700 1 -- 172.21.15.43:0/176702430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f558819fc60 msgr2=0x7f55881a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:49.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.075+0000 7f557fe00700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f558819fc60 0x7f55881a2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.075+0000 7f557fe00700 1 -- 172.21.15.43:0/176702430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5588104140 msgr2=0x7f558819a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.075+0000 7f557fe00700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5588104140 0x7f558819a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.075+0000 7f557fe00700 1 -- 172.21.15.43:0/176702430 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55881a27f0 con 0x7f5588105c10 2024-05-27T07:12:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.076+0000 7f557fe00700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5588105c10 0x7f558819ad90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5578006010 tx=0x7f5578009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.076+0000 7f5586000700 1 -- 172.21.15.43:0/176702430 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5578026070 con 0x7f5588105c10 2024-05-27T07:12:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.076+0000 7f5586000700 1 -- 172.21.15.43:0/176702430 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5578003fe0 con 0x7f5588105c10 2024-05-27T07:12:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.076+0000 7f5586000700 1 -- 172.21.15.43:0/176702430 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557800fba0 con 0x7f5588105c10 2024-05-27T07:12:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.076+0000 7f558e000700 1 -- 172.21.15.43:0/176702430 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55881a2a80 con 0x7f5588105c10 2024-05-27T07:12:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.076+0000 7f558e000700 1 -- 172.21.15.43:0/176702430 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55881a2e80 con 0x7f5588105c10 2024-05-27T07:12:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.078+0000 7f558e000700 1 -- 172.21.15.43:0/176702430 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f558802e5e0 con 0x7f5588105c10 2024-05-27T07:12:49.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.080+0000 7f5586000700 1 -- 172.21.15.43:0/176702430 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f5578029070 con 0x7f5588105c10 2024-05-27T07:12:49.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.082+0000 7f5586000700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55680b2880 0x7f55680b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:49.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.082+0000 7f5586000700 1 -- 172.21.15.43:0/176702430 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f55780e36c0 con 0x7f5588105c10 2024-05-27T07:12:49.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.082+0000 7f558cc00700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55680b2880 0x7f55680b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:49.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.083+0000 7f558cc00700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55680b2880 0x7f55680b4d90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5570005950 tx=0x7f55700058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.085+0000 7f5586000700 1 -- 172.21.15.43:0/176702430 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5578013070 con 0x7f5588105c10 2024-05-27T07:12:49.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.337+0000 7f558e000700 1 -- 172.21.15.43:0/176702430 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f5588063bf0 con 0x7f55680b2880 2024-05-27T07:12:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.382+0000 7f5586000700 1 -- 172.21.15.43:0/176702430 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f5588063bf0 con 0x7f55680b2880 2024-05-27T07:12:49.373 INFO:teuthology.orchestra.run.smithi043.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-27T07:12:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:mds.a smithi043 running (4m) 2m ago 4m 82.2M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 840760a352fe 2024-05-27T07:12:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:mds.b smithi132 running (4m) 2m ago 4m 92.4M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 22b3908559b9 2024-05-27T07:12:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:mds.c smithi179 running (5m) 2m ago 5m 83.4M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 f3c9408632d8 2024-05-27T07:12:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:mds.d smithi043 running (5m) 2m ago 5m 80.5M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 d390a0d3996b 2024-05-27T07:12:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:mds.e smithi132 running (5m) 2m ago 5m 83.7M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 983414c01141 2024-05-27T07:12:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:mds.f smithi179 running (5m) 2m ago 5m 82.3M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 374a158fddff 2024-05-27T07:12:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:mds.g smithi043 running (4m) 2m ago 4m 82.2M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 e4fbe8dc111d 2024-05-27T07:12:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:mds.h smithi132 running (5m) 2m ago 5m 83.5M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 b73a5f07d84b 2024-05-27T07:12:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:mds.i smithi179 running (5m) 2m ago 5m 83.3M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 0c675283f0dc 2024-05-27T07:12:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:mds.j smithi043 running (5m) 2m ago 5m 80.5M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 6726de02f165 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:mds.k smithi132 running (5m) 2m ago 5m 83.7M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 5e67a7a8e86d 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:mds.l smithi179 running (5m) 2m ago 5m 82.4M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 83caa31734dd 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.x smithi043 *:9283,8765 running (15m) 2m ago 15m 598M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 0ae7c08d9f68 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.y smithi132 *:8443,8765 running (13m) 2m ago 13m 534M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 a66e6a350575 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.z smithi179 *:8443,8765 running (13m) 2m ago 13m 534M - 19.0.0-3872-g38f9d900 e36ea4fc8a58 dc3f23bc3228 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a smithi043 running (15m) 2m ago 16m 118M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 83c5e162e73a 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b smithi132 running (13m) 2m ago 13m 108M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 0f4e4ee5f492 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c smithi179 running (13m) 2m ago 13m 108M 2048M 19.0.0-3872-g38f9d900 e36ea4fc8a58 554dd2bd2723 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 smithi043 running (12m) 2m ago 12m 114M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 2934ad7a01cf 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 smithi132 running (12m) 2m ago 12m 113M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 4627b716a906 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 smithi179 running (11m) 2m ago 11m 121M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 f9e6b6d1a1a1 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 smithi043 running (11m) 2m ago 11m 111M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 f7e9ba69131c 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 smithi132 running (10m) 2m ago 10m 111M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 b63730da29c2 2024-05-27T07:12:49.375 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 smithi179 running (9m) 2m ago 9m 111M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 9da519f3c2d7 2024-05-27T07:12:49.376 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 smithi043 running (9m) 2m ago 9m 116M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 829e3cbd626c 2024-05-27T07:12:49.376 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 smithi132 running (8m) 2m ago 8m 121M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 c7835527819a 2024-05-27T07:12:49.376 INFO:teuthology.orchestra.run.smithi043.stdout:osd.8 smithi179 running (7m) 2m ago 7m 119M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 eaeee19a212e 2024-05-27T07:12:49.376 INFO:teuthology.orchestra.run.smithi043.stdout:osd.9 smithi043 running (7m) 2m ago 7m 119M 4273M 19.0.0-3872-g38f9d900 e36ea4fc8a58 b32a02e58555 2024-05-27T07:12:49.376 INFO:teuthology.orchestra.run.smithi043.stdout:osd.10 smithi132 running (6m) 2m ago 6m 110M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 2cc3dcf0860b 2024-05-27T07:12:49.376 INFO:teuthology.orchestra.run.smithi043.stdout:osd.11 smithi179 running (6m) 2m ago 6m 116M 4272M 19.0.0-3872-g38f9d900 e36ea4fc8a58 6ffa1e6f5c5f 2024-05-27T07:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 -- 172.21.15.43:0/176702430 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55680b2880 msgr2=0x7f55680b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55680b2880 0x7f55680b4d90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5570005950 tx=0x7f55700058e0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 -- 172.21.15.43:0/176702430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5588105c10 msgr2=0x7f558819ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5588105c10 0x7f558819ad90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5578006010 tx=0x7f5578009510 comp rx=0 tx=0).stop 2024-05-27T07:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 -- 172.21.15.43:0/176702430 shutdown_connections 2024-05-27T07:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f55680b2880 0x7f55680b4d90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f558819fc60 0x7f55881a2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5588104140 0x7f558819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 --2- 172.21.15.43:0/176702430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5588105c10 0x7f558819ad90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:49.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 -- 172.21.15.43:0/176702430 >> 172.21.15.43:0/176702430 conn(0x7f55880ff7a0 msgr2=0x7f55881085e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:49.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.386+0000 7f558e000700 1 -- 172.21.15.43:0/176702430 shutdown_connections 2024-05-27T07:12:49.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:49.387+0000 7f558e000700 1 -- 172.21.15.43:0/176702430 wait complete. 2024-05-27T07:12:50.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:50 smithi132 ceph-mon[35283]: pgmap v528: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:50.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:50 smithi043 ceph-mon[31531]: pgmap v528: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:50.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-27T07:12:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:50 smithi179 ceph-mon[35888]: pgmap v528: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:50.833 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:51.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:51 smithi043 ceph-mon[31531]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:51.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:51 smithi179 ceph-mon[35888]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:51.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:51 smithi132 ceph-mon[35283]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:52.242 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:52 smithi043 ceph-mon[31531]: pgmap v529: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:52.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:52 smithi179 ceph-mon[35888]: pgmap v529: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:52.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:52 smithi132 ceph-mon[35283]: pgmap v529: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:53.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.130+0000 7f2a61000700 1 -- 172.21.15.43:0/793361451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c0776d0 msgr2=0x7f2a5c075ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:53.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.130+0000 7f2a61000700 1 --2- 172.21.15.43:0/793361451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c0776d0 0x7f2a5c075ba0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009990 tx=0x7f2a4c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.130+0000 7f2a61000700 1 -- 172.21.15.43:0/793361451 shutdown_connections 2024-05-27T07:12:53.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.130+0000 7f2a61000700 1 --2- 172.21.15.43:0/793361451 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a5c10bff0 0x7f2a5c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.130+0000 7f2a61000700 1 --2- 172.21.15.43:0/793361451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c0776d0 0x7f2a5c075ba0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.130+0000 7f2a61000700 1 --2- 172.21.15.43:0/793361451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a5c0760e0 0x7f2a5c076570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.130+0000 7f2a61000700 1 -- 172.21.15.43:0/793361451 >> 172.21.15.43:0/793361451 conn(0x7f2a5c0ff7a0 msgr2=0x7f2a5c101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:53.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.130+0000 7f2a61000700 1 -- 172.21.15.43:0/793361451 shutdown_connections 2024-05-27T07:12:53.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.130+0000 7f2a61000700 1 -- 172.21.15.43:0/793361451 wait complete. 2024-05-27T07:12:53.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.131+0000 7f2a61000700 1 Processor -- start 2024-05-27T07:12:53.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a61000700 1 -- start start 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a61000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a5c0760e0 0x7f2a5c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a61000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a5c0776d0 0x7f2a5c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a61000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c10bff0 0x7f2a5c1a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a61000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c111c70 con 0x7f2a5c10bff0 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a61000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c111af0 con 0x7f2a5c0776d0 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a61000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c111df0 con 0x7f2a5c0760e0 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a5be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c10bff0 0x7f2a5c1a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a5be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c10bff0 0x7f2a5c1a3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44238/0 (socket says 172.21.15.43:44238) 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a5be00700 1 -- 172.21.15.43:0/1048649609 learned_addr learned my addr 172.21.15.43:0/1048649609 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.132+0000 7f2a5b400700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a5c0760e0 0x7f2a5c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.133+0000 7f2a5aa00700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a5c0776d0 0x7f2a5c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.133+0000 7f2a5be00700 1 -- 172.21.15.43:0/1048649609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a5c0760e0 msgr2=0x7f2a5c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:53.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.133+0000 7f2a5be00700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a5c0760e0 0x7f2a5c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.133+0000 7f2a5be00700 1 -- 172.21.15.43:0/1048649609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a5c0776d0 msgr2=0x7f2a5c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:53.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.133+0000 7f2a5be00700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a5c0776d0 0x7f2a5c19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.133+0000 7f2a5be00700 1 -- 172.21.15.43:0/1048649609 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5c1a43d0 con 0x7f2a5c10bff0 2024-05-27T07:12:53.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.133+0000 7f2a5aa00700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a5c0776d0 0x7f2a5c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:53.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.133+0000 7f2a5be00700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c10bff0 0x7f2a5c1a3d50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2a54007ee0 tx=0x7f2a5400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:53.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.133+0000 7f2a53e00700 1 -- 172.21.15.43:0/1048649609 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5400dbd0 con 0x7f2a5c10bff0 2024-05-27T07:12:53.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.134+0000 7f2a53e00700 1 -- 172.21.15.43:0/1048649609 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a5400dd70 con 0x7f2a5c10bff0 2024-05-27T07:12:53.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.134+0000 7f2a53e00700 1 -- 172.21.15.43:0/1048649609 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a540058d0 con 0x7f2a5c10bff0 2024-05-27T07:12:53.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.134+0000 7f2a61000700 1 -- 172.21.15.43:0/1048649609 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5c1a46c0 con 0x7f2a5c10bff0 2024-05-27T07:12:53.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.134+0000 7f2a61000700 1 -- 172.21.15.43:0/1048649609 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5c1a4bb0 con 0x7f2a5c10bff0 2024-05-27T07:12:53.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.135+0000 7f2a61000700 1 -- 172.21.15.43:0/1048649609 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a5c0512b0 con 0x7f2a5c10bff0 2024-05-27T07:12:53.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.137+0000 7f2a53e00700 1 -- 172.21.15.43:0/1048649609 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2a5401a020 con 0x7f2a5c10bff0 2024-05-27T07:12:53.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.139+0000 7f2a53e00700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a400b2900 0x7f2a400b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:53.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.139+0000 7f2a53e00700 1 -- 172.21.15.43:0/1048649609 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f2a540e2410 con 0x7f2a5c10bff0 2024-05-27T07:12:53.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.139+0000 7f2a5b400700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a400b2900 0x7f2a400b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:53.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.140+0000 7f2a5b400700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a400b2900 0x7f2a400b4e10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c006010 tx=0x7f2a4c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:53.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.141+0000 7f2a53e00700 1 -- 172.21.15.43:0/1048649609 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2a54010070 con 0x7f2a5c10bff0 2024-05-27T07:12:53.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.398+0000 7f2a61000700 1 -- 172.21.15.43:0/1048649609 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f2a5c063bf0 con 0x7f2a400b2900 2024-05-27T07:12:53.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.406+0000 7f2a53e00700 1 -- 172.21.15.43:0/1048649609 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f2a5c063bf0 con 0x7f2a400b2900 2024-05-27T07:12:53.398 INFO:teuthology.orchestra.run.smithi043.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-27T07:12:53.398 INFO:teuthology.orchestra.run.smithi043.stdout:mds.all 12/12 2m ago 5m smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-05-27T07:12:53.398 INFO:teuthology.orchestra.run.smithi043.stdout:mgr 3/3 2m ago 13m smithi043=x;smithi132=y;smithi179=z;count:3 2024-05-27T07:12:53.398 INFO:teuthology.orchestra.run.smithi043.stdout:mon 3/3 2m ago 14m smithi043:172.21.15.43=a;smithi132:172.21.15.132=b;smithi179:172.21.15.179=c;count:3 2024-05-27T07:12:53.398 INFO:teuthology.orchestra.run.smithi043.stdout:osd 12 2m ago - 2024-05-27T07:12:53.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.409+0000 7f2a61000700 1 -- 172.21.15.43:0/1048649609 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a400b2900 msgr2=0x7f2a400b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:53.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.409+0000 7f2a61000700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a400b2900 0x7f2a400b4e10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c006010 tx=0x7f2a4c01a040 comp rx=0 tx=0).stop 2024-05-27T07:12:53.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.409+0000 7f2a61000700 1 -- 172.21.15.43:0/1048649609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c10bff0 msgr2=0x7f2a5c1a3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:53.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.409+0000 7f2a61000700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c10bff0 0x7f2a5c1a3d50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2a54007ee0 tx=0x7f2a5400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.409+0000 7f2a61000700 1 -- 172.21.15.43:0/1048649609 shutdown_connections 2024-05-27T07:12:53.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.409+0000 7f2a61000700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a400b2900 0x7f2a400b4e10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.409+0000 7f2a61000700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c10bff0 0x7f2a5c1a3d50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.409+0000 7f2a61000700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a5c0760e0 0x7f2a5c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.410+0000 7f2a61000700 1 --2- 172.21.15.43:0/1048649609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a5c0776d0 0x7f2a5c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:53.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.410+0000 7f2a61000700 1 -- 172.21.15.43:0/1048649609 >> 172.21.15.43:0/1048649609 conn(0x7f2a5c0ff7a0 msgr2=0x7f2a5c107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:53.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.410+0000 7f2a61000700 1 -- 172.21.15.43:0/1048649609 shutdown_connections 2024-05-27T07:12:53.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:53.410+0000 7f2a61000700 1 -- 172.21.15.43:0/1048649609 wait complete. 2024-05-27T07:12:54.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-27T07:12:54.426 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:54 smithi043 ceph-mon[31531]: pgmap v530: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:54 smithi179 ceph-mon[35888]: pgmap v530: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:54.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:54 smithi132 ceph-mon[35283]: pgmap v530: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:12:54.830 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:55.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:55 smithi043 ceph-mon[31531]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:55 smithi179 ceph-mon[35888]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:55.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:55 smithi132 ceph-mon[35283]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:56 smithi043 ceph-mon[31531]: pgmap v531: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:56.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:56 smithi179 ceph-mon[35888]: pgmap v531: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:56.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:56 smithi132 ceph-mon[35283]: pgmap v531: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.288+0000 7fe537000700 1 -- 172.21.15.43:0/3397496755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe530104140 msgr2=0x7fe530104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.288+0000 7fe537000700 1 --2- 172.21.15.43:0/3397496755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe530104140 0x7fe530104550 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe524009990 tx=0x7fe524009ca0 comp rx=0 tx=0).stop 2024-05-27T07:12:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.288+0000 7fe537000700 1 -- 172.21.15.43:0/3397496755 shutdown_connections 2024-05-27T07:12:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.288+0000 7fe537000700 1 --2- 172.21.15.43:0/3397496755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe530105c10 0x7fe53010c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.288+0000 7fe537000700 1 --2- 172.21.15.43:0/3397496755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe530104140 0x7fe530104550 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.288+0000 7fe537000700 1 --2- 172.21.15.43:0/3397496755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe530105270 0x7fe530105680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.288+0000 7fe537000700 1 -- 172.21.15.43:0/3397496755 >> 172.21.15.43:0/3397496755 conn(0x7fe5300ff7a0 msgr2=0x7fe530101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.288+0000 7fe537000700 1 -- 172.21.15.43:0/3397496755 shutdown_connections 2024-05-27T07:12:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.288+0000 7fe537000700 1 -- 172.21.15.43:0/3397496755 wait complete. 2024-05-27T07:12:57.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:57 smithi043 ceph-mon[31531]: pgmap v532: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:57.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.290+0000 7fe537000700 1 Processor -- start 2024-05-27T07:12:57.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.290+0000 7fe537000700 1 -- start start 2024-05-27T07:12:57.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.291+0000 7fe537000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe530104140 0x7fe53019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:57.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.291+0000 7fe537000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe530105270 0x7fe53019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:57.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.291+0000 7fe537000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe530105c10 0x7fe5301a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:57.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.291+0000 7fe537000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe53010fa40 con 0x7fe530105270 2024-05-27T07:12:57.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.291+0000 7fe537000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe53010f8c0 con 0x7fe530104140 2024-05-27T07:12:57.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.291+0000 7fe537000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe53010fbc0 con 0x7fe530105c10 2024-05-27T07:12:57.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.292+0000 7fe535200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe530105270 0x7fe53019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:57.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.292+0000 7fe535200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe530105270 0x7fe53019ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57334/0 (socket says 172.21.15.43:57334) 2024-05-27T07:12:57.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.292+0000 7fe535200700 1 -- 172.21.15.43:0/4047905049 learned_addr learned my addr 172.21.15.43:0/4047905049 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:12:57.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.292+0000 7fe535c00700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe530104140 0x7fe53019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:57.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.292+0000 7fe536600700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe530105c10 0x7fe5301a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:57.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.292+0000 7fe535200700 1 -- 172.21.15.43:0/4047905049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe530105c10 msgr2=0x7fe5301a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:57.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.292+0000 7fe535200700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe530105c10 0x7fe5301a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:57.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.292+0000 7fe535200700 1 -- 172.21.15.43:0/4047905049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe530104140 msgr2=0x7fe53019a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:57.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.292+0000 7fe535200700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe530104140 0x7fe53019a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:57.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.292+0000 7fe535200700 1 -- 172.21.15.43:0/4047905049 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5301a21a0 con 0x7fe530105270 2024-05-27T07:12:57.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.293+0000 7fe536600700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe530105c10 0x7fe5301a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:12:57.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.293+0000 7fe535200700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe530105270 0x7fe53019ab80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00d950 tx=0x7fe51c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:57.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.293+0000 7fe52e000700 1 -- 172.21.15.43:0/4047905049 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c016070 con 0x7fe530105270 2024-05-27T07:12:57.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.293+0000 7fe52e000700 1 -- 172.21.15.43:0/4047905049 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe51c009dd0 con 0x7fe530105270 2024-05-27T07:12:57.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.293+0000 7fe52e000700 1 -- 172.21.15.43:0/4047905049 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c00f3c0 con 0x7fe530105270 2024-05-27T07:12:57.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.293+0000 7fe537000700 1 -- 172.21.15.43:0/4047905049 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5301a2490 con 0x7fe530105270 2024-05-27T07:12:57.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.294+0000 7fe537000700 1 -- 172.21.15.43:0/4047905049 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5301a29d0 con 0x7fe530105270 2024-05-27T07:12:57.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.295+0000 7fe537000700 1 -- 172.21.15.43:0/4047905049 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5300512b0 con 0x7fe530105270 2024-05-27T07:12:57.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.298+0000 7fe52e000700 1 -- 172.21.15.43:0/4047905049 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fe51c019070 con 0x7fe530105270 2024-05-27T07:12:57.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.299+0000 7fe52e000700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5180b2880 0x7fe5180b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:12:57.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.299+0000 7fe52e000700 1 -- 172.21.15.43:0/4047905049 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fe51c0e2230 con 0x7fe530105270 2024-05-27T07:12:57.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.299+0000 7fe535c00700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5180b2880 0x7fe5180b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:12:57.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.300+0000 7fe535c00700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5180b2880 0x7fe5180b4d90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe524006010 tx=0x7fe52401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:12:57.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.302+0000 7fe52e000700 1 -- 172.21.15.43:0/4047905049 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe51c0a9b60 con 0x7fe530105270 2024-05-27T07:12:57.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:57 smithi179 ceph-mon[35888]: pgmap v532: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.542+0000 7fe537000700 1 -- 172.21.15.43:0/4047905049 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fe530063bf0 con 0x7fe5180b2880 2024-05-27T07:12:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.546+0000 7fe52e000700 1 -- 172.21.15.43:0/4047905049 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fe530063bf0 con 0x7fe5180b2880 2024-05-27T07:12:57.537 INFO:teuthology.orchestra.run.smithi043.stdout:HOST ADDR LABELS STATUS 2024-05-27T07:12:57.537 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 172.21.15.43 2024-05-27T07:12:57.537 INFO:teuthology.orchestra.run.smithi043.stdout:smithi132 172.21.15.132 2024-05-27T07:12:57.537 INFO:teuthology.orchestra.run.smithi043.stdout:smithi179 172.21.15.179 2024-05-27T07:12:57.537 INFO:teuthology.orchestra.run.smithi043.stdout:3 hosts in cluster 2024-05-27T07:12:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.549+0000 7fe537000700 1 -- 172.21.15.43:0/4047905049 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5180b2880 msgr2=0x7fe5180b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.549+0000 7fe537000700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5180b2880 0x7fe5180b4d90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe524006010 tx=0x7fe52401a040 comp rx=0 tx=0).stop 2024-05-27T07:12:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.549+0000 7fe537000700 1 -- 172.21.15.43:0/4047905049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe530105270 msgr2=0x7fe53019ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:12:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.549+0000 7fe537000700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe530105270 0x7fe53019ab80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00d950 tx=0x7fe51c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:12:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.549+0000 7fe537000700 1 -- 172.21.15.43:0/4047905049 shutdown_connections 2024-05-27T07:12:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.549+0000 7fe537000700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe5180b2880 0x7fe5180b4d90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.549+0000 7fe537000700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe530105c10 0x7fe5301a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.550+0000 7fe537000700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe530104140 0x7fe53019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.550+0000 7fe537000700 1 --2- 172.21.15.43:0/4047905049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe530105270 0x7fe53019ab80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:12:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.550+0000 7fe537000700 1 -- 172.21.15.43:0/4047905049 >> 172.21.15.43:0/4047905049 conn(0x7fe5300ff7a0 msgr2=0x7fe5301084a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:12:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.550+0000 7fe537000700 1 -- 172.21.15.43:0/4047905049 shutdown_connections 2024-05-27T07:12:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:12:57.550+0000 7fe537000700 1 -- 172.21.15.43:0/4047905049 wait complete. 2024-05-27T07:12:57.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:57 smithi132 ceph-mon[35283]: pgmap v532: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:12:58.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:58 smithi043 ceph-mon[31531]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:58.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-27T07:12:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:58 smithi179 ceph-mon[35888]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:58.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:58 smithi132 ceph-mon[35283]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:12:58.924 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:12:59.416 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:12:59 smithi043 ceph-mon[31531]: pgmap v533: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:59.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:12:59 smithi179 ceph-mon[35888]: pgmap v533: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:12:59.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:12:59 smithi132 ceph-mon[35283]: pgmap v533: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.355+0000 7fe698a00700 1 -- 172.21.15.43:0/1721672696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe694107e50 msgr2=0x7fe69410e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.355+0000 7fe698a00700 1 --2- 172.21.15.43:0/1721672696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe694107e50 0x7fe69410e700 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009990 tx=0x7fe68c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:13:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.355+0000 7fe698a00700 1 -- 172.21.15.43:0/1721672696 shutdown_connections 2024-05-27T07:13:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.355+0000 7fe698a00700 1 --2- 172.21.15.43:0/1721672696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe694107e50 0x7fe69410e700 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.355+0000 7fe698a00700 1 --2- 172.21.15.43:0/1721672696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe694106380 0x7fe694106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.355+0000 7fe698a00700 1 --2- 172.21.15.43:0/1721672696 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6941074b0 0x7fe6941078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.355+0000 7fe698a00700 1 -- 172.21.15.43:0/1721672696 >> 172.21.15.43:0/1721672696 conn(0x7fe694101a00 msgr2=0x7fe694103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.356+0000 7fe698a00700 1 -- 172.21.15.43:0/1721672696 shutdown_connections 2024-05-27T07:13:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.356+0000 7fe698a00700 1 -- 172.21.15.43:0/1721672696 wait complete. 2024-05-27T07:13:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.357+0000 7fe698a00700 1 Processor -- start 2024-05-27T07:13:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.357+0000 7fe698a00700 1 -- start start 2024-05-27T07:13:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.358+0000 7fe698a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe694106380 0x7fe69419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.358+0000 7fe698a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6941074b0 0x7fe69419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.358+0000 7fe698a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe694107e50 0x7fe6941a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.358+0000 7fe698a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe694111cf0 con 0x7fe6941074b0 2024-05-27T07:13:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.358+0000 7fe698a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe694111b70 con 0x7fe694107e50 2024-05-27T07:13:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.358+0000 7fe698a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe694111e70 con 0x7fe694106380 2024-05-27T07:13:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.358+0000 7fe68aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6941074b0 0x7fe69419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.358+0000 7fe68aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6941074b0 0x7fe69419cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57354/0 (socket says 172.21.15.43:57354) 2024-05-27T07:13:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.358+0000 7fe68aa00700 1 -- 172.21.15.43:0/1806874836 learned_addr learned my addr 172.21.15.43:0/1806874836 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.359+0000 7fe693400700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe694106380 0x7fe69419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.359+0000 7fe693e00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe694107e50 0x7fe6941a3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.359+0000 7fe68aa00700 1 -- 172.21.15.43:0/1806874836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe694106380 msgr2=0x7fe69419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.359+0000 7fe68aa00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe694106380 0x7fe69419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.359+0000 7fe68aa00700 1 -- 172.21.15.43:0/1806874836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe694107e50 msgr2=0x7fe6941a3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.359+0000 7fe68aa00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe694107e50 0x7fe6941a3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.359+0000 7fe68aa00700 1 -- 172.21.15.43:0/1806874836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6941a4300 con 0x7fe6941074b0 2024-05-27T07:13:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.359+0000 7fe693e00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe694107e50 0x7fe6941a3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.359+0000 7fe68aa00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6941074b0 0x7fe69419cd70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00d950 tx=0x7fe67c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.360+0000 7fe690c00700 1 -- 172.21.15.43:0/1806874836 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67c016070 con 0x7fe6941074b0 2024-05-27T07:13:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.360+0000 7fe698a00700 1 -- 172.21.15.43:0/1806874836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6941a45f0 con 0x7fe6941074b0 2024-05-27T07:13:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.360+0000 7fe690c00700 1 -- 172.21.15.43:0/1806874836 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe67c009dd0 con 0x7fe6941074b0 2024-05-27T07:13:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.360+0000 7fe698a00700 1 -- 172.21.15.43:0/1806874836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6941a4b80 con 0x7fe6941074b0 2024-05-27T07:13:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.360+0000 7fe690c00700 1 -- 172.21.15.43:0/1806874836 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67c020060 con 0x7fe6941074b0 2024-05-27T07:13:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.361+0000 7fe698a00700 1 -- 172.21.15.43:0/1806874836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6940512b0 con 0x7fe6941074b0 2024-05-27T07:13:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.364+0000 7fe690c00700 1 -- 172.21.15.43:0/1806874836 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fe67c019070 con 0x7fe6941074b0 2024-05-27T07:13:01.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.366+0000 7fe690c00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe6780b2880 0x7fe6780b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:01.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.366+0000 7fe690c00700 1 -- 172.21.15.43:0/1806874836 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fe67c0e25c0 con 0x7fe6941074b0 2024-05-27T07:13:01.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.367+0000 7fe693400700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe6780b2880 0x7fe6780b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:01.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.367+0000 7fe693400700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe6780b2880 0x7fe6780b4d90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe6941073b0 tx=0x7fe684007a30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:01.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.371+0000 7fe690c00700 1 -- 172.21.15.43:0/1806874836 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe67c0a9ef0 con 0x7fe6941074b0 2024-05-27T07:13:01.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.619+0000 7fe698a00700 1 -- 172.21.15.43:0/1806874836 --> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fe694063bf0 con 0x7fe6780b2880 2024-05-27T07:13:01.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.623+0000 7fe690c00700 1 -- 172.21.15.43:0/1806874836 <== mgr.14152 v2:172.21.15.43:6800/1809594794 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fe694063bf0 con 0x7fe6780b2880 2024-05-27T07:13:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-27T07:13:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300GH480DGN 447G No 7m ago Has a FileSystem, LVM detected 2024-05-27T07:13:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:smithi132 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232000B400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-05-27T07:13:01.614 INFO:teuthology.orchestra.run.smithi043.stdout:smithi179 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303009J400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-05-27T07:13:01.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.626+0000 7fe698a00700 1 -- 172.21.15.43:0/1806874836 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe6780b2880 msgr2=0x7fe6780b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:01.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.626+0000 7fe698a00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe6780b2880 0x7fe6780b4d90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe6941073b0 tx=0x7fe684007a30 comp rx=0 tx=0).stop 2024-05-27T07:13:01.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.626+0000 7fe698a00700 1 -- 172.21.15.43:0/1806874836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6941074b0 msgr2=0x7fe69419cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:01.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.626+0000 7fe698a00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6941074b0 0x7fe69419cd70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00d950 tx=0x7fe67c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:13:01.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.626+0000 7fe698a00700 1 -- 172.21.15.43:0/1806874836 shutdown_connections 2024-05-27T07:13:01.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.626+0000 7fe698a00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe6780b2880 0x7fe6780b4d90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:01.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.626+0000 7fe698a00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe694107e50 0x7fe6941a3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:01.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.626+0000 7fe698a00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe694106380 0x7fe69419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:01.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.626+0000 7fe698a00700 1 --2- 172.21.15.43:0/1806874836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6941074b0 0x7fe69419cd70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:01.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.626+0000 7fe698a00700 1 -- 172.21.15.43:0/1806874836 >> 172.21.15.43:0/1806874836 conn(0x7fe694101a00 msgr2=0x7fe69410a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:01.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.627+0000 7fe698a00700 1 -- 172.21.15.43:0/1806874836 shutdown_connections 2024-05-27T07:13:01.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:01.627+0000 7fe698a00700 1 -- 172.21.15.43:0/1806874836 wait complete. 2024-05-27T07:13:02.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:01 smithi043 ceph-mon[31531]: pgmap v534: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:02.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:01 smithi179 ceph-mon[35888]: pgmap v534: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:02.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:01 smithi132 ceph-mon[35283]: pgmap v534: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:02.592 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-27T07:13:02.604 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-05-27T07:13:02.604 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-27T07:13:03.087 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:03.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:02 smithi043 ceph-mon[31531]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:13:03.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:02 smithi179 ceph-mon[35888]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:13:03.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:02 smithi132 ceph-mon[35283]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:13:04.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:03 smithi043 ceph-mon[31531]: pgmap v535: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:04.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:03 smithi179 ceph-mon[35888]: pgmap v535: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:03 smithi132 ceph-mon[35283]: pgmap v535: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.494+0000 7f2729a00700 1 -- 172.21.15.43:0/779384761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f272410c1e0 msgr2=0x7f272410e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.494+0000 7f2729a00700 1 --2- 172.21.15.43:0/779384761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f272410c1e0 0x7f272410e6f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f271c00b0a0 tx=0x7f271c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:13:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.494+0000 7f2729a00700 1 -- 172.21.15.43:0/779384761 shutdown_connections 2024-05-27T07:13:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.494+0000 7f2729a00700 1 --2- 172.21.15.43:0/779384761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f272410c1e0 0x7f272410e6f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.494+0000 7f2729a00700 1 --2- 172.21.15.43:0/779384761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27240776d0 0x7f2724075ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.494+0000 7f2729a00700 1 --2- 172.21.15.43:0/779384761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27240760e0 0x7f2724076570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.494+0000 7f2729a00700 1 -- 172.21.15.43:0/779384761 >> 172.21.15.43:0/779384761 conn(0x7f27240ff9b0 msgr2=0x7f2724101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.495+0000 7f2729a00700 1 -- 172.21.15.43:0/779384761 shutdown_connections 2024-05-27T07:13:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.495+0000 7f2729a00700 1 -- 172.21.15.43:0/779384761 wait complete. 2024-05-27T07:13:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.496+0000 7f2729a00700 1 Processor -- start 2024-05-27T07:13:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.496+0000 7f2729a00700 1 -- start start 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.496+0000 7f2729a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27240760e0 0x7f272419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.496+0000 7f2729a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27240776d0 0x7f272419cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.496+0000 7f2729a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f272410c1e0 0x7f27241a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.496+0000 7f2729a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2724111e90 con 0x7f272410c1e0 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.496+0000 7f2729a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2724111d10 con 0x7f27240776d0 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.496+0000 7f2729a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2724112010 con 0x7f27240760e0 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.497+0000 7f2723400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f272410c1e0 0x7f27241a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.497+0000 7f2723400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f272410c1e0 0x7f27241a3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49724/0 (socket says 172.21.15.43:49724) 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.497+0000 7f2723400700 1 -- 172.21.15.43:0/1918241382 learned_addr learned my addr 172.21.15.43:0/1918241382 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.497+0000 7f2722a00700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27240760e0 0x7f272419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.497+0000 7f271be00700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27240776d0 0x7f272419cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.497+0000 7f2723400700 1 -- 172.21.15.43:0/1918241382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27240760e0 msgr2=0x7f272419ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.497+0000 7f2723400700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27240760e0 0x7f272419ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.497+0000 7f2723400700 1 -- 172.21.15.43:0/1918241382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27240776d0 msgr2=0x7f272419cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.497+0000 7f2723400700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27240776d0 0x7f272419cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.497+0000 7f2723400700 1 -- 172.21.15.43:0/1918241382 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27241a44e0 con 0x7f272410c1e0 2024-05-27T07:13:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.498+0000 7f2723400700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f272410c1e0 0x7f27241a3ef0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f271c0062a0 tx=0x7f271c004f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.498+0000 7f271b400700 1 -- 172.21.15.43:0/1918241382 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c01f030 con 0x7f272410c1e0 2024-05-27T07:13:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.498+0000 7f271b400700 1 -- 172.21.15.43:0/1918241382 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f271c007a40 con 0x7f272410c1e0 2024-05-27T07:13:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.498+0000 7f2729a00700 1 -- 172.21.15.43:0/1918241382 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27241a4770 con 0x7f272410c1e0 2024-05-27T07:13:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.498+0000 7f271b400700 1 -- 172.21.15.43:0/1918241382 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c004710 con 0x7f272410c1e0 2024-05-27T07:13:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.498+0000 7f2729a00700 1 -- 172.21.15.43:0/1918241382 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27241a4c20 con 0x7f272410c1e0 2024-05-27T07:13:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.500+0000 7f2729a00700 1 -- 172.21.15.43:0/1918241382 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27240512b0 con 0x7f272410c1e0 2024-05-27T07:13:05.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.502+0000 7f271b400700 1 -- 172.21.15.43:0/1918241382 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f271c003a00 con 0x7f272410c1e0 2024-05-27T07:13:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.504+0000 7f271b400700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f27080b2880 0x7f27080b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.504+0000 7f271b400700 1 -- 172.21.15.43:0/1918241382 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f271c00f020 con 0x7f272410c1e0 2024-05-27T07:13:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.504+0000 7f2722a00700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f27080b2880 0x7f27080b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.505+0000 7f2722a00700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f27080b2880 0x7f27080b4d90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2714005950 tx=0x7f271400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:05.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.509+0000 7f271b400700 1 -- 172.21.15.43:0/1918241382 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f271c029020 con 0x7f272410c1e0 2024-05-27T07:13:05.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.800+0000 7f2729a00700 1 -- 172.21.15.43:0/1918241382 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f27240643c0 con 0x7f272410c1e0 2024-05-27T07:13:05.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.801+0000 7f271b400700 1 -- 172.21.15.43:0/1918241382 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) ==== 76+0+3096 (secure 0 0 0) 0x7f271c0ab330 con 0x7f272410c1e0 2024-05-27T07:13:05.793 INFO:teuthology.orchestra.run.smithi043.stdout:e29 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout:legacy client fscid: 1 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout:Filesystem 'cephfs' (1) 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout:fs_name cephfs 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 29 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout:created 2024-05-27T07:08:25.128097+0000 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout:modified 2024-05-27T07:09:57.102577+0000 2024-05-27T07:13:05.794 INFO:teuthology.orchestra.run.smithi043.stdout:tableserver 0 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:root 0 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:session_timeout 300 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:session_autoclose 300 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:max_file_size 1099511627776 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:max_xattr_size 65536 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:required_client_features {} 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:last_failure 0 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:last_failure_osd_epoch 0 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:max_mds 5 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:in 0,1,2,3,4 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:up {0=34484,1=34448,2=34424,3=34403,4=24484} 2024-05-27T07:13:05.795 INFO:teuthology.orchestra.run.smithi043.stdout:failed 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:damaged 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:stopped 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:data_pools [3,4] 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:metadata_pool 2 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:inline_data disabled 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:balancer 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:bal_rank_mask -1 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:standby_count_wanted 0 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:qdb_cluster leader: 34484 members: 24484,34403,34424,34448,34484 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.b{0:34484} state up:active seq 8 addr [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.c{0:24469} state up:standby-replay seq 1 addr [v2:172.21.15.179:6836/4061292168,v1:172.21.15.179:6837/4061292168] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.h{1:34448} state up:active seq 33 addr [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.796 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.f{1:24457} state up:standby-replay seq 1 addr [v2:172.21.15.179:6834/2538262781,v1:172.21.15.179:6835/2538262781] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.e{2:34424} state up:active seq 36 addr [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.l{2:24412} state up:standby-replay seq 1 addr [v2:172.21.15.179:6832/4280267074,v1:172.21.15.179:6833/4280267074] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.k{3:34403} state up:active seq 40 addr [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.a{3:14613} state up:standby-replay seq 1 addr [v2:172.21.15.43:6840/2697504369,v1:172.21.15.43:6841/2697504369] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.i{4:24484} state up:active seq 33 addr [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.g{4:14601} state up:standby-replay seq 1 addr [v2:172.21.15.43:6838/807126288,v1:172.21.15.43:6839/807126288] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout:Standby daemons: 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.j{-1:14544} state up:standby seq 1 addr [v2:172.21.15.43:6834/3664629623,v1:172.21.15.43:6835/3664629623] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.797 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.d{-1:14556} state up:standby seq 1 addr [v2:172.21.15.43:6836/3666274467,v1:172.21.15.43:6837/3666274467] compat {c=[1],r=[1],i=[1fff]}] 2024-05-27T07:13:05.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 -- 172.21.15.43:0/1918241382 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f27080b2880 msgr2=0x7f27080b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:05.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f27080b2880 0x7f27080b4d90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2714005950 tx=0x7f271400a300 comp rx=0 tx=0).stop 2024-05-27T07:13:05.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 -- 172.21.15.43:0/1918241382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f272410c1e0 msgr2=0x7f27241a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:05.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f272410c1e0 0x7f27241a3ef0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f271c0062a0 tx=0x7f271c004f40 comp rx=0 tx=0).stop 2024-05-27T07:13:05.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 -- 172.21.15.43:0/1918241382 shutdown_connections 2024-05-27T07:13:05.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f27080b2880 0x7f27080b4d90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:05.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f272410c1e0 0x7f27241a3ef0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:05.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27240760e0 0x7f272419ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:05.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 --2- 172.21.15.43:0/1918241382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27240776d0 0x7f272419cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:05.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 -- 172.21.15.43:0/1918241382 >> 172.21.15.43:0/1918241382 conn(0x7f27240ff9b0 msgr2=0x7f2724108170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:05.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.806+0000 7f2729a00700 1 -- 172.21.15.43:0/1918241382 shutdown_connections 2024-05-27T07:13:05.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:05.807+0000 7f2729a00700 1 -- 172.21.15.43:0/1918241382 wait complete. 2024-05-27T07:13:05.799 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-05-27T07:13:06.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:05 smithi043 ceph-mon[31531]: pgmap v536: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:06.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:05 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1918241382' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:13:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:05 smithi179 ceph-mon[35888]: pgmap v536: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:05 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1918241382' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:13:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:05 smithi132 ceph-mon[35283]: pgmap v536: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:05 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1918241382' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-27T07:13:06.731 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-27T07:13:07.197 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:08.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:07 smithi043 ceph-mon[31531]: pgmap v537: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:07 smithi179 ceph-mon[35888]: pgmap v537: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:08.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:07 smithi132 ceph-mon[35283]: pgmap v537: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.759+0000 7f1f6e600700 1 -- 172.21.15.43:0/2050992196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f6810c410 msgr2=0x7f1f68112cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.759+0000 7f1f6e600700 1 --2- 172.21.15.43:0/2050992196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f6810c410 0x7f1f68112cc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000b0a0 tx=0x7f1f6000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:13:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.759+0000 7f1f6e600700 1 -- 172.21.15.43:0/2050992196 shutdown_connections 2024-05-27T07:13:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.759+0000 7f1f6e600700 1 --2- 172.21.15.43:0/2050992196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f6810c410 0x7f1f68112cc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.759+0000 7f1f6e600700 1 --2- 172.21.15.43:0/2050992196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6810ab00 0x7f1f6810af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.759+0000 7f1f6e600700 1 --2- 172.21.15.43:0/2050992196 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f6810ba70 0x7f1f6810be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.759+0000 7f1f6e600700 1 -- 172.21.15.43:0/2050992196 >> 172.21.15.43:0/2050992196 conn(0x7f1f680799b0 msgr2=0x7f1f6807be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.759+0000 7f1f6e600700 1 -- 172.21.15.43:0/2050992196 shutdown_connections 2024-05-27T07:13:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.759+0000 7f1f6e600700 1 -- 172.21.15.43:0/2050992196 wait complete. 2024-05-27T07:13:09.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.760+0000 7f1f6e600700 1 Processor -- start 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.761+0000 7f1f6e600700 1 -- start start 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.761+0000 7f1f6e600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f6810ab00 0x7f1f681a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.761+0000 7f1f6e600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6810ba70 0x7f1f681a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.761+0000 7f1f6e600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f6810c410 0x7f1f681a82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.761+0000 7f1f6e600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1f681162d0 con 0x7f1f6810ab00 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.761+0000 7f1f6e600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68116150 con 0x7f1f6810c410 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.761+0000 7f1f6e600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68116450 con 0x7f1f6810ba70 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.761+0000 7f1f67400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f6810ab00 0x7f1f681a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.761+0000 7f1f67400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f6810ab00 0x7f1f681a0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49742/0 (socket says 172.21.15.43:49742) 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.761+0000 7f1f67400700 1 -- 172.21.15.43:0/1121041271 learned_addr learned my addr 172.21.15.43:0/1121041271 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:09.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.762+0000 7f1f67e00700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f6810c410 0x7f1f681a82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:09.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.762+0000 7f1f5fe00700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6810ba70 0x7f1f681a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:09.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.762+0000 7f1f67400700 1 -- 172.21.15.43:0/1121041271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6810ba70 msgr2=0x7f1f681a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:09.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.762+0000 7f1f67400700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6810ba70 0x7f1f681a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:09.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.762+0000 7f1f67400700 1 -- 172.21.15.43:0/1121041271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f6810c410 msgr2=0x7f1f681a82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:09.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.762+0000 7f1f67400700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f6810c410 0x7f1f681a82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:09.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.762+0000 7f1f67400700 1 -- 172.21.15.43:0/1121041271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f681a8950 con 0x7f1f6810ab00 2024-05-27T07:13:09.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.762+0000 7f1f67400700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f6810ab00 0x7f1f681a0df0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1f5800d950 tx=0x7f1f5800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:09.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.763+0000 7f1f64c00700 1 -- 172.21.15.43:0/1121041271 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58017070 con 0x7f1f6810ab00 2024-05-27T07:13:09.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.763+0000 7f1f64c00700 1 -- 172.21.15.43:0/1121041271 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f58009dc0 con 0x7f1f6810ab00 2024-05-27T07:13:09.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.763+0000 7f1f6e600700 1 -- 172.21.15.43:0/1121041271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f681a8c40 con 0x7f1f6810ab00 2024-05-27T07:13:09.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.763+0000 7f1f64c00700 1 -- 172.21.15.43:0/1121041271 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58010de0 con 0x7f1f6810ab00 2024-05-27T07:13:09.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.763+0000 7f1f6e600700 1 -- 172.21.15.43:0/1121041271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f681a9130 con 0x7f1f6810ab00 2024-05-27T07:13:09.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.764+0000 7f1f6e600700 1 -- 172.21.15.43:0/1121041271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f680512b0 con 0x7f1f6810ab00 2024-05-27T07:13:09.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.767+0000 7f1f64c00700 1 -- 172.21.15.43:0/1121041271 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f1f5801a070 con 0x7f1f6810ab00 2024-05-27T07:13:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.769+0000 7f1f64c00700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1f4c0b2880 0x7f1f4c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.769+0000 7f1f64c00700 1 -- 172.21.15.43:0/1121041271 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f1f580e1ee0 con 0x7f1f6810ab00 2024-05-27T07:13:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.769+0000 7f1f5fe00700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1f4c0b2880 0x7f1f4c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.770+0000 7f1f5fe00700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1f4c0b2880 0x7f1f4c0b4d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1f50005950 tx=0x7f1f500058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:09.771+0000 7f1f64c00700 1 -- 172.21.15.43:0/1121041271 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1f580a9810 con 0x7f1f6810ab00 2024-05-27T07:13:10.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.009+0000 7f1f6e600700 1 -- 172.21.15.43:0/1121041271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f1f680643c0 con 0x7f1f6810ab00 2024-05-27T07:13:10.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.012+0000 7f1f64c00700 1 -- 172.21.15.43:0/1121041271 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5653 (secure 0 0 0) 0x7f1f580a9110 con 0x7f1f6810ab00 2024-05-27T07:13:10.003 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:09 smithi043 ceph-mon[31531]: pgmap v538: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:10.003 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 90 2024-05-27T07:13:10.003 INFO:teuthology.orchestra.run.smithi043.stdout:fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T07:13:10.003 INFO:teuthology.orchestra.run.smithi043.stdout:created 2024-05-27T06:56:43.552142+0000 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:modified 2024-05-27T07:09:54.019617+0000 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:crush_version 27 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:full_ratio 0.95 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:backfillfull_ratio 0.9 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:nearfull_ratio 0.85 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:require_min_compat_client luminous 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:min_compat_client luminous 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:require_osd_release squid 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:stretch_mode_enabled false 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-05-27T07:13:10.004 INFO:teuthology.orchestra.run.smithi043.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-05-27T07:13:10.005 INFO:teuthology.orchestra.run.smithi043.stdout:max_osd 12 2024-05-27T07:13:10.005 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6802/3368788577,v1:172.21.15.43:6803/3368788577] [v2:172.21.15.43:6804/3368788577,v1:172.21.15.43:6805/3368788577] exists,up 92381bc8-d8af-4c8d-b55c-2311e7de0211 2024-05-27T07:13:10.005 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 up in weight 1 up_from 13 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6800/3434174667,v1:172.21.15.132:6801/3434174667] [v2:172.21.15.132:6802/3434174667,v1:172.21.15.132:6803/3434174667] exists,up 8749b216-d75b-4f50-be38-9d9c7c67fe57 2024-05-27T07:13:10.005 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 up in weight 1 up_from 18 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6800/2313942243,v1:172.21.15.179:6801/2313942243] [v2:172.21.15.179:6802/2313942243,v1:172.21.15.179:6803/2313942243] exists,up 97582b6e-2b42-48b6-bf39-511a1121e069 2024-05-27T07:13:10.005 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 up in weight 1 up_from 26 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6810/902648995,v1:172.21.15.43:6811/902648995] [v2:172.21.15.43:6812/902648995,v1:172.21.15.43:6813/902648995] exists,up 8247fd20-bc21-430a-a3b9-bd5221468995 2024-05-27T07:13:10.005 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 up in weight 1 up_from 32 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6808/4132674108,v1:172.21.15.132:6809/4132674108] [v2:172.21.15.132:6810/4132674108,v1:172.21.15.132:6811/4132674108] exists,up 8f6a847d-4753-4061-bf71-27d740189512 2024-05-27T07:13:10.005 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 up in weight 1 up_from 37 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6808/1714122228,v1:172.21.15.179:6809/1714122228] [v2:172.21.15.179:6810/1714122228,v1:172.21.15.179:6811/1714122228] exists,up dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12 2024-05-27T07:13:10.005 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 up in weight 1 up_from 43 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6818/2712807127,v1:172.21.15.43:6819/2712807127] [v2:172.21.15.43:6820/2712807127,v1:172.21.15.43:6821/2712807127] exists,up a722fe71-7939-43d7-9c9e-72f2c7e559be 2024-05-27T07:13:10.005 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 up in weight 1 up_from 48 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6816/2427045470,v1:172.21.15.132:6817/2427045470] [v2:172.21.15.132:6818/2427045470,v1:172.21.15.132:6819/2427045470] exists,up 3b6b6895-34fe-4afc-90c1-a0d30b365713 2024-05-27T07:13:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:osd.8 up in weight 1 up_from 54 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6816/2617404629,v1:172.21.15.179:6817/2617404629] [v2:172.21.15.179:6818/2617404629,v1:172.21.15.179:6819/2617404629] exists,up 10a2bd5e-5ac8-441f-b2e5-deb1f9239b17 2024-05-27T07:13:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:osd.9 up in weight 1 up_from 60 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6826/3342932073,v1:172.21.15.43:6827/3342932073] [v2:172.21.15.43:6828/3342932073,v1:172.21.15.43:6829/3342932073] exists,up 19e77076-5052-4d5f-a35b-00bb595cf435 2024-05-27T07:13:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:osd.10 up in weight 1 up_from 65 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6824/3139130496,v1:172.21.15.132:6825/3139130496] [v2:172.21.15.132:6826/3139130496,v1:172.21.15.132:6827/3139130496] exists,up 78b7db29-142a-4f61-a360-8139d14f676b 2024-05-27T07:13:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:osd.11 up in weight 1 up_from 71 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6824/55249493,v1:172.21.15.179:6825/55249493] [v2:172.21.15.179:6826/55249493,v1:172.21.15.179:6827/55249493] exists,up dfe7c464-567c-4836-8042-4bb9daedb76b 2024-05-27T07:13:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 2.e [2,1] 2024-05-27T07:13:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 2.10 [8,11] 2024-05-27T07:13:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 2.14 [2,4] 2024-05-27T07:13:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 2.1c [8,11] 2024-05-27T07:13:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 3.0 [5,9] 2024-05-27T07:13:10.006 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 3.c [3,9] 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 3.17 [4,8] 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 3.36 [4,9] 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.2 [9,1] 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.8 [5,8] 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.12 [9,10] 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.18 [9,8] 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6801/1651567761 expires 2024-05-28T06:57:52.506320+0000 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6800/1651567761 expires 2024-05-28T06:57:52.506320+0000 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/1332970913 expires 2024-05-28T06:57:18.380825+0000 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/1752945868 expires 2024-05-28T06:57:18.380825+0000 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/2426463283 expires 2024-05-28T06:57:18.380825+0000 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/2081880966 expires 2024-05-28T06:57:52.506320+0000 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6800/3521718752 expires 2024-05-28T06:57:18.380825+0000 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/830904719 expires 2024-05-28T06:57:52.506320+0000 2024-05-27T07:13:10.007 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6801/3521718752 expires 2024-05-28T06:57:18.380825+0000 2024-05-27T07:13:10.008 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/338672925 expires 2024-05-28T06:57:52.506320+0000 2024-05-27T07:13:10.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.015+0000 7f1f6e600700 1 -- 172.21.15.43:0/1121041271 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1f4c0b2880 msgr2=0x7f1f4c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.015+0000 7f1f6e600700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1f4c0b2880 0x7f1f4c0b4d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1f50005950 tx=0x7f1f500058e0 comp rx=0 tx=0).stop 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.016+0000 7f1f6e600700 1 -- 172.21.15.43:0/1121041271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f6810ab00 msgr2=0x7f1f681a0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.016+0000 7f1f6e600700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f6810ab00 0x7f1f681a0df0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1f5800d950 tx=0x7f1f5800dc60 comp rx=0 tx=0).stop 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.016+0000 7f1f6e600700 1 -- 172.21.15.43:0/1121041271 shutdown_connections 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.016+0000 7f1f6e600700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f1f4c0b2880 0x7f1f4c0b4d90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.016+0000 7f1f6e600700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f6810c410 0x7f1f681a82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.016+0000 7f1f6e600700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f6810ab00 0x7f1f681a0df0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.016+0000 7f1f6e600700 1 --2- 172.21.15.43:0/1121041271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6810ba70 0x7f1f681a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.016+0000 7f1f6e600700 1 -- 172.21.15.43:0/1121041271 >> 172.21.15.43:0/1121041271 conn(0x7f1f680799b0 msgr2=0x7f1f6810eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.016+0000 7f1f6e600700 1 -- 172.21.15.43:0/1121041271 shutdown_connections 2024-05-27T07:13:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:10.016+0000 7f1f6e600700 1 -- 172.21.15.43:0/1121041271 wait complete. 2024-05-27T07:13:10.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:09 smithi179 ceph-mon[35888]: pgmap v538: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:10.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:09 smithi132 ceph-mon[35283]: pgmap v538: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:10.949 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-27T07:13:10.961 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:13:11.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:10 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1121041271' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:13:11.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:10 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1121041271' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:13:11.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:10 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1121041271' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-27T07:13:11.438 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:12.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:11 smithi043 ceph-mon[31531]: pgmap v539: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:11 smithi132 ceph-mon[35283]: pgmap v539: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:11 smithi179 ceph-mon[35888]: pgmap v539: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.877+0000 7ff4fe400700 1 -- 172.21.15.43:0/4112704609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4f810af40 msgr2=0x7ff4f810b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.877+0000 7ff4fe400700 1 --2- 172.21.15.43:0/4112704609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4f810af40 0x7ff4f810b350 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40099c0 tx=0x7ff4e4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.877+0000 7ff4fe400700 1 -- 172.21.15.43:0/4112704609 shutdown_connections 2024-05-27T07:13:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.877+0000 7ff4fe400700 1 --2- 172.21.15.43:0/4112704609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4f810b890 0x7ff4f81131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.877+0000 7ff4fe400700 1 --2- 172.21.15.43:0/4112704609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4f8106610 0x7ff4f8106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.877+0000 7ff4fe400700 1 --2- 172.21.15.43:0/4112704609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4f810af40 0x7ff4f810b350 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.877+0000 7ff4fe400700 1 -- 172.21.15.43:0/4112704609 >> 172.21.15.43:0/4112704609 conn(0x7ff4f8102020 msgr2=0x7ff4f8104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.877+0000 7ff4fe400700 1 -- 172.21.15.43:0/4112704609 shutdown_connections 2024-05-27T07:13:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.878+0000 7ff4fe400700 1 -- 172.21.15.43:0/4112704609 wait complete. 2024-05-27T07:13:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.879+0000 7ff4fe400700 1 Processor -- start 2024-05-27T07:13:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.879+0000 7ff4fe400700 1 -- start start 2024-05-27T07:13:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.880+0000 7ff4fe400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4f8106610 0x7ff4f819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.880+0000 7ff4fe400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4f810af40 0x7ff4f819d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.880+0000 7ff4fe400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4f810b890 0x7ff4f81a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.880+0000 7ff4fe400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f81163c0 con 0x7ff4f8106610 2024-05-27T07:13:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.880+0000 7ff4fe400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8116240 con 0x7ff4f810b890 2024-05-27T07:13:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.880+0000 7ff4fe400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8116540 con 0x7ff4f810af40 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.880+0000 7ff4fd000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4f8106610 0x7ff4f819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.880+0000 7ff4fd000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4f8106610 0x7ff4f819cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49758/0 (socket says 172.21.15.43:49758) 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.880+0000 7ff4fd000700 1 -- 172.21.15.43:0/4004594455 learned_addr learned my addr 172.21.15.43:0/4004594455 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.881+0000 7ff4f7e00700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4f810af40 0x7ff4f819d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.881+0000 7ff4fda00700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4f810b890 0x7ff4f81a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.881+0000 7ff4fd000700 1 -- 172.21.15.43:0/4004594455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4f810af40 msgr2=0x7ff4f819d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.881+0000 7ff4fd000700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4f810af40 0x7ff4f819d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.881+0000 7ff4fd000700 1 -- 172.21.15.43:0/4004594455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4f810b890 msgr2=0x7ff4f81a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.881+0000 7ff4fd000700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4f810b890 0x7ff4f81a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.881+0000 7ff4fd000700 1 -- 172.21.15.43:0/4004594455 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f81a1bf0 con 0x7ff4f8106610 2024-05-27T07:13:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.881+0000 7ff4fd000700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4f8106610 0x7ff4f819cdd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00c9b0 tx=0x7ff4ec00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.882+0000 7ff4f5600700 1 -- 172.21.15.43:0/4004594455 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec007b00 con 0x7ff4f8106610 2024-05-27T07:13:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.882+0000 7ff4f5600700 1 -- 172.21.15.43:0/4004594455 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4ec007ca0 con 0x7ff4f8106610 2024-05-27T07:13:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.882+0000 7ff4f5600700 1 -- 172.21.15.43:0/4004594455 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec018660 con 0x7ff4f8106610 2024-05-27T07:13:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.882+0000 7ff4fe400700 1 -- 172.21.15.43:0/4004594455 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f81a1ee0 con 0x7ff4f8106610 2024-05-27T07:13:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.882+0000 7ff4fe400700 1 -- 172.21.15.43:0/4004594455 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f81a2440 con 0x7ff4f8106610 2024-05-27T07:13:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.883+0000 7ff4fe400700 1 -- 172.21.15.43:0/4004594455 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f80512b0 con 0x7ff4f8106610 2024-05-27T07:13:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.886+0000 7ff4f5600700 1 -- 172.21.15.43:0/4004594455 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7ff4ec00f450 con 0x7ff4f8106610 2024-05-27T07:13:13.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.887+0000 7ff4f5600700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff4e00b2900 0x7ff4e00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:13.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.887+0000 7ff4f5600700 1 -- 172.21.15.43:0/4004594455 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7ff4ec0e28a0 con 0x7ff4f8106610 2024-05-27T07:13:13.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.888+0000 7ff4f7e00700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff4e00b2900 0x7ff4e00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:13.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.888+0000 7ff4f7e00700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff4e00b2900 0x7ff4e00b4e10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009990 tx=0x7ff4e4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:13.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:13.890+0000 7ff4f5600700 1 -- 172.21.15.43:0/4004594455 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff4ec014030 con 0x7ff4f8106610 2024-05-27T07:13:14.132 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:13 smithi043 ceph-mon[31531]: pgmap v540: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:14.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.188+0000 7ff4fe400700 1 -- 172.21.15.43:0/4004594455 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4f802e5e0 con 0x7ff4f8106610 2024-05-27T07:13:14.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.189+0000 7ff4f5600700 1 -- 172.21.15.43:0/4004594455 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7ff4ec0aa2e0 con 0x7ff4f8106610 2024-05-27T07:13:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:13:14.180 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:09:57.102577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[24484,34403,34424,34448,34484]},"id":1}]} 2024-05-27T07:13:14.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 -- 172.21.15.43:0/4004594455 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff4e00b2900 msgr2=0x7ff4e00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:14.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff4e00b2900 0x7ff4e00b4e10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009990 tx=0x7ff4e4005fb0 comp rx=0 tx=0).stop 2024-05-27T07:13:14.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 -- 172.21.15.43:0/4004594455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4f8106610 msgr2=0x7ff4f819cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:14.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4f8106610 0x7ff4f819cdd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00c9b0 tx=0x7ff4ec00cd70 comp rx=0 tx=0).stop 2024-05-27T07:13:14.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 -- 172.21.15.43:0/4004594455 shutdown_connections 2024-05-27T07:13:14.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7ff4e00b2900 0x7ff4e00b4e10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:14.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4f810b890 0x7ff4f81a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:14.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4f8106610 0x7ff4f819cdd0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:14.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 --2- 172.21.15.43:0/4004594455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4f810af40 0x7ff4f819d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:14.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 -- 172.21.15.43:0/4004594455 >> 172.21.15.43:0/4004594455 conn(0x7ff4f8102020 msgr2=0x7ff4f810a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:14.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 -- 172.21.15.43:0/4004594455 shutdown_connections 2024-05-27T07:13:14.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:14.193+0000 7ff4fe400700 1 -- 172.21.15.43:0/4004594455 wait complete. 2024-05-27T07:13:14.185 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-05-27T07:13:14.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:13 smithi132 ceph-mon[35283]: pgmap v540: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:14.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:13 smithi179 ceph-mon[35888]: pgmap v540: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:15.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:14 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4004594455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:13:15.144 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:13:15.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:14 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4004594455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:13:15.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:14 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4004594455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:13:15.621 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:16.136 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:16 smithi043 ceph-mon[31531]: pgmap v541: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:16.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:16 smithi132 ceph-mon[35283]: pgmap v541: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:16.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:16 smithi179 ceph-mon[35888]: pgmap v541: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.914+0000 7fe423600700 1 -- 172.21.15.43:0/3354332940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe41c110bf0 msgr2=0x7fe41c113010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.914+0000 7fe423600700 1 --2- 172.21.15.43:0/3354332940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe41c110bf0 0x7fe41c113010 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe41400b0a0 tx=0x7fe41400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:13:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.914+0000 7fe423600700 1 -- 172.21.15.43:0/3354332940 shutdown_connections 2024-05-27T07:13:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.914+0000 7fe423600700 1 --2- 172.21.15.43:0/3354332940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe41c110bf0 0x7fe41c113010 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.914+0000 7fe423600700 1 --2- 172.21.15.43:0/3354332940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe41c076be0 0x7fe41c076ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.914+0000 7fe423600700 1 --2- 172.21.15.43:0/3354332940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c077530 0x7fe41c1106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.914+0000 7fe423600700 1 -- 172.21.15.43:0/3354332940 >> 172.21.15.43:0/3354332940 conn(0x7fe41c0ffe90 msgr2=0x7fe41c1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.914+0000 7fe423600700 1 -- 172.21.15.43:0/3354332940 shutdown_connections 2024-05-27T07:13:17.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.914+0000 7fe423600700 1 -- 172.21.15.43:0/3354332940 wait complete. 2024-05-27T07:13:17.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.916+0000 7fe423600700 1 Processor -- start 2024-05-27T07:13:17.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.916+0000 7fe423600700 1 -- start start 2024-05-27T07:13:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.917+0000 7fe423600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe41c076be0 0x7fe41c19cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.917+0000 7fe423600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe41c077530 0x7fe41c19d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.917+0000 7fe423600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c110bf0 0x7fe41c1a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.917+0000 7fe423600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c116580 con 0x7fe41c076be0 2024-05-27T07:13:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.917+0000 7fe423600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c116400 con 0x7fe41c077530 2024-05-27T07:13:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.917+0000 7fe423600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c116700 con 0x7fe41c110bf0 2024-05-27T07:13:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.917+0000 7fe420c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe41c076be0 0x7fe41c19cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.917+0000 7fe420c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe41c076be0 0x7fe41c19cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55762/0 (socket says 172.21.15.43:55762) 2024-05-27T07:13:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.917+0000 7fe420c00700 1 -- 172.21.15.43:0/1164583987 learned_addr learned my addr 172.21.15.43:0/1164583987 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.918+0000 7fe41be00700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe41c077530 0x7fe41c19d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.918+0000 7fe421600700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c110bf0 0x7fe41c1a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.918+0000 7fe420c00700 1 -- 172.21.15.43:0/1164583987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c110bf0 msgr2=0x7fe41c1a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.918+0000 7fe420c00700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c110bf0 0x7fe41c1a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.918+0000 7fe420c00700 1 -- 172.21.15.43:0/1164583987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe41c077530 msgr2=0x7fe41c19d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.918+0000 7fe420c00700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe41c077530 0x7fe41c19d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.918+0000 7fe420c00700 1 -- 172.21.15.43:0/1164583987 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe41c1a1d90 con 0x7fe41c076be0 2024-05-27T07:13:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.918+0000 7fe41be00700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe41c077530 0x7fe41c19d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.918+0000 7fe420c00700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe41c076be0 0x7fe41c19cf70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe41000c9b0 tx=0x7fe41000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.919+0000 7fe419600700 1 -- 172.21.15.43:0/1164583987 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe410007b00 con 0x7fe41c076be0 2024-05-27T07:13:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.919+0000 7fe419600700 1 -- 172.21.15.43:0/1164583987 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe410007ca0 con 0x7fe41c076be0 2024-05-27T07:13:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.919+0000 7fe423600700 1 -- 172.21.15.43:0/1164583987 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe41c1a2080 con 0x7fe41c076be0 2024-05-27T07:13:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.919+0000 7fe419600700 1 -- 172.21.15.43:0/1164583987 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe410018660 con 0x7fe41c076be0 2024-05-27T07:13:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.919+0000 7fe423600700 1 -- 172.21.15.43:0/1164583987 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe41c1a9a50 con 0x7fe41c076be0 2024-05-27T07:13:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.920+0000 7fe423600700 1 -- 172.21.15.43:0/1164583987 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3fc005330 con 0x7fe41c076be0 2024-05-27T07:13:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.923+0000 7fe419600700 1 -- 172.21.15.43:0/1164583987 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fe4100188c0 con 0x7fe41c076be0 2024-05-27T07:13:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.924+0000 7fe419600700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4040b2900 0x7fe4040b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.925+0000 7fe419600700 1 -- 172.21.15.43:0/1164583987 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fe4100e27f0 con 0x7fe41c076be0 2024-05-27T07:13:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.925+0000 7fe41be00700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4040b2900 0x7fe4040b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.925+0000 7fe41be00700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4040b2900 0x7fe4040b4e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe408005e10 tx=0x7fe408005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:17.927+0000 7fe419600700 1 -- 172.21.15.43:0/1164583987 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe410014030 con 0x7fe41c076be0 2024-05-27T07:13:18.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:18 smithi043 ceph-mon[31531]: pgmap v542: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:18.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.227+0000 7fe423600700 1 -- 172.21.15.43:0/1164583987 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3fc005620 con 0x7fe41c076be0 2024-05-27T07:13:18.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.228+0000 7fe419600700 1 -- 172.21.15.43:0/1164583987 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11938 (secure 0 0 0) 0x7fe4100aa230 con 0x7fe41c076be0 2024-05-27T07:13:18.219 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:13:18.219 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:09:57.102577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[24484,34403,34424,34448,34484]},"id":1}]} 2024-05-27T07:13:18.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.231+0000 7fe423600700 1 -- 172.21.15.43:0/1164583987 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4040b2900 msgr2=0x7fe4040b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:18.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.231+0000 7fe423600700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4040b2900 0x7fe4040b4e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe408005e10 tx=0x7fe408005d60 comp rx=0 tx=0).stop 2024-05-27T07:13:18.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.231+0000 7fe423600700 1 -- 172.21.15.43:0/1164583987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe41c076be0 msgr2=0x7fe41c19cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:18.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.231+0000 7fe423600700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe41c076be0 0x7fe41c19cf70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe41000c9b0 tx=0x7fe41000cd70 comp rx=0 tx=0).stop 2024-05-27T07:13:18.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.232+0000 7fe423600700 1 -- 172.21.15.43:0/1164583987 shutdown_connections 2024-05-27T07:13:18.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.232+0000 7fe423600700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fe4040b2900 0x7fe4040b4e10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:18.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.232+0000 7fe423600700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c110bf0 0x7fe41c1a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:18.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.232+0000 7fe423600700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe41c076be0 0x7fe41c19cf70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:18.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.232+0000 7fe423600700 1 --2- 172.21.15.43:0/1164583987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe41c077530 0x7fe41c19d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:18.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.232+0000 7fe423600700 1 -- 172.21.15.43:0/1164583987 >> 172.21.15.43:0/1164583987 conn(0x7fe41c0ffe90 msgr2=0x7fe41c100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:18.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.232+0000 7fe423600700 1 -- 172.21.15.43:0/1164583987 shutdown_connections 2024-05-27T07:13:18.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:18.232+0000 7fe423600700 1 -- 172.21.15.43:0/1164583987 wait complete. 2024-05-27T07:13:18.224 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-05-27T07:13:18.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:18 smithi132 ceph-mon[35283]: pgmap v542: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:18 smithi179 ceph-mon[35888]: pgmap v542: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:19.180 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:13:19.258 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:19 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1164583987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:13:19.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:19 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1164583987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:13:19.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:19 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1164583987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:13:19.666 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:20.170 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:20 smithi043 ceph-mon[31531]: pgmap v543: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:20.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:20 smithi132 ceph-mon[35283]: pgmap v543: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:20.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:20 smithi179 ceph-mon[35888]: pgmap v543: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:21.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.995+0000 7fed8da00700 1 -- 172.21.15.43:0/3654292197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed88115360 msgr2=0x7fed88117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:21.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.995+0000 7fed8da00700 1 --2- 172.21.15.43:0/3654292197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed88115360 0x7fed88117780 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fed8000b0a0 tx=0x7fed8000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.995+0000 7fed8da00700 1 -- 172.21.15.43:0/3654292197 shutdown_connections 2024-05-27T07:13:21.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.995+0000 7fed8da00700 1 --2- 172.21.15.43:0/3654292197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed88115360 0x7fed88117780 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.995+0000 7fed8da00700 1 --2- 172.21.15.43:0/3654292197 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed88076c60 0x7fed88077070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.995+0000 7fed8da00700 1 --2- 172.21.15.43:0/3654292197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed880775b0 0x7fed88114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.995+0000 7fed8da00700 1 -- 172.21.15.43:0/3654292197 >> 172.21.15.43:0/3654292197 conn(0x7fed880ffeb0 msgr2=0x7fed88102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:21.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.996+0000 7fed8da00700 1 -- 172.21.15.43:0/3654292197 shutdown_connections 2024-05-27T07:13:21.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.996+0000 7fed8da00700 1 -- 172.21.15.43:0/3654292197 wait complete. 2024-05-27T07:13:21.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.997+0000 7fed8da00700 1 Processor -- start 2024-05-27T07:13:21.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.997+0000 7fed8da00700 1 -- start start 2024-05-27T07:13:21.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.997+0000 7fed8da00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed88076c60 0x7fed881a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:21.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.997+0000 7fed8da00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed880775b0 0x7fed881a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:21.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.997+0000 7fed8da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed88115360 0x7fed881a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.997+0000 7fed8da00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fed8811a910 con 0x7fed88115360 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.997+0000 7fed8da00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fed8811a790 con 0x7fed880775b0 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.997+0000 7fed8da00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fed8811aa90 con 0x7fed88076c60 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.998+0000 7fed87400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed88115360 0x7fed881a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.998+0000 7fed87400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed88115360 0x7fed881a5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55788/0 (socket says 172.21.15.43:55788) 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.998+0000 7fed87400700 1 -- 172.21.15.43:0/757906285 learned_addr learned my addr 172.21.15.43:0/757906285 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.998+0000 7fed86a00700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed88076c60 0x7fed881a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.998+0000 7fed86000700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed880775b0 0x7fed881a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.998+0000 7fed87400700 1 -- 172.21.15.43:0/757906285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed88076c60 msgr2=0x7fed881a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.998+0000 7fed87400700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed88076c60 0x7fed881a12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.998+0000 7fed87400700 1 -- 172.21.15.43:0/757906285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed880775b0 msgr2=0x7fed881a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.998+0000 7fed87400700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed880775b0 0x7fed881a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.998+0000 7fed87400700 1 -- 172.21.15.43:0/757906285 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed881a6100 con 0x7fed88115360 2024-05-27T07:13:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.999+0000 7fed87400700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed88115360 0x7fed881a5b10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fed800062a0 tx=0x7fed80004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.999+0000 7fed7f400700 1 -- 172.21.15.43:0/757906285 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80021070 con 0x7fed88115360 2024-05-27T07:13:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.999+0000 7fed7f400700 1 -- 172.21.15.43:0/757906285 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed80007d00 con 0x7fed88115360 2024-05-27T07:13:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.999+0000 7fed8da00700 1 -- 172.21.15.43:0/757906285 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed881a6390 con 0x7fed88115360 2024-05-27T07:13:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.999+0000 7fed7f400700 1 -- 172.21.15.43:0/757906285 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80005280 con 0x7fed88115360 2024-05-27T07:13:21.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:21.999+0000 7fed8da00700 1 -- 172.21.15.43:0/757906285 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed881a6820 con 0x7fed88115360 2024-05-27T07:13:21.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.000+0000 7fed8da00700 1 -- 172.21.15.43:0/757906285 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed880512b0 con 0x7fed88115360 2024-05-27T07:13:21.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.003+0000 7fed7f400700 1 -- 172.21.15.43:0/757906285 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7fed80024070 con 0x7fed88115360 2024-05-27T07:13:21.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.004+0000 7fed7f400700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed6c0b2880 0x7fed6c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:21.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.005+0000 7fed7f400700 1 -- 172.21.15.43:0/757906285 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fed800e3cd0 con 0x7fed88115360 2024-05-27T07:13:21.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.005+0000 7fed86a00700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed6c0b2880 0x7fed6c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:21.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.005+0000 7fed86a00700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed6c0b2880 0x7fed6c0b4d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fed78005ea0 tx=0x7fed78005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:21.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.007+0000 7fed7f400700 1 -- 172.21.15.43:0/757906285 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fed800ab690 con 0x7fed88115360 2024-05-27T07:13:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.248+0000 7fed8da00700 1 -- 172.21.15.43:0/757906285 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed8802e5e0 con 0x7fed88115360 2024-05-27T07:13:22.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.253+0000 7fed7f400700 1 -- 172.21.15.43:0/757906285 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24748 (secure 0 0 0) 0x7fed800aaf90 con 0x7fed88115360 2024-05-27T07:13:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:13:22.244 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":90,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:09:54.019617+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 -- 172.21.15.43:0/757906285 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed6c0b2880 msgr2=0x7fed6c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed6c0b2880 0x7fed6c0b4d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fed78005ea0 tx=0x7fed78005e30 comp rx=0 tx=0).stop 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 -- 172.21.15.43:0/757906285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed88115360 msgr2=0x7fed881a5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed88115360 0x7fed881a5b10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fed800062a0 tx=0x7fed80004ed0 comp rx=0 tx=0).stop 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 -- 172.21.15.43:0/757906285 shutdown_connections 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed6c0b2880 0x7fed6c0b4d90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed88115360 0x7fed881a5b10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed88076c60 0x7fed881a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 --2- 172.21.15.43:0/757906285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed880775b0 0x7fed881a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 -- 172.21.15.43:0/757906285 >> 172.21.15.43:0/757906285 conn(0x7fed880ffeb0 msgr2=0x7fed881006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 -- 172.21.15.43:0/757906285 shutdown_connections 2024-05-27T07:13:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:22.256+0000 7fed8da00700 1 -- 172.21.15.43:0/757906285 wait complete. 2024-05-27T07:13:22.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:22 smithi043 ceph-mon[31531]: pgmap v544: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:22.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:22 smithi132 ceph-mon[35283]: pgmap v544: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:22 smithi179 ceph-mon[35888]: pgmap v544: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:23.206 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-27T07:08:25.128097+0000', 'modified': '2024-05-27T07:09:57.102577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 34484, 'mds_1': 34448, 'mds_2': 34424, 'mds_3': 34403, 'mds_4': 24484}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14601': {'gid': 14601, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.43:6839/807126288', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6838', 'nonce': 807126288}, {'type': 'v1', 'addr': '172.21.15.43:6839', 'nonce': 807126288}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14613': {'gid': 14613, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.43:6841/2697504369', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6840', 'nonce': 2697504369}, {'type': 'v1', 'addr': '172.21.15.43:6841', 'nonce': 2697504369}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24412': {'gid': 24412, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6833/4280267074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6832', 'nonce': 4280267074}, {'type': 'v1', 'addr': '172.21.15.179:6833', 'nonce': 4280267074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6835/2538262781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6834', 'nonce': 2538262781}, {'type': 'v1', 'addr': '172.21.15.179:6835', 'nonce': 2538262781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6837/4061292168', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6836', 'nonce': 4061292168}, {'type': 'v1', 'addr': '172.21.15.179:6837', 'nonce': 4061292168}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 33, 'addr': '172.21.15.179:6839/93338133', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6838', 'nonce': 93338133}, {'type': 'v1', 'addr': '172.21.15.179:6839', 'nonce': 93338133}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34403': {'gid': 34403, 'name': 'k', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.132:6833/512644396', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6832', 'nonce': 512644396}, {'type': 'v1', 'addr': '172.21.15.132:6833', 'nonce': 512644396}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34424': {'gid': 34424, 'name': 'e', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 36, 'addr': '172.21.15.132:6835/1074601310', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6834', 'nonce': 1074601310}, {'type': 'v1', 'addr': '172.21.15.132:6835', 'nonce': 1074601310}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34448': {'gid': 34448, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 33, 'addr': '172.21.15.132:6837/2777087908', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6836', 'nonce': 2777087908}, {'type': 'v1', 'addr': '172.21.15.132:6837', 'nonce': 2777087908}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34484': {'gid': 34484, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.132:6839/4104367678', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 4104367678}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 4104367678}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34484, 'qdb_cluster': [24484, 34403, 34424, 34448, 34484]} 2024-05-27T07:13:23.206 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-27T07:13:23.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-27T07:13:23.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:23 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/757906285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:13:23.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:23 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/757906285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:13:23.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:23 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/757906285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:13:23.706 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:24.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:24 smithi043 ceph-mon[31531]: pgmap v545: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:24 smithi132 ceph-mon[35283]: pgmap v545: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:24.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:24 smithi179 ceph-mon[35888]: pgmap v545: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:25.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.952+0000 7f3620800700 1 -- 172.21.15.43:0/2455444368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f361810a980 msgr2=0x7f361810ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:25.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.952+0000 7f3620800700 1 --2- 172.21.15.43:0/2455444368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f361810a980 0x7f361810ad90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3608009990 tx=0x7f3608009ca0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.952+0000 7f3620800700 1 -- 172.21.15.43:0/2455444368 shutdown_connections 2024-05-27T07:13:25.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.952+0000 7f3620800700 1 --2- 172.21.15.43:0/2455444368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f361810b2d0 0x7f3618117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.952+0000 7f3620800700 1 --2- 172.21.15.43:0/2455444368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3618110960 0x7f3618110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.952+0000 7f3620800700 1 --2- 172.21.15.43:0/2455444368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f361810a980 0x7f361810ad90 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.952+0000 7f3620800700 1 -- 172.21.15.43:0/2455444368 >> 172.21.15.43:0/2455444368 conn(0x7f3618077c70 msgr2=0x7f36180780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:25.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.952+0000 7f3620800700 1 -- 172.21.15.43:0/2455444368 shutdown_connections 2024-05-27T07:13:25.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.953+0000 7f3620800700 1 -- 172.21.15.43:0/2455444368 wait complete. 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.954+0000 7f3620800700 1 Processor -- start 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.954+0000 7f3620800700 1 -- start start 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.954+0000 7f3620800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f361810a980 0x7f36181a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.954+0000 7f3620800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f361810b2d0 0x7f36181a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.954+0000 7f3620800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3618110960 0x7f36181a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.954+0000 7f3620800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f361811a980 con 0x7f3618110960 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.954+0000 7f3620800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f361811a800 con 0x7f361810a980 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.954+0000 7f3620800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f361811ab00 con 0x7f361810b2d0 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.955+0000 7f361e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3618110960 0x7f36181a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.955+0000 7f361e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3618110960 0x7f36181a5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33578/0 (socket says 172.21.15.43:33578) 2024-05-27T07:13:25.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.955+0000 7f361e800700 1 -- 172.21.15.43:0/603416712 learned_addr learned my addr 172.21.15.43:0/603416712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:25.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.955+0000 7f361d400700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f361810b2d0 0x7f36181a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.955+0000 7f361de00700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f361810a980 0x7f36181a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.956+0000 7f361e800700 1 -- 172.21.15.43:0/603416712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f361810b2d0 msgr2=0x7f36181a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:25.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.956+0000 7f361e800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f361810b2d0 0x7f36181a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.956+0000 7f361e800700 1 -- 172.21.15.43:0/603416712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f361810a980 msgr2=0x7f36181a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:25.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.956+0000 7f361e800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f361810a980 0x7f36181a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.956+0000 7f361e800700 1 -- 172.21.15.43:0/603416712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36181a61d0 con 0x7f3618110960 2024-05-27T07:13:25.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.956+0000 7f361e800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3618110960 0x7f36181a5be0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3614007d70 tx=0x7f361400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:25.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.957+0000 7f3612a00700 1 -- 172.21.15.43:0/603416712 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3614016020 con 0x7f3618110960 2024-05-27T07:13:25.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.957+0000 7f3620800700 1 -- 172.21.15.43:0/603416712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36181a64c0 con 0x7f3618110960 2024-05-27T07:13:25.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.957+0000 7f3612a00700 1 -- 172.21.15.43:0/603416712 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f361400ac90 con 0x7f3618110960 2024-05-27T07:13:25.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.957+0000 7f3612a00700 1 -- 172.21.15.43:0/603416712 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36140115e0 con 0x7f3618110960 2024-05-27T07:13:25.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.957+0000 7f3620800700 1 -- 172.21.15.43:0/603416712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36181a69b0 con 0x7f3618110960 2024-05-27T07:13:25.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.961+0000 7f3612a00700 1 -- 172.21.15.43:0/603416712 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f3614019020 con 0x7f3618110960 2024-05-27T07:13:25.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.962+0000 7f362f432b80 1 Processor -- start 2024-05-27T07:13:25.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.962+0000 7f362f432b80 1 -- start start 2024-05-27T07:13:25.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.962+0000 7f362f432b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfa65df0 0x5566cfbea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.962+0000 7f362f432b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfbeace0 0x5566cfbe4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.962+0000 7f362f432b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbe5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.962+0000 7f362f432b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5566cfb12df0 con 0x5566cfbe4d60 2024-05-27T07:13:25.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.962+0000 7f362f432b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5566cfb02990 con 0x5566cfbeace0 2024-05-27T07:13:25.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.962+0000 7f362f432b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5566cfb069d0 con 0x5566cfa65df0 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.963+0000 7f361d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbe5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.963+0000 7f361d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbe5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33588/0 (socket says 172.21.15.43:33588) 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.963+0000 7f361d400700 1 -- 172.21.15.43:0/1915631370 learned_addr learned my addr 172.21.15.43:0/1915631370 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.963+0000 7f361d400700 1 --2- 172.21.15.43:0/1915631370 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfa65df0 0x5566cfbea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.963+0000 7f361de00700 1 --2- 172.21.15.43:0/1915631370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfbeace0 0x5566cfbe4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.963+0000 7f3612a00700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f36000b2880 0x7f36000b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.964+0000 7f3612a00700 1 -- 172.21.15.43:0/603416712 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f36140e25a0 con 0x7f3618110960 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.964+0000 7f361d400700 1 -- 172.21.15.43:0/1915631370 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfa65df0 msgr2=0x5566cfbea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.964+0000 7f361d400700 1 --2- 172.21.15.43:0/1915631370 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfa65df0 0x5566cfbea7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.964+0000 7f361d400700 1 -- 172.21.15.43:0/1915631370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfbeace0 msgr2=0x5566cfbe4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.964+0000 7f361d400700 1 --2- 172.21.15.43:0/1915631370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfbeace0 0x5566cfbe4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.964+0000 7f361d400700 1 -- 172.21.15.43:0/1915631370 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566cfacb3d0 con 0x5566cfbe4d60 2024-05-27T07:13:25.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.964+0000 7f361e800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f36000b2880 0x7f36000b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.964+0000 7f361d400700 1 --2- 172.21.15.43:0/1915631370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbe5170 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3608005950 tx=0x7f36080058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d823062766496c1c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.964+0000 7f3607400700 1 -- 172.21.15.43:0/1915631370 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608021040 con 0x5566cfbe4d60 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.965+0000 7f3607400700 1 -- 172.21.15.43:0/1915631370 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f360800bc60 con 0x5566cfbe4d60 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.965+0000 7f3607400700 1 -- 172.21.15.43:0/1915631370 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360800f960 con 0x5566cfbe4d60 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.966+0000 7f362f432b80 1 -- 172.21.15.43:0/1915631370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 msgr2=0x5566cfbe5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.966+0000 7f362f432b80 1 --2- 172.21.15.43:0/1915631370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbe5170 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3608005950 tx=0x7f36080058e0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.966+0000 7f361e800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f36000b2880 0x7f36000b4d90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3614018ff0 tx=0x7f36140129d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.966+0000 7f362f432b80 1 -- 172.21.15.43:0/1915631370 shutdown_connections 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.966+0000 7f362f432b80 1 --2- 172.21.15.43:0/1915631370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbe5170 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.966+0000 7f362f432b80 1 --2- 172.21.15.43:0/1915631370 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfa65df0 0x5566cfbea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.966+0000 7f362f432b80 1 --2- 172.21.15.43:0/1915631370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfbeace0 0x5566cfbe4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.966+0000 7f362f432b80 1 -- 172.21.15.43:0/1915631370 >> 172.21.15.43:0/1915631370 conn(0x5566cfac5f70 msgr2=0x5566cfbdf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.966+0000 7f362f432b80 1 -- 172.21.15.43:0/1915631370 shutdown_connections 2024-05-27T07:13:25.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.966+0000 7f362f432b80 1 -- 172.21.15.43:0/1915631370 wait complete. 2024-05-27T07:13:25.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.967+0000 7f362f432b80 1 -- start start 2024-05-27T07:13:25.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.967+0000 7f362f432b80 1 Processor -- start 2024-05-27T07:13:25.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f362f432b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f362f432b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbeeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f362f432b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbeefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f362f432b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5566cfa08f50 con 0x5566cfbe4d60 2024-05-27T07:13:25.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f362f432b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5566cfaa3710 con 0x5566cfa65df0 2024-05-27T07:13:25.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f362f432b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5566cfbf1750 con 0x5566cfbeace0 2024-05-27T07:13:25.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f362f432b80 10 client.0 _ll_register_callbacks cb 0x5566cfaacc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:13:25.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f361de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbeeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f361de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbeeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33594/0 (socket says 172.21.15.43:33594) 2024-05-27T07:13:25.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f361de00700 1 -- 172.21.15.43:0/2725042821 learned_addr learned my addr 172.21.15.43:0/2725042821 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f361d400700 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.968+0000 7f361d400700 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbeefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f361de00700 1 -- 172.21.15.43:0/2725042821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 msgr2=0x5566cfbeefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f361de00700 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbeefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f361de00700 1 -- 172.21.15.43:0/2725042821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 msgr2=0x5566cfbee560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f361de00700 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbee560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f361de00700 1 -- 172.21.15.43:0/2725042821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566cfa52880 con 0x5566cfbe4d60 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f361d400700 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f361de00700 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbeeaa0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f360c00bfd0 tx=0x7f360c0117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2aa8864e91ae73f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f35fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f35fbe00700 1 -- 172.21.15.43:0/2725042821 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c004830 con 0x5566cfbe4d60 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f35fbe00700 1 -- 172.21.15.43:0/2725042821 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f360c0049d0 con 0x5566cfbe4d60 2024-05-27T07:13:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.969+0000 7f362f432b80 1 -- 172.21.15.43:0/2725042821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f360800bc60 con 0x5566cfbe4d60 2024-05-27T07:13:25.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f35fbe00700 1 -- 172.21.15.43:0/2725042821 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f360c025050 con 0x5566cfbe4d60 2024-05-27T07:13:25.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f362f432b80 10 client.15219 fetch_fsmap learned FSMap version 29 2024-05-27T07:13:25.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f362f432b80 1 -- 172.21.15.43:0/2725042821 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5566cfad3010 con 0x5566cfbe4d60 2024-05-27T07:13:25.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f35fbe00700 1 -- 172.21.15.43:0/2725042821 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f360800bc60 con 0x5566cfbe4d60 2024-05-27T07:13:25.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f35fbe00700 1 -- 172.21.15.43:0/2725042821 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f360c01a710 con 0x5566cfbe4d60 2024-05-27T07:13:25.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f362f432b80 10 client.15219 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:13:25.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f362f432b80 10 client.15219 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:13:25.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f362f432b80 20 client.15219 populate_metadata read hostname 'smithi043' 2024-05-27T07:13:25.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f362f432b80 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x5566cfbfe7f0 0x5566cfbffc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f362f432b80 4 client.15219 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:13:25.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.970+0000 7f362f432b80 1 -- 172.21.15.43:0/2725042821 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566cfb14d40 con 0x5566cfbfe7f0 2024-05-27T07:13:25.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.971+0000 7f361d400700 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x5566cfbfe7f0 0x5566cfbffc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.972+0000 7f361d400700 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x5566cfbfe7f0 0x5566cfbffc60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fede009fc681e5d1 server_cookie=a74dc135a65af2c4 in_seq=0 out_seq=0 2024-05-27T07:13:25.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.972+0000 7f35fbe00700 10 client.15219 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:13:25.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f35fbe00700 1 -- 172.21.15.43:0/2725042821 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5566cfb14d40 con 0x5566cfbfe7f0 2024-05-27T07:13:25.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f35fbe00700 10 client.15219 handle_command_reply: tid=0 2024-05-27T07:13:25.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 client.15219 shutdown 2024-05-27T07:13:25.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 -- 172.21.15.43:0/2725042821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 msgr2=0x5566cfbeeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:25.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbeeaa0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f360c00bfd0 tx=0x7f360c0117d0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 -- 172.21.15.43:0/2725042821 shutdown_connections 2024-05-27T07:13:25.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x5566cfbfe7f0 0x5566cfbffc60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbeefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 --2- 172.21.15.43:0/2725042821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbeeaa0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 -- 172.21.15.43:0/2725042821 >> 172.21.15.43:0/2725042821 conn(0x5566cfac5f70 msgr2=0x5566cfbdf530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:25.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f35fbe00700 0 client.15219 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:13:25.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 -- 172.21.15.43:0/2725042821 shutdown_connections 2024-05-27T07:13:25.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 1 -- 172.21.15.43:0/2725042821 wait complete. 2024-05-27T07:13:25.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:25.975+0000 7f362f432b80 20 client.15219 trim_cache size 0 max 16384 2024-05-27T07:13:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.001+0000 7f362f432b80 1 Processor -- start 2024-05-27T07:13:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.001+0000 7f362f432b80 1 -- start start 2024-05-27T07:13:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.001+0000 7f362f432b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbe9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.001+0000 7f362f432b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbee520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.001+0000 7f362f432b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbeea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.001+0000 7f362f432b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5566cfbfdfa0 con 0x5566cfbe4d60 2024-05-27T07:13:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.001+0000 7f362f432b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5566cfbfe670 con 0x5566cfa65df0 2024-05-27T07:13:25.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.001+0000 7f362f432b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5566cfbfed40 con 0x5566cfbeace0 2024-05-27T07:13:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbe9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbee520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33596/0 (socket says 172.21.15.43:33596) 2024-05-27T07:13:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361de00700 1 -- 172.21.15.43:0/908143836 learned_addr learned my addr 172.21.15.43:0/908143836 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361d400700 1 --2- 172.21.15.43:0/908143836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbeea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361de00700 1 -- 172.21.15.43:0/908143836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 msgr2=0x5566cfbeea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361de00700 1 --2- 172.21.15.43:0/908143836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbeea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361de00700 1 -- 172.21.15.43:0/908143836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 msgr2=0x5566cfbe9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:25.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361de00700 1 --2- 172.21.15.43:0/908143836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbe9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361de00700 1 -- 172.21.15.43:0/908143836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566cfbfe490 con 0x5566cfbe4d60 2024-05-27T07:13:25.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.002+0000 7f361d400700 1 --2- 172.21.15.43:0/908143836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbe9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.003+0000 7f361de00700 1 --2- 172.21.15.43:0/908143836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbee520 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f360c000c00 tx=0x7f360c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b0690ceac002636 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.003+0000 7f3606a00700 1 -- 172.21.15.43:0/908143836 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c02d070 con 0x5566cfbe4d60 2024-05-27T07:13:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.003+0000 7f3606a00700 1 -- 172.21.15.43:0/908143836 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f360c0044b0 con 0x5566cfbe4d60 2024-05-27T07:13:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.003+0000 7f3606a00700 1 -- 172.21.15.43:0/908143836 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c0052c0 con 0x5566cfbe4d60 2024-05-27T07:13:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.004+0000 7f362f432b80 1 -- 172.21.15.43:0/908143836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 msgr2=0x5566cfbee520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.004+0000 7f362f432b80 1 --2- 172.21.15.43:0/908143836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbee520 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f360c000c00 tx=0x7f360c00b480 comp rx=0 tx=0).stop 2024-05-27T07:13:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.004+0000 7f362f432b80 1 -- 172.21.15.43:0/908143836 shutdown_connections 2024-05-27T07:13:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.004+0000 7f362f432b80 1 --2- 172.21.15.43:0/908143836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbeea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.004+0000 7f362f432b80 1 --2- 172.21.15.43:0/908143836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbe9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.004+0000 7f362f432b80 1 --2- 172.21.15.43:0/908143836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbee520 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.004+0000 7f362f432b80 1 -- 172.21.15.43:0/908143836 >> 172.21.15.43:0/908143836 conn(0x5566cfac5f70 msgr2=0x5566cfbe96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:25.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.004+0000 7f362f432b80 1 -- 172.21.15.43:0/908143836 shutdown_connections 2024-05-27T07:13:25.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.004+0000 7f362f432b80 1 -- 172.21.15.43:0/908143836 wait complete. 2024-05-27T07:13:25.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.005+0000 7f362f432b80 1 -- start start 2024-05-27T07:13:25.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.005+0000 7f362f432b80 1 Processor -- start 2024-05-27T07:13:25.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.005+0000 7f362f432b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbdf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f362f432b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbdf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f362f432b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbe3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:25.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f362f432b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5566cfbffae0 con 0x5566cfbe4d60 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f362f432b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5566cfbff410 con 0x5566cfa65df0 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f362f432b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5566cfc001b0 con 0x5566cfbeace0 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f362f432b80 10 client.0 _ll_register_callbacks cb 0x5566cfaacc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f361de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbdf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f361de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbdf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33602/0 (socket says 172.21.15.43:33602) 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f361de00700 1 -- 172.21.15.43:0/1097312957 learned_addr learned my addr 172.21.15.43:0/1097312957 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f361d400700 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbdf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f361d400700 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbe3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f361de00700 1 -- 172.21.15.43:0/1097312957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 msgr2=0x5566cfbe3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f361de00700 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbe3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f361de00700 1 -- 172.21.15.43:0/1097312957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 msgr2=0x5566cfbdf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f361de00700 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbdf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.006+0000 7f361de00700 1 -- 172.21.15.43:0/1097312957 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566cfbdfd60 con 0x5566cfbe4d60 2024-05-27T07:13:25.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.007+0000 7f361de00700 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbdf720 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f360c01ab30 tx=0x7f360c01a3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c913e8294fb8703 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:25.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.007+0000 7f35fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.007+0000 7f35fbe00700 1 -- 172.21.15.43:0/1097312957 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c02d030 con 0x5566cfbe4d60 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.007+0000 7f35fbe00700 1 -- 172.21.15.43:0/1097312957 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f360c005040 con 0x5566cfbe4d60 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.007+0000 7f362f432b80 1 -- 172.21.15.43:0/1097312957 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f360c0044b0 con 0x5566cfbe4d60 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.007+0000 7f35fbe00700 1 -- 172.21.15.43:0/1097312957 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f360c042050 con 0x5566cfbe4d60 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.007+0000 7f362f432b80 10 client.15225 fetch_fsmap learned FSMap version 29 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.008+0000 7f362f432b80 1 -- 172.21.15.43:0/1097312957 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5566cfc6a4c0 con 0x5566cfbe4d60 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.008+0000 7f35fbe00700 1 -- 172.21.15.43:0/1097312957 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f360c0044b0 con 0x5566cfbe4d60 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.008+0000 7f35fbe00700 1 -- 172.21.15.43:0/1097312957 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f360c004bc0 con 0x5566cfbe4d60 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.008+0000 7f362f432b80 10 client.15225 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.008+0000 7f362f432b80 10 client.15225 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-27T07:13:26.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.008+0000 7f362f432b80 20 client.15225 populate_metadata read hostname 'smithi043' 2024-05-27T07:13:26.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.008+0000 7f362f432b80 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x5566cfc6c5b0 0x5566cfc6da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:26.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.008+0000 7f362f432b80 4 client.15225 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:13:26.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.008+0000 7f362f432b80 1 -- 172.21.15.43:0/1097312957 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "status"}) -- 0x5566cfb0e520 con 0x5566cfc6c5b0 2024-05-27T07:13:26.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.009+0000 7f361d400700 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x5566cfc6c5b0 0x5566cfc6da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:26.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.010+0000 7f361d400700 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x5566cfc6c5b0 0x5566cfc6da20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=885b2b3391782ecc server_cookie=7d404fbc1d664ab6 in_seq=0 out_seq=0 2024-05-27T07:13:26.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.010+0000 7f35fbe00700 10 client.15225 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:13:26.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.010+0000 7f35fbe00700 1 -- 172.21.15.43:0/1097312957 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5566cfb0e520 con 0x5566cfc6c5b0 2024-05-27T07:13:26.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.010+0000 7f35fbe00700 10 client.15225 handle_command_reply: tid=0 2024-05-27T07:13:26.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.010+0000 7f362f432b80 1 client.15225 shutdown 2024-05-27T07:13:26.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 1 -- 172.21.15.43:0/1097312957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 msgr2=0x5566cfbdf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:26.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbdf720 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f360c01ab30 tx=0x7f360c01a3e0 comp rx=0 tx=0).stop 2024-05-27T07:13:26.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 1 -- 172.21.15.43:0/1097312957 shutdown_connections 2024-05-27T07:13:26.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x5566cfc6c5b0 0x5566cfc6da20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:26.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566cfbeace0 0x5566cfbe3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:26.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566cfa65df0 0x5566cfbdf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:26.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 1 --2- 172.21.15.43:0/1097312957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566cfbe4d60 0x5566cfbdf720 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:26.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 1 -- 172.21.15.43:0/1097312957 >> 172.21.15.43:0/1097312957 conn(0x5566cfac5f70 msgr2=0x5566cfbf9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:26.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f35fbe00700 0 client.15225 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:13:26.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 1 -- 172.21.15.43:0/1097312957 shutdown_connections 2024-05-27T07:13:26.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 1 -- 172.21.15.43:0/1097312957 wait complete. 2024-05-27T07:13:26.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.011+0000 7f362f432b80 20 client.15225 trim_cache size 0 max 16384 2024-05-27T07:13:26.003 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:13:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 4, 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24484, 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 214.004040678, 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 29, 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 90, 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 338.25714521800001 2024-05-27T07:13:26.004 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 -- 172.21.15.43:0/603416712 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f36000b2880 msgr2=0x7f36000b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f36000b2880 0x7f36000b4d90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3614018ff0 tx=0x7f36140129d0 comp rx=0 tx=0).stop 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 -- 172.21.15.43:0/603416712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3618110960 msgr2=0x7f36181a5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3618110960 0x7f36181a5be0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3614007d70 tx=0x7f361400a340 comp rx=0 tx=0).stop 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 -- 172.21.15.43:0/603416712 shutdown_connections 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f36000b2880 0x7f36000b4d90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3618110960 0x7f36181a5be0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f361810a980 0x7f36181a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 --2- 172.21.15.43:0/603416712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f361810b2d0 0x7f36181a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 -- 172.21.15.43:0/603416712 >> 172.21.15.43:0/603416712 conn(0x7f3618077c70 msgr2=0x7f361807b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 -- 172.21.15.43:0/603416712 shutdown_connections 2024-05-27T07:13:26.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:26.012+0000 7f3620800700 1 -- 172.21.15.43:0/603416712 wait complete. 2024-05-27T07:13:26.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:26 smithi043 ceph-mon[31531]: pgmap v546: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:26.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:26 smithi132 ceph-mon[35283]: pgmap v546: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:26 smithi179 ceph-mon[35888]: pgmap v546: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:26.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.k status 2024-05-27T07:13:27.382 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:28.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:28 smithi043 ceph-mon[31531]: pgmap v547: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:28.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:28 smithi132 ceph-mon[35283]: pgmap v547: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:28 smithi179 ceph-mon[35888]: pgmap v547: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:29.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.871+0000 7f7ce9800700 1 -- 172.21.15.43:0/2151402254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4108d20 msgr2=0x7f7ce4109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:29.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.871+0000 7f7ce9800700 1 --2- 172.21.15.43:0/2151402254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4108d20 0x7f7ce4109130 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0099c0 tx=0x7f7ccc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.871+0000 7f7ce9800700 1 -- 172.21.15.43:0/2151402254 shutdown_connections 2024-05-27T07:13:29.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.871+0000 7f7ce9800700 1 --2- 172.21.15.43:0/2151402254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4109670 0x7f7ce4110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.871+0000 7f7ce9800700 1 --2- 172.21.15.43:0/2151402254 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce41043f0 0x7f7ce4104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.871+0000 7f7ce9800700 1 --2- 172.21.15.43:0/2151402254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4108d20 0x7f7ce4109130 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.871+0000 7f7ce9800700 1 -- 172.21.15.43:0/2151402254 >> 172.21.15.43:0/2151402254 conn(0x7f7ce4079d00 msgr2=0x7f7ce407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:29.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.872+0000 7f7ce9800700 1 -- 172.21.15.43:0/2151402254 shutdown_connections 2024-05-27T07:13:29.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.872+0000 7f7ce9800700 1 -- 172.21.15.43:0/2151402254 wait complete. 2024-05-27T07:13:29.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.873+0000 7f7ce9800700 1 Processor -- start 2024-05-27T07:13:29.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.873+0000 7f7ce9800700 1 -- start start 2024-05-27T07:13:29.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.873+0000 7f7ce9800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce41043f0 0x7f7ce41a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.873+0000 7f7ce9800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4108d20 0x7f7ce41a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.873+0000 7f7ce9800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4109670 0x7f7ce419aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.873+0000 7f7ce9800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce4114180 con 0x7f7ce4108d20 2024-05-27T07:13:29.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.873+0000 7f7ce9800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce4114000 con 0x7f7ce4109670 2024-05-27T07:13:29.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.873+0000 7f7ce9800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce4114300 con 0x7f7ce41043f0 2024-05-27T07:13:29.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7cdb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4108d20 0x7f7ce41a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7cdb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4108d20 0x7f7ce41a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33622/0 (socket says 172.21.15.43:33622) 2024-05-27T07:13:29.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7cdb400700 1 -- 172.21.15.43:0/2229033807 learned_addr learned my addr 172.21.15.43:0/2229033807 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:29.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce41043f0 0x7f7ce41a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4109670 0x7f7ce419aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7cdb400700 1 -- 172.21.15.43:0/2229033807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce41043f0 msgr2=0x7f7ce41a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:29.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7cdb400700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce41043f0 0x7f7ce41a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7cdb400700 1 -- 172.21.15.43:0/2229033807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4109670 msgr2=0x7f7ce419aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:29.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7cdb400700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4109670 0x7f7ce419aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7cdb400700 1 -- 172.21.15.43:0/2229033807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce419b170 con 0x7f7ce4108d20 2024-05-27T07:13:29.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce41043f0 0x7f7ce41a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:29.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.874+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4109670 0x7f7ce419aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:29.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.875+0000 7f7cdb400700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4108d20 0x7f7ce41a0de0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00b580 tx=0x7f7ccc00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:29.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.875+0000 7f7ce1600700 1 -- 172.21.15.43:0/2229033807 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc027070 con 0x7f7ce4108d20 2024-05-27T07:13:29.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.875+0000 7f7ce1600700 1 -- 172.21.15.43:0/2229033807 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7ccc0056d0 con 0x7f7ce4108d20 2024-05-27T07:13:29.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.875+0000 7f7ce9800700 1 -- 172.21.15.43:0/2229033807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce419b400 con 0x7f7ce4108d20 2024-05-27T07:13:29.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.875+0000 7f7ce1600700 1 -- 172.21.15.43:0/2229033807 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc0173f0 con 0x7f7ce4108d20 2024-05-27T07:13:29.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.875+0000 7f7ce9800700 1 -- 172.21.15.43:0/2229033807 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce419b8e0 con 0x7f7ce4108d20 2024-05-27T07:13:29.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.879+0000 7f7ce1600700 1 -- 172.21.15.43:0/2229033807 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f7ccc02a070 con 0x7f7ce4108d20 2024-05-27T07:13:29.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.880+0000 7f7cf9a34b80 1 Processor -- start 2024-05-27T07:13:29.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.880+0000 7f7cf9a34b80 1 -- start start 2024-05-27T07:13:29.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.880+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0be7edf0 0x562e0bffce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.881+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c007a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.881+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd7d0 0x562e0c009fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.881+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562e0bf2bdf0 con 0x562e0be7edf0 2024-05-27T07:13:29.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.881+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562e0bf1b990 con 0x562e0bffd7d0 2024-05-27T07:13:29.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.881+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562e0bf1f9d0 con 0x562e0bffd3c0 2024-05-27T07:13:29.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.881+0000 7f7ce8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0be7edf0 0x562e0bffce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.881+0000 7f7ce8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0be7edf0 0x562e0bffce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33628/0 (socket says 172.21.15.43:33628) 2024-05-27T07:13:29.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.881+0000 7f7ce8e00700 1 -- 172.21.15.43:0/3720431478 learned_addr learned my addr 172.21.15.43:0/3720431478 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:29.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.881+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/3720431478 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c007a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.881+0000 7f7cdb400700 1 --2- 172.21.15.43:0/3720431478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd7d0 0x562e0c009fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7ce8e00700 1 -- 172.21.15.43:0/3720431478 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 msgr2=0x562e0c007a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:29.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/3720431478 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c007a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7ce8e00700 1 -- 172.21.15.43:0/3720431478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd7d0 msgr2=0x562e0c009fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:29.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/3720431478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd7d0 0x562e0c009fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7ce8e00700 1 -- 172.21.15.43:0/3720431478 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e0bee43d0 con 0x562e0be7edf0 2024-05-27T07:13:29.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7cdb400700 1 --2- 172.21.15.43:0/3720431478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd7d0 0x562e0c009fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:29.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7ce1600700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7cc80b2880 0x7f7cc80b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7ce1600700 1 -- 172.21.15.43:0/2229033807 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f7ccc0e35a0 con 0x7f7ce4108d20 2024-05-27T07:13:29.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7cc80b2880 0x7f7cc80b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/3720431478 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0be7edf0 0x562e0bffce80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc008470 tx=0x7f7cdc008780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=339f1fb9593ea29 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:29.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.882+0000 7f7cd9600700 1 -- 172.21.15.43:0/3720431478 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc012970 con 0x562e0be7edf0 2024-05-27T07:13:29.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.883+0000 7f7cd9600700 1 -- 172.21.15.43:0/3720431478 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cdc016070 con 0x562e0be7edf0 2024-05-27T07:13:29.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.883+0000 7f7cd9600700 1 -- 172.21.15.43:0/3720431478 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc01bdf0 con 0x562e0be7edf0 2024-05-27T07:13:29.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.884+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/3720431478 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0be7edf0 msgr2=0x562e0bffce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:29.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.884+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7cc80b2880 0x7f7cc80b4d90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x562e0bffe3e0 tx=0x7f7cd400c330 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:29.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.884+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/3720431478 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0be7edf0 0x562e0bffce80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc008470 tx=0x7f7cdc008780 comp rx=0 tx=0).stop 2024-05-27T07:13:29.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.884+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/3720431478 shutdown_connections 2024-05-27T07:13:29.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.884+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/3720431478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd7d0 0x562e0c009fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.884+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/3720431478 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0be7edf0 0x562e0bffce80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.884+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/3720431478 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c007a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.884+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/3720431478 >> 172.21.15.43:0/3720431478 conn(0x562e0bedef70 msgr2=0x562e0bff8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:29.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.884+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/3720431478 shutdown_connections 2024-05-27T07:13:29.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.884+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/3720431478 wait complete. 2024-05-27T07:13:29.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.885+0000 7f7cf9a34b80 1 -- start start 2024-05-27T07:13:29.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.885+0000 7f7cf9a34b80 1 Processor -- start 2024-05-27T07:13:29.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0be7edf0 0x562e0c007570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c007ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c007ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562e0be21f50 con 0x562e0bffd7d0 2024-05-27T07:13:29.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562e0bebc710 con 0x562e0be7edf0 2024-05-27T07:13:29.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562e0c00a990 con 0x562e0bffd3c0 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7cf9a34b80 10 client.0 _ll_register_callbacks cb 0x562e0bec5c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7ce8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c007ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7ce8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c007ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33634/0 (socket says 172.21.15.43:33634) 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7ce8e00700 1 -- 172.21.15.43:0/1323520782 learned_addr learned my addr 172.21.15.43:0/1323520782 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7cdb400700 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c007ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.886+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0be7edf0 0x562e0c007570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.887+0000 7f7ce8e00700 1 -- 172.21.15.43:0/1323520782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 msgr2=0x562e0c007ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.887+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c007ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.887+0000 7f7ce8e00700 1 -- 172.21.15.43:0/1323520782 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0be7edf0 msgr2=0x562e0c007570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.887+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0be7edf0 0x562e0c007570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.887+0000 7f7ce8e00700 1 -- 172.21.15.43:0/1323520782 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e0be6b880 con 0x562e0bffd7d0 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.887+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c007ff0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc01b6a0 tx=0x7f7cdc004b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca9ba1b834d33af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.887+0000 7f7cd2000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.887+0000 7f7cd2000700 1 -- 172.21.15.43:0/1323520782 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc029060 con 0x562e0bffd7d0 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.887+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/1323520782 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cdc016070 con 0x562e0bffd7d0 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.887+0000 7f7cd2000700 1 -- 172.21.15.43:0/1323520782 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f7cdc012b80 con 0x562e0bffd7d0 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.888+0000 7f7cd2000700 1 -- 172.21.15.43:0/1323520782 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7cdc030050 con 0x562e0bffd7d0 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.888+0000 7f7cf9a34b80 10 client.15237 fetch_fsmap learned FSMap version 29 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.888+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/1323520782 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e0beec010 con 0x562e0bffd7d0 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.888+0000 7f7cd2000700 1 -- 172.21.15.43:0/1323520782 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cdc016070 con 0x562e0bffd7d0 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.888+0000 7f7cd2000700 1 -- 172.21.15.43:0/1323520782 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7cdc0233e0 con 0x562e0bffd7d0 2024-05-27T07:13:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.888+0000 7f7cf9a34b80 10 client.15237 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:13:29.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.888+0000 7f7cf9a34b80 10 client.15237 resolve_mds: resolved name 'k' to daemon mds.k 2024-05-27T07:13:29.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.888+0000 7f7cf9a34b80 20 client.15237 populate_metadata read hostname 'smithi043' 2024-05-27T07:13:29.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.888+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562e0c0174e0 0x562e0c018950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.889+0000 7f7cf9a34b80 4 client.15237 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:13:29.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.889+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/1323520782 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e0bf2dd40 con 0x562e0c0174e0 2024-05-27T07:13:29.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.889+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562e0c0174e0 0x562e0c018950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.890+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562e0c0174e0 0x562e0c018950 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a2bdf254bdebfe0c server_cookie=ac5222cfd7913e13 in_seq=0 out_seq=0 2024-05-27T07:13:29.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.890+0000 7f7cd2000700 10 client.15237 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cd2000700 1 -- 172.21.15.43:0/1323520782 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562e0bf2dd40 con 0x562e0c0174e0 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cd2000700 10 client.15237 handle_command_reply: tid=0 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cf9a34b80 1 client.15237 shutdown 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/1323520782 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 msgr2=0x562e0c007ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c007ff0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc01b6a0 tx=0x7f7cdc004b30 comp rx=0 tx=0).stop 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/1323520782 shutdown_connections 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c007ff0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562e0c0174e0 0x562e0c018950 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0be7edf0 0x562e0c007570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/1323520782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c007ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/1323520782 >> 172.21.15.43:0/1323520782 conn(0x562e0bedef70 msgr2=0x562e0bff7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cd2000700 0 client.15237 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:13:29.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.893+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/1323520782 shutdown_connections 2024-05-27T07:13:29.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.894+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/1323520782 wait complete. 2024-05-27T07:13:29.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.894+0000 7f7cf9a34b80 20 client.15237 trim_cache size 0 max 16384 2024-05-27T07:13:29.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.917+0000 7f7cf9a34b80 1 Processor -- start 2024-05-27T07:13:29.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.917+0000 7f7cf9a34b80 1 -- start start 2024-05-27T07:13:29.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.917+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0be7edf0 0x562e0bffce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.918+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c009a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.918+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c002d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.918+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562e0c016c90 con 0x562e0bffd7d0 2024-05-27T07:13:29.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.918+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562e0c017360 con 0x562e0be7edf0 2024-05-27T07:13:29.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.918+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562e0c017a30 con 0x562e0bffd3c0 2024-05-27T07:13:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.918+0000 7f7ce8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c002d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.918+0000 7f7ce8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c002d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33638/0 (socket says 172.21.15.43:33638) 2024-05-27T07:13:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.918+0000 7f7ce8e00700 1 -- 172.21.15.43:0/2870021334 learned_addr learned my addr 172.21.15.43:0/2870021334 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.918+0000 7f7cdb400700 1 --2- 172.21.15.43:0/2870021334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c009a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.918+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/2870021334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0be7edf0 0x562e0bffce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.919+0000 7f7ce8e00700 1 -- 172.21.15.43:0/2870021334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 msgr2=0x562e0c009a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.919+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/2870021334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c009a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.919+0000 7f7ce8e00700 1 -- 172.21.15.43:0/2870021334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0be7edf0 msgr2=0x562e0bffce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:29.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.919+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/2870021334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0be7edf0 0x562e0bffce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.919+0000 7f7ce8e00700 1 -- 172.21.15.43:0/2870021334 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e0bffa300 con 0x562e0bffd7d0 2024-05-27T07:13:29.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.919+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/2870021334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c002d40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0074b0 tx=0x7f7cdc008780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9490c1bb84fd56a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:29.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.919+0000 7f7cd8c00700 1 -- 172.21.15.43:0/2870021334 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc029070 con 0x562e0bffd7d0 2024-05-27T07:13:29.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.919+0000 7f7cd8c00700 1 -- 172.21.15.43:0/2870021334 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cdc016030 con 0x562e0bffd7d0 2024-05-27T07:13:29.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.919+0000 7f7cd8c00700 1 -- 172.21.15.43:0/2870021334 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc024480 con 0x562e0bffd7d0 2024-05-27T07:13:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.920+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/2870021334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 msgr2=0x562e0c002d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.920+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/2870021334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c002d40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0074b0 tx=0x7f7cdc008780 comp rx=0 tx=0).stop 2024-05-27T07:13:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.920+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/2870021334 shutdown_connections 2024-05-27T07:13:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.920+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/2870021334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c002d40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.920+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/2870021334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0be7edf0 0x562e0bffce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.920+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/2870021334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0bffd3c0 0x562e0c009a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.921+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/2870021334 >> 172.21.15.43:0/2870021334 conn(0x562e0bedef70 msgr2=0x562e0c005e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.921+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/2870021334 shutdown_connections 2024-05-27T07:13:29.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.921+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/2870021334 wait complete. 2024-05-27T07:13:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.921+0000 7f7cf9a34b80 1 -- start start 2024-05-27T07:13:29.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.921+0000 7f7cf9a34b80 1 Processor -- start 2024-05-27T07:13:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.922+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0be7edf0 0x562e0c022360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.922+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd3c0 0x562e0c01b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.922+0000 7f7cf9a34b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c01bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.922+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562e0c0187d0 con 0x562e0bffd7d0 2024-05-27T07:13:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.922+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562e0c018100 con 0x562e0bffd3c0 2024-05-27T07:13:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.922+0000 7f7cf9a34b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562e0c018ea0 con 0x562e0be7edf0 2024-05-27T07:13:29.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.922+0000 7f7cf9a34b80 10 client.0 _ll_register_callbacks cb 0x562e0bec5c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.922+0000 7f7ce8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c01bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.922+0000 7f7ce8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c01bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33644/0 (socket says 172.21.15.43:33644) 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.922+0000 7f7ce8e00700 1 -- 172.21.15.43:0/965528500 learned_addr learned my addr 172.21.15.43:0/965528500 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.923+0000 7f7cdb400700 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd3c0 0x562e0c01b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.923+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0be7edf0 0x562e0c022360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.923+0000 7f7ce8e00700 1 -- 172.21.15.43:0/965528500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0be7edf0 msgr2=0x562e0c022360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.923+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0be7edf0 0x562e0c022360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.923+0000 7f7ce8e00700 1 -- 172.21.15.43:0/965528500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd3c0 msgr2=0x562e0c01b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.923+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd3c0 0x562e0c01b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.923+0000 7f7ce8e00700 1 -- 172.21.15.43:0/965528500 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e0c01c220 con 0x562e0bffd7d0 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.923+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0be7edf0 0x562e0c022360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.923+0000 7f7ce8e00700 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c01bbe0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc023750 tx=0x7f7cdc007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2515a250ca50c61b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.923+0000 7f7cd2000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:13:29.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.924+0000 7f7cd2000700 1 -- 172.21.15.43:0/965528500 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc042060 con 0x562e0bffd7d0 2024-05-27T07:13:29.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.924+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/965528500 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cdc016030 con 0x562e0bffd7d0 2024-05-27T07:13:29.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.924+0000 7f7cd2000700 1 -- 172.21.15.43:0/965528500 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f7cdc01b680 con 0x562e0bffd7d0 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.924+0000 7f7cd2000700 1 -- 172.21.15.43:0/965528500 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7cdc049050 con 0x562e0bffd7d0 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.924+0000 7f7cf9a34b80 10 client.15243 fetch_fsmap learned FSMap version 29 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.924+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/965528500 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e0c020c10 con 0x562e0bffd7d0 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.924+0000 7f7cd2000700 1 -- 172.21.15.43:0/965528500 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cdc016030 con 0x562e0bffd7d0 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.925+0000 7f7cd2000700 1 -- 172.21.15.43:0/965528500 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7cdc03d6f0 con 0x562e0bffd7d0 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.925+0000 7f7cf9a34b80 10 client.15243 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.925+0000 7f7cf9a34b80 10 client.15243 resolve_mds: resolved name 'k' to daemon mds.k 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.925+0000 7f7cf9a34b80 20 client.15243 populate_metadata read hostname 'smithi043' 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.925+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562e0c084b90 0x562e0c086000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.925+0000 7f7cf9a34b80 4 client.15243 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.925+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/965528500 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "status"}) -- 0x562e0bf27520 con 0x562e0c084b90 2024-05-27T07:13:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.925+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562e0c084b90 0x562e0c086000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:29.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.926+0000 7f7ce3e00700 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562e0c084b90 0x562e0c086000 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fa116013ad33d46d server_cookie=28c6d5f308f5de5e in_seq=0 out_seq=0 2024-05-27T07:13:29.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.926+0000 7f7cd2000700 10 client.15243 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:13:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cd2000700 1 -- 172.21.15.43:0/965528500 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562e0bf27520 con 0x562e0c084b90 2024-05-27T07:13:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cd2000700 10 client.15243 handle_command_reply: tid=0 2024-05-27T07:13:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cf9a34b80 1 client.15243 shutdown 2024-05-27T07:13:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/965528500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 msgr2=0x562e0c01bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c01bbe0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc023750 tx=0x7f7cdc007900 comp rx=0 tx=0).stop 2024-05-27T07:13:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/965528500 shutdown_connections 2024-05-27T07:13:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e0bffd7d0 0x562e0c01bbe0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e0be7edf0 0x562e0c022360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562e0c084b90 0x562e0c086000 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cf9a34b80 1 --2- 172.21.15.43:0/965528500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e0bffd3c0 0x562e0c01b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/965528500 >> 172.21.15.43:0/965528500 conn(0x562e0bedef70 msgr2=0x562e0c007530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.927+0000 7f7cd2000700 0 client.15243 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:13:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.928+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/965528500 shutdown_connections 2024-05-27T07:13:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.928+0000 7f7cf9a34b80 1 -- 172.21.15.43:0/965528500 wait complete. 2024-05-27T07:13:29.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.928+0000 7f7cf9a34b80 20 client.15243 trim_cache size 0 max 16384 2024-05-27T07:13:29.920 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:13:29.920 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:13:29.920 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 3, 2024-05-27T07:13:29.920 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 34403, 2024-05-27T07:13:29.921 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:13:29.921 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:13:29.921 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:13:29.921 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 218.93804954999999, 2024-05-27T07:13:29.921 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 29, 2024-05-27T07:13:29.921 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-05-27T07:13:29.921 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 90, 2024-05-27T07:13:29.921 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 373.96914116300002 2024-05-27T07:13:29.921 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:13:29.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.928+0000 7f7ce9800700 1 -- 172.21.15.43:0/2229033807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7cc80b2880 msgr2=0x7f7cc80b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.928+0000 7f7ce9800700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7cc80b2880 0x7f7cc80b4d90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x562e0bffe3e0 tx=0x7f7cd400c330 comp rx=0 tx=0).stop 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.928+0000 7f7ce9800700 1 -- 172.21.15.43:0/2229033807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4108d20 msgr2=0x7f7ce41a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.928+0000 7f7ce9800700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4108d20 0x7f7ce41a0de0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00b580 tx=0x7f7ccc00f740 comp rx=0 tx=0).stop 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.929+0000 7f7ce9800700 1 -- 172.21.15.43:0/2229033807 shutdown_connections 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.929+0000 7f7ce9800700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f7cc80b2880 0x7f7cc80b4d90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.929+0000 7f7ce9800700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4109670 0x7f7ce419aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.929+0000 7f7ce9800700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce41043f0 0x7f7ce41a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.929+0000 7f7ce9800700 1 --2- 172.21.15.43:0/2229033807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4108d20 0x7f7ce41a0de0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.929+0000 7f7ce9800700 1 -- 172.21.15.43:0/2229033807 >> 172.21.15.43:0/2229033807 conn(0x7f7ce4079d00 msgr2=0x7f7ce4101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.929+0000 7f7ce9800700 1 -- 172.21.15.43:0/2229033807 shutdown_connections 2024-05-27T07:13:29.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:29.929+0000 7f7ce9800700 1 -- 172.21.15.43:0/2229033807 wait complete. 2024-05-27T07:13:30.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:30 smithi132 ceph-mon[35283]: pgmap v548: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:30.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:30 smithi043 ceph-mon[31531]: pgmap v548: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:30.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:30 smithi179 ceph-mon[35888]: pgmap v548: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:30.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.e status 2024-05-27T07:13:31.394 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:32.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:32 smithi043 ceph-mon[31531]: pgmap v549: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:32.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:32 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:32.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:32 smithi132 ceph-mon[35283]: pgmap v549: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:32.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:32 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:32.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:32 smithi179 ceph-mon[35888]: pgmap v549: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:32.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:32 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.827+0000 7f82e8e00700 1 -- 172.21.15.43:0/1640447950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e4108730 msgr2=0x7f82e4110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.828+0000 7f82e8e00700 1 --2- 172.21.15.43:0/1640447950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e4108730 0x7f82e4110090 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00b0d0 tx=0x7f82dc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.828+0000 7f82e8e00700 1 -- 172.21.15.43:0/1640447950 shutdown_connections 2024-05-27T07:13:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.828+0000 7f82e8e00700 1 --2- 172.21.15.43:0/1640447950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e4108730 0x7f82e4110090 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.828+0000 7f82e8e00700 1 --2- 172.21.15.43:0/1640447950 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e41034b0 0x7f82e41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.828+0000 7f82e8e00700 1 --2- 172.21.15.43:0/1640447950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82e4107de0 0x7f82e41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.828+0000 7f82e8e00700 1 -- 172.21.15.43:0/1640447950 >> 172.21.15.43:0/1640447950 conn(0x7f82e40fedb0 msgr2=0x7f82e41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.828+0000 7f82e8e00700 1 -- 172.21.15.43:0/1640447950 shutdown_connections 2024-05-27T07:13:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.828+0000 7f82e8e00700 1 -- 172.21.15.43:0/1640447950 wait complete. 2024-05-27T07:13:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.829+0000 7f82e8e00700 1 Processor -- start 2024-05-27T07:13:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.829+0000 7f82e8e00700 1 -- start start 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e8e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e41034b0 0x7f82e419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e4107de0 0x7f82e419d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e8e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82e4108730 0x7f82e41a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e8e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f82e4113260 con 0x7f82e4107de0 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e8e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f82e41130e0 con 0x7f82e41034b0 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e8e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f82e41133e0 con 0x7f82e4108730 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e4107de0 0x7f82e419d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e4107de0 0x7f82e419d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33652/0 (socket says 172.21.15.43:33652) 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e1400700 1 -- 172.21.15.43:0/580464459 learned_addr learned my addr 172.21.15.43:0/580464459 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e2800700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82e4108730 0x7f82e41a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.830+0000 7f82e1e00700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e41034b0 0x7f82e419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.831+0000 7f82e1400700 1 -- 172.21.15.43:0/580464459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82e4108730 msgr2=0x7f82e41a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.831+0000 7f82e1400700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82e4108730 0x7f82e41a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.831+0000 7f82e1400700 1 -- 172.21.15.43:0/580464459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e41034b0 msgr2=0x7f82e419cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.831+0000 7f82e1400700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e41034b0 0x7f82e419cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.831+0000 7f82e1400700 1 -- 172.21.15.43:0/580464459 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e41a1e00 con 0x7f82e4107de0 2024-05-27T07:13:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.831+0000 7f82e2800700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82e4108730 0x7f82e41a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:13:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.831+0000 7f82e1400700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e4107de0 0x7f82e419d4d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f82d000d950 tx=0x7f82d000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.832+0000 7f82daa00700 1 -- 172.21.15.43:0/580464459 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0016070 con 0x7f82e4107de0 2024-05-27T07:13:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.832+0000 7f82e8e00700 1 -- 172.21.15.43:0/580464459 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e41a20f0 con 0x7f82e4107de0 2024-05-27T07:13:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.832+0000 7f82daa00700 1 -- 172.21.15.43:0/580464459 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f82d0004830 con 0x7f82e4107de0 2024-05-27T07:13:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.832+0000 7f82daa00700 1 -- 172.21.15.43:0/580464459 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d00053c0 con 0x7f82e4107de0 2024-05-27T07:13:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.832+0000 7f82e8e00700 1 -- 172.21.15.43:0/580464459 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e41a9a10 con 0x7f82e4107de0 2024-05-27T07:13:33.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.837+0000 7f82daa00700 1 -- 172.21.15.43:0/580464459 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f82d00055a0 con 0x7f82e4107de0 2024-05-27T07:13:33.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.837+0000 7f82f7887b80 1 Processor -- start 2024-05-27T07:13:33.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.837+0000 7f82f7887b80 1 -- start start 2024-05-27T07:13:33.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.838+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18b9b9a0 0x563e18c2e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.838+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c2e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.838+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c3b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.838+0000 7f82f7887b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563e18b5cdf0 con 0x563e18aafdf0 2024-05-27T07:13:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.838+0000 7f82f7887b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563e18b4c990 con 0x563e18c2ee20 2024-05-27T07:13:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.838+0000 7f82f7887b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x563e18b509d0 con 0x563e18b9b9a0 2024-05-27T07:13:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.838+0000 7f82e1e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c2e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.839+0000 7f82e1e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c2e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33666/0 (socket says 172.21.15.43:33666) 2024-05-27T07:13:33.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.839+0000 7f82e1e00700 1 -- 172.21.15.43:0/284803300 learned_addr learned my addr 172.21.15.43:0/284803300 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:33.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.839+0000 7f82e2800700 1 --2- 172.21.15.43:0/284803300 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18b9b9a0 0x563e18c2e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.839+0000 7f82e1400700 1 --2- 172.21.15.43:0/284803300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c3b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.839+0000 7f82daa00700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f82c80b28c0 0x7f82c80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.839+0000 7f82e2800700 1 -- 172.21.15.43:0/284803300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 msgr2=0x563e18c3b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:33.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.839+0000 7f82e2800700 1 --2- 172.21.15.43:0/284803300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c3b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.839+0000 7f82e2800700 1 -- 172.21.15.43:0/284803300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 msgr2=0x563e18c2e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:33.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.839+0000 7f82daa00700 1 -- 172.21.15.43:0/580464459 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f82d00e2120 con 0x7f82e4107de0 2024-05-27T07:13:33.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.839+0000 7f82e2800700 1 --2- 172.21.15.43:0/284803300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c2e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.840+0000 7f82e2800700 1 -- 172.21.15.43:0/284803300 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e18b153d0 con 0x563e18b9b9a0 2024-05-27T07:13:33.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.840+0000 7f82e1e00700 1 --2- 172.21.15.43:0/284803300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c2e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:13:33.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.840+0000 7f82e2800700 1 --2- 172.21.15.43:0/284803300 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18b9b9a0 0x563e18c2e2e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009fd0 tx=0x7f82dc009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a351c71a0e9a006 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:33.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.840+0000 7f82e2800700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f82c80b28c0 0x7f82c80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.841+0000 7f82cf400700 1 -- 172.21.15.43:0/284803300 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0077c0 con 0x563e18b9b9a0 2024-05-27T07:13:33.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.841+0000 7f82cf400700 1 -- 172.21.15.43:0/284803300 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f82dc01c410 con 0x563e18b9b9a0 2024-05-27T07:13:33.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.841+0000 7f82cf400700 1 -- 172.21.15.43:0/284803300 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc024660 con 0x563e18b9b9a0 2024-05-27T07:13:33.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.842+0000 7f82f7887b80 1 -- 172.21.15.43:0/284803300 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18b9b9a0 msgr2=0x563e18c2e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:33.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.842+0000 7f82e2800700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f82c80b28c0 0x7f82c80b4dd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009610 tx=0x7f82dc01cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:33.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.842+0000 7f82f7887b80 1 --2- 172.21.15.43:0/284803300 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18b9b9a0 0x563e18c2e2e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009fd0 tx=0x7f82dc009b80 comp rx=0 tx=0).stop 2024-05-27T07:13:33.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.842+0000 7f82f7887b80 1 -- 172.21.15.43:0/284803300 shutdown_connections 2024-05-27T07:13:33.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.842+0000 7f82f7887b80 1 --2- 172.21.15.43:0/284803300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c3b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.842+0000 7f82f7887b80 1 --2- 172.21.15.43:0/284803300 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18b9b9a0 0x563e18c2e2e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.842+0000 7f82f7887b80 1 --2- 172.21.15.43:0/284803300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c2e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.842+0000 7f82f7887b80 1 -- 172.21.15.43:0/284803300 >> 172.21.15.43:0/284803300 conn(0x563e18b49340 msgr2=0x563e18c2a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:33.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.842+0000 7f82f7887b80 1 -- 172.21.15.43:0/284803300 shutdown_connections 2024-05-27T07:13:33.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.842+0000 7f82f7887b80 1 -- 172.21.15.43:0/284803300 wait complete. 2024-05-27T07:13:33.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.843+0000 7f82f7887b80 1 -- start start 2024-05-27T07:13:33.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.843+0000 7f82f7887b80 1 Processor -- start 2024-05-27T07:13:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.844+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c38530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.844+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18b9b9a0 0x563e18c38a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.844+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18c2ee20 0x563e18c38fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.844+0000 7f82f7887b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563e18a52f50 con 0x563e18aafdf0 2024-05-27T07:13:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.844+0000 7f82f7887b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563e18aed710 con 0x563e18b9b9a0 2024-05-27T07:13:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.844+0000 7f82f7887b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x563e18c3bad0 con 0x563e18c2ee20 2024-05-27T07:13:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.844+0000 7f82f7887b80 10 client.0 _ll_register_callbacks cb 0x563e18af6c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:13:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.845+0000 7f82e1e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c38530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.845+0000 7f82e1e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c38530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33674/0 (socket says 172.21.15.43:33674) 2024-05-27T07:13:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.845+0000 7f82e1e00700 1 -- 172.21.15.43:0/322693418 learned_addr learned my addr 172.21.15.43:0/322693418 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.845+0000 7f82e1400700 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18b9b9a0 0x563e18c38a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.845+0000 7f82e2800700 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18c2ee20 0x563e18c38fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.845+0000 7f82e1e00700 1 -- 172.21.15.43:0/322693418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18c2ee20 msgr2=0x563e18c38fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:33.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.845+0000 7f82e1e00700 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18c2ee20 0x563e18c38fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.845+0000 7f82e1e00700 1 -- 172.21.15.43:0/322693418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18b9b9a0 msgr2=0x563e18c38a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:33.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.845+0000 7f82e1e00700 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18b9b9a0 0x563e18c38a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.846+0000 7f82e1e00700 1 -- 172.21.15.43:0/322693418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e18a9c880 con 0x563e18aafdf0 2024-05-27T07:13:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.846+0000 7f82e2800700 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18c2ee20 0x563e18c38fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.846+0000 7f82e1e00700 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c38530 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f82d400bb70 tx=0x7f82d400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80df02bcdedf8fb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.846+0000 7f82c3e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:13:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.846+0000 7f82c3e00700 1 -- 172.21.15.43:0/322693418 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4003ca0 con 0x563e18aafdf0 2024-05-27T07:13:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.846+0000 7f82f7887b80 1 -- 172.21.15.43:0/322693418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82dc01c410 con 0x563e18aafdf0 2024-05-27T07:13:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.847+0000 7f82c3e00700 1 -- 172.21.15.43:0/322693418 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f82d401b470 con 0x563e18aafdf0 2024-05-27T07:13:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.847+0000 7f82c3e00700 1 -- 172.21.15.43:0/322693418 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82d401b6b0 con 0x563e18aafdf0 2024-05-27T07:13:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.847+0000 7f82f7887b80 10 client.15255 fetch_fsmap learned FSMap version 29 2024-05-27T07:13:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.847+0000 7f82f7887b80 1 -- 172.21.15.43:0/322693418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e18b1d010 con 0x563e18aafdf0 2024-05-27T07:13:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.847+0000 7f82c3e00700 1 -- 172.21.15.43:0/322693418 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f82dc01c410 con 0x563e18aafdf0 2024-05-27T07:13:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.847+0000 7f82c3e00700 1 -- 172.21.15.43:0/322693418 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f82d4011850 con 0x563e18aafdf0 2024-05-27T07:13:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.848+0000 7f82f7887b80 10 client.15255 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:13:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.848+0000 7f82f7887b80 10 client.15255 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:13:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.848+0000 7f82f7887b80 20 client.15255 populate_metadata read hostname 'smithi043' 2024-05-27T07:13:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.848+0000 7f82f7887b80 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x563e18c48ae0 0x563e18c49f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.848+0000 7f82f7887b80 4 client.15255 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:13:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.848+0000 7f82f7887b80 1 -- 172.21.15.43:0/322693418 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e18b5ed40 con 0x563e18c48ae0 2024-05-27T07:13:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.849+0000 7f82e1400700 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x563e18c48ae0 0x563e18c49f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.850+0000 7f82c3e00700 10 client.15255 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:13:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.850+0000 7f82e1400700 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x563e18c48ae0 0x563e18c49f50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54871e7013a379ad server_cookie=9e639b9dc16f43af in_seq=0 out_seq=0 2024-05-27T07:13:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.852+0000 7f82c3e00700 1 -- 172.21.15.43:0/322693418 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x563e18b5ed40 con 0x563e18c48ae0 2024-05-27T07:13:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.852+0000 7f82c3e00700 10 client.15255 handle_command_reply: tid=0 2024-05-27T07:13:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.852+0000 7f82f7887b80 1 client.15255 shutdown 2024-05-27T07:13:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.852+0000 7f82f7887b80 1 -- 172.21.15.43:0/322693418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 msgr2=0x563e18c38530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.852+0000 7f82f7887b80 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c38530 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f82d400bb70 tx=0x7f82d400be80 comp rx=0 tx=0).stop 2024-05-27T07:13:33.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.853+0000 7f82f7887b80 1 -- 172.21.15.43:0/322693418 shutdown_connections 2024-05-27T07:13:33.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.853+0000 7f82f7887b80 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x563e18c48ae0 0x563e18c49f50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.853+0000 7f82f7887b80 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18c2ee20 0x563e18c38fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.853+0000 7f82f7887b80 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18aafdf0 0x563e18c38530 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.853+0000 7f82f7887b80 1 --2- 172.21.15.43:0/322693418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18b9b9a0 0x563e18c38a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.853+0000 7f82f7887b80 1 -- 172.21.15.43:0/322693418 >> 172.21.15.43:0/322693418 conn(0x563e18b49340 msgr2=0x563e18c2e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:33.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.853+0000 7f82c3e00700 0 client.15255 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:13:33.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.853+0000 7f82f7887b80 1 -- 172.21.15.43:0/322693418 shutdown_connections 2024-05-27T07:13:33.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.853+0000 7f82f7887b80 1 -- 172.21.15.43:0/322693418 wait complete. 2024-05-27T07:13:33.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.853+0000 7f82f7887b80 20 client.15255 trim_cache size 0 max 16384 2024-05-27T07:13:33.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.877+0000 7f82f7887b80 1 Processor -- start 2024-05-27T07:13:33.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.877+0000 7f82f7887b80 1 -- start start 2024-05-27T07:13:33.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.877+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 0x563e18c2a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.877+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c44e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c31fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82f7887b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563e18c48290 con 0x563e18b9b9a0 2024-05-27T07:13:33.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82f7887b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563e18c48960 con 0x563e18c2ee20 2024-05-27T07:13:33.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82f7887b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x563e18c49030 con 0x563e18aafdf0 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82e1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c44e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82e1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c44e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33688/0 (socket says 172.21.15.43:33688) 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82e1400700 1 -- 172.21.15.43:0/3526913203 learned_addr learned my addr 172.21.15.43:0/3526913203 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82e1e00700 1 --2- 172.21.15.43:0/3526913203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 0x563e18c2a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82e2800700 1 --2- 172.21.15.43:0/3526913203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c31fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82e1400700 1 -- 172.21.15.43:0/3526913203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 msgr2=0x563e18c2a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82e1400700 1 --2- 172.21.15.43:0/3526913203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 0x563e18c2a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82e1400700 1 -- 172.21.15.43:0/3526913203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 msgr2=0x563e18c31fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82e1400700 1 --2- 172.21.15.43:0/3526913203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c31fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.878+0000 7f82e1400700 1 -- 172.21.15.43:0/3526913203 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e18c464b0 con 0x563e18b9b9a0 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.879+0000 7f82e1e00700 1 --2- 172.21.15.43:0/3526913203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 0x563e18c2a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:33.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.879+0000 7f82e1400700 1 --2- 172.21.15.43:0/3526913203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c44e00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f82d00a5430 tx=0x7f82d0091040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aaaf287af861052 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.879+0000 7f82cea00700 1 -- 172.21.15.43:0/3526913203 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d00a8d00 con 0x563e18b9b9a0 2024-05-27T07:13:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.879+0000 7f82cea00700 1 -- 172.21.15.43:0/3526913203 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f82d00a8ec0 con 0x563e18b9b9a0 2024-05-27T07:13:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.879+0000 7f82cea00700 1 -- 172.21.15.43:0/3526913203 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d00a8b20 con 0x563e18b9b9a0 2024-05-27T07:13:33.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.880+0000 7f82f7887b80 1 -- 172.21.15.43:0/3526913203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 msgr2=0x563e18c44e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.880+0000 7f82f7887b80 1 --2- 172.21.15.43:0/3526913203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c44e00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f82d00a5430 tx=0x7f82d0091040 comp rx=0 tx=0).stop 2024-05-27T07:13:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.880+0000 7f82f7887b80 1 -- 172.21.15.43:0/3526913203 shutdown_connections 2024-05-27T07:13:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.880+0000 7f82f7887b80 1 --2- 172.21.15.43:0/3526913203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c31fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.880+0000 7f82f7887b80 1 --2- 172.21.15.43:0/3526913203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 0x563e18c2a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.880+0000 7f82f7887b80 1 --2- 172.21.15.43:0/3526913203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c44e00 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.880+0000 7f82f7887b80 1 -- 172.21.15.43:0/3526913203 >> 172.21.15.43:0/3526913203 conn(0x563e18b49340 msgr2=0x563e18c29f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.881+0000 7f82f7887b80 1 -- 172.21.15.43:0/3526913203 shutdown_connections 2024-05-27T07:13:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.881+0000 7f82f7887b80 1 -- 172.21.15.43:0/3526913203 wait complete. 2024-05-27T07:13:33.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.881+0000 7f82f7887b80 1 -- start start 2024-05-27T07:13:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.881+0000 7f82f7887b80 1 Processor -- start 2024-05-27T07:13:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.882+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 0x563e18c32e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.882+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c333d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.882+0000 7f82f7887b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c33910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.882+0000 7f82f7887b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563e18c49dd0 con 0x563e18b9b9a0 2024-05-27T07:13:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.882+0000 7f82f7887b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563e18c49700 con 0x563e18c2ee20 2024-05-27T07:13:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.882+0000 7f82f7887b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x563e18c4a4a0 con 0x563e18aafdf0 2024-05-27T07:13:33.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.882+0000 7f82f7887b80 10 client.0 _ll_register_callbacks cb 0x563e18af6c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:13:33.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.882+0000 7f82e1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82e1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c333d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33702/0 (socket says 172.21.15.43:33702) 2024-05-27T07:13:33.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82e1400700 1 -- 172.21.15.43:0/3806740761 learned_addr learned my addr 172.21.15.43:0/3806740761 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:33.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82e2800700 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c33910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82e1e00700 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 0x563e18c32e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82e1400700 1 -- 172.21.15.43:0/3806740761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 msgr2=0x563e18c32e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82e1400700 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 0x563e18c32e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82e1400700 1 -- 172.21.15.43:0/3806740761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 msgr2=0x563e18c33910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82e1400700 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c33910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82e1400700 1 -- 172.21.15.43:0/3806740761 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e18c39320 con 0x563e18b9b9a0 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82e1400700 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c333d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f82d00dfc00 tx=0x7f82d00a8bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195882cc9e8511e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.883+0000 7f82c3e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.884+0000 7f82c3e00700 1 -- 172.21.15.43:0/3806740761 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d00a9ab0 con 0x563e18b9b9a0 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.884+0000 7f82c3e00700 1 -- 172.21.15.43:0/3806740761 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f82d00b1f90 con 0x563e18b9b9a0 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.884+0000 7f82f7887b80 1 -- 172.21.15.43:0/3806740761 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d00a8ec0 con 0x563e18b9b9a0 2024-05-27T07:13:33.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.884+0000 7f82c3e00700 1 -- 172.21.15.43:0/3806740761 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82d00b2210 con 0x563e18b9b9a0 2024-05-27T07:13:33.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.884+0000 7f82f7887b80 10 client.15261 fetch_fsmap learned FSMap version 29 2024-05-27T07:13:33.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.884+0000 7f82f7887b80 1 -- 172.21.15.43:0/3806740761 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e18cb4480 con 0x563e18b9b9a0 2024-05-27T07:13:33.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.884+0000 7f82c3e00700 1 -- 172.21.15.43:0/3806740761 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f82d00a8ec0 con 0x563e18b9b9a0 2024-05-27T07:13:33.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.885+0000 7f82c3e00700 1 -- 172.21.15.43:0/3806740761 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f82d00e40d0 con 0x563e18b9b9a0 2024-05-27T07:13:33.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.885+0000 7f82f7887b80 10 client.15261 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:13:33.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.885+0000 7f82f7887b80 10 client.15261 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-27T07:13:33.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.885+0000 7f82f7887b80 20 client.15261 populate_metadata read hostname 'smithi043' 2024-05-27T07:13:33.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.885+0000 7f82f7887b80 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x563e18cb64c0 0x563e18cb7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:33.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.885+0000 7f82f7887b80 4 client.15261 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:13:33.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.885+0000 7f82f7887b80 1 -- 172.21.15.43:0/3806740761 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "status"}) -- 0x563e18b58520 con 0x563e18cb64c0 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.886+0000 7f82e1e00700 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x563e18cb64c0 0x563e18cb7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.886+0000 7f82e1e00700 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x563e18cb64c0 0x563e18cb7930 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=279f2d15e815f64d server_cookie=f71feae2c46fde3e in_seq=0 out_seq=0 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.887+0000 7f82c3e00700 10 client.15261 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.887+0000 7f82c3e00700 1 -- 172.21.15.43:0/3806740761 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563e18b58520 con 0x563e18cb64c0 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.887+0000 7f82c3e00700 10 client.15261 handle_command_reply: tid=0 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.887+0000 7f82f7887b80 1 client.15261 shutdown 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.887+0000 7f82f7887b80 1 -- 172.21.15.43:0/3806740761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 msgr2=0x563e18c333d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.887+0000 7f82f7887b80 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c333d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f82d00dfc00 tx=0x7f82d00a8bb0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.887+0000 7f82f7887b80 1 -- 172.21.15.43:0/3806740761 shutdown_connections 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.887+0000 7f82f7887b80 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x563e18cb64c0 0x563e18cb7930 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.887+0000 7f82f7887b80 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563e18c2ee20 0x563e18c33910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.888+0000 7f82f7887b80 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563e18aafdf0 0x563e18c32e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.888+0000 7f82f7887b80 1 --2- 172.21.15.43:0/3806740761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e18b9b9a0 0x563e18c333d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.888+0000 7f82f7887b80 1 -- 172.21.15.43:0/3806740761 >> 172.21.15.43:0/3806740761 conn(0x563e18b49340 msgr2=0x563e18c36a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:33.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.888+0000 7f82c3e00700 0 client.15261 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:13:33.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.888+0000 7f82f7887b80 1 -- 172.21.15.43:0/3806740761 shutdown_connections 2024-05-27T07:13:33.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.888+0000 7f82f7887b80 1 -- 172.21.15.43:0/3806740761 wait complete. 2024-05-27T07:13:33.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.888+0000 7f82f7887b80 20 client.15261 trim_cache size 0 max 16384 2024-05-27T07:13:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:13:33.881 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 2, 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 34424, 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 223.924017741, 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 29, 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 90, 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 363.52078566099999 2024-05-27T07:13:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:13:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 -- 172.21.15.43:0/580464459 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f82c80b28c0 msgr2=0x7f82c80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f82c80b28c0 0x7f82c80b4dd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009610 tx=0x7f82dc01cf90 comp rx=0 tx=0).stop 2024-05-27T07:13:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 -- 172.21.15.43:0/580464459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e4107de0 msgr2=0x7f82e419d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e4107de0 0x7f82e419d4d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f82d000d950 tx=0x7f82d000dc60 comp rx=0 tx=0).stop 2024-05-27T07:13:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 -- 172.21.15.43:0/580464459 shutdown_connections 2024-05-27T07:13:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f82c80b28c0 0x7f82c80b4dd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82e4108730 0x7f82e41a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e41034b0 0x7f82e419cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 --2- 172.21.15.43:0/580464459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e4107de0 0x7f82e419d4d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:33.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 -- 172.21.15.43:0/580464459 >> 172.21.15.43:0/580464459 conn(0x7f82e40fedb0 msgr2=0x7f82e406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:33.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 -- 172.21.15.43:0/580464459 shutdown_connections 2024-05-27T07:13:33.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:33.889+0000 7f82e8e00700 1 -- 172.21.15.43:0/580464459 wait complete. 2024-05-27T07:13:34.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:34 smithi043 ceph-mon[31531]: pgmap v550: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:34.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:34.156 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:34.156 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:34 smithi043 ceph-mon[31531]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:13:34.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:34 smithi132 ceph-mon[35283]: pgmap v550: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:34.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:34.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:34.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:34 smithi132 ceph-mon[35283]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:13:34.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:34 smithi179 ceph-mon[35888]: pgmap v550: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:34.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:13:34.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:13:34.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:34 smithi179 ceph-mon[35888]: from='mgr.14152 172.21.15.43:0/3659261064' entity='mgr.x' 2024-05-27T07:13:34.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-05-27T07:13:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:36 smithi132 ceph-mon[35283]: pgmap v551: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:36.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:36 smithi043 ceph-mon[31531]: pgmap v551: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:36 smithi179 ceph-mon[35888]: pgmap v551: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:37.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.602+0000 7efc9d400700 1 -- 172.21.15.43:0/533728127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc98109860 msgr2=0x7efc981111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:37.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.602+0000 7efc9d400700 1 --2- 172.21.15.43:0/533728127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc98109860 0x7efc981111c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efc9000b0d0 tx=0x7efc9000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.602+0000 7efc9d400700 1 -- 172.21.15.43:0/533728127 shutdown_connections 2024-05-27T07:13:37.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.602+0000 7efc9d400700 1 --2- 172.21.15.43:0/533728127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc98109860 0x7efc981111c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.602+0000 7efc9d400700 1 --2- 172.21.15.43:0/533728127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc981045e0 0x7efc981049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.602+0000 7efc9d400700 1 --2- 172.21.15.43:0/533728127 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc98108f10 0x7efc98109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.602+0000 7efc9d400700 1 -- 172.21.15.43:0/533728127 >> 172.21.15.43:0/533728127 conn(0x7efc98100030 msgr2=0x7efc98102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:37.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.602+0000 7efc9d400700 1 -- 172.21.15.43:0/533728127 shutdown_connections 2024-05-27T07:13:37.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.603+0000 7efc9d400700 1 -- 172.21.15.43:0/533728127 wait complete. 2024-05-27T07:13:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.604+0000 7efc9d400700 1 Processor -- start 2024-05-27T07:13:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.604+0000 7efc9d400700 1 -- start start 2024-05-27T07:13:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc9d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc981045e0 0x7efc98075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc9d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc98108f10 0x7efc98075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc9d400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc98109860 0x7efc98076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc9d400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efc98114340 con 0x7efc98108f10 2024-05-27T07:13:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc9d400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efc981141c0 con 0x7efc981045e0 2024-05-27T07:13:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc9d400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efc981144c0 con 0x7efc98109860 2024-05-27T07:13:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc8de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc98108f10 0x7efc98075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc8de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc98108f10 0x7efc98075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49526/0 (socket says 172.21.15.43:49526) 2024-05-27T07:13:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc8de00700 1 -- 172.21.15.43:0/909006273 learned_addr learned my addr 172.21.15.43:0/909006273 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc97200700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc98109860 0x7efc98076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.605+0000 7efc96800700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc981045e0 0x7efc98075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.606+0000 7efc8de00700 1 -- 172.21.15.43:0/909006273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc98109860 msgr2=0x7efc98076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.606+0000 7efc8de00700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc98109860 0x7efc98076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.606+0000 7efc8de00700 1 -- 172.21.15.43:0/909006273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc981045e0 msgr2=0x7efc98075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.606+0000 7efc8de00700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc981045e0 0x7efc98075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.606+0000 7efc8de00700 1 -- 172.21.15.43:0/909006273 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc981a9020 con 0x7efc98108f10 2024-05-27T07:13:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.606+0000 7efc96800700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc981045e0 0x7efc98075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.606+0000 7efc8de00700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc98108f10 0x7efc98075ff0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efc8000d980 tx=0x7efc8000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.606+0000 7efc8fe00700 1 -- 172.21.15.43:0/909006273 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80017070 con 0x7efc98108f10 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.606+0000 7efc8fe00700 1 -- 172.21.15.43:0/909006273 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc80009dc0 con 0x7efc98108f10 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.606+0000 7efc9d400700 1 -- 172.21.15.43:0/909006273 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc981a9310 con 0x7efc98108f10 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.607+0000 7efc8fe00700 1 -- 172.21.15.43:0/909006273 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8000f3c0 con 0x7efc98108f10 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.607+0000 7efc9d400700 1 -- 172.21.15.43:0/909006273 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc981a9850 con 0x7efc98108f10 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.611+0000 7efc8fe00700 1 -- 172.21.15.43:0/909006273 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7efc80009930 con 0x7efc98108f10 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.612+0000 7efcabf68b80 1 Processor -- start 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.612+0000 7efcabf68b80 1 -- start start 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.612+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x56353460b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.612+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x56353460b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.612+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 0x5635346180c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.612+0000 7efcabf68b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563534539df0 con 0x56353448cdf0 2024-05-27T07:13:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.612+0000 7efcabf68b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563534529990 con 0x56353460be20 2024-05-27T07:13:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.612+0000 7efcabf68b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56353452d9d0 con 0x5635345789a0 2024-05-27T07:13:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.612+0000 7efc8fe00700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efc7c0b2880 0x7efc7c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.613+0000 7efc96800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x56353460b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.613+0000 7efc97200700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efc7c0b2880 0x7efc7c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.613+0000 7efc8fe00700 1 -- 172.21.15.43:0/909006273 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7efc800e2050 con 0x7efc98108f10 2024-05-27T07:13:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.613+0000 7efc96800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x56353460b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49540/0 (socket says 172.21.15.43:49540) 2024-05-27T07:13:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.613+0000 7efc96800700 1 -- 172.21.15.43:0/1356495771 learned_addr learned my addr 172.21.15.43:0/1356495771 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.613+0000 7efc97200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x56353460b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.613+0000 7efc8de00700 1 --2- 172.21.15.43:0/1356495771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 0x5635346180c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc96800700 1 -- 172.21.15.43:0/1356495771 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 msgr2=0x56353460b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc96800700 1 --2- 172.21.15.43:0/1356495771 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x56353460b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc96800700 1 -- 172.21.15.43:0/1356495771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 msgr2=0x5635346180c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc96800700 1 --2- 172.21.15.43:0/1356495771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 0x5635346180c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc96800700 1 -- 172.21.15.43:0/1356495771 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635344f23d0 con 0x56353448cdf0 2024-05-27T07:13:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc8de00700 1 --2- 172.21.15.43:0/1356495771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 0x5635346180c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc97200700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efc7c0b2880 0x7efc7c0b4d90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efc90009e60 tx=0x7efc900090d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc96800700 1 --2- 172.21.15.43:0/1356495771 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x56353460b820 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efc88008d30 tx=0x7efc8800e630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2daa016231da3e77 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc87e00700 1 -- 172.21.15.43:0/1356495771 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc88014070 con 0x56353448cdf0 2024-05-27T07:13:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc87e00700 1 -- 172.21.15.43:0/1356495771 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc88004cc0 con 0x56353448cdf0 2024-05-27T07:13:37.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.614+0000 7efc87e00700 1 -- 172.21.15.43:0/1356495771 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc880056b0 con 0x56353448cdf0 2024-05-27T07:13:37.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.615+0000 7efcabf68b80 1 -- 172.21.15.43:0/1356495771 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 msgr2=0x56353460b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:37.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.615+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1356495771 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x56353460b820 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efc88008d30 tx=0x7efc8800e630 comp rx=0 tx=0).stop 2024-05-27T07:13:37.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.616+0000 7efcabf68b80 1 -- 172.21.15.43:0/1356495771 shutdown_connections 2024-05-27T07:13:37.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.616+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1356495771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 0x5635346180c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.616+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1356495771 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x56353460b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.616+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1356495771 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x56353460b820 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.616+0000 7efcabf68b80 1 -- 172.21.15.43:0/1356495771 >> 172.21.15.43:0/1356495771 conn(0x563534526340 msgr2=0x563534607470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:37.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.616+0000 7efcabf68b80 1 -- 172.21.15.43:0/1356495771 shutdown_connections 2024-05-27T07:13:37.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.616+0000 7efcabf68b80 1 -- 172.21.15.43:0/1356495771 wait complete. 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.616+0000 7efcabf68b80 1 -- start start 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.617+0000 7efcabf68b80 1 Processor -- start 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.617+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 0x563534615630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.617+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5635345789a0 0x563534615b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.617+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56353460be20 0x5635346160b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.617+0000 7efcabf68b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56353442ff50 con 0x5635345789a0 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.617+0000 7efcabf68b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5635344ca710 con 0x56353448cdf0 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.617+0000 7efcabf68b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x563534618b30 con 0x56353460be20 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.617+0000 7efcabf68b80 10 client.0 _ll_register_callbacks cb 0x5635344d3c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc8de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5635345789a0 0x563534615b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc8de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5635345789a0 0x563534615b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49556/0 (socket says 172.21.15.43:49556) 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc8de00700 1 -- 172.21.15.43:0/1698651981 learned_addr learned my addr 172.21.15.43:0/1698651981 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc96800700 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 0x563534615630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc97200700 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56353460be20 0x5635346160b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc8de00700 1 -- 172.21.15.43:0/1698651981 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56353460be20 msgr2=0x5635346160b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc8de00700 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56353460be20 0x5635346160b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc8de00700 1 -- 172.21.15.43:0/1698651981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 msgr2=0x563534615630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc8de00700 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 0x563534615630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc8de00700 1 -- 172.21.15.43:0/1698651981 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563534479880 con 0x5635345789a0 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc97200700 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56353460be20 0x5635346160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc8de00700 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5635345789a0 0x563534615b70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efc800a6550 tx=0x7efc800108b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b51036a5a2a94341 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.618+0000 7efc84c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.619+0000 7efc84c00700 1 -- 172.21.15.43:0/1698651981 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80020040 con 0x5635345789a0 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.619+0000 7efc84c00700 1 -- 172.21.15.43:0/1698651981 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7efc800a58d0 con 0x5635345789a0 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.619+0000 7efcabf68b80 1 -- 172.21.15.43:0/1698651981 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc88004cc0 con 0x5635345789a0 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.619+0000 7efc84c00700 1 -- 172.21.15.43:0/1698651981 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efc800a6950 con 0x5635345789a0 2024-05-27T07:13:37.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.619+0000 7efcabf68b80 10 client.15273 fetch_fsmap learned FSMap version 29 2024-05-27T07:13:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.619+0000 7efcabf68b80 1 -- 172.21.15.43:0/1698651981 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5635344fa010 con 0x5635345789a0 2024-05-27T07:13:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.619+0000 7efc84c00700 1 -- 172.21.15.43:0/1698651981 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc88004cc0 con 0x5635345789a0 2024-05-27T07:13:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.619+0000 7efc84c00700 1 -- 172.21.15.43:0/1698651981 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efc800a4c50 con 0x5635345789a0 2024-05-27T07:13:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.620+0000 7efcabf68b80 10 client.15273 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:13:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.620+0000 7efcabf68b80 10 client.15273 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:13:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.620+0000 7efcabf68b80 20 client.15273 populate_metadata read hostname 'smithi043' 2024-05-27T07:13:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.620+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x563534625a50 0x563534626ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.620+0000 7efcabf68b80 4 client.15273 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:13:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.620+0000 7efcabf68b80 1 -- 172.21.15.43:0/1698651981 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56353453bd40 con 0x563534625a50 2024-05-27T07:13:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.621+0000 7efc96800700 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x563534625a50 0x563534626ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.621+0000 7efc96800700 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x563534625a50 0x563534626ec0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ae4fb8a82e8c188d server_cookie=7b92b3fa4a82fce9 in_seq=0 out_seq=0 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.621+0000 7efc84c00700 10 client.15273 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efc84c00700 1 -- 172.21.15.43:0/1698651981 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56353453bd40 con 0x563534625a50 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efc84c00700 10 client.15273 handle_command_reply: tid=0 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efcabf68b80 1 client.15273 shutdown 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efcabf68b80 1 -- 172.21.15.43:0/1698651981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5635345789a0 msgr2=0x563534615b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5635345789a0 0x563534615b70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efc800a6550 tx=0x7efc800108b0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efcabf68b80 1 -- 172.21.15.43:0/1698651981 shutdown_connections 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x563534625a50 0x563534626ec0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56353460be20 0x5635346160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 0x563534615630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1698651981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5635345789a0 0x563534615b70 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efcabf68b80 1 -- 172.21.15.43:0/1698651981 >> 172.21.15.43:0/1698651981 conn(0x563534526340 msgr2=0x56353460b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efc84c00700 0 client.15273 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.624+0000 7efcabf68b80 1 -- 172.21.15.43:0/1698651981 shutdown_connections 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.625+0000 7efcabf68b80 1 -- 172.21.15.43:0/1698651981 wait complete. 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.625+0000 7efcabf68b80 20 client.15273 trim_cache size 0 max 16384 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.648+0000 7efcabf68b80 1 Processor -- start 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.648+0000 7efcabf68b80 1 -- start start 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.649+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x563534621dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.649+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x563534606f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.649+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 0x563534607440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.649+0000 7efcabf68b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563534625200 con 0x56353448cdf0 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.649+0000 7efcabf68b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5635346258d0 con 0x56353460be20 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.649+0000 7efcabf68b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x563534625fa0 con 0x5635345789a0 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.649+0000 7efc96800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x563534621dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.649+0000 7efc96800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x563534621dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49560/0 (socket says 172.21.15.43:49560) 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.649+0000 7efc96800700 1 -- 172.21.15.43:0/1862092805 learned_addr learned my addr 172.21.15.43:0/1862092805 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.650+0000 7efc8de00700 1 --2- 172.21.15.43:0/1862092805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x563534606f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.650+0000 7efc97200700 1 --2- 172.21.15.43:0/1862092805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 0x563534607440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.650+0000 7efc96800700 1 -- 172.21.15.43:0/1862092805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 msgr2=0x563534606f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.650+0000 7efc96800700 1 --2- 172.21.15.43:0/1862092805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x563534606f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.650+0000 7efc96800700 1 -- 172.21.15.43:0/1862092805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 msgr2=0x563534607440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.650+0000 7efc96800700 1 --2- 172.21.15.43:0/1862092805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 0x563534607440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.650+0000 7efc96800700 1 -- 172.21.15.43:0/1862092805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563534616420 con 0x56353448cdf0 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.650+0000 7efc96800700 1 --2- 172.21.15.43:0/1862092805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x563534621dd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efc8800e980 tx=0x7efc88008f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbbd846bff8efd1b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.651+0000 7efc87400700 1 -- 172.21.15.43:0/1862092805 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc88014040 con 0x56353448cdf0 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.651+0000 7efc87400700 1 -- 172.21.15.43:0/1862092805 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc88004ba0 con 0x56353448cdf0 2024-05-27T07:13:37.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.651+0000 7efc87400700 1 -- 172.21.15.43:0/1862092805 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc88010430 con 0x56353448cdf0 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.651+0000 7efcabf68b80 1 -- 172.21.15.43:0/1862092805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 msgr2=0x563534621dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.651+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1862092805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x563534621dd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efc8800e980 tx=0x7efc88008f00 comp rx=0 tx=0).stop 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.652+0000 7efcabf68b80 1 -- 172.21.15.43:0/1862092805 shutdown_connections 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.652+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1862092805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353460be20 0x563534607440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.652+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1862092805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353448cdf0 0x563534621dd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.652+0000 7efcabf68b80 1 --2- 172.21.15.43:0/1862092805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x563534606f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.652+0000 7efcabf68b80 1 -- 172.21.15.43:0/1862092805 >> 172.21.15.43:0/1862092805 conn(0x563534526340 msgr2=0x563534606490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.652+0000 7efcabf68b80 1 -- 172.21.15.43:0/1862092805 shutdown_connections 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.652+0000 7efcabf68b80 1 -- 172.21.15.43:0/1862092805 wait complete. 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.652+0000 7efcabf68b80 1 -- start start 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.652+0000 7efcabf68b80 1 Processor -- start 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.653+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 0x563534630750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.653+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x56353460f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.653+0000 7efcabf68b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353460be20 0x56353460fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.653+0000 7efcabf68b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563534626d40 con 0x56353460be20 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.653+0000 7efcabf68b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563534626670 con 0x56353448cdf0 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.653+0000 7efcabf68b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x563534627410 con 0x5635345789a0 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.653+0000 7efcabf68b80 10 client.0 _ll_register_callbacks cb 0x5635344d3c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.653+0000 7efc97200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353460be20 0x56353460fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.653+0000 7efc97200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353460be20 0x56353460fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49562/0 (socket says 172.21.15.43:49562) 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.653+0000 7efc97200700 1 -- 172.21.15.43:0/142132928 learned_addr learned my addr 172.21.15.43:0/142132928 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.654+0000 7efc96800700 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 0x563534630750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.654+0000 7efc8de00700 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x56353460f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.654+0000 7efc97200700 1 -- 172.21.15.43:0/142132928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 msgr2=0x56353460f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.654+0000 7efc97200700 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x56353460f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.654+0000 7efc97200700 1 -- 172.21.15.43:0/142132928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 msgr2=0x563534630750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.654+0000 7efc97200700 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 0x563534630750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.654+0000 7efc97200700 1 -- 172.21.15.43:0/142132928 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563534610270 con 0x56353460be20 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.654+0000 7efc96800700 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 0x563534630750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.654+0000 7efc97200700 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353460be20 0x56353460fd30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efc9000b820 tx=0x7efc9000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7c85a25e648e36 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.654+0000 7efc84c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.655+0000 7efc84c00700 1 -- 172.21.15.43:0/142132928 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9001ec20 con 0x56353460be20 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.655+0000 7efc84c00700 1 -- 172.21.15.43:0/142132928 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7efc9001edc0 con 0x56353460be20 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.655+0000 7efcabf68b80 1 -- 172.21.15.43:0/142132928 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc88004ba0 con 0x56353460be20 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.655+0000 7efc84c00700 1 -- 172.21.15.43:0/142132928 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efc90037410 con 0x56353460be20 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.655+0000 7efcabf68b80 10 client.15279 fetch_fsmap learned FSMap version 29 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.655+0000 7efcabf68b80 1 -- 172.21.15.43:0/142132928 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56353462cf70 con 0x56353460be20 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.655+0000 7efc84c00700 1 -- 172.21.15.43:0/142132928 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc88004ba0 con 0x56353460be20 2024-05-27T07:13:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.656+0000 7efc84c00700 1 -- 172.21.15.43:0/142132928 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efc90011070 con 0x56353460be20 2024-05-27T07:13:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.656+0000 7efcabf68b80 10 client.15279 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:13:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.656+0000 7efcabf68b80 10 client.15279 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-27T07:13:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.656+0000 7efcabf68b80 20 client.15279 populate_metadata read hostname 'smithi043' 2024-05-27T07:13:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.656+0000 7efcabf68b80 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5635346932d0 0x563534694740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.656+0000 7efcabf68b80 4 client.15279 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:13:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.656+0000 7efcabf68b80 1 -- 172.21.15.43:0/142132928 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "status"}) -- 0x563534535520 con 0x5635346932d0 2024-05-27T07:13:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.656+0000 7efc96800700 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5635346932d0 0x563534694740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.657+0000 7efc96800700 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5635346932d0 0x563534694740 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb71a4c6f62ed7b2 server_cookie=f782e7080813b055 in_seq=0 out_seq=0 2024-05-27T07:13:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.657+0000 7efc84c00700 10 client.15279 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:13:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efc84c00700 1 -- 172.21.15.43:0/142132928 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x563534535520 con 0x5635346932d0 2024-05-27T07:13:37.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efc84c00700 10 client.15279 handle_command_reply: tid=0 2024-05-27T07:13:37.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efcabf68b80 1 client.15279 shutdown 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efcabf68b80 1 -- 172.21.15.43:0/142132928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353460be20 msgr2=0x56353460fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efcabf68b80 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353460be20 0x56353460fd30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efc9000b820 tx=0x7efc9000b040 comp rx=0 tx=0).stop 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efcabf68b80 1 -- 172.21.15.43:0/142132928 shutdown_connections 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efcabf68b80 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5635346932d0 0x563534694740 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efcabf68b80 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56353460be20 0x56353460fd30 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efcabf68b80 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56353448cdf0 0x563534630750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efcabf68b80 1 --2- 172.21.15.43:0/142132928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5635345789a0 0x56353460f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efc84c00700 0 client.15279 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.658+0000 7efcabf68b80 1 -- 172.21.15.43:0/142132928 >> 172.21.15.43:0/142132928 conn(0x563534526340 msgr2=0x56353460b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.659+0000 7efcabf68b80 1 -- 172.21.15.43:0/142132928 shutdown_connections 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.659+0000 7efcabf68b80 1 -- 172.21.15.43:0/142132928 wait complete. 2024-05-27T07:13:37.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.659+0000 7efcabf68b80 20 client.15279 trim_cache size 0 max 16384 2024-05-27T07:13:37.783 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:13:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:13:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 1, 2024-05-27T07:13:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 34448, 2024-05-27T07:13:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:13:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:13:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:13:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 228.708975068, 2024-05-27T07:13:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 29, 2024-05-27T07:13:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-05-27T07:13:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 90, 2024-05-27T07:13:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 354.254891413 2024-05-27T07:13:37.784 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:13:37.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.659+0000 7efc9d400700 1 -- 172.21.15.43:0/909006273 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efc7c0b2880 msgr2=0x7efc7c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:37.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.659+0000 7efc9d400700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efc7c0b2880 0x7efc7c0b4d90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efc90009e60 tx=0x7efc900090d0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.659+0000 7efc9d400700 1 -- 172.21.15.43:0/909006273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc98108f10 msgr2=0x7efc98075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:37.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.660+0000 7efc9d400700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc98108f10 0x7efc98075ff0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efc8000d980 tx=0x7efc8000dc90 comp rx=0 tx=0).stop 2024-05-27T07:13:37.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.660+0000 7efc9d400700 1 -- 172.21.15.43:0/909006273 shutdown_connections 2024-05-27T07:13:37.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.660+0000 7efc9d400700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7efc7c0b2880 0x7efc7c0b4d90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.660+0000 7efc9d400700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc98109860 0x7efc98076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.660+0000 7efc9d400700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc981045e0 0x7efc98075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.660+0000 7efc9d400700 1 --2- 172.21.15.43:0/909006273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc98108f10 0x7efc98075ff0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:37.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.660+0000 7efc9d400700 1 -- 172.21.15.43:0/909006273 >> 172.21.15.43:0/909006273 conn(0x7efc98100030 msgr2=0x7efc9810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:37.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.660+0000 7efc9d400700 1 -- 172.21.15.43:0/909006273 shutdown_connections 2024-05-27T07:13:37.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:37.660+0000 7efc9d400700 1 -- 172.21.15.43:0/909006273 wait complete. 2024-05-27T07:13:38.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:38 smithi132 ceph-mon[35283]: pgmap v552: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:38.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:38 smithi043 ceph-mon[31531]: pgmap v552: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:38.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:38 smithi179 ceph-mon[35888]: pgmap v552: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:38.690 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-27T07:13:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:40.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:40 smithi043 ceph-mon[31531]: pgmap v553: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:40.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:40 smithi179 ceph-mon[35888]: pgmap v553: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:40.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:40 smithi132 ceph-mon[35283]: pgmap v553: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-27T07:13:41.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.610+0000 7f2a7a600700 1 -- 172.21.15.43:0/3332938836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a74106610 msgr2=0x7f2a74106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:41.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.610+0000 7f2a7a600700 1 --2- 172.21.15.43:0/3332938836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a74106610 0x7f2a74106a20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2a680099c0 tx=0x7f2a68009cd0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.610+0000 7f2a7a600700 1 -- 172.21.15.43:0/3332938836 shutdown_connections 2024-05-27T07:13:41.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.610+0000 7f2a7a600700 1 --2- 172.21.15.43:0/3332938836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a7410b890 0x7f2a741131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.610+0000 7f2a7a600700 1 --2- 172.21.15.43:0/3332938836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a74106610 0x7f2a74106a20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.610+0000 7f2a7a600700 1 --2- 172.21.15.43:0/3332938836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a7410af40 0x7f2a7410b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.610+0000 7f2a7a600700 1 -- 172.21.15.43:0/3332938836 >> 172.21.15.43:0/3332938836 conn(0x7f2a74102020 msgr2=0x7f2a74104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:41.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.610+0000 7f2a7a600700 1 -- 172.21.15.43:0/3332938836 shutdown_connections 2024-05-27T07:13:41.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.610+0000 7f2a7a600700 1 -- 172.21.15.43:0/3332938836 wait complete. 2024-05-27T07:13:41.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.612+0000 7f2a7a600700 1 Processor -- start 2024-05-27T07:13:41.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.612+0000 7f2a7a600700 1 -- start start 2024-05-27T07:13:41.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.612+0000 7f2a7a600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a74106610 0x7f2a7419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.612+0000 7f2a7a600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a7410af40 0x7f2a7419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.612+0000 7f2a7a600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a7410b890 0x7f2a741a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.612+0000 7f2a7a600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2a741163e0 con 0x7f2a7410b890 2024-05-27T07:13:41.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.612+0000 7f2a7a600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2a74116260 con 0x7f2a74106610 2024-05-27T07:13:41.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.612+0000 7f2a7a600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2a74116560 con 0x7f2a7410af40 2024-05-27T07:13:41.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a79c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a7410b890 0x7f2a741a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a79c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a7410b890 0x7f2a741a15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49570/0 (socket says 172.21.15.43:49570) 2024-05-27T07:13:41.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a79c00700 1 -- 172.21.15.43:0/2122534160 learned_addr learned my addr 172.21.15.43:0/2122534160 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:41.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a79200700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a74106610 0x7f2a7419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a73e00700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a7410af40 0x7f2a7419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a79c00700 1 -- 172.21.15.43:0/2122534160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a7410af40 msgr2=0x7f2a7419d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:41.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a79c00700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a7410af40 0x7f2a7419d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a79c00700 1 -- 172.21.15.43:0/2122534160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a74106610 msgr2=0x7f2a7419cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:41.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a79c00700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a74106610 0x7f2a7419cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a79c00700 1 -- 172.21.15.43:0/2122534160 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a741a1c60 con 0x7f2a7410b890 2024-05-27T07:13:41.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.613+0000 7f2a79200700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a74106610 0x7f2a7419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:41.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.614+0000 7f2a79c00700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a7410b890 0x7f2a741a15c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00ef30 tx=0x7f2a6c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:41.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.614+0000 7f2a71600700 1 -- 172.21.15.43:0/2122534160 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c0092e0 con 0x7f2a7410b890 2024-05-27T07:13:41.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.614+0000 7f2a71600700 1 -- 172.21.15.43:0/2122534160 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a6c004bb0 con 0x7f2a7410b890 2024-05-27T07:13:41.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.614+0000 7f2a7a600700 1 -- 172.21.15.43:0/2122534160 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a741a1ef0 con 0x7f2a7410b890 2024-05-27T07:13:41.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.614+0000 7f2a71600700 1 -- 172.21.15.43:0/2122534160 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c004020 con 0x7f2a7410b890 2024-05-27T07:13:41.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.614+0000 7f2a7a600700 1 -- 172.21.15.43:0/2122534160 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a741a9870 con 0x7f2a7410b890 2024-05-27T07:13:41.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.618+0000 7f2a71600700 1 -- 172.21.15.43:0/2122534160 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7f2a6c01b030 con 0x7f2a7410b890 2024-05-27T07:13:41.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.619+0000 7f2a8a77cb80 1 Processor -- start 2024-05-27T07:13:41.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.619+0000 7f2a8a77cb80 1 -- start start 2024-05-27T07:13:41.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556254585df0 0x556254703e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x55625470ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5562547047d0 0x556254710fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556254632df0 con 0x5562547043c0 2024-05-27T07:13:41.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556254622990 con 0x556254585df0 2024-05-27T07:13:41.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5562546269d0 con 0x5562547047d0 2024-05-27T07:13:41.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a71600700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a5c0b2880 0x7f2a5c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a79200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x55625470ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a71600700 1 -- 172.21.15.43:0/2122534160 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f2a6c0e2110 con 0x7f2a7410b890 2024-05-27T07:13:41.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a79200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x55625470ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49584/0 (socket says 172.21.15.43:49584) 2024-05-27T07:13:41.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.620+0000 7f2a79200700 1 -- 172.21.15.43:0/2796385178 learned_addr learned my addr 172.21.15.43:0/2796385178 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:41.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.621+0000 7f2a73e00700 1 --2- 172.21.15.43:0/2796385178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556254585df0 0x556254703e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.621+0000 7f2a79c00700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a5c0b2880 0x7f2a5c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.621+0000 7f2a73e00700 1 --2- 172.21.15.43:0/2796385178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5562547047d0 0x556254710fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.621+0000 7f2a79200700 1 -- 172.21.15.43:0/2796385178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5562547047d0 msgr2=0x556254710fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:41.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.621+0000 7f2a79200700 1 --2- 172.21.15.43:0/2796385178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5562547047d0 0x556254710fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.621+0000 7f2a79200700 1 -- 172.21.15.43:0/2796385178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556254585df0 msgr2=0x556254703e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:41.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.621+0000 7f2a79200700 1 --2- 172.21.15.43:0/2796385178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556254585df0 0x556254703e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.621+0000 7f2a79200700 1 -- 172.21.15.43:0/2796385178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562545eb3d0 con 0x5562547043c0 2024-05-27T07:13:41.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.621+0000 7f2a73e00700 1 --2- 172.21.15.43:0/2796385178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556254585df0 0x556254703e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:41.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.621+0000 7f2a79200700 1 --2- 172.21.15.43:0/2796385178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x55625470ea80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2a68015000 tx=0x7f2a68009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c171304dd672f5d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:41.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.622+0000 7f2a79c00700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a5c0b2880 0x7f2a5c0b4d90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c007fd0 tx=0x7f2a6c009480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:41.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.622+0000 7f2a66000700 1 -- 172.21.15.43:0/2796385178 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68026070 con 0x5562547043c0 2024-05-27T07:13:41.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.622+0000 7f2a66000700 1 -- 172.21.15.43:0/2796385178 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a680037c0 con 0x5562547043c0 2024-05-27T07:13:41.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.622+0000 7f2a66000700 1 -- 172.21.15.43:0/2796385178 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6800bbb0 con 0x5562547043c0 2024-05-27T07:13:41.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.623+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2796385178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 msgr2=0x55625470ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.623+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/2796385178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x55625470ea80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2a68015000 tx=0x7f2a68009510 comp rx=0 tx=0).stop 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.623+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2796385178 shutdown_connections 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.623+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/2796385178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5562547047d0 0x556254710fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.623+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/2796385178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556254585df0 0x556254703e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.623+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/2796385178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x55625470ea80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.623+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2796385178 >> 172.21.15.43:0/2796385178 conn(0x5562545e5f70 msgr2=0x5562546ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.623+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2796385178 shutdown_connections 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.623+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2796385178 wait complete. 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.624+0000 7f2a8a77cb80 1 -- start start 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.624+0000 7f2a8a77cb80 1 Processor -- start 2024-05-27T07:13:41.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556254585df0 0x55625470e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547043c0 0x55625470ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547047d0 0x55625470ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556254528f50 con 0x5562547047d0 2024-05-27T07:13:41.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5562545c3710 con 0x5562547043c0 2024-05-27T07:13:41.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5562547118e0 con 0x556254585df0 2024-05-27T07:13:41.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a8a77cb80 10 client.0 _ll_register_callbacks cb 0x5562545ccc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:13:41.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a73e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547047d0 0x55625470ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a73e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547047d0 0x55625470ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49586/0 (socket says 172.21.15.43:49586) 2024-05-27T07:13:41.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a73e00700 1 -- 172.21.15.43:0/246677338 learned_addr learned my addr 172.21.15.43:0/246677338 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a73e00700 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556254585df0 0x55625470e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.625+0000 7f2a79200700 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547043c0 0x55625470ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.626+0000 7f2a73e00700 1 -- 172.21.15.43:0/246677338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556254585df0 msgr2=0x55625470e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.626+0000 7f2a73e00700 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556254585df0 0x55625470e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.626+0000 7f2a73e00700 1 -- 172.21.15.43:0/246677338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547043c0 msgr2=0x55625470ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.626+0000 7f2a73e00700 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547043c0 0x55625470ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.626+0000 7f2a73e00700 1 -- 172.21.15.43:0/246677338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556254572880 con 0x5562547047d0 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.626+0000 7f2a73e00700 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547047d0 0x55625470ef60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2a7406cea0 tx=0x7f2a600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b5bba4b98ebb34 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.626+0000 7f2a5aa00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.626+0000 7f2a5aa00700 1 -- 172.21.15.43:0/246677338 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a60010e60 con 0x5562547047d0 2024-05-27T07:13:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.626+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/246677338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a680037c0 con 0x5562547047d0 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.626+0000 7f2a5aa00700 1 -- 172.21.15.43:0/246677338 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f2a6000fdf0 con 0x5562547047d0 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a5aa00700 1 -- 172.21.15.43:0/246677338 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a60028050 con 0x5562547047d0 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a8a77cb80 10 client.15291 fetch_fsmap learned FSMap version 29 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/246677338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5562545f3010 con 0x5562547047d0 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a5aa00700 1 -- 172.21.15.43:0/246677338 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a680037c0 con 0x5562547047d0 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a5aa00700 1 -- 172.21.15.43:0/246677338 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2a60003b60 con 0x5562547047d0 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a8a77cb80 10 client.15291 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a8a77cb80 10 client.15291 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a8a77cb80 20 client.15291 populate_metadata read hostname 'smithi043' 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55625471e8a0 0x55625471fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a8a77cb80 4 client.15291 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:13:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.627+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/246677338 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556254634d40 con 0x55625471e8a0 2024-05-27T07:13:41.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.628+0000 7f2a79200700 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55625471e8a0 0x55625471fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.629+0000 7f2a79200700 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55625471e8a0 0x55625471fd10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d28d5d323390eeea server_cookie=9b56e3da5db9178b in_seq=0 out_seq=0 2024-05-27T07:13:41.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.629+0000 7f2a5aa00700 10 client.15291 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:13:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.631+0000 7f2a5aa00700 1 -- 172.21.15.43:0/246677338 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556254634d40 con 0x55625471e8a0 2024-05-27T07:13:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.631+0000 7f2a5aa00700 10 client.15291 handle_command_reply: tid=0 2024-05-27T07:13:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.631+0000 7f2a8a77cb80 1 client.15291 shutdown 2024-05-27T07:13:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/246677338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547047d0 msgr2=0x55625470ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547047d0 0x55625470ef60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2a7406cea0 tx=0x7f2a600047c0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/246677338 shutdown_connections 2024-05-27T07:13:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55625471e8a0 0x55625471fd10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547047d0 0x55625470ef60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556254585df0 0x55625470e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/246677338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547043c0 0x55625470ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/246677338 >> 172.21.15.43:0/246677338 conn(0x5562545e5f70 msgr2=0x5562546fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:41.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a5aa00700 0 client.15291 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:13:41.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/246677338 shutdown_connections 2024-05-27T07:13:41.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/246677338 wait complete. 2024-05-27T07:13:41.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.632+0000 7f2a8a77cb80 20 client.15291 trim_cache size 0 max 16384 2024-05-27T07:13:41.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.656+0000 7f2a8a77cb80 1 Processor -- start 2024-05-27T07:13:41.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.656+0000 7f2a8a77cb80 1 -- start start 2024-05-27T07:13:41.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.656+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556254585df0 0x556254708770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.656+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x556254709d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547047d0 0x55625471abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55625471e050 con 0x5562547043c0 2024-05-27T07:13:41.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55625471e720 con 0x5562547047d0 2024-05-27T07:13:41.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55625471edf0 con 0x556254585df0 2024-05-27T07:13:41.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a79200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x556254709d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a79200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x556254709d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49598/0 (socket says 172.21.15.43:49598) 2024-05-27T07:13:41.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a79200700 1 -- 172.21.15.43:0/3655227180 learned_addr learned my addr 172.21.15.43:0/3655227180 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:41.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a73e00700 1 --2- 172.21.15.43:0/3655227180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556254585df0 0x556254708770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a73e00700 1 --2- 172.21.15.43:0/3655227180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547047d0 0x55625471abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a79200700 1 -- 172.21.15.43:0/3655227180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556254585df0 msgr2=0x556254708770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:41.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.657+0000 7f2a79200700 1 --2- 172.21.15.43:0/3655227180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556254585df0 0x556254708770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.658+0000 7f2a79200700 1 -- 172.21.15.43:0/3655227180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547047d0 msgr2=0x55625471abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:41.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.658+0000 7f2a79200700 1 --2- 172.21.15.43:0/3655227180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547047d0 0x55625471abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.658+0000 7f2a79200700 1 -- 172.21.15.43:0/3655227180 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55625471c330 con 0x5562547043c0 2024-05-27T07:13:41.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.658+0000 7f2a79200700 1 --2- 172.21.15.43:0/3655227180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x556254709d40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800b800 tx=0x7f2a68009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57031e653f7619b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:41.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.658+0000 7f2a65600700 1 -- 172.21.15.43:0/3655227180 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68026040 con 0x5562547043c0 2024-05-27T07:13:41.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.658+0000 7f2a65600700 1 -- 172.21.15.43:0/3655227180 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a68027040 con 0x5562547043c0 2024-05-27T07:13:41.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.658+0000 7f2a65600700 1 -- 172.21.15.43:0/3655227180 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68003c30 con 0x5562547043c0 2024-05-27T07:13:41.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.659+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/3655227180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 msgr2=0x556254709d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:41.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.659+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/3655227180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x556254709d40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800b800 tx=0x7f2a68009cd0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.659+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/3655227180 shutdown_connections 2024-05-27T07:13:41.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.659+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/3655227180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547047d0 0x55625471abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.659+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/3655227180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556254585df0 0x556254708770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.659+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/3655227180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5562547043c0 0x556254709d40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.659+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/3655227180 >> 172.21.15.43:0/3655227180 conn(0x5562545e5f70 msgr2=0x55625470cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:41.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.659+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/3655227180 shutdown_connections 2024-05-27T07:13:41.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.659+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/3655227180 wait complete. 2024-05-27T07:13:41.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.660+0000 7f2a8a77cb80 1 -- start start 2024-05-27T07:13:41.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.660+0000 7f2a8a77cb80 1 Processor -- start 2024-05-27T07:13:41.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556254585df0 0x5562547299f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5562547043c0 0x556254722d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a8a77cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547047d0 0x556254723450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55625471fb90 con 0x556254585df0 2024-05-27T07:13:41.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55625471f4c0 con 0x5562547047d0 2024-05-27T07:13:41.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a8a77cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556254720260 con 0x5562547043c0 2024-05-27T07:13:41.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a8a77cb80 10 client.0 _ll_register_callbacks cb 0x5562545ccc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:13:41.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a73e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556254585df0 0x5562547299f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a73e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556254585df0 0x5562547299f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49608/0 (socket says 172.21.15.43:49608) 2024-05-27T07:13:41.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a73e00700 1 -- 172.21.15.43:0/2136961278 learned_addr learned my addr 172.21.15.43:0/2136961278 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:41.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.661+0000 7f2a79200700 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5562547043c0 0x556254722d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.662+0000 7f2a73e00700 1 -- 172.21.15.43:0/2136961278 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5562547043c0 msgr2=0x556254722d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:41.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.662+0000 7f2a73e00700 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5562547043c0 0x556254722d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.662+0000 7f2a73e00700 1 -- 172.21.15.43:0/2136961278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547047d0 msgr2=0x556254723450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:13:41.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.662+0000 7f2a73e00700 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547047d0 0x556254723450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.662+0000 7f2a73e00700 1 -- 172.21.15.43:0/2136961278 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55625470f2d0 con 0x556254585df0 2024-05-27T07:13:41.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.662+0000 7f2a73e00700 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556254585df0 0x5562547299f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2a60010af0 tx=0x7f2a60010930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8587f47dab6b394a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:41.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.662+0000 7f2a5aa00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:13:41.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.662+0000 7f2a5aa00700 1 -- 172.21.15.43:0/2136961278 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a60010e60 con 0x556254585df0 2024-05-27T07:13:41.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.662+0000 7f2a5aa00700 1 -- 172.21.15.43:0/2136961278 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f2a6001a040 con 0x556254585df0 2024-05-27T07:13:41.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.662+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2136961278 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a68027040 con 0x556254585df0 2024-05-27T07:13:41.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a5aa00700 1 -- 172.21.15.43:0/2136961278 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a6003d050 con 0x556254585df0 2024-05-27T07:13:41.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a8a77cb80 10 client.15297 fetch_fsmap learned FSMap version 29 2024-05-27T07:13:41.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2136961278 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556254701300 con 0x556254585df0 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a5aa00700 1 -- 172.21.15.43:0/2136961278 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2a68027040 con 0x556254585df0 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a5aa00700 1 -- 172.21.15.43:0/2136961278 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2a600042c0 con 0x556254585df0 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a8a77cb80 10 client.15297 fetch_fsmap finished waiting for FSMap version 29 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a8a77cb80 10 client.15297 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a8a77cb80 20 client.15297 populate_metadata read hostname 'smithi043' 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55625478c1e0 0x55625478d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a8a77cb80 4 client.15297 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.663+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2136961278 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "status"}) -- 0x55625462e520 con 0x55625478c1e0 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.664+0000 7f2a79200700 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55625478c1e0 0x55625478d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.665+0000 7f2a79200700 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55625478c1e0 0x55625478d650 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27ce027d6cd39ce8 server_cookie=a6d2620e61003a5a in_seq=0 out_seq=0 2024-05-27T07:13:41.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.665+0000 7f2a5aa00700 10 client.15297 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:13:41.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a5aa00700 1 -- 172.21.15.43:0/2136961278 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55625462e520 con 0x55625478c1e0 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a5aa00700 10 client.15297 handle_command_reply: tid=0 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a8a77cb80 1 client.15297 shutdown 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2136961278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556254585df0 msgr2=0x5562547299f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556254585df0 0x5562547299f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2a60010af0 tx=0x7f2a60010930 comp rx=0 tx=0).stop 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2136961278 shutdown_connections 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5562547047d0 0x556254723450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55625478c1e0 0x55625478d650 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556254585df0 0x5562547299f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a5aa00700 0 client.15297 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a8a77cb80 1 --2- 172.21.15.43:0/2136961278 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5562547043c0 0x556254722d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.666+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2136961278 >> 172.21.15.43:0/2136961278 conn(0x5562545e5f70 msgr2=0x5562546ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:41.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.667+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2136961278 shutdown_connections 2024-05-27T07:13:41.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.667+0000 7f2a8a77cb80 1 -- 172.21.15.43:0/2136961278 wait complete. 2024-05-27T07:13:41.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.667+0000 7f2a8a77cb80 20 client.15297 trim_cache size 0 max 16384 2024-05-27T07:13:41.661 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:13:41.661 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "1bb64d42-1bf6-11ef-bc9b-c7b262605968", 2024-05-27T07:13:41.661 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-05-27T07:13:41.661 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 34484, 2024-05-27T07:13:41.661 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-05-27T07:13:41.661 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-05-27T07:13:41.662 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-05-27T07:13:41.662 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 315.58169828000001, 2024-05-27T07:13:41.662 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 29, 2024-05-27T07:13:41.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 90, 2024-05-27T07:13:41.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 90, 2024-05-27T07:13:41.662 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 340.16879041599998 2024-05-27T07:13:41.662 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:13:41.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.667+0000 7f2a7a600700 1 -- 172.21.15.43:0/2122534160 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a5c0b2880 msgr2=0x7f2a5c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:41.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.667+0000 7f2a7a600700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a5c0b2880 0x7f2a5c0b4d90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c007fd0 tx=0x7f2a6c009480 comp rx=0 tx=0).stop 2024-05-27T07:13:41.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.668+0000 7f2a7a600700 1 -- 172.21.15.43:0/2122534160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a7410b890 msgr2=0x7f2a741a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:41.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.668+0000 7f2a7a600700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a7410b890 0x7f2a741a15c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00ef30 tx=0x7f2a6c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.668+0000 7f2a7a600700 1 -- 172.21.15.43:0/2122534160 shutdown_connections 2024-05-27T07:13:41.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.668+0000 7f2a7a600700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f2a5c0b2880 0x7f2a5c0b4d90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.668+0000 7f2a7a600700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a7410b890 0x7f2a741a15c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.668+0000 7f2a7a600700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2a74106610 0x7f2a7419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.668+0000 7f2a7a600700 1 --2- 172.21.15.43:0/2122534160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a7410af40 0x7f2a7419d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:41.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.668+0000 7f2a7a600700 1 -- 172.21.15.43:0/2122534160 >> 172.21.15.43:0/2122534160 conn(0x7f2a74102020 msgr2=0x7f2a7410a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:41.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.668+0000 7f2a7a600700 1 -- 172.21.15.43:0/2122534160 shutdown_connections 2024-05-27T07:13:41.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:41.668+0000 7f2a7a600700 1 -- 172.21.15.43:0/2122534160 wait complete. 2024-05-27T07:13:42.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:42 smithi043 ceph-mon[31531]: pgmap v554: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:42.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:42 smithi179 ceph-mon[35888]: pgmap v554: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:42 smithi132 ceph-mon[35283]: pgmap v554: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-27T07:13:42.625 INFO:teuthology.run_tasks:Running task sequential... 2024-05-27T07:13:42.635 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-27T07:13:42.635 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-27T07:13:42.636 INFO:teuthology.task.print:Enabling mgr modules 2024-05-27T07:13:42.636 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-27T07:13:42.637 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:13:42.637 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-05-27T07:13:42.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-27T07:13:42.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.940+0000 7feb70e00640 1 -- 172.21.15.43:0/2138217530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb6c106ea0 msgr2=0x7feb6c1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:42.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.940+0000 7feb70e00640 1 --2- 172.21.15.43:0/2138217530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb6c106ea0 0x7feb6c1092c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7feb54009900 tx=0x7feb5402a3f0 comp rx=0 tx=0).stop 2024-05-27T07:13:42.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.940+0000 7feb70e00640 1 -- 172.21.15.43:0/2138217530 shutdown_connections 2024-05-27T07:13:42.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.940+0000 7feb70e00640 1 --2- 172.21.15.43:0/2138217530 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb6c109800 0x7feb6c10bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:42.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.940+0000 7feb70e00640 1 --2- 172.21.15.43:0/2138217530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb6c106ea0 0x7feb6c1092c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:42.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.940+0000 7feb70e00640 1 --2- 172.21.15.43:0/2138217530 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb6c104540 0x7feb6c106960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:42.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.940+0000 7feb70e00640 1 -- 172.21.15.43:0/2138217530 >> 172.21.15.43:0/2138217530 conn(0x7feb6c0fdb40 msgr2=0x7feb6c0ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:42.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.940+0000 7feb70e00640 1 -- 172.21.15.43:0/2138217530 shutdown_connections 2024-05-27T07:13:42.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.940+0000 7feb70e00640 1 -- 172.21.15.43:0/2138217530 wait complete. 2024-05-27T07:13:42.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.941+0000 7feb70e00640 1 Processor -- start 2024-05-27T07:13:42.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.941+0000 7feb70e00640 1 -- start start 2024-05-27T07:13:42.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb70e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb6c104540 0x7feb6c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:42.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb70e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb6c106ea0 0x7feb6c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:42.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb70e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb6c109800 0x7feb6c1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:42.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb70e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c10f460 con 0x7feb6c109800 2024-05-27T07:13:42.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb70e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c10f2e0 con 0x7feb6c104540 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb70e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c10f5e0 con 0x7feb6c106ea0 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb6be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb6c109800 0x7feb6c1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb6be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb6c109800 0x7feb6c1a15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49612/0 (socket says 172.21.15.43:49612) 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb6be00640 1 -- 172.21.15.43:0/788254461 learned_addr learned my addr 172.21.15.43:0/788254461 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb63e00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb6c106ea0 0x7feb6c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.942+0000 7feb6b400640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb6c104540 0x7feb6c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb6be00640 1 -- 172.21.15.43:0/788254461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb6c106ea0 msgr2=0x7feb6c19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb6be00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb6c106ea0 0x7feb6c19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb6be00640 1 -- 172.21.15.43:0/788254461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb6c104540 msgr2=0x7feb6c19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb6be00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb6c104540 0x7feb6c19a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:42.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb6be00640 1 -- 172.21.15.43:0/788254461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6c1a1b80 con 0x7feb6c109800 2024-05-27T07:13:42.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb6be00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb6c109800 0x7feb6c1a15c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7feb6400ccb0 tx=0x7feb6400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:42.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb68c00640 1 -- 172.21.15.43:0/788254461 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64003970 con 0x7feb6c109800 2024-05-27T07:13:42.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb68c00640 1 -- 172.21.15.43:0/788254461 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb64004030 con 0x7feb6c109800 2024-05-27T07:13:42.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb70e00640 1 -- 172.21.15.43:0/788254461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb6c1a1e70 con 0x7feb6c109800 2024-05-27T07:13:42.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb68c00640 1 -- 172.21.15.43:0/788254461 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6400f6b0 con 0x7feb6c109800 2024-05-27T07:13:42.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.943+0000 7feb70e00640 1 -- 172.21.15.43:0/788254461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb6c1a22d0 con 0x7feb6c109800 2024-05-27T07:13:42.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.945+0000 7feb70e00640 1 -- 172.21.15.43:0/788254461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb6c0ff660 con 0x7feb6c109800 2024-05-27T07:13:42.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.947+0000 7feb68c00640 1 -- 172.21.15.43:0/788254461 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 149652+0+0 (secure 0 0 0) 0x7feb64003b10 con 0x7feb6c109800 2024-05-27T07:13:42.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.949+0000 7feb68c00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7feb500b27e0 0x7feb500b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:42.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.949+0000 7feb68c00640 1 -- 172.21.15.43:0/788254461 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7feb640e2b60 con 0x7feb6c109800 2024-05-27T07:13:42.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.949+0000 7feb6b400640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7feb500b27e0 0x7feb500b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:42.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.950+0000 7feb6b400640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7feb500b27e0 0x7feb500b4cf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feb5c005fd0 tx=0x7feb5c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:42.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:42.952+0000 7feb68c00640 1 -- 172.21.15.43:0/788254461 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7feb64014070 con 0x7feb6c109800 2024-05-27T07:13:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:43.219+0000 7feb70e00640 1 -- 172.21.15.43:0/788254461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7feb6c02ddb0 con 0x7feb6c109800 2024-05-27T07:13:44.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.118+0000 7feb68c00640 1 -- 172.21.15.43:0/788254461 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7feb640aa490 con 0x7feb6c109800 2024-05-27T07:13:44.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.123+0000 7feb70e00640 1 -- 172.21.15.43:0/788254461 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7feb500b27e0 msgr2=0x7feb500b4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:44.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.123+0000 7feb70e00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7feb500b27e0 0x7feb500b4cf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feb5c005fd0 tx=0x7feb5c005850 comp rx=0 tx=0).stop 2024-05-27T07:13:44.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.123+0000 7feb70e00640 1 -- 172.21.15.43:0/788254461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb6c109800 msgr2=0x7feb6c1a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:44.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.123+0000 7feb70e00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb6c109800 0x7feb6c1a15c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7feb6400ccb0 tx=0x7feb6400a620 comp rx=0 tx=0).stop 2024-05-27T07:13:44.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.124+0000 7feb70e00640 1 -- 172.21.15.43:0/788254461 shutdown_connections 2024-05-27T07:13:44.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.124+0000 7feb70e00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7feb500b27e0 0x7feb500b4cf0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:44.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.124+0000 7feb70e00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb6c109800 0x7feb6c1a15c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:44.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.124+0000 7feb70e00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb6c106ea0 0x7feb6c19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:44.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.124+0000 7feb70e00640 1 --2- 172.21.15.43:0/788254461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb6c104540 0x7feb6c19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:44.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.124+0000 7feb70e00640 1 -- 172.21.15.43:0/788254461 >> 172.21.15.43:0/788254461 conn(0x7feb6c0fdb40 msgr2=0x7feb6c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:44.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.124+0000 7feb70e00640 1 -- 172.21.15.43:0/788254461 shutdown_connections 2024-05-27T07:13:44.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:44.124+0000 7feb70e00640 1 -- 172.21.15.43:0/788254461 wait complete. 2024-05-27T07:13:44.158 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-05-27T07:13:44.169 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-05-27T07:13:44.169 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-05-27T07:13:44.169 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-05-27T07:13:44.169 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T07:13:44.169 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T07:13:44.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:44.169 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-05-27T07:13:44.216 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-27T07:13:44.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:44.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-27T07:13:44.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:44.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link delete ceph-brx 2024-05-27T07:13:44.393 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: ignoring --setuser ceph since I am not root 2024-05-27T07:13:44.393 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: ignoring --setgroup ceph since I am not root 2024-05-27T07:13:44.393 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:44 smithi132 ceph-mgr[36667]: -- 172.21.15.132:0/2861842181 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5608e77c85a0 con 0x5608ec552900 2024-05-27T07:13:44.393 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:44 smithi132 ceph-mon[35283]: pgmap v555: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:44.393 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:44 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/788254461' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:13:44.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:44 smithi043 ceph-mon[31531]: pgmap v555: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:44.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:44 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/788254461' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:13:44.396 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: ignoring --setuser ceph since I am not root 2024-05-27T07:13:44.396 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: ignoring --setgroup ceph since I am not root 2024-05-27T07:13:44.444 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-05-27T07:13:44.444 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-05-27T07:13:44.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:44 smithi179 ceph-mon[35888]: pgmap v555: 129 pgs: 129 active+clean; 647 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:13:44.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:44 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/788254461' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-27T07:13:44.459 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: ignoring --setuser ceph since I am not root 2024-05-27T07:13:44.459 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: ignoring --setgroup ceph since I am not root 2024-05-27T07:13:44.459 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:44 smithi179 ceph-mgr[37622]: -- 172.21.15.179:0/3315951564 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562a5078c5a0 con 0x562a55520900 2024-05-27T07:13:44.702 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:44.573+0000 7fb9f4453200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:13:44.773 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:44.525+0000 7fb6cec25200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:13:44.785 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:44.537+0000 7f1fd4902200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T07:13:44.890 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:45.091 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:44.793+0000 7f1fd4902200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:13:45.091 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:44.926+0000 7f1fd4902200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:13:45.140 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:44.782+0000 7fb6cec25200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:13:45.140 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:44.915+0000 7fb6cec25200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:13:45.140 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:44.844+0000 7fb9f4453200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:13:45.140 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:44.992+0000 7fb9f4453200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T07:13:45.420 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:45.312+0000 7fb6cec25200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:13:45.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:45 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/788254461' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:13:45.420 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:45 smithi179 ceph-mon[35888]: mgrmap e19: x(active, since 15m), standbys: z, y 2024-05-27T07:13:45.428 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:45.324+0000 7f1fd4902200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:13:45.429 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:45 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/788254461' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:13:45.429 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:45 smithi132 ceph-mon[35283]: mgrmap e19: x(active, since 15m), standbys: z, y 2024-05-27T07:13:45.430 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:45 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/788254461' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-27T07:13:45.430 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:45 smithi043 ceph-mon[31531]: mgrmap e19: x(active, since 15m), standbys: z, y 2024-05-27T07:13:45.703 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:45 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:45.438+0000 7fb9f4453200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T07:13:45.703 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:45 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:45.559+0000 7fb9f4453200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:13:45.708 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:45.428+0000 7fb6cec25200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:13:45.841 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:45.437+0000 7f1fd4902200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T07:13:47.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.287+0000 7fcb0cc00700 1 -- 172.21.15.43:0/2873776405 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081045e0 msgr2=0x7fcb081049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:47.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.287+0000 7fcb0cc00700 1 --2- 172.21.15.43:0/2873776405 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081045e0 0x7fcb081049f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80099c0 tx=0x7fcaf8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.287+0000 7fcb0cc00700 1 -- 172.21.15.43:0/2873776405 shutdown_connections 2024-05-27T07:13:47.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.287+0000 7fcb0cc00700 1 --2- 172.21.15.43:0/2873776405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb08109860 0x7fcb081111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.287+0000 7fcb0cc00700 1 --2- 172.21.15.43:0/2873776405 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081045e0 0x7fcb081049f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.287+0000 7fcb0cc00700 1 --2- 172.21.15.43:0/2873776405 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb08108f10 0x7fcb08109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.287+0000 7fcb0cc00700 1 -- 172.21.15.43:0/2873776405 >> 172.21.15.43:0/2873776405 conn(0x7fcb08100030 msgr2=0x7fcb08102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:47.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.287+0000 7fcb0cc00700 1 -- 172.21.15.43:0/2873776405 shutdown_connections 2024-05-27T07:13:47.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.288+0000 7fcb0cc00700 1 -- 172.21.15.43:0/2873776405 wait complete. 2024-05-27T07:13:47.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.289+0000 7fcb0cc00700 1 Processor -- start 2024-05-27T07:13:47.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.289+0000 7fcb0cc00700 1 -- start start 2024-05-27T07:13:47.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.290+0000 7fcb0cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081045e0 0x7fcb08075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:47.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.290+0000 7fcb0cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb08108f10 0x7fcb08075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:47.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.290+0000 7fcb0cc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb08109860 0x7fcb08076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:47.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.290+0000 7fcb0cc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08114390 con 0x7fcb081045e0 2024-05-27T07:13:47.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.290+0000 7fcb0cc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08114210 con 0x7fcb08108f10 2024-05-27T07:13:47.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.290+0000 7fcb0cc00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08114510 con 0x7fcb08109860 2024-05-27T07:13:47.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.290+0000 7fcb05e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081045e0 0x7fcb08075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:47.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.290+0000 7fcb05e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081045e0 0x7fcb08075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40996/0 (socket says 172.21.15.43:40996) 2024-05-27T07:13:47.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.290+0000 7fcb05e00700 1 -- 172.21.15.43:0/4089781234 learned_addr learned my addr 172.21.15.43:0/4089781234 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:47.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.291+0000 7fcb06800700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb08109860 0x7fcb08076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.291+0000 7fcb05400700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb08108f10 0x7fcb08075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.291+0000 7fcb05e00700 1 -- 172.21.15.43:0/4089781234 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb08109860 msgr2=0x7fcb08076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.291+0000 7fcb05e00700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb08109860 0x7fcb08076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.291+0000 7fcb05e00700 1 -- 172.21.15.43:0/4089781234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb08108f10 msgr2=0x7fcb08075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.291+0000 7fcb05e00700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb08108f10 0x7fcb08075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.291+0000 7fcb05e00700 1 -- 172.21.15.43:0/4089781234 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb081a9140 con 0x7fcb081045e0 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.291+0000 7fcb05400700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb08108f10 0x7fcb08075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.291+0000 7fcb05e00700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081045e0 0x7fcb08075ab0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8000c00 tx=0x7fcaf800f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.292+0000 7fcafea00700 1 -- 172.21.15.43:0/4089781234 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8027070 con 0x7fcb081045e0 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.292+0000 7fcafea00700 1 -- 172.21.15.43:0/4089781234 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcaf800fe50 con 0x7fcb081045e0 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.292+0000 7fcb0cc00700 1 -- 172.21.15.43:0/4089781234 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb081a93d0 con 0x7fcb081045e0 2024-05-27T07:13:47.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.292+0000 7fcafea00700 1 -- 172.21.15.43:0/4089781234 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80173f0 con 0x7fcb081045e0 2024-05-27T07:13:47.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.292+0000 7fcb0cc00700 1 -- 172.21.15.43:0/4089781234 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb081a9820 con 0x7fcb081045e0 2024-05-27T07:13:47.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.293+0000 7fcb0cc00700 1 -- 172.21.15.43:0/4089781234 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae8005330 con 0x7fcb081045e0 2024-05-27T07:13:47.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.297+0000 7fcafea00700 1 -- 172.21.15.43:0/4089781234 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7fcaf802a070 con 0x7fcb081045e0 2024-05-27T07:13:47.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.299+0000 7fcafea00700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcaec0b28d0 0x7fcaec0b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:47.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.299+0000 7fcb05400700 1 -- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcaec0b28d0 msgr2=0x7fcaec0b4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1809594794 2024-05-27T07:13:47.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.299+0000 7fcafea00700 1 -- 172.21.15.43:0/4089781234 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fcaf80e3500 con 0x7fcb081045e0 2024-05-27T07:13:47.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.299+0000 7fcb05400700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcaec0b28d0 0x7fcaec0b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:13:47.291 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:47 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:47.223+0000 7fb9f4453200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:13:47.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.304+0000 7fcafea00700 1 -- 172.21.15.43:0/4089781234 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcaf8013070 con 0x7fcb081045e0 2024-05-27T07:13:47.341 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:46 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:46.982+0000 7f1fd4902200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:13:47.341 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:47 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:47.212+0000 7f1fd4902200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:13:47.434 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:46 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:46.994+0000 7fb6cec25200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T07:13:47.434 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:47.224+0000 7fb6cec25200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:13:47.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.500+0000 7fcb05400700 1 -- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcaec0b28d0 msgr2=0x7fcaec0b4de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1809594794 2024-05-27T07:13:47.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.500+0000 7fcb05400700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcaec0b28d0 0x7fcaec0b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:13:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.585+0000 7fcb0cc00700 1 -- 172.21.15.43:0/4089781234 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcae8005e30 con 0x7fcb081045e0 2024-05-27T07:13:47.577 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:47 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:47.470+0000 7fb9f4453200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T07:13:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.586+0000 7fcafea00700 1 -- 172.21.15.43:0/4089781234 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v29) ==== 53+0+88 (secure 0 0 0) 0x7fcaf80aaec0 con 0x7fcb081045e0 2024-05-27T07:13:47.578 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-27T07:13:47.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 -- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcaec0b28d0 msgr2=0x7fcaec0b4de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:13:47.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcaec0b28d0 0x7fcaec0b4de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 -- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081045e0 msgr2=0x7fcb08075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081045e0 0x7fcb08075ab0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8000c00 tx=0x7fcaf800f510 comp rx=0 tx=0).stop 2024-05-27T07:13:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 -- 172.21.15.43:0/4089781234 shutdown_connections 2024-05-27T07:13:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fcaec0b28d0 0x7fcaec0b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb08109860 0x7fcb08076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081045e0 0x7fcb08075ab0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 --2- 172.21.15.43:0/4089781234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb08108f10 0x7fcb08075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 -- 172.21.15.43:0/4089781234 >> 172.21.15.43:0/4089781234 conn(0x7fcb08100030 msgr2=0x7fcb08102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 -- 172.21.15.43:0/4089781234 shutdown_connections 2024-05-27T07:13:47.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:47.589+0000 7fcb0cc00700 1 -- 172.21.15.43:0/4089781234 wait complete. 2024-05-27T07:13:47.709 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:47.443+0000 7fb6cec25200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:13:47.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:47 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4089781234' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:13:47.810 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:47 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:47.432+0000 7f1fd4902200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:13:47.814 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:47 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4089781234' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:13:47.953 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:47 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:47.701+0000 7fb9f4453200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T07:13:47.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:47 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4089781234' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-27T07:13:48.091 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:47 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:47.818+0000 7f1fd4902200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:13:48.091 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:47 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:47.942+0000 7f1fd4902200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:13:48.209 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:47.837+0000 7fb6cec25200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:13:48.209 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:47.961+0000 7fb6cec25200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:13:48.449 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:48 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:48.198+0000 7f1fd4902200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:13:48.452 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:48.126+0000 7fb9f4453200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T07:13:48.453 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:48.267+0000 7fb9f4453200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T07:13:48.468 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:48.218+0000 7fb6cec25200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:13:48.543 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-27T07:13:48.543 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T07:13:48.543 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-05-27T07:13:48.543 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-27T07:13:48.543 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T07:13:48.543 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-27T07:13:48.544 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T07:13:48.544 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T07:13:48.544 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-27T07:13:48.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:48.544 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-05-27T07:13:48.593 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T07:13:48.593 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T07:13:48.593 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T07:13:48.593 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:13:48.593 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-05-27T07:13:48.593 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:13:48.593 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:13:48.593 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-05-27T07:13:48.593 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-05-27T07:13:48.593 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:13:48.594 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-05-27T07:13:48.594 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-05-27T07:13:48.594 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T07:13:48.594 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:13:48.594 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-05-27T07:13:48.594 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-05-27T07:13:48.594 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 41573sec preferred_lft 41573sec 2024-05-27T07:13:48.594 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-05-27T07:13:48.594 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-05-27T07:13:48.594 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-27T07:13:48.595 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-05-27T07:13:48.595 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-05-27T07:13:48.595 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T07:13:48.596 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:13:48.596 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-05-27T07:13:48.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add name ceph-brx type bridge 2024-05-27T07:13:48.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip addr flush dev ceph-brx 2024-05-27T07:13:48.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set ceph-brx up 2024-05-27T07:13:48.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T07:13:48.596 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-05-27T07:13:48.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:13:48.746 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:48.544+0000 7fb9f4453200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T07:13:48.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:13:48.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:48.816 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T07:13:48.841 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:48 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:48.457+0000 7f1fd4902200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:13:48.875 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-05-27T07:13:48.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:48.880 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-05-27T07:13:48.923 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-05-27T07:13:48.923 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T07:13:48.923 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-27T07:13:48.923 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-27T07:13:48.923 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T07:13:48.923 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:13:48.923 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-05-27T07:13:48.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-27T07:13:48.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-27T07:13:48.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-27T07:13:48.924 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-05-27T07:13:48.959 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:48.476+0000 7fb6cec25200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:13:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:13:49.091 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:48.824+0000 7fb9f4453200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T07:13:49.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:13:49.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:49.170 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-05-27T07:13:49.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:49.218 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list-id 2024-05-27T07:13:49.263 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:13:49.263 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-05-27T07:13:49.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:13:49.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-27T07:13:49.263 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-05-27T07:13:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:13:49.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:13:49.475 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-27T07:13:49.475 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:13:49.475 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-05-27T07:13:49.475 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-27T07:13:49.475 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T07:13:49.475 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-27T07:13:49.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-27T07:13:49.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-27T07:13:49.476 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-05-27T07:13:49.550 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:49 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:49.267+0000 7f1fd4902200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:13:49.567 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:49.284+0000 7fb6cec25200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:13:49.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:13:49.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:13:49.735 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T07:13:49.735 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-05-27T07:13:49.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-05-27T07:13:49.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T07:13:49.735 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-05-27T07:13:49.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T07:13:49.841 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:49 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:49.558+0000 7f1fd4902200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:13:49.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:13:49.934 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-05-27T07:13:49.935 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:13:49.935 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-27T07:13:49.952 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:49 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:49.690+0000 7fb9f4453200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-27T07:13:49.958 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:49.576+0000 7fb6cec25200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:13:49.977 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-27T07:13:49.978 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T07:13:49.978 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-27T07:13:50.069 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-05-27T07:13:50.144 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:13:50.190 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-05-27T07:13:50.191 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-05-27T07:13:50.191 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-05-27T07:13:50.191 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-05-27T07:13:50.191 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-05-27T07:13:50.191 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-05-27T07:13:50.191 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-05-27T07:13:50.191 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-05-27T07:13:50.191 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-05-27T07:13:50.191 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/bpf 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:13:50.192 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fb962e0896f306e6bdf309817399bbb3496e7c31069501c545db8fe3e7bba85d/merged 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/6741f2331dca4059307dfc7f032fc0462f11aef53f9941537451b7c8713b2447/merged 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/0 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d94143aab52c0cf4e74c4ce713e61b04a988bdf6896e24080ad3b89fe8435b77/merged 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fe62c8ab8b1aeaa91d01ab9fe0bf51ede8e7f0357669aaf589eab6a2ce1bac66/merged 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/202cde062bc89da2b696fbfe1f7746610c02b5423d4469ab514a3cbb53acd07f/merged 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/df5388b1b92ef39ed24085545b944bc3ed739d064b74cb685878895bf4818382/merged 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/5fbcb271e6c21124319502ad3025fc6461d8a6854221c4825ab7f11c56ec4a21/merged 2024-05-27T07:13:50.193 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/484787d0f6d97e5d65195de880768fe8e92235f422c129fa06f3d45a43eee6fb/merged 2024-05-27T07:13:50.194 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/1b86423c0412d7d490dab40bc424d2df4478d6849737c098ddd5feb887413f9b/merged 2024-05-27T07:13:50.194 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d011ecf4d242761a3d35b1dc879cbeda1e6484d35d2d8ca10cd410c5f937d874/merged 2024-05-27T07:13:50.194 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-05-27T07:13:50.194 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:13:50.194 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:13:50.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:50.195 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-05-27T07:13:50.236 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-27T07:13:50.236 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/40de48eb-1580-4e20-8968-b75b2a70113a --client_fs=cephfs) 2024-05-27T07:13:50.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-05-27T07:13:50.339 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:50.009+0000 7fb9f4453200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-27T07:13:50.353 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:13:50.397 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/bpf 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-05-27T07:13:50.398 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-05-27T07:13:50.399 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fb962e0896f306e6bdf309817399bbb3496e7c31069501c545db8fe3e7bba85d/merged 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/6741f2331dca4059307dfc7f032fc0462f11aef53f9941537451b7c8713b2447/merged 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/0 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d94143aab52c0cf4e74c4ce713e61b04a988bdf6896e24080ad3b89fe8435b77/merged 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fe62c8ab8b1aeaa91d01ab9fe0bf51ede8e7f0357669aaf589eab6a2ce1bac66/merged 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/202cde062bc89da2b696fbfe1f7746610c02b5423d4469ab514a3cbb53acd07f/merged 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/df5388b1b92ef39ed24085545b944bc3ed739d064b74cb685878895bf4818382/merged 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/5fbcb271e6c21124319502ad3025fc6461d8a6854221c4825ab7f11c56ec4a21/merged 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/484787d0f6d97e5d65195de880768fe8e92235f422c129fa06f3d45a43eee6fb/merged 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/1b86423c0412d7d490dab40bc424d2df4478d6849737c098ddd5feb887413f9b/merged 2024-05-27T07:13:50.400 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d011ecf4d242761a3d35b1dc879cbeda1e6484d35d2d8ca10cd410c5f937d874/merged 2024-05-27T07:13:50.401 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-05-27T07:13:50.401 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:13:50.401 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:13:50.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:50.402 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-05-27T07:13:50.459 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:50.147+0000 7fb6cec25200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:13:50.459 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:50.276+0000 7fb6cec25200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:13:50.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:2024-05-27T07:13:50.542+0000 7f8dc681c480 -1 init, newargv = 0x7f8dac056f20 newargc=15 2024-05-27T07:13:50.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[88755]: starting ceph client 2024-05-27T07:13:50.589 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[88755]: starting fuse 2024-05-27T07:13:50.591 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:50 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:50.129+0000 7f1fd4902200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:13:50.591 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:50 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:50.258+0000 7f1fd4902200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:13:50.953 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:50.626+0000 7fb9f4453200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T07:13:50.953 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:50.768+0000 7fb9f4453200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T07:13:51.091 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:50 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:50.739+0000 7f1fd4902200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:13:51.209 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:50.760+0000 7fb6cec25200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:13:51.453 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:51.270+0000 7fb9f4453200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-27T07:13:51.587 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-05-27T07:13:51.591 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:51.199+0000 7f1fd4902200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:13:51.591 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:51.326+0000 7f1fd4902200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:13:51.654 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T07:13:51.668 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:51.226+0000 7fb6cec25200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:13:51.668 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:51.353+0000 7fb6cec25200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:13:51.669 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:51.677+0000 7fb6cec25200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:13:51.699 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-05-27T07:13:51.699 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-05-27T07:13:51.699 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-05-27T07:13:51.699 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-05-27T07:13:51.699 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-05-27T07:13:51.699 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-05-27T07:13:51.699 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-05-27T07:13:51.699 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/bpf 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-27T07:13:51.700 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fb962e0896f306e6bdf309817399bbb3496e7c31069501c545db8fe3e7bba85d/merged 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/6741f2331dca4059307dfc7f032fc0462f11aef53f9941537451b7c8713b2447/merged 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/0 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d94143aab52c0cf4e74c4ce713e61b04a988bdf6896e24080ad3b89fe8435b77/merged 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fe62c8ab8b1aeaa91d01ab9fe0bf51ede8e7f0357669aaf589eab6a2ce1bac66/merged 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/202cde062bc89da2b696fbfe1f7746610c02b5423d4469ab514a3cbb53acd07f/merged 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/df5388b1b92ef39ed24085545b944bc3ed739d064b74cb685878895bf4818382/merged 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/5fbcb271e6c21124319502ad3025fc6461d8a6854221c4825ab7f11c56ec4a21/merged 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/484787d0f6d97e5d65195de880768fe8e92235f422c129fa06f3d45a43eee6fb/merged 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/1b86423c0412d7d490dab40bc424d2df4478d6849737c098ddd5feb887413f9b/merged 2024-05-27T07:13:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d011ecf4d242761a3d35b1dc879cbeda1e6484d35d2d8ca10cd410c5f937d874/merged 2024-05-27T07:13:51.702 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-05-27T07:13:51.702 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:13:51.702 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:13:51.702 INFO:teuthology.orchestra.run.smithi043.stdout:/home/ubuntu/cephtest/mnt.0 2024-05-27T07:13:51.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:51.703 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-05-27T07:13:51.792 INFO:teuthology.orchestra.run.smithi043.stdout:119 2024-05-27T07:13:51.792 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-05-27T07:13:51.793 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.0")) 2024-05-27T07:13:51.794 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-05-27T07:13:51.894 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:51.763+0000 7fb9f4453200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T07:13:51.948 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.0.88755.asok 2024-05-27T07:13:51.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T07:13:51.958 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:51.652+0000 7f1fd4902200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:13:51.958 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:51.835+0000 7f1fd4902200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:13:51.959 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:51.860+0000 7fb6cec25200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:13:51.963 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.88755.asok 2024-05-27T07:13:51.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:51.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.88755.asok status 2024-05-27T07:13:52.203 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:51.901+0000 7fb9f4453200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T07:13:52.261 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:13:52.261 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "38f9d900c83c63799fbdbe61acc9a11b0d3554a6", 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "0", 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "88755", 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/volumes/qa/sv_0/40de48eb-1580-4e20-8968-b75b2a70113a" 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 25090, 2024-05-27T07:13:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 25090 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2918082699 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-05-27T07:13:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2918082699 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.25090 172.21.15.43:0/2918082699", 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "172.21.15.43:0/2918082699", 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 5, 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 29, 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 90, 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-05-27T07:13:52.264 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:13:52.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:52.283 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-05-27T07:13:52.283 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-27T07:13:52.337 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-05-27T07:13:52.338 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-27T07:13:52.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:13:52.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-27T07:13:52.341 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:51.965+0000 7f1fd4902200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:13:52.390 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:51.990+0000 7fb6cec25200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:13:52.424 INFO:teuthology.run_tasks:Running task exec... 2024-05-27T07:13:52.437 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:13:52.438 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-05-27T07:13:52.438 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-05-27T07:13:52.508 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:52.255+0000 7fb9f4453200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T07:13:52.508 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:52.451+0000 7fb9f4453200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T07:13:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.715+0000 7fed97e00640 1 -- 172.21.15.43:0/1067500576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed90109ba0 msgr2=0x7fed9010bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.715+0000 7fed97e00640 1 --2- 172.21.15.43:0/1067500576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed90109ba0 0x7fed9010bfc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fed88009900 tx=0x7fed8802a410 comp rx=0 tx=0).stop 2024-05-27T07:13:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.715+0000 7fed97e00640 1 -- 172.21.15.43:0/1067500576 shutdown_connections 2024-05-27T07:13:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.715+0000 7fed97e00640 1 --2- 172.21.15.43:0/1067500576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed90109ba0 0x7fed9010bfc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.715+0000 7fed97e00640 1 --2- 172.21.15.43:0/1067500576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed90104ea0 0x7fed901052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.715+0000 7fed97e00640 1 --2- 172.21.15.43:0/1067500576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed90103c60 0x7fed90104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.715+0000 7fed97e00640 1 -- 172.21.15.43:0/1067500576 >> 172.21.15.43:0/1067500576 conn(0x7fed900fee50 msgr2=0x7fed90101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.715+0000 7fed97e00640 1 -- 172.21.15.43:0/1067500576 shutdown_connections 2024-05-27T07:13:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.715+0000 7fed97e00640 1 -- 172.21.15.43:0/1067500576 wait complete. 2024-05-27T07:13:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.716+0000 7fed97e00640 1 Processor -- start 2024-05-27T07:13:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.717+0000 7fed97e00640 1 -- start start 2024-05-27T07:13:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.717+0000 7fed97e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed90103c60 0x7fed90077a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.717+0000 7fed97e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed90104ea0 0x7fed90076120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.717+0000 7fed97e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed90109ba0 0x7fed90076660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.717+0000 7fed97e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fed9010f690 con 0x7fed90109ba0 2024-05-27T07:13:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.717+0000 7fed97e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fed9010f510 con 0x7fed90104ea0 2024-05-27T07:13:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.717+0000 7fed97e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fed9010f810 con 0x7fed90103c60 2024-05-27T07:13:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.717+0000 7fed95e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed90109ba0 0x7fed90076660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.717+0000 7fed95e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed90109ba0 0x7fed90076660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41012/0 (socket says 172.21.15.43:41012) 2024-05-27T07:13:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.717+0000 7fed95e00640 1 -- 172.21.15.43:0/2444333805 learned_addr learned my addr 172.21.15.43:0/2444333805 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.718+0000 7fed94a00640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed90104ea0 0x7fed90076120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.718+0000 7fed95400640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed90103c60 0x7fed90077a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:52.710 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:52.398+0000 7fb6cec25200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:13:52.711 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:52.525+0000 7fb6cec25200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:13:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.718+0000 7fed95e00640 1 -- 172.21.15.43:0/2444333805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed90103c60 msgr2=0x7fed90077a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.718+0000 7fed95e00640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed90103c60 0x7fed90077a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.718+0000 7fed95e00640 1 -- 172.21.15.43:0/2444333805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed90104ea0 msgr2=0x7fed90076120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.718+0000 7fed95e00640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed90104ea0 0x7fed90076120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.718+0000 7fed95e00640 1 -- 172.21.15.43:0/2444333805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed90076d90 con 0x7fed90109ba0 2024-05-27T07:13:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.718+0000 7fed95e00640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed90109ba0 0x7fed90076660 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fed8802a930 tx=0x7fed88003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.719+0000 7fed8e000640 1 -- 172.21.15.43:0/2444333805 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8803f070 con 0x7fed90109ba0 2024-05-27T07:13:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.719+0000 7fed8e000640 1 -- 172.21.15.43:0/2444333805 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed88031b90 con 0x7fed90109ba0 2024-05-27T07:13:52.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.719+0000 7fed8e000640 1 -- 172.21.15.43:0/2444333805 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88035030 con 0x7fed90109ba0 2024-05-27T07:13:52.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.719+0000 7fed97e00640 1 -- 172.21.15.43:0/2444333805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed901a42c0 con 0x7fed90109ba0 2024-05-27T07:13:52.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.719+0000 7fed97e00640 1 -- 172.21.15.43:0/2444333805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed901a46c0 con 0x7fed90109ba0 2024-05-27T07:13:52.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.720+0000 7fed97e00640 1 -- 172.21.15.43:0/2444333805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed90050860 con 0x7fed90109ba0 2024-05-27T07:13:52.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.724+0000 7fed8e000640 1 -- 172.21.15.43:0/2444333805 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7fed88042070 con 0x7fed90109ba0 2024-05-27T07:13:52.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.727+0000 7fed8e000640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed780b2830 0x7fed780b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:52.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.727+0000 7fed95400640 1 -- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed780b2830 msgr2=0x7fed780b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1809594794 2024-05-27T07:13:52.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.727+0000 7fed8e000640 1 -- 172.21.15.43:0/2444333805 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fed88103420 con 0x7fed90109ba0 2024-05-27T07:13:52.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.727+0000 7fed95400640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed780b2830 0x7fed780b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:13:52.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.729+0000 7fed8e000640 1 -- 172.21.15.43:0/2444333805 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fed880cad50 con 0x7fed90109ba0 2024-05-27T07:13:52.841 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:52 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:52.371+0000 7f1fd4902200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:13:52.841 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:52 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:52.500+0000 7f1fd4902200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:13:52.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.928+0000 7fed95400640 1 -- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed780b2830 msgr2=0x7fed780b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1809594794 2024-05-27T07:13:52.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.928+0000 7fed95400640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed780b2830 0x7fed780b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:13:52.953 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:52.593+0000 7fb9f4453200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T07:13:52.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:52.988+0000 7fed97e00640 1 -- 172.21.15.43:0/2444333805 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fed901089b0 con 0x7fed90109ba0 2024-05-27T07:13:53.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.043+0000 7fed8e000640 1 -- 172.21.15.43:0/2444333805 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v30) ==== 117+0+0 (secure 0 0 0) 0x7fed880ca650 con 0x7fed90109ba0 2024-05-27T07:13:53.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.046+0000 7fed97e00640 1 -- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed780b2830 msgr2=0x7fed780b4d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:13:53.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.046+0000 7fed97e00640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed780b2830 0x7fed780b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.046+0000 7fed97e00640 1 -- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed90109ba0 msgr2=0x7fed90076660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.046+0000 7fed97e00640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed90109ba0 0x7fed90076660 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fed8802a930 tx=0x7fed88003b90 comp rx=0 tx=0).stop 2024-05-27T07:13:53.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.046+0000 7fed97e00640 1 -- 172.21.15.43:0/2444333805 shutdown_connections 2024-05-27T07:13:53.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.046+0000 7fed97e00640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fed780b2830 0x7fed780b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.046+0000 7fed97e00640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed90104ea0 0x7fed90076120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.046+0000 7fed97e00640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed90109ba0 0x7fed90076660 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.046+0000 7fed97e00640 1 --2- 172.21.15.43:0/2444333805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed90103c60 0x7fed90077a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.047+0000 7fed97e00640 1 -- 172.21.15.43:0/2444333805 >> 172.21.15.43:0/2444333805 conn(0x7fed900fee50 msgr2=0x7fed90105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:53.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.047+0000 7fed97e00640 1 -- 172.21.15.43:0/2444333805 shutdown_connections 2024-05-27T07:13:53.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.047+0000 7fed97e00640 1 -- 172.21.15.43:0/2444333805 wait complete. 2024-05-27T07:13:53.063 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-27T07:13:53.077 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-27T07:13:53.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T07:13:53.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:53 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2444333805' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-27T07:13:53.341 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:53 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:53.270+0000 7f1fd4902200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:13:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.395+0000 7fb107200640 1 -- 172.21.15.43:0/2149620214 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb108104c40 msgr2=0x7fb108105050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.395+0000 7fb107200640 1 --2- 172.21.15.43:0/2149620214 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb108104c40 0x7fb108105050 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009900 tx=0x7fb0f002a410 comp rx=0 tx=0).stop 2024-05-27T07:13:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.396+0000 7fb107200640 1 -- 172.21.15.43:0/2149620214 shutdown_connections 2024-05-27T07:13:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.396+0000 7fb107200640 1 --2- 172.21.15.43:0/2149620214 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb108109940 0x7fb10810bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.396+0000 7fb107200640 1 --2- 172.21.15.43:0/2149620214 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb108104c40 0x7fb108105050 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.396+0000 7fb107200640 1 --2- 172.21.15.43:0/2149620214 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb108103a00 0x7fb108104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.396+0000 7fb107200640 1 -- 172.21.15.43:0/2149620214 >> 172.21.15.43:0/2149620214 conn(0x7fb1080febd0 msgr2=0x7fb1081010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.396+0000 7fb107200640 1 -- 172.21.15.43:0/2149620214 shutdown_connections 2024-05-27T07:13:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.396+0000 7fb107200640 1 -- 172.21.15.43:0/2149620214 wait complete. 2024-05-27T07:13:53.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.397+0000 7fb107200640 1 Processor -- start 2024-05-27T07:13:53.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.398+0000 7fb107200640 1 -- start start 2024-05-27T07:13:53.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.398+0000 7fb107200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb108103a00 0x7fb10819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.398+0000 7fb107200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb108104c40 0x7fb10819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.398+0000 7fb107200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb108109940 0x7fb1081a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.398+0000 7fb107200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb10810f400 con 0x7fb108109940 2024-05-27T07:13:53.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.398+0000 7fb107200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb10810f280 con 0x7fb108104c40 2024-05-27T07:13:53.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.398+0000 7fb107200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb10810f580 con 0x7fb108103a00 2024-05-27T07:13:53.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.398+0000 7fb106800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb108109940 0x7fb1081a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.398+0000 7fb106800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb108109940 0x7fb1081a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41036/0 (socket says 172.21.15.43:41036) 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.398+0000 7fb106800640 1 -- 172.21.15.43:0/3935481832 learned_addr learned my addr 172.21.15.43:0/3935481832 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.399+0000 7fb0ffe00640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb108104c40 0x7fb10819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.399+0000 7fb105e00640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb108103a00 0x7fb10819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.399+0000 7fb106800640 1 -- 172.21.15.43:0/3935481832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb108103a00 msgr2=0x7fb10819a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.399+0000 7fb106800640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb108103a00 0x7fb10819a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.399+0000 7fb106800640 1 -- 172.21.15.43:0/3935481832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb108104c40 msgr2=0x7fb10819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.399+0000 7fb106800640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb108104c40 0x7fb10819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.399+0000 7fb106800640 1 -- 172.21.15.43:0/3935481832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1081a1b90 con 0x7fb108109940 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.399+0000 7fb105e00640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb108103a00 0x7fb10819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.399+0000 7fb106800640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb108109940 0x7fb1081a1520 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb10000eb10 tx=0x7fb10000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.400+0000 7fb0fea00640 1 -- 172.21.15.43:0/3935481832 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100004b20 con 0x7fb108109940 2024-05-27T07:13:53.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.400+0000 7fb0fea00640 1 -- 172.21.15.43:0/3935481832 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb100004cc0 con 0x7fb108109940 2024-05-27T07:13:53.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.400+0000 7fb107200640 1 -- 172.21.15.43:0/3935481832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1081a1e80 con 0x7fb108109940 2024-05-27T07:13:53.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.400+0000 7fb0fea00640 1 -- 172.21.15.43:0/3935481832 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100010690 con 0x7fb108109940 2024-05-27T07:13:53.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.400+0000 7fb107200640 1 -- 172.21.15.43:0/3935481832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1081a2290 con 0x7fb108109940 2024-05-27T07:13:53.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.401+0000 7fb107200640 1 -- 172.21.15.43:0/3935481832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb108108750 con 0x7fb108109940 2024-05-27T07:13:53.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.405+0000 7fb0fea00640 1 -- 172.21.15.43:0/3935481832 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7fb100005000 con 0x7fb108109940 2024-05-27T07:13:53.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.408+0000 7fb0fea00640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0ec0b2830 0x7fb0ec0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.408+0000 7fb105e00640 1 -- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0ec0b2830 msgr2=0x7fb0ec0b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1809594794 2024-05-27T07:13:53.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.408+0000 7fb0fea00640 1 -- 172.21.15.43:0/3935481832 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7fb1000e2e50 con 0x7fb108109940 2024-05-27T07:13:53.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.408+0000 7fb105e00640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0ec0b2830 0x7fb0ec0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:13:53.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.409+0000 7fb0fea00640 1 -- 172.21.15.43:0/3935481832 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb100054df0 con 0x7fb108109940 2024-05-27T07:13:53.453 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:53 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:53.021+0000 7fb9f4453200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T07:13:53.453 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:53 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:53.160+0000 7fb9f4453200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T07:13:53.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:53 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2444333805' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-27T07:13:53.458 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:53.296+0000 7fb6cec25200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:13:53.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:53 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2444333805' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-27T07:13:53.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.609+0000 7fb105e00640 1 -- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0ec0b2830 msgr2=0x7fb0ec0b4d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1809594794 2024-05-27T07:13:53.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.609+0000 7fb105e00640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0ec0b2830 0x7fb0ec0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:13:53.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.620+0000 7fb107200640 1 -- 172.21.15.43:0/3935481832 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb10802ddf0 con 0x7fb108109940 2024-05-27T07:13:53.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.623+0000 7fb0fea00640 1 -- 172.21.15.43:0/3935481832 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24748 (secure 0 0 0) 0x7fb1000aa780 con 0x7fb108109940 2024-05-27T07:13:53.615 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:13:53.615 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":90,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:09:54.019617+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:13:53.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.625+0000 7fb107200640 1 -- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0ec0b2830 msgr2=0x7fb0ec0b4d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:13:53.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.625+0000 7fb107200640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0ec0b2830 0x7fb0ec0b4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.625+0000 7fb107200640 1 -- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb108109940 msgr2=0x7fb1081a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.625+0000 7fb107200640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb108109940 0x7fb1081a1520 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb10000eb10 tx=0x7fb10000c2d0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.626+0000 7fb107200640 1 -- 172.21.15.43:0/3935481832 shutdown_connections 2024-05-27T07:13:53.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.626+0000 7fb107200640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7fb0ec0b2830 0x7fb0ec0b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.626+0000 7fb107200640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb108104c40 0x7fb10819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.626+0000 7fb107200640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb108109940 0x7fb1081a1520 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.626+0000 7fb107200640 1 --2- 172.21.15.43:0/3935481832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb108103a00 0x7fb10819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.626+0000 7fb107200640 1 -- 172.21.15.43:0/3935481832 >> 172.21.15.43:0/3935481832 conn(0x7fb1080febd0 msgr2=0x7fb108105920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:53.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.626+0000 7fb107200640 1 -- 172.21.15.43:0/3935481832 shutdown_connections 2024-05-27T07:13:53.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.626+0000 7fb107200640 1 -- 172.21.15.43:0/3935481832 wait complete. 2024-05-27T07:13:53.641 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:01:14.696118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T07:08:16.033058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T07:08:20.929248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-27T07:08:34.823559+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-27T07:13:53.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T07:13:53.841 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:13:53 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y[36663]: 2024-05-27T07:13:53.564+0000 7f1fd4902200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:13:53.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.922+0000 7f94b6c00640 1 -- 172.21.15.43:0/1005712119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b0108a40 msgr2=0x7f94b010ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.922+0000 7f94b6c00640 1 --2- 172.21.15.43:0/1005712119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b0108a40 0x7f94b010ae60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009ca0 tx=0x7f94a802a8b0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.922+0000 7f94b6c00640 1 -- 172.21.15.43:0/1005712119 shutdown_connections 2024-05-27T07:13:53.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.922+0000 7f94b6c00640 1 --2- 172.21.15.43:0/1005712119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b0108a40 0x7f94b010ae60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.922+0000 7f94b6c00640 1 --2- 172.21.15.43:0/1005712119 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94b0103d40 0x7f94b0104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.922+0000 7f94b6c00640 1 --2- 172.21.15.43:0/1005712119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94b0102b00 0x7f94b0103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.922+0000 7f94b6c00640 1 -- 172.21.15.43:0/1005712119 >> 172.21.15.43:0/1005712119 conn(0x7f94b00fdcf0 msgr2=0x7f94b01001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:53.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.922+0000 7f94b6c00640 1 -- 172.21.15.43:0/1005712119 shutdown_connections 2024-05-27T07:13:53.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.922+0000 7f94b6c00640 1 -- 172.21.15.43:0/1005712119 wait complete. 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.924+0000 7f94b6c00640 1 Processor -- start 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.924+0000 7f94b6c00640 1 -- start start 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.924+0000 7f94b6c00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94b0102b00 0x7f94b0199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.924+0000 7f94b6c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b0103d40 0x7f94b0199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.924+0000 7f94b6c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94b0108a40 0x7f94b01a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.924+0000 7f94b6c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f94b010e510 con 0x7f94b0103d40 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.924+0000 7f94b6c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f94b010e390 con 0x7f94b0108a40 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.924+0000 7f94b6c00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f94b010e690 con 0x7f94b0102b00 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94af400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b0103d40 0x7f94b0199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94af400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b0103d40 0x7f94b0199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41044/0 (socket says 172.21.15.43:41044) 2024-05-27T07:13:53.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94af400640 1 -- 172.21.15.43:0/908830850 learned_addr learned my addr 172.21.15.43:0/908830850 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94afe00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94b0102b00 0x7f94b0199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94b4c00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94b0108a40 0x7f94b01a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94af400640 1 -- 172.21.15.43:0/908830850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94b0102b00 msgr2=0x7f94b0199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94af400640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94b0102b00 0x7f94b0199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94af400640 1 -- 172.21.15.43:0/908830850 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94b0108a40 msgr2=0x7f94b01a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94af400640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94b0108a40 0x7f94b01a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94af400640 1 -- 172.21.15.43:0/908830850 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b006a0b0 con 0x7f94b0103d40 2024-05-27T07:13:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94b4c00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94b0108a40 0x7f94b01a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.925+0000 7f94afe00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94b0102b00 0x7f94b0199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:13:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.926+0000 7f94af400640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b0103d40 0x7f94b0199950 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f94a000e9f0 tx=0x7f94a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:53.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.926+0000 7f94acc00640 1 -- 172.21.15.43:0/908830850 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a000cd80 con 0x7f94b0103d40 2024-05-27T07:13:53.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.926+0000 7f94acc00640 1 -- 172.21.15.43:0/908830850 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94a00108c0 con 0x7f94b0103d40 2024-05-27T07:13:53.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.926+0000 7f94b6c00640 1 -- 172.21.15.43:0/908830850 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b006a3a0 con 0x7f94b0103d40 2024-05-27T07:13:53.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.926+0000 7f94acc00640 1 -- 172.21.15.43:0/908830850 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a00187c0 con 0x7f94b0103d40 2024-05-27T07:13:53.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.926+0000 7f94b6c00640 1 -- 172.21.15.43:0/908830850 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b006a940 con 0x7f94b0103d40 2024-05-27T07:13:53.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.927+0000 7f94b6c00640 1 -- 172.21.15.43:0/908830850 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b006c4e0 con 0x7f94b0103d40 2024-05-27T07:13:53.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.930+0000 7f94acc00640 1 -- 172.21.15.43:0/908830850 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) ==== 149669+0+0 (secure 0 0 0) 0x7f94a0010430 con 0x7f94b0103d40 2024-05-27T07:13:53.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.934+0000 7f94acc00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f94940b28c0 0x7f94940b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:53.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.934+0000 7f94afe00640 1 -- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f94940b28c0 msgr2=0x7f94940b4dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1809594794 2024-05-27T07:13:53.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.934+0000 7f94acc00640 1 -- 172.21.15.43:0/908830850 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9150+0+0 (secure 0 0 0) 0x7f94a00e2f70 con 0x7f94b0103d40 2024-05-27T07:13:53.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.934+0000 7f94afe00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f94940b28c0 0x7f94940b4dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:13:53.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:53.934+0000 7f94acc00640 1 -- 172.21.15.43:0/908830850 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f94a00a6430 con 0x7f94b0103d40 2024-05-27T07:13:53.959 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:13:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z[37618]: 2024-05-27T07:13:53.587+0000 7fb6cec25200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:13:54.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.097+0000 7f94acc00640 1 -- 172.21.15.43:0/908830850 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 20) ==== 149669+0+0 (secure 0 0 0) 0x7f94a00aa400 con 0x7f94b0103d40 2024-05-27T07:13:54.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.135+0000 7f94afe00640 1 -- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f94940b28c0 msgr2=0x7f94940b4dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1809594794 2024-05-27T07:13:54.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.135+0000 7f94afe00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f94940b28c0 0x7f94940b4dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:13:54.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.208+0000 7f94b6c00640 1 -- 172.21.15.43:0/908830850 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94b006aba0 con 0x7f94b0103d40 2024-05-27T07:13:54.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.209+0000 7f94acc00640 1 -- 172.21.15.43:0/908830850 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f94a00aa8a0 con 0x7f94b0103d40 2024-05-27T07:13:54.201 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 -- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f94940b28c0 msgr2=0x7f94940b4dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f94940b28c0 0x7f94940b4dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 -- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b0103d40 msgr2=0x7f94b0199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b0103d40 0x7f94b0199950 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f94a000e9f0 tx=0x7f94a000eeb0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 -- 172.21.15.43:0/908830850 shutdown_connections 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:6800/1809594794,v1:172.21.15.43:6801/1809594794] conn(0x7f94940b28c0 0x7f94940b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94b0108a40 0x7f94b01a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b0103d40 0x7f94b0199950 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 --2- 172.21.15.43:0/908830850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94b0102b00 0x7f94b0199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 -- 172.21.15.43:0/908830850 >> 172.21.15.43:0/908830850 conn(0x7f94b00fdcf0 msgr2=0x7f94b0104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 -- 172.21.15.43:0/908830850 shutdown_connections 2024-05-27T07:13:54.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.212+0000 7f94b6c00640 1 -- 172.21.15.43:0/908830850 wait complete. 2024-05-27T07:13:54.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T07:13:54.290 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.024+0000 7fb9f4453200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2444333805' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: Standby manager daemon y restarted 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: Standby manager daemon y started 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: Standby manager daemon z restarted 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: Standby manager daemon z started 2024-05-27T07:13:54.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:13:54.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:13:54.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:13:54.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:13:54.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:54 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3935481832' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:13:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2444333805' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-27T07:13:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: Standby manager daemon y restarted 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: Standby manager daemon y started 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: Standby manager daemon z restarted 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: Standby manager daemon z started 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:13:54.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:13:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:13:54.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:54 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3935481832' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2444333805' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: Standby manager daemon y restarted 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: Standby manager daemon y started 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.132:0/336535091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: Standby manager daemon z restarted 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: Standby manager daemon z started 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-27T07:13:54.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-27T07:13:54.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-27T07:13:54.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: from='mgr.? 172.21.15.179:0/1310194995' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-27T07:13:54.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:54 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3935481832' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:13:54.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.521+0000 7f6919a00640 1 -- 172.21.15.43:0/4087632418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914106e90 msgr2=0x7f69141072a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:54.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.521+0000 7f6919a00640 1 --2- 172.21.15.43:0/4087632418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914106e90 0x7f69141072a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6900009900 tx=0x7f690002a400 comp rx=0 tx=0).stop 2024-05-27T07:13:54.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.522+0000 7f6919a00640 1 -- 172.21.15.43:0/4087632418 shutdown_connections 2024-05-27T07:13:54.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.522+0000 7f6919a00640 1 --2- 172.21.15.43:0/4087632418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f691410bb90 0x7f691410dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.522+0000 7f6919a00640 1 --2- 172.21.15.43:0/4087632418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914106e90 0x7f69141072a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.522+0000 7f6919a00640 1 --2- 172.21.15.43:0/4087632418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6914105c50 0x7f6914106950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.522+0000 7f6919a00640 1 -- 172.21.15.43:0/4087632418 >> 172.21.15.43:0/4087632418 conn(0x7f6914100e40 msgr2=0x7f69141032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:54.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.522+0000 7f6919a00640 1 -- 172.21.15.43:0/4087632418 shutdown_connections 2024-05-27T07:13:54.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.522+0000 7f6919a00640 1 -- 172.21.15.43:0/4087632418 wait complete. 2024-05-27T07:13:54.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.523+0000 7f6919a00640 1 Processor -- start 2024-05-27T07:13:54.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.523+0000 7f6919a00640 1 -- start start 2024-05-27T07:13:54.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.523+0000 7f6919a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105c50 0x7f691419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:54.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6919a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914106e90 0x7f691419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:54.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6919a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410bb90 0x7f69141a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:54.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6919a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6914111600 con 0x7f6914105c50 2024-05-27T07:13:54.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6919a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6914111480 con 0x7f691410bb90 2024-05-27T07:13:54.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6919a00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6914111780 con 0x7f6914106e90 2024-05-27T07:13:54.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6913e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105c50 0x7f691419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:54.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6913e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105c50 0x7f691419c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49126/0 (socket says 172.21.15.43:49126) 2024-05-27T07:13:54.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6913e00640 1 -- 172.21.15.43:0/2566954755 learned_addr learned my addr 172.21.15.43:0/2566954755 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:54.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6913400640 1 --2- 172.21.15.43:0/2566954755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914106e90 0x7f691419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:54.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6919000640 1 --2- 172.21.15.43:0/2566954755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410bb90 0x7f69141a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:54.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6913e00640 1 -- 172.21.15.43:0/2566954755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914106e90 msgr2=0x7f691419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:54.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6913e00640 1 --2- 172.21.15.43:0/2566954755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914106e90 0x7f691419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6913e00640 1 -- 172.21.15.43:0/2566954755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410bb90 msgr2=0x7f69141a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6913e00640 1 --2- 172.21.15.43:0/2566954755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410bb90 0x7f69141a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6913e00640 1 -- 172.21.15.43:0/2566954755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69141a3d30 con 0x7f6914105c50 2024-05-27T07:13:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.524+0000 7f6919000640 1 --2- 172.21.15.43:0/2566954755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410bb90 0x7f69141a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.525+0000 7f6913e00640 1 --2- 172.21.15.43:0/2566954755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105c50 0x7f691419c2a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6904005fd0 tx=0x7f690400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.525+0000 7f6910c00640 1 -- 172.21.15.43:0/2566954755 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6904017070 con 0x7f6914105c50 2024-05-27T07:13:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.525+0000 7f6910c00640 1 -- 172.21.15.43:0/2566954755 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6904009c30 con 0x7f6914105c50 2024-05-27T07:13:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.525+0000 7f6919a00640 1 -- 172.21.15.43:0/2566954755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69141a3fc0 con 0x7f6914105c50 2024-05-27T07:13:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.525+0000 7f6910c00640 1 -- 172.21.15.43:0/2566954755 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6904003c90 con 0x7f6914105c50 2024-05-27T07:13:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.525+0000 7f6919a00640 1 -- 172.21.15.43:0/2566954755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69141a44d0 con 0x7f6914105c50 2024-05-27T07:13:54.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.527+0000 7f6919a00640 1 -- 172.21.15.43:0/2566954755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f40052c0 con 0x7f6914105c50 2024-05-27T07:13:54.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.530+0000 7f6910c00640 1 -- 172.21.15.43:0/2566954755 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 21) ==== 149337+0+0 (secure 0 0 0) 0x7f6904003e30 con 0x7f6914105c50 2024-05-27T07:13:54.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.532+0000 7f6910c00640 1 -- 172.21.15.43:0/2566954755 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f69040e24d0 con 0x7f6914105c50 2024-05-27T07:13:54.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.538+0000 7f6910c00640 1 -- 172.21.15.43:0/2566954755 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6904020030 con 0x7f6914105c50 2024-05-27T07:13:54.631 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.384+0000 7fb9f4453200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T07:13:54.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.854+0000 7f6919a00640 1 -- 172.21.15.43:0/2566954755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f68f4005880 con 0x7f6914105c50 2024-05-27T07:13:54.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.855+0000 7f6910c00640 1 -- 172.21.15.43:0/2566954755 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f69040aa1a0 con 0x7f6914105c50 2024-05-27T07:13:54.847 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-05-27T07:13:54.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.857+0000 7f6919a00640 1 -- 172.21.15.43:0/2566954755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105c50 msgr2=0x7f691419c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:54.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.857+0000 7f6919a00640 1 --2- 172.21.15.43:0/2566954755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105c50 0x7f691419c2a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6904005fd0 tx=0x7f690400dde0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.857+0000 7f6919a00640 1 -- 172.21.15.43:0/2566954755 shutdown_connections 2024-05-27T07:13:54.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.857+0000 7f6919a00640 1 --2- 172.21.15.43:0/2566954755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f691410bb90 0x7f69141a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.857+0000 7f6919a00640 1 --2- 172.21.15.43:0/2566954755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6914106e90 0x7f691419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.857+0000 7f6919a00640 1 --2- 172.21.15.43:0/2566954755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6914105c50 0x7f691419c2a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:54.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.857+0000 7f6919a00640 1 -- 172.21.15.43:0/2566954755 >> 172.21.15.43:0/2566954755 conn(0x7f6914100e40 msgr2=0x7f6914107b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:54.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.858+0000 7f6919a00640 1 -- 172.21.15.43:0/2566954755 shutdown_connections 2024-05-27T07:13:54.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:54.858+0000 7f6919a00640 1 -- 172.21.15.43:0/2566954755 wait complete. 2024-05-27T07:13:54.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T07:13:55.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.150+0000 7fbd77200640 1 -- 172.21.15.43:0/1587846635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd78104c70 msgr2=0x7fbd78105080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.150+0000 7fbd77200640 1 --2- 172.21.15.43:0/1587846635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd78104c70 0x7fbd78105080 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009900 tx=0x7fbd6402a410 comp rx=0 tx=0).stop 2024-05-27T07:13:55.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.150+0000 7fbd77200640 1 -- 172.21.15.43:0/1587846635 shutdown_connections 2024-05-27T07:13:55.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.150+0000 7fbd77200640 1 --2- 172.21.15.43:0/1587846635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd78109970 0x7fbd7810bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.150+0000 7fbd77200640 1 --2- 172.21.15.43:0/1587846635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd78104c70 0x7fbd78105080 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.150+0000 7fbd77200640 1 --2- 172.21.15.43:0/1587846635 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78103a30 0x7fbd78104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.150+0000 7fbd77200640 1 -- 172.21.15.43:0/1587846635 >> 172.21.15.43:0/1587846635 conn(0x7fbd780fec40 msgr2=0x7fbd781010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:55.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.151+0000 7fbd77200640 1 -- 172.21.15.43:0/1587846635 shutdown_connections 2024-05-27T07:13:55.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.151+0000 7fbd77200640 1 -- 172.21.15.43:0/1587846635 wait complete. 2024-05-27T07:13:55.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd77200640 1 Processor -- start 2024-05-27T07:13:55.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd77200640 1 -- start start 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd77200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78103a30 0x7fbd7819c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd77200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd78104c70 0x7fbd7819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd77200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd78109970 0x7fbd781a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd77200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7810f400 con 0x7fbd78109970 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd77200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7810f280 con 0x7fbd78104c70 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd77200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7810f580 con 0x7fbd78103a30 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd76800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd78109970 0x7fbd781a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd76800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd78109970 0x7fbd781a3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49216/0 (socket says 172.21.15.43:49216) 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.152+0000 7fbd76800640 1 -- 172.21.15.43:0/1459045714 learned_addr learned my addr 172.21.15.43:0/1459045714 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.153+0000 7fbd75e00640 1 --2- 172.21.15.43:0/1459045714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78103a30 0x7fbd7819c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.153+0000 7fbd75400640 1 --2- 172.21.15.43:0/1459045714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd78104c70 0x7fbd7819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:55.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.153+0000 7fbd76800640 1 -- 172.21.15.43:0/1459045714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78103a30 msgr2=0x7fbd7819c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.153+0000 7fbd76800640 1 --2- 172.21.15.43:0/1459045714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78103a30 0x7fbd7819c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.153+0000 7fbd76800640 1 -- 172.21.15.43:0/1459045714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd78104c70 msgr2=0x7fbd7819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.153+0000 7fbd76800640 1 --2- 172.21.15.43:0/1459045714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd78104c70 0x7fbd7819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.153+0000 7fbd76800640 1 -- 172.21.15.43:0/1459045714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd781a3dd0 con 0x7fbd78109970 2024-05-27T07:13:55.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.153+0000 7fbd76800640 1 --2- 172.21.15.43:0/1459045714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd78109970 0x7fbd781a3760 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbd7000eb10 tx=0x7fbd7000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:55.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.154+0000 7fbd6ea00640 1 -- 172.21.15.43:0/1459045714 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70004b20 con 0x7fbd78109970 2024-05-27T07:13:55.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.154+0000 7fbd77200640 1 -- 172.21.15.43:0/1459045714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd781a40c0 con 0x7fbd78109970 2024-05-27T07:13:55.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.154+0000 7fbd6ea00640 1 -- 172.21.15.43:0/1459045714 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd70004cc0 con 0x7fbd78109970 2024-05-27T07:13:55.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.154+0000 7fbd77200640 1 -- 172.21.15.43:0/1459045714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd781a4610 con 0x7fbd78109970 2024-05-27T07:13:55.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.154+0000 7fbd6ea00640 1 -- 172.21.15.43:0/1459045714 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70010690 con 0x7fbd78109970 2024-05-27T07:13:55.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.156+0000 7fbd77200640 1 -- 172.21.15.43:0/1459045714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd580052c0 con 0x7fbd78109970 2024-05-27T07:13:55.149 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: mgrmap e20: x(active, since 16m), standbys: z, y 2024-05-27T07:13:55.149 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/908830850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:13:55.149 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: Active manager daemon x restarted 2024-05-27T07:13:55.149 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: Activating manager daemon x 2024-05-27T07:13:55.149 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: mgrmap e21: x(active, starting, since 0.0212419s), standbys: z, y 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:13:55.150 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:13:55.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:13:55.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:13:55.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:13:55.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:13:55.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:13:55.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:13:55.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:13:55.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:13:55.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:13:55.151 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:13:55.152 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:13:55.152 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:13:55.152 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:13:55.152 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: Manager daemon x is now available 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:13:55.153 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:55 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2566954755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:13:55.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.158+0000 7fbd6ea00640 1 -- 172.21.15.43:0/1459045714 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 21) ==== 149337+0+0 (secure 0 0 0) 0x7fbd70005020 con 0x7fbd78109970 2024-05-27T07:13:55.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.161+0000 7fbd6ea00640 1 -- 172.21.15.43:0/1459045714 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fbd700e2540 con 0x7fbd78109970 2024-05-27T07:13:55.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.164+0000 7fbd6ea00640 1 -- 172.21.15.43:0/1459045714 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbd700aa1d0 con 0x7fbd78109970 2024-05-27T07:13:55.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: mgrmap e20: x(active, since 16m), standbys: z, y 2024-05-27T07:13:55.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/908830850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:13:55.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: Active manager daemon x restarted 2024-05-27T07:13:55.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: Activating manager daemon x 2024-05-27T07:13:55.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:13:55.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: mgrmap e21: x(active, starting, since 0.0212419s), standbys: z, y 2024-05-27T07:13:55.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:13:55.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.987+0000 7fb944c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.987+0000 7fb944c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.987+0000 7fb944c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.987+0000 7fb944c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:13:55.205 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.987+0000 7fb944c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:13:55.206 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.987+0000 7fb948800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:13:55.206 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.987+0000 7fb948800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:13:55.206 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.987+0000 7fb948800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:13:55.206 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.987+0000 7fb948800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:13:55.206 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:13:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:13:54.987+0000 7fb948800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:13:55.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:13:55.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:13:55.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:13:55.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: Manager daemon x is now available 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:13:55.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:55 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2566954755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:13:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.376+0000 7fbd77200640 1 -- 172.21.15.43:0/1459045714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbd58005dc0 con 0x7fbd78109970 2024-05-27T07:13:55.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.377+0000 7fbd6ea00640 1 -- 172.21.15.43:0/1459045714 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fbd700b2250 con 0x7fbd78109970 2024-05-27T07:13:55.369 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-05-27T07:13:55.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.380+0000 7fbd77200640 1 -- 172.21.15.43:0/1459045714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd78109970 msgr2=0x7fbd781a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.380+0000 7fbd77200640 1 --2- 172.21.15.43:0/1459045714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd78109970 0x7fbd781a3760 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbd7000eb10 tx=0x7fbd7000c2d0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.380+0000 7fbd77200640 1 -- 172.21.15.43:0/1459045714 shutdown_connections 2024-05-27T07:13:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.380+0000 7fbd77200640 1 --2- 172.21.15.43:0/1459045714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd78104c70 0x7fbd7819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.380+0000 7fbd77200640 1 --2- 172.21.15.43:0/1459045714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd78109970 0x7fbd781a3760 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.380+0000 7fbd77200640 1 --2- 172.21.15.43:0/1459045714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78103a30 0x7fbd7819c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.380+0000 7fbd77200640 1 -- 172.21.15.43:0/1459045714 >> 172.21.15.43:0/1459045714 conn(0x7fbd780fec40 msgr2=0x7fbd780ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.380+0000 7fbd77200640 1 -- 172.21.15.43:0/1459045714 shutdown_connections 2024-05-27T07:13:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.380+0000 7fbd77200640 1 -- 172.21.15.43:0/1459045714 wait complete. 2024-05-27T07:13:55.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-27T07:13:55.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: mgrmap e20: x(active, since 16m), standbys: z, y 2024-05-27T07:13:55.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/908830850' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:13:55.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: Active manager daemon x restarted 2024-05-27T07:13:55.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: Activating manager daemon x 2024-05-27T07:13:55.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: osdmap e91: 12 total, 12 up, 12 in 2024-05-27T07:13:55.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: mgrmap e21: x(active, starting, since 0.0212419s), standbys: z, y 2024-05-27T07:13:55.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-27T07:13:55.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-27T07:13:55.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-27T07:13:55.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-27T07:13:55.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-27T07:13:55.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-27T07:13:55.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-27T07:13:55.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-27T07:13:55.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-27T07:13:55.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-27T07:13:55.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: Manager daemon x is now available 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:13:55.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-27T07:13:55.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-27T07:13:55.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:55 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2566954755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:13:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.664+0000 7f4137800640 1 -- 172.21.15.43:0/410190876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4130104e40 msgr2=0x7f4130105250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.664+0000 7f4137800640 1 --2- 172.21.15.43:0/410190876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4130104e40 0x7f4130105250 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f411c009900 tx=0x7f411c02a410 comp rx=0 tx=0).stop 2024-05-27T07:13:55.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.664+0000 7f4137800640 1 -- 172.21.15.43:0/410190876 shutdown_connections 2024-05-27T07:13:55.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.664+0000 7f4137800640 1 --2- 172.21.15.43:0/410190876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4130109b40 0x7f413010bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.664+0000 7f4137800640 1 --2- 172.21.15.43:0/410190876 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4130104e40 0x7f4130105250 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.664+0000 7f4137800640 1 --2- 172.21.15.43:0/410190876 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4130103c00 0x7f4130104900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.664+0000 7f4137800640 1 -- 172.21.15.43:0/410190876 >> 172.21.15.43:0/410190876 conn(0x7f41300fedf0 msgr2=0x7f41301012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:55.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.664+0000 7f4137800640 1 -- 172.21.15.43:0/410190876 shutdown_connections 2024-05-27T07:13:55.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.664+0000 7f4137800640 1 -- 172.21.15.43:0/410190876 wait complete. 2024-05-27T07:13:55.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.665+0000 7f4137800640 1 Processor -- start 2024-05-27T07:13:55.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.665+0000 7f4137800640 1 -- start start 2024-05-27T07:13:55.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4137800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4130103c00 0x7f413019c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:55.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4137800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4130104e40 0x7f413019c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:55.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4137800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4130109b40 0x7f41301a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:55.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4137800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f413010f5d0 con 0x7f4130103c00 2024-05-27T07:13:55.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4137800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f413010f450 con 0x7f4130109b40 2024-05-27T07:13:55.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4137800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f413010f750 con 0x7f4130104e40 2024-05-27T07:13:55.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4134e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4130103c00 0x7f413019c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:55.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4134e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4130103c00 0x7f413019c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49232/0 (socket says 172.21.15.43:49232) 2024-05-27T07:13:55.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4134e00640 1 -- 172.21.15.43:0/999416781 learned_addr learned my addr 172.21.15.43:0/999416781 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:55.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4135800640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4130109b40 0x7f41301a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:55.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f412fe00640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4130104e40 0x7f413019c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:55.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4134e00640 1 -- 172.21.15.43:0/999416781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4130104e40 msgr2=0x7f413019c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4134e00640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4130104e40 0x7f413019c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4134e00640 1 -- 172.21.15.43:0/999416781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4130109b40 msgr2=0x7f41301a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4134e00640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4130109b40 0x7f41301a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4134e00640 1 -- 172.21.15.43:0/999416781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41301a3fc0 con 0x7f4130103c00 2024-05-27T07:13:55.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f4135800640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4130109b40 0x7f41301a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:55.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.666+0000 7f412fe00640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4130104e40 0x7f413019c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:13:55.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.667+0000 7f4134e00640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4130103c00 0x7f413019c480 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4124005d00 tx=0x7f412400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:55.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.667+0000 7f412d600640 1 -- 172.21.15.43:0/999416781 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124017070 con 0x7f4130103c00 2024-05-27T07:13:55.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.667+0000 7f412d600640 1 -- 172.21.15.43:0/999416781 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4124009c30 con 0x7f4130103c00 2024-05-27T07:13:55.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.667+0000 7f412d600640 1 -- 172.21.15.43:0/999416781 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412400f620 con 0x7f4130103c00 2024-05-27T07:13:55.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.667+0000 7f4137800640 1 -- 172.21.15.43:0/999416781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41301a42b0 con 0x7f4130103c00 2024-05-27T07:13:55.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.667+0000 7f4137800640 1 -- 172.21.15.43:0/999416781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41301a47b0 con 0x7f4130103c00 2024-05-27T07:13:55.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.668+0000 7f4137800640 1 -- 172.21.15.43:0/999416781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41301a4a10 con 0x7f4130103c00 2024-05-27T07:13:55.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.671+0000 7f412d600640 1 -- 172.21.15.43:0/999416781 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 22) ==== 149464+0+0 (secure 0 0 0) 0x7f412401a070 con 0x7f4130103c00 2024-05-27T07:13:55.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.672+0000 7f412d600640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41180b25e0 0x7f41180b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:55.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.673+0000 7f412d600640 1 -- 172.21.15.43:0/999416781 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f41240e1d60 con 0x7f4130103c00 2024-05-27T07:13:55.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.673+0000 7f412fe00640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41180b25e0 0x7f41180b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:55.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.674+0000 7f412fe00640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41180b25e0 0x7f41180b4af0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f411c005f70 tx=0x7f411c005720 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:55.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.682+0000 7f412d600640 1 -- 172.21.15.43:0/999416781 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4124009de0 con 0x7f4130103c00 2024-05-27T07:13:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.912+0000 7f4137800640 1 -- 172.21.15.43:0/999416781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f41301a4be0 con 0x7f4130103c00 2024-05-27T07:13:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.912+0000 7f412d600640 1 -- 172.21.15.43:0/999416781 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f41240a98c0 con 0x7f4130103c00 2024-05-27T07:13:55.904 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-05-27T07:13:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 -- 172.21.15.43:0/999416781 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41180b25e0 msgr2=0x7f41180b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41180b25e0 0x7f41180b4af0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f411c005f70 tx=0x7f411c005720 comp rx=0 tx=0).stop 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 -- 172.21.15.43:0/999416781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4130103c00 msgr2=0x7f413019c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4130103c00 0x7f413019c480 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4124005d00 tx=0x7f412400dde0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 -- 172.21.15.43:0/999416781 shutdown_connections 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41180b25e0 0x7f41180b4af0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4130109b40 0x7f41301a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4130104e40 0x7f413019c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 --2- 172.21.15.43:0/999416781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4130103c00 0x7f413019c480 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 -- 172.21.15.43:0/999416781 >> 172.21.15.43:0/999416781 conn(0x7f41300fedf0 msgr2=0x7f41300ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 -- 172.21.15.43:0/999416781 shutdown_connections 2024-05-27T07:13:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:55.915+0000 7f4137800640 1 -- 172.21.15.43:0/999416781 wait complete. 2024-05-27T07:13:55.940 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-27T07:13:55.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:13:56.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:56 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1459045714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:13:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:56 smithi043 ceph-mon[31531]: mgrmap e22: x(active, since 1.03441s), standbys: z, y 2024-05-27T07:13:56.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:56 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/999416781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:13:56.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:56 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1459045714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:13:56.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:56 smithi132 ceph-mon[35283]: mgrmap e22: x(active, since 1.03441s), standbys: z, y 2024-05-27T07:13:56.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:56 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/999416781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:13:56.380 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:56 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1459045714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:13:56.380 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:56 smithi179 ceph-mon[35888]: mgrmap e22: x(active, since 1.03441s), standbys: z, y 2024-05-27T07:13:56.380 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:56 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/999416781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:13:56.515 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:13:57.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:57 smithi043 ceph-mon[31531]: [27/May/2024:07:13:55] ENGINE Bus STARTING 2024-05-27T07:13:57.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:57 smithi043 ceph-mon[31531]: [27/May/2024:07:13:56] ENGINE Serving on http://172.21.15.43:8765 2024-05-27T07:13:57.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:57 smithi043 ceph-mon[31531]: [27/May/2024:07:13:56] ENGINE Serving on https://172.21.15.43:7150 2024-05-27T07:13:57.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:57 smithi043 ceph-mon[31531]: [27/May/2024:07:13:56] ENGINE Bus STARTED 2024-05-27T07:13:57.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:57 smithi132 ceph-mon[35283]: [27/May/2024:07:13:55] ENGINE Bus STARTING 2024-05-27T07:13:57.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:57 smithi132 ceph-mon[35283]: [27/May/2024:07:13:56] ENGINE Serving on http://172.21.15.43:8765 2024-05-27T07:13:57.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:57 smithi132 ceph-mon[35283]: [27/May/2024:07:13:56] ENGINE Serving on https://172.21.15.43:7150 2024-05-27T07:13:57.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:57 smithi132 ceph-mon[35283]: [27/May/2024:07:13:56] ENGINE Bus STARTED 2024-05-27T07:13:57.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:57 smithi179 ceph-mon[35888]: [27/May/2024:07:13:55] ENGINE Bus STARTING 2024-05-27T07:13:57.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:57 smithi179 ceph-mon[35888]: [27/May/2024:07:13:56] ENGINE Serving on http://172.21.15.43:8765 2024-05-27T07:13:57.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:57 smithi179 ceph-mon[35888]: [27/May/2024:07:13:56] ENGINE Serving on https://172.21.15.43:7150 2024-05-27T07:13:57.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:57 smithi179 ceph-mon[35888]: [27/May/2024:07:13:56] ENGINE Bus STARTED 2024-05-27T07:13:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:58 smithi043 ceph-mon[31531]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:13:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:58 smithi043 ceph-mon[31531]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-27T07:13:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:58 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:58 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:58 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:13:58 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:58 smithi179 ceph-mon[35888]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:13:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:58 smithi179 ceph-mon[35888]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-27T07:13:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:58 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:58 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:58 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:13:58 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:58 smithi132 ceph-mon[35283]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:13:58.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:58 smithi132 ceph-mon[35283]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-05-27T07:13:58.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:58 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:58 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:58 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:58.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:13:58 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:13:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.220+0000 7f2465600700 1 -- 172.21.15.43:0/991604091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f246010e700 msgr2=0x7f2460110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.220+0000 7f2465600700 1 --2- 172.21.15.43:0/991604091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f246010e700 0x7f2460110be0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2458009f90 tx=0x7f245800b330 comp rx=0 tx=0).stop 2024-05-27T07:13:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.220+0000 7f2465600700 1 -- 172.21.15.43:0/991604091 shutdown_connections 2024-05-27T07:13:59.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.220+0000 7f2465600700 1 --2- 172.21.15.43:0/991604091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f246010e700 0x7f2460110be0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.220+0000 7f2465600700 1 --2- 172.21.15.43:0/991604091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2460105320 0x7f2460105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.220+0000 7f2465600700 1 --2- 172.21.15.43:0/991604091 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2460105c70 0x7f246010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.220+0000 7f2465600700 1 -- 172.21.15.43:0/991604091 >> 172.21.15.43:0/991604091 conn(0x7f2460078ab0 msgr2=0x7f2460078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.221+0000 7f2465600700 1 -- 172.21.15.43:0/991604091 shutdown_connections 2024-05-27T07:13:59.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.221+0000 7f2465600700 1 -- 172.21.15.43:0/991604091 wait complete. 2024-05-27T07:13:59.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.222+0000 7f2465600700 1 Processor -- start 2024-05-27T07:13:59.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.222+0000 7f2465600700 1 -- start start 2024-05-27T07:13:59.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f2465600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2460105320 0x7f246019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:59.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f2465600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2460105c70 0x7f246019d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:59.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f2465600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f246010e700 0x7f24601a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:59.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f2465600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2460114100 con 0x7f2460105c70 2024-05-27T07:13:59.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f2465600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2460113f80 con 0x7f2460105320 2024-05-27T07:13:59.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f2465600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2460114280 con 0x7f246010e700 2024-05-27T07:13:59.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f245f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2460105c70 0x7f246019d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:59.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f245f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2460105c70 0x7f246019d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49282/0 (socket says 172.21.15.43:49282) 2024-05-27T07:13:59.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f245f400700 1 -- 172.21.15.43:0/802665225 learned_addr learned my addr 172.21.15.43:0/802665225 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:13:59.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f245fe00700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2460105320 0x7f246019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:59.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.223+0000 7f2464c00700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f246010e700 0x7f24601a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:59.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.224+0000 7f245f400700 1 -- 172.21.15.43:0/802665225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f246010e700 msgr2=0x7f24601a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:59.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.224+0000 7f245f400700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f246010e700 0x7f24601a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:59.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.224+0000 7f245f400700 1 -- 172.21.15.43:0/802665225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2460105320 msgr2=0x7f246019cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:59.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.224+0000 7f245f400700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2460105320 0x7f246019cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:59.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.224+0000 7f245f400700 1 -- 172.21.15.43:0/802665225 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24601a1be0 con 0x7f2460105c70 2024-05-27T07:13:59.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.224+0000 7f2464c00700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f246010e700 0x7f24601a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:13:59.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.224+0000 7f245f400700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2460105c70 0x7f246019d2b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f244c00e9c0 tx=0x7f244c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:59.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.225+0000 7f245cc00700 1 -- 172.21.15.43:0/802665225 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c004a30 con 0x7f2460105c70 2024-05-27T07:13:59.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.225+0000 7f245cc00700 1 -- 172.21.15.43:0/802665225 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f244c00cc50 con 0x7f2460105c70 2024-05-27T07:13:59.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.225+0000 7f2465600700 1 -- 172.21.15.43:0/802665225 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24601a1ed0 con 0x7f2460105c70 2024-05-27T07:13:59.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.225+0000 7f245cc00700 1 -- 172.21.15.43:0/802665225 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c005420 con 0x7f2460105c70 2024-05-27T07:13:59.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.225+0000 7f2465600700 1 -- 172.21.15.43:0/802665225 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24601a97f0 con 0x7f2460105c70 2024-05-27T07:13:59.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.226+0000 7f2465600700 1 -- 172.21.15.43:0/802665225 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24600512b0 con 0x7f2460105c70 2024-05-27T07:13:59.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.231+0000 7f245cc00700 1 -- 172.21.15.43:0/802665225 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f244c0055c0 con 0x7f2460105c70 2024-05-27T07:13:59.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.233+0000 7f245cc00700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24440b2a80 0x7f24440b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:13:59.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.233+0000 7f245cc00700 1 -- 172.21.15.43:0/802665225 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f244c0e2d80 con 0x7f2460105c70 2024-05-27T07:13:59.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.233+0000 7f245fe00700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24440b2a80 0x7f24440b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:13:59.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.234+0000 7f245cc00700 1 -- 172.21.15.43:0/802665225 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f244c0aa240 con 0x7f2460105c70 2024-05-27T07:13:59.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.235+0000 7f245fe00700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24440b2a80 0x7f24440b4f90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2450009c60 tx=0x7f2450009260 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:13:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.544+0000 7f2465600700 1 -- 172.21.15.43:0/802665225 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f246019e070 con 0x7f2460105c70 2024-05-27T07:13:59.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.545+0000 7f245cc00700 1 -- 172.21.15.43:0/802665225 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11938 (secure 0 0 0) 0x7f244c077c10 con 0x7f2460105c70 2024-05-27T07:13:59.537 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:13:59.537 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:13:52.990426+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34484,34448,34424,24484,34403]},"id":1}]} 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 -- 172.21.15.43:0/802665225 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24440b2a80 msgr2=0x7f24440b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24440b2a80 0x7f24440b4f90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2450009c60 tx=0x7f2450009260 comp rx=0 tx=0).stop 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 -- 172.21.15.43:0/802665225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2460105c70 msgr2=0x7f246019d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2460105c70 0x7f246019d2b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f244c00e9c0 tx=0x7f244c00ecd0 comp rx=0 tx=0).stop 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 -- 172.21.15.43:0/802665225 shutdown_connections 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24440b2a80 0x7f24440b4f90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f246010e700 0x7f24601a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2460105320 0x7f246019cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 --2- 172.21.15.43:0/802665225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2460105c70 0x7f246019d2b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 -- 172.21.15.43:0/802665225 >> 172.21.15.43:0/802665225 conn(0x7f2460078ab0 msgr2=0x7f2460102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 -- 172.21.15.43:0/802665225 shutdown_connections 2024-05-27T07:13:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:13:59.549+0000 7f2465600700 1 -- 172.21.15.43:0/802665225 wait complete. 2024-05-27T07:13:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 30 2024-05-27T07:14:00.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:14:00.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-27T07:14:00.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/802665225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:14:00.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:00.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:00.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:14:00.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-27T07:14:00.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/802665225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:14:00.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:00.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:00.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:14:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-05-27T07:14:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/802665225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:14:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:00.619 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-27T07:14:00.619 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-27T07:14:00.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:14:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.911+0000 7fe2cb200640 1 -- 172.21.15.43:0/1508987981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2cc109940 msgr2=0x7fe2cc10bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.911+0000 7fe2cb200640 1 --2- 172.21.15.43:0/1508987981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2cc109940 0x7fe2cc10bd60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009ca0 tx=0x7fe2c402a8c0 comp rx=0 tx=0).stop 2024-05-27T07:14:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.911+0000 7fe2cb200640 1 -- 172.21.15.43:0/1508987981 shutdown_connections 2024-05-27T07:14:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.911+0000 7fe2cb200640 1 --2- 172.21.15.43:0/1508987981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2cc109940 0x7fe2cc10bd60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.911+0000 7fe2cb200640 1 --2- 172.21.15.43:0/1508987981 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2cc104c40 0x7fe2cc105050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.911+0000 7fe2cb200640 1 --2- 172.21.15.43:0/1508987981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2cc103a00 0x7fe2cc104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.911+0000 7fe2cb200640 1 -- 172.21.15.43:0/1508987981 >> 172.21.15.43:0/1508987981 conn(0x7fe2cc0febd0 msgr2=0x7fe2cc1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.912+0000 7fe2cb200640 1 -- 172.21.15.43:0/1508987981 shutdown_connections 2024-05-27T07:14:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.912+0000 7fe2cb200640 1 -- 172.21.15.43:0/1508987981 wait complete. 2024-05-27T07:14:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.913+0000 7fe2cb200640 1 Processor -- start 2024-05-27T07:14:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.913+0000 7fe2cb200640 1 -- start start 2024-05-27T07:14:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.913+0000 7fe2cb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2cc103a00 0x7fe2cc19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.913+0000 7fe2cb200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2cc104c40 0x7fe2cc19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.913+0000 7fe2cb200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2cc109940 0x7fe2cc1a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.913+0000 7fe2cb200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe2cc10f430 con 0x7fe2cc103a00 2024-05-27T07:14:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.913+0000 7fe2cb200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2cc10f2b0 con 0x7fe2cc104c40 2024-05-27T07:14:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.913+0000 7fe2cb200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe2cc10f5b0 con 0x7fe2cc109940 2024-05-27T07:14:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.914+0000 7fe2c9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2cc103a00 0x7fe2cc19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.914+0000 7fe2c9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2cc103a00 0x7fe2cc19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49300/0 (socket says 172.21.15.43:49300) 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.914+0000 7fe2c9e00640 1 -- 172.21.15.43:0/34111623 learned_addr learned my addr 172.21.15.43:0/34111623 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.914+0000 7fe2c3e00640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2cc104c40 0x7fe2cc19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.914+0000 7fe2ca800640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2cc109940 0x7fe2cc1a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.914+0000 7fe2c9e00640 1 -- 172.21.15.43:0/34111623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2cc109940 msgr2=0x7fe2cc1a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.914+0000 7fe2c9e00640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2cc109940 0x7fe2cc1a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.914+0000 7fe2c9e00640 1 -- 172.21.15.43:0/34111623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2cc104c40 msgr2=0x7fe2cc19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.914+0000 7fe2c9e00640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2cc104c40 0x7fe2cc19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.914+0000 7fe2c9e00640 1 -- 172.21.15.43:0/34111623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2cc1a3dc0 con 0x7fe2cc103a00 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.915+0000 7fe2c9e00640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2cc103a00 0x7fe2cc19c280 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc005fd0 tx=0x7fe2bc00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.915+0000 7fe2c2a00640 1 -- 172.21.15.43:0/34111623 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc017070 con 0x7fe2cc103a00 2024-05-27T07:14:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.915+0000 7fe2c2a00640 1 -- 172.21.15.43:0/34111623 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2bc009c30 con 0x7fe2cc103a00 2024-05-27T07:14:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.915+0000 7fe2cb200640 1 -- 172.21.15.43:0/34111623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2cc1a40b0 con 0x7fe2cc103a00 2024-05-27T07:14:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.915+0000 7fe2c2a00640 1 -- 172.21.15.43:0/34111623 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc003c90 con 0x7fe2cc103a00 2024-05-27T07:14:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.915+0000 7fe2cb200640 1 -- 172.21.15.43:0/34111623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2cc1a4600 con 0x7fe2cc103a00 2024-05-27T07:14:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.917+0000 7fe2cb200640 1 -- 172.21.15.43:0/34111623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2cc1a4860 con 0x7fe2cc103a00 2024-05-27T07:14:00.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.919+0000 7fe2c2a00640 1 -- 172.21.15.43:0/34111623 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7fe2bc003e30 con 0x7fe2cc103a00 2024-05-27T07:14:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.921+0000 7fe2c2a00640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2b00b29e0 0x7fe2b00b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.921+0000 7fe2c2a00640 1 -- 172.21.15.43:0/34111623 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fe2bc0e3130 con 0x7fe2cc103a00 2024-05-27T07:14:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.922+0000 7fe2c3e00640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2b00b29e0 0x7fe2b00b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.923+0000 7fe2c3e00640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2b00b29e0 0x7fe2b00b4ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4005fd0 tx=0x7fe2b4005cf0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:00.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:00.924+0000 7fe2c2a00640 1 -- 172.21.15.43:0/34111623 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe2bc020030 con 0x7fe2cc103a00 2024-05-27T07:14:01.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.145+0000 7fe2cb200640 1 -- 172.21.15.43:0/34111623 --> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe2cc100700 con 0x7fe2b00b29e0 2024-05-27T07:14:01.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.163+0000 7fe2c2a00640 1 -- 172.21.15.43:0/34111623 <== mgr.35033 v2:172.21.15.43:6800/2368318638 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339527 (secure 0 0 0) 0x7fe2cc100700 con 0x7fe2b00b29e0 2024-05-27T07:14:01.162 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:14:01.165 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-05-27T07:14:01.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.169+0000 7fe2cb200640 1 -- 172.21.15.43:0/34111623 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2b00b29e0 msgr2=0x7fe2b00b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:01.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.169+0000 7fe2cb200640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2b00b29e0 0x7fe2b00b4ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4005fd0 tx=0x7fe2b4005cf0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.169+0000 7fe2cb200640 1 -- 172.21.15.43:0/34111623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2cc103a00 msgr2=0x7fe2cc19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:01.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.169+0000 7fe2cb200640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2cc103a00 0x7fe2cc19c280 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc005fd0 tx=0x7fe2bc00dde0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.169+0000 7fe2cb200640 1 -- 172.21.15.43:0/34111623 shutdown_connections 2024-05-27T07:14:01.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.169+0000 7fe2cb200640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2b00b29e0 0x7fe2b00b4ef0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.169+0000 7fe2cb200640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2cc104c40 0x7fe2cc19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.169+0000 7fe2cb200640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2cc109940 0x7fe2cc1a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.169+0000 7fe2cb200640 1 --2- 172.21.15.43:0/34111623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2cc103a00 0x7fe2cc19c280 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.169+0000 7fe2cb200640 1 -- 172.21.15.43:0/34111623 >> 172.21.15.43:0/34111623 conn(0x7fe2cc0febd0 msgr2=0x7fe2cc0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:01.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.170+0000 7fe2cb200640 1 -- 172.21.15.43:0/34111623 shutdown_connections 2024-05-27T07:14:01.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.170+0000 7fe2cb200640 1 -- 172.21.15.43:0/34111623 wait complete. 2024-05-27T07:14:01.189 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-27T07:14:00.440276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1195241,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2642,"num_read_kb":1465,"num_write":518,"num_write_kb":3632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953452,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369236,"kb_used_data":15836,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124572588,"statfs":{"total":1151940427776,"available":1151562330112,"internally_reserved":0,"allocated":16216064,"data_stored":9659013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.977748"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551640+0000","last_change":"2024-05-27T07:08:22.053540+0000","last_active":"2024-05-27T07:13:54.551640+0000","last_peered":"2024-05-27T07:13:54.551640+0000","last_clean":"2024-05-27T07:13:54.551640+0000","last_became_active":"2024-05-27T07:08:22.051872+0000","last_became_peered":"2024-05-27T07:08:22.051872+0000","last_unstale":"2024-05-27T07:13:54.551640+0000","last_undegraded":"2024-05-27T07:13:54.551640+0000","last_fullsized":"2024-05-27T07:13:54.551640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:02.787957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000328+0000","last_change":"2024-05-27T07:08:22.049341+0000","last_active":"2024-05-27T07:13:55.000328+0000","last_peered":"2024-05-27T07:13:55.000328+0000","last_clean":"2024-05-27T07:13:55.000328+0000","last_became_active":"2024-05-27T07:08:22.048905+0000","last_became_peered":"2024-05-27T07:08:22.048905+0000","last_unstale":"2024-05-27T07:13:55.000328+0000","last_undegraded":"2024-05-27T07:13:55.000328+0000","last_fullsized":"2024-05-27T07:13:55.000328+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:27.271336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.999833+0000","last_change":"2024-05-27T07:08:22.053925+0000","last_active":"2024-05-27T07:13:54.999833+0000","last_peered":"2024-05-27T07:13:54.999833+0000","last_clean":"2024-05-27T07:13:54.999833+0000","last_became_active":"2024-05-27T07:08:22.053409+0000","last_became_peered":"2024-05-27T07:08:22.053409+0000","last_unstale":"2024-05-27T07:13:54.999833+0000","last_undegraded":"2024-05-27T07:13:54.999833+0000","last_fullsized":"2024-05-27T07:13:54.999833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:07:55.439966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835634+0000","last_change":"2024-05-27T07:08:22.048214+0000","last_active":"2024-05-27T07:13:54.835634+0000","last_peered":"2024-05-27T07:13:54.835634+0000","last_clean":"2024-05-27T07:13:54.835634+0000","last_became_active":"2024-05-27T07:08:22.047791+0000","last_became_peered":"2024-05-27T07:08:22.047791+0000","last_unstale":"2024-05-27T07:13:54.835634+0000","last_undegraded":"2024-05-27T07:13:54.835634+0000","last_fullsized":"2024-05-27T07:13:54.835634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:23:56.381008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572980+0000","last_change":"2024-05-27T07:08:22.037883+0000","last_active":"2024-05-27T07:13:54.572980+0000","last_peered":"2024-05-27T07:13:54.572980+0000","last_clean":"2024-05-27T07:13:54.572980+0000","last_became_active":"2024-05-27T07:08:22.037089+0000","last_became_peered":"2024-05-27T07:08:22.037089+0000","last_unstale":"2024-05-27T07:13:54.572980+0000","last_undegraded":"2024-05-27T07:13:54.572980+0000","last_fullsized":"2024-05-27T07:13:54.572980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:21.929086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.478833+0000","last_change":"2024-05-27T07:08:22.061833+0000","last_active":"2024-05-27T07:13:54.478833+0000","last_peered":"2024-05-27T07:13:54.478833+0000","last_clean":"2024-05-27T07:13:54.478833+0000","last_became_active":"2024-05-27T07:08:22.057692+0000","last_became_peered":"2024-05-27T07:08:22.057692+0000","last_unstale":"2024-05-27T07:13:54.478833+0000","last_undegraded":"2024-05-27T07:13:54.478833+0000","last_fullsized":"2024-05-27T07:13:54.478833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:38:33.656872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779346+0000","last_change":"2024-05-27T07:08:22.047880+0000","last_active":"2024-05-27T07:13:54.779346+0000","last_peered":"2024-05-27T07:13:54.779346+0000","last_clean":"2024-05-27T07:13:54.779346+0000","last_became_active":"2024-05-27T07:08:22.047482+0000","last_became_peered":"2024-05-27T07:08:22.047482+0000","last_unstale":"2024-05-27T07:13:54.779346+0000","last_undegraded":"2024-05-27T07:13:54.779346+0000","last_fullsized":"2024-05-27T07:13:54.779346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:07.751588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.451911+0000","last_change":"2024-05-27T07:08:22.055970+0000","last_active":"2024-05-27T07:13:54.451911+0000","last_peered":"2024-05-27T07:13:54.451911+0000","last_clean":"2024-05-27T07:13:54.451911+0000","last_became_active":"2024-05-27T07:08:22.055591+0000","last_became_peered":"2024-05-27T07:08:22.055591+0000","last_unstale":"2024-05-27T07:13:54.451911+0000","last_undegraded":"2024-05-27T07:13:54.451911+0000","last_fullsized":"2024-05-27T07:13:54.451911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:58:46.571473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412531+0000","last_change":"2024-05-27T07:08:22.059929+0000","last_active":"2024-05-27T07:13:54.412531+0000","last_peered":"2024-05-27T07:13:54.412531+0000","last_clean":"2024-05-27T07:13:54.412531+0000","last_became_active":"2024-05-27T07:08:22.059118+0000","last_became_peered":"2024-05-27T07:08:22.059118+0000","last_unstale":"2024-05-27T07:13:54.412531+0000","last_undegraded":"2024-05-27T07:13:54.412531+0000","last_fullsized":"2024-05-27T07:13:54.412531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:29.231220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.479644+0000","last_change":"2024-05-27T07:08:53.979269+0000","last_active":"2024-05-27T07:13:54.479644+0000","last_peered":"2024-05-27T07:13:54.479644+0000","last_clean":"2024-05-27T07:13:54.479644+0000","last_became_active":"2024-05-27T07:08:53.978433+0000","last_became_peered":"2024-05-27T07:08:53.978433+0000","last_unstale":"2024-05-27T07:13:54.479644+0000","last_undegraded":"2024-05-27T07:13:54.479644+0000","last_fullsized":"2024-05-27T07:13:54.479644+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:57:37.600911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780920+0000","last_change":"2024-05-27T07:08:22.053499+0000","last_active":"2024-05-27T07:13:54.780920+0000","last_peered":"2024-05-27T07:13:54.780920+0000","last_clean":"2024-05-27T07:13:54.780920+0000","last_became_active":"2024-05-27T07:08:22.052428+0000","last_became_peered":"2024-05-27T07:08:22.052428+0000","last_unstale":"2024-05-27T07:13:54.780920+0000","last_undegraded":"2024-05-27T07:13:54.780920+0000","last_fullsized":"2024-05-27T07:13:54.780920+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:04.352262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835640+0000","last_change":"2024-05-27T07:08:22.058882+0000","last_active":"2024-05-27T07:13:54.835640+0000","last_peered":"2024-05-27T07:13:54.835640+0000","last_clean":"2024-05-27T07:13:54.835640+0000","last_became_active":"2024-05-27T07:08:22.057874+0000","last_became_peered":"2024-05-27T07:08:22.057874+0000","last_unstale":"2024-05-27T07:13:54.835640+0000","last_undegraded":"2024-05-27T07:13:54.835640+0000","last_fullsized":"2024-05-27T07:13:54.835640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:52:11.289488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.828041+0000","last_change":"2024-05-27T07:08:22.054315+0000","last_active":"2024-05-27T07:13:54.828041+0000","last_peered":"2024-05-27T07:13:54.828041+0000","last_clean":"2024-05-27T07:13:54.828041+0000","last_became_active":"2024-05-27T07:08:22.053944+0000","last_became_peered":"2024-05-27T07:08:22.053944+0000","last_unstale":"2024-05-27T07:13:54.828041+0000","last_undegraded":"2024-05-27T07:13:54.828041+0000","last_fullsized":"2024-05-27T07:13:54.828041+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:43:06.720147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.998761+0000","last_change":"2024-05-27T07:08:22.054649+0000","last_active":"2024-05-27T07:13:54.998761+0000","last_peered":"2024-05-27T07:13:54.998761+0000","last_clean":"2024-05-27T07:13:54.998761+0000","last_became_active":"2024-05-27T07:08:22.051834+0000","last_became_peered":"2024-05-27T07:08:22.051834+0000","last_unstale":"2024-05-27T07:13:54.998761+0000","last_undegraded":"2024-05-27T07:13:54.998761+0000","last_fullsized":"2024-05-27T07:13:54.998761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:16:56.893458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.998595+0000","last_change":"2024-05-27T07:08:22.052231+0000","last_active":"2024-05-27T07:13:54.998595+0000","last_peered":"2024-05-27T07:13:54.998595+0000","last_clean":"2024-05-27T07:13:54.998595+0000","last_became_active":"2024-05-27T07:08:22.051495+0000","last_became_peered":"2024-05-27T07:08:22.051495+0000","last_unstale":"2024-05-27T07:13:54.998595+0000","last_undegraded":"2024-05-27T07:13:54.998595+0000","last_fullsized":"2024-05-27T07:13:54.998595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:30:21.036874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410977+0000","last_change":"2024-05-27T07:08:22.056847+0000","last_active":"2024-05-27T07:13:54.410977+0000","last_peered":"2024-05-27T07:13:54.410977+0000","last_clean":"2024-05-27T07:13:54.410977+0000","last_became_active":"2024-05-27T07:08:22.056127+0000","last_became_peered":"2024-05-27T07:08:22.056127+0000","last_unstale":"2024-05-27T07:13:54.410977+0000","last_undegraded":"2024-05-27T07:13:54.410977+0000","last_fullsized":"2024-05-27T07:13:54.410977+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:04.757452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551330+0000","last_change":"2024-05-27T07:08:22.058078+0000","last_active":"2024-05-27T07:13:54.551330+0000","last_peered":"2024-05-27T07:13:54.551330+0000","last_clean":"2024-05-27T07:13:54.551330+0000","last_became_active":"2024-05-27T07:08:22.056886+0000","last_became_peered":"2024-05-27T07:08:22.056886+0000","last_unstale":"2024-05-27T07:13:54.551330+0000","last_undegraded":"2024-05-27T07:13:54.551330+0000","last_fullsized":"2024-05-27T07:13:54.551330+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.104572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.407614+0000","last_change":"2024-05-27T07:08:22.051492+0000","last_active":"2024-05-27T07:13:54.407614+0000","last_peered":"2024-05-27T07:13:54.407614+0000","last_clean":"2024-05-27T07:13:54.407614+0000","last_became_active":"2024-05-27T07:08:22.051091+0000","last_became_peered":"2024-05-27T07:08:22.051091+0000","last_unstale":"2024-05-27T07:13:54.407614+0000","last_undegraded":"2024-05-27T07:13:54.407614+0000","last_fullsized":"2024-05-27T07:13:54.407614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:47.816312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411384+0000","last_change":"2024-05-27T07:08:22.050714+0000","last_active":"2024-05-27T07:13:54.411384+0000","last_peered":"2024-05-27T07:13:54.411384+0000","last_clean":"2024-05-27T07:13:54.411384+0000","last_became_active":"2024-05-27T07:08:22.050322+0000","last_became_peered":"2024-05-27T07:08:22.050322+0000","last_unstale":"2024-05-27T07:13:54.411384+0000","last_undegraded":"2024-05-27T07:13:54.411384+0000","last_fullsized":"2024-05-27T07:13:54.411384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:01.867467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410108+0000","last_change":"2024-05-27T07:08:36.859916+0000","last_active":"2024-05-27T07:13:54.410108+0000","last_peered":"2024-05-27T07:13:54.410108+0000","last_clean":"2024-05-27T07:13:54.410108+0000","last_became_active":"2024-05-27T07:08:36.859429+0000","last_became_peered":"2024-05-27T07:08:36.859429+0000","last_unstale":"2024-05-27T07:13:54.410108+0000","last_undegraded":"2024-05-27T07:13:54.410108+0000","last_fullsized":"2024-05-27T07:13:54.410108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:14.026604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'14","reported_seq":305,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410345+0000","last_change":"2024-05-27T07:08:18.008912+0000","last_active":"2024-05-27T07:13:54.410345+0000","last_peered":"2024-05-27T07:13:54.410345+0000","last_clean":"2024-05-27T07:13:54.410345+0000","last_became_active":"2024-05-27T07:08:18.007603+0000","last_became_peered":"2024-05-27T07:08:18.007603+0000","last_unstale":"2024-05-27T07:13:54.410345+0000","last_undegraded":"2024-05-27T07:13:54.410345+0000","last_fullsized":"2024-05-27T07:13:54.410345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:17:40.681754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":243,"num_read_kb":243,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453059+0000","last_change":"2024-05-27T07:08:22.055426+0000","last_active":"2024-05-27T07:13:54.453059+0000","last_peered":"2024-05-27T07:13:54.453059+0000","last_clean":"2024-05-27T07:13:54.453059+0000","last_became_active":"2024-05-27T07:08:22.055063+0000","last_became_peered":"2024-05-27T07:08:22.055063+0000","last_unstale":"2024-05-27T07:13:54.453059+0000","last_undegraded":"2024-05-27T07:13:54.453059+0000","last_fullsized":"2024-05-27T07:13:54.453059+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:06:49.923192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834122+0000","last_change":"2024-05-27T07:08:53.982756+0000","last_active":"2024-05-27T07:13:54.834122+0000","last_peered":"2024-05-27T07:13:54.834122+0000","last_clean":"2024-05-27T07:13:54.834122+0000","last_became_active":"2024-05-27T07:08:53.981816+0000","last_became_peered":"2024-05-27T07:08:53.981816+0000","last_unstale":"2024-05-27T07:13:54.834122+0000","last_undegraded":"2024-05-27T07:13:54.834122+0000","last_fullsized":"2024-05-27T07:13:54.834122+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:09.984488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453284+0000","last_change":"2024-05-27T07:08:18.307659+0000","last_active":"2024-05-27T07:13:54.453284+0000","last_peered":"2024-05-27T07:13:54.453284+0000","last_clean":"2024-05-27T07:13:54.453284+0000","last_became_active":"2024-05-27T07:08:18.307239+0000","last_became_peered":"2024-05-27T07:08:18.307239+0000","last_unstale":"2024-05-27T07:13:54.453284+0000","last_undegraded":"2024-05-27T07:13:54.453284+0000","last_fullsized":"2024-05-27T07:13:54.453284+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:09:02.816813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000794+0000","last_change":"2024-05-27T07:08:22.050403+0000","last_active":"2024-05-27T07:13:55.000794+0000","last_peered":"2024-05-27T07:13:55.000794+0000","last_clean":"2024-05-27T07:13:55.000794+0000","last_became_active":"2024-05-27T07:08:22.049939+0000","last_became_peered":"2024-05-27T07:08:22.049939+0000","last_unstale":"2024-05-27T07:13:55.000794+0000","last_undegraded":"2024-05-27T07:13:55.000794+0000","last_fullsized":"2024-05-27T07:13:55.000794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:45.857508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.833966+0000","last_change":"2024-05-27T07:08:36.864458+0000","last_active":"2024-05-27T07:13:54.833966+0000","last_peered":"2024-05-27T07:13:54.833966+0000","last_clean":"2024-05-27T07:13:54.833966+0000","last_became_active":"2024-05-27T07:08:36.863894+0000","last_became_peered":"2024-05-27T07:08:36.863894+0000","last_unstale":"2024-05-27T07:13:54.833966+0000","last_undegraded":"2024-05-27T07:13:54.833966+0000","last_fullsized":"2024-05-27T07:13:54.833966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:49.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'13","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573904+0000","last_change":"2024-05-27T07:08:18.016051+0000","last_active":"2024-05-27T07:13:54.573904+0000","last_peered":"2024-05-27T07:13:54.573904+0000","last_clean":"2024-05-27T07:13:54.573904+0000","last_became_active":"2024-05-27T07:08:18.015590+0000","last_became_peered":"2024-05-27T07:08:18.015590+0000","last_unstale":"2024-05-27T07:13:54.573904+0000","last_undegraded":"2024-05-27T07:13:54.573904+0000","last_fullsized":"2024-05-27T07:13:54.573904+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:33.396027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411799+0000","last_change":"2024-05-27T07:08:22.055603+0000","last_active":"2024-05-27T07:13:54.411799+0000","last_peered":"2024-05-27T07:13:54.411799+0000","last_clean":"2024-05-27T07:13:54.411799+0000","last_became_active":"2024-05-27T07:08:22.054543+0000","last_became_peered":"2024-05-27T07:08:22.054543+0000","last_unstale":"2024-05-27T07:13:54.411799+0000","last_undegraded":"2024-05-27T07:13:54.411799+0000","last_fullsized":"2024-05-27T07:13:54.411799+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:29:01.738598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.833715+0000","last_change":"2024-05-27T07:08:36.870358+0000","last_active":"2024-05-27T07:13:54.833715+0000","last_peered":"2024-05-27T07:13:54.833715+0000","last_clean":"2024-05-27T07:13:54.833715+0000","last_became_active":"2024-05-27T07:08:36.869457+0000","last_became_peered":"2024-05-27T07:08:36.869457+0000","last_unstale":"2024-05-27T07:13:54.833715+0000","last_undegraded":"2024-05-27T07:13:54.833715+0000","last_fullsized":"2024-05-27T07:13:54.833715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:50.510098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550540+0000","last_change":"2024-05-27T07:08:53.985562+0000","last_active":"2024-05-27T07:13:54.550540+0000","last_peered":"2024-05-27T07:13:54.550540+0000","last_clean":"2024-05-27T07:13:54.550540+0000","last_became_active":"2024-05-27T07:08:53.977778+0000","last_became_peered":"2024-05-27T07:08:53.977778+0000","last_unstale":"2024-05-27T07:13:54.550540+0000","last_undegraded":"2024-05-27T07:13:54.550540+0000","last_fullsized":"2024-05-27T07:13:54.550540+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:44.455508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584633+0000","last_change":"2024-05-27T07:08:22.057622+0000","last_active":"2024-05-27T07:13:54.584633+0000","last_peered":"2024-05-27T07:13:54.584633+0000","last_clean":"2024-05-27T07:13:54.584633+0000","last_became_active":"2024-05-27T07:08:22.056544+0000","last_became_peered":"2024-05-27T07:08:22.056544+0000","last_unstale":"2024-05-27T07:13:54.584633+0000","last_undegraded":"2024-05-27T07:13:54.584633+0000","last_fullsized":"2024-05-27T07:13:54.584633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:38.484560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410554+0000","last_change":"2024-05-27T07:08:36.864984+0000","last_active":"2024-05-27T07:13:54.410554+0000","last_peered":"2024-05-27T07:13:54.410554+0000","last_clean":"2024-05-27T07:13:54.410554+0000","last_became_active":"2024-05-27T07:08:36.864302+0000","last_became_peered":"2024-05-27T07:08:36.864302+0000","last_unstale":"2024-05-27T07:13:54.410554+0000","last_undegraded":"2024-05-27T07:13:54.410554+0000","last_fullsized":"2024-05-27T07:13:54.410554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:44.183562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584240+0000","last_change":"2024-05-27T07:08:18.017918+0000","last_active":"2024-05-27T07:13:54.584240+0000","last_peered":"2024-05-27T07:13:54.584240+0000","last_clean":"2024-05-27T07:13:54.584240+0000","last_became_active":"2024-05-27T07:08:18.016174+0000","last_became_peered":"2024-05-27T07:08:18.016174+0000","last_unstale":"2024-05-27T07:13:54.584240+0000","last_undegraded":"2024-05-27T07:13:54.584240+0000","last_fullsized":"2024-05-27T07:13:54.584240+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:21.906187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573084+0000","last_change":"2024-05-27T07:08:22.043172+0000","last_active":"2024-05-27T07:13:54.573084+0000","last_peered":"2024-05-27T07:13:54.573084+0000","last_clean":"2024-05-27T07:13:54.573084+0000","last_became_active":"2024-05-27T07:08:22.041399+0000","last_became_peered":"2024-05-27T07:08:22.041399+0000","last_unstale":"2024-05-27T07:13:54.573084+0000","last_undegraded":"2024-05-27T07:13:54.573084+0000","last_fullsized":"2024-05-27T07:13:54.573084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:19.651330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410576+0000","last_change":"2024-05-27T07:08:36.866352+0000","last_active":"2024-05-27T07:13:54.410576+0000","last_peered":"2024-05-27T07:13:54.410576+0000","last_clean":"2024-05-27T07:13:54.410576+0000","last_became_active":"2024-05-27T07:08:36.865447+0000","last_became_peered":"2024-05-27T07:08:36.865447+0000","last_unstale":"2024-05-27T07:13:54.410576+0000","last_undegraded":"2024-05-27T07:13:54.410576+0000","last_fullsized":"2024-05-27T07:13:54.410576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:55:42.634374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.478763+0000","last_change":"2024-05-27T07:08:18.022142+0000","last_active":"2024-05-27T07:13:54.478763+0000","last_peered":"2024-05-27T07:13:54.478763+0000","last_clean":"2024-05-27T07:13:54.478763+0000","last_became_active":"2024-05-27T07:08:18.021453+0000","last_became_peered":"2024-05-27T07:08:18.021453+0000","last_unstale":"2024-05-27T07:13:54.478763+0000","last_undegraded":"2024-05-27T07:13:54.478763+0000","last_fullsized":"2024-05-27T07:13:54.478763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:01:09.049440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584299+0000","last_change":"2024-05-27T07:08:22.055742+0000","last_active":"2024-05-27T07:13:54.584299+0000","last_peered":"2024-05-27T07:13:54.584299+0000","last_clean":"2024-05-27T07:13:54.584299+0000","last_became_active":"2024-05-27T07:08:22.052077+0000","last_became_peered":"2024-05-27T07:08:22.052077+0000","last_unstale":"2024-05-27T07:13:54.584299+0000","last_undegraded":"2024-05-27T07:13:54.584299+0000","last_fullsized":"2024-05-27T07:13:54.584299+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:09:13.418027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551028+0000","last_change":"2024-05-27T07:08:36.874225+0000","last_active":"2024-05-27T07:13:54.551028+0000","last_peered":"2024-05-27T07:13:54.551028+0000","last_clean":"2024-05-27T07:13:54.551028+0000","last_became_active":"2024-05-27T07:08:36.873745+0000","last_became_peered":"2024-05-27T07:08:36.873745+0000","last_unstale":"2024-05-27T07:13:54.551028+0000","last_undegraded":"2024-05-27T07:13:54.551028+0000","last_fullsized":"2024-05-27T07:13:54.551028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:40.633667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827100+0000","last_change":"2024-05-27T07:08:18.009678+0000","last_active":"2024-05-27T07:13:54.827100+0000","last_peered":"2024-05-27T07:13:54.827100+0000","last_clean":"2024-05-27T07:13:54.827100+0000","last_became_active":"2024-05-27T07:08:18.006502+0000","last_became_peered":"2024-05-27T07:08:18.006502+0000","last_unstale":"2024-05-27T07:13:54.827100+0000","last_undegraded":"2024-05-27T07:13:54.827100+0000","last_fullsized":"2024-05-27T07:13:54.827100+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:37.457044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452037+0000","last_change":"2024-05-27T07:08:22.047302+0000","last_active":"2024-05-27T07:13:54.452037+0000","last_peered":"2024-05-27T07:13:54.452037+0000","last_clean":"2024-05-27T07:13:54.452037+0000","last_became_active":"2024-05-27T07:08:22.045932+0000","last_became_peered":"2024-05-27T07:08:22.045932+0000","last_unstale":"2024-05-27T07:13:54.452037+0000","last_undegraded":"2024-05-27T07:13:54.452037+0000","last_fullsized":"2024-05-27T07:13:54.452037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:27.129539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000221+0000","last_change":"2024-05-27T07:08:36.870512+0000","last_active":"2024-05-27T07:13:55.000221+0000","last_peered":"2024-05-27T07:13:55.000221+0000","last_clean":"2024-05-27T07:13:55.000221+0000","last_became_active":"2024-05-27T07:08:36.870061+0000","last_became_peered":"2024-05-27T07:08:36.870061+0000","last_unstale":"2024-05-27T07:13:55.000221+0000","last_undegraded":"2024-05-27T07:13:55.000221+0000","last_fullsized":"2024-05-27T07:13:55.000221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:03.608758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583370+0000","last_change":"2024-05-27T07:08:18.018420+0000","last_active":"2024-05-27T07:13:54.583370+0000","last_peered":"2024-05-27T07:13:54.583370+0000","last_clean":"2024-05-27T07:13:54.583370+0000","last_became_active":"2024-05-27T07:08:18.017567+0000","last_became_peered":"2024-05-27T07:08:18.017567+0000","last_unstale":"2024-05-27T07:13:54.583370+0000","last_undegraded":"2024-05-27T07:13:54.583370+0000","last_fullsized":"2024-05-27T07:13:54.583370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:37.214408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.404452+0000","last_change":"2024-05-27T07:08:22.050943+0000","last_active":"2024-05-27T07:13:54.404452+0000","last_peered":"2024-05-27T07:13:54.404452+0000","last_clean":"2024-05-27T07:13:54.404452+0000","last_became_active":"2024-05-27T07:08:22.047706+0000","last_became_peered":"2024-05-27T07:08:22.047706+0000","last_unstale":"2024-05-27T07:13:54.404452+0000","last_undegraded":"2024-05-27T07:13:54.404452+0000","last_fullsized":"2024-05-27T07:13:54.404452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:52:09.739528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452211+0000","last_change":"2024-05-27T07:08:36.867487+0000","last_active":"2024-05-27T07:13:54.452211+0000","last_peered":"2024-05-27T07:13:54.452211+0000","last_clean":"2024-05-27T07:13:54.452211+0000","last_became_active":"2024-05-27T07:08:36.867032+0000","last_became_peered":"2024-05-27T07:08:36.867032+0000","last_unstale":"2024-05-27T07:13:54.452211+0000","last_undegraded":"2024-05-27T07:13:54.452211+0000","last_fullsized":"2024-05-27T07:13:54.452211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:26:24.912382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'7","reported_seq":539,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.792547+0000","last_change":"2024-05-27T07:08:18.020168+0000","last_active":"2024-05-27T07:13:54.792547+0000","last_peered":"2024-05-27T07:13:54.792547+0000","last_clean":"2024-05-27T07:13:54.792547+0000","last_became_active":"2024-05-27T07:08:18.018309+0000","last_became_peered":"2024-05-27T07:08:18.018309+0000","last_unstale":"2024-05-27T07:13:54.792547+0000","last_undegraded":"2024-05-27T07:13:54.792547+0000","last_fullsized":"2024-05-27T07:13:54.792547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:48:16.839510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":484,"num_read_kb":484,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412438+0000","last_change":"2024-05-27T07:08:22.054888+0000","last_active":"2024-05-27T07:13:54.412438+0000","last_peered":"2024-05-27T07:13:54.412438+0000","last_clean":"2024-05-27T07:13:54.412438+0000","last_became_active":"2024-05-27T07:08:22.054417+0000","last_became_peered":"2024-05-27T07:08:22.054417+0000","last_unstale":"2024-05-27T07:13:54.412438+0000","last_undegraded":"2024-05-27T07:13:54.412438+0000","last_fullsized":"2024-05-27T07:13:54.412438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:43:00.866755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.404284+0000","last_change":"2024-05-27T07:08:36.854516+0000","last_active":"2024-05-27T07:13:54.404284+0000","last_peered":"2024-05-27T07:13:54.404284+0000","last_clean":"2024-05-27T07:13:54.404284+0000","last_became_active":"2024-05-27T07:08:36.853978+0000","last_became_peered":"2024-05-27T07:08:36.853978+0000","last_unstale":"2024-05-27T07:13:54.404284+0000","last_undegraded":"2024-05-27T07:13:54.404284+0000","last_fullsized":"2024-05-27T07:13:54.404284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:45.299100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.781001+0000","last_change":"2024-05-27T07:08:18.010539+0000","last_active":"2024-05-27T07:13:54.781001+0000","last_peered":"2024-05-27T07:13:54.781001+0000","last_clean":"2024-05-27T07:13:54.781001+0000","last_became_active":"2024-05-27T07:08:18.009643+0000","last_became_peered":"2024-05-27T07:08:18.009643+0000","last_unstale":"2024-05-27T07:13:54.781001+0000","last_undegraded":"2024-05-27T07:13:54.781001+0000","last_fullsized":"2024-05-27T07:13:54.781001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:48:40.335403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000623+0000","last_change":"2024-05-27T07:08:53.980165+0000","last_active":"2024-05-27T07:13:55.000623+0000","last_peered":"2024-05-27T07:13:55.000623+0000","last_clean":"2024-05-27T07:13:55.000623+0000","last_became_active":"2024-05-27T07:08:53.978565+0000","last_became_peered":"2024-05-27T07:08:53.978565+0000","last_unstale":"2024-05-27T07:13:55.000623+0000","last_undegraded":"2024-05-27T07:13:55.000623+0000","last_fullsized":"2024-05-27T07:13:55.000623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:56:33.513961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834628+0000","last_change":"2024-05-27T07:08:36.857814+0000","last_active":"2024-05-27T07:13:54.834628+0000","last_peered":"2024-05-27T07:13:54.834628+0000","last_clean":"2024-05-27T07:13:54.834628+0000","last_became_active":"2024-05-27T07:08:36.856790+0000","last_became_peered":"2024-05-27T07:08:36.856790+0000","last_unstale":"2024-05-27T07:13:54.834628+0000","last_undegraded":"2024-05-27T07:13:54.834628+0000","last_fullsized":"2024-05-27T07:13:54.834628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:39:10.322261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584453+0000","last_change":"2024-05-27T07:08:18.308032+0000","last_active":"2024-05-27T07:13:54.584453+0000","last_peered":"2024-05-27T07:13:54.584453+0000","last_clean":"2024-05-27T07:13:54.584453+0000","last_became_active":"2024-05-27T07:08:18.306867+0000","last_became_peered":"2024-05-27T07:08:18.306867+0000","last_unstale":"2024-05-27T07:13:54.584453+0000","last_undegraded":"2024-05-27T07:13:54.584453+0000","last_fullsized":"2024-05-27T07:13:54.584453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:37.816237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412154+0000","last_change":"2024-05-27T07:08:22.057940+0000","last_active":"2024-05-27T07:13:54.412154+0000","last_peered":"2024-05-27T07:13:54.412154+0000","last_clean":"2024-05-27T07:13:54.412154+0000","last_became_active":"2024-05-27T07:08:22.055109+0000","last_became_peered":"2024-05-27T07:08:22.055109+0000","last_unstale":"2024-05-27T07:13:54.412154+0000","last_undegraded":"2024-05-27T07:13:54.412154+0000","last_fullsized":"2024-05-27T07:13:54.412154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:46.905064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.478388+0000","last_change":"2024-05-27T07:08:53.980209+0000","last_active":"2024-05-27T07:13:54.478388+0000","last_peered":"2024-05-27T07:13:54.478388+0000","last_clean":"2024-05-27T07:13:54.478388+0000","last_became_active":"2024-05-27T07:08:53.979353+0000","last_became_peered":"2024-05-27T07:08:53.979353+0000","last_unstale":"2024-05-27T07:13:54.478388+0000","last_undegraded":"2024-05-27T07:13:54.478388+0000","last_fullsized":"2024-05-27T07:13:54.478388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:06.163563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'41","reported_seq":600,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.802694+0000","last_change":"2024-05-27T07:08:53.996491+0000","last_active":"2024-05-27T07:13:54.802694+0000","last_peered":"2024-05-27T07:13:54.802694+0000","last_clean":"2024-05-27T07:13:54.802694+0000","last_became_active":"2024-05-27T07:08:53.976013+0000","last_became_peered":"2024-05-27T07:08:53.976013+0000","last_unstale":"2024-05-27T07:13:54.802694+0000","last_undegraded":"2024-05-27T07:13:54.802694+0000","last_fullsized":"2024-05-27T07:13:54.802694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:44:16.592270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69583,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":495,"num_read_kb":311,"num_write":43,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573510+0000","last_change":"2024-05-27T07:08:22.057231+0000","last_active":"2024-05-27T07:13:54.573510+0000","last_peered":"2024-05-27T07:13:54.573510+0000","last_clean":"2024-05-27T07:13:54.573510+0000","last_became_active":"2024-05-27T07:08:22.056837+0000","last_became_peered":"2024-05-27T07:08:22.056837+0000","last_unstale":"2024-05-27T07:13:54.573510+0000","last_undegraded":"2024-05-27T07:13:54.573510+0000","last_fullsized":"2024-05-27T07:13:54.573510+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:10:28.645103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.585422+0000","last_change":"2024-05-27T07:08:36.870307+0000","last_active":"2024-05-27T07:13:54.585422+0000","last_peered":"2024-05-27T07:13:54.585422+0000","last_clean":"2024-05-27T07:13:54.585422+0000","last_became_active":"2024-05-27T07:08:36.869390+0000","last_became_peered":"2024-05-27T07:08:36.869390+0000","last_unstale":"2024-05-27T07:13:54.585422+0000","last_undegraded":"2024-05-27T07:13:54.585422+0000","last_fullsized":"2024-05-27T07:13:54.585422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:43:45.228885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410841+0000","last_change":"2024-05-27T07:08:18.014458+0000","last_active":"2024-05-27T07:13:54.410841+0000","last_peered":"2024-05-27T07:13:54.410841+0000","last_clean":"2024-05-27T07:13:54.410841+0000","last_became_active":"2024-05-27T07:08:18.013365+0000","last_became_peered":"2024-05-27T07:08:18.013365+0000","last_unstale":"2024-05-27T07:13:54.410841+0000","last_undegraded":"2024-05-27T07:13:54.410841+0000","last_fullsized":"2024-05-27T07:13:54.410841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:41.352618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827630+0000","last_change":"2024-05-27T07:08:22.043354+0000","last_active":"2024-05-27T07:13:54.827630+0000","last_peered":"2024-05-27T07:13:54.827630+0000","last_clean":"2024-05-27T07:13:54.827630+0000","last_became_active":"2024-05-27T07:08:22.042876+0000","last_became_peered":"2024-05-27T07:08:22.042876+0000","last_unstale":"2024-05-27T07:13:54.827630+0000","last_undegraded":"2024-05-27T07:13:54.827630+0000","last_fullsized":"2024-05-27T07:13:54.827630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:52.575568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550548+0000","last_change":"2024-05-27T07:08:36.869579+0000","last_active":"2024-05-27T07:13:54.550548+0000","last_peered":"2024-05-27T07:13:54.550548+0000","last_clean":"2024-05-27T07:13:54.550548+0000","last_became_active":"2024-05-27T07:08:36.869055+0000","last_became_peered":"2024-05-27T07:08:36.869055+0000","last_unstale":"2024-05-27T07:13:54.550548+0000","last_undegraded":"2024-05-27T07:13:54.550548+0000","last_fullsized":"2024-05-27T07:13:54.550548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.870079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584312+0000","last_change":"2024-05-27T07:08:18.015322+0000","last_active":"2024-05-27T07:13:54.584312+0000","last_peered":"2024-05-27T07:13:54.584312+0000","last_clean":"2024-05-27T07:13:54.584312+0000","last_became_active":"2024-05-27T07:08:18.014535+0000","last_became_peered":"2024-05-27T07:08:18.014535+0000","last_unstale":"2024-05-27T07:13:54.584312+0000","last_undegraded":"2024-05-27T07:13:54.584312+0000","last_fullsized":"2024-05-27T07:13:54.584312+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:35.116140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779909+0000","last_change":"2024-05-27T07:08:22.056181+0000","last_active":"2024-05-27T07:13:54.779909+0000","last_peered":"2024-05-27T07:13:54.779909+0000","last_clean":"2024-05-27T07:13:54.779909+0000","last_became_active":"2024-05-27T07:08:22.055755+0000","last_became_peered":"2024-05-27T07:08:22.055755+0000","last_unstale":"2024-05-27T07:13:54.779909+0000","last_undegraded":"2024-05-27T07:13:54.779909+0000","last_fullsized":"2024-05-27T07:13:54.779909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:38:35.634116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.409346+0000","last_change":"2024-05-27T07:08:36.863184+0000","last_active":"2024-05-27T07:13:54.409346+0000","last_peered":"2024-05-27T07:13:54.409346+0000","last_clean":"2024-05-27T07:13:54.409346+0000","last_became_active":"2024-05-27T07:08:36.862760+0000","last_became_peered":"2024-05-27T07:08:36.862760+0000","last_unstale":"2024-05-27T07:13:54.409346+0000","last_undegraded":"2024-05-27T07:13:54.409346+0000","last_fullsized":"2024-05-27T07:13:54.409346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:48.379157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":302,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834697+0000","last_change":"2024-05-27T07:08:18.019119+0000","last_active":"2024-05-27T07:13:54.834697+0000","last_peered":"2024-05-27T07:13:54.834697+0000","last_clean":"2024-05-27T07:13:54.834697+0000","last_became_active":"2024-05-27T07:08:18.018408+0000","last_became_peered":"2024-05-27T07:08:18.018408+0000","last_unstale":"2024-05-27T07:13:54.834697+0000","last_undegraded":"2024-05-27T07:13:54.834697+0000","last_fullsized":"2024-05-27T07:13:54.834697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:19.978328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":248,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779408+0000","last_change":"2024-05-27T07:08:22.059464+0000","last_active":"2024-05-27T07:13:54.779408+0000","last_peered":"2024-05-27T07:13:54.779408+0000","last_clean":"2024-05-27T07:13:54.779408+0000","last_became_active":"2024-05-27T07:08:22.058890+0000","last_became_peered":"2024-05-27T07:08:22.058890+0000","last_unstale":"2024-05-27T07:13:54.779408+0000","last_undegraded":"2024-05-27T07:13:54.779408+0000","last_fullsized":"2024-05-27T07:13:54.779408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:30.122552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.828201+0000","last_change":"2024-05-27T07:08:36.865884+0000","last_active":"2024-05-27T07:13:54.828201+0000","last_peered":"2024-05-27T07:13:54.828201+0000","last_clean":"2024-05-27T07:13:54.828201+0000","last_became_active":"2024-05-27T07:08:36.865443+0000","last_became_peered":"2024-05-27T07:08:36.865443+0000","last_unstale":"2024-05-27T07:13:54.828201+0000","last_undegraded":"2024-05-27T07:13:54.828201+0000","last_fullsized":"2024-05-27T07:13:54.828201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:47:15.026632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550142+0000","last_change":"2024-05-27T07:08:53.994809+0000","last_active":"2024-05-27T07:13:54.550142+0000","last_peered":"2024-05-27T07:13:54.550142+0000","last_clean":"2024-05-27T07:13:54.550142+0000","last_became_active":"2024-05-27T07:08:53.977628+0000","last_became_peered":"2024-05-27T07:08:53.977628+0000","last_unstale":"2024-05-27T07:13:54.550142+0000","last_undegraded":"2024-05-27T07:13:54.550142+0000","last_fullsized":"2024-05-27T07:13:54.550142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:13:37.176319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583849+0000","last_change":"2024-05-27T07:08:22.050733+0000","last_active":"2024-05-27T07:13:54.583849+0000","last_peered":"2024-05-27T07:13:54.583849+0000","last_clean":"2024-05-27T07:13:54.583849+0000","last_became_active":"2024-05-27T07:08:22.050321+0000","last_became_peered":"2024-05-27T07:08:22.050321+0000","last_unstale":"2024-05-27T07:13:54.583849+0000","last_undegraded":"2024-05-27T07:13:54.583849+0000","last_fullsized":"2024-05-27T07:13:54.583849+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:44.590595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583996+0000","last_change":"2024-05-27T07:08:36.867784+0000","last_active":"2024-05-27T07:13:54.583996+0000","last_peered":"2024-05-27T07:13:54.583996+0000","last_clean":"2024-05-27T07:13:54.583996+0000","last_became_active":"2024-05-27T07:08:36.867290+0000","last_became_peered":"2024-05-27T07:08:36.867290+0000","last_unstale":"2024-05-27T07:13:54.583996+0000","last_undegraded":"2024-05-27T07:13:54.583996+0000","last_fullsized":"2024-05-27T07:13:54.583996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:44:07.649640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573792+0000","last_change":"2024-05-27T07:08:18.013193+0000","last_active":"2024-05-27T07:13:54.573792+0000","last_peered":"2024-05-27T07:13:54.573792+0000","last_clean":"2024-05-27T07:13:54.573792+0000","last_became_active":"2024-05-27T07:08:18.011767+0000","last_became_peered":"2024-05-27T07:08:18.011767+0000","last_unstale":"2024-05-27T07:13:54.573792+0000","last_undegraded":"2024-05-27T07:13:54.573792+0000","last_fullsized":"2024-05-27T07:13:54.573792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:15:03.679719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550880+0000","last_change":"2024-05-27T07:08:22.067145+0000","last_active":"2024-05-27T07:13:54.550880+0000","last_peered":"2024-05-27T07:13:54.550880+0000","last_clean":"2024-05-27T07:13:54.550880+0000","last_became_active":"2024-05-27T07:08:22.066750+0000","last_became_peered":"2024-05-27T07:08:22.066750+0000","last_unstale":"2024-05-27T07:13:54.550880+0000","last_undegraded":"2024-05-27T07:13:54.550880+0000","last_fullsized":"2024-05-27T07:13:54.550880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:41:35.333311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572917+0000","last_change":"2024-05-27T07:08:53.982497+0000","last_active":"2024-05-27T07:13:54.572917+0000","last_peered":"2024-05-27T07:13:54.572917+0000","last_clean":"2024-05-27T07:13:54.572917+0000","last_became_active":"2024-05-27T07:08:53.981920+0000","last_became_peered":"2024-05-27T07:08:53.981920+0000","last_unstale":"2024-05-27T07:13:54.572917+0000","last_undegraded":"2024-05-27T07:13:54.572917+0000","last_fullsized":"2024-05-27T07:13:54.572917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:43:40.095357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.406501+0000","last_change":"2024-05-27T07:08:18.308330+0000","last_active":"2024-05-27T07:13:54.406501+0000","last_peered":"2024-05-27T07:13:54.406501+0000","last_clean":"2024-05-27T07:13:54.406501+0000","last_became_active":"2024-05-27T07:08:18.306961+0000","last_became_peered":"2024-05-27T07:08:18.306961+0000","last_unstale":"2024-05-27T07:13:54.406501+0000","last_undegraded":"2024-05-27T07:13:54.406501+0000","last_fullsized":"2024-05-27T07:13:54.406501+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:27.875936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.479309+0000","last_change":"2024-05-27T07:08:22.047562+0000","last_active":"2024-05-27T07:13:54.479309+0000","last_peered":"2024-05-27T07:13:54.479309+0000","last_clean":"2024-05-27T07:13:54.479309+0000","last_became_active":"2024-05-27T07:08:22.046977+0000","last_became_peered":"2024-05-27T07:08:22.046977+0000","last_unstale":"2024-05-27T07:13:54.479309+0000","last_undegraded":"2024-05-27T07:13:54.479309+0000","last_fullsized":"2024-05-27T07:13:54.479309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:49:43.376617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827427+0000","last_change":"2024-05-27T07:08:36.867382+0000","last_active":"2024-05-27T07:13:54.827427+0000","last_peered":"2024-05-27T07:13:54.827427+0000","last_clean":"2024-05-27T07:13:54.827427+0000","last_became_active":"2024-05-27T07:08:36.866296+0000","last_became_peered":"2024-05-27T07:08:36.866296+0000","last_unstale":"2024-05-27T07:13:54.827427+0000","last_undegraded":"2024-05-27T07:13:54.827427+0000","last_fullsized":"2024-05-27T07:13:54.827427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:14.305169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'7","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780505+0000","last_change":"2024-05-27T07:08:18.012031+0000","last_active":"2024-05-27T07:13:54.780505+0000","last_peered":"2024-05-27T07:13:54.780505+0000","last_clean":"2024-05-27T07:13:54.780505+0000","last_became_active":"2024-05-27T07:08:18.009593+0000","last_became_peered":"2024-05-27T07:08:18.009593+0000","last_unstale":"2024-05-27T07:13:54.780505+0000","last_undegraded":"2024-05-27T07:13:54.780505+0000","last_fullsized":"2024-05-27T07:13:54.780505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:25:22.316575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1472,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.479213+0000","last_change":"2024-05-27T07:08:22.061136+0000","last_active":"2024-05-27T07:13:54.479213+0000","last_peered":"2024-05-27T07:13:54.479213+0000","last_clean":"2024-05-27T07:13:54.479213+0000","last_became_active":"2024-05-27T07:08:22.060735+0000","last_became_peered":"2024-05-27T07:08:22.060735+0000","last_unstale":"2024-05-27T07:13:54.479213+0000","last_undegraded":"2024-05-27T07:13:54.479213+0000","last_fullsized":"2024-05-27T07:13:54.479213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:50.392130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827972+0000","last_change":"2024-05-27T07:08:36.932187+0000","last_active":"2024-05-27T07:13:54.827972+0000","last_peered":"2024-05-27T07:13:54.827972+0000","last_clean":"2024-05-27T07:13:54.827972+0000","last_became_active":"2024-05-27T07:08:36.931214+0000","last_became_peered":"2024-05-27T07:08:36.931214+0000","last_unstale":"2024-05-27T07:13:54.827972+0000","last_undegraded":"2024-05-27T07:13:54.827972+0000","last_fullsized":"2024-05-27T07:13:54.827972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:20:59.645352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.404178+0000","last_change":"2024-05-27T07:09:54.082674+0000","last_active":"2024-05-27T07:13:54.404178+0000","last_peered":"2024-05-27T07:13:54.404178+0000","last_clean":"2024-05-27T07:13:54.404178+0000","last_became_active":"2024-05-27T07:09:54.081616+0000","last_became_peered":"2024-05-27T07:09:54.081616+0000","last_unstale":"2024-05-27T07:13:54.404178+0000","last_undegraded":"2024-05-27T07:13:54.404178+0000","last_fullsized":"2024-05-27T07:13:54.404178+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:38:11.230026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827139+0000","last_change":"2024-05-27T07:08:18.306439+0000","last_active":"2024-05-27T07:13:54.827139+0000","last_peered":"2024-05-27T07:13:54.827139+0000","last_clean":"2024-05-27T07:13:54.827139+0000","last_became_active":"2024-05-27T07:08:18.305978+0000","last_became_peered":"2024-05-27T07:08:18.305978+0000","last_unstale":"2024-05-27T07:13:54.827139+0000","last_undegraded":"2024-05-27T07:13:54.827139+0000","last_fullsized":"2024-05-27T07:13:54.827139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:20:22.648922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":243,"num_read_kb":243,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780681+0000","last_change":"2024-05-27T07:08:36.866247+0000","last_active":"2024-05-27T07:13:54.780681+0000","last_peered":"2024-05-27T07:13:54.780681+0000","last_clean":"2024-05-27T07:13:54.780681+0000","last_became_active":"2024-05-27T07:08:36.865516+0000","last_became_peered":"2024-05-27T07:08:36.865516+0000","last_unstale":"2024-05-27T07:13:54.780681+0000","last_undegraded":"2024-05-27T07:13:54.780681+0000","last_fullsized":"2024-05-27T07:13:54.780681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:39:41.235352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452547+0000","last_change":"2024-05-27T07:08:22.045312+0000","last_active":"2024-05-27T07:13:54.452547+0000","last_peered":"2024-05-27T07:13:54.452547+0000","last_clean":"2024-05-27T07:13:54.452547+0000","last_became_active":"2024-05-27T07:08:22.044724+0000","last_became_peered":"2024-05-27T07:08:22.044724+0000","last_unstale":"2024-05-27T07:13:54.452547+0000","last_undegraded":"2024-05-27T07:13:54.452547+0000","last_fullsized":"2024-05-27T07:13:54.452547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:05.487794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834127+0000","last_change":"2024-05-27T07:08:18.019333+0000","last_active":"2024-05-27T07:13:54.834127+0000","last_peered":"2024-05-27T07:13:54.834127+0000","last_clean":"2024-05-27T07:13:54.834127+0000","last_became_active":"2024-05-27T07:08:18.018816+0000","last_became_peered":"2024-05-27T07:08:18.018816+0000","last_unstale":"2024-05-27T07:13:54.834127+0000","last_undegraded":"2024-05-27T07:13:54.834127+0000","last_fullsized":"2024-05-27T07:13:54.834127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:09.421346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.778974+0000","last_change":"2024-05-27T07:06:16.865587+0000","last_active":"2024-05-27T07:13:54.778974+0000","last_peered":"2024-05-27T07:13:54.778974+0000","last_clean":"2024-05-27T07:13:54.778974+0000","last_became_active":"2024-05-27T07:06:16.850646+0000","last_became_peered":"2024-05-27T07:06:16.850646+0000","last_unstale":"2024-05-27T07:13:54.778974+0000","last_undegraded":"2024-05-27T07:13:54.778974+0000","last_fullsized":"2024-05-27T07:13:54.778974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_clean_scrub_stamp":"2024-05-27T07:01:14.931400+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:37.272619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584459+0000","last_change":"2024-05-27T07:08:36.934696+0000","last_active":"2024-05-27T07:13:54.584459+0000","last_peered":"2024-05-27T07:13:54.584459+0000","last_clean":"2024-05-27T07:13:54.584459+0000","last_became_active":"2024-05-27T07:08:36.934008+0000","last_became_peered":"2024-05-27T07:08:36.934008+0000","last_unstale":"2024-05-27T07:13:54.584459+0000","last_undegraded":"2024-05-27T07:13:54.584459+0000","last_fullsized":"2024-05-27T07:13:54.584459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:39.099180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.826925+0000","last_change":"2024-05-27T07:08:22.054479+0000","last_active":"2024-05-27T07:13:54.826925+0000","last_peered":"2024-05-27T07:13:54.826925+0000","last_clean":"2024-05-27T07:13:54.826925+0000","last_became_active":"2024-05-27T07:08:22.054127+0000","last_became_peered":"2024-05-27T07:08:22.054127+0000","last_unstale":"2024-05-27T07:13:54.826925+0000","last_undegraded":"2024-05-27T07:13:54.826925+0000","last_fullsized":"2024-05-27T07:13:54.826925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:37:25.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779394+0000","last_change":"2024-05-27T07:08:18.010944+0000","last_active":"2024-05-27T07:13:54.779394+0000","last_peered":"2024-05-27T07:13:54.779394+0000","last_clean":"2024-05-27T07:13:54.779394+0000","last_became_active":"2024-05-27T07:08:18.009816+0000","last_became_peered":"2024-05-27T07:08:18.009816+0000","last_unstale":"2024-05-27T07:13:54.779394+0000","last_undegraded":"2024-05-27T07:13:54.779394+0000","last_fullsized":"2024-05-27T07:13:54.779394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:34:26.061787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550526+0000","last_change":"2024-05-27T07:08:36.870618+0000","last_active":"2024-05-27T07:13:54.550526+0000","last_peered":"2024-05-27T07:13:54.550526+0000","last_clean":"2024-05-27T07:13:54.550526+0000","last_became_active":"2024-05-27T07:08:36.869799+0000","last_became_peered":"2024-05-27T07:08:36.869799+0000","last_unstale":"2024-05-27T07:13:54.550526+0000","last_undegraded":"2024-05-27T07:13:54.550526+0000","last_fullsized":"2024-05-27T07:13:54.550526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:53.094337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573571+0000","last_change":"2024-05-27T07:08:18.308834+0000","last_active":"2024-05-27T07:13:54.573571+0000","last_peered":"2024-05-27T07:13:54.573571+0000","last_clean":"2024-05-27T07:13:54.573571+0000","last_became_active":"2024-05-27T07:08:18.306953+0000","last_became_peered":"2024-05-27T07:08:18.306953+0000","last_unstale":"2024-05-27T07:13:54.573571+0000","last_undegraded":"2024-05-27T07:13:54.573571+0000","last_fullsized":"2024-05-27T07:13:54.573571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:26:14.747631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827621+0000","last_change":"2024-05-27T07:08:22.048219+0000","last_active":"2024-05-27T07:13:54.827621+0000","last_peered":"2024-05-27T07:13:54.827621+0000","last_clean":"2024-05-27T07:13:54.827621+0000","last_became_active":"2024-05-27T07:08:22.047848+0000","last_became_peered":"2024-05-27T07:08:22.047848+0000","last_unstale":"2024-05-27T07:13:54.827621+0000","last_undegraded":"2024-05-27T07:13:54.827621+0000","last_fullsized":"2024-05-27T07:13:54.827621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:51.707603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.451911+0000","last_change":"2024-05-27T07:08:36.932747+0000","last_active":"2024-05-27T07:13:54.451911+0000","last_peered":"2024-05-27T07:13:54.451911+0000","last_clean":"2024-05-27T07:13:54.451911+0000","last_became_active":"2024-05-27T07:08:36.932050+0000","last_became_peered":"2024-05-27T07:08:36.932050+0000","last_unstale":"2024-05-27T07:13:54.451911+0000","last_undegraded":"2024-05-27T07:13:54.451911+0000","last_fullsized":"2024-05-27T07:13:54.451911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:26:07.015651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'52","reported_seq":123,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.842091+0000","last_change":"2024-05-27T07:08:18.008844+0000","last_active":"2024-05-27T07:13:54.842091+0000","last_peered":"2024-05-27T07:13:54.842091+0000","last_clean":"2024-05-27T07:13:54.842091+0000","last_became_active":"2024-05-27T07:08:18.008140+0000","last_became_peered":"2024-05-27T07:08:18.008140+0000","last_unstale":"2024-05-27T07:13:54.842091+0000","last_undegraded":"2024-05-27T07:13:54.842091+0000","last_fullsized":"2024-05-27T07:13:54.842091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:22:22.075341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":93,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453192+0000","last_change":"2024-05-27T07:08:22.050855+0000","last_active":"2024-05-27T07:13:54.453192+0000","last_peered":"2024-05-27T07:13:54.453192+0000","last_clean":"2024-05-27T07:13:54.453192+0000","last_became_active":"2024-05-27T07:08:22.048482+0000","last_became_peered":"2024-05-27T07:08:22.048482+0000","last_unstale":"2024-05-27T07:13:54.453192+0000","last_undegraded":"2024-05-27T07:13:54.453192+0000","last_fullsized":"2024-05-27T07:13:54.453192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:04.319884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.409858+0000","last_change":"2024-05-27T07:08:36.862470+0000","last_active":"2024-05-27T07:13:54.409858+0000","last_peered":"2024-05-27T07:13:54.409858+0000","last_clean":"2024-05-27T07:13:54.409858+0000","last_became_active":"2024-05-27T07:08:36.861951+0000","last_became_peered":"2024-05-27T07:08:36.861951+0000","last_unstale":"2024-05-27T07:13:54.409858+0000","last_undegraded":"2024-05-27T07:13:54.409858+0000","last_fullsized":"2024-05-27T07:13:54.409858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:56:31.659951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835285+0000","last_change":"2024-05-27T07:08:18.438518+0000","last_active":"2024-05-27T07:13:54.835285+0000","last_peered":"2024-05-27T07:13:54.835285+0000","last_clean":"2024-05-27T07:13:54.835285+0000","last_became_active":"2024-05-27T07:08:18.437588+0000","last_became_peered":"2024-05-27T07:08:18.437588+0000","last_unstale":"2024-05-27T07:13:54.835285+0000","last_undegraded":"2024-05-27T07:13:54.835285+0000","last_fullsized":"2024-05-27T07:13:54.835285+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:52:12.895009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000455+0000","last_change":"2024-05-27T07:08:22.064188+0000","last_active":"2024-05-27T07:13:55.000455+0000","last_peered":"2024-05-27T07:13:55.000455+0000","last_clean":"2024-05-27T07:13:55.000455+0000","last_became_active":"2024-05-27T07:08:22.063868+0000","last_became_peered":"2024-05-27T07:08:22.063868+0000","last_unstale":"2024-05-27T07:13:55.000455+0000","last_undegraded":"2024-05-27T07:13:55.000455+0000","last_fullsized":"2024-05-27T07:13:55.000455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:43:17.227079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834073+0000","last_change":"2024-05-27T07:08:36.934922+0000","last_active":"2024-05-27T07:13:54.834073+0000","last_peered":"2024-05-27T07:13:54.834073+0000","last_clean":"2024-05-27T07:13:54.834073+0000","last_became_active":"2024-05-27T07:08:36.934000+0000","last_became_peered":"2024-05-27T07:08:36.934000+0000","last_unstale":"2024-05-27T07:13:54.834073+0000","last_undegraded":"2024-05-27T07:13:54.834073+0000","last_fullsized":"2024-05-27T07:13:54.834073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:19:00.570557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412244+0000","last_change":"2024-05-27T07:08:18.308743+0000","last_active":"2024-05-27T07:13:54.412244+0000","last_peered":"2024-05-27T07:13:54.412244+0000","last_clean":"2024-05-27T07:13:54.412244+0000","last_became_active":"2024-05-27T07:08:18.307597+0000","last_became_peered":"2024-05-27T07:08:18.307597+0000","last_unstale":"2024-05-27T07:13:54.412244+0000","last_undegraded":"2024-05-27T07:13:54.412244+0000","last_fullsized":"2024-05-27T07:13:54.412244+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:39.914998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572836+0000","last_change":"2024-05-27T07:08:22.049132+0000","last_active":"2024-05-27T07:13:54.572836+0000","last_peered":"2024-05-27T07:13:54.572836+0000","last_clean":"2024-05-27T07:13:54.572836+0000","last_became_active":"2024-05-27T07:08:22.048606+0000","last_became_peered":"2024-05-27T07:08:22.048606+0000","last_unstale":"2024-05-27T07:13:54.572836+0000","last_undegraded":"2024-05-27T07:13:54.572836+0000","last_fullsized":"2024-05-27T07:13:54.572836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:58.253461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.409892+0000","last_change":"2024-05-27T07:08:36.867286+0000","last_active":"2024-05-27T07:13:54.409892+0000","last_peered":"2024-05-27T07:13:54.409892+0000","last_clean":"2024-05-27T07:13:54.409892+0000","last_became_active":"2024-05-27T07:08:36.866732+0000","last_became_peered":"2024-05-27T07:08:36.866732+0000","last_unstale":"2024-05-27T07:13:54.409892+0000","last_undegraded":"2024-05-27T07:13:54.409892+0000","last_fullsized":"2024-05-27T07:13:54.409892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:14:18.386441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452354+0000","last_change":"2024-05-27T07:08:18.010026+0000","last_active":"2024-05-27T07:13:54.452354+0000","last_peered":"2024-05-27T07:13:54.452354+0000","last_clean":"2024-05-27T07:13:54.452354+0000","last_became_active":"2024-05-27T07:08:18.009502+0000","last_became_peered":"2024-05-27T07:08:18.009502+0000","last_unstale":"2024-05-27T07:13:54.452354+0000","last_undegraded":"2024-05-27T07:13:54.452354+0000","last_fullsized":"2024-05-27T07:13:54.452354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:28:27.500933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550700+0000","last_change":"2024-05-27T07:08:22.054536+0000","last_active":"2024-05-27T07:13:54.550700+0000","last_peered":"2024-05-27T07:13:54.550700+0000","last_clean":"2024-05-27T07:13:54.550700+0000","last_became_active":"2024-05-27T07:08:22.054141+0000","last_became_peered":"2024-05-27T07:08:22.054141+0000","last_unstale":"2024-05-27T07:13:54.550700+0000","last_undegraded":"2024-05-27T07:13:54.550700+0000","last_fullsized":"2024-05-27T07:13:54.550700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:53.413683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.836642+0000","last_change":"2024-05-27T07:08:36.859721+0000","last_active":"2024-05-27T07:13:54.836642+0000","last_peered":"2024-05-27T07:13:54.836642+0000","last_clean":"2024-05-27T07:13:54.836642+0000","last_became_active":"2024-05-27T07:08:36.858770+0000","last_became_peered":"2024-05-27T07:08:36.858770+0000","last_unstale":"2024-05-27T07:13:54.836642+0000","last_undegraded":"2024-05-27T07:13:54.836642+0000","last_fullsized":"2024-05-27T07:13:54.836642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:32.583869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411052+0000","last_change":"2024-05-27T07:08:18.008762+0000","last_active":"2024-05-27T07:13:54.411052+0000","last_peered":"2024-05-27T07:13:54.411052+0000","last_clean":"2024-05-27T07:13:54.411052+0000","last_became_active":"2024-05-27T07:08:18.007680+0000","last_became_peered":"2024-05-27T07:08:18.007680+0000","last_unstale":"2024-05-27T07:13:54.411052+0000","last_undegraded":"2024-05-27T07:13:54.411052+0000","last_fullsized":"2024-05-27T07:13:54.411052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:22.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452861+0000","last_change":"2024-05-27T07:08:22.051955+0000","last_active":"2024-05-27T07:13:54.452861+0000","last_peered":"2024-05-27T07:13:54.452861+0000","last_clean":"2024-05-27T07:13:54.452861+0000","last_became_active":"2024-05-27T07:08:22.051618+0000","last_became_peered":"2024-05-27T07:08:22.051618+0000","last_unstale":"2024-05-27T07:13:54.452861+0000","last_undegraded":"2024-05-27T07:13:54.452861+0000","last_fullsized":"2024-05-27T07:13:54.452861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:13.219186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835409+0000","last_change":"2024-05-27T07:08:36.932048+0000","last_active":"2024-05-27T07:13:54.835409+0000","last_peered":"2024-05-27T07:13:54.835409+0000","last_clean":"2024-05-27T07:13:54.835409+0000","last_became_active":"2024-05-27T07:08:36.930777+0000","last_became_peered":"2024-05-27T07:08:36.930777+0000","last_unstale":"2024-05-27T07:13:54.835409+0000","last_undegraded":"2024-05-27T07:13:54.835409+0000","last_fullsized":"2024-05-27T07:13:54.835409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:32.284487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573040+0000","last_change":"2024-05-27T07:08:18.011343+0000","last_active":"2024-05-27T07:13:54.573040+0000","last_peered":"2024-05-27T07:13:54.573040+0000","last_clean":"2024-05-27T07:13:54.573040+0000","last_became_active":"2024-05-27T07:08:18.009988+0000","last_became_peered":"2024-05-27T07:08:18.009988+0000","last_unstale":"2024-05-27T07:13:54.573040+0000","last_undegraded":"2024-05-27T07:13:54.573040+0000","last_fullsized":"2024-05-27T07:13:54.573040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:39.620568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779670+0000","last_change":"2024-05-27T07:08:22.051630+0000","last_active":"2024-05-27T07:13:54.779670+0000","last_peered":"2024-05-27T07:13:54.779670+0000","last_clean":"2024-05-27T07:13:54.779670+0000","last_became_active":"2024-05-27T07:08:22.049140+0000","last_became_peered":"2024-05-27T07:08:22.049140+0000","last_unstale":"2024-05-27T07:13:54.779670+0000","last_undegraded":"2024-05-27T07:13:54.779670+0000","last_fullsized":"2024-05-27T07:13:54.779670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:51.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411238+0000","last_change":"2024-05-27T07:08:36.870668+0000","last_active":"2024-05-27T07:13:54.411238+0000","last_peered":"2024-05-27T07:13:54.411238+0000","last_clean":"2024-05-27T07:13:54.411238+0000","last_became_active":"2024-05-27T07:08:36.869986+0000","last_became_peered":"2024-05-27T07:08:36.869986+0000","last_unstale":"2024-05-27T07:13:54.411238+0000","last_undegraded":"2024-05-27T07:13:54.411238+0000","last_fullsized":"2024-05-27T07:13:54.411238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:59.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'9","reported_seq":304,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.831814+0000","last_change":"2024-05-27T07:08:18.439082+0000","last_active":"2024-05-27T07:13:54.831814+0000","last_peered":"2024-05-27T07:13:54.831814+0000","last_clean":"2024-05-27T07:13:54.831814+0000","last_became_active":"2024-05-27T07:08:18.437844+0000","last_became_peered":"2024-05-27T07:08:18.437844+0000","last_unstale":"2024-05-27T07:13:54.831814+0000","last_undegraded":"2024-05-27T07:13:54.831814+0000","last_fullsized":"2024-05-27T07:13:54.831814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:38:30.276747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2586,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":247,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.781353+0000","last_change":"2024-05-27T07:08:22.052308+0000","last_active":"2024-05-27T07:13:54.781353+0000","last_peered":"2024-05-27T07:13:54.781353+0000","last_clean":"2024-05-27T07:13:54.781353+0000","last_became_active":"2024-05-27T07:08:22.051615+0000","last_became_peered":"2024-05-27T07:08:22.051615+0000","last_unstale":"2024-05-27T07:13:54.781353+0000","last_undegraded":"2024-05-27T07:13:54.781353+0000","last_fullsized":"2024-05-27T07:13:54.781353+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:14.241454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.998936+0000","last_change":"2024-05-27T07:08:36.861082+0000","last_active":"2024-05-27T07:13:54.998936+0000","last_peered":"2024-05-27T07:13:54.998936+0000","last_clean":"2024-05-27T07:13:54.998936+0000","last_became_active":"2024-05-27T07:08:36.860548+0000","last_became_peered":"2024-05-27T07:08:36.860548+0000","last_unstale":"2024-05-27T07:13:54.998936+0000","last_undegraded":"2024-05-27T07:13:54.998936+0000","last_fullsized":"2024-05-27T07:13:54.998936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:46:32.446464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.838063+0000","last_change":"2024-05-27T07:08:18.018218+0000","last_active":"2024-05-27T07:13:54.838063+0000","last_peered":"2024-05-27T07:13:54.838063+0000","last_clean":"2024-05-27T07:13:54.838063+0000","last_became_active":"2024-05-27T07:08:18.017200+0000","last_became_peered":"2024-05-27T07:08:18.017200+0000","last_unstale":"2024-05-27T07:13:54.838063+0000","last_undegraded":"2024-05-27T07:13:54.838063+0000","last_fullsized":"2024-05-27T07:13:54.838063+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:22.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780497+0000","last_change":"2024-05-27T07:09:54.052709+0000","last_active":"2024-05-27T07:13:54.780497+0000","last_peered":"2024-05-27T07:13:54.780497+0000","last_clean":"2024-05-27T07:13:54.780497+0000","last_became_active":"2024-05-27T07:09:54.051425+0000","last_became_peered":"2024-05-27T07:09:54.051425+0000","last_unstale":"2024-05-27T07:13:54.780497+0000","last_undegraded":"2024-05-27T07:13:54.780497+0000","last_fullsized":"2024-05-27T07:13:54.780497+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:51:47.166228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572880+0000","last_change":"2024-05-27T07:08:53.979067+0000","last_active":"2024-05-27T07:13:54.572880+0000","last_peered":"2024-05-27T07:13:54.572880+0000","last_clean":"2024-05-27T07:13:54.572880+0000","last_became_active":"2024-05-27T07:08:53.977620+0000","last_became_peered":"2024-05-27T07:08:53.977620+0000","last_unstale":"2024-05-27T07:13:54.572880+0000","last_undegraded":"2024-05-27T07:13:54.572880+0000","last_fullsized":"2024-05-27T07:13:54.572880+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:31.619861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.828067+0000","last_change":"2024-05-27T07:08:54.001216+0000","last_active":"2024-05-27T07:13:54.828067+0000","last_peered":"2024-05-27T07:13:54.828067+0000","last_clean":"2024-05-27T07:13:54.828067+0000","last_became_active":"2024-05-27T07:08:53.982121+0000","last_became_peered":"2024-05-27T07:08:53.982121+0000","last_unstale":"2024-05-27T07:13:54.828067+0000","last_undegraded":"2024-05-27T07:13:54.828067+0000","last_fullsized":"2024-05-27T07:13:54.828067+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:01.474383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.781463+0000","last_change":"2024-05-27T07:08:22.053334+0000","last_active":"2024-05-27T07:13:54.781463+0000","last_peered":"2024-05-27T07:13:54.781463+0000","last_clean":"2024-05-27T07:13:54.781463+0000","last_became_active":"2024-05-27T07:08:22.051660+0000","last_became_peered":"2024-05-27T07:08:22.051660+0000","last_unstale":"2024-05-27T07:13:54.781463+0000","last_undegraded":"2024-05-27T07:13:54.781463+0000","last_fullsized":"2024-05-27T07:13:54.781463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:07:19.554620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550386+0000","last_change":"2024-05-27T07:08:22.064044+0000","last_active":"2024-05-27T07:13:54.550386+0000","last_peered":"2024-05-27T07:13:54.550386+0000","last_clean":"2024-05-27T07:13:54.550386+0000","last_became_active":"2024-05-27T07:08:22.063539+0000","last_became_peered":"2024-05-27T07:08:22.063539+0000","last_unstale":"2024-05-27T07:13:54.550386+0000","last_undegraded":"2024-05-27T07:13:54.550386+0000","last_fullsized":"2024-05-27T07:13:54.550386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:07.291250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410502+0000","last_change":"2024-05-27T07:08:22.059727+0000","last_active":"2024-05-27T07:13:54.410502+0000","last_peered":"2024-05-27T07:13:54.410502+0000","last_clean":"2024-05-27T07:13:54.410502+0000","last_became_active":"2024-05-27T07:08:22.058933+0000","last_became_peered":"2024-05-27T07:08:22.058933+0000","last_unstale":"2024-05-27T07:13:54.410502+0000","last_undegraded":"2024-05-27T07:13:54.410502+0000","last_fullsized":"2024-05-27T07:13:54.410502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:14.828031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410019+0000","last_change":"2024-05-27T07:08:22.044871+0000","last_active":"2024-05-27T07:13:54.410019+0000","last_peered":"2024-05-27T07:13:54.410019+0000","last_clean":"2024-05-27T07:13:54.410019+0000","last_became_active":"2024-05-27T07:08:22.044394+0000","last_became_peered":"2024-05-27T07:08:22.044394+0000","last_unstale":"2024-05-27T07:13:54.410019+0000","last_undegraded":"2024-05-27T07:13:54.410019+0000","last_fullsized":"2024-05-27T07:13:54.410019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:11.151064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.999243+0000","last_change":"2024-05-27T07:08:22.048318+0000","last_active":"2024-05-27T07:13:54.999243+0000","last_peered":"2024-05-27T07:13:54.999243+0000","last_clean":"2024-05-27T07:13:54.999243+0000","last_became_active":"2024-05-27T07:08:22.047983+0000","last_became_peered":"2024-05-27T07:08:22.047983+0000","last_unstale":"2024-05-27T07:13:54.999243+0000","last_undegraded":"2024-05-27T07:13:54.999243+0000","last_fullsized":"2024-05-27T07:13:54.999243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:11.479943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827805+0000","last_change":"2024-05-27T07:08:22.056951+0000","last_active":"2024-05-27T07:13:54.827805+0000","last_peered":"2024-05-27T07:13:54.827805+0000","last_clean":"2024-05-27T07:13:54.827805+0000","last_became_active":"2024-05-27T07:08:22.053701+0000","last_became_peered":"2024-05-27T07:08:22.053701+0000","last_unstale":"2024-05-27T07:13:54.827805+0000","last_undegraded":"2024-05-27T07:13:54.827805+0000","last_fullsized":"2024-05-27T07:13:54.827805+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:22:31.041730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551197+0000","last_change":"2024-05-27T07:08:22.062745+0000","last_active":"2024-05-27T07:13:54.551197+0000","last_peered":"2024-05-27T07:13:54.551197+0000","last_clean":"2024-05-27T07:13:54.551197+0000","last_became_active":"2024-05-27T07:08:22.062339+0000","last_became_peered":"2024-05-27T07:08:22.062339+0000","last_unstale":"2024-05-27T07:13:54.551197+0000","last_undegraded":"2024-05-27T07:13:54.551197+0000","last_fullsized":"2024-05-27T07:13:54.551197+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:58:02.934021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411753+0000","last_change":"2024-05-27T07:08:22.052621+0000","last_active":"2024-05-27T07:13:54.411753+0000","last_peered":"2024-05-27T07:13:54.411753+0000","last_clean":"2024-05-27T07:13:54.411753+0000","last_became_active":"2024-05-27T07:08:22.052070+0000","last_became_peered":"2024-05-27T07:08:22.052070+0000","last_unstale":"2024-05-27T07:13:54.411753+0000","last_undegraded":"2024-05-27T07:13:54.411753+0000","last_fullsized":"2024-05-27T07:13:54.411753+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:34.491068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453362+0000","last_change":"2024-05-27T07:08:22.055704+0000","last_active":"2024-05-27T07:13:54.453362+0000","last_peered":"2024-05-27T07:13:54.453362+0000","last_clean":"2024-05-27T07:13:54.453362+0000","last_became_active":"2024-05-27T07:08:22.055251+0000","last_became_peered":"2024-05-27T07:08:22.055251+0000","last_unstale":"2024-05-27T07:13:54.453362+0000","last_undegraded":"2024-05-27T07:13:54.453362+0000","last_fullsized":"2024-05-27T07:13:54.453362+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:28.074825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583483+0000","last_change":"2024-05-27T07:08:22.052160+0000","last_active":"2024-05-27T07:13:54.583483+0000","last_peered":"2024-05-27T07:13:54.583483+0000","last_clean":"2024-05-27T07:13:54.583483+0000","last_became_active":"2024-05-27T07:08:22.049495+0000","last_became_peered":"2024-05-27T07:08:22.049495+0000","last_unstale":"2024-05-27T07:13:54.583483+0000","last_undegraded":"2024-05-27T07:13:54.583483+0000","last_fullsized":"2024-05-27T07:13:54.583483+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:54:59.595031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452440+0000","last_change":"2024-05-27T07:08:22.053828+0000","last_active":"2024-05-27T07:13:54.452440+0000","last_peered":"2024-05-27T07:13:54.452440+0000","last_clean":"2024-05-27T07:13:54.452440+0000","last_became_active":"2024-05-27T07:08:22.053491+0000","last_became_peered":"2024-05-27T07:08:22.053491+0000","last_unstale":"2024-05-27T07:13:54.452440+0000","last_undegraded":"2024-05-27T07:13:54.452440+0000","last_fullsized":"2024-05-27T07:13:54.452440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:20.568772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411658+0000","last_change":"2024-05-27T07:08:22.050791+0000","last_active":"2024-05-27T07:13:54.411658+0000","last_peered":"2024-05-27T07:13:54.411658+0000","last_clean":"2024-05-27T07:13:54.411658+0000","last_became_active":"2024-05-27T07:08:22.050323+0000","last_became_peered":"2024-05-27T07:08:22.050323+0000","last_unstale":"2024-05-27T07:13:54.411658+0000","last_undegraded":"2024-05-27T07:13:54.411658+0000","last_fullsized":"2024-05-27T07:13:54.411658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:32.400758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827770+0000","last_change":"2024-05-27T07:08:22.041286+0000","last_active":"2024-05-27T07:13:54.827770+0000","last_peered":"2024-05-27T07:13:54.827770+0000","last_clean":"2024-05-27T07:13:54.827770+0000","last_became_active":"2024-05-27T07:08:22.040811+0000","last_became_peered":"2024-05-27T07:08:22.040811+0000","last_unstale":"2024-05-27T07:13:54.827770+0000","last_undegraded":"2024-05-27T07:13:54.827770+0000","last_fullsized":"2024-05-27T07:13:54.827770+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:40:09.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550805+0000","last_change":"2024-05-27T07:08:22.066490+0000","last_active":"2024-05-27T07:13:54.550805+0000","last_peered":"2024-05-27T07:13:54.550805+0000","last_clean":"2024-05-27T07:13:54.550805+0000","last_became_active":"2024-05-27T07:08:22.064779+0000","last_became_peered":"2024-05-27T07:08:22.064779+0000","last_unstale":"2024-05-27T07:13:54.550805+0000","last_undegraded":"2024-05-27T07:13:54.550805+0000","last_fullsized":"2024-05-27T07:13:54.550805+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:23.967856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604639,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2561,"num_read_kb":1396,"num_write":393,"num_write_kb":1480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1813917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":314,"ondisk_log_size":314,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678103,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":520386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874333,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28860,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716292,"statfs":{"total":95995035648,"available":95965483008,"internally_reserved":0,"allocated":1822720,"data_stored":1247308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36784,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708368,"statfs":{"total":95995035648,"available":95957368832,"internally_reserved":0,"allocated":1478656,"data_stored":925775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234093,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1458176,"data_stored":921363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430324,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32816,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712336,"statfs":{"total":95995035648,"available":95961432064,"internally_reserved":0,"allocated":1675264,"data_stored":1126958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593853,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":514594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1126400,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":526839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411482,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1073152,"data_stored":527126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28576,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716576,"statfs":{"total":95995035648,"available":95965773824,"internally_reserved":0,"allocated":1597440,"data_stored":1048491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953610,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":598649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790083,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1111156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":72219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":203944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:14:01.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:14:01.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.402 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.402 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.402 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:01 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 176.4M 2024-05-27T07:14:01.402 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:01 smithi043 ceph-mon[31531]: Unable to set osd_memory_target on smithi132 to 185069158: error parsing value: Value '185069158' is below minimum 939524096 2024-05-27T07:14:01.402 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.402 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:01 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi179 to 176.4M 2024-05-27T07:14:01.402 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:01 smithi043 ceph-mon[31531]: Unable to set osd_memory_target on smithi179 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-05-27T07:14:01.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:01 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi132 to 176.4M 2024-05-27T07:14:01.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:01 smithi179 ceph-mon[35888]: Unable to set osd_memory_target on smithi132 to 185069158: error parsing value: Value '185069158' is below minimum 939524096 2024-05-27T07:14:01.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:01 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi179 to 176.4M 2024-05-27T07:14:01.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:01 smithi179 ceph-mon[35888]: Unable to set osd_memory_target on smithi179 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-05-27T07:14:01.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.529+0000 7f2bf5400640 1 -- 172.21.15.43:0/1424819977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bf0105e30 msgr2=0x7f2bf0106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:01.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.529+0000 7f2bf5400640 1 --2- 172.21.15.43:0/1424819977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bf0105e30 0x7f2bf0106b30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009ca0 tx=0x7f2be002a8c0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.529+0000 7f2bf5400640 1 -- 172.21.15.43:0/1424819977 shutdown_connections 2024-05-27T07:14:01.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.529+0000 7f2bf5400640 1 --2- 172.21.15.43:0/1424819977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bf0107070 0x7f2bf0107480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.529+0000 7f2bf5400640 1 --2- 172.21.15.43:0/1424819977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2bf010bd70 0x7f2bf010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.529+0000 7f2bf5400640 1 --2- 172.21.15.43:0/1424819977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bf0105e30 0x7f2bf0106b30 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.529+0000 7f2bf5400640 1 -- 172.21.15.43:0/1424819977 >> 172.21.15.43:0/1424819977 conn(0x7f2bf0101040 msgr2=0x7f2bf01034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.529+0000 7f2bf5400640 1 -- 172.21.15.43:0/1424819977 shutdown_connections 2024-05-27T07:14:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.529+0000 7f2bf5400640 1 -- 172.21.15.43:0/1424819977 wait complete. 2024-05-27T07:14:01.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.531+0000 7f2bf5400640 1 Processor -- start 2024-05-27T07:14:01.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.531+0000 7f2bf5400640 1 -- start start 2024-05-27T07:14:01.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.531+0000 7f2bf5400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2bf0105e30 0x7f2bf019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:01.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.531+0000 7f2bf5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bf0107070 0x7f2bf019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:01.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.531+0000 7f2bf5400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bf010bd70 0x7f2bf01a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:01.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.531+0000 7f2bf5400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf0111830 con 0x7f2bf0107070 2024-05-27T07:14:01.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.531+0000 7f2bf5400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf01116b0 con 0x7f2bf010bd70 2024-05-27T07:14:01.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.531+0000 7f2bf5400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf01119b0 con 0x7f2bf0105e30 2024-05-27T07:14:01.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bede00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bf0107070 0x7f2bf019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:01.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bede00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bf0107070 0x7f2bf019ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49316/0 (socket says 172.21.15.43:49316) 2024-05-27T07:14:01.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bede00640 1 -- 172.21.15.43:0/627332169 learned_addr learned my addr 172.21.15.43:0/627332169 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:01.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bee800640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2bf0105e30 0x7f2bf019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bef200640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bf010bd70 0x7f2bf01a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bede00640 1 -- 172.21.15.43:0/627332169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2bf0105e30 msgr2=0x7f2bf019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bede00640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2bf0105e30 0x7f2bf019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bede00640 1 -- 172.21.15.43:0/627332169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bf010bd70 msgr2=0x7f2bf01a39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bede00640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bf010bd70 0x7f2bf01a39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bede00640 1 -- 172.21.15.43:0/627332169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf01a3f80 con 0x7f2bf0107070 2024-05-27T07:14:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.532+0000 7f2bef200640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bf010bd70 0x7f2bf01a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:14:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.533+0000 7f2bede00640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bf0107070 0x7f2bf019ca30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00c920 tx=0x7f2bdc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.533+0000 7f2be7400640 1 -- 172.21.15.43:0/627332169 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc007c70 con 0x7f2bf0107070 2024-05-27T07:14:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.533+0000 7f2be7400640 1 -- 172.21.15.43:0/627332169 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2bdc007e10 con 0x7f2bf0107070 2024-05-27T07:14:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.533+0000 7f2bf5400640 1 -- 172.21.15.43:0/627332169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf01a4270 con 0x7f2bf0107070 2024-05-27T07:14:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.533+0000 7f2be7400640 1 -- 172.21.15.43:0/627332169 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc018710 con 0x7f2bf0107070 2024-05-27T07:14:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.533+0000 7f2bf5400640 1 -- 172.21.15.43:0/627332169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf01a47c0 con 0x7f2bf0107070 2024-05-27T07:14:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.535+0000 7f2bf5400640 1 -- 172.21.15.43:0/627332169 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bf010ab80 con 0x7f2bf0107070 2024-05-27T07:14:01.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.539+0000 7f2be7400640 1 -- 172.21.15.43:0/627332169 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f2bdc00f450 con 0x7f2bf0107070 2024-05-27T07:14:01.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.541+0000 7f2be7400640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bd40b29f0 0x7f2bd40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:01.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.541+0000 7f2be7400640 1 -- 172.21.15.43:0/627332169 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f2bdc0e2fc0 con 0x7f2bf0107070 2024-05-27T07:14:01.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.541+0000 7f2bee800640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bd40b29f0 0x7f2bd40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:01.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.542+0000 7f2be7400640 1 -- 172.21.15.43:0/627332169 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2bdc014030 con 0x7f2bf0107070 2024-05-27T07:14:01.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.542+0000 7f2bee800640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bd40b29f0 0x7f2bd40b4f00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2be0000f80 tx=0x7f2be0031000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:01.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:01 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi132 to 176.4M 2024-05-27T07:14:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:01 smithi132 ceph-mon[35283]: Unable to set osd_memory_target on smithi132 to 185069158: error parsing value: Value '185069158' is below minimum 939524096 2024-05-27T07:14:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:01 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi179 to 176.4M 2024-05-27T07:14:01.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:01 smithi132 ceph-mon[35283]: Unable to set osd_memory_target on smithi179 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-05-27T07:14:01.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.753+0000 7f2bf5400640 1 -- 172.21.15.43:0/627332169 --> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2bf0063100 con 0x7f2bd40b29f0 2024-05-27T07:14:01.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.776+0000 7f2be7400640 1 -- 172.21.15.43:0/627332169 <== mgr.35033 v2:172.21.15.43:6800/2368318638 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339527 (secure 0 0 0) 0x7f2bf0063100 con 0x7f2bd40b29f0 2024-05-27T07:14:01.771 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:14:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-05-27T07:14:01.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.782+0000 7f2bf5400640 1 -- 172.21.15.43:0/627332169 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bd40b29f0 msgr2=0x7f2bd40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:01.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.782+0000 7f2bf5400640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bd40b29f0 0x7f2bd40b4f00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2be0000f80 tx=0x7f2be0031000 comp rx=0 tx=0).stop 2024-05-27T07:14:01.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.782+0000 7f2bf5400640 1 -- 172.21.15.43:0/627332169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bf0107070 msgr2=0x7f2bf019ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:01.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.782+0000 7f2bf5400640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bf0107070 0x7f2bf019ca30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00c920 tx=0x7f2bdc00cde0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.782+0000 7f2bf5400640 1 -- 172.21.15.43:0/627332169 shutdown_connections 2024-05-27T07:14:01.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.782+0000 7f2bf5400640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bd40b29f0 0x7f2bd40b4f00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.782+0000 7f2bf5400640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2bf010bd70 0x7f2bf01a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.782+0000 7f2bf5400640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2bf0107070 0x7f2bf019ca30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.782+0000 7f2bf5400640 1 --2- 172.21.15.43:0/627332169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2bf0105e30 0x7f2bf019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:01.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.782+0000 7f2bf5400640 1 -- 172.21.15.43:0/627332169 >> 172.21.15.43:0/627332169 conn(0x7f2bf0101040 msgr2=0x7f2bf0107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:01.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.783+0000 7f2bf5400640 1 -- 172.21.15.43:0/627332169 shutdown_connections 2024-05-27T07:14:01.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:01.783+0000 7f2bf5400640 1 -- 172.21.15.43:0/627332169 wait complete. 2024-05-27T07:14:01.801 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-27T07:14:00.440276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1195241,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2642,"num_read_kb":1465,"num_write":518,"num_write_kb":3632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953452,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369236,"kb_used_data":15836,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124572588,"statfs":{"total":1151940427776,"available":1151562330112,"internally_reserved":0,"allocated":16216064,"data_stored":9659013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.977748"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551640+0000","last_change":"2024-05-27T07:08:22.053540+0000","last_active":"2024-05-27T07:13:54.551640+0000","last_peered":"2024-05-27T07:13:54.551640+0000","last_clean":"2024-05-27T07:13:54.551640+0000","last_became_active":"2024-05-27T07:08:22.051872+0000","last_became_peered":"2024-05-27T07:08:22.051872+0000","last_unstale":"2024-05-27T07:13:54.551640+0000","last_undegraded":"2024-05-27T07:13:54.551640+0000","last_fullsized":"2024-05-27T07:13:54.551640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:02.787957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000328+0000","last_change":"2024-05-27T07:08:22.049341+0000","last_active":"2024-05-27T07:13:55.000328+0000","last_peered":"2024-05-27T07:13:55.000328+0000","last_clean":"2024-05-27T07:13:55.000328+0000","last_became_active":"2024-05-27T07:08:22.048905+0000","last_became_peered":"2024-05-27T07:08:22.048905+0000","last_unstale":"2024-05-27T07:13:55.000328+0000","last_undegraded":"2024-05-27T07:13:55.000328+0000","last_fullsized":"2024-05-27T07:13:55.000328+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:27.271336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.999833+0000","last_change":"2024-05-27T07:08:22.053925+0000","last_active":"2024-05-27T07:13:54.999833+0000","last_peered":"2024-05-27T07:13:54.999833+0000","last_clean":"2024-05-27T07:13:54.999833+0000","last_became_active":"2024-05-27T07:08:22.053409+0000","last_became_peered":"2024-05-27T07:08:22.053409+0000","last_unstale":"2024-05-27T07:13:54.999833+0000","last_undegraded":"2024-05-27T07:13:54.999833+0000","last_fullsized":"2024-05-27T07:13:54.999833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:07:55.439966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835634+0000","last_change":"2024-05-27T07:08:22.048214+0000","last_active":"2024-05-27T07:13:54.835634+0000","last_peered":"2024-05-27T07:13:54.835634+0000","last_clean":"2024-05-27T07:13:54.835634+0000","last_became_active":"2024-05-27T07:08:22.047791+0000","last_became_peered":"2024-05-27T07:08:22.047791+0000","last_unstale":"2024-05-27T07:13:54.835634+0000","last_undegraded":"2024-05-27T07:13:54.835634+0000","last_fullsized":"2024-05-27T07:13:54.835634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:23:56.381008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572980+0000","last_change":"2024-05-27T07:08:22.037883+0000","last_active":"2024-05-27T07:13:54.572980+0000","last_peered":"2024-05-27T07:13:54.572980+0000","last_clean":"2024-05-27T07:13:54.572980+0000","last_became_active":"2024-05-27T07:08:22.037089+0000","last_became_peered":"2024-05-27T07:08:22.037089+0000","last_unstale":"2024-05-27T07:13:54.572980+0000","last_undegraded":"2024-05-27T07:13:54.572980+0000","last_fullsized":"2024-05-27T07:13:54.572980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:21.929086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.478833+0000","last_change":"2024-05-27T07:08:22.061833+0000","last_active":"2024-05-27T07:13:54.478833+0000","last_peered":"2024-05-27T07:13:54.478833+0000","last_clean":"2024-05-27T07:13:54.478833+0000","last_became_active":"2024-05-27T07:08:22.057692+0000","last_became_peered":"2024-05-27T07:08:22.057692+0000","last_unstale":"2024-05-27T07:13:54.478833+0000","last_undegraded":"2024-05-27T07:13:54.478833+0000","last_fullsized":"2024-05-27T07:13:54.478833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:38:33.656872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779346+0000","last_change":"2024-05-27T07:08:22.047880+0000","last_active":"2024-05-27T07:13:54.779346+0000","last_peered":"2024-05-27T07:13:54.779346+0000","last_clean":"2024-05-27T07:13:54.779346+0000","last_became_active":"2024-05-27T07:08:22.047482+0000","last_became_peered":"2024-05-27T07:08:22.047482+0000","last_unstale":"2024-05-27T07:13:54.779346+0000","last_undegraded":"2024-05-27T07:13:54.779346+0000","last_fullsized":"2024-05-27T07:13:54.779346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:07.751588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.451911+0000","last_change":"2024-05-27T07:08:22.055970+0000","last_active":"2024-05-27T07:13:54.451911+0000","last_peered":"2024-05-27T07:13:54.451911+0000","last_clean":"2024-05-27T07:13:54.451911+0000","last_became_active":"2024-05-27T07:08:22.055591+0000","last_became_peered":"2024-05-27T07:08:22.055591+0000","last_unstale":"2024-05-27T07:13:54.451911+0000","last_undegraded":"2024-05-27T07:13:54.451911+0000","last_fullsized":"2024-05-27T07:13:54.451911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:58:46.571473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412531+0000","last_change":"2024-05-27T07:08:22.059929+0000","last_active":"2024-05-27T07:13:54.412531+0000","last_peered":"2024-05-27T07:13:54.412531+0000","last_clean":"2024-05-27T07:13:54.412531+0000","last_became_active":"2024-05-27T07:08:22.059118+0000","last_became_peered":"2024-05-27T07:08:22.059118+0000","last_unstale":"2024-05-27T07:13:54.412531+0000","last_undegraded":"2024-05-27T07:13:54.412531+0000","last_fullsized":"2024-05-27T07:13:54.412531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:29.231220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.479644+0000","last_change":"2024-05-27T07:08:53.979269+0000","last_active":"2024-05-27T07:13:54.479644+0000","last_peered":"2024-05-27T07:13:54.479644+0000","last_clean":"2024-05-27T07:13:54.479644+0000","last_became_active":"2024-05-27T07:08:53.978433+0000","last_became_peered":"2024-05-27T07:08:53.978433+0000","last_unstale":"2024-05-27T07:13:54.479644+0000","last_undegraded":"2024-05-27T07:13:54.479644+0000","last_fullsized":"2024-05-27T07:13:54.479644+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:57:37.600911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780920+0000","last_change":"2024-05-27T07:08:22.053499+0000","last_active":"2024-05-27T07:13:54.780920+0000","last_peered":"2024-05-27T07:13:54.780920+0000","last_clean":"2024-05-27T07:13:54.780920+0000","last_became_active":"2024-05-27T07:08:22.052428+0000","last_became_peered":"2024-05-27T07:08:22.052428+0000","last_unstale":"2024-05-27T07:13:54.780920+0000","last_undegraded":"2024-05-27T07:13:54.780920+0000","last_fullsized":"2024-05-27T07:13:54.780920+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:04.352262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835640+0000","last_change":"2024-05-27T07:08:22.058882+0000","last_active":"2024-05-27T07:13:54.835640+0000","last_peered":"2024-05-27T07:13:54.835640+0000","last_clean":"2024-05-27T07:13:54.835640+0000","last_became_active":"2024-05-27T07:08:22.057874+0000","last_became_peered":"2024-05-27T07:08:22.057874+0000","last_unstale":"2024-05-27T07:13:54.835640+0000","last_undegraded":"2024-05-27T07:13:54.835640+0000","last_fullsized":"2024-05-27T07:13:54.835640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:52:11.289488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.828041+0000","last_change":"2024-05-27T07:08:22.054315+0000","last_active":"2024-05-27T07:13:54.828041+0000","last_peered":"2024-05-27T07:13:54.828041+0000","last_clean":"2024-05-27T07:13:54.828041+0000","last_became_active":"2024-05-27T07:08:22.053944+0000","last_became_peered":"2024-05-27T07:08:22.053944+0000","last_unstale":"2024-05-27T07:13:54.828041+0000","last_undegraded":"2024-05-27T07:13:54.828041+0000","last_fullsized":"2024-05-27T07:13:54.828041+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:43:06.720147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.998761+0000","last_change":"2024-05-27T07:08:22.054649+0000","last_active":"2024-05-27T07:13:54.998761+0000","last_peered":"2024-05-27T07:13:54.998761+0000","last_clean":"2024-05-27T07:13:54.998761+0000","last_became_active":"2024-05-27T07:08:22.051834+0000","last_became_peered":"2024-05-27T07:08:22.051834+0000","last_unstale":"2024-05-27T07:13:54.998761+0000","last_undegraded":"2024-05-27T07:13:54.998761+0000","last_fullsized":"2024-05-27T07:13:54.998761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:16:56.893458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.998595+0000","last_change":"2024-05-27T07:08:22.052231+0000","last_active":"2024-05-27T07:13:54.998595+0000","last_peered":"2024-05-27T07:13:54.998595+0000","last_clean":"2024-05-27T07:13:54.998595+0000","last_became_active":"2024-05-27T07:08:22.051495+0000","last_became_peered":"2024-05-27T07:08:22.051495+0000","last_unstale":"2024-05-27T07:13:54.998595+0000","last_undegraded":"2024-05-27T07:13:54.998595+0000","last_fullsized":"2024-05-27T07:13:54.998595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:30:21.036874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410977+0000","last_change":"2024-05-27T07:08:22.056847+0000","last_active":"2024-05-27T07:13:54.410977+0000","last_peered":"2024-05-27T07:13:54.410977+0000","last_clean":"2024-05-27T07:13:54.410977+0000","last_became_active":"2024-05-27T07:08:22.056127+0000","last_became_peered":"2024-05-27T07:08:22.056127+0000","last_unstale":"2024-05-27T07:13:54.410977+0000","last_undegraded":"2024-05-27T07:13:54.410977+0000","last_fullsized":"2024-05-27T07:13:54.410977+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:04.757452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551330+0000","last_change":"2024-05-27T07:08:22.058078+0000","last_active":"2024-05-27T07:13:54.551330+0000","last_peered":"2024-05-27T07:13:54.551330+0000","last_clean":"2024-05-27T07:13:54.551330+0000","last_became_active":"2024-05-27T07:08:22.056886+0000","last_became_peered":"2024-05-27T07:08:22.056886+0000","last_unstale":"2024-05-27T07:13:54.551330+0000","last_undegraded":"2024-05-27T07:13:54.551330+0000","last_fullsized":"2024-05-27T07:13:54.551330+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.104572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.407614+0000","last_change":"2024-05-27T07:08:22.051492+0000","last_active":"2024-05-27T07:13:54.407614+0000","last_peered":"2024-05-27T07:13:54.407614+0000","last_clean":"2024-05-27T07:13:54.407614+0000","last_became_active":"2024-05-27T07:08:22.051091+0000","last_became_peered":"2024-05-27T07:08:22.051091+0000","last_unstale":"2024-05-27T07:13:54.407614+0000","last_undegraded":"2024-05-27T07:13:54.407614+0000","last_fullsized":"2024-05-27T07:13:54.407614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:47.816312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411384+0000","last_change":"2024-05-27T07:08:22.050714+0000","last_active":"2024-05-27T07:13:54.411384+0000","last_peered":"2024-05-27T07:13:54.411384+0000","last_clean":"2024-05-27T07:13:54.411384+0000","last_became_active":"2024-05-27T07:08:22.050322+0000","last_became_peered":"2024-05-27T07:08:22.050322+0000","last_unstale":"2024-05-27T07:13:54.411384+0000","last_undegraded":"2024-05-27T07:13:54.411384+0000","last_fullsized":"2024-05-27T07:13:54.411384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:01.867467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410108+0000","last_change":"2024-05-27T07:08:36.859916+0000","last_active":"2024-05-27T07:13:54.410108+0000","last_peered":"2024-05-27T07:13:54.410108+0000","last_clean":"2024-05-27T07:13:54.410108+0000","last_became_active":"2024-05-27T07:08:36.859429+0000","last_became_peered":"2024-05-27T07:08:36.859429+0000","last_unstale":"2024-05-27T07:13:54.410108+0000","last_undegraded":"2024-05-27T07:13:54.410108+0000","last_fullsized":"2024-05-27T07:13:54.410108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:14.026604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'14","reported_seq":305,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410345+0000","last_change":"2024-05-27T07:08:18.008912+0000","last_active":"2024-05-27T07:13:54.410345+0000","last_peered":"2024-05-27T07:13:54.410345+0000","last_clean":"2024-05-27T07:13:54.410345+0000","last_became_active":"2024-05-27T07:08:18.007603+0000","last_became_peered":"2024-05-27T07:08:18.007603+0000","last_unstale":"2024-05-27T07:13:54.410345+0000","last_undegraded":"2024-05-27T07:13:54.410345+0000","last_fullsized":"2024-05-27T07:13:54.410345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:17:40.681754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":243,"num_read_kb":243,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453059+0000","last_change":"2024-05-27T07:08:22.055426+0000","last_active":"2024-05-27T07:13:54.453059+0000","last_peered":"2024-05-27T07:13:54.453059+0000","last_clean":"2024-05-27T07:13:54.453059+0000","last_became_active":"2024-05-27T07:08:22.055063+0000","last_became_peered":"2024-05-27T07:08:22.055063+0000","last_unstale":"2024-05-27T07:13:54.453059+0000","last_undegraded":"2024-05-27T07:13:54.453059+0000","last_fullsized":"2024-05-27T07:13:54.453059+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:06:49.923192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834122+0000","last_change":"2024-05-27T07:08:53.982756+0000","last_active":"2024-05-27T07:13:54.834122+0000","last_peered":"2024-05-27T07:13:54.834122+0000","last_clean":"2024-05-27T07:13:54.834122+0000","last_became_active":"2024-05-27T07:08:53.981816+0000","last_became_peered":"2024-05-27T07:08:53.981816+0000","last_unstale":"2024-05-27T07:13:54.834122+0000","last_undegraded":"2024-05-27T07:13:54.834122+0000","last_fullsized":"2024-05-27T07:13:54.834122+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:09.984488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453284+0000","last_change":"2024-05-27T07:08:18.307659+0000","last_active":"2024-05-27T07:13:54.453284+0000","last_peered":"2024-05-27T07:13:54.453284+0000","last_clean":"2024-05-27T07:13:54.453284+0000","last_became_active":"2024-05-27T07:08:18.307239+0000","last_became_peered":"2024-05-27T07:08:18.307239+0000","last_unstale":"2024-05-27T07:13:54.453284+0000","last_undegraded":"2024-05-27T07:13:54.453284+0000","last_fullsized":"2024-05-27T07:13:54.453284+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:09:02.816813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000794+0000","last_change":"2024-05-27T07:08:22.050403+0000","last_active":"2024-05-27T07:13:55.000794+0000","last_peered":"2024-05-27T07:13:55.000794+0000","last_clean":"2024-05-27T07:13:55.000794+0000","last_became_active":"2024-05-27T07:08:22.049939+0000","last_became_peered":"2024-05-27T07:08:22.049939+0000","last_unstale":"2024-05-27T07:13:55.000794+0000","last_undegraded":"2024-05-27T07:13:55.000794+0000","last_fullsized":"2024-05-27T07:13:55.000794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:45.857508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.833966+0000","last_change":"2024-05-27T07:08:36.864458+0000","last_active":"2024-05-27T07:13:54.833966+0000","last_peered":"2024-05-27T07:13:54.833966+0000","last_clean":"2024-05-27T07:13:54.833966+0000","last_became_active":"2024-05-27T07:08:36.863894+0000","last_became_peered":"2024-05-27T07:08:36.863894+0000","last_unstale":"2024-05-27T07:13:54.833966+0000","last_undegraded":"2024-05-27T07:13:54.833966+0000","last_fullsized":"2024-05-27T07:13:54.833966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:49.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'13","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573904+0000","last_change":"2024-05-27T07:08:18.016051+0000","last_active":"2024-05-27T07:13:54.573904+0000","last_peered":"2024-05-27T07:13:54.573904+0000","last_clean":"2024-05-27T07:13:54.573904+0000","last_became_active":"2024-05-27T07:08:18.015590+0000","last_became_peered":"2024-05-27T07:08:18.015590+0000","last_unstale":"2024-05-27T07:13:54.573904+0000","last_undegraded":"2024-05-27T07:13:54.573904+0000","last_fullsized":"2024-05-27T07:13:54.573904+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:33.396027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411799+0000","last_change":"2024-05-27T07:08:22.055603+0000","last_active":"2024-05-27T07:13:54.411799+0000","last_peered":"2024-05-27T07:13:54.411799+0000","last_clean":"2024-05-27T07:13:54.411799+0000","last_became_active":"2024-05-27T07:08:22.054543+0000","last_became_peered":"2024-05-27T07:08:22.054543+0000","last_unstale":"2024-05-27T07:13:54.411799+0000","last_undegraded":"2024-05-27T07:13:54.411799+0000","last_fullsized":"2024-05-27T07:13:54.411799+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:29:01.738598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.833715+0000","last_change":"2024-05-27T07:08:36.870358+0000","last_active":"2024-05-27T07:13:54.833715+0000","last_peered":"2024-05-27T07:13:54.833715+0000","last_clean":"2024-05-27T07:13:54.833715+0000","last_became_active":"2024-05-27T07:08:36.869457+0000","last_became_peered":"2024-05-27T07:08:36.869457+0000","last_unstale":"2024-05-27T07:13:54.833715+0000","last_undegraded":"2024-05-27T07:13:54.833715+0000","last_fullsized":"2024-05-27T07:13:54.833715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:50.510098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550540+0000","last_change":"2024-05-27T07:08:53.985562+0000","last_active":"2024-05-27T07:13:54.550540+0000","last_peered":"2024-05-27T07:13:54.550540+0000","last_clean":"2024-05-27T07:13:54.550540+0000","last_became_active":"2024-05-27T07:08:53.977778+0000","last_became_peered":"2024-05-27T07:08:53.977778+0000","last_unstale":"2024-05-27T07:13:54.550540+0000","last_undegraded":"2024-05-27T07:13:54.550540+0000","last_fullsized":"2024-05-27T07:13:54.550540+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:44.455508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584633+0000","last_change":"2024-05-27T07:08:22.057622+0000","last_active":"2024-05-27T07:13:54.584633+0000","last_peered":"2024-05-27T07:13:54.584633+0000","last_clean":"2024-05-27T07:13:54.584633+0000","last_became_active":"2024-05-27T07:08:22.056544+0000","last_became_peered":"2024-05-27T07:08:22.056544+0000","last_unstale":"2024-05-27T07:13:54.584633+0000","last_undegraded":"2024-05-27T07:13:54.584633+0000","last_fullsized":"2024-05-27T07:13:54.584633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:38.484560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410554+0000","last_change":"2024-05-27T07:08:36.864984+0000","last_active":"2024-05-27T07:13:54.410554+0000","last_peered":"2024-05-27T07:13:54.410554+0000","last_clean":"2024-05-27T07:13:54.410554+0000","last_became_active":"2024-05-27T07:08:36.864302+0000","last_became_peered":"2024-05-27T07:08:36.864302+0000","last_unstale":"2024-05-27T07:13:54.410554+0000","last_undegraded":"2024-05-27T07:13:54.410554+0000","last_fullsized":"2024-05-27T07:13:54.410554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:44.183562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584240+0000","last_change":"2024-05-27T07:08:18.017918+0000","last_active":"2024-05-27T07:13:54.584240+0000","last_peered":"2024-05-27T07:13:54.584240+0000","last_clean":"2024-05-27T07:13:54.584240+0000","last_became_active":"2024-05-27T07:08:18.016174+0000","last_became_peered":"2024-05-27T07:08:18.016174+0000","last_unstale":"2024-05-27T07:13:54.584240+0000","last_undegraded":"2024-05-27T07:13:54.584240+0000","last_fullsized":"2024-05-27T07:13:54.584240+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:21.906187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573084+0000","last_change":"2024-05-27T07:08:22.043172+0000","last_active":"2024-05-27T07:13:54.573084+0000","last_peered":"2024-05-27T07:13:54.573084+0000","last_clean":"2024-05-27T07:13:54.573084+0000","last_became_active":"2024-05-27T07:08:22.041399+0000","last_became_peered":"2024-05-27T07:08:22.041399+0000","last_unstale":"2024-05-27T07:13:54.573084+0000","last_undegraded":"2024-05-27T07:13:54.573084+0000","last_fullsized":"2024-05-27T07:13:54.573084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:19.651330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410576+0000","last_change":"2024-05-27T07:08:36.866352+0000","last_active":"2024-05-27T07:13:54.410576+0000","last_peered":"2024-05-27T07:13:54.410576+0000","last_clean":"2024-05-27T07:13:54.410576+0000","last_became_active":"2024-05-27T07:08:36.865447+0000","last_became_peered":"2024-05-27T07:08:36.865447+0000","last_unstale":"2024-05-27T07:13:54.410576+0000","last_undegraded":"2024-05-27T07:13:54.410576+0000","last_fullsized":"2024-05-27T07:13:54.410576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:55:42.634374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.478763+0000","last_change":"2024-05-27T07:08:18.022142+0000","last_active":"2024-05-27T07:13:54.478763+0000","last_peered":"2024-05-27T07:13:54.478763+0000","last_clean":"2024-05-27T07:13:54.478763+0000","last_became_active":"2024-05-27T07:08:18.021453+0000","last_became_peered":"2024-05-27T07:08:18.021453+0000","last_unstale":"2024-05-27T07:13:54.478763+0000","last_undegraded":"2024-05-27T07:13:54.478763+0000","last_fullsized":"2024-05-27T07:13:54.478763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:01:09.049440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584299+0000","last_change":"2024-05-27T07:08:22.055742+0000","last_active":"2024-05-27T07:13:54.584299+0000","last_peered":"2024-05-27T07:13:54.584299+0000","last_clean":"2024-05-27T07:13:54.584299+0000","last_became_active":"2024-05-27T07:08:22.052077+0000","last_became_peered":"2024-05-27T07:08:22.052077+0000","last_unstale":"2024-05-27T07:13:54.584299+0000","last_undegraded":"2024-05-27T07:13:54.584299+0000","last_fullsized":"2024-05-27T07:13:54.584299+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:09:13.418027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551028+0000","last_change":"2024-05-27T07:08:36.874225+0000","last_active":"2024-05-27T07:13:54.551028+0000","last_peered":"2024-05-27T07:13:54.551028+0000","last_clean":"2024-05-27T07:13:54.551028+0000","last_became_active":"2024-05-27T07:08:36.873745+0000","last_became_peered":"2024-05-27T07:08:36.873745+0000","last_unstale":"2024-05-27T07:13:54.551028+0000","last_undegraded":"2024-05-27T07:13:54.551028+0000","last_fullsized":"2024-05-27T07:13:54.551028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:40.633667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827100+0000","last_change":"2024-05-27T07:08:18.009678+0000","last_active":"2024-05-27T07:13:54.827100+0000","last_peered":"2024-05-27T07:13:54.827100+0000","last_clean":"2024-05-27T07:13:54.827100+0000","last_became_active":"2024-05-27T07:08:18.006502+0000","last_became_peered":"2024-05-27T07:08:18.006502+0000","last_unstale":"2024-05-27T07:13:54.827100+0000","last_undegraded":"2024-05-27T07:13:54.827100+0000","last_fullsized":"2024-05-27T07:13:54.827100+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:37.457044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452037+0000","last_change":"2024-05-27T07:08:22.047302+0000","last_active":"2024-05-27T07:13:54.452037+0000","last_peered":"2024-05-27T07:13:54.452037+0000","last_clean":"2024-05-27T07:13:54.452037+0000","last_became_active":"2024-05-27T07:08:22.045932+0000","last_became_peered":"2024-05-27T07:08:22.045932+0000","last_unstale":"2024-05-27T07:13:54.452037+0000","last_undegraded":"2024-05-27T07:13:54.452037+0000","last_fullsized":"2024-05-27T07:13:54.452037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:27.129539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000221+0000","last_change":"2024-05-27T07:08:36.870512+0000","last_active":"2024-05-27T07:13:55.000221+0000","last_peered":"2024-05-27T07:13:55.000221+0000","last_clean":"2024-05-27T07:13:55.000221+0000","last_became_active":"2024-05-27T07:08:36.870061+0000","last_became_peered":"2024-05-27T07:08:36.870061+0000","last_unstale":"2024-05-27T07:13:55.000221+0000","last_undegraded":"2024-05-27T07:13:55.000221+0000","last_fullsized":"2024-05-27T07:13:55.000221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:03.608758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583370+0000","last_change":"2024-05-27T07:08:18.018420+0000","last_active":"2024-05-27T07:13:54.583370+0000","last_peered":"2024-05-27T07:13:54.583370+0000","last_clean":"2024-05-27T07:13:54.583370+0000","last_became_active":"2024-05-27T07:08:18.017567+0000","last_became_peered":"2024-05-27T07:08:18.017567+0000","last_unstale":"2024-05-27T07:13:54.583370+0000","last_undegraded":"2024-05-27T07:13:54.583370+0000","last_fullsized":"2024-05-27T07:13:54.583370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:37.214408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.404452+0000","last_change":"2024-05-27T07:08:22.050943+0000","last_active":"2024-05-27T07:13:54.404452+0000","last_peered":"2024-05-27T07:13:54.404452+0000","last_clean":"2024-05-27T07:13:54.404452+0000","last_became_active":"2024-05-27T07:08:22.047706+0000","last_became_peered":"2024-05-27T07:08:22.047706+0000","last_unstale":"2024-05-27T07:13:54.404452+0000","last_undegraded":"2024-05-27T07:13:54.404452+0000","last_fullsized":"2024-05-27T07:13:54.404452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:52:09.739528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452211+0000","last_change":"2024-05-27T07:08:36.867487+0000","last_active":"2024-05-27T07:13:54.452211+0000","last_peered":"2024-05-27T07:13:54.452211+0000","last_clean":"2024-05-27T07:13:54.452211+0000","last_became_active":"2024-05-27T07:08:36.867032+0000","last_became_peered":"2024-05-27T07:08:36.867032+0000","last_unstale":"2024-05-27T07:13:54.452211+0000","last_undegraded":"2024-05-27T07:13:54.452211+0000","last_fullsized":"2024-05-27T07:13:54.452211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:26:24.912382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'7","reported_seq":539,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.792547+0000","last_change":"2024-05-27T07:08:18.020168+0000","last_active":"2024-05-27T07:13:54.792547+0000","last_peered":"2024-05-27T07:13:54.792547+0000","last_clean":"2024-05-27T07:13:54.792547+0000","last_became_active":"2024-05-27T07:08:18.018309+0000","last_became_peered":"2024-05-27T07:08:18.018309+0000","last_unstale":"2024-05-27T07:13:54.792547+0000","last_undegraded":"2024-05-27T07:13:54.792547+0000","last_fullsized":"2024-05-27T07:13:54.792547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:48:16.839510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":484,"num_read_kb":484,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412438+0000","last_change":"2024-05-27T07:08:22.054888+0000","last_active":"2024-05-27T07:13:54.412438+0000","last_peered":"2024-05-27T07:13:54.412438+0000","last_clean":"2024-05-27T07:13:54.412438+0000","last_became_active":"2024-05-27T07:08:22.054417+0000","last_became_peered":"2024-05-27T07:08:22.054417+0000","last_unstale":"2024-05-27T07:13:54.412438+0000","last_undegraded":"2024-05-27T07:13:54.412438+0000","last_fullsized":"2024-05-27T07:13:54.412438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:43:00.866755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.404284+0000","last_change":"2024-05-27T07:08:36.854516+0000","last_active":"2024-05-27T07:13:54.404284+0000","last_peered":"2024-05-27T07:13:54.404284+0000","last_clean":"2024-05-27T07:13:54.404284+0000","last_became_active":"2024-05-27T07:08:36.853978+0000","last_became_peered":"2024-05-27T07:08:36.853978+0000","last_unstale":"2024-05-27T07:13:54.404284+0000","last_undegraded":"2024-05-27T07:13:54.404284+0000","last_fullsized":"2024-05-27T07:13:54.404284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:45.299100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.781001+0000","last_change":"2024-05-27T07:08:18.010539+0000","last_active":"2024-05-27T07:13:54.781001+0000","last_peered":"2024-05-27T07:13:54.781001+0000","last_clean":"2024-05-27T07:13:54.781001+0000","last_became_active":"2024-05-27T07:08:18.009643+0000","last_became_peered":"2024-05-27T07:08:18.009643+0000","last_unstale":"2024-05-27T07:13:54.781001+0000","last_undegraded":"2024-05-27T07:13:54.781001+0000","last_fullsized":"2024-05-27T07:13:54.781001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:48:40.335403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000623+0000","last_change":"2024-05-27T07:08:53.980165+0000","last_active":"2024-05-27T07:13:55.000623+0000","last_peered":"2024-05-27T07:13:55.000623+0000","last_clean":"2024-05-27T07:13:55.000623+0000","last_became_active":"2024-05-27T07:08:53.978565+0000","last_became_peered":"2024-05-27T07:08:53.978565+0000","last_unstale":"2024-05-27T07:13:55.000623+0000","last_undegraded":"2024-05-27T07:13:55.000623+0000","last_fullsized":"2024-05-27T07:13:55.000623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:56:33.513961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834628+0000","last_change":"2024-05-27T07:08:36.857814+0000","last_active":"2024-05-27T07:13:54.834628+0000","last_peered":"2024-05-27T07:13:54.834628+0000","last_clean":"2024-05-27T07:13:54.834628+0000","last_became_active":"2024-05-27T07:08:36.856790+0000","last_became_peered":"2024-05-27T07:08:36.856790+0000","last_unstale":"2024-05-27T07:13:54.834628+0000","last_undegraded":"2024-05-27T07:13:54.834628+0000","last_fullsized":"2024-05-27T07:13:54.834628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:39:10.322261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584453+0000","last_change":"2024-05-27T07:08:18.308032+0000","last_active":"2024-05-27T07:13:54.584453+0000","last_peered":"2024-05-27T07:13:54.584453+0000","last_clean":"2024-05-27T07:13:54.584453+0000","last_became_active":"2024-05-27T07:08:18.306867+0000","last_became_peered":"2024-05-27T07:08:18.306867+0000","last_unstale":"2024-05-27T07:13:54.584453+0000","last_undegraded":"2024-05-27T07:13:54.584453+0000","last_fullsized":"2024-05-27T07:13:54.584453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:37.816237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412154+0000","last_change":"2024-05-27T07:08:22.057940+0000","last_active":"2024-05-27T07:13:54.412154+0000","last_peered":"2024-05-27T07:13:54.412154+0000","last_clean":"2024-05-27T07:13:54.412154+0000","last_became_active":"2024-05-27T07:08:22.055109+0000","last_became_peered":"2024-05-27T07:08:22.055109+0000","last_unstale":"2024-05-27T07:13:54.412154+0000","last_undegraded":"2024-05-27T07:13:54.412154+0000","last_fullsized":"2024-05-27T07:13:54.412154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:46.905064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.478388+0000","last_change":"2024-05-27T07:08:53.980209+0000","last_active":"2024-05-27T07:13:54.478388+0000","last_peered":"2024-05-27T07:13:54.478388+0000","last_clean":"2024-05-27T07:13:54.478388+0000","last_became_active":"2024-05-27T07:08:53.979353+0000","last_became_peered":"2024-05-27T07:08:53.979353+0000","last_unstale":"2024-05-27T07:13:54.478388+0000","last_undegraded":"2024-05-27T07:13:54.478388+0000","last_fullsized":"2024-05-27T07:13:54.478388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:06.163563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'41","reported_seq":600,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.802694+0000","last_change":"2024-05-27T07:08:53.996491+0000","last_active":"2024-05-27T07:13:54.802694+0000","last_peered":"2024-05-27T07:13:54.802694+0000","last_clean":"2024-05-27T07:13:54.802694+0000","last_became_active":"2024-05-27T07:08:53.976013+0000","last_became_peered":"2024-05-27T07:08:53.976013+0000","last_unstale":"2024-05-27T07:13:54.802694+0000","last_undegraded":"2024-05-27T07:13:54.802694+0000","last_fullsized":"2024-05-27T07:13:54.802694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:44:16.592270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69583,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":495,"num_read_kb":311,"num_write":43,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573510+0000","last_change":"2024-05-27T07:08:22.057231+0000","last_active":"2024-05-27T07:13:54.573510+0000","last_peered":"2024-05-27T07:13:54.573510+0000","last_clean":"2024-05-27T07:13:54.573510+0000","last_became_active":"2024-05-27T07:08:22.056837+0000","last_became_peered":"2024-05-27T07:08:22.056837+0000","last_unstale":"2024-05-27T07:13:54.573510+0000","last_undegraded":"2024-05-27T07:13:54.573510+0000","last_fullsized":"2024-05-27T07:13:54.573510+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:10:28.645103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.585422+0000","last_change":"2024-05-27T07:08:36.870307+0000","last_active":"2024-05-27T07:13:54.585422+0000","last_peered":"2024-05-27T07:13:54.585422+0000","last_clean":"2024-05-27T07:13:54.585422+0000","last_became_active":"2024-05-27T07:08:36.869390+0000","last_became_peered":"2024-05-27T07:08:36.869390+0000","last_unstale":"2024-05-27T07:13:54.585422+0000","last_undegraded":"2024-05-27T07:13:54.585422+0000","last_fullsized":"2024-05-27T07:13:54.585422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:43:45.228885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410841+0000","last_change":"2024-05-27T07:08:18.014458+0000","last_active":"2024-05-27T07:13:54.410841+0000","last_peered":"2024-05-27T07:13:54.410841+0000","last_clean":"2024-05-27T07:13:54.410841+0000","last_became_active":"2024-05-27T07:08:18.013365+0000","last_became_peered":"2024-05-27T07:08:18.013365+0000","last_unstale":"2024-05-27T07:13:54.410841+0000","last_undegraded":"2024-05-27T07:13:54.410841+0000","last_fullsized":"2024-05-27T07:13:54.410841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:41.352618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827630+0000","last_change":"2024-05-27T07:08:22.043354+0000","last_active":"2024-05-27T07:13:54.827630+0000","last_peered":"2024-05-27T07:13:54.827630+0000","last_clean":"2024-05-27T07:13:54.827630+0000","last_became_active":"2024-05-27T07:08:22.042876+0000","last_became_peered":"2024-05-27T07:08:22.042876+0000","last_unstale":"2024-05-27T07:13:54.827630+0000","last_undegraded":"2024-05-27T07:13:54.827630+0000","last_fullsized":"2024-05-27T07:13:54.827630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:52.575568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550548+0000","last_change":"2024-05-27T07:08:36.869579+0000","last_active":"2024-05-27T07:13:54.550548+0000","last_peered":"2024-05-27T07:13:54.550548+0000","last_clean":"2024-05-27T07:13:54.550548+0000","last_became_active":"2024-05-27T07:08:36.869055+0000","last_became_peered":"2024-05-27T07:08:36.869055+0000","last_unstale":"2024-05-27T07:13:54.550548+0000","last_undegraded":"2024-05-27T07:13:54.550548+0000","last_fullsized":"2024-05-27T07:13:54.550548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.870079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584312+0000","last_change":"2024-05-27T07:08:18.015322+0000","last_active":"2024-05-27T07:13:54.584312+0000","last_peered":"2024-05-27T07:13:54.584312+0000","last_clean":"2024-05-27T07:13:54.584312+0000","last_became_active":"2024-05-27T07:08:18.014535+0000","last_became_peered":"2024-05-27T07:08:18.014535+0000","last_unstale":"2024-05-27T07:13:54.584312+0000","last_undegraded":"2024-05-27T07:13:54.584312+0000","last_fullsized":"2024-05-27T07:13:54.584312+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:35.116140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779909+0000","last_change":"2024-05-27T07:08:22.056181+0000","last_active":"2024-05-27T07:13:54.779909+0000","last_peered":"2024-05-27T07:13:54.779909+0000","last_clean":"2024-05-27T07:13:54.779909+0000","last_became_active":"2024-05-27T07:08:22.055755+0000","last_became_peered":"2024-05-27T07:08:22.055755+0000","last_unstale":"2024-05-27T07:13:54.779909+0000","last_undegraded":"2024-05-27T07:13:54.779909+0000","last_fullsized":"2024-05-27T07:13:54.779909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:38:35.634116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.409346+0000","last_change":"2024-05-27T07:08:36.863184+0000","last_active":"2024-05-27T07:13:54.409346+0000","last_peered":"2024-05-27T07:13:54.409346+0000","last_clean":"2024-05-27T07:13:54.409346+0000","last_became_active":"2024-05-27T07:08:36.862760+0000","last_became_peered":"2024-05-27T07:08:36.862760+0000","last_unstale":"2024-05-27T07:13:54.409346+0000","last_undegraded":"2024-05-27T07:13:54.409346+0000","last_fullsized":"2024-05-27T07:13:54.409346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:48.379157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":302,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834697+0000","last_change":"2024-05-27T07:08:18.019119+0000","last_active":"2024-05-27T07:13:54.834697+0000","last_peered":"2024-05-27T07:13:54.834697+0000","last_clean":"2024-05-27T07:13:54.834697+0000","last_became_active":"2024-05-27T07:08:18.018408+0000","last_became_peered":"2024-05-27T07:08:18.018408+0000","last_unstale":"2024-05-27T07:13:54.834697+0000","last_undegraded":"2024-05-27T07:13:54.834697+0000","last_fullsized":"2024-05-27T07:13:54.834697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:19.978328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":248,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779408+0000","last_change":"2024-05-27T07:08:22.059464+0000","last_active":"2024-05-27T07:13:54.779408+0000","last_peered":"2024-05-27T07:13:54.779408+0000","last_clean":"2024-05-27T07:13:54.779408+0000","last_became_active":"2024-05-27T07:08:22.058890+0000","last_became_peered":"2024-05-27T07:08:22.058890+0000","last_unstale":"2024-05-27T07:13:54.779408+0000","last_undegraded":"2024-05-27T07:13:54.779408+0000","last_fullsized":"2024-05-27T07:13:54.779408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:30.122552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.828201+0000","last_change":"2024-05-27T07:08:36.865884+0000","last_active":"2024-05-27T07:13:54.828201+0000","last_peered":"2024-05-27T07:13:54.828201+0000","last_clean":"2024-05-27T07:13:54.828201+0000","last_became_active":"2024-05-27T07:08:36.865443+0000","last_became_peered":"2024-05-27T07:08:36.865443+0000","last_unstale":"2024-05-27T07:13:54.828201+0000","last_undegraded":"2024-05-27T07:13:54.828201+0000","last_fullsized":"2024-05-27T07:13:54.828201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:47:15.026632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550142+0000","last_change":"2024-05-27T07:08:53.994809+0000","last_active":"2024-05-27T07:13:54.550142+0000","last_peered":"2024-05-27T07:13:54.550142+0000","last_clean":"2024-05-27T07:13:54.550142+0000","last_became_active":"2024-05-27T07:08:53.977628+0000","last_became_peered":"2024-05-27T07:08:53.977628+0000","last_unstale":"2024-05-27T07:13:54.550142+0000","last_undegraded":"2024-05-27T07:13:54.550142+0000","last_fullsized":"2024-05-27T07:13:54.550142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:13:37.176319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583849+0000","last_change":"2024-05-27T07:08:22.050733+0000","last_active":"2024-05-27T07:13:54.583849+0000","last_peered":"2024-05-27T07:13:54.583849+0000","last_clean":"2024-05-27T07:13:54.583849+0000","last_became_active":"2024-05-27T07:08:22.050321+0000","last_became_peered":"2024-05-27T07:08:22.050321+0000","last_unstale":"2024-05-27T07:13:54.583849+0000","last_undegraded":"2024-05-27T07:13:54.583849+0000","last_fullsized":"2024-05-27T07:13:54.583849+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:44.590595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583996+0000","last_change":"2024-05-27T07:08:36.867784+0000","last_active":"2024-05-27T07:13:54.583996+0000","last_peered":"2024-05-27T07:13:54.583996+0000","last_clean":"2024-05-27T07:13:54.583996+0000","last_became_active":"2024-05-27T07:08:36.867290+0000","last_became_peered":"2024-05-27T07:08:36.867290+0000","last_unstale":"2024-05-27T07:13:54.583996+0000","last_undegraded":"2024-05-27T07:13:54.583996+0000","last_fullsized":"2024-05-27T07:13:54.583996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:44:07.649640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573792+0000","last_change":"2024-05-27T07:08:18.013193+0000","last_active":"2024-05-27T07:13:54.573792+0000","last_peered":"2024-05-27T07:13:54.573792+0000","last_clean":"2024-05-27T07:13:54.573792+0000","last_became_active":"2024-05-27T07:08:18.011767+0000","last_became_peered":"2024-05-27T07:08:18.011767+0000","last_unstale":"2024-05-27T07:13:54.573792+0000","last_undegraded":"2024-05-27T07:13:54.573792+0000","last_fullsized":"2024-05-27T07:13:54.573792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:15:03.679719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550880+0000","last_change":"2024-05-27T07:08:22.067145+0000","last_active":"2024-05-27T07:13:54.550880+0000","last_peered":"2024-05-27T07:13:54.550880+0000","last_clean":"2024-05-27T07:13:54.550880+0000","last_became_active":"2024-05-27T07:08:22.066750+0000","last_became_peered":"2024-05-27T07:08:22.066750+0000","last_unstale":"2024-05-27T07:13:54.550880+0000","last_undegraded":"2024-05-27T07:13:54.550880+0000","last_fullsized":"2024-05-27T07:13:54.550880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:41:35.333311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572917+0000","last_change":"2024-05-27T07:08:53.982497+0000","last_active":"2024-05-27T07:13:54.572917+0000","last_peered":"2024-05-27T07:13:54.572917+0000","last_clean":"2024-05-27T07:13:54.572917+0000","last_became_active":"2024-05-27T07:08:53.981920+0000","last_became_peered":"2024-05-27T07:08:53.981920+0000","last_unstale":"2024-05-27T07:13:54.572917+0000","last_undegraded":"2024-05-27T07:13:54.572917+0000","last_fullsized":"2024-05-27T07:13:54.572917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:43:40.095357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.406501+0000","last_change":"2024-05-27T07:08:18.308330+0000","last_active":"2024-05-27T07:13:54.406501+0000","last_peered":"2024-05-27T07:13:54.406501+0000","last_clean":"2024-05-27T07:13:54.406501+0000","last_became_active":"2024-05-27T07:08:18.306961+0000","last_became_peered":"2024-05-27T07:08:18.306961+0000","last_unstale":"2024-05-27T07:13:54.406501+0000","last_undegraded":"2024-05-27T07:13:54.406501+0000","last_fullsized":"2024-05-27T07:13:54.406501+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:27.875936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.479309+0000","last_change":"2024-05-27T07:08:22.047562+0000","last_active":"2024-05-27T07:13:54.479309+0000","last_peered":"2024-05-27T07:13:54.479309+0000","last_clean":"2024-05-27T07:13:54.479309+0000","last_became_active":"2024-05-27T07:08:22.046977+0000","last_became_peered":"2024-05-27T07:08:22.046977+0000","last_unstale":"2024-05-27T07:13:54.479309+0000","last_undegraded":"2024-05-27T07:13:54.479309+0000","last_fullsized":"2024-05-27T07:13:54.479309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:49:43.376617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827427+0000","last_change":"2024-05-27T07:08:36.867382+0000","last_active":"2024-05-27T07:13:54.827427+0000","last_peered":"2024-05-27T07:13:54.827427+0000","last_clean":"2024-05-27T07:13:54.827427+0000","last_became_active":"2024-05-27T07:08:36.866296+0000","last_became_peered":"2024-05-27T07:08:36.866296+0000","last_unstale":"2024-05-27T07:13:54.827427+0000","last_undegraded":"2024-05-27T07:13:54.827427+0000","last_fullsized":"2024-05-27T07:13:54.827427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:14.305169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'7","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780505+0000","last_change":"2024-05-27T07:08:18.012031+0000","last_active":"2024-05-27T07:13:54.780505+0000","last_peered":"2024-05-27T07:13:54.780505+0000","last_clean":"2024-05-27T07:13:54.780505+0000","last_became_active":"2024-05-27T07:08:18.009593+0000","last_became_peered":"2024-05-27T07:08:18.009593+0000","last_unstale":"2024-05-27T07:13:54.780505+0000","last_undegraded":"2024-05-27T07:13:54.780505+0000","last_fullsized":"2024-05-27T07:13:54.780505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:25:22.316575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1472,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.479213+0000","last_change":"2024-05-27T07:08:22.061136+0000","last_active":"2024-05-27T07:13:54.479213+0000","last_peered":"2024-05-27T07:13:54.479213+0000","last_clean":"2024-05-27T07:13:54.479213+0000","last_became_active":"2024-05-27T07:08:22.060735+0000","last_became_peered":"2024-05-27T07:08:22.060735+0000","last_unstale":"2024-05-27T07:13:54.479213+0000","last_undegraded":"2024-05-27T07:13:54.479213+0000","last_fullsized":"2024-05-27T07:13:54.479213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:50.392130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827972+0000","last_change":"2024-05-27T07:08:36.932187+0000","last_active":"2024-05-27T07:13:54.827972+0000","last_peered":"2024-05-27T07:13:54.827972+0000","last_clean":"2024-05-27T07:13:54.827972+0000","last_became_active":"2024-05-27T07:08:36.931214+0000","last_became_peered":"2024-05-27T07:08:36.931214+0000","last_unstale":"2024-05-27T07:13:54.827972+0000","last_undegraded":"2024-05-27T07:13:54.827972+0000","last_fullsized":"2024-05-27T07:13:54.827972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:20:59.645352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.404178+0000","last_change":"2024-05-27T07:09:54.082674+0000","last_active":"2024-05-27T07:13:54.404178+0000","last_peered":"2024-05-27T07:13:54.404178+0000","last_clean":"2024-05-27T07:13:54.404178+0000","last_became_active":"2024-05-27T07:09:54.081616+0000","last_became_peered":"2024-05-27T07:09:54.081616+0000","last_unstale":"2024-05-27T07:13:54.404178+0000","last_undegraded":"2024-05-27T07:13:54.404178+0000","last_fullsized":"2024-05-27T07:13:54.404178+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:38:11.230026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827139+0000","last_change":"2024-05-27T07:08:18.306439+0000","last_active":"2024-05-27T07:13:54.827139+0000","last_peered":"2024-05-27T07:13:54.827139+0000","last_clean":"2024-05-27T07:13:54.827139+0000","last_became_active":"2024-05-27T07:08:18.305978+0000","last_became_peered":"2024-05-27T07:08:18.305978+0000","last_unstale":"2024-05-27T07:13:54.827139+0000","last_undegraded":"2024-05-27T07:13:54.827139+0000","last_fullsized":"2024-05-27T07:13:54.827139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:20:22.648922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":243,"num_read_kb":243,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780681+0000","last_change":"2024-05-27T07:08:36.866247+0000","last_active":"2024-05-27T07:13:54.780681+0000","last_peered":"2024-05-27T07:13:54.780681+0000","last_clean":"2024-05-27T07:13:54.780681+0000","last_became_active":"2024-05-27T07:08:36.865516+0000","last_became_peered":"2024-05-27T07:08:36.865516+0000","last_unstale":"2024-05-27T07:13:54.780681+0000","last_undegraded":"2024-05-27T07:13:54.780681+0000","last_fullsized":"2024-05-27T07:13:54.780681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:39:41.235352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452547+0000","last_change":"2024-05-27T07:08:22.045312+0000","last_active":"2024-05-27T07:13:54.452547+0000","last_peered":"2024-05-27T07:13:54.452547+0000","last_clean":"2024-05-27T07:13:54.452547+0000","last_became_active":"2024-05-27T07:08:22.044724+0000","last_became_peered":"2024-05-27T07:08:22.044724+0000","last_unstale":"2024-05-27T07:13:54.452547+0000","last_undegraded":"2024-05-27T07:13:54.452547+0000","last_fullsized":"2024-05-27T07:13:54.452547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:05.487794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834127+0000","last_change":"2024-05-27T07:08:18.019333+0000","last_active":"2024-05-27T07:13:54.834127+0000","last_peered":"2024-05-27T07:13:54.834127+0000","last_clean":"2024-05-27T07:13:54.834127+0000","last_became_active":"2024-05-27T07:08:18.018816+0000","last_became_peered":"2024-05-27T07:08:18.018816+0000","last_unstale":"2024-05-27T07:13:54.834127+0000","last_undegraded":"2024-05-27T07:13:54.834127+0000","last_fullsized":"2024-05-27T07:13:54.834127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:09.421346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.778974+0000","last_change":"2024-05-27T07:06:16.865587+0000","last_active":"2024-05-27T07:13:54.778974+0000","last_peered":"2024-05-27T07:13:54.778974+0000","last_clean":"2024-05-27T07:13:54.778974+0000","last_became_active":"2024-05-27T07:06:16.850646+0000","last_became_peered":"2024-05-27T07:06:16.850646+0000","last_unstale":"2024-05-27T07:13:54.778974+0000","last_undegraded":"2024-05-27T07:13:54.778974+0000","last_fullsized":"2024-05-27T07:13:54.778974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_clean_scrub_stamp":"2024-05-27T07:01:14.931400+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:37.272619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584459+0000","last_change":"2024-05-27T07:08:36.934696+0000","last_active":"2024-05-27T07:13:54.584459+0000","last_peered":"2024-05-27T07:13:54.584459+0000","last_clean":"2024-05-27T07:13:54.584459+0000","last_became_active":"2024-05-27T07:08:36.934008+0000","last_became_peered":"2024-05-27T07:08:36.934008+0000","last_unstale":"2024-05-27T07:13:54.584459+0000","last_undegraded":"2024-05-27T07:13:54.584459+0000","last_fullsized":"2024-05-27T07:13:54.584459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:39.099180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.826925+0000","last_change":"2024-05-27T07:08:22.054479+0000","last_active":"2024-05-27T07:13:54.826925+0000","last_peered":"2024-05-27T07:13:54.826925+0000","last_clean":"2024-05-27T07:13:54.826925+0000","last_became_active":"2024-05-27T07:08:22.054127+0000","last_became_peered":"2024-05-27T07:08:22.054127+0000","last_unstale":"2024-05-27T07:13:54.826925+0000","last_undegraded":"2024-05-27T07:13:54.826925+0000","last_fullsized":"2024-05-27T07:13:54.826925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:37:25.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779394+0000","last_change":"2024-05-27T07:08:18.010944+0000","last_active":"2024-05-27T07:13:54.779394+0000","last_peered":"2024-05-27T07:13:54.779394+0000","last_clean":"2024-05-27T07:13:54.779394+0000","last_became_active":"2024-05-27T07:08:18.009816+0000","last_became_peered":"2024-05-27T07:08:18.009816+0000","last_unstale":"2024-05-27T07:13:54.779394+0000","last_undegraded":"2024-05-27T07:13:54.779394+0000","last_fullsized":"2024-05-27T07:13:54.779394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:34:26.061787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550526+0000","last_change":"2024-05-27T07:08:36.870618+0000","last_active":"2024-05-27T07:13:54.550526+0000","last_peered":"2024-05-27T07:13:54.550526+0000","last_clean":"2024-05-27T07:13:54.550526+0000","last_became_active":"2024-05-27T07:08:36.869799+0000","last_became_peered":"2024-05-27T07:08:36.869799+0000","last_unstale":"2024-05-27T07:13:54.550526+0000","last_undegraded":"2024-05-27T07:13:54.550526+0000","last_fullsized":"2024-05-27T07:13:54.550526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:53.094337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573571+0000","last_change":"2024-05-27T07:08:18.308834+0000","last_active":"2024-05-27T07:13:54.573571+0000","last_peered":"2024-05-27T07:13:54.573571+0000","last_clean":"2024-05-27T07:13:54.573571+0000","last_became_active":"2024-05-27T07:08:18.306953+0000","last_became_peered":"2024-05-27T07:08:18.306953+0000","last_unstale":"2024-05-27T07:13:54.573571+0000","last_undegraded":"2024-05-27T07:13:54.573571+0000","last_fullsized":"2024-05-27T07:13:54.573571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:26:14.747631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827621+0000","last_change":"2024-05-27T07:08:22.048219+0000","last_active":"2024-05-27T07:13:54.827621+0000","last_peered":"2024-05-27T07:13:54.827621+0000","last_clean":"2024-05-27T07:13:54.827621+0000","last_became_active":"2024-05-27T07:08:22.047848+0000","last_became_peered":"2024-05-27T07:08:22.047848+0000","last_unstale":"2024-05-27T07:13:54.827621+0000","last_undegraded":"2024-05-27T07:13:54.827621+0000","last_fullsized":"2024-05-27T07:13:54.827621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:51.707603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.451911+0000","last_change":"2024-05-27T07:08:36.932747+0000","last_active":"2024-05-27T07:13:54.451911+0000","last_peered":"2024-05-27T07:13:54.451911+0000","last_clean":"2024-05-27T07:13:54.451911+0000","last_became_active":"2024-05-27T07:08:36.932050+0000","last_became_peered":"2024-05-27T07:08:36.932050+0000","last_unstale":"2024-05-27T07:13:54.451911+0000","last_undegraded":"2024-05-27T07:13:54.451911+0000","last_fullsized":"2024-05-27T07:13:54.451911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:26:07.015651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'52","reported_seq":123,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.842091+0000","last_change":"2024-05-27T07:08:18.008844+0000","last_active":"2024-05-27T07:13:54.842091+0000","last_peered":"2024-05-27T07:13:54.842091+0000","last_clean":"2024-05-27T07:13:54.842091+0000","last_became_active":"2024-05-27T07:08:18.008140+0000","last_became_peered":"2024-05-27T07:08:18.008140+0000","last_unstale":"2024-05-27T07:13:54.842091+0000","last_undegraded":"2024-05-27T07:13:54.842091+0000","last_fullsized":"2024-05-27T07:13:54.842091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:22:22.075341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":93,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453192+0000","last_change":"2024-05-27T07:08:22.050855+0000","last_active":"2024-05-27T07:13:54.453192+0000","last_peered":"2024-05-27T07:13:54.453192+0000","last_clean":"2024-05-27T07:13:54.453192+0000","last_became_active":"2024-05-27T07:08:22.048482+0000","last_became_peered":"2024-05-27T07:08:22.048482+0000","last_unstale":"2024-05-27T07:13:54.453192+0000","last_undegraded":"2024-05-27T07:13:54.453192+0000","last_fullsized":"2024-05-27T07:13:54.453192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:04.319884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.409858+0000","last_change":"2024-05-27T07:08:36.862470+0000","last_active":"2024-05-27T07:13:54.409858+0000","last_peered":"2024-05-27T07:13:54.409858+0000","last_clean":"2024-05-27T07:13:54.409858+0000","last_became_active":"2024-05-27T07:08:36.861951+0000","last_became_peered":"2024-05-27T07:08:36.861951+0000","last_unstale":"2024-05-27T07:13:54.409858+0000","last_undegraded":"2024-05-27T07:13:54.409858+0000","last_fullsized":"2024-05-27T07:13:54.409858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:56:31.659951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835285+0000","last_change":"2024-05-27T07:08:18.438518+0000","last_active":"2024-05-27T07:13:54.835285+0000","last_peered":"2024-05-27T07:13:54.835285+0000","last_clean":"2024-05-27T07:13:54.835285+0000","last_became_active":"2024-05-27T07:08:18.437588+0000","last_became_peered":"2024-05-27T07:08:18.437588+0000","last_unstale":"2024-05-27T07:13:54.835285+0000","last_undegraded":"2024-05-27T07:13:54.835285+0000","last_fullsized":"2024-05-27T07:13:54.835285+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:52:12.895009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000455+0000","last_change":"2024-05-27T07:08:22.064188+0000","last_active":"2024-05-27T07:13:55.000455+0000","last_peered":"2024-05-27T07:13:55.000455+0000","last_clean":"2024-05-27T07:13:55.000455+0000","last_became_active":"2024-05-27T07:08:22.063868+0000","last_became_peered":"2024-05-27T07:08:22.063868+0000","last_unstale":"2024-05-27T07:13:55.000455+0000","last_undegraded":"2024-05-27T07:13:55.000455+0000","last_fullsized":"2024-05-27T07:13:55.000455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:43:17.227079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834073+0000","last_change":"2024-05-27T07:08:36.934922+0000","last_active":"2024-05-27T07:13:54.834073+0000","last_peered":"2024-05-27T07:13:54.834073+0000","last_clean":"2024-05-27T07:13:54.834073+0000","last_became_active":"2024-05-27T07:08:36.934000+0000","last_became_peered":"2024-05-27T07:08:36.934000+0000","last_unstale":"2024-05-27T07:13:54.834073+0000","last_undegraded":"2024-05-27T07:13:54.834073+0000","last_fullsized":"2024-05-27T07:13:54.834073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:19:00.570557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412244+0000","last_change":"2024-05-27T07:08:18.308743+0000","last_active":"2024-05-27T07:13:54.412244+0000","last_peered":"2024-05-27T07:13:54.412244+0000","last_clean":"2024-05-27T07:13:54.412244+0000","last_became_active":"2024-05-27T07:08:18.307597+0000","last_became_peered":"2024-05-27T07:08:18.307597+0000","last_unstale":"2024-05-27T07:13:54.412244+0000","last_undegraded":"2024-05-27T07:13:54.412244+0000","last_fullsized":"2024-05-27T07:13:54.412244+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:39.914998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572836+0000","last_change":"2024-05-27T07:08:22.049132+0000","last_active":"2024-05-27T07:13:54.572836+0000","last_peered":"2024-05-27T07:13:54.572836+0000","last_clean":"2024-05-27T07:13:54.572836+0000","last_became_active":"2024-05-27T07:08:22.048606+0000","last_became_peered":"2024-05-27T07:08:22.048606+0000","last_unstale":"2024-05-27T07:13:54.572836+0000","last_undegraded":"2024-05-27T07:13:54.572836+0000","last_fullsized":"2024-05-27T07:13:54.572836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:58.253461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.409892+0000","last_change":"2024-05-27T07:08:36.867286+0000","last_active":"2024-05-27T07:13:54.409892+0000","last_peered":"2024-05-27T07:13:54.409892+0000","last_clean":"2024-05-27T07:13:54.409892+0000","last_became_active":"2024-05-27T07:08:36.866732+0000","last_became_peered":"2024-05-27T07:08:36.866732+0000","last_unstale":"2024-05-27T07:13:54.409892+0000","last_undegraded":"2024-05-27T07:13:54.409892+0000","last_fullsized":"2024-05-27T07:13:54.409892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:14:18.386441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452354+0000","last_change":"2024-05-27T07:08:18.010026+0000","last_active":"2024-05-27T07:13:54.452354+0000","last_peered":"2024-05-27T07:13:54.452354+0000","last_clean":"2024-05-27T07:13:54.452354+0000","last_became_active":"2024-05-27T07:08:18.009502+0000","last_became_peered":"2024-05-27T07:08:18.009502+0000","last_unstale":"2024-05-27T07:13:54.452354+0000","last_undegraded":"2024-05-27T07:13:54.452354+0000","last_fullsized":"2024-05-27T07:13:54.452354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:28:27.500933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550700+0000","last_change":"2024-05-27T07:08:22.054536+0000","last_active":"2024-05-27T07:13:54.550700+0000","last_peered":"2024-05-27T07:13:54.550700+0000","last_clean":"2024-05-27T07:13:54.550700+0000","last_became_active":"2024-05-27T07:08:22.054141+0000","last_became_peered":"2024-05-27T07:08:22.054141+0000","last_unstale":"2024-05-27T07:13:54.550700+0000","last_undegraded":"2024-05-27T07:13:54.550700+0000","last_fullsized":"2024-05-27T07:13:54.550700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:53.413683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.836642+0000","last_change":"2024-05-27T07:08:36.859721+0000","last_active":"2024-05-27T07:13:54.836642+0000","last_peered":"2024-05-27T07:13:54.836642+0000","last_clean":"2024-05-27T07:13:54.836642+0000","last_became_active":"2024-05-27T07:08:36.858770+0000","last_became_peered":"2024-05-27T07:08:36.858770+0000","last_unstale":"2024-05-27T07:13:54.836642+0000","last_undegraded":"2024-05-27T07:13:54.836642+0000","last_fullsized":"2024-05-27T07:13:54.836642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:32.583869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411052+0000","last_change":"2024-05-27T07:08:18.008762+0000","last_active":"2024-05-27T07:13:54.411052+0000","last_peered":"2024-05-27T07:13:54.411052+0000","last_clean":"2024-05-27T07:13:54.411052+0000","last_became_active":"2024-05-27T07:08:18.007680+0000","last_became_peered":"2024-05-27T07:08:18.007680+0000","last_unstale":"2024-05-27T07:13:54.411052+0000","last_undegraded":"2024-05-27T07:13:54.411052+0000","last_fullsized":"2024-05-27T07:13:54.411052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:22.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452861+0000","last_change":"2024-05-27T07:08:22.051955+0000","last_active":"2024-05-27T07:13:54.452861+0000","last_peered":"2024-05-27T07:13:54.452861+0000","last_clean":"2024-05-27T07:13:54.452861+0000","last_became_active":"2024-05-27T07:08:22.051618+0000","last_became_peered":"2024-05-27T07:08:22.051618+0000","last_unstale":"2024-05-27T07:13:54.452861+0000","last_undegraded":"2024-05-27T07:13:54.452861+0000","last_fullsized":"2024-05-27T07:13:54.452861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:13.219186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835409+0000","last_change":"2024-05-27T07:08:36.932048+0000","last_active":"2024-05-27T07:13:54.835409+0000","last_peered":"2024-05-27T07:13:54.835409+0000","last_clean":"2024-05-27T07:13:54.835409+0000","last_became_active":"2024-05-27T07:08:36.930777+0000","last_became_peered":"2024-05-27T07:08:36.930777+0000","last_unstale":"2024-05-27T07:13:54.835409+0000","last_undegraded":"2024-05-27T07:13:54.835409+0000","last_fullsized":"2024-05-27T07:13:54.835409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:32.284487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573040+0000","last_change":"2024-05-27T07:08:18.011343+0000","last_active":"2024-05-27T07:13:54.573040+0000","last_peered":"2024-05-27T07:13:54.573040+0000","last_clean":"2024-05-27T07:13:54.573040+0000","last_became_active":"2024-05-27T07:08:18.009988+0000","last_became_peered":"2024-05-27T07:08:18.009988+0000","last_unstale":"2024-05-27T07:13:54.573040+0000","last_undegraded":"2024-05-27T07:13:54.573040+0000","last_fullsized":"2024-05-27T07:13:54.573040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:39.620568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779670+0000","last_change":"2024-05-27T07:08:22.051630+0000","last_active":"2024-05-27T07:13:54.779670+0000","last_peered":"2024-05-27T07:13:54.779670+0000","last_clean":"2024-05-27T07:13:54.779670+0000","last_became_active":"2024-05-27T07:08:22.049140+0000","last_became_peered":"2024-05-27T07:08:22.049140+0000","last_unstale":"2024-05-27T07:13:54.779670+0000","last_undegraded":"2024-05-27T07:13:54.779670+0000","last_fullsized":"2024-05-27T07:13:54.779670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:51.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411238+0000","last_change":"2024-05-27T07:08:36.870668+0000","last_active":"2024-05-27T07:13:54.411238+0000","last_peered":"2024-05-27T07:13:54.411238+0000","last_clean":"2024-05-27T07:13:54.411238+0000","last_became_active":"2024-05-27T07:08:36.869986+0000","last_became_peered":"2024-05-27T07:08:36.869986+0000","last_unstale":"2024-05-27T07:13:54.411238+0000","last_undegraded":"2024-05-27T07:13:54.411238+0000","last_fullsized":"2024-05-27T07:13:54.411238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:59.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'9","reported_seq":304,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.831814+0000","last_change":"2024-05-27T07:08:18.439082+0000","last_active":"2024-05-27T07:13:54.831814+0000","last_peered":"2024-05-27T07:13:54.831814+0000","last_clean":"2024-05-27T07:13:54.831814+0000","last_became_active":"2024-05-27T07:08:18.437844+0000","last_became_peered":"2024-05-27T07:08:18.437844+0000","last_unstale":"2024-05-27T07:13:54.831814+0000","last_undegraded":"2024-05-27T07:13:54.831814+0000","last_fullsized":"2024-05-27T07:13:54.831814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:38:30.276747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2586,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":247,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.781353+0000","last_change":"2024-05-27T07:08:22.052308+0000","last_active":"2024-05-27T07:13:54.781353+0000","last_peered":"2024-05-27T07:13:54.781353+0000","last_clean":"2024-05-27T07:13:54.781353+0000","last_became_active":"2024-05-27T07:08:22.051615+0000","last_became_peered":"2024-05-27T07:08:22.051615+0000","last_unstale":"2024-05-27T07:13:54.781353+0000","last_undegraded":"2024-05-27T07:13:54.781353+0000","last_fullsized":"2024-05-27T07:13:54.781353+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:14.241454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.998936+0000","last_change":"2024-05-27T07:08:36.861082+0000","last_active":"2024-05-27T07:13:54.998936+0000","last_peered":"2024-05-27T07:13:54.998936+0000","last_clean":"2024-05-27T07:13:54.998936+0000","last_became_active":"2024-05-27T07:08:36.860548+0000","last_became_peered":"2024-05-27T07:08:36.860548+0000","last_unstale":"2024-05-27T07:13:54.998936+0000","last_undegraded":"2024-05-27T07:13:54.998936+0000","last_fullsized":"2024-05-27T07:13:54.998936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:46:32.446464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.838063+0000","last_change":"2024-05-27T07:08:18.018218+0000","last_active":"2024-05-27T07:13:54.838063+0000","last_peered":"2024-05-27T07:13:54.838063+0000","last_clean":"2024-05-27T07:13:54.838063+0000","last_became_active":"2024-05-27T07:08:18.017200+0000","last_became_peered":"2024-05-27T07:08:18.017200+0000","last_unstale":"2024-05-27T07:13:54.838063+0000","last_undegraded":"2024-05-27T07:13:54.838063+0000","last_fullsized":"2024-05-27T07:13:54.838063+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:22.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780497+0000","last_change":"2024-05-27T07:09:54.052709+0000","last_active":"2024-05-27T07:13:54.780497+0000","last_peered":"2024-05-27T07:13:54.780497+0000","last_clean":"2024-05-27T07:13:54.780497+0000","last_became_active":"2024-05-27T07:09:54.051425+0000","last_became_peered":"2024-05-27T07:09:54.051425+0000","last_unstale":"2024-05-27T07:13:54.780497+0000","last_undegraded":"2024-05-27T07:13:54.780497+0000","last_fullsized":"2024-05-27T07:13:54.780497+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:51:47.166228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572880+0000","last_change":"2024-05-27T07:08:53.979067+0000","last_active":"2024-05-27T07:13:54.572880+0000","last_peered":"2024-05-27T07:13:54.572880+0000","last_clean":"2024-05-27T07:13:54.572880+0000","last_became_active":"2024-05-27T07:08:53.977620+0000","last_became_peered":"2024-05-27T07:08:53.977620+0000","last_unstale":"2024-05-27T07:13:54.572880+0000","last_undegraded":"2024-05-27T07:13:54.572880+0000","last_fullsized":"2024-05-27T07:13:54.572880+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:31.619861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.828067+0000","last_change":"2024-05-27T07:08:54.001216+0000","last_active":"2024-05-27T07:13:54.828067+0000","last_peered":"2024-05-27T07:13:54.828067+0000","last_clean":"2024-05-27T07:13:54.828067+0000","last_became_active":"2024-05-27T07:08:53.982121+0000","last_became_peered":"2024-05-27T07:08:53.982121+0000","last_unstale":"2024-05-27T07:13:54.828067+0000","last_undegraded":"2024-05-27T07:13:54.828067+0000","last_fullsized":"2024-05-27T07:13:54.828067+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:01.474383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.781463+0000","last_change":"2024-05-27T07:08:22.053334+0000","last_active":"2024-05-27T07:13:54.781463+0000","last_peered":"2024-05-27T07:13:54.781463+0000","last_clean":"2024-05-27T07:13:54.781463+0000","last_became_active":"2024-05-27T07:08:22.051660+0000","last_became_peered":"2024-05-27T07:08:22.051660+0000","last_unstale":"2024-05-27T07:13:54.781463+0000","last_undegraded":"2024-05-27T07:13:54.781463+0000","last_fullsized":"2024-05-27T07:13:54.781463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:07:19.554620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550386+0000","last_change":"2024-05-27T07:08:22.064044+0000","last_active":"2024-05-27T07:13:54.550386+0000","last_peered":"2024-05-27T07:13:54.550386+0000","last_clean":"2024-05-27T07:13:54.550386+0000","last_became_active":"2024-05-27T07:08:22.063539+0000","last_became_peered":"2024-05-27T07:08:22.063539+0000","last_unstale":"2024-05-27T07:13:54.550386+0000","last_undegraded":"2024-05-27T07:13:54.550386+0000","last_fullsized":"2024-05-27T07:13:54.550386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:07.291250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410502+0000","last_change":"2024-05-27T07:08:22.059727+0000","last_active":"2024-05-27T07:13:54.410502+0000","last_peered":"2024-05-27T07:13:54.410502+0000","last_clean":"2024-05-27T07:13:54.410502+0000","last_became_active":"2024-05-27T07:08:22.058933+0000","last_became_peered":"2024-05-27T07:08:22.058933+0000","last_unstale":"2024-05-27T07:13:54.410502+0000","last_undegraded":"2024-05-27T07:13:54.410502+0000","last_fullsized":"2024-05-27T07:13:54.410502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:14.828031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410019+0000","last_change":"2024-05-27T07:08:22.044871+0000","last_active":"2024-05-27T07:13:54.410019+0000","last_peered":"2024-05-27T07:13:54.410019+0000","last_clean":"2024-05-27T07:13:54.410019+0000","last_became_active":"2024-05-27T07:08:22.044394+0000","last_became_peered":"2024-05-27T07:08:22.044394+0000","last_unstale":"2024-05-27T07:13:54.410019+0000","last_undegraded":"2024-05-27T07:13:54.410019+0000","last_fullsized":"2024-05-27T07:13:54.410019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:11.151064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.999243+0000","last_change":"2024-05-27T07:08:22.048318+0000","last_active":"2024-05-27T07:13:54.999243+0000","last_peered":"2024-05-27T07:13:54.999243+0000","last_clean":"2024-05-27T07:13:54.999243+0000","last_became_active":"2024-05-27T07:08:22.047983+0000","last_became_peered":"2024-05-27T07:08:22.047983+0000","last_unstale":"2024-05-27T07:13:54.999243+0000","last_undegraded":"2024-05-27T07:13:54.999243+0000","last_fullsized":"2024-05-27T07:13:54.999243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:11.479943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827805+0000","last_change":"2024-05-27T07:08:22.056951+0000","last_active":"2024-05-27T07:13:54.827805+0000","last_peered":"2024-05-27T07:13:54.827805+0000","last_clean":"2024-05-27T07:13:54.827805+0000","last_became_active":"2024-05-27T07:08:22.053701+0000","last_became_peered":"2024-05-27T07:08:22.053701+0000","last_unstale":"2024-05-27T07:13:54.827805+0000","last_undegraded":"2024-05-27T07:13:54.827805+0000","last_fullsized":"2024-05-27T07:13:54.827805+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:22:31.041730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551197+0000","last_change":"2024-05-27T07:08:22.062745+0000","last_active":"2024-05-27T07:13:54.551197+0000","last_peered":"2024-05-27T07:13:54.551197+0000","last_clean":"2024-05-27T07:13:54.551197+0000","last_became_active":"2024-05-27T07:08:22.062339+0000","last_became_peered":"2024-05-27T07:08:22.062339+0000","last_unstale":"2024-05-27T07:13:54.551197+0000","last_undegraded":"2024-05-27T07:13:54.551197+0000","last_fullsized":"2024-05-27T07:13:54.551197+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:58:02.934021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411753+0000","last_change":"2024-05-27T07:08:22.052621+0000","last_active":"2024-05-27T07:13:54.411753+0000","last_peered":"2024-05-27T07:13:54.411753+0000","last_clean":"2024-05-27T07:13:54.411753+0000","last_became_active":"2024-05-27T07:08:22.052070+0000","last_became_peered":"2024-05-27T07:08:22.052070+0000","last_unstale":"2024-05-27T07:13:54.411753+0000","last_undegraded":"2024-05-27T07:13:54.411753+0000","last_fullsized":"2024-05-27T07:13:54.411753+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:34.491068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453362+0000","last_change":"2024-05-27T07:08:22.055704+0000","last_active":"2024-05-27T07:13:54.453362+0000","last_peered":"2024-05-27T07:13:54.453362+0000","last_clean":"2024-05-27T07:13:54.453362+0000","last_became_active":"2024-05-27T07:08:22.055251+0000","last_became_peered":"2024-05-27T07:08:22.055251+0000","last_unstale":"2024-05-27T07:13:54.453362+0000","last_undegraded":"2024-05-27T07:13:54.453362+0000","last_fullsized":"2024-05-27T07:13:54.453362+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:28.074825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583483+0000","last_change":"2024-05-27T07:08:22.052160+0000","last_active":"2024-05-27T07:13:54.583483+0000","last_peered":"2024-05-27T07:13:54.583483+0000","last_clean":"2024-05-27T07:13:54.583483+0000","last_became_active":"2024-05-27T07:08:22.049495+0000","last_became_peered":"2024-05-27T07:08:22.049495+0000","last_unstale":"2024-05-27T07:13:54.583483+0000","last_undegraded":"2024-05-27T07:13:54.583483+0000","last_fullsized":"2024-05-27T07:13:54.583483+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:54:59.595031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452440+0000","last_change":"2024-05-27T07:08:22.053828+0000","last_active":"2024-05-27T07:13:54.452440+0000","last_peered":"2024-05-27T07:13:54.452440+0000","last_clean":"2024-05-27T07:13:54.452440+0000","last_became_active":"2024-05-27T07:08:22.053491+0000","last_became_peered":"2024-05-27T07:08:22.053491+0000","last_unstale":"2024-05-27T07:13:54.452440+0000","last_undegraded":"2024-05-27T07:13:54.452440+0000","last_fullsized":"2024-05-27T07:13:54.452440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:20.568772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411658+0000","last_change":"2024-05-27T07:08:22.050791+0000","last_active":"2024-05-27T07:13:54.411658+0000","last_peered":"2024-05-27T07:13:54.411658+0000","last_clean":"2024-05-27T07:13:54.411658+0000","last_became_active":"2024-05-27T07:08:22.050323+0000","last_became_peered":"2024-05-27T07:08:22.050323+0000","last_unstale":"2024-05-27T07:13:54.411658+0000","last_undegraded":"2024-05-27T07:13:54.411658+0000","last_fullsized":"2024-05-27T07:13:54.411658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:32.400758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827770+0000","last_change":"2024-05-27T07:08:22.041286+0000","last_active":"2024-05-27T07:13:54.827770+0000","last_peered":"2024-05-27T07:13:54.827770+0000","last_clean":"2024-05-27T07:13:54.827770+0000","last_became_active":"2024-05-27T07:08:22.040811+0000","last_became_peered":"2024-05-27T07:08:22.040811+0000","last_unstale":"2024-05-27T07:13:54.827770+0000","last_undegraded":"2024-05-27T07:13:54.827770+0000","last_fullsized":"2024-05-27T07:13:54.827770+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:40:09.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550805+0000","last_change":"2024-05-27T07:08:22.066490+0000","last_active":"2024-05-27T07:13:54.550805+0000","last_peered":"2024-05-27T07:13:54.550805+0000","last_clean":"2024-05-27T07:13:54.550805+0000","last_became_active":"2024-05-27T07:08:22.064779+0000","last_became_peered":"2024-05-27T07:08:22.064779+0000","last_unstale":"2024-05-27T07:13:54.550805+0000","last_undegraded":"2024-05-27T07:13:54.550805+0000","last_fullsized":"2024-05-27T07:13:54.550805+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:23.967856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604639,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2561,"num_read_kb":1396,"num_write":393,"num_write_kb":1480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1813917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":314,"ondisk_log_size":314,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678103,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":520386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874333,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28860,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716292,"statfs":{"total":95995035648,"available":95965483008,"internally_reserved":0,"allocated":1822720,"data_stored":1247308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36784,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708368,"statfs":{"total":95995035648,"available":95957368832,"internally_reserved":0,"allocated":1478656,"data_stored":925775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234093,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1458176,"data_stored":921363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430324,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32816,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712336,"statfs":{"total":95995035648,"available":95961432064,"internally_reserved":0,"allocated":1675264,"data_stored":1126958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593853,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":514594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1126400,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":526839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411482,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1073152,"data_stored":527126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28576,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716576,"statfs":{"total":95995035648,"available":95965773824,"internally_reserved":0,"allocated":1597440,"data_stored":1048491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953610,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":598649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790083,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1111156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":72219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":203944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:14:01.806 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-27T07:14:01.807 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:14:02.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.151+0000 7f9217a00640 1 -- 172.21.15.43:0/1583649663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9210105e30 msgr2=0x7f9210106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:02.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.151+0000 7f9217a00640 1 --2- 172.21.15.43:0/1583649663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9210105e30 0x7f9210106b30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9204009900 tx=0x7f920402a400 comp rx=0 tx=0).stop 2024-05-27T07:14:02.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.151+0000 7f9217a00640 1 -- 172.21.15.43:0/1583649663 shutdown_connections 2024-05-27T07:14:02.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.151+0000 7f9217a00640 1 --2- 172.21.15.43:0/1583649663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f921010bd70 0x7f921010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:02.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.151+0000 7f9217a00640 1 --2- 172.21.15.43:0/1583649663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9210107070 0x7f9210107480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:02.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.151+0000 7f9217a00640 1 --2- 172.21.15.43:0/1583649663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9210105e30 0x7f9210106b30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:02.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.151+0000 7f9217a00640 1 -- 172.21.15.43:0/1583649663 >> 172.21.15.43:0/1583649663 conn(0x7f9210101040 msgr2=0x7f92101034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:02.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.151+0000 7f9217a00640 1 -- 172.21.15.43:0/1583649663 shutdown_connections 2024-05-27T07:14:02.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.151+0000 7f9217a00640 1 -- 172.21.15.43:0/1583649663 wait complete. 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.152+0000 7f9217a00640 1 Processor -- start 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f9217a00640 1 -- start start 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f9217a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9210105e30 0x7f921019c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f9217a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9210107070 0x7f921019c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f9217a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f921010bd70 0x7f92101a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f9217a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9210111810 con 0x7f9210107070 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f9217a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9210111690 con 0x7f921010bd70 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f9217a00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9210111990 con 0x7f9210105e30 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f920fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9210107070 0x7f921019c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f920fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9210107070 0x7f921019c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49326/0 (socket says 172.21.15.43:49326) 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f920fe00640 1 -- 172.21.15.43:0/2361298864 learned_addr learned my addr 172.21.15.43:0/2361298864 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:02.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.153+0000 7f9215000640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9210105e30 0x7f921019c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:02.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.154+0000 7f9215a00640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f921010bd70 0x7f92101a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:02.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.154+0000 7f920fe00640 1 -- 172.21.15.43:0/2361298864 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9210105e30 msgr2=0x7f921019c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:02.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.154+0000 7f920fe00640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9210105e30 0x7f921019c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:02.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.154+0000 7f920fe00640 1 -- 172.21.15.43:0/2361298864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f921010bd70 msgr2=0x7f92101a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:02.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.154+0000 7f920fe00640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f921010bd70 0x7f92101a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:02.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.154+0000 7f920fe00640 1 -- 172.21.15.43:0/2361298864 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92101a3fe0 con 0x7f9210107070 2024-05-27T07:14:02.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.154+0000 7f920fe00640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9210107070 0x7f921019c9e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00ea60 tx=0x7f91fc00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:02.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.155+0000 7f920d600640 1 -- 172.21.15.43:0/2361298864 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc00cd20 con 0x7f9210107070 2024-05-27T07:14:02.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.155+0000 7f9217a00640 1 -- 172.21.15.43:0/2361298864 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92101a42d0 con 0x7f9210107070 2024-05-27T07:14:02.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.155+0000 7f920d600640 1 -- 172.21.15.43:0/2361298864 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91fc0035f0 con 0x7f9210107070 2024-05-27T07:14:02.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.155+0000 7f9217a00640 1 -- 172.21.15.43:0/2361298864 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92101a4870 con 0x7f9210107070 2024-05-27T07:14:02.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.155+0000 7f920d600640 1 -- 172.21.15.43:0/2361298864 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc005220 con 0x7f9210107070 2024-05-27T07:14:02.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.156+0000 7f9217a00640 1 -- 172.21.15.43:0/2361298864 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f921010ab80 con 0x7f9210107070 2024-05-27T07:14:02.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.159+0000 7f920d600640 1 -- 172.21.15.43:0/2361298864 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f91fc0053c0 con 0x7f9210107070 2024-05-27T07:14:02.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.162+0000 7f920d600640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91f80b29f0 0x7f91f80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:02.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.162+0000 7f920d600640 1 -- 172.21.15.43:0/2361298864 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f91fc0e31c0 con 0x7f9210107070 2024-05-27T07:14:02.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.162+0000 7f9215000640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91f80b29f0 0x7f91f80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:02.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.163+0000 7f9215000640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91f80b29f0 0x7f91f80b4f00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f920402a920 tx=0x7f9204031420 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:02.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.164+0000 7f920d600640 1 -- 172.21.15.43:0/2361298864 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f91fc0aa680 con 0x7f9210107070 2024-05-27T07:14:02.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.402+0000 7f9217a00640 1 -- 172.21.15.43:0/2361298864 --> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9210063100 con 0x7f91f80b29f0 2024-05-27T07:14:02.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.419+0000 7f920d600640 1 -- 172.21.15.43:0/2361298864 <== mgr.35033 v2:172.21.15.43:6800/2368318638 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339527 (secure 0 0 0) 0x7f9210063100 con 0x7f91f80b29f0 2024-05-27T07:14:02.413 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:14:02.415 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 -- 172.21.15.43:0/2361298864 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91f80b29f0 msgr2=0x7f91f80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91f80b29f0 0x7f91f80b4f00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f920402a920 tx=0x7f9204031420 comp rx=0 tx=0).stop 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 -- 172.21.15.43:0/2361298864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9210107070 msgr2=0x7f921019c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9210107070 0x7f921019c9e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f91fc00ea60 tx=0x7f91fc00ee60 comp rx=0 tx=0).stop 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 -- 172.21.15.43:0/2361298864 shutdown_connections 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91f80b29f0 0x7f91f80b4f00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f921010bd70 0x7f92101a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9210107070 0x7f921019c9e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 --2- 172.21.15.43:0/2361298864 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9210105e30 0x7f921019c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 -- 172.21.15.43:0/2361298864 >> 172.21.15.43:0/2361298864 conn(0x7f9210101040 msgr2=0x7f9210107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:02.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 -- 172.21.15.43:0/2361298864 shutdown_connections 2024-05-27T07:14:02.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:02.424+0000 7f9217a00640 1 -- 172.21.15.43:0/2361298864 wait complete. 2024-05-27T07:14:02.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:02 smithi043 ceph-mon[31531]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:14:02.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:02 smithi043 ceph-mon[31531]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:14:02.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:02 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:02.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:02 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:02.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:02 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:02 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:02 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:02 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:02 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:02.439 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:02 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:02.440 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-27T07:14:00.440276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1195241,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2642,"num_read_kb":1465,"num_write":518,"num_write_kb":3632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953452,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369236,"kb_used_data":15836,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124572588,"statfs":{"total":1151940427776,"available":1151562330112,"internally_reserved":0,"allocated":16216064,"data_stored":9659013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.977748"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551640+0000","last_change":"2024-05-27T07:08:22.053540+0000","last_active":"2024-05-27T07:13:54.551640+0000","last_peered":"2024-05-27T07:13:54.551640+0000","last_clean":"2024-05-27T07:13:54.551640+0000","last_became_active":"2024-05-27T07:08:22.051872+0000","last_became_peered":"2024-05-27T07:08:22.051872+0000","last_unstale":"2024-05-27T07:13:54.551640+0000","last_undegraded":"2024-05-27T07:13:54.551640+0000","last_fullsized":"2024-05-27T07:13:54.551640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:46:02.787957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000328+0000","last_change":"2024-05-27T07:08:22.049341+0000","last_active":"2024-05-27T07:13:55.000328+0000","last_peered":"2024-05-27T07:13:55.000328+0000","last_clean":"2024-05-27T07:13:55.000328+0000","last_became_active":"2024-05-27T07:08:22.048905+0000","last_became_peered":"2024-05-27T07:08:22.048905+0000","last_unstale":"2024-05-27T07:13:55.000328+0000","last_undegraded":"2024-05-27T07:13:55.000328+0000","last_fullsized":"2024-05-27T07:13:55.000328+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:10:27.271336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.999833+0000","last_change":"2024-05-27T07:08:22.053925+0000","last_active":"2024-05-27T07:13:54.999833+0000","last_peered":"2024-05-27T07:13:54.999833+0000","last_clean":"2024-05-27T07:13:54.999833+0000","last_became_active":"2024-05-27T07:08:22.053409+0000","last_became_peered":"2024-05-27T07:08:22.053409+0000","last_unstale":"2024-05-27T07:13:54.999833+0000","last_undegraded":"2024-05-27T07:13:54.999833+0000","last_fullsized":"2024-05-27T07:13:54.999833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:07:55.439966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835634+0000","last_change":"2024-05-27T07:08:22.048214+0000","last_active":"2024-05-27T07:13:54.835634+0000","last_peered":"2024-05-27T07:13:54.835634+0000","last_clean":"2024-05-27T07:13:54.835634+0000","last_became_active":"2024-05-27T07:08:22.047791+0000","last_became_peered":"2024-05-27T07:08:22.047791+0000","last_unstale":"2024-05-27T07:13:54.835634+0000","last_undegraded":"2024-05-27T07:13:54.835634+0000","last_fullsized":"2024-05-27T07:13:54.835634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:23:56.381008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572980+0000","last_change":"2024-05-27T07:08:22.037883+0000","last_active":"2024-05-27T07:13:54.572980+0000","last_peered":"2024-05-27T07:13:54.572980+0000","last_clean":"2024-05-27T07:13:54.572980+0000","last_became_active":"2024-05-27T07:08:22.037089+0000","last_became_peered":"2024-05-27T07:08:22.037089+0000","last_unstale":"2024-05-27T07:13:54.572980+0000","last_undegraded":"2024-05-27T07:13:54.572980+0000","last_fullsized":"2024-05-27T07:13:54.572980+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:40:21.929086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.478833+0000","last_change":"2024-05-27T07:08:22.061833+0000","last_active":"2024-05-27T07:13:54.478833+0000","last_peered":"2024-05-27T07:13:54.478833+0000","last_clean":"2024-05-27T07:13:54.478833+0000","last_became_active":"2024-05-27T07:08:22.057692+0000","last_became_peered":"2024-05-27T07:08:22.057692+0000","last_unstale":"2024-05-27T07:13:54.478833+0000","last_undegraded":"2024-05-27T07:13:54.478833+0000","last_fullsized":"2024-05-27T07:13:54.478833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:38:33.656872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779346+0000","last_change":"2024-05-27T07:08:22.047880+0000","last_active":"2024-05-27T07:13:54.779346+0000","last_peered":"2024-05-27T07:13:54.779346+0000","last_clean":"2024-05-27T07:13:54.779346+0000","last_became_active":"2024-05-27T07:08:22.047482+0000","last_became_peered":"2024-05-27T07:08:22.047482+0000","last_unstale":"2024-05-27T07:13:54.779346+0000","last_undegraded":"2024-05-27T07:13:54.779346+0000","last_fullsized":"2024-05-27T07:13:54.779346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:20:07.751588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.451911+0000","last_change":"2024-05-27T07:08:22.055970+0000","last_active":"2024-05-27T07:13:54.451911+0000","last_peered":"2024-05-27T07:13:54.451911+0000","last_clean":"2024-05-27T07:13:54.451911+0000","last_became_active":"2024-05-27T07:08:22.055591+0000","last_became_peered":"2024-05-27T07:08:22.055591+0000","last_unstale":"2024-05-27T07:13:54.451911+0000","last_undegraded":"2024-05-27T07:13:54.451911+0000","last_fullsized":"2024-05-27T07:13:54.451911+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:58:46.571473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412531+0000","last_change":"2024-05-27T07:08:22.059929+0000","last_active":"2024-05-27T07:13:54.412531+0000","last_peered":"2024-05-27T07:13:54.412531+0000","last_clean":"2024-05-27T07:13:54.412531+0000","last_became_active":"2024-05-27T07:08:22.059118+0000","last_became_peered":"2024-05-27T07:08:22.059118+0000","last_unstale":"2024-05-27T07:13:54.412531+0000","last_undegraded":"2024-05-27T07:13:54.412531+0000","last_fullsized":"2024-05-27T07:13:54.412531+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:29.231220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.479644+0000","last_change":"2024-05-27T07:08:53.979269+0000","last_active":"2024-05-27T07:13:54.479644+0000","last_peered":"2024-05-27T07:13:54.479644+0000","last_clean":"2024-05-27T07:13:54.479644+0000","last_became_active":"2024-05-27T07:08:53.978433+0000","last_became_peered":"2024-05-27T07:08:53.978433+0000","last_unstale":"2024-05-27T07:13:54.479644+0000","last_undegraded":"2024-05-27T07:13:54.479644+0000","last_fullsized":"2024-05-27T07:13:54.479644+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:57:37.600911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780920+0000","last_change":"2024-05-27T07:08:22.053499+0000","last_active":"2024-05-27T07:13:54.780920+0000","last_peered":"2024-05-27T07:13:54.780920+0000","last_clean":"2024-05-27T07:13:54.780920+0000","last_became_active":"2024-05-27T07:08:22.052428+0000","last_became_peered":"2024-05-27T07:08:22.052428+0000","last_unstale":"2024-05-27T07:13:54.780920+0000","last_undegraded":"2024-05-27T07:13:54.780920+0000","last_fullsized":"2024-05-27T07:13:54.780920+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:04.352262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835640+0000","last_change":"2024-05-27T07:08:22.058882+0000","last_active":"2024-05-27T07:13:54.835640+0000","last_peered":"2024-05-27T07:13:54.835640+0000","last_clean":"2024-05-27T07:13:54.835640+0000","last_became_active":"2024-05-27T07:08:22.057874+0000","last_became_peered":"2024-05-27T07:08:22.057874+0000","last_unstale":"2024-05-27T07:13:54.835640+0000","last_undegraded":"2024-05-27T07:13:54.835640+0000","last_fullsized":"2024-05-27T07:13:54.835640+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:52:11.289488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.828041+0000","last_change":"2024-05-27T07:08:22.054315+0000","last_active":"2024-05-27T07:13:54.828041+0000","last_peered":"2024-05-27T07:13:54.828041+0000","last_clean":"2024-05-27T07:13:54.828041+0000","last_became_active":"2024-05-27T07:08:22.053944+0000","last_became_peered":"2024-05-27T07:08:22.053944+0000","last_unstale":"2024-05-27T07:13:54.828041+0000","last_undegraded":"2024-05-27T07:13:54.828041+0000","last_fullsized":"2024-05-27T07:13:54.828041+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:43:06.720147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.998761+0000","last_change":"2024-05-27T07:08:22.054649+0000","last_active":"2024-05-27T07:13:54.998761+0000","last_peered":"2024-05-27T07:13:54.998761+0000","last_clean":"2024-05-27T07:13:54.998761+0000","last_became_active":"2024-05-27T07:08:22.051834+0000","last_became_peered":"2024-05-27T07:08:22.051834+0000","last_unstale":"2024-05-27T07:13:54.998761+0000","last_undegraded":"2024-05-27T07:13:54.998761+0000","last_fullsized":"2024-05-27T07:13:54.998761+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:16:56.893458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.998595+0000","last_change":"2024-05-27T07:08:22.052231+0000","last_active":"2024-05-27T07:13:54.998595+0000","last_peered":"2024-05-27T07:13:54.998595+0000","last_clean":"2024-05-27T07:13:54.998595+0000","last_became_active":"2024-05-27T07:08:22.051495+0000","last_became_peered":"2024-05-27T07:08:22.051495+0000","last_unstale":"2024-05-27T07:13:54.998595+0000","last_undegraded":"2024-05-27T07:13:54.998595+0000","last_fullsized":"2024-05-27T07:13:54.998595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:30:21.036874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410977+0000","last_change":"2024-05-27T07:08:22.056847+0000","last_active":"2024-05-27T07:13:54.410977+0000","last_peered":"2024-05-27T07:13:54.410977+0000","last_clean":"2024-05-27T07:13:54.410977+0000","last_became_active":"2024-05-27T07:08:22.056127+0000","last_became_peered":"2024-05-27T07:08:22.056127+0000","last_unstale":"2024-05-27T07:13:54.410977+0000","last_undegraded":"2024-05-27T07:13:54.410977+0000","last_fullsized":"2024-05-27T07:13:54.410977+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:45:04.757452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551330+0000","last_change":"2024-05-27T07:08:22.058078+0000","last_active":"2024-05-27T07:13:54.551330+0000","last_peered":"2024-05-27T07:13:54.551330+0000","last_clean":"2024-05-27T07:13:54.551330+0000","last_became_active":"2024-05-27T07:08:22.056886+0000","last_became_peered":"2024-05-27T07:08:22.056886+0000","last_unstale":"2024-05-27T07:13:54.551330+0000","last_undegraded":"2024-05-27T07:13:54.551330+0000","last_fullsized":"2024-05-27T07:13:54.551330+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.104572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.407614+0000","last_change":"2024-05-27T07:08:22.051492+0000","last_active":"2024-05-27T07:13:54.407614+0000","last_peered":"2024-05-27T07:13:54.407614+0000","last_clean":"2024-05-27T07:13:54.407614+0000","last_became_active":"2024-05-27T07:08:22.051091+0000","last_became_peered":"2024-05-27T07:08:22.051091+0000","last_unstale":"2024-05-27T07:13:54.407614+0000","last_undegraded":"2024-05-27T07:13:54.407614+0000","last_fullsized":"2024-05-27T07:13:54.407614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:16:47.816312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411384+0000","last_change":"2024-05-27T07:08:22.050714+0000","last_active":"2024-05-27T07:13:54.411384+0000","last_peered":"2024-05-27T07:13:54.411384+0000","last_clean":"2024-05-27T07:13:54.411384+0000","last_became_active":"2024-05-27T07:08:22.050322+0000","last_became_peered":"2024-05-27T07:08:22.050322+0000","last_unstale":"2024-05-27T07:13:54.411384+0000","last_undegraded":"2024-05-27T07:13:54.411384+0000","last_fullsized":"2024-05-27T07:13:54.411384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:01.867467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410108+0000","last_change":"2024-05-27T07:08:36.859916+0000","last_active":"2024-05-27T07:13:54.410108+0000","last_peered":"2024-05-27T07:13:54.410108+0000","last_clean":"2024-05-27T07:13:54.410108+0000","last_became_active":"2024-05-27T07:08:36.859429+0000","last_became_peered":"2024-05-27T07:08:36.859429+0000","last_unstale":"2024-05-27T07:13:54.410108+0000","last_undegraded":"2024-05-27T07:13:54.410108+0000","last_fullsized":"2024-05-27T07:13:54.410108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:14.026604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'14","reported_seq":305,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410345+0000","last_change":"2024-05-27T07:08:18.008912+0000","last_active":"2024-05-27T07:13:54.410345+0000","last_peered":"2024-05-27T07:13:54.410345+0000","last_clean":"2024-05-27T07:13:54.410345+0000","last_became_active":"2024-05-27T07:08:18.007603+0000","last_became_peered":"2024-05-27T07:08:18.007603+0000","last_unstale":"2024-05-27T07:13:54.410345+0000","last_undegraded":"2024-05-27T07:13:54.410345+0000","last_fullsized":"2024-05-27T07:13:54.410345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:17:40.681754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":243,"num_read_kb":243,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453059+0000","last_change":"2024-05-27T07:08:22.055426+0000","last_active":"2024-05-27T07:13:54.453059+0000","last_peered":"2024-05-27T07:13:54.453059+0000","last_clean":"2024-05-27T07:13:54.453059+0000","last_became_active":"2024-05-27T07:08:22.055063+0000","last_became_peered":"2024-05-27T07:08:22.055063+0000","last_unstale":"2024-05-27T07:13:54.453059+0000","last_undegraded":"2024-05-27T07:13:54.453059+0000","last_fullsized":"2024-05-27T07:13:54.453059+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:06:49.923192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834122+0000","last_change":"2024-05-27T07:08:53.982756+0000","last_active":"2024-05-27T07:13:54.834122+0000","last_peered":"2024-05-27T07:13:54.834122+0000","last_clean":"2024-05-27T07:13:54.834122+0000","last_became_active":"2024-05-27T07:08:53.981816+0000","last_became_peered":"2024-05-27T07:08:53.981816+0000","last_unstale":"2024-05-27T07:13:54.834122+0000","last_undegraded":"2024-05-27T07:13:54.834122+0000","last_fullsized":"2024-05-27T07:13:54.834122+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:09.984488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453284+0000","last_change":"2024-05-27T07:08:18.307659+0000","last_active":"2024-05-27T07:13:54.453284+0000","last_peered":"2024-05-27T07:13:54.453284+0000","last_clean":"2024-05-27T07:13:54.453284+0000","last_became_active":"2024-05-27T07:08:18.307239+0000","last_became_peered":"2024-05-27T07:08:18.307239+0000","last_unstale":"2024-05-27T07:13:54.453284+0000","last_undegraded":"2024-05-27T07:13:54.453284+0000","last_fullsized":"2024-05-27T07:13:54.453284+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:09:02.816813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000794+0000","last_change":"2024-05-27T07:08:22.050403+0000","last_active":"2024-05-27T07:13:55.000794+0000","last_peered":"2024-05-27T07:13:55.000794+0000","last_clean":"2024-05-27T07:13:55.000794+0000","last_became_active":"2024-05-27T07:08:22.049939+0000","last_became_peered":"2024-05-27T07:08:22.049939+0000","last_unstale":"2024-05-27T07:13:55.000794+0000","last_undegraded":"2024-05-27T07:13:55.000794+0000","last_fullsized":"2024-05-27T07:13:55.000794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:45.857508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.833966+0000","last_change":"2024-05-27T07:08:36.864458+0000","last_active":"2024-05-27T07:13:54.833966+0000","last_peered":"2024-05-27T07:13:54.833966+0000","last_clean":"2024-05-27T07:13:54.833966+0000","last_became_active":"2024-05-27T07:08:36.863894+0000","last_became_peered":"2024-05-27T07:08:36.863894+0000","last_unstale":"2024-05-27T07:13:54.833966+0000","last_undegraded":"2024-05-27T07:13:54.833966+0000","last_fullsized":"2024-05-27T07:13:54.833966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:49.448209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'13","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573904+0000","last_change":"2024-05-27T07:08:18.016051+0000","last_active":"2024-05-27T07:13:54.573904+0000","last_peered":"2024-05-27T07:13:54.573904+0000","last_clean":"2024-05-27T07:13:54.573904+0000","last_became_active":"2024-05-27T07:08:18.015590+0000","last_became_peered":"2024-05-27T07:08:18.015590+0000","last_unstale":"2024-05-27T07:13:54.573904+0000","last_undegraded":"2024-05-27T07:13:54.573904+0000","last_fullsized":"2024-05-27T07:13:54.573904+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:39:33.396027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411799+0000","last_change":"2024-05-27T07:08:22.055603+0000","last_active":"2024-05-27T07:13:54.411799+0000","last_peered":"2024-05-27T07:13:54.411799+0000","last_clean":"2024-05-27T07:13:54.411799+0000","last_became_active":"2024-05-27T07:08:22.054543+0000","last_became_peered":"2024-05-27T07:08:22.054543+0000","last_unstale":"2024-05-27T07:13:54.411799+0000","last_undegraded":"2024-05-27T07:13:54.411799+0000","last_fullsized":"2024-05-27T07:13:54.411799+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:29:01.738598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.833715+0000","last_change":"2024-05-27T07:08:36.870358+0000","last_active":"2024-05-27T07:13:54.833715+0000","last_peered":"2024-05-27T07:13:54.833715+0000","last_clean":"2024-05-27T07:13:54.833715+0000","last_became_active":"2024-05-27T07:08:36.869457+0000","last_became_peered":"2024-05-27T07:08:36.869457+0000","last_unstale":"2024-05-27T07:13:54.833715+0000","last_undegraded":"2024-05-27T07:13:54.833715+0000","last_fullsized":"2024-05-27T07:13:54.833715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:50.510098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550540+0000","last_change":"2024-05-27T07:08:53.985562+0000","last_active":"2024-05-27T07:13:54.550540+0000","last_peered":"2024-05-27T07:13:54.550540+0000","last_clean":"2024-05-27T07:13:54.550540+0000","last_became_active":"2024-05-27T07:08:53.977778+0000","last_became_peered":"2024-05-27T07:08:53.977778+0000","last_unstale":"2024-05-27T07:13:54.550540+0000","last_undegraded":"2024-05-27T07:13:54.550540+0000","last_fullsized":"2024-05-27T07:13:54.550540+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:20:44.455508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584633+0000","last_change":"2024-05-27T07:08:22.057622+0000","last_active":"2024-05-27T07:13:54.584633+0000","last_peered":"2024-05-27T07:13:54.584633+0000","last_clean":"2024-05-27T07:13:54.584633+0000","last_became_active":"2024-05-27T07:08:22.056544+0000","last_became_peered":"2024-05-27T07:08:22.056544+0000","last_unstale":"2024-05-27T07:13:54.584633+0000","last_undegraded":"2024-05-27T07:13:54.584633+0000","last_fullsized":"2024-05-27T07:13:54.584633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:38.484560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410554+0000","last_change":"2024-05-27T07:08:36.864984+0000","last_active":"2024-05-27T07:13:54.410554+0000","last_peered":"2024-05-27T07:13:54.410554+0000","last_clean":"2024-05-27T07:13:54.410554+0000","last_became_active":"2024-05-27T07:08:36.864302+0000","last_became_peered":"2024-05-27T07:08:36.864302+0000","last_unstale":"2024-05-27T07:13:54.410554+0000","last_undegraded":"2024-05-27T07:13:54.410554+0000","last_fullsized":"2024-05-27T07:13:54.410554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:44.183562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":299,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584240+0000","last_change":"2024-05-27T07:08:18.017918+0000","last_active":"2024-05-27T07:13:54.584240+0000","last_peered":"2024-05-27T07:13:54.584240+0000","last_clean":"2024-05-27T07:13:54.584240+0000","last_became_active":"2024-05-27T07:08:18.016174+0000","last_became_peered":"2024-05-27T07:08:18.016174+0000","last_unstale":"2024-05-27T07:13:54.584240+0000","last_undegraded":"2024-05-27T07:13:54.584240+0000","last_fullsized":"2024-05-27T07:13:54.584240+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:21.906187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573084+0000","last_change":"2024-05-27T07:08:22.043172+0000","last_active":"2024-05-27T07:13:54.573084+0000","last_peered":"2024-05-27T07:13:54.573084+0000","last_clean":"2024-05-27T07:13:54.573084+0000","last_became_active":"2024-05-27T07:08:22.041399+0000","last_became_peered":"2024-05-27T07:08:22.041399+0000","last_unstale":"2024-05-27T07:13:54.573084+0000","last_undegraded":"2024-05-27T07:13:54.573084+0000","last_fullsized":"2024-05-27T07:13:54.573084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:49:19.651330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410576+0000","last_change":"2024-05-27T07:08:36.866352+0000","last_active":"2024-05-27T07:13:54.410576+0000","last_peered":"2024-05-27T07:13:54.410576+0000","last_clean":"2024-05-27T07:13:54.410576+0000","last_became_active":"2024-05-27T07:08:36.865447+0000","last_became_peered":"2024-05-27T07:08:36.865447+0000","last_unstale":"2024-05-27T07:13:54.410576+0000","last_undegraded":"2024-05-27T07:13:54.410576+0000","last_fullsized":"2024-05-27T07:13:54.410576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:55:42.634374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.478763+0000","last_change":"2024-05-27T07:08:18.022142+0000","last_active":"2024-05-27T07:13:54.478763+0000","last_peered":"2024-05-27T07:13:54.478763+0000","last_clean":"2024-05-27T07:13:54.478763+0000","last_became_active":"2024-05-27T07:08:18.021453+0000","last_became_peered":"2024-05-27T07:08:18.021453+0000","last_unstale":"2024-05-27T07:13:54.478763+0000","last_undegraded":"2024-05-27T07:13:54.478763+0000","last_fullsized":"2024-05-27T07:13:54.478763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:01:09.049440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584299+0000","last_change":"2024-05-27T07:08:22.055742+0000","last_active":"2024-05-27T07:13:54.584299+0000","last_peered":"2024-05-27T07:13:54.584299+0000","last_clean":"2024-05-27T07:13:54.584299+0000","last_became_active":"2024-05-27T07:08:22.052077+0000","last_became_peered":"2024-05-27T07:08:22.052077+0000","last_unstale":"2024-05-27T07:13:54.584299+0000","last_undegraded":"2024-05-27T07:13:54.584299+0000","last_fullsized":"2024-05-27T07:13:54.584299+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:09:13.418027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551028+0000","last_change":"2024-05-27T07:08:36.874225+0000","last_active":"2024-05-27T07:13:54.551028+0000","last_peered":"2024-05-27T07:13:54.551028+0000","last_clean":"2024-05-27T07:13:54.551028+0000","last_became_active":"2024-05-27T07:08:36.873745+0000","last_became_peered":"2024-05-27T07:08:36.873745+0000","last_unstale":"2024-05-27T07:13:54.551028+0000","last_undegraded":"2024-05-27T07:13:54.551028+0000","last_fullsized":"2024-05-27T07:13:54.551028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:14:40.633667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827100+0000","last_change":"2024-05-27T07:08:18.009678+0000","last_active":"2024-05-27T07:13:54.827100+0000","last_peered":"2024-05-27T07:13:54.827100+0000","last_clean":"2024-05-27T07:13:54.827100+0000","last_became_active":"2024-05-27T07:08:18.006502+0000","last_became_peered":"2024-05-27T07:08:18.006502+0000","last_unstale":"2024-05-27T07:13:54.827100+0000","last_undegraded":"2024-05-27T07:13:54.827100+0000","last_fullsized":"2024-05-27T07:13:54.827100+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:13:37.457044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452037+0000","last_change":"2024-05-27T07:08:22.047302+0000","last_active":"2024-05-27T07:13:54.452037+0000","last_peered":"2024-05-27T07:13:54.452037+0000","last_clean":"2024-05-27T07:13:54.452037+0000","last_became_active":"2024-05-27T07:08:22.045932+0000","last_became_peered":"2024-05-27T07:08:22.045932+0000","last_unstale":"2024-05-27T07:13:54.452037+0000","last_undegraded":"2024-05-27T07:13:54.452037+0000","last_fullsized":"2024-05-27T07:13:54.452037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:39:27.129539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000221+0000","last_change":"2024-05-27T07:08:36.870512+0000","last_active":"2024-05-27T07:13:55.000221+0000","last_peered":"2024-05-27T07:13:55.000221+0000","last_clean":"2024-05-27T07:13:55.000221+0000","last_became_active":"2024-05-27T07:08:36.870061+0000","last_became_peered":"2024-05-27T07:08:36.870061+0000","last_unstale":"2024-05-27T07:13:55.000221+0000","last_undegraded":"2024-05-27T07:13:55.000221+0000","last_fullsized":"2024-05-27T07:13:55.000221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:31:03.608758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583370+0000","last_change":"2024-05-27T07:08:18.018420+0000","last_active":"2024-05-27T07:13:54.583370+0000","last_peered":"2024-05-27T07:13:54.583370+0000","last_clean":"2024-05-27T07:13:54.583370+0000","last_became_active":"2024-05-27T07:08:18.017567+0000","last_became_peered":"2024-05-27T07:08:18.017567+0000","last_unstale":"2024-05-27T07:13:54.583370+0000","last_undegraded":"2024-05-27T07:13:54.583370+0000","last_fullsized":"2024-05-27T07:13:54.583370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:37.214408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.404452+0000","last_change":"2024-05-27T07:08:22.050943+0000","last_active":"2024-05-27T07:13:54.404452+0000","last_peered":"2024-05-27T07:13:54.404452+0000","last_clean":"2024-05-27T07:13:54.404452+0000","last_became_active":"2024-05-27T07:08:22.047706+0000","last_became_peered":"2024-05-27T07:08:22.047706+0000","last_unstale":"2024-05-27T07:13:54.404452+0000","last_undegraded":"2024-05-27T07:13:54.404452+0000","last_fullsized":"2024-05-27T07:13:54.404452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:52:09.739528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452211+0000","last_change":"2024-05-27T07:08:36.867487+0000","last_active":"2024-05-27T07:13:54.452211+0000","last_peered":"2024-05-27T07:13:54.452211+0000","last_clean":"2024-05-27T07:13:54.452211+0000","last_became_active":"2024-05-27T07:08:36.867032+0000","last_became_peered":"2024-05-27T07:08:36.867032+0000","last_unstale":"2024-05-27T07:13:54.452211+0000","last_undegraded":"2024-05-27T07:13:54.452211+0000","last_fullsized":"2024-05-27T07:13:54.452211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:26:24.912382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'7","reported_seq":539,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.792547+0000","last_change":"2024-05-27T07:08:18.020168+0000","last_active":"2024-05-27T07:13:54.792547+0000","last_peered":"2024-05-27T07:13:54.792547+0000","last_clean":"2024-05-27T07:13:54.792547+0000","last_became_active":"2024-05-27T07:08:18.018309+0000","last_became_peered":"2024-05-27T07:08:18.018309+0000","last_unstale":"2024-05-27T07:13:54.792547+0000","last_undegraded":"2024-05-27T07:13:54.792547+0000","last_fullsized":"2024-05-27T07:13:54.792547+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:48:16.839510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":484,"num_read_kb":484,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412438+0000","last_change":"2024-05-27T07:08:22.054888+0000","last_active":"2024-05-27T07:13:54.412438+0000","last_peered":"2024-05-27T07:13:54.412438+0000","last_clean":"2024-05-27T07:13:54.412438+0000","last_became_active":"2024-05-27T07:08:22.054417+0000","last_became_peered":"2024-05-27T07:08:22.054417+0000","last_unstale":"2024-05-27T07:13:54.412438+0000","last_undegraded":"2024-05-27T07:13:54.412438+0000","last_fullsized":"2024-05-27T07:13:54.412438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:43:00.866755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.404284+0000","last_change":"2024-05-27T07:08:36.854516+0000","last_active":"2024-05-27T07:13:54.404284+0000","last_peered":"2024-05-27T07:13:54.404284+0000","last_clean":"2024-05-27T07:13:54.404284+0000","last_became_active":"2024-05-27T07:08:36.853978+0000","last_became_peered":"2024-05-27T07:08:36.853978+0000","last_unstale":"2024-05-27T07:13:54.404284+0000","last_undegraded":"2024-05-27T07:13:54.404284+0000","last_fullsized":"2024-05-27T07:13:54.404284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:35:45.299100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.781001+0000","last_change":"2024-05-27T07:08:18.010539+0000","last_active":"2024-05-27T07:13:54.781001+0000","last_peered":"2024-05-27T07:13:54.781001+0000","last_clean":"2024-05-27T07:13:54.781001+0000","last_became_active":"2024-05-27T07:08:18.009643+0000","last_became_peered":"2024-05-27T07:08:18.009643+0000","last_unstale":"2024-05-27T07:13:54.781001+0000","last_undegraded":"2024-05-27T07:13:54.781001+0000","last_fullsized":"2024-05-27T07:13:54.781001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:48:40.335403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000623+0000","last_change":"2024-05-27T07:08:53.980165+0000","last_active":"2024-05-27T07:13:55.000623+0000","last_peered":"2024-05-27T07:13:55.000623+0000","last_clean":"2024-05-27T07:13:55.000623+0000","last_became_active":"2024-05-27T07:08:53.978565+0000","last_became_peered":"2024-05-27T07:08:53.978565+0000","last_unstale":"2024-05-27T07:13:55.000623+0000","last_undegraded":"2024-05-27T07:13:55.000623+0000","last_fullsized":"2024-05-27T07:13:55.000623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:56:33.513961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834628+0000","last_change":"2024-05-27T07:08:36.857814+0000","last_active":"2024-05-27T07:13:54.834628+0000","last_peered":"2024-05-27T07:13:54.834628+0000","last_clean":"2024-05-27T07:13:54.834628+0000","last_became_active":"2024-05-27T07:08:36.856790+0000","last_became_peered":"2024-05-27T07:08:36.856790+0000","last_unstale":"2024-05-27T07:13:54.834628+0000","last_undegraded":"2024-05-27T07:13:54.834628+0000","last_fullsized":"2024-05-27T07:13:54.834628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:39:10.322261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584453+0000","last_change":"2024-05-27T07:08:18.308032+0000","last_active":"2024-05-27T07:13:54.584453+0000","last_peered":"2024-05-27T07:13:54.584453+0000","last_clean":"2024-05-27T07:13:54.584453+0000","last_became_active":"2024-05-27T07:08:18.306867+0000","last_became_peered":"2024-05-27T07:08:18.306867+0000","last_unstale":"2024-05-27T07:13:54.584453+0000","last_undegraded":"2024-05-27T07:13:54.584453+0000","last_fullsized":"2024-05-27T07:13:54.584453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:56:37.816237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412154+0000","last_change":"2024-05-27T07:08:22.057940+0000","last_active":"2024-05-27T07:13:54.412154+0000","last_peered":"2024-05-27T07:13:54.412154+0000","last_clean":"2024-05-27T07:13:54.412154+0000","last_became_active":"2024-05-27T07:08:22.055109+0000","last_became_peered":"2024-05-27T07:08:22.055109+0000","last_unstale":"2024-05-27T07:13:54.412154+0000","last_undegraded":"2024-05-27T07:13:54.412154+0000","last_fullsized":"2024-05-27T07:13:54.412154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:00:46.905064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.478388+0000","last_change":"2024-05-27T07:08:53.980209+0000","last_active":"2024-05-27T07:13:54.478388+0000","last_peered":"2024-05-27T07:13:54.478388+0000","last_clean":"2024-05-27T07:13:54.478388+0000","last_became_active":"2024-05-27T07:08:53.979353+0000","last_became_peered":"2024-05-27T07:08:53.979353+0000","last_unstale":"2024-05-27T07:13:54.478388+0000","last_undegraded":"2024-05-27T07:13:54.478388+0000","last_fullsized":"2024-05-27T07:13:54.478388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:19:06.163563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'41","reported_seq":600,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.802694+0000","last_change":"2024-05-27T07:08:53.996491+0000","last_active":"2024-05-27T07:13:54.802694+0000","last_peered":"2024-05-27T07:13:54.802694+0000","last_clean":"2024-05-27T07:13:54.802694+0000","last_became_active":"2024-05-27T07:08:53.976013+0000","last_became_peered":"2024-05-27T07:08:53.976013+0000","last_unstale":"2024-05-27T07:13:54.802694+0000","last_undegraded":"2024-05-27T07:13:54.802694+0000","last_fullsized":"2024-05-27T07:13:54.802694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:44:16.592270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69583,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":495,"num_read_kb":311,"num_write":43,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573510+0000","last_change":"2024-05-27T07:08:22.057231+0000","last_active":"2024-05-27T07:13:54.573510+0000","last_peered":"2024-05-27T07:13:54.573510+0000","last_clean":"2024-05-27T07:13:54.573510+0000","last_became_active":"2024-05-27T07:08:22.056837+0000","last_became_peered":"2024-05-27T07:08:22.056837+0000","last_unstale":"2024-05-27T07:13:54.573510+0000","last_undegraded":"2024-05-27T07:13:54.573510+0000","last_fullsized":"2024-05-27T07:13:54.573510+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:10:28.645103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.585422+0000","last_change":"2024-05-27T07:08:36.870307+0000","last_active":"2024-05-27T07:13:54.585422+0000","last_peered":"2024-05-27T07:13:54.585422+0000","last_clean":"2024-05-27T07:13:54.585422+0000","last_became_active":"2024-05-27T07:08:36.869390+0000","last_became_peered":"2024-05-27T07:08:36.869390+0000","last_unstale":"2024-05-27T07:13:54.585422+0000","last_undegraded":"2024-05-27T07:13:54.585422+0000","last_fullsized":"2024-05-27T07:13:54.585422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:43:45.228885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410841+0000","last_change":"2024-05-27T07:08:18.014458+0000","last_active":"2024-05-27T07:13:54.410841+0000","last_peered":"2024-05-27T07:13:54.410841+0000","last_clean":"2024-05-27T07:13:54.410841+0000","last_became_active":"2024-05-27T07:08:18.013365+0000","last_became_peered":"2024-05-27T07:08:18.013365+0000","last_unstale":"2024-05-27T07:13:54.410841+0000","last_undegraded":"2024-05-27T07:13:54.410841+0000","last_fullsized":"2024-05-27T07:13:54.410841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:16:41.352618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827630+0000","last_change":"2024-05-27T07:08:22.043354+0000","last_active":"2024-05-27T07:13:54.827630+0000","last_peered":"2024-05-27T07:13:54.827630+0000","last_clean":"2024-05-27T07:13:54.827630+0000","last_became_active":"2024-05-27T07:08:22.042876+0000","last_became_peered":"2024-05-27T07:08:22.042876+0000","last_unstale":"2024-05-27T07:13:54.827630+0000","last_undegraded":"2024-05-27T07:13:54.827630+0000","last_fullsized":"2024-05-27T07:13:54.827630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:33:52.575568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550548+0000","last_change":"2024-05-27T07:08:36.869579+0000","last_active":"2024-05-27T07:13:54.550548+0000","last_peered":"2024-05-27T07:13:54.550548+0000","last_clean":"2024-05-27T07:13:54.550548+0000","last_became_active":"2024-05-27T07:08:36.869055+0000","last_became_peered":"2024-05-27T07:08:36.869055+0000","last_unstale":"2024-05-27T07:13:54.550548+0000","last_undegraded":"2024-05-27T07:13:54.550548+0000","last_fullsized":"2024-05-27T07:13:54.550548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:02:10.870079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584312+0000","last_change":"2024-05-27T07:08:18.015322+0000","last_active":"2024-05-27T07:13:54.584312+0000","last_peered":"2024-05-27T07:13:54.584312+0000","last_clean":"2024-05-27T07:13:54.584312+0000","last_became_active":"2024-05-27T07:08:18.014535+0000","last_became_peered":"2024-05-27T07:08:18.014535+0000","last_unstale":"2024-05-27T07:13:54.584312+0000","last_undegraded":"2024-05-27T07:13:54.584312+0000","last_fullsized":"2024-05-27T07:13:54.584312+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:35.116140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779909+0000","last_change":"2024-05-27T07:08:22.056181+0000","last_active":"2024-05-27T07:13:54.779909+0000","last_peered":"2024-05-27T07:13:54.779909+0000","last_clean":"2024-05-27T07:13:54.779909+0000","last_became_active":"2024-05-27T07:08:22.055755+0000","last_became_peered":"2024-05-27T07:08:22.055755+0000","last_unstale":"2024-05-27T07:13:54.779909+0000","last_undegraded":"2024-05-27T07:13:54.779909+0000","last_fullsized":"2024-05-27T07:13:54.779909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:38:35.634116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.409346+0000","last_change":"2024-05-27T07:08:36.863184+0000","last_active":"2024-05-27T07:13:54.409346+0000","last_peered":"2024-05-27T07:13:54.409346+0000","last_clean":"2024-05-27T07:13:54.409346+0000","last_became_active":"2024-05-27T07:08:36.862760+0000","last_became_peered":"2024-05-27T07:08:36.862760+0000","last_unstale":"2024-05-27T07:13:54.409346+0000","last_undegraded":"2024-05-27T07:13:54.409346+0000","last_fullsized":"2024-05-27T07:13:54.409346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:55:48.379157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":302,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834697+0000","last_change":"2024-05-27T07:08:18.019119+0000","last_active":"2024-05-27T07:13:54.834697+0000","last_peered":"2024-05-27T07:13:54.834697+0000","last_clean":"2024-05-27T07:13:54.834697+0000","last_became_active":"2024-05-27T07:08:18.018408+0000","last_became_peered":"2024-05-27T07:08:18.018408+0000","last_unstale":"2024-05-27T07:13:54.834697+0000","last_undegraded":"2024-05-27T07:13:54.834697+0000","last_fullsized":"2024-05-27T07:13:54.834697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:47:19.978328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":248,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779408+0000","last_change":"2024-05-27T07:08:22.059464+0000","last_active":"2024-05-27T07:13:54.779408+0000","last_peered":"2024-05-27T07:13:54.779408+0000","last_clean":"2024-05-27T07:13:54.779408+0000","last_became_active":"2024-05-27T07:08:22.058890+0000","last_became_peered":"2024-05-27T07:08:22.058890+0000","last_unstale":"2024-05-27T07:13:54.779408+0000","last_undegraded":"2024-05-27T07:13:54.779408+0000","last_fullsized":"2024-05-27T07:13:54.779408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:49:30.122552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.828201+0000","last_change":"2024-05-27T07:08:36.865884+0000","last_active":"2024-05-27T07:13:54.828201+0000","last_peered":"2024-05-27T07:13:54.828201+0000","last_clean":"2024-05-27T07:13:54.828201+0000","last_became_active":"2024-05-27T07:08:36.865443+0000","last_became_peered":"2024-05-27T07:08:36.865443+0000","last_unstale":"2024-05-27T07:13:54.828201+0000","last_undegraded":"2024-05-27T07:13:54.828201+0000","last_fullsized":"2024-05-27T07:13:54.828201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:47:15.026632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550142+0000","last_change":"2024-05-27T07:08:53.994809+0000","last_active":"2024-05-27T07:13:54.550142+0000","last_peered":"2024-05-27T07:13:54.550142+0000","last_clean":"2024-05-27T07:13:54.550142+0000","last_became_active":"2024-05-27T07:08:53.977628+0000","last_became_peered":"2024-05-27T07:08:53.977628+0000","last_unstale":"2024-05-27T07:13:54.550142+0000","last_undegraded":"2024-05-27T07:13:54.550142+0000","last_fullsized":"2024-05-27T07:13:54.550142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:13:37.176319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583849+0000","last_change":"2024-05-27T07:08:22.050733+0000","last_active":"2024-05-27T07:13:54.583849+0000","last_peered":"2024-05-27T07:13:54.583849+0000","last_clean":"2024-05-27T07:13:54.583849+0000","last_became_active":"2024-05-27T07:08:22.050321+0000","last_became_peered":"2024-05-27T07:08:22.050321+0000","last_unstale":"2024-05-27T07:13:54.583849+0000","last_undegraded":"2024-05-27T07:13:54.583849+0000","last_fullsized":"2024-05-27T07:13:54.583849+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:41:44.590595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583996+0000","last_change":"2024-05-27T07:08:36.867784+0000","last_active":"2024-05-27T07:13:54.583996+0000","last_peered":"2024-05-27T07:13:54.583996+0000","last_clean":"2024-05-27T07:13:54.583996+0000","last_became_active":"2024-05-27T07:08:36.867290+0000","last_became_peered":"2024-05-27T07:08:36.867290+0000","last_unstale":"2024-05-27T07:13:54.583996+0000","last_undegraded":"2024-05-27T07:13:54.583996+0000","last_fullsized":"2024-05-27T07:13:54.583996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:44:07.649640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573792+0000","last_change":"2024-05-27T07:08:18.013193+0000","last_active":"2024-05-27T07:13:54.573792+0000","last_peered":"2024-05-27T07:13:54.573792+0000","last_clean":"2024-05-27T07:13:54.573792+0000","last_became_active":"2024-05-27T07:08:18.011767+0000","last_became_peered":"2024-05-27T07:08:18.011767+0000","last_unstale":"2024-05-27T07:13:54.573792+0000","last_undegraded":"2024-05-27T07:13:54.573792+0000","last_fullsized":"2024-05-27T07:13:54.573792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:15:03.679719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550880+0000","last_change":"2024-05-27T07:08:22.067145+0000","last_active":"2024-05-27T07:13:54.550880+0000","last_peered":"2024-05-27T07:13:54.550880+0000","last_clean":"2024-05-27T07:13:54.550880+0000","last_became_active":"2024-05-27T07:08:22.066750+0000","last_became_peered":"2024-05-27T07:08:22.066750+0000","last_unstale":"2024-05-27T07:13:54.550880+0000","last_undegraded":"2024-05-27T07:13:54.550880+0000","last_fullsized":"2024-05-27T07:13:54.550880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:41:35.333311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572917+0000","last_change":"2024-05-27T07:08:53.982497+0000","last_active":"2024-05-27T07:13:54.572917+0000","last_peered":"2024-05-27T07:13:54.572917+0000","last_clean":"2024-05-27T07:13:54.572917+0000","last_became_active":"2024-05-27T07:08:53.981920+0000","last_became_peered":"2024-05-27T07:08:53.981920+0000","last_unstale":"2024-05-27T07:13:54.572917+0000","last_undegraded":"2024-05-27T07:13:54.572917+0000","last_fullsized":"2024-05-27T07:13:54.572917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:43:40.095357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.406501+0000","last_change":"2024-05-27T07:08:18.308330+0000","last_active":"2024-05-27T07:13:54.406501+0000","last_peered":"2024-05-27T07:13:54.406501+0000","last_clean":"2024-05-27T07:13:54.406501+0000","last_became_active":"2024-05-27T07:08:18.306961+0000","last_became_peered":"2024-05-27T07:08:18.306961+0000","last_unstale":"2024-05-27T07:13:54.406501+0000","last_undegraded":"2024-05-27T07:13:54.406501+0000","last_fullsized":"2024-05-27T07:13:54.406501+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:44:27.875936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.479309+0000","last_change":"2024-05-27T07:08:22.047562+0000","last_active":"2024-05-27T07:13:54.479309+0000","last_peered":"2024-05-27T07:13:54.479309+0000","last_clean":"2024-05-27T07:13:54.479309+0000","last_became_active":"2024-05-27T07:08:22.046977+0000","last_became_peered":"2024-05-27T07:08:22.046977+0000","last_unstale":"2024-05-27T07:13:54.479309+0000","last_undegraded":"2024-05-27T07:13:54.479309+0000","last_fullsized":"2024-05-27T07:13:54.479309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:49:43.376617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827427+0000","last_change":"2024-05-27T07:08:36.867382+0000","last_active":"2024-05-27T07:13:54.827427+0000","last_peered":"2024-05-27T07:13:54.827427+0000","last_clean":"2024-05-27T07:13:54.827427+0000","last_became_active":"2024-05-27T07:08:36.866296+0000","last_became_peered":"2024-05-27T07:08:36.866296+0000","last_unstale":"2024-05-27T07:13:54.827427+0000","last_undegraded":"2024-05-27T07:13:54.827427+0000","last_fullsized":"2024-05-27T07:13:54.827427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:14.305169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'7","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780505+0000","last_change":"2024-05-27T07:08:18.012031+0000","last_active":"2024-05-27T07:13:54.780505+0000","last_peered":"2024-05-27T07:13:54.780505+0000","last_clean":"2024-05-27T07:13:54.780505+0000","last_became_active":"2024-05-27T07:08:18.009593+0000","last_became_peered":"2024-05-27T07:08:18.009593+0000","last_unstale":"2024-05-27T07:13:54.780505+0000","last_undegraded":"2024-05-27T07:13:54.780505+0000","last_fullsized":"2024-05-27T07:13:54.780505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:25:22.316575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1472,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.479213+0000","last_change":"2024-05-27T07:08:22.061136+0000","last_active":"2024-05-27T07:13:54.479213+0000","last_peered":"2024-05-27T07:13:54.479213+0000","last_clean":"2024-05-27T07:13:54.479213+0000","last_became_active":"2024-05-27T07:08:22.060735+0000","last_became_peered":"2024-05-27T07:08:22.060735+0000","last_unstale":"2024-05-27T07:13:54.479213+0000","last_undegraded":"2024-05-27T07:13:54.479213+0000","last_fullsized":"2024-05-27T07:13:54.479213+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:45:50.392130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827972+0000","last_change":"2024-05-27T07:08:36.932187+0000","last_active":"2024-05-27T07:13:54.827972+0000","last_peered":"2024-05-27T07:13:54.827972+0000","last_clean":"2024-05-27T07:13:54.827972+0000","last_became_active":"2024-05-27T07:08:36.931214+0000","last_became_peered":"2024-05-27T07:08:36.931214+0000","last_unstale":"2024-05-27T07:13:54.827972+0000","last_undegraded":"2024-05-27T07:13:54.827972+0000","last_fullsized":"2024-05-27T07:13:54.827972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:20:59.645352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.404178+0000","last_change":"2024-05-27T07:09:54.082674+0000","last_active":"2024-05-27T07:13:54.404178+0000","last_peered":"2024-05-27T07:13:54.404178+0000","last_clean":"2024-05-27T07:13:54.404178+0000","last_became_active":"2024-05-27T07:09:54.081616+0000","last_became_peered":"2024-05-27T07:09:54.081616+0000","last_unstale":"2024-05-27T07:13:54.404178+0000","last_undegraded":"2024-05-27T07:13:54.404178+0000","last_fullsized":"2024-05-27T07:13:54.404178+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:38:11.230026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":298,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827139+0000","last_change":"2024-05-27T07:08:18.306439+0000","last_active":"2024-05-27T07:13:54.827139+0000","last_peered":"2024-05-27T07:13:54.827139+0000","last_clean":"2024-05-27T07:13:54.827139+0000","last_became_active":"2024-05-27T07:08:18.305978+0000","last_became_peered":"2024-05-27T07:08:18.305978+0000","last_unstale":"2024-05-27T07:13:54.827139+0000","last_undegraded":"2024-05-27T07:13:54.827139+0000","last_fullsized":"2024-05-27T07:13:54.827139+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:20:22.648922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":243,"num_read_kb":243,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780681+0000","last_change":"2024-05-27T07:08:36.866247+0000","last_active":"2024-05-27T07:13:54.780681+0000","last_peered":"2024-05-27T07:13:54.780681+0000","last_clean":"2024-05-27T07:13:54.780681+0000","last_became_active":"2024-05-27T07:08:36.865516+0000","last_became_peered":"2024-05-27T07:08:36.865516+0000","last_unstale":"2024-05-27T07:13:54.780681+0000","last_undegraded":"2024-05-27T07:13:54.780681+0000","last_fullsized":"2024-05-27T07:13:54.780681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:39:41.235352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452547+0000","last_change":"2024-05-27T07:08:22.045312+0000","last_active":"2024-05-27T07:13:54.452547+0000","last_peered":"2024-05-27T07:13:54.452547+0000","last_clean":"2024-05-27T07:13:54.452547+0000","last_became_active":"2024-05-27T07:08:22.044724+0000","last_became_peered":"2024-05-27T07:08:22.044724+0000","last_unstale":"2024-05-27T07:13:54.452547+0000","last_undegraded":"2024-05-27T07:13:54.452547+0000","last_fullsized":"2024-05-27T07:13:54.452547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:53:05.487794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834127+0000","last_change":"2024-05-27T07:08:18.019333+0000","last_active":"2024-05-27T07:13:54.834127+0000","last_peered":"2024-05-27T07:13:54.834127+0000","last_clean":"2024-05-27T07:13:54.834127+0000","last_became_active":"2024-05-27T07:08:18.018816+0000","last_became_peered":"2024-05-27T07:08:18.018816+0000","last_unstale":"2024-05-27T07:13:54.834127+0000","last_undegraded":"2024-05-27T07:13:54.834127+0000","last_fullsized":"2024-05-27T07:13:54.834127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:17:09.421346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.778974+0000","last_change":"2024-05-27T07:06:16.865587+0000","last_active":"2024-05-27T07:13:54.778974+0000","last_peered":"2024-05-27T07:13:54.778974+0000","last_clean":"2024-05-27T07:13:54.778974+0000","last_became_active":"2024-05-27T07:06:16.850646+0000","last_became_peered":"2024-05-27T07:06:16.850646+0000","last_unstale":"2024-05-27T07:13:54.778974+0000","last_undegraded":"2024-05-27T07:13:54.778974+0000","last_fullsized":"2024-05-27T07:13:54.778974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:01:14.931400+0000","last_clean_scrub_stamp":"2024-05-27T07:01:14.931400+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:37.272619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.584459+0000","last_change":"2024-05-27T07:08:36.934696+0000","last_active":"2024-05-27T07:13:54.584459+0000","last_peered":"2024-05-27T07:13:54.584459+0000","last_clean":"2024-05-27T07:13:54.584459+0000","last_became_active":"2024-05-27T07:08:36.934008+0000","last_became_peered":"2024-05-27T07:08:36.934008+0000","last_unstale":"2024-05-27T07:13:54.584459+0000","last_undegraded":"2024-05-27T07:13:54.584459+0000","last_fullsized":"2024-05-27T07:13:54.584459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:39.099180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.826925+0000","last_change":"2024-05-27T07:08:22.054479+0000","last_active":"2024-05-27T07:13:54.826925+0000","last_peered":"2024-05-27T07:13:54.826925+0000","last_clean":"2024-05-27T07:13:54.826925+0000","last_became_active":"2024-05-27T07:08:22.054127+0000","last_became_peered":"2024-05-27T07:08:22.054127+0000","last_unstale":"2024-05-27T07:13:54.826925+0000","last_undegraded":"2024-05-27T07:13:54.826925+0000","last_fullsized":"2024-05-27T07:13:54.826925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:37:25.452176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779394+0000","last_change":"2024-05-27T07:08:18.010944+0000","last_active":"2024-05-27T07:13:54.779394+0000","last_peered":"2024-05-27T07:13:54.779394+0000","last_clean":"2024-05-27T07:13:54.779394+0000","last_became_active":"2024-05-27T07:08:18.009816+0000","last_became_peered":"2024-05-27T07:08:18.009816+0000","last_unstale":"2024-05-27T07:13:54.779394+0000","last_undegraded":"2024-05-27T07:13:54.779394+0000","last_fullsized":"2024-05-27T07:13:54.779394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:34:26.061787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550526+0000","last_change":"2024-05-27T07:08:36.870618+0000","last_active":"2024-05-27T07:13:54.550526+0000","last_peered":"2024-05-27T07:13:54.550526+0000","last_clean":"2024-05-27T07:13:54.550526+0000","last_became_active":"2024-05-27T07:08:36.869799+0000","last_became_peered":"2024-05-27T07:08:36.869799+0000","last_unstale":"2024-05-27T07:13:54.550526+0000","last_undegraded":"2024-05-27T07:13:54.550526+0000","last_fullsized":"2024-05-27T07:13:54.550526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:53.094337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573571+0000","last_change":"2024-05-27T07:08:18.308834+0000","last_active":"2024-05-27T07:13:54.573571+0000","last_peered":"2024-05-27T07:13:54.573571+0000","last_clean":"2024-05-27T07:13:54.573571+0000","last_became_active":"2024-05-27T07:08:18.306953+0000","last_became_peered":"2024-05-27T07:08:18.306953+0000","last_unstale":"2024-05-27T07:13:54.573571+0000","last_undegraded":"2024-05-27T07:13:54.573571+0000","last_fullsized":"2024-05-27T07:13:54.573571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:26:14.747631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827621+0000","last_change":"2024-05-27T07:08:22.048219+0000","last_active":"2024-05-27T07:13:54.827621+0000","last_peered":"2024-05-27T07:13:54.827621+0000","last_clean":"2024-05-27T07:13:54.827621+0000","last_became_active":"2024-05-27T07:08:22.047848+0000","last_became_peered":"2024-05-27T07:08:22.047848+0000","last_unstale":"2024-05-27T07:13:54.827621+0000","last_undegraded":"2024-05-27T07:13:54.827621+0000","last_fullsized":"2024-05-27T07:13:54.827621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:17:51.707603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.451911+0000","last_change":"2024-05-27T07:08:36.932747+0000","last_active":"2024-05-27T07:13:54.451911+0000","last_peered":"2024-05-27T07:13:54.451911+0000","last_clean":"2024-05-27T07:13:54.451911+0000","last_became_active":"2024-05-27T07:08:36.932050+0000","last_became_peered":"2024-05-27T07:08:36.932050+0000","last_unstale":"2024-05-27T07:13:54.451911+0000","last_undegraded":"2024-05-27T07:13:54.451911+0000","last_fullsized":"2024-05-27T07:13:54.451911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:26:07.015651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'52","reported_seq":123,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.842091+0000","last_change":"2024-05-27T07:08:18.008844+0000","last_active":"2024-05-27T07:13:54.842091+0000","last_peered":"2024-05-27T07:13:54.842091+0000","last_clean":"2024-05-27T07:13:54.842091+0000","last_became_active":"2024-05-27T07:08:18.008140+0000","last_became_peered":"2024-05-27T07:08:18.008140+0000","last_unstale":"2024-05-27T07:13:54.842091+0000","last_undegraded":"2024-05-27T07:13:54.842091+0000","last_fullsized":"2024-05-27T07:13:54.842091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:22:22.075341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":93,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453192+0000","last_change":"2024-05-27T07:08:22.050855+0000","last_active":"2024-05-27T07:13:54.453192+0000","last_peered":"2024-05-27T07:13:54.453192+0000","last_clean":"2024-05-27T07:13:54.453192+0000","last_became_active":"2024-05-27T07:08:22.048482+0000","last_became_peered":"2024-05-27T07:08:22.048482+0000","last_unstale":"2024-05-27T07:13:54.453192+0000","last_undegraded":"2024-05-27T07:13:54.453192+0000","last_fullsized":"2024-05-27T07:13:54.453192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:12:04.319884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.409858+0000","last_change":"2024-05-27T07:08:36.862470+0000","last_active":"2024-05-27T07:13:54.409858+0000","last_peered":"2024-05-27T07:13:54.409858+0000","last_clean":"2024-05-27T07:13:54.409858+0000","last_became_active":"2024-05-27T07:08:36.861951+0000","last_became_peered":"2024-05-27T07:08:36.861951+0000","last_unstale":"2024-05-27T07:13:54.409858+0000","last_undegraded":"2024-05-27T07:13:54.409858+0000","last_fullsized":"2024-05-27T07:13:54.409858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:56:31.659951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835285+0000","last_change":"2024-05-27T07:08:18.438518+0000","last_active":"2024-05-27T07:13:54.835285+0000","last_peered":"2024-05-27T07:13:54.835285+0000","last_clean":"2024-05-27T07:13:54.835285+0000","last_became_active":"2024-05-27T07:08:18.437588+0000","last_became_peered":"2024-05-27T07:08:18.437588+0000","last_unstale":"2024-05-27T07:13:54.835285+0000","last_undegraded":"2024-05-27T07:13:54.835285+0000","last_fullsized":"2024-05-27T07:13:54.835285+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:52:12.895009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:55.000455+0000","last_change":"2024-05-27T07:08:22.064188+0000","last_active":"2024-05-27T07:13:55.000455+0000","last_peered":"2024-05-27T07:13:55.000455+0000","last_clean":"2024-05-27T07:13:55.000455+0000","last_became_active":"2024-05-27T07:08:22.063868+0000","last_became_peered":"2024-05-27T07:08:22.063868+0000","last_unstale":"2024-05-27T07:13:55.000455+0000","last_undegraded":"2024-05-27T07:13:55.000455+0000","last_fullsized":"2024-05-27T07:13:55.000455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:43:17.227079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.834073+0000","last_change":"2024-05-27T07:08:36.934922+0000","last_active":"2024-05-27T07:13:54.834073+0000","last_peered":"2024-05-27T07:13:54.834073+0000","last_clean":"2024-05-27T07:13:54.834073+0000","last_became_active":"2024-05-27T07:08:36.934000+0000","last_became_peered":"2024-05-27T07:08:36.934000+0000","last_unstale":"2024-05-27T07:13:54.834073+0000","last_undegraded":"2024-05-27T07:13:54.834073+0000","last_fullsized":"2024-05-27T07:13:54.834073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:19:00.570557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.412244+0000","last_change":"2024-05-27T07:08:18.308743+0000","last_active":"2024-05-27T07:13:54.412244+0000","last_peered":"2024-05-27T07:13:54.412244+0000","last_clean":"2024-05-27T07:13:54.412244+0000","last_became_active":"2024-05-27T07:08:18.307597+0000","last_became_peered":"2024-05-27T07:08:18.307597+0000","last_unstale":"2024-05-27T07:13:54.412244+0000","last_undegraded":"2024-05-27T07:13:54.412244+0000","last_fullsized":"2024-05-27T07:13:54.412244+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:39.914998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572836+0000","last_change":"2024-05-27T07:08:22.049132+0000","last_active":"2024-05-27T07:13:54.572836+0000","last_peered":"2024-05-27T07:13:54.572836+0000","last_clean":"2024-05-27T07:13:54.572836+0000","last_became_active":"2024-05-27T07:08:22.048606+0000","last_became_peered":"2024-05-27T07:08:22.048606+0000","last_unstale":"2024-05-27T07:13:54.572836+0000","last_undegraded":"2024-05-27T07:13:54.572836+0000","last_fullsized":"2024-05-27T07:13:54.572836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:58.253461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.409892+0000","last_change":"2024-05-27T07:08:36.867286+0000","last_active":"2024-05-27T07:13:54.409892+0000","last_peered":"2024-05-27T07:13:54.409892+0000","last_clean":"2024-05-27T07:13:54.409892+0000","last_became_active":"2024-05-27T07:08:36.866732+0000","last_became_peered":"2024-05-27T07:08:36.866732+0000","last_unstale":"2024-05-27T07:13:54.409892+0000","last_undegraded":"2024-05-27T07:13:54.409892+0000","last_fullsized":"2024-05-27T07:13:54.409892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:14:18.386441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452354+0000","last_change":"2024-05-27T07:08:18.010026+0000","last_active":"2024-05-27T07:13:54.452354+0000","last_peered":"2024-05-27T07:13:54.452354+0000","last_clean":"2024-05-27T07:13:54.452354+0000","last_became_active":"2024-05-27T07:08:18.009502+0000","last_became_peered":"2024-05-27T07:08:18.009502+0000","last_unstale":"2024-05-27T07:13:54.452354+0000","last_undegraded":"2024-05-27T07:13:54.452354+0000","last_fullsized":"2024-05-27T07:13:54.452354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:28:27.500933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550700+0000","last_change":"2024-05-27T07:08:22.054536+0000","last_active":"2024-05-27T07:13:54.550700+0000","last_peered":"2024-05-27T07:13:54.550700+0000","last_clean":"2024-05-27T07:13:54.550700+0000","last_became_active":"2024-05-27T07:08:22.054141+0000","last_became_peered":"2024-05-27T07:08:22.054141+0000","last_unstale":"2024-05-27T07:13:54.550700+0000","last_undegraded":"2024-05-27T07:13:54.550700+0000","last_fullsized":"2024-05-27T07:13:54.550700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:06:53.413683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.836642+0000","last_change":"2024-05-27T07:08:36.859721+0000","last_active":"2024-05-27T07:13:54.836642+0000","last_peered":"2024-05-27T07:13:54.836642+0000","last_clean":"2024-05-27T07:13:54.836642+0000","last_became_active":"2024-05-27T07:08:36.858770+0000","last_became_peered":"2024-05-27T07:08:36.858770+0000","last_unstale":"2024-05-27T07:13:54.836642+0000","last_undegraded":"2024-05-27T07:13:54.836642+0000","last_fullsized":"2024-05-27T07:13:54.836642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:27:32.583869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411052+0000","last_change":"2024-05-27T07:08:18.008762+0000","last_active":"2024-05-27T07:13:54.411052+0000","last_peered":"2024-05-27T07:13:54.411052+0000","last_clean":"2024-05-27T07:13:54.411052+0000","last_became_active":"2024-05-27T07:08:18.007680+0000","last_became_peered":"2024-05-27T07:08:18.007680+0000","last_unstale":"2024-05-27T07:13:54.411052+0000","last_undegraded":"2024-05-27T07:13:54.411052+0000","last_fullsized":"2024-05-27T07:13:54.411052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:47:22.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452861+0000","last_change":"2024-05-27T07:08:22.051955+0000","last_active":"2024-05-27T07:13:54.452861+0000","last_peered":"2024-05-27T07:13:54.452861+0000","last_clean":"2024-05-27T07:13:54.452861+0000","last_became_active":"2024-05-27T07:08:22.051618+0000","last_became_peered":"2024-05-27T07:08:22.051618+0000","last_unstale":"2024-05-27T07:13:54.452861+0000","last_undegraded":"2024-05-27T07:13:54.452861+0000","last_fullsized":"2024-05-27T07:13:54.452861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:13.219186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.835409+0000","last_change":"2024-05-27T07:08:36.932048+0000","last_active":"2024-05-27T07:13:54.835409+0000","last_peered":"2024-05-27T07:13:54.835409+0000","last_clean":"2024-05-27T07:13:54.835409+0000","last_became_active":"2024-05-27T07:08:36.930777+0000","last_became_peered":"2024-05-27T07:08:36.930777+0000","last_unstale":"2024-05-27T07:13:54.835409+0000","last_undegraded":"2024-05-27T07:13:54.835409+0000","last_fullsized":"2024-05-27T07:13:54.835409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:32.284487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.573040+0000","last_change":"2024-05-27T07:08:18.011343+0000","last_active":"2024-05-27T07:13:54.573040+0000","last_peered":"2024-05-27T07:13:54.573040+0000","last_clean":"2024-05-27T07:13:54.573040+0000","last_became_active":"2024-05-27T07:08:18.009988+0000","last_became_peered":"2024-05-27T07:08:18.009988+0000","last_unstale":"2024-05-27T07:13:54.573040+0000","last_undegraded":"2024-05-27T07:13:54.573040+0000","last_fullsized":"2024-05-27T07:13:54.573040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:28:39.620568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.779670+0000","last_change":"2024-05-27T07:08:22.051630+0000","last_active":"2024-05-27T07:13:54.779670+0000","last_peered":"2024-05-27T07:13:54.779670+0000","last_clean":"2024-05-27T07:13:54.779670+0000","last_became_active":"2024-05-27T07:08:22.049140+0000","last_became_peered":"2024-05-27T07:08:22.049140+0000","last_unstale":"2024-05-27T07:13:54.779670+0000","last_undegraded":"2024-05-27T07:13:54.779670+0000","last_fullsized":"2024-05-27T07:13:54.779670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:30:51.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411238+0000","last_change":"2024-05-27T07:08:36.870668+0000","last_active":"2024-05-27T07:13:54.411238+0000","last_peered":"2024-05-27T07:13:54.411238+0000","last_clean":"2024-05-27T07:13:54.411238+0000","last_became_active":"2024-05-27T07:08:36.869986+0000","last_became_peered":"2024-05-27T07:08:36.869986+0000","last_unstale":"2024-05-27T07:13:54.411238+0000","last_undegraded":"2024-05-27T07:13:54.411238+0000","last_fullsized":"2024-05-27T07:13:54.411238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:19:59.440993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'9","reported_seq":304,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.831814+0000","last_change":"2024-05-27T07:08:18.439082+0000","last_active":"2024-05-27T07:13:54.831814+0000","last_peered":"2024-05-27T07:13:54.831814+0000","last_clean":"2024-05-27T07:13:54.831814+0000","last_became_active":"2024-05-27T07:08:18.437844+0000","last_became_peered":"2024-05-27T07:08:18.437844+0000","last_unstale":"2024-05-27T07:13:54.831814+0000","last_undegraded":"2024-05-27T07:13:54.831814+0000","last_fullsized":"2024-05-27T07:13:54.831814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:38:30.276747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2586,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":247,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.781353+0000","last_change":"2024-05-27T07:08:22.052308+0000","last_active":"2024-05-27T07:13:54.781353+0000","last_peered":"2024-05-27T07:13:54.781353+0000","last_clean":"2024-05-27T07:13:54.781353+0000","last_became_active":"2024-05-27T07:08:22.051615+0000","last_became_peered":"2024-05-27T07:08:22.051615+0000","last_unstale":"2024-05-27T07:13:54.781353+0000","last_undegraded":"2024-05-27T07:13:54.781353+0000","last_fullsized":"2024-05-27T07:13:54.781353+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:21:14.241454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.998936+0000","last_change":"2024-05-27T07:08:36.861082+0000","last_active":"2024-05-27T07:13:54.998936+0000","last_peered":"2024-05-27T07:13:54.998936+0000","last_clean":"2024-05-27T07:13:54.998936+0000","last_became_active":"2024-05-27T07:08:36.860548+0000","last_became_peered":"2024-05-27T07:08:36.860548+0000","last_unstale":"2024-05-27T07:13:54.998936+0000","last_undegraded":"2024-05-27T07:13:54.998936+0000","last_fullsized":"2024-05-27T07:13:54.998936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:46:32.446464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.838063+0000","last_change":"2024-05-27T07:08:18.018218+0000","last_active":"2024-05-27T07:13:54.838063+0000","last_peered":"2024-05-27T07:13:54.838063+0000","last_clean":"2024-05-27T07:13:54.838063+0000","last_became_active":"2024-05-27T07:08:18.017200+0000","last_became_peered":"2024-05-27T07:08:18.017200+0000","last_unstale":"2024-05-27T07:13:54.838063+0000","last_undegraded":"2024-05-27T07:13:54.838063+0000","last_fullsized":"2024-05-27T07:13:54.838063+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:14:22.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.780497+0000","last_change":"2024-05-27T07:09:54.052709+0000","last_active":"2024-05-27T07:13:54.780497+0000","last_peered":"2024-05-27T07:13:54.780497+0000","last_clean":"2024-05-27T07:13:54.780497+0000","last_became_active":"2024-05-27T07:09:54.051425+0000","last_became_peered":"2024-05-27T07:09:54.051425+0000","last_unstale":"2024-05-27T07:13:54.780497+0000","last_undegraded":"2024-05-27T07:13:54.780497+0000","last_fullsized":"2024-05-27T07:13:54.780497+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:51:47.166228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.572880+0000","last_change":"2024-05-27T07:08:53.979067+0000","last_active":"2024-05-27T07:13:54.572880+0000","last_peered":"2024-05-27T07:13:54.572880+0000","last_clean":"2024-05-27T07:13:54.572880+0000","last_became_active":"2024-05-27T07:08:53.977620+0000","last_became_peered":"2024-05-27T07:08:53.977620+0000","last_unstale":"2024-05-27T07:13:54.572880+0000","last_undegraded":"2024-05-27T07:13:54.572880+0000","last_fullsized":"2024-05-27T07:13:54.572880+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:35.793129+0000","last_clean_scrub_stamp":"2024-05-27T07:08:35.793129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:02:31.619861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.828067+0000","last_change":"2024-05-27T07:08:54.001216+0000","last_active":"2024-05-27T07:13:54.828067+0000","last_peered":"2024-05-27T07:13:54.828067+0000","last_clean":"2024-05-27T07:13:54.828067+0000","last_became_active":"2024-05-27T07:08:53.982121+0000","last_became_peered":"2024-05-27T07:08:53.982121+0000","last_unstale":"2024-05-27T07:13:54.828067+0000","last_undegraded":"2024-05-27T07:13:54.828067+0000","last_fullsized":"2024-05-27T07:13:54.828067+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:16.952575+0000","last_clean_scrub_stamp":"2024-05-27T07:08:16.952575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:50:01.474383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.781463+0000","last_change":"2024-05-27T07:08:22.053334+0000","last_active":"2024-05-27T07:13:54.781463+0000","last_peered":"2024-05-27T07:13:54.781463+0000","last_clean":"2024-05-27T07:13:54.781463+0000","last_became_active":"2024-05-27T07:08:22.051660+0000","last_became_peered":"2024-05-27T07:08:22.051660+0000","last_unstale":"2024-05-27T07:13:54.781463+0000","last_undegraded":"2024-05-27T07:13:54.781463+0000","last_fullsized":"2024-05-27T07:13:54.781463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:07:19.554620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550386+0000","last_change":"2024-05-27T07:08:22.064044+0000","last_active":"2024-05-27T07:13:54.550386+0000","last_peered":"2024-05-27T07:13:54.550386+0000","last_clean":"2024-05-27T07:13:54.550386+0000","last_became_active":"2024-05-27T07:08:22.063539+0000","last_became_peered":"2024-05-27T07:08:22.063539+0000","last_unstale":"2024-05-27T07:13:54.550386+0000","last_undegraded":"2024-05-27T07:13:54.550386+0000","last_fullsized":"2024-05-27T07:13:54.550386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:23:07.291250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410502+0000","last_change":"2024-05-27T07:08:22.059727+0000","last_active":"2024-05-27T07:13:54.410502+0000","last_peered":"2024-05-27T07:13:54.410502+0000","last_clean":"2024-05-27T07:13:54.410502+0000","last_became_active":"2024-05-27T07:08:22.058933+0000","last_became_peered":"2024-05-27T07:08:22.058933+0000","last_unstale":"2024-05-27T07:13:54.410502+0000","last_undegraded":"2024-05-27T07:13:54.410502+0000","last_fullsized":"2024-05-27T07:13:54.410502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:58:14.828031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.410019+0000","last_change":"2024-05-27T07:08:22.044871+0000","last_active":"2024-05-27T07:13:54.410019+0000","last_peered":"2024-05-27T07:13:54.410019+0000","last_clean":"2024-05-27T07:13:54.410019+0000","last_became_active":"2024-05-27T07:08:22.044394+0000","last_became_peered":"2024-05-27T07:08:22.044394+0000","last_unstale":"2024-05-27T07:13:54.410019+0000","last_undegraded":"2024-05-27T07:13:54.410019+0000","last_fullsized":"2024-05-27T07:13:54.410019+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:26:11.151064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.999243+0000","last_change":"2024-05-27T07:08:22.048318+0000","last_active":"2024-05-27T07:13:54.999243+0000","last_peered":"2024-05-27T07:13:54.999243+0000","last_clean":"2024-05-27T07:13:54.999243+0000","last_became_active":"2024-05-27T07:08:22.047983+0000","last_became_peered":"2024-05-27T07:08:22.047983+0000","last_unstale":"2024-05-27T07:13:54.999243+0000","last_undegraded":"2024-05-27T07:13:54.999243+0000","last_fullsized":"2024-05-27T07:13:54.999243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:17:11.479943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827805+0000","last_change":"2024-05-27T07:08:22.056951+0000","last_active":"2024-05-27T07:13:54.827805+0000","last_peered":"2024-05-27T07:13:54.827805+0000","last_clean":"2024-05-27T07:13:54.827805+0000","last_became_active":"2024-05-27T07:08:22.053701+0000","last_became_peered":"2024-05-27T07:08:22.053701+0000","last_unstale":"2024-05-27T07:13:54.827805+0000","last_undegraded":"2024-05-27T07:13:54.827805+0000","last_fullsized":"2024-05-27T07:13:54.827805+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:22:31.041730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.551197+0000","last_change":"2024-05-27T07:08:22.062745+0000","last_active":"2024-05-27T07:13:54.551197+0000","last_peered":"2024-05-27T07:13:54.551197+0000","last_clean":"2024-05-27T07:13:54.551197+0000","last_became_active":"2024-05-27T07:08:22.062339+0000","last_became_peered":"2024-05-27T07:08:22.062339+0000","last_unstale":"2024-05-27T07:13:54.551197+0000","last_undegraded":"2024-05-27T07:13:54.551197+0000","last_fullsized":"2024-05-27T07:13:54.551197+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:58:02.934021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411753+0000","last_change":"2024-05-27T07:08:22.052621+0000","last_active":"2024-05-27T07:13:54.411753+0000","last_peered":"2024-05-27T07:13:54.411753+0000","last_clean":"2024-05-27T07:13:54.411753+0000","last_became_active":"2024-05-27T07:08:22.052070+0000","last_became_peered":"2024-05-27T07:08:22.052070+0000","last_unstale":"2024-05-27T07:13:54.411753+0000","last_undegraded":"2024-05-27T07:13:54.411753+0000","last_fullsized":"2024-05-27T07:13:54.411753+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:19:34.491068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.453362+0000","last_change":"2024-05-27T07:08:22.055704+0000","last_active":"2024-05-27T07:13:54.453362+0000","last_peered":"2024-05-27T07:13:54.453362+0000","last_clean":"2024-05-27T07:13:54.453362+0000","last_became_active":"2024-05-27T07:08:22.055251+0000","last_became_peered":"2024-05-27T07:08:22.055251+0000","last_unstale":"2024-05-27T07:13:54.453362+0000","last_undegraded":"2024-05-27T07:13:54.453362+0000","last_fullsized":"2024-05-27T07:13:54.453362+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:27:28.074825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.583483+0000","last_change":"2024-05-27T07:08:22.052160+0000","last_active":"2024-05-27T07:13:54.583483+0000","last_peered":"2024-05-27T07:13:54.583483+0000","last_clean":"2024-05-27T07:13:54.583483+0000","last_became_active":"2024-05-27T07:08:22.049495+0000","last_became_peered":"2024-05-27T07:08:22.049495+0000","last_unstale":"2024-05-27T07:13:54.583483+0000","last_undegraded":"2024-05-27T07:13:54.583483+0000","last_fullsized":"2024-05-27T07:13:54.583483+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:54:59.595031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.452440+0000","last_change":"2024-05-27T07:08:22.053828+0000","last_active":"2024-05-27T07:13:54.452440+0000","last_peered":"2024-05-27T07:13:54.452440+0000","last_clean":"2024-05-27T07:13:54.452440+0000","last_became_active":"2024-05-27T07:08:22.053491+0000","last_became_peered":"2024-05-27T07:08:22.053491+0000","last_unstale":"2024-05-27T07:13:54.452440+0000","last_undegraded":"2024-05-27T07:13:54.452440+0000","last_fullsized":"2024-05-27T07:13:54.452440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:20.568772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.411658+0000","last_change":"2024-05-27T07:08:22.050791+0000","last_active":"2024-05-27T07:13:54.411658+0000","last_peered":"2024-05-27T07:13:54.411658+0000","last_clean":"2024-05-27T07:13:54.411658+0000","last_became_active":"2024-05-27T07:08:22.050323+0000","last_became_peered":"2024-05-27T07:08:22.050323+0000","last_unstale":"2024-05-27T07:13:54.411658+0000","last_undegraded":"2024-05-27T07:13:54.411658+0000","last_fullsized":"2024-05-27T07:13:54.411658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:32:32.400758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.827770+0000","last_change":"2024-05-27T07:08:22.041286+0000","last_active":"2024-05-27T07:13:54.827770+0000","last_peered":"2024-05-27T07:13:54.827770+0000","last_clean":"2024-05-27T07:13:54.827770+0000","last_became_active":"2024-05-27T07:08:22.040811+0000","last_became_peered":"2024-05-27T07:08:22.040811+0000","last_unstale":"2024-05-27T07:13:54.827770+0000","last_undegraded":"2024-05-27T07:13:54.827770+0000","last_fullsized":"2024-05-27T07:13:54.827770+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:40:09.872734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-27T07:13:54.550805+0000","last_change":"2024-05-27T07:08:22.066490+0000","last_active":"2024-05-27T07:13:54.550805+0000","last_peered":"2024-05-27T07:13:54.550805+0000","last_clean":"2024-05-27T07:13:54.550805+0000","last_became_active":"2024-05-27T07:08:22.064779+0000","last_became_peered":"2024-05-27T07:08:22.064779+0000","last_unstale":"2024-05-27T07:13:54.550805+0000","last_undegraded":"2024-05-27T07:13:54.550805+0000","last_fullsized":"2024-05-27T07:13:54.550805+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T07:08:20.989947+0000","last_clean_scrub_stamp":"2024-05-27T07:08:20.989947+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:34:23.967856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604639,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2561,"num_read_kb":1396,"num_write":393,"num_write_kb":1480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1813917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":314,"ondisk_log_size":314,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678103,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":520386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874333,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28860,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716292,"statfs":{"total":95995035648,"available":95965483008,"internally_reserved":0,"allocated":1822720,"data_stored":1247308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36784,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708368,"statfs":{"total":95995035648,"available":95957368832,"internally_reserved":0,"allocated":1478656,"data_stored":925775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234093,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1458176,"data_stored":921363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430324,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32816,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712336,"statfs":{"total":95995035648,"available":95961432064,"internally_reserved":0,"allocated":1675264,"data_stored":1126958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593853,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":514594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1126400,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":526839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411482,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1073152,"data_stored":527126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28576,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716576,"statfs":{"total":95995035648,"available":95965773824,"internally_reserved":0,"allocated":1597440,"data_stored":1048491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953610,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":598649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790083,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1111156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":72219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":203944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T07:14:02.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:14:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:02 smithi179 ceph-mon[35888]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:14:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:02 smithi179 ceph-mon[35888]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:14:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:02 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:02 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:02 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:02 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:02 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:02 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:02 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:02 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:02.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:02 smithi132 ceph-mon[35283]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:14:02.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:02 smithi132 ceph-mon[35283]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:14:02.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:02 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:02.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:02 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:02.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:02 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:02 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:02 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:02 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:14:02.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:02 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:02.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:02 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:02.910 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:03.330 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:03 smithi043 ceph-mon[31531]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:14:03.330 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:03 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi043 to 177.5M 2024-05-27T07:14:03.330 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:03 smithi043 ceph-mon[31531]: Unable to set osd_memory_target on smithi043 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:14:03.330 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:03 smithi043 ceph-mon[31531]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-27T07:14:03.331 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:03 smithi043 ceph-mon[31531]: Updating smithi132:/etc/ceph/ceph.conf 2024-05-27T07:14:03.331 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:03 smithi043 ceph-mon[31531]: Updating smithi179:/etc/ceph/ceph.conf 2024-05-27T07:14:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:03 smithi179 ceph-mon[35888]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:14:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:03 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi043 to 177.5M 2024-05-27T07:14:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:03 smithi179 ceph-mon[35888]: Unable to set osd_memory_target on smithi043 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:14:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:03 smithi179 ceph-mon[35888]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-27T07:14:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:03 smithi179 ceph-mon[35888]: Updating smithi132:/etc/ceph/ceph.conf 2024-05-27T07:14:03.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:03 smithi179 ceph-mon[35888]: Updating smithi179:/etc/ceph/ceph.conf 2024-05-27T07:14:03.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:03 smithi132 ceph-mon[35283]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:14:03.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:03 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi043 to 177.5M 2024-05-27T07:14:03.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:03 smithi132 ceph-mon[35283]: Unable to set osd_memory_target on smithi043 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:14:03.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:03 smithi132 ceph-mon[35283]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-27T07:14:03.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:03 smithi132 ceph-mon[35283]: Updating smithi132:/etc/ceph/ceph.conf 2024-05-27T07:14:03.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:03 smithi132 ceph-mon[35283]: Updating smithi179:/etc/ceph/ceph.conf 2024-05-27T07:14:04.441 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:04 smithi043 ceph-mon[31531]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:14:04.441 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:04 smithi043 ceph-mon[31531]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-05-27T07:14:04.442 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:04 smithi043 ceph-mon[31531]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:14:04.442 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:04 smithi043 ceph-mon[31531]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:14:04.442 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:04 smithi043 ceph-mon[31531]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:14:04.442 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:04 smithi043 ceph-mon[31531]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:04.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:04 smithi179 ceph-mon[35888]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:14:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:04 smithi179 ceph-mon[35888]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-05-27T07:14:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:04 smithi179 ceph-mon[35888]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:14:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:04 smithi179 ceph-mon[35888]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:14:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:04 smithi179 ceph-mon[35888]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:14:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:04 smithi179 ceph-mon[35888]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:04.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:04 smithi132 ceph-mon[35283]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-27T07:14:04.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:04 smithi132 ceph-mon[35283]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-05-27T07:14:04.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:04 smithi132 ceph-mon[35283]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:14:04.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:04 smithi132 ceph-mon[35283]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:14:04.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:04 smithi132 ceph-mon[35283]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-27T07:14:04.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:04 smithi132 ceph-mon[35283]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:05.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.359+0000 7f4660200700 1 -- 172.21.15.43:0/991269299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658109850 msgr2=0x7f46581111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:05.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.359+0000 7f4660200700 1 --2- 172.21.15.43:0/991269299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658109850 0x7f46581111b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f465000b0d0 tx=0x7f465000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:14:05.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.359+0000 7f4660200700 1 -- 172.21.15.43:0/991269299 shutdown_connections 2024-05-27T07:14:05.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.359+0000 7f4660200700 1 --2- 172.21.15.43:0/991269299 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658109850 0x7f46581111b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:05.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.359+0000 7f4660200700 1 --2- 172.21.15.43:0/991269299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46581045d0 0x7f46581049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:05.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.359+0000 7f4660200700 1 --2- 172.21.15.43:0/991269299 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4658108f00 0x7f4658109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:05.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.359+0000 7f4660200700 1 -- 172.21.15.43:0/991269299 >> 172.21.15.43:0/991269299 conn(0x7f46580ffe60 msgr2=0x7f46581022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:05.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.360+0000 7f4660200700 1 -- 172.21.15.43:0/991269299 shutdown_connections 2024-05-27T07:14:05.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.360+0000 7f4660200700 1 -- 172.21.15.43:0/991269299 wait complete. 2024-05-27T07:14:05.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:05.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:05.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:14:05.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:14:05.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:14:05.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.361+0000 7f4660200700 1 Processor -- start 2024-05-27T07:14:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.362+0000 7f4660200700 1 -- start start 2024-05-27T07:14:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.362+0000 7f4660200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581045d0 0x7f465819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.362+0000 7f4660200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4658108f00 0x7f465819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.362+0000 7f4660200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658109850 0x7f465819f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:05.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.362+0000 7f4660200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4658114380 con 0x7f4658109850 2024-05-27T07:14:05.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.362+0000 7f4660200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4658114200 con 0x7f4658108f00 2024-05-27T07:14:05.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.362+0000 7f4660200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4658114500 con 0x7f46581045d0 2024-05-27T07:14:05.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.363+0000 7f465d800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581045d0 0x7f465819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:05.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.363+0000 7f465ce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4658108f00 0x7f465819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:05.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.363+0000 7f465e200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658109850 0x7f465819f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:05.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.363+0000 7f465ce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4658108f00 0x7f465819b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:60370/0 (socket says 172.21.15.43:60370) 2024-05-27T07:14:05.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.363+0000 7f465ce00700 1 -- 172.21.15.43:0/3582615302 learned_addr learned my addr 172.21.15.43:0/3582615302 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:05.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.363+0000 7f465d800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581045d0 0x7f465819ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:54758/0 (socket says 172.21.15.43:54758) 2024-05-27T07:14:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.364+0000 7f465e200700 1 -- 172.21.15.43:0/3582615302 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581045d0 msgr2=0x7f465819ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.364+0000 7f465e200700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581045d0 0x7f465819ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.364+0000 7f465e200700 1 -- 172.21.15.43:0/3582615302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4658108f00 msgr2=0x7f465819b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.364+0000 7f465e200700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4658108f00 0x7f465819b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.364+0000 7f465e200700 1 -- 172.21.15.43:0/3582615302 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465819fc10 con 0x7f4658109850 2024-05-27T07:14:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.364+0000 7f465ce00700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4658108f00 0x7f465819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:14:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.364+0000 7f465e200700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658109850 0x7f465819f570 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f465000b820 tx=0x7f4650007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.364+0000 7f4656000700 1 -- 172.21.15.43:0/3582615302 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465001f070 con 0x7f4658109850 2024-05-27T07:14:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.365+0000 7f4656000700 1 -- 172.21.15.43:0/3582615302 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4650005500 con 0x7f4658109850 2024-05-27T07:14:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.365+0000 7f4656000700 1 -- 172.21.15.43:0/3582615302 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465001b760 con 0x7f4658109850 2024-05-27T07:14:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.365+0000 7f4660200700 1 -- 172.21.15.43:0/3582615302 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f465819fea0 con 0x7f4658109850 2024-05-27T07:14:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.365+0000 7f4660200700 1 -- 172.21.15.43:0/3582615302 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46581a0330 con 0x7f4658109850 2024-05-27T07:14:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.366+0000 7f4660200700 1 -- 172.21.15.43:0/3582615302 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46580512b0 con 0x7f4658109850 2024-05-27T07:14:05.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.371+0000 7f4656000700 1 -- 172.21.15.43:0/3582615302 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f4650022070 con 0x7f4658109850 2024-05-27T07:14:05.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.372+0000 7f4656000700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46400b2a80 0x7f46400b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:05.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.373+0000 7f4656000700 1 -- 172.21.15.43:0/3582615302 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f465000f020 con 0x7f4658109850 2024-05-27T07:14:05.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.373+0000 7f465d800700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46400b2a80 0x7f46400b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:05.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.373+0000 7f4656000700 1 -- 172.21.15.43:0/3582615302 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4650029020 con 0x7f4658109850 2024-05-27T07:14:05.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.374+0000 7f465d800700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46400b2a80 0x7f46400b4f90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f464c005950 tx=0x7f464c00b410 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:05.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-05-27T07:14:05.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: Updating smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:14:05.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: Updating smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:14:05.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: Updating smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-27T07:14:05.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:05.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.681+0000 7f4660200700 1 -- 172.21.15.43:0/3582615302 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f465802e5e0 con 0x7f4658109850 2024-05-27T07:14:05.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.682+0000 7f4656000700 1 -- 172.21.15.43:0/3582615302 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11938 (secure 0 0 0) 0x7f46500ab880 con 0x7f4658109850 2024-05-27T07:14:05.673 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:14:05.673 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:13:52.990426+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34484,34448,34424,24484,34403]},"id":1}]} 2024-05-27T07:14:05.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.686+0000 7f4660200700 1 -- 172.21.15.43:0/3582615302 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46400b2a80 msgr2=0x7f46400b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:05.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.686+0000 7f4660200700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46400b2a80 0x7f46400b4f90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f464c005950 tx=0x7f464c00b410 comp rx=0 tx=0).stop 2024-05-27T07:14:05.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.686+0000 7f4660200700 1 -- 172.21.15.43:0/3582615302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658109850 msgr2=0x7f465819f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:05.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.686+0000 7f4660200700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658109850 0x7f465819f570 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f465000b820 tx=0x7f4650007d10 comp rx=0 tx=0).stop 2024-05-27T07:14:05.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.686+0000 7f4660200700 1 -- 172.21.15.43:0/3582615302 shutdown_connections 2024-05-27T07:14:05.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.686+0000 7f4660200700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46400b2a80 0x7f46400b4f90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:05.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.686+0000 7f4660200700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4658109850 0x7f465819f570 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:05.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.686+0000 7f4660200700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46581045d0 0x7f465819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:05.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.686+0000 7f4660200700 1 --2- 172.21.15.43:0/3582615302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4658108f00 0x7f465819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:05.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.686+0000 7f4660200700 1 -- 172.21.15.43:0/3582615302 >> 172.21.15.43:0/3582615302 conn(0x7f46580ffe60 msgr2=0x7f4658107150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:05.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.687+0000 7f4660200700 1 -- 172.21.15.43:0/3582615302 shutdown_connections 2024-05-27T07:14:05.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:05.687+0000 7f4660200700 1 -- 172.21.15.43:0/3582615302 wait complete. 2024-05-27T07:14:05.678 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 30 2024-05-27T07:14:06.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:06 smithi043 ceph-mon[31531]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 0 B/s wr, 14 op/s 2024-05-27T07:14:06.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:06.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:06.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:06.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:06.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:06 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3582615302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:14:06.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:06 smithi179 ceph-mon[35888]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 0 B/s wr, 14 op/s 2024-05-27T07:14:06.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:06.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:06.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:06.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:06.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:06 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3582615302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:14:06.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:06 smithi132 ceph-mon[35283]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 0 B/s wr, 14 op/s 2024-05-27T07:14:06.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:06.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:06.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:06.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:06.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:06 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3582615302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:14:06.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-27T07:14:07.145 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:08.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:08 smithi179 ceph-mon[35888]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 0 B/s wr, 16 op/s 2024-05-27T07:14:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:08 smithi043 ceph-mon[31531]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 0 B/s wr, 16 op/s 2024-05-27T07:14:08.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:08 smithi132 ceph-mon[35283]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 0 B/s wr, 16 op/s 2024-05-27T07:14:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.642+0000 7fb8d2a00700 1 -- 172.21.15.43:0/2196363319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8cc10b890 msgr2=0x7fb8cc1131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.642+0000 7fb8d2a00700 1 --2- 172.21.15.43:0/2196363319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8cc10b890 0x7fb8cc1131f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb8c400b0d0 tx=0x7fb8c400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.642+0000 7fb8d2a00700 1 -- 172.21.15.43:0/2196363319 shutdown_connections 2024-05-27T07:14:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.642+0000 7fb8d2a00700 1 --2- 172.21.15.43:0/2196363319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8cc10b890 0x7fb8cc1131f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.642+0000 7fb8d2a00700 1 --2- 172.21.15.43:0/2196363319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8cc106610 0x7fb8cc106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.642+0000 7fb8d2a00700 1 --2- 172.21.15.43:0/2196363319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8cc10af40 0x7fb8cc10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.642+0000 7fb8d2a00700 1 -- 172.21.15.43:0/2196363319 >> 172.21.15.43:0/2196363319 conn(0x7fb8cc102020 msgr2=0x7fb8cc104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.642+0000 7fb8d2a00700 1 -- 172.21.15.43:0/2196363319 shutdown_connections 2024-05-27T07:14:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.642+0000 7fb8d2a00700 1 -- 172.21.15.43:0/2196363319 wait complete. 2024-05-27T07:14:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.643+0000 7fb8d2a00700 1 Processor -- start 2024-05-27T07:14:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.643+0000 7fb8d2a00700 1 -- start start 2024-05-27T07:14:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.644+0000 7fb8d2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8cc106610 0x7fb8cc1a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.644+0000 7fb8d2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8cc10af40 0x7fb8cc1a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.644+0000 7fb8d2a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8cc10b890 0x7fb8cc19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.644+0000 7fb8d2a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc116390 con 0x7fb8cc10af40 2024-05-27T07:14:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.644+0000 7fb8d2a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc116210 con 0x7fb8cc106610 2024-05-27T07:14:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.644+0000 7fb8d2a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc116510 con 0x7fb8cc10b890 2024-05-27T07:14:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.644+0000 7fb8d0c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8cc10af40 0x7fb8cc1a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.644+0000 7fb8d0c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8cc10af40 0x7fb8cc1a2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44780/0 (socket says 172.21.15.43:44780) 2024-05-27T07:14:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.644+0000 7fb8d0c00700 1 -- 172.21.15.43:0/4244002596 learned_addr learned my addr 172.21.15.43:0/4244002596 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.645+0000 7fb8d1600700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8cc106610 0x7fb8cc1a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.645+0000 7fb8d2000700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8cc10b890 0x7fb8cc19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.645+0000 7fb8d0c00700 1 -- 172.21.15.43:0/4244002596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8cc10b890 msgr2=0x7fb8cc19cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.645+0000 7fb8d0c00700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8cc10b890 0x7fb8cc19cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.645+0000 7fb8d0c00700 1 -- 172.21.15.43:0/4244002596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8cc106610 msgr2=0x7fb8cc1a29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.645+0000 7fb8d0c00700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8cc106610 0x7fb8cc1a29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.645+0000 7fb8d0c00700 1 -- 172.21.15.43:0/4244002596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc19d280 con 0x7fb8cc10af40 2024-05-27T07:14:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.645+0000 7fb8d0c00700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8cc10af40 0x7fb8cc1a2f00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800d950 tx=0x7fb8b800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.646+0000 7fb8ca000700 1 -- 172.21.15.43:0/4244002596 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8016070 con 0x7fb8cc10af40 2024-05-27T07:14:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.646+0000 7fb8ca000700 1 -- 172.21.15.43:0/4244002596 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8b8004830 con 0x7fb8cc10af40 2024-05-27T07:14:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.646+0000 7fb8d2a00700 1 -- 172.21.15.43:0/4244002596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8cc19d570 con 0x7fb8cc10af40 2024-05-27T07:14:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.646+0000 7fb8ca000700 1 -- 172.21.15.43:0/4244002596 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b80053e0 con 0x7fb8cc10af40 2024-05-27T07:14:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.646+0000 7fb8d2a00700 1 -- 172.21.15.43:0/4244002596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8cc1a9770 con 0x7fb8cc10af40 2024-05-27T07:14:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.647+0000 7fb8d2a00700 1 -- 172.21.15.43:0/4244002596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8cc107ac0 con 0x7fb8cc10af40 2024-05-27T07:14:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.651+0000 7fb8ca000700 1 -- 172.21.15.43:0/4244002596 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7fb8b8005580 con 0x7fb8cc10af40 2024-05-27T07:14:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.653+0000 7fb8ca000700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8b40b2a80 0x7fb8b40b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.653+0000 7fb8ca000700 1 -- 172.21.15.43:0/4244002596 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fb8b80e2de0 con 0x7fb8cc10af40 2024-05-27T07:14:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.653+0000 7fb8d1600700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8b40b2a80 0x7fb8b40b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.654+0000 7fb8d1600700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8b40b2a80 0x7fb8b40b4f90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0004630 tx=0x7fb8c000b3c0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.656+0000 7fb8ca000700 1 -- 172.21.15.43:0/4244002596 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb8b801f020 con 0x7fb8cc10af40 2024-05-27T07:14:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.896+0000 7fb8d2a00700 1 -- 172.21.15.43:0/4244002596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb8cc02e5e0 con 0x7fb8cc10af40 2024-05-27T07:14:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.899+0000 7fb8ca000700 1 -- 172.21.15.43:0/4244002596 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+25126 (secure 0 0 0) 0x7fb8b80aa2a0 con 0x7fb8cc10af40 2024-05-27T07:14:09.891 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:14:09.891 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":91,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:13:54.391605+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6800/1809594794":"2024-05-28T07:13:54.391569+0000","172.21.15.43:0/2266081183":"2024-05-28T07:13:54.391569+0000","172.21.15.43:0/3331041602":"2024-05-28T07:13:54.391569+0000","172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/3399450175":"2024-05-28T07:13:54.391569+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:6801/1809594794":"2024-05-28T07:13:54.391569+0000","172.21.15.43:0/3253329315":"2024-05-28T07:13:54.391569+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:14:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 -- 172.21.15.43:0/4244002596 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8b40b2a80 msgr2=0x7fb8b40b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8b40b2a80 0x7fb8b40b4f90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0004630 tx=0x7fb8c000b3c0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 -- 172.21.15.43:0/4244002596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8cc10af40 msgr2=0x7fb8cc1a2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8cc10af40 0x7fb8cc1a2f00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800d950 tx=0x7fb8b800dc60 comp rx=0 tx=0).stop 2024-05-27T07:14:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 -- 172.21.15.43:0/4244002596 shutdown_connections 2024-05-27T07:14:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8b40b2a80 0x7fb8b40b4f90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8cc10b890 0x7fb8cc19cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8cc106610 0x7fb8cc1a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 --2- 172.21.15.43:0/4244002596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8cc10af40 0x7fb8cc1a2f00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 -- 172.21.15.43:0/4244002596 >> 172.21.15.43:0/4244002596 conn(0x7fb8cc102020 msgr2=0x7fb8cc10a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.903+0000 7fb8d2a00700 1 -- 172.21.15.43:0/4244002596 shutdown_connections 2024-05-27T07:14:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:09.904+0000 7fb8d2a00700 1 -- 172.21.15.43:0/4244002596 wait complete. 2024-05-27T07:14:10.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:10 smithi043 ceph-mon[31531]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-05-27T07:14:10.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:10 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4244002596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:14:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:10 smithi179 ceph-mon[35888]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-05-27T07:14:10.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:10 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4244002596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:14:10.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:10 smithi132 ceph-mon[35283]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-05-27T07:14:10.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:10 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4244002596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:14:10.889 DEBUG:tasks.fwd_scrub:Yielding 2024-05-27T07:14:10.890 INFO:teuthology.run_tasks:Running task exec... 2024-05-27T07:14:10.900 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:14:10.900 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-05-27T07:14:10.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-05-27T07:14:10.900 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-27T07:14:10.900 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:14:10.901 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:14:11.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.188+0000 7f1862000640 1 -- 172.21.15.43:0/2381335402 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185c106430 msgr2=0x7f185c112800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:11.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.188+0000 7f1862000640 1 --2- 172.21.15.43:0/2381335402 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185c106430 0x7f185c112800 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1854009ca0 tx=0x7f185402a8c0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.189+0000 7f1862000640 1 -- 172.21.15.43:0/2381335402 shutdown_connections 2024-05-27T07:14:11.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.189+0000 7f1862000640 1 --2- 172.21.15.43:0/2381335402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f185c10c180 0x7f185c105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.189+0000 7f1862000640 1 --2- 172.21.15.43:0/2381335402 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185c106430 0x7f185c112800 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.189+0000 7f1862000640 1 --2- 172.21.15.43:0/2381335402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c10bce0 0x7f185c105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.189+0000 7f1862000640 1 -- 172.21.15.43:0/2381335402 >> 172.21.15.43:0/2381335402 conn(0x7f185c101380 msgr2=0x7f185c1037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:11.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.189+0000 7f1862000640 1 -- 172.21.15.43:0/2381335402 shutdown_connections 2024-05-27T07:14:11.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.189+0000 7f1862000640 1 -- 172.21.15.43:0/2381335402 wait complete. 2024-05-27T07:14:11.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.191+0000 7f1862000640 1 Processor -- start 2024-05-27T07:14:11.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.191+0000 7f1862000640 1 -- start start 2024-05-27T07:14:11.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f1862000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185c106430 0x7f185c1a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:11.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f1862000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c10bce0 0x7f185c1a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:11.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f1862000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f185c10c180 0x7f185c19c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:11.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f1862000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f185c115df0 con 0x7f185c106430 2024-05-27T07:14:11.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f1862000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f185c115c70 con 0x7f185c10c180 2024-05-27T07:14:11.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f1862000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f185c115f70 con 0x7f185c10bce0 2024-05-27T07:14:11.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f1860c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185c106430 0x7f185c1a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:11.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f1860c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185c106430 0x7f185c1a2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44798/0 (socket says 172.21.15.43:44798) 2024-05-27T07:14:11.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f1860c00640 1 -- 172.21.15.43:0/271862313 learned_addr learned my addr 172.21.15.43:0/271862313 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:11.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f1861600640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f185c10c180 0x7f185c19c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:11.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.192+0000 7f185be00640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c10bce0 0x7f185c1a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1860c00640 1 -- 172.21.15.43:0/271862313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c10bce0 msgr2=0x7f185c1a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1860c00640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c10bce0 0x7f185c1a2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1860c00640 1 -- 172.21.15.43:0/271862313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f185c10c180 msgr2=0x7f185c19c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1860c00640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f185c10c180 0x7f185c19c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1860c00640 1 -- 172.21.15.43:0/271862313 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185c19ce40 con 0x7f185c106430 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1861600640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f185c10c180 0x7f185c19c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1860c00640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185c106430 0x7f185c1a2540 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f185000e9f0 tx=0x7f185000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1859600640 1 -- 172.21.15.43:0/271862313 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185000ccc0 con 0x7f185c106430 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1859600640 1 -- 172.21.15.43:0/271862313 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f18500054b0 con 0x7f185c106430 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1859600640 1 -- 172.21.15.43:0/271862313 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850010430 con 0x7f185c106430 2024-05-27T07:14:11.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1862000640 1 -- 172.21.15.43:0/271862313 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185c19d130 con 0x7f185c106430 2024-05-27T07:14:11.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.193+0000 7f1862000640 1 -- 172.21.15.43:0/271862313 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f185c1a9330 con 0x7f185c106430 2024-05-27T07:14:11.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.195+0000 7f1862000640 1 -- 172.21.15.43:0/271862313 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185c050860 con 0x7f185c106430 2024-05-27T07:14:11.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.197+0000 7f1859600640 1 -- 172.21.15.43:0/271862313 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f18500046c0 con 0x7f185c106430 2024-05-27T07:14:11.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.200+0000 7f1859600640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18440b29e0 0x7f18440b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:11.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.200+0000 7f1859600640 1 -- 172.21.15.43:0/271862313 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f18500e3200 con 0x7f185c106430 2024-05-27T07:14:11.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.200+0000 7f185be00640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18440b29e0 0x7f18440b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:11.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.201+0000 7f185be00640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18440b29e0 0x7f18440b4ef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f185c10a1c0 tx=0x7f1848005ef0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:11.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.203+0000 7f1859600640 1 -- 172.21.15.43:0/271862313 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1850054df0 con 0x7f185c106430 2024-05-27T07:14:11.361 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:11.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.425+0000 7f1862000640 1 -- 172.21.15.43:0/271862313 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f185c19dc20 con 0x7f185c106430 2024-05-27T07:14:11.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.432+0000 7f1859600640 1 -- 172.21.15.43:0/271862313 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v29)=0 v29) ==== 129+0+0 (secure 0 0 0) 0x7f18500aa6c0 con 0x7f185c106430 2024-05-27T07:14:11.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 -- 172.21.15.43:0/271862313 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18440b29e0 msgr2=0x7f18440b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:11.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18440b29e0 0x7f18440b4ef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f185c10a1c0 tx=0x7f1848005ef0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 -- 172.21.15.43:0/271862313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185c106430 msgr2=0x7f185c1a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:11.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185c106430 0x7f185c1a2540 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f185000e9f0 tx=0x7f185000eeb0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 -- 172.21.15.43:0/271862313 shutdown_connections 2024-05-27T07:14:11.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18440b29e0 0x7f18440b4ef0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f185c10c180 0x7f185c19c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c10bce0 0x7f185c1a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 --2- 172.21.15.43:0/271862313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185c106430 0x7f185c1a2540 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 -- 172.21.15.43:0/271862313 >> 172.21.15.43:0/271862313 conn(0x7f185c101380 msgr2=0x7f185c1086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:11.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 -- 172.21.15.43:0/271862313 shutdown_connections 2024-05-27T07:14:11.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.435+0000 7f1862000640 1 -- 172.21.15.43:0/271862313 wait complete. 2024-05-27T07:14:11.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-27T07:14:11.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.738+0000 7f7b6da00640 1 -- 172.21.15.43:0/1446388070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b68108b90 msgr2=0x7f7b68102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:11.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.738+0000 7f7b6da00640 1 --2- 172.21.15.43:0/1446388070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b68108b90 0x7f7b68102730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009900 tx=0x7f7b5802a410 comp rx=0 tx=0).stop 2024-05-27T07:14:11.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.738+0000 7f7b6da00640 1 -- 172.21.15.43:0/1446388070 shutdown_connections 2024-05-27T07:14:11.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.738+0000 7f7b6da00640 1 --2- 172.21.15.43:0/1446388070 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b68109030 0x7f7b68102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.738+0000 7f7b6da00640 1 --2- 172.21.15.43:0/1446388070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b681032e0 0x7f7b6810f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.738+0000 7f7b6da00640 1 --2- 172.21.15.43:0/1446388070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b68108b90 0x7f7b68102730 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.738+0000 7f7b6da00640 1 -- 172.21.15.43:0/1446388070 >> 172.21.15.43:0/1446388070 conn(0x7f7b680782d0 msgr2=0x7f7b68078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:11.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.738+0000 7f7b6da00640 1 -- 172.21.15.43:0/1446388070 shutdown_connections 2024-05-27T07:14:11.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.738+0000 7f7b6da00640 1 -- 172.21.15.43:0/1446388070 wait complete. 2024-05-27T07:14:11.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.739+0000 7f7b6da00640 1 Processor -- start 2024-05-27T07:14:11.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.739+0000 7f7b6da00640 1 -- start start 2024-05-27T07:14:11.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b6da00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b681032e0 0x7f7b681a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b6da00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b68108b90 0x7f7b681a2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b6da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b68109030 0x7f7b6819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b6da00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7b68112c30 con 0x7f7b68109030 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b6da00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7b68112ab0 con 0x7f7b681032e0 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b6da00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7b68112db0 con 0x7f7b68108b90 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b67400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b68109030 0x7f7b6819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b67400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b68109030 0x7f7b6819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44824/0 (socket says 172.21.15.43:44824) 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b67400640 1 -- 172.21.15.43:0/2448542880 learned_addr learned my addr 172.21.15.43:0/2448542880 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b66a00640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b681032e0 0x7f7b681a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.740+0000 7f7b5fe00640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b68108b90 0x7f7b681a2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.741+0000 7f7b67400640 1 -- 172.21.15.43:0/2448542880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b68108b90 msgr2=0x7f7b681a2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:11.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.741+0000 7f7b67400640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b68108b90 0x7f7b681a2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.741+0000 7f7b67400640 1 -- 172.21.15.43:0/2448542880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b681032e0 msgr2=0x7f7b681a25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:11.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.741+0000 7f7b67400640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b681032e0 0x7f7b681a25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.741+0000 7f7b67400640 1 -- 172.21.15.43:0/2448542880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6819cef0 con 0x7f7b68109030 2024-05-27T07:14:11.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.741+0000 7f7b67400640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b68109030 0x7f7b6819c810 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000eaa0 tx=0x7f7b6000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:11.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.741+0000 7f7b5f400640 1 -- 172.21.15.43:0/2448542880 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6000cdb0 con 0x7f7b68109030 2024-05-27T07:14:11.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.741+0000 7f7b6da00640 1 -- 172.21.15.43:0/2448542880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b6819d1e0 con 0x7f7b68109030 2024-05-27T07:14:11.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.741+0000 7f7b5f400640 1 -- 172.21.15.43:0/2448542880 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b60005020 con 0x7f7b68109030 2024-05-27T07:14:11.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.742+0000 7f7b6da00640 1 -- 172.21.15.43:0/2448542880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b681a9430 con 0x7f7b68109030 2024-05-27T07:14:11.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.742+0000 7f7b5f400640 1 -- 172.21.15.43:0/2448542880 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6001f060 con 0x7f7b68109030 2024-05-27T07:14:11.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.743+0000 7f7b6da00640 1 -- 172.21.15.43:0/2448542880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b68103e70 con 0x7f7b68109030 2024-05-27T07:14:11.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.747+0000 7f7b5f400640 1 -- 172.21.15.43:0/2448542880 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f7b60004a40 con 0x7f7b68109030 2024-05-27T07:14:11.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.749+0000 7f7b5f400640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7b480b29e0 0x7f7b480b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:11.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.750+0000 7f7b5f400640 1 -- 172.21.15.43:0/2448542880 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f7b600e2c20 con 0x7f7b68109030 2024-05-27T07:14:11.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.750+0000 7f7b66a00640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7b480b29e0 0x7f7b480b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:11.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.751+0000 7f7b66a00640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7b480b29e0 0x7f7b480b4ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7b58004770 tx=0x7f7b580058d0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:11.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.752+0000 7f7b5f400640 1 -- 172.21.15.43:0/2448542880 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7b60016030 con 0x7f7b68109030 2024-05-27T07:14:11.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.974+0000 7f7b6da00640 1 -- 172.21.15.43:0/2448542880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f7b681a9600 con 0x7f7b68109030 2024-05-27T07:14:11.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.981+0000 7f7b5f400640 1 -- 172.21.15.43:0/2448542880 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7f7b600aa0e0 con 0x7f7b68109030 2024-05-27T07:14:11.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.985+0000 7f7b6da00640 1 -- 172.21.15.43:0/2448542880 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7b480b29e0 msgr2=0x7f7b480b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:11.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.985+0000 7f7b6da00640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7b480b29e0 0x7f7b480b4ef0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7b58004770 tx=0x7f7b580058d0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.985+0000 7f7b6da00640 1 -- 172.21.15.43:0/2448542880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b68109030 msgr2=0x7f7b6819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:11.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.985+0000 7f7b6da00640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b68109030 0x7f7b6819c810 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000eaa0 tx=0x7f7b6000ef60 comp rx=0 tx=0).stop 2024-05-27T07:14:11.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.985+0000 7f7b6da00640 1 -- 172.21.15.43:0/2448542880 shutdown_connections 2024-05-27T07:14:11.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.985+0000 7f7b6da00640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7b480b29e0 0x7f7b480b4ef0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.985+0000 7f7b6da00640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b68109030 0x7f7b6819c810 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.985+0000 7f7b6da00640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b68108b90 0x7f7b681a2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.985+0000 7f7b6da00640 1 --2- 172.21.15.43:0/2448542880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b681032e0 0x7f7b681a25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:11.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.985+0000 7f7b6da00640 1 -- 172.21.15.43:0/2448542880 >> 172.21.15.43:0/2448542880 conn(0x7f7b680782d0 msgr2=0x7f7b6806a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:11.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.986+0000 7f7b6da00640 1 -- 172.21.15.43:0/2448542880 shutdown_connections 2024-05-27T07:14:11.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:11.986+0000 7f7b6da00640 1 -- 172.21.15.43:0/2448542880 wait complete. 2024-05-27T07:14:12.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-27T07:14:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.288+0000 7f139be00640 1 -- 172.21.15.43:0/538789885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139c075480 msgr2=0x7f139c075890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.288+0000 7f139be00640 1 --2- 172.21.15.43:0/538789885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139c075480 0x7f139c075890 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f138c009900 tx=0x7f138c02a480 comp rx=0 tx=0).stop 2024-05-27T07:14:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.288+0000 7f139be00640 1 -- 172.21.15.43:0/538789885 shutdown_connections 2024-05-27T07:14:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.288+0000 7f139be00640 1 --2- 172.21.15.43:0/538789885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f139c10d990 0x7f139c10fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.288+0000 7f139be00640 1 --2- 172.21.15.43:0/538789885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f139c1102f0 0x7f139c112710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.288+0000 7f139be00640 1 --2- 172.21.15.43:0/538789885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139c075480 0x7f139c075890 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.288+0000 7f139be00640 1 -- 172.21.15.43:0/538789885 >> 172.21.15.43:0/538789885 conn(0x7f139c079240 msgr2=0x7f139c079680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.288+0000 7f139be00640 1 -- 172.21.15.43:0/538789885 shutdown_connections 2024-05-27T07:14:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.288+0000 7f139be00640 1 -- 172.21.15.43:0/538789885 wait complete. 2024-05-27T07:14:12.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.289+0000 7f139be00640 1 Processor -- start 2024-05-27T07:14:12.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.289+0000 7f139be00640 1 -- start start 2024-05-27T07:14:12.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.290+0000 7f139be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f139c075480 0x7f139c1a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:12.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.290+0000 7f139be00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f139c10d990 0x7f139c1a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:12.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.290+0000 7f139be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139c1102f0 0x7f139c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:12.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.290+0000 7f139be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f139c115d80 con 0x7f139c1102f0 2024-05-27T07:14:12.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.290+0000 7f139be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f139c115c00 con 0x7f139c075480 2024-05-27T07:14:12.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.290+0000 7f139be00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f139c115f00 con 0x7f139c10d990 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.290+0000 7f139b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139c1102f0 0x7f139c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.290+0000 7f139b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139c1102f0 0x7f139c19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44846/0 (socket says 172.21.15.43:44846) 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.290+0000 7f139b400640 1 -- 172.21.15.43:0/2827742192 learned_addr learned my addr 172.21.15.43:0/2827742192 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.290+0000 7f139aa00640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f139c075480 0x7f139c1a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.291+0000 7f139a000640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f139c10d990 0x7f139c1a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.291+0000 7f139b400640 1 -- 172.21.15.43:0/2827742192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f139c10d990 msgr2=0x7f139c1a2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.291+0000 7f139b400640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f139c10d990 0x7f139c1a2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.291+0000 7f139b400640 1 -- 172.21.15.43:0/2827742192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f139c075480 msgr2=0x7f139c1a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.291+0000 7f139b400640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f139c075480 0x7f139c1a2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.291+0000 7f139b400640 1 -- 172.21.15.43:0/2827742192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139c19ce30 con 0x7f139c1102f0 2024-05-27T07:14:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.291+0000 7f139b400640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139c1102f0 0x7f139c19c700 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f139400eaa0 tx=0x7f139400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:12.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.292+0000 7f1393400640 1 -- 172.21.15.43:0/2827742192 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139400cdb0 con 0x7f139c1102f0 2024-05-27T07:14:12.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.292+0000 7f139be00640 1 -- 172.21.15.43:0/2827742192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f139c19d0c0 con 0x7f139c1102f0 2024-05-27T07:14:12.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.292+0000 7f139be00640 1 -- 172.21.15.43:0/2827742192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f139c1a9320 con 0x7f139c1102f0 2024-05-27T07:14:12.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.292+0000 7f1393400640 1 -- 172.21.15.43:0/2827742192 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1394005020 con 0x7f139c1102f0 2024-05-27T07:14:12.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.292+0000 7f1393400640 1 -- 172.21.15.43:0/2827742192 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13940116d0 con 0x7f139c1102f0 2024-05-27T07:14:12.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.293+0000 7f139be00640 1 -- 172.21.15.43:0/2827742192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f137c005290 con 0x7f139c1102f0 2024-05-27T07:14:12.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.298+0000 7f1393400640 1 -- 172.21.15.43:0/2827742192 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f1394004a40 con 0x7f139c1102f0 2024-05-27T07:14:12.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.299+0000 7f1393400640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13800b2a60 0x7f13800b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:12.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.299+0000 7f1393400640 1 -- 172.21.15.43:0/2827742192 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f13940e2df0 con 0x7f139c1102f0 2024-05-27T07:14:12.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.300+0000 7f139aa00640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13800b2a60 0x7f13800b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:12.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.301+0000 7f139aa00640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13800b2a60 0x7f13800b4f70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f138c0057c0 tx=0x7f138c000bc0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:12.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.302+0000 7f1393400640 1 -- 172.21.15.43:0/2827742192 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1394016030 con 0x7f139c1102f0 2024-05-27T07:14:12.411 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 0 B/s wr, 15 op/s 2024-05-27T07:14:12.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/271862313' entity='client.admin' 2024-05-27T07:14:12.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:12.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:12.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:12.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:12.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2448542880' entity='client.admin' 2024-05-27T07:14:12.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:12.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:12.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:12.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:12 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 0 B/s wr, 15 op/s 2024-05-27T07:14:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/271862313' entity='client.admin' 2024-05-27T07:14:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2448542880' entity='client.admin' 2024-05-27T07:14:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:12.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:12.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:12 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:12.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.515+0000 7f139be00640 1 -- 172.21.15.43:0/2827742192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f137c005580 con 0x7f139c1102f0 2024-05-27T07:14:12.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.522+0000 7f1393400640 1 -- 172.21.15.43:0/2827742192 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f13940a9ce0 con 0x7f139c1102f0 2024-05-27T07:14:12.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.525+0000 7f139be00640 1 -- 172.21.15.43:0/2827742192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13800b2a60 msgr2=0x7f13800b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:12.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13800b2a60 0x7f13800b4f70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f138c0057c0 tx=0x7f138c000bc0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 -- 172.21.15.43:0/2827742192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139c1102f0 msgr2=0x7f139c19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:12.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139c1102f0 0x7f139c19c700 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f139400eaa0 tx=0x7f139400ef60 comp rx=0 tx=0).stop 2024-05-27T07:14:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 -- 172.21.15.43:0/2827742192 shutdown_connections 2024-05-27T07:14:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13800b2a60 0x7f13800b4f70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f139c1102f0 0x7f139c19c700 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f139c10d990 0x7f139c1a2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 --2- 172.21.15.43:0/2827742192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f139c075480 0x7f139c1a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 -- 172.21.15.43:0/2827742192 >> 172.21.15.43:0/2827742192 conn(0x7f139c079240 msgr2=0x7f139c102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:12.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 -- 172.21.15.43:0/2827742192 shutdown_connections 2024-05-27T07:14:12.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.526+0000 7f139be00640 1 -- 172.21.15.43:0/2827742192 wait complete. 2024-05-27T07:14:12.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-27T07:14:12.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: pgmap v11: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 0 B/s wr, 15 op/s 2024-05-27T07:14:12.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/271862313' entity='client.admin' 2024-05-27T07:14:12.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:12.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:12.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:12.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:12.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2448542880' entity='client.admin' 2024-05-27T07:14:12.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:12.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:12.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:12.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:12 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.836+0000 7f5d59800640 1 -- 172.21.15.43:0/2700773770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d5410a340 msgr2=0x7f5d5410a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.836+0000 7f5d59800640 1 --2- 172.21.15.43:0/2700773770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d5410a340 0x7f5d5410a750 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009900 tx=0x7f5d4402a410 comp rx=0 tx=0).stop 2024-05-27T07:14:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.837+0000 7f5d59800640 1 -- 172.21.15.43:0/2700773770 shutdown_connections 2024-05-27T07:14:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.837+0000 7f5d59800640 1 --2- 172.21.15.43:0/2700773770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d5410f390 0x7f5d54116b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.837+0000 7f5d59800640 1 --2- 172.21.15.43:0/2700773770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d5410e8d0 0x7f5d5410ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.837+0000 7f5d59800640 1 --2- 172.21.15.43:0/2700773770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d5410a340 0x7f5d5410a750 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.837+0000 7f5d59800640 1 -- 172.21.15.43:0/2700773770 >> 172.21.15.43:0/2700773770 conn(0x7f5d54076cd0 msgr2=0x7f5d54077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.837+0000 7f5d59800640 1 -- 172.21.15.43:0/2700773770 shutdown_connections 2024-05-27T07:14:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.837+0000 7f5d59800640 1 -- 172.21.15.43:0/2700773770 wait complete. 2024-05-27T07:14:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.838+0000 7f5d59800640 1 Processor -- start 2024-05-27T07:14:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.838+0000 7f5d59800640 1 -- start start 2024-05-27T07:14:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d59800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d5410a340 0x7f5d541a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d59800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d5410e8d0 0x7f5d541a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d59800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d5410f390 0x7f5d541a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d59800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5411a150 con 0x7f5d5410a340 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d59800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5d54119fd0 con 0x7f5d5410f390 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d59800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5411a2d0 con 0x7f5d5410e8d0 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d53e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d5410a340 0x7f5d541a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d53e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d5410a340 0x7f5d541a68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44868/0 (socket says 172.21.15.43:44868) 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d53e00640 1 -- 172.21.15.43:0/3365234467 learned_addr learned my addr 172.21.15.43:0/3365234467 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d4be00640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d5410e8d0 0x7f5d541a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.839+0000 7f5d58e00640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d5410f390 0x7f5d541a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.840+0000 7f5d53e00640 1 -- 172.21.15.43:0/3365234467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d5410e8d0 msgr2=0x7f5d541a6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.840+0000 7f5d53e00640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d5410e8d0 0x7f5d541a6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.840+0000 7f5d53e00640 1 -- 172.21.15.43:0/3365234467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d5410f390 msgr2=0x7f5d541a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.840+0000 7f5d53e00640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d5410f390 0x7f5d541a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.840+0000 7f5d53e00640 1 -- 172.21.15.43:0/3365234467 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d541a1190 con 0x7f5d5410a340 2024-05-27T07:14:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.840+0000 7f5d53e00640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d5410a340 0x7f5d541a68c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009230 tx=0x7f5d44004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.840+0000 7f5d51600640 1 -- 172.21.15.43:0/3365234467 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4403f030 con 0x7f5d5410a340 2024-05-27T07:14:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.840+0000 7f5d51600640 1 -- 172.21.15.43:0/3365234467 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d44003a80 con 0x7f5d5410a340 2024-05-27T07:14:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.840+0000 7f5d59800640 1 -- 172.21.15.43:0/3365234467 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d541a1420 con 0x7f5d5410a340 2024-05-27T07:14:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.841+0000 7f5d51600640 1 -- 172.21.15.43:0/3365234467 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44035070 con 0x7f5d5410a340 2024-05-27T07:14:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.841+0000 7f5d59800640 1 -- 172.21.15.43:0/3365234467 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d541ad760 con 0x7f5d5410a340 2024-05-27T07:14:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.842+0000 7f5d59800640 1 -- 172.21.15.43:0/3365234467 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d5406a230 con 0x7f5d5410a340 2024-05-27T07:14:12.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.844+0000 7f5d51600640 1 -- 172.21.15.43:0/3365234467 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f5d44042030 con 0x7f5d5410a340 2024-05-27T07:14:12.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.846+0000 7f5d51600640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d380b2ae0 0x7f5d380b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:12.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.846+0000 7f5d51600640 1 -- 172.21.15.43:0/3365234467 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f5d44103390 con 0x7f5d5410a340 2024-05-27T07:14:12.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.847+0000 7f5d4be00640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d380b2ae0 0x7f5d380b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:12.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.848+0000 7f5d4be00640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d380b2ae0 0x7f5d380b4ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c005fd0 tx=0x7f5d3c005850 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:12.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:12.851+0000 7f5d51600640 1 -- 172.21.15.43:0/3365234467 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5d440ca850 con 0x7f5d5410a340 2024-05-27T07:14:13.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.101+0000 7f5d59800640 1 -- 172.21.15.43:0/3365234467 --> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f5d541081f0 con 0x7f5d380b2ae0 2024-05-27T07:14:13.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.268+0000 7f5d51600640 1 -- 172.21.15.43:0/3365234467 <== mgr.35033 v2:172.21.15.43:6800/2368318638 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5d541081f0 con 0x7f5d380b2ae0 2024-05-27T07:14:13.259 INFO:teuthology.orchestra.run.smithi043.stdout:Schedule set for path / 2024-05-27T07:14:13.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 -- 172.21.15.43:0/3365234467 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d380b2ae0 msgr2=0x7f5d380b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d380b2ae0 0x7f5d380b4ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c005fd0 tx=0x7f5d3c005850 comp rx=0 tx=0).stop 2024-05-27T07:14:13.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 -- 172.21.15.43:0/3365234467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d5410a340 msgr2=0x7f5d541a68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d5410a340 0x7f5d541a68c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009230 tx=0x7f5d44004270 comp rx=0 tx=0).stop 2024-05-27T07:14:13.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 -- 172.21.15.43:0/3365234467 shutdown_connections 2024-05-27T07:14:13.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d380b2ae0 0x7f5d380b4ff0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d5410f390 0x7f5d541a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d5410e8d0 0x7f5d541a6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 --2- 172.21.15.43:0/3365234467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d5410a340 0x7f5d541a68c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 -- 172.21.15.43:0/3365234467 >> 172.21.15.43:0/3365234467 conn(0x7f5d54076cd0 msgr2=0x7f5d54107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:13.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 -- 172.21.15.43:0/3365234467 shutdown_connections 2024-05-27T07:14:13.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.271+0000 7f5d59800640 1 -- 172.21.15.43:0/3365234467 wait complete. 2024-05-27T07:14:13.288 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-27T07:14:13.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.579+0000 7ff4ac800640 1 -- 172.21.15.43:0/3107382226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4a4110450 msgr2=0x7ff4a41128b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.579+0000 7ff4ac800640 1 --2- 172.21.15.43:0/3107382226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4a4110450 0x7ff4a41128b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009ca0 tx=0x7ff4a002a940 comp rx=0 tx=0).stop 2024-05-27T07:14:13.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.579+0000 7ff4ac800640 1 -- 172.21.15.43:0/3107382226 shutdown_connections 2024-05-27T07:14:13.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.579+0000 7ff4ac800640 1 --2- 172.21.15.43:0/3107382226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4a410daf0 0x7ff4a410ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.579+0000 7ff4ac800640 1 --2- 172.21.15.43:0/3107382226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4a4110450 0x7ff4a41128b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.579+0000 7ff4ac800640 1 --2- 172.21.15.43:0/3107382226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4a4076770 0x7ff4a4076b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.579+0000 7ff4ac800640 1 -- 172.21.15.43:0/3107382226 >> 172.21.15.43:0/3107382226 conn(0x7ff4a40ff320 msgr2=0x7ff4a4101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:13.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.579+0000 7ff4ac800640 1 -- 172.21.15.43:0/3107382226 shutdown_connections 2024-05-27T07:14:13.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.579+0000 7ff4ac800640 1 -- 172.21.15.43:0/3107382226 wait complete. 2024-05-27T07:14:13.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.580+0000 7ff4ac800640 1 Processor -- start 2024-05-27T07:14:13.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.581+0000 7ff4ac800640 1 -- start start 2024-05-27T07:14:13.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.581+0000 7ff4ac800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4a4076770 0x7ff4a41a2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.581+0000 7ff4ac800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4a410daf0 0x7ff4a41a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.581+0000 7ff4ac800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4a4110450 0x7ff4a419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.581+0000 7ff4ac800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a4115f10 con 0x7ff4a410daf0 2024-05-27T07:14:13.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.581+0000 7ff4ac800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a4115d90 con 0x7ff4a4110450 2024-05-27T07:14:13.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.581+0000 7ff4ac800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a4116090 con 0x7ff4a4076770 2024-05-27T07:14:13.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.582+0000 7ff4a9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4a410daf0 0x7ff4a41a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.582+0000 7ff4a9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4a410daf0 0x7ff4a41a2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44884/0 (socket says 172.21.15.43:44884) 2024-05-27T07:14:13.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.582+0000 7ff4a9400640 1 -- 172.21.15.43:0/1654877314 learned_addr learned my addr 172.21.15.43:0/1654877314 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:13.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.582+0000 7ff4a9e00640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4a4076770 0x7ff4a41a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.582+0000 7ff4aa800640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4a4110450 0x7ff4a419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.582+0000 7ff4a9400640 1 -- 172.21.15.43:0/1654877314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4a4076770 msgr2=0x7ff4a41a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.582+0000 7ff4a9400640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4a4076770 0x7ff4a41a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.582+0000 7ff4a9400640 1 -- 172.21.15.43:0/1654877314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4a4110450 msgr2=0x7ff4a419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.582+0000 7ff4a9400640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4a4110450 0x7ff4a419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.582+0000 7ff4a9400640 1 -- 172.21.15.43:0/1654877314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a419cf50 con 0x7ff4a410daf0 2024-05-27T07:14:13.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.583+0000 7ff4a9400640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4a410daf0 0x7ff4a41a2b80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff49400d950 tx=0x7ff49400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:13.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.583+0000 7ff49ea00640 1 -- 172.21.15.43:0/1654877314 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff494017070 con 0x7ff4a410daf0 2024-05-27T07:14:13.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.583+0000 7ff49ea00640 1 -- 172.21.15.43:0/1654877314 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff494009c30 con 0x7ff4a410daf0 2024-05-27T07:14:13.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.583+0000 7ff49ea00640 1 -- 172.21.15.43:0/1654877314 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49400f620 con 0x7ff4a410daf0 2024-05-27T07:14:13.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.583+0000 7ff4ac800640 1 -- 172.21.15.43:0/1654877314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a419d240 con 0x7ff4a410daf0 2024-05-27T07:14:13.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.584+0000 7ff4ac800640 1 -- 172.21.15.43:0/1654877314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a41a9490 con 0x7ff4a410daf0 2024-05-27T07:14:13.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.586+0000 7ff4ac800640 1 -- 172.21.15.43:0/1654877314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a40778b0 con 0x7ff4a410daf0 2024-05-27T07:14:13.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.588+0000 7ff49ea00640 1 -- 172.21.15.43:0/1654877314 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7ff494010460 con 0x7ff4a410daf0 2024-05-27T07:14:13.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.590+0000 7ff49ea00640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff48c0b29e0 0x7ff48c0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.591+0000 7ff49ea00640 1 -- 172.21.15.43:0/1654877314 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff4940e2670 con 0x7ff4a410daf0 2024-05-27T07:14:13.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.591+0000 7ff4a9e00640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff48c0b29e0 0x7ff48c0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.592+0000 7ff4a9e00640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff48c0b29e0 0x7ff48c0b4ef0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff4a419dba0 tx=0x7ff49800b390 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:13.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.595+0000 7ff49ea00640 1 -- 172.21.15.43:0/1654877314 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff494009de0 con 0x7ff4a410daf0 2024-05-27T07:14:13.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.819+0000 7ff1eb200700 1 -- 172.21.15.43:0/2525376185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1ec105300 msgr2=0x7ff1ec105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.819+0000 7ff1eb200700 1 --2- 172.21.15.43:0/2525376185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1ec105300 0x7ff1ec105710 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009990 tx=0x7ff1dc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.819+0000 7ff1eb200700 1 -- 172.21.15.43:0/2525376185 shutdown_connections 2024-05-27T07:14:13.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.819+0000 7ff1eb200700 1 --2- 172.21.15.43:0/2525376185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1ec10e730 0x7ff1ec110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.819+0000 7ff1eb200700 1 --2- 172.21.15.43:0/2525376185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1ec105300 0x7ff1ec105710 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.819+0000 7ff1eb200700 1 --2- 172.21.15.43:0/2525376185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1ec105c50 0x7ff1ec10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.819+0000 7ff1eb200700 1 -- 172.21.15.43:0/2525376185 >> 172.21.15.43:0/2525376185 conn(0x7ff1ec078ab0 msgr2=0x7ff1ec078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:13.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.820+0000 7ff1eb200700 1 -- 172.21.15.43:0/2525376185 shutdown_connections 2024-05-27T07:14:13.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.820+0000 7ff1eb200700 1 -- 172.21.15.43:0/2525376185 wait complete. 2024-05-27T07:14:13.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:13 smithi043 ceph-mon[31531]: pgmap v12: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 0 B/s wr, 18 op/s 2024-05-27T07:14:13.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:13 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2827742192' entity='client.admin' 2024-05-27T07:14:13.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:13 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:13.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:13 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:13.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:13 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:13.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:13 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:13.812 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:13 smithi043 ceph-mon[31531]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:13.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.821+0000 7ff1eb200700 1 Processor -- start 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.821+0000 7ff1eb200700 1 -- start start 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.822+0000 7ff1eb200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1ec105300 0x7ff1ec1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.822+0000 7ff1eb200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1ec105c50 0x7ff1ec1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.822+0000 7ff1eb200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1ec10e730 0x7ff1ec19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.822+0000 7ff1eb200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec1140f0 con 0x7ff1ec105c50 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.822+0000 7ff1eb200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec113f70 con 0x7ff1ec105300 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.822+0000 7ff1eb200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec114270 con 0x7ff1ec10e730 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.822+0000 7ff1e9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1ec105c50 0x7ff1ec1a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.822+0000 7ff1e9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1ec105c50 0x7ff1ec1a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44894/0 (socket says 172.21.15.43:44894) 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.822+0000 7ff1e9400700 1 -- 172.21.15.43:0/900607816 learned_addr learned my addr 172.21.15.43:0/900607816 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.823+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1ec105300 0x7ff1ec1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.823+0000 7ff1ea800700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1ec10e730 0x7ff1ec19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.823+0000 7ff1e9400700 1 -- 172.21.15.43:0/900607816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1ec10e730 msgr2=0x7ff1ec19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.823+0000 7ff1e9400700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1ec10e730 0x7ff1ec19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.823+0000 7ff1e9400700 1 -- 172.21.15.43:0/900607816 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1ec105300 msgr2=0x7ff1ec1a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.823+0000 7ff1e9400700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1ec105300 0x7ff1ec1a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.823+0000 7ff1e9400700 1 -- 172.21.15.43:0/900607816 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec19b020 con 0x7ff1ec105c50 2024-05-27T07:14:13.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.823+0000 7ff1ea800700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1ec10e730 0x7ff1ec19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:14:13.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.823+0000 7ff1e9400700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1ec105c50 0x7ff1ec1a0c90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800d950 tx=0x7ff1d800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:13.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.824+0000 7ff1e2a00700 1 -- 172.21.15.43:0/900607816 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8016070 con 0x7ff1ec105c50 2024-05-27T07:14:13.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.824+0000 7ff1e2a00700 1 -- 172.21.15.43:0/900607816 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1d8010460 con 0x7ff1ec105c50 2024-05-27T07:14:13.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.824+0000 7ff1eb200700 1 -- 172.21.15.43:0/900607816 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ec19b310 con 0x7ff1ec105c50 2024-05-27T07:14:13.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.824+0000 7ff1e2a00700 1 -- 172.21.15.43:0/900607816 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d800f620 con 0x7ff1ec105c50 2024-05-27T07:14:13.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.824+0000 7ff1eb200700 1 -- 172.21.15.43:0/900607816 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ec1a75e0 con 0x7ff1ec105c50 2024-05-27T07:14:13.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.828+0000 7ff4ac800640 1 -- 172.21.15.43:0/1654877314 --> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7ff4a4110f70 con 0x7ff48c0b29e0 2024-05-27T07:14:13.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.828+0000 7ff1e2a00700 1 -- 172.21.15.43:0/900607816 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7ff1d8019070 con 0x7ff1ec105c50 2024-05-27T07:14:13.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.828+0000 7ff1ff4fab80 1 Processor -- start 2024-05-27T07:14:13.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.829+0000 7ff1ff4fab80 1 -- start start 2024-05-27T07:14:13.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.829+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd7da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.829+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebbf9ec0 0x55edebd7df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.829+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 0x55edebd897a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.829+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55edebca6ec0 con 0x55edebce5a70 2024-05-27T07:14:13.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.829+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55edebc96a60 con 0x55edebbf9ec0 2024-05-27T07:14:13.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.829+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55edebc9aaa0 con 0x55edebd7e580 2024-05-27T07:14:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.829+0000 7ff1ea800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd7da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.830+0000 7ff1ea800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd7da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44896/0 (socket says 172.21.15.43:44896) 2024-05-27T07:14:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.830+0000 7ff1ea800700 1 -- 172.21.15.43:0/1760000778 learned_addr learned my addr 172.21.15.43:0/1760000778 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.830+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/1760000778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebbf9ec0 0x55edebd7df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.830+0000 7ff1e9400700 1 --2- 172.21.15.43:0/1760000778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 0x55edebd897a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.830+0000 7ff1ea800700 1 -- 172.21.15.43:0/1760000778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 msgr2=0x55edebd897a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.830+0000 7ff1ea800700 1 --2- 172.21.15.43:0/1760000778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 0x55edebd897a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.830+0000 7ff1ea800700 1 -- 172.21.15.43:0/1760000778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebbf9ec0 msgr2=0x55edebd7df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.830+0000 7ff1ea800700 1 --2- 172.21.15.43:0/1760000778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebbf9ec0 0x55edebd7df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.830+0000 7ff1ea800700 1 -- 172.21.15.43:0/1760000778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edebc5f4a0 con 0x55edebce5a70 2024-05-27T07:14:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.831+0000 7ff1e2a00700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1d00b2a80 0x7ff1d00b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.831+0000 7ff1e9400700 1 --2- 172.21.15.43:0/1760000778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 0x55edebd897a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:14:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.831+0000 7ff1e2a00700 1 -- 172.21.15.43:0/900607816 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff1d80e27b0 con 0x7ff1ec105c50 2024-05-27T07:14:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.831+0000 7ff1ea800700 1 --2- 172.21.15.43:0/1760000778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd7da20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009cd0 tx=0x7ff1e4007500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551fd3ae6486d9db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.831+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1d00b2a80 0x7ff1d00b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.832+0000 7ff1d7400700 1 -- 172.21.15.43:0/1760000778 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4019070 con 0x55edebce5a70 2024-05-27T07:14:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.832+0000 7ff1d7400700 1 -- 172.21.15.43:0/1760000778 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1e4007e40 con 0x55edebce5a70 2024-05-27T07:14:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.832+0000 7ff1d7400700 1 -- 172.21.15.43:0/1760000778 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e40138c0 con 0x55edebce5a70 2024-05-27T07:14:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.833+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1760000778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 msgr2=0x55edebd7da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.833+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1d00b2a80 0x7ff1d00b4f90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00b5c0 tx=0x7ff1dc005940 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.833+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/1760000778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd7da20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009cd0 tx=0x7ff1e4007500 comp rx=0 tx=0).stop 2024-05-27T07:14:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.833+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1760000778 shutdown_connections 2024-05-27T07:14:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.833+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/1760000778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 0x55edebd897a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.833+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/1760000778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd7da20 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.833+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/1760000778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebbf9ec0 0x55edebd7df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.833+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1760000778 >> 172.21.15.43:0/1760000778 conn(0x55edebc93410 msgr2=0x55edebd745f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.833+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1760000778 shutdown_connections 2024-05-27T07:14:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.833+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1760000778 wait complete. 2024-05-27T07:14:13.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.834+0000 7ff1ff4fab80 1 -- start start 2024-05-27T07:14:13.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.834+0000 7ff1ff4fab80 1 Processor -- start 2024-05-27T07:14:13.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.835+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebbf9ec0 0x55edebd84710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.835+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebce5a70 0x55edebd84c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.835+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebd7e580 0x55edebd85190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.835+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55edebb9d020 con 0x55edebd7e580 2024-05-27T07:14:13.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.835+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55edebc377e0 con 0x55edebce5a70 2024-05-27T07:14:13.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.835+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55edebd8a270 con 0x55edebbf9ec0 2024-05-27T07:14:13.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.835+0000 7ff1ff4fab80 10 client.0 _ll_register_callbacks cb 0x55edebc40cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:14:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.835+0000 7ff1ea800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebd7e580 0x55edebd85190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1ea800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebd7e580 0x55edebd85190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44900/0 (socket says 172.21.15.43:44900) 2024-05-27T07:14:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1ea800700 1 -- 172.21.15.43:0/1888899163 learned_addr learned my addr 172.21.15.43:0/1888899163 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebbf9ec0 0x55edebd84710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1e9400700 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebce5a70 0x55edebd84c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1ea800700 1 -- 172.21.15.43:0/1888899163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebbf9ec0 msgr2=0x55edebd84710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1ea800700 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebbf9ec0 0x55edebd84710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1ea800700 1 -- 172.21.15.43:0/1888899163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebce5a70 msgr2=0x55edebd84c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1ea800700 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebce5a70 0x55edebd84c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1ea800700 1 -- 172.21.15.43:0/1888899163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edebbe6950 con 0x55edebd7e580 2024-05-27T07:14:13.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1ea800700 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebd7e580 0x55edebd85190 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4013400 tx=0x7ff1e40146b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e8342c3fd39773 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:13.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.836+0000 7ff1cbe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.837+0000 7ff1cbe00700 1 -- 172.21.15.43:0/1888899163 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4019040 con 0x55edebd7e580 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.837+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1888899163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55edebca8e10 con 0x55edebd7e580 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.837+0000 7ff1cbe00700 1 -- 172.21.15.43:0/1888899163 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff1e4014aa0 con 0x55edebd7e580 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.837+0000 7ff1cbe00700 1 -- 172.21.15.43:0/1888899163 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff1e402c050 con 0x55edebd7e580 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.837+0000 7ff1ff4fab80 10 client.15486 fetch_fsmap learned FSMap version 30 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.837+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1888899163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55edebc670e0 con 0x55edebd7e580 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.838+0000 7ff1cbe00700 1 -- 172.21.15.43:0/1888899163 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55edebca8e10 con 0x55edebd7e580 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.838+0000 7ff1cbe00700 1 -- 172.21.15.43:0/1888899163 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff1e4013d10 con 0x55edebd7e580 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.838+0000 7ff1ff4fab80 10 client.15486 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.838+0000 7ff1ff4fab80 10 client.15486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:14:13.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.838+0000 7ff1ff4fab80 20 client.15486 populate_metadata read hostname 'smithi043' 2024-05-27T07:14:13.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.838+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55edebd96f60 0x55edebd99380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.838+0000 7ff1ff4fab80 4 client.15486 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:14:13.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.838+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1888899163 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edebca25f0 con 0x55edebd96f60 2024-05-27T07:14:13.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.839+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55edebd96f60 0x55edebd99380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.840+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55edebd96f60 0x55edebd99380 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b6b5a2dbc2fb434 server_cookie=e769132729303dc7 in_seq=0 out_seq=0 2024-05-27T07:14:13.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.840+0000 7ff1cbe00700 10 client.15486 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:13.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.842+0000 7ff1cbe00700 1 -- 172.21.15.43:0/1888899163 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55edebca25f0 con 0x55edebd96f60 2024-05-27T07:14:13.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.842+0000 7ff1cbe00700 10 client.15486 handle_command_reply: tid=0 2024-05-27T07:14:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.842+0000 7ff1ff4fab80 1 client.15486 shutdown 2024-05-27T07:14:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.842+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1888899163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebd7e580 msgr2=0x55edebd85190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.842+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebd7e580 0x55edebd85190 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4013400 tx=0x7ff1e40146b0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.843+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1888899163 shutdown_connections 2024-05-27T07:14:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.843+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55edebd96f60 0x55edebd99380 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.843+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebd7e580 0x55edebd85190 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.843+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebbf9ec0 0x55edebd84710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.843+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/1888899163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebce5a70 0x55edebd84c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.843+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1888899163 >> 172.21.15.43:0/1888899163 conn(0x55edebc93410 msgr2=0x55edebd74140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.843+0000 7ff1cbe00700 0 client.15486 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.843+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1888899163 shutdown_connections 2024-05-27T07:14:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.843+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/1888899163 wait complete. 2024-05-27T07:14:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.843+0000 7ff1ff4fab80 20 client.15486 trim_cache size 0 max 16384 2024-05-27T07:14:13.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:13 smithi132 ceph-mon[35283]: pgmap v12: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 0 B/s wr, 18 op/s 2024-05-27T07:14:13.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:13 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2827742192' entity='client.admin' 2024-05-27T07:14:13.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:13 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:13.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:13 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:13.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:13 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:13 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:13 smithi132 ceph-mon[35283]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.869+0000 7ff1ff4fab80 1 Processor -- start 2024-05-27T07:14:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.869+0000 7ff1ff4fab80 1 -- start start 2024-05-27T07:14:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.869+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebbf9ec0 0x55edebd7cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.869+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebce5a70 0x55edebd93310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.869+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 0x55edebd864d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.869+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec1ae1e0 con 0x55edebbf9ec0 2024-05-27T07:14:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.869+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec1afd20 con 0x55edebce5a70 2024-05-27T07:14:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.869+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec1ae8b0 con 0x55edebd7e580 2024-05-27T07:14:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebbf9ec0 0x55edebd7cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebbf9ec0 0x55edebd7cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44914/0 (socket says 172.21.15.43:44914) 2024-05-27T07:14:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1e9e00700 1 -- 172.21.15.43:0/2879528710 learned_addr learned my addr 172.21.15.43:0/2879528710 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1e9400700 1 --2- 172.21.15.43:0/2879528710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebce5a70 0x55edebd93310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1ea800700 1 --2- 172.21.15.43:0/2879528710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 0x55edebd864d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1e9e00700 1 -- 172.21.15.43:0/2879528710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 msgr2=0x55edebd864d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/2879528710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 0x55edebd864d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1e9e00700 1 -- 172.21.15.43:0/2879528710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebce5a70 msgr2=0x55edebd93310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/2879528710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebce5a70 0x55edebd93310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1e9e00700 1 -- 172.21.15.43:0/2879528710 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edebd82c80 con 0x55edebbf9ec0 2024-05-27T07:14:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1ea800700 1 --2- 172.21.15.43:0/2879528710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 0x55edebd864d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:14:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.870+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/2879528710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebbf9ec0 0x55edebd7cfe0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc003730 tx=0x7ff1dc022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae59235282da2d60 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.871+0000 7ff1d6a00700 1 -- 172.21.15.43:0/2879528710 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc003e20 con 0x55edebbf9ec0 2024-05-27T07:14:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.871+0000 7ff1d6a00700 1 -- 172.21.15.43:0/2879528710 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1dc00bc60 con 0x55edebbf9ec0 2024-05-27T07:14:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.871+0000 7ff1d6a00700 1 -- 172.21.15.43:0/2879528710 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0329e0 con 0x55edebbf9ec0 2024-05-27T07:14:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.872+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2879528710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebbf9ec0 msgr2=0x55edebd7cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.872+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/2879528710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebbf9ec0 0x55edebd7cfe0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc003730 tx=0x7ff1dc022000 comp rx=0 tx=0).stop 2024-05-27T07:14:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.872+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2879528710 shutdown_connections 2024-05-27T07:14:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.872+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/2879528710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebd7e580 0x55edebd864d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.872+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/2879528710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebbf9ec0 0x55edebd7cfe0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.872+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/2879528710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebce5a70 0x55edebd93310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.872+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2879528710 >> 172.21.15.43:0/2879528710 conn(0x55edebc93410 msgr2=0x55edebd85dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.872+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2879528710 shutdown_connections 2024-05-27T07:14:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.872+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2879528710 wait complete. 2024-05-27T07:14:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.873+0000 7ff1ff4fab80 1 -- start start 2024-05-27T07:14:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.873+0000 7ff1ff4fab80 1 Processor -- start 2024-05-27T07:14:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebbf9ec0 0x55edebd8f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd8acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1ff4fab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebd7e580 0x55edebd8b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec1af650 con 0x55edebce5a70 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec1aef80 con 0x55edebd7e580 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1ff4fab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec1ac160 con 0x55edebbf9ec0 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1ff4fab80 10 client.0 _ll_register_callbacks cb 0x55edebc40cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1e9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd8acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1e9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd8acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44916/0 (socket says 172.21.15.43:44916) 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1e9400700 1 -- 172.21.15.43:0/2958673818 learned_addr learned my addr 172.21.15.43:0/2958673818 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebbf9ec0 0x55edebd8f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1ea800700 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebd7e580 0x55edebd8b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.874+0000 7ff1e9400700 1 -- 172.21.15.43:0/2958673818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebbf9ec0 msgr2=0x55edebd8f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.875+0000 7ff1e9400700 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebbf9ec0 0x55edebd8f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.875+0000 7ff1e9400700 1 -- 172.21.15.43:0/2958673818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebd7e580 msgr2=0x55edebd8b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.875+0000 7ff1e9400700 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebd7e580 0x55edebd8b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.875+0000 7ff1e9400700 1 -- 172.21.15.43:0/2958673818 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edebd8b740 con 0x55edebce5a70 2024-05-27T07:14:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.875+0000 7ff1ea800700 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebd7e580 0x55edebd8b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:14:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.875+0000 7ff1e9400700 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd8acc0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80a9270 tx=0x7ff1d8010890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959cc15df645642e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.875+0000 7ff1cbe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:14:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.875+0000 7ff1cbe00700 1 -- 172.21.15.43:0/2958673818 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8020040 con 0x55edebce5a70 2024-05-27T07:14:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.875+0000 7ff1cbe00700 1 -- 172.21.15.43:0/2958673818 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff1d80a97d0 con 0x55edebce5a70 2024-05-27T07:14:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.875+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2958673818 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1dc00bc60 con 0x55edebce5a70 2024-05-27T07:14:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.876+0000 7ff1cbe00700 1 -- 172.21.15.43:0/2958673818 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff1d80e25d0 con 0x55edebce5a70 2024-05-27T07:14:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.876+0000 7ff1ff4fab80 10 client.15492 fetch_fsmap learned FSMap version 30 2024-05-27T07:14:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.876+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2958673818 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55edebe01550 con 0x55edebce5a70 2024-05-27T07:14:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.876+0000 7ff1cbe00700 1 -- 172.21.15.43:0/2958673818 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1dc00bc60 con 0x55edebce5a70 2024-05-27T07:14:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.877+0000 7ff1cbe00700 1 -- 172.21.15.43:0/2958673818 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff1d80e4ce0 con 0x55edebce5a70 2024-05-27T07:14:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.877+0000 7ff1ff4fab80 10 client.15492 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:14:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.877+0000 7ff1ff4fab80 10 client.15492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:14:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.877+0000 7ff1ff4fab80 20 client.15492 populate_metadata read hostname 'smithi043' 2024-05-27T07:14:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.877+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55edebe01c60 0x55edebe04080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.877+0000 7ff1ff4fab80 4 client.15492 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:14:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.877+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2958673818 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55edebb192d0 con 0x55edebe01c60 2024-05-27T07:14:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.877+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55edebe01c60 0x55edebe04080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.878+0000 7ff1e9e00700 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55edebe01c60 0x55edebe04080 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3df9baf4b5042de server_cookie=549d85cf910339df in_seq=0 out_seq=0 2024-05-27T07:14:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.878+0000 7ff1cbe00700 10 client.15492 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.882+0000 7ff1cbe00700 1 -- 172.21.15.43:0/2958673818 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55edebb192d0 con 0x55edebe01c60 2024-05-27T07:14:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.882+0000 7ff1cbe00700 10 client.15492 handle_command_reply: tid=0 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.882+0000 7ff1ff4fab80 1 client.15492 shutdown 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2958673818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 msgr2=0x55edebd8acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd8acc0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80a9270 tx=0x7ff1d8010890 comp rx=0 tx=0).stop 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2958673818 shutdown_connections 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55edebd7e580 0x55edebd8b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55edebbf9ec0 0x55edebd8f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55edebe01c60 0x55edebe04080 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 1 --2- 172.21.15.43:0/2958673818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55edebce5a70 0x55edebd8acc0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2958673818 >> 172.21.15.43:0/2958673818 conn(0x55edebc93410 msgr2=0x55edebd7cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1cbe00700 0 client.15492 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2958673818 shutdown_connections 2024-05-27T07:14:13.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 1 -- 172.21.15.43:0/2958673818 wait complete. 2024-05-27T07:14:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.883+0000 7ff1ff4fab80 20 client.15492 trim_cache size 0 max 16384 2024-05-27T07:14:13.875 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:14:13.875 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:14:13.875 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "0dddcec2-9b72-479b-b95a-8ba6d8666c03", 2024-05-27T07:14:13.875 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:14:13.875 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:14:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 -- 172.21.15.43:0/900607816 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1d00b2a80 msgr2=0x7ff1d00b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1d00b2a80 0x7ff1d00b4f90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00b5c0 tx=0x7ff1dc005940 comp rx=0 tx=0).stop 2024-05-27T07:14:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 -- 172.21.15.43:0/900607816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1ec105c50 msgr2=0x7ff1ec1a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1ec105c50 0x7ff1ec1a0c90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800d950 tx=0x7ff1d800dd10 comp rx=0 tx=0).stop 2024-05-27T07:14:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 -- 172.21.15.43:0/900607816 shutdown_connections 2024-05-27T07:14:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1d00b2a80 0x7ff1d00b4f90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1ec10e730 0x7ff1ec19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1ec105300 0x7ff1ec1a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 --2- 172.21.15.43:0/900607816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1ec105c50 0x7ff1ec1a0c90 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 -- 172.21.15.43:0/900607816 >> 172.21.15.43:0/900607816 conn(0x7ff1ec078ab0 msgr2=0x7ff1ec0fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.884+0000 7ff1eb200700 1 -- 172.21.15.43:0/900607816 shutdown_connections 2024-05-27T07:14:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.885+0000 7ff1eb200700 1 -- 172.21.15.43:0/900607816 wait complete. 2024-05-27T07:14:13.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.964+0000 7ff49ea00640 1 -- 172.21.15.43:0/1654877314 <== mgr.35033 v2:172.21.15.43:6800/2368318638 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7ff4a4110f70 con 0x7ff48c0b29e0 2024-05-27T07:14:13.955 INFO:teuthology.orchestra.run.smithi043.stdout:Retention added to path / 2024-05-27T07:14:13.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 -- 172.21.15.43:0/1654877314 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff48c0b29e0 msgr2=0x7ff48c0b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff48c0b29e0 0x7ff48c0b4ef0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff4a419dba0 tx=0x7ff49800b390 comp rx=0 tx=0).stop 2024-05-27T07:14:13.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 -- 172.21.15.43:0/1654877314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4a410daf0 msgr2=0x7ff4a41a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:13.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4a410daf0 0x7ff4a41a2b80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff49400d950 tx=0x7ff49400de10 comp rx=0 tx=0).stop 2024-05-27T07:14:13.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 -- 172.21.15.43:0/1654877314 shutdown_connections 2024-05-27T07:14:13.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff48c0b29e0 0x7ff48c0b4ef0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4a4110450 0x7ff4a419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4a410daf0 0x7ff4a41a2b80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 --2- 172.21.15.43:0/1654877314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4a4076770 0x7ff4a41a2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:13.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 -- 172.21.15.43:0/1654877314 >> 172.21.15.43:0/1654877314 conn(0x7ff4a40ff320 msgr2=0x7ff4a4110860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:13.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 -- 172.21.15.43:0/1654877314 shutdown_connections 2024-05-27T07:14:13.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:13.967+0000 7ff4ac800640 1 -- 172.21.15.43:0/1654877314 wait complete. 2024-05-27T07:14:13.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:13 smithi179 ceph-mon[35888]: pgmap v12: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 0 B/s wr, 18 op/s 2024-05-27T07:14:13.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:13 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2827742192' entity='client.admin' 2024-05-27T07:14:13.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:13 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:14:13.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:13 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:14:13.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:13 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:14:13.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:13 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:14:13.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:13 smithi179 ceph-mon[35888]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:13.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-27T07:14:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.244+0000 7f298be00640 1 -- 172.21.15.43:0/2436839951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f298c10f390 msgr2=0x7f298c116b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.244+0000 7f298be00640 1 --2- 172.21.15.43:0/2436839951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f298c10f390 0x7f298c116b30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2984009ca0 tx=0x7f298402a8c0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.244+0000 7f298be00640 1 -- 172.21.15.43:0/2436839951 shutdown_connections 2024-05-27T07:14:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.245+0000 7f298be00640 1 --2- 172.21.15.43:0/2436839951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f298c10f390 0x7f298c116b30 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.245+0000 7f298be00640 1 --2- 172.21.15.43:0/2436839951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f298c10e8d0 0x7f298c10ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.245+0000 7f298be00640 1 --2- 172.21.15.43:0/2436839951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f298c10a340 0x7f298c10a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.245+0000 7f298be00640 1 -- 172.21.15.43:0/2436839951 >> 172.21.15.43:0/2436839951 conn(0x7f298c076cd0 msgr2=0x7f298c077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.245+0000 7f298be00640 1 -- 172.21.15.43:0/2436839951 shutdown_connections 2024-05-27T07:14:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.245+0000 7f298be00640 1 -- 172.21.15.43:0/2436839951 wait complete. 2024-05-27T07:14:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.246+0000 7f298be00640 1 Processor -- start 2024-05-27T07:14:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.246+0000 7f298be00640 1 -- start start 2024-05-27T07:14:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.246+0000 7f298be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f298c10a340 0x7f298c1a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.246+0000 7f298be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f298c10e8d0 0x7f298c1a6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.246+0000 7f298be00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f298c10f390 0x7f298c1a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.246+0000 7f298be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f298c11a1a0 con 0x7f298c10a340 2024-05-27T07:14:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.246+0000 7f298be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f298c11a020 con 0x7f298c10e8d0 2024-05-27T07:14:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.246+0000 7f298be00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f298c11a320 con 0x7f298c10f390 2024-05-27T07:14:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f298c10a340 0x7f298c1a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f298c10a340 0x7f298c1a6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44934/0 (socket says 172.21.15.43:44934) 2024-05-27T07:14:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298aa00640 1 -- 172.21.15.43:0/2198694117 learned_addr learned my addr 172.21.15.43:0/2198694117 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298a000640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f298c10e8d0 0x7f298c1a6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298b400640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f298c10f390 0x7f298c1a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298aa00640 1 -- 172.21.15.43:0/2198694117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f298c10f390 msgr2=0x7f298c1a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298aa00640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f298c10f390 0x7f298c1a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298aa00640 1 -- 172.21.15.43:0/2198694117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f298c10e8d0 msgr2=0x7f298c1a6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298aa00640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f298c10e8d0 0x7f298c1a6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298aa00640 1 -- 172.21.15.43:0/2198694117 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298c1a1040 con 0x7f298c10a340 2024-05-27T07:14:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298b400640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f298c10f390 0x7f298c1a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:14:14.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.247+0000 7f298aa00640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f298c10a340 0x7f298c1a6790 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f297c00e9f0 tx=0x7f297c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:14.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.248+0000 7f2983400640 1 -- 172.21.15.43:0/2198694117 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c00cd80 con 0x7f298c10a340 2024-05-27T07:14:14.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.248+0000 7f2983400640 1 -- 172.21.15.43:0/2198694117 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f297c0108c0 con 0x7f298c10a340 2024-05-27T07:14:14.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.248+0000 7f2983400640 1 -- 172.21.15.43:0/2198694117 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c0187c0 con 0x7f298c10a340 2024-05-27T07:14:14.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.248+0000 7f298be00640 1 -- 172.21.15.43:0/2198694117 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f298c1a1330 con 0x7f298c10a340 2024-05-27T07:14:14.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.248+0000 7f298be00640 1 -- 172.21.15.43:0/2198694117 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f298c1ad610 con 0x7f298c10a340 2024-05-27T07:14:14.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.249+0000 7f298be00640 1 -- 172.21.15.43:0/2198694117 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f298c06a230 con 0x7f298c10a340 2024-05-27T07:14:14.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.252+0000 7f2983400640 1 -- 172.21.15.43:0/2198694117 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f297c010430 con 0x7f298c10a340 2024-05-27T07:14:14.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.253+0000 7f2983400640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29700b29e0 0x7f29700b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.253+0000 7f2983400640 1 -- 172.21.15.43:0/2198694117 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f297c0e3500 con 0x7f298c10a340 2024-05-27T07:14:14.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.254+0000 7f298a000640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29700b29e0 0x7f29700b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.254+0000 7f298a000640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29700b29e0 0x7f29700b4ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2978005fd0 tx=0x7f297800b510 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:14.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.255+0000 7f2983400640 1 -- 172.21.15.43:0/2198694117 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f297c0aa9c0 con 0x7f298c10a340 2024-05-27T07:14:14.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.515+0000 7f298be00640 1 -- 172.21.15.43:0/2198694117 --> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f298c063100 con 0x7f29700b29e0 2024-05-27T07:14:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.533+0000 7f2983400640 1 -- 172.21.15.43:0/2198694117 <== mgr.35033 v2:172.21.15.43:6800/2368318638 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f298c063100 con 0x7f29700b29e0 2024-05-27T07:14:14.524 INFO:teuthology.orchestra.run.smithi043.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-27T00:00:00", "created": "2024-05-27T07:14:13", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-27T07:14:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 -- 172.21.15.43:0/2198694117 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29700b29e0 msgr2=0x7f29700b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29700b29e0 0x7f29700b4ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2978005fd0 tx=0x7f297800b510 comp rx=0 tx=0).stop 2024-05-27T07:14:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 -- 172.21.15.43:0/2198694117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f298c10a340 msgr2=0x7f298c1a6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f298c10a340 0x7f298c1a6790 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f297c00e9f0 tx=0x7f297c00eeb0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 -- 172.21.15.43:0/2198694117 shutdown_connections 2024-05-27T07:14:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29700b29e0 0x7f29700b4ef0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f298c10e8d0 0x7f298c1a6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f298c10f390 0x7f298c1a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 --2- 172.21.15.43:0/2198694117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f298c10a340 0x7f298c1a6790 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 -- 172.21.15.43:0/2198694117 >> 172.21.15.43:0/2198694117 conn(0x7f298c076cd0 msgr2=0x7f298c107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 -- 172.21.15.43:0/2198694117 shutdown_connections 2024-05-27T07:14:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.536+0000 7f298be00640 1 -- 172.21.15.43:0/2198694117 wait complete. 2024-05-27T07:14:14.550 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:14 smithi043 ceph-mon[31531]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:14.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-27T07:14:14.778 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:14 smithi132 ceph-mon[35283]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:14.839 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:14:14.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.849+0000 7f1dcbe00640 1 -- 172.21.15.43:0/2617098610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1dc410a500 msgr2=0x7f1dc410a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.849+0000 7f1dcbe00640 1 --2- 172.21.15.43:0/2617098610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1dc410a500 0x7f1dc410a910 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1db8009900 tx=0x7f1db802a3f0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.849+0000 7f1dcbe00640 1 -- 172.21.15.43:0/2617098610 shutdown_connections 2024-05-27T07:14:14.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.849+0000 7f1dcbe00640 1 --2- 172.21.15.43:0/2617098610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dc410ea80 0x7f1dc410eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.849+0000 7f1dcbe00640 1 --2- 172.21.15.43:0/2617098610 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dc410f540 0x7f1dc4116ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.849+0000 7f1dcbe00640 1 --2- 172.21.15.43:0/2617098610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1dc410a500 0x7f1dc410a910 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.849+0000 7f1dcbe00640 1 -- 172.21.15.43:0/2617098610 >> 172.21.15.43:0/2617098610 conn(0x7f1dc4077190 msgr2=0x7f1dc40775d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:14.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.849+0000 7f1dcbe00640 1 -- 172.21.15.43:0/2617098610 shutdown_connections 2024-05-27T07:14:14.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.849+0000 7f1dcbe00640 1 -- 172.21.15.43:0/2617098610 wait complete. 2024-05-27T07:14:14.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.850+0000 7f1dcbe00640 1 Processor -- start 2024-05-27T07:14:14.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.851+0000 7f1dcbe00640 1 -- start start 2024-05-27T07:14:14.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.851+0000 7f1dcbe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dc410a500 0x7f1dc41a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.851+0000 7f1dcbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1dc410ea80 0x7f1dc41a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.851+0000 7f1dcbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dc410f540 0x7f1dc41a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.851+0000 7f1dcbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc411a370 con 0x7f1dc410ea80 2024-05-27T07:14:14.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.851+0000 7f1dcbe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc411a1f0 con 0x7f1dc410f540 2024-05-27T07:14:14.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.851+0000 7f1dcbe00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc411a4f0 con 0x7f1dc410a500 2024-05-27T07:14:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.851+0000 7f1dc8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1dc410ea80 0x7f1dc41a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.851+0000 7f1dc8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1dc410ea80 0x7f1dc41a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55644/0 (socket says 172.21.15.43:55644) 2024-05-27T07:14:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.851+0000 7f1dc8a00640 1 -- 172.21.15.43:0/4062162200 learned_addr learned my addr 172.21.15.43:0/4062162200 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.852+0000 7f1dc9400640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dc410a500 0x7f1dc41a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.852+0000 7f1dc9e00640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dc410f540 0x7f1dc41a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.852+0000 7f1dc8a00640 1 -- 172.21.15.43:0/4062162200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dc410a500 msgr2=0x7f1dc41a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.852+0000 7f1dc8a00640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dc410a500 0x7f1dc41a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.852+0000 7f1dc8a00640 1 -- 172.21.15.43:0/4062162200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dc410f540 msgr2=0x7f1dc41a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.852+0000 7f1dc8a00640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dc410f540 0x7f1dc41a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.852+0000 7f1dc8a00640 1 -- 172.21.15.43:0/4062162200 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc41a5c10 con 0x7f1dc410ea80 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.852+0000 7f1dc8a00640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1dc410ea80 0x7f1dc41a1310 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1db000d770 tx=0x7f1db000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.853+0000 7f1dc2000640 1 -- 172.21.15.43:0/4062162200 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db0017070 con 0x7f1dc410ea80 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.853+0000 7f1dc2000640 1 -- 172.21.15.43:0/4062162200 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1db0009c30 con 0x7f1dc410ea80 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.853+0000 7f1dc2000640 1 -- 172.21.15.43:0/4062162200 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db000f620 con 0x7f1dc410ea80 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.853+0000 7f1dcbe00640 1 -- 172.21.15.43:0/4062162200 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc41a5f00 con 0x7f1dc410ea80 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.853+0000 7f1dcbe00640 1 -- 172.21.15.43:0/4062162200 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc41a6400 con 0x7f1dc410ea80 2024-05-27T07:14:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.854+0000 7f1dcbe00640 1 -- 172.21.15.43:0/4062162200 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc4050860 con 0x7f1dc410ea80 2024-05-27T07:14:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.857+0000 7f1dc2000640 1 -- 172.21.15.43:0/4062162200 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f1db001a070 con 0x7f1dc410ea80 2024-05-27T07:14:14.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.859+0000 7f1dc2000640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1dac0b2a60 0x7f1dac0b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:14.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.859+0000 7f1dc2000640 1 -- 172.21.15.43:0/4062162200 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f1db00e2680 con 0x7f1dc410ea80 2024-05-27T07:14:14.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.859+0000 7f1dc9400640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1dac0b2a60 0x7f1dac0b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:14.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.860+0000 7f1dc9400640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1dac0b2a60 0x7f1dac0b4f70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1db8006010 tx=0x7f1db8000bc0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:14.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:14.863+0000 7f1dc2000640 1 -- 172.21.15.43:0/4062162200 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1dc41a6400 con 0x7f1dc410ea80 2024-05-27T07:14:14.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:14 smithi179 ceph-mon[35888]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:15.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.103+0000 7f1dcbe00640 1 -- 172.21.15.43:0/4062162200 --> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f1dc407ab50 con 0x7f1dac0b2a60 2024-05-27T07:14:15.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.117+0000 7f1dc2000640 1 -- 172.21.15.43:0/4062162200 <== mgr.35033 v2:172.21.15.43:6800/2368318638 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f1dc407ab50 con 0x7f1dac0b2a60 2024-05-27T07:14:15.108 INFO:teuthology.orchestra.run.smithi043.stdout:/ 1m 6m3h 2024-05-27T07:14:15.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 -- 172.21.15.43:0/4062162200 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1dac0b2a60 msgr2=0x7f1dac0b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:15.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1dac0b2a60 0x7f1dac0b4f70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1db8006010 tx=0x7f1db8000bc0 comp rx=0 tx=0).stop 2024-05-27T07:14:15.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 -- 172.21.15.43:0/4062162200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1dc410ea80 msgr2=0x7f1dc41a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:15.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1dc410ea80 0x7f1dc41a1310 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1db000d770 tx=0x7f1db000dc30 comp rx=0 tx=0).stop 2024-05-27T07:14:15.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 -- 172.21.15.43:0/4062162200 shutdown_connections 2024-05-27T07:14:15.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1dac0b2a60 0x7f1dac0b4f70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:15.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dc410f540 0x7f1dc41a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:15.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1dc410ea80 0x7f1dc41a1310 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:15.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 --2- 172.21.15.43:0/4062162200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dc410a500 0x7f1dc41a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:15.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 -- 172.21.15.43:0/4062162200 >> 172.21.15.43:0/4062162200 conn(0x7f1dc4077190 msgr2=0x7f1dc407a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:15.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 -- 172.21.15.43:0/4062162200 shutdown_connections 2024-05-27T07:14:15.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:15.120+0000 7f1dcbe00640 1 -- 172.21.15.43:0/4062162200 wait complete. 2024-05-27T07:14:15.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-27T07:14:15.259 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-27T07:14:15.271 INFO:teuthology.run_tasks:Running task workunit... 2024-05-27T07:14:15.314 INFO:tasks.workunit:Pulling workunits from ref 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:14:15.314 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-27T07:14:15.314 INFO:tasks.workunit:timeout=3h 2024-05-27T07:14:15.314 INFO:tasks.workunit:cleanup=True 2024-05-27T07:14:15.315 DEBUG:teuthology.orchestra.run.smithi043:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-27T07:14:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:15.357 INFO:teuthology.orchestra.run.smithi043.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-27T07:14:15.358 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-05-27T07:14:15.358 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-05-27T07:14:15.358 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T07:14:15.358 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:fusefs_t:s0 2024-05-27T07:14:15.358 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-05-27 07:10:06.577259597 +0000 2024-05-27T07:14:15.358 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-05-27 07:10:06.577259597 +0000 2024-05-27T07:14:15.358 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-05-27 07:13:52.425286592 +0000 2024-05-27T07:14:15.358 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2024-05-27T07:14:15.358 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-27T07:14:15.359 DEBUG:teuthology.orchestra.run.smithi043:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-27T07:14:15.506 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T07:14:15.567 INFO:tasks.workunit.client.0.smithi043.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-27T07:14:15.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:15 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:14:15.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:15 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:14:15.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:15 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:14:15.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:15 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:14:15.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:15 smithi043 ceph-mon[31531]: pgmap v13: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:15.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:15 smithi043 ceph-mon[31531]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:15.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:15 smithi043 ceph-mon[31531]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:15.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:15 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:14:15.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:15 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:14:15.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:15 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:14:15.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:15 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:14:15.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:15 smithi132 ceph-mon[35283]: pgmap v13: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:15.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:15 smithi132 ceph-mon[35283]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:15.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:15 smithi132 ceph-mon[35283]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:15 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:14:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:15 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:14:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:15 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:14:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:15 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:14:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:15 smithi179 ceph-mon[35888]: pgmap v13: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:15.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:15 smithi179 ceph-mon[35888]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:15 smithi179 ceph-mon[35888]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:14:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.693+0000 7fbd84a00700 1 -- 172.21.15.43:0/1388926484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80110b70 msgr2=0x7fbd80112fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.693+0000 7fbd84a00700 1 --2- 172.21.15.43:0/1388926484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80110b70 0x7fbd80112fd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800b0a0 tx=0x7fbd7800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.693+0000 7fbd84a00700 1 -- 172.21.15.43:0/1388926484 shutdown_connections 2024-05-27T07:14:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.693+0000 7fbd84a00700 1 --2- 172.21.15.43:0/1388926484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80110b70 0x7fbd80112fd0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.693+0000 7fbd84a00700 1 --2- 172.21.15.43:0/1388926484 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd80076e30 0x7fbd80077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.693+0000 7fbd84a00700 1 --2- 172.21.15.43:0/1388926484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd80077780 0x7fbd80110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.693+0000 7fbd84a00700 1 -- 172.21.15.43:0/1388926484 >> 172.21.15.43:0/1388926484 conn(0x7fbd800ffe90 msgr2=0x7fbd801022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.693+0000 7fbd84a00700 1 -- 172.21.15.43:0/1388926484 shutdown_connections 2024-05-27T07:14:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.693+0000 7fbd84a00700 1 -- 172.21.15.43:0/1388926484 wait complete. 2024-05-27T07:14:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.695+0000 7fbd84a00700 1 Processor -- start 2024-05-27T07:14:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.695+0000 7fbd84a00700 1 -- start start 2024-05-27T07:14:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.695+0000 7fbd84a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd80076e30 0x7fbd8019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.695+0000 7fbd84a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80077780 0x7fbd8019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.695+0000 7fbd84a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd80110b70 0x7fbd801a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.695+0000 7fbd84a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbd80116550 con 0x7fbd80077780 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.695+0000 7fbd84a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbd801163d0 con 0x7fbd80076e30 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.695+0000 7fbd84a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbd801166d0 con 0x7fbd80110b70 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.696+0000 7fbd7d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80077780 0x7fbd8019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.696+0000 7fbd7d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80077780 0x7fbd8019d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55662/0 (socket says 172.21.15.43:55662) 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.696+0000 7fbd7d400700 1 -- 172.21.15.43:0/1867366212 learned_addr learned my addr 172.21.15.43:0/1867366212 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.696+0000 7fbd7de00700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd80076e30 0x7fbd8019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.696+0000 7fbd7e800700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd80110b70 0x7fbd801a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.696+0000 7fbd7d400700 1 -- 172.21.15.43:0/1867366212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd80110b70 msgr2=0x7fbd801a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.696+0000 7fbd7d400700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd80110b70 0x7fbd801a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.696+0000 7fbd7d400700 1 -- 172.21.15.43:0/1867366212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd80076e30 msgr2=0x7fbd8019ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.696+0000 7fbd7d400700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd80076e30 0x7fbd8019ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.696+0000 7fbd7d400700 1 -- 172.21.15.43:0/1867366212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd801a1ca0 con 0x7fbd80077780 2024-05-27T07:14:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.697+0000 7fbd7d400700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80077780 0x7fbd8019d420 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00c9b0 tx=0x7fbd6c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.697+0000 7fbd76a00700 1 -- 172.21.15.43:0/1867366212 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c007b00 con 0x7fbd80077780 2024-05-27T07:14:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.697+0000 7fbd76a00700 1 -- 172.21.15.43:0/1867366212 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd6c007ca0 con 0x7fbd80077780 2024-05-27T07:14:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.697+0000 7fbd76a00700 1 -- 172.21.15.43:0/1867366212 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c018660 con 0x7fbd80077780 2024-05-27T07:14:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.697+0000 7fbd84a00700 1 -- 172.21.15.43:0/1867366212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd801a1f90 con 0x7fbd80077780 2024-05-27T07:14:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.697+0000 7fbd84a00700 1 -- 172.21.15.43:0/1867366212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd801a2480 con 0x7fbd80077780 2024-05-27T07:14:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.702+0000 7fbd76a00700 1 -- 172.21.15.43:0/1867366212 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7fbd6c00f450 con 0x7fbd80077780 2024-05-27T07:14:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.702+0000 7fbd9362fb80 1 Processor -- start 2024-05-27T07:14:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.703+0000 7fbd9362fb80 1 -- start start 2024-05-27T07:14:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.703+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 0x558e2ef6f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.703+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2ef69870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.703+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 0x558e2ef6a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.703+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558e2ee97e40 con 0x558e2ef6fd30 2024-05-27T07:14:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.703+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558e2ee879e0 con 0x558e2edeae40 2024-05-27T07:14:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.703+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x558e2ee8ba20 con 0x558e2ef69db0 2024-05-27T07:14:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.704+0000 7fbd7de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2ef69870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.704+0000 7fbd7de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2ef69870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55676/0 (socket says 172.21.15.43:55676) 2024-05-27T07:14:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.704+0000 7fbd7de00700 1 -- 172.21.15.43:0/3071514392 learned_addr learned my addr 172.21.15.43:0/3071514392 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.704+0000 7fbd7e800700 1 --2- 172.21.15.43:0/3071514392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 0x558e2ef6f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.704+0000 7fbd7d400700 1 --2- 172.21.15.43:0/3071514392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 0x558e2ef6a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.704+0000 7fbd7de00700 1 -- 172.21.15.43:0/3071514392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 msgr2=0x558e2ef6a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.704+0000 7fbd7de00700 1 --2- 172.21.15.43:0/3071514392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 0x558e2ef6a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.704+0000 7fbd7de00700 1 -- 172.21.15.43:0/3071514392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 msgr2=0x558e2ef6f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.704+0000 7fbd7de00700 1 --2- 172.21.15.43:0/3071514392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 0x558e2ef6f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.704+0000 7fbd7de00700 1 -- 172.21.15.43:0/3071514392 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2ee50420 con 0x558e2ef6fd30 2024-05-27T07:14:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.705+0000 7fbd76a00700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbd640b2a80 0x7fbd640b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.705+0000 7fbd76a00700 1 -- 172.21.15.43:0/1867366212 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fbd6c0e2bb0 con 0x7fbd80077780 2024-05-27T07:14:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.705+0000 7fbd7de00700 1 --2- 172.21.15.43:0/3071514392 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2ef69870 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbd70008c60 tx=0x7fbd70008f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c3d65ea94ff723 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.705+0000 7fbd7e800700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbd640b2a80 0x7fbd640b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.705+0000 7fbd6b400700 1 -- 172.21.15.43:0/3071514392 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70014070 con 0x558e2ef6fd30 2024-05-27T07:14:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.705+0000 7fbd6b400700 1 -- 172.21.15.43:0/3071514392 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd7000ed70 con 0x558e2ef6fd30 2024-05-27T07:14:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.705+0000 7fbd6b400700 1 -- 172.21.15.43:0/3071514392 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70010430 con 0x558e2ef6fd30 2024-05-27T07:14:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.706+0000 7fbd7e800700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbd640b2a80 0x7fbd640b4f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x558e2ef6ab30 tx=0x7fbd78009b80 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.706+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3071514392 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 msgr2=0x558e2ef69870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.706+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3071514392 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2ef69870 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbd70008c60 tx=0x7fbd70008f70 comp rx=0 tx=0).stop 2024-05-27T07:14:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.707+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3071514392 shutdown_connections 2024-05-27T07:14:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.707+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3071514392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 0x558e2ef6a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.707+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3071514392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 0x558e2ef6f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.707+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3071514392 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2ef69870 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.707+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3071514392 >> 172.21.15.43:0/3071514392 conn(0x558e2ee4afc0 msgr2=0x558e2ef648d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.707+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3071514392 shutdown_connections 2024-05-27T07:14:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.707+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3071514392 wait complete. 2024-05-27T07:14:17.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.707+0000 7fbd9362fb80 1 -- start start 2024-05-27T07:14:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.708+0000 7fbd9362fb80 1 Processor -- start 2024-05-27T07:14:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.708+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2edeae40 0x558e2ef73580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.708+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2ef69db0 0x558e2ef73ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.708+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef6fd30 0x558e2ef74000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.708+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558e2ed8dfa0 con 0x558e2edeae40 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.708+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558e2ee28760 con 0x558e2ef69db0 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.708+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x558e2ef76800 con 0x558e2ef6fd30 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd9362fb80 10 client.0 _ll_register_callbacks cb 0x558e2ee31c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2edeae40 0x558e2ef73580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2edeae40 0x558e2ef73580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55684/0 (socket says 172.21.15.43:55684) 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7de00700 1 -- 172.21.15.43:0/3086189522 learned_addr learned my addr 172.21.15.43:0/3086189522 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7d400700 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2ef69db0 0x558e2ef73ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7e800700 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef6fd30 0x558e2ef74000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7de00700 1 -- 172.21.15.43:0/3086189522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef6fd30 msgr2=0x558e2ef74000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7de00700 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef6fd30 0x558e2ef74000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7de00700 1 -- 172.21.15.43:0/3086189522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2ef69db0 msgr2=0x558e2ef73ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7de00700 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2ef69db0 0x558e2ef73ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7de00700 1 -- 172.21.15.43:0/3086189522 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e2edd78d0 con 0x558e2edeae40 2024-05-27T07:14:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.709+0000 7fbd7e800700 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef6fd30 0x558e2ef74000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:14:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.710+0000 7fbd7de00700 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2edeae40 0x558e2ef73580 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbd70008c30 tx=0x7fbd70010710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7bbb46429a9a51 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.710+0000 7fbd5fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:14:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.710+0000 7fbd5fe00700 1 -- 172.21.15.43:0/3086189522 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70014070 con 0x558e2edeae40 2024-05-27T07:14:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.710+0000 7fbd5fe00700 1 -- 172.21.15.43:0/3086189522 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fbd70010d40 con 0x558e2edeae40 2024-05-27T07:14:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.710+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3086189522 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd7000ed70 con 0x558e2edeae40 2024-05-27T07:14:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.710+0000 7fbd5fe00700 1 -- 172.21.15.43:0/3086189522 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbd7002d050 con 0x558e2edeae40 2024-05-27T07:14:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.710+0000 7fbd9362fb80 10 client.15516 fetch_fsmap learned FSMap version 30 2024-05-27T07:14:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.710+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3086189522 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558e2ee58060 con 0x558e2edeae40 2024-05-27T07:14:17.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.711+0000 7fbd5fe00700 1 -- 172.21.15.43:0/3086189522 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd7000ed70 con 0x558e2edeae40 2024-05-27T07:14:17.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.711+0000 7fbd5fe00700 1 -- 172.21.15.43:0/3086189522 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fbd700039d0 con 0x558e2edeae40 2024-05-27T07:14:17.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.711+0000 7fbd9362fb80 10 client.15516 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:14:17.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.711+0000 7fbd9362fb80 10 client.15516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:14:17.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.711+0000 7fbd9362fb80 20 client.15516 populate_metadata read hostname 'smithi043' 2024-05-27T07:14:17.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.711+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558e2ef836b0 0x558e2ef85ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.711+0000 7fbd9362fb80 4 client.15516 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:14:17.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.711+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3086189522 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e2ee99d90 con 0x558e2ef836b0 2024-05-27T07:14:17.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.712+0000 7fbd7d400700 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558e2ef836b0 0x558e2ef85ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.713+0000 7fbd7d400700 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558e2ef836b0 0x558e2ef85ad0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=434cf0ee5388ba6d server_cookie=482140695d93a9ef in_seq=0 out_seq=0 2024-05-27T07:14:17.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.713+0000 7fbd5fe00700 10 client.15516 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:17.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.715+0000 7fbd5fe00700 1 -- 172.21.15.43:0/3086189522 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558e2ee99d90 con 0x558e2ef836b0 2024-05-27T07:14:17.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.715+0000 7fbd5fe00700 10 client.15516 handle_command_reply: tid=0 2024-05-27T07:14:17.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.715+0000 7fbd9362fb80 1 client.15516 shutdown 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.715+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3086189522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2edeae40 msgr2=0x558e2ef73580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.715+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2edeae40 0x558e2ef73580 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbd70008c30 tx=0x7fbd70010710 comp rx=0 tx=0).stop 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.715+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3086189522 shutdown_connections 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.716+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef6fd30 0x558e2ef74000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.716+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558e2ef836b0 0x558e2ef85ad0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.716+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2edeae40 0x558e2ef73580 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.716+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3086189522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2ef69db0 0x558e2ef73ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.716+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3086189522 >> 172.21.15.43:0/3086189522 conn(0x558e2ee4afc0 msgr2=0x558e2ef64580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.716+0000 7fbd5fe00700 0 client.15516 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.716+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3086189522 shutdown_connections 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.716+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3086189522 wait complete. 2024-05-27T07:14:17.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.716+0000 7fbd9362fb80 20 client.15516 trim_cache size 0 max 16384 2024-05-27T07:14:17.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:17 smithi179 ceph-mon[35888]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-05-27T07:14:17.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.738+0000 7fbd9362fb80 1 Processor -- start 2024-05-27T07:14:17.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.739+0000 7fbd9362fb80 1 -- start start 2024-05-27T07:14:17.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.739+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2edeae40 0x558e2ef69870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.739+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef69db0 0x558e2ef71150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.739+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2ef6fd30 0x558e2ef6db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.739+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbd801b25b0 con 0x558e2ef69db0 2024-05-27T07:14:17.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.739+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbd801b40f0 con 0x558e2ef6fd30 2024-05-27T07:14:17.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.739+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbd801b2c80 con 0x558e2edeae40 2024-05-27T07:14:17.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.739+0000 7fbd7d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef69db0 0x558e2ef71150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.740+0000 7fbd7d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef69db0 0x558e2ef71150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55698/0 (socket says 172.21.15.43:55698) 2024-05-27T07:14:17.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.740+0000 7fbd7d400700 1 -- 172.21.15.43:0/3467187395 learned_addr learned my addr 172.21.15.43:0/3467187395 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:17.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.740+0000 7fbd7de00700 1 --2- 172.21.15.43:0/3467187395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2edeae40 0x558e2ef69870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.740+0000 7fbd7e800700 1 --2- 172.21.15.43:0/3467187395 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2ef6fd30 0x558e2ef6db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.740+0000 7fbd7d400700 1 -- 172.21.15.43:0/3467187395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2edeae40 msgr2=0x558e2ef69870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:17.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.740+0000 7fbd7d400700 1 --2- 172.21.15.43:0/3467187395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2edeae40 0x558e2ef69870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.740+0000 7fbd7d400700 1 -- 172.21.15.43:0/3467187395 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2ef6fd30 msgr2=0x558e2ef6db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:17.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.740+0000 7fbd7d400700 1 --2- 172.21.15.43:0/3467187395 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2ef6fd30 0x558e2ef6db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.740+0000 7fbd7d400700 1 -- 172.21.15.43:0/3467187395 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2ef6a1c0 con 0x558e2ef69db0 2024-05-27T07:14:17.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.741+0000 7fbd7d400700 1 --2- 172.21.15.43:0/3467187395 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef69db0 0x558e2ef71150 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00cfd0 tx=0x7fbd6c0cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98393e294b5d9c2f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:17.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.741+0000 7fbd6aa00700 1 -- 172.21.15.43:0/3467187395 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c08ae60 con 0x558e2ef69db0 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.741+0000 7fbd6aa00700 1 -- 172.21.15.43:0/3467187395 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd6c0b2110 con 0x558e2ef69db0 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.741+0000 7fbd6aa00700 1 -- 172.21.15.43:0/3467187395 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c00f970 con 0x558e2ef69db0 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.742+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3467187395 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef69db0 msgr2=0x558e2ef71150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.742+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3467187395 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef69db0 0x558e2ef71150 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00cfd0 tx=0x7fbd6c0cc040 comp rx=0 tx=0).stop 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.742+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3467187395 shutdown_connections 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.742+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3467187395 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2ef6fd30 0x558e2ef6db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.742+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3467187395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2edeae40 0x558e2ef69870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.742+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/3467187395 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef69db0 0x558e2ef71150 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.742+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3467187395 >> 172.21.15.43:0/3467187395 conn(0x558e2ee4afc0 msgr2=0x558e2ef73360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.742+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3467187395 shutdown_connections 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.742+0000 7fbd9362fb80 1 -- 172.21.15.43:0/3467187395 wait complete. 2024-05-27T07:14:17.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.742+0000 7fbd9362fb80 1 -- start start 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.743+0000 7fbd9362fb80 1 Processor -- start 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.743+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 0x558e2ef85cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.743+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 0x558e2ef86200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.743+0000 7fbd9362fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2efebf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.743+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbd801b3a20 con 0x558e2ef6fd30 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.743+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbd801b3350 con 0x558e2edeae40 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.743+0000 7fbd9362fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbd801b0530 con 0x558e2ef69db0 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.743+0000 7fbd9362fb80 10 client.0 _ll_register_callbacks cb 0x558e2ee31c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2efebf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2efebf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55704/0 (socket says 172.21.15.43:55704) 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7e800700 1 -- 172.21.15.43:0/329595179 learned_addr learned my addr 172.21.15.43:0/329595179 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7d400700 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 0x558e2ef86200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7de00700 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 0x558e2ef85cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7e800700 1 -- 172.21.15.43:0/329595179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 msgr2=0x558e2ef86200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7e800700 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 0x558e2ef86200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7e800700 1 -- 172.21.15.43:0/329595179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 msgr2=0x558e2ef85cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7e800700 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 0x558e2ef85cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7e800700 1 -- 172.21.15.43:0/329595179 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e2efec450 con 0x558e2ef6fd30 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7de00700 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 0x558e2ef85cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.744+0000 7fbd7d400700 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 0x558e2ef86200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.745+0000 7fbd7e800700 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2efebf10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800b0a0 tx=0x7fbd780099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c4b0485311cc6d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.745+0000 7fbd5fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.745+0000 7fbd5fe00700 1 -- 172.21.15.43:0/329595179 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd78005240 con 0x558e2ef6fd30 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.745+0000 7fbd9362fb80 1 -- 172.21.15.43:0/329595179 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558e2ee93570 con 0x558e2ef6fd30 2024-05-27T07:14:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.745+0000 7fbd5fe00700 1 -- 172.21.15.43:0/329595179 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fbd780053e0 con 0x558e2ef6fd30 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.745+0000 7fbd5fe00700 1 -- 172.21.15.43:0/329595179 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbd7803c050 con 0x558e2ef6fd30 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.746+0000 7fbd9362fb80 10 client.15522 fetch_fsmap learned FSMap version 30 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.746+0000 7fbd9362fb80 1 -- 172.21.15.43:0/329595179 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558e2efed610 con 0x558e2ef6fd30 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.746+0000 7fbd5fe00700 1 -- 172.21.15.43:0/329595179 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x558e2ee93570 con 0x558e2ef6fd30 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.746+0000 7fbd5fe00700 1 -- 172.21.15.43:0/329595179 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fbd78011070 con 0x558e2ef6fd30 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.746+0000 7fbd9362fb80 10 client.15522 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.746+0000 7fbd9362fb80 10 client.15522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.746+0000 7fbd9362fb80 20 client.15522 populate_metadata read hostname 'smithi043' 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.746+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558e2efedd20 0x558e2eff0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.746+0000 7fbd9362fb80 4 client.15522 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.746+0000 7fbd9362fb80 1 -- 172.21.15.43:0/329595179 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558e2ed0a250 con 0x558e2efedd20 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.747+0000 7fbd7de00700 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558e2efedd20 0x558e2eff0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.748+0000 7fbd7de00700 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558e2efedd20 0x558e2eff0140 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff4eb2bd923abcf1 server_cookie=1e74eff3a497996d in_seq=0 out_seq=0 2024-05-27T07:14:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.748+0000 7fbd5fe00700 10 client.15522 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.748+0000 7fbd5fe00700 1 -- 172.21.15.43:0/329595179 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558e2ed0a250 con 0x558e2efedd20 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.748+0000 7fbd5fe00700 10 client.15522 handle_command_reply: tid=0 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.748+0000 7fbd9362fb80 1 client.15522 shutdown 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 1 -- 172.21.15.43:0/329595179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 msgr2=0x558e2efebf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2efebf10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800b0a0 tx=0x7fbd780099e0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 1 -- 172.21.15.43:0/329595179 shutdown_connections 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558e2efedd20 0x558e2eff0140 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558e2ef6fd30 0x558e2efebf10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558e2edeae40 0x558e2ef85cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 1 --2- 172.21.15.43:0/329595179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558e2ef69db0 0x558e2ef86200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd5fe00700 0 client.15522 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 1 -- 172.21.15.43:0/329595179 >> 172.21.15.43:0/329595179 conn(0x558e2ee4afc0 msgr2=0x558e2ef73d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 1 -- 172.21.15.43:0/329595179 shutdown_connections 2024-05-27T07:14:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 1 -- 172.21.15.43:0/329595179 wait complete. 2024-05-27T07:14:17.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.749+0000 7fbd9362fb80 20 client.15522 trim_cache size 0 max 16384 2024-05-27T07:14:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:14:17.743 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:14:17.743 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:14:17.743 INFO:teuthology.orchestra.run.smithi043.stdout: "0dddcec2-9b72-479b-b95a-8ba6d8666c03": { 2024-05-27T07:14:17.743 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:14:17.743 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "0dddcec2-9b72-479b-b95a-8ba6d8666c03", 2024-05-27T07:14:17.744 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:14:17.744 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:14:17.744 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:14:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:14:17.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 -- 172.21.15.43:0/1867366212 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbd640b2a80 msgr2=0x7fbd640b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbd640b2a80 0x7fbd640b4f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x558e2ef6ab30 tx=0x7fbd78009b80 comp rx=0 tx=0).stop 2024-05-27T07:14:17.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 -- 172.21.15.43:0/1867366212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80077780 msgr2=0x7fbd8019d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:17.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80077780 0x7fbd8019d420 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00c9b0 tx=0x7fbd6c00cd70 comp rx=0 tx=0).stop 2024-05-27T07:14:17.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 -- 172.21.15.43:0/1867366212 shutdown_connections 2024-05-27T07:14:17.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbd640b2a80 0x7fbd640b4f90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd80110b70 0x7fbd801a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbd80076e30 0x7fbd8019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 --2- 172.21.15.43:0/1867366212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd80077780 0x7fbd8019d420 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:17.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 -- 172.21.15.43:0/1867366212 >> 172.21.15.43:0/1867366212 conn(0x7fbd800ffe90 msgr2=0x7fbd80100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:17.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 -- 172.21.15.43:0/1867366212 shutdown_connections 2024-05-27T07:14:17.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:17.750+0000 7fbd84a00700 1 -- 172.21.15.43:0/1867366212 wait complete. 2024-05-27T07:14:17.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:17 smithi043 ceph-mon[31531]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-05-27T07:14:17.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:17 smithi132 ceph-mon[35283]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-05-27T07:14:18.667 INFO:tasks.cephfs.filesystem:scrub status for tag:0dddcec2-9b72-479b-b95a-8ba6d8666c03 - {'path': '/', 'tag': '0dddcec2-9b72-479b-b95a-8ba6d8666c03', 'options': 'recursive,force'} 2024-05-27T07:14:19.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:19 smithi132 ceph-mon[35283]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-05-27T07:14:19.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:19 smithi043 ceph-mon[31531]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-05-27T07:14:19.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:19 smithi179 ceph-mon[35888]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-05-27T07:14:21.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:21 smithi132 ceph-mon[35283]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 41 op/s 2024-05-27T07:14:21.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:21 smithi043 ceph-mon[31531]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 41 op/s 2024-05-27T07:14:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:21 smithi179 ceph-mon[35888]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 41 op/s 2024-05-27T07:14:23.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:23 smithi132 ceph-mon[35283]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:14:23.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:23 smithi043 ceph-mon[31531]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:14:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:23 smithi179 ceph-mon[35888]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:14:25.826 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:25 smithi132 ceph-mon[35283]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-27T07:14:25.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:25 smithi043 ceph-mon[31531]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-27T07:14:25.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:25 smithi179 ceph-mon[35888]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-05-27T07:14:26.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:26 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:14:26.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:26 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:14:26.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:26 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:14:27.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:27 smithi132 ceph-mon[35283]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:14:27.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:27 smithi043 ceph-mon[31531]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:14:27.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:27 smithi179 ceph-mon[35888]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-05-27T07:14:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:29 smithi132 ceph-mon[35283]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-05-27T07:14:29.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:29 smithi043 ceph-mon[31531]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-05-27T07:14:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:29 smithi179 ceph-mon[35888]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-05-27T07:14:31.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:31 smithi043 ceph-mon[31531]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-27T07:14:31.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:31 smithi179 ceph-mon[35888]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-27T07:14:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:31 smithi132 ceph-mon[35283]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-27T07:14:33.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:33 smithi043 ceph-mon[31531]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-27T07:14:33.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:33 smithi179 ceph-mon[35888]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-27T07:14:34.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:33 smithi132 ceph-mon[35283]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-05-27T07:14:35.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:35 smithi043 ceph-mon[31531]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 8 op/s 2024-05-27T07:14:35.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:35 smithi179 ceph-mon[35888]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 8 op/s 2024-05-27T07:14:36.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:35 smithi132 ceph-mon[35283]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 8 op/s 2024-05-27T07:14:37.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:37 smithi043 ceph-mon[31531]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 596 B/s wr, 12 op/s 2024-05-27T07:14:37.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:37 smithi179 ceph-mon[35888]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 596 B/s wr, 12 op/s 2024-05-27T07:14:38.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:37 smithi132 ceph-mon[35283]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 596 B/s wr, 12 op/s 2024-05-27T07:14:39.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:39 smithi179 ceph-mon[35888]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:39.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:39 smithi043 ceph-mon[31531]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:39 smithi132 ceph-mon[35283]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:41.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:41 smithi043 ceph-mon[31531]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-05-27T07:14:41.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:41 smithi179 ceph-mon[35888]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-05-27T07:14:42.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:41 smithi132 ceph-mon[35283]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-05-27T07:14:43.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:43 smithi043 ceph-mon[31531]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-27T07:14:43.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:43 smithi179 ceph-mon[35888]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-27T07:14:44.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:43 smithi132 ceph-mon[35283]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-27T07:14:45.893 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:45 smithi132 ceph-mon[35283]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:45.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:45 smithi043 ceph-mon[31531]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:45.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:45 smithi179 ceph-mon[35888]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:47.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:47 smithi043 ceph-mon[31531]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:14:47.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:47 smithi179 ceph-mon[35888]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:14:48.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:47 smithi132 ceph-mon[35283]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:14:48.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:14:49.148 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:49.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:49 smithi043 ceph-mon[31531]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:49.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:49 smithi179 ceph-mon[35888]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:50.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:49 smithi132 ceph-mon[35283]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:51.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.596+0000 7f4e0c000700 1 -- 172.21.15.43:0/2361578633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e04109860 msgr2=0x7f4e041111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:51.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.596+0000 7f4e0c000700 1 --2- 172.21.15.43:0/2361578633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e04109860 0x7f4e041111c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00b0d0 tx=0x7f4dfc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.596+0000 7f4e0c000700 1 -- 172.21.15.43:0/2361578633 shutdown_connections 2024-05-27T07:14:51.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.596+0000 7f4e0c000700 1 --2- 172.21.15.43:0/2361578633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e04109860 0x7f4e041111c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.596+0000 7f4e0c000700 1 --2- 172.21.15.43:0/2361578633 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e041045e0 0x7f4e041049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.596+0000 7f4e0c000700 1 --2- 172.21.15.43:0/2361578633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e04108f10 0x7f4e04109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.596+0000 7f4e0c000700 1 -- 172.21.15.43:0/2361578633 >> 172.21.15.43:0/2361578633 conn(0x7f4e04100030 msgr2=0x7f4e04102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:51.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.597+0000 7f4e0c000700 1 -- 172.21.15.43:0/2361578633 shutdown_connections 2024-05-27T07:14:51.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.597+0000 7f4e0c000700 1 -- 172.21.15.43:0/2361578633 wait complete. 2024-05-27T07:14:51.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.598+0000 7f4e0c000700 1 Processor -- start 2024-05-27T07:14:51.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.599+0000 7f4e0c000700 1 -- start start 2024-05-27T07:14:51.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.599+0000 7f4e0c000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e041045e0 0x7f4e0419ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.599+0000 7f4e0c000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e04108f10 0x7f4e0419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.599+0000 7f4e0c000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e04109860 0x7f4e0419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.599+0000 7f4e0c000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4e04114340 con 0x7f4e041045e0 2024-05-27T07:14:51.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.599+0000 7f4e0c000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4e041141c0 con 0x7f4e04108f10 2024-05-27T07:14:51.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.599+0000 7f4e0c000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4e041144c0 con 0x7f4e04109860 2024-05-27T07:14:51.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e09600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e041045e0 0x7f4e0419ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e09600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e041045e0 0x7f4e0419ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45990/0 (socket says 172.21.15.43:45990) 2024-05-27T07:14:51.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e09600700 1 -- 172.21.15.43:0/3934251368 learned_addr learned my addr 172.21.15.43:0/3934251368 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:51.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e08c00700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e04108f10 0x7f4e0419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e0a000700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e04109860 0x7f4e0419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e09600700 1 -- 172.21.15.43:0/3934251368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e04109860 msgr2=0x7f4e0419f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:51.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e09600700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e04109860 0x7f4e0419f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e09600700 1 -- 172.21.15.43:0/3934251368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e04108f10 msgr2=0x7f4e0419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:51.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e09600700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e04108f10 0x7f4e0419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e09600700 1 -- 172.21.15.43:0/3934251368 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0419fbe0 con 0x7f4e041045e0 2024-05-27T07:14:51.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.600+0000 7f4e08c00700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e04108f10 0x7f4e0419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:14:51.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.601+0000 7f4e09600700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e041045e0 0x7f4e0419ad10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4df800c920 tx=0x7f4df800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:51.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.601+0000 7f4e02000700 1 -- 172.21.15.43:0/3934251368 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8007b00 con 0x7f4e041045e0 2024-05-27T07:14:51.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.601+0000 7f4e02000700 1 -- 172.21.15.43:0/3934251368 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4df8004cc0 con 0x7f4e041045e0 2024-05-27T07:14:51.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.601+0000 7f4e02000700 1 -- 172.21.15.43:0/3934251368 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df80054f0 con 0x7f4e041045e0 2024-05-27T07:14:51.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.601+0000 7f4e0c000700 1 -- 172.21.15.43:0/3934251368 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0419fed0 con 0x7f4e041045e0 2024-05-27T07:14:51.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.601+0000 7f4e0c000700 1 -- 172.21.15.43:0/3934251368 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e041a77f0 con 0x7f4e041045e0 2024-05-27T07:14:51.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.606+0000 7f4e02000700 1 -- 172.21.15.43:0/3934251368 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f4df8007ca0 con 0x7f4e041045e0 2024-05-27T07:14:51.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.607+0000 7f4e1aadfb80 1 Processor -- start 2024-05-27T07:14:51.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.607+0000 7f4e1aadfb80 1 -- start start 2024-05-27T07:14:51.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.608+0000 7f4e02000700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4dec0b2a80 0x7f4dec0b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.608+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fc3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.608+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fbdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.608+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fbe1c0 0x55aee9fbe5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.608+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55aee9ee7e40 con 0x55aee9fc4110 2024-05-27T07:14:51.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.608+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55aee9ed79e0 con 0x55aee9e3ae40 2024-05-27T07:14:51.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.608+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55aee9edba20 con 0x55aee9fbe1c0 2024-05-27T07:14:51.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.608+0000 7f4e02000700 1 -- 172.21.15.43:0/3934251368 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f4df80e27b0 con 0x7f4e041045e0 2024-05-27T07:14:51.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.608+0000 7f4e0a000700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4dec0b2a80 0x7f4dec0b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.608+0000 7f4e08c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fbdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.609+0000 7f4e08c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fbdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46000/0 (socket says 172.21.15.43:46000) 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.609+0000 7f4e08c00700 1 -- 172.21.15.43:0/2431167528 learned_addr learned my addr 172.21.15.43:0/2431167528 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.609+0000 7f4e09600700 1 --2- 172.21.15.43:0/2431167528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fc3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.609+0000 7f4e08c00700 1 -- 172.21.15.43:0/2431167528 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fbe1c0 msgr2=0x55aee9fbe5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.609+0000 7f4e08c00700 1 --2- 172.21.15.43:0/2431167528 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fbe1c0 0x55aee9fbe5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.609+0000 7f4e08c00700 1 -- 172.21.15.43:0/2431167528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 msgr2=0x55aee9fc3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.609+0000 7f4e08c00700 1 --2- 172.21.15.43:0/2431167528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fc3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.609+0000 7f4e08c00700 1 -- 172.21.15.43:0/2431167528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aee9ea0420 con 0x55aee9fc4110 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.609+0000 7f4e0a000700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4dec0b2a80 0x7f4dec0b4f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009fd0 tx=0x7f4dfc009780 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:51.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.610+0000 7f4e08c00700 1 --2- 172.21.15.43:0/2431167528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fbdc80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4df400dcb0 tx=0x7f4df400f360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784daf52072d6759 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:51.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.610+0000 7f4df2a00700 1 -- 172.21.15.43:0/2431167528 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df400fb60 con 0x55aee9fc4110 2024-05-27T07:14:51.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.610+0000 7f4df2a00700 1 -- 172.21.15.43:0/2431167528 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4df400fd00 con 0x55aee9fc4110 2024-05-27T07:14:51.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.610+0000 7f4df2a00700 1 -- 172.21.15.43:0/2431167528 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df4010630 con 0x55aee9fc4110 2024-05-27T07:14:51.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.611+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2431167528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 msgr2=0x55aee9fbdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:51.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.611+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2431167528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fbdc80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4df400dcb0 tx=0x7f4df400f360 comp rx=0 tx=0).stop 2024-05-27T07:14:51.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.612+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2431167528 shutdown_connections 2024-05-27T07:14:51.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.612+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2431167528 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fbe1c0 0x55aee9fbe5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.612+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2431167528 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fc3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.612+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2431167528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fbdc80 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.612+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2431167528 >> 172.21.15.43:0/2431167528 conn(0x55aee9e9afc0 msgr2=0x55aee9fb48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:51.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.612+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2431167528 shutdown_connections 2024-05-27T07:14:51.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.612+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2431167528 wait complete. 2024-05-27T07:14:51.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.613+0000 7f4e1aadfb80 1 -- start start 2024-05-27T07:14:51.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.613+0000 7f4e1aadfb80 1 Processor -- start 2024-05-27T07:14:51.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.614+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fc7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.614+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fbe1c0 0x55aee9fc7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.614+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fc8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.614+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55aee9dddfa0 con 0x55aee9fc4110 2024-05-27T07:14:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.614+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55aee9e78760 con 0x55aee9e3ae40 2024-05-27T07:14:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.614+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55aee9fcac10 con 0x55aee9fbe1c0 2024-05-27T07:14:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.614+0000 7f4e1aadfb80 10 client.0 _ll_register_callbacks cb 0x55aee9e81c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:14:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.614+0000 7f4e0a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fc8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.614+0000 7f4e0a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fc8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46012/0 (socket says 172.21.15.43:46012) 2024-05-27T07:14:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.614+0000 7f4e0a000700 1 -- 172.21.15.43:0/3408537591 learned_addr learned my addr 172.21.15.43:0/3408537591 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.615+0000 7f4e09600700 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fc7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.615+0000 7f4e08c00700 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fbe1c0 0x55aee9fc7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.615+0000 7f4e0a000700 1 -- 172.21.15.43:0/3408537591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fbe1c0 msgr2=0x55aee9fc7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.615+0000 7f4e0a000700 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fbe1c0 0x55aee9fc7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.615+0000 7f4e0a000700 1 -- 172.21.15.43:0/3408537591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 msgr2=0x55aee9fc7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.615+0000 7f4e0a000700 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fc7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.615+0000 7f4e0a000700 1 -- 172.21.15.43:0/3408537591 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aee9e278d0 con 0x55aee9fc4110 2024-05-27T07:14:51.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.616+0000 7f4e0a000700 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fc8480 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc004d00 tx=0x7f4dfc0099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cdb1d9b977dd2c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:51.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.616+0000 7f4de7400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:14:51.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.616+0000 7f4de7400700 1 -- 172.21.15.43:0/3408537591 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc031060 con 0x55aee9fc4110 2024-05-27T07:14:51.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.616+0000 7f4de7400700 1 -- 172.21.15.43:0/3408537591 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f4dfc007c50 con 0x55aee9fc4110 2024-05-27T07:14:51.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.616+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/3408537591 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4df400fd00 con 0x55aee9fc4110 2024-05-27T07:14:51.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.616+0000 7f4de7400700 1 -- 172.21.15.43:0/3408537591 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4dfc005040 con 0x55aee9fc4110 2024-05-27T07:14:51.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.616+0000 7f4e1aadfb80 10 client.15534 fetch_fsmap learned FSMap version 30 2024-05-27T07:14:51.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.616+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/3408537591 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55aee9ea8060 con 0x55aee9fc4110 2024-05-27T07:14:51.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.617+0000 7f4de7400700 1 -- 172.21.15.43:0/3408537591 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4df400fd00 con 0x55aee9fc4110 2024-05-27T07:14:51.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.617+0000 7f4de7400700 1 -- 172.21.15.43:0/3408537591 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4dfc011070 con 0x55aee9fc4110 2024-05-27T07:14:51.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.617+0000 7f4e1aadfb80 10 client.15534 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:14:51.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.617+0000 7f4e1aadfb80 10 client.15534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:14:51.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.617+0000 7f4e1aadfb80 20 client.15534 populate_metadata read hostname 'smithi043' 2024-05-27T07:14:51.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.617+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55aee9fd7b60 0x55aee9fd9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.617+0000 7f4e1aadfb80 4 client.15534 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:14:51.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.617+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/3408537591 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aee9ee9d90 con 0x55aee9fd7b60 2024-05-27T07:14:51.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.618+0000 7f4e09600700 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55aee9fd7b60 0x55aee9fd9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.619+0000 7f4e09600700 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55aee9fd7b60 0x55aee9fd9fe0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e51831f79a93d52c server_cookie=7122948466007818 in_seq=0 out_seq=0 2024-05-27T07:14:51.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.619+0000 7f4de7400700 10 client.15534 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4de7400700 1 -- 172.21.15.43:0/3408537591 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55aee9ee9d90 con 0x55aee9fd7b60 2024-05-27T07:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4de7400700 10 client.15534 handle_command_reply: tid=0 2024-05-27T07:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4e1aadfb80 1 client.15534 shutdown 2024-05-27T07:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/3408537591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 msgr2=0x55aee9fc8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fc8480 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc004d00 tx=0x7f4dfc0099e0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/3408537591 shutdown_connections 2024-05-27T07:14:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55aee9fd7b60 0x55aee9fd9fe0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fc4110 0x55aee9fc8480 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fc7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/3408537591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fbe1c0 0x55aee9fc7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.621+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/3408537591 >> 172.21.15.43:0/3408537591 conn(0x55aee9e9afc0 msgr2=0x55aee9fb4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.622+0000 7f4de7400700 0 client.15534 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.622+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/3408537591 shutdown_connections 2024-05-27T07:14:51.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.622+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/3408537591 wait complete. 2024-05-27T07:14:51.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.622+0000 7f4e1aadfb80 20 client.15534 trim_cache size 0 max 16384 2024-05-27T07:14:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.668+0000 7f4e1aadfb80 1 Processor -- start 2024-05-27T07:14:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.669+0000 7f4e1aadfb80 1 -- start start 2024-05-27T07:14:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.669+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9e3ae40 0x55aee9fc9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.669+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fbe1c0 0x55aee9fd3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.669+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9fc4110 0x55aee9fb8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.669+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4e041ae1d0 con 0x55aee9fbe1c0 2024-05-27T07:14:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.669+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4e041afd10 con 0x55aee9fc4110 2024-05-27T07:14:51.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.669+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4e041ae8a0 con 0x55aee9e3ae40 2024-05-27T07:14:51.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e08c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fbe1c0 0x55aee9fd3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e08c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fbe1c0 0x55aee9fd3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46028/0 (socket says 172.21.15.43:46028) 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e08c00700 1 -- 172.21.15.43:0/2946777357 learned_addr learned my addr 172.21.15.43:0/2946777357 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e09600700 1 --2- 172.21.15.43:0/2946777357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9e3ae40 0x55aee9fc9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e0a000700 1 --2- 172.21.15.43:0/2946777357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9fc4110 0x55aee9fb8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e08c00700 1 -- 172.21.15.43:0/2946777357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9e3ae40 msgr2=0x55aee9fc9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e08c00700 1 --2- 172.21.15.43:0/2946777357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9e3ae40 0x55aee9fc9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e08c00700 1 -- 172.21.15.43:0/2946777357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9fc4110 msgr2=0x55aee9fb8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e08c00700 1 --2- 172.21.15.43:0/2946777357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9fc4110 0x55aee9fb8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e08c00700 1 -- 172.21.15.43:0/2946777357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aee9fb6030 con 0x55aee9fbe1c0 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.670+0000 7f4e09600700 1 --2- 172.21.15.43:0/2946777357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9e3ae40 0x55aee9fc9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:14:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.671+0000 7f4e08c00700 1 --2- 172.21.15.43:0/2946777357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fbe1c0 0x55aee9fd3c70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4df400d4c0 tx=0x7f4df4004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a173a84c5a903cc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:51.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.675+0000 7f4df2000700 1 -- 172.21.15.43:0/2946777357 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df4029070 con 0x55aee9fbe1c0 2024-05-27T07:14:51.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.675+0000 7f4df2000700 1 -- 172.21.15.43:0/2946777357 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4df400fdf0 con 0x55aee9fbe1c0 2024-05-27T07:14:51.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.675+0000 7f4df2000700 1 -- 172.21.15.43:0/2946777357 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df4005530 con 0x55aee9fbe1c0 2024-05-27T07:14:51.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.676+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2946777357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fbe1c0 msgr2=0x55aee9fd3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:51.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.676+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2946777357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fbe1c0 0x55aee9fd3c70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4df400d4c0 tx=0x7f4df4004080 comp rx=0 tx=0).stop 2024-05-27T07:14:51.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.677+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2946777357 shutdown_connections 2024-05-27T07:14:51.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.677+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2946777357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9fc4110 0x55aee9fb8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.677+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2946777357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9e3ae40 0x55aee9fc9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.677+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2946777357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fbe1c0 0x55aee9fd3c70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.677+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2946777357 >> 172.21.15.43:0/2946777357 conn(0x55aee9e9afc0 msgr2=0x55aee9fb8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:51.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.677+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2946777357 shutdown_connections 2024-05-27T07:14:51.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.677+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2946777357 wait complete. 2024-05-27T07:14:51.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.677+0000 7f4e1aadfb80 1 -- start start 2024-05-27T07:14:51.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.678+0000 7f4e1aadfb80 1 Processor -- start 2024-05-27T07:14:51.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.678+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fdf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.678+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fc4110 0x55aee9fdf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.678+0000 7f4e1aadfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fdfc90 0x55aeea041670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.678+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4e041af640 con 0x55aee9fdfc90 2024-05-27T07:14:51.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.678+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4e041aef70 con 0x55aee9e3ae40 2024-05-27T07:14:51.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.678+0000 7f4e1aadfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4e041ac170 con 0x55aee9fc4110 2024-05-27T07:14:51.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.678+0000 7f4e1aadfb80 10 client.0 _ll_register_callbacks cb 0x55aee9e81c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:14:51.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e0a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fdfc90 0x55aeea041670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e0a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fdfc90 0x55aeea041670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46030/0 (socket says 172.21.15.43:46030) 2024-05-27T07:14:51.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e0a000700 1 -- 172.21.15.43:0/2064493958 learned_addr learned my addr 172.21.15.43:0/2064493958 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:51.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e08c00700 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fc4110 0x55aee9fdf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e09600700 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fdf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e0a000700 1 -- 172.21.15.43:0/2064493958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fc4110 msgr2=0x55aee9fdf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:51.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e0a000700 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fc4110 0x55aee9fdf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e0a000700 1 -- 172.21.15.43:0/2064493958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 msgr2=0x55aee9fdf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:51.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e0a000700 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fdf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e0a000700 1 -- 172.21.15.43:0/2064493958 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aee9fd09f0 con 0x55aee9fdfc90 2024-05-27T07:14:51.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.679+0000 7f4e09600700 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fdf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:14:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.680+0000 7f4e0a000700 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fdfc90 0x55aeea041670 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00d010 tx=0x7f4dfc007b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd54d63dc42d9b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.680+0000 7f4de7400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:14:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.680+0000 7f4de7400700 1 -- 172.21.15.43:0/2064493958 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc041020 con 0x55aee9fdfc90 2024-05-27T07:14:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.680+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2064493958 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4df400fdf0 con 0x55aee9fdfc90 2024-05-27T07:14:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.680+0000 7f4de7400700 1 -- 172.21.15.43:0/2064493958 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f4dfc01e9b0 con 0x55aee9fdfc90 2024-05-27T07:14:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4de7400700 1 -- 172.21.15.43:0/2064493958 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4dfc031040 con 0x55aee9fdfc90 2024-05-27T07:14:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4e1aadfb80 10 client.15540 fetch_fsmap learned FSMap version 30 2024-05-27T07:14:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2064493958 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55aeea042bd0 con 0x55aee9fdfc90 2024-05-27T07:14:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4de7400700 1 -- 172.21.15.43:0/2064493958 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4df400fdf0 con 0x55aee9fdfc90 2024-05-27T07:14:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4de7400700 1 -- 172.21.15.43:0/2064493958 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4dfc011040 con 0x55aee9fdfc90 2024-05-27T07:14:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4e1aadfb80 10 client.15540 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:14:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4e1aadfb80 10 client.15540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:14:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4e1aadfb80 20 client.15540 populate_metadata read hostname 'smithi043' 2024-05-27T07:14:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55aeea0432d0 0x55aeea0456f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4e1aadfb80 4 client.15540 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:14:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.681+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2064493958 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aee9ee3570 con 0x55aeea0432d0 2024-05-27T07:14:51.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.682+0000 7f4e09600700 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55aeea0432d0 0x55aeea0456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:51.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.683+0000 7f4e09600700 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55aeea0432d0 0x55aeea0456f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc0e3ee2a64f24b5 server_cookie=f985bc72101aa109 in_seq=0 out_seq=0 2024-05-27T07:14:51.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.683+0000 7f4de7400700 10 client.15540 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4de7400700 1 -- 172.21.15.43:0/2064493958 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55aee9ee3570 con 0x55aeea0432d0 2024-05-27T07:14:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4de7400700 10 client.15540 handle_command_reply: tid=0 2024-05-27T07:14:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 client.15540 shutdown 2024-05-27T07:14:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2064493958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fdfc90 msgr2=0x55aeea041670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fdfc90 0x55aeea041670 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00d010 tx=0x7f4dfc007b50 comp rx=0 tx=0).stop 2024-05-27T07:14:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2064493958 shutdown_connections 2024-05-27T07:14:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55aeea0432d0 0x55aeea0456f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aee9fdfc90 0x55aeea041670 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aee9e3ae40 0x55aee9fdf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 --2- 172.21.15.43:0/2064493958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aee9fc4110 0x55aee9fdf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2064493958 >> 172.21.15.43:0/2064493958 conn(0x55aee9e9afc0 msgr2=0x55aee9fc1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4de7400700 0 client.15540 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:14:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2064493958 shutdown_connections 2024-05-27T07:14:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.684+0000 7f4e1aadfb80 1 -- 172.21.15.43:0/2064493958 wait complete. 2024-05-27T07:14:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.685+0000 7f4e1aadfb80 20 client.15540 trim_cache size 0 max 16384 2024-05-27T07:14:51.676 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:14:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:14:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:14:51.677 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:14:51.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.685+0000 7f4e0c000700 1 -- 172.21.15.43:0/3934251368 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4dec0b2a80 msgr2=0x7f4dec0b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:51.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.685+0000 7f4e0c000700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4dec0b2a80 0x7f4dec0b4f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009fd0 tx=0x7f4dfc009780 comp rx=0 tx=0).stop 2024-05-27T07:14:51.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.685+0000 7f4e0c000700 1 -- 172.21.15.43:0/3934251368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e041045e0 msgr2=0x7f4e0419ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:51.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.685+0000 7f4e0c000700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e041045e0 0x7f4e0419ad10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4df800c920 tx=0x7f4df800cce0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.686+0000 7f4e0c000700 1 -- 172.21.15.43:0/3934251368 shutdown_connections 2024-05-27T07:14:51.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.686+0000 7f4e0c000700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4dec0b2a80 0x7f4dec0b4f90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.686+0000 7f4e0c000700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e04109860 0x7f4e0419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.686+0000 7f4e0c000700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e041045e0 0x7f4e0419ad10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.686+0000 7f4e0c000700 1 --2- 172.21.15.43:0/3934251368 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e04108f10 0x7f4e0419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:51.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.686+0000 7f4e0c000700 1 -- 172.21.15.43:0/3934251368 >> 172.21.15.43:0/3934251368 conn(0x7f4e04100030 msgr2=0x7f4e04108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:51.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.686+0000 7f4e0c000700 1 -- 172.21.15.43:0/3934251368 shutdown_connections 2024-05-27T07:14:51.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:51.686+0000 7f4e0c000700 1 -- 172.21.15.43:0/3934251368 wait complete. 2024-05-27T07:14:51.948 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:51 smithi043 ceph-mon[31531]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-05-27T07:14:51.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:51 smithi179 ceph-mon[35888]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-05-27T07:14:52.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:51 smithi132 ceph-mon[35283]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-05-27T07:14:52.560 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:14:52.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:14:52.990 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:53.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:53 smithi043 ceph-mon[31531]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-27T07:14:53.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:53 smithi179 ceph-mon[35888]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-27T07:14:54.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:53 smithi132 ceph-mon[35283]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-27T07:14:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.491+0000 7f44b0200700 1 -- 172.21.15.43:0/2590615834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a8109860 msgr2=0x7f44a81111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.491+0000 7f44b0200700 1 --2- 172.21.15.43:0/2590615834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a8109860 0x7f44a81111c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f44a000b0d0 tx=0x7f44a000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.491+0000 7f44b0200700 1 -- 172.21.15.43:0/2590615834 shutdown_connections 2024-05-27T07:14:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.491+0000 7f44b0200700 1 --2- 172.21.15.43:0/2590615834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a8109860 0x7f44a81111c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.491+0000 7f44b0200700 1 --2- 172.21.15.43:0/2590615834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f44a81045e0 0x7f44a81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.491+0000 7f44b0200700 1 --2- 172.21.15.43:0/2590615834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44a8108f10 0x7f44a8109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.491+0000 7f44b0200700 1 -- 172.21.15.43:0/2590615834 >> 172.21.15.43:0/2590615834 conn(0x7f44a8100030 msgr2=0x7f44a8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.491+0000 7f44b0200700 1 -- 172.21.15.43:0/2590615834 shutdown_connections 2024-05-27T07:14:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.492+0000 7f44b0200700 1 -- 172.21.15.43:0/2590615834 wait complete. 2024-05-27T07:14:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.493+0000 7f44b0200700 1 Processor -- start 2024-05-27T07:14:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.493+0000 7f44b0200700 1 -- start start 2024-05-27T07:14:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.493+0000 7f44b0200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f44a81045e0 0x7f44a819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.493+0000 7f44b0200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a8108f10 0x7f44a819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.493+0000 7f44b0200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44a8109860 0x7f44a819f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.493+0000 7f44b0200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f44a81143b0 con 0x7f44a8108f10 2024-05-27T07:14:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.493+0000 7f44b0200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f44a8114230 con 0x7f44a81045e0 2024-05-27T07:14:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.493+0000 7f44b0200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f44a8114530 con 0x7f44a8109860 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.494+0000 7f44ace00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a8108f10 0x7f44a819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.494+0000 7f44ace00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a8108f10 0x7f44a819b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45464/0 (socket says 172.21.15.43:45464) 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.494+0000 7f44ace00700 1 -- 172.21.15.43:0/3530221443 learned_addr learned my addr 172.21.15.43:0/3530221443 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.494+0000 7f44ad800700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f44a81045e0 0x7f44a819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.494+0000 7f44ae200700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44a8109860 0x7f44a819f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.494+0000 7f44ace00700 1 -- 172.21.15.43:0/3530221443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44a8109860 msgr2=0x7f44a819f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.494+0000 7f44ace00700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44a8109860 0x7f44a819f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.494+0000 7f44ace00700 1 -- 172.21.15.43:0/3530221443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f44a81045e0 msgr2=0x7f44a819ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.494+0000 7f44ace00700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f44a81045e0 0x7f44a819ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.494+0000 7f44ace00700 1 -- 172.21.15.43:0/3530221443 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a819fc50 con 0x7f44a8108f10 2024-05-27T07:14:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.495+0000 7f44ace00700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a8108f10 0x7f44a819b320 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f449400d950 tx=0x7f449400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.495+0000 7f44a6000700 1 -- 172.21.15.43:0/3530221443 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4494017070 con 0x7f44a8108f10 2024-05-27T07:14:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.495+0000 7f44a6000700 1 -- 172.21.15.43:0/3530221443 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f449400de30 con 0x7f44a8108f10 2024-05-27T07:14:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.495+0000 7f44b0200700 1 -- 172.21.15.43:0/3530221443 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a819ff40 con 0x7f44a8108f10 2024-05-27T07:14:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.495+0000 7f44a6000700 1 -- 172.21.15.43:0/3530221443 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4494005310 con 0x7f44a8108f10 2024-05-27T07:14:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.495+0000 7f44b0200700 1 -- 172.21.15.43:0/3530221443 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a81a7860 con 0x7f44a8108f10 2024-05-27T07:14:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.496+0000 7f44b0200700 1 -- 172.21.15.43:0/3530221443 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a80512b0 con 0x7f44a8108f10 2024-05-27T07:14:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.499+0000 7f44a6000700 1 -- 172.21.15.43:0/3530221443 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7f4494004750 con 0x7f44a8108f10 2024-05-27T07:14:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.500+0000 7f44a6000700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f44900b2b00 0x7f44900b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.500+0000 7f44a6000700 1 -- 172.21.15.43:0/3530221443 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f44940e27a0 con 0x7f44a8108f10 2024-05-27T07:14:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.501+0000 7f44ad800700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f44900b2b00 0x7f44900b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.502+0000 7f44ad800700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f44900b2b00 0x7f44900b5010 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f449c005950 tx=0x7f449c0058e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.503+0000 7f44a6000700 1 -- 172.21.15.43:0/3530221443 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f44940a9ce0 con 0x7f44a8108f10 2024-05-27T07:14:55.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.796+0000 7f44b0200700 1 -- 172.21.15.43:0/3530221443 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44a802e5e0 con 0x7f44a8108f10 2024-05-27T07:14:55.788 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:55 smithi043 ceph-mon[31531]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:55.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.798+0000 7f44a6000700 1 -- 172.21.15.43:0/3530221443 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11938 (secure 0 0 0) 0x7f44940a95e0 con 0x7f44a8108f10 2024-05-27T07:14:55.789 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:14:55.790 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:13:52.990426+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34484,34448,34424,24484,34403]},"id":1}]} 2024-05-27T07:14:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.801+0000 7f44b0200700 1 -- 172.21.15.43:0/3530221443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f44900b2b00 msgr2=0x7f44900b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.801+0000 7f44b0200700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f44900b2b00 0x7f44900b5010 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f449c005950 tx=0x7f449c0058e0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.801+0000 7f44b0200700 1 -- 172.21.15.43:0/3530221443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a8108f10 msgr2=0x7f44a819b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.801+0000 7f44b0200700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a8108f10 0x7f44a819b320 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f449400d950 tx=0x7f449400dd10 comp rx=0 tx=0).stop 2024-05-27T07:14:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.801+0000 7f44b0200700 1 -- 172.21.15.43:0/3530221443 shutdown_connections 2024-05-27T07:14:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.801+0000 7f44b0200700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f44900b2b00 0x7f44900b5010 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.801+0000 7f44b0200700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44a8109860 0x7f44a819f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.801+0000 7f44b0200700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f44a81045e0 0x7f44a819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.801+0000 7f44b0200700 1 --2- 172.21.15.43:0/3530221443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a8108f10 0x7f44a819b320 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.801+0000 7f44b0200700 1 -- 172.21.15.43:0/3530221443 >> 172.21.15.43:0/3530221443 conn(0x7f44a8100030 msgr2=0x7f44a810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.802+0000 7f44b0200700 1 -- 172.21.15.43:0/3530221443 shutdown_connections 2024-05-27T07:14:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:55.802+0000 7f44b0200700 1 -- 172.21.15.43:0/3530221443 wait complete. 2024-05-27T07:14:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 30 2024-05-27T07:14:55.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:55 smithi179 ceph-mon[35888]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:56.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:55 smithi132 ceph-mon[35283]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:56.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:14:56.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:56 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3530221443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:14:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:56 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3530221443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:14:57.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:56 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3530221443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:14:57.283 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:14:57.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:57 smithi043 ceph-mon[31531]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:14:57.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:57 smithi179 ceph-mon[35888]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:14:58.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:57 smithi132 ceph-mon[35283]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:14:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.840+0000 7ff113200700 1 -- 172.21.15.43:0/786815252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10c10e6f0 msgr2=0x7ff10c110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.840+0000 7ff113200700 1 --2- 172.21.15.43:0/786815252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10c10e6f0 0x7ff10c110bd0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff10800b0a0 tx=0x7ff10800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.833 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:14:59 smithi043 ceph-mon[31531]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:14:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.840+0000 7ff113200700 1 -- 172.21.15.43:0/786815252 shutdown_connections 2024-05-27T07:14:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.840+0000 7ff113200700 1 --2- 172.21.15.43:0/786815252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10c10e6f0 0x7ff10c110bd0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.840+0000 7ff113200700 1 --2- 172.21.15.43:0/786815252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10c105280 0x7ff10c105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.840+0000 7ff113200700 1 --2- 172.21.15.43:0/786815252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10c105bd0 0x7ff10c10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.840+0000 7ff113200700 1 -- 172.21.15.43:0/786815252 >> 172.21.15.43:0/786815252 conn(0x7ff10c078ab0 msgr2=0x7ff10c078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.841+0000 7ff113200700 1 -- 172.21.15.43:0/786815252 shutdown_connections 2024-05-27T07:14:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.841+0000 7ff113200700 1 -- 172.21.15.43:0/786815252 wait complete. 2024-05-27T07:14:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.842+0000 7ff113200700 1 Processor -- start 2024-05-27T07:14:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.842+0000 7ff113200700 1 -- start start 2024-05-27T07:14:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff113200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10c105280 0x7ff10c1a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff113200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10c105bd0 0x7ff10c1a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff113200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10c10e6f0 0x7ff10c19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff113200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c114150 con 0x7ff10c105280 2024-05-27T07:14:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff113200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c113fd0 con 0x7ff10c10e6f0 2024-05-27T07:14:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff113200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c1142d0 con 0x7ff10c105bd0 2024-05-27T07:14:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff111e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10c105280 0x7ff10c1a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff111e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10c105280 0x7ff10c1a29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45472/0 (socket says 172.21.15.43:45472) 2024-05-27T07:14:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff111e00700 1 -- 172.21.15.43:0/418712857 learned_addr learned my addr 172.21.15.43:0/418712857 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff111400700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10c105bd0 0x7ff10c1a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.843+0000 7ff112800700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10c10e6f0 0x7ff10c19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff111e00700 1 -- 172.21.15.43:0/418712857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10c105bd0 msgr2=0x7ff10c1a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:59.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff111e00700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10c105bd0 0x7ff10c1a2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff111e00700 1 -- 172.21.15.43:0/418712857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10c10e6f0 msgr2=0x7ff10c19cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:59.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff111e00700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10c10e6f0 0x7ff10c19cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff111e00700 1 -- 172.21.15.43:0/418712857 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10c19d280 con 0x7ff10c105280 2024-05-27T07:14:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff111400700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10c105bd0 0x7ff10c1a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:14:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff111e00700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10c105280 0x7ff10c1a29c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff10000d950 tx=0x7ff10000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff106a00700 1 -- 172.21.15.43:0/418712857 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff100016070 con 0x7ff10c105280 2024-05-27T07:14:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff106a00700 1 -- 172.21.15.43:0/418712857 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff100010460 con 0x7ff10c105280 2024-05-27T07:14:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff113200700 1 -- 172.21.15.43:0/418712857 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10c19d570 con 0x7ff10c105280 2024-05-27T07:14:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.845+0000 7ff113200700 1 -- 172.21.15.43:0/418712857 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff10c1a9770 con 0x7ff10c105280 2024-05-27T07:14:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.844+0000 7ff106a00700 1 -- 172.21.15.43:0/418712857 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10000f620 con 0x7ff10c105280 2024-05-27T07:14:59.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.849+0000 7ff106a00700 1 -- 172.21.15.43:0/418712857 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 24) ==== 149828+0+0 (secure 0 0 0) 0x7ff100019070 con 0x7ff10c105280 2024-05-27T07:14:59.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.850+0000 7ff123372b80 1 Processor -- start 2024-05-27T07:14:59.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.850+0000 7ff123372b80 1 -- start start 2024-05-27T07:14:59.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff123372b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91caded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff123372b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91cb8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff106a00700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff0f40b2a80 0x7ff0f40b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff123372b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cbb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff123372b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55da91bdce40 con 0x55da91cae820 2024-05-27T07:14:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff123372b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55da91bcc9e0 con 0x55da91b2fe40 2024-05-27T07:14:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff123372b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55da91bd0a20 con 0x55da91cae410 2024-05-27T07:14:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff106a00700 1 -- 172.21.15.43:0/418712857 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff1000e27b0 con 0x7ff10c105280 2024-05-27T07:14:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff112800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cbb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff112800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cbb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45486/0 (socket says 172.21.15.43:45486) 2024-05-27T07:14:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff112800700 1 -- 172.21.15.43:0/3227086742 learned_addr learned my addr 172.21.15.43:0/3227086742 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff111e00700 1 --2- 172.21.15.43:0/3227086742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91cb8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff112800700 1 --2- 172.21.15.43:0/3227086742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91caded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.851+0000 7ff111400700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff0f40b2a80 0x7ff0f40b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.852+0000 7ff112800700 1 -- 172.21.15.43:0/3227086742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 msgr2=0x55da91cb8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.852+0000 7ff112800700 1 --2- 172.21.15.43:0/3227086742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91cb8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.852+0000 7ff112800700 1 -- 172.21.15.43:0/3227086742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 msgr2=0x55da91caded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.852+0000 7ff112800700 1 --2- 172.21.15.43:0/3227086742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91caded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.852+0000 7ff112800700 1 -- 172.21.15.43:0/3227086742 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da91b95420 con 0x55da91cae820 2024-05-27T07:14:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.852+0000 7ff112800700 1 --2- 172.21.15.43:0/3227086742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cbb020 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff108015000 tx=0x7ff108009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0dcade28cc7ed94 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.853+0000 7ff111400700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff0f40b2a80 0x7ff0f40b4f90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc005e10 tx=0x7ff0fc005d60 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.853+0000 7ff0fb400700 1 -- 172.21.15.43:0/3227086742 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108004d00 con 0x55da91cae820 2024-05-27T07:14:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.853+0000 7ff0fb400700 1 -- 172.21.15.43:0/3227086742 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff108007a50 con 0x55da91cae820 2024-05-27T07:14:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.853+0000 7ff0fb400700 1 -- 172.21.15.43:0/3227086742 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10801d6b0 con 0x55da91cae820 2024-05-27T07:14:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.854+0000 7ff123372b80 1 -- 172.21.15.43:0/3227086742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 msgr2=0x55da91cbb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.854+0000 7ff123372b80 1 --2- 172.21.15.43:0/3227086742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cbb020 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff108015000 tx=0x7ff108009800 comp rx=0 tx=0).stop 2024-05-27T07:14:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.854+0000 7ff123372b80 1 -- 172.21.15.43:0/3227086742 shutdown_connections 2024-05-27T07:14:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.854+0000 7ff123372b80 1 --2- 172.21.15.43:0/3227086742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cbb020 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.854+0000 7ff123372b80 1 --2- 172.21.15.43:0/3227086742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91caded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.854+0000 7ff123372b80 1 --2- 172.21.15.43:0/3227086742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91cb8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.854+0000 7ff123372b80 1 -- 172.21.15.43:0/3227086742 >> 172.21.15.43:0/3227086742 conn(0x55da91b8ffc0 msgr2=0x55da91ca9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.854+0000 7ff123372b80 1 -- 172.21.15.43:0/3227086742 shutdown_connections 2024-05-27T07:14:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.854+0000 7ff123372b80 1 -- 172.21.15.43:0/3227086742 wait complete. 2024-05-27T07:14:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.855+0000 7ff123372b80 1 -- start start 2024-05-27T07:14:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.855+0000 7ff123372b80 1 Processor -- start 2024-05-27T07:14:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.856+0000 7ff123372b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91cb85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.856+0000 7ff123372b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91cb8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.856+0000 7ff123372b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.856+0000 7ff123372b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55da91ad2fa0 con 0x55da91cae820 2024-05-27T07:14:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.856+0000 7ff123372b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55da91b6d760 con 0x55da91b2fe40 2024-05-27T07:14:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.856+0000 7ff123372b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55da91cbb930 con 0x55da91cae410 2024-05-27T07:14:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.856+0000 7ff123372b80 10 client.0 _ll_register_callbacks cb 0x55da91b76c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:14:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.856+0000 7ff112800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.857+0000 7ff112800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45500/0 (socket says 172.21.15.43:45500) 2024-05-27T07:14:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.857+0000 7ff112800700 1 -- 172.21.15.43:0/1493375203 learned_addr learned my addr 172.21.15.43:0/1493375203 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.857+0000 7ff111e00700 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91cb85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.857+0000 7ff111400700 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91cb8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.857+0000 7ff112800700 1 -- 172.21.15.43:0/1493375203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 msgr2=0x55da91cb8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.857+0000 7ff112800700 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91cb8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.857+0000 7ff112800700 1 -- 172.21.15.43:0/1493375203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 msgr2=0x55da91cb85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.857+0000 7ff112800700 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91cb85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.857+0000 7ff112800700 1 -- 172.21.15.43:0/1493375203 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da91b1c8d0 con 0x55da91cae820 2024-05-27T07:14:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.858+0000 7ff112800700 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb9060 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff108002230 tx=0x7ff108007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89ee3bccc7aa843 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.858+0000 7ff0efe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:14:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.858+0000 7ff0efe00700 1 -- 172.21.15.43:0/1493375203 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10802e060 con 0x55da91cae820 2024-05-27T07:14:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.858+0000 7ff123372b80 1 -- 172.21.15.43:0/1493375203 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff108007a50 con 0x55da91cae820 2024-05-27T07:14:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.858+0000 7ff0efe00700 1 -- 172.21.15.43:0/1493375203 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff108029660 con 0x55da91cae820 2024-05-27T07:14:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.858+0000 7ff0efe00700 1 -- 172.21.15.43:0/1493375203 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff108035050 con 0x55da91cae820 2024-05-27T07:14:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.859+0000 7ff123372b80 10 client.15558 fetch_fsmap learned FSMap version 30 2024-05-27T07:14:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.859+0000 7ff123372b80 1 -- 172.21.15.43:0/1493375203 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55da91b9d060 con 0x55da91cae820 2024-05-27T07:14:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.859+0000 7ff0efe00700 1 -- 172.21.15.43:0/1493375203 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff108007a50 con 0x55da91cae820 2024-05-27T07:14:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.859+0000 7ff0efe00700 1 -- 172.21.15.43:0/1493375203 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff108011070 con 0x55da91cae820 2024-05-27T07:14:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.859+0000 7ff123372b80 10 client.15558 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:14:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.859+0000 7ff123372b80 10 client.15558 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:14:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.859+0000 7ff123372b80 20 client.15558 populate_metadata read hostname 'smithi043' 2024-05-27T07:14:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.859+0000 7ff123372b80 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55da91cc8790 0x55da91ccabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.859+0000 7ff123372b80 4 client.15558 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:14:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.859+0000 7ff123372b80 1 -- 172.21.15.43:0/1493375203 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da91bded90 con 0x55da91cc8790 2024-05-27T07:14:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.860+0000 7ff111e00700 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55da91cc8790 0x55da91ccabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.861+0000 7ff111e00700 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55da91cc8790 0x55da91ccabb0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=91e257cdfe5e2cbb server_cookie=36b2fc4c66d8cf05 in_seq=0 out_seq=0 2024-05-27T07:14:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.861+0000 7ff0efe00700 10 client.15558 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:14:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff0efe00700 1 -- 172.21.15.43:0/1493375203 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55da91bded90 con 0x55da91cc8790 2024-05-27T07:14:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff0efe00700 10 client.15558 handle_command_reply: tid=0 2024-05-27T07:14:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff123372b80 1 client.15558 shutdown 2024-05-27T07:14:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff123372b80 1 -- 172.21.15.43:0/1493375203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 msgr2=0x55da91cb9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff123372b80 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb9060 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff108002230 tx=0x7ff108007850 comp rx=0 tx=0).stop 2024-05-27T07:14:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff123372b80 1 -- 172.21.15.43:0/1493375203 shutdown_connections 2024-05-27T07:14:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff123372b80 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55da91cc8790 0x55da91ccabb0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff123372b80 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb9060 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff123372b80 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91cb85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff123372b80 1 --2- 172.21.15.43:0/1493375203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91cb8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff123372b80 1 -- 172.21.15.43:0/1493375203 >> 172.21.15.43:0/1493375203 conn(0x55da91b8ffc0 msgr2=0x55da91ca8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.863+0000 7ff0efe00700 0 client.15558 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:14:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.864+0000 7ff123372b80 1 -- 172.21.15.43:0/1493375203 shutdown_connections 2024-05-27T07:14:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.864+0000 7ff123372b80 1 -- 172.21.15.43:0/1493375203 wait complete. 2024-05-27T07:14:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.865+0000 7ff123372b80 20 client.15558 trim_cache size 0 max 16384 2024-05-27T07:14:59.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.884+0000 7ff123372b80 1 Processor -- start 2024-05-27T07:14:59.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.884+0000 7ff123372b80 1 -- start start 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff123372b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91b2fe40 0x55da91cbaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff123372b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91cae410 0x55da91cb8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff123372b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff123372b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c1b24c0 con 0x55da91cae820 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff123372b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c1b4000 con 0x55da91cae410 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff123372b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c1b2b90 con 0x55da91b2fe40 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff112800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff112800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45512/0 (socket says 172.21.15.43:45512) 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff112800700 1 -- 172.21.15.43:0/3576167650 learned_addr learned my addr 172.21.15.43:0/3576167650 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff111e00700 1 --2- 172.21.15.43:0/3576167650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91b2fe40 0x55da91cbaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.885+0000 7ff111400700 1 --2- 172.21.15.43:0/3576167650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91cae410 0x55da91cb8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.886+0000 7ff112800700 1 -- 172.21.15.43:0/3576167650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91b2fe40 msgr2=0x55da91cbaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:59.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.886+0000 7ff112800700 1 --2- 172.21.15.43:0/3576167650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91b2fe40 0x55da91cbaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.886+0000 7ff112800700 1 -- 172.21.15.43:0/3576167650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91cae410 msgr2=0x55da91cb8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:59.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.886+0000 7ff112800700 1 --2- 172.21.15.43:0/3576167650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91cae410 0x55da91cb8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.886+0000 7ff112800700 1 -- 172.21.15.43:0/3576167650 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da91cab350 con 0x55da91cae820 2024-05-27T07:14:59.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.886+0000 7ff112800700 1 --2- 172.21.15.43:0/3576167650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb3d90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff10800b3f0 tx=0x7ff10801dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f10469aaf10412 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:59.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.886+0000 7ff0faa00700 1 -- 172.21.15.43:0/3576167650 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10802e070 con 0x55da91cae820 2024-05-27T07:14:59.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.886+0000 7ff0faa00700 1 -- 172.21.15.43:0/3576167650 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1080078c0 con 0x55da91cae820 2024-05-27T07:14:59.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.886+0000 7ff0faa00700 1 -- 172.21.15.43:0/3576167650 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108029b50 con 0x55da91cae820 2024-05-27T07:14:59.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.887+0000 7ff123372b80 1 -- 172.21.15.43:0/3576167650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 msgr2=0x55da91cb3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:59.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.887+0000 7ff123372b80 1 --2- 172.21.15.43:0/3576167650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb3d90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff10800b3f0 tx=0x7ff10801dda0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.887+0000 7ff123372b80 1 -- 172.21.15.43:0/3576167650 shutdown_connections 2024-05-27T07:14:59.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.887+0000 7ff123372b80 1 --2- 172.21.15.43:0/3576167650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91cb3d90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.887+0000 7ff123372b80 1 --2- 172.21.15.43:0/3576167650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91b2fe40 0x55da91cbaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.887+0000 7ff123372b80 1 --2- 172.21.15.43:0/3576167650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91cae410 0x55da91cb8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.887+0000 7ff123372b80 1 -- 172.21.15.43:0/3576167650 >> 172.21.15.43:0/3576167650 conn(0x55da91b8ffc0 msgr2=0x55da91cc3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:59.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.888+0000 7ff123372b80 1 -- 172.21.15.43:0/3576167650 shutdown_connections 2024-05-27T07:14:59.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.888+0000 7ff123372b80 1 -- 172.21.15.43:0/3576167650 wait complete. 2024-05-27T07:14:59.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.888+0000 7ff123372b80 1 -- start start 2024-05-27T07:14:59.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.888+0000 7ff123372b80 1 Processor -- start 2024-05-27T07:14:59.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff123372b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91ccdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff123372b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91ccb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff123372b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91ccb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff123372b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c1b3930 con 0x55da91cae820 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff123372b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c1b3260 con 0x55da91b2fe40 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff123372b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c1b0440 con 0x55da91cae410 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff123372b80 10 client.0 _ll_register_callbacks cb 0x55da91b76c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff112800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91ccb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff112800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91ccb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45520/0 (socket says 172.21.15.43:45520) 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff112800700 1 -- 172.21.15.43:0/2389005909 learned_addr learned my addr 172.21.15.43:0/2389005909 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.889+0000 7ff111e00700 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91ccdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.890+0000 7ff111400700 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91ccb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.890+0000 7ff112800700 1 -- 172.21.15.43:0/2389005909 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 msgr2=0x55da91ccb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.890+0000 7ff112800700 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91ccb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.890+0000 7ff112800700 1 -- 172.21.15.43:0/2389005909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 msgr2=0x55da91ccdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:14:59.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.890+0000 7ff112800700 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91ccdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.890+0000 7ff112800700 1 -- 172.21.15.43:0/2389005909 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da91ccbdc0 con 0x55da91cae820 2024-05-27T07:14:59.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.890+0000 7ff111400700 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91ccb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:14:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.890+0000 7ff112800700 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91ccb880 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff10800d010 tx=0x7ff10801d450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abad2f1e3f2eb5f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:14:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.890+0000 7ff0efe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:14:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.891+0000 7ff0efe00700 1 -- 172.21.15.43:0/2389005909 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108046060 con 0x55da91cae820 2024-05-27T07:14:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.891+0000 7ff0efe00700 1 -- 172.21.15.43:0/2389005909 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff108029460 con 0x55da91cae820 2024-05-27T07:14:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.891+0000 7ff123372b80 1 -- 172.21.15.43:0/2389005909 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1080078c0 con 0x55da91cae820 2024-05-27T07:14:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.891+0000 7ff0efe00700 1 -- 172.21.15.43:0/2389005909 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff10804d050 con 0x55da91cae820 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.891+0000 7ff123372b80 10 client.15564 fetch_fsmap learned FSMap version 30 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.891+0000 7ff123372b80 1 -- 172.21.15.43:0/2389005909 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55da91d32be0 con 0x55da91cae820 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.891+0000 7ff0efe00700 1 -- 172.21.15.43:0/2389005909 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1080078c0 con 0x55da91cae820 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.892+0000 7ff0efe00700 1 -- 172.21.15.43:0/2389005909 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff108011040 con 0x55da91cae820 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.892+0000 7ff123372b80 10 client.15564 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.892+0000 7ff123372b80 10 client.15564 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.892+0000 7ff123372b80 20 client.15564 populate_metadata read hostname 'smithi043' 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.892+0000 7ff123372b80 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55da91d33120 0x55da91d35540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.892+0000 7ff123372b80 4 client.15564 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.892+0000 7ff123372b80 1 -- 172.21.15.43:0/2389005909 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da91bd8570 con 0x55da91d33120 2024-05-27T07:14:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.893+0000 7ff111e00700 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55da91d33120 0x55da91d35540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:14:59.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.894+0000 7ff111e00700 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55da91d33120 0x55da91d35540 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1c522ee53dc15d8b server_cookie=73f72aaa804953fb in_seq=0 out_seq=0 2024-05-27T07:14:59.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.894+0000 7ff0efe00700 10 client.15564 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.894+0000 7ff0efe00700 1 -- 172.21.15.43:0/2389005909 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da91bd8570 con 0x55da91d33120 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.894+0000 7ff0efe00700 10 client.15564 handle_command_reply: tid=0 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.894+0000 7ff123372b80 1 client.15564 shutdown 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.894+0000 7ff123372b80 1 -- 172.21.15.43:0/2389005909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 msgr2=0x55da91ccb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.894+0000 7ff123372b80 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91ccb880 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff10800d010 tx=0x7ff10801d450 comp rx=0 tx=0).stop 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.894+0000 7ff123372b80 1 -- 172.21.15.43:0/2389005909 shutdown_connections 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.895+0000 7ff123372b80 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55da91d33120 0x55da91d35540 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.895+0000 7ff123372b80 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da91cae820 0x55da91ccb880 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.895+0000 7ff123372b80 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da91b2fe40 0x55da91ccdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.895+0000 7ff123372b80 1 --2- 172.21.15.43:0/2389005909 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da91cae410 0x55da91ccb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.895+0000 7ff123372b80 1 -- 172.21.15.43:0/2389005909 >> 172.21.15.43:0/2389005909 conn(0x55da91b8ffc0 msgr2=0x55da91cb8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.895+0000 7ff0efe00700 0 client.15564 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:14:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.895+0000 7ff123372b80 1 -- 172.21.15.43:0/2389005909 shutdown_connections 2024-05-27T07:14:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.895+0000 7ff123372b80 1 -- 172.21.15.43:0/2389005909 wait complete. 2024-05-27T07:14:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.895+0000 7ff123372b80 20 client.15564 trim_cache size 0 max 16384 2024-05-27T07:14:59.888 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 -- 172.21.15.43:0/418712857 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff0f40b2a80 msgr2=0x7ff0f40b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff0f40b2a80 0x7ff0f40b4f90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc005e10 tx=0x7ff0fc005d60 comp rx=0 tx=0).stop 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 -- 172.21.15.43:0/418712857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10c105280 msgr2=0x7ff10c1a29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10c105280 0x7ff10c1a29c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff10000d950 tx=0x7ff10000dd10 comp rx=0 tx=0).stop 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 -- 172.21.15.43:0/418712857 shutdown_connections 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff0f40b2a80 0x7ff0f40b4f90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff10c10e6f0 0x7ff10c19cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff10c105280 0x7ff10c1a29c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 --2- 172.21.15.43:0/418712857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff10c105bd0 0x7ff10c1a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 -- 172.21.15.43:0/418712857 >> 172.21.15.43:0/418712857 conn(0x7ff10c078ab0 msgr2=0x7ff10c102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 -- 172.21.15.43:0/418712857 shutdown_connections 2024-05-27T07:14:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:14:59.896+0000 7ff113200700 1 -- 172.21.15.43:0/418712857 wait complete. 2024-05-27T07:14:59.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:14:59 smithi179 ceph-mon[35888]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:00.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:14:59 smithi132 ceph-mon[35283]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:00.970 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:15:01.203 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:15:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:15:00.986+0000 7fb919a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:15:01.203 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:15:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:15:00.986+0000 7fb919a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:15:01.203 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:15:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:15:00.986+0000 7fb919a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:15:01.203 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:15:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:15:00.986+0000 7fb919a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:15:01.203 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:15:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:15:00.986+0000 7fb919a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-27T07:15:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:01.914 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:01 smithi043 ceph-mon[31531]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:01.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:01 smithi179 ceph-mon[35888]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:02.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:01 smithi132 ceph-mon[35283]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:03.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.804+0000 7fba46800700 1 -- 172.21.15.43:0/2702742060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba40109670 msgr2=0x7fba40110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:03.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.804+0000 7fba46800700 1 --2- 172.21.15.43:0/2702742060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba40109670 0x7fba40110fd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fba380099c0 tx=0x7fba38009cd0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.797 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:03 smithi043 ceph-mon[31531]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:15:03.797 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:03 smithi043 ceph-mon[31531]: mgrmap e25: x(active, since 68s), standbys: z, y 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.804+0000 7fba46800700 1 -- 172.21.15.43:0/2702742060 shutdown_connections 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.804+0000 7fba46800700 1 --2- 172.21.15.43:0/2702742060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba40109670 0x7fba40110fd0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.804+0000 7fba46800700 1 --2- 172.21.15.43:0/2702742060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba401043f0 0x7fba40104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.804+0000 7fba46800700 1 --2- 172.21.15.43:0/2702742060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba40108d20 0x7fba40109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.804+0000 7fba46800700 1 -- 172.21.15.43:0/2702742060 >> 172.21.15.43:0/2702742060 conn(0x7fba40079d00 msgr2=0x7fba4007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.804+0000 7fba46800700 1 -- 172.21.15.43:0/2702742060 shutdown_connections 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.804+0000 7fba46800700 1 -- 172.21.15.43:0/2702742060 wait complete. 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.806+0000 7fba46800700 1 Processor -- start 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.806+0000 7fba46800700 1 -- start start 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.806+0000 7fba46800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba401043f0 0x7fba401a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.806+0000 7fba46800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba40108d20 0x7fba401a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.806+0000 7fba46800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba40109670 0x7fba4019cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.806+0000 7fba46800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fba40114150 con 0x7fba401043f0 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.806+0000 7fba46800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fba40113fd0 con 0x7fba40109670 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.806+0000 7fba46800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fba401142d0 con 0x7fba40108d20 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba45400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba401043f0 0x7fba401a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba45400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba401043f0 0x7fba401a29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45552/0 (socket says 172.21.15.43:45552) 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba45400700 1 -- 172.21.15.43:0/3746734664 learned_addr learned my addr 172.21.15.43:0/3746734664 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba44a00700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba40108d20 0x7fba401a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba45e00700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba40109670 0x7fba4019cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba45400700 1 -- 172.21.15.43:0/3746734664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba40108d20 msgr2=0x7fba401a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba45400700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba40108d20 0x7fba401a2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba45400700 1 -- 172.21.15.43:0/3746734664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba40109670 msgr2=0x7fba4019cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:03.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba45400700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba40109670 0x7fba4019cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba45400700 1 -- 172.21.15.43:0/3746734664 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4019d280 con 0x7fba401043f0 2024-05-27T07:15:03.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.807+0000 7fba45e00700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba40109670 0x7fba4019cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:15:03.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.808+0000 7fba45400700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba401043f0 0x7fba401a29c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fba340060b0 tx=0x7fba3400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:03.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.808+0000 7fba3e000700 1 -- 172.21.15.43:0/3746734664 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3401a070 con 0x7fba401043f0 2024-05-27T07:15:03.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.808+0000 7fba3e000700 1 -- 172.21.15.43:0/3746734664 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba3400ddd0 con 0x7fba401043f0 2024-05-27T07:15:03.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.808+0000 7fba46800700 1 -- 172.21.15.43:0/3746734664 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4019d570 con 0x7fba401043f0 2024-05-27T07:15:03.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.808+0000 7fba3e000700 1 -- 172.21.15.43:0/3746734664 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba34014690 con 0x7fba401043f0 2024-05-27T07:15:03.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.808+0000 7fba46800700 1 -- 172.21.15.43:0/3746734664 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba401a9770 con 0x7fba401043f0 2024-05-27T07:15:03.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.814+0000 7fba3e000700 1 -- 172.21.15.43:0/3746734664 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7fba3401d070 con 0x7fba401043f0 2024-05-27T07:15:03.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.815+0000 7fba568e6b80 1 Processor -- start 2024-05-27T07:15:03.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.815+0000 7fba568e6b80 1 -- start start 2024-05-27T07:15:03.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 0x55d064f39750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064db9e40 0x55d064f39c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064f432c0 0x55d064f456e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba568e6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d064e66e40 con 0x55d064f432c0 2024-05-27T07:15:03.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba568e6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d064e569e0 con 0x55d064ea59f0 2024-05-27T07:15:03.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba568e6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d064e5aa20 con 0x55d064db9e40 2024-05-27T07:15:03.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba3e000700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba280b2b10 0x7fba280b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba44a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064f432c0 0x55d064f456e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba44a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064f432c0 0x55d064f456e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45566/0 (socket says 172.21.15.43:45566) 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba44a00700 1 -- 172.21.15.43:0/2172714510 learned_addr learned my addr 172.21.15.43:0/2172714510 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba3e000700 1 -- 172.21.15.43:0/3746734664 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fba340e2480 con 0x7fba401043f0 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba45e00700 1 --2- 172.21.15.43:0/2172714510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 0x55d064f39750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.816+0000 7fba45400700 1 --2- 172.21.15.43:0/2172714510 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064db9e40 0x55d064f39c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.817+0000 7fba45e00700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba280b2b10 0x7fba280b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.817+0000 7fba44a00700 1 -- 172.21.15.43:0/2172714510 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064db9e40 msgr2=0x55d064f39c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.817+0000 7fba44a00700 1 --2- 172.21.15.43:0/2172714510 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064db9e40 0x55d064f39c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.817+0000 7fba44a00700 1 -- 172.21.15.43:0/2172714510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 msgr2=0x55d064f39750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.817+0000 7fba44a00700 1 --2- 172.21.15.43:0/2172714510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 0x55d064f39750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.817+0000 7fba44a00700 1 -- 172.21.15.43:0/2172714510 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d064e1f420 con 0x55d064f432c0 2024-05-27T07:15:03.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.818+0000 7fba44a00700 1 --2- 172.21.15.43:0/2172714510 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064f432c0 0x55d064f456e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00bc30 tx=0x7fba2c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8188e5a8ff1a2e44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:03.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.818+0000 7fba45e00700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba280b2b10 0x7fba280b5020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fba38009750 tx=0x7fba38009f50 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:03.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.818+0000 7fba32a00700 1 -- 172.21.15.43:0/2172714510 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c010ca0 con 0x55d064f432c0 2024-05-27T07:15:03.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.818+0000 7fba32a00700 1 -- 172.21.15.43:0/2172714510 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba2c010e40 con 0x55d064f432c0 2024-05-27T07:15:03.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.818+0000 7fba32a00700 1 -- 172.21.15.43:0/2172714510 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c005020 con 0x55d064f432c0 2024-05-27T07:15:03.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.819+0000 7fba568e6b80 1 -- 172.21.15.43:0/2172714510 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064f432c0 msgr2=0x55d064f456e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:03.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.819+0000 7fba568e6b80 1 --2- 172.21.15.43:0/2172714510 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064f432c0 0x55d064f456e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00bc30 tx=0x7fba2c00bf40 comp rx=0 tx=0).stop 2024-05-27T07:15:03.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.819+0000 7fba568e6b80 1 -- 172.21.15.43:0/2172714510 shutdown_connections 2024-05-27T07:15:03.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.819+0000 7fba568e6b80 1 --2- 172.21.15.43:0/2172714510 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064f432c0 0x55d064f456e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.819+0000 7fba568e6b80 1 --2- 172.21.15.43:0/2172714510 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 0x55d064f39750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.819+0000 7fba568e6b80 1 --2- 172.21.15.43:0/2172714510 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064db9e40 0x55d064f39c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.819+0000 7fba568e6b80 1 -- 172.21.15.43:0/2172714510 >> 172.21.15.43:0/2172714510 conn(0x55d064e53390 msgr2=0x55d064f34710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:03.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.819+0000 7fba568e6b80 1 -- 172.21.15.43:0/2172714510 shutdown_connections 2024-05-27T07:15:03.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.819+0000 7fba568e6b80 1 -- 172.21.15.43:0/2172714510 wait complete. 2024-05-27T07:15:03.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.820+0000 7fba568e6b80 1 -- start start 2024-05-27T07:15:03.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.820+0000 7fba568e6b80 1 Processor -- start 2024-05-27T07:15:03.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.821+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064db9e40 0x55d064f40200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.821+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 0x55d064f40740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.821+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064f40c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.821+0000 7fba568e6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d064d5cfa0 con 0x55d064db9e40 2024-05-27T07:15:03.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.821+0000 7fba568e6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d064df7760 con 0x55d064ea59f0 2024-05-27T07:15:03.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.821+0000 7fba568e6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d064f45e00 con 0x55d064f432c0 2024-05-27T07:15:03.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.821+0000 7fba568e6b80 10 client.0 _ll_register_callbacks cb 0x55d064e00c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:03.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.821+0000 7fba45400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064db9e40 0x55d064f40200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.821+0000 7fba45400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064db9e40 0x55d064f40200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45578/0 (socket says 172.21.15.43:45578) 2024-05-27T07:15:03.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.821+0000 7fba45400700 1 -- 172.21.15.43:0/3935783073 learned_addr learned my addr 172.21.15.43:0/3935783073 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:03.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.822+0000 7fba44a00700 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 0x55d064f40740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.822+0000 7fba45e00700 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064f40c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.822+0000 7fba45400700 1 -- 172.21.15.43:0/3935783073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 msgr2=0x55d064f40c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.822+0000 7fba45400700 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064f40c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.822+0000 7fba45400700 1 -- 172.21.15.43:0/3935783073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 msgr2=0x55d064f40740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.822+0000 7fba45400700 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 0x55d064f40740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.822+0000 7fba45400700 1 -- 172.21.15.43:0/3935783073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d064da68d0 con 0x55d064db9e40 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.822+0000 7fba44a00700 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 0x55d064f40740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.822+0000 7fba45400700 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064db9e40 0x55d064f40200 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fba34009510 tx=0x7fba340a96f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921442e71819ed28 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.822+0000 7fba27400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.823+0000 7fba27400700 1 -- 172.21.15.43:0/3935783073 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba340c2020 con 0x55d064db9e40 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.823+0000 7fba27400700 1 -- 172.21.15.43:0/3935783073 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fba340e20b0 con 0x55d064db9e40 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.823+0000 7fba568e6b80 1 -- 172.21.15.43:0/3935783073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba2c010e40 con 0x55d064db9e40 2024-05-27T07:15:03.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.823+0000 7fba27400700 1 -- 172.21.15.43:0/3935783073 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fba340e22b0 con 0x55d064db9e40 2024-05-27T07:15:03.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.823+0000 7fba568e6b80 10 client.15582 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:03.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.823+0000 7fba568e6b80 1 -- 172.21.15.43:0/3935783073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d064e27060 con 0x55d064db9e40 2024-05-27T07:15:03.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.823+0000 7fba27400700 1 -- 172.21.15.43:0/3935783073 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba2c010e40 con 0x55d064db9e40 2024-05-27T07:15:03.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.824+0000 7fba27400700 1 -- 172.21.15.43:0/3935783073 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fba340a83c0 con 0x55d064db9e40 2024-05-27T07:15:03.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.824+0000 7fba568e6b80 10 client.15582 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:03.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.824+0000 7fba568e6b80 10 client.15582 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:15:03.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.824+0000 7fba568e6b80 20 client.15582 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:03.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.824+0000 7fba568e6b80 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d064f52c90 0x55d064f55110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.824+0000 7fba568e6b80 4 client.15582 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:03.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.824+0000 7fba568e6b80 1 -- 172.21.15.43:0/3935783073 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d064e68d90 con 0x55d064f52c90 2024-05-27T07:15:03.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.824+0000 7fba44a00700 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d064f52c90 0x55d064f55110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.825+0000 7fba44a00700 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d064f52c90 0x55d064f55110 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=478aa21c1f088013 server_cookie=c3885427487ba154 in_seq=0 out_seq=0 2024-05-27T07:15:03.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.825+0000 7fba27400700 10 client.15582 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:15:03.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.827+0000 7fba27400700 1 -- 172.21.15.43:0/3935783073 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d064e68d90 con 0x55d064f52c90 2024-05-27T07:15:03.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.827+0000 7fba27400700 10 client.15582 handle_command_reply: tid=0 2024-05-27T07:15:03.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 client.15582 shutdown 2024-05-27T07:15:03.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 -- 172.21.15.43:0/3935783073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064db9e40 msgr2=0x55d064f40200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:03.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064db9e40 0x55d064f40200 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fba34009510 tx=0x7fba340a96f0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 -- 172.21.15.43:0/3935783073 shutdown_connections 2024-05-27T07:15:03.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064f40c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064db9e40 0x55d064f40200 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d064f52c90 0x55d064f55110 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 --2- 172.21.15.43:0/3935783073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064ea59f0 0x55d064f40740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba27400700 0 client.15582 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:15:03.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 -- 172.21.15.43:0/3935783073 >> 172.21.15.43:0/3935783073 conn(0x55d064e53390 msgr2=0x55d064f39750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:03.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 -- 172.21.15.43:0/3935783073 shutdown_connections 2024-05-27T07:15:03.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 1 -- 172.21.15.43:0/3935783073 wait complete. 2024-05-27T07:15:03.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.828+0000 7fba568e6b80 20 client.15582 trim_cache size 0 max 16384 2024-05-27T07:15:03.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.858+0000 7fba568e6b80 1 Processor -- start 2024-05-27T07:15:03.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.859+0000 7fba568e6b80 1 -- start start 2024-05-27T07:15:03.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.859+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064db9e40 0x55d064f3f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.859+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f34740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.859+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064f4eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.859+0000 7fba568e6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fba401b2440 con 0x55d064ea59f0 2024-05-27T07:15:03.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.859+0000 7fba568e6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fba401b3f80 con 0x55d064db9e40 2024-05-27T07:15:03.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.859+0000 7fba568e6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fba401b2b10 con 0x55d064f432c0 2024-05-27T07:15:03.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.859+0000 7fba44a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f34740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.859+0000 7fba44a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f34740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45586/0 (socket says 172.21.15.43:45586) 2024-05-27T07:15:03.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.859+0000 7fba44a00700 1 -- 172.21.15.43:0/1274853769 learned_addr learned my addr 172.21.15.43:0/1274853769 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:03.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.860+0000 7fba45e00700 1 --2- 172.21.15.43:0/1274853769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064f4eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.860+0000 7fba45400700 1 --2- 172.21.15.43:0/1274853769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064db9e40 0x55d064f3f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.860+0000 7fba44a00700 1 -- 172.21.15.43:0/1274853769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 msgr2=0x55d064f4eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:03.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.860+0000 7fba44a00700 1 --2- 172.21.15.43:0/1274853769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064f4eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.860+0000 7fba44a00700 1 -- 172.21.15.43:0/1274853769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064db9e40 msgr2=0x55d064f3f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:03.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.860+0000 7fba44a00700 1 --2- 172.21.15.43:0/1274853769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064db9e40 0x55d064f3f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.860+0000 7fba44a00700 1 -- 172.21.15.43:0/1274853769 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d064f40ff0 con 0x55d064ea59f0 2024-05-27T07:15:03.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.860+0000 7fba44a00700 1 --2- 172.21.15.43:0/1274853769 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f34740 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0106b0 tx=0x7fba2c00bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ba5bab6ec8a1a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:03.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.861+0000 7fba32000700 1 -- 172.21.15.43:0/1274853769 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c011440 con 0x55d064ea59f0 2024-05-27T07:15:03.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.861+0000 7fba32000700 1 -- 172.21.15.43:0/1274853769 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba2c011a70 con 0x55d064ea59f0 2024-05-27T07:15:03.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.861+0000 7fba32000700 1 -- 172.21.15.43:0/1274853769 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c027940 con 0x55d064ea59f0 2024-05-27T07:15:03.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.862+0000 7fba568e6b80 1 -- 172.21.15.43:0/1274853769 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 msgr2=0x55d064f34740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:03.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.862+0000 7fba568e6b80 1 --2- 172.21.15.43:0/1274853769 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f34740 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0106b0 tx=0x7fba2c00bfb0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.862+0000 7fba568e6b80 1 -- 172.21.15.43:0/1274853769 shutdown_connections 2024-05-27T07:15:03.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.862+0000 7fba568e6b80 1 --2- 172.21.15.43:0/1274853769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064f4eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.862+0000 7fba568e6b80 1 --2- 172.21.15.43:0/1274853769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064db9e40 0x55d064f3f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.862+0000 7fba568e6b80 1 --2- 172.21.15.43:0/1274853769 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f34740 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.862+0000 7fba568e6b80 1 -- 172.21.15.43:0/1274853769 >> 172.21.15.43:0/1274853769 conn(0x55d064e53390 msgr2=0x55d064f3b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:03.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.862+0000 7fba568e6b80 1 -- 172.21.15.43:0/1274853769 shutdown_connections 2024-05-27T07:15:03.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.863+0000 7fba568e6b80 1 -- 172.21.15.43:0/1274853769 wait complete. 2024-05-27T07:15:03.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.863+0000 7fba568e6b80 1 -- start start 2024-05-27T07:15:03.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.863+0000 7fba568e6b80 1 Processor -- start 2024-05-27T07:15:03.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064db9e40 0x55d064f46830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f46d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba568e6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064fb5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba568e6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fba401b38b0 con 0x55d064ea59f0 2024-05-27T07:15:03.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba568e6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fba401b31e0 con 0x55d064db9e40 2024-05-27T07:15:03.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba568e6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fba401b03c0 con 0x55d064f432c0 2024-05-27T07:15:03.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba568e6b80 10 client.0 _ll_register_callbacks cb 0x55d064e00c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:03.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba44a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f46d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba44a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f46d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45598/0 (socket says 172.21.15.43:45598) 2024-05-27T07:15:03.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba44a00700 1 -- 172.21.15.43:0/163959441 learned_addr learned my addr 172.21.15.43:0/163959441 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:03.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba45400700 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064db9e40 0x55d064f46830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.864+0000 7fba45e00700 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064fb5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba44a00700 1 -- 172.21.15.43:0/163959441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 msgr2=0x55d064fb5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:03.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba44a00700 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064fb5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba44a00700 1 -- 172.21.15.43:0/163959441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064db9e40 msgr2=0x55d064f46830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:03.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba44a00700 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064db9e40 0x55d064f46830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba44a00700 1 -- 172.21.15.43:0/163959441 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d064fb5d40 con 0x55d064ea59f0 2024-05-27T07:15:03.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba45e00700 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064fb5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:15:03.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba44a00700 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f46d70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00bbf0 tx=0x7fba2c027420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b7c7903c40113b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:03.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba27400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:03.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba27400700 1 -- 172.21.15.43:0/163959441 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c011e20 con 0x55d064ea59f0 2024-05-27T07:15:03.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba27400700 1 -- 172.21.15.43:0/163959441 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fba2c011440 con 0x55d064ea59f0 2024-05-27T07:15:03.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.865+0000 7fba568e6b80 1 -- 172.21.15.43:0/163959441 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba2c011a70 con 0x55d064ea59f0 2024-05-27T07:15:03.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.866+0000 7fba27400700 1 -- 172.21.15.43:0/163959441 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fba2c044050 con 0x55d064ea59f0 2024-05-27T07:15:03.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.866+0000 7fba568e6b80 10 client.15588 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:03.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.866+0000 7fba568e6b80 1 -- 172.21.15.43:0/163959441 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d064fbd050 con 0x55d064ea59f0 2024-05-27T07:15:03.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.866+0000 7fba27400700 1 -- 172.21.15.43:0/163959441 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba2c011a70 con 0x55d064ea59f0 2024-05-27T07:15:03.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.866+0000 7fba27400700 1 -- 172.21.15.43:0/163959441 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fba2c03b470 con 0x55d064ea59f0 2024-05-27T07:15:03.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.867+0000 7fba568e6b80 10 client.15588 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:03.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.867+0000 7fba568e6b80 10 client.15588 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:15:03.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.867+0000 7fba568e6b80 20 client.15588 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:03.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.867+0000 7fba568e6b80 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d064fbd4b0 0x55d064fbf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:03.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.867+0000 7fba568e6b80 4 client.15588 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:15:03.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.867+0000 7fba568e6b80 1 -- 172.21.15.43:0/163959441 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d064e62570 con 0x55d064fbd4b0 2024-05-27T07:15:03.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.867+0000 7fba45400700 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d064fbd4b0 0x55d064fbf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:03.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.868+0000 7fba45400700 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d064fbd4b0 0x55d064fbf8d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=784b3975c427c1bf server_cookie=d64fbd5a40f577fb in_seq=0 out_seq=0 2024-05-27T07:15:03.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.868+0000 7fba27400700 10 client.15588 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:15:03.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba27400700 1 -- 172.21.15.43:0/163959441 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d064e62570 con 0x55d064fbd4b0 2024-05-27T07:15:03.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba27400700 10 client.15588 handle_command_reply: tid=0 2024-05-27T07:15:03.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba568e6b80 1 client.15588 shutdown 2024-05-27T07:15:03.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba568e6b80 1 -- 172.21.15.43:0/163959441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 msgr2=0x55d064f46d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:03.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba568e6b80 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f46d70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00bbf0 tx=0x7fba2c027420 comp rx=0 tx=0).stop 2024-05-27T07:15:03.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba568e6b80 1 -- 172.21.15.43:0/163959441 shutdown_connections 2024-05-27T07:15:03.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba568e6b80 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d064f432c0 0x55d064fb5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba568e6b80 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d064fbd4b0 0x55d064fbf8d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba568e6b80 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d064db9e40 0x55d064f46830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba568e6b80 1 --2- 172.21.15.43:0/163959441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d064ea59f0 0x55d064f46d70 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba27400700 0 client.15588 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:15:03.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.869+0000 7fba568e6b80 1 -- 172.21.15.43:0/163959441 >> 172.21.15.43:0/163959441 conn(0x55d064e53390 msgr2=0x55d064f3b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:03.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.870+0000 7fba568e6b80 1 -- 172.21.15.43:0/163959441 shutdown_connections 2024-05-27T07:15:03.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.870+0000 7fba568e6b80 1 -- 172.21.15.43:0/163959441 wait complete. 2024-05-27T07:15:03.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.870+0000 7fba568e6b80 20 client.15588 trim_cache size 0 max 16384 2024-05-27T07:15:03.862 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:15:03.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 -- 172.21.15.43:0/3746734664 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba280b2b10 msgr2=0x7fba280b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:03.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba280b2b10 0x7fba280b5020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fba38009750 tx=0x7fba38009f50 comp rx=0 tx=0).stop 2024-05-27T07:15:03.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 -- 172.21.15.43:0/3746734664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba401043f0 msgr2=0x7fba401a29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:03.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba401043f0 0x7fba401a29c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fba340060b0 tx=0x7fba3400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 -- 172.21.15.43:0/3746734664 shutdown_connections 2024-05-27T07:15:03.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba280b2b10 0x7fba280b5020 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba40109670 0x7fba4019cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba401043f0 0x7fba401a29c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 --2- 172.21.15.43:0/3746734664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba40108d20 0x7fba401a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:03.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 -- 172.21.15.43:0/3746734664 >> 172.21.15.43:0/3746734664 conn(0x7fba40079d00 msgr2=0x7fba4010f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:03.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 -- 172.21.15.43:0/3746734664 shutdown_connections 2024-05-27T07:15:03.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:03.871+0000 7fba46800700 1 -- 172.21.15.43:0/3746734664 wait complete. 2024-05-27T07:15:04.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:03 smithi132 ceph-mon[35283]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:15:04.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:03 smithi132 ceph-mon[35283]: mgrmap e25: x(active, since 68s), standbys: z, y 2024-05-27T07:15:04.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:03 smithi179 ceph-mon[35888]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-05-27T07:15:04.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:03 smithi179 ceph-mon[35888]: mgrmap e25: x(active, since 68s), standbys: z, y 2024-05-27T07:15:05.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:15:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:05.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:05 smithi043 ceph-mon[31531]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:06.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:05 smithi132 ceph-mon[35283]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:06.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:05 smithi179 ceph-mon[35888]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:07.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.948+0000 7f502be00700 1 -- 172.21.15.43:0/2438637814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f502c108d20 msgr2=0x7f502c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.948+0000 7f502be00700 1 --2- 172.21.15.43:0/2438637814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f502c108d20 0x7f502c109130 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f50180099c0 tx=0x7f5018009cd0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.948+0000 7f502be00700 1 -- 172.21.15.43:0/2438637814 shutdown_connections 2024-05-27T07:15:07.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.948+0000 7f502be00700 1 --2- 172.21.15.43:0/2438637814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502c109670 0x7f502c110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.948+0000 7f502be00700 1 --2- 172.21.15.43:0/2438637814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f502c1043f0 0x7f502c104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.948+0000 7f502be00700 1 --2- 172.21.15.43:0/2438637814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f502c108d20 0x7f502c109130 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.948+0000 7f502be00700 1 -- 172.21.15.43:0/2438637814 >> 172.21.15.43:0/2438637814 conn(0x7f502c079d00 msgr2=0x7f502c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:07.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.948+0000 7f502be00700 1 -- 172.21.15.43:0/2438637814 shutdown_connections 2024-05-27T07:15:07.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.948+0000 7f502be00700 1 -- 172.21.15.43:0/2438637814 wait complete. 2024-05-27T07:15:07.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.950+0000 7f502be00700 1 Processor -- start 2024-05-27T07:15:07.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.950+0000 7f502be00700 1 -- start start 2024-05-27T07:15:07.942 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:07 smithi043 ceph-mon[31531]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:15:07.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.951+0000 7f502be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f502c1043f0 0x7f502c075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.951+0000 7f502be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502c108d20 0x7f502c075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.951+0000 7f502be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f502c109670 0x7f502c076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.951+0000 7f502be00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f502c1141f0 con 0x7f502c109670 2024-05-27T07:15:07.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.951+0000 7f502be00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f502c114070 con 0x7f502c108d20 2024-05-27T07:15:07.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.951+0000 7f502be00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f502c114370 con 0x7f502c1043f0 2024-05-27T07:15:07.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.951+0000 7f502b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f502c109670 0x7f502c076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.951+0000 7f502b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f502c109670 0x7f502c076530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45348/0 (socket says 172.21.15.43:45348) 2024-05-27T07:15:07.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.951+0000 7f502b400700 1 -- 172.21.15.43:0/3454030512 learned_addr learned my addr 172.21.15.43:0/3454030512 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:07.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.951+0000 7f502aa00700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f502c1043f0 0x7f502c075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.952+0000 7f502a000700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502c108d20 0x7f502c075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.952+0000 7f502b400700 1 -- 172.21.15.43:0/3454030512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f502c1043f0 msgr2=0x7f502c075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.952+0000 7f502b400700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f502c1043f0 0x7f502c075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.952+0000 7f502b400700 1 -- 172.21.15.43:0/3454030512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502c108d20 msgr2=0x7f502c075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.952+0000 7f502b400700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502c108d20 0x7f502c075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.952+0000 7f502b400700 1 -- 172.21.15.43:0/3454030512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502c1a9040 con 0x7f502c109670 2024-05-27T07:15:07.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.953+0000 7f502b400700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f502c109670 0x7f502c076530 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f502400b970 tx=0x7f502400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:07.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.953+0000 7f5023400700 1 -- 172.21.15.43:0/3454030512 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502400d860 con 0x7f502c109670 2024-05-27T07:15:07.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.953+0000 7f5023400700 1 -- 172.21.15.43:0/3454030512 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f502400e430 con 0x7f502c109670 2024-05-27T07:15:07.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.953+0000 7f5023400700 1 -- 172.21.15.43:0/3454030512 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5024017660 con 0x7f502c109670 2024-05-27T07:15:07.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.953+0000 7f502be00700 1 -- 172.21.15.43:0/3454030512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f502c1a9330 con 0x7f502c109670 2024-05-27T07:15:07.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.953+0000 7f502be00700 1 -- 172.21.15.43:0/3454030512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f502c1a97f0 con 0x7f502c109670 2024-05-27T07:15:07.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.957+0000 7f5023400700 1 -- 172.21.15.43:0/3454030512 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7f502400da00 con 0x7f502c109670 2024-05-27T07:15:07.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.958+0000 7f5040455b80 1 Processor -- start 2024-05-27T07:15:07.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.959+0000 7f5040455b80 1 -- start start 2024-05-27T07:15:07.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.959+0000 7f5040455b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555766f22e40 0x5557670a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.959+0000 7f5040455b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1410 0x5557670abad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.959+0000 7f5040455b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1820 0x5557670ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.959+0000 7f5040455b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555766fcfe40 con 0x5557670a1410 2024-05-27T07:15:07.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.959+0000 7f5040455b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555766fbf9e0 con 0x5557670a1820 2024-05-27T07:15:07.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.959+0000 7f5040455b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555766fc3a20 con 0x555766f22e40 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f5023400700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50100b2b10 0x7f50100b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f502aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1410 0x5557670abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f502aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1410 0x5557670abad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45352/0 (socket says 172.21.15.43:45352) 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f502aa00700 1 -- 172.21.15.43:0/2753720917 learned_addr learned my addr 172.21.15.43:0/2753720917 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f5023400700 1 -- 172.21.15.43:0/3454030512 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f50240e28b0 con 0x7f502c109670 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f502b400700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50100b2b10 0x7f50100b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f502aa00700 1 -- 172.21.15.43:0/2753720917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555766f22e40 msgr2=0x5557670a0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f502aa00700 1 --2- 172.21.15.43:0/2753720917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555766f22e40 0x5557670a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f502aa00700 1 -- 172.21.15.43:0/2753720917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1820 msgr2=0x5557670ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f502aa00700 1 --2- 172.21.15.43:0/2753720917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1820 0x5557670ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.960+0000 7f502aa00700 1 -- 172.21.15.43:0/2753720917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555766f88420 con 0x5557670a1410 2024-05-27T07:15:07.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.961+0000 7f502aa00700 1 --2- 172.21.15.43:0/2753720917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1410 0x5557670abad0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f501c00bd70 tx=0x7f501c0106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b190b3b6dfcd5b3a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:07.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.961+0000 7f502b400700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50100b2b10 0x7f50100b5020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f50240dfb30 tx=0x7f502400ebb0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:07.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.961+0000 7f5017e00700 1 -- 172.21.15.43:0/2753720917 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c01c480 con 0x5557670a1410 2024-05-27T07:15:07.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.961+0000 7f5017e00700 1 -- 172.21.15.43:0/2753720917 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f501c01cab0 con 0x5557670a1410 2024-05-27T07:15:07.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.961+0000 7f5017e00700 1 -- 172.21.15.43:0/2753720917 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c011820 con 0x5557670a1410 2024-05-27T07:15:07.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.962+0000 7f5040455b80 1 -- 172.21.15.43:0/2753720917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1410 msgr2=0x5557670abad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.962+0000 7f5040455b80 1 --2- 172.21.15.43:0/2753720917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1410 0x5557670abad0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f501c00bd70 tx=0x7f501c0106b0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.962+0000 7f5040455b80 1 -- 172.21.15.43:0/2753720917 shutdown_connections 2024-05-27T07:15:07.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.962+0000 7f5040455b80 1 --2- 172.21.15.43:0/2753720917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1820 0x5557670ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.962+0000 7f5040455b80 1 --2- 172.21.15.43:0/2753720917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555766f22e40 0x5557670a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.962+0000 7f5040455b80 1 --2- 172.21.15.43:0/2753720917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1410 0x5557670abad0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.962+0000 7f5040455b80 1 -- 172.21.15.43:0/2753720917 >> 172.21.15.43:0/2753720917 conn(0x555766f82fc0 msgr2=0x55576709cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:07.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.962+0000 7f5040455b80 1 -- 172.21.15.43:0/2753720917 shutdown_connections 2024-05-27T07:15:07.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.963+0000 7f5040455b80 1 -- 172.21.15.43:0/2753720917 wait complete. 2024-05-27T07:15:07.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.963+0000 7f5040455b80 1 -- start start 2024-05-27T07:15:07.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.964+0000 7f5040455b80 1 Processor -- start 2024-05-27T07:15:07.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.964+0000 7f5040455b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555766f22e40 0x5557670ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.964+0000 7f5040455b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1410 0x5557670abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.964+0000 7f5040455b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670ac070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.964+0000 7f5040455b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555766ec5fa0 con 0x5557670a1820 2024-05-27T07:15:07.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.964+0000 7f5040455b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555766f60760 con 0x555766f22e40 2024-05-27T07:15:07.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.964+0000 7f5040455b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5557670ae990 con 0x5557670a1410 2024-05-27T07:15:07.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.964+0000 7f5040455b80 10 client.0 _ll_register_callbacks cb 0x555766f69c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670ac070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45368/0 (socket says 172.21.15.43:45368) 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502a000700 1 -- 172.21.15.43:0/3131399211 learned_addr learned my addr 172.21.15.43:0/3131399211 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502a000700 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555766f22e40 0x5557670ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502aa00700 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1410 0x5557670abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502a000700 1 -- 172.21.15.43:0/3131399211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1410 msgr2=0x5557670abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502a000700 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1410 0x5557670abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502a000700 1 -- 172.21.15.43:0/3131399211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555766f22e40 msgr2=0x5557670ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502a000700 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555766f22e40 0x5557670ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502a000700 1 -- 172.21.15.43:0/3131399211 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555766f0f8d0 con 0x5557670a1820 2024-05-27T07:15:07.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.965+0000 7f502aa00700 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1410 0x5557670abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:07.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.966+0000 7f5014c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.966+0000 7f502a000700 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670ac070 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f502c06c3b0 tx=0x7f501801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b582cc0a665277f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.966+0000 7f5014c00700 1 -- 172.21.15.43:0/3131399211 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50180050d0 con 0x5557670a1820 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.966+0000 7f5014c00700 1 -- 172.21.15.43:0/3131399211 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f5018005270 con 0x5557670a1820 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.966+0000 7f5040455b80 1 -- 172.21.15.43:0/3131399211 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f501c01cab0 con 0x5557670a1820 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.966+0000 7f5014c00700 1 -- 172.21.15.43:0/3131399211 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5018005470 con 0x5557670a1820 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.966+0000 7f5040455b80 10 client.15600 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.966+0000 7f5040455b80 1 -- 172.21.15.43:0/3131399211 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555766f90060 con 0x5557670a1820 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.967+0000 7f5014c00700 1 -- 172.21.15.43:0/3131399211 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f501c01cab0 con 0x5557670a1820 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.967+0000 7f5014c00700 1 -- 172.21.15.43:0/3131399211 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f50180039f0 con 0x5557670a1820 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.967+0000 7f5040455b80 10 client.15600 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.967+0000 7f5040455b80 10 client.15600 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.967+0000 7f5040455b80 20 client.15600 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.967+0000 7f5040455b80 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5557670bb8a0 0x5557670bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.967+0000 7f5040455b80 4 client.15600 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:07.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.967+0000 7f5040455b80 1 -- 172.21.15.43:0/3131399211 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555766fd1d90 con 0x5557670bb8a0 2024-05-27T07:15:07.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.968+0000 7f502aa00700 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5557670bb8a0 0x5557670bdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.969+0000 7f502aa00700 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5557670bb8a0 0x5557670bdd20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cd5ac961514e7dd1 server_cookie=1413bdcee7d62115 in_seq=0 out_seq=0 2024-05-27T07:15:07.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.969+0000 7f5014c00700 10 client.15600 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:15:07.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5014c00700 1 -- 172.21.15.43:0/3131399211 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555766fd1d90 con 0x5557670bb8a0 2024-05-27T07:15:07.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5014c00700 10 client.15600 handle_command_reply: tid=0 2024-05-27T07:15:07.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5040455b80 1 client.15600 shutdown 2024-05-27T07:15:07.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5040455b80 1 -- 172.21.15.43:0/3131399211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 msgr2=0x5557670ac070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5040455b80 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670ac070 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f502c06c3b0 tx=0x7f501801a040 comp rx=0 tx=0).stop 2024-05-27T07:15:07.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5040455b80 1 -- 172.21.15.43:0/3131399211 shutdown_connections 2024-05-27T07:15:07.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5040455b80 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5557670bb8a0 0x5557670bdd20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5040455b80 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670ac070 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5040455b80 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555766f22e40 0x5557670ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5040455b80 1 --2- 172.21.15.43:0/3131399211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1410 0x5557670abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5014c00700 0 client.15600 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:15:07.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.971+0000 7f5040455b80 1 -- 172.21.15.43:0/3131399211 >> 172.21.15.43:0/3131399211 conn(0x555766f82fc0 msgr2=0x55576709bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:07.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.972+0000 7f5040455b80 1 -- 172.21.15.43:0/3131399211 shutdown_connections 2024-05-27T07:15:07.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.972+0000 7f5040455b80 1 -- 172.21.15.43:0/3131399211 wait complete. 2024-05-27T07:15:07.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.972+0000 7f5040455b80 20 client.15600 trim_cache size 0 max 16384 2024-05-27T07:15:07.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.989+0000 7f5040455b80 1 Processor -- start 2024-05-27T07:15:07.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.989+0000 7f5040455b80 1 -- start start 2024-05-27T07:15:07.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.989+0000 7f5040455b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555766f22e40 0x55576709ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.989+0000 7f5040455b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1410 0x5557670ada90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.989+0000 7f5040455b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.989+0000 7f5040455b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f502c1b24f0 con 0x5557670a1820 2024-05-27T07:15:07.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.989+0000 7f5040455b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f502c1b4030 con 0x5557670a1410 2024-05-27T07:15:07.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.989+0000 7f5040455b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f502c1b2bc0 con 0x555766f22e40 2024-05-27T07:15:07.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670a6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45380/0 (socket says 172.21.15.43:45380) 2024-05-27T07:15:07.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502a000700 1 -- 172.21.15.43:0/1512011587 learned_addr learned my addr 172.21.15.43:0/1512011587 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:07.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502aa00700 1 --2- 172.21.15.43:0/1512011587 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1410 0x5557670ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502a000700 1 --2- 172.21.15.43:0/1512011587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555766f22e40 0x55576709ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502a000700 1 -- 172.21.15.43:0/1512011587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555766f22e40 msgr2=0x55576709ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:07.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502a000700 1 --2- 172.21.15.43:0/1512011587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555766f22e40 0x55576709ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502a000700 1 -- 172.21.15.43:0/1512011587 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1410 msgr2=0x5557670ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:07.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502a000700 1 --2- 172.21.15.43:0/1512011587 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1410 0x5557670ada90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502a000700 1 -- 172.21.15.43:0/1512011587 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557670b7a30 con 0x5557670a1820 2024-05-27T07:15:07.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.990+0000 7f502a000700 1 --2- 172.21.15.43:0/1512011587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670a6dd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x555767096f90 tx=0x7f5018029040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab140fb9b7db34cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:07.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.991+0000 7f5017400700 1 -- 172.21.15.43:0/1512011587 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50180052d0 con 0x5557670a1820 2024-05-27T07:15:07.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.991+0000 7f5017400700 1 -- 172.21.15.43:0/1512011587 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5018029540 con 0x5557670a1820 2024-05-27T07:15:07.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.991+0000 7f5017400700 1 -- 172.21.15.43:0/1512011587 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501800fa40 con 0x5557670a1820 2024-05-27T07:15:07.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.992+0000 7f5040455b80 1 -- 172.21.15.43:0/1512011587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 msgr2=0x5557670a6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.992+0000 7f5040455b80 1 --2- 172.21.15.43:0/1512011587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670a6dd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x555767096f90 tx=0x7f5018029040 comp rx=0 tx=0).stop 2024-05-27T07:15:07.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.992+0000 7f5040455b80 1 -- 172.21.15.43:0/1512011587 shutdown_connections 2024-05-27T07:15:07.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.992+0000 7f5040455b80 1 --2- 172.21.15.43:0/1512011587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5557670a1820 0x5557670a6dd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.992+0000 7f5040455b80 1 --2- 172.21.15.43:0/1512011587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555766f22e40 0x55576709ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.992+0000 7f5040455b80 1 --2- 172.21.15.43:0/1512011587 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1410 0x5557670ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.992+0000 7f5040455b80 1 -- 172.21.15.43:0/1512011587 >> 172.21.15.43:0/1512011587 conn(0x555766f82fc0 msgr2=0x55576709c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:07.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.992+0000 7f5040455b80 1 -- 172.21.15.43:0/1512011587 shutdown_connections 2024-05-27T07:15:07.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.992+0000 7f5040455b80 1 -- 172.21.15.43:0/1512011587 wait complete. 2024-05-27T07:15:07.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.993+0000 7f5040455b80 1 -- start start 2024-05-27T07:15:07.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.993+0000 7f5040455b80 1 Processor -- start 2024-05-27T07:15:07.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.994+0000 7f5040455b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555766f22e40 0x5557670b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.994+0000 7f5040455b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1410 0x5557670af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.994+0000 7f5040455b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1820 0x5557670afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.994+0000 7f5040455b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f502c1b3960 con 0x555766f22e40 2024-05-27T07:15:07.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.994+0000 7f5040455b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f502c1b3290 con 0x5557670a1410 2024-05-27T07:15:07.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.994+0000 7f5040455b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f502c1b0470 con 0x5557670a1820 2024-05-27T07:15:07.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.994+0000 7f5040455b80 10 client.0 _ll_register_callbacks cb 0x555766f69c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:07.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.995+0000 7f502a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555766f22e40 0x5557670b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.995+0000 7f502a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555766f22e40 0x5557670b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45396/0 (socket says 172.21.15.43:45396) 2024-05-27T07:15:07.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.995+0000 7f502a000700 1 -- 172.21.15.43:0/3344299080 learned_addr learned my addr 172.21.15.43:0/3344299080 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:07.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.995+0000 7f502aa00700 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1410 0x5557670af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.995+0000 7f502a000700 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1820 0x5557670afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.995+0000 7f502a000700 1 -- 172.21.15.43:0/3344299080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1820 msgr2=0x5557670afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:07.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.995+0000 7f502a000700 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1820 0x5557670afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.995+0000 7f502a000700 1 -- 172.21.15.43:0/3344299080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1410 msgr2=0x5557670af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:07.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.995+0000 7f502a000700 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1410 0x5557670af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.995+0000 7f502a000700 1 -- 172.21.15.43:0/3344299080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557670b00d0 con 0x555766f22e40 2024-05-27T07:15:07.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.996+0000 7f502a000700 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555766f22e40 0x5557670b41f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5018009d10 tx=0x7f5018005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafa37d016e7a759 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:07.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.996+0000 7f5014c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:07.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.996+0000 7f5014c00700 1 -- 172.21.15.43:0/3344299080 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50180292b0 con 0x555766f22e40 2024-05-27T07:15:07.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.996+0000 7f5014c00700 1 -- 172.21.15.43:0/3344299080 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f501800f7e0 con 0x555766f22e40 2024-05-27T07:15:07.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.996+0000 7f5040455b80 1 -- 172.21.15.43:0/3344299080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5018029540 con 0x555766f22e40 2024-05-27T07:15:07.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.997+0000 7f5014c00700 1 -- 172.21.15.43:0/3344299080 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f501800f9e0 con 0x555766f22e40 2024-05-27T07:15:07.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.997+0000 7f5040455b80 10 client.15606 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:07.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.997+0000 7f5040455b80 1 -- 172.21.15.43:0/3344299080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555767126180 con 0x555766f22e40 2024-05-27T07:15:07.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.997+0000 7f5014c00700 1 -- 172.21.15.43:0/3344299080 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5018029540 con 0x555766f22e40 2024-05-27T07:15:07.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.997+0000 7f5014c00700 1 -- 172.21.15.43:0/3344299080 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5018036450 con 0x555766f22e40 2024-05-27T07:15:07.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.998+0000 7f5040455b80 10 client.15606 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:07.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.998+0000 7f5040455b80 10 client.15606 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:15:07.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.998+0000 7f5040455b80 20 client.15606 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:07.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.998+0000 7f5040455b80 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555767126590 0x5557671289b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:07.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.998+0000 7f5040455b80 4 client.15606 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:15:07.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.998+0000 7f5040455b80 1 -- 172.21.15.43:0/3344299080 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555766fcb570 con 0x555767126590 2024-05-27T07:15:07.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:07.999+0000 7f502aa00700 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555767126590 0x5557671289b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.000+0000 7f502aa00700 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555767126590 0x5557671289b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=50872ccff302bd53 server_cookie=16c36c94950db96d in_seq=0 out_seq=0 2024-05-27T07:15:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.000+0000 7f5014c00700 10 client.15606 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:15:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.000+0000 7f5014c00700 1 -- 172.21.15.43:0/3344299080 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555766fcb570 con 0x555767126590 2024-05-27T07:15:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.000+0000 7f5014c00700 10 client.15606 handle_command_reply: tid=0 2024-05-27T07:15:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.000+0000 7f5040455b80 1 client.15606 shutdown 2024-05-27T07:15:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.000+0000 7f5040455b80 1 -- 172.21.15.43:0/3344299080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555766f22e40 msgr2=0x5557670b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.000+0000 7f5040455b80 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555766f22e40 0x5557670b41f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5018009d10 tx=0x7f5018005ce0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.000+0000 7f5040455b80 1 -- 172.21.15.43:0/3344299080 shutdown_connections 2024-05-27T07:15:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.000+0000 7f5040455b80 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555767126590 0x5557671289b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.001+0000 7f5040455b80 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5557670a1820 0x5557670afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.001+0000 7f5040455b80 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555766f22e40 0x5557670b41f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.001+0000 7f5040455b80 1 --2- 172.21.15.43:0/3344299080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5557670a1410 0x5557670af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.001+0000 7f5040455b80 1 -- 172.21.15.43:0/3344299080 >> 172.21.15.43:0/3344299080 conn(0x555766f82fc0 msgr2=0x55576709c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.001+0000 7f5014c00700 0 client.15606 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:15:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.001+0000 7f5040455b80 1 -- 172.21.15.43:0/3344299080 shutdown_connections 2024-05-27T07:15:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.001+0000 7f5040455b80 1 -- 172.21.15.43:0/3344299080 wait complete. 2024-05-27T07:15:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.001+0000 7f5040455b80 20 client.15606 trim_cache size 0 max 16384 2024-05-27T07:15:07.994 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:15:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 -- 172.21.15.43:0/3454030512 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50100b2b10 msgr2=0x7f50100b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50100b2b10 0x7f50100b5020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f50240dfb30 tx=0x7f502400ebb0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 -- 172.21.15.43:0/3454030512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f502c109670 msgr2=0x7f502c076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f502c109670 0x7f502c076530 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f502400b970 tx=0x7f502400bc80 comp rx=0 tx=0).stop 2024-05-27T07:15:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 -- 172.21.15.43:0/3454030512 shutdown_connections 2024-05-27T07:15:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50100b2b10 0x7f50100b5020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f502c109670 0x7f502c076530 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f502c1043f0 0x7f502c075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 --2- 172.21.15.43:0/3454030512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f502c108d20 0x7f502c075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 -- 172.21.15.43:0/3454030512 >> 172.21.15.43:0/3454030512 conn(0x7f502c079d00 msgr2=0x7f502c101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 -- 172.21.15.43:0/3454030512 shutdown_connections 2024-05-27T07:15:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:08.002+0000 7f502be00700 1 -- 172.21.15.43:0/3454030512 wait complete. 2024-05-27T07:15:08.011 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:07 smithi179 ceph-mon[35888]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:15:08.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:07 smithi132 ceph-mon[35283]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:15:08.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:15:09.457 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:09.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:09 smithi043 ceph-mon[31531]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:15:10.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:09 smithi132 ceph-mon[35283]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:15:10.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:09 smithi179 ceph-mon[35888]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:15:12.082 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:11 smithi043 ceph-mon[31531]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:15:12.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:11 smithi132 ceph-mon[35283]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:15:12.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:11 smithi179 ceph-mon[35888]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:15:13.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:13 smithi043 ceph-mon[31531]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:15:13.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:13 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.036+0000 7f0be9600700 1 -- 172.21.15.43:0/1775696190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0be410a920 msgr2=0x7f0be410ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.036+0000 7f0be9600700 1 --2- 172.21.15.43:0/1775696190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0be410a920 0x7f0be410ad30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00098f0 tx=0x7f0bd0009c00 comp rx=0 tx=0).stop 2024-05-27T07:15:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.036+0000 7f0be9600700 1 -- 172.21.15.43:0/1775696190 shutdown_connections 2024-05-27T07:15:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.036+0000 7f0be9600700 1 --2- 172.21.15.43:0/1775696190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0be410b270 0x7f0be41198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.036+0000 7f0be9600700 1 --2- 172.21.15.43:0/1775696190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0be4111bc0 0x7f0be4111fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.036+0000 7f0be9600700 1 --2- 172.21.15.43:0/1775696190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0be410a920 0x7f0be410ad30 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.036+0000 7f0be9600700 1 -- 172.21.15.43:0/1775696190 >> 172.21.15.43:0/1775696190 conn(0x7f0be406f4c0 msgr2=0x7f0be406f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.036+0000 7f0be9600700 1 -- 172.21.15.43:0/1775696190 shutdown_connections 2024-05-27T07:15:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.036+0000 7f0be9600700 1 -- 172.21.15.43:0/1775696190 wait complete. 2024-05-27T07:15:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.038+0000 7f0be9600700 1 Processor -- start 2024-05-27T07:15:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.038+0000 7f0be9600700 1 -- start start 2024-05-27T07:15:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.039+0000 7f0be9600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0be410a920 0x7f0be41a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.039+0000 7f0be9600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0be410b270 0x7f0be41a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.039+0000 7f0be9600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0be4111bc0 0x7f0be41a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.039+0000 7f0be9600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0be411a980 con 0x7f0be410a920 2024-05-27T07:15:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.039+0000 7f0be9600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0be411a800 con 0x7f0be410b270 2024-05-27T07:15:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.039+0000 7f0be9600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0be411ab00 con 0x7f0be4111bc0 2024-05-27T07:15:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.039+0000 7f0be3400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0be4111bc0 0x7f0be41a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.040+0000 7f0be3400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0be4111bc0 0x7f0be41a5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:39990/0 (socket says 172.21.15.43:39990) 2024-05-27T07:15:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.040+0000 7f0be3400700 1 -- 172.21.15.43:0/545959695 learned_addr learned my addr 172.21.15.43:0/545959695 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.040+0000 7f0be2000700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0be410b270 0x7f0be41a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.041+0000 7f0be3400700 1 -- 172.21.15.43:0/545959695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0be410b270 msgr2=0x7f0be41a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.041+0000 7f0be3400700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0be410b270 0x7f0be41a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.041+0000 7f0be2a00700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0be410a920 0x7f0be41a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.041+0000 7f0be3400700 1 -- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0be410a920 msgr2=0x7f0be41a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.041+0000 7f0be3400700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0be410a920 0x7f0be41a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.041+0000 7f0be3400700 1 -- 172.21.15.43:0/545959695 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be41a6340 con 0x7f0be4111bc0 2024-05-27T07:15:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.041+0000 7f0be2a00700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0be410a920 0x7f0be41a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:15:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.041+0000 7f0be3400700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0be4111bc0 0x7f0be41a5ca0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00c110 tx=0x7f0bdc00c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.042+0000 7f0bdb400700 1 -- 172.21.15.43:0/545959695 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc00cd90 con 0x7f0be4111bc0 2024-05-27T07:15:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.042+0000 7f0bdb400700 1 -- 172.21.15.43:0/545959695 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0bdc004bb0 con 0x7f0be4111bc0 2024-05-27T07:15:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.042+0000 7f0bdb400700 1 -- 172.21.15.43:0/545959695 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc007c80 con 0x7f0be4111bc0 2024-05-27T07:15:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.042+0000 7f0be9600700 1 -- 172.21.15.43:0/545959695 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be41a65d0 con 0x7f0be4111bc0 2024-05-27T07:15:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.042+0000 7f0be9600700 1 -- 172.21.15.43:0/545959695 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be41a6a90 con 0x7f0be4111bc0 2024-05-27T07:15:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.047+0000 7f0bf81f1b80 1 Processor -- start 2024-05-27T07:15:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.049+0000 7f0bdb400700 1 -- 172.21.15.43:0/545959695 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7f0bdc01a070 con 0x7f0be4111bc0 2024-05-27T07:15:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.050+0000 7f0bf81f1b80 1 -- start start 2024-05-27T07:15:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.050+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 0x557862f04330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.050+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 0x557862f04870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.050+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 0x557862f11110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.050+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557862e32e40 con 0x557862f04e70 2024-05-27T07:15:14.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.050+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557862e229e0 con 0x557862e719f0 2024-05-27T07:15:14.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.050+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557862e26a20 con 0x557862d85e40 2024-05-27T07:15:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.051+0000 7f0be2a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 0x557862f04870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.051+0000 7f0be2a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 0x557862f04870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:39998/0 (socket says 172.21.15.43:39998) 2024-05-27T07:15:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.051+0000 7f0be2a00700 1 -- 172.21.15.43:0/209713316 learned_addr learned my addr 172.21.15.43:0/209713316 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.052+0000 7f0bdb400700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0bc80b6f20 0x7f0bc80b9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.052+0000 7f0be2000700 1 --2- 172.21.15.43:0/209713316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 0x557862f11110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.052+0000 7f0be2000700 1 --2- 172.21.15.43:0/209713316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 0x557862f04330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.052+0000 7f0bdb400700 1 -- 172.21.15.43:0/545959695 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f0bdc0e2220 con 0x7f0be4111bc0 2024-05-27T07:15:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.052+0000 7f0be2a00700 1 -- 172.21.15.43:0/209713316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 msgr2=0x557862f04330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.052+0000 7f0be2a00700 1 --2- 172.21.15.43:0/209713316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 0x557862f04330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.052+0000 7f0be2a00700 1 -- 172.21.15.43:0/209713316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 msgr2=0x557862f11110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:14.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.052+0000 7f0be2a00700 1 --2- 172.21.15.43:0/209713316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 0x557862f11110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.052+0000 7f0be2a00700 1 -- 172.21.15.43:0/209713316 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557862deb420 con 0x557862d85e40 2024-05-27T07:15:14.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.052+0000 7f0be3400700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0bc80b6f20 0x7f0bc80b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.053+0000 7f0be2a00700 1 --2- 172.21.15.43:0/209713316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 0x557862f04870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009c90 tx=0x7f0bd400f360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59fe33ff98186ae9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.054+0000 7f0be3400700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0bc80b6f20 0x7f0bc80b9430 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0df4a0 tx=0x7f0bdc00ca00 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.054+0000 7f0bcfe00700 1 -- 172.21.15.43:0/209713316 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd400fc30 con 0x557862d85e40 2024-05-27T07:15:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.054+0000 7f0bcfe00700 1 -- 172.21.15.43:0/209713316 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0bd400fdd0 con 0x557862d85e40 2024-05-27T07:15:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.054+0000 7f0bcfe00700 1 -- 172.21.15.43:0/209713316 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4011440 con 0x557862d85e40 2024-05-27T07:15:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.055+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/209713316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 msgr2=0x557862f04870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.055+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/209713316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 0x557862f04870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009c90 tx=0x7f0bd400f360 comp rx=0 tx=0).stop 2024-05-27T07:15:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.055+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/209713316 shutdown_connections 2024-05-27T07:15:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.055+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/209713316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 0x557862f11110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.055+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/209713316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 0x557862f04330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.055+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/209713316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 0x557862f04870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.055+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/209713316 >> 172.21.15.43:0/209713316 conn(0x557862e1f390 msgr2=0x557862f004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.055+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/209713316 shutdown_connections 2024-05-27T07:15:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.056+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/209713316 wait complete. 2024-05-27T07:15:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.056+0000 7f0bf81f1b80 1 -- start start 2024-05-27T07:15:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.056+0000 7f0bf81f1b80 1 Processor -- start 2024-05-27T07:15:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.057+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862d85e40 0x557862f1d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.057+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 0x557862f1d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.057+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862f04e70 0x557862f17600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.057+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557862d28fa0 con 0x557862d85e40 2024-05-27T07:15:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.057+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557862dc3760 con 0x557862e719f0 2024-05-27T07:15:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.057+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557862f11b80 con 0x557862f04e70 2024-05-27T07:15:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.057+0000 7f0bf81f1b80 10 client.0 _ll_register_callbacks cb 0x557862dccc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.058+0000 7f0be2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 0x557862f1d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.058+0000 7f0be2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 0x557862f1d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58380/0 (socket says 172.21.15.43:58380) 2024-05-27T07:15:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.058+0000 7f0be2a00700 1 -- 172.21.15.43:0/4089498044 learned_addr learned my addr 172.21.15.43:0/4089498044 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.058+0000 7f0be2000700 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862d85e40 0x557862f1d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.059+0000 7f0be2a00700 1 -- 172.21.15.43:0/4089498044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862f04e70 msgr2=0x557862f17600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:15:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.059+0000 7f0be2a00700 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862f04e70 0x557862f17600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.059+0000 7f0be2a00700 1 -- 172.21.15.43:0/4089498044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862d85e40 msgr2=0x557862f1d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.059+0000 7f0be2a00700 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862d85e40 0x557862f1d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.059+0000 7f0be2a00700 1 -- 172.21.15.43:0/4089498044 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557862d728d0 con 0x557862e719f0 2024-05-27T07:15:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.059+0000 7f0be2000700 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862d85e40 0x557862f1d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.059+0000 7f0be2a00700 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 0x557862f1d740 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009c60 tx=0x7f0bd400fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28ed7ebbdc8183a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.059+0000 7f0bccc00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:15:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.060+0000 7f0bccc00700 1 -- 172.21.15.43:0/4089498044 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4004750 con 0x557862e719f0 2024-05-27T07:15:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.060+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/4089498044 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bd400fdd0 con 0x557862e719f0 2024-05-27T07:15:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.060+0000 7f0bccc00700 1 -- 172.21.15.43:0/4089498044 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f0bd40048f0 con 0x557862e719f0 2024-05-27T07:15:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.061+0000 7f0bccc00700 1 -- 172.21.15.43:0/4089498044 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0bd4004af0 con 0x557862e719f0 2024-05-27T07:15:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.061+0000 7f0bf81f1b80 10 client.35291 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.061+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/4089498044 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557862df3060 con 0x557862e719f0 2024-05-27T07:15:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.063+0000 7f0bccc00700 1 -- 172.21.15.43:0/4089498044 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0bd400fdd0 con 0x557862e719f0 2024-05-27T07:15:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.063+0000 7f0bccc00700 1 -- 172.21.15.43:0/4089498044 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0bd4011bb0 con 0x557862e719f0 2024-05-27T07:15:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.064+0000 7f0bf81f1b80 10 client.35291 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.064+0000 7f0bf81f1b80 10 client.35291 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:15:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.064+0000 7f0bf81f1b80 20 client.35291 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.064+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x557862f271d0 0x557862f295f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.065+0000 7f0bf81f1b80 4 client.35291 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.065+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/4089498044 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557862e34d90 con 0x557862f271d0 2024-05-27T07:15:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.065+0000 7f0be2000700 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x557862f271d0 0x557862f295f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.067+0000 7f0be2000700 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x557862f271d0 0x557862f295f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9cbc29c697c754d7 server_cookie=b5791d3feefcfbd6 in_seq=0 out_seq=0 2024-05-27T07:15:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.067+0000 7f0bccc00700 10 client.35291 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:15:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.068+0000 7f0bccc00700 1 -- 172.21.15.43:0/4089498044 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557862e34d90 con 0x557862f271d0 2024-05-27T07:15:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.068+0000 7f0bccc00700 10 client.35291 handle_command_reply: tid=0 2024-05-27T07:15:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.068+0000 7f0bf81f1b80 1 client.35291 shutdown 2024-05-27T07:15:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.069+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/4089498044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 msgr2=0x557862f1d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.069+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 0x557862f1d740 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009c60 tx=0x7f0bd400fc60 comp rx=0 tx=0).stop 2024-05-27T07:15:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.069+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/4089498044 shutdown_connections 2024-05-27T07:15:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.069+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x557862f271d0 0x557862f295f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.069+0000 7f0bccc00700 0 client.35291 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:15:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.069+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862f04e70 0x557862f17600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.069+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862d85e40 0x557862f1d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.069+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/4089498044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862e719f0 0x557862f1d740 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.069+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/4089498044 >> 172.21.15.43:0/4089498044 conn(0x557862e1f390 msgr2=0x557862f04310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:14.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.070+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/4089498044 shutdown_connections 2024-05-27T07:15:14.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.070+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/4089498044 wait complete. 2024-05-27T07:15:14.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.070+0000 7f0bf81f1b80 20 client.35291 trim_cache size 0 max 16384 2024-05-27T07:15:14.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:13 smithi132 ceph-mon[35283]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:15:14.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:13 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:14.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.102+0000 7f0bf81f1b80 1 Processor -- start 2024-05-27T07:15:14.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.102+0000 7f0bf81f1b80 1 -- start start 2024-05-27T07:15:14.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.102+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 0x557862f173e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.103+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862e719f0 0x557862f17920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.103+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862f04e70 0x557862f10720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.103+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0be41b6be0 con 0x557862e719f0 2024-05-27T07:15:14.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.103+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0be41b8720 con 0x557862f04e70 2024-05-27T07:15:14.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.103+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0be41b72b0 con 0x557862d85e40 2024-05-27T07:15:14.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.103+0000 7f0be2000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862f04e70 0x557862f10720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.103+0000 7f0be2000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862f04e70 0x557862f10720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58388/0 (socket says 172.21.15.43:58388) 2024-05-27T07:15:14.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.103+0000 7f0be2000700 1 -- 172.21.15.43:0/135061726 learned_addr learned my addr 172.21.15.43:0/135061726 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:14.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.103+0000 7f0be2000700 1 --2- 172.21.15.43:0/135061726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 0x557862f173e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.104+0000 7f0be2a00700 1 --2- 172.21.15.43:0/135061726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862e719f0 0x557862f17920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.104+0000 7f0be2000700 1 -- 172.21.15.43:0/135061726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 msgr2=0x557862f173e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:14.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.104+0000 7f0be2000700 1 --2- 172.21.15.43:0/135061726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 0x557862f173e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.104+0000 7f0be2000700 1 -- 172.21.15.43:0/135061726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862e719f0 msgr2=0x557862f17920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:14.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.104+0000 7f0be2000700 1 --2- 172.21.15.43:0/135061726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862e719f0 0x557862f17920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.104+0000 7f0be2000700 1 -- 172.21.15.43:0/135061726 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557862efc240 con 0x557862f04e70 2024-05-27T07:15:14.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.105+0000 7f0be2000700 1 --2- 172.21.15.43:0/135061726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862f04e70 0x557862f10720 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0009fd0 tx=0x7f0bd0019040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fba561bf5ed7141e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:14.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.105+0000 7f0bcf400700 1 -- 172.21.15.43:0/135061726 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0005360 con 0x557862f04e70 2024-05-27T07:15:14.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.106+0000 7f0bcf400700 1 -- 172.21.15.43:0/135061726 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0bd0005500 con 0x557862f04e70 2024-05-27T07:15:14.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.106+0000 7f0bcf400700 1 -- 172.21.15.43:0/135061726 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd000b7c0 con 0x557862f04e70 2024-05-27T07:15:14.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.107+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/135061726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862f04e70 msgr2=0x557862f10720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.107+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/135061726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862f04e70 0x557862f10720 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0009fd0 tx=0x7f0bd0019040 comp rx=0 tx=0).stop 2024-05-27T07:15:14.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.107+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/135061726 shutdown_connections 2024-05-27T07:15:14.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.107+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/135061726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862f04e70 0x557862f10720 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.107+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/135061726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862d85e40 0x557862f173e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.107+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/135061726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862e719f0 0x557862f17920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.107+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/135061726 >> 172.21.15.43:0/135061726 conn(0x557862e1f390 msgr2=0x557862f040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:14.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.107+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/135061726 shutdown_connections 2024-05-27T07:15:14.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.108+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/135061726 wait complete. 2024-05-27T07:15:14.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.108+0000 7f0bf81f1b80 1 -- start start 2024-05-27T07:15:14.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.109+0000 7f0bf81f1b80 1 Processor -- start 2024-05-27T07:15:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.109+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862d85e40 0x557862f202b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.109+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862e719f0 0x557862f207f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.109+0000 7f0bf81f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 0x557862f20d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.109+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0be41b8050 con 0x557862f04e70 2024-05-27T07:15:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.109+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0be41b7980 con 0x557862d85e40 2024-05-27T07:15:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.109+0000 7f0bf81f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0be41b4b60 con 0x557862e719f0 2024-05-27T07:15:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.109+0000 7f0bf81f1b80 10 client.0 _ll_register_callbacks cb 0x557862dccc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:14.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.110+0000 7f0be2a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862e719f0 0x557862f207f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.110+0000 7f0be2000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 0x557862f20d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.110+0000 7f0be2a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862e719f0 0x557862f207f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:40020/0 (socket says 172.21.15.43:40020) 2024-05-27T07:15:14.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.110+0000 7f0be2a00700 1 -- 172.21.15.43:0/3653320747 learned_addr learned my addr 172.21.15.43:0/3653320747 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.110+0000 7f0be2000700 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862d85e40 0x557862f202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.111+0000 7f0be2000700 1 -- 172.21.15.43:0/3653320747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862e719f0 msgr2=0x557862f207f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.111+0000 7f0be2000700 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862e719f0 0x557862f207f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.111+0000 7f0be2000700 1 -- 172.21.15.43:0/3653320747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862d85e40 msgr2=0x557862f202b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.111+0000 7f0be2000700 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862d85e40 0x557862f202b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.111+0000 7f0be2000700 1 -- 172.21.15.43:0/3653320747 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557862f8fc00 con 0x557862f04e70 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.111+0000 7f0be2000700 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 0x557862f20d30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0009fd0 tx=0x7f0bd000f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202bf821ccaf827b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.111+0000 7f0bccc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.112+0000 7f0bccc00700 1 -- 172.21.15.43:0/3653320747 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd00046d0 con 0x557862f04e70 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.112+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/3653320747 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bd0005500 con 0x557862f04e70 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.112+0000 7f0bccc00700 1 -- 172.21.15.43:0/3653320747 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f0bd0039040 con 0x557862f04e70 2024-05-27T07:15:14.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.112+0000 7f0bccc00700 1 -- 172.21.15.43:0/3653320747 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0bd00046d0 con 0x557862f04e70 2024-05-27T07:15:14.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.113+0000 7f0bf81f1b80 10 client.15621 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:14.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.113+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/3653320747 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557862f90e60 con 0x557862f04e70 2024-05-27T07:15:14.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.114+0000 7f0bccc00700 1 -- 172.21.15.43:0/3653320747 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0bd0005500 con 0x557862f04e70 2024-05-27T07:15:14.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.114+0000 7f0bccc00700 1 -- 172.21.15.43:0/3653320747 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0bd003c020 con 0x557862f04e70 2024-05-27T07:15:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.115+0000 7f0bf81f1b80 10 client.15621 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.115+0000 7f0bf81f1b80 10 client.15621 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:15:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.115+0000 7f0bf81f1b80 20 client.15621 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.115+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x557862f271d0 0x557862f93310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.115+0000 7f0bf81f1b80 4 client.15621 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:15:14.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.115+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/3653320747 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557862e2e570 con 0x557862f271d0 2024-05-27T07:15:14.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.115+0000 7f0be2a00700 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x557862f271d0 0x557862f93310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:14.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.116+0000 7f0be2a00700 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x557862f271d0 0x557862f93310 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a4dc1a72c92af2e4 server_cookie=680ebccb1ee026a9 in_seq=0 out_seq=0 2024-05-27T07:15:14.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.116+0000 7f0bccc00700 10 client.15621 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:15:14.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bccc00700 1 -- 172.21.15.43:0/3653320747 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557862e2e570 con 0x557862f271d0 2024-05-27T07:15:14.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bccc00700 10 client.15621 handle_command_reply: tid=0 2024-05-27T07:15:14.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bf81f1b80 1 client.15621 shutdown 2024-05-27T07:15:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/3653320747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 msgr2=0x557862f20d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 0x557862f20d30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0009fd0 tx=0x7f0bd000f8a0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/3653320747 shutdown_connections 2024-05-27T07:15:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x557862f271d0 0x557862f93310 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557862f04e70 0x557862f20d30 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557862d85e40 0x557862f202b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bf81f1b80 1 --2- 172.21.15.43:0/3653320747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557862e719f0 0x557862f207f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/3653320747 >> 172.21.15.43:0/3653320747 conn(0x557862e1f390 msgr2=0x557862f040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.117+0000 7f0bccc00700 0 client.15621 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:15:14.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.118+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/3653320747 shutdown_connections 2024-05-27T07:15:14.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.118+0000 7f0bf81f1b80 1 -- 172.21.15.43:0/3653320747 wait complete. 2024-05-27T07:15:14.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.119+0000 7f0bf81f1b80 20 client.15621 trim_cache size 0 max 16384 2024-05-27T07:15:14.110 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:15:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.119+0000 7f0be9600700 1 -- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0bc80b6f20 msgr2=0x7f0bc80b9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.119+0000 7f0be9600700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0bc80b6f20 0x7f0bc80b9430 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0df4a0 tx=0x7f0bdc00ca00 comp rx=0 tx=0).stop 2024-05-27T07:15:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.119+0000 7f0be9600700 1 -- 172.21.15.43:0/545959695 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0be4111bc0 msgr2=0x7f0be41a5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.119+0000 7f0be9600700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0be4111bc0 0x7f0be41a5ca0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00c110 tx=0x7f0bdc00c4d0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.120+0000 7f0be9600700 1 -- 172.21.15.43:0/545959695 shutdown_connections 2024-05-27T07:15:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.120+0000 7f0be9600700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0bc80b6f20 0x7f0bc80b9430 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.120+0000 7f0be9600700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0be4111bc0 0x7f0be41a5ca0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.120+0000 7f0be9600700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0be410a920 0x7f0be41a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.120+0000 7f0be9600700 1 --2- 172.21.15.43:0/545959695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0be410b270 0x7f0be41a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:14.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.120+0000 7f0be9600700 1 -- 172.21.15.43:0/545959695 >> 172.21.15.43:0/545959695 conn(0x7f0be406f4c0 msgr2=0x7f0be41108e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:14.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.121+0000 7f0be9600700 1 -- 172.21.15.43:0/545959695 shutdown_connections 2024-05-27T07:15:14.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:14.121+0000 7f0be9600700 1 -- 172.21.15.43:0/545959695 wait complete. 2024-05-27T07:15:14.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:13 smithi179 ceph-mon[35888]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:15:14.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:13 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:15:15.039 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:14 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:15.039 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:14 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:15.039 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:14 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:15:15.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:15:15.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:14 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:15.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:14 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:15.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:14 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:15:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:14 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:15:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:14 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:15:15.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:14 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:15:15.527 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:15.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:15 smithi043 ceph-mon[31531]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:15:16.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:15 smithi132 ceph-mon[35283]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:15:16.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:15 smithi179 ceph-mon[35888]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-05-27T07:15:17.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.775+0000 7f762b200700 1 -- 172.21.15.43:0/4131561673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f762c1043f0 msgr2=0x7f762c104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:17.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.775+0000 7f762b200700 1 --2- 172.21.15.43:0/4131561673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f762c1043f0 0x7f762c104800 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f761c009a40 tx=0x7f761c009d50 comp rx=0 tx=0).stop 2024-05-27T07:15:17.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.775+0000 7f762b200700 1 -- 172.21.15.43:0/4131561673 shutdown_connections 2024-05-27T07:15:17.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.775+0000 7f762b200700 1 --2- 172.21.15.43:0/4131561673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f762c109670 0x7f762c110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.775+0000 7f762b200700 1 --2- 172.21.15.43:0/4131561673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f762c1043f0 0x7f762c104800 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.775+0000 7f762b200700 1 --2- 172.21.15.43:0/4131561673 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f762c108d20 0x7f762c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.775+0000 7f762b200700 1 -- 172.21.15.43:0/4131561673 >> 172.21.15.43:0/4131561673 conn(0x7f762c079d00 msgr2=0x7f762c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:17.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.775+0000 7f762b200700 1 -- 172.21.15.43:0/4131561673 shutdown_connections 2024-05-27T07:15:17.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.775+0000 7f762b200700 1 -- 172.21.15.43:0/4131561673 wait complete. 2024-05-27T07:15:17.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.776+0000 7f762b200700 1 Processor -- start 2024-05-27T07:15:17.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.777+0000 7f762b200700 1 -- start start 2024-05-27T07:15:17.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.777+0000 7f762b200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f762c1043f0 0x7f762c19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.777+0000 7f762b200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f762c108d20 0x7f762c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.777+0000 7f762b200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f762c109670 0x7f762c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.777+0000 7f762b200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f762c112020 con 0x7f762c1043f0 2024-05-27T07:15:17.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.777+0000 7f762b200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f762c111ea0 con 0x7f762c108d20 2024-05-27T07:15:17.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.777+0000 7f762b200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f762c1121a0 con 0x7f762c109670 2024-05-27T07:15:17.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.778+0000 7f7629e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f762c1043f0 0x7f762c19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.778+0000 7f7629e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f762c1043f0 0x7f762c19e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50744/0 (socket says 172.21.15.43:50744) 2024-05-27T07:15:17.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.778+0000 7f7629e00700 1 -- 172.21.15.43:0/109463557 learned_addr learned my addr 172.21.15.43:0/109463557 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:17.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.778+0000 7f7629400700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f762c108d20 0x7f762c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.779+0000 7f7629e00700 1 -- 172.21.15.43:0/109463557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f762c109670 msgr2=0x7f762c1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:17.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.779+0000 7f7629e00700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f762c109670 0x7f762c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.779+0000 7f7629e00700 1 -- 172.21.15.43:0/109463557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f762c108d20 msgr2=0x7f762c19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:17.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.779+0000 7f7629e00700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f762c108d20 0x7f762c19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.779+0000 7f7629e00700 1 -- 172.21.15.43:0/109463557 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762c198ed0 con 0x7f762c1043f0 2024-05-27T07:15:17.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.779+0000 7f7629e00700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f762c1043f0 0x7f762c19e560 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f761c005950 tx=0x7f761c00ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:17.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.780+0000 7f7622a00700 1 -- 172.21.15.43:0/109463557 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c026070 con 0x7f762c1043f0 2024-05-27T07:15:17.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.780+0000 7f7622a00700 1 -- 172.21.15.43:0/109463557 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f761c005da0 con 0x7f762c1043f0 2024-05-27T07:15:17.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.780+0000 7f762b200700 1 -- 172.21.15.43:0/109463557 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f762c199160 con 0x7f762c1043f0 2024-05-27T07:15:17.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.780+0000 7f7622a00700 1 -- 172.21.15.43:0/109463557 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c022810 con 0x7f762c1043f0 2024-05-27T07:15:17.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.780+0000 7f762b200700 1 -- 172.21.15.43:0/109463557 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f762c1a53c0 con 0x7f762c1043f0 2024-05-27T07:15:17.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.785+0000 7f763fd89b80 1 Processor -- start 2024-05-27T07:15:17.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.785+0000 7f763fd89b80 1 -- start start 2024-05-27T07:15:17.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.787+0000 7f7622a00700 1 -- 172.21.15.43:0/109463557 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7f761c029070 con 0x7f762c1043f0 2024-05-27T07:15:17.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.787+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c60659f0 0x5653c60f9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.787+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c60f9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.787+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 0x5653c61056e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.787+0000 7f763fd89b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5653c6026e40 con 0x5653c61032c0 2024-05-27T07:15:17.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.787+0000 7f763fd89b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5653c60169e0 con 0x5653c60659f0 2024-05-27T07:15:17.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.787+0000 7f763fd89b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5653c601aa20 con 0x5653c5f79e40 2024-05-27T07:15:17.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.788+0000 7f7629400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 0x5653c61056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.788+0000 7f7629400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 0x5653c61056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50754/0 (socket says 172.21.15.43:50754) 2024-05-27T07:15:17.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.788+0000 7f7629400700 1 -- 172.21.15.43:0/762623121 learned_addr learned my addr 172.21.15.43:0/762623121 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:17.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.788+0000 7f762a800700 1 --2- 172.21.15.43:0/762623121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c60659f0 0x5653c60f9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.788+0000 7f7629e00700 1 --2- 172.21.15.43:0/762623121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c60f9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.789+0000 7f7629400700 1 -- 172.21.15.43:0/762623121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 msgr2=0x5653c60f9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:17.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.789+0000 7f7629400700 1 --2- 172.21.15.43:0/762623121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c60f9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.789+0000 7f7629400700 1 -- 172.21.15.43:0/762623121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c60659f0 msgr2=0x5653c60f9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:17.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.789+0000 7f7629400700 1 --2- 172.21.15.43:0/762623121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c60659f0 0x5653c60f9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.789+0000 7f7629400700 1 -- 172.21.15.43:0/762623121 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c5fdf420 con 0x5653c61032c0 2024-05-27T07:15:17.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.789+0000 7f7622a00700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f76100b2b90 0x7f76100b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.790+0000 7f7629400700 1 --2- 172.21.15.43:0/762623121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 0x5653c61056e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f761800bcf0 tx=0x7f76180103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab8f04b3c6f371c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:17.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.790+0000 7f7622a00700 1 -- 172.21.15.43:0/109463557 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f761c0e8760 con 0x7f762c1043f0 2024-05-27T07:15:17.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.790+0000 7f762a800700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f76100b2b90 0x7f76100b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.790+0000 7f7617400700 1 -- 172.21.15.43:0/762623121 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7618010d10 con 0x5653c61032c0 2024-05-27T07:15:17.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.790+0000 7f7617400700 1 -- 172.21.15.43:0/762623121 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f76180044c0 con 0x5653c61032c0 2024-05-27T07:15:17.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.790+0000 7f7617400700 1 -- 172.21.15.43:0/762623121 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7618005280 con 0x5653c61032c0 2024-05-27T07:15:17.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.792+0000 7f762a800700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f76100b2b90 0x7f76100b50a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7624009510 tx=0x7f7624009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:17.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.792+0000 7f763fd89b80 1 -- 172.21.15.43:0/762623121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 msgr2=0x5653c61056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:17.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.792+0000 7f763fd89b80 1 --2- 172.21.15.43:0/762623121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 0x5653c61056e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f761800bcf0 tx=0x7f76180103d0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.792+0000 7f763fd89b80 1 -- 172.21.15.43:0/762623121 shutdown_connections 2024-05-27T07:15:17.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.792+0000 7f763fd89b80 1 --2- 172.21.15.43:0/762623121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 0x5653c61056e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.792+0000 7f763fd89b80 1 --2- 172.21.15.43:0/762623121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c60659f0 0x5653c60f9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.792+0000 7f763fd89b80 1 --2- 172.21.15.43:0/762623121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c60f9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.792+0000 7f763fd89b80 1 -- 172.21.15.43:0/762623121 >> 172.21.15.43:0/762623121 conn(0x5653c6013390 msgr2=0x5653c60f4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:17.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.792+0000 7f763fd89b80 1 -- 172.21.15.43:0/762623121 shutdown_connections 2024-05-27T07:15:17.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.793+0000 7f763fd89b80 1 -- 172.21.15.43:0/762623121 wait complete. 2024-05-27T07:15:17.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.793+0000 7f763fd89b80 1 -- start start 2024-05-27T07:15:17.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.794+0000 7f763fd89b80 1 Processor -- start 2024-05-27T07:15:17.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.794+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c6100240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.794+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c60659f0 0x5653c6100780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.794+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 0x5653c6100cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.795+0000 7f763fd89b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5653c5f1cfa0 con 0x5653c61032c0 2024-05-27T07:15:17.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.795+0000 7f763fd89b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5653c5fb7760 con 0x5653c60659f0 2024-05-27T07:15:17.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.795+0000 7f763fd89b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5653c6105e60 con 0x5653c5f79e40 2024-05-27T07:15:17.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.795+0000 7f763fd89b80 10 client.0 _ll_register_callbacks cb 0x5653c5fc0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:17.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.795+0000 7f7629e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c6100240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.795+0000 7f762a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 0x5653c6100cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.795+0000 7f7629e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c6100240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:52906/0 (socket says 172.21.15.43:52906) 2024-05-27T07:15:17.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.795+0000 7f7629e00700 1 -- 172.21.15.43:0/3843503634 learned_addr learned my addr 172.21.15.43:0/3843503634 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:17.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.795+0000 7f7629400700 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c60659f0 0x5653c6100780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.796+0000 7f7629e00700 1 -- 172.21.15.43:0/3843503634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c60659f0 msgr2=0x5653c6100780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:17.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.796+0000 7f7629e00700 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c60659f0 0x5653c6100780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.796+0000 7f7629e00700 1 -- 172.21.15.43:0/3843503634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 msgr2=0x5653c6100cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:17.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.796+0000 7f7629e00700 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 0x5653c6100cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.796+0000 7f7629e00700 1 -- 172.21.15.43:0/3843503634 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c5f668d0 con 0x5653c5f79e40 2024-05-27T07:15:17.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.797+0000 7f7629e00700 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c6100240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f761c0ab660 tx=0x7f761c0afaf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4179c6323dca389 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:17.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.797+0000 7f760be00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:15:17.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.797+0000 7f760be00700 1 -- 172.21.15.43:0/3843503634 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c0aef30 con 0x5653c5f79e40 2024-05-27T07:15:17.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.797+0000 7f763fd89b80 1 -- 172.21.15.43:0/3843503634 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76180044c0 con 0x5653c5f79e40 2024-05-27T07:15:17.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.797+0000 7f760be00700 1 -- 172.21.15.43:0/3843503634 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f761c0b7f20 con 0x5653c5f79e40 2024-05-27T07:15:17.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.798+0000 7f760be00700 1 -- 172.21.15.43:0/3843503634 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f761c0aed50 con 0x5653c5f79e40 2024-05-27T07:15:17.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.798+0000 7f763fd89b80 10 client.25339 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:17.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.798+0000 7f763fd89b80 1 -- 172.21.15.43:0/3843503634 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5653c5fe7060 con 0x5653c5f79e40 2024-05-27T07:15:17.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.798+0000 7f760be00700 1 -- 172.21.15.43:0/3843503634 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f761c0af0f0 con 0x5653c5f79e40 2024-05-27T07:15:17.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.799+0000 7f760be00700 1 -- 172.21.15.43:0/3843503634 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f761c0e8020 con 0x5653c5f79e40 2024-05-27T07:15:17.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.799+0000 7f763fd89b80 10 client.25339 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:17.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.799+0000 7f763fd89b80 10 client.25339 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:15:17.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.799+0000 7f763fd89b80 20 client.25339 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:17.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.799+0000 7f763fd89b80 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5653c6112d00 0x5653c6115170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.799+0000 7f763fd89b80 4 client.25339 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:17.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.799+0000 7f763fd89b80 1 -- 172.21.15.43:0/3843503634 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f76180044c0 con 0x5653c6112d00 2024-05-27T07:15:17.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.800+0000 7f7629400700 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5653c6112d00 0x5653c6115170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.801+0000 7f7629400700 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5653c6112d00 0x5653c6115170 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e1d3b3de9c49cef server_cookie=9d79439492a53525 in_seq=0 out_seq=0 2024-05-27T07:15:17.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.801+0000 7f760be00700 10 client.25339 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:17.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f760be00700 1 -- 172.21.15.43:0/3843503634 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f76180044c0 con 0x5653c6112d00 2024-05-27T07:15:17.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f760be00700 10 client.25339 handle_command_reply: tid=0 2024-05-27T07:15:17.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f763fd89b80 1 client.25339 shutdown 2024-05-27T07:15:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f763fd89b80 1 -- 172.21.15.43:0/3843503634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 msgr2=0x5653c6100240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f763fd89b80 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c6100240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f761c0ab660 tx=0x7f761c0afaf0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f763fd89b80 1 -- 172.21.15.43:0/3843503634 shutdown_connections 2024-05-27T07:15:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f763fd89b80 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5653c6112d00 0x5653c6115170 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f763fd89b80 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c61032c0 0x5653c6100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f763fd89b80 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c6100240 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f763fd89b80 1 --2- 172.21.15.43:0/3843503634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c60659f0 0x5653c6100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f763fd89b80 1 -- 172.21.15.43:0/3843503634 >> 172.21.15.43:0/3843503634 conn(0x5653c6013390 msgr2=0x5653c60f97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.803+0000 7f760be00700 0 client.25339 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:17.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.804+0000 7f763fd89b80 1 -- 172.21.15.43:0/3843503634 shutdown_connections 2024-05-27T07:15:17.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.804+0000 7f763fd89b80 1 -- 172.21.15.43:0/3843503634 wait complete. 2024-05-27T07:15:17.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.804+0000 7f763fd89b80 20 client.25339 trim_cache size 0 max 16384 2024-05-27T07:15:17.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.830+0000 7f763fd89b80 1 Processor -- start 2024-05-27T07:15:17.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.830+0000 7f763fd89b80 1 -- start start 2024-05-27T07:15:17.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.830+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c5f79e40 0x5653c60fb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.830+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c60f4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.830+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c61032c0 0x5653c610ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.830+0000 7f763fd89b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f762c1ae070 con 0x5653c60659f0 2024-05-27T07:15:17.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.830+0000 7f763fd89b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f762c1afbb0 con 0x5653c5f79e40 2024-05-27T07:15:17.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.830+0000 7f763fd89b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f762c1ae740 con 0x5653c61032c0 2024-05-27T07:15:17.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.831+0000 7f7629400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c60f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.831+0000 7f7629400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c60f4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50776/0 (socket says 172.21.15.43:50776) 2024-05-27T07:15:17.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.831+0000 7f7629400700 1 -- 172.21.15.43:0/842396067 learned_addr learned my addr 172.21.15.43:0/842396067 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:17.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.831+0000 7f7629e00700 1 --2- 172.21.15.43:0/842396067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c5f79e40 0x5653c60fb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.831+0000 7f7629400700 1 -- 172.21.15.43:0/842396067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c61032c0 msgr2=0x5653c610ef00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:15:17.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.831+0000 7f7629400700 1 --2- 172.21.15.43:0/842396067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c61032c0 0x5653c610ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.831+0000 7f7629400700 1 -- 172.21.15.43:0/842396067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c5f79e40 msgr2=0x5653c60fb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:17.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.831+0000 7f7629400700 1 --2- 172.21.15.43:0/842396067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c5f79e40 0x5653c60fb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.831+0000 7f7629400700 1 -- 172.21.15.43:0/842396067 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c6101030 con 0x5653c60659f0 2024-05-27T07:15:17.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.832+0000 7f7629400700 1 --2- 172.21.15.43:0/842396067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c60f4740 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f761800bcf0 tx=0x7f761800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805da7b307e561f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:17.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.832+0000 7f7616a00700 1 -- 172.21.15.43:0/842396067 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7618010d10 con 0x5653c60659f0 2024-05-27T07:15:17.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.832+0000 7f7616a00700 1 -- 172.21.15.43:0/842396067 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f76180298c0 con 0x5653c60659f0 2024-05-27T07:15:17.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.832+0000 7f7616a00700 1 -- 172.21.15.43:0/842396067 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7618004f20 con 0x5653c60659f0 2024-05-27T07:15:17.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.833+0000 7f763fd89b80 1 -- 172.21.15.43:0/842396067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 msgr2=0x5653c60f4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:17.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.833+0000 7f763fd89b80 1 --2- 172.21.15.43:0/842396067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c60f4740 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f761800bcf0 tx=0x7f761800b410 comp rx=0 tx=0).stop 2024-05-27T07:15:17.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.833+0000 7f763fd89b80 1 -- 172.21.15.43:0/842396067 shutdown_connections 2024-05-27T07:15:17.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.833+0000 7f763fd89b80 1 --2- 172.21.15.43:0/842396067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c61032c0 0x5653c610ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.833+0000 7f763fd89b80 1 --2- 172.21.15.43:0/842396067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c5f79e40 0x5653c60fb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.833+0000 7f763fd89b80 1 --2- 172.21.15.43:0/842396067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c60f4740 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.833+0000 7f763fd89b80 1 -- 172.21.15.43:0/842396067 >> 172.21.15.43:0/842396067 conn(0x5653c6013390 msgr2=0x5653c60fb240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:17.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.833+0000 7f763fd89b80 1 -- 172.21.15.43:0/842396067 shutdown_connections 2024-05-27T07:15:17.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.834+0000 7f763fd89b80 1 -- 172.21.15.43:0/842396067 wait complete. 2024-05-27T07:15:17.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.834+0000 7f763fd89b80 1 -- start start 2024-05-27T07:15:17.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.834+0000 7f763fd89b80 1 Processor -- start 2024-05-27T07:15:17.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.835+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c61066b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.835+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c6106bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.835+0000 7f763fd89b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c61032c0 0x5653c617b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.835+0000 7f763fd89b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f762c1af4e0 con 0x5653c60659f0 2024-05-27T07:15:17.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.835+0000 7f763fd89b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f762c1aee10 con 0x5653c61032c0 2024-05-27T07:15:17.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.835+0000 7f763fd89b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f762c1abff0 con 0x5653c5f79e40 2024-05-27T07:15:17.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.835+0000 7f763fd89b80 10 client.0 _ll_register_callbacks cb 0x5653c5fc0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:17.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.836+0000 7f7629400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c6106bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.836+0000 7f7629400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c6106bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50786/0 (socket says 172.21.15.43:50786) 2024-05-27T07:15:17.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.836+0000 7f7629400700 1 -- 172.21.15.43:0/2287737050 learned_addr learned my addr 172.21.15.43:0/2287737050 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:17.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.836+0000 7f7629e00700 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c61066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.836+0000 7f762a800700 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c61032c0 0x5653c617b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.836+0000 7f7629400700 1 -- 172.21.15.43:0/2287737050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 msgr2=0x5653c61066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:17.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.836+0000 7f7629400700 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c61066b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.836+0000 7f7629400700 1 -- 172.21.15.43:0/2287737050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c61032c0 msgr2=0x5653c617b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:17.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.836+0000 7f7629400700 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c61032c0 0x5653c617b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.836+0000 7f7629400700 1 -- 172.21.15.43:0/2287737050 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c617bd30 con 0x5653c60659f0 2024-05-27T07:15:17.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.837+0000 7f7629400700 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c6106bf0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7618010410 tx=0x7f7618029ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5436efd2f49ba8f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:17.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.837+0000 7f762a800700 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c61032c0 0x5653c617b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:15:17.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.837+0000 7f760be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:17.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.837+0000 7f760be00700 1 -- 172.21.15.43:0/2287737050 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7618010b10 con 0x5653c60659f0 2024-05-27T07:15:17.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.837+0000 7f760be00700 1 -- 172.21.15.43:0/2287737050 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f7618010cb0 con 0x5653c60659f0 2024-05-27T07:15:17.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.837+0000 7f763fd89b80 1 -- 172.21.15.43:0/2287737050 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76180298c0 con 0x5653c60659f0 2024-05-27T07:15:17.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.838+0000 7f760be00700 1 -- 172.21.15.43:0/2287737050 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7618043050 con 0x5653c60659f0 2024-05-27T07:15:17.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.838+0000 7f763fd89b80 10 client.15639 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:17.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.838+0000 7f763fd89b80 1 -- 172.21.15.43:0/2287737050 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5653c6185770 con 0x5653c60659f0 2024-05-27T07:15:17.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.838+0000 7f760be00700 1 -- 172.21.15.43:0/2287737050 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7618036070 con 0x5653c60659f0 2024-05-27T07:15:17.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.838+0000 7f760be00700 1 -- 172.21.15.43:0/2287737050 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7618011440 con 0x5653c60659f0 2024-05-27T07:15:17.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.839+0000 7f763fd89b80 10 client.15639 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:17.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.839+0000 7f763fd89b80 10 client.15639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:15:17.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.839+0000 7f763fd89b80 20 client.15639 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:17.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.839+0000 7f763fd89b80 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5653c6185c00 0x5653c6188020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:17.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.839+0000 7f763fd89b80 4 client.15639 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:15:17.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.839+0000 7f763fd89b80 1 -- 172.21.15.43:0/2287737050 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f76180298c0 con 0x5653c6185c00 2024-05-27T07:15:17.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.839+0000 7f7629e00700 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5653c6185c00 0x5653c6188020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:17.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.840+0000 7f7629e00700 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5653c6185c00 0x5653c6188020 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ffdc5c81b94bf1 server_cookie=6e352cf911642444 in_seq=0 out_seq=0 2024-05-27T07:15:17.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.840+0000 7f760be00700 10 client.15639 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:17.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f760be00700 1 -- 172.21.15.43:0/2287737050 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f76180298c0 con 0x5653c6185c00 2024-05-27T07:15:17.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f760be00700 10 client.15639 handle_command_reply: tid=0 2024-05-27T07:15:17.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f763fd89b80 1 client.15639 shutdown 2024-05-27T07:15:17.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f763fd89b80 1 -- 172.21.15.43:0/2287737050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 msgr2=0x5653c6106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:17.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f763fd89b80 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c6106bf0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7618010410 tx=0x7f7618029ca0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f763fd89b80 1 -- 172.21.15.43:0/2287737050 shutdown_connections 2024-05-27T07:15:17.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f763fd89b80 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653c61032c0 0x5653c617b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f763fd89b80 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5653c5f79e40 0x5653c61066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.834 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:15:17.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f763fd89b80 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5653c6185c00 0x5653c6188020 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f763fd89b80 1 --2- 172.21.15.43:0/2287737050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653c60659f0 0x5653c6106bf0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f763fd89b80 1 -- 172.21.15.43:0/2287737050 >> 172.21.15.43:0/2287737050 conn(0x5653c6013390 msgr2=0x5653c6100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:17.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f760be00700 0 client.15639 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:17.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.841+0000 7f763fd89b80 1 -- 172.21.15.43:0/2287737050 shutdown_connections 2024-05-27T07:15:17.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.842+0000 7f763fd89b80 1 -- 172.21.15.43:0/2287737050 wait complete. 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.842+0000 7f763fd89b80 20 client.15639 trim_cache size 0 max 16384 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.842+0000 7f762b200700 1 -- 172.21.15.43:0/109463557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f76100b2b90 msgr2=0x7f76100b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.842+0000 7f762b200700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f76100b2b90 0x7f76100b50a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7624009510 tx=0x7f7624009040 comp rx=0 tx=0).stop 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.842+0000 7f762b200700 1 -- 172.21.15.43:0/109463557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f762c1043f0 msgr2=0x7f762c19e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.842+0000 7f762b200700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f762c1043f0 0x7f762c19e560 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f761c005950 tx=0x7f761c00ba00 comp rx=0 tx=0).stop 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.843+0000 7f762b200700 1 -- 172.21.15.43:0/109463557 shutdown_connections 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.843+0000 7f762b200700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f76100b2b90 0x7f76100b50a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.843+0000 7f762b200700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f762c109670 0x7f762c1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.843+0000 7f762b200700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f762c1043f0 0x7f762c19e560 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.843+0000 7f762b200700 1 --2- 172.21.15.43:0/109463557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f762c108d20 0x7f762c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:17.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.843+0000 7f762b200700 1 -- 172.21.15.43:0/109463557 >> 172.21.15.43:0/109463557 conn(0x7f762c079d00 msgr2=0x7f762c101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:17.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.843+0000 7f762b200700 1 -- 172.21.15.43:0/109463557 shutdown_connections 2024-05-27T07:15:17.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:17.843+0000 7f762b200700 1 -- 172.21.15.43:0/109463557 wait complete. 2024-05-27T07:15:18.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:17 smithi043 ceph-mon[31531]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:15:18.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:17 smithi179 ceph-mon[35888]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:15:18.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:17 smithi132 ceph-mon[35283]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-05-27T07:15:19.752 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:15:19.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:15:20.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:19 smithi043 ceph-mon[31531]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-27T07:15:20.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:19 smithi179 ceph-mon[35888]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-27T07:15:20.231 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:20.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:19 smithi132 ceph-mon[35283]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-27T07:15:22.212 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:21 smithi043 ceph-mon[31531]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-05-27T07:15:22.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:21 smithi132 ceph-mon[35283]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-05-27T07:15:22.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:21 smithi179 ceph-mon[35888]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-05-27T07:15:22.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.671+0000 7fcbfb200700 1 -- 172.21.15.43:0/1213089050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbfc105280 msgr2=0x7fcbfc105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:22.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.671+0000 7fcbfb200700 1 --2- 172.21.15.43:0/1213089050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbfc105280 0x7fcbfc105690 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009990 tx=0x7fcbec009ca0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.671+0000 7fcbfb200700 1 -- 172.21.15.43:0/1213089050 shutdown_connections 2024-05-27T07:15:22.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.671+0000 7fcbfb200700 1 --2- 172.21.15.43:0/1213089050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbfc10e6f0 0x7fcbfc110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.671+0000 7fcbfb200700 1 --2- 172.21.15.43:0/1213089050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbfc105280 0x7fcbfc105690 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.671+0000 7fcbfb200700 1 --2- 172.21.15.43:0/1213089050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbfc105bd0 0x7fcbfc10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.671+0000 7fcbfb200700 1 -- 172.21.15.43:0/1213089050 >> 172.21.15.43:0/1213089050 conn(0x7fcbfc078ab0 msgr2=0x7fcbfc078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:22.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.672+0000 7fcbfb200700 1 -- 172.21.15.43:0/1213089050 shutdown_connections 2024-05-27T07:15:22.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.672+0000 7fcbfb200700 1 -- 172.21.15.43:0/1213089050 wait complete. 2024-05-27T07:15:22.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.673+0000 7fcbfb200700 1 Processor -- start 2024-05-27T07:15:22.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.673+0000 7fcbfb200700 1 -- start start 2024-05-27T07:15:22.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.674+0000 7fcbfb200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbfc105280 0x7fcbfc19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.674+0000 7fcbfb200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbfc105bd0 0x7fcbfc19d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.674+0000 7fcbfb200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbfc10e6f0 0x7fcbfc1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.674+0000 7fcbfb200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc114140 con 0x7fcbfc10e6f0 2024-05-27T07:15:22.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.674+0000 7fcbfb200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc113fc0 con 0x7fcbfc105bd0 2024-05-27T07:15:22.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.674+0000 7fcbfb200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc1142c0 con 0x7fcbfc105280 2024-05-27T07:15:22.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.674+0000 7fcbf9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbfc105280 0x7fcbfc19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.674+0000 7fcbf9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbfc105bd0 0x7fcbfc19d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.674+0000 7fcbf9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbfc105280 0x7fcbfc19cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:52930/0 (socket says 172.21.15.43:52930) 2024-05-27T07:15:22.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.674+0000 7fcbf9e00700 1 -- 172.21.15.43:0/4076014729 learned_addr learned my addr 172.21.15.43:0/4076014729 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:22.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.675+0000 7fcbfa800700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbfc10e6f0 0x7fcbfc1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.675+0000 7fcbfa800700 1 -- 172.21.15.43:0/4076014729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbfc105280 msgr2=0x7fcbfc19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:22.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.675+0000 7fcbfa800700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbfc105280 0x7fcbfc19cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.675+0000 7fcbfa800700 1 -- 172.21.15.43:0/4076014729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbfc105bd0 msgr2=0x7fcbfc19d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:22.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.675+0000 7fcbfa800700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbfc105bd0 0x7fcbfc19d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.675+0000 7fcbfa800700 1 -- 172.21.15.43:0/4076014729 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc1a1c60 con 0x7fcbfc10e6f0 2024-05-27T07:15:22.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.676+0000 7fcbfa800700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbfc10e6f0 0x7fcbfc1a15c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400efc0 tx=0x7fcbf400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:22.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.676+0000 7fcbf2a00700 1 -- 172.21.15.43:0/4076014729 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40092e0 con 0x7fcbfc10e6f0 2024-05-27T07:15:22.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.676+0000 7fcbfb200700 1 -- 172.21.15.43:0/4076014729 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbfc1a1ef0 con 0x7fcbfc10e6f0 2024-05-27T07:15:22.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.677+0000 7fcbfb200700 1 -- 172.21.15.43:0/4076014729 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbfc1060c0 con 0x7fcbfc10e6f0 2024-05-27T07:15:22.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.677+0000 7fcbf2a00700 1 -- 172.21.15.43:0/4076014729 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbf4004bb0 con 0x7fcbfc10e6f0 2024-05-27T07:15:22.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.677+0000 7fcbf2a00700 1 -- 172.21.15.43:0/4076014729 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4004020 con 0x7fcbfc10e6f0 2024-05-27T07:15:22.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.683+0000 7fcbf2a00700 1 -- 172.21.15.43:0/4076014729 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7fcbf401b030 con 0x7fcbfc10e6f0 2024-05-27T07:15:22.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.685+0000 7fcc0f9abb80 1 Processor -- start 2024-05-27T07:15:22.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.685+0000 7fcc0f9abb80 1 -- start start 2024-05-27T07:15:22.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.686+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474c56a70 0x563474cea7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.686+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474b6aec0 0x563474cead10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.686+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474cf4340 0x563474cf6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.686+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563474c17ec0 con 0x563474cf4340 2024-05-27T07:15:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.686+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563474c07a60 con 0x563474b6aec0 2024-05-27T07:15:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.686+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x563474c0baa0 con 0x563474c56a70 2024-05-27T07:15:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.686+0000 7fcbf2a00700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fcbe00b2c10 0x7fcbe00b5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.686+0000 7fcbf2a00700 1 -- 172.21.15.43:0/4076014729 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fcbf40e2560 con 0x7fcbfc10e6f0 2024-05-27T07:15:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.687+0000 7fcbf9e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474b6aec0 0x563474cead10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.687+0000 7fcbf9e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474b6aec0 0x563474cead10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:53774/0 (socket says 172.21.15.43:53774) 2024-05-27T07:15:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.687+0000 7fcbf9e00700 1 -- 172.21.15.43:0/1535384566 learned_addr learned my addr 172.21.15.43:0/1535384566 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.687+0000 7fcbf9400700 1 --2- 172.21.15.43:0/1535384566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474c56a70 0x563474cea7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.688+0000 7fcbf9e00700 1 -- 172.21.15.43:0/1535384566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474c56a70 msgr2=0x563474cea7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.688+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/1535384566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474c56a70 0x563474cea7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.688+0000 7fcbfa800700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fcbe00b2c10 0x7fcbe00b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.688+0000 7fcbf9e00700 1 -- 172.21.15.43:0/1535384566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474cf4340 msgr2=0x563474cf6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.688+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/1535384566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474cf4340 0x563474cf6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.688+0000 7fcbf9e00700 1 -- 172.21.15.43:0/1535384566 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563474bd04a0 con 0x563474b6aec0 2024-05-27T07:15:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.689+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/1535384566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474b6aec0 0x563474cead10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0229c0 tx=0x7fcbec00b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b29b8e25bab74d62 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.689+0000 7fcbfa800700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fcbe00b2c10 0x7fcbe00b5120 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40df7e0 tx=0x7fcbf4003f70 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.689+0000 7fcbe7400700 1 -- 172.21.15.43:0/1535384566 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec02f070 con 0x563474b6aec0 2024-05-27T07:15:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.690+0000 7fcbe7400700 1 -- 172.21.15.43:0/1535384566 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbec003680 con 0x563474b6aec0 2024-05-27T07:15:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.690+0000 7fcbe7400700 1 -- 172.21.15.43:0/1535384566 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec02f070 con 0x563474b6aec0 2024-05-27T07:15:22.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.691+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1535384566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474b6aec0 msgr2=0x563474cead10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:22.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.691+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1535384566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474b6aec0 0x563474cead10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0229c0 tx=0x7fcbec00b260 comp rx=0 tx=0).stop 2024-05-27T07:15:22.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.691+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1535384566 shutdown_connections 2024-05-27T07:15:22.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.691+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1535384566 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474cf4340 0x563474cf6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.691+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1535384566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474c56a70 0x563474cea7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.691+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1535384566 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474b6aec0 0x563474cead10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.691+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1535384566 >> 172.21.15.43:0/1535384566 conn(0x563474c04410 msgr2=0x563474ce5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.692+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1535384566 shutdown_connections 2024-05-27T07:15:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.692+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1535384566 wait complete. 2024-05-27T07:15:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.693+0000 7fcc0f9abb80 1 -- start start 2024-05-27T07:15:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.693+0000 7fcc0f9abb80 1 Processor -- start 2024-05-27T07:15:22.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.694+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474cf1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.694+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474c56a70 0x563474cf17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.694+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474cf4340 0x563474cf1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.694+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563474b0e020 con 0x563474b6aec0 2024-05-27T07:15:22.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.694+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563474ba87e0 con 0x563474cf4340 2024-05-27T07:15:22.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.694+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x563474cf6e20 con 0x563474c56a70 2024-05-27T07:15:22.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.694+0000 7fcc0f9abb80 10 client.0 _ll_register_callbacks cb 0x563474bb1cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:22.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.695+0000 7fcbf9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474cf1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.695+0000 7fcbf9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474cf1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50838/0 (socket says 172.21.15.43:50838) 2024-05-27T07:15:22.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.695+0000 7fcbf9400700 1 -- 172.21.15.43:0/1949237940 learned_addr learned my addr 172.21.15.43:0/1949237940 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:22.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.695+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474c56a70 0x563474cf17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.696+0000 7fcbf9400700 1 -- 172.21.15.43:0/1949237940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474c56a70 msgr2=0x563474cf17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:22.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.696+0000 7fcbf9400700 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474c56a70 0x563474cf17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.696+0000 7fcbf9400700 1 -- 172.21.15.43:0/1949237940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474cf4340 msgr2=0x563474cf1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:22.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.696+0000 7fcbf9400700 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474cf4340 0x563474cf1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.696+0000 7fcbf9400700 1 -- 172.21.15.43:0/1949237940 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563474b57950 con 0x563474b6aec0 2024-05-27T07:15:22.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.697+0000 7fcbf9400700 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474cf1290 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8007fb0 tx=0x7fcbe80137f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6da7900a7164ca3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:22.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.697+0000 7fcbdbe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:22.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.697+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1949237940 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8004250 con 0x563474b6aec0 2024-05-27T07:15:22.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.697+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1949237940 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fcbe80043f0 con 0x563474b6aec0 2024-05-27T07:15:22.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.698+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1949237940 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbec003680 con 0x563474b6aec0 2024-05-27T07:15:22.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.698+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1949237940 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcbe8003680 con 0x563474b6aec0 2024-05-27T07:15:22.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.698+0000 7fcc0f9abb80 10 client.15648 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:22.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.698+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1949237940 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563474bd80e0 con 0x563474b6aec0 2024-05-27T07:15:22.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.699+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1949237940 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbe8017030 con 0x563474b6aec0 2024-05-27T07:15:22.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.699+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1949237940 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcbe801c920 con 0x563474b6aec0 2024-05-27T07:15:22.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.699+0000 7fcc0f9abb80 10 client.15648 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:22.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.699+0000 7fcc0f9abb80 10 client.15648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:15:22.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.699+0000 7fcc0f9abb80 20 client.15648 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:22.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.699+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x563474d03bf0 0x563474d06010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.699+0000 7fcc0f9abb80 4 client.15648 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:22.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.699+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1949237940 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcbec003680 con 0x563474d03bf0 2024-05-27T07:15:22.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.700+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x563474d03bf0 0x563474d06010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.701+0000 7fcbdbe00700 10 client.15648 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:22.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.701+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x563474d03bf0 0x563474d06010 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d05aafc9222e114f server_cookie=f05aa1bddafac39a in_seq=0 out_seq=0 2024-05-27T07:15:22.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.703+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1949237940 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7fcbec003680 con 0x563474d03bf0 2024-05-27T07:15:22.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.703+0000 7fcbdbe00700 10 client.15648 handle_command_reply: tid=0 2024-05-27T07:15:22.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.704+0000 7fcc0f9abb80 1 client.15648 shutdown 2024-05-27T07:15:22.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.704+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1949237940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 msgr2=0x563474cf1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:22.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.704+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474cf1290 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8007fb0 tx=0x7fcbe80137f0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.704+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1949237940 shutdown_connections 2024-05-27T07:15:22.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.704+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474cf4340 0x563474cf1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.704+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x563474d03bf0 0x563474d06010 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.704+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474cf1290 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.704+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1949237940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474c56a70 0x563474cf17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.704+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1949237940 >> 172.21.15.43:0/1949237940 conn(0x563474c04410 msgr2=0x563474ce53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:22.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.704+0000 7fcbdbe00700 0 client.15648 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:22.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.705+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1949237940 shutdown_connections 2024-05-27T07:15:22.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.705+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1949237940 wait complete. 2024-05-27T07:15:22.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.705+0000 7fcc0f9abb80 20 client.15648 trim_cache size 0 max 16384 2024-05-27T07:15:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.730+0000 7fcc0f9abb80 1 Processor -- start 2024-05-27T07:15:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.730+0000 7fcc0f9abb80 1 -- start start 2024-05-27T07:15:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.730+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474ce57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.730+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474cfff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.730+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474cf4340 0x563474d0b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.730+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc00ecc0 con 0x563474b6aec0 2024-05-27T07:15:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.730+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc010800 con 0x563474c56a70 2024-05-27T07:15:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.730+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc00f390 con 0x563474cf4340 2024-05-27T07:15:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.731+0000 7fcbf9e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474cfff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.731+0000 7fcbf9e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474cfff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:53792/0 (socket says 172.21.15.43:53792) 2024-05-27T07:15:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.732+0000 7fcbf9e00700 1 -- 172.21.15.43:0/2528172815 learned_addr learned my addr 172.21.15.43:0/2528172815 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.732+0000 7fcbf9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474ce57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.732+0000 7fcbf9e00700 1 -- 172.21.15.43:0/2528172815 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474cf4340 msgr2=0x563474d0b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.732+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/2528172815 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474cf4340 0x563474d0b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.732+0000 7fcbf9e00700 1 -- 172.21.15.43:0/2528172815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 msgr2=0x563474ce57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.732+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/2528172815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474ce57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.732+0000 7fcbf9e00700 1 -- 172.21.15.43:0/2528172815 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563474cebde0 con 0x563474c56a70 2024-05-27T07:15:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.732+0000 7fcbf9400700 1 --2- 172.21.15.43:0/2528172815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474ce57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.734+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/2528172815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474cfff50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcbec022d80 tx=0x7fcbec005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d501c3bef9b0bd31 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.734+0000 7fcbe6a00700 1 -- 172.21.15.43:0/2528172815 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec02f040 con 0x563474c56a70 2024-05-27T07:15:22.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.734+0000 7fcbe6a00700 1 -- 172.21.15.43:0/2528172815 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbec030040 con 0x563474c56a70 2024-05-27T07:15:22.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.734+0000 7fcbe6a00700 1 -- 172.21.15.43:0/2528172815 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec00baa0 con 0x563474c56a70 2024-05-27T07:15:22.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.736+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/2528172815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 msgr2=0x563474cfff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:22.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.736+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/2528172815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474cfff50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcbec022d80 tx=0x7fcbec005040 comp rx=0 tx=0).stop 2024-05-27T07:15:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.736+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/2528172815 shutdown_connections 2024-05-27T07:15:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.736+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/2528172815 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474cf4340 0x563474d0b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.736+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/2528172815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474b6aec0 0x563474ce57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.736+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/2528172815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474cfff50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.736+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/2528172815 >> 172.21.15.43:0/2528172815 conn(0x563474c04410 msgr2=0x563474ce9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.736+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/2528172815 shutdown_connections 2024-05-27T07:15:22.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.737+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/2528172815 wait complete. 2024-05-27T07:15:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.737+0000 7fcc0f9abb80 1 -- start start 2024-05-27T07:15:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.738+0000 7fcc0f9abb80 1 Processor -- start 2024-05-27T07:15:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.739+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474b6aec0 0x563474d679c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.739+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474d67f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.739+0000 7fcc0f9abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474cf4340 0x563474cf8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.739+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc010130 con 0x563474cf4340 2024-05-27T07:15:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.739+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc00fa60 con 0x563474c56a70 2024-05-27T07:15:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.739+0000 7fcc0f9abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc00cc40 con 0x563474b6aec0 2024-05-27T07:15:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.739+0000 7fcc0f9abb80 10 client.0 _ll_register_callbacks cb 0x563474bb1cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.740+0000 7fcbf9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474cf4340 0x563474cf8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.740+0000 7fcbf9400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474b6aec0 0x563474d679c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.740+0000 7fcbf9e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474d67f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.740+0000 7fcbf9400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474b6aec0 0x563474d679c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:52968/0 (socket says 172.21.15.43:52968) 2024-05-27T07:15:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.740+0000 7fcbf9e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474d67f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:53808/0 (socket says 172.21.15.43:53808) 2024-05-27T07:15:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.740+0000 7fcbf9400700 1 -- 172.21.15.43:0/1856122999 learned_addr learned my addr 172.21.15.43:0/1856122999 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.741+0000 7fcbf9400700 1 -- 172.21.15.43:0/1856122999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474b6aec0 msgr2=0x563474d679c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.741+0000 7fcbf9400700 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474b6aec0 0x563474d679c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.741+0000 7fcbf9400700 1 -- 172.21.15.43:0/1856122999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 msgr2=0x563474d67f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.741+0000 7fcbf9400700 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474d67f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.741+0000 7fcbf9400700 1 -- 172.21.15.43:0/1856122999 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563474cf2080 con 0x563474cf4340 2024-05-27T07:15:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.742+0000 7fcbf9400700 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474cf4340 0x563474cf8640 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8007d00 tx=0x7fcbe8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4dee0239542f10 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.742+0000 7fcbdbe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.743+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1856122999 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe80052c0 con 0x563474cf4340 2024-05-27T07:15:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.743+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1856122999 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbec030040 con 0x563474cf4340 2024-05-27T07:15:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.743+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1856122999 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fcbe802f060 con 0x563474cf4340 2024-05-27T07:15:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.744+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1856122999 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcbe8036050 con 0x563474cf4340 2024-05-27T07:15:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.744+0000 7fcc0f9abb80 10 client.15654 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.744+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1856122999 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563474d6eed0 con 0x563474cf4340 2024-05-27T07:15:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.745+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1856122999 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcbec030040 con 0x563474cf4340 2024-05-27T07:15:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.745+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1856122999 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcbe8026480 con 0x563474cf4340 2024-05-27T07:15:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.746+0000 7fcc0f9abb80 10 client.15654 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.746+0000 7fcc0f9abb80 10 client.15654 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:15:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.746+0000 7fcc0f9abb80 20 client.15654 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.746+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x563474d6f5b0 0x563474d719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.746+0000 7fcc0f9abb80 4 client.15654 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:15:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.746+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1856122999 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563474c19e10 con 0x563474d6f5b0 2024-05-27T07:15:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.746+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x563474d6f5b0 0x563474d719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.748+0000 7fcbf9e00700 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x563474d6f5b0 0x563474d719d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df51a4155493e446 server_cookie=89dfd82b6655180b in_seq=0 out_seq=0 2024-05-27T07:15:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.748+0000 7fcbdbe00700 10 client.15654 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.752+0000 7fcbdbe00700 1 -- 172.21.15.43:0/1856122999 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563474c19e10 con 0x563474d6f5b0 2024-05-27T07:15:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.752+0000 7fcbdbe00700 10 client.15654 handle_command_reply: tid=0 2024-05-27T07:15:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.752+0000 7fcc0f9abb80 1 client.15654 shutdown 2024-05-27T07:15:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.753+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1856122999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474cf4340 msgr2=0x563474cf8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.753+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474cf4340 0x563474cf8640 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8007d00 tx=0x7fcbe8000bc0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.754+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1856122999 shutdown_connections 2024-05-27T07:15:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.754+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x563474d6f5b0 0x563474d719d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.754+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563474cf4340 0x563474cf8640 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.754+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563474b6aec0 0x563474d679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.754+0000 7fcc0f9abb80 1 --2- 172.21.15.43:0/1856122999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563474c56a70 0x563474d67f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.754+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1856122999 >> 172.21.15.43:0/1856122999 conn(0x563474c04410 msgr2=0x563474ce9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.754+0000 7fcbdbe00700 0 client.15654 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.755+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1856122999 shutdown_connections 2024-05-27T07:15:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.755+0000 7fcc0f9abb80 1 -- 172.21.15.43:0/1856122999 wait complete. 2024-05-27T07:15:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.756+0000 7fcc0f9abb80 20 client.15654 trim_cache size 0 max 16384 2024-05-27T07:15:22.748 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:15:22.748 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:15:22.748 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "2f4f7078-795f-4d11-99a2-07bf8d855062", 2024-05-27T07:15:22.748 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:15:22.748 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:15:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.757+0000 7fcbfb200700 1 -- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fcbe00b2c10 msgr2=0x7fcbe00b5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.757+0000 7fcbfb200700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fcbe00b2c10 0x7fcbe00b5120 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40df7e0 tx=0x7fcbf4003f70 comp rx=0 tx=0).stop 2024-05-27T07:15:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.757+0000 7fcbfb200700 1 -- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbfc10e6f0 msgr2=0x7fcbfc1a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.757+0000 7fcbfb200700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbfc10e6f0 0x7fcbfc1a15c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400efc0 tx=0x7fcbf400c3a0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.757+0000 7fcbfb200700 1 -- 172.21.15.43:0/4076014729 shutdown_connections 2024-05-27T07:15:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.757+0000 7fcbfb200700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fcbe00b2c10 0x7fcbe00b5120 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.757+0000 7fcbfb200700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbfc10e6f0 0x7fcbfc1a15c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.757+0000 7fcbfb200700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbfc105280 0x7fcbfc19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.757+0000 7fcbfb200700 1 --2- 172.21.15.43:0/4076014729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcbfc105bd0 0x7fcbfc19d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.757+0000 7fcbfb200700 1 -- 172.21.15.43:0/4076014729 >> 172.21.15.43:0/4076014729 conn(0x7fcbfc078ab0 msgr2=0x7fcbfc102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.759+0000 7fcbfb200700 1 -- 172.21.15.43:0/4076014729 shutdown_connections 2024-05-27T07:15:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:22.759+0000 7fcbfb200700 1 -- 172.21.15.43:0/4076014729 wait complete. 2024-05-27T07:15:23.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:15:24.083 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:24.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:23 smithi043 ceph-mon[31531]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:24.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:23 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:15:24.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:23 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:24.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:23 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:15:24.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:23 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:23 smithi132 ceph-mon[35283]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:23 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:15:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:23 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:23 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:15:24.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:23 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:24.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:23 smithi179 ceph-mon[35888]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:24.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:23 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:15:24.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:23 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:24.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:23 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:15:24.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:23 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:26.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:25 smithi043 ceph-mon[31531]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:26.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:25 smithi132 ceph-mon[35283]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:25 smithi179 ceph-mon[35888]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:26.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.681+0000 7feec9000700 1 -- 172.21.15.43:0/3118256501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feec41045e0 msgr2=0x7feec41049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:26.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.681+0000 7feec9000700 1 --2- 172.21.15.43:0/3118256501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feec41045e0 0x7feec41049f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7feeb40099c0 tx=0x7feeb4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.682+0000 7feec9000700 1 -- 172.21.15.43:0/3118256501 shutdown_connections 2024-05-27T07:15:26.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.682+0000 7feec9000700 1 --2- 172.21.15.43:0/3118256501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec4109860 0x7feec41111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.682+0000 7feec9000700 1 --2- 172.21.15.43:0/3118256501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feec41045e0 0x7feec41049f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.682+0000 7feec9000700 1 --2- 172.21.15.43:0/3118256501 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feec4108f10 0x7feec4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.682+0000 7feec9000700 1 -- 172.21.15.43:0/3118256501 >> 172.21.15.43:0/3118256501 conn(0x7feec4100030 msgr2=0x7feec4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:26.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.683+0000 7feec9000700 1 -- 172.21.15.43:0/3118256501 shutdown_connections 2024-05-27T07:15:26.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.683+0000 7feec9000700 1 -- 172.21.15.43:0/3118256501 wait complete. 2024-05-27T07:15:26.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.685+0000 7feec9000700 1 Processor -- start 2024-05-27T07:15:26.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.685+0000 7feec9000700 1 -- start start 2024-05-27T07:15:26.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.686+0000 7feec9000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec41045e0 0x7feec419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.686+0000 7feec9000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feec4108f10 0x7feec419d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.686+0000 7feec9000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feec4109860 0x7feec41a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.686+0000 7feec9000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feec4114390 con 0x7feec4109860 2024-05-27T07:15:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.686+0000 7feec9000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7feec4114210 con 0x7feec41045e0 2024-05-27T07:15:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.686+0000 7feec9000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7feec4114510 con 0x7feec4108f10 2024-05-27T07:15:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.686+0000 7feec2800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feec4109860 0x7feec41a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.686+0000 7feec2800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feec4109860 0x7feec41a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47172/0 (socket says 172.21.15.43:47172) 2024-05-27T07:15:26.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.687+0000 7feec2800700 1 -- 172.21.15.43:0/3320841750 learned_addr learned my addr 172.21.15.43:0/3320841750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:26.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.687+0000 7feec1e00700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec41045e0 0x7feec419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.687+0000 7feec1400700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feec4108f10 0x7feec419d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.687+0000 7feec1e00700 1 -- 172.21.15.43:0/3320841750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feec4108f10 msgr2=0x7feec419d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:26.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.687+0000 7feec1e00700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feec4108f10 0x7feec419d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.687+0000 7feec1e00700 1 -- 172.21.15.43:0/3320841750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feec4109860 msgr2=0x7feec41a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:26.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.687+0000 7feec1e00700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feec4109860 0x7feec41a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.687+0000 7feec1e00700 1 -- 172.21.15.43:0/3320841750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec41a1e70 con 0x7feec41045e0 2024-05-27T07:15:26.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.688+0000 7feec2800700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feec4109860 0x7feec41a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:26.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.688+0000 7feec1e00700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec41045e0 0x7feec419cfa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feeb40098e0 tx=0x7feeb40057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:26.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.689+0000 7feebaa00700 1 -- 172.21.15.43:0/3320841750 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4027070 con 0x7feec41045e0 2024-05-27T07:15:26.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.689+0000 7feec9000700 1 -- 172.21.15.43:0/3320841750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec41a2100 con 0x7feec41045e0 2024-05-27T07:15:26.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.689+0000 7feec9000700 1 -- 172.21.15.43:0/3320841750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec410b110 con 0x7feec41045e0 2024-05-27T07:15:26.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.689+0000 7feebaa00700 1 -- 172.21.15.43:0/3320841750 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feeb400fc80 con 0x7feec41045e0 2024-05-27T07:15:26.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.689+0000 7feebaa00700 1 -- 172.21.15.43:0/3320841750 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4022d90 con 0x7feec41045e0 2024-05-27T07:15:26.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.696+0000 7feed7a5ab80 1 Processor -- start 2024-05-27T07:15:26.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.696+0000 7feed7a5ab80 1 -- start start 2024-05-27T07:15:26.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.696+0000 7feebaa00700 1 -- 172.21.15.43:0/3320841750 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) ==== 149885+0+0 (secure 0 0 0) 0x7feeb400f7f0 con 0x7feec41045e0 2024-05-27T07:15:26.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.697+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d0251087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.697+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d025102870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.697+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 0x55d0251031c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.697+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d025030e40 con 0x55d024f83e40 2024-05-27T07:15:26.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.697+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d0250209e0 con 0x55d025102db0 2024-05-27T07:15:26.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.697+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d025024a20 con 0x55d025108d30 2024-05-27T07:15:26.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.698+0000 7feec1400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 0x55d0251031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.698+0000 7feec2800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d0251087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.698+0000 7feec1400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 0x55d0251031c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41692/0 (socket says 172.21.15.43:41692) 2024-05-27T07:15:26.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.698+0000 7feec1400700 1 -- 172.21.15.43:0/236275704 learned_addr learned my addr 172.21.15.43:0/236275704 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:26.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.698+0000 7feec1e00700 1 --2- 172.21.15.43:0/236275704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d025102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.698+0000 7feec2800700 1 -- 172.21.15.43:0/236275704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 msgr2=0x55d025102870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:26.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.698+0000 7feec2800700 1 --2- 172.21.15.43:0/236275704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d025102870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.698+0000 7feec2800700 1 -- 172.21.15.43:0/236275704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 msgr2=0x55d0251031c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:26.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.698+0000 7feec2800700 1 --2- 172.21.15.43:0/236275704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 0x55d0251031c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.698+0000 7feec2800700 1 -- 172.21.15.43:0/236275704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d024fe9420 con 0x55d024f83e40 2024-05-27T07:15:26.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.699+0000 7feec2800700 1 --2- 172.21.15.43:0/236275704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d0251087f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7feebc0076c0 tx=0x7feebc0091b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e840ef995d644d98 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:26.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.699+0000 7feebaa00700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feea80b2b10 0x7feea80b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.699+0000 7feebaa00700 1 -- 172.21.15.43:0/3320841750 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7feeb40e3670 con 0x7feec41045e0 2024-05-27T07:15:26.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.700+0000 7feeaf400700 1 -- 172.21.15.43:0/236275704 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc004bb0 con 0x55d024f83e40 2024-05-27T07:15:26.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.700+0000 7feec1e00700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feea80b2b10 0x7feea80b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.700+0000 7feeaf400700 1 -- 172.21.15.43:0/236275704 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feebc004d50 con 0x55d024f83e40 2024-05-27T07:15:26.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.700+0000 7feeaf400700 1 -- 172.21.15.43:0/236275704 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc004bb0 con 0x55d024f83e40 2024-05-27T07:15:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.701+0000 7feec1e00700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feea80b2b10 0x7feea80b5020 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feeb40e0930 tx=0x7feeb40e9530 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.701+0000 7feed7a5ab80 1 -- 172.21.15.43:0/236275704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 msgr2=0x55d0251087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.701+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/236275704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d0251087f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7feebc0076c0 tx=0x7feebc0091b0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.702+0000 7feed7a5ab80 1 -- 172.21.15.43:0/236275704 shutdown_connections 2024-05-27T07:15:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.702+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/236275704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 0x55d0251031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.702+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/236275704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d0251087f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.702+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/236275704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d025102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.702+0000 7feed7a5ab80 1 -- 172.21.15.43:0/236275704 >> 172.21.15.43:0/236275704 conn(0x55d024fe3fc0 msgr2=0x55d0250fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.702+0000 7feed7a5ab80 1 -- 172.21.15.43:0/236275704 shutdown_connections 2024-05-27T07:15:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.702+0000 7feed7a5ab80 1 -- 172.21.15.43:0/236275704 wait complete. 2024-05-27T07:15:26.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.703+0000 7feed7a5ab80 1 -- start start 2024-05-27T07:15:26.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.703+0000 7feed7a5ab80 1 Processor -- start 2024-05-27T07:15:26.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.704+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02510c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.704+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 0x55d02510c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.704+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02510ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.704+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d024f26fa0 con 0x55d024f83e40 2024-05-27T07:15:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.704+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d024fc1760 con 0x55d025102db0 2024-05-27T07:15:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.704+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d02510f800 con 0x55d025108d30 2024-05-27T07:15:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.704+0000 7feed7a5ab80 10 client.0 _ll_register_callbacks cb 0x55d024fcac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.705+0000 7feec2800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02510c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.705+0000 7feec2800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02510c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47194/0 (socket says 172.21.15.43:47194) 2024-05-27T07:15:26.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.705+0000 7feec2800700 1 -- 172.21.15.43:0/1549191042 learned_addr learned my addr 172.21.15.43:0/1549191042 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:26.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.705+0000 7feec1400700 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 0x55d02510c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.705+0000 7feec2800700 1 -- 172.21.15.43:0/1549191042 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 msgr2=0x55d02510ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:26.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.705+0000 7feec2800700 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02510ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.705+0000 7feec2800700 1 -- 172.21.15.43:0/1549191042 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 msgr2=0x55d02510c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:26.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.705+0000 7feec2800700 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 0x55d02510c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.705+0000 7feec2800700 1 -- 172.21.15.43:0/1549191042 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d024f708d0 con 0x55d024f83e40 2024-05-27T07:15:26.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.706+0000 7feec2800700 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02510c450 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7feebc0096e0 tx=0x7feebc014ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6559d140c20c2e59 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:26.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.706+0000 7feea3e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:26.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.706+0000 7feea3e00700 1 -- 172.21.15.43:0/1549191042 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc016720 con 0x55d024f83e40 2024-05-27T07:15:26.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.706+0000 7feea3e00700 1 -- 172.21.15.43:0/1549191042 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7feebc0168c0 con 0x55d024f83e40 2024-05-27T07:15:26.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.706+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1549191042 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feebc004d50 con 0x55d024f83e40 2024-05-27T07:15:26.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.707+0000 7feea3e00700 1 -- 172.21.15.43:0/1549191042 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feebc02f050 con 0x55d024f83e40 2024-05-27T07:15:26.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.707+0000 7feed7a5ab80 10 client.15666 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:26.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.707+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1549191042 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d024ff1060 con 0x55d024f83e40 2024-05-27T07:15:26.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.707+0000 7feea3e00700 1 -- 172.21.15.43:0/1549191042 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feebc01e410 con 0x55d024f83e40 2024-05-27T07:15:26.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.708+0000 7feea3e00700 1 -- 172.21.15.43:0/1549191042 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7feebc0294f0 con 0x55d024f83e40 2024-05-27T07:15:26.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.708+0000 7feed7a5ab80 10 client.15666 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:26.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.708+0000 7feed7a5ab80 10 client.15666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:15:26.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.708+0000 7feed7a5ab80 20 client.15666 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:26.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.708+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d02511c640 0x55d02511ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.708+0000 7feed7a5ab80 4 client.15666 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:26.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.708+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1549191042 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feebc004d50 con 0x55d02511c640 2024-05-27T07:15:26.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.709+0000 7feec1400700 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d02511c640 0x55d02511ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.710+0000 7feea3e00700 10 client.15666 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:26.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.710+0000 7feec1400700 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d02511c640 0x55d02511ea60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef03602ed135a1e4 server_cookie=ada8d90d19e32414 in_seq=0 out_seq=0 2024-05-27T07:15:26.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feea3e00700 1 -- 172.21.15.43:0/1549191042 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7feebc004d50 con 0x55d02511c640 2024-05-27T07:15:26.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feea3e00700 10 client.15666 handle_command_reply: tid=0 2024-05-27T07:15:26.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feed7a5ab80 1 client.15666 shutdown 2024-05-27T07:15:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1549191042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 msgr2=0x55d02510c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02510c450 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7feebc0096e0 tx=0x7feebc014ed0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1549191042 shutdown_connections 2024-05-27T07:15:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02510ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d02511c640 0x55d02511ea60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02510c450 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1549191042 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d025102db0 0x55d02510c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1549191042 >> 172.21.15.43:0/1549191042 conn(0x55d024fe3fc0 msgr2=0x55d0250fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:26.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.713+0000 7feea3e00700 0 client.15666 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:26.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.714+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1549191042 shutdown_connections 2024-05-27T07:15:26.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.714+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1549191042 wait complete. 2024-05-27T07:15:26.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.714+0000 7feed7a5ab80 20 client.15666 trim_cache size 0 max 16384 2024-05-27T07:15:26.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.757+0000 7feed7a5ab80 1 Processor -- start 2024-05-27T07:15:26.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.757+0000 7feed7a5ab80 1 -- start start 2024-05-27T07:15:26.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.757+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d024f83e40 0x55d025102870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.757+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d025102db0 0x55d025118b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.757+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02510ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.757+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feea400ecc0 con 0x55d025102db0 2024-05-27T07:15:26.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.757+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7feea4010800 con 0x55d024f83e40 2024-05-27T07:15:26.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.757+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7feea400f390 con 0x55d025108d30 2024-05-27T07:15:26.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.758+0000 7feec1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d025102db0 0x55d025118b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.758+0000 7feec1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d025102db0 0x55d025118b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47196/0 (socket says 172.21.15.43:47196) 2024-05-27T07:15:26.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.758+0000 7feec1400700 1 -- 172.21.15.43:0/306729392 learned_addr learned my addr 172.21.15.43:0/306729392 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:26.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.758+0000 7feec2800700 1 --2- 172.21.15.43:0/306729392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d024f83e40 0x55d025102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.758+0000 7feec2800700 1 --2- 172.21.15.43:0/306729392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02510ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.758+0000 7feec1400700 1 -- 172.21.15.43:0/306729392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 msgr2=0x55d02510ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:26.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.758+0000 7feec1400700 1 --2- 172.21.15.43:0/306729392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02510ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.758+0000 7feec1400700 1 -- 172.21.15.43:0/306729392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d024f83e40 msgr2=0x55d025102870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:26.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.758+0000 7feec1400700 1 --2- 172.21.15.43:0/306729392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d024f83e40 0x55d025102870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.758+0000 7feec1400700 1 -- 172.21.15.43:0/306729392 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02510a9c0 con 0x55d025102db0 2024-05-27T07:15:26.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.759+0000 7feec2800700 1 --2- 172.21.15.43:0/306729392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02510ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:26.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.759+0000 7feec1400700 1 --2- 172.21.15.43:0/306729392 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d025102db0 0x55d025118b00 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7feeb0017010 tx=0x7feeb00073d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2067bb2a750a62 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:26.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.759+0000 7feeaea00700 1 -- 172.21.15.43:0/306729392 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0007e50 con 0x55d025102db0 2024-05-27T07:15:26.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.760+0000 7feeaea00700 1 -- 172.21.15.43:0/306729392 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feeb0003b10 con 0x55d025102db0 2024-05-27T07:15:26.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.760+0000 7feeaea00700 1 -- 172.21.15.43:0/306729392 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0021ad0 con 0x55d025102db0 2024-05-27T07:15:26.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.761+0000 7feed7a5ab80 1 -- 172.21.15.43:0/306729392 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d025102db0 msgr2=0x55d025118b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:26.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.761+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/306729392 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d025102db0 0x55d025118b00 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7feeb0017010 tx=0x7feeb00073d0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.761+0000 7feed7a5ab80 1 -- 172.21.15.43:0/306729392 shutdown_connections 2024-05-27T07:15:26.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.761+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/306729392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02510ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.761+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/306729392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d024f83e40 0x55d025102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.761+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/306729392 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d025102db0 0x55d025118b00 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.761+0000 7feed7a5ab80 1 -- 172.21.15.43:0/306729392 >> 172.21.15.43:0/306729392 conn(0x55d024fe3fc0 msgr2=0x55d025106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:26.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.761+0000 7feed7a5ab80 1 -- 172.21.15.43:0/306729392 shutdown_connections 2024-05-27T07:15:26.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.761+0000 7feed7a5ab80 1 -- 172.21.15.43:0/306729392 wait complete. 2024-05-27T07:15:26.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.762+0000 7feed7a5ab80 1 -- start start 2024-05-27T07:15:26.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.762+0000 7feed7a5ab80 1 Processor -- start 2024-05-27T07:15:26.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.763+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02511f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.763+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02511fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.763+0000 7feed7a5ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0251200c0 0x55d025185eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.763+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feea4010130 con 0x55d024f83e40 2024-05-27T07:15:26.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.763+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7feea400fa60 con 0x55d0251200c0 2024-05-27T07:15:26.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.763+0000 7feed7a5ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7feea400cc40 con 0x55d025108d30 2024-05-27T07:15:26.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.763+0000 7feed7a5ab80 10 client.0 _ll_register_callbacks cb 0x55d024fcac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:26.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.764+0000 7feec1400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02511fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.764+0000 7feec2800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02511f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.764+0000 7feec1400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02511fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:55972/0 (socket says 172.21.15.43:55972) 2024-05-27T07:15:26.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.764+0000 7feec1400700 1 -- 172.21.15.43:0/1053066157 learned_addr learned my addr 172.21.15.43:0/1053066157 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:26.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.764+0000 7feec2800700 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0251200c0 0x55d025185eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.764+0000 7feec2800700 1 -- 172.21.15.43:0/1053066157 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 msgr2=0x55d02511fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:26.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.764+0000 7feec2800700 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02511fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.764+0000 7feec2800700 1 -- 172.21.15.43:0/1053066157 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0251200c0 msgr2=0x55d025185eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:26.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.764+0000 7feec2800700 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0251200c0 0x55d025185eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.764+0000 7feec2800700 1 -- 172.21.15.43:0/1053066157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0251864c0 con 0x55d024f83e40 2024-05-27T07:15:26.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.765+0000 7feec1400700 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02511fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:26.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.765+0000 7feec2800700 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02511f640 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7feebc0298f0 tx=0x7feebc007cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156439e0f89735a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:26.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.765+0000 7feea3e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:26.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.765+0000 7feea3e00700 1 -- 172.21.15.43:0/1053066157 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feebc016260 con 0x55d024f83e40 2024-05-27T07:15:26.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.765+0000 7feea3e00700 1 -- 172.21.15.43:0/1053066157 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7feebc01e410 con 0x55d024f83e40 2024-05-27T07:15:26.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.765+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1053066157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeb0003b10 con 0x55d024f83e40 2024-05-27T07:15:26.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.766+0000 7feea3e00700 1 -- 172.21.15.43:0/1053066157 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feebc03d050 con 0x55d024f83e40 2024-05-27T07:15:26.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.766+0000 7feed7a5ab80 10 client.15672 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:26.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.766+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1053066157 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d025187730 con 0x55d024f83e40 2024-05-27T07:15:26.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.766+0000 7feea3e00700 1 -- 172.21.15.43:0/1053066157 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feeb0003b10 con 0x55d024f83e40 2024-05-27T07:15:26.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.767+0000 7feea3e00700 1 -- 172.21.15.43:0/1053066157 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7feebc0291c0 con 0x55d024f83e40 2024-05-27T07:15:26.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.767+0000 7feed7a5ab80 10 client.15672 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:26.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.767+0000 7feed7a5ab80 10 client.15672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:15:26.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.767+0000 7feed7a5ab80 20 client.15672 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:26.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.767+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d025187d80 0x55d02518a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:26.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.767+0000 7feed7a5ab80 4 client.15672 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:15:26.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.767+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1053066157 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d025032d90 con 0x55d025187d80 2024-05-27T07:15:26.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.768+0000 7feec1400700 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d025187d80 0x55d02518a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:26.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.769+0000 7feea3e00700 10 client.15672 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:26.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.769+0000 7feec1400700 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d025187d80 0x55d02518a1a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=453ef8f5bd9ae517 server_cookie=7456401ac3be1862 in_seq=0 out_seq=0 2024-05-27T07:15:26.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.769+0000 7feea3e00700 1 -- 172.21.15.43:0/1053066157 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d025032d90 con 0x55d025187d80 2024-05-27T07:15:26.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.769+0000 7feea3e00700 10 client.15672 handle_command_reply: tid=0 2024-05-27T07:15:26.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.769+0000 7feed7a5ab80 1 client.15672 shutdown 2024-05-27T07:15:26.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1053066157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 msgr2=0x55d02511f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:26.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02511f640 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7feebc0298f0 tx=0x7feebc007cd0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1053066157 shutdown_connections 2024-05-27T07:15:26.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0251200c0 0x55d025185eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d025187d80 0x55d02518a1a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d024f83e40 0x55d02511f640 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 1 --2- 172.21.15.43:0/1053066157 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d025108d30 0x55d02511fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1053066157 >> 172.21.15.43:0/1053066157 conn(0x55d024fe3fc0 msgr2=0x55d02510cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:26.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feea3e00700 0 client.15672 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:26.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1053066157 shutdown_connections 2024-05-27T07:15:26.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 1 -- 172.21.15.43:0/1053066157 wait complete. 2024-05-27T07:15:26.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.770+0000 7feed7a5ab80 20 client.15672 trim_cache size 0 max 16384 2024-05-27T07:15:26.762 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:15:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:15:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:15:26.762 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.771+0000 7feec9000700 1 -- 172.21.15.43:0/3320841750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feea80b2b10 msgr2=0x7feea80b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.771+0000 7feec9000700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feea80b2b10 0x7feea80b5020 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feeb40e0930 tx=0x7feeb40e9530 comp rx=0 tx=0).stop 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.771+0000 7feec9000700 1 -- 172.21.15.43:0/3320841750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec41045e0 msgr2=0x7feec419cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.771+0000 7feec9000700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec41045e0 0x7feec419cfa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feeb40098e0 tx=0x7feeb40057c0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.772+0000 7feec9000700 1 -- 172.21.15.43:0/3320841750 shutdown_connections 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.772+0000 7feec9000700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feea80b2b10 0x7feea80b5020 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.772+0000 7feec9000700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feec4109860 0x7feec41a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.772+0000 7feec9000700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec41045e0 0x7feec419cfa0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.772+0000 7feec9000700 1 --2- 172.21.15.43:0/3320841750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feec4108f10 0x7feec419d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.772+0000 7feec9000700 1 -- 172.21.15.43:0/3320841750 >> 172.21.15.43:0/3320841750 conn(0x7feec4100030 msgr2=0x7feec410a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.772+0000 7feec9000700 1 -- 172.21.15.43:0/3320841750 shutdown_connections 2024-05-27T07:15:26.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:26.772+0000 7feec9000700 1 -- 172.21.15.43:0/3320841750 wait complete. 2024-05-27T07:15:27.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:26 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:15:27.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:15:27.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:15:27.724 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:15:27.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:15:28.328 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:28.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:28 smithi132 ceph-mon[35283]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 13 op/s 2024-05-27T07:15:28.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:28 smithi132 ceph-mon[35283]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-27T07:15:28.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:28 smithi043 ceph-mon[31531]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 13 op/s 2024-05-27T07:15:28.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:28 smithi043 ceph-mon[31531]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-27T07:15:28.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:28 smithi179 ceph-mon[35888]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 13 op/s 2024-05-27T07:15:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:28 smithi179 ceph-mon[35888]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-05-27T07:15:30.103 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:30 smithi043 ceph-mon[31531]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-05-27T07:15:30.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:30 smithi132 ceph-mon[35283]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-05-27T07:15:30.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:30 smithi179 ceph-mon[35888]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-05-27T07:15:30.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.735+0000 7f79c7e00700 1 -- 172.21.15.43:0/3053947860 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79c8109670 msgr2=0x7f79c8110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:30.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.735+0000 7f79c7e00700 1 --2- 172.21.15.43:0/3053947860 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79c8109670 0x7f79c8110fd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f79c000b0d0 tx=0x7f79c000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:15:30.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.735+0000 7f79c7e00700 1 -- 172.21.15.43:0/3053947860 shutdown_connections 2024-05-27T07:15:30.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.735+0000 7f79c7e00700 1 --2- 172.21.15.43:0/3053947860 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79c8109670 0x7f79c8110fd0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:30.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.735+0000 7f79c7e00700 1 --2- 172.21.15.43:0/3053947860 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79c81043f0 0x7f79c8104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:30.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.735+0000 7f79c7e00700 1 --2- 172.21.15.43:0/3053947860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f79c8108d20 0x7f79c8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:30.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.735+0000 7f79c7e00700 1 -- 172.21.15.43:0/3053947860 >> 172.21.15.43:0/3053947860 conn(0x7f79c8079d00 msgr2=0x7f79c807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:30.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.735+0000 7f79c7e00700 1 -- 172.21.15.43:0/3053947860 shutdown_connections 2024-05-27T07:15:30.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.735+0000 7f79c7e00700 1 -- 172.21.15.43:0/3053947860 wait complete. 2024-05-27T07:15:30.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.737+0000 7f79c7e00700 1 Processor -- start 2024-05-27T07:15:30.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.737+0000 7f79c7e00700 1 -- start start 2024-05-27T07:15:30.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.738+0000 7f79c7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79c81043f0 0x7f79c81a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:30.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.738+0000 7f79c7e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79c8108d20 0x7f79c81a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:30.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.738+0000 7f79c7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f79c8109670 0x7f79c81acf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:30.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.738+0000 7f79c7e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f79c81141a0 con 0x7f79c81043f0 2024-05-27T07:15:30.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.738+0000 7f79c7e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8114020 con 0x7f79c8109670 2024-05-27T07:15:30.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.738+0000 7f79c7e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8114320 con 0x7f79c8108d20 2024-05-27T07:15:30.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.739+0000 7f79c6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79c81043f0 0x7f79c81a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:30.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.739+0000 7f79c6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79c81043f0 0x7f79c81a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47214/0 (socket says 172.21.15.43:47214) 2024-05-27T07:15:30.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.739+0000 7f79c6a00700 1 -- 172.21.15.43:0/1406319096 learned_addr learned my addr 172.21.15.43:0/1406319096 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:30.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.739+0000 7f79c6000700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79c8108d20 0x7f79c81a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:30.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.739+0000 7f79c7400700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f79c8109670 0x7f79c81acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:30.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.740+0000 7f79c6a00700 1 -- 172.21.15.43:0/1406319096 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79c8108d20 msgr2=0x7f79c81a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:30.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.740+0000 7f79c6a00700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79c8108d20 0x7f79c81a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:30.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.740+0000 7f79c6a00700 1 -- 172.21.15.43:0/1406319096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f79c8109670 msgr2=0x7f79c81acf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:30.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.740+0000 7f79c6a00700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f79c8109670 0x7f79c81acf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:30.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.740+0000 7f79c6a00700 1 -- 172.21.15.43:0/1406319096 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c81ad440 con 0x7f79c81043f0 2024-05-27T07:15:30.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.740+0000 7f79c7400700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f79c8109670 0x7f79c81acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:30.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.740+0000 7f79c6000700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79c8108d20 0x7f79c81a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:30.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.740+0000 7f79c6a00700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79c81043f0 0x7f79c81a1e80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f79b800d950 tx=0x7f79b800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:30.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.741+0000 7f79cca00700 1 -- 172.21.15.43:0/1406319096 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b8016070 con 0x7f79c81043f0 2024-05-27T07:15:30.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.741+0000 7f79cca00700 1 -- 172.21.15.43:0/1406319096 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79b8010460 con 0x7f79c81043f0 2024-05-27T07:15:30.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.741+0000 7f79cca00700 1 -- 172.21.15.43:0/1406319096 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b800f620 con 0x7f79c81043f0 2024-05-27T07:15:30.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.741+0000 7f79c7e00700 1 -- 172.21.15.43:0/1406319096 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c81ad610 con 0x7f79c81043f0 2024-05-27T07:15:30.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.741+0000 7f79c7e00700 1 -- 172.21.15.43:0/1406319096 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c81adaa0 con 0x7f79c81043f0 2024-05-27T07:15:30.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.743+0000 7f79c7e00700 1 -- 172.21.15.43:0/1406319096 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c81058a0 con 0x7f79c81043f0 2024-05-27T07:15:30.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.747+0000 7f79cca00700 1 -- 172.21.15.43:0/1406319096 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f79b8019070 con 0x7f79c81043f0 2024-05-27T07:15:30.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.750+0000 7f79cca00700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79ac0b2a70 0x7f79ac0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:30.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.750+0000 7f79cca00700 1 -- 172.21.15.43:0/1406319096 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f79b80e2540 con 0x7f79c81043f0 2024-05-27T07:15:30.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.751+0000 7f79c6000700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79ac0b2a70 0x7f79ac0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:30.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.752+0000 7f79c6000700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79ac0b2a70 0x7f79ac0b4f80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f79b4005e10 tx=0x7f79b4005d60 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:30.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:30.754+0000 7f79cca00700 1 -- 172.21.15.43:0/1406319096 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f79b80a9bb0 con 0x7f79c81043f0 2024-05-27T07:15:31.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.192+0000 7f79c7e00700 1 -- 172.21.15.43:0/1406319096 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79c81addc0 con 0x7f79c81043f0 2024-05-27T07:15:31.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.193+0000 7f79cca00700 1 -- 172.21.15.43:0/1406319096 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11938 (secure 0 0 0) 0x7f79b80a94b0 con 0x7f79c81043f0 2024-05-27T07:15:31.185 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:15:31.185 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:13:52.990426+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/4061292168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":4061292168},{"type":"v1","addr":"172.21.15.179:6837","nonce":4061292168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34484,34448,34424,24484,34403]},"id":1}]} 2024-05-27T07:15:31.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.197+0000 7f79c7e00700 1 -- 172.21.15.43:0/1406319096 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79ac0b2a70 msgr2=0x7f79ac0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:31.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.197+0000 7f79c7e00700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79ac0b2a70 0x7f79ac0b4f80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f79b4005e10 tx=0x7f79b4005d60 comp rx=0 tx=0).stop 2024-05-27T07:15:31.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.197+0000 7f79c7e00700 1 -- 172.21.15.43:0/1406319096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79c81043f0 msgr2=0x7f79c81a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:31.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.197+0000 7f79c7e00700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79c81043f0 0x7f79c81a1e80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f79b800d950 tx=0x7f79b800dd10 comp rx=0 tx=0).stop 2024-05-27T07:15:31.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.197+0000 7f79c7e00700 1 -- 172.21.15.43:0/1406319096 shutdown_connections 2024-05-27T07:15:31.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.197+0000 7f79c7e00700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79ac0b2a70 0x7f79ac0b4f80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.197+0000 7f79c7e00700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f79c8109670 0x7f79c81acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.197+0000 7f79c7e00700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79c81043f0 0x7f79c81a1e80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.197+0000 7f79c7e00700 1 --2- 172.21.15.43:0/1406319096 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79c8108d20 0x7f79c81a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:31.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.197+0000 7f79c7e00700 1 -- 172.21.15.43:0/1406319096 >> 172.21.15.43:0/1406319096 conn(0x7f79c8079d00 msgr2=0x7f79c8102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:31.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.198+0000 7f79c7e00700 1 -- 172.21.15.43:0/1406319096 shutdown_connections 2024-05-27T07:15:31.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:31.198+0000 7f79c7e00700 1 -- 172.21.15.43:0/1406319096 wait complete. 2024-05-27T07:15:31.191 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 30 2024-05-27T07:15:32.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:15:32.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:32 smithi132 ceph-mon[35283]: pgmap v51: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 13 op/s 2024-05-27T07:15:32.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:32 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1406319096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:15:32.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:32 smithi043 ceph-mon[31531]: pgmap v51: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 13 op/s 2024-05-27T07:15:32.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:32 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1406319096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:15:32.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:32 smithi179 ceph-mon[35888]: pgmap v51: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 13 op/s 2024-05-27T07:15:32.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:32 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1406319096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:15:32.656 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:34.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:34 smithi132 ceph-mon[35283]: pgmap v52: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-05-27T07:15:34.404 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:34 smithi043 ceph-mon[31531]: pgmap v52: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-05-27T07:15:34.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:34 smithi179 ceph-mon[35888]: pgmap v52: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-05-27T07:15:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:34.998+0000 7f64f0c00700 1 -- 172.21.15.43:0/1384932455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64ec110ca0 msgr2=0x7f64ec1130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:34.998+0000 7f64f0c00700 1 --2- 172.21.15.43:0/1384932455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64ec110ca0 0x7f64ec1130c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f64e400b0a0 tx=0x7f64e400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:34.998+0000 7f64f0c00700 1 -- 172.21.15.43:0/1384932455 shutdown_connections 2024-05-27T07:15:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:34.998+0000 7f64f0c00700 1 --2- 172.21.15.43:0/1384932455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64ec110ca0 0x7f64ec1130c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:34.998+0000 7f64f0c00700 1 --2- 172.21.15.43:0/1384932455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64ec1030f0 0x7f64ec103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:34.998+0000 7f64f0c00700 1 --2- 172.21.15.43:0/1384932455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec103a40 0x7f64ec110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:34.998+0000 7f64f0c00700 1 -- 172.21.15.43:0/1384932455 >> 172.21.15.43:0/1384932455 conn(0x7f64ec0fedb0 msgr2=0x7f64ec1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:34.999+0000 7f64f0c00700 1 -- 172.21.15.43:0/1384932455 shutdown_connections 2024-05-27T07:15:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:34.999+0000 7f64f0c00700 1 -- 172.21.15.43:0/1384932455 wait complete. 2024-05-27T07:15:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.000+0000 7f64f0c00700 1 Processor -- start 2024-05-27T07:15:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.001+0000 7f64f0c00700 1 -- start start 2024-05-27T07:15:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.001+0000 7f64f0c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64ec1030f0 0x7f64ec19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.001+0000 7f64f0c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64ec103a40 0x7f64ec19d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.001+0000 7f64f0c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec110ca0 0x7f64ec1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.001+0000 7f64f0c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec116560 con 0x7f64ec103a40 2024-05-27T07:15:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.001+0000 7f64f0c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec1163e0 con 0x7f64ec110ca0 2024-05-27T07:15:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.001+0000 7f64f0c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec1166e0 con 0x7f64ec1030f0 2024-05-27T07:15:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.002+0000 7f64e9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64ec1030f0 0x7f64ec19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.002+0000 7f64e9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64ec1030f0 0x7f64ec19cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:34202/0 (socket says 172.21.15.43:34202) 2024-05-27T07:15:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.002+0000 7f64e9e00700 1 -- 172.21.15.43:0/644586213 learned_addr learned my addr 172.21.15.43:0/644586213 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.002+0000 7f64ea800700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec110ca0 0x7f64ec1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.002+0000 7f64e9400700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64ec103a40 0x7f64ec19d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.003+0000 7f64e9e00700 1 -- 172.21.15.43:0/644586213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec110ca0 msgr2=0x7f64ec1a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.003+0000 7f64e9e00700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec110ca0 0x7f64ec1a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.003+0000 7f64e9e00700 1 -- 172.21.15.43:0/644586213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64ec103a40 msgr2=0x7f64ec19d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.003+0000 7f64e9e00700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64ec103a40 0x7f64ec19d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.003+0000 7f64e9e00700 1 -- 172.21.15.43:0/644586213 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ec1a1dc0 con 0x7f64ec1030f0 2024-05-27T07:15:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.003+0000 7f64e9e00700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64ec1030f0 0x7f64ec19cef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00d950 tx=0x7f64dc00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.004+0000 7f64e2a00700 1 -- 172.21.15.43:0/644586213 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc016070 con 0x7f64ec1030f0 2024-05-27T07:15:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.004+0000 7f64f0c00700 1 -- 172.21.15.43:0/644586213 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ec1a2050 con 0x7f64ec1030f0 2024-05-27T07:15:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.004+0000 7f64f0c00700 1 -- 172.21.15.43:0/644586213 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ec1a99d0 con 0x7f64ec1030f0 2024-05-27T07:15:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.004+0000 7f64e2a00700 1 -- 172.21.15.43:0/644586213 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64dc009dd0 con 0x7f64ec1030f0 2024-05-27T07:15:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.004+0000 7f64e2a00700 1 -- 172.21.15.43:0/644586213 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc020060 con 0x7f64ec1030f0 2024-05-27T07:15:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.009+0000 7f64e2a00700 1 -- 172.21.15.43:0/644586213 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f64dc019070 con 0x7f64ec1030f0 2024-05-27T07:15:35.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.011+0000 7f64ff64eb80 1 Processor -- start 2024-05-27T07:15:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.011+0000 7f64ff64eb80 1 -- start start 2024-05-27T07:15:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.011+0000 7f64e2a00700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f64d00b2af0 0x7f64d00b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.011+0000 7f64e2a00700 1 -- 172.21.15.43:0/644586213 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f64dc0e2480 con 0x7f64ec1030f0 2024-05-27T07:15:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.012+0000 7f64ea800700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f64d00b2af0 0x7f64d00b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.012+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade589e40 0x561ade70e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.012+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade70ed30 0x561ade708870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.012+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 0x561ade7091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.012+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561ade636e40 con 0x561ade70ed30 2024-05-27T07:15:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.012+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561ade6269e0 con 0x561ade589e40 2024-05-27T07:15:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.012+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561ade62aa20 con 0x561ade708db0 2024-05-27T07:15:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.013+0000 7f64e9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade70ed30 0x561ade708870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.013+0000 7f64e9e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade589e40 0x561ade70e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.014+0000 7f64e9e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade589e40 0x561ade70e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:60618/0 (socket says 172.21.15.43:60618) 2024-05-27T07:15:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.014+0000 7f64e9e00700 1 -- 172.21.15.43:0/1698830177 learned_addr learned my addr 172.21.15.43:0/1698830177 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.014+0000 7f64ea800700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f64d00b2af0 0x7f64d00b5000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009e60 tx=0x7f64e4006210 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.014+0000 7f64ea800700 1 --2- 172.21.15.43:0/1698830177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 0x561ade7091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.015+0000 7f64e9e00700 1 -- 172.21.15.43:0/1698830177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 msgr2=0x561ade7091c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.015+0000 7f64e9e00700 1 --2- 172.21.15.43:0/1698830177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 0x561ade7091c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.015+0000 7f64e9e00700 1 -- 172.21.15.43:0/1698830177 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade70ed30 msgr2=0x561ade708870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:35.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.015+0000 7f64e9e00700 1 --2- 172.21.15.43:0/1698830177 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade70ed30 0x561ade708870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.015+0000 7f64e9e00700 1 -- 172.21.15.43:0/1698830177 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ade5ef420 con 0x561ade589e40 2024-05-27T07:15:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.015+0000 7f64e9400700 1 --2- 172.21.15.43:0/1698830177 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade70ed30 0x561ade708870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.016+0000 7f64e9e00700 1 --2- 172.21.15.43:0/1698830177 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade589e40 0x561ade70e7f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0a90b0 tx=0x7f64dc0a9b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aec8ad4bb6b27acc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.017+0000 7f64d7400700 1 -- 172.21.15.43:0/1698830177 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc0ad760 con 0x561ade589e40 2024-05-27T07:15:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.017+0000 7f64d7400700 1 -- 172.21.15.43:0/1698830177 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64dc0b2210 con 0x561ade589e40 2024-05-27T07:15:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.017+0000 7f64d7400700 1 -- 172.21.15.43:0/1698830177 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc0a8610 con 0x561ade589e40 2024-05-27T07:15:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.018+0000 7f64ff64eb80 1 -- 172.21.15.43:0/1698830177 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade589e40 msgr2=0x561ade70e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:35.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.018+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/1698830177 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade589e40 0x561ade70e7f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0a90b0 tx=0x7f64dc0a9b30 comp rx=0 tx=0).stop 2024-05-27T07:15:35.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.022+0000 7f64ff64eb80 1 -- 172.21.15.43:0/1698830177 shutdown_connections 2024-05-27T07:15:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.022+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/1698830177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 0x561ade7091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.022+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/1698830177 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade589e40 0x561ade70e7f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.022+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/1698830177 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade70ed30 0x561ade708870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.022+0000 7f64ff64eb80 1 -- 172.21.15.43:0/1698830177 >> 172.21.15.43:0/1698830177 conn(0x561ade5e9fc0 msgr2=0x561ade7038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.022+0000 7f64ff64eb80 1 -- 172.21.15.43:0/1698830177 shutdown_connections 2024-05-27T07:15:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.022+0000 7f64ff64eb80 1 -- 172.21.15.43:0/1698830177 wait complete. 2024-05-27T07:15:35.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.023+0000 7f64ff64eb80 1 -- start start 2024-05-27T07:15:35.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.024+0000 7f64ff64eb80 1 Processor -- start 2024-05-27T07:15:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.024+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 0x561ade7104e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.024+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 0x561ade710a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.024+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade70ed30 0x561ade710f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.024+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561ade52cfa0 con 0x561ade589e40 2024-05-27T07:15:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.024+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561ade5c7760 con 0x561ade70ed30 2024-05-27T07:15:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.024+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561ade7267b0 con 0x561ade708db0 2024-05-27T07:15:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.025+0000 7f64ff64eb80 10 client.0 _ll_register_callbacks cb 0x561ade5d0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.025+0000 7f64e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 0x561ade7104e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.025+0000 7f64e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 0x561ade7104e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35872/0 (socket says 172.21.15.43:35872) 2024-05-27T07:15:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.025+0000 7f64e9e00700 1 -- 172.21.15.43:0/3727000142 learned_addr learned my addr 172.21.15.43:0/3727000142 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.025+0000 7f64ea800700 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade70ed30 0x561ade710f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.025+0000 7f64e9400700 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 0x561ade710a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.026+0000 7f64e9e00700 1 -- 172.21.15.43:0/3727000142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 msgr2=0x561ade710a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.026+0000 7f64e9e00700 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 0x561ade710a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.026+0000 7f64e9e00700 1 -- 172.21.15.43:0/3727000142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade70ed30 msgr2=0x561ade710f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.026+0000 7f64e9e00700 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade70ed30 0x561ade710f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.026+0000 7f64e9e00700 1 -- 172.21.15.43:0/3727000142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ade5768d0 con 0x561ade589e40 2024-05-27T07:15:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.026+0000 7f64e9400700 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 0x561ade710a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.027+0000 7f64e9e00700 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 0x561ade7104e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0a9340 tx=0x7f64dc0b1fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987f5b27c0783988 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:35.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.027+0000 7f64cbe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:35.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.027+0000 7f64cbe00700 1 -- 172.21.15.43:0/3727000142 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc0c8060 con 0x561ade589e40 2024-05-27T07:15:35.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.027+0000 7f64cbe00700 1 -- 172.21.15.43:0/3727000142 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f64dc0a9820 con 0x561ade589e40 2024-05-27T07:15:35.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.027+0000 7f64ff64eb80 1 -- 172.21.15.43:0/3727000142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64dc0b2210 con 0x561ade589e40 2024-05-27T07:15:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.028+0000 7f64cbe00700 1 -- 172.21.15.43:0/3727000142 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f64dc0ad760 con 0x561ade589e40 2024-05-27T07:15:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.028+0000 7f64ff64eb80 10 client.15690 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.028+0000 7f64ff64eb80 1 -- 172.21.15.43:0/3727000142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561ade5f7060 con 0x561ade589e40 2024-05-27T07:15:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.028+0000 7f64cbe00700 1 -- 172.21.15.43:0/3727000142 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64dc0b2210 con 0x561ade589e40 2024-05-27T07:15:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.029+0000 7f64cbe00700 1 -- 172.21.15.43:0/3727000142 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f64dc0b0ae0 con 0x561ade589e40 2024-05-27T07:15:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.029+0000 7f64ff64eb80 10 client.15690 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.029+0000 7f64ff64eb80 10 client.15690 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:15:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.029+0000 7f64ff64eb80 20 client.15690 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.029+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561ade733560 0x561ade735980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.029+0000 7f64ff64eb80 4 client.15690 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.029+0000 7f64ff64eb80 1 -- 172.21.15.43:0/3727000142 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ade638d90 con 0x561ade733560 2024-05-27T07:15:35.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.030+0000 7f64e9400700 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561ade733560 0x561ade735980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.031+0000 7f64e9400700 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561ade733560 0x561ade735980 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=abe49822a9f05efb server_cookie=c89df8da5404db49 in_seq=0 out_seq=0 2024-05-27T07:15:35.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.031+0000 7f64cbe00700 10 client.15690 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:15:35.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.033+0000 7f64cbe00700 1 -- 172.21.15.43:0/3727000142 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561ade638d90 con 0x561ade733560 2024-05-27T07:15:35.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.033+0000 7f64cbe00700 10 client.15690 handle_command_reply: tid=0 2024-05-27T07:15:35.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.033+0000 7f64ff64eb80 1 client.15690 shutdown 2024-05-27T07:15:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.034+0000 7f64ff64eb80 1 -- 172.21.15.43:0/3727000142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 msgr2=0x561ade7104e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.034+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 0x561ade7104e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0a9340 tx=0x7f64dc0b1fb0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.034+0000 7f64ff64eb80 1 -- 172.21.15.43:0/3727000142 shutdown_connections 2024-05-27T07:15:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.034+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561ade733560 0x561ade735980 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.034+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade70ed30 0x561ade710f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.034+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 0x561ade7104e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.034+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/3727000142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade708db0 0x561ade710a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.034+0000 7f64ff64eb80 1 -- 172.21.15.43:0/3727000142 >> 172.21.15.43:0/3727000142 conn(0x561ade5e9fc0 msgr2=0x561ade703580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:35.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.034+0000 7f64cbe00700 0 client.15690 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:15:35.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.035+0000 7f64ff64eb80 1 -- 172.21.15.43:0/3727000142 shutdown_connections 2024-05-27T07:15:35.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.035+0000 7f64ff64eb80 1 -- 172.21.15.43:0/3727000142 wait complete. 2024-05-27T07:15:35.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.035+0000 7f64ff64eb80 20 client.15690 trim_cache size 0 max 16384 2024-05-27T07:15:35.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.066+0000 7f64ff64eb80 1 Processor -- start 2024-05-27T07:15:35.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.066+0000 7f64ff64eb80 1 -- start start 2024-05-27T07:15:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.067+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade589e40 0x561ade72fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.067+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade708db0 0x561ade70ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.067+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade70ed30 0x561ade7109d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.067+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc00eca0 con 0x561ade708db0 2024-05-27T07:15:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.067+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc0107e0 con 0x561ade70ed30 2024-05-27T07:15:35.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.067+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc00f370 con 0x561ade589e40 2024-05-27T07:15:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.068+0000 7f64e9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade589e40 0x561ade72fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.068+0000 7f64e9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade589e40 0x561ade72fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:34228/0 (socket says 172.21.15.43:34228) 2024-05-27T07:15:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.068+0000 7f64e9e00700 1 -- 172.21.15.43:0/1777273226 learned_addr learned my addr 172.21.15.43:0/1777273226 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.068+0000 7f64e9400700 1 --2- 172.21.15.43:0/1777273226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade708db0 0x561ade70ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.068+0000 7f64ea800700 1 --2- 172.21.15.43:0/1777273226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade70ed30 0x561ade7109d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.069+0000 7f64e9400700 1 -- 172.21.15.43:0/1777273226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade589e40 msgr2=0x561ade72fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:35.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.069+0000 7f64e9400700 1 --2- 172.21.15.43:0/1777273226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade589e40 0x561ade72fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.069+0000 7f64e9400700 1 -- 172.21.15.43:0/1777273226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade70ed30 msgr2=0x561ade7109d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:35.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.069+0000 7f64e9400700 1 --2- 172.21.15.43:0/1777273226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade70ed30 0x561ade7109d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.069+0000 7f64e9400700 1 -- 172.21.15.43:0/1777273226 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ade70f140 con 0x561ade708db0 2024-05-27T07:15:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.069+0000 7f64e9400700 1 --2- 172.21.15.43:0/1777273226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade708db0 0x561ade70ab60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x561ade626ae0 tx=0x7f64d80137f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b1dcdaf7715f46 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.070+0000 7f64d6a00700 1 -- 172.21.15.43:0/1777273226 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d8013c80 con 0x561ade708db0 2024-05-27T07:15:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.070+0000 7f64d6a00700 1 -- 172.21.15.43:0/1777273226 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64d8013e20 con 0x561ade708db0 2024-05-27T07:15:35.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.070+0000 7f64d6a00700 1 -- 172.21.15.43:0/1777273226 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d801d430 con 0x561ade708db0 2024-05-27T07:15:35.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.071+0000 7f64ff64eb80 1 -- 172.21.15.43:0/1777273226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade708db0 msgr2=0x561ade70ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:35.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.071+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/1777273226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade708db0 0x561ade70ab60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x561ade626ae0 tx=0x7f64d80137f0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.071+0000 7f64ff64eb80 1 -- 172.21.15.43:0/1777273226 shutdown_connections 2024-05-27T07:15:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.071+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/1777273226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade70ed30 0x561ade7109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.071+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/1777273226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade589e40 0x561ade72fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.071+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/1777273226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade708db0 0x561ade70ab60 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.071+0000 7f64ff64eb80 1 -- 172.21.15.43:0/1777273226 >> 172.21.15.43:0/1777273226 conn(0x561ade5e9fc0 msgr2=0x561ade7102c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.072+0000 7f64ff64eb80 1 -- 172.21.15.43:0/1777273226 shutdown_connections 2024-05-27T07:15:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.072+0000 7f64ff64eb80 1 -- 172.21.15.43:0/1777273226 wait complete. 2024-05-27T07:15:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.072+0000 7f64ff64eb80 1 -- start start 2024-05-27T07:15:35.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.073+0000 7f64ff64eb80 1 Processor -- start 2024-05-27T07:15:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.074+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 0x561ade738d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.074+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade708db0 0x561ade736150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.074+0000 7f64ff64eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade70ed30 0x561ade736690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.074+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc010110 con 0x561ade589e40 2024-05-27T07:15:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.074+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc00fa40 con 0x561ade708db0 2024-05-27T07:15:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.074+0000 7f64ff64eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc00cc20 con 0x561ade70ed30 2024-05-27T07:15:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.074+0000 7f64ff64eb80 10 client.0 _ll_register_callbacks cb 0x561ade5d0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.074+0000 7f64e9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade708db0 0x561ade736150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.075+0000 7f64ea800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade70ed30 0x561ade736690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.075+0000 7f64ea800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade70ed30 0x561ade736690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:34244/0 (socket says 172.21.15.43:34244) 2024-05-27T07:15:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.075+0000 7f64ea800700 1 -- 172.21.15.43:0/2156681070 learned_addr learned my addr 172.21.15.43:0/2156681070 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.075+0000 7f64e9e00700 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 0x561ade738d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.076+0000 7f64ea800700 1 -- 172.21.15.43:0/2156681070 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade708db0 msgr2=0x561ade736150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.076+0000 7f64ea800700 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade708db0 0x561ade736150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.076+0000 7f64ea800700 1 -- 172.21.15.43:0/2156681070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 msgr2=0x561ade738d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.076+0000 7f64ea800700 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 0x561ade738d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.076+0000 7f64ea800700 1 -- 172.21.15.43:0/2156681070 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ade736bd0 con 0x561ade70ed30 2024-05-27T07:15:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.077+0000 7f64ea800700 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade70ed30 0x561ade736690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f64e400b0a0 tx=0x7f64e40099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa27ca21abf3a690 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.077+0000 7f64cbe00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:15:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.077+0000 7f64cbe00700 1 -- 172.21.15.43:0/2156681070 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4038060 con 0x561ade70ed30 2024-05-27T07:15:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.077+0000 7f64cbe00700 1 -- 172.21.15.43:0/2156681070 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f64e40050c0 con 0x561ade70ed30 2024-05-27T07:15:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.077+0000 7f64ff64eb80 1 -- 172.21.15.43:0/2156681070 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64d8013e20 con 0x561ade70ed30 2024-05-27T07:15:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.078+0000 7f64cbe00700 1 -- 172.21.15.43:0/2156681070 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f64e40052c0 con 0x561ade70ed30 2024-05-27T07:15:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.078+0000 7f64ff64eb80 10 client.25393 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.078+0000 7f64ff64eb80 1 -- 172.21.15.43:0/2156681070 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561ade7a6160 con 0x561ade70ed30 2024-05-27T07:15:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.079+0000 7f64cbe00700 1 -- 172.21.15.43:0/2156681070 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64e40058f0 con 0x561ade70ed30 2024-05-27T07:15:35.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.079+0000 7f64cbe00700 1 -- 172.21.15.43:0/2156681070 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f64e4011070 con 0x561ade70ed30 2024-05-27T07:15:35.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.079+0000 7f64ff64eb80 10 client.25393 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.079+0000 7f64ff64eb80 10 client.25393 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:15:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.079+0000 7f64ff64eb80 20 client.25393 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.079+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561ade7a6870 0x561ade7a8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.079+0000 7f64ff64eb80 4 client.25393 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:15:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.079+0000 7f64ff64eb80 1 -- 172.21.15.43:0/2156681070 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f64d8013e20 con 0x561ade7a6870 2024-05-27T07:15:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.080+0000 7f64e9e00700 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561ade7a6870 0x561ade7a8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:35.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.081+0000 7f64e9e00700 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561ade7a6870 0x561ade7a8c90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=99f3a4352d77416d server_cookie=d6b0a285723cb7cc in_seq=0 out_seq=0 2024-05-27T07:15:35.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.081+0000 7f64cbe00700 10 client.25393 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:15:35.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64cbe00700 1 -- 172.21.15.43:0/2156681070 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f64d8013e20 con 0x561ade7a6870 2024-05-27T07:15:35.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64cbe00700 10 client.25393 handle_command_reply: tid=0 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64ff64eb80 1 client.25393 shutdown 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64ff64eb80 1 -- 172.21.15.43:0/2156681070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade70ed30 msgr2=0x561ade736690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade70ed30 0x561ade736690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f64e400b0a0 tx=0x7f64e40099e0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64ff64eb80 1 -- 172.21.15.43:0/2156681070 shutdown_connections 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561ade7a6870 0x561ade7a8c90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561ade70ed30 0x561ade736690 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561ade589e40 0x561ade738d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64ff64eb80 1 --2- 172.21.15.43:0/2156681070 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561ade708db0 0x561ade736150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.082+0000 7f64ff64eb80 1 -- 172.21.15.43:0/2156681070 >> 172.21.15.43:0/2156681070 conn(0x561ade5e9fc0 msgr2=0x561ade70a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.083+0000 7f64cbe00700 0 client.25393 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.083+0000 7f64ff64eb80 1 -- 172.21.15.43:0/2156681070 shutdown_connections 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.083+0000 7f64ff64eb80 1 -- 172.21.15.43:0/2156681070 wait complete. 2024-05-27T07:15:35.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.083+0000 7f64ff64eb80 20 client.25393 trim_cache size 0 max 16384 2024-05-27T07:15:35.075 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.084+0000 7f64f0c00700 1 -- 172.21.15.43:0/644586213 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f64d00b2af0 msgr2=0x7f64d00b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.084+0000 7f64f0c00700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f64d00b2af0 0x7f64d00b5000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009e60 tx=0x7f64e4006210 comp rx=0 tx=0).stop 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.084+0000 7f64f0c00700 1 -- 172.21.15.43:0/644586213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64ec1030f0 msgr2=0x7f64ec19cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.084+0000 7f64f0c00700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64ec1030f0 0x7f64ec19cef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00d950 tx=0x7f64dc00dc60 comp rx=0 tx=0).stop 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.085+0000 7f64f0c00700 1 -- 172.21.15.43:0/644586213 shutdown_connections 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.085+0000 7f64f0c00700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f64d00b2af0 0x7f64d00b5000 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.085+0000 7f64f0c00700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64ec110ca0 0x7f64ec1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.085+0000 7f64f0c00700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64ec1030f0 0x7f64ec19cef0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.085+0000 7f64f0c00700 1 --2- 172.21.15.43:0/644586213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64ec103a40 0x7f64ec19d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.085+0000 7f64f0c00700 1 -- 172.21.15.43:0/644586213 >> 172.21.15.43:0/644586213 conn(0x7f64ec0fedb0 msgr2=0x7f64ec112700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.085+0000 7f64f0c00700 1 -- 172.21.15.43:0/644586213 shutdown_connections 2024-05-27T07:15:35.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:35.085+0000 7f64f0c00700 1 -- 172.21.15.43:0/644586213 wait complete. 2024-05-27T07:15:35.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:15:36.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:36 smithi043 ceph-mon[31531]: pgmap v53: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-05-27T07:15:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:36 smithi132 ceph-mon[35283]: pgmap v53: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-05-27T07:15:36.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:36 smithi179 ceph-mon[35888]: pgmap v53: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-05-27T07:15:36.588 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:38.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:38 smithi132 ceph-mon[35283]: pgmap v54: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 13 op/s 2024-05-27T07:15:38.447 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:38 smithi043 ceph-mon[31531]: pgmap v54: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 13 op/s 2024-05-27T07:15:38.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:38 smithi179 ceph-mon[35888]: pgmap v54: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 13 op/s 2024-05-27T07:15:39.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.182+0000 7f3608600700 1 -- 172.21.15.43:0/3420607190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36001045e0 msgr2=0x7f36001049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:39.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.182+0000 7f3608600700 1 --2- 172.21.15.43:0/3420607190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36001045e0 0x7f36001049f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f35f40099c0 tx=0x7f35f4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.182+0000 7f3608600700 1 -- 172.21.15.43:0/3420607190 shutdown_connections 2024-05-27T07:15:39.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.182+0000 7f3608600700 1 --2- 172.21.15.43:0/3420607190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3600109860 0x7f36001111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.182+0000 7f3608600700 1 --2- 172.21.15.43:0/3420607190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36001045e0 0x7f36001049f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.182+0000 7f3608600700 1 --2- 172.21.15.43:0/3420607190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3600108f10 0x7f3600109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.182+0000 7f3608600700 1 -- 172.21.15.43:0/3420607190 >> 172.21.15.43:0/3420607190 conn(0x7f3600100030 msgr2=0x7f3600102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:39.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.182+0000 7f3608600700 1 -- 172.21.15.43:0/3420607190 shutdown_connections 2024-05-27T07:15:39.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.182+0000 7f3608600700 1 -- 172.21.15.43:0/3420607190 wait complete. 2024-05-27T07:15:39.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.184+0000 7f3608600700 1 Processor -- start 2024-05-27T07:15:39.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.184+0000 7f3608600700 1 -- start start 2024-05-27T07:15:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.185+0000 7f3608600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36001045e0 0x7f3600199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.185+0000 7f3608600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3600108f10 0x7f3600199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.185+0000 7f3608600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3600109860 0x7f36001a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.185+0000 7f3608600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3600114390 con 0x7f36001045e0 2024-05-27T07:15:39.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.185+0000 7f3608600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3600114210 con 0x7f3600109860 2024-05-27T07:15:39.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.185+0000 7f3608600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3600114510 con 0x7f3600108f10 2024-05-27T07:15:39.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.185+0000 7f3605c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36001045e0 0x7f3600199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.185+0000 7f3605c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36001045e0 0x7f3600199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35906/0 (socket says 172.21.15.43:35906) 2024-05-27T07:15:39.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.186+0000 7f3605c00700 1 -- 172.21.15.43:0/3416514834 learned_addr learned my addr 172.21.15.43:0/3416514834 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:39.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.186+0000 7f3605200700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3600108f10 0x7f3600199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.186+0000 7f3606600700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3600109860 0x7f36001a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.186+0000 7f3605c00700 1 -- 172.21.15.43:0/3416514834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3600108f10 msgr2=0x7f3600199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:39.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.186+0000 7f3605c00700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3600108f10 0x7f3600199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.186+0000 7f3605c00700 1 -- 172.21.15.43:0/3416514834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3600109860 msgr2=0x7f36001a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:39.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.186+0000 7f3605c00700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3600109860 0x7f36001a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.186+0000 7f3605c00700 1 -- 172.21.15.43:0/3416514834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36001a0e00 con 0x7f36001045e0 2024-05-27T07:15:39.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.186+0000 7f3606600700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3600109860 0x7f36001a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:39.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.187+0000 7f3605c00700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36001045e0 0x7f3600199a30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f35f4000c00 tx=0x7f35f400f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:39.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.187+0000 7f35fe000700 1 -- 172.21.15.43:0/3416514834 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4027070 con 0x7f36001045e0 2024-05-27T07:15:39.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.187+0000 7f3608600700 1 -- 172.21.15.43:0/3416514834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36001a1090 con 0x7f36001045e0 2024-05-27T07:15:39.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.187+0000 7f3608600700 1 -- 172.21.15.43:0/3416514834 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36001a15c0 con 0x7f36001045e0 2024-05-27T07:15:39.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.188+0000 7f35fe000700 1 -- 172.21.15.43:0/3416514834 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35f400fda0 con 0x7f36001045e0 2024-05-27T07:15:39.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.188+0000 7f35fe000700 1 -- 172.21.15.43:0/3416514834 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4022e50 con 0x7f36001045e0 2024-05-27T07:15:39.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.193+0000 7f35fe000700 1 -- 172.21.15.43:0/3416514834 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f35f402a070 con 0x7f36001045e0 2024-05-27T07:15:39.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.194+0000 7f3617222b80 1 Processor -- start 2024-05-27T07:15:39.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.194+0000 7f3617222b80 1 -- start start 2024-05-27T07:15:39.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.194+0000 7f3617222b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 0x55c9427be7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.194+0000 7f3617222b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427b8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.195+0000 7f3617222b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427b91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.195+0000 7f3617222b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c9426e6e40 con 0x55c9427bed30 2024-05-27T07:15:39.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.195+0000 7f3617222b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c9426d69e0 con 0x55c942639e40 2024-05-27T07:15:39.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.195+0000 7f3617222b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c9426daa20 con 0x55c9427b8db0 2024-05-27T07:15:39.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.195+0000 7f3605c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.195+0000 7f3605c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427b8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35910/0 (socket says 172.21.15.43:35910) 2024-05-27T07:15:39.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.195+0000 7f3605c00700 1 -- 172.21.15.43:0/204408498 learned_addr learned my addr 172.21.15.43:0/204408498 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:39.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.195+0000 7f3606600700 1 --2- 172.21.15.43:0/204408498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 0x55c9427be7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.195+0000 7f3605200700 1 --2- 172.21.15.43:0/204408498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427b91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.196+0000 7f35fe000700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f35e80b2c10 0x7f35e80b5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.196+0000 7f3605200700 1 -- 172.21.15.43:0/204408498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 msgr2=0x55c9427be7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:39.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.196+0000 7f3605200700 1 --2- 172.21.15.43:0/204408498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 0x55c9427be7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.196+0000 7f3605200700 1 -- 172.21.15.43:0/204408498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 msgr2=0x55c9427b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:39.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.196+0000 7f35fe000700 1 -- 172.21.15.43:0/3416514834 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f35f40e34f0 con 0x7f36001045e0 2024-05-27T07:15:39.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.196+0000 7f3605200700 1 --2- 172.21.15.43:0/204408498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427b8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.196+0000 7f3605200700 1 -- 172.21.15.43:0/204408498 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c94269f420 con 0x55c9427b8db0 2024-05-27T07:15:39.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.196+0000 7f3605c00700 1 --2- 172.21.15.43:0/204408498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:39.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.197+0000 7f3605200700 1 --2- 172.21.15.43:0/204408498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427b91c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f35ec005950 tx=0x7f35ec00bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=644cdec2d4379ad9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:39.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.197+0000 7f35f2a00700 1 -- 172.21.15.43:0/204408498 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec010c70 con 0x55c9427b8db0 2024-05-27T07:15:39.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.197+0000 7f35f2a00700 1 -- 172.21.15.43:0/204408498 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35ec010e10 con 0x55c9427b8db0 2024-05-27T07:15:39.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.198+0000 7f35f2a00700 1 -- 172.21.15.43:0/204408498 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec005020 con 0x55c9427b8db0 2024-05-27T07:15:39.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.198+0000 7f3606600700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f35e80b2c10 0x7f35e80b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.199+0000 7f3617222b80 1 -- 172.21.15.43:0/204408498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 msgr2=0x55c9427b91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:39.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.199+0000 7f3617222b80 1 --2- 172.21.15.43:0/204408498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427b91c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f35ec005950 tx=0x7f35ec00bf60 comp rx=0 tx=0).stop 2024-05-27T07:15:39.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.199+0000 7f3617222b80 1 -- 172.21.15.43:0/204408498 shutdown_connections 2024-05-27T07:15:39.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.199+0000 7f3617222b80 1 --2- 172.21.15.43:0/204408498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427b91c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.199+0000 7f3617222b80 1 --2- 172.21.15.43:0/204408498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 0x55c9427be7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.199+0000 7f3617222b80 1 --2- 172.21.15.43:0/204408498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.199+0000 7f3617222b80 1 -- 172.21.15.43:0/204408498 >> 172.21.15.43:0/204408498 conn(0x55c942699fc0 msgr2=0x55c9427b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.199+0000 7f3617222b80 1 -- 172.21.15.43:0/204408498 shutdown_connections 2024-05-27T07:15:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.199+0000 7f3617222b80 1 -- 172.21.15.43:0/204408498 wait complete. 2024-05-27T07:15:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.199+0000 7f3606600700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f35e80b2c10 0x7f35e80b5120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55c9427b9b30 tx=0x7f35f8004ad0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.200+0000 7f3617222b80 1 -- start start 2024-05-27T07:15:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.200+0000 7f3617222b80 1 Processor -- start 2024-05-27T07:15:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.201+0000 7f3617222b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.201+0000 7f3617222b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.201+0000 7f3617222b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 0x55c9427c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.201+0000 7f3617222b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c9425dcfa0 con 0x55c942639e40 2024-05-27T07:15:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.201+0000 7f3617222b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c942677760 con 0x55c9427bed30 2024-05-27T07:15:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.201+0000 7f3617222b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c9427c5800 con 0x55c9427b8db0 2024-05-27T07:15:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.201+0000 7f3617222b80 10 client.0 _ll_register_callbacks cb 0x55c942680c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.202+0000 7f3605c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.202+0000 7f3605c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427c23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35926/0 (socket says 172.21.15.43:35926) 2024-05-27T07:15:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.202+0000 7f3605c00700 1 -- 172.21.15.43:0/408547833 learned_addr learned my addr 172.21.15.43:0/408547833 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.202+0000 7f3605200700 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.202+0000 7f3606600700 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 0x55c9427c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.203+0000 7f3605c00700 1 -- 172.21.15.43:0/408547833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 msgr2=0x55c9427c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.203+0000 7f3605c00700 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.203+0000 7f3605c00700 1 -- 172.21.15.43:0/408547833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 msgr2=0x55c9427c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.203+0000 7f3605c00700 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 0x55c9427c2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.203+0000 7f3605c00700 1 -- 172.21.15.43:0/408547833 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9426268d0 con 0x55c942639e40 2024-05-27T07:15:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.203+0000 7f3606600700 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 0x55c9427c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.203+0000 7f3605c00700 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427c23f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f35f40058e0 tx=0x7f35f4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a978b9ce43fcda92 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.203+0000 7f35e7400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.204+0000 7f35e7400700 1 -- 172.21.15.43:0/408547833 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f40a63b0 con 0x55c942639e40 2024-05-27T07:15:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.204+0000 7f3617222b80 1 -- 172.21.15.43:0/408547833 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35ec010e10 con 0x55c942639e40 2024-05-27T07:15:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.204+0000 7f35e7400700 1 -- 172.21.15.43:0/408547833 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f35f40aa750 con 0x55c942639e40 2024-05-27T07:15:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.204+0000 7f35e7400700 1 -- 172.21.15.43:0/408547833 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f35f40e0d60 con 0x55c942639e40 2024-05-27T07:15:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.204+0000 7f3617222b80 10 client.15708 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.204+0000 7f3617222b80 1 -- 172.21.15.43:0/408547833 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c9426a7060 con 0x55c942639e40 2024-05-27T07:15:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.205+0000 7f35e7400700 1 -- 172.21.15.43:0/408547833 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35ec010e10 con 0x55c942639e40 2024-05-27T07:15:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.205+0000 7f35e7400700 1 -- 172.21.15.43:0/408547833 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f35f40e3100 con 0x55c942639e40 2024-05-27T07:15:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.205+0000 7f3617222b80 10 client.15708 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.205+0000 7f3617222b80 10 client.15708 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:15:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.205+0000 7f3617222b80 20 client.15708 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:39.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.205+0000 7f3617222b80 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c9427d2640 0x55c9427d4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.205+0000 7f3617222b80 4 client.15708 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:39.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.205+0000 7f3617222b80 1 -- 172.21.15.43:0/408547833 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9426e8d90 con 0x55c9427d2640 2024-05-27T07:15:39.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.206+0000 7f3605200700 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c9427d2640 0x55c9427d4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.207+0000 7f3605200700 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c9427d2640 0x55c9427d4ab0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6c55fba9d0a47483 server_cookie=6452895dbc283cad in_seq=0 out_seq=0 2024-05-27T07:15:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.207+0000 7f35e7400700 10 client.15708 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:15:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.209+0000 7f35e7400700 1 -- 172.21.15.43:0/408547833 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c9426e8d90 con 0x55c9427d2640 2024-05-27T07:15:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.209+0000 7f35e7400700 10 client.15708 handle_command_reply: tid=0 2024-05-27T07:15:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.210+0000 7f3617222b80 1 client.15708 shutdown 2024-05-27T07:15:39.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.210+0000 7f3617222b80 1 -- 172.21.15.43:0/408547833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 msgr2=0x55c9427c23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:39.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.210+0000 7f3617222b80 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427c23f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f35f40058e0 tx=0x7f35f4009310 comp rx=0 tx=0).stop 2024-05-27T07:15:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.210+0000 7f3617222b80 1 -- 172.21.15.43:0/408547833 shutdown_connections 2024-05-27T07:15:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.210+0000 7f3617222b80 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c9427d2640 0x55c9427d4ab0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.210+0000 7f3617222b80 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 0x55c9427c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.210+0000 7f3617222b80 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427c23f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.210+0000 7f3617222b80 1 --2- 172.21.15.43:0/408547833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.210+0000 7f3617222b80 1 -- 172.21.15.43:0/408547833 >> 172.21.15.43:0/408547833 conn(0x55c942699fc0 msgr2=0x55c9427b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.210+0000 7f35e7400700 0 client.15708 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:15:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.211+0000 7f3617222b80 1 -- 172.21.15.43:0/408547833 shutdown_connections 2024-05-27T07:15:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.211+0000 7f3617222b80 1 -- 172.21.15.43:0/408547833 wait complete. 2024-05-27T07:15:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.211+0000 7f3617222b80 20 client.15708 trim_cache size 0 max 16384 2024-05-27T07:15:39.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.243+0000 7f3617222b80 1 Processor -- start 2024-05-27T07:15:39.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.243+0000 7f3617222b80 1 -- start start 2024-05-27T07:15:39.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.243+0000 7f3617222b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 0x55c9427b8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.243+0000 7f3617222b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427c2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.243+0000 7f3617222b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427c4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.243+0000 7f3617222b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f36001bf1e0 con 0x55c9427bed30 2024-05-27T07:15:39.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.243+0000 7f3617222b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f36001c0d20 con 0x55c942639e40 2024-05-27T07:15:39.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.243+0000 7f3617222b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f36001bf8b0 con 0x55c9427b8db0 2024-05-27T07:15:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.244+0000 7f3606600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427c4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.244+0000 7f3606600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427c4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35928/0 (socket says 172.21.15.43:35928) 2024-05-27T07:15:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.244+0000 7f3606600700 1 -- 172.21.15.43:0/2169958986 learned_addr learned my addr 172.21.15.43:0/2169958986 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.244+0000 7f3605200700 1 --2- 172.21.15.43:0/2169958986 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427c2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.244+0000 7f3605c00700 1 --2- 172.21.15.43:0/2169958986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 0x55c9427b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.244+0000 7f3606600700 1 -- 172.21.15.43:0/2169958986 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 msgr2=0x55c9427c2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.245+0000 7f3606600700 1 --2- 172.21.15.43:0/2169958986 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427c2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.245+0000 7f3606600700 1 -- 172.21.15.43:0/2169958986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 msgr2=0x55c9427b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.245+0000 7f3606600700 1 --2- 172.21.15.43:0/2169958986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 0x55c9427b8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.245+0000 7f3606600700 1 -- 172.21.15.43:0/2169958986 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9427c1160 con 0x55c9427bed30 2024-05-27T07:15:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.245+0000 7f3605c00700 1 --2- 172.21.15.43:0/2169958986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 0x55c9427b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.245+0000 7f3606600700 1 --2- 172.21.15.43:0/2169958986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427c4890 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f35f8017880 tx=0x7f35f8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c495a703e7bc6ba5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.246+0000 7f35f2000700 1 -- 172.21.15.43:0/2169958986 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f801d030 con 0x55c9427bed30 2024-05-27T07:15:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.246+0000 7f35f2000700 1 -- 172.21.15.43:0/2169958986 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35f8028070 con 0x55c9427bed30 2024-05-27T07:15:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.246+0000 7f35f2000700 1 -- 172.21.15.43:0/2169958986 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f8016870 con 0x55c9427bed30 2024-05-27T07:15:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.247+0000 7f3617222b80 1 -- 172.21.15.43:0/2169958986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 msgr2=0x55c9427c4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.247+0000 7f3617222b80 1 --2- 172.21.15.43:0/2169958986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427c4890 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f35f8017880 tx=0x7f35f8015000 comp rx=0 tx=0).stop 2024-05-27T07:15:39.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.247+0000 7f3617222b80 1 -- 172.21.15.43:0/2169958986 shutdown_connections 2024-05-27T07:15:39.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.247+0000 7f3617222b80 1 --2- 172.21.15.43:0/2169958986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c9427bed30 0x55c9427c4890 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.247+0000 7f3617222b80 1 --2- 172.21.15.43:0/2169958986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c942639e40 0x55c9427b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.247+0000 7f3617222b80 1 --2- 172.21.15.43:0/2169958986 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427c2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.247+0000 7f3617222b80 1 -- 172.21.15.43:0/2169958986 >> 172.21.15.43:0/2169958986 conn(0x55c942699fc0 msgr2=0x55c9427bc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:39.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.247+0000 7f3617222b80 1 -- 172.21.15.43:0/2169958986 shutdown_connections 2024-05-27T07:15:39.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.247+0000 7f3617222b80 1 -- 172.21.15.43:0/2169958986 wait complete. 2024-05-27T07:15:39.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.248+0000 7f3617222b80 1 -- start start 2024-05-27T07:15:39.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.248+0000 7f3617222b80 1 Processor -- start 2024-05-27T07:15:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.249+0000 7f3617222b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427d9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.249+0000 7f3617222b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427da350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.249+0000 7f3617222b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 0x55c9427da890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.249+0000 7f3617222b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f36001c0650 con 0x55c942639e40 2024-05-27T07:15:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.249+0000 7f3617222b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f36001bff80 con 0x55c9427bed30 2024-05-27T07:15:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.249+0000 7f3617222b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f36001bd160 con 0x55c9427b8db0 2024-05-27T07:15:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.249+0000 7f3617222b80 10 client.0 _ll_register_callbacks cb 0x55c942680c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.250+0000 7f3605c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427d9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.250+0000 7f3605c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427d9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35936/0 (socket says 172.21.15.43:35936) 2024-05-27T07:15:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.250+0000 7f3605c00700 1 -- 172.21.15.43:0/403862414 learned_addr learned my addr 172.21.15.43:0/403862414 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.250+0000 7f3606600700 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 0x55c9427da890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.250+0000 7f3605200700 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427da350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.250+0000 7f3605c00700 1 -- 172.21.15.43:0/403862414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 msgr2=0x55c9427da350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.250+0000 7f3605c00700 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427da350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.250+0000 7f3605c00700 1 -- 172.21.15.43:0/403862414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 msgr2=0x55c9427da890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.250+0000 7f3605c00700 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 0x55c9427da890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.250+0000 7f3605c00700 1 -- 172.21.15.43:0/403862414 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c94283c180 con 0x55c942639e40 2024-05-27T07:15:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.251+0000 7f3605c00700 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427d9e10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f35f40e6d80 tx=0x7f35f40098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c957c05e1aea35c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.251+0000 7f35e7400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.251+0000 7f35e7400700 1 -- 172.21.15.43:0/403862414 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f40a9c60 con 0x55c942639e40 2024-05-27T07:15:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.251+0000 7f3617222b80 1 -- 172.21.15.43:0/403862414 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35f8028070 con 0x55c942639e40 2024-05-27T07:15:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.252+0000 7f35e7400700 1 -- 172.21.15.43:0/403862414 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f35f40a9750 con 0x55c942639e40 2024-05-27T07:15:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.252+0000 7f35e7400700 1 -- 172.21.15.43:0/403862414 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f35f40a9e40 con 0x55c942639e40 2024-05-27T07:15:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.252+0000 7f3617222b80 10 client.15714 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.252+0000 7f3617222b80 1 -- 172.21.15.43:0/403862414 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c94283d360 con 0x55c942639e40 2024-05-27T07:15:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.252+0000 7f35e7400700 1 -- 172.21.15.43:0/403862414 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f35f8028070 con 0x55c942639e40 2024-05-27T07:15:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.252+0000 7f35e7400700 1 -- 172.21.15.43:0/403862414 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f35f40cb300 con 0x55c942639e40 2024-05-27T07:15:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.253+0000 7f3617222b80 10 client.15714 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.253+0000 7f3617222b80 10 client.15714 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:15:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.253+0000 7f3617222b80 20 client.15714 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.253+0000 7f3617222b80 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c94283d8e0 0x55c94283fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.253+0000 7f3617222b80 4 client.15714 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:15:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.253+0000 7f3617222b80 1 -- 172.21.15.43:0/403862414 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c9426e2570 con 0x55c94283d8e0 2024-05-27T07:15:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.253+0000 7f3605200700 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c94283d8e0 0x55c94283fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.254+0000 7f3605200700 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c94283d8e0 0x55c94283fd00 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f7411ffdcc36092e server_cookie=3db0cee04ef970f9 in_seq=0 out_seq=0 2024-05-27T07:15:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.254+0000 7f35e7400700 10 client.15714 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:15:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.255+0000 7f35e7400700 1 -- 172.21.15.43:0/403862414 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c9426e2570 con 0x55c94283d8e0 2024-05-27T07:15:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.255+0000 7f35e7400700 10 client.15714 handle_command_reply: tid=0 2024-05-27T07:15:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.255+0000 7f3617222b80 1 client.15714 shutdown 2024-05-27T07:15:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.255+0000 7f3617222b80 1 -- 172.21.15.43:0/403862414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 msgr2=0x55c9427d9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.255+0000 7f3617222b80 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427d9e10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f35f40e6d80 tx=0x7f35f40098f0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.256+0000 7f3617222b80 1 -- 172.21.15.43:0/403862414 shutdown_connections 2024-05-27T07:15:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.256+0000 7f3617222b80 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c94283d8e0 0x55c94283fd00 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.256+0000 7f3617222b80 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c9427bed30 0x55c9427da890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.256+0000 7f3617222b80 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c942639e40 0x55c9427d9e10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.256+0000 7f35e7400700 0 client.15714 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:15:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.256+0000 7f3617222b80 1 --2- 172.21.15.43:0/403862414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c9427b8db0 0x55c9427da350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.256+0000 7f3617222b80 1 -- 172.21.15.43:0/403862414 >> 172.21.15.43:0/403862414 conn(0x55c942699fc0 msgr2=0x55c9427bc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:39.248 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:15:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.256+0000 7f3617222b80 1 -- 172.21.15.43:0/403862414 shutdown_connections 2024-05-27T07:15:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.256+0000 7f3617222b80 1 -- 172.21.15.43:0/403862414 wait complete. 2024-05-27T07:15:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.256+0000 7f3617222b80 20 client.15714 trim_cache size 0 max 16384 2024-05-27T07:15:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 -- 172.21.15.43:0/3416514834 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f35e80b2c10 msgr2=0x7f35e80b5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f35e80b2c10 0x7f35e80b5120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55c9427b9b30 tx=0x7f35f8004ad0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 -- 172.21.15.43:0/3416514834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36001045e0 msgr2=0x7f3600199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36001045e0 0x7f3600199a30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f35f4000c00 tx=0x7f35f400f460 comp rx=0 tx=0).stop 2024-05-27T07:15:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 -- 172.21.15.43:0/3416514834 shutdown_connections 2024-05-27T07:15:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f35e80b2c10 0x7f35e80b5120 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3600109860 0x7f36001a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36001045e0 0x7f3600199a30 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 --2- 172.21.15.43:0/3416514834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3600108f10 0x7f3600199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 -- 172.21.15.43:0/3416514834 >> 172.21.15.43:0/3416514834 conn(0x7f3600100030 msgr2=0x7f360010a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:39.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 -- 172.21.15.43:0/3416514834 shutdown_connections 2024-05-27T07:15:39.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:39.257+0000 7f3608600700 1 -- 172.21.15.43:0/3416514834 wait complete. 2024-05-27T07:15:40.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:40 smithi043 ceph-mon[31531]: pgmap v55: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:40.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:15:40.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:40 smithi132 ceph-mon[35283]: pgmap v55: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:40.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:40 smithi179 ceph-mon[35888]: pgmap v55: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:42.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:42 smithi043 ceph-mon[31531]: pgmap v56: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:42.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:42 smithi132 ceph-mon[35283]: pgmap v56: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:42.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:42 smithi179 ceph-mon[35888]: pgmap v56: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:43.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.546+0000 7f5289800700 1 -- 172.21.15.43:0/3360838859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5284105cb0 msgr2=0x7f528410e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.546+0000 7f5289800700 1 --2- 172.21.15.43:0/3360838859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5284105cb0 0x7f528410e200 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5270009990 tx=0x7f5270009ca0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.546+0000 7f5289800700 1 -- 172.21.15.43:0/3360838859 shutdown_connections 2024-05-27T07:15:43.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.546+0000 7f5289800700 1 --2- 172.21.15.43:0/3360838859 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f528410e740 0x7f5284110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.546+0000 7f5289800700 1 --2- 172.21.15.43:0/3360838859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5284105360 0x7f5284105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.546+0000 7f5289800700 1 --2- 172.21.15.43:0/3360838859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5284105cb0 0x7f528410e200 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.546+0000 7f5289800700 1 -- 172.21.15.43:0/3360838859 >> 172.21.15.43:0/3360838859 conn(0x7f5284078ab0 msgr2=0x7f5284078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:43.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.546+0000 7f5289800700 1 -- 172.21.15.43:0/3360838859 shutdown_connections 2024-05-27T07:15:43.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.547+0000 7f5289800700 1 -- 172.21.15.43:0/3360838859 wait complete. 2024-05-27T07:15:43.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.548+0000 7f5289800700 1 Processor -- start 2024-05-27T07:15:43.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.548+0000 7f5289800700 1 -- start start 2024-05-27T07:15:43.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.548+0000 7f5289800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5284105360 0x7f528419cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.548+0000 7f5289800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5284105cb0 0x7f528419d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.548+0000 7f5289800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f528410e740 0x7f52841a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.548+0000 7f5289800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5284114200 con 0x7f5284105cb0 2024-05-27T07:15:43.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.548+0000 7f5289800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5284114080 con 0x7f528410e740 2024-05-27T07:15:43.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.548+0000 7f5289800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5284114380 con 0x7f5284105360 2024-05-27T07:15:43.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.549+0000 7f5283e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5284105360 0x7f528419cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.549+0000 7f5288e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f528410e740 0x7f52841a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.549+0000 7f5283e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5284105360 0x7f528419cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:34292/0 (socket says 172.21.15.43:34292) 2024-05-27T07:15:43.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.549+0000 7f5283e00700 1 -- 172.21.15.43:0/3936378288 learned_addr learned my addr 172.21.15.43:0/3936378288 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:43.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.550+0000 7f5283400700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5284105cb0 0x7f528419d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.550+0000 7f5283e00700 1 -- 172.21.15.43:0/3936378288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f528410e740 msgr2=0x7f52841a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.550+0000 7f5283e00700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f528410e740 0x7f52841a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.550+0000 7f5283e00700 1 -- 172.21.15.43:0/3936378288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5284105cb0 msgr2=0x7f528419d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.550+0000 7f5283e00700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5284105cb0 0x7f528419d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.550+0000 7f5283e00700 1 -- 172.21.15.43:0/3936378288 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52841a1dc0 con 0x7f5284105360 2024-05-27T07:15:43.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.551+0000 7f5283e00700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5284105360 0x7f528419cef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f527400b530 tx=0x7f527400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:43.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.551+0000 7f5280c00700 1 -- 172.21.15.43:0/3936378288 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274015020 con 0x7f5284105360 2024-05-27T07:15:43.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.551+0000 7f5289800700 1 -- 172.21.15.43:0/3936378288 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52841a20b0 con 0x7f5284105360 2024-05-27T07:15:43.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.551+0000 7f5280c00700 1 -- 172.21.15.43:0/3936378288 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5274011cc0 con 0x7f5284105360 2024-05-27T07:15:43.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.551+0000 7f5289800700 1 -- 172.21.15.43:0/3936378288 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52841a99d0 con 0x7f5284105360 2024-05-27T07:15:43.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.552+0000 7f5280c00700 1 -- 172.21.15.43:0/3936378288 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274010c80 con 0x7f5284105360 2024-05-27T07:15:43.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.558+0000 7f5299959b80 1 Processor -- start 2024-05-27T07:15:43.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.558+0000 7f5299959b80 1 -- start start 2024-05-27T07:15:43.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.558+0000 7f5299959b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 0x55f04964d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.558+0000 7f5299959b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0494cde40 0x55f04964dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.558+0000 7f5299959b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0496572c0 0x55f0496596e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.558+0000 7f5299959b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f04957ae40 con 0x55f0496572c0 2024-05-27T07:15:43.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.558+0000 7f5299959b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f04956a9e0 con 0x55f0494cde40 2024-05-27T07:15:43.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.558+0000 7f5299959b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f04956ea20 con 0x55f0495b99f0 2024-05-27T07:15:43.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.559+0000 7f5283400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0496572c0 0x55f0496596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.559+0000 7f5283400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0496572c0 0x55f0496596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35964/0 (socket says 172.21.15.43:35964) 2024-05-27T07:15:43.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.559+0000 7f5283400700 1 -- 172.21.15.43:0/3695407135 learned_addr learned my addr 172.21.15.43:0/3695407135 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:43.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.559+0000 7f5280c00700 1 -- 172.21.15.43:0/3936378288 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5274018020 con 0x7f5284105360 2024-05-27T07:15:43.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.559+0000 7f5288e00700 1 --2- 172.21.15.43:0/3695407135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 0x55f04964d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.559+0000 7f5283400700 1 -- 172.21.15.43:0/3695407135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 msgr2=0x55f04964d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:43.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.559+0000 7f5283400700 1 --2- 172.21.15.43:0/3695407135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 0x55f04964d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.559+0000 7f5283400700 1 -- 172.21.15.43:0/3695407135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0494cde40 msgr2=0x55f04964dc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:15:43.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.559+0000 7f5283400700 1 --2- 172.21.15.43:0/3695407135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0494cde40 0x55f04964dc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.559+0000 7f5283400700 1 -- 172.21.15.43:0/3695407135 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f049533420 con 0x55f0496572c0 2024-05-27T07:15:43.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.560+0000 7f5288e00700 1 --2- 172.21.15.43:0/3695407135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 0x55f04964d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:15:43.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.560+0000 7f5283400700 1 --2- 172.21.15.43:0/3695407135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0496572c0 0x55f0496596e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5270000c00 tx=0x7f5270009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c4f726556aec45 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:43.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.561+0000 7f5279600700 1 -- 172.21.15.43:0/3695407135 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5270027070 con 0x55f0496572c0 2024-05-27T07:15:43.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.561+0000 7f5279600700 1 -- 172.21.15.43:0/3695407135 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5270005540 con 0x55f0496572c0 2024-05-27T07:15:43.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.561+0000 7f5279600700 1 -- 172.21.15.43:0/3695407135 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527000bd10 con 0x55f0496572c0 2024-05-27T07:15:43.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.561+0000 7f5280c00700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52680b2a70 0x7f52680b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.562+0000 7f5280c00700 1 -- 172.21.15.43:0/3936378288 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f52740e1fa0 con 0x7f5284105360 2024-05-27T07:15:43.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.562+0000 7f5288e00700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52680b2a70 0x7f52680b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.562+0000 7f5299959b80 1 -- 172.21.15.43:0/3695407135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0496572c0 msgr2=0x55f0496596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.562+0000 7f5299959b80 1 --2- 172.21.15.43:0/3695407135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0496572c0 0x55f0496596e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5270000c00 tx=0x7f5270009f90 comp rx=0 tx=0).stop 2024-05-27T07:15:43.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.562+0000 7f5299959b80 1 -- 172.21.15.43:0/3695407135 shutdown_connections 2024-05-27T07:15:43.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.562+0000 7f5299959b80 1 --2- 172.21.15.43:0/3695407135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0496572c0 0x55f0496596e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.562+0000 7f5299959b80 1 --2- 172.21.15.43:0/3695407135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 0x55f04964d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.562+0000 7f5299959b80 1 --2- 172.21.15.43:0/3695407135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0494cde40 0x55f04964dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.562+0000 7f5299959b80 1 -- 172.21.15.43:0/3695407135 >> 172.21.15.43:0/3695407135 conn(0x55f049567390 msgr2=0x55f049648710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:43.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.563+0000 7f5299959b80 1 -- 172.21.15.43:0/3695407135 shutdown_connections 2024-05-27T07:15:43.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.563+0000 7f5299959b80 1 -- 172.21.15.43:0/3695407135 wait complete. 2024-05-27T07:15:43.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.564+0000 7f5288e00700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52680b2a70 0x7f52680b4f80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x55f049649a90 tx=0x7f527c009650 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:43.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.564+0000 7f5299959b80 1 -- start start 2024-05-27T07:15:43.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.564+0000 7f5299959b80 1 Processor -- start 2024-05-27T07:15:43.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5299959b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0494cde40 0x55f049654340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5299959b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049654880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5299959b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0496572c0 0x55f049654dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5299959b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f049470fa0 con 0x55f0495b99f0 2024-05-27T07:15:43.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5299959b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f04950b760 con 0x55f0494cde40 2024-05-27T07:15:43.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5299959b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f049659da0 con 0x55f0496572c0 2024-05-27T07:15:43.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5299959b80 10 client.0 _ll_register_callbacks cb 0x55f049514c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:43.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5283400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049654880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5283400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049654880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35974/0 (socket says 172.21.15.43:35974) 2024-05-27T07:15:43.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5283400700 1 -- 172.21.15.43:0/703185853 learned_addr learned my addr 172.21.15.43:0/703185853 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:43.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5283e00700 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0494cde40 0x55f049654340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.565+0000 7f5288e00700 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0496572c0 0x55f049654dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.566+0000 7f5283400700 1 -- 172.21.15.43:0/703185853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0496572c0 msgr2=0x55f049654dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:43.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.566+0000 7f5283400700 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0496572c0 0x55f049654dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.566+0000 7f5283400700 1 -- 172.21.15.43:0/703185853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0494cde40 msgr2=0x55f049654340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:43.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.566+0000 7f5283400700 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0494cde40 0x55f049654340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.566+0000 7f5283400700 1 -- 172.21.15.43:0/703185853 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0494ba8d0 con 0x55f0495b99f0 2024-05-27T07:15:43.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.566+0000 7f5288e00700 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0496572c0 0x55f049654dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:43.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.566+0000 7f5283400700 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049654880 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f52700059d0 tx=0x7f5270003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55271c2e20bb0d3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:43.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.566+0000 7f526e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:43.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.567+0000 7f526e000700 1 -- 172.21.15.43:0/703185853 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5270031060 con 0x55f0495b99f0 2024-05-27T07:15:43.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.567+0000 7f526e000700 1 -- 172.21.15.43:0/703185853 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f527000b910 con 0x55f0495b99f0 2024-05-27T07:15:43.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.567+0000 7f5299959b80 1 -- 172.21.15.43:0/703185853 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5270005540 con 0x55f0495b99f0 2024-05-27T07:15:43.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.567+0000 7f526e000700 1 -- 172.21.15.43:0/703185853 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5270027040 con 0x55f0495b99f0 2024-05-27T07:15:43.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.567+0000 7f5299959b80 10 client.15726 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:43.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.567+0000 7f5299959b80 1 -- 172.21.15.43:0/703185853 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f04953b060 con 0x55f0495b99f0 2024-05-27T07:15:43.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.568+0000 7f526e000700 1 -- 172.21.15.43:0/703185853 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5270005540 con 0x55f0495b99f0 2024-05-27T07:15:43.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.569+0000 7f526e000700 1 -- 172.21.15.43:0/703185853 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5270022b20 con 0x55f0495b99f0 2024-05-27T07:15:43.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.569+0000 7f5299959b80 10 client.15726 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:43.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.569+0000 7f5299959b80 10 client.15726 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:15:43.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.569+0000 7f5299959b80 20 client.15726 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:43.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.569+0000 7f5299959b80 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f04966f380 0x55f0496717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.569+0000 7f5299959b80 4 client.15726 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:43.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.569+0000 7f5299959b80 1 -- 172.21.15.43:0/703185853 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f04957cd90 con 0x55f04966f380 2024-05-27T07:15:43.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.570+0000 7f5283e00700 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f04966f380 0x55f0496717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.571+0000 7f5283e00700 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f04966f380 0x55f0496717a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d29b36ee4dfd54f6 server_cookie=285161de4c9b90d3 in_seq=0 out_seq=0 2024-05-27T07:15:43.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.571+0000 7f526e000700 10 client.15726 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:15:43.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f526e000700 1 -- 172.21.15.43:0/703185853 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f04957cd90 con 0x55f04966f380 2024-05-27T07:15:43.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f526e000700 10 client.15726 handle_command_reply: tid=0 2024-05-27T07:15:43.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 client.15726 shutdown 2024-05-27T07:15:43.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 -- 172.21.15.43:0/703185853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 msgr2=0x55f049654880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049654880 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f52700059d0 tx=0x7f5270003aa0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 -- 172.21.15.43:0/703185853 shutdown_connections 2024-05-27T07:15:43.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f04966f380 0x55f0496717a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0496572c0 0x55f049654dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0494cde40 0x55f049654340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 --2- 172.21.15.43:0/703185853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049654880 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 -- 172.21.15.43:0/703185853 >> 172.21.15.43:0/703185853 conn(0x55f049567390 msgr2=0x55f04964d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:43.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f526e000700 0 client.15726 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:15:43.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 -- 172.21.15.43:0/703185853 shutdown_connections 2024-05-27T07:15:43.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.573+0000 7f5299959b80 1 -- 172.21.15.43:0/703185853 wait complete. 2024-05-27T07:15:43.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.574+0000 7f5299959b80 20 client.15726 trim_cache size 0 max 16384 2024-05-27T07:15:43.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.590+0000 7f5299959b80 1 Processor -- start 2024-05-27T07:15:43.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.590+0000 7f5299959b80 1 -- start start 2024-05-27T07:15:43.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.591+0000 7f5299959b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0494cde40 0x55f04964fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.591+0000 7f5299959b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 0x55f04966b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.591+0000 7f5299959b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0496572c0 0x55f049648740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.591+0000 7f5299959b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f52841ae1b0 con 0x55f0494cde40 2024-05-27T07:15:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.591+0000 7f5299959b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52841afcf0 con 0x55f0496572c0 2024-05-27T07:15:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.591+0000 7f5299959b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f52841ae880 con 0x55f0495b99f0 2024-05-27T07:15:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.591+0000 7f5283e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0494cde40 0x55f04964fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.591+0000 7f5283e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0494cde40 0x55f04964fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35986/0 (socket says 172.21.15.43:35986) 2024-05-27T07:15:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.591+0000 7f5283e00700 1 -- 172.21.15.43:0/2537402861 learned_addr learned my addr 172.21.15.43:0/2537402861 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.592+0000 7f5288e00700 1 --2- 172.21.15.43:0/2537402861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0496572c0 0x55f049648740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.592+0000 7f5283400700 1 --2- 172.21.15.43:0/2537402861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 0x55f04966b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.592+0000 7f5283e00700 1 -- 172.21.15.43:0/2537402861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 msgr2=0x55f04966b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.592+0000 7f5283e00700 1 --2- 172.21.15.43:0/2537402861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 0x55f04966b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.592+0000 7f5283e00700 1 -- 172.21.15.43:0/2537402861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0496572c0 msgr2=0x55f049648740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.592+0000 7f5283e00700 1 --2- 172.21.15.43:0/2537402861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0496572c0 0x55f049648740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.592+0000 7f5283e00700 1 -- 172.21.15.43:0/2537402861 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f049655130 con 0x55f0494cde40 2024-05-27T07:15:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.592+0000 7f5283e00700 1 --2- 172.21.15.43:0/2537402861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0494cde40 0x55f04964fba0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f52740a8ce0 tx=0x7f5274004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe08480fab4a23a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.593+0000 7f5278c00700 1 -- 172.21.15.43:0/2537402861 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52740ad390 con 0x55f0494cde40 2024-05-27T07:15:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.593+0000 7f5278c00700 1 -- 172.21.15.43:0/2537402861 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52740bd070 con 0x55f0494cde40 2024-05-27T07:15:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.593+0000 7f5278c00700 1 -- 172.21.15.43:0/2537402861 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52740a9240 con 0x55f0494cde40 2024-05-27T07:15:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.594+0000 7f5299959b80 1 -- 172.21.15.43:0/2537402861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0494cde40 msgr2=0x55f04964fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.594+0000 7f5299959b80 1 --2- 172.21.15.43:0/2537402861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0494cde40 0x55f04964fba0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f52740a8ce0 tx=0x7f5274004750 comp rx=0 tx=0).stop 2024-05-27T07:15:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.594+0000 7f5299959b80 1 -- 172.21.15.43:0/2537402861 shutdown_connections 2024-05-27T07:15:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.594+0000 7f5299959b80 1 --2- 172.21.15.43:0/2537402861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0496572c0 0x55f049648740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.594+0000 7f5299959b80 1 --2- 172.21.15.43:0/2537402861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0494cde40 0x55f04964fba0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.594+0000 7f5299959b80 1 --2- 172.21.15.43:0/2537402861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0495b99f0 0x55f04966b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.594+0000 7f5299959b80 1 -- 172.21.15.43:0/2537402861 >> 172.21.15.43:0/2537402861 conn(0x55f049567390 msgr2=0x55f04964c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.594+0000 7f5299959b80 1 -- 172.21.15.43:0/2537402861 shutdown_connections 2024-05-27T07:15:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.595+0000 7f5299959b80 1 -- 172.21.15.43:0/2537402861 wait complete. 2024-05-27T07:15:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.595+0000 7f5299959b80 1 -- start start 2024-05-27T07:15:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.595+0000 7f5299959b80 1 Processor -- start 2024-05-27T07:15:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.596+0000 7f5299959b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0494cde40 0x55f049671620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.596+0000 7f5299959b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049671b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.596+0000 7f5299959b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0496572c0 0x55f0496720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.596+0000 7f5299959b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f52841af620 con 0x55f0495b99f0 2024-05-27T07:15:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.596+0000 7f5299959b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52841aef50 con 0x55f0496572c0 2024-05-27T07:15:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.596+0000 7f5299959b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f52841ac170 con 0x55f0494cde40 2024-05-27T07:15:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.596+0000 7f5299959b80 10 client.0 _ll_register_callbacks cb 0x55f049514c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.597+0000 7f5283400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049671b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.597+0000 7f5283e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0494cde40 0x55f049671620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.597+0000 7f5283400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049671b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35998/0 (socket says 172.21.15.43:35998) 2024-05-27T07:15:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.597+0000 7f5283400700 1 -- 172.21.15.43:0/2271367999 learned_addr learned my addr 172.21.15.43:0/2271367999 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.597+0000 7f5288e00700 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0496572c0 0x55f0496720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.598+0000 7f5283e00700 1 -- 172.21.15.43:0/2271367999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0496572c0 msgr2=0x55f0496720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.598+0000 7f5283e00700 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0496572c0 0x55f0496720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.598+0000 7f5283e00700 1 -- 172.21.15.43:0/2271367999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 msgr2=0x55f049671b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.598+0000 7f5283e00700 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049671b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.598+0000 7f5283e00700 1 -- 172.21.15.43:0/2271367999 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0496d7c60 con 0x55f0494cde40 2024-05-27T07:15:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.598+0000 7f5283e00700 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0494cde40 0x55f049671620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f52740b2000 tx=0x7f52740be270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0c0c2cc9b69a6b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.598+0000 7f526e000700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:15:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.599+0000 7f526e000700 1 -- 172.21.15.43:0/2271367999 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52740df7d0 con 0x55f0494cde40 2024-05-27T07:15:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.599+0000 7f5299959b80 1 -- 172.21.15.43:0/2271367999 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52740bd070 con 0x55f0494cde40 2024-05-27T07:15:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.599+0000 7f526e000700 1 -- 172.21.15.43:0/2271367999 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f52740a9240 con 0x55f0494cde40 2024-05-27T07:15:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.600+0000 7f526e000700 1 -- 172.21.15.43:0/2271367999 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f52740b1e40 con 0x55f0494cde40 2024-05-27T07:15:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.600+0000 7f5299959b80 10 client.25423 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.600+0000 7f5299959b80 1 -- 172.21.15.43:0/2271367999 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f0496d8ec0 con 0x55f0494cde40 2024-05-27T07:15:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.600+0000 7f526e000700 1 -- 172.21.15.43:0/2271367999 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52740bd070 con 0x55f0494cde40 2024-05-27T07:15:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.601+0000 7f526e000700 1 -- 172.21.15.43:0/2271367999 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f52740a8240 con 0x55f0494cde40 2024-05-27T07:15:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.601+0000 7f5299959b80 10 client.25423 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.601+0000 7f5299959b80 10 client.25423 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:15:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.601+0000 7f5299959b80 20 client.25423 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.601+0000 7f5299959b80 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f0496d94d0 0x55f0496db8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.601+0000 7f5299959b80 4 client.25423 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:15:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.601+0000 7f5299959b80 1 -- 172.21.15.43:0/2271367999 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f049576570 con 0x55f0496d94d0 2024-05-27T07:15:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.602+0000 7f5283400700 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f0496d94d0 0x55f0496db8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.603+0000 7f5283400700 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f0496d94d0 0x55f0496db8f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1054030b35d7d494 server_cookie=cf9dae33b3bc1ad6 in_seq=0 out_seq=0 2024-05-27T07:15:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.603+0000 7f526e000700 10 client.25423 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:15:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.603+0000 7f526e000700 1 -- 172.21.15.43:0/2271367999 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f049576570 con 0x55f0496d94d0 2024-05-27T07:15:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.603+0000 7f526e000700 10 client.25423 handle_command_reply: tid=0 2024-05-27T07:15:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.603+0000 7f5299959b80 1 client.25423 shutdown 2024-05-27T07:15:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.604+0000 7f5299959b80 1 -- 172.21.15.43:0/2271367999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0494cde40 msgr2=0x55f049671620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.604+0000 7f5299959b80 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0494cde40 0x55f049671620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f52740b2000 tx=0x7f52740be270 comp rx=0 tx=0).stop 2024-05-27T07:15:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.604+0000 7f5299959b80 1 -- 172.21.15.43:0/2271367999 shutdown_connections 2024-05-27T07:15:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.604+0000 7f5299959b80 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f0496d94d0 0x55f0496db8f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.604+0000 7f5299959b80 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0496572c0 0x55f0496720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.604+0000 7f5299959b80 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0494cde40 0x55f049671620 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.604+0000 7f526e000700 0 client.25423 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:15:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.604+0000 7f5299959b80 1 --2- 172.21.15.43:0/2271367999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0495b99f0 0x55f049671b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.604+0000 7f5299959b80 1 -- 172.21.15.43:0/2271367999 >> 172.21.15.43:0/2271367999 conn(0x55f049567390 msgr2=0x55f04964c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.605+0000 7f5299959b80 1 -- 172.21.15.43:0/2271367999 shutdown_connections 2024-05-27T07:15:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.605+0000 7f5299959b80 1 -- 172.21.15.43:0/2271367999 wait complete. 2024-05-27T07:15:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.605+0000 7f5299959b80 20 client.25423 trim_cache size 0 max 16384 2024-05-27T07:15:43.596 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:15:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.606+0000 7f5289800700 1 -- 172.21.15.43:0/3936378288 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52680b2a70 msgr2=0x7f52680b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.606+0000 7f5289800700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52680b2a70 0x7f52680b4f80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x55f049649a90 tx=0x7f527c009650 comp rx=0 tx=0).stop 2024-05-27T07:15:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.606+0000 7f5289800700 1 -- 172.21.15.43:0/3936378288 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5284105360 msgr2=0x7f528419cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.606+0000 7f5289800700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5284105360 0x7f528419cef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f527400b530 tx=0x7f527400b840 comp rx=0 tx=0).stop 2024-05-27T07:15:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.607+0000 7f5289800700 1 -- 172.21.15.43:0/3936378288 shutdown_connections 2024-05-27T07:15:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.607+0000 7f5289800700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52680b2a70 0x7f52680b4f80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.607+0000 7f5289800700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f528410e740 0x7f52841a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.607+0000 7f5289800700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5284105360 0x7f528419cef0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.607+0000 7f5289800700 1 --2- 172.21.15.43:0/3936378288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5284105cb0 0x7f528419d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.607+0000 7f5289800700 1 -- 172.21.15.43:0/3936378288 >> 172.21.15.43:0/3936378288 conn(0x7f5284078ab0 msgr2=0x7f5284103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.607+0000 7f5289800700 1 -- 172.21.15.43:0/3936378288 shutdown_connections 2024-05-27T07:15:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:43.607+0000 7f5289800700 1 -- 172.21.15.43:0/3936378288 wait complete. 2024-05-27T07:15:44.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:44 smithi132 ceph-mon[35283]: pgmap v57: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:44.438 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:44 smithi043 ceph-mon[31531]: pgmap v57: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:44.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:44 smithi179 ceph-mon[35888]: pgmap v57: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:44.499 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:15:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:46.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:46 smithi132 ceph-mon[35283]: pgmap v58: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:46.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:46 smithi043 ceph-mon[31531]: pgmap v58: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:46.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:46 smithi179 ceph-mon[35888]: pgmap v58: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:47.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.559+0000 7ff45fe00700 1 -- 172.21.15.43:0/3754725106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4601043f0 msgr2=0x7ff460104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.559+0000 7ff45fe00700 1 --2- 172.21.15.43:0/3754725106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4601043f0 0x7ff460104800 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff4500099c0 tx=0x7ff450009cd0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.559+0000 7ff45fe00700 1 -- 172.21.15.43:0/3754725106 shutdown_connections 2024-05-27T07:15:47.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.559+0000 7ff45fe00700 1 --2- 172.21.15.43:0/3754725106 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff460109670 0x7ff460110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.559+0000 7ff45fe00700 1 --2- 172.21.15.43:0/3754725106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff4601043f0 0x7ff460104800 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.559+0000 7ff45fe00700 1 --2- 172.21.15.43:0/3754725106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff460108d20 0x7ff460109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.559+0000 7ff45fe00700 1 -- 172.21.15.43:0/3754725106 >> 172.21.15.43:0/3754725106 conn(0x7ff460079d00 msgr2=0x7ff46007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:47.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.561+0000 7ff45fe00700 1 -- 172.21.15.43:0/3754725106 shutdown_connections 2024-05-27T07:15:47.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.561+0000 7ff45fe00700 1 -- 172.21.15.43:0/3754725106 wait complete. 2024-05-27T07:15:47.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.563+0000 7ff45fe00700 1 Processor -- start 2024-05-27T07:15:47.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.563+0000 7ff45fe00700 1 -- start start 2024-05-27T07:15:47.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.563+0000 7ff45fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4601043f0 0x7ff460075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.563+0000 7ff45fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff460108d20 0x7ff460075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.564+0000 7ff45fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff460109670 0x7ff460076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.564+0000 7ff45fe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff4601141a0 con 0x7ff460109670 2024-05-27T07:15:47.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.564+0000 7ff45fe00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff460114020 con 0x7ff4601043f0 2024-05-27T07:15:47.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.564+0000 7ff45fe00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff460114320 con 0x7ff460108d20 2024-05-27T07:15:47.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.564+0000 7ff45ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4601043f0 0x7ff460075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.564+0000 7ff45ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4601043f0 0x7ff460075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:43960/0 (socket says 172.21.15.43:43960) 2024-05-27T07:15:47.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.564+0000 7ff45ea00700 1 -- 172.21.15.43:0/2970453353 learned_addr learned my addr 172.21.15.43:0/2970453353 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:47.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.564+0000 7ff45e000700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff460108d20 0x7ff460075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.564+0000 7ff45f400700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff460109670 0x7ff460076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.565+0000 7ff45ea00700 1 -- 172.21.15.43:0/2970453353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff460108d20 msgr2=0x7ff460075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.565+0000 7ff45ea00700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff460108d20 0x7ff460075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.565+0000 7ff45ea00700 1 -- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff460109670 msgr2=0x7ff460076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.565+0000 7ff45ea00700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff460109670 0x7ff460076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.565+0000 7ff45ea00700 1 -- 172.21.15.43:0/2970453353 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4601a8f90 con 0x7ff4601043f0 2024-05-27T07:15:47.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.565+0000 7ff45f400700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff460109670 0x7ff460076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:47.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.565+0000 7ff45ea00700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4601043f0 0x7ff460075ab0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff450009f50 tx=0x7ff45000f460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:47.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.566+0000 7ff457400700 1 -- 172.21.15.43:0/2970453353 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff450027070 con 0x7ff4601043f0 2024-05-27T07:15:47.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.566+0000 7ff45fe00700 1 -- 172.21.15.43:0/2970453353 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4601a9220 con 0x7ff4601043f0 2024-05-27T07:15:47.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.566+0000 7ff45fe00700 1 -- 172.21.15.43:0/2970453353 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4601a96d0 con 0x7ff4601043f0 2024-05-27T07:15:47.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.567+0000 7ff457400700 1 -- 172.21.15.43:0/2970453353 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff45000fd20 con 0x7ff4601043f0 2024-05-27T07:15:47.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.567+0000 7ff457400700 1 -- 172.21.15.43:0/2970453353 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4500173f0 con 0x7ff4601043f0 2024-05-27T07:15:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.572+0000 7ff474299b80 1 Processor -- start 2024-05-27T07:15:47.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.573+0000 7ff474299b80 1 -- start start 2024-05-27T07:15:47.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.573+0000 7ff474299b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 0x555f26993690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.573+0000 7ff474299b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f26993bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.573+0000 7ff474299b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 0x555f269a7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.573+0000 7ff474299b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555f268c0e40 con 0x555f26813e40 2024-05-27T07:15:47.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.573+0000 7ff474299b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555f268b09e0 con 0x555f268ff9f0 2024-05-27T07:15:47.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.573+0000 7ff474299b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555f268b4a20 con 0x555f269a5ad0 2024-05-27T07:15:47.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.574+0000 7ff45f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 0x555f26993690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.574+0000 7ff45f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 0x555f26993690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:43966/0 (socket says 172.21.15.43:43966) 2024-05-27T07:15:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.574+0000 7ff45f400700 1 -- 172.21.15.43:0/3336087221 learned_addr learned my addr 172.21.15.43:0/3336087221 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.574+0000 7ff45e000700 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 0x555f269a7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.574+0000 7ff457400700 1 -- 172.21.15.43:0/2970453353 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff45002a070 con 0x7ff4601043f0 2024-05-27T07:15:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.575+0000 7ff45ea00700 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f26993bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.575+0000 7ff45f400700 1 -- 172.21.15.43:0/3336087221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 msgr2=0x555f269a7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.575+0000 7ff45f400700 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 0x555f269a7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.575+0000 7ff45f400700 1 -- 172.21.15.43:0/3336087221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 msgr2=0x555f26993bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.575+0000 7ff45f400700 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f26993bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.575+0000 7ff45f400700 1 -- 172.21.15.43:0/3336087221 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f26879420 con 0x555f268ff9f0 2024-05-27T07:15:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.575+0000 7ff45ea00700 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f26993bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:15:47.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.575+0000 7ff45e000700 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 0x555f269a7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:47.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.575+0000 7ff45f400700 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 0x555f26993690 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff458009dd0 tx=0x7ff4580077e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4ca2e7c583e3e30 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:47.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.576+0000 7ff44be00700 1 -- 172.21.15.43:0/3336087221 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458014610 con 0x555f268ff9f0 2024-05-27T07:15:47.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.576+0000 7ff44be00700 1 -- 172.21.15.43:0/3336087221 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff458019070 con 0x555f268ff9f0 2024-05-27T07:15:47.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.576+0000 7ff44be00700 1 -- 172.21.15.43:0/3336087221 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458013910 con 0x555f268ff9f0 2024-05-27T07:15:47.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.577+0000 7ff474299b80 1 -- 172.21.15.43:0/3336087221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 msgr2=0x555f26993690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.577+0000 7ff474299b80 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 0x555f26993690 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff458009dd0 tx=0x7ff4580077e0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.578+0000 7ff474299b80 1 -- 172.21.15.43:0/3336087221 shutdown_connections 2024-05-27T07:15:47.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.578+0000 7ff474299b80 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 0x555f269a7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.578+0000 7ff474299b80 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 0x555f26993690 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.578+0000 7ff474299b80 1 --2- 172.21.15.43:0/3336087221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f26993bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.578+0000 7ff474299b80 1 -- 172.21.15.43:0/3336087221 >> 172.21.15.43:0/3336087221 conn(0x555f268ad390 msgr2=0x555f2698e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:47.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.578+0000 7ff474299b80 1 -- 172.21.15.43:0/3336087221 shutdown_connections 2024-05-27T07:15:47.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.578+0000 7ff457400700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff4440b2a00 0x7ff4440b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.578+0000 7ff474299b80 1 -- 172.21.15.43:0/3336087221 wait complete. 2024-05-27T07:15:47.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.578+0000 7ff457400700 1 -- 172.21.15.43:0/2970453353 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff4500e3690 con 0x7ff4601043f0 2024-05-27T07:15:47.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.579+0000 7ff474299b80 1 -- start start 2024-05-27T07:15:47.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.579+0000 7ff45e000700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff4440b2a00 0x7ff4440b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.579+0000 7ff474299b80 1 Processor -- start 2024-05-27T07:15:47.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.580+0000 7ff474299b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f26813e40 0x555f269a2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.580+0000 7ff474299b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 0x555f269a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.580+0000 7ff474299b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f269a5ad0 0x555f269a3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.580+0000 7ff474299b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555f267b6fa0 con 0x555f269a5ad0 2024-05-27T07:15:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.580+0000 7ff474299b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555f26851760 con 0x555f268ff9f0 2024-05-27T07:15:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.580+0000 7ff474299b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555f269a8580 con 0x555f26813e40 2024-05-27T07:15:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.580+0000 7ff474299b80 10 client.0 _ll_register_callbacks cb 0x555f2685ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.581+0000 7ff45ea00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f26813e40 0x555f269a2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.581+0000 7ff45ea00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f26813e40 0x555f269a2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:35068/0 (socket says 172.21.15.43:35068) 2024-05-27T07:15:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.581+0000 7ff45ea00700 1 -- 172.21.15.43:0/869200157 learned_addr learned my addr 172.21.15.43:0/869200157 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.581+0000 7ff45f400700 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f269a5ad0 0x555f269a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.581+0000 7ff45e000700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff4440b2a00 0x7ff4440b4f10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff46007a3a0 tx=0x7ff44c00a300 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.581+0000 7ff45e000700 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 0x555f269a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.582+0000 7ff45ea00700 1 -- 172.21.15.43:0/869200157 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 msgr2=0x555f269a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.582+0000 7ff45ea00700 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 0x555f269a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.582+0000 7ff45ea00700 1 -- 172.21.15.43:0/869200157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f269a5ad0 msgr2=0x555f269a3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:47.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.582+0000 7ff45ea00700 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f269a5ad0 0x555f269a3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.582+0000 7ff45ea00700 1 -- 172.21.15.43:0/869200157 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f268008d0 con 0x555f26813e40 2024-05-27T07:15:47.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.582+0000 7ff45f400700 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f269a5ad0 0x555f269a3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:47.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.582+0000 7ff45ea00700 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f26813e40 0x555f269a2990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff450009990 tx=0x7ff4500aab90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5cbdfbf4072b20 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.582+0000 7ff448c00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:15:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.583+0000 7ff448c00700 1 -- 172.21.15.43:0/869200157 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4500a9fa0 con 0x555f26813e40 2024-05-27T07:15:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.583+0000 7ff474299b80 1 -- 172.21.15.43:0/869200157 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff458019070 con 0x555f26813e40 2024-05-27T07:15:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.583+0000 7ff448c00700 1 -- 172.21.15.43:0/869200157 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff4500afb90 con 0x555f26813e40 2024-05-27T07:15:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.583+0000 7ff448c00700 1 -- 172.21.15.43:0/869200157 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff4500a9dc0 con 0x555f26813e40 2024-05-27T07:15:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.583+0000 7ff474299b80 10 client.25435 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.583+0000 7ff474299b80 1 -- 172.21.15.43:0/869200157 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555f26881060 con 0x555f26813e40 2024-05-27T07:15:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.584+0000 7ff448c00700 1 -- 172.21.15.43:0/869200157 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff4500e2030 con 0x555f26813e40 2024-05-27T07:15:47.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.584+0000 7ff448c00700 1 -- 172.21.15.43:0/869200157 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff4500e3280 con 0x555f26813e40 2024-05-27T07:15:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.585+0000 7ff474299b80 10 client.25435 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.585+0000 7ff474299b80 10 client.25435 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:15:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.585+0000 7ff474299b80 20 client.25435 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.585+0000 7ff474299b80 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x555f269b55d0 0x555f269b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.585+0000 7ff474299b80 4 client.25435 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.585+0000 7ff474299b80 1 -- 172.21.15.43:0/869200157 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f268c2d90 con 0x555f269b55d0 2024-05-27T07:15:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.586+0000 7ff45e000700 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x555f269b55d0 0x555f269b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.587+0000 7ff45e000700 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x555f269b55d0 0x555f269b7a80 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=18289a14429561dc server_cookie=30defb613b180277 in_seq=0 out_seq=0 2024-05-27T07:15:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.587+0000 7ff448c00700 10 client.25435 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:15:47.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.589+0000 7ff448c00700 1 -- 172.21.15.43:0/869200157 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555f268c2d90 con 0x555f269b55d0 2024-05-27T07:15:47.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.589+0000 7ff448c00700 10 client.25435 handle_command_reply: tid=0 2024-05-27T07:15:47.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.589+0000 7ff474299b80 1 client.25435 shutdown 2024-05-27T07:15:47.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.589+0000 7ff474299b80 1 -- 172.21.15.43:0/869200157 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f26813e40 msgr2=0x555f269a2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.589+0000 7ff474299b80 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f26813e40 0x555f269a2990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff450009990 tx=0x7ff4500aab90 comp rx=0 tx=0).stop 2024-05-27T07:15:47.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.589+0000 7ff474299b80 1 -- 172.21.15.43:0/869200157 shutdown_connections 2024-05-27T07:15:47.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.589+0000 7ff474299b80 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x555f269b55d0 0x555f269b7a80 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.590+0000 7ff448c00700 0 client.25435 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:15:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.590+0000 7ff474299b80 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f269a5ad0 0x555f269a3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.590+0000 7ff474299b80 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f26813e40 0x555f269a2990 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.590+0000 7ff474299b80 1 --2- 172.21.15.43:0/869200157 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f268ff9f0 0x555f269a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.590+0000 7ff474299b80 1 -- 172.21.15.43:0/869200157 >> 172.21.15.43:0/869200157 conn(0x555f268ad390 msgr2=0x555f26993670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.590+0000 7ff474299b80 1 -- 172.21.15.43:0/869200157 shutdown_connections 2024-05-27T07:15:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.590+0000 7ff474299b80 1 -- 172.21.15.43:0/869200157 wait complete. 2024-05-27T07:15:47.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.591+0000 7ff474299b80 20 client.25435 trim_cache size 0 max 16384 2024-05-27T07:15:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.622+0000 7ff474299b80 1 Processor -- start 2024-05-27T07:15:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.622+0000 7ff474299b80 1 -- start start 2024-05-27T07:15:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.623+0000 7ff474299b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f26813e40 0x555f2699e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.623+0000 7ff474299b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f268ff9f0 0x555f2698e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.623+0000 7ff474299b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 0x555f269b1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.623+0000 7ff474299b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff44000ec20 con 0x555f268ff9f0 2024-05-27T07:15:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.623+0000 7ff474299b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff440010760 con 0x555f26813e40 2024-05-27T07:15:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.623+0000 7ff474299b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff44000f2f0 con 0x555f269a5ad0 2024-05-27T07:15:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.623+0000 7ff45e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f268ff9f0 0x555f2698e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.623+0000 7ff45e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f268ff9f0 0x555f2698e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32920/0 (socket says 172.21.15.43:32920) 2024-05-27T07:15:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.623+0000 7ff45e000700 1 -- 172.21.15.43:0/4136811885 learned_addr learned my addr 172.21.15.43:0/4136811885 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.624+0000 7ff45f400700 1 --2- 172.21.15.43:0/4136811885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 0x555f269b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.624+0000 7ff45ea00700 1 --2- 172.21.15.43:0/4136811885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f26813e40 0x555f2699e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.624+0000 7ff45e000700 1 -- 172.21.15.43:0/4136811885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 msgr2=0x555f269b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:47.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.624+0000 7ff45e000700 1 --2- 172.21.15.43:0/4136811885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 0x555f269b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.624+0000 7ff45e000700 1 -- 172.21.15.43:0/4136811885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f26813e40 msgr2=0x555f2699e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:47.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.624+0000 7ff45e000700 1 --2- 172.21.15.43:0/4136811885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f26813e40 0x555f2699e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.624+0000 7ff45e000700 1 -- 172.21.15.43:0/4136811885 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f269b1bc0 con 0x555f268ff9f0 2024-05-27T07:15:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.625+0000 7ff45e000700 1 --2- 172.21.15.43:0/4136811885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f268ff9f0 0x555f2698e740 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff46006c3b0 tx=0x7ff44c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b66f47cfb179cd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.625+0000 7ff44b400700 1 -- 172.21.15.43:0/4136811885 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c01bb00 con 0x555f268ff9f0 2024-05-27T07:15:47.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.625+0000 7ff44b400700 1 -- 172.21.15.43:0/4136811885 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff44c02d070 con 0x555f268ff9f0 2024-05-27T07:15:47.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.625+0000 7ff44b400700 1 -- 172.21.15.43:0/4136811885 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c00f680 con 0x555f268ff9f0 2024-05-27T07:15:47.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.627+0000 7ff474299b80 1 -- 172.21.15.43:0/4136811885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f268ff9f0 msgr2=0x555f2698e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.627+0000 7ff474299b80 1 --2- 172.21.15.43:0/4136811885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f268ff9f0 0x555f2698e740 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff46006c3b0 tx=0x7ff44c019000 comp rx=0 tx=0).stop 2024-05-27T07:15:47.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.628+0000 7ff474299b80 1 -- 172.21.15.43:0/4136811885 shutdown_connections 2024-05-27T07:15:47.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.628+0000 7ff474299b80 1 --2- 172.21.15.43:0/4136811885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f269a5ad0 0x555f269b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.628+0000 7ff474299b80 1 --2- 172.21.15.43:0/4136811885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f26813e40 0x555f2699e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.628+0000 7ff474299b80 1 --2- 172.21.15.43:0/4136811885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f268ff9f0 0x555f2698e740 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.628+0000 7ff474299b80 1 -- 172.21.15.43:0/4136811885 >> 172.21.15.43:0/4136811885 conn(0x555f268ad390 msgr2=0x555f2699da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:47.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.628+0000 7ff474299b80 1 -- 172.21.15.43:0/4136811885 shutdown_connections 2024-05-27T07:15:47.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.628+0000 7ff474299b80 1 -- 172.21.15.43:0/4136811885 wait complete. 2024-05-27T07:15:47.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.629+0000 7ff474299b80 1 -- start start 2024-05-27T07:15:47.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.629+0000 7ff474299b80 1 Processor -- start 2024-05-27T07:15:47.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.630+0000 7ff474299b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f269a8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.630+0000 7ff474299b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f268ff9f0 0x555f269a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.630+0000 7ff474299b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f269a5ad0 0x555f269a92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.630+0000 7ff474299b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff440010090 con 0x555f26813e40 2024-05-27T07:15:47.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.630+0000 7ff474299b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff44000f9c0 con 0x555f269a5ad0 2024-05-27T07:15:47.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.630+0000 7ff474299b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff44000cba0 con 0x555f268ff9f0 2024-05-27T07:15:47.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.630+0000 7ff474299b80 10 client.0 _ll_register_callbacks cb 0x555f2685ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:47.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.630+0000 7ff45ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f269a8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.631+0000 7ff45ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f269a8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32930/0 (socket says 172.21.15.43:32930) 2024-05-27T07:15:47.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.631+0000 7ff45ea00700 1 -- 172.21.15.43:0/4110303468 learned_addr learned my addr 172.21.15.43:0/4110303468 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:47.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.631+0000 7ff45e000700 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f268ff9f0 0x555f269a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.631+0000 7ff45f400700 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f269a5ad0 0x555f269a92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.632+0000 7ff45f400700 1 -- 172.21.15.43:0/4110303468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f268ff9f0 msgr2=0x555f269a8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:47.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.632+0000 7ff45f400700 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f268ff9f0 0x555f269a8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.632+0000 7ff45f400700 1 -- 172.21.15.43:0/4110303468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 msgr2=0x555f269a8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:47.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.632+0000 7ff45f400700 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f269a8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.632+0000 7ff45f400700 1 -- 172.21.15.43:0/4110303468 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f26a1e360 con 0x555f269a5ad0 2024-05-27T07:15:47.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.632+0000 7ff45ea00700 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f269a8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:47.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.632+0000 7ff45e000700 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f268ff9f0 0x555f269a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:15:47.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.632+0000 7ff45f400700 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f269a5ad0 0x555f269a92f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff458009500 tx=0x7ff458003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8baed696d885743 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:47.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.632+0000 7ff448c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:15:47.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.633+0000 7ff448c00700 1 -- 172.21.15.43:0/4110303468 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45802c060 con 0x555f269a5ad0 2024-05-27T07:15:47.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.633+0000 7ff448c00700 1 -- 172.21.15.43:0/4110303468 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff4580138a0 con 0x555f269a5ad0 2024-05-27T07:15:47.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.633+0000 7ff474299b80 1 -- 172.21.15.43:0/4110303468 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff44c02d070 con 0x555f269a5ad0 2024-05-27T07:15:47.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.634+0000 7ff448c00700 1 -- 172.21.15.43:0/4110303468 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff458033050 con 0x555f269a5ad0 2024-05-27T07:15:47.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.634+0000 7ff474299b80 10 client.35414 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:47.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.634+0000 7ff474299b80 1 -- 172.21.15.43:0/4110303468 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555f26a1f5c0 con 0x555f269a5ad0 2024-05-27T07:15:47.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.634+0000 7ff448c00700 1 -- 172.21.15.43:0/4110303468 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff458003fd0 con 0x555f269a5ad0 2024-05-27T07:15:47.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.635+0000 7ff448c00700 1 -- 172.21.15.43:0/4110303468 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff458005040 con 0x555f269a5ad0 2024-05-27T07:15:47.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.635+0000 7ff474299b80 10 client.35414 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:47.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.635+0000 7ff474299b80 10 client.35414 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:15:47.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.635+0000 7ff474299b80 20 client.35414 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:47.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.635+0000 7ff474299b80 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x555f269b55d0 0x555f26a21970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:47.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.635+0000 7ff474299b80 4 client.35414 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:15:47.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.635+0000 7ff474299b80 1 -- 172.21.15.43:0/4110303468 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff44c02d070 con 0x555f269b55d0 2024-05-27T07:15:47.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.636+0000 7ff45ea00700 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x555f269b55d0 0x555f26a21970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:47.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.637+0000 7ff45ea00700 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x555f269b55d0 0x555f26a21970 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6dde941c2822dd09 server_cookie=55e44b632692d251 in_seq=0 out_seq=0 2024-05-27T07:15:47.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.637+0000 7ff448c00700 10 client.35414 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:15:47.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff448c00700 1 -- 172.21.15.43:0/4110303468 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff44c02d070 con 0x555f269b55d0 2024-05-27T07:15:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff448c00700 10 client.35414 handle_command_reply: tid=0 2024-05-27T07:15:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff474299b80 1 client.35414 shutdown 2024-05-27T07:15:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff474299b80 1 -- 172.21.15.43:0/4110303468 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f269a5ad0 msgr2=0x555f269a92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff474299b80 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f269a5ad0 0x555f269a92f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff458009500 tx=0x7ff458003c10 comp rx=0 tx=0).stop 2024-05-27T07:15:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff474299b80 1 -- 172.21.15.43:0/4110303468 shutdown_connections 2024-05-27T07:15:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff474299b80 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x555f269b55d0 0x555f26a21970 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff474299b80 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555f269a5ad0 0x555f269a92f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff474299b80 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555f26813e40 0x555f269a8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff474299b80 1 --2- 172.21.15.43:0/4110303468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555f268ff9f0 0x555f269a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff474299b80 1 -- 172.21.15.43:0/4110303468 >> 172.21.15.43:0/4110303468 conn(0x555f268ad390 msgr2=0x555f26992120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:47.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.638+0000 7ff448c00700 0 client.35414 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:15:47.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.639+0000 7ff474299b80 1 -- 172.21.15.43:0/4110303468 shutdown_connections 2024-05-27T07:15:47.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.639+0000 7ff474299b80 1 -- 172.21.15.43:0/4110303468 wait complete. 2024-05-27T07:15:47.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.639+0000 7ff474299b80 20 client.35414 trim_cache size 0 max 16384 2024-05-27T07:15:47.632 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:15:47.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.640+0000 7ff45fe00700 1 -- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff4440b2a00 msgr2=0x7ff4440b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.640+0000 7ff45fe00700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff4440b2a00 0x7ff4440b4f10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff46007a3a0 tx=0x7ff44c00a300 comp rx=0 tx=0).stop 2024-05-27T07:15:47.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.640+0000 7ff45fe00700 1 -- 172.21.15.43:0/2970453353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4601043f0 msgr2=0x7ff460075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.640+0000 7ff45fe00700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4601043f0 0x7ff460075ab0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff450009f50 tx=0x7ff45000f460 comp rx=0 tx=0).stop 2024-05-27T07:15:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.640+0000 7ff45fe00700 1 -- 172.21.15.43:0/2970453353 shutdown_connections 2024-05-27T07:15:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.640+0000 7ff45fe00700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff4440b2a00 0x7ff4440b4f10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.640+0000 7ff45fe00700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff460109670 0x7ff460076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.640+0000 7ff45fe00700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4601043f0 0x7ff460075ab0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.640+0000 7ff45fe00700 1 --2- 172.21.15.43:0/2970453353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff460108d20 0x7ff460075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.641+0000 7ff45fe00700 1 -- 172.21.15.43:0/2970453353 >> 172.21.15.43:0/2970453353 conn(0x7ff460079d00 msgr2=0x7ff460101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.641+0000 7ff45fe00700 1 -- 172.21.15.43:0/2970453353 shutdown_connections 2024-05-27T07:15:47.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:47.641+0000 7ff45fe00700 1 -- 172.21.15.43:0/2970453353 wait complete. 2024-05-27T07:15:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:48 smithi132 ceph-mon[35283]: pgmap v59: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:48.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:48 smithi043 ceph-mon[31531]: pgmap v59: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:48.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:48 smithi179 ceph-mon[35888]: pgmap v59: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:48.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:15:49.019 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:50.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:50 smithi043 ceph-mon[31531]: pgmap v60: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:50.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:50 smithi132 ceph-mon[35283]: pgmap v60: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:50.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:50 smithi179 ceph-mon[35888]: pgmap v60: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:51.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.807+0000 7ff1d1200700 1 -- 172.21.15.43:0/695063632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1cc108d30 msgr2=0x7ff1cc109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.807+0000 7ff1d1200700 1 --2- 172.21.15.43:0/695063632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1cc108d30 0x7ff1cc109140 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff1b80099c0 tx=0x7ff1b8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.808+0000 7ff1d1200700 1 -- 172.21.15.43:0/695063632 shutdown_connections 2024-05-27T07:15:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.808+0000 7ff1d1200700 1 --2- 172.21.15.43:0/695063632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1cc109680 0x7ff1cc110fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.808+0000 7ff1d1200700 1 --2- 172.21.15.43:0/695063632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc104400 0x7ff1cc104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.808+0000 7ff1d1200700 1 --2- 172.21.15.43:0/695063632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1cc108d30 0x7ff1cc109140 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.808+0000 7ff1d1200700 1 -- 172.21.15.43:0/695063632 >> 172.21.15.43:0/695063632 conn(0x7ff1cc079c10 msgr2=0x7ff1cc07a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.808+0000 7ff1d1200700 1 -- 172.21.15.43:0/695063632 shutdown_connections 2024-05-27T07:15:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.808+0000 7ff1d1200700 1 -- 172.21.15.43:0/695063632 wait complete. 2024-05-27T07:15:51.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.809+0000 7ff1d1200700 1 Processor -- start 2024-05-27T07:15:51.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.809+0000 7ff1d1200700 1 -- start start 2024-05-27T07:15:51.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.810+0000 7ff1d1200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1cc104400 0x7ff1cc19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.810+0000 7ff1d1200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc108d30 0x7ff1cc19d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.810+0000 7ff1d1200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1cc109680 0x7ff1cc1a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.810+0000 7ff1d1200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc114190 con 0x7ff1cc109680 2024-05-27T07:15:51.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.810+0000 7ff1d1200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc114010 con 0x7ff1cc104400 2024-05-27T07:15:51.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.810+0000 7ff1d1200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc114310 con 0x7ff1cc108d30 2024-05-27T07:15:51.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.810+0000 7ff1cbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1cc109680 0x7ff1cc1a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.811+0000 7ff1cbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1cc109680 0x7ff1cc1a15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32942/0 (socket says 172.21.15.43:32942) 2024-05-27T07:15:51.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.811+0000 7ff1cbe00700 1 -- 172.21.15.43:0/2249181585 learned_addr learned my addr 172.21.15.43:0/2249181585 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:51.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.811+0000 7ff1cb400700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1cc104400 0x7ff1cc19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.811+0000 7ff1caa00700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc108d30 0x7ff1cc19d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.811+0000 7ff1cbe00700 1 -- 172.21.15.43:0/2249181585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc108d30 msgr2=0x7ff1cc19d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.811+0000 7ff1cbe00700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc108d30 0x7ff1cc19d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.811+0000 7ff1cbe00700 1 -- 172.21.15.43:0/2249181585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1cc104400 msgr2=0x7ff1cc19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.811+0000 7ff1cbe00700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1cc104400 0x7ff1cc19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.811+0000 7ff1cbe00700 1 -- 172.21.15.43:0/2249181585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc1a1bc0 con 0x7ff1cc109680 2024-05-27T07:15:51.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.811+0000 7ff1caa00700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc108d30 0x7ff1cc19d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:51.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.812+0000 7ff1cbe00700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1cc109680 0x7ff1cc1a15d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4007d70 tx=0x7ff1c400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:51.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.812+0000 7ff1c3e00700 1 -- 172.21.15.43:0/2249181585 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4016020 con 0x7ff1cc109680 2024-05-27T07:15:51.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.812+0000 7ff1c3e00700 1 -- 172.21.15.43:0/2249181585 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1c400ad40 con 0x7ff1cc109680 2024-05-27T07:15:51.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.812+0000 7ff1d1200700 1 -- 172.21.15.43:0/2249181585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1cc1a1eb0 con 0x7ff1cc109680 2024-05-27T07:15:51.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.812+0000 7ff1d1200700 1 -- 172.21.15.43:0/2249181585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1cc1a2370 con 0x7ff1cc109680 2024-05-27T07:15:51.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.812+0000 7ff1c3e00700 1 -- 172.21.15.43:0/2249181585 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40115e0 con 0x7ff1cc109680 2024-05-27T07:15:51.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.816+0000 7ff1c3e00700 1 -- 172.21.15.43:0/2249181585 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff1c4019020 con 0x7ff1cc109680 2024-05-27T07:15:51.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.817+0000 7ff1e124eb80 1 Processor -- start 2024-05-27T07:15:51.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.818+0000 7ff1e124eb80 1 -- start start 2024-05-27T07:15:51.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.818+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda8edae40 0x55fda9058ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.818+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059410 0x55fda9063ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.818+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda9059820 0x55fda9066020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.818+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fda8f87e40 con 0x55fda8edae40 2024-05-27T07:15:51.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.818+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fda8f779e0 con 0x55fda9059410 2024-05-27T07:15:51.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.818+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55fda8f7ba20 con 0x55fda9059820 2024-05-27T07:15:51.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.819+0000 7ff1caa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda8edae40 0x55fda9058ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.819+0000 7ff1caa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda8edae40 0x55fda9058ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32954/0 (socket says 172.21.15.43:32954) 2024-05-27T07:15:51.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.819+0000 7ff1caa00700 1 -- 172.21.15.43:0/3195679059 learned_addr learned my addr 172.21.15.43:0/3195679059 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:51.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.819+0000 7ff1cb400700 1 --2- 172.21.15.43:0/3195679059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059410 0x55fda9063ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.819+0000 7ff1c3e00700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1b00b29f0 0x7ff1b00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.819+0000 7ff1caa00700 1 --2- 172.21.15.43:0/3195679059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda9059820 0x55fda9066020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.819+0000 7ff1c3e00700 1 -- 172.21.15.43:0/2249181585 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff1c40e2760 con 0x7ff1cc109680 2024-05-27T07:15:51.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.820+0000 7ff1caa00700 1 -- 172.21.15.43:0/3195679059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda9059820 msgr2=0x55fda9066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:51.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.820+0000 7ff1caa00700 1 --2- 172.21.15.43:0/3195679059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda9059820 0x55fda9066020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.820+0000 7ff1caa00700 1 -- 172.21.15.43:0/3195679059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059410 msgr2=0x55fda9063ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:51.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.820+0000 7ff1caa00700 1 --2- 172.21.15.43:0/3195679059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059410 0x55fda9063ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.820+0000 7ff1caa00700 1 -- 172.21.15.43:0/3195679059 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda8f40420 con 0x55fda8edae40 2024-05-27T07:15:51.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.820+0000 7ff1caa00700 1 --2- 172.21.15.43:0/3195679059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda8edae40 0x55fda9058ed0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc06ce90 tx=0x7ff1b8009690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac50a229126ffc5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:51.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.820+0000 7ff1cbe00700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1b00b29f0 0x7ff1b00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.820+0000 7ff1c0c00700 1 -- 172.21.15.43:0/3195679059 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b800bc70 con 0x55fda8edae40 2024-05-27T07:15:51.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.820+0000 7ff1c0c00700 1 -- 172.21.15.43:0/3195679059 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1b8022040 con 0x55fda8edae40 2024-05-27T07:15:51.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.820+0000 7ff1c0c00700 1 -- 172.21.15.43:0/3195679059 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b8005490 con 0x55fda8edae40 2024-05-27T07:15:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.821+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3195679059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda8edae40 msgr2=0x55fda9058ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.821+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/3195679059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda8edae40 0x55fda9058ed0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc06ce90 tx=0x7ff1b8009690 comp rx=0 tx=0).stop 2024-05-27T07:15:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.821+0000 7ff1cbe00700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1b00b29f0 0x7ff1b00b4f00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40a9730 tx=0x7ff1c4007ec0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.822+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3195679059 shutdown_connections 2024-05-27T07:15:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.822+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/3195679059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda9059820 0x55fda9066020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.822+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/3195679059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda8edae40 0x55fda9058ed0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.822+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/3195679059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059410 0x55fda9063ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.822+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3195679059 >> 172.21.15.43:0/3195679059 conn(0x55fda8f3afc0 msgr2=0x55fda9054d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:51.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.822+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3195679059 shutdown_connections 2024-05-27T07:15:51.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.822+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3195679059 wait complete. 2024-05-27T07:15:51.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.822+0000 7ff1e124eb80 1 -- start start 2024-05-27T07:15:51.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.823+0000 7ff1e124eb80 1 Processor -- start 2024-05-27T07:15:51.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.823+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda9063620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.823+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda9063b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.823+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 0x55fda90640a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.823+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fda8e7dfa0 con 0x55fda9059410 2024-05-27T07:15:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.823+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fda8f18760 con 0x55fda9059820 2024-05-27T07:15:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.823+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55fda9066930 con 0x55fda8edae40 2024-05-27T07:15:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.823+0000 7ff1e124eb80 10 client.0 _ll_register_callbacks cb 0x55fda8f21c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1cb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda9063b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1cb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda9063b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32956/0 (socket says 172.21.15.43:32956) 2024-05-27T07:15:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1cb400700 1 -- 172.21.15.43:0/3642040045 learned_addr learned my addr 172.21.15.43:0/3642040045 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1caa00700 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda9063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1caa00700 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 0x55fda90640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1cb400700 1 -- 172.21.15.43:0/3642040045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 msgr2=0x55fda9063620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1cb400700 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda9063620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1cb400700 1 -- 172.21.15.43:0/3642040045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 msgr2=0x55fda90640a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1cb400700 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 0x55fda90640a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1cb400700 1 -- 172.21.15.43:0/3642040045 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda8ec78d0 con 0x55fda9059410 2024-05-27T07:15:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.824+0000 7ff1caa00700 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 0x55fda90640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.825+0000 7ff1cb400700 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda9063b60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc010a50 tx=0x7ff1bc010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23081d8747754541 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.825+0000 7ff1b5600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.825+0000 7ff1b5600700 1 -- 172.21.15.43:0/3642040045 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc004260 con 0x55fda9059410 2024-05-27T07:15:51.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.825+0000 7ff1b5600700 1 -- 172.21.15.43:0/3642040045 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff1bc004400 con 0x55fda9059410 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.825+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3642040045 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1b8022040 con 0x55fda9059410 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1b5600700 1 -- 172.21.15.43:0/3642040045 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff1bc00f830 con 0x55fda9059410 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1e124eb80 10 client.15759 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3642040045 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fda8f48060 con 0x55fda9059410 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1b5600700 1 -- 172.21.15.43:0/3642040045 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1b8022040 con 0x55fda9059410 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1b5600700 1 -- 172.21.15.43:0/3642040045 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff1bc003c70 con 0x55fda9059410 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1e124eb80 10 client.15759 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1e124eb80 10 client.15759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1e124eb80 20 client.15759 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fda9073a30 0x55fda9075e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1e124eb80 4 client.15759 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:51.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.826+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3642040045 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fda8f89d90 con 0x55fda9073a30 2024-05-27T07:15:51.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.827+0000 7ff1caa00700 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fda9073a30 0x55fda9075e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.828+0000 7ff1b5600700 10 client.15759 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:51.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.828+0000 7ff1caa00700 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fda9073a30 0x55fda9075e50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f30086b983463c server_cookie=4d2e4b7e8004e3d5 in_seq=0 out_seq=0 2024-05-27T07:15:51.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.830+0000 7ff1b5600700 1 -- 172.21.15.43:0/3642040045 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fda8f89d90 con 0x55fda9073a30 2024-05-27T07:15:51.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.830+0000 7ff1b5600700 10 client.15759 handle_command_reply: tid=0 2024-05-27T07:15:51.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.830+0000 7ff1e124eb80 1 client.15759 shutdown 2024-05-27T07:15:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3642040045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 msgr2=0x55fda9063b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda9063b60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc010a50 tx=0x7ff1bc010e10 comp rx=0 tx=0).stop 2024-05-27T07:15:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3642040045 shutdown_connections 2024-05-27T07:15:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 0x55fda90640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda9063620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fda9073a30 0x55fda9075e50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/3642040045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda9063b60 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3642040045 >> 172.21.15.43:0/3642040045 conn(0x55fda8f3afc0 msgr2=0x55fda9053ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1b5600700 0 client.15759 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:51.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3642040045 shutdown_connections 2024-05-27T07:15:51.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 1 -- 172.21.15.43:0/3642040045 wait complete. 2024-05-27T07:15:51.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.831+0000 7ff1e124eb80 20 client.15759 trim_cache size 0 max 16384 2024-05-27T07:15:51.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.853+0000 7ff1e124eb80 1 Processor -- start 2024-05-27T07:15:51.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.853+0000 7ff1e124eb80 1 -- start start 2024-05-27T07:15:51.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.853+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda905ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.853+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda906fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.853+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 0x55fda907b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.853+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac00ecc0 con 0x55fda9059410 2024-05-27T07:15:51.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.853+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac010800 con 0x55fda9059820 2024-05-27T07:15:51.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.853+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac00f390 con 0x55fda8edae40 2024-05-27T07:15:51.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.854+0000 7ff1cb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda906fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.854+0000 7ff1cb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda906fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32962/0 (socket says 172.21.15.43:32962) 2024-05-27T07:15:51.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.854+0000 7ff1cb400700 1 -- 172.21.15.43:0/2396175970 learned_addr learned my addr 172.21.15.43:0/2396175970 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:51.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.854+0000 7ff1caa00700 1 --2- 172.21.15.43:0/2396175970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda905ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.854+0000 7ff1caa00700 1 --2- 172.21.15.43:0/2396175970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 0x55fda907b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.855+0000 7ff1cb400700 1 -- 172.21.15.43:0/2396175970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 msgr2=0x55fda905ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:51.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.855+0000 7ff1cb400700 1 --2- 172.21.15.43:0/2396175970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda905ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.855+0000 7ff1cb400700 1 -- 172.21.15.43:0/2396175970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 msgr2=0x55fda907b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:51.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.855+0000 7ff1cb400700 1 --2- 172.21.15.43:0/2396175970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 0x55fda907b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.855+0000 7ff1cb400700 1 -- 172.21.15.43:0/2396175970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda9056350 con 0x55fda9059410 2024-05-27T07:15:51.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.855+0000 7ff1cb400700 1 --2- 172.21.15.43:0/2396175970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda906fa60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc01a6e0 tx=0x7ff1bc00f780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e05888f0a7ae5f47 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:51.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.855+0000 7ff1b7e00700 1 -- 172.21.15.43:0/2396175970 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc02e070 con 0x55fda9059410 2024-05-27T07:15:51.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.855+0000 7ff1b7e00700 1 -- 172.21.15.43:0/2396175970 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1bc00f9d0 con 0x55fda9059410 2024-05-27T07:15:51.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.855+0000 7ff1b7e00700 1 -- 172.21.15.43:0/2396175970 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc003680 con 0x55fda9059410 2024-05-27T07:15:51.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.856+0000 7ff1e124eb80 1 -- 172.21.15.43:0/2396175970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 msgr2=0x55fda906fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.856+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/2396175970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda906fa60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc01a6e0 tx=0x7ff1bc00f780 comp rx=0 tx=0).stop 2024-05-27T07:15:51.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.857+0000 7ff1e124eb80 1 -- 172.21.15.43:0/2396175970 shutdown_connections 2024-05-27T07:15:51.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.857+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/2396175970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059820 0x55fda907b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.857+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/2396175970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda905ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.857+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/2396175970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059410 0x55fda906fa60 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.857+0000 7ff1e124eb80 1 -- 172.21.15.43:0/2396175970 >> 172.21.15.43:0/2396175970 conn(0x55fda8f3afc0 msgr2=0x55fda9061f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:51.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.857+0000 7ff1e124eb80 1 -- 172.21.15.43:0/2396175970 shutdown_connections 2024-05-27T07:15:51.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.857+0000 7ff1e124eb80 1 -- 172.21.15.43:0/2396175970 wait complete. 2024-05-27T07:15:51.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.857+0000 7ff1e124eb80 1 -- start start 2024-05-27T07:15:51.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.858+0000 7ff1e124eb80 1 Processor -- start 2024-05-27T07:15:51.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.858+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda9067010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.858+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059410 0x55fda9067550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.858+0000 7ff1e124eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059820 0x55fda90dc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.858+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac010130 con 0x55fda9059820 2024-05-27T07:15:51.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.858+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac00fa60 con 0x55fda9059410 2024-05-27T07:15:51.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.858+0000 7ff1e124eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac00cc40 con 0x55fda8edae40 2024-05-27T07:15:51.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.858+0000 7ff1e124eb80 10 client.0 _ll_register_callbacks cb 0x55fda8f21c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.859+0000 7ff1caa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059820 0x55fda90dc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.859+0000 7ff1caa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059820 0x55fda90dc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32964/0 (socket says 172.21.15.43:32964) 2024-05-27T07:15:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.859+0000 7ff1caa00700 1 -- 172.21.15.43:0/1930193335 learned_addr learned my addr 172.21.15.43:0/1930193335 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.859+0000 7ff1cb400700 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059410 0x55fda9067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.859+0000 7ff1caa00700 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda9067010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.859+0000 7ff1caa00700 1 -- 172.21.15.43:0/1930193335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 msgr2=0x55fda9067010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.859+0000 7ff1caa00700 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda9067010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.859+0000 7ff1caa00700 1 -- 172.21.15.43:0/1930193335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059410 msgr2=0x55fda9067550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.860+0000 7ff1caa00700 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059410 0x55fda9067550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.860+0000 7ff1caa00700 1 -- 172.21.15.43:0/1930193335 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda90dca10 con 0x55fda9059820 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.860+0000 7ff1caa00700 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059820 0x55fda90dc4d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800ba80 tx=0x7ff1bc00ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a684b49c8acb02bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.860+0000 7ff1b5600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.860+0000 7ff1b5600700 1 -- 172.21.15.43:0/1930193335 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b8005620 con 0x55fda9059820 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.860+0000 7ff1b5600700 1 -- 172.21.15.43:0/1930193335 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7ff1b8004770 con 0x55fda9059820 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.860+0000 7ff1e124eb80 1 -- 172.21.15.43:0/1930193335 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1bc00f9d0 con 0x55fda9059820 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.861+0000 7ff1b5600700 1 -- 172.21.15.43:0/1930193335 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff1b8004970 con 0x55fda9059820 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.861+0000 7ff1e124eb80 10 client.15765 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.861+0000 7ff1e124eb80 1 -- 172.21.15.43:0/1930193335 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fda90ddb30 con 0x55fda9059820 2024-05-27T07:15:51.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.861+0000 7ff1b5600700 1 -- 172.21.15.43:0/1930193335 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff1bc00f9d0 con 0x55fda9059820 2024-05-27T07:15:51.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.861+0000 7ff1b5600700 1 -- 172.21.15.43:0/1930193335 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff1b8025020 con 0x55fda9059820 2024-05-27T07:15:51.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.861+0000 7ff1e124eb80 10 client.15765 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:51.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.861+0000 7ff1e124eb80 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:15:51.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.861+0000 7ff1e124eb80 20 client.15765 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:51.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.862+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fda90dddb0 0x55fda90e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:51.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.862+0000 7ff1e124eb80 4 client.15765 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:15:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.862+0000 7ff1e124eb80 1 -- 172.21.15.43:0/1930193335 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fda8f83570 con 0x55fda90dddb0 2024-05-27T07:15:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.862+0000 7ff1cb400700 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fda90dddb0 0x55fda90e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.863+0000 7ff1cb400700 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fda90dddb0 0x55fda90e01d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b50375fd98639aa4 server_cookie=6a254ef34630e05d in_seq=0 out_seq=0 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.863+0000 7ff1b5600700 10 client.15765 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1b5600700 1 -- 172.21.15.43:0/1930193335 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fda8f83570 con 0x55fda90dddb0 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1b5600700 10 client.15765 handle_command_reply: tid=0 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 client.15765 shutdown 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 -- 172.21.15.43:0/1930193335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059820 msgr2=0x55fda90dc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059820 0x55fda90dc4d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800ba80 tx=0x7ff1bc00ae50 comp rx=0 tx=0).stop 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 -- 172.21.15.43:0/1930193335 shutdown_connections 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fda90dddb0 0x55fda90e01d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fda9059820 0x55fda90dc4d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fda8edae40 0x55fda9067010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 --2- 172.21.15.43:0/1930193335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fda9059410 0x55fda9067550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 -- 172.21.15.43:0/1930193335 >> 172.21.15.43:0/1930193335 conn(0x55fda8f3afc0 msgr2=0x55fda905a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1b5600700 0 client.15765 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 -- 172.21.15.43:0/1930193335 shutdown_connections 2024-05-27T07:15:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.864+0000 7ff1e124eb80 1 -- 172.21.15.43:0/1930193335 wait complete. 2024-05-27T07:15:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.865+0000 7ff1e124eb80 20 client.15765 trim_cache size 0 max 16384 2024-05-27T07:15:51.856 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:15:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.865+0000 7ff1d1200700 1 -- 172.21.15.43:0/2249181585 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1b00b29f0 msgr2=0x7ff1b00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.865+0000 7ff1d1200700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1b00b29f0 0x7ff1b00b4f00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40a9730 tx=0x7ff1c4007ec0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.865+0000 7ff1d1200700 1 -- 172.21.15.43:0/2249181585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1cc109680 msgr2=0x7ff1cc1a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.865+0000 7ff1d1200700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1cc109680 0x7ff1cc1a15d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4007d70 tx=0x7ff1c400a340 comp rx=0 tx=0).stop 2024-05-27T07:15:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.866+0000 7ff1d1200700 1 -- 172.21.15.43:0/2249181585 shutdown_connections 2024-05-27T07:15:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.866+0000 7ff1d1200700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff1b00b29f0 0x7ff1b00b4f00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.866+0000 7ff1d1200700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1cc109680 0x7ff1cc1a15d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.866+0000 7ff1d1200700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1cc104400 0x7ff1cc19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.866+0000 7ff1d1200700 1 --2- 172.21.15.43:0/2249181585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc108d30 0x7ff1cc19d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:51.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.866+0000 7ff1d1200700 1 -- 172.21.15.43:0/2249181585 >> 172.21.15.43:0/2249181585 conn(0x7ff1cc079c10 msgr2=0x7ff1cc110cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:51.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.866+0000 7ff1d1200700 1 -- 172.21.15.43:0/2249181585 shutdown_connections 2024-05-27T07:15:51.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:51.866+0000 7ff1d1200700 1 -- 172.21.15.43:0/2249181585 wait complete. 2024-05-27T07:15:52.301 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:52 smithi043 ceph-mon[31531]: pgmap v61: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:52.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:52 smithi132 ceph-mon[35283]: pgmap v61: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:52.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:52 smithi179 ceph-mon[35888]: pgmap v61: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:52.740 INFO:tasks.workunit.client.0.smithi043.stderr:Updating files: 55% (6790/12271) Updating files: 56% (6872/12271) Updating files: 57% (6995/12271) Updating files: 58% (7118/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 90% (11057/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-27T07:15:53.819 INFO:tasks.workunit.client.0.smithi043.stderr:Note: switching to '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'. 2024-05-27T07:15:53.819 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-05-27T07:15:53.819 INFO:tasks.workunit.client.0.smithi043.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-27T07:15:53.819 INFO:tasks.workunit.client.0.smithi043.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-27T07:15:53.819 INFO:tasks.workunit.client.0.smithi043.stderr:state without impacting any branches by switching back to a branch. 2024-05-27T07:15:53.819 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr: git switch -c 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr:Or undo this operation with: 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr: git switch - 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-05-27T07:15:53.820 INFO:tasks.workunit.client.0.smithi043.stderr:HEAD is now at 38f9d900c83 Merge PR #55758 into wip-vshankar-testing-20240521.063558-debug 2024-05-27T07:15:53.834 DEBUG:teuthology.orchestra.run.smithi043:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-27T07:15:53.840 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:15:53.840 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:15:53.896 INFO:tasks.workunit.client.0.smithi043.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-27T07:15:53.904 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T07:15:53.904 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-27T07:15:54.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:54 smithi043 ceph-mon[31531]: pgmap v62: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:54.325 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:54.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:54 smithi132 ceph-mon[35283]: pgmap v62: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:54 smithi179 ceph-mon[35888]: pgmap v62: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:55.455 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-27T07:15:55.631 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-27T07:15:55.723 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T07:15:55.729 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T07:15:55.729 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-27T07:15:55.813 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T07:15:55.823 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T07:15:55.824 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-27T07:15:55.926 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-05-27T07:15:55.928 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-05-27T07:15:55.928 DEBUG:teuthology.orchestra.run.smithi043:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-05-27T07:15:56.010 INFO:tasks.workunit.client.0.smithi043.stderr:+ dbench 1 2024-05-27T07:15:56.023 INFO:tasks.workunit.client.0.smithi043.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-27T07:15:56.024 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-05-27T07:15:56.024 INFO:tasks.workunit.client.0.smithi043.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-27T07:15:56.073 INFO:tasks.workunit.client.0.smithi043.stdout:0 of 1 processes prepared for launch 0 sec 2024-05-27T07:15:56.173 INFO:tasks.workunit.client.0.smithi043.stdout:1 of 1 processes prepared for launch 0 sec 2024-05-27T07:15:56.173 INFO:tasks.workunit.client.0.smithi043.stdout:releasing clients 2024-05-27T07:15:56.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:56 smithi043 ceph-mon[31531]: pgmap v63: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:56.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:56 smithi132 ceph-mon[35283]: pgmap v63: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:56.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:56 smithi179 ceph-mon[35888]: pgmap v63: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:15:56.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.882+0000 7f9828000700 1 -- 172.21.15.43:0/1249025958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98201030b0 msgr2=0x7f98201034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:56.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.882+0000 7f9828000700 1 --2- 172.21.15.43:0/1249025958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98201030b0 0x7f98201034c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9814009990 tx=0x7f9814009ca0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.882+0000 7f9828000700 1 -- 172.21.15.43:0/1249025958 shutdown_connections 2024-05-27T07:15:56.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.882+0000 7f9828000700 1 --2- 172.21.15.43:0/1249025958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9820110c60 0x7f9820113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.882+0000 7f9828000700 1 --2- 172.21.15.43:0/1249025958 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98201030b0 0x7f98201034c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.882+0000 7f9828000700 1 --2- 172.21.15.43:0/1249025958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9820103a00 0x7f9820110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.882+0000 7f9828000700 1 -- 172.21.15.43:0/1249025958 >> 172.21.15.43:0/1249025958 conn(0x7f98200fedb0 msgr2=0x7f98201011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:56.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.882+0000 7f9828000700 1 -- 172.21.15.43:0/1249025958 shutdown_connections 2024-05-27T07:15:56.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.882+0000 7f9828000700 1 -- 172.21.15.43:0/1249025958 wait complete. 2024-05-27T07:15:56.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.884+0000 7f9828000700 1 Processor -- start 2024-05-27T07:15:56.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.884+0000 7f9828000700 1 -- start start 2024-05-27T07:15:56.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9828000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98201030b0 0x7f982019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9828000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9820103a00 0x7f982019d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9828000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9820110c60 0x7f98201a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9828000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9820116550 con 0x7f9820110c60 2024-05-27T07:15:56.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9828000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98201163d0 con 0x7f98201030b0 2024-05-27T07:15:56.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9828000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f98201166d0 con 0x7f9820103a00 2024-05-27T07:15:56.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9826000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9820110c60 0x7f98201a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9826000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9820110c60 0x7f98201a17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50680/0 (socket says 172.21.15.43:50680) 2024-05-27T07:15:56.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9826000700 1 -- 172.21.15.43:0/4060715714 learned_addr learned my addr 172.21.15.43:0/4060715714 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:56.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9824c00700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9820103a00 0x7f982019d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.885+0000 7f9825600700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98201030b0 0x7f982019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.886+0000 7f9826000700 1 -- 172.21.15.43:0/4060715714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9820103a00 msgr2=0x7f982019d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:56.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.886+0000 7f9826000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9820103a00 0x7f982019d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.886+0000 7f9826000700 1 -- 172.21.15.43:0/4060715714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98201030b0 msgr2=0x7f982019cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:56.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.886+0000 7f9826000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98201030b0 0x7f982019cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.886+0000 7f9826000700 1 -- 172.21.15.43:0/4060715714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98201a1de0 con 0x7f9820110c60 2024-05-27T07:15:56.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.886+0000 7f9826000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9820110c60 0x7f98201a17f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9818007d70 tx=0x7f981800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:56.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.887+0000 7f981e000700 1 -- 172.21.15.43:0/4060715714 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9818016020 con 0x7f9820110c60 2024-05-27T07:15:56.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.887+0000 7f981e000700 1 -- 172.21.15.43:0/4060715714 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f981800ad40 con 0x7f9820110c60 2024-05-27T07:15:56.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.887+0000 7f9828000700 1 -- 172.21.15.43:0/4060715714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98201a20d0 con 0x7f9820110c60 2024-05-27T07:15:56.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.887+0000 7f981e000700 1 -- 172.21.15.43:0/4060715714 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98180115e0 con 0x7f9820110c60 2024-05-27T07:15:56.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.887+0000 7f9828000700 1 -- 172.21.15.43:0/4060715714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98201a25c0 con 0x7f9820110c60 2024-05-27T07:15:56.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.892+0000 7f981e000700 1 -- 172.21.15.43:0/4060715714 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9818019020 con 0x7f9820110c60 2024-05-27T07:15:56.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.893+0000 7f9836af7b80 1 Processor -- start 2024-05-27T07:15:56.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.893+0000 7f9836af7b80 1 -- start start 2024-05-27T07:15:56.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b3535ec0 0x5571b36b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b36b4ef0 0x5571b36c1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f9836af7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5571b35e2ec0 con 0x5571b3621a70 2024-05-27T07:15:56.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f9836af7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5571b35d2a60 con 0x5571b36b4ef0 2024-05-27T07:15:56.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f9836af7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5571b35d6aa0 con 0x5571b3535ec0 2024-05-27T07:15:56.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f981e000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98080b29f0 0x7f98080b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f9824c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f9824c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36b43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50696/0 (socket says 172.21.15.43:50696) 2024-05-27T07:15:56.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f9824c00700 1 -- 172.21.15.43:0/3150735005 learned_addr learned my addr 172.21.15.43:0/3150735005 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:56.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.894+0000 7f981e000700 1 -- 172.21.15.43:0/4060715714 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f98180e27f0 con 0x7f9820110c60 2024-05-27T07:15:56.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.895+0000 7f9824c00700 1 --2- 172.21.15.43:0/3150735005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b36b4ef0 0x5571b36c1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.895+0000 7f9825600700 1 --2- 172.21.15.43:0/3150735005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b3535ec0 0x5571b36b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.896+0000 7f9826000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98080b29f0 0x7f98080b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.896+0000 7f9824c00700 1 -- 172.21.15.43:0/3150735005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b3535ec0 msgr2=0x5571b36b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:56.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.896+0000 7f9824c00700 1 --2- 172.21.15.43:0/3150735005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b3535ec0 0x5571b36b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.896+0000 7f9824c00700 1 -- 172.21.15.43:0/3150735005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b36b4ef0 msgr2=0x5571b36c1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:56.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.896+0000 7f9824c00700 1 --2- 172.21.15.43:0/3150735005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b36b4ef0 0x5571b36c1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.896+0000 7f9824c00700 1 -- 172.21.15.43:0/3150735005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571b359b4a0 con 0x5571b3621a70 2024-05-27T07:15:56.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.896+0000 7f9824c00700 1 --2- 172.21.15.43:0/3150735005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36b43b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f980c00bef0 tx=0x7f980c005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae65c637fddcb444 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:56.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.897+0000 7f9826000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98080b29f0 0x7f98080b4f00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f98180a9530 tx=0x7f9818007ec0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:56.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.897+0000 7f9812a00700 1 -- 172.21.15.43:0/3150735005 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c012110 con 0x5571b3621a70 2024-05-27T07:15:56.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.897+0000 7f9812a00700 1 -- 172.21.15.43:0/3150735005 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f980c0044c0 con 0x5571b3621a70 2024-05-27T07:15:56.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.897+0000 7f9812a00700 1 -- 172.21.15.43:0/3150735005 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c005280 con 0x5571b3621a70 2024-05-27T07:15:56.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.898+0000 7f9836af7b80 1 -- 172.21.15.43:0/3150735005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 msgr2=0x5571b36b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:56.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.898+0000 7f9836af7b80 1 --2- 172.21.15.43:0/3150735005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36b43b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f980c00bef0 tx=0x7f980c005d20 comp rx=0 tx=0).stop 2024-05-27T07:15:56.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.898+0000 7f9836af7b80 1 -- 172.21.15.43:0/3150735005 shutdown_connections 2024-05-27T07:15:56.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.898+0000 7f9836af7b80 1 --2- 172.21.15.43:0/3150735005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b36b4ef0 0x5571b36c1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.898+0000 7f9836af7b80 1 --2- 172.21.15.43:0/3150735005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36b43b0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.898+0000 7f9836af7b80 1 --2- 172.21.15.43:0/3150735005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b3535ec0 0x5571b36b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.898+0000 7f9836af7b80 1 -- 172.21.15.43:0/3150735005 >> 172.21.15.43:0/3150735005 conn(0x5571b35cf410 msgr2=0x5571b36b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:56.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.899+0000 7f9836af7b80 1 -- 172.21.15.43:0/3150735005 shutdown_connections 2024-05-27T07:15:56.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.899+0000 7f9836af7b80 1 -- 172.21.15.43:0/3150735005 wait complete. 2024-05-27T07:15:56.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.899+0000 7f9836af7b80 1 -- start start 2024-05-27T07:15:56.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.899+0000 7f9836af7b80 1 Processor -- start 2024-05-27T07:15:56.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.900+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36be6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.900+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b3621a70 0x5571b36bec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.900+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b36b4ef0 0x5571b36bf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.900+0000 7f9836af7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5571b34d9020 con 0x5571b36b4ef0 2024-05-27T07:15:56.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.900+0000 7f9836af7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5571b35737e0 con 0x5571b3535ec0 2024-05-27T07:15:56.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.900+0000 7f9836af7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5571b36c1c00 con 0x5571b3621a70 2024-05-27T07:15:56.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.900+0000 7f9836af7b80 10 client.0 _ll_register_callbacks cb 0x5571b357ccf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:56.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.901+0000 7f9824c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b36b4ef0 0x5571b36bf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.901+0000 7f9825600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b3621a70 0x5571b36bec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.901+0000 7f9824c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b36b4ef0 0x5571b36bf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50708/0 (socket says 172.21.15.43:50708) 2024-05-27T07:15:56.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.901+0000 7f9824c00700 1 -- 172.21.15.43:0/3300533680 learned_addr learned my addr 172.21.15.43:0/3300533680 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:56.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.901+0000 7f9824c00700 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36be6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.901+0000 7f9824c00700 1 -- 172.21.15.43:0/3300533680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b3621a70 msgr2=0x5571b36bec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:56.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.901+0000 7f9824c00700 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b3621a70 0x5571b36bec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.901+0000 7f9824c00700 1 -- 172.21.15.43:0/3300533680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 msgr2=0x5571b36be6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:56.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.901+0000 7f9824c00700 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36be6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.901+0000 7f9824c00700 1 -- 172.21.15.43:0/3300533680 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571b3522950 con 0x5571b36b4ef0 2024-05-27T07:15:56.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.902+0000 7f9824c00700 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b36b4ef0 0x5571b36bf160 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f980c00bad0 tx=0x7f980c00bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d739c7d2c80db3c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:56.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.902+0000 7f9807400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:56.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.902+0000 7f9807400700 1 -- 172.21.15.43:0/3300533680 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c027040 con 0x5571b36b4ef0 2024-05-27T07:15:56.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.902+0000 7f9807400700 1 -- 172.21.15.43:0/3300533680 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f980c004750 con 0x5571b36b4ef0 2024-05-27T07:15:56.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.902+0000 7f9836af7b80 1 -- 172.21.15.43:0/3300533680 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f980c0044c0 con 0x5571b36b4ef0 2024-05-27T07:15:56.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.903+0000 7f9807400700 1 -- 172.21.15.43:0/3300533680 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f980c004950 con 0x5571b36b4ef0 2024-05-27T07:15:56.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.903+0000 7f9836af7b80 10 client.15777 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:56.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.903+0000 7f9836af7b80 1 -- 172.21.15.43:0/3300533680 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5571b35a30e0 con 0x5571b36b4ef0 2024-05-27T07:15:56.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.903+0000 7f9807400700 1 -- 172.21.15.43:0/3300533680 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f980c0044c0 con 0x5571b36b4ef0 2024-05-27T07:15:56.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.903+0000 7f9807400700 1 -- 172.21.15.43:0/3300533680 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f980c01c020 con 0x5571b36b4ef0 2024-05-27T07:15:56.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.903+0000 7f9836af7b80 10 client.15777 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:56.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.904+0000 7f9836af7b80 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:15:56.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.904+0000 7f9836af7b80 20 client.15777 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:56.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.904+0000 7f9836af7b80 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5571b36cea00 0x5571b36d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.904+0000 7f9836af7b80 4 client.15777 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:15:56.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.904+0000 7f9836af7b80 1 -- 172.21.15.43:0/3300533680 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571b35e4e10 con 0x5571b36cea00 2024-05-27T07:15:56.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.904+0000 7f9825600700 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5571b36cea00 0x5571b36d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.905+0000 7f9825600700 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5571b36cea00 0x5571b36d0e20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=180799ea55f5d796 server_cookie=1c5fabfe28ab51d3 in_seq=0 out_seq=0 2024-05-27T07:15:56.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.905+0000 7f9807400700 10 client.15777 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:56.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9807400700 1 -- 172.21.15.43:0/3300533680 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5571b35e4e10 con 0x5571b36cea00 2024-05-27T07:15:56.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9807400700 10 client.15777 handle_command_reply: tid=0 2024-05-27T07:15:56.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9836af7b80 1 client.15777 shutdown 2024-05-27T07:15:56.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9836af7b80 1 -- 172.21.15.43:0/3300533680 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b36b4ef0 msgr2=0x5571b36bf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:56.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9836af7b80 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b36b4ef0 0x5571b36bf160 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f980c00bad0 tx=0x7f980c00bc60 comp rx=0 tx=0).stop 2024-05-27T07:15:56.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9836af7b80 1 -- 172.21.15.43:0/3300533680 shutdown_connections 2024-05-27T07:15:56.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9836af7b80 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5571b36cea00 0x5571b36d0e20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9836af7b80 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b36b4ef0 0x5571b36bf160 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9836af7b80 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36be6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9836af7b80 1 --2- 172.21.15.43:0/3300533680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b3621a70 0x5571b36bec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9807400700 0 client.15777 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:56.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.908+0000 7f9836af7b80 1 -- 172.21.15.43:0/3300533680 >> 172.21.15.43:0/3300533680 conn(0x5571b35cf410 msgr2=0x5571b36b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:56.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.909+0000 7f9836af7b80 1 -- 172.21.15.43:0/3300533680 shutdown_connections 2024-05-27T07:15:56.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.909+0000 7f9836af7b80 1 -- 172.21.15.43:0/3300533680 wait complete. 2024-05-27T07:15:56.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.909+0000 7f9836af7b80 20 client.15777 trim_cache size 0 max 16384 2024-05-27T07:15:56.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.936+0000 7f9836af7b80 1 Processor -- start 2024-05-27T07:15:56.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.936+0000 7f9836af7b80 1 -- start start 2024-05-27T07:15:56.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.936+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.937+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.937+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b36b4ef0 0x5571b36d63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.937+0000 7f9836af7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f98201b26c0 con 0x5571b3621a70 2024-05-27T07:15:56.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.937+0000 7f9836af7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98201b4200 con 0x5571b3535ec0 2024-05-27T07:15:56.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.937+0000 7f9836af7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f98201b2d90 con 0x5571b36b4ef0 2024-05-27T07:15:56.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.937+0000 7f9825600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.937+0000 7f9825600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50712/0 (socket says 172.21.15.43:50712) 2024-05-27T07:15:56.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.937+0000 7f9825600700 1 -- 172.21.15.43:0/4187700259 learned_addr learned my addr 172.21.15.43:0/4187700259 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:56.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9824c00700 1 --2- 172.21.15.43:0/4187700259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9825600700 1 -- 172.21.15.43:0/4187700259 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b36b4ef0 msgr2=0x5571b36d63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:56.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9825600700 1 --2- 172.21.15.43:0/4187700259 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b36b4ef0 0x5571b36d63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9825600700 1 -- 172.21.15.43:0/4187700259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 msgr2=0x5571b36bf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:56.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9825600700 1 --2- 172.21.15.43:0/4187700259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36bf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9825600700 1 -- 172.21.15.43:0/4187700259 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571b36b5880 con 0x5571b3621a70 2024-05-27T07:15:56.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9824c00700 1 --2- 172.21.15.43:0/4187700259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36bf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:15:56.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9825600700 1 --2- 172.21.15.43:0/4187700259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36caca0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9814009750 tx=0x7f9814005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b150ce15a0c3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:56.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9812000700 1 -- 172.21.15.43:0/4187700259 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98140037c0 con 0x5571b3621a70 2024-05-27T07:15:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9812000700 1 -- 172.21.15.43:0/4187700259 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9814003df0 con 0x5571b3621a70 2024-05-27T07:15:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.938+0000 7f9812000700 1 -- 172.21.15.43:0/4187700259 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981400f460 con 0x5571b3621a70 2024-05-27T07:15:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.940+0000 7f9836af7b80 1 -- 172.21.15.43:0/4187700259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 msgr2=0x5571b36caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.940+0000 7f9836af7b80 1 --2- 172.21.15.43:0/4187700259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36caca0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9814009750 tx=0x7f9814005040 comp rx=0 tx=0).stop 2024-05-27T07:15:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.940+0000 7f9836af7b80 1 -- 172.21.15.43:0/4187700259 shutdown_connections 2024-05-27T07:15:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.940+0000 7f9836af7b80 1 --2- 172.21.15.43:0/4187700259 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b36b4ef0 0x5571b36d63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.940+0000 7f9836af7b80 1 --2- 172.21.15.43:0/4187700259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36bf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.940+0000 7f9836af7b80 1 --2- 172.21.15.43:0/4187700259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36caca0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.940+0000 7f9836af7b80 1 -- 172.21.15.43:0/4187700259 >> 172.21.15.43:0/4187700259 conn(0x5571b35cf410 msgr2=0x5571b36b85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.940+0000 7f9836af7b80 1 -- 172.21.15.43:0/4187700259 shutdown_connections 2024-05-27T07:15:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.940+0000 7f9836af7b80 1 -- 172.21.15.43:0/4187700259 wait complete. 2024-05-27T07:15:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.941+0000 7f9836af7b80 1 -- start start 2024-05-27T07:15:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.941+0000 7f9836af7b80 1 Processor -- start 2024-05-27T07:15:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.941+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.941+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.941+0000 7f9836af7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b36b4ef0 0x5571b3740ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.941+0000 7f9836af7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f98201b3b30 con 0x5571b3621a70 2024-05-27T07:15:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.941+0000 7f9836af7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98201b3460 con 0x5571b3535ec0 2024-05-27T07:15:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.941+0000 7f9836af7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f98201b0640 con 0x5571b36b4ef0 2024-05-27T07:15:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9836af7b80 10 client.0 _ll_register_callbacks cb 0x5571b357ccf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:15:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9825600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9825600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36c34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50728/0 (socket says 172.21.15.43:50728) 2024-05-27T07:15:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9825600700 1 -- 172.21.15.43:0/2075728900 learned_addr learned my addr 172.21.15.43:0/2075728900 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:15:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9824c00700 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b36b4ef0 0x5571b3740ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9824c00700 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9825600700 1 -- 172.21.15.43:0/2075728900 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b36b4ef0 msgr2=0x5571b3740ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9825600700 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b36b4ef0 0x5571b3740ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9825600700 1 -- 172.21.15.43:0/2075728900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 msgr2=0x5571b36c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9825600700 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9825600700 1 -- 172.21.15.43:0/2075728900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571b36c3db0 con 0x5571b3621a70 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.942+0000 7f9824c00700 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b36b4ef0 0x5571b3740ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.943+0000 7f9825600700 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36c34d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f981400b5c0 tx=0x7f9814005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b16ae466fbede81 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.943+0000 7f9807400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.943+0000 7f9807400700 1 -- 172.21.15.43:0/2075728900 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981403c060 con 0x5571b3621a70 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.943+0000 7f9807400700 1 -- 172.21.15.43:0/2075728900 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7f981400b770 con 0x5571b3621a70 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.943+0000 7f9836af7b80 1 -- 172.21.15.43:0/2075728900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9814003df0 con 0x5571b3621a70 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.943+0000 7f9807400700 1 -- 172.21.15.43:0/2075728900 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f981400b970 con 0x5571b3621a70 2024-05-27T07:15:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.943+0000 7f9836af7b80 10 client.15783 fetch_fsmap learned FSMap version 30 2024-05-27T07:15:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.944+0000 7f9836af7b80 1 -- 172.21.15.43:0/2075728900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5571b3742030 con 0x5571b3621a70 2024-05-27T07:15:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.944+0000 7f9807400700 1 -- 172.21.15.43:0/2075728900 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9814003df0 con 0x5571b3621a70 2024-05-27T07:15:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.944+0000 7f9807400700 1 -- 172.21.15.43:0/2075728900 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f981400fa80 con 0x5571b3621a70 2024-05-27T07:15:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.944+0000 7f9836af7b80 10 client.15783 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:15:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.944+0000 7f9836af7b80 10 client.15783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:15:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.944+0000 7f9836af7b80 20 client.15783 populate_metadata read hostname 'smithi043' 2024-05-27T07:15:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.944+0000 7f9836af7b80 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5571b37425f0 0x5571b3744a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:15:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.944+0000 7f9836af7b80 4 client.15783 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:15:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.944+0000 7f9836af7b80 1 -- 172.21.15.43:0/2075728900 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5571b35de5f0 con 0x5571b37425f0 2024-05-27T07:15:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.945+0000 7f9824c00700 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5571b37425f0 0x5571b3744a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:15:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.946+0000 7f9824c00700 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5571b37425f0 0x5571b3744a10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19e6a0ee8104eca9 server_cookie=bd6197392e9af292 in_seq=0 out_seq=0 2024-05-27T07:15:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.946+0000 7f9807400700 10 client.15783 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.951+0000 7f9807400700 1 -- 172.21.15.43:0/2075728900 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5571b35de5f0 con 0x5571b37425f0 2024-05-27T07:15:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.951+0000 7f9807400700 10 client.15783 handle_command_reply: tid=0 2024-05-27T07:15:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.951+0000 7f9836af7b80 1 client.15783 shutdown 2024-05-27T07:15:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 1 -- 172.21.15.43:0/2075728900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 msgr2=0x5571b36c34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36c34d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f981400b5c0 tx=0x7f9814005790 comp rx=0 tx=0).stop 2024-05-27T07:15:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 1 -- 172.21.15.43:0/2075728900 shutdown_connections 2024-05-27T07:15:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571b36b4ef0 0x5571b3740ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571b3535ec0 0x5571b36c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5571b37425f0 0x5571b3744a10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 1 --2- 172.21.15.43:0/2075728900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5571b3621a70 0x5571b36c34d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 1 -- 172.21.15.43:0/2075728900 >> 172.21.15.43:0/2075728900 conn(0x5571b35cf410 msgr2=0x5571b36bd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9807400700 0 client.15783 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:15:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 1 -- 172.21.15.43:0/2075728900 shutdown_connections 2024-05-27T07:15:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 1 -- 172.21.15.43:0/2075728900 wait complete. 2024-05-27T07:15:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.952+0000 7f9836af7b80 20 client.15783 trim_cache size 0 max 16384 2024-05-27T07:15:56.944 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:15:56.944 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:15:56.944 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "98944aad-55cd-4eda-9108-0ef2ca9b5c45", 2024-05-27T07:15:56.944 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:15:56.944 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:15:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 -- 172.21.15.43:0/4060715714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98080b29f0 msgr2=0x7f98080b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98080b29f0 0x7f98080b4f00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f98180a9530 tx=0x7f9818007ec0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 -- 172.21.15.43:0/4060715714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9820110c60 msgr2=0x7f98201a17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:15:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9820110c60 0x7f98201a17f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9818007d70 tx=0x7f981800a340 comp rx=0 tx=0).stop 2024-05-27T07:15:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 -- 172.21.15.43:0/4060715714 shutdown_connections 2024-05-27T07:15:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98080b29f0 0x7f98080b4f00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9820110c60 0x7f98201a17f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98201030b0 0x7f982019cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 --2- 172.21.15.43:0/4060715714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9820103a00 0x7f982019d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:15:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 -- 172.21.15.43:0/4060715714 >> 172.21.15.43:0/4060715714 conn(0x7f98200fedb0 msgr2=0x7f9820112870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:15:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 -- 172.21.15.43:0/4060715714 shutdown_connections 2024-05-27T07:15:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:15:56.953+0000 7f9828000700 1 -- 172.21.15.43:0/4060715714 wait complete. 2024-05-27T07:15:57.173 INFO:tasks.workunit.client.0.smithi043.stdout: 1 271 10.18 MB/sec warmup 1 sec latency 35.350 ms 2024-05-27T07:15:57.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:15:58.173 INFO:tasks.workunit.client.0.smithi043.stdout: 1 501 8.73 MB/sec warmup 2 sec latency 33.015 ms 2024-05-27T07:15:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:58 smithi043 ceph-mon[31531]: pgmap v64: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:58 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:15:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:58 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:58 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:15:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:15:58 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:58 smithi132 ceph-mon[35283]: pgmap v64: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:58 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:15:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:58 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:58 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:15:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:15:58 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:58 smithi179 ceph-mon[35888]: pgmap v64: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-05-27T07:15:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:58 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:15:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:58 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:58 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:15:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:15:58 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:15:58.482 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:15:59.173 INFO:tasks.workunit.client.0.smithi043.stdout: 1 743 7.11 MB/sec warmup 3 sec latency 51.401 ms 2024-05-27T07:16:00.173 INFO:tasks.workunit.client.0.smithi043.stdout: 1 880 5.42 MB/sec warmup 4 sec latency 57.911 ms 2024-05-27T07:16:00.336 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:00 smithi043 ceph-mon[31531]: pgmap v65: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:16:00.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:00 smithi132 ceph-mon[35283]: pgmap v65: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:16:00.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:00 smithi179 ceph-mon[35888]: pgmap v65: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-27T07:16:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.955+0000 7fe299600700 1 -- 172.21.15.43:0/1547499779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe294106560 msgr2=0x7fe294106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.955+0000 7fe299600700 1 --2- 172.21.15.43:0/1547499779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe294106560 0x7fe294106970 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009990 tx=0x7fe27c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.955+0000 7fe299600700 1 -- 172.21.15.43:0/1547499779 shutdown_connections 2024-05-27T07:16:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.955+0000 7fe299600700 1 --2- 172.21.15.43:0/1547499779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe294106eb0 0x7fe294113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.955+0000 7fe299600700 1 --2- 172.21.15.43:0/1547499779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe29410c5a0 0x7fe29410c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.955+0000 7fe299600700 1 --2- 172.21.15.43:0/1547499779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe294106560 0x7fe294106970 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.955+0000 7fe299600700 1 -- 172.21.15.43:0/1547499779 >> 172.21.15.43:0/1547499779 conn(0x7fe294102040 msgr2=0x7fe294104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.955+0000 7fe299600700 1 -- 172.21.15.43:0/1547499779 shutdown_connections 2024-05-27T07:16:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.955+0000 7fe299600700 1 -- 172.21.15.43:0/1547499779 wait complete. 2024-05-27T07:16:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.956+0000 7fe299600700 1 Processor -- start 2024-05-27T07:16:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.957+0000 7fe299600700 1 -- start start 2024-05-27T07:16:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.957+0000 7fe299600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe294106560 0x7fe29419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.957+0000 7fe299600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe294106eb0 0x7fe29419d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.957+0000 7fe299600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29410c5a0 0x7fe2941a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.957+0000 7fe299600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe294116580 con 0x7fe29410c5a0 2024-05-27T07:16:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.957+0000 7fe299600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe294116400 con 0x7fe294106560 2024-05-27T07:16:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.957+0000 7fe299600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe294116700 con 0x7fe294106eb0 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.957+0000 7fe293400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29410c5a0 0x7fe2941a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.958+0000 7fe293400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29410c5a0 0x7fe2941a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50746/0 (socket says 172.21.15.43:50746) 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.958+0000 7fe293400700 1 -- 172.21.15.43:0/1185377673 learned_addr learned my addr 172.21.15.43:0/1185377673 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.958+0000 7fe292a00700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe294106560 0x7fe29419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.958+0000 7fe28be00700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe294106eb0 0x7fe29419d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.958+0000 7fe293400700 1 -- 172.21.15.43:0/1185377673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe294106eb0 msgr2=0x7fe29419d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.958+0000 7fe293400700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe294106eb0 0x7fe29419d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.958+0000 7fe293400700 1 -- 172.21.15.43:0/1185377673 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe294106560 msgr2=0x7fe29419cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.958+0000 7fe293400700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe294106560 0x7fe29419cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.958+0000 7fe293400700 1 -- 172.21.15.43:0/1185377673 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2941a1e40 con 0x7fe29410c5a0 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.958+0000 7fe293400700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29410c5a0 0x7fe2941a17a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe28c007ee0 tx=0x7fe28c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:00.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.959+0000 7fe28b400700 1 -- 172.21.15.43:0/1185377673 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c00ddd0 con 0x7fe29410c5a0 2024-05-27T07:16:00.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.959+0000 7fe299600700 1 -- 172.21.15.43:0/1185377673 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2941a2130 con 0x7fe29410c5a0 2024-05-27T07:16:00.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.959+0000 7fe28b400700 1 -- 172.21.15.43:0/1185377673 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe28c004020 con 0x7fe29410c5a0 2024-05-27T07:16:00.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.959+0000 7fe28b400700 1 -- 172.21.15.43:0/1185377673 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c015660 con 0x7fe29410c5a0 2024-05-27T07:16:00.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.959+0000 7fe299600700 1 -- 172.21.15.43:0/1185377673 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2941a9a50 con 0x7fe29410c5a0 2024-05-27T07:16:00.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.963+0000 7fe28b400700 1 -- 172.21.15.43:0/1185377673 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe28c0041d0 con 0x7fe29410c5a0 2024-05-27T07:16:00.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.965+0000 7fe2a80b4b80 1 Processor -- start 2024-05-27T07:16:00.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.965+0000 7fe2a80b4b80 1 -- start start 2024-05-27T07:16:00.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.965+0000 7fe28b400700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2780b2a70 0x7fe2780b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.965+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6319f0 0x55f23d6c4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.965+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6c4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.965+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6c4e70 0x55f23d6d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.965+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f23d5f2e40 con 0x55f23d6c4e70 2024-05-27T07:16:00.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.965+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f23d5e29e0 con 0x55f23d545e40 2024-05-27T07:16:00.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.965+0000 7fe28b400700 1 -- 172.21.15.43:0/1185377673 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fe28c0e24f0 con 0x7fe29410c5a0 2024-05-27T07:16:00.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.965+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f23d5e6a20 con 0x55f23d6319f0 2024-05-27T07:16:00.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.966+0000 7fe28be00700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2780b2a70 0x7fe2780b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.966+0000 7fe293400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6c4e70 0x55f23d6d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.966+0000 7fe293400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6c4e70 0x55f23d6d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50748/0 (socket says 172.21.15.43:50748) 2024-05-27T07:16:00.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.966+0000 7fe293400700 1 -- 172.21.15.43:0/718897870 learned_addr learned my addr 172.21.15.43:0/718897870 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:00.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.966+0000 7fe292a00700 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6319f0 0x55f23d6c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.966+0000 7fe28be00700 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.967+0000 7fe28be00700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2780b2a70 0x7fe2780b4f80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2941056c0 tx=0x7fe27c004950 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.967+0000 7fe293400700 1 -- 172.21.15.43:0/718897870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6319f0 msgr2=0x55f23d6c4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.967+0000 7fe293400700 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6319f0 0x55f23d6c4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.967+0000 7fe293400700 1 -- 172.21.15.43:0/718897870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 msgr2=0x55f23d6c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.967+0000 7fe293400700 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6c4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.967+0000 7fe293400700 1 -- 172.21.15.43:0/718897870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23d5ab420 con 0x55f23d6c4e70 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.967+0000 7fe292a00700 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6319f0 0x55f23d6c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.967+0000 7fe28be00700 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.967+0000 7fe293400700 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6c4e70 0x55f23d6d1110 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe28c004a20 tx=0x7fe28c0a9b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34624cb371d8a68 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.968+0000 7fe283e00700 1 -- 172.21.15.43:0/718897870 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c0ad7d0 con 0x55f23d6c4e70 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.968+0000 7fe283e00700 1 -- 172.21.15.43:0/718897870 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe28c0b2280 con 0x55f23d6c4e70 2024-05-27T07:16:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.968+0000 7fe283e00700 1 -- 172.21.15.43:0/718897870 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c0a98e0 con 0x55f23d6c4e70 2024-05-27T07:16:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.969+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/718897870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6c4e70 msgr2=0x55f23d6d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.969+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6c4e70 0x55f23d6d1110 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe28c004a20 tx=0x7fe28c0a9b20 comp rx=0 tx=0).stop 2024-05-27T07:16:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.969+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/718897870 shutdown_connections 2024-05-27T07:16:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.969+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6c4e70 0x55f23d6d1110 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.969+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6319f0 0x55f23d6c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.969+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/718897870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.969+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/718897870 >> 172.21.15.43:0/718897870 conn(0x55f23d5df390 msgr2=0x55f23d6c04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:00.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.969+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/718897870 shutdown_connections 2024-05-27T07:16:00.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.970+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/718897870 wait complete. 2024-05-27T07:16:00.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.970+0000 7fe2a80b4b80 1 -- start start 2024-05-27T07:16:00.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.971+0000 7fe2a80b4b80 1 Processor -- start 2024-05-27T07:16:00.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.971+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d545e40 0x55f23d6ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.971+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.971+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d6c4e70 0x55f23d6cf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.971+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f23d4e8fa0 con 0x55f23d6319f0 2024-05-27T07:16:00.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.971+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f23d583760 con 0x55f23d6c4e70 2024-05-27T07:16:00.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.971+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f23d6d1bd0 con 0x55f23d545e40 2024-05-27T07:16:00.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.971+0000 7fe2a80b4b80 10 client.0 _ll_register_callbacks cb 0x55f23d58cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:00.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.972+0000 7fe28be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.972+0000 7fe28be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50752/0 (socket says 172.21.15.43:50752) 2024-05-27T07:16:00.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.972+0000 7fe28be00700 1 -- 172.21.15.43:0/4098787156 learned_addr learned my addr 172.21.15.43:0/4098787156 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:00.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.972+0000 7fe292a00700 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d545e40 0x55f23d6ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.972+0000 7fe293400700 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d6c4e70 0x55f23d6cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.972+0000 7fe28be00700 1 -- 172.21.15.43:0/4098787156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d545e40 msgr2=0x55f23d6ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:00.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.972+0000 7fe28be00700 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d545e40 0x55f23d6ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.972+0000 7fe28be00700 1 -- 172.21.15.43:0/4098787156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d6c4e70 msgr2=0x55f23d6cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:00.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.972+0000 7fe28be00700 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d6c4e70 0x55f23d6cf140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.972+0000 7fe28be00700 1 -- 172.21.15.43:0/4098787156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23d5328d0 con 0x55f23d6319f0 2024-05-27T07:16:00.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.973+0000 7fe28be00700 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6cec00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe294075b70 tx=0x7fe27c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a648c4fdcdd96188 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:00.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.973+0000 7fe280c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:00.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.973+0000 7fe280c00700 1 -- 172.21.15.43:0/4098787156 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c034060 con 0x55f23d6319f0 2024-05-27T07:16:00.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.973+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/4098787156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe28c0b2280 con 0x55f23d6319f0 2024-05-27T07:16:00.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.973+0000 7fe280c00700 1 -- 172.21.15.43:0/4098787156 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fe27c00f460 con 0x55f23d6319f0 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.973+0000 7fe280c00700 1 -- 172.21.15.43:0/4098787156 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe27c03b050 con 0x55f23d6319f0 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.973+0000 7fe2a80b4b80 10 client.15795 fetch_fsmap learned FSMap version 30 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.973+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/4098787156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f23d5b3060 con 0x55f23d6319f0 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.974+0000 7fe280c00700 1 -- 172.21.15.43:0/4098787156 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe28c0b2280 con 0x55f23d6319f0 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.974+0000 7fe280c00700 1 -- 172.21.15.43:0/4098787156 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fe27c02e030 con 0x55f23d6319f0 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.974+0000 7fe2a80b4b80 10 client.15795 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.974+0000 7fe2a80b4b80 10 client.15795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.974+0000 7fe2a80b4b80 20 client.15795 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.974+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f23d6dec30 0x55f23d6e1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.974+0000 7fe2a80b4b80 4 client.15795 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:16:00.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.974+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/4098787156 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f23d5f4d90 con 0x55f23d6dec30 2024-05-27T07:16:00.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.975+0000 7fe292a00700 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f23d6dec30 0x55f23d6e1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:00.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.976+0000 7fe292a00700 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f23d6dec30 0x55f23d6e1050 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e4411e7f4d83bf1 server_cookie=1565f7d25d4ab1f3 in_seq=0 out_seq=0 2024-05-27T07:16:00.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.976+0000 7fe280c00700 10 client.15795 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:16:00.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.979+0000 7fe280c00700 1 -- 172.21.15.43:0/4098787156 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f23d5f4d90 con 0x55f23d6dec30 2024-05-27T07:16:00.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.979+0000 7fe280c00700 10 client.15795 handle_command_reply: tid=0 2024-05-27T07:16:00.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.979+0000 7fe2a80b4b80 1 client.15795 shutdown 2024-05-27T07:16:00.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/4098787156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 msgr2=0x55f23d6cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:00.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6cec00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe294075b70 tx=0x7fe27c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/4098787156 shutdown_connections 2024-05-27T07:16:00.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d6c4e70 0x55f23d6cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d545e40 0x55f23d6ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f23d6dec30 0x55f23d6e1050 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/4098787156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6cec00 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:00.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/4098787156 >> 172.21.15.43:0/4098787156 conn(0x55f23d5df390 msgr2=0x55f23d6c43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:00.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe280c00700 0 client.15795 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:16:00.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/4098787156 shutdown_connections 2024-05-27T07:16:00.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/4098787156 wait complete. 2024-05-27T07:16:00.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:00.980+0000 7fe2a80b4b80 20 client.15795 trim_cache size 0 max 16384 2024-05-27T07:16:00.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.007+0000 7fe2a80b4b80 1 Processor -- start 2024-05-27T07:16:00.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.007+0000 7fe2a80b4b80 1 -- start start 2024-05-27T07:16:00.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.008+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.008+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.008+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6c4e70 0x55f23d6c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.008+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941b2640 con 0x55f23d6319f0 2024-05-27T07:16:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.008+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941b4180 con 0x55f23d545e40 2024-05-27T07:16:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.008+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941b2d10 con 0x55f23d6c4e70 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.008+0000 7fe28be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.008+0000 7fe28be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6c9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50758/0 (socket says 172.21.15.43:50758) 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.008+0000 7fe28be00700 1 -- 172.21.15.43:0/3378853666 learned_addr learned my addr 172.21.15.43:0/3378853666 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.009+0000 7fe292a00700 1 --2- 172.21.15.43:0/3378853666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.009+0000 7fe293400700 1 --2- 172.21.15.43:0/3378853666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6c4e70 0x55f23d6c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.009+0000 7fe28be00700 1 -- 172.21.15.43:0/3378853666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6c4e70 msgr2=0x55f23d6c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.009+0000 7fe28be00700 1 --2- 172.21.15.43:0/3378853666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6c4e70 0x55f23d6c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.009+0000 7fe28be00700 1 -- 172.21.15.43:0/3378853666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 msgr2=0x55f23d6cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.009+0000 7fe28be00700 1 --2- 172.21.15.43:0/3378853666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.009+0000 7fe28be00700 1 -- 172.21.15.43:0/3378853666 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23d6c5800 con 0x55f23d6319f0 2024-05-27T07:16:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.009+0000 7fe28be00700 1 --2- 172.21.15.43:0/3378853666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6c9e90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009350 tx=0x7fe27c003720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0f02def4d2544a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.010+0000 7fe283400700 1 -- 172.21.15.43:0/3378853666 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c034070 con 0x55f23d6319f0 2024-05-27T07:16:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.010+0000 7fe283400700 1 -- 172.21.15.43:0/3378853666 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe27c022040 con 0x55f23d6319f0 2024-05-27T07:16:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.010+0000 7fe283400700 1 -- 172.21.15.43:0/3378853666 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c028bc0 con 0x55f23d6319f0 2024-05-27T07:16:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.011+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3378853666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 msgr2=0x55f23d6c9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:01.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.011+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/3378853666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6c9e90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009350 tx=0x7fe27c003720 comp rx=0 tx=0).stop 2024-05-27T07:16:01.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.011+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3378853666 shutdown_connections 2024-05-27T07:16:01.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.011+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/3378853666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6c4e70 0x55f23d6c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.011+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/3378853666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.011+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/3378853666 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6c9e90 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.011+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3378853666 >> 172.21.15.43:0/3378853666 conn(0x55f23d5df390 msgr2=0x55f23d6c5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.011+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3378853666 shutdown_connections 2024-05-27T07:16:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.011+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3378853666 wait complete. 2024-05-27T07:16:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.012+0000 7fe2a80b4b80 1 -- start start 2024-05-27T07:16:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.012+0000 7fe2a80b4b80 1 Processor -- start 2024-05-27T07:16:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.013+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6d6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.013+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6d7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.013+0000 7fe2a80b4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6c4e70 0x55f23d6d1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.013+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941b3ab0 con 0x55f23d6319f0 2024-05-27T07:16:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.013+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941b33e0 con 0x55f23d545e40 2024-05-27T07:16:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.013+0000 7fe2a80b4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941b05c0 con 0x55f23d6c4e70 2024-05-27T07:16:01.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.013+0000 7fe2a80b4b80 10 client.0 _ll_register_callbacks cb 0x55f23d58cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.013+0000 7fe28be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6d7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.013+0000 7fe28be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6d7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50768/0 (socket says 172.21.15.43:50768) 2024-05-27T07:16:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.013+0000 7fe28be00700 1 -- 172.21.15.43:0/3716273130 learned_addr learned my addr 172.21.15.43:0/3716273130 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.014+0000 7fe293400700 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6c4e70 0x55f23d6d1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.014+0000 7fe292a00700 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6d6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.014+0000 7fe28be00700 1 -- 172.21.15.43:0/3716273130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6c4e70 msgr2=0x55f23d6d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.014+0000 7fe28be00700 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6c4e70 0x55f23d6d1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.014+0000 7fe28be00700 1 -- 172.21.15.43:0/3716273130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 msgr2=0x55f23d6d6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.014+0000 7fe28be00700 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6d6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.014+0000 7fe28be00700 1 -- 172.21.15.43:0/3716273130 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23d6ca130 con 0x55f23d6319f0 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.014+0000 7fe292a00700 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6d6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.014+0000 7fe28be00700 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6d7400 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0284b0 tx=0x7fe27c0284e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3527e505c7baff8f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.014+0000 7fe280c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.015+0000 7fe280c00700 1 -- 172.21.15.43:0/3716273130 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c034040 con 0x55f23d6319f0 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.015+0000 7fe280c00700 1 -- 172.21.15.43:0/3716273130 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9408+0+0 (secure 0 0 0) 0x7fe27c004230 con 0x55f23d6319f0 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.015+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3716273130 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe27c022040 con 0x55f23d6319f0 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe280c00700 1 -- 172.21.15.43:0/3716273130 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe27c04e050 con 0x55f23d6319f0 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe2a80b4b80 10 client.15801 fetch_fsmap learned FSMap version 30 2024-05-27T07:16:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3716273130 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f23d749b70 con 0x55f23d6319f0 2024-05-27T07:16:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe280c00700 1 -- 172.21.15.43:0/3716273130 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe27c022040 con 0x55f23d6319f0 2024-05-27T07:16:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe280c00700 1 -- 172.21.15.43:0/3716273130 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fe27c02e070 con 0x55f23d6319f0 2024-05-27T07:16:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe2a80b4b80 10 client.15801 fetch_fsmap finished waiting for FSMap version 30 2024-05-27T07:16:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe2a80b4b80 10 client.15801 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:16:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe2a80b4b80 20 client.15801 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f23d74a170 0x55f23d74c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe2a80b4b80 4 client.15801 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:16:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.016+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3716273130 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f23d5ee570 con 0x55f23d74a170 2024-05-27T07:16:01.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.017+0000 7fe292a00700 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f23d74a170 0x55f23d74c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:01.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.019+0000 7fe292a00700 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f23d74a170 0x55f23d74c590 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6dbee9746acf4fe server_cookie=bdf104613cebf8b5 in_seq=0 out_seq=0 2024-05-27T07:16:01.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.019+0000 7fe280c00700 10 client.15801 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:16:01.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe280c00700 1 -- 172.21.15.43:0/3716273130 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f23d5ee570 con 0x55f23d74a170 2024-05-27T07:16:01.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe280c00700 10 client.15801 handle_command_reply: tid=0 2024-05-27T07:16:01.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe2a80b4b80 1 client.15801 shutdown 2024-05-27T07:16:01.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3716273130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 msgr2=0x55f23d6d7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:01.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6d7400 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0284b0 tx=0x7fe27c0284e0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3716273130 shutdown_connections 2024-05-27T07:16:01.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f23d6c4e70 0x55f23d6d1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f23d545e40 0x55f23d6d6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f23d74a170 0x55f23d74c590 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe2a80b4b80 1 --2- 172.21.15.43:0/3716273130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f23d6319f0 0x55f23d6d7400 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3716273130 >> 172.21.15.43:0/3716273130 conn(0x55f23d5df390 msgr2=0x55f23d6ce4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe280c00700 0 client.15801 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:16:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.020+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3716273130 shutdown_connections 2024-05-27T07:16:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.021+0000 7fe2a80b4b80 1 -- 172.21.15.43:0/3716273130 wait complete. 2024-05-27T07:16:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.021+0000 7fe2a80b4b80 20 client.15801 trim_cache size 0 max 16384 2024-05-27T07:16:01.012 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:16:01.013 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:16:01.013 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:16:01.013 INFO:teuthology.orchestra.run.smithi043.stdout: "98944aad-55cd-4eda-9108-0ef2ca9b5c45": { 2024-05-27T07:16:01.013 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:16:01.013 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "98944aad-55cd-4eda-9108-0ef2ca9b5c45", 2024-05-27T07:16:01.013 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:16:01.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:16:01.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:16:01.014 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:16:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 -- 172.21.15.43:0/1185377673 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2780b2a70 msgr2=0x7fe2780b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2780b2a70 0x7fe2780b4f80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2941056c0 tx=0x7fe27c004950 comp rx=0 tx=0).stop 2024-05-27T07:16:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 -- 172.21.15.43:0/1185377673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29410c5a0 msgr2=0x7fe2941a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29410c5a0 0x7fe2941a17a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe28c007ee0 tx=0x7fe28c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 -- 172.21.15.43:0/1185377673 shutdown_connections 2024-05-27T07:16:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2780b2a70 0x7fe2780b4f80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29410c5a0 0x7fe2941a17a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe294106560 0x7fe29419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 --2- 172.21.15.43:0/1185377673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe294106eb0 0x7fe29419d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:01.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 -- 172.21.15.43:0/1185377673 >> 172.21.15.43:0/1185377673 conn(0x7fe294102040 msgr2=0x7fe294103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:01.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 -- 172.21.15.43:0/1185377673 shutdown_connections 2024-05-27T07:16:01.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:01.022+0000 7fe299600700 1 -- 172.21.15.43:0/1185377673 wait complete. 2024-05-27T07:16:01.173 INFO:tasks.workunit.client.0.smithi043.stdout: 1 1033 4.58 MB/sec warmup 5 sec latency 102.980 ms 2024-05-27T07:16:01.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd0ca000700 time 2024-05-27T07:16:01.245530+0000 2024-05-27T07:16:01.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:01.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:01.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd0dfbd5ba6] 2024-05-27T07:16:01.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd0dfbd5dd1] 2024-05-27T07:16:01.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562aac5e123d] 2024-05-27T07:16:01.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562aac5cc971] 2024-05-27T07:16:01.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 5: (SnapRealm::check_cache() const+0x13e) [0x562aac5cafa0] 2024-05-27T07:16:01.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562aac5cb651] 2024-05-27T07:16:01.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562aac560347] 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562aac25c87e] 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562aac261e94] 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562aac26c969] 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 11: (MDLog::_replay_thread()+0x1f38) [0x562aac60548c] 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562aac60e9cb] 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 13: (Thread::entry_wrapper()+0x43) [0x7fd0dfba472d] 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd0dfba4749] 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd0de7481ca] 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 16: clone() 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: *** Caught signal (Aborted) ** 2024-05-27T07:16:01.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: in thread 7fd0ca000700 thread_name:md_log_replay 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.249+0000 7fd0ca000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd0ca000700 time 2024-05-27T07:16:01.245530+0000 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd0dfbd5ba6] 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd0dfbd5dd1] 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562aac5e123d] 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562aac5cc971] 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 5: (SnapRealm::check_cache() const+0x13e) [0x562aac5cafa0] 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562aac5cb651] 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562aac560347] 2024-05-27T07:16:01.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562aac25c87e] 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562aac261e94] 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562aac26c969] 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 11: (MDLog::_replay_thread()+0x1f38) [0x562aac60548c] 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562aac60e9cb] 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 13: (Thread::entry_wrapper()+0x43) [0x7fd0dfba472d] 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd0dfba4749] 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd0de7481ca] 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 16: clone() 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd0de752d20] 2024-05-27T07:16:01.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2: gsignal() 2024-05-27T07:16:01.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 3: abort() 2024-05-27T07:16:01.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd0dfbd5ced] 2024-05-27T07:16:01.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd0dfbd5dd1] 2024-05-27T07:16:01.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562aac5e123d] 2024-05-27T07:16:01.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562aac5cc971] 2024-05-27T07:16:01.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 8: (SnapRealm::check_cache() const+0x13e) [0x562aac5cafa0] 2024-05-27T07:16:01.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562aac5cb651] 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562aac560347] 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562aac25c87e] 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562aac261e94] 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562aac26c969] 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 14: (MDLog::_replay_thread()+0x1f38) [0x562aac60548c] 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562aac60e9cb] 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 16: (Thread::entry_wrapper()+0x43) [0x7fd0dfba472d] 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd0dfba4749] 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd0de7481ca] 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 19: clone() 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.253+0000 7fd0ca000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: in thread 7fd0ca000700 thread_name:md_log_replay 2024-05-27T07:16:01.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd0de752d20] 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2: gsignal() 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 3: abort() 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd0dfbd5ced] 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd0dfbd5dd1] 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562aac5e123d] 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562aac5cc971] 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 8: (SnapRealm::check_cache() const+0x13e) [0x562aac5cafa0] 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562aac5cb651] 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562aac560347] 2024-05-27T07:16:01.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562aac25c87e] 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562aac261e94] 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562aac26c969] 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 14: (MDLog::_replay_thread()+0x1f38) [0x562aac60548c] 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562aac60e9cb] 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 16: (Thread::entry_wrapper()+0x43) [0x7fd0dfba472d] 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd0dfba4749] 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd0de7481ca] 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 19: clone() 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: -1> 2024-05-27T07:16:01.249+0000 7fd0ca000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd0ca000700 time 2024-05-27T07:16:01.245530+0000 2024-05-27T07:16:01.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:01.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:01.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd0dfbd5ba6] 2024-05-27T07:16:01.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd0dfbd5dd1] 2024-05-27T07:16:01.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562aac5e123d] 2024-05-27T07:16:01.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562aac5cc971] 2024-05-27T07:16:01.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 5: (SnapRealm::check_cache() const+0x13e) [0x562aac5cafa0] 2024-05-27T07:16:01.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562aac5cb651] 2024-05-27T07:16:01.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562aac560347] 2024-05-27T07:16:01.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562aac25c87e] 2024-05-27T07:16:01.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562aac261e94] 2024-05-27T07:16:01.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562aac26c969] 2024-05-27T07:16:01.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 11: (MDLog::_replay_thread()+0x1f38) [0x562aac60548c] 2024-05-27T07:16:01.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562aac60e9cb] 2024-05-27T07:16:01.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 13: (Thread::entry_wrapper()+0x43) [0x7fd0dfba472d] 2024-05-27T07:16:01.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd0dfba4749] 2024-05-27T07:16:01.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd0de7481ca] 2024-05-27T07:16:01.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 16: clone() 2024-05-27T07:16:01.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 0> 2024-05-27T07:16:01.253+0000 7fd0ca000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: in thread 7fd0ca000700 thread_name:md_log_replay 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd0de752d20] 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2: gsignal() 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 3: abort() 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd0dfbd5ced] 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd0dfbd5dd1] 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562aac5e123d] 2024-05-27T07:16:01.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562aac5cc971] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 8: (SnapRealm::check_cache() const+0x13e) [0x562aac5cafa0] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562aac5cb651] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562aac560347] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562aac25c87e] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562aac261e94] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562aac26c969] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 14: (MDLog::_replay_thread()+0x1f38) [0x562aac60548c] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562aac60e9cb] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 16: (Thread::entry_wrapper()+0x43) [0x7fd0dfba472d] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd0dfba4749] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd0de7481ca] 2024-05-27T07:16:01.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 19: clone() 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: -1> 2024-05-27T07:16:01.249+0000 7fd0ca000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd0ca000700 time 2024-05-27T07:16:01.245530+0000 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd0dfbd5ba6] 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd0dfbd5dd1] 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562aac5e123d] 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562aac5cc971] 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 5: (SnapRealm::check_cache() const+0x13e) [0x562aac5cafa0] 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562aac5cb651] 2024-05-27T07:16:01.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562aac560347] 2024-05-27T07:16:01.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562aac25c87e] 2024-05-27T07:16:01.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562aac261e94] 2024-05-27T07:16:01.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562aac26c969] 2024-05-27T07:16:01.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 11: (MDLog::_replay_thread()+0x1f38) [0x562aac60548c] 2024-05-27T07:16:01.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562aac60e9cb] 2024-05-27T07:16:01.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 13: (Thread::entry_wrapper()+0x43) [0x7fd0dfba472d] 2024-05-27T07:16:01.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd0dfba4749] 2024-05-27T07:16:01.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd0de7481ca] 2024-05-27T07:16:01.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 16: clone() 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 0> 2024-05-27T07:16:01.253+0000 7fd0ca000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: in thread 7fd0ca000700 thread_name:md_log_replay 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd0de752d20] 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2: gsignal() 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 3: abort() 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd0dfbd5ced] 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd0dfbd5dd1] 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562aac5e123d] 2024-05-27T07:16:01.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562aac5cc971] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 8: (SnapRealm::check_cache() const+0x13e) [0x562aac5cafa0] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562aac5cb651] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562aac560347] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562aac25c87e] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562aac261e94] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562aac26c969] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 14: (MDLog::_replay_thread()+0x1f38) [0x562aac60548c] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562aac60e9cb] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 16: (Thread::entry_wrapper()+0x43) [0x7fd0dfba472d] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd0dfba4749] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd0de7481ca] 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 19: clone() 2024-05-27T07:16:01.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:01.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[66138]: 2024-05-27T07:16:01.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:01 smithi179 podman[70637]: 2024-05-27 07:16:01.412487923 +0000 UTC m=+0.041742454 container died f3c9408632d818fef921c5d9ffdea7d82874431699e8705a9d69f1339ec1dcb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:16:01.986 INFO:tasks.cephfs.filesystem:scrub status for tag:98944aad-55cd-4eda-9108-0ef2ca9b5c45 - {'path': '/', 'tag': '98944aad-55cd-4eda-9108-0ef2ca9b5c45', 'options': 'recursive,force'} 2024-05-27T07:16:02.173 INFO:tasks.workunit.client.0.smithi043.stdout: 1 1157 3.83 MB/sec warmup 6 sec latency 55.164 ms 2024-05-27T07:16:02.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:02 smithi043 ceph-mon[31531]: pgmap v66: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 1.8 MiB/s wr, 42 op/s 2024-05-27T07:16:02.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:02 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:16:02.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:02 smithi132 ceph-mon[35283]: pgmap v66: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 1.8 MiB/s wr, 42 op/s 2024-05-27T07:16:02.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:02 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:16:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:02 smithi179 ceph-mon[35888]: pgmap v66: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 1.8 MiB/s wr, 42 op/s 2024-05-27T07:16:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:02 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:16:02.459 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:02 smithi179 podman[70637]: 2024-05-27 07:16:02.283313837 +0000 UTC m=+0.912568387 container remove f3c9408632d818fef921c5d9ffdea7d82874431699e8705a9d69f1339ec1dcb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:16:02.459 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:02 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:16:03.174 INFO:tasks.workunit.client.0.smithi043.stdout: 1 1280 3.31 MB/sec warmup 7 sec latency 51.817 ms 2024-05-27T07:16:03.209 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:02 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:16:03.209 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:02 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.327s CPU time. 2024-05-27T07:16:04.174 INFO:tasks.workunit.client.0.smithi043.stdout: 1 1461 3.25 MB/sec warmup 8 sec latency 80.150 ms 2024-05-27T07:16:04.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:04 smithi043 ceph-mon[31531]: pgmap v67: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.8 MiB/s wr, 38 op/s 2024-05-27T07:16:04.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:04 smithi179 ceph-mon[35888]: pgmap v67: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.8 MiB/s wr, 38 op/s 2024-05-27T07:16:04.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:04 smithi132 ceph-mon[35283]: pgmap v67: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.8 MiB/s wr, 38 op/s 2024-05-27T07:16:05.174 INFO:tasks.workunit.client.0.smithi043.stdout: 1 1585 2.91 MB/sec warmup 9 sec latency 51.648 ms 2024-05-27T07:16:06.174 INFO:tasks.workunit.client.0.smithi043.stdout: 1 1707 2.62 MB/sec warmup 10 sec latency 54.583 ms 2024-05-27T07:16:06.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:06 smithi043 ceph-mon[31531]: pgmap v68: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.8 MiB/s wr, 38 op/s 2024-05-27T07:16:06.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:06 smithi179 ceph-mon[35888]: pgmap v68: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.8 MiB/s wr, 38 op/s 2024-05-27T07:16:06.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:06 smithi132 ceph-mon[35283]: pgmap v68: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.8 MiB/s wr, 38 op/s 2024-05-27T07:16:07.174 INFO:tasks.workunit.client.0.smithi043.stdout: 1 1858 2.40 MB/sec warmup 11 sec latency 58.961 ms 2024-05-27T07:16:08.174 INFO:tasks.workunit.client.0.smithi043.stdout: 1 1997 2.21 MB/sec warmup 12 sec latency 57.034 ms 2024-05-27T07:16:08.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:08 smithi043 ceph-mon[31531]: pgmap v69: 129 pgs: 129 active+clean; 23 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.9 MiB/s wr, 53 op/s 2024-05-27T07:16:08.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:08 smithi179 ceph-mon[35888]: pgmap v69: 129 pgs: 129 active+clean; 23 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.9 MiB/s wr, 53 op/s 2024-05-27T07:16:08.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:08 smithi132 ceph-mon[35283]: pgmap v69: 129 pgs: 129 active+clean; 23 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.9 MiB/s wr, 53 op/s 2024-05-27T07:16:09.174 INFO:tasks.workunit.client.0.smithi043.stdout: 1 2126 2.05 MB/sec warmup 13 sec latency 58.540 ms 2024-05-27T07:16:10.175 INFO:tasks.workunit.client.0.smithi043.stdout: 1 2296 1.98 MB/sec warmup 14 sec latency 56.025 ms 2024-05-27T07:16:10.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:10 smithi043 ceph-mon[31531]: pgmap v70: 129 pgs: 129 active+clean; 23 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.9 MiB/s wr, 48 op/s 2024-05-27T07:16:10.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:10 smithi179 ceph-mon[35888]: pgmap v70: 129 pgs: 129 active+clean; 23 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.9 MiB/s wr, 48 op/s 2024-05-27T07:16:10.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:10 smithi132 ceph-mon[35283]: pgmap v70: 129 pgs: 129 active+clean; 23 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.9 MiB/s wr, 48 op/s 2024-05-27T07:16:11.175 INFO:tasks.workunit.client.0.smithi043.stdout: 1 2388 1.86 MB/sec warmup 15 sec latency 56.443 ms 2024-05-27T07:16:12.175 INFO:tasks.workunit.client.0.smithi043.stdout: 1 2492 1.80 MB/sec warmup 16 sec latency 52.357 ms 2024-05-27T07:16:12.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:12 smithi043 ceph-mon[31531]: pgmap v71: 129 pgs: 129 active+clean; 24 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.9 MiB/s wr, 61 op/s 2024-05-27T07:16:12.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:12 smithi179 ceph-mon[35888]: pgmap v71: 129 pgs: 129 active+clean; 24 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.9 MiB/s wr, 61 op/s 2024-05-27T07:16:12.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:12 smithi132 ceph-mon[35283]: pgmap v71: 129 pgs: 129 active+clean; 24 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 1.9 MiB/s wr, 61 op/s 2024-05-27T07:16:13.175 INFO:tasks.workunit.client.0.smithi043.stdout: 1 2581 1.70 MB/sec warmup 17 sec latency 249.143 ms 2024-05-27T07:16:13.371 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:13 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:13.371 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:12 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:16:13.371 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:12 smithi179 systemd[1]: Stopped Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:13.372 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:12 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 3.327s CPU time. 2024-05-27T07:16:13.372 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:12 smithi179 systemd[1]: Starting Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:16:13.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:13 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:13.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:13 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:13.709 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:13 smithi179 podman[70730]: 2024-05-27 07:16:13.286642901 +0000 UTC m=+0.034736946 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:13.709 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:13 smithi179 podman[70730]: 2024-05-27 07:16:13.459428717 +0000 UTC m=+0.207522789 container create 7b07c41873431a22938686740934ba7d5ce87bfb5ee5df98142edbae071af368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T07:16:14.166 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:13 smithi179 podman[70730]: 2024-05-27 07:16:13.730342975 +0000 UTC m=+0.478437147 container init 7b07c41873431a22938686740934ba7d5ce87bfb5ee5df98142edbae071af368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:16:14.166 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:13 smithi179 podman[70730]: 2024-05-27 07:16:13.742285381 +0000 UTC m=+0.490379483 container start 7b07c41873431a22938686740934ba7d5ce87bfb5ee5df98142edbae071af368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-27T07:16:14.166 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:13 smithi179 ceph-mds[70751]: -- 172.21.15.179:0/148036499 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ca646da40 con 0x564ca64b5a80 2024-05-27T07:16:14.166 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: starting mds.c at 2024-05-27T07:16:14.166 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:13 smithi179 bash[70730]: 7b07c41873431a22938686740934ba7d5ce87bfb5ee5df98142edbae071af368 2024-05-27T07:16:14.166 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:13 smithi179 systemd[1]: Started Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:14.175 INFO:tasks.workunit.client.0.smithi043.stdout: 1 2760 1.78 MB/sec warmup 18 sec latency 53.187 ms 2024-05-27T07:16:14.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:14 smithi043 ceph-mon[31531]: pgmap v72: 129 pgs: 129 active+clean; 24 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 180 KiB/s wr, 26 op/s 2024-05-27T07:16:14.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:14 smithi043 ceph-mon[31531]: daemon mds.c restarted 2024-05-27T07:16:14.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-mon[35888]: pgmap v72: 129 pgs: 129 active+clean; 24 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 180 KiB/s wr, 26 op/s 2024-05-27T07:16:14.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-mon[35888]: daemon mds.c restarted 2024-05-27T07:16:14.460 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1195e00700 time 2024-05-27T07:16:14.288166+0000 2024-05-27T07:16:14.460 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:14.460 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:14.460 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f11ab85dba6] 2024-05-27T07:16:14.460 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f11ab85ddd1] 2024-05-27T07:16:14.460 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564c99d2423d] 2024-05-27T07:16:14.460 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564c99d0f971] 2024-05-27T07:16:14.460 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 5: (SnapRealm::check_cache() const+0x13e) [0x564c99d0dfa0] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564c99d0e651] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564c99ca3347] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564c9999f87e] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564c999a4e94] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564c999af969] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 11: (MDLog::_replay_thread()+0x1f38) [0x564c99d4848c] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564c99d519cb] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 13: (Thread::entry_wrapper()+0x43) [0x7f11ab82c72d] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 14: (Thread::_entry_func(void*)+0xd) [0x7f11ab82c749] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f11aa3d01ca] 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 16: clone() 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: *** Caught signal (Aborted) ** 2024-05-27T07:16:14.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: in thread 7f1195e00700 thread_name:md_log_replay 2024-05-27T07:16:14.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:14.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f11aa3dad20] 2024-05-27T07:16:14.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2: gsignal() 2024-05-27T07:16:14.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 3: abort() 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f11ab85dced] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f11ab85ddd1] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564c99d2423d] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564c99d0f971] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 8: (SnapRealm::check_cache() const+0x13e) [0x564c99d0dfa0] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564c99d0e651] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564c99ca3347] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564c9999f87e] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564c999a4e94] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564c999af969] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 14: (MDLog::_replay_thread()+0x1f38) [0x564c99d4848c] 2024-05-27T07:16:14.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564c99d519cb] 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 16: (Thread::entry_wrapper()+0x43) [0x7f11ab82c72d] 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 17: (Thread::_entry_func(void*)+0xd) [0x7f11ab82c749] 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f11aa3d01ca] 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 19: clone() 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.290+0000 7f1195e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1195e00700 time 2024-05-27T07:16:14.288166+0000 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f11ab85dba6] 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f11ab85ddd1] 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564c99d2423d] 2024-05-27T07:16:14.464 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564c99d0f971] 2024-05-27T07:16:14.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 5: (SnapRealm::check_cache() const+0x13e) [0x564c99d0dfa0] 2024-05-27T07:16:14.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564c99d0e651] 2024-05-27T07:16:14.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564c99ca3347] 2024-05-27T07:16:14.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564c9999f87e] 2024-05-27T07:16:14.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564c999a4e94] 2024-05-27T07:16:14.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564c999af969] 2024-05-27T07:16:14.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 11: (MDLog::_replay_thread()+0x1f38) [0x564c99d4848c] 2024-05-27T07:16:14.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564c99d519cb] 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 13: (Thread::entry_wrapper()+0x43) [0x7f11ab82c72d] 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 14: (Thread::_entry_func(void*)+0xd) [0x7f11ab82c749] 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f11aa3d01ca] 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 16: clone() 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.293+0000 7f1195e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: in thread 7f1195e00700 thread_name:md_log_replay 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f11aa3dad20] 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2: gsignal() 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 3: abort() 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f11ab85dced] 2024-05-27T07:16:14.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f11ab85ddd1] 2024-05-27T07:16:14.467 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564c99d2423d] 2024-05-27T07:16:14.467 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564c99d0f971] 2024-05-27T07:16:14.467 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 8: (SnapRealm::check_cache() const+0x13e) [0x564c99d0dfa0] 2024-05-27T07:16:14.467 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564c99d0e651] 2024-05-27T07:16:14.467 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564c99ca3347] 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564c9999f87e] 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564c999a4e94] 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564c999af969] 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 14: (MDLog::_replay_thread()+0x1f38) [0x564c99d4848c] 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564c99d519cb] 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 16: (Thread::entry_wrapper()+0x43) [0x7f11ab82c72d] 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 17: (Thread::_entry_func(void*)+0xd) [0x7f11ab82c749] 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f11aa3d01ca] 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 19: clone() 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:14.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: -1> 2024-05-27T07:16:14.290+0000 7f1195e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1195e00700 time 2024-05-27T07:16:14.288166+0000 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f11ab85dba6] 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f11ab85ddd1] 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564c99d2423d] 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564c99d0f971] 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 5: (SnapRealm::check_cache() const+0x13e) [0x564c99d0dfa0] 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564c99d0e651] 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564c99ca3347] 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564c9999f87e] 2024-05-27T07:16:14.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564c999a4e94] 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564c999af969] 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 11: (MDLog::_replay_thread()+0x1f38) [0x564c99d4848c] 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564c99d519cb] 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 13: (Thread::entry_wrapper()+0x43) [0x7f11ab82c72d] 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 14: (Thread::_entry_func(void*)+0xd) [0x7f11ab82c749] 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f11aa3d01ca] 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 16: clone() 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 0> 2024-05-27T07:16:14.293+0000 7f1195e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: in thread 7f1195e00700 thread_name:md_log_replay 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:14.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f11aa3dad20] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2: gsignal() 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 3: abort() 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f11ab85dced] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f11ab85ddd1] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564c99d2423d] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564c99d0f971] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 8: (SnapRealm::check_cache() const+0x13e) [0x564c99d0dfa0] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564c99d0e651] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564c99ca3347] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564c9999f87e] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564c999a4e94] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564c999af969] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 14: (MDLog::_replay_thread()+0x1f38) [0x564c99d4848c] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564c99d519cb] 2024-05-27T07:16:14.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 16: (Thread::entry_wrapper()+0x43) [0x7f11ab82c72d] 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 17: (Thread::_entry_func(void*)+0xd) [0x7f11ab82c749] 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f11aa3d01ca] 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 19: clone() 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: -1> 2024-05-27T07:16:14.290+0000 7f1195e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1195e00700 time 2024-05-27T07:16:14.288166+0000 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f11ab85dba6] 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f11ab85ddd1] 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564c99d2423d] 2024-05-27T07:16:14.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564c99d0f971] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 5: (SnapRealm::check_cache() const+0x13e) [0x564c99d0dfa0] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564c99d0e651] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564c99ca3347] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564c9999f87e] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564c999a4e94] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564c999af969] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 11: (MDLog::_replay_thread()+0x1f38) [0x564c99d4848c] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564c99d519cb] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 13: (Thread::entry_wrapper()+0x43) [0x7f11ab82c72d] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 14: (Thread::_entry_func(void*)+0xd) [0x7f11ab82c749] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f11aa3d01ca] 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 16: clone() 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 0> 2024-05-27T07:16:14.293+0000 7f1195e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: in thread 7f1195e00700 thread_name:md_log_replay 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f11aa3dad20] 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2: gsignal() 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 3: abort() 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f11ab85dced] 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f11ab85ddd1] 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564c99d2423d] 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564c99d0f971] 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 8: (SnapRealm::check_cache() const+0x13e) [0x564c99d0dfa0] 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564c99d0e651] 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564c99ca3347] 2024-05-27T07:16:14.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564c9999f87e] 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564c999a4e94] 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564c999af969] 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 14: (MDLog::_replay_thread()+0x1f38) [0x564c99d4848c] 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564c99d519cb] 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 16: (Thread::entry_wrapper()+0x43) [0x7f11ab82c72d] 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 17: (Thread::_entry_func(void*)+0xd) [0x7f11ab82c749] 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f11aa3d01ca] 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 19: clone() 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[70747]: 2024-05-27T07:16:14.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:14 smithi179 podman[70783]: 2024-05-27 07:16:14.414805292 +0000 UTC m=+0.035892594 container died 7b07c41873431a22938686740934ba7d5ce87bfb5ee5df98142edbae071af368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:16:14.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:14 smithi132 ceph-mon[35283]: pgmap v72: 129 pgs: 129 active+clean; 24 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 180 KiB/s wr, 26 op/s 2024-05-27T07:16:14.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:14 smithi132 ceph-mon[35283]: daemon mds.c restarted 2024-05-27T07:16:15.175 INFO:tasks.workunit.client.0.smithi043.stdout: 1 2971 1.76 MB/sec warmup 19 sec latency 56.955 ms 2024-05-27T07:16:15.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:15 smithi043 ceph-mon[31531]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:16:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:15 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6836/1504635552,v1:172.21.15.179:6837/1504635552] up:boot 2024-05-27T07:16:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:15 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:15 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:15 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:15 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:15.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:15 smithi179 ceph-mon[35888]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:16:15.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:15 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6836/1504635552,v1:172.21.15.179:6837/1504635552] up:boot 2024-05-27T07:16:15.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:15 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:15.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:15 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:15.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:15 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:15.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:15 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:15.459 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:15 smithi179 podman[70783]: 2024-05-27 07:16:15.177227231 +0000 UTC m=+0.798314593 container remove 7b07c41873431a22938686740934ba7d5ce87bfb5ee5df98142edbae071af368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:16:15.459 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:15 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:16:15.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:15 smithi132 ceph-mon[35283]: osdmap e92: 12 total, 12 up, 12 in 2024-05-27T07:16:15.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:15 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6836/1504635552,v1:172.21.15.179:6837/1504635552] up:boot 2024-05-27T07:16:15.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:15 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:15.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:15 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:15.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:15 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:15.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:15 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:16:15.959 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:15 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:16:15.959 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:15 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.471s CPU time. 2024-05-27T07:16:16.175 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3205 1.71 MB/sec warmup 20 sec latency 47.597 ms 2024-05-27T07:16:16.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:16 smithi043 ceph-mon[31531]: pgmap v74: 129 pgs: 129 active+clean; 24 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 216 KiB/s wr, 31 op/s 2024-05-27T07:16:16.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:16 smithi179 ceph-mon[35888]: pgmap v74: 129 pgs: 129 active+clean; 24 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 216 KiB/s wr, 31 op/s 2024-05-27T07:16:16.493 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:16 smithi132 ceph-mon[35283]: pgmap v74: 129 pgs: 129 active+clean; 24 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 216 KiB/s wr, 31 op/s 2024-05-27T07:16:17.175 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3299 1.68 MB/sec warmup 21 sec latency 51.758 ms 2024-05-27T07:16:17.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:17.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:17.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:16:17.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:17.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:17.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:16:17.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:16:17.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:16:17.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:16:18.176 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3325 1.61 MB/sec warmup 22 sec latency 56.652 ms 2024-05-27T07:16:18.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:18 smithi132 ceph-mon[35283]: pgmap v75: 129 pgs: 129 active+clean; 26 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 437 KiB/s wr, 37 op/s 2024-05-27T07:16:18.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:18 smithi043 ceph-mon[31531]: pgmap v75: 129 pgs: 129 active+clean; 26 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 437 KiB/s wr, 37 op/s 2024-05-27T07:16:18.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:18 smithi179 ceph-mon[35888]: pgmap v75: 129 pgs: 129 active+clean; 26 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 437 KiB/s wr, 37 op/s 2024-05-27T07:16:19.176 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3378 1.54 MB/sec warmup 23 sec latency 57.810 ms 2024-05-27T07:16:20.176 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3558 1.60 MB/sec warmup 24 sec latency 59.738 ms 2024-05-27T07:16:20.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:20 smithi132 ceph-mon[35283]: pgmap v76: 129 pgs: 129 active+clean; 26 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 437 KiB/s wr, 37 op/s 2024-05-27T07:16:20.706 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:20 smithi043 ceph-mon[31531]: pgmap v76: 129 pgs: 129 active+clean; 26 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 437 KiB/s wr, 37 op/s 2024-05-27T07:16:20.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:20 smithi179 ceph-mon[35888]: pgmap v76: 129 pgs: 129 active+clean; 26 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 437 KiB/s wr, 37 op/s 2024-05-27T07:16:21.176 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3676 1.55 MB/sec warmup 25 sec latency 55.680 ms 2024-05-27T07:16:22.176 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3785 1.53 MB/sec warmup 26 sec latency 54.672 ms 2024-05-27T07:16:22.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:22 smithi132 ceph-mon[35283]: pgmap v77: 129 pgs: 129 active+clean; 27 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 555 KiB/s wr, 40 op/s 2024-05-27T07:16:22.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:22 smithi043 ceph-mon[31531]: pgmap v77: 129 pgs: 129 active+clean; 27 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 555 KiB/s wr, 40 op/s 2024-05-27T07:16:22.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:22 smithi179 ceph-mon[35888]: pgmap v77: 129 pgs: 129 active+clean; 27 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 555 KiB/s wr, 40 op/s 2024-05-27T07:16:23.176 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3840 1.47 MB/sec warmup 27 sec latency 46.337 ms 2024-05-27T07:16:24.176 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3917 1.43 MB/sec warmup 28 sec latency 46.915 ms 2024-05-27T07:16:24.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:24 smithi132 ceph-mon[35283]: pgmap v78: 129 pgs: 129 active+clean; 27 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 555 KiB/s wr, 40 op/s 2024-05-27T07:16:24.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:24 smithi043 ceph-mon[31531]: pgmap v78: 129 pgs: 129 active+clean; 27 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 555 KiB/s wr, 40 op/s 2024-05-27T07:16:24.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:24 smithi179 ceph-mon[35888]: pgmap v78: 129 pgs: 129 active+clean; 27 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 555 KiB/s wr, 40 op/s 2024-05-27T07:16:25.176 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3971 1.38 MB/sec warmup 29 sec latency 44.884 ms 2024-05-27T07:16:25.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:25 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:25.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:25 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:25.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:25 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:26.111 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:25 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:16:26.111 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:25 smithi179 systemd[1]: Stopped Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:26.111 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:25 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.471s CPU time. 2024-05-27T07:16:26.111 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:25 smithi179 systemd[1]: Starting Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:16:26.176 INFO:tasks.workunit.client.0.smithi043.stdout: 1 4028 1.34 MB/sec warmup 30 sec latency 57.257 ms 2024-05-27T07:16:26.447 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:26 smithi179 ceph-mon[35888]: pgmap v79: 129 pgs: 129 active+clean; 27 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 537 KiB/s wr, 39 op/s 2024-05-27T07:16:26.447 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:26 smithi179 podman[71012]: 2024-05-27 07:16:26.025375825 +0000 UTC m=+0.033811584 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:26.448 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:26 smithi179 podman[71012]: 2024-05-27 07:16:26.156940166 +0000 UTC m=+0.165375946 container create 713c2a2e8f411241998122d832ed64a43ed13a093bbe1baa88db852d4b42cb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T07:16:26.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:26 smithi043 ceph-mon[31531]: pgmap v79: 129 pgs: 129 active+clean; 27 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 537 KiB/s wr, 39 op/s 2024-05-27T07:16:26.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:26 smithi179 podman[71012]: 2024-05-27 07:16:26.457216664 +0000 UTC m=+0.465652465 container init 713c2a2e8f411241998122d832ed64a43ed13a093bbe1baa88db852d4b42cb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:16:26.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:26 smithi179 podman[71012]: 2024-05-27 07:16:26.468204033 +0000 UTC m=+0.476639800 container start 713c2a2e8f411241998122d832ed64a43ed13a093bbe1baa88db852d4b42cb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-27T07:16:26.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:26 smithi179 bash[71012]: 713c2a2e8f411241998122d832ed64a43ed13a093bbe1baa88db852d4b42cb1d 2024-05-27T07:16:26.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:26 smithi179 systemd[1]: Started Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:26.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: starting mds.c at 2024-05-27T07:16:26.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:26 smithi132 ceph-mon[35283]: pgmap v79: 129 pgs: 129 active+clean; 27 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 537 KiB/s wr, 39 op/s 2024-05-27T07:16:27.176 INFO:tasks.workunit.client.0.smithi043.stdout: 1 4097 1.29 MB/sec warmup 31 sec latency 61.025 ms 2024-05-27T07:16:27.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:27 smithi043 ceph-mon[31531]: daemon mds.c restarted 2024-05-27T07:16:27.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-mon[35888]: daemon mds.c restarted 2024-05-27T07:16:27.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa7e3c00700 time 2024-05-27T07:16:27.490616+0000 2024-05-27T07:16:27.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:27.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:27.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa7f963fba6] 2024-05-27T07:16:27.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa7f963fdd1] 2024-05-27T07:16:27.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b46cd6223d] 2024-05-27T07:16:27.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b46cd4d971] 2024-05-27T07:16:27.710 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b46cd4bfa0] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b46cd4c651] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b46cce1347] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b46c9dd87e] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b46c9e2e94] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b46c9ed969] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b46cd8648c] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b46cd8f9cb] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 13: (Thread::entry_wrapper()+0x43) [0x7fa7f960e72d] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa7f960e749] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa7f81b21ca] 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 16: clone() 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: *** Caught signal (Aborted) ** 2024-05-27T07:16:27.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: in thread 7fa7e3c00700 thread_name:md_log_replay 2024-05-27T07:16:27.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.492+0000 7fa7e3c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa7e3c00700 time 2024-05-27T07:16:27.490616+0000 2024-05-27T07:16:27.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:27.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:27.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa7f963fba6] 2024-05-27T07:16:27.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa7f963fdd1] 2024-05-27T07:16:27.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b46cd6223d] 2024-05-27T07:16:27.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b46cd4d971] 2024-05-27T07:16:27.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b46cd4bfa0] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b46cd4c651] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b46cce1347] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b46c9dd87e] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b46c9e2e94] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b46c9ed969] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b46cd8648c] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b46cd8f9cb] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 13: (Thread::entry_wrapper()+0x43) [0x7fa7f960e72d] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa7f960e749] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa7f81b21ca] 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 16: clone() 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:27.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa7f81bcd20] 2024-05-27T07:16:27.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2: gsignal() 2024-05-27T07:16:27.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 3: abort() 2024-05-27T07:16:27.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa7f963fced] 2024-05-27T07:16:27.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa7f963fdd1] 2024-05-27T07:16:27.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b46cd6223d] 2024-05-27T07:16:27.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b46cd4d971] 2024-05-27T07:16:27.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b46cd4bfa0] 2024-05-27T07:16:27.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b46cd4c651] 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b46cce1347] 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b46c9dd87e] 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b46c9e2e94] 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b46c9ed969] 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b46cd8648c] 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b46cd8f9cb] 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 16: (Thread::entry_wrapper()+0x43) [0x7fa7f960e72d] 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa7f960e749] 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa7f81b21ca] 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 19: clone() 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.496+0000 7fa7e3c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: in thread 7fa7e3c00700 thread_name:md_log_replay 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:27.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa7f81bcd20] 2024-05-27T07:16:27.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2: gsignal() 2024-05-27T07:16:27.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 3: abort() 2024-05-27T07:16:27.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa7f963fced] 2024-05-27T07:16:27.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa7f963fdd1] 2024-05-27T07:16:27.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b46cd6223d] 2024-05-27T07:16:27.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b46cd4d971] 2024-05-27T07:16:27.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b46cd4bfa0] 2024-05-27T07:16:27.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b46cd4c651] 2024-05-27T07:16:27.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b46cce1347] 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b46c9dd87e] 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b46c9e2e94] 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b46c9ed969] 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b46cd8648c] 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b46cd8f9cb] 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 16: (Thread::entry_wrapper()+0x43) [0x7fa7f960e72d] 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa7f960e749] 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa7f81b21ca] 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 19: clone() 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:27.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: -1> 2024-05-27T07:16:27.492+0000 7fa7e3c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa7e3c00700 time 2024-05-27T07:16:27.490616+0000 2024-05-27T07:16:27.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:27.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:27.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa7f963fba6] 2024-05-27T07:16:27.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa7f963fdd1] 2024-05-27T07:16:27.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b46cd6223d] 2024-05-27T07:16:27.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b46cd4d971] 2024-05-27T07:16:27.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b46cd4bfa0] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b46cd4c651] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b46cce1347] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b46c9dd87e] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b46c9e2e94] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b46c9ed969] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b46cd8648c] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b46cd8f9cb] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 13: (Thread::entry_wrapper()+0x43) [0x7fa7f960e72d] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa7f960e749] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa7f81b21ca] 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 16: clone() 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 0> 2024-05-27T07:16:27.496+0000 7fa7e3c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: in thread 7fa7e3c00700 thread_name:md_log_replay 2024-05-27T07:16:27.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa7f81bcd20] 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2: gsignal() 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 3: abort() 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa7f963fced] 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa7f963fdd1] 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b46cd6223d] 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b46cd4d971] 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b46cd4bfa0] 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b46cd4c651] 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b46cce1347] 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b46c9dd87e] 2024-05-27T07:16:27.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b46c9e2e94] 2024-05-27T07:16:27.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b46c9ed969] 2024-05-27T07:16:27.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b46cd8648c] 2024-05-27T07:16:27.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b46cd8f9cb] 2024-05-27T07:16:27.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 16: (Thread::entry_wrapper()+0x43) [0x7fa7f960e72d] 2024-05-27T07:16:27.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa7f960e749] 2024-05-27T07:16:27.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa7f81b21ca] 2024-05-27T07:16:27.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 19: clone() 2024-05-27T07:16:27.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: -1> 2024-05-27T07:16:27.492+0000 7fa7e3c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fa7e3c00700 time 2024-05-27T07:16:27.490616+0000 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa7f963fba6] 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa7f963fdd1] 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b46cd6223d] 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b46cd4d971] 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b46cd4bfa0] 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b46cd4c651] 2024-05-27T07:16:27.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b46cce1347] 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b46c9dd87e] 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b46c9e2e94] 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b46c9ed969] 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b46cd8648c] 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b46cd8f9cb] 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 13: (Thread::entry_wrapper()+0x43) [0x7fa7f960e72d] 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 14: (Thread::_entry_func(void*)+0xd) [0x7fa7f960e749] 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa7f81b21ca] 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 16: clone() 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 0> 2024-05-27T07:16:27.496+0000 7fa7e3c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: in thread 7fa7e3c00700 thread_name:md_log_replay 2024-05-27T07:16:27.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa7f81bcd20] 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2: gsignal() 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 3: abort() 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa7f963fced] 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa7f963fdd1] 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b46cd6223d] 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b46cd4d971] 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b46cd4bfa0] 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b46cd4c651] 2024-05-27T07:16:27.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b46cce1347] 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b46c9dd87e] 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b46c9e2e94] 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b46c9ed969] 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b46cd8648c] 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b46cd8f9cb] 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 16: (Thread::entry_wrapper()+0x43) [0x7fa7f960e72d] 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 17: (Thread::_entry_func(void*)+0xd) [0x7fa7f960e749] 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa7f81b21ca] 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 19: clone() 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71027]: 2024-05-27T07:16:27.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:27 smithi179 podman[71064]: 2024-05-27 07:16:27.612134986 +0000 UTC m=+0.032897805 container died 713c2a2e8f411241998122d832ed64a43ed13a093bbe1baa88db852d4b42cb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5) 2024-05-27T07:16:27.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:27 smithi132 ceph-mon[35283]: daemon mds.c restarted 2024-05-27T07:16:28.177 INFO:tasks.workunit.client.0.smithi043.stdout: 1 4162 1.25 MB/sec warmup 32 sec latency 42.405 ms 2024-05-27T07:16:28.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:28 smithi043 ceph-mon[31531]: pgmap v80: 129 pgs: 129 active+clean; 28 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 630 KiB/s wr, 55 op/s 2024-05-27T07:16:28.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:28 smithi043 ceph-mon[31531]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:16:28.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:28 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6836/3449501518,v1:172.21.15.179:6837/3449501518] up:boot 2024-05-27T07:16:28.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:28 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:28.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:28 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:28.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:28 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:28.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd516800700 time 2024-05-27T07:16:28.251804+0000 2024-05-27T07:16:28.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:28.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd52c2a9ba6] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd52c2a9dd1] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd31f0b23d] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cd31ef6971] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cd31ef4fa0] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd31ef5651] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd31e8a347] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd31b8687e] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cd31b8be94] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cd31b96969] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cd31f2f48c] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cd31f389cb] 2024-05-27T07:16:28.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 13: (Thread::entry_wrapper()+0x43) [0x7fd52c27872d] 2024-05-27T07:16:28.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd52c278749] 2024-05-27T07:16:28.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd52ae1c1ca] 2024-05-27T07:16:28.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 16: clone() 2024-05-27T07:16:28.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: *** Caught signal (Aborted) ** 2024-05-27T07:16:28.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: in thread 7fd516800700 thread_name:md_log_replay 2024-05-27T07:16:28.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:28 smithi179 ceph-mon[35888]: pgmap v80: 129 pgs: 129 active+clean; 28 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 630 KiB/s wr, 55 op/s 2024-05-27T07:16:28.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:28 smithi179 ceph-mon[35888]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:16:28.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:28 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6836/3449501518,v1:172.21.15.179:6837/3449501518] up:boot 2024-05-27T07:16:28.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:28 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:28.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:28 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:28.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:28 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:28.709 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:28 smithi179 podman[71064]: 2024-05-27 07:16:28.479552129 +0000 UTC m=+0.900314989 container remove 713c2a2e8f411241998122d832ed64a43ed13a093bbe1baa88db852d4b42cb1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-27T07:16:28.709 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:28 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:16:28.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.253+0000 7fd516800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd516800700 time 2024-05-27T07:16:28.251804+0000 2024-05-27T07:16:28.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:28.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:28.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd52c2a9ba6] 2024-05-27T07:16:28.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd52c2a9dd1] 2024-05-27T07:16:28.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd31f0b23d] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cd31ef6971] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cd31ef4fa0] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd31ef5651] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd31e8a347] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd31b8687e] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cd31b8be94] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cd31b96969] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cd31f2f48c] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cd31f389cb] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 13: (Thread::entry_wrapper()+0x43) [0x7fd52c27872d] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd52c278749] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd52ae1c1ca] 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 16: clone() 2024-05-27T07:16:28.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:28.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd52ae26d20] 2024-05-27T07:16:28.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2: gsignal() 2024-05-27T07:16:28.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 3: abort() 2024-05-27T07:16:28.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd52c2a9ced] 2024-05-27T07:16:28.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd52c2a9dd1] 2024-05-27T07:16:28.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd31f0b23d] 2024-05-27T07:16:28.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cd31ef6971] 2024-05-27T07:16:28.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cd31ef4fa0] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd31ef5651] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd31e8a347] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd31b8687e] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cd31b8be94] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cd31b96969] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cd31f2f48c] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cd31f389cb] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 16: (Thread::entry_wrapper()+0x43) [0x7fd52c27872d] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd52c278749] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd52ae1c1ca] 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 19: clone() 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.257+0000 7fd516800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: in thread 7fd516800700 thread_name:md_log_replay 2024-05-27T07:16:28.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd52ae26d20] 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2: gsignal() 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 3: abort() 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd52c2a9ced] 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd52c2a9dd1] 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd31f0b23d] 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cd31ef6971] 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cd31ef4fa0] 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd31ef5651] 2024-05-27T07:16:28.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd31e8a347] 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd31b8687e] 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cd31b8be94] 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cd31b96969] 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cd31f2f48c] 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cd31f389cb] 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 16: (Thread::entry_wrapper()+0x43) [0x7fd52c27872d] 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd52c278749] 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd52ae1c1ca] 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 19: clone() 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: -1> 2024-05-27T07:16:28.253+0000 7fd516800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd516800700 time 2024-05-27T07:16:28.251804+0000 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd52c2a9ba6] 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd52c2a9dd1] 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd31f0b23d] 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cd31ef6971] 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cd31ef4fa0] 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd31ef5651] 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd31e8a347] 2024-05-27T07:16:28.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd31b8687e] 2024-05-27T07:16:28.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cd31b8be94] 2024-05-27T07:16:28.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cd31b96969] 2024-05-27T07:16:28.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cd31f2f48c] 2024-05-27T07:16:28.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cd31f389cb] 2024-05-27T07:16:28.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 13: (Thread::entry_wrapper()+0x43) [0x7fd52c27872d] 2024-05-27T07:16:28.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd52c278749] 2024-05-27T07:16:28.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd52ae1c1ca] 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 16: clone() 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 0> 2024-05-27T07:16:28.257+0000 7fd516800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: in thread 7fd516800700 thread_name:md_log_replay 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd52ae26d20] 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2: gsignal() 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 3: abort() 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd52c2a9ced] 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd52c2a9dd1] 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd31f0b23d] 2024-05-27T07:16:28.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cd31ef6971] 2024-05-27T07:16:28.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cd31ef4fa0] 2024-05-27T07:16:28.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd31ef5651] 2024-05-27T07:16:28.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd31e8a347] 2024-05-27T07:16:28.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd31b8687e] 2024-05-27T07:16:28.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cd31b8be94] 2024-05-27T07:16:28.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cd31b96969] 2024-05-27T07:16:28.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cd31f2f48c] 2024-05-27T07:16:28.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cd31f389cb] 2024-05-27T07:16:28.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 16: (Thread::entry_wrapper()+0x43) [0x7fd52c27872d] 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd52c278749] 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd52ae1c1ca] 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 19: clone() 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: -8> 2024-05-27T07:16:28.253+0000 7fd516800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd516800700 time 2024-05-27T07:16:28.251804+0000 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd52c2a9ba6] 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd52c2a9dd1] 2024-05-27T07:16:28.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd31f0b23d] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cd31ef6971] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cd31ef4fa0] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd31ef5651] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd31e8a347] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd31b8687e] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cd31b8be94] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cd31b96969] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cd31f2f48c] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cd31f389cb] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 13: (Thread::entry_wrapper()+0x43) [0x7fd52c27872d] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd52c278749] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd52ae1c1ca] 2024-05-27T07:16:28.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 16: clone() 2024-05-27T07:16:28.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: -7> 2024-05-27T07:16:28.257+0000 7fd516800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:28.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: in thread 7fd516800700 thread_name:md_log_replay 2024-05-27T07:16:28.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd52ae26d20] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2: gsignal() 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 3: abort() 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd52c2a9ced] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd52c2a9dd1] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cd31f0b23d] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cd31ef6971] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cd31ef4fa0] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cd31ef5651] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cd31e8a347] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cd31b8687e] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cd31b8be94] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cd31b96969] 2024-05-27T07:16:28.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cd31f2f48c] 2024-05-27T07:16:28.724 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cd31f389cb] 2024-05-27T07:16:28.724 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 16: (Thread::entry_wrapper()+0x43) [0x7fd52c27872d] 2024-05-27T07:16:28.724 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd52c278749] 2024-05-27T07:16:28.724 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd52ae1c1ca] 2024-05-27T07:16:28.724 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 19: clone() 2024-05-27T07:16:28.724 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:28.724 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[70418]: 2024-05-27T07:16:28.724 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:28 smithi043 podman[95599]: 2024-05-27 07:16:28.424982304 +0000 UTC m=+0.051686892 container died e4fbe8dc111d5ce54b2d92a894dc66b85cc65722fb3013600561e12e8fe27f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:16:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:28 smithi132 ceph-mon[35283]: pgmap v80: 129 pgs: 129 active+clean; 28 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 630 KiB/s wr, 55 op/s 2024-05-27T07:16:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:28 smithi132 ceph-mon[35283]: osdmap e93: 12 total, 12 up, 12 in 2024-05-27T07:16:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:28 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6836/3449501518,v1:172.21.15.179:6837/3449501518] up:boot 2024-05-27T07:16:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:28 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:28 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:28 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:29.177 INFO:tasks.workunit.client.0.smithi043.stdout: 1 4277 1.22 MB/sec warmup 33 sec latency 55.469 ms 2024-05-27T07:16:29.209 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:28 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:16:29.209 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:28 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.509s CPU time. 2024-05-27T07:16:29.702 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:29 smithi043 podman[95599]: 2024-05-27 07:16:29.306672758 +0000 UTC m=+0.933377375 container remove e4fbe8dc111d5ce54b2d92a894dc66b85cc65722fb3013600561e12e8fe27f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:16:29.702 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:29 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:16:30.177 INFO:tasks.workunit.client.0.smithi043.stdout: 1 4424 1.20 MB/sec warmup 34 sec latency 56.477 ms 2024-05-27T07:16:30.202 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:29 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:16:30.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:29 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 3.211s CPU time. 2024-05-27T07:16:30.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:30 smithi043 ceph-mon[31531]: pgmap v82: 129 pgs: 129 active+clean; 28 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 367 KiB/s wr, 44 op/s 2024-05-27T07:16:30.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:30 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:30.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:30 smithi179 ceph-mon[35888]: pgmap v82: 129 pgs: 129 active+clean; 28 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 367 KiB/s wr, 44 op/s 2024-05-27T07:16:30.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:30 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:30.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:30 smithi132 ceph-mon[35283]: pgmap v82: 129 pgs: 129 active+clean; 28 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 367 KiB/s wr, 44 op/s 2024-05-27T07:16:30.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:30 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:31.177 INFO:tasks.workunit.client.0.smithi043.stdout: 1 4511 1.17 MB/sec warmup 35 sec latency 51.444 ms 2024-05-27T07:16:31.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:16:32.177 INFO:tasks.workunit.client.0.smithi043.stdout: 1 4631 1.17 MB/sec warmup 36 sec latency 52.989 ms 2024-05-27T07:16:32.495 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:32.663 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:32 smithi043 ceph-mon[31531]: pgmap v83: 129 pgs: 129 active+clean; 28 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 306 KiB/s wr, 54 op/s 2024-05-27T07:16:32.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:32 smithi179 ceph-mon[35888]: pgmap v83: 129 pgs: 129 active+clean; 28 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 306 KiB/s wr, 54 op/s 2024-05-27T07:16:32.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:32 smithi132 ceph-mon[35283]: pgmap v83: 129 pgs: 129 active+clean; 28 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 306 KiB/s wr, 54 op/s 2024-05-27T07:16:33.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 4747 1.14 MB/sec warmup 37 sec latency 51.364 ms 2024-05-27T07:16:34.177 INFO:tasks.workunit.client.0.smithi043.stdout: 1 4884 1.14 MB/sec warmup 38 sec latency 52.968 ms 2024-05-27T07:16:34.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:34 smithi043 ceph-mon[31531]: pgmap v84: 129 pgs: 129 active+clean; 28 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 306 KiB/s wr, 54 op/s 2024-05-27T07:16:34.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:34 smithi179 ceph-mon[35888]: pgmap v84: 129 pgs: 129 active+clean; 28 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 306 KiB/s wr, 54 op/s 2024-05-27T07:16:34.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:34 smithi132 ceph-mon[35283]: pgmap v84: 129 pgs: 129 active+clean; 28 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 306 KiB/s wr, 54 op/s 2024-05-27T07:16:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.990+0000 7fe0e9800700 1 -- 172.21.15.43:0/3228734841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0e4105bd0 msgr2=0x7fe0e410e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.990+0000 7fe0e9800700 1 --2- 172.21.15.43:0/3228734841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0e4105bd0 0x7fe0e410e120 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009990 tx=0x7fe0cc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.991+0000 7fe0e9800700 1 -- 172.21.15.43:0/3228734841 shutdown_connections 2024-05-27T07:16:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.991+0000 7fe0e9800700 1 --2- 172.21.15.43:0/3228734841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0e410e6f0 0x7fe0e4110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.991+0000 7fe0e9800700 1 --2- 172.21.15.43:0/3228734841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0e4105280 0x7fe0e4105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.991+0000 7fe0e9800700 1 --2- 172.21.15.43:0/3228734841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0e4105bd0 0x7fe0e410e120 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.991+0000 7fe0e9800700 1 -- 172.21.15.43:0/3228734841 >> 172.21.15.43:0/3228734841 conn(0x7fe0e4078ab0 msgr2=0x7fe0e4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.991+0000 7fe0e9800700 1 -- 172.21.15.43:0/3228734841 shutdown_connections 2024-05-27T07:16:34.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.991+0000 7fe0e9800700 1 -- 172.21.15.43:0/3228734841 wait complete. 2024-05-27T07:16:34.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.992+0000 7fe0e9800700 1 Processor -- start 2024-05-27T07:16:34.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.992+0000 7fe0e9800700 1 -- start start 2024-05-27T07:16:34.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.993+0000 7fe0e9800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0e4105280 0x7fe0e419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:34.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.993+0000 7fe0e9800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0e4105bd0 0x7fe0e419d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:34.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.993+0000 7fe0e9800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0e410e6f0 0x7fe0e41a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:34.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.993+0000 7fe0e9800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4114150 con 0x7fe0e410e6f0 2024-05-27T07:16:34.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.993+0000 7fe0e9800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4113fd0 con 0x7fe0e4105bd0 2024-05-27T07:16:34.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.993+0000 7fe0e9800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e41142d0 con 0x7fe0e4105280 2024-05-27T07:16:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.993+0000 7fe0e8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0e410e6f0 0x7fe0e41a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.993+0000 7fe0e3e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0e4105280 0x7fe0e419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.993+0000 7fe0e8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0e410e6f0 0x7fe0e41a15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35196/0 (socket says 172.21.15.43:35196) 2024-05-27T07:16:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.993+0000 7fe0e8e00700 1 -- 172.21.15.43:0/1608187184 learned_addr learned my addr 172.21.15.43:0/1608187184 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.994+0000 7fe0db400700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0e4105bd0 0x7fe0e419d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.994+0000 7fe0e8e00700 1 -- 172.21.15.43:0/1608187184 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0e4105280 msgr2=0x7fe0e419cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.994+0000 7fe0e8e00700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0e4105280 0x7fe0e419cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.994+0000 7fe0e8e00700 1 -- 172.21.15.43:0/1608187184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0e4105bd0 msgr2=0x7fe0e419d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.994+0000 7fe0e8e00700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0e4105bd0 0x7fe0e419d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.994+0000 7fe0e8e00700 1 -- 172.21.15.43:0/1608187184 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e41a1c70 con 0x7fe0e410e6f0 2024-05-27T07:16:34.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.995+0000 7fe0e8e00700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0e410e6f0 0x7fe0e41a15d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc007ee0 tx=0x7fe0dc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:34.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.995+0000 7fe0e1600700 1 -- 172.21.15.43:0/1608187184 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc004bb0 con 0x7fe0e410e6f0 2024-05-27T07:16:34.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.995+0000 7fe0e1600700 1 -- 172.21.15.43:0/1608187184 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0dc004d50 con 0x7fe0e410e6f0 2024-05-27T07:16:34.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.995+0000 7fe0e1600700 1 -- 172.21.15.43:0/1608187184 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0058b0 con 0x7fe0e410e6f0 2024-05-27T07:16:34.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.995+0000 7fe0e9800700 1 -- 172.21.15.43:0/1608187184 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e41a1f60 con 0x7fe0e410e6f0 2024-05-27T07:16:34.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:34.995+0000 7fe0e9800700 1 -- 172.21.15.43:0/1608187184 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e41a9880 con 0x7fe0e410e6f0 2024-05-27T07:16:34.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.000+0000 7fe0e1600700 1 -- 172.21.15.43:0/1608187184 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe0dc01a020 con 0x7fe0e410e6f0 2024-05-27T07:16:34.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.001+0000 7fe0f98f7b80 1 Processor -- start 2024-05-27T07:16:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.001+0000 7fe0f98f7b80 1 -- start start 2024-05-27T07:16:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.002+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 0x56540b1bced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.002+0000 7fe0e1600700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe0c80b2a70 0x7fe0c80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.002+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b1bd410 0x56540b1c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.002+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 0x56540b1ca020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.002+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56540b0ebe40 con 0x56540b1bd820 2024-05-27T07:16:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.002+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56540b0db9e0 con 0x56540b1bd410 2024-05-27T07:16:34.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.002+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56540b0dfa20 con 0x56540b03ee40 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.002+0000 7fe0e1600700 1 -- 172.21.15.43:0/1608187184 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9408+0+0 (secure 0 0 0) 0x7fe0dc0e1fb0 con 0x7fe0e410e6f0 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.002+0000 7fe0e3e00700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe0c80b2a70 0x7fe0c80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.003+0000 7fe0db400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 0x56540b1bced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.003+0000 7fe0db400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 0x56540b1bced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:34592/0 (socket says 172.21.15.43:34592) 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.003+0000 7fe0db400700 1 -- 172.21.15.43:0/1322612544 learned_addr learned my addr 172.21.15.43:0/1322612544 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.003+0000 7fe0db400700 1 --2- 172.21.15.43:0/1322612544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b1bd410 0x56540b1c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.003+0000 7fe0e3e00700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe0c80b2a70 0x7fe0c80b4f80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4005950 tx=0x7fe0d400b410 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.003+0000 7fe0db400700 1 -- 172.21.15.43:0/1322612544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b1bd410 msgr2=0x56540b1c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.003+0000 7fe0db400700 1 --2- 172.21.15.43:0/1322612544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b1bd410 0x56540b1c7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.003+0000 7fe0db400700 1 -- 172.21.15.43:0/1322612544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 msgr2=0x56540b1ca020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.003+0000 7fe0db400700 1 --2- 172.21.15.43:0/1322612544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 0x56540b1ca020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.003+0000 7fe0db400700 1 -- 172.21.15.43:0/1322612544 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56540b0a4420 con 0x56540b03ee40 2024-05-27T07:16:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.004+0000 7fe0db400700 1 --2- 172.21.15.43:0/1322612544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 0x56540b1bced0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc015000 tx=0x7fe0cc01a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf14d393844d080a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.004+0000 7fe0d9600700 1 -- 172.21.15.43:0/1322612544 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc00bb30 con 0x56540b03ee40 2024-05-27T07:16:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.004+0000 7fe0d9600700 1 -- 172.21.15.43:0/1322612544 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0cc0055a0 con 0x56540b03ee40 2024-05-27T07:16:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.004+0000 7fe0d9600700 1 -- 172.21.15.43:0/1322612544 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc003cb0 con 0x56540b03ee40 2024-05-27T07:16:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.005+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1322612544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 msgr2=0x56540b1bced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.005+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/1322612544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 0x56540b1bced0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc015000 tx=0x7fe0cc01a040 comp rx=0 tx=0).stop 2024-05-27T07:16:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.006+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1322612544 shutdown_connections 2024-05-27T07:16:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.006+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/1322612544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 0x56540b1ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.006+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/1322612544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 0x56540b1bced0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.006+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/1322612544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b1bd410 0x56540b1c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.006+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1322612544 >> 172.21.15.43:0/1322612544 conn(0x56540b09efc0 msgr2=0x56540b1b8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.006+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1322612544 shutdown_connections 2024-05-27T07:16:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.006+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1322612544 wait complete. 2024-05-27T07:16:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.007+0000 7fe0f98f7b80 1 -- start start 2024-05-27T07:16:34.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.007+0000 7fe0f98f7b80 1 Processor -- start 2024-05-27T07:16:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.007+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 0x56540b1c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.007+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.007+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b1bd820 0x56540b1c8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.007+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56540afe1fa0 con 0x56540b1bd410 2024-05-27T07:16:34.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.007+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56540b07c760 con 0x56540b1bd820 2024-05-27T07:16:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.007+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56540b1ca930 con 0x56540b03ee40 2024-05-27T07:16:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0f98f7b80 10 client.0 _ll_register_callbacks cb 0x56540b085c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0db400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0db400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35212/0 (socket says 172.21.15.43:35212) 2024-05-27T07:16:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0db400700 1 -- 172.21.15.43:0/3511692825 learned_addr learned my addr 172.21.15.43:0/3511692825 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:35.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0e3e00700 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 0x56540b1c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0e8e00700 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b1bd820 0x56540b1c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0db400700 1 -- 172.21.15.43:0/3511692825 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 msgr2=0x56540b1c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0db400700 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 0x56540b1c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0db400700 1 -- 172.21.15.43:0/3511692825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b1bd820 msgr2=0x56540b1c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0db400700 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b1bd820 0x56540b1c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.008+0000 7fe0db400700 1 -- 172.21.15.43:0/3511692825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56540b02b8d0 con 0x56540b1bd410 2024-05-27T07:16:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.009+0000 7fe0db400700 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c7b10 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009990 tx=0x7fe0cc009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1195767490d3b8d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.009+0000 7fe0d2000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.009+0000 7fe0d2000700 1 -- 172.21.15.43:0/3511692825 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc034060 con 0x56540b1bd410 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.009+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/3511692825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0cc0055a0 con 0x56540b1bd410 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.009+0000 7fe0d2000700 1 -- 172.21.15.43:0/3511692825 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9408+0+0 (secure 0 0 0) 0x7fe0cc003bd0 con 0x56540b1bd410 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.009+0000 7fe0d2000700 1 -- 172.21.15.43:0/3511692825 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe0cc03b050 con 0x56540b1bd410 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.010+0000 7fe0f98f7b80 10 client.15813 fetch_fsmap learned FSMap version 37 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.010+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/3511692825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56540b0ac060 con 0x56540b1bd410 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.010+0000 7fe0d2000700 1 -- 172.21.15.43:0/3511692825 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0cc0055a0 con 0x56540b1bd410 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.010+0000 7fe0d2000700 1 -- 172.21.15.43:0/3511692825 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7fe0cc00fc60 con 0x56540b1bd410 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.010+0000 7fe0f98f7b80 10 client.15813 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.010+0000 7fe0f98f7b80 10 client.15813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.010+0000 7fe0f98f7b80 20 client.15813 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.010+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56540b1d7780 0x56540b1d9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.010+0000 7fe0f98f7b80 4 client.15813 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:16:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.010+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/3511692825 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56540b0edd90 con 0x56540b1d7780 2024-05-27T07:16:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.011+0000 7fe0e3e00700 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56540b1d7780 0x56540b1d9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.012+0000 7fe0e3e00700 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56540b1d7780 0x56540b1d9ba0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a022f3812cb817bd server_cookie=3454be677deeb965 in_seq=0 out_seq=0 2024-05-27T07:16:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.012+0000 7fe0d2000700 10 client.15813 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:16:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0d2000700 1 -- 172.21.15.43:0/3511692825 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56540b0edd90 con 0x56540b1d7780 2024-05-27T07:16:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0d2000700 10 client.15813 handle_command_reply: tid=0 2024-05-27T07:16:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0f98f7b80 1 client.15813 shutdown 2024-05-27T07:16:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/3511692825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 msgr2=0x56540b1c7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c7b10 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009990 tx=0x7fe0cc009310 comp rx=0 tx=0).stop 2024-05-27T07:16:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/3511692825 shutdown_connections 2024-05-27T07:16:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b1bd820 0x56540b1c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b03ee40 0x56540b1c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56540b1d7780 0x56540b1d9ba0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/3511692825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c7b10 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/3511692825 >> 172.21.15.43:0/3511692825 conn(0x56540b09efc0 msgr2=0x56540b1b7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.015+0000 7fe0d2000700 0 client.15813 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:16:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.016+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/3511692825 shutdown_connections 2024-05-27T07:16:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.016+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/3511692825 wait complete. 2024-05-27T07:16:35.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.016+0000 7fe0f98f7b80 20 client.15813 trim_cache size 0 max 16384 2024-05-27T07:16:35.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.040+0000 7fe0f98f7b80 1 Processor -- start 2024-05-27T07:16:35.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.040+0000 7fe0f98f7b80 1 -- start start 2024-05-27T07:16:35.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.041+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b03ee40 0x56540b1b8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:35.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.041+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:35.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.041+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b1bd820 0x56540b1c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:35.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.041+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e41b2450 con 0x56540b1bd410 2024-05-27T07:16:35.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.041+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e41b4050 con 0x56540b03ee40 2024-05-27T07:16:35.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.041+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e41b2b20 con 0x56540b1bd820 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.041+0000 7fe0db400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.041+0000 7fe0db400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35216/0 (socket says 172.21.15.43:35216) 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.041+0000 7fe0db400700 1 -- 172.21.15.43:0/2517823416 learned_addr learned my addr 172.21.15.43:0/2517823416 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.041+0000 7fe0e3e00700 1 --2- 172.21.15.43:0/2517823416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b03ee40 0x56540b1b8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.042+0000 7fe0db400700 1 -- 172.21.15.43:0/2517823416 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b1bd820 msgr2=0x56540b1c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.042+0000 7fe0db400700 1 --2- 172.21.15.43:0/2517823416 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b1bd820 0x56540b1c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.042+0000 7fe0db400700 1 -- 172.21.15.43:0/2517823416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b03ee40 msgr2=0x56540b1b8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.042+0000 7fe0db400700 1 --2- 172.21.15.43:0/2517823416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b03ee40 0x56540b1b8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.042+0000 7fe0db400700 1 -- 172.21.15.43:0/2517823416 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56540b1ba350 con 0x56540b1bd410 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.042+0000 7fe0db400700 1 --2- 172.21.15.43:0/2517823416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c2e30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc005b00 tx=0x7fe0cc009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee50607004bb124 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.042+0000 7fe0d8c00700 1 -- 172.21.15.43:0/2517823416 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc034040 con 0x56540b1bd410 2024-05-27T07:16:35.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.042+0000 7fe0d8c00700 1 -- 172.21.15.43:0/2517823416 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0cc02f680 con 0x56540b1bd410 2024-05-27T07:16:35.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.042+0000 7fe0d8c00700 1 -- 172.21.15.43:0/2517823416 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc003e30 con 0x56540b1bd410 2024-05-27T07:16:35.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.043+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/2517823416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 msgr2=0x56540b1c2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:35.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.043+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/2517823416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c2e30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc005b00 tx=0x7fe0cc009810 comp rx=0 tx=0).stop 2024-05-27T07:16:35.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.043+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/2517823416 shutdown_connections 2024-05-27T07:16:35.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.043+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/2517823416 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b1bd820 0x56540b1c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.043+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/2517823416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b03ee40 0x56540b1b8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.043+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/2517823416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd410 0x56540b1c2e30 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.043+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/2517823416 >> 172.21.15.43:0/2517823416 conn(0x56540b09efc0 msgr2=0x56540b1d2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:35.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.044+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/2517823416 shutdown_connections 2024-05-27T07:16:35.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.044+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/2517823416 wait complete. 2024-05-27T07:16:35.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.044+0000 7fe0f98f7b80 1 -- start start 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.044+0000 7fe0f98f7b80 1 Processor -- start 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.045+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b03ee40 0x56540b1dd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.045+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b1bd410 0x56540b1da5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.045+0000 7fe0f98f7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 0x56540b1dab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.045+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e41b3920 con 0x56540b1bd820 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.045+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e41b31f0 con 0x56540b03ee40 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.045+0000 7fe0f98f7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e41b03d0 con 0x56540b1bd410 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.045+0000 7fe0f98f7b80 10 client.0 _ll_register_callbacks cb 0x56540b085c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.045+0000 7fe0e8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 0x56540b1dab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.045+0000 7fe0e8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 0x56540b1dab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35228/0 (socket says 172.21.15.43:35228) 2024-05-27T07:16:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.045+0000 7fe0e8e00700 1 -- 172.21.15.43:0/1942657194 learned_addr learned my addr 172.21.15.43:0/1942657194 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.046+0000 7fe0e3e00700 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b03ee40 0x56540b1dd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.046+0000 7fe0db400700 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b1bd410 0x56540b1da5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.046+0000 7fe0e8e00700 1 -- 172.21.15.43:0/1942657194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b1bd410 msgr2=0x56540b1da5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.046+0000 7fe0e8e00700 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b1bd410 0x56540b1da5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.046+0000 7fe0e8e00700 1 -- 172.21.15.43:0/1942657194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b03ee40 msgr2=0x56540b1dd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.046+0000 7fe0e8e00700 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b03ee40 0x56540b1dd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.046+0000 7fe0e8e00700 1 -- 172.21.15.43:0/1942657194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56540b1bf7b0 con 0x56540b1bd820 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.046+0000 7fe0e8e00700 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 0x56540b1dab30 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0a8be0 tx=0x7fe0dc0a8950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558e37818184ae4a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.046+0000 7fe0d2000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.046+0000 7fe0d2000700 1 -- 172.21.15.43:0/1942657194 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0a95e0 con 0x56540b1bd820 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0d2000700 1 -- 172.21.15.43:0/1942657194 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9408+0+0 (secure 0 0 0) 0x7fe0dc0a9350 con 0x56540b1bd820 2024-05-27T07:16:35.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1942657194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0cc02f680 con 0x56540b1bd820 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0d2000700 1 -- 172.21.15.43:0/1942657194 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe0dc0a8ee0 con 0x56540b1bd820 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0f98f7b80 10 client.15819 fetch_fsmap learned FSMap version 37 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1942657194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56540b241d30 con 0x56540b1bd820 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0d2000700 1 -- 172.21.15.43:0/1942657194 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe0cc02f680 con 0x56540b1bd820 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0d2000700 1 -- 172.21.15.43:0/1942657194 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7fe0dc0a4850 con 0x56540b1bd820 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0f98f7b80 10 client.15819 fetch_fsmap finished waiting for FSMap version 37 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0f98f7b80 10 client.15819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0f98f7b80 20 client.15819 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56540b242420 0x56540b244840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0f98f7b80 4 client.15819 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:16:35.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.047+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1942657194 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56540b0e7570 con 0x56540b242420 2024-05-27T07:16:35.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.048+0000 7fe0e3e00700 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56540b242420 0x56540b244840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:35.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.050+0000 7fe0e3e00700 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56540b242420 0x56540b244840 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a650c892e9ffeb48 server_cookie=c001c32fe7670fab in_seq=0 out_seq=0 2024-05-27T07:16:35.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.050+0000 7fe0d2000700 10 client.15819 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:16:35.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.050+0000 7fe0d2000700 1 -- 172.21.15.43:0/1942657194 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56540b0e7570 con 0x56540b242420 2024-05-27T07:16:35.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.050+0000 7fe0d2000700 10 client.15819 handle_command_reply: tid=0 2024-05-27T07:16:35.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.050+0000 7fe0f98f7b80 1 client.15819 shutdown 2024-05-27T07:16:35.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1942657194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 msgr2=0x56540b1dab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:35.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 0x56540b1dab30 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0a8be0 tx=0x7fe0dc0a8950 comp rx=0 tx=0).stop 2024-05-27T07:16:35.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1942657194 shutdown_connections 2024-05-27T07:16:35.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56540b242420 0x56540b244840 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56540b1bd820 0x56540b1dab30 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56540b03ee40 0x56540b1dd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 1 --2- 172.21.15.43:0/1942657194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56540b1bd410 0x56540b1da5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1942657194 >> 172.21.15.43:0/1942657194 conn(0x56540b09efc0 msgr2=0x56540b1be780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:35.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0d2000700 0 client.15819 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:16:35.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1942657194 shutdown_connections 2024-05-27T07:16:35.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 1 -- 172.21.15.43:0/1942657194 wait complete. 2024-05-27T07:16:35.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.051+0000 7fe0f98f7b80 20 client.15819 trim_cache size 0 max 16384 2024-05-27T07:16:35.044 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:16:35.044 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:16:35.044 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:16:35.044 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:16:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.052+0000 7fe0e9800700 1 -- 172.21.15.43:0/1608187184 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe0c80b2a70 msgr2=0x7fe0c80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.052+0000 7fe0e9800700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe0c80b2a70 0x7fe0c80b4f80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4005950 tx=0x7fe0d400b410 comp rx=0 tx=0).stop 2024-05-27T07:16:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.052+0000 7fe0e9800700 1 -- 172.21.15.43:0/1608187184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0e410e6f0 msgr2=0x7fe0e41a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.052+0000 7fe0e9800700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0e410e6f0 0x7fe0e41a15d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc007ee0 tx=0x7fe0dc00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.052+0000 7fe0e9800700 1 -- 172.21.15.43:0/1608187184 shutdown_connections 2024-05-27T07:16:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.052+0000 7fe0e9800700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe0c80b2a70 0x7fe0c80b4f80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.052+0000 7fe0e9800700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe0e410e6f0 0x7fe0e41a15d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.052+0000 7fe0e9800700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0e4105280 0x7fe0e419cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.052+0000 7fe0e9800700 1 --2- 172.21.15.43:0/1608187184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0e4105bd0 0x7fe0e419d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:35.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.052+0000 7fe0e9800700 1 -- 172.21.15.43:0/1608187184 >> 172.21.15.43:0/1608187184 conn(0x7fe0e4078ab0 msgr2=0x7fe0e40ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:35.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.053+0000 7fe0e9800700 1 -- 172.21.15.43:0/1608187184 shutdown_connections 2024-05-27T07:16:35.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:35.053+0000 7fe0e9800700 1 -- 172.21.15.43:0/1608187184 wait complete. 2024-05-27T07:16:35.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 5056 1.16 MB/sec warmup 39 sec latency 50.055 ms 2024-05-27T07:16:36.152 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:16:36.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:16:36.177 INFO:tasks.workunit.client.0.smithi043.stdout: 1 5175 1.13 MB/sec warmup 40 sec latency 50.123 ms 2024-05-27T07:16:36.643 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:36.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:36 smithi043 ceph-mon[31531]: pgmap v85: 129 pgs: 129 active+clean; 28 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 306 KiB/s wr, 54 op/s 2024-05-27T07:16:36.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:36 smithi179 ceph-mon[35888]: pgmap v85: 129 pgs: 129 active+clean; 28 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 306 KiB/s wr, 54 op/s 2024-05-27T07:16:36.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:36 smithi132 ceph-mon[35283]: pgmap v85: 129 pgs: 129 active+clean; 28 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 306 KiB/s wr, 54 op/s 2024-05-27T07:16:37.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 5342 1.11 MB/sec warmup 41 sec latency 50.579 ms 2024-05-27T07:16:38.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 5463 1.08 MB/sec warmup 42 sec latency 59.904 ms 2024-05-27T07:16:38.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:38 smithi043 ceph-mon[31531]: pgmap v86: 129 pgs: 129 active+clean; 26 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 272 KiB/s wr, 55 op/s 2024-05-27T07:16:38.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:38 smithi179 ceph-mon[35888]: pgmap v86: 129 pgs: 129 active+clean; 26 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 272 KiB/s wr, 55 op/s 2024-05-27T07:16:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:38 smithi132 ceph-mon[35283]: pgmap v86: 129 pgs: 129 active+clean; 26 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 272 KiB/s wr, 55 op/s 2024-05-27T07:16:39.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.141+0000 7f4eb3000700 1 -- 172.21.15.43:0/1153904427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eac0764a0 msgr2=0x7f4eac114c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:39.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.141+0000 7f4eb3000700 1 --2- 172.21.15.43:0/1153904427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eac0764a0 0x7f4eac114c30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4e980098f0 tx=0x7f4e98009c00 comp rx=0 tx=0).stop 2024-05-27T07:16:39.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.141+0000 7f4eb3000700 1 -- 172.21.15.43:0/1153904427 shutdown_connections 2024-05-27T07:16:39.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.141+0000 7f4eb3000700 1 --2- 172.21.15.43:0/1153904427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4eac115170 0x7f4eac117590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.141+0000 7f4eb3000700 1 --2- 172.21.15.43:0/1153904427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4eac075b50 0x7f4eac075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.141+0000 7f4eb3000700 1 --2- 172.21.15.43:0/1153904427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eac0764a0 0x7f4eac114c30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.141+0000 7f4eb3000700 1 -- 172.21.15.43:0/1153904427 >> 172.21.15.43:0/1153904427 conn(0x7f4eac079d00 msgr2=0x7f4eac07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:39.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.141+0000 7f4eb3000700 1 -- 172.21.15.43:0/1153904427 shutdown_connections 2024-05-27T07:16:39.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.141+0000 7f4eb3000700 1 -- 172.21.15.43:0/1153904427 wait complete. 2024-05-27T07:16:39.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.143+0000 7f4eb3000700 1 Processor -- start 2024-05-27T07:16:39.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.143+0000 7f4eb3000700 1 -- start start 2024-05-27T07:16:39.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.144+0000 7f4eb3000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4eac075b50 0x7f4eac1a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:39.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.144+0000 7f4eb3000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eac0764a0 0x7f4eac1a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:39.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.144+0000 7f4eb3000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4eac115170 0x7f4eac1a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:39.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.144+0000 7f4eb3000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac11a780 con 0x7f4eac075b50 2024-05-27T07:16:39.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.144+0000 7f4eb3000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac11a600 con 0x7f4eac115170 2024-05-27T07:16:39.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.144+0000 7f4eb3000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac11a900 con 0x7f4eac0764a0 2024-05-27T07:16:39.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.144+0000 7f4eb1c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4eac075b50 0x7f4eac1a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:39.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.144+0000 7f4eb1c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4eac075b50 0x7f4eac1a12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35256/0 (socket says 172.21.15.43:35256) 2024-05-27T07:16:39.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.144+0000 7f4eb1c00700 1 -- 172.21.15.43:0/3465735090 learned_addr learned my addr 172.21.15.43:0/3465735090 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:39.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.144+0000 7f4eb1200700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eac0764a0 0x7f4eac1a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:39.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.145+0000 7f4eb2600700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4eac115170 0x7f4eac1a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:39.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.145+0000 7f4eb1c00700 1 -- 172.21.15.43:0/3465735090 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eac0764a0 msgr2=0x7f4eac1a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:39.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.145+0000 7f4eb1c00700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eac0764a0 0x7f4eac1a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.145+0000 7f4eb1c00700 1 -- 172.21.15.43:0/3465735090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4eac115170 msgr2=0x7f4eac1a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:39.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.145+0000 7f4eb1c00700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4eac115170 0x7f4eac1a5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.145+0000 7f4eb1c00700 1 -- 172.21.15.43:0/3465735090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eac1a6180 con 0x7f4eac075b50 2024-05-27T07:16:39.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.145+0000 7f4eb1200700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eac0764a0 0x7f4eac1a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:16:39.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.145+0000 7f4eb1c00700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4eac075b50 0x7f4eac1a12b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000ec10 tx=0x7f4ea000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:39.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.145+0000 7f4eaa000700 1 -- 172.21.15.43:0/3465735090 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea000cd40 con 0x7f4eac075b50 2024-05-27T07:16:39.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.146+0000 7f4eaa000700 1 -- 172.21.15.43:0/3465735090 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ea00044b0 con 0x7f4eac075b50 2024-05-27T07:16:39.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.146+0000 7f4eb3000700 1 -- 172.21.15.43:0/3465735090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eac1a6470 con 0x7f4eac075b50 2024-05-27T07:16:39.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.146+0000 7f4eaa000700 1 -- 172.21.15.43:0/3465735090 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0010430 con 0x7f4eac075b50 2024-05-27T07:16:39.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.146+0000 7f4eb3000700 1 -- 172.21.15.43:0/3465735090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eac1add90 con 0x7f4eac075b50 2024-05-27T07:16:39.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.147+0000 7f4eb3000700 1 -- 172.21.15.43:0/3465735090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eac077290 con 0x7f4eac075b50 2024-05-27T07:16:39.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.150+0000 7f4eaa000700 1 -- 172.21.15.43:0/3465735090 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4ea0004000 con 0x7f4eac075b50 2024-05-27T07:16:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.152+0000 7f4eaa000700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e940b2a70 0x7f4e940b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.152+0000 7f4eaa000700 1 -- 172.21.15.43:0/3465735090 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9408+0+0 (secure 0 0 0) 0x7f4ea00e2d90 con 0x7f4eac075b50 2024-05-27T07:16:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.152+0000 7f4eb1200700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e940b2a70 0x7f4e940b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:39.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.153+0000 7f4eb1200700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e940b2a70 0x7f4e940b4f80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e980098c0 tx=0x7f4e98019040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:39.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.155+0000 7f4eaa000700 1 -- 172.21.15.43:0/3465735090 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4ea00aa450 con 0x7f4eac075b50 2024-05-27T07:16:39.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 5619 1.06 MB/sec warmup 43 sec latency 50.710 ms 2024-05-27T07:16:39.359 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:38 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:16:39.359 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:38 smithi179 systemd[1]: Stopped Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:39.360 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:38 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.509s CPU time. 2024-05-27T07:16:39.360 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:38 smithi179 systemd[1]: Starting Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:16:39.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.467+0000 7f4eb3000700 1 -- 172.21.15.43:0/3465735090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4eac1a2610 con 0x7f4eac075b50 2024-05-27T07:16:39.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.469+0000 7f4eaa000700 1 -- 172.21.15.43:0/3465735090 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11944 (secure 0 0 0) 0x7f4ea00a9d50 con 0x7f4eac075b50 2024-05-27T07:16:39.461 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:16:39.461 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/3666274467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":3666274467},{"type":"v1","addr":"172.21.15.43:6837","nonce":3666274467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:16:28.424899+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6839/807126288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":807126288},{"type":"v1","addr":"172.21.15.43:6839","nonce":807126288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25498":{"gid":25498,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/3449501518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":3449501518},{"type":"v1","addr":"172.21.15.179:6837","nonce":3449501518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[24484,34403,34424,34448,34484]},"id":1}]} 2024-05-27T07:16:39.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.471+0000 7f4eb3000700 1 -- 172.21.15.43:0/3465735090 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e940b2a70 msgr2=0x7f4e940b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:39.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.471+0000 7f4eb3000700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e940b2a70 0x7f4e940b4f80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e980098c0 tx=0x7f4e98019040 comp rx=0 tx=0).stop 2024-05-27T07:16:39.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.471+0000 7f4eb3000700 1 -- 172.21.15.43:0/3465735090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4eac075b50 msgr2=0x7f4eac1a12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:39.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.471+0000 7f4eb3000700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4eac075b50 0x7f4eac1a12b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000ec10 tx=0x7f4ea000ef20 comp rx=0 tx=0).stop 2024-05-27T07:16:39.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.472+0000 7f4eb3000700 1 -- 172.21.15.43:0/3465735090 shutdown_connections 2024-05-27T07:16:39.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.472+0000 7f4eb3000700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e940b2a70 0x7f4e940b4f80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.472+0000 7f4eb3000700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4eac115170 0x7f4eac1a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.472+0000 7f4eb3000700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4eac075b50 0x7f4eac1a12b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.472+0000 7f4eb3000700 1 --2- 172.21.15.43:0/3465735090 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eac0764a0 0x7f4eac1a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:39.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.472+0000 7f4eb3000700 1 -- 172.21.15.43:0/3465735090 >> 172.21.15.43:0/3465735090 conn(0x7f4eac079d00 msgr2=0x7f4eac102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:39.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.472+0000 7f4eb3000700 1 -- 172.21.15.43:0/3465735090 shutdown_connections 2024-05-27T07:16:39.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:39.472+0000 7f4eb3000700 1 -- 172.21.15.43:0/3465735090 wait complete. 2024-05-27T07:16:39.465 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 37 2024-05-27T07:16:39.673 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:39 smithi179 podman[71164]: 2024-05-27 07:16:39.273240031 +0000 UTC m=+0.032330162 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:39.674 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:39 smithi179 podman[71164]: 2024-05-27 07:16:39.39699579 +0000 UTC m=+0.156085958 container create 068e97d0bfef637a104dea2ea490dea0427363f88d33eb8134e2d24858a200f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T07:16:39.960 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:39 smithi179 podman[71164]: 2024-05-27 07:16:39.69177809 +0000 UTC m=+0.450868247 container init 068e97d0bfef637a104dea2ea490dea0427363f88d33eb8134e2d24858a200f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T07:16:39.960 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:39 smithi179 podman[71164]: 2024-05-27 07:16:39.701060132 +0000 UTC m=+0.460150279 container start 068e97d0bfef637a104dea2ea490dea0427363f88d33eb8134e2d24858a200f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-27T07:16:39.961 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:39 smithi179 bash[71164]: 068e97d0bfef637a104dea2ea490dea0427363f88d33eb8134e2d24858a200f3 2024-05-27T07:16:39.961 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:39 smithi179 ceph-mds[71193]: -- 172.21.15.179:0/567857269 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5647f07b3a40 con 0x5647f07fb600 2024-05-27T07:16:39.961 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:39 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: starting mds.c at 2024-05-27T07:16:39.961 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:39 smithi179 systemd[1]: Started Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:40.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 5762 1.04 MB/sec warmup 44 sec latency 57.842 ms 2024-05-27T07:16:40.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:39 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:16:40.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:39 smithi043 systemd[1]: Stopped Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:40.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:39 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 3.211s CPU time. 2024-05-27T07:16:40.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:39 smithi043 systemd[1]: Starting Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:16:40.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:16:40.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:40 smithi043 ceph-mon[31531]: pgmap v87: 129 pgs: 129 active+clean; 26 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 244 KiB/s wr, 49 op/s 2024-05-27T07:16:40.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:40 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3465735090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:40.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:40 smithi043 ceph-mon[31531]: daemon mds.c restarted 2024-05-27T07:16:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-mon[35888]: pgmap v87: 129 pgs: 129 active+clean; 26 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 244 KiB/s wr, 49 op/s 2024-05-27T07:16:40.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3465735090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:40.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-mon[35888]: daemon mds.c restarted 2024-05-27T07:16:40.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6d48c00700 time 2024-05-27T07:16:40.530200+0000 2024-05-27T07:16:40.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:40.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:40.711 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6d5e708ba6] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6d5e708dd1] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e53fb23d] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5647e53e6971] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 5: (SnapRealm::check_cache() const+0x13e) [0x5647e53e4fa0] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e53e5651] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e537a347] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e507687e] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5647e507be94] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5647e5086969] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 11: (MDLog::_replay_thread()+0x1f38) [0x5647e541f48c] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5647e54289cb] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 13: (Thread::entry_wrapper()+0x43) [0x7f6d5e6d772d] 2024-05-27T07:16:40.712 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6d5e6d7749] 2024-05-27T07:16:40.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6d5d27b1ca] 2024-05-27T07:16:40.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 16: clone() 2024-05-27T07:16:40.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: *** Caught signal (Aborted) ** 2024-05-27T07:16:40.713 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: in thread 7f6d48c00700 thread_name:md_log_replay 2024-05-27T07:16:40.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.532+0000 7f6d48c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6d48c00700 time 2024-05-27T07:16:40.530200+0000 2024-05-27T07:16:40.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:40.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:40.714 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6d5e708ba6] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6d5e708dd1] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e53fb23d] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5647e53e6971] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 5: (SnapRealm::check_cache() const+0x13e) [0x5647e53e4fa0] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e53e5651] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e537a347] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e507687e] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5647e507be94] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5647e5086969] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 11: (MDLog::_replay_thread()+0x1f38) [0x5647e541f48c] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5647e54289cb] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 13: (Thread::entry_wrapper()+0x43) [0x7f6d5e6d772d] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6d5e6d7749] 2024-05-27T07:16:40.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6d5d27b1ca] 2024-05-27T07:16:40.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 16: clone() 2024-05-27T07:16:40.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:40.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6d5d285d20] 2024-05-27T07:16:40.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2: gsignal() 2024-05-27T07:16:40.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 3: abort() 2024-05-27T07:16:40.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6d5e708ced] 2024-05-27T07:16:40.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6d5e708dd1] 2024-05-27T07:16:40.716 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e53fb23d] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5647e53e6971] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 8: (SnapRealm::check_cache() const+0x13e) [0x5647e53e4fa0] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e53e5651] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e537a347] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e507687e] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5647e507be94] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5647e5086969] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 14: (MDLog::_replay_thread()+0x1f38) [0x5647e541f48c] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5647e54289cb] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 16: (Thread::entry_wrapper()+0x43) [0x7f6d5e6d772d] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6d5e6d7749] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6d5d27b1ca] 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 19: clone() 2024-05-27T07:16:40.717 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.536+0000 7f6d48c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: in thread 7f6d48c00700 thread_name:md_log_replay 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6d5d285d20] 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2: gsignal() 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 3: abort() 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6d5e708ced] 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6d5e708dd1] 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e53fb23d] 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5647e53e6971] 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 8: (SnapRealm::check_cache() const+0x13e) [0x5647e53e4fa0] 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e53e5651] 2024-05-27T07:16:40.718 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e537a347] 2024-05-27T07:16:40.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e507687e] 2024-05-27T07:16:40.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5647e507be94] 2024-05-27T07:16:40.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5647e5086969] 2024-05-27T07:16:40.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 14: (MDLog::_replay_thread()+0x1f38) [0x5647e541f48c] 2024-05-27T07:16:40.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5647e54289cb] 2024-05-27T07:16:40.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 16: (Thread::entry_wrapper()+0x43) [0x7f6d5e6d772d] 2024-05-27T07:16:40.719 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6d5e6d7749] 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6d5d27b1ca] 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 19: clone() 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: -1> 2024-05-27T07:16:40.532+0000 7f6d48c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6d48c00700 time 2024-05-27T07:16:40.530200+0000 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6d5e708ba6] 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6d5e708dd1] 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e53fb23d] 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5647e53e6971] 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 5: (SnapRealm::check_cache() const+0x13e) [0x5647e53e4fa0] 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e53e5651] 2024-05-27T07:16:40.720 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e537a347] 2024-05-27T07:16:40.721 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e507687e] 2024-05-27T07:16:40.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5647e507be94] 2024-05-27T07:16:40.722 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5647e5086969] 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 11: (MDLog::_replay_thread()+0x1f38) [0x5647e541f48c] 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5647e54289cb] 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 13: (Thread::entry_wrapper()+0x43) [0x7f6d5e6d772d] 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6d5e6d7749] 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6d5d27b1ca] 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 16: clone() 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 0> 2024-05-27T07:16:40.536+0000 7f6d48c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: in thread 7f6d48c00700 thread_name:md_log_replay 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6d5d285d20] 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2: gsignal() 2024-05-27T07:16:40.723 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 3: abort() 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6d5e708ced] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6d5e708dd1] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e53fb23d] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5647e53e6971] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 8: (SnapRealm::check_cache() const+0x13e) [0x5647e53e4fa0] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e53e5651] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e537a347] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e507687e] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5647e507be94] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5647e5086969] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 14: (MDLog::_replay_thread()+0x1f38) [0x5647e541f48c] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5647e54289cb] 2024-05-27T07:16:40.724 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 16: (Thread::entry_wrapper()+0x43) [0x7f6d5e6d772d] 2024-05-27T07:16:40.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6d5e6d7749] 2024-05-27T07:16:40.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6d5d27b1ca] 2024-05-27T07:16:40.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 19: clone() 2024-05-27T07:16:40.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:40.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.725 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: -1> 2024-05-27T07:16:40.532+0000 7f6d48c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6d48c00700 time 2024-05-27T07:16:40.530200+0000 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6d5e708ba6] 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6d5e708dd1] 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e53fb23d] 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5647e53e6971] 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 5: (SnapRealm::check_cache() const+0x13e) [0x5647e53e4fa0] 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e53e5651] 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e537a347] 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e507687e] 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5647e507be94] 2024-05-27T07:16:40.726 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5647e5086969] 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 11: (MDLog::_replay_thread()+0x1f38) [0x5647e541f48c] 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5647e54289cb] 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 13: (Thread::entry_wrapper()+0x43) [0x7f6d5e6d772d] 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6d5e6d7749] 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6d5d27b1ca] 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 16: clone() 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 0> 2024-05-27T07:16:40.536+0000 7f6d48c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: in thread 7f6d48c00700 thread_name:md_log_replay 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6d5d285d20] 2024-05-27T07:16:40.727 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2: gsignal() 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 3: abort() 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6d5e708ced] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6d5e708dd1] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5647e53fb23d] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5647e53e6971] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 8: (SnapRealm::check_cache() const+0x13e) [0x5647e53e4fa0] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5647e53e5651] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5647e537a347] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5647e507687e] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5647e507be94] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5647e5086969] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 14: (MDLog::_replay_thread()+0x1f38) [0x5647e541f48c] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5647e54289cb] 2024-05-27T07:16:40.728 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 16: (Thread::entry_wrapper()+0x43) [0x7f6d5e6d772d] 2024-05-27T07:16:40.729 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6d5e6d7749] 2024-05-27T07:16:40.729 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6d5d27b1ca] 2024-05-27T07:16:40.729 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 19: clone() 2024-05-27T07:16:40.729 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:40.729 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71184]: 2024-05-27T07:16:40.729 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:40 smithi179 podman[71228]: 2024-05-27 07:16:40.639867199 +0000 UTC m=+0.032836162 container died 068e97d0bfef637a104dea2ea490dea0427363f88d33eb8134e2d24858a200f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-27T07:16:40.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:40 smithi132 ceph-mon[35283]: pgmap v87: 129 pgs: 129 active+clean; 26 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 244 KiB/s wr, 49 op/s 2024-05-27T07:16:40.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:40 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3465735090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:16:40.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:40 smithi132 ceph-mon[35283]: daemon mds.c restarted 2024-05-27T07:16:40.952 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:40 smithi043 podman[96033]: 2024-05-27 07:16:40.793490501 +0000 UTC m=+0.035050310 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:41.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 5924 1.04 MB/sec warmup 45 sec latency 51.087 ms 2024-05-27T07:16:41.355 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:41 smithi043 podman[96033]: 2024-05-27 07:16:41.183612503 +0000 UTC m=+0.425172343 container create c9321fd2eb1cb8856b759f07e46943e2555ae11f9e461d73ed903a2c498b7e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:16:41.648 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:41.702 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:41 smithi043 podman[96033]: 2024-05-27 07:16:41.463018649 +0000 UTC m=+0.704578495 container init c9321fd2eb1cb8856b759f07e46943e2555ae11f9e461d73ed903a2c498b7e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:41 smithi043 podman[96033]: 2024-05-27 07:16:41.474592468 +0000 UTC m=+0.716152341 container start c9321fd2eb1cb8856b759f07e46943e2555ae11f9e461d73ed903a2c498b7e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:41 smithi043 bash[96033]: c9321fd2eb1cb8856b759f07e46943e2555ae11f9e461d73ed903a2c498b7e2c 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:41 smithi043 systemd[1]: Started Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:41 smithi043 ceph-mds[96075]: -- 172.21.15.43:0/4234783444 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557c84511a40 con 0x557c84559180 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:41 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: starting mds.g at 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:41 smithi043 ceph-mon[31531]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:41 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6836/1939368333,v1:172.21.15.179:6837/1939368333] up:boot 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:41 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:41 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:41 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:41.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:41 smithi043 ceph-mon[31531]: pgmap v89: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 219 KiB/s wr, 47 op/s 2024-05-27T07:16:41.709 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:41 smithi179 podman[71228]: 2024-05-27 07:16:41.38104591 +0000 UTC m=+0.774014905 container remove 068e97d0bfef637a104dea2ea490dea0427363f88d33eb8134e2d24858a200f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0) 2024-05-27T07:16:41.709 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:41 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:16:41.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:41 smithi179 ceph-mon[35888]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:16:41.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:41 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6836/1939368333,v1:172.21.15.179:6837/1939368333] up:boot 2024-05-27T07:16:41.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:41 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:41.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:41 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:41.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:41 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:41.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:41 smithi179 ceph-mon[35888]: pgmap v89: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 219 KiB/s wr, 47 op/s 2024-05-27T07:16:41.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:41 smithi132 ceph-mon[35283]: osdmap e94: 12 total, 12 up, 12 in 2024-05-27T07:16:41.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:41 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6836/1939368333,v1:172.21.15.179:6837/1939368333] up:boot 2024-05-27T07:16:41.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:41 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:41.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:41 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:41.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:41 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:41.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:41 smithi132 ceph-mon[35283]: pgmap v89: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 219 KiB/s wr, 47 op/s 2024-05-27T07:16:42.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 6007 1.03 MB/sec warmup 46 sec latency 65.736 ms 2024-05-27T07:16:42.209 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:41 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:16:42.209 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:41 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.434s CPU time. 2024-05-27T07:16:42.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:42 smithi043 ceph-mon[31531]: daemon mds.g restarted 2024-05-27T07:16:42.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:42 smithi179 ceph-mon[35888]: daemon mds.g restarted 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f544d400700 time 2024-05-27T07:16:42.542904+0000 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5462f64ba6] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5462f64dd1] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557c73ec023d] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557c73eab971] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 5: (SnapRealm::check_cache() const+0x13e) [0x557c73ea9fa0] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557c73eaa651] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557c73e3f347] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557c73b3b87e] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557c73b40e94] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557c73b4b969] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 11: (MDLog::_replay_thread()+0x1f38) [0x557c73ee448c] 2024-05-27T07:16:42.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557c73eed9cb] 2024-05-27T07:16:42.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 13: (Thread::entry_wrapper()+0x43) [0x7f5462f3372d] 2024-05-27T07:16:42.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5462f33749] 2024-05-27T07:16:42.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5461ad71ca] 2024-05-27T07:16:42.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 16: clone() 2024-05-27T07:16:42.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.544+0000 7f544d400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f544d400700 time 2024-05-27T07:16:42.542904+0000 2024-05-27T07:16:42.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:42.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:42.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5462f64ba6] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5462f64dd1] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557c73ec023d] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557c73eab971] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 5: (SnapRealm::check_cache() const+0x13e) [0x557c73ea9fa0] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557c73eaa651] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557c73e3f347] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557c73b3b87e] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557c73b40e94] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557c73b4b969] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 11: (MDLog::_replay_thread()+0x1f38) [0x557c73ee448c] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557c73eed9cb] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 13: (Thread::entry_wrapper()+0x43) [0x7f5462f3372d] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5462f33749] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5461ad71ca] 2024-05-27T07:16:42.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 16: clone() 2024-05-27T07:16:42.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: *** Caught signal (Aborted) ** 2024-05-27T07:16:42.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: in thread 7f544d400700 thread_name:md_log_replay 2024-05-27T07:16:42.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:42.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5461ae1d20] 2024-05-27T07:16:42.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2: gsignal() 2024-05-27T07:16:42.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 3: abort() 2024-05-27T07:16:42.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5462f64ced] 2024-05-27T07:16:42.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5462f64dd1] 2024-05-27T07:16:42.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557c73ec023d] 2024-05-27T07:16:42.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557c73eab971] 2024-05-27T07:16:42.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 8: (SnapRealm::check_cache() const+0x13e) [0x557c73ea9fa0] 2024-05-27T07:16:42.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557c73eaa651] 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557c73e3f347] 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557c73b3b87e] 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557c73b40e94] 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557c73b4b969] 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 14: (MDLog::_replay_thread()+0x1f38) [0x557c73ee448c] 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557c73eed9cb] 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 16: (Thread::entry_wrapper()+0x43) [0x7f5462f3372d] 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5462f33749] 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5461ad71ca] 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 19: clone() 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.547+0000 7f544d400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: in thread 7f544d400700 thread_name:md_log_replay 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:42.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5461ae1d20] 2024-05-27T07:16:42.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2: gsignal() 2024-05-27T07:16:42.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 3: abort() 2024-05-27T07:16:42.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5462f64ced] 2024-05-27T07:16:42.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5462f64dd1] 2024-05-27T07:16:42.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557c73ec023d] 2024-05-27T07:16:42.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557c73eab971] 2024-05-27T07:16:42.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 8: (SnapRealm::check_cache() const+0x13e) [0x557c73ea9fa0] 2024-05-27T07:16:42.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557c73eaa651] 2024-05-27T07:16:42.720 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557c73e3f347] 2024-05-27T07:16:42.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557c73b3b87e] 2024-05-27T07:16:42.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557c73b40e94] 2024-05-27T07:16:42.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557c73b4b969] 2024-05-27T07:16:42.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 14: (MDLog::_replay_thread()+0x1f38) [0x557c73ee448c] 2024-05-27T07:16:42.721 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557c73eed9cb] 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 16: (Thread::entry_wrapper()+0x43) [0x7f5462f3372d] 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5462f33749] 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5461ad71ca] 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 19: clone() 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: -1> 2024-05-27T07:16:42.544+0000 7f544d400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f544d400700 time 2024-05-27T07:16:42.542904+0000 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:42.722 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5462f64ba6] 2024-05-27T07:16:42.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5462f64dd1] 2024-05-27T07:16:42.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557c73ec023d] 2024-05-27T07:16:42.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557c73eab971] 2024-05-27T07:16:42.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 5: (SnapRealm::check_cache() const+0x13e) [0x557c73ea9fa0] 2024-05-27T07:16:42.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557c73eaa651] 2024-05-27T07:16:42.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557c73e3f347] 2024-05-27T07:16:42.723 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557c73b3b87e] 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557c73b40e94] 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557c73b4b969] 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 11: (MDLog::_replay_thread()+0x1f38) [0x557c73ee448c] 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557c73eed9cb] 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 13: (Thread::entry_wrapper()+0x43) [0x7f5462f3372d] 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5462f33749] 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5461ad71ca] 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 16: clone() 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 0> 2024-05-27T07:16:42.547+0000 7f544d400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: in thread 7f544d400700 thread_name:md_log_replay 2024-05-27T07:16:42.725 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5461ae1d20] 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2: gsignal() 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 3: abort() 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5462f64ced] 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5462f64dd1] 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557c73ec023d] 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557c73eab971] 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 8: (SnapRealm::check_cache() const+0x13e) [0x557c73ea9fa0] 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557c73eaa651] 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557c73e3f347] 2024-05-27T07:16:42.726 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557c73b3b87e] 2024-05-27T07:16:42.727 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557c73b40e94] 2024-05-27T07:16:42.727 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557c73b4b969] 2024-05-27T07:16:42.727 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 14: (MDLog::_replay_thread()+0x1f38) [0x557c73ee448c] 2024-05-27T07:16:42.727 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557c73eed9cb] 2024-05-27T07:16:42.727 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 16: (Thread::entry_wrapper()+0x43) [0x7f5462f3372d] 2024-05-27T07:16:42.727 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5462f33749] 2024-05-27T07:16:42.727 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5461ad71ca] 2024-05-27T07:16:42.727 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 19: clone() 2024-05-27T07:16:42.727 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:42.727 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: -1> 2024-05-27T07:16:42.544+0000 7f544d400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f544d400700 time 2024-05-27T07:16:42.542904+0000 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5462f64ba6] 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5462f64dd1] 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557c73ec023d] 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 4: (SnapRealm::build_snap_set() const+0x323) [0x557c73eab971] 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 5: (SnapRealm::check_cache() const+0x13e) [0x557c73ea9fa0] 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557c73eaa651] 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557c73e3f347] 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557c73b3b87e] 2024-05-27T07:16:42.728 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557c73b40e94] 2024-05-27T07:16:42.729 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x557c73b4b969] 2024-05-27T07:16:42.729 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 11: (MDLog::_replay_thread()+0x1f38) [0x557c73ee448c] 2024-05-27T07:16:42.729 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 12: (MDLog::ReplayThread::entry()+0x11) [0x557c73eed9cb] 2024-05-27T07:16:42.729 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 13: (Thread::entry_wrapper()+0x43) [0x7f5462f3372d] 2024-05-27T07:16:42.729 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5462f33749] 2024-05-27T07:16:42.729 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5461ad71ca] 2024-05-27T07:16:42.729 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 16: clone() 2024-05-27T07:16:42.729 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 0> 2024-05-27T07:16:42.547+0000 7f544d400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: in thread 7f544d400700 thread_name:md_log_replay 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5461ae1d20] 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2: gsignal() 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 3: abort() 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5462f64ced] 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5462f64dd1] 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x557c73ec023d] 2024-05-27T07:16:42.730 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 7: (SnapRealm::build_snap_set() const+0x323) [0x557c73eab971] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 8: (SnapRealm::check_cache() const+0x13e) [0x557c73ea9fa0] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x557c73eaa651] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x557c73e3f347] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x557c73b3b87e] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x557c73b40e94] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x557c73b4b969] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 14: (MDLog::_replay_thread()+0x1f38) [0x557c73ee448c] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 15: (MDLog::ReplayThread::entry()+0x11) [0x557c73eed9cb] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 16: (Thread::entry_wrapper()+0x43) [0x7f5462f3372d] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5462f33749] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5461ad71ca] 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 19: clone() 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:42.731 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96071]: 2024-05-27T07:16:42.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:42 smithi132 ceph-mon[35283]: daemon mds.g restarted 2024-05-27T07:16:43.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 6134 1.02 MB/sec warmup 47 sec latency 60.760 ms 2024-05-27T07:16:43.391 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:43 smithi043 podman[96152]: 2024-05-27 07:16:43.068263719 +0000 UTC m=+0.463606265 container died c9321fd2eb1cb8856b759f07e46943e2555ae11f9e461d73ed903a2c498b7e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-27T07:16:43.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:43 smithi043 ceph-mon[31531]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:16:43.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:43 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6838/3451846239,v1:172.21.15.43:6839/3451846239] up:boot 2024-05-27T07:16:43.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:43 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:43.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:43 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:16:43.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:43 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:43.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:43 smithi043 ceph-mon[31531]: pgmap v91: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 274 KiB/s wr, 59 op/s 2024-05-27T07:16:43.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:43 smithi179 ceph-mon[35888]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:16:43.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:43 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6838/3451846239,v1:172.21.15.43:6839/3451846239] up:boot 2024-05-27T07:16:43.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:43 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:43.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:43 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:16:43.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:43 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:43.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:43 smithi179 ceph-mon[35888]: pgmap v91: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 274 KiB/s wr, 59 op/s 2024-05-27T07:16:43.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:43 smithi132 ceph-mon[35283]: osdmap e95: 12 total, 12 up, 12 in 2024-05-27T07:16:43.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:43 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6838/3451846239,v1:172.21.15.43:6839/3451846239] up:boot 2024-05-27T07:16:43.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:43 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:43.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:43 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:16:43.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:43 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:43.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:43 smithi132 ceph-mon[35283]: pgmap v91: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 274 KiB/s wr, 59 op/s 2024-05-27T07:16:44.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 6317 1.06 MB/sec warmup 48 sec latency 58.025 ms 2024-05-27T07:16:44.202 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:43 smithi043 podman[96152]: 2024-05-27 07:16:43.836927329 +0000 UTC m=+1.232269843 container remove c9321fd2eb1cb8856b759f07e46943e2555ae11f9e461d73ed903a2c498b7e2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-27T07:16:44.202 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:43 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:16:44.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.711+0000 7fb4b8a00700 1 -- 172.21.15.43:0/2700818914 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b4077550 msgr2=0x7fb4b4110690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:44.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.711+0000 7fb4b8a00700 1 --2- 172.21.15.43:0/2700818914 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b4077550 0x7fb4b4110690 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009990 tx=0x7fb4a0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.711+0000 7fb4b8a00700 1 -- 172.21.15.43:0/2700818914 shutdown_connections 2024-05-27T07:16:44.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.711+0000 7fb4b8a00700 1 --2- 172.21.15.43:0/2700818914 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4b4110bd0 0x7fb4b4112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.711+0000 7fb4b8a00700 1 --2- 172.21.15.43:0/2700818914 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b4076c00 0x7fb4b4077010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.711+0000 7fb4b8a00700 1 --2- 172.21.15.43:0/2700818914 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b4077550 0x7fb4b4110690 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.711+0000 7fb4b8a00700 1 -- 172.21.15.43:0/2700818914 >> 172.21.15.43:0/2700818914 conn(0x7fb4b40ffe90 msgr2=0x7fb4b41022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:44.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.712+0000 7fb4b8a00700 1 -- 172.21.15.43:0/2700818914 shutdown_connections 2024-05-27T07:16:44.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.712+0000 7fb4b8a00700 1 -- 172.21.15.43:0/2700818914 wait complete. 2024-05-27T07:16:44.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.713+0000 7fb4b8a00700 1 Processor -- start 2024-05-27T07:16:44.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.713+0000 7fb4b8a00700 1 -- start start 2024-05-27T07:16:44.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b8a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4b4076c00 0x7fb4b419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b4077550 0x7fb4b419d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b8a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b4110bd0 0x7fb4b41a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b8a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b4116590 con 0x7fb4b4110bd0 2024-05-27T07:16:44.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b8a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b4116410 con 0x7fb4b4077550 2024-05-27T07:16:44.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b8a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b4116710 con 0x7fb4b4076c00 2024-05-27T07:16:44.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b2800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b4110bd0 0x7fb4b41a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b2800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b4110bd0 0x7fb4b41a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54490/0 (socket says 172.21.15.43:54490) 2024-05-27T07:16:44.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b2800700 1 -- 172.21.15.43:0/793129943 learned_addr learned my addr 172.21.15.43:0/793129943 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:44.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4b4076c00 0x7fb4b419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.714+0000 7fb4b1400700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b4077550 0x7fb4b419d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.715+0000 7fb4b2800700 1 -- 172.21.15.43:0/793129943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4b4076c00 msgr2=0x7fb4b419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:44.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.715+0000 7fb4b2800700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4b4076c00 0x7fb4b419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.715+0000 7fb4b2800700 1 -- 172.21.15.43:0/793129943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b4077550 msgr2=0x7fb4b419d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:44.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.715+0000 7fb4b2800700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b4077550 0x7fb4b419d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.715+0000 7fb4b2800700 1 -- 172.21.15.43:0/793129943 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b41a1e30 con 0x7fb4b4110bd0 2024-05-27T07:16:44.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.716+0000 7fb4b2800700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b4110bd0 0x7fb4b41a1790 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0094f0 tx=0x7fb4ac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:44.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.716+0000 7fb4aaa00700 1 -- 172.21.15.43:0/793129943 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac00ddd0 con 0x7fb4b4110bd0 2024-05-27T07:16:44.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.716+0000 7fb4aaa00700 1 -- 172.21.15.43:0/793129943 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4ac009e20 con 0x7fb4b4110bd0 2024-05-27T07:16:44.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.716+0000 7fb4b8a00700 1 -- 172.21.15.43:0/793129943 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b41a2120 con 0x7fb4b4110bd0 2024-05-27T07:16:44.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.717+0000 7fb4aaa00700 1 -- 172.21.15.43:0/793129943 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac015400 con 0x7fb4b4110bd0 2024-05-27T07:16:44.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.717+0000 7fb4b8a00700 1 -- 172.21.15.43:0/793129943 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b41a9a40 con 0x7fb4b4110bd0 2024-05-27T07:16:44.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.720+0000 7fb4aaa00700 1 -- 172.21.15.43:0/793129943 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb4ac0155a0 con 0x7fb4b4110bd0 2024-05-27T07:16:44.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.722+0000 7fb4c7538b80 1 Processor -- start 2024-05-27T07:16:44.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.722+0000 7fb4c7538b80 1 -- start start 2024-05-27T07:16:44.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.722+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb38ee40 0x557bdb5137f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.722+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 0x557bdb50d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.722+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb50e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.722+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557bdb43be40 con 0x557bdb50ddb0 2024-05-27T07:16:44.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.722+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557bdb42b9e0 con 0x557bdb513d30 2024-05-27T07:16:44.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.722+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557bdb42fa20 con 0x557bdb38ee40 2024-05-27T07:16:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.723+0000 7fb4b1400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb38ee40 0x557bdb5137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.723+0000 7fb4b1e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 0x557bdb50d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.723+0000 7fb4b1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb50e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.723+0000 7fb4b1e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 0x557bdb50d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41278/0 (socket says 172.21.15.43:41278) 2024-05-27T07:16:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.723+0000 7fb4b1e00700 1 -- 172.21.15.43:0/2893424781 learned_addr learned my addr 172.21.15.43:0/2893424781 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.723+0000 7fb4aaa00700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4980b2a70 0x7fb4980b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.724+0000 7fb4b1400700 1 -- 172.21.15.43:0/2893424781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb38ee40 msgr2=0x557bdb5137f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:44.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.724+0000 7fb4b1400700 1 --2- 172.21.15.43:0/2893424781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb38ee40 0x557bdb5137f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.724+0000 7fb4b1400700 1 -- 172.21.15.43:0/2893424781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 msgr2=0x557bdb50d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:44.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.724+0000 7fb4b1400700 1 --2- 172.21.15.43:0/2893424781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 0x557bdb50d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.724+0000 7fb4b1400700 1 -- 172.21.15.43:0/2893424781 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bdb3f4420 con 0x557bdb50ddb0 2024-05-27T07:16:44.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.724+0000 7fb4b1400700 1 --2- 172.21.15.43:0/2893424781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb50e1c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000b5c0 tx=0x7fb4a0005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4978c76a0af6349a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:44.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.724+0000 7fb4b2800700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4980b2a70 0x7fb4980b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.724+0000 7fb4aaa00700 1 -- 172.21.15.43:0/793129943 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9408+0+0 (secure 0 0 0) 0x7fb4ac0e2420 con 0x7fb4b4110bd0 2024-05-27T07:16:44.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.725+0000 7fb49f400700 1 -- 172.21.15.43:0/2893424781 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a000bb60 con 0x557bdb50ddb0 2024-05-27T07:16:44.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.725+0000 7fb49f400700 1 -- 172.21.15.43:0/2893424781 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4a00055a0 con 0x557bdb50ddb0 2024-05-27T07:16:44.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.725+0000 7fb49f400700 1 -- 172.21.15.43:0/2893424781 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0003c80 con 0x557bdb50ddb0 2024-05-27T07:16:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.726+0000 7fb4c7538b80 1 -- 172.21.15.43:0/2893424781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 msgr2=0x557bdb50e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.726+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/2893424781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb50e1c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000b5c0 tx=0x7fb4a0005a90 comp rx=0 tx=0).stop 2024-05-27T07:16:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.726+0000 7fb4b2800700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4980b2a70 0x7fb4980b4f80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0a53d0 tx=0x7fb4ac005b70 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.726+0000 7fb4c7538b80 1 -- 172.21.15.43:0/2893424781 shutdown_connections 2024-05-27T07:16:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.726+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/2893424781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb50e1c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.726+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/2893424781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb38ee40 0x557bdb5137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.726+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/2893424781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 0x557bdb50d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.726+0000 7fb4c7538b80 1 -- 172.21.15.43:0/2893424781 >> 172.21.15.43:0/2893424781 conn(0x557bdb3eefc0 msgr2=0x557bdb5088d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:44.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.726+0000 7fb4c7538b80 1 -- 172.21.15.43:0/2893424781 shutdown_connections 2024-05-27T07:16:44.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.726+0000 7fb4c7538b80 1 -- 172.21.15.43:0/2893424781 wait complete. 2024-05-27T07:16:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.727+0000 7fb4c7538b80 1 -- start start 2024-05-27T07:16:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.727+0000 7fb4c7538b80 1 Processor -- start 2024-05-27T07:16:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb5175c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb50ddb0 0x557bdb517b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb513d30 0x557bdb518040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557bdb331fa0 con 0x557bdb38ee40 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557bdb3cc760 con 0x557bdb50ddb0 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557bdb51a7a0 con 0x557bdb513d30 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4c7538b80 10 client.0 _ll_register_callbacks cb 0x557bdb3d5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4b1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb5175c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4b1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb5175c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54522/0 (socket says 172.21.15.43:54522) 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4b1400700 1 -- 172.21.15.43:0/1824836856 learned_addr learned my addr 172.21.15.43:0/1824836856 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.728+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb50ddb0 0x557bdb517b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.729+0000 7fb4b1400700 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb513d30 0x557bdb518040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.729+0000 7fb4b1400700 1 -- 172.21.15.43:0/1824836856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb513d30 msgr2=0x557bdb518040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:44.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.729+0000 7fb4b1400700 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb513d30 0x557bdb518040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.729+0000 7fb4b1400700 1 -- 172.21.15.43:0/1824836856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb50ddb0 msgr2=0x557bdb517b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:44.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.729+0000 7fb4b1400700 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb50ddb0 0x557bdb517b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.729+0000 7fb4b1400700 1 -- 172.21.15.43:0/1824836856 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bdb37b8d0 con 0x557bdb38ee40 2024-05-27T07:16:44.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.729+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb50ddb0 0x557bdb517b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:16:44.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.729+0000 7fb4b1400700 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb5175c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0005d40 tx=0x7fb4a0009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cccc1f3726fd8d75 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:44.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.729+0000 7fb493e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:44.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.730+0000 7fb493e00700 1 -- 172.21.15.43:0/1824836856 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0004ea0 con 0x557bdb38ee40 2024-05-27T07:16:44.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.730+0000 7fb493e00700 1 -- 172.21.15.43:0/1824836856 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9408+0+0 (secure 0 0 0) 0x7fb4a000f460 con 0x557bdb38ee40 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.730+0000 7fb4c7538b80 1 -- 172.21.15.43:0/1824836856 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4a00055a0 con 0x557bdb38ee40 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.730+0000 7fb493e00700 1 -- 172.21.15.43:0/1824836856 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb4a003d050 con 0x557bdb38ee40 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.730+0000 7fb4c7538b80 10 client.15843 fetch_fsmap learned FSMap version 41 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.730+0000 7fb4c7538b80 1 -- 172.21.15.43:0/1824836856 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x557bdb3fc060 con 0x557bdb38ee40 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.730+0000 7fb493e00700 1 -- 172.21.15.43:0/1824836856 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4a00055a0 con 0x557bdb38ee40 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.731+0000 7fb493e00700 1 -- 172.21.15.43:0/1824836856 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7fb4a00276d0 con 0x557bdb38ee40 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.731+0000 7fb4c7538b80 10 client.15843 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.731+0000 7fb4c7538b80 10 client.15843 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.731+0000 7fb4c7538b80 20 client.15843 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.731+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x557bdb5275f0 0x557bdb529a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.731+0000 7fb4c7538b80 4 client.15843 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:16:44.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.731+0000 7fb4c7538b80 1 -- 172.21.15.43:0/1824836856 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bdb43dd90 con 0x557bdb5275f0 2024-05-27T07:16:44.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.731+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x557bdb5275f0 0x557bdb529a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.732+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x557bdb5275f0 0x557bdb529a10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ff81981cac50a79b server_cookie=45562c10a3767bfd in_seq=0 out_seq=0 2024-05-27T07:16:44.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.732+0000 7fb493e00700 10 client.15843 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:16:44.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.734+0000 7fb493e00700 1 -- 172.21.15.43:0/1824836856 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557bdb43dd90 con 0x557bdb5275f0 2024-05-27T07:16:44.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.734+0000 7fb493e00700 10 client.15843 handle_command_reply: tid=0 2024-05-27T07:16:44.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb4c7538b80 1 client.15843 shutdown 2024-05-27T07:16:44.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb4c7538b80 1 -- 172.21.15.43:0/1824836856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 msgr2=0x557bdb5175c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:44.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb5175c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0005d40 tx=0x7fb4a0009510 comp rx=0 tx=0).stop 2024-05-27T07:16:44.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb4c7538b80 1 -- 172.21.15.43:0/1824836856 shutdown_connections 2024-05-27T07:16:44.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x557bdb5275f0 0x557bdb529a10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb513d30 0x557bdb518040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb5175c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/1824836856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb50ddb0 0x557bdb517b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb4c7538b80 1 -- 172.21.15.43:0/1824836856 >> 172.21.15.43:0/1824836856 conn(0x557bdb3eefc0 msgr2=0x557bdb508580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:44.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb493e00700 0 client.15843 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:16:44.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.735+0000 7fb4c7538b80 1 -- 172.21.15.43:0/1824836856 shutdown_connections 2024-05-27T07:16:44.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.736+0000 7fb4c7538b80 1 -- 172.21.15.43:0/1824836856 wait complete. 2024-05-27T07:16:44.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.736+0000 7fb4c7538b80 20 client.15843 trim_cache size 0 max 16384 2024-05-27T07:16:44.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.755+0000 7fb4c7538b80 1 Processor -- start 2024-05-27T07:16:44.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.755+0000 7fb4c7538b80 1 -- start start 2024-05-27T07:16:44.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.755+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb38ee40 0x557bdb50d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.755+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb517fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.755+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb513d30 0x557bdb512d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.755+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b41b25e0 con 0x557bdb50ddb0 2024-05-27T07:16:44.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.755+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b41b41e0 con 0x557bdb38ee40 2024-05-27T07:16:44.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.755+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b41b2cb0 con 0x557bdb513d30 2024-05-27T07:16:44.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.756+0000 7fb4b1e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb517fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.756+0000 7fb4b1e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb517fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54532/0 (socket says 172.21.15.43:54532) 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.756+0000 7fb4b1e00700 1 -- 172.21.15.43:0/2142371619 learned_addr learned my addr 172.21.15.43:0/2142371619 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.756+0000 7fb4b1400700 1 --2- 172.21.15.43:0/2142371619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb38ee40 0x557bdb50d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.756+0000 7fb4b1400700 1 --2- 172.21.15.43:0/2142371619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb513d30 0x557bdb512d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.756+0000 7fb4b1e00700 1 -- 172.21.15.43:0/2142371619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb513d30 msgr2=0x557bdb512d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.756+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/2142371619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb513d30 0x557bdb512d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.756+0000 7fb4b1e00700 1 -- 172.21.15.43:0/2142371619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb38ee40 msgr2=0x557bdb50d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.756+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/2142371619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb38ee40 0x557bdb50d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.756+0000 7fb4b1e00700 1 -- 172.21.15.43:0/2142371619 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bdb51a3c0 con 0x557bdb50ddb0 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.757+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/2142371619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb517fb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4000c00 tx=0x7fb4a4013740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3eecbe7d569b40 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.757+0000 7fb49ea00700 1 -- 172.21.15.43:0/2142371619 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40130d0 con 0x557bdb50ddb0 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.757+0000 7fb49ea00700 1 -- 172.21.15.43:0/2142371619 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4a4013270 con 0x557bdb50ddb0 2024-05-27T07:16:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.757+0000 7fb49ea00700 1 -- 172.21.15.43:0/2142371619 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40038e0 con 0x557bdb50ddb0 2024-05-27T07:16:44.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.758+0000 7fb4c7538b80 1 -- 172.21.15.43:0/2142371619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 msgr2=0x557bdb517fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:44.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.758+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/2142371619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb517fb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4000c00 tx=0x7fb4a4013740 comp rx=0 tx=0).stop 2024-05-27T07:16:44.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.758+0000 7fb4c7538b80 1 -- 172.21.15.43:0/2142371619 shutdown_connections 2024-05-27T07:16:44.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.758+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/2142371619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb513d30 0x557bdb512d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.758+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/2142371619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb38ee40 0x557bdb50d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.758+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/2142371619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb50ddb0 0x557bdb517fb0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.758+0000 7fb4c7538b80 1 -- 172.21.15.43:0/2142371619 >> 172.21.15.43:0/2142371619 conn(0x557bdb3eefc0 msgr2=0x557bdb511140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:44.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.759+0000 7fb4c7538b80 1 -- 172.21.15.43:0/2142371619 shutdown_connections 2024-05-27T07:16:44.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.759+0000 7fb4c7538b80 1 -- 172.21.15.43:0/2142371619 wait complete. 2024-05-27T07:16:44.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.759+0000 7fb4c7538b80 1 -- start start 2024-05-27T07:16:44.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.759+0000 7fb4c7538b80 1 Processor -- start 2024-05-27T07:16:44.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.760+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb52cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.760+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb50ddb0 0x557bdb529600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.760+0000 7fb4c7538b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 0x557bdb529b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.760+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b41b3ab0 con 0x557bdb38ee40 2024-05-27T07:16:44.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.760+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b41b3380 con 0x557bdb513d30 2024-05-27T07:16:44.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.760+0000 7fb4c7538b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b41b0560 con 0x557bdb50ddb0 2024-05-27T07:16:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.760+0000 7fb4c7538b80 10 client.0 _ll_register_callbacks cb 0x557bdb3d5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.761+0000 7fb4b1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb52cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.761+0000 7fb4b1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb52cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54542/0 (socket says 172.21.15.43:54542) 2024-05-27T07:16:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.761+0000 7fb4b1400700 1 -- 172.21.15.43:0/3234378853 learned_addr learned my addr 172.21.15.43:0/3234378853 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.761+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb50ddb0 0x557bdb529600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.761+0000 7fb4b1400700 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 0x557bdb529b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.761+0000 7fb4b1400700 1 -- 172.21.15.43:0/3234378853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb50ddb0 msgr2=0x557bdb529600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.761+0000 7fb4b1400700 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb50ddb0 0x557bdb529600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.761+0000 7fb4b1400700 1 -- 172.21.15.43:0/3234378853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 msgr2=0x557bdb529b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.761+0000 7fb4b1400700 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 0x557bdb529b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.761+0000 7fb4b1400700 1 -- 172.21.15.43:0/3234378853 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bdb52a080 con 0x557bdb38ee40 2024-05-27T07:16:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.762+0000 7fb4b1400700 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb52cbb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0027c40 tx=0x7fb4a000bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3a6cd1ae9359b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.762+0000 7fb493e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.762+0000 7fb493e00700 1 -- 172.21.15.43:0/3234378853 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0027640 con 0x557bdb38ee40 2024-05-27T07:16:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.762+0000 7fb493e00700 1 -- 172.21.15.43:0/3234378853 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9408+0+0 (secure 0 0 0) 0x7fb4a00277e0 con 0x557bdb38ee40 2024-05-27T07:16:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.762+0000 7fb4c7538b80 1 -- 172.21.15.43:0/3234378853 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4a4013270 con 0x557bdb38ee40 2024-05-27T07:16:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.763+0000 7fb493e00700 1 -- 172.21.15.43:0/3234378853 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fb4a004f050 con 0x557bdb38ee40 2024-05-27T07:16:44.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.763+0000 7fb4c7538b80 10 client.15849 fetch_fsmap learned FSMap version 41 2024-05-27T07:16:44.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.763+0000 7fb4c7538b80 1 -- 172.21.15.43:0/3234378853 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x557bdb590af0 con 0x557bdb38ee40 2024-05-27T07:16:44.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.763+0000 7fb493e00700 1 -- 172.21.15.43:0/3234378853 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4a4013270 con 0x557bdb38ee40 2024-05-27T07:16:44.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.763+0000 7fb493e00700 1 -- 172.21.15.43:0/3234378853 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7fb4a004a030 con 0x557bdb38ee40 2024-05-27T07:16:44.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.764+0000 7fb4c7538b80 10 client.15849 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:16:44.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.764+0000 7fb4c7538b80 10 client.15849 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:16:44.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.764+0000 7fb4c7538b80 20 client.15849 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.764+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x557bdb591110 0x557bdb593530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.764+0000 7fb4c7538b80 4 client.15849 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:16:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.764+0000 7fb4c7538b80 1 -- 172.21.15.43:0/3234378853 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557bdb437570 con 0x557bdb591110 2024-05-27T07:16:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.765+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x557bdb591110 0x557bdb593530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:44.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.766+0000 7fb4b1e00700 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x557bdb591110 0x557bdb593530 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=97189494c125989a server_cookie=5145698e8315c8a7 in_seq=0 out_seq=0 2024-05-27T07:16:44.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.766+0000 7fb493e00700 10 client.15849 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:16:44.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb493e00700 1 -- 172.21.15.43:0/3234378853 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557bdb437570 con 0x557bdb591110 2024-05-27T07:16:44.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb493e00700 10 client.15849 handle_command_reply: tid=0 2024-05-27T07:16:44.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb4c7538b80 1 client.15849 shutdown 2024-05-27T07:16:44.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb4c7538b80 1 -- 172.21.15.43:0/3234378853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 msgr2=0x557bdb52cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:44.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb52cbb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0027c40 tx=0x7fb4a000bef0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb4c7538b80 1 -- 172.21.15.43:0/3234378853 shutdown_connections 2024-05-27T07:16:44.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x557bdb591110 0x557bdb593530 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557bdb513d30 0x557bdb529b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bdb38ee40 0x557bdb52cbb0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb4c7538b80 1 --2- 172.21.15.43:0/3234378853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557bdb50ddb0 0x557bdb529600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.767+0000 7fb4c7538b80 1 -- 172.21.15.43:0/3234378853 >> 172.21.15.43:0/3234378853 conn(0x557bdb3eefc0 msgr2=0x557bdb511b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.768+0000 7fb493e00700 0 client.15849 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:16:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.768+0000 7fb4c7538b80 1 -- 172.21.15.43:0/3234378853 shutdown_connections 2024-05-27T07:16:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.768+0000 7fb4c7538b80 1 -- 172.21.15.43:0/3234378853 wait complete. 2024-05-27T07:16:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.768+0000 7fb4c7538b80 20 client.15849 trim_cache size 0 max 16384 2024-05-27T07:16:44.761 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:16:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 -- 172.21.15.43:0/793129943 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4980b2a70 msgr2=0x7fb4980b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4980b2a70 0x7fb4980b4f80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0a53d0 tx=0x7fb4ac005b70 comp rx=0 tx=0).stop 2024-05-27T07:16:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 -- 172.21.15.43:0/793129943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b4110bd0 msgr2=0x7fb4b41a1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b4110bd0 0x7fb4b41a1790 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0094f0 tx=0x7fb4ac00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 -- 172.21.15.43:0/793129943 shutdown_connections 2024-05-27T07:16:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4980b2a70 0x7fb4980b4f80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b4110bd0 0x7fb4b41a1790 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4b4076c00 0x7fb4b419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 --2- 172.21.15.43:0/793129943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b4077550 0x7fb4b419d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 -- 172.21.15.43:0/793129943 >> 172.21.15.43:0/793129943 conn(0x7fb4b40ffe90 msgr2=0x7fb4b4100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 -- 172.21.15.43:0/793129943 shutdown_connections 2024-05-27T07:16:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:44.769+0000 7fb4b8a00700 1 -- 172.21.15.43:0/793129943 wait complete. 2024-05-27T07:16:44.788 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:44 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:16:44.788 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:44 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.535s CPU time. 2024-05-27T07:16:45.178 INFO:tasks.workunit.client.0.smithi043.stdout: 1 6528 1.07 MB/sec warmup 49 sec latency 53.355 ms 2024-05-27T07:16:45.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:45 smithi132 ceph-mon[35283]: pgmap v92: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 64 KiB/s wr, 26 op/s 2024-05-27T07:16:45.851 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:45 smithi043 ceph-mon[31531]: pgmap v92: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 64 KiB/s wr, 26 op/s 2024-05-27T07:16:45.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:16:45.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:45 smithi179 ceph-mon[35888]: pgmap v92: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 64 KiB/s wr, 26 op/s 2024-05-27T07:16:46.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 6743 1.07 MB/sec warmup 50 sec latency 55.170 ms 2024-05-27T07:16:46.350 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:47.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 6852 1.07 MB/sec warmup 51 sec latency 55.263 ms 2024-05-27T07:16:48.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 6881 1.05 MB/sec warmup 52 sec latency 55.876 ms 2024-05-27T07:16:48.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:48 smithi043 ceph-mon[31531]: pgmap v93: 129 pgs: 129 active+clean; 29 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 521 KiB/s wr, 58 op/s 2024-05-27T07:16:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:48 smithi132 ceph-mon[35283]: pgmap v93: 129 pgs: 129 active+clean; 29 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 521 KiB/s wr, 58 op/s 2024-05-27T07:16:48.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:48 smithi179 ceph-mon[35888]: pgmap v93: 129 pgs: 129 active+clean; 29 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 521 KiB/s wr, 58 op/s 2024-05-27T07:16:48.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.823+0000 7f9878a00700 1 -- 172.21.15.43:0/2623068590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9874115360 msgr2=0x7f9874117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:48.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.823+0000 7f9878a00700 1 --2- 172.21.15.43:0/2623068590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9874115360 0x7f9874117780 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f986c00b0a0 tx=0x7f986c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.823+0000 7f9878a00700 1 -- 172.21.15.43:0/2623068590 shutdown_connections 2024-05-27T07:16:48.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.823+0000 7f9878a00700 1 --2- 172.21.15.43:0/2623068590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9874115360 0x7f9874117780 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.823+0000 7f9878a00700 1 --2- 172.21.15.43:0/2623068590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9874076c60 0x7f9874077070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.823+0000 7f9878a00700 1 --2- 172.21.15.43:0/2623068590 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98740775b0 0x7f9874114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.823+0000 7f9878a00700 1 -- 172.21.15.43:0/2623068590 >> 172.21.15.43:0/2623068590 conn(0x7f98740ffeb0 msgr2=0x7f9874102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:48.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.823+0000 7f9878a00700 1 -- 172.21.15.43:0/2623068590 shutdown_connections 2024-05-27T07:16:48.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.823+0000 7f9878a00700 1 -- 172.21.15.43:0/2623068590 wait complete. 2024-05-27T07:16:48.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.825+0000 7f9878a00700 1 Processor -- start 2024-05-27T07:16:48.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.826+0000 7f9878a00700 1 -- start start 2024-05-27T07:16:48.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.826+0000 7f9878a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9874076c60 0x7f98741a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.826+0000 7f9878a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98740775b0 0x7f98741a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.826+0000 7f9878a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9874115360 0x7f98741a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.826+0000 7f9878a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f987411a910 con 0x7f9874115360 2024-05-27T07:16:48.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.826+0000 7f9878a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f987411a790 con 0x7f9874076c60 2024-05-27T07:16:48.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.826+0000 7f9878a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f987411aa90 con 0x7f98740775b0 2024-05-27T07:16:48.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.827+0000 7f9872800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9874115360 0x7f98741a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.827+0000 7f9872800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9874115360 0x7f98741a5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54554/0 (socket says 172.21.15.43:54554) 2024-05-27T07:16:48.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.827+0000 7f9872800700 1 -- 172.21.15.43:0/3859853310 learned_addr learned my addr 172.21.15.43:0/3859853310 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:48.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.827+0000 7f9871400700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98740775b0 0x7f98741a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.827+0000 7f9871e00700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9874076c60 0x7f98741a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.827+0000 7f9872800700 1 -- 172.21.15.43:0/3859853310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98740775b0 msgr2=0x7f98741a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:48.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.827+0000 7f9872800700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98740775b0 0x7f98741a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.827+0000 7f9872800700 1 -- 172.21.15.43:0/3859853310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9874076c60 msgr2=0x7f98741a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:48.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.827+0000 7f9872800700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9874076c60 0x7f98741a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.827+0000 7f9872800700 1 -- 172.21.15.43:0/3859853310 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98741a61d0 con 0x7f9874115360 2024-05-27T07:16:48.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.828+0000 7f9872800700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9874115360 0x7f98741a5b30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f986c00b830 tx=0x7f986c004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:48.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.828+0000 7f986aa00700 1 -- 172.21.15.43:0/3859853310 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c021070 con 0x7f9874115360 2024-05-27T07:16:48.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.828+0000 7f986aa00700 1 -- 172.21.15.43:0/3859853310 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f986c007c80 con 0x7f9874115360 2024-05-27T07:16:48.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.828+0000 7f986aa00700 1 -- 172.21.15.43:0/3859853310 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c005260 con 0x7f9874115360 2024-05-27T07:16:48.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.828+0000 7f9878a00700 1 -- 172.21.15.43:0/3859853310 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98741a6460 con 0x7f9874115360 2024-05-27T07:16:48.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.828+0000 7f9878a00700 1 -- 172.21.15.43:0/3859853310 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98741a68f0 con 0x7f9874115360 2024-05-27T07:16:48.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.832+0000 7f986aa00700 1 -- 172.21.15.43:0/3859853310 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f986c024070 con 0x7f9874115360 2024-05-27T07:16:48.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.833+0000 7f98875efb80 1 Processor -- start 2024-05-27T07:16:48.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.833+0000 7f98875efb80 1 -- start start 2024-05-27T07:16:48.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f98875efb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e982330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f98875efb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e803e40 0x55a23e982870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f98875efb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e982e70 0x55a23e98f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f98875efb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a23e8b0e40 con 0x55a23e982e70 2024-05-27T07:16:48.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f98875efb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a23e8a09e0 con 0x55a23e8ef9f0 2024-05-27T07:16:48.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f98875efb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a23e8a4a20 con 0x55a23e803e40 2024-05-27T07:16:48.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f986aa00700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98580b2a70 0x7f98580b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f986aa00700 1 -- 172.21.15.43:0/3859853310 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9408+0+0 (secure 0 0 0) 0x7f986c00f020 con 0x7f9874115360 2024-05-27T07:16:48.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f9871400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e982e70 0x55a23e98f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f9871400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e982e70 0x55a23e98f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54556/0 (socket says 172.21.15.43:54556) 2024-05-27T07:16:48.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f9871400700 1 -- 172.21.15.43:0/2952110441 learned_addr learned my addr 172.21.15.43:0/2952110441 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:48.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f9871e00700 1 --2- 172.21.15.43:0/2952110441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e803e40 0x55a23e982870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.834+0000 7f9872800700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98580b2a70 0x7f98580b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.835+0000 7f9871400700 1 --2- 172.21.15.43:0/2952110441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e982330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.835+0000 7f9871400700 1 -- 172.21.15.43:0/2952110441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e803e40 msgr2=0x55a23e982870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:48.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.835+0000 7f9871400700 1 --2- 172.21.15.43:0/2952110441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e803e40 0x55a23e982870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.835+0000 7f9871400700 1 -- 172.21.15.43:0/2952110441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 msgr2=0x55a23e982330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:48.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.835+0000 7f9871400700 1 --2- 172.21.15.43:0/2952110441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e982330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.835+0000 7f9871400700 1 -- 172.21.15.43:0/2952110441 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a23e869420 con 0x55a23e982e70 2024-05-27T07:16:48.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.835+0000 7f9872800700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98580b2a70 0x7f98580b4f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f986c0a70b0 tx=0x7f986c009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:48.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.835+0000 7f9871400700 1 --2- 172.21.15.43:0/2952110441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e982e70 0x55a23e98f110 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f98600149a0 tx=0x7f986000c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebc6124282bde5c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:48.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.836+0000 7f985f400700 1 -- 172.21.15.43:0/2952110441 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860012450 con 0x55a23e982e70 2024-05-27T07:16:48.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.836+0000 7f985f400700 1 -- 172.21.15.43:0/2952110441 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9860012a80 con 0x55a23e982e70 2024-05-27T07:16:48.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.836+0000 7f985f400700 1 -- 172.21.15.43:0/2952110441 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98600118d0 con 0x55a23e982e70 2024-05-27T07:16:48.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.837+0000 7f98875efb80 1 -- 172.21.15.43:0/2952110441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e982e70 msgr2=0x55a23e98f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:48.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.837+0000 7f98875efb80 1 --2- 172.21.15.43:0/2952110441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e982e70 0x55a23e98f110 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f98600149a0 tx=0x7f986000c4d0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.837+0000 7f98875efb80 1 -- 172.21.15.43:0/2952110441 shutdown_connections 2024-05-27T07:16:48.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.837+0000 7f98875efb80 1 --2- 172.21.15.43:0/2952110441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e982e70 0x55a23e98f110 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.837+0000 7f98875efb80 1 --2- 172.21.15.43:0/2952110441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e982330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.837+0000 7f98875efb80 1 --2- 172.21.15.43:0/2952110441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e803e40 0x55a23e982870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.837+0000 7f98875efb80 1 -- 172.21.15.43:0/2952110441 >> 172.21.15.43:0/2952110441 conn(0x55a23e89d390 msgr2=0x55a23e97e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:48.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.837+0000 7f98875efb80 1 -- 172.21.15.43:0/2952110441 shutdown_connections 2024-05-27T07:16:48.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.837+0000 7f98875efb80 1 -- 172.21.15.43:0/2952110441 wait complete. 2024-05-27T07:16:48.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.838+0000 7f98875efb80 1 -- start start 2024-05-27T07:16:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.838+0000 7f98875efb80 1 Processor -- start 2024-05-27T07:16:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f98875efb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e803e40 0x55a23e98c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f98875efb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e8ef9f0 0x55a23e98cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f98875efb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e982e70 0x55a23e98d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f98875efb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a23e7a6fa0 con 0x55a23e8ef9f0 2024-05-27T07:16:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f98875efb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a23e841760 con 0x55a23e982e70 2024-05-27T07:16:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f98875efb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a23e98fb80 con 0x55a23e803e40 2024-05-27T07:16:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f98875efb80 10 client.0 _ll_register_callbacks cb 0x55a23e84ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:48.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f9871e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e8ef9f0 0x55a23e98cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f9871e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e8ef9f0 0x55a23e98cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54566/0 (socket says 172.21.15.43:54566) 2024-05-27T07:16:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f9871e00700 1 -- 172.21.15.43:0/1273558298 learned_addr learned my addr 172.21.15.43:0/1273558298 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.839+0000 7f9871400700 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e803e40 0x55a23e98c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.840+0000 7f9871400700 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e982e70 0x55a23e98d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.840+0000 7f9871e00700 1 -- 172.21.15.43:0/1273558298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e803e40 msgr2=0x55a23e98c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:48.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.840+0000 7f9871e00700 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e803e40 0x55a23e98c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.840+0000 7f9871e00700 1 -- 172.21.15.43:0/1273558298 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e982e70 msgr2=0x55a23e98d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:48.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.840+0000 7f9871e00700 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e982e70 0x55a23e98d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.840+0000 7f9871e00700 1 -- 172.21.15.43:0/1273558298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a23e7f08d0 con 0x55a23e8ef9f0 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.840+0000 7f9871e00700 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e8ef9f0 0x55a23e98cbb0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f986400bfd0 tx=0x7f98640127e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c28d5b589555d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.840+0000 7f9853e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.841+0000 7f9853e00700 1 -- 172.21.15.43:0/1273558298 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864004440 con 0x55a23e8ef9f0 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.841+0000 7f9853e00700 1 -- 172.21.15.43:0/1273558298 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9408+0+0 (secure 0 0 0) 0x7f9864003680 con 0x55a23e8ef9f0 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.841+0000 7f98875efb80 1 -- 172.21.15.43:0/1273558298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9860012a80 con 0x55a23e8ef9f0 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.841+0000 7f9853e00700 1 -- 172.21.15.43:0/1273558298 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f9864003880 con 0x55a23e8ef9f0 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.841+0000 7f98875efb80 10 client.15861 fetch_fsmap learned FSMap version 41 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.841+0000 7f98875efb80 1 -- 172.21.15.43:0/1273558298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55a23e871060 con 0x55a23e8ef9f0 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.841+0000 7f9853e00700 1 -- 172.21.15.43:0/1273558298 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9860012a80 con 0x55a23e8ef9f0 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.842+0000 7f9853e00700 1 -- 172.21.15.43:0/1273558298 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7f9864019020 con 0x55a23e8ef9f0 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.842+0000 7f98875efb80 10 client.15861 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.842+0000 7f98875efb80 10 client.15861 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.842+0000 7f98875efb80 20 client.15861 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:48.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.842+0000 7f98875efb80 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55a23e99cbb0 0x55a23e99efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.842+0000 7f98875efb80 4 client.15861 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:16:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.842+0000 7f98875efb80 1 -- 172.21.15.43:0/1273558298 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a23e8b2d90 con 0x55a23e99cbb0 2024-05-27T07:16:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.843+0000 7f9871400700 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55a23e99cbb0 0x55a23e99efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.844+0000 7f9871400700 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55a23e99cbb0 0x55a23e99efd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7e0b7f685187173e server_cookie=f7ae805d6424c3b9 in_seq=0 out_seq=0 2024-05-27T07:16:48.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.844+0000 7f9853e00700 10 client.15861 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:16:48.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f9853e00700 1 -- 172.21.15.43:0/1273558298 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a23e8b2d90 con 0x55a23e99cbb0 2024-05-27T07:16:48.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f9853e00700 10 client.15861 handle_command_reply: tid=0 2024-05-27T07:16:48.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 client.15861 shutdown 2024-05-27T07:16:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 -- 172.21.15.43:0/1273558298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e8ef9f0 msgr2=0x55a23e98cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e8ef9f0 0x55a23e98cbb0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f986400bfd0 tx=0x7f98640127e0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 -- 172.21.15.43:0/1273558298 shutdown_connections 2024-05-27T07:16:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55a23e99cbb0 0x55a23e99efd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e982e70 0x55a23e98d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e803e40 0x55a23e98c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 --2- 172.21.15.43:0/1273558298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e8ef9f0 0x55a23e98cbb0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f9853e00700 0 client.15861 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:16:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 -- 172.21.15.43:0/1273558298 >> 172.21.15.43:0/1273558298 conn(0x55a23e89d390 msgr2=0x55a23e982370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:48.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 -- 172.21.15.43:0/1273558298 shutdown_connections 2024-05-27T07:16:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.846+0000 7f98875efb80 1 -- 172.21.15.43:0/1273558298 wait complete. 2024-05-27T07:16:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.847+0000 7f98875efb80 20 client.15861 trim_cache size 0 max 16384 2024-05-27T07:16:48.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.865+0000 7f98875efb80 1 Processor -- start 2024-05-27T07:16:48.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.865+0000 7f98875efb80 1 -- start start 2024-05-27T07:16:48.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.866+0000 7f98875efb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e987e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.866+0000 7f98875efb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e982790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.866+0000 7f98875efb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e982e70 0x55a23e9865f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.866+0000 7f98875efb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b6aa0 con 0x55a23e803e40 2024-05-27T07:16:48.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.866+0000 7f98875efb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b86a0 con 0x55a23e8ef9f0 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.866+0000 7f98875efb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b7170 con 0x55a23e982e70 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.866+0000 7f9871400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e987e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.866+0000 7f9871400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e987e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54572/0 (socket says 172.21.15.43:54572) 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.866+0000 7f9871400700 1 -- 172.21.15.43:0/1777326562 learned_addr learned my addr 172.21.15.43:0/1777326562 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.866+0000 7f9871e00700 1 --2- 172.21.15.43:0/1777326562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e982790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.867+0000 7f9871400700 1 --2- 172.21.15.43:0/1777326562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e982e70 0x55a23e9865f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.867+0000 7f9871400700 1 -- 172.21.15.43:0/1777326562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e982e70 msgr2=0x55a23e9865f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.867+0000 7f9871400700 1 --2- 172.21.15.43:0/1777326562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e982e70 0x55a23e9865f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.867+0000 7f9871400700 1 -- 172.21.15.43:0/1777326562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 msgr2=0x55a23e982790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.867+0000 7f9871400700 1 --2- 172.21.15.43:0/1777326562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e982790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.867+0000 7f9871400700 1 -- 172.21.15.43:0/1777326562 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a23e97f000 con 0x55a23e803e40 2024-05-27T07:16:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.867+0000 7f9871400700 1 --2- 172.21.15.43:0/1777326562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e987e60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9860014970 tx=0x7f986000c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97ed71c06c08f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.867+0000 7f985ea00700 1 -- 172.21.15.43:0/1777326562 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860012650 con 0x55a23e803e40 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.867+0000 7f985ea00700 1 -- 172.21.15.43:0/1777326562 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98600054d0 con 0x55a23e803e40 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.867+0000 7f985ea00700 1 -- 172.21.15.43:0/1777326562 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860004b00 con 0x55a23e803e40 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.868+0000 7f98875efb80 1 -- 172.21.15.43:0/1777326562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 msgr2=0x55a23e987e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.868+0000 7f98875efb80 1 --2- 172.21.15.43:0/1777326562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e987e60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9860014970 tx=0x7f986000c4d0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.868+0000 7f98875efb80 1 -- 172.21.15.43:0/1777326562 shutdown_connections 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.868+0000 7f98875efb80 1 --2- 172.21.15.43:0/1777326562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e982e70 0x55a23e9865f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.868+0000 7f98875efb80 1 --2- 172.21.15.43:0/1777326562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e987e60 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.868+0000 7f98875efb80 1 --2- 172.21.15.43:0/1777326562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e982790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.868+0000 7f98875efb80 1 -- 172.21.15.43:0/1777326562 >> 172.21.15.43:0/1777326562 conn(0x55a23e89d390 msgr2=0x55a23e985ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.869+0000 7f98875efb80 1 -- 172.21.15.43:0/1777326562 shutdown_connections 2024-05-27T07:16:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.869+0000 7f98875efb80 1 -- 172.21.15.43:0/1777326562 wait complete. 2024-05-27T07:16:48.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.869+0000 7f98875efb80 1 -- start start 2024-05-27T07:16:48.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.870+0000 7f98875efb80 1 Processor -- start 2024-05-27T07:16:48.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.870+0000 7f98875efb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e990430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.870+0000 7f98875efb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e990970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.870+0000 7f98875efb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e982e70 0x55a23ea05640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.870+0000 7f98875efb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b7f70 con 0x55a23e803e40 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.870+0000 7f98875efb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b7840 con 0x55a23e8ef9f0 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.870+0000 7f98875efb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b4a20 con 0x55a23e982e70 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.870+0000 7f98875efb80 10 client.0 _ll_register_callbacks cb 0x55a23e84ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.870+0000 7f9871400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e990430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.871+0000 7f9871400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e990430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54580/0 (socket says 172.21.15.43:54580) 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.871+0000 7f9871400700 1 -- 172.21.15.43:0/752637240 learned_addr learned my addr 172.21.15.43:0/752637240 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.871+0000 7f9871e00700 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e990970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.871+0000 7f9871400700 1 -- 172.21.15.43:0/752637240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e982e70 msgr2=0x55a23ea05640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.871+0000 7f9871400700 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e982e70 0x55a23ea05640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.871+0000 7f9871400700 1 -- 172.21.15.43:0/752637240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 msgr2=0x55a23e990970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.871+0000 7f9871400700 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e990970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.871+0000 7f9871400700 1 -- 172.21.15.43:0/752637240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a23ea05b80 con 0x55a23e803e40 2024-05-27T07:16:48.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.871+0000 7f9871400700 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e990430 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9860005750 tx=0x7f9860014310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa036d3105e175c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:48.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.871+0000 7f9853e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:48.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.872+0000 7f9853e00700 1 -- 172.21.15.43:0/752637240 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860034070 con 0x55a23e803e40 2024-05-27T07:16:48.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.872+0000 7f98875efb80 1 -- 172.21.15.43:0/752637240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98600054d0 con 0x55a23e803e40 2024-05-27T07:16:48.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.872+0000 7f9853e00700 1 -- 172.21.15.43:0/752637240 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9408+0+0 (secure 0 0 0) 0x7f9860004020 con 0x55a23e803e40 2024-05-27T07:16:48.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.872+0000 7f9853e00700 1 -- 172.21.15.43:0/752637240 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f986003d050 con 0x55a23e803e40 2024-05-27T07:16:48.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.872+0000 7f98875efb80 10 client.15867 fetch_fsmap learned FSMap version 41 2024-05-27T07:16:48.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.872+0000 7f98875efb80 1 -- 172.21.15.43:0/752637240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55a23ea06d00 con 0x55a23e803e40 2024-05-27T07:16:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.872+0000 7f9853e00700 1 -- 172.21.15.43:0/752637240 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98600054d0 con 0x55a23e803e40 2024-05-27T07:16:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.873+0000 7f9853e00700 1 -- 172.21.15.43:0/752637240 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7f9860037020 con 0x55a23e803e40 2024-05-27T07:16:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.873+0000 7f98875efb80 10 client.15867 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:16:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.873+0000 7f98875efb80 10 client.15867 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:16:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.873+0000 7f98875efb80 20 client.15867 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:48.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.873+0000 7f98875efb80 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55a23ea070f0 0x55a23ea09510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:48.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.873+0000 7f98875efb80 4 client.15867 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:16:48.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.873+0000 7f98875efb80 1 -- 172.21.15.43:0/752637240 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a23e8ac570 con 0x55a23ea070f0 2024-05-27T07:16:48.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.874+0000 7f9871e00700 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55a23ea070f0 0x55a23ea09510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:48.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.875+0000 7f9871e00700 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55a23ea070f0 0x55a23ea09510 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eb9534fe734350da server_cookie=5318a2262019acd1 in_seq=0 out_seq=0 2024-05-27T07:16:48.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.875+0000 7f9853e00700 10 client.15867 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:16:48.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f9853e00700 1 -- 172.21.15.43:0/752637240 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a23e8ac570 con 0x55a23ea070f0 2024-05-27T07:16:48.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f9853e00700 10 client.15867 handle_command_reply: tid=0 2024-05-27T07:16:48.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 client.15867 shutdown 2024-05-27T07:16:48.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 -- 172.21.15.43:0/752637240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 msgr2=0x55a23e990430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e990430 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9860005750 tx=0x7f9860014310 comp rx=0 tx=0).stop 2024-05-27T07:16:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 -- 172.21.15.43:0/752637240 shutdown_connections 2024-05-27T07:16:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a23e982e70 0x55a23ea05640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a23e803e40 0x55a23e990430 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55a23ea070f0 0x55a23ea09510 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 --2- 172.21.15.43:0/752637240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a23e8ef9f0 0x55a23e990970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 -- 172.21.15.43:0/752637240 >> 172.21.15.43:0/752637240 conn(0x55a23e89d390 msgr2=0x55a23e98c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f9853e00700 0 client.15867 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:16:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 -- 172.21.15.43:0/752637240 shutdown_connections 2024-05-27T07:16:48.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 1 -- 172.21.15.43:0/752637240 wait complete. 2024-05-27T07:16:48.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.876+0000 7f98875efb80 20 client.15867 trim_cache size 0 max 16384 2024-05-27T07:16:48.869 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:16:48.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.877+0000 7f9878a00700 1 -- 172.21.15.43:0/3859853310 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98580b2a70 msgr2=0x7f98580b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:48.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.877+0000 7f9878a00700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98580b2a70 0x7f98580b4f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f986c0a70b0 tx=0x7f986c009040 comp rx=0 tx=0).stop 2024-05-27T07:16:48.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.877+0000 7f9878a00700 1 -- 172.21.15.43:0/3859853310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9874115360 msgr2=0x7f98741a5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:48.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.877+0000 7f9878a00700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9874115360 0x7f98741a5b30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f986c00b830 tx=0x7f986c004db0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.877+0000 7f9878a00700 1 -- 172.21.15.43:0/3859853310 shutdown_connections 2024-05-27T07:16:48.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.877+0000 7f9878a00700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98580b2a70 0x7f98580b4f80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.877+0000 7f9878a00700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9874115360 0x7f98741a5b30 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.877+0000 7f9878a00700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9874076c60 0x7f98741a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.877+0000 7f9878a00700 1 --2- 172.21.15.43:0/3859853310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98740775b0 0x7f98741a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:48.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.877+0000 7f9878a00700 1 -- 172.21.15.43:0/3859853310 >> 172.21.15.43:0/3859853310 conn(0x7f98740ffeb0 msgr2=0x7f9874101fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:48.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.878+0000 7f9878a00700 1 -- 172.21.15.43:0/3859853310 shutdown_connections 2024-05-27T07:16:48.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:48.878+0000 7f9878a00700 1 -- 172.21.15.43:0/3859853310 wait complete. 2024-05-27T07:16:49.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 6948 1.03 MB/sec warmup 53 sec latency 51.346 ms 2024-05-27T07:16:49.798 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:16:50.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7135 1.07 MB/sec warmup 54 sec latency 50.433 ms 2024-05-27T07:16:50.310 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:50.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:50 smithi132 ceph-mon[35283]: pgmap v94: 129 pgs: 129 active+clean; 29 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 515 KiB/s wr, 57 op/s 2024-05-27T07:16:50.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:50 smithi043 ceph-mon[31531]: pgmap v94: 129 pgs: 129 active+clean; 29 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 515 KiB/s wr, 57 op/s 2024-05-27T07:16:50.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:50 smithi179 ceph-mon[35888]: pgmap v94: 129 pgs: 129 active+clean; 29 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 515 KiB/s wr, 57 op/s 2024-05-27T07:16:51.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7261 1.05 MB/sec warmup 55 sec latency 52.290 ms 2024-05-27T07:16:52.132 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:52 smithi043 ceph-mon[31531]: pgmap v95: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 536 KiB/s wr, 44 op/s 2024-05-27T07:16:52.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7349 1.05 MB/sec warmup 56 sec latency 57.105 ms 2024-05-27T07:16:52.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:52 smithi132 ceph-mon[35283]: pgmap v95: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 536 KiB/s wr, 44 op/s 2024-05-27T07:16:52.365 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:52 smithi179 ceph-mon[35888]: pgmap v95: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 536 KiB/s wr, 44 op/s 2024-05-27T07:16:52.365 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:51 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:16:52.365 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:51 smithi179 systemd[1]: Stopped Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:52.365 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:51 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.434s CPU time. 2024-05-27T07:16:52.365 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:51 smithi179 systemd[1]: Starting Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:16:52.708 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:52 smithi179 podman[71332]: 2024-05-27 07:16:52.280105302 +0000 UTC m=+0.037063665 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:52.709 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:52 smithi179 podman[71332]: 2024-05-27 07:16:52.54038573 +0000 UTC m=+0.297344134 container create 15be4a9744dcb85aaab1423d03a3f1ac10d3c70ab0fc8a660ae3a5c133ab4e2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T07:16:52.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.779+0000 7f2ba3e00700 1 -- 172.21.15.43:0/3206348909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ba410b890 msgr2=0x7f2ba41131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.779+0000 7f2ba3e00700 1 --2- 172.21.15.43:0/3206348909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ba410b890 0x7f2ba41131f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c0099c0 tx=0x7f2b9c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.780+0000 7f2ba3e00700 1 -- 172.21.15.43:0/3206348909 shutdown_connections 2024-05-27T07:16:52.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.780+0000 7f2ba3e00700 1 --2- 172.21.15.43:0/3206348909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ba410b890 0x7f2ba41131f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.780+0000 7f2ba3e00700 1 --2- 172.21.15.43:0/3206348909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba4106610 0x7f2ba4106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.780+0000 7f2ba3e00700 1 --2- 172.21.15.43:0/3206348909 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ba410af40 0x7f2ba410b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.780+0000 7f2ba3e00700 1 -- 172.21.15.43:0/3206348909 >> 172.21.15.43:0/3206348909 conn(0x7f2ba4102020 msgr2=0x7f2ba4104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:52.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.780+0000 7f2ba3e00700 1 -- 172.21.15.43:0/3206348909 shutdown_connections 2024-05-27T07:16:52.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.780+0000 7f2ba3e00700 1 -- 172.21.15.43:0/3206348909 wait complete. 2024-05-27T07:16:52.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.781+0000 7f2ba3e00700 1 Processor -- start 2024-05-27T07:16:52.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.781+0000 7f2ba3e00700 1 -- start start 2024-05-27T07:16:52.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.781+0000 7f2ba3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba4106610 0x7f2ba419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.782+0000 7f2ba3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ba410af40 0x7f2ba419d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.782+0000 7f2ba3e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ba410b890 0x7f2ba41a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.782+0000 7f2ba3e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba41163c0 con 0x7f2ba410af40 2024-05-27T07:16:52.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.782+0000 7f2ba3e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba4116240 con 0x7f2ba4106610 2024-05-27T07:16:52.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.782+0000 7f2ba3e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba4116540 con 0x7f2ba410b890 2024-05-27T07:16:52.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.782+0000 7f2b9a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ba410af40 0x7f2ba419d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.782+0000 7f2b9a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ba410af40 0x7f2ba419d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54594/0 (socket says 172.21.15.43:54594) 2024-05-27T07:16:52.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.782+0000 7f2b9a000700 1 -- 172.21.15.43:0/2979363862 learned_addr learned my addr 172.21.15.43:0/2979363862 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:52.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.782+0000 7f2ba3400700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ba410b890 0x7f2ba41a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.782+0000 7f2ba2a00700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba4106610 0x7f2ba419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.783+0000 7f2b9a000700 1 -- 172.21.15.43:0/2979363862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ba410b890 msgr2=0x7f2ba41a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.783+0000 7f2b9a000700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ba410b890 0x7f2ba41a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.783+0000 7f2b9a000700 1 -- 172.21.15.43:0/2979363862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba4106610 msgr2=0x7f2ba419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.783+0000 7f2b9a000700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba4106610 0x7f2ba419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.783+0000 7f2b9a000700 1 -- 172.21.15.43:0/2979363862 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba41a1be0 con 0x7f2ba410af40 2024-05-27T07:16:52.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.783+0000 7f2ba2a00700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba4106610 0x7f2ba419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:16:52.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.783+0000 7f2b9a000700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ba410af40 0x7f2ba419d2b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00d950 tx=0x7f2b8c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:52.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.783+0000 7f2ba8a00700 1 -- 172.21.15.43:0/2979363862 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c016070 con 0x7f2ba410af40 2024-05-27T07:16:52.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.784+0000 7f2ba8a00700 1 -- 172.21.15.43:0/2979363862 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b8c00de30 con 0x7f2ba410af40 2024-05-27T07:16:52.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.784+0000 7f2ba3e00700 1 -- 172.21.15.43:0/2979363862 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba41a1ed0 con 0x7f2ba410af40 2024-05-27T07:16:52.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.784+0000 7f2ba8a00700 1 -- 172.21.15.43:0/2979363862 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c005240 con 0x7f2ba410af40 2024-05-27T07:16:52.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.784+0000 7f2ba3e00700 1 -- 172.21.15.43:0/2979363862 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba41a97f0 con 0x7f2ba410af40 2024-05-27T07:16:52.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.788+0000 7f2ba8a00700 1 -- 172.21.15.43:0/2979363862 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f2b8c0049a0 con 0x7f2ba410af40 2024-05-27T07:16:52.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.790+0000 7f2bb88fbb80 1 Processor -- start 2024-05-27T07:16:52.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.790+0000 7f2bb88fbb80 1 -- start start 2024-05-27T07:16:52.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.790+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a72750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.790+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a72c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.790+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433a7e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.790+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f43399fe40 con 0x55f4339de9f0 2024-05-27T07:16:52.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.790+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f43398f9e0 con 0x55f4338f2e40 2024-05-27T07:16:52.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.790+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f433993a20 con 0x55f433a7c2c0 2024-05-27T07:16:52.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.791+0000 7f2ba2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a72c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.791+0000 7f2b9a000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433a7e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.791+0000 7f2ba3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a72750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.791+0000 7f2ba3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a72750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54602/0 (socket says 172.21.15.43:54602) 2024-05-27T07:16:52.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.791+0000 7f2ba3400700 1 -- 172.21.15.43:0/3675363977 learned_addr learned my addr 172.21.15.43:0/3675363977 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:52.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.791+0000 7f2ba8a00700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2b880b2a70 0x7f2b880b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.792+0000 7f2ba8a00700 1 -- 172.21.15.43:0/2979363862 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9408+0+0 (secure 0 0 0) 0x7f2b8c0e2170 con 0x7f2ba410af40 2024-05-27T07:16:52.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.792+0000 7f2b9a000700 1 -- 172.21.15.43:0/3675363977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 msgr2=0x55f433a72c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:52.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.792+0000 7f2b9a000700 1 --2- 172.21.15.43:0/3675363977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a72c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.792+0000 7f2b9a000700 1 -- 172.21.15.43:0/3675363977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 msgr2=0x55f433a72750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:52.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.792+0000 7f2b9a000700 1 --2- 172.21.15.43:0/3675363977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a72750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.792+0000 7f2b9a000700 1 -- 172.21.15.43:0/3675363977 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f433958420 con 0x55f433a7c2c0 2024-05-27T07:16:52.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.792+0000 7f2ba3400700 1 --2- 172.21.15.43:0/3675363977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a72750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:16:52.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.792+0000 7f2ba3400700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2b880b2a70 0x7f2b880b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.793+0000 7f2b9a000700 1 --2- 172.21.15.43:0/3675363977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433a7e6e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0df510 tx=0x7f2b8c01b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa2169d346d24bb1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:52.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.793+0000 7f2ba3400700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2b880b2a70 0x7f2b880b4f80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c006010 tx=0x7f2b9c009310 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:52.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.793+0000 7f2b98c00700 1 -- 172.21.15.43:0/3675363977 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c0a8810 con 0x55f433a7c2c0 2024-05-27T07:16:52.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.793+0000 7f2b98c00700 1 -- 172.21.15.43:0/3675363977 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b8c0a8bb0 con 0x55f433a7c2c0 2024-05-27T07:16:52.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.793+0000 7f2b98c00700 1 -- 172.21.15.43:0/3675363977 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c0b1a70 con 0x55f433a7c2c0 2024-05-27T07:16:52.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.795+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3675363977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 msgr2=0x55f433a7e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.795+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/3675363977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433a7e6e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0df510 tx=0x7f2b8c01b000 comp rx=0 tx=0).stop 2024-05-27T07:16:52.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.795+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3675363977 shutdown_connections 2024-05-27T07:16:52.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.795+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/3675363977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433a7e6e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.795+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/3675363977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a72750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.795+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/3675363977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a72c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.795+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3675363977 >> 172.21.15.43:0/3675363977 conn(0x55f43398c390 msgr2=0x55f433a6d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:52.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.795+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3675363977 shutdown_connections 2024-05-27T07:16:52.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.795+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3675363977 wait complete. 2024-05-27T07:16:52.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.796+0000 7f2bb88fbb80 1 -- start start 2024-05-27T07:16:52.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.796+0000 7f2bb88fbb80 1 Processor -- start 2024-05-27T07:16:52.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f4338f2e40 0x55f433a79210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4339de9f0 0x55f433a79750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f433a7c2c0 0x55f433a79c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f433895fa0 con 0x55f433a7c2c0 2024-05-27T07:16:52.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f433930760 con 0x55f4339de9f0 2024-05-27T07:16:52.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f433a7eda0 con 0x55f4338f2e40 2024-05-27T07:16:52.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2bb88fbb80 10 client.0 _ll_register_callbacks cb 0x55f433939c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:52.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2ba2a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f4338f2e40 0x55f433a79210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2ba2a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f4338f2e40 0x55f433a79210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:41830/0 (socket says 172.21.15.43:41830) 2024-05-27T07:16:52.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2ba2a00700 1 -- 172.21.15.43:0/4204429023 learned_addr learned my addr 172.21.15.43:0/4204429023 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:52.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2ba3400700 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f433a7c2c0 0x55f433a79c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.797+0000 7f2b9a000700 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4339de9f0 0x55f433a79750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.798+0000 7f2ba2a00700 1 -- 172.21.15.43:0/4204429023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4339de9f0 msgr2=0x55f433a79750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:52.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.798+0000 7f2ba2a00700 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4339de9f0 0x55f433a79750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.798+0000 7f2ba2a00700 1 -- 172.21.15.43:0/4204429023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f433a7c2c0 msgr2=0x55f433a79c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:52.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.798+0000 7f2ba2a00700 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f433a7c2c0 0x55f433a79c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.798+0000 7f2ba2a00700 1 -- 172.21.15.43:0/4204429023 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4338df8d0 con 0x55f4338f2e40 2024-05-27T07:16:52.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.798+0000 7f2ba3400700 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f433a7c2c0 0x55f433a79c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:16:52.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.799+0000 7f2ba2a00700 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f4338f2e40 0x55f433a79210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b94007b60 tx=0x7f2b94007e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a3d76d689947a12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:52.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.799+0000 7f2b91600700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:16:52.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.799+0000 7f2b91600700 1 -- 172.21.15.43:0/4204429023 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9401a020 con 0x55f4338f2e40 2024-05-27T07:16:52.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.799+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/4204429023 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b8c0a8bb0 con 0x55f4338f2e40 2024-05-27T07:16:52.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.799+0000 7f2b91600700 1 -- 172.21.15.43:0/4204429023 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9408+0+0 (secure 0 0 0) 0x7f2b940118d0 con 0x55f4338f2e40 2024-05-27T07:16:52.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.800+0000 7f2b91600700 1 -- 172.21.15.43:0/4204429023 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f2b94011ad0 con 0x55f4338f2e40 2024-05-27T07:16:52.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.800+0000 7f2bb88fbb80 10 client.25576 fetch_fsmap learned FSMap version 41 2024-05-27T07:16:52.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.800+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/4204429023 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f433960060 con 0x55f4338f2e40 2024-05-27T07:16:52.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.800+0000 7f2b91600700 1 -- 172.21.15.43:0/4204429023 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b8c0a8bb0 con 0x55f4338f2e40 2024-05-27T07:16:52.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.801+0000 7f2b91600700 1 -- 172.21.15.43:0/4204429023 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7f2b94004370 con 0x55f4338f2e40 2024-05-27T07:16:52.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.801+0000 7f2bb88fbb80 10 client.25576 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:16:52.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.801+0000 7f2bb88fbb80 10 client.25576 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:16:52.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.801+0000 7f2bb88fbb80 20 client.25576 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:52.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.801+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f433a8bc40 0x55f433a8e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.801+0000 7f2bb88fbb80 4 client.25576 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:16:52.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.801+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/4204429023 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4339a1d90 con 0x55f433a8bc40 2024-05-27T07:16:52.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.802+0000 7f2b9a000700 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f433a8bc40 0x55f433a8e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.803+0000 7f2b9a000700 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f433a8bc40 0x55f433a8e0e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f75429678f99b1b server_cookie=3dae5e0f8e8a2332 in_seq=0 out_seq=0 2024-05-27T07:16:52.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.803+0000 7f2b91600700 10 client.25576 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2b91600700 1 -- 172.21.15.43:0/4204429023 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f4339a1d90 con 0x55f433a8bc40 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2b91600700 10 client.25576 handle_command_reply: tid=0 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2bb88fbb80 1 client.25576 shutdown 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/4204429023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f4338f2e40 msgr2=0x55f433a79210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f4338f2e40 0x55f433a79210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b94007b60 tx=0x7f2b94007e70 comp rx=0 tx=0).stop 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/4204429023 shutdown_connections 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f433a8bc40 0x55f433a8e0e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f433a7c2c0 0x55f433a79c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f4338f2e40 0x55f433a79210 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2b91600700 0 client.25576 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:16:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/4204429023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4339de9f0 0x55f433a79750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/4204429023 >> 172.21.15.43:0/4204429023 conn(0x55f43398c390 msgr2=0x55f433a6d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:52.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.805+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/4204429023 shutdown_connections 2024-05-27T07:16:52.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.806+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/4204429023 wait complete. 2024-05-27T07:16:52.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.806+0000 7f2bb88fbb80 20 client.25576 trim_cache size 0 max 16384 2024-05-27T07:16:52.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.825+0000 7f2bb88fbb80 1 Processor -- start 2024-05-27T07:16:52.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.826+0000 7f2bb88fbb80 1 -- start start 2024-05-27T07:16:52.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.826+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a78530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.826+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a87e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.826+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433a938a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.826+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba41b2590 con 0x55f4339de9f0 2024-05-27T07:16:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.826+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba41b4190 con 0x55f4338f2e40 2024-05-27T07:16:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.826+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba41b2c60 con 0x55f433a7c2c0 2024-05-27T07:16:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.826+0000 7f2b9a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a87e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.827+0000 7f2b9a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a87e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54620/0 (socket says 172.21.15.43:54620) 2024-05-27T07:16:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.827+0000 7f2b9a000700 1 -- 172.21.15.43:0/506771648 learned_addr learned my addr 172.21.15.43:0/506771648 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:52.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.827+0000 7f2ba2a00700 1 --2- 172.21.15.43:0/506771648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a78530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.827+0000 7f2ba3400700 1 --2- 172.21.15.43:0/506771648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433a938a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.827+0000 7f2b9a000700 1 -- 172.21.15.43:0/506771648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 msgr2=0x55f433a938a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.827+0000 7f2b9a000700 1 --2- 172.21.15.43:0/506771648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433a938a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.827+0000 7f2b9a000700 1 -- 172.21.15.43:0/506771648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 msgr2=0x55f433a78530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.827+0000 7f2b9a000700 1 --2- 172.21.15.43:0/506771648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a78530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.827+0000 7f2b9a000700 1 -- 172.21.15.43:0/506771648 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f433a7a000 con 0x55f4339de9f0 2024-05-27T07:16:52.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.828+0000 7f2b9a000700 1 --2- 172.21.15.43:0/506771648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a87e40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0a8da0 tx=0x7f2b8c0ad040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4acc00fe05440d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:52.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.828+0000 7f2b93e00700 1 -- 172.21.15.43:0/506771648 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c0ad450 con 0x55f4339de9f0 2024-05-27T07:16:52.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.828+0000 7f2b93e00700 1 -- 172.21.15.43:0/506771648 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b8c0a8bb0 con 0x55f4339de9f0 2024-05-27T07:16:52.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.829+0000 7f2b93e00700 1 -- 172.21.15.43:0/506771648 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c0b1cb0 con 0x55f4339de9f0 2024-05-27T07:16:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.829+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/506771648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 msgr2=0x55f433a87e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.829+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/506771648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a87e40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0a8da0 tx=0x7f2b8c0ad040 comp rx=0 tx=0).stop 2024-05-27T07:16:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.829+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/506771648 shutdown_connections 2024-05-27T07:16:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.829+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/506771648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433a938a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.829+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/506771648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a78530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.829+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/506771648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a87e40 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.829+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/506771648 >> 172.21.15.43:0/506771648 conn(0x55f43398c390 msgr2=0x55f433a74840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:52.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.829+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/506771648 shutdown_connections 2024-05-27T07:16:52.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.829+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/506771648 wait complete. 2024-05-27T07:16:52.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.830+0000 7f2bb88fbb80 1 -- start start 2024-05-27T07:16:52.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.830+0000 7f2bb88fbb80 1 Processor -- start 2024-05-27T07:16:52.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a7f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a7f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2bb88fbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433afcf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba41b3a60 con 0x55f4339de9f0 2024-05-27T07:16:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba41b3330 con 0x55f4338f2e40 2024-05-27T07:16:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2bb88fbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba41b0510 con 0x55f433a7c2c0 2024-05-27T07:16:52.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2bb88fbb80 10 client.0 _ll_register_callbacks cb 0x55f433939c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2b9a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a7f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2b9a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a7f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54622/0 (socket says 172.21.15.43:54622) 2024-05-27T07:16:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2b9a000700 1 -- 172.21.15.43:0/3362744788 learned_addr learned my addr 172.21.15.43:0/3362744788 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.831+0000 7f2ba2a00700 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a7f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.832+0000 7f2ba3400700 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433afcf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.832+0000 7f2b9a000700 1 -- 172.21.15.43:0/3362744788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 msgr2=0x55f433afcf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.832+0000 7f2b9a000700 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433afcf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.832+0000 7f2b9a000700 1 -- 172.21.15.43:0/3362744788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 msgr2=0x55f433a7f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.832+0000 7f2b9a000700 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a7f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.832+0000 7f2b9a000700 1 -- 172.21.15.43:0/3362744788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f433a7fe60 con 0x55f4339de9f0 2024-05-27T07:16:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.832+0000 7f2ba3400700 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433afcf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:16:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.832+0000 7f2b9a000700 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a7f5d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0b20c0 tx=0x7f2b8c0a5720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32927da394a3b2ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.832+0000 7f2b91600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.833+0000 7f2b91600700 1 -- 172.21.15.43:0/3362744788 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c0ad450 con 0x55f4339de9f0 2024-05-27T07:16:52.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.833+0000 7f2b91600700 1 -- 172.21.15.43:0/3362744788 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9408+0+0 (secure 0 0 0) 0x7f2b8c0b1a70 con 0x55f4339de9f0 2024-05-27T07:16:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.833+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3362744788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b8c0a8bb0 con 0x55f4339de9f0 2024-05-27T07:16:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.833+0000 7f2b91600700 1 -- 172.21.15.43:0/3362744788 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f2b8c0b1c70 con 0x55f4339de9f0 2024-05-27T07:16:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.833+0000 7f2bb88fbb80 10 client.15885 fetch_fsmap learned FSMap version 41 2024-05-27T07:16:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.833+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3362744788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f433afe4c0 con 0x55f4339de9f0 2024-05-27T07:16:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.833+0000 7f2b91600700 1 -- 172.21.15.43:0/3362744788 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b8c0a8bb0 con 0x55f4339de9f0 2024-05-27T07:16:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.833+0000 7f2b91600700 1 -- 172.21.15.43:0/3362744788 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 41) ==== 8378+0+0 (secure 0 0 0) 0x7f2b8c009b40 con 0x55f4339de9f0 2024-05-27T07:16:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.834+0000 7f2bb88fbb80 10 client.15885 fetch_fsmap finished waiting for FSMap version 41 2024-05-27T07:16:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.834+0000 7f2bb88fbb80 10 client.15885 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:16:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.834+0000 7f2bb88fbb80 20 client.15885 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:52.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.834+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f433afe690 0x55f433b00ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:52.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.834+0000 7f2bb88fbb80 4 client.15885 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:16:52.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.834+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3362744788 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f43399b570 con 0x55f433afe690 2024-05-27T07:16:52.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.834+0000 7f2ba2a00700 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f433afe690 0x55f433b00ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:52.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.835+0000 7f2ba2a00700 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f433afe690 0x55f433b00ab0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cfb97d07f6d79e server_cookie=3cc3718993993185 in_seq=0 out_seq=0 2024-05-27T07:16:52.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.835+0000 7f2b91600700 10 client.15885 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:16:52.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2b91600700 1 -- 172.21.15.43:0/3362744788 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f43399b570 con 0x55f433afe690 2024-05-27T07:16:52.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2b91600700 10 client.15885 handle_command_reply: tid=0 2024-05-27T07:16:52.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 client.15885 shutdown 2024-05-27T07:16:52.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3362744788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 msgr2=0x55f433a7f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a7f5d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0b20c0 tx=0x7f2b8c0a5720 comp rx=0 tx=0).stop 2024-05-27T07:16:52.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3362744788 shutdown_connections 2024-05-27T07:16:52.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f433afe690 0x55f433b00ab0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f433a7c2c0 0x55f433afcf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f4338f2e40 0x55f433a7f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 --2- 172.21.15.43:0/3362744788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f4339de9f0 0x55f433a7f5d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2b91600700 0 client.15885 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:16:52.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3362744788 >> 172.21.15.43:0/3362744788 conn(0x55f43398c390 msgr2=0x55f433a71120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:52.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3362744788 shutdown_connections 2024-05-27T07:16:52.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.836+0000 7f2bb88fbb80 1 -- 172.21.15.43:0/3362744788 wait complete. 2024-05-27T07:16:52.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2bb88fbb80 20 client.15885 trim_cache size 0 max 16384 2024-05-27T07:16:52.830 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:16:52.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2ba3e00700 1 -- 172.21.15.43:0/2979363862 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2b880b2a70 msgr2=0x7f2b880b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2ba3e00700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2b880b2a70 0x7f2b880b4f80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c006010 tx=0x7f2b9c009310 comp rx=0 tx=0).stop 2024-05-27T07:16:52.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2ba3e00700 1 -- 172.21.15.43:0/2979363862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ba410af40 msgr2=0x7f2ba419d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:52.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2ba3e00700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ba410af40 0x7f2ba419d2b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00d950 tx=0x7f2b8c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:16:52.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2ba3e00700 1 -- 172.21.15.43:0/2979363862 shutdown_connections 2024-05-27T07:16:52.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2ba3e00700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2b880b2a70 0x7f2b880b4f80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2ba3e00700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ba410b890 0x7f2ba41a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2ba3e00700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba4106610 0x7f2ba419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:52.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2ba3e00700 1 --2- 172.21.15.43:0/2979363862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ba410af40 0x7f2ba419d2b0 secure :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00d950 tx=0x7f2b8c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:16:52.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.837+0000 7f2ba3e00700 1 -- 172.21.15.43:0/2979363862 >> 172.21.15.43:0/2979363862 conn(0x7f2ba4102020 msgr2=0x7f2ba410a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:52.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.838+0000 7f2ba3e00700 1 -- 172.21.15.43:0/2979363862 shutdown_connections 2024-05-27T07:16:52.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:52.838+0000 7f2ba3e00700 1 -- 172.21.15.43:0/2979363862 wait complete. 2024-05-27T07:16:53.140 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-mon[35888]: daemon mds.c restarted 2024-05-27T07:16:53.141 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:52 smithi179 podman[71332]: 2024-05-27 07:16:52.837327637 +0000 UTC m=+0.594286044 container init 15be4a9744dcb85aaab1423d03a3f1ac10d3c70ab0fc8a660ae3a5c133ab4e2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-27T07:16:53.141 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:52 smithi179 podman[71332]: 2024-05-27 07:16:52.848207468 +0000 UTC m=+0.605165901 container start 15be4a9744dcb85aaab1423d03a3f1ac10d3c70ab0fc8a660ae3a5c133ab4e2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-05-27T07:16:53.141 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:52 smithi179 ceph-mds[71351]: -- 172.21.15.179:0/2383580438 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558167d124e0 con 0x558167d5d180 2024-05-27T07:16:53.141 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:52 smithi179 ceph-mds[71351]: -- 172.21.15.179:0/2383580438 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558167d15a40 con 0x558167d5d180 2024-05-27T07:16:53.141 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: starting mds.c at 2024-05-27T07:16:53.141 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:52 smithi179 bash[71332]: 15be4a9744dcb85aaab1423d03a3f1ac10d3c70ab0fc8a660ae3a5c133ab4e2b 2024-05-27T07:16:53.141 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:52 smithi179 systemd[1]: Started Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:53.141 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f94a6800700 time 2024-05-27T07:16:53.149965+0000 2024-05-27T07:16:53.141 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:53.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:53 smithi043 ceph-mon[31531]: daemon mds.c restarted 2024-05-27T07:16:53.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7415 1.03 MB/sec warmup 57 sec latency 45.914 ms 2024-05-27T07:16:53.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:53 smithi132 ceph-mon[35283]: daemon mds.c restarted 2024-05-27T07:16:53.461 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f94bc3f5ba6] 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f94bc3f5dd1] 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55814583323d] 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55814581e971] 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 5: (SnapRealm::check_cache() const+0x13e) [0x55814581cfa0] 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55814581d651] 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5581457b2347] 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5581454ae87e] 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5581454b3e94] 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5581454be969] 2024-05-27T07:16:53.462 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 11: (MDLog::_replay_thread()+0x1f38) [0x55814585748c] 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5581458609cb] 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 13: (Thread::entry_wrapper()+0x43) [0x7f94bc3c472d] 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 14: (Thread::_entry_func(void*)+0xd) [0x7f94bc3c4749] 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f94baf681ca] 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 16: clone() 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.153+0000 7f94a6800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f94a6800700 time 2024-05-27T07:16:53.149965+0000 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f94bc3f5ba6] 2024-05-27T07:16:53.463 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f94bc3f5dd1] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55814583323d] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55814581e971] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 5: (SnapRealm::check_cache() const+0x13e) [0x55814581cfa0] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55814581d651] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5581457b2347] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5581454ae87e] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5581454b3e94] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5581454be969] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 11: (MDLog::_replay_thread()+0x1f38) [0x55814585748c] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5581458609cb] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 13: (Thread::entry_wrapper()+0x43) [0x7f94bc3c472d] 2024-05-27T07:16:53.465 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 14: (Thread::_entry_func(void*)+0xd) [0x7f94bc3c4749] 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f94baf681ca] 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 16: clone() 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: *** Caught signal (Aborted) ** 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: in thread 7f94a6800700 thread_name:md_log_replay 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f94baf72d20] 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2: gsignal() 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 3: abort() 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f94bc3f5ced] 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f94bc3f5dd1] 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55814583323d] 2024-05-27T07:16:53.466 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55814581e971] 2024-05-27T07:16:53.467 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 8: (SnapRealm::check_cache() const+0x13e) [0x55814581cfa0] 2024-05-27T07:16:53.467 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55814581d651] 2024-05-27T07:16:53.467 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5581457b2347] 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5581454ae87e] 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5581454b3e94] 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5581454be969] 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 14: (MDLog::_replay_thread()+0x1f38) [0x55814585748c] 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5581458609cb] 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 16: (Thread::entry_wrapper()+0x43) [0x7f94bc3c472d] 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 17: (Thread::_entry_func(void*)+0xd) [0x7f94bc3c4749] 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f94baf681ca] 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 19: clone() 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.158+0000 7f94a6800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: in thread 7f94a6800700 thread_name:md_log_replay 2024-05-27T07:16:53.468 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f94baf72d20] 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2: gsignal() 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 3: abort() 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f94bc3f5ced] 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f94bc3f5dd1] 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55814583323d] 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55814581e971] 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 8: (SnapRealm::check_cache() const+0x13e) [0x55814581cfa0] 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55814581d651] 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5581457b2347] 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5581454ae87e] 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5581454b3e94] 2024-05-27T07:16:53.469 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5581454be969] 2024-05-27T07:16:53.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 14: (MDLog::_replay_thread()+0x1f38) [0x55814585748c] 2024-05-27T07:16:53.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5581458609cb] 2024-05-27T07:16:53.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 16: (Thread::entry_wrapper()+0x43) [0x7f94bc3c472d] 2024-05-27T07:16:53.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 17: (Thread::_entry_func(void*)+0xd) [0x7f94bc3c4749] 2024-05-27T07:16:53.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f94baf681ca] 2024-05-27T07:16:53.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 19: clone() 2024-05-27T07:16:53.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:53.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.470 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: -1> 2024-05-27T07:16:53.153+0000 7f94a6800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f94a6800700 time 2024-05-27T07:16:53.149965+0000 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f94bc3f5ba6] 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f94bc3f5dd1] 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55814583323d] 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55814581e971] 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 5: (SnapRealm::check_cache() const+0x13e) [0x55814581cfa0] 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55814581d651] 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5581457b2347] 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5581454ae87e] 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5581454b3e94] 2024-05-27T07:16:53.471 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5581454be969] 2024-05-27T07:16:53.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 11: (MDLog::_replay_thread()+0x1f38) [0x55814585748c] 2024-05-27T07:16:53.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5581458609cb] 2024-05-27T07:16:53.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 13: (Thread::entry_wrapper()+0x43) [0x7f94bc3c472d] 2024-05-27T07:16:53.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 14: (Thread::_entry_func(void*)+0xd) [0x7f94bc3c4749] 2024-05-27T07:16:53.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f94baf681ca] 2024-05-27T07:16:53.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 16: clone() 2024-05-27T07:16:53.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 0> 2024-05-27T07:16:53.158+0000 7f94a6800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:53.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: in thread 7f94a6800700 thread_name:md_log_replay 2024-05-27T07:16:53.472 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f94baf72d20] 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2: gsignal() 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 3: abort() 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f94bc3f5ced] 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f94bc3f5dd1] 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55814583323d] 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55814581e971] 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 8: (SnapRealm::check_cache() const+0x13e) [0x55814581cfa0] 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55814581d651] 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5581457b2347] 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5581454ae87e] 2024-05-27T07:16:53.473 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5581454b3e94] 2024-05-27T07:16:53.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5581454be969] 2024-05-27T07:16:53.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 14: (MDLog::_replay_thread()+0x1f38) [0x55814585748c] 2024-05-27T07:16:53.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5581458609cb] 2024-05-27T07:16:53.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 16: (Thread::entry_wrapper()+0x43) [0x7f94bc3c472d] 2024-05-27T07:16:53.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 17: (Thread::_entry_func(void*)+0xd) [0x7f94bc3c4749] 2024-05-27T07:16:53.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f94baf681ca] 2024-05-27T07:16:53.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 19: clone() 2024-05-27T07:16:53.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:53.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.474 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: -1> 2024-05-27T07:16:53.153+0000 7f94a6800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f94a6800700 time 2024-05-27T07:16:53.149965+0000 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f94bc3f5ba6] 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f94bc3f5dd1] 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55814583323d] 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55814581e971] 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 5: (SnapRealm::check_cache() const+0x13e) [0x55814581cfa0] 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55814581d651] 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5581457b2347] 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5581454ae87e] 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5581454b3e94] 2024-05-27T07:16:53.475 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5581454be969] 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 11: (MDLog::_replay_thread()+0x1f38) [0x55814585748c] 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5581458609cb] 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 13: (Thread::entry_wrapper()+0x43) [0x7f94bc3c472d] 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 14: (Thread::_entry_func(void*)+0xd) [0x7f94bc3c4749] 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f94baf681ca] 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 16: clone() 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 0> 2024-05-27T07:16:53.158+0000 7f94a6800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: in thread 7f94a6800700 thread_name:md_log_replay 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:53.476 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f94baf72d20] 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2: gsignal() 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 3: abort() 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f94bc3f5ced] 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f94bc3f5dd1] 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55814583323d] 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55814581e971] 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 8: (SnapRealm::check_cache() const+0x13e) [0x55814581cfa0] 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55814581d651] 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5581457b2347] 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5581454ae87e] 2024-05-27T07:16:53.477 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5581454b3e94] 2024-05-27T07:16:53.478 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5581454be969] 2024-05-27T07:16:53.478 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 14: (MDLog::_replay_thread()+0x1f38) [0x55814585748c] 2024-05-27T07:16:53.478 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5581458609cb] 2024-05-27T07:16:53.478 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 16: (Thread::entry_wrapper()+0x43) [0x7f94bc3c472d] 2024-05-27T07:16:53.478 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 17: (Thread::_entry_func(void*)+0xd) [0x7f94bc3c4749] 2024-05-27T07:16:53.478 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f94baf681ca] 2024-05-27T07:16:53.478 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 19: clone() 2024-05-27T07:16:53.478 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:53.478 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71347]: 2024-05-27T07:16:53.478 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:53 smithi179 podman[71382]: 2024-05-27 07:16:53.276775355 +0000 UTC m=+0.035456270 container died 15be4a9744dcb85aaab1423d03a3f1ac10d3c70ab0fc8a660ae3a5c133ab4e2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:16:53.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:16:54.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7487 1.02 MB/sec warmup 58 sec latency 56.298 ms 2024-05-27T07:16:54.193 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:54 smithi043 ceph-mon[31531]: pgmap v96: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 533 KiB/s wr, 44 op/s 2024-05-27T07:16:54.194 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:54 smithi043 ceph-mon[31531]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:16:54.194 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:54 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6836/3266558594,v1:172.21.15.179:6837/3266558594] up:boot 2024-05-27T07:16:54.194 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:54 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:54.194 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:54.194 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:54 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:54.332 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:54 smithi132 ceph-mon[35283]: pgmap v96: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 533 KiB/s wr, 44 op/s 2024-05-27T07:16:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:54 smithi132 ceph-mon[35283]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:16:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:54 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6836/3266558594,v1:172.21.15.179:6837/3266558594] up:boot 2024-05-27T07:16:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:54 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:54.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:54 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:54.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:54 smithi179 ceph-mon[35888]: pgmap v96: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 533 KiB/s wr, 44 op/s 2024-05-27T07:16:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:54 smithi179 ceph-mon[35888]: osdmap e96: 12 total, 12 up, 12 in 2024-05-27T07:16:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:54 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6836/3266558594,v1:172.21.15.179:6837/3266558594] up:boot 2024-05-27T07:16:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:54 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:16:54.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:54 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:54.459 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:54 smithi179 podman[71382]: 2024-05-27 07:16:54.192054684 +0000 UTC m=+0.950735630 container remove 15be4a9744dcb85aaab1423d03a3f1ac10d3c70ab0fc8a660ae3a5c133ab4e2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T07:16:54.459 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:54 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:16:55.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7553 1.00 MB/sec warmup 59 sec latency 37.774 ms 2024-05-27T07:16:55.202 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:54 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:16:55.202 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:54 smithi043 systemd[1]: Stopped Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:55.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:54 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.535s CPU time. 2024-05-27T07:16:55.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:54 smithi043 systemd[1]: Starting Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:16:55.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:54 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:16:55.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:16:54 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.575s CPU time. 2024-05-27T07:16:56.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7640 0.99 MB/sec warmup 60 sec latency 53.021 ms 2024-05-27T07:16:56.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:56 smithi043 ceph-mon[31531]: pgmap v98: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 536 KiB/s wr, 44 op/s 2024-05-27T07:16:56.202 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:55 smithi043 podman[96796]: 2024-05-27 07:16:55.861765542 +0000 UTC m=+0.036476315 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:16:56.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:55 smithi043 podman[96796]: 2024-05-27 07:16:55.971033666 +0000 UTC m=+0.145744457 container create 138dec11e441d78d34b378135de5149590c06ccfcefc8b798132f5d7fe7cc438 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T07:16:56.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:56 smithi132 ceph-mon[35283]: pgmap v98: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 536 KiB/s wr, 44 op/s 2024-05-27T07:16:56.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:56 smithi179 ceph-mon[35888]: pgmap v98: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 536 KiB/s wr, 44 op/s 2024-05-27T07:16:56.533 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:56 smithi043 podman[96796]: 2024-05-27 07:16:56.310047107 +0000 UTC m=+0.484757916 container init 138dec11e441d78d34b378135de5149590c06ccfcefc8b798132f5d7fe7cc438 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-27T07:16:56.533 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:56 smithi043 podman[96796]: 2024-05-27 07:16:56.321393295 +0000 UTC m=+0.496104086 container start 138dec11e441d78d34b378135de5149590c06ccfcefc8b798132f5d7fe7cc438 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-27T07:16:56.533 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:56 smithi043 ceph-mds[96832]: -- 172.21.15.43:0/3219265713 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5601d8c75a40 con 0x5601d8cbd600 2024-05-27T07:16:56.533 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: starting mds.g at 2024-05-27T07:16:56.871 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:56 smithi043 bash[96796]: 138dec11e441d78d34b378135de5149590c06ccfcefc8b798132f5d7fe7cc438 2024-05-27T07:16:56.872 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:56 smithi043 systemd[1]: Started Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:16:57.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:57 smithi043 ceph-mon[31531]: daemon mds.g restarted 2024-05-27T07:16:57.151 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe32ec00700 time 2024-05-27T07:16:57.104376+0000 2024-05-27T07:16:57.151 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe344807ba6] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe344807dd1] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5601a5eec23d] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5601a5ed7971] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 5: (SnapRealm::check_cache() const+0x13e) [0x5601a5ed5fa0] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5601a5ed6651] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5601a5e6b347] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5601a5b6787e] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5601a5b6ce94] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5601a5b77969] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 11: (MDLog::_replay_thread()+0x1f38) [0x5601a5f1048c] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5601a5f199cb] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 13: (Thread::entry_wrapper()+0x43) [0x7fe3447d672d] 2024-05-27T07:16:57.152 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe3447d6749] 2024-05-27T07:16:57.153 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe34337a1ca] 2024-05-27T07:16:57.153 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 16: clone() 2024-05-27T07:16:57.153 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: *** Caught signal (Aborted) ** 2024-05-27T07:16:57.153 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: in thread 7fe32ec00700 thread_name:md_log_replay 2024-05-27T07:16:57.153 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.106+0000 7fe32ec00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe32ec00700 time 2024-05-27T07:16:57.104376+0000 2024-05-27T07:16:57.153 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:57.153 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.153 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe344807ba6] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe344807dd1] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5601a5eec23d] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5601a5ed7971] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 5: (SnapRealm::check_cache() const+0x13e) [0x5601a5ed5fa0] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5601a5ed6651] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5601a5e6b347] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5601a5b6787e] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5601a5b6ce94] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5601a5b77969] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 11: (MDLog::_replay_thread()+0x1f38) [0x5601a5f1048c] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5601a5f199cb] 2024-05-27T07:16:57.154 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 13: (Thread::entry_wrapper()+0x43) [0x7fe3447d672d] 2024-05-27T07:16:57.155 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe3447d6749] 2024-05-27T07:16:57.155 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe34337a1ca] 2024-05-27T07:16:57.155 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 16: clone() 2024-05-27T07:16:57.155 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.155 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:57.155 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe343384d20] 2024-05-27T07:16:57.155 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2: gsignal() 2024-05-27T07:16:57.155 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 3: abort() 2024-05-27T07:16:57.155 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe344807ced] 2024-05-27T07:16:57.155 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe344807dd1] 2024-05-27T07:16:57.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.160+0000 7f7ddda00700 1 -- 172.21.15.43:0/2117122544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dd8108f10 msgr2=0x7f7dd8109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.160+0000 7f7ddda00700 1 --2- 172.21.15.43:0/2117122544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dd8108f10 0x7f7dd8109320 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40099c0 tx=0x7f7dc4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.160+0000 7f7ddda00700 1 -- 172.21.15.43:0/2117122544 shutdown_connections 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.160+0000 7f7ddda00700 1 --2- 172.21.15.43:0/2117122544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd8109860 0x7f7dd81111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.160+0000 7f7ddda00700 1 --2- 172.21.15.43:0/2117122544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7dd81045e0 0x7f7dd81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.160+0000 7f7ddda00700 1 --2- 172.21.15.43:0/2117122544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dd8108f10 0x7f7dd8109320 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.160+0000 7f7ddda00700 1 -- 172.21.15.43:0/2117122544 >> 172.21.15.43:0/2117122544 conn(0x7f7dd8100030 msgr2=0x7f7dd8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.160+0000 7f7ddda00700 1 -- 172.21.15.43:0/2117122544 shutdown_connections 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.160+0000 7f7ddda00700 1 -- 172.21.15.43:0/2117122544 wait complete. 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.162+0000 7f7ddda00700 1 Processor -- start 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.162+0000 7f7ddda00700 1 -- start start 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.163+0000 7f7ddda00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd81045e0 0x7f7dd819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.163+0000 7f7ddda00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dd8108f10 0x7f7dd819d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.163+0000 7f7ddda00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7dd8109860 0x7f7dd81a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.163+0000 7f7ddda00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd8114370 con 0x7f7dd8108f10 2024-05-27T07:16:57.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.163+0000 7f7ddda00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd81141f0 con 0x7f7dd8109860 2024-05-27T07:16:57.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.163+0000 7f7ddda00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd81144f0 con 0x7f7dd81045e0 2024-05-27T07:16:57.157 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5601a5eec23d] 2024-05-27T07:16:57.157 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5601a5ed7971] 2024-05-27T07:16:57.157 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 8: (SnapRealm::check_cache() const+0x13e) [0x5601a5ed5fa0] 2024-05-27T07:16:57.157 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5601a5ed6651] 2024-05-27T07:16:57.157 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5601a5e6b347] 2024-05-27T07:16:57.157 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5601a5b6787e] 2024-05-27T07:16:57.157 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5601a5b6ce94] 2024-05-27T07:16:57.157 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5601a5b77969] 2024-05-27T07:16:57.157 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 14: (MDLog::_replay_thread()+0x1f38) [0x5601a5f1048c] 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5601a5f199cb] 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 16: (Thread::entry_wrapper()+0x43) [0x7fe3447d672d] 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe3447d6749] 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe34337a1ca] 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 19: clone() 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.109+0000 7fe32ec00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: in thread 7fe32ec00700 thread_name:md_log_replay 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe343384d20] 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2: gsignal() 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 3: abort() 2024-05-27T07:16:57.158 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe344807ced] 2024-05-27T07:16:57.159 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe344807dd1] 2024-05-27T07:16:57.159 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5601a5eec23d] 2024-05-27T07:16:57.159 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5601a5ed7971] 2024-05-27T07:16:57.159 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 8: (SnapRealm::check_cache() const+0x13e) [0x5601a5ed5fa0] 2024-05-27T07:16:57.159 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5601a5ed6651] 2024-05-27T07:16:57.159 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5601a5e6b347] 2024-05-27T07:16:57.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.163+0000 7f7dd6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dd8108f10 0x7f7dd819d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.159 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5601a5b6787e] 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5601a5b6ce94] 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5601a5b77969] 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 14: (MDLog::_replay_thread()+0x1f38) [0x5601a5f1048c] 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5601a5f199cb] 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 16: (Thread::entry_wrapper()+0x43) [0x7fe3447d672d] 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe3447d6749] 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe34337a1ca] 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 19: clone() 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: -1> 2024-05-27T07:16:57.106+0000 7fe32ec00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe32ec00700 time 2024-05-27T07:16:57.104376+0000 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:57.160 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.161 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:57.161 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe344807ba6] 2024-05-27T07:16:57.161 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe344807dd1] 2024-05-27T07:16:57.161 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5601a5eec23d] 2024-05-27T07:16:57.161 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5601a5ed7971] 2024-05-27T07:16:57.161 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 5: (SnapRealm::check_cache() const+0x13e) [0x5601a5ed5fa0] 2024-05-27T07:16:57.161 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5601a5ed6651] 2024-05-27T07:16:57.161 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5601a5e6b347] 2024-05-27T07:16:57.161 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5601a5b6787e] 2024-05-27T07:16:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.163+0000 7f7dd6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dd8108f10 0x7f7dd819d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43250/0 (socket says 172.21.15.43:43250) 2024-05-27T07:16:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.163+0000 7f7dd6000700 1 -- 172.21.15.43:0/1398889280 learned_addr learned my addr 172.21.15.43:0/1398889280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.163+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd81045e0 0x7f7dd819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.164+0000 7f7dd7400700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7dd8109860 0x7f7dd81a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.164+0000 7f7dd6a00700 1 -- 172.21.15.43:0/1398889280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7dd8109860 msgr2=0x7f7dd81a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.164+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7dd8109860 0x7f7dd81a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.162 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5601a5b6ce94] 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5601a5b77969] 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 11: (MDLog::_replay_thread()+0x1f38) [0x5601a5f1048c] 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5601a5f199cb] 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 13: (Thread::entry_wrapper()+0x43) [0x7fe3447d672d] 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe3447d6749] 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe34337a1ca] 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 16: clone() 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 0> 2024-05-27T07:16:57.109+0000 7fe32ec00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: in thread 7fe32ec00700 thread_name:md_log_replay 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe343384d20] 2024-05-27T07:16:57.163 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2: gsignal() 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 3: abort() 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe344807ced] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe344807dd1] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5601a5eec23d] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5601a5ed7971] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 8: (SnapRealm::check_cache() const+0x13e) [0x5601a5ed5fa0] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5601a5ed6651] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5601a5e6b347] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5601a5b6787e] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5601a5b6ce94] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5601a5b77969] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 14: (MDLog::_replay_thread()+0x1f38) [0x5601a5f1048c] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5601a5f199cb] 2024-05-27T07:16:57.164 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 16: (Thread::entry_wrapper()+0x43) [0x7fe3447d672d] 2024-05-27T07:16:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.164+0000 7f7dd6a00700 1 -- 172.21.15.43:0/1398889280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dd8108f10 msgr2=0x7f7dd819d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.165+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dd8108f10 0x7f7dd819d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.165+0000 7f7dd6a00700 1 -- 172.21.15.43:0/1398889280 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd81a1ef0 con 0x7f7dd81045e0 2024-05-27T07:16:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.165+0000 7f7dd6000700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dd8108f10 0x7f7dd819d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:16:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.165+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd81045e0 0x7f7dd819d020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800c970 tx=0x7f7dc800cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe3447d6749] 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe34337a1ca] 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 19: clone() 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: -1> 2024-05-27T07:16:57.106+0000 7fe32ec00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe32ec00700 time 2024-05-27T07:16:57.104376+0000 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe344807ba6] 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe344807dd1] 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5601a5eec23d] 2024-05-27T07:16:57.166 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5601a5ed7971] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 5: (SnapRealm::check_cache() const+0x13e) [0x5601a5ed5fa0] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5601a5ed6651] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5601a5e6b347] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5601a5b6787e] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5601a5b6ce94] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5601a5b77969] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 11: (MDLog::_replay_thread()+0x1f38) [0x5601a5f1048c] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5601a5f199cb] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 13: (Thread::entry_wrapper()+0x43) [0x7fe3447d672d] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe3447d6749] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe34337a1ca] 2024-05-27T07:16:57.167 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 16: clone() 2024-05-27T07:16:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.166+0000 7f7dcf400700 1 -- 172.21.15.43:0/1398889280 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8004a30 con 0x7f7dd81045e0 2024-05-27T07:16:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.166+0000 7f7ddda00700 1 -- 172.21.15.43:0/1398889280 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd81a21e0 con 0x7f7dd81045e0 2024-05-27T07:16:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.166+0000 7f7ddda00700 1 -- 172.21.15.43:0/1398889280 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd81a9b00 con 0x7f7dd81045e0 2024-05-27T07:16:57.168 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.168 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 0> 2024-05-27T07:16:57.109+0000 7fe32ec00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:16:57.168 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: in thread 7fe32ec00700 thread_name:md_log_replay 2024-05-27T07:16:57.168 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.168 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe343384d20] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2: gsignal() 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 3: abort() 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe344807ced] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe344807dd1] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5601a5eec23d] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5601a5ed7971] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 8: (SnapRealm::check_cache() const+0x13e) [0x5601a5ed5fa0] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5601a5ed6651] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5601a5e6b347] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5601a5b6787e] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5601a5b6ce94] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5601a5b77969] 2024-05-27T07:16:57.169 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 14: (MDLog::_replay_thread()+0x1f38) [0x5601a5f1048c] 2024-05-27T07:16:57.170 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5601a5f199cb] 2024-05-27T07:16:57.170 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 16: (Thread::entry_wrapper()+0x43) [0x7fe3447d672d] 2024-05-27T07:16:57.170 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe3447d6749] 2024-05-27T07:16:57.170 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe34337a1ca] 2024-05-27T07:16:57.170 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 19: clone() 2024-05-27T07:16:57.170 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:16:57.170 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[96825]: 2024-05-27T07:16:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.166+0000 7f7dcf400700 1 -- 172.21.15.43:0/1398889280 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7dc80054b0 con 0x7f7dd81045e0 2024-05-27T07:16:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.166+0000 7f7dcf400700 1 -- 172.21.15.43:0/1398889280 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc800eae0 con 0x7f7dd81045e0 2024-05-27T07:16:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.172+0000 7f7dcf400700 1 -- 172.21.15.43:0/1398889280 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7dc800ec80 con 0x7f7dd81045e0 2024-05-27T07:16:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.173+0000 7f7dec538b80 1 Processor -- start 2024-05-27T07:16:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.174+0000 7f7dec538b80 1 -- start start 2024-05-27T07:16:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.174+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef7f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.174+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef79870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.174+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef79db0 0x55a16ef7a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.174+0000 7f7dec538b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a16eea7e40 con 0x55a16ef7fd30 2024-05-27T07:16:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.174+0000 7f7dec538b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a16ee979e0 con 0x55a16ef79db0 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.174+0000 7f7dec538b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a16ee9ba20 con 0x55a16edfae40 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.175+0000 7f7dd6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef79870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.175+0000 7f7dd6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef79870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43264/0 (socket says 172.21.15.43:43264) 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.175+0000 7f7dd6a00700 1 -- 172.21.15.43:0/679033750 learned_addr learned my addr 172.21.15.43:0/679033750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.175+0000 7f7dd6000700 1 --2- 172.21.15.43:0/679033750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef79db0 0x55a16ef7a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.175+0000 7f7dd7400700 1 --2- 172.21.15.43:0/679033750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef7f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.175+0000 7f7dd6a00700 1 -- 172.21.15.43:0/679033750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 msgr2=0x55a16ef7f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.175+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/679033750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef7f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.175+0000 7f7dd6a00700 1 -- 172.21.15.43:0/679033750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef79db0 msgr2=0x55a16ef7a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.176+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/679033750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef79db0 0x55a16ef7a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.176+0000 7f7dd6a00700 1 -- 172.21.15.43:0/679033750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a16ee60420 con 0x55a16ef7fd30 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.176+0000 7f7dd6000700 1 --2- 172.21.15.43:0/679033750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef79db0 0x55a16ef7a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:16:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.176+0000 7f7dcf400700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7dbc0b2af0 0x7f7dbc0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.176+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/679033750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef79870 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80df590 tx=0x7f7dc80ea3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7846381eb33a6b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.176+0000 7f7dd7400700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7dbc0b2af0 0x7f7dbc0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.176+0000 7f7dcf400700 1 -- 172.21.15.43:0/1398889280 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9408+0+0 (secure 0 0 0) 0x7f7dc80e22b0 con 0x7f7dd81045e0 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.177+0000 7f7dc3e00700 1 -- 172.21.15.43:0/679033750 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc80a50a0 con 0x55a16ef7fd30 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.178+0000 7f7dc3e00700 1 -- 172.21.15.43:0/679033750 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7dc80a8cf0 con 0x55a16ef7fd30 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.178+0000 7f7dc3e00700 1 -- 172.21.15.43:0/679033750 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc80a8950 con 0x55a16ef7fd30 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.179+0000 7f7dec538b80 1 -- 172.21.15.43:0/679033750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 msgr2=0x55a16ef79870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.179+0000 7f7dd7400700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7dbc0b2af0 0x7f7dbc0b5000 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x55a16ef7ab30 tx=0x7f7dd000c040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.179+0000 7f7dec538b80 1 --2- 172.21.15.43:0/679033750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef79870 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80df590 tx=0x7f7dc80ea3f0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.179+0000 7f7dec538b80 1 -- 172.21.15.43:0/679033750 shutdown_connections 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.179+0000 7f7dec538b80 1 --2- 172.21.15.43:0/679033750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef79db0 0x55a16ef7a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.179+0000 7f7dec538b80 1 --2- 172.21.15.43:0/679033750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef7f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.179+0000 7f7dec538b80 1 --2- 172.21.15.43:0/679033750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef79870 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.179+0000 7f7dec538b80 1 -- 172.21.15.43:0/679033750 >> 172.21.15.43:0/679033750 conn(0x55a16ee5afc0 msgr2=0x55a16ef748d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.179+0000 7f7dec538b80 1 -- 172.21.15.43:0/679033750 shutdown_connections 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.179+0000 7f7dec538b80 1 -- 172.21.15.43:0/679033750 wait complete. 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.180+0000 7f7dec538b80 1 -- start start 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.180+0000 7f7dec538b80 1 Processor -- start 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.181+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16edfae40 0x55a16ef83520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.181+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16ef79db0 0x55a16ef83a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.181+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef83fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.181+0000 7f7dec538b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a16ed9dfa0 con 0x55a16ef7fd30 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.181+0000 7f7dec538b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a16ee38760 con 0x55a16edfae40 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.181+0000 7f7dec538b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a16ef86800 con 0x55a16ef79db0 2024-05-27T07:16:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.181+0000 7f7dec538b80 10 client.0 _ll_register_callbacks cb 0x55a16ee41c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.182+0000 7f7dd6a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16edfae40 0x55a16ef83520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.182+0000 7f7dd6a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16edfae40 0x55a16ef83520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:34084/0 (socket says 172.21.15.43:34084) 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.182+0000 7f7dd6a00700 1 -- 172.21.15.43:0/2016145703 learned_addr learned my addr 172.21.15.43:0/2016145703 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.182+0000 7f7dd6000700 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16ef79db0 0x55a16ef83a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.182+0000 7f7dd7400700 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef83fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.183+0000 7f7dd7400700 1 -- 172.21.15.43:0/2016145703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16ef79db0 msgr2=0x55a16ef83a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.183+0000 7f7dd7400700 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16ef79db0 0x55a16ef83a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.183+0000 7f7dd7400700 1 -- 172.21.15.43:0/2016145703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16edfae40 msgr2=0x55a16ef83520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.183+0000 7f7dd7400700 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16edfae40 0x55a16ef83520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.183+0000 7f7dd7400700 1 -- 172.21.15.43:0/2016145703 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a16ede78d0 con 0x55a16ef7fd30 2024-05-27T07:16:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.183+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16edfae40 0x55a16ef83520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:16:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.184+0000 7f7dd7400700 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef83fa0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0016e10 tx=0x7f7dd0004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434d706114e56aac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.184+0000 7f7dc0c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.184+0000 7f7dc0c00700 1 -- 172.21.15.43:0/2016145703 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd001d050 con 0x55a16ef7fd30 2024-05-27T07:16:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.184+0000 7f7dc0c00700 1 -- 172.21.15.43:0/2016145703 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9408+0+0 (secure 0 0 0) 0x7f7dd0022440 con 0x55a16ef7fd30 2024-05-27T07:16:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.185+0000 7f7dec538b80 1 -- 172.21.15.43:0/2016145703 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dc80a8cf0 con 0x55a16ef7fd30 2024-05-27T07:16:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.185+0000 7f7dc0c00700 1 -- 172.21.15.43:0/2016145703 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f7dd0030050 con 0x55a16ef7fd30 2024-05-27T07:16:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.185+0000 7f7dec538b80 10 client.15909 fetch_fsmap learned FSMap version 45 2024-05-27T07:16:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.185+0000 7f7dec538b80 1 -- 172.21.15.43:0/2016145703 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a16ee68060 con 0x55a16ef7fd30 2024-05-27T07:16:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.186+0000 7f7dc0c00700 1 -- 172.21.15.43:0/2016145703 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7dc80a8cf0 con 0x55a16ef7fd30 2024-05-27T07:16:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.186+0000 7f7dc0c00700 1 -- 172.21.15.43:0/2016145703 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7f7dd002b3d0 con 0x55a16ef7fd30 2024-05-27T07:16:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.187+0000 7f7dec538b80 10 client.15909 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:16:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.187+0000 7f7dec538b80 10 client.15909 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:16:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.187+0000 7f7dec538b80 20 client.15909 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.187+0000 7f7dec538b80 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a16ef93820 0x55a16ef95c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.187+0000 7f7dec538b80 4 client.15909 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:16:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.187+0000 7f7dec538b80 1 -- 172.21.15.43:0/2016145703 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a16eea9d90 con 0x55a16ef93820 2024-05-27T07:16:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.188+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a16ef93820 0x55a16ef95c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.180 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7714 0.97 MB/sec warmup 61 sec latency 30.028 ms 2024-05-27T07:16:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.189+0000 7f7dc0c00700 10 client.15909 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:16:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.190+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a16ef93820 0x55a16ef95c40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=21085c76d3bc02ef server_cookie=cf78ede55ab49a25 in_seq=0 out_seq=0 2024-05-27T07:16:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.192+0000 7f7dc0c00700 1 -- 172.21.15.43:0/2016145703 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a16eea9d90 con 0x55a16ef93820 2024-05-27T07:16:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.192+0000 7f7dc0c00700 10 client.15909 handle_command_reply: tid=0 2024-05-27T07:16:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.192+0000 7f7dec538b80 1 client.15909 shutdown 2024-05-27T07:16:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.192+0000 7f7dec538b80 1 -- 172.21.15.43:0/2016145703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 msgr2=0x55a16ef83fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.192+0000 7f7dec538b80 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef83fa0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0016e10 tx=0x7f7dd0004b40 comp rx=0 tx=0).stop 2024-05-27T07:16:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.193+0000 7f7dec538b80 1 -- 172.21.15.43:0/2016145703 shutdown_connections 2024-05-27T07:16:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.193+0000 7f7dec538b80 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a16ef93820 0x55a16ef95c40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.193+0000 7f7dec538b80 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef7fd30 0x55a16ef83fa0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.193+0000 7f7dec538b80 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16edfae40 0x55a16ef83520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.193+0000 7f7dec538b80 1 --2- 172.21.15.43:0/2016145703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16ef79db0 0x55a16ef83a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.193+0000 7f7dec538b80 1 -- 172.21.15.43:0/2016145703 >> 172.21.15.43:0/2016145703 conn(0x55a16ee5afc0 msgr2=0x55a16ef74580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.193+0000 7f7dc0c00700 0 client.15909 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:16:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.194+0000 7f7dec538b80 1 -- 172.21.15.43:0/2016145703 shutdown_connections 2024-05-27T07:16:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.194+0000 7f7dec538b80 1 -- 172.21.15.43:0/2016145703 wait complete. 2024-05-27T07:16:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.194+0000 7f7dec538b80 20 client.15909 trim_cache size 0 max 16384 2024-05-27T07:16:57.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.216+0000 7f7dec538b80 1 Processor -- start 2024-05-27T07:16:57.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.216+0000 7f7dec538b80 1 -- start start 2024-05-27T07:16:57.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.216+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef79870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.216+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef79db0 0x55a16ef85880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.216+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 0x55a16ef83300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.216+0000 7f7dec538b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7db800ee20 con 0x55a16ef79db0 2024-05-27T07:16:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.216+0000 7f7dec538b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7db8010a20 con 0x55a16ef7fd30 2024-05-27T07:16:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.216+0000 7f7dec538b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7db800f4f0 con 0x55a16edfae40 2024-05-27T07:16:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef79db0 0x55a16ef85880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef79db0 0x55a16ef85880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43282/0 (socket says 172.21.15.43:43282) 2024-05-27T07:16:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd6000700 1 -- 172.21.15.43:0/2043102093 learned_addr learned my addr 172.21.15.43:0/2043102093 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:57.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd7400700 1 --2- 172.21.15.43:0/2043102093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 0x55a16ef83300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/2043102093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef79870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd6000700 1 -- 172.21.15.43:0/2043102093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 msgr2=0x55a16ef79870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd6000700 1 --2- 172.21.15.43:0/2043102093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef79870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd6000700 1 -- 172.21.15.43:0/2043102093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 msgr2=0x55a16ef83300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd6000700 1 --2- 172.21.15.43:0/2043102093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 0x55a16ef83300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd6000700 1 -- 172.21.15.43:0/2043102093 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a16ef85dc0 con 0x55a16ef79db0 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.217+0000 7f7dd7400700 1 --2- 172.21.15.43:0/2043102093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 0x55a16ef83300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.218+0000 7f7dd6000700 1 --2- 172.21.15.43:0/2043102093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef79db0 0x55a16ef85880 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40058e0 tx=0x7f7dc4005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766c5be62e5b45d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.218+0000 7f7dc3400700 1 -- 172.21.15.43:0/2043102093 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4038020 con 0x55a16ef79db0 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.218+0000 7f7dc3400700 1 -- 172.21.15.43:0/2043102093 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7dc4005600 con 0x55a16ef79db0 2024-05-27T07:16:57.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.218+0000 7f7dc3400700 1 -- 172.21.15.43:0/2043102093 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc400f460 con 0x55a16ef79db0 2024-05-27T07:16:57.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.219+0000 7f7dec538b80 1 -- 172.21.15.43:0/2043102093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef79db0 msgr2=0x55a16ef85880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.219+0000 7f7dec538b80 1 --2- 172.21.15.43:0/2043102093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef79db0 0x55a16ef85880 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40058e0 tx=0x7f7dc4005040 comp rx=0 tx=0).stop 2024-05-27T07:16:57.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.220+0000 7f7dec538b80 1 -- 172.21.15.43:0/2043102093 shutdown_connections 2024-05-27T07:16:57.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.220+0000 7f7dec538b80 1 --2- 172.21.15.43:0/2043102093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 0x55a16ef83300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.220+0000 7f7dec538b80 1 --2- 172.21.15.43:0/2043102093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef79870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.220+0000 7f7dec538b80 1 --2- 172.21.15.43:0/2043102093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef79db0 0x55a16ef85880 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.220+0000 7f7dec538b80 1 -- 172.21.15.43:0/2043102093 >> 172.21.15.43:0/2043102093 conn(0x55a16ee5afc0 msgr2=0x55a16ef821a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:57.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.220+0000 7f7dec538b80 1 -- 172.21.15.43:0/2043102093 shutdown_connections 2024-05-27T07:16:57.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.220+0000 7f7dec538b80 1 -- 172.21.15.43:0/2043102093 wait complete. 2024-05-27T07:16:57.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.220+0000 7f7dec538b80 1 -- start start 2024-05-27T07:16:57.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.220+0000 7f7dec538b80 1 Processor -- start 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.221+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef8baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.221+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 0x55a16ef8c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.221+0000 7f7dec538b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef9a410 0x55a16ef9a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.221+0000 7f7dec538b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7db80102f0 con 0x55a16ef9a410 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.221+0000 7f7dec538b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7db800fbc0 con 0x55a16ef7fd30 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.221+0000 7f7dec538b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7db800cd60 con 0x55a16edfae40 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.221+0000 7f7dec538b80 10 client.0 _ll_register_callbacks cb 0x55a16ee41c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.221+0000 7f7dd7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef9a410 0x55a16ef9a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.221+0000 7f7dd7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef9a410 0x55a16ef9a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43290/0 (socket says 172.21.15.43:43290) 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.222+0000 7f7dd7400700 1 -- 172.21.15.43:0/983899357 learned_addr learned my addr 172.21.15.43:0/983899357 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.222+0000 7f7dd6000700 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 0x55a16ef8c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.222+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef8baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.222+0000 7f7dd7400700 1 -- 172.21.15.43:0/983899357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 msgr2=0x55a16ef8baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.222+0000 7f7dd7400700 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef8baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.222+0000 7f7dd7400700 1 -- 172.21.15.43:0/983899357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 msgr2=0x55a16ef8c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:16:57.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.222+0000 7f7dd7400700 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 0x55a16ef8c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.222+0000 7f7dd7400700 1 -- 172.21.15.43:0/983899357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a16ef8c5a0 con 0x55a16ef9a410 2024-05-27T07:16:57.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.222+0000 7f7dd7400700 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef9a410 0x55a16ef9a820 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7dd002b990 tx=0x7f7dd0008db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38087287850d2e71 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:16:57.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.222+0000 7f7dc0c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:16:57.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.223+0000 7f7dc0c00700 1 -- 172.21.15.43:0/983899357 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd003a070 con 0x55a16ef9a410 2024-05-27T07:16:57.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.223+0000 7f7dec538b80 1 -- 172.21.15.43:0/983899357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dc4005600 con 0x55a16ef9a410 2024-05-27T07:16:57.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.223+0000 7f7dc0c00700 1 -- 172.21.15.43:0/983899357 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9408+0+0 (secure 0 0 0) 0x7f7dd0005e60 con 0x55a16ef9a410 2024-05-27T07:16:57.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.223+0000 7f7dc0c00700 1 -- 172.21.15.43:0/983899357 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f7dd0047050 con 0x55a16ef9a410 2024-05-27T07:16:57.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.223+0000 7f7dec538b80 10 client.15915 fetch_fsmap learned FSMap version 45 2024-05-27T07:16:57.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.223+0000 7f7dec538b80 1 -- 172.21.15.43:0/983899357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a16effdb00 con 0x55a16ef9a410 2024-05-27T07:16:57.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.224+0000 7f7dc0c00700 1 -- 172.21.15.43:0/983899357 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7dc4005600 con 0x55a16ef9a410 2024-05-27T07:16:57.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.224+0000 7f7dc0c00700 1 -- 172.21.15.43:0/983899357 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7f7dd0022ae0 con 0x55a16ef9a410 2024-05-27T07:16:57.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.224+0000 7f7dec538b80 10 client.15915 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:16:57.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.224+0000 7f7dec538b80 10 client.15915 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:16:57.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.224+0000 7f7dec538b80 20 client.15915 populate_metadata read hostname 'smithi043' 2024-05-27T07:16:57.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.224+0000 7f7dec538b80 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a16effe270 0x55a16f000690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:16:57.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.224+0000 7f7dec538b80 4 client.15915 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:16:57.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.224+0000 7f7dec538b80 1 -- 172.21.15.43:0/983899357 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a16eea3570 con 0x55a16effe270 2024-05-27T07:16:57.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.225+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a16effe270 0x55a16f000690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:16:57.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.226+0000 7f7dd6a00700 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a16effe270 0x55a16f000690 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2098d733a24e341f server_cookie=bcc1234064e100f3 in_seq=0 out_seq=0 2024-05-27T07:16:57.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.226+0000 7f7dc0c00700 10 client.15915 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:16:57.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.226+0000 7f7dc0c00700 1 -- 172.21.15.43:0/983899357 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a16eea3570 con 0x55a16effe270 2024-05-27T07:16:57.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.226+0000 7f7dc0c00700 10 client.15915 handle_command_reply: tid=0 2024-05-27T07:16:57.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.226+0000 7f7dec538b80 1 client.15915 shutdown 2024-05-27T07:16:57.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.226+0000 7f7dec538b80 1 -- 172.21.15.43:0/983899357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef9a410 msgr2=0x55a16ef9a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.226+0000 7f7dec538b80 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef9a410 0x55a16ef9a820 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7dd002b990 tx=0x7f7dd0008db0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.227+0000 7f7dec538b80 1 -- 172.21.15.43:0/983899357 shutdown_connections 2024-05-27T07:16:57.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.227+0000 7f7dec538b80 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a16effe270 0x55a16f000690 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.227+0000 7f7dec538b80 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a16ef9a410 0x55a16ef9a820 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.227+0000 7f7dec538b80 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a16edfae40 0x55a16ef8baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.227+0000 7f7dec538b80 1 --2- 172.21.15.43:0/983899357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a16ef7fd30 0x55a16ef8c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.227+0000 7f7dec538b80 1 -- 172.21.15.43:0/983899357 >> 172.21.15.43:0/983899357 conn(0x55a16ee5afc0 msgr2=0x55a16ef821a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:57.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.227+0000 7f7dc0c00700 0 client.15915 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:16:57.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.227+0000 7f7dec538b80 1 -- 172.21.15.43:0/983899357 shutdown_connections 2024-05-27T07:16:57.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.227+0000 7f7dec538b80 1 -- 172.21.15.43:0/983899357 wait complete. 2024-05-27T07:16:57.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.227+0000 7f7dec538b80 20 client.15915 trim_cache size 0 max 16384 2024-05-27T07:16:57.219 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:16:57.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.228+0000 7f7ddda00700 1 -- 172.21.15.43:0/1398889280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7dbc0b2af0 msgr2=0x7f7dbc0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.228+0000 7f7ddda00700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7dbc0b2af0 0x7f7dbc0b5000 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x55a16ef7ab30 tx=0x7f7dd000c040 comp rx=0 tx=0).stop 2024-05-27T07:16:57.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.228+0000 7f7ddda00700 1 -- 172.21.15.43:0/1398889280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd81045e0 msgr2=0x7f7dd819d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:16:57.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.228+0000 7f7ddda00700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd81045e0 0x7f7dd819d020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800c970 tx=0x7f7dc800cc80 comp rx=0 tx=0).stop 2024-05-27T07:16:57.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.228+0000 7f7ddda00700 1 -- 172.21.15.43:0/1398889280 shutdown_connections 2024-05-27T07:16:57.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.228+0000 7f7ddda00700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7dbc0b2af0 0x7f7dbc0b5000 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.228+0000 7f7ddda00700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7dd8109860 0x7f7dd81a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.228+0000 7f7ddda00700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd81045e0 0x7f7dd819d020 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.228+0000 7f7ddda00700 1 --2- 172.21.15.43:0/1398889280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7dd8108f10 0x7f7dd819d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:16:57.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.228+0000 7f7ddda00700 1 -- 172.21.15.43:0/1398889280 >> 172.21.15.43:0/1398889280 conn(0x7f7dd8100030 msgr2=0x7f7dd810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:16:57.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.229+0000 7f7ddda00700 1 -- 172.21.15.43:0/1398889280 shutdown_connections 2024-05-27T07:16:57.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:16:57.229+0000 7f7ddda00700 1 -- 172.21.15.43:0/1398889280 wait complete. 2024-05-27T07:16:57.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:57 smithi132 ceph-mon[35283]: daemon mds.g restarted 2024-05-27T07:16:57.452 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:57 smithi043 podman[96872]: 2024-05-27 07:16:57.201179706 +0000 UTC m=+0.045405127 container died 138dec11e441d78d34b378135de5149590c06ccfcefc8b798132f5d7fe7cc438 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-27T07:16:57.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:57 smithi179 ceph-mon[35888]: daemon mds.g restarted 2024-05-27T07:16:58.179 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7831 0.96 MB/sec warmup 62 sec latency 65.799 ms 2024-05-27T07:16:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:58 smithi132 ceph-mon[35283]: pgmap v99: 129 pgs: 129 active+clean; 31 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 363 KiB/s wr, 46 op/s 2024-05-27T07:16:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:58 smithi132 ceph-mon[35283]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:16:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:58 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6838/1651750867,v1:172.21.15.43:6839/1651750867] up:boot 2024-05-27T07:16:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:58 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:58 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:16:58.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:16:58 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:58.371 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:58 smithi043 podman[96872]: 2024-05-27 07:16:58.196179894 +0000 UTC m=+1.040405362 container remove 138dec11e441d78d34b378135de5149590c06ccfcefc8b798132f5d7fe7cc438 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-05-27T07:16:58.371 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:58 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:16:58.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:58 smithi043 ceph-mon[31531]: pgmap v99: 129 pgs: 129 active+clean; 31 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 363 KiB/s wr, 46 op/s 2024-05-27T07:16:58.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:58 smithi043 ceph-mon[31531]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:16:58.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:58 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6838/1651750867,v1:172.21.15.43:6839/1651750867] up:boot 2024-05-27T07:16:58.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:58 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:58.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:58 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:16:58.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:16:58 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:58 smithi179 ceph-mon[35888]: pgmap v99: 129 pgs: 129 active+clean; 31 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 363 KiB/s wr, 46 op/s 2024-05-27T07:16:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:58 smithi179 ceph-mon[35888]: osdmap e97: 12 total, 12 up, 12 in 2024-05-27T07:16:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:58 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6838/1651750867,v1:172.21.15.43:6839/1651750867] up:boot 2024-05-27T07:16:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:58 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:16:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:58 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:16:58.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:16:58 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:16:58.840 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:16:59.180 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7975 0.95 MB/sec warmup 63 sec latency 51.671 ms 2024-05-27T07:16:59.336 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:16:59.452 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:59 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:16:59.453 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:16:59 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.590s CPU time. 2024-05-27T07:17:00.180 INFO:tasks.workunit.client.0.smithi043.stdout: 1 8090 0.94 MB/sec warmup 64 sec latency 55.403 ms 2024-05-27T07:17:00.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:00 smithi043 ceph-mon[31531]: pgmap v101: 129 pgs: 129 active+clean; 31 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 240 KiB/s wr, 33 op/s 2024-05-27T07:17:00.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:00 smithi132 ceph-mon[35283]: pgmap v101: 129 pgs: 129 active+clean; 31 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 240 KiB/s wr, 33 op/s 2024-05-27T07:17:00.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:00 smithi179 ceph-mon[35888]: pgmap v101: 129 pgs: 129 active+clean; 31 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 240 KiB/s wr, 33 op/s 2024-05-27T07:17:01.180 INFO:tasks.workunit.client.0.smithi043.stdout: 1 8210 0.94 MB/sec warmup 65 sec latency 76.103 ms 2024-05-27T07:17:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.763+0000 7f21c2a00700 1 -- 172.21.15.43:0/2805721553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21bc1109a0 msgr2=0x7f21bc112e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.763+0000 7f21c2a00700 1 --2- 172.21.15.43:0/2805721553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21bc1109a0 0x7f21bc112e00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f21b400b0a0 tx=0x7f21b400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.763+0000 7f21c2a00700 1 -- 172.21.15.43:0/2805721553 shutdown_connections 2024-05-27T07:17:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.763+0000 7f21c2a00700 1 --2- 172.21.15.43:0/2805721553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21bc1109a0 0x7f21bc112e00 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.763+0000 7f21c2a00700 1 --2- 172.21.15.43:0/2805721553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21bc075d20 0x7f21bc076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.763+0000 7f21c2a00700 1 --2- 172.21.15.43:0/2805721553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21bc076670 0x7f21bc110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.763+0000 7f21c2a00700 1 -- 172.21.15.43:0/2805721553 >> 172.21.15.43:0/2805721553 conn(0x7f21bc079d00 msgr2=0x7f21bc07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:01.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.763+0000 7f21c2a00700 1 -- 172.21.15.43:0/2805721553 shutdown_connections 2024-05-27T07:17:01.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.763+0000 7f21c2a00700 1 -- 172.21.15.43:0/2805721553 wait complete. 2024-05-27T07:17:01.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.764+0000 7f21c2a00700 1 Processor -- start 2024-05-27T07:17:01.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.764+0000 7f21c2a00700 1 -- start start 2024-05-27T07:17:01.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.765+0000 7f21c2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21bc075d20 0x7f21bc19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.765+0000 7f21c2a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21bc076670 0x7f21bc19d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.765+0000 7f21c2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21bc1109a0 0x7f21bc1a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.765+0000 7f21c2a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc116330 con 0x7f21bc075d20 2024-05-27T07:17:01.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.765+0000 7f21c2a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1161b0 con 0x7f21bc1109a0 2024-05-27T07:17:01.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.765+0000 7f21c2a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1164b0 con 0x7f21bc076670 2024-05-27T07:17:01.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.765+0000 7f21c1600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21bc075d20 0x7f21bc19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.765+0000 7f21c1600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21bc075d20 0x7f21bc19cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43308/0 (socket says 172.21.15.43:43308) 2024-05-27T07:17:01.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.765+0000 7f21c1600700 1 -- 172.21.15.43:0/1572936109 learned_addr learned my addr 172.21.15.43:0/1572936109 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:01.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.766+0000 7f21c2000700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21bc1109a0 0x7f21bc1a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.766+0000 7f21c0c00700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21bc076670 0x7f21bc19d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.766+0000 7f21c1600700 1 -- 172.21.15.43:0/1572936109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21bc076670 msgr2=0x7f21bc19d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.766+0000 7f21c1600700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21bc076670 0x7f21bc19d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.766+0000 7f21c1600700 1 -- 172.21.15.43:0/1572936109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21bc1109a0 msgr2=0x7f21bc1a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.766+0000 7f21c1600700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21bc1109a0 0x7f21bc1a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.766+0000 7f21c1600700 1 -- 172.21.15.43:0/1572936109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21bc1a1ba0 con 0x7f21bc075d20 2024-05-27T07:17:01.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.766+0000 7f21c1600700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21bc075d20 0x7f21bc19cd80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f21b000d950 tx=0x7f21b000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:01.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.767+0000 7f21ba000700 1 -- 172.21.15.43:0/1572936109 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b0016070 con 0x7f21bc075d20 2024-05-27T07:17:01.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.767+0000 7f21ba000700 1 -- 172.21.15.43:0/1572936109 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21b0009dd0 con 0x7f21bc075d20 2024-05-27T07:17:01.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.767+0000 7f21c2a00700 1 -- 172.21.15.43:0/1572936109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21bc1a1e30 con 0x7f21bc075d20 2024-05-27T07:17:01.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.767+0000 7f21ba000700 1 -- 172.21.15.43:0/1572936109 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b000f3c0 con 0x7f21bc075d20 2024-05-27T07:17:01.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.767+0000 7f21c2a00700 1 -- 172.21.15.43:0/1572936109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21bc1a2370 con 0x7f21bc075d20 2024-05-27T07:17:01.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.771+0000 7f21ba000700 1 -- 172.21.15.43:0/1572936109 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f21b0019070 con 0x7f21bc075d20 2024-05-27T07:17:01.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.773+0000 7f21d2ac3b80 1 Processor -- start 2024-05-27T07:17:01.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.773+0000 7f21d2ac3b80 1 -- start start 2024-05-27T07:17:01.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.773+0000 7f21ba000700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21a40b2b40 0x7f21a40b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.773+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d3f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.773+0000 7f21ba000700 1 -- 172.21.15.43:0/1572936109 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9408+0+0 (secure 0 0 0) 0x7f21b00e2500 con 0x7f21bc075d20 2024-05-27T07:17:01.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.773+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2bbfe40 0x5618d2d3fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.774+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2d4b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.774+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5618d2c6ce40 con 0x5618d2d492c0 2024-05-27T07:17:01.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.774+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5618d2c5c9e0 con 0x5618d2cab9f0 2024-05-27T07:17:01.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.774+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5618d2c60a20 con 0x5618d2bbfe40 2024-05-27T07:17:01.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.774+0000 7f21c2000700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21a40b2b40 0x7f21a40b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.774+0000 7f21c1600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d3f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.774+0000 7f21c0c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2bbfe40 0x5618d2d3fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.774+0000 7f21c2000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2d4b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.774+0000 7f21c0c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2bbfe40 0x5618d2d3fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:56158/0 (socket says 172.21.15.43:56158) 2024-05-27T07:17:01.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.774+0000 7f21c0c00700 1 -- 172.21.15.43:0/740706083 learned_addr learned my addr 172.21.15.43:0/740706083 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:01.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.775+0000 7f21c2000700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21a40b2b40 0x7f21a40b5050 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009900 tx=0x7f21b4009540 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:01.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.775+0000 7f21c2000700 1 -- 172.21.15.43:0/740706083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2bbfe40 msgr2=0x5618d2d3fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:01.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.775+0000 7f21c2000700 1 --2- 172.21.15.43:0/740706083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2bbfe40 0x5618d2d3fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.775+0000 7f21c2000700 1 -- 172.21.15.43:0/740706083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 msgr2=0x5618d2d3f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:01.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.775+0000 7f21c2000700 1 --2- 172.21.15.43:0/740706083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d3f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.775+0000 7f21c2000700 1 -- 172.21.15.43:0/740706083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618d2c25420 con 0x5618d2d492c0 2024-05-27T07:17:01.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.775+0000 7f21c1600700 1 --2- 172.21.15.43:0/740706083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d3f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:17:01.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.775+0000 7f21c0c00700 1 --2- 172.21.15.43:0/740706083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2bbfe40 0x5618d2d3fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:17:01.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.775+0000 7f21c2000700 1 --2- 172.21.15.43:0/740706083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2d4b6e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f21b400b070 tx=0x7f21b4029000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b322884073dace4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:01.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.776+0000 7f21aea00700 1 -- 172.21.15.43:0/740706083 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b4030070 con 0x5618d2d492c0 2024-05-27T07:17:01.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.776+0000 7f21aea00700 1 -- 172.21.15.43:0/740706083 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21b4007b20 con 0x5618d2d492c0 2024-05-27T07:17:01.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.776+0000 7f21aea00700 1 -- 172.21.15.43:0/740706083 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b401e440 con 0x5618d2d492c0 2024-05-27T07:17:01.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.777+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/740706083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 msgr2=0x5618d2d4b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.777+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/740706083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2d4b6e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f21b400b070 tx=0x7f21b4029000 comp rx=0 tx=0).stop 2024-05-27T07:17:01.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.777+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/740706083 shutdown_connections 2024-05-27T07:17:01.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.777+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/740706083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2d4b6e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.777+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/740706083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d3f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.777+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/740706083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2bbfe40 0x5618d2d3fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.777+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/740706083 >> 172.21.15.43:0/740706083 conn(0x5618d2c59390 msgr2=0x5618d2d3a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:01.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.777+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/740706083 shutdown_connections 2024-05-27T07:17:01.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.777+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/740706083 wait complete. 2024-05-27T07:17:01.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.778+0000 7f21d2ac3b80 1 -- start start 2024-05-27T07:17:01.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.778+0000 7f21d2ac3b80 1 Processor -- start 2024-05-27T07:17:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d46260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d467a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2d492c0 0x5618d2d46ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5618d2b62fa0 con 0x5618d2bbfe40 2024-05-27T07:17:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5618d2bfd760 con 0x5618d2cab9f0 2024-05-27T07:17:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5618d2d4bda0 con 0x5618d2d492c0 2024-05-27T07:17:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21d2ac3b80 10 client.0 _ll_register_callbacks cb 0x5618d2c06c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21c1600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d46260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21c1600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d46260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43324/0 (socket says 172.21.15.43:43324) 2024-05-27T07:17:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21c1600700 1 -- 172.21.15.43:0/2020745185 learned_addr learned my addr 172.21.15.43:0/2020745185 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:01.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21c0c00700 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.779+0000 7f21c2000700 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2d492c0 0x5618d2d46ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.780+0000 7f21c1600700 1 -- 172.21.15.43:0/2020745185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2d492c0 msgr2=0x5618d2d46ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:01.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.780+0000 7f21c1600700 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2d492c0 0x5618d2d46ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.780+0000 7f21c1600700 1 -- 172.21.15.43:0/2020745185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 msgr2=0x5618d2d467a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:01.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.780+0000 7f21c1600700 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d467a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.780+0000 7f21c1600700 1 -- 172.21.15.43:0/2020745185 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618d2bac8d0 con 0x5618d2bbfe40 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.780+0000 7f21c1600700 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d46260 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f21b00a66e0 tx=0x7f21b00108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747fbaef89965779 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.780+0000 7f21a3400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.780+0000 7f21a3400700 1 -- 172.21.15.43:0/2020745185 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b00c3060 con 0x5618d2bbfe40 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.780+0000 7f21a3400700 1 -- 172.21.15.43:0/2020745185 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9408+0+0 (secure 0 0 0) 0x7f21b00a96d0 con 0x5618d2bbfe40 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.780+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2020745185 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21b4007b20 con 0x5618d2bbfe40 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.781+0000 7f21a3400700 1 -- 172.21.15.43:0/2020745185 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f21b0020040 con 0x5618d2bbfe40 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.781+0000 7f21d2ac3b80 10 client.15927 fetch_fsmap learned FSMap version 45 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.781+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2020745185 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5618d2c2d060 con 0x5618d2bbfe40 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.781+0000 7f21a3400700 1 -- 172.21.15.43:0/2020745185 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21b4007b20 con 0x5618d2bbfe40 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.781+0000 7f21a3400700 1 -- 172.21.15.43:0/2020745185 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7f21b00b1e40 con 0x5618d2bbfe40 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.782+0000 7f21d2ac3b80 10 client.15927 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:17:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.782+0000 7f21d2ac3b80 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:17:01.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.782+0000 7f21d2ac3b80 20 client.15927 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:01.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.782+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5618d2d58c90 0x5618d2d5b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.782+0000 7f21d2ac3b80 4 client.15927 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:17:01.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.782+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2020745185 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618d2c6ed90 con 0x5618d2d58c90 2024-05-27T07:17:01.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.782+0000 7f21c0c00700 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5618d2d58c90 0x5618d2d5b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.783+0000 7f21c0c00700 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5618d2d58c90 0x5618d2d5b0b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc4e091a7d3a9337 server_cookie=78c7abcc96a312bf in_seq=0 out_seq=0 2024-05-27T07:17:01.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.783+0000 7f21a3400700 10 client.15927 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:01.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21a3400700 1 -- 172.21.15.43:0/2020745185 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5618d2c6ed90 con 0x5618d2d58c90 2024-05-27T07:17:01.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21a3400700 10 client.15927 handle_command_reply: tid=0 2024-05-27T07:17:01.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21d2ac3b80 1 client.15927 shutdown 2024-05-27T07:17:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2020745185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 msgr2=0x5618d2d46260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d46260 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f21b00a66e0 tx=0x7f21b00108d0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2020745185 shutdown_connections 2024-05-27T07:17:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2d492c0 0x5618d2d46ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5618d2d58c90 0x5618d2d5b0b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d46260 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/2020745185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d467a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2020745185 >> 172.21.15.43:0/2020745185 conn(0x5618d2c59390 msgr2=0x5618d2d3a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.788+0000 7f21a3400700 0 client.15927 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.789+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2020745185 shutdown_connections 2024-05-27T07:17:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.789+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2020745185 wait complete. 2024-05-27T07:17:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.789+0000 7f21d2ac3b80 20 client.15927 trim_cache size 0 max 16384 2024-05-27T07:17:01.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.808+0000 7f21d2ac3b80 1 Processor -- start 2024-05-27T07:17:01.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.808+0000 7f21d2ac3b80 1 -- start start 2024-05-27T07:17:01.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.808+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d54ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.808+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d3e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.808+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2d492c0 0x5618d2d608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.808+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1b2340 con 0x5618d2bbfe40 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.808+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1b3f40 con 0x5618d2cab9f0 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.808+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1b2a10 con 0x5618d2d492c0 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c1600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d54ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c1600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d54ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43334/0 (socket says 172.21.15.43:43334) 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c1600700 1 -- 172.21.15.43:0/2259915365 learned_addr learned my addr 172.21.15.43:0/2259915365 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c0c00700 1 --2- 172.21.15.43:0/2259915365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d3e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c2000700 1 --2- 172.21.15.43:0/2259915365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2d492c0 0x5618d2d608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c1600700 1 -- 172.21.15.43:0/2259915365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2d492c0 msgr2=0x5618d2d608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c1600700 1 --2- 172.21.15.43:0/2259915365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2d492c0 0x5618d2d608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c1600700 1 -- 172.21.15.43:0/2259915365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 msgr2=0x5618d2d3e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c1600700 1 --2- 172.21.15.43:0/2259915365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d3e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c1600700 1 -- 172.21.15.43:0/2259915365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618d2d47050 con 0x5618d2bbfe40 2024-05-27T07:17:01.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.809+0000 7f21c1600700 1 --2- 172.21.15.43:0/2259915365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d54ea0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f21b00a93b0 tx=0x7f21b00a8ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2a4f100ab0cba3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.810+0000 7f21ae000700 1 -- 172.21.15.43:0/2259915365 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b00c3040 con 0x5618d2bbfe40 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.810+0000 7f21ae000700 1 -- 172.21.15.43:0/2259915365 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21b00e2330 con 0x5618d2bbfe40 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.810+0000 7f21ae000700 1 -- 172.21.15.43:0/2259915365 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b00a9930 con 0x5618d2bbfe40 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.811+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2259915365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 msgr2=0x5618d2d54ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.811+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/2259915365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d54ea0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f21b00a93b0 tx=0x7f21b00a8ef0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.811+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2259915365 shutdown_connections 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.811+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/2259915365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2d492c0 0x5618d2d608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.811+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/2259915365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2bbfe40 0x5618d2d54ea0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.811+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/2259915365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2cab9f0 0x5618d2d3e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.811+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2259915365 >> 172.21.15.43:0/2259915365 conn(0x5618d2c59390 msgr2=0x5618d2d3e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.811+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2259915365 shutdown_connections 2024-05-27T07:17:01.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.811+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/2259915365 wait complete. 2024-05-27T07:17:01.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.811+0000 7f21d2ac3b80 1 -- start start 2024-05-27T07:17:01.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.812+0000 7f21d2ac3b80 1 Processor -- start 2024-05-27T07:17:01.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.812+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2bbfe40 0x5618d2d4c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.812+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2cab9f0 0x5618d2d4c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.812+0000 7f21d2ac3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2dc1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.812+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1b3810 con 0x5618d2d492c0 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.812+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1b30e0 con 0x5618d2bbfe40 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.812+0000 7f21d2ac3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1b02c0 con 0x5618d2cab9f0 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.812+0000 7f21d2ac3b80 10 client.0 _ll_register_callbacks cb 0x5618d2c06c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.813+0000 7f21c2000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2dc1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.813+0000 7f21c2000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2dc1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43340/0 (socket says 172.21.15.43:43340) 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.813+0000 7f21c2000700 1 -- 172.21.15.43:0/3875897684 learned_addr learned my addr 172.21.15.43:0/3875897684 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.813+0000 7f21c1600700 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2bbfe40 0x5618d2d4c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.813+0000 7f21c0c00700 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2cab9f0 0x5618d2d4c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.813+0000 7f21c2000700 1 -- 172.21.15.43:0/3875897684 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2cab9f0 msgr2=0x5618d2d4c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.813+0000 7f21c2000700 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2cab9f0 0x5618d2d4c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.813+0000 7f21c2000700 1 -- 172.21.15.43:0/3875897684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2bbfe40 msgr2=0x5618d2d4c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.813+0000 7f21c2000700 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2bbfe40 0x5618d2d4c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.813+0000 7f21c2000700 1 -- 172.21.15.43:0/3875897684 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618d2d4cfc0 con 0x5618d2d492c0 2024-05-27T07:17:01.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.814+0000 7f21c2000700 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2dc1770 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f21b4004d30 tx=0x7f21b4007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec1e105ba318f59 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:01.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.814+0000 7f21a3400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:01.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.814+0000 7f21a3400700 1 -- 172.21.15.43:0/3875897684 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b403a070 con 0x5618d2d492c0 2024-05-27T07:17:01.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.814+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/3875897684 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21b00e2330 con 0x5618d2d492c0 2024-05-27T07:17:01.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.814+0000 7f21a3400700 1 -- 172.21.15.43:0/3875897684 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9408+0+0 (secure 0 0 0) 0x7f21b401f440 con 0x5618d2d492c0 2024-05-27T07:17:01.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.814+0000 7f21a3400700 1 -- 172.21.15.43:0/3875897684 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f21b4042050 con 0x5618d2d492c0 2024-05-27T07:17:01.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.814+0000 7f21d2ac3b80 10 client.15933 fetch_fsmap learned FSMap version 45 2024-05-27T07:17:01.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.814+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/3875897684 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5618d2dc2cf0 con 0x5618d2d492c0 2024-05-27T07:17:01.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.815+0000 7f21a3400700 1 -- 172.21.15.43:0/3875897684 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21b00e2330 con 0x5618d2d492c0 2024-05-27T07:17:01.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.815+0000 7f21a3400700 1 -- 172.21.15.43:0/3875897684 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7f21b403d030 con 0x5618d2d492c0 2024-05-27T07:17:01.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.815+0000 7f21d2ac3b80 10 client.15933 fetch_fsmap finished waiting for FSMap version 45 2024-05-27T07:17:01.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.815+0000 7f21d2ac3b80 10 client.15933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:17:01.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.815+0000 7f21d2ac3b80 20 client.15933 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:01.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.815+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5618d2dc2fd0 0x5618d2dc53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:01.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.815+0000 7f21d2ac3b80 4 client.15933 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:17:01.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.815+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/3875897684 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5618d2c68570 con 0x5618d2dc2fd0 2024-05-27T07:17:01.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.816+0000 7f21c1600700 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5618d2dc2fd0 0x5618d2dc53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:01.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.817+0000 7f21c1600700 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5618d2dc2fd0 0x5618d2dc53f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff99826a70200a9a server_cookie=54ebc7e0b37147bf in_seq=0 out_seq=0 2024-05-27T07:17:01.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.817+0000 7f21a3400700 10 client.15933 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:01.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.817+0000 7f21a3400700 1 -- 172.21.15.43:0/3875897684 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5618d2c68570 con 0x5618d2dc2fd0 2024-05-27T07:17:01.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.817+0000 7f21a3400700 10 client.15933 handle_command_reply: tid=0 2024-05-27T07:17:01.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 client.15933 shutdown 2024-05-27T07:17:01.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/3875897684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 msgr2=0x5618d2dc1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2dc1770 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f21b4004d30 tx=0x7f21b4007de0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/3875897684 shutdown_connections 2024-05-27T07:17:01.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5618d2dc2fd0 0x5618d2dc53f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5618d2d492c0 0x5618d2dc1770 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5618d2bbfe40 0x5618d2d4c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 --2- 172.21.15.43:0/3875897684 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5618d2cab9f0 0x5618d2d4c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/3875897684 >> 172.21.15.43:0/3875897684 conn(0x5618d2c59390 msgr2=0x5618d2d46040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:01.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21a3400700 0 client.15933 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:01.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/3875897684 shutdown_connections 2024-05-27T07:17:01.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 1 -- 172.21.15.43:0/3875897684 wait complete. 2024-05-27T07:17:01.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.818+0000 7f21d2ac3b80 20 client.15933 trim_cache size 0 max 16384 2024-05-27T07:17:01.810 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:17:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.819+0000 7f21c2a00700 1 -- 172.21.15.43:0/1572936109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21a40b2b40 msgr2=0x7f21a40b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.819+0000 7f21c2a00700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21a40b2b40 0x7f21a40b5050 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009900 tx=0x7f21b4009540 comp rx=0 tx=0).stop 2024-05-27T07:17:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.819+0000 7f21c2a00700 1 -- 172.21.15.43:0/1572936109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21bc075d20 msgr2=0x7f21bc19cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.819+0000 7f21c2a00700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21bc075d20 0x7f21bc19cd80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f21b000d950 tx=0x7f21b000dc60 comp rx=0 tx=0).stop 2024-05-27T07:17:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.819+0000 7f21c2a00700 1 -- 172.21.15.43:0/1572936109 shutdown_connections 2024-05-27T07:17:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.819+0000 7f21c2a00700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21a40b2b40 0x7f21a40b5050 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.819+0000 7f21c2a00700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21bc1109a0 0x7f21bc1a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.819+0000 7f21c2a00700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21bc075d20 0x7f21bc19cd80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.819+0000 7f21c2a00700 1 --2- 172.21.15.43:0/1572936109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21bc076670 0x7f21bc19d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.819+0000 7f21c2a00700 1 -- 172.21.15.43:0/1572936109 >> 172.21.15.43:0/1572936109 conn(0x7f21bc079d00 msgr2=0x7f21bc100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:01.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.820+0000 7f21c2a00700 1 -- 172.21.15.43:0/1572936109 shutdown_connections 2024-05-27T07:17:01.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:01.820+0000 7f21c2a00700 1 -- 172.21.15.43:0/1572936109 wait complete. 2024-05-27T07:17:02.180 INFO:tasks.workunit.client.0.smithi043.stdout: 1 8328 0.93 MB/sec warmup 66 sec latency 64.034 ms 2024-05-27T07:17:02.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:02 smithi132 ceph-mon[35283]: pgmap v102: 129 pgs: 129 active+clean; 29 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 538 KiB/s wr, 83 op/s 2024-05-27T07:17:02.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:02 smithi043 ceph-mon[31531]: pgmap v102: 129 pgs: 129 active+clean; 29 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 538 KiB/s wr, 83 op/s 2024-05-27T07:17:02.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:02 smithi179 ceph-mon[35888]: pgmap v102: 129 pgs: 129 active+clean; 29 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 538 KiB/s wr, 83 op/s 2024-05-27T07:17:03.180 INFO:tasks.workunit.client.0.smithi043.stdout: 1 8494 0.96 MB/sec warmup 67 sec latency 57.456 ms 2024-05-27T07:17:03.731 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:17:03.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:17:04.180 INFO:tasks.workunit.client.0.smithi043.stdout: 1 8613 0.95 MB/sec warmup 68 sec latency 60.531 ms 2024-05-27T07:17:04.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:04 smithi043 ceph-mon[31531]: pgmap v103: 129 pgs: 129 active+clean; 29 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 453 KiB/s wr, 70 op/s 2024-05-27T07:17:04.233 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:04.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:04 smithi132 ceph-mon[35283]: pgmap v103: 129 pgs: 129 active+clean; 29 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 453 KiB/s wr, 70 op/s 2024-05-27T07:17:04.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:04 smithi179 ceph-mon[35888]: pgmap v103: 129 pgs: 129 active+clean; 29 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 453 KiB/s wr, 70 op/s 2024-05-27T07:17:05.181 INFO:tasks.workunit.client.0.smithi043.stdout: 1 8751 0.93 MB/sec warmup 69 sec latency 55.360 ms 2024-05-27T07:17:05.359 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:04 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:17:05.359 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:04 smithi179 systemd[1]: Stopped Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:05.359 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:04 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.575s CPU time. 2024-05-27T07:17:05.359 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:04 smithi179 systemd[1]: Starting Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:17:05.709 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:05 smithi179 podman[71485]: 2024-05-27 07:17:05.273625676 +0000 UTC m=+0.034993131 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:17:05.709 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:05 smithi179 podman[71485]: 2024-05-27 07:17:05.392398735 +0000 UTC m=+0.153766227 container create fc67d5b4987a426ff6d76f1127586dba799175ec1a00274272bd8a443882d6f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0) 2024-05-27T07:17:06.056 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:05 smithi179 podman[71485]: 2024-05-27 07:17:05.722694756 +0000 UTC m=+0.484062231 container init fc67d5b4987a426ff6d76f1127586dba799175ec1a00274272bd8a443882d6f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-27T07:17:06.056 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:05 smithi179 podman[71485]: 2024-05-27 07:17:05.73268618 +0000 UTC m=+0.494053652 container start fc67d5b4987a426ff6d76f1127586dba799175ec1a00274272bd8a443882d6f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-27T07:17:06.057 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:05 smithi179 bash[71485]: fc67d5b4987a426ff6d76f1127586dba799175ec1a00274272bd8a443882d6f2 2024-05-27T07:17:06.057 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:05 smithi179 systemd[1]: Started Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:06.057 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:05 smithi179 ceph-mds[71505]: -- 172.21.15.179:0/2109772404 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d1b7b71a40 con 0x55d1b7bb9600 2024-05-27T07:17:06.057 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: starting mds.c at 2024-05-27T07:17:06.180 INFO:tasks.workunit.client.0.smithi043.stdout: 1 8914 0.92 MB/sec warmup 70 sec latency 50.233 ms 2024-05-27T07:17:06.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-mon[35888]: pgmap v104: 129 pgs: 129 active+clean; 29 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 430 KiB/s wr, 66 op/s 2024-05-27T07:17:06.191 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-mon[35888]: daemon mds.c restarted 2024-05-27T07:17:06.192 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-mon[35888]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:17:06.192 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6836/3754939540,v1:172.21.15.179:6837/3754939540] up:boot 2024-05-27T07:17:06.192 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:06.192 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:17:06.193 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9060c00700 time 2024-05-27T07:17:06.128480+0000 2024-05-27T07:17:06.193 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:06.193 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f90766ecba6] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f90766ecdd1] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d188d2723d] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d188d12971] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d188d10fa0] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d188d11651] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d188ca6347] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d1889a287e] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d1889a7e94] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d1889b2969] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d188d4b48c] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d188d549cb] 2024-05-27T07:17:06.194 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 13: (Thread::entry_wrapper()+0x43) [0x7f90766bb72d] 2024-05-27T07:17:06.195 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 14: (Thread::_entry_func(void*)+0xd) [0x7f90766bb749] 2024-05-27T07:17:06.195 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f907525f1ca] 2024-05-27T07:17:06.195 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 16: clone() 2024-05-27T07:17:06.195 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.130+0000 7f9060c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9060c00700 time 2024-05-27T07:17:06.128480+0000 2024-05-27T07:17:06.195 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:06.195 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.195 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:06.195 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f90766ecba6] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f90766ecdd1] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d188d2723d] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d188d12971] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d188d10fa0] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d188d11651] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d188ca6347] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d1889a287e] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d1889a7e94] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d1889b2969] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d188d4b48c] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d188d549cb] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 13: (Thread::entry_wrapper()+0x43) [0x7f90766bb72d] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 14: (Thread::_entry_func(void*)+0xd) [0x7f90766bb749] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f907525f1ca] 2024-05-27T07:17:06.196 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 16: clone() 2024-05-27T07:17:06.197 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.197 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: *** Caught signal (Aborted) ** 2024-05-27T07:17:06.197 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: in thread 7f9060c00700 thread_name:md_log_replay 2024-05-27T07:17:06.197 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:06.197 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9075269d20] 2024-05-27T07:17:06.197 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2: gsignal() 2024-05-27T07:17:06.197 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 3: abort() 2024-05-27T07:17:06.197 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f90766ecced] 2024-05-27T07:17:06.197 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f90766ecdd1] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d188d2723d] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d188d12971] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d188d10fa0] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d188d11651] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d188ca6347] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d1889a287e] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d1889a7e94] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d1889b2969] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d188d4b48c] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d188d549cb] 2024-05-27T07:17:06.198 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 16: (Thread::entry_wrapper()+0x43) [0x7f90766bb72d] 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 17: (Thread::_entry_func(void*)+0xd) [0x7f90766bb749] 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f907525f1ca] 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 19: clone() 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.133+0000 7f9060c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: in thread 7f9060c00700 thread_name:md_log_replay 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9075269d20] 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2: gsignal() 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 3: abort() 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f90766ecced] 2024-05-27T07:17:06.199 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f90766ecdd1] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d188d2723d] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d188d12971] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d188d10fa0] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d188d11651] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d188ca6347] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d1889a287e] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d1889a7e94] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d1889b2969] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d188d4b48c] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d188d549cb] 2024-05-27T07:17:06.200 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 16: (Thread::entry_wrapper()+0x43) [0x7f90766bb72d] 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 17: (Thread::_entry_func(void*)+0xd) [0x7f90766bb749] 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f907525f1ca] 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 19: clone() 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: -1> 2024-05-27T07:17:06.130+0000 7f9060c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9060c00700 time 2024-05-27T07:17:06.128480+0000 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f90766ecba6] 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f90766ecdd1] 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d188d2723d] 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d188d12971] 2024-05-27T07:17:06.201 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d188d10fa0] 2024-05-27T07:17:06.202 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d188d11651] 2024-05-27T07:17:06.202 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d188ca6347] 2024-05-27T07:17:06.202 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d1889a287e] 2024-05-27T07:17:06.202 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d1889a7e94] 2024-05-27T07:17:06.202 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d1889b2969] 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d188d4b48c] 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d188d549cb] 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 13: (Thread::entry_wrapper()+0x43) [0x7f90766bb72d] 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 14: (Thread::_entry_func(void*)+0xd) [0x7f90766bb749] 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f907525f1ca] 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 16: clone() 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 0> 2024-05-27T07:17:06.133+0000 7f9060c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: in thread 7f9060c00700 thread_name:md_log_replay 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.203 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9075269d20] 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2: gsignal() 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 3: abort() 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f90766ecced] 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f90766ecdd1] 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d188d2723d] 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d188d12971] 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d188d10fa0] 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d188d11651] 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d188ca6347] 2024-05-27T07:17:06.204 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d1889a287e] 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d1889a7e94] 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d1889b2969] 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d188d4b48c] 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d188d549cb] 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 16: (Thread::entry_wrapper()+0x43) [0x7f90766bb72d] 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 17: (Thread::_entry_func(void*)+0xd) [0x7f90766bb749] 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f907525f1ca] 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 19: clone() 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: -1> 2024-05-27T07:17:06.130+0000 7f9060c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9060c00700 time 2024-05-27T07:17:06.128480+0000 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.205 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f90766ecba6] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f90766ecdd1] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d188d2723d] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d188d12971] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d188d10fa0] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d188d11651] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d188ca6347] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d1889a287e] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d1889a7e94] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d1889b2969] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d188d4b48c] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d188d549cb] 2024-05-27T07:17:06.206 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 13: (Thread::entry_wrapper()+0x43) [0x7f90766bb72d] 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 14: (Thread::_entry_func(void*)+0xd) [0x7f90766bb749] 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f907525f1ca] 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 16: clone() 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 0> 2024-05-27T07:17:06.133+0000 7f9060c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: in thread 7f9060c00700 thread_name:md_log_replay 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9075269d20] 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2: gsignal() 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 3: abort() 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f90766ecced] 2024-05-27T07:17:06.207 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f90766ecdd1] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d188d2723d] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d188d12971] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d188d10fa0] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d188d11651] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d188ca6347] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d1889a287e] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d1889a7e94] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d1889b2969] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d188d4b48c] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d188d549cb] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 16: (Thread::entry_wrapper()+0x43) [0x7f90766bb72d] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 17: (Thread::_entry_func(void*)+0xd) [0x7f90766bb749] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f907525f1ca] 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 19: clone() 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:06.208 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c[71501]: 2024-05-27T07:17:06.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:06 smithi043 ceph-mon[31531]: pgmap v104: 129 pgs: 129 active+clean; 29 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 430 KiB/s wr, 66 op/s 2024-05-27T07:17:06.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:06 smithi043 ceph-mon[31531]: daemon mds.c restarted 2024-05-27T07:17:06.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:06 smithi043 ceph-mon[31531]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:17:06.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:06 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6836/3754939540,v1:172.21.15.179:6837/3754939540] up:boot 2024-05-27T07:17:06.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:06 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:06.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:17:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:06 smithi132 ceph-mon[35283]: pgmap v104: 129 pgs: 129 active+clean; 29 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 430 KiB/s wr, 66 op/s 2024-05-27T07:17:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:06 smithi132 ceph-mon[35283]: daemon mds.c restarted 2024-05-27T07:17:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:06 smithi132 ceph-mon[35283]: osdmap e98: 12 total, 12 up, 12 in 2024-05-27T07:17:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:06 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6836/3754939540,v1:172.21.15.179:6837/3754939540] up:boot 2024-05-27T07:17:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:06 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:06.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-27T07:17:06.459 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:06 smithi179 podman[71536]: 2024-05-27 07:17:06.284326142 +0000 UTC m=+0.033057120 container died fc67d5b4987a426ff6d76f1127586dba799175ec1a00274272bd8a443882d6f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-27T07:17:06.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.774+0000 7fbdce800700 1 -- 172.21.15.43:0/1737040039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdc8109860 msgr2=0x7fbdc81111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:06.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.774+0000 7fbdce800700 1 --2- 172.21.15.43:0/1737040039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdc8109860 0x7fbdc81111c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00099c0 tx=0x7fbdc0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.774+0000 7fbdce800700 1 -- 172.21.15.43:0/1737040039 shutdown_connections 2024-05-27T07:17:06.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.774+0000 7fbdce800700 1 --2- 172.21.15.43:0/1737040039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdc8109860 0x7fbdc81111c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.774+0000 7fbdce800700 1 --2- 172.21.15.43:0/1737040039 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdc81045e0 0x7fbdc81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.774+0000 7fbdce800700 1 --2- 172.21.15.43:0/1737040039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc8108f10 0x7fbdc8109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.774+0000 7fbdce800700 1 -- 172.21.15.43:0/1737040039 >> 172.21.15.43:0/1737040039 conn(0x7fbdc8100030 msgr2=0x7fbdc8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:06.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.774+0000 7fbdce800700 1 -- 172.21.15.43:0/1737040039 shutdown_connections 2024-05-27T07:17:06.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.774+0000 7fbdce800700 1 -- 172.21.15.43:0/1737040039 wait complete. 2024-05-27T07:17:06.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.776+0000 7fbdce800700 1 Processor -- start 2024-05-27T07:17:06.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.776+0000 7fbdce800700 1 -- start start 2024-05-27T07:17:06.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.776+0000 7fbdce800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdc81045e0 0x7fbdc819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdce800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdc8108f10 0x7fbdc819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdce800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc8109860 0x7fbdc819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdce800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc8114340 con 0x7fbdc81045e0 2024-05-27T07:17:06.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdce800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc81141c0 con 0x7fbdc8109860 2024-05-27T07:17:06.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdce800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc81144c0 con 0x7fbdc8108f10 2024-05-27T07:17:06.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdc7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdc81045e0 0x7fbdc819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdc7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdc81045e0 0x7fbdc819ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38530/0 (socket says 172.21.15.43:38530) 2024-05-27T07:17:06.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdc7400700 1 -- 172.21.15.43:0/3245704423 learned_addr learned my addr 172.21.15.43:0/3245704423 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:06.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdbea00700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdc8108f10 0x7fbdc819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdc7400700 1 -- 172.21.15.43:0/3245704423 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdc8108f10 msgr2=0x7fbdc819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:06.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdc7400700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdc8108f10 0x7fbdc819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdc7400700 1 -- 172.21.15.43:0/3245704423 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc8109860 msgr2=0x7fbdc819f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:06.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdc7400700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc8109860 0x7fbdc819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.777+0000 7fbdc7400700 1 -- 172.21.15.43:0/3245704423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc819fc20 con 0x7fbdc81045e0 2024-05-27T07:17:06.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.778+0000 7fbdc7400700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdc81045e0 0x7fbdc819ad50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80060b0 tx=0x7fbdb8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:06.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.778+0000 7fbdc4c00700 1 -- 172.21.15.43:0/3245704423 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb801a070 con 0x7fbdc81045e0 2024-05-27T07:17:06.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.778+0000 7fbdc4c00700 1 -- 172.21.15.43:0/3245704423 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdb800de20 con 0x7fbdc81045e0 2024-05-27T07:17:06.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.778+0000 7fbdce800700 1 -- 172.21.15.43:0/3245704423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc819ff10 con 0x7fbdc81045e0 2024-05-27T07:17:06.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.778+0000 7fbdc4c00700 1 -- 172.21.15.43:0/3245704423 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8014690 con 0x7fbdc81045e0 2024-05-27T07:17:06.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.778+0000 7fbdce800700 1 -- 172.21.15.43:0/3245704423 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc81a7830 con 0x7fbdc81045e0 2024-05-27T07:17:06.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.782+0000 7fbdc4c00700 1 -- 172.21.15.43:0/3245704423 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fbdb801d070 con 0x7fbdc81045e0 2024-05-27T07:17:06.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.784+0000 7fbddd2b3b80 1 Processor -- start 2024-05-27T07:17:06.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.784+0000 7fbdc4c00700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbdac0b29f0 0x7fbdac0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.784+0000 7fbddd2b3b80 1 -- start start 2024-05-27T07:17:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.784+0000 7fbdc4c00700 1 -- 172.21.15.43:0/3245704423 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fbdb80e21e0 con 0x7fbdc81045e0 2024-05-27T07:17:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.785+0000 7fbdc7e00700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbdac0b29f0 0x7fbdac0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.785+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e420c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.785+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e421160 0x55717e41ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.785+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e41b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.785+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55717e344ec0 con 0x55717e421160 2024-05-27T07:17:06.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.785+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55717e334a60 con 0x55717e41b240 2024-05-27T07:17:06.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.785+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55717e338aa0 con 0x55717e297ec0 2024-05-27T07:17:06.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.785+0000 7fbdbea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e421160 0x55717e41ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.785+0000 7fbdbea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e421160 0x55717e41ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38542/0 (socket says 172.21.15.43:38542) 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.785+0000 7fbdbea00700 1 -- 172.21.15.43:0/679054077 learned_addr learned my addr 172.21.15.43:0/679054077 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.786+0000 7fbdc7400700 1 --2- 172.21.15.43:0/679054077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e420c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.786+0000 7fbdc7e00700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbdac0b29f0 0x7fbdac0b4f00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00058e0 tx=0x7fbdc0005960 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.786+0000 7fbdc7e00700 1 --2- 172.21.15.43:0/679054077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e41b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.786+0000 7fbdbea00700 1 -- 172.21.15.43:0/679054077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 msgr2=0x55717e420c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.786+0000 7fbdbea00700 1 --2- 172.21.15.43:0/679054077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e420c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.786+0000 7fbdbea00700 1 -- 172.21.15.43:0/679054077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 msgr2=0x55717e41b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.786+0000 7fbdbea00700 1 --2- 172.21.15.43:0/679054077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e41b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.786+0000 7fbdbea00700 1 -- 172.21.15.43:0/679054077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55717e2fd4a0 con 0x55717e421160 2024-05-27T07:17:06.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.786+0000 7fbdc7400700 1 --2- 172.21.15.43:0/679054077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e420c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:06.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.787+0000 7fbdbea00700 1 --2- 172.21.15.43:0/679054077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e421160 0x55717e41ad00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbdb000cb90 tx=0x7fbdb000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9650c5124833d494 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:06.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.787+0000 7fbdbcc00700 1 -- 172.21.15.43:0/679054077 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb000ed70 con 0x55717e421160 2024-05-27T07:17:06.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.787+0000 7fbdbcc00700 1 -- 172.21.15.43:0/679054077 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdb00044b0 con 0x55717e421160 2024-05-27T07:17:06.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.787+0000 7fbdbcc00700 1 -- 172.21.15.43:0/679054077 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0010430 con 0x55717e421160 2024-05-27T07:17:06.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.788+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/679054077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e421160 msgr2=0x55717e41ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:06.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.788+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/679054077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e421160 0x55717e41ad00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbdb000cb90 tx=0x7fbdb000cea0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.788+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/679054077 shutdown_connections 2024-05-27T07:17:06.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.788+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/679054077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e41b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.788+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/679054077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e420c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.788+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/679054077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e421160 0x55717e41ad00 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.788+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/679054077 >> 172.21.15.43:0/679054077 conn(0x55717e2f8040 msgr2=0x55717e411950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:06.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.788+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/679054077 shutdown_connections 2024-05-27T07:17:06.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.788+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/679054077 wait complete. 2024-05-27T07:17:06.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.789+0000 7fbddd2b3b80 1 -- start start 2024-05-27T07:17:06.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.789+0000 7fbddd2b3b80 1 Processor -- start 2024-05-27T07:17:06.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e424920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e41b240 0x55717e424e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e421160 0x55717e4253a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55717e23b020 con 0x55717e41b240 2024-05-27T07:17:06.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55717e2d57e0 con 0x55717e421160 2024-05-27T07:17:06.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55717e427c40 con 0x55717e297ec0 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbddd2b3b80 10 client.0 _ll_register_callbacks cb 0x55717e2decf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbdbea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e41b240 0x55717e424e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbdbea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e41b240 0x55717e424e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38544/0 (socket says 172.21.15.43:38544) 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbdbea00700 1 -- 172.21.15.43:0/3901047326 learned_addr learned my addr 172.21.15.43:0/3901047326 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbdc7e00700 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e421160 0x55717e4253a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.790+0000 7fbdc7400700 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e424920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbdbea00700 1 -- 172.21.15.43:0/3901047326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 msgr2=0x55717e424920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbdbea00700 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e424920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbdbea00700 1 -- 172.21.15.43:0/3901047326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e421160 msgr2=0x55717e4253a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbdbea00700 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e421160 0x55717e4253a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbdbea00700 1 -- 172.21.15.43:0/3901047326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55717e284950 con 0x55717e41b240 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbdc7e00700 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e421160 0x55717e4253a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:17:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbdbea00700 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e41b240 0x55717e424e60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbdb000c7c0 tx=0x7fbdb0003ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5c6cfb927fe96c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:06.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbdb5600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbdb5600700 1 -- 172.21.15.43:0/3901047326 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0003de0 con 0x55717e41b240 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbdb5600700 1 -- 172.21.15.43:0/3901047326 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fbdb0003f80 con 0x55717e41b240 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.791+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/3901047326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdb00044b0 con 0x55717e41b240 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbdb5600700 1 -- 172.21.15.43:0/3901047326 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fbdb002f050 con 0x55717e41b240 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbddd2b3b80 10 client.15948 fetch_fsmap learned FSMap version 46 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/3901047326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55717e3050e0 con 0x55717e41b240 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbdb5600700 1 -- 172.21.15.43:0/3901047326 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdb00044b0 con 0x55717e41b240 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbdb5600700 1 -- 172.21.15.43:0/3901047326 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 46) ==== 8378+0+0 (secure 0 0 0) 0x7fbdb0005260 con 0x55717e41b240 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbddd2b3b80 10 client.15948 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbddd2b3b80 10 client.15948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbddd2b3b80 20 client.15948 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:06.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55717e434a30 0x55717e436e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbddd2b3b80 4 client.15948 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:17:06.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.792+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/3901047326 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55717e346e10 con 0x55717e434a30 2024-05-27T07:17:06.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.793+0000 7fbdc7400700 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55717e434a30 0x55717e436e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.794+0000 7fbdc7400700 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55717e434a30 0x55717e436e50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e26a4a55d6c6f798 server_cookie=e998a513229c323d in_seq=0 out_seq=0 2024-05-27T07:17:06.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.794+0000 7fbdb5600700 10 client.15948 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:06.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.796+0000 7fbdb5600700 1 -- 172.21.15.43:0/3901047326 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55717e346e10 con 0x55717e434a30 2024-05-27T07:17:06.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.796+0000 7fbdb5600700 10 client.15948 handle_command_reply: tid=0 2024-05-27T07:17:06.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.796+0000 7fbddd2b3b80 1 client.15948 shutdown 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/3901047326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e41b240 msgr2=0x55717e424e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e41b240 0x55717e424e60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbdb000c7c0 tx=0x7fbdb0003ab0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/3901047326 shutdown_connections 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e421160 0x55717e4253a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e424920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55717e434a30 0x55717e436e50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/3901047326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e41b240 0x55717e424e60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/3901047326 >> 172.21.15.43:0/3901047326 conn(0x55717e2f8040 msgr2=0x55717e411600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbdb5600700 0 client.15948 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/3901047326 shutdown_connections 2024-05-27T07:17:06.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.797+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/3901047326 wait complete. 2024-05-27T07:17:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.798+0000 7fbddd2b3b80 20 client.15948 trim_cache size 0 max 16384 2024-05-27T07:17:06.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.822+0000 7fbddd2b3b80 1 Processor -- start 2024-05-27T07:17:06.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.822+0000 7fbddd2b3b80 1 -- start start 2024-05-27T07:17:06.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.823+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e2f8040 0x55717e416310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.823+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e297ec0 0x55717e426c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.823+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e430d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.823+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbda400ecc0 con 0x55717e297ec0 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.823+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbda40108c0 con 0x55717e41b240 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.823+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbda400f390 con 0x55717e2f8040 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.823+0000 7fbdbea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e297ec0 0x55717e426c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.823+0000 7fbdbea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e297ec0 0x55717e426c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38548/0 (socket says 172.21.15.43:38548) 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.823+0000 7fbdbea00700 1 -- 172.21.15.43:0/2848042182 learned_addr learned my addr 172.21.15.43:0/2848042182 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.824+0000 7fbdc7e00700 1 --2- 172.21.15.43:0/2848042182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e430d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.824+0000 7fbdc7400700 1 --2- 172.21.15.43:0/2848042182 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e2f8040 0x55717e416310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.824+0000 7fbdbea00700 1 -- 172.21.15.43:0/2848042182 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e2f8040 msgr2=0x55717e416310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.824+0000 7fbdbea00700 1 --2- 172.21.15.43:0/2848042182 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e2f8040 0x55717e416310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.824+0000 7fbdbea00700 1 -- 172.21.15.43:0/2848042182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 msgr2=0x55717e430d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:06.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.824+0000 7fbdbea00700 1 --2- 172.21.15.43:0/2848042182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e430d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.824+0000 7fbdbea00700 1 -- 172.21.15.43:0/2848042182 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55717e425710 con 0x55717e297ec0 2024-05-27T07:17:06.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.824+0000 7fbdbea00700 1 --2- 172.21.15.43:0/2848042182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e297ec0 0x55717e426c80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbdb000e630 tx=0x7fbdb0010700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47be230408e27b58 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.824+0000 7fbdb7e00700 1 -- 172.21.15.43:0/2848042182 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0038030 con 0x55717e297ec0 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.825+0000 7fbdb7e00700 1 -- 172.21.15.43:0/2848042182 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdb0010870 con 0x55717e297ec0 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.825+0000 7fbdb7e00700 1 -- 172.21.15.43:0/2848042182 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0004810 con 0x55717e297ec0 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.825+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2848042182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e297ec0 msgr2=0x55717e426c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.825+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/2848042182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e297ec0 0x55717e426c80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbdb000e630 tx=0x7fbdb0010700 comp rx=0 tx=0).stop 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.825+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2848042182 shutdown_connections 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.825+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/2848042182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e430d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.825+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/2848042182 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e2f8040 0x55717e416310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.825+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/2848042182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e297ec0 0x55717e426c80 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.825+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2848042182 >> 172.21.15.43:0/2848042182 conn(0x55717e41b920 msgr2=0x55717e41e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.826+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2848042182 shutdown_connections 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.826+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2848042182 wait complete. 2024-05-27T07:17:06.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.826+0000 7fbddd2b3b80 1 -- start start 2024-05-27T07:17:06.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.826+0000 7fbddd2b3b80 1 Processor -- start 2024-05-27T07:17:06.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.827+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e42cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.827+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e2f8040 0x55717e42d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.827+0000 7fbddd2b3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e429f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.827+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbda4010190 con 0x55717e2f8040 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.827+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbda400fa60 con 0x55717e41b240 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.827+0000 7fbddd2b3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbda400cc40 con 0x55717e297ec0 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.827+0000 7fbddd2b3b80 10 client.0 _ll_register_callbacks cb 0x55717e2decf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.827+0000 7fbdbea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e2f8040 0x55717e42d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.828+0000 7fbdbea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e2f8040 0x55717e42d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38556/0 (socket says 172.21.15.43:38556) 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.828+0000 7fbdc7400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e42cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.828+0000 7fbdbea00700 1 -- 172.21.15.43:0/2241854438 learned_addr learned my addr 172.21.15.43:0/2241854438 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.828+0000 7fbdc7e00700 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e429f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.828+0000 7fbdbea00700 1 -- 172.21.15.43:0/2241854438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 msgr2=0x55717e42cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.828+0000 7fbdbea00700 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e42cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.828+0000 7fbdbea00700 1 -- 172.21.15.43:0/2241854438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 msgr2=0x55717e429f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:06.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.828+0000 7fbdbea00700 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e429f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.828+0000 7fbdbea00700 1 -- 172.21.15.43:0/2241854438 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55717e42a470 con 0x55717e2f8040 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.829+0000 7fbdbea00700 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e2f8040 0x55717e42d4a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fbdb000cb90 tx=0x7fbdb0010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e7e2abd230d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.829+0000 7fbdb5600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.829+0000 7fbdb5600700 1 -- 172.21.15.43:0/2241854438 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0038040 con 0x55717e2f8040 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.829+0000 7fbdb5600700 1 -- 172.21.15.43:0/2241854438 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fbdb0005000 con 0x55717e2f8040 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.829+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2241854438 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdb0010870 con 0x55717e2f8040 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.829+0000 7fbdb5600700 1 -- 172.21.15.43:0/2241854438 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fbdb004a050 con 0x55717e2f8040 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.829+0000 7fbddd2b3b80 10 client.15954 fetch_fsmap learned FSMap version 46 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.829+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2241854438 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55717e49fa20 con 0x55717e2f8040 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.830+0000 7fbdb5600700 1 -- 172.21.15.43:0/2241854438 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdb0010870 con 0x55717e2f8040 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.830+0000 7fbdb5600700 1 -- 172.21.15.43:0/2241854438 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 46) ==== 8378+0+0 (secure 0 0 0) 0x7fbdb0003de0 con 0x55717e2f8040 2024-05-27T07:17:06.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.830+0000 7fbddd2b3b80 10 client.15954 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:17:06.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.830+0000 7fbddd2b3b80 10 client.15954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:17:06.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.830+0000 7fbddd2b3b80 20 client.15954 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:06.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.830+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55717e421160 0x55717e4a2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:06.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.830+0000 7fbddd2b3b80 4 client.15954 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:17:06.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.830+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2241854438 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55717e3405f0 con 0x55717e421160 2024-05-27T07:17:06.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.831+0000 7fbdc7400700 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55717e421160 0x55717e4a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:06.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.831+0000 7fbdc7400700 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55717e421160 0x55717e4a2100 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a10f554167238da server_cookie=f655973e012640bc in_seq=0 out_seq=0 2024-05-27T07:17:06.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.832+0000 7fbdb5600700 10 client.15954 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:06.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.835+0000 7fbdb5600700 1 -- 172.21.15.43:0/2241854438 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55717e3405f0 con 0x55717e421160 2024-05-27T07:17:06.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.835+0000 7fbdb5600700 10 client.15954 handle_command_reply: tid=0 2024-05-27T07:17:06.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.835+0000 7fbddd2b3b80 1 client.15954 shutdown 2024-05-27T07:17:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2241854438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e2f8040 msgr2=0x55717e42d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e2f8040 0x55717e42d4a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fbdb000cb90 tx=0x7fbdb0010c50 comp rx=0 tx=0).stop 2024-05-27T07:17:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2241854438 shutdown_connections 2024-05-27T07:17:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55717e41b240 0x55717e429f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55717e297ec0 0x55717e42cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55717e421160 0x55717e4a2100 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 1 --2- 172.21.15.43:0/2241854438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55717e2f8040 0x55717e42d4a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2241854438 >> 172.21.15.43:0/2241854438 conn(0x55717e41b920 msgr2=0x55717e416070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbdb5600700 0 client.15954 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2241854438 shutdown_connections 2024-05-27T07:17:06.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 1 -- 172.21.15.43:0/2241854438 wait complete. 2024-05-27T07:17:06.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.836+0000 7fbddd2b3b80 20 client.15954 trim_cache size 0 max 16384 2024-05-27T07:17:06.828 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:17:06.828 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:17:06.828 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "7c2441f4-56c2-441a-a3d3-1abca46b5d64", 2024-05-27T07:17:06.828 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:17:06.828 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.837+0000 7fbdce800700 1 -- 172.21.15.43:0/3245704423 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbdac0b29f0 msgr2=0x7fbdac0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.837+0000 7fbdce800700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbdac0b29f0 0x7fbdac0b4f00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00058e0 tx=0x7fbdc0005960 comp rx=0 tx=0).stop 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.837+0000 7fbdce800700 1 -- 172.21.15.43:0/3245704423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdc81045e0 msgr2=0x7fbdc819ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.837+0000 7fbdce800700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdc81045e0 0x7fbdc819ad50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80060b0 tx=0x7fbdb8007f90 comp rx=0 tx=0).stop 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.837+0000 7fbdce800700 1 -- 172.21.15.43:0/3245704423 shutdown_connections 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.837+0000 7fbdce800700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbdac0b29f0 0x7fbdac0b4f00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.837+0000 7fbdce800700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc8109860 0x7fbdc819f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.837+0000 7fbdce800700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdc81045e0 0x7fbdc819ad50 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.837+0000 7fbdce800700 1 --2- 172.21.15.43:0/3245704423 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdc8108f10 0x7fbdc819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.837+0000 7fbdce800700 1 -- 172.21.15.43:0/3245704423 >> 172.21.15.43:0/3245704423 conn(0x7fbdc8100030 msgr2=0x7fbdc81086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.838+0000 7fbdce800700 1 -- 172.21.15.43:0/3245704423 shutdown_connections 2024-05-27T07:17:06.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:06.838+0000 7fbdce800700 1 -- 172.21.15.43:0/3245704423 wait complete. 2024-05-27T07:17:07.181 INFO:tasks.workunit.client.0.smithi043.stdout: 1 9033 0.91 MB/sec warmup 71 sec latency 58.758 ms 2024-05-27T07:17:07.459 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:07 smithi179 podman[71536]: 2024-05-27 07:17:07.018011972 +0000 UTC m=+0.766742960 container remove fc67d5b4987a426ff6d76f1127586dba799175ec1a00274272bd8a443882d6f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:17:07.459 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:07 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:17:07.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:17:07.959 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:07 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:17:07.959 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:07 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.577s CPU time. 2024-05-27T07:17:08.181 INFO:tasks.workunit.client.0.smithi043.stdout: 1 9187 0.90 MB/sec warmup 72 sec latency 53.945 ms 2024-05-27T07:17:08.310 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:08.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: pgmap v106: 129 pgs: 129 active+clean; 29 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 329 KiB/s wr, 78 op/s 2024-05-27T07:17:08.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: pgmap v106: 129 pgs: 129 active+clean; 29 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 329 KiB/s wr, 78 op/s 2024-05-27T07:17:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:17:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:17:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:17:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:17:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:17:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:17:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:17:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:17:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: pgmap v106: 129 pgs: 129 active+clean; 29 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 329 KiB/s wr, 78 op/s 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:17:08.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:17:08.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:17:08.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:08.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:17:09.181 INFO:tasks.workunit.client.0.smithi043.stdout: 1 9395 0.90 MB/sec warmup 73 sec latency 52.605 ms 2024-05-27T07:17:09.518 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:09 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:17:09.519 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:09 smithi043 systemd[1]: Stopped Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:09.519 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:09 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.590s CPU time. 2024-05-27T07:17:09.519 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:09 smithi043 systemd[1]: Starting Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:17:10.181 INFO:tasks.workunit.client.0.smithi043.stdout: 1 9488 0.89 MB/sec warmup 74 sec latency 56.039 ms 2024-05-27T07:17:10.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:10 smithi043 ceph-mon[31531]: pgmap v107: 129 pgs: 129 active+clean; 29 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 312 KiB/s wr, 74 op/s 2024-05-27T07:17:10.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:10 smithi132 ceph-mon[35283]: pgmap v107: 129 pgs: 129 active+clean; 29 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 312 KiB/s wr, 74 op/s 2024-05-27T07:17:10.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:10 smithi179 ceph-mon[35888]: pgmap v107: 129 pgs: 129 active+clean; 29 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 312 KiB/s wr, 74 op/s 2024-05-27T07:17:10.886 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:10 smithi043 podman[97490]: 2024-05-27 07:17:10.412883814 +0000 UTC m=+0.278035659 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:17:10.886 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:10 smithi043 podman[97490]: 2024-05-27 07:17:10.587920258 +0000 UTC m=+0.453072128 container create a4ec09669ee058322333bd82362dbe4acd0f2f3cc5dec1bd085ece20dc96c1cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:17:10.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.981+0000 7f6af2c00700 1 -- 172.21.15.43:0/363831923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aec105530 msgr2=0x7f6aec105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:10.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.981+0000 7f6af2c00700 1 --2- 172.21.15.43:0/363831923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aec105530 0x7f6aec105940 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009990 tx=0x7f6adc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.981+0000 7f6af2c00700 1 -- 172.21.15.43:0/363831923 shutdown_connections 2024-05-27T07:17:10.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.981+0000 7f6af2c00700 1 --2- 172.21.15.43:0/363831923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6aec10e910 0x7f6aec110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.981+0000 7f6af2c00700 1 --2- 172.21.15.43:0/363831923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aec105530 0x7f6aec105940 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.981+0000 7f6af2c00700 1 --2- 172.21.15.43:0/363831923 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aec105e80 0x7f6aec10e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.981+0000 7f6af2c00700 1 -- 172.21.15.43:0/363831923 >> 172.21.15.43:0/363831923 conn(0x7f6aec0fee40 msgr2=0x7f6aec101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:10.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.981+0000 7f6af2c00700 1 -- 172.21.15.43:0/363831923 shutdown_connections 2024-05-27T07:17:10.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.981+0000 7f6af2c00700 1 -- 172.21.15.43:0/363831923 wait complete. 2024-05-27T07:17:10.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.982+0000 7f6af2c00700 1 Processor -- start 2024-05-27T07:17:10.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.983+0000 7f6af2c00700 1 -- start start 2024-05-27T07:17:10.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.983+0000 7f6af2c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aec105530 0x7f6aec19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.983+0000 7f6af2c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aec105e80 0x7f6aec19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.983+0000 7f6af2c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6aec10e910 0x7f6aec1a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.983+0000 7f6af2c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec114350 con 0x7f6aec105e80 2024-05-27T07:17:10.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.983+0000 7f6af2c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1141d0 con 0x7f6aec10e910 2024-05-27T07:17:10.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.983+0000 7f6af2c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1144d0 con 0x7f6aec105530 2024-05-27T07:17:10.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.984+0000 7f6aeb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aec105e80 0x7f6aec19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.984+0000 7f6aeb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aec105e80 0x7f6aec19d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38588/0 (socket says 172.21.15.43:38588) 2024-05-27T07:17:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.984+0000 7f6aeb400700 1 -- 172.21.15.43:0/1819348788 learned_addr learned my addr 172.21.15.43:0/1819348788 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:10.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.984+0000 7f6aebe00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aec105530 0x7f6aec19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.984+0000 7f6af0c00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6aec10e910 0x7f6aec1a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.984+0000 7f6aeb400700 1 -- 172.21.15.43:0/1819348788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aec105530 msgr2=0x7f6aec19cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.984+0000 7f6aeb400700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aec105530 0x7f6aec19cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.984+0000 7f6aeb400700 1 -- 172.21.15.43:0/1819348788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6aec10e910 msgr2=0x7f6aec1a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.984+0000 7f6aeb400700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6aec10e910 0x7f6aec1a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.984+0000 7f6aeb400700 1 -- 172.21.15.43:0/1819348788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec1a1df0 con 0x7f6aec105e80 2024-05-27T07:17:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.985+0000 7f6aeb400700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aec105e80 0x7f6aec19d4c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800d950 tx=0x7f6ad800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.985+0000 7f6ae8c00700 1 -- 172.21.15.43:0/1819348788 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8016070 con 0x7f6aec105e80 2024-05-27T07:17:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.985+0000 7f6ae8c00700 1 -- 172.21.15.43:0/1819348788 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ad8009dd0 con 0x7f6aec105e80 2024-05-27T07:17:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.985+0000 7f6af2c00700 1 -- 172.21.15.43:0/1819348788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aec1a20e0 con 0x7f6aec105e80 2024-05-27T07:17:10.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.985+0000 7f6ae8c00700 1 -- 172.21.15.43:0/1819348788 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad800f3c0 con 0x7f6aec105e80 2024-05-27T07:17:10.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.985+0000 7f6af2c00700 1 -- 172.21.15.43:0/1819348788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aec1a9a00 con 0x7f6aec105e80 2024-05-27T07:17:10.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.991+0000 7f6b01732b80 1 Processor -- start 2024-05-27T07:17:10.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.991+0000 7f6ae8c00700 1 -- 172.21.15.43:0/1819348788 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6ad8019070 con 0x7f6aec105e80 2024-05-27T07:17:10.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.991+0000 7f6b01732b80 1 -- start start 2024-05-27T07:17:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a6554e40 0x55f2a66d97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d9d30 0x55f2a66d3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66d41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6b01732b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f2a6601e40 con 0x55f2a66d9d30 2024-05-27T07:17:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6b01732b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f2a65f19e0 con 0x55f2a66d3db0 2024-05-27T07:17:10.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6b01732b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f2a65f5a20 con 0x55f2a6554e40 2024-05-27T07:17:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6aebe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d9d30 0x55f2a66d3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6aebe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d9d30 0x55f2a66d3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38600/0 (socket says 172.21.15.43:38600) 2024-05-27T07:17:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6aebe00700 1 -- 172.21.15.43:0/930568292 learned_addr learned my addr 172.21.15.43:0/930568292 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6af0c00700 1 --2- 172.21.15.43:0/930568292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a6554e40 0x55f2a66d97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.992+0000 7f6aeb400700 1 --2- 172.21.15.43:0/930568292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66d41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.993+0000 7f6aebe00700 1 -- 172.21.15.43:0/930568292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a6554e40 msgr2=0x55f2a66d97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.993+0000 7f6aebe00700 1 --2- 172.21.15.43:0/930568292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a6554e40 0x55f2a66d97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.993+0000 7f6aebe00700 1 -- 172.21.15.43:0/930568292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 msgr2=0x55f2a66d41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.993+0000 7f6aebe00700 1 --2- 172.21.15.43:0/930568292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66d41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.993+0000 7f6aebe00700 1 -- 172.21.15.43:0/930568292 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a65ba420 con 0x55f2a66d9d30 2024-05-27T07:17:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.993+0000 7f6aebe00700 1 --2- 172.21.15.43:0/930568292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d9d30 0x55f2a66d3870 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6adc015000 tx=0x7f6adc005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee56c5874c34e7c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.993+0000 7f6ae8c00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6ad00b29f0 0x7f6ad00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.994+0000 7f6ae1600700 1 -- 172.21.15.43:0/930568292 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc027070 con 0x55f2a66d9d30 2024-05-27T07:17:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.994+0000 7f6ae1600700 1 -- 172.21.15.43:0/930568292 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6adc003c50 con 0x55f2a66d9d30 2024-05-27T07:17:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.994+0000 7f6ae1600700 1 -- 172.21.15.43:0/930568292 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc00b9f0 con 0x55f2a66d9d30 2024-05-27T07:17:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.994+0000 7f6ae8c00700 1 -- 172.21.15.43:0/1819348788 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6ad80e2640 con 0x7f6aec105e80 2024-05-27T07:17:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.994+0000 7f6af0c00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6ad00b29f0 0x7f6ad00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.995+0000 7f6af0c00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6ad00b29f0 0x7f6ad00b4f00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55f2a66d4b30 tx=0x7f6ae4009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.995+0000 7f6b01732b80 1 -- 172.21.15.43:0/930568292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d9d30 msgr2=0x55f2a66d3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.995+0000 7f6b01732b80 1 --2- 172.21.15.43:0/930568292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d9d30 0x55f2a66d3870 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6adc015000 tx=0x7f6adc005f00 comp rx=0 tx=0).stop 2024-05-27T07:17:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.995+0000 7f6b01732b80 1 -- 172.21.15.43:0/930568292 shutdown_connections 2024-05-27T07:17:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.995+0000 7f6b01732b80 1 --2- 172.21.15.43:0/930568292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66d41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.995+0000 7f6b01732b80 1 --2- 172.21.15.43:0/930568292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a6554e40 0x55f2a66d97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.995+0000 7f6b01732b80 1 --2- 172.21.15.43:0/930568292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d9d30 0x55f2a66d3870 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.995+0000 7f6b01732b80 1 -- 172.21.15.43:0/930568292 >> 172.21.15.43:0/930568292 conn(0x55f2a65b4fc0 msgr2=0x55f2a66ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:10.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.995+0000 7f6b01732b80 1 -- 172.21.15.43:0/930568292 shutdown_connections 2024-05-27T07:17:10.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.995+0000 7f6b01732b80 1 -- 172.21.15.43:0/930568292 wait complete. 2024-05-27T07:17:10.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.996+0000 7f6b01732b80 1 -- start start 2024-05-27T07:17:10.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.996+0000 7f6b01732b80 1 Processor -- start 2024-05-27T07:17:10.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.997+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66dd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.997+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66dd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.997+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66ddee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.997+0000 7f6b01732b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f2a64f7fa0 con 0x55f2a6554e40 2024-05-27T07:17:10.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.997+0000 7f6b01732b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f2a6592760 con 0x55f2a66d3db0 2024-05-27T07:17:10.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.997+0000 7f6b01732b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f2a66e07a0 con 0x55f2a66d9d30 2024-05-27T07:17:10.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.997+0000 7f6b01732b80 10 client.0 _ll_register_callbacks cb 0x55f2a659bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:10.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.997+0000 7f6aebe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66dd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.997+0000 7f6aebe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66dd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38604/0 (socket says 172.21.15.43:38604) 2024-05-27T07:17:10.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.997+0000 7f6aebe00700 1 -- 172.21.15.43:0/1647252949 learned_addr learned my addr 172.21.15.43:0/1647252949 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:10.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.998+0000 7f6aeb400700 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66dd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.998+0000 7f6af0c00700 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66ddee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.998+0000 7f6aebe00700 1 -- 172.21.15.43:0/1647252949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 msgr2=0x55f2a66ddee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:10.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.998+0000 7f6aebe00700 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66ddee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.998+0000 7f6aebe00700 1 -- 172.21.15.43:0/1647252949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 msgr2=0x55f2a66dd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:10.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.998+0000 7f6aebe00700 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66dd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.998+0000 7f6aebe00700 1 -- 172.21.15.43:0/1647252949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2a65418d0 con 0x55f2a6554e40 2024-05-27T07:17:10.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.998+0000 7f6aeb400700 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66dd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:10.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.999+0000 7f6aebe00700 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66dd460 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0098e0 tx=0x7f6adc005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfde8f8199c877b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:10.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.999+0000 7f6ad6000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:10.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.999+0000 7f6ad6000700 1 -- 172.21.15.43:0/1647252949 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc031060 con 0x55f2a6554e40 2024-05-27T07:17:10.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.999+0000 7f6b01732b80 1 -- 172.21.15.43:0/1647252949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6adc003c50 con 0x55f2a6554e40 2024-05-27T07:17:10.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.999+0000 7f6ad6000700 1 -- 172.21.15.43:0/1647252949 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6adc0039c0 con 0x55f2a6554e40 2024-05-27T07:17:10.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.999+0000 7f6ad6000700 1 -- 172.21.15.43:0/1647252949 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6adc038050 con 0x55f2a6554e40 2024-05-27T07:17:10.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.999+0000 7f6b01732b80 10 client.15966 fetch_fsmap learned FSMap version 46 2024-05-27T07:17:10.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:10.999+0000 7f6b01732b80 1 -- 172.21.15.43:0/1647252949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55f2a65c2060 con 0x55f2a6554e40 2024-05-27T07:17:10.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.000+0000 7f6ad6000700 1 -- 172.21.15.43:0/1647252949 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6adc003c50 con 0x55f2a6554e40 2024-05-27T07:17:10.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.000+0000 7f6ad6000700 1 -- 172.21.15.43:0/1647252949 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 46) ==== 8378+0+0 (secure 0 0 0) 0x7f6adc022700 con 0x55f2a6554e40 2024-05-27T07:17:10.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.000+0000 7f6b01732b80 10 client.15966 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:17:10.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.000+0000 7f6b01732b80 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:17:10.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.000+0000 7f6b01732b80 20 client.15966 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:10.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.000+0000 7f6b01732b80 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f2a66ed4c0 0x55f2a66ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:10.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.000+0000 7f6b01732b80 4 client.15966 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:17:10.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.000+0000 7f6b01732b80 1 -- 172.21.15.43:0/1647252949 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2a6603d90 con 0x55f2a66ed4c0 2024-05-27T07:17:10.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.001+0000 7f6aeb400700 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f2a66ed4c0 0x55f2a66ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:10.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.002+0000 7f6aeb400700 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f2a66ed4c0 0x55f2a66ef8e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c036bab1115ddfd1 server_cookie=7b90beaac96a656c in_seq=0 out_seq=0 2024-05-27T07:17:10.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.002+0000 7f6ad6000700 10 client.15966 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:10.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.004+0000 7f6ad6000700 1 -- 172.21.15.43:0/1647252949 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f2a6603d90 con 0x55f2a66ed4c0 2024-05-27T07:17:10.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.004+0000 7f6ad6000700 10 client.15966 handle_command_reply: tid=0 2024-05-27T07:17:10.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.004+0000 7f6b01732b80 1 client.15966 shutdown 2024-05-27T07:17:10.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 1 -- 172.21.15.43:0/1647252949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 msgr2=0x55f2a66dd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66dd460 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0098e0 tx=0x7f6adc005810 comp rx=0 tx=0).stop 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 1 -- 172.21.15.43:0/1647252949 shutdown_connections 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66ddee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f2a66ed4c0 0x55f2a66ef8e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66dd460 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 1 --2- 172.21.15.43:0/1647252949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66dd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 1 -- 172.21.15.43:0/1647252949 >> 172.21.15.43:0/1647252949 conn(0x55f2a65b4fc0 msgr2=0x55f2a66ce580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6ad6000700 0 client.15966 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 1 -- 172.21.15.43:0/1647252949 shutdown_connections 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 1 -- 172.21.15.43:0/1647252949 wait complete. 2024-05-27T07:17:10.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.005+0000 7f6b01732b80 20 client.15966 trim_cache size 0 max 16384 2024-05-27T07:17:11.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.028+0000 7f6b01732b80 1 Processor -- start 2024-05-27T07:17:11.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.029+0000 7f6b01732b80 1 -- start start 2024-05-27T07:17:11.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.029+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66d3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.029+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66e9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.029+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66df900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.029+0000 7f6b01732b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1b2640 con 0x55f2a6554e40 2024-05-27T07:17:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.029+0000 7f6b01732b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1b4240 con 0x55f2a66d3db0 2024-05-27T07:17:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.029+0000 7f6b01732b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1b2d10 con 0x55f2a66d9d30 2024-05-27T07:17:11.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.029+0000 7f6aebe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66d3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:11.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.029+0000 7f6aebe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66d3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38606/0 (socket says 172.21.15.43:38606) 2024-05-27T07:17:11.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.029+0000 7f6aebe00700 1 -- 172.21.15.43:0/3768088615 learned_addr learned my addr 172.21.15.43:0/3768088615 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:11.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.030+0000 7f6af0c00700 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66df900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:11.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.030+0000 7f6aeb400700 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66e9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:11.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.030+0000 7f6aebe00700 1 -- 172.21.15.43:0/3768088615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 msgr2=0x55f2a66df900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:11.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.030+0000 7f6aebe00700 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66df900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.030+0000 7f6aebe00700 1 -- 172.21.15.43:0/3768088615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 msgr2=0x55f2a66e9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:11.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.030+0000 7f6aebe00700 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66e9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.030+0000 7f6aebe00700 1 -- 172.21.15.43:0/3768088615 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a66de250 con 0x55f2a6554e40 2024-05-27T07:17:11.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.030+0000 7f6af0c00700 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66df900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:11.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.030+0000 7f6aeb400700 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66e9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:11.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.031+0000 7f6aebe00700 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66d3870 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c00 tx=0x7f6adc00b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479b4506273555c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:11.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.031+0000 7f6ae0c00700 1 -- 172.21.15.43:0/3768088615 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc031040 con 0x55f2a6554e40 2024-05-27T07:17:11.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.031+0000 7f6ae0c00700 1 -- 172.21.15.43:0/3768088615 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6adc0037c0 con 0x55f2a6554e40 2024-05-27T07:17:11.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.031+0000 7f6ae0c00700 1 -- 172.21.15.43:0/3768088615 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc027040 con 0x55f2a6554e40 2024-05-27T07:17:11.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.032+0000 7f6b01732b80 1 -- 172.21.15.43:0/3768088615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 msgr2=0x55f2a66d3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:11.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.032+0000 7f6b01732b80 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66d3870 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c00 tx=0x7f6adc00b5d0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.032+0000 7f6b01732b80 1 -- 172.21.15.43:0/3768088615 shutdown_connections 2024-05-27T07:17:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.032+0000 7f6b01732b80 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66df900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.032+0000 7f6b01732b80 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a6554e40 0x55f2a66d3870 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.032+0000 7f6b01732b80 1 --2- 172.21.15.43:0/3768088615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a66d3db0 0x55f2a66e9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.032+0000 7f6b01732b80 1 -- 172.21.15.43:0/3768088615 >> 172.21.15.43:0/3768088615 conn(0x55f2a65b4fc0 msgr2=0x55f2a66d8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.033+0000 7f6b01732b80 1 -- 172.21.15.43:0/3768088615 shutdown_connections 2024-05-27T07:17:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.033+0000 7f6b01732b80 1 -- 172.21.15.43:0/3768088615 wait complete. 2024-05-27T07:17:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.033+0000 7f6b01732b80 1 -- start start 2024-05-27T07:17:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.034+0000 7f6b01732b80 1 Processor -- start 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.034+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a6554e40 0x55f2a66f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.034+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d3db0 0x55f2a66efd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.034+0000 7f6b01732b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66f02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.034+0000 7f6b01732b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1b3b10 con 0x55f2a66d3db0 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.034+0000 7f6b01732b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1b33e0 con 0x55f2a6554e40 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.034+0000 7f6b01732b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1b05c0 con 0x55f2a66d9d30 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.034+0000 7f6b01732b80 10 client.0 _ll_register_callbacks cb 0x55f2a659bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.035+0000 7f6aebe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a6554e40 0x55f2a66f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.035+0000 7f6aebe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a6554e40 0x55f2a66f29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:51838/0 (socket says 172.21.15.43:51838) 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.035+0000 7f6aebe00700 1 -- 172.21.15.43:0/4191177131 learned_addr learned my addr 172.21.15.43:0/4191177131 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.035+0000 7f6aeb400700 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d3db0 0x55f2a66efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:11.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.035+0000 7f6af0c00700 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66f02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.036+0000 7f6aeb400700 1 -- 172.21.15.43:0/4191177131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 msgr2=0x55f2a66f02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.036+0000 7f6aeb400700 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66f02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.036+0000 7f6aeb400700 1 -- 172.21.15.43:0/4191177131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a6554e40 msgr2=0x55f2a66f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.036+0000 7f6aeb400700 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a6554e40 0x55f2a66f29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.036+0000 7f6aeb400700 1 -- 172.21.15.43:0/4191177131 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2a66f07f0 con 0x55f2a66d3db0 2024-05-27T07:17:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.037+0000 7f6aeb400700 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d3db0 0x55f2a66efd70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80df9e0 tx=0x7f6ad8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ed6e5176164243 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.037+0000 7f6ad6000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.037+0000 7f6ad6000700 1 -- 172.21.15.43:0/4191177131 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8020040 con 0x55f2a66d3db0 2024-05-27T07:17:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.037+0000 7f6b01732b80 1 -- 172.21.15.43:0/4191177131 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6adc0037c0 con 0x55f2a66d3db0 2024-05-27T07:17:11.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.037+0000 7f6ad6000700 1 -- 172.21.15.43:0/4191177131 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6ad80106a0 con 0x55f2a66d3db0 2024-05-27T07:17:11.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6ad6000700 1 -- 172.21.15.43:0/4191177131 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6ad80a9570 con 0x55f2a66d3db0 2024-05-27T07:17:11.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6b01732b80 10 client.15972 fetch_fsmap learned FSMap version 46 2024-05-27T07:17:11.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6b01732b80 1 -- 172.21.15.43:0/4191177131 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55f2a6757a50 con 0x55f2a66d3db0 2024-05-27T07:17:11.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6ad6000700 1 -- 172.21.15.43:0/4191177131 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6adc0037c0 con 0x55f2a66d3db0 2024-05-27T07:17:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6ad6000700 1 -- 172.21.15.43:0/4191177131 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 46) ==== 8378+0+0 (secure 0 0 0) 0x7f6ad80a97d0 con 0x55f2a66d3db0 2024-05-27T07:17:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6b01732b80 10 client.15972 fetch_fsmap finished waiting for FSMap version 46 2024-05-27T07:17:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6b01732b80 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:17:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6b01732b80 20 client.15972 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6b01732b80 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f2a6758090 0x55f2a675a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6b01732b80 4 client.15972 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:17:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.038+0000 7f6b01732b80 1 -- 172.21.15.43:0/4191177131 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f2a65fd570 con 0x55f2a6758090 2024-05-27T07:17:11.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.039+0000 7f6aebe00700 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f2a6758090 0x55f2a675a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.040+0000 7f6aebe00700 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f2a6758090 0x55f2a675a4b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77f48b01cfd3f7de server_cookie=f93291d96944471f in_seq=0 out_seq=0 2024-05-27T07:17:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.040+0000 7f6ad6000700 10 client.15972 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6ad6000700 1 -- 172.21.15.43:0/4191177131 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f2a65fd570 con 0x55f2a6758090 2024-05-27T07:17:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6ad6000700 10 client.15972 handle_command_reply: tid=0 2024-05-27T07:17:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6b01732b80 1 client.15972 shutdown 2024-05-27T07:17:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6b01732b80 1 -- 172.21.15.43:0/4191177131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d3db0 msgr2=0x55f2a66efd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6b01732b80 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d3db0 0x55f2a66efd70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80df9e0 tx=0x7f6ad8005dc0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6b01732b80 1 -- 172.21.15.43:0/4191177131 shutdown_connections 2024-05-27T07:17:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6b01732b80 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2a66d9d30 0x55f2a66f02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6b01732b80 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2a6554e40 0x55f2a66f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6b01732b80 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f2a6758090 0x55f2a675a4b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6b01732b80 1 --2- 172.21.15.43:0/4191177131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2a66d3db0 0x55f2a66efd70 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.041+0000 7f6b01732b80 1 -- 172.21.15.43:0/4191177131 >> 172.21.15.43:0/4191177131 conn(0x55f2a65b4fc0 msgr2=0x55f2a66d7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.042+0000 7f6ad6000700 0 client.15972 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:11.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.042+0000 7f6b01732b80 1 -- 172.21.15.43:0/4191177131 shutdown_connections 2024-05-27T07:17:11.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.042+0000 7f6b01732b80 1 -- 172.21.15.43:0/4191177131 wait complete. 2024-05-27T07:17:11.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.042+0000 7f6b01732b80 20 client.15972 trim_cache size 0 max 16384 2024-05-27T07:17:11.033 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:17:11.033 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:17:11.033 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:17:11.034 INFO:teuthology.orchestra.run.smithi043.stdout: "7c2441f4-56c2-441a-a3d3-1abca46b5d64": { 2024-05-27T07:17:11.034 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:17:11.034 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "7c2441f4-56c2-441a-a3d3-1abca46b5d64", 2024-05-27T07:17:11.034 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:17:11.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:17:11.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:17:11.034 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:17:11.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.043+0000 7f6af2c00700 1 -- 172.21.15.43:0/1819348788 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6ad00b29f0 msgr2=0x7f6ad00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.043+0000 7f6af2c00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6ad00b29f0 0x7f6ad00b4f00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55f2a66d4b30 tx=0x7f6ae4009040 comp rx=0 tx=0).stop 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.043+0000 7f6af2c00700 1 -- 172.21.15.43:0/1819348788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aec105e80 msgr2=0x7f6aec19d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.043+0000 7f6af2c00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aec105e80 0x7f6aec19d4c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800d950 tx=0x7f6ad800dc60 comp rx=0 tx=0).stop 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.043+0000 7f6af2c00700 1 -- 172.21.15.43:0/1819348788 shutdown_connections 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.043+0000 7f6af2c00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6ad00b29f0 0x7f6ad00b4f00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.043+0000 7f6af2c00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6aec10e910 0x7f6aec1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.043+0000 7f6af2c00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aec105530 0x7f6aec19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.043+0000 7f6af2c00700 1 --2- 172.21.15.43:0/1819348788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6aec105e80 0x7f6aec19d4c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.043+0000 7f6af2c00700 1 -- 172.21.15.43:0/1819348788 >> 172.21.15.43:0/1819348788 conn(0x7f6aec0fee40 msgr2=0x7f6aec1031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.044+0000 7f6af2c00700 1 -- 172.21.15.43:0/1819348788 shutdown_connections 2024-05-27T07:17:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:11.044+0000 7f6af2c00700 1 -- 172.21.15.43:0/1819348788 wait complete. 2024-05-27T07:17:11.181 INFO:tasks.workunit.client.0.smithi043.stdout: 1 9581 0.89 MB/sec warmup 75 sec latency 48.840 ms 2024-05-27T07:17:11.202 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:11 smithi043 podman[97490]: 2024-05-27 07:17:11.034669272 +0000 UTC m=+0.899821127 container init a4ec09669ee058322333bd82362dbe4acd0f2f3cc5dec1bd085ece20dc96c1cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-27T07:17:11.202 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:11 smithi043 podman[97490]: 2024-05-27 07:17:11.046307021 +0000 UTC m=+0.911458860 container start a4ec09669ee058322333bd82362dbe4acd0f2f3cc5dec1bd085ece20dc96c1cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-27T07:17:11.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:11 smithi043 bash[97490]: a4ec09669ee058322333bd82362dbe4acd0f2f3cc5dec1bd085ece20dc96c1cb 2024-05-27T07:17:11.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:11 smithi043 systemd[1]: Started Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:11.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:11 smithi043 ceph-mds[97564]: -- 172.21.15.43:0/2468004212 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5637f0b99a40 con 0x5637f0be1600 2024-05-27T07:17:11.203 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:11 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: starting mds.g at 2024-05-27T07:17:12.181 INFO:tasks.workunit.client.0.smithi043.stdout: 1 9683 0.88 MB/sec warmup 76 sec latency 192.088 ms 2024-05-27T07:17:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:12 smithi132 ceph-mon[35283]: pgmap v108: 129 pgs: 129 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 234 KiB/s wr, 56 op/s 2024-05-27T07:17:12.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:12 smithi132 ceph-mon[35283]: daemon mds.g restarted 2024-05-27T07:17:12.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:12 smithi043 ceph-mon[31531]: pgmap v108: 129 pgs: 129 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 234 KiB/s wr, 56 op/s 2024-05-27T07:17:12.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:12 smithi043 ceph-mon[31531]: daemon mds.g restarted 2024-05-27T07:17:12.454 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fed2f800700 time 2024-05-27T07:17:12.171481+0000 2024-05-27T07:17:12.454 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:12.454 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:12.454 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fed45337ba6] 2024-05-27T07:17:12.454 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fed45337dd1] 2024-05-27T07:17:12.454 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5637bf73223d] 2024-05-27T07:17:12.454 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5637bf71d971] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 5: (SnapRealm::check_cache() const+0x13e) [0x5637bf71bfa0] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5637bf71c651] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5637bf6b1347] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5637bf3ad87e] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5637bf3b2e94] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5637bf3bd969] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 11: (MDLog::_replay_thread()+0x1f38) [0x5637bf75648c] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5637bf75f9cb] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 13: (Thread::entry_wrapper()+0x43) [0x7fed4530672d] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 14: (Thread::_entry_func(void*)+0xd) [0x7fed45306749] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fed43eaa1ca] 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 16: clone() 2024-05-27T07:17:12.455 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.173+0000 7fed2f800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fed2f800700 time 2024-05-27T07:17:12.171481+0000 2024-05-27T07:17:12.456 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:12.456 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.456 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:12.456 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fed45337ba6] 2024-05-27T07:17:12.456 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fed45337dd1] 2024-05-27T07:17:12.456 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5637bf73223d] 2024-05-27T07:17:12.456 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5637bf71d971] 2024-05-27T07:17:12.456 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 5: (SnapRealm::check_cache() const+0x13e) [0x5637bf71bfa0] 2024-05-27T07:17:12.456 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5637bf71c651] 2024-05-27T07:17:12.456 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5637bf6b1347] 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5637bf3ad87e] 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5637bf3b2e94] 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5637bf3bd969] 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 11: (MDLog::_replay_thread()+0x1f38) [0x5637bf75648c] 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5637bf75f9cb] 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 13: (Thread::entry_wrapper()+0x43) [0x7fed4530672d] 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 14: (Thread::_entry_func(void*)+0xd) [0x7fed45306749] 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fed43eaa1ca] 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 16: clone() 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: *** Caught signal (Aborted) ** 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: in thread 7fed2f800700 thread_name:md_log_replay 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fed43eb4d20] 2024-05-27T07:17:12.457 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2: gsignal() 2024-05-27T07:17:12.458 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 3: abort() 2024-05-27T07:17:12.458 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fed45337ced] 2024-05-27T07:17:12.458 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fed45337dd1] 2024-05-27T07:17:12.458 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5637bf73223d] 2024-05-27T07:17:12.458 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5637bf71d971] 2024-05-27T07:17:12.458 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 8: (SnapRealm::check_cache() const+0x13e) [0x5637bf71bfa0] 2024-05-27T07:17:12.458 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5637bf71c651] 2024-05-27T07:17:12.458 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5637bf6b1347] 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5637bf3ad87e] 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5637bf3b2e94] 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5637bf3bd969] 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 14: (MDLog::_replay_thread()+0x1f38) [0x5637bf75648c] 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5637bf75f9cb] 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 16: (Thread::entry_wrapper()+0x43) [0x7fed4530672d] 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 17: (Thread::_entry_func(void*)+0xd) [0x7fed45306749] 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fed43eaa1ca] 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 19: clone() 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.178+0000 7fed2f800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: in thread 7fed2f800700 thread_name:md_log_replay 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.459 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:12.460 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fed43eb4d20] 2024-05-27T07:17:12.460 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2: gsignal() 2024-05-27T07:17:12.460 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 3: abort() 2024-05-27T07:17:12.460 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fed45337ced] 2024-05-27T07:17:12.460 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fed45337dd1] 2024-05-27T07:17:12.460 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5637bf73223d] 2024-05-27T07:17:12.460 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5637bf71d971] 2024-05-27T07:17:12.460 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 8: (SnapRealm::check_cache() const+0x13e) [0x5637bf71bfa0] 2024-05-27T07:17:12.460 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5637bf71c651] 2024-05-27T07:17:12.460 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5637bf6b1347] 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:12 smithi179 ceph-mon[35888]: pgmap v108: 129 pgs: 129 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 234 KiB/s wr, 56 op/s 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:12 smithi179 ceph-mon[35888]: daemon mds.g restarted 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5637bf3ad87e] 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5637bf3b2e94] 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5637bf3bd969] 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 14: (MDLog::_replay_thread()+0x1f38) [0x5637bf75648c] 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5637bf75f9cb] 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 16: (Thread::entry_wrapper()+0x43) [0x7fed4530672d] 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 17: (Thread::_entry_func(void*)+0xd) [0x7fed45306749] 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fed43eaa1ca] 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 19: clone() 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:12.461 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: -1> 2024-05-27T07:17:12.173+0000 7fed2f800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fed2f800700 time 2024-05-27T07:17:12.171481+0000 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fed45337ba6] 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fed45337dd1] 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5637bf73223d] 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5637bf71d971] 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 5: (SnapRealm::check_cache() const+0x13e) [0x5637bf71bfa0] 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5637bf71c651] 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5637bf6b1347] 2024-05-27T07:17:12.462 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5637bf3ad87e] 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5637bf3b2e94] 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5637bf3bd969] 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 11: (MDLog::_replay_thread()+0x1f38) [0x5637bf75648c] 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5637bf75f9cb] 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 13: (Thread::entry_wrapper()+0x43) [0x7fed4530672d] 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 14: (Thread::_entry_func(void*)+0xd) [0x7fed45306749] 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fed43eaa1ca] 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 16: clone() 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 0> 2024-05-27T07:17:12.178+0000 7fed2f800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: in thread 7fed2f800700 thread_name:md_log_replay 2024-05-27T07:17:12.463 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fed43eb4d20] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2: gsignal() 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 3: abort() 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fed45337ced] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fed45337dd1] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5637bf73223d] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5637bf71d971] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 8: (SnapRealm::check_cache() const+0x13e) [0x5637bf71bfa0] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5637bf71c651] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5637bf6b1347] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5637bf3ad87e] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5637bf3b2e94] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5637bf3bd969] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 14: (MDLog::_replay_thread()+0x1f38) [0x5637bf75648c] 2024-05-27T07:17:12.464 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5637bf75f9cb] 2024-05-27T07:17:12.465 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 16: (Thread::entry_wrapper()+0x43) [0x7fed4530672d] 2024-05-27T07:17:12.465 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 17: (Thread::_entry_func(void*)+0xd) [0x7fed45306749] 2024-05-27T07:17:12.465 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fed43eaa1ca] 2024-05-27T07:17:12.465 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 19: clone() 2024-05-27T07:17:12.465 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:12.465 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.465 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: -1> 2024-05-27T07:17:12.173+0000 7fed2f800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fed2f800700 time 2024-05-27T07:17:12.171481+0000 2024-05-27T07:17:12.465 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:12.465 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.465 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fed45337ba6] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fed45337dd1] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5637bf73223d] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5637bf71d971] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 5: (SnapRealm::check_cache() const+0x13e) [0x5637bf71bfa0] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5637bf71c651] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5637bf6b1347] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5637bf3ad87e] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5637bf3b2e94] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5637bf3bd969] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 11: (MDLog::_replay_thread()+0x1f38) [0x5637bf75648c] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5637bf75f9cb] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 13: (Thread::entry_wrapper()+0x43) [0x7fed4530672d] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 14: (Thread::_entry_func(void*)+0xd) [0x7fed45306749] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fed43eaa1ca] 2024-05-27T07:17:12.466 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 16: clone() 2024-05-27T07:17:12.467 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.467 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 0> 2024-05-27T07:17:12.178+0000 7fed2f800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:12.467 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: in thread 7fed2f800700 thread_name:md_log_replay 2024-05-27T07:17:12.467 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.467 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:12.467 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fed43eb4d20] 2024-05-27T07:17:12.467 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2: gsignal() 2024-05-27T07:17:12.467 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 3: abort() 2024-05-27T07:17:12.467 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fed45337ced] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fed45337dd1] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5637bf73223d] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5637bf71d971] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 8: (SnapRealm::check_cache() const+0x13e) [0x5637bf71bfa0] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5637bf71c651] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5637bf6b1347] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5637bf3ad87e] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5637bf3b2e94] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5637bf3bd969] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 14: (MDLog::_replay_thread()+0x1f38) [0x5637bf75648c] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5637bf75f9cb] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 16: (Thread::entry_wrapper()+0x43) [0x7fed4530672d] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 17: (Thread::_entry_func(void*)+0xd) [0x7fed45306749] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fed43eaa1ca] 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 19: clone() 2024-05-27T07:17:12.468 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:12.469 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97514]: 2024-05-27T07:17:12.888 INFO:tasks.cephfs.filesystem:scrub status for tag:7c2441f4-56c2-441a-a3d3-1abca46b5d64 - {'path': '/', 'tag': '7c2441f4-56c2-441a-a3d3-1abca46b5d64', 'options': 'recursive,force'} 2024-05-27T07:17:13.079 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:12 smithi043 podman[97610]: 2024-05-27 07:17:12.912921697 +0000 UTC m=+0.679401887 container died a4ec09669ee058322333bd82362dbe4acd0f2f3cc5dec1bd085ece20dc96c1cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T07:17:13.181 INFO:tasks.workunit.client.0.smithi043.stdout: 1 9864 0.91 MB/sec warmup 77 sec latency 57.714 ms 2024-05-27T07:17:13.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:13 smithi132 ceph-mon[35283]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:17:13.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:13 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6838/495866589,v1:172.21.15.43:6839/495866589] up:boot 2024-05-27T07:17:13.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:13 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:17:13.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:13 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:17:13.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:13 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:13.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:13 smithi043 ceph-mon[31531]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:17:13.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:13 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6838/495866589,v1:172.21.15.43:6839/495866589] up:boot 2024-05-27T07:17:13.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:13 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:17:13.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:13 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:17:13.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:13 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:13 smithi179 ceph-mon[35888]: osdmap e99: 12 total, 12 up, 12 in 2024-05-27T07:17:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:13 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6838/495866589,v1:172.21.15.43:6839/495866589] up:boot 2024-05-27T07:17:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:13 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-05-27T07:17:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:13 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:17:13.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:13 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:13.952 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:13 smithi043 podman[97610]: 2024-05-27 07:17:13.648522088 +0000 UTC m=+1.415002291 container remove a4ec09669ee058322333bd82362dbe4acd0f2f3cc5dec1bd085ece20dc96c1cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:17:13.953 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:13 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:17:14.181 INFO:tasks.workunit.client.0.smithi043.stdout: 1 10070 0.92 MB/sec warmup 78 sec latency 52.411 ms 2024-05-27T07:17:14.452 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:14 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:17:14.453 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:14 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.629s CPU time. 2024-05-27T07:17:14.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:14 smithi043 ceph-mon[31531]: pgmap v110: 129 pgs: 129 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 292 KiB/s wr, 70 op/s 2024-05-27T07:17:14.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:14 smithi179 ceph-mon[35888]: pgmap v110: 129 pgs: 129 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 292 KiB/s wr, 70 op/s 2024-05-27T07:17:14.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:14 smithi132 ceph-mon[35283]: pgmap v110: 129 pgs: 129 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 292 KiB/s wr, 70 op/s 2024-05-27T07:17:15.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 10276 0.92 MB/sec warmup 79 sec latency 53.480 ms 2024-05-27T07:17:16.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 10390 0.92 MB/sec warmup 80 sec latency 56.676 ms 2024-05-27T07:17:16.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:16 smithi043 ceph-mon[31531]: pgmap v111: 129 pgs: 129 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 188 KiB/s wr, 25 op/s 2024-05-27T07:17:16.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:16 smithi179 ceph-mon[35888]: pgmap v111: 129 pgs: 129 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 188 KiB/s wr, 25 op/s 2024-05-27T07:17:16.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:16 smithi132 ceph-mon[35283]: pgmap v111: 129 pgs: 129 active+clean; 30 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 188 KiB/s wr, 25 op/s 2024-05-27T07:17:17.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 10419 0.91 MB/sec warmup 81 sec latency 59.985 ms 2024-05-27T07:17:17.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:17 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:17:17.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:17 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:17:17.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:17.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:17 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:17:17.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:17 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:17:17.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:17.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:17 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:17:17.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:17 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:17:17.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:17:18.139 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:17 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:17:18.139 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:17 smithi179 systemd[1]: Stopped Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:18.139 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:17 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.577s CPU time. 2024-05-27T07:17:18.139 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:17 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Start request repeated too quickly. 2024-05-27T07:17:18.139 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:17 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-05-27T07:17:18.139 INFO:journalctl@ceph.mds.c.smithi179.stdout:May 27 07:17:17 smithi179 systemd[1]: Failed to start Ceph mds.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:18.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 10476 0.90 MB/sec warmup 82 sec latency 55.981 ms 2024-05-27T07:17:18.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:18 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:17:18.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:18 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:17:18.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:18 smithi043 ceph-mon[31531]: pgmap v112: 129 pgs: 129 active+clean; 33 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 452 KiB/s wr, 46 op/s 2024-05-27T07:17:18.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:18 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:17:18.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:18 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:17:18.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:18 smithi179 ceph-mon[35888]: pgmap v112: 129 pgs: 129 active+clean; 33 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 452 KiB/s wr, 46 op/s 2024-05-27T07:17:18.492 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:18 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:17:18.492 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:18 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:17:18.493 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:18 smithi132 ceph-mon[35283]: pgmap v112: 129 pgs: 129 active+clean; 33 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 452 KiB/s wr, 46 op/s 2024-05-27T07:17:19.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 10658 0.92 MB/sec warmup 83 sec latency 52.721 ms 2024-05-27T07:17:19.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:19 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:19.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:19 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:19.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:19 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:17:19.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:19 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:19.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:19 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:19.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:19 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:17:19.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:19 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:17:19.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:19 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:17:19.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:19 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:17:20.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 10775 0.92 MB/sec warmup 84 sec latency 58.939 ms 2024-05-27T07:17:20.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:20 smithi043 ceph-mon[31531]: pgmap v113: 129 pgs: 129 active+clean; 33 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 452 KiB/s wr, 46 op/s 2024-05-27T07:17:20.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:20 smithi179 ceph-mon[35888]: pgmap v113: 129 pgs: 129 active+clean; 33 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 452 KiB/s wr, 46 op/s 2024-05-27T07:17:20.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:20 smithi132 ceph-mon[35283]: pgmap v113: 129 pgs: 129 active+clean; 33 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 452 KiB/s wr, 46 op/s 2024-05-27T07:17:21.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 10879 0.92 MB/sec warmup 85 sec latency 59.796 ms 2024-05-27T07:17:21.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:21 smithi043 ceph-mon[31531]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:17:21.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:21 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:21.454 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4d1a800700 time 2024-05-27T07:17:21.123192+0000 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4d30300ba6] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d30300dd1] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b93f523d] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5634b93e0971] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 5: (SnapRealm::check_cache() const+0x13e) [0x5634b93defa0] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b93df651] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9374347] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b907087e] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9075e94] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5634b9080969] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 11: (MDLog::_replay_thread()+0x1f38) [0x5634b941948c] 2024-05-27T07:17:21.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5634b94229cb] 2024-05-27T07:17:21.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 13: (Thread::entry_wrapper()+0x43) [0x7f4d302cf72d] 2024-05-27T07:17:21.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4d302cf749] 2024-05-27T07:17:21.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4d2ee731ca] 2024-05-27T07:17:21.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 16: clone() 2024-05-27T07:17:21.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: *** Caught signal (Aborted) ** 2024-05-27T07:17:21.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: in thread 7f4d1a800700 thread_name:md_log_replay 2024-05-27T07:17:21.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.126+0000 7f4d1a800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4d1a800700 time 2024-05-27T07:17:21.123192+0000 2024-05-27T07:17:21.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:21.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4d30300ba6] 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d30300dd1] 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b93f523d] 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5634b93e0971] 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 5: (SnapRealm::check_cache() const+0x13e) [0x5634b93defa0] 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b93df651] 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9374347] 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b907087e] 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9075e94] 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5634b9080969] 2024-05-27T07:17:21.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 11: (MDLog::_replay_thread()+0x1f38) [0x5634b941948c] 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5634b94229cb] 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 13: (Thread::entry_wrapper()+0x43) [0x7f4d302cf72d] 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4d302cf749] 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4d2ee731ca] 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 16: clone() 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d2ee7dd20] 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2: gsignal() 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 3: abort() 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4d30300ced] 2024-05-27T07:17:21.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d30300dd1] 2024-05-27T07:17:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:21 smithi179 ceph-mon[35888]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:17:21.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:21 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:21.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b93f523d] 2024-05-27T07:17:21.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5634b93e0971] 2024-05-27T07:17:21.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 8: (SnapRealm::check_cache() const+0x13e) [0x5634b93defa0] 2024-05-27T07:17:21.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b93df651] 2024-05-27T07:17:21.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9374347] 2024-05-27T07:17:21.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b907087e] 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9075e94] 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5634b9080969] 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 14: (MDLog::_replay_thread()+0x1f38) [0x5634b941948c] 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5634b94229cb] 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 16: (Thread::entry_wrapper()+0x43) [0x7f4d302cf72d] 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4d302cf749] 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4d2ee731ca] 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 19: clone() 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.130+0000 7f4d1a800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: in thread 7f4d1a800700 thread_name:md_log_replay 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d2ee7dd20] 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2: gsignal() 2024-05-27T07:17:21.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 3: abort() 2024-05-27T07:17:21.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4d30300ced] 2024-05-27T07:17:21.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d30300dd1] 2024-05-27T07:17:21.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b93f523d] 2024-05-27T07:17:21.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5634b93e0971] 2024-05-27T07:17:21.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 8: (SnapRealm::check_cache() const+0x13e) [0x5634b93defa0] 2024-05-27T07:17:21.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b93df651] 2024-05-27T07:17:21.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9374347] 2024-05-27T07:17:21.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b907087e] 2024-05-27T07:17:21.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9075e94] 2024-05-27T07:17:21.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5634b9080969] 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 14: (MDLog::_replay_thread()+0x1f38) [0x5634b941948c] 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5634b94229cb] 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 16: (Thread::entry_wrapper()+0x43) [0x7f4d302cf72d] 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4d302cf749] 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4d2ee731ca] 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 19: clone() 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: -1> 2024-05-27T07:17:21.126+0000 7f4d1a800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4d1a800700 time 2024-05-27T07:17:21.123192+0000 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:21.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4d30300ba6] 2024-05-27T07:17:21.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d30300dd1] 2024-05-27T07:17:21.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b93f523d] 2024-05-27T07:17:21.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5634b93e0971] 2024-05-27T07:17:21.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 5: (SnapRealm::check_cache() const+0x13e) [0x5634b93defa0] 2024-05-27T07:17:21.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b93df651] 2024-05-27T07:17:21.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9374347] 2024-05-27T07:17:21.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b907087e] 2024-05-27T07:17:21.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9075e94] 2024-05-27T07:17:21.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5634b9080969] 2024-05-27T07:17:21.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 11: (MDLog::_replay_thread()+0x1f38) [0x5634b941948c] 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5634b94229cb] 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 13: (Thread::entry_wrapper()+0x43) [0x7f4d302cf72d] 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4d302cf749] 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4d2ee731ca] 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 16: clone() 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 0> 2024-05-27T07:17:21.130+0000 7f4d1a800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: in thread 7f4d1a800700 thread_name:md_log_replay 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d2ee7dd20] 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2: gsignal() 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 3: abort() 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4d30300ced] 2024-05-27T07:17:21.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d30300dd1] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b93f523d] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5634b93e0971] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 8: (SnapRealm::check_cache() const+0x13e) [0x5634b93defa0] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b93df651] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9374347] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b907087e] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9075e94] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5634b9080969] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 14: (MDLog::_replay_thread()+0x1f38) [0x5634b941948c] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5634b94229cb] 2024-05-27T07:17:21.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 16: (Thread::entry_wrapper()+0x43) [0x7f4d302cf72d] 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4d302cf749] 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4d2ee731ca] 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 19: clone() 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: -1> 2024-05-27T07:17:21.126+0000 7f4d1a800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4d1a800700 time 2024-05-27T07:17:21.123192+0000 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4d30300ba6] 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d30300dd1] 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b93f523d] 2024-05-27T07:17:21.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5634b93e0971] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 5: (SnapRealm::check_cache() const+0x13e) [0x5634b93defa0] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b93df651] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9374347] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b907087e] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9075e94] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5634b9080969] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 11: (MDLog::_replay_thread()+0x1f38) [0x5634b941948c] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5634b94229cb] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 13: (Thread::entry_wrapper()+0x43) [0x7f4d302cf72d] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 14: (Thread::_entry_func(void*)+0xd) [0x7f4d302cf749] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f4d2ee731ca] 2024-05-27T07:17:21.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 16: clone() 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 0> 2024-05-27T07:17:21.130+0000 7f4d1a800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: in thread 7f4d1a800700 thread_name:md_log_replay 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d2ee7dd20] 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2: gsignal() 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 3: abort() 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4d30300ced] 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4d30300dd1] 2024-05-27T07:17:21.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5634b93f523d] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5634b93e0971] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 8: (SnapRealm::check_cache() const+0x13e) [0x5634b93defa0] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5634b93df651] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5634b9374347] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5634b907087e] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5634b9075e94] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5634b9080969] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 14: (MDLog::_replay_thread()+0x1f38) [0x5634b941948c] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5634b94229cb] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 16: (Thread::entry_wrapper()+0x43) [0x7f4d302cf72d] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 17: (Thread::_entry_func(void*)+0xd) [0x7f4d302cf749] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f4d2ee731ca] 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 19: clone() 2024-05-27T07:17:21.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:21.470 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[69434]: 2024-05-27T07:17:21.470 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:21 smithi043 podman[97791]: 2024-05-27 07:17:21.272318285 +0000 UTC m=+0.040179979 container died d390a0d3996b1f15a11e739cd567d555832d7b4091e91ce8b6f996835013e269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T07:17:21.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:21 smithi132 ceph-mon[35283]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-05-27T07:17:21.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:21 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:22.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 10932 0.91 MB/sec warmup 86 sec latency 52.987 ms 2024-05-27T07:17:22.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:22 smithi043 ceph-mon[31531]: pgmap v114: 129 pgs: 129 active+clean; 34 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 446 KiB/s wr, 41 op/s 2024-05-27T07:17:22.452 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:22 smithi043 podman[97791]: 2024-05-27 07:17:22.113605139 +0000 UTC m=+0.881466851 container remove d390a0d3996b1f15a11e739cd567d555832d7b4091e91ce8b6f996835013e269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240521) 2024-05-27T07:17:22.453 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:22 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:17:22.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:22 smithi179 ceph-mon[35888]: pgmap v114: 129 pgs: 129 active+clean; 34 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 446 KiB/s wr, 41 op/s 2024-05-27T07:17:22.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:22 smithi132 ceph-mon[35283]: pgmap v114: 129 pgs: 129 active+clean; 34 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 446 KiB/s wr, 41 op/s 2024-05-27T07:17:22.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:22 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:17:22.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:22 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 2.020s CPU time. 2024-05-27T07:17:23.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 11004 0.90 MB/sec warmup 87 sec latency 52.987 ms 2024-05-27T07:17:24.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 11056 0.89 MB/sec warmup 88 sec latency 50.416 ms 2024-05-27T07:17:24.435 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:24 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:17:24.436 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:24 smithi043 systemd[1]: Stopped Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:24.436 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:24 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.629s CPU time. 2024-05-27T07:17:24.436 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:24 smithi043 systemd[1]: Starting Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:17:24.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:24 smithi043 ceph-mon[31531]: pgmap v115: 129 pgs: 129 active+clean; 34 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 426 KiB/s wr, 40 op/s 2024-05-27T07:17:24.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:24 smithi179 ceph-mon[35888]: pgmap v115: 129 pgs: 129 active+clean; 34 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 426 KiB/s wr, 40 op/s 2024-05-27T07:17:24.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:24 smithi132 ceph-mon[35283]: pgmap v115: 129 pgs: 129 active+clean; 34 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 426 KiB/s wr, 40 op/s 2024-05-27T07:17:24.702 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:24 smithi043 podman[97888]: 2024-05-27 07:17:24.521006885 +0000 UTC m=+0.035153365 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:17:24.702 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:24 smithi043 podman[97888]: 2024-05-27 07:17:24.678600321 +0000 UTC m=+0.192746871 container create 4128f38957a76f0ae5285af6c9487de2bd9fe290510b0b05b2d5ab2a71f07f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:17:25.182 INFO:tasks.workunit.client.0.smithi043.stdout: 1 11124 0.88 MB/sec warmup 89 sec latency 40.473 ms 2024-05-27T07:17:25.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:25 smithi043 ceph-mon[31531]: daemon mds.g restarted 2024-05-27T07:17:25.262 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 podman[97888]: 2024-05-27 07:17:25.021392882 +0000 UTC m=+0.535539406 container init 4128f38957a76f0ae5285af6c9487de2bd9fe290510b0b05b2d5ab2a71f07f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:17:25.262 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 podman[97888]: 2024-05-27 07:17:25.033900107 +0000 UTC m=+0.548046651 container start 4128f38957a76f0ae5285af6c9487de2bd9fe290510b0b05b2d5ab2a71f07f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 bash[97888]: 4128f38957a76f0ae5285af6c9487de2bd9fe290510b0b05b2d5ab2a71f07f58 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 systemd[1]: Started Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-mds[97908]: -- 172.21.15.43:0/4031255386 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562af1b0fa40 con 0x562af1b57180 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: starting mds.g at 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff119000700 time 2024-05-27T07:17:25.269981+0000 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff12eb92ba6] 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff12eb92dd1] 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ac533623d] 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562ac5321971] 2024-05-27T07:17:25.263 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 5: (SnapRealm::check_cache() const+0x13e) [0x562ac531ffa0] 2024-05-27T07:17:25.264 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ac5320651] 2024-05-27T07:17:25.264 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ac52b5347] 2024-05-27T07:17:25.264 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ac4fb187e] 2024-05-27T07:17:25.264 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ac4fb6e94] 2024-05-27T07:17:25.264 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562ac4fc1969] 2024-05-27T07:17:25.264 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 11: (MDLog::_replay_thread()+0x1f38) [0x562ac535a48c] 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562ac53639cb] 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 13: (Thread::entry_wrapper()+0x43) [0x7ff12eb6172d] 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff12eb61749] 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff12d7051ca] 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 16: clone() 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.271+0000 7ff119000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff119000700 time 2024-05-27T07:17:25.269981+0000 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff12eb92ba6] 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff12eb92dd1] 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ac533623d] 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562ac5321971] 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 5: (SnapRealm::check_cache() const+0x13e) [0x562ac531ffa0] 2024-05-27T07:17:25.265 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ac5320651] 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ac52b5347] 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ac4fb187e] 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ac4fb6e94] 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562ac4fc1969] 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 11: (MDLog::_replay_thread()+0x1f38) [0x562ac535a48c] 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562ac53639cb] 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 13: (Thread::entry_wrapper()+0x43) [0x7ff12eb6172d] 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff12eb61749] 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff12d7051ca] 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 16: clone() 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: *** Caught signal (Aborted) ** 2024-05-27T07:17:25.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: in thread 7ff119000700 thread_name:md_log_replay 2024-05-27T07:17:25.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:25 smithi179 ceph-mon[35888]: daemon mds.g restarted 2024-05-27T07:17:25.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:25 smithi132 ceph-mon[35283]: daemon mds.g restarted 2024-05-27T07:17:25.673 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:25.673 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff12d70fd20] 2024-05-27T07:17:25.673 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2: gsignal() 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 3: abort() 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff12eb92ced] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff12eb92dd1] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ac533623d] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562ac5321971] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 8: (SnapRealm::check_cache() const+0x13e) [0x562ac531ffa0] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ac5320651] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ac52b5347] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ac4fb187e] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ac4fb6e94] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562ac4fc1969] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 14: (MDLog::_replay_thread()+0x1f38) [0x562ac535a48c] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562ac53639cb] 2024-05-27T07:17:25.674 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 16: (Thread::entry_wrapper()+0x43) [0x7ff12eb6172d] 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff12eb61749] 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff12d7051ca] 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 19: clone() 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.275+0000 7ff119000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: in thread 7ff119000700 thread_name:md_log_replay 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff12d70fd20] 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2: gsignal() 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 3: abort() 2024-05-27T07:17:25.675 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff12eb92ced] 2024-05-27T07:17:25.678 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff12eb92dd1] 2024-05-27T07:17:25.678 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ac533623d] 2024-05-27T07:17:25.678 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562ac5321971] 2024-05-27T07:17:25.678 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 8: (SnapRealm::check_cache() const+0x13e) [0x562ac531ffa0] 2024-05-27T07:17:25.678 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ac5320651] 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ac52b5347] 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ac4fb187e] 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ac4fb6e94] 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562ac4fc1969] 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 14: (MDLog::_replay_thread()+0x1f38) [0x562ac535a48c] 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562ac53639cb] 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 16: (Thread::entry_wrapper()+0x43) [0x7ff12eb6172d] 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff12eb61749] 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff12d7051ca] 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 19: clone() 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: -1> 2024-05-27T07:17:25.271+0000 7ff119000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff119000700 time 2024-05-27T07:17:25.269981+0000 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:25.679 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.680 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:25.680 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff12eb92ba6] 2024-05-27T07:17:25.680 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff12eb92dd1] 2024-05-27T07:17:25.680 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ac533623d] 2024-05-27T07:17:25.684 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562ac5321971] 2024-05-27T07:17:25.684 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 5: (SnapRealm::check_cache() const+0x13e) [0x562ac531ffa0] 2024-05-27T07:17:25.684 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ac5320651] 2024-05-27T07:17:25.684 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ac52b5347] 2024-05-27T07:17:25.684 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ac4fb187e] 2024-05-27T07:17:25.684 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ac4fb6e94] 2024-05-27T07:17:25.684 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562ac4fc1969] 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 11: (MDLog::_replay_thread()+0x1f38) [0x562ac535a48c] 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562ac53639cb] 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 13: (Thread::entry_wrapper()+0x43) [0x7ff12eb6172d] 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff12eb61749] 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff12d7051ca] 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 16: clone() 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 0> 2024-05-27T07:17:25.275+0000 7ff119000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: in thread 7ff119000700 thread_name:md_log_replay 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff12d70fd20] 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2: gsignal() 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 3: abort() 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff12eb92ced] 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff12eb92dd1] 2024-05-27T07:17:25.685 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ac533623d] 2024-05-27T07:17:25.686 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562ac5321971] 2024-05-27T07:17:25.686 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 8: (SnapRealm::check_cache() const+0x13e) [0x562ac531ffa0] 2024-05-27T07:17:25.686 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ac5320651] 2024-05-27T07:17:25.686 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ac52b5347] 2024-05-27T07:17:25.688 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ac4fb187e] 2024-05-27T07:17:25.688 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ac4fb6e94] 2024-05-27T07:17:25.688 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562ac4fc1969] 2024-05-27T07:17:25.688 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 14: (MDLog::_replay_thread()+0x1f38) [0x562ac535a48c] 2024-05-27T07:17:25.688 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562ac53639cb] 2024-05-27T07:17:25.688 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 16: (Thread::entry_wrapper()+0x43) [0x7ff12eb6172d] 2024-05-27T07:17:25.688 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff12eb61749] 2024-05-27T07:17:25.688 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff12d7051ca] 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 19: clone() 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: -1> 2024-05-27T07:17:25.271+0000 7ff119000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff119000700 time 2024-05-27T07:17:25.269981+0000 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff12eb92ba6] 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff12eb92dd1] 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ac533623d] 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562ac5321971] 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 5: (SnapRealm::check_cache() const+0x13e) [0x562ac531ffa0] 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ac5320651] 2024-05-27T07:17:25.689 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ac52b5347] 2024-05-27T07:17:25.690 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ac4fb187e] 2024-05-27T07:17:25.690 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ac4fb6e94] 2024-05-27T07:17:25.690 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562ac4fc1969] 2024-05-27T07:17:25.690 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 11: (MDLog::_replay_thread()+0x1f38) [0x562ac535a48c] 2024-05-27T07:17:25.690 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562ac53639cb] 2024-05-27T07:17:25.690 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 13: (Thread::entry_wrapper()+0x43) [0x7ff12eb6172d] 2024-05-27T07:17:25.690 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff12eb61749] 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff12d7051ca] 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 16: clone() 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 0> 2024-05-27T07:17:25.275+0000 7ff119000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: in thread 7ff119000700 thread_name:md_log_replay 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff12d70fd20] 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2: gsignal() 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 3: abort() 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff12eb92ced] 2024-05-27T07:17:25.691 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff12eb92dd1] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562ac533623d] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562ac5321971] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 8: (SnapRealm::check_cache() const+0x13e) [0x562ac531ffa0] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562ac5320651] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562ac52b5347] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562ac4fb187e] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562ac4fb6e94] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562ac4fc1969] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 14: (MDLog::_replay_thread()+0x1f38) [0x562ac535a48c] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562ac53639cb] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 16: (Thread::entry_wrapper()+0x43) [0x7ff12eb6172d] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff12eb61749] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff12d7051ca] 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 19: clone() 2024-05-27T07:17:25.692 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:25.693 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[97904]: 2024-05-27T07:17:25.693 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:25 smithi043 podman[97939]: 2024-05-27 07:17:25.367660231 +0000 UTC m=+0.034815660 container died 4128f38957a76f0ae5285af6c9487de2bd9fe290510b0b05b2d5ab2a71f07f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:17:26.183 INFO:tasks.workunit.client.0.smithi043.stdout: 1 11217 0.87 MB/sec warmup 90 sec latency 51.501 ms 2024-05-27T07:17:26.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:26 smithi043 ceph-mon[31531]: pgmap v116: 129 pgs: 129 active+clean; 34 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 371 KiB/s wr, 34 op/s 2024-05-27T07:17:26.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:26 smithi043 ceph-mon[31531]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:17:26.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:26 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6836/3697385247,v1:172.21.15.43:6837/3697385247] up:boot 2024-05-27T07:17:26.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:26 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:26.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:26 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:17:26.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:26 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:26.453 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:26 smithi043 podman[97939]: 2024-05-27 07:17:26.159050092 +0000 UTC m=+0.826205558 container remove 4128f38957a76f0ae5285af6c9487de2bd9fe290510b0b05b2d5ab2a71f07f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:17:26.453 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:26 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:17:26.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:26 smithi179 ceph-mon[35888]: pgmap v116: 129 pgs: 129 active+clean; 34 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 371 KiB/s wr, 34 op/s 2024-05-27T07:17:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:26 smithi179 ceph-mon[35888]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:17:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:26 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6836/3697385247,v1:172.21.15.43:6837/3697385247] up:boot 2024-05-27T07:17:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:26 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:26 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:17:26.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:26 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:26.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:26 smithi132 ceph-mon[35283]: pgmap v116: 129 pgs: 129 active+clean; 34 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 371 KiB/s wr, 34 op/s 2024-05-27T07:17:26.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:26 smithi132 ceph-mon[35283]: osdmap e100: 12 total, 12 up, 12 in 2024-05-27T07:17:26.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:26 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6836/3697385247,v1:172.21.15.43:6837/3697385247] up:boot 2024-05-27T07:17:26.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:26 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:26.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:26 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:17:26.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:26 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:26.952 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:26 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:17:26.952 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:26 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.477s CPU time. 2024-05-27T07:17:27.183 INFO:tasks.workunit.client.0.smithi043.stdout: 1 11306 0.86 MB/sec warmup 91 sec latency 51.290 ms 2024-05-27T07:17:28.183 INFO:tasks.workunit.client.0.smithi043.stdout: 1 11431 0.86 MB/sec warmup 92 sec latency 51.646 ms 2024-05-27T07:17:28.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:28 smithi179 ceph-mon[35888]: pgmap v118: 129 pgs: 129 active+clean; 34 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 346 KiB/s wr, 43 op/s 2024-05-27T07:17:28.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:28 smithi132 ceph-mon[35283]: pgmap v118: 129 pgs: 129 active+clean; 34 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 346 KiB/s wr, 43 op/s 2024-05-27T07:17:28.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:28 smithi043 ceph-mon[31531]: pgmap v118: 129 pgs: 129 active+clean; 34 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 346 KiB/s wr, 43 op/s 2024-05-27T07:17:29.183 INFO:tasks.workunit.client.0.smithi043.stdout: 1 11539 0.85 MB/sec warmup 93 sec latency 66.313 ms 2024-05-27T07:17:30.183 INFO:tasks.workunit.client.0.smithi043.stdout: 1 11641 0.84 MB/sec warmup 94 sec latency 53.547 ms 2024-05-27T07:17:30.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:30 smithi132 ceph-mon[35283]: pgmap v119: 129 pgs: 129 active+clean; 34 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 346 KiB/s wr, 43 op/s 2024-05-27T07:17:30.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:30 smithi043 ceph-mon[31531]: pgmap v119: 129 pgs: 129 active+clean; 34 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 346 KiB/s wr, 43 op/s 2024-05-27T07:17:30.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:30 smithi179 ceph-mon[35888]: pgmap v119: 129 pgs: 129 active+clean; 34 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 346 KiB/s wr, 43 op/s 2024-05-27T07:17:31.183 INFO:tasks.workunit.client.0.smithi043.stdout: 1 11757 0.85 MB/sec warmup 95 sec latency 52.609 ms 2024-05-27T07:17:32.183 INFO:tasks.workunit.client.0.smithi043.stdout: 1 11857 0.84 MB/sec warmup 96 sec latency 53.893 ms 2024-05-27T07:17:32.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:32 smithi132 ceph-mon[35283]: pgmap v120: 129 pgs: 129 active+clean; 33 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 319 KiB/s wr, 56 op/s 2024-05-27T07:17:32.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:32 smithi043 ceph-mon[31531]: pgmap v120: 129 pgs: 129 active+clean; 33 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 319 KiB/s wr, 56 op/s 2024-05-27T07:17:32.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:32 smithi179 ceph-mon[35888]: pgmap v120: 129 pgs: 129 active+clean; 33 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 319 KiB/s wr, 56 op/s 2024-05-27T07:17:33.121 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:32 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:17:33.121 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:32 smithi043 systemd[1]: Stopped Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:33.121 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:32 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 2.020s CPU time. 2024-05-27T07:17:33.121 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:32 smithi043 systemd[1]: Starting Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:17:33.183 INFO:tasks.workunit.client.0.smithi043.stdout: 1 12041 0.86 MB/sec warmup 97 sec latency 50.394 ms 2024-05-27T07:17:33.438 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:33 smithi043 podman[98030]: 2024-05-27 07:17:33.035862717 +0000 UTC m=+0.036184055 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:17:33.439 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:33 smithi043 podman[98030]: 2024-05-27 07:17:33.168398334 +0000 UTC m=+0.168719696 container create c40557e01013152ce9b8c1735ec99847ab2cc4d876a686b0707abbb154a0e4f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:17:33.439 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:33 smithi043 podman[98030]: 2024-05-27 07:17:33.447587789 +0000 UTC m=+0.447909195 container init c40557e01013152ce9b8c1735ec99847ab2cc4d876a686b0707abbb154a0e4f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:17:33.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:33 smithi043 podman[98030]: 2024-05-27 07:17:33.458630347 +0000 UTC m=+0.458951716 container start c40557e01013152ce9b8c1735ec99847ab2cc4d876a686b0707abbb154a0e4f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:17:33.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:33 smithi043 bash[98030]: c40557e01013152ce9b8c1735ec99847ab2cc4d876a686b0707abbb154a0e4f6 2024-05-27T07:17:33.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:33 smithi043 systemd[1]: Started Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:33.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: starting mds.d at 2024-05-27T07:17:34.183 INFO:tasks.workunit.client.0.smithi043.stdout: 1 12151 0.85 MB/sec warmup 98 sec latency 50.871 ms 2024-05-27T07:17:34.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:34 smithi043 ceph-mon[31531]: pgmap v121: 129 pgs: 129 active+clean; 33 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 319 KiB/s wr, 56 op/s 2024-05-27T07:17:34.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:34 smithi043 ceph-mon[31531]: daemon mds.d restarted 2024-05-27T07:17:34.513 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0e6d600700 time 2024-05-27T07:17:34.382388+0000 2024-05-27T07:17:34.513 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:34.513 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0e831e3ba6] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0e831e3dd1] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bd05c6323d] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bd05c4e971] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bd05c4cfa0] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bd05c4d651] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bd05be2347] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bd058de87e] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bd058e3e94] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bd058ee969] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bd05c8748c] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bd05c909cb] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 13: (Thread::entry_wrapper()+0x43) [0x7f0e831b272d] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0e831b2749] 2024-05-27T07:17:34.514 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0e81d561ca] 2024-05-27T07:17:34.515 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 16: clone() 2024-05-27T07:17:34.515 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.385+0000 7f0e6d600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0e6d600700 time 2024-05-27T07:17:34.382388+0000 2024-05-27T07:17:34.515 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:34.515 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.515 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:34.515 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0e831e3ba6] 2024-05-27T07:17:34.515 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0e831e3dd1] 2024-05-27T07:17:34.515 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bd05c6323d] 2024-05-27T07:17:34.515 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bd05c4e971] 2024-05-27T07:17:34.515 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bd05c4cfa0] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bd05c4d651] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bd05be2347] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bd058de87e] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bd058e3e94] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bd058ee969] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bd05c8748c] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bd05c909cb] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 13: (Thread::entry_wrapper()+0x43) [0x7f0e831b272d] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0e831b2749] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0e81d561ca] 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 16: clone() 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: *** Caught signal (Aborted) ** 2024-05-27T07:17:34.516 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: in thread 7f0e6d600700 thread_name:md_log_replay 2024-05-27T07:17:34.517 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:34.517 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0e81d60d20] 2024-05-27T07:17:34.517 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2: gsignal() 2024-05-27T07:17:34.517 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 3: abort() 2024-05-27T07:17:34.517 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0e831e3ced] 2024-05-27T07:17:34.517 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0e831e3dd1] 2024-05-27T07:17:34.517 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bd05c6323d] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bd05c4e971] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bd05c4cfa0] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bd05c4d651] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bd05be2347] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bd058de87e] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bd058e3e94] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bd058ee969] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bd05c8748c] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bd05c909cb] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 16: (Thread::entry_wrapper()+0x43) [0x7f0e831b272d] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0e831b2749] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0e81d561ca] 2024-05-27T07:17:34.518 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 19: clone() 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.390+0000 7f0e6d600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: in thread 7f0e6d600700 thread_name:md_log_replay 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0e81d60d20] 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2: gsignal() 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 3: abort() 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0e831e3ced] 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0e831e3dd1] 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bd05c6323d] 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bd05c4e971] 2024-05-27T07:17:34.519 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bd05c4cfa0] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bd05c4d651] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bd05be2347] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bd058de87e] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bd058e3e94] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bd058ee969] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bd05c8748c] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bd05c909cb] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 16: (Thread::entry_wrapper()+0x43) [0x7f0e831b272d] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0e831b2749] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0e81d561ca] 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 19: clone() 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.520 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: -1> 2024-05-27T07:17:34.385+0000 7f0e6d600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0e6d600700 time 2024-05-27T07:17:34.382388+0000 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0e831e3ba6] 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0e831e3dd1] 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bd05c6323d] 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bd05c4e971] 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bd05c4cfa0] 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bd05c4d651] 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bd05be2347] 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bd058de87e] 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bd058e3e94] 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bd058ee969] 2024-05-27T07:17:34.521 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bd05c8748c] 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bd05c909cb] 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 13: (Thread::entry_wrapper()+0x43) [0x7f0e831b272d] 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0e831b2749] 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0e81d561ca] 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 16: clone() 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 0> 2024-05-27T07:17:34.390+0000 7f0e6d600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: in thread 7f0e6d600700 thread_name:md_log_replay 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0e81d60d20] 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2: gsignal() 2024-05-27T07:17:34.522 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 3: abort() 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0e831e3ced] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0e831e3dd1] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bd05c6323d] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bd05c4e971] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bd05c4cfa0] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bd05c4d651] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bd05be2347] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bd058de87e] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bd058e3e94] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bd058ee969] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bd05c8748c] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bd05c909cb] 2024-05-27T07:17:34.523 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 16: (Thread::entry_wrapper()+0x43) [0x7f0e831b272d] 2024-05-27T07:17:34.524 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0e831b2749] 2024-05-27T07:17:34.524 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0e81d561ca] 2024-05-27T07:17:34.524 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 19: clone() 2024-05-27T07:17:34.524 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:34.524 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.524 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: -1> 2024-05-27T07:17:34.385+0000 7f0e6d600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0e6d600700 time 2024-05-27T07:17:34.382388+0000 2024-05-27T07:17:34.524 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:34.524 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0e831e3ba6] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0e831e3dd1] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bd05c6323d] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bd05c4e971] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bd05c4cfa0] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bd05c4d651] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bd05be2347] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bd058de87e] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bd058e3e94] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bd058ee969] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bd05c8748c] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bd05c909cb] 2024-05-27T07:17:34.525 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 13: (Thread::entry_wrapper()+0x43) [0x7f0e831b272d] 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 14: (Thread::_entry_func(void*)+0xd) [0x7f0e831b2749] 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0e81d561ca] 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 16: clone() 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 0> 2024-05-27T07:17:34.390+0000 7f0e6d600700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: in thread 7f0e6d600700 thread_name:md_log_replay 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0e81d60d20] 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2: gsignal() 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 3: abort() 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0e831e3ced] 2024-05-27T07:17:34.526 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0e831e3dd1] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bd05c6323d] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bd05c4e971] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bd05c4cfa0] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bd05c4d651] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bd05be2347] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bd058de87e] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bd058e3e94] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bd058ee969] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bd05c8748c] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bd05c909cb] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 16: (Thread::entry_wrapper()+0x43) [0x7f0e831b272d] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 17: (Thread::_entry_func(void*)+0xd) [0x7f0e831b2749] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0e81d561ca] 2024-05-27T07:17:34.527 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 19: clone() 2024-05-27T07:17:34.528 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:34.528 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98045]: 2024-05-27T07:17:34.528 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:34 smithi043 podman[98080]: 2024-05-27 07:17:34.522635341 +0000 UTC m=+0.036265278 container died c40557e01013152ce9b8c1735ec99847ab2cc4d876a686b0707abbb154a0e4f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-27T07:17:34.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:34 smithi132 ceph-mon[35283]: pgmap v121: 129 pgs: 129 active+clean; 33 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 319 KiB/s wr, 56 op/s 2024-05-27T07:17:34.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:34 smithi132 ceph-mon[35283]: daemon mds.d restarted 2024-05-27T07:17:34.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:34 smithi179 ceph-mon[35888]: pgmap v121: 129 pgs: 129 active+clean; 33 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 319 KiB/s wr, 56 op/s 2024-05-27T07:17:34.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:34 smithi179 ceph-mon[35888]: daemon mds.d restarted 2024-05-27T07:17:35.184 INFO:tasks.workunit.client.0.smithi043.stdout: 1 12272 0.84 MB/sec warmup 99 sec latency 52.009 ms 2024-05-27T07:17:35.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:35 smithi043 ceph-mon[31531]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:17:35.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:35 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6836/1176845155,v1:172.21.15.43:6837/1176845155] up:boot 2024-05-27T07:17:35.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:35 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:35.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:35 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:17:35.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:35 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:35.572 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:35 smithi043 podman[98080]: 2024-05-27 07:17:35.337081467 +0000 UTC m=+0.850711400 container remove c40557e01013152ce9b8c1735ec99847ab2cc4d876a686b0707abbb154a0e4f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:17:35.572 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:35 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:17:35.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:35 smithi132 ceph-mon[35283]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:17:35.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:35 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6836/1176845155,v1:172.21.15.43:6837/1176845155] up:boot 2024-05-27T07:17:35.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:35 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:35.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:35 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:17:35.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:35 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:35.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:35 smithi179 ceph-mon[35888]: osdmap e101: 12 total, 12 up, 12 in 2024-05-27T07:17:35.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:35 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6836/1176845155,v1:172.21.15.43:6837/1176845155] up:boot 2024-05-27T07:17:35.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:35 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:35.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:35 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:17:35.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:35 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:35.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:35 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:17:35.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:35 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.586s CPU time. 2024-05-27T07:17:36.184 INFO:tasks.workunit.client.0.smithi043.stdout: 1 12450 0.84 MB/sec warmup 100 sec latency 50.159 ms 2024-05-27T07:17:36.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:36 smithi132 ceph-mon[35283]: pgmap v123: 129 pgs: 129 active+clean; 33 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 341 KiB/s wr, 60 op/s 2024-05-27T07:17:36.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:36 smithi043 ceph-mon[31531]: pgmap v123: 129 pgs: 129 active+clean; 33 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 341 KiB/s wr, 60 op/s 2024-05-27T07:17:36.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:36 smithi179 ceph-mon[35888]: pgmap v123: 129 pgs: 129 active+clean; 33 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 341 KiB/s wr, 60 op/s 2024-05-27T07:17:37.142 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:36 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:17:37.142 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:36 smithi043 systemd[1]: Stopped Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:37.142 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:36 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.477s CPU time. 2024-05-27T07:17:37.142 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:36 smithi043 systemd[1]: Starting Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:17:37.184 INFO:tasks.workunit.client.0.smithi043.stdout: 1 12570 0.83 MB/sec warmup 101 sec latency 54.845 ms 2024-05-27T07:17:37.442 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:37 smithi043 podman[98172]: 2024-05-27 07:17:37.056763806 +0000 UTC m=+0.039797595 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:17:37.442 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:37 smithi043 podman[98172]: 2024-05-27 07:17:37.196758559 +0000 UTC m=+0.179792372 container create 9486b170a072c78bc466419b947ce1d98a0c80c05810356e7bf52d1635c9d105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-27T07:17:37.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:37 smithi043 podman[98172]: 2024-05-27 07:17:37.451465416 +0000 UTC m=+0.434499256 container init 9486b170a072c78bc466419b947ce1d98a0c80c05810356e7bf52d1635c9d105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:17:37.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:37 smithi043 podman[98172]: 2024-05-27 07:17:37.461617807 +0000 UTC m=+0.444651627 container start 9486b170a072c78bc466419b947ce1d98a0c80c05810356e7bf52d1635c9d105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-27T07:17:37.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:37 smithi043 bash[98172]: 9486b170a072c78bc466419b947ce1d98a0c80c05810356e7bf52d1635c9d105 2024-05-27T07:17:37.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:37 smithi043 systemd[1]: Started Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:37.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:37 smithi043 ceph-mds[98191]: -- 172.21.15.43:0/557538478 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a3f208fa40 con 0x55a3f20d7600 2024-05-27T07:17:37.703 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: starting mds.g at 2024-05-27T07:17:38.184 INFO:tasks.workunit.client.0.smithi043.stdout: 1 12718 0.82 MB/sec warmup 102 sec latency 65.888 ms 2024-05-27T07:17:38.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:38 smithi132 ceph-mon[35283]: pgmap v124: 129 pgs: 129 active+clean; 33 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 287 KiB/s wr, 56 op/s 2024-05-27T07:17:38.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:38 smithi132 ceph-mon[35283]: daemon mds.g restarted 2024-05-27T07:17:38.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:38 smithi043 ceph-mon[31531]: pgmap v124: 129 pgs: 129 active+clean; 33 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 287 KiB/s wr, 56 op/s 2024-05-27T07:17:38.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:38 smithi043 ceph-mon[31531]: daemon mds.g restarted 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7502e00700 time 2024-05-27T07:17:38.345451+0000 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7518948ba6] 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7518948dd1] 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a3d8a5323d] 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a3d8a3e971] 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a3d8a3cfa0] 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a3d8a3d651] 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a3d89d2347] 2024-05-27T07:17:38.704 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a3d86ce87e] 2024-05-27T07:17:38.705 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a3d86d3e94] 2024-05-27T07:17:38.705 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a3d86de969] 2024-05-27T07:17:38.705 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a3d8a7748c] 2024-05-27T07:17:38.705 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a3d8a809cb] 2024-05-27T07:17:38.705 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 13: (Thread::entry_wrapper()+0x43) [0x7f751891772d] 2024-05-27T07:17:38.705 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7518917749] 2024-05-27T07:17:38.705 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f75174bb1ca] 2024-05-27T07:17:38.705 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 16: clone() 2024-05-27T07:17:38.705 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: *** Caught signal (Aborted) ** 2024-05-27T07:17:38.705 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: in thread 7f7502e00700 thread_name:md_log_replay 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.347+0000 7f7502e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7502e00700 time 2024-05-27T07:17:38.345451+0000 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7518948ba6] 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7518948dd1] 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a3d8a5323d] 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a3d8a3e971] 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a3d8a3cfa0] 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a3d8a3d651] 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a3d89d2347] 2024-05-27T07:17:38.706 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a3d86ce87e] 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a3d86d3e94] 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a3d86de969] 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a3d8a7748c] 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a3d8a809cb] 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 13: (Thread::entry_wrapper()+0x43) [0x7f751891772d] 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7518917749] 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f75174bb1ca] 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 16: clone() 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f75174c5d20] 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2: gsignal() 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 3: abort() 2024-05-27T07:17:38.707 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7518948ced] 2024-05-27T07:17:38.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7518948dd1] 2024-05-27T07:17:38.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a3d8a5323d] 2024-05-27T07:17:38.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a3d8a3e971] 2024-05-27T07:17:38.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a3d8a3cfa0] 2024-05-27T07:17:38.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a3d8a3d651] 2024-05-27T07:17:38.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a3d89d2347] 2024-05-27T07:17:38.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a3d86ce87e] 2024-05-27T07:17:38.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a3d86d3e94] 2024-05-27T07:17:38.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a3d86de969] 2024-05-27T07:17:38.708 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a3d8a7748c] 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a3d8a809cb] 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 16: (Thread::entry_wrapper()+0x43) [0x7f751891772d] 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7518917749] 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f75174bb1ca] 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 19: clone() 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.351+0000 7f7502e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: in thread 7f7502e00700 thread_name:md_log_replay 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f75174c5d20] 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2: gsignal() 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 3: abort() 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7518948ced] 2024-05-27T07:17:38.709 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7518948dd1] 2024-05-27T07:17:38.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a3d8a5323d] 2024-05-27T07:17:38.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a3d8a3e971] 2024-05-27T07:17:38.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a3d8a3cfa0] 2024-05-27T07:17:38.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a3d8a3d651] 2024-05-27T07:17:38.710 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a3d89d2347] 2024-05-27T07:17:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:38 smithi179 ceph-mon[35888]: pgmap v124: 129 pgs: 129 active+clean; 33 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 287 KiB/s wr, 56 op/s 2024-05-27T07:17:38.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:38 smithi179 ceph-mon[35888]: daemon mds.g restarted 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a3d86ce87e] 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a3d86d3e94] 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a3d86de969] 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a3d8a7748c] 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a3d8a809cb] 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 16: (Thread::entry_wrapper()+0x43) [0x7f751891772d] 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7518917749] 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f75174bb1ca] 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 19: clone() 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.711 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: -1> 2024-05-27T07:17:38.347+0000 7f7502e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7502e00700 time 2024-05-27T07:17:38.345451+0000 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7518948ba6] 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7518948dd1] 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a3d8a5323d] 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a3d8a3e971] 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a3d8a3cfa0] 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a3d8a3d651] 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a3d89d2347] 2024-05-27T07:17:38.712 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a3d86ce87e] 2024-05-27T07:17:38.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a3d86d3e94] 2024-05-27T07:17:38.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a3d86de969] 2024-05-27T07:17:38.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a3d8a7748c] 2024-05-27T07:17:38.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a3d8a809cb] 2024-05-27T07:17:38.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 13: (Thread::entry_wrapper()+0x43) [0x7f751891772d] 2024-05-27T07:17:38.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7518917749] 2024-05-27T07:17:38.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f75174bb1ca] 2024-05-27T07:17:38.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 16: clone() 2024-05-27T07:17:38.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.713 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 0> 2024-05-27T07:17:38.351+0000 7f7502e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: in thread 7f7502e00700 thread_name:md_log_replay 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f75174c5d20] 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2: gsignal() 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 3: abort() 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7518948ced] 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7518948dd1] 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a3d8a5323d] 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a3d8a3e971] 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a3d8a3cfa0] 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a3d8a3d651] 2024-05-27T07:17:38.714 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a3d89d2347] 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a3d86ce87e] 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a3d86d3e94] 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a3d86de969] 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a3d8a7748c] 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a3d8a809cb] 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 16: (Thread::entry_wrapper()+0x43) [0x7f751891772d] 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7518917749] 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f75174bb1ca] 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 19: clone() 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: -1> 2024-05-27T07:17:38.347+0000 7f7502e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f7502e00700 time 2024-05-27T07:17:38.345451+0000 2024-05-27T07:17:38.715 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7518948ba6] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7518948dd1] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a3d8a5323d] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a3d8a3e971] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a3d8a3cfa0] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a3d8a3d651] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a3d89d2347] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a3d86ce87e] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a3d86d3e94] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a3d86de969] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a3d8a7748c] 2024-05-27T07:17:38.716 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a3d8a809cb] 2024-05-27T07:17:38.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 13: (Thread::entry_wrapper()+0x43) [0x7f751891772d] 2024-05-27T07:17:38.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 14: (Thread::_entry_func(void*)+0xd) [0x7f7518917749] 2024-05-27T07:17:38.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f75174bb1ca] 2024-05-27T07:17:38.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 16: clone() 2024-05-27T07:17:38.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 0> 2024-05-27T07:17:38.351+0000 7f7502e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:38.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: in thread 7f7502e00700 thread_name:md_log_replay 2024-05-27T07:17:38.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:38.717 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f75174c5d20] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2: gsignal() 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 3: abort() 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7518948ced] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7518948dd1] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a3d8a5323d] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a3d8a3e971] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a3d8a3cfa0] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a3d8a3d651] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a3d89d2347] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a3d86ce87e] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a3d86d3e94] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a3d86de969] 2024-05-27T07:17:38.718 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a3d8a7748c] 2024-05-27T07:17:38.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a3d8a809cb] 2024-05-27T07:17:38.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 16: (Thread::entry_wrapper()+0x43) [0x7f751891772d] 2024-05-27T07:17:38.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 17: (Thread::_entry_func(void*)+0xd) [0x7f7518917749] 2024-05-27T07:17:38.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f75174bb1ca] 2024-05-27T07:17:38.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 19: clone() 2024-05-27T07:17:38.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:38.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g[98187]: 2024-05-27T07:17:38.719 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:38 smithi043 podman[98222]: 2024-05-27 07:17:38.44540918 +0000 UTC m=+0.035371489 container died 9486b170a072c78bc466419b947ce1d98a0c80c05810356e7bf52d1635c9d105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=) 2024-05-27T07:17:39.184 INFO:tasks.workunit.client.0.smithi043.stdout: 1 12875 0.82 MB/sec warmup 103 sec latency 58.101 ms 2024-05-27T07:17:39.548 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:39 smithi179 ceph-mon[35888]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:17:39.549 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:39 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6836/1084201245,v1:172.21.15.43:6837/1084201245] up:boot 2024-05-27T07:17:39.549 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:39 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:39.549 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:39 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:17:39.549 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:39 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:39.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:39 smithi132 ceph-mon[35283]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:17:39.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:39 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6836/1084201245,v1:172.21.15.43:6837/1084201245] up:boot 2024-05-27T07:17:39.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:39 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:39.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:39 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:17:39.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:39 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:39.702 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:39 smithi043 podman[98222]: 2024-05-27 07:17:39.265177965 +0000 UTC m=+0.855140291 container remove 9486b170a072c78bc466419b947ce1d98a0c80c05810356e7bf52d1635c9d105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-g, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-27T07:17:39.702 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:39 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:17:39.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:39 smithi043 ceph-mon[31531]: osdmap e102: 12 total, 12 up, 12 in 2024-05-27T07:17:39.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:39 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6836/1084201245,v1:172.21.15.43:6837/1084201245] up:boot 2024-05-27T07:17:39.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:39 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:39.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:39 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-27T07:17:39.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:39 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:40.184 INFO:tasks.workunit.client.0.smithi043.stdout: 1 13016 0.82 MB/sec warmup 104 sec latency 56.803 ms 2024-05-27T07:17:40.202 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:39 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:17:40.202 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:39 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.501s CPU time. 2024-05-27T07:17:40.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:40 smithi132 ceph-mon[35283]: pgmap v126: 129 pgs: 129 active+clean; 33 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 201 KiB/s wr, 32 op/s 2024-05-27T07:17:40.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:40 smithi043 ceph-mon[31531]: pgmap v126: 129 pgs: 129 active+clean; 33 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 201 KiB/s wr, 32 op/s 2024-05-27T07:17:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:40 smithi179 ceph-mon[35888]: pgmap v126: 129 pgs: 129 active+clean; 33 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 201 KiB/s wr, 32 op/s 2024-05-27T07:17:41.184 INFO:tasks.workunit.client.0.smithi043.stdout: 1 13088 0.81 MB/sec warmup 105 sec latency 58.986 ms 2024-05-27T07:17:42.184 INFO:tasks.workunit.client.0.smithi043.stdout: 1 13204 0.81 MB/sec warmup 106 sec latency 59.228 ms 2024-05-27T07:17:42.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:42 smithi132 ceph-mon[35283]: pgmap v127: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 392 KiB/s wr, 58 op/s 2024-05-27T07:17:42.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:42 smithi043 ceph-mon[31531]: pgmap v127: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 392 KiB/s wr, 58 op/s 2024-05-27T07:17:42.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:42 smithi179 ceph-mon[35888]: pgmap v127: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 392 KiB/s wr, 58 op/s 2024-05-27T07:17:42.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:17:43.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 13273 0.81 MB/sec warmup 107 sec latency 54.688 ms 2024-05-27T07:17:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:44.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 13579 0.84 MB/sec warmup 108 sec latency 54.533 ms 2024-05-27T07:17:44.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:44 smithi132 ceph-mon[35283]: pgmap v128: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 378 KiB/s wr, 56 op/s 2024-05-27T07:17:44.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:44 smithi043 ceph-mon[31531]: pgmap v128: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 378 KiB/s wr, 56 op/s 2024-05-27T07:17:44.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:44 smithi179 ceph-mon[35888]: pgmap v128: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 378 KiB/s wr, 56 op/s 2024-05-27T07:17:45.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 13681 0.83 MB/sec warmup 109 sec latency 53.288 ms 2024-05-27T07:17:45.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.857+0000 7f2115a00700 1 -- 172.21.15.43:0/3200113532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f211010af10 msgr2=0x7f211010b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:45.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.857+0000 7f2115a00700 1 --2- 172.21.15.43:0/3200113532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f211010af10 0x7f211010b320 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f20f80099c0 tx=0x7f20f8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.857+0000 7f2115a00700 1 -- 172.21.15.43:0/3200113532 shutdown_connections 2024-05-27T07:17:45.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.857+0000 7f2115a00700 1 --2- 172.21.15.43:0/3200113532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f211010b860 0x7f21101131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.858+0000 7f2115a00700 1 --2- 172.21.15.43:0/3200113532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2110106630 0x7f2110106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.858+0000 7f2115a00700 1 --2- 172.21.15.43:0/3200113532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f211010af10 0x7f211010b320 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.858+0000 7f2115a00700 1 -- 172.21.15.43:0/3200113532 >> 172.21.15.43:0/3200113532 conn(0x7f2110102020 msgr2=0x7f2110104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.858+0000 7f2115a00700 1 -- 172.21.15.43:0/3200113532 shutdown_connections 2024-05-27T07:17:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.858+0000 7f2115a00700 1 -- 172.21.15.43:0/3200113532 wait complete. 2024-05-27T07:17:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.859+0000 7f2115a00700 1 Processor -- start 2024-05-27T07:17:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.859+0000 7f2115a00700 1 -- start start 2024-05-27T07:17:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.860+0000 7f2115a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2110106630 0x7f211019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.860+0000 7f2115a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f211010af10 0x7f211019d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.860+0000 7f2115a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f211010b860 0x7f21101a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.860+0000 7f2115a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2110116370 con 0x7f211010af10 2024-05-27T07:17:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.860+0000 7f2115a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f21101161f0 con 0x7f211010b860 2024-05-27T07:17:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.860+0000 7f2115a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f21101164f0 con 0x7f2110106630 2024-05-27T07:17:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.860+0000 7f2107400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f211010af10 0x7f211019d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.860+0000 7f2107400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f211010af10 0x7f211019d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33020/0 (socket says 172.21.15.43:33020) 2024-05-27T07:17:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.860+0000 7f2107400700 1 -- 172.21.15.43:0/730749840 learned_addr learned my addr 172.21.15.43:0/730749840 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.861+0000 7f2115000700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f211010b860 0x7f21101a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.861+0000 7f210fe00700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2110106630 0x7f211019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.861+0000 7f2107400700 1 -- 172.21.15.43:0/730749840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2110106630 msgr2=0x7f211019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.861+0000 7f2107400700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2110106630 0x7f211019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.861+0000 7f2107400700 1 -- 172.21.15.43:0/730749840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f211010b860 msgr2=0x7f21101a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.861+0000 7f2107400700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f211010b860 0x7f21101a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.861+0000 7f2107400700 1 -- 172.21.15.43:0/730749840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21101a1c70 con 0x7f211010af10 2024-05-27T07:17:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.861+0000 7f2107400700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f211010af10 0x7f211019d340 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f20f800b580 tx=0x7f20f800f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.862+0000 7f210d600700 1 -- 172.21.15.43:0/730749840 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8027070 con 0x7f211010af10 2024-05-27T07:17:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.862+0000 7f210d600700 1 -- 172.21.15.43:0/730749840 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f20f800fda0 con 0x7f211010af10 2024-05-27T07:17:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.862+0000 7f2115a00700 1 -- 172.21.15.43:0/730749840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21101a1f00 con 0x7f211010af10 2024-05-27T07:17:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.862+0000 7f210d600700 1 -- 172.21.15.43:0/730749840 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f80173f0 con 0x7f211010af10 2024-05-27T07:17:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.862+0000 7f2115a00700 1 -- 172.21.15.43:0/730749840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21101a23e0 con 0x7f211010af10 2024-05-27T07:17:45.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.866+0000 7f210d600700 1 -- 172.21.15.43:0/730749840 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f20f802a070 con 0x7f211010af10 2024-05-27T07:17:45.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.867+0000 7f2125c11b80 1 Processor -- start 2024-05-27T07:17:45.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.867+0000 7f2125c11b80 1 -- start start 2024-05-27T07:17:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.867+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 0x560b32dafed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.867+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0410 0x560b32dbaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.867+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0820 0x560b32dbd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.868+0000 7f2125c11b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560b32cdee40 con 0x560b32db0410 2024-05-27T07:17:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.868+0000 7f2125c11b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560b32cce9e0 con 0x560b32db0820 2024-05-27T07:17:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.868+0000 7f2125c11b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560b32cd2a20 con 0x560b32c31e40 2024-05-27T07:17:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.868+0000 7f210fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0410 0x560b32dbaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.868+0000 7f210fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0410 0x560b32dbaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33026/0 (socket says 172.21.15.43:33026) 2024-05-27T07:17:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.868+0000 7f210fe00700 1 -- 172.21.15.43:0/2568305172 learned_addr learned my addr 172.21.15.43:0/2568305172 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.868+0000 7f2115000700 1 --2- 172.21.15.43:0/2568305172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 0x560b32dafed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.868+0000 7f210d600700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f20f40b29f0 0x7f20f40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.868+0000 7f2107400700 1 --2- 172.21.15.43:0/2568305172 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0820 0x560b32dbd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.869+0000 7f210fe00700 1 -- 172.21.15.43:0/2568305172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 msgr2=0x560b32dafed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.869+0000 7f210fe00700 1 --2- 172.21.15.43:0/2568305172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 0x560b32dafed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.869+0000 7f210fe00700 1 -- 172.21.15.43:0/2568305172 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0820 msgr2=0x560b32dbd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.869+0000 7f210fe00700 1 --2- 172.21.15.43:0/2568305172 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0820 0x560b32dbd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.869+0000 7f210fe00700 1 -- 172.21.15.43:0/2568305172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b32c97420 con 0x560b32db0410 2024-05-27T07:17:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.869+0000 7f2115000700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f20f40b29f0 0x7f20f40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.869+0000 7f210d600700 1 -- 172.21.15.43:0/730749840 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9408+0+0 (secure 0 0 0) 0x7f20f80e3920 con 0x7f211010af10 2024-05-27T07:17:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.869+0000 7f210fe00700 1 --2- 172.21.15.43:0/2568305172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0410 0x560b32dbaad0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f210000bd70 tx=0x7f210000f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fac7294467c74c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:45.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.870+0000 7f2105600700 1 -- 172.21.15.43:0/2568305172 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100015070 con 0x560b32db0410 2024-05-27T07:17:45.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.870+0000 7f2105600700 1 -- 172.21.15.43:0/2568305172 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f210000fe50 con 0x560b32db0410 2024-05-27T07:17:45.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.870+0000 7f2105600700 1 -- 172.21.15.43:0/2568305172 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21000116a0 con 0x560b32db0410 2024-05-27T07:17:45.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.870+0000 7f2115000700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f20f40b29f0 0x7f20f40b4f00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x560b32dac690 tx=0x7f2108004b40 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:45.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.871+0000 7f2125c11b80 1 -- 172.21.15.43:0/2568305172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0410 msgr2=0x560b32dbaad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.871+0000 7f2125c11b80 1 --2- 172.21.15.43:0/2568305172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0410 0x560b32dbaad0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f210000bd70 tx=0x7f210000f640 comp rx=0 tx=0).stop 2024-05-27T07:17:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.871+0000 7f2125c11b80 1 -- 172.21.15.43:0/2568305172 shutdown_connections 2024-05-27T07:17:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.871+0000 7f2125c11b80 1 --2- 172.21.15.43:0/2568305172 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0820 0x560b32dbd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.871+0000 7f2125c11b80 1 --2- 172.21.15.43:0/2568305172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 0x560b32dafed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.871+0000 7f2125c11b80 1 --2- 172.21.15.43:0/2568305172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0410 0x560b32dbaad0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.871+0000 7f2125c11b80 1 -- 172.21.15.43:0/2568305172 >> 172.21.15.43:0/2568305172 conn(0x560b32c91fc0 msgr2=0x560b32dabd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.871+0000 7f2125c11b80 1 -- 172.21.15.43:0/2568305172 shutdown_connections 2024-05-27T07:17:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.871+0000 7f2125c11b80 1 -- 172.21.15.43:0/2568305172 wait complete. 2024-05-27T07:17:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.872+0000 7f2125c11b80 1 -- start start 2024-05-27T07:17:45.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.872+0000 7f2125c11b80 1 Processor -- start 2024-05-27T07:17:45.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.873+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 0x560b32dba5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.873+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 0x560b32dbaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.873+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0820 0x560b32dbb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.873+0000 7f2125c11b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560b32bd4fa0 con 0x560b32db0820 2024-05-27T07:17:45.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.873+0000 7f2125c11b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560b32c6f760 con 0x560b32db0410 2024-05-27T07:17:45.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.873+0000 7f2125c11b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560b32dbd990 con 0x560b32c31e40 2024-05-27T07:17:45.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.873+0000 7f2125c11b80 10 client.0 _ll_register_callbacks cb 0x560b32c78c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:45.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.873+0000 7f2115000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0820 0x560b32dbb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.874+0000 7f2115000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0820 0x560b32dbb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33028/0 (socket says 172.21.15.43:33028) 2024-05-27T07:17:45.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.874+0000 7f2115000700 1 -- 172.21.15.43:0/844589904 learned_addr learned my addr 172.21.15.43:0/844589904 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:45.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.874+0000 7f210fe00700 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 0x560b32dba5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.874+0000 7f2107400700 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 0x560b32dbaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.874+0000 7f2115000700 1 -- 172.21.15.43:0/844589904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 msgr2=0x560b32dba5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:45.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.874+0000 7f2115000700 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 0x560b32dba5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.874+0000 7f2115000700 1 -- 172.21.15.43:0/844589904 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 msgr2=0x560b32dbaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:45.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.874+0000 7f2115000700 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 0x560b32dbaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.874+0000 7f2115000700 1 -- 172.21.15.43:0/844589904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b32c1e8d0 con 0x560b32db0820 2024-05-27T07:17:45.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.874+0000 7f210fe00700 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 0x560b32dba5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:45.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.875+0000 7f2115000700 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0820 0x560b32dbb020 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f210801ae10 tx=0x7f2108004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75aeb6006bcf304 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:45.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.875+0000 7f20fe000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:45.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.875+0000 7f20fe000700 1 -- 172.21.15.43:0/844589904 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108012040 con 0x560b32db0820 2024-05-27T07:17:45.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.875+0000 7f20fe000700 1 -- 172.21.15.43:0/844589904 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9408+0+0 (secure 0 0 0) 0x7f2108017b50 con 0x560b32db0820 2024-05-27T07:17:45.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.875+0000 7f2125c11b80 1 -- 172.21.15.43:0/844589904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f210000fe50 con 0x560b32db0820 2024-05-27T07:17:45.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.875+0000 7f20fe000700 1 -- 172.21.15.43:0/844589904 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f2108017d50 con 0x560b32db0820 2024-05-27T07:17:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.875+0000 7f2125c11b80 10 client.16008 fetch_fsmap learned FSMap version 55 2024-05-27T07:17:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.875+0000 7f2125c11b80 1 -- 172.21.15.43:0/844589904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x560b32c9f060 con 0x560b32db0820 2024-05-27T07:17:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.876+0000 7f20fe000700 1 -- 172.21.15.43:0/844589904 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f210000fe50 con 0x560b32db0820 2024-05-27T07:17:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.876+0000 7f20fe000700 1 -- 172.21.15.43:0/844589904 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 55) ==== 7773+0+0 (secure 0 0 0) 0x7f2108018420 con 0x560b32db0820 2024-05-27T07:17:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.876+0000 7f2125c11b80 10 client.16008 fetch_fsmap finished waiting for FSMap version 55 2024-05-27T07:17:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.876+0000 7f2125c11b80 10 client.16008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:17:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.876+0000 7f2125c11b80 20 client.16008 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.876+0000 7f2125c11b80 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560b32dcaa00 0x560b32dcced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.876+0000 7f2125c11b80 4 client.16008 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:17:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.876+0000 7f2125c11b80 1 -- 172.21.15.43:0/844589904 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b32ce0d90 con 0x560b32dcaa00 2024-05-27T07:17:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.877+0000 7f210fe00700 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560b32dcaa00 0x560b32dcced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.878+0000 7f210fe00700 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560b32dcaa00 0x560b32dcced0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bf369a2e36cb981 server_cookie=5893fe3bd5e192cd in_seq=0 out_seq=0 2024-05-27T07:17:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.878+0000 7f20fe000700 10 client.16008 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.882+0000 7f20fe000700 1 -- 172.21.15.43:0/844589904 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560b32ce0d90 con 0x560b32dcaa00 2024-05-27T07:17:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.882+0000 7f20fe000700 10 client.16008 handle_command_reply: tid=0 2024-05-27T07:17:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.882+0000 7f2125c11b80 1 client.16008 shutdown 2024-05-27T07:17:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 1 -- 172.21.15.43:0/844589904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0820 msgr2=0x560b32dbb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0820 0x560b32dbb020 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f210801ae10 tx=0x7f2108004ad0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 1 -- 172.21.15.43:0/844589904 shutdown_connections 2024-05-27T07:17:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560b32dcaa00 0x560b32dcced0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32db0820 0x560b32dbb020 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32c31e40 0x560b32dba5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 1 --2- 172.21.15.43:0/844589904 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 0x560b32dbaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f20fe000700 0 client.16008 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 1 -- 172.21.15.43:0/844589904 >> 172.21.15.43:0/844589904 conn(0x560b32c91fc0 msgr2=0x560b32daaff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 1 -- 172.21.15.43:0/844589904 shutdown_connections 2024-05-27T07:17:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 1 -- 172.21.15.43:0/844589904 wait complete. 2024-05-27T07:17:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.883+0000 7f2125c11b80 20 client.16008 trim_cache size 0 max 16384 2024-05-27T07:17:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.906+0000 7f2125c11b80 1 Processor -- start 2024-05-27T07:17:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.907+0000 7f2125c11b80 1 -- start start 2024-05-27T07:17:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.907+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32db5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.907+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 0x560b32dbca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.907+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32db0820 0x560b32dbac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.907+0000 7f2125c11b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f21101b3f80 con 0x560b32c31e40 2024-05-27T07:17:45.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.907+0000 7f2125c11b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f21101b1cb0 con 0x560b32db0410 2024-05-27T07:17:45.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.907+0000 7f2125c11b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f21101b38b0 con 0x560b32db0820 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.907+0000 7f210fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32db5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.907+0000 7f210fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32db5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33040/0 (socket says 172.21.15.43:33040) 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.907+0000 7f210fe00700 1 -- 172.21.15.43:0/2674379356 learned_addr learned my addr 172.21.15.43:0/2674379356 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.908+0000 7f2107400700 1 --2- 172.21.15.43:0/2674379356 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 0x560b32dbca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.908+0000 7f2115000700 1 --2- 172.21.15.43:0/2674379356 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32db0820 0x560b32dbac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.908+0000 7f210fe00700 1 -- 172.21.15.43:0/2674379356 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32db0820 msgr2=0x560b32dbac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.908+0000 7f210fe00700 1 --2- 172.21.15.43:0/2674379356 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32db0820 0x560b32dbac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.908+0000 7f210fe00700 1 -- 172.21.15.43:0/2674379356 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 msgr2=0x560b32dbca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.908+0000 7f210fe00700 1 --2- 172.21.15.43:0/2674379356 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 0x560b32dbca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.908+0000 7f210fe00700 1 -- 172.21.15.43:0/2674379356 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b32dad350 con 0x560b32c31e40 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.908+0000 7f2107400700 1 --2- 172.21.15.43:0/2674379356 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 0x560b32dbca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:45.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.908+0000 7f210fe00700 1 --2- 172.21.15.43:0/2674379356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32db5d90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f210000fa40 tx=0x7f210001ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d265aa8ed7d0c2e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:45.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.909+0000 7f2104c00700 1 -- 172.21.15.43:0/2674379356 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100015070 con 0x560b32c31e40 2024-05-27T07:17:45.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.909+0000 7f2104c00700 1 -- 172.21.15.43:0/2674379356 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f210002c030 con 0x560b32c31e40 2024-05-27T07:17:45.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.909+0000 7f2104c00700 1 -- 172.21.15.43:0/2674379356 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100004710 con 0x560b32c31e40 2024-05-27T07:17:45.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.910+0000 7f2125c11b80 1 -- 172.21.15.43:0/2674379356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 msgr2=0x560b32db5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:45.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.910+0000 7f2125c11b80 1 --2- 172.21.15.43:0/2674379356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32db5d90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f210000fa40 tx=0x7f210001ae40 comp rx=0 tx=0).stop 2024-05-27T07:17:45.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.910+0000 7f2125c11b80 1 -- 172.21.15.43:0/2674379356 shutdown_connections 2024-05-27T07:17:45.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.910+0000 7f2125c11b80 1 --2- 172.21.15.43:0/2674379356 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32db0820 0x560b32dbac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.910+0000 7f2125c11b80 1 --2- 172.21.15.43:0/2674379356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32db5d90 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.910+0000 7f2125c11b80 1 --2- 172.21.15.43:0/2674379356 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0410 0x560b32dbca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.910+0000 7f2125c11b80 1 -- 172.21.15.43:0/2674379356 >> 172.21.15.43:0/2674379356 conn(0x560b32c91fc0 msgr2=0x560b32dba560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:45.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.910+0000 7f2125c11b80 1 -- 172.21.15.43:0/2674379356 shutdown_connections 2024-05-27T07:17:45.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.910+0000 7f2125c11b80 1 -- 172.21.15.43:0/2674379356 wait complete. 2024-05-27T07:17:45.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.911+0000 7f2125c11b80 1 -- start start 2024-05-27T07:17:45.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.911+0000 7f2125c11b80 1 Processor -- start 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32dbf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32db0410 0x560b32dbf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f2125c11b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0820 0x560b32e34560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f2125c11b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f21101b2a50 con 0x560b32c31e40 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f2125c11b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f21101b2380 con 0x560b32db0820 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f2125c11b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f21101b3180 con 0x560b32db0410 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f2125c11b80 10 client.0 _ll_register_callbacks cb 0x560b32c78c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f210fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32dbf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f210fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32dbf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33054/0 (socket says 172.21.15.43:33054) 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f210fe00700 1 -- 172.21.15.43:0/1348717734 learned_addr learned my addr 172.21.15.43:0/1348717734 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.912+0000 7f2107400700 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32db0410 0x560b32dbf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.913+0000 7f2115000700 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0820 0x560b32e34560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.913+0000 7f210fe00700 1 -- 172.21.15.43:0/1348717734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32db0410 msgr2=0x560b32dbf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:45.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.913+0000 7f210fe00700 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32db0410 0x560b32dbf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.913+0000 7f210fe00700 1 -- 172.21.15.43:0/1348717734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0820 msgr2=0x560b32e34560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:45.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.913+0000 7f210fe00700 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0820 0x560b32e34560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.913+0000 7f210fe00700 1 -- 172.21.15.43:0/1348717734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b32e34aa0 con 0x560b32c31e40 2024-05-27T07:17:45.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.913+0000 7f210fe00700 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32dbf130 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f210001aeb0 tx=0x7f210000bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e47e0d548fbfc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:45.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.913+0000 7f20fe000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:45.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.914+0000 7f20fe000700 1 -- 172.21.15.43:0/1348717734 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100015070 con 0x560b32c31e40 2024-05-27T07:17:45.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.914+0000 7f20fe000700 1 -- 172.21.15.43:0/1348717734 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9408+0+0 (secure 0 0 0) 0x7f2100005480 con 0x560b32c31e40 2024-05-27T07:17:45.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.914+0000 7f2125c11b80 1 -- 172.21.15.43:0/1348717734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f210002c030 con 0x560b32c31e40 2024-05-27T07:17:45.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.914+0000 7f20fe000700 1 -- 172.21.15.43:0/1348717734 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f2100005680 con 0x560b32c31e40 2024-05-27T07:17:45.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.914+0000 7f2125c11b80 10 client.16014 fetch_fsmap learned FSMap version 55 2024-05-27T07:17:45.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.914+0000 7f2125c11b80 1 -- 172.21.15.43:0/1348717734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x560b32e35ca0 con 0x560b32c31e40 2024-05-27T07:17:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.914+0000 7f20fe000700 1 -- 172.21.15.43:0/1348717734 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f210002c030 con 0x560b32c31e40 2024-05-27T07:17:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.915+0000 7f20fe000700 1 -- 172.21.15.43:0/1348717734 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 55) ==== 7773+0+0 (secure 0 0 0) 0x7f2100003bb0 con 0x560b32c31e40 2024-05-27T07:17:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.915+0000 7f2125c11b80 10 client.16014 fetch_fsmap finished waiting for FSMap version 55 2024-05-27T07:17:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.915+0000 7f2125c11b80 10 client.16014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:17:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.915+0000 7f2125c11b80 20 client.16014 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.915+0000 7f2125c11b80 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560b32e36340 0x560b32e38760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.915+0000 7f2125c11b80 4 client.16014 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:17:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.915+0000 7f2125c11b80 1 -- 172.21.15.43:0/1348717734 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560b32cda570 con 0x560b32e36340 2024-05-27T07:17:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.916+0000 7f2107400700 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560b32e36340 0x560b32e38760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:45.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.916+0000 7f2107400700 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560b32e36340 0x560b32e38760 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9674ed39533364ff server_cookie=8584446b4740dc6d in_seq=0 out_seq=0 2024-05-27T07:17:45.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.916+0000 7f20fe000700 10 client.16014 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:45.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f20fe000700 1 -- 172.21.15.43:0/1348717734 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560b32cda570 con 0x560b32e36340 2024-05-27T07:17:45.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f20fe000700 10 client.16014 handle_command_reply: tid=0 2024-05-27T07:17:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f2125c11b80 1 client.16014 shutdown 2024-05-27T07:17:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f2125c11b80 1 -- 172.21.15.43:0/1348717734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 msgr2=0x560b32dbf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f2125c11b80 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32dbf130 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f210001aeb0 tx=0x7f210000bc90 comp rx=0 tx=0).stop 2024-05-27T07:17:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f2125c11b80 1 -- 172.21.15.43:0/1348717734 shutdown_connections 2024-05-27T07:17:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f2125c11b80 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560b32db0820 0x560b32e34560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f2125c11b80 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560b32e36340 0x560b32e38760 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f2125c11b80 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b32c31e40 0x560b32dbf130 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f2125c11b80 1 --2- 172.21.15.43:0/1348717734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b32db0410 0x560b32dbf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.919+0000 7f2125c11b80 1 -- 172.21.15.43:0/1348717734 >> 172.21.15.43:0/1348717734 conn(0x560b32c91fc0 msgr2=0x560b32dba380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.920+0000 7f20fe000700 0 client.16014 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.920+0000 7f2125c11b80 1 -- 172.21.15.43:0/1348717734 shutdown_connections 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.920+0000 7f2125c11b80 1 -- 172.21.15.43:0/1348717734 wait complete. 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.920+0000 7f2125c11b80 20 client.16014 trim_cache size 0 max 16384 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:17:45.912 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:17:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 -- 172.21.15.43:0/730749840 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f20f40b29f0 msgr2=0x7f20f40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f20f40b29f0 0x7f20f40b4f00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x560b32dac690 tx=0x7f2108004b40 comp rx=0 tx=0).stop 2024-05-27T07:17:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 -- 172.21.15.43:0/730749840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f211010af10 msgr2=0x7f211019d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f211010af10 0x7f211019d340 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f20f800b580 tx=0x7f20f800f460 comp rx=0 tx=0).stop 2024-05-27T07:17:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 -- 172.21.15.43:0/730749840 shutdown_connections 2024-05-27T07:17:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f20f40b29f0 0x7f20f40b4f00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f211010b860 0x7f21101a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2110106630 0x7f211019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 --2- 172.21.15.43:0/730749840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f211010af10 0x7f211019d340 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 -- 172.21.15.43:0/730749840 >> 172.21.15.43:0/730749840 conn(0x7f2110102020 msgr2=0x7f211010c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 -- 172.21.15.43:0/730749840 shutdown_connections 2024-05-27T07:17:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:45.921+0000 7f2115a00700 1 -- 172.21.15.43:0/730749840 wait complete. 2024-05-27T07:17:46.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 13939 0.84 MB/sec warmup 110 sec latency 53.603 ms 2024-05-27T07:17:46.202 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:45 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:17:46.202 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:45 smithi043 systemd[1]: Stopped Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:46.202 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:45 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.586s CPU time. 2024-05-27T07:17:46.202 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:45 smithi043 systemd[1]: Starting Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:17:46.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:46 smithi132 ceph-mon[35283]: pgmap v129: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 314 KiB/s wr, 46 op/s 2024-05-27T07:17:46.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:46 smithi043 ceph-mon[31531]: pgmap v129: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 314 KiB/s wr, 46 op/s 2024-05-27T07:17:46.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:46 smithi179 ceph-mon[35888]: pgmap v129: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 314 KiB/s wr, 46 op/s 2024-05-27T07:17:46.879 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:17:46.879 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:17:47.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 13971 0.84 MB/sec warmup 111 sec latency 52.519 ms 2024-05-27T07:17:47.394 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:47 smithi043 podman[98504]: 2024-05-27 07:17:47.039284965 +0000 UTC m=+0.035785288 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:17:47.394 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:47 smithi043 podman[98504]: 2024-05-27 07:17:47.162822306 +0000 UTC m=+0.159322656 container create 9fa7906c7c399918df129b008775e288646466a506d1bcdbec6c9bd447854bf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-05-27T07:17:47.625 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:47.702 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:47 smithi043 podman[98504]: 2024-05-27 07:17:47.400916039 +0000 UTC m=+0.397416428 container init 9fa7906c7c399918df129b008775e288646466a506d1bcdbec6c9bd447854bf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:17:47.702 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:47 smithi043 podman[98504]: 2024-05-27 07:17:47.414885459 +0000 UTC m=+0.411385827 container start 9fa7906c7c399918df129b008775e288646466a506d1bcdbec6c9bd447854bf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:17:47.702 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:47 smithi043 bash[98504]: 9fa7906c7c399918df129b008775e288646466a506d1bcdbec6c9bd447854bf6 2024-05-27T07:17:47.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:47 smithi043 systemd[1]: Started Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:47.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:47 smithi043 ceph-mds[98546]: -- 172.21.15.43:0/2289834946 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627d2f95a40 con 0x5627d2fdd180 2024-05-27T07:17:47.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:47 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: starting mds.d at 2024-05-27T07:17:48.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 14037 0.83 MB/sec warmup 112 sec latency 42.769 ms 2024-05-27T07:17:48.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:48 smithi043 ceph-mon[31531]: pgmap v130: 129 pgs: 129 active+clean; 36 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 449 KiB/s wr, 44 op/s 2024-05-27T07:17:48.498 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:48 smithi043 ceph-mon[31531]: daemon mds.d restarted 2024-05-27T07:17:48.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:48 smithi179 ceph-mon[35888]: pgmap v130: 129 pgs: 129 active+clean; 36 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 449 KiB/s wr, 44 op/s 2024-05-27T07:17:48.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:48 smithi179 ceph-mon[35888]: daemon mds.d restarted 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1e1a200700 time 2024-05-27T07:17:48.466016+0000 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1e2fd16ba6] 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1e2fd16dd1] 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5627c2c4d23d] 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5627c2c38971] 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 5: (SnapRealm::check_cache() const+0x13e) [0x5627c2c36fa0] 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5627c2c37651] 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627c2bcc347] 2024-05-27T07:17:48.716 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627c28c887e] 2024-05-27T07:17:48.717 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5627c28cde94] 2024-05-27T07:17:48.717 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5627c28d8969] 2024-05-27T07:17:48.717 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 11: (MDLog::_replay_thread()+0x1f38) [0x5627c2c7148c] 2024-05-27T07:17:48.717 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5627c2c7a9cb] 2024-05-27T07:17:48.717 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 13: (Thread::entry_wrapper()+0x43) [0x7f1e2fce572d] 2024-05-27T07:17:48.717 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1e2fce5749] 2024-05-27T07:17:48.717 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1e2e8891ca] 2024-05-27T07:17:48.717 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 16: clone() 2024-05-27T07:17:48.717 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: *** Caught signal (Aborted) ** 2024-05-27T07:17:48.717 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: in thread 7f1e1a200700 thread_name:md_log_replay 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.467+0000 7f1e1a200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1e1a200700 time 2024-05-27T07:17:48.466016+0000 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1e2fd16ba6] 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1e2fd16dd1] 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5627c2c4d23d] 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5627c2c38971] 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 5: (SnapRealm::check_cache() const+0x13e) [0x5627c2c36fa0] 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5627c2c37651] 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627c2bcc347] 2024-05-27T07:17:48.718 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627c28c887e] 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5627c28cde94] 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5627c28d8969] 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 11: (MDLog::_replay_thread()+0x1f38) [0x5627c2c7148c] 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5627c2c7a9cb] 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 13: (Thread::entry_wrapper()+0x43) [0x7f1e2fce572d] 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1e2fce5749] 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1e2e8891ca] 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 16: clone() 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1e2e893d20] 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2: gsignal() 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 3: abort() 2024-05-27T07:17:48.719 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1e2fd16ced] 2024-05-27T07:17:48.720 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1e2fd16dd1] 2024-05-27T07:17:48.720 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5627c2c4d23d] 2024-05-27T07:17:48.720 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5627c2c38971] 2024-05-27T07:17:48.720 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 8: (SnapRealm::check_cache() const+0x13e) [0x5627c2c36fa0] 2024-05-27T07:17:48.720 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5627c2c37651] 2024-05-27T07:17:48.720 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627c2bcc347] 2024-05-27T07:17:48.721 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627c28c887e] 2024-05-27T07:17:48.721 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5627c28cde94] 2024-05-27T07:17:48.721 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5627c28d8969] 2024-05-27T07:17:48.721 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 14: (MDLog::_replay_thread()+0x1f38) [0x5627c2c7148c] 2024-05-27T07:17:48.721 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5627c2c7a9cb] 2024-05-27T07:17:48.721 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 16: (Thread::entry_wrapper()+0x43) [0x7f1e2fce572d] 2024-05-27T07:17:48.721 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1e2fce5749] 2024-05-27T07:17:48.721 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1e2e8891ca] 2024-05-27T07:17:48.722 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 19: clone() 2024-05-27T07:17:48.722 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.471+0000 7f1e1a200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:48.722 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: in thread 7f1e1a200700 thread_name:md_log_replay 2024-05-27T07:17:48.722 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.722 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:48.723 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1e2e893d20] 2024-05-27T07:17:48.723 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2: gsignal() 2024-05-27T07:17:48.723 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 3: abort() 2024-05-27T07:17:48.723 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1e2fd16ced] 2024-05-27T07:17:48.723 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1e2fd16dd1] 2024-05-27T07:17:48.724 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5627c2c4d23d] 2024-05-27T07:17:48.724 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5627c2c38971] 2024-05-27T07:17:48.724 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 8: (SnapRealm::check_cache() const+0x13e) [0x5627c2c36fa0] 2024-05-27T07:17:48.724 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5627c2c37651] 2024-05-27T07:17:48.724 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627c2bcc347] 2024-05-27T07:17:48.724 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627c28c887e] 2024-05-27T07:17:48.724 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5627c28cde94] 2024-05-27T07:17:48.725 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5627c28d8969] 2024-05-27T07:17:48.725 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 14: (MDLog::_replay_thread()+0x1f38) [0x5627c2c7148c] 2024-05-27T07:17:48.725 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5627c2c7a9cb] 2024-05-27T07:17:48.725 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 16: (Thread::entry_wrapper()+0x43) [0x7f1e2fce572d] 2024-05-27T07:17:48.725 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1e2fce5749] 2024-05-27T07:17:48.725 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1e2e8891ca] 2024-05-27T07:17:48.726 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 19: clone() 2024-05-27T07:17:48.726 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:48.726 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.726 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: -1> 2024-05-27T07:17:48.467+0000 7f1e1a200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1e1a200700 time 2024-05-27T07:17:48.466016+0000 2024-05-27T07:17:48.726 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:48.727 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.727 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:48.727 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1e2fd16ba6] 2024-05-27T07:17:48.727 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1e2fd16dd1] 2024-05-27T07:17:48.727 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5627c2c4d23d] 2024-05-27T07:17:48.727 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5627c2c38971] 2024-05-27T07:17:48.728 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 5: (SnapRealm::check_cache() const+0x13e) [0x5627c2c36fa0] 2024-05-27T07:17:48.728 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5627c2c37651] 2024-05-27T07:17:48.728 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627c2bcc347] 2024-05-27T07:17:48.728 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627c28c887e] 2024-05-27T07:17:48.729 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5627c28cde94] 2024-05-27T07:17:48.730 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5627c28d8969] 2024-05-27T07:17:48.730 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 11: (MDLog::_replay_thread()+0x1f38) [0x5627c2c7148c] 2024-05-27T07:17:48.730 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5627c2c7a9cb] 2024-05-27T07:17:48.730 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 13: (Thread::entry_wrapper()+0x43) [0x7f1e2fce572d] 2024-05-27T07:17:48.730 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1e2fce5749] 2024-05-27T07:17:48.731 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1e2e8891ca] 2024-05-27T07:17:48.731 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 16: clone() 2024-05-27T07:17:48.731 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.731 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 0> 2024-05-27T07:17:48.471+0000 7f1e1a200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:48.731 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: in thread 7f1e1a200700 thread_name:md_log_replay 2024-05-27T07:17:48.731 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.732 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:48.732 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1e2e893d20] 2024-05-27T07:17:48.732 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2: gsignal() 2024-05-27T07:17:48.732 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 3: abort() 2024-05-27T07:17:48.732 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1e2fd16ced] 2024-05-27T07:17:48.732 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1e2fd16dd1] 2024-05-27T07:17:48.733 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5627c2c4d23d] 2024-05-27T07:17:48.733 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5627c2c38971] 2024-05-27T07:17:48.733 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 8: (SnapRealm::check_cache() const+0x13e) [0x5627c2c36fa0] 2024-05-27T07:17:48.733 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5627c2c37651] 2024-05-27T07:17:48.733 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627c2bcc347] 2024-05-27T07:17:48.733 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627c28c887e] 2024-05-27T07:17:48.734 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5627c28cde94] 2024-05-27T07:17:48.734 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5627c28d8969] 2024-05-27T07:17:48.734 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 14: (MDLog::_replay_thread()+0x1f38) [0x5627c2c7148c] 2024-05-27T07:17:48.734 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5627c2c7a9cb] 2024-05-27T07:17:48.734 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 16: (Thread::entry_wrapper()+0x43) [0x7f1e2fce572d] 2024-05-27T07:17:48.734 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1e2fce5749] 2024-05-27T07:17:48.735 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1e2e8891ca] 2024-05-27T07:17:48.735 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 19: clone() 2024-05-27T07:17:48.735 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:48.735 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.735 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: -1> 2024-05-27T07:17:48.467+0000 7f1e1a200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1e1a200700 time 2024-05-27T07:17:48.466016+0000 2024-05-27T07:17:48.735 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:48.736 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.736 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:48.736 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1e2fd16ba6] 2024-05-27T07:17:48.736 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1e2fd16dd1] 2024-05-27T07:17:48.736 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5627c2c4d23d] 2024-05-27T07:17:48.736 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5627c2c38971] 2024-05-27T07:17:48.737 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 5: (SnapRealm::check_cache() const+0x13e) [0x5627c2c36fa0] 2024-05-27T07:17:48.737 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5627c2c37651] 2024-05-27T07:17:48.737 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627c2bcc347] 2024-05-27T07:17:48.737 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627c28c887e] 2024-05-27T07:17:48.737 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5627c28cde94] 2024-05-27T07:17:48.737 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5627c28d8969] 2024-05-27T07:17:48.738 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 11: (MDLog::_replay_thread()+0x1f38) [0x5627c2c7148c] 2024-05-27T07:17:48.738 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5627c2c7a9cb] 2024-05-27T07:17:48.738 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 13: (Thread::entry_wrapper()+0x43) [0x7f1e2fce572d] 2024-05-27T07:17:48.738 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1e2fce5749] 2024-05-27T07:17:48.738 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1e2e8891ca] 2024-05-27T07:17:48.739 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 16: clone() 2024-05-27T07:17:48.739 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.739 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 0> 2024-05-27T07:17:48.471+0000 7f1e1a200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:48.739 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: in thread 7f1e1a200700 thread_name:md_log_replay 2024-05-27T07:17:48.739 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.739 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:48.740 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1e2e893d20] 2024-05-27T07:17:48.740 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2: gsignal() 2024-05-27T07:17:48.740 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 3: abort() 2024-05-27T07:17:48.740 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1e2fd16ced] 2024-05-27T07:17:48.740 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1e2fd16dd1] 2024-05-27T07:17:48.740 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5627c2c4d23d] 2024-05-27T07:17:48.741 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5627c2c38971] 2024-05-27T07:17:48.741 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 8: (SnapRealm::check_cache() const+0x13e) [0x5627c2c36fa0] 2024-05-27T07:17:48.741 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5627c2c37651] 2024-05-27T07:17:48.741 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5627c2bcc347] 2024-05-27T07:17:48.741 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5627c28c887e] 2024-05-27T07:17:48.742 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5627c28cde94] 2024-05-27T07:17:48.742 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5627c28d8969] 2024-05-27T07:17:48.742 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 14: (MDLog::_replay_thread()+0x1f38) [0x5627c2c7148c] 2024-05-27T07:17:48.742 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5627c2c7a9cb] 2024-05-27T07:17:48.742 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 16: (Thread::entry_wrapper()+0x43) [0x7f1e2fce572d] 2024-05-27T07:17:48.742 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1e2fce5749] 2024-05-27T07:17:48.743 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1e2e8891ca] 2024-05-27T07:17:48.743 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 19: clone() 2024-05-27T07:17:48.743 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:48.743 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[98531]: 2024-05-27T07:17:48.743 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:48 smithi043 podman[98623]: 2024-05-27 07:17:48.723882584 +0000 UTC m=+0.176014705 container died 9fa7906c7c399918df129b008775e288646466a506d1bcdbec6c9bd447854bf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:17:48.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:48 smithi132 ceph-mon[35283]: pgmap v130: 129 pgs: 129 active+clean; 36 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 449 KiB/s wr, 44 op/s 2024-05-27T07:17:48.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:48 smithi132 ceph-mon[35283]: daemon mds.d restarted 2024-05-27T07:17:49.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 14218 0.85 MB/sec warmup 113 sec latency 55.829 ms 2024-05-27T07:17:49.546 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:49 smithi043 ceph-mon[31531]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:17:49.546 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:49 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6836/1218271778,v1:172.21.15.43:6837/1218271778] up:boot 2024-05-27T07:17:49.546 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:49 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:49.546 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:49 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:17:49.546 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:49 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:49 smithi179 ceph-mon[35888]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:17:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:49 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6836/1218271778,v1:172.21.15.43:6837/1218271778] up:boot 2024-05-27T07:17:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:49 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:49 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:17:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:49 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:49 smithi132 ceph-mon[35283]: osdmap e103: 12 total, 12 up, 12 in 2024-05-27T07:17:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:49 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6836/1218271778,v1:172.21.15.43:6837/1218271778] up:boot 2024-05-27T07:17:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:49 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-05-27T07:17:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:49 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:17:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:49 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:17:50.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 14343 0.84 MB/sec warmup 114 sec latency 52.420 ms 2024-05-27T07:17:50.323 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:50 smithi043 podman[98623]: 2024-05-27 07:17:50.009124398 +0000 UTC m=+1.461256501 container remove 9fa7906c7c399918df129b008775e288646466a506d1bcdbec6c9bd447854bf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-05-27T07:17:50.323 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:50 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:17:50.323 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:49 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:17:50.323 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:49 smithi043 systemd[1]: Stopped Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:50.324 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:49 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Consumed 1.501s CPU time. 2024-05-27T07:17:50.324 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:49 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Start request repeated too quickly. 2024-05-27T07:17:50.324 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:49 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-05-27T07:17:50.324 INFO:journalctl@ceph.mds.g.smithi043.stdout:May 27 07:17:49 smithi043 systemd[1]: Failed to start Ceph mds.g for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:17:50.687 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:50 smithi043 ceph-mon[31531]: pgmap v132: 129 pgs: 129 active+clean; 36 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 449 KiB/s wr, 44 op/s 2024-05-27T07:17:50.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:50 smithi179 ceph-mon[35888]: pgmap v132: 129 pgs: 129 active+clean; 36 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 449 KiB/s wr, 44 op/s 2024-05-27T07:17:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.820+0000 7fbfd7000700 1 -- 172.21.15.43:0/959556320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfd0109860 msgr2=0x7fbfd01111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.820+0000 7fbfd7000700 1 --2- 172.21.15.43:0/959556320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfd0109860 0x7fbfd01111c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fbfc800b0d0 tx=0x7fbfc800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.821+0000 7fbfd7000700 1 -- 172.21.15.43:0/959556320 shutdown_connections 2024-05-27T07:17:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.821+0000 7fbfd7000700 1 --2- 172.21.15.43:0/959556320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfd0109860 0x7fbfd01111c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.821+0000 7fbfd7000700 1 --2- 172.21.15.43:0/959556320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbfd01045e0 0x7fbfd01049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.821+0000 7fbfd7000700 1 --2- 172.21.15.43:0/959556320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd0108f10 0x7fbfd0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.821+0000 7fbfd7000700 1 -- 172.21.15.43:0/959556320 >> 172.21.15.43:0/959556320 conn(0x7fbfd0100030 msgr2=0x7fbfd0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.821+0000 7fbfd7000700 1 -- 172.21.15.43:0/959556320 shutdown_connections 2024-05-27T07:17:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.821+0000 7fbfd7000700 1 -- 172.21.15.43:0/959556320 wait complete. 2024-05-27T07:17:50.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.822+0000 7fbfd7000700 1 Processor -- start 2024-05-27T07:17:50.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.822+0000 7fbfd7000700 1 -- start start 2024-05-27T07:17:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.823+0000 7fbfd7000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd01045e0 0x7fbfd0075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.823+0000 7fbfd7000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfd0108f10 0x7fbfd0075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.823+0000 7fbfd7000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbfd0109860 0x7fbfd0076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.823+0000 7fbfd7000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0114390 con 0x7fbfd0108f10 2024-05-27T07:17:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.823+0000 7fbfd7000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0114210 con 0x7fbfd0109860 2024-05-27T07:17:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.823+0000 7fbfd7000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0114510 con 0x7fbfd01045e0 2024-05-27T07:17:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.823+0000 7fbfc7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfd0108f10 0x7fbfd0075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.824+0000 7fbfc7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfd0108f10 0x7fbfd0075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33092/0 (socket says 172.21.15.43:33092) 2024-05-27T07:17:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.824+0000 7fbfc7e00700 1 -- 172.21.15.43:0/3826444753 learned_addr learned my addr 172.21.15.43:0/3826444753 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.824+0000 7fbfcfe00700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd01045e0 0x7fbfd0075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.824+0000 7fbfd5000700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbfd0109860 0x7fbfd0076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.824+0000 7fbfc7e00700 1 -- 172.21.15.43:0/3826444753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd01045e0 msgr2=0x7fbfd0075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.824+0000 7fbfc7e00700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd01045e0 0x7fbfd0075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.824+0000 7fbfc7e00700 1 -- 172.21.15.43:0/3826444753 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbfd0109860 msgr2=0x7fbfd0076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.824+0000 7fbfc7e00700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbfd0109860 0x7fbfd0076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.824+0000 7fbfc7e00700 1 -- 172.21.15.43:0/3826444753 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd01a9070 con 0x7fbfd0108f10 2024-05-27T07:17:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.825+0000 7fbfc7e00700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfd0108f10 0x7fbfd0075ff0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800d980 tx=0x7fbfb800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.825+0000 7fbfcd600700 1 -- 172.21.15.43:0/3826444753 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8017070 con 0x7fbfd0108f10 2024-05-27T07:17:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.825+0000 7fbfcd600700 1 -- 172.21.15.43:0/3826444753 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbfb800fd20 con 0x7fbfd0108f10 2024-05-27T07:17:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.825+0000 7fbfcd600700 1 -- 172.21.15.43:0/3826444753 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb800e3b0 con 0x7fbfd0108f10 2024-05-27T07:17:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.825+0000 7fbfd7000700 1 -- 172.21.15.43:0/3826444753 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd01a9360 con 0x7fbfd0108f10 2024-05-27T07:17:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.825+0000 7fbfd7000700 1 -- 172.21.15.43:0/3826444753 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd01a9850 con 0x7fbfd0108f10 2024-05-27T07:17:50.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.827+0000 7fbfd7000700 1 -- 172.21.15.43:0/3826444753 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd00512b0 con 0x7fbfd0108f10 2024-05-27T07:17:50.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.830+0000 7fbfcd600700 1 -- 172.21.15.43:0/3826444753 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fbfb800f890 con 0x7fbfd0108f10 2024-05-27T07:17:50.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.831+0000 7fbfcd600700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbfb40b2af0 0x7fbfb40b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:50.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.832+0000 7fbfcd600700 1 -- 172.21.15.43:0/3826444753 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9408+0+0 (secure 0 0 0) 0x7fbfb80e2340 con 0x7fbfd0108f10 2024-05-27T07:17:50.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.832+0000 7fbfcfe00700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbfb40b2af0 0x7fbfb40b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:50.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.833+0000 7fbfcfe00700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbfb40b2af0 0x7fbfb40b5000 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000ac50 tx=0x7fbfc000a380 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:50.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:50.834+0000 7fbfcd600700 1 -- 172.21.15.43:0/3826444753 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbfb80a99f0 con 0x7fbfd0108f10 2024-05-27T07:17:50.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:50 smithi132 ceph-mon[35283]: pgmap v132: 129 pgs: 129 active+clean; 36 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 449 KiB/s wr, 44 op/s 2024-05-27T07:17:50.952 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:50 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:17:50.952 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:17:50 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.655s CPU time. 2024-05-27T07:17:51.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.142+0000 7fbfd7000700 1 -- 172.21.15.43:0/3826444753 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfd002e5e0 con 0x7fbfd0108f10 2024-05-27T07:17:51.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.144+0000 7fbfcd600700 1 -- 172.21.15.43:0/3826444753 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+11119 (secure 0 0 0) 0x7fbfb80a92f0 con 0x7fbfd0108f10 2024-05-27T07:17:51.136 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:17:51.136 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/3664629623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3664629623},{"type":"v1","addr":"172.21.15.43:6835","nonce":3664629623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":57,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:17:48.359271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_15996":{"gid":15996,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1084201245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1084201245},{"type":"v1","addr":"172.21.15.43:6837","nonce":1084201245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16020":{"gid":16020,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1218271778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1218271778},{"type":"v1","addr":"172.21.15.43:6837","nonce":1218271778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[24484,34403,34424,34448,34484]},"id":1}]} 2024-05-27T07:17:51.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.148+0000 7fbfd7000700 1 -- 172.21.15.43:0/3826444753 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbfb40b2af0 msgr2=0x7fbfb40b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:51.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.148+0000 7fbfd7000700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbfb40b2af0 0x7fbfb40b5000 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000ac50 tx=0x7fbfc000a380 comp rx=0 tx=0).stop 2024-05-27T07:17:51.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.148+0000 7fbfd7000700 1 -- 172.21.15.43:0/3826444753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfd0108f10 msgr2=0x7fbfd0075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:51.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.148+0000 7fbfd7000700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfd0108f10 0x7fbfd0075ff0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800d980 tx=0x7fbfb800dc90 comp rx=0 tx=0).stop 2024-05-27T07:17:51.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.148+0000 7fbfd7000700 1 -- 172.21.15.43:0/3826444753 shutdown_connections 2024-05-27T07:17:51.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.149+0000 7fbfd7000700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbfb40b2af0 0x7fbfb40b5000 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:51.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.149+0000 7fbfd7000700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbfd0109860 0x7fbfd0076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:51.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.149+0000 7fbfd7000700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd01045e0 0x7fbfd0075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:51.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.149+0000 7fbfd7000700 1 --2- 172.21.15.43:0/3826444753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfd0108f10 0x7fbfd0075ff0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:51.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.149+0000 7fbfd7000700 1 -- 172.21.15.43:0/3826444753 >> 172.21.15.43:0/3826444753 conn(0x7fbfd0100030 msgr2=0x7fbfd010a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:51.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.149+0000 7fbfd7000700 1 -- 172.21.15.43:0/3826444753 shutdown_connections 2024-05-27T07:17:51.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:51.149+0000 7fbfd7000700 1 -- 172.21.15.43:0/3826444753 wait complete. 2024-05-27T07:17:51.142 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 57 2024-05-27T07:17:51.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 14444 0.84 MB/sec warmup 115 sec latency 47.542 ms 2024-05-27T07:17:51.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:51 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3826444753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:17:51.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:51 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3826444753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:17:51.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:51 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3826444753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:17:52.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:17:52.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 14505 0.84 MB/sec warmup 116 sec latency 43.310 ms 2024-05-27T07:17:52.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:52 smithi043 ceph-mon[31531]: pgmap v133: 129 pgs: 129 active+clean; 37 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 458 KiB/s wr, 44 op/s 2024-05-27T07:17:52.505 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:52.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:52 smithi179 ceph-mon[35888]: pgmap v133: 129 pgs: 129 active+clean; 37 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 458 KiB/s wr, 44 op/s 2024-05-27T07:17:52.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:52 smithi132 ceph-mon[35283]: pgmap v133: 129 pgs: 129 active+clean; 37 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 458 KiB/s wr, 44 op/s 2024-05-27T07:17:53.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 14577 0.83 MB/sec warmup 117 sec latency 49.041 ms 2024-05-27T07:17:54.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 14637 0.83 MB/sec warmup 118 sec latency 50.401 ms 2024-05-27T07:17:54.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:54 smithi043 ceph-mon[31531]: pgmap v134: 129 pgs: 129 active+clean; 37 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 458 KiB/s wr, 44 op/s 2024-05-27T07:17:54.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:54 smithi179 ceph-mon[35888]: pgmap v134: 129 pgs: 129 active+clean; 37 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 458 KiB/s wr, 44 op/s 2024-05-27T07:17:54.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:54 smithi132 ceph-mon[35283]: pgmap v134: 129 pgs: 129 active+clean; 37 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 458 KiB/s wr, 44 op/s 2024-05-27T07:17:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.084+0000 7f05d5000700 1 -- 172.21.15.43:0/2270856832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05d0105bd0 msgr2=0x7f05d010e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.084+0000 7f05d5000700 1 --2- 172.21.15.43:0/2270856832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05d0105bd0 0x7f05d010e120 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009990 tx=0x7f05bc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.085+0000 7f05d5000700 1 -- 172.21.15.43:0/2270856832 shutdown_connections 2024-05-27T07:17:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.085+0000 7f05d5000700 1 --2- 172.21.15.43:0/2270856832 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05d010e6f0 0x7f05d0110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.085+0000 7f05d5000700 1 --2- 172.21.15.43:0/2270856832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05d0105280 0x7f05d0105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.085+0000 7f05d5000700 1 --2- 172.21.15.43:0/2270856832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05d0105bd0 0x7f05d010e120 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.085+0000 7f05d5000700 1 -- 172.21.15.43:0/2270856832 >> 172.21.15.43:0/2270856832 conn(0x7f05d0078af0 msgr2=0x7f05d0078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.085+0000 7f05d5000700 1 -- 172.21.15.43:0/2270856832 shutdown_connections 2024-05-27T07:17:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.085+0000 7f05d5000700 1 -- 172.21.15.43:0/2270856832 wait complete. 2024-05-27T07:17:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.086+0000 7f05d5000700 1 Processor -- start 2024-05-27T07:17:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.086+0000 7f05d5000700 1 -- start start 2024-05-27T07:17:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.086+0000 7f05d5000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05d0105280 0x7f05d019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.086+0000 7f05d5000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05d0105bd0 0x7f05d019d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05d5000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05d010e6f0 0x7f05d01a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05d5000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f05d0114100 con 0x7f05d0105280 2024-05-27T07:17:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05d5000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f05d0113f80 con 0x7f05d0105bd0 2024-05-27T07:17:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05d5000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f05d0114280 con 0x7f05d010e6f0 2024-05-27T07:17:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05cf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05d0105280 0x7f05d019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05cf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05d0105280 0x7f05d019cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47062/0 (socket says 172.21.15.43:47062) 2024-05-27T07:17:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05cf400700 1 -- 172.21.15.43:0/4156103445 learned_addr learned my addr 172.21.15.43:0/4156103445 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05cea00700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05d0105bd0 0x7f05d019d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05cfe00700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05d010e6f0 0x7f05d01a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05cf400700 1 -- 172.21.15.43:0/4156103445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05d010e6f0 msgr2=0x7f05d01a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05cf400700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05d010e6f0 0x7f05d01a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05cf400700 1 -- 172.21.15.43:0/4156103445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05d0105bd0 msgr2=0x7f05d019d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.087+0000 7f05cf400700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05d0105bd0 0x7f05d019d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.088+0000 7f05cf400700 1 -- 172.21.15.43:0/4156103445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d01a1c20 con 0x7f05d0105280 2024-05-27T07:17:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.088+0000 7f05cf400700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05d0105280 0x7f05d019cd50 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f05c000d950 tx=0x7f05c000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.088+0000 7f05c7e00700 1 -- 172.21.15.43:0/4156103445 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c0016070 con 0x7f05d0105280 2024-05-27T07:17:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.088+0000 7f05d5000700 1 -- 172.21.15.43:0/4156103445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d01a1f10 con 0x7f05d0105280 2024-05-27T07:17:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.088+0000 7f05c7e00700 1 -- 172.21.15.43:0/4156103445 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05c0010460 con 0x7f05d0105280 2024-05-27T07:17:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.088+0000 7f05d5000700 1 -- 172.21.15.43:0/4156103445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d01a9830 con 0x7f05d0105280 2024-05-27T07:17:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.088+0000 7f05c7e00700 1 -- 172.21.15.43:0/4156103445 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c000f620 con 0x7f05d0105280 2024-05-27T07:17:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.093+0000 7f05c7e00700 1 -- 172.21.15.43:0/4156103445 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f05c0019070 con 0x7f05d0105280 2024-05-27T07:17:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.094+0000 7f05e51c9b80 1 Processor -- start 2024-05-27T07:17:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.094+0000 7f05e51c9b80 1 -- start start 2024-05-27T07:17:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.094+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b197967ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.094+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b197972ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.094+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197975020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b197896e40 con 0x55b197968820 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b1978869e0 con 0x55b197968410 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b19788aa20 con 0x55b1977e9e40 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05c7e00700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f05b40b29f0 0x7f05b40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05cea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197975020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05cea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197975020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47076/0 (socket says 172.21.15.43:47076) 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05cea00700 1 -- 172.21.15.43:0/3977977739 learned_addr learned my addr 172.21.15.43:0/3977977739 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05cfe00700 1 --2- 172.21.15.43:0/3977977739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b197967ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05c7e00700 1 -- 172.21.15.43:0/4156103445 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9408+0+0 (secure 0 0 0) 0x7f05c00e25d0 con 0x7f05d0105280 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05cf400700 1 --2- 172.21.15.43:0/3977977739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b197972ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.095+0000 7f05cfe00700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f05b40b29f0 0x7f05b40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.096+0000 7f05cea00700 1 -- 172.21.15.43:0/3977977739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 msgr2=0x55b197967ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:55.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.096+0000 7f05cea00700 1 --2- 172.21.15.43:0/3977977739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b197967ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.096+0000 7f05cea00700 1 -- 172.21.15.43:0/3977977739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 msgr2=0x55b197972ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:55.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.096+0000 7f05cea00700 1 --2- 172.21.15.43:0/3977977739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b197972ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.096+0000 7f05cea00700 1 -- 172.21.15.43:0/3977977739 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b19784f420 con 0x55b197968820 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.096+0000 7f05cea00700 1 --2- 172.21.15.43:0/3977977739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197975020 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f05bc006010 tx=0x7f05bc0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc4c4b3218a2c8e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.096+0000 7f05cfe00700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f05b40b29f0 0x7f05b40b4f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55b197964690 tx=0x7f05c8009680 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.097+0000 7f05c4c00700 1 -- 172.21.15.43:0/3977977739 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc026070 con 0x55b197968820 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.097+0000 7f05c4c00700 1 -- 172.21.15.43:0/3977977739 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05bc003c50 con 0x55b197968820 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.097+0000 7f05c4c00700 1 -- 172.21.15.43:0/3977977739 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc00b7d0 con 0x55b197968820 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.098+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3977977739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 msgr2=0x55b197975020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.098+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3977977739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197975020 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f05bc006010 tx=0x7f05bc0096a0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.098+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3977977739 shutdown_connections 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.098+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3977977739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197975020 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.098+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3977977739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b197967ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.098+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3977977739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b197972ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.098+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3977977739 >> 172.21.15.43:0/3977977739 conn(0x55b197849fc0 msgr2=0x55b197963d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:55.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.098+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3977977739 shutdown_connections 2024-05-27T07:17:55.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.098+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3977977739 wait complete. 2024-05-27T07:17:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.099+0000 7f05e51c9b80 1 -- start start 2024-05-27T07:17:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.099+0000 7f05e51c9b80 1 Processor -- start 2024-05-27T07:17:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b197972560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b197972aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197972fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b19778cfa0 con 0x55b197968820 2024-05-27T07:17:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b197827760 con 0x55b197968410 2024-05-27T07:17:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b197975930 con 0x55b1977e9e40 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05e51c9b80 10 client.0 _ll_register_callbacks cb 0x55b197830c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05cf400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b197972560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05cfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197972fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05cf400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b197972560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:53814/0 (socket says 172.21.15.43:53814) 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05cf400700 1 -- 172.21.15.43:0/3973072932 learned_addr learned my addr 172.21.15.43:0/3973072932 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.100+0000 7f05cea00700 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b197972aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.101+0000 7f05cfe00700 1 -- 172.21.15.43:0/3973072932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 msgr2=0x55b197972560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.101+0000 7f05cfe00700 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b197972560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.101+0000 7f05cfe00700 1 -- 172.21.15.43:0/3973072932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 msgr2=0x55b197972aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.101+0000 7f05cfe00700 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b197972aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.101+0000 7f05cfe00700 1 -- 172.21.15.43:0/3973072932 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1977d68d0 con 0x55b197968820 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.101+0000 7f05cfe00700 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197972fe0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f05c80060b0 tx=0x7f05c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b4ac063043d3ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.101+0000 7f05b9600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.101+0000 7f05b9600700 1 -- 172.21.15.43:0/3973072932 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8004bb0 con 0x55b197968820 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.102+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3973072932 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05bc003c50 con 0x55b197968820 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.102+0000 7f05b9600700 1 -- 172.21.15.43:0/3973072932 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9408+0+0 (secure 0 0 0) 0x7f05c800f070 con 0x55b197968820 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.102+0000 7f05b9600700 1 -- 172.21.15.43:0/3973072932 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f05c802f050 con 0x55b197968820 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.102+0000 7f05e51c9b80 10 client.16038 fetch_fsmap learned FSMap version 57 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.102+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3973072932 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55b197857060 con 0x55b197968820 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.102+0000 7f05b9600700 1 -- 172.21.15.43:0/3973072932 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05bc003c50 con 0x55b197968820 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.103+0000 7f05b9600700 1 -- 172.21.15.43:0/3973072932 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 57) ==== 7773+0+0 (secure 0 0 0) 0x7f05c8017430 con 0x55b197968820 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.103+0000 7f05e51c9b80 10 client.16038 fetch_fsmap finished waiting for FSMap version 57 2024-05-27T07:17:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.103+0000 7f05e51c9b80 10 client.16038 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:17:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.103+0000 7f05e51c9b80 20 client.16038 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.103+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b197982850 0x55b197984ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.103+0000 7f05e51c9b80 4 client.16038 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:17:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.103+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3973072932 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b197898d90 con 0x55b197982850 2024-05-27T07:17:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.103+0000 7f05cf400700 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b197982850 0x55b197984ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.104+0000 7f05cf400700 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b197982850 0x55b197984ce0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b7cc0c900b6c1b58 server_cookie=6a77bfa20daf0048 in_seq=0 out_seq=0 2024-05-27T07:17:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.104+0000 7f05b9600700 10 client.16038 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:17:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.106+0000 7f05b9600700 1 -- 172.21.15.43:0/3973072932 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b197898d90 con 0x55b197982850 2024-05-27T07:17:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.106+0000 7f05b9600700 10 client.16038 handle_command_reply: tid=0 2024-05-27T07:17:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 client.16038 shutdown 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3973072932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 msgr2=0x55b197972fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197972fe0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f05c80060b0 tx=0x7f05c8004ad0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3973072932 shutdown_connections 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b197982850 0x55b197984ce0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197968820 0x55b197972fe0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b197972560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3973072932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b197972aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3973072932 >> 172.21.15.43:0/3973072932 conn(0x55b197849fc0 msgr2=0x55b197962ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05b9600700 0 client.16038 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3973072932 shutdown_connections 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.107+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3973072932 wait complete. 2024-05-27T07:17:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.108+0000 7f05e51c9b80 20 client.16038 trim_cache size 0 max 16384 2024-05-27T07:17:55.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.124+0000 7f05e51c9b80 1 Processor -- start 2024-05-27T07:17:55.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.124+0000 7f05e51c9b80 1 -- start start 2024-05-27T07:17:55.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.124+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b197974a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.124+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1977e9e40 0x55b19796dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.124+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b197968410 0x55b19797e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.124+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b3fc0 con 0x55b197849fc0 2024-05-27T07:17:55.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.124+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b1cf0 con 0x55b1977e9e40 2024-05-27T07:17:55.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.124+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b38f0 con 0x55b197968410 2024-05-27T07:17:55.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.125+0000 7f05cf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b197974a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.125+0000 7f05cf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b197974a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47098/0 (socket says 172.21.15.43:47098) 2024-05-27T07:17:55.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.125+0000 7f05cf400700 1 -- 172.21.15.43:0/3337502075 learned_addr learned my addr 172.21.15.43:0/3337502075 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:55.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.125+0000 7f05cea00700 1 --2- 172.21.15.43:0/3337502075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1977e9e40 0x55b19796dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.125+0000 7f05cfe00700 1 --2- 172.21.15.43:0/3337502075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b197968410 0x55b19797e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.125+0000 7f05cf400700 1 -- 172.21.15.43:0/3337502075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b197968410 msgr2=0x55b19797e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.125+0000 7f05cf400700 1 --2- 172.21.15.43:0/3337502075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b197968410 0x55b19797e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.125+0000 7f05cf400700 1 -- 172.21.15.43:0/3337502075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1977e9e40 msgr2=0x55b19796dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.125+0000 7f05cf400700 1 --2- 172.21.15.43:0/3337502075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1977e9e40 0x55b19796dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.125+0000 7f05cf400700 1 -- 172.21.15.43:0/3337502075 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b197973350 con 0x55b197849fc0 2024-05-27T07:17:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.126+0000 7f05cf400700 1 --2- 172.21.15.43:0/3337502075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b197974a00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f05c00a91f0 tx=0x7f05c00e4a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd656140e0328c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.126+0000 7f05bbe00700 1 -- 172.21.15.43:0/3337502075 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c00e2040 con 0x55b197849fc0 2024-05-27T07:17:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.126+0000 7f05bbe00700 1 -- 172.21.15.43:0/3337502075 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05c00b23a0 con 0x55b197849fc0 2024-05-27T07:17:55.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.126+0000 7f05bbe00700 1 -- 172.21.15.43:0/3337502075 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c0020040 con 0x55b197849fc0 2024-05-27T07:17:55.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.127+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3337502075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 msgr2=0x55b197974a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:55.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.127+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3337502075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b197974a00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f05c00a91f0 tx=0x7f05c00e4a60 comp rx=0 tx=0).stop 2024-05-27T07:17:55.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.127+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3337502075 shutdown_connections 2024-05-27T07:17:55.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.127+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3337502075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b197968410 0x55b19797e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.127+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3337502075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b197974a00 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.127+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/3337502075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b1977e9e40 0x55b19796dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.127+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3337502075 >> 172.21.15.43:0/3337502075 conn(0x55b19795cf60 msgr2=0x55b197970e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.127+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3337502075 shutdown_connections 2024-05-27T07:17:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.127+0000 7f05e51c9b80 1 -- 172.21.15.43:0/3337502075 wait complete. 2024-05-27T07:17:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.128+0000 7f05e51c9b80 1 -- start start 2024-05-27T07:17:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.128+0000 7f05e51c9b80 1 Processor -- start 2024-05-27T07:17:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b19797aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b1979763a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05e51c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b1979768e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b2a90 con 0x55b197849fc0 2024-05-27T07:17:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b23c0 con 0x55b197968410 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05e51c9b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01b31c0 con 0x55b1977e9e40 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05e51c9b80 10 client.0 _ll_register_callbacks cb 0x55b197830c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05cea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b1979763a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05cea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b1979763a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47106/0 (socket says 172.21.15.43:47106) 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05cea00700 1 -- 172.21.15.43:0/275935419 learned_addr learned my addr 172.21.15.43:0/275935419 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05cfe00700 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b1979768e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.129+0000 7f05cf400700 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b19797aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.130+0000 7f05cea00700 1 -- 172.21.15.43:0/275935419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 msgr2=0x55b19797aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.130+0000 7f05cea00700 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b19797aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.130+0000 7f05cea00700 1 -- 172.21.15.43:0/275935419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 msgr2=0x55b1979768e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.130+0000 7f05cea00700 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b1979768e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.130+0000 7f05cea00700 1 -- 172.21.15.43:0/275935419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b197976e20 con 0x55b197849fc0 2024-05-27T07:17:55.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.130+0000 7f05cea00700 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b1979763a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x55b197968cb0 tx=0x7f05bc009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7870168aabf168f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:55.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.130+0000 7f05b9600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.130+0000 7f05b9600700 1 -- 172.21.15.43:0/275935419 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc02f020 con 0x55b197849fc0 2024-05-27T07:17:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.131+0000 7f05b9600700 1 -- 172.21.15.43:0/275935419 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9408+0+0 (secure 0 0 0) 0x7f05bc0039d0 con 0x55b197849fc0 2024-05-27T07:17:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.131+0000 7f05e51c9b80 1 -- 172.21.15.43:0/275935419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05c00b23a0 con 0x55b197849fc0 2024-05-27T07:17:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.131+0000 7f05b9600700 1 -- 172.21.15.43:0/275935419 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f05bc03a050 con 0x55b197849fc0 2024-05-27T07:17:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.131+0000 7f05e51c9b80 10 client.16044 fetch_fsmap learned FSMap version 57 2024-05-27T07:17:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.131+0000 7f05e51c9b80 1 -- 172.21.15.43:0/275935419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55b1979ecff0 con 0x55b197849fc0 2024-05-27T07:17:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.131+0000 7f05b9600700 1 -- 172.21.15.43:0/275935419 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f05c00b23a0 con 0x55b197849fc0 2024-05-27T07:17:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.132+0000 7f05b9600700 1 -- 172.21.15.43:0/275935419 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 57) ==== 7773+0+0 (secure 0 0 0) 0x7f05bc00f460 con 0x55b197849fc0 2024-05-27T07:17:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.132+0000 7f05e51c9b80 10 client.16044 fetch_fsmap finished waiting for FSMap version 57 2024-05-27T07:17:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.132+0000 7f05e51c9b80 10 client.16044 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:17:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.132+0000 7f05e51c9b80 20 client.16044 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.132+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b197968820 0x55b1979ef1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.132+0000 7f05e51c9b80 4 client.16044 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:17:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.132+0000 7f05e51c9b80 1 -- 172.21.15.43:0/275935419 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b197892570 con 0x55b197968820 2024-05-27T07:17:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.132+0000 7f05cf400700 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b197968820 0x55b1979ef1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:55.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.133+0000 7f05cf400700 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b197968820 0x55b1979ef1d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=379d09766f3d1c19 server_cookie=b34e0f204650c602 in_seq=0 out_seq=0 2024-05-27T07:17:55.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.133+0000 7f05b9600700 10 client.16044 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:17:55.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05b9600700 1 -- 172.21.15.43:0/275935419 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b197892570 con 0x55b197968820 2024-05-27T07:17:55.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05b9600700 10 client.16044 handle_command_reply: tid=0 2024-05-27T07:17:55.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05e51c9b80 1 client.16044 shutdown 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05e51c9b80 1 -- 172.21.15.43:0/275935419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 msgr2=0x55b1979763a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b1979763a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x55b197968cb0 tx=0x7f05bc009d50 comp rx=0 tx=0).stop 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05e51c9b80 1 -- 172.21.15.43:0/275935419 shutdown_connections 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b197968820 0x55b1979ef1d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b197968410 0x55b1979768e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b1977e9e40 0x55b19797aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05e51c9b80 1 --2- 172.21.15.43:0/275935419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b197849fc0 0x55b1979763a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05b9600700 0 client.16044 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.134+0000 7f05e51c9b80 1 -- 172.21.15.43:0/275935419 >> 172.21.15.43:0/275935419 conn(0x55b19795cf60 msgr2=0x55b197972340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.135+0000 7f05e51c9b80 1 -- 172.21.15.43:0/275935419 shutdown_connections 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.135+0000 7f05e51c9b80 1 -- 172.21.15.43:0/275935419 wait complete. 2024-05-27T07:17:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.135+0000 7f05e51c9b80 20 client.16044 trim_cache size 0 max 16384 2024-05-27T07:17:55.128 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:17:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.135+0000 7f05d5000700 1 -- 172.21.15.43:0/4156103445 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f05b40b29f0 msgr2=0x7f05b40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.135+0000 7f05d5000700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f05b40b29f0 0x7f05b40b4f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55b197964690 tx=0x7f05c8009680 comp rx=0 tx=0).stop 2024-05-27T07:17:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.135+0000 7f05d5000700 1 -- 172.21.15.43:0/4156103445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05d0105280 msgr2=0x7f05d019cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.135+0000 7f05d5000700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05d0105280 0x7f05d019cd50 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f05c000d950 tx=0x7f05c000dd10 comp rx=0 tx=0).stop 2024-05-27T07:17:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.136+0000 7f05d5000700 1 -- 172.21.15.43:0/4156103445 shutdown_connections 2024-05-27T07:17:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.136+0000 7f05d5000700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f05b40b29f0 0x7f05b40b4f00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.136+0000 7f05d5000700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05d010e6f0 0x7f05d01a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.136+0000 7f05d5000700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05d0105280 0x7f05d019cd50 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.136+0000 7f05d5000700 1 --2- 172.21.15.43:0/4156103445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05d0105bd0 0x7f05d019d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.136+0000 7f05d5000700 1 -- 172.21.15.43:0/4156103445 >> 172.21.15.43:0/4156103445 conn(0x7f05d0078af0 msgr2=0x7f05d0102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.136+0000 7f05d5000700 1 -- 172.21.15.43:0/4156103445 shutdown_connections 2024-05-27T07:17:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:55.136+0000 7f05d5000700 1 -- 172.21.15.43:0/4156103445 wait complete. 2024-05-27T07:17:55.185 INFO:tasks.workunit.client.0.smithi043.stdout: 1 14721 0.82 MB/sec warmup 119 sec latency 49.586 ms 2024-05-27T07:17:56.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:17:56.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff788e00700 time 2024-05-27T07:17:56.073609+0000 2024-05-27T07:17:56.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:56.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:56.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff79e8d9ba6] 2024-05-27T07:17:56.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff79e8d9dd1] 2024-05-27T07:17:56.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dfbd2223d] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555dfbd0d971] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 5: (SnapRealm::check_cache() const+0x13e) [0x555dfbd0bfa0] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dfbd0c651] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dfbca1347] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dfb99d87e] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555dfb9a2e94] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555dfb9ad969] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 11: (MDLog::_replay_thread()+0x1f38) [0x555dfbd4648c] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555dfbd4f9cb] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 13: (Thread::entry_wrapper()+0x43) [0x7ff79e8a872d] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff79e8a8749] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff79d44c1ca] 2024-05-27T07:17:56.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 16: clone() 2024-05-27T07:17:56.407 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.076+0000 7ff788e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff788e00700 time 2024-05-27T07:17:56.073609+0000 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff79e8d9ba6] 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff79e8d9dd1] 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dfbd2223d] 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555dfbd0d971] 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 5: (SnapRealm::check_cache() const+0x13e) [0x555dfbd0bfa0] 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dfbd0c651] 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dfbca1347] 2024-05-27T07:17:56.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dfb99d87e] 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555dfb9a2e94] 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555dfb9ad969] 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 11: (MDLog::_replay_thread()+0x1f38) [0x555dfbd4648c] 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555dfbd4f9cb] 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 13: (Thread::entry_wrapper()+0x43) [0x7ff79e8a872d] 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff79e8a8749] 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff79d44c1ca] 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 16: clone() 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: *** Caught signal (Aborted) ** 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: in thread 7ff788e00700 thread_name:md_log_replay 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff79d456d20] 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2: gsignal() 2024-05-27T07:17:56.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 3: abort() 2024-05-27T07:17:56.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff79e8d9ced] 2024-05-27T07:17:56.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff79e8d9dd1] 2024-05-27T07:17:56.411 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dfbd2223d] 2024-05-27T07:17:56.411 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555dfbd0d971] 2024-05-27T07:17:56.411 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 8: (SnapRealm::check_cache() const+0x13e) [0x555dfbd0bfa0] 2024-05-27T07:17:56.411 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dfbd0c651] 2024-05-27T07:17:56.411 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dfbca1347] 2024-05-27T07:17:56.411 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dfb99d87e] 2024-05-27T07:17:56.411 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555dfb9a2e94] 2024-05-27T07:17:56.411 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555dfb9ad969] 2024-05-27T07:17:56.411 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 14: (MDLog::_replay_thread()+0x1f38) [0x555dfbd4648c] 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555dfbd4f9cb] 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 16: (Thread::entry_wrapper()+0x43) [0x7ff79e8a872d] 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff79e8a8749] 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff79d44c1ca] 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 19: clone() 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.081+0000 7ff788e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: in thread 7ff788e00700 thread_name:md_log_replay 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff79d456d20] 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2: gsignal() 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 3: abort() 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff79e8d9ced] 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff79e8d9dd1] 2024-05-27T07:17:56.412 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dfbd2223d] 2024-05-27T07:17:56.413 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555dfbd0d971] 2024-05-27T07:17:56.413 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 8: (SnapRealm::check_cache() const+0x13e) [0x555dfbd0bfa0] 2024-05-27T07:17:56.413 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dfbd0c651] 2024-05-27T07:17:56.413 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dfbca1347] 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dfb99d87e] 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555dfb9a2e94] 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555dfb9ad969] 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 14: (MDLog::_replay_thread()+0x1f38) [0x555dfbd4648c] 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555dfbd4f9cb] 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 16: (Thread::entry_wrapper()+0x43) [0x7ff79e8a872d] 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff79e8a8749] 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff79d44c1ca] 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 19: clone() 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:56.414 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: -1> 2024-05-27T07:17:56.076+0000 7ff788e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff788e00700 time 2024-05-27T07:17:56.073609+0000 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff79e8d9ba6] 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff79e8d9dd1] 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dfbd2223d] 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555dfbd0d971] 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 5: (SnapRealm::check_cache() const+0x13e) [0x555dfbd0bfa0] 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dfbd0c651] 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dfbca1347] 2024-05-27T07:17:56.415 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dfb99d87e] 2024-05-27T07:17:56.416 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555dfb9a2e94] 2024-05-27T07:17:56.416 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555dfb9ad969] 2024-05-27T07:17:56.416 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 11: (MDLog::_replay_thread()+0x1f38) [0x555dfbd4648c] 2024-05-27T07:17:56.416 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555dfbd4f9cb] 2024-05-27T07:17:56.416 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 13: (Thread::entry_wrapper()+0x43) [0x7ff79e8a872d] 2024-05-27T07:17:56.416 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff79e8a8749] 2024-05-27T07:17:56.416 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff79d44c1ca] 2024-05-27T07:17:56.416 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 16: clone() 2024-05-27T07:17:56.416 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 0> 2024-05-27T07:17:56.081+0000 7ff788e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: in thread 7ff788e00700 thread_name:md_log_replay 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff79d456d20] 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2: gsignal() 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 3: abort() 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff79e8d9ced] 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff79e8d9dd1] 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dfbd2223d] 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555dfbd0d971] 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 8: (SnapRealm::check_cache() const+0x13e) [0x555dfbd0bfa0] 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dfbd0c651] 2024-05-27T07:17:56.417 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dfbca1347] 2024-05-27T07:17:56.418 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dfb99d87e] 2024-05-27T07:17:56.418 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555dfb9a2e94] 2024-05-27T07:17:56.418 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555dfb9ad969] 2024-05-27T07:17:56.418 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 14: (MDLog::_replay_thread()+0x1f38) [0x555dfbd4648c] 2024-05-27T07:17:56.418 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555dfbd4f9cb] 2024-05-27T07:17:56.418 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 16: (Thread::entry_wrapper()+0x43) [0x7ff79e8a872d] 2024-05-27T07:17:56.418 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff79e8a8749] 2024-05-27T07:17:56.418 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff79d44c1ca] 2024-05-27T07:17:56.418 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 19: clone() 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: -1> 2024-05-27T07:17:56.076+0000 7ff788e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff788e00700 time 2024-05-27T07:17:56.073609+0000 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff79e8d9ba6] 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff79e8d9dd1] 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dfbd2223d] 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 4: (SnapRealm::build_snap_set() const+0x323) [0x555dfbd0d971] 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 5: (SnapRealm::check_cache() const+0x13e) [0x555dfbd0bfa0] 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dfbd0c651] 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dfbca1347] 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dfb99d87e] 2024-05-27T07:17:56.419 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555dfb9a2e94] 2024-05-27T07:17:56.420 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555dfb9ad969] 2024-05-27T07:17:56.420 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 11: (MDLog::_replay_thread()+0x1f38) [0x555dfbd4648c] 2024-05-27T07:17:56.420 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 12: (MDLog::ReplayThread::entry()+0x11) [0x555dfbd4f9cb] 2024-05-27T07:17:56.420 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 13: (Thread::entry_wrapper()+0x43) [0x7ff79e8a872d] 2024-05-27T07:17:56.420 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff79e8a8749] 2024-05-27T07:17:56.420 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff79d44c1ca] 2024-05-27T07:17:56.420 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 16: clone() 2024-05-27T07:17:56.421 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.421 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 0> 2024-05-27T07:17:56.081+0000 7ff788e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:17:56.421 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: in thread 7ff788e00700 thread_name:md_log_replay 2024-05-27T07:17:56.421 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.421 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:17:56.421 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff79d456d20] 2024-05-27T07:17:56.421 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2: gsignal() 2024-05-27T07:17:56.421 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 3: abort() 2024-05-27T07:17:56.421 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff79e8d9ced] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff79e8d9dd1] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x555dfbd2223d] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 7: (SnapRealm::build_snap_set() const+0x323) [0x555dfbd0d971] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 8: (SnapRealm::check_cache() const+0x13e) [0x555dfbd0bfa0] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x555dfbd0c651] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555dfbca1347] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x555dfb99d87e] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555dfb9a2e94] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555dfb9ad969] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 14: (MDLog::_replay_thread()+0x1f38) [0x555dfbd4648c] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 15: (MDLog::ReplayThread::entry()+0x11) [0x555dfbd4f9cb] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 16: (Thread::entry_wrapper()+0x43) [0x7ff79e8a872d] 2024-05-27T07:17:56.422 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff79e8a8749] 2024-05-27T07:17:56.423 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff79d44c1ca] 2024-05-27T07:17:56.423 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 19: clone() 2024-05-27T07:17:56.423 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:17:56.423 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[68985]: 2024-05-27T07:17:56.423 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 podman[98928]: 2024-05-27 07:17:56.221591857 +0000 UTC m=+0.043918416 container died 6726de02f1655b0d4317dda56f7105b6e72f405b0d7bc928610f0e5deae3c591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-05-27T07:17:56.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:56 smithi043 ceph-mon[31531]: pgmap v135: 129 pgs: 129 active+clean; 37 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 458 KiB/s wr, 44 op/s 2024-05-27T07:17:56.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:56 smithi043 ceph-mon[31531]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:17:56.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:56 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:17:56.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:56 smithi179 ceph-mon[35888]: pgmap v135: 129 pgs: 129 active+clean; 37 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 458 KiB/s wr, 44 op/s 2024-05-27T07:17:56.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:56 smithi179 ceph-mon[35888]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:17:56.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:56 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:17:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:56 smithi132 ceph-mon[35283]: pgmap v135: 129 pgs: 129 active+clean; 37 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 458 KiB/s wr, 44 op/s 2024-05-27T07:17:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:56 smithi132 ceph-mon[35283]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-05-27T07:17:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:56 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:17:57.064 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:17:57.186 INFO:tasks.workunit.client.0.smithi043.stdout: 1 14918 0.23 MB/sec execute 1 sec latency 49.341 ms 2024-05-27T07:17:57.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 podman[98928]: 2024-05-27 07:17:56.964819768 +0000 UTC m=+0.787146354 container remove 6726de02f1655b0d4317dda56f7105b6e72f405b0d7bc928610f0e5deae3c591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:17:57.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:56 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:17:57.953 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:57 smithi043 podman[98987]: 2024-05-27 07:17:57.69494773 +0000 UTC m=+0.679453093 container died fac880fb235ec4c74b03ece5c3f210c84df8ee11339fcf25bd23a1e27a040124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=reverent_moore, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T07:17:58.186 INFO:tasks.workunit.client.0.smithi043.stdout: 1 15060 0.30 MB/sec execute 2 sec latency 48.411 ms 2024-05-27T07:17:58.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:17:58 smithi043 ceph-mon[31531]: pgmap v136: 129 pgs: 129 active+clean; 38 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 358 KiB/s wr, 45 op/s 2024-05-27T07:17:58.702 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:58 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:17:58.702 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:17:58 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.919s CPU time. 2024-05-27T07:17:58.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:17:58 smithi179 ceph-mon[35888]: pgmap v136: 129 pgs: 129 active+clean; 38 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 358 KiB/s wr, 45 op/s 2024-05-27T07:17:58.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:17:58 smithi132 ceph-mon[35283]: pgmap v136: 129 pgs: 129 active+clean; 38 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 358 KiB/s wr, 45 op/s 2024-05-27T07:17:59.186 INFO:tasks.workunit.client.0.smithi043.stdout: 1 15154 0.28 MB/sec execute 3 sec latency 60.049 ms 2024-05-27T07:17:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.359+0000 7ff728e00700 1 -- 172.21.15.43:0/1589773486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff72410c5b0 msgr2=0x7ff72410c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.359+0000 7ff728e00700 1 --2- 172.21.15.43:0/1589773486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff72410c5b0 0x7ff72410c9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff714009990 tx=0x7ff714009ca0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.359+0000 7ff728e00700 1 -- 172.21.15.43:0/1589773486 shutdown_connections 2024-05-27T07:17:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.359+0000 7ff728e00700 1 --2- 172.21.15.43:0/1589773486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724106ec0 0x7ff724113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.359+0000 7ff728e00700 1 --2- 172.21.15.43:0/1589773486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff72410c5b0 0x7ff72410c9c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.359+0000 7ff728e00700 1 --2- 172.21.15.43:0/1589773486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff724106570 0x7ff724106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.359+0000 7ff728e00700 1 -- 172.21.15.43:0/1589773486 >> 172.21.15.43:0/1589773486 conn(0x7ff724102070 msgr2=0x7ff7241044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.359+0000 7ff728e00700 1 -- 172.21.15.43:0/1589773486 shutdown_connections 2024-05-27T07:17:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.359+0000 7ff728e00700 1 -- 172.21.15.43:0/1589773486 wait complete. 2024-05-27T07:17:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.361+0000 7ff728e00700 1 Processor -- start 2024-05-27T07:17:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.361+0000 7ff728e00700 1 -- start start 2024-05-27T07:17:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.361+0000 7ff728e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff724106570 0x7ff72419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.361+0000 7ff728e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724106ec0 0x7ff72419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.361+0000 7ff728e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff72410c5b0 0x7ff7241a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.361+0000 7ff728e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff7241165c0 con 0x7ff724106570 2024-05-27T07:17:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.361+0000 7ff728e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff724116440 con 0x7ff724106ec0 2024-05-27T07:17:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.361+0000 7ff728e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff724116740 con 0x7ff72410c5b0 2024-05-27T07:17:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.362+0000 7ff721e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff724106570 0x7ff72419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.362+0000 7ff721e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff724106570 0x7ff72419d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47112/0 (socket says 172.21.15.43:47112) 2024-05-27T07:17:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.362+0000 7ff721e00700 1 -- 172.21.15.43:0/704642074 learned_addr learned my addr 172.21.15.43:0/704642074 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.362+0000 7ff721400700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724106ec0 0x7ff72419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.362+0000 7ff722800700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff72410c5b0 0x7ff7241a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.362+0000 7ff721e00700 1 -- 172.21.15.43:0/704642074 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff72410c5b0 msgr2=0x7ff7241a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.362+0000 7ff721e00700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff72410c5b0 0x7ff7241a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.362+0000 7ff721e00700 1 -- 172.21.15.43:0/704642074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724106ec0 msgr2=0x7ff72419d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.362+0000 7ff721e00700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724106ec0 0x7ff72419d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.362+0000 7ff721e00700 1 -- 172.21.15.43:0/704642074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7241a1ee0 con 0x7ff724106570 2024-05-27T07:17:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.363+0000 7ff721e00700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff724106570 0x7ff72419d010 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff714000c00 tx=0x7ff714005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.363+0000 7ff71aa00700 1 -- 172.21.15.43:0/704642074 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714027070 con 0x7ff724106570 2024-05-27T07:17:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.363+0000 7ff71aa00700 1 -- 172.21.15.43:0/704642074 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff71400fb80 con 0x7ff724106570 2024-05-27T07:17:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.363+0000 7ff728e00700 1 -- 172.21.15.43:0/704642074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7241a2170 con 0x7ff724106570 2024-05-27T07:17:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.363+0000 7ff71aa00700 1 -- 172.21.15.43:0/704642074 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714022c00 con 0x7ff724106570 2024-05-27T07:17:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.363+0000 7ff728e00700 1 -- 172.21.15.43:0/704642074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7241a2650 con 0x7ff724106570 2024-05-27T07:17:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.368+0000 7ff71aa00700 1 -- 172.21.15.43:0/704642074 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff71402a070 con 0x7ff724106570 2024-05-27T07:17:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.369+0000 7ff7378c7b80 1 Processor -- start 2024-05-27T07:17:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.369+0000 7ff7378c7b80 1 -- start start 2024-05-27T07:17:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd58ce40 0x5615cd7117f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd70b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff71aa00700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff7080b29f0 0x7ff7080b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd70bdb0 0x5615cd70c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5615cd639e40 con 0x5615cd711d30 2024-05-27T07:17:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5615cd6299e0 con 0x5615cd58ce40 2024-05-27T07:17:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5615cd62da20 con 0x5615cd70bdb0 2024-05-27T07:17:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff721400700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff7080b29f0 0x7ff7080b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff71aa00700 1 -- 172.21.15.43:0/704642074 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9408+0+0 (secure 0 0 0) 0x7ff7140e39a0 con 0x7ff724106570 2024-05-27T07:17:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff721e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd70b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff722800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd58ce40 0x5615cd7117f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff721e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd70b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47126/0 (socket says 172.21.15.43:47126) 2024-05-27T07:17:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff721e00700 1 -- 172.21.15.43:0/1033815067 learned_addr learned my addr 172.21.15.43:0/1033815067 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.370+0000 7ff722800700 1 --2- 172.21.15.43:0/1033815067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd70bdb0 0x5615cd70c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.371+0000 7ff721e00700 1 -- 172.21.15.43:0/1033815067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd70bdb0 msgr2=0x5615cd70c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.371+0000 7ff721e00700 1 --2- 172.21.15.43:0/1033815067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd70bdb0 0x5615cd70c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.371+0000 7ff721e00700 1 -- 172.21.15.43:0/1033815067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd58ce40 msgr2=0x5615cd7117f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.371+0000 7ff721e00700 1 --2- 172.21.15.43:0/1033815067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd58ce40 0x5615cd7117f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.371+0000 7ff721e00700 1 -- 172.21.15.43:0/1033815067 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615cd5f2420 con 0x5615cd711d30 2024-05-27T07:17:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.371+0000 7ff721400700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff7080b29f0 0x7ff7080b4f00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff710005950 tx=0x7ff7100058e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.371+0000 7ff721e00700 1 --2- 172.21.15.43:0/1033815067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd70b870 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff7140a69e0 tx=0x7ff7140a6950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b451cbf37b05550a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.372+0000 7ff70f400700 1 -- 172.21.15.43:0/1033815067 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7140aa470 con 0x5615cd711d30 2024-05-27T07:17:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.372+0000 7ff70f400700 1 -- 172.21.15.43:0/1033815067 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7140ab060 con 0x5615cd711d30 2024-05-27T07:17:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.372+0000 7ff70f400700 1 -- 172.21.15.43:0/1033815067 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7140aa960 con 0x5615cd711d30 2024-05-27T07:17:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.373+0000 7ff7378c7b80 1 -- 172.21.15.43:0/1033815067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 msgr2=0x5615cd70b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.373+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/1033815067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd70b870 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff7140a69e0 tx=0x7ff7140a6950 comp rx=0 tx=0).stop 2024-05-27T07:17:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.373+0000 7ff7378c7b80 1 -- 172.21.15.43:0/1033815067 shutdown_connections 2024-05-27T07:17:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.373+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/1033815067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd70bdb0 0x5615cd70c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.373+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/1033815067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd58ce40 0x5615cd7117f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.373+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/1033815067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd70b870 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.373+0000 7ff7378c7b80 1 -- 172.21.15.43:0/1033815067 >> 172.21.15.43:0/1033815067 conn(0x5615cd5ecfc0 msgr2=0x5615cd7068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.373+0000 7ff7378c7b80 1 -- 172.21.15.43:0/1033815067 shutdown_connections 2024-05-27T07:17:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.373+0000 7ff7378c7b80 1 -- 172.21.15.43:0/1033815067 wait complete. 2024-05-27T07:17:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.374+0000 7ff7378c7b80 1 -- start start 2024-05-27T07:17:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.374+0000 7ff7378c7b80 1 Processor -- start 2024-05-27T07:17:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.375+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd58ce40 0x5615cd715580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.375+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 0x5615cd715ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.375+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd716000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.375+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5615cd52ffa0 con 0x5615cd711d30 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.375+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5615cd5ca760 con 0x5615cd70bdb0 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.375+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5615cd7185c0 con 0x5615cd58ce40 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.375+0000 7ff7378c7b80 10 client.0 _ll_register_callbacks cb 0x5615cd5d3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.375+0000 7ff722800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd716000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.375+0000 7ff722800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd716000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47142/0 (socket says 172.21.15.43:47142) 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.375+0000 7ff722800700 1 -- 172.21.15.43:0/551866649 learned_addr learned my addr 172.21.15.43:0/551866649 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.376+0000 7ff721400700 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 0x5615cd715ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.376+0000 7ff721e00700 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd58ce40 0x5615cd715580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.376+0000 7ff722800700 1 -- 172.21.15.43:0/551866649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd58ce40 msgr2=0x5615cd715580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.376+0000 7ff722800700 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd58ce40 0x5615cd715580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.376+0000 7ff722800700 1 -- 172.21.15.43:0/551866649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 msgr2=0x5615cd715ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.376+0000 7ff722800700 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 0x5615cd715ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.376+0000 7ff722800700 1 -- 172.21.15.43:0/551866649 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615cd5798d0 con 0x5615cd711d30 2024-05-27T07:17:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.376+0000 7ff721400700 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 0x5615cd715ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:17:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.376+0000 7ff722800700 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd716000 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00aad0 tx=0x7ff71c00ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa95644c41849e22 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.376+0000 7ff703e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.377+0000 7ff703e00700 1 -- 172.21.15.43:0/551866649 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c01d020 con 0x5615cd711d30 2024-05-27T07:17:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.377+0000 7ff703e00700 1 -- 172.21.15.43:0/551866649 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9408+0+0 (secure 0 0 0) 0x7ff71c008d90 con 0x5615cd711d30 2024-05-27T07:17:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.377+0000 7ff7378c7b80 1 -- 172.21.15.43:0/551866649 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7140ab060 con 0x5615cd711d30 2024-05-27T07:17:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.377+0000 7ff703e00700 1 -- 172.21.15.43:0/551866649 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7ff71c004bb0 con 0x5615cd711d30 2024-05-27T07:17:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.377+0000 7ff7378c7b80 10 client.16056 fetch_fsmap learned FSMap version 58 2024-05-27T07:17:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.377+0000 7ff7378c7b80 1 -- 172.21.15.43:0/551866649 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5615cd5fa060 con 0x5615cd711d30 2024-05-27T07:17:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.378+0000 7ff703e00700 1 -- 172.21.15.43:0/551866649 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7140ab060 con 0x5615cd711d30 2024-05-27T07:17:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.378+0000 7ff703e00700 1 -- 172.21.15.43:0/551866649 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 58) ==== 7168+0+0 (secure 0 0 0) 0x7ff71c0193f0 con 0x5615cd711d30 2024-05-27T07:17:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.378+0000 7ff7378c7b80 10 client.16056 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:17:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.378+0000 7ff7378c7b80 10 client.16056 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:17:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.378+0000 7ff7378c7b80 20 client.16056 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.378+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5615cd725660 0x5615cd727ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.378+0000 7ff7378c7b80 4 client.16056 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:17:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.378+0000 7ff7378c7b80 1 -- 172.21.15.43:0/551866649 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615cd63bd90 con 0x5615cd725660 2024-05-27T07:17:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.379+0000 7ff721e00700 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5615cd725660 0x5615cd727ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.380+0000 7ff721e00700 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5615cd725660 0x5615cd727ad0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b48cc831bb263ad8 server_cookie=e6360aed333a2dc8 in_seq=0 out_seq=0 2024-05-27T07:17:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.380+0000 7ff703e00700 10 client.16056 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:17:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff703e00700 1 -- 172.21.15.43:0/551866649 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5615cd63bd90 con 0x5615cd725660 2024-05-27T07:17:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff703e00700 10 client.16056 handle_command_reply: tid=0 2024-05-27T07:17:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 client.16056 shutdown 2024-05-27T07:17:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 -- 172.21.15.43:0/551866649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 msgr2=0x5615cd716000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd716000 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00aad0 tx=0x7ff71c00ae90 comp rx=0 tx=0).stop 2024-05-27T07:17:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 -- 172.21.15.43:0/551866649 shutdown_connections 2024-05-27T07:17:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd711d30 0x5615cd716000 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd58ce40 0x5615cd715580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5615cd725660 0x5615cd727ad0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/551866649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 0x5615cd715ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff703e00700 0 client.16056 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:17:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 -- 172.21.15.43:0/551866649 >> 172.21.15.43:0/551866649 conn(0x5615cd5ecfc0 msgr2=0x5615cd706580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 -- 172.21.15.43:0/551866649 shutdown_connections 2024-05-27T07:17:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.382+0000 7ff7378c7b80 1 -- 172.21.15.43:0/551866649 wait complete. 2024-05-27T07:17:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.383+0000 7ff7378c7b80 20 client.16056 trim_cache size 0 max 16384 2024-05-27T07:17:59.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.402+0000 7ff7378c7b80 1 Processor -- start 2024-05-27T07:17:59.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.402+0000 7ff7378c7b80 1 -- start start 2024-05-27T07:17:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd58ce40 0x5615cd70b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 0x5615cd717a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd711d30 0x5615cd721620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff7241b10f0 con 0x5615cd58ce40 2024-05-27T07:17:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff7241b3a90 con 0x5615cd70bdb0 2024-05-27T07:17:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff7241b17c0 con 0x5615cd711d30 2024-05-27T07:17:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff721e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd58ce40 0x5615cd70b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff721e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd58ce40 0x5615cd70b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47144/0 (socket says 172.21.15.43:47144) 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff721e00700 1 -- 172.21.15.43:0/1224677906 learned_addr learned my addr 172.21.15.43:0/1224677906 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff722800700 1 --2- 172.21.15.43:0/1224677906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd711d30 0x5615cd721620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.403+0000 7ff721400700 1 --2- 172.21.15.43:0/1224677906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 0x5615cd717a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.404+0000 7ff721e00700 1 -- 172.21.15.43:0/1224677906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd711d30 msgr2=0x5615cd721620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.404+0000 7ff721e00700 1 --2- 172.21.15.43:0/1224677906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd711d30 0x5615cd721620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.404+0000 7ff721e00700 1 -- 172.21.15.43:0/1224677906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 msgr2=0x5615cd717a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.404+0000 7ff721e00700 1 --2- 172.21.15.43:0/1224677906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 0x5615cd717a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.404+0000 7ff721e00700 1 -- 172.21.15.43:0/1224677906 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615cd708320 con 0x5615cd58ce40 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.404+0000 7ff721e00700 1 --2- 172.21.15.43:0/1224677906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd58ce40 0x5615cd70b870 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff7140098e0 tx=0x7ff714021fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b146dd2a01ab2f2c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.404+0000 7ff70ea00700 1 -- 172.21.15.43:0/1224677906 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7140aed10 con 0x5615cd58ce40 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.404+0000 7ff70ea00700 1 -- 172.21.15.43:0/1224677906 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7140b3730 con 0x5615cd58ce40 2024-05-27T07:17:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.404+0000 7ff70ea00700 1 -- 172.21.15.43:0/1224677906 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7140b10b0 con 0x5615cd58ce40 2024-05-27T07:17:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.405+0000 7ff7378c7b80 1 -- 172.21.15.43:0/1224677906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd58ce40 msgr2=0x5615cd70b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.405+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/1224677906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd58ce40 0x5615cd70b870 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff7140098e0 tx=0x7ff714021fc0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.405+0000 7ff7378c7b80 1 -- 172.21.15.43:0/1224677906 shutdown_connections 2024-05-27T07:17:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.405+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/1224677906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd711d30 0x5615cd721620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.405+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/1224677906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd58ce40 0x5615cd70b870 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.405+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/1224677906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd70bdb0 0x5615cd717a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.406+0000 7ff7378c7b80 1 -- 172.21.15.43:0/1224677906 >> 172.21.15.43:0/1224677906 conn(0x5615cd5ecfc0 msgr2=0x5615cd70f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:59.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.406+0000 7ff7378c7b80 1 -- 172.21.15.43:0/1224677906 shutdown_connections 2024-05-27T07:17:59.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.406+0000 7ff7378c7b80 1 -- 172.21.15.43:0/1224677906 wait complete. 2024-05-27T07:17:59.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.406+0000 7ff7378c7b80 1 -- start start 2024-05-27T07:17:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.406+0000 7ff7378c7b80 1 Processor -- start 2024-05-27T07:17:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.407+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd58ce40 0x5615cd71d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.407+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd70bdb0 0x5615cd71dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.407+0000 7ff7378c7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd711d30 0x5615cd72c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.407+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff7241b1e90 con 0x5615cd70bdb0 2024-05-27T07:17:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.407+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff7241b33c0 con 0x5615cd58ce40 2024-05-27T07:17:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.407+0000 7ff7378c7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff7241b2560 con 0x5615cd711d30 2024-05-27T07:17:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.407+0000 7ff7378c7b80 10 client.0 _ll_register_callbacks cb 0x5615cd5d3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:17:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.407+0000 7ff721400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd70bdb0 0x5615cd71dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.407+0000 7ff721400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd70bdb0 0x5615cd71dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47152/0 (socket says 172.21.15.43:47152) 2024-05-27T07:17:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.407+0000 7ff721400700 1 -- 172.21.15.43:0/509078120 learned_addr learned my addr 172.21.15.43:0/509078120 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:17:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.408+0000 7ff721e00700 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd58ce40 0x5615cd71d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.408+0000 7ff721400700 1 -- 172.21.15.43:0/509078120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd711d30 msgr2=0x5615cd72c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.408+0000 7ff721400700 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd711d30 0x5615cd72c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.408+0000 7ff721400700 1 -- 172.21.15.43:0/509078120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd58ce40 msgr2=0x5615cd71d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:17:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.408+0000 7ff721400700 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd58ce40 0x5615cd71d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.408+0000 7ff721400700 1 -- 172.21.15.43:0/509078120 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615cd71e310 con 0x5615cd70bdb0 2024-05-27T07:17:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.408+0000 7ff721400700 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd70bdb0 0x5615cd71dd50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff72406d110 tx=0x7ff710004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218b770d903cf81c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:17:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.408+0000 7ff703e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.409+0000 7ff703e00700 1 -- 172.21.15.43:0/509078120 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710029670 con 0x5615cd70bdb0 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.409+0000 7ff703e00700 1 -- 172.21.15.43:0/509078120 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9408+0+0 (secure 0 0 0) 0x7ff710029810 con 0x5615cd70bdb0 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.409+0000 7ff7378c7b80 1 -- 172.21.15.43:0/509078120 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7140b3730 con 0x5615cd70bdb0 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.409+0000 7ff703e00700 1 -- 172.21.15.43:0/509078120 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7ff710032050 con 0x5615cd70bdb0 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.409+0000 7ff7378c7b80 10 client.16062 fetch_fsmap learned FSMap version 58 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.409+0000 7ff7378c7b80 1 -- 172.21.15.43:0/509078120 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5615cd78f640 con 0x5615cd70bdb0 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.409+0000 7ff703e00700 1 -- 172.21.15.43:0/509078120 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7140b3730 con 0x5615cd70bdb0 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.409+0000 7ff703e00700 1 -- 172.21.15.43:0/509078120 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 58) ==== 7168+0+0 (secure 0 0 0) 0x7ff7100209f0 con 0x5615cd70bdb0 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.410+0000 7ff7378c7b80 10 client.16062 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.410+0000 7ff7378c7b80 10 client.16062 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.410+0000 7ff7378c7b80 20 client.16062 populate_metadata read hostname 'smithi043' 2024-05-27T07:17:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.410+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5615cd78fdb0 0x5615cd7921d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:17:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.410+0000 7ff7378c7b80 4 client.16062 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:17:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.410+0000 7ff7378c7b80 1 -- 172.21.15.43:0/509078120 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615cd635570 con 0x5615cd78fdb0 2024-05-27T07:17:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.410+0000 7ff721e00700 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5615cd78fdb0 0x5615cd7921d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:17:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.411+0000 7ff721e00700 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5615cd78fdb0 0x5615cd7921d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4381ec92ff9ee756 server_cookie=a47695e338fd3281 in_seq=0 out_seq=0 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.411+0000 7ff703e00700 10 client.16062 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff703e00700 1 -- 172.21.15.43:0/509078120 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615cd635570 con 0x5615cd78fdb0 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff703e00700 10 client.16062 handle_command_reply: tid=0 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff7378c7b80 1 client.16062 shutdown 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff7378c7b80 1 -- 172.21.15.43:0/509078120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd70bdb0 msgr2=0x5615cd71dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd70bdb0 0x5615cd71dd50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff72406d110 tx=0x7ff710004750 comp rx=0 tx=0).stop 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff7378c7b80 1 -- 172.21.15.43:0/509078120 shutdown_connections 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615cd711d30 0x5615cd72c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5615cd78fdb0 0x5615cd7921d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615cd58ce40 0x5615cd71d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff7378c7b80 1 --2- 172.21.15.43:0/509078120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615cd70bdb0 0x5615cd71dd50 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff703e00700 0 client.16062 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:17:59.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.412+0000 7ff7378c7b80 1 -- 172.21.15.43:0/509078120 >> 172.21.15.43:0/509078120 conn(0x5615cd5ecfc0 msgr2=0x5615cd712430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:59.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.413+0000 7ff7378c7b80 1 -- 172.21.15.43:0/509078120 shutdown_connections 2024-05-27T07:17:59.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.413+0000 7ff7378c7b80 1 -- 172.21.15.43:0/509078120 wait complete. 2024-05-27T07:17:59.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.413+0000 7ff7378c7b80 20 client.16062 trim_cache size 0 max 16384 2024-05-27T07:17:59.406 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:17:59.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 -- 172.21.15.43:0/704642074 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff7080b29f0 msgr2=0x7ff7080b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:59.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff7080b29f0 0x7ff7080b4f00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff710005950 tx=0x7ff7100058e0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 -- 172.21.15.43:0/704642074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff724106570 msgr2=0x7ff72419d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:17:59.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff724106570 0x7ff72419d010 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff714000c00 tx=0x7ff714005ef0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 -- 172.21.15.43:0/704642074 shutdown_connections 2024-05-27T07:17:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff7080b29f0 0x7ff7080b4f00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff72410c5b0 0x7ff7241a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff724106570 0x7ff72419d010 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 --2- 172.21.15.43:0/704642074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff724106ec0 0x7ff72419d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:17:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 -- 172.21.15.43:0/704642074 >> 172.21.15.43:0/704642074 conn(0x7ff724102070 msgr2=0x7ff724104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:17:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 -- 172.21.15.43:0/704642074 shutdown_connections 2024-05-27T07:17:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:17:59.414+0000 7ff728e00700 1 -- 172.21.15.43:0/704642074 wait complete. 2024-05-27T07:18:00.186 INFO:tasks.workunit.client.0.smithi043.stdout: 1 15288 0.49 MB/sec execute 4 sec latency 55.036 ms 2024-05-27T07:18:00.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:18:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:00 smithi043 ceph-mon[31531]: pgmap v137: 129 pgs: 129 active+clean; 38 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 350 KiB/s wr, 44 op/s 2024-05-27T07:18:00.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:00 smithi179 ceph-mon[35888]: pgmap v137: 129 pgs: 129 active+clean; 38 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 350 KiB/s wr, 44 op/s 2024-05-27T07:18:00.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:00 smithi132 ceph-mon[35283]: pgmap v137: 129 pgs: 129 active+clean; 38 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 350 KiB/s wr, 44 op/s 2024-05-27T07:18:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:01.186 INFO:tasks.workunit.client.0.smithi043.stdout: 1 15403 0.42 MB/sec execute 5 sec latency 53.392 ms 2024-05-27T07:18:01.257 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:00 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:18:01.257 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:00 smithi043 systemd[1]: Stopped Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:01.257 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:00 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.655s CPU time. 2024-05-27T07:18:01.257 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:00 smithi043 systemd[1]: Starting Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:18:02.186 INFO:tasks.workunit.client.0.smithi043.stdout: 1 15587 0.84 MB/sec execute 6 sec latency 50.220 ms 2024-05-27T07:18:02.594 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:02 smithi043 podman[99290]: 2024-05-27 07:18:02.385728989 +0000 UTC m=+0.039319648 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:18:02.594 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:02 smithi043 podman[99290]: 2024-05-27 07:18:02.497514468 +0000 UTC m=+0.151105128 container create b6685cfcacb02452c2aeff593013481cc7034dff2f0e5073343d27a21d5716f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-05-27T07:18:02.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:02 smithi043 ceph-mon[31531]: pgmap v138: 129 pgs: 129 active+clean; 36 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 480 KiB/s wr, 64 op/s 2024-05-27T07:18:02.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:02 smithi179 ceph-mon[35888]: pgmap v138: 129 pgs: 129 active+clean; 36 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 480 KiB/s wr, 64 op/s 2024-05-27T07:18:02.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:02 smithi132 ceph-mon[35283]: pgmap v138: 129 pgs: 129 active+clean; 36 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 480 KiB/s wr, 64 op/s 2024-05-27T07:18:03.180 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:02 smithi043 podman[99290]: 2024-05-27 07:18:02.93917788 +0000 UTC m=+0.592768583 container init b6685cfcacb02452c2aeff593013481cc7034dff2f0e5073343d27a21d5716f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-27T07:18:03.180 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:02 smithi043 podman[99290]: 2024-05-27 07:18:02.949726444 +0000 UTC m=+0.603317118 container start b6685cfcacb02452c2aeff593013481cc7034dff2f0e5073343d27a21d5716f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True) 2024-05-27T07:18:03.180 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-mds[99331]: -- 172.21.15.43:0/3174346007 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bf47fbba40 con 0x55bf48003180 2024-05-27T07:18:03.180 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: starting mds.d at 2024-05-27T07:18:03.186 INFO:tasks.workunit.client.0.smithi043.stdout: 1 15724 0.73 MB/sec execute 7 sec latency 70.548 ms 2024-05-27T07:18:03.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.330+0000 7f154b200700 1 -- 172.21.15.43:0/3350293158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f154c10e6f0 msgr2=0x7f154c110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:03.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.330+0000 7f154b200700 1 --2- 172.21.15.43:0/3350293158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f154c10e6f0 0x7f154c110bd0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f154400b0a0 tx=0x7f154400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.331+0000 7f154b200700 1 -- 172.21.15.43:0/3350293158 shutdown_connections 2024-05-27T07:18:03.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.331+0000 7f154b200700 1 --2- 172.21.15.43:0/3350293158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f154c10e6f0 0x7f154c110bd0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.331+0000 7f154b200700 1 --2- 172.21.15.43:0/3350293158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f154c105280 0x7f154c105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.331+0000 7f154b200700 1 --2- 172.21.15.43:0/3350293158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f154c105bd0 0x7f154c10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.331+0000 7f154b200700 1 -- 172.21.15.43:0/3350293158 >> 172.21.15.43:0/3350293158 conn(0x7f154c078ab0 msgr2=0x7f154c078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:03.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.331+0000 7f154b200700 1 -- 172.21.15.43:0/3350293158 shutdown_connections 2024-05-27T07:18:03.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.331+0000 7f154b200700 1 -- 172.21.15.43:0/3350293158 wait complete. 2024-05-27T07:18:03.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.332+0000 7f154b200700 1 Processor -- start 2024-05-27T07:18:03.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.332+0000 7f154b200700 1 -- start start 2024-05-27T07:18:03.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.333+0000 7f154b200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f154c105280 0x7f154c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.333+0000 7f154b200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f154c105bd0 0x7f154c19d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.333+0000 7f154b200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f154c10e6f0 0x7f154c1a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.333+0000 7f154b200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f154c114150 con 0x7f154c105bd0 2024-05-27T07:18:03.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.333+0000 7f154b200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f154c113fd0 con 0x7f154c10e6f0 2024-05-27T07:18:03.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.333+0000 7f154b200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f154c1142d0 con 0x7f154c105280 2024-05-27T07:18:03.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.333+0000 7f1543e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f154c105bd0 0x7f154c19d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.333+0000 7f1543e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f154c105bd0 0x7f154c19d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47178/0 (socket says 172.21.15.43:47178) 2024-05-27T07:18:03.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.333+0000 7f1543e00700 1 -- 172.21.15.43:0/163174557 learned_addr learned my addr 172.21.15.43:0/163174557 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:03.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.334+0000 7f1549e00700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f154c105280 0x7f154c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.334+0000 7f154a800700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f154c10e6f0 0x7f154c1a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.334+0000 7f1543e00700 1 -- 172.21.15.43:0/163174557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f154c105280 msgr2=0x7f154c19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:03.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.334+0000 7f1543e00700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f154c105280 0x7f154c19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.334+0000 7f1543e00700 1 -- 172.21.15.43:0/163174557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f154c10e6f0 msgr2=0x7f154c1a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:03.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.334+0000 7f1543e00700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f154c10e6f0 0x7f154c1a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.334+0000 7f1543e00700 1 -- 172.21.15.43:0/163174557 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154c1a1cf0 con 0x7f154c105bd0 2024-05-27T07:18:03.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.334+0000 7f1543e00700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f154c105bd0 0x7f154c19d3c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f153400e9a0 tx=0x7f153400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:03.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.335+0000 7f1542a00700 1 -- 172.21.15.43:0/163174557 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153400cbd0 con 0x7f154c105bd0 2024-05-27T07:18:03.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.335+0000 7f1542a00700 1 -- 172.21.15.43:0/163174557 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f153400cd70 con 0x7f154c105bd0 2024-05-27T07:18:03.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.335+0000 7f154b200700 1 -- 172.21.15.43:0/163174557 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f154c1a1fe0 con 0x7f154c105bd0 2024-05-27T07:18:03.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.335+0000 7f1542a00700 1 -- 172.21.15.43:0/163174557 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1534018680 con 0x7f154c105bd0 2024-05-27T07:18:03.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.335+0000 7f154b200700 1 -- 172.21.15.43:0/163174557 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f154c1a9900 con 0x7f154c105bd0 2024-05-27T07:18:03.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.340+0000 7f1542a00700 1 -- 172.21.15.43:0/163174557 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1534010950 con 0x7f154c105bd0 2024-05-27T07:18:03.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.340+0000 7f155f6fdb80 1 Processor -- start 2024-05-27T07:18:03.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.341+0000 7f155f6fdb80 1 -- start start 2024-05-27T07:18:03.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.341+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 0x55a292561ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.341+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a29256cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.341+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 0x55a29256f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.341+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a292490e40 con 0x55a292562410 2024-05-27T07:18:03.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.341+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a2924809e0 con 0x55a2923e3e40 2024-05-27T07:18:03.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.341+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a292484a20 con 0x55a292562820 2024-05-27T07:18:03.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1549e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a29256cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1549e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a29256cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47186/0 (socket says 172.21.15.43:47186) 2024-05-27T07:18:03.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1549e00700 1 -- 172.21.15.43:0/556728063 learned_addr learned my addr 172.21.15.43:0/556728063 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:03.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f154a800700 1 --2- 172.21.15.43:0/556728063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 0x55a292561ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1543e00700 1 --2- 172.21.15.43:0/556728063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 0x55a29256f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1549e00700 1 -- 172.21.15.43:0/556728063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 msgr2=0x55a29256f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:03.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1549e00700 1 --2- 172.21.15.43:0/556728063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 0x55a29256f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1542a00700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f15300b2a70 0x7f15300b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1549e00700 1 -- 172.21.15.43:0/556728063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 msgr2=0x55a292561ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:03.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1549e00700 1 --2- 172.21.15.43:0/556728063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 0x55a292561ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1549e00700 1 -- 172.21.15.43:0/556728063 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a292449420 con 0x55a292562410 2024-05-27T07:18:03.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1543e00700 1 --2- 172.21.15.43:0/556728063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 0x55a29256f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:18:03.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.342+0000 7f1542a00700 1 -- 172.21.15.43:0/163174557 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9408+0+0 (secure 0 0 0) 0x7f15340e2bf0 con 0x7f154c105bd0 2024-05-27T07:18:03.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.343+0000 7f154a800700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f15300b2a70 0x7f15300b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.343+0000 7f1549e00700 1 --2- 172.21.15.43:0/556728063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a29256cad0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f153c009d10 tx=0x7f153c0116c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da64e8dac456d0d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:03.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.343+0000 7f153b400700 1 -- 172.21.15.43:0/556728063 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c009e60 con 0x55a292562410 2024-05-27T07:18:03.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.343+0000 7f153b400700 1 -- 172.21.15.43:0/556728063 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f153c011e20 con 0x55a292562410 2024-05-27T07:18:03.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.343+0000 7f153b400700 1 -- 172.21.15.43:0/556728063 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c00f7e0 con 0x55a292562410 2024-05-27T07:18:03.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.345+0000 7f155f6fdb80 1 -- 172.21.15.43:0/556728063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 msgr2=0x55a29256cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:03.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.345+0000 7f154a800700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f15300b2a70 0x7f15300b4f80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55a29255e690 tx=0x7f15440099e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:03.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.345+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/556728063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a29256cad0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f153c009d10 tx=0x7f153c0116c0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.345+0000 7f155f6fdb80 1 -- 172.21.15.43:0/556728063 shutdown_connections 2024-05-27T07:18:03.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.345+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/556728063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 0x55a29256f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.345+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/556728063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 0x55a292561ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.345+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/556728063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a29256cad0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.345+0000 7f155f6fdb80 1 -- 172.21.15.43:0/556728063 >> 172.21.15.43:0/556728063 conn(0x55a292443fc0 msgr2=0x55a29255dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:03.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.345+0000 7f155f6fdb80 1 -- 172.21.15.43:0/556728063 shutdown_connections 2024-05-27T07:18:03.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.345+0000 7f155f6fdb80 1 -- 172.21.15.43:0/556728063 wait complete. 2024-05-27T07:18:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.346+0000 7f155f6fdb80 1 -- start start 2024-05-27T07:18:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.346+0000 7f155f6fdb80 1 Processor -- start 2024-05-27T07:18:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 0x55a29256c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a292562410 0x55a29256ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a29256cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a292386fa0 con 0x55a292562820 2024-05-27T07:18:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a292421760 con 0x55a292562410 2024-05-27T07:18:03.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a29256f990 con 0x55a2923e3e40 2024-05-27T07:18:03.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f155f6fdb80 10 client.0 _ll_register_callbacks cb 0x55a29242ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:03.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f154a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a29256cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f154a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a29256cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47200/0 (socket says 172.21.15.43:47200) 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f154a800700 1 -- 172.21.15.43:0/1278316636 learned_addr learned my addr 172.21.15.43:0/1278316636 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f1543e00700 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a292562410 0x55a29256ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.347+0000 7f1549e00700 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 0x55a29256c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.348+0000 7f154a800700 1 -- 172.21.15.43:0/1278316636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 msgr2=0x55a29256c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.348+0000 7f154a800700 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 0x55a29256c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.348+0000 7f154a800700 1 -- 172.21.15.43:0/1278316636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a292562410 msgr2=0x55a29256ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.348+0000 7f154a800700 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a292562410 0x55a29256ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.348+0000 7f154a800700 1 -- 172.21.15.43:0/1278316636 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2923d08d0 con 0x55a292562820 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.348+0000 7f1543e00700 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a292562410 0x55a29256ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.348+0000 7f1549e00700 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 0x55a29256c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.348+0000 7f154a800700 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a29256cf40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1544015000 tx=0x7f154400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ebd0b7bbf873e08 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.348+0000 7f152fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.349+0000 7f152fe00700 1 -- 172.21.15.43:0/1278316636 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15440079d0 con 0x55a292562820 2024-05-27T07:18:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.349+0000 7f152fe00700 1 -- 172.21.15.43:0/1278316636 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9408+0+0 (secure 0 0 0) 0x7f1544007b70 con 0x55a292562820 2024-05-27T07:18:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.349+0000 7f155f6fdb80 1 -- 172.21.15.43:0/1278316636 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f153c011e20 con 0x55a292562820 2024-05-27T07:18:03.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.349+0000 7f152fe00700 1 -- 172.21.15.43:0/1278316636 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f1544038050 con 0x55a292562820 2024-05-27T07:18:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.349+0000 7f155f6fdb80 10 client.16080 fetch_fsmap learned FSMap version 58 2024-05-27T07:18:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.349+0000 7f155f6fdb80 1 -- 172.21.15.43:0/1278316636 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55a292451060 con 0x55a292562820 2024-05-27T07:18:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.350+0000 7f152fe00700 1 -- 172.21.15.43:0/1278316636 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f153c011e20 con 0x55a292562820 2024-05-27T07:18:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.350+0000 7f152fe00700 1 -- 172.21.15.43:0/1278316636 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 58) ==== 7168+0+0 (secure 0 0 0) 0x7f1544025070 con 0x55a292562820 2024-05-27T07:18:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.350+0000 7f155f6fdb80 10 client.16080 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:18:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.350+0000 7f155f6fdb80 10 client.16080 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:18:03.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.350+0000 7f155f6fdb80 20 client.16080 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:03.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.350+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a29257c820 0x55a29257ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.350+0000 7f155f6fdb80 4 client.16080 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:18:03.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.350+0000 7f155f6fdb80 1 -- 172.21.15.43:0/1278316636 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a292492d90 con 0x55a29257c820 2024-05-27T07:18:03.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.351+0000 7f1549e00700 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a29257c820 0x55a29257ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.352+0000 7f1549e00700 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a29257c820 0x55a29257ecf0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=90f0d12ea0a9a328 server_cookie=8a3b6970dc23a700 in_seq=0 out_seq=0 2024-05-27T07:18:03.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.352+0000 7f152fe00700 10 client.16080 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:18:03.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f152fe00700 1 -- 172.21.15.43:0/1278316636 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a292492d90 con 0x55a29257c820 2024-05-27T07:18:03.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f152fe00700 10 client.16080 handle_command_reply: tid=0 2024-05-27T07:18:03.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f155f6fdb80 1 client.16080 shutdown 2024-05-27T07:18:03.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f155f6fdb80 1 -- 172.21.15.43:0/1278316636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 msgr2=0x55a29256cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:03.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a29256cf40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1544015000 tx=0x7f154400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f155f6fdb80 1 -- 172.21.15.43:0/1278316636 shutdown_connections 2024-05-27T07:18:03.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a29257c820 0x55a29257ecf0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a29256cf40 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 0x55a29256c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/1278316636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a292562410 0x55a29256ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f155f6fdb80 1 -- 172.21.15.43:0/1278316636 >> 172.21.15.43:0/1278316636 conn(0x55a292443fc0 msgr2=0x55a29255cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:03.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.354+0000 7f152fe00700 0 client.16080 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:18:03.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.355+0000 7f155f6fdb80 1 -- 172.21.15.43:0/1278316636 shutdown_connections 2024-05-27T07:18:03.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.355+0000 7f155f6fdb80 1 -- 172.21.15.43:0/1278316636 wait complete. 2024-05-27T07:18:03.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.355+0000 7f155f6fdb80 20 client.16080 trim_cache size 0 max 16384 2024-05-27T07:18:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.380+0000 7f155f6fdb80 1 Processor -- start 2024-05-27T07:18:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.380+0000 7f155f6fdb80 1 -- start start 2024-05-27T07:18:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.380+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 0x55a29256e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.380+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a292578a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.380+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 0x55a29256c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.380+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f154c1b0f80 con 0x55a292562410 2024-05-27T07:18:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.381+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f154c1b3920 con 0x55a2923e3e40 2024-05-27T07:18:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.381+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f154c1b1650 con 0x55a292562820 2024-05-27T07:18:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.381+0000 7f1543e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a292578a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.381+0000 7f1543e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a292578a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47210/0 (socket says 172.21.15.43:47210) 2024-05-27T07:18:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.381+0000 7f1543e00700 1 -- 172.21.15.43:0/3938537410 learned_addr learned my addr 172.21.15.43:0/3938537410 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.381+0000 7f154a800700 1 --2- 172.21.15.43:0/3938537410 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 0x55a29256c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.381+0000 7f1549e00700 1 --2- 172.21.15.43:0/3938537410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 0x55a29256e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.382+0000 7f1543e00700 1 -- 172.21.15.43:0/3938537410 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 msgr2=0x55a29256c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:03.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.382+0000 7f1543e00700 1 --2- 172.21.15.43:0/3938537410 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 0x55a29256c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.382+0000 7f1543e00700 1 -- 172.21.15.43:0/3938537410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 msgr2=0x55a29256e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:03.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.382+0000 7f1543e00700 1 --2- 172.21.15.43:0/3938537410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 0x55a29256e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.382+0000 7f1543e00700 1 -- 172.21.15.43:0/3938537410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a29256aa30 con 0x55a292562410 2024-05-27T07:18:03.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.382+0000 7f1549e00700 1 --2- 172.21.15.43:0/3938537410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 0x55a29256e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:18:03.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.382+0000 7f1543e00700 1 --2- 172.21.15.43:0/3938537410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a292578a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f15340a9ab0 tx=0x7f153400efa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a044b3497bbf3ca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:03.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.382+0000 7f153aa00700 1 -- 172.21.15.43:0/3938537410 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15340a9630 con 0x55a292562410 2024-05-27T07:18:03.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.383+0000 7f153aa00700 1 -- 172.21.15.43:0/3938537410 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f15340aae50 con 0x55a292562410 2024-05-27T07:18:03.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.383+0000 7f153aa00700 1 -- 172.21.15.43:0/3938537410 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15340a9f90 con 0x55a292562410 2024-05-27T07:18:03.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.384+0000 7f155f6fdb80 1 -- 172.21.15.43:0/3938537410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 msgr2=0x55a292578a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:03.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.384+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/3938537410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a292578a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f15340a9ab0 tx=0x7f153400efa0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.384+0000 7f155f6fdb80 1 -- 172.21.15.43:0/3938537410 shutdown_connections 2024-05-27T07:18:03.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.384+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/3938537410 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a292562820 0x55a29256c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.384+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/3938537410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2923e3e40 0x55a29256e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.384+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/3938537410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562410 0x55a292578a90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.384+0000 7f155f6fdb80 1 -- 172.21.15.43:0/3938537410 >> 172.21.15.43:0/3938537410 conn(0x55a292443fc0 msgr2=0x55a29256c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:03.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.384+0000 7f155f6fdb80 1 -- 172.21.15.43:0/3938537410 shutdown_connections 2024-05-27T07:18:03.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.384+0000 7f155f6fdb80 1 -- 172.21.15.43:0/3938537410 wait complete. 2024-05-27T07:18:03.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.385+0000 7f155f6fdb80 1 -- start start 2024-05-27T07:18:03.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.385+0000 7f155f6fdb80 1 Processor -- start 2024-05-27T07:18:03.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.385+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 0x55a292570060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.385+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a292562410 0x55a2925705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f155f6fdb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a2925e53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f154c1b1d20 con 0x55a292562820 2024-05-27T07:18:03.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f154c1b3250 con 0x55a292562410 2024-05-27T07:18:03.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f155f6fdb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f154c1b23f0 con 0x55a2923e3e40 2024-05-27T07:18:03.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f155f6fdb80 10 client.0 _ll_register_callbacks cb 0x55a29242ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:03.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f154a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a2925e53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f154a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a2925e53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47214/0 (socket says 172.21.15.43:47214) 2024-05-27T07:18:03.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f154a800700 1 -- 172.21.15.43:0/844875295 learned_addr learned my addr 172.21.15.43:0/844875295 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:03.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f1549e00700 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 0x55a292570060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f154a800700 1 -- 172.21.15.43:0/844875295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 msgr2=0x55a292570060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:03.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f154a800700 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 0x55a292570060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f154a800700 1 -- 172.21.15.43:0/844875295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a292562410 msgr2=0x55a2925705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:03.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f154a800700 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a292562410 0x55a2925705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.386+0000 7f154a800700 1 -- 172.21.15.43:0/844875295 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2925e5900 con 0x55a292562820 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.387+0000 7f154a800700 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a2925e53c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1544005a60 tx=0x7f1544027f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ad8f166a644747 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.387+0000 7f152fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.387+0000 7f152fe00700 1 -- 172.21.15.43:0/844875295 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544005e70 con 0x55a292562820 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.387+0000 7f152fe00700 1 -- 172.21.15.43:0/844875295 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9408+0+0 (secure 0 0 0) 0x7f15440077d0 con 0x55a292562820 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.387+0000 7f155f6fdb80 1 -- 172.21.15.43:0/844875295 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15340aae50 con 0x55a292562820 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.387+0000 7f152fe00700 1 -- 172.21.15.43:0/844875295 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f15440094e0 con 0x55a292562820 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.387+0000 7f155f6fdb80 10 client.16086 fetch_fsmap learned FSMap version 58 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.387+0000 7f155f6fdb80 1 -- 172.21.15.43:0/844875295 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55a2925e6a20 con 0x55a292562820 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.388+0000 7f152fe00700 1 -- 172.21.15.43:0/844875295 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f15340aae50 con 0x55a292562820 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.388+0000 7f152fe00700 1 -- 172.21.15.43:0/844875295 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 58) ==== 7168+0+0 (secure 0 0 0) 0x7f1544025040 con 0x55a292562820 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.388+0000 7f155f6fdb80 10 client.16086 fetch_fsmap finished waiting for FSMap version 58 2024-05-27T07:18:03.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.388+0000 7f155f6fdb80 10 client.16086 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:18:03.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.388+0000 7f155f6fdb80 20 client.16086 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:03.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.388+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a2925e70c0 0x55a2925e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:03.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.388+0000 7f155f6fdb80 4 client.16086 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:18:03.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.388+0000 7f155f6fdb80 1 -- 172.21.15.43:0/844875295 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a29248c570 con 0x55a2925e70c0 2024-05-27T07:18:03.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.389+0000 7f1549e00700 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a2925e70c0 0x55a2925e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:03.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.390+0000 7f1549e00700 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a2925e70c0 0x55a2925e94e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60f184499c83856e server_cookie=51f114710d271650 in_seq=0 out_seq=0 2024-05-27T07:18:03.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.390+0000 7f152fe00700 10 client.16086 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:18:03.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.391+0000 7f152fe00700 1 -- 172.21.15.43:0/844875295 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a29248c570 con 0x55a2925e70c0 2024-05-27T07:18:03.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.391+0000 7f152fe00700 10 client.16086 handle_command_reply: tid=0 2024-05-27T07:18:03.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.391+0000 7f155f6fdb80 1 client.16086 shutdown 2024-05-27T07:18:03.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.391+0000 7f155f6fdb80 1 -- 172.21.15.43:0/844875295 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 msgr2=0x55a2925e53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:03.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.391+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a2925e53c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1544005a60 tx=0x7f1544027f60 comp rx=0 tx=0).stop 2024-05-27T07:18:03.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.391+0000 7f155f6fdb80 1 -- 172.21.15.43:0/844875295 shutdown_connections 2024-05-27T07:18:03.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.391+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a2925e70c0 0x55a2925e94e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.392+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a292562820 0x55a2925e53c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.392+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a2923e3e40 0x55a292570060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.392+0000 7f155f6fdb80 1 --2- 172.21.15.43:0/844875295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a292562410 0x55a2925705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.392+0000 7f155f6fdb80 1 -- 172.21.15.43:0/844875295 >> 172.21.15.43:0/844875295 conn(0x55a292443fc0 msgr2=0x55a29256ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:03.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.392+0000 7f152fe00700 0 client.16086 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:18:03.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.392+0000 7f155f6fdb80 1 -- 172.21.15.43:0/844875295 shutdown_connections 2024-05-27T07:18:03.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.392+0000 7f155f6fdb80 1 -- 172.21.15.43:0/844875295 wait complete. 2024-05-27T07:18:03.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.392+0000 7f155f6fdb80 20 client.16086 trim_cache size 0 max 16384 2024-05-27T07:18:03.384 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:18:03.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 -- 172.21.15.43:0/163174557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f15300b2a70 msgr2=0x7f15300b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:03.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f15300b2a70 0x7f15300b4f80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55a29255e690 tx=0x7f15440099e0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 -- 172.21.15.43:0/163174557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f154c105bd0 msgr2=0x7f154c19d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:03.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f154c105bd0 0x7f154c19d3c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f153400e9a0 tx=0x7f153400ed60 comp rx=0 tx=0).stop 2024-05-27T07:18:03.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 -- 172.21.15.43:0/163174557 shutdown_connections 2024-05-27T07:18:03.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f15300b2a70 0x7f15300b4f80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f154c10e6f0 0x7f154c1a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f154c105280 0x7f154c19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 --2- 172.21.15.43:0/163174557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f154c105bd0 0x7f154c19d3c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:03.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 -- 172.21.15.43:0/163174557 >> 172.21.15.43:0/163174557 conn(0x7f154c078ab0 msgr2=0x7f154c104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:03.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 -- 172.21.15.43:0/163174557 shutdown_connections 2024-05-27T07:18:03.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:03.393+0000 7f154b200700 1 -- 172.21.15.43:0/163174557 wait complete. 2024-05-27T07:18:03.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:03 smithi043 ceph-mon[31531]: daemon mds.d restarted 2024-05-27T07:18:03.452 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 bash[99290]: b6685cfcacb02452c2aeff593013481cc7034dff2f0e5073343d27a21d5716f5 2024-05-27T07:18:03.452 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 systemd[1]: Started Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:03.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:03 smithi179 ceph-mon[35888]: daemon mds.d restarted 2024-05-27T07:18:03.837 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb1ef200700 time 2024-05-27T07:18:03.557587+0000 2024-05-27T07:18:03.900 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:03.900 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:03.900 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb204d84ba6] 2024-05-27T07:18:03.900 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb204d84dd1] 2024-05-27T07:18:03.900 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bf1985b23d] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bf19846971] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bf19844fa0] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bf19845651] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bf197da347] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bf194d687e] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bf194dbe94] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bf194e6969] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bf1987f48c] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bf198889cb] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 13: (Thread::entry_wrapper()+0x43) [0x7fb204d5372d] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb204d53749] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb2038f71ca] 2024-05-27T07:18:03.901 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 16: clone() 2024-05-27T07:18:03.902 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: *** Caught signal (Aborted) ** 2024-05-27T07:18:03.902 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: in thread 7fb1ef200700 thread_name:md_log_replay 2024-05-27T07:18:03.902 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:03 smithi132 ceph-mon[35283]: daemon mds.d restarted 2024-05-27T07:18:03.903 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.558+0000 7fb1ef200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb1ef200700 time 2024-05-27T07:18:03.557587+0000 2024-05-27T07:18:03.903 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:03.903 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.903 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb204d84ba6] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb204d84dd1] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bf1985b23d] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bf19846971] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bf19844fa0] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bf19845651] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bf197da347] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bf194d687e] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bf194dbe94] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bf194e6969] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bf1987f48c] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bf198889cb] 2024-05-27T07:18:03.904 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 13: (Thread::entry_wrapper()+0x43) [0x7fb204d5372d] 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb204d53749] 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb2038f71ca] 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 16: clone() 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb203901d20] 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2: gsignal() 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 3: abort() 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb204d84ced] 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb204d84dd1] 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bf1985b23d] 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bf19846971] 2024-05-27T07:18:03.905 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bf19844fa0] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bf19845651] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bf197da347] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bf194d687e] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bf194dbe94] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bf194e6969] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bf1987f48c] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bf198889cb] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 16: (Thread::entry_wrapper()+0x43) [0x7fb204d5372d] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb204d53749] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb2038f71ca] 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 19: clone() 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.561+0000 7fb1ef200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:03.906 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: in thread 7fb1ef200700 thread_name:md_log_replay 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb203901d20] 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2: gsignal() 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 3: abort() 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb204d84ced] 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb204d84dd1] 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bf1985b23d] 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bf19846971] 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bf19844fa0] 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bf19845651] 2024-05-27T07:18:03.907 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bf197da347] 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bf194d687e] 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bf194dbe94] 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bf194e6969] 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bf1987f48c] 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bf198889cb] 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 16: (Thread::entry_wrapper()+0x43) [0x7fb204d5372d] 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb204d53749] 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb2038f71ca] 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 19: clone() 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: -1> 2024-05-27T07:18:03.558+0000 7fb1ef200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb1ef200700 time 2024-05-27T07:18:03.557587+0000 2024-05-27T07:18:03.908 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb204d84ba6] 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb204d84dd1] 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bf1985b23d] 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bf19846971] 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bf19844fa0] 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bf19845651] 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bf197da347] 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bf194d687e] 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bf194dbe94] 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bf194e6969] 2024-05-27T07:18:03.909 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bf1987f48c] 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bf198889cb] 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 13: (Thread::entry_wrapper()+0x43) [0x7fb204d5372d] 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb204d53749] 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb2038f71ca] 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 16: clone() 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 0> 2024-05-27T07:18:03.561+0000 7fb1ef200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: in thread 7fb1ef200700 thread_name:md_log_replay 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb203901d20] 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2: gsignal() 2024-05-27T07:18:03.910 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 3: abort() 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb204d84ced] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb204d84dd1] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bf1985b23d] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bf19846971] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bf19844fa0] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bf19845651] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bf197da347] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bf194d687e] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bf194dbe94] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bf194e6969] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bf1987f48c] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bf198889cb] 2024-05-27T07:18:03.911 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 16: (Thread::entry_wrapper()+0x43) [0x7fb204d5372d] 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb204d53749] 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb2038f71ca] 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 19: clone() 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: -1> 2024-05-27T07:18:03.558+0000 7fb1ef200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb1ef200700 time 2024-05-27T07:18:03.557587+0000 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb204d84ba6] 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb204d84dd1] 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bf1985b23d] 2024-05-27T07:18:03.912 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55bf19846971] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 5: (SnapRealm::check_cache() const+0x13e) [0x55bf19844fa0] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bf19845651] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bf197da347] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bf194d687e] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bf194dbe94] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55bf194e6969] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 11: (MDLog::_replay_thread()+0x1f38) [0x55bf1987f48c] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55bf198889cb] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 13: (Thread::entry_wrapper()+0x43) [0x7fb204d5372d] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb204d53749] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb2038f71ca] 2024-05-27T07:18:03.913 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 16: clone() 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 0> 2024-05-27T07:18:03.561+0000 7fb1ef200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: in thread 7fb1ef200700 thread_name:md_log_replay 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb203901d20] 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2: gsignal() 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 3: abort() 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb204d84ced] 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb204d84dd1] 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55bf1985b23d] 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55bf19846971] 2024-05-27T07:18:03.914 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 8: (SnapRealm::check_cache() const+0x13e) [0x55bf19844fa0] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55bf19845651] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55bf197da347] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55bf194d687e] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55bf194dbe94] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55bf194e6969] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 14: (MDLog::_replay_thread()+0x1f38) [0x55bf1987f48c] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55bf198889cb] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 16: (Thread::entry_wrapper()+0x43) [0x7fb204d5372d] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb204d53749] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb2038f71ca] 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 19: clone() 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:03.915 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[99323]: 2024-05-27T07:18:04.186 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:03 smithi043 podman[99422]: 2024-05-27 07:18:03.879070076 +0000 UTC m=+0.231153434 container died b6685cfcacb02452c2aeff593013481cc7034dff2f0e5073343d27a21d5716f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T07:18:04.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 15854 0.65 MB/sec execute 8 sec latency 58.490 ms 2024-05-27T07:18:04.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:04 smithi043 ceph-mon[31531]: pgmap v139: 129 pgs: 129 active+clean; 36 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 345 KiB/s wr, 47 op/s 2024-05-27T07:18:04.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:04 smithi043 ceph-mon[31531]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:18:04.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:04 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/2492917843,v1:172.21.15.43:6835/2492917843] up:boot 2024-05-27T07:18:04.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:04 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:04.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:04 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:18:04.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:04 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:04.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:04 smithi179 ceph-mon[35888]: pgmap v139: 129 pgs: 129 active+clean; 36 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 345 KiB/s wr, 47 op/s 2024-05-27T07:18:04.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:04 smithi179 ceph-mon[35888]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:18:04.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:04 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/2492917843,v1:172.21.15.43:6835/2492917843] up:boot 2024-05-27T07:18:04.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:04 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:04.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:04 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:18:04.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:04 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:04.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:04 smithi132 ceph-mon[35283]: pgmap v139: 129 pgs: 129 active+clean; 36 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 345 KiB/s wr, 47 op/s 2024-05-27T07:18:04.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:04 smithi132 ceph-mon[35283]: osdmap e104: 12 total, 12 up, 12 in 2024-05-27T07:18:04.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:04 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/2492917843,v1:172.21.15.43:6835/2492917843] up:boot 2024-05-27T07:18:04.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:04 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:04.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:04 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:18:04.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:04 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:05.159 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:04 smithi043 podman[99422]: 2024-05-27 07:18:04.766570338 +0000 UTC m=+1.118653690 container remove b6685cfcacb02452c2aeff593013481cc7034dff2f0e5073343d27a21d5716f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2) 2024-05-27T07:18:05.159 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:04 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:18:05.186 INFO:tasks.workunit.client.0.smithi043.stdout: 1 16021 0.60 MB/sec execute 9 sec latency 53.091 ms 2024-05-27T07:18:05.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:18:05.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:05 smithi043 ceph-mon[31531]: pgmap v141: 129 pgs: 129 active+clean; 36 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 414 KiB/s wr, 57 op/s 2024-05-27T07:18:05.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:05 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:18:05.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:05 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.637s CPU time. 2024-05-27T07:18:05.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:05 smithi179 ceph-mon[35888]: pgmap v141: 129 pgs: 129 active+clean; 36 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 414 KiB/s wr, 57 op/s 2024-05-27T07:18:05.737 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:05.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:05 smithi132 ceph-mon[35283]: pgmap v141: 129 pgs: 129 active+clean; 36 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 414 KiB/s wr, 57 op/s 2024-05-27T07:18:06.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 16183 0.55 MB/sec execute 10 sec latency 45.551 ms 2024-05-27T07:18:07.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 16333 0.52 MB/sec execute 11 sec latency 51.951 ms 2024-05-27T07:18:07.847 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:07 smithi043 ceph-mon[31531]: pgmap v142: 129 pgs: 129 active+clean; 36 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 326 KiB/s wr, 76 op/s 2024-05-27T07:18:07.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:07 smithi179 ceph-mon[35888]: pgmap v142: 129 pgs: 129 active+clean; 36 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 326 KiB/s wr, 76 op/s 2024-05-27T07:18:08.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:07 smithi132 ceph-mon[35283]: pgmap v142: 129 pgs: 129 active+clean; 36 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 326 KiB/s wr, 76 op/s 2024-05-27T07:18:08.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 16495 0.57 MB/sec execute 12 sec latency 52.142 ms 2024-05-27T07:18:08.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.228+0000 7f46dde00700 1 -- 172.21.15.43:0/685615003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d8106eb0 msgr2=0x7f46d8113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.228+0000 7f46dde00700 1 --2- 172.21.15.43:0/685615003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d8106eb0 0x7f46d8113340 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f46d000b0a0 tx=0x7f46d000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.229+0000 7f46dde00700 1 -- 172.21.15.43:0/685615003 shutdown_connections 2024-05-27T07:18:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.229+0000 7f46dde00700 1 --2- 172.21.15.43:0/685615003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d8106eb0 0x7f46d8113340 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.229+0000 7f46dde00700 1 --2- 172.21.15.43:0/685615003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46d810c5a0 0x7f46d810c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.229+0000 7f46dde00700 1 --2- 172.21.15.43:0/685615003 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46d8106560 0x7f46d8106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.229+0000 7f46dde00700 1 -- 172.21.15.43:0/685615003 >> 172.21.15.43:0/685615003 conn(0x7f46d8102040 msgr2=0x7f46d8104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.229+0000 7f46dde00700 1 -- 172.21.15.43:0/685615003 shutdown_connections 2024-05-27T07:18:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.229+0000 7f46dde00700 1 -- 172.21.15.43:0/685615003 wait complete. 2024-05-27T07:18:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.230+0000 7f46dde00700 1 Processor -- start 2024-05-27T07:18:08.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.230+0000 7f46dde00700 1 -- start start 2024-05-27T07:18:08.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.231+0000 7f46dde00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d8106560 0x7f46d819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.231+0000 7f46dde00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46d8106eb0 0x7f46d819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.231+0000 7f46dde00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46d810c5a0 0x7f46d81a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.231+0000 7f46dde00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f46d81165a0 con 0x7f46d8106560 2024-05-27T07:18:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.231+0000 7f46dde00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46d8116420 con 0x7f46d810c5a0 2024-05-27T07:18:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.231+0000 7f46dde00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f46d8116720 con 0x7f46d8106eb0 2024-05-27T07:18:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.231+0000 7f46d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d8106560 0x7f46d819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.231+0000 7f46d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d8106560 0x7f46d819ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37912/0 (socket says 172.21.15.43:37912) 2024-05-27T07:18:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.231+0000 7f46d6a00700 1 -- 172.21.15.43:0/1047882579 learned_addr learned my addr 172.21.15.43:0/1047882579 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.232+0000 7f46d6000700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46d8106eb0 0x7f46d819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.232+0000 7f46d7400700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46d810c5a0 0x7f46d81a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.232+0000 7f46d6a00700 1 -- 172.21.15.43:0/1047882579 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46d8106eb0 msgr2=0x7f46d819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.232+0000 7f46d6a00700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46d8106eb0 0x7f46d819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.232+0000 7f46d6a00700 1 -- 172.21.15.43:0/1047882579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46d810c5a0 msgr2=0x7f46d81a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.232+0000 7f46d6a00700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46d810c5a0 0x7f46d81a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.232+0000 7f46d6a00700 1 -- 172.21.15.43:0/1047882579 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d81a1d80 con 0x7f46d8106560 2024-05-27T07:18:08.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.232+0000 7f46d6000700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46d8106eb0 0x7f46d819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:08.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.232+0000 7f46d6a00700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d8106560 0x7f46d819ceb0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f46c800d980 tx=0x7f46c800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:08.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.233+0000 7f46dca00700 1 -- 172.21.15.43:0/1047882579 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c8017070 con 0x7f46d8106560 2024-05-27T07:18:08.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.233+0000 7f46dca00700 1 -- 172.21.15.43:0/1047882579 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46c8010460 con 0x7f46d8106560 2024-05-27T07:18:08.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.233+0000 7f46dde00700 1 -- 172.21.15.43:0/1047882579 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d81a2070 con 0x7f46d8106560 2024-05-27T07:18:08.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.233+0000 7f46dca00700 1 -- 172.21.15.43:0/1047882579 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c800de60 con 0x7f46d8106560 2024-05-27T07:18:08.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.233+0000 7f46dde00700 1 -- 172.21.15.43:0/1047882579 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d81a9990 con 0x7f46d8106560 2024-05-27T07:18:08.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.237+0000 7f46dca00700 1 -- 172.21.15.43:0/1047882579 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f46c8010610 con 0x7f46d8106560 2024-05-27T07:18:08.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.238+0000 7f46ec85bb80 1 Processor -- start 2024-05-27T07:18:08.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.238+0000 7f46ec85bb80 1 -- start start 2024-05-27T07:18:08.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46dca00700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46bc0b29f0 0x7f46bc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641ba3e40 0x55a641d287f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d28d30 0x55a641d22870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d22db0 0x55a641d231c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a641c50e40 con 0x55a641d28d30 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a641c409e0 con 0x55a641ba3e40 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a641c44a20 con 0x55a641d22db0 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46d7400700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46bc0b29f0 0x7f46bc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46dca00700 1 -- 172.21.15.43:0/1047882579 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9408+0+0 (secure 0 0 0) 0x7f46c80e2530 con 0x7f46d8106560 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46d6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d28d30 0x55a641d22870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46d6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d28d30 0x55a641d22870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37924/0 (socket says 172.21.15.43:37924) 2024-05-27T07:18:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.239+0000 7f46d6000700 1 -- 172.21.15.43:0/3341835319 learned_addr learned my addr 172.21.15.43:0/3341835319 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.240+0000 7f46d6a00700 1 --2- 172.21.15.43:0/3341835319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641ba3e40 0x55a641d287f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.240+0000 7f46d7400700 1 --2- 172.21.15.43:0/3341835319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d22db0 0x55a641d231c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.240+0000 7f46d6000700 1 -- 172.21.15.43:0/3341835319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d22db0 msgr2=0x55a641d231c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.240+0000 7f46d6000700 1 --2- 172.21.15.43:0/3341835319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d22db0 0x55a641d231c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.240+0000 7f46d6000700 1 -- 172.21.15.43:0/3341835319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641ba3e40 msgr2=0x55a641d287f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.240+0000 7f46d6000700 1 --2- 172.21.15.43:0/3341835319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641ba3e40 0x55a641d287f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.240+0000 7f46d6000700 1 -- 172.21.15.43:0/3341835319 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a641c09420 con 0x55a641d28d30 2024-05-27T07:18:08.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.240+0000 7f46d6a00700 1 --2- 172.21.15.43:0/3341835319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641ba3e40 0x55a641d287f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:08.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.241+0000 7f46d6000700 1 --2- 172.21.15.43:0/3341835319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d28d30 0x55a641d22870 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f46c400bd60 tx=0x7f46c400f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa117a46baa3802b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:08.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.241+0000 7f46d7400700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46bc0b29f0 0x7f46bc0b4f00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009fd0 tx=0x7f46d00093a0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:08.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.241+0000 7f46ccc00700 1 -- 172.21.15.43:0/3341835319 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4015070 con 0x55a641d28d30 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.241+0000 7f46ccc00700 1 -- 172.21.15.43:0/3341835319 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46c400fe50 con 0x55a641d28d30 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.241+0000 7f46ccc00700 1 -- 172.21.15.43:0/3341835319 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4010630 con 0x55a641d28d30 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.242+0000 7f46ec85bb80 1 -- 172.21.15.43:0/3341835319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d28d30 msgr2=0x55a641d22870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.242+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/3341835319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d28d30 0x55a641d22870 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f46c400bd60 tx=0x7f46c400f640 comp rx=0 tx=0).stop 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.242+0000 7f46ec85bb80 1 -- 172.21.15.43:0/3341835319 shutdown_connections 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.242+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/3341835319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d22db0 0x55a641d231c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.242+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/3341835319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641ba3e40 0x55a641d287f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.242+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/3341835319 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d28d30 0x55a641d22870 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.242+0000 7f46ec85bb80 1 -- 172.21.15.43:0/3341835319 >> 172.21.15.43:0/3341835319 conn(0x55a641c03fc0 msgr2=0x55a641d1d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.242+0000 7f46ec85bb80 1 -- 172.21.15.43:0/3341835319 shutdown_connections 2024-05-27T07:18:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.242+0000 7f46ec85bb80 1 -- 172.21.15.43:0/3341835319 wait complete. 2024-05-27T07:18:08.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.243+0000 7f46ec85bb80 1 -- start start 2024-05-27T07:18:08.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.243+0000 7f46ec85bb80 1 Processor -- start 2024-05-27T07:18:08.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.244+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641ba3e40 0x55a641d2c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.244+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d22db0 0x55a641d2ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.244+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d28d30 0x55a641d2cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.244+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a641b46fa0 con 0x55a641d22db0 2024-05-27T07:18:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.244+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a641be1760 con 0x55a641d28d30 2024-05-27T07:18:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.244+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a641d2f800 con 0x55a641ba3e40 2024-05-27T07:18:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.244+0000 7f46ec85bb80 10 client.0 _ll_register_callbacks cb 0x55a641beac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.244+0000 7f46d6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d22db0 0x55a641d2ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.244+0000 7f46d6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d22db0 0x55a641d2ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37926/0 (socket says 172.21.15.43:37926) 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.244+0000 7f46d6000700 1 -- 172.21.15.43:0/1816881746 learned_addr learned my addr 172.21.15.43:0/1816881746 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.245+0000 7f46d6a00700 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641ba3e40 0x55a641d2c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.245+0000 7f46d6000700 1 -- 172.21.15.43:0/1816881746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641ba3e40 msgr2=0x55a641d2c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.245+0000 7f46d6000700 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641ba3e40 0x55a641d2c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.245+0000 7f46d6000700 1 -- 172.21.15.43:0/1816881746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d28d30 msgr2=0x55a641d2cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.245+0000 7f46d6000700 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d28d30 0x55a641d2cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.245+0000 7f46d6000700 1 -- 172.21.15.43:0/1816881746 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a641b908d0 con 0x55a641d22db0 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.245+0000 7f46d6000700 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d22db0 0x55a641d2ca20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f46c4011d70 tx=0x7f46c400fd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd5bd563fafd503 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.245+0000 7f46c1600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:08.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.245+0000 7f46c1600700 1 -- 172.21.15.43:0/1816881746 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4015040 con 0x55a641d22db0 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.246+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1816881746 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46c400fe50 con 0x55a641d22db0 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.246+0000 7f46c1600700 1 -- 172.21.15.43:0/1816881746 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9408+0+0 (secure 0 0 0) 0x7f46c4005040 con 0x55a641d22db0 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.246+0000 7f46c1600700 1 -- 172.21.15.43:0/1816881746 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f46c4030050 con 0x55a641d22db0 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.246+0000 7f46ec85bb80 10 client.16098 fetch_fsmap learned FSMap version 60 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.246+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1816881746 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55a641c11060 con 0x55a641d22db0 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.246+0000 7f46c1600700 1 -- 172.21.15.43:0/1816881746 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46c400fe50 con 0x55a641d22db0 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.247+0000 7f46c1600700 1 -- 172.21.15.43:0/1816881746 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 60) ==== 7168+0+0 (secure 0 0 0) 0x7f46c402b030 con 0x55a641d22db0 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.247+0000 7f46ec85bb80 10 client.16098 fetch_fsmap finished waiting for FSMap version 60 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.247+0000 7f46ec85bb80 10 client.16098 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.247+0000 7f46ec85bb80 20 client.16098 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.247+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a641d3c660 0x55a641d3eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.247+0000 7f46ec85bb80 4 client.16098 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:18:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.247+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1816881746 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a641c52d90 con 0x55a641d3c660 2024-05-27T07:18:08.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.247+0000 7f46d6a00700 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a641d3c660 0x55a641d3eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.248+0000 7f46d6a00700 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a641d3c660 0x55a641d3eaf0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3fc48185fd15c771 server_cookie=7800208540af7a6d in_seq=0 out_seq=0 2024-05-27T07:18:08.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.248+0000 7f46c1600700 10 client.16098 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:18:08.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46c1600700 1 -- 172.21.15.43:0/1816881746 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a641c52d90 con 0x55a641d3c660 2024-05-27T07:18:08.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46c1600700 10 client.16098 handle_command_reply: tid=0 2024-05-27T07:18:08.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46ec85bb80 1 client.16098 shutdown 2024-05-27T07:18:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1816881746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d22db0 msgr2=0x55a641d2ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d22db0 0x55a641d2ca20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f46c4011d70 tx=0x7f46c400fd70 comp rx=0 tx=0).stop 2024-05-27T07:18:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1816881746 shutdown_connections 2024-05-27T07:18:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a641d3c660 0x55a641d3eaf0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d28d30 0x55a641d2cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641ba3e40 0x55a641d2c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46c1600700 0 client.16098 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:18:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/1816881746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641d22db0 0x55a641d2ca20 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1816881746 >> 172.21.15.43:0/1816881746 conn(0x55a641c03fc0 msgr2=0x55a641d1d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.251+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1816881746 shutdown_connections 2024-05-27T07:18:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.252+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1816881746 wait complete. 2024-05-27T07:18:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.252+0000 7f46ec85bb80 20 client.16098 trim_cache size 0 max 16384 2024-05-27T07:18:08.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.270+0000 7f46ec85bb80 1 Processor -- start 2024-05-27T07:18:08.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.270+0000 7f46ec85bb80 1 -- start start 2024-05-27T07:18:08.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.271+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d22870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.271+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d22db0 0x55a641d2c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.271+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d28d30 0x55a641d2c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.271+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f46d81b1070 con 0x55a641ba3e40 2024-05-27T07:18:08.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.271+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46d81b3a10 con 0x55a641d22db0 2024-05-27T07:18:08.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.271+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f46d81b1740 con 0x55a641d28d30 2024-05-27T07:18:08.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.271+0000 7f46d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d22870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.271+0000 7f46d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d22870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37942/0 (socket says 172.21.15.43:37942) 2024-05-27T07:18:08.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.271+0000 7f46d6a00700 1 -- 172.21.15.43:0/1700050869 learned_addr learned my addr 172.21.15.43:0/1700050869 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:08.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.272+0000 7f46d6000700 1 --2- 172.21.15.43:0/1700050869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d22db0 0x55a641d2c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.272+0000 7f46d6a00700 1 -- 172.21.15.43:0/1700050869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d28d30 msgr2=0x55a641d2c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:08.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.272+0000 7f46d6a00700 1 --2- 172.21.15.43:0/1700050869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d28d30 0x55a641d2c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.272+0000 7f46d6a00700 1 -- 172.21.15.43:0/1700050869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d22db0 msgr2=0x55a641d2c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:08.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.272+0000 7f46d6a00700 1 --2- 172.21.15.43:0/1700050869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d22db0 0x55a641d2c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.272+0000 7f46d6a00700 1 -- 172.21.15.43:0/1700050869 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a641d28090 con 0x55a641ba3e40 2024-05-27T07:18:08.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.272+0000 7f46d6a00700 1 --2- 172.21.15.43:0/1700050869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d22870 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f46c80a9160 tx=0x7f46c80a8ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f28dfd78c815a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:08.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.272+0000 7f46c3e00700 1 -- 172.21.15.43:0/1700050869 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c80c2070 con 0x55a641ba3e40 2024-05-27T07:18:08.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.272+0000 7f46c3e00700 1 -- 172.21.15.43:0/1700050869 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46c80b22c0 con 0x55a641ba3e40 2024-05-27T07:18:08.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.272+0000 7f46c3e00700 1 -- 172.21.15.43:0/1700050869 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c8020040 con 0x55a641ba3e40 2024-05-27T07:18:08.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.273+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1700050869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 msgr2=0x55a641d22870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.273+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/1700050869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d22870 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f46c80a9160 tx=0x7f46c80a8ec0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.273+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1700050869 shutdown_connections 2024-05-27T07:18:08.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.273+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/1700050869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d28d30 0x55a641d2c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.273+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/1700050869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d22870 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.273+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/1700050869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d22db0 0x55a641d2c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.273+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1700050869 >> 172.21.15.43:0/1700050869 conn(0x55a641c03fc0 msgr2=0x55a641d2add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:08.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.274+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1700050869 shutdown_connections 2024-05-27T07:18:08.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.274+0000 7f46ec85bb80 1 -- 172.21.15.43:0/1700050869 wait complete. 2024-05-27T07:18:08.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.274+0000 7f46ec85bb80 1 -- start start 2024-05-27T07:18:08.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.274+0000 7f46ec85bb80 1 Processor -- start 2024-05-27T07:18:08.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.275+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d34a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.275+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d22db0 0x55a641d34fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.275+0000 7f46ec85bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d28d30 0x55a641d432a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.275+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f46d81b1e10 con 0x55a641ba3e40 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.275+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46d81b3340 con 0x55a641d28d30 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.275+0000 7f46ec85bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f46d81b24e0 con 0x55a641d22db0 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.275+0000 7f46ec85bb80 10 client.0 _ll_register_callbacks cb 0x55a641beac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d34a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d34a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37948/0 (socket says 172.21.15.43:37948) 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d6a00700 1 -- 172.21.15.43:0/2854980675 learned_addr learned my addr 172.21.15.43:0/2854980675 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d7400700 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d28d30 0x55a641d432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d6000700 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d22db0 0x55a641d34fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d6a00700 1 -- 172.21.15.43:0/2854980675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d22db0 msgr2=0x55a641d34fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d6a00700 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d22db0 0x55a641d34fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d6a00700 1 -- 172.21.15.43:0/2854980675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d28d30 msgr2=0x55a641d432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d6a00700 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d28d30 0x55a641d432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d6a00700 1 -- 172.21.15.43:0/2854980675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a641d44d00 con 0x55a641ba3e40 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46d6a00700 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d34a90 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f46c80a9110 tx=0x7f46c80a9560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71ddd0396bd661f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.276+0000 7f46c1600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.277+0000 7f46c1600700 1 -- 172.21.15.43:0/2854980675 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c8020070 con 0x55a641ba3e40 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.277+0000 7f46c1600700 1 -- 172.21.15.43:0/2854980675 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9408+0+0 (secure 0 0 0) 0x7f46c80a98c0 con 0x55a641ba3e40 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.277+0000 7f46ec85bb80 1 -- 172.21.15.43:0/2854980675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46c80b22c0 con 0x55a641ba3e40 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.277+0000 7f46c1600700 1 -- 172.21.15.43:0/2854980675 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f46c80c2070 con 0x55a641ba3e40 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.277+0000 7f46ec85bb80 10 client.16104 fetch_fsmap learned FSMap version 60 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.277+0000 7f46ec85bb80 1 -- 172.21.15.43:0/2854980675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x55a641da6600 con 0x55a641ba3e40 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.278+0000 7f46c1600700 1 -- 172.21.15.43:0/2854980675 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46c80b22c0 con 0x55a641ba3e40 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.278+0000 7f46c1600700 1 -- 172.21.15.43:0/2854980675 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 60) ==== 7168+0+0 (secure 0 0 0) 0x7f46c80b0b90 con 0x55a641ba3e40 2024-05-27T07:18:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.278+0000 7f46ec85bb80 10 client.16104 fetch_fsmap finished waiting for FSMap version 60 2024-05-27T07:18:08.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.278+0000 7f46ec85bb80 10 client.16104 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:18:08.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.278+0000 7f46ec85bb80 20 client.16104 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:08.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.278+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a641d3c660 0x55a641da8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:08.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.278+0000 7f46ec85bb80 4 client.16104 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:18:08.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.278+0000 7f46ec85bb80 1 -- 172.21.15.43:0/2854980675 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a641c4c570 con 0x55a641d3c660 2024-05-27T07:18:08.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.279+0000 7f46d6000700 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a641d3c660 0x55a641da8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.279+0000 7f46d6000700 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a641d3c660 0x55a641da8cb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=61f7df59a8f3091b server_cookie=29db3de7ca0e09a6 in_seq=0 out_seq=0 2024-05-27T07:18:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.279+0000 7f46c1600700 10 client.16104 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:18:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46c1600700 1 -- 172.21.15.43:0/2854980675 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a641c4c570 con 0x55a641d3c660 2024-05-27T07:18:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46c1600700 10 client.16104 handle_command_reply: tid=0 2024-05-27T07:18:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46ec85bb80 1 client.16104 shutdown 2024-05-27T07:18:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46ec85bb80 1 -- 172.21.15.43:0/2854980675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 msgr2=0x55a641d34a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d34a90 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f46c80a9110 tx=0x7f46c80a9560 comp rx=0 tx=0).stop 2024-05-27T07:18:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46ec85bb80 1 -- 172.21.15.43:0/2854980675 shutdown_connections 2024-05-27T07:18:08.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a641d3c660 0x55a641da8cb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a641d28d30 0x55a641d432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a641ba3e40 0x55a641d34a90 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46ec85bb80 1 --2- 172.21.15.43:0/2854980675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a641d22db0 0x55a641d34fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46c1600700 0 client.16104 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:18:08.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.280+0000 7f46ec85bb80 1 -- 172.21.15.43:0/2854980675 >> 172.21.15.43:0/2854980675 conn(0x55a641c03fc0 msgr2=0x55a641d26140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:08.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.281+0000 7f46ec85bb80 1 -- 172.21.15.43:0/2854980675 shutdown_connections 2024-05-27T07:18:08.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.281+0000 7f46ec85bb80 1 -- 172.21.15.43:0/2854980675 wait complete. 2024-05-27T07:18:08.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.281+0000 7f46ec85bb80 20 client.16104 trim_cache size 0 max 16384 2024-05-27T07:18:08.273 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:18:08.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 -- 172.21.15.43:0/1047882579 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46bc0b29f0 msgr2=0x7f46bc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46bc0b29f0 0x7f46bc0b4f00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009fd0 tx=0x7f46d00093a0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 -- 172.21.15.43:0/1047882579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d8106560 msgr2=0x7f46d819ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:08.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d8106560 0x7f46d819ceb0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f46c800d980 tx=0x7f46c800dd40 comp rx=0 tx=0).stop 2024-05-27T07:18:08.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 -- 172.21.15.43:0/1047882579 shutdown_connections 2024-05-27T07:18:08.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46bc0b29f0 0x7f46bc0b4f00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46d810c5a0 0x7f46d81a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46d8106560 0x7f46d819ceb0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 --2- 172.21.15.43:0/1047882579 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46d8106eb0 0x7f46d819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:08.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 -- 172.21.15.43:0/1047882579 >> 172.21.15.43:0/1047882579 conn(0x7f46d8102040 msgr2=0x7f46d81035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:08.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 -- 172.21.15.43:0/1047882579 shutdown_connections 2024-05-27T07:18:08.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:08.282+0000 7f46dde00700 1 -- 172.21.15.43:0/1047882579 wait complete. 2024-05-27T07:18:08.953 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:08 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:18:08.953 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:08 smithi043 systemd[1]: Stopped Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:08.953 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:08 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.919s CPU time. 2024-05-27T07:18:08.953 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:08 smithi043 systemd[1]: Starting Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:18:09.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 16594 0.54 MB/sec execute 13 sec latency 49.643 ms 2024-05-27T07:18:09.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:18:09.599 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:09 smithi043 podman[99716]: 2024-05-27 07:18:09.470421017 +0000 UTC m=+0.035893601 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:18:09.599 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:09 smithi043 podman[99716]: 2024-05-27 07:18:09.608841112 +0000 UTC m=+0.174313684 container create 890ad32d7d9844db98af845d8f9d47ca3715fb05264e35664655d0d59a1a6500 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:18:09.854 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:09 smithi043 ceph-mon[31531]: pgmap v143: 129 pgs: 129 active+clean; 36 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 326 KiB/s wr, 76 op/s 2024-05-27T07:18:09.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:09 smithi179 ceph-mon[35888]: pgmap v143: 129 pgs: 129 active+clean; 36 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 326 KiB/s wr, 76 op/s 2024-05-27T07:18:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:10.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:09 smithi132 ceph-mon[35283]: pgmap v143: 129 pgs: 129 active+clean; 36 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 326 KiB/s wr, 76 op/s 2024-05-27T07:18:10.157 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:09 smithi043 podman[99716]: 2024-05-27 07:18:09.863750338 +0000 UTC m=+0.429223030 container init 890ad32d7d9844db98af845d8f9d47ca3715fb05264e35664655d0d59a1a6500 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-27T07:18:10.157 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:09 smithi043 podman[99716]: 2024-05-27 07:18:09.873348121 +0000 UTC m=+0.438820767 container start 890ad32d7d9844db98af845d8f9d47ca3715fb05264e35664655d0d59a1a6500 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:18:10.157 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:09 smithi043 bash[99716]: 890ad32d7d9844db98af845d8f9d47ca3715fb05264e35664655d0d59a1a6500 2024-05-27T07:18:10.158 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:09 smithi043 systemd[1]: Started Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:10.158 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:09 smithi043 ceph-mds[99759]: -- 172.21.15.43:0/38411735 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55698952da40 con 0x556989575180 2024-05-27T07:18:10.158 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:09 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: starting mds.j at 2024-05-27T07:18:10.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 16701 0.55 MB/sec execute 14 sec latency 42.475 ms 2024-05-27T07:18:10.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:10 smithi043 ceph-mon[31531]: daemon mds.j restarted 2024-05-27T07:18:10.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:10 smithi179 ceph-mon[35888]: daemon mds.j restarted 2024-05-27T07:18:10.999 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcf37200700 time 2024-05-27T07:18:10.725814+0000 2024-05-27T07:18:10.999 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:10.999 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:10.999 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcf4cdbcba6] 2024-05-27T07:18:10.999 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcf4cdbcdd1] 2024-05-27T07:18:10.999 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5569802b423d] 2024-05-27T07:18:10.999 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55698029f971] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 5: (SnapRealm::check_cache() const+0x13e) [0x55698029dfa0] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55698029e651] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556980233347] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55697ff2f87e] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55697ff34e94] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55697ff3f969] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 11: (MDLog::_replay_thread()+0x1f38) [0x5569802d848c] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5569802e19cb] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 13: (Thread::entry_wrapper()+0x43) [0x7fcf4cd8b72d] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcf4cd8b749] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcf4b92f1ca] 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 16: clone() 2024-05-27T07:18:11.000 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:10.727+0000 7fcf37200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcf37200700 time 2024-05-27T07:18:10.725814+0000 2024-05-27T07:18:11.001 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:11.001 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.001 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:11.001 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcf4cdbcba6] 2024-05-27T07:18:11.001 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcf4cdbcdd1] 2024-05-27T07:18:11.001 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5569802b423d] 2024-05-27T07:18:11.001 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55698029f971] 2024-05-27T07:18:11.001 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 5: (SnapRealm::check_cache() const+0x13e) [0x55698029dfa0] 2024-05-27T07:18:11.001 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55698029e651] 2024-05-27T07:18:11.001 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556980233347] 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55697ff2f87e] 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55697ff34e94] 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55697ff3f969] 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 11: (MDLog::_replay_thread()+0x1f38) [0x5569802d848c] 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5569802e19cb] 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 13: (Thread::entry_wrapper()+0x43) [0x7fcf4cd8b72d] 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcf4cd8b749] 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcf4b92f1ca] 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 16: clone() 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: *** Caught signal (Aborted) ** 2024-05-27T07:18:11.002 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: in thread 7fcf37200700 thread_name:md_log_replay 2024-05-27T07:18:11.003 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:11.003 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcf4b939d20] 2024-05-27T07:18:11.003 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2: gsignal() 2024-05-27T07:18:11.003 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 3: abort() 2024-05-27T07:18:11.003 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcf4cdbcced] 2024-05-27T07:18:11.003 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcf4cdbcdd1] 2024-05-27T07:18:11.003 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5569802b423d] 2024-05-27T07:18:11.003 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55698029f971] 2024-05-27T07:18:11.003 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 8: (SnapRealm::check_cache() const+0x13e) [0x55698029dfa0] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55698029e651] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556980233347] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55697ff2f87e] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55697ff34e94] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55697ff3f969] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 14: (MDLog::_replay_thread()+0x1f38) [0x5569802d848c] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5569802e19cb] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 16: (Thread::entry_wrapper()+0x43) [0x7fcf4cd8b72d] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcf4cd8b749] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcf4b92f1ca] 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 19: clone() 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:10.730+0000 7fcf37200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:11.004 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: in thread 7fcf37200700 thread_name:md_log_replay 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcf4b939d20] 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2: gsignal() 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 3: abort() 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcf4cdbcced] 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcf4cdbcdd1] 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5569802b423d] 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55698029f971] 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 8: (SnapRealm::check_cache() const+0x13e) [0x55698029dfa0] 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55698029e651] 2024-05-27T07:18:11.005 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556980233347] 2024-05-27T07:18:11.006 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55697ff2f87e] 2024-05-27T07:18:11.006 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55697ff34e94] 2024-05-27T07:18:11.006 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55697ff3f969] 2024-05-27T07:18:11.006 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 14: (MDLog::_replay_thread()+0x1f38) [0x5569802d848c] 2024-05-27T07:18:11.006 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5569802e19cb] 2024-05-27T07:18:11.006 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 16: (Thread::entry_wrapper()+0x43) [0x7fcf4cd8b72d] 2024-05-27T07:18:11.006 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcf4cd8b749] 2024-05-27T07:18:11.006 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcf4b92f1ca] 2024-05-27T07:18:11.006 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 19: clone() 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: -1> 2024-05-27T07:18:10.727+0000 7fcf37200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcf37200700 time 2024-05-27T07:18:10.725814+0000 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcf4cdbcba6] 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcf4cdbcdd1] 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5569802b423d] 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55698029f971] 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 5: (SnapRealm::check_cache() const+0x13e) [0x55698029dfa0] 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55698029e651] 2024-05-27T07:18:11.007 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556980233347] 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55697ff2f87e] 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55697ff34e94] 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55697ff3f969] 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 11: (MDLog::_replay_thread()+0x1f38) [0x5569802d848c] 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5569802e19cb] 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 13: (Thread::entry_wrapper()+0x43) [0x7fcf4cd8b72d] 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcf4cd8b749] 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcf4b92f1ca] 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 16: clone() 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 0> 2024-05-27T07:18:10.730+0000 7fcf37200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: in thread 7fcf37200700 thread_name:md_log_replay 2024-05-27T07:18:11.008 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcf4b939d20] 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2: gsignal() 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 3: abort() 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcf4cdbcced] 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcf4cdbcdd1] 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5569802b423d] 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55698029f971] 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 8: (SnapRealm::check_cache() const+0x13e) [0x55698029dfa0] 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55698029e651] 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556980233347] 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55697ff2f87e] 2024-05-27T07:18:11.009 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55697ff34e94] 2024-05-27T07:18:11.010 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55697ff3f969] 2024-05-27T07:18:11.010 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 14: (MDLog::_replay_thread()+0x1f38) [0x5569802d848c] 2024-05-27T07:18:11.010 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5569802e19cb] 2024-05-27T07:18:11.010 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 16: (Thread::entry_wrapper()+0x43) [0x7fcf4cd8b72d] 2024-05-27T07:18:11.010 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcf4cd8b749] 2024-05-27T07:18:11.010 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcf4b92f1ca] 2024-05-27T07:18:11.010 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 19: clone() 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: -1> 2024-05-27T07:18:10.727+0000 7fcf37200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcf37200700 time 2024-05-27T07:18:10.725814+0000 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcf4cdbcba6] 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcf4cdbcdd1] 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5569802b423d] 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55698029f971] 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 5: (SnapRealm::check_cache() const+0x13e) [0x55698029dfa0] 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55698029e651] 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556980233347] 2024-05-27T07:18:11.011 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55697ff2f87e] 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55697ff34e94] 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55697ff3f969] 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 11: (MDLog::_replay_thread()+0x1f38) [0x5569802d848c] 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5569802e19cb] 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 13: (Thread::entry_wrapper()+0x43) [0x7fcf4cd8b72d] 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcf4cd8b749] 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcf4b92f1ca] 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 16: clone() 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 0> 2024-05-27T07:18:10.730+0000 7fcf37200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: in thread 7fcf37200700 thread_name:md_log_replay 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.012 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcf4b939d20] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2: gsignal() 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 3: abort() 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcf4cdbcced] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcf4cdbcdd1] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5569802b423d] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55698029f971] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 8: (SnapRealm::check_cache() const+0x13e) [0x55698029dfa0] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55698029e651] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x556980233347] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55697ff2f87e] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55697ff34e94] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55697ff3f969] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 14: (MDLog::_replay_thread()+0x1f38) [0x5569802d848c] 2024-05-27T07:18:11.013 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5569802e19cb] 2024-05-27T07:18:11.014 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 16: (Thread::entry_wrapper()+0x43) [0x7fcf4cd8b72d] 2024-05-27T07:18:11.014 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcf4cd8b749] 2024-05-27T07:18:11.014 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcf4b92f1ca] 2024-05-27T07:18:11.014 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 19: clone() 2024-05-27T07:18:11.014 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:11.014 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[99743]: 2024-05-27T07:18:11.014 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:11 smithi043 podman[99839]: 2024-05-27 07:18:11.006480012 +0000 UTC m=+0.226536339 container died 890ad32d7d9844db98af845d8f9d47ca3715fb05264e35664655d0d59a1a6500 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:18:11.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:10 smithi132 ceph-mon[35283]: daemon mds.j restarted 2024-05-27T07:18:11.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 16813 0.54 MB/sec execute 15 sec latency 57.473 ms 2024-05-27T07:18:11.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:11 smithi043 ceph-mon[31531]: pgmap v144: 129 pgs: 129 active+clean; 38 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 346 KiB/s wr, 67 op/s 2024-05-27T07:18:11.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:11 smithi043 ceph-mon[31531]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:18:11.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:11 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/1813962286,v1:172.21.15.43:6835/1813962286] up:boot 2024-05-27T07:18:11.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:11 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:11.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:11 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:11.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:11 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:11 smithi179 ceph-mon[35888]: pgmap v144: 129 pgs: 129 active+clean; 38 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 346 KiB/s wr, 67 op/s 2024-05-27T07:18:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:11 smithi179 ceph-mon[35888]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:18:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:11 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/1813962286,v1:172.21.15.43:6835/1813962286] up:boot 2024-05-27T07:18:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:11 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:11 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:11.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:11 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:12.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:11 smithi132 ceph-mon[35283]: pgmap v144: 129 pgs: 129 active+clean; 38 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 346 KiB/s wr, 67 op/s 2024-05-27T07:18:12.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:11 smithi132 ceph-mon[35283]: osdmap e105: 12 total, 12 up, 12 in 2024-05-27T07:18:12.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:11 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/1813962286,v1:172.21.15.43:6835/1813962286] up:boot 2024-05-27T07:18:12.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:11 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:12.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:11 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:12.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:11 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:12.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 17094 0.74 MB/sec execute 16 sec latency 58.178 ms 2024-05-27T07:18:12.202 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:11 smithi043 podman[99839]: 2024-05-27 07:18:11.89266387 +0000 UTC m=+1.112720157 container remove 890ad32d7d9844db98af845d8f9d47ca3715fb05264e35664655d0d59a1a6500 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:18:12.202 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:11 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:18:13.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 17213 0.74 MB/sec execute 17 sec latency 58.048 ms 2024-05-27T07:18:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.424+0000 7f6f7ec00700 1 -- 172.21.15.43:0/3078865508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f78105bf0 msgr2=0x7f6f7810e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.424+0000 7f6f7ec00700 1 --2- 172.21.15.43:0/3078865508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f78105bf0 0x7f6f7810e140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6f64009990 tx=0x7f6f64009ca0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.424+0000 7f6f7ec00700 1 -- 172.21.15.43:0/3078865508 shutdown_connections 2024-05-27T07:18:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.424+0000 7f6f7ec00700 1 --2- 172.21.15.43:0/3078865508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f7810e710 0x7f6f78110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.424+0000 7f6f7ec00700 1 --2- 172.21.15.43:0/3078865508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f781052a0 0x7f6f781056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.424+0000 7f6f7ec00700 1 --2- 172.21.15.43:0/3078865508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f78105bf0 0x7f6f7810e140 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.424+0000 7f6f7ec00700 1 -- 172.21.15.43:0/3078865508 >> 172.21.15.43:0/3078865508 conn(0x7f6f78078c30 msgr2=0x7f6f78079070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.425+0000 7f6f7ec00700 1 -- 172.21.15.43:0/3078865508 shutdown_connections 2024-05-27T07:18:13.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.425+0000 7f6f7ec00700 1 -- 172.21.15.43:0/3078865508 wait complete. 2024-05-27T07:18:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.426+0000 7f6f7ec00700 1 Processor -- start 2024-05-27T07:18:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.426+0000 7f6f7ec00700 1 -- start start 2024-05-27T07:18:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.427+0000 7f6f7ec00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f781052a0 0x7f6f7819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.427+0000 7f6f7ec00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f78105bf0 0x7f6f7819d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.427+0000 7f6f7ec00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f7810e710 0x7f6f781a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.427+0000 7f6f7ec00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f78114190 con 0x7f6f7810e710 2024-05-27T07:18:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.427+0000 7f6f7ec00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f78114010 con 0x7f6f781052a0 2024-05-27T07:18:13.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.427+0000 7f6f7ec00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6f78114310 con 0x7f6f78105bf0 2024-05-27T07:18:13.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.427+0000 7f6f7e200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f7810e710 0x7f6f781a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.427+0000 7f6f7e200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f7810e710 0x7f6f781a15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37998/0 (socket says 172.21.15.43:37998) 2024-05-27T07:18:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.427+0000 7f6f7e200700 1 -- 172.21.15.43:0/4144012183 learned_addr learned my addr 172.21.15.43:0/4144012183 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.427+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f78105bf0 0x7f6f7819d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.428+0000 7f6f7d800700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f781052a0 0x7f6f7819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.428+0000 7f6f7e200700 1 -- 172.21.15.43:0/4144012183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f78105bf0 msgr2=0x7f6f7819d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.428+0000 7f6f7e200700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f78105bf0 0x7f6f7819d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.428+0000 7f6f7e200700 1 -- 172.21.15.43:0/4144012183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f781052a0 msgr2=0x7f6f7819cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.428+0000 7f6f7e200700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f781052a0 0x7f6f7819cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.428+0000 7f6f7e200700 1 -- 172.21.15.43:0/4144012183 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f781a1c90 con 0x7f6f7810e710 2024-05-27T07:18:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.428+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f78105bf0 0x7f6f7819d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.428+0000 7f6f7e200700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f7810e710 0x7f6f781a15f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6f70007ee0 tx=0x7f6f7000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.429+0000 7f6f76000700 1 -- 172.21.15.43:0/4144012183 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7000dbd0 con 0x7f6f7810e710 2024-05-27T07:18:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.429+0000 7f6f7ec00700 1 -- 172.21.15.43:0/4144012183 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f781a1f80 con 0x7f6f7810e710 2024-05-27T07:18:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.429+0000 7f6f76000700 1 -- 172.21.15.43:0/4144012183 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f7000dd70 con 0x7f6f7810e710 2024-05-27T07:18:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.429+0000 7f6f76000700 1 -- 172.21.15.43:0/4144012183 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f700058d0 con 0x7f6f7810e710 2024-05-27T07:18:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.429+0000 7f6f7ec00700 1 -- 172.21.15.43:0/4144012183 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f781a98a0 con 0x7f6f7810e710 2024-05-27T07:18:13.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.433+0000 7f6f76000700 1 -- 172.21.15.43:0/4144012183 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6f7001a020 con 0x7f6f7810e710 2024-05-27T07:18:13.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.434+0000 7f6f8ecfcb80 1 Processor -- start 2024-05-27T07:18:13.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.434+0000 7f6f8ecfcb80 1 -- start start 2024-05-27T07:18:13.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632283029f0 0x563228396750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563228216e40 0x563228396c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283a02c0 0x5632283a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5632282c3e40 con 0x563228216e40 2024-05-27T07:18:13.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5632282b39e0 con 0x5632283a02c0 2024-05-27T07:18:13.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5632282b7a20 con 0x5632283029f0 2024-05-27T07:18:13.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f76000700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f600b29f0 0x7f6f600b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f7d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563228216e40 0x563228396c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f7d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563228216e40 0x563228396c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38002/0 (socket says 172.21.15.43:38002) 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f76000700 1 -- 172.21.15.43:0/4144012183 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9408+0+0 (secure 0 0 0) 0x7f6f700e21f0 con 0x7f6f7810e710 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.435+0000 7f6f7d800700 1 -- 172.21.15.43:0/266619988 learned_addr learned my addr 172.21.15.43:0/266619988 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.436+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/266619988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632283029f0 0x563228396750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.436+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/266619988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283a02c0 0x5632283a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.436+0000 7f6f7e200700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f600b29f0 0x7f6f600b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.436+0000 7f6f7d800700 1 -- 172.21.15.43:0/266619988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632283029f0 msgr2=0x563228396750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.436+0000 7f6f7d800700 1 --2- 172.21.15.43:0/266619988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632283029f0 0x563228396750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.436+0000 7f6f7d800700 1 -- 172.21.15.43:0/266619988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283a02c0 msgr2=0x5632283a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.436+0000 7f6f7d800700 1 --2- 172.21.15.43:0/266619988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283a02c0 0x5632283a26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.436+0000 7f6f7d800700 1 -- 172.21.15.43:0/266619988 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56322827c420 con 0x563228216e40 2024-05-27T07:18:13.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.436+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/266619988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283a02c0 0x5632283a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:18:13.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.437+0000 7f6f7d800700 1 --2- 172.21.15.43:0/266619988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563228216e40 0x563228396c90 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00bd30 tx=0x7f6f6c00f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b627d8900b5911a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:13.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.437+0000 7f6f7e200700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f600b29f0 0x7f6f600b4f00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6f700a51a0 tx=0x7f6f70004e80 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:13.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.437+0000 7f6f6aa00700 1 -- 172.21.15.43:0/266619988 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c015070 con 0x563228216e40 2024-05-27T07:18:13.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.437+0000 7f6f6aa00700 1 -- 172.21.15.43:0/266619988 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f6c00fe50 con 0x563228216e40 2024-05-27T07:18:13.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.437+0000 7f6f6aa00700 1 -- 172.21.15.43:0/266619988 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c011440 con 0x563228216e40 2024-05-27T07:18:13.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.438+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/266619988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563228216e40 msgr2=0x563228396c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.438+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/266619988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563228216e40 0x563228396c90 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00bd30 tx=0x7f6f6c00f640 comp rx=0 tx=0).stop 2024-05-27T07:18:13.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.438+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/266619988 shutdown_connections 2024-05-27T07:18:13.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.438+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/266619988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283a02c0 0x5632283a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.438+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/266619988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632283029f0 0x563228396750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.438+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/266619988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563228216e40 0x563228396c90 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.438+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/266619988 >> 172.21.15.43:0/266619988 conn(0x5632282b0390 msgr2=0x563228391710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:13.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.438+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/266619988 shutdown_connections 2024-05-27T07:18:13.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.439+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/266619988 wait complete. 2024-05-27T07:18:13.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.439+0000 7f6f8ecfcb80 1 -- start start 2024-05-27T07:18:13.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.439+0000 7f6f8ecfcb80 1 Processor -- start 2024-05-27T07:18:13.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.440+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563228216e40 0x56322839d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.440+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632283029f0 0x56322839d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.440+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x56322839dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.440+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5632281b9fa0 con 0x5632283a02c0 2024-05-27T07:18:13.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.440+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x563228254760 con 0x563228216e40 2024-05-27T07:18:13.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.440+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5632283a2e00 con 0x5632283029f0 2024-05-27T07:18:13.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.440+0000 7f6f8ecfcb80 10 client.0 _ll_register_callbacks cb 0x56322825dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:13.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.440+0000 7f6f7ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x56322839dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.441+0000 7f6f7ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x56322839dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38004/0 (socket says 172.21.15.43:38004) 2024-05-27T07:18:13.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.441+0000 7f6f7ce00700 1 -- 172.21.15.43:0/2916071015 learned_addr learned my addr 172.21.15.43:0/2916071015 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:13.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.441+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563228216e40 0x56322839d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.441+0000 7f6f7d800700 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632283029f0 0x56322839d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.441+0000 7f6f7ce00700 1 -- 172.21.15.43:0/2916071015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632283029f0 msgr2=0x56322839d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.441+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632283029f0 0x56322839d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.441+0000 7f6f7ce00700 1 -- 172.21.15.43:0/2916071015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563228216e40 msgr2=0x56322839d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.441+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563228216e40 0x56322839d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.441+0000 7f6f7ce00700 1 -- 172.21.15.43:0/2916071015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632282038d0 con 0x5632283a02c0 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.442+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x56322839dcd0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x56322839cad0 tx=0x7f6f64005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a1844aff6354e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.442+0000 7f6f5f400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.442+0000 7f6f5f400700 1 -- 172.21.15.43:0/2916071015 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640039c0 con 0x5632283a02c0 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.442+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/2916071015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f6c00fe50 con 0x5632283a02c0 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.442+0000 7f6f5f400700 1 -- 172.21.15.43:0/2916071015 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9408+0+0 (secure 0 0 0) 0x7f6f64003b60 con 0x5632283a02c0 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.442+0000 7f6f5f400700 1 -- 172.21.15.43:0/2916071015 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f6f64003d60 con 0x5632283a02c0 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.442+0000 7f6f8ecfcb80 10 client.16122 fetch_fsmap learned FSMap version 62 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.442+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/2916071015 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x563228284060 con 0x5632283a02c0 2024-05-27T07:18:13.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.443+0000 7f6f5f400700 1 -- 172.21.15.43:0/2916071015 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f6c00fe50 con 0x5632283a02c0 2024-05-27T07:18:13.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.443+0000 7f6f5f400700 1 -- 172.21.15.43:0/2916071015 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 62) ==== 7168+0+0 (secure 0 0 0) 0x7f6f6400f460 con 0x5632283a02c0 2024-05-27T07:18:13.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.443+0000 7f6f8ecfcb80 10 client.16122 fetch_fsmap finished waiting for FSMap version 62 2024-05-27T07:18:13.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.443+0000 7f6f8ecfcb80 10 client.16122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:18:13.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.443+0000 7f6f8ecfcb80 20 client.16122 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:13.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.443+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5632283afc50 0x5632283b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.443+0000 7f6f8ecfcb80 4 client.16122 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:18:13.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.443+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/2916071015 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632282c5d90 con 0x5632283afc50 2024-05-27T07:18:13.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.444+0000 7f6f7d800700 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5632283afc50 0x5632283b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.445+0000 7f6f7d800700 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5632283afc50 0x5632283b20d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8e9b431eac666e8 server_cookie=4af88a615aa9382f in_seq=0 out_seq=0 2024-05-27T07:18:13.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.445+0000 7f6f5f400700 10 client.16122 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:13.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.447+0000 7f6f5f400700 1 -- 172.21.15.43:0/2916071015 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5632282c5d90 con 0x5632283afc50 2024-05-27T07:18:13.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.447+0000 7f6f5f400700 10 client.16122 handle_command_reply: tid=0 2024-05-27T07:18:13.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.447+0000 7f6f8ecfcb80 1 client.16122 shutdown 2024-05-27T07:18:13.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.447+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/2916071015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 msgr2=0x56322839dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.447+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x56322839dcd0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x56322839cad0 tx=0x7f6f64005fb0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.447+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/2916071015 shutdown_connections 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.447+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5632283afc50 0x5632283b20d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.448+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x56322839dcd0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.448+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563228216e40 0x56322839d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.448+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/2916071015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632283029f0 0x56322839d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.448+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/2916071015 >> 172.21.15.43:0/2916071015 conn(0x5632282b0390 msgr2=0x5632283967f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.448+0000 7f6f5f400700 0 client.16122 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.448+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/2916071015 shutdown_connections 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.448+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/2916071015 wait complete. 2024-05-27T07:18:13.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.448+0000 7f6f8ecfcb80 20 client.16122 trim_cache size 0 max 16384 2024-05-27T07:18:13.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.467+0000 7f6f8ecfcb80 1 Processor -- start 2024-05-27T07:18:13.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.467+0000 7f6f8ecfcb80 1 -- start start 2024-05-27T07:18:13.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.467+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563228216e40 0x563228395830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.468+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 0x5632283abea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.468+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.468+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f781b0ed0 con 0x5632283a02c0 2024-05-27T07:18:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.468+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f781b3870 con 0x5632283029f0 2024-05-27T07:18:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.468+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6f781b15a0 con 0x563228216e40 2024-05-27T07:18:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.468+0000 7f6f7ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.468+0000 7f6f7d800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 0x5632283abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.468+0000 7f6f7ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283b78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38006/0 (socket says 172.21.15.43:38006) 2024-05-27T07:18:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.468+0000 7f6f7ce00700 1 -- 172.21.15.43:0/1215167818 learned_addr learned my addr 172.21.15.43:0/1215167818 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:13.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.468+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/1215167818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563228216e40 0x563228395830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.469+0000 7f6f7d800700 1 -- 172.21.15.43:0/1215167818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563228216e40 msgr2=0x563228395830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:13.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.469+0000 7f6f7d800700 1 --2- 172.21.15.43:0/1215167818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563228216e40 0x563228395830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.469+0000 7f6f7d800700 1 -- 172.21.15.43:0/1215167818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 msgr2=0x5632283b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:13.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.469+0000 7f6f7d800700 1 --2- 172.21.15.43:0/1215167818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.469+0000 7f6f7d800700 1 -- 172.21.15.43:0/1215167818 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563228395d70 con 0x5632283029f0 2024-05-27T07:18:13.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.470+0000 7f6f7d800700 1 --2- 172.21.15.43:0/1215167818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 0x5632283abea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c005310 tx=0x7f6f6c00b620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af1466f20d22f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:13.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.470+0000 7f6f6a000700 1 -- 172.21.15.43:0/1215167818 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c015040 con 0x5632283029f0 2024-05-27T07:18:13.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.470+0000 7f6f6a000700 1 -- 172.21.15.43:0/1215167818 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f6c005540 con 0x5632283029f0 2024-05-27T07:18:13.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.470+0000 7f6f6a000700 1 -- 172.21.15.43:0/1215167818 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c0116a0 con 0x5632283029f0 2024-05-27T07:18:13.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.471+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/1215167818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 msgr2=0x5632283abea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.471+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/1215167818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 0x5632283abea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c005310 tx=0x7f6f6c00b620 comp rx=0 tx=0).stop 2024-05-27T07:18:13.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.471+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/1215167818 shutdown_connections 2024-05-27T07:18:13.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.471+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/1215167818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.471+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/1215167818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563228216e40 0x563228395830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.471+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/1215167818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 0x5632283abea0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.471+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/1215167818 >> 172.21.15.43:0/1215167818 conn(0x5632282b0390 msgr2=0x563228395120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:13.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.471+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/1215167818 shutdown_connections 2024-05-27T07:18:13.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.471+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/1215167818 wait complete. 2024-05-27T07:18:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.472+0000 7f6f8ecfcb80 1 -- start start 2024-05-27T07:18:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.472+0000 7f6f8ecfcb80 1 Processor -- start 2024-05-27T07:18:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563228216e40 0x5632283a8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 0x5632283a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f8ecfcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f781b1c70 con 0x5632283a02c0 2024-05-27T07:18:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f781b31a0 con 0x5632283029f0 2024-05-27T07:18:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f8ecfcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6f781b2340 con 0x563228216e40 2024-05-27T07:18:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f8ecfcb80 10 client.0 _ll_register_callbacks cb 0x56322825dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f7ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f7ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283a3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38014/0 (socket says 172.21.15.43:38014) 2024-05-27T07:18:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f7ce00700 1 -- 172.21.15.43:0/134112486 learned_addr learned my addr 172.21.15.43:0/134112486 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.473+0000 7f6f7d800700 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 0x5632283a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563228216e40 0x5632283a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f7ce00700 1 -- 172.21.15.43:0/134112486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563228216e40 msgr2=0x5632283a8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563228216e40 0x5632283a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f7ce00700 1 -- 172.21.15.43:0/134112486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 msgr2=0x5632283a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 0x5632283a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f7ce00700 1 -- 172.21.15.43:0/134112486 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632283a46a0 con 0x5632283a02c0 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f7d800700 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 0x5632283a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f7ce00700 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283a3fb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6f64001eb0 tx=0x7f6f64009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3a314b700e4e1e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f5f400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f5f400700 1 -- 172.21.15.43:0/134112486 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6403f060 con 0x5632283a02c0 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.474+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/134112486 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f6c005540 con 0x5632283a02c0 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.475+0000 7f6f5f400700 1 -- 172.21.15.43:0/134112486 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9408+0+0 (secure 0 0 0) 0x7f6f640039c0 con 0x5632283a02c0 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.475+0000 7f6f5f400700 1 -- 172.21.15.43:0/134112486 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f6f64046050 con 0x5632283a02c0 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.475+0000 7f6f8ecfcb80 10 client.16128 fetch_fsmap learned FSMap version 62 2024-05-27T07:18:13.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.475+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/134112486 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x56322841a630 con 0x5632283a02c0 2024-05-27T07:18:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.475+0000 7f6f5f400700 1 -- 172.21.15.43:0/134112486 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f6c005540 con 0x5632283a02c0 2024-05-27T07:18:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.475+0000 7f6f5f400700 1 -- 172.21.15.43:0/134112486 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 62) ==== 7168+0+0 (secure 0 0 0) 0x7f6f6403a550 con 0x5632283a02c0 2024-05-27T07:18:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.475+0000 7f6f8ecfcb80 10 client.16128 fetch_fsmap finished waiting for FSMap version 62 2024-05-27T07:18:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.475+0000 7f6f8ecfcb80 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:18:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.475+0000 7f6f8ecfcb80 20 client.16128 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.476+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56322841ac50 0x56322841d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.476+0000 7f6f8ecfcb80 4 client.16128 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.476+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/134112486 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632282bf570 con 0x56322841ac50 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.476+0000 7f6f7d800700 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56322841ac50 0x56322841d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.477+0000 7f6f7d800700 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56322841ac50 0x56322841d070 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8340033d322b78ce server_cookie=7f6ea4b1f4447c69 in_seq=0 out_seq=0 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.477+0000 7f6f5f400700 10 client.16128 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f5f400700 1 -- 172.21.15.43:0/134112486 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632282bf570 con 0x56322841ac50 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f5f400700 10 client.16128 handle_command_reply: tid=0 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 client.16128 shutdown 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/134112486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 msgr2=0x5632283a3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283a3fb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6f64001eb0 tx=0x7f6f64009570 comp rx=0 tx=0).stop 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/134112486 shutdown_connections 2024-05-27T07:18:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56322841ac50 0x56322841d070 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632283a02c0 0x5632283a3fb0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563228216e40 0x5632283a8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 --2- 172.21.15.43:0/134112486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5632283029f0 0x5632283a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/134112486 >> 172.21.15.43:0/134112486 conn(0x5632282b0390 msgr2=0x563228390fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f5f400700 0 client.16128 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/134112486 shutdown_connections 2024-05-27T07:18:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 1 -- 172.21.15.43:0/134112486 wait complete. 2024-05-27T07:18:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.478+0000 7f6f8ecfcb80 20 client.16128 trim_cache size 0 max 16384 2024-05-27T07:18:13.471 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:18:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.479+0000 7f6f7ec00700 1 -- 172.21.15.43:0/4144012183 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f600b29f0 msgr2=0x7f6f600b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.479+0000 7f6f7ec00700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f600b29f0 0x7f6f600b4f00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6f700a51a0 tx=0x7f6f70004e80 comp rx=0 tx=0).stop 2024-05-27T07:18:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.479+0000 7f6f7ec00700 1 -- 172.21.15.43:0/4144012183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f7810e710 msgr2=0x7f6f781a15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.479+0000 7f6f7ec00700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f7810e710 0x7f6f781a15f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6f70007ee0 tx=0x7f6f7000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.479+0000 7f6f7ec00700 1 -- 172.21.15.43:0/4144012183 shutdown_connections 2024-05-27T07:18:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.479+0000 7f6f7ec00700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f600b29f0 0x7f6f600b4f00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.479+0000 7f6f7ec00700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f7810e710 0x7f6f781a15f0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.479+0000 7f6f7ec00700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f781052a0 0x7f6f7819cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.479+0000 7f6f7ec00700 1 --2- 172.21.15.43:0/4144012183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f78105bf0 0x7f6f7819d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:13.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.479+0000 7f6f7ec00700 1 -- 172.21.15.43:0/4144012183 >> 172.21.15.43:0/4144012183 conn(0x7f6f78078c30 msgr2=0x7f6f78102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:13.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.480+0000 7f6f7ec00700 1 -- 172.21.15.43:0/4144012183 shutdown_connections 2024-05-27T07:18:13.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:13.480+0000 7f6f7ec00700 1 -- 172.21.15.43:0/4144012183 wait complete. 2024-05-27T07:18:13.636 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:13 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:18:13.636 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:13 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.495s CPU time. 2024-05-27T07:18:13.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:13 smithi043 ceph-mon[31531]: pgmap v146: 129 pgs: 129 active+clean; 38 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 379 KiB/s wr, 73 op/s 2024-05-27T07:18:13.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:13 smithi179 ceph-mon[35888]: pgmap v146: 129 pgs: 129 active+clean; 38 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 379 KiB/s wr, 73 op/s 2024-05-27T07:18:14.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:13 smithi132 ceph-mon[35283]: pgmap v146: 129 pgs: 129 active+clean; 38 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 379 KiB/s wr, 73 op/s 2024-05-27T07:18:14.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 17475 0.80 MB/sec execute 18 sec latency 54.230 ms 2024-05-27T07:18:15.187 INFO:tasks.workunit.client.0.smithi043.stdout: 1 17501 0.76 MB/sec execute 19 sec latency 204.645 ms 2024-05-27T07:18:15.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:15 smithi043 ceph-mon[31531]: pgmap v147: 129 pgs: 129 active+clean; 38 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 346 KiB/s wr, 67 op/s 2024-05-27T07:18:15.952 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:15 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:18:15.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:15 smithi043 systemd[1]: Stopped Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:15.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:15 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.637s CPU time. 2024-05-27T07:18:15.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:15 smithi043 systemd[1]: Starting Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:18:15.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:15 smithi179 ceph-mon[35888]: pgmap v147: 129 pgs: 129 active+clean; 38 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 346 KiB/s wr, 67 op/s 2024-05-27T07:18:16.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:15 smithi132 ceph-mon[35283]: pgmap v147: 129 pgs: 129 active+clean; 38 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 346 KiB/s wr, 67 op/s 2024-05-27T07:18:16.188 INFO:tasks.workunit.client.0.smithi043.stdout: 1 17528 0.72 MB/sec execute 20 sec latency 59.210 ms 2024-05-27T07:18:16.215 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:18:16.215 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:18:16.400 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:16 smithi043 podman[100017]: 2024-05-27 07:18:16.053503695 +0000 UTC m=+0.041303302 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:18:16.400 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:16 smithi043 podman[100017]: 2024-05-27 07:18:16.228285014 +0000 UTC m=+0.216084615 container create 40e9a696bd1f736da2644ec1607b6b1ecfb999dda5c589c6de1034c940a68901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:18:16.790 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:16.941 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:16 smithi043 podman[100017]: 2024-05-27 07:18:16.523681677 +0000 UTC m=+0.511481268 container init 40e9a696bd1f736da2644ec1607b6b1ecfb999dda5c589c6de1034c940a68901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-27T07:18:16.941 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:16 smithi043 podman[100017]: 2024-05-27 07:18:16.533188599 +0000 UTC m=+0.520988164 container start 40e9a696bd1f736da2644ec1607b6b1ecfb999dda5c589c6de1034c940a68901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:18:16.941 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:16 smithi043 bash[100017]: 40e9a696bd1f736da2644ec1607b6b1ecfb999dda5c589c6de1034c940a68901 2024-05-27T07:18:16.942 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:16 smithi043 systemd[1]: Started Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:16.942 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:16 smithi043 ceph-mds[100055]: -- 172.21.15.43:0/1306031890 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562de3c7da40 con 0x562de3cc5180 2024-05-27T07:18:16.942 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:16 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: starting mds.d at 2024-05-27T07:18:17.188 INFO:tasks.workunit.client.0.smithi043.stdout: 1 17723 0.83 MB/sec execute 21 sec latency 48.865 ms 2024-05-27T07:18:17.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:17 smithi043 ceph-mon[31531]: pgmap v148: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 460 KiB/s wr, 43 op/s 2024-05-27T07:18:17.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:17 smithi043 ceph-mon[31531]: daemon mds.d restarted 2024-05-27T07:18:17.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5aecc00700 time 2024-05-27T07:18:17.805631+0000 2024-05-27T07:18:17.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:17.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:17.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5b027ffba6] 2024-05-27T07:18:17.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b027ffdd1] 2024-05-27T07:18:17.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562dd907823d] 2024-05-27T07:18:17.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562dd9063971] 2024-05-27T07:18:17.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 5: (SnapRealm::check_cache() const+0x13e) [0x562dd9061fa0] 2024-05-27T07:18:17.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562dd9062651] 2024-05-27T07:18:17.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562dd8ff7347] 2024-05-27T07:18:17.954 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562dd8cf387e] 2024-05-27T07:18:17.954 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562dd8cf8e94] 2024-05-27T07:18:17.954 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562dd8d03969] 2024-05-27T07:18:17.954 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 11: (MDLog::_replay_thread()+0x1f38) [0x562dd909c48c] 2024-05-27T07:18:17.954 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562dd90a59cb] 2024-05-27T07:18:17.954 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 13: (Thread::entry_wrapper()+0x43) [0x7f5b027ce72d] 2024-05-27T07:18:17.954 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5b027ce749] 2024-05-27T07:18:17.954 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5b013721ca] 2024-05-27T07:18:17.954 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 16: clone() 2024-05-27T07:18:17.954 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.808+0000 7f5aecc00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5aecc00700 time 2024-05-27T07:18:17.805631+0000 2024-05-27T07:18:17.955 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:17.955 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.955 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:17.955 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5b027ffba6] 2024-05-27T07:18:17.955 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b027ffdd1] 2024-05-27T07:18:17.955 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562dd907823d] 2024-05-27T07:18:17.955 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562dd9063971] 2024-05-27T07:18:17.955 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 5: (SnapRealm::check_cache() const+0x13e) [0x562dd9061fa0] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562dd9062651] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562dd8ff7347] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562dd8cf387e] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562dd8cf8e94] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562dd8d03969] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 11: (MDLog::_replay_thread()+0x1f38) [0x562dd909c48c] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562dd90a59cb] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 13: (Thread::entry_wrapper()+0x43) [0x7f5b027ce72d] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5b027ce749] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5b013721ca] 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 16: clone() 2024-05-27T07:18:17.956 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: *** Caught signal (Aborted) ** 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: in thread 7f5aecc00700 thread_name:md_log_replay 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5b0137cd20] 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2: gsignal() 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 3: abort() 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5b027ffced] 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b027ffdd1] 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562dd907823d] 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562dd9063971] 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 8: (SnapRealm::check_cache() const+0x13e) [0x562dd9061fa0] 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562dd9062651] 2024-05-27T07:18:17.957 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562dd8ff7347] 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562dd8cf387e] 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562dd8cf8e94] 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562dd8d03969] 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 14: (MDLog::_replay_thread()+0x1f38) [0x562dd909c48c] 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562dd90a59cb] 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 16: (Thread::entry_wrapper()+0x43) [0x7f5b027ce72d] 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5b027ce749] 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5b013721ca] 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 19: clone() 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.812+0000 7f5aecc00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: in thread 7f5aecc00700 thread_name:md_log_replay 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.958 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:17.959 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5b0137cd20] 2024-05-27T07:18:17.959 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2: gsignal() 2024-05-27T07:18:17.959 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 3: abort() 2024-05-27T07:18:17.959 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5b027ffced] 2024-05-27T07:18:17.959 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b027ffdd1] 2024-05-27T07:18:17.959 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562dd907823d] 2024-05-27T07:18:17.959 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562dd9063971] 2024-05-27T07:18:17.959 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 8: (SnapRealm::check_cache() const+0x13e) [0x562dd9061fa0] 2024-05-27T07:18:17.959 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562dd9062651] 2024-05-27T07:18:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:17 smithi179 ceph-mon[35888]: pgmap v148: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 460 KiB/s wr, 43 op/s 2024-05-27T07:18:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:17 smithi179 ceph-mon[35888]: daemon mds.d restarted 2024-05-27T07:18:17.960 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562dd8ff7347] 2024-05-27T07:18:17.960 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562dd8cf387e] 2024-05-27T07:18:17.960 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562dd8cf8e94] 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562dd8d03969] 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 14: (MDLog::_replay_thread()+0x1f38) [0x562dd909c48c] 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562dd90a59cb] 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 16: (Thread::entry_wrapper()+0x43) [0x7f5b027ce72d] 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5b027ce749] 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5b013721ca] 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 19: clone() 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: -1> 2024-05-27T07:18:17.808+0000 7f5aecc00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5aecc00700 time 2024-05-27T07:18:17.805631+0000 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:17.961 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5b027ffba6] 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b027ffdd1] 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562dd907823d] 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562dd9063971] 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 5: (SnapRealm::check_cache() const+0x13e) [0x562dd9061fa0] 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562dd9062651] 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562dd8ff7347] 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562dd8cf387e] 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562dd8cf8e94] 2024-05-27T07:18:17.962 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562dd8d03969] 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 11: (MDLog::_replay_thread()+0x1f38) [0x562dd909c48c] 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562dd90a59cb] 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 13: (Thread::entry_wrapper()+0x43) [0x7f5b027ce72d] 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5b027ce749] 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5b013721ca] 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 16: clone() 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 0> 2024-05-27T07:18:17.812+0000 7f5aecc00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: in thread 7f5aecc00700 thread_name:md_log_replay 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5b0137cd20] 2024-05-27T07:18:17.963 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2: gsignal() 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 3: abort() 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5b027ffced] 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b027ffdd1] 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562dd907823d] 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562dd9063971] 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 8: (SnapRealm::check_cache() const+0x13e) [0x562dd9061fa0] 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562dd9062651] 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562dd8ff7347] 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562dd8cf387e] 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562dd8cf8e94] 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562dd8d03969] 2024-05-27T07:18:17.964 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 14: (MDLog::_replay_thread()+0x1f38) [0x562dd909c48c] 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562dd90a59cb] 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 16: (Thread::entry_wrapper()+0x43) [0x7f5b027ce72d] 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5b027ce749] 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5b013721ca] 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 19: clone() 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: -1> 2024-05-27T07:18:17.808+0000 7f5aecc00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f5aecc00700 time 2024-05-27T07:18:17.805631+0000 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.965 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5b027ffba6] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b027ffdd1] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562dd907823d] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562dd9063971] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 5: (SnapRealm::check_cache() const+0x13e) [0x562dd9061fa0] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562dd9062651] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562dd8ff7347] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562dd8cf387e] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562dd8cf8e94] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562dd8d03969] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 11: (MDLog::_replay_thread()+0x1f38) [0x562dd909c48c] 2024-05-27T07:18:17.966 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562dd90a59cb] 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 13: (Thread::entry_wrapper()+0x43) [0x7f5b027ce72d] 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 14: (Thread::_entry_func(void*)+0xd) [0x7f5b027ce749] 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5b013721ca] 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 16: clone() 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 0> 2024-05-27T07:18:17.812+0000 7f5aecc00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: in thread 7f5aecc00700 thread_name:md_log_replay 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5b0137cd20] 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2: gsignal() 2024-05-27T07:18:17.967 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 3: abort() 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5b027ffced] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5b027ffdd1] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562dd907823d] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562dd9063971] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 8: (SnapRealm::check_cache() const+0x13e) [0x562dd9061fa0] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562dd9062651] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562dd8ff7347] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562dd8cf387e] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562dd8cf8e94] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562dd8d03969] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 14: (MDLog::_replay_thread()+0x1f38) [0x562dd909c48c] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562dd90a59cb] 2024-05-27T07:18:17.968 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 16: (Thread::entry_wrapper()+0x43) [0x7f5b027ce72d] 2024-05-27T07:18:17.969 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 17: (Thread::_entry_func(void*)+0xd) [0x7f5b027ce749] 2024-05-27T07:18:17.969 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5b013721ca] 2024-05-27T07:18:17.969 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 19: clone() 2024-05-27T07:18:17.969 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:17.969 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100049]: 2024-05-27T07:18:18.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:17 smithi132 ceph-mon[35283]: pgmap v148: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 460 KiB/s wr, 43 op/s 2024-05-27T07:18:18.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:17 smithi132 ceph-mon[35283]: daemon mds.d restarted 2024-05-27T07:18:18.188 INFO:tasks.workunit.client.0.smithi043.stdout: 1 17850 0.81 MB/sec execute 22 sec latency 52.846 ms 2024-05-27T07:18:18.587 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:18 smithi043 podman[100159]: 2024-05-27 07:18:18.296421125 +0000 UTC m=+0.374079443 container died 40e9a696bd1f736da2644ec1607b6b1ecfb999dda5c589c6de1034c940a68901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-27T07:18:18.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:18 smithi043 ceph-mon[31531]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:18:18.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:18 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/3808824737,v1:172.21.15.43:6835/3808824737] up:boot 2024-05-27T07:18:18.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:18 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:18.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:18 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:18:18.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:18 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:18.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:18 smithi179 ceph-mon[35888]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:18:18.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:18 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/3808824737,v1:172.21.15.43:6835/3808824737] up:boot 2024-05-27T07:18:18.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:18 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:18.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:18 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:18:18.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:18 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:18.992 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:18 smithi132 ceph-mon[35283]: osdmap e106: 12 total, 12 up, 12 in 2024-05-27T07:18:18.992 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:18 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/3808824737,v1:172.21.15.43:6835/3808824737] up:boot 2024-05-27T07:18:18.992 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:18 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:18.992 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:18 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:18:18.992 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:18 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:19.189 INFO:tasks.workunit.client.0.smithi043.stdout: 1 17973 0.82 MB/sec execute 23 sec latency 50.872 ms 2024-05-27T07:18:19.359 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:19 smithi043 podman[100159]: 2024-05-27 07:18:19.041547248 +0000 UTC m=+1.119205590 container remove 40e9a696bd1f736da2644ec1607b6b1ecfb999dda5c589c6de1034c940a68901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T07:18:19.359 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:19 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:18:19.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.780+0000 7fa109e00700 1 -- 172.21.15.43:0/1324818925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa104075d20 msgr2=0x7fa104076130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:19.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.780+0000 7fa109e00700 1 --2- 172.21.15.43:0/1324818925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa104075d20 0x7fa104076130 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009990 tx=0x7fa0f4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.781+0000 7fa109e00700 1 -- 172.21.15.43:0/1324818925 shutdown_connections 2024-05-27T07:18:19.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.781+0000 7fa109e00700 1 --2- 172.21.15.43:0/1324818925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa104110b90 0x7fa104112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.781+0000 7fa109e00700 1 --2- 172.21.15.43:0/1324818925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa104075d20 0x7fa104076130 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.781+0000 7fa109e00700 1 --2- 172.21.15.43:0/1324818925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa104076670 0x7fa104110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.781+0000 7fa109e00700 1 -- 172.21.15.43:0/1324818925 >> 172.21.15.43:0/1324818925 conn(0x7fa104100030 msgr2=0x7fa104102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:19.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.782+0000 7fa109e00700 1 -- 172.21.15.43:0/1324818925 shutdown_connections 2024-05-27T07:18:19.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.782+0000 7fa109e00700 1 -- 172.21.15.43:0/1324818925 wait complete. 2024-05-27T07:18:19.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.783+0000 7fa109e00700 1 Processor -- start 2024-05-27T07:18:19.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.783+0000 7fa109e00700 1 -- start start 2024-05-27T07:18:19.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.784+0000 7fa109e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa104075d20 0x7fa1041042b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.784+0000 7fa109e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa104076670 0x7fa1041047f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.784+0000 7fa109e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa104110b90 0x7fa104104d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.784+0000 7fa109e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa1041165d0 con 0x7fa104076670 2024-05-27T07:18:19.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.784+0000 7fa109e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa104116450 con 0x7fa104075d20 2024-05-27T07:18:19.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.784+0000 7fa109e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa104116750 con 0x7fa104110b90 2024-05-27T07:18:19.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.785+0000 7fa102000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa104076670 0x7fa1041047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.785+0000 7fa102000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa104076670 0x7fa1041047f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57724/0 (socket says 172.21.15.43:57724) 2024-05-27T07:18:19.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.785+0000 7fa102000700 1 -- 172.21.15.43:0/302946879 learned_addr learned my addr 172.21.15.43:0/302946879 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:19.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.785+0000 7fa102a00700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa104075d20 0x7fa1041042b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.785+0000 7fa103400700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa104110b90 0x7fa104104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.786+0000 7fa102000700 1 -- 172.21.15.43:0/302946879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa104110b90 msgr2=0x7fa104104d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:19.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.786+0000 7fa102000700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa104110b90 0x7fa104104d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.786+0000 7fa102000700 1 -- 172.21.15.43:0/302946879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa104075d20 msgr2=0x7fa1041042b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:19.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.786+0000 7fa102000700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa104075d20 0x7fa1041042b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.786+0000 7fa102000700 1 -- 172.21.15.43:0/302946879 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1041ad560 con 0x7fa104076670 2024-05-27T07:18:19.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.786+0000 7fa102a00700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa104075d20 0x7fa1041042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:18:19.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.786+0000 7fa102000700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa104076670 0x7fa1041047f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000d950 tx=0x7fa0f000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:19.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.787+0000 7fa108a00700 1 -- 172.21.15.43:0/302946879 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0016070 con 0x7fa104076670 2024-05-27T07:18:19.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.787+0000 7fa108a00700 1 -- 172.21.15.43:0/302946879 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0f000fd20 con 0x7fa104076670 2024-05-27T07:18:19.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.787+0000 7fa108a00700 1 -- 172.21.15.43:0/302946879 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000e3b0 con 0x7fa104076670 2024-05-27T07:18:19.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.787+0000 7fa109e00700 1 -- 172.21.15.43:0/302946879 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1041ad7f0 con 0x7fa104076670 2024-05-27T07:18:19.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.788+0000 7fa109e00700 1 -- 172.21.15.43:0/302946879 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1041adce0 con 0x7fa104076670 2024-05-27T07:18:19.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.795+0000 7fa108a00700 1 -- 172.21.15.43:0/302946879 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa0f0019070 con 0x7fa104076670 2024-05-27T07:18:19.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.795+0000 7fa118963b80 1 Processor -- start 2024-05-27T07:18:19.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.795+0000 7fa118963b80 1 -- start start 2024-05-27T07:18:19.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.797+0000 7fa108a00700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0e80b2c00 0x7fa0e80b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.797+0000 7fa108a00700 1 -- 172.21.15.43:0/302946879 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9408+0+0 (secure 0 0 0) 0x7fa0f00e23a0 con 0x7fa104076670 2024-05-27T07:18:19.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.797+0000 7fa103400700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0e80b2c00 0x7fa0e80b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.797+0000 7fa118963b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 0x55e7e82393b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.797+0000 7fa118963b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e80baec0 0x55e7e82398f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.797+0000 7fa118963b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e8239ef0 0x55e7e8246190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.797+0000 7fa118963b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e7e8167ec0 con 0x55e7e8239ef0 2024-05-27T07:18:19.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.797+0000 7fa118963b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e7e8157a60 con 0x55e7e81a6a70 2024-05-27T07:18:19.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.797+0000 7fa118963b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e7e815baa0 con 0x55e7e80baec0 2024-05-27T07:18:19.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.798+0000 7fa102000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e80baec0 0x55e7e82398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.798+0000 7fa102000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e80baec0 0x55e7e82398f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:58688/0 (socket says 172.21.15.43:58688) 2024-05-27T07:18:19.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.798+0000 7fa102000700 1 -- 172.21.15.43:0/5331073 learned_addr learned my addr 172.21.15.43:0/5331073 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:19.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.798+0000 7fa102a00700 1 --2- 172.21.15.43:0/5331073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 0x55e7e82393b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.799+0000 7fa103400700 1 --2- 172.21.15.43:0/5331073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e8239ef0 0x55e7e8246190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.799+0000 7fa102000700 1 -- 172.21.15.43:0/5331073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 msgr2=0x55e7e82393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:19.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.799+0000 7fa102000700 1 --2- 172.21.15.43:0/5331073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 0x55e7e82393b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.799+0000 7fa102000700 1 -- 172.21.15.43:0/5331073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e8239ef0 msgr2=0x55e7e8246190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:19.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.799+0000 7fa102000700 1 --2- 172.21.15.43:0/5331073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e8239ef0 0x55e7e8246190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.800+0000 7fa102000700 1 -- 172.21.15.43:0/5331073 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7e81204a0 con 0x55e7e80baec0 2024-05-27T07:18:19.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.800+0000 7fa102a00700 1 --2- 172.21.15.43:0/5331073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 0x55e7e82393b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:18:19.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.800+0000 7fa103400700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0e80b2c00 0x7fa0e80b5110 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa104077bc0 tx=0x7fa0fc009bd0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:19.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.800+0000 7fa102000700 1 --2- 172.21.15.43:0/5331073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e80baec0 0x55e7e82398f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0005cf0 tx=0x7fa0f00a8d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7d2e79a2f7e1df9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:19.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.800+0000 7fa0f8c00700 1 -- 172.21.15.43:0/5331073 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0020040 con 0x55e7e80baec0 2024-05-27T07:18:19.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.801+0000 7fa0f8c00700 1 -- 172.21.15.43:0/5331073 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0f00a92d0 con 0x55e7e80baec0 2024-05-27T07:18:19.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.801+0000 7fa0f8c00700 1 -- 172.21.15.43:0/5331073 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f00b1ca0 con 0x55e7e80baec0 2024-05-27T07:18:19.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.802+0000 7fa118963b80 1 -- 172.21.15.43:0/5331073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e80baec0 msgr2=0x55e7e82398f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:19.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.802+0000 7fa118963b80 1 --2- 172.21.15.43:0/5331073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e80baec0 0x55e7e82398f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0005cf0 tx=0x7fa0f00a8d50 comp rx=0 tx=0).stop 2024-05-27T07:18:19.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.802+0000 7fa118963b80 1 -- 172.21.15.43:0/5331073 shutdown_connections 2024-05-27T07:18:19.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.802+0000 7fa118963b80 1 --2- 172.21.15.43:0/5331073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e8239ef0 0x55e7e8246190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.802+0000 7fa118963b80 1 --2- 172.21.15.43:0/5331073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 0x55e7e82393b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.802+0000 7fa118963b80 1 --2- 172.21.15.43:0/5331073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e80baec0 0x55e7e82398f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.802+0000 7fa118963b80 1 -- 172.21.15.43:0/5331073 >> 172.21.15.43:0/5331073 conn(0x55e7e8154410 msgr2=0x55e7e8235540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:19.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.802+0000 7fa118963b80 1 -- 172.21.15.43:0/5331073 shutdown_connections 2024-05-27T07:18:19.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.803+0000 7fa118963b80 1 -- 172.21.15.43:0/5331073 wait complete. 2024-05-27T07:18:19.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.803+0000 7fa118963b80 1 -- start start 2024-05-27T07:18:19.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.804+0000 7fa118963b80 1 Processor -- start 2024-05-27T07:18:19.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.805+0000 7fa118963b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e80baec0 0x55e7e8243700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.805+0000 7fa118963b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e81a6a70 0x55e7e8243c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.805+0000 7fa118963b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e8239ef0 0x55e7e8244180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.805+0000 7fa118963b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e7e805e020 con 0x55e7e81a6a70 2024-05-27T07:18:19.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.805+0000 7fa118963b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e7e80f87e0 con 0x55e7e80baec0 2024-05-27T07:18:19.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.805+0000 7fa118963b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e7e8246c00 con 0x55e7e8239ef0 2024-05-27T07:18:19.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.805+0000 7fa118963b80 10 client.0 _ll_register_callbacks cb 0x55e7e8101cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:19.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.805+0000 7fa102a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e80baec0 0x55e7e8243700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.805+0000 7fa102000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e81a6a70 0x55e7e8243c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.806+0000 7fa102a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e80baec0 0x55e7e8243700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:36868/0 (socket says 172.21.15.43:36868) 2024-05-27T07:18:19.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.806+0000 7fa102a00700 1 -- 172.21.15.43:0/3480211143 learned_addr learned my addr 172.21.15.43:0/3480211143 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:19.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.806+0000 7fa103400700 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e8239ef0 0x55e7e8244180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.806+0000 7fa102000700 1 -- 172.21.15.43:0/3480211143 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e8239ef0 msgr2=0x55e7e8244180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:19.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.806+0000 7fa102000700 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e8239ef0 0x55e7e8244180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.806+0000 7fa102000700 1 -- 172.21.15.43:0/3480211143 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e80baec0 msgr2=0x55e7e8243700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:19.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.806+0000 7fa102000700 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e80baec0 0x55e7e8243700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.806+0000 7fa102000700 1 -- 172.21.15.43:0/3480211143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7e80a7950 con 0x55e7e81a6a70 2024-05-27T07:18:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.807+0000 7fa102000700 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e81a6a70 0x55e7e8243c40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x55e7e82468d0 tx=0x7fa0f00108b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9048a948e3a3a9a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.807+0000 7fa0ed600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.807+0000 7fa0ed600700 1 -- 172.21.15.43:0/3480211143 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f00c8060 con 0x55e7e81a6a70 2024-05-27T07:18:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.807+0000 7fa118963b80 1 -- 172.21.15.43:0/3480211143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0f00a92d0 con 0x55e7e81a6a70 2024-05-27T07:18:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.807+0000 7fa0ed600700 1 -- 172.21.15.43:0/3480211143 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9408+0+0 (secure 0 0 0) 0x7fa0f0010460 con 0x55e7e81a6a70 2024-05-27T07:18:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.807+0000 7fa0ed600700 1 -- 172.21.15.43:0/3480211143 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fa0f00c2030 con 0x55e7e81a6a70 2024-05-27T07:18:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.808+0000 7fa118963b80 10 client.16146 fetch_fsmap learned FSMap version 64 2024-05-27T07:18:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.808+0000 7fa118963b80 1 -- 172.21.15.43:0/3480211143 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55e7e81280e0 con 0x55e7e81a6a70 2024-05-27T07:18:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.808+0000 7fa0ed600700 1 -- 172.21.15.43:0/3480211143 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0f00a92d0 con 0x55e7e81a6a70 2024-05-27T07:18:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.808+0000 7fa0ed600700 1 -- 172.21.15.43:0/3480211143 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 64) ==== 7168+0+0 (secure 0 0 0) 0x7fa0f00c3bc0 con 0x55e7e81a6a70 2024-05-27T07:18:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.809+0000 7fa118963b80 10 client.16146 fetch_fsmap finished waiting for FSMap version 64 2024-05-27T07:18:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.809+0000 7fa118963b80 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:18:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.809+0000 7fa118963b80 20 client.16146 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.809+0000 7fa118963b80 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e7e8253a20 0x55e7e8255e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.809+0000 7fa118963b80 4 client.16146 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:18:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.809+0000 7fa118963b80 1 -- 172.21.15.43:0/3480211143 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7e8169e10 con 0x55e7e8253a20 2024-05-27T07:18:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.810+0000 7fa102a00700 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e7e8253a20 0x55e7e8255e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.811+0000 7fa102a00700 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e7e8253a20 0x55e7e8255e40 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a37c89d37c89c686 server_cookie=68c43ac9b2cfce74 in_seq=0 out_seq=0 2024-05-27T07:18:19.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.811+0000 7fa0ed600700 10 client.16146 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:19.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.813+0000 7fa0ed600700 1 -- 172.21.15.43:0/3480211143 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e7e8169e10 con 0x55e7e8253a20 2024-05-27T07:18:19.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.813+0000 7fa0ed600700 10 client.16146 handle_command_reply: tid=0 2024-05-27T07:18:19.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.813+0000 7fa118963b80 1 client.16146 shutdown 2024-05-27T07:18:19.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.814+0000 7fa118963b80 1 -- 172.21.15.43:0/3480211143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e81a6a70 msgr2=0x55e7e8243c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:19.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.814+0000 7fa118963b80 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e81a6a70 0x55e7e8243c40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x55e7e82468d0 tx=0x7fa0f00108b0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.815+0000 7fa118963b80 1 -- 172.21.15.43:0/3480211143 shutdown_connections 2024-05-27T07:18:19.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.815+0000 7fa118963b80 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e8239ef0 0x55e7e8244180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.815+0000 7fa118963b80 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e80baec0 0x55e7e8243700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.815+0000 7fa118963b80 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e7e8253a20 0x55e7e8255e40 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.815+0000 7fa118963b80 1 --2- 172.21.15.43:0/3480211143 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e81a6a70 0x55e7e8243c40 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.815+0000 7fa118963b80 1 -- 172.21.15.43:0/3480211143 >> 172.21.15.43:0/3480211143 conn(0x55e7e8154410 msgr2=0x55e7e8239390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:19.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.815+0000 7fa0ed600700 0 client.16146 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:19.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.815+0000 7fa118963b80 1 -- 172.21.15.43:0/3480211143 shutdown_connections 2024-05-27T07:18:19.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.815+0000 7fa118963b80 1 -- 172.21.15.43:0/3480211143 wait complete. 2024-05-27T07:18:19.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.817+0000 7fa118963b80 20 client.16146 trim_cache size 0 max 16384 2024-05-27T07:18:19.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.847+0000 7fa118963b80 1 Processor -- start 2024-05-27T07:18:19.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.847+0000 7fa118963b80 1 -- start start 2024-05-27T07:18:19.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.847+0000 7fa118963b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e823d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.847+0000 7fa118963b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 0x55e7e824fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.847+0000 7fa118963b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e8239ef0 0x55e7e8243dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.847+0000 7fa118963b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa1041b5570 con 0x55e7e80baec0 2024-05-27T07:18:19.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.847+0000 7fa118963b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa1041b7f10 con 0x55e7e81a6a70 2024-05-27T07:18:19.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.847+0000 7fa118963b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa1041b5c40 con 0x55e7e8239ef0 2024-05-27T07:18:19.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.848+0000 7fa102a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e823d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.848+0000 7fa102a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e823d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57762/0 (socket says 172.21.15.43:57762) 2024-05-27T07:18:19.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.848+0000 7fa102a00700 1 -- 172.21.15.43:0/4227143932 learned_addr learned my addr 172.21.15.43:0/4227143932 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:19.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.848+0000 7fa103400700 1 --2- 172.21.15.43:0/4227143932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e8239ef0 0x55e7e8243dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.848+0000 7fa102000700 1 --2- 172.21.15.43:0/4227143932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 0x55e7e824fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.848+0000 7fa102a00700 1 -- 172.21.15.43:0/4227143932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e8239ef0 msgr2=0x55e7e8243dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:19.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.849+0000 7fa102a00700 1 --2- 172.21.15.43:0/4227143932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e8239ef0 0x55e7e8243dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.849+0000 7fa102a00700 1 -- 172.21.15.43:0/4227143932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 msgr2=0x55e7e824fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:19.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.849+0000 7fa102a00700 1 --2- 172.21.15.43:0/4227143932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 0x55e7e824fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.849+0000 7fa102a00700 1 -- 172.21.15.43:0/4227143932 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7e8236a40 con 0x55e7e80baec0 2024-05-27T07:18:19.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.849+0000 7fa102a00700 1 --2- 172.21.15.43:0/4227143932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e823d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40098e0 tx=0x7fa0f4005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378632c7b8c7c211 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:19.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.850+0000 7fa0efe00700 1 -- 172.21.15.43:0/4227143932 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40053f0 con 0x55e7e80baec0 2024-05-27T07:18:19.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.850+0000 7fa0efe00700 1 -- 172.21.15.43:0/4227143932 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0f4005590 con 0x55e7e80baec0 2024-05-27T07:18:19.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.850+0000 7fa0efe00700 1 -- 172.21.15.43:0/4227143932 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f400fa60 con 0x55e7e80baec0 2024-05-27T07:18:19.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.851+0000 7fa118963b80 1 -- 172.21.15.43:0/4227143932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 msgr2=0x55e7e823d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:19.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.851+0000 7fa118963b80 1 --2- 172.21.15.43:0/4227143932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e823d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40098e0 tx=0x7fa0f4005040 comp rx=0 tx=0).stop 2024-05-27T07:18:19.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.851+0000 7fa118963b80 1 -- 172.21.15.43:0/4227143932 shutdown_connections 2024-05-27T07:18:19.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.851+0000 7fa118963b80 1 --2- 172.21.15.43:0/4227143932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e8239ef0 0x55e7e8243dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.851+0000 7fa118963b80 1 --2- 172.21.15.43:0/4227143932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e823d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.851+0000 7fa118963b80 1 --2- 172.21.15.43:0/4227143932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e81a6a70 0x55e7e824fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.851+0000 7fa118963b80 1 -- 172.21.15.43:0/4227143932 >> 172.21.15.43:0/4227143932 conn(0x55e7e8154410 msgr2=0x55e7e82436c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:19.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.852+0000 7fa118963b80 1 -- 172.21.15.43:0/4227143932 shutdown_connections 2024-05-27T07:18:19.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.852+0000 7fa118963b80 1 -- 172.21.15.43:0/4227143932 wait complete. 2024-05-27T07:18:19.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.852+0000 7fa118963b80 1 -- start start 2024-05-27T07:18:19.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.853+0000 7fa118963b80 1 Processor -- start 2024-05-27T07:18:19.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa118963b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e8259250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa118963b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e81a6a70 0x55e7e8257630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa118963b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e8239ef0 0x55e7e8257b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa118963b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa1041b6310 con 0x55e7e80baec0 2024-05-27T07:18:19.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa118963b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa1041b7840 con 0x55e7e8239ef0 2024-05-27T07:18:19.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa118963b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa1041b69e0 con 0x55e7e81a6a70 2024-05-27T07:18:19.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa118963b80 10 client.0 _ll_register_callbacks cb 0x55e7e8101cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:19.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa102a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e8259250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa102a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e8259250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57772/0 (socket says 172.21.15.43:57772) 2024-05-27T07:18:19.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa102a00700 1 -- 172.21.15.43:0/2941107263 learned_addr learned my addr 172.21.15.43:0/2941107263 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:19.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.854+0000 7fa102000700 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e81a6a70 0x55e7e8257630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.855+0000 7fa103400700 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e8239ef0 0x55e7e8257b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.855+0000 7fa102a00700 1 -- 172.21.15.43:0/2941107263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e81a6a70 msgr2=0x55e7e8257630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:19.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.855+0000 7fa102a00700 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e81a6a70 0x55e7e8257630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.855+0000 7fa102a00700 1 -- 172.21.15.43:0/2941107263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e8239ef0 msgr2=0x55e7e8257b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:19.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.855+0000 7fa102a00700 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e8239ef0 0x55e7e8257b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.855+0000 7fa102a00700 1 -- 172.21.15.43:0/2941107263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7e82312c0 con 0x55e7e80baec0 2024-05-27T07:18:19.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.855+0000 7fa102a00700 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e8259250 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009350 tx=0x7fa0f40057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d2537943840a2e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:19.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.855+0000 7fa0ed600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:19.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.856+0000 7fa0ed600700 1 -- 172.21.15.43:0/2941107263 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4036070 con 0x55e7e80baec0 2024-05-27T07:18:19.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.856+0000 7fa0ed600700 1 -- 172.21.15.43:0/2941107263 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9408+0+0 (secure 0 0 0) 0x7fa0f4003950 con 0x55e7e80baec0 2024-05-27T07:18:19.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.856+0000 7fa118963b80 1 -- 172.21.15.43:0/2941107263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0f4005590 con 0x55e7e80baec0 2024-05-27T07:18:19.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.856+0000 7fa0ed600700 1 -- 172.21.15.43:0/2941107263 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fa0f4040050 con 0x55e7e80baec0 2024-05-27T07:18:19.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.856+0000 7fa118963b80 10 client.16152 fetch_fsmap learned FSMap version 64 2024-05-27T07:18:19.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.856+0000 7fa118963b80 1 -- 172.21.15.43:0/2941107263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x55e7e82bed60 con 0x55e7e80baec0 2024-05-27T07:18:19.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.857+0000 7fa0ed600700 1 -- 172.21.15.43:0/2941107263 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa0f4005590 con 0x55e7e80baec0 2024-05-27T07:18:19.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.857+0000 7fa0ed600700 1 -- 172.21.15.43:0/2941107263 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 64) ==== 7168+0+0 (secure 0 0 0) 0x7fa0f403b440 con 0x55e7e80baec0 2024-05-27T07:18:19.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.857+0000 7fa118963b80 10 client.16152 fetch_fsmap finished waiting for FSMap version 64 2024-05-27T07:18:19.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.857+0000 7fa118963b80 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:18:19.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.857+0000 7fa118963b80 20 client.16152 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:19.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.858+0000 7fa118963b80 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e7e82bf4c0 0x55e7e82c18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:19.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.858+0000 7fa118963b80 4 client.16152 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:18:19.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.858+0000 7fa118963b80 1 -- 172.21.15.43:0/2941107263 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e7e81635f0 con 0x55e7e82bf4c0 2024-05-27T07:18:19.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.859+0000 7fa102000700 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e7e82bf4c0 0x55e7e82c18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:19.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.860+0000 7fa102000700 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e7e82bf4c0 0x55e7e82c18e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc6d2961d3bef164 server_cookie=3cd469094726faf1 in_seq=0 out_seq=0 2024-05-27T07:18:19.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.860+0000 7fa0ed600700 10 client.16152 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:19.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.863+0000 7fa0ed600700 1 -- 172.21.15.43:0/2941107263 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e7e81635f0 con 0x55e7e82bf4c0 2024-05-27T07:18:19.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.863+0000 7fa0ed600700 10 client.16152 handle_command_reply: tid=0 2024-05-27T07:18:19.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.863+0000 7fa118963b80 1 client.16152 shutdown 2024-05-27T07:18:19.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.863+0000 7fa118963b80 1 -- 172.21.15.43:0/2941107263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 msgr2=0x55e7e8259250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:19.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.863+0000 7fa118963b80 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e8259250 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009350 tx=0x7fa0f40057d0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.864+0000 7fa118963b80 1 -- 172.21.15.43:0/2941107263 shutdown_connections 2024-05-27T07:18:19.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.864+0000 7fa118963b80 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7e8239ef0 0x55e7e8257b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.864+0000 7fa118963b80 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e7e82bf4c0 0x55e7e82c18e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.864+0000 7fa118963b80 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e7e80baec0 0x55e7e8259250 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.864+0000 7fa118963b80 1 --2- 172.21.15.43:0/2941107263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e7e81a6a70 0x55e7e8257630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.864+0000 7fa0ed600700 0 client.16152 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:19.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.864+0000 7fa118963b80 1 -- 172.21.15.43:0/2941107263 >> 172.21.15.43:0/2941107263 conn(0x55e7e8154410 msgr2=0x55e7e8245740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:19.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.864+0000 7fa118963b80 1 -- 172.21.15.43:0/2941107263 shutdown_connections 2024-05-27T07:18:19.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.865+0000 7fa118963b80 1 -- 172.21.15.43:0/2941107263 wait complete. 2024-05-27T07:18:19.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.865+0000 7fa118963b80 20 client.16152 trim_cache size 0 max 16384 2024-05-27T07:18:19.856 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:18:19.856 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:18:19.856 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "caf3e5c3-5d19-4858-8737-82df5624fc47", 2024-05-27T07:18:19.856 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:18:19.856 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:18:19.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.866+0000 7fa109e00700 1 -- 172.21.15.43:0/302946879 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0e80b2c00 msgr2=0x7fa0e80b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:19.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.866+0000 7fa109e00700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0e80b2c00 0x7fa0e80b5110 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa104077bc0 tx=0x7fa0fc009bd0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.866+0000 7fa109e00700 1 -- 172.21.15.43:0/302946879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa104076670 msgr2=0x7fa1041047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:19.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.866+0000 7fa109e00700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa104076670 0x7fa1041047f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000d950 tx=0x7fa0f000dc60 comp rx=0 tx=0).stop 2024-05-27T07:18:19.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.866+0000 7fa109e00700 1 -- 172.21.15.43:0/302946879 shutdown_connections 2024-05-27T07:18:19.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.866+0000 7fa109e00700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0e80b2c00 0x7fa0e80b5110 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.866+0000 7fa109e00700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa104110b90 0x7fa104104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.866+0000 7fa109e00700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa104075d20 0x7fa1041042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.866+0000 7fa109e00700 1 --2- 172.21.15.43:0/302946879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa104076670 0x7fa1041047f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:19.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.866+0000 7fa109e00700 1 -- 172.21.15.43:0/302946879 >> 172.21.15.43:0/302946879 conn(0x7fa104100030 msgr2=0x7fa1041006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:19.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.867+0000 7fa109e00700 1 -- 172.21.15.43:0/302946879 shutdown_connections 2024-05-27T07:18:19.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:19.867+0000 7fa109e00700 1 -- 172.21.15.43:0/302946879 wait complete. 2024-05-27T07:18:19.952 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:19 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:18:19.952 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:19 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.614s CPU time. 2024-05-27T07:18:19.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:19 smithi043 ceph-mon[31531]: pgmap v150: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 278 KiB/s wr, 26 op/s 2024-05-27T07:18:19.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:19 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:19.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:19 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:18:19.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:19 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:18:19.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:19 smithi179 ceph-mon[35888]: pgmap v150: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 278 KiB/s wr, 26 op/s 2024-05-27T07:18:19.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:19 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:19.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:19 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:18:19.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:19 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:18:20.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:19 smithi132 ceph-mon[35283]: pgmap v150: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 278 KiB/s wr, 26 op/s 2024-05-27T07:18:20.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:19 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:18:20.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:19 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:18:20.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:19 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:18:20.188 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18049 0.78 MB/sec execute 24 sec latency 40.734 ms 2024-05-27T07:18:20.925 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:20 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:18:20.925 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:20 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:20.926 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:20 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:18:20.926 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:20 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:18:20.926 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:20 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:18:20.926 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:20 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:20.926 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:20 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:20.926 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:20 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:18:20.926 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:20 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:20.926 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:20 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:20.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:18:20.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:20 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:18:20.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:20 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:20.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:20 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:18:20.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:20 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:18:20.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:20 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:18:20.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:20 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:20 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:20 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:18:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:20 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:20.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:20 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:21.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:20 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:18:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:20 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:20 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:18:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:20 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:18:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:20 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:18:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:20 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:20 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:20 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:18:21.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:20 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:21.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:20 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:21.188 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18122 0.76 MB/sec execute 25 sec latency 48.005 ms 2024-05-27T07:18:21.429 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:21.917 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:21 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:18:21.918 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:21 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:21.918 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:21 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:18:21.918 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:21 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:21.918 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:21 smithi043 ceph-mon[31531]: pgmap v151: 129 pgs: 129 active+clean; 41 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 509 KiB/s wr, 48 op/s 2024-05-27T07:18:21.918 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:21 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:21.918 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:21 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:21.918 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:21 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:18:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:21 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:18:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:21 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:21 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:18:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:21 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:21.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:21 smithi179 ceph-mon[35888]: pgmap v151: 129 pgs: 129 active+clean; 41 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 509 KiB/s wr, 48 op/s 2024-05-27T07:18:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:21 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:21 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:21 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:18:22.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:21 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:18:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:21 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:21 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:18:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:21 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:18:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:21 smithi132 ceph-mon[35283]: pgmap v151: 129 pgs: 129 active+clean; 41 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 509 KiB/s wr, 48 op/s 2024-05-27T07:18:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:21 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:18:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:21 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:18:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:21 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:18:22.188 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18185 0.73 MB/sec execute 26 sec latency 38.883 ms 2024-05-27T07:18:23.188 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18246 0.71 MB/sec execute 27 sec latency 58.823 ms 2024-05-27T07:18:23.479 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:23 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:18:23.479 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:23 smithi043 systemd[1]: Stopped Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:23.479 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:23 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.495s CPU time. 2024-05-27T07:18:23.479 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:23 smithi043 systemd[1]: Starting Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:18:23.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.905+0000 7fee03c00700 1 -- 172.21.15.43:0/2791964502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fedfc109860 msgr2=0x7fedfc1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:23.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.905+0000 7fee03c00700 1 --2- 172.21.15.43:0/2791964502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fedfc109860 0x7fedfc1111c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fedf400b0d0 tx=0x7fedf400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.905+0000 7fee03c00700 1 -- 172.21.15.43:0/2791964502 shutdown_connections 2024-05-27T07:18:23.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.905+0000 7fee03c00700 1 --2- 172.21.15.43:0/2791964502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fedfc109860 0x7fedfc1111c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.905+0000 7fee03c00700 1 --2- 172.21.15.43:0/2791964502 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedfc1045e0 0x7fedfc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.905+0000 7fee03c00700 1 --2- 172.21.15.43:0/2791964502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedfc108f10 0x7fedfc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.905+0000 7fee03c00700 1 -- 172.21.15.43:0/2791964502 >> 172.21.15.43:0/2791964502 conn(0x7fedfc100030 msgr2=0x7fedfc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:23.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.905+0000 7fee03c00700 1 -- 172.21.15.43:0/2791964502 shutdown_connections 2024-05-27T07:18:23.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.905+0000 7fee03c00700 1 -- 172.21.15.43:0/2791964502 wait complete. 2024-05-27T07:18:23.898 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:23 smithi043 ceph-mon[31531]: pgmap v152: 129 pgs: 129 active+clean; 41 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 505 KiB/s wr, 48 op/s 2024-05-27T07:18:23.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.907+0000 7fee03c00700 1 Processor -- start 2024-05-27T07:18:23.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.907+0000 7fee03c00700 1 -- start start 2024-05-27T07:18:23.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.907+0000 7fee03c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedfc1045e0 0x7fedfc19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.908+0000 7fee03c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fedfc108f10 0x7fedfc19d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.908+0000 7fee03c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedfc109860 0x7fedfc1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.908+0000 7fee03c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc114390 con 0x7fedfc108f10 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.908+0000 7fee03c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc114210 con 0x7fedfc1045e0 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.908+0000 7fee03c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc114510 con 0x7fedfc109860 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.908+0000 7fedfbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fedfc108f10 0x7fedfc19d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.908+0000 7fee01200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedfc1045e0 0x7fedfc19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.908+0000 7fedfbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fedfc108f10 0x7fedfc19d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57798/0 (socket says 172.21.15.43:57798) 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.908+0000 7fedfbe00700 1 -- 172.21.15.43:0/1196521109 learned_addr learned my addr 172.21.15.43:0/1196521109 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.909+0000 7fee01c00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedfc109860 0x7fedfc1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.909+0000 7fedfbe00700 1 -- 172.21.15.43:0/1196521109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedfc109860 msgr2=0x7fedfc1a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.909+0000 7fedfbe00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedfc109860 0x7fedfc1a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.909+0000 7fedfbe00700 1 -- 172.21.15.43:0/1196521109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedfc1045e0 msgr2=0x7fedfc19cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.909+0000 7fedfbe00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedfc1045e0 0x7fedfc19cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.909+0000 7fedfbe00700 1 -- 172.21.15.43:0/1196521109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedfc1a1e50 con 0x7fedfc108f10 2024-05-27T07:18:23.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.909+0000 7fedfbe00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fedfc108f10 0x7fedfc19d520 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fede800d980 tx=0x7fede800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:23.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.910+0000 7fedf9600700 1 -- 172.21.15.43:0/1196521109 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8017070 con 0x7fedfc108f10 2024-05-27T07:18:23.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.910+0000 7fedf9600700 1 -- 172.21.15.43:0/1196521109 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fede8009dc0 con 0x7fedfc108f10 2024-05-27T07:18:23.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.910+0000 7fedf9600700 1 -- 172.21.15.43:0/1196521109 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede800f3c0 con 0x7fedfc108f10 2024-05-27T07:18:23.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.910+0000 7fee03c00700 1 -- 172.21.15.43:0/1196521109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedfc1a2140 con 0x7fedfc108f10 2024-05-27T07:18:23.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.910+0000 7fee03c00700 1 -- 172.21.15.43:0/1196521109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedfc1a9a60 con 0x7fedfc108f10 2024-05-27T07:18:23.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.914+0000 7fedf9600700 1 -- 172.21.15.43:0/1196521109 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fede8009930 con 0x7fedfc108f10 2024-05-27T07:18:23.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.916+0000 7fee1280db80 1 Processor -- start 2024-05-27T07:18:23.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.916+0000 7fee1280db80 1 -- start start 2024-05-27T07:18:23.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.916+0000 7fee1280db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f8900a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.916+0000 7fee1280db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f89004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fee1280db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f89004db0 0x562f890051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fee1280db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562f88f32e40 con 0x562f88e85e40 2024-05-27T07:18:23.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fee1280db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562f88f229e0 con 0x562f8900ad30 2024-05-27T07:18:23.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fee1280db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562f88f26a20 con 0x562f89004db0 2024-05-27T07:18:23.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fedf9600700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fede40b2a70 0x7fede40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fee01c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f8900a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fee01c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f8900a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57800/0 (socket says 172.21.15.43:57800) 2024-05-27T07:18:23.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fee01c00700 1 -- 172.21.15.43:0/321321059 learned_addr learned my addr 172.21.15.43:0/321321059 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:23.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fedf9600700 1 -- 172.21.15.43:0/1196521109 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9408+0+0 (secure 0 0 0) 0x7fede80e2400 con 0x7fedfc108f10 2024-05-27T07:18:23.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fee01200700 1 --2- 172.21.15.43:0/321321059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f89004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.917+0000 7fedfbe00700 1 --2- 172.21.15.43:0/321321059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f89004db0 0x562f890051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.918+0000 7fee01c00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fede40b2a70 0x7fede40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.918+0000 7fee01c00700 1 -- 172.21.15.43:0/321321059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f89004db0 msgr2=0x562f890051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:23.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.918+0000 7fee01c00700 1 --2- 172.21.15.43:0/321321059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f89004db0 0x562f890051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.918+0000 7fee01c00700 1 -- 172.21.15.43:0/321321059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 msgr2=0x562f89004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:23.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.918+0000 7fee01c00700 1 --2- 172.21.15.43:0/321321059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f89004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.918+0000 7fee01c00700 1 -- 172.21.15.43:0/321321059 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f88eeb420 con 0x562f88e85e40 2024-05-27T07:18:23.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.918+0000 7fee01c00700 1 --2- 172.21.15.43:0/321321059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f8900a7f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009e60 tx=0x7fedf40090d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd4f35a7815dac6a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:23.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.919+0000 7fedee000700 1 -- 172.21.15.43:0/321321059 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4004c50 con 0x562f88e85e40 2024-05-27T07:18:23.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.919+0000 7fedee000700 1 -- 172.21.15.43:0/321321059 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fedf4004df0 con 0x562f88e85e40 2024-05-27T07:18:23.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.919+0000 7fedee000700 1 -- 172.21.15.43:0/321321059 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf401f440 con 0x562f88e85e40 2024-05-27T07:18:23.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.919+0000 7fee01c00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fede40b2a70 0x7fede40b4f80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fedf402a000 tx=0x7fedf4028000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:23.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.920+0000 7fee1280db80 1 -- 172.21.15.43:0/321321059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 msgr2=0x562f8900a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:23.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.920+0000 7fee1280db80 1 --2- 172.21.15.43:0/321321059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f8900a7f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009e60 tx=0x7fedf40090d0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.920+0000 7fee1280db80 1 -- 172.21.15.43:0/321321059 shutdown_connections 2024-05-27T07:18:23.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.920+0000 7fee1280db80 1 --2- 172.21.15.43:0/321321059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f89004db0 0x562f890051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.920+0000 7fee1280db80 1 --2- 172.21.15.43:0/321321059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f8900a7f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.920+0000 7fee1280db80 1 --2- 172.21.15.43:0/321321059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f89004870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.920+0000 7fee1280db80 1 -- 172.21.15.43:0/321321059 >> 172.21.15.43:0/321321059 conn(0x562f88ee5fc0 msgr2=0x562f88fff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:23.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.920+0000 7fee1280db80 1 -- 172.21.15.43:0/321321059 shutdown_connections 2024-05-27T07:18:23.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.920+0000 7fee1280db80 1 -- 172.21.15.43:0/321321059 wait complete. 2024-05-27T07:18:23.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.921+0000 7fee1280db80 1 -- start start 2024-05-27T07:18:23.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.921+0000 7fee1280db80 1 Processor -- start 2024-05-27T07:18:23.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.922+0000 7fee1280db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f88e85e40 0x562f8900e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.922+0000 7fee1280db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f8900eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.922+0000 7fee1280db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f8900ad30 0x562f8900f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.922+0000 7fee1280db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562f88e28fa0 con 0x562f89004db0 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.922+0000 7fee1280db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562f88ec3760 con 0x562f88e85e40 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.922+0000 7fee1280db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562f89011800 con 0x562f8900ad30 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.922+0000 7fee1280db80 10 client.0 _ll_register_callbacks cb 0x562f88eccc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.922+0000 7fedfbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f8900eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.922+0000 7fedfbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f8900eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57810/0 (socket says 172.21.15.43:57810) 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.922+0000 7fedfbe00700 1 -- 172.21.15.43:0/3822762163 learned_addr learned my addr 172.21.15.43:0/3822762163 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.923+0000 7fee01200700 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f88e85e40 0x562f8900e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.923+0000 7fedfbe00700 1 -- 172.21.15.43:0/3822762163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f8900ad30 msgr2=0x562f8900f000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.923+0000 7fedfbe00700 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f8900ad30 0x562f8900f000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.923+0000 7fedfbe00700 1 -- 172.21.15.43:0/3822762163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f88e85e40 msgr2=0x562f8900e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:23.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.923+0000 7fedfbe00700 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f88e85e40 0x562f8900e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.923+0000 7fedfbe00700 1 -- 172.21.15.43:0/3822762163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f88e728d0 con 0x562f89004db0 2024-05-27T07:18:23.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.923+0000 7fedfbe00700 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f8900eac0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fede800d950 tx=0x7fede80aed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07facff3a723720 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:23.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.923+0000 7fede2a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:23.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.924+0000 7fede2a00700 1 -- 172.21.15.43:0/3822762163 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8020040 con 0x562f89004db0 2024-05-27T07:18:23.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.924+0000 7fede2a00700 1 -- 172.21.15.43:0/3822762163 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9408+0+0 (secure 0 0 0) 0x7fede80a5920 con 0x562f89004db0 2024-05-27T07:18:23.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.924+0000 7fee1280db80 1 -- 172.21.15.43:0/3822762163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedf4004df0 con 0x562f89004db0 2024-05-27T07:18:23.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.924+0000 7fede2a00700 1 -- 172.21.15.43:0/3822762163 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fede80a8e40 con 0x562f89004db0 2024-05-27T07:18:23.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.925+0000 7fee1280db80 10 client.16164 fetch_fsmap learned FSMap version 64 2024-05-27T07:18:23.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.925+0000 7fee1280db80 1 -- 172.21.15.43:0/3822762163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x562f88ef3060 con 0x562f89004db0 2024-05-27T07:18:23.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.926+0000 7fede2a00700 1 -- 172.21.15.43:0/3822762163 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fedf4004df0 con 0x562f89004db0 2024-05-27T07:18:23.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.926+0000 7fede2a00700 1 -- 172.21.15.43:0/3822762163 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 64) ==== 7168+0+0 (secure 0 0 0) 0x7fede80b1d00 con 0x562f89004db0 2024-05-27T07:18:23.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.926+0000 7fee1280db80 10 client.16164 fetch_fsmap finished waiting for FSMap version 64 2024-05-27T07:18:23.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.926+0000 7fee1280db80 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:18:23.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.926+0000 7fee1280db80 20 client.16164 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.926+0000 7fee1280db80 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562f8901e790 0x562f89020be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.926+0000 7fee1280db80 4 client.16164 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:18:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.926+0000 7fee1280db80 1 -- 172.21.15.43:0/3822762163 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f88f34d90 con 0x562f8901e790 2024-05-27T07:18:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.927+0000 7fee01200700 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562f8901e790 0x562f89020be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.928+0000 7fee01200700 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562f8901e790 0x562f89020be0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2215f684177f746b server_cookie=bb18307bf5d044f2 in_seq=0 out_seq=0 2024-05-27T07:18:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.928+0000 7fede2a00700 10 client.16164 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fede2a00700 1 -- 172.21.15.43:0/3822762163 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562f88f34d90 con 0x562f8901e790 2024-05-27T07:18:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fede2a00700 10 client.16164 handle_command_reply: tid=0 2024-05-27T07:18:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fee1280db80 1 client.16164 shutdown 2024-05-27T07:18:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fee1280db80 1 -- 172.21.15.43:0/3822762163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 msgr2=0x562f8900eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fee1280db80 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f8900eac0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fede800d950 tx=0x7fede80aed60 comp rx=0 tx=0).stop 2024-05-27T07:18:23.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fee1280db80 1 -- 172.21.15.43:0/3822762163 shutdown_connections 2024-05-27T07:18:23.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fee1280db80 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f8900ad30 0x562f8900f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fee1280db80 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f88e85e40 0x562f8900e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fee1280db80 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562f8901e790 0x562f89020be0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fee1280db80 1 --2- 172.21.15.43:0/3822762163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f8900eac0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fee1280db80 1 -- 172.21.15.43:0/3822762163 >> 172.21.15.43:0/3822762163 conn(0x562f88ee5fc0 msgr2=0x562f88fff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:23.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fede2a00700 0 client.16164 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:23.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.930+0000 7fee1280db80 1 -- 172.21.15.43:0/3822762163 shutdown_connections 2024-05-27T07:18:23.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.931+0000 7fee1280db80 1 -- 172.21.15.43:0/3822762163 wait complete. 2024-05-27T07:18:23.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.931+0000 7fee1280db80 20 client.16164 trim_cache size 0 max 16384 2024-05-27T07:18:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.955+0000 7fee1280db80 1 Processor -- start 2024-05-27T07:18:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.955+0000 7fee1280db80 1 -- start start 2024-05-27T07:18:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.956+0000 7fee1280db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f89004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.956+0000 7fee1280db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f89004db0 0x562f89010a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.956+0000 7fee1280db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f8900ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.956+0000 7fee1280db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc1b1170 con 0x562f88e85e40 2024-05-27T07:18:23.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.956+0000 7fee1280db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc1b3b10 con 0x562f8900ad30 2024-05-27T07:18:23.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.956+0000 7fee1280db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc1b1840 con 0x562f89004db0 2024-05-27T07:18:23.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.956+0000 7fee01200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f89004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.956+0000 7fee01200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f89004870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57818/0 (socket says 172.21.15.43:57818) 2024-05-27T07:18:23.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.956+0000 7fee01200700 1 -- 172.21.15.43:0/2823866257 learned_addr learned my addr 172.21.15.43:0/2823866257 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:23.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.957+0000 7fedfbe00700 1 --2- 172.21.15.43:0/2823866257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f89004db0 0x562f89010a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.957+0000 7fee01c00700 1 --2- 172.21.15.43:0/2823866257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f8900ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.957+0000 7fee01200700 1 -- 172.21.15.43:0/2823866257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f89004db0 msgr2=0x562f89010a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:23.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.957+0000 7fee01200700 1 --2- 172.21.15.43:0/2823866257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f89004db0 0x562f89010a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.957+0000 7fee01200700 1 -- 172.21.15.43:0/2823866257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 msgr2=0x562f8900ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:23.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.957+0000 7fee01200700 1 --2- 172.21.15.43:0/2823866257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f8900ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.957+0000 7fee01200700 1 -- 172.21.15.43:0/2823866257 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f890051c0 con 0x562f88e85e40 2024-05-27T07:18:23.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.957+0000 7fee01200700 1 --2- 172.21.15.43:0/2823866257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f89004870 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x562f89021310 tx=0x7fedf000bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1920486ec88d54c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:23.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.957+0000 7feded600700 1 -- 172.21.15.43:0/2823866257 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf0003be0 con 0x562f88e85e40 2024-05-27T07:18:23.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.958+0000 7feded600700 1 -- 172.21.15.43:0/2823866257 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fedf0003d80 con 0x562f88e85e40 2024-05-27T07:18:23.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.958+0000 7feded600700 1 -- 172.21.15.43:0/2823866257 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf00056d0 con 0x562f88e85e40 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.958+0000 7fee1280db80 1 -- 172.21.15.43:0/2823866257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 msgr2=0x562f89004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.958+0000 7fee1280db80 1 --2- 172.21.15.43:0/2823866257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f89004870 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x562f89021310 tx=0x7fedf000bde0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.959+0000 7fee1280db80 1 -- 172.21.15.43:0/2823866257 shutdown_connections 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.959+0000 7fee1280db80 1 --2- 172.21.15.43:0/2823866257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f8900ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.959+0000 7fee1280db80 1 --2- 172.21.15.43:0/2823866257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f88e85e40 0x562f89004870 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.959+0000 7fee1280db80 1 --2- 172.21.15.43:0/2823866257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f89004db0 0x562f89010a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.959+0000 7fee1280db80 1 -- 172.21.15.43:0/2823866257 >> 172.21.15.43:0/2823866257 conn(0x562f88ee5fc0 msgr2=0x562f8900e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.959+0000 7fee1280db80 1 -- 172.21.15.43:0/2823866257 shutdown_connections 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.959+0000 7fee1280db80 1 -- 172.21.15.43:0/2823866257 wait complete. 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.959+0000 7fee1280db80 1 -- start start 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.960+0000 7fee1280db80 1 Processor -- start 2024-05-27T07:18:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.960+0000 7fee1280db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f88e85e40 0x562f89017110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.960+0000 7fee1280db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f89013a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.960+0000 7fee1280db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f89013fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.960+0000 7fee1280db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc1b1f10 con 0x562f89004db0 2024-05-27T07:18:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.960+0000 7fee1280db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc1b3440 con 0x562f8900ad30 2024-05-27T07:18:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.960+0000 7fee1280db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc1b25e0 con 0x562f88e85e40 2024-05-27T07:18:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.960+0000 7fee1280db80 10 client.0 _ll_register_callbacks cb 0x562f88eccc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.961+0000 7fedfbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f89013a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.961+0000 7fedfbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f89013a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57822/0 (socket says 172.21.15.43:57822) 2024-05-27T07:18:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.961+0000 7fedfbe00700 1 -- 172.21.15.43:0/907388233 learned_addr learned my addr 172.21.15.43:0/907388233 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.961+0000 7fee01200700 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f88e85e40 0x562f89017110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.961+0000 7fee01c00700 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f89013fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.961+0000 7fedfbe00700 1 -- 172.21.15.43:0/907388233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f88e85e40 msgr2=0x562f89017110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.961+0000 7fedfbe00700 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f88e85e40 0x562f89017110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.961+0000 7fedfbe00700 1 -- 172.21.15.43:0/907388233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 msgr2=0x562f89013fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.961+0000 7fedfbe00700 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f89013fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.961+0000 7fedfbe00700 1 -- 172.21.15.43:0/907388233 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f890146a0 con 0x562f89004db0 2024-05-27T07:18:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.962+0000 7fedfbe00700 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f89013a60 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fede80a92c0 tx=0x7fede80a9180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a57f40e9f519c9b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.962+0000 7fede2a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.962+0000 7fede2a00700 1 -- 172.21.15.43:0/907388233 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8020070 con 0x562f89004db0 2024-05-27T07:18:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.962+0000 7fede2a00700 1 -- 172.21.15.43:0/907388233 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9408+0+0 (secure 0 0 0) 0x7fede8010460 con 0x562f89004db0 2024-05-27T07:18:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.962+0000 7fee1280db80 1 -- 172.21.15.43:0/907388233 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedf0003d80 con 0x562f89004db0 2024-05-27T07:18:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.962+0000 7fede2a00700 1 -- 172.21.15.43:0/907388233 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fede80a9a30 con 0x562f89004db0 2024-05-27T07:18:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.962+0000 7fee1280db80 10 client.16170 fetch_fsmap learned FSMap version 64 2024-05-27T07:18:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.962+0000 7fee1280db80 1 -- 172.21.15.43:0/907388233 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=4+}) -- 0x562f8908a120 con 0x562f89004db0 2024-05-27T07:18:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.963+0000 7fede2a00700 1 -- 172.21.15.43:0/907388233 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fedf0003d80 con 0x562f89004db0 2024-05-27T07:18:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.963+0000 7fede2a00700 1 -- 172.21.15.43:0/907388233 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 64) ==== 7168+0+0 (secure 0 0 0) 0x7fede80a9cc0 con 0x562f89004db0 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.963+0000 7fee1280db80 10 client.16170 fetch_fsmap finished waiting for FSMap version 64 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.963+0000 7fee1280db80 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.963+0000 7fee1280db80 20 client.16170 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.963+0000 7fee1280db80 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562f8908a740 0x562f8908cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.963+0000 7fee1280db80 4 client.16170 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.963+0000 7fee1280db80 1 -- 172.21.15.43:0/907388233 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562f88f2e570 con 0x562f8908a740 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.964+0000 7fee01200700 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562f8908a740 0x562f8908cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.965+0000 7fee01200700 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562f8908a740 0x562f8908cb60 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6ce76a55e6772fc server_cookie=10f5ba1ba544317f in_seq=0 out_seq=0 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.965+0000 7fede2a00700 10 client.16170 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fede2a00700 1 -- 172.21.15.43:0/907388233 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562f88f2e570 con 0x562f8908a740 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fede2a00700 10 client.16170 handle_command_reply: tid=0 2024-05-27T07:18:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 client.16170 shutdown 2024-05-27T07:18:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 -- 172.21.15.43:0/907388233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 msgr2=0x562f89013a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f89013a60 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fede80a92c0 tx=0x7fede80a9180 comp rx=0 tx=0).stop 2024-05-27T07:18:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 -- 172.21.15.43:0/907388233 shutdown_connections 2024-05-27T07:18:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562f8900ad30 0x562f89013fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562f88e85e40 0x562f89017110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562f8908a740 0x562f8908cb60 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 --2- 172.21.15.43:0/907388233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562f89004db0 0x562f89013a60 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 -- 172.21.15.43:0/907388233 >> 172.21.15.43:0/907388233 conn(0x562f88ee5fc0 msgr2=0x562f89009e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fede2a00700 0 client.16170 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 -- 172.21.15.43:0/907388233 shutdown_connections 2024-05-27T07:18:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.966+0000 7fee1280db80 1 -- 172.21.15.43:0/907388233 wait complete. 2024-05-27T07:18:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.967+0000 7fee1280db80 20 client.16170 trim_cache size 0 max 16384 2024-05-27T07:18:23.959 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:18:23.959 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:18:23.959 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:18:23.959 INFO:teuthology.orchestra.run.smithi043.stdout: "caf3e5c3-5d19-4858-8737-82df5624fc47": { 2024-05-27T07:18:23.959 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:18:23.959 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "caf3e5c3-5d19-4858-8737-82df5624fc47", 2024-05-27T07:18:23.959 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:18:23.959 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:18:23.959 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:18:23.959 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:18:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.967+0000 7fee03c00700 1 -- 172.21.15.43:0/1196521109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fede40b2a70 msgr2=0x7fede40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.967+0000 7fee03c00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fede40b2a70 0x7fede40b4f80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fedf402a000 tx=0x7fedf4028000 comp rx=0 tx=0).stop 2024-05-27T07:18:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.967+0000 7fee03c00700 1 -- 172.21.15.43:0/1196521109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fedfc108f10 msgr2=0x7fedfc19d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.967+0000 7fee03c00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fedfc108f10 0x7fedfc19d520 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fede800d980 tx=0x7fede800dc90 comp rx=0 tx=0).stop 2024-05-27T07:18:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.968+0000 7fee03c00700 1 -- 172.21.15.43:0/1196521109 shutdown_connections 2024-05-27T07:18:23.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.968+0000 7fee03c00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fede40b2a70 0x7fede40b4f80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.968+0000 7fee03c00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedfc109860 0x7fedfc1a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.968+0000 7fee03c00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fedfc1045e0 0x7fedfc19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.968+0000 7fee03c00700 1 --2- 172.21.15.43:0/1196521109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fedfc108f10 0x7fedfc19d520 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:23.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.968+0000 7fee03c00700 1 -- 172.21.15.43:0/1196521109 >> 172.21.15.43:0/1196521109 conn(0x7fedfc100030 msgr2=0x7fedfc110e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:23.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.968+0000 7fee03c00700 1 -- 172.21.15.43:0/1196521109 shutdown_connections 2024-05-27T07:18:23.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:23.968+0000 7fee03c00700 1 -- 172.21.15.43:0/1196521109 wait complete. 2024-05-27T07:18:23.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:23 smithi179 ceph-mon[35888]: pgmap v152: 129 pgs: 129 active+clean; 41 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 505 KiB/s wr, 48 op/s 2024-05-27T07:18:24.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:23 smithi132 ceph-mon[35283]: pgmap v152: 129 pgs: 129 active+clean; 41 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 505 KiB/s wr, 48 op/s 2024-05-27T07:18:24.188 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18344 0.68 MB/sec execute 28 sec latency 42.887 ms 2024-05-27T07:18:24.203 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:23 smithi043 podman[100670]: 2024-05-27 07:18:23.879205185 +0000 UTC m=+0.046687533 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:18:24.203 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:24 smithi043 podman[100670]: 2024-05-27 07:18:24.003803069 +0000 UTC m=+0.171285495 container create 4b55efc4bbddd38006737ebec1738a1bb95af9e6bc1f4e5a4f71396cd3e75c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:18:24.952 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:24 smithi043 podman[100670]: 2024-05-27 07:18:24.612225593 +0000 UTC m=+0.779708036 container init 4b55efc4bbddd38006737ebec1738a1bb95af9e6bc1f4e5a4f71396cd3e75c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:18:24.953 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:24 smithi043 podman[100670]: 2024-05-27 07:18:24.625410439 +0000 UTC m=+0.792892794 container start 4b55efc4bbddd38006737ebec1738a1bb95af9e6bc1f4e5a4f71396cd3e75c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T07:18:24.953 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:24 smithi043 bash[100670]: 4b55efc4bbddd38006737ebec1738a1bb95af9e6bc1f4e5a4f71396cd3e75c68 2024-05-27T07:18:24.953 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:24 smithi043 systemd[1]: Started Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:24.953 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:24 smithi043 ceph-mds[100753]: -- 172.21.15.43:0/1192057746 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b203fda40 con 0x559b20445a80 2024-05-27T07:18:24.953 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: starting mds.j at 2024-05-27T07:18:25.188 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18466 0.67 MB/sec execute 29 sec latency 71.312 ms 2024-05-27T07:18:25.230 INFO:tasks.cephfs.filesystem:scrub status for tag:caf3e5c3-5d19-4858-8737-82df5624fc47 - {'path': '/', 'tag': 'caf3e5c3-5d19-4858-8737-82df5624fc47', 'options': 'recursive,force'} 2024-05-27T07:18:25.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:25 smithi043 ceph-mon[31531]: pgmap v153: 129 pgs: 129 active+clean; 41 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 505 KiB/s wr, 48 op/s 2024-05-27T07:18:25.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:25 smithi043 ceph-mon[31531]: daemon mds.j restarted 2024-05-27T07:18:25.955 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3ec3a00700 time 2024-05-27T07:18:25.779568+0000 2024-05-27T07:18:25.955 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:25.955 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:25.955 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3ed952dba6] 2024-05-27T07:18:25.955 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ed952ddd1] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559aff29e23d] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559aff289971] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 5: (SnapRealm::check_cache() const+0x13e) [0x559aff287fa0] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559aff288651] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559aff21d347] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559afef1987e] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559afef1ee94] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559afef29969] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 11: (MDLog::_replay_thread()+0x1f38) [0x559aff2c248c] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559aff2cb9cb] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 13: (Thread::entry_wrapper()+0x43) [0x7f3ed94fc72d] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3ed94fc749] 2024-05-27T07:18:25.956 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3ed80a01ca] 2024-05-27T07:18:25.957 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 16: clone() 2024-05-27T07:18:25.957 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.781+0000 7f3ec3a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3ec3a00700 time 2024-05-27T07:18:25.779568+0000 2024-05-27T07:18:25.957 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:25.957 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.957 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:25.957 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3ed952dba6] 2024-05-27T07:18:25.957 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ed952ddd1] 2024-05-27T07:18:25.958 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559aff29e23d] 2024-05-27T07:18:25.958 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559aff289971] 2024-05-27T07:18:25.958 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 5: (SnapRealm::check_cache() const+0x13e) [0x559aff287fa0] 2024-05-27T07:18:25.958 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559aff288651] 2024-05-27T07:18:25.958 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559aff21d347] 2024-05-27T07:18:25.958 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559afef1987e] 2024-05-27T07:18:25.958 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559afef1ee94] 2024-05-27T07:18:25.958 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559afef29969] 2024-05-27T07:18:25.958 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 11: (MDLog::_replay_thread()+0x1f38) [0x559aff2c248c] 2024-05-27T07:18:25.958 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559aff2cb9cb] 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 13: (Thread::entry_wrapper()+0x43) [0x7f3ed94fc72d] 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3ed94fc749] 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3ed80a01ca] 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 16: clone() 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: *** Caught signal (Aborted) ** 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: in thread 7f3ec3a00700 thread_name:md_log_replay 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3ed80aad20] 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2: gsignal() 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 3: abort() 2024-05-27T07:18:25.959 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3ed952dced] 2024-05-27T07:18:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:25 smithi179 ceph-mon[35888]: pgmap v153: 129 pgs: 129 active+clean; 41 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 505 KiB/s wr, 48 op/s 2024-05-27T07:18:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:25 smithi179 ceph-mon[35888]: daemon mds.j restarted 2024-05-27T07:18:25.960 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ed952ddd1] 2024-05-27T07:18:25.960 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559aff29e23d] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559aff289971] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 8: (SnapRealm::check_cache() const+0x13e) [0x559aff287fa0] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559aff288651] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559aff21d347] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559afef1987e] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559afef1ee94] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559afef29969] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 14: (MDLog::_replay_thread()+0x1f38) [0x559aff2c248c] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559aff2cb9cb] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 16: (Thread::entry_wrapper()+0x43) [0x7f3ed94fc72d] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3ed94fc749] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3ed80a01ca] 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 19: clone() 2024-05-27T07:18:25.961 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.785+0000 7f3ec3a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: in thread 7f3ec3a00700 thread_name:md_log_replay 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3ed80aad20] 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2: gsignal() 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 3: abort() 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3ed952dced] 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ed952ddd1] 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559aff29e23d] 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559aff289971] 2024-05-27T07:18:25.962 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 8: (SnapRealm::check_cache() const+0x13e) [0x559aff287fa0] 2024-05-27T07:18:25.963 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559aff288651] 2024-05-27T07:18:25.963 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559aff21d347] 2024-05-27T07:18:25.963 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559afef1987e] 2024-05-27T07:18:25.963 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559afef1ee94] 2024-05-27T07:18:25.963 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559afef29969] 2024-05-27T07:18:25.963 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 14: (MDLog::_replay_thread()+0x1f38) [0x559aff2c248c] 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559aff2cb9cb] 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 16: (Thread::entry_wrapper()+0x43) [0x7f3ed94fc72d] 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3ed94fc749] 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3ed80a01ca] 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 19: clone() 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: -1> 2024-05-27T07:18:25.781+0000 7f3ec3a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3ec3a00700 time 2024-05-27T07:18:25.779568+0000 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3ed952dba6] 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ed952ddd1] 2024-05-27T07:18:25.964 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559aff29e23d] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559aff289971] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 5: (SnapRealm::check_cache() const+0x13e) [0x559aff287fa0] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559aff288651] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559aff21d347] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559afef1987e] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559afef1ee94] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559afef29969] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 11: (MDLog::_replay_thread()+0x1f38) [0x559aff2c248c] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559aff2cb9cb] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 13: (Thread::entry_wrapper()+0x43) [0x7f3ed94fc72d] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3ed94fc749] 2024-05-27T07:18:25.965 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3ed80a01ca] 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 16: clone() 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 0> 2024-05-27T07:18:25.785+0000 7f3ec3a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: in thread 7f3ec3a00700 thread_name:md_log_replay 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3ed80aad20] 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2: gsignal() 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 3: abort() 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3ed952dced] 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ed952ddd1] 2024-05-27T07:18:25.966 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559aff29e23d] 2024-05-27T07:18:25.967 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559aff289971] 2024-05-27T07:18:25.967 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 8: (SnapRealm::check_cache() const+0x13e) [0x559aff287fa0] 2024-05-27T07:18:25.967 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559aff288651] 2024-05-27T07:18:25.967 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559aff21d347] 2024-05-27T07:18:25.967 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559afef1987e] 2024-05-27T07:18:25.967 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559afef1ee94] 2024-05-27T07:18:25.967 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559afef29969] 2024-05-27T07:18:25.967 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 14: (MDLog::_replay_thread()+0x1f38) [0x559aff2c248c] 2024-05-27T07:18:25.967 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559aff2cb9cb] 2024-05-27T07:18:25.968 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 16: (Thread::entry_wrapper()+0x43) [0x7f3ed94fc72d] 2024-05-27T07:18:25.968 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3ed94fc749] 2024-05-27T07:18:25.968 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3ed80a01ca] 2024-05-27T07:18:25.968 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 19: clone() 2024-05-27T07:18:25.968 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:25.968 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.968 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: -1> 2024-05-27T07:18:25.781+0000 7f3ec3a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3ec3a00700 time 2024-05-27T07:18:25.779568+0000 2024-05-27T07:18:25.968 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:25.968 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.968 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3ed952dba6] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ed952ddd1] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559aff29e23d] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559aff289971] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 5: (SnapRealm::check_cache() const+0x13e) [0x559aff287fa0] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559aff288651] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559aff21d347] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559afef1987e] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559afef1ee94] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559afef29969] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 11: (MDLog::_replay_thread()+0x1f38) [0x559aff2c248c] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559aff2cb9cb] 2024-05-27T07:18:25.969 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 13: (Thread::entry_wrapper()+0x43) [0x7f3ed94fc72d] 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3ed94fc749] 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3ed80a01ca] 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 16: clone() 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 0> 2024-05-27T07:18:25.785+0000 7f3ec3a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: in thread 7f3ec3a00700 thread_name:md_log_replay 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3ed80aad20] 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2: gsignal() 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 3: abort() 2024-05-27T07:18:25.970 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3ed952dced] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ed952ddd1] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559aff29e23d] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559aff289971] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 8: (SnapRealm::check_cache() const+0x13e) [0x559aff287fa0] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559aff288651] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559aff21d347] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x559afef1987e] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x559afef1ee94] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559afef29969] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 14: (MDLog::_replay_thread()+0x1f38) [0x559aff2c248c] 2024-05-27T07:18:25.971 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559aff2cb9cb] 2024-05-27T07:18:25.972 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 16: (Thread::entry_wrapper()+0x43) [0x7f3ed94fc72d] 2024-05-27T07:18:25.972 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3ed94fc749] 2024-05-27T07:18:25.972 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3ed80a01ca] 2024-05-27T07:18:25.972 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 19: clone() 2024-05-27T07:18:25.972 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:25.972 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[100749]: 2024-05-27T07:18:25.972 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:25 smithi043 podman[100786]: 2024-05-27 07:18:25.880731917 +0000 UTC m=+0.038618332 container died 4b55efc4bbddd38006737ebec1738a1bb95af9e6bc1f4e5a4f71396cd3e75c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-27T07:18:26.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:25 smithi132 ceph-mon[35283]: pgmap v153: 129 pgs: 129 active+clean; 41 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 505 KiB/s wr, 48 op/s 2024-05-27T07:18:26.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:25 smithi132 ceph-mon[35283]: daemon mds.j restarted 2024-05-27T07:18:26.188 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18608 0.66 MB/sec execute 30 sec latency 60.658 ms 2024-05-27T07:18:26.952 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:26 smithi043 podman[100786]: 2024-05-27 07:18:26.754820463 +0000 UTC m=+0.912706939 container remove 4b55efc4bbddd38006737ebec1738a1bb95af9e6bc1f4e5a4f71396cd3e75c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:18:26.952 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:26 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:18:26.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:26 smithi043 ceph-mon[31531]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:18:26.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:26 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/2816978221,v1:172.21.15.43:6835/2816978221] up:boot 2024-05-27T07:18:26.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:26 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:26.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:26 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:26.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:26 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:26.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:26 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:18:26.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:26 smithi179 ceph-mon[35888]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:18:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:26 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/2816978221,v1:172.21.15.43:6835/2816978221] up:boot 2024-05-27T07:18:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:26 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:26 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:26 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:26.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:26 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:18:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:26 smithi132 ceph-mon[35283]: osdmap e107: 12 total, 12 up, 12 in 2024-05-27T07:18:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:26 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/2816978221,v1:172.21.15.43:6835/2816978221] up:boot 2024-05-27T07:18:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:26 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:26 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:26 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:26 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:18:27.189 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18712 0.64 MB/sec execute 31 sec latency 51.450 ms 2024-05-27T07:18:27.702 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:27 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:18:27.702 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:27 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.437s CPU time. 2024-05-27T07:18:28.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:18:28.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:18:28.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:18:28.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:27 smithi132 ceph-mon[35283]: pgmap v155: 129 pgs: 129 active+clean; 42 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 469 KiB/s wr, 59 op/s 2024-05-27T07:18:28.189 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18828 0.66 MB/sec execute 32 sec latency 55.382 ms 2024-05-27T07:18:28.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:18:28.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:18:28.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:18:28.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:27 smithi043 ceph-mon[31531]: pgmap v155: 129 pgs: 129 active+clean; 42 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 469 KiB/s wr, 59 op/s 2024-05-27T07:18:28.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:18:28.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:18:28.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:18:28.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:27 smithi179 ceph-mon[35888]: pgmap v155: 129 pgs: 129 active+clean; 42 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 469 KiB/s wr, 59 op/s 2024-05-27T07:18:29.189 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18925 0.64 MB/sec execute 33 sec latency 63.579 ms 2024-05-27T07:18:30.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:29 smithi132 ceph-mon[35283]: pgmap v156: 129 pgs: 129 active+clean; 42 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 418 KiB/s wr, 52 op/s 2024-05-27T07:18:30.189 INFO:tasks.workunit.client.0.smithi043.stdout: 1 19035 0.63 MB/sec execute 34 sec latency 51.309 ms 2024-05-27T07:18:30.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:29 smithi043 ceph-mon[31531]: pgmap v156: 129 pgs: 129 active+clean; 42 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 418 KiB/s wr, 52 op/s 2024-05-27T07:18:30.202 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:29 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:18:30.202 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:29 smithi043 systemd[1]: Stopped Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:30.202 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:29 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.614s CPU time. 2024-05-27T07:18:30.202 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:29 smithi043 systemd[1]: Starting Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:18:30.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:29 smithi179 ceph-mon[35888]: pgmap v156: 129 pgs: 129 active+clean; 42 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 418 KiB/s wr, 52 op/s 2024-05-27T07:18:30.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:30 smithi043 podman[100878]: 2024-05-27 07:18:30.290466879 +0000 UTC m=+0.037832334 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:18:30.703 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:30 smithi043 podman[100878]: 2024-05-27 07:18:30.43941905 +0000 UTC m=+0.186784599 container create d8ffe0d58f6fc6607a674be080ecad5c11b65f2ae2c92628c69f7ba4babfff90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:18:31.182 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:30 smithi043 podman[100878]: 2024-05-27 07:18:30.71983454 +0000 UTC m=+0.467200075 container init d8ffe0d58f6fc6607a674be080ecad5c11b65f2ae2c92628c69f7ba4babfff90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-27T07:18:31.182 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:30 smithi043 podman[100878]: 2024-05-27 07:18:30.734951731 +0000 UTC m=+0.482317243 container start d8ffe0d58f6fc6607a674be080ecad5c11b65f2ae2c92628c69f7ba4babfff90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-05-27T07:18:31.182 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:30 smithi043 bash[100878]: d8ffe0d58f6fc6607a674be080ecad5c11b65f2ae2c92628c69f7ba4babfff90 2024-05-27T07:18:31.182 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:30 smithi043 systemd[1]: Started Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:31.182 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:30 smithi043 ceph-mds[100896]: -- 172.21.15.43:0/1768226285 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563fee41fa40 con 0x563fee467a80 2024-05-27T07:18:31.183 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:30 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: starting mds.d at 2024-05-27T07:18:31.183 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffa50c00700 time 2024-05-27T07:18:31.190955+0000 2024-05-27T07:18:31.183 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:31.189 INFO:tasks.workunit.client.0.smithi043.stdout: 1 19232 0.69 MB/sec execute 35 sec latency 49.361 ms 2024-05-27T07:18:31.454 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:31.454 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffa66794ba6] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa66794dd1] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fb9c8123d] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563fb9c6c971] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 5: (SnapRealm::check_cache() const+0x13e) [0x563fb9c6afa0] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fb9c6b651] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fb9c00347] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fb98fc87e] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563fb9901e94] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563fb990c969] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 11: (MDLog::_replay_thread()+0x1f38) [0x563fb9ca548c] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563fb9cae9cb] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 13: (Thread::entry_wrapper()+0x43) [0x7ffa6676372d] 2024-05-27T07:18:31.455 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffa66763749] 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffa653071ca] 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 16: clone() 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: *** Caught signal (Aborted) ** 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: in thread 7ffa50c00700 thread_name:md_log_replay 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffa65311d20] 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2: gsignal() 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 3: abort() 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffa66794ced] 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa66794dd1] 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fb9c8123d] 2024-05-27T07:18:31.456 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563fb9c6c971] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 8: (SnapRealm::check_cache() const+0x13e) [0x563fb9c6afa0] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fb9c6b651] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fb9c00347] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fb98fc87e] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563fb9901e94] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563fb990c969] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 14: (MDLog::_replay_thread()+0x1f38) [0x563fb9ca548c] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563fb9cae9cb] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 16: (Thread::entry_wrapper()+0x43) [0x7ffa6676372d] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffa66763749] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffa653071ca] 2024-05-27T07:18:31.457 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 19: clone() 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.194+0000 7ffa50c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffa50c00700 time 2024-05-27T07:18:31.190955+0000 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffa66794ba6] 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa66794dd1] 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fb9c8123d] 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563fb9c6c971] 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 5: (SnapRealm::check_cache() const+0x13e) [0x563fb9c6afa0] 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fb9c6b651] 2024-05-27T07:18:31.458 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fb9c00347] 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fb98fc87e] 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563fb9901e94] 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563fb990c969] 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 11: (MDLog::_replay_thread()+0x1f38) [0x563fb9ca548c] 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563fb9cae9cb] 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 13: (Thread::entry_wrapper()+0x43) [0x7ffa6676372d] 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffa66763749] 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffa653071ca] 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 16: clone() 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.198+0000 7ffa50c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:31.459 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: in thread 7ffa50c00700 thread_name:md_log_replay 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffa65311d20] 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2: gsignal() 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 3: abort() 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffa66794ced] 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa66794dd1] 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fb9c8123d] 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563fb9c6c971] 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 8: (SnapRealm::check_cache() const+0x13e) [0x563fb9c6afa0] 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fb9c6b651] 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fb9c00347] 2024-05-27T07:18:31.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fb98fc87e] 2024-05-27T07:18:31.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563fb9901e94] 2024-05-27T07:18:31.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563fb990c969] 2024-05-27T07:18:31.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 14: (MDLog::_replay_thread()+0x1f38) [0x563fb9ca548c] 2024-05-27T07:18:31.461 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563fb9cae9cb] 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 16: (Thread::entry_wrapper()+0x43) [0x7ffa6676372d] 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffa66763749] 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffa653071ca] 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 19: clone() 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: -1> 2024-05-27T07:18:31.194+0000 7ffa50c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffa50c00700 time 2024-05-27T07:18:31.190955+0000 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:31.462 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffa66794ba6] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa66794dd1] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fb9c8123d] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563fb9c6c971] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 5: (SnapRealm::check_cache() const+0x13e) [0x563fb9c6afa0] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fb9c6b651] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fb9c00347] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fb98fc87e] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563fb9901e94] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563fb990c969] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 11: (MDLog::_replay_thread()+0x1f38) [0x563fb9ca548c] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563fb9cae9cb] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 13: (Thread::entry_wrapper()+0x43) [0x7ffa6676372d] 2024-05-27T07:18:31.463 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffa66763749] 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffa653071ca] 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 16: clone() 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 0> 2024-05-27T07:18:31.198+0000 7ffa50c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: in thread 7ffa50c00700 thread_name:md_log_replay 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffa65311d20] 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2: gsignal() 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 3: abort() 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffa66794ced] 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa66794dd1] 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fb9c8123d] 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563fb9c6c971] 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 8: (SnapRealm::check_cache() const+0x13e) [0x563fb9c6afa0] 2024-05-27T07:18:31.464 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fb9c6b651] 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fb9c00347] 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fb98fc87e] 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563fb9901e94] 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563fb990c969] 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 14: (MDLog::_replay_thread()+0x1f38) [0x563fb9ca548c] 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563fb9cae9cb] 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 16: (Thread::entry_wrapper()+0x43) [0x7ffa6676372d] 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffa66763749] 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffa653071ca] 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 19: clone() 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:31.465 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: -1> 2024-05-27T07:18:31.194+0000 7ffa50c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffa50c00700 time 2024-05-27T07:18:31.190955+0000 2024-05-27T07:18:31.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:31.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:31.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffa66794ba6] 2024-05-27T07:18:31.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa66794dd1] 2024-05-27T07:18:31.466 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fb9c8123d] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 4: (SnapRealm::build_snap_set() const+0x323) [0x563fb9c6c971] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 5: (SnapRealm::check_cache() const+0x13e) [0x563fb9c6afa0] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fb9c6b651] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fb9c00347] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fb98fc87e] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563fb9901e94] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x563fb990c969] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 11: (MDLog::_replay_thread()+0x1f38) [0x563fb9ca548c] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 12: (MDLog::ReplayThread::entry()+0x11) [0x563fb9cae9cb] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 13: (Thread::entry_wrapper()+0x43) [0x7ffa6676372d] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffa66763749] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffa653071ca] 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 16: clone() 2024-05-27T07:18:31.467 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 0> 2024-05-27T07:18:31.198+0000 7ffa50c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: in thread 7ffa50c00700 thread_name:md_log_replay 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffa65311d20] 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2: gsignal() 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 3: abort() 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffa66794ced] 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa66794dd1] 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x563fb9c8123d] 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 7: (SnapRealm::build_snap_set() const+0x323) [0x563fb9c6c971] 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 8: (SnapRealm::check_cache() const+0x13e) [0x563fb9c6afa0] 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x563fb9c6b651] 2024-05-27T07:18:31.468 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x563fb9c00347] 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x563fb98fc87e] 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x563fb9901e94] 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x563fb990c969] 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 14: (MDLog::_replay_thread()+0x1f38) [0x563fb9ca548c] 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 15: (MDLog::ReplayThread::entry()+0x11) [0x563fb9cae9cb] 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 16: (Thread::entry_wrapper()+0x43) [0x7ffa6676372d] 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffa66763749] 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffa653071ca] 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 19: clone() 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d[100892]: 2024-05-27T07:18:31.469 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:31 smithi043 podman[100927]: 2024-05-27 07:18:31.319111978 +0000 UTC m=+0.038951161 container died d8ffe0d58f6fc6607a674be080ecad5c11b65f2ae2c92628c69f7ba4babfff90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-27T07:18:31.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:31 smithi043 ceph-mon[31531]: pgmap v157: 129 pgs: 129 active+clean; 40 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 338 KiB/s wr, 55 op/s 2024-05-27T07:18:31.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:31 smithi043 ceph-mon[31531]: daemon mds.d restarted 2024-05-27T07:18:31.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:31 smithi043 ceph-mon[31531]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:18:31.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:31 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/2223818817,v1:172.21.15.43:6835/2223818817] up:boot 2024-05-27T07:18:31.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:31 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:31.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:31 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:18:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:31 smithi132 ceph-mon[35283]: pgmap v157: 129 pgs: 129 active+clean; 40 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 338 KiB/s wr, 55 op/s 2024-05-27T07:18:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:31 smithi132 ceph-mon[35283]: daemon mds.d restarted 2024-05-27T07:18:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:31 smithi132 ceph-mon[35283]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:18:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:31 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/2223818817,v1:172.21.15.43:6835/2223818817] up:boot 2024-05-27T07:18:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:31 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:32.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:31 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:18:32.189 INFO:tasks.workunit.client.0.smithi043.stdout: 1 19353 0.68 MB/sec execute 36 sec latency 54.155 ms 2024-05-27T07:18:32.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:31 smithi179 ceph-mon[35888]: pgmap v157: 129 pgs: 129 active+clean; 40 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 338 KiB/s wr, 55 op/s 2024-05-27T07:18:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:31 smithi179 ceph-mon[35888]: daemon mds.d restarted 2024-05-27T07:18:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:31 smithi179 ceph-mon[35888]: osdmap e108: 12 total, 12 up, 12 in 2024-05-27T07:18:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:31 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/2223818817,v1:172.21.15.43:6835/2223818817] up:boot 2024-05-27T07:18:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:31 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:32.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:31 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-27T07:18:32.452 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:32 smithi043 podman[100927]: 2024-05-27 07:18:32.099176165 +0000 UTC m=+0.819015392 container remove d8ffe0d58f6fc6607a674be080ecad5c11b65f2ae2c92628c69f7ba4babfff90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=) 2024-05-27T07:18:32.453 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:32 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:18:32.952 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:32 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:18:32.953 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:32 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.671s CPU time. 2024-05-27T07:18:33.189 INFO:tasks.workunit.client.0.smithi043.stdout: 1 19479 0.66 MB/sec execute 37 sec latency 53.162 ms 2024-05-27T07:18:34.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:33 smithi132 ceph-mon[35283]: pgmap v159: 129 pgs: 129 active+clean; 40 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 422 KiB/s wr, 69 op/s 2024-05-27T07:18:34.189 INFO:tasks.workunit.client.0.smithi043.stdout: 1 19585 0.64 MB/sec execute 38 sec latency 343.313 ms 2024-05-27T07:18:34.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:33 smithi043 ceph-mon[31531]: pgmap v159: 129 pgs: 129 active+clean; 40 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 422 KiB/s wr, 69 op/s 2024-05-27T07:18:34.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:33 smithi179 ceph-mon[35888]: pgmap v159: 129 pgs: 129 active+clean; 40 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 422 KiB/s wr, 69 op/s 2024-05-27T07:18:35.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:34 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:35.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:34 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:35.189 INFO:tasks.workunit.client.0.smithi043.stdout: 1 19713 0.63 MB/sec execute 39 sec latency 59.645 ms 2024-05-27T07:18:35.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:34 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:36.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:35 smithi132 ceph-mon[35283]: pgmap v160: 129 pgs: 129 active+clean; 40 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 229 KiB/s wr, 33 op/s 2024-05-27T07:18:36.190 INFO:tasks.workunit.client.0.smithi043.stdout: 1 19829 0.62 MB/sec execute 40 sec latency 56.223 ms 2024-05-27T07:18:36.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:35 smithi043 ceph-mon[31531]: pgmap v160: 129 pgs: 129 active+clean; 40 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 229 KiB/s wr, 33 op/s 2024-05-27T07:18:36.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:35 smithi179 ceph-mon[35888]: pgmap v160: 129 pgs: 129 active+clean; 40 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 229 KiB/s wr, 33 op/s 2024-05-27T07:18:37.190 INFO:tasks.workunit.client.0.smithi043.stdout: 1 19987 0.61 MB/sec execute 41 sec latency 55.558 ms 2024-05-27T07:18:37.801 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:37 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:18:37.801 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:37 smithi043 systemd[1]: Stopped Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:37.801 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:37 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.437s CPU time. 2024-05-27T07:18:37.801 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:37 smithi043 systemd[1]: Starting Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:18:38.070 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:37 smithi043 podman[101017]: 2024-05-27 07:18:37.79380239 +0000 UTC m=+0.040931993 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:18:38.070 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:37 smithi043 podman[101017]: 2024-05-27 07:18:37.908442876 +0000 UTC m=+0.155572516 container create 29098eaad90ee0e3fdc59fa0696a0ca366cb4ef85ce9a0346b5337fa7ee12a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-27T07:18:38.070 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:37 smithi043 ceph-mon[31531]: pgmap v161: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 265 KiB/s wr, 53 op/s 2024-05-27T07:18:38.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:37 smithi132 ceph-mon[35283]: pgmap v161: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 265 KiB/s wr, 53 op/s 2024-05-27T07:18:38.190 INFO:tasks.workunit.client.0.smithi043.stdout: 1 20146 0.62 MB/sec execute 42 sec latency 56.090 ms 2024-05-27T07:18:38.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:37 smithi179 ceph-mon[35888]: pgmap v161: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 265 KiB/s wr, 53 op/s 2024-05-27T07:18:38.454 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 podman[101017]: 2024-05-27 07:18:38.179469356 +0000 UTC m=+0.426599073 container init 29098eaad90ee0e3fdc59fa0696a0ca366cb4ef85ce9a0346b5337fa7ee12a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:18:38.454 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 podman[101017]: 2024-05-27 07:18:38.191147971 +0000 UTC m=+0.438277624 container start 29098eaad90ee0e3fdc59fa0696a0ca366cb4ef85ce9a0346b5337fa7ee12a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS) 2024-05-27T07:18:38.454 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 bash[101017]: 29098eaad90ee0e3fdc59fa0696a0ca366cb4ef85ce9a0346b5337fa7ee12a16 2024-05-27T07:18:38.454 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 systemd[1]: Started Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:38.454 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-mds[101035]: -- 172.21.15.43:0/3389910411 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564fe5db3a40 con 0x564fe5dfba80 2024-05-27T07:18:38.454 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: starting mds.j at 2024-05-27T07:18:39.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:38 smithi132 ceph-mon[35283]: daemon mds.j restarted 2024-05-27T07:18:39.110 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:38 smithi179 ceph-mon[35888]: daemon mds.j restarted 2024-05-27T07:18:39.190 INFO:tasks.workunit.client.0.smithi043.stdout: 1 20223 0.61 MB/sec execute 43 sec latency 49.736 ms 2024-05-27T07:18:39.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:38 smithi043 ceph-mon[31531]: daemon mds.j restarted 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0255e00700 time 2024-05-27T07:18:38.912368+0000 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f026b93bba6] 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f026b93bdd1] 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564fd4a3823d] 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564fd4a23971] 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 5: (SnapRealm::check_cache() const+0x13e) [0x564fd4a21fa0] 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564fd4a22651] 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564fd49b7347] 2024-05-27T07:18:39.204 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564fd46b387e] 2024-05-27T07:18:39.205 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564fd46b8e94] 2024-05-27T07:18:39.205 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564fd46c3969] 2024-05-27T07:18:39.205 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 11: (MDLog::_replay_thread()+0x1f38) [0x564fd4a5c48c] 2024-05-27T07:18:39.205 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564fd4a659cb] 2024-05-27T07:18:39.205 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 13: (Thread::entry_wrapper()+0x43) [0x7f026b90a72d] 2024-05-27T07:18:39.205 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 14: (Thread::_entry_func(void*)+0xd) [0x7f026b90a749] 2024-05-27T07:18:39.205 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f026a4ae1ca] 2024-05-27T07:18:39.205 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 16: clone() 2024-05-27T07:18:39.205 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:38.915+0000 7f0255e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0255e00700 time 2024-05-27T07:18:38.912368+0000 2024-05-27T07:18:39.205 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f026b93bba6] 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f026b93bdd1] 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564fd4a3823d] 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564fd4a23971] 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 5: (SnapRealm::check_cache() const+0x13e) [0x564fd4a21fa0] 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564fd4a22651] 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564fd49b7347] 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564fd46b387e] 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564fd46b8e94] 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564fd46c3969] 2024-05-27T07:18:39.206 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 11: (MDLog::_replay_thread()+0x1f38) [0x564fd4a5c48c] 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564fd4a659cb] 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 13: (Thread::entry_wrapper()+0x43) [0x7f026b90a72d] 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 14: (Thread::_entry_func(void*)+0xd) [0x7f026b90a749] 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f026a4ae1ca] 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 16: clone() 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: *** Caught signal (Aborted) ** 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: in thread 7f0255e00700 thread_name:md_log_replay 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f026a4b8d20] 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2: gsignal() 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 3: abort() 2024-05-27T07:18:39.207 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f026b93bced] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f026b93bdd1] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564fd4a3823d] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564fd4a23971] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 8: (SnapRealm::check_cache() const+0x13e) [0x564fd4a21fa0] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564fd4a22651] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564fd49b7347] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564fd46b387e] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564fd46b8e94] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564fd46c3969] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 14: (MDLog::_replay_thread()+0x1f38) [0x564fd4a5c48c] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564fd4a659cb] 2024-05-27T07:18:39.208 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 16: (Thread::entry_wrapper()+0x43) [0x7f026b90a72d] 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 17: (Thread::_entry_func(void*)+0xd) [0x7f026b90a749] 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f026a4ae1ca] 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 19: clone() 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:38.921+0000 7f0255e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: in thread 7f0255e00700 thread_name:md_log_replay 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f026a4b8d20] 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2: gsignal() 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 3: abort() 2024-05-27T07:18:39.209 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f026b93bced] 2024-05-27T07:18:39.210 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f026b93bdd1] 2024-05-27T07:18:39.210 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564fd4a3823d] 2024-05-27T07:18:39.210 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564fd4a23971] 2024-05-27T07:18:39.210 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 8: (SnapRealm::check_cache() const+0x13e) [0x564fd4a21fa0] 2024-05-27T07:18:39.210 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564fd4a22651] 2024-05-27T07:18:39.210 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564fd49b7347] 2024-05-27T07:18:39.210 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564fd46b387e] 2024-05-27T07:18:39.210 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564fd46b8e94] 2024-05-27T07:18:39.210 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564fd46c3969] 2024-05-27T07:18:39.210 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 14: (MDLog::_replay_thread()+0x1f38) [0x564fd4a5c48c] 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564fd4a659cb] 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 16: (Thread::entry_wrapper()+0x43) [0x7f026b90a72d] 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 17: (Thread::_entry_func(void*)+0xd) [0x7f026b90a749] 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f026a4ae1ca] 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 19: clone() 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: -1> 2024-05-27T07:18:38.915+0000 7f0255e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0255e00700 time 2024-05-27T07:18:38.912368+0000 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f026b93bba6] 2024-05-27T07:18:39.211 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f026b93bdd1] 2024-05-27T07:18:39.212 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564fd4a3823d] 2024-05-27T07:18:39.212 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564fd4a23971] 2024-05-27T07:18:39.212 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 5: (SnapRealm::check_cache() const+0x13e) [0x564fd4a21fa0] 2024-05-27T07:18:39.212 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564fd4a22651] 2024-05-27T07:18:39.212 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564fd49b7347] 2024-05-27T07:18:39.212 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564fd46b387e] 2024-05-27T07:18:39.212 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564fd46b8e94] 2024-05-27T07:18:39.212 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564fd46c3969] 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 11: (MDLog::_replay_thread()+0x1f38) [0x564fd4a5c48c] 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564fd4a659cb] 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 13: (Thread::entry_wrapper()+0x43) [0x7f026b90a72d] 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 14: (Thread::_entry_func(void*)+0xd) [0x7f026b90a749] 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f026a4ae1ca] 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 16: clone() 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 0> 2024-05-27T07:18:38.921+0000 7f0255e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: in thread 7f0255e00700 thread_name:md_log_replay 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f026a4b8d20] 2024-05-27T07:18:39.213 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2: gsignal() 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 3: abort() 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f026b93bced] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f026b93bdd1] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564fd4a3823d] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564fd4a23971] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 8: (SnapRealm::check_cache() const+0x13e) [0x564fd4a21fa0] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564fd4a22651] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564fd49b7347] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564fd46b387e] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564fd46b8e94] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564fd46c3969] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 14: (MDLog::_replay_thread()+0x1f38) [0x564fd4a5c48c] 2024-05-27T07:18:39.214 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564fd4a659cb] 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 16: (Thread::entry_wrapper()+0x43) [0x7f026b90a72d] 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 17: (Thread::_entry_func(void*)+0xd) [0x7f026b90a749] 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f026a4ae1ca] 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 19: clone() 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: -1> 2024-05-27T07:18:38.915+0000 7f0255e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f0255e00700 time 2024-05-27T07:18:38.912368+0000 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:39.215 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f026b93bba6] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f026b93bdd1] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564fd4a3823d] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 4: (SnapRealm::build_snap_set() const+0x323) [0x564fd4a23971] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 5: (SnapRealm::check_cache() const+0x13e) [0x564fd4a21fa0] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564fd4a22651] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564fd49b7347] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564fd46b387e] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564fd46b8e94] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564fd46c3969] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 11: (MDLog::_replay_thread()+0x1f38) [0x564fd4a5c48c] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 12: (MDLog::ReplayThread::entry()+0x11) [0x564fd4a659cb] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 13: (Thread::entry_wrapper()+0x43) [0x7f026b90a72d] 2024-05-27T07:18:39.216 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 14: (Thread::_entry_func(void*)+0xd) [0x7f026b90a749] 2024-05-27T07:18:39.217 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f026a4ae1ca] 2024-05-27T07:18:39.217 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 16: clone() 2024-05-27T07:18:39.217 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.217 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 0> 2024-05-27T07:18:38.921+0000 7f0255e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:39.217 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: in thread 7f0255e00700 thread_name:md_log_replay 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f026a4b8d20] 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2: gsignal() 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 3: abort() 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f026b93bced] 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f026b93bdd1] 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x564fd4a3823d] 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 7: (SnapRealm::build_snap_set() const+0x323) [0x564fd4a23971] 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 8: (SnapRealm::check_cache() const+0x13e) [0x564fd4a21fa0] 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x564fd4a22651] 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x564fd49b7347] 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564fd46b387e] 2024-05-27T07:18:39.218 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564fd46b8e94] 2024-05-27T07:18:39.219 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564fd46c3969] 2024-05-27T07:18:39.219 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 14: (MDLog::_replay_thread()+0x1f38) [0x564fd4a5c48c] 2024-05-27T07:18:39.219 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 15: (MDLog::ReplayThread::entry()+0x11) [0x564fd4a659cb] 2024-05-27T07:18:39.219 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 16: (Thread::entry_wrapper()+0x43) [0x7f026b90a72d] 2024-05-27T07:18:39.219 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 17: (Thread::_entry_func(void*)+0xd) [0x7f026b90a749] 2024-05-27T07:18:39.219 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f026a4ae1ca] 2024-05-27T07:18:39.219 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 19: clone() 2024-05-27T07:18:39.219 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:39.219 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101031]: 2024-05-27T07:18:39.219 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:39 smithi043 podman[101066]: 2024-05-27 07:18:39.027740157 +0000 UTC m=+0.039352429 container died 29098eaad90ee0e3fdc59fa0696a0ca366cb4ef85ce9a0346b5337fa7ee12a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0) 2024-05-27T07:18:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:39 smithi132 ceph-mon[35283]: pgmap v162: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 265 KiB/s wr, 53 op/s 2024-05-27T07:18:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:39 smithi132 ceph-mon[35283]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:18:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:39 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/1147340542,v1:172.21.15.43:6835/1147340542] up:boot 2024-05-27T07:18:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:39 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:39 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:40.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:39 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:40.190 INFO:tasks.workunit.client.0.smithi043.stdout: 1 20349 0.62 MB/sec execute 44 sec latency 52.138 ms 2024-05-27T07:18:40.202 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:39 smithi043 podman[101066]: 2024-05-27 07:18:39.826630583 +0000 UTC m=+0.838242889 container remove 29098eaad90ee0e3fdc59fa0696a0ca366cb4ef85ce9a0346b5337fa7ee12a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-27T07:18:40.202 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:39 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:18:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:39 smithi043 ceph-mon[31531]: pgmap v162: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 265 KiB/s wr, 53 op/s 2024-05-27T07:18:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:39 smithi043 ceph-mon[31531]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:18:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:39 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/1147340542,v1:172.21.15.43:6835/1147340542] up:boot 2024-05-27T07:18:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:39 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:39 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:40.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:39 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:39 smithi179 ceph-mon[35888]: pgmap v162: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 265 KiB/s wr, 53 op/s 2024-05-27T07:18:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:39 smithi179 ceph-mon[35888]: osdmap e109: 12 total, 12 up, 12 in 2024-05-27T07:18:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:39 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/1147340542,v1:172.21.15.43:6835/1147340542] up:boot 2024-05-27T07:18:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:39 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:39 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:40.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:39 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:40.702 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:40 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:18:40.702 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:40 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.575s CPU time. 2024-05-27T07:18:41.190 INFO:tasks.workunit.client.0.smithi043.stdout: 1 20527 0.67 MB/sec execute 45 sec latency 57.271 ms 2024-05-27T07:18:42.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:41 smithi132 ceph-mon[35283]: pgmap v164: 129 pgs: 129 active+clean; 42 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 315 KiB/s wr, 46 op/s 2024-05-27T07:18:42.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:41 smithi043 ceph-mon[31531]: pgmap v164: 129 pgs: 129 active+clean; 42 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 315 KiB/s wr, 46 op/s 2024-05-27T07:18:42.190 INFO:tasks.workunit.client.0.smithi043.stdout: 1 20712 0.69 MB/sec execute 46 sec latency 51.396 ms 2024-05-27T07:18:42.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:41 smithi179 ceph-mon[35888]: pgmap v164: 129 pgs: 129 active+clean; 42 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 315 KiB/s wr, 46 op/s 2024-05-27T07:18:43.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:42 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:43.134 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:42 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:43.135 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:42 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:18:43.135 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:42 smithi043 systemd[1]: Stopped Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:43.135 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:42 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Consumed 1.671s CPU time. 2024-05-27T07:18:43.135 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:42 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-05-27T07:18:43.135 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:42 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-05-27T07:18:43.135 INFO:journalctl@ceph.mds.d.smithi043.stdout:May 27 07:18:42 smithi043 systemd[1]: Failed to start Ceph mds.d for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:43.190 INFO:tasks.workunit.client.0.smithi043.stdout: 1 20947 0.69 MB/sec execute 47 sec latency 52.290 ms 2024-05-27T07:18:43.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:42 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-05-27T07:18:44.190 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21065 0.70 MB/sec execute 48 sec latency 54.962 ms 2024-05-27T07:18:44.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:43 smithi043 ceph-mon[31531]: pgmap v165: 129 pgs: 129 active+clean; 42 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 300 KiB/s wr, 44 op/s 2024-05-27T07:18:44.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:43 smithi179 ceph-mon[35888]: pgmap v165: 129 pgs: 129 active+clean; 42 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 300 KiB/s wr, 44 op/s 2024-05-27T07:18:44.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:43 smithi132 ceph-mon[35283]: pgmap v165: 129 pgs: 129 active+clean; 42 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 300 KiB/s wr, 44 op/s 2024-05-27T07:18:45.190 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21093 0.69 MB/sec execute 49 sec latency 54.528 ms 2024-05-27T07:18:46.191 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21116 0.67 MB/sec execute 50 sec latency 61.216 ms 2024-05-27T07:18:46.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:45 smithi043 ceph-mon[31531]: pgmap v166: 129 pgs: 129 active+clean; 42 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 300 KiB/s wr, 44 op/s 2024-05-27T07:18:46.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:45 smithi179 ceph-mon[35888]: pgmap v166: 129 pgs: 129 active+clean; 42 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 300 KiB/s wr, 44 op/s 2024-05-27T07:18:46.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:45 smithi132 ceph-mon[35283]: pgmap v166: 129 pgs: 129 active+clean; 42 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 300 KiB/s wr, 44 op/s 2024-05-27T07:18:47.191 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21290 0.72 MB/sec execute 51 sec latency 51.072 ms 2024-05-27T07:18:47.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:46 smithi043 ceph-mon[31531]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:18:47.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:46 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:47.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:46 smithi179 ceph-mon[35888]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:18:47.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:46 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:47.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:46 smithi132 ceph-mon[35283]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-05-27T07:18:47.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:46 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:48.191 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21404 0.71 MB/sec execute 52 sec latency 59.443 ms 2024-05-27T07:18:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:48 smithi132 ceph-mon[35283]: pgmap v167: 129 pgs: 129 active+clean; 44 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 449 KiB/s wr, 41 op/s 2024-05-27T07:18:48.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:48 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:48.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:48 smithi043 ceph-mon[31531]: pgmap v167: 129 pgs: 129 active+clean; 44 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 449 KiB/s wr, 41 op/s 2024-05-27T07:18:48.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:48 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:48.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:48 smithi179 ceph-mon[35888]: pgmap v167: 129 pgs: 129 active+clean; 44 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 449 KiB/s wr, 41 op/s 2024-05-27T07:18:48.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:48 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:49.191 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21491 0.70 MB/sec execute 53 sec latency 51.101 ms 2024-05-27T07:18:50.191 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21576 0.70 MB/sec execute 54 sec latency 50.180 ms 2024-05-27T07:18:50.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:50 smithi132 ceph-mon[35283]: pgmap v168: 129 pgs: 129 active+clean; 44 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 449 KiB/s wr, 41 op/s 2024-05-27T07:18:50.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:50 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:50.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:50 smithi043 ceph-mon[31531]: pgmap v168: 129 pgs: 129 active+clean; 44 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 449 KiB/s wr, 41 op/s 2024-05-27T07:18:50.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:50 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:50.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:50 smithi179 ceph-mon[35888]: pgmap v168: 129 pgs: 129 active+clean; 44 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 449 KiB/s wr, 41 op/s 2024-05-27T07:18:50.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:50 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:50.866 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:50 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:18:50.866 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:50 smithi043 systemd[1]: Stopped Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:50.866 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:50 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.575s CPU time. 2024-05-27T07:18:50.866 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:50 smithi043 systemd[1]: Starting Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:18:51.182 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:50 smithi043 podman[101169]: 2024-05-27 07:18:50.781509985 +0000 UTC m=+0.034561832 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:18:51.182 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:50 smithi043 podman[101169]: 2024-05-27 07:18:50.92781096 +0000 UTC m=+0.180862849 container create d41213e3f5d6038d3bfd1acc46f6e0b825e2f3eee9396505a850bd5f0330012f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:18:51.182 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:51 smithi043 podman[101169]: 2024-05-27 07:18:51.191224268 +0000 UTC m=+0.444276188 container init d41213e3f5d6038d3bfd1acc46f6e0b825e2f3eee9396505a850bd5f0330012f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:18:51.191 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21639 0.69 MB/sec execute 55 sec latency 40.538 ms 2024-05-27T07:18:51.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:51 smithi043 podman[101169]: 2024-05-27 07:18:51.202234949 +0000 UTC m=+0.455286836 container start d41213e3f5d6038d3bfd1acc46f6e0b825e2f3eee9396505a850bd5f0330012f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:18:51.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:51 smithi043 bash[101169]: d41213e3f5d6038d3bfd1acc46f6e0b825e2f3eee9396505a850bd5f0330012f 2024-05-27T07:18:51.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:51 smithi043 systemd[1]: Started Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:18:51.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:51 smithi043 ceph-mds[101188]: -- 172.21.15.43:0/541140528 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c22f165a40 con 0x55c22f1ad600 2024-05-27T07:18:51.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: starting mds.j at 2024-05-27T07:18:52.191 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21703 0.68 MB/sec execute 56 sec latency 68.245 ms 2024-05-27T07:18:52.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:52 smithi043 ceph-mon[31531]: pgmap v169: 129 pgs: 129 active+clean; 44 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 608 KiB/s wr, 53 op/s 2024-05-27T07:18:52.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:52 smithi043 ceph-mon[31531]: daemon mds.j restarted 2024-05-27T07:18:52.359 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7feb8e800700 time 2024-05-27T07:18:52.251999+0000 2024-05-27T07:18:52.359 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:52.359 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:52.359 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7feba424cba6] 2024-05-27T07:18:52.359 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7feba424cdd1] 2024-05-27T07:18:52.359 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c20bde023d] 2024-05-27T07:18:52.359 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c20bdcb971] 2024-05-27T07:18:52.359 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c20bdc9fa0] 2024-05-27T07:18:52.359 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c20bdca651] 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c20bd5f347] 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c20ba5b87e] 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c20ba60e94] 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c20ba6b969] 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c20be0448c] 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c20be0d9cb] 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 13: (Thread::entry_wrapper()+0x43) [0x7feba421b72d] 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 14: (Thread::_entry_func(void*)+0xd) [0x7feba421b749] 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7feba2dbf1ca] 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 16: clone() 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: *** Caught signal (Aborted) ** 2024-05-27T07:18:52.360 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: in thread 7feb8e800700 thread_name:md_log_replay 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feba2dc9d20] 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2: gsignal() 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 3: abort() 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7feba424cced] 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7feba424cdd1] 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c20bde023d] 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c20bdcb971] 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c20bdc9fa0] 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c20bdca651] 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c20bd5f347] 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c20ba5b87e] 2024-05-27T07:18:52.361 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c20ba60e94] 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c20ba6b969] 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c20be0448c] 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c20be0d9cb] 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 16: (Thread::entry_wrapper()+0x43) [0x7feba421b72d] 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 17: (Thread::_entry_func(void*)+0xd) [0x7feba421b749] 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7feba2dbf1ca] 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 19: clone() 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.255+0000 7feb8e800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7feb8e800700 time 2024-05-27T07:18:52.251999+0000 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7feba424cba6] 2024-05-27T07:18:52.362 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7feba424cdd1] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c20bde023d] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c20bdcb971] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c20bdc9fa0] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c20bdca651] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c20bd5f347] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c20ba5b87e] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c20ba60e94] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c20ba6b969] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c20be0448c] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c20be0d9cb] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 13: (Thread::entry_wrapper()+0x43) [0x7feba421b72d] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 14: (Thread::_entry_func(void*)+0xd) [0x7feba421b749] 2024-05-27T07:18:52.363 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7feba2dbf1ca] 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 16: clone() 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.259+0000 7feb8e800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: in thread 7feb8e800700 thread_name:md_log_replay 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feba2dc9d20] 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2: gsignal() 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 3: abort() 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7feba424cced] 2024-05-27T07:18:52.364 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7feba424cdd1] 2024-05-27T07:18:52.365 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c20bde023d] 2024-05-27T07:18:52.365 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c20bdcb971] 2024-05-27T07:18:52.365 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c20bdc9fa0] 2024-05-27T07:18:52.365 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c20bdca651] 2024-05-27T07:18:52.366 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c20bd5f347] 2024-05-27T07:18:52.366 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c20ba5b87e] 2024-05-27T07:18:52.366 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c20ba60e94] 2024-05-27T07:18:52.366 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c20ba6b969] 2024-05-27T07:18:52.366 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c20be0448c] 2024-05-27T07:18:52.366 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c20be0d9cb] 2024-05-27T07:18:52.366 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 16: (Thread::entry_wrapper()+0x43) [0x7feba421b72d] 2024-05-27T07:18:52.366 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 17: (Thread::_entry_func(void*)+0xd) [0x7feba421b749] 2024-05-27T07:18:52.366 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7feba2dbf1ca] 2024-05-27T07:18:52.366 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 19: clone() 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: -1> 2024-05-27T07:18:52.255+0000 7feb8e800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7feb8e800700 time 2024-05-27T07:18:52.251999+0000 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7feba424cba6] 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7feba424cdd1] 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c20bde023d] 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c20bdcb971] 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c20bdc9fa0] 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c20bdca651] 2024-05-27T07:18:52.367 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c20bd5f347] 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c20ba5b87e] 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c20ba60e94] 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c20ba6b969] 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c20be0448c] 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c20be0d9cb] 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 13: (Thread::entry_wrapper()+0x43) [0x7feba421b72d] 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 14: (Thread::_entry_func(void*)+0xd) [0x7feba421b749] 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7feba2dbf1ca] 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 16: clone() 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 0> 2024-05-27T07:18:52.259+0000 7feb8e800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: in thread 7feb8e800700 thread_name:md_log_replay 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.368 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feba2dc9d20] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2: gsignal() 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 3: abort() 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7feba424cced] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7feba424cdd1] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c20bde023d] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c20bdcb971] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c20bdc9fa0] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c20bdca651] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c20bd5f347] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c20ba5b87e] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c20ba60e94] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c20ba6b969] 2024-05-27T07:18:52.369 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c20be0448c] 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c20be0d9cb] 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 16: (Thread::entry_wrapper()+0x43) [0x7feba421b72d] 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 17: (Thread::_entry_func(void*)+0xd) [0x7feba421b749] 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7feba2dbf1ca] 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 19: clone() 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: -1> 2024-05-27T07:18:52.255+0000 7feb8e800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7feb8e800700 time 2024-05-27T07:18:52.251999+0000 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:52.370 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7feba424cba6] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7feba424cdd1] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c20bde023d] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55c20bdcb971] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 5: (SnapRealm::check_cache() const+0x13e) [0x55c20bdc9fa0] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c20bdca651] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c20bd5f347] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c20ba5b87e] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c20ba60e94] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55c20ba6b969] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 11: (MDLog::_replay_thread()+0x1f38) [0x55c20be0448c] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55c20be0d9cb] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 13: (Thread::entry_wrapper()+0x43) [0x7feba421b72d] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 14: (Thread::_entry_func(void*)+0xd) [0x7feba421b749] 2024-05-27T07:18:52.371 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7feba2dbf1ca] 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 16: clone() 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 0> 2024-05-27T07:18:52.259+0000 7feb8e800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: in thread 7feb8e800700 thread_name:md_log_replay 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7feba2dc9d20] 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2: gsignal() 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 3: abort() 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7feba424cced] 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7feba424cdd1] 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55c20bde023d] 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55c20bdcb971] 2024-05-27T07:18:52.372 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 8: (SnapRealm::check_cache() const+0x13e) [0x55c20bdc9fa0] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55c20bdca651] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55c20bd5f347] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55c20ba5b87e] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55c20ba60e94] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55c20ba6b969] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 14: (MDLog::_replay_thread()+0x1f38) [0x55c20be0448c] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55c20be0d9cb] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 16: (Thread::entry_wrapper()+0x43) [0x7feba421b72d] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 17: (Thread::_entry_func(void*)+0xd) [0x7feba421b749] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7feba2dbf1ca] 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 19: clone() 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:18:52.373 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101184]: 2024-05-27T07:18:52.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:52 smithi179 ceph-mon[35888]: pgmap v169: 129 pgs: 129 active+clean; 44 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 608 KiB/s wr, 53 op/s 2024-05-27T07:18:52.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:52 smithi179 ceph-mon[35888]: daemon mds.j restarted 2024-05-27T07:18:52.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:52 smithi132 ceph-mon[35283]: pgmap v169: 129 pgs: 129 active+clean; 44 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 608 KiB/s wr, 53 op/s 2024-05-27T07:18:52.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:52 smithi132 ceph-mon[35283]: daemon mds.j restarted 2024-05-27T07:18:52.702 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:52 smithi043 podman[101219]: 2024-05-27 07:18:52.373966142 +0000 UTC m=+0.034770461 container died d41213e3f5d6038d3bfd1acc46f6e0b825e2f3eee9396505a850bd5f0330012f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T07:18:53.191 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21754 0.67 MB/sec execute 57 sec latency 51.043 ms 2024-05-27T07:18:53.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:53 smithi043 podman[101219]: 2024-05-27 07:18:53.213210201 +0000 UTC m=+0.874014565 container remove d41213e3f5d6038d3bfd1acc46f6e0b825e2f3eee9396505a850bd5f0330012f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:18:53.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:53 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:18:53.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:53 smithi043 ceph-mon[31531]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:18:53.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:53 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/3078967333,v1:172.21.15.43:6835/3078967333] up:boot 2024-05-27T07:18:53.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:53 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:53.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:53 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:53.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:53 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:53.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:53 smithi179 ceph-mon[35888]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:18:53.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:53 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/3078967333,v1:172.21.15.43:6835/3078967333] up:boot 2024-05-27T07:18:53.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:53 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:53.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:53 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:53.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:53 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:53.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:53 smithi132 ceph-mon[35283]: osdmap e110: 12 total, 12 up, 12 in 2024-05-27T07:18:53.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:53 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/3078967333,v1:172.21.15.43:6835/3078967333] up:boot 2024-05-27T07:18:53.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:53 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:18:53.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:53 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:18:53.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:53 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:18:54.131 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:53 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:18:54.131 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:18:53 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.638s CPU time. 2024-05-27T07:18:54.191 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21811 0.66 MB/sec execute 58 sec latency 56.360 ms 2024-05-27T07:18:54.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:54 smithi043 ceph-mon[31531]: pgmap v171: 129 pgs: 129 active+clean; 44 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 477 KiB/s wr, 41 op/s 2024-05-27T07:18:54.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:54 smithi179 ceph-mon[35888]: pgmap v171: 129 pgs: 129 active+clean; 44 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 477 KiB/s wr, 41 op/s 2024-05-27T07:18:54.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:54 smithi132 ceph-mon[35283]: pgmap v171: 129 pgs: 129 active+clean; 44 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 477 KiB/s wr, 41 op/s 2024-05-27T07:18:55.192 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21889 0.65 MB/sec execute 59 sec latency 57.814 ms 2024-05-27T07:18:55.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:18:55.749 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:18:56.192 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21985 0.64 MB/sec execute 60 sec latency 52.007 ms 2024-05-27T07:18:56.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:56 smithi043 ceph-mon[31531]: pgmap v172: 129 pgs: 129 active+clean; 44 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 477 KiB/s wr, 41 op/s 2024-05-27T07:18:56.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:56 smithi179 ceph-mon[35888]: pgmap v172: 129 pgs: 129 active+clean; 44 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 477 KiB/s wr, 41 op/s 2024-05-27T07:18:56.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:56 smithi132 ceph-mon[35283]: pgmap v172: 129 pgs: 129 active+clean; 44 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 477 KiB/s wr, 41 op/s 2024-05-27T07:18:57.192 INFO:tasks.workunit.client.0.smithi043.stdout: 1 22109 0.63 MB/sec execute 61 sec latency 62.053 ms 2024-05-27T07:18:58.192 INFO:tasks.workunit.client.0.smithi043.stdout: 1 22237 0.63 MB/sec execute 62 sec latency 57.268 ms 2024-05-27T07:18:58.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.370+0000 7f5248e00700 1 -- 172.21.15.43:0/2336015050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5244107de0 msgr2=0x7f52441081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.370+0000 7f5248e00700 1 --2- 172.21.15.43:0/2336015050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5244107de0 0x7f52441081f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f52300099c0 tx=0x7f5230009cd0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.370+0000 7f5248e00700 1 -- 172.21.15.43:0/2336015050 shutdown_connections 2024-05-27T07:18:58.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.370+0000 7f5248e00700 1 --2- 172.21.15.43:0/2336015050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5244108730 0x7f5244110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.370+0000 7f5248e00700 1 --2- 172.21.15.43:0/2336015050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52441034b0 0x7f52441038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.370+0000 7f5248e00700 1 --2- 172.21.15.43:0/2336015050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5244107de0 0x7f52441081f0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.370+0000 7f5248e00700 1 -- 172.21.15.43:0/2336015050 >> 172.21.15.43:0/2336015050 conn(0x7f52440fedb0 msgr2=0x7f52441011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:58.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.370+0000 7f5248e00700 1 -- 172.21.15.43:0/2336015050 shutdown_connections 2024-05-27T07:18:58.362 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:18:58 smithi043 ceph-mon[31531]: pgmap v173: 129 pgs: 129 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 377 KiB/s wr, 45 op/s 2024-05-27T07:18:58.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.371+0000 7f5248e00700 1 -- 172.21.15.43:0/2336015050 wait complete. 2024-05-27T07:18:58.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.372+0000 7f5248e00700 1 Processor -- start 2024-05-27T07:18:58.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.372+0000 7f5248e00700 1 -- start start 2024-05-27T07:18:58.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.373+0000 7f5248e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52441034b0 0x7f52440ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.373+0000 7f5248e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5244107de0 0x7f52440ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.373+0000 7f5248e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5244108730 0x7f52440ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.373+0000 7f5248e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5244113240 con 0x7f52441034b0 2024-05-27T07:18:58.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.373+0000 7f5248e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52441130c0 con 0x7f5244108730 2024-05-27T07:18:58.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.373+0000 7f5248e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f52441133c0 con 0x7f5244107de0 2024-05-27T07:18:58.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.373+0000 7f5241e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52441034b0 0x7f52440ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.373+0000 7f5241e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52441034b0 0x7f52440ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32896/0 (socket says 172.21.15.43:32896) 2024-05-27T07:18:58.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.373+0000 7f5241e00700 1 -- 172.21.15.43:0/4040084063 learned_addr learned my addr 172.21.15.43:0/4040084063 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:58.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.374+0000 7f5241400700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5244107de0 0x7f52440ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.374+0000 7f5242800700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5244108730 0x7f52440ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.374+0000 7f5241e00700 1 -- 172.21.15.43:0/4040084063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5244107de0 msgr2=0x7f52440ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.374+0000 7f5241e00700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5244107de0 0x7f52440ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.374+0000 7f5241e00700 1 -- 172.21.15.43:0/4040084063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5244108730 msgr2=0x7f52440ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.374+0000 7f5241e00700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5244108730 0x7f52440ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.374+0000 7f5241e00700 1 -- 172.21.15.43:0/4040084063 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52441a90c0 con 0x7f52441034b0 2024-05-27T07:18:58.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.374+0000 7f5241e00700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52441034b0 0x7f52440ff290 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f523400d950 tx=0x7f523400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:58.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.375+0000 7f523aa00700 1 -- 172.21.15.43:0/4040084063 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234016070 con 0x7f52441034b0 2024-05-27T07:18:58.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.375+0000 7f523aa00700 1 -- 172.21.15.43:0/4040084063 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5234010460 con 0x7f52441034b0 2024-05-27T07:18:58.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.375+0000 7f5248e00700 1 -- 172.21.15.43:0/4040084063 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52441a93b0 con 0x7f52441034b0 2024-05-27T07:18:58.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.375+0000 7f523aa00700 1 -- 172.21.15.43:0/4040084063 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523400f620 con 0x7f52441034b0 2024-05-27T07:18:58.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.375+0000 7f5248e00700 1 -- 172.21.15.43:0/4040084063 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52441a98c0 con 0x7f52441034b0 2024-05-27T07:18:58.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.379+0000 7f523aa00700 1 -- 172.21.15.43:0/4040084063 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5234019070 con 0x7f52441034b0 2024-05-27T07:18:58.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.380+0000 7f52579eeb80 1 Processor -- start 2024-05-27T07:18:58.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.381+0000 7f52579eeb80 1 -- start start 2024-05-27T07:18:58.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.381+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 0x5566b83eb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.381+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83eb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.381+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83ebe70 0x5566b83f8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.381+0000 7f52579eeb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5566b8319e40 con 0x5566b826ce40 2024-05-27T07:18:58.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.381+0000 7f52579eeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5566b83099e0 con 0x5566b83ebe70 2024-05-27T07:18:58.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.381+0000 7f52579eeb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5566b830da20 con 0x5566b83589f0 2024-05-27T07:18:58.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.382+0000 7f523aa00700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52280b29f0 0x7f52280b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.382+0000 7f5241e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83eb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.382+0000 7f5241e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83eb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32900/0 (socket says 172.21.15.43:32900) 2024-05-27T07:18:58.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.382+0000 7f5241e00700 1 -- 172.21.15.43:0/545169400 learned_addr learned my addr 172.21.15.43:0/545169400 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:58.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.382+0000 7f523aa00700 1 -- 172.21.15.43:0/4040084063 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9408+0+0 (secure 0 0 0) 0x7f52340e2610 con 0x7f52441034b0 2024-05-27T07:18:58.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.382+0000 7f5242800700 1 --2- 172.21.15.43:0/545169400 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 0x5566b83eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.382+0000 7f5241400700 1 --2- 172.21.15.43:0/545169400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83ebe70 0x5566b83f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.382+0000 7f5242800700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52280b29f0 0x7f52280b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.383+0000 7f5241e00700 1 -- 172.21.15.43:0/545169400 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 msgr2=0x5566b83eb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:58.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.383+0000 7f5241e00700 1 --2- 172.21.15.43:0/545169400 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 0x5566b83eb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.383+0000 7f5241e00700 1 -- 172.21.15.43:0/545169400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83ebe70 msgr2=0x5566b83f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:58.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.383+0000 7f5241e00700 1 --2- 172.21.15.43:0/545169400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83ebe70 0x5566b83f8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.383+0000 7f5241e00700 1 -- 172.21.15.43:0/545169400 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566b82d2420 con 0x5566b826ce40 2024-05-27T07:18:58.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.383+0000 7f5241e00700 1 --2- 172.21.15.43:0/545169400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83eb870 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f52340a94d0 tx=0x7f5234075e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd3a7b0c1ffd787 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:58.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.383+0000 7f522f400700 1 -- 172.21.15.43:0/545169400 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234091110 con 0x5566b826ce40 2024-05-27T07:18:58.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.383+0000 7f5242800700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52280b29f0 0x7f52280b4f00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x5566b83edff0 tx=0x7f523c009680 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:58.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.383+0000 7f522f400700 1 -- 172.21.15.43:0/545169400 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52340dfe30 con 0x5566b826ce40 2024-05-27T07:18:58.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.383+0000 7f522f400700 1 -- 172.21.15.43:0/545169400 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523406ecc0 con 0x5566b826ce40 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.384+0000 7f52579eeb80 1 -- 172.21.15.43:0/545169400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 msgr2=0x5566b83eb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.384+0000 7f52579eeb80 1 --2- 172.21.15.43:0/545169400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83eb870 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f52340a94d0 tx=0x7f5234075e70 comp rx=0 tx=0).stop 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.385+0000 7f52579eeb80 1 -- 172.21.15.43:0/545169400 shutdown_connections 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.385+0000 7f52579eeb80 1 --2- 172.21.15.43:0/545169400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83ebe70 0x5566b83f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.385+0000 7f52579eeb80 1 --2- 172.21.15.43:0/545169400 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 0x5566b83eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.385+0000 7f52579eeb80 1 --2- 172.21.15.43:0/545169400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83eb870 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.385+0000 7f52579eeb80 1 -- 172.21.15.43:0/545169400 >> 172.21.15.43:0/545169400 conn(0x5566b8306390 msgr2=0x5566b83e74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.385+0000 7f52579eeb80 1 -- 172.21.15.43:0/545169400 shutdown_connections 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.385+0000 7f52579eeb80 1 -- 172.21.15.43:0/545169400 wait complete. 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.385+0000 7f52579eeb80 1 -- start start 2024-05-27T07:18:58.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.386+0000 7f52579eeb80 1 Processor -- start 2024-05-27T07:18:58.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.386+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83f5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.386+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 0x5566b83f5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.386+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83ebe70 0x5566b83f6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.386+0000 7f52579eeb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5566b820ffa0 con 0x5566b826ce40 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.386+0000 7f52579eeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5566b82aa760 con 0x5566b83ebe70 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.386+0000 7f52579eeb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5566b83f8b80 con 0x5566b83589f0 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.386+0000 7f52579eeb80 10 client.0 _ll_register_callbacks cb 0x5566b82b3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5241e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5241e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83f5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32904/0 (socket says 172.21.15.43:32904) 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5241e00700 1 -- 172.21.15.43:0/3348782652 learned_addr learned my addr 172.21.15.43:0/3348782652 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5241400700 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 0x5566b83f5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5242800700 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83ebe70 0x5566b83f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5241e00700 1 -- 172.21.15.43:0/3348782652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 msgr2=0x5566b83f5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5241e00700 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 0x5566b83f5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5241e00700 1 -- 172.21.15.43:0/3348782652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83ebe70 msgr2=0x5566b83f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5241e00700 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83ebe70 0x5566b83f6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5241e00700 1 -- 172.21.15.43:0/3348782652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566b82598d0 con 0x5566b826ce40 2024-05-27T07:18:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.387+0000 7f5241400700 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 0x5566b83f5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.388+0000 7f5241e00700 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83f5710 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f52340a55c0 tx=0x7f52340a9b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c0452ecd0012b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:58.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.388+0000 7f5223e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.388+0000 7f5223e00700 1 -- 172.21.15.43:0/3348782652 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52340e2070 con 0x5566b826ce40 2024-05-27T07:18:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.388+0000 7f5223e00700 1 -- 172.21.15.43:0/3348782652 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9408+0+0 (secure 0 0 0) 0x7f52340b1f10 con 0x5566b826ce40 2024-05-27T07:18:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.388+0000 7f52579eeb80 1 -- 172.21.15.43:0/3348782652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52340dfe30 con 0x5566b826ce40 2024-05-27T07:18:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.388+0000 7f5223e00700 1 -- 172.21.15.43:0/3348782652 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f5234091110 con 0x5566b826ce40 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.388+0000 7f52579eeb80 10 client.16206 fetch_fsmap learned FSMap version 76 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.388+0000 7f52579eeb80 1 -- 172.21.15.43:0/3348782652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x5566b82da060 con 0x5566b826ce40 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.389+0000 7f5223e00700 1 -- 172.21.15.43:0/3348782652 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52340dfe30 con 0x5566b826ce40 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.389+0000 7f5223e00700 1 -- 172.21.15.43:0/3348782652 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 76) ==== 6559+0+0 (secure 0 0 0) 0x7f52340a87a0 con 0x5566b826ce40 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.389+0000 7f52579eeb80 10 client.16206 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.389+0000 7f52579eeb80 10 client.16206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.389+0000 7f52579eeb80 20 client.16206 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.389+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5566b84059b0 0x5566b8407dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.389+0000 7f52579eeb80 4 client.16206 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.389+0000 7f52579eeb80 1 -- 172.21.15.43:0/3348782652 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566b831bd90 con 0x5566b84059b0 2024-05-27T07:18:58.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.390+0000 7f5241400700 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5566b84059b0 0x5566b8407dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.391+0000 7f5223e00700 10 client.16206 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:58.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.391+0000 7f5241400700 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5566b84059b0 0x5566b8407dd0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdb9ec96fbd27873 server_cookie=1d37a777ecc9304f in_seq=0 out_seq=0 2024-05-27T07:18:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.393+0000 7f5223e00700 1 -- 172.21.15.43:0/3348782652 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5566b831bd90 con 0x5566b84059b0 2024-05-27T07:18:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.393+0000 7f5223e00700 10 client.16206 handle_command_reply: tid=0 2024-05-27T07:18:58.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.393+0000 7f52579eeb80 1 client.16206 shutdown 2024-05-27T07:18:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 1 -- 172.21.15.43:0/3348782652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 msgr2=0x5566b83f5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83f5710 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f52340a55c0 tx=0x7f52340a9b00 comp rx=0 tx=0).stop 2024-05-27T07:18:58.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 1 -- 172.21.15.43:0/3348782652 shutdown_connections 2024-05-27T07:18:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83ebe70 0x5566b83f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5566b84059b0 0x5566b8407dd0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b826ce40 0x5566b83f5710 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3348782652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83589f0 0x5566b83f5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f5223e00700 0 client.16206 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 1 -- 172.21.15.43:0/3348782652 >> 172.21.15.43:0/3348782652 conn(0x5566b8306390 msgr2=0x5566b83eb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 1 -- 172.21.15.43:0/3348782652 shutdown_connections 2024-05-27T07:18:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 1 -- 172.21.15.43:0/3348782652 wait complete. 2024-05-27T07:18:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.394+0000 7f52579eeb80 20 client.16206 trim_cache size 0 max 16384 2024-05-27T07:18:58.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.417+0000 7f52579eeb80 1 Processor -- start 2024-05-27T07:18:58.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.417+0000 7f52579eeb80 1 -- start start 2024-05-27T07:18:58.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.417+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b826ce40 0x5566b83f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.417+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83589f0 0x5566b83eb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.418+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83ebe70 0x5566b83f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.418+0000 7f52579eeb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f52441b4140 con 0x5566b83ebe70 2024-05-27T07:18:58.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.418+0000 7f52579eeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52441b1190 con 0x5566b83589f0 2024-05-27T07:18:58.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.418+0000 7f52579eeb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f52441b33a0 con 0x5566b826ce40 2024-05-27T07:18:58.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.418+0000 7f5242800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83ebe70 0x5566b83f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.418+0000 7f5242800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83ebe70 0x5566b83f5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32906/0 (socket says 172.21.15.43:32906) 2024-05-27T07:18:58.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.418+0000 7f5242800700 1 -- 172.21.15.43:0/4020444342 learned_addr learned my addr 172.21.15.43:0/4020444342 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:58.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.418+0000 7f5241e00700 1 --2- 172.21.15.43:0/4020444342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b826ce40 0x5566b83f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.418+0000 7f5241400700 1 --2- 172.21.15.43:0/4020444342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83589f0 0x5566b83eb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.419+0000 7f5242800700 1 -- 172.21.15.43:0/4020444342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b826ce40 msgr2=0x5566b83f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.419+0000 7f5242800700 1 --2- 172.21.15.43:0/4020444342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b826ce40 0x5566b83f0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.419+0000 7f5242800700 1 -- 172.21.15.43:0/4020444342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83589f0 msgr2=0x5566b83eb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.419+0000 7f5242800700 1 --2- 172.21.15.43:0/4020444342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83589f0 0x5566b83eb7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.419+0000 7f5242800700 1 -- 172.21.15.43:0/4020444342 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566b83e89c0 con 0x5566b83ebe70 2024-05-27T07:18:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.419+0000 7f5242800700 1 --2- 172.21.15.43:0/4020444342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83ebe70 0x5566b83f5de0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f523c0189b0 tx=0x7f523c009270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2bf7bc907d7bac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.419+0000 7f522ea00700 1 -- 172.21.15.43:0/4020444342 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c004bb0 con 0x5566b83ebe70 2024-05-27T07:18:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.420+0000 7f522ea00700 1 -- 172.21.15.43:0/4020444342 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f523c004d50 con 0x5566b83ebe70 2024-05-27T07:18:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.420+0000 7f522ea00700 1 -- 172.21.15.43:0/4020444342 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c0051e0 con 0x5566b83ebe70 2024-05-27T07:18:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.421+0000 7f52579eeb80 1 -- 172.21.15.43:0/4020444342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83ebe70 msgr2=0x5566b83f5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.421+0000 7f52579eeb80 1 --2- 172.21.15.43:0/4020444342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83ebe70 0x5566b83f5de0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f523c0189b0 tx=0x7f523c009270 comp rx=0 tx=0).stop 2024-05-27T07:18:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.421+0000 7f52579eeb80 1 -- 172.21.15.43:0/4020444342 shutdown_connections 2024-05-27T07:18:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.421+0000 7f52579eeb80 1 --2- 172.21.15.43:0/4020444342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83ebe70 0x5566b83f5de0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.421+0000 7f52579eeb80 1 --2- 172.21.15.43:0/4020444342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b826ce40 0x5566b83f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.421+0000 7f52579eeb80 1 --2- 172.21.15.43:0/4020444342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b83589f0 0x5566b83eb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.421+0000 7f52579eeb80 1 -- 172.21.15.43:0/4020444342 >> 172.21.15.43:0/4020444342 conn(0x5566b8306390 msgr2=0x5566b83f56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.421+0000 7f52579eeb80 1 -- 172.21.15.43:0/4020444342 shutdown_connections 2024-05-27T07:18:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.421+0000 7f52579eeb80 1 -- 172.21.15.43:0/4020444342 wait complete. 2024-05-27T07:18:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.421+0000 7f52579eeb80 1 -- start start 2024-05-27T07:18:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.422+0000 7f52579eeb80 1 Processor -- start 2024-05-27T07:18:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.422+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b826ce40 0x5566b83fe470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.422+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83589f0 0x5566b83efeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.422+0000 7f52579eeb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83ebe70 0x5566b83f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.422+0000 7f52579eeb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f52441b2cd0 con 0x5566b83589f0 2024-05-27T07:18:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.422+0000 7f52579eeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52441b1860 con 0x5566b826ce40 2024-05-27T07:18:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.422+0000 7f52579eeb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f52441b1f30 con 0x5566b83ebe70 2024-05-27T07:18:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.422+0000 7f52579eeb80 10 client.0 _ll_register_callbacks cb 0x5566b82b3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:18:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5241400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83589f0 0x5566b83efeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5241400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83589f0 0x5566b83efeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32918/0 (socket says 172.21.15.43:32918) 2024-05-27T07:18:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5241400700 1 -- 172.21.15.43:0/3761918671 learned_addr learned my addr 172.21.15.43:0/3761918671 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:18:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5241e00700 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b826ce40 0x5566b83fe470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5242800700 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83ebe70 0x5566b83f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5241400700 1 -- 172.21.15.43:0/3761918671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83ebe70 msgr2=0x5566b83f03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5241400700 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83ebe70 0x5566b83f03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5241400700 1 -- 172.21.15.43:0/3761918671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b826ce40 msgr2=0x5566b83fe470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5241400700 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b826ce40 0x5566b83fe470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5241400700 1 -- 172.21.15.43:0/3761918671 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566b83f0b20 con 0x5566b83589f0 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.423+0000 7f5241e00700 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b826ce40 0x5566b83fe470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.424+0000 7f5241400700 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83589f0 0x5566b83efeb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5230005b40 tx=0x7f5230005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341bc18d44121427 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.424+0000 7f5223e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.424+0000 7f5223e00700 1 -- 172.21.15.43:0/3761918671 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523000bba0 con 0x5566b83589f0 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.424+0000 7f5223e00700 1 -- 172.21.15.43:0/3761918671 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9408+0+0 (secure 0 0 0) 0x7f523000bd40 con 0x5566b83589f0 2024-05-27T07:18:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.424+0000 7f52579eeb80 1 -- 172.21.15.43:0/3761918671 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f523c004d50 con 0x5566b83589f0 2024-05-27T07:18:58.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.424+0000 7f5223e00700 1 -- 172.21.15.43:0/3761918671 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f5230003680 con 0x5566b83589f0 2024-05-27T07:18:58.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.424+0000 7f52579eeb80 10 client.16212 fetch_fsmap learned FSMap version 76 2024-05-27T07:18:58.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.424+0000 7f52579eeb80 1 -- 172.21.15.43:0/3761918671 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x5566b8478b80 con 0x5566b83589f0 2024-05-27T07:18:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.425+0000 7f5223e00700 1 -- 172.21.15.43:0/3761918671 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f523c004d50 con 0x5566b83589f0 2024-05-27T07:18:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.425+0000 7f5223e00700 1 -- 172.21.15.43:0/3761918671 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 76) ==== 6559+0+0 (secure 0 0 0) 0x7f523000f7d0 con 0x5566b83589f0 2024-05-27T07:18:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.425+0000 7f52579eeb80 10 client.16212 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:18:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.425+0000 7f52579eeb80 10 client.16212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:18:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.425+0000 7f52579eeb80 20 client.16212 populate_metadata read hostname 'smithi043' 2024-05-27T07:18:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.425+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5566b8479200 0x5566b847b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:18:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.425+0000 7f52579eeb80 4 client.16212 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:18:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.425+0000 7f52579eeb80 1 -- 172.21.15.43:0/3761918671 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5566b8315570 con 0x5566b8479200 2024-05-27T07:18:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.426+0000 7f5241e00700 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5566b8479200 0x5566b847b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:18:58.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.427+0000 7f5241e00700 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5566b8479200 0x5566b847b620 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10ebf45ca2404b3f server_cookie=e5e915bdbb83ff3e in_seq=0 out_seq=0 2024-05-27T07:18:58.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.427+0000 7f5223e00700 10 client.16212 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f5223e00700 1 -- 172.21.15.43:0/3761918671 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5566b8315570 con 0x5566b8479200 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f5223e00700 10 client.16212 handle_command_reply: tid=0 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 client.16212 shutdown 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 -- 172.21.15.43:0/3761918671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83589f0 msgr2=0x5566b83efeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83589f0 0x5566b83efeb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5230005b40 tx=0x7f5230005ad0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 -- 172.21.15.43:0/3761918671 shutdown_connections 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5566b83ebe70 0x5566b83f03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5566b826ce40 0x5566b83fe470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5566b8479200 0x5566b847b620 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 --2- 172.21.15.43:0/3761918671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5566b83589f0 0x5566b83efeb0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 -- 172.21.15.43:0/3761918671 >> 172.21.15.43:0/3761918671 conn(0x5566b8306390 msgr2=0x5566b83f6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:58.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f5223e00700 0 client.16212 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:18:58.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 -- 172.21.15.43:0/3761918671 shutdown_connections 2024-05-27T07:18:58.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.430+0000 7f52579eeb80 1 -- 172.21.15.43:0/3761918671 wait complete. 2024-05-27T07:18:58.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.431+0000 7f52579eeb80 20 client.16212 trim_cache size 0 max 16384 2024-05-27T07:18:58.422 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:18:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:18:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:18:58.422 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:18:58.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 -- 172.21.15.43:0/4040084063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52280b29f0 msgr2=0x7f52280b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52280b29f0 0x7f52280b4f00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x5566b83edff0 tx=0x7f523c009680 comp rx=0 tx=0).stop 2024-05-27T07:18:58.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 -- 172.21.15.43:0/4040084063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52441034b0 msgr2=0x7f52440ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:18:58.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52441034b0 0x7f52440ff290 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f523400d950 tx=0x7f523400dd10 comp rx=0 tx=0).stop 2024-05-27T07:18:58.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 -- 172.21.15.43:0/4040084063 shutdown_connections 2024-05-27T07:18:58.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52280b29f0 0x7f52280b4f00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5244108730 0x7f52440ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52441034b0 0x7f52440ff290 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 --2- 172.21.15.43:0/4040084063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5244107de0 0x7f52440ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:18:58.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 -- 172.21.15.43:0/4040084063 >> 172.21.15.43:0/4040084063 conn(0x7f52440fedb0 msgr2=0x7f524406b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:18:58.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 -- 172.21.15.43:0/4040084063 shutdown_connections 2024-05-27T07:18:58.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:18:58.432+0000 7f5248e00700 1 -- 172.21.15.43:0/4040084063 wait complete. 2024-05-27T07:18:58.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:18:58 smithi179 ceph-mon[35888]: pgmap v173: 129 pgs: 129 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 377 KiB/s wr, 45 op/s 2024-05-27T07:18:58.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:18:58 smithi132 ceph-mon[35283]: pgmap v173: 129 pgs: 129 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 377 KiB/s wr, 45 op/s 2024-05-27T07:18:59.193 INFO:tasks.workunit.client.0.smithi043.stdout: 1 22362 0.64 MB/sec execute 63 sec latency 51.003 ms 2024-05-27T07:18:59.387 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:18:59.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:18:59.876 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:00.192 INFO:tasks.workunit.client.0.smithi043.stdout: 1 22481 0.63 MB/sec execute 64 sec latency 47.706 ms 2024-05-27T07:19:00.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:00 smithi043 ceph-mon[31531]: pgmap v174: 129 pgs: 129 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 377 KiB/s wr, 45 op/s 2024-05-27T07:19:00.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:00 smithi179 ceph-mon[35888]: pgmap v174: 129 pgs: 129 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 377 KiB/s wr, 45 op/s 2024-05-27T07:19:00.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:00 smithi132 ceph-mon[35283]: pgmap v174: 129 pgs: 129 active+clean; 45 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 377 KiB/s wr, 45 op/s 2024-05-27T07:19:01.192 INFO:tasks.workunit.client.0.smithi043.stdout: 1 22582 0.62 MB/sec execute 65 sec latency 51.065 ms 2024-05-27T07:19:02.192 INFO:tasks.workunit.client.0.smithi043.stdout: 1 22752 0.66 MB/sec execute 66 sec latency 61.354 ms 2024-05-27T07:19:02.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.226+0000 7f211fe00700 1 -- 172.21.15.43:0/3133312667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2120108d20 msgr2=0x7f2120109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:02.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.226+0000 7f211fe00700 1 --2- 172.21.15.43:0/3133312667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2120108d20 0x7f2120109130 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f21080099c0 tx=0x7f2108009cd0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.226+0000 7f211fe00700 1 -- 172.21.15.43:0/3133312667 shutdown_connections 2024-05-27T07:19:02.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.226+0000 7f211fe00700 1 --2- 172.21.15.43:0/3133312667 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2120109670 0x7f2120110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.226+0000 7f211fe00700 1 --2- 172.21.15.43:0/3133312667 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21201043f0 0x7f2120104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.226+0000 7f211fe00700 1 --2- 172.21.15.43:0/3133312667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2120108d20 0x7f2120109130 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.226+0000 7f211fe00700 1 -- 172.21.15.43:0/3133312667 >> 172.21.15.43:0/3133312667 conn(0x7f2120079d00 msgr2=0x7f212007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:02.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.226+0000 7f211fe00700 1 -- 172.21.15.43:0/3133312667 shutdown_connections 2024-05-27T07:19:02.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.226+0000 7f211fe00700 1 -- 172.21.15.43:0/3133312667 wait complete. 2024-05-27T07:19:02.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.227+0000 7f211fe00700 1 Processor -- start 2024-05-27T07:19:02.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.227+0000 7f211fe00700 1 -- start start 2024-05-27T07:19:02.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21201043f0 0x7f21201a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:02.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2120108d20 0x7f21201a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:02.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2120109670 0x7f212019aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:02.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211fe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2120114180 con 0x7f2120109670 2024-05-27T07:19:02.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211fe00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2120114000 con 0x7f21201043f0 2024-05-27T07:19:02.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211fe00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2120114300 con 0x7f2120108d20 2024-05-27T07:19:02.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2120109670 0x7f212019aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:02.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2120109670 0x7f212019aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32928/0 (socket says 172.21.15.43:32928) 2024-05-27T07:19:02.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211f400700 1 -- 172.21.15.43:0/2141205704 learned_addr learned my addr 172.21.15.43:0/2141205704 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:02.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211e000700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2120108d20 0x7f21201a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:02.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.228+0000 7f211ea00700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21201043f0 0x7f21201a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:02.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.229+0000 7f211f400700 1 -- 172.21.15.43:0/2141205704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2120108d20 msgr2=0x7f21201a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:02.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.229+0000 7f211f400700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2120108d20 0x7f21201a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.229+0000 7f211f400700 1 -- 172.21.15.43:0/2141205704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21201043f0 msgr2=0x7f21201a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:02.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.229+0000 7f211f400700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21201043f0 0x7f21201a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.229+0000 7f211f400700 1 -- 172.21.15.43:0/2141205704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212019b0d0 con 0x7f2120109670 2024-05-27T07:19:02.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.229+0000 7f211e000700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2120108d20 0x7f21201a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:19:02.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.229+0000 7f211f400700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2120109670 0x7f212019aa20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f211800c970 tx=0x7f211800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.229+0000 7f2117400700 1 -- 172.21.15.43:0/2141205704 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2118007b00 con 0x7f2120109670 2024-05-27T07:19:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.230+0000 7f211fe00700 1 -- 172.21.15.43:0/2141205704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f212019b3c0 con 0x7f2120109670 2024-05-27T07:19:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.230+0000 7f2117400700 1 -- 172.21.15.43:0/2141205704 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2118004cc0 con 0x7f2120109670 2024-05-27T07:19:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.230+0000 7f2117400700 1 -- 172.21.15.43:0/2141205704 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21180054c0 con 0x7f2120109670 2024-05-27T07:19:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.230+0000 7f211fe00700 1 -- 172.21.15.43:0/2141205704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21201a7690 con 0x7f2120109670 2024-05-27T07:19:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.231+0000 7f211fe00700 1 -- 172.21.15.43:0/2141205704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21200512b0 con 0x7f2120109670 2024-05-27T07:19:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.234+0000 7f2117400700 1 -- 172.21.15.43:0/2141205704 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f2118004750 con 0x7f2120109670 2024-05-27T07:19:02.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.235+0000 7f2117400700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21040b2a70 0x7f21040b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:02.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.235+0000 7f2117400700 1 -- 172.21.15.43:0/2141205704 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9408+0+0 (secure 0 0 0) 0x7f21180e2890 con 0x7f2120109670 2024-05-27T07:19:02.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.236+0000 7f211ea00700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21040b2a70 0x7f21040b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:02.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.237+0000 7f211ea00700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21040b2a70 0x7f21040b4f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f21100060b0 tx=0x7f21100077b0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.238+0000 7f2117400700 1 -- 172.21.15.43:0/2141205704 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2118014030 con 0x7f2120109670 2024-05-27T07:19:02.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:02 smithi043 ceph-mon[31531]: pgmap v175: 129 pgs: 129 active+clean; 43 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 332 KiB/s wr, 54 op/s 2024-05-27T07:19:02.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:02 smithi179 ceph-mon[35888]: pgmap v175: 129 pgs: 129 active+clean; 43 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 332 KiB/s wr, 54 op/s 2024-05-27T07:19:02.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.540+0000 7f211fe00700 1 -- 172.21.15.43:0/2141205704 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f212002e5e0 con 0x7f2120109670 2024-05-27T07:19:02.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.542+0000 7f2117400700 1 -- 172.21.15.43:0/2141205704 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+9456 (secure 0 0 0) 0x7f21180a9f40 con 0x7f2120109670 2024-05-27T07:19:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:19:02.532 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:18:52.128990+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16194":{"gid":16194,"name":"j","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/3078967333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":3078967333},{"type":"v1","addr":"172.21.15.43:6835","nonce":3078967333}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34448,34484,34424,34403,24484]},"id":1}]} 2024-05-27T07:19:02.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.545+0000 7f211fe00700 1 -- 172.21.15.43:0/2141205704 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21040b2a70 msgr2=0x7f21040b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:02.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.545+0000 7f211fe00700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21040b2a70 0x7f21040b4f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f21100060b0 tx=0x7f21100077b0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.545+0000 7f211fe00700 1 -- 172.21.15.43:0/2141205704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2120109670 msgr2=0x7f212019aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:02.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.545+0000 7f211fe00700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2120109670 0x7f212019aa20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f211800c970 tx=0x7f211800cd30 comp rx=0 tx=0).stop 2024-05-27T07:19:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.545+0000 7f211fe00700 1 -- 172.21.15.43:0/2141205704 shutdown_connections 2024-05-27T07:19:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.545+0000 7f211fe00700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21040b2a70 0x7f21040b4f80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.545+0000 7f211fe00700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2120109670 0x7f212019aa20 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.545+0000 7f211fe00700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21201043f0 0x7f21201a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.545+0000 7f211fe00700 1 --2- 172.21.15.43:0/2141205704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2120108d20 0x7f21201a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.545+0000 7f211fe00700 1 -- 172.21.15.43:0/2141205704 >> 172.21.15.43:0/2141205704 conn(0x7f2120079d00 msgr2=0x7f2120101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.546+0000 7f211fe00700 1 -- 172.21.15.43:0/2141205704 shutdown_connections 2024-05-27T07:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:02.546+0000 7f211fe00700 1 -- 172.21.15.43:0/2141205704 wait complete. 2024-05-27T07:19:02.537 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 76 2024-05-27T07:19:02.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:02 smithi132 ceph-mon[35283]: pgmap v175: 129 pgs: 129 active+clean; 43 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 332 KiB/s wr, 54 op/s 2024-05-27T07:19:03.192 INFO:tasks.workunit.client.0.smithi043.stdout: 1 22883 0.65 MB/sec execute 67 sec latency 51.767 ms 2024-05-27T07:19:03.446 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:03 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2141205704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:19:03.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:03 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2141205704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:19:03.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:19:03.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:03 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2141205704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:19:04.002 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:04.193 INFO:tasks.workunit.client.0.smithi043.stdout: 1 23021 0.64 MB/sec execute 68 sec latency 51.827 ms 2024-05-27T07:19:04.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:04 smithi043 ceph-mon[31531]: pgmap v176: 129 pgs: 129 active+clean; 43 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 317 KiB/s wr, 52 op/s 2024-05-27T07:19:04.354 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:03 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:19:04.354 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:03 smithi043 systemd[1]: Stopped Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:19:04.354 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:03 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.638s CPU time. 2024-05-27T07:19:04.355 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:03 smithi043 systemd[1]: Starting Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:19:04.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:04 smithi179 ceph-mon[35888]: pgmap v176: 129 pgs: 129 active+clean; 43 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 317 KiB/s wr, 52 op/s 2024-05-27T07:19:04.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:04 smithi132 ceph-mon[35283]: pgmap v176: 129 pgs: 129 active+clean; 43 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 317 KiB/s wr, 52 op/s 2024-05-27T07:19:05.193 INFO:tasks.workunit.client.0.smithi043.stdout: 1 23188 0.63 MB/sec execute 69 sec latency 78.848 ms 2024-05-27T07:19:05.452 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:05 smithi043 podman[101732]: 2024-05-27 07:19:05.100969182 +0000 UTC m=+0.169767658 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:19:05.878 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:05 smithi043 podman[101732]: 2024-05-27 07:19:05.666657262 +0000 UTC m=+0.735455724 container create b156c4e838ae090ba3de8fa67a5efe8164b267d584c5329cda8d181e9648d401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:19:06.193 INFO:tasks.workunit.client.0.smithi043.stdout: 1 23342 0.63 MB/sec execute 70 sec latency 56.946 ms 2024-05-27T07:19:06.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:06 smithi043 ceph-mon[31531]: pgmap v177: 129 pgs: 129 active+clean; 43 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 277 KiB/s wr, 45 op/s 2024-05-27T07:19:06.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:06 smithi043 podman[101732]: 2024-05-27 07:19:06.217566151 +0000 UTC m=+1.286364672 container init b156c4e838ae090ba3de8fa67a5efe8164b267d584c5329cda8d181e9648d401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-27T07:19:06.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:06 smithi043 podman[101732]: 2024-05-27 07:19:06.228744935 +0000 UTC m=+1.297543377 container start b156c4e838ae090ba3de8fa67a5efe8164b267d584c5329cda8d181e9648d401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True) 2024-05-27T07:19:06.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:06 smithi043 bash[101732]: b156c4e838ae090ba3de8fa67a5efe8164b267d584c5329cda8d181e9648d401 2024-05-27T07:19:06.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:06 smithi043 systemd[1]: Started Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:19:06.453 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:06 smithi043 ceph-mds[101776]: -- 172.21.15.43:0/768330893 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d4605e3a40 con 0x55d46062ba80 2024-05-27T07:19:06.454 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:06 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: starting mds.j at 2024-05-27T07:19:06.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:06 smithi179 ceph-mon[35888]: pgmap v177: 129 pgs: 129 active+clean; 43 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 277 KiB/s wr, 45 op/s 2024-05-27T07:19:06.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:06 smithi132 ceph-mon[35283]: pgmap v177: 129 pgs: 129 active+clean; 43 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 277 KiB/s wr, 45 op/s 2024-05-27T07:19:06.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.702+0000 7f7c7b000700 1 -- 172.21.15.43:0/110661084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c7410c5b0 msgr2=0x7f7c7410c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.702+0000 7f7c7b000700 1 --2- 172.21.15.43:0/110661084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c7410c5b0 0x7f7c7410c9c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009990 tx=0x7f7c64009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.702+0000 7f7c7b000700 1 -- 172.21.15.43:0/110661084 shutdown_connections 2024-05-27T07:19:06.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.702+0000 7f7c7b000700 1 --2- 172.21.15.43:0/110661084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c74106ec0 0x7f7c74113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.702+0000 7f7c7b000700 1 --2- 172.21.15.43:0/110661084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c7410c5b0 0x7f7c7410c9c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.702+0000 7f7c7b000700 1 --2- 172.21.15.43:0/110661084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c74106570 0x7f7c74106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.702+0000 7f7c7b000700 1 -- 172.21.15.43:0/110661084 >> 172.21.15.43:0/110661084 conn(0x7f7c74102070 msgr2=0x7f7c741044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:06.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.702+0000 7f7c7b000700 1 -- 172.21.15.43:0/110661084 shutdown_connections 2024-05-27T07:19:06.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.702+0000 7f7c7b000700 1 -- 172.21.15.43:0/110661084 wait complete. 2024-05-27T07:19:06.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.703+0000 7f7c7b000700 1 Processor -- start 2024-05-27T07:19:06.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.704+0000 7f7c7b000700 1 -- start start 2024-05-27T07:19:06.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.704+0000 7f7c7b000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c74106570 0x7f7c7419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.704+0000 7f7c7b000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c74106ec0 0x7f7c7419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.704+0000 7f7c7b000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c7410c5b0 0x7f7c741a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.704+0000 7f7c7b000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74116570 con 0x7f7c74106570 2024-05-27T07:19:06.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.704+0000 7f7c7b000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741163f0 con 0x7f7c7410c5b0 2024-05-27T07:19:06.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.704+0000 7f7c7b000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741166f0 con 0x7f7c74106ec0 2024-05-27T07:19:06.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.704+0000 7f7c73e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c74106570 0x7f7c7419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.704+0000 7f7c73e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c74106570 0x7f7c7419cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50550/0 (socket says 172.21.15.43:50550) 2024-05-27T07:19:06.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.705+0000 7f7c73e00700 1 -- 172.21.15.43:0/1066525651 learned_addr learned my addr 172.21.15.43:0/1066525651 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:06.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.705+0000 7f7c73400700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c74106ec0 0x7f7c7419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.705+0000 7f7c79000700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c7410c5b0 0x7f7c741a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.705+0000 7f7c73e00700 1 -- 172.21.15.43:0/1066525651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c74106ec0 msgr2=0x7f7c7419d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.705+0000 7f7c73e00700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c74106ec0 0x7f7c7419d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.705+0000 7f7c73e00700 1 -- 172.21.15.43:0/1066525651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c7410c5b0 msgr2=0x7f7c741a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.705+0000 7f7c73e00700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c7410c5b0 0x7f7c741a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.705+0000 7f7c73e00700 1 -- 172.21.15.43:0/1066525651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c741a1df0 con 0x7f7c74106570 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.705+0000 7f7c79000700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c7410c5b0 0x7f7c741a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.705+0000 7f7c73e00700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c74106570 0x7f7c7419cf20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7c64000c00 tx=0x7f7c64005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.706+0000 7f7c70c00700 1 -- 172.21.15.43:0/1066525651 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64027070 con 0x7f7c74106570 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.706+0000 7f7c70c00700 1 -- 172.21.15.43:0/1066525651 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c6400fd40 con 0x7f7c74106570 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.706+0000 7f7c7b000700 1 -- 172.21.15.43:0/1066525651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c741a2080 con 0x7f7c74106570 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.706+0000 7f7c70c00700 1 -- 172.21.15.43:0/1066525651 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64022ca0 con 0x7f7c74106570 2024-05-27T07:19:06.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.706+0000 7f7c7b000700 1 -- 172.21.15.43:0/1066525651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c741a2560 con 0x7f7c74106570 2024-05-27T07:19:06.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.710+0000 7f7c70c00700 1 -- 172.21.15.43:0/1066525651 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7c6402a070 con 0x7f7c74106570 2024-05-27T07:19:06.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.711+0000 7f7c89ac6b80 1 Processor -- start 2024-05-27T07:19:06.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.712+0000 7f7c89ac6b80 1 -- start start 2024-05-27T07:19:06.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.712+0000 7f7c70c00700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7c580b29f0 0x7f7c580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.712+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e84330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.712+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67d05e40 0x55ac67e84870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.712+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67e84e70 0x55ac67e91110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.712+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ac67db2e40 con 0x55ac67df19f0 2024-05-27T07:19:06.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.712+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ac67da29e0 con 0x55ac67d05e40 2024-05-27T07:19:06.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.712+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ac67da6a20 con 0x55ac67e84e70 2024-05-27T07:19:06.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.712+0000 7f7c70c00700 1 -- 172.21.15.43:0/1066525651 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9408+0+0 (secure 0 0 0) 0x7f7c640e3930 con 0x7f7c74106570 2024-05-27T07:19:06.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c73e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e84330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c73e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e84330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50552/0 (socket says 172.21.15.43:50552) 2024-05-27T07:19:06.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c73e00700 1 -- 172.21.15.43:0/3733471217 learned_addr learned my addr 172.21.15.43:0/3733471217 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:06.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c79000700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7c580b29f0 0x7f7c580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c73400700 1 --2- 172.21.15.43:0/3733471217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67d05e40 0x55ac67e84870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c79000700 1 --2- 172.21.15.43:0/3733471217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67e84e70 0x55ac67e91110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c73e00700 1 -- 172.21.15.43:0/3733471217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67e84e70 msgr2=0x55ac67e91110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:06.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c73e00700 1 --2- 172.21.15.43:0/3733471217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67e84e70 0x55ac67e91110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c73e00700 1 -- 172.21.15.43:0/3733471217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67d05e40 msgr2=0x55ac67e84870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:06.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c73e00700 1 --2- 172.21.15.43:0/3733471217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67d05e40 0x55ac67e84870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.713+0000 7f7c73e00700 1 -- 172.21.15.43:0/3733471217 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac67d6b420 con 0x55ac67df19f0 2024-05-27T07:19:06.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.714+0000 7f7c73e00700 1 --2- 172.21.15.43:0/3733471217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e84330 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7c640aa5f0 tx=0x7f7c64009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5b7d46b990fd05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:06.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.714+0000 7f7c79000700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7c580b29f0 0x7f7c580b4f00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0094f0 tx=0x7f7c6c0091a0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:06.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.714+0000 7f7c69600700 1 -- 172.21.15.43:0/3733471217 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640aaff0 con 0x55ac67df19f0 2024-05-27T07:19:06.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.714+0000 7f7c69600700 1 -- 172.21.15.43:0/3733471217 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c640aa8f0 con 0x55ac67df19f0 2024-05-27T07:19:06.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.714+0000 7f7c69600700 1 -- 172.21.15.43:0/3733471217 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640aab50 con 0x55ac67df19f0 2024-05-27T07:19:06.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.716+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3733471217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 msgr2=0x55ac67e84330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.716+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/3733471217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e84330 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7c640aa5f0 tx=0x7f7c64009310 comp rx=0 tx=0).stop 2024-05-27T07:19:06.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.716+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3733471217 shutdown_connections 2024-05-27T07:19:06.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.716+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/3733471217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67e84e70 0x55ac67e91110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.716+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/3733471217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e84330 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.716+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/3733471217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67d05e40 0x55ac67e84870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.716+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3733471217 >> 172.21.15.43:0/3733471217 conn(0x55ac67d9f390 msgr2=0x55ac67e804c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:06.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.716+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3733471217 shutdown_connections 2024-05-27T07:19:06.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.716+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3733471217 wait complete. 2024-05-27T07:19:06.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.717+0000 7f7c89ac6b80 1 -- start start 2024-05-27T07:19:06.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.717+0000 7f7c89ac6b80 1 Processor -- start 2024-05-27T07:19:06.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.718+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e8e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.718+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e8eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.718+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67e84e70 0x55ac67e8f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.718+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ac67ca8fa0 con 0x55ac67df19f0 2024-05-27T07:19:06.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.718+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ac67d43760 con 0x55ac67e84e70 2024-05-27T07:19:06.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.718+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ac67e91940 con 0x55ac67d05e40 2024-05-27T07:19:06.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.718+0000 7f7c89ac6b80 10 client.0 _ll_register_callbacks cb 0x55ac67d4cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:06.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.718+0000 7f7c73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e8eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.718+0000 7f7c73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e8eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50558/0 (socket says 172.21.15.43:50558) 2024-05-27T07:19:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.718+0000 7f7c73400700 1 -- 172.21.15.43:0/3827514307 learned_addr learned my addr 172.21.15.43:0/3827514307 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.719+0000 7f7c73e00700 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e8e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.719+0000 7f7c79000700 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67e84e70 0x55ac67e8f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.719+0000 7f7c73400700 1 -- 172.21.15.43:0/3827514307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 msgr2=0x55ac67e8e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.719+0000 7f7c73400700 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e8e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.719+0000 7f7c73400700 1 -- 172.21.15.43:0/3827514307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67e84e70 msgr2=0x55ac67e8f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.719+0000 7f7c73400700 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67e84e70 0x55ac67e8f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.719+0000 7f7c73400700 1 -- 172.21.15.43:0/3827514307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac67cf28d0 con 0x55ac67df19f0 2024-05-27T07:19:06.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.719+0000 7f7c73400700 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e8eba0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000be70 tx=0x7f7c600116c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6621549093eca4d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:06.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.719+0000 7f7c5e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:06.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.720+0000 7f7c5e000700 1 -- 172.21.15.43:0/3827514307 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60011e70 con 0x55ac67df19f0 2024-05-27T07:19:06.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.720+0000 7f7c5e000700 1 -- 172.21.15.43:0/3827514307 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9408+0+0 (secure 0 0 0) 0x7f7c60004020 con 0x55ac67df19f0 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.720+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3827514307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c640aa8f0 con 0x55ac67df19f0 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.720+0000 7f7c5e000700 1 -- 172.21.15.43:0/3827514307 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f7c60024050 con 0x55ac67df19f0 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.720+0000 7f7c89ac6b80 10 client.16236 fetch_fsmap learned FSMap version 76 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.720+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3827514307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55ac67d73060 con 0x55ac67df19f0 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.721+0000 7f7c5e000700 1 -- 172.21.15.43:0/3827514307 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c640aa8f0 con 0x55ac67df19f0 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.721+0000 7f7c5e000700 1 -- 172.21.15.43:0/3827514307 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 76) ==== 6559+0+0 (secure 0 0 0) 0x7f7c6001b480 con 0x55ac67df19f0 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.721+0000 7f7c89ac6b80 10 client.16236 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.721+0000 7f7c89ac6b80 10 client.16236 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.721+0000 7f7c89ac6b80 20 client.16236 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.721+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ac67e9ea70 0x55ac67ea0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.721+0000 7f7c89ac6b80 4 client.16236 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.721+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3827514307 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac67db4d90 con 0x55ac67e9ea70 2024-05-27T07:19:06.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.722+0000 7f7c73e00700 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ac67e9ea70 0x55ac67ea0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.723+0000 7f7c73e00700 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ac67e9ea70 0x55ac67ea0ec0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f7fc5df9d30de8c6 server_cookie=39eab305d7749071 in_seq=0 out_seq=0 2024-05-27T07:19:06.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.723+0000 7f7c5e000700 10 client.16236 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:19:06.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c5e000700 1 -- 172.21.15.43:0/3827514307 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ac67db4d90 con 0x55ac67e9ea70 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c5e000700 10 client.16236 handle_command_reply: tid=0 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c89ac6b80 1 client.16236 shutdown 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3827514307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 msgr2=0x55ac67e8eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e8eba0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000be70 tx=0x7f7c600116c0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3827514307 shutdown_connections 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ac67e9ea70 0x55ac67ea0ec0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67e84e70 0x55ac67e8f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e8e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/3827514307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e8eba0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3827514307 >> 172.21.15.43:0/3827514307 conn(0x55ac67d9f390 msgr2=0x55ac67e84370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c5e000700 0 client.16236 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.725+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3827514307 shutdown_connections 2024-05-27T07:19:06.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.726+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/3827514307 wait complete. 2024-05-27T07:19:06.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.726+0000 7f7c89ac6b80 20 client.16236 trim_cache size 0 max 16384 2024-05-27T07:19:06.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.744+0000 7f7c89ac6b80 1 Processor -- start 2024-05-27T07:19:06.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.744+0000 7f7c89ac6b80 1 -- start start 2024-05-27T07:19:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.744+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e848c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.744+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67df19f0 0x55ac67e89e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.744+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67e84e70 0x55ac67e9a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.744+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741b40a0 con 0x55ac67e84e70 2024-05-27T07:19:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.744+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741b10f0 con 0x55ac67df19f0 2024-05-27T07:19:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.744+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741b3300 con 0x55ac67d05e40 2024-05-27T07:19:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.745+0000 7f7c79000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67e84e70 0x55ac67e9a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.745+0000 7f7c79000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67e84e70 0x55ac67e9a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50564/0 (socket says 172.21.15.43:50564) 2024-05-27T07:19:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.745+0000 7f7c79000700 1 -- 172.21.15.43:0/2366509264 learned_addr learned my addr 172.21.15.43:0/2366509264 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:06.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.745+0000 7f7c73400700 1 --2- 172.21.15.43:0/2366509264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67df19f0 0x55ac67e89e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.745+0000 7f7c73e00700 1 --2- 172.21.15.43:0/2366509264 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e848c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.745+0000 7f7c79000700 1 -- 172.21.15.43:0/2366509264 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 msgr2=0x55ac67e848c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:06.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.745+0000 7f7c79000700 1 --2- 172.21.15.43:0/2366509264 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e848c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.745+0000 7f7c79000700 1 -- 172.21.15.43:0/2366509264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67df19f0 msgr2=0x55ac67e89e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:06.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.745+0000 7f7c79000700 1 --2- 172.21.15.43:0/2366509264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67df19f0 0x55ac67e89e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.745+0000 7f7c79000700 1 -- 172.21.15.43:0/2366509264 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac67e8cbd0 con 0x55ac67e84e70 2024-05-27T07:19:06.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.746+0000 7f7c79000700 1 --2- 172.21.15.43:0/2366509264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67e84e70 0x55ac67e9a9a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0185f0 tx=0x7f7c6c009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c83f74885b15c6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:06.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.746+0000 7f7c68c00700 1 -- 172.21.15.43:0/2366509264 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0233f0 con 0x55ac67e84e70 2024-05-27T07:19:06.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.746+0000 7f7c68c00700 1 -- 172.21.15.43:0/2366509264 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c6c028070 con 0x55ac67e84e70 2024-05-27T07:19:06.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.746+0000 7f7c68c00700 1 -- 172.21.15.43:0/2366509264 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0177e0 con 0x55ac67e84e70 2024-05-27T07:19:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.747+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/2366509264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67e84e70 msgr2=0x55ac67e9a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.747+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/2366509264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67e84e70 0x55ac67e9a9a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0185f0 tx=0x7f7c6c009620 comp rx=0 tx=0).stop 2024-05-27T07:19:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.747+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/2366509264 shutdown_connections 2024-05-27T07:19:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.747+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/2366509264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67e84e70 0x55ac67e9a9a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.747+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/2366509264 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e848c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.747+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/2366509264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67df19f0 0x55ac67e89e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.747+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/2366509264 >> 172.21.15.43:0/2366509264 conn(0x55ac67d9f390 msgr2=0x55ac67e841b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.747+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/2366509264 shutdown_connections 2024-05-27T07:19:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.747+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/2366509264 wait complete. 2024-05-27T07:19:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.748+0000 7f7c89ac6b80 1 -- start start 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.748+0000 7f7c89ac6b80 1 Processor -- start 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e89d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e88100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c89ac6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67e84e70 0x55ac67e88700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741b2c30 con 0x55ac67df19f0 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741b17c0 con 0x55ac67e84e70 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c89ac6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741b1e90 con 0x55ac67d05e40 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c89ac6b80 10 client.0 _ll_register_callbacks cb 0x55ac67d4cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e88100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e88100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50578/0 (socket says 172.21.15.43:50578) 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c73400700 1 -- 172.21.15.43:0/1220138186 learned_addr learned my addr 172.21.15.43:0/1220138186 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c73e00700 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e89d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.749+0000 7f7c79000700 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67e84e70 0x55ac67e88700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.750+0000 7f7c73400700 1 -- 172.21.15.43:0/1220138186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 msgr2=0x55ac67e89d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.750+0000 7f7c73400700 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e89d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.750+0000 7f7c73400700 1 -- 172.21.15.43:0/1220138186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67e84e70 msgr2=0x55ac67e88700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.750+0000 7f7c73400700 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67e84e70 0x55ac67e88700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.750+0000 7f7c73400700 1 -- 172.21.15.43:0/1220138186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac67e88ef0 con 0x55ac67df19f0 2024-05-27T07:19:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.750+0000 7f7c73400700 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e88100 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7c6001b840 tx=0x7f7c6001b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82586ad8ec71471 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.750+0000 7f7c5e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.751+0000 7f7c5e000700 1 -- 172.21.15.43:0/1220138186 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6001bd80 con 0x55ac67df19f0 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.751+0000 7f7c5e000700 1 -- 172.21.15.43:0/1220138186 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9408+0+0 (secure 0 0 0) 0x7f7c60011ac0 con 0x55ac67df19f0 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.751+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/1220138186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c6c028070 con 0x55ac67df19f0 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.751+0000 7f7c5e000700 1 -- 172.21.15.43:0/1220138186 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f7c60039050 con 0x55ac67df19f0 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.751+0000 7f7c89ac6b80 10 client.16242 fetch_fsmap learned FSMap version 76 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.751+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/1220138186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=4+}) -- 0x55ac67e96b70 con 0x55ac67df19f0 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.752+0000 7f7c5e000700 1 -- 172.21.15.43:0/1220138186 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c6c028070 con 0x55ac67df19f0 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.752+0000 7f7c5e000700 1 -- 172.21.15.43:0/1220138186 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 76) ==== 6559+0+0 (secure 0 0 0) 0x7f7c60035a80 con 0x55ac67df19f0 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.752+0000 7f7c89ac6b80 10 client.16242 fetch_fsmap finished waiting for FSMap version 76 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.752+0000 7f7c89ac6b80 10 client.16242 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.752+0000 7f7c89ac6b80 20 client.16242 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.752+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ac67e97280 0x55ac67f0b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:06.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.752+0000 7f7c89ac6b80 4 client.16242 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:19:06.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.752+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/1220138186 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ac67e7f120 con 0x55ac67e97280 2024-05-27T07:19:06.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.753+0000 7f7c73e00700 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ac67e97280 0x55ac67f0b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:06.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.754+0000 7f7c73e00700 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ac67e97280 0x55ac67f0b540 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7e9a0b026c2b4054 server_cookie=4dfe96f352d7937d in_seq=0 out_seq=0 2024-05-27T07:19:06.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.754+0000 7f7c5e000700 10 client.16242 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:19:06.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.754+0000 7f7c5e000700 1 -- 172.21.15.43:0/1220138186 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ac67e7f120 con 0x55ac67e97280 2024-05-27T07:19:06.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.754+0000 7f7c5e000700 10 client.16242 handle_command_reply: tid=0 2024-05-27T07:19:06.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.754+0000 7f7c89ac6b80 1 client.16242 shutdown 2024-05-27T07:19:06.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.754+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/1220138186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 msgr2=0x55ac67e88100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e88100 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7c6001b840 tx=0x7f7c6001b6b0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/1220138186 shutdown_connections 2024-05-27T07:19:06.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ac67e97280 0x55ac67f0b540 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ac67e84e70 0x55ac67e88700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ac67d05e40 0x55ac67e89d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c89ac6b80 1 --2- 172.21.15.43:0/1220138186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac67df19f0 0x55ac67e88100 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/1220138186 >> 172.21.15.43:0/1220138186 conn(0x55ac67d9f390 msgr2=0x55ac67e7fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:06.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c5e000700 0 client.16242 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:19:06.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/1220138186 shutdown_connections 2024-05-27T07:19:06.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c89ac6b80 1 -- 172.21.15.43:0/1220138186 wait complete. 2024-05-27T07:19:06.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.755+0000 7f7c89ac6b80 20 client.16242 trim_cache size 0 max 16384 2024-05-27T07:19:06.747 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:19:06.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 -- 172.21.15.43:0/1066525651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7c580b29f0 msgr2=0x7f7c580b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7c580b29f0 0x7f7c580b4f00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0094f0 tx=0x7f7c6c0091a0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 -- 172.21.15.43:0/1066525651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c74106570 msgr2=0x7f7c7419cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:06.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c74106570 0x7f7c7419cf20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7c64000c00 tx=0x7f7c64005ef0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 -- 172.21.15.43:0/1066525651 shutdown_connections 2024-05-27T07:19:06.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f7c580b29f0 0x7f7c580b4f00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c7410c5b0 0x7f7c741a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c74106570 0x7f7c7419cf20 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 --2- 172.21.15.43:0/1066525651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c74106ec0 0x7f7c7419d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:06.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 -- 172.21.15.43:0/1066525651 >> 172.21.15.43:0/1066525651 conn(0x7f7c74102070 msgr2=0x7f7c74104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:06.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.756+0000 7f7c7b000700 1 -- 172.21.15.43:0/1066525651 shutdown_connections 2024-05-27T07:19:06.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:06.757+0000 7f7c7b000700 1 -- 172.21.15.43:0/1066525651 wait complete. 2024-05-27T07:19:07.193 INFO:tasks.workunit.client.0.smithi043.stdout: 1 23502 0.62 MB/sec execute 71 sec latency 53.279 ms 2024-05-27T07:19:07.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:07 smithi043 ceph-mon[31531]: daemon mds.j restarted 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff924a00700 time 2024-05-27T07:19:07.363809+0000 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff93a477ba6] 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff93a477dd1] 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d459fee23d] 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d459fd9971] 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d459fd7fa0] 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d459fd8651] 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d459f6d347] 2024-05-27T07:19:07.401 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d459c6987e] 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d459c6ee94] 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d459c79969] 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d45a01248c] 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d45a01b9cb] 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 13: (Thread::entry_wrapper()+0x43) [0x7ff93a44672d] 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff93a446749] 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff938fea1ca] 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 16: clone() 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: *** Caught signal (Aborted) ** 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: in thread 7ff924a00700 thread_name:md_log_replay 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:19:07.402 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff938ff4d20] 2024-05-27T07:19:07.403 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2: gsignal() 2024-05-27T07:19:07.403 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 3: abort() 2024-05-27T07:19:07.403 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff93a477ced] 2024-05-27T07:19:07.403 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff93a477dd1] 2024-05-27T07:19:07.403 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d459fee23d] 2024-05-27T07:19:07.403 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d459fd9971] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d459fd7fa0] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d459fd8651] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d459f6d347] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d459c6987e] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d459c6ee94] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d459c79969] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d45a01248c] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d45a01b9cb] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 16: (Thread::entry_wrapper()+0x43) [0x7ff93a44672d] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff93a446749] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff938fea1ca] 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 19: clone() 2024-05-27T07:19:07.404 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.366+0000 7ff924a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff924a00700 time 2024-05-27T07:19:07.363809+0000 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff93a477ba6] 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff93a477dd1] 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d459fee23d] 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d459fd9971] 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d459fd7fa0] 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d459fd8651] 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d459f6d347] 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d459c6987e] 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d459c6ee94] 2024-05-27T07:19:07.405 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d459c79969] 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d45a01248c] 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d45a01b9cb] 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 13: (Thread::entry_wrapper()+0x43) [0x7ff93a44672d] 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff93a446749] 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff938fea1ca] 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 16: clone() 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.371+0000 7ff924a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: in thread 7ff924a00700 thread_name:md_log_replay 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:19:07.406 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff938ff4d20] 2024-05-27T07:19:07.407 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2: gsignal() 2024-05-27T07:19:07.407 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 3: abort() 2024-05-27T07:19:07.407 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff93a477ced] 2024-05-27T07:19:07.407 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff93a477dd1] 2024-05-27T07:19:07.407 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d459fee23d] 2024-05-27T07:19:07.407 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d459fd9971] 2024-05-27T07:19:07.407 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d459fd7fa0] 2024-05-27T07:19:07.407 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d459fd8651] 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d459f6d347] 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d459c6987e] 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d459c6ee94] 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d459c79969] 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d45a01248c] 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d45a01b9cb] 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 16: (Thread::entry_wrapper()+0x43) [0x7ff93a44672d] 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff93a446749] 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff938fea1ca] 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 19: clone() 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:19:07.408 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: -1> 2024-05-27T07:19:07.366+0000 7ff924a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff924a00700 time 2024-05-27T07:19:07.363809+0000 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff93a477ba6] 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff93a477dd1] 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d459fee23d] 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d459fd9971] 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d459fd7fa0] 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d459fd8651] 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d459f6d347] 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d459c6987e] 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d459c6ee94] 2024-05-27T07:19:07.409 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d459c79969] 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d45a01248c] 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d45a01b9cb] 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 13: (Thread::entry_wrapper()+0x43) [0x7ff93a44672d] 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff93a446749] 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff938fea1ca] 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 16: clone() 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 0> 2024-05-27T07:19:07.371+0000 7ff924a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: in thread 7ff924a00700 thread_name:md_log_replay 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.410 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:19:07.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:07 smithi179 ceph-mon[35888]: daemon mds.j restarted 2024-05-27T07:19:07.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:07 smithi132 ceph-mon[35283]: daemon mds.j restarted 2024-05-27T07:19:07.663 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff938ff4d20] 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2: gsignal() 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 3: abort() 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff93a477ced] 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff93a477dd1] 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d459fee23d] 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d459fd9971] 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d459fd7fa0] 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d459fd8651] 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d459f6d347] 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d459c6987e] 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d459c6ee94] 2024-05-27T07:19:07.664 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d459c79969] 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d45a01248c] 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d45a01b9cb] 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 16: (Thread::entry_wrapper()+0x43) [0x7ff93a44672d] 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff93a446749] 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff938fea1ca] 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 19: clone() 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: -1> 2024-05-27T07:19:07.366+0000 7ff924a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ff924a00700 time 2024-05-27T07:19:07.363809+0000 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:19:07.665 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.666 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:19:07.666 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff93a477ba6] 2024-05-27T07:19:07.666 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff93a477dd1] 2024-05-27T07:19:07.666 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d459fee23d] 2024-05-27T07:19:07.666 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55d459fd9971] 2024-05-27T07:19:07.666 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 5: (SnapRealm::check_cache() const+0x13e) [0x55d459fd7fa0] 2024-05-27T07:19:07.666 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d459fd8651] 2024-05-27T07:19:07.666 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d459f6d347] 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d459c6987e] 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d459c6ee94] 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55d459c79969] 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 11: (MDLog::_replay_thread()+0x1f38) [0x55d45a01248c] 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55d45a01b9cb] 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 13: (Thread::entry_wrapper()+0x43) [0x7ff93a44672d] 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 14: (Thread::_entry_func(void*)+0xd) [0x7ff93a446749] 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff938fea1ca] 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 16: clone() 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 0> 2024-05-27T07:19:07.371+0000 7ff924a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: in thread 7ff924a00700 thread_name:md_log_replay 2024-05-27T07:19:07.667 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.668 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:19:07.668 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff938ff4d20] 2024-05-27T07:19:07.668 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2: gsignal() 2024-05-27T07:19:07.668 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 3: abort() 2024-05-27T07:19:07.668 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff93a477ced] 2024-05-27T07:19:07.668 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff93a477dd1] 2024-05-27T07:19:07.668 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55d459fee23d] 2024-05-27T07:19:07.668 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55d459fd9971] 2024-05-27T07:19:07.668 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 8: (SnapRealm::check_cache() const+0x13e) [0x55d459fd7fa0] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55d459fd8651] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55d459f6d347] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55d459c6987e] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55d459c6ee94] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55d459c79969] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 14: (MDLog::_replay_thread()+0x1f38) [0x55d45a01248c] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55d45a01b9cb] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 16: (Thread::entry_wrapper()+0x43) [0x7ff93a44672d] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 17: (Thread::_entry_func(void*)+0xd) [0x7ff93a446749] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff938fea1ca] 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 19: clone() 2024-05-27T07:19:07.669 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:19:07.670 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j[101766]: 2024-05-27T07:19:07.731 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:19:07.952 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:07 smithi043 podman[101868]: 2024-05-27 07:19:07.698311265 +0000 UTC m=+0.231934954 container died b156c4e838ae090ba3de8fa67a5efe8164b267d584c5329cda8d181e9648d401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:19:08.193 INFO:tasks.workunit.client.0.smithi043.stdout: 1 23663 0.63 MB/sec execute 72 sec latency 55.314 ms 2024-05-27T07:19:08.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:08 smithi043 ceph-mon[31531]: pgmap v178: 129 pgs: 129 active+clean; 44 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 371 KiB/s wr, 81 op/s 2024-05-27T07:19:08.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:08 smithi043 ceph-mon[31531]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:19:08.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:08 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/3009933220,v1:172.21.15.43:6835/3009933220] up:boot 2024-05-27T07:19:08.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:08 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:19:08.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:08 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:19:08.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:08 smithi043 ceph-mon[31531]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:19:08.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:08 smithi179 ceph-mon[35888]: pgmap v178: 129 pgs: 129 active+clean; 44 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 371 KiB/s wr, 81 op/s 2024-05-27T07:19:08.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:08 smithi179 ceph-mon[35888]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:19:08.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:08 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/3009933220,v1:172.21.15.43:6835/3009933220] up:boot 2024-05-27T07:19:08.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:08 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:19:08.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:08 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:19:08.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:08 smithi179 ceph-mon[35888]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:19:08.480 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:08.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:08 smithi132 ceph-mon[35283]: pgmap v178: 129 pgs: 129 active+clean; 44 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 371 KiB/s wr, 81 op/s 2024-05-27T07:19:08.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:08 smithi132 ceph-mon[35283]: osdmap e111: 12 total, 12 up, 12 in 2024-05-27T07:19:08.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:08 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/3009933220,v1:172.21.15.43:6835/3009933220] up:boot 2024-05-27T07:19:08.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:08 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:19:08.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:08 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-27T07:19:08.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:08 smithi132 ceph-mon[35283]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-05-27T07:19:08.632 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:08 smithi043 podman[101868]: 2024-05-27 07:19:08.354046479 +0000 UTC m=+0.887670168 container remove b156c4e838ae090ba3de8fa67a5efe8164b267d584c5329cda8d181e9648d401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-j, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:19:08.632 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:08 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:19:09.193 INFO:tasks.workunit.client.0.smithi043.stdout: 1 23755 0.62 MB/sec execute 73 sec latency 48.959 ms 2024-05-27T07:19:09.202 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:09 smithi043 podman[101929]: 2024-05-27 07:19:09.05046872 +0000 UTC m=+0.630112391 container died e827c0c1e466666cff41c08f9458481c5c0d072d591a33b25b867ed3ca079d73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=silly_brahmagupta, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-05-27T07:19:10.194 INFO:tasks.workunit.client.0.smithi043.stdout: 1 23878 0.62 MB/sec execute 74 sec latency 51.330 ms 2024-05-27T07:19:10.379 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:09 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:19:10.379 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:09 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.806s CPU time. 2024-05-27T07:19:10.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:10 smithi043 ceph-mon[31531]: pgmap v180: 129 pgs: 129 active+clean; 44 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 333 KiB/s wr, 73 op/s 2024-05-27T07:19:10.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:10 smithi132 ceph-mon[35283]: pgmap v180: 129 pgs: 129 active+clean; 44 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 333 KiB/s wr, 73 op/s 2024-05-27T07:19:10.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:10 smithi179 ceph-mon[35888]: pgmap v180: 129 pgs: 129 active+clean; 44 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 333 KiB/s wr, 73 op/s 2024-05-27T07:19:10.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.787+0000 7facc2a00700 1 -- 172.21.15.43:0/2190547440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facbc109860 msgr2=0x7facbc1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.787+0000 7facc2a00700 1 --2- 172.21.15.43:0/2190547440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facbc109860 0x7facbc1111c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7facb400b0d0 tx=0x7facb400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.787+0000 7facc2a00700 1 -- 172.21.15.43:0/2190547440 shutdown_connections 2024-05-27T07:19:10.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.787+0000 7facc2a00700 1 --2- 172.21.15.43:0/2190547440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facbc109860 0x7facbc1111c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.787+0000 7facc2a00700 1 --2- 172.21.15.43:0/2190547440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facbc1045e0 0x7facbc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.787+0000 7facc2a00700 1 --2- 172.21.15.43:0/2190547440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facbc108f10 0x7facbc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.787+0000 7facc2a00700 1 -- 172.21.15.43:0/2190547440 >> 172.21.15.43:0/2190547440 conn(0x7facbc100030 msgr2=0x7facbc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:10.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.787+0000 7facc2a00700 1 -- 172.21.15.43:0/2190547440 shutdown_connections 2024-05-27T07:19:10.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.787+0000 7facc2a00700 1 -- 172.21.15.43:0/2190547440 wait complete. 2024-05-27T07:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.789+0000 7facc2a00700 1 Processor -- start 2024-05-27T07:19:10.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.789+0000 7facc2a00700 1 -- start start 2024-05-27T07:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.789+0000 7facc2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facbc1045e0 0x7facbc19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.789+0000 7facc2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facbc108f10 0x7facbc19d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.790+0000 7facc2a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facbc109860 0x7facbc1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.790+0000 7facc2a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7facbc114390 con 0x7facbc108f10 2024-05-27T07:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.790+0000 7facc2a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7facbc114210 con 0x7facbc1045e0 2024-05-27T07:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.790+0000 7facc2a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7facbc114510 con 0x7facbc109860 2024-05-27T07:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.790+0000 7facbb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facbc108f10 0x7facbc19d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.790+0000 7facbb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facbc108f10 0x7facbc19d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50598/0 (socket says 172.21.15.43:50598) 2024-05-27T07:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.790+0000 7facbb400700 1 -- 172.21.15.43:0/1923002813 learned_addr learned my addr 172.21.15.43:0/1923002813 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:10.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.790+0000 7facbbe00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facbc1045e0 0x7facbc19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.790+0000 7facc0a00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facbc109860 0x7facbc1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.791+0000 7facbb400700 1 -- 172.21.15.43:0/1923002813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facbc109860 msgr2=0x7facbc1a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.791+0000 7facbb400700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facbc109860 0x7facbc1a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.791+0000 7facbb400700 1 -- 172.21.15.43:0/1923002813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facbc1045e0 msgr2=0x7facbc19cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.791+0000 7facbb400700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facbc1045e0 0x7facbc19cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.791+0000 7facbb400700 1 -- 172.21.15.43:0/1923002813 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facbc1a1e50 con 0x7facbc108f10 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.791+0000 7facc0a00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facbc109860 0x7facbc1a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.791+0000 7facbb400700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facbc108f10 0x7facbc19d520 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7faca800c8f0 tx=0x7faca800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.792+0000 7facb8c00700 1 -- 172.21.15.43:0/1923002813 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca8007940 con 0x7facbc108f10 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.792+0000 7facc2a00700 1 -- 172.21.15.43:0/1923002813 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facbc1a2140 con 0x7facbc108f10 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.792+0000 7facb8c00700 1 -- 172.21.15.43:0/1923002813 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faca800f450 con 0x7facbc108f10 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.792+0000 7facb8c00700 1 -- 172.21.15.43:0/1923002813 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca8018660 con 0x7facbc108f10 2024-05-27T07:19:10.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.792+0000 7facc2a00700 1 -- 172.21.15.43:0/1923002813 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facbc1a9a60 con 0x7facbc108f10 2024-05-27T07:19:10.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.797+0000 7facb8c00700 1 -- 172.21.15.43:0/1923002813 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7faca800f900 con 0x7facbc108f10 2024-05-27T07:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.798+0000 7facd155bb80 1 Processor -- start 2024-05-27T07:19:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.798+0000 7facd155bb80 1 -- start start 2024-05-27T07:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facd155bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252dbf9f0 0x562252e52330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facd155bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252cd3e40 0x562252e52870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facd155bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252e52e70 0x562252e5f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facd155bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562252d80e40 con 0x562252dbf9f0 2024-05-27T07:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facd155bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562252d709e0 con 0x562252cd3e40 2024-05-27T07:19:10.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facd155bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562252d74a20 con 0x562252e52e70 2024-05-27T07:19:10.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facc0a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252dbf9f0 0x562252e52330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facb8c00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7faca00b29f0 0x7faca00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facc0a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252dbf9f0 0x562252e52330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50612/0 (socket says 172.21.15.43:50612) 2024-05-27T07:19:10.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facc0a00700 1 -- 172.21.15.43:0/535295835 learned_addr learned my addr 172.21.15.43:0/535295835 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:10.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.799+0000 7facbbe00700 1 --2- 172.21.15.43:0/535295835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252cd3e40 0x562252e52870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.800+0000 7facb8c00700 1 -- 172.21.15.43:0/1923002813 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7faca80e2800 con 0x7facbc108f10 2024-05-27T07:19:10.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.800+0000 7facbbe00700 1 -- 172.21.15.43:0/535295835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252e52e70 msgr2=0x562252e5f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:10.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.800+0000 7facbbe00700 1 --2- 172.21.15.43:0/535295835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252e52e70 0x562252e5f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.800+0000 7facbbe00700 1 -- 172.21.15.43:0/535295835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252dbf9f0 msgr2=0x562252e52330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:10.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.801+0000 7facbbe00700 1 --2- 172.21.15.43:0/535295835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252dbf9f0 0x562252e52330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.801+0000 7facbbe00700 1 -- 172.21.15.43:0/535295835 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562252d39420 con 0x562252cd3e40 2024-05-27T07:19:10.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.801+0000 7facc0a00700 1 --2- 172.21.15.43:0/535295835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252dbf9f0 0x562252e52330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:19:10.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.801+0000 7facc0a00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7faca00b29f0 0x7faca00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.801+0000 7facbbe00700 1 --2- 172.21.15.43:0/535295835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252cd3e40 0x562252e52870 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7facac00bc80 tx=0x7facac00bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9682dce278de26e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:10.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.802+0000 7facb1600700 1 -- 172.21.15.43:0/535295835 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac010ca0 con 0x562252cd3e40 2024-05-27T07:19:10.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.802+0000 7facb1600700 1 -- 172.21.15.43:0/535295835 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facac010e40 con 0x562252cd3e40 2024-05-27T07:19:10.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.802+0000 7facb1600700 1 -- 172.21.15.43:0/535295835 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac005020 con 0x562252cd3e40 2024-05-27T07:19:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.803+0000 7facd155bb80 1 -- 172.21.15.43:0/535295835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252cd3e40 msgr2=0x562252e52870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.803+0000 7facd155bb80 1 --2- 172.21.15.43:0/535295835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252cd3e40 0x562252e52870 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7facac00bc80 tx=0x7facac00bf90 comp rx=0 tx=0).stop 2024-05-27T07:19:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.803+0000 7facc0a00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7faca00b29f0 0x7faca00b4f00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x562252e54ff0 tx=0x7facb40099e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.804+0000 7facd155bb80 1 -- 172.21.15.43:0/535295835 shutdown_connections 2024-05-27T07:19:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.804+0000 7facd155bb80 1 --2- 172.21.15.43:0/535295835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252e52e70 0x562252e5f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.804+0000 7facd155bb80 1 --2- 172.21.15.43:0/535295835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252dbf9f0 0x562252e52330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.804+0000 7facd155bb80 1 --2- 172.21.15.43:0/535295835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252cd3e40 0x562252e52870 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.804+0000 7facd155bb80 1 -- 172.21.15.43:0/535295835 >> 172.21.15.43:0/535295835 conn(0x562252d6d390 msgr2=0x562252e4e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.804+0000 7facd155bb80 1 -- 172.21.15.43:0/535295835 shutdown_connections 2024-05-27T07:19:10.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.804+0000 7facd155bb80 1 -- 172.21.15.43:0/535295835 wait complete. 2024-05-27T07:19:10.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.805+0000 7facd155bb80 1 -- start start 2024-05-27T07:19:10.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.805+0000 7facd155bb80 1 Processor -- start 2024-05-27T07:19:10.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.806+0000 7facd155bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252cd3e40 0x562252e5c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.806+0000 7facd155bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e5cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.806+0000 7facd155bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e5d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.806+0000 7facd155bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562252c76fa0 con 0x562252e52e70 2024-05-27T07:19:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.806+0000 7facd155bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562252d11760 con 0x562252dbf9f0 2024-05-27T07:19:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.806+0000 7facd155bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562252e5fb80 con 0x562252cd3e40 2024-05-27T07:19:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.806+0000 7facd155bb80 10 client.0 _ll_register_callbacks cb 0x562252d1ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.806+0000 7facc0a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e5d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.806+0000 7facc0a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e5d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50626/0 (socket says 172.21.15.43:50626) 2024-05-27T07:19:10.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.806+0000 7facc0a00700 1 -- 172.21.15.43:0/2307227327 learned_addr learned my addr 172.21.15.43:0/2307227327 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:10.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.807+0000 7facbb400700 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e5cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.807+0000 7facbbe00700 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252cd3e40 0x562252e5c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.807+0000 7facbb400700 1 -- 172.21.15.43:0/2307227327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252cd3e40 msgr2=0x562252e5c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:10.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.807+0000 7facbb400700 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252cd3e40 0x562252e5c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.807+0000 7facbb400700 1 -- 172.21.15.43:0/2307227327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 msgr2=0x562252e5d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.807+0000 7facbb400700 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e5d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.807+0000 7facbb400700 1 -- 172.21.15.43:0/2307227327 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562252cc08d0 con 0x562252dbf9f0 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.807+0000 7facc0a00700 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e5d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.808+0000 7facbb400700 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e5cbc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faca80dfba0 tx=0x7faca80e00d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d828c5ff5936c53 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.808+0000 7faca6000700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.808+0000 7faca6000700 1 -- 172.21.15.43:0/2307227327 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca80b0e60 con 0x562252dbf9f0 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.808+0000 7facd155bb80 1 -- 172.21.15.43:0/2307227327 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facac010e40 con 0x562252dbf9f0 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.809+0000 7faca6000700 1 -- 172.21.15.43:0/2307227327 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7faca80e2430 con 0x562252dbf9f0 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.809+0000 7faca6000700 1 -- 172.21.15.43:0/2307227327 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7faca80e2630 con 0x562252dbf9f0 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.809+0000 7facd155bb80 10 client.35846 fetch_fsmap learned FSMap version 78 2024-05-27T07:19:10.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.809+0000 7facd155bb80 1 -- 172.21.15.43:0/2307227327 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x562252d41060 con 0x562252dbf9f0 2024-05-27T07:19:10.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.810+0000 7faca6000700 1 -- 172.21.15.43:0/2307227327 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facac010e40 con 0x562252dbf9f0 2024-05-27T07:19:10.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.810+0000 7faca6000700 1 -- 172.21.15.43:0/2307227327 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 78) ==== 6559+0+0 (secure 0 0 0) 0x7faca80e1070 con 0x562252dbf9f0 2024-05-27T07:19:10.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.810+0000 7facd155bb80 10 client.35846 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:19:10.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.810+0000 7facd155bb80 10 client.35846 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:19:10.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.810+0000 7facd155bb80 20 client.35846 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:10.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.810+0000 7facd155bb80 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562252e75490 0x562252e778e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.810+0000 7facd155bb80 4 client.35846 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:19:10.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.810+0000 7facd155bb80 1 -- 172.21.15.43:0/2307227327 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562252d82d90 con 0x562252e75490 2024-05-27T07:19:10.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.811+0000 7facbbe00700 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562252e75490 0x562252e778e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.812+0000 7facbbe00700 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562252e75490 0x562252e778e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aa9292024f17f6bb server_cookie=19a234dce9f0bad4 in_seq=0 out_seq=0 2024-05-27T07:19:10.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.812+0000 7faca6000700 10 client.35846 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:19:10.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.814+0000 7faca6000700 1 -- 172.21.15.43:0/2307227327 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562252d82d90 con 0x562252e75490 2024-05-27T07:19:10.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.814+0000 7faca6000700 10 client.35846 handle_command_reply: tid=0 2024-05-27T07:19:10.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.814+0000 7facd155bb80 1 client.35846 shutdown 2024-05-27T07:19:10.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.814+0000 7facd155bb80 1 -- 172.21.15.43:0/2307227327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 msgr2=0x562252e5cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.814+0000 7facd155bb80 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e5cbc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faca80dfba0 tx=0x7faca80e00d0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.814+0000 7facd155bb80 1 -- 172.21.15.43:0/2307227327 shutdown_connections 2024-05-27T07:19:10.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.814+0000 7facd155bb80 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e5d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.814+0000 7facd155bb80 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252cd3e40 0x562252e5c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.814+0000 7facd155bb80 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562252e75490 0x562252e778e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.814+0000 7facd155bb80 1 --2- 172.21.15.43:0/2307227327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e5cbc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.815+0000 7faca6000700 0 client.35846 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:19:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.815+0000 7facd155bb80 1 -- 172.21.15.43:0/2307227327 >> 172.21.15.43:0/2307227327 conn(0x562252d6d390 msgr2=0x562252e523b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.815+0000 7facd155bb80 1 -- 172.21.15.43:0/2307227327 shutdown_connections 2024-05-27T07:19:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.815+0000 7facd155bb80 1 -- 172.21.15.43:0/2307227327 wait complete. 2024-05-27T07:19:10.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.815+0000 7facd155bb80 20 client.35846 trim_cache size 0 max 16384 2024-05-27T07:19:10.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.833+0000 7facd155bb80 1 Processor -- start 2024-05-27T07:19:10.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.833+0000 7facd155bb80 1 -- start start 2024-05-27T07:19:10.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.833+0000 7facd155bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252cd3e40 0x562252e57e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.833+0000 7facd155bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e5b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.833+0000 7facd155bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e52830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.833+0000 7facd155bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1afc70 con 0x562252e52e70 2024-05-27T07:19:10.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.833+0000 7facd155bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1accc0 con 0x562252dbf9f0 2024-05-27T07:19:10.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.833+0000 7facd155bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1aeed0 con 0x562252cd3e40 2024-05-27T07:19:10.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.834+0000 7facc0a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e52830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.834+0000 7facc0a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e52830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50640/0 (socket says 172.21.15.43:50640) 2024-05-27T07:19:10.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.834+0000 7facc0a00700 1 -- 172.21.15.43:0/1086160801 learned_addr learned my addr 172.21.15.43:0/1086160801 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:10.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.834+0000 7facbb400700 1 --2- 172.21.15.43:0/1086160801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e5b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.834+0000 7facbbe00700 1 --2- 172.21.15.43:0/1086160801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252cd3e40 0x562252e57e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.834+0000 7facc0a00700 1 -- 172.21.15.43:0/1086160801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252cd3e40 msgr2=0x562252e57e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:10.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.834+0000 7facc0a00700 1 --2- 172.21.15.43:0/1086160801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252cd3e40 0x562252e57e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.834+0000 7facc0a00700 1 -- 172.21.15.43:0/1086160801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 msgr2=0x562252e5b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:10.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.834+0000 7facc0a00700 1 --2- 172.21.15.43:0/1086160801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e5b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.834+0000 7facc0a00700 1 -- 172.21.15.43:0/1086160801 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562252e4f000 con 0x562252e52e70 2024-05-27T07:19:10.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.835+0000 7facc0a00700 1 --2- 172.21.15.43:0/1086160801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e52830 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7facb4009fd0 tx=0x7facb4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308b6ae6e001d0de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:10.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.835+0000 7facb0c00700 1 -- 172.21.15.43:0/1086160801 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb401e8c0 con 0x562252e52e70 2024-05-27T07:19:10.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.835+0000 7facb0c00700 1 -- 172.21.15.43:0/1086160801 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facb4022070 con 0x562252e52e70 2024-05-27T07:19:10.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.835+0000 7facb0c00700 1 -- 172.21.15.43:0/1086160801 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb402ac90 con 0x562252e52e70 2024-05-27T07:19:10.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.836+0000 7facd155bb80 1 -- 172.21.15.43:0/1086160801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 msgr2=0x562252e52830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.836+0000 7facd155bb80 1 --2- 172.21.15.43:0/1086160801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e52830 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7facb4009fd0 tx=0x7facb4020000 comp rx=0 tx=0).stop 2024-05-27T07:19:10.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.836+0000 7facd155bb80 1 -- 172.21.15.43:0/1086160801 shutdown_connections 2024-05-27T07:19:10.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.836+0000 7facd155bb80 1 --2- 172.21.15.43:0/1086160801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252e52e70 0x562252e52830 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.836+0000 7facd155bb80 1 --2- 172.21.15.43:0/1086160801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252cd3e40 0x562252e57e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.836+0000 7facd155bb80 1 --2- 172.21.15.43:0/1086160801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e5b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.836+0000 7facd155bb80 1 -- 172.21.15.43:0/1086160801 >> 172.21.15.43:0/1086160801 conn(0x562252d6d390 msgr2=0x562252e52120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:10.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.837+0000 7facd155bb80 1 -- 172.21.15.43:0/1086160801 shutdown_connections 2024-05-27T07:19:10.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.837+0000 7facd155bb80 1 -- 172.21.15.43:0/1086160801 wait complete. 2024-05-27T07:19:10.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.838+0000 7facd155bb80 1 -- start start 2024-05-27T07:19:10.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.838+0000 7facd155bb80 1 Processor -- start 2024-05-27T07:19:10.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facd155bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252cd3e40 0x562252e563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facd155bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e56910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facd155bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252e52e70 0x562252e56e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facd155bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1ae800 con 0x562252cd3e40 2024-05-27T07:19:10.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facd155bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1ad390 con 0x562252dbf9f0 2024-05-27T07:19:10.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facd155bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1ada60 con 0x562252e52e70 2024-05-27T07:19:10.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facd155bb80 10 client.0 _ll_register_callbacks cb 0x562252d1ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:10.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facbbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252cd3e40 0x562252e563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facbbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252cd3e40 0x562252e563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50652/0 (socket says 172.21.15.43:50652) 2024-05-27T07:19:10.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facbbe00700 1 -- 172.21.15.43:0/1419788491 learned_addr learned my addr 172.21.15.43:0/1419788491 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:10.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facc0a00700 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252e52e70 0x562252e56e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.839+0000 7facbb400700 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e56910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.840+0000 7facbbe00700 1 -- 172.21.15.43:0/1419788491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252e52e70 msgr2=0x562252e56e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:10.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.840+0000 7facbbe00700 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252e52e70 0x562252e56e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.840+0000 7facbbe00700 1 -- 172.21.15.43:0/1419788491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 msgr2=0x562252e56910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:10.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.840+0000 7facbbe00700 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e56910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.840+0000 7facbbe00700 1 -- 172.21.15.43:0/1419788491 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562252e63e40 con 0x562252cd3e40 2024-05-27T07:19:10.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.840+0000 7facbbe00700 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252cd3e40 0x562252e563d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7facac00bbd0 tx=0x7facac005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58630c4c3c13c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:10.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.840+0000 7faca6000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:10.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.841+0000 7faca6000700 1 -- 172.21.15.43:0/1419788491 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac03d020 con 0x562252cd3e40 2024-05-27T07:19:10.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.841+0000 7faca6000700 1 -- 172.21.15.43:0/1419788491 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7facac027860 con 0x562252cd3e40 2024-05-27T07:19:10.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.841+0000 7facd155bb80 1 -- 172.21.15.43:0/1419788491 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facb4022070 con 0x562252cd3e40 2024-05-27T07:19:10.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.841+0000 7faca6000700 1 -- 172.21.15.43:0/1419788491 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7facac048050 con 0x562252cd3e40 2024-05-27T07:19:10.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.841+0000 7facd155bb80 10 client.16260 fetch_fsmap learned FSMap version 78 2024-05-27T07:19:10.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.841+0000 7facd155bb80 1 -- 172.21.15.43:0/1419788491 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x562252e650e0 con 0x562252cd3e40 2024-05-27T07:19:10.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.842+0000 7faca6000700 1 -- 172.21.15.43:0/1419788491 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7facb4022070 con 0x562252cd3e40 2024-05-27T07:19:10.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.842+0000 7faca6000700 1 -- 172.21.15.43:0/1419788491 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 78) ==== 6559+0+0 (secure 0 0 0) 0x7facac011980 con 0x562252cd3e40 2024-05-27T07:19:10.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.842+0000 7facd155bb80 10 client.16260 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:19:10.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.842+0000 7facd155bb80 10 client.16260 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:19:10.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.842+0000 7facd155bb80 20 client.16260 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:10.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.842+0000 7facd155bb80 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562252e65820 0x562252e6af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:10.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.842+0000 7facd155bb80 4 client.16260 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:19:10.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.842+0000 7facd155bb80 1 -- 172.21.15.43:0/1419788491 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562252d7c570 con 0x562252e65820 2024-05-27T07:19:10.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.843+0000 7facbb400700 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562252e65820 0x562252e6af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.844+0000 7faca6000700 10 client.16260 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:19:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.844+0000 7facbb400700 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562252e65820 0x562252e6af40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b1d8170dfc4597a9 server_cookie=e3b891e5c606588f in_seq=0 out_seq=0 2024-05-27T07:19:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7faca6000700 1 -- 172.21.15.43:0/1419788491 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562252d7c570 con 0x562252e65820 2024-05-27T07:19:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7faca6000700 10 client.16260 handle_command_reply: tid=0 2024-05-27T07:19:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 client.16260 shutdown 2024-05-27T07:19:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 -- 172.21.15.43:0/1419788491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252cd3e40 msgr2=0x562252e563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252cd3e40 0x562252e563d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7facac00bbd0 tx=0x7facac005db0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 -- 172.21.15.43:0/1419788491 shutdown_connections 2024-05-27T07:19:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562252e52e70 0x562252e56e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562252cd3e40 0x562252e563d0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562252e65820 0x562252e6af40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 --2- 172.21.15.43:0/1419788491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562252dbf9f0 0x562252e56910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 -- 172.21.15.43:0/1419788491 >> 172.21.15.43:0/1419788491 conn(0x562252d6d390 msgr2=0x562252e53ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7faca6000700 0 client.16260 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:19:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 -- 172.21.15.43:0/1419788491 shutdown_connections 2024-05-27T07:19:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.845+0000 7facd155bb80 1 -- 172.21.15.43:0/1419788491 wait complete. 2024-05-27T07:19:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.846+0000 7facd155bb80 20 client.16260 trim_cache size 0 max 16384 2024-05-27T07:19:10.839 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:19:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.846+0000 7facc2a00700 1 -- 172.21.15.43:0/1923002813 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7faca00b29f0 msgr2=0x7faca00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.846+0000 7facc2a00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7faca00b29f0 0x7faca00b4f00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x562252e54ff0 tx=0x7facb40099e0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.846+0000 7facc2a00700 1 -- 172.21.15.43:0/1923002813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facbc108f10 msgr2=0x7facbc19d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.846+0000 7facc2a00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facbc108f10 0x7facbc19d520 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7faca800c8f0 tx=0x7faca800cc00 comp rx=0 tx=0).stop 2024-05-27T07:19:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.847+0000 7facc2a00700 1 -- 172.21.15.43:0/1923002813 shutdown_connections 2024-05-27T07:19:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.847+0000 7facc2a00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7faca00b29f0 0x7faca00b4f00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.847+0000 7facc2a00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facbc109860 0x7facbc1a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.847+0000 7facc2a00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facbc1045e0 0x7facbc19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.847+0000 7facc2a00700 1 --2- 172.21.15.43:0/1923002813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facbc108f10 0x7facbc19d520 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:10.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.847+0000 7facc2a00700 1 -- 172.21.15.43:0/1923002813 >> 172.21.15.43:0/1923002813 conn(0x7facbc100030 msgr2=0x7facbc110e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:10.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.847+0000 7facc2a00700 1 -- 172.21.15.43:0/1923002813 shutdown_connections 2024-05-27T07:19:10.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:10.847+0000 7facc2a00700 1 -- 172.21.15.43:0/1923002813 wait complete. 2024-05-27T07:19:11.193 INFO:tasks.workunit.client.0.smithi043.stdout: 1 24073 0.66 MB/sec execute 75 sec latency 50.069 ms 2024-05-27T07:19:11.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:19:12.193 INFO:tasks.workunit.client.0.smithi043.stdout: 1 24261 0.67 MB/sec execute 76 sec latency 67.469 ms 2024-05-27T07:19:12.291 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:12.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:12 smithi132 ceph-mon[35283]: pgmap v181: 129 pgs: 129 active+clean; 46 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 369 KiB/s wr, 68 op/s 2024-05-27T07:19:12.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:12 smithi043 ceph-mon[31531]: pgmap v181: 129 pgs: 129 active+clean; 46 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 369 KiB/s wr, 68 op/s 2024-05-27T07:19:12.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:12 smithi179 ceph-mon[35888]: pgmap v181: 129 pgs: 129 active+clean; 46 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 369 KiB/s wr, 68 op/s 2024-05-27T07:19:13.193 INFO:tasks.workunit.client.0.smithi043.stdout: 1 24499 0.67 MB/sec execute 77 sec latency 51.044 ms 2024-05-27T07:19:14.194 INFO:tasks.workunit.client.0.smithi043.stdout: 1 24615 0.68 MB/sec execute 78 sec latency 51.932 ms 2024-05-27T07:19:14.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:14 smithi043 ceph-mon[31531]: pgmap v182: 129 pgs: 129 active+clean; 46 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 369 KiB/s wr, 68 op/s 2024-05-27T07:19:14.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:14 smithi132 ceph-mon[35283]: pgmap v182: 129 pgs: 129 active+clean; 46 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 369 KiB/s wr, 68 op/s 2024-05-27T07:19:14.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:14 smithi179 ceph-mon[35888]: pgmap v182: 129 pgs: 129 active+clean; 46 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 369 KiB/s wr, 68 op/s 2024-05-27T07:19:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.783+0000 7f4168400700 1 -- 172.21.15.43:0/3729401687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4160102650 msgr2=0x7f4160102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.783+0000 7f4168400700 1 --2- 172.21.15.43:0/3729401687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4160102650 0x7f4160102a60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4154009990 tx=0x7f4154009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.783+0000 7f4168400700 1 -- 172.21.15.43:0/3729401687 shutdown_connections 2024-05-27T07:19:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.783+0000 7f4168400700 1 --2- 172.21.15.43:0/3729401687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4160110d10 0x7f4160113130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.783+0000 7f4168400700 1 --2- 172.21.15.43:0/3729401687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4160102650 0x7f4160102a60 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.783+0000 7f4168400700 1 --2- 172.21.15.43:0/3729401687 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f416010e3b0 0x7f41601107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.783+0000 7f4168400700 1 -- 172.21.15.43:0/3729401687 >> 172.21.15.43:0/3729401687 conn(0x7f41600fe0c0 msgr2=0x7f41601004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.784+0000 7f4168400700 1 -- 172.21.15.43:0/3729401687 shutdown_connections 2024-05-27T07:19:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.784+0000 7f4168400700 1 -- 172.21.15.43:0/3729401687 wait complete. 2024-05-27T07:19:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.785+0000 7f4168400700 1 Processor -- start 2024-05-27T07:19:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.786+0000 7f4168400700 1 -- start start 2024-05-27T07:19:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.786+0000 7f4168400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4160102650 0x7f416019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.786+0000 7f4168400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f416010e3b0 0x7f416019d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.786+0000 7f4168400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4160110d10 0x7f41601a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.786+0000 7f4168400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4160116530 con 0x7f4160110d10 2024-05-27T07:19:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.786+0000 7f4168400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f41601163b0 con 0x7f416010e3b0 2024-05-27T07:19:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.786+0000 7f4168400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f41601166b0 con 0x7f4160102650 2024-05-27T07:19:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.787+0000 7f4166400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4160110d10 0x7f41601a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.787+0000 7f4166400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4160110d10 0x7f41601a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54896/0 (socket says 172.21.15.43:54896) 2024-05-27T07:19:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.787+0000 7f4166400700 1 -- 172.21.15.43:0/23833223 learned_addr learned my addr 172.21.15.43:0/23833223 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.787+0000 7f4165a00700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4160102650 0x7f416019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.787+0000 7f4165000700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f416010e3b0 0x7f416019d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.787+0000 7f4166400700 1 -- 172.21.15.43:0/23833223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4160102650 msgr2=0x7f416019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:14.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.787+0000 7f4166400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4160102650 0x7f416019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.787+0000 7f4166400700 1 -- 172.21.15.43:0/23833223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f416010e3b0 msgr2=0x7f416019d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:14.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.787+0000 7f4166400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f416010e3b0 0x7f416019d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.787+0000 7f4166400700 1 -- 172.21.15.43:0/23833223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41601a1e70 con 0x7f4160110d10 2024-05-27T07:19:14.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.788+0000 7f4166400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4160110d10 0x7f41601a17d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4158007ee0 tx=0x7f415800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:14.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.788+0000 7f415e000700 1 -- 172.21.15.43:0/23833223 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158004bb0 con 0x7f4160110d10 2024-05-27T07:19:14.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.788+0000 7f415e000700 1 -- 172.21.15.43:0/23833223 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4158004d50 con 0x7f4160110d10 2024-05-27T07:19:14.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.788+0000 7f415e000700 1 -- 172.21.15.43:0/23833223 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580058b0 con 0x7f4160110d10 2024-05-27T07:19:14.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.788+0000 7f4168400700 1 -- 172.21.15.43:0/23833223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41601a2160 con 0x7f4160110d10 2024-05-27T07:19:14.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.789+0000 7f4168400700 1 -- 172.21.15.43:0/23833223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41601a9a80 con 0x7f4160110d10 2024-05-27T07:19:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.793+0000 7f415e000700 1 -- 172.21.15.43:0/23833223 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f415801a020 con 0x7f4160110d10 2024-05-27T07:19:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.793+0000 7f4176fa4b80 1 Processor -- start 2024-05-27T07:19:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.794+0000 7f4176fa4b80 1 -- start start 2024-05-27T07:19:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.794+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615140c49f0 0x561514157330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.794+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561513fd8e40 0x561514157870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.794+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561514157e70 0x561514164110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.794+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561514085e40 con 0x5615140c49f0 2024-05-27T07:19:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.794+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5615140759e0 con 0x561514157e70 2024-05-27T07:19:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.794+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561514079a20 con 0x561513fd8e40 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.794+0000 7f4165000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615140c49f0 0x561514157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f4165000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615140c49f0 0x561514157330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54900/0 (socket says 172.21.15.43:54900) 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f4165000700 1 -- 172.21.15.43:0/1680093664 learned_addr learned my addr 172.21.15.43:0/1680093664 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f4165a00700 1 --2- 172.21.15.43:0/1680093664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561513fd8e40 0x561514157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f4165000700 1 --2- 172.21.15.43:0/1680093664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561514157e70 0x561514164110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f415e000700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41480b2a70 0x7f41480b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f4165000700 1 -- 172.21.15.43:0/1680093664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561513fd8e40 msgr2=0x561514157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f4165000700 1 --2- 172.21.15.43:0/1680093664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561513fd8e40 0x561514157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f4165000700 1 -- 172.21.15.43:0/1680093664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561514157e70 msgr2=0x561514164110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f4165000700 1 --2- 172.21.15.43:0/1680093664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561514157e70 0x561514164110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f4165000700 1 -- 172.21.15.43:0/1680093664 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56151403e420 con 0x5615140c49f0 2024-05-27T07:19:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.795+0000 7f415e000700 1 -- 172.21.15.43:0/23833223 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f41580e1fb0 con 0x7f4160110d10 2024-05-27T07:19:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.796+0000 7f4165000700 1 --2- 172.21.15.43:0/1680093664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615140c49f0 0x561514157330 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f414c005d90 tx=0x7f414c005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c2f844ad68d710 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.796+0000 7f4166400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41480b2a70 0x7f41480b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.796+0000 7f4152a00700 1 -- 172.21.15.43:0/1680093664 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c004030 con 0x5615140c49f0 2024-05-27T07:19:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.796+0000 7f4152a00700 1 -- 172.21.15.43:0/1680093664 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f414c004830 con 0x5615140c49f0 2024-05-27T07:19:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.796+0000 7f4152a00700 1 -- 172.21.15.43:0/1680093664 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c005280 con 0x5615140c49f0 2024-05-27T07:19:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.797+0000 7f4176fa4b80 1 -- 172.21.15.43:0/1680093664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615140c49f0 msgr2=0x561514157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.797+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/1680093664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615140c49f0 0x561514157330 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f414c005d90 tx=0x7f414c005d20 comp rx=0 tx=0).stop 2024-05-27T07:19:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.797+0000 7f4166400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41480b2a70 0x7f41480b4f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f41580df3b0 tx=0x7f415800b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.797+0000 7f4176fa4b80 1 -- 172.21.15.43:0/1680093664 shutdown_connections 2024-05-27T07:19:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.797+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/1680093664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561514157e70 0x561514164110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.797+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/1680093664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5615140c49f0 0x561514157330 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.797+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/1680093664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561513fd8e40 0x561514157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.797+0000 7f4176fa4b80 1 -- 172.21.15.43:0/1680093664 >> 172.21.15.43:0/1680093664 conn(0x561514072390 msgr2=0x5615141534c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.798+0000 7f4176fa4b80 1 -- 172.21.15.43:0/1680093664 shutdown_connections 2024-05-27T07:19:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.798+0000 7f4176fa4b80 1 -- 172.21.15.43:0/1680093664 wait complete. 2024-05-27T07:19:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.798+0000 7f4176fa4b80 1 -- start start 2024-05-27T07:19:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.799+0000 7f4176fa4b80 1 Processor -- start 2024-05-27T07:19:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.799+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514161700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.799+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514161c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.799+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.799+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561513f7bfa0 con 0x561513fd8e40 2024-05-27T07:19:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.799+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561514016760 con 0x5615140c49f0 2024-05-27T07:19:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.799+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561514164b80 con 0x561514157e70 2024-05-27T07:19:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.799+0000 7f4176fa4b80 10 client.0 _ll_register_callbacks cb 0x56151401fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.800+0000 7f4165000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.800+0000 7f4165000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514161700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54904/0 (socket says 172.21.15.43:54904) 2024-05-27T07:19:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.800+0000 7f4165000700 1 -- 172.21.15.43:0/4279710193 learned_addr learned my addr 172.21.15.43:0/4279710193 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.800+0000 7f4165a00700 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.800+0000 7f4165000700 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.800+0000 7f4165000700 1 -- 172.21.15.43:0/4279710193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 msgr2=0x561514162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.800+0000 7f4165000700 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.800+0000 7f4165000700 1 -- 172.21.15.43:0/4279710193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 msgr2=0x561514161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.800+0000 7f4165000700 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514161c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.800+0000 7f4165000700 1 -- 172.21.15.43:0/4279710193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561513fc58d0 con 0x561513fd8e40 2024-05-27T07:19:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.801+0000 7f4165000700 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514161700 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f414c00ba00 tx=0x7f414c004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1d6f7c6690538c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.801+0000 7f4147400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.801+0000 7f4147400700 1 -- 172.21.15.43:0/4279710193 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c028020 con 0x561513fd8e40 2024-05-27T07:19:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.801+0000 7f4147400700 1 -- 172.21.15.43:0/4279710193 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f414c004460 con 0x561513fd8e40 2024-05-27T07:19:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.801+0000 7f4176fa4b80 1 -- 172.21.15.43:0/4279710193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f414c004830 con 0x561513fd8e40 2024-05-27T07:19:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.801+0000 7f4147400700 1 -- 172.21.15.43:0/4279710193 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f414c030050 con 0x561513fd8e40 2024-05-27T07:19:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.802+0000 7f4176fa4b80 10 client.16272 fetch_fsmap learned FSMap version 78 2024-05-27T07:19:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.802+0000 7f4176fa4b80 1 -- 172.21.15.43:0/4279710193 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x561514046060 con 0x561513fd8e40 2024-05-27T07:19:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.802+0000 7f4147400700 1 -- 172.21.15.43:0/4279710193 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f414c004830 con 0x561513fd8e40 2024-05-27T07:19:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.802+0000 7f4147400700 1 -- 172.21.15.43:0/4279710193 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 78) ==== 6559+0+0 (secure 0 0 0) 0x7f414c003840 con 0x561513fd8e40 2024-05-27T07:19:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.802+0000 7f4176fa4b80 10 client.16272 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:19:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.802+0000 7f4176fa4b80 10 client.16272 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:19:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.802+0000 7f4176fa4b80 20 client.16272 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.802+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561514171a70 0x561514173ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.802+0000 7f4176fa4b80 4 client.16272 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:19:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.802+0000 7f4176fa4b80 1 -- 172.21.15.43:0/4279710193 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561514087d90 con 0x561514171a70 2024-05-27T07:19:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.803+0000 7f4165a00700 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561514171a70 0x561514173ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.804+0000 7f4165a00700 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561514171a70 0x561514173ed0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=23b8a82ed17110e1 server_cookie=6b36fe20a1f1cd1d in_seq=0 out_seq=0 2024-05-27T07:19:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.804+0000 7f4147400700 10 client.16272 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:19:14.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.806+0000 7f4147400700 1 -- 172.21.15.43:0/4279710193 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561514087d90 con 0x561514171a70 2024-05-27T07:19:14.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.806+0000 7f4147400700 10 client.16272 handle_command_reply: tid=0 2024-05-27T07:19:14.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.806+0000 7f4176fa4b80 1 client.16272 shutdown 2024-05-27T07:19:14.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.806+0000 7f4176fa4b80 1 -- 172.21.15.43:0/4279710193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 msgr2=0x561514161700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:14.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.806+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514161700 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f414c00ba00 tx=0x7f414c004f90 comp rx=0 tx=0).stop 2024-05-27T07:19:14.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.806+0000 7f4176fa4b80 1 -- 172.21.15.43:0/4279710193 shutdown_connections 2024-05-27T07:19:14.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.806+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561514171a70 0x561514173ed0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.807+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.807+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514161700 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.807+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/4279710193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514161c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.807+0000 7f4176fa4b80 1 -- 172.21.15.43:0/4279710193 >> 172.21.15.43:0/4279710193 conn(0x561514072390 msgr2=0x5615141573b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:14.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.807+0000 7f4147400700 0 client.16272 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:19:14.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.807+0000 7f4176fa4b80 1 -- 172.21.15.43:0/4279710193 shutdown_connections 2024-05-27T07:19:14.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.807+0000 7f4176fa4b80 1 -- 172.21.15.43:0/4279710193 wait complete. 2024-05-27T07:19:14.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.807+0000 7f4176fa4b80 20 client.16272 trim_cache size 0 max 16384 2024-05-27T07:19:14.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.826+0000 7f4176fa4b80 1 Processor -- start 2024-05-27T07:19:14.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.826+0000 7f4176fa4b80 1 -- start start 2024-05-27T07:19:14.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.826+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x56151415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.826+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.826+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514179690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.826+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f41601b4140 con 0x561513fd8e40 2024-05-27T07:19:14.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.826+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f41601b1190 con 0x5615140c49f0 2024-05-27T07:19:14.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.826+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f41601b33a0 con 0x561514157e70 2024-05-27T07:19:14.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.827+0000 7f4165000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x56151415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.827+0000 7f4165000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x56151415cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54918/0 (socket says 172.21.15.43:54918) 2024-05-27T07:19:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.827+0000 7f4165000700 1 -- 172.21.15.43:0/2611058732 learned_addr learned my addr 172.21.15.43:0/2611058732 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.827+0000 7f4165a00700 1 --2- 172.21.15.43:0/2611058732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.827+0000 7f4165000700 1 -- 172.21.15.43:0/2611058732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 msgr2=0x561514179690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.827+0000 7f4165000700 1 --2- 172.21.15.43:0/2611058732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514179690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.827+0000 7f4165000700 1 -- 172.21.15.43:0/2611058732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 msgr2=0x561514160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:14.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.827+0000 7f4165000700 1 --2- 172.21.15.43:0/2611058732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.827+0000 7f4165000700 1 -- 172.21.15.43:0/2611058732 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561514154000 con 0x561513fd8e40 2024-05-27T07:19:14.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.827+0000 7f4165a00700 1 --2- 172.21.15.43:0/2611058732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:19:14.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.828+0000 7f4165000700 1 --2- 172.21.15.43:0/2611058732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x56151415cef0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f414c00ba90 tx=0x7f414c00b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6db254f1f73924d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:14.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.828+0000 7f4152000700 1 -- 172.21.15.43:0/2611058732 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c028040 con 0x561513fd8e40 2024-05-27T07:19:14.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.828+0000 7f4152000700 1 -- 172.21.15.43:0/2611058732 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f414c025070 con 0x561513fd8e40 2024-05-27T07:19:14.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.828+0000 7f4152000700 1 -- 172.21.15.43:0/2611058732 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c003880 con 0x561513fd8e40 2024-05-27T07:19:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.829+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2611058732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 msgr2=0x56151415cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.829+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/2611058732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x56151415cef0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f414c00ba90 tx=0x7f414c00b630 comp rx=0 tx=0).stop 2024-05-27T07:19:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.829+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2611058732 shutdown_connections 2024-05-27T07:19:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.829+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/2611058732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514179690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.829+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/2611058732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x56151415cef0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.829+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/2611058732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.829+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2611058732 >> 172.21.15.43:0/2611058732 conn(0x561514072390 msgr2=0x56151415fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.829+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2611058732 shutdown_connections 2024-05-27T07:19:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.829+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2611058732 wait complete. 2024-05-27T07:19:14.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.830+0000 7f4176fa4b80 1 -- start start 2024-05-27T07:19:14.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.830+0000 7f4176fa4b80 1 Processor -- start 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514164dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514165310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4176fa4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514165850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f41601b2cd0 con 0x561513fd8e40 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f41601b1860 con 0x5615140c49f0 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4176fa4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f41601b1f30 con 0x561514157e70 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4176fa4b80 10 client.0 _ll_register_callbacks cb 0x56151401fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4165000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514164dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4165000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514164dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54922/0 (socket says 172.21.15.43:54922) 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4165000700 1 -- 172.21.15.43:0/2043984121 learned_addr learned my addr 172.21.15.43:0/2043984121 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:14.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.831+0000 7f4165a00700 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514165310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.832+0000 7f4165000700 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514165850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.832+0000 7f4165000700 1 -- 172.21.15.43:0/2043984121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 msgr2=0x561514165850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:14.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.832+0000 7f4165000700 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514165850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.832+0000 7f4165000700 1 -- 172.21.15.43:0/2043984121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 msgr2=0x561514165310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.832+0000 7f4165000700 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514165310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.832+0000 7f4165000700 1 -- 172.21.15.43:0/2043984121 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615141da550 con 0x561513fd8e40 2024-05-27T07:19:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.832+0000 7f4165000700 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514164dd0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f414c004020 tx=0x7f414c003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348147f2d530de66 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.832+0000 7f4147400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.833+0000 7f4147400700 1 -- 172.21.15.43:0/2043984121 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c040060 con 0x561513fd8e40 2024-05-27T07:19:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.833+0000 7f4147400700 1 -- 172.21.15.43:0/2043984121 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f414c003880 con 0x561513fd8e40 2024-05-27T07:19:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.833+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2043984121 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f414c025070 con 0x561513fd8e40 2024-05-27T07:19:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.833+0000 7f4147400700 1 -- 172.21.15.43:0/2043984121 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f414c047050 con 0x561513fd8e40 2024-05-27T07:19:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.833+0000 7f4176fa4b80 10 client.16278 fetch_fsmap learned FSMap version 78 2024-05-27T07:19:14.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.833+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2043984121 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x5615141db6a0 con 0x561513fd8e40 2024-05-27T07:19:14.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.834+0000 7f4147400700 1 -- 172.21.15.43:0/2043984121 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f414c025070 con 0x561513fd8e40 2024-05-27T07:19:14.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.834+0000 7f4147400700 1 -- 172.21.15.43:0/2043984121 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 78) ==== 6559+0+0 (secure 0 0 0) 0x7f414c021450 con 0x561513fd8e40 2024-05-27T07:19:14.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.834+0000 7f4176fa4b80 10 client.16278 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:19:14.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.834+0000 7f4176fa4b80 10 client.16278 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:19:14.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.834+0000 7f4176fa4b80 20 client.16278 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:14.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.834+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5615141dbc90 0x5615141de0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:14.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.834+0000 7f4176fa4b80 4 client.16278 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:19:14.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.834+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2043984121 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561514081570 con 0x5615141dbc90 2024-05-27T07:19:14.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.835+0000 7f4165a00700 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5615141dbc90 0x5615141de0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:14.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.836+0000 7f4165a00700 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5615141dbc90 0x5615141de0b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=126b269c27bb51a3 server_cookie=706507a9b021af07 in_seq=0 out_seq=0 2024-05-27T07:19:14.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.836+0000 7f4147400700 10 client.16278 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:19:14.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.836+0000 7f4147400700 1 -- 172.21.15.43:0/2043984121 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561514081570 con 0x5615141dbc90 2024-05-27T07:19:14.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.836+0000 7f4147400700 10 client.16278 handle_command_reply: tid=0 2024-05-27T07:19:14.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.836+0000 7f4176fa4b80 1 client.16278 shutdown 2024-05-27T07:19:14.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2043984121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 msgr2=0x561514164dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:14.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514164dd0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f414c004020 tx=0x7f414c003fb0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2043984121 shutdown_connections 2024-05-27T07:19:14.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5615141dbc90 0x5615141de0b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561514157e70 0x561514165850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561513fd8e40 0x561514164dd0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 1 --2- 172.21.15.43:0/2043984121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5615140c49f0 0x561514165310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2043984121 >> 172.21.15.43:0/2043984121 conn(0x561514072390 msgr2=0x5615141616c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:14.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4147400700 0 client.16278 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:19:14.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2043984121 shutdown_connections 2024-05-27T07:19:14.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 1 -- 172.21.15.43:0/2043984121 wait complete. 2024-05-27T07:19:14.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.837+0000 7f4176fa4b80 20 client.16278 trim_cache size 0 max 16384 2024-05-27T07:19:14.830 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:19:14.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.838+0000 7f4168400700 1 -- 172.21.15.43:0/23833223 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41480b2a70 msgr2=0x7f41480b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:14.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.838+0000 7f4168400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41480b2a70 0x7f41480b4f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f41580df3b0 tx=0x7f415800b040 comp rx=0 tx=0).stop 2024-05-27T07:19:14.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.838+0000 7f4168400700 1 -- 172.21.15.43:0/23833223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4160110d10 msgr2=0x7f41601a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:14.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.838+0000 7f4168400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4160110d10 0x7f41601a17d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4158007ee0 tx=0x7f415800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.838+0000 7f4168400700 1 -- 172.21.15.43:0/23833223 shutdown_connections 2024-05-27T07:19:14.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.838+0000 7f4168400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f41480b2a70 0x7f41480b4f80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.838+0000 7f4168400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4160110d10 0x7f41601a17d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.838+0000 7f4168400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4160102650 0x7f416019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.838+0000 7f4168400700 1 --2- 172.21.15.43:0/23833223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f416010e3b0 0x7f416019d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:14.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.838+0000 7f4168400700 1 -- 172.21.15.43:0/23833223 >> 172.21.15.43:0/23833223 conn(0x7f41600fe0c0 msgr2=0x7f4160105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:14.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.839+0000 7f4168400700 1 -- 172.21.15.43:0/23833223 shutdown_connections 2024-05-27T07:19:14.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:14.839+0000 7f4168400700 1 -- 172.21.15.43:0/23833223 wait complete. 2024-05-27T07:19:15.194 INFO:tasks.workunit.client.0.smithi043.stdout: 1 24646 0.67 MB/sec execute 79 sec latency 54.324 ms 2024-05-27T07:19:15.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:19:16.194 INFO:tasks.workunit.client.0.smithi043.stdout: 1 24699 0.66 MB/sec execute 80 sec latency 66.604 ms 2024-05-27T07:19:16.245 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:16.583 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:16 smithi043 ceph-mon[31531]: pgmap v183: 129 pgs: 129 active+clean; 46 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 368 KiB/s wr, 68 op/s 2024-05-27T07:19:16.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:16 smithi132 ceph-mon[35283]: pgmap v183: 129 pgs: 129 active+clean; 46 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 368 KiB/s wr, 68 op/s 2024-05-27T07:19:16.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:16 smithi179 ceph-mon[35888]: pgmap v183: 129 pgs: 129 active+clean; 46 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 368 KiB/s wr, 68 op/s 2024-05-27T07:19:17.194 INFO:tasks.workunit.client.0.smithi043.stdout: 1 24882 0.69 MB/sec execute 81 sec latency 50.137 ms 2024-05-27T07:19:18.194 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25001 0.68 MB/sec execute 82 sec latency 53.191 ms 2024-05-27T07:19:18.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:18 smithi132 ceph-mon[35283]: pgmap v184: 129 pgs: 129 active+clean; 48 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 481 KiB/s wr, 45 op/s 2024-05-27T07:19:18.686 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:18 smithi043 ceph-mon[31531]: pgmap v184: 129 pgs: 129 active+clean; 48 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 481 KiB/s wr, 45 op/s 2024-05-27T07:19:18.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:18 smithi179 ceph-mon[35888]: pgmap v184: 129 pgs: 129 active+clean; 48 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 481 KiB/s wr, 45 op/s 2024-05-27T07:19:19.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25107 0.69 MB/sec execute 83 sec latency 50.228 ms 2024-05-27T07:19:19.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.278+0000 7fefb2600700 1 -- 172.21.15.43:0/2361029382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefac10e8f0 msgr2=0x7fefac110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.278+0000 7fefb2600700 1 --2- 172.21.15.43:0/2361029382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefac10e8f0 0x7fefac110db0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fefa400b0a0 tx=0x7fefa400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.278+0000 7fefb2600700 1 -- 172.21.15.43:0/2361029382 shutdown_connections 2024-05-27T07:19:19.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.278+0000 7fefb2600700 1 --2- 172.21.15.43:0/2361029382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefac10e8f0 0x7fefac110db0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.278+0000 7fefb2600700 1 --2- 172.21.15.43:0/2361029382 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac105510 0x7fefac105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.278+0000 7fefb2600700 1 --2- 172.21.15.43:0/2361029382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefac105e60 0x7fefac10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.278+0000 7fefb2600700 1 -- 172.21.15.43:0/2361029382 >> 172.21.15.43:0/2361029382 conn(0x7fefac0fedb0 msgr2=0x7fefac1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:19.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.278+0000 7fefb2600700 1 -- 172.21.15.43:0/2361029382 shutdown_connections 2024-05-27T07:19:19.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.279+0000 7fefb2600700 1 -- 172.21.15.43:0/2361029382 wait complete. 2024-05-27T07:19:19.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.280+0000 7fefb2600700 1 Processor -- start 2024-05-27T07:19:19.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.280+0000 7fefb2600700 1 -- start start 2024-05-27T07:19:19.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.281+0000 7fefb2600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefac105510 0x7fefac1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.281+0000 7fefb2600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefac105e60 0x7fefac1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.281+0000 7fefb2600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac10e8f0 0x7fefac19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.281+0000 7fefb2600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fefac1142e0 con 0x7fefac105e60 2024-05-27T07:19:19.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.281+0000 7fefb2600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fefac114160 con 0x7fefac10e8f0 2024-05-27T07:19:19.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.281+0000 7fefb2600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fefac114460 con 0x7fefac105510 2024-05-27T07:19:19.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.281+0000 7fefab400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefac105510 0x7fefac1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.281+0000 7fefabe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac10e8f0 0x7fefac19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.282+0000 7fefabe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac10e8f0 0x7fefac19aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58620/0 (socket says 172.21.15.43:58620) 2024-05-27T07:19:19.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.282+0000 7fefabe00700 1 -- 172.21.15.43:0/2930065896 learned_addr learned my addr 172.21.15.43:0/2930065896 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:19.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.282+0000 7fefaaa00700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefac105e60 0x7fefac1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.282+0000 7fefab400700 1 -- 172.21.15.43:0/2930065896 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac10e8f0 msgr2=0x7fefac19aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.282+0000 7fefab400700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac10e8f0 0x7fefac19aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.283+0000 7fefab400700 1 -- 172.21.15.43:0/2930065896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefac105e60 msgr2=0x7fefac1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.283+0000 7fefab400700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefac105e60 0x7fefac1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.283+0000 7fefab400700 1 -- 172.21.15.43:0/2930065896 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefac19b130 con 0x7fefac105510 2024-05-27T07:19:19.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.283+0000 7fefab400700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefac105510 0x7fefac1a0860 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00d950 tx=0x7fef9c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:19.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.284+0000 7fefa3e00700 1 -- 172.21.15.43:0/2930065896 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c016070 con 0x7fefac105510 2024-05-27T07:19:19.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.284+0000 7fefa3e00700 1 -- 172.21.15.43:0/2930065896 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef9c00fd20 con 0x7fefac105510 2024-05-27T07:19:19.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.284+0000 7fefb2600700 1 -- 172.21.15.43:0/2930065896 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefac19b3c0 con 0x7fefac105510 2024-05-27T07:19:19.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.284+0000 7fefa3e00700 1 -- 172.21.15.43:0/2930065896 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c00e3b0 con 0x7fefac105510 2024-05-27T07:19:19.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.284+0000 7fefb2600700 1 -- 172.21.15.43:0/2930065896 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefac1a76f0 con 0x7fefac105510 2024-05-27T07:19:19.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.289+0000 7fefc1192b80 1 Processor -- start 2024-05-27T07:19:19.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.289+0000 7fefc1192b80 1 -- start start 2024-05-27T07:19:19.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516faf9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516faf9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516faf9d00 0x56516fb0a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefc1192b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56516fa27e40 con 0x56516f97ae40 2024-05-27T07:19:19.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefc1192b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56516fa179e0 con 0x56516fa669f0 2024-05-27T07:19:19.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefc1192b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56516fa1ba20 con 0x56516faf9d00 2024-05-27T07:19:19.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefa3e00700 1 -- 172.21.15.43:0/2930065896 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fef9c019070 con 0x7fefac105510 2024-05-27T07:19:19.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516faf9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefabe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516faf9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516faf9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54950/0 (socket says 172.21.15.43:54950) 2024-05-27T07:19:19.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefab400700 1 -- 172.21.15.43:0/4189492917 learned_addr learned my addr 172.21.15.43:0/4189492917 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:19.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.290+0000 7fefaaa00700 1 --2- 172.21.15.43:0/4189492917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516faf9d00 0x56516fb0a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.291+0000 7fefab400700 1 -- 172.21.15.43:0/4189492917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516faf9d00 msgr2=0x56516fb0a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:19.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.291+0000 7fefab400700 1 --2- 172.21.15.43:0/4189492917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516faf9d00 0x56516fb0a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.291+0000 7fefab400700 1 -- 172.21.15.43:0/4189492917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 msgr2=0x56516faf9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:19.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.291+0000 7fefab400700 1 --2- 172.21.15.43:0/4189492917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516faf9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.291+0000 7fefab400700 1 -- 172.21.15.43:0/4189492917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56516f9e0420 con 0x56516f97ae40 2024-05-27T07:19:19.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.291+0000 7fefab400700 1 --2- 172.21.15.43:0/4189492917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516faf9770 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0df6c0 tx=0x7fef9c0ea880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed63d3f03452d903 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:19.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.292+0000 7fefa0c00700 1 -- 172.21.15.43:0/4189492917 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c0e4520 con 0x56516f97ae40 2024-05-27T07:19:19.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.292+0000 7fefa0c00700 1 -- 172.21.15.43:0/4189492917 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef9c0e2c60 con 0x56516f97ae40 2024-05-27T07:19:19.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.292+0000 7fefa0c00700 1 -- 172.21.15.43:0/4189492917 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c0f1ab0 con 0x56516f97ae40 2024-05-27T07:19:19.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.292+0000 7fefa3e00700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fef900b29f0 0x7fef900b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.292+0000 7fefa3e00700 1 -- 172.21.15.43:0/2930065896 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fef9c0e23a0 con 0x7fefac105510 2024-05-27T07:19:19.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.293+0000 7fefabe00700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fef900b29f0 0x7fef900b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.293+0000 7fefc1192b80 1 -- 172.21.15.43:0/4189492917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 msgr2=0x56516faf9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.293+0000 7fefc1192b80 1 --2- 172.21.15.43:0/4189492917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516faf9770 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0df6c0 tx=0x7fef9c0ea880 comp rx=0 tx=0).stop 2024-05-27T07:19:19.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.293+0000 7fefc1192b80 1 -- 172.21.15.43:0/4189492917 shutdown_connections 2024-05-27T07:19:19.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.293+0000 7fefc1192b80 1 --2- 172.21.15.43:0/4189492917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516faf9d00 0x56516fb0a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.293+0000 7fefc1192b80 1 --2- 172.21.15.43:0/4189492917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516faf9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.293+0000 7fefc1192b80 1 --2- 172.21.15.43:0/4189492917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516faf9770 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.293+0000 7fefc1192b80 1 -- 172.21.15.43:0/4189492917 >> 172.21.15.43:0/4189492917 conn(0x56516fa14390 msgr2=0x56516faf54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:19.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.293+0000 7fefc1192b80 1 -- 172.21.15.43:0/4189492917 shutdown_connections 2024-05-27T07:19:19.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.294+0000 7fefabe00700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fef900b29f0 0x7fef900b4f00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fefa4009e60 tx=0x7fefa4006210 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:19.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.294+0000 7fefc1192b80 1 -- 172.21.15.43:0/4189492917 wait complete. 2024-05-27T07:19:19.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.295+0000 7fefc1192b80 1 -- start start 2024-05-27T07:19:19.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.295+0000 7fefc1192b80 1 Processor -- start 2024-05-27T07:19:19.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516f97ae40 0x56516fb07b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516fb080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb08610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefc1192b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56516f91dfa0 con 0x56516faf9d00 2024-05-27T07:19:19.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefc1192b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56516f9b8760 con 0x56516fa669f0 2024-05-27T07:19:19.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefc1192b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56516fb0afa0 con 0x56516f97ae40 2024-05-27T07:19:19.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefc1192b80 10 client.0 _ll_register_callbacks cb 0x56516f9c1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:19.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefabe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb08610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefabe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb08610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54960/0 (socket says 172.21.15.43:54960) 2024-05-27T07:19:19.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefabe00700 1 -- 172.21.15.43:0/2826798727 learned_addr learned my addr 172.21.15.43:0/2826798727 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:19.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefab400700 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516f97ae40 0x56516fb07b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.296+0000 7fefaaa00700 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516fb080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.297+0000 7fefabe00700 1 -- 172.21.15.43:0/2826798727 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516f97ae40 msgr2=0x56516fb07b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:19.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.297+0000 7fefabe00700 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516f97ae40 0x56516fb07b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.297+0000 7fefabe00700 1 -- 172.21.15.43:0/2826798727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 msgr2=0x56516fb080d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:19.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.297+0000 7fefabe00700 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516fb080d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.297+0000 7fefabe00700 1 -- 172.21.15.43:0/2826798727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56516f9678d0 con 0x56516faf9d00 2024-05-27T07:19:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.297+0000 7fefabe00700 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb08610 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fefa400d010 tx=0x7fefa40099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe27f3dbe95f4b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.297+0000 7fef95600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.298+0000 7fef95600700 1 -- 172.21.15.43:0/2826798727 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa402d070 con 0x56516faf9d00 2024-05-27T07:19:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.298+0000 7fef95600700 1 -- 172.21.15.43:0/2826798727 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fefa4005040 con 0x56516faf9d00 2024-05-27T07:19:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.298+0000 7fefc1192b80 1 -- 172.21.15.43:0/2826798727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef9c0e2c60 con 0x56516faf9d00 2024-05-27T07:19:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.298+0000 7fef95600700 1 -- 172.21.15.43:0/2826798727 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7fefa4007dd0 con 0x56516faf9d00 2024-05-27T07:19:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.298+0000 7fefc1192b80 10 client.16290 fetch_fsmap learned FSMap version 78 2024-05-27T07:19:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.298+0000 7fefc1192b80 1 -- 172.21.15.43:0/2826798727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x56516f9e8060 con 0x56516faf9d00 2024-05-27T07:19:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.298+0000 7fef95600700 1 -- 172.21.15.43:0/2826798727 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef9c0e2c60 con 0x56516faf9d00 2024-05-27T07:19:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.299+0000 7fef95600700 1 -- 172.21.15.43:0/2826798727 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 78) ==== 6559+0+0 (secure 0 0 0) 0x7fefa4011070 con 0x56516faf9d00 2024-05-27T07:19:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.299+0000 7fefc1192b80 10 client.16290 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:19:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.299+0000 7fefc1192b80 10 client.16290 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:19:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.299+0000 7fefc1192b80 20 client.16290 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.299+0000 7fefc1192b80 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x56516fb18140 0x56516fb1a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.299+0000 7fefc1192b80 4 client.16290 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:19:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.299+0000 7fefc1192b80 1 -- 172.21.15.43:0/2826798727 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56516fa29d90 con 0x56516fb18140 2024-05-27T07:19:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.300+0000 7fefab400700 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x56516fb18140 0x56516fb1a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.301+0000 7fef95600700 10 client.16290 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:19:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.301+0000 7fefab400700 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x56516fb18140 0x56516fb1a5f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8043489c019d2034 server_cookie=9943ae766fd97a82 in_seq=0 out_seq=0 2024-05-27T07:19:19.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.303+0000 7fef95600700 1 -- 172.21.15.43:0/2826798727 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56516fa29d90 con 0x56516fb18140 2024-05-27T07:19:19.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.303+0000 7fef95600700 10 client.16290 handle_command_reply: tid=0 2024-05-27T07:19:19.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.303+0000 7fefc1192b80 1 client.16290 shutdown 2024-05-27T07:19:19.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 1 -- 172.21.15.43:0/2826798727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 msgr2=0x56516fb08610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb08610 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fefa400d010 tx=0x7fefa40099e0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 1 -- 172.21.15.43:0/2826798727 shutdown_connections 2024-05-27T07:19:19.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x56516fb18140 0x56516fb1a5f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb08610 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516f97ae40 0x56516fb07b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fef95600700 0 client.16290 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:19:19.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 1 --2- 172.21.15.43:0/2826798727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516fb080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 1 -- 172.21.15.43:0/2826798727 >> 172.21.15.43:0/2826798727 conn(0x56516fa14390 msgr2=0x56516faf9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:19.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 1 -- 172.21.15.43:0/2826798727 shutdown_connections 2024-05-27T07:19:19.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 1 -- 172.21.15.43:0/2826798727 wait complete. 2024-05-27T07:19:19.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.304+0000 7fefc1192b80 20 client.16290 trim_cache size 0 max 16384 2024-05-27T07:19:19.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.322+0000 7fefc1192b80 1 Processor -- start 2024-05-27T07:19:19.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.322+0000 7fefc1192b80 1 -- start start 2024-05-27T07:19:19.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.323+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516fb06b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.323+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516fb14040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.323+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516faf9d00 0x56516fb1fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.323+0000 7fefc1192b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fefac1afc30 con 0x56516f97ae40 2024-05-27T07:19:19.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.323+0000 7fefc1192b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fefac1acc80 con 0x56516fa669f0 2024-05-27T07:19:19.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.323+0000 7fefc1192b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fefac1aee90 con 0x56516faf9d00 2024-05-27T07:19:19.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.323+0000 7fefab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516fb06b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.323+0000 7fefab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516fb06b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54970/0 (socket says 172.21.15.43:54970) 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.323+0000 7fefab400700 1 -- 172.21.15.43:0/1301605992 learned_addr learned my addr 172.21.15.43:0/1301605992 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.323+0000 7fefaaa00700 1 --2- 172.21.15.43:0/1301605992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516fb14040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.324+0000 7fefabe00700 1 --2- 172.21.15.43:0/1301605992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516faf9d00 0x56516fb1fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.324+0000 7fefab400700 1 -- 172.21.15.43:0/1301605992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516faf9d00 msgr2=0x56516fb1fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.324+0000 7fefab400700 1 --2- 172.21.15.43:0/1301605992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516faf9d00 0x56516fb1fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.324+0000 7fefab400700 1 -- 172.21.15.43:0/1301605992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 msgr2=0x56516fb14040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.324+0000 7fefab400700 1 --2- 172.21.15.43:0/1301605992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516fb14040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.324+0000 7fefab400700 1 -- 172.21.15.43:0/1301605992 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56516faf1240 con 0x56516f97ae40 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.324+0000 7fefab400700 1 --2- 172.21.15.43:0/1301605992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516fb06b90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0a5390 tx=0x7fef9c0f1650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f03e02932d79ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.324+0000 7fef97e00700 1 -- 172.21.15.43:0/1301605992 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c020030 con 0x56516f97ae40 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.324+0000 7fef97e00700 1 -- 172.21.15.43:0/1301605992 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef9c0a8e20 con 0x56516f97ae40 2024-05-27T07:19:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.324+0000 7fef97e00700 1 -- 172.21.15.43:0/1301605992 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c0a8c60 con 0x56516f97ae40 2024-05-27T07:19:19.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.325+0000 7fefc1192b80 1 -- 172.21.15.43:0/1301605992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 msgr2=0x56516fb06b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.325+0000 7fefc1192b80 1 --2- 172.21.15.43:0/1301605992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516fb06b90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0a5390 tx=0x7fef9c0f1650 comp rx=0 tx=0).stop 2024-05-27T07:19:19.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.326+0000 7fefc1192b80 1 -- 172.21.15.43:0/1301605992 shutdown_connections 2024-05-27T07:19:19.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.326+0000 7fefc1192b80 1 --2- 172.21.15.43:0/1301605992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516faf9d00 0x56516fb1fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.326+0000 7fefc1192b80 1 --2- 172.21.15.43:0/1301605992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516f97ae40 0x56516fb06b90 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.326+0000 7fefc1192b80 1 --2- 172.21.15.43:0/1301605992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516fa669f0 0x56516fb14040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.326+0000 7fefc1192b80 1 -- 172.21.15.43:0/1301605992 >> 172.21.15.43:0/1301605992 conn(0x56516fa14390 msgr2=0x56516fb06480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:19.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.326+0000 7fefc1192b80 1 -- 172.21.15.43:0/1301605992 shutdown_connections 2024-05-27T07:19:19.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.326+0000 7fefc1192b80 1 -- 172.21.15.43:0/1301605992 wait complete. 2024-05-27T07:19:19.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.327+0000 7fefc1192b80 1 -- start start 2024-05-27T07:19:19.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.327+0000 7fefc1192b80 1 Processor -- start 2024-05-27T07:19:19.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516f97ae40 0x56516fb01bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516fa669f0 0x56516fb020f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefc1192b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb0f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefc1192b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fefac1ae7c0 con 0x56516faf9d00 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefc1192b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fefac1ad350 con 0x56516f97ae40 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefc1192b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fefac1ada20 con 0x56516fa669f0 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefc1192b80 10 client.0 _ll_register_callbacks cb 0x56516f9c1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefabe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb0f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefabe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb0f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54982/0 (socket says 172.21.15.43:54982) 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefabe00700 1 -- 172.21.15.43:0/1495779097 learned_addr learned my addr 172.21.15.43:0/1495779097 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefaaa00700 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516fa669f0 0x56516fb020f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefab400700 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516f97ae40 0x56516fb01bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefabe00700 1 -- 172.21.15.43:0/1495779097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516fa669f0 msgr2=0x56516fb020f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:19.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefabe00700 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516fa669f0 0x56516fb020f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefabe00700 1 -- 172.21.15.43:0/1495779097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516f97ae40 msgr2=0x56516fb01bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:19.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.328+0000 7fefabe00700 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516f97ae40 0x56516fb01bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.329+0000 7fefabe00700 1 -- 172.21.15.43:0/1495779097 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56516fb08980 con 0x56516faf9d00 2024-05-27T07:19:19.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.329+0000 7fefabe00700 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb0f230 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fefa400b070 tx=0x7fefa4007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2541be1f694b7b79 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:19.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.329+0000 7fef95600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:19.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.329+0000 7fef95600700 1 -- 172.21.15.43:0/1495779097 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa402d040 con 0x56516faf9d00 2024-05-27T07:19:19.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.329+0000 7fef95600700 1 -- 172.21.15.43:0/1495779097 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fefa401c5b0 con 0x56516faf9d00 2024-05-27T07:19:19.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.329+0000 7fefc1192b80 1 -- 172.21.15.43:0/1495779097 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef9c0a8e20 con 0x56516faf9d00 2024-05-27T07:19:19.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fef95600700 1 -- 172.21.15.43:0/1495779097 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7fefa401c7b0 con 0x56516faf9d00 2024-05-27T07:19:19.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fefc1192b80 10 client.16296 fetch_fsmap learned FSMap version 78 2024-05-27T07:19:19.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fefc1192b80 1 -- 172.21.15.43:0/1495779097 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x56516faf9ae0 con 0x56516faf9d00 2024-05-27T07:19:19.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fef95600700 1 -- 172.21.15.43:0/1495779097 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef9c0a8e20 con 0x56516faf9d00 2024-05-27T07:19:19.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fef95600700 1 -- 172.21.15.43:0/1495779097 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 78) ==== 6559+0+0 (secure 0 0 0) 0x7fefa403c070 con 0x56516faf9d00 2024-05-27T07:19:19.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fefc1192b80 10 client.16296 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:19:19.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fefc1192b80 10 client.16296 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:19:19.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fefc1192b80 20 client.16296 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:19.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fefc1192b80 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x56516fb10b40 0x56516fb84cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:19.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fefc1192b80 4 client.16296 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:19:19.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.330+0000 7fefc1192b80 1 -- 172.21.15.43:0/1495779097 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56516fa23570 con 0x56516fb10b40 2024-05-27T07:19:19.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.331+0000 7fefab400700 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x56516fb10b40 0x56516fb84cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:19.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.332+0000 7fefab400700 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x56516fb10b40 0x56516fb84cd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f76d69d002c9d60f server_cookie=d09d8df9ed355ebe in_seq=0 out_seq=0 2024-05-27T07:19:19.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.332+0000 7fef95600700 10 client.16296 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:19:19.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.334+0000 7fef95600700 1 -- 172.21.15.43:0/1495779097 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56516fa23570 con 0x56516fb10b40 2024-05-27T07:19:19.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.334+0000 7fef95600700 10 client.16296 handle_command_reply: tid=0 2024-05-27T07:19:19.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.334+0000 7fefc1192b80 1 client.16296 shutdown 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fefc1192b80 1 -- 172.21.15.43:0/1495779097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 msgr2=0x56516fb0f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fefc1192b80 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb0f230 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fefa400b070 tx=0x7fefa4007bd0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fefc1192b80 1 -- 172.21.15.43:0/1495779097 shutdown_connections 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fefc1192b80 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x56516fb10b40 0x56516fb84cd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fef95600700 0 client.16296 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fefc1192b80 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56516faf9d00 0x56516fb0f230 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fefc1192b80 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56516f97ae40 0x56516fb01bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fefc1192b80 1 --2- 172.21.15.43:0/1495779097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56516fa669f0 0x56516fb020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fefc1192b80 1 -- 172.21.15.43:0/1495779097 >> 172.21.15.43:0/1495779097 conn(0x56516fa14390 msgr2=0x56516fb091d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fefc1192b80 1 -- 172.21.15.43:0/1495779097 shutdown_connections 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.335+0000 7fefc1192b80 1 -- 172.21.15.43:0/1495779097 wait complete. 2024-05-27T07:19:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.336+0000 7fefc1192b80 20 client.16296 trim_cache size 0 max 16384 2024-05-27T07:19:19.328 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:19:19.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.336+0000 7fefb2600700 1 -- 172.21.15.43:0/2930065896 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fef900b29f0 msgr2=0x7fef900b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.336+0000 7fefb2600700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fef900b29f0 0x7fef900b4f00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fefa4009e60 tx=0x7fefa4006210 comp rx=0 tx=0).stop 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.336+0000 7fefb2600700 1 -- 172.21.15.43:0/2930065896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefac105510 msgr2=0x7fefac1a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.336+0000 7fefb2600700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefac105510 0x7fefac1a0860 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00d950 tx=0x7fef9c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.337+0000 7fefb2600700 1 -- 172.21.15.43:0/2930065896 shutdown_connections 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.337+0000 7fefb2600700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fef900b29f0 0x7fef900b4f00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.337+0000 7fefb2600700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac10e8f0 0x7fefac19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.337+0000 7fefb2600700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefac105510 0x7fefac1a0860 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.337+0000 7fefb2600700 1 --2- 172.21.15.43:0/2930065896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fefac105e60 0x7fefac1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.337+0000 7fefb2600700 1 -- 172.21.15.43:0/2930065896 >> 172.21.15.43:0/2930065896 conn(0x7fefac0fedb0 msgr2=0x7fefac0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.337+0000 7fefb2600700 1 -- 172.21.15.43:0/2930065896 shutdown_connections 2024-05-27T07:19:19.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:19.337+0000 7fefb2600700 1 -- 172.21.15.43:0/2930065896 wait complete. 2024-05-27T07:19:20.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:19:20.194 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25163 0.68 MB/sec execute 84 sec latency 44.296 ms 2024-05-27T07:19:20.454 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:20 smithi043 ceph-mon[31531]: pgmap v185: 129 pgs: 129 active+clean; 48 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 422 KiB/s wr, 39 op/s 2024-05-27T07:19:20.454 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:20 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:19:20.454 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:20 smithi043 systemd[1]: Stopped Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:19:20.454 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:20 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Consumed 1.806s CPU time. 2024-05-27T07:19:20.455 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:20 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Start request repeated too quickly. 2024-05-27T07:19:20.455 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:20 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-05-27T07:19:20.455 INFO:journalctl@ceph.mds.j.smithi043.stdout:May 27 07:19:20 smithi043 systemd[1]: Failed to start Ceph mds.j for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:19:20.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:20 smithi132 ceph-mon[35283]: pgmap v185: 129 pgs: 129 active+clean; 48 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 422 KiB/s wr, 39 op/s 2024-05-27T07:19:20.628 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:20.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:20 smithi179 ceph-mon[35888]: pgmap v185: 129 pgs: 129 active+clean; 48 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 422 KiB/s wr, 39 op/s 2024-05-27T07:19:21.194 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25242 0.68 MB/sec execute 85 sec latency 50.470 ms 2024-05-27T07:19:22.194 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25312 0.67 MB/sec execute 86 sec latency 47.133 ms 2024-05-27T07:19:22.464 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:22 smithi043 ceph-mon[31531]: pgmap v186: 129 pgs: 129 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 638 KiB/s wr, 57 op/s 2024-05-27T07:19:22.464 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:22 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:22.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:22 smithi132 ceph-mon[35283]: pgmap v186: 129 pgs: 129 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 638 KiB/s wr, 57 op/s 2024-05-27T07:19:22.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:22 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:22.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:22 smithi179 ceph-mon[35888]: pgmap v186: 129 pgs: 129 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 638 KiB/s wr, 57 op/s 2024-05-27T07:19:22.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:22 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:19:22.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.854+0000 7f52c3200700 1 -- 172.21.15.43:0/2733069610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52c4105280 msgr2=0x7f52c4105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:22.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.854+0000 7f52c3200700 1 --2- 172.21.15.43:0/2733069610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52c4105280 0x7f52c4105690 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f52b4009990 tx=0x7f52b4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.854+0000 7f52c3200700 1 -- 172.21.15.43:0/2733069610 shutdown_connections 2024-05-27T07:19:22.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.854+0000 7f52c3200700 1 --2- 172.21.15.43:0/2733069610 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52c410e6f0 0x7f52c4110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.854+0000 7f52c3200700 1 --2- 172.21.15.43:0/2733069610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52c4105280 0x7f52c4105690 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.854+0000 7f52c3200700 1 --2- 172.21.15.43:0/2733069610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52c4105bd0 0x7f52c410e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.854+0000 7f52c3200700 1 -- 172.21.15.43:0/2733069610 >> 172.21.15.43:0/2733069610 conn(0x7f52c4078ab0 msgr2=0x7f52c4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:22.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.854+0000 7f52c3200700 1 -- 172.21.15.43:0/2733069610 shutdown_connections 2024-05-27T07:19:22.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.854+0000 7f52c3200700 1 -- 172.21.15.43:0/2733069610 wait complete. 2024-05-27T07:19:22.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.855+0000 7f52c3200700 1 Processor -- start 2024-05-27T07:19:22.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.856+0000 7f52c3200700 1 -- start start 2024-05-27T07:19:22.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.856+0000 7f52c3200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52c4105280 0x7f52c4102f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.856+0000 7f52c3200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52c4105bd0 0x7f52c41034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.856+0000 7f52c3200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52c410e6f0 0x7f52c41039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.856+0000 7f52c3200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f52c4114140 con 0x7f52c4105280 2024-05-27T07:19:22.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.856+0000 7f52c3200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52c4113fc0 con 0x7f52c4105bd0 2024-05-27T07:19:22.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.856+0000 7f52c3200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41142c0 con 0x7f52c410e6f0 2024-05-27T07:19:22.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.857+0000 7f52c1e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52c4105280 0x7f52c4102f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.857+0000 7f52c1400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52c4105bd0 0x7f52c41034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.857+0000 7f52c1e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52c4105280 0x7f52c4102f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55002/0 (socket says 172.21.15.43:55002) 2024-05-27T07:19:22.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.857+0000 7f52c1e00700 1 -- 172.21.15.43:0/4201050443 learned_addr learned my addr 172.21.15.43:0/4201050443 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:22.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.857+0000 7f52c2800700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52c410e6f0 0x7f52c41039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.858+0000 7f52c1e00700 1 -- 172.21.15.43:0/4201050443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52c410e6f0 msgr2=0x7f52c41039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:22.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.858+0000 7f52c1e00700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52c410e6f0 0x7f52c41039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.858+0000 7f52c1e00700 1 -- 172.21.15.43:0/4201050443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52c4105bd0 msgr2=0x7f52c41034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:22.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.858+0000 7f52c1e00700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52c4105bd0 0x7f52c41034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.858+0000 7f52c1e00700 1 -- 172.21.15.43:0/4201050443 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c41a8f10 con 0x7f52c4105280 2024-05-27T07:19:22.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.858+0000 7f52c1e00700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52c4105280 0x7f52c4102f70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f52b4000c00 tx=0x7f52b40057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:22.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.858+0000 7f52baa00700 1 -- 172.21.15.43:0/4201050443 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b4027070 con 0x7f52c4105280 2024-05-27T07:19:22.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.858+0000 7f52c3200700 1 -- 172.21.15.43:0/4201050443 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c41a91a0 con 0x7f52c4105280 2024-05-27T07:19:22.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.858+0000 7f52baa00700 1 -- 172.21.15.43:0/4201050443 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52b400fb00 con 0x7f52c4105280 2024-05-27T07:19:22.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.859+0000 7f52baa00700 1 -- 172.21.15.43:0/4201050443 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b4022c00 con 0x7f52c4105280 2024-05-27T07:19:22.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.859+0000 7f52c3200700 1 -- 172.21.15.43:0/4201050443 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c41a95f0 con 0x7f52c4105280 2024-05-27T07:19:22.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.865+0000 7f52d7626b80 1 Processor -- start 2024-05-27T07:19:22.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.865+0000 7f52d7626b80 1 -- start start 2024-05-27T07:19:22.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.865+0000 7f52baa00700 1 -- 172.21.15.43:0/4201050443 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f52b402a070 con 0x7f52c4105280 2024-05-27T07:19:22.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.865+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb96084ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.865+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085410 0x55eb9608fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.865+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085820 0x55eb96092020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.865+0000 7f52d7626b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55eb95fb3e40 con 0x55eb96085820 2024-05-27T07:19:22.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.865+0000 7f52d7626b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55eb95fa39e0 con 0x55eb96085410 2024-05-27T07:19:22.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.865+0000 7f52d7626b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55eb95fa7a20 con 0x55eb95f06e40 2024-05-27T07:19:22.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.866+0000 7f52c1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085820 0x55eb96092020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.866+0000 7f52c1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085820 0x55eb96092020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55008/0 (socket says 172.21.15.43:55008) 2024-05-27T07:19:22.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.866+0000 7f52c1400700 1 -- 172.21.15.43:0/1790343855 learned_addr learned my addr 172.21.15.43:0/1790343855 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:22.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.866+0000 7f52c2800700 1 --2- 172.21.15.43:0/1790343855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb96084ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.866+0000 7f52c1e00700 1 --2- 172.21.15.43:0/1790343855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085410 0x55eb9608fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.866+0000 7f52c1400700 1 -- 172.21.15.43:0/1790343855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 msgr2=0x55eb96084ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:22.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.866+0000 7f52c1400700 1 --2- 172.21.15.43:0/1790343855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb96084ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.867+0000 7f52c1400700 1 -- 172.21.15.43:0/1790343855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085410 msgr2=0x55eb9608fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:22.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.867+0000 7f52c1400700 1 --2- 172.21.15.43:0/1790343855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085410 0x55eb9608fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.867+0000 7f52c1400700 1 -- 172.21.15.43:0/1790343855 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb95f6c420 con 0x55eb96085820 2024-05-27T07:19:22.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.867+0000 7f52c1400700 1 --2- 172.21.15.43:0/1790343855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085820 0x55eb96092020 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f52b000bd30 tx=0x7f52b00106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36dd1613fe35595b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:22.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.868+0000 7f52baa00700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52a80b29f0 0x7f52a80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.868+0000 7f52af400700 1 -- 172.21.15.43:0/1790343855 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0010e10 con 0x55eb96085820 2024-05-27T07:19:22.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.868+0000 7f52baa00700 1 -- 172.21.15.43:0/4201050443 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f52b40e3960 con 0x7f52c4105280 2024-05-27T07:19:22.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.868+0000 7f52af400700 1 -- 172.21.15.43:0/1790343855 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52b00044c0 con 0x55eb96085820 2024-05-27T07:19:22.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.868+0000 7f52af400700 1 -- 172.21.15.43:0/1790343855 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0011440 con 0x55eb96085820 2024-05-27T07:19:22.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.868+0000 7f52c2800700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52a80b29f0 0x7f52a80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.869+0000 7f52d7626b80 1 -- 172.21.15.43:0/1790343855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085820 msgr2=0x55eb96092020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:22.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.869+0000 7f52c2800700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52a80b29f0 0x7f52a80b4f00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x55eb96081690 tx=0x7f52bc009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:22.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.869+0000 7f52d7626b80 1 --2- 172.21.15.43:0/1790343855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085820 0x55eb96092020 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f52b000bd30 tx=0x7f52b00106b0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.870+0000 7f52d7626b80 1 -- 172.21.15.43:0/1790343855 shutdown_connections 2024-05-27T07:19:22.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.870+0000 7f52d7626b80 1 --2- 172.21.15.43:0/1790343855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085820 0x55eb96092020 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.870+0000 7f52d7626b80 1 --2- 172.21.15.43:0/1790343855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb96084ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.870+0000 7f52d7626b80 1 --2- 172.21.15.43:0/1790343855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085410 0x55eb9608fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.870+0000 7f52d7626b80 1 -- 172.21.15.43:0/1790343855 >> 172.21.15.43:0/1790343855 conn(0x55eb95f66fc0 msgr2=0x55eb96080d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:22.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.870+0000 7f52d7626b80 1 -- 172.21.15.43:0/1790343855 shutdown_connections 2024-05-27T07:19:22.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.870+0000 7f52d7626b80 1 -- 172.21.15.43:0/1790343855 wait complete. 2024-05-27T07:19:22.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.871+0000 7f52d7626b80 1 -- start start 2024-05-27T07:19:22.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.871+0000 7f52d7626b80 1 Processor -- start 2024-05-27T07:19:22.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.871+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb9608f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 0x55eb9608fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9608ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52d7626b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55eb95ea9fa0 con 0x55eb96085410 2024-05-27T07:19:22.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52d7626b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55eb95f44760 con 0x55eb96085820 2024-05-27T07:19:22.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52d7626b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55eb96092990 con 0x55eb95f06e40 2024-05-27T07:19:22.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52d7626b80 10 client.0 _ll_register_callbacks cb 0x55eb95f4dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:22.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52c1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 0x55eb9608fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52c1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 0x55eb9608fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55016/0 (socket says 172.21.15.43:55016) 2024-05-27T07:19:22.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52c1400700 1 -- 172.21.15.43:0/3905624957 learned_addr learned my addr 172.21.15.43:0/3905624957 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:22.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52c1e00700 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb9608f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.872+0000 7f52c2800700 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9608ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.873+0000 7f52c1400700 1 -- 172.21.15.43:0/3905624957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 msgr2=0x55eb9608f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:22.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.873+0000 7f52c1400700 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb9608f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.873+0000 7f52c1400700 1 -- 172.21.15.43:0/3905624957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 msgr2=0x55eb9608ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:22.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.873+0000 7f52c1400700 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9608ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.873+0000 7f52c1400700 1 -- 172.21.15.43:0/3905624957 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb95ef38d0 con 0x55eb96085410 2024-05-27T07:19:22.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.873+0000 7f52c2800700 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9608ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:19:22.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.873+0000 7f52c1400700 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 0x55eb9608fa60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f52b0005d60 tx=0x7f52b0005320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e086f5c3f6e4fc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.873+0000 7f52a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.874+0000 7f52a3e00700 1 -- 172.21.15.43:0/3905624957 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0004950 con 0x55eb96085410 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.874+0000 7f52d7626b80 1 -- 172.21.15.43:0/3905624957 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52b00044c0 con 0x55eb96085410 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.874+0000 7f52a3e00700 1 -- 172.21.15.43:0/3905624957 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f52b0004af0 con 0x55eb96085410 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.874+0000 7f52a3e00700 1 -- 172.21.15.43:0/3905624957 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f52b002e050 con 0x55eb96085410 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.874+0000 7f52d7626b80 10 client.16308 fetch_fsmap learned FSMap version 78 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.874+0000 7f52d7626b80 1 -- 172.21.15.43:0/3905624957 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55eb95f74060 con 0x55eb96085410 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.875+0000 7f52a3e00700 1 -- 172.21.15.43:0/3905624957 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52b00044c0 con 0x55eb96085410 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.875+0000 7f52a3e00700 1 -- 172.21.15.43:0/3905624957 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 78) ==== 6559+0+0 (secure 0 0 0) 0x7f52b0023020 con 0x55eb96085410 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.875+0000 7f52d7626b80 10 client.16308 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.875+0000 7f52d7626b80 10 client.16308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.875+0000 7f52d7626b80 20 client.16308 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:22.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.875+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55eb9609f960 0x55eb960a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.875+0000 7f52d7626b80 4 client.16308 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:19:22.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.875+0000 7f52d7626b80 1 -- 172.21.15.43:0/3905624957 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb95fb5d90 con 0x55eb9609f960 2024-05-27T07:19:22.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.876+0000 7f52c1e00700 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55eb9609f960 0x55eb960a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.876+0000 7f52c1e00700 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55eb9609f960 0x55eb960a1db0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28e5118b8a322212 server_cookie=8fbb926908266c54 in_seq=0 out_seq=0 2024-05-27T07:19:22.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.876+0000 7f52a3e00700 10 client.16308 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:22.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52a3e00700 1 -- 172.21.15.43:0/3905624957 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55eb95fb5d90 con 0x55eb9609f960 2024-05-27T07:19:22.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52a3e00700 10 client.16308 handle_command_reply: tid=0 2024-05-27T07:19:22.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52d7626b80 1 client.16308 shutdown 2024-05-27T07:19:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52d7626b80 1 -- 172.21.15.43:0/3905624957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 msgr2=0x55eb9608fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 0x55eb9608fa60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f52b0005d60 tx=0x7f52b0005320 comp rx=0 tx=0).stop 2024-05-27T07:19:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52d7626b80 1 -- 172.21.15.43:0/3905624957 shutdown_connections 2024-05-27T07:19:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9608ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb9608f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55eb9609f960 0x55eb960a1db0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3905624957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 0x55eb9608fa60 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52d7626b80 1 -- 172.21.15.43:0/3905624957 >> 172.21.15.43:0/3905624957 conn(0x55eb95f66fc0 msgr2=0x55eb9607fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:22.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.881+0000 7f52a3e00700 0 client.16308 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:22.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.882+0000 7f52d7626b80 1 -- 172.21.15.43:0/3905624957 shutdown_connections 2024-05-27T07:19:22.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.882+0000 7f52d7626b80 1 -- 172.21.15.43:0/3905624957 wait complete. 2024-05-27T07:19:22.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.882+0000 7f52d7626b80 20 client.16308 trim_cache size 0 max 16384 2024-05-27T07:19:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.914+0000 7f52d7626b80 1 Processor -- start 2024-05-27T07:19:22.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.914+0000 7f52d7626b80 1 -- start start 2024-05-27T07:19:22.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.915+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb960897c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.915+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 0x55eb9608e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.915+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9609ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.916+0000 7f52d7626b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41b3d70 con 0x55eb96085410 2024-05-27T07:19:22.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.916+0000 7f52d7626b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41b0dc0 con 0x55eb96085820 2024-05-27T07:19:22.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.916+0000 7f52d7626b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41b2fd0 con 0x55eb95f06e40 2024-05-27T07:19:22.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.916+0000 7f52c1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 0x55eb9608e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.916+0000 7f52c2800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9609ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.916+0000 7f52c2800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9609ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58696/0 (socket says 172.21.15.43:58696) 2024-05-27T07:19:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.916+0000 7f52c2800700 1 -- 172.21.15.43:0/1444972957 learned_addr learned my addr 172.21.15.43:0/1444972957 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.917+0000 7f52c1e00700 1 --2- 172.21.15.43:0/1444972957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb960897c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.917+0000 7f52c2800700 1 -- 172.21.15.43:0/1444972957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 msgr2=0x55eb960897c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:22.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.917+0000 7f52c2800700 1 --2- 172.21.15.43:0/1444972957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb960897c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.917+0000 7f52c2800700 1 -- 172.21.15.43:0/1444972957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 msgr2=0x55eb9608e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:22.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.917+0000 7f52c2800700 1 --2- 172.21.15.43:0/1444972957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 0x55eb9608e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.917+0000 7f52c2800700 1 -- 172.21.15.43:0/1444972957 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb9608da90 con 0x55eb96085820 2024-05-27T07:19:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.918+0000 7f52c2800700 1 --2- 172.21.15.43:0/1444972957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9609ba60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009550 tx=0x7f52bc015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fcf8e7ae7ea9006 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.919+0000 7f52aea00700 1 -- 172.21.15.43:0/1444972957 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc01e040 con 0x55eb96085820 2024-05-27T07:19:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.919+0000 7f52aea00700 1 -- 172.21.15.43:0/1444972957 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52bc028070 con 0x55eb96085820 2024-05-27T07:19:22.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.920+0000 7f52aea00700 1 -- 172.21.15.43:0/1444972957 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc023410 con 0x55eb96085820 2024-05-27T07:19:22.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.921+0000 7f52d7626b80 1 -- 172.21.15.43:0/1444972957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 msgr2=0x55eb9609ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.921+0000 7f52d7626b80 1 --2- 172.21.15.43:0/1444972957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9609ba60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009550 tx=0x7f52bc015000 comp rx=0 tx=0).stop 2024-05-27T07:19:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.921+0000 7f52d7626b80 1 -- 172.21.15.43:0/1444972957 shutdown_connections 2024-05-27T07:19:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.921+0000 7f52d7626b80 1 --2- 172.21.15.43:0/1444972957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085820 0x55eb9609ba60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.921+0000 7f52d7626b80 1 --2- 172.21.15.43:0/1444972957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb95f06e40 0x55eb960897c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.921+0000 7f52d7626b80 1 --2- 172.21.15.43:0/1444972957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb96085410 0x55eb9608e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.921+0000 7f52d7626b80 1 -- 172.21.15.43:0/1444972957 >> 172.21.15.43:0/1444972957 conn(0x55eb95f66fc0 msgr2=0x55eb9608f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.921+0000 7f52d7626b80 1 -- 172.21.15.43:0/1444972957 shutdown_connections 2024-05-27T07:19:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.922+0000 7f52d7626b80 1 -- 172.21.15.43:0/1444972957 wait complete. 2024-05-27T07:19:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.923+0000 7f52d7626b80 1 -- start start 2024-05-27T07:19:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.923+0000 7f52d7626b80 1 Processor -- start 2024-05-27T07:19:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.924+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb95f06e40 0x55eb960933f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.924+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085410 0x55eb96093930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.924+0000 7f52d7626b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb96085820 0x55eb9610ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.924+0000 7f52d7626b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41b2900 con 0x55eb95f06e40 2024-05-27T07:19:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.924+0000 7f52d7626b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41b1490 con 0x55eb96085410 2024-05-27T07:19:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.924+0000 7f52d7626b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41b1b60 con 0x55eb96085820 2024-05-27T07:19:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.924+0000 7f52d7626b80 10 client.0 _ll_register_callbacks cb 0x55eb95f4dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.925+0000 7f52c2800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb96085820 0x55eb9610ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.925+0000 7f52c2800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb96085820 0x55eb9610ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:56426/0 (socket says 172.21.15.43:56426) 2024-05-27T07:19:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.925+0000 7f52c2800700 1 -- 172.21.15.43:0/3012951902 learned_addr learned my addr 172.21.15.43:0/3012951902 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.925+0000 7f52c1e00700 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb95f06e40 0x55eb960933f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.926+0000 7f52c1e00700 1 -- 172.21.15.43:0/3012951902 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb96085820 msgr2=0x55eb9610ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.926+0000 7f52c1e00700 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb96085820 0x55eb9610ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.926+0000 7f52c1e00700 1 -- 172.21.15.43:0/3012951902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085410 msgr2=0x55eb96093930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.926+0000 7f52c1e00700 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085410 0x55eb96093930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.926+0000 7f52c1e00700 1 -- 172.21.15.43:0/3012951902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb9610b1a0 con 0x55eb95f06e40 2024-05-27T07:19:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.926+0000 7f52c1e00700 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb95f06e40 0x55eb960933f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f52b40e0d40 tx=0x7f52b4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1426a73d1ac0e4f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.926+0000 7f52a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.927+0000 7f52a3e00700 1 -- 172.21.15.43:0/3012951902 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b40e5e60 con 0x55eb95f06e40 2024-05-27T07:19:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.927+0000 7f52a3e00700 1 -- 172.21.15.43:0/3012951902 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f52b40b1070 con 0x55eb95f06e40 2024-05-27T07:19:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.928+0000 7f52d7626b80 1 -- 172.21.15.43:0/3012951902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52bc028070 con 0x55eb95f06e40 2024-05-27T07:19:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.928+0000 7f52a3e00700 1 -- 172.21.15.43:0/3012951902 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f52b40b1270 con 0x55eb95f06e40 2024-05-27T07:19:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.928+0000 7f52d7626b80 10 client.16314 fetch_fsmap learned FSMap version 78 2024-05-27T07:19:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.928+0000 7f52d7626b80 1 -- 172.21.15.43:0/3012951902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=4+}) -- 0x55eb96112520 con 0x55eb95f06e40 2024-05-27T07:19:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.928+0000 7f52a3e00700 1 -- 172.21.15.43:0/3012951902 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f52bc028070 con 0x55eb95f06e40 2024-05-27T07:19:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.929+0000 7f52a3e00700 1 -- 172.21.15.43:0/3012951902 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 78) ==== 6559+0+0 (secure 0 0 0) 0x7f52b40e0fc0 con 0x55eb95f06e40 2024-05-27T07:19:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.929+0000 7f52d7626b80 10 client.16314 fetch_fsmap finished waiting for FSMap version 78 2024-05-27T07:19:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.929+0000 7f52d7626b80 10 client.16314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:19:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.929+0000 7f52d7626b80 20 client.16314 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.929+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55eb96112b80 0x55eb96114fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.929+0000 7f52d7626b80 4 client.16314 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:19:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.929+0000 7f52d7626b80 1 -- 172.21.15.43:0/3012951902 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb95faf570 con 0x55eb96112b80 2024-05-27T07:19:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.930+0000 7f52c1400700 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55eb96112b80 0x55eb96114fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.931+0000 7f52c1400700 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55eb96112b80 0x55eb96114fa0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=414080635e590f25 server_cookie=648a720ae86205e8 in_seq=0 out_seq=0 2024-05-27T07:19:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.931+0000 7f52a3e00700 10 client.16314 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.933+0000 7f52a3e00700 1 -- 172.21.15.43:0/3012951902 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb95faf570 con 0x55eb96112b80 2024-05-27T07:19:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.933+0000 7f52a3e00700 10 client.16314 handle_command_reply: tid=0 2024-05-27T07:19:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.933+0000 7f52d7626b80 1 client.16314 shutdown 2024-05-27T07:19:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.933+0000 7f52d7626b80 1 -- 172.21.15.43:0/3012951902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb95f06e40 msgr2=0x55eb960933f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.933+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb95f06e40 0x55eb960933f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f52b40e0d40 tx=0x7f52b4009310 comp rx=0 tx=0).stop 2024-05-27T07:19:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.934+0000 7f52d7626b80 1 -- 172.21.15.43:0/3012951902 shutdown_connections 2024-05-27T07:19:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.934+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eb96085820 0x55eb9610ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.934+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55eb96112b80 0x55eb96114fa0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.934+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eb95f06e40 0x55eb960933f0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.934+0000 7f52d7626b80 1 --2- 172.21.15.43:0/3012951902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb96085410 0x55eb96093930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.934+0000 7f52d7626b80 1 -- 172.21.15.43:0/3012951902 >> 172.21.15.43:0/3012951902 conn(0x55eb95f66fc0 msgr2=0x55eb9608fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.934+0000 7f52a3e00700 0 client.16314 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.934+0000 7f52d7626b80 1 -- 172.21.15.43:0/3012951902 shutdown_connections 2024-05-27T07:19:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.934+0000 7f52d7626b80 1 -- 172.21.15.43:0/3012951902 wait complete. 2024-05-27T07:19:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.934+0000 7f52d7626b80 20 client.16314 trim_cache size 0 max 16384 2024-05-27T07:19:22.927 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:19:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.935+0000 7f52c3200700 1 -- 172.21.15.43:0/4201050443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52a80b29f0 msgr2=0x7f52a80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.935+0000 7f52c3200700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52a80b29f0 0x7f52a80b4f00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x55eb96081690 tx=0x7f52bc009040 comp rx=0 tx=0).stop 2024-05-27T07:19:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.935+0000 7f52c3200700 1 -- 172.21.15.43:0/4201050443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52c4105280 msgr2=0x7f52c4102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.935+0000 7f52c3200700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52c4105280 0x7f52c4102f70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f52b4000c00 tx=0x7f52b40057a0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.935+0000 7f52c3200700 1 -- 172.21.15.43:0/4201050443 shutdown_connections 2024-05-27T07:19:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.935+0000 7f52c3200700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52a80b29f0 0x7f52a80b4f00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.935+0000 7f52c3200700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52c410e6f0 0x7f52c41039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.935+0000 7f52c3200700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f52c4105280 0x7f52c4102f70 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.935+0000 7f52c3200700 1 --2- 172.21.15.43:0/4201050443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52c4105bd0 0x7f52c41034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.935+0000 7f52c3200700 1 -- 172.21.15.43:0/4201050443 >> 172.21.15.43:0/4201050443 conn(0x7f52c4078ab0 msgr2=0x7f52c40fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.936+0000 7f52c3200700 1 -- 172.21.15.43:0/4201050443 shutdown_connections 2024-05-27T07:19:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:22.936+0000 7f52c3200700 1 -- 172.21.15.43:0/4201050443 wait complete. 2024-05-27T07:19:23.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25409 0.66 MB/sec execute 87 sec latency 44.540 ms 2024-05-27T07:19:23.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:23 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:23 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:23 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:23 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:23 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:23 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:23 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:23 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:23 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:23 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:23 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:23 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:24.194 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25487 0.65 MB/sec execute 88 sec latency 46.671 ms 2024-05-27T07:19:24.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:24 smithi043 ceph-mon[31531]: pgmap v187: 129 pgs: 129 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 425 KiB/s wr, 36 op/s 2024-05-27T07:19:24.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:24 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:24.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:24 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:24.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:24 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:24.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:24 smithi179 ceph-mon[35888]: pgmap v187: 129 pgs: 129 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 425 KiB/s wr, 36 op/s 2024-05-27T07:19:24.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:24 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:24.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:24 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:24.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:24 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:24.786 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:19:24.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:19:24.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:24 smithi132 ceph-mon[35283]: pgmap v187: 129 pgs: 129 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 425 KiB/s wr, 36 op/s 2024-05-27T07:19:24.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:24 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:19:24.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:24 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:19:24.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:24 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:19:25.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25626 0.65 MB/sec execute 89 sec latency 46.780 ms 2024-05-27T07:19:25.263 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:26.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25758 0.65 MB/sec execute 90 sec latency 61.686 ms 2024-05-27T07:19:26.566 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:26 smithi043 ceph-mon[31531]: pgmap v188: 129 pgs: 129 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 425 KiB/s wr, 36 op/s 2024-05-27T07:19:26.566 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:26 smithi043 ceph-mon[31531]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:19:26.566 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:26 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:19:26.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:26 smithi179 ceph-mon[35888]: pgmap v188: 129 pgs: 129 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 425 KiB/s wr, 36 op/s 2024-05-27T07:19:26.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:26 smithi179 ceph-mon[35888]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:19:26.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:26 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:19:26.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:26 smithi132 ceph-mon[35283]: pgmap v188: 129 pgs: 129 active+clean; 49 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 425 KiB/s wr, 36 op/s 2024-05-27T07:19:26.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:26 smithi132 ceph-mon[35283]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-05-27T07:19:26.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:26 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:19:27.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25874 0.64 MB/sec execute 91 sec latency 51.023 ms 2024-05-27T07:19:27.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.560+0000 7fb38f200700 1 -- 172.21.15.43:0/607879131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb390109670 msgr2=0x7fb390110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.560+0000 7fb38f200700 1 --2- 172.21.15.43:0/607879131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb390109670 0x7fb390110fd0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb38800b0d0 tx=0x7fb38800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.560+0000 7fb38f200700 1 -- 172.21.15.43:0/607879131 shutdown_connections 2024-05-27T07:19:27.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.560+0000 7fb38f200700 1 --2- 172.21.15.43:0/607879131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb390109670 0x7fb390110fd0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.560+0000 7fb38f200700 1 --2- 172.21.15.43:0/607879131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3901043f0 0x7fb390104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.560+0000 7fb38f200700 1 --2- 172.21.15.43:0/607879131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb390108d20 0x7fb390109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.560+0000 7fb38f200700 1 -- 172.21.15.43:0/607879131 >> 172.21.15.43:0/607879131 conn(0x7fb390079d00 msgr2=0x7fb39007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:27.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.561+0000 7fb38f200700 1 -- 172.21.15.43:0/607879131 shutdown_connections 2024-05-27T07:19:27.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.561+0000 7fb38f200700 1 -- 172.21.15.43:0/607879131 wait complete. 2024-05-27T07:19:27.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.562+0000 7fb38f200700 1 Processor -- start 2024-05-27T07:19:27.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.562+0000 7fb38f200700 1 -- start start 2024-05-27T07:19:27.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.563+0000 7fb38f200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3901043f0 0x7fb390075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.563+0000 7fb38f200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb390108d20 0x7fb390075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.563+0000 7fb38f200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb390109670 0x7fb390076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.563+0000 7fb38f200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb390114150 con 0x7fb390108d20 2024-05-27T07:19:27.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.563+0000 7fb38f200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb390113fd0 con 0x7fb390109670 2024-05-27T07:19:27.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.563+0000 7fb38f200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901142d0 con 0x7fb3901043f0 2024-05-27T07:19:27.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.563+0000 7fb38d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb390108d20 0x7fb390075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.563+0000 7fb38d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb390108d20 0x7fb390075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36144/0 (socket says 172.21.15.43:36144) 2024-05-27T07:19:27.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.563+0000 7fb38d400700 1 -- 172.21.15.43:0/3098691145 learned_addr learned my addr 172.21.15.43:0/3098691145 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:27.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.564+0000 7fb38de00700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3901043f0 0x7fb390075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.564+0000 7fb38e800700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb390109670 0x7fb390076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.564+0000 7fb38d400700 1 -- 172.21.15.43:0/3098691145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3901043f0 msgr2=0x7fb390075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.564+0000 7fb38d400700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3901043f0 0x7fb390075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.564+0000 7fb38d400700 1 -- 172.21.15.43:0/3098691145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb390109670 msgr2=0x7fb390076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.564+0000 7fb38d400700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb390109670 0x7fb390076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.564+0000 7fb38d400700 1 -- 172.21.15.43:0/3098691145 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3901a8ee0 con 0x7fb390108d20 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.565+0000 7fb38d400700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb390108d20 0x7fb390075ff0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00c8f0 tx=0x7fb37c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.565+0000 7fb386a00700 1 -- 172.21.15.43:0/3098691145 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c0078d0 con 0x7fb390108d20 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.565+0000 7fb386a00700 1 -- 172.21.15.43:0/3098691145 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb37c00f450 con 0x7fb390108d20 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.565+0000 7fb38f200700 1 -- 172.21.15.43:0/3098691145 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3901a91d0 con 0x7fb390108d20 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.565+0000 7fb386a00700 1 -- 172.21.15.43:0/3098691145 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c018660 con 0x7fb390108d20 2024-05-27T07:19:27.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.565+0000 7fb38f200700 1 -- 172.21.15.43:0/3098691145 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3901a9760 con 0x7fb390108d20 2024-05-27T07:19:27.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.570+0000 7fb3a3aeab80 1 Processor -- start 2024-05-27T07:19:27.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.571+0000 7fb3a3aeab80 1 -- start start 2024-05-27T07:19:27.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.571+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c461f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.571+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c462490 0x56217c46cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.571+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c4628a0 0x56217c46f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.571+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56217c390ec0 con 0x56217c2e3ec0 2024-05-27T07:19:27.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.571+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56217c380a60 con 0x56217c4628a0 2024-05-27T07:19:27.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.571+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56217c384aa0 con 0x56217c462490 2024-05-27T07:19:27.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.572+0000 7fb38e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c461f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.572+0000 7fb386a00700 1 -- 172.21.15.43:0/3098691145 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb37c00f900 con 0x7fb390108d20 2024-05-27T07:19:27.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.572+0000 7fb38e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c461f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36160/0 (socket says 172.21.15.43:36160) 2024-05-27T07:19:27.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.572+0000 7fb38e800700 1 -- 172.21.15.43:0/2477406237 learned_addr learned my addr 172.21.15.43:0/2477406237 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:27.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.572+0000 7fb38de00700 1 --2- 172.21.15.43:0/2477406237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c462490 0x56217c46cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.572+0000 7fb38e800700 1 -- 172.21.15.43:0/2477406237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c462490 msgr2=0x56217c46cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:27.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.572+0000 7fb38e800700 1 --2- 172.21.15.43:0/2477406237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c462490 0x56217c46cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.572+0000 7fb38e800700 1 -- 172.21.15.43:0/2477406237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c4628a0 msgr2=0x56217c46f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:27.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.572+0000 7fb38e800700 1 --2- 172.21.15.43:0/2477406237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c4628a0 0x56217c46f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.572+0000 7fb38e800700 1 -- 172.21.15.43:0/2477406237 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56217c3494a0 con 0x56217c2e3ec0 2024-05-27T07:19:27.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.573+0000 7fb38e800700 1 --2- 172.21.15.43:0/2477406237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c461f50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb388009cb0 tx=0x7fb388009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb4ceb0f00b23ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:27.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.573+0000 7fb37b400700 1 -- 172.21.15.43:0/2477406237 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388009350 con 0x56217c2e3ec0 2024-05-27T07:19:27.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.573+0000 7fb37b400700 1 -- 172.21.15.43:0/2477406237 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb388020070 con 0x56217c2e3ec0 2024-05-27T07:19:27.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.573+0000 7fb37b400700 1 -- 172.21.15.43:0/2477406237 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388005810 con 0x56217c2e3ec0 2024-05-27T07:19:27.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.574+0000 7fb386a00700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3740b2a70 0x7fb3740b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.574+0000 7fb386a00700 1 -- 172.21.15.43:0/3098691145 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fb37c0e2720 con 0x7fb390108d20 2024-05-27T07:19:27.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.574+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/2477406237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 msgr2=0x56217c461f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.574+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/2477406237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c461f50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb388009cb0 tx=0x7fb388009ce0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.574+0000 7fb38de00700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3740b2a70 0x7fb3740b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.575+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/2477406237 shutdown_connections 2024-05-27T07:19:27.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.575+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/2477406237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c4628a0 0x56217c46f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.575+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/2477406237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c461f50 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.575+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/2477406237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c462490 0x56217c46cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.575+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/2477406237 >> 172.21.15.43:0/2477406237 conn(0x56217c344040 msgr2=0x56217c45ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:27.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.575+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/2477406237 shutdown_connections 2024-05-27T07:19:27.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.575+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/2477406237 wait complete. 2024-05-27T07:19:27.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.575+0000 7fb38de00700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3740b2a70 0x7fb3740b4f80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x56217c4634b0 tx=0x7fb38000b410 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:27.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.576+0000 7fb3a3aeab80 1 -- start start 2024-05-27T07:19:27.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.576+0000 7fb3a3aeab80 1 Processor -- start 2024-05-27T07:19:27.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.576+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c46c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c462490 0x56217c46cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c4628a0 0x56217c46d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56217c287020 con 0x56217c2e3ec0 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56217c3217e0 con 0x56217c462490 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56217c46f9b0 con 0x56217c4628a0 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb3a3aeab80 10 client.0 _ll_register_callbacks cb 0x56217c32acf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb38de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c46c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb38de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c46c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36170/0 (socket says 172.21.15.43:36170) 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb38de00700 1 -- 172.21.15.43:0/3542389992 learned_addr learned my addr 172.21.15.43:0/3542389992 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb38d400700 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c462490 0x56217c46cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.577+0000 7fb38e800700 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c4628a0 0x56217c46d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.578+0000 7fb38de00700 1 -- 172.21.15.43:0/3542389992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c4628a0 msgr2=0x56217c46d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:27.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.578+0000 7fb38de00700 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c4628a0 0x56217c46d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.578+0000 7fb38de00700 1 -- 172.21.15.43:0/3542389992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c462490 msgr2=0x56217c46cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:27.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.578+0000 7fb38de00700 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c462490 0x56217c46cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.578+0000 7fb38de00700 1 -- 172.21.15.43:0/3542389992 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56217c2d0950 con 0x56217c2e3ec0 2024-05-27T07:19:27.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.578+0000 7fb38de00700 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c46c680 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb380005d90 tx=0x7fb3800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3084dfb8061eee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:27.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.578+0000 7fb36fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:27.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.578+0000 7fb36fe00700 1 -- 172.21.15.43:0/3542389992 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38001fad0 con 0x56217c2e3ec0 2024-05-27T07:19:27.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.578+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3542389992 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb388020070 con 0x56217c2e3ec0 2024-05-27T07:19:27.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.578+0000 7fb36fe00700 1 -- 172.21.15.43:0/3542389992 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fb380017030 con 0x56217c2e3ec0 2024-05-27T07:19:27.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.579+0000 7fb36fe00700 1 -- 172.21.15.43:0/3542389992 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fb380032050 con 0x56217c2e3ec0 2024-05-27T07:19:27.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.579+0000 7fb3a3aeab80 10 client.16326 fetch_fsmap learned FSMap version 79 2024-05-27T07:19:27.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.579+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3542389992 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x56217c3510e0 con 0x56217c2e3ec0 2024-05-27T07:19:27.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.579+0000 7fb36fe00700 1 -- 172.21.15.43:0/3542389992 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb388020070 con 0x56217c2e3ec0 2024-05-27T07:19:27.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.579+0000 7fb36fe00700 1 -- 172.21.15.43:0/3542389992 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fb3800050b0 con 0x56217c2e3ec0 2024-05-27T07:19:27.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.579+0000 7fb3a3aeab80 10 client.16326 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:19:27.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.579+0000 7fb3a3aeab80 10 client.16326 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:19:27.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.580+0000 7fb3a3aeab80 20 client.16326 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:27.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.580+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56217c47cce0 0x56217c47f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.580+0000 7fb3a3aeab80 4 client.16326 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:19:27.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.580+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3542389992 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56217c392e10 con 0x56217c47cce0 2024-05-27T07:19:27.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.580+0000 7fb38d400700 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56217c47cce0 0x56217c47f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.581+0000 7fb38d400700 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56217c47cce0 0x56217c47f1b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab31e3a003e9690b server_cookie=ad0f4be6da45e47b in_seq=0 out_seq=0 2024-05-27T07:19:27.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.581+0000 7fb36fe00700 10 client.16326 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:27.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb36fe00700 1 -- 172.21.15.43:0/3542389992 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56217c392e10 con 0x56217c47cce0 2024-05-27T07:19:27.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb36fe00700 10 client.16326 handle_command_reply: tid=0 2024-05-27T07:19:27.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb3a3aeab80 1 client.16326 shutdown 2024-05-27T07:19:27.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3542389992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 msgr2=0x56217c46c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c46c680 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb380005d90 tx=0x7fb3800047c0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3542389992 shutdown_connections 2024-05-27T07:19:27.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c4628a0 0x56217c46d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56217c47cce0 0x56217c47f1b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c2e3ec0 0x56217c46c680 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3542389992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c462490 0x56217c46cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3542389992 >> 172.21.15.43:0/3542389992 conn(0x56217c344040 msgr2=0x56217c45d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:27.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.584+0000 7fb36fe00700 0 client.16326 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:27.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.585+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3542389992 shutdown_connections 2024-05-27T07:19:27.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.585+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3542389992 wait complete. 2024-05-27T07:19:27.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.585+0000 7fb3a3aeab80 20 client.16326 trim_cache size 0 max 16384 2024-05-27T07:19:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.610+0000 7fb3a3aeab80 1 Processor -- start 2024-05-27T07:19:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.610+0000 7fb3a3aeab80 1 -- start start 2024-05-27T07:19:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.610+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c2e3ec0 0x56217c466820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.610+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c462490 0x56217c478a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.610+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c4628a0 0x56217c484970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.610+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901b2fc0 con 0x56217c4628a0 2024-05-27T07:19:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.610+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901b3690 con 0x56217c462490 2024-05-27T07:19:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.610+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901b0db0 con 0x56217c2e3ec0 2024-05-27T07:19:27.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.611+0000 7fb38e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c4628a0 0x56217c484970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.611+0000 7fb38e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c4628a0 0x56217c484970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36178/0 (socket says 172.21.15.43:36178) 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.611+0000 7fb38e800700 1 -- 172.21.15.43:0/2578652414 learned_addr learned my addr 172.21.15.43:0/2578652414 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.611+0000 7fb38d400700 1 --2- 172.21.15.43:0/2578652414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c462490 0x56217c478a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.611+0000 7fb38de00700 1 --2- 172.21.15.43:0/2578652414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c2e3ec0 0x56217c466820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.611+0000 7fb38e800700 1 -- 172.21.15.43:0/2578652414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c2e3ec0 msgr2=0x56217c466820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.611+0000 7fb38e800700 1 --2- 172.21.15.43:0/2578652414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c2e3ec0 0x56217c466820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.611+0000 7fb38e800700 1 -- 172.21.15.43:0/2578652414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c462490 msgr2=0x56217c478a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.611+0000 7fb38e800700 1 --2- 172.21.15.43:0/2578652414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c462490 0x56217c478a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.611+0000 7fb38e800700 1 -- 172.21.15.43:0/2578652414 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56217c461ac0 con 0x56217c4628a0 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.612+0000 7fb38e800700 1 --2- 172.21.15.43:0/2578652414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c4628a0 0x56217c484970 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb388005bd0 tx=0x7fb388009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7f70c0898b73c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.612+0000 7fb37aa00700 1 -- 172.21.15.43:0/2578652414 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3880093d0 con 0x56217c4628a0 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.612+0000 7fb37aa00700 1 -- 172.21.15.43:0/2578652414 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb388029070 con 0x56217c4628a0 2024-05-27T07:19:27.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.612+0000 7fb37aa00700 1 -- 172.21.15.43:0/2578652414 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3880075c0 con 0x56217c4628a0 2024-05-27T07:19:27.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.613+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/2578652414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c4628a0 msgr2=0x56217c484970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.613+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/2578652414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c4628a0 0x56217c484970 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb388005bd0 tx=0x7fb388009f20 comp rx=0 tx=0).stop 2024-05-27T07:19:27.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.613+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/2578652414 shutdown_connections 2024-05-27T07:19:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.613+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/2578652414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c4628a0 0x56217c484970 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.613+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/2578652414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c2e3ec0 0x56217c466820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.613+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/2578652414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c462490 0x56217c478a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.613+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/2578652414 >> 172.21.15.43:0/2578652414 conn(0x56217c344040 msgr2=0x56217c466ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.614+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/2578652414 shutdown_connections 2024-05-27T07:19:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.614+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/2578652414 wait complete. 2024-05-27T07:19:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.614+0000 7fb3a3aeab80 1 -- start start 2024-05-27T07:19:27.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.614+0000 7fb3a3aeab80 1 Processor -- start 2024-05-27T07:19:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.615+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c2e3ec0 0x56217c474c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.615+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c462490 0x56217c4751b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.615+0000 7fb3a3aeab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c4628a0 0x56217c471c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.615+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901b1480 con 0x56217c462490 2024-05-27T07:19:27.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.615+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901b28f0 con 0x56217c2e3ec0 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.615+0000 7fb3a3aeab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb3901b1b50 con 0x56217c4628a0 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.615+0000 7fb3a3aeab80 10 client.0 _ll_register_callbacks cb 0x56217c32acf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.615+0000 7fb38d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c462490 0x56217c4751b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.615+0000 7fb38d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c462490 0x56217c4751b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36190/0 (socket says 172.21.15.43:36190) 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.615+0000 7fb38d400700 1 -- 172.21.15.43:0/3335937585 learned_addr learned my addr 172.21.15.43:0/3335937585 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.616+0000 7fb38de00700 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c2e3ec0 0x56217c474c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.616+0000 7fb38e800700 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c4628a0 0x56217c471c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.616+0000 7fb38d400700 1 -- 172.21.15.43:0/3335937585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c4628a0 msgr2=0x56217c471c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.616+0000 7fb38d400700 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c4628a0 0x56217c471c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.616+0000 7fb38d400700 1 -- 172.21.15.43:0/3335937585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c2e3ec0 msgr2=0x56217c474c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.616+0000 7fb38d400700 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c2e3ec0 0x56217c474c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.616+0000 7fb38d400700 1 -- 172.21.15.43:0/3335937585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56217c472180 con 0x56217c462490 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.616+0000 7fb38d400700 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c462490 0x56217c4751b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00c8c0 tx=0x7fb37c00fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=662794dc489f4c9f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.616+0000 7fb36fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.617+0000 7fb36fe00700 1 -- 172.21.15.43:0/3335937585 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c0ada00 con 0x56217c462490 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.617+0000 7fb36fe00700 1 -- 172.21.15.43:0/3335937585 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fb37c0a5c40 con 0x56217c462490 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.617+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3335937585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb388029070 con 0x56217c462490 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.617+0000 7fb36fe00700 1 -- 172.21.15.43:0/3335937585 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fb37c0b24b0 con 0x56217c462490 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.617+0000 7fb3a3aeab80 10 client.16332 fetch_fsmap learned FSMap version 79 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.617+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3335937585 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x56217c4e8050 con 0x56217c462490 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.618+0000 7fb36fe00700 1 -- 172.21.15.43:0/3335937585 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb388029070 con 0x56217c462490 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.618+0000 7fb36fe00700 1 -- 172.21.15.43:0/3335937585 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fb37c0a98b0 con 0x56217c462490 2024-05-27T07:19:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.618+0000 7fb3a3aeab80 10 client.16332 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:19:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.618+0000 7fb3a3aeab80 10 client.16332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:19:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.618+0000 7fb3a3aeab80 20 client.16332 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.618+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56217c4e8710 0x56217c4eab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.618+0000 7fb3a3aeab80 4 client.16332 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:19:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.618+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3335937585 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56217c38c5f0 con 0x56217c4e8710 2024-05-27T07:19:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.618+0000 7fb38de00700 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56217c4e8710 0x56217c4eab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.619+0000 7fb38de00700 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56217c4e8710 0x56217c4eab30 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baefba20358d3bd6 server_cookie=a0ba0cafaa482b49 in_seq=0 out_seq=0 2024-05-27T07:19:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.619+0000 7fb36fe00700 10 client.16332 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb36fe00700 1 -- 172.21.15.43:0/3335937585 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56217c38c5f0 con 0x56217c4e8710 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb36fe00700 10 client.16332 handle_command_reply: tid=0 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb3a3aeab80 1 client.16332 shutdown 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3335937585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c462490 msgr2=0x56217c4751b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c462490 0x56217c4751b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00c8c0 tx=0x7fb37c00fbf0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3335937585 shutdown_connections 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56217c4628a0 0x56217c471c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56217c2e3ec0 0x56217c474c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56217c4e8710 0x56217c4eab30 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb3a3aeab80 1 --2- 172.21.15.43:0/3335937585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56217c462490 0x56217c4751b0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3335937585 >> 172.21.15.43:0/3335937585 conn(0x56217c344040 msgr2=0x56217c466240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb36fe00700 0 client.16332 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.624+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3335937585 shutdown_connections 2024-05-27T07:19:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.625+0000 7fb3a3aeab80 1 -- 172.21.15.43:0/3335937585 wait complete. 2024-05-27T07:19:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.625+0000 7fb3a3aeab80 20 client.16332 trim_cache size 0 max 16384 2024-05-27T07:19:27.617 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:19:27.617 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:19:27.617 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "6df5d3b3-9f43-4a95-a58f-62acf2d6c1c6", 2024-05-27T07:19:27.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:19:27.617 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:19:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 -- 172.21.15.43:0/3098691145 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3740b2a70 msgr2=0x7fb3740b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3740b2a70 0x7fb3740b4f80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x56217c4634b0 tx=0x7fb38000b410 comp rx=0 tx=0).stop 2024-05-27T07:19:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 -- 172.21.15.43:0/3098691145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb390108d20 msgr2=0x7fb390075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb390108d20 0x7fb390075ff0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00c8f0 tx=0x7fb37c00cc00 comp rx=0 tx=0).stop 2024-05-27T07:19:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 -- 172.21.15.43:0/3098691145 shutdown_connections 2024-05-27T07:19:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3740b2a70 0x7fb3740b4f80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb390109670 0x7fb390076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3901043f0 0x7fb390075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 --2- 172.21.15.43:0/3098691145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb390108d20 0x7fb390075ff0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 -- 172.21.15.43:0/3098691145 >> 172.21.15.43:0/3098691145 conn(0x7fb390079d00 msgr2=0x7fb390101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 -- 172.21.15.43:0/3098691145 shutdown_connections 2024-05-27T07:19:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:27.626+0000 7fb38f200700 1 -- 172.21.15.43:0/3098691145 wait complete. 2024-05-27T07:19:28.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25996 0.65 MB/sec execute 92 sec latency 61.419 ms 2024-05-27T07:19:28.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:28 smithi043 ceph-mon[31531]: pgmap v189: 129 pgs: 129 active+clean; 48 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 562 KiB/s wr, 61 op/s 2024-05-27T07:19:28.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:28 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:19:28.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:28 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:28.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:28 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:19:28.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:28 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:28.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:19:28.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:28 smithi179 ceph-mon[35888]: pgmap v189: 129 pgs: 129 active+clean; 48 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 562 KiB/s wr, 61 op/s 2024-05-27T07:19:28.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:28 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:19:28.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:28 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:28.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:28 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:19:28.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:28 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:28 smithi132 ceph-mon[35283]: pgmap v189: 129 pgs: 129 active+clean; 48 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 562 KiB/s wr, 61 op/s 2024-05-27T07:19:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:28 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:19:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:28 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:28 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:19:28.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:28 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:19:29.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 26112 0.64 MB/sec execute 93 sec latency 53.526 ms 2024-05-27T07:19:29.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:19:29.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:19:29.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:19:29.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:19:29.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:19:29.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:19:29.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:29 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:19:29.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:19:29.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:19:29.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:19:29.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:19:29.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:19:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:29 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:19:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:19:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:19:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:19:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:19:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:19:29.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:29.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:29 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:19:30.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 26280 0.66 MB/sec execute 94 sec latency 69.741 ms 2024-05-27T07:19:30.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:30 smithi043 ceph-mon[31531]: pgmap v190: 129 pgs: 129 active+clean; 48 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 374 KiB/s wr, 44 op/s 2024-05-27T07:19:30.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:30 smithi179 ceph-mon[35888]: pgmap v190: 129 pgs: 129 active+clean; 48 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 374 KiB/s wr, 44 op/s 2024-05-27T07:19:30.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:30 smithi132 ceph-mon[35283]: pgmap v190: 129 pgs: 129 active+clean; 48 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 374 KiB/s wr, 44 op/s 2024-05-27T07:19:31.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 26383 0.66 MB/sec execute 95 sec latency 59.544 ms 2024-05-27T07:19:31.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.454+0000 7fe70be00700 1 -- 172.21.15.43:0/3387806211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe70c108d20 msgr2=0x7fe70c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.454+0000 7fe70be00700 1 --2- 172.21.15.43:0/3387806211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe70c108d20 0x7fe70c109130 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40099c0 tx=0x7fe6f4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.454+0000 7fe70be00700 1 -- 172.21.15.43:0/3387806211 shutdown_connections 2024-05-27T07:19:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.454+0000 7fe70be00700 1 --2- 172.21.15.43:0/3387806211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe70c109670 0x7fe70c110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.454+0000 7fe70be00700 1 --2- 172.21.15.43:0/3387806211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe70c1043f0 0x7fe70c104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.454+0000 7fe70be00700 1 --2- 172.21.15.43:0/3387806211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe70c108d20 0x7fe70c109130 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.454+0000 7fe70be00700 1 -- 172.21.15.43:0/3387806211 >> 172.21.15.43:0/3387806211 conn(0x7fe70c079d00 msgr2=0x7fe70c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.454+0000 7fe70be00700 1 -- 172.21.15.43:0/3387806211 shutdown_connections 2024-05-27T07:19:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.454+0000 7fe70be00700 1 -- 172.21.15.43:0/3387806211 wait complete. 2024-05-27T07:19:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.455+0000 7fe70be00700 1 Processor -- start 2024-05-27T07:19:31.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.456+0000 7fe70be00700 1 -- start start 2024-05-27T07:19:31.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.456+0000 7fe70be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe70c1043f0 0x7fe70c075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.456+0000 7fe70be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe70c108d20 0x7fe70c075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.456+0000 7fe70be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe70c109670 0x7fe70c076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.456+0000 7fe70be00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c114180 con 0x7fe70c108d20 2024-05-27T07:19:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.456+0000 7fe70be00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c114000 con 0x7fe70c1043f0 2024-05-27T07:19:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.456+0000 7fe70be00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c114300 con 0x7fe70c109670 2024-05-27T07:19:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.457+0000 7fe702000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe70c108d20 0x7fe70c075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.457+0000 7fe70aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe70c1043f0 0x7fe70c075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.457+0000 7fe702000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe70c108d20 0x7fe70c075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36208/0 (socket says 172.21.15.43:36208) 2024-05-27T07:19:31.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.457+0000 7fe702000700 1 -- 172.21.15.43:0/414223385 learned_addr learned my addr 172.21.15.43:0/414223385 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.457+0000 7fe70b400700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe70c109670 0x7fe70c076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.457+0000 7fe702000700 1 -- 172.21.15.43:0/414223385 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe70c109670 msgr2=0x7fe70c076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.457+0000 7fe702000700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe70c109670 0x7fe70c076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.457+0000 7fe702000700 1 -- 172.21.15.43:0/414223385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe70c1043f0 msgr2=0x7fe70c075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.457+0000 7fe702000700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe70c1043f0 0x7fe70c075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.457+0000 7fe702000700 1 -- 172.21.15.43:0/414223385 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70c1a8fd0 con 0x7fe70c108d20 2024-05-27T07:19:31.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.458+0000 7fe702000700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe70c108d20 0x7fe70c075ff0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400b580 tx=0x7fe6f400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:31.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.458+0000 7fe703e00700 1 -- 172.21.15.43:0/414223385 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4027070 con 0x7fe70c108d20 2024-05-27T07:19:31.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.458+0000 7fe70be00700 1 -- 172.21.15.43:0/414223385 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe70c1a9260 con 0x7fe70c108d20 2024-05-27T07:19:31.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.458+0000 7fe703e00700 1 -- 172.21.15.43:0/414223385 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6f40056d0 con 0x7fe70c108d20 2024-05-27T07:19:31.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.459+0000 7fe70be00700 1 -- 172.21.15.43:0/414223385 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe70c1a96c0 con 0x7fe70c108d20 2024-05-27T07:19:31.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.459+0000 7fe703e00700 1 -- 172.21.15.43:0/414223385 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4022de0 con 0x7fe70c108d20 2024-05-27T07:19:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.463+0000 7fe703e00700 1 -- 172.21.15.43:0/414223385 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe6f402a070 con 0x7fe70c108d20 2024-05-27T07:19:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.464+0000 7fe720619b80 1 Processor -- start 2024-05-27T07:19:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.465+0000 7fe720619b80 1 -- start start 2024-05-27T07:19:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.465+0000 7fe703e00700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe6f00b29f0 0x7fe6f00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.465+0000 7fe720619b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd288ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.465+0000 7fe720619b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d410 0x561cd2897ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.465+0000 7fe720619b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 0x561cd289a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.465+0000 7fe703e00700 1 -- 172.21.15.43:0/414223385 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fe6f40e35c0 con 0x7fe70c108d20 2024-05-27T07:19:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.465+0000 7fe720619b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561cd27bbe40 con 0x561cd270ee40 2024-05-27T07:19:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.465+0000 7fe720619b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561cd27ab9e0 con 0x561cd288d820 2024-05-27T07:19:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.465+0000 7fe720619b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561cd27afa20 con 0x561cd288d410 2024-05-27T07:19:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.465+0000 7fe70b400700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe6f00b29f0 0x7fe6f00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe70aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd288ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe70aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd288ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36214/0 (socket says 172.21.15.43:36214) 2024-05-27T07:19:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe702000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d410 0x561cd2897ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe70aa00700 1 -- 172.21.15.43:0/2171695136 learned_addr learned my addr 172.21.15.43:0/2171695136 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe70b400700 1 --2- 172.21.15.43:0/2171695136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 0x561cd289a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe70aa00700 1 -- 172.21.15.43:0/2171695136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d410 msgr2=0x561cd2897ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe70aa00700 1 --2- 172.21.15.43:0/2171695136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d410 0x561cd2897ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe70aa00700 1 -- 172.21.15.43:0/2171695136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 msgr2=0x561cd289a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe70aa00700 1 --2- 172.21.15.43:0/2171695136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 0x561cd289a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe70aa00700 1 -- 172.21.15.43:0/2171695136 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cd2774420 con 0x561cd270ee40 2024-05-27T07:19:31.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.466+0000 7fe70b400700 1 --2- 172.21.15.43:0/2171695136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 0x561cd289a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:19:31.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.467+0000 7fe70b400700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe6f00b29f0 0x7fe6f00b4f00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe7040059d0 tx=0x7fe7040137f0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:31.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.467+0000 7fe70aa00700 1 --2- 172.21.15.43:0/2171695136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd288ced0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00a730 tx=0x7fe6fc007990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdab392471c00863 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:31.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.467+0000 7fe6fbe00700 1 -- 172.21.15.43:0/2171695136 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc014600 con 0x561cd270ee40 2024-05-27T07:19:31.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.467+0000 7fe6fbe00700 1 -- 172.21.15.43:0/2171695136 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6fc019070 con 0x561cd270ee40 2024-05-27T07:19:31.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.467+0000 7fe6fbe00700 1 -- 172.21.15.43:0/2171695136 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc008960 con 0x561cd270ee40 2024-05-27T07:19:31.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.468+0000 7fe720619b80 1 -- 172.21.15.43:0/2171695136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 msgr2=0x561cd288ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.468+0000 7fe720619b80 1 --2- 172.21.15.43:0/2171695136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd288ced0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00a730 tx=0x7fe6fc007990 comp rx=0 tx=0).stop 2024-05-27T07:19:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.469+0000 7fe720619b80 1 -- 172.21.15.43:0/2171695136 shutdown_connections 2024-05-27T07:19:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.469+0000 7fe720619b80 1 --2- 172.21.15.43:0/2171695136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 0x561cd289a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.469+0000 7fe720619b80 1 --2- 172.21.15.43:0/2171695136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd288ced0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.469+0000 7fe720619b80 1 --2- 172.21.15.43:0/2171695136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d410 0x561cd2897ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.469+0000 7fe720619b80 1 -- 172.21.15.43:0/2171695136 >> 172.21.15.43:0/2171695136 conn(0x561cd276efc0 msgr2=0x561cd2888d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:31.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.469+0000 7fe720619b80 1 -- 172.21.15.43:0/2171695136 shutdown_connections 2024-05-27T07:19:31.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.469+0000 7fe720619b80 1 -- 172.21.15.43:0/2171695136 wait complete. 2024-05-27T07:19:31.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.470+0000 7fe720619b80 1 -- start start 2024-05-27T07:19:31.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.470+0000 7fe720619b80 1 Processor -- start 2024-05-27T07:19:31.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.470+0000 7fe720619b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd2897450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.470+0000 7fe720619b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 0x561cd2897990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.470+0000 7fe720619b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d820 0x561cd2897ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.471+0000 7fe720619b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561cd26b1fa0 con 0x561cd270ee40 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.471+0000 7fe720619b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561cd274c760 con 0x561cd288d410 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.471+0000 7fe720619b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561cd289a980 con 0x561cd288d820 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.471+0000 7fe720619b80 10 client.0 _ll_register_callbacks cb 0x561cd2755c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.471+0000 7fe70aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd2897450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.471+0000 7fe70aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd2897450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36218/0 (socket says 172.21.15.43:36218) 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.471+0000 7fe70aa00700 1 -- 172.21.15.43:0/1411818569 learned_addr learned my addr 172.21.15.43:0/1411818569 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.471+0000 7fe70b400700 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d820 0x561cd2897ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.471+0000 7fe702000700 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 0x561cd2897990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.472+0000 7fe70aa00700 1 -- 172.21.15.43:0/1411818569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d820 msgr2=0x561cd2897ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.472+0000 7fe70aa00700 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d820 0x561cd2897ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.472+0000 7fe70aa00700 1 -- 172.21.15.43:0/1411818569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 msgr2=0x561cd2897990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.472+0000 7fe70aa00700 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 0x561cd2897990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.472+0000 7fe70aa00700 1 -- 172.21.15.43:0/1411818569 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cd26fb8d0 con 0x561cd270ee40 2024-05-27T07:19:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.472+0000 7fe70aa00700 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd2897450 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc007f70 tx=0x7fe6fc0098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f976e943f7481e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.472+0000 7fe6f8c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.472+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1411818569 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc009d40 con 0x561cd270ee40 2024-05-27T07:19:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.472+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1411818569 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fe6fc003a00 con 0x561cd270ee40 2024-05-27T07:19:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.472+0000 7fe720619b80 1 -- 172.21.15.43:0/1411818569 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6fc019070 con 0x561cd270ee40 2024-05-27T07:19:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.473+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1411818569 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fe6fc031050 con 0x561cd270ee40 2024-05-27T07:19:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.473+0000 7fe720619b80 10 client.16344 fetch_fsmap learned FSMap version 79 2024-05-27T07:19:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.473+0000 7fe720619b80 1 -- 172.21.15.43:0/1411818569 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x561cd277c060 con 0x561cd270ee40 2024-05-27T07:19:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.473+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1411818569 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6fc019070 con 0x561cd270ee40 2024-05-27T07:19:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.473+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1411818569 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fe6fc004520 con 0x561cd270ee40 2024-05-27T07:19:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.473+0000 7fe720619b80 10 client.16344 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:19:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.473+0000 7fe720619b80 10 client.16344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:19:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.473+0000 7fe720619b80 20 client.16344 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.474+0000 7fe720619b80 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561cd28a79c0 0x561cd28a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.474+0000 7fe720619b80 4 client.16344 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:19:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.474+0000 7fe720619b80 1 -- 172.21.15.43:0/1411818569 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cd27bdd90 con 0x561cd28a79c0 2024-05-27T07:19:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.474+0000 7fe702000700 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561cd28a79c0 0x561cd28a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.475+0000 7fe702000700 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561cd28a79c0 0x561cd28a9e60 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6607ec1f8c5a886 server_cookie=9c000ab29d7f1c4e in_seq=0 out_seq=0 2024-05-27T07:19:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.475+0000 7fe6f8c00700 10 client.16344 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.479+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1411818569 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561cd27bdd90 con 0x561cd28a79c0 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.479+0000 7fe6f8c00700 10 client.16344 handle_command_reply: tid=0 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.479+0000 7fe720619b80 1 client.16344 shutdown 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.479+0000 7fe720619b80 1 -- 172.21.15.43:0/1411818569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 msgr2=0x561cd2897450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.479+0000 7fe720619b80 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd2897450 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc007f70 tx=0x7fe6fc0098a0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.480+0000 7fe720619b80 1 -- 172.21.15.43:0/1411818569 shutdown_connections 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.480+0000 7fe720619b80 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d820 0x561cd2897ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.480+0000 7fe720619b80 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561cd28a79c0 0x561cd28a9e60 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.480+0000 7fe720619b80 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd2897450 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.480+0000 7fe720619b80 1 --2- 172.21.15.43:0/1411818569 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 0x561cd2897990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.480+0000 7fe6f8c00700 0 client.16344 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.480+0000 7fe720619b80 1 -- 172.21.15.43:0/1411818569 >> 172.21.15.43:0/1411818569 conn(0x561cd276efc0 msgr2=0x561cd2887ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.480+0000 7fe720619b80 1 -- 172.21.15.43:0/1411818569 shutdown_connections 2024-05-27T07:19:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.480+0000 7fe720619b80 1 -- 172.21.15.43:0/1411818569 wait complete. 2024-05-27T07:19:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.480+0000 7fe720619b80 20 client.16344 trim_cache size 0 max 16384 2024-05-27T07:19:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.504+0000 7fe720619b80 1 Processor -- start 2024-05-27T07:19:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.504+0000 7fe720619b80 1 -- start start 2024-05-27T07:19:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.505+0000 7fe720619b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28998f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.505+0000 7fe720619b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 0x561cd28a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.505+0000 7fe720619b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d820 0x561cd28af620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.505+0000 7fe720619b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1b30e0 con 0x561cd270ee40 2024-05-27T07:19:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.505+0000 7fe720619b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1b37b0 con 0x561cd288d410 2024-05-27T07:19:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.505+0000 7fe720619b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1b0ed0 con 0x561cd288d820 2024-05-27T07:19:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.505+0000 7fe70aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.505+0000 7fe70aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36226/0 (socket says 172.21.15.43:36226) 2024-05-27T07:19:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.505+0000 7fe70aa00700 1 -- 172.21.15.43:0/3003292837 learned_addr learned my addr 172.21.15.43:0/3003292837 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.506+0000 7fe702000700 1 --2- 172.21.15.43:0/3003292837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 0x561cd28a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.506+0000 7fe70b400700 1 --2- 172.21.15.43:0/3003292837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d820 0x561cd28af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.506+0000 7fe70aa00700 1 -- 172.21.15.43:0/3003292837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d820 msgr2=0x561cd28af620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.506+0000 7fe70aa00700 1 --2- 172.21.15.43:0/3003292837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d820 0x561cd28af620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.506+0000 7fe70aa00700 1 -- 172.21.15.43:0/3003292837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 msgr2=0x561cd28a3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.506+0000 7fe70aa00700 1 --2- 172.21.15.43:0/3003292837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 0x561cd28a3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.506+0000 7fe70aa00700 1 -- 172.21.15.43:0/3003292837 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cd288a350 con 0x561cd270ee40 2024-05-27T07:19:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.506+0000 7fe702000700 1 --2- 172.21.15.43:0/3003292837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 0x561cd28a3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:19:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.506+0000 7fe70aa00700 1 --2- 172.21.15.43:0/3003292837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28998f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00ad20 tx=0x7fe6fc004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35a51d7efffa38a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.507+0000 7fe6fb400700 1 -- 172.21.15.43:0/3003292837 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc0285a0 con 0x561cd270ee40 2024-05-27T07:19:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.507+0000 7fe6fb400700 1 -- 172.21.15.43:0/3003292837 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6fc019040 con 0x561cd270ee40 2024-05-27T07:19:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.507+0000 7fe6fb400700 1 -- 172.21.15.43:0/3003292837 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc0099c0 con 0x561cd270ee40 2024-05-27T07:19:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.508+0000 7fe720619b80 1 -- 172.21.15.43:0/3003292837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 msgr2=0x561cd28998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.508+0000 7fe720619b80 1 --2- 172.21.15.43:0/3003292837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28998f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00ad20 tx=0x7fe6fc004a70 comp rx=0 tx=0).stop 2024-05-27T07:19:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.508+0000 7fe720619b80 1 -- 172.21.15.43:0/3003292837 shutdown_connections 2024-05-27T07:19:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.508+0000 7fe720619b80 1 --2- 172.21.15.43:0/3003292837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d820 0x561cd28af620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.508+0000 7fe720619b80 1 --2- 172.21.15.43:0/3003292837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28998f0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.508+0000 7fe720619b80 1 --2- 172.21.15.43:0/3003292837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d410 0x561cd28a3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.508+0000 7fe720619b80 1 -- 172.21.15.43:0/3003292837 >> 172.21.15.43:0/3003292837 conn(0x561cd276efc0 msgr2=0x561cd2897c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.508+0000 7fe720619b80 1 -- 172.21.15.43:0/3003292837 shutdown_connections 2024-05-27T07:19:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.508+0000 7fe720619b80 1 -- 172.21.15.43:0/3003292837 wait complete. 2024-05-27T07:19:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.509+0000 7fe720619b80 1 -- start start 2024-05-27T07:19:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.509+0000 7fe720619b80 1 Processor -- start 2024-05-27T07:19:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.509+0000 7fe720619b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe720619b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d410 0x561cd289b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe720619b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 0x561cd289bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe720619b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1b15a0 con 0x561cd270ee40 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe720619b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1b2a10 con 0x561cd288d820 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe720619b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1b1c70 con 0x561cd288d410 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe720619b80 10 client.0 _ll_register_callbacks cb 0x561cd2755c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe70aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe70aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28a03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36234/0 (socket says 172.21.15.43:36234) 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe70aa00700 1 -- 172.21.15.43:0/1508774413 learned_addr learned my addr 172.21.15.43:0/1508774413 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe702000700 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d410 0x561cd289b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.510+0000 7fe70b400700 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 0x561cd289bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.511+0000 7fe70aa00700 1 -- 172.21.15.43:0/1508774413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d410 msgr2=0x561cd289b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.511+0000 7fe70aa00700 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d410 0x561cd289b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.511+0000 7fe70aa00700 1 -- 172.21.15.43:0/1508774413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 msgr2=0x561cd289bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.511+0000 7fe70aa00700 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 0x561cd289bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.511+0000 7fe70aa00700 1 -- 172.21.15.43:0/1508774413 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cd2889620 con 0x561cd270ee40 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.511+0000 7fe70aa00700 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28a03c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc0284a0 tx=0x7fe6fc029f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa57be8dc3e391f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.511+0000 7fe6f8c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.512+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1508774413 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc005d70 con 0x561cd270ee40 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.512+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1508774413 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fe6fc0099c0 con 0x561cd270ee40 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.512+0000 7fe720619b80 1 -- 172.21.15.43:0/1508774413 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6fc019040 con 0x561cd270ee40 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.512+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1508774413 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fe6fc046050 con 0x561cd270ee40 2024-05-27T07:19:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.512+0000 7fe720619b80 10 client.16350 fetch_fsmap learned FSMap version 79 2024-05-27T07:19:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.512+0000 7fe720619b80 1 -- 172.21.15.43:0/1508774413 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x561cd29122d0 con 0x561cd270ee40 2024-05-27T07:19:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.513+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1508774413 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6fc019040 con 0x561cd270ee40 2024-05-27T07:19:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.513+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1508774413 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fe6fc005680 con 0x561cd270ee40 2024-05-27T07:19:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.513+0000 7fe720619b80 10 client.16350 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:19:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.513+0000 7fe720619b80 10 client.16350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:19:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.513+0000 7fe720619b80 20 client.16350 populate_metadata read hostname 'smithi043' 2024-05-27T07:19:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.513+0000 7fe720619b80 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561cd2912810 0x561cd2914c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:19:31.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.513+0000 7fe720619b80 4 client.16350 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:19:31.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.513+0000 7fe720619b80 1 -- 172.21.15.43:0/1508774413 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561cd27b7570 con 0x561cd2912810 2024-05-27T07:19:31.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.514+0000 7fe702000700 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561cd2912810 0x561cd2914c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:19:31.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.515+0000 7fe702000700 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561cd2912810 0x561cd2914c30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=420d5d3cee943356 server_cookie=5191157151eaa193 in_seq=0 out_seq=0 2024-05-27T07:19:31.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.515+0000 7fe6f8c00700 10 client.16350 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:31.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe6f8c00700 1 -- 172.21.15.43:0/1508774413 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561cd27b7570 con 0x561cd2912810 2024-05-27T07:19:31.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe6f8c00700 10 client.16350 handle_command_reply: tid=0 2024-05-27T07:19:31.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe720619b80 1 client.16350 shutdown 2024-05-27T07:19:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe720619b80 1 -- 172.21.15.43:0/1508774413 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 msgr2=0x561cd28a03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe720619b80 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28a03c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc0284a0 tx=0x7fe6fc029f80 comp rx=0 tx=0).stop 2024-05-27T07:19:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe720619b80 1 -- 172.21.15.43:0/1508774413 shutdown_connections 2024-05-27T07:19:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe720619b80 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561cd288d820 0x561cd289bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe720619b80 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561cd2912810 0x561cd2914c30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe720619b80 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561cd270ee40 0x561cd28a03c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe720619b80 1 --2- 172.21.15.43:0/1508774413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561cd288d410 0x561cd289b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe720619b80 1 -- 172.21.15.43:0/1508774413 >> 172.21.15.43:0/1508774413 conn(0x561cd276efc0 msgr2=0x561cd2897230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.516+0000 7fe6f8c00700 0 client.16350 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:19:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.517+0000 7fe720619b80 1 -- 172.21.15.43:0/1508774413 shutdown_connections 2024-05-27T07:19:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.517+0000 7fe720619b80 1 -- 172.21.15.43:0/1508774413 wait complete. 2024-05-27T07:19:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.517+0000 7fe720619b80 20 client.16350 trim_cache size 0 max 16384 2024-05-27T07:19:31.510 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:19:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:19:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:19:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "6df5d3b3-9f43-4a95-a58f-62acf2d6c1c6": { 2024-05-27T07:19:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:19:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "6df5d3b3-9f43-4a95-a58f-62acf2d6c1c6", 2024-05-27T07:19:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:19:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:19:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:19:31.510 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:19:31.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.518+0000 7fe70be00700 1 -- 172.21.15.43:0/414223385 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe6f00b29f0 msgr2=0x7fe6f00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.518+0000 7fe70be00700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe6f00b29f0 0x7fe6f00b4f00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe7040059d0 tx=0x7fe7040137f0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.518+0000 7fe70be00700 1 -- 172.21.15.43:0/414223385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe70c108d20 msgr2=0x7fe70c075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:19:31.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.518+0000 7fe70be00700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe70c108d20 0x7fe70c075ff0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400b580 tx=0x7fe6f400f690 comp rx=0 tx=0).stop 2024-05-27T07:19:31.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.518+0000 7fe70be00700 1 -- 172.21.15.43:0/414223385 shutdown_connections 2024-05-27T07:19:31.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.518+0000 7fe70be00700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe6f00b29f0 0x7fe6f00b4f00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.518+0000 7fe70be00700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe70c109670 0x7fe70c076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.518+0000 7fe70be00700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe70c1043f0 0x7fe70c075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.518+0000 7fe70be00700 1 --2- 172.21.15.43:0/414223385 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe70c108d20 0x7fe70c075ff0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:19:31.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.518+0000 7fe70be00700 1 -- 172.21.15.43:0/414223385 >> 172.21.15.43:0/414223385 conn(0x7fe70c079d00 msgr2=0x7fe70c101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:19:31.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.519+0000 7fe70be00700 1 -- 172.21.15.43:0/414223385 shutdown_connections 2024-05-27T07:19:31.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:19:31.519+0000 7fe70be00700 1 -- 172.21.15.43:0/414223385 wait complete. 2024-05-27T07:19:32.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 26496 0.65 MB/sec execute 96 sec latency 57.576 ms 2024-05-27T07:19:32.389 INFO:tasks.cephfs.filesystem:scrub status for tag:6df5d3b3-9f43-4a95-a58f-62acf2d6c1c6 - {'path': '/', 'tag': '6df5d3b3-9f43-4a95-a58f-62acf2d6c1c6', 'options': 'recursive,force'} 2024-05-27T07:19:32.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:32 smithi043 ceph-mon[31531]: pgmap v191: 129 pgs: 129 active+clean; 48 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 546 KiB/s wr, 68 op/s 2024-05-27T07:19:32.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:32 smithi179 ceph-mon[35888]: pgmap v191: 129 pgs: 129 active+clean; 48 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 546 KiB/s wr, 68 op/s 2024-05-27T07:19:32.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:32 smithi132 ceph-mon[35283]: pgmap v191: 129 pgs: 129 active+clean; 48 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 546 KiB/s wr, 68 op/s 2024-05-27T07:19:33.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 26592 0.65 MB/sec execute 97 sec latency 67.448 ms 2024-05-27T07:19:34.195 INFO:tasks.workunit.client.0.smithi043.stdout: 1 26735 0.64 MB/sec execute 98 sec latency 64.556 ms 2024-05-27T07:19:34.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:34 smithi043 ceph-mon[31531]: pgmap v192: 129 pgs: 129 active+clean; 48 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 309 KiB/s wr, 48 op/s 2024-05-27T07:19:34.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:34 smithi179 ceph-mon[35888]: pgmap v192: 129 pgs: 129 active+clean; 48 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 309 KiB/s wr, 48 op/s 2024-05-27T07:19:34.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:34 smithi132 ceph-mon[35283]: pgmap v192: 129 pgs: 129 active+clean; 48 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 309 KiB/s wr, 48 op/s 2024-05-27T07:19:35.196 INFO:tasks.workunit.client.0.smithi043.stdout: 1 26855 0.64 MB/sec execute 99 sec latency 68.692 ms 2024-05-27T07:19:36.196 INFO:tasks.workunit.client.0.smithi043.stdout: 1 26967 0.63 MB/sec execute 100 sec latency 55.596 ms 2024-05-27T07:19:36.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:36 smithi043 ceph-mon[31531]: pgmap v193: 129 pgs: 129 active+clean; 48 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 309 KiB/s wr, 48 op/s 2024-05-27T07:19:36.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:36 smithi179 ceph-mon[35888]: pgmap v193: 129 pgs: 129 active+clean; 48 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 309 KiB/s wr, 48 op/s 2024-05-27T07:19:36.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:36 smithi132 ceph-mon[35283]: pgmap v193: 129 pgs: 129 active+clean; 48 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 309 KiB/s wr, 48 op/s 2024-05-27T07:19:37.196 INFO:tasks.workunit.client.0.smithi043.stdout: 1 27098 0.63 MB/sec execute 101 sec latency 68.802 ms 2024-05-27T07:19:37.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:37 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:19:37.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:37 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:19:37.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:37 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:19:37.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:37 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:19:37.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:37 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:19:37.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:37 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:19:38.196 INFO:tasks.workunit.client.0.smithi043.stdout: 1 27234 0.63 MB/sec execute 102 sec latency 59.149 ms 2024-05-27T07:19:38.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:38 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:19:38.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:38 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:19:38.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:38 smithi043 ceph-mon[31531]: pgmap v194: 129 pgs: 129 active+clean; 49 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 366 KiB/s wr, 65 op/s 2024-05-27T07:19:38.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:38 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:19:38.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:38 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:19:38.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:38 smithi179 ceph-mon[35888]: pgmap v194: 129 pgs: 129 active+clean; 49 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 366 KiB/s wr, 65 op/s 2024-05-27T07:19:38.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:38 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:19:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:38 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:19:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:38 smithi132 ceph-mon[35283]: pgmap v194: 129 pgs: 129 active+clean; 49 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 366 KiB/s wr, 65 op/s 2024-05-27T07:19:39.196 INFO:tasks.workunit.client.0.smithi043.stdout: 1 27305 0.63 MB/sec execute 103 sec latency 56.380 ms 2024-05-27T07:19:39.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:39 smithi132 ceph-mon[35283]: pgmap v195: 129 pgs: 129 active+clean; 49 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 229 KiB/s wr, 40 op/s 2024-05-27T07:19:39.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:39 smithi043 ceph-mon[31531]: pgmap v195: 129 pgs: 129 active+clean; 49 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 229 KiB/s wr, 40 op/s 2024-05-27T07:19:39.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:39 smithi179 ceph-mon[35888]: pgmap v195: 129 pgs: 129 active+clean; 49 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 229 KiB/s wr, 40 op/s 2024-05-27T07:19:40.196 INFO:tasks.workunit.client.0.smithi043.stdout: 1 27380 0.63 MB/sec execute 104 sec latency 57.834 ms 2024-05-27T07:19:41.196 INFO:tasks.workunit.client.0.smithi043.stdout: 1 27489 0.62 MB/sec execute 105 sec latency 58.080 ms 2024-05-27T07:19:41.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:41 smithi043 ceph-mon[31531]: pgmap v196: 129 pgs: 129 active+clean; 50 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 422 KiB/s wr, 57 op/s 2024-05-27T07:19:41.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:41 smithi179 ceph-mon[35888]: pgmap v196: 129 pgs: 129 active+clean; 50 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 422 KiB/s wr, 57 op/s 2024-05-27T07:19:42.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:41 smithi132 ceph-mon[35283]: pgmap v196: 129 pgs: 129 active+clean; 50 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 422 KiB/s wr, 57 op/s 2024-05-27T07:19:42.196 INFO:tasks.workunit.client.0.smithi043.stdout: 1 27750 0.65 MB/sec execute 106 sec latency 69.443 ms 2024-05-27T07:19:43.196 INFO:tasks.workunit.client.0.smithi043.stdout: 1 27861 0.65 MB/sec execute 107 sec latency 50.540 ms 2024-05-27T07:19:43.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:43 smithi043 ceph-mon[31531]: pgmap v197: 129 pgs: 129 active+clean; 50 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 250 KiB/s wr, 33 op/s 2024-05-27T07:19:43.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:43 smithi179 ceph-mon[35888]: pgmap v197: 129 pgs: 129 active+clean; 50 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 250 KiB/s wr, 33 op/s 2024-05-27T07:19:44.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:43 smithi132 ceph-mon[35283]: pgmap v197: 129 pgs: 129 active+clean; 50 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 250 KiB/s wr, 33 op/s 2024-05-27T07:19:44.197 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28081 0.65 MB/sec execute 108 sec latency 56.153 ms 2024-05-27T07:19:45.197 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28168 0.66 MB/sec execute 109 sec latency 59.233 ms 2024-05-27T07:19:45.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:45 smithi043 ceph-mon[31531]: pgmap v198: 129 pgs: 129 active+clean; 50 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 250 KiB/s wr, 33 op/s 2024-05-27T07:19:45.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:45 smithi179 ceph-mon[35888]: pgmap v198: 129 pgs: 129 active+clean; 50 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 250 KiB/s wr, 33 op/s 2024-05-27T07:19:46.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:45 smithi132 ceph-mon[35283]: pgmap v198: 129 pgs: 129 active+clean; 50 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 250 KiB/s wr, 33 op/s 2024-05-27T07:19:46.197 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28194 0.65 MB/sec execute 110 sec latency 67.019 ms 2024-05-27T07:19:47.197 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28244 0.65 MB/sec execute 111 sec latency 52.079 ms 2024-05-27T07:19:47.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:47 smithi043 ceph-mon[31531]: pgmap v199: 129 pgs: 129 active+clean; 52 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 443 KiB/s wr, 50 op/s 2024-05-27T07:19:47.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:47 smithi179 ceph-mon[35888]: pgmap v199: 129 pgs: 129 active+clean; 52 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 443 KiB/s wr, 50 op/s 2024-05-27T07:19:48.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:47 smithi132 ceph-mon[35283]: pgmap v199: 129 pgs: 129 active+clean; 52 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 443 KiB/s wr, 50 op/s 2024-05-27T07:19:48.197 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28423 0.67 MB/sec execute 112 sec latency 52.664 ms 2024-05-27T07:19:49.197 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28534 0.67 MB/sec execute 113 sec latency 54.908 ms 2024-05-27T07:19:49.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:49 smithi043 ceph-mon[31531]: pgmap v200: 129 pgs: 129 active+clean; 52 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 386 KiB/s wr, 33 op/s 2024-05-27T07:19:49.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:49 smithi179 ceph-mon[35888]: pgmap v200: 129 pgs: 129 active+clean; 52 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 386 KiB/s wr, 33 op/s 2024-05-27T07:19:50.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:49 smithi132 ceph-mon[35283]: pgmap v200: 129 pgs: 129 active+clean; 52 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 386 KiB/s wr, 33 op/s 2024-05-27T07:19:50.197 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28640 0.67 MB/sec execute 114 sec latency 56.372 ms 2024-05-27T07:19:51.198 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28691 0.66 MB/sec execute 115 sec latency 49.074 ms 2024-05-27T07:19:51.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:51 smithi043 ceph-mon[31531]: pgmap v201: 129 pgs: 129 active+clean; 53 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 606 KiB/s wr, 49 op/s 2024-05-27T07:19:51.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:51 smithi179 ceph-mon[35888]: pgmap v201: 129 pgs: 129 active+clean; 53 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 606 KiB/s wr, 49 op/s 2024-05-27T07:19:52.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:51 smithi132 ceph-mon[35283]: pgmap v201: 129 pgs: 129 active+clean; 53 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 606 KiB/s wr, 49 op/s 2024-05-27T07:19:52.198 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28743 0.66 MB/sec execute 116 sec latency 54.959 ms 2024-05-27T07:19:53.198 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28807 0.65 MB/sec execute 117 sec latency 56.233 ms 2024-05-27T07:19:53.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:53 smithi179 ceph-mon[35888]: pgmap v202: 129 pgs: 129 active+clean; 53 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 412 KiB/s wr, 33 op/s 2024-05-27T07:19:54.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:53 smithi132 ceph-mon[35283]: pgmap v202: 129 pgs: 129 active+clean; 53 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 412 KiB/s wr, 33 op/s 2024-05-27T07:19:54.166 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:53 smithi043 ceph-mon[31531]: pgmap v202: 129 pgs: 129 active+clean; 53 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 412 KiB/s wr, 33 op/s 2024-05-27T07:19:54.198 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28864 0.65 MB/sec execute 118 sec latency 49.665 ms 2024-05-27T07:19:55.198 INFO:tasks.workunit.client.0.smithi043.stdout: 1 28929 0.64 MB/sec execute 119 sec latency 51.710 ms 2024-05-27T07:19:55.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:55 smithi043 ceph-mon[31531]: pgmap v203: 129 pgs: 129 active+clean; 53 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 413 KiB/s wr, 33 op/s 2024-05-27T07:19:56.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:55 smithi132 ceph-mon[35283]: pgmap v203: 129 pgs: 129 active+clean; 53 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 413 KiB/s wr, 33 op/s 2024-05-27T07:19:56.198 INFO:tasks.workunit.client.0.smithi043.stdout: 1 29007 0.64 MB/sec execute 120 sec latency 49.629 ms 2024-05-27T07:19:56.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:55 smithi179 ceph-mon[35888]: pgmap v203: 129 pgs: 129 active+clean; 53 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 413 KiB/s wr, 33 op/s 2024-05-27T07:19:57.198 INFO:tasks.workunit.client.0.smithi043.stdout: 1 29090 0.63 MB/sec execute 121 sec latency 64.540 ms 2024-05-27T07:19:58.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:57 smithi132 ceph-mon[35283]: pgmap v204: 129 pgs: 129 active+clean; 54 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 510 KiB/s wr, 52 op/s 2024-05-27T07:19:58.198 INFO:tasks.workunit.client.0.smithi043.stdout: 1 29214 0.63 MB/sec execute 122 sec latency 62.980 ms 2024-05-27T07:19:58.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:57 smithi043 ceph-mon[31531]: pgmap v204: 129 pgs: 129 active+clean; 54 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 510 KiB/s wr, 52 op/s 2024-05-27T07:19:58.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:57 smithi179 ceph-mon[35888]: pgmap v204: 129 pgs: 129 active+clean; 54 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 510 KiB/s wr, 52 op/s 2024-05-27T07:19:59.198 INFO:tasks.workunit.client.0.smithi043.stdout: 1 29329 0.63 MB/sec execute 123 sec latency 58.622 ms 2024-05-27T07:20:00.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:19:59 smithi132 ceph-mon[35283]: pgmap v205: 129 pgs: 129 active+clean; 54 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 317 KiB/s wr, 34 op/s 2024-05-27T07:20:00.198 INFO:tasks.workunit.client.0.smithi043.stdout: 1 29424 0.62 MB/sec execute 124 sec latency 51.602 ms 2024-05-27T07:20:00.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:19:59 smithi043 ceph-mon[31531]: pgmap v205: 129 pgs: 129 active+clean; 54 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 317 KiB/s wr, 34 op/s 2024-05-27T07:20:00.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:19:59 smithi179 ceph-mon[35888]: pgmap v205: 129 pgs: 129 active+clean; 54 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 317 KiB/s wr, 34 op/s 2024-05-27T07:20:01.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:00 smithi132 ceph-mon[35283]: overall HEALTH_OK 2024-05-27T07:20:01.199 INFO:tasks.workunit.client.0.smithi043.stdout: 1 29538 0.63 MB/sec execute 125 sec latency 56.685 ms 2024-05-27T07:20:01.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:00 smithi043 ceph-mon[31531]: overall HEALTH_OK 2024-05-27T07:20:01.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:00 smithi179 ceph-mon[35888]: overall HEALTH_OK 2024-05-27T07:20:02.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:01 smithi132 ceph-mon[35283]: pgmap v206: 129 pgs: 129 active+clean; 52 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 452 KiB/s wr, 74 op/s 2024-05-27T07:20:02.199 INFO:tasks.workunit.client.0.smithi043.stdout: 1 29637 0.62 MB/sec execute 126 sec latency 65.301 ms 2024-05-27T07:20:02.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:01 smithi043 ceph-mon[31531]: pgmap v206: 129 pgs: 129 active+clean; 52 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 452 KiB/s wr, 74 op/s 2024-05-27T07:20:02.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:01 smithi179 ceph-mon[35888]: pgmap v206: 129 pgs: 129 active+clean; 52 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 452 KiB/s wr, 74 op/s 2024-05-27T07:20:02.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:20:02.897 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:03.199 INFO:tasks.workunit.client.0.smithi043.stdout: 1 29820 0.64 MB/sec execute 127 sec latency 54.158 ms 2024-05-27T07:20:04.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:03 smithi132 ceph-mon[35283]: pgmap v207: 129 pgs: 129 active+clean; 52 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 232 KiB/s wr, 58 op/s 2024-05-27T07:20:04.105 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:03 smithi043 ceph-mon[31531]: pgmap v207: 129 pgs: 129 active+clean; 52 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 232 KiB/s wr, 58 op/s 2024-05-27T07:20:04.199 INFO:tasks.workunit.client.0.smithi043.stdout: 1 29911 0.64 MB/sec execute 128 sec latency 69.588 ms 2024-05-27T07:20:04.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:03 smithi179 ceph-mon[35888]: pgmap v207: 129 pgs: 129 active+clean; 52 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 232 KiB/s wr, 58 op/s 2024-05-27T07:20:05.199 INFO:tasks.workunit.client.0.smithi043.stdout: 1 30026 0.63 MB/sec execute 129 sec latency 58.197 ms 2024-05-27T07:20:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.445+0000 7f3b79a00700 1 -- 172.21.15.43:0/2740251839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b74105470 msgr2=0x7f3b74105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.445+0000 7f3b79a00700 1 --2- 172.21.15.43:0/2740251839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b74105470 0x7f3b74105880 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f3b64009990 tx=0x7f3b64009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.445+0000 7f3b79a00700 1 -- 172.21.15.43:0/2740251839 shutdown_connections 2024-05-27T07:20:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.445+0000 7f3b79a00700 1 --2- 172.21.15.43:0/2740251839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b7410e930 0x7f3b74110e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.445+0000 7f3b79a00700 1 --2- 172.21.15.43:0/2740251839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b74105470 0x7f3b74105880 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.445+0000 7f3b79a00700 1 --2- 172.21.15.43:0/2740251839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b74105e50 0x7f3b7410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.445+0000 7f3b79a00700 1 -- 172.21.15.43:0/2740251839 >> 172.21.15.43:0/2740251839 conn(0x7f3b740fef40 msgr2=0x7f3b74101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.445+0000 7f3b79a00700 1 -- 172.21.15.43:0/2740251839 shutdown_connections 2024-05-27T07:20:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.445+0000 7f3b79a00700 1 -- 172.21.15.43:0/2740251839 wait complete. 2024-05-27T07:20:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.446+0000 7f3b79a00700 1 Processor -- start 2024-05-27T07:20:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.447+0000 7f3b79a00700 1 -- start start 2024-05-27T07:20:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.447+0000 7f3b79a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b74105470 0x7f3b74103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.447+0000 7f3b79a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b74105e50 0x7f3b741036a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.447+0000 7f3b79a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b7410e930 0x7f3b74103be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.447+0000 7f3b79a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3b74114390 con 0x7f3b7410e930 2024-05-27T07:20:05.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.447+0000 7f3b79a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3b74114210 con 0x7f3b74105470 2024-05-27T07:20:05.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.447+0000 7f3b79a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3b74114510 con 0x7f3b74105e50 2024-05-27T07:20:05.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.448+0000 7f3b73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b7410e930 0x7f3b74103be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.448+0000 7f3b73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b7410e930 0x7f3b74103be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52978/0 (socket says 172.21.15.43:52978) 2024-05-27T07:20:05.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.448+0000 7f3b73400700 1 -- 172.21.15.43:0/936971501 learned_addr learned my addr 172.21.15.43:0/936971501 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:05.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.448+0000 7f3b72a00700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b74105470 0x7f3b74103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.448+0000 7f3b72000700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b74105e50 0x7f3b741036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.448+0000 7f3b73400700 1 -- 172.21.15.43:0/936971501 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b74105e50 msgr2=0x7f3b741036a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:05.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.448+0000 7f3b73400700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b74105e50 0x7f3b741036a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.448+0000 7f3b73400700 1 -- 172.21.15.43:0/936971501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b74105470 msgr2=0x7f3b74103160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:05.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.448+0000 7f3b73400700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b74105470 0x7f3b74103160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.449+0000 7f3b73400700 1 -- 172.21.15.43:0/936971501 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b741a9160 con 0x7f3b7410e930 2024-05-27T07:20:05.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.449+0000 7f3b72a00700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b74105470 0x7f3b74103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:05.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.449+0000 7f3b73400700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b7410e930 0x7f3b74103be0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00ada0 tx=0x7f3b6c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:05.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.449+0000 7f3b6b400700 1 -- 172.21.15.43:0/936971501 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c017020 con 0x7f3b7410e930 2024-05-27T07:20:05.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.449+0000 7f3b6b400700 1 -- 172.21.15.43:0/936971501 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b6c00ce10 con 0x7f3b7410e930 2024-05-27T07:20:05.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.449+0000 7f3b79a00700 1 -- 172.21.15.43:0/936971501 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b741a9450 con 0x7f3b7410e930 2024-05-27T07:20:05.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.449+0000 7f3b6b400700 1 -- 172.21.15.43:0/936971501 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c005280 con 0x7f3b7410e930 2024-05-27T07:20:05.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.450+0000 7f3b79a00700 1 -- 172.21.15.43:0/936971501 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b741a9940 con 0x7f3b7410e930 2024-05-27T07:20:05.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.454+0000 7f3b6b400700 1 -- 172.21.15.43:0/936971501 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3b6c00c980 con 0x7f3b7410e930 2024-05-27T07:20:05.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.455+0000 7f3b885b8b80 1 Processor -- start 2024-05-27T07:20:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.455+0000 7f3b885b8b80 1 -- start start 2024-05-27T07:20:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.455+0000 7f3b6b400700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3b580b29f0 0x7f3b580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.456+0000 7f3b6b400700 1 -- 172.21.15.43:0/936971501 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f3b6c0e20b0 con 0x7f3b7410e930 2024-05-27T07:20:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.456+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab86330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.456+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab86870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.456+0000 7f3b72000700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3b580b29f0 0x7f3b580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.456+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab93110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.456+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56333aab4e40 con 0x56333ab86e70 2024-05-27T07:20:05.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.456+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56333aaa49e0 con 0x56333aa07e40 2024-05-27T07:20:05.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.456+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56333aaa8a20 con 0x56333aaf39f0 2024-05-27T07:20:05.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.456+0000 7f3b73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab93110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.457+0000 7f3b73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab93110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52990/0 (socket says 172.21.15.43:52990) 2024-05-27T07:20:05.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.457+0000 7f3b73400700 1 -- 172.21.15.43:0/214088548 learned_addr learned my addr 172.21.15.43:0/214088548 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:05.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.457+0000 7f3b72a00700 1 --2- 172.21.15.43:0/214088548 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab86330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.457+0000 7f3b72000700 1 --2- 172.21.15.43:0/214088548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab86870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.457+0000 7f3b73400700 1 -- 172.21.15.43:0/214088548 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 msgr2=0x56333ab86330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:05.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.457+0000 7f3b73400700 1 --2- 172.21.15.43:0/214088548 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab86330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.457+0000 7f3b73400700 1 -- 172.21.15.43:0/214088548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 msgr2=0x56333ab86870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:05.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.457+0000 7f3b73400700 1 --2- 172.21.15.43:0/214088548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab86870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.457+0000 7f3b73400700 1 -- 172.21.15.43:0/214088548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56333aa6d420 con 0x56333ab86e70 2024-05-27T07:20:05.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.457+0000 7f3b72000700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3b580b29f0 0x7f3b580b4f00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3b741025e0 tx=0x7f3b600106b0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:05.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.458+0000 7f3b73400700 1 --2- 172.21.15.43:0/214088548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab93110 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0a50f0 tx=0x7f3b6c00cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17920daa8d368c9d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:05.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.458+0000 7f3b5fe00700 1 -- 172.21.15.43:0/214088548 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c0a8b80 con 0x56333ab86e70 2024-05-27T07:20:05.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.458+0000 7f3b5fe00700 1 -- 172.21.15.43:0/214088548 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b6c0a9770 con 0x56333ab86e70 2024-05-27T07:20:05.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.458+0000 7f3b5fe00700 1 -- 172.21.15.43:0/214088548 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c0a92d0 con 0x56333ab86e70 2024-05-27T07:20:05.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.459+0000 7f3b885b8b80 1 -- 172.21.15.43:0/214088548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 msgr2=0x56333ab93110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:05.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.459+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/214088548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab93110 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0a50f0 tx=0x7f3b6c00cd40 comp rx=0 tx=0).stop 2024-05-27T07:20:05.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.459+0000 7f3b885b8b80 1 -- 172.21.15.43:0/214088548 shutdown_connections 2024-05-27T07:20:05.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.459+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/214088548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab93110 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.459+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/214088548 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab86330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.459+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/214088548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab86870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.459+0000 7f3b885b8b80 1 -- 172.21.15.43:0/214088548 >> 172.21.15.43:0/214088548 conn(0x56333aaa1390 msgr2=0x56333ab824c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:05.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.459+0000 7f3b885b8b80 1 -- 172.21.15.43:0/214088548 shutdown_connections 2024-05-27T07:20:05.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.459+0000 7f3b885b8b80 1 -- 172.21.15.43:0/214088548 wait complete. 2024-05-27T07:20:05.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.460+0000 7f3b885b8b80 1 -- start start 2024-05-27T07:20:05.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.460+0000 7f3b885b8b80 1 Processor -- start 2024-05-27T07:20:05.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.461+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab906e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.461+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aaf39f0 0x56333ab90c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.461+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333ab86e70 0x56333ab91160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.461+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56333a9aafa0 con 0x56333aaf39f0 2024-05-27T07:20:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.461+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56333aa45760 con 0x56333aa07e40 2024-05-27T07:20:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.461+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56333ab93b80 con 0x56333ab86e70 2024-05-27T07:20:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.461+0000 7f3b885b8b80 10 client.0 _ll_register_callbacks cb 0x56333aa4ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.461+0000 7f3b72000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aaf39f0 0x56333ab90c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.461+0000 7f3b72000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aaf39f0 0x56333ab90c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52992/0 (socket says 172.21.15.43:52992) 2024-05-27T07:20:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.461+0000 7f3b72000700 1 -- 172.21.15.43:0/1147189027 learned_addr learned my addr 172.21.15.43:0/1147189027 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.462+0000 7f3b73400700 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333ab86e70 0x56333ab91160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.462+0000 7f3b72a00700 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab906e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.462+0000 7f3b72000700 1 -- 172.21.15.43:0/1147189027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333ab86e70 msgr2=0x56333ab91160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.462+0000 7f3b72000700 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333ab86e70 0x56333ab91160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.462+0000 7f3b72000700 1 -- 172.21.15.43:0/1147189027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 msgr2=0x56333ab906e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.462+0000 7f3b72000700 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab906e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.462+0000 7f3b72000700 1 -- 172.21.15.43:0/1147189027 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56333a9f48d0 con 0x56333aaf39f0 2024-05-27T07:20:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.462+0000 7f3b73400700 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333ab86e70 0x56333ab91160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.462+0000 7f3b72000700 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aaf39f0 0x56333ab90c20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3b600136e0 tx=0x7f3b600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa4b49f9c0d976d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.462+0000 7f3b5cc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.463+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1147189027 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b60010ea0 con 0x56333aaf39f0 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.463+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1147189027 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f3b60028670 con 0x56333aaf39f0 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.463+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1147189027 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b6c0a9770 con 0x56333aaf39f0 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.463+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1147189027 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f3b60028870 con 0x56333aaf39f0 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.463+0000 7f3b885b8b80 10 client.16362 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.463+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1147189027 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x56333aa75060 con 0x56333aaf39f0 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.464+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1147189027 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b6c0a9770 con 0x56333aaf39f0 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.464+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1147189027 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f3b600119e0 con 0x56333aaf39f0 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.464+0000 7f3b885b8b80 10 client.16362 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.464+0000 7f3b885b8b80 10 client.16362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.464+0000 7f3b885b8b80 20 client.16362 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.464+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56333aba0eb0 0x56333aba3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.464+0000 7f3b885b8b80 4 client.16362 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:20:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.464+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1147189027 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56333aab6d90 con 0x56333aba0eb0 2024-05-27T07:20:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.465+0000 7f3b72a00700 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56333aba0eb0 0x56333aba3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.466+0000 7f3b72a00700 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56333aba0eb0 0x56333aba3300 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abad41314873a873 server_cookie=8197248c4d6684f0 in_seq=0 out_seq=0 2024-05-27T07:20:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.466+0000 7f3b5cc00700 10 client.16362 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:05.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1147189027 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56333aab6d90 con 0x56333aba0eb0 2024-05-27T07:20:05.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b5cc00700 10 client.16362 handle_command_reply: tid=0 2024-05-27T07:20:05.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b885b8b80 1 client.16362 shutdown 2024-05-27T07:20:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1147189027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aaf39f0 msgr2=0x56333ab90c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aaf39f0 0x56333ab90c20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3b600136e0 tx=0x7f3b600047c0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1147189027 shutdown_connections 2024-05-27T07:20:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333ab86e70 0x56333ab91160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab906e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56333aba0eb0 0x56333aba3300 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1147189027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aaf39f0 0x56333ab90c20 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1147189027 >> 172.21.15.43:0/1147189027 conn(0x56333aaa1390 msgr2=0x56333ab86370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.469+0000 7f3b5cc00700 0 client.16362 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.470+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1147189027 shutdown_connections 2024-05-27T07:20:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.470+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1147189027 wait complete. 2024-05-27T07:20:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.470+0000 7f3b885b8b80 20 client.16362 trim_cache size 0 max 16384 2024-05-27T07:20:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.496+0000 7f3b885b8b80 1 Processor -- start 2024-05-27T07:20:05.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.496+0000 7f3b885b8b80 1 -- start start 2024-05-27T07:20:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.496+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aa07e40 0x56333ab8bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.496+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab86790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.496+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333ab86e70 0x56333ab9cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.496+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3b741b3360 con 0x56333aa07e40 2024-05-27T07:20:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.496+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3b741b3a30 con 0x56333ab86e70 2024-05-27T07:20:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.496+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3b741b1150 con 0x56333aaf39f0 2024-05-27T07:20:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.497+0000 7f3b72a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aa07e40 0x56333ab8bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.497+0000 7f3b72a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aa07e40 0x56333ab8bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52998/0 (socket says 172.21.15.43:52998) 2024-05-27T07:20:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.497+0000 7f3b72a00700 1 -- 172.21.15.43:0/2261908309 learned_addr learned my addr 172.21.15.43:0/2261908309 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:05.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.497+0000 7f3b72000700 1 --2- 172.21.15.43:0/2261908309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab86790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.497+0000 7f3b73400700 1 --2- 172.21.15.43:0/2261908309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333ab86e70 0x56333ab9cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.497+0000 7f3b72a00700 1 -- 172.21.15.43:0/2261908309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 msgr2=0x56333ab86790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.497+0000 7f3b72a00700 1 --2- 172.21.15.43:0/2261908309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab86790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.497+0000 7f3b72a00700 1 -- 172.21.15.43:0/2261908309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333ab86e70 msgr2=0x56333ab9cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.497+0000 7f3b72a00700 1 --2- 172.21.15.43:0/2261908309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333ab86e70 0x56333ab9cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.497+0000 7f3b72a00700 1 -- 172.21.15.43:0/2261908309 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56333ab7e240 con 0x56333aa07e40 2024-05-27T07:20:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.498+0000 7f3b72a00700 1 --2- 172.21.15.43:0/2261908309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aa07e40 0x56333ab8bef0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3b64009750 tx=0x7f3b64005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b72b32fa114f68 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.498+0000 7f3b5f400700 1 -- 172.21.15.43:0/2261908309 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b640037c0 con 0x56333aa07e40 2024-05-27T07:20:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.498+0000 7f3b5f400700 1 -- 172.21.15.43:0/2261908309 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b64003df0 con 0x56333aa07e40 2024-05-27T07:20:05.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.498+0000 7f3b5f400700 1 -- 172.21.15.43:0/2261908309 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b640246e0 con 0x56333aa07e40 2024-05-27T07:20:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.499+0000 7f3b885b8b80 1 -- 172.21.15.43:0/2261908309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aa07e40 msgr2=0x56333ab8bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.499+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/2261908309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aa07e40 0x56333ab8bef0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3b64009750 tx=0x7f3b64005040 comp rx=0 tx=0).stop 2024-05-27T07:20:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.499+0000 7f3b885b8b80 1 -- 172.21.15.43:0/2261908309 shutdown_connections 2024-05-27T07:20:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.499+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/2261908309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333ab86e70 0x56333ab9cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.499+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/2261908309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333aa07e40 0x56333ab8bef0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.499+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/2261908309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab86790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.499+0000 7f3b885b8b80 1 -- 172.21.15.43:0/2261908309 >> 172.21.15.43:0/2261908309 conn(0x56333aaa1390 msgr2=0x56333ab91d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.499+0000 7f3b885b8b80 1 -- 172.21.15.43:0/2261908309 shutdown_connections 2024-05-27T07:20:05.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.500+0000 7f3b885b8b80 1 -- 172.21.15.43:0/2261908309 wait complete. 2024-05-27T07:20:05.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.500+0000 7f3b885b8b80 1 -- start start 2024-05-27T07:20:05.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.501+0000 7f3b885b8b80 1 Processor -- start 2024-05-27T07:20:05.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.501+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab990e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.501+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab99680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.501+0000 7f3b885b8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab8abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.501+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3b741b1820 con 0x56333ab86e70 2024-05-27T07:20:05.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.501+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3b741b2c90 con 0x56333aa07e40 2024-05-27T07:20:05.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.501+0000 7f3b885b8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3b741b1ef0 con 0x56333aaf39f0 2024-05-27T07:20:05.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.501+0000 7f3b885b8b80 10 client.0 _ll_register_callbacks cb 0x56333aa4ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:05.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.502+0000 7f3b73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab8abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.502+0000 7f3b73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab8abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53008/0 (socket says 172.21.15.43:53008) 2024-05-27T07:20:05.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.502+0000 7f3b73400700 1 -- 172.21.15.43:0/1090669632 learned_addr learned my addr 172.21.15.43:0/1090669632 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:05.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.502+0000 7f3b72a00700 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.502+0000 7f3b72000700 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab99680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.503+0000 7f3b73400700 1 -- 172.21.15.43:0/1090669632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 msgr2=0x56333ab99680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:05.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.503+0000 7f3b73400700 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab99680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.503+0000 7f3b73400700 1 -- 172.21.15.43:0/1090669632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 msgr2=0x56333ab990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:05.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.503+0000 7f3b73400700 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.503+0000 7f3b73400700 1 -- 172.21.15.43:0/1090669632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56333ab8b380 con 0x56333ab86e70 2024-05-27T07:20:05.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.503+0000 7f3b73400700 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab8abe0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0a9230 tx=0x7f3b6c0ad2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15be0f668b03fef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:05.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.503+0000 7f3b5cc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:05.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.503+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1090669632 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c06c070 con 0x56333ab86e70 2024-05-27T07:20:05.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.503+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1090669632 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f3b6c0b1bc0 con 0x56333ab86e70 2024-05-27T07:20:05.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.503+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1090669632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b64003df0 con 0x56333ab86e70 2024-05-27T07:20:05.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.504+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1090669632 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f3b6c0a8b80 con 0x56333ab86e70 2024-05-27T07:20:05.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.504+0000 7f3b885b8b80 10 client.16368 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.504+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1090669632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x56333ab989c0 con 0x56333ab86e70 2024-05-27T07:20:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.504+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1090669632 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b64003df0 con 0x56333ab86e70 2024-05-27T07:20:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.504+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1090669632 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f3b6c012070 con 0x56333ab86e70 2024-05-27T07:20:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.504+0000 7f3b885b8b80 10 client.16368 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.504+0000 7f3b885b8b80 10 client.16368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:20:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.504+0000 7f3b885b8b80 20 client.16368 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:05.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.504+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56333ac0baf0 0x56333ac0df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:05.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.504+0000 7f3b885b8b80 4 client.16368 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:20:05.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.505+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1090669632 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56333aab0570 con 0x56333ac0baf0 2024-05-27T07:20:05.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.505+0000 7f3b72a00700 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56333ac0baf0 0x56333ac0df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.507+0000 7f3b72a00700 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56333ac0baf0 0x56333ac0df10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74a708f7b1e13f97 server_cookie=7a2e3821781723c1 in_seq=0 out_seq=0 2024-05-27T07:20:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.507+0000 7f3b5cc00700 10 client.16368 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.507+0000 7f3b5cc00700 1 -- 172.21.15.43:0/1090669632 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56333aab0570 con 0x56333ac0baf0 2024-05-27T07:20:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.507+0000 7f3b5cc00700 10 client.16368 handle_command_reply: tid=0 2024-05-27T07:20:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.507+0000 7f3b885b8b80 1 client.16368 shutdown 2024-05-27T07:20:05.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1090669632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 msgr2=0x56333ab8abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:05.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab8abe0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0a9230 tx=0x7f3b6c0ad2f0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1090669632 shutdown_connections 2024-05-27T07:20:05.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56333ac0baf0 0x56333ac0df10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56333ab86e70 0x56333ab8abe0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56333aa07e40 0x56333ab990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 1 --2- 172.21.15.43:0/1090669632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56333aaf39f0 0x56333ab99680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1090669632 >> 172.21.15.43:0/1090669632 conn(0x56333aaa1390 msgr2=0x56333ab90ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:05.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b5cc00700 0 client.16368 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:05.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1090669632 shutdown_connections 2024-05-27T07:20:05.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 1 -- 172.21.15.43:0/1090669632 wait complete. 2024-05-27T07:20:05.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.508+0000 7f3b885b8b80 20 client.16368 trim_cache size 0 max 16384 2024-05-27T07:20:05.500 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:20:05.500 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:20:05.500 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:20:05.500 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:20:05.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.509+0000 7f3b79a00700 1 -- 172.21.15.43:0/936971501 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3b580b29f0 msgr2=0x7f3b580b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.509+0000 7f3b79a00700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3b580b29f0 0x7f3b580b4f00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3b741025e0 tx=0x7f3b600106b0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.509+0000 7f3b79a00700 1 -- 172.21.15.43:0/936971501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b7410e930 msgr2=0x7f3b74103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.509+0000 7f3b79a00700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b7410e930 0x7f3b74103be0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00ada0 tx=0x7f3b6c00c330 comp rx=0 tx=0).stop 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.509+0000 7f3b79a00700 1 -- 172.21.15.43:0/936971501 shutdown_connections 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.509+0000 7f3b79a00700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3b580b29f0 0x7f3b580b4f00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.509+0000 7f3b79a00700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b7410e930 0x7f3b74103be0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.509+0000 7f3b79a00700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b74105470 0x7f3b74103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.509+0000 7f3b79a00700 1 --2- 172.21.15.43:0/936971501 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b74105e50 0x7f3b741036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.509+0000 7f3b79a00700 1 -- 172.21.15.43:0/936971501 >> 172.21.15.43:0/936971501 conn(0x7f3b740fef40 msgr2=0x7f3b740ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.510+0000 7f3b79a00700 1 -- 172.21.15.43:0/936971501 shutdown_connections 2024-05-27T07:20:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:05.510+0000 7f3b79a00700 1 -- 172.21.15.43:0/936971501 wait complete. 2024-05-27T07:20:06.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:05 smithi132 ceph-mon[35283]: pgmap v208: 129 pgs: 129 active+clean; 52 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 232 KiB/s wr, 58 op/s 2024-05-27T07:20:06.199 INFO:tasks.workunit.client.0.smithi043.stdout: 1 30129 0.63 MB/sec execute 130 sec latency 66.671 ms 2024-05-27T07:20:06.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:05 smithi043 ceph-mon[31531]: pgmap v208: 129 pgs: 129 active+clean; 52 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 232 KiB/s wr, 58 op/s 2024-05-27T07:20:06.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:05 smithi179 ceph-mon[35888]: pgmap v208: 129 pgs: 129 active+clean; 52 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 232 KiB/s wr, 58 op/s 2024-05-27T07:20:06.462 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:20:06.463 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:20:06.978 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:07.199 INFO:tasks.workunit.client.0.smithi043.stdout: 1 30282 0.63 MB/sec execute 131 sec latency 59.174 ms 2024-05-27T07:20:08.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:07 smithi132 ceph-mon[35283]: pgmap v209: 129 pgs: 129 active+clean; 53 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 387 KiB/s wr, 78 op/s 2024-05-27T07:20:08.189 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:07 smithi043 ceph-mon[31531]: pgmap v209: 129 pgs: 129 active+clean; 53 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 387 KiB/s wr, 78 op/s 2024-05-27T07:20:08.199 INFO:tasks.workunit.client.0.smithi043.stdout: 1 30421 0.62 MB/sec execute 132 sec latency 55.241 ms 2024-05-27T07:20:08.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:07 smithi179 ceph-mon[35888]: pgmap v209: 129 pgs: 129 active+clean; 53 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 387 KiB/s wr, 78 op/s 2024-05-27T07:20:09.200 INFO:tasks.workunit.client.0.smithi043.stdout: 1 30559 0.62 MB/sec execute 133 sec latency 53.880 ms 2024-05-27T07:20:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.402+0000 7f00c8000700 1 -- 172.21.15.43:0/1135511907 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c0108f10 msgr2=0x7f00c0109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.402+0000 7f00c8000700 1 --2- 172.21.15.43:0/1135511907 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c0108f10 0x7f00c0109320 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0099c0 tx=0x7f00ac009cd0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.402+0000 7f00c8000700 1 -- 172.21.15.43:0/1135511907 shutdown_connections 2024-05-27T07:20:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.402+0000 7f00c8000700 1 --2- 172.21.15.43:0/1135511907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00c0109860 0x7f00c01111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.402+0000 7f00c8000700 1 --2- 172.21.15.43:0/1135511907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00c01045e0 0x7f00c01049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.402+0000 7f00c8000700 1 --2- 172.21.15.43:0/1135511907 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c0108f10 0x7f00c0109320 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.402+0000 7f00c8000700 1 -- 172.21.15.43:0/1135511907 >> 172.21.15.43:0/1135511907 conn(0x7f00c0100030 msgr2=0x7f00c0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.402+0000 7f00c8000700 1 -- 172.21.15.43:0/1135511907 shutdown_connections 2024-05-27T07:20:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.402+0000 7f00c8000700 1 -- 172.21.15.43:0/1135511907 wait complete. 2024-05-27T07:20:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.404+0000 7f00c8000700 1 Processor -- start 2024-05-27T07:20:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.404+0000 7f00c8000700 1 -- start start 2024-05-27T07:20:09.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.404+0000 7f00c8000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00c01045e0 0x7f00c019ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:09.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.404+0000 7f00c8000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c0108f10 0x7f00c019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:09.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.405+0000 7f00c8000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00c0109860 0x7f00c019f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:09.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.405+0000 7f00c8000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f00c0114320 con 0x7f00c0108f10 2024-05-27T07:20:09.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.405+0000 7f00c8000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f00c01141a0 con 0x7f00c0109860 2024-05-27T07:20:09.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.405+0000 7f00c8000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f00c01144a0 con 0x7f00c01045e0 2024-05-27T07:20:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.405+0000 7f00c4c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c0108f10 0x7f00c019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.405+0000 7f00c4c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c0108f10 0x7f00c019b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53030/0 (socket says 172.21.15.43:53030) 2024-05-27T07:20:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.405+0000 7f00c4c00700 1 -- 172.21.15.43:0/1694318192 learned_addr learned my addr 172.21.15.43:0/1694318192 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.405+0000 7f00c5600700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00c01045e0 0x7f00c019ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.405+0000 7f00c6000700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00c0109860 0x7f00c019f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.406+0000 7f00c4c00700 1 -- 172.21.15.43:0/1694318192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00c01045e0 msgr2=0x7f00c019ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.406+0000 7f00c4c00700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00c01045e0 0x7f00c019ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.406+0000 7f00c4c00700 1 -- 172.21.15.43:0/1694318192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00c0109860 msgr2=0x7f00c019f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.406+0000 7f00c4c00700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00c0109860 0x7f00c019f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.406+0000 7f00c4c00700 1 -- 172.21.15.43:0/1694318192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c019fbf0 con 0x7f00c0108f10 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.406+0000 7f00c4c00700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c0108f10 0x7f00c019b310 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00b580 tx=0x7f00ac0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.406+0000 7f00be000700 1 -- 172.21.15.43:0/1694318192 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac027070 con 0x7f00c0108f10 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.407+0000 7f00be000700 1 -- 172.21.15.43:0/1694318192 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f00ac00fd30 con 0x7f00c0108f10 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.407+0000 7f00c8000700 1 -- 172.21.15.43:0/1694318192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c019fe80 con 0x7f00c0108f10 2024-05-27T07:20:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.407+0000 7f00be000700 1 -- 172.21.15.43:0/1694318192 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac0173f0 con 0x7f00c0108f10 2024-05-27T07:20:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.407+0000 7f00c8000700 1 -- 172.21.15.43:0/1694318192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c01a7800 con 0x7f00c0108f10 2024-05-27T07:20:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.408+0000 7f00c8000700 1 -- 172.21.15.43:0/1694318192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c0105a90 con 0x7f00c0108f10 2024-05-27T07:20:09.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.411+0000 7f00be000700 1 -- 172.21.15.43:0/1694318192 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f00ac02a070 con 0x7f00c0108f10 2024-05-27T07:20:09.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.412+0000 7f00be000700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f00a80b2a70 0x7f00a80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:09.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.413+0000 7f00be000700 1 -- 172.21.15.43:0/1694318192 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f00ac0e3780 con 0x7f00c0108f10 2024-05-27T07:20:09.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.413+0000 7f00c5600700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f00a80b2a70 0x7f00a80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:09.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.414+0000 7f00c5600700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f00a80b2a70 0x7f00a80b4f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009550 tx=0x7f00b40091b0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.418+0000 7f00be000700 1 -- 172.21.15.43:0/1694318192 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f00ac013070 con 0x7f00c0108f10 2024-05-27T07:20:09.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.731+0000 7f00c8000700 1 -- 172.21.15.43:0/1694318192 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00c019c080 con 0x7f00c0108f10 2024-05-27T07:20:09.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.732+0000 7f00be000700 1 -- 172.21.15.43:0/1694318192 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+8623 (secure 0 0 0) 0x7f00ac0aae40 con 0x7f00c0108f10 2024-05-27T07:20:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:20:09.723 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":79,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:19:26.072115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34403,24484,34424,34484,34448]},"id":1}]} 2024-05-27T07:20:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.735+0000 7f00c8000700 1 -- 172.21.15.43:0/1694318192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f00a80b2a70 msgr2=0x7f00a80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.735+0000 7f00c8000700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f00a80b2a70 0x7f00a80b4f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009550 tx=0x7f00b40091b0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.735+0000 7f00c8000700 1 -- 172.21.15.43:0/1694318192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c0108f10 msgr2=0x7f00c019b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:09.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.735+0000 7f00c8000700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c0108f10 0x7f00c019b310 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00b580 tx=0x7f00ac0057c0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.736+0000 7f00c8000700 1 -- 172.21.15.43:0/1694318192 shutdown_connections 2024-05-27T07:20:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.736+0000 7f00c8000700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f00a80b2a70 0x7f00a80b4f80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.736+0000 7f00c8000700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00c0109860 0x7f00c019f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.736+0000 7f00c8000700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00c01045e0 0x7f00c019ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.736+0000 7f00c8000700 1 --2- 172.21.15.43:0/1694318192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f00c0108f10 0x7f00c019b310 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.736+0000 7f00c8000700 1 -- 172.21.15.43:0/1694318192 >> 172.21.15.43:0/1694318192 conn(0x7f00c0100030 msgr2=0x7f00c0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.736+0000 7f00c8000700 1 -- 172.21.15.43:0/1694318192 shutdown_connections 2024-05-27T07:20:09.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:09.736+0000 7f00c8000700 1 -- 172.21.15.43:0/1694318192 wait complete. 2024-05-27T07:20:09.728 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 79 2024-05-27T07:20:10.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:09 smithi132 ceph-mon[35283]: pgmap v210: 129 pgs: 129 active+clean; 53 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 290 KiB/s wr, 59 op/s 2024-05-27T07:20:10.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:09 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1694318192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:20:10.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:09 smithi043 ceph-mon[31531]: pgmap v210: 129 pgs: 129 active+clean; 53 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 290 KiB/s wr, 59 op/s 2024-05-27T07:20:10.119 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:09 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1694318192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:20:10.199 INFO:tasks.workunit.client.0.smithi043.stdout: 1 30722 0.62 MB/sec execute 134 sec latency 73.395 ms 2024-05-27T07:20:10.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:09 smithi179 ceph-mon[35888]: pgmap v210: 129 pgs: 129 active+clean; 53 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 290 KiB/s wr, 59 op/s 2024-05-27T07:20:10.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:09 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1694318192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:20:10.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:20:11.053 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:11.200 INFO:tasks.workunit.client.0.smithi043.stdout: 1 30813 0.62 MB/sec execute 135 sec latency 52.117 ms 2024-05-27T07:20:11.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:11 smithi043 ceph-mon[31531]: pgmap v211: 129 pgs: 129 active+clean; 54 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 421 KiB/s wr, 74 op/s 2024-05-27T07:20:12.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:11 smithi132 ceph-mon[35283]: pgmap v211: 129 pgs: 129 active+clean; 54 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 421 KiB/s wr, 74 op/s 2024-05-27T07:20:12.200 INFO:tasks.workunit.client.0.smithi043.stdout: 1 30892 0.62 MB/sec execute 136 sec latency 69.366 ms 2024-05-27T07:20:12.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:11 smithi179 ceph-mon[35888]: pgmap v211: 129 pgs: 129 active+clean; 54 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 421 KiB/s wr, 74 op/s 2024-05-27T07:20:13.200 INFO:tasks.workunit.client.0.smithi043.stdout: 1 31013 0.62 MB/sec execute 137 sec latency 52.850 ms 2024-05-27T07:20:13.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.541+0000 7fe307000700 1 -- 172.21.15.43:0/448445549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3001109a0 msgr2=0x7fe300112e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:13.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.541+0000 7fe307000700 1 --2- 172.21.15.43:0/448445549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3001109a0 0x7fe300112e00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800b0a0 tx=0x7fe2f800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.542+0000 7fe307000700 1 -- 172.21.15.43:0/448445549 shutdown_connections 2024-05-27T07:20:13.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.542+0000 7fe307000700 1 --2- 172.21.15.43:0/448445549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3001109a0 0x7fe300112e00 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.542+0000 7fe307000700 1 --2- 172.21.15.43:0/448445549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300075d20 0x7fe300076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.542+0000 7fe307000700 1 --2- 172.21.15.43:0/448445549 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe300076670 0x7fe300110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.542+0000 7fe307000700 1 -- 172.21.15.43:0/448445549 >> 172.21.15.43:0/448445549 conn(0x7fe300079d00 msgr2=0x7fe30007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:13.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.542+0000 7fe307000700 1 -- 172.21.15.43:0/448445549 shutdown_connections 2024-05-27T07:20:13.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.542+0000 7fe307000700 1 -- 172.21.15.43:0/448445549 wait complete. 2024-05-27T07:20:13.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.543+0000 7fe307000700 1 Processor -- start 2024-05-27T07:20:13.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.544+0000 7fe307000700 1 -- start start 2024-05-27T07:20:13.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.544+0000 7fe307000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe300075d20 0x7fe3001a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.544+0000 7fe307000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300076670 0x7fe3001a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.544+0000 7fe307000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3001109a0 0x7fe30019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.544+0000 7fe307000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe300116330 con 0x7fe3001109a0 2024-05-27T07:20:13.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.544+0000 7fe307000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001161b0 con 0x7fe300076670 2024-05-27T07:20:13.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.544+0000 7fe307000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001164b0 con 0x7fe300075d20 2024-05-27T07:20:13.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe306600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3001109a0 0x7fe30019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe306600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3001109a0 0x7fe30019cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53044/0 (socket says 172.21.15.43:53044) 2024-05-27T07:20:13.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe306600700 1 -- 172.21.15.43:0/3733670186 learned_addr learned my addr 172.21.15.43:0/3733670186 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:13.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe305c00700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe300075d20 0x7fe3001a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe305200700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300076670 0x7fe3001a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe306600700 1 -- 172.21.15.43:0/3733670186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe300075d20 msgr2=0x7fe3001a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:13.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe306600700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe300075d20 0x7fe3001a29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe306600700 1 -- 172.21.15.43:0/3733670186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300076670 msgr2=0x7fe3001a2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:13.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe306600700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300076670 0x7fe3001a2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe306600700 1 -- 172.21.15.43:0/3733670186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30019d2c0 con 0x7fe3001109a0 2024-05-27T07:20:13.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.545+0000 7fe305c00700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe300075d20 0x7fe3001a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:13.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.546+0000 7fe306600700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3001109a0 0x7fe30019cb90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800b780 tx=0x7fe2f8004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:13.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.546+0000 7fe2fe000700 1 -- 172.21.15.43:0/3733670186 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8021070 con 0x7fe3001109a0 2024-05-27T07:20:13.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.546+0000 7fe307000700 1 -- 172.21.15.43:0/3733670186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30019d550 con 0x7fe3001109a0 2024-05-27T07:20:13.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.546+0000 7fe2fe000700 1 -- 172.21.15.43:0/3733670186 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2f8007b80 con 0x7fe3001109a0 2024-05-27T07:20:13.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.546+0000 7fe2fe000700 1 -- 172.21.15.43:0/3733670186 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80052a0 con 0x7fe3001109a0 2024-05-27T07:20:13.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.546+0000 7fe307000700 1 -- 172.21.15.43:0/3733670186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3001a9760 con 0x7fe3001109a0 2024-05-27T07:20:13.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.550+0000 7fe2fe000700 1 -- 172.21.15.43:0/3733670186 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe2f8024070 con 0x7fe3001109a0 2024-05-27T07:20:13.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.552+0000 7fe3170d4b80 1 Processor -- start 2024-05-27T07:20:13.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.552+0000 7fe3170d4b80 1 -- start start 2024-05-27T07:20:13.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.553+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557303076e40 0x5573031f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.553+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x5573031ffad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.553+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303202020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.553+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557303123e40 con 0x5573031f5820 2024-05-27T07:20:13.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.553+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5573031139e0 con 0x5573031f5410 2024-05-27T07:20:13.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.553+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557303117a20 con 0x557303076e40 2024-05-27T07:20:13.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.553+0000 7fe2fe000700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2e80b29f0 0x7fe2e80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.554+0000 7fe2fe000700 1 -- 172.21.15.43:0/3733670186 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fe2f800f020 con 0x7fe3001109a0 2024-05-27T07:20:13.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.554+0000 7fe305200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303202020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.554+0000 7fe306600700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2e80b29f0 0x7fe2e80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.554+0000 7fe305200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303202020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53050/0 (socket says 172.21.15.43:53050) 2024-05-27T07:20:13.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.554+0000 7fe305200700 1 -- 172.21.15.43:0/2210617338 learned_addr learned my addr 172.21.15.43:0/2210617338 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:13.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.554+0000 7fe305200700 1 --2- 172.21.15.43:0/2210617338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557303076e40 0x5573031f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.554+0000 7fe305c00700 1 --2- 172.21.15.43:0/2210617338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x5573031ffad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.554+0000 7fe305200700 1 -- 172.21.15.43:0/2210617338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557303076e40 msgr2=0x5573031f4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:13.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.554+0000 7fe305200700 1 --2- 172.21.15.43:0/2210617338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557303076e40 0x5573031f4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.554+0000 7fe305200700 1 -- 172.21.15.43:0/2210617338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 msgr2=0x5573031ffad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:13.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.555+0000 7fe305200700 1 --2- 172.21.15.43:0/2210617338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x5573031ffad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.555+0000 7fe305200700 1 -- 172.21.15.43:0/2210617338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573030dc420 con 0x5573031f5820 2024-05-27T07:20:13.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.555+0000 7fe305c00700 1 --2- 172.21.15.43:0/2210617338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x5573031ffad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:13.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.555+0000 7fe306600700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2e80b29f0 0x7fe2e80b4f00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80a7040 tx=0x7fe2f8009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:13.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.555+0000 7fe305200700 1 --2- 172.21.15.43:0/2210617338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303202020 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x5573031f1690 tx=0x7fe2ec0096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85324cdebde267fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:13.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.555+0000 7fe2f2a00700 1 -- 172.21.15.43:0/2210617338 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec012d20 con 0x5573031f5820 2024-05-27T07:20:13.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.555+0000 7fe2f2a00700 1 -- 172.21.15.43:0/2210617338 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2ec0044c0 con 0x5573031f5820 2024-05-27T07:20:13.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.555+0000 7fe2f2a00700 1 -- 172.21.15.43:0/2210617338 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec00f450 con 0x5573031f5820 2024-05-27T07:20:13.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.556+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2210617338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 msgr2=0x557303202020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:13.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.556+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/2210617338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303202020 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x5573031f1690 tx=0x7fe2ec0096d0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.557+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2210617338 shutdown_connections 2024-05-27T07:20:13.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.557+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/2210617338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303202020 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.557+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/2210617338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557303076e40 0x5573031f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.557+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/2210617338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x5573031ffad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.557+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2210617338 >> 172.21.15.43:0/2210617338 conn(0x5573030d6fc0 msgr2=0x5573031f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:13.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.557+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2210617338 shutdown_connections 2024-05-27T07:20:13.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.557+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2210617338 wait complete. 2024-05-27T07:20:13.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.557+0000 7fe3170d4b80 1 -- start start 2024-05-27T07:20:13.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.558+0000 7fe3170d4b80 1 Processor -- start 2024-05-27T07:20:13.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.558+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557303076e40 0x5573031ff5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.558+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5410 0x5573031ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.558+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303200020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.558+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557303019fa0 con 0x5573031f5820 2024-05-27T07:20:13.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.558+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5573030b4760 con 0x557303076e40 2024-05-27T07:20:13.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.558+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557303202990 con 0x5573031f5410 2024-05-27T07:20:13.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.558+0000 7fe3170d4b80 10 client.0 _ll_register_callbacks cb 0x5573030bdc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:13.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.559+0000 7fe305200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303200020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.559+0000 7fe305c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5410 0x5573031ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.559+0000 7fe305200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303200020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53064/0 (socket says 172.21.15.43:53064) 2024-05-27T07:20:13.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.559+0000 7fe305c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5410 0x5573031ffae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:45644/0 (socket says 172.21.15.43:45644) 2024-05-27T07:20:13.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.559+0000 7fe305200700 1 -- 172.21.15.43:0/2997424911 learned_addr learned my addr 172.21.15.43:0/2997424911 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:13.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.559+0000 7fe305200700 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557303076e40 0x5573031ff5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.560+0000 7fe305c00700 1 -- 172.21.15.43:0/2997424911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557303076e40 msgr2=0x5573031ff5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:13.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.560+0000 7fe305c00700 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557303076e40 0x5573031ff5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.560+0000 7fe305c00700 1 -- 172.21.15.43:0/2997424911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 msgr2=0x557303200020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:13.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.560+0000 7fe305c00700 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303200020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.560+0000 7fe305c00700 1 -- 172.21.15.43:0/2997424911 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573030638d0 con 0x5573031f5410 2024-05-27T07:20:13.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.560+0000 7fe305200700 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303200020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:20:13.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.560+0000 7fe305c00700 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5410 0x5573031ffae0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x5573031f2050 tx=0x7fe2f40116c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a71b37c4bb6a64e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:13.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.560+0000 7fe2e7400700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:20:13.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.561+0000 7fe2e7400700 1 -- 172.21.15.43:0/2997424911 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4004220 con 0x5573031f5410 2024-05-27T07:20:13.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.561+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2997424911 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2ec0044c0 con 0x5573031f5410 2024-05-27T07:20:13.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.561+0000 7fe2e7400700 1 -- 172.21.15.43:0/2997424911 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fe2f40043c0 con 0x5573031f5410 2024-05-27T07:20:13.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.561+0000 7fe2e7400700 1 -- 172.21.15.43:0/2997424911 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fe2f4003680 con 0x5573031f5410 2024-05-27T07:20:13.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.561+0000 7fe3170d4b80 10 client.26029 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:13.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.561+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2997424911 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x5573030e4060 con 0x5573031f5410 2024-05-27T07:20:13.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.562+0000 7fe2e7400700 1 -- 172.21.15.43:0/2997424911 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2ec0044c0 con 0x5573031f5410 2024-05-27T07:20:13.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.562+0000 7fe2e7400700 1 -- 172.21.15.43:0/2997424911 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fe2f40127c0 con 0x5573031f5410 2024-05-27T07:20:13.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.562+0000 7fe3170d4b80 10 client.26029 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:13.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.562+0000 7fe3170d4b80 10 client.26029 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:20:13.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.562+0000 7fe3170d4b80 20 client.26029 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:13.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.562+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55730320fbc0 0x557303212010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.562+0000 7fe3170d4b80 4 client.26029 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:20:13.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.562+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2997424911 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557303125d90 con 0x55730320fbc0 2024-05-27T07:20:13.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.563+0000 7fe305200700 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55730320fbc0 0x557303212010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.564+0000 7fe305200700 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55730320fbc0 0x557303212010 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=84049b0984ad0035 server_cookie=dea8449977c1d5c0 in_seq=0 out_seq=0 2024-05-27T07:20:13.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.564+0000 7fe2e7400700 10 client.26029 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:20:13.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe2e7400700 1 -- 172.21.15.43:0/2997424911 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557303125d90 con 0x55730320fbc0 2024-05-27T07:20:13.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe2e7400700 10 client.26029 handle_command_reply: tid=0 2024-05-27T07:20:13.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe3170d4b80 1 client.26029 shutdown 2024-05-27T07:20:13.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2997424911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5410 msgr2=0x5573031ffae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:13.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5410 0x5573031ffae0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x5573031f2050 tx=0x7fe2f40116c0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2997424911 shutdown_connections 2024-05-27T07:20:13.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55730320fbc0 0x557303212010 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5573031f5820 0x557303200020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557303076e40 0x5573031ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/2997424911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5410 0x5573031ffae0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2997424911 >> 172.21.15.43:0/2997424911 conn(0x5573030d6fc0 msgr2=0x5573031efff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:13.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.566+0000 7fe2e7400700 0 client.26029 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:20:13.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.567+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2997424911 shutdown_connections 2024-05-27T07:20:13.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.567+0000 7fe3170d4b80 1 -- 172.21.15.43:0/2997424911 wait complete. 2024-05-27T07:20:13.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.567+0000 7fe3170d4b80 20 client.26029 trim_cache size 0 max 16384 2024-05-27T07:20:13.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.585+0000 7fe3170d4b80 1 Processor -- start 2024-05-27T07:20:13.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.585+0000 7fe3170d4b80 1 -- start start 2024-05-27T07:20:13.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.585+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x5573031f97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.585+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x55730320ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.585+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5820 0x5573031ffa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.585+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b3090 con 0x557303076e40 2024-05-27T07:20:13.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.585+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b3760 con 0x5573031f5410 2024-05-27T07:20:13.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.585+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b0e80 con 0x5573031f5820 2024-05-27T07:20:13.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.586+0000 7fe305200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x5573031f97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.586+0000 7fe305200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x5573031f97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53078/0 (socket says 172.21.15.43:53078) 2024-05-27T07:20:13.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.586+0000 7fe305200700 1 -- 172.21.15.43:0/4003051241 learned_addr learned my addr 172.21.15.43:0/4003051241 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:13.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.586+0000 7fe305c00700 1 --2- 172.21.15.43:0/4003051241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x55730320ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.586+0000 7fe305200700 1 --2- 172.21.15.43:0/4003051241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5820 0x5573031ffa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.587+0000 7fe305200700 1 -- 172.21.15.43:0/4003051241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5820 msgr2=0x5573031ffa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:13.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.587+0000 7fe305200700 1 --2- 172.21.15.43:0/4003051241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5820 0x5573031ffa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.587+0000 7fe305200700 1 -- 172.21.15.43:0/4003051241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 msgr2=0x55730320ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:13.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.587+0000 7fe305200700 1 --2- 172.21.15.43:0/4003051241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x55730320ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.587+0000 7fe305200700 1 -- 172.21.15.43:0/4003051241 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573031f2350 con 0x557303076e40 2024-05-27T07:20:13.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.587+0000 7fe305c00700 1 --2- 172.21.15.43:0/4003051241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x55730320ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:13.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.587+0000 7fe305200700 1 --2- 172.21.15.43:0/4003051241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x5573031f97a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec012cf0 tx=0x7fe2ec017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50459e9c544b47ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:13.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.587+0000 7fe2f2000700 1 -- 172.21.15.43:0/4003051241 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec0052e0 con 0x557303076e40 2024-05-27T07:20:13.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.587+0000 7fe2f2000700 1 -- 172.21.15.43:0/4003051241 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2ec034020 con 0x557303076e40 2024-05-27T07:20:13.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.587+0000 7fe2f2000700 1 -- 172.21.15.43:0/4003051241 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec004e60 con 0x557303076e40 2024-05-27T07:20:13.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.588+0000 7fe3170d4b80 1 -- 172.21.15.43:0/4003051241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 msgr2=0x5573031f97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:13.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.588+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/4003051241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x5573031f97a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec012cf0 tx=0x7fe2ec017040 comp rx=0 tx=0).stop 2024-05-27T07:20:13.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.588+0000 7fe3170d4b80 1 -- 172.21.15.43:0/4003051241 shutdown_connections 2024-05-27T07:20:13.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.588+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/4003051241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5820 0x5573031ffa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.588+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/4003051241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x5573031f97a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.588+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/4003051241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x55730320ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.588+0000 7fe3170d4b80 1 -- 172.21.15.43:0/4003051241 >> 172.21.15.43:0/4003051241 conn(0x5573030d6fc0 msgr2=0x5573031ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:13.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.589+0000 7fe3170d4b80 1 -- 172.21.15.43:0/4003051241 shutdown_connections 2024-05-27T07:20:13.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.589+0000 7fe3170d4b80 1 -- 172.21.15.43:0/4003051241 wait complete. 2024-05-27T07:20:13.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.589+0000 7fe3170d4b80 1 -- start start 2024-05-27T07:20:13.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.590+0000 7fe3170d4b80 1 Processor -- start 2024-05-27T07:20:13.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.590+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x557303203230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.590+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x557303203770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.590+0000 7fe3170d4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5820 0x557303278690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.590+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b1550 con 0x557303076e40 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.590+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b29c0 con 0x5573031f5410 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.590+0000 7fe3170d4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001b1c20 con 0x5573031f5820 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.590+0000 7fe3170d4b80 10 client.0 _ll_register_callbacks cb 0x5573030bdc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.591+0000 7fe305200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x557303203230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.591+0000 7fe305200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x557303203230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53094/0 (socket says 172.21.15.43:53094) 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.591+0000 7fe305200700 1 -- 172.21.15.43:0/3022312793 learned_addr learned my addr 172.21.15.43:0/3022312793 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.591+0000 7fe305c00700 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x557303203770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.591+0000 7fe305200700 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5820 0x557303278690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.591+0000 7fe305200700 1 -- 172.21.15.43:0/3022312793 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5820 msgr2=0x557303278690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:13.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.591+0000 7fe305200700 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5820 0x557303278690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.591+0000 7fe305200700 1 -- 172.21.15.43:0/3022312793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 msgr2=0x557303203770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:13.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.591+0000 7fe305200700 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x557303203770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.591+0000 7fe305200700 1 -- 172.21.15.43:0/3022312793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557303278bd0 con 0x557303076e40 2024-05-27T07:20:13.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.592+0000 7fe305200700 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x557303203230 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009cb0 tx=0x7fe2ec009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec16bf75e2c5106 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:13.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.592+0000 7fe2e7400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:13.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.592+0000 7fe2e7400700 1 -- 172.21.15.43:0/3022312793 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec0055e0 con 0x557303076e40 2024-05-27T07:20:13.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.592+0000 7fe3170d4b80 1 -- 172.21.15.43:0/3022312793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2ec034020 con 0x557303076e40 2024-05-27T07:20:13.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.592+0000 7fe2e7400700 1 -- 172.21.15.43:0/3022312793 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fe2ec035040 con 0x557303076e40 2024-05-27T07:20:13.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.592+0000 7fe2e7400700 1 -- 172.21.15.43:0/3022312793 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fe2ec003b50 con 0x557303076e40 2024-05-27T07:20:13.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.592+0000 7fe3170d4b80 10 client.16392 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:13.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.592+0000 7fe3170d4b80 1 -- 172.21.15.43:0/3022312793 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x557303279d50 con 0x557303076e40 2024-05-27T07:20:13.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.593+0000 7fe2e7400700 1 -- 172.21.15.43:0/3022312793 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe2ec034020 con 0x557303076e40 2024-05-27T07:20:13.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.593+0000 7fe2e7400700 1 -- 172.21.15.43:0/3022312793 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fe2ec038070 con 0x557303076e40 2024-05-27T07:20:13.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.593+0000 7fe3170d4b80 10 client.16392 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:13.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.593+0000 7fe3170d4b80 10 client.16392 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:20:13.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.593+0000 7fe3170d4b80 20 client.16392 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:13.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.593+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55730327a380 0x55730327c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:13.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.593+0000 7fe3170d4b80 4 client.16392 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:20:13.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.593+0000 7fe3170d4b80 1 -- 172.21.15.43:0/3022312793 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55730311f570 con 0x55730327a380 2024-05-27T07:20:13.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.594+0000 7fe305c00700 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55730327a380 0x55730327c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.595+0000 7fe305c00700 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55730327a380 0x55730327c7a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f4c332a0a3589118 server_cookie=b047dd0b3ae54830 in_seq=0 out_seq=0 2024-05-27T07:20:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.595+0000 7fe2e7400700 10 client.16392 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:20:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.595+0000 7fe2e7400700 1 -- 172.21.15.43:0/3022312793 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55730311f570 con 0x55730327a380 2024-05-27T07:20:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.595+0000 7fe2e7400700 10 client.16392 handle_command_reply: tid=0 2024-05-27T07:20:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.595+0000 7fe3170d4b80 1 client.16392 shutdown 2024-05-27T07:20:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 1 -- 172.21.15.43:0/3022312793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 msgr2=0x557303203230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x557303203230 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009cb0 tx=0x7fe2ec009860 comp rx=0 tx=0).stop 2024-05-27T07:20:13.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 1 -- 172.21.15.43:0/3022312793 shutdown_connections 2024-05-27T07:20:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55730327a380 0x55730327c7a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573031f5820 0x557303278690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557303076e40 0x557303203230 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 1 --2- 172.21.15.43:0/3022312793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5573031f5410 0x557303203770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 1 -- 172.21.15.43:0/3022312793 >> 172.21.15.43:0/3022312793 conn(0x5573030d6fc0 msgr2=0x5573031ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe2e7400700 0 client.16392 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:20:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 1 -- 172.21.15.43:0/3022312793 shutdown_connections 2024-05-27T07:20:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 1 -- 172.21.15.43:0/3022312793 wait complete. 2024-05-27T07:20:13.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.596+0000 7fe3170d4b80 20 client.16392 trim_cache size 0 max 16384 2024-05-27T07:20:13.587 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:20:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 -- 172.21.15.43:0/3733670186 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2e80b29f0 msgr2=0x7fe2e80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2e80b29f0 0x7fe2e80b4f00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80a7040 tx=0x7fe2f8009040 comp rx=0 tx=0).stop 2024-05-27T07:20:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 -- 172.21.15.43:0/3733670186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3001109a0 msgr2=0x7fe30019cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3001109a0 0x7fe30019cb90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800b780 tx=0x7fe2f8004d00 comp rx=0 tx=0).stop 2024-05-27T07:20:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 -- 172.21.15.43:0/3733670186 shutdown_connections 2024-05-27T07:20:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe2e80b29f0 0x7fe2e80b4f00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3001109a0 0x7fe30019cb90 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe300075d20 0x7fe3001a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 --2- 172.21.15.43:0/3733670186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300076670 0x7fe3001a2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 -- 172.21.15.43:0/3733670186 >> 172.21.15.43:0/3733670186 conn(0x7fe300079d00 msgr2=0x7fe300100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 -- 172.21.15.43:0/3733670186 shutdown_connections 2024-05-27T07:20:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:13.597+0000 7fe307000700 1 -- 172.21.15.43:0/3733670186 wait complete. 2024-05-27T07:20:14.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:13 smithi132 ceph-mon[35283]: pgmap v212: 129 pgs: 129 active+clean; 54 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 286 KiB/s wr, 35 op/s 2024-05-27T07:20:14.200 INFO:tasks.workunit.client.0.smithi043.stdout: 1 31286 0.64 MB/sec execute 138 sec latency 55.560 ms 2024-05-27T07:20:14.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:13 smithi043 ceph-mon[31531]: pgmap v212: 129 pgs: 129 active+clean; 54 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 286 KiB/s wr, 35 op/s 2024-05-27T07:20:14.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:13 smithi179 ceph-mon[35888]: pgmap v212: 129 pgs: 129 active+clean; 54 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 286 KiB/s wr, 35 op/s 2024-05-27T07:20:14.501 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:20:15.014 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:15.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 31411 0.64 MB/sec execute 139 sec latency 52.218 ms 2024-05-27T07:20:16.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:15 smithi132 ceph-mon[35283]: pgmap v213: 129 pgs: 129 active+clean; 54 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 286 KiB/s wr, 35 op/s 2024-05-27T07:20:16.200 INFO:tasks.workunit.client.0.smithi043.stdout: 1 31681 0.65 MB/sec execute 140 sec latency 48.816 ms 2024-05-27T07:20:16.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:15 smithi043 ceph-mon[31531]: pgmap v213: 129 pgs: 129 active+clean; 54 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 286 KiB/s wr, 35 op/s 2024-05-27T07:20:16.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:15 smithi179 ceph-mon[35888]: pgmap v213: 129 pgs: 129 active+clean; 54 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 286 KiB/s wr, 35 op/s 2024-05-27T07:20:17.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 31721 0.64 MB/sec execute 141 sec latency 58.434 ms 2024-05-27T07:20:17.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.473+0000 7f5001800700 1 -- 172.21.15.43:0/1783953765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ffc105bf0 msgr2=0x7f4ffc10e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.473+0000 7f5001800700 1 --2- 172.21.15.43:0/1783953765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ffc105bf0 0x7f4ffc10e140 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009990 tx=0x7f4fe8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.473+0000 7f5001800700 1 -- 172.21.15.43:0/1783953765 shutdown_connections 2024-05-27T07:20:17.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.473+0000 7f5001800700 1 --2- 172.21.15.43:0/1783953765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc10e710 0x7f4ffc110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.473+0000 7f5001800700 1 --2- 172.21.15.43:0/1783953765 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ffc1052a0 0x7f4ffc1056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.473+0000 7f5001800700 1 --2- 172.21.15.43:0/1783953765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ffc105bf0 0x7f4ffc10e140 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.473+0000 7f5001800700 1 -- 172.21.15.43:0/1783953765 >> 172.21.15.43:0/1783953765 conn(0x7f4ffc078c30 msgr2=0x7f4ffc079070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:17.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.474+0000 7f5001800700 1 -- 172.21.15.43:0/1783953765 shutdown_connections 2024-05-27T07:20:17.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.474+0000 7f5001800700 1 -- 172.21.15.43:0/1783953765 wait complete. 2024-05-27T07:20:17.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.475+0000 7f5001800700 1 Processor -- start 2024-05-27T07:20:17.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.475+0000 7f5001800700 1 -- start start 2024-05-27T07:20:17.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f5001800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ffc1052a0 0x7f4ffc19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f5001800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc105bf0 0x7f4ffc19d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f5001800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ffc10e710 0x7f4ffc1a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f5001800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc114120 con 0x7f4ffc1052a0 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f5001800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc113fa0 con 0x7f4ffc105bf0 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f5001800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1142a0 con 0x7f4ffc10e710 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ffc1052a0 0x7f4ffc19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ffc1052a0 0x7f4ffc19cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44692/0 (socket says 172.21.15.43:44692) 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f4ffbe00700 1 -- 172.21.15.43:0/203217110 learned_addr learned my addr 172.21.15.43:0/203217110 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f4ffb400700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc105bf0 0x7f4ffc19d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.476+0000 7f5000e00700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ffc10e710 0x7f4ffc1a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.477+0000 7f4ffbe00700 1 -- 172.21.15.43:0/203217110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ffc10e710 msgr2=0x7f4ffc1a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.477+0000 7f4ffbe00700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ffc10e710 0x7f4ffc1a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.477+0000 7f4ffbe00700 1 -- 172.21.15.43:0/203217110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc105bf0 msgr2=0x7f4ffc19d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.477+0000 7f4ffbe00700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc105bf0 0x7f4ffc19d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.477+0000 7f4ffbe00700 1 -- 172.21.15.43:0/203217110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ffc1a1c00 con 0x7f4ffc1052a0 2024-05-27T07:20:17.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.477+0000 7f4ffbe00700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ffc1052a0 0x7f4ffc19cd30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00d950 tx=0x7f4fec00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:17.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.478+0000 7f4ff8c00700 1 -- 172.21.15.43:0/203217110 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec016070 con 0x7f4ffc1052a0 2024-05-27T07:20:17.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.478+0000 7f4ff8c00700 1 -- 172.21.15.43:0/203217110 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4fec009dd0 con 0x7f4ffc1052a0 2024-05-27T07:20:17.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.478+0000 7f5001800700 1 -- 172.21.15.43:0/203217110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ffc1a1ef0 con 0x7f4ffc1052a0 2024-05-27T07:20:17.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.478+0000 7f4ff8c00700 1 -- 172.21.15.43:0/203217110 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec00f3c0 con 0x7f4ffc1052a0 2024-05-27T07:20:17.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.478+0000 7f5001800700 1 -- 172.21.15.43:0/203217110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ffc1a9810 con 0x7f4ffc1052a0 2024-05-27T07:20:17.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.482+0000 7f4ff8c00700 1 -- 172.21.15.43:0/203217110 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4fec019070 con 0x7f4ffc1052a0 2024-05-27T07:20:17.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.483+0000 7f50118bfb80 1 Processor -- start 2024-05-27T07:20:17.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.484+0000 7f50118bfb80 1 -- start start 2024-05-27T07:20:17.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.484+0000 7f4ff8c00700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4fe00b29f0 0x7f4fe00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.484+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680b7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.484+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.484+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.484+0000 7f50118bfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559e67fe4e40 con 0x559e67f37e40 2024-05-27T07:20:17.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.484+0000 7f50118bfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559e67fd49e0 con 0x559e680c12c0 2024-05-27T07:20:17.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.484+0000 7f50118bfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x559e67fd8a20 con 0x559e680239f0 2024-05-27T07:20:17.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f5000e00700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4fe00b29f0 0x7f4fe00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f4ffb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f4ff8c00700 1 -- 172.21.15.43:0/203217110 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f4fec0e2600 con 0x7f4ffc1052a0 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f4ffb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680b7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44696/0 (socket says 172.21.15.43:44696) 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f4ffb400700 1 -- 172.21.15.43:0/2752921543 learned_addr learned my addr 172.21.15.43:0/2752921543 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f4ffbe00700 1 --2- 172.21.15.43:0/2752921543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f5000e00700 1 --2- 172.21.15.43:0/2752921543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f4ffb400700 1 -- 172.21.15.43:0/2752921543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 msgr2=0x559e680b7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f4ffb400700 1 --2- 172.21.15.43:0/2752921543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680b7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f4ffb400700 1 -- 172.21.15.43:0/2752921543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 msgr2=0x559e680c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f4ffb400700 1 --2- 172.21.15.43:0/2752921543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f4ffb400700 1 -- 172.21.15.43:0/2752921543 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e67f9d420 con 0x559e67f37e40 2024-05-27T07:20:17.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.485+0000 7f5000e00700 1 --2- 172.21.15.43:0/2752921543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:20:17.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.486+0000 7f4ffb400700 1 --2- 172.21.15.43:0/2752921543 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680b7c90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009990 tx=0x7f4fe8005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60127a793a04fcc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:17.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.486+0000 7f5000e00700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4fe00b29f0 0x7f4fe00b4f00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x559e680b9210 tx=0x7f4ff4009920 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:17.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.486+0000 7f4ff1600700 1 -- 172.21.15.43:0/2752921543 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8003680 con 0x559e67f37e40 2024-05-27T07:20:17.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.486+0000 7f4ff1600700 1 -- 172.21.15.43:0/2752921543 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4fe8003cb0 con 0x559e67f37e40 2024-05-27T07:20:17.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.486+0000 7f4ff1600700 1 -- 172.21.15.43:0/2752921543 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe800be00 con 0x559e67f37e40 2024-05-27T07:20:17.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.487+0000 7f50118bfb80 1 -- 172.21.15.43:0/2752921543 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 msgr2=0x559e680b7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.487+0000 7f50118bfb80 1 --2- 172.21.15.43:0/2752921543 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680b7c90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009990 tx=0x7f4fe8005f00 comp rx=0 tx=0).stop 2024-05-27T07:20:17.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.488+0000 7f50118bfb80 1 -- 172.21.15.43:0/2752921543 shutdown_connections 2024-05-27T07:20:17.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.488+0000 7f50118bfb80 1 --2- 172.21.15.43:0/2752921543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.488+0000 7f50118bfb80 1 --2- 172.21.15.43:0/2752921543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680b7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.488+0000 7f50118bfb80 1 --2- 172.21.15.43:0/2752921543 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680b7c90 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.488+0000 7f50118bfb80 1 -- 172.21.15.43:0/2752921543 >> 172.21.15.43:0/2752921543 conn(0x559e67fd1390 msgr2=0x559e680b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:17.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.488+0000 7f50118bfb80 1 -- 172.21.15.43:0/2752921543 shutdown_connections 2024-05-27T07:20:17.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.488+0000 7f50118bfb80 1 -- 172.21.15.43:0/2752921543 wait complete. 2024-05-27T07:20:17.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.488+0000 7f50118bfb80 1 -- start start 2024-05-27T07:20:17.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.489+0000 7f50118bfb80 1 Processor -- start 2024-05-27T07:20:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.489+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680be180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.489+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680be6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.489+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680bec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.489+0000 7f50118bfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559e67edafa0 con 0x559e67f37e40 2024-05-27T07:20:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.489+0000 7f50118bfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559e67f75760 con 0x559e680c12c0 2024-05-27T07:20:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.489+0000 7f50118bfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x559e680c3d40 con 0x559e680239f0 2024-05-27T07:20:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.489+0000 7f50118bfb80 10 client.0 _ll_register_callbacks cb 0x559e67f7ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.490+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680be180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.490+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680be180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44710/0 (socket says 172.21.15.43:44710) 2024-05-27T07:20:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.490+0000 7f4ffbe00700 1 -- 172.21.15.43:0/1456540469 learned_addr learned my addr 172.21.15.43:0/1456540469 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:17.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.490+0000 7f4ffb400700 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680be6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.490+0000 7f5000e00700 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680bec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.490+0000 7f4ffbe00700 1 -- 172.21.15.43:0/1456540469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 msgr2=0x559e680be6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:17.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.490+0000 7f4ffbe00700 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680be6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.490+0000 7f4ffbe00700 1 -- 172.21.15.43:0/1456540469 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 msgr2=0x559e680bec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:17.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.490+0000 7f4ffbe00700 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680bec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.490+0000 7f4ffbe00700 1 -- 172.21.15.43:0/1456540469 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e67f248d0 con 0x559e67f37e40 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.491+0000 7f4ffbe00700 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680be180 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0a67e0 tx=0x7f4fec0108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8dbe0e957521 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.491+0000 7f4fe6000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.491+0000 7f4fe6000700 1 -- 172.21.15.43:0/1456540469 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec020070 con 0x559e67f37e40 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.491+0000 7f4fe6000700 1 -- 172.21.15.43:0/1456540469 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f4fec0a5b60 con 0x559e67f37e40 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.491+0000 7f50118bfb80 1 -- 172.21.15.43:0/1456540469 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fe8003cb0 con 0x559e67f37e40 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.491+0000 7f4fe6000700 1 -- 172.21.15.43:0/1456540469 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f4fec0ad920 con 0x559e67f37e40 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.491+0000 7f50118bfb80 10 client.16404 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.491+0000 7f50118bfb80 1 -- 172.21.15.43:0/1456540469 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x559e67fa5060 con 0x559e67f37e40 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.492+0000 7f4fe6000700 1 -- 172.21.15.43:0/1456540469 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4fe8003cb0 con 0x559e67f37e40 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.492+0000 7f4fe6000700 1 -- 172.21.15.43:0/1456540469 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f4fec0dfc60 con 0x559e67f37e40 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.492+0000 7f50118bfb80 10 client.16404 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.492+0000 7f50118bfb80 10 client.16404 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.492+0000 7f50118bfb80 20 client.16404 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.492+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559e680d0bb0 0x559e680d3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.492+0000 7f50118bfb80 4 client.16404 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:20:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.492+0000 7f50118bfb80 1 -- 172.21.15.43:0/1456540469 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e67fe6d90 con 0x559e680d0bb0 2024-05-27T07:20:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.493+0000 7f4ffb400700 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559e680d0bb0 0x559e680d3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.494+0000 7f4ffb400700 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559e680d0bb0 0x559e680d3060 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c0ab0b5438409866 server_cookie=675696f9ae4b02dd in_seq=0 out_seq=0 2024-05-27T07:20:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.494+0000 7f4fe6000700 10 client.16404 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:20:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f4fe6000700 1 -- 172.21.15.43:0/1456540469 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559e67fe6d90 con 0x559e680d0bb0 2024-05-27T07:20:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f4fe6000700 10 client.16404 handle_command_reply: tid=0 2024-05-27T07:20:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f50118bfb80 1 client.16404 shutdown 2024-05-27T07:20:17.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f50118bfb80 1 -- 172.21.15.43:0/1456540469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 msgr2=0x559e680be180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680be180 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0a67e0 tx=0x7f4fec0108d0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f50118bfb80 1 -- 172.21.15.43:0/1456540469 shutdown_connections 2024-05-27T07:20:17.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559e680d0bb0 0x559e680d3060 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680bec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680be180 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1456540469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680be6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.496+0000 7f50118bfb80 1 -- 172.21.15.43:0/1456540469 >> 172.21.15.43:0/1456540469 conn(0x559e67fd1390 msgr2=0x559e680b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:17.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.497+0000 7f4fe6000700 0 client.16404 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:20:17.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.497+0000 7f50118bfb80 1 -- 172.21.15.43:0/1456540469 shutdown_connections 2024-05-27T07:20:17.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.497+0000 7f50118bfb80 1 -- 172.21.15.43:0/1456540469 wait complete. 2024-05-27T07:20:17.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.497+0000 7f50118bfb80 20 client.16404 trim_cache size 0 max 16384 2024-05-27T07:20:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.514+0000 7f50118bfb80 1 Processor -- start 2024-05-27T07:20:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.514+0000 7f50118bfb80 1 -- start start 2024-05-27T07:20:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.514+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e67f37e40 0x559e680b9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.514+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680cce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.515+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e680c12c0 0x559e680d8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.515+0000 7f50118bfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b3070 con 0x559e680c12c0 2024-05-27T07:20:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.515+0000 7f50118bfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b3740 con 0x559e67f37e40 2024-05-27T07:20:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.515+0000 7f50118bfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b0e60 con 0x559e680239f0 2024-05-27T07:20:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.515+0000 7f5000e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e680c12c0 0x559e680d8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.515+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e67f37e40 0x559e680b9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.515+0000 7f5000e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e680c12c0 0x559e680d8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44712/0 (socket says 172.21.15.43:44712) 2024-05-27T07:20:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.515+0000 7f5000e00700 1 -- 172.21.15.43:0/3892951023 learned_addr learned my addr 172.21.15.43:0/3892951023 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.515+0000 7f4ffb400700 1 --2- 172.21.15.43:0/3892951023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.516+0000 7f5000e00700 1 -- 172.21.15.43:0/3892951023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 msgr2=0x559e680cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.516+0000 7f5000e00700 1 --2- 172.21.15.43:0/3892951023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680cce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.516+0000 7f5000e00700 1 -- 172.21.15.43:0/3892951023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e67f37e40 msgr2=0x559e680b9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.516+0000 7f5000e00700 1 --2- 172.21.15.43:0/3892951023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e67f37e40 0x559e680b9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.516+0000 7f5000e00700 1 -- 172.21.15.43:0/3892951023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e680c0a20 con 0x559e680c12c0 2024-05-27T07:20:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.517+0000 7f5000e00700 1 --2- 172.21.15.43:0/3892951023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e680c12c0 0x559e680d8860 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40194a0 tx=0x7f4ff4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69800650823eb93b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.517+0000 7f4ff0c00700 1 -- 172.21.15.43:0/3892951023 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4019c40 con 0x559e680c12c0 2024-05-27T07:20:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.517+0000 7f4ff0c00700 1 -- 172.21.15.43:0/3892951023 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ff402a070 con 0x559e680c12c0 2024-05-27T07:20:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.517+0000 7f4ff0c00700 1 -- 172.21.15.43:0/3892951023 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4017630 con 0x559e680c12c0 2024-05-27T07:20:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.518+0000 7f50118bfb80 1 -- 172.21.15.43:0/3892951023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e680c12c0 msgr2=0x559e680d8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.518+0000 7f50118bfb80 1 --2- 172.21.15.43:0/3892951023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e680c12c0 0x559e680d8860 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40194a0 tx=0x7f4ff4018000 comp rx=0 tx=0).stop 2024-05-27T07:20:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.518+0000 7f50118bfb80 1 -- 172.21.15.43:0/3892951023 shutdown_connections 2024-05-27T07:20:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.518+0000 7f50118bfb80 1 --2- 172.21.15.43:0/3892951023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e680c12c0 0x559e680d8860 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.518+0000 7f50118bfb80 1 --2- 172.21.15.43:0/3892951023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e67f37e40 0x559e680b9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.518+0000 7f50118bfb80 1 --2- 172.21.15.43:0/3892951023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.518+0000 7f50118bfb80 1 -- 172.21.15.43:0/3892951023 >> 172.21.15.43:0/3892951023 conn(0x559e67fd1390 msgr2=0x559e680b9240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.518+0000 7f50118bfb80 1 -- 172.21.15.43:0/3892951023 shutdown_connections 2024-05-27T07:20:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.518+0000 7f50118bfb80 1 -- 172.21.15.43:0/3892951023 wait complete. 2024-05-27T07:20:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.519+0000 7f50118bfb80 1 -- start start 2024-05-27T07:20:17.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.519+0000 7f50118bfb80 1 Processor -- start 2024-05-27T07:20:17.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f50118bfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680c4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f50118bfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b1530 con 0x559e67f37e40 2024-05-27T07:20:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f50118bfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b29a0 con 0x559e680c12c0 2024-05-27T07:20:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f50118bfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b1c00 con 0x559e680239f0 2024-05-27T07:20:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f50118bfb80 10 client.0 _ll_register_callbacks cb 0x559e680d3750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680c97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44714/0 (socket says 172.21.15.43:44714) 2024-05-27T07:20:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f4ffbe00700 1 -- 172.21.15.43:0/1624750152 learned_addr learned my addr 172.21.15.43:0/1624750152 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.520+0000 7f4ffb400700 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f5000e00700 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f4ffbe00700 1 -- 172.21.15.43:0/1624750152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 msgr2=0x559e680c3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f4ffbe00700 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680c3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f4ffbe00700 1 -- 172.21.15.43:0/1624750152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 msgr2=0x559e680c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f4ffbe00700 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680c4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f4ffbe00700 1 -- 172.21.15.43:0/1624750152 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e680c4d00 con 0x559e67f37e40 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f5000e00700 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f4ffb400700 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680c3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f4ffbe00700 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680c97c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0a9270 tx=0x7f4fec0dfe90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8167cceb89bc60 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f4fe6000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f4fe6000700 1 -- 172.21.15.43:0/1624750152 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec0c3060 con 0x559e67f37e40 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.521+0000 7f4fe6000700 1 -- 172.21.15.43:0/1624750152 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f4fec0a5b60 con 0x559e67f37e40 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.522+0000 7f50118bfb80 1 -- 172.21.15.43:0/1624750152 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ff402a070 con 0x559e67f37e40 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.522+0000 7f4fe6000700 1 -- 172.21.15.43:0/1624750152 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f4fec0c3e60 con 0x559e67f37e40 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.522+0000 7f50118bfb80 10 client.16410 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.522+0000 7f50118bfb80 1 -- 172.21.15.43:0/1624750152 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x559e6813a680 con 0x559e67f37e40 2024-05-27T07:20:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.522+0000 7f4fe6000700 1 -- 172.21.15.43:0/1624750152 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4ff402a070 con 0x559e67f37e40 2024-05-27T07:20:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.522+0000 7f4fe6000700 1 -- 172.21.15.43:0/1624750152 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f4fec0a87d0 con 0x559e67f37e40 2024-05-27T07:20:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.522+0000 7f50118bfb80 10 client.16410 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.522+0000 7f50118bfb80 10 client.16410 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:20:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.522+0000 7f50118bfb80 20 client.16410 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.522+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559e6813abe0 0x559e6813d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.523+0000 7f50118bfb80 4 client.16410 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:20:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.523+0000 7f50118bfb80 1 -- 172.21.15.43:0/1624750152 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e67fe0570 con 0x559e6813abe0 2024-05-27T07:20:17.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.523+0000 7f4ffb400700 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559e6813abe0 0x559e6813d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:17.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.524+0000 7f4ffb400700 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559e6813abe0 0x559e6813d000 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57906355b2931947 server_cookie=c210862a8d71df56 in_seq=0 out_seq=0 2024-05-27T07:20:17.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.524+0000 7f4fe6000700 10 client.16410 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:20:17.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f4fe6000700 1 -- 172.21.15.43:0/1624750152 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559e67fe0570 con 0x559e6813abe0 2024-05-27T07:20:17.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f4fe6000700 10 client.16410 handle_command_reply: tid=0 2024-05-27T07:20:17.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 client.16410 shutdown 2024-05-27T07:20:17.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 -- 172.21.15.43:0/1624750152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 msgr2=0x559e680c97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680c97c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0a9270 tx=0x7f4fec0dfe90 comp rx=0 tx=0).stop 2024-05-27T07:20:17.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 -- 172.21.15.43:0/1624750152 shutdown_connections 2024-05-27T07:20:17.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559e6813abe0 0x559e6813d000 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e680c12c0 0x559e680c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559e67f37e40 0x559e680c97c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 --2- 172.21.15.43:0/1624750152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e680239f0 0x559e680c3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 -- 172.21.15.43:0/1624750152 >> 172.21.15.43:0/1624750152 conn(0x559e67fd1390 msgr2=0x559e680b7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:17.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f4fe6000700 0 client.16410 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:20:17.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 -- 172.21.15.43:0/1624750152 shutdown_connections 2024-05-27T07:20:17.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.525+0000 7f50118bfb80 1 -- 172.21.15.43:0/1624750152 wait complete. 2024-05-27T07:20:17.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f50118bfb80 20 client.16410 trim_cache size 0 max 16384 2024-05-27T07:20:17.517 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:20:17.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f5001800700 1 -- 172.21.15.43:0/203217110 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4fe00b29f0 msgr2=0x7f4fe00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f5001800700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4fe00b29f0 0x7f4fe00b4f00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x559e680b9210 tx=0x7f4ff4009920 comp rx=0 tx=0).stop 2024-05-27T07:20:17.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f5001800700 1 -- 172.21.15.43:0/203217110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ffc1052a0 msgr2=0x7f4ffc19cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:17.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f5001800700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ffc1052a0 0x7f4ffc19cd30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00d950 tx=0x7f4fec00dc60 comp rx=0 tx=0).stop 2024-05-27T07:20:17.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f5001800700 1 -- 172.21.15.43:0/203217110 shutdown_connections 2024-05-27T07:20:17.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f5001800700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4fe00b29f0 0x7f4fe00b4f00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f5001800700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ffc10e710 0x7f4ffc1a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f5001800700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ffc1052a0 0x7f4ffc19cd30 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f5001800700 1 --2- 172.21.15.43:0/203217110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc105bf0 0x7f4ffc19d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:17.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.526+0000 7f5001800700 1 -- 172.21.15.43:0/203217110 >> 172.21.15.43:0/203217110 conn(0x7f4ffc078c30 msgr2=0x7f4ffc103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:17.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.527+0000 7f5001800700 1 -- 172.21.15.43:0/203217110 shutdown_connections 2024-05-27T07:20:17.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:17.527+0000 7f5001800700 1 -- 172.21.15.43:0/203217110 wait complete. 2024-05-27T07:20:17.798 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:17 smithi043 ceph-mon[31531]: pgmap v214: 129 pgs: 129 active+clean; 57 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 546 KiB/s wr, 55 op/s 2024-05-27T07:20:18.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:17 smithi132 ceph-mon[35283]: pgmap v214: 129 pgs: 129 active+clean; 57 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 546 KiB/s wr, 55 op/s 2024-05-27T07:20:18.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 31747 0.64 MB/sec execute 142 sec latency 56.396 ms 2024-05-27T07:20:18.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:17 smithi179 ceph-mon[35888]: pgmap v214: 129 pgs: 129 active+clean; 57 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 546 KiB/s wr, 55 op/s 2024-05-27T07:20:18.516 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:20:19.003 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:19.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 31915 0.66 MB/sec execute 143 sec latency 55.769 ms 2024-05-27T07:20:20.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:19 smithi132 ceph-mon[35283]: pgmap v215: 129 pgs: 129 active+clean; 57 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 392 KiB/s wr, 35 op/s 2024-05-27T07:20:20.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 31986 0.65 MB/sec execute 144 sec latency 235.402 ms 2024-05-27T07:20:20.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:19 smithi043 ceph-mon[31531]: pgmap v215: 129 pgs: 129 active+clean; 57 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 392 KiB/s wr, 35 op/s 2024-05-27T07:20:20.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:19 smithi179 ceph-mon[35888]: pgmap v215: 129 pgs: 129 active+clean; 57 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 392 KiB/s wr, 35 op/s 2024-05-27T07:20:21.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32103 0.65 MB/sec execute 145 sec latency 56.303 ms 2024-05-27T07:20:22.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:21 smithi132 ceph-mon[35283]: pgmap v216: 129 pgs: 129 active+clean; 57 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 542 KiB/s wr, 49 op/s 2024-05-27T07:20:22.278 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:21 smithi179 ceph-mon[35888]: pgmap v216: 129 pgs: 129 active+clean; 57 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 542 KiB/s wr, 49 op/s 2024-05-27T07:20:22.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.174+0000 7fc840e00700 1 -- 172.21.15.43:0/1735720977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83c10b890 msgr2=0x7fc83c1131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.174+0000 7fc840e00700 1 --2- 172.21.15.43:0/1735720977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83c10b890 0x7fc83c1131f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc83400b0d0 tx=0x7fc83400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.174+0000 7fc840e00700 1 -- 172.21.15.43:0/1735720977 shutdown_connections 2024-05-27T07:20:22.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.174+0000 7fc840e00700 1 --2- 172.21.15.43:0/1735720977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83c10b890 0x7fc83c1131f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.174+0000 7fc840e00700 1 --2- 172.21.15.43:0/1735720977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc83c106610 0x7fc83c106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.174+0000 7fc840e00700 1 --2- 172.21.15.43:0/1735720977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc83c10af40 0x7fc83c10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.174+0000 7fc840e00700 1 -- 172.21.15.43:0/1735720977 >> 172.21.15.43:0/1735720977 conn(0x7fc83c102020 msgr2=0x7fc83c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:22.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.175+0000 7fc840e00700 1 -- 172.21.15.43:0/1735720977 shutdown_connections 2024-05-27T07:20:22.279 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:21 smithi043 ceph-mon[31531]: pgmap v216: 129 pgs: 129 active+clean; 57 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 542 KiB/s wr, 49 op/s 2024-05-27T07:20:22.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.175+0000 7fc840e00700 1 -- 172.21.15.43:0/1735720977 wait complete. 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.176+0000 7fc840e00700 1 Processor -- start 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.176+0000 7fc840e00700 1 -- start start 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc840e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83c106610 0x7fc83c19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc840e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc83c10af40 0x7fc83c19d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc840e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc83c10b890 0x7fc83c1a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc840e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1163c0 con 0x7fc83c106610 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc840e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c116240 con 0x7fc83c10b890 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc840e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c116540 con 0x7fc83c10af40 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc83b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83c106610 0x7fc83c19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc83b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83c106610 0x7fc83c19cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44738/0 (socket says 172.21.15.43:44738) 2024-05-27T07:20:22.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc83b400700 1 -- 172.21.15.43:0/1088707517 learned_addr learned my addr 172.21.15.43:0/1088707517 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:22.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc83aa00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc83c10af40 0x7fc83c19d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.177+0000 7fc83be00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc83c10b890 0x7fc83c1a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc83b400700 1 -- 172.21.15.43:0/1088707517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc83c10af40 msgr2=0x7fc83c19d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc83b400700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc83c10af40 0x7fc83c19d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc83b400700 1 -- 172.21.15.43:0/1088707517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc83c10b890 msgr2=0x7fc83c1a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc83b400700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc83c10b890 0x7fc83c1a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc83b400700 1 -- 172.21.15.43:0/1088707517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c1a1c00 con 0x7fc83c106610 2024-05-27T07:20:22.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc83be00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc83c10b890 0x7fc83c1a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc83b400700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83c106610 0x7fc83c19cd30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00d950 tx=0x7fc82c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc833e00700 1 -- 172.21.15.43:0/1088707517 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c016070 con 0x7fc83c106610 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc833e00700 1 -- 172.21.15.43:0/1088707517 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc82c009dd0 con 0x7fc83c106610 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc833e00700 1 -- 172.21.15.43:0/1088707517 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c00f3c0 con 0x7fc83c106610 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.178+0000 7fc840e00700 1 -- 172.21.15.43:0/1088707517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83c1a1ef0 con 0x7fc83c106610 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.179+0000 7fc840e00700 1 -- 172.21.15.43:0/1088707517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83c1a9810 con 0x7fc83c106610 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.183+0000 7fc833e00700 1 -- 172.21.15.43:0/1088707517 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc82c019070 con 0x7fc83c106610 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.184+0000 7fc850ee0b80 1 Processor -- start 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.185+0000 7fc850ee0b80 1 -- start start 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.185+0000 7fc833e00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc8200b2a70 0x7fc8200b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.185+0000 7fc833e00700 1 -- 172.21.15.43:0/1088707517 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fc82c0e25b0 con 0x7fc83c106610 2024-05-27T07:20:22.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.185+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8ab750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.185+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef72be40 0x5607ef8abc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.185+0000 7fc83be00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc8200b2a70 0x7fc8200b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.185+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.185+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5607ef7d8e40 con 0x5607ef8b52c0 2024-05-27T07:20:22.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.185+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5607ef7c89e0 con 0x5607ef72be40 2024-05-27T07:20:22.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.185+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5607ef7cca20 con 0x5607ef8179f0 2024-05-27T07:20:22.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.186+0000 7fc83be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.186+0000 7fc83be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8b76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44748/0 (socket says 172.21.15.43:44748) 2024-05-27T07:20:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.186+0000 7fc83be00700 1 -- 172.21.15.43:0/149610364 learned_addr learned my addr 172.21.15.43:0/149610364 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.186+0000 7fc83aa00700 1 --2- 172.21.15.43:0/149610364 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef72be40 0x5607ef8abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.186+0000 7fc83b400700 1 --2- 172.21.15.43:0/149610364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8ab750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.186+0000 7fc83be00700 1 -- 172.21.15.43:0/149610364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 msgr2=0x5607ef8ab750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.186+0000 7fc83be00700 1 --2- 172.21.15.43:0/149610364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8ab750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.186+0000 7fc83be00700 1 -- 172.21.15.43:0/149610364 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef72be40 msgr2=0x5607ef8abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.186+0000 7fc83be00700 1 --2- 172.21.15.43:0/149610364 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef72be40 0x5607ef8abc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.186+0000 7fc83be00700 1 -- 172.21.15.43:0/149610364 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607ef791420 con 0x5607ef8b52c0 2024-05-27T07:20:22.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.187+0000 7fc83be00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc8200b2a70 0x7fc8200b4f80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc834015000 tx=0x7fc834009310 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:22.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.187+0000 7fc83be00700 1 --2- 172.21.15.43:0/149610364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8b76e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc834009900 tx=0x7fc834027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58decfce34c01ef3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:22.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.187+0000 7fc830c00700 1 -- 172.21.15.43:0/149610364 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83402f070 con 0x5607ef8b52c0 2024-05-27T07:20:22.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.187+0000 7fc830c00700 1 -- 172.21.15.43:0/149610364 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc834005050 con 0x5607ef8b52c0 2024-05-27T07:20:22.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.187+0000 7fc830c00700 1 -- 172.21.15.43:0/149610364 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83401f6d0 con 0x5607ef8b52c0 2024-05-27T07:20:22.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.188+0000 7fc850ee0b80 1 -- 172.21.15.43:0/149610364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 msgr2=0x5607ef8b76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.188+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/149610364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8b76e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc834009900 tx=0x7fc834027000 comp rx=0 tx=0).stop 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.188+0000 7fc850ee0b80 1 -- 172.21.15.43:0/149610364 shutdown_connections 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.188+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/149610364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8b76e0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.188+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/149610364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8ab750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.188+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/149610364 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef72be40 0x5607ef8abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.188+0000 7fc850ee0b80 1 -- 172.21.15.43:0/149610364 >> 172.21.15.43:0/149610364 conn(0x5607ef7c5390 msgr2=0x5607ef8a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.189+0000 7fc850ee0b80 1 -- 172.21.15.43:0/149610364 shutdown_connections 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.189+0000 7fc850ee0b80 1 -- 172.21.15.43:0/149610364 wait complete. 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.189+0000 7fc850ee0b80 1 -- start start 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.190+0000 7fc850ee0b80 1 Processor -- start 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.190+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.190+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.190+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 0x5607ef8b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.190+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5607ef6cefa0 con 0x5607ef72be40 2024-05-27T07:20:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.190+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5607ef769760 con 0x5607ef8b52c0 2024-05-27T07:20:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.190+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5607ef8b7da0 con 0x5607ef8179f0 2024-05-27T07:20:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc850ee0b80 10 client.0 _ll_register_callbacks cb 0x5607ef772c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8b22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44756/0 (socket says 172.21.15.43:44756) 2024-05-27T07:20:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83b400700 1 -- 172.21.15.43:0/1680890346 learned_addr learned my addr 172.21.15.43:0/1680890346 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83aa00700 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83be00700 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 0x5607ef8b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83b400700 1 -- 172.21.15.43:0/1680890346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 msgr2=0x5607ef8b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83b400700 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8b27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83b400700 1 -- 172.21.15.43:0/1680890346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 msgr2=0x5607ef8b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83b400700 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 0x5607ef8b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83b400700 1 -- 172.21.15.43:0/1680890346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607ef7188d0 con 0x5607ef72be40 2024-05-27T07:20:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.191+0000 7fc83be00700 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 0x5607ef8b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.192+0000 7fc83b400700 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8b22a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0df8f0 tx=0x7fc82c0108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff38e686646d4e4f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.192+0000 7fc825600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.192+0000 7fc825600700 1 -- 172.21.15.43:0/1680890346 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c020030 con 0x5607ef72be40 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.192+0000 7fc825600700 1 -- 172.21.15.43:0/1680890346 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fc82c0e2390 con 0x5607ef72be40 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.192+0000 7fc850ee0b80 1 -- 172.21.15.43:0/1680890346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc834005050 con 0x5607ef72be40 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.192+0000 7fc825600700 1 -- 172.21.15.43:0/1680890346 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fc82c0b2340 con 0x5607ef72be40 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.193+0000 7fc850ee0b80 10 client.16422 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.193+0000 7fc850ee0b80 1 -- 172.21.15.43:0/1680890346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x5607ef799060 con 0x5607ef72be40 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.193+0000 7fc825600700 1 -- 172.21.15.43:0/1680890346 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc834005050 con 0x5607ef72be40 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.193+0000 7fc825600700 1 -- 172.21.15.43:0/1680890346 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fc82c0b1f40 con 0x5607ef72be40 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.193+0000 7fc850ee0b80 10 client.16422 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.193+0000 7fc850ee0b80 10 client.16422 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.193+0000 7fc850ee0b80 20 client.16422 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.194+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5607ef8c5160 0x5607ef8c7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.194+0000 7fc850ee0b80 4 client.16422 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.194+0000 7fc850ee0b80 1 -- 172.21.15.43:0/1680890346 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607ef7dad90 con 0x5607ef8c5160 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.194+0000 7fc83aa00700 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5607ef8c5160 0x5607ef8c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.195+0000 7fc83aa00700 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5607ef8c5160 0x5607ef8c7610 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=977a1e5f46da0d07 server_cookie=f7ac1ebd2b7fab77 in_seq=0 out_seq=0 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.195+0000 7fc825600700 10 client.16422 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc825600700 1 -- 172.21.15.43:0/1680890346 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5607ef7dad90 con 0x5607ef8c5160 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc825600700 10 client.16422 handle_command_reply: tid=0 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc850ee0b80 1 client.16422 shutdown 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc850ee0b80 1 -- 172.21.15.43:0/1680890346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 msgr2=0x5607ef8b22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8b22a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0df8f0 tx=0x7fc82c0108d0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc850ee0b80 1 -- 172.21.15.43:0/1680890346 shutdown_connections 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5607ef8c5160 0x5607ef8c7610 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 0x5607ef8b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc825600700 0 client.16422 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:20:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8b22a0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/1680890346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc850ee0b80 1 -- 172.21.15.43:0/1680890346 >> 172.21.15.43:0/1680890346 conn(0x5607ef7c5390 msgr2=0x5607ef8ab7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.198+0000 7fc850ee0b80 1 -- 172.21.15.43:0/1680890346 shutdown_connections 2024-05-27T07:20:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.199+0000 7fc850ee0b80 1 -- 172.21.15.43:0/1680890346 wait complete. 2024-05-27T07:20:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.199+0000 7fc850ee0b80 20 client.16422 trim_cache size 0 max 16384 2024-05-27T07:20:22.297 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32205 0.65 MB/sec execute 146 sec latency 52.337 ms 2024-05-27T07:20:22.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.216+0000 7fc850ee0b80 1 Processor -- start 2024-05-27T07:20:22.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.216+0000 7fc850ee0b80 1 -- start start 2024-05-27T07:20:22.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef72be40 0x5607ef8a6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8179f0 0x5607ef8ad950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8c0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1b3030 con 0x5607ef8b52c0 2024-05-27T07:20:22.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1b3700 con 0x5607ef8179f0 2024-05-27T07:20:22.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1b0e20 con 0x5607ef72be40 2024-05-27T07:20:22.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc83be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8c0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc83be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8c0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44758/0 (socket says 172.21.15.43:44758) 2024-05-27T07:20:22.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc83be00700 1 -- 172.21.15.43:0/2426164856 learned_addr learned my addr 172.21.15.43:0/2426164856 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:22.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc83b400700 1 --2- 172.21.15.43:0/2426164856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef72be40 0x5607ef8a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.217+0000 7fc83aa00700 1 --2- 172.21.15.43:0/2426164856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8179f0 0x5607ef8ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.218+0000 7fc83be00700 1 -- 172.21.15.43:0/2426164856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef72be40 msgr2=0x5607ef8a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:22.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.218+0000 7fc83be00700 1 --2- 172.21.15.43:0/2426164856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef72be40 0x5607ef8a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.218+0000 7fc83be00700 1 -- 172.21.15.43:0/2426164856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8179f0 msgr2=0x5607ef8ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:22.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.218+0000 7fc83be00700 1 --2- 172.21.15.43:0/2426164856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8179f0 0x5607ef8ad950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.218+0000 7fc83be00700 1 -- 172.21.15.43:0/2426164856 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607ef8ade90 con 0x5607ef8b52c0 2024-05-27T07:20:22.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.218+0000 7fc83be00700 1 --2- 172.21.15.43:0/2426164856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8c0e40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc83402d040 tx=0x7fc834007ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f37be8e2ca072c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:22.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.218+0000 7fc827e00700 1 -- 172.21.15.43:0/2426164856 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83402f050 con 0x5607ef8b52c0 2024-05-27T07:20:22.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.218+0000 7fc827e00700 1 -- 172.21.15.43:0/2426164856 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc83403b070 con 0x5607ef8b52c0 2024-05-27T07:20:22.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.218+0000 7fc827e00700 1 -- 172.21.15.43:0/2426164856 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8340058a0 con 0x5607ef8b52c0 2024-05-27T07:20:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.219+0000 7fc850ee0b80 1 -- 172.21.15.43:0/2426164856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 msgr2=0x5607ef8c0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.219+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/2426164856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8c0e40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc83402d040 tx=0x7fc834007ab0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.219+0000 7fc850ee0b80 1 -- 172.21.15.43:0/2426164856 shutdown_connections 2024-05-27T07:20:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.219+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/2426164856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef8b52c0 0x5607ef8c0e40 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.219+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/2426164856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef72be40 0x5607ef8a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.219+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/2426164856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8179f0 0x5607ef8ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.219+0000 7fc850ee0b80 1 -- 172.21.15.43:0/2426164856 >> 172.21.15.43:0/2426164856 conn(0x5607ef7c5390 msgr2=0x5607ef8ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.220+0000 7fc850ee0b80 1 -- 172.21.15.43:0/2426164856 shutdown_connections 2024-05-27T07:20:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.220+0000 7fc850ee0b80 1 -- 172.21.15.43:0/2426164856 wait complete. 2024-05-27T07:20:22.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.220+0000 7fc850ee0b80 1 -- start start 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.220+0000 7fc850ee0b80 1 Processor -- start 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.221+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8bd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.221+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.221+0000 7fc850ee0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 0x5607ef8afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.221+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1b14f0 con 0x5607ef72be40 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.221+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1b2960 con 0x5607ef8b52c0 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.221+0000 7fc850ee0b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1b1bc0 con 0x5607ef8179f0 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.221+0000 7fc850ee0b80 10 client.0 _ll_register_callbacks cb 0x5607ef772c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.221+0000 7fc83b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8bd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.221+0000 7fc83b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8bd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44768/0 (socket says 172.21.15.43:44768) 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.221+0000 7fc83b400700 1 -- 172.21.15.43:0/391946829 learned_addr learned my addr 172.21.15.43:0/391946829 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc83aa00700 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc83be00700 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 0x5607ef8afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc83b400700 1 -- 172.21.15.43:0/391946829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 msgr2=0x5607ef8af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:22.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc83b400700 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc83b400700 1 -- 172.21.15.43:0/391946829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 msgr2=0x5607ef8afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:22.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc83b400700 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 0x5607ef8afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc83b400700 1 -- 172.21.15.43:0/391946829 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607ef8b0140 con 0x5607ef72be40 2024-05-27T07:20:22.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc83aa00700 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:22.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc83be00700 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 0x5607ef8afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:22.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc83b400700 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8bd4f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0a6750 tx=0x7fc82c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f172b1597e1126 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:22.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc825600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:22.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.222+0000 7fc825600700 1 -- 172.21.15.43:0/391946829 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c020070 con 0x5607ef72be40 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.223+0000 7fc825600700 1 -- 172.21.15.43:0/391946829 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7fc82c0a5ad0 con 0x5607ef72be40 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.223+0000 7fc850ee0b80 1 -- 172.21.15.43:0/391946829 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc83403b070 con 0x5607ef72be40 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.223+0000 7fc825600700 1 -- 172.21.15.43:0/391946829 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fc82c08c050 con 0x5607ef72be40 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.223+0000 7fc850ee0b80 10 client.16428 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.223+0000 7fc850ee0b80 1 -- 172.21.15.43:0/391946829 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x5607ef92eca0 con 0x5607ef72be40 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.223+0000 7fc825600700 1 -- 172.21.15.43:0/391946829 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc83403b070 con 0x5607ef72be40 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.223+0000 7fc825600700 1 -- 172.21.15.43:0/391946829 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fc82c0a9740 con 0x5607ef72be40 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.224+0000 7fc850ee0b80 10 client.16428 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.224+0000 7fc850ee0b80 10 client.16428 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.224+0000 7fc850ee0b80 20 client.16428 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.224+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5607ef92f060 0x5607ef931480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.224+0000 7fc850ee0b80 4 client.16428 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:20:22.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.224+0000 7fc850ee0b80 1 -- 172.21.15.43:0/391946829 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5607ef7d4570 con 0x5607ef92f060 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.224+0000 7fc83aa00700 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5607ef92f060 0x5607ef931480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.225+0000 7fc83aa00700 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5607ef92f060 0x5607ef931480 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fe2839d28d3b86d5 server_cookie=ee1a94de6de0aa30 in_seq=0 out_seq=0 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.225+0000 7fc825600700 10 client.16428 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc825600700 1 -- 172.21.15.43:0/391946829 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5607ef7d4570 con 0x5607ef92f060 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc825600700 10 client.16428 handle_command_reply: tid=0 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc850ee0b80 1 client.16428 shutdown 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc850ee0b80 1 -- 172.21.15.43:0/391946829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 msgr2=0x5607ef8bd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8bd4f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0a6750 tx=0x7fc82c000bc0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc850ee0b80 1 -- 172.21.15.43:0/391946829 shutdown_connections 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5607ef92f060 0x5607ef931480 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607ef8b52c0 0x5607ef8afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5607ef72be40 0x5607ef8bd4f0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc850ee0b80 1 --2- 172.21.15.43:0/391946829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ef8179f0 0x5607ef8af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc825600700 0 client.16428 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:20:22.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.226+0000 7fc850ee0b80 1 -- 172.21.15.43:0/391946829 >> 172.21.15.43:0/391946829 conn(0x5607ef7c5390 msgr2=0x5607ef8aa120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:22.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.227+0000 7fc850ee0b80 1 -- 172.21.15.43:0/391946829 shutdown_connections 2024-05-27T07:20:22.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.227+0000 7fc850ee0b80 1 -- 172.21.15.43:0/391946829 wait complete. 2024-05-27T07:20:22.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.227+0000 7fc850ee0b80 20 client.16428 trim_cache size 0 max 16384 2024-05-27T07:20:22.311 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:20:22.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 -- 172.21.15.43:0/1088707517 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc8200b2a70 msgr2=0x7fc8200b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc8200b2a70 0x7fc8200b4f80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc834015000 tx=0x7fc834009310 comp rx=0 tx=0).stop 2024-05-27T07:20:22.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 -- 172.21.15.43:0/1088707517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83c106610 msgr2=0x7fc83c19cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:22.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83c106610 0x7fc83c19cd30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00d950 tx=0x7fc82c00dc60 comp rx=0 tx=0).stop 2024-05-27T07:20:22.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 -- 172.21.15.43:0/1088707517 shutdown_connections 2024-05-27T07:20:22.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc8200b2a70 0x7fc8200b4f80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc83c10b890 0x7fc83c1a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc83c106610 0x7fc83c19cd30 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 --2- 172.21.15.43:0/1088707517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc83c10af40 0x7fc83c19d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:22.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 -- 172.21.15.43:0/1088707517 >> 172.21.15.43:0/1088707517 conn(0x7fc83c102020 msgr2=0x7fc83c10a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:22.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 -- 172.21.15.43:0/1088707517 shutdown_connections 2024-05-27T07:20:22.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:22.228+0000 7fc840e00700 1 -- 172.21.15.43:0/1088707517 wait complete. 2024-05-27T07:20:23.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32262 0.65 MB/sec execute 147 sec latency 45.508 ms 2024-05-27T07:20:23.215 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:20:23.713 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:24.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:23 smithi132 ceph-mon[35283]: pgmap v217: 129 pgs: 129 active+clean; 57 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 411 KiB/s wr, 34 op/s 2024-05-27T07:20:24.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:23 smithi043 ceph-mon[31531]: pgmap v217: 129 pgs: 129 active+clean; 57 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 411 KiB/s wr, 34 op/s 2024-05-27T07:20:24.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32338 0.65 MB/sec execute 148 sec latency 55.478 ms 2024-05-27T07:20:24.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:23 smithi179 ceph-mon[35888]: pgmap v217: 129 pgs: 129 active+clean; 57 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 411 KiB/s wr, 34 op/s 2024-05-27T07:20:25.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:24 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:25.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32413 0.64 MB/sec execute 149 sec latency 40.260 ms 2024-05-27T07:20:25.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:24 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:25.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:24 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:20:26.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:25 smithi132 ceph-mon[35283]: pgmap v218: 129 pgs: 129 active+clean; 57 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 411 KiB/s wr, 34 op/s 2024-05-27T07:20:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.181+0000 7f94abe00700 1 -- 172.21.15.43:0/1784188828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94ac109690 msgr2=0x7f94ac110ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.181+0000 7f94abe00700 1 --2- 172.21.15.43:0/1784188828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94ac109690 0x7f94ac110ff0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f94a400b0d0 tx=0x7f94a400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.181+0000 7f94abe00700 1 -- 172.21.15.43:0/1784188828 shutdown_connections 2024-05-27T07:20:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.181+0000 7f94abe00700 1 --2- 172.21.15.43:0/1784188828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94ac109690 0x7f94ac110ff0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.181+0000 7f94abe00700 1 --2- 172.21.15.43:0/1784188828 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94ac104410 0x7f94ac104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.181+0000 7f94abe00700 1 --2- 172.21.15.43:0/1784188828 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94ac108d40 0x7f94ac109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.181+0000 7f94abe00700 1 -- 172.21.15.43:0/1784188828 >> 172.21.15.43:0/1784188828 conn(0x7f94ac07aca0 msgr2=0x7f94ac07b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:26.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.181+0000 7f94abe00700 1 -- 172.21.15.43:0/1784188828 shutdown_connections 2024-05-27T07:20:26.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.181+0000 7f94abe00700 1 -- 172.21.15.43:0/1784188828 wait complete. 2024-05-27T07:20:26.173 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:25 smithi043 ceph-mon[31531]: pgmap v218: 129 pgs: 129 active+clean; 57 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 411 KiB/s wr, 34 op/s 2024-05-27T07:20:26.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.183+0000 7f94abe00700 1 Processor -- start 2024-05-27T07:20:26.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.183+0000 7f94abe00700 1 -- start start 2024-05-27T07:20:26.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94abe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94ac104410 0x7f94ac1a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94abe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94ac108d40 0x7f94ac1a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94abe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94ac109690 0x7f94ac19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94abe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1141c0 con 0x7f94ac108d40 2024-05-27T07:20:26.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94abe00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac114040 con 0x7f94ac109690 2024-05-27T07:20:26.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94abe00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac114340 con 0x7f94ac104410 2024-05-27T07:20:26.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94aa000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94ac108d40 0x7f94ac1a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94aa000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94ac108d40 0x7f94ac1a0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59798/0 (socket says 172.21.15.43:59798) 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94aa000700 1 -- 172.21.15.43:0/2036848439 learned_addr learned my addr 172.21.15.43:0/2036848439 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94ab400700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94ac109690 0x7f94ac19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.184+0000 7f94aaa00700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94ac104410 0x7f94ac1a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.185+0000 7f94aa000700 1 -- 172.21.15.43:0/2036848439 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94ac104410 msgr2=0x7f94ac1a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.185+0000 7f94aa000700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94ac104410 0x7f94ac1a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.185+0000 7f94aa000700 1 -- 172.21.15.43:0/2036848439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94ac109690 msgr2=0x7f94ac19a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.185+0000 7f94aa000700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94ac109690 0x7f94ac19a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.185+0000 7f94aa000700 1 -- 172.21.15.43:0/2036848439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94ac19b090 con 0x7f94ac108d40 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.185+0000 7f94aa000700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94ac108d40 0x7f94ac1a0cb0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f949800d980 tx=0x7f949800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.185+0000 7f94a3400700 1 -- 172.21.15.43:0/2036848439 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498017070 con 0x7f94ac108d40 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.185+0000 7f94abe00700 1 -- 172.21.15.43:0/2036848439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94ac19b380 con 0x7f94ac108d40 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.185+0000 7f94a3400700 1 -- 172.21.15.43:0/2036848439 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9498009dc0 con 0x7f94ac108d40 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.185+0000 7f94a3400700 1 -- 172.21.15.43:0/2036848439 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949800f3c0 con 0x7f94ac108d40 2024-05-27T07:20:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.186+0000 7f94abe00700 1 -- 172.21.15.43:0/2036848439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94ac1a7600 con 0x7f94ac108d40 2024-05-27T07:20:26.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.190+0000 7f94a3400700 1 -- 172.21.15.43:0/2036848439 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9498009930 con 0x7f94ac108d40 2024-05-27T07:20:26.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.190+0000 7f94c076bb80 1 Processor -- start 2024-05-27T07:20:26.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.190+0000 7f94c076bb80 1 -- start start 2024-05-27T07:20:26.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965410e40 0x5629655932e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593820 0x56296559dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x5629655a0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94c076bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5629654bde40 con 0x562965593c30 2024-05-27T07:20:26.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94c076bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5629654ad9e0 con 0x562965410e40 2024-05-27T07:20:26.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94c076bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5629654b1a20 con 0x562965593820 2024-05-27T07:20:26.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94aa000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x5629655a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94ab400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965410e40 0x5629655932e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94aa000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x5629655a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59808/0 (socket says 172.21.15.43:59808) 2024-05-27T07:20:26.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94aa000700 1 -- 172.21.15.43:0/420181475 learned_addr learned my addr 172.21.15.43:0/420181475 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:26.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.191+0000 7f94aaa00700 1 --2- 172.21.15.43:0/420181475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593820 0x56296559dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.192+0000 7f94aa000700 1 -- 172.21.15.43:0/420181475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593820 msgr2=0x56296559dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.192+0000 7f94aa000700 1 --2- 172.21.15.43:0/420181475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593820 0x56296559dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.192+0000 7f94aa000700 1 -- 172.21.15.43:0/420181475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965410e40 msgr2=0x5629655932e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.192+0000 7f94aa000700 1 --2- 172.21.15.43:0/420181475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965410e40 0x5629655932e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.192+0000 7f94a3400700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94900b29f0 0x7f94900b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.192+0000 7f94aa000700 1 -- 172.21.15.43:0/420181475 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562965476420 con 0x562965593c30 2024-05-27T07:20:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.192+0000 7f94a3400700 1 -- 172.21.15.43:0/2036848439 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f94980e2480 con 0x7f94ac108d40 2024-05-27T07:20:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.192+0000 7f94ab400700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94900b29f0 0x7f94900b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.192+0000 7f94aa000700 1 --2- 172.21.15.43:0/420181475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x5629655a0430 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f94980df760 tx=0x7f94980a5a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fdcfc91cb44016 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:26.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.193+0000 7f9497e00700 1 -- 172.21.15.43:0/420181475 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498020030 con 0x562965593c30 2024-05-27T07:20:26.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.193+0000 7f9497e00700 1 -- 172.21.15.43:0/420181475 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94980a8ec0 con 0x562965593c30 2024-05-27T07:20:26.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.193+0000 7f9497e00700 1 -- 172.21.15.43:0/420181475 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94980a7ec0 con 0x562965593c30 2024-05-27T07:20:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.194+0000 7f94c076bb80 1 -- 172.21.15.43:0/420181475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 msgr2=0x5629655a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.194+0000 7f94c076bb80 1 --2- 172.21.15.43:0/420181475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x5629655a0430 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f94980df760 tx=0x7f94980a5a00 comp rx=0 tx=0).stop 2024-05-27T07:20:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.194+0000 7f94ab400700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94900b29f0 0x7f94900b4f00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009900 tx=0x7f94a400b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.194+0000 7f94c076bb80 1 -- 172.21.15.43:0/420181475 shutdown_connections 2024-05-27T07:20:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.194+0000 7f94c076bb80 1 --2- 172.21.15.43:0/420181475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x5629655a0430 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.194+0000 7f94c076bb80 1 --2- 172.21.15.43:0/420181475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965410e40 0x5629655932e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.194+0000 7f94c076bb80 1 --2- 172.21.15.43:0/420181475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593820 0x56296559dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.194+0000 7f94c076bb80 1 -- 172.21.15.43:0/420181475 >> 172.21.15.43:0/420181475 conn(0x562965470fc0 msgr2=0x56296558ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.194+0000 7f94c076bb80 1 -- 172.21.15.43:0/420181475 shutdown_connections 2024-05-27T07:20:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.194+0000 7f94c076bb80 1 -- 172.21.15.43:0/420181475 wait complete. 2024-05-27T07:20:26.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.195+0000 7f94c076bb80 1 -- start start 2024-05-27T07:20:26.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.195+0000 7f94c076bb80 1 Processor -- start 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965410e40 0x56296559d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593820 0x56296559dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 0x56296559e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94c076bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5629653b3fa0 con 0x562965593820 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94c076bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56296544e760 con 0x562965410e40 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94c076bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5629655a0d40 con 0x562965593c30 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94c076bb80 10 client.0 _ll_register_callbacks cb 0x562965457c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94aa000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593820 0x56296559dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94aa000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593820 0x56296559dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59822/0 (socket says 172.21.15.43:59822) 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94aa000700 1 -- 172.21.15.43:0/2141709932 learned_addr learned my addr 172.21.15.43:0/2141709932 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94aaa00700 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965410e40 0x56296559d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94ab400700 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 0x56296559e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94aa000700 1 -- 172.21.15.43:0/2141709932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 msgr2=0x56296559e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:26.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94aa000700 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 0x56296559e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94aa000700 1 -- 172.21.15.43:0/2141709932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965410e40 msgr2=0x56296559d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:26.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94aa000700 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965410e40 0x56296559d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94aa000700 1 -- 172.21.15.43:0/2141709932 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629653fd8d0 con 0x562965593820 2024-05-27T07:20:26.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.196+0000 7f94ab400700 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 0x56296559e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:20:26.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.197+0000 7f94aa000700 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593820 0x56296559dec0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f94980910f0 tx=0x7f9498010950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f3c8dd3843de4c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:26.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.197+0000 7f9494c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:26.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.197+0000 7f9494c00700 1 -- 172.21.15.43:0/2141709932 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498020070 con 0x562965593820 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.197+0000 7f9494c00700 1 -- 172.21.15.43:0/2141709932 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f9498010680 con 0x562965593820 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.197+0000 7f94c076bb80 1 -- 172.21.15.43:0/2141709932 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94980a8ec0 con 0x562965593820 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.197+0000 7f9494c00700 1 -- 172.21.15.43:0/2141709932 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f9498009c20 con 0x562965593820 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.197+0000 7f94c076bb80 10 client.16440 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.197+0000 7f94c076bb80 1 -- 172.21.15.43:0/2141709932 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x56296547e060 con 0x562965593820 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.198+0000 7f9494c00700 1 -- 172.21.15.43:0/2141709932 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94980a8ec0 con 0x562965593820 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.198+0000 7f9494c00700 1 -- 172.21.15.43:0/2141709932 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f94980dfaa0 con 0x562965593820 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.198+0000 7f94c076bb80 10 client.16440 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.198+0000 7f94c076bb80 10 client.16440 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.198+0000 7f94c076bb80 20 client.16440 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.198+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5629655adbe0 0x5629655b0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.198+0000 7f94c076bb80 4 client.16440 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:20:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.198+0000 7f94c076bb80 1 -- 172.21.15.43:0/2141709932 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629654bfd90 con 0x5629655adbe0 2024-05-27T07:20:26.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.199+0000 7f94aaa00700 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5629655adbe0 0x5629655b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.199+0000 7f9494c00700 10 client.16440 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:20:26.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.199+0000 7f94aaa00700 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5629655adbe0 0x5629655b0030 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=45bd49a705690d68 server_cookie=6fdf369a8f789328 in_seq=0 out_seq=0 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.200+0000 7f9494c00700 1 -- 172.21.15.43:0/2141709932 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5629654bfd90 con 0x5629655adbe0 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.200+0000 7f9494c00700 10 client.16440 handle_command_reply: tid=0 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.200+0000 7f94c076bb80 1 client.16440 shutdown 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.200+0000 7f94c076bb80 1 -- 172.21.15.43:0/2141709932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593820 msgr2=0x56296559dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.200+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593820 0x56296559dec0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f94980910f0 tx=0x7f9498010950 comp rx=0 tx=0).stop 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.201+0000 7f94c076bb80 1 -- 172.21.15.43:0/2141709932 shutdown_connections 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.201+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5629655adbe0 0x5629655b0030 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.201+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 0x56296559e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.201+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965410e40 0x56296559d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.201+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2141709932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593820 0x56296559dec0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.201+0000 7f9494c00700 0 client.16440 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.201+0000 7f94c076bb80 1 -- 172.21.15.43:0/2141709932 >> 172.21.15.43:0/2141709932 conn(0x562965470fc0 msgr2=0x562965589ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.201+0000 7f94c076bb80 1 -- 172.21.15.43:0/2141709932 shutdown_connections 2024-05-27T07:20:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.201+0000 7f94c076bb80 1 -- 172.21.15.43:0/2141709932 wait complete. 2024-05-27T07:20:26.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.201+0000 7f94c076bb80 20 client.16440 trim_cache size 0 max 16384 2024-05-27T07:20:26.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32509 0.64 MB/sec execute 150 sec latency 43.253 ms 2024-05-27T07:20:26.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:25 smithi179 ceph-mon[35888]: pgmap v218: 129 pgs: 129 active+clean; 57 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 411 KiB/s wr, 34 op/s 2024-05-27T07:20:26.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.219+0000 7f94c076bb80 1 Processor -- start 2024-05-27T07:20:26.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.219+0000 7f94c076bb80 1 -- start start 2024-05-27T07:20:26.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.219+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965410e40 0x562965597bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.219+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 0x56296558ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.219+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x56296559cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.219+0000 7f94c076bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1aed70 con 0x562965593c30 2024-05-27T07:20:26.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.219+0000 7f94c076bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1af440 con 0x562965593820 2024-05-27T07:20:26.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.219+0000 7f94c076bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1acb60 con 0x562965410e40 2024-05-27T07:20:26.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94ab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x56296559cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94ab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x56296559cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59834/0 (socket says 172.21.15.43:59834) 2024-05-27T07:20:26.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94ab400700 1 -- 172.21.15.43:0/4092039025 learned_addr learned my addr 172.21.15.43:0/4092039025 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:26.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94aa000700 1 --2- 172.21.15.43:0/4092039025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 0x56296558ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94aaa00700 1 --2- 172.21.15.43:0/4092039025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965410e40 0x562965597bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94ab400700 1 -- 172.21.15.43:0/4092039025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965410e40 msgr2=0x562965597bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94ab400700 1 --2- 172.21.15.43:0/4092039025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965410e40 0x562965597bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94ab400700 1 -- 172.21.15.43:0/4092039025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 msgr2=0x56296558ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94ab400700 1 --2- 172.21.15.43:0/4092039025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 0x56296558ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94ab400700 1 -- 172.21.15.43:0/4092039025 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56296558c350 con 0x562965593c30 2024-05-27T07:20:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.220+0000 7f94aa000700 1 --2- 172.21.15.43:0/4092039025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 0x56296558ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:20:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.221+0000 7f94ab400700 1 --2- 172.21.15.43:0/4092039025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x56296559cf00 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f94a4015000 tx=0x7f94a4021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16654b25cebc839c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.221+0000 7f9497400700 1 -- 172.21.15.43:0/4092039025 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a4037070 con 0x562965593c30 2024-05-27T07:20:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.221+0000 7f9497400700 1 -- 172.21.15.43:0/4092039025 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94a4009410 con 0x562965593c30 2024-05-27T07:20:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.222+0000 7f9497400700 1 -- 172.21.15.43:0/4092039025 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a4032a30 con 0x562965593c30 2024-05-27T07:20:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.222+0000 7f94c076bb80 1 -- 172.21.15.43:0/4092039025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 msgr2=0x56296559cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.222+0000 7f94c076bb80 1 --2- 172.21.15.43:0/4092039025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x56296559cf00 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f94a4015000 tx=0x7f94a4021000 comp rx=0 tx=0).stop 2024-05-27T07:20:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.222+0000 7f94c076bb80 1 -- 172.21.15.43:0/4092039025 shutdown_connections 2024-05-27T07:20:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.222+0000 7f94c076bb80 1 --2- 172.21.15.43:0/4092039025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965593c30 0x56296559cf00 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.222+0000 7f94c076bb80 1 --2- 172.21.15.43:0/4092039025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965410e40 0x562965597bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.222+0000 7f94c076bb80 1 --2- 172.21.15.43:0/4092039025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 0x56296558ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.222+0000 7f94c076bb80 1 -- 172.21.15.43:0/4092039025 >> 172.21.15.43:0/4092039025 conn(0x562965470fc0 msgr2=0x56296559eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:26.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.222+0000 7f94c076bb80 1 -- 172.21.15.43:0/4092039025 shutdown_connections 2024-05-27T07:20:26.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.222+0000 7f94c076bb80 1 -- 172.21.15.43:0/4092039025 wait complete. 2024-05-27T07:20:26.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.223+0000 7f94c076bb80 1 -- start start 2024-05-27T07:20:26.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.223+0000 7f94c076bb80 1 Processor -- start 2024-05-27T07:20:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965410e40 0x5629655a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 0x5629655a1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94c076bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 0x5629655a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94c076bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1ad230 con 0x562965410e40 2024-05-27T07:20:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94c076bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1ae6a0 con 0x562965593820 2024-05-27T07:20:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94c076bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1ad900 con 0x562965593c30 2024-05-27T07:20:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94c076bb80 10 client.0 _ll_register_callbacks cb 0x562965457c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94aaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965410e40 0x5629655a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94aaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965410e40 0x5629655a0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59838/0 (socket says 172.21.15.43:59838) 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94aaa00700 1 -- 172.21.15.43:0/2434676835 learned_addr learned my addr 172.21.15.43:0/2434676835 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94aa000700 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 0x5629655a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.224+0000 7f94ab400700 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 0x5629655a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.225+0000 7f94aaa00700 1 -- 172.21.15.43:0/2434676835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 msgr2=0x5629655a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.225+0000 7f94aaa00700 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 0x5629655a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.225+0000 7f94aaa00700 1 -- 172.21.15.43:0/2434676835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 msgr2=0x5629655a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.225+0000 7f94aaa00700 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 0x5629655a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.225+0000 7f94aaa00700 1 -- 172.21.15.43:0/2434676835 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629656168a0 con 0x562965410e40 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.225+0000 7f94aaa00700 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965410e40 0x5629655a0fd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x562965594840 tx=0x7f949c012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32eafe41f809fc9b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.225+0000 7f9494c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.225+0000 7f9494c00700 1 -- 172.21.15.43:0/2434676835 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c01e7c0 con 0x562965410e40 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.225+0000 7f9494c00700 1 -- 172.21.15.43:0/2434676835 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f949c01e960 con 0x562965410e40 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.225+0000 7f94c076bb80 1 -- 172.21.15.43:0/2434676835 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94a4009410 con 0x562965410e40 2024-05-27T07:20:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f9494c00700 1 -- 172.21.15.43:0/2434676835 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f949c02e050 con 0x562965410e40 2024-05-27T07:20:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f94c076bb80 10 client.16446 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f94c076bb80 1 -- 172.21.15.43:0/2434676835 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x562965617b60 con 0x562965410e40 2024-05-27T07:20:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f9494c00700 1 -- 172.21.15.43:0/2434676835 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94a4009410 con 0x562965410e40 2024-05-27T07:20:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f9494c00700 1 -- 172.21.15.43:0/2434676835 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f949c011d20 con 0x562965410e40 2024-05-27T07:20:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f94c076bb80 10 client.16446 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f94c076bb80 10 client.16446 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:20:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f94c076bb80 20 client.16446 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x562965618270 0x56296561a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f94c076bb80 4 client.16446 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:20:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.226+0000 7f94c076bb80 1 -- 172.21.15.43:0/2434676835 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5629654b9570 con 0x562965618270 2024-05-27T07:20:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.227+0000 7f94aa000700 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x562965618270 0x56296561a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.228+0000 7f94aa000700 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x562965618270 0x56296561a690 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4fcf6e9a37c9904 server_cookie=cba9b836b85684a6 in_seq=0 out_seq=0 2024-05-27T07:20:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.228+0000 7f9494c00700 10 client.16446 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:20:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f9494c00700 1 -- 172.21.15.43:0/2434676835 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5629654b9570 con 0x562965618270 2024-05-27T07:20:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f9494c00700 10 client.16446 handle_command_reply: tid=0 2024-05-27T07:20:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f94c076bb80 1 client.16446 shutdown 2024-05-27T07:20:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f94c076bb80 1 -- 172.21.15.43:0/2434676835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965410e40 msgr2=0x5629655a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965410e40 0x5629655a0fd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x562965594840 tx=0x7f949c012730 comp rx=0 tx=0).stop 2024-05-27T07:20:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f94c076bb80 1 -- 172.21.15.43:0/2434676835 shutdown_connections 2024-05-27T07:20:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x562965618270 0x56296561a690 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562965593c30 0x5629655a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562965410e40 0x5629655a0fd0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f94c076bb80 1 --2- 172.21.15.43:0/2434676835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562965593820 0x5629655a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f9494c00700 0 client.16446 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:20:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f94c076bb80 1 -- 172.21.15.43:0/2434676835 >> 172.21.15.43:0/2434676835 conn(0x562965470fc0 msgr2=0x56296559f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.229+0000 7f94c076bb80 1 -- 172.21.15.43:0/2434676835 shutdown_connections 2024-05-27T07:20:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94c076bb80 1 -- 172.21.15.43:0/2434676835 wait complete. 2024-05-27T07:20:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94c076bb80 20 client.16446 trim_cache size 0 max 16384 2024-05-27T07:20:26.221 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:20:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94abe00700 1 -- 172.21.15.43:0/2036848439 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94900b29f0 msgr2=0x7f94900b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94abe00700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94900b29f0 0x7f94900b4f00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009900 tx=0x7f94a400b040 comp rx=0 tx=0).stop 2024-05-27T07:20:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94abe00700 1 -- 172.21.15.43:0/2036848439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94ac108d40 msgr2=0x7f94ac1a0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94abe00700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94ac108d40 0x7f94ac1a0cb0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f949800d980 tx=0x7f949800dc90 comp rx=0 tx=0).stop 2024-05-27T07:20:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94abe00700 1 -- 172.21.15.43:0/2036848439 shutdown_connections 2024-05-27T07:20:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94abe00700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94900b29f0 0x7f94900b4f00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94abe00700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94ac109690 0x7f94ac19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94abe00700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94ac104410 0x7f94ac1a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.230+0000 7f94abe00700 1 --2- 172.21.15.43:0/2036848439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94ac108d40 0x7f94ac1a0cb0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.231+0000 7f94abe00700 1 -- 172.21.15.43:0/2036848439 >> 172.21.15.43:0/2036848439 conn(0x7f94ac07aca0 msgr2=0x7f94ac101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.231+0000 7f94abe00700 1 -- 172.21.15.43:0/2036848439 shutdown_connections 2024-05-27T07:20:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:26.231+0000 7f94abe00700 1 -- 172.21.15.43:0/2036848439 wait complete. 2024-05-27T07:20:26.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:26 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:26.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:26 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:26.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:26 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:20:27.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:26 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:26 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:27.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:26 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:20:27.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:20:27.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32578 0.63 MB/sec execute 151 sec latency 57.470 ms 2024-05-27T07:20:27.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:26 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:20:27.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:26 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:20:27.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:26 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:20:27.651 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:28.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:27 smithi043 ceph-mon[31531]: pgmap v219: 129 pgs: 129 active+clean; 59 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 610 KiB/s wr, 56 op/s 2024-05-27T07:20:28.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32720 0.63 MB/sec execute 152 sec latency 45.555 ms 2024-05-27T07:20:28.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:27 smithi132 ceph-mon[35283]: pgmap v219: 129 pgs: 129 active+clean; 59 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 610 KiB/s wr, 56 op/s 2024-05-27T07:20:28.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:27 smithi179 ceph-mon[35888]: pgmap v219: 129 pgs: 129 active+clean; 59 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 610 KiB/s wr, 56 op/s 2024-05-27T07:20:29.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32838 0.63 MB/sec execute 153 sec latency 57.194 ms 2024-05-27T07:20:30.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.111+0000 7f3eaf000700 1 -- 172.21.15.43:0/4170742947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ea8075ad0 msgr2=0x7f3ea8075ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:30.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.111+0000 7f3eaf000700 1 --2- 172.21.15.43:0/4170742947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ea8075ad0 0x7f3ea8075ee0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009990 tx=0x7f3e98009ca0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.111+0000 7f3eaf000700 1 -- 172.21.15.43:0/4170742947 shutdown_connections 2024-05-27T07:20:30.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.111+0000 7f3eaf000700 1 --2- 172.21.15.43:0/4170742947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ea8110c10 0x7f3ea8113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.111+0000 7f3eaf000700 1 --2- 172.21.15.43:0/4170742947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ea8075ad0 0x7f3ea8075ee0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.111+0000 7f3eaf000700 1 --2- 172.21.15.43:0/4170742947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ea8076420 0x7f3ea81106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.111+0000 7f3eaf000700 1 -- 172.21.15.43:0/4170742947 >> 172.21.15.43:0/4170742947 conn(0x7f3ea8100030 msgr2=0x7f3ea8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:30.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.111+0000 7f3eaf000700 1 -- 172.21.15.43:0/4170742947 shutdown_connections 2024-05-27T07:20:30.102 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:29 smithi043 ceph-mon[31531]: pgmap v220: 129 pgs: 129 active+clean; 59 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 350 KiB/s wr, 36 op/s 2024-05-27T07:20:30.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.112+0000 7f3eaf000700 1 -- 172.21.15.43:0/4170742947 wait complete. 2024-05-27T07:20:30.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.113+0000 7f3eaf000700 1 Processor -- start 2024-05-27T07:20:30.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.113+0000 7f3eaf000700 1 -- start start 2024-05-27T07:20:30.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.114+0000 7f3eaf000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ea8075ad0 0x7f3ea819cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.114+0000 7f3eaf000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ea8076420 0x7f3ea819d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.114+0000 7f3eaf000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ea8110c10 0x7f3ea81a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.114+0000 7f3eaf000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea8116560 con 0x7f3ea8076420 2024-05-27T07:20:30.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.114+0000 7f3eaf000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea81163e0 con 0x7f3ea8075ad0 2024-05-27T07:20:30.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.114+0000 7f3eaf000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea81166e0 con 0x7f3ea8110c10 2024-05-27T07:20:30.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.114+0000 7f3ea7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ea8076420 0x7f3ea819d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.114+0000 7f3ea7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ea8076420 0x7f3ea819d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59842/0 (socket says 172.21.15.43:59842) 2024-05-27T07:20:30.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.114+0000 7f3ea7400700 1 -- 172.21.15.43:0/655419554 learned_addr learned my addr 172.21.15.43:0/655419554 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:30.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.114+0000 7f3ea7e00700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ea8075ad0 0x7f3ea819cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.115+0000 7f3ead000700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ea8110c10 0x7f3ea81a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.115+0000 7f3ea7400700 1 -- 172.21.15.43:0/655419554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ea8110c10 msgr2=0x7f3ea81a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:30.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.115+0000 7f3ea7400700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ea8110c10 0x7f3ea81a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.115+0000 7f3ea7400700 1 -- 172.21.15.43:0/655419554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ea8075ad0 msgr2=0x7f3ea819cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:30.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.115+0000 7f3ea7400700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ea8075ad0 0x7f3ea819cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.115+0000 7f3ea7400700 1 -- 172.21.15.43:0/655419554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea81a1d70 con 0x7f3ea8076420 2024-05-27T07:20:30.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.115+0000 7f3ead000700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ea8110c10 0x7f3ea81a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:30.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.115+0000 7f3ea7400700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ea8076420 0x7f3ea819d4f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400d950 tx=0x7f3e9400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:30.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.116+0000 7f3ea4c00700 1 -- 172.21.15.43:0/655419554 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94017070 con 0x7f3ea8076420 2024-05-27T07:20:30.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.116+0000 7f3eaf000700 1 -- 172.21.15.43:0/655419554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea81a2060 con 0x7f3ea8076420 2024-05-27T07:20:30.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.116+0000 7f3ea4c00700 1 -- 172.21.15.43:0/655419554 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e94005020 con 0x7f3ea8076420 2024-05-27T07:20:30.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.116+0000 7f3ea4c00700 1 -- 172.21.15.43:0/655419554 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94009a20 con 0x7f3ea8076420 2024-05-27T07:20:30.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.116+0000 7f3eaf000700 1 -- 172.21.15.43:0/655419554 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea81a2550 con 0x7f3ea8076420 2024-05-27T07:20:30.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.121+0000 7f3ea4c00700 1 -- 172.21.15.43:0/655419554 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3e94009bc0 con 0x7f3ea8076420 2024-05-27T07:20:30.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.122+0000 7f3ebdc08b80 1 Processor -- start 2024-05-27T07:20:30.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.123+0000 7f3ebdc08b80 1 -- start start 2024-05-27T07:20:30.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.123+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac397f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.123+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac33870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.123+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac341c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.123+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cadab61e40 con 0x55cadac33db0 2024-05-27T07:20:30.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.123+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cadab519e0 con 0x55cadac39d30 2024-05-27T07:20:30.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.123+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55cadab55a20 con 0x55cadaab4e40 2024-05-27T07:20:30.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.124+0000 7f3ea4c00700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3e8c0b29f0 0x7f3e8c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.124+0000 7f3ea7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.124+0000 7f3ead000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.124+0000 7f3ead000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac397f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:51202/0 (socket says 172.21.15.43:51202) 2024-05-27T07:20:30.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.124+0000 7f3ead000700 1 -- 172.21.15.43:0/3302404082 learned_addr learned my addr 172.21.15.43:0/3302404082 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:30.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.124+0000 7f3ea7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac33870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.124+0000 7f3ead000700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3e8c0b29f0 0x7f3e8c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.124+0000 7f3ea4c00700 1 -- 172.21.15.43:0/655419554 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f3e940e2250 con 0x7f3ea8076420 2024-05-27T07:20:30.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.125+0000 7f3ea7400700 1 -- 172.21.15.43:0/3302404082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 msgr2=0x55cadac397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:30.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.125+0000 7f3ea7400700 1 --2- 172.21.15.43:0/3302404082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.125+0000 7f3ea7400700 1 -- 172.21.15.43:0/3302404082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 msgr2=0x55cadac33870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:30.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.125+0000 7f3ea7400700 1 --2- 172.21.15.43:0/3302404082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac33870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.125+0000 7f3ea7400700 1 -- 172.21.15.43:0/3302404082 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cadab1a420 con 0x55cadac33db0 2024-05-27T07:20:30.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.125+0000 7f3ead000700 1 --2- 172.21.15.43:0/3302404082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:30.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.125+0000 7f3ea7400700 1 --2- 172.21.15.43:0/3302404082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac341c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3e940df5f0 tx=0x7f3e94076d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c0582902b9d022 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:30.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.125+0000 7f3ead000700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3e8c0b29f0 0x7f3e8c0b4f00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x55cadac34b30 tx=0x7f3ea00076d0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:30.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.125+0000 7f3e9d600700 1 -- 172.21.15.43:0/3302404082 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9406eb80 con 0x55cadac33db0 2024-05-27T07:20:30.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.126+0000 7f3e9d600700 1 -- 172.21.15.43:0/3302404082 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e940a9880 con 0x55cadac33db0 2024-05-27T07:20:30.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.126+0000 7f3e9d600700 1 -- 172.21.15.43:0/3302404082 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94056020 con 0x55cadac33db0 2024-05-27T07:20:30.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.127+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/3302404082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 msgr2=0x55cadac341c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:30.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.127+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/3302404082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac341c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3e940df5f0 tx=0x7f3e94076d50 comp rx=0 tx=0).stop 2024-05-27T07:20:30.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.127+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/3302404082 shutdown_connections 2024-05-27T07:20:30.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.127+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/3302404082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac341c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.127+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/3302404082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.127+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/3302404082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac33870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.127+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/3302404082 >> 172.21.15.43:0/3302404082 conn(0x55cadab14fc0 msgr2=0x55cadac2e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:30.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.127+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/3302404082 shutdown_connections 2024-05-27T07:20:30.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.127+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/3302404082 wait complete. 2024-05-27T07:20:30.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.128+0000 7f3ebdc08b80 1 -- start start 2024-05-27T07:20:30.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.128+0000 7f3ebdc08b80 1 Processor -- start 2024-05-27T07:20:30.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadaab4e40 0x55cadac3d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadac33db0 0x55cadac3da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac3df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cadaa57fa0 con 0x55cadaab4e40 2024-05-27T07:20:30.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cadaaf2760 con 0x55cadac39d30 2024-05-27T07:20:30.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55cadac40800 con 0x55cadac33db0 2024-05-27T07:20:30.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ebdc08b80 10 client.0 _ll_register_callbacks cb 0x55cadaafbc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:30.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ea7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadaab4e40 0x55cadac3d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ea7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadaab4e40 0x55cadac3d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59862/0 (socket says 172.21.15.43:59862) 2024-05-27T07:20:30.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ea7e00700 1 -- 172.21.15.43:0/4003276742 learned_addr learned my addr 172.21.15.43:0/4003276742 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:30.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.129+0000 7f3ead000700 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac3df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3ea7400700 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadac33db0 0x55cadac3da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3ea7e00700 1 -- 172.21.15.43:0/4003276742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadac33db0 msgr2=0x55cadac3da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3ea7e00700 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadac33db0 0x55cadac3da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3ea7e00700 1 -- 172.21.15.43:0/4003276742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 msgr2=0x55cadac3df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3ea7e00700 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac3df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3ea7e00700 1 -- 172.21.15.43:0/4003276742 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cadaaa18d0 con 0x55cadaab4e40 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3ead000700 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac3df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3ea7e00700 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadaab4e40 0x55cadac3d4f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009990 tx=0x7f3e98009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ff551c7c5fe9b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3e92000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3e92000700 1 -- 172.21.15.43:0/4003276742 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e98003bd0 con 0x55cadaab4e40 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3e92000700 1 -- 172.21.15.43:0/4003276742 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f3e98003d70 con 0x55cadaab4e40 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.130+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/4003276742 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e940a9880 con 0x55cadaab4e40 2024-05-27T07:20:30.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.131+0000 7f3e92000700 1 -- 172.21.15.43:0/4003276742 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f3e98031050 con 0x55cadaab4e40 2024-05-27T07:20:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.131+0000 7f3ebdc08b80 10 client.16458 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.131+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/4003276742 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55cadab22060 con 0x55cadaab4e40 2024-05-27T07:20:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.131+0000 7f3e92000700 1 -- 172.21.15.43:0/4003276742 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e940a9880 con 0x55cadaab4e40 2024-05-27T07:20:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.131+0000 7f3e92000700 1 -- 172.21.15.43:0/4003276742 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f3e980237c0 con 0x55cadaab4e40 2024-05-27T07:20:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.131+0000 7f3ebdc08b80 10 client.16458 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.131+0000 7f3ebdc08b80 10 client.16458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:20:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.132+0000 7f3ebdc08b80 20 client.16458 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.132+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cadac4d970 0x55cadac4fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.132+0000 7f3ebdc08b80 4 client.16458 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:20:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.132+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/4003276742 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cadab63d90 con 0x55cadac4d970 2024-05-27T07:20:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.132+0000 7f3ea7400700 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cadac4d970 0x55cadac4fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.133+0000 7f3ea7400700 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cadac4d970 0x55cadac4fe40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=614ce75e7aa165a server_cookie=259640f640ad52f6 in_seq=0 out_seq=0 2024-05-27T07:20:30.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.133+0000 7f3e92000700 10 client.16458 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:30.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.135+0000 7f3e92000700 1 -- 172.21.15.43:0/4003276742 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cadab63d90 con 0x55cadac4d970 2024-05-27T07:20:30.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.135+0000 7f3e92000700 10 client.16458 handle_command_reply: tid=0 2024-05-27T07:20:30.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.135+0000 7f3ebdc08b80 1 client.16458 shutdown 2024-05-27T07:20:30.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.136+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/4003276742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadaab4e40 msgr2=0x55cadac3d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.136+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadaab4e40 0x55cadac3d4f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009990 tx=0x7f3e98009640 comp rx=0 tx=0).stop 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.136+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/4003276742 shutdown_connections 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.136+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac3df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.136+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cadac4d970 0x55cadac4fe40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.136+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadaab4e40 0x55cadac3d4f0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.136+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/4003276742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadac33db0 0x55cadac3da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.136+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/4003276742 >> 172.21.15.43:0/4003276742 conn(0x55cadab14fc0 msgr2=0x55cadac2e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.136+0000 7f3e92000700 0 client.16458 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.136+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/4003276742 shutdown_connections 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.137+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/4003276742 wait complete. 2024-05-27T07:20:30.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.137+0000 7f3ebdc08b80 20 client.16458 trim_cache size 0 max 16384 2024-05-27T07:20:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.155+0000 7f3ebdc08b80 1 Processor -- start 2024-05-27T07:20:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.155+0000 7f3ebdc08b80 1 -- start start 2024-05-27T07:20:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.155+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac33870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.155+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac3c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.155+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac38cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.155+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea81b3250 con 0x55cadac33db0 2024-05-27T07:20:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.155+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea81b3920 con 0x55cadac39d30 2024-05-27T07:20:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.155+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea81b1040 con 0x55cadaab4e40 2024-05-27T07:20:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.156+0000 7f3ea7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac3c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.156+0000 7f3ea7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac3c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59868/0 (socket says 172.21.15.43:59868) 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.156+0000 7f3ea7400700 1 -- 172.21.15.43:0/875161951 learned_addr learned my addr 172.21.15.43:0/875161951 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.156+0000 7f3ea7e00700 1 --2- 172.21.15.43:0/875161951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac33870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.156+0000 7f3ead000700 1 --2- 172.21.15.43:0/875161951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac38cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.156+0000 7f3ea7400700 1 -- 172.21.15.43:0/875161951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 msgr2=0x55cadac33870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.156+0000 7f3ea7400700 1 --2- 172.21.15.43:0/875161951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac33870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.156+0000 7f3ea7400700 1 -- 172.21.15.43:0/875161951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 msgr2=0x55cadac38cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.156+0000 7f3ea7400700 1 --2- 172.21.15.43:0/875161951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac38cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.156+0000 7f3ea7400700 1 -- 172.21.15.43:0/875161951 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cadac30350 con 0x55cadac33db0 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.157+0000 7f3ea7400700 1 --2- 172.21.15.43:0/875161951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac3c810 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f3e940a8e80 tx=0x7f3e940053e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782aa8b510ab26d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.157+0000 7f3e9cc00700 1 -- 172.21.15.43:0/875161951 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94020040 con 0x55cadac33db0 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.157+0000 7f3e9cc00700 1 -- 172.21.15.43:0/875161951 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e940ad530 con 0x55cadac33db0 2024-05-27T07:20:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.157+0000 7f3e9cc00700 1 -- 172.21.15.43:0/875161951 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e940a93e0 con 0x55cadac33db0 2024-05-27T07:20:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.158+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/875161951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 msgr2=0x55cadac3c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.158+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/875161951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac3c810 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f3e940a8e80 tx=0x7f3e940053e0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.158+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/875161951 shutdown_connections 2024-05-27T07:20:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.158+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/875161951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac39d30 0x55cadac38cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.158+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/875161951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac33870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.158+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/875161951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac33db0 0x55cadac3c810 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.158+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/875161951 >> 172.21.15.43:0/875161951 conn(0x55cadab14fc0 msgr2=0x55cadac37210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.159+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/875161951 shutdown_connections 2024-05-27T07:20:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.159+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/875161951 wait complete. 2024-05-27T07:20:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.159+0000 7f3ebdc08b80 1 -- start start 2024-05-27T07:20:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.160+0000 7f3ebdc08b80 1 Processor -- start 2024-05-27T07:20:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.160+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac45a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.160+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac33db0 0x55cadac45f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.160+0000 7f3ebdc08b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac39d30 0x55cadac54630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.160+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea81b1710 con 0x55cadac39d30 2024-05-27T07:20:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.160+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea81b2b80 con 0x55cadac33db0 2024-05-27T07:20:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.160+0000 7f3ebdc08b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea81b1de0 con 0x55cadaab4e40 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.160+0000 7f3ebdc08b80 10 client.0 _ll_register_callbacks cb 0x55cadaafbc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.161+0000 7f3ead000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac39d30 0x55cadac54630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.161+0000 7f3ead000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac39d30 0x55cadac54630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59870/0 (socket says 172.21.15.43:59870) 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.161+0000 7f3ead000700 1 -- 172.21.15.43:0/959878461 learned_addr learned my addr 172.21.15.43:0/959878461 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.161+0000 7f3ea7400700 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac33db0 0x55cadac45f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.161+0000 7f3ea7e00700 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac45a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.161+0000 7f3ead000700 1 -- 172.21.15.43:0/959878461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 msgr2=0x55cadac45a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.161+0000 7f3ead000700 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac45a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.161+0000 7f3ead000700 1 -- 172.21.15.43:0/959878461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac33db0 msgr2=0x55cadac45f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.161+0000 7f3ead000700 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac33db0 0x55cadac45f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.161+0000 7f3ead000700 1 -- 172.21.15.43:0/959878461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cadac3e2e0 con 0x55cadac39d30 2024-05-27T07:20:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.162+0000 7f3ead000700 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac39d30 0x55cadac54630 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0015c60 tx=0x7f3ea0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6900730f991403 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.162+0000 7f3e92000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.162+0000 7f3e92000700 1 -- 172.21.15.43:0/959878461 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0026070 con 0x55cadac39d30 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.162+0000 7f3e92000700 1 -- 172.21.15.43:0/959878461 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f3ea0004db0 con 0x55cadac39d30 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.162+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/959878461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cadab5d570 con 0x55cadac39d30 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.162+0000 7f3e92000700 1 -- 172.21.15.43:0/959878461 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f3ea0031050 con 0x55cadac39d30 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.162+0000 7f3ebdc08b80 10 client.16464 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.162+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/959878461 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55cadacb7830 con 0x55cadac39d30 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.163+0000 7f3e92000700 1 -- 172.21.15.43:0/959878461 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55cadab5d570 con 0x55cadac39d30 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.163+0000 7f3e92000700 1 -- 172.21.15.43:0/959878461 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f3ea0016040 con 0x55cadac39d30 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.163+0000 7f3ebdc08b80 10 client.16464 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.163+0000 7f3ebdc08b80 10 client.16464 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.163+0000 7f3ebdc08b80 20 client.16464 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.163+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cadacb7f70 0x55cadacba390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.163+0000 7f3ebdc08b80 4 client.16464 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:20:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.163+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/959878461 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cada9d4250 con 0x55cadacb7f70 2024-05-27T07:20:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.164+0000 7f3ea7e00700 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cadacb7f70 0x55cadacba390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.165+0000 7f3ea7e00700 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cadacb7f70 0x55cadacba390 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24b3cde59e569dc9 server_cookie=fcfaa5a3633a6164 in_seq=0 out_seq=0 2024-05-27T07:20:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.165+0000 7f3e92000700 10 client.16464 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.165+0000 7f3e92000700 1 -- 172.21.15.43:0/959878461 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cada9d4250 con 0x55cadacb7f70 2024-05-27T07:20:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.165+0000 7f3e92000700 10 client.16464 handle_command_reply: tid=0 2024-05-27T07:20:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.165+0000 7f3ebdc08b80 1 client.16464 shutdown 2024-05-27T07:20:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/959878461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac39d30 msgr2=0x55cadac54630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac39d30 0x55cadac54630 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0015c60 tx=0x7f3ea0004ad0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/959878461 shutdown_connections 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cadacb7f70 0x55cadacba390 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cadac39d30 0x55cadac54630 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cadaab4e40 0x55cadac45a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 1 --2- 172.21.15.43:0/959878461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cadac33db0 0x55cadac45f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3e92000700 0 client.16464 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/959878461 >> 172.21.15.43:0/959878461 conn(0x55cadab14fc0 msgr2=0x55cadac37210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/959878461 shutdown_connections 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 1 -- 172.21.15.43:0/959878461 wait complete. 2024-05-27T07:20:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.166+0000 7f3ebdc08b80 20 client.16464 trim_cache size 0 max 16384 2024-05-27T07:20:30.158 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:20:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.167+0000 7f3eaf000700 1 -- 172.21.15.43:0/655419554 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3e8c0b29f0 msgr2=0x7f3e8c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.167+0000 7f3eaf000700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3e8c0b29f0 0x7f3e8c0b4f00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x55cadac34b30 tx=0x7f3ea00076d0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.167+0000 7f3eaf000700 1 -- 172.21.15.43:0/655419554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ea8076420 msgr2=0x7f3ea819d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.167+0000 7f3eaf000700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ea8076420 0x7f3ea819d4f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400d950 tx=0x7f3e9400dc60 comp rx=0 tx=0).stop 2024-05-27T07:20:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.167+0000 7f3eaf000700 1 -- 172.21.15.43:0/655419554 shutdown_connections 2024-05-27T07:20:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.167+0000 7f3eaf000700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3e8c0b29f0 0x7f3e8c0b4f00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.167+0000 7f3eaf000700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ea8110c10 0x7f3ea81a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.167+0000 7f3eaf000700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ea8075ad0 0x7f3ea819cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.167+0000 7f3eaf000700 1 --2- 172.21.15.43:0/655419554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ea8076420 0x7f3ea819d4f0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.167+0000 7f3eaf000700 1 -- 172.21.15.43:0/655419554 >> 172.21.15.43:0/655419554 conn(0x7f3ea8100030 msgr2=0x7f3ea8100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.168+0000 7f3eaf000700 1 -- 172.21.15.43:0/655419554 shutdown_connections 2024-05-27T07:20:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:30.168+0000 7f3eaf000700 1 -- 172.21.15.43:0/655419554 wait complete. 2024-05-27T07:20:30.201 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32954 0.63 MB/sec execute 154 sec latency 50.196 ms 2024-05-27T07:20:30.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:29 smithi132 ceph-mon[35283]: pgmap v220: 129 pgs: 129 active+clean; 59 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 350 KiB/s wr, 36 op/s 2024-05-27T07:20:30.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:29 smithi179 ceph-mon[35888]: pgmap v220: 129 pgs: 129 active+clean; 59 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 350 KiB/s wr, 36 op/s 2024-05-27T07:20:31.202 INFO:tasks.workunit.client.0.smithi043.stdout: 1 33079 0.63 MB/sec execute 155 sec latency 56.865 ms 2024-05-27T07:20:32.005 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:20:32.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:20:32.202 INFO:tasks.workunit.client.0.smithi043.stdout: 1 33177 0.63 MB/sec execute 156 sec latency 53.829 ms 2024-05-27T07:20:32.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:31 smithi132 ceph-mon[35283]: pgmap v221: 129 pgs: 129 active+clean; 57 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 477 KiB/s wr, 61 op/s 2024-05-27T07:20:32.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:31 smithi043 ceph-mon[31531]: pgmap v221: 129 pgs: 129 active+clean; 57 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 477 KiB/s wr, 61 op/s 2024-05-27T07:20:32.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:31 smithi179 ceph-mon[35888]: pgmap v221: 129 pgs: 129 active+clean; 57 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 477 KiB/s wr, 61 op/s 2024-05-27T07:20:32.501 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:33.202 INFO:tasks.workunit.client.0.smithi043.stdout: 1 33366 0.64 MB/sec execute 157 sec latency 61.104 ms 2024-05-27T07:20:34.202 INFO:tasks.workunit.client.0.smithi043.stdout: 1 33432 0.64 MB/sec execute 158 sec latency 327.489 ms 2024-05-27T07:20:34.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:34 smithi132 ceph-mon[35283]: pgmap v222: 129 pgs: 129 active+clean; 57 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 327 KiB/s wr, 47 op/s 2024-05-27T07:20:34.426 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:34 smithi043 ceph-mon[31531]: pgmap v222: 129 pgs: 129 active+clean; 57 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 327 KiB/s wr, 47 op/s 2024-05-27T07:20:34.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:34 smithi179 ceph-mon[35888]: pgmap v222: 129 pgs: 129 active+clean; 57 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 327 KiB/s wr, 47 op/s 2024-05-27T07:20:35.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.091+0000 7f3ff2600700 1 -- 172.21.15.43:0/4181828738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fec1045d0 msgr2=0x7f3fec1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:35.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.091+0000 7f3ff2600700 1 --2- 172.21.15.43:0/4181828738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fec1045d0 0x7f3fec1049e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0099c0 tx=0x7f3fdc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.092+0000 7f3ff2600700 1 -- 172.21.15.43:0/4181828738 shutdown_connections 2024-05-27T07:20:35.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.092+0000 7f3ff2600700 1 --2- 172.21.15.43:0/4181828738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fec109850 0x7f3fec1111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.092+0000 7f3ff2600700 1 --2- 172.21.15.43:0/4181828738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fec1045d0 0x7f3fec1049e0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.092+0000 7f3ff2600700 1 --2- 172.21.15.43:0/4181828738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fec108f00 0x7f3fec109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.092+0000 7f3ff2600700 1 -- 172.21.15.43:0/4181828738 >> 172.21.15.43:0/4181828738 conn(0x7f3fec0ffe60 msgr2=0x7f3fec1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:35.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.092+0000 7f3ff2600700 1 -- 172.21.15.43:0/4181828738 shutdown_connections 2024-05-27T07:20:35.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.092+0000 7f3ff2600700 1 -- 172.21.15.43:0/4181828738 wait complete. 2024-05-27T07:20:35.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.093+0000 7f3ff2600700 1 Processor -- start 2024-05-27T07:20:35.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.093+0000 7f3ff2600700 1 -- start start 2024-05-27T07:20:35.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3ff2600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fec1045d0 0x7f3fec075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3ff2600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fec108f00 0x7f3fec075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3ff2600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fec109850 0x7f3fec076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3ff2600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec114330 con 0x7f3fec1045d0 2024-05-27T07:20:35.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3ff2600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec1141b0 con 0x7f3fec109850 2024-05-27T07:20:35.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3ff2600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec1144b0 con 0x7f3fec108f00 2024-05-27T07:20:35.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3feb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fec1045d0 0x7f3fec075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3feb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fec1045d0 0x7f3fec075ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39570/0 (socket says 172.21.15.43:39570) 2024-05-27T07:20:35.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3feb400700 1 -- 172.21.15.43:0/3498853555 learned_addr learned my addr 172.21.15.43:0/3498853555 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:35.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3feaa00700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fec108f00 0x7f3fec075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.094+0000 7f3febe00700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fec109850 0x7f3fec076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.095+0000 7f3feb400700 1 -- 172.21.15.43:0/3498853555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fec108f00 msgr2=0x7f3fec075ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.095+0000 7f3feb400700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fec108f00 0x7f3fec075ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.095+0000 7f3feb400700 1 -- 172.21.15.43:0/3498853555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fec109850 msgr2=0x7f3fec076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.095+0000 7f3feb400700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fec109850 0x7f3fec076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.095+0000 7f3feb400700 1 -- 172.21.15.43:0/3498853555 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fec1a90d0 con 0x7f3fec1045d0 2024-05-27T07:20:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.095+0000 7f3feb400700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fec1045d0 0x7f3fec075ab0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc000c00 tx=0x7f3fdc00f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.095+0000 7f3fe3e00700 1 -- 172.21.15.43:0/3498853555 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc00f940 con 0x7f3fec1045d0 2024-05-27T07:20:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.096+0000 7f3fe3e00700 1 -- 172.21.15.43:0/3498853555 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3fdc004830 con 0x7f3fec1045d0 2024-05-27T07:20:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.096+0000 7f3fe3e00700 1 -- 172.21.15.43:0/3498853555 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc0054b0 con 0x7f3fec1045d0 2024-05-27T07:20:35.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.096+0000 7f3ff2600700 1 -- 172.21.15.43:0/3498853555 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fec1a9360 con 0x7f3fec1045d0 2024-05-27T07:20:35.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.096+0000 7f3ff2600700 1 -- 172.21.15.43:0/3498853555 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fec1a9700 con 0x7f3fec1045d0 2024-05-27T07:20:35.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.100+0000 7f3fe3e00700 1 -- 172.21.15.43:0/3498853555 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3fdc029060 con 0x7f3fec1045d0 2024-05-27T07:20:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.101+0000 7f4001161b80 1 Processor -- start 2024-05-27T07:20:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.101+0000 7f4001161b80 1 -- start start 2024-05-27T07:20:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.101+0000 7f4001161b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.101+0000 7f4001161b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f092310ec0 0x55f09248f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.101+0000 7f4001161b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f09248fef0 0x55f09249c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.101+0000 7f4001161b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f0923bdec0 con 0x55f0923fca70 2024-05-27T07:20:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.101+0000 7f4001161b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f0923ada60 con 0x55f09248fef0 2024-05-27T07:20:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.101+0000 7f4001161b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f0923b1aa0 con 0x55f092310ec0 2024-05-27T07:20:35.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.102+0000 7f3febe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.102+0000 7f3febe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39586/0 (socket says 172.21.15.43:39586) 2024-05-27T07:20:35.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.102+0000 7f3febe00700 1 -- 172.21.15.43:0/2947399867 learned_addr learned my addr 172.21.15.43:0/2947399867 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:35.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.102+0000 7f3feaa00700 1 --2- 172.21.15.43:0/2947399867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f09248fef0 0x55f09249c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.102+0000 7f3feb400700 1 --2- 172.21.15.43:0/2947399867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f092310ec0 0x55f09248f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.102+0000 7f3fe3e00700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3fd00b29f0 0x7f3fd00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.103+0000 7f3febe00700 1 -- 172.21.15.43:0/2947399867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f092310ec0 msgr2=0x55f09248f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:35.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.103+0000 7f3febe00700 1 --2- 172.21.15.43:0/2947399867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f092310ec0 0x55f09248f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.103+0000 7f3febe00700 1 -- 172.21.15.43:0/2947399867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f09248fef0 msgr2=0x55f09249c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:35.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.103+0000 7f3fe3e00700 1 -- 172.21.15.43:0/3498853555 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f3fdc0e3580 con 0x7f3fec1045d0 2024-05-27T07:20:35.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.103+0000 7f3febe00700 1 --2- 172.21.15.43:0/2947399867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f09248fef0 0x55f09249c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.103+0000 7f3febe00700 1 -- 172.21.15.43:0/2947399867 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0923764a0 con 0x55f0923fca70 2024-05-27T07:20:35.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.103+0000 7f3febe00700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3fd00b29f0 0x7f3fd00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.103+0000 7f3febe00700 1 --2- 172.21.15.43:0/2947399867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248f3b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009e80 tx=0x7f3fe40077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8db0498e2be4d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.104+0000 7f3fe0c00700 1 -- 172.21.15.43:0/2947399867 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4018dd0 con 0x55f0923fca70 2024-05-27T07:20:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.104+0000 7f3fe0c00700 1 -- 172.21.15.43:0/2947399867 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3fe4017860 con 0x55f0923fca70 2024-05-27T07:20:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.104+0000 7f3fe0c00700 1 -- 172.21.15.43:0/2947399867 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe40165d0 con 0x55f0923fca70 2024-05-27T07:20:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.105+0000 7f4001161b80 1 -- 172.21.15.43:0/2947399867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 msgr2=0x55f09248f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.105+0000 7f4001161b80 1 --2- 172.21.15.43:0/2947399867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248f3b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009e80 tx=0x7f3fe40077e0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.105+0000 7f3febe00700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3fd00b29f0 0x7f3fd00b4f00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4020000 tx=0x7f3fe401d000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.105+0000 7f4001161b80 1 -- 172.21.15.43:0/2947399867 shutdown_connections 2024-05-27T07:20:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.105+0000 7f4001161b80 1 --2- 172.21.15.43:0/2947399867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f09248fef0 0x55f09249c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.105+0000 7f4001161b80 1 --2- 172.21.15.43:0/2947399867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248f3b0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.105+0000 7f4001161b80 1 --2- 172.21.15.43:0/2947399867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f092310ec0 0x55f09248f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.105+0000 7f4001161b80 1 -- 172.21.15.43:0/2947399867 >> 172.21.15.43:0/2947399867 conn(0x55f0923aa410 msgr2=0x55f09248b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.105+0000 7f4001161b80 1 -- 172.21.15.43:0/2947399867 shutdown_connections 2024-05-27T07:20:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.105+0000 7f4001161b80 1 -- 172.21.15.43:0/2947399867 wait complete. 2024-05-27T07:20:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.106+0000 7f4001161b80 1 -- start start 2024-05-27T07:20:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.106+0000 7f4001161b80 1 Processor -- start 2024-05-27T07:20:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f4001161b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f092310ec0 0x55f0924997d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f4001161b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0923fca70 0x55f092499d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f4001161b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f09249a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f4001161b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f0922b4020 con 0x55f09248fef0 2024-05-27T07:20:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f4001161b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f09234e7e0 con 0x55f0923fca70 2024-05-27T07:20:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f4001161b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f09249cc00 con 0x55f092310ec0 2024-05-27T07:20:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f4001161b80 10 client.0 _ll_register_callbacks cb 0x55f092357cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f3febe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f09249a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f3febe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f09249a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39592/0 (socket says 172.21.15.43:39592) 2024-05-27T07:20:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f3febe00700 1 -- 172.21.15.43:0/3979063498 learned_addr learned my addr 172.21.15.43:0/3979063498 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f3feaa00700 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0923fca70 0x55f092499d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.107+0000 7f3feb400700 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f092310ec0 0x55f0924997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.108+0000 7f3febe00700 1 -- 172.21.15.43:0/3979063498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f092310ec0 msgr2=0x55f0924997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.108+0000 7f3febe00700 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f092310ec0 0x55f0924997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.108+0000 7f3febe00700 1 -- 172.21.15.43:0/3979063498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0923fca70 msgr2=0x55f092499d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.108+0000 7f3febe00700 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0923fca70 0x55f092499d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.108+0000 7f3febe00700 1 -- 172.21.15.43:0/3979063498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0922fd950 con 0x55f09248fef0 2024-05-27T07:20:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.108+0000 7f3febe00700 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f09249a250 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4018040 tx=0x7f3fe4017680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aea029c06fbf8f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.108+0000 7f3fd5600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.108+0000 7f3fd5600700 1 -- 172.21.15.43:0/3979063498 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4031060 con 0x55f09248fef0 2024-05-27T07:20:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.109+0000 7f3fd5600700 1 -- 172.21.15.43:0/3979063498 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f3fe4018d50 con 0x55f09248fef0 2024-05-27T07:20:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.109+0000 7f4001161b80 1 -- 172.21.15.43:0/3979063498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fe4017860 con 0x55f09248fef0 2024-05-27T07:20:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.109+0000 7f3fd5600700 1 -- 172.21.15.43:0/3979063498 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f3fe4038050 con 0x55f09248fef0 2024-05-27T07:20:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.109+0000 7f4001161b80 10 client.16476 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.109+0000 7f4001161b80 1 -- 172.21.15.43:0/3979063498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55f09237e0e0 con 0x55f09248fef0 2024-05-27T07:20:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.109+0000 7f3fd5600700 1 -- 172.21.15.43:0/3979063498 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3fe4017860 con 0x55f09248fef0 2024-05-27T07:20:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.109+0000 7f3fd5600700 1 -- 172.21.15.43:0/3979063498 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f3fe402b030 con 0x55f09248fef0 2024-05-27T07:20:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.110+0000 7f4001161b80 10 client.16476 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:35.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.110+0000 7f4001161b80 10 client.16476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:20:35.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.110+0000 7f4001161b80 20 client.16476 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:35.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.110+0000 7f4001161b80 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f0924a9b90 0x55f0924ac020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.110+0000 7f4001161b80 4 client.16476 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:20:35.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.110+0000 7f4001161b80 1 -- 172.21.15.43:0/3979063498 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0923bfe10 con 0x55f0924a9b90 2024-05-27T07:20:35.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.110+0000 7f3feb400700 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f0924a9b90 0x55f0924ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.111+0000 7f3feb400700 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f0924a9b90 0x55f0924ac020 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ebf05b7e32592d9 server_cookie=a2b9f872b3e71cb4 in_seq=0 out_seq=0 2024-05-27T07:20:35.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.111+0000 7f3fd5600700 10 client.16476 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:35.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f3fd5600700 1 -- 172.21.15.43:0/3979063498 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f0923bfe10 con 0x55f0924a9b90 2024-05-27T07:20:35.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f3fd5600700 10 client.16476 handle_command_reply: tid=0 2024-05-27T07:20:35.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 client.16476 shutdown 2024-05-27T07:20:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 -- 172.21.15.43:0/3979063498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 msgr2=0x55f09249a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f09249a250 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4018040 tx=0x7f3fe4017680 comp rx=0 tx=0).stop 2024-05-27T07:20:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 -- 172.21.15.43:0/3979063498 shutdown_connections 2024-05-27T07:20:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f0924a9b90 0x55f0924ac020 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f09249a250 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f092310ec0 0x55f0924997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 --2- 172.21.15.43:0/3979063498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f0923fca70 0x55f092499d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 -- 172.21.15.43:0/3979063498 >> 172.21.15.43:0/3979063498 conn(0x55f0923aa410 msgr2=0x55f09248f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f3fd5600700 0 client.16476 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 -- 172.21.15.43:0/3979063498 shutdown_connections 2024-05-27T07:20:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.114+0000 7f4001161b80 1 -- 172.21.15.43:0/3979063498 wait complete. 2024-05-27T07:20:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.115+0000 7f4001161b80 20 client.16476 trim_cache size 0 max 16384 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.140+0000 7f4001161b80 1 Processor -- start 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.140+0000 7f4001161b80 1 -- start start 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.140+0000 7f4001161b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f092310ec0 0x55f09249a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.140+0000 7f4001161b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.140+0000 7f4001161b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f09248fef0 0x55f0924a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.140+0000 7f4001161b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec1b31c0 con 0x55f0923fca70 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.140+0000 7f4001161b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec1b3890 con 0x55f092310ec0 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.140+0000 7f4001161b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec1b0fb0 con 0x55f09248fef0 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.141+0000 7f3feaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.141+0000 7f3feaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39606/0 (socket says 172.21.15.43:39606) 2024-05-27T07:20:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.141+0000 7f3feaa00700 1 -- 172.21.15.43:0/277298996 learned_addr learned my addr 172.21.15.43:0/277298996 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.141+0000 7f3feb400700 1 --2- 172.21.15.43:0/277298996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f092310ec0 0x55f09249a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.141+0000 7f3febe00700 1 --2- 172.21.15.43:0/277298996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f09248fef0 0x55f0924a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.141+0000 7f3feaa00700 1 -- 172.21.15.43:0/277298996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f09248fef0 msgr2=0x55f0924a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.141+0000 7f3feaa00700 1 --2- 172.21.15.43:0/277298996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f09248fef0 0x55f0924a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.141+0000 7f3feaa00700 1 -- 172.21.15.43:0/277298996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f092310ec0 msgr2=0x55f09249a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.141+0000 7f3feaa00700 1 --2- 172.21.15.43:0/277298996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f092310ec0 0x55f09249a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.141+0000 7f3feaa00700 1 -- 172.21.15.43:0/277298996 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0924ac5b0 con 0x55f0923fca70 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.142+0000 7f3feaa00700 1 --2- 172.21.15.43:0/277298996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248b570 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800be70 tx=0x7f3fd8012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4047fe376c1bbd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.142+0000 7f3fd7e00700 1 -- 172.21.15.43:0/277298996 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8004030 con 0x55f0923fca70 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.142+0000 7f3fd7e00700 1 -- 172.21.15.43:0/277298996 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3fd8003680 con 0x55f0923fca70 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.142+0000 7f3fd7e00700 1 -- 172.21.15.43:0/277298996 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd801b760 con 0x55f0923fca70 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.143+0000 7f4001161b80 1 -- 172.21.15.43:0/277298996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 msgr2=0x55f09248b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.143+0000 7f4001161b80 1 --2- 172.21.15.43:0/277298996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248b570 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800be70 tx=0x7f3fd8012730 comp rx=0 tx=0).stop 2024-05-27T07:20:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.143+0000 7f4001161b80 1 -- 172.21.15.43:0/277298996 shutdown_connections 2024-05-27T07:20:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.143+0000 7f4001161b80 1 --2- 172.21.15.43:0/277298996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f09248fef0 0x55f0924a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.143+0000 7f4001161b80 1 --2- 172.21.15.43:0/277298996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f092310ec0 0x55f09249a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.143+0000 7f4001161b80 1 --2- 172.21.15.43:0/277298996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f0923fca70 0x55f09248b570 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.143+0000 7f4001161b80 1 -- 172.21.15.43:0/277298996 >> 172.21.15.43:0/277298996 conn(0x55f0923aa410 msgr2=0x55f092490f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.143+0000 7f4001161b80 1 -- 172.21.15.43:0/277298996 shutdown_connections 2024-05-27T07:20:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.143+0000 7f4001161b80 1 -- 172.21.15.43:0/277298996 wait complete. 2024-05-27T07:20:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.144+0000 7f4001161b80 1 -- start start 2024-05-27T07:20:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.144+0000 7f4001161b80 1 Processor -- start 2024-05-27T07:20:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.145+0000 7f4001161b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f092310ec0 0x55f0924a2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.145+0000 7f4001161b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0923fca70 0x55f0924940a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.145+0000 7f4001161b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f092494620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.145+0000 7f4001161b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec1b1680 con 0x55f09248fef0 2024-05-27T07:20:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.145+0000 7f4001161b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec1b2af0 con 0x55f092310ec0 2024-05-27T07:20:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.145+0000 7f4001161b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec1b1d50 con 0x55f0923fca70 2024-05-27T07:20:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.145+0000 7f4001161b80 10 client.0 _ll_register_callbacks cb 0x55f092357cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.145+0000 7f3febe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f092494620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.145+0000 7f3febe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f092494620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39612/0 (socket says 172.21.15.43:39612) 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.145+0000 7f3febe00700 1 -- 172.21.15.43:0/1080873459 learned_addr learned my addr 172.21.15.43:0/1080873459 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.146+0000 7f3feb400700 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f092310ec0 0x55f0924a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.146+0000 7f3feaa00700 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0923fca70 0x55f0924940a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.146+0000 7f3febe00700 1 -- 172.21.15.43:0/1080873459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0923fca70 msgr2=0x55f0924940a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.146+0000 7f3febe00700 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0923fca70 0x55f0924940a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.146+0000 7f3febe00700 1 -- 172.21.15.43:0/1080873459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f092310ec0 msgr2=0x55f0924a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.146+0000 7f3febe00700 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f092310ec0 0x55f0924a2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.146+0000 7f3febe00700 1 -- 172.21.15.43:0/1080873459 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f092494dc0 con 0x55f09248fef0 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.146+0000 7f3febe00700 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f092494620 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009710 tx=0x7f3fe4006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17540e47419b10c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.146+0000 7f3fd5600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.147+0000 7f3fd5600700 1 -- 172.21.15.43:0/1080873459 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4040020 con 0x55f09248fef0 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.147+0000 7f3fd5600700 1 -- 172.21.15.43:0/1080873459 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f3fe4018b50 con 0x55f09248fef0 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.147+0000 7f4001161b80 1 -- 172.21.15.43:0/1080873459 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fd8003680 con 0x55f09248fef0 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.147+0000 7f3fd5600700 1 -- 172.21.15.43:0/1080873459 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f3fe4048050 con 0x55f09248fef0 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.147+0000 7f4001161b80 10 client.16482 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.147+0000 7f4001161b80 1 -- 172.21.15.43:0/1080873459 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55f0925146b0 con 0x55f09248fef0 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.147+0000 7f3fd5600700 1 -- 172.21.15.43:0/1080873459 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3fd8003680 con 0x55f09248fef0 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.148+0000 7f3fd5600700 1 -- 172.21.15.43:0/1080873459 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f3fe402b040 con 0x55f09248fef0 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.148+0000 7f4001161b80 10 client.16482 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.148+0000 7f4001161b80 10 client.16482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.148+0000 7f4001161b80 20 client.16482 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.148+0000 7f4001161b80 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f092514bf0 0x55f092517010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.148+0000 7f4001161b80 4 client.16482 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:20:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.148+0000 7f4001161b80 1 -- 172.21.15.43:0/1080873459 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f0923b95f0 con 0x55f092514bf0 2024-05-27T07:20:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.149+0000 7f3feb400700 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f092514bf0 0x55f092517010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.149+0000 7f3feb400700 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f092514bf0 0x55f092517010 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47ef010f18a24bb8 server_cookie=2a3db66a4a2fa744 in_seq=0 out_seq=0 2024-05-27T07:20:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.149+0000 7f3fd5600700 10 client.16482 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.154+0000 7f3fd5600700 1 -- 172.21.15.43:0/1080873459 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f0923b95f0 con 0x55f092514bf0 2024-05-27T07:20:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.154+0000 7f3fd5600700 10 client.16482 handle_command_reply: tid=0 2024-05-27T07:20:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.154+0000 7f4001161b80 1 client.16482 shutdown 2024-05-27T07:20:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.154+0000 7f4001161b80 1 -- 172.21.15.43:0/1080873459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 msgr2=0x55f092494620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.154+0000 7f4001161b80 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f092494620 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009710 tx=0x7f3fe4006000 comp rx=0 tx=0).stop 2024-05-27T07:20:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.154+0000 7f4001161b80 1 -- 172.21.15.43:0/1080873459 shutdown_connections 2024-05-27T07:20:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.154+0000 7f4001161b80 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55f092514bf0 0x55f092517010 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.155+0000 7f4001161b80 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f09248fef0 0x55f092494620 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.155+0000 7f4001161b80 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f092310ec0 0x55f0924a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.155+0000 7f4001161b80 1 --2- 172.21.15.43:0/1080873459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f0923fca70 0x55f0924940a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.155+0000 7f4001161b80 1 -- 172.21.15.43:0/1080873459 >> 172.21.15.43:0/1080873459 conn(0x55f0923aa410 msgr2=0x55f092499fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.155+0000 7f3fd5600700 0 client.16482 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.155+0000 7f4001161b80 1 -- 172.21.15.43:0/1080873459 shutdown_connections 2024-05-27T07:20:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.155+0000 7f4001161b80 1 -- 172.21.15.43:0/1080873459 wait complete. 2024-05-27T07:20:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.155+0000 7f4001161b80 20 client.16482 trim_cache size 0 max 16384 2024-05-27T07:20:35.147 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:20:35.148 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:20:35.148 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "80c73548-c338-4802-9ffb-85a179f8bbec", 2024-05-27T07:20:35.148 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:20:35.148 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:20:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 -- 172.21.15.43:0/3498853555 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3fd00b29f0 msgr2=0x7f3fd00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3fd00b29f0 0x7f3fd00b4f00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4020000 tx=0x7f3fe401d000 comp rx=0 tx=0).stop 2024-05-27T07:20:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 -- 172.21.15.43:0/3498853555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fec1045d0 msgr2=0x7f3fec075ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fec1045d0 0x7f3fec075ab0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc000c00 tx=0x7f3fdc00f510 comp rx=0 tx=0).stop 2024-05-27T07:20:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 -- 172.21.15.43:0/3498853555 shutdown_connections 2024-05-27T07:20:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3fd00b29f0 0x7f3fd00b4f00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fec109850 0x7f3fec076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fec1045d0 0x7f3fec075ab0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 --2- 172.21.15.43:0/3498853555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fec108f00 0x7f3fec075ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 -- 172.21.15.43:0/3498853555 >> 172.21.15.43:0/3498853555 conn(0x7f3fec0ffe60 msgr2=0x7f3fec1072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 -- 172.21.15.43:0/3498853555 shutdown_connections 2024-05-27T07:20:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:35.156+0000 7f3ff2600700 1 -- 172.21.15.43:0/3498853555 wait complete. 2024-05-27T07:20:35.202 INFO:tasks.workunit.client.0.smithi043.stdout: 1 33532 0.63 MB/sec execute 159 sec latency 54.003 ms 2024-05-27T07:20:36.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: pgmap v223: 129 pgs: 129 active+clean; 57 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 327 KiB/s wr, 47 op/s 2024-05-27T07:20:36.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:20:36.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:20:36.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.202 INFO:tasks.workunit.client.0.smithi043.stdout: 1 33656 0.63 MB/sec execute 160 sec latency 64.106 ms 2024-05-27T07:20:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: pgmap v223: 129 pgs: 129 active+clean; 57 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 327 KiB/s wr, 47 op/s 2024-05-27T07:20:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:20:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:20:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:20:36.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.342 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:20:36.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: pgmap v223: 129 pgs: 129 active+clean; 57 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 327 KiB/s wr, 47 op/s 2024-05-27T07:20:36.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:20:36.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:20:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:20:37.202 INFO:tasks.workunit.client.0.smithi043.stdout: 1 33808 0.63 MB/sec execute 161 sec latency 62.502 ms 2024-05-27T07:20:38.202 INFO:tasks.workunit.client.0.smithi043.stdout: 1 33943 0.62 MB/sec execute 162 sec latency 67.142 ms 2024-05-27T07:20:38.306 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:38 smithi043 ceph-mon[31531]: pgmap v224: 129 pgs: 129 active+clean; 58 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 481 KiB/s wr, 68 op/s 2024-05-27T07:20:38.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:38 smithi132 ceph-mon[35283]: pgmap v224: 129 pgs: 129 active+clean; 58 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 481 KiB/s wr, 68 op/s 2024-05-27T07:20:38.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:38 smithi179 ceph-mon[35888]: pgmap v224: 129 pgs: 129 active+clean; 58 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 481 KiB/s wr, 68 op/s 2024-05-27T07:20:39.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 34107 0.62 MB/sec execute 163 sec latency 69.128 ms 2024-05-27T07:20:39.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.259+0000 7f83dd200700 1 -- 172.21.15.43:0/1263089979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83d8107c00 msgr2=0x7f83d8108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:39.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.259+0000 7f83dd200700 1 --2- 172.21.15.43:0/1263089979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83d8107c00 0x7f83d8108010 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f83c40099c0 tx=0x7f83c4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.259+0000 7f83dd200700 1 -- 172.21.15.43:0/1263089979 shutdown_connections 2024-05-27T07:20:39.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.259+0000 7f83dd200700 1 --2- 172.21.15.43:0/1263089979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d8108550 0x7f83d810feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.259+0000 7f83dd200700 1 --2- 172.21.15.43:0/1263089979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83d81032d0 0x7f83d81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.259+0000 7f83dd200700 1 --2- 172.21.15.43:0/1263089979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83d8107c00 0x7f83d8108010 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.259+0000 7f83dd200700 1 -- 172.21.15.43:0/1263089979 >> 172.21.15.43:0/1263089979 conn(0x7f83d8078ab0 msgr2=0x7f83d8078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:39.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.259+0000 7f83dd200700 1 -- 172.21.15.43:0/1263089979 shutdown_connections 2024-05-27T07:20:39.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.259+0000 7f83dd200700 1 -- 172.21.15.43:0/1263089979 wait complete. 2024-05-27T07:20:39.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.261+0000 7f83dd200700 1 Processor -- start 2024-05-27T07:20:39.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.261+0000 7f83dd200700 1 -- start start 2024-05-27T07:20:39.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.261+0000 7f83dd200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d81032d0 0x7f83d81a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.261+0000 7f83dd200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83d8107c00 0x7f83d81a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.261+0000 7f83dd200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83d8108550 0x7f83d819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.261+0000 7f83dd200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f83d8113060 con 0x7f83d8107c00 2024-05-27T07:20:39.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.261+0000 7f83dd200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f83d8112ee0 con 0x7f83d8108550 2024-05-27T07:20:39.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.261+0000 7f83dd200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81131e0 con 0x7f83d81032d0 2024-05-27T07:20:39.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.262+0000 7f83d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83d8107c00 0x7f83d81a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.262+0000 7f83d6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83d8107c00 0x7f83d81a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39624/0 (socket says 172.21.15.43:39624) 2024-05-27T07:20:39.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.262+0000 7f83d6a00700 1 -- 172.21.15.43:0/69583000 learned_addr learned my addr 172.21.15.43:0/69583000 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:39.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.262+0000 7f83d7400700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d81032d0 0x7f83d81a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.262+0000 7f83d7e00700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83d8108550 0x7f83d819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.263+0000 7f83d6a00700 1 -- 172.21.15.43:0/69583000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d81032d0 msgr2=0x7f83d81a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:39.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.263+0000 7f83d6a00700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d81032d0 0x7f83d81a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.263+0000 7f83d6a00700 1 -- 172.21.15.43:0/69583000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83d8108550 msgr2=0x7f83d819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:39.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.263+0000 7f83d6a00700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83d8108550 0x7f83d819a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.263+0000 7f83d6a00700 1 -- 172.21.15.43:0/69583000 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d819b030 con 0x7f83d8107c00 2024-05-27T07:20:39.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.263+0000 7f83d7e00700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83d8108550 0x7f83d819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:39.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.263+0000 7f83d6a00700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83d8107c00 0x7f83d81a0d10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f83c400b580 tx=0x7f83c40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:39.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.263+0000 7f83cfe00700 1 -- 172.21.15.43:0/69583000 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4027070 con 0x7f83d8107c00 2024-05-27T07:20:39.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.263+0000 7f83dd200700 1 -- 172.21.15.43:0/69583000 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d819b2c0 con 0x7f83d8107c00 2024-05-27T07:20:39.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.264+0000 7f83cfe00700 1 -- 172.21.15.43:0/69583000 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f83c400fd30 con 0x7f83d8107c00 2024-05-27T07:20:39.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.264+0000 7f83dd200700 1 -- 172.21.15.43:0/69583000 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d819b7a0 con 0x7f83d8107c00 2024-05-27T07:20:39.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.264+0000 7f83cfe00700 1 -- 172.21.15.43:0/69583000 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4022df0 con 0x7f83d8107c00 2024-05-27T07:20:39.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.268+0000 7f83cfe00700 1 -- 172.21.15.43:0/69583000 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f83c402a070 con 0x7f83d8107c00 2024-05-27T07:20:39.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.269+0000 7f83ed2dbb80 1 Processor -- start 2024-05-27T07:20:39.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.269+0000 7f83ed2dbb80 1 -- start start 2024-05-27T07:20:39.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.270+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b84ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.270+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85410 0x560e28b8fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.270+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85820 0x560e28b92020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.270+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560e28ab3e40 con 0x560e28b85410 2024-05-27T07:20:39.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.270+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560e28aa39e0 con 0x560e28a06e40 2024-05-27T07:20:39.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.270+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560e28aa7a20 con 0x560e28b85820 2024-05-27T07:20:39.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.270+0000 7f83d7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85410 0x560e28b8fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.270+0000 7f83d7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85410 0x560e28b8fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39632/0 (socket says 172.21.15.43:39632) 2024-05-27T07:20:39.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.270+0000 7f83d7400700 1 -- 172.21.15.43:0/480848126 learned_addr learned my addr 172.21.15.43:0/480848126 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:39.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83d7e00700 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b84ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83cfe00700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f83bc0b29f0 0x7f83bc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83d6a00700 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85820 0x560e28b92020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83d7400700 1 -- 172.21.15.43:0/480848126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85820 msgr2=0x560e28b92020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:39.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83d7400700 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85820 0x560e28b92020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83cfe00700 1 -- 172.21.15.43:0/69583000 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f83c40e3710 con 0x7f83d8107c00 2024-05-27T07:20:39.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83d7400700 1 -- 172.21.15.43:0/480848126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 msgr2=0x560e28b84ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:39.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83d7400700 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b84ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83d7400700 1 -- 172.21.15.43:0/480848126 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e28a6c420 con 0x560e28b85410 2024-05-27T07:20:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83d6a00700 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85820 0x560e28b92020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:20:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83d7e00700 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b84ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.271+0000 7f83d7e00700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f83bc0b29f0 0x7f83bc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.272+0000 7f83d7400700 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85410 0x560e28b8fad0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f83c800bd30 tx=0x7f83c80106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c90b7ab6fd28b16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:39.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.272+0000 7f83ccc00700 1 -- 172.21.15.43:0/480848126 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c801c480 con 0x560e28b85410 2024-05-27T07:20:39.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.272+0000 7f83ccc00700 1 -- 172.21.15.43:0/480848126 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f83c801cab0 con 0x560e28b85410 2024-05-27T07:20:39.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.272+0000 7f83ccc00700 1 -- 172.21.15.43:0/480848126 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8011800 con 0x560e28b85410 2024-05-27T07:20:39.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.273+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/480848126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85410 msgr2=0x560e28b8fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:39.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.273+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85410 0x560e28b8fad0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f83c800bd30 tx=0x7f83c80106b0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.273+0000 7f83d7e00700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f83bc0b29f0 0x7f83bc0b4f00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x560e28b81690 tx=0x7f83d0004ad0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:39.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.273+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/480848126 shutdown_connections 2024-05-27T07:20:39.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.273+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85820 0x560e28b92020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.273+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b84ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.273+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/480848126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85410 0x560e28b8fad0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.273+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/480848126 >> 172.21.15.43:0/480848126 conn(0x560e28a66fc0 msgr2=0x560e28b80d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:39.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.274+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/480848126 shutdown_connections 2024-05-27T07:20:39.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.274+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/480848126 wait complete. 2024-05-27T07:20:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.275+0000 7f83ed2dbb80 1 -- start start 2024-05-27T07:20:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.275+0000 7f83ed2dbb80 1 Processor -- start 2024-05-27T07:20:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.275+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28a06e40 0x560e28b8f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.275+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85410 0x560e28b8fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.275+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28b85820 0x560e28b8ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.275+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560e289a9fa0 con 0x560e28a06e40 2024-05-27T07:20:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.275+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560e28a44760 con 0x560e28b85820 2024-05-27T07:20:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.275+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560e28b92990 con 0x560e28b85410 2024-05-27T07:20:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83ed2dbb80 10 client.0 _ll_register_callbacks cb 0x560e28a4dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:39.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83d7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28a06e40 0x560e28b8f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83d7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28a06e40 0x560e28b8f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39638/0 (socket says 172.21.15.43:39638) 2024-05-27T07:20:39.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83d7400700 1 -- 172.21.15.43:0/1421413616 learned_addr learned my addr 172.21.15.43:0/1421413616 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83d6a00700 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85410 0x560e28b8fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83d7e00700 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28b85820 0x560e28b8ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83d7400700 1 -- 172.21.15.43:0/1421413616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85410 msgr2=0x560e28b8fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83d7400700 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85410 0x560e28b8fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83d7400700 1 -- 172.21.15.43:0/1421413616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28b85820 msgr2=0x560e28b8ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83d7400700 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28b85820 0x560e28b8ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.276+0000 7f83d7400700 1 -- 172.21.15.43:0/1421413616 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e289f38d0 con 0x560e28a06e40 2024-05-27T07:20:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.277+0000 7f83d7400700 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28a06e40 0x560e28b8f500 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f83c800b900 tx=0x7f83c801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3456edff9d2058a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.277+0000 7f83c1600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:20:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.277+0000 7f83c1600700 1 -- 172.21.15.43:0/1421413616 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8004230 con 0x560e28a06e40 2024-05-27T07:20:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.277+0000 7f83c1600700 1 -- 172.21.15.43:0/1421413616 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f83c80043d0 con 0x560e28a06e40 2024-05-27T07:20:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.277+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1421413616 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83c801cab0 con 0x560e28a06e40 2024-05-27T07:20:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.278+0000 7f83c1600700 1 -- 172.21.15.43:0/1421413616 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f83c8005050 con 0x560e28a06e40 2024-05-27T07:20:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.278+0000 7f83ed2dbb80 10 client.16494 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.278+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1421413616 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x560e28a74060 con 0x560e28a06e40 2024-05-27T07:20:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.278+0000 7f83c1600700 1 -- 172.21.15.43:0/1421413616 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f83c801cab0 con 0x560e28a06e40 2024-05-27T07:20:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.278+0000 7f83c1600700 1 -- 172.21.15.43:0/1421413616 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f83c8031020 con 0x560e28a06e40 2024-05-27T07:20:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.278+0000 7f83ed2dbb80 10 client.16494 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.278+0000 7f83ed2dbb80 10 client.16494 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:20:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.278+0000 7f83ed2dbb80 20 client.16494 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.279+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560e28b9f8f0 0x560e28ba1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.279+0000 7f83ed2dbb80 4 client.16494 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:20:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.279+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1421413616 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e28ab5d90 con 0x560e28b9f8f0 2024-05-27T07:20:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.279+0000 7f83d6a00700 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560e28b9f8f0 0x560e28ba1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.280+0000 7f83d6a00700 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560e28b9f8f0 0x560e28ba1dc0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=815e766ebbeb17af server_cookie=eb476ac18e49a3e8 in_seq=0 out_seq=0 2024-05-27T07:20:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.280+0000 7f83c1600700 10 client.16494 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83c1600700 1 -- 172.21.15.43:0/1421413616 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560e28ab5d90 con 0x560e28b9f8f0 2024-05-27T07:20:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83c1600700 10 client.16494 handle_command_reply: tid=0 2024-05-27T07:20:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83ed2dbb80 1 client.16494 shutdown 2024-05-27T07:20:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1421413616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28a06e40 msgr2=0x560e28b8f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28a06e40 0x560e28b8f500 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f83c800b900 tx=0x7f83c801c530 comp rx=0 tx=0).stop 2024-05-27T07:20:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1421413616 shutdown_connections 2024-05-27T07:20:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28b85820 0x560e28b8ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560e28b9f8f0 0x560e28ba1dc0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28a06e40 0x560e28b8f500 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1421413616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85410 0x560e28b8fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.283+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1421413616 >> 172.21.15.43:0/1421413616 conn(0x560e28a66fc0 msgr2=0x560e28b7fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.284+0000 7f83c1600700 0 client.16494 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.284+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1421413616 shutdown_connections 2024-05-27T07:20:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.284+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1421413616 wait complete. 2024-05-27T07:20:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.284+0000 7f83ed2dbb80 20 client.16494 trim_cache size 0 max 16384 2024-05-27T07:20:39.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.311+0000 7f83ed2dbb80 1 Processor -- start 2024-05-27T07:20:39.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.311+0000 7f83ed2dbb80 1 -- start start 2024-05-27T07:20:39.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.312+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28a06e40 0x560e28b8e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.312+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28b85410 0x560e28b8acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.312+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 0x560e28b9bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.312+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81b30a0 con 0x560e28b85820 2024-05-27T07:20:39.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.312+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81b3770 con 0x560e28b85410 2024-05-27T07:20:39.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.312+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81b0e90 con 0x560e28a06e40 2024-05-27T07:20:39.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.312+0000 7f83d7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 0x560e28b9bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.312+0000 7f83d7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 0x560e28b9bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39652/0 (socket says 172.21.15.43:39652) 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.313+0000 7f83d7e00700 1 -- 172.21.15.43:0/4248643268 learned_addr learned my addr 172.21.15.43:0/4248643268 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.313+0000 7f83d6a00700 1 --2- 172.21.15.43:0/4248643268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28b85410 0x560e28b8acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.313+0000 7f83d7400700 1 --2- 172.21.15.43:0/4248643268 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28a06e40 0x560e28b8e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.313+0000 7f83d7e00700 1 -- 172.21.15.43:0/4248643268 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28a06e40 msgr2=0x560e28b8e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.313+0000 7f83d7e00700 1 --2- 172.21.15.43:0/4248643268 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28a06e40 0x560e28b8e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.313+0000 7f83d7e00700 1 -- 172.21.15.43:0/4248643268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28b85410 msgr2=0x560e28b8acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.313+0000 7f83d7e00700 1 --2- 172.21.15.43:0/4248643268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28b85410 0x560e28b8acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.313+0000 7f83d7e00700 1 -- 172.21.15.43:0/4248643268 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e28b8da70 con 0x560e28b85820 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.313+0000 7f83d7400700 1 --2- 172.21.15.43:0/4248643268 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28a06e40 0x560e28b8e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:20:39.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.314+0000 7f83d7e00700 1 --2- 172.21.15.43:0/4248643268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 0x560e28b9bac0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f83d0015740 tx=0x7f83d0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aeee00145ae44af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:39.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.314+0000 7f83c3e00700 1 -- 172.21.15.43:0/4248643268 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0021620 con 0x560e28b85820 2024-05-27T07:20:39.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.314+0000 7f83c3e00700 1 -- 172.21.15.43:0/4248643268 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f83d0026070 con 0x560e28b85820 2024-05-27T07:20:39.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.314+0000 7f83c3e00700 1 -- 172.21.15.43:0/4248643268 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0020a50 con 0x560e28b85820 2024-05-27T07:20:39.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.315+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/4248643268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 msgr2=0x560e28b9bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:39.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.315+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/4248643268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 0x560e28b9bac0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f83d0015740 tx=0x7f83d0013000 comp rx=0 tx=0).stop 2024-05-27T07:20:39.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.315+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/4248643268 shutdown_connections 2024-05-27T07:20:39.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.315+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/4248643268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 0x560e28b9bac0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.315+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/4248643268 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28a06e40 0x560e28b8e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.315+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/4248643268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28b85410 0x560e28b8acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.315+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/4248643268 >> 172.21.15.43:0/4248643268 conn(0x560e28a66fc0 msgr2=0x560e28b8f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:39.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.315+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/4248643268 shutdown_connections 2024-05-27T07:20:39.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.315+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/4248643268 wait complete. 2024-05-27T07:20:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.316+0000 7f83ed2dbb80 1 -- start start 2024-05-27T07:20:39.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.316+0000 7f83ed2dbb80 1 Processor -- start 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.317+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b93070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.317+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85410 0x560e28b935b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.317+0000 7f83ed2dbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 0x560e28c08480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.317+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81b1560 con 0x560e28b85820 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.317+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81b29d0 con 0x560e28a06e40 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.317+0000 7f83ed2dbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f83d81b1c30 con 0x560e28b85410 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.317+0000 7f83ed2dbb80 10 client.0 _ll_register_callbacks cb 0x560e28a4dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.318+0000 7f83d7400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b93070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.318+0000 7f83d7400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b93070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40270/0 (socket says 172.21.15.43:40270) 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.318+0000 7f83d6a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85410 0x560e28b935b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.318+0000 7f83d7400700 1 -- 172.21.15.43:0/1097582228 learned_addr learned my addr 172.21.15.43:0/1097582228 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:20:39.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.318+0000 7f83d7e00700 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 0x560e28c08480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.318+0000 7f83d7400700 1 -- 172.21.15.43:0/1097582228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85410 msgr2=0x560e28b935b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:39.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.318+0000 7f83d7400700 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85410 0x560e28b935b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.318+0000 7f83d7400700 1 -- 172.21.15.43:0/1097582228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 msgr2=0x560e28c08480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:20:39.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.318+0000 7f83d7400700 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 0x560e28c08480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.318+0000 7f83d7400700 1 -- 172.21.15.43:0/1097582228 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e28c089c0 con 0x560e28a06e40 2024-05-27T07:20:39.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.319+0000 7f83d7400700 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b93070 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f83c80037f0 tx=0x7f83c80056b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ecd4a2d344d7d5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:20:39.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.319+0000 7f83c1600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.319+0000 7f83c1600700 1 -- 172.21.15.43:0/1097582228 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8036020 con 0x560e28a06e40 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.319+0000 7f83c1600700 1 -- 172.21.15.43:0/1097582228 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9408+0+0 (secure 0 0 0) 0x7f83c8024390 con 0x560e28a06e40 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.319+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1097582228 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83d0026070 con 0x560e28a06e40 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.320+0000 7f83c1600700 1 -- 172.21.15.43:0/1097582228 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f83c803e050 con 0x560e28a06e40 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.320+0000 7f83ed2dbb80 10 client.36059 fetch_fsmap learned FSMap version 79 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.320+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1097582228 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x560e28c09af0 con 0x560e28a06e40 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.321+0000 7f83c1600700 1 -- 172.21.15.43:0/1097582228 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f83d0026070 con 0x560e28a06e40 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.321+0000 7f83c1600700 1 -- 172.21.15.43:0/1097582228 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f83c8039020 con 0x560e28a06e40 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.321+0000 7f83ed2dbb80 10 client.36059 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.321+0000 7f83ed2dbb80 10 client.36059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.321+0000 7f83ed2dbb80 20 client.36059 populate_metadata read hostname 'smithi043' 2024-05-27T07:20:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.321+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560e28c0a140 0x560e28c0c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:20:39.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.321+0000 7f83ed2dbb80 4 client.36059 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:20:39.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.321+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1097582228 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560e28aaf570 con 0x560e28c0a140 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.322+0000 7f83d6a00700 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560e28c0a140 0x560e28c0c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.323+0000 7f83d6a00700 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560e28c0a140 0x560e28c0c560 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58f2e8460314582 server_cookie=6c710a40b0edaf9f in_seq=0 out_seq=0 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.323+0000 7f83c1600700 10 client.36059 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.323+0000 7f83c1600700 1 -- 172.21.15.43:0/1097582228 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560e28aaf570 con 0x560e28c0a140 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.323+0000 7f83c1600700 10 client.36059 handle_command_reply: tid=0 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.323+0000 7f83ed2dbb80 1 client.36059 shutdown 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.323+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1097582228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 msgr2=0x560e28b93070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.323+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b93070 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f83c80037f0 tx=0x7f83c80056b0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.324+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1097582228 shutdown_connections 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.324+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560e28c0a140 0x560e28c0c560 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.324+0000 7f83c1600700 0 client.36059 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:20:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.324+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560e28b85820 0x560e28c08480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.324+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560e28a06e40 0x560e28b93070 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.324+0000 7f83ed2dbb80 1 --2- 172.21.15.43:0/1097582228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560e28b85410 0x560e28b935b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.324+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1097582228 >> 172.21.15.43:0/1097582228 conn(0x560e28a66fc0 msgr2=0x560e28b8e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:39.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.324+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1097582228 shutdown_connections 2024-05-27T07:20:39.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.324+0000 7f83ed2dbb80 1 -- 172.21.15.43:0/1097582228 wait complete. 2024-05-27T07:20:39.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.324+0000 7f83ed2dbb80 20 client.36059 trim_cache size 0 max 16384 2024-05-27T07:20:39.317 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:20:39.317 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:20:39.317 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:20:39.317 INFO:teuthology.orchestra.run.smithi043.stdout: "80c73548-c338-4802-9ffb-85a179f8bbec": { 2024-05-27T07:20:39.317 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:20:39.317 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "80c73548-c338-4802-9ffb-85a179f8bbec", 2024-05-27T07:20:39.318 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:20:39.318 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:20:39.318 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:20:39.318 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.325+0000 7f83dd200700 1 -- 172.21.15.43:0/69583000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f83bc0b29f0 msgr2=0x7f83bc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.325+0000 7f83dd200700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f83bc0b29f0 0x7f83bc0b4f00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x560e28b81690 tx=0x7f83d0004ad0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.325+0000 7f83dd200700 1 -- 172.21.15.43:0/69583000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83d8107c00 msgr2=0x7f83d81a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.325+0000 7f83dd200700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83d8107c00 0x7f83d81a0d10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f83c400b580 tx=0x7f83c40057c0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.326+0000 7f83dd200700 1 -- 172.21.15.43:0/69583000 shutdown_connections 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.326+0000 7f83dd200700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f83bc0b29f0 0x7f83bc0b4f00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.326+0000 7f83dd200700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83d8108550 0x7f83d819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.326+0000 7f83dd200700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d81032d0 0x7f83d81a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.326+0000 7f83dd200700 1 --2- 172.21.15.43:0/69583000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83d8107c00 0x7f83d81a0d10 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.326+0000 7f83dd200700 1 -- 172.21.15.43:0/69583000 >> 172.21.15.43:0/69583000 conn(0x7f83d8078ab0 msgr2=0x7f83d8106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.326+0000 7f83dd200700 1 -- 172.21.15.43:0/69583000 shutdown_connections 2024-05-27T07:20:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:20:39.326+0000 7f83dd200700 1 -- 172.21.15.43:0/69583000 wait complete. 2024-05-27T07:20:40.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 34272 0.62 MB/sec execute 164 sec latency 50.593 ms 2024-05-27T07:20:40.286 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:40 smithi043 ceph-mon[31531]: pgmap v225: 129 pgs: 129 active+clean; 58 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 282 KiB/s wr, 46 op/s 2024-05-27T07:20:40.290 INFO:tasks.cephfs.filesystem:scrub status for tag:80c73548-c338-4802-9ffb-85a179f8bbec - {'path': '/', 'tag': '80c73548-c338-4802-9ffb-85a179f8bbec', 'options': 'recursive,force'} 2024-05-27T07:20:40.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:40 smithi132 ceph-mon[35283]: pgmap v225: 129 pgs: 129 active+clean; 58 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 282 KiB/s wr, 46 op/s 2024-05-27T07:20:40.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:40 smithi179 ceph-mon[35888]: pgmap v225: 129 pgs: 129 active+clean; 58 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 282 KiB/s wr, 46 op/s 2024-05-27T07:20:41.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 34355 0.62 MB/sec execute 165 sec latency 52.876 ms 2024-05-27T07:20:42.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 34420 0.62 MB/sec execute 166 sec latency 72.209 ms 2024-05-27T07:20:42.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:42 smithi132 ceph-mon[35283]: pgmap v226: 129 pgs: 129 active+clean; 58 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 416 KiB/s wr, 61 op/s 2024-05-27T07:20:42.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:42 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:20:42.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:42 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:20:42.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:42 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:20:42.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:42 smithi043 ceph-mon[31531]: pgmap v226: 129 pgs: 129 active+clean; 58 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 416 KiB/s wr, 61 op/s 2024-05-27T07:20:42.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:42 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:20:42.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:42 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:20:42.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:42 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:20:42.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:42 smithi179 ceph-mon[35888]: pgmap v226: 129 pgs: 129 active+clean; 58 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 416 KiB/s wr, 61 op/s 2024-05-27T07:20:42.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:42 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:20:42.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:42 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:20:42.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:42 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:20:43.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 34544 0.62 MB/sec execute 167 sec latency 59.348 ms 2024-05-27T07:20:43.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:43 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:20:43.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:43 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:20:43.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:43 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:20:44.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 34723 0.63 MB/sec execute 168 sec latency 53.041 ms 2024-05-27T07:20:44.341 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:44 smithi132 ceph-mon[35283]: pgmap v227: 129 pgs: 129 active+clean; 58 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 288 KiB/s wr, 36 op/s 2024-05-27T07:20:44.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:44 smithi043 ceph-mon[31531]: pgmap v227: 129 pgs: 129 active+clean; 58 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 288 KiB/s wr, 36 op/s 2024-05-27T07:20:44.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:44 smithi179 ceph-mon[35888]: pgmap v227: 129 pgs: 129 active+clean; 58 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 288 KiB/s wr, 36 op/s 2024-05-27T07:20:45.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 34896 0.64 MB/sec execute 169 sec latency 51.394 ms 2024-05-27T07:20:46.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 34993 0.64 MB/sec execute 170 sec latency 56.504 ms 2024-05-27T07:20:46.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:46 smithi132 ceph-mon[35283]: pgmap v228: 129 pgs: 129 active+clean; 58 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 288 KiB/s wr, 36 op/s 2024-05-27T07:20:46.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:46 smithi043 ceph-mon[31531]: pgmap v228: 129 pgs: 129 active+clean; 58 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 288 KiB/s wr, 36 op/s 2024-05-27T07:20:46.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:46 smithi179 ceph-mon[35888]: pgmap v228: 129 pgs: 129 active+clean; 58 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 288 KiB/s wr, 36 op/s 2024-05-27T07:20:47.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35244 0.64 MB/sec execute 171 sec latency 56.237 ms 2024-05-27T07:20:48.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35278 0.64 MB/sec execute 172 sec latency 59.615 ms 2024-05-27T07:20:48.412 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:48 smithi043 ceph-mon[31531]: pgmap v229: 129 pgs: 129 active+clean; 61 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 538 KiB/s wr, 54 op/s 2024-05-27T07:20:48.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:48 smithi179 ceph-mon[35888]: pgmap v229: 129 pgs: 129 active+clean; 61 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 538 KiB/s wr, 54 op/s 2024-05-27T07:20:48.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:48 smithi132 ceph-mon[35283]: pgmap v229: 129 pgs: 129 active+clean; 61 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 538 KiB/s wr, 54 op/s 2024-05-27T07:20:49.203 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35302 0.64 MB/sec execute 173 sec latency 52.497 ms 2024-05-27T07:20:50.204 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35464 0.65 MB/sec execute 174 sec latency 51.654 ms 2024-05-27T07:20:50.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:50 smithi043 ceph-mon[31531]: pgmap v230: 129 pgs: 129 active+clean; 61 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 384 KiB/s wr, 33 op/s 2024-05-27T07:20:50.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:50 smithi179 ceph-mon[35888]: pgmap v230: 129 pgs: 129 active+clean; 61 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 384 KiB/s wr, 33 op/s 2024-05-27T07:20:50.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:50 smithi132 ceph-mon[35283]: pgmap v230: 129 pgs: 129 active+clean; 61 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 384 KiB/s wr, 33 op/s 2024-05-27T07:20:51.204 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35582 0.65 MB/sec execute 175 sec latency 56.179 ms 2024-05-27T07:20:52.204 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35670 0.64 MB/sec execute 176 sec latency 51.451 ms 2024-05-27T07:20:52.392 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:52 smithi132 ceph-mon[35283]: pgmap v231: 129 pgs: 129 active+clean; 62 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 538 KiB/s wr, 47 op/s 2024-05-27T07:20:52.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:52 smithi043 ceph-mon[31531]: pgmap v231: 129 pgs: 129 active+clean; 62 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 538 KiB/s wr, 47 op/s 2024-05-27T07:20:52.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:52 smithi179 ceph-mon[35888]: pgmap v231: 129 pgs: 129 active+clean; 62 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 538 KiB/s wr, 47 op/s 2024-05-27T07:20:53.204 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35754 0.65 MB/sec execute 177 sec latency 51.313 ms 2024-05-27T07:20:54.204 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35809 0.64 MB/sec execute 178 sec latency 56.103 ms 2024-05-27T07:20:54.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:54 smithi043 ceph-mon[31531]: pgmap v232: 129 pgs: 129 active+clean; 62 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 404 KiB/s wr, 32 op/s 2024-05-27T07:20:54.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:54 smithi179 ceph-mon[35888]: pgmap v232: 129 pgs: 129 active+clean; 62 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 404 KiB/s wr, 32 op/s 2024-05-27T07:20:54.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:54 smithi132 ceph-mon[35283]: pgmap v232: 129 pgs: 129 active+clean; 62 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 404 KiB/s wr, 32 op/s 2024-05-27T07:20:55.204 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35874 0.64 MB/sec execute 179 sec latency 52.189 ms 2024-05-27T07:20:56.205 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35926 0.64 MB/sec execute 180 sec latency 49.697 ms 2024-05-27T07:20:56.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:56 smithi043 ceph-mon[31531]: pgmap v233: 129 pgs: 129 active+clean; 62 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 404 KiB/s wr, 32 op/s 2024-05-27T07:20:56.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:56 smithi179 ceph-mon[35888]: pgmap v233: 129 pgs: 129 active+clean; 62 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 404 KiB/s wr, 32 op/s 2024-05-27T07:20:56.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:56 smithi132 ceph-mon[35283]: pgmap v233: 129 pgs: 129 active+clean; 62 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 404 KiB/s wr, 32 op/s 2024-05-27T07:20:57.205 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35982 0.63 MB/sec execute 181 sec latency 53.850 ms 2024-05-27T07:20:58.205 INFO:tasks.workunit.client.0.smithi043.stdout: 1 36055 0.63 MB/sec execute 182 sec latency 57.574 ms 2024-05-27T07:20:58.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:20:58 smithi043 ceph-mon[31531]: pgmap v234: 129 pgs: 129 active+clean; 63 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 591 KiB/s wr, 52 op/s 2024-05-27T07:20:58.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:20:58 smithi179 ceph-mon[35888]: pgmap v234: 129 pgs: 129 active+clean; 63 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 591 KiB/s wr, 52 op/s 2024-05-27T07:20:58.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:20:58 smithi132 ceph-mon[35283]: pgmap v234: 129 pgs: 129 active+clean; 63 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 591 KiB/s wr, 52 op/s 2024-05-27T07:20:59.205 INFO:tasks.workunit.client.0.smithi043.stdout: 1 36119 0.63 MB/sec execute 183 sec latency 54.490 ms 2024-05-27T07:21:00.205 INFO:tasks.workunit.client.0.smithi043.stdout: 1 36227 0.63 MB/sec execute 184 sec latency 55.610 ms 2024-05-27T07:21:00.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:00 smithi043 ceph-mon[31531]: pgmap v235: 129 pgs: 129 active+clean; 63 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 341 KiB/s wr, 33 op/s 2024-05-27T07:21:00.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:00 smithi179 ceph-mon[35888]: pgmap v235: 129 pgs: 129 active+clean; 63 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 341 KiB/s wr, 33 op/s 2024-05-27T07:21:00.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:00 smithi132 ceph-mon[35283]: pgmap v235: 129 pgs: 129 active+clean; 63 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 341 KiB/s wr, 33 op/s 2024-05-27T07:21:01.205 INFO:tasks.workunit.client.0.smithi043.stdout: 1 36336 0.62 MB/sec execute 185 sec latency 74.028 ms 2024-05-27T07:21:02.205 INFO:tasks.workunit.client.0.smithi043.stdout: 1 36447 0.62 MB/sec execute 186 sec latency 64.596 ms 2024-05-27T07:21:02.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:02 smithi043 ceph-mon[31531]: pgmap v236: 129 pgs: 129 active+clean; 62 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 460 KiB/s wr, 70 op/s 2024-05-27T07:21:02.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:02 smithi179 ceph-mon[35888]: pgmap v236: 129 pgs: 129 active+clean; 62 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 460 KiB/s wr, 70 op/s 2024-05-27T07:21:02.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:02 smithi132 ceph-mon[35283]: pgmap v236: 129 pgs: 129 active+clean; 62 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 460 KiB/s wr, 70 op/s 2024-05-27T07:21:03.205 INFO:tasks.workunit.client.0.smithi043.stdout: 1 36563 0.62 MB/sec execute 187 sec latency 75.886 ms 2024-05-27T07:21:03.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:03 smithi043 ceph-mon[31531]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:21:03.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:03 smithi179 ceph-mon[35888]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:21:03.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:03 smithi132 ceph-mon[35283]: osdmap e112: 12 total, 12 up, 12 in 2024-05-27T07:21:04.205 INFO:tasks.workunit.client.0.smithi043.stdout: 1 36665 0.62 MB/sec execute 188 sec latency 59.965 ms 2024-05-27T07:21:04.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:04 smithi043 ceph-mon[31531]: pgmap v238: 129 pgs: 129 active+clean; 62 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 367 KiB/s wr, 68 op/s 2024-05-27T07:21:04.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:04 smithi179 ceph-mon[35888]: pgmap v238: 129 pgs: 129 active+clean; 62 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 367 KiB/s wr, 68 op/s 2024-05-27T07:21:04.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:04 smithi132 ceph-mon[35283]: pgmap v238: 129 pgs: 129 active+clean; 62 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 367 KiB/s wr, 68 op/s 2024-05-27T07:21:05.205 INFO:tasks.workunit.client.0.smithi043.stdout: 1 36760 0.62 MB/sec execute 189 sec latency 60.637 ms 2024-05-27T07:21:06.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 36918 0.63 MB/sec execute 190 sec latency 57.724 ms 2024-05-27T07:21:06.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:06 smithi043 ceph-mon[31531]: pgmap v239: 129 pgs: 129 active+clean; 62 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 367 KiB/s wr, 68 op/s 2024-05-27T07:21:06.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:06 smithi179 ceph-mon[35888]: pgmap v239: 129 pgs: 129 active+clean; 62 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 367 KiB/s wr, 68 op/s 2024-05-27T07:21:06.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:06 smithi132 ceph-mon[35283]: pgmap v239: 129 pgs: 129 active+clean; 62 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 367 KiB/s wr, 68 op/s 2024-05-27T07:21:07.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 37007 0.63 MB/sec execute 191 sec latency 66.824 ms 2024-05-27T07:21:08.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 37117 0.62 MB/sec execute 192 sec latency 70.931 ms 2024-05-27T07:21:08.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:08 smithi179 ceph-mon[35888]: pgmap v240: 129 pgs: 129 active+clean; 63 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 394 KiB/s wr, 73 op/s 2024-05-27T07:21:08.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:08 smithi132 ceph-mon[35283]: pgmap v240: 129 pgs: 129 active+clean; 63 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 394 KiB/s wr, 73 op/s 2024-05-27T07:21:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:08 smithi043 ceph-mon[31531]: pgmap v240: 129 pgs: 129 active+clean; 63 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 394 KiB/s wr, 73 op/s 2024-05-27T07:21:09.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 37203 0.62 MB/sec execute 193 sec latency 71.400 ms 2024-05-27T07:21:10.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 37337 0.62 MB/sec execute 194 sec latency 67.595 ms 2024-05-27T07:21:10.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:21:10.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:10 smithi132 ceph-mon[35283]: pgmap v241: 129 pgs: 129 active+clean; 63 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 394 KiB/s wr, 73 op/s 2024-05-27T07:21:10.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:10 smithi043 ceph-mon[31531]: pgmap v241: 129 pgs: 129 active+clean; 63 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 394 KiB/s wr, 73 op/s 2024-05-27T07:21:10.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:10 smithi179 ceph-mon[35888]: pgmap v241: 129 pgs: 129 active+clean; 63 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 394 KiB/s wr, 73 op/s 2024-05-27T07:21:10.783 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:11.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 37482 0.62 MB/sec execute 195 sec latency 56.070 ms 2024-05-27T07:21:12.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 37599 0.61 MB/sec execute 196 sec latency 69.408 ms 2024-05-27T07:21:12.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:12 smithi043 ceph-mon[31531]: pgmap v242: 129 pgs: 129 active+clean; 64 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 327 KiB/s wr, 48 op/s 2024-05-27T07:21:12.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:12 smithi043 ceph-mon[31531]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:21:12.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:12 smithi179 ceph-mon[35888]: pgmap v242: 129 pgs: 129 active+clean; 64 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 327 KiB/s wr, 48 op/s 2024-05-27T07:21:12.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:12 smithi179 ceph-mon[35888]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:21:12.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:12 smithi132 ceph-mon[35283]: pgmap v242: 129 pgs: 129 active+clean; 64 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 327 KiB/s wr, 48 op/s 2024-05-27T07:21:12.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:12 smithi132 ceph-mon[35283]: osdmap e113: 12 total, 12 up, 12 in 2024-05-27T07:21:13.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.166+0000 7fa48ce00700 1 -- 172.21.15.43:0/1227648244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa488110ad0 msgr2=0x7fa488112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.166+0000 7fa48ce00700 1 --2- 172.21.15.43:0/1227648244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa488110ad0 0x7fa488112ef0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa48000b0a0 tx=0x7fa48000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.167+0000 7fa48ce00700 1 -- 172.21.15.43:0/1227648244 shutdown_connections 2024-05-27T07:21:13.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.167+0000 7fa48ce00700 1 --2- 172.21.15.43:0/1227648244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa488110ad0 0x7fa488112ef0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.167+0000 7fa48ce00700 1 --2- 172.21.15.43:0/1227648244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa488102f20 0x7fa488103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.167+0000 7fa48ce00700 1 --2- 172.21.15.43:0/1227648244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa488103870 0x7fa488110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.167+0000 7fa48ce00700 1 -- 172.21.15.43:0/1227648244 >> 172.21.15.43:0/1227648244 conn(0x7fa488078ab0 msgr2=0x7fa488078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.167+0000 7fa48ce00700 1 -- 172.21.15.43:0/1227648244 shutdown_connections 2024-05-27T07:21:13.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.167+0000 7fa48ce00700 1 -- 172.21.15.43:0/1227648244 wait complete. 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.168+0000 7fa48ce00700 1 Processor -- start 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.168+0000 7fa48ce00700 1 -- start start 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.169+0000 7fa48ce00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa488102f20 0x7fa48810d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.169+0000 7fa48ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa488103870 0x7fa48810df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.169+0000 7fa48ce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa488110ad0 0x7fa48810e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.169+0000 7fa48ce00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa488116350 con 0x7fa488103870 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.169+0000 7fa48ce00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa4881161d0 con 0x7fa488110ad0 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.169+0000 7fa48ce00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa4881164d0 con 0x7fa488102f20 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.169+0000 7fa486a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa488103870 0x7fa48810df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.169+0000 7fa486a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa488103870 0x7fa48810df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58514/0 (socket says 172.21.15.43:58514) 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.169+0000 7fa486a00700 1 -- 172.21.15.43:0/2993019999 learned_addr learned my addr 172.21.15.43:0/2993019999 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:13.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.170+0000 7fa487e00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa488110ad0 0x7fa48810e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.170+0000 7fa487400700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa488102f20 0x7fa48810d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.170+0000 7fa486a00700 1 -- 172.21.15.43:0/2993019999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa488102f20 msgr2=0x7fa48810d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.170+0000 7fa486a00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa488102f20 0x7fa48810d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.170+0000 7fa486a00700 1 -- 172.21.15.43:0/2993019999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa488110ad0 msgr2=0x7fa48810e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.170+0000 7fa486a00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa488110ad0 0x7fa48810e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.170+0000 7fa486a00700 1 -- 172.21.15.43:0/2993019999 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4881ad250 con 0x7fa488103870 2024-05-27T07:21:13.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.171+0000 7fa486a00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa488103870 0x7fa48810df10 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa47400d950 tx=0x7fa47400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.171+0000 7fa47fe00700 1 -- 172.21.15.43:0/2993019999 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa474017070 con 0x7fa488103870 2024-05-27T07:21:13.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.171+0000 7fa47fe00700 1 -- 172.21.15.43:0/2993019999 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa474009730 con 0x7fa488103870 2024-05-27T07:21:13.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.171+0000 7fa47fe00700 1 -- 172.21.15.43:0/2993019999 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa474003a00 con 0x7fa488103870 2024-05-27T07:21:13.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.171+0000 7fa48ce00700 1 -- 172.21.15.43:0/2993019999 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4881ad540 con 0x7fa488103870 2024-05-27T07:21:13.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.171+0000 7fa48ce00700 1 -- 172.21.15.43:0/2993019999 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4881ada30 con 0x7fa488103870 2024-05-27T07:21:13.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.176+0000 7fa47fe00700 1 -- 172.21.15.43:0/2993019999 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa474003ba0 con 0x7fa488103870 2024-05-27T07:21:13.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.177+0000 7fa49cf6bb80 1 Processor -- start 2024-05-27T07:21:13.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.177+0000 7fa49cf6bb80 1 -- start start 2024-05-27T07:21:13.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789cffe40 0x564789e7ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789e7e410 0x564789e88ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e8b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564789dace40 con 0x564789e7e410 2024-05-27T07:21:13.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564789d9c9e0 con 0x564789cffe40 2024-05-27T07:21:13.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564789da0a20 con 0x564789e7e820 2024-05-27T07:21:13.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa487400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789e7e410 0x564789e88ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa487400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789e7e410 0x564789e88ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58518/0 (socket says 172.21.15.43:58518) 2024-05-27T07:21:13.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa487400700 1 -- 172.21.15.43:0/2871473594 learned_addr learned my addr 172.21.15.43:0/2871473594 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:13.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa487e00700 1 --2- 172.21.15.43:0/2871473594 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789cffe40 0x564789e7ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.178+0000 7fa486a00700 1 --2- 172.21.15.43:0/2871473594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e8b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.179+0000 7fa487400700 1 -- 172.21.15.43:0/2871473594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 msgr2=0x564789e8b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.179+0000 7fa487400700 1 --2- 172.21.15.43:0/2871473594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e8b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.179+0000 7fa47fe00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa46c0b29f0 0x7fa46c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.179+0000 7fa487400700 1 -- 172.21.15.43:0/2871473594 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789cffe40 msgr2=0x564789e7ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.179+0000 7fa487400700 1 --2- 172.21.15.43:0/2871473594 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789cffe40 0x564789e7ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.179+0000 7fa487400700 1 -- 172.21.15.43:0/2871473594 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564789d65420 con 0x564789e7e410 2024-05-27T07:21:13.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.179+0000 7fa47fe00700 1 -- 172.21.15.43:0/2993019999 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7fa4740e23d0 con 0x7fa488103870 2024-05-27T07:21:13.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.179+0000 7fa487e00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa46c0b29f0 0x7fa46c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.179+0000 7fa487400700 1 --2- 172.21.15.43:0/2871473594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789e7e410 0x564789e88ad0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa47800bd70 tx=0x7fa4780106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7acc983033ff2f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.180+0000 7fa47cc00700 1 -- 172.21.15.43:0/2871473594 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478010e10 con 0x564789e7e410 2024-05-27T07:21:13.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.180+0000 7fa47cc00700 1 -- 172.21.15.43:0/2871473594 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa47801c910 con 0x564789e7e410 2024-05-27T07:21:13.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.180+0000 7fa47cc00700 1 -- 172.21.15.43:0/2871473594 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478011870 con 0x564789e7e410 2024-05-27T07:21:13.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.181+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/2871473594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789e7e410 msgr2=0x564789e88ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.181+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/2871473594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789e7e410 0x564789e88ad0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa47800bd70 tx=0x7fa4780106b0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.181+0000 7fa487e00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa46c0b29f0 0x7fa46c0b4f00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x564789e7a690 tx=0x7fa4800099e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.181+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/2871473594 shutdown_connections 2024-05-27T07:21:13.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.181+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/2871473594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e8b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.181+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/2871473594 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789cffe40 0x564789e7ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.181+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/2871473594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789e7e410 0x564789e88ad0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.181+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/2871473594 >> 172.21.15.43:0/2871473594 conn(0x564789d5ffc0 msgr2=0x564789e79d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.181+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/2871473594 shutdown_connections 2024-05-27T07:21:13.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.181+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/2871473594 wait complete. 2024-05-27T07:21:13.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.182+0000 7fa49cf6bb80 1 -- start start 2024-05-27T07:21:13.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.182+0000 7fa49cf6bb80 1 Processor -- start 2024-05-27T07:21:13.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.183+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e88500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.183+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 0x564789e88a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.183+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e88f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.183+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564789ca2fa0 con 0x564789cffe40 2024-05-27T07:21:13.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.183+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564789d3d760 con 0x564789e7e410 2024-05-27T07:21:13.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.183+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564789e8b990 con 0x564789e7e820 2024-05-27T07:21:13.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.183+0000 7fa49cf6bb80 10 client.0 _ll_register_callbacks cb 0x564789d46c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:13.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.183+0000 7fa487400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e88500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa487400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e88500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58534/0 (socket says 172.21.15.43:58534) 2024-05-27T07:21:13.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa487400700 1 -- 172.21.15.43:0/3988395726 learned_addr learned my addr 172.21.15.43:0/3988395726 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa486a00700 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 0x564789e88a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa487400700 1 -- 172.21.15.43:0/3988395726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 msgr2=0x564789e88f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa487e00700 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e88f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa487400700 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e88f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa487400700 1 -- 172.21.15.43:0/3988395726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 msgr2=0x564789e88a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa487400700 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 0x564789e88a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa487400700 1 -- 172.21.15.43:0/3988395726 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564789cec8d0 con 0x564789cffe40 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa487e00700 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e88f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.184+0000 7fa486a00700 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 0x564789e88a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.185+0000 7fa487400700 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e88500 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa47800bd40 tx=0x7fa478010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc53d08c943160b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.185+0000 7fa471600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:13.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.185+0000 7fa471600700 1 -- 172.21.15.43:0/3988395726 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478029060 con 0x564789cffe40 2024-05-27T07:21:13.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.185+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/3988395726 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa47801c910 con 0x564789cffe40 2024-05-27T07:21:13.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.185+0000 7fa471600700 1 -- 172.21.15.43:0/3988395726 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7fa47801ccc0 con 0x564789cffe40 2024-05-27T07:21:13.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.186+0000 7fa471600700 1 -- 172.21.15.43:0/3988395726 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fa478030050 con 0x564789cffe40 2024-05-27T07:21:13.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.186+0000 7fa49cf6bb80 10 client.16512 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:13.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.186+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/3988395726 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x564789d6d060 con 0x564789cffe40 2024-05-27T07:21:13.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.186+0000 7fa471600700 1 -- 172.21.15.43:0/3988395726 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa47801c910 con 0x564789cffe40 2024-05-27T07:21:13.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.186+0000 7fa471600700 1 -- 172.21.15.43:0/3988395726 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fa478003a80 con 0x564789cffe40 2024-05-27T07:21:13.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.187+0000 7fa49cf6bb80 10 client.16512 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:13.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.187+0000 7fa49cf6bb80 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:21:13.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.187+0000 7fa49cf6bb80 20 client.16512 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:13.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.187+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x564789e98ad0 0x564789e9afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.187+0000 7fa49cf6bb80 4 client.16512 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:13.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.187+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/3988395726 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564789daed90 con 0x564789e98ad0 2024-05-27T07:21:13.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.188+0000 7fa486a00700 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x564789e98ad0 0x564789e9afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.189+0000 7fa486a00700 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x564789e98ad0 0x564789e9afa0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f462e4853f8cbc7 server_cookie=268c1a8956147170 in_seq=0 out_seq=0 2024-05-27T07:21:13.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.189+0000 7fa471600700 10 client.16512 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:13.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.192+0000 7fa471600700 1 -- 172.21.15.43:0/3988395726 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564789daed90 con 0x564789e98ad0 2024-05-27T07:21:13.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.192+0000 7fa471600700 10 client.16512 handle_command_reply: tid=0 2024-05-27T07:21:13.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.192+0000 7fa49cf6bb80 1 client.16512 shutdown 2024-05-27T07:21:13.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.192+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/3988395726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 msgr2=0x564789e88500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.192+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e88500 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa47800bd40 tx=0x7fa478010ef0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.193+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/3988395726 shutdown_connections 2024-05-27T07:21:13.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.193+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e88f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.193+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x564789e98ad0 0x564789e9afa0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.193+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e88500 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.193+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/3988395726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 0x564789e88a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.193+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/3988395726 >> 172.21.15.43:0/3988395726 conn(0x564789d5ffc0 msgr2=0x564789e78ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.193+0000 7fa471600700 0 client.16512 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:13.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.193+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/3988395726 shutdown_connections 2024-05-27T07:21:13.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.193+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/3988395726 wait complete. 2024-05-27T07:21:13.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.194+0000 7fa49cf6bb80 20 client.16512 trim_cache size 0 max 16384 2024-05-27T07:21:13.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 37758 0.61 MB/sec execute 197 sec latency 53.439 ms 2024-05-27T07:21:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.219+0000 7fa49cf6bb80 1 Processor -- start 2024-05-27T07:21:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.219+0000 7fa49cf6bb80 1 -- start start 2024-05-27T07:21:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.219+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e83cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.219+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 0x564789e94ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.219+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e889f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.219+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa46800f9e0 con 0x564789cffe40 2024-05-27T07:21:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.219+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa4680100b0 con 0x564789e7e410 2024-05-27T07:21:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.219+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa46800d7d0 con 0x564789e7e820 2024-05-27T07:21:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa487400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e83cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa487400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e83cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58540/0 (socket says 172.21.15.43:58540) 2024-05-27T07:21:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa487400700 1 -- 172.21.15.43:0/735136731 learned_addr learned my addr 172.21.15.43:0/735136731 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa487e00700 1 --2- 172.21.15.43:0/735136731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e889f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa486a00700 1 --2- 172.21.15.43:0/735136731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 0x564789e94ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa487400700 1 -- 172.21.15.43:0/735136731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 msgr2=0x564789e889f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa487400700 1 --2- 172.21.15.43:0/735136731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e889f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa487400700 1 -- 172.21.15.43:0/735136731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 msgr2=0x564789e94ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa487400700 1 --2- 172.21.15.43:0/735136731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 0x564789e94ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa487400700 1 -- 172.21.15.43:0/735136731 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564789e86a70 con 0x564789cffe40 2024-05-27T07:21:13.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.220+0000 7fa486a00700 1 --2- 172.21.15.43:0/735136731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 0x564789e94ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:13.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.221+0000 7fa487400700 1 --2- 172.21.15.43:0/735136731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e83cf0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa478003e80 tx=0x7fa478019e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fdc8a7650d8354 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.221+0000 7fa473e00700 1 -- 172.21.15.43:0/735136731 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478029040 con 0x564789cffe40 2024-05-27T07:21:13.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.221+0000 7fa473e00700 1 -- 172.21.15.43:0/735136731 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa47800b410 con 0x564789cffe40 2024-05-27T07:21:13.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.221+0000 7fa473e00700 1 -- 172.21.15.43:0/735136731 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478011d60 con 0x564789cffe40 2024-05-27T07:21:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.222+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/735136731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 msgr2=0x564789e83cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.222+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/735136731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e83cf0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa478003e80 tx=0x7fa478019e90 comp rx=0 tx=0).stop 2024-05-27T07:21:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.222+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/735136731 shutdown_connections 2024-05-27T07:21:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.222+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/735136731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e820 0x564789e889f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.222+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/735136731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e83cf0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.222+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/735136731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e410 0x564789e94ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.222+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/735136731 >> 172.21.15.43:0/735136731 conn(0x564789d5ffc0 msgr2=0x564789e882e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.222+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/735136731 shutdown_connections 2024-05-27T07:21:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.222+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/735136731 wait complete. 2024-05-27T07:21:13.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.223+0000 7fa49cf6bb80 1 -- start start 2024-05-27T07:21:13.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.223+0000 7fa49cf6bb80 1 Processor -- start 2024-05-27T07:21:13.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.224+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e91500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.224+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e410 0x564789e8c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.224+0000 7fa49cf6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e820 0x564789e8cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.224+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa46800dea0 con 0x564789cffe40 2024-05-27T07:21:13.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.224+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa46800f310 con 0x564789e7e820 2024-05-27T07:21:13.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.224+0000 7fa49cf6bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa46800e570 con 0x564789e7e410 2024-05-27T07:21:13.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.224+0000 7fa49cf6bb80 10 client.0 _ll_register_callbacks cb 0x564789d46c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:13.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.224+0000 7fa487400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e91500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.224+0000 7fa487400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e91500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58554/0 (socket says 172.21.15.43:58554) 2024-05-27T07:21:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.224+0000 7fa487400700 1 -- 172.21.15.43:0/1594685273 learned_addr learned my addr 172.21.15.43:0/1594685273 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.225+0000 7fa486a00700 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e410 0x564789e8c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.225+0000 7fa487e00700 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e820 0x564789e8cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.225+0000 7fa487400700 1 -- 172.21.15.43:0/1594685273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e410 msgr2=0x564789e8c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.225+0000 7fa487400700 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e410 0x564789e8c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.225+0000 7fa487400700 1 -- 172.21.15.43:0/1594685273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e820 msgr2=0x564789e8cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.225+0000 7fa487400700 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e820 0x564789e8cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.225+0000 7fa487400700 1 -- 172.21.15.43:0/1594685273 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564789e8d6a0 con 0x564789cffe40 2024-05-27T07:21:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.226+0000 7fa487400700 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e91500 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa478005d90 tx=0x7fa4780053b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a2adf68e0ed340 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.226+0000 7fa471600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:13.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.226+0000 7fa471600700 1 -- 172.21.15.43:0/1594685273 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478029040 con 0x564789cffe40 2024-05-27T07:21:13.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.226+0000 7fa471600700 1 -- 172.21.15.43:0/1594685273 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7fa478003a90 con 0x564789cffe40 2024-05-27T07:21:13.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.226+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/1594685273 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564789da8570 con 0x564789cffe40 2024-05-27T07:21:13.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.226+0000 7fa471600700 1 -- 172.21.15.43:0/1594685273 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fa478003c90 con 0x564789cffe40 2024-05-27T07:21:13.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.226+0000 7fa49cf6bb80 10 client.16518 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:13.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.226+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/1594685273 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x564789f035f0 con 0x564789cffe40 2024-05-27T07:21:13.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.227+0000 7fa471600700 1 -- 172.21.15.43:0/1594685273 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x564789da8570 con 0x564789cffe40 2024-05-27T07:21:13.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.227+0000 7fa471600700 1 -- 172.21.15.43:0/1594685273 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fa478011640 con 0x564789cffe40 2024-05-27T07:21:13.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.227+0000 7fa49cf6bb80 10 client.16518 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:13.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.227+0000 7fa49cf6bb80 10 client.16518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:21:13.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.227+0000 7fa49cf6bb80 20 client.16518 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:13.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.227+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x564789f03a00 0x564789f05e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:13.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.227+0000 7fa49cf6bb80 4 client.16518 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:21:13.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.227+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/1594685273 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564789c1f250 con 0x564789f03a00 2024-05-27T07:21:13.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.228+0000 7fa486a00700 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x564789f03a00 0x564789f05e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:13.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.229+0000 7fa486a00700 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x564789f03a00 0x564789f05e20 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6edf94067597b19 server_cookie=a45d9b38697f3bf9 in_seq=0 out_seq=0 2024-05-27T07:21:13.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.229+0000 7fa471600700 10 client.16518 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:13.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.229+0000 7fa471600700 1 -- 172.21.15.43:0/1594685273 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564789c1f250 con 0x564789f03a00 2024-05-27T07:21:13.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.229+0000 7fa471600700 10 client.16518 handle_command_reply: tid=0 2024-05-27T07:21:13.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.229+0000 7fa49cf6bb80 1 client.16518 shutdown 2024-05-27T07:21:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.229+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/1594685273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 msgr2=0x564789e91500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.229+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e91500 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa478005d90 tx=0x7fa4780053b0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.230+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/1594685273 shutdown_connections 2024-05-27T07:21:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.230+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564789e7e820 0x564789e8cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.230+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x564789f03a00 0x564789f05e20 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.230+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564789cffe40 0x564789e91500 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.230+0000 7fa49cf6bb80 1 --2- 172.21.15.43:0/1594685273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564789e7e410 0x564789e8c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.230+0000 7fa471600700 0 client.16518 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.230+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/1594685273 >> 172.21.15.43:0/1594685273 conn(0x564789d5ffc0 msgr2=0x564789e88d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.230+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/1594685273 shutdown_connections 2024-05-27T07:21:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.230+0000 7fa49cf6bb80 1 -- 172.21.15.43:0/1594685273 wait complete. 2024-05-27T07:21:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.230+0000 7fa49cf6bb80 20 client.16518 trim_cache size 0 max 16384 2024-05-27T07:21:13.221 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:21:13.221 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:21:13.221 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:21:13.221 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:21:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 -- 172.21.15.43:0/2993019999 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa46c0b29f0 msgr2=0x7fa46c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa46c0b29f0 0x7fa46c0b4f00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x564789e7a690 tx=0x7fa4800099e0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 -- 172.21.15.43:0/2993019999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa488103870 msgr2=0x7fa48810df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa488103870 0x7fa48810df10 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa47400d950 tx=0x7fa47400dc60 comp rx=0 tx=0).stop 2024-05-27T07:21:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 -- 172.21.15.43:0/2993019999 shutdown_connections 2024-05-27T07:21:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa46c0b29f0 0x7fa46c0b4f00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa488110ad0 0x7fa48810e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa488102f20 0x7fa48810d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 --2- 172.21.15.43:0/2993019999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa488103870 0x7fa48810df10 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:13.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 -- 172.21.15.43:0/2993019999 >> 172.21.15.43:0/2993019999 conn(0x7fa488078ab0 msgr2=0x7fa488111e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:13.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 -- 172.21.15.43:0/2993019999 shutdown_connections 2024-05-27T07:21:13.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:13.231+0000 7fa48ce00700 1 -- 172.21.15.43:0/2993019999 wait complete. 2024-05-27T07:21:14.031 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:21:14.031 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:21:14.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 37880 0.61 MB/sec execute 198 sec latency 47.006 ms 2024-05-27T07:21:14.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:14 smithi043 ceph-mon[31531]: pgmap v244: 129 pgs: 129 active+clean; 64 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 327 KiB/s wr, 48 op/s 2024-05-27T07:21:14.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:14 smithi179 ceph-mon[35888]: pgmap v244: 129 pgs: 129 active+clean; 64 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 327 KiB/s wr, 48 op/s 2024-05-27T07:21:14.505 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:14.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:14 smithi132 ceph-mon[35283]: pgmap v244: 129 pgs: 129 active+clean; 64 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 327 KiB/s wr, 48 op/s 2024-05-27T07:21:15.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 37982 0.62 MB/sec execute 199 sec latency 47.823 ms 2024-05-27T07:21:16.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 38099 0.61 MB/sec execute 200 sec latency 60.442 ms 2024-05-27T07:21:16.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:16 smithi043 ceph-mon[31531]: pgmap v245: 129 pgs: 129 active+clean; 64 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 327 KiB/s wr, 48 op/s 2024-05-27T07:21:16.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:16 smithi179 ceph-mon[35888]: pgmap v245: 129 pgs: 129 active+clean; 64 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 327 KiB/s wr, 48 op/s 2024-05-27T07:21:16.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:16 smithi132 ceph-mon[35283]: pgmap v245: 129 pgs: 129 active+clean; 64 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 327 KiB/s wr, 48 op/s 2024-05-27T07:21:16.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.822+0000 7f696e800700 1 -- 172.21.15.43:0/4274774435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6968076670 msgr2=0x7f6968110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:16.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.822+0000 7f696e800700 1 --2- 172.21.15.43:0/4274774435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6968076670 0x7f6968110460 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f6954009990 tx=0x7f6954009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.822+0000 7f696e800700 1 -- 172.21.15.43:0/4274774435 shutdown_connections 2024-05-27T07:21:16.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.822+0000 7f696e800700 1 --2- 172.21.15.43:0/4274774435 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69681109a0 0x7f6968112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.822+0000 7f696e800700 1 --2- 172.21.15.43:0/4274774435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6968075d20 0x7f6968076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.822+0000 7f696e800700 1 --2- 172.21.15.43:0/4274774435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6968076670 0x7f6968110460 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.822+0000 7f696e800700 1 -- 172.21.15.43:0/4274774435 >> 172.21.15.43:0/4274774435 conn(0x7f6968079d00 msgr2=0x7f696807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:16.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.822+0000 7f696e800700 1 -- 172.21.15.43:0/4274774435 shutdown_connections 2024-05-27T07:21:16.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.822+0000 7f696e800700 1 -- 172.21.15.43:0/4274774435 wait complete. 2024-05-27T07:21:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.824+0000 7f696e800700 1 Processor -- start 2024-05-27T07:21:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.824+0000 7f696e800700 1 -- start start 2024-05-27T07:21:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6968075d20 0x7f696819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6968076670 0x7f696819d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696e800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69681109a0 0x7f69681a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696e800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f69681163a0 con 0x7f6968075d20 2024-05-27T07:21:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696e800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6968116220 con 0x7f6968076670 2024-05-27T07:21:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696e800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6968116520 con 0x7f69681109a0 2024-05-27T07:21:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6968075d20 0x7f696819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:16.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6968075d20 0x7f696819cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37302/0 (socket says 172.21.15.43:37302) 2024-05-27T07:21:16.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696d400700 1 -- 172.21.15.43:0/481945901 learned_addr learned my addr 172.21.15.43:0/481945901 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:16.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696ca00700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6968076670 0x7f696819d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696de00700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69681109a0 0x7f69681a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696d400700 1 -- 172.21.15.43:0/481945901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69681109a0 msgr2=0x7f69681a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696d400700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69681109a0 0x7f69681a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.825+0000 7f696d400700 1 -- 172.21.15.43:0/481945901 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6968076670 msgr2=0x7f696819d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.826+0000 7f696d400700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6968076670 0x7f696819d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.826+0000 7f696d400700 1 -- 172.21.15.43:0/481945901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69681a1cc0 con 0x7f6968075d20 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.826+0000 7f696de00700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69681109a0 0x7f69681a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.826+0000 7f696d400700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6968075d20 0x7f696819cdf0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f695c00d950 tx=0x7f695c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.826+0000 7f6966000700 1 -- 172.21.15.43:0/481945901 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c016070 con 0x7f6968075d20 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.826+0000 7f6966000700 1 -- 172.21.15.43:0/481945901 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f695c010460 con 0x7f6968075d20 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.826+0000 7f6966000700 1 -- 172.21.15.43:0/481945901 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c00f620 con 0x7f6968075d20 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.826+0000 7f696e800700 1 -- 172.21.15.43:0/481945901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69681a1fb0 con 0x7f6968075d20 2024-05-27T07:21:16.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.827+0000 7f696e800700 1 -- 172.21.15.43:0/481945901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69681a98d0 con 0x7f6968075d20 2024-05-27T07:21:16.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.828+0000 7f696e800700 1 -- 172.21.15.43:0/481945901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6968077290 con 0x7f6968075d20 2024-05-27T07:21:16.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.830+0000 7f6966000700 1 -- 172.21.15.43:0/481945901 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f695c019070 con 0x7f6968075d20 2024-05-27T07:21:16.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.832+0000 7f6966000700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f69500b2a70 0x7f69500b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:16.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.832+0000 7f6966000700 1 -- 172.21.15.43:0/481945901 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f695c0e2580 con 0x7f6968075d20 2024-05-27T07:21:16.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.832+0000 7f696ca00700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f69500b2a70 0x7f69500b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:16.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.833+0000 7f696ca00700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f69500b2a70 0x7f69500b4f80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f695400b5c0 tx=0x7f6954005fb0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:16.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:16.837+0000 7f6966000700 1 -- 172.21.15.43:0/481945901 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f695c0a9b90 con 0x7f6968075d20 2024-05-27T07:21:17.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.165+0000 7f696e800700 1 -- 172.21.15.43:0/481945901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f696819e150 con 0x7f6968075d20 2024-05-27T07:21:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.167+0000 7f6966000700 1 -- 172.21.15.43:0/481945901 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+8623 (secure 0 0 0) 0x7f695c0a9490 con 0x7f6968075d20 2024-05-27T07:21:17.157 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:21:17.158 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":79,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:19:26.072115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/4280267074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":4280267074},{"type":"v1","addr":"172.21.15.179:6833","nonce":4280267074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6835/2538262781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2538262781},{"type":"v1","addr":"172.21.15.179:6835","nonce":2538262781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34403,24484,34424,34484,34448]},"id":1}]} 2024-05-27T07:21:17.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.170+0000 7f696e800700 1 -- 172.21.15.43:0/481945901 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f69500b2a70 msgr2=0x7f69500b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.170+0000 7f696e800700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f69500b2a70 0x7f69500b4f80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f695400b5c0 tx=0x7f6954005fb0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.170+0000 7f696e800700 1 -- 172.21.15.43:0/481945901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6968075d20 msgr2=0x7f696819cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.170+0000 7f696e800700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6968075d20 0x7f696819cdf0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f695c00d950 tx=0x7f695c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:21:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.171+0000 7f696e800700 1 -- 172.21.15.43:0/481945901 shutdown_connections 2024-05-27T07:21:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.171+0000 7f696e800700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f69500b2a70 0x7f69500b4f80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.171+0000 7f696e800700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69681109a0 0x7f69681a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.171+0000 7f696e800700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6968075d20 0x7f696819cdf0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.171+0000 7f696e800700 1 --2- 172.21.15.43:0/481945901 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6968076670 0x7f696819d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.171+0000 7f696e800700 1 -- 172.21.15.43:0/481945901 >> 172.21.15.43:0/481945901 conn(0x7f6968079d00 msgr2=0x7f6968100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.171+0000 7f696e800700 1 -- 172.21.15.43:0/481945901 shutdown_connections 2024-05-27T07:21:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:17.171+0000 7f696e800700 1 -- 172.21.15.43:0/481945901 wait complete. 2024-05-27T07:21:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 79 2024-05-27T07:21:17.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 38285 0.63 MB/sec execute 201 sec latency 54.671 ms 2024-05-27T07:21:18.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:21:18.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:18 smithi043 ceph-mon[31531]: pgmap v246: 129 pgs: 129 active+clean; 65 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 335 KiB/s wr, 41 op/s 2024-05-27T07:21:18.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:18 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/481945901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:18.206 INFO:tasks.workunit.client.0.smithi043.stdout: 1 38482 0.63 MB/sec execute 202 sec latency 56.677 ms 2024-05-27T07:21:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:18 smithi179 ceph-mon[35888]: pgmap v246: 129 pgs: 129 active+clean; 65 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 335 KiB/s wr, 41 op/s 2024-05-27T07:21:18.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:18 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/481945901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:18.512 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:18.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:18 smithi132 ceph-mon[35283]: pgmap v246: 129 pgs: 129 active+clean; 65 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 335 KiB/s wr, 41 op/s 2024-05-27T07:21:18.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:18 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/481945901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:21:19.207 INFO:tasks.workunit.client.0.smithi043.stdout: 1 38716 0.63 MB/sec execute 203 sec latency 52.350 ms 2024-05-27T07:21:20.207 INFO:tasks.workunit.client.0.smithi043.stdout: 1 38807 0.63 MB/sec execute 204 sec latency 56.175 ms 2024-05-27T07:21:20.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:20 smithi043 ceph-mon[31531]: pgmap v247: 129 pgs: 129 active+clean; 65 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 335 KiB/s wr, 41 op/s 2024-05-27T07:21:20.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:20 smithi179 ceph-mon[35888]: pgmap v247: 129 pgs: 129 active+clean; 65 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 335 KiB/s wr, 41 op/s 2024-05-27T07:21:20.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:20 smithi132 ceph-mon[35283]: pgmap v247: 129 pgs: 129 active+clean; 65 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 335 KiB/s wr, 41 op/s 2024-05-27T07:21:20.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.823+0000 7f028de00700 1 -- 172.21.15.43:0/3392454438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0288076670 msgr2=0x7f0288110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:20.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.823+0000 7f028de00700 1 --2- 172.21.15.43:0/3392454438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0288076670 0x7f0288110460 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f0274009990 tx=0x7f0274009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.823+0000 7f028de00700 1 -- 172.21.15.43:0/3392454438 shutdown_connections 2024-05-27T07:21:20.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.823+0000 7f028de00700 1 --2- 172.21.15.43:0/3392454438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02881109a0 0x7f0288112e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.823+0000 7f028de00700 1 --2- 172.21.15.43:0/3392454438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0288075d20 0x7f0288076130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.823+0000 7f028de00700 1 --2- 172.21.15.43:0/3392454438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0288076670 0x7f0288110460 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.823+0000 7f028de00700 1 -- 172.21.15.43:0/3392454438 >> 172.21.15.43:0/3392454438 conn(0x7f0288079d00 msgr2=0x7f028807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:20.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.823+0000 7f028de00700 1 -- 172.21.15.43:0/3392454438 shutdown_connections 2024-05-27T07:21:20.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.824+0000 7f028de00700 1 -- 172.21.15.43:0/3392454438 wait complete. 2024-05-27T07:21:20.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.825+0000 7f028de00700 1 Processor -- start 2024-05-27T07:21:20.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.825+0000 7f028de00700 1 -- start start 2024-05-27T07:21:20.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.826+0000 7f028de00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0288075d20 0x7f028819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.826+0000 7f028de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0288076670 0x7f028819d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.826+0000 7f028de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02881109a0 0x7f02881a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.826+0000 7f028de00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f02881163a0 con 0x7f02881109a0 2024-05-27T07:21:20.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.826+0000 7f028de00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0288116220 con 0x7f0288076670 2024-05-27T07:21:20.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.826+0000 7f028de00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0288116520 con 0x7f0288075d20 2024-05-27T07:21:20.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.826+0000 7f028d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02881109a0 0x7f02881a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.826+0000 7f028d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02881109a0 0x7f02881a15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37328/0 (socket says 172.21.15.43:37328) 2024-05-27T07:21:20.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.826+0000 7f028d400700 1 -- 172.21.15.43:0/789679389 learned_addr learned my addr 172.21.15.43:0/789679389 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:20.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.826+0000 7f028ca00700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0288075d20 0x7f028819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.827+0000 7f027fe00700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0288076670 0x7f028819d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.827+0000 7f028d400700 1 -- 172.21.15.43:0/789679389 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0288075d20 msgr2=0x7f028819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:20.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.827+0000 7f028d400700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0288075d20 0x7f028819cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.827+0000 7f028d400700 1 -- 172.21.15.43:0/789679389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0288076670 msgr2=0x7f028819d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:20.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.827+0000 7f028d400700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0288076670 0x7f028819d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.827+0000 7f028d400700 1 -- 172.21.15.43:0/789679389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02881a1bd0 con 0x7f02881109a0 2024-05-27T07:21:20.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.827+0000 7f027fe00700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0288076670 0x7f028819d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:20.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.828+0000 7f028d400700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02881109a0 0x7f02881a15e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f02800094f0 tx=0x7f028000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:20.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.828+0000 7f0286000700 1 -- 172.21.15.43:0/789679389 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028000ddd0 con 0x7f02881109a0 2024-05-27T07:21:20.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.828+0000 7f0286000700 1 -- 172.21.15.43:0/789679389 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0280009e20 con 0x7f02881109a0 2024-05-27T07:21:20.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.828+0000 7f0286000700 1 -- 172.21.15.43:0/789679389 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0280015400 con 0x7f02881109a0 2024-05-27T07:21:20.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.828+0000 7f028de00700 1 -- 172.21.15.43:0/789679389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02881a1ec0 con 0x7f02881109a0 2024-05-27T07:21:20.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.828+0000 7f028de00700 1 -- 172.21.15.43:0/789679389 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02881a23b0 con 0x7f02881109a0 2024-05-27T07:21:20.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.832+0000 7f0286000700 1 -- 172.21.15.43:0/789679389 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0280005020 con 0x7f02881109a0 2024-05-27T07:21:20.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.834+0000 7f029e01fb80 1 Processor -- start 2024-05-27T07:21:20.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.834+0000 7f029e01fb80 1 -- start start 2024-05-27T07:21:20.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.835+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec0244750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.835+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0244c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.835+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec02506e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.835+0000 7f029e01fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55aec0171e40 con 0x55aec024e2c0 2024-05-27T07:21:20.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.835+0000 7f029e01fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55aec01619e0 con 0x55aec01b09f0 2024-05-27T07:21:20.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.835+0000 7f029e01fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55aec0165a20 con 0x55aec00c4e40 2024-05-27T07:21:20.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.835+0000 7f0286000700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02640b2a70 0x7f02640b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.836+0000 7f028ca00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0244c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.836+0000 7f027fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec0244750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.836+0000 7f0286000700 1 -- 172.21.15.43:0/789679389 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f02800e23f0 con 0x7f02881109a0 2024-05-27T07:21:20.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.836+0000 7f027fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec0244750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58930/0 (socket says 172.21.15.43:58930) 2024-05-27T07:21:20.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.836+0000 7f028ca00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0244c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:58980/0 (socket says 172.21.15.43:58980) 2024-05-27T07:21:20.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.836+0000 7f027fe00700 1 -- 172.21.15.43:0/1059856917 learned_addr learned my addr 172.21.15.43:0/1059856917 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:20.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.836+0000 7f027fe00700 1 --2- 172.21.15.43:0/1059856917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec02506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.836+0000 7f028d400700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02640b2a70 0x7f02640b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.837+0000 7f028ca00700 1 -- 172.21.15.43:0/1059856917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 msgr2=0x55aec0244750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.837+0000 7f028ca00700 1 --2- 172.21.15.43:0/1059856917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec0244750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.837+0000 7f028ca00700 1 -- 172.21.15.43:0/1059856917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 msgr2=0x55aec02506e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.837+0000 7f028ca00700 1 --2- 172.21.15.43:0/1059856917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec02506e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.837+0000 7f028ca00700 1 -- 172.21.15.43:0/1059856917 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aec012a420 con 0x55aec00c4e40 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.837+0000 7f027fe00700 1 --2- 172.21.15.43:0/1059856917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec02506e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.837+0000 7f028d400700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02640b2a70 0x7f02640b4f80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0280009710 tx=0x7f028000df90 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.837+0000 7f028ca00700 1 --2- 172.21.15.43:0/1059856917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0244c90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f026c00ac60 tx=0x7f026c00af70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e5c45f3de7a721a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.838+0000 7f027e000700 1 -- 172.21.15.43:0/1059856917 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c00fde0 con 0x55aec00c4e40 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.838+0000 7f027e000700 1 -- 172.21.15.43:0/1059856917 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f026c0044c0 con 0x55aec00c4e40 2024-05-27T07:21:20.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.838+0000 7f027e000700 1 -- 172.21.15.43:0/1059856917 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c005280 con 0x55aec00c4e40 2024-05-27T07:21:20.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.839+0000 7f029e01fb80 1 -- 172.21.15.43:0/1059856917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 msgr2=0x55aec0244c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:20.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.839+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1059856917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0244c90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f026c00ac60 tx=0x7f026c00af70 comp rx=0 tx=0).stop 2024-05-27T07:21:20.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.839+0000 7f029e01fb80 1 -- 172.21.15.43:0/1059856917 shutdown_connections 2024-05-27T07:21:20.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.839+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1059856917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec02506e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.839+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1059856917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec0244750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.839+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1059856917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0244c90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.839+0000 7f029e01fb80 1 -- 172.21.15.43:0/1059856917 >> 172.21.15.43:0/1059856917 conn(0x55aec015e390 msgr2=0x55aec023f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:20.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.839+0000 7f029e01fb80 1 -- 172.21.15.43:0/1059856917 shutdown_connections 2024-05-27T07:21:20.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.839+0000 7f029e01fb80 1 -- 172.21.15.43:0/1059856917 wait complete. 2024-05-27T07:21:20.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.840+0000 7f029e01fb80 1 -- start start 2024-05-27T07:21:20.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.840+0000 7f029e01fb80 1 Processor -- start 2024-05-27T07:21:20.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.841+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec024b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.841+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec01b09f0 0x55aec024b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.841+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec024e2c0 0x55aec024bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.841+0000 7f029e01fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55aec0067fa0 con 0x55aec01b09f0 2024-05-27T07:21:20.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.841+0000 7f029e01fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55aec0102760 con 0x55aec024e2c0 2024-05-27T07:21:20.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.841+0000 7f029e01fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55aec0250e00 con 0x55aec00c4e40 2024-05-27T07:21:20.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.841+0000 7f029e01fb80 10 client.0 _ll_register_callbacks cb 0x55aec010bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:20.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.841+0000 7f028ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec01b09f0 0x55aec024b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.841+0000 7f028ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec01b09f0 0x55aec024b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37350/0 (socket says 172.21.15.43:37350) 2024-05-27T07:21:20.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.841+0000 7f028ca00700 1 -- 172.21.15.43:0/1967824652 learned_addr learned my addr 172.21.15.43:0/1967824652 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.842+0000 7f027fe00700 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec024b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.842+0000 7f027fe00700 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec024e2c0 0x55aec024bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.842+0000 7f028ca00700 1 -- 172.21.15.43:0/1967824652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 msgr2=0x55aec024b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.842+0000 7f028ca00700 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec024b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.842+0000 7f028ca00700 1 -- 172.21.15.43:0/1967824652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec024e2c0 msgr2=0x55aec024bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.842+0000 7f028ca00700 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec024e2c0 0x55aec024bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.842+0000 7f028ca00700 1 -- 172.21.15.43:0/1967824652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aec00b18d0 con 0x55aec01b09f0 2024-05-27T07:21:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.842+0000 7f027fe00700 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec024e2c0 0x55aec024bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.843+0000 7f028ca00700 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec01b09f0 0x55aec024b6c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f026c004d40 tx=0x7f026c004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f71d7bb78770a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:20.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.843+0000 7f027ae00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:20.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.843+0000 7f027ae00700 1 -- 172.21.15.43:0/1967824652 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c01d040 con 0x55aec01b09f0 2024-05-27T07:21:20.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.843+0000 7f027ae00700 1 -- 172.21.15.43:0/1967824652 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f026c005150 con 0x55aec01b09f0 2024-05-27T07:21:20.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.843+0000 7f029e01fb80 1 -- 172.21.15.43:0/1967824652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f026c0044c0 con 0x55aec01b09f0 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.843+0000 7f027ae00700 1 -- 172.21.15.43:0/1967824652 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f026c030050 con 0x55aec01b09f0 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.843+0000 7f029e01fb80 10 client.16536 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.843+0000 7f029e01fb80 1 -- 172.21.15.43:0/1967824652 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55aec0132060 con 0x55aec01b09f0 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.844+0000 7f027ae00700 1 -- 172.21.15.43:0/1967824652 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f026c0044c0 con 0x55aec01b09f0 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.844+0000 7f027ae00700 1 -- 172.21.15.43:0/1967824652 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f026c018b30 con 0x55aec01b09f0 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.844+0000 7f029e01fb80 10 client.16536 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.844+0000 7f029e01fb80 10 client.16536 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.844+0000 7f029e01fb80 20 client.16536 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.844+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55aec025dec0 0x55aec0260350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.844+0000 7f029e01fb80 4 client.16536 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:20.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.844+0000 7f029e01fb80 1 -- 172.21.15.43:0/1967824652 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aec0173d90 con 0x55aec025dec0 2024-05-27T07:21:20.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.845+0000 7f027fe00700 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55aec025dec0 0x55aec0260350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.846+0000 7f027fe00700 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55aec025dec0 0x55aec0260350 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=204a11903c66680b server_cookie=605c504997c67cd5 in_seq=0 out_seq=0 2024-05-27T07:21:20.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.846+0000 7f027ae00700 10 client.16536 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:21:20.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f027ae00700 1 -- 172.21.15.43:0/1967824652 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55aec0173d90 con 0x55aec025dec0 2024-05-27T07:21:20.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f027ae00700 10 client.16536 handle_command_reply: tid=0 2024-05-27T07:21:20.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f029e01fb80 1 client.16536 shutdown 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f029e01fb80 1 -- 172.21.15.43:0/1967824652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec01b09f0 msgr2=0x55aec024b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec01b09f0 0x55aec024b6c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f026c004d40 tx=0x7f026c004bb0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f029e01fb80 1 -- 172.21.15.43:0/1967824652 shutdown_connections 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55aec025dec0 0x55aec0260350 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec024e2c0 0x55aec024bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec024b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1967824652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec01b09f0 0x55aec024b6c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f027ae00700 0 client.16536 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.848+0000 7f029e01fb80 1 -- 172.21.15.43:0/1967824652 >> 172.21.15.43:0/1967824652 conn(0x55aec015e390 msgr2=0x55aec023f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.849+0000 7f029e01fb80 1 -- 172.21.15.43:0/1967824652 shutdown_connections 2024-05-27T07:21:20.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.849+0000 7f029e01fb80 1 -- 172.21.15.43:0/1967824652 wait complete. 2024-05-27T07:21:20.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.849+0000 7f029e01fb80 20 client.16536 trim_cache size 0 max 16384 2024-05-27T07:21:20.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.867+0000 7f029e01fb80 1 Processor -- start 2024-05-27T07:21:20.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.868+0000 7f029e01fb80 1 -- start start 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.868+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0259f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.868+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec0244750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.868+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0244c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.868+0000 7f029e01fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f02881b31c0 con 0x55aec024e2c0 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.868+0000 7f029e01fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f02881b3890 con 0x55aec01b09f0 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.868+0000 7f029e01fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f02881b0fb0 con 0x55aec00c4e40 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.869+0000 7f027fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0244c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.869+0000 7f027fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0244c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37358/0 (socket says 172.21.15.43:37358) 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.869+0000 7f027fe00700 1 -- 172.21.15.43:0/1902950959 learned_addr learned my addr 172.21.15.43:0/1902950959 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.869+0000 7f027fe00700 1 --2- 172.21.15.43:0/1902950959 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0259f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.869+0000 7f028ca00700 1 --2- 172.21.15.43:0/1902950959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec0244750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.869+0000 7f027fe00700 1 -- 172.21.15.43:0/1902950959 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 msgr2=0x55aec0259f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:20.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.869+0000 7f027fe00700 1 --2- 172.21.15.43:0/1902950959 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0259f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.869+0000 7f027fe00700 1 -- 172.21.15.43:0/1902950959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 msgr2=0x55aec0244750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.869+0000 7f027fe00700 1 --2- 172.21.15.43:0/1902950959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec0244750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.869+0000 7f027fe00700 1 -- 172.21.15.43:0/1902950959 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aec023fc50 con 0x55aec024e2c0 2024-05-27T07:21:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.870+0000 7f027fe00700 1 --2- 172.21.15.43:0/1902950959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0244c90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0274000c00 tx=0x7f0274005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52e5443b48640c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.870+0000 7f027d600700 1 -- 172.21.15.43:0/1902950959 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027400bd00 con 0x55aec024e2c0 2024-05-27T07:21:20.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.870+0000 7f027d600700 1 -- 172.21.15.43:0/1902950959 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f027402e040 con 0x55aec024e2c0 2024-05-27T07:21:20.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.870+0000 7f027d600700 1 -- 172.21.15.43:0/1902950959 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027400f6c0 con 0x55aec024e2c0 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.871+0000 7f029e01fb80 1 -- 172.21.15.43:0/1902950959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 msgr2=0x55aec0244c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.871+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1902950959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0244c90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0274000c00 tx=0x7f0274005fb0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.871+0000 7f029e01fb80 1 -- 172.21.15.43:0/1902950959 shutdown_connections 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.871+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1902950959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0244c90 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.871+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1902950959 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0259f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.871+0000 7f029e01fb80 1 --2- 172.21.15.43:0/1902950959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec0244750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.871+0000 7f029e01fb80 1 -- 172.21.15.43:0/1902950959 >> 172.21.15.43:0/1902950959 conn(0x55aec015e390 msgr2=0x55aec0249a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.871+0000 7f029e01fb80 1 -- 172.21.15.43:0/1902950959 shutdown_connections 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.871+0000 7f029e01fb80 1 -- 172.21.15.43:0/1902950959 wait complete. 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.872+0000 7f029e01fb80 1 -- start start 2024-05-27T07:21:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.872+0000 7f029e01fb80 1 Processor -- start 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0256990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec02510e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f029e01fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0251730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f029e01fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f02881b1680 con 0x55aec024e2c0 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f029e01fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f02881b2af0 con 0x55aec01b09f0 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f029e01fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f02881b1d50 con 0x55aec00c4e40 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f029e01fb80 10 client.0 _ll_register_callbacks cb 0x55aec010bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f027fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0251730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f027fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0251730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37366/0 (socket says 172.21.15.43:37366) 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f027fe00700 1 -- 172.21.15.43:0/2640817551 learned_addr learned my addr 172.21.15.43:0/2640817551 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.873+0000 7f028ca00700 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec02510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.874+0000 7f027fe00700 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0256990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.874+0000 7f027fe00700 1 -- 172.21.15.43:0/2640817551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 msgr2=0x55aec0256990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.874+0000 7f027fe00700 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0256990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.874+0000 7f027fe00700 1 -- 172.21.15.43:0/2640817551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 msgr2=0x55aec02510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.874+0000 7f027fe00700 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec02510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.874+0000 7f027fe00700 1 -- 172.21.15.43:0/2640817551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aec0251f00 con 0x55aec024e2c0 2024-05-27T07:21:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.874+0000 7f028ca00700 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec02510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:20.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.874+0000 7f027fe00700 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0251730 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f0274005850 tx=0x7f0274005210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6791ee559ff87a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:20.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.874+0000 7f027ae00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:20.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.874+0000 7f027ae00700 1 -- 172.21.15.43:0/2640817551 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027400f460 con 0x55aec024e2c0 2024-05-27T07:21:20.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.875+0000 7f029e01fb80 1 -- 172.21.15.43:0/2640817551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f027402e040 con 0x55aec024e2c0 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.875+0000 7f027ae00700 1 -- 172.21.15.43:0/2640817551 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f027400f600 con 0x55aec024e2c0 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.875+0000 7f027ae00700 1 -- 172.21.15.43:0/2640817551 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f027400f8a0 con 0x55aec024e2c0 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.875+0000 7f029e01fb80 10 client.16542 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.875+0000 7f029e01fb80 1 -- 172.21.15.43:0/2640817551 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55aec02c7a40 con 0x55aec024e2c0 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.875+0000 7f027ae00700 1 -- 172.21.15.43:0/2640817551 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f027402e040 con 0x55aec024e2c0 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.875+0000 7f027ae00700 1 -- 172.21.15.43:0/2640817551 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f0274033020 con 0x55aec024e2c0 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.876+0000 7f029e01fb80 10 client.16542 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.876+0000 7f029e01fb80 10 client.16542 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.876+0000 7f029e01fb80 20 client.16542 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.876+0000 7f029e01fb80 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55aec02c8010 0x55aec02ca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.876+0000 7f029e01fb80 4 client.16542 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.876+0000 7f029e01fb80 1 -- 172.21.15.43:0/2640817551 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aec016d570 con 0x55aec02c8010 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.876+0000 7f028ca00700 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55aec02c8010 0x55aec02ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.877+0000 7f028ca00700 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55aec02c8010 0x55aec02ca430 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=27b99e05f7856bc8 server_cookie=6356a2bc8c532e21 in_seq=0 out_seq=0 2024-05-27T07:21:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.877+0000 7f027ae00700 10 client.16542 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:21:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f027ae00700 1 -- 172.21.15.43:0/2640817551 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aec016d570 con 0x55aec02c8010 2024-05-27T07:21:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f027ae00700 10 client.16542 handle_command_reply: tid=0 2024-05-27T07:21:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f029e01fb80 1 client.16542 shutdown 2024-05-27T07:21:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f029e01fb80 1 -- 172.21.15.43:0/2640817551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 msgr2=0x55aec0251730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f029e01fb80 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0251730 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f0274005850 tx=0x7f0274005210 comp rx=0 tx=0).stop 2024-05-27T07:21:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f029e01fb80 1 -- 172.21.15.43:0/2640817551 shutdown_connections 2024-05-27T07:21:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f029e01fb80 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55aec02c8010 0x55aec02ca430 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f029e01fb80 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aec024e2c0 0x55aec0251730 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f029e01fb80 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aec00c4e40 0x55aec0256990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f029e01fb80 1 --2- 172.21.15.43:0/2640817551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aec01b09f0 0x55aec02510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f029e01fb80 1 -- 172.21.15.43:0/2640817551 >> 172.21.15.43:0/2640817551 conn(0x55aec015e390 msgr2=0x55aec023ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:20.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.878+0000 7f027ae00700 0 client.16542 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:21:20.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.879+0000 7f029e01fb80 1 -- 172.21.15.43:0/2640817551 shutdown_connections 2024-05-27T07:21:20.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.879+0000 7f029e01fb80 1 -- 172.21.15.43:0/2640817551 wait complete. 2024-05-27T07:21:20.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.879+0000 7f029e01fb80 20 client.16542 trim_cache size 0 max 16384 2024-05-27T07:21:20.871 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 -- 172.21.15.43:0/789679389 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02640b2a70 msgr2=0x7f02640b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02640b2a70 0x7f02640b4f80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0280009710 tx=0x7f028000df90 comp rx=0 tx=0).stop 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 -- 172.21.15.43:0/789679389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02881109a0 msgr2=0x7f02881a15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02881109a0 0x7f02881a15e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f02800094f0 tx=0x7f028000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 -- 172.21.15.43:0/789679389 shutdown_connections 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02640b2a70 0x7f02640b4f80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02881109a0 0x7f02881a15e0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0288075d20 0x7f028819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 --2- 172.21.15.43:0/789679389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0288076670 0x7f028819d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 -- 172.21.15.43:0/789679389 >> 172.21.15.43:0/789679389 conn(0x7f0288079d00 msgr2=0x7f0288100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 -- 172.21.15.43:0/789679389 shutdown_connections 2024-05-27T07:21:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:20.880+0000 7f028de00700 1 -- 172.21.15.43:0/789679389 wait complete. 2024-05-27T07:21:21.207 INFO:tasks.workunit.client.0.smithi043.stdout: 1 38834 0.63 MB/sec execute 205 sec latency 55.025 ms 2024-05-27T07:21:21.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:21:22.177 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:22.207 INFO:tasks.workunit.client.0.smithi043.stdout: 1 38901 0.63 MB/sec execute 206 sec latency 62.259 ms 2024-05-27T07:21:22.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:22 smithi043 ceph-mon[31531]: pgmap v248: 129 pgs: 129 active+clean; 67 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 502 KiB/s wr, 42 op/s 2024-05-27T07:21:22.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:22 smithi179 ceph-mon[35888]: pgmap v248: 129 pgs: 129 active+clean; 67 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 502 KiB/s wr, 42 op/s 2024-05-27T07:21:22.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:22 smithi132 ceph-mon[35283]: pgmap v248: 129 pgs: 129 active+clean; 67 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 502 KiB/s wr, 42 op/s 2024-05-27T07:21:23.207 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39121 0.64 MB/sec execute 207 sec latency 57.941 ms 2024-05-27T07:21:24.207 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39215 0.64 MB/sec execute 208 sec latency 54.960 ms 2024-05-27T07:21:24.414 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:24 smithi043 ceph-mon[31531]: pgmap v249: 129 pgs: 129 active+clean; 67 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 435 KiB/s wr, 36 op/s 2024-05-27T07:21:24.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:24 smithi179 ceph-mon[35888]: pgmap v249: 129 pgs: 129 active+clean; 67 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 435 KiB/s wr, 36 op/s 2024-05-27T07:21:24.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:24 smithi132 ceph-mon[35283]: pgmap v249: 129 pgs: 129 active+clean; 67 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 435 KiB/s wr, 36 op/s 2024-05-27T07:21:24.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.682+0000 7f3a57400700 1 -- 172.21.15.43:0/3740262633 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44003b40 con 0x7f3a58105e20 2024-05-27T07:21:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.683+0000 7f3a5f800700 1 -- 172.21.15.43:0/3740262633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a58105e20 msgr2=0x7f3a5810e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.683+0000 7f3a5f800700 1 --2- 172.21.15.43:0/3740262633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a58105e20 0x7f3a5810e370 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009990 tx=0x7f3a44009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.684+0000 7f3a5f800700 1 -- 172.21.15.43:0/3740262633 shutdown_connections 2024-05-27T07:21:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.684+0000 7f3a5f800700 1 --2- 172.21.15.43:0/3740262633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a5810e900 0x7f3a58110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.684+0000 7f3a5f800700 1 --2- 172.21.15.43:0/3740262633 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581054d0 0x7f3a581058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.684+0000 7f3a5f800700 1 --2- 172.21.15.43:0/3740262633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a58105e20 0x7f3a5810e370 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.684+0000 7f3a5f800700 1 -- 172.21.15.43:0/3740262633 >> 172.21.15.43:0/3740262633 conn(0x7f3a580fef40 msgr2=0x7f3a58101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.684+0000 7f3a5f800700 1 -- 172.21.15.43:0/3740262633 shutdown_connections 2024-05-27T07:21:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.684+0000 7f3a5f800700 1 -- 172.21.15.43:0/3740262633 wait complete. 2024-05-27T07:21:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.685+0000 7f3a5f800700 1 Processor -- start 2024-05-27T07:21:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.685+0000 7f3a5f800700 1 -- start start 2024-05-27T07:21:24.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a5f800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581054d0 0x7f3a5819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a5f800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a58105e20 0x7f3a5819d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a5f800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a5810e900 0x7f3a581a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a5f800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3a58114300 con 0x7f3a5810e900 2024-05-27T07:21:24.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a5f800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3a58114180 con 0x7f3a58105e20 2024-05-27T07:21:24.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a5f800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3a58114480 con 0x7f3a581054d0 2024-05-27T07:21:24.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a5d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a5810e900 0x7f3a581a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a5d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a5810e900 0x7f3a581a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46538/0 (socket says 172.21.15.43:46538) 2024-05-27T07:21:24.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a5d800700 1 -- 172.21.15.43:0/2986715271 learned_addr learned my addr 172.21.15.43:0/2986715271 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581054d0 0x7f3a5819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.686+0000 7f3a57e00700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a58105e20 0x7f3a5819d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.687+0000 7f3a5d800700 1 -- 172.21.15.43:0/2986715271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581054d0 msgr2=0x7f3a5819cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.687+0000 7f3a5d800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581054d0 0x7f3a5819cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.687+0000 7f3a5d800700 1 -- 172.21.15.43:0/2986715271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a58105e20 msgr2=0x7f3a5819d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.687+0000 7f3a5d800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a58105e20 0x7f3a5819d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.687+0000 7f3a5d800700 1 -- 172.21.15.43:0/2986715271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a581a1d90 con 0x7f3a5810e900 2024-05-27T07:21:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.687+0000 7f3a5d800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a5810e900 0x7f3a581a17a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3a50007d70 tx=0x7f3a5000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.687+0000 7f3a55600700 1 -- 172.21.15.43:0/2986715271 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a50016020 con 0x7f3a5810e900 2024-05-27T07:21:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.688+0000 7f3a55600700 1 -- 172.21.15.43:0/2986715271 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a5000ad40 con 0x7f3a5810e900 2024-05-27T07:21:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.688+0000 7f3a55600700 1 -- 172.21.15.43:0/2986715271 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a500115e0 con 0x7f3a5810e900 2024-05-27T07:21:24.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.688+0000 7f3a5f800700 1 -- 172.21.15.43:0/2986715271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a581a2080 con 0x7f3a5810e900 2024-05-27T07:21:24.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.688+0000 7f3a5f800700 1 -- 172.21.15.43:0/2986715271 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a581a2570 con 0x7f3a5810e900 2024-05-27T07:21:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.692+0000 7f3a55600700 1 -- 172.21.15.43:0/2986715271 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3a50019020 con 0x7f3a5810e900 2024-05-27T07:21:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.693+0000 7f3a6e311b80 1 Processor -- start 2024-05-27T07:21:24.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.693+0000 7f3a6e311b80 1 -- start start 2024-05-27T07:21:24.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.693+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d118a3e40 0x556d11a287f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.693+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d11a28d30 0x556d11a22870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a22db0 0x556d11a231c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556d11950e40 con 0x556d11a28d30 2024-05-27T07:21:24.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556d119409e0 con 0x556d118a3e40 2024-05-27T07:21:24.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556d11944a20 con 0x556d11a22db0 2024-05-27T07:21:24.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a55600700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3a400b2a70 0x7f3a400b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a5ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d11a28d30 0x556d11a22870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a55600700 1 -- 172.21.15.43:0/2986715271 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f3a500e28a0 con 0x7f3a5810e900 2024-05-27T07:21:24.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a5d800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3a400b2a70 0x7f3a400b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a5ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d11a28d30 0x556d11a22870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46542/0 (socket says 172.21.15.43:46542) 2024-05-27T07:21:24.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a5ce00700 1 -- 172.21.15.43:0/2355038817 learned_addr learned my addr 172.21.15.43:0/2355038817 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:24.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a57e00700 1 --2- 172.21.15.43:0/2355038817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d118a3e40 0x556d11a287f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.694+0000 7f3a57e00700 1 --2- 172.21.15.43:0/2355038817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a22db0 0x556d11a231c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.695+0000 7f3a5ce00700 1 -- 172.21.15.43:0/2355038817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a22db0 msgr2=0x556d11a231c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:24.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.695+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/2355038817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a22db0 0x556d11a231c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.695+0000 7f3a5ce00700 1 -- 172.21.15.43:0/2355038817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d118a3e40 msgr2=0x556d11a287f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:24.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.695+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/2355038817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d118a3e40 0x556d11a287f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.695+0000 7f3a5ce00700 1 -- 172.21.15.43:0/2355038817 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d11909420 con 0x556d11a28d30 2024-05-27T07:21:24.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.695+0000 7f3a5d800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3a400b2a70 0x7f3a400b4f80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3a50012c40 tx=0x7f3a500b0f90 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:24.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.695+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/2355038817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d11a28d30 0x556d11a22870 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c00bd70 tx=0x7f3a4c00f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832f2b378ec236a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:24.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.696+0000 7f3a4a000700 1 -- 172.21.15.43:0/2355038817 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c015070 con 0x556d11a28d30 2024-05-27T07:21:24.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.696+0000 7f3a4a000700 1 -- 172.21.15.43:0/2355038817 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a4c00fda0 con 0x556d11a28d30 2024-05-27T07:21:24.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.696+0000 7f3a4a000700 1 -- 172.21.15.43:0/2355038817 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c0116a0 con 0x556d11a28d30 2024-05-27T07:21:24.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.697+0000 7f3a6e311b80 1 -- 172.21.15.43:0/2355038817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d11a28d30 msgr2=0x556d11a22870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:24.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.697+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/2355038817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d11a28d30 0x556d11a22870 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c00bd70 tx=0x7f3a4c00f640 comp rx=0 tx=0).stop 2024-05-27T07:21:24.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.697+0000 7f3a6e311b80 1 -- 172.21.15.43:0/2355038817 shutdown_connections 2024-05-27T07:21:24.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.697+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/2355038817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a22db0 0x556d11a231c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.697+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/2355038817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d118a3e40 0x556d11a287f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.697+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/2355038817 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d11a28d30 0x556d11a22870 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.697+0000 7f3a6e311b80 1 -- 172.21.15.43:0/2355038817 >> 172.21.15.43:0/2355038817 conn(0x556d11903fc0 msgr2=0x556d11a1d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:24.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.697+0000 7f3a6e311b80 1 -- 172.21.15.43:0/2355038817 shutdown_connections 2024-05-27T07:21:24.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.697+0000 7f3a6e311b80 1 -- 172.21.15.43:0/2355038817 wait complete. 2024-05-27T07:21:24.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.698+0000 7f3a6e311b80 1 -- start start 2024-05-27T07:21:24.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.698+0000 7f3a6e311b80 1 Processor -- start 2024-05-27T07:21:24.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.699+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a2c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.699+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a22db0 0x556d11a2c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.699+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a28d30 0x556d11a2cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.699+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556d11846fa0 con 0x556d118a3e40 2024-05-27T07:21:24.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.699+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556d118e1760 con 0x556d11a28d30 2024-05-27T07:21:24.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.699+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556d11a2f800 con 0x556d11a22db0 2024-05-27T07:21:24.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.699+0000 7f3a6e311b80 10 client.0 _ll_register_callbacks cb 0x556d118eac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:24.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a57e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a2c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a57e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a2c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46554/0 (socket says 172.21.15.43:46554) 2024-05-27T07:21:24.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a57e00700 1 -- 172.21.15.43:0/3269802390 learned_addr learned my addr 172.21.15.43:0/3269802390 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:24.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a22db0 0x556d11a2c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a57e00700 1 -- 172.21.15.43:0/3269802390 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a22db0 msgr2=0x556d11a2c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:24.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a57e00700 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a22db0 0x556d11a2c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a57e00700 1 -- 172.21.15.43:0/3269802390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a28d30 msgr2=0x556d11a2cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:24.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a57e00700 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a28d30 0x556d11a2cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a57e00700 1 -- 172.21.15.43:0/3269802390 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d118908d0 con 0x556d118a3e40 2024-05-27T07:21:24.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a57e00700 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a2c460 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f3a58076cf0 tx=0x7f3a44005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618bcf2bbd6ed160 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:24.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.700+0000 7f3a3ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:24.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.701+0000 7f3a3ea00700 1 -- 172.21.15.43:0/3269802390 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4402b060 con 0x556d118a3e40 2024-05-27T07:21:24.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.701+0000 7f3a3ea00700 1 -- 172.21.15.43:0/3269802390 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f3a44003b10 con 0x556d118a3e40 2024-05-27T07:21:24.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.701+0000 7f3a6e311b80 1 -- 172.21.15.43:0/3269802390 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a4c00fda0 con 0x556d118a3e40 2024-05-27T07:21:24.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.701+0000 7f3a3ea00700 1 -- 172.21.15.43:0/3269802390 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f3a44032050 con 0x556d118a3e40 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.701+0000 7f3a6e311b80 10 client.16554 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.701+0000 7f3a6e311b80 1 -- 172.21.15.43:0/3269802390 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x556d11911060 con 0x556d118a3e40 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.702+0000 7f3a3ea00700 1 -- 172.21.15.43:0/3269802390 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a4c00fda0 con 0x556d118a3e40 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.702+0000 7f3a3ea00700 1 -- 172.21.15.43:0/3269802390 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f3a44027bc0 con 0x556d118a3e40 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.702+0000 7f3a6e311b80 10 client.16554 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.702+0000 7f3a6e311b80 10 client.16554 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.702+0000 7f3a6e311b80 20 client.16554 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.702+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x556d11a3c880 0x556d11a3ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.702+0000 7f3a6e311b80 4 client.16554 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.702+0000 7f3a6e311b80 1 -- 172.21.15.43:0/3269802390 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d11952d90 con 0x556d11a3c880 2024-05-27T07:21:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.703+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x556d11a3c880 0x556d11a3ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.704+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x556d11a3c880 0x556d11a3ed10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4e53e80313fbf0cf server_cookie=e8c106900e2b5682 in_seq=0 out_seq=0 2024-05-27T07:21:24.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.704+0000 7f3a3ea00700 10 client.16554 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:21:24.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a3ea00700 1 -- 172.21.15.43:0/3269802390 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556d11952d90 con 0x556d11a3c880 2024-05-27T07:21:24.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a3ea00700 10 client.16554 handle_command_reply: tid=0 2024-05-27T07:21:24.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a6e311b80 1 client.16554 shutdown 2024-05-27T07:21:24.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a6e311b80 1 -- 172.21.15.43:0/3269802390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 msgr2=0x556d11a2c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:24.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a2c460 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f3a58076cf0 tx=0x7f3a44005730 comp rx=0 tx=0).stop 2024-05-27T07:21:24.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a6e311b80 1 -- 172.21.15.43:0/3269802390 shutdown_connections 2024-05-27T07:21:24.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x556d11a3c880 0x556d11a3ed10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a28d30 0x556d11a2cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a2c460 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/3269802390 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a22db0 0x556d11a2c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a6e311b80 1 -- 172.21.15.43:0/3269802390 >> 172.21.15.43:0/3269802390 conn(0x556d11903fc0 msgr2=0x556d11a1d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:24.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.706+0000 7f3a3ea00700 0 client.16554 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:21:24.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.707+0000 7f3a6e311b80 1 -- 172.21.15.43:0/3269802390 shutdown_connections 2024-05-27T07:21:24.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.707+0000 7f3a6e311b80 1 -- 172.21.15.43:0/3269802390 wait complete. 2024-05-27T07:21:24.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.707+0000 7f3a6e311b80 20 client.16554 trim_cache size 0 max 16384 2024-05-27T07:21:24.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.727+0000 7f3a6e311b80 1 Processor -- start 2024-05-27T07:21:24.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.727+0000 7f3a6e311b80 1 -- start start 2024-05-27T07:21:24.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.727+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a22870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.727+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 0x556d11a2b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.727+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a28d30 0x556d11a2e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.727+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3a581b3390 con 0x556d118a3e40 2024-05-27T07:21:24.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.727+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3a581b3a60 con 0x556d11a22db0 2024-05-27T07:21:24.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.727+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3a581b1180 con 0x556d11a28d30 2024-05-27T07:21:24.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a57e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a22870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a57e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a22870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46560/0 (socket says 172.21.15.43:46560) 2024-05-27T07:21:24.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a57e00700 1 -- 172.21.15.43:0/163122557 learned_addr learned my addr 172.21.15.43:0/163122557 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:24.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/163122557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 0x556d11a2b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a57e00700 1 --2- 172.21.15.43:0/163122557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a28d30 0x556d11a2e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a57e00700 1 -- 172.21.15.43:0/163122557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a28d30 msgr2=0x556d11a2e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:24.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a57e00700 1 --2- 172.21.15.43:0/163122557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a28d30 0x556d11a2e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a57e00700 1 -- 172.21.15.43:0/163122557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 msgr2=0x556d11a2b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:24.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a57e00700 1 --2- 172.21.15.43:0/163122557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 0x556d11a2b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a57e00700 1 -- 172.21.15.43:0/163122557 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d11a2a9d0 con 0x556d118a3e40 2024-05-27T07:21:24.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.728+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/163122557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 0x556d11a2b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:24.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.729+0000 7f3a57e00700 1 --2- 172.21.15.43:0/163122557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a22870 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009990 tx=0x7f3a4400f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c16d6870e62df8d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:24.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.729+0000 7f3a49600700 1 -- 172.21.15.43:0/163122557 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4402b040 con 0x556d118a3e40 2024-05-27T07:21:24.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.729+0000 7f3a49600700 1 -- 172.21.15.43:0/163122557 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a4400f790 con 0x556d118a3e40 2024-05-27T07:21:24.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.729+0000 7f3a49600700 1 -- 172.21.15.43:0/163122557 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4402d560 con 0x556d118a3e40 2024-05-27T07:21:24.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.730+0000 7f3a6e311b80 1 -- 172.21.15.43:0/163122557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 msgr2=0x556d11a22870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:24.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.730+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/163122557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a22870 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009990 tx=0x7f3a4400f460 comp rx=0 tx=0).stop 2024-05-27T07:21:24.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.730+0000 7f3a6e311b80 1 -- 172.21.15.43:0/163122557 shutdown_connections 2024-05-27T07:21:24.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.730+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/163122557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a28d30 0x556d11a2e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.730+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/163122557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a22870 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.730+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/163122557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 0x556d11a2b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.730+0000 7f3a6e311b80 1 -- 172.21.15.43:0/163122557 >> 172.21.15.43:0/163122557 conn(0x556d11903fc0 msgr2=0x556d11a26140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:24.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.730+0000 7f3a6e311b80 1 -- 172.21.15.43:0/163122557 shutdown_connections 2024-05-27T07:21:24.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.730+0000 7f3a6e311b80 1 -- 172.21.15.43:0/163122557 wait complete. 2024-05-27T07:21:24.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.731+0000 7f3a6e311b80 1 -- start start 2024-05-27T07:21:24.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.731+0000 7f3a6e311b80 1 Processor -- start 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.732+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a34a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.732+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 0x556d11a34f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.732+0000 7f3a6e311b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a28d30 0x556d11a434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.732+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3a581b1850 con 0x556d118a3e40 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.732+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3a581b2cc0 con 0x556d11a22db0 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.732+0000 7f3a6e311b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3a581b1f20 con 0x556d11a28d30 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.732+0000 7f3a6e311b80 10 client.0 _ll_register_callbacks cb 0x556d118eac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.732+0000 7f3a57e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a34a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.732+0000 7f3a57e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a34a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46562/0 (socket says 172.21.15.43:46562) 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.732+0000 7f3a57e00700 1 -- 172.21.15.43:0/1038646253 learned_addr learned my addr 172.21.15.43:0/1038646253 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.733+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 0x556d11a34f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.733+0000 7f3a57e00700 1 -- 172.21.15.43:0/1038646253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a28d30 msgr2=0x556d11a434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:24.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.733+0000 7f3a57e00700 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a28d30 0x556d11a434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.733+0000 7f3a57e00700 1 -- 172.21.15.43:0/1038646253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 msgr2=0x556d11a34f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:24.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.733+0000 7f3a57e00700 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 0x556d11a34f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.733+0000 7f3a57e00700 1 -- 172.21.15.43:0/1038646253 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d11a44ea0 con 0x556d118a3e40 2024-05-27T07:21:24.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.733+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 0x556d11a34f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.733+0000 7f3a57e00700 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a34a50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3a440058b0 tx=0x7f3a4400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ccaea3ae885ae4e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.733+0000 7f3a3ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.734+0000 7f3a3ea00700 1 -- 172.21.15.43:0/1038646253 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4402b040 con 0x556d118a3e40 2024-05-27T07:21:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.734+0000 7f3a3ea00700 1 -- 172.21.15.43:0/1038646253 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f3a4400fc90 con 0x556d118a3e40 2024-05-27T07:21:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.734+0000 7f3a6e311b80 1 -- 172.21.15.43:0/1038646253 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a4400f790 con 0x556d118a3e40 2024-05-27T07:21:24.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.734+0000 7f3a3ea00700 1 -- 172.21.15.43:0/1038646253 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f3a4402d300 con 0x556d118a3e40 2024-05-27T07:21:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.734+0000 7f3a6e311b80 10 client.16560 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.734+0000 7f3a6e311b80 1 -- 172.21.15.43:0/1038646253 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x556d11aa6720 con 0x556d118a3e40 2024-05-27T07:21:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.735+0000 7f3a3ea00700 1 -- 172.21.15.43:0/1038646253 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3a4400f790 con 0x556d118a3e40 2024-05-27T07:21:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.735+0000 7f3a3ea00700 1 -- 172.21.15.43:0/1038646253 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f3a44035070 con 0x556d118a3e40 2024-05-27T07:21:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.735+0000 7f3a6e311b80 10 client.16560 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.735+0000 7f3a6e311b80 10 client.16560 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:21:24.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.735+0000 7f3a6e311b80 20 client.16560 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:24.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.735+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x556d11aa6d10 0x556d11aa9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:24.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.735+0000 7f3a6e311b80 4 client.16560 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:21:24.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.735+0000 7f3a6e311b80 1 -- 172.21.15.43:0/1038646253 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d1194c570 con 0x556d11aa6d10 2024-05-27T07:21:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.736+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x556d11aa6d10 0x556d11aa9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a5ce00700 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x556d11aa6d10 0x556d11aa9130 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7bc55bf8c51ec82e server_cookie=70c849ed17bb6170 in_seq=0 out_seq=0 2024-05-27T07:21:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a3ea00700 10 client.16560 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:21:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a3ea00700 1 -- 172.21.15.43:0/1038646253 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d1194c570 con 0x556d11aa6d10 2024-05-27T07:21:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a3ea00700 10 client.16560 handle_command_reply: tid=0 2024-05-27T07:21:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a6e311b80 1 client.16560 shutdown 2024-05-27T07:21:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a6e311b80 1 -- 172.21.15.43:0/1038646253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 msgr2=0x556d11a34a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a34a50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3a440058b0 tx=0x7f3a4400f690 comp rx=0 tx=0).stop 2024-05-27T07:21:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a6e311b80 1 -- 172.21.15.43:0/1038646253 shutdown_connections 2024-05-27T07:21:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d11a28d30 0x556d11a434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556d118a3e40 0x556d11a34a50 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x556d11aa6d10 0x556d11aa9130 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a6e311b80 1 --2- 172.21.15.43:0/1038646253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556d11a22db0 0x556d11a34f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a6e311b80 1 -- 172.21.15.43:0/1038646253 >> 172.21.15.43:0/1038646253 conn(0x556d11903fc0 msgr2=0x556d11a2c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.737+0000 7f3a3ea00700 0 client.16560 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:21:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.738+0000 7f3a6e311b80 1 -- 172.21.15.43:0/1038646253 shutdown_connections 2024-05-27T07:21:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.738+0000 7f3a6e311b80 1 -- 172.21.15.43:0/1038646253 wait complete. 2024-05-27T07:21:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.738+0000 7f3a6e311b80 20 client.16560 trim_cache size 0 max 16384 2024-05-27T07:21:24.729 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:21:24.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 -- 172.21.15.43:0/2986715271 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3a400b2a70 msgr2=0x7f3a400b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:24.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3a400b2a70 0x7f3a400b4f80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3a50012c40 tx=0x7f3a500b0f90 comp rx=0 tx=0).stop 2024-05-27T07:21:24.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 -- 172.21.15.43:0/2986715271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a5810e900 msgr2=0x7f3a581a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:24.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a5810e900 0x7f3a581a17a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3a50007d70 tx=0x7f3a5000a340 comp rx=0 tx=0).stop 2024-05-27T07:21:24.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 -- 172.21.15.43:0/2986715271 shutdown_connections 2024-05-27T07:21:24.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3a400b2a70 0x7f3a400b4f80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a5810e900 0x7f3a581a17a0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581054d0 0x7f3a5819cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 --2- 172.21.15.43:0/2986715271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a58105e20 0x7f3a5819d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 -- 172.21.15.43:0/2986715271 >> 172.21.15.43:0/2986715271 conn(0x7f3a580fef40 msgr2=0x7f3a58103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 -- 172.21.15.43:0/2986715271 shutdown_connections 2024-05-27T07:21:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:24.739+0000 7f3a5f800700 1 -- 172.21.15.43:0/2986715271 wait complete. 2024-05-27T07:21:25.207 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39307 0.64 MB/sec execute 209 sec latency 47.827 ms 2024-05-27T07:21:25.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:21:26.142 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:26.207 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39379 0.64 MB/sec execute 210 sec latency 40.117 ms 2024-05-27T07:21:26.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:26 smithi043 ceph-mon[31531]: pgmap v250: 129 pgs: 129 active+clean; 67 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 419 KiB/s wr, 35 op/s 2024-05-27T07:21:26.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:26 smithi179 ceph-mon[35888]: pgmap v250: 129 pgs: 129 active+clean; 67 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 419 KiB/s wr, 35 op/s 2024-05-27T07:21:26.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:26 smithi132 ceph-mon[35283]: pgmap v250: 129 pgs: 129 active+clean; 67 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 419 KiB/s wr, 35 op/s 2024-05-27T07:21:27.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39470 0.63 MB/sec execute 211 sec latency 49.906 ms 2024-05-27T07:21:27.317 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:27 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:27.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:27 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:27.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:27 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:21:28.207 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39542 0.63 MB/sec execute 212 sec latency 46.707 ms 2024-05-27T07:21:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.264+0000 7f54b7200700 1 -- 172.21.15.43:0/2878136827 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b81043f0 msgr2=0x7f54b8104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.264+0000 7f54b7200700 1 --2- 172.21.15.43:0/2878136827 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b81043f0 0x7f54b8104800 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f54a80099c0 tx=0x7f54a8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.264+0000 7f54b7200700 1 -- 172.21.15.43:0/2878136827 shutdown_connections 2024-05-27T07:21:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.264+0000 7f54b7200700 1 --2- 172.21.15.43:0/2878136827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54b8109670 0x7f54b8110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.264+0000 7f54b7200700 1 --2- 172.21.15.43:0/2878136827 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b81043f0 0x7f54b8104800 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.264+0000 7f54b7200700 1 --2- 172.21.15.43:0/2878136827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54b8108d20 0x7f54b8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.264+0000 7f54b7200700 1 -- 172.21.15.43:0/2878136827 >> 172.21.15.43:0/2878136827 conn(0x7f54b8079d00 msgr2=0x7f54b807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.264+0000 7f54b7200700 1 -- 172.21.15.43:0/2878136827 shutdown_connections 2024-05-27T07:21:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.265+0000 7f54b7200700 1 -- 172.21.15.43:0/2878136827 wait complete. 2024-05-27T07:21:28.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:28 smithi043 ceph-mon[31531]: pgmap v251: 129 pgs: 129 active+clean; 68 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 664 KiB/s wr, 55 op/s 2024-05-27T07:21:28.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:28 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:28.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:28 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:28.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:28 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:21:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.266+0000 7f54b7200700 1 Processor -- start 2024-05-27T07:21:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.266+0000 7f54b7200700 1 -- start start 2024-05-27T07:21:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.266+0000 7f54b7200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b81043f0 0x7f54b81a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.266+0000 7f54b7200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54b8108d20 0x7f54b81a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.266+0000 7f54b7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54b8109670 0x7f54b819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.266+0000 7f54b7200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81141a0 con 0x7f54b81043f0 2024-05-27T07:21:28.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.266+0000 7f54b7200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f54b8114020 con 0x7f54b8109670 2024-05-27T07:21:28.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.266+0000 7f54b7200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f54b8114320 con 0x7f54b8108d20 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.267+0000 7f54b5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b81043f0 0x7f54b81a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.267+0000 7f54b5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b81043f0 0x7f54b81a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46568/0 (socket says 172.21.15.43:46568) 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.267+0000 7f54b5e00700 1 -- 172.21.15.43:0/2187025251 learned_addr learned my addr 172.21.15.43:0/2187025251 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.267+0000 7f54b6800700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54b8109670 0x7f54b819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.267+0000 7f54b5400700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54b8108d20 0x7f54b81a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.267+0000 7f54b5e00700 1 -- 172.21.15.43:0/2187025251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54b8108d20 msgr2=0x7f54b81a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.267+0000 7f54b5e00700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54b8108d20 0x7f54b81a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.267+0000 7f54b5e00700 1 -- 172.21.15.43:0/2187025251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54b8109670 msgr2=0x7f54b819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.267+0000 7f54b5e00700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54b8109670 0x7f54b819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.267+0000 7f54b5e00700 1 -- 172.21.15.43:0/2187025251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b819b050 con 0x7f54b81043f0 2024-05-27T07:21:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.268+0000 7f54b5e00700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b81043f0 0x7f54b81a0780 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f54a8005910 tx=0x7f54a8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.268+0000 7f54aea00700 1 -- 172.21.15.43:0/2187025251 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8009e60 con 0x7f54b81043f0 2024-05-27T07:21:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.268+0000 7f54aea00700 1 -- 172.21.15.43:0/2187025251 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f54a8023070 con 0x7f54b81043f0 2024-05-27T07:21:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.268+0000 7f54b7200700 1 -- 172.21.15.43:0/2187025251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b819b2e0 con 0x7f54b81043f0 2024-05-27T07:21:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.268+0000 7f54aea00700 1 -- 172.21.15.43:0/2187025251 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a801fd30 con 0x7f54b81043f0 2024-05-27T07:21:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.268+0000 7f54b7200700 1 -- 172.21.15.43:0/2187025251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b80759e0 con 0x7f54b81043f0 2024-05-27T07:21:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.273+0000 7f54aea00700 1 -- 172.21.15.43:0/2187025251 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f54a801f4d0 con 0x7f54b81043f0 2024-05-27T07:21:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.273+0000 7f54cb693b80 1 Processor -- start 2024-05-27T07:21:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.274+0000 7f54cb693b80 1 -- start start 2024-05-27T07:21:28.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.274+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 0x55d32a120ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.274+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a121410 0x55d32a12bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.274+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121820 0x55d32a12e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.274+0000 7f54cb693b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d32a04fe40 con 0x55d32a121820 2024-05-27T07:21:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.274+0000 7f54cb693b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d32a03f9e0 con 0x55d329fa2e40 2024-05-27T07:21:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.274+0000 7f54cb693b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d32a043a20 con 0x55d32a121410 2024-05-27T07:21:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.275+0000 7f54b6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 0x55d32a120ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.275+0000 7f54aea00700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f549c0b2a70 0x7f549c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.275+0000 7f54b5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121820 0x55d32a12e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.275+0000 7f54b6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 0x55d32a120ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:55344/0 (socket says 172.21.15.43:55344) 2024-05-27T07:21:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.275+0000 7f54b6800700 1 -- 172.21.15.43:0/784082174 learned_addr learned my addr 172.21.15.43:0/784082174 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:28.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.275+0000 7f54b5e00700 1 --2- 172.21.15.43:0/784082174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a121410 0x55d32a12bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.275+0000 7f54aea00700 1 -- 172.21.15.43:0/2187025251 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f54a80e7b80 con 0x7f54b81043f0 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.275+0000 7f54b6800700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f549c0b2a70 0x7f549c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.276+0000 7f54b6800700 1 -- 172.21.15.43:0/784082174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a121410 msgr2=0x55d32a12bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.276+0000 7f54b6800700 1 --2- 172.21.15.43:0/784082174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a121410 0x55d32a12bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.276+0000 7f54b6800700 1 -- 172.21.15.43:0/784082174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121820 msgr2=0x55d32a12e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.276+0000 7f54b6800700 1 --2- 172.21.15.43:0/784082174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121820 0x55d32a12e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.276+0000 7f54b6800700 1 -- 172.21.15.43:0/784082174 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d32a008420 con 0x55d329fa2e40 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.276+0000 7f54b6800700 1 --2- 172.21.15.43:0/784082174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 0x55d32a120ed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009fc0 tx=0x7f54b0004320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf751fcae72376f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.276+0000 7f54b6800700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f549c0b2a70 0x7f549c0b4f80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f54b001b000 tx=0x7f54b001d000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.277+0000 7f54a3400700 1 -- 172.21.15.43:0/784082174 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b001e040 con 0x55d329fa2e40 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.277+0000 7f54a3400700 1 -- 172.21.15.43:0/784082174 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f54b0007d50 con 0x55d329fa2e40 2024-05-27T07:21:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.277+0000 7f54a3400700 1 -- 172.21.15.43:0/784082174 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b001ab60 con 0x55d329fa2e40 2024-05-27T07:21:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.278+0000 7f54cb693b80 1 -- 172.21.15.43:0/784082174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 msgr2=0x55d32a120ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.278+0000 7f54cb693b80 1 --2- 172.21.15.43:0/784082174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 0x55d32a120ed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009fc0 tx=0x7f54b0004320 comp rx=0 tx=0).stop 2024-05-27T07:21:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.278+0000 7f54cb693b80 1 -- 172.21.15.43:0/784082174 shutdown_connections 2024-05-27T07:21:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.278+0000 7f54cb693b80 1 --2- 172.21.15.43:0/784082174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121820 0x55d32a12e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.278+0000 7f54cb693b80 1 --2- 172.21.15.43:0/784082174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 0x55d32a120ed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.278+0000 7f54cb693b80 1 --2- 172.21.15.43:0/784082174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a121410 0x55d32a12bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.278+0000 7f54cb693b80 1 -- 172.21.15.43:0/784082174 >> 172.21.15.43:0/784082174 conn(0x55d32a002fc0 msgr2=0x55d32a11cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.278+0000 7f54cb693b80 1 -- 172.21.15.43:0/784082174 shutdown_connections 2024-05-27T07:21:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.278+0000 7f54cb693b80 1 -- 172.21.15.43:0/784082174 wait complete. 2024-05-27T07:21:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.279+0000 7f54cb693b80 1 -- start start 2024-05-27T07:21:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.279+0000 7f54cb693b80 1 Processor -- start 2024-05-27T07:21:28.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d329fa2e40 0x55d32a12b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a121410 0x55d32a12bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121820 0x55d32a12c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54cb693b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d329f45fa0 con 0x55d329fa2e40 2024-05-27T07:21:28.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54cb693b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d329fe0760 con 0x55d32a121820 2024-05-27T07:21:28.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54cb693b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d32a12e930 con 0x55d32a121410 2024-05-27T07:21:28.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54cb693b80 10 client.0 _ll_register_callbacks cb 0x55d329fe9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:28.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54b5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d329fa2e40 0x55d32a12b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54b5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d329fa2e40 0x55d32a12b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46590/0 (socket says 172.21.15.43:46590) 2024-05-27T07:21:28.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54b5e00700 1 -- 172.21.15.43:0/2605628968 learned_addr learned my addr 172.21.15.43:0/2605628968 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:28.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54b6800700 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121820 0x55d32a12c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.280+0000 7f54b5400700 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a121410 0x55d32a12bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.281+0000 7f54b5e00700 1 -- 172.21.15.43:0/2605628968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a121410 msgr2=0x55d32a12bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.281+0000 7f54b5e00700 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a121410 0x55d32a12bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.281+0000 7f54b5e00700 1 -- 172.21.15.43:0/2605628968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121820 msgr2=0x55d32a12c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.281+0000 7f54b5e00700 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121820 0x55d32a12c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.281+0000 7f54b5e00700 1 -- 172.21.15.43:0/2605628968 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d329f8f8d0 con 0x55d329fa2e40 2024-05-27T07:21:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.281+0000 7f54b6800700 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121820 0x55d32a12c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.281+0000 7f54b5e00700 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d329fa2e40 0x55d32a12b690 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f54a80e5010 tx=0x7f54a8078470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388217c267c422fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.281+0000 7f5497e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.281+0000 7f5497e00700 1 -- 172.21.15.43:0/2605628968 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a80aa360 con 0x55d329fa2e40 2024-05-27T07:21:28.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.282+0000 7f5497e00700 1 -- 172.21.15.43:0/2605628968 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f54a80b7720 con 0x55d329fa2e40 2024-05-27T07:21:28.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.282+0000 7f54cb693b80 1 -- 172.21.15.43:0/2605628968 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54b0007d50 con 0x55d329fa2e40 2024-05-27T07:21:28.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.282+0000 7f5497e00700 1 -- 172.21.15.43:0/2605628968 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f54a80b72e0 con 0x55d329fa2e40 2024-05-27T07:21:28.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.282+0000 7f54cb693b80 10 client.16572 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:28.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.282+0000 7f54cb693b80 1 -- 172.21.15.43:0/2605628968 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55d32a010060 con 0x55d329fa2e40 2024-05-27T07:21:28.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.282+0000 7f5497e00700 1 -- 172.21.15.43:0/2605628968 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f54b0007d50 con 0x55d329fa2e40 2024-05-27T07:21:28.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.283+0000 7f5497e00700 1 -- 172.21.15.43:0/2605628968 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f54a80aa040 con 0x55d329fa2e40 2024-05-27T07:21:28.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.283+0000 7f54cb693b80 10 client.16572 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:28.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.283+0000 7f54cb693b80 10 client.16572 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:21:28.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.283+0000 7f54cb693b80 20 client.16572 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:28.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.283+0000 7f54cb693b80 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d32a13b8b0 0x55d32a13dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.283+0000 7f54cb693b80 4 client.16572 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:28.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.283+0000 7f54cb693b80 1 -- 172.21.15.43:0/2605628968 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d32a051d90 con 0x55d32a13b8b0 2024-05-27T07:21:28.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.283+0000 7f54b5400700 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d32a13b8b0 0x55d32a13dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.284+0000 7f54b5400700 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d32a13b8b0 0x55d32a13dcf0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b3b0213312008ba server_cookie=1377007805a3e04e in_seq=0 out_seq=0 2024-05-27T07:21:28.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.284+0000 7f5497e00700 10 client.16572 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:21:28.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.285+0000 7f5497e00700 1 -- 172.21.15.43:0/2605628968 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d32a051d90 con 0x55d32a13b8b0 2024-05-27T07:21:28.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.285+0000 7f5497e00700 10 client.16572 handle_command_reply: tid=0 2024-05-27T07:21:28.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.285+0000 7f54cb693b80 1 client.16572 shutdown 2024-05-27T07:21:28.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 1 -- 172.21.15.43:0/2605628968 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d329fa2e40 msgr2=0x55d32a12b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:28.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d329fa2e40 0x55d32a12b690 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f54a80e5010 tx=0x7f54a8078470 comp rx=0 tx=0).stop 2024-05-27T07:21:28.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 1 -- 172.21.15.43:0/2605628968 shutdown_connections 2024-05-27T07:21:28.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d32a13b8b0 0x55d32a13dcf0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121820 0x55d32a12c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d329fa2e40 0x55d32a12b690 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 1 --2- 172.21.15.43:0/2605628968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a121410 0x55d32a12bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f5497e00700 0 client.16572 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:21:28.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 1 -- 172.21.15.43:0/2605628968 >> 172.21.15.43:0/2605628968 conn(0x55d32a002fc0 msgr2=0x55d32a11bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:28.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 1 -- 172.21.15.43:0/2605628968 shutdown_connections 2024-05-27T07:21:28.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 1 -- 172.21.15.43:0/2605628968 wait complete. 2024-05-27T07:21:28.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.286+0000 7f54cb693b80 20 client.16572 trim_cache size 0 max 16384 2024-05-27T07:21:28.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.307+0000 7f54cb693b80 1 Processor -- start 2024-05-27T07:21:28.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.307+0000 7f54cb693b80 1 -- start start 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.307+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a002fc0 0x55d32a12db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.307+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 0x55d32a126e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.307+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121410 0x55d32a12bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.307+0000 7f54cb693b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81aed10 con 0x55d32a121410 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.307+0000 7f54cb693b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81af3e0 con 0x55d329fa2e40 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.307+0000 7f54cb693b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81acb00 con 0x55d32a002fc0 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.308+0000 7f54b6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121410 0x55d32a12bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.308+0000 7f54b6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121410 0x55d32a12bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46598/0 (socket says 172.21.15.43:46598) 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.308+0000 7f54b6800700 1 -- 172.21.15.43:0/805609774 learned_addr learned my addr 172.21.15.43:0/805609774 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.308+0000 7f54b5e00700 1 --2- 172.21.15.43:0/805609774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a002fc0 0x55d32a12db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.308+0000 7f54b5400700 1 --2- 172.21.15.43:0/805609774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 0x55d32a126e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.308+0000 7f54b6800700 1 -- 172.21.15.43:0/805609774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a002fc0 msgr2=0x55d32a12db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:28.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.308+0000 7f54b6800700 1 --2- 172.21.15.43:0/805609774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a002fc0 0x55d32a12db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.308+0000 7f54b6800700 1 -- 172.21.15.43:0/805609774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 msgr2=0x55d32a126e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:28.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.308+0000 7f54b6800700 1 --2- 172.21.15.43:0/805609774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 0x55d32a126e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.308+0000 7f54b6800700 1 -- 172.21.15.43:0/805609774 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d32a11e350 con 0x55d32a121410 2024-05-27T07:21:28.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.309+0000 7f54b6800700 1 --2- 172.21.15.43:0/805609774 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121410 0x55d32a12bd60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009220 tx=0x7f54b0009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce4b58723029806 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:28.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.309+0000 7f54a2a00700 1 -- 172.21.15.43:0/805609774 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0033070 con 0x55d32a121410 2024-05-27T07:21:28.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.309+0000 7f54a2a00700 1 -- 172.21.15.43:0/805609774 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f54b0027070 con 0x55d32a121410 2024-05-27T07:21:28.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.309+0000 7f54a2a00700 1 -- 172.21.15.43:0/805609774 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b001e040 con 0x55d32a121410 2024-05-27T07:21:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.310+0000 7f54cb693b80 1 -- 172.21.15.43:0/805609774 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121410 msgr2=0x55d32a12bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.310+0000 7f54cb693b80 1 --2- 172.21.15.43:0/805609774 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121410 0x55d32a12bd60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009220 tx=0x7f54b0009f50 comp rx=0 tx=0).stop 2024-05-27T07:21:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.310+0000 7f54cb693b80 1 -- 172.21.15.43:0/805609774 shutdown_connections 2024-05-27T07:21:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.310+0000 7f54cb693b80 1 --2- 172.21.15.43:0/805609774 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a121410 0x55d32a12bd60 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.310+0000 7f54cb693b80 1 --2- 172.21.15.43:0/805609774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d32a002fc0 0x55d32a12db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.310+0000 7f54cb693b80 1 --2- 172.21.15.43:0/805609774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d329fa2e40 0x55d32a126e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.310+0000 7f54cb693b80 1 -- 172.21.15.43:0/805609774 >> 172.21.15.43:0/805609774 conn(0x55d32a115f60 msgr2=0x55d32a12b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.311+0000 7f54cb693b80 1 -- 172.21.15.43:0/805609774 shutdown_connections 2024-05-27T07:21:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.311+0000 7f54cb693b80 1 -- 172.21.15.43:0/805609774 wait complete. 2024-05-27T07:21:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.311+0000 7f54cb693b80 1 -- start start 2024-05-27T07:21:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.311+0000 7f54cb693b80 1 Processor -- start 2024-05-27T07:21:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.312+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d329fa2e40 0x55d32a133f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.312+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a002fc0 0x55d32a12f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.312+0000 7f54cb693b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121410 0x55d32a12f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.312+0000 7f54cb693b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81ad1d0 con 0x55d32a002fc0 2024-05-27T07:21:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.312+0000 7f54cb693b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81ae640 con 0x55d32a121410 2024-05-27T07:21:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.312+0000 7f54cb693b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81ad8a0 con 0x55d329fa2e40 2024-05-27T07:21:28.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.312+0000 7f54cb693b80 10 client.0 _ll_register_callbacks cb 0x55d329fe9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:28.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.313+0000 7f54b5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a002fc0 0x55d32a12f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.313+0000 7f54b5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a002fc0 0x55d32a12f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46602/0 (socket says 172.21.15.43:46602) 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.313+0000 7f54b5400700 1 -- 172.21.15.43:0/576091255 learned_addr learned my addr 172.21.15.43:0/576091255 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.313+0000 7f54b5e00700 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d329fa2e40 0x55d32a133f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.313+0000 7f54b6800700 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121410 0x55d32a12f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.313+0000 7f54b5400700 1 -- 172.21.15.43:0/576091255 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d329fa2e40 msgr2=0x55d32a133f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.313+0000 7f54b5400700 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d329fa2e40 0x55d32a133f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.313+0000 7f54b5400700 1 -- 172.21.15.43:0/576091255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121410 msgr2=0x55d32a12f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.313+0000 7f54b5400700 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121410 0x55d32a12f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.313+0000 7f54b5400700 1 -- 172.21.15.43:0/576091255 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d32a12fd50 con 0x55d32a002fc0 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.314+0000 7f54b5400700 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a002fc0 0x55d32a12f2d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f54a401efd0 tx=0x7f54a4009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8be8585456861b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.314+0000 7f5497e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:28.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.314+0000 7f5497e00700 1 -- 172.21.15.43:0/576091255 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a401e380 con 0x55d32a002fc0 2024-05-27T07:21:28.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.314+0000 7f5497e00700 1 -- 172.21.15.43:0/576091255 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f54a401e520 con 0x55d32a002fc0 2024-05-27T07:21:28.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.314+0000 7f54cb693b80 1 -- 172.21.15.43:0/576091255 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54b0027070 con 0x55d32a002fc0 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.314+0000 7f5497e00700 1 -- 172.21.15.43:0/576091255 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f54a402f050 con 0x55d32a002fc0 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.314+0000 7f54cb693b80 10 client.16578 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.315+0000 7f54cb693b80 1 -- 172.21.15.43:0/576091255 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55d32a1a5fa0 con 0x55d32a002fc0 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.315+0000 7f5497e00700 1 -- 172.21.15.43:0/576091255 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f54b0027070 con 0x55d32a002fc0 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.315+0000 7f5497e00700 1 -- 172.21.15.43:0/576091255 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f54a4012870 con 0x55d32a002fc0 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.315+0000 7f54cb693b80 10 client.16578 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.315+0000 7f54cb693b80 10 client.16578 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.315+0000 7f54cb693b80 20 client.16578 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.315+0000 7f54cb693b80 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d32a121820 0x55d32a1a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.315+0000 7f54cb693b80 4 client.16578 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:21:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.315+0000 7f54cb693b80 1 -- 172.21.15.43:0/576091255 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d32a04b570 con 0x55d32a121820 2024-05-27T07:21:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.316+0000 7f54b5e00700 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d32a121820 0x55d32a1a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.317+0000 7f54b5e00700 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d32a121820 0x55d32a1a86a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5752f6e3b2536a14 server_cookie=d733a860241b7aa1 in_seq=0 out_seq=0 2024-05-27T07:21:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.317+0000 7f5497e00700 10 client.16578 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:21:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.317+0000 7f5497e00700 1 -- 172.21.15.43:0/576091255 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d32a04b570 con 0x55d32a121820 2024-05-27T07:21:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.317+0000 7f5497e00700 10 client.16578 handle_command_reply: tid=0 2024-05-27T07:21:28.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.317+0000 7f54cb693b80 1 client.16578 shutdown 2024-05-27T07:21:28.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 1 -- 172.21.15.43:0/576091255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a002fc0 msgr2=0x55d32a12f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:28.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a002fc0 0x55d32a12f2d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f54a401efd0 tx=0x7f54a4009e70 comp rx=0 tx=0).stop 2024-05-27T07:21:28.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 1 -- 172.21.15.43:0/576091255 shutdown_connections 2024-05-27T07:21:28.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d32a121820 0x55d32a1a86a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d32a121410 0x55d32a12f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d329fa2e40 0x55d32a133f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 1 --2- 172.21.15.43:0/576091255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d32a002fc0 0x55d32a12f2d0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 1 -- 172.21.15.43:0/576091255 >> 172.21.15.43:0/576091255 conn(0x55d32a115f60 msgr2=0x55d32a1369c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:28.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f5497e00700 0 client.16578 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:21:28.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 1 -- 172.21.15.43:0/576091255 shutdown_connections 2024-05-27T07:21:28.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 1 -- 172.21.15.43:0/576091255 wait complete. 2024-05-27T07:21:28.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.318+0000 7f54cb693b80 20 client.16578 trim_cache size 0 max 16384 2024-05-27T07:21:28.309 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:21:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.319+0000 7f54b7200700 1 -- 172.21.15.43:0/2187025251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f549c0b2a70 msgr2=0x7f549c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.319+0000 7f54b7200700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f549c0b2a70 0x7f549c0b4f80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f54b001b000 tx=0x7f54b001d000 comp rx=0 tx=0).stop 2024-05-27T07:21:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.319+0000 7f54b7200700 1 -- 172.21.15.43:0/2187025251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b81043f0 msgr2=0x7f54b81a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.319+0000 7f54b7200700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b81043f0 0x7f54b81a0780 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f54a8005910 tx=0x7f54a8005e80 comp rx=0 tx=0).stop 2024-05-27T07:21:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.320+0000 7f54b7200700 1 -- 172.21.15.43:0/2187025251 shutdown_connections 2024-05-27T07:21:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.320+0000 7f54b7200700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f549c0b2a70 0x7f549c0b4f80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.320+0000 7f54b7200700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54b8109670 0x7f54b819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.320+0000 7f54b7200700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54b81043f0 0x7f54b81a0780 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.320+0000 7f54b7200700 1 --2- 172.21.15.43:0/2187025251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54b8108d20 0x7f54b81a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.320+0000 7f54b7200700 1 -- 172.21.15.43:0/2187025251 >> 172.21.15.43:0/2187025251 conn(0x7f54b8079d00 msgr2=0x7f54b8101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.320+0000 7f54b7200700 1 -- 172.21.15.43:0/2187025251 shutdown_connections 2024-05-27T07:21:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:28.320+0000 7f54b7200700 1 -- 172.21.15.43:0/2187025251 wait complete. 2024-05-27T07:21:28.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:28 smithi179 ceph-mon[35888]: pgmap v251: 129 pgs: 129 active+clean; 68 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 664 KiB/s wr, 55 op/s 2024-05-27T07:21:28.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:28 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:28.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:28 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:28.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:28 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:21:28.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:28 smithi132 ceph-mon[35283]: pgmap v251: 129 pgs: 129 active+clean; 68 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 664 KiB/s wr, 55 op/s 2024-05-27T07:21:28.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:28 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:21:28.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:28 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:21:28.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:28 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:21:29.207 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39637 0.63 MB/sec execute 213 sec latency 49.315 ms 2024-05-27T07:21:29.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:21:29.733 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:30.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39748 0.63 MB/sec execute 214 sec latency 50.657 ms 2024-05-27T07:21:30.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:30 smithi043 ceph-mon[31531]: pgmap v252: 129 pgs: 129 active+clean; 68 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 449 KiB/s wr, 36 op/s 2024-05-27T07:21:30.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:30 smithi179 ceph-mon[35888]: pgmap v252: 129 pgs: 129 active+clean; 68 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 449 KiB/s wr, 36 op/s 2024-05-27T07:21:30.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:30 smithi132 ceph-mon[35283]: pgmap v252: 129 pgs: 129 active+clean; 68 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 449 KiB/s wr, 36 op/s 2024-05-27T07:21:31.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39880 0.63 MB/sec execute 215 sec latency 48.568 ms 2024-05-27T07:21:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.107+0000 7f9c7c200700 1 -- 172.21.15.43:0/1433803431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c741031c0 msgr2=0x7f9c74111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.107+0000 7f9c7c200700 1 --2- 172.21.15.43:0/1433803431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c741031c0 0x7f9c74111160 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00b0a0 tx=0x7f9c6c00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.107+0000 7f9c7c200700 1 -- 172.21.15.43:0/1433803431 shutdown_connections 2024-05-27T07:21:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.107+0000 7f9c7c200700 1 --2- 172.21.15.43:0/1433803431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c741031c0 0x7f9c74111160 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.107+0000 7f9c7c200700 1 --2- 172.21.15.43:0/1433803431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c741092f0 0x7f9c74109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.107+0000 7f9c7c200700 1 --2- 172.21.15.43:0/1433803431 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c74102870 0x7f9c74102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.107+0000 7f9c7c200700 1 -- 172.21.15.43:0/1433803431 >> 172.21.15.43:0/1433803431 conn(0x7f9c740fe6e0 msgr2=0x7f9c74100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:32.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.107+0000 7f9c7c200700 1 -- 172.21.15.43:0/1433803431 shutdown_connections 2024-05-27T07:21:32.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.108+0000 7f9c7c200700 1 -- 172.21.15.43:0/1433803431 wait complete. 2024-05-27T07:21:32.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.109+0000 7f9c7c200700 1 Processor -- start 2024-05-27T07:21:32.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.109+0000 7f9c7c200700 1 -- start start 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.110+0000 7f9c7c200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c74102870 0x7f9c7419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.110+0000 7f9c7c200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c741031c0 0x7f9c7419b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.110+0000 7f9c7c200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c741092f0 0x7f9c7419f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.110+0000 7f9c7c200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9c74114490 con 0x7f9c741031c0 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.110+0000 7f9c7c200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9c74114310 con 0x7f9c74102870 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.110+0000 7f9c7c200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9c74114610 con 0x7f9c741092f0 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.110+0000 7f9c78e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c741031c0 0x7f9c7419b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.110+0000 7f9c78e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c741031c0 0x7f9c7419b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46622/0 (socket says 172.21.15.43:46622) 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.110+0000 7f9c78e00700 1 -- 172.21.15.43:0/2385895706 learned_addr learned my addr 172.21.15.43:0/2385895706 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.111+0000 7f9c79800700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c74102870 0x7f9c7419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.111+0000 7f9c7a200700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c741092f0 0x7f9c7419f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.111+0000 7f9c78e00700 1 -- 172.21.15.43:0/2385895706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c741092f0 msgr2=0x7f9c7419f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:32.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.111+0000 7f9c78e00700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c741092f0 0x7f9c7419f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.111+0000 7f9c78e00700 1 -- 172.21.15.43:0/2385895706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c74102870 msgr2=0x7f9c7419ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:32.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.111+0000 7f9c78e00700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c74102870 0x7f9c7419ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.111+0000 7f9c78e00700 1 -- 172.21.15.43:0/2385895706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c7419fcb0 con 0x7f9c741031c0 2024-05-27T07:21:32.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.111+0000 7f9c78e00700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c741031c0 0x7f9c7419b380 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9c6000d950 tx=0x7f9c6000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:32.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.112+0000 7f9c72000700 1 -- 172.21.15.43:0/2385895706 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c60016070 con 0x7f9c741031c0 2024-05-27T07:21:32.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.112+0000 7f9c7c200700 1 -- 172.21.15.43:0/2385895706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c7419ffa0 con 0x7f9c741031c0 2024-05-27T07:21:32.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.112+0000 7f9c72000700 1 -- 172.21.15.43:0/2385895706 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c60004830 con 0x7f9c741031c0 2024-05-27T07:21:32.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.112+0000 7f9c72000700 1 -- 172.21.15.43:0/2385895706 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c600053e0 con 0x7f9c741031c0 2024-05-27T07:21:32.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.112+0000 7f9c7c200700 1 -- 172.21.15.43:0/2385895706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c741a78c0 con 0x7f9c741031c0 2024-05-27T07:21:32.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.116+0000 7f9c72000700 1 -- 172.21.15.43:0/2385895706 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9c60005580 con 0x7f9c741031c0 2024-05-27T07:21:32.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.117+0000 7f9c8ad44b80 1 Processor -- start 2024-05-27T07:21:32.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.117+0000 7f9c8ad44b80 1 -- start start 2024-05-27T07:21:32.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.118+0000 7f9c72000700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c5c0b2a70 0x7f9c5c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.118+0000 7f9c72000700 1 -- 172.21.15.43:0/2385895706 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f9c600e2790 con 0x7f9c741031c0 2024-05-27T07:21:32.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.118+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d5959f0 0x55820d6283f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.118+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d4a9e40 0x55820d628930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.118+0000 7f9c7a200700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c5c0b2a70 0x7f9c5c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.118+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d639930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.118+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55820d556e40 con 0x55820d637510 2024-05-27T07:21:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.118+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55820d5469e0 con 0x55820d4a9e40 2024-05-27T07:21:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.118+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55820d54aa20 con 0x55820d5959f0 2024-05-27T07:21:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.118+0000 7f9c7a200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d639930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.119+0000 7f9c7a200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d639930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46626/0 (socket says 172.21.15.43:46626) 2024-05-27T07:21:32.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.119+0000 7f9c7a200700 1 -- 172.21.15.43:0/1122379057 learned_addr learned my addr 172.21.15.43:0/1122379057 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:32.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.119+0000 7f9c79800700 1 --2- 172.21.15.43:0/1122379057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d5959f0 0x55820d6283f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.119+0000 7f9c78e00700 1 --2- 172.21.15.43:0/1122379057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d4a9e40 0x55820d628930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.119+0000 7f9c7a200700 1 -- 172.21.15.43:0/1122379057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d5959f0 msgr2=0x55820d6283f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:32.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.119+0000 7f9c7a200700 1 --2- 172.21.15.43:0/1122379057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d5959f0 0x55820d6283f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.119+0000 7f9c7a200700 1 -- 172.21.15.43:0/1122379057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d4a9e40 msgr2=0x55820d628930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:32.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.119+0000 7f9c7a200700 1 --2- 172.21.15.43:0/1122379057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d4a9e40 0x55820d628930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.119+0000 7f9c7a200700 1 -- 172.21.15.43:0/1122379057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55820d50f420 con 0x55820d637510 2024-05-27T07:21:32.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.119+0000 7f9c7a200700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c5c0b2a70 0x7f9c5c0b4f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c015000 tx=0x7f9c6c009d60 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:32.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.120+0000 7f9c7a200700 1 --2- 172.21.15.43:0/1122379057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d639930 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c004c90 tx=0x7f9c6c025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24bfdb4eccaf392f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:32.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.120+0000 7f9c66a00700 1 -- 172.21.15.43:0/1122379057 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c009b80 con 0x55820d637510 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.120+0000 7f9c66a00700 1 -- 172.21.15.43:0/1122379057 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c6c0094c0 con 0x55820d637510 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.120+0000 7f9c66a00700 1 -- 172.21.15.43:0/1122379057 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c0208e0 con 0x55820d637510 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.121+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/1122379057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 msgr2=0x55820d639930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.121+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/1122379057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d639930 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c004c90 tx=0x7f9c6c025000 comp rx=0 tx=0).stop 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.121+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/1122379057 shutdown_connections 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.121+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/1122379057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d639930 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.121+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/1122379057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d5959f0 0x55820d6283f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.121+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/1122379057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d4a9e40 0x55820d628930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.121+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/1122379057 >> 172.21.15.43:0/1122379057 conn(0x55820d543390 msgr2=0x55820d6244c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.121+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/1122379057 shutdown_connections 2024-05-27T07:21:32.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.121+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/1122379057 wait complete. 2024-05-27T07:21:32.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.122+0000 7f9c8ad44b80 1 -- start start 2024-05-27T07:21:32.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.122+0000 7f9c8ad44b80 1 Processor -- start 2024-05-27T07:21:32.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d634570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d5959f0 0x55820d634ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d637510 0x55820d634ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55820d44cfa0 con 0x55820d4a9e40 2024-05-27T07:21:32.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55820d4e7760 con 0x55820d637510 2024-05-27T07:21:32.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55820d639fc0 con 0x55820d5959f0 2024-05-27T07:21:32.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c8ad44b80 10 client.0 _ll_register_callbacks cb 0x55820d4f0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:32.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c79800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d634570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c79800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d634570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46636/0 (socket says 172.21.15.43:46636) 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c79800700 1 -- 172.21.15.43:0/3126329536 learned_addr learned my addr 172.21.15.43:0/3126329536 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.123+0000 7f9c78e00700 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d5959f0 0x55820d634ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.124+0000 7f9c7a200700 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d637510 0x55820d634ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.124+0000 7f9c79800700 1 -- 172.21.15.43:0/3126329536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d5959f0 msgr2=0x55820d634ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.124+0000 7f9c79800700 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d5959f0 0x55820d634ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.124+0000 7f9c79800700 1 -- 172.21.15.43:0/3126329536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d637510 msgr2=0x55820d634ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.124+0000 7f9c79800700 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d637510 0x55820d634ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.124+0000 7f9c79800700 1 -- 172.21.15.43:0/3126329536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55820d4968d0 con 0x55820d4a9e40 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.124+0000 7f9c79800700 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d634570 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9c68012730 tx=0x7f9c68012af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2fa7be5e9eaec2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.124+0000 7f9c5b400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.124+0000 7f9c5b400700 1 -- 172.21.15.43:0/3126329536 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c68004050 con 0x55820d4a9e40 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.125+0000 7f9c5b400700 1 -- 172.21.15.43:0/3126329536 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f9c680041f0 con 0x55820d4a9e40 2024-05-27T07:21:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.125+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/3126329536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c6c0094c0 con 0x55820d4a9e40 2024-05-27T07:21:32.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.125+0000 7f9c5b400700 1 -- 172.21.15.43:0/3126329536 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f9c68025050 con 0x55820d4a9e40 2024-05-27T07:21:32.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.125+0000 7f9c8ad44b80 10 client.16590 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:32.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.125+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/3126329536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55820d517060 con 0x55820d4a9e40 2024-05-27T07:21:32.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.125+0000 7f9c5b400700 1 -- 172.21.15.43:0/3126329536 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c6c0094c0 con 0x55820d4a9e40 2024-05-27T07:21:32.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.126+0000 7f9c5b400700 1 -- 172.21.15.43:0/3126329536 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f9c6800f560 con 0x55820d4a9e40 2024-05-27T07:21:32.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.126+0000 7f9c8ad44b80 10 client.16590 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:32.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.126+0000 7f9c8ad44b80 10 client.16590 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:21:32.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.126+0000 7f9c8ad44b80 20 client.16590 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:32.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.126+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55820d647220 0x55820d6496d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.126+0000 7f9c8ad44b80 4 client.16590 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:32.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.126+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/3126329536 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55820d558d90 con 0x55820d647220 2024-05-27T07:21:32.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.127+0000 7f9c78e00700 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55820d647220 0x55820d6496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.128+0000 7f9c78e00700 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55820d647220 0x55820d6496d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a1e8f7d5842c977c server_cookie=7112e47d5a64011 in_seq=0 out_seq=0 2024-05-27T07:21:32.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.128+0000 7f9c5b400700 10 client.16590 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:21:32.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c5b400700 1 -- 172.21.15.43:0/3126329536 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55820d558d90 con 0x55820d647220 2024-05-27T07:21:32.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c5b400700 10 client.16590 handle_command_reply: tid=0 2024-05-27T07:21:32.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 client.16590 shutdown 2024-05-27T07:21:32.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/3126329536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 msgr2=0x55820d634570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:32.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d634570 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9c68012730 tx=0x7f9c68012af0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/3126329536 shutdown_connections 2024-05-27T07:21:32.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55820d647220 0x55820d6496d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d637510 0x55820d634ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d634570 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/3126329536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d5959f0 0x55820d634ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c5b400700 0 client.16590 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:21:32.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/3126329536 >> 172.21.15.43:0/3126329536 conn(0x55820d543390 msgr2=0x55820d6284f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:32.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/3126329536 shutdown_connections 2024-05-27T07:21:32.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/3126329536 wait complete. 2024-05-27T07:21:32.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.130+0000 7f9c8ad44b80 20 client.16590 trim_cache size 0 max 16384 2024-05-27T07:21:32.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.150+0000 7f9c8ad44b80 1 Processor -- start 2024-05-27T07:21:32.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.150+0000 7f9c8ad44b80 1 -- start start 2024-05-27T07:21:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.151+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d628b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.151+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d5959f0 0x55820d6334d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.151+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d637510 0x55820d643060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.151+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9c741aeff0 con 0x55820d4a9e40 2024-05-27T07:21:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.151+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9c741af6c0 con 0x55820d5959f0 2024-05-27T07:21:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.151+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9c741acde0 con 0x55820d637510 2024-05-27T07:21:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.151+0000 7f9c79800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d628b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.151+0000 7f9c79800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d628b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46640/0 (socket says 172.21.15.43:46640) 2024-05-27T07:21:32.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.151+0000 7f9c79800700 1 -- 172.21.15.43:0/1629197546 learned_addr learned my addr 172.21.15.43:0/1629197546 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.152+0000 7f9c78e00700 1 --2- 172.21.15.43:0/1629197546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d5959f0 0x55820d6334d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.152+0000 7f9c7a200700 1 --2- 172.21.15.43:0/1629197546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d637510 0x55820d643060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.152+0000 7f9c79800700 1 -- 172.21.15.43:0/1629197546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d637510 msgr2=0x55820d643060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.152+0000 7f9c79800700 1 --2- 172.21.15.43:0/1629197546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d637510 0x55820d643060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.152+0000 7f9c79800700 1 -- 172.21.15.43:0/1629197546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d5959f0 msgr2=0x55820d6334d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.152+0000 7f9c79800700 1 --2- 172.21.15.43:0/1629197546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d5959f0 0x55820d6334d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.152+0000 7f9c79800700 1 -- 172.21.15.43:0/1629197546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55820d6259c0 con 0x55820d4a9e40 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.152+0000 7f9c7a200700 1 --2- 172.21.15.43:0/1629197546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d637510 0x55820d643060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.152+0000 7f9c79800700 1 --2- 172.21.15.43:0/1629197546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d628b00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800f9b0 tx=0x7f9c6800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741392e791d6acba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.153+0000 7f9c66000700 1 -- 172.21.15.43:0/1629197546 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6802e070 con 0x55820d4a9e40 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.153+0000 7f9c66000700 1 -- 172.21.15.43:0/1629197546 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c6800fba0 con 0x55820d4a9e40 2024-05-27T07:21:32.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.153+0000 7f9c66000700 1 -- 172.21.15.43:0/1629197546 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6801b830 con 0x55820d4a9e40 2024-05-27T07:21:32.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.154+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/1629197546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 msgr2=0x55820d628b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:32.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.154+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/1629197546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d628b00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800f9b0 tx=0x7f9c6800f790 comp rx=0 tx=0).stop 2024-05-27T07:21:32.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.154+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/1629197546 shutdown_connections 2024-05-27T07:21:32.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.154+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/1629197546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d637510 0x55820d643060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.154+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/1629197546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d4a9e40 0x55820d628b00 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.154+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/1629197546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d5959f0 0x55820d6334d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.154+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/1629197546 >> 172.21.15.43:0/1629197546 conn(0x55820d543390 msgr2=0x55820d6283f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.154+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/1629197546 shutdown_connections 2024-05-27T07:21:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.154+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/1629197546 wait complete. 2024-05-27T07:21:32.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.155+0000 7f9c8ad44b80 1 -- start start 2024-05-27T07:21:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.155+0000 7f9c8ad44b80 1 Processor -- start 2024-05-27T07:21:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d4a9e40 0x55820d637b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d5959f0 0x55820d638040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c8ad44b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d63e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9c741ad4b0 con 0x55820d637510 2024-05-27T07:21:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9c741ae920 con 0x55820d5959f0 2024-05-27T07:21:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c8ad44b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9c741adb80 con 0x55820d4a9e40 2024-05-27T07:21:32.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c8ad44b80 10 client.0 _ll_register_callbacks cb 0x55820d4f0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c7a200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d63e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c7a200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d63e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46648/0 (socket says 172.21.15.43:46648) 2024-05-27T07:21:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c7a200700 1 -- 172.21.15.43:0/709974499 learned_addr learned my addr 172.21.15.43:0/709974499 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c79800700 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d4a9e40 0x55820d637b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.156+0000 7f9c78e00700 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d5959f0 0x55820d638040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.157+0000 7f9c7a200700 1 -- 172.21.15.43:0/709974499 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d4a9e40 msgr2=0x55820d637b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.157+0000 7f9c7a200700 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d4a9e40 0x55820d637b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.157+0000 7f9c7a200700 1 -- 172.21.15.43:0/709974499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d5959f0 msgr2=0x55820d638040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.157+0000 7f9c7a200700 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d5959f0 0x55820d638040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.157+0000 7f9c7a200700 1 -- 172.21.15.43:0/709974499 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55820d638920 con 0x55820d637510 2024-05-27T07:21:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.157+0000 7f9c7a200700 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d63e250 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009fd0 tx=0x7f9c6c00bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71218dd3df483792 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.157+0000 7f9c5b400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.157+0000 7f9c5b400700 1 -- 172.21.15.43:0/709974499 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c03a060 con 0x55820d637510 2024-05-27T07:21:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.157+0000 7f9c5b400700 1 -- 172.21.15.43:0/709974499 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7f9c6c00f040 con 0x55820d637510 2024-05-27T07:21:32.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.157+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/709974499 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c6800fba0 con 0x55820d637510 2024-05-27T07:21:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.158+0000 7f9c5b400700 1 -- 172.21.15.43:0/709974499 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f9c6c041050 con 0x55820d637510 2024-05-27T07:21:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.158+0000 7f9c8ad44b80 10 client.16596 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.158+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/709974499 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55820d63f750 con 0x55820d637510 2024-05-27T07:21:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.158+0000 7f9c5b400700 1 -- 172.21.15.43:0/709974499 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c6800fba0 con 0x55820d637510 2024-05-27T07:21:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.158+0000 7f9c5b400700 1 -- 172.21.15.43:0/709974499 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7f9c6c011070 con 0x55820d637510 2024-05-27T07:21:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.158+0000 7f9c8ad44b80 10 client.16596 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.158+0000 7f9c8ad44b80 10 client.16596 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:21:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.158+0000 7f9c8ad44b80 20 client.16596 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.158+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55820d6b1d60 0x55820d6b4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:32.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.159+0000 7f9c8ad44b80 4 client.16596 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:21:32.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.159+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/709974499 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55820d552570 con 0x55820d6b1d60 2024-05-27T07:21:32.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.159+0000 7f9c79800700 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55820d6b1d60 0x55820d6b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:32.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.161+0000 7f9c79800700 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55820d6b1d60 0x55820d6b4180 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f1b4d8eee94ccc27 server_cookie=1e5ad7dac2b7d718 in_seq=0 out_seq=0 2024-05-27T07:21:32.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.161+0000 7f9c5b400700 10 client.16596 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:21:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c5b400700 1 -- 172.21.15.43:0/709974499 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55820d552570 con 0x55820d6b1d60 2024-05-27T07:21:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c5b400700 10 client.16596 handle_command_reply: tid=0 2024-05-27T07:21:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c8ad44b80 1 client.16596 shutdown 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/709974499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 msgr2=0x55820d63e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d63e250 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009fd0 tx=0x7f9c6c00bdc0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/709974499 shutdown_connections 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55820d6b1d60 0x55820d6b4180 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55820d637510 0x55820d63e250 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55820d4a9e40 0x55820d637b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c8ad44b80 1 --2- 172.21.15.43:0/709974499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55820d5959f0 0x55820d638040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c5b400700 0 client.16596 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/709974499 >> 172.21.15.43:0/709974499 conn(0x55820d543390 msgr2=0x55820d62f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.162+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/709974499 shutdown_connections 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.163+0000 7f9c8ad44b80 1 -- 172.21.15.43:0/709974499 wait complete. 2024-05-27T07:21:32.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.163+0000 7f9c8ad44b80 20 client.16596 trim_cache size 0 max 16384 2024-05-27T07:21:32.154 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:21:32.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 -- 172.21.15.43:0/2385895706 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c5c0b2a70 msgr2=0x7f9c5c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:32.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c5c0b2a70 0x7f9c5c0b4f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c015000 tx=0x7f9c6c009d60 comp rx=0 tx=0).stop 2024-05-27T07:21:32.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 -- 172.21.15.43:0/2385895706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c741031c0 msgr2=0x7f9c7419b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:32.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c741031c0 0x7f9c7419b380 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9c6000d950 tx=0x7f9c6000dc60 comp rx=0 tx=0).stop 2024-05-27T07:21:32.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 -- 172.21.15.43:0/2385895706 shutdown_connections 2024-05-27T07:21:32.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c5c0b2a70 0x7f9c5c0b4f80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c741092f0 0x7f9c7419f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c74102870 0x7f9c7419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 --2- 172.21.15.43:0/2385895706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c741031c0 0x7f9c7419b380 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:32.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 -- 172.21.15.43:0/2385895706 >> 172.21.15.43:0/2385895706 conn(0x7f9c740fe6e0 msgr2=0x7f9c7410a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:32.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 -- 172.21.15.43:0/2385895706 shutdown_connections 2024-05-27T07:21:32.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:32.164+0000 7f9c7c200700 1 -- 172.21.15.43:0/2385895706 wait complete. 2024-05-27T07:21:32.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39995 0.62 MB/sec execute 216 sec latency 60.268 ms 2024-05-27T07:21:32.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:32 smithi043 ceph-mon[31531]: pgmap v253: 129 pgs: 129 active+clean; 69 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 597 KiB/s wr, 60 op/s 2024-05-27T07:21:32.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:32 smithi179 ceph-mon[35888]: pgmap v253: 129 pgs: 129 active+clean; 69 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 597 KiB/s wr, 60 op/s 2024-05-27T07:21:32.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:32 smithi132 ceph-mon[35283]: pgmap v253: 129 pgs: 129 active+clean; 69 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 597 KiB/s wr, 60 op/s 2024-05-27T07:21:32.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:21:33.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 40114 0.63 MB/sec execute 217 sec latency 54.704 ms 2024-05-27T07:21:33.457 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:34.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 40237 0.62 MB/sec execute 218 sec latency 48.994 ms 2024-05-27T07:21:34.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:34 smithi043 ceph-mon[31531]: pgmap v254: 129 pgs: 129 active+clean; 69 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 394 KiB/s wr, 42 op/s 2024-05-27T07:21:34.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:34 smithi179 ceph-mon[35888]: pgmap v254: 129 pgs: 129 active+clean; 69 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 394 KiB/s wr, 42 op/s 2024-05-27T07:21:34.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:34 smithi132 ceph-mon[35283]: pgmap v254: 129 pgs: 129 active+clean; 69 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 394 KiB/s wr, 42 op/s 2024-05-27T07:21:35.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 40345 0.62 MB/sec execute 219 sec latency 50.300 ms 2024-05-27T07:21:35.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.902+0000 7fad85c00700 1 -- 172.21.15.43:0/2915504206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad80106610 msgr2=0x7fad80106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:35.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.902+0000 7fad85c00700 1 --2- 172.21.15.43:0/2915504206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad80106610 0x7fad80106a20 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fad700099c0 tx=0x7fad70009cd0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.902+0000 7fad85c00700 1 -- 172.21.15.43:0/2915504206 shutdown_connections 2024-05-27T07:21:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.902+0000 7fad85c00700 1 --2- 172.21.15.43:0/2915504206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad8010b890 0x7fad801131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.902+0000 7fad85c00700 1 --2- 172.21.15.43:0/2915504206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad80106610 0x7fad80106a20 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.902+0000 7fad85c00700 1 --2- 172.21.15.43:0/2915504206 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad8010af40 0x7fad8010b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.902+0000 7fad85c00700 1 -- 172.21.15.43:0/2915504206 >> 172.21.15.43:0/2915504206 conn(0x7fad80102020 msgr2=0x7fad80104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.902+0000 7fad85c00700 1 -- 172.21.15.43:0/2915504206 shutdown_connections 2024-05-27T07:21:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.902+0000 7fad85c00700 1 -- 172.21.15.43:0/2915504206 wait complete. 2024-05-27T07:21:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.904+0000 7fad85c00700 1 Processor -- start 2024-05-27T07:21:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.904+0000 7fad85c00700 1 -- start start 2024-05-27T07:21:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.905+0000 7fad85c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad80106610 0x7fad8019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.905+0000 7fad85c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad8010af40 0x7fad8019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.905+0000 7fad85c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad8010b890 0x7fad801a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.905+0000 7fad85c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fad801163c0 con 0x7fad80106610 2024-05-27T07:21:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.905+0000 7fad85c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fad80116240 con 0x7fad8010af40 2024-05-27T07:21:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.905+0000 7fad85c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fad80116540 con 0x7fad8010b890 2024-05-27T07:21:35.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.905+0000 7fad7fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad80106610 0x7fad8019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.906+0000 7fad7fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad80106610 0x7fad8019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57518/0 (socket says 172.21.15.43:57518) 2024-05-27T07:21:35.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.906+0000 7fad7fe00700 1 -- 172.21.15.43:0/76103811 learned_addr learned my addr 172.21.15.43:0/76103811 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:35.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.906+0000 7fad77400700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad8010af40 0x7fad8019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.906+0000 7fad85200700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad8010b890 0x7fad801a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.906+0000 7fad7fe00700 1 -- 172.21.15.43:0/76103811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad8010b890 msgr2=0x7fad801a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:35.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.906+0000 7fad7fe00700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad8010b890 0x7fad801a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.906+0000 7fad7fe00700 1 -- 172.21.15.43:0/76103811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad8010af40 msgr2=0x7fad8019d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:35.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.906+0000 7fad7fe00700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad8010af40 0x7fad8019d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.906+0000 7fad7fe00700 1 -- 172.21.15.43:0/76103811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad801a1ca0 con 0x7fad80106610 2024-05-27T07:21:35.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.906+0000 7fad77400700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad8010af40 0x7fad8019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:35.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.907+0000 7fad7fe00700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad80106610 0x7fad8019cdd0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fad70000c00 tx=0x7fad7000f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.907+0000 7fad7d600700 1 -- 172.21.15.43:0/76103811 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70027070 con 0x7fad80106610 2024-05-27T07:21:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.907+0000 7fad7d600700 1 -- 172.21.15.43:0/76103811 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad7000fe50 con 0x7fad80106610 2024-05-27T07:21:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.907+0000 7fad85c00700 1 -- 172.21.15.43:0/76103811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad801a1f30 con 0x7fad80106610 2024-05-27T07:21:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.907+0000 7fad7d600700 1 -- 172.21.15.43:0/76103811 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad700173f0 con 0x7fad80106610 2024-05-27T07:21:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.907+0000 7fad85c00700 1 -- 172.21.15.43:0/76103811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad801a2410 con 0x7fad80106610 2024-05-27T07:21:35.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.911+0000 7fad7d600700 1 -- 172.21.15.43:0/76103811 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fad7002a070 con 0x7fad80106610 2024-05-27T07:21:35.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.912+0000 7fad95d41b80 1 Processor -- start 2024-05-27T07:21:35.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.912+0000 7fad95d41b80 1 -- start start 2024-05-27T07:21:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.913+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x556867471750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.913+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867471c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.913+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 0x55686747d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.913+0000 7fad95d41b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55686739ee40 con 0x5568672f1e40 2024-05-27T07:21:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.913+0000 7fad95d41b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55686738e9e0 con 0x5568673dd9f0 2024-05-27T07:21:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.913+0000 7fad95d41b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556867392a20 con 0x55686747b2c0 2024-05-27T07:21:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.913+0000 7fad7d600700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fad640b29f0 0x7fad640b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.913+0000 7fad7fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867471c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.913+0000 7fad7fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867471c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57524/0 (socket says 172.21.15.43:57524) 2024-05-27T07:21:35.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.913+0000 7fad7fe00700 1 -- 172.21.15.43:0/2882830479 learned_addr learned my addr 172.21.15.43:0/2882830479 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.914+0000 7fad85200700 1 --2- 172.21.15.43:0/2882830479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x556867471750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.914+0000 7fad77400700 1 --2- 172.21.15.43:0/2882830479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 0x55686747d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.914+0000 7fad85200700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fad640b29f0 0x7fad640b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.914+0000 7fad7d600700 1 -- 172.21.15.43:0/76103811 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7fad700e39a0 con 0x7fad80106610 2024-05-27T07:21:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.914+0000 7fad7fe00700 1 -- 172.21.15.43:0/2882830479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 msgr2=0x55686747d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.914+0000 7fad7fe00700 1 --2- 172.21.15.43:0/2882830479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 0x55686747d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.914+0000 7fad7fe00700 1 -- 172.21.15.43:0/2882830479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 msgr2=0x556867471750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.914+0000 7fad7fe00700 1 --2- 172.21.15.43:0/2882830479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x556867471750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.914+0000 7fad7fe00700 1 -- 172.21.15.43:0/2882830479 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556867357420 con 0x5568672f1e40 2024-05-27T07:21:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.915+0000 7fad7fe00700 1 --2- 172.21.15.43:0/2882830479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867471c90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fad700aa5e0 tx=0x7fad70009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623a04fa15530b44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.915+0000 7fad85200700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fad640b29f0 0x7fad640b4f00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55686746da90 tx=0x7fad78009670 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.915+0000 7fad75600700 1 -- 172.21.15.43:0/2882830479 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad700aec90 con 0x5568672f1e40 2024-05-27T07:21:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.915+0000 7fad75600700 1 -- 172.21.15.43:0/2882830479 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad700b36b0 con 0x5568672f1e40 2024-05-27T07:21:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.915+0000 7fad75600700 1 -- 172.21.15.43:0/2882830479 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70022690 con 0x5568672f1e40 2024-05-27T07:21:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.916+0000 7fad95d41b80 1 -- 172.21.15.43:0/2882830479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 msgr2=0x556867471c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.916+0000 7fad95d41b80 1 --2- 172.21.15.43:0/2882830479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867471c90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fad700aa5e0 tx=0x7fad70009310 comp rx=0 tx=0).stop 2024-05-27T07:21:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.916+0000 7fad95d41b80 1 -- 172.21.15.43:0/2882830479 shutdown_connections 2024-05-27T07:21:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.916+0000 7fad95d41b80 1 --2- 172.21.15.43:0/2882830479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 0x55686747d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.916+0000 7fad95d41b80 1 --2- 172.21.15.43:0/2882830479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x556867471750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.916+0000 7fad95d41b80 1 --2- 172.21.15.43:0/2882830479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867471c90 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.916+0000 7fad95d41b80 1 -- 172.21.15.43:0/2882830479 >> 172.21.15.43:0/2882830479 conn(0x55686738b390 msgr2=0x55686746c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.916+0000 7fad95d41b80 1 -- 172.21.15.43:0/2882830479 shutdown_connections 2024-05-27T07:21:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.917+0000 7fad95d41b80 1 -- 172.21.15.43:0/2882830479 wait complete. 2024-05-27T07:21:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.917+0000 7fad95d41b80 1 -- start start 2024-05-27T07:21:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.917+0000 7fad95d41b80 1 Processor -- start 2024-05-27T07:21:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.918+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5568672f1e40 0x556867478200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.918+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x556867478740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.918+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867478c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.918+0000 7fad95d41b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556867294fa0 con 0x55686747b2c0 2024-05-27T07:21:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.918+0000 7fad95d41b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55686732f760 con 0x5568673dd9f0 2024-05-27T07:21:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.918+0000 7fad95d41b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55686739ee40 con 0x5568672f1e40 2024-05-27T07:21:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.918+0000 7fad95d41b80 10 client.0 _ll_register_callbacks cb 0x556867338c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.918+0000 7fad85200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867478c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.919+0000 7fad85200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867478c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57536/0 (socket says 172.21.15.43:57536) 2024-05-27T07:21:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.919+0000 7fad85200700 1 -- 172.21.15.43:0/1422377770 learned_addr learned my addr 172.21.15.43:0/1422377770 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.919+0000 7fad77400700 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x556867478740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.919+0000 7fad7fe00700 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5568672f1e40 0x556867478200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.919+0000 7fad85200700 1 -- 172.21.15.43:0/1422377770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5568672f1e40 msgr2=0x556867478200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.919+0000 7fad85200700 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5568672f1e40 0x556867478200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.919+0000 7fad85200700 1 -- 172.21.15.43:0/1422377770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 msgr2=0x556867478740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.919+0000 7fad85200700 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x556867478740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.919+0000 7fad85200700 1 -- 172.21.15.43:0/1422377770 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568672de8d0 con 0x55686747b2c0 2024-05-27T07:21:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.920+0000 7fad85200700 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867478c80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fad78018c20 tx=0x7fad78004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1210db9bdd484b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.920+0000 7fad6e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.920+0000 7fad6e000700 1 -- 172.21.15.43:0/1422377770 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78004bb0 con 0x55686747b2c0 2024-05-27T07:21:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.920+0000 7fad6e000700 1 -- 172.21.15.43:0/1422377770 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7fad78012040 con 0x55686747b2c0 2024-05-27T07:21:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.920+0000 7fad95d41b80 1 -- 172.21.15.43:0/1422377770 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad700b36b0 con 0x55686747b2c0 2024-05-27T07:21:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.920+0000 7fad6e000700 1 -- 172.21.15.43:0/1422377770 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fad78031050 con 0x55686747b2c0 2024-05-27T07:21:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.920+0000 7fad95d41b80 10 client.16608 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.920+0000 7fad95d41b80 1 -- 172.21.15.43:0/1422377770 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55686735f060 con 0x55686747b2c0 2024-05-27T07:21:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.921+0000 7fad6e000700 1 -- 172.21.15.43:0/1422377770 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad700b36b0 con 0x55686747b2c0 2024-05-27T07:21:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.921+0000 7fad6e000700 1 -- 172.21.15.43:0/1422377770 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fad780052b0 con 0x55686747b2c0 2024-05-27T07:21:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.921+0000 7fad95d41b80 10 client.16608 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.921+0000 7fad95d41b80 10 client.16608 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:21:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.921+0000 7fad95d41b80 20 client.16608 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.921+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55686748ac70 0x55686748d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.921+0000 7fad95d41b80 4 client.16608 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.921+0000 7fad95d41b80 1 -- 172.21.15.43:0/1422377770 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568673a0d90 con 0x55686748ac70 2024-05-27T07:21:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.922+0000 7fad7fe00700 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55686748ac70 0x55686748d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.923+0000 7fad7fe00700 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55686748ac70 0x55686748d0e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76ffddf4548c087c server_cookie=2bbaf6114ff68411 in_seq=0 out_seq=0 2024-05-27T07:21:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.923+0000 7fad6e000700 10 client.16608 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.925+0000 7fad6e000700 1 -- 172.21.15.43:0/1422377770 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5568673a0d90 con 0x55686748ac70 2024-05-27T07:21:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.925+0000 7fad6e000700 10 client.16608 handle_command_reply: tid=0 2024-05-27T07:21:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.925+0000 7fad95d41b80 1 client.16608 shutdown 2024-05-27T07:21:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.925+0000 7fad95d41b80 1 -- 172.21.15.43:0/1422377770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 msgr2=0x556867478c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.925+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867478c80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fad78018c20 tx=0x7fad78004ad0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.926+0000 7fad95d41b80 1 -- 172.21.15.43:0/1422377770 shutdown_connections 2024-05-27T07:21:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.926+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55686748ac70 0x55686748d0e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.926+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867478c80 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.926+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5568672f1e40 0x556867478200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.926+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1422377770 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x556867478740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.926+0000 7fad95d41b80 1 -- 172.21.15.43:0/1422377770 >> 172.21.15.43:0/1422377770 conn(0x55686738b390 msgr2=0x55686746c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.926+0000 7fad6e000700 0 client.16608 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.926+0000 7fad95d41b80 1 -- 172.21.15.43:0/1422377770 shutdown_connections 2024-05-27T07:21:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.926+0000 7fad95d41b80 1 -- 172.21.15.43:0/1422377770 wait complete. 2024-05-27T07:21:35.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.926+0000 7fad95d41b80 20 client.16608 trim_cache size 0 max 16384 2024-05-27T07:21:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.944+0000 7fad95d41b80 1 Processor -- start 2024-05-27T07:21:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.944+0000 7fad95d41b80 1 -- start start 2024-05-27T07:21:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.944+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5568672f1e40 0x556867473950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.945+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x55686746c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.945+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867486b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.945+0000 7fad95d41b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fad801b3160 con 0x55686747b2c0 2024-05-27T07:21:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.945+0000 7fad95d41b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fad801b3830 con 0x5568673dd9f0 2024-05-27T07:21:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.945+0000 7fad95d41b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fad801b0f50 con 0x5568672f1e40 2024-05-27T07:21:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.945+0000 7fad85200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867486b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.945+0000 7fad85200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867486b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57548/0 (socket says 172.21.15.43:57548) 2024-05-27T07:21:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.945+0000 7fad85200700 1 -- 172.21.15.43:0/1396446210 learned_addr learned my addr 172.21.15.43:0/1396446210 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.945+0000 7fad7fe00700 1 --2- 172.21.15.43:0/1396446210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5568672f1e40 0x556867473950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.946+0000 7fad77400700 1 --2- 172.21.15.43:0/1396446210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x55686746c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.946+0000 7fad85200700 1 -- 172.21.15.43:0/1396446210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5568672f1e40 msgr2=0x556867473950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.946+0000 7fad85200700 1 --2- 172.21.15.43:0/1396446210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5568672f1e40 0x556867473950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.946+0000 7fad85200700 1 -- 172.21.15.43:0/1396446210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 msgr2=0x55686746c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.946+0000 7fad85200700 1 --2- 172.21.15.43:0/1396446210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x55686746c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.946+0000 7fad85200700 1 -- 172.21.15.43:0/1396446210 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568674724a0 con 0x55686747b2c0 2024-05-27T07:21:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.946+0000 7fad85200700 1 --2- 172.21.15.43:0/1396446210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867486b70 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fad78018bf0 tx=0x7fad78005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=825b5c916ec9462b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.946+0000 7fad74c00700 1 -- 172.21.15.43:0/1396446210 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78005c60 con 0x55686747b2c0 2024-05-27T07:21:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.946+0000 7fad74c00700 1 -- 172.21.15.43:0/1396446210 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad78005e00 con 0x55686747b2c0 2024-05-27T07:21:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.946+0000 7fad74c00700 1 -- 172.21.15.43:0/1396446210 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78003c60 con 0x55686747b2c0 2024-05-27T07:21:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.947+0000 7fad95d41b80 1 -- 172.21.15.43:0/1396446210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 msgr2=0x556867486b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.947+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1396446210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867486b70 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fad78018bf0 tx=0x7fad78005b20 comp rx=0 tx=0).stop 2024-05-27T07:21:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.947+0000 7fad95d41b80 1 -- 172.21.15.43:0/1396446210 shutdown_connections 2024-05-27T07:21:35.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.947+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1396446210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55686747b2c0 0x556867486b70 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.947+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1396446210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5568672f1e40 0x556867473950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.947+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1396446210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x55686746c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.947+0000 7fad95d41b80 1 -- 172.21.15.43:0/1396446210 >> 172.21.15.43:0/1396446210 conn(0x55686738b390 msgr2=0x556867473240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:35.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.948+0000 7fad95d41b80 1 -- 172.21.15.43:0/1396446210 shutdown_connections 2024-05-27T07:21:35.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.948+0000 7fad95d41b80 1 -- 172.21.15.43:0/1396446210 wait complete. 2024-05-27T07:21:35.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.948+0000 7fad95d41b80 1 -- start start 2024-05-27T07:21:35.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.948+0000 7fad95d41b80 1 Processor -- start 2024-05-27T07:21:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.949+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867483420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.949+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x55686747dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.949+0000 7fad95d41b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 0x55686747e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.949+0000 7fad95d41b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fad801b1620 con 0x5568672f1e40 2024-05-27T07:21:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.949+0000 7fad95d41b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fad801b2a90 con 0x5568673dd9f0 2024-05-27T07:21:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.949+0000 7fad95d41b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fad801b1cf0 con 0x55686747b2c0 2024-05-27T07:21:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.949+0000 7fad95d41b80 10 client.0 _ll_register_callbacks cb 0x556867338c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.949+0000 7fad7fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867483420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.949+0000 7fad7fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867483420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57558/0 (socket says 172.21.15.43:57558) 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.949+0000 7fad7fe00700 1 -- 172.21.15.43:0/1839552795 learned_addr learned my addr 172.21.15.43:0/1839552795 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.950+0000 7fad77400700 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x55686747dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.950+0000 7fad85200700 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 0x55686747e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.950+0000 7fad7fe00700 1 -- 172.21.15.43:0/1839552795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 msgr2=0x55686747e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.950+0000 7fad7fe00700 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 0x55686747e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.950+0000 7fad7fe00700 1 -- 172.21.15.43:0/1839552795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 msgr2=0x55686747dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.950+0000 7fad7fe00700 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x55686747dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.950+0000 7fad7fe00700 1 -- 172.21.15.43:0/1839552795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55686747e910 con 0x5568672f1e40 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.950+0000 7fad85200700 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 0x55686747e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.950+0000 7fad7fe00700 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867483420 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x55686746d970 tx=0x7fad700b3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc216e341d48b9b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.950+0000 7fad6e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.951+0000 7fad6e000700 1 -- 172.21.15.43:0/1839552795 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad700bf060 con 0x5568672f1e40 2024-05-27T07:21:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.951+0000 7fad6e000700 1 -- 172.21.15.43:0/1839552795 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7fad700a6ed0 con 0x5568672f1e40 2024-05-27T07:21:35.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.951+0000 7fad95d41b80 1 -- 172.21.15.43:0/1839552795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad78005e00 con 0x5568672f1e40 2024-05-27T07:21:35.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.951+0000 7fad6e000700 1 -- 172.21.15.43:0/1839552795 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fad700aa230 con 0x5568672f1e40 2024-05-27T07:21:35.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.951+0000 7fad95d41b80 10 client.16614 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:35.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.951+0000 7fad95d41b80 1 -- 172.21.15.43:0/1839552795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x5568674f4600 con 0x5568672f1e40 2024-05-27T07:21:35.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.951+0000 7fad6e000700 1 -- 172.21.15.43:0/1839552795 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fad78005e00 con 0x5568672f1e40 2024-05-27T07:21:35.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.951+0000 7fad6e000700 1 -- 172.21.15.43:0/1839552795 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fad70070170 con 0x5568672f1e40 2024-05-27T07:21:35.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.952+0000 7fad95d41b80 10 client.16614 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:35.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.952+0000 7fad95d41b80 10 client.16614 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:21:35.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.952+0000 7fad95d41b80 20 client.16614 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:35.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.952+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5568674f4bb0 0x5568674f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:35.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.952+0000 7fad95d41b80 4 client.16614 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:21:35.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.952+0000 7fad95d41b80 1 -- 172.21.15.43:0/1839552795 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556867472160 con 0x5568674f4bb0 2024-05-27T07:21:35.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.952+0000 7fad77400700 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5568674f4bb0 0x5568674f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:35.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.954+0000 7fad77400700 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5568674f4bb0 0x5568674f6fd0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e10f84cdff78dc0 server_cookie=b6b7a6234b1cf960 in_seq=0 out_seq=0 2024-05-27T07:21:35.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.954+0000 7fad6e000700 10 client.16614 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.954+0000 7fad6e000700 1 -- 172.21.15.43:0/1839552795 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556867472160 con 0x5568674f4bb0 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.954+0000 7fad6e000700 10 client.16614 handle_command_reply: tid=0 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.954+0000 7fad95d41b80 1 client.16614 shutdown 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 1 -- 172.21.15.43:0/1839552795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 msgr2=0x556867483420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867483420 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x55686746d970 tx=0x7fad700b3040 comp rx=0 tx=0).stop 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 1 -- 172.21.15.43:0/1839552795 shutdown_connections 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55686747b2c0 0x55686747e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5568674f4bb0 0x5568674f6fd0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5568672f1e40 0x556867483420 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 1 --2- 172.21.15.43:0/1839552795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5568673dd9f0 0x55686747dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 1 -- 172.21.15.43:0/1839552795 >> 172.21.15.43:0/1839552795 conn(0x55686738b390 msgr2=0x556867470120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad6e000700 0 client.16614 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 1 -- 172.21.15.43:0/1839552795 shutdown_connections 2024-05-27T07:21:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 1 -- 172.21.15.43:0/1839552795 wait complete. 2024-05-27T07:21:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.955+0000 7fad95d41b80 20 client.16614 trim_cache size 0 max 16384 2024-05-27T07:21:35.947 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 -- 172.21.15.43:0/76103811 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fad640b29f0 msgr2=0x7fad640b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fad640b29f0 0x7fad640b4f00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55686746da90 tx=0x7fad78009670 comp rx=0 tx=0).stop 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 -- 172.21.15.43:0/76103811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad80106610 msgr2=0x7fad8019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad80106610 0x7fad8019cdd0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fad70000c00 tx=0x7fad7000f510 comp rx=0 tx=0).stop 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 -- 172.21.15.43:0/76103811 shutdown_connections 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fad640b29f0 0x7fad640b4f00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad8010b890 0x7fad801a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad80106610 0x7fad8019cdd0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 --2- 172.21.15.43:0/76103811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad8010af40 0x7fad8019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 -- 172.21.15.43:0/76103811 >> 172.21.15.43:0/76103811 conn(0x7fad80102020 msgr2=0x7fad8010c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 -- 172.21.15.43:0/76103811 shutdown_connections 2024-05-27T07:21:35.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:35.956+0000 7fad85c00700 1 -- 172.21.15.43:0/76103811 wait complete. 2024-05-27T07:21:36.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 40516 0.63 MB/sec execute 220 sec latency 61.421 ms 2024-05-27T07:21:36.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:36 smithi132 ceph-mon[35283]: pgmap v255: 129 pgs: 129 active+clean; 69 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 394 KiB/s wr, 42 op/s 2024-05-27T07:21:36.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:36 smithi043 ceph-mon[31531]: pgmap v255: 129 pgs: 129 active+clean; 69 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 394 KiB/s wr, 42 op/s 2024-05-27T07:21:36.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:36 smithi179 ceph-mon[35888]: pgmap v255: 129 pgs: 129 active+clean; 69 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 394 KiB/s wr, 42 op/s 2024-05-27T07:21:37.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 40615 0.63 MB/sec execute 221 sec latency 68.678 ms 2024-05-27T07:21:37.953 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:21:37.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:21:38.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 40715 0.63 MB/sec execute 222 sec latency 65.057 ms 2024-05-27T07:21:38.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:38 smithi043 ceph-mon[31531]: pgmap v256: 129 pgs: 129 active+clean; 68 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 586 KiB/s wr, 67 op/s 2024-05-27T07:21:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:38.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:38 smithi132 ceph-mon[35283]: pgmap v256: 129 pgs: 129 active+clean; 68 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 586 KiB/s wr, 67 op/s 2024-05-27T07:21:38.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:38 smithi179 ceph-mon[35888]: pgmap v256: 129 pgs: 129 active+clean; 68 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 586 KiB/s wr, 67 op/s 2024-05-27T07:21:39.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 40868 0.62 MB/sec execute 223 sec latency 73.061 ms 2024-05-27T07:21:40.208 INFO:tasks.workunit.client.0.smithi043.stdout: 1 40972 0.62 MB/sec execute 224 sec latency 57.879 ms 2024-05-27T07:21:40.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:40 smithi043 ceph-mon[31531]: pgmap v257: 129 pgs: 129 active+clean; 68 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 341 KiB/s wr, 47 op/s 2024-05-27T07:21:40.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:40 smithi132 ceph-mon[35283]: pgmap v257: 129 pgs: 129 active+clean; 68 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 341 KiB/s wr, 47 op/s 2024-05-27T07:21:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:40 smithi179 ceph-mon[35888]: pgmap v257: 129 pgs: 129 active+clean; 68 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 341 KiB/s wr, 47 op/s 2024-05-27T07:21:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.958+0000 7effad600700 1 -- 172.21.15.43:0/3423020856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa81030f0 msgr2=0x7effa8103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.958+0000 7effad600700 1 --2- 172.21.15.43:0/3423020856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa81030f0 0x7effa8103500 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7eff98009990 tx=0x7eff98009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.958+0000 7effad600700 1 -- 172.21.15.43:0/3423020856 shutdown_connections 2024-05-27T07:21:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.958+0000 7effad600700 1 --2- 172.21.15.43:0/3423020856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa8110ca0 0x7effa81130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.958+0000 7effad600700 1 --2- 172.21.15.43:0/3423020856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa81030f0 0x7effa8103500 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.958+0000 7effad600700 1 --2- 172.21.15.43:0/3423020856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7effa8103a40 0x7effa8110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.958+0000 7effad600700 1 -- 172.21.15.43:0/3423020856 >> 172.21.15.43:0/3423020856 conn(0x7effa80fedb0 msgr2=0x7effa81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:40.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.958+0000 7effad600700 1 -- 172.21.15.43:0/3423020856 shutdown_connections 2024-05-27T07:21:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.958+0000 7effad600700 1 -- 172.21.15.43:0/3423020856 wait complete. 2024-05-27T07:21:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.959+0000 7effad600700 1 Processor -- start 2024-05-27T07:21:40.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.959+0000 7effad600700 1 -- start start 2024-05-27T07:21:40.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effad600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7effa81030f0 0x7effa819cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effad600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8103a40 0x7effa819d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effad600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa8110ca0 0x7effa81a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effad600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7effa8116500 con 0x7effa8103a40 2024-05-27T07:21:40.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effad600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7effa8116380 con 0x7effa81030f0 2024-05-27T07:21:40.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effad600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7effa8116680 con 0x7effa8110ca0 2024-05-27T07:21:40.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effa6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8103a40 0x7effa819d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effa6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8103a40 0x7effa819d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57580/0 (socket says 172.21.15.43:57580) 2024-05-27T07:21:40.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effa6000700 1 -- 172.21.15.43:0/2847537662 learned_addr learned my addr 172.21.15.43:0/2847537662 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:40.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effa6a00700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7effa81030f0 0x7effa819cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.960+0000 7effa7400700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa8110ca0 0x7effa81a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.961+0000 7effa6000700 1 -- 172.21.15.43:0/2847537662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa8110ca0 msgr2=0x7effa81a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:40.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.961+0000 7effa6000700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa8110ca0 0x7effa81a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.961+0000 7effa6000700 1 -- 172.21.15.43:0/2847537662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7effa81030f0 msgr2=0x7effa819cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.961+0000 7effa6000700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7effa81030f0 0x7effa819cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.961+0000 7effa6000700 1 -- 172.21.15.43:0/2847537662 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa81a1d70 con 0x7effa8103a40 2024-05-27T07:21:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.961+0000 7effa6000700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8103a40 0x7effa819d4f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7eff9400d980 tx=0x7eff9400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.962+0000 7eff9f400700 1 -- 172.21.15.43:0/2847537662 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94017030 con 0x7effa8103a40 2024-05-27T07:21:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.962+0000 7eff9f400700 1 -- 172.21.15.43:0/2847537662 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7eff940044b0 con 0x7effa8103a40 2024-05-27T07:21:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.962+0000 7effad600700 1 -- 172.21.15.43:0/2847537662 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effa81a2060 con 0x7effa8103a40 2024-05-27T07:21:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.962+0000 7eff9f400700 1 -- 172.21.15.43:0/2847537662 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff940038e0 con 0x7effa8103a40 2024-05-27T07:21:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.962+0000 7effad600700 1 -- 172.21.15.43:0/2847537662 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effa81a9a30 con 0x7effa8103a40 2024-05-27T07:21:40.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.968+0000 7effbc15cb80 1 Processor -- start 2024-05-27T07:21:40.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.968+0000 7eff9f400700 1 -- 172.21.15.43:0/2847537662 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7eff9401a070 con 0x7effa8103a40 2024-05-27T07:21:40.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.968+0000 7effbc15cb80 1 -- start start 2024-05-27T07:21:40.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.968+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 0x55afb18383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.968+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb16b9ec0 0x55afb18388f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.968+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb1845190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.968+0000 7effbc15cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55afb1766ec0 con 0x55afb1838ef0 2024-05-27T07:21:40.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.968+0000 7effbc15cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55afb1756a60 con 0x55afb16b9ec0 2024-05-27T07:21:40.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.968+0000 7effbc15cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55afb175aaa0 con 0x55afb17a5a70 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.969+0000 7effa6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb1845190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.969+0000 7effa7400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 0x55afb18383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.969+0000 7effa7400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 0x55afb18383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:47206/0 (socket says 172.21.15.43:47206) 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.969+0000 7effa7400700 1 -- 172.21.15.43:0/962480008 learned_addr learned my addr 172.21.15.43:0/962480008 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.969+0000 7effa6a00700 1 --2- 172.21.15.43:0/962480008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb16b9ec0 0x55afb18388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.970+0000 7eff9f400700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7eff8c0b2a70 0x7eff8c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.970+0000 7effa6000700 1 -- 172.21.15.43:0/962480008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 msgr2=0x55afb18383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.970+0000 7effa6000700 1 --2- 172.21.15.43:0/962480008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 0x55afb18383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.970+0000 7effa6000700 1 -- 172.21.15.43:0/962480008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb16b9ec0 msgr2=0x55afb18388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.970+0000 7effa6000700 1 --2- 172.21.15.43:0/962480008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb16b9ec0 0x55afb18388f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.970+0000 7effa6000700 1 -- 172.21.15.43:0/962480008 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afb171f4a0 con 0x55afb1838ef0 2024-05-27T07:21:40.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.970+0000 7eff9f400700 1 -- 172.21.15.43:0/2847537662 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7eff940e2290 con 0x7effa8103a40 2024-05-27T07:21:40.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.970+0000 7effa7400700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7eff8c0b2a70 0x7eff8c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.970+0000 7effa6000700 1 --2- 172.21.15.43:0/962480008 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb1845190 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7eff940df5b0 tx=0x7eff940729e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6fda38c9005eb30 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:40.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.971+0000 7eff93e00700 1 -- 172.21.15.43:0/962480008 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff940a8a90 con 0x55afb1838ef0 2024-05-27T07:21:40.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.971+0000 7eff93e00700 1 -- 172.21.15.43:0/962480008 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7eff940a8c50 con 0x55afb1838ef0 2024-05-27T07:21:40.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.971+0000 7eff93e00700 1 -- 172.21.15.43:0/962480008 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff940ae680 con 0x55afb1838ef0 2024-05-27T07:21:40.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.972+0000 7effbc15cb80 1 -- 172.21.15.43:0/962480008 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 msgr2=0x55afb1845190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:40.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.972+0000 7effbc15cb80 1 --2- 172.21.15.43:0/962480008 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb1845190 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7eff940df5b0 tx=0x7eff940729e0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.972+0000 7effa7400700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7eff8c0b2a70 0x7eff8c0b4f80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55afb183b070 tx=0x7effa0009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:40.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.972+0000 7effbc15cb80 1 -- 172.21.15.43:0/962480008 shutdown_connections 2024-05-27T07:21:40.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.972+0000 7effbc15cb80 1 --2- 172.21.15.43:0/962480008 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb1845190 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.972+0000 7effbc15cb80 1 --2- 172.21.15.43:0/962480008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 0x55afb18383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.972+0000 7effbc15cb80 1 --2- 172.21.15.43:0/962480008 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb16b9ec0 0x55afb18388f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.972+0000 7effbc15cb80 1 -- 172.21.15.43:0/962480008 >> 172.21.15.43:0/962480008 conn(0x55afb1753410 msgr2=0x55afb1834540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:40.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.972+0000 7effbc15cb80 1 -- 172.21.15.43:0/962480008 shutdown_connections 2024-05-27T07:21:40.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.973+0000 7effbc15cb80 1 -- 172.21.15.43:0/962480008 wait complete. 2024-05-27T07:21:40.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.973+0000 7effbc15cb80 1 -- start start 2024-05-27T07:21:40.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.973+0000 7effbc15cb80 1 Processor -- start 2024-05-27T07:21:40.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.974+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 0x55afb1842630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.974+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1842b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.974+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18430b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.974+0000 7effbc15cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55afb165d020 con 0x55afb1838ef0 2024-05-27T07:21:40.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.974+0000 7effbc15cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55afb16f77e0 con 0x55afb17a5a70 2024-05-27T07:21:40.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.974+0000 7effbc15cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55afb1845ba0 con 0x55afb16b9ec0 2024-05-27T07:21:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.974+0000 7effbc15cb80 10 client.0 _ll_register_callbacks cb 0x55afb1700cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa6000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1842b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18430b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57594/0 (socket says 172.21.15.43:57594) 2024-05-27T07:21:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa7400700 1 -- 172.21.15.43:0/1612052118 learned_addr learned my addr 172.21.15.43:0/1612052118 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa6000700 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1842b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41180/0 (socket says 172.21.15.43:41180) 2024-05-27T07:21:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa6a00700 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 0x55afb1842630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa6000700 1 -- 172.21.15.43:0/1612052118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 msgr2=0x55afb1842630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa6000700 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 0x55afb1842630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa6000700 1 -- 172.21.15.43:0/1612052118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 msgr2=0x55afb18430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa6000700 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18430b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.975+0000 7effa6000700 1 -- 172.21.15.43:0/1612052118 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afb16a6950 con 0x55afb17a5a70 2024-05-27T07:21:40.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.976+0000 7effa6000700 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1842b70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7eff940a63b0 tx=0x7eff9400fc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f371f885b121f2f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:40.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.976+0000 7eff90c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:21:40.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.976+0000 7eff90c00700 1 -- 172.21.15.43:0/1612052118 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff940a88b0 con 0x55afb17a5a70 2024-05-27T07:21:40.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.977+0000 7effbc15cb80 1 -- 172.21.15.43:0/1612052118 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff940a8c50 con 0x55afb17a5a70 2024-05-27T07:21:40.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.977+0000 7eff90c00700 1 -- 172.21.15.43:0/1612052118 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7eff940ae910 con 0x55afb17a5a70 2024-05-27T07:21:40.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.977+0000 7eff90c00700 1 -- 172.21.15.43:0/1612052118 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7eff940a8a90 con 0x55afb17a5a70 2024-05-27T07:21:40.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.977+0000 7effbc15cb80 10 client.36167 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:40.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.977+0000 7effbc15cb80 1 -- 172.21.15.43:0/1612052118 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55afb17270e0 con 0x55afb17a5a70 2024-05-27T07:21:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.978+0000 7eff90c00700 1 -- 172.21.15.43:0/1612052118 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7eff940a8c50 con 0x55afb17a5a70 2024-05-27T07:21:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.978+0000 7eff90c00700 1 -- 172.21.15.43:0/1612052118 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7eff94024060 con 0x55afb17a5a70 2024-05-27T07:21:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.978+0000 7effbc15cb80 10 client.36167 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:40.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.978+0000 7effbc15cb80 10 client.36167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:21:40.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.978+0000 7effbc15cb80 20 client.36167 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:40.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.978+0000 7effbc15cb80 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55afb18528f0 0x55afb1854d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.978+0000 7effbc15cb80 4 client.36167 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:40.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.978+0000 7effbc15cb80 1 -- 172.21.15.43:0/1612052118 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afb1768e10 con 0x55afb18528f0 2024-05-27T07:21:40.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.979+0000 7effa6a00700 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55afb18528f0 0x55afb1854d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.980+0000 7effa6a00700 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55afb18528f0 0x55afb1854d10 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86d5afd652c5013 server_cookie=659acf6316e14c15 in_seq=0 out_seq=0 2024-05-27T07:21:40.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.980+0000 7eff90c00700 10 client.36167 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:40.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7eff90c00700 1 -- 172.21.15.43:0/1612052118 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55afb1768e10 con 0x55afb18528f0 2024-05-27T07:21:40.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7eff90c00700 10 client.36167 handle_command_reply: tid=0 2024-05-27T07:21:40.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7effbc15cb80 1 client.36167 shutdown 2024-05-27T07:21:40.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7effbc15cb80 1 -- 172.21.15.43:0/1612052118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 msgr2=0x55afb1842b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:40.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7effbc15cb80 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1842b70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7eff940a63b0 tx=0x7eff9400fc10 comp rx=0 tx=0).stop 2024-05-27T07:21:40.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7effbc15cb80 1 -- 172.21.15.43:0/1612052118 shutdown_connections 2024-05-27T07:21:40.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7effbc15cb80 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55afb18528f0 0x55afb1854d10 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7effbc15cb80 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7effbc15cb80 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 0x55afb1842630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7effbc15cb80 1 --2- 172.21.15.43:0/1612052118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1842b70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:40.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7effbc15cb80 1 -- 172.21.15.43:0/1612052118 >> 172.21.15.43:0/1612052118 conn(0x55afb1753410 msgr2=0x55afb1838390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:40.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.983+0000 7eff90c00700 0 client.36167 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:40.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.984+0000 7effbc15cb80 1 -- 172.21.15.43:0/1612052118 shutdown_connections 2024-05-27T07:21:40.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.984+0000 7effbc15cb80 1 -- 172.21.15.43:0/1612052118 wait complete. 2024-05-27T07:21:40.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:40.984+0000 7effbc15cb80 20 client.36167 trim_cache size 0 max 16384 2024-05-27T07:21:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.007+0000 7effbc15cb80 1 Processor -- start 2024-05-27T07:21:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.007+0000 7effbc15cb80 1 -- start start 2024-05-27T07:21:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.007+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb16b9ec0 0x55afb184ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.008+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 0x55afb183de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.008+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb183a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.008+0000 7effbc15cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7effa81b3310 con 0x55afb1838ef0 2024-05-27T07:21:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.008+0000 7effbc15cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7effa81b39e0 con 0x55afb16b9ec0 2024-05-27T07:21:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.008+0000 7effbc15cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7effa81b1100 con 0x55afb17a5a70 2024-05-27T07:21:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.008+0000 7effa7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb183a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.008+0000 7effa7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb183a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57598/0 (socket says 172.21.15.43:57598) 2024-05-27T07:21:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.008+0000 7effa7400700 1 -- 172.21.15.43:0/940815195 learned_addr learned my addr 172.21.15.43:0/940815195 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.008+0000 7effa6000700 1 --2- 172.21.15.43:0/940815195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 0x55afb183de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.008+0000 7effa6a00700 1 --2- 172.21.15.43:0/940815195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb16b9ec0 0x55afb184ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.009+0000 7effa7400700 1 -- 172.21.15.43:0/940815195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 msgr2=0x55afb183de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.009+0000 7effa7400700 1 --2- 172.21.15.43:0/940815195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 0x55afb183de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.009+0000 7effa7400700 1 -- 172.21.15.43:0/940815195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb16b9ec0 msgr2=0x55afb184ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.009+0000 7effa7400700 1 --2- 172.21.15.43:0/940815195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb16b9ec0 0x55afb184ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.009+0000 7effa7400700 1 -- 172.21.15.43:0/940815195 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afb18302c0 con 0x55afb1838ef0 2024-05-27T07:21:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.009+0000 7effa7400700 1 --2- 172.21.15.43:0/940815195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb183a980 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7effa0004a40 tx=0x7effa0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2971715607ca72ab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.010+0000 7eff93400700 1 -- 172.21.15.43:0/940815195 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa0027070 con 0x55afb1838ef0 2024-05-27T07:21:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.010+0000 7eff93400700 1 -- 172.21.15.43:0/940815195 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effa0015c00 con 0x55afb1838ef0 2024-05-27T07:21:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.010+0000 7eff93400700 1 -- 172.21.15.43:0/940815195 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa0023c00 con 0x55afb1838ef0 2024-05-27T07:21:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.011+0000 7effbc15cb80 1 -- 172.21.15.43:0/940815195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 msgr2=0x55afb183a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.011+0000 7effbc15cb80 1 --2- 172.21.15.43:0/940815195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb183a980 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7effa0004a40 tx=0x7effa0015000 comp rx=0 tx=0).stop 2024-05-27T07:21:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.011+0000 7effbc15cb80 1 -- 172.21.15.43:0/940815195 shutdown_connections 2024-05-27T07:21:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.011+0000 7effbc15cb80 1 --2- 172.21.15.43:0/940815195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb183a980 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.011+0000 7effbc15cb80 1 --2- 172.21.15.43:0/940815195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb16b9ec0 0x55afb184ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.011+0000 7effbc15cb80 1 --2- 172.21.15.43:0/940815195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb17a5a70 0x55afb183de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.011+0000 7effbc15cb80 1 -- 172.21.15.43:0/940815195 >> 172.21.15.43:0/940815195 conn(0x55afb1753410 msgr2=0x55afb1841590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.011+0000 7effbc15cb80 1 -- 172.21.15.43:0/940815195 shutdown_connections 2024-05-27T07:21:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.011+0000 7effbc15cb80 1 -- 172.21.15.43:0/940815195 wait complete. 2024-05-27T07:21:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.011+0000 7effbc15cb80 1 -- start start 2024-05-27T07:21:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.012+0000 7effbc15cb80 1 Processor -- start 2024-05-27T07:21:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.012+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 0x55afb1853a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.012+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1853f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.012+0000 7effbc15cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18bc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.012+0000 7effbc15cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7effa81b17d0 con 0x55afb1838ef0 2024-05-27T07:21:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.012+0000 7effbc15cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7effa81b2c40 con 0x55afb17a5a70 2024-05-27T07:21:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.012+0000 7effbc15cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7effa81b1ea0 con 0x55afb16b9ec0 2024-05-27T07:21:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.012+0000 7effbc15cb80 10 client.0 _ll_register_callbacks cb 0x55afb1700cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18bc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18bc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57608/0 (socket says 172.21.15.43:57608) 2024-05-27T07:21:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa7400700 1 -- 172.21.15.43:0/22175095 learned_addr learned my addr 172.21.15.43:0/22175095 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa6a00700 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 0x55afb1853a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa6000700 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1853f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa7400700 1 -- 172.21.15.43:0/22175095 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 msgr2=0x55afb1853a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa7400700 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 0x55afb1853a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa7400700 1 -- 172.21.15.43:0/22175095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 msgr2=0x55afb1853f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa7400700 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1853f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa7400700 1 -- 172.21.15.43:0/22175095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afb1843420 con 0x55afb1838ef0 2024-05-27T07:21:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa6a00700 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 0x55afb1853a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.013+0000 7effa6000700 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1853f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:21:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.014+0000 7effa7400700 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18bc150 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7effa0016980 tx=0x7effa0005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3041aae90d715726 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.014+0000 7eff90c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.014+0000 7eff90c00700 1 -- 172.21.15.43:0/22175095 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa003a060 con 0x55afb1838ef0 2024-05-27T07:21:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.014+0000 7effbc15cb80 1 -- 172.21.15.43:0/22175095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effa0015c00 con 0x55afb1838ef0 2024-05-27T07:21:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.014+0000 7eff90c00700 1 -- 172.21.15.43:0/22175095 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7effa0023820 con 0x55afb1838ef0 2024-05-27T07:21:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.014+0000 7eff90c00700 1 -- 172.21.15.43:0/22175095 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7effa0041050 con 0x55afb1838ef0 2024-05-27T07:21:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.014+0000 7effbc15cb80 10 client.16632 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:41.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.014+0000 7effbc15cb80 1 -- 172.21.15.43:0/22175095 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x55afb18bd570 con 0x55afb1838ef0 2024-05-27T07:21:41.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.015+0000 7eff90c00700 1 -- 172.21.15.43:0/22175095 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effa0015c00 con 0x55afb1838ef0 2024-05-27T07:21:41.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.015+0000 7eff90c00700 1 -- 172.21.15.43:0/22175095 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7effa003da60 con 0x55afb1838ef0 2024-05-27T07:21:41.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.015+0000 7effbc15cb80 10 client.16632 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:41.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.015+0000 7effbc15cb80 10 client.16632 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:21:41.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.015+0000 7effbc15cb80 20 client.16632 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:41.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.015+0000 7effbc15cb80 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55afb18bdba0 0x55afb18bffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:41.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.015+0000 7effbc15cb80 4 client.16632 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:21:41.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.015+0000 7effbc15cb80 1 -- 172.21.15.43:0/22175095 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55afb17625f0 con 0x55afb18bdba0 2024-05-27T07:21:41.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.016+0000 7effa6a00700 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55afb18bdba0 0x55afb18bffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:41.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.017+0000 7effa6a00700 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55afb18bdba0 0x55afb18bffc0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0b7c1d9fd534910 server_cookie=c6c75cb97a17db8 in_seq=0 out_seq=0 2024-05-27T07:21:41.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.017+0000 7eff90c00700 10 client.16632 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:41.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.021+0000 7eff90c00700 1 -- 172.21.15.43:0/22175095 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55afb17625f0 con 0x55afb18bdba0 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.021+0000 7eff90c00700 10 client.16632 handle_command_reply: tid=0 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.021+0000 7effbc15cb80 1 client.16632 shutdown 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 1 -- 172.21.15.43:0/22175095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 msgr2=0x55afb18bc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18bc150 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7effa0016980 tx=0x7effa0005b90 comp rx=0 tx=0).stop 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 1 -- 172.21.15.43:0/22175095 shutdown_connections 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55afb18bdba0 0x55afb18bffc0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afb1838ef0 0x55afb18bc150 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55afb16b9ec0 0x55afb1853a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 1 --2- 172.21.15.43:0/22175095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55afb17a5a70 0x55afb1853f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7eff90c00700 0 client.16632 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 1 -- 172.21.15.43:0/22175095 >> 172.21.15.43:0/22175095 conn(0x55afb1753410 msgr2=0x55afb1839f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 1 -- 172.21.15.43:0/22175095 shutdown_connections 2024-05-27T07:21:41.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 1 -- 172.21.15.43:0/22175095 wait complete. 2024-05-27T07:21:41.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.022+0000 7effbc15cb80 20 client.16632 trim_cache size 0 max 16384 2024-05-27T07:21:41.014 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:21:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:21:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "dd418cd3-0880-4c2b-a731-d0c0e5d4d7e5", 2024-05-27T07:21:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:21:41.014 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:21:41.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.023+0000 7effad600700 1 -- 172.21.15.43:0/2847537662 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7eff8c0b2a70 msgr2=0x7eff8c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:41.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.023+0000 7effad600700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7eff8c0b2a70 0x7eff8c0b4f80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55afb183b070 tx=0x7effa0009040 comp rx=0 tx=0).stop 2024-05-27T07:21:41.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.023+0000 7effad600700 1 -- 172.21.15.43:0/2847537662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8103a40 msgr2=0x7effa819d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:41.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.023+0000 7effad600700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8103a40 0x7effa819d4f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7eff9400d980 tx=0x7eff9400dc90 comp rx=0 tx=0).stop 2024-05-27T07:21:41.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.023+0000 7effad600700 1 -- 172.21.15.43:0/2847537662 shutdown_connections 2024-05-27T07:21:41.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.023+0000 7effad600700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7eff8c0b2a70 0x7eff8c0b4f80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.023+0000 7effad600700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa8110ca0 0x7effa81a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.023+0000 7effad600700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7effa81030f0 0x7effa819cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.023+0000 7effad600700 1 --2- 172.21.15.43:0/2847537662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7effa8103a40 0x7effa819d4f0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:41.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.023+0000 7effad600700 1 -- 172.21.15.43:0/2847537662 >> 172.21.15.43:0/2847537662 conn(0x7effa80fedb0 msgr2=0x7effa8101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:41.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.024+0000 7effad600700 1 -- 172.21.15.43:0/2847537662 shutdown_connections 2024-05-27T07:21:41.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:41.024+0000 7effad600700 1 -- 172.21.15.43:0/2847537662 wait complete. 2024-05-27T07:21:41.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 41123 0.62 MB/sec execute 225 sec latency 65.823 ms 2024-05-27T07:21:41.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:21:42.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 41257 0.62 MB/sec execute 226 sec latency 70.438 ms 2024-05-27T07:21:42.364 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:21:42.512 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: pgmap v258: 129 pgs: 129 active+clean; 68 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 412 KiB/s wr, 66 op/s 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:21:42.513 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:21:42.514 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.514 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.514 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.514 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.520 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: pgmap v258: 129 pgs: 129 active+clean; 68 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 412 KiB/s wr, 66 op/s 2024-05-27T07:21:42.520 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:21:42.520 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.520 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: pgmap v258: 129 pgs: 129 active+clean; 68 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 412 KiB/s wr, 66 op/s 2024-05-27T07:21:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:21:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:21:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:21:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:21:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:21:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:21:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:21:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:21:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:42.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:21:43.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 41403 0.62 MB/sec execute 227 sec latency 125.392 ms 2024-05-27T07:21:44.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 41488 0.62 MB/sec execute 228 sec latency 53.064 ms 2024-05-27T07:21:44.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:44 smithi043 ceph-mon[31531]: pgmap v259: 129 pgs: 129 active+clean; 68 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 264 KiB/s wr, 43 op/s 2024-05-27T07:21:44.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:44 smithi132 ceph-mon[35283]: pgmap v259: 129 pgs: 129 active+clean; 68 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 264 KiB/s wr, 43 op/s 2024-05-27T07:21:44.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:44 smithi179 ceph-mon[35888]: pgmap v259: 129 pgs: 129 active+clean; 68 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 264 KiB/s wr, 43 op/s 2024-05-27T07:21:44.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.922+0000 7fc682000700 1 -- 172.21.15.43:0/3597020358 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc67c105280 msgr2=0x7fc67c105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.922+0000 7fc682000700 1 --2- 172.21.15.43:0/3597020358 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc67c105280 0x7fc67c105690 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc670009990 tx=0x7fc670009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.922+0000 7fc682000700 1 -- 172.21.15.43:0/3597020358 shutdown_connections 2024-05-27T07:21:44.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.922+0000 7fc682000700 1 --2- 172.21.15.43:0/3597020358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c10e6f0 0x7fc67c110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.922+0000 7fc682000700 1 --2- 172.21.15.43:0/3597020358 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc67c105280 0x7fc67c105690 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.922+0000 7fc682000700 1 --2- 172.21.15.43:0/3597020358 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc67c105bd0 0x7fc67c10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.922+0000 7fc682000700 1 -- 172.21.15.43:0/3597020358 >> 172.21.15.43:0/3597020358 conn(0x7fc67c078ab0 msgr2=0x7fc67c078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:44.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.922+0000 7fc682000700 1 -- 172.21.15.43:0/3597020358 shutdown_connections 2024-05-27T07:21:44.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.922+0000 7fc682000700 1 -- 172.21.15.43:0/3597020358 wait complete. 2024-05-27T07:21:44.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.923+0000 7fc682000700 1 Processor -- start 2024-05-27T07:21:44.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.924+0000 7fc682000700 1 -- start start 2024-05-27T07:21:44.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.924+0000 7fc682000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc67c105280 0x7fc67c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.924+0000 7fc682000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c105bd0 0x7fc67c19d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.924+0000 7fc682000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc67c10e6f0 0x7fc67c1a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.924+0000 7fc682000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1140f0 con 0x7fc67c10e6f0 2024-05-27T07:21:44.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.924+0000 7fc682000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c113f70 con 0x7fc67c105bd0 2024-05-27T07:21:44.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.924+0000 7fc682000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c114270 con 0x7fc67c105280 2024-05-27T07:21:44.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.925+0000 7fc681600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc67c10e6f0 0x7fc67c1a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.925+0000 7fc681600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc67c10e6f0 0x7fc67c1a1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48292/0 (socket says 172.21.15.43:48292) 2024-05-27T07:21:44.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.925+0000 7fc681600700 1 -- 172.21.15.43:0/2608342091 learned_addr learned my addr 172.21.15.43:0/2608342091 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:44.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.925+0000 7fc680c00700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc67c105280 0x7fc67c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.925+0000 7fc67be00700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c105bd0 0x7fc67c19d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.925+0000 7fc681600700 1 -- 172.21.15.43:0/2608342091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc67c105280 msgr2=0x7fc67c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.925+0000 7fc681600700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc67c105280 0x7fc67c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.925+0000 7fc681600700 1 -- 172.21.15.43:0/2608342091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c105bd0 msgr2=0x7fc67c19d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.925+0000 7fc681600700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c105bd0 0x7fc67c19d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.925+0000 7fc681600700 1 -- 172.21.15.43:0/2608342091 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67c1a1b40 con 0x7fc67c10e6f0 2024-05-27T07:21:44.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.926+0000 7fc681600700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc67c10e6f0 0x7fc67c1a1550 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc674007d70 tx=0x7fc67400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:44.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.926+0000 7fc679600700 1 -- 172.21.15.43:0/2608342091 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674016020 con 0x7fc67c10e6f0 2024-05-27T07:21:44.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.926+0000 7fc679600700 1 -- 172.21.15.43:0/2608342091 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc67400ac90 con 0x7fc67c10e6f0 2024-05-27T07:21:44.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.926+0000 7fc682000700 1 -- 172.21.15.43:0/2608342091 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc67c1a1e30 con 0x7fc67c10e6f0 2024-05-27T07:21:44.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.927+0000 7fc679600700 1 -- 172.21.15.43:0/2608342091 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6740115e0 con 0x7fc67c10e6f0 2024-05-27T07:21:44.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.927+0000 7fc682000700 1 -- 172.21.15.43:0/2608342091 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc67c1a2370 con 0x7fc67c10e6f0 2024-05-27T07:21:44.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.933+0000 7fc692135b80 1 Processor -- start 2024-05-27T07:21:44.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.933+0000 7fc679600700 1 -- 172.21.15.43:0/2608342091 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc674019020 con 0x7fc67c10e6f0 2024-05-27T07:21:44.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.933+0000 7fc692135b80 1 -- start start 2024-05-27T07:21:44.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.934+0000 7fc692135b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a8ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.934+0000 7fc692135b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e410 0x5609a1a98ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.934+0000 7fc692135b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e820 0x5609a1a9b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.934+0000 7fc692135b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5609a19bce40 con 0x5609a190fe40 2024-05-27T07:21:44.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.934+0000 7fc692135b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5609a19ac9e0 con 0x5609a1a8e410 2024-05-27T07:21:44.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.934+0000 7fc692135b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5609a19b0a20 con 0x5609a1a8e820 2024-05-27T07:21:44.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.934+0000 7fc67be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a8ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.934+0000 7fc67be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a8ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48294/0 (socket says 172.21.15.43:48294) 2024-05-27T07:21:44.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.934+0000 7fc67be00700 1 -- 172.21.15.43:0/599918093 learned_addr learned my addr 172.21.15.43:0/599918093 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:44.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.934+0000 7fc680c00700 1 --2- 172.21.15.43:0/599918093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e410 0x5609a1a98ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.935+0000 7fc67be00700 1 --2- 172.21.15.43:0/599918093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e820 0x5609a1a9b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.935+0000 7fc67be00700 1 -- 172.21.15.43:0/599918093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e820 msgr2=0x5609a1a9b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:44.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.935+0000 7fc67be00700 1 --2- 172.21.15.43:0/599918093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e820 0x5609a1a9b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.935+0000 7fc67be00700 1 -- 172.21.15.43:0/599918093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e410 msgr2=0x5609a1a98ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:44.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.935+0000 7fc67be00700 1 --2- 172.21.15.43:0/599918093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e410 0x5609a1a98ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.935+0000 7fc67be00700 1 -- 172.21.15.43:0/599918093 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609a1975420 con 0x5609a190fe40 2024-05-27T07:21:44.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.935+0000 7fc679600700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc6640b2a70 0x7fc6640b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.935+0000 7fc67be00700 1 --2- 172.21.15.43:0/599918093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a8ded0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc668013640 tx=0x7fc66800a870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0f6c38d6d93022 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:44.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.936+0000 7fc679600700 1 -- 172.21.15.43:0/2608342091 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7fc6740e2740 con 0x7fc67c10e6f0 2024-05-27T07:21:44.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.936+0000 7fc680c00700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc6640b2a70 0x7fc6640b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.936+0000 7fc66e000700 1 -- 172.21.15.43:0/599918093 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc668004030 con 0x5609a190fe40 2024-05-27T07:21:44.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.936+0000 7fc66e000700 1 -- 172.21.15.43:0/599918093 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc668003680 con 0x5609a190fe40 2024-05-27T07:21:44.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.936+0000 7fc66e000700 1 -- 172.21.15.43:0/599918093 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66800f890 con 0x5609a190fe40 2024-05-27T07:21:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.938+0000 7fc692135b80 1 -- 172.21.15.43:0/599918093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 msgr2=0x5609a1a8ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.938+0000 7fc692135b80 1 --2- 172.21.15.43:0/599918093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a8ded0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc668013640 tx=0x7fc66800a870 comp rx=0 tx=0).stop 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.938+0000 7fc680c00700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc6640b2a70 0x7fc6640b4f80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc67000b5c0 tx=0x7fc670005040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.938+0000 7fc692135b80 1 -- 172.21.15.43:0/599918093 shutdown_connections 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.938+0000 7fc692135b80 1 --2- 172.21.15.43:0/599918093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e820 0x5609a1a9b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.938+0000 7fc692135b80 1 --2- 172.21.15.43:0/599918093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a8ded0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.938+0000 7fc692135b80 1 --2- 172.21.15.43:0/599918093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e410 0x5609a1a98ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.938+0000 7fc692135b80 1 -- 172.21.15.43:0/599918093 >> 172.21.15.43:0/599918093 conn(0x5609a196ffc0 msgr2=0x5609a1a89d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.938+0000 7fc692135b80 1 -- 172.21.15.43:0/599918093 shutdown_connections 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.938+0000 7fc692135b80 1 -- 172.21.15.43:0/599918093 wait complete. 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.939+0000 7fc692135b80 1 -- start start 2024-05-27T07:21:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.939+0000 7fc692135b80 1 Processor -- start 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.940+0000 7fc692135b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a98600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.940+0000 7fc692135b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e410 0x5609a1a98b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.940+0000 7fc692135b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e820 0x5609a1a99080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.940+0000 7fc692135b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5609a18b2fa0 con 0x5609a190fe40 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.940+0000 7fc692135b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5609a194d760 con 0x5609a1a8e410 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.940+0000 7fc692135b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5609a1a9b990 con 0x5609a1a8e820 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.940+0000 7fc692135b80 10 client.0 _ll_register_callbacks cb 0x5609a1956c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.940+0000 7fc680c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a98600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.940+0000 7fc680c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a98600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48302/0 (socket says 172.21.15.43:48302) 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.940+0000 7fc680c00700 1 -- 172.21.15.43:0/1336691520 learned_addr learned my addr 172.21.15.43:0/1336691520 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.941+0000 7fc681600700 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e820 0x5609a1a99080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.941+0000 7fc67be00700 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e410 0x5609a1a98b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.941+0000 7fc680c00700 1 -- 172.21.15.43:0/1336691520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e820 msgr2=0x5609a1a99080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.941+0000 7fc680c00700 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e820 0x5609a1a99080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.941+0000 7fc680c00700 1 -- 172.21.15.43:0/1336691520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e410 msgr2=0x5609a1a98b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.941+0000 7fc680c00700 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e410 0x5609a1a98b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.941+0000 7fc680c00700 1 -- 172.21.15.43:0/1336691520 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609a18fc8d0 con 0x5609a190fe40 2024-05-27T07:21:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.941+0000 7fc680c00700 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a98600 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc670015000 tx=0x7fc670009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4b4afb0b23d3db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.941+0000 7fc662a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.942+0000 7fc662a00700 1 -- 172.21.15.43:0/1336691520 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670035060 con 0x5609a190fe40 2024-05-27T07:21:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.942+0000 7fc692135b80 1 -- 172.21.15.43:0/1336691520 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc668003680 con 0x5609a190fe40 2024-05-27T07:21:44.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.942+0000 7fc662a00700 1 -- 172.21.15.43:0/1336691520 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7fc670025680 con 0x5609a190fe40 2024-05-27T07:21:44.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.942+0000 7fc662a00700 1 -- 172.21.15.43:0/1336691520 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fc67003c050 con 0x5609a190fe40 2024-05-27T07:21:44.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.942+0000 7fc692135b80 10 client.16644 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:44.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.942+0000 7fc692135b80 1 -- 172.21.15.43:0/1336691520 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x5609a197d060 con 0x5609a190fe40 2024-05-27T07:21:44.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.943+0000 7fc662a00700 1 -- 172.21.15.43:0/1336691520 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc668003680 con 0x5609a190fe40 2024-05-27T07:21:44.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.943+0000 7fc662a00700 1 -- 172.21.15.43:0/1336691520 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fc670031c10 con 0x5609a190fe40 2024-05-27T07:21:44.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.943+0000 7fc692135b80 10 client.16644 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:44.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.943+0000 7fc692135b80 10 client.16644 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:21:44.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.943+0000 7fc692135b80 20 client.16644 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:44.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.943+0000 7fc692135b80 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5609a1aa8c10 0x5609a1aab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.943+0000 7fc692135b80 4 client.16644 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:21:44.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.943+0000 7fc692135b80 1 -- 172.21.15.43:0/1336691520 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609a19bed90 con 0x5609a1aa8c10 2024-05-27T07:21:44.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.944+0000 7fc67be00700 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5609a1aa8c10 0x5609a1aab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.945+0000 7fc67be00700 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5609a1aa8c10 0x5609a1aab0c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dda7c3e86bc20e8f server_cookie=99f3177c654be8f2 in_seq=0 out_seq=0 2024-05-27T07:21:44.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.945+0000 7fc662a00700 10 client.16644 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:44.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.947+0000 7fc662a00700 1 -- 172.21.15.43:0/1336691520 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5609a19bed90 con 0x5609a1aa8c10 2024-05-27T07:21:44.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.947+0000 7fc662a00700 10 client.16644 handle_command_reply: tid=0 2024-05-27T07:21:44.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.947+0000 7fc692135b80 1 client.16644 shutdown 2024-05-27T07:21:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.947+0000 7fc692135b80 1 -- 172.21.15.43:0/1336691520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 msgr2=0x5609a1a98600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.947+0000 7fc692135b80 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a98600 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc670015000 tx=0x7fc670009ca0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.947+0000 7fc692135b80 1 -- 172.21.15.43:0/1336691520 shutdown_connections 2024-05-27T07:21:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.947+0000 7fc692135b80 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e820 0x5609a1a99080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.947+0000 7fc692135b80 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5609a1aa8c10 0x5609a1aab0c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.948+0000 7fc692135b80 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1a98600 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.948+0000 7fc692135b80 1 --2- 172.21.15.43:0/1336691520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e410 0x5609a1a98b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.948+0000 7fc692135b80 1 -- 172.21.15.43:0/1336691520 >> 172.21.15.43:0/1336691520 conn(0x5609a196ffc0 msgr2=0x5609a1a88ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.948+0000 7fc662a00700 0 client.16644 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.948+0000 7fc692135b80 1 -- 172.21.15.43:0/1336691520 shutdown_connections 2024-05-27T07:21:44.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.948+0000 7fc692135b80 1 -- 172.21.15.43:0/1336691520 wait complete. 2024-05-27T07:21:44.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.948+0000 7fc692135b80 20 client.16644 trim_cache size 0 max 16384 2024-05-27T07:21:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.974+0000 7fc692135b80 1 Processor -- start 2024-05-27T07:21:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.974+0000 7fc692135b80 1 -- start start 2024-05-27T07:21:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.974+0000 7fc692135b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a190fe40 0x5609a1aab290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.974+0000 7fc692135b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e410 0x5609a1a9aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.974+0000 7fc692135b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a1a8e820 0x5609a1a93dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.974+0000 7fc692135b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b30a0 con 0x5609a1a8e820 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.974+0000 7fc692135b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b3770 con 0x5609a190fe40 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.974+0000 7fc692135b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b0e90 con 0x5609a1a8e410 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.975+0000 7fc681600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a1a8e820 0x5609a1a93dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.975+0000 7fc681600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a1a8e820 0x5609a1a93dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48310/0 (socket says 172.21.15.43:48310) 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.975+0000 7fc681600700 1 -- 172.21.15.43:0/3961078269 learned_addr learned my addr 172.21.15.43:0/3961078269 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.975+0000 7fc680c00700 1 --2- 172.21.15.43:0/3961078269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a190fe40 0x5609a1aab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.975+0000 7fc67be00700 1 --2- 172.21.15.43:0/3961078269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e410 0x5609a1a9aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.975+0000 7fc681600700 1 -- 172.21.15.43:0/3961078269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e410 msgr2=0x5609a1a9aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.975+0000 7fc681600700 1 --2- 172.21.15.43:0/3961078269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e410 0x5609a1a9aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.975+0000 7fc681600700 1 -- 172.21.15.43:0/3961078269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a190fe40 msgr2=0x5609a1aab290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.975+0000 7fc681600700 1 --2- 172.21.15.43:0/3961078269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a190fe40 0x5609a1aab290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.975+0000 7fc681600700 1 -- 172.21.15.43:0/3961078269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609a1a8b350 con 0x5609a1a8e820 2024-05-27T07:21:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.976+0000 7fc681600700 1 --2- 172.21.15.43:0/3961078269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a1a8e820 0x5609a1a93dd0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc6740a69d0 tx=0x7fc6740a8fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f13a955cb57bf74 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.976+0000 7fc66d600700 1 -- 172.21.15.43:0/3961078269 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6740a9270 con 0x5609a1a8e820 2024-05-27T07:21:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.976+0000 7fc66d600700 1 -- 172.21.15.43:0/3961078269 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc674091300 con 0x5609a1a8e820 2024-05-27T07:21:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.976+0000 7fc66d600700 1 -- 172.21.15.43:0/3961078269 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6740a9c20 con 0x5609a1a8e820 2024-05-27T07:21:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.977+0000 7fc692135b80 1 -- 172.21.15.43:0/3961078269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a1a8e820 msgr2=0x5609a1a93dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.977+0000 7fc692135b80 1 --2- 172.21.15.43:0/3961078269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a1a8e820 0x5609a1a93dd0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc6740a69d0 tx=0x7fc6740a8fa0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.978+0000 7fc692135b80 1 -- 172.21.15.43:0/3961078269 shutdown_connections 2024-05-27T07:21:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.978+0000 7fc692135b80 1 --2- 172.21.15.43:0/3961078269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a1a8e820 0x5609a1a93dd0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.978+0000 7fc692135b80 1 --2- 172.21.15.43:0/3961078269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a190fe40 0x5609a1aab290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.978+0000 7fc692135b80 1 --2- 172.21.15.43:0/3961078269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e410 0x5609a1a9aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.978+0000 7fc692135b80 1 -- 172.21.15.43:0/3961078269 >> 172.21.15.43:0/3961078269 conn(0x5609a196ffc0 msgr2=0x5609a1a985c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.978+0000 7fc692135b80 1 -- 172.21.15.43:0/3961078269 shutdown_connections 2024-05-27T07:21:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.978+0000 7fc692135b80 1 -- 172.21.15.43:0/3961078269 wait complete. 2024-05-27T07:21:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.979+0000 7fc692135b80 1 -- start start 2024-05-27T07:21:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.979+0000 7fc692135b80 1 Processor -- start 2024-05-27T07:21:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc692135b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1aa1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc692135b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e410 0x5609a1a9c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc692135b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e820 0x5609a1a9cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc692135b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b1560 con 0x5609a190fe40 2024-05-27T07:21:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc692135b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b29d0 con 0x5609a1a8e820 2024-05-27T07:21:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc692135b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc67c1b1c30 con 0x5609a1a8e410 2024-05-27T07:21:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc692135b80 10 client.0 _ll_register_callbacks cb 0x5609a1956c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:21:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc680c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1aa1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc680c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1aa1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48326/0 (socket says 172.21.15.43:48326) 2024-05-27T07:21:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc680c00700 1 -- 172.21.15.43:0/1331924400 learned_addr learned my addr 172.21.15.43:0/1331924400 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:21:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.980+0000 7fc67be00700 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e410 0x5609a1a9c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.981+0000 7fc681600700 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e820 0x5609a1a9cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.981+0000 7fc680c00700 1 -- 172.21.15.43:0/1331924400 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e410 msgr2=0x5609a1a9c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.981+0000 7fc680c00700 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e410 0x5609a1a9c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.981+0000 7fc680c00700 1 -- 172.21.15.43:0/1331924400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e820 msgr2=0x5609a1a9cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.981+0000 7fc680c00700 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e820 0x5609a1a9cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.981+0000 7fc680c00700 1 -- 172.21.15.43:0/1331924400 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609a1a9d260 con 0x5609a190fe40 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.981+0000 7fc680c00700 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1aa1290 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fc670009990 tx=0x7fc67000ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38010009334f9bfe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.981+0000 7fc662a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.982+0000 7fc662a00700 1 -- 172.21.15.43:0/1331924400 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670045060 con 0x5609a190fe40 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.982+0000 7fc662a00700 1 -- 172.21.15.43:0/1331924400 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9496+0+0 (secure 0 0 0) 0x7fc67000fb30 con 0x5609a190fe40 2024-05-27T07:21:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.982+0000 7fc692135b80 1 -- 172.21.15.43:0/1331924400 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc674091300 con 0x5609a190fe40 2024-05-27T07:21:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.982+0000 7fc662a00700 1 -- 172.21.15.43:0/1331924400 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7fc67004c050 con 0x5609a190fe40 2024-05-27T07:21:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.982+0000 7fc692135b80 10 client.16650 fetch_fsmap learned FSMap version 79 2024-05-27T07:21:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.982+0000 7fc692135b80 1 -- 172.21.15.43:0/1331924400 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=4+}) -- 0x5609a1b1bde0 con 0x5609a190fe40 2024-05-27T07:21:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.983+0000 7fc662a00700 1 -- 172.21.15.43:0/1331924400 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc674091300 con 0x5609a190fe40 2024-05-27T07:21:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.983+0000 7fc662a00700 1 -- 172.21.15.43:0/1331924400 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 79) ==== 5966+0+0 (secure 0 0 0) 0x7fc670048a60 con 0x5609a190fe40 2024-05-27T07:21:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.983+0000 7fc692135b80 10 client.16650 fetch_fsmap finished waiting for FSMap version 79 2024-05-27T07:21:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.983+0000 7fc692135b80 10 client.16650 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:21:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.983+0000 7fc692135b80 20 client.16650 populate_metadata read hostname 'smithi043' 2024-05-27T07:21:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.983+0000 7fc692135b80 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5609a1b1c430 0x5609a1b1e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:21:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.983+0000 7fc692135b80 4 client.16650 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:21:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.983+0000 7fc692135b80 1 -- 172.21.15.43:0/1331924400 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5609a19b8570 con 0x5609a1b1c430 2024-05-27T07:21:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.984+0000 7fc67be00700 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5609a1b1c430 0x5609a1b1e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:21:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.985+0000 7fc67be00700 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5609a1b1c430 0x5609a1b1e850 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18dbb3191c27eb60 server_cookie=4c35f415a381253 in_seq=0 out_seq=0 2024-05-27T07:21:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.985+0000 7fc662a00700 10 client.16650 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.988+0000 7fc662a00700 1 -- 172.21.15.43:0/1331924400 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5609a19b8570 con 0x5609a1b1c430 2024-05-27T07:21:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.988+0000 7fc662a00700 10 client.16650 handle_command_reply: tid=0 2024-05-27T07:21:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.988+0000 7fc692135b80 1 client.16650 shutdown 2024-05-27T07:21:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.988+0000 7fc692135b80 1 -- 172.21.15.43:0/1331924400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 msgr2=0x5609a1aa1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.988+0000 7fc692135b80 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1aa1290 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fc670009990 tx=0x7fc67000ff90 comp rx=0 tx=0).stop 2024-05-27T07:21:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.988+0000 7fc692135b80 1 -- 172.21.15.43:0/1331924400 shutdown_connections 2024-05-27T07:21:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.988+0000 7fc692135b80 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5609a1a8e820 0x5609a1a9cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.989+0000 7fc692135b80 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5609a1b1c430 0x5609a1b1e850 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.989+0000 7fc692135b80 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609a190fe40 0x5609a1aa1290 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.989+0000 7fc692135b80 1 --2- 172.21.15.43:0/1331924400 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5609a1a8e410 0x5609a1a9c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.989+0000 7fc662a00700 0 client.16650 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:21:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.989+0000 7fc692135b80 1 -- 172.21.15.43:0/1331924400 >> 172.21.15.43:0/1331924400 conn(0x5609a196ffc0 msgr2=0x5609a1aa3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:44.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.989+0000 7fc692135b80 1 -- 172.21.15.43:0/1331924400 shutdown_connections 2024-05-27T07:21:44.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.989+0000 7fc692135b80 1 -- 172.21.15.43:0/1331924400 wait complete. 2024-05-27T07:21:44.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.989+0000 7fc692135b80 20 client.16650 trim_cache size 0 max 16384 2024-05-27T07:21:44.980 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:21:44.980 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:21:44.980 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:21:44.980 INFO:teuthology.orchestra.run.smithi043.stdout: "dd418cd3-0880-4c2b-a731-d0c0e5d4d7e5": { 2024-05-27T07:21:44.980 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:21:44.981 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "dd418cd3-0880-4c2b-a731-d0c0e5d4d7e5", 2024-05-27T07:21:44.981 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:21:44.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:21:44.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:21:44.981 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:21:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 -- 172.21.15.43:0/2608342091 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc6640b2a70 msgr2=0x7fc6640b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc6640b2a70 0x7fc6640b4f80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc67000b5c0 tx=0x7fc670005040 comp rx=0 tx=0).stop 2024-05-27T07:21:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 -- 172.21.15.43:0/2608342091 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc67c10e6f0 msgr2=0x7fc67c1a1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:21:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc67c10e6f0 0x7fc67c1a1550 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc674007d70 tx=0x7fc67400a340 comp rx=0 tx=0).stop 2024-05-27T07:21:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 -- 172.21.15.43:0/2608342091 shutdown_connections 2024-05-27T07:21:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc6640b2a70 0x7fc6640b4f80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc67c10e6f0 0x7fc67c1a1550 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc67c105280 0x7fc67c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 --2- 172.21.15.43:0/2608342091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc67c105bd0 0x7fc67c19d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:21:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 -- 172.21.15.43:0/2608342091 >> 172.21.15.43:0/2608342091 conn(0x7fc67c078ab0 msgr2=0x7fc67c102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:21:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 -- 172.21.15.43:0/2608342091 shutdown_connections 2024-05-27T07:21:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:21:44.990+0000 7fc682000700 1 -- 172.21.15.43:0/2608342091 wait complete. 2024-05-27T07:21:45.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 41617 0.62 MB/sec execute 229 sec latency 50.108 ms 2024-05-27T07:21:45.930 INFO:tasks.cephfs.filesystem:scrub status for tag:dd418cd3-0880-4c2b-a731-d0c0e5d4d7e5 - {'path': '/', 'tag': 'dd418cd3-0880-4c2b-a731-d0c0e5d4d7e5', 'options': 'recursive,force'} 2024-05-27T07:21:46.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 41801 0.63 MB/sec execute 230 sec latency 55.249 ms 2024-05-27T07:21:46.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:46 smithi132 ceph-mon[35283]: pgmap v260: 129 pgs: 129 active+clean; 68 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 264 KiB/s wr, 43 op/s 2024-05-27T07:21:46.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:46 smithi043 ceph-mon[31531]: pgmap v260: 129 pgs: 129 active+clean; 68 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 264 KiB/s wr, 43 op/s 2024-05-27T07:21:46.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:46 smithi179 ceph-mon[35888]: pgmap v260: 129 pgs: 129 active+clean; 68 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 264 KiB/s wr, 43 op/s 2024-05-27T07:21:47.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 41980 0.63 MB/sec execute 231 sec latency 61.094 ms 2024-05-27T07:21:47.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:47 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:21:47.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:47 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:21:47.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:47 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:21:47.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:47 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:21:47.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:47 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:21:47.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:47 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:21:47.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:47 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:21:47.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:47 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:21:47.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:47 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:21:48.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42078 0.63 MB/sec execute 232 sec latency 51.146 ms 2024-05-27T07:21:48.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:48 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:21:48.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:48 smithi132 ceph-mon[35283]: pgmap v261: 129 pgs: 129 active+clean; 70 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 486 KiB/s wr, 61 op/s 2024-05-27T07:21:48.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:48 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:21:48.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:48 smithi043 ceph-mon[31531]: pgmap v261: 129 pgs: 129 active+clean; 70 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 486 KiB/s wr, 61 op/s 2024-05-27T07:21:48.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:48 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:21:48.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:48 smithi179 ceph-mon[35888]: pgmap v261: 129 pgs: 129 active+clean; 70 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 486 KiB/s wr, 61 op/s 2024-05-27T07:21:49.209 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42335 0.64 MB/sec execute 233 sec latency 54.405 ms 2024-05-27T07:21:50.210 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42364 0.63 MB/sec execute 234 sec latency 56.976 ms 2024-05-27T07:21:50.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:50 smithi132 ceph-mon[35283]: pgmap v262: 129 pgs: 129 active+clean; 70 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 293 KiB/s wr, 36 op/s 2024-05-27T07:21:50.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:50 smithi043 ceph-mon[31531]: pgmap v262: 129 pgs: 129 active+clean; 70 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 293 KiB/s wr, 36 op/s 2024-05-27T07:21:50.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:50 smithi179 ceph-mon[35888]: pgmap v262: 129 pgs: 129 active+clean; 70 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 293 KiB/s wr, 36 op/s 2024-05-27T07:21:51.210 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42387 0.63 MB/sec execute 235 sec latency 68.432 ms 2024-05-27T07:21:52.210 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42544 0.64 MB/sec execute 236 sec latency 59.082 ms 2024-05-27T07:21:52.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:52 smithi132 ceph-mon[35283]: pgmap v263: 129 pgs: 129 active+clean; 72 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 482 KiB/s wr, 52 op/s 2024-05-27T07:21:52.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:52 smithi043 ceph-mon[31531]: pgmap v263: 129 pgs: 129 active+clean; 72 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 482 KiB/s wr, 52 op/s 2024-05-27T07:21:52.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:52 smithi179 ceph-mon[35888]: pgmap v263: 129 pgs: 129 active+clean; 72 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 482 KiB/s wr, 52 op/s 2024-05-27T07:21:53.210 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42662 0.64 MB/sec execute 237 sec latency 56.284 ms 2024-05-27T07:21:54.210 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42751 0.64 MB/sec execute 238 sec latency 50.982 ms 2024-05-27T07:21:54.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:54 smithi132 ceph-mon[35283]: pgmap v264: 129 pgs: 129 active+clean; 72 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 410 KiB/s wr, 33 op/s 2024-05-27T07:21:54.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:54 smithi043 ceph-mon[31531]: pgmap v264: 129 pgs: 129 active+clean; 72 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 410 KiB/s wr, 33 op/s 2024-05-27T07:21:54.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:54 smithi179 ceph-mon[35888]: pgmap v264: 129 pgs: 129 active+clean; 72 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 410 KiB/s wr, 33 op/s 2024-05-27T07:21:55.210 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42843 0.64 MB/sec execute 239 sec latency 56.022 ms 2024-05-27T07:21:56.211 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42903 0.64 MB/sec execute 240 sec latency 43.016 ms 2024-05-27T07:21:56.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:56 smithi043 ceph-mon[31531]: pgmap v265: 129 pgs: 129 active+clean; 72 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 410 KiB/s wr, 33 op/s 2024-05-27T07:21:56.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:56 smithi132 ceph-mon[35283]: pgmap v265: 129 pgs: 129 active+clean; 72 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 410 KiB/s wr, 33 op/s 2024-05-27T07:21:56.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:56 smithi179 ceph-mon[35888]: pgmap v265: 129 pgs: 129 active+clean; 72 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 410 KiB/s wr, 33 op/s 2024-05-27T07:21:57.210 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42969 0.63 MB/sec execute 241 sec latency 57.795 ms 2024-05-27T07:21:58.211 INFO:tasks.workunit.client.0.smithi043.stdout: 1 43019 0.63 MB/sec execute 242 sec latency 47.861 ms 2024-05-27T07:21:58.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:21:58 smithi132 ceph-mon[35283]: pgmap v266: 129 pgs: 129 active+clean; 73 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 653 KiB/s wr, 51 op/s 2024-05-27T07:21:58.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:21:58 smithi043 ceph-mon[31531]: pgmap v266: 129 pgs: 129 active+clean; 73 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 653 KiB/s wr, 51 op/s 2024-05-27T07:21:58.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:21:58 smithi179 ceph-mon[35888]: pgmap v266: 129 pgs: 129 active+clean; 73 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 653 KiB/s wr, 51 op/s 2024-05-27T07:21:59.211 INFO:tasks.workunit.client.0.smithi043.stdout: 1 43072 0.63 MB/sec execute 243 sec latency 50.968 ms 2024-05-27T07:22:00.211 INFO:tasks.workunit.client.0.smithi043.stdout: 1 43147 0.63 MB/sec execute 244 sec latency 52.151 ms 2024-05-27T07:22:00.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:00 smithi132 ceph-mon[35283]: pgmap v267: 129 pgs: 129 active+clean; 73 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 431 KiB/s wr, 33 op/s 2024-05-27T07:22:00.652 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:00 smithi179 ceph-mon[35888]: pgmap v267: 129 pgs: 129 active+clean; 73 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 431 KiB/s wr, 33 op/s 2024-05-27T07:22:00.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:00 smithi043 ceph-mon[31531]: pgmap v267: 129 pgs: 129 active+clean; 73 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 431 KiB/s wr, 33 op/s 2024-05-27T07:22:00.960 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb89c400700 time 2024-05-27T07:22:00.661532+0000 2024-05-27T07:22:00.960 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:00.960 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:00.960 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb8b1ec7ba6] 2024-05-27T07:22:00.960 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8b1ec7dd1] 2024-05-27T07:22:00.960 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56480a33723d] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56480a322971] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 5: (SnapRealm::check_cache() const+0x13e) [0x56480a320fa0] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56480a321651] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56480a2b6347] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564809fb287e] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564809fb7e94] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564809fc2969] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 11: (MDLog::_replay_thread()+0x1f38) [0x56480a35b48c] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56480a3649cb] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 13: (Thread::entry_wrapper()+0x43) [0x7fb8b1e9672d] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb8b1e96749] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb8b0a3a1ca] 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 16: clone() 2024-05-27T07:22:00.961 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: *** Caught signal (Aborted) ** 2024-05-27T07:22:00.962 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: in thread 7fb89c400700 thread_name:md_log_replay 2024-05-27T07:22:00.962 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.664+0000 7fb89c400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb89c400700 time 2024-05-27T07:22:00.661532+0000 2024-05-27T07:22:00.962 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:00.962 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.962 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:00.962 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb8b1ec7ba6] 2024-05-27T07:22:00.962 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8b1ec7dd1] 2024-05-27T07:22:00.962 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56480a33723d] 2024-05-27T07:22:00.962 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56480a322971] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 5: (SnapRealm::check_cache() const+0x13e) [0x56480a320fa0] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56480a321651] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56480a2b6347] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564809fb287e] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564809fb7e94] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564809fc2969] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 11: (MDLog::_replay_thread()+0x1f38) [0x56480a35b48c] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56480a3649cb] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 13: (Thread::entry_wrapper()+0x43) [0x7fb8b1e9672d] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb8b1e96749] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb8b0a3a1ca] 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 16: clone() 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.963 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:00.964 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb8b0a44d20] 2024-05-27T07:22:00.964 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2: gsignal() 2024-05-27T07:22:00.964 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 3: abort() 2024-05-27T07:22:00.964 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb8b1ec7ced] 2024-05-27T07:22:00.964 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8b1ec7dd1] 2024-05-27T07:22:00.964 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56480a33723d] 2024-05-27T07:22:00.964 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56480a322971] 2024-05-27T07:22:00.964 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 8: (SnapRealm::check_cache() const+0x13e) [0x56480a320fa0] 2024-05-27T07:22:00.964 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56480a321651] 2024-05-27T07:22:00.964 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56480a2b6347] 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564809fb287e] 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564809fb7e94] 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564809fc2969] 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 14: (MDLog::_replay_thread()+0x1f38) [0x56480a35b48c] 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56480a3649cb] 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 16: (Thread::entry_wrapper()+0x43) [0x7fb8b1e9672d] 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb8b1e96749] 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb8b0a3a1ca] 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 19: clone() 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.669+0000 7fb89c400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: in thread 7fb89c400700 thread_name:md_log_replay 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb8b0a44d20] 2024-05-27T07:22:00.965 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2: gsignal() 2024-05-27T07:22:00.966 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 3: abort() 2024-05-27T07:22:00.966 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb8b1ec7ced] 2024-05-27T07:22:00.966 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8b1ec7dd1] 2024-05-27T07:22:00.966 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56480a33723d] 2024-05-27T07:22:00.966 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56480a322971] 2024-05-27T07:22:00.966 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 8: (SnapRealm::check_cache() const+0x13e) [0x56480a320fa0] 2024-05-27T07:22:00.966 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56480a321651] 2024-05-27T07:22:00.966 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56480a2b6347] 2024-05-27T07:22:00.967 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564809fb287e] 2024-05-27T07:22:00.967 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564809fb7e94] 2024-05-27T07:22:00.967 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564809fc2969] 2024-05-27T07:22:00.967 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 14: (MDLog::_replay_thread()+0x1f38) [0x56480a35b48c] 2024-05-27T07:22:00.967 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56480a3649cb] 2024-05-27T07:22:00.967 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 16: (Thread::entry_wrapper()+0x43) [0x7fb8b1e9672d] 2024-05-27T07:22:00.967 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb8b1e96749] 2024-05-27T07:22:00.967 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb8b0a3a1ca] 2024-05-27T07:22:00.968 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 19: clone() 2024-05-27T07:22:00.968 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:00.968 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.968 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: -1> 2024-05-27T07:22:00.664+0000 7fb89c400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb89c400700 time 2024-05-27T07:22:00.661532+0000 2024-05-27T07:22:00.968 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:00.968 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.968 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:00.968 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb8b1ec7ba6] 2024-05-27T07:22:00.968 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8b1ec7dd1] 2024-05-27T07:22:00.969 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56480a33723d] 2024-05-27T07:22:00.969 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56480a322971] 2024-05-27T07:22:00.969 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 5: (SnapRealm::check_cache() const+0x13e) [0x56480a320fa0] 2024-05-27T07:22:00.969 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56480a321651] 2024-05-27T07:22:00.969 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56480a2b6347] 2024-05-27T07:22:00.969 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564809fb287e] 2024-05-27T07:22:00.969 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564809fb7e94] 2024-05-27T07:22:00.969 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564809fc2969] 2024-05-27T07:22:00.969 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 11: (MDLog::_replay_thread()+0x1f38) [0x56480a35b48c] 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56480a3649cb] 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 13: (Thread::entry_wrapper()+0x43) [0x7fb8b1e9672d] 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb8b1e96749] 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb8b0a3a1ca] 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 16: clone() 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 0> 2024-05-27T07:22:00.669+0000 7fb89c400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: in thread 7fb89c400700 thread_name:md_log_replay 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb8b0a44d20] 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2: gsignal() 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 3: abort() 2024-05-27T07:22:00.970 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb8b1ec7ced] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8b1ec7dd1] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56480a33723d] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56480a322971] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 8: (SnapRealm::check_cache() const+0x13e) [0x56480a320fa0] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56480a321651] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56480a2b6347] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564809fb287e] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564809fb7e94] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564809fc2969] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 14: (MDLog::_replay_thread()+0x1f38) [0x56480a35b48c] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56480a3649cb] 2024-05-27T07:22:00.971 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 16: (Thread::entry_wrapper()+0x43) [0x7fb8b1e9672d] 2024-05-27T07:22:00.972 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb8b1e96749] 2024-05-27T07:22:00.972 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb8b0a3a1ca] 2024-05-27T07:22:00.972 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 19: clone() 2024-05-27T07:22:00.972 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:00.972 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.972 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: -1> 2024-05-27T07:22:00.664+0000 7fb89c400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb89c400700 time 2024-05-27T07:22:00.661532+0000 2024-05-27T07:22:00.972 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:00.972 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb8b1ec7ba6] 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8b1ec7dd1] 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56480a33723d] 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56480a322971] 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 5: (SnapRealm::check_cache() const+0x13e) [0x56480a320fa0] 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56480a321651] 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56480a2b6347] 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564809fb287e] 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564809fb7e94] 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x564809fc2969] 2024-05-27T07:22:00.973 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 11: (MDLog::_replay_thread()+0x1f38) [0x56480a35b48c] 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56480a3649cb] 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 13: (Thread::entry_wrapper()+0x43) [0x7fb8b1e9672d] 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb8b1e96749] 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb8b0a3a1ca] 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 16: clone() 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 0> 2024-05-27T07:22:00.669+0000 7fb89c400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: in thread 7fb89c400700 thread_name:md_log_replay 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb8b0a44d20] 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2: gsignal() 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 3: abort() 2024-05-27T07:22:00.974 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb8b1ec7ced] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb8b1ec7dd1] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56480a33723d] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56480a322971] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 8: (SnapRealm::check_cache() const+0x13e) [0x56480a320fa0] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56480a321651] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56480a2b6347] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x564809fb287e] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x564809fb7e94] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x564809fc2969] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 14: (MDLog::_replay_thread()+0x1f38) [0x56480a35b48c] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56480a3649cb] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 16: (Thread::entry_wrapper()+0x43) [0x7fb8b1e9672d] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb8b1e96749] 2024-05-27T07:22:00.975 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb8b0a3a1ca] 2024-05-27T07:22:00.976 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 19: clone() 2024-05-27T07:22:00.976 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:00.976 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[65832]: 2024-05-27T07:22:00.976 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:00 smithi179 podman[72657]: 2024-05-27 07:22:00.81326508 +0000 UTC m=+0.039907585 container died 374a158fddff45b23791ef94bb5347cf67b5e12456c9a1154f2986f68f5a70cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T07:22:01.211 INFO:tasks.workunit.client.0.smithi043.stdout: 1 43209 0.62 MB/sec execute 245 sec latency 55.148 ms 2024-05-27T07:22:01.462 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffa69c00700 time 2024-05-27T07:22:01.003233+0000 2024-05-27T07:22:01.462 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:01.462 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:01.462 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffa7f802ba6] 2024-05-27T07:22:01.462 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa7f802dd1] 2024-05-27T07:22:01.462 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eb9115023d] 2024-05-27T07:22:01.462 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55eb9113b971] 2024-05-27T07:22:01.462 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 5: (SnapRealm::check_cache() const+0x13e) [0x55eb91139fa0] 2024-05-27T07:22:01.462 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eb9113a651] 2024-05-27T07:22:01.462 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eb910cf347] 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eb90dcb87e] 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eb90dd0e94] 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55eb90ddb969] 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 11: (MDLog::_replay_thread()+0x1f38) [0x55eb9117448c] 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55eb9117d9cb] 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 13: (Thread::entry_wrapper()+0x43) [0x7ffa7f7d172d] 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffa7f7d1749] 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffa7e3751ca] 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 16: clone() 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: *** Caught signal (Aborted) ** 2024-05-27T07:22:01.463 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: in thread 7ffa69c00700 thread_name:md_log_replay 2024-05-27T07:22:01.465 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.006+0000 7ffa69c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffa69c00700 time 2024-05-27T07:22:01.003233+0000 2024-05-27T07:22:01.465 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:01.465 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.465 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:01.465 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffa7f802ba6] 2024-05-27T07:22:01.465 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa7f802dd1] 2024-05-27T07:22:01.465 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eb9115023d] 2024-05-27T07:22:01.465 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55eb9113b971] 2024-05-27T07:22:01.465 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 5: (SnapRealm::check_cache() const+0x13e) [0x55eb91139fa0] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eb9113a651] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eb910cf347] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eb90dcb87e] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eb90dd0e94] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55eb90ddb969] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 11: (MDLog::_replay_thread()+0x1f38) [0x55eb9117448c] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55eb9117d9cb] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 13: (Thread::entry_wrapper()+0x43) [0x7ffa7f7d172d] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffa7f7d1749] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffa7e3751ca] 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 16: clone() 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:01.466 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffa7e37fd20] 2024-05-27T07:22:01.467 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2: gsignal() 2024-05-27T07:22:01.467 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 3: abort() 2024-05-27T07:22:01.467 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffa7f802ced] 2024-05-27T07:22:01.467 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa7f802dd1] 2024-05-27T07:22:01.468 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eb9115023d] 2024-05-27T07:22:01.468 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55eb9113b971] 2024-05-27T07:22:01.468 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 8: (SnapRealm::check_cache() const+0x13e) [0x55eb91139fa0] 2024-05-27T07:22:01.468 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eb9113a651] 2024-05-27T07:22:01.468 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eb910cf347] 2024-05-27T07:22:01.468 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eb90dcb87e] 2024-05-27T07:22:01.468 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eb90dd0e94] 2024-05-27T07:22:01.468 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55eb90ddb969] 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 14: (MDLog::_replay_thread()+0x1f38) [0x55eb9117448c] 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55eb9117d9cb] 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 16: (Thread::entry_wrapper()+0x43) [0x7ffa7f7d172d] 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffa7f7d1749] 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffa7e3751ca] 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 19: clone() 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.009+0000 7ffa69c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: in thread 7ffa69c00700 thread_name:md_log_replay 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffa7e37fd20] 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2: gsignal() 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 3: abort() 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffa7f802ced] 2024-05-27T07:22:01.469 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa7f802dd1] 2024-05-27T07:22:01.470 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eb9115023d] 2024-05-27T07:22:01.470 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55eb9113b971] 2024-05-27T07:22:01.470 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 8: (SnapRealm::check_cache() const+0x13e) [0x55eb91139fa0] 2024-05-27T07:22:01.470 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eb9113a651] 2024-05-27T07:22:01.470 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eb910cf347] 2024-05-27T07:22:01.470 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eb90dcb87e] 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eb90dd0e94] 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55eb90ddb969] 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 14: (MDLog::_replay_thread()+0x1f38) [0x55eb9117448c] 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55eb9117d9cb] 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 16: (Thread::entry_wrapper()+0x43) [0x7ffa7f7d172d] 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffa7f7d1749] 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffa7e3751ca] 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 19: clone() 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: -1> 2024-05-27T07:22:01.006+0000 7ffa69c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffa69c00700 time 2024-05-27T07:22:01.003233+0000 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.471 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:01.472 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffa7f802ba6] 2024-05-27T07:22:01.472 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa7f802dd1] 2024-05-27T07:22:01.472 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eb9115023d] 2024-05-27T07:22:01.472 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55eb9113b971] 2024-05-27T07:22:01.472 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 5: (SnapRealm::check_cache() const+0x13e) [0x55eb91139fa0] 2024-05-27T07:22:01.472 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eb9113a651] 2024-05-27T07:22:01.472 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eb910cf347] 2024-05-27T07:22:01.472 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eb90dcb87e] 2024-05-27T07:22:01.472 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eb90dd0e94] 2024-05-27T07:22:01.472 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55eb90ddb969] 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 11: (MDLog::_replay_thread()+0x1f38) [0x55eb9117448c] 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55eb9117d9cb] 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 13: (Thread::entry_wrapper()+0x43) [0x7ffa7f7d172d] 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffa7f7d1749] 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffa7e3751ca] 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 16: clone() 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 0> 2024-05-27T07:22:01.009+0000 7ffa69c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: in thread 7ffa69c00700 thread_name:md_log_replay 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffa7e37fd20] 2024-05-27T07:22:01.473 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2: gsignal() 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 3: abort() 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffa7f802ced] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa7f802dd1] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eb9115023d] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55eb9113b971] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 8: (SnapRealm::check_cache() const+0x13e) [0x55eb91139fa0] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eb9113a651] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eb910cf347] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eb90dcb87e] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eb90dd0e94] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55eb90ddb969] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 14: (MDLog::_replay_thread()+0x1f38) [0x55eb9117448c] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55eb9117d9cb] 2024-05-27T07:22:01.474 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 16: (Thread::entry_wrapper()+0x43) [0x7ffa7f7d172d] 2024-05-27T07:22:01.475 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffa7f7d1749] 2024-05-27T07:22:01.475 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffa7e3751ca] 2024-05-27T07:22:01.475 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 19: clone() 2024-05-27T07:22:01.475 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:01.475 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.475 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: -1> 2024-05-27T07:22:01.006+0000 7ffa69c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7ffa69c00700 time 2024-05-27T07:22:01.003233+0000 2024-05-27T07:22:01.475 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ffa7f802ba6] 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa7f802dd1] 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eb9115023d] 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55eb9113b971] 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 5: (SnapRealm::check_cache() const+0x13e) [0x55eb91139fa0] 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eb9113a651] 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eb910cf347] 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eb90dcb87e] 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eb90dd0e94] 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55eb90ddb969] 2024-05-27T07:22:01.476 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 11: (MDLog::_replay_thread()+0x1f38) [0x55eb9117448c] 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55eb9117d9cb] 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 13: (Thread::entry_wrapper()+0x43) [0x7ffa7f7d172d] 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 14: (Thread::_entry_func(void*)+0xd) [0x7ffa7f7d1749] 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ffa7e3751ca] 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 16: clone() 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 0> 2024-05-27T07:22:01.009+0000 7ffa69c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: in thread 7ffa69c00700 thread_name:md_log_replay 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ffa7e37fd20] 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2: gsignal() 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 3: abort() 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ffa7f802ced] 2024-05-27T07:22:01.477 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ffa7f802dd1] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55eb9115023d] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55eb9113b971] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 8: (SnapRealm::check_cache() const+0x13e) [0x55eb91139fa0] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55eb9113a651] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55eb910cf347] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55eb90dcb87e] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55eb90dd0e94] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55eb90ddb969] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 14: (MDLog::_replay_thread()+0x1f38) [0x55eb9117448c] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55eb9117d9cb] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 16: (Thread::entry_wrapper()+0x43) [0x7ffa7f7d172d] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 17: (Thread::_entry_func(void*)+0xd) [0x7ffa7f7d1749] 2024-05-27T07:22:01.478 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ffa7e3751ca] 2024-05-27T07:22:01.479 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 19: clone() 2024-05-27T07:22:01.479 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:01.479 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[65523]: 2024-05-27T07:22:01.479 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:01 smithi179 podman[72671]: 2024-05-27 07:22:01.212449738 +0000 UTC m=+0.116004663 container died 83caa31734dd4fa0f625f2deea24cdd45f088f96815618ce547f710a5e63a7ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:22:02.211 INFO:tasks.workunit.client.0.smithi043.stdout: 1 43319 0.62 MB/sec execute 246 sec latency 59.787 ms 2024-05-27T07:22:02.314 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:02 smithi179 podman[72657]: 2024-05-27 07:22:02.047410012 +0000 UTC m=+1.274052554 container remove 374a158fddff45b23791ef94bb5347cf67b5e12456c9a1154f2986f68f5a70cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-27T07:22:02.314 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:02 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:22:02.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:02 smithi132 ceph-mon[35283]: pgmap v268: 129 pgs: 129 active+clean; 74 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 543 KiB/s wr, 53 op/s 2024-05-27T07:22:02.634 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:02 smithi179 ceph-mon[35888]: pgmap v268: 129 pgs: 129 active+clean; 74 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 543 KiB/s wr, 53 op/s 2024-05-27T07:22:02.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:02 smithi043 ceph-mon[31531]: pgmap v268: 129 pgs: 129 active+clean; 74 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 543 KiB/s wr, 53 op/s 2024-05-27T07:22:02.898 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:02 smithi179 podman[72671]: 2024-05-27 07:22:02.643148318 +0000 UTC m=+1.546703289 container remove 83caa31734dd4fa0f625f2deea24cdd45f088f96815618ce547f710a5e63a7ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:22:02.898 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:02 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:22:03.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:03 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:22:03.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:03 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 5.497s CPU time. 2024-05-27T07:22:03.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:02 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:22:03.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:02 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 6.403s CPU time. 2024-05-27T07:22:03.211 INFO:tasks.workunit.client.0.smithi043.stdout: 1 43461 0.62 MB/sec execute 247 sec latency 55.471 ms 2024-05-27T07:22:03.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:03 smithi043 ceph-mon[31531]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:22:03.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:03 smithi179 ceph-mon[35888]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:22:03.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:03 smithi132 ceph-mon[35283]: osdmap e114: 12 total, 12 up, 12 in 2024-05-27T07:22:04.211 INFO:tasks.workunit.client.0.smithi043.stdout: 1 43541 0.62 MB/sec execute 248 sec latency 64.097 ms 2024-05-27T07:22:04.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:04 smithi043 ceph-mon[31531]: pgmap v270: 129 pgs: 129 active+clean; 74 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 426 KiB/s wr, 46 op/s 2024-05-27T07:22:04.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:04 smithi179 ceph-mon[35888]: pgmap v270: 129 pgs: 129 active+clean; 74 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 426 KiB/s wr, 46 op/s 2024-05-27T07:22:04.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:04 smithi132 ceph-mon[35283]: pgmap v270: 129 pgs: 129 active+clean; 74 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 426 KiB/s wr, 46 op/s 2024-05-27T07:22:05.211 INFO:tasks.workunit.client.0.smithi043.stdout: 1 43656 0.62 MB/sec execute 249 sec latency 59.021 ms 2024-05-27T07:22:06.212 INFO:tasks.workunit.client.0.smithi043.stdout: 1 43765 0.62 MB/sec execute 250 sec latency 50.000 ms 2024-05-27T07:22:06.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:06 smithi043 ceph-mon[31531]: pgmap v271: 129 pgs: 129 active+clean; 74 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 426 KiB/s wr, 46 op/s 2024-05-27T07:22:06.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:06 smithi179 ceph-mon[35888]: pgmap v271: 129 pgs: 129 active+clean; 74 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 426 KiB/s wr, 46 op/s 2024-05-27T07:22:06.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:06 smithi132 ceph-mon[35283]: pgmap v271: 129 pgs: 129 active+clean; 74 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 426 KiB/s wr, 46 op/s 2024-05-27T07:22:07.212 INFO:tasks.workunit.client.0.smithi043.stdout: 1 43856 0.62 MB/sec execute 251 sec latency 54.852 ms 2024-05-27T07:22:08.212 INFO:tasks.workunit.client.0.smithi043.stdout: 1 44016 0.63 MB/sec execute 252 sec latency 72.800 ms 2024-05-27T07:22:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:08 smithi043 ceph-mon[31531]: pgmap v272: 129 pgs: 129 active+clean; 74 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 442 KiB/s wr, 70 op/s 2024-05-27T07:22:08.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:08 smithi179 ceph-mon[35888]: pgmap v272: 129 pgs: 129 active+clean; 74 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 442 KiB/s wr, 70 op/s 2024-05-27T07:22:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:08 smithi132 ceph-mon[35283]: pgmap v272: 129 pgs: 129 active+clean; 74 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 442 KiB/s wr, 70 op/s 2024-05-27T07:22:09.212 INFO:tasks.workunit.client.0.smithi043.stdout: 1 44104 0.62 MB/sec execute 253 sec latency 66.521 ms 2024-05-27T07:22:10.212 INFO:tasks.workunit.client.0.smithi043.stdout: 1 44209 0.62 MB/sec execute 254 sec latency 66.492 ms 2024-05-27T07:22:10.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:10 smithi043 ceph-mon[31531]: pgmap v273: 129 pgs: 129 active+clean; 74 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 442 KiB/s wr, 70 op/s 2024-05-27T07:22:10.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:10 smithi179 ceph-mon[35888]: pgmap v273: 129 pgs: 129 active+clean; 74 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 442 KiB/s wr, 70 op/s 2024-05-27T07:22:10.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:10 smithi132 ceph-mon[35283]: pgmap v273: 129 pgs: 129 active+clean; 74 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 442 KiB/s wr, 70 op/s 2024-05-27T07:22:11.212 INFO:tasks.workunit.client.0.smithi043.stdout: 1 44303 0.62 MB/sec execute 255 sec latency 65.468 ms 2024-05-27T07:22:12.212 INFO:tasks.workunit.client.0.smithi043.stdout: 1 44440 0.62 MB/sec execute 256 sec latency 67.912 ms 2024-05-27T07:22:12.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:12 smithi043 ceph-mon[31531]: pgmap v274: 129 pgs: 129 active+clean; 75 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 408 KiB/s wr, 66 op/s 2024-05-27T07:22:12.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:12 smithi043 ceph-mon[31531]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:22:12.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:12 smithi179 ceph-mon[35888]: pgmap v274: 129 pgs: 129 active+clean; 75 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 408 KiB/s wr, 66 op/s 2024-05-27T07:22:12.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:12 smithi179 ceph-mon[35888]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:22:12.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:12 smithi132 ceph-mon[35283]: pgmap v274: 129 pgs: 129 active+clean; 75 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 408 KiB/s wr, 66 op/s 2024-05-27T07:22:12.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:12 smithi132 ceph-mon[35283]: osdmap e115: 12 total, 12 up, 12 in 2024-05-27T07:22:13.141 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:13 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:22:13.142 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:12 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:22:13.142 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:12 smithi179 systemd[1]: Stopped Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:13.142 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:12 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 6.403s CPU time. 2024-05-27T07:22:13.142 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:13 smithi179 systemd[1]: Starting Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:22:13.212 INFO:tasks.workunit.client.0.smithi043.stdout: 1 44553 0.62 MB/sec execute 257 sec latency 59.339 ms 2024-05-27T07:22:13.400 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:13 smithi179 podman[72810]: 2024-05-27 07:22:13.285555301 +0000 UTC m=+0.037696993 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:22:13.400 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:13 smithi179 systemd[1]: Stopped Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:13.400 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:13 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 5.497s CPU time. 2024-05-27T07:22:13.401 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:13 smithi179 systemd[1]: Starting Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:22:13.689 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:13 smithi179 podman[72810]: 2024-05-27 07:22:13.410427389 +0000 UTC m=+0.162569142 container create 9fe19b34fd46080e52ea3e111b2951cd3900f08d5d0968aa30d8db22d3725e2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0) 2024-05-27T07:22:13.689 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:13 smithi179 podman[72810]: 2024-05-27 07:22:13.698692737 +0000 UTC m=+0.450834512 container init 9fe19b34fd46080e52ea3e111b2951cd3900f08d5d0968aa30d8db22d3725e2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T07:22:13.958 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:13 smithi179 podman[72864]: 2024-05-27 07:22:13.847065839 +0000 UTC m=+0.112242525 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:22:13.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:13 smithi179 podman[72810]: 2024-05-27 07:22:13.711352338 +0000 UTC m=+0.463494020 container start 9fe19b34fd46080e52ea3e111b2951cd3900f08d5d0968aa30d8db22d3725e2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5) 2024-05-27T07:22:13.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:13 smithi179 ceph-mds[72863]: -- 172.21.15.179:0/629651035 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55873f15fa40 con 0x55873f1a7a80 2024-05-27T07:22:13.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: starting mds.f at 2024-05-27T07:22:13.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:13 smithi179 bash[72810]: 9fe19b34fd46080e52ea3e111b2951cd3900f08d5d0968aa30d8db22d3725e2f 2024-05-27T07:22:13.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:13 smithi179 systemd[1]: Started Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:14.213 INFO:tasks.workunit.client.0.smithi043.stdout: 1 44663 0.61 MB/sec execute 258 sec latency 73.970 ms 2024-05-27T07:22:14.274 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:13 smithi179 podman[72864]: 2024-05-27 07:22:13.973110937 +0000 UTC m=+0.238287661 container create c4de5d9b4f10f6da4e4b8165bca8a62634bf72c1c0932db25a96baf201c525a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, ceph=True) 2024-05-27T07:22:14.274 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:14 smithi179 ceph-mon[35888]: pgmap v276: 129 pgs: 129 active+clean; 75 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 408 KiB/s wr, 66 op/s 2024-05-27T07:22:14.275 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:14 smithi179 ceph-mon[35888]: daemon mds.f restarted 2024-05-27T07:22:14.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:14 smithi043 ceph-mon[31531]: pgmap v276: 129 pgs: 129 active+clean; 75 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 408 KiB/s wr, 66 op/s 2024-05-27T07:22:14.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:14 smithi043 ceph-mon[31531]: daemon mds.f restarted 2024-05-27T07:22:14.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:14 smithi132 ceph-mon[35283]: pgmap v276: 129 pgs: 129 active+clean; 75 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 408 KiB/s wr, 66 op/s 2024-05-27T07:22:14.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:14 smithi132 ceph-mon[35283]: daemon mds.f restarted 2024-05-27T07:22:14.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:14 smithi179 podman[72864]: 2024-05-27 07:22:14.285048704 +0000 UTC m=+0.550225438 container init c4de5d9b4f10f6da4e4b8165bca8a62634bf72c1c0932db25a96baf201c525a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-27T07:22:14.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:14 smithi179 podman[72864]: 2024-05-27 07:22:14.291799894 +0000 UTC m=+0.556976583 container start c4de5d9b4f10f6da4e4b8165bca8a62634bf72c1c0932db25a96baf201c525a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:22:14.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:14 smithi179 bash[72864]: c4de5d9b4f10f6da4e4b8165bca8a62634bf72c1c0932db25a96baf201c525a3 2024-05-27T07:22:14.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:14 smithi179 systemd[1]: Started Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:14.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:14 smithi179 ceph-mds[72914]: -- 172.21.15.179:0/3974252244 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56210bbf1a40 con 0x56210bc39600 2024-05-27T07:22:14.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: starting mds.l at 2024-05-27T07:22:14.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1e9ac00700 time 2024-05-27T07:22:14.284332+0000 2024-05-27T07:22:14.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:14.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:14.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1eb0748ba6] 2024-05-27T07:22:14.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1eb0748dd1] 2024-05-27T07:22:14.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55870f4cb23d] 2024-05-27T07:22:14.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55870f4b6971] 2024-05-27T07:22:14.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 5: (SnapRealm::check_cache() const+0x13e) [0x55870f4b4fa0] 2024-05-27T07:22:14.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55870f4b5651] 2024-05-27T07:22:14.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55870f44a347] 2024-05-27T07:22:14.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55870f14687e] 2024-05-27T07:22:14.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55870f14be94] 2024-05-27T07:22:14.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55870f156969] 2024-05-27T07:22:14.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 11: (MDLog::_replay_thread()+0x1f38) [0x55870f4ef48c] 2024-05-27T07:22:14.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55870f4f89cb] 2024-05-27T07:22:14.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 13: (Thread::entry_wrapper()+0x43) [0x7f1eb071772d] 2024-05-27T07:22:14.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1eb0717749] 2024-05-27T07:22:14.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1eaf2bb1ca] 2024-05-27T07:22:14.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 16: clone() 2024-05-27T07:22:14.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.286+0000 7f1e9ac00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1e9ac00700 time 2024-05-27T07:22:14.284332+0000 2024-05-27T07:22:14.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:14.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:14.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1eb0748ba6] 2024-05-27T07:22:14.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1eb0748dd1] 2024-05-27T07:22:14.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55870f4cb23d] 2024-05-27T07:22:14.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55870f4b6971] 2024-05-27T07:22:14.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 5: (SnapRealm::check_cache() const+0x13e) [0x55870f4b4fa0] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55870f4b5651] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55870f44a347] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55870f14687e] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55870f14be94] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55870f156969] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 11: (MDLog::_replay_thread()+0x1f38) [0x55870f4ef48c] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55870f4f89cb] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 13: (Thread::entry_wrapper()+0x43) [0x7f1eb071772d] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1eb0717749] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1eaf2bb1ca] 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 16: clone() 2024-05-27T07:22:14.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: *** Caught signal (Aborted) ** 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: in thread 7f1e9ac00700 thread_name:md_log_replay 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1eaf2c5d20] 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2: gsignal() 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 3: abort() 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1eb0748ced] 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1eb0748dd1] 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55870f4cb23d] 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55870f4b6971] 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 8: (SnapRealm::check_cache() const+0x13e) [0x55870f4b4fa0] 2024-05-27T07:22:14.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55870f4b5651] 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55870f44a347] 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55870f14687e] 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55870f14be94] 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55870f156969] 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 14: (MDLog::_replay_thread()+0x1f38) [0x55870f4ef48c] 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55870f4f89cb] 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 16: (Thread::entry_wrapper()+0x43) [0x7f1eb071772d] 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1eb0717749] 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1eaf2bb1ca] 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 19: clone() 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.290+0000 7f1e9ac00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: in thread 7f1e9ac00700 thread_name:md_log_replay 2024-05-27T07:22:14.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1eaf2c5d20] 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2: gsignal() 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 3: abort() 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1eb0748ced] 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1eb0748dd1] 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55870f4cb23d] 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55870f4b6971] 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 8: (SnapRealm::check_cache() const+0x13e) [0x55870f4b4fa0] 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55870f4b5651] 2024-05-27T07:22:14.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55870f44a347] 2024-05-27T07:22:14.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55870f14687e] 2024-05-27T07:22:14.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55870f14be94] 2024-05-27T07:22:14.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55870f156969] 2024-05-27T07:22:14.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 14: (MDLog::_replay_thread()+0x1f38) [0x55870f4ef48c] 2024-05-27T07:22:14.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55870f4f89cb] 2024-05-27T07:22:14.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 16: (Thread::entry_wrapper()+0x43) [0x7f1eb071772d] 2024-05-27T07:22:14.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1eb0717749] 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1eaf2bb1ca] 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 19: clone() 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: -1> 2024-05-27T07:22:14.286+0000 7f1e9ac00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1e9ac00700 time 2024-05-27T07:22:14.284332+0000 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1eb0748ba6] 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1eb0748dd1] 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55870f4cb23d] 2024-05-27T07:22:14.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55870f4b6971] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 5: (SnapRealm::check_cache() const+0x13e) [0x55870f4b4fa0] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55870f4b5651] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55870f44a347] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55870f14687e] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55870f14be94] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55870f156969] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 11: (MDLog::_replay_thread()+0x1f38) [0x55870f4ef48c] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55870f4f89cb] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 13: (Thread::entry_wrapper()+0x43) [0x7f1eb071772d] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1eb0717749] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1eaf2bb1ca] 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 16: clone() 2024-05-27T07:22:14.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 0> 2024-05-27T07:22:14.290+0000 7f1e9ac00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: in thread 7f1e9ac00700 thread_name:md_log_replay 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1eaf2c5d20] 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2: gsignal() 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 3: abort() 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1eb0748ced] 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1eb0748dd1] 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55870f4cb23d] 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55870f4b6971] 2024-05-27T07:22:14.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 8: (SnapRealm::check_cache() const+0x13e) [0x55870f4b4fa0] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55870f4b5651] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55870f44a347] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55870f14687e] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55870f14be94] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55870f156969] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 14: (MDLog::_replay_thread()+0x1f38) [0x55870f4ef48c] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55870f4f89cb] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 16: (Thread::entry_wrapper()+0x43) [0x7f1eb071772d] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1eb0717749] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1eaf2bb1ca] 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 19: clone() 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:14.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: -1> 2024-05-27T07:22:14.286+0000 7f1e9ac00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f1e9ac00700 time 2024-05-27T07:22:14.284332+0000 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1eb0748ba6] 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1eb0748dd1] 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55870f4cb23d] 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55870f4b6971] 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 5: (SnapRealm::check_cache() const+0x13e) [0x55870f4b4fa0] 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55870f4b5651] 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55870f44a347] 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55870f14687e] 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55870f14be94] 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55870f156969] 2024-05-27T07:22:14.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 11: (MDLog::_replay_thread()+0x1f38) [0x55870f4ef48c] 2024-05-27T07:22:14.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55870f4f89cb] 2024-05-27T07:22:14.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 13: (Thread::entry_wrapper()+0x43) [0x7f1eb071772d] 2024-05-27T07:22:14.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 14: (Thread::_entry_func(void*)+0xd) [0x7f1eb0717749] 2024-05-27T07:22:14.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1eaf2bb1ca] 2024-05-27T07:22:14.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 16: clone() 2024-05-27T07:22:14.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 0> 2024-05-27T07:22:14.290+0000 7f1e9ac00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:14.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: in thread 7f1e9ac00700 thread_name:md_log_replay 2024-05-27T07:22:14.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1eaf2c5d20] 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2: gsignal() 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 3: abort() 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1eb0748ced] 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1eb0748dd1] 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55870f4cb23d] 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55870f4b6971] 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 8: (SnapRealm::check_cache() const+0x13e) [0x55870f4b4fa0] 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55870f4b5651] 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55870f44a347] 2024-05-27T07:22:14.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55870f14687e] 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55870f14be94] 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55870f156969] 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 14: (MDLog::_replay_thread()+0x1f38) [0x55870f4ef48c] 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55870f4f89cb] 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 16: (Thread::entry_wrapper()+0x43) [0x7f1eb071772d] 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 17: (Thread::_entry_func(void*)+0xd) [0x7f1eb0717749] 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1eaf2bb1ca] 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 19: clone() 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[72848]: 2024-05-27T07:22:14.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:14 smithi179 podman[72915]: 2024-05-27 07:22:14.410192643 +0000 UTC m=+0.032787841 container died 9fe19b34fd46080e52ea3e111b2951cd3900f08d5d0968aa30d8db22d3725e2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:22:15.213 INFO:tasks.workunit.client.0.smithi043.stdout: 1 44782 0.61 MB/sec execute 259 sec latency 76.805 ms 2024-05-27T07:22:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:15 smithi043 ceph-mon[31531]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:22:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:15 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/2130135734,v1:172.21.15.179:6833/2130135734] up:boot 2024-05-27T07:22:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:15 smithi043 ceph-mon[31531]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:15 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:15 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:15.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:15 smithi043 ceph-mon[31531]: daemon mds.l restarted 2024-05-27T07:22:15.458 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:15 smithi179 podman[72915]: 2024-05-27 07:22:15.265482989 +0000 UTC m=+0.888078179 container remove 9fe19b34fd46080e52ea3e111b2951cd3900f08d5d0968aa30d8db22d3725e2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:22:15.458 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:15 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:22:15.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:15 smithi179 ceph-mon[35888]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:22:15.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:15 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/2130135734,v1:172.21.15.179:6833/2130135734] up:boot 2024-05-27T07:22:15.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:15 smithi179 ceph-mon[35888]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:15.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:15 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:15.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:15 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:15.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:15 smithi179 ceph-mon[35888]: daemon mds.l restarted 2024-05-27T07:22:15.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:15 smithi132 ceph-mon[35283]: osdmap e116: 12 total, 12 up, 12 in 2024-05-27T07:22:15.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:15 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/2130135734,v1:172.21.15.179:6833/2130135734] up:boot 2024-05-27T07:22:15.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:15 smithi132 ceph-mon[35283]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:15.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:15 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:15.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:15 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:15.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:15 smithi132 ceph-mon[35283]: daemon mds.l restarted 2024-05-27T07:22:15.931 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:22:16.171 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:15 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:22:16.171 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:15 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.503s CPU time. 2024-05-27T07:22:16.173 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f41000700 time 2024-05-27T07:22:15.778597+0000 2024-05-27T07:22:16.173 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:16.173 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:16.173 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f56aecba6] 2024-05-27T07:22:16.173 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f56aecdd1] 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620da1a423d] 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620da18f971] 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620da18dfa0] 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620da18e651] 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620da123347] 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620d9e1f87e] 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620d9e24e94] 2024-05-27T07:22:16.174 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620d9e2f969] 2024-05-27T07:22:16.175 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620da1c848c] 2024-05-27T07:22:16.175 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620da1d19cb] 2024-05-27T07:22:16.175 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f56abb72d] 2024-05-27T07:22:16.175 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f56abb749] 2024-05-27T07:22:16.175 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f5565f1ca] 2024-05-27T07:22:16.175 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 16: clone() 2024-05-27T07:22:16.175 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: *** Caught signal (Aborted) ** 2024-05-27T07:22:16.175 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: in thread 7f6f41000700 thread_name:md_log_replay 2024-05-27T07:22:16.177 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:15.780+0000 7f6f41000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f41000700 time 2024-05-27T07:22:15.778597+0000 2024-05-27T07:22:16.177 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:16.177 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.177 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:16.177 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f56aecba6] 2024-05-27T07:22:16.177 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f56aecdd1] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620da1a423d] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620da18f971] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620da18dfa0] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620da18e651] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620da123347] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620d9e1f87e] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620d9e24e94] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620d9e2f969] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620da1c848c] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620da1d19cb] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f56abb72d] 2024-05-27T07:22:16.178 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f56abb749] 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f5565f1ca] 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 16: clone() 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f55669d20] 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2: gsignal() 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 3: abort() 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f56aecced] 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f56aecdd1] 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620da1a423d] 2024-05-27T07:22:16.179 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620da18f971] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620da18dfa0] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620da18e651] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620da123347] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620d9e1f87e] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620d9e24e94] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620d9e2f969] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620da1c848c] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620da1d19cb] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f56abb72d] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f56abb749] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f5565f1ca] 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 19: clone() 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:15.782+0000 7f6f41000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:16.180 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: in thread 7f6f41000700 thread_name:md_log_replay 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f55669d20] 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2: gsignal() 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 3: abort() 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f56aecced] 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f56aecdd1] 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620da1a423d] 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620da18f971] 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620da18dfa0] 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620da18e651] 2024-05-27T07:22:16.181 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620da123347] 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620d9e1f87e] 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620d9e24e94] 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620d9e2f969] 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620da1c848c] 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620da1d19cb] 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f56abb72d] 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f56abb749] 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f5565f1ca] 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 19: clone() 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:16.182 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: -1> 2024-05-27T07:22:15.780+0000 7f6f41000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f41000700 time 2024-05-27T07:22:15.778597+0000 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f56aecba6] 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f56aecdd1] 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620da1a423d] 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620da18f971] 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620da18dfa0] 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620da18e651] 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620da123347] 2024-05-27T07:22:16.183 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620d9e1f87e] 2024-05-27T07:22:16.184 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620d9e24e94] 2024-05-27T07:22:16.184 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620d9e2f969] 2024-05-27T07:22:16.184 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620da1c848c] 2024-05-27T07:22:16.184 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620da1d19cb] 2024-05-27T07:22:16.184 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f56abb72d] 2024-05-27T07:22:16.184 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f56abb749] 2024-05-27T07:22:16.184 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f5565f1ca] 2024-05-27T07:22:16.184 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 16: clone() 2024-05-27T07:22:16.184 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.184 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 0> 2024-05-27T07:22:15.782+0000 7f6f41000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:16.185 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: in thread 7f6f41000700 thread_name:md_log_replay 2024-05-27T07:22:16.185 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.185 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:16.185 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f55669d20] 2024-05-27T07:22:16.185 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2: gsignal() 2024-05-27T07:22:16.185 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 3: abort() 2024-05-27T07:22:16.185 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f56aecced] 2024-05-27T07:22:16.185 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f56aecdd1] 2024-05-27T07:22:16.185 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620da1a423d] 2024-05-27T07:22:16.186 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620da18f971] 2024-05-27T07:22:16.186 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620da18dfa0] 2024-05-27T07:22:16.186 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620da18e651] 2024-05-27T07:22:16.186 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620da123347] 2024-05-27T07:22:16.186 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620d9e1f87e] 2024-05-27T07:22:16.186 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620d9e24e94] 2024-05-27T07:22:16.187 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620d9e2f969] 2024-05-27T07:22:16.187 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620da1c848c] 2024-05-27T07:22:16.187 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620da1d19cb] 2024-05-27T07:22:16.187 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f56abb72d] 2024-05-27T07:22:16.187 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f56abb749] 2024-05-27T07:22:16.187 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f5565f1ca] 2024-05-27T07:22:16.188 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 19: clone() 2024-05-27T07:22:16.188 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:16.188 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.188 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: -1> 2024-05-27T07:22:15.780+0000 7f6f41000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f6f41000700 time 2024-05-27T07:22:15.778597+0000 2024-05-27T07:22:16.188 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:16.188 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.188 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:16.188 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f56aecba6] 2024-05-27T07:22:16.188 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f56aecdd1] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620da1a423d] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5620da18f971] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 5: (SnapRealm::check_cache() const+0x13e) [0x5620da18dfa0] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620da18e651] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620da123347] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620d9e1f87e] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620d9e24e94] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5620d9e2f969] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 11: (MDLog::_replay_thread()+0x1f38) [0x5620da1c848c] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5620da1d19cb] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 13: (Thread::entry_wrapper()+0x43) [0x7f6f56abb72d] 2024-05-27T07:22:16.189 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 14: (Thread::_entry_func(void*)+0xd) [0x7f6f56abb749] 2024-05-27T07:22:16.190 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f6f5565f1ca] 2024-05-27T07:22:16.190 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 16: clone() 2024-05-27T07:22:16.190 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.190 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 0> 2024-05-27T07:22:15.782+0000 7f6f41000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:16.190 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: in thread 7f6f41000700 thread_name:md_log_replay 2024-05-27T07:22:16.190 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.190 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:16.190 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f55669d20] 2024-05-27T07:22:16.190 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2: gsignal() 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 3: abort() 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f56aecced] 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f56aecdd1] 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5620da1a423d] 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5620da18f971] 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 8: (SnapRealm::check_cache() const+0x13e) [0x5620da18dfa0] 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5620da18e651] 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5620da123347] 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5620d9e1f87e] 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5620d9e24e94] 2024-05-27T07:22:16.191 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5620d9e2f969] 2024-05-27T07:22:16.192 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 14: (MDLog::_replay_thread()+0x1f38) [0x5620da1c848c] 2024-05-27T07:22:16.192 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5620da1d19cb] 2024-05-27T07:22:16.192 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 16: (Thread::entry_wrapper()+0x43) [0x7f6f56abb72d] 2024-05-27T07:22:16.192 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 17: (Thread::_entry_func(void*)+0xd) [0x7f6f56abb749] 2024-05-27T07:22:16.192 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f6f5565f1ca] 2024-05-27T07:22:16.192 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 19: clone() 2024-05-27T07:22:16.192 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:16.192 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[72898]: 2024-05-27T07:22:16.192 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:15 smithi179 podman[72988]: 2024-05-27 07:22:15.89110498 +0000 UTC m=+0.030659694 container died c4de5d9b4f10f6da4e4b8165bca8a62634bf72c1c0932db25a96baf201c525a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T07:22:16.213 INFO:tasks.workunit.client.0.smithi043.stdout: 1 44934 0.61 MB/sec execute 260 sec latency 58.584 ms 2024-05-27T07:22:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:16.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:16 smithi043 ceph-mon[31531]: pgmap v278: 129 pgs: 129 active+clean; 75 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 126 KiB/s wr, 25 op/s 2024-05-27T07:22:16.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:16 smithi043 ceph-mon[31531]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:22:16.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:16 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/2196267080,v1:172.21.15.179:6833/2196267080] up:boot 2024-05-27T07:22:16.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:16 smithi043 ceph-mon[31531]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:16.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:16 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:16.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:16 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:16.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:16 smithi179 ceph-mon[35888]: pgmap v278: 129 pgs: 129 active+clean; 75 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 126 KiB/s wr, 25 op/s 2024-05-27T07:22:16.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:16 smithi179 ceph-mon[35888]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:22:16.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:16 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/2196267080,v1:172.21.15.179:6833/2196267080] up:boot 2024-05-27T07:22:16.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:16 smithi179 ceph-mon[35888]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:16.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:16 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:16.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:16 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:16.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:16 smithi132 ceph-mon[35283]: pgmap v278: 129 pgs: 129 active+clean; 75 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 126 KiB/s wr, 25 op/s 2024-05-27T07:22:16.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:16 smithi132 ceph-mon[35283]: osdmap e117: 12 total, 12 up, 12 in 2024-05-27T07:22:16.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:16 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/2196267080,v1:172.21.15.179:6833/2196267080] up:boot 2024-05-27T07:22:16.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:16 smithi132 ceph-mon[35283]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:16.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:16 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:16.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:16 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:17.195 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:16 smithi179 podman[72988]: 2024-05-27 07:22:16.737607238 +0000 UTC m=+0.877161982 container remove c4de5d9b4f10f6da4e4b8165bca8a62634bf72c1c0932db25a96baf201c525a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-27T07:22:17.195 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:16 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:22:17.213 INFO:tasks.workunit.client.0.smithi043.stdout: 1 45027 0.61 MB/sec execute 261 sec latency 55.547 ms 2024-05-27T07:22:17.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:17 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:22:17.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:17 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.347s CPU time. 2024-05-27T07:22:18.213 INFO:tasks.workunit.client.0.smithi043.stdout: 1 45148 0.61 MB/sec execute 262 sec latency 55.144 ms 2024-05-27T07:22:18.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:18 smithi043 ceph-mon[31531]: pgmap v280: 129 pgs: 129 active+clean; 75 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 123 KiB/s wr, 27 op/s 2024-05-27T07:22:18.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:18 smithi179 ceph-mon[35888]: pgmap v280: 129 pgs: 129 active+clean; 75 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 123 KiB/s wr, 27 op/s 2024-05-27T07:22:18.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:18 smithi132 ceph-mon[35283]: pgmap v280: 129 pgs: 129 active+clean; 75 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 123 KiB/s wr, 27 op/s 2024-05-27T07:22:18.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.909+0000 7f254f200700 1 -- 172.21.15.43:0/3008372429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25501032e0 msgr2=0x7f25501036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:18.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.909+0000 7f254f200700 1 --2- 172.21.15.43:0/3008372429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25501032e0 0x7f25501036f0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f25400099c0 tx=0x7f2540009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.909+0000 7f254f200700 1 -- 172.21.15.43:0/3008372429 shutdown_connections 2024-05-27T07:22:18.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.909+0000 7f254f200700 1 --2- 172.21.15.43:0/3008372429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2550108560 0x7f255010fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.909+0000 7f254f200700 1 --2- 172.21.15.43:0/3008372429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25501032e0 0x7f25501036f0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.909+0000 7f254f200700 1 --2- 172.21.15.43:0/3008372429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2550107c10 0x7f2550108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.909+0000 7f254f200700 1 -- 172.21.15.43:0/3008372429 >> 172.21.15.43:0/3008372429 conn(0x7f2550078af0 msgr2=0x7f2550078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:18.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.909+0000 7f254f200700 1 -- 172.21.15.43:0/3008372429 shutdown_connections 2024-05-27T07:22:18.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.909+0000 7f254f200700 1 -- 172.21.15.43:0/3008372429 wait complete. 2024-05-27T07:22:18.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.910+0000 7f254f200700 1 Processor -- start 2024-05-27T07:22:18.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.911+0000 7f254f200700 1 -- start start 2024-05-27T07:22:18.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.911+0000 7f254f200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25501032e0 0x7f25501a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.911+0000 7f254f200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2550107c10 0x7f25501a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.911+0000 7f254f200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2550108560 0x7f255019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.911+0000 7f254f200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2550113040 con 0x7f2550107c10 2024-05-27T07:22:18.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.911+0000 7f254f200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2550112ec0 con 0x7f25501032e0 2024-05-27T07:22:18.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.911+0000 7f254f200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f25501131c0 con 0x7f2550108560 2024-05-27T07:22:18.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.911+0000 7f254d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2550107c10 0x7f25501a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.911+0000 7f254d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2550107c10 0x7f25501a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48386/0 (socket says 172.21.15.43:48386) 2024-05-27T07:22:18.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.911+0000 7f254d400700 1 -- 172.21.15.43:0/446596004 learned_addr learned my addr 172.21.15.43:0/446596004 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:18.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.912+0000 7f254de00700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25501032e0 0x7f25501a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.912+0000 7f254e800700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2550108560 0x7f255019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.912+0000 7f254d400700 1 -- 172.21.15.43:0/446596004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2550108560 msgr2=0x7f255019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:18.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.912+0000 7f254d400700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2550108560 0x7f255019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.912+0000 7f254d400700 1 -- 172.21.15.43:0/446596004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25501032e0 msgr2=0x7f25501a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:18.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.912+0000 7f254d400700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25501032e0 0x7f25501a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.912+0000 7f254d400700 1 -- 172.21.15.43:0/446596004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255019afb0 con 0x7f2550107c10 2024-05-27T07:22:18.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.912+0000 7f254d400700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2550107c10 0x7f25501a0c90 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f253c00c9e0 tx=0x7f253c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:18.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.913+0000 7f2546a00700 1 -- 172.21.15.43:0/446596004 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c007aa0 con 0x7f2550107c10 2024-05-27T07:22:18.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.913+0000 7f2546a00700 1 -- 172.21.15.43:0/446596004 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f253c007c40 con 0x7f2550107c10 2024-05-27T07:22:18.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.913+0000 7f254f200700 1 -- 172.21.15.43:0/446596004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f255019b2a0 con 0x7f2550107c10 2024-05-27T07:22:18.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.913+0000 7f254f200700 1 -- 172.21.15.43:0/446596004 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25501a75e0 con 0x7f2550107c10 2024-05-27T07:22:18.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.913+0000 7f2546a00700 1 -- 172.21.15.43:0/446596004 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c018660 con 0x7f2550107c10 2024-05-27T07:22:18.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.918+0000 7f2546a00700 1 -- 172.21.15.43:0/446596004 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f253c004830 con 0x7f2550107c10 2024-05-27T07:22:18.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.918+0000 7f2563cf6b80 1 Processor -- start 2024-05-27T07:22:18.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.919+0000 7f2563cf6b80 1 -- start start 2024-05-27T07:22:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.919+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.919+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bac0e40 0x55a47bc43df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.919+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc54ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.919+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a47bb6de40 con 0x55a47bbac9f0 2024-05-27T07:22:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.919+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a47bb5d9e0 con 0x55a47bc44330 2024-05-27T07:22:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.919+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a47bb61a20 con 0x55a47bac0e40 2024-05-27T07:22:18.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.919+0000 7f254e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.920+0000 7f254e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc438b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48388/0 (socket says 172.21.15.43:48388) 2024-05-27T07:22:18.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.920+0000 7f254e800700 1 -- 172.21.15.43:0/3000732517 learned_addr learned my addr 172.21.15.43:0/3000732517 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:18.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.920+0000 7f254d400700 1 --2- 172.21.15.43:0/3000732517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc54ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.920+0000 7f254de00700 1 --2- 172.21.15.43:0/3000732517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bac0e40 0x55a47bc43df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.920+0000 7f254e800700 1 -- 172.21.15.43:0/3000732517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bac0e40 msgr2=0x55a47bc43df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:18.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.920+0000 7f254e800700 1 --2- 172.21.15.43:0/3000732517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bac0e40 0x55a47bc43df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.920+0000 7f254e800700 1 -- 172.21.15.43:0/3000732517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 msgr2=0x55a47bc54ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:18.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.920+0000 7f254e800700 1 --2- 172.21.15.43:0/3000732517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc54ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.920+0000 7f2546a00700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f25340b2a70 0x7f25340b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.920+0000 7f254e800700 1 -- 172.21.15.43:0/3000732517 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a47bb26420 con 0x55a47bbac9f0 2024-05-27T07:22:18.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.921+0000 7f2546a00700 1 -- 172.21.15.43:0/446596004 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9408+0+0 (secure 0 0 0) 0x7f253c0e2850 con 0x7f2550107c10 2024-05-27T07:22:18.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.921+0000 7f254e800700 1 --2- 172.21.15.43:0/3000732517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc438b0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2548009d20 tx=0x7f25480077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a022efd02ad37393 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:18.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.921+0000 7f254de00700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f25340b2a70 0x7f25340b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.921+0000 7f253b400700 1 -- 172.21.15.43:0/3000732517 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548004bb0 con 0x55a47bbac9f0 2024-05-27T07:22:18.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.922+0000 7f253b400700 1 -- 172.21.15.43:0/3000732517 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2548004d50 con 0x55a47bbac9f0 2024-05-27T07:22:18.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.922+0000 7f253b400700 1 -- 172.21.15.43:0/3000732517 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548014670 con 0x55a47bbac9f0 2024-05-27T07:22:18.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.923+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3000732517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 msgr2=0x55a47bc438b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:18.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.923+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/3000732517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc438b0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2548009d20 tx=0x7f25480077e0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.923+0000 7f254de00700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f25340b2a70 0x7f25340b4f80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2540009990 tx=0x7f2540005fb0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:18.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.923+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3000732517 shutdown_connections 2024-05-27T07:22:18.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.923+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/3000732517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc54ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.923+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/3000732517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc438b0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.923+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/3000732517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bac0e40 0x55a47bc43df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.923+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3000732517 >> 172.21.15.43:0/3000732517 conn(0x55a47bb5a390 msgr2=0x55a47bc3b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:18.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.923+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3000732517 shutdown_connections 2024-05-27T07:22:18.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.923+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3000732517 wait complete. 2024-05-27T07:22:18.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.924+0000 7f2563cf6b80 1 -- start start 2024-05-27T07:22:18.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.924+0000 7f2563cf6b80 1 Processor -- start 2024-05-27T07:22:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.924+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bac0e40 0x55a47bc52430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.924+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bbac9f0 0x55a47bc52970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.925+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc52eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.925+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a47ba63fa0 con 0x55a47bac0e40 2024-05-27T07:22:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.925+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a47bafe760 con 0x55a47bc44330 2024-05-27T07:22:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.925+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a47bc55620 con 0x55a47bbac9f0 2024-05-27T07:22:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.925+0000 7f2563cf6b80 10 client.0 _ll_register_callbacks cb 0x55a47bb07c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.925+0000 7f254de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bac0e40 0x55a47bc52430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.925+0000 7f254de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bac0e40 0x55a47bc52430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48394/0 (socket says 172.21.15.43:48394) 2024-05-27T07:22:18.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.925+0000 7f254de00700 1 -- 172.21.15.43:0/4283839445 learned_addr learned my addr 172.21.15.43:0/4283839445 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.925+0000 7f254e800700 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc52eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.925+0000 7f254d400700 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bbac9f0 0x55a47bc52970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.926+0000 7f254de00700 1 -- 172.21.15.43:0/4283839445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bbac9f0 msgr2=0x55a47bc52970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.926+0000 7f254de00700 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bbac9f0 0x55a47bc52970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.926+0000 7f254de00700 1 -- 172.21.15.43:0/4283839445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 msgr2=0x55a47bc52eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.926+0000 7f254de00700 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc52eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.926+0000 7f254de00700 1 -- 172.21.15.43:0/4283839445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a47baad8d0 con 0x55a47bac0e40 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.926+0000 7f254d400700 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bbac9f0 0x55a47bc52970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.926+0000 7f254de00700 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bac0e40 0x55a47bc52430 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f2540015000 tx=0x7f2540009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30d4f66f17d1b91 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.926+0000 7f252fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.927+0000 7f252fe00700 1 -- 172.21.15.43:0/4283839445 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540035060 con 0x55a47bac0e40 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.927+0000 7f2563cf6b80 1 -- 172.21.15.43:0/4283839445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2548004d50 con 0x55a47bac0e40 2024-05-27T07:22:18.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.927+0000 7f252fe00700 1 -- 172.21.15.43:0/4283839445 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9408+0+0 (secure 0 0 0) 0x7f2540003c90 con 0x55a47bac0e40 2024-05-27T07:22:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.927+0000 7f252fe00700 1 -- 172.21.15.43:0/4283839445 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f254003c050 con 0x55a47bac0e40 2024-05-27T07:22:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.927+0000 7f2563cf6b80 10 client.16665 fetch_fsmap learned FSMap version 83 2024-05-27T07:22:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.927+0000 7f2563cf6b80 1 -- 172.21.15.43:0/4283839445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=4+}) -- 0x55a47bb2e060 con 0x55a47bac0e40 2024-05-27T07:22:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.928+0000 7f252fe00700 1 -- 172.21.15.43:0/4283839445 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2548004d50 con 0x55a47bac0e40 2024-05-27T07:22:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.928+0000 7f252fe00700 1 -- 172.21.15.43:0/4283839445 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 83) ==== 5966+0+0 (secure 0 0 0) 0x7f254000f460 con 0x55a47bac0e40 2024-05-27T07:22:18.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.928+0000 7f2563cf6b80 10 client.16665 fetch_fsmap finished waiting for FSMap version 83 2024-05-27T07:22:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.928+0000 7f2563cf6b80 10 client.16665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.928+0000 7f2563cf6b80 20 client.16665 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.928+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a47bc628e0 0x55a47bc64d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.928+0000 7f2563cf6b80 4 client.16665 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.928+0000 7f2563cf6b80 1 -- 172.21.15.43:0/4283839445 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a47bb6fd90 con 0x55a47bc628e0 2024-05-27T07:22:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.929+0000 7f254d400700 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a47bc628e0 0x55a47bc64d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.930+0000 7f254d400700 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a47bc628e0 0x55a47bc64d90 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a6baed11d97d4f1 server_cookie=b86155f71b2308b7 in_seq=0 out_seq=0 2024-05-27T07:22:18.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.930+0000 7f252fe00700 10 client.16665 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:18.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.932+0000 7f252fe00700 1 -- 172.21.15.43:0/4283839445 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a47bb6fd90 con 0x55a47bc628e0 2024-05-27T07:22:18.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.932+0000 7f252fe00700 10 client.16665 handle_command_reply: tid=0 2024-05-27T07:22:18.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.932+0000 7f2563cf6b80 1 client.16665 shutdown 2024-05-27T07:22:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.932+0000 7f2563cf6b80 1 -- 172.21.15.43:0/4283839445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bac0e40 msgr2=0x55a47bc52430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.932+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bac0e40 0x55a47bc52430 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f2540015000 tx=0x7f2540009310 comp rx=0 tx=0).stop 2024-05-27T07:22:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.933+0000 7f2563cf6b80 1 -- 172.21.15.43:0/4283839445 shutdown_connections 2024-05-27T07:22:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.933+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc52eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.933+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a47bc628e0 0x55a47bc64d90 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.933+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bac0e40 0x55a47bc52430 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.933+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/4283839445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bbac9f0 0x55a47bc52970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.933+0000 7f2563cf6b80 1 -- 172.21.15.43:0/4283839445 >> 172.21.15.43:0/4283839445 conn(0x55a47bb5a390 msgr2=0x55a47bc437f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.933+0000 7f252fe00700 0 client.16665 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:18.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.933+0000 7f2563cf6b80 1 -- 172.21.15.43:0/4283839445 shutdown_connections 2024-05-27T07:22:18.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.933+0000 7f2563cf6b80 1 -- 172.21.15.43:0/4283839445 wait complete. 2024-05-27T07:22:18.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.933+0000 7f2563cf6b80 20 client.16665 trim_cache size 0 max 16384 2024-05-27T07:22:18.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.957+0000 7f2563cf6b80 1 Processor -- start 2024-05-27T07:22:18.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.957+0000 7f2563cf6b80 1 -- start start 2024-05-27T07:22:18.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.958+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bac0e40 0x55a47bc3b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.958+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc3ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.958+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bc44330 0x55a47bc4fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.958+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f253000f8e0 con 0x55a47bbac9f0 2024-05-27T07:22:18.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.958+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f253000ffb0 con 0x55a47bac0e40 2024-05-27T07:22:18.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.958+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f253000d6d0 con 0x55a47bc44330 2024-05-27T07:22:18.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.958+0000 7f254d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc3ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.958+0000 7f254d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc3ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48410/0 (socket says 172.21.15.43:48410) 2024-05-27T07:22:18.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.958+0000 7f254d400700 1 -- 172.21.15.43:0/2203693489 learned_addr learned my addr 172.21.15.43:0/2203693489 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.959+0000 7f254de00700 1 --2- 172.21.15.43:0/2203693489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bac0e40 0x55a47bc3b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.959+0000 7f254e800700 1 --2- 172.21.15.43:0/2203693489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bc44330 0x55a47bc4fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.959+0000 7f254d400700 1 -- 172.21.15.43:0/2203693489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bc44330 msgr2=0x55a47bc4fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.959+0000 7f254d400700 1 --2- 172.21.15.43:0/2203693489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bc44330 0x55a47bc4fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.959+0000 7f254d400700 1 -- 172.21.15.43:0/2203693489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bac0e40 msgr2=0x55a47bc3b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.959+0000 7f254d400700 1 --2- 172.21.15.43:0/2203693489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bac0e40 0x55a47bc3b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.959+0000 7f254d400700 1 -- 172.21.15.43:0/2203693489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a47bc37490 con 0x55a47bbac9f0 2024-05-27T07:22:18.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.960+0000 7f254d400700 1 --2- 172.21.15.43:0/2203693489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc3ff80 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f253c0a5800 tx=0x7f253c0e4ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50b783d53db533d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:18.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.960+0000 7f253aa00700 1 -- 172.21.15.43:0/2203693489 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c0a8ef0 con 0x55a47bbac9f0 2024-05-27T07:22:18.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.960+0000 7f253aa00700 1 -- 172.21.15.43:0/2203693489 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f253c0a9290 con 0x55a47bbac9f0 2024-05-27T07:22:18.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.960+0000 7f253aa00700 1 -- 172.21.15.43:0/2203693489 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c0dfe70 con 0x55a47bbac9f0 2024-05-27T07:22:18.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.961+0000 7f2563cf6b80 1 -- 172.21.15.43:0/2203693489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 msgr2=0x55a47bc3ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:18.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.961+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/2203693489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc3ff80 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f253c0a5800 tx=0x7f253c0e4ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.962+0000 7f2563cf6b80 1 -- 172.21.15.43:0/2203693489 shutdown_connections 2024-05-27T07:22:18.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.962+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/2203693489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bc44330 0x55a47bc4fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.962+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/2203693489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bac0e40 0x55a47bc3b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.962+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/2203693489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc3ff80 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.962+0000 7f2563cf6b80 1 -- 172.21.15.43:0/2203693489 >> 172.21.15.43:0/2203693489 conn(0x55a47bb5a390 msgr2=0x55a47bc3b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:18.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.962+0000 7f2563cf6b80 1 -- 172.21.15.43:0/2203693489 shutdown_connections 2024-05-27T07:22:18.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.962+0000 7f2563cf6b80 1 -- 172.21.15.43:0/2203693489 wait complete. 2024-05-27T07:22:18.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.963+0000 7f2563cf6b80 1 -- start start 2024-05-27T07:22:18.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.963+0000 7f2563cf6b80 1 Processor -- start 2024-05-27T07:22:18.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.963+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bac0e40 0x55a47bc69540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.963+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc69a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.963+0000 7f2563cf6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc59910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f253000dda0 con 0x55a47bbac9f0 2024-05-27T07:22:18.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f253000f210 con 0x55a47bc44330 2024-05-27T07:22:18.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f2563cf6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f253000e470 con 0x55a47bac0e40 2024-05-27T07:22:18.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f2563cf6b80 10 client.0 _ll_register_callbacks cb 0x55a47bb07c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:18.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f254d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc69a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f254d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc69a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48412/0 (socket says 172.21.15.43:48412) 2024-05-27T07:22:18.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f254d400700 1 -- 172.21.15.43:0/3190526460 learned_addr learned my addr 172.21.15.43:0/3190526460 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:18.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f254de00700 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bac0e40 0x55a47bc69540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f254e800700 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc59910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f254d400700 1 -- 172.21.15.43:0/3190526460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bac0e40 msgr2=0x55a47bc69540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:18.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f254d400700 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bac0e40 0x55a47bc69540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f254d400700 1 -- 172.21.15.43:0/3190526460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 msgr2=0x55a47bc59910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:18.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f254d400700 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc59910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.964+0000 7f254d400700 1 -- 172.21.15.43:0/3190526460 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a47bc6a280 con 0x55a47bbac9f0 2024-05-27T07:22:18.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.965+0000 7f254e800700 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc59910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:18.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.965+0000 7f254d400700 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc69a80 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f253c00c9b0 tx=0x7f253c0e82a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d69ee2cd581987c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:18.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.965+0000 7f252fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:18.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.965+0000 7f252fe00700 1 -- 172.21.15.43:0/3190526460 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c0e2690 con 0x55a47bbac9f0 2024-05-27T07:22:18.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.965+0000 7f252fe00700 1 -- 172.21.15.43:0/3190526460 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9408+0+0 (secure 0 0 0) 0x7f253c0dfe70 con 0x55a47bbac9f0 2024-05-27T07:22:18.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.965+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3190526460 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f253c0a9290 con 0x55a47bbac9f0 2024-05-27T07:22:18.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.965+0000 7f252fe00700 1 -- 172.21.15.43:0/3190526460 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f253c0adb30 con 0x55a47bbac9f0 2024-05-27T07:22:18.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.966+0000 7f2563cf6b80 10 client.16671 fetch_fsmap learned FSMap version 83 2024-05-27T07:22:18.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.966+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3190526460 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=4+}) -- 0x55a47bc5adf0 con 0x55a47bbac9f0 2024-05-27T07:22:18.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.966+0000 7f252fe00700 1 -- 172.21.15.43:0/3190526460 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f253c0a9290 con 0x55a47bbac9f0 2024-05-27T07:22:18.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.966+0000 7f252fe00700 1 -- 172.21.15.43:0/3190526460 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 83) ==== 5966+0+0 (secure 0 0 0) 0x7f253c0a99e0 con 0x55a47bbac9f0 2024-05-27T07:22:18.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.966+0000 7f2563cf6b80 10 client.16671 fetch_fsmap finished waiting for FSMap version 83 2024-05-27T07:22:18.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.966+0000 7f2563cf6b80 10 client.16671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:18.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.966+0000 7f2563cf6b80 20 client.16671 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:18.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.966+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a47bc5b2c0 0x55a47bccf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:18.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.966+0000 7f2563cf6b80 4 client.16671 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:22:18.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.966+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3190526460 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a47bb69570 con 0x55a47bc5b2c0 2024-05-27T07:22:18.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.967+0000 7f254de00700 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a47bc5b2c0 0x55a47bccf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:18.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.968+0000 7f254de00700 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a47bc5b2c0 0x55a47bccf430 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e3960df992b0e98 server_cookie=12051a63d1fb210b in_seq=0 out_seq=0 2024-05-27T07:22:18.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.968+0000 7f252fe00700 10 client.16671 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:18.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f252fe00700 1 -- 172.21.15.43:0/3190526460 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a47bb69570 con 0x55a47bc5b2c0 2024-05-27T07:22:18.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f252fe00700 10 client.16671 handle_command_reply: tid=0 2024-05-27T07:22:18.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f2563cf6b80 1 client.16671 shutdown 2024-05-27T07:22:18.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3190526460 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 msgr2=0x55a47bc69a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:18.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc69a80 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f253c00c9b0 tx=0x7f253c0e82a0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3190526460 shutdown_connections 2024-05-27T07:22:18.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a47bc44330 0x55a47bc59910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a47bac0e40 0x55a47bc69540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a47bc5b2c0 0x55a47bccf430 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f2563cf6b80 1 --2- 172.21.15.43:0/3190526460 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a47bbac9f0 0x55a47bc69a80 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.969+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3190526460 >> 172.21.15.43:0/3190526460 conn(0x55a47bb5a390 msgr2=0x55a47bc454f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:18.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.970+0000 7f252fe00700 0 client.16671 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:18.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.970+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3190526460 shutdown_connections 2024-05-27T07:22:18.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.970+0000 7f2563cf6b80 1 -- 172.21.15.43:0/3190526460 wait complete. 2024-05-27T07:22:18.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.970+0000 7f2563cf6b80 20 client.16671 trim_cache size 0 max 16384 2024-05-27T07:22:18.971 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:22:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:22:18.972 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:22:18.972 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:22:18.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 -- 172.21.15.43:0/446596004 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f25340b2a70 msgr2=0x7f25340b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:18.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f25340b2a70 0x7f25340b4f80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2540009990 tx=0x7f2540005fb0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 -- 172.21.15.43:0/446596004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2550107c10 msgr2=0x7f25501a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:18.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2550107c10 0x7f25501a0c90 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f253c00c9e0 tx=0x7f253c00ccf0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 -- 172.21.15.43:0/446596004 shutdown_connections 2024-05-27T07:22:18.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f25340b2a70 0x7f25340b4f80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2550108560 0x7f255019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25501032e0 0x7f25501a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 --2- 172.21.15.43:0/446596004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2550107c10 0x7f25501a0c90 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:18.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 -- 172.21.15.43:0/446596004 >> 172.21.15.43:0/446596004 conn(0x7f2550078af0 msgr2=0x7f2550101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:18.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 -- 172.21.15.43:0/446596004 shutdown_connections 2024-05-27T07:22:18.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:18.971+0000 7f254f200700 1 -- 172.21.15.43:0/446596004 wait complete. 2024-05-27T07:22:19.213 INFO:tasks.workunit.client.0.smithi043.stdout: 1 45342 0.62 MB/sec execute 263 sec latency 51.143 ms 2024-05-27T07:22:19.810 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:22:19.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:22:20.213 INFO:tasks.workunit.client.0.smithi043.stdout: 1 45538 0.63 MB/sec execute 264 sec latency 50.307 ms 2024-05-27T07:22:20.281 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:20.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:20 smithi043 ceph-mon[31531]: pgmap v281: 129 pgs: 129 active+clean; 75 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 98 KiB/s wr, 22 op/s 2024-05-27T07:22:20.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:20 smithi179 ceph-mon[35888]: pgmap v281: 129 pgs: 129 active+clean; 75 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 98 KiB/s wr, 22 op/s 2024-05-27T07:22:20.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:20 smithi132 ceph-mon[35283]: pgmap v281: 129 pgs: 129 active+clean; 75 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 98 KiB/s wr, 22 op/s 2024-05-27T07:22:21.213 INFO:tasks.workunit.client.0.smithi043.stdout: 1 45783 0.63 MB/sec execute 265 sec latency 50.784 ms 2024-05-27T07:22:22.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 45889 0.63 MB/sec execute 266 sec latency 74.037 ms 2024-05-27T07:22:22.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:22 smithi043 ceph-mon[31531]: pgmap v282: 129 pgs: 129 active+clean; 78 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 617 KiB/s wr, 46 op/s 2024-05-27T07:22:22.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:22 smithi179 ceph-mon[35888]: pgmap v282: 129 pgs: 129 active+clean; 78 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 617 KiB/s wr, 46 op/s 2024-05-27T07:22:22.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:22 smithi132 ceph-mon[35283]: pgmap v282: 129 pgs: 129 active+clean; 78 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 617 KiB/s wr, 46 op/s 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.763+0000 7fd928a00700 1 -- 172.21.15.43:0/3204197064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd924107c10 msgr2=0x7fd924108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.763+0000 7fd928a00700 1 --2- 172.21.15.43:0/3204197064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd924107c10 0x7fd924108020 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd9100099c0 tx=0x7fd910009cd0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.763+0000 7fd928a00700 1 -- 172.21.15.43:0/3204197064 shutdown_connections 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.763+0000 7fd928a00700 1 --2- 172.21.15.43:0/3204197064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd924108560 0x7fd92410fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.763+0000 7fd928a00700 1 --2- 172.21.15.43:0/3204197064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9241032e0 0x7fd9241036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.763+0000 7fd928a00700 1 --2- 172.21.15.43:0/3204197064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd924107c10 0x7fd924108020 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.763+0000 7fd928a00700 1 -- 172.21.15.43:0/3204197064 >> 172.21.15.43:0/3204197064 conn(0x7fd924078af0 msgr2=0x7fd924078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.763+0000 7fd928a00700 1 -- 172.21.15.43:0/3204197064 shutdown_connections 2024-05-27T07:22:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.763+0000 7fd928a00700 1 -- 172.21.15.43:0/3204197064 wait complete. 2024-05-27T07:22:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.765+0000 7fd928a00700 1 Processor -- start 2024-05-27T07:22:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.765+0000 7fd928a00700 1 -- start start 2024-05-27T07:22:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.765+0000 7fd928a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9241032e0 0x7fd92419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.765+0000 7fd928a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd924107c10 0x7fd92419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.765+0000 7fd928a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd924108560 0x7fd9241a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.765+0000 7fd928a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd924113070 con 0x7fd924108560 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.765+0000 7fd928a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd924112ef0 con 0x7fd924107c10 2024-05-27T07:22:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.765+0000 7fd928a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd9241131f0 con 0x7fd9241032e0 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.766+0000 7fd923e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd924108560 0x7fd9241a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.766+0000 7fd923e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd924108560 0x7fd9241a15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48432/0 (socket says 172.21.15.43:48432) 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.766+0000 7fd923e00700 1 -- 172.21.15.43:0/2881117743 learned_addr learned my addr 172.21.15.43:0/2881117743 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.766+0000 7fd922a00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd924107c10 0x7fd92419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.766+0000 7fd923400700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9241032e0 0x7fd92419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.767+0000 7fd923e00700 1 -- 172.21.15.43:0/2881117743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9241032e0 msgr2=0x7fd92419cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.767+0000 7fd923e00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9241032e0 0x7fd92419cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.767+0000 7fd923e00700 1 -- 172.21.15.43:0/2881117743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd924107c10 msgr2=0x7fd92419d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.767+0000 7fd923e00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd924107c10 0x7fd92419d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.767+0000 7fd923e00700 1 -- 172.21.15.43:0/2881117743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9241a1c60 con 0x7fd924108560 2024-05-27T07:22:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.767+0000 7fd923e00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd924108560 0x7fd9241a15c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fd91c007ee0 tx=0x7fd91c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.767+0000 7fd91be00700 1 -- 172.21.15.43:0/2881117743 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c004bb0 con 0x7fd924108560 2024-05-27T07:22:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.767+0000 7fd91be00700 1 -- 172.21.15.43:0/2881117743 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd91c004d50 con 0x7fd924108560 2024-05-27T07:22:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.767+0000 7fd928a00700 1 -- 172.21.15.43:0/2881117743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9241a1f50 con 0x7fd924108560 2024-05-27T07:22:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.768+0000 7fd91be00700 1 -- 172.21.15.43:0/2881117743 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c0058b0 con 0x7fd924108560 2024-05-27T07:22:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.768+0000 7fd928a00700 1 -- 172.21.15.43:0/2881117743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9241a9870 con 0x7fd924108560 2024-05-27T07:22:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.769+0000 7fd928a00700 1 -- 172.21.15.43:0/2881117743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd904005330 con 0x7fd924108560 2024-05-27T07:22:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.773+0000 7fd91be00700 1 -- 172.21.15.43:0/2881117743 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd91c01a020 con 0x7fd924108560 2024-05-27T07:22:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.775+0000 7fd91be00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9080b2a70 0x7fd9080b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.775+0000 7fd91be00700 1 -- 172.21.15.43:0/2881117743 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9408+0+0 (secure 0 0 0) 0x7fd91c0e24f0 con 0x7fd924108560 2024-05-27T07:22:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.775+0000 7fd923400700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9080b2a70 0x7fd9080b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:22.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.776+0000 7fd923400700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9080b2a70 0x7fd9080b4f80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd914005950 tx=0x7fd91400b410 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:22.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:22.778+0000 7fd91be00700 1 -- 172.21.15.43:0/2881117743 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd91c010070 con 0x7fd924108560 2024-05-27T07:22:23.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.096+0000 7fd928a00700 1 -- 172.21.15.43:0/2881117743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd904005620 con 0x7fd924108560 2024-05-27T07:22:23.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.098+0000 7fd91be00700 1 -- 172.21.15.43:0/2881117743 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+8623 (secure 0 0 0) 0x7fd91c0a9b60 con 0x7fd924108560 2024-05-27T07:22:23.088 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:22:23.088 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:22:15.226346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26263":{"gid":26263,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/2130135734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2130135734},{"type":"v1","addr":"172.21.15.179:6833","nonce":2130135734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26269":{"gid":26269,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6833/2196267080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2196267080},{"type":"v1","addr":"172.21.15.179:6833","nonce":2196267080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34403,24484,34424,34484,34448]},"id":1}]} 2024-05-27T07:22:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.101+0000 7fd928a00700 1 -- 172.21.15.43:0/2881117743 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9080b2a70 msgr2=0x7fd9080b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.101+0000 7fd928a00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9080b2a70 0x7fd9080b4f80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd914005950 tx=0x7fd91400b410 comp rx=0 tx=0).stop 2024-05-27T07:22:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.101+0000 7fd928a00700 1 -- 172.21.15.43:0/2881117743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd924108560 msgr2=0x7fd9241a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.101+0000 7fd928a00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd924108560 0x7fd9241a15c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fd91c007ee0 tx=0x7fd91c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.102+0000 7fd928a00700 1 -- 172.21.15.43:0/2881117743 shutdown_connections 2024-05-27T07:22:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.102+0000 7fd928a00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9080b2a70 0x7fd9080b4f80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.102+0000 7fd928a00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd924108560 0x7fd9241a15c0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.102+0000 7fd928a00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9241032e0 0x7fd92419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.102+0000 7fd928a00700 1 --2- 172.21.15.43:0/2881117743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd924107c10 0x7fd92419d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.102+0000 7fd928a00700 1 -- 172.21.15.43:0/2881117743 >> 172.21.15.43:0/2881117743 conn(0x7fd924078af0 msgr2=0x7fd92410ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.102+0000 7fd928a00700 1 -- 172.21.15.43:0/2881117743 shutdown_connections 2024-05-27T07:22:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:23.102+0000 7fd928a00700 1 -- 172.21.15.43:0/2881117743 wait complete. 2024-05-27T07:22:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 83 2024-05-27T07:22:23.213 INFO:tasks.workunit.client.0.smithi043.stdout: 1 45921 0.63 MB/sec execute 267 sec latency 55.002 ms 2024-05-27T07:22:23.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:23 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2881117743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:23.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:23 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2881117743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:23.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:23 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2881117743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:22:23.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:22:24.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 45975 0.62 MB/sec execute 268 sec latency 52.326 ms 2024-05-27T07:22:24.407 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:24.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:24 smithi179 ceph-mon[35888]: pgmap v283: 129 pgs: 129 active+clean; 78 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 579 KiB/s wr, 43 op/s 2024-05-27T07:22:24.563 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:24 smithi043 ceph-mon[31531]: pgmap v283: 129 pgs: 129 active+clean; 78 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 579 KiB/s wr, 43 op/s 2024-05-27T07:22:24.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:24 smithi132 ceph-mon[35283]: pgmap v283: 129 pgs: 129 active+clean; 78 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 579 KiB/s wr, 43 op/s 2024-05-27T07:22:25.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46166 0.63 MB/sec execute 269 sec latency 55.470 ms 2024-05-27T07:22:26.211 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:25 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:22:26.211 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:25 smithi179 systemd[1]: Stopped Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:26.211 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:25 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.503s CPU time. 2024-05-27T07:22:26.211 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:25 smithi179 systemd[1]: Starting Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:22:26.213 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46284 0.63 MB/sec execute 270 sec latency 59.648 ms 2024-05-27T07:22:26.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:26 smithi043 ceph-mon[31531]: pgmap v284: 129 pgs: 129 active+clean; 78 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 494 KiB/s wr, 37 op/s 2024-05-27T07:22:26.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:26 smithi132 ceph-mon[35283]: pgmap v284: 129 pgs: 129 active+clean; 78 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 494 KiB/s wr, 37 op/s 2024-05-27T07:22:26.610 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:26 smithi179 ceph-mon[35888]: pgmap v284: 129 pgs: 129 active+clean; 78 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 494 KiB/s wr, 37 op/s 2024-05-27T07:22:26.610 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:26 smithi179 podman[73083]: 2024-05-27 07:22:26.275191803 +0000 UTC m=+0.032369764 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:22:26.610 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:26 smithi179 podman[73083]: 2024-05-27 07:22:26.405894843 +0000 UTC m=+0.163072842 container create a03c510326457f394974eb5dbcb6c785f21ddee9721bc4b7b3023b53516e2b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521) 2024-05-27T07:22:27.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:26 smithi179 podman[73083]: 2024-05-27 07:22:26.74570341 +0000 UTC m=+0.502881442 container init a03c510326457f394974eb5dbcb6c785f21ddee9721bc4b7b3023b53516e2b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True) 2024-05-27T07:22:27.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:26 smithi179 podman[73083]: 2024-05-27 07:22:26.75787707 +0000 UTC m=+0.515055046 container start a03c510326457f394974eb5dbcb6c785f21ddee9721bc4b7b3023b53516e2b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:22:27.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:26 smithi179 bash[73083]: a03c510326457f394974eb5dbcb6c785f21ddee9721bc4b7b3023b53516e2b94 2024-05-27T07:22:27.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:26 smithi179 systemd[1]: Started Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:27.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:26 smithi179 ceph-mds[73102]: -- 172.21.15.179:0/3271364664 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b125da1a40 con 0x55b125de9a80 2024-05-27T07:22:27.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: starting mds.f at 2024-05-27T07:22:27.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46384 0.63 MB/sec execute 271 sec latency 51.345 ms 2024-05-27T07:22:27.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:27 smithi043 ceph-mon[31531]: daemon mds.f restarted 2024-05-27T07:22:27.484 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:27 smithi179 ceph-mon[35888]: daemon mds.f restarted 2024-05-27T07:22:27.484 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:27 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:22:27.484 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:27 smithi179 systemd[1]: Stopped Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:27.484 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:27 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.347s CPU time. 2024-05-27T07:22:27.485 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3eaae00700 time 2024-05-27T07:22:27.370931+0000 2024-05-27T07:22:27.485 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:27.485 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:27.485 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3ec0848ba6] 2024-05-27T07:22:27.485 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ec0848dd1] 2024-05-27T07:22:27.485 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b10e27323d] 2024-05-27T07:22:27.485 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b10e25e971] 2024-05-27T07:22:27.485 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b10e25cfa0] 2024-05-27T07:22:27.485 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b10e25d651] 2024-05-27T07:22:27.485 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b10e1f2347] 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b10deee87e] 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b10def3e94] 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b10defe969] 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b10e29748c] 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b10e2a09cb] 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 13: (Thread::entry_wrapper()+0x43) [0x7f3ec081772d] 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3ec0817749] 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3ebf3bb1ca] 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 16: clone() 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.372+0000 7f3eaae00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3eaae00700 time 2024-05-27T07:22:27.370931+0000 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:27.486 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3ec0848ba6] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ec0848dd1] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b10e27323d] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b10e25e971] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b10e25cfa0] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b10e25d651] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b10e1f2347] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b10deee87e] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b10def3e94] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b10defe969] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b10e29748c] 2024-05-27T07:22:27.487 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b10e2a09cb] 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 13: (Thread::entry_wrapper()+0x43) [0x7f3ec081772d] 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3ec0817749] 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3ebf3bb1ca] 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 16: clone() 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: *** Caught signal (Aborted) ** 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: in thread 7f3eaae00700 thread_name:md_log_replay 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3ebf3c5d20] 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2: gsignal() 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 3: abort() 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3ec0848ced] 2024-05-27T07:22:27.488 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ec0848dd1] 2024-05-27T07:22:27.490 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b10e27323d] 2024-05-27T07:22:27.490 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b10e25e971] 2024-05-27T07:22:27.490 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b10e25cfa0] 2024-05-27T07:22:27.490 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b10e25d651] 2024-05-27T07:22:27.490 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b10e1f2347] 2024-05-27T07:22:27.490 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b10deee87e] 2024-05-27T07:22:27.490 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b10def3e94] 2024-05-27T07:22:27.490 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b10defe969] 2024-05-27T07:22:27.490 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b10e29748c] 2024-05-27T07:22:27.490 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b10e2a09cb] 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 16: (Thread::entry_wrapper()+0x43) [0x7f3ec081772d] 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3ec0817749] 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3ebf3bb1ca] 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 19: clone() 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.376+0000 7f3eaae00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: in thread 7f3eaae00700 thread_name:md_log_replay 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3ebf3c5d20] 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2: gsignal() 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 3: abort() 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3ec0848ced] 2024-05-27T07:22:27.491 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ec0848dd1] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b10e27323d] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b10e25e971] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b10e25cfa0] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b10e25d651] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b10e1f2347] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b10deee87e] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b10def3e94] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b10defe969] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b10e29748c] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b10e2a09cb] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 16: (Thread::entry_wrapper()+0x43) [0x7f3ec081772d] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3ec0817749] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3ebf3bb1ca] 2024-05-27T07:22:27.492 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 19: clone() 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: -1> 2024-05-27T07:22:27.372+0000 7f3eaae00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3eaae00700 time 2024-05-27T07:22:27.370931+0000 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3ec0848ba6] 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ec0848dd1] 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b10e27323d] 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b10e25e971] 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b10e25cfa0] 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b10e25d651] 2024-05-27T07:22:27.493 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b10e1f2347] 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b10deee87e] 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b10def3e94] 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b10defe969] 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b10e29748c] 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b10e2a09cb] 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 13: (Thread::entry_wrapper()+0x43) [0x7f3ec081772d] 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3ec0817749] 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3ebf3bb1ca] 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 16: clone() 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 0> 2024-05-27T07:22:27.376+0000 7f3eaae00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: in thread 7f3eaae00700 thread_name:md_log_replay 2024-05-27T07:22:27.494 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3ebf3c5d20] 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2: gsignal() 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 3: abort() 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3ec0848ced] 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ec0848dd1] 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b10e27323d] 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b10e25e971] 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b10e25cfa0] 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b10e25d651] 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b10e1f2347] 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b10deee87e] 2024-05-27T07:22:27.495 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b10def3e94] 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b10defe969] 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b10e29748c] 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b10e2a09cb] 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 16: (Thread::entry_wrapper()+0x43) [0x7f3ec081772d] 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3ec0817749] 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3ebf3bb1ca] 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 19: clone() 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: -1> 2024-05-27T07:22:27.372+0000 7f3eaae00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3eaae00700 time 2024-05-27T07:22:27.370931+0000 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:27.496 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3ec0848ba6] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ec0848dd1] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b10e27323d] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55b10e25e971] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 5: (SnapRealm::check_cache() const+0x13e) [0x55b10e25cfa0] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b10e25d651] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b10e1f2347] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b10deee87e] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b10def3e94] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55b10defe969] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 11: (MDLog::_replay_thread()+0x1f38) [0x55b10e29748c] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55b10e2a09cb] 2024-05-27T07:22:27.497 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 13: (Thread::entry_wrapper()+0x43) [0x7f3ec081772d] 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3ec0817749] 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3ebf3bb1ca] 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 16: clone() 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 0> 2024-05-27T07:22:27.376+0000 7f3eaae00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: in thread 7f3eaae00700 thread_name:md_log_replay 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3ebf3c5d20] 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2: gsignal() 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 3: abort() 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3ec0848ced] 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3ec0848dd1] 2024-05-27T07:22:27.498 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55b10e27323d] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55b10e25e971] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 8: (SnapRealm::check_cache() const+0x13e) [0x55b10e25cfa0] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55b10e25d651] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55b10e1f2347] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55b10deee87e] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55b10def3e94] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55b10defe969] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 14: (MDLog::_replay_thread()+0x1f38) [0x55b10e29748c] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55b10e2a09cb] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 16: (Thread::entry_wrapper()+0x43) [0x7f3ec081772d] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3ec0817749] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3ebf3bb1ca] 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 19: clone() 2024-05-27T07:22:27.499 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:27.500 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73097]: 2024-05-27T07:22:27.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:27 smithi132 ceph-mon[35283]: daemon mds.f restarted 2024-05-27T07:22:27.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.725+0000 7fbf12000700 1 -- 172.21.15.43:0/1894927026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0c1034b0 msgr2=0x7fbf0c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.725+0000 7fbf12000700 1 --2- 172.21.15.43:0/1894927026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0c1034b0 0x7fbf0c1038c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00b0d0 tx=0x7fbefc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.725+0000 7fbf12000700 1 -- 172.21.15.43:0/1894927026 shutdown_connections 2024-05-27T07:22:27.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.725+0000 7fbf12000700 1 --2- 172.21.15.43:0/1894927026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf0c108730 0x7fbf0c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.725+0000 7fbf12000700 1 --2- 172.21.15.43:0/1894927026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0c1034b0 0x7fbf0c1038c0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.725+0000 7fbf12000700 1 --2- 172.21.15.43:0/1894927026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf0c107de0 0x7fbf0c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.725+0000 7fbf12000700 1 -- 172.21.15.43:0/1894927026 >> 172.21.15.43:0/1894927026 conn(0x7fbf0c0fedb0 msgr2=0x7fbf0c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:27.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.725+0000 7fbf12000700 1 -- 172.21.15.43:0/1894927026 shutdown_connections 2024-05-27T07:22:27.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.725+0000 7fbf12000700 1 -- 172.21.15.43:0/1894927026 wait complete. 2024-05-27T07:22:27.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.726+0000 7fbf12000700 1 Processor -- start 2024-05-27T07:22:27.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.727+0000 7fbf12000700 1 -- start start 2024-05-27T07:22:27.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.727+0000 7fbf12000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf0c107de0 0x7fbf0c0ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.727+0000 7fbf12000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0c108730 0x7fbf0c0ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.727+0000 7fbf12000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf0c0ffd10 0x7fbf0c1a9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.727+0000 7fbf12000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c113210 con 0x7fbf0c108730 2024-05-27T07:22:27.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.727+0000 7fbf12000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c113090 con 0x7fbf0c107de0 2024-05-27T07:22:27.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.727+0000 7fbf12000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c113390 con 0x7fbf0c0ffd10 2024-05-27T07:22:27.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf02a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0c108730 0x7fbf0c0ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf02a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0c108730 0x7fbf0c0ff7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38736/0 (socket says 172.21.15.43:38736) 2024-05-27T07:22:27.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf02a00700 1 -- 172.21.15.43:0/1514292863 learned_addr learned my addr 172.21.15.43:0/1514292863 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:27.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf0be00700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf0c0ffd10 0x7fbf0c1a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf0b400700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf0c107de0 0x7fbf0c0ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf02a00700 1 -- 172.21.15.43:0/1514292863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf0c0ffd10 msgr2=0x7fbf0c1a9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf02a00700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf0c0ffd10 0x7fbf0c1a9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf02a00700 1 -- 172.21.15.43:0/1514292863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf0c107de0 msgr2=0x7fbf0c0ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf02a00700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf0c107de0 0x7fbf0c0ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf02a00700 1 -- 172.21.15.43:0/1514292863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c1a96d0 con 0x7fbf0c108730 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.728+0000 7fbf0be00700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf0c0ffd10 0x7fbf0c1a9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.729+0000 7fbf02a00700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0c108730 0x7fbf0c0ff7d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbef400c920 tx=0x7fbef400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.729+0000 7fbf08c00700 1 -- 172.21.15.43:0/1514292863 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4007ac0 con 0x7fbf0c108730 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.729+0000 7fbf12000700 1 -- 172.21.15.43:0/1514292863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0c1a99c0 con 0x7fbf0c108730 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.729+0000 7fbf08c00700 1 -- 172.21.15.43:0/1514292863 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbef4007c60 con 0x7fbf0c108730 2024-05-27T07:22:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.729+0000 7fbf08c00700 1 -- 172.21.15.43:0/1514292863 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4018660 con 0x7fbf0c108730 2024-05-27T07:22:27.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.729+0000 7fbf12000700 1 -- 172.21.15.43:0/1514292863 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0c1a9f00 con 0x7fbf0c108730 2024-05-27T07:22:27.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.733+0000 7fbf08c00700 1 -- 172.21.15.43:0/1514292863 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fbef400f450 con 0x7fbf0c108730 2024-05-27T07:22:27.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.734+0000 7fbf20bc6b80 1 Processor -- start 2024-05-27T07:22:27.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.735+0000 7fbf20bc6b80 1 -- start start 2024-05-27T07:22:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.735+0000 7fbf08c00700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbef00b29f0 0x7fbef00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.735+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df757f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.735+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df6f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.735+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df701c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.735+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564d9de9de40 con 0x564d9ddf0e40 2024-05-27T07:22:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.735+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564d9de8d9e0 con 0x564d9df6fdb0 2024-05-27T07:22:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.735+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564d9de91a20 con 0x564d9df75d30 2024-05-27T07:22:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.735+0000 7fbf08c00700 1 -- 172.21.15.43:0/1514292863 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9408+0+0 (secure 0 0 0) 0x7fbef40e2930 con 0x7fbf0c108730 2024-05-27T07:22:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf0be00700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbef00b29f0 0x7fbef00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf0b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df757f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf0b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df757f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38740/0 (socket says 172.21.15.43:38740) 2024-05-27T07:22:27.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf0b400700 1 -- 172.21.15.43:0/2759891136 learned_addr learned my addr 172.21.15.43:0/2759891136 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:27.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf02a00700 1 --2- 172.21.15.43:0/2759891136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df6f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf0be00700 1 --2- 172.21.15.43:0/2759891136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf0b400700 1 -- 172.21.15.43:0/2759891136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 msgr2=0x564d9df6f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf0b400700 1 --2- 172.21.15.43:0/2759891136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df6f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf0b400700 1 -- 172.21.15.43:0/2759891136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 msgr2=0x564d9df701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf0b400700 1 --2- 172.21.15.43:0/2759891136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df701c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.736+0000 7fbf0b400700 1 -- 172.21.15.43:0/2759891136 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d9de56420 con 0x564d9ddf0e40 2024-05-27T07:22:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.737+0000 7fbf0b400700 1 --2- 172.21.15.43:0/2759891136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df757f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fbefc000f80 tx=0x7fbefc009150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7ea7e68088d33e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.737+0000 7fbf0be00700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbef00b29f0 0x7fbef00b4f00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbf040059d0 tx=0x7fbf04005f10 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.737+0000 7fbf00c00700 1 -- 172.21.15.43:0/2759891136 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc004a50 con 0x564d9ddf0e40 2024-05-27T07:22:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.737+0000 7fbf00c00700 1 -- 172.21.15.43:0/2759891136 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbefc004bf0 con 0x564d9ddf0e40 2024-05-27T07:22:27.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.737+0000 7fbf00c00700 1 -- 172.21.15.43:0/2759891136 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc005260 con 0x564d9ddf0e40 2024-05-27T07:22:27.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.738+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/2759891136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 msgr2=0x564d9df757f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.738+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/2759891136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df757f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fbefc000f80 tx=0x7fbefc009150 comp rx=0 tx=0).stop 2024-05-27T07:22:27.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.738+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/2759891136 shutdown_connections 2024-05-27T07:22:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.738+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/2759891136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.738+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/2759891136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df757f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.738+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/2759891136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df6f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.738+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/2759891136 >> 172.21.15.43:0/2759891136 conn(0x564d9de50fc0 msgr2=0x564d9df6a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.739+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/2759891136 shutdown_connections 2024-05-27T07:22:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.739+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/2759891136 wait complete. 2024-05-27T07:22:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.739+0000 7fbf20bc6b80 1 -- start start 2024-05-27T07:22:27.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.740+0000 7fbf20bc6b80 1 Processor -- start 2024-05-27T07:22:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.740+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9ddf0e40 0x564d9df79590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.740+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df79ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.740+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df75d30 0x564d9df7a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.740+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564d9dd93fa0 con 0x564d9df75d30 2024-05-27T07:22:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.740+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564d9de2e760 con 0x564d9df6fdb0 2024-05-27T07:22:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.740+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564d9df7c7a0 con 0x564d9ddf0e40 2024-05-27T07:22:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.740+0000 7fbf20bc6b80 10 client.0 _ll_register_callbacks cb 0x564d9de37c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:27.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.741+0000 7fbf0be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df75d30 0x564d9df7a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.741+0000 7fbf0be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df75d30 0x564d9df7a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38744/0 (socket says 172.21.15.43:38744) 2024-05-27T07:22:27.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.741+0000 7fbf0be00700 1 -- 172.21.15.43:0/1040333532 learned_addr learned my addr 172.21.15.43:0/1040333532 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:27.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.741+0000 7fbf0b400700 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9ddf0e40 0x564d9df79590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.741+0000 7fbf02a00700 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df79ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.741+0000 7fbf0be00700 1 -- 172.21.15.43:0/1040333532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9ddf0e40 msgr2=0x564d9df79590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.741+0000 7fbf0be00700 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9ddf0e40 0x564d9df79590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.742+0000 7fbf0be00700 1 -- 172.21.15.43:0/1040333532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 msgr2=0x564d9df79ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:27.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.742+0000 7fbf0be00700 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df79ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.742+0000 7fbf0be00700 1 -- 172.21.15.43:0/1040333532 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d9dddd8d0 con 0x564d9df75d30 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.742+0000 7fbf0be00700 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df75d30 0x564d9df7a010 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbf040117d0 tx=0x7fbf040047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fa7ea50b395c97 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.742+0000 7fbef9600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.742+0000 7fbef9600700 1 -- 172.21.15.43:0/1040333532 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04023070 con 0x564d9df75d30 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.742+0000 7fbef9600700 1 -- 172.21.15.43:0/1040333532 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9408+0+0 (secure 0 0 0) 0x7fbf04004830 con 0x564d9df75d30 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.742+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1040333532 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbefc004bf0 con 0x564d9df75d30 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbef9600700 1 -- 172.21.15.43:0/1040333532 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fbf0402e050 con 0x564d9df75d30 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbf20bc6b80 10 client.16695 fetch_fsmap learned FSMap version 85 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1040333532 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=4+}) -- 0x564d9de5e060 con 0x564d9df75d30 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbef9600700 1 -- 172.21.15.43:0/1040333532 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbefc004bf0 con 0x564d9df75d30 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbef9600700 1 -- 172.21.15.43:0/1040333532 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 85) ==== 5966+0+0 (secure 0 0 0) 0x7fbf04028ba0 con 0x564d9df75d30 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbf20bc6b80 10 client.16695 fetch_fsmap finished waiting for FSMap version 85 2024-05-27T07:22:27.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbf20bc6b80 10 client.16695 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:22:27.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbf20bc6b80 20 client.16695 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:27.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x564d9df89960 0x564d9df8be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbf20bc6b80 4 client.16695 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:27.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.743+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1040333532 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d9de9fd90 con 0x564d9df89960 2024-05-27T07:22:27.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.744+0000 7fbf0b400700 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x564d9df89960 0x564d9df8be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.746+0000 7fbf0b400700 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x564d9df89960 0x564d9df8be10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=adcbbc737e565359 server_cookie=eb60cdf113619e3d in_seq=0 out_seq=0 2024-05-27T07:22:27.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.746+0000 7fbef9600700 10 client.16695 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:22:27.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbef9600700 1 -- 172.21.15.43:0/1040333532 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564d9de9fd90 con 0x564d9df89960 2024-05-27T07:22:27.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbef9600700 10 client.16695 handle_command_reply: tid=0 2024-05-27T07:22:27.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbf20bc6b80 1 client.16695 shutdown 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1040333532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df75d30 msgr2=0x564d9df7a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df75d30 0x564d9df7a010 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbf040117d0 tx=0x7fbf040047c0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1040333532 shutdown_connections 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x564d9df89960 0x564d9df8be10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df75d30 0x564d9df7a010 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9ddf0e40 0x564d9df79590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1040333532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df79ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1040333532 >> 172.21.15.43:0/1040333532 conn(0x564d9de50fc0 msgr2=0x564d9df6a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.750+0000 7fbef9600700 0 client.16695 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.751+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1040333532 shutdown_connections 2024-05-27T07:22:27.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.751+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1040333532 wait complete. 2024-05-27T07:22:27.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.751+0000 7fbf20bc6b80 20 client.16695 trim_cache size 0 max 16384 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.769+0000 7fbf20bc6b80 1 Processor -- start 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.769+0000 7fbf20bc6b80 1 -- start start 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.769+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9ddf0e40 0x564d9df6f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.769+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df6fdb0 0x564d9df784f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.769+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df85800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.769+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1b3870 con 0x564d9df6fdb0 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.769+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1b3f40 con 0x564d9ddf0e40 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.769+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1b1660 con 0x564d9df75d30 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf02a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df6fdb0 0x564d9df784f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf02a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df6fdb0 0x564d9df784f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38758/0 (socket says 172.21.15.43:38758) 2024-05-27T07:22:27.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf02a00700 1 -- 172.21.15.43:0/650853068 learned_addr learned my addr 172.21.15.43:0/650853068 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf0be00700 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df85800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf0b400700 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9ddf0e40 0x564d9df6f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf02a00700 1 -- 172.21.15.43:0/650853068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 msgr2=0x564d9df85800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf02a00700 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df85800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf02a00700 1 -- 172.21.15.43:0/650853068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9ddf0e40 msgr2=0x564d9df6f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf02a00700 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9ddf0e40 0x564d9df6f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf02a00700 1 -- 172.21.15.43:0/650853068 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d9df7a380 con 0x564d9df6fdb0 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.770+0000 7fbf0be00700 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df85800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.771+0000 7fbf0b400700 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9ddf0e40 0x564d9df6f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.771+0000 7fbf02a00700 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df6fdb0 0x564d9df784f0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fbef40b2190 tx=0x7fbef40a5170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4938dd7dc3c8907b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.771+0000 7fbefbe00700 1 -- 172.21.15.43:0/650853068 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef40adc10 con 0x564d9df6fdb0 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.771+0000 7fbefbe00700 1 -- 172.21.15.43:0/650853068 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbef40b26c0 con 0x564d9df6fdb0 2024-05-27T07:22:27.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.771+0000 7fbefbe00700 1 -- 172.21.15.43:0/650853068 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef40a9b70 con 0x564d9df6fdb0 2024-05-27T07:22:27.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.772+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/650853068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df6fdb0 msgr2=0x564d9df784f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.772+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df6fdb0 0x564d9df784f0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fbef40b2190 tx=0x7fbef40a5170 comp rx=0 tx=0).stop 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.772+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/650853068 shutdown_connections 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.772+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df85800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.772+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9ddf0e40 0x564d9df6f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.772+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/650853068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9df6fdb0 0x564d9df784f0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.772+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/650853068 >> 172.21.15.43:0/650853068 conn(0x564d9de50fc0 msgr2=0x564d9df73140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.772+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/650853068 shutdown_connections 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.772+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/650853068 wait complete. 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.773+0000 7fbf20bc6b80 1 -- start start 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.773+0000 7fbf20bc6b80 1 Processor -- start 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df819f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df81f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf20bc6b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df90660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1b1d30 con 0x564d9ddf0e40 2024-05-27T07:22:27.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1b31a0 con 0x564d9df6fdb0 2024-05-27T07:22:27.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf20bc6b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1b2400 con 0x564d9df75d30 2024-05-27T07:22:27.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf20bc6b80 10 client.0 _ll_register_callbacks cb 0x564d9df8c500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:27.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf0b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df819f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf0b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df819f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38768/0 (socket says 172.21.15.43:38768) 2024-05-27T07:22:27.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf0b400700 1 -- 172.21.15.43:0/1002504387 learned_addr learned my addr 172.21.15.43:0/1002504387 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:27.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf0b400700 1 -- 172.21.15.43:0/1002504387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 msgr2=0x564d9df90660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:27.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf0b400700 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df90660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf0b400700 1 -- 172.21.15.43:0/1002504387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 msgr2=0x564d9df81f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.774+0000 7fbf02a00700 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df81f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.775+0000 7fbf0b400700 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df81f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.775+0000 7fbf0b400700 1 -- 172.21.15.43:0/1002504387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d9df90ba0 con 0x564d9ddf0e40 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.775+0000 7fbf02a00700 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df81f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.775+0000 7fbf0b400700 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df819f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbefc002230 tx=0x7fbefc009270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6883e02eb1d84c64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.775+0000 7fbef9600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.775+0000 7fbef9600700 1 -- 172.21.15.43:0/1002504387 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0075c0 con 0x564d9ddf0e40 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.775+0000 7fbef9600700 1 -- 172.21.15.43:0/1002504387 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9408+0+0 (secure 0 0 0) 0x7fbefc007760 con 0x564d9ddf0e40 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.775+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1002504387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbef40b26c0 con 0x564d9ddf0e40 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbef9600700 1 -- 172.21.15.43:0/1002504387 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fbefc03c050 con 0x564d9ddf0e40 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbf20bc6b80 10 client.16701 fetch_fsmap learned FSMap version 85 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1002504387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=4+}) -- 0x564d9dff3b00 con 0x564d9ddf0e40 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbef9600700 1 -- 172.21.15.43:0/1002504387 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbef40b26c0 con 0x564d9ddf0e40 2024-05-27T07:22:27.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbef9600700 1 -- 172.21.15.43:0/1002504387 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 85) ==== 5966+0+0 (secure 0 0 0) 0x7fbefc011070 con 0x564d9ddf0e40 2024-05-27T07:22:27.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbf20bc6b80 10 client.16701 fetch_fsmap finished waiting for FSMap version 85 2024-05-27T07:22:27.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbf20bc6b80 10 client.16701 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:22:27.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbf20bc6b80 20 client.16701 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:27.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x564d9dff4150 0x564d9dff6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:27.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbf20bc6b80 4 client.16701 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:22:27.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.776+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1002504387 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d9de99570 con 0x564d9dff4150 2024-05-27T07:22:27.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.777+0000 7fbf02a00700 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x564d9dff4150 0x564d9dff6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:27.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.778+0000 7fbf02a00700 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x564d9dff4150 0x564d9dff6570 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7fa0c5a5a99edf25 server_cookie=1df9269a28c6988f in_seq=0 out_seq=0 2024-05-27T07:22:27.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.778+0000 7fbef9600700 10 client.16701 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:22:27.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbef9600700 1 -- 172.21.15.43:0/1002504387 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d9de99570 con 0x564d9dff4150 2024-05-27T07:22:27.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbef9600700 10 client.16701 handle_command_reply: tid=0 2024-05-27T07:22:27.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbf20bc6b80 1 client.16701 shutdown 2024-05-27T07:22:27.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1002504387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 msgr2=0x564d9df819f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df819f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbefc002230 tx=0x7fbefc009270 comp rx=0 tx=0).stop 2024-05-27T07:22:27.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1002504387 shutdown_connections 2024-05-27T07:22:27.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x564d9dff4150 0x564d9dff6570 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d9df75d30 0x564d9df90660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d9ddf0e40 0x564d9df819f0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbf20bc6b80 1 --2- 172.21.15.43:0/1002504387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d9df6fdb0 0x564d9df81f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbef9600700 0 client.16701 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:22:27.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1002504387 >> 172.21.15.43:0/1002504387 conn(0x564d9de50fc0 msgr2=0x564d9df73140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:27.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.779+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1002504387 shutdown_connections 2024-05-27T07:22:27.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.780+0000 7fbf20bc6b80 1 -- 172.21.15.43:0/1002504387 wait complete. 2024-05-27T07:22:27.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.780+0000 7fbf20bc6b80 20 client.16701 trim_cache size 0 max 16384 2024-05-27T07:22:27.771 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.780+0000 7fbf12000700 1 -- 172.21.15.43:0/1514292863 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbef00b29f0 msgr2=0x7fbef00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.780+0000 7fbf12000700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbef00b29f0 0x7fbef00b4f00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbf040059d0 tx=0x7fbf04005f10 comp rx=0 tx=0).stop 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.781+0000 7fbf12000700 1 -- 172.21.15.43:0/1514292863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0c108730 msgr2=0x7fbf0c0ff7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.781+0000 7fbf12000700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0c108730 0x7fbf0c0ff7d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbef400c920 tx=0x7fbef400cce0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.781+0000 7fbf12000700 1 -- 172.21.15.43:0/1514292863 shutdown_connections 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.781+0000 7fbf12000700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbef00b29f0 0x7fbef00b4f00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.781+0000 7fbf12000700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf0c0ffd10 0x7fbf0c1a9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.781+0000 7fbf12000700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf0c107de0 0x7fbf0c0ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.781+0000 7fbf12000700 1 --2- 172.21.15.43:0/1514292863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf0c108730 0x7fbf0c0ff7d0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.781+0000 7fbf12000700 1 -- 172.21.15.43:0/1514292863 >> 172.21.15.43:0/1514292863 conn(0x7fbf0c0fedb0 msgr2=0x7fbf0c101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.781+0000 7fbf12000700 1 -- 172.21.15.43:0/1514292863 shutdown_connections 2024-05-27T07:22:27.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:27.781+0000 7fbf12000700 1 -- 172.21.15.43:0/1514292863 wait complete. 2024-05-27T07:22:27.851 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:27 smithi179 systemd[1]: Starting Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:22:27.851 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:27 smithi179 podman[73133]: 2024-05-27 07:22:27.519080459 +0000 UTC m=+0.039215529 container died a03c510326457f394974eb5dbcb6c785f21ddee9721bc4b7b3023b53516e2b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-27T07:22:28.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46444 0.63 MB/sec execute 272 sec latency 67.076 ms 2024-05-27T07:22:28.530 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:28 smithi179 podman[73133]: 2024-05-27 07:22:28.258053238 +0000 UTC m=+0.778188338 container remove a03c510326457f394974eb5dbcb6c785f21ddee9721bc4b7b3023b53516e2b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:22:28.534 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:28 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:22:28.534 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:28 smithi179 ceph-mon[35888]: pgmap v285: 129 pgs: 129 active+clean; 79 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 614 KiB/s wr, 47 op/s 2024-05-27T07:22:28.534 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:28 smithi179 ceph-mon[35888]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:22:28.534 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:28 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/459265898,v1:172.21.15.179:6833/459265898] up:boot 2024-05-27T07:22:28.534 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:28 smithi179 ceph-mon[35888]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:28.535 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:28 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:28.535 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:28 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:28.569 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:28 smithi043 ceph-mon[31531]: pgmap v285: 129 pgs: 129 active+clean; 79 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 614 KiB/s wr, 47 op/s 2024-05-27T07:22:28.569 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:28 smithi043 ceph-mon[31531]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:22:28.569 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:28 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/459265898,v1:172.21.15.179:6833/459265898] up:boot 2024-05-27T07:22:28.569 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:28 smithi043 ceph-mon[31531]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:28.570 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:28 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:28.570 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:28 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:28.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:28 smithi132 ceph-mon[35283]: pgmap v285: 129 pgs: 129 active+clean; 79 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 614 KiB/s wr, 47 op/s 2024-05-27T07:22:28.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:28 smithi132 ceph-mon[35283]: osdmap e118: 12 total, 12 up, 12 in 2024-05-27T07:22:28.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:28 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/459265898,v1:172.21.15.179:6833/459265898] up:boot 2024-05-27T07:22:28.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:28 smithi132 ceph-mon[35283]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:28.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:28 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:28.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:28 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:28.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:22:28.803 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:28 smithi179 podman[73204]: 2024-05-27 07:22:28.446375197 +0000 UTC m=+0.034668585 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:22:28.803 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:28 smithi179 podman[73204]: 2024-05-27 07:22:28.643845027 +0000 UTC m=+0.232138402 container create 399e5555e7257e577e3a56e8e149e7f494661110ee449ca298b00152fde4cfa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5) 2024-05-27T07:22:29.131 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:29.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:29 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:22:29.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:29 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.544s CPU time. 2024-05-27T07:22:29.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:28 smithi179 podman[73204]: 2024-05-27 07:22:28.923141059 +0000 UTC m=+0.511434506 container init 399e5555e7257e577e3a56e8e149e7f494661110ee449ca298b00152fde4cfa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD) 2024-05-27T07:22:29.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:28 smithi179 podman[73204]: 2024-05-27 07:22:28.932568528 +0000 UTC m=+0.520861902 container start 399e5555e7257e577e3a56e8e149e7f494661110ee449ca298b00152fde4cfa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-27T07:22:29.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-mds[73302]: -- 172.21.15.179:0/3336351498 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5621d54eba40 con 0x5621d5533a80 2024-05-27T07:22:29.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: starting mds.l at 2024-05-27T07:22:29.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 bash[73204]: 399e5555e7257e577e3a56e8e149e7f494661110ee449ca298b00152fde4cfa1 2024-05-27T07:22:29.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 systemd[1]: Started Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:29.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46529 0.63 MB/sec execute 273 sec latency 43.405 ms 2024-05-27T07:22:29.505 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:29 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:29.505 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:29 smithi043 ceph-mon[31531]: daemon mds.l restarted 2024-05-27T07:22:29.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:29 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:29.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:29 smithi132 ceph-mon[35283]: daemon mds.l restarted 2024-05-27T07:22:29.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:29 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:22:29.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:29 smithi179 ceph-mon[35888]: daemon mds.l restarted 2024-05-27T07:22:30.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe3c2e00700 time 2024-05-27T07:22:29.834590+0000 2024-05-27T07:22:30.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:30.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:30.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe3d89dbba6] 2024-05-27T07:22:30.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe3d89dbdd1] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562196f3f23d] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562196f2a971] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 5: (SnapRealm::check_cache() const+0x13e) [0x562196f28fa0] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562196f29651] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562196ebe347] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562196bba87e] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562196bbfe94] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562196bca969] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 11: (MDLog::_replay_thread()+0x1f38) [0x562196f6348c] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562196f6c9cb] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 13: (Thread::entry_wrapper()+0x43) [0x7fe3d89aa72d] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe3d89aa749] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe3d754e1ca] 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 16: clone() 2024-05-27T07:22:30.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: *** Caught signal (Aborted) ** 2024-05-27T07:22:30.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: in thread 7fe3c2e00700 thread_name:md_log_replay 2024-05-27T07:22:30.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:29.835+0000 7fe3c2e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe3c2e00700 time 2024-05-27T07:22:29.834590+0000 2024-05-27T07:22:30.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:30.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:30.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe3d89dbba6] 2024-05-27T07:22:30.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe3d89dbdd1] 2024-05-27T07:22:30.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562196f3f23d] 2024-05-27T07:22:30.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562196f2a971] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 5: (SnapRealm::check_cache() const+0x13e) [0x562196f28fa0] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562196f29651] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562196ebe347] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562196bba87e] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562196bbfe94] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562196bca969] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 11: (MDLog::_replay_thread()+0x1f38) [0x562196f6348c] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562196f6c9cb] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 13: (Thread::entry_wrapper()+0x43) [0x7fe3d89aa72d] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe3d89aa749] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe3d754e1ca] 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 16: clone() 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:30.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe3d7558d20] 2024-05-27T07:22:30.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2: gsignal() 2024-05-27T07:22:30.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 3: abort() 2024-05-27T07:22:30.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe3d89dbced] 2024-05-27T07:22:30.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe3d89dbdd1] 2024-05-27T07:22:30.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46598 0.63 MB/sec execute 274 sec latency 47.933 ms 2024-05-27T07:22:30.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562196f3f23d] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562196f2a971] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 8: (SnapRealm::check_cache() const+0x13e) [0x562196f28fa0] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562196f29651] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562196ebe347] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562196bba87e] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562196bbfe94] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562196bca969] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 14: (MDLog::_replay_thread()+0x1f38) [0x562196f6348c] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562196f6c9cb] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 16: (Thread::entry_wrapper()+0x43) [0x7fe3d89aa72d] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe3d89aa749] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe3d754e1ca] 2024-05-27T07:22:30.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 19: clone() 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:29.837+0000 7fe3c2e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: in thread 7fe3c2e00700 thread_name:md_log_replay 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe3d7558d20] 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2: gsignal() 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 3: abort() 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe3d89dbced] 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe3d89dbdd1] 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562196f3f23d] 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562196f2a971] 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 8: (SnapRealm::check_cache() const+0x13e) [0x562196f28fa0] 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562196f29651] 2024-05-27T07:22:30.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562196ebe347] 2024-05-27T07:22:30.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562196bba87e] 2024-05-27T07:22:30.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562196bbfe94] 2024-05-27T07:22:30.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562196bca969] 2024-05-27T07:22:30.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 14: (MDLog::_replay_thread()+0x1f38) [0x562196f6348c] 2024-05-27T07:22:30.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562196f6c9cb] 2024-05-27T07:22:30.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 16: (Thread::entry_wrapper()+0x43) [0x7fe3d89aa72d] 2024-05-27T07:22:30.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe3d89aa749] 2024-05-27T07:22:30.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe3d754e1ca] 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 19: clone() 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: -1> 2024-05-27T07:22:29.835+0000 7fe3c2e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe3c2e00700 time 2024-05-27T07:22:29.834590+0000 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe3d89dbba6] 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe3d89dbdd1] 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562196f3f23d] 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562196f2a971] 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 5: (SnapRealm::check_cache() const+0x13e) [0x562196f28fa0] 2024-05-27T07:22:30.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562196f29651] 2024-05-27T07:22:30.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562196ebe347] 2024-05-27T07:22:30.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562196bba87e] 2024-05-27T07:22:30.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562196bbfe94] 2024-05-27T07:22:30.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562196bca969] 2024-05-27T07:22:30.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 11: (MDLog::_replay_thread()+0x1f38) [0x562196f6348c] 2024-05-27T07:22:30.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562196f6c9cb] 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 13: (Thread::entry_wrapper()+0x43) [0x7fe3d89aa72d] 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe3d89aa749] 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe3d754e1ca] 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 16: clone() 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 0> 2024-05-27T07:22:29.837+0000 7fe3c2e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: in thread 7fe3c2e00700 thread_name:md_log_replay 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe3d7558d20] 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2: gsignal() 2024-05-27T07:22:30.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 3: abort() 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe3d89dbced] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe3d89dbdd1] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562196f3f23d] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562196f2a971] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 8: (SnapRealm::check_cache() const+0x13e) [0x562196f28fa0] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562196f29651] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562196ebe347] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562196bba87e] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562196bbfe94] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562196bca969] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 14: (MDLog::_replay_thread()+0x1f38) [0x562196f6348c] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562196f6c9cb] 2024-05-27T07:22:30.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 16: (Thread::entry_wrapper()+0x43) [0x7fe3d89aa72d] 2024-05-27T07:22:30.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe3d89aa749] 2024-05-27T07:22:30.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe3d754e1ca] 2024-05-27T07:22:30.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 19: clone() 2024-05-27T07:22:30.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:30.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: -1> 2024-05-27T07:22:29.835+0000 7fe3c2e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fe3c2e00700 time 2024-05-27T07:22:29.834590+0000 2024-05-27T07:22:30.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:30.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:30.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe3d89dbba6] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe3d89dbdd1] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562196f3f23d] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 4: (SnapRealm::build_snap_set() const+0x323) [0x562196f2a971] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 5: (SnapRealm::check_cache() const+0x13e) [0x562196f28fa0] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562196f29651] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562196ebe347] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562196bba87e] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562196bbfe94] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x562196bca969] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 11: (MDLog::_replay_thread()+0x1f38) [0x562196f6348c] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 12: (MDLog::ReplayThread::entry()+0x11) [0x562196f6c9cb] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 13: (Thread::entry_wrapper()+0x43) [0x7fe3d89aa72d] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 14: (Thread::_entry_func(void*)+0xd) [0x7fe3d89aa749] 2024-05-27T07:22:30.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe3d754e1ca] 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 16: clone() 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 0> 2024-05-27T07:22:29.837+0000 7fe3c2e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: in thread 7fe3c2e00700 thread_name:md_log_replay 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe3d7558d20] 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2: gsignal() 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 3: abort() 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe3d89dbced] 2024-05-27T07:22:30.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe3d89dbdd1] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x562196f3f23d] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 7: (SnapRealm::build_snap_set() const+0x323) [0x562196f2a971] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 8: (SnapRealm::check_cache() const+0x13e) [0x562196f28fa0] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x562196f29651] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x562196ebe347] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x562196bba87e] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x562196bbfe94] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x562196bca969] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 14: (MDLog::_replay_thread()+0x1f38) [0x562196f6348c] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 15: (MDLog::ReplayThread::entry()+0x11) [0x562196f6c9cb] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 16: (Thread::entry_wrapper()+0x43) [0x7fe3d89aa72d] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 17: (Thread::_entry_func(void*)+0xd) [0x7fe3d89aa749] 2024-05-27T07:22:30.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe3d754e1ca] 2024-05-27T07:22:30.227 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 19: clone() 2024-05-27T07:22:30.227 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:30.227 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73298]: 2024-05-27T07:22:30.227 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:29 smithi179 podman[73366]: 2024-05-27 07:22:29.948501537 +0000 UTC m=+0.033104474 container died 399e5555e7257e577e3a56e8e149e7f494661110ee449ca298b00152fde4cfa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-27T07:22:30.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:30 smithi043 ceph-mon[31531]: pgmap v287: 129 pgs: 129 active+clean; 79 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 634 KiB/s wr, 37 op/s 2024-05-27T07:22:30.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:30 smithi043 ceph-mon[31531]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:22:30.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:30 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/1100556609,v1:172.21.15.179:6833/1100556609] up:boot 2024-05-27T07:22:30.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:30 smithi043 ceph-mon[31531]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:30.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:30 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:30.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:30 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:30.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:30 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:30.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:30 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:30.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:30 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:22:30.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:30 smithi132 ceph-mon[35283]: pgmap v287: 129 pgs: 129 active+clean; 79 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 634 KiB/s wr, 37 op/s 2024-05-27T07:22:30.630 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:30 smithi132 ceph-mon[35283]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:22:30.630 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:30 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/1100556609,v1:172.21.15.179:6833/1100556609] up:boot 2024-05-27T07:22:30.630 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:30 smithi132 ceph-mon[35283]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:30.631 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:30 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:30.631 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:30 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:30.631 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:30 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:30.631 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:30 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:30.631 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:30 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:22:30.632 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:30 smithi179 ceph-mon[35888]: pgmap v287: 129 pgs: 129 active+clean; 79 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 634 KiB/s wr, 37 op/s 2024-05-27T07:22:30.632 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:30 smithi179 ceph-mon[35888]: osdmap e119: 12 total, 12 up, 12 in 2024-05-27T07:22:30.632 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:30 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/1100556609,v1:172.21.15.179:6833/1100556609] up:boot 2024-05-27T07:22:30.632 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:30 smithi179 ceph-mon[35888]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:30.632 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:30 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:30.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:30 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:30.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:30 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:22:30.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:30 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:22:30.633 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:30 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:22:31.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46670 0.63 MB/sec execute 275 sec latency 52.584 ms 2024-05-27T07:22:31.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.452+0000 7f927ae00700 1 -- 172.21.15.43:0/3593609786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f92741045e0 msgr2=0x7f92741049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:31.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.452+0000 7f927ae00700 1 --2- 172.21.15.43:0/3593609786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f92741045e0 0x7f92741049f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92640098f0 tx=0x7f9264009c00 comp rx=0 tx=0).stop 2024-05-27T07:22:31.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.452+0000 7f927ae00700 1 -- 172.21.15.43:0/3593609786 shutdown_connections 2024-05-27T07:22:31.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.452+0000 7f927ae00700 1 --2- 172.21.15.43:0/3593609786 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9274109860 0x7f92741111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.452+0000 7f927ae00700 1 --2- 172.21.15.43:0/3593609786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f92741045e0 0x7f92741049f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.452+0000 7f927ae00700 1 --2- 172.21.15.43:0/3593609786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9274108f10 0x7f9274109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.452+0000 7f927ae00700 1 -- 172.21.15.43:0/3593609786 >> 172.21.15.43:0/3593609786 conn(0x7f9274100030 msgr2=0x7f9274102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:31.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.452+0000 7f927ae00700 1 -- 172.21.15.43:0/3593609786 shutdown_connections 2024-05-27T07:22:31.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.452+0000 7f927ae00700 1 -- 172.21.15.43:0/3593609786 wait complete. 2024-05-27T07:22:31.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.454+0000 7f927ae00700 1 Processor -- start 2024-05-27T07:22:31.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.454+0000 7f927ae00700 1 -- start start 2024-05-27T07:22:31.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.454+0000 7f927ae00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92741045e0 0x7f927419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.454+0000 7f927ae00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9274108f10 0x7f927419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.454+0000 7f927ae00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9274109860 0x7f92741a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.454+0000 7f927ae00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9274114390 con 0x7f92741045e0 2024-05-27T07:22:31.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.454+0000 7f927ae00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9274114210 con 0x7f9274108f10 2024-05-27T07:22:31.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.454+0000 7f927ae00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9274114510 con 0x7f9274109860 2024-05-27T07:22:31.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9273e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92741045e0 0x7f927419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9273e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92741045e0 0x7f927419d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38792/0 (socket says 172.21.15.43:38792) 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9273e00700 1 -- 172.21.15.43:0/3993620746 learned_addr learned my addr 172.21.15.43:0/3993620746 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9273400700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9274108f10 0x7f927419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9278e00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9274109860 0x7f92741a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9273e00700 1 -- 172.21.15.43:0/3993620746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9274109860 msgr2=0x7f92741a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9273e00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9274109860 0x7f92741a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9273e00700 1 -- 172.21.15.43:0/3993620746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9274108f10 msgr2=0x7f927419d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9273e00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9274108f10 0x7f927419d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9273e00700 1 -- 172.21.15.43:0/3993620746 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92741a1ed0 con 0x7f92741045e0 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.455+0000 7f9278e00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9274109860 0x7f92741a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:31.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.456+0000 7f9273e00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92741045e0 0x7f927419d000 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9264005950 tx=0x7f9264009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:31.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.456+0000 7f9270c00700 1 -- 172.21.15.43:0/3993620746 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926400b790 con 0x7f92741045e0 2024-05-27T07:22:31.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.456+0000 7f927ae00700 1 -- 172.21.15.43:0/3993620746 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92741a2160 con 0x7f92741045e0 2024-05-27T07:22:31.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.456+0000 7f9270c00700 1 -- 172.21.15.43:0/3993620746 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f92640054d0 con 0x7f92741045e0 2024-05-27T07:22:31.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.456+0000 7f9270c00700 1 -- 172.21.15.43:0/3993620746 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264003ab0 con 0x7f92741045e0 2024-05-27T07:22:31.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.456+0000 7f927ae00700 1 -- 172.21.15.43:0/3993620746 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92741a9ae0 con 0x7f92741045e0 2024-05-27T07:22:31.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.462+0000 7f9270c00700 1 -- 172.21.15.43:0/3993620746 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f926400b930 con 0x7f92741045e0 2024-05-27T07:22:31.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.463+0000 7f92898adb80 1 Processor -- start 2024-05-27T07:22:31.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.463+0000 7f92898adb80 1 -- start start 2024-05-27T07:22:31.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f92898adb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e908330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f92898adb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e789e40 0x560f0e908870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f92898adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e908e70 0x560f0e915110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f92898adb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560f0e836e40 con 0x560f0e8759f0 2024-05-27T07:22:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f92898adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560f0e8269e0 con 0x560f0e908e70 2024-05-27T07:22:31.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f92898adb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560f0e82aa20 con 0x560f0e789e40 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f9278e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e908330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f9270c00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f92580b29f0 0x7f92580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f9278e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e908330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38806/0 (socket says 172.21.15.43:38806) 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f9278e00700 1 -- 172.21.15.43:0/3704357250 learned_addr learned my addr 172.21.15.43:0/3704357250 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f9273400700 1 --2- 172.21.15.43:0/3704357250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e908e70 0x560f0e915110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f9270c00700 1 -- 172.21.15.43:0/3993620746 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9408+0+0 (secure 0 0 0) 0x7f92640e3b50 con 0x7f92741045e0 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.464+0000 7f9273e00700 1 --2- 172.21.15.43:0/3704357250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e789e40 0x560f0e908870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.465+0000 7f9278e00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f92580b29f0 0x7f92580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.465+0000 7f9278e00700 1 -- 172.21.15.43:0/3704357250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e789e40 msgr2=0x560f0e908870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.465+0000 7f9278e00700 1 --2- 172.21.15.43:0/3704357250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e789e40 0x560f0e908870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.465+0000 7f9278e00700 1 -- 172.21.15.43:0/3704357250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e908e70 msgr2=0x560f0e915110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.465+0000 7f9278e00700 1 --2- 172.21.15.43:0/3704357250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e908e70 0x560f0e915110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.465+0000 7f9278e00700 1 -- 172.21.15.43:0/3704357250 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f0e7ef420 con 0x560f0e8759f0 2024-05-27T07:22:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.465+0000 7f9273400700 1 --2- 172.21.15.43:0/3704357250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e908e70 0x560f0e915110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.465+0000 7f9278e00700 1 --2- 172.21.15.43:0/3704357250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e908330 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f926c009fd0 tx=0x7f926c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5080559570e596e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.466+0000 7f9278e00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f92580b29f0 0x7f92580b4f00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f926c01c000 tx=0x7f926c01e000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:31.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.466+0000 7f9269600700 1 -- 172.21.15.43:0/3704357250 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c017d90 con 0x560f0e8759f0 2024-05-27T07:22:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.466+0000 7f9269600700 1 -- 172.21.15.43:0/3704357250 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f926c01abe0 con 0x560f0e8759f0 2024-05-27T07:22:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.466+0000 7f9269600700 1 -- 172.21.15.43:0/3704357250 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c023960 con 0x560f0e8759f0 2024-05-27T07:22:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.467+0000 7f92898adb80 1 -- 172.21.15.43:0/3704357250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 msgr2=0x560f0e908330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.467+0000 7f92898adb80 1 --2- 172.21.15.43:0/3704357250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e908330 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f926c009fd0 tx=0x7f926c007e00 comp rx=0 tx=0).stop 2024-05-27T07:22:31.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.467+0000 7f92898adb80 1 -- 172.21.15.43:0/3704357250 shutdown_connections 2024-05-27T07:22:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.467+0000 7f92898adb80 1 --2- 172.21.15.43:0/3704357250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e908e70 0x560f0e915110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.467+0000 7f92898adb80 1 --2- 172.21.15.43:0/3704357250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e908330 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.467+0000 7f92898adb80 1 --2- 172.21.15.43:0/3704357250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e789e40 0x560f0e908870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.467+0000 7f92898adb80 1 -- 172.21.15.43:0/3704357250 >> 172.21.15.43:0/3704357250 conn(0x560f0e823390 msgr2=0x560f0e9044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.467+0000 7f92898adb80 1 -- 172.21.15.43:0/3704357250 shutdown_connections 2024-05-27T07:22:31.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.467+0000 7f92898adb80 1 -- 172.21.15.43:0/3704357250 wait complete. 2024-05-27T07:22:31.459 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:30 smithi179 podman[73366]: 2024-05-27 07:22:30.99478039 +0000 UTC m=+1.079383373 container remove 399e5555e7257e577e3a56e8e149e7f494661110ee449ca298b00152fde4cfa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:22:31.459 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:31 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:22:31.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.468+0000 7f92898adb80 1 -- start start 2024-05-27T07:22:31.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.468+0000 7f92898adb80 1 Processor -- start 2024-05-27T07:22:31.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f92898adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e789e40 0x560f0e912710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f92898adb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e8759f0 0x560f0e912c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f92898adb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e908e70 0x560f0e913190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f92898adb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560f0e72cfa0 con 0x560f0e908e70 2024-05-27T07:22:31.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f92898adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560f0e7c7760 con 0x560f0e789e40 2024-05-27T07:22:31.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f92898adb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560f0e915b20 con 0x560f0e8759f0 2024-05-27T07:22:31.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f92898adb80 10 client.0 _ll_register_callbacks cb 0x560f0e7d0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f9278e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e908e70 0x560f0e913190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f9278e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e908e70 0x560f0e913190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38814/0 (socket says 172.21.15.43:38814) 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f9278e00700 1 -- 172.21.15.43:0/166810987 learned_addr learned my addr 172.21.15.43:0/166810987 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f9273400700 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e8759f0 0x560f0e912c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.469+0000 7f9273e00700 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e789e40 0x560f0e912710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.470+0000 7f9278e00700 1 -- 172.21.15.43:0/166810987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e8759f0 msgr2=0x560f0e912c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.470+0000 7f9278e00700 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e8759f0 0x560f0e912c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.470+0000 7f9278e00700 1 -- 172.21.15.43:0/166810987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e789e40 msgr2=0x560f0e912710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.470+0000 7f9278e00700 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e789e40 0x560f0e912710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.470+0000 7f9278e00700 1 -- 172.21.15.43:0/166810987 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f0e7768d0 con 0x560f0e908e70 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.470+0000 7f9278e00700 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e908e70 0x560f0e913190 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f926c009a00 tx=0x7f926c01ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25343aae7b578a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:31.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.470+0000 7f925e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:31.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.470+0000 7f925e000700 1 -- 172.21.15.43:0/166810987 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c031070 con 0x560f0e908e70 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.470+0000 7f925e000700 1 -- 172.21.15.43:0/166810987 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9408+0+0 (secure 0 0 0) 0x7f926c004a70 con 0x560f0e908e70 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.471+0000 7f92898adb80 1 -- 172.21.15.43:0/166810987 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f926c01abe0 con 0x560f0e908e70 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.471+0000 7f925e000700 1 -- 172.21.15.43:0/166810987 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f926c03d050 con 0x560f0e908e70 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.471+0000 7f92898adb80 10 client.16719 fetch_fsmap learned FSMap version 87 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.471+0000 7f92898adb80 1 -- 172.21.15.43:0/166810987 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x560f0e7f7060 con 0x560f0e908e70 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.471+0000 7f925e000700 1 -- 172.21.15.43:0/166810987 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f926c01abe0 con 0x560f0e908e70 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.471+0000 7f925e000700 1 -- 172.21.15.43:0/166810987 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 87) ==== 5966+0+0 (secure 0 0 0) 0x7f926c021020 con 0x560f0e908e70 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.472+0000 7f92898adb80 10 client.16719 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.472+0000 7f92898adb80 10 client.16719 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.472+0000 7f92898adb80 20 client.16719 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:31.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.472+0000 7f92898adb80 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560f0e922c40 0x560f0e925090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.472+0000 7f92898adb80 4 client.16719 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.472+0000 7f92898adb80 1 -- 172.21.15.43:0/166810987 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f0e838d90 con 0x560f0e922c40 2024-05-27T07:22:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.472+0000 7f9273e00700 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560f0e922c40 0x560f0e925090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.473+0000 7f9273e00700 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560f0e922c40 0x560f0e925090 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=91d87aea6f7be86e server_cookie=aae024189745b50c in_seq=0 out_seq=0 2024-05-27T07:22:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.473+0000 7f925e000700 10 client.16719 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:22:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.475+0000 7f925e000700 1 -- 172.21.15.43:0/166810987 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560f0e838d90 con 0x560f0e922c40 2024-05-27T07:22:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.475+0000 7f925e000700 10 client.16719 handle_command_reply: tid=0 2024-05-27T07:22:31.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 client.16719 shutdown 2024-05-27T07:22:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 -- 172.21.15.43:0/166810987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e908e70 msgr2=0x560f0e913190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e908e70 0x560f0e913190 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f926c009a00 tx=0x7f926c01ae40 comp rx=0 tx=0).stop 2024-05-27T07:22:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 -- 172.21.15.43:0/166810987 shutdown_connections 2024-05-27T07:22:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e908e70 0x560f0e913190 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560f0e922c40 0x560f0e925090 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e789e40 0x560f0e912710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 --2- 172.21.15.43:0/166810987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e8759f0 0x560f0e912c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 -- 172.21.15.43:0/166810987 >> 172.21.15.43:0/166810987 conn(0x560f0e823390 msgr2=0x560f0e908350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f925e000700 0 client.16719 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:22:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 -- 172.21.15.43:0/166810987 shutdown_connections 2024-05-27T07:22:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 1 -- 172.21.15.43:0/166810987 wait complete. 2024-05-27T07:22:31.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.476+0000 7f92898adb80 20 client.16719 trim_cache size 0 max 16384 2024-05-27T07:22:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.494+0000 7f92898adb80 1 Processor -- start 2024-05-27T07:22:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.495+0000 7f92898adb80 1 -- start start 2024-05-27T07:22:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.495+0000 7f92898adb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e789e40 0x560f0e91ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.495+0000 7f92898adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e8759f0 0x560f0e903e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.495+0000 7f92898adb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 0x560f0e9043c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.495+0000 7f92898adb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f92741b3230 con 0x560f0e789e40 2024-05-27T07:22:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.495+0000 7f92898adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f92741b3900 con 0x560f0e8759f0 2024-05-27T07:22:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.495+0000 7f92898adb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f92741b1020 con 0x560f0e908e70 2024-05-27T07:22:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.495+0000 7f9273e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e789e40 0x560f0e91ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9273e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e789e40 0x560f0e91ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38828/0 (socket says 172.21.15.43:38828) 2024-05-27T07:22:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9273e00700 1 -- 172.21.15.43:0/94864167 learned_addr learned my addr 172.21.15.43:0/94864167 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9278e00700 1 --2- 172.21.15.43:0/94864167 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 0x560f0e9043c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9273400700 1 --2- 172.21.15.43:0/94864167 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e8759f0 0x560f0e903e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9273e00700 1 -- 172.21.15.43:0/94864167 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 msgr2=0x560f0e9043c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9273e00700 1 --2- 172.21.15.43:0/94864167 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 0x560f0e9043c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9273e00700 1 -- 172.21.15.43:0/94864167 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e8759f0 msgr2=0x560f0e903e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9273e00700 1 --2- 172.21.15.43:0/94864167 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e8759f0 0x560f0e903e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9273e00700 1 -- 172.21.15.43:0/94864167 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f0e913500 con 0x560f0e789e40 2024-05-27T07:22:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9273400700 1 --2- 172.21.15.43:0/94864167 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e8759f0 0x560f0e903e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.496+0000 7f9273e00700 1 --2- 172.21.15.43:0/94864167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e789e40 0x560f0e91ebc0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f92640a6b90 tx=0x7f92640e5fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9351e03dfb6cee94 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.497+0000 7f9268c00700 1 -- 172.21.15.43:0/94864167 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92640b3220 con 0x560f0e789e40 2024-05-27T07:22:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.497+0000 7f9268c00700 1 -- 172.21.15.43:0/94864167 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f92640aa620 con 0x560f0e789e40 2024-05-27T07:22:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.497+0000 7f9268c00700 1 -- 172.21.15.43:0/94864167 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92640b38e0 con 0x560f0e789e40 2024-05-27T07:22:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.498+0000 7f92898adb80 1 -- 172.21.15.43:0/94864167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e789e40 msgr2=0x560f0e91ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.498+0000 7f92898adb80 1 --2- 172.21.15.43:0/94864167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e789e40 0x560f0e91ebc0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f92640a6b90 tx=0x7f92640e5fa0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.498+0000 7f92898adb80 1 -- 172.21.15.43:0/94864167 shutdown_connections 2024-05-27T07:22:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.498+0000 7f92898adb80 1 --2- 172.21.15.43:0/94864167 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 0x560f0e9043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.498+0000 7f92898adb80 1 --2- 172.21.15.43:0/94864167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e789e40 0x560f0e91ebc0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.498+0000 7f92898adb80 1 --2- 172.21.15.43:0/94864167 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e8759f0 0x560f0e903e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.498+0000 7f92898adb80 1 -- 172.21.15.43:0/94864167 >> 172.21.15.43:0/94864167 conn(0x560f0e823390 msgr2=0x560f0e911000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.498+0000 7f92898adb80 1 -- 172.21.15.43:0/94864167 shutdown_connections 2024-05-27T07:22:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.498+0000 7f92898adb80 1 -- 172.21.15.43:0/94864167 wait complete. 2024-05-27T07:22:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.499+0000 7f92898adb80 1 -- start start 2024-05-27T07:22:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.499+0000 7f92898adb80 1 Processor -- start 2024-05-27T07:22:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.499+0000 7f92898adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e789e40 0x560f0e90d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.499+0000 7f92898adb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e90bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.499+0000 7f92898adb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 0x560f0e90c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.499+0000 7f92898adb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f92741b16f0 con 0x560f0e8759f0 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.499+0000 7f92898adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f92741b2b60 con 0x560f0e789e40 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.499+0000 7f92898adb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f92741b1dc0 con 0x560f0e908e70 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.499+0000 7f92898adb80 10 client.0 _ll_register_callbacks cb 0x560f0e7d0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9273400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e90bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9273400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e90bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38838/0 (socket says 172.21.15.43:38838) 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9273400700 1 -- 172.21.15.43:0/1439566823 learned_addr learned my addr 172.21.15.43:0/1439566823 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9273e00700 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e789e40 0x560f0e90d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9278e00700 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 0x560f0e90c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9273400700 1 -- 172.21.15.43:0/1439566823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 msgr2=0x560f0e90c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9273400700 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 0x560f0e90c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9273400700 1 -- 172.21.15.43:0/1439566823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e789e40 msgr2=0x560f0e90d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9273400700 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e789e40 0x560f0e90d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9273400700 1 -- 172.21.15.43:0/1439566823 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f0e90c850 con 0x560f0e8759f0 2024-05-27T07:22:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.500+0000 7f9278e00700 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 0x560f0e90c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.501+0000 7f9273400700 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e90bc70 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f927406c670 tx=0x7f92600109f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac08ab002a3fa38 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.501+0000 7f925e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.501+0000 7f925e000700 1 -- 172.21.15.43:0/1439566823 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260004520 con 0x560f0e8759f0 2024-05-27T07:22:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.501+0000 7f925e000700 1 -- 172.21.15.43:0/1439566823 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9408+0+0 (secure 0 0 0) 0x7f9260004830 con 0x560f0e8759f0 2024-05-27T07:22:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.501+0000 7f92898adb80 1 -- 172.21.15.43:0/1439566823 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92640aa620 con 0x560f0e8759f0 2024-05-27T07:22:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.501+0000 7f925e000700 1 -- 172.21.15.43:0/1439566823 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f9260004a30 con 0x560f0e8759f0 2024-05-27T07:22:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.501+0000 7f92898adb80 10 client.16725 fetch_fsmap learned FSMap version 87 2024-05-27T07:22:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.501+0000 7f92898adb80 1 -- 172.21.15.43:0/1439566823 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x560f0e91abe0 con 0x560f0e8759f0 2024-05-27T07:22:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.502+0000 7f925e000700 1 -- 172.21.15.43:0/1439566823 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f92640aa620 con 0x560f0e8759f0 2024-05-27T07:22:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.502+0000 7f925e000700 1 -- 172.21.15.43:0/1439566823 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 87) ==== 5966+0+0 (secure 0 0 0) 0x7f9260003710 con 0x560f0e8759f0 2024-05-27T07:22:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.502+0000 7f92898adb80 10 client.16725 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:22:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.502+0000 7f92898adb80 10 client.16725 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:22:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.502+0000 7f92898adb80 20 client.16725 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.502+0000 7f92898adb80 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560f0e91b3a0 0x560f0e98f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.502+0000 7f92898adb80 4 client.16725 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:22:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.502+0000 7f92898adb80 1 -- 172.21.15.43:0/1439566823 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f0e832570 con 0x560f0e91b3a0 2024-05-27T07:22:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.503+0000 7f9273e00700 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560f0e91b3a0 0x560f0e98f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.503+0000 7f9273e00700 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560f0e91b3a0 0x560f0e98f750 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b13667b090cc0d7c server_cookie=5620a7ee1b60b3bd in_seq=0 out_seq=0 2024-05-27T07:22:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.503+0000 7f925e000700 10 client.16725 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:22:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f925e000700 1 -- 172.21.15.43:0/1439566823 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f0e832570 con 0x560f0e91b3a0 2024-05-27T07:22:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f925e000700 10 client.16725 handle_command_reply: tid=0 2024-05-27T07:22:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f92898adb80 1 client.16725 shutdown 2024-05-27T07:22:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f92898adb80 1 -- 172.21.15.43:0/1439566823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 msgr2=0x560f0e90bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f92898adb80 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e90bc70 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f927406c670 tx=0x7f92600109f0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f92898adb80 1 -- 172.21.15.43:0/1439566823 shutdown_connections 2024-05-27T07:22:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f92898adb80 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f0e908e70 0x560f0e90c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f92898adb80 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560f0e91b3a0 0x560f0e98f750 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f92898adb80 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f0e789e40 0x560f0e90d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f92898adb80 1 --2- 172.21.15.43:0/1439566823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f0e8759f0 0x560f0e90bc70 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f92898adb80 1 -- 172.21.15.43:0/1439566823 >> 172.21.15.43:0/1439566823 conn(0x560f0e823390 msgr2=0x560f0e903d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.504+0000 7f925e000700 0 client.16725 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:22:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.505+0000 7f92898adb80 1 -- 172.21.15.43:0/1439566823 shutdown_connections 2024-05-27T07:22:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.505+0000 7f92898adb80 1 -- 172.21.15.43:0/1439566823 wait complete. 2024-05-27T07:22:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.505+0000 7f92898adb80 20 client.16725 trim_cache size 0 max 16384 2024-05-27T07:22:31.496 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:22:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 -- 172.21.15.43:0/3993620746 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f92580b29f0 msgr2=0x7f92580b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f92580b29f0 0x7f92580b4f00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f926c01c000 tx=0x7f926c01e000 comp rx=0 tx=0).stop 2024-05-27T07:22:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 -- 172.21.15.43:0/3993620746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92741045e0 msgr2=0x7f927419d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92741045e0 0x7f927419d000 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9264005950 tx=0x7f9264009480 comp rx=0 tx=0).stop 2024-05-27T07:22:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 -- 172.21.15.43:0/3993620746 shutdown_connections 2024-05-27T07:22:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f92580b29f0 0x7f92580b4f00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9274109860 0x7f92741a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92741045e0 0x7f927419d000 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 --2- 172.21.15.43:0/3993620746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9274108f10 0x7f927419d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 -- 172.21.15.43:0/3993620746 >> 172.21.15.43:0/3993620746 conn(0x7f9274100030 msgr2=0x7f9274100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 -- 172.21.15.43:0/3993620746 shutdown_connections 2024-05-27T07:22:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:31.506+0000 7f927ae00700 1 -- 172.21.15.43:0/3993620746 wait complete. 2024-05-27T07:22:31.958 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:31 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:22:31.958 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:31 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.551s CPU time. 2024-05-27T07:22:32.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46746 0.62 MB/sec execute 276 sec latency 58.040 ms 2024-05-27T07:22:32.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:22:32.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:32 smithi132 ceph-mon[35283]: pgmap v289: 129 pgs: 129 active+clean; 81 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 605 KiB/s wr, 54 op/s 2024-05-27T07:22:32.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:32 smithi043 ceph-mon[31531]: pgmap v289: 129 pgs: 129 active+clean; 81 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 605 KiB/s wr, 54 op/s 2024-05-27T07:22:32.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:32 smithi179 ceph-mon[35888]: pgmap v289: 129 pgs: 129 active+clean; 81 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 605 KiB/s wr, 54 op/s 2024-05-27T07:22:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:33.215 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46865 0.62 MB/sec execute 277 sec latency 54.795 ms 2024-05-27T07:22:34.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46981 0.62 MB/sec execute 278 sec latency 53.584 ms 2024-05-27T07:22:34.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:34 smithi043 ceph-mon[31531]: pgmap v290: 129 pgs: 129 active+clean; 81 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 605 KiB/s wr, 54 op/s 2024-05-27T07:22:34.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:34 smithi132 ceph-mon[35283]: pgmap v290: 129 pgs: 129 active+clean; 81 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 605 KiB/s wr, 54 op/s 2024-05-27T07:22:34.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:34 smithi179 ceph-mon[35888]: pgmap v290: 129 pgs: 129 active+clean; 81 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 605 KiB/s wr, 54 op/s 2024-05-27T07:22:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.096+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3365939375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5d410b890 msgr2=0x7ff5d41131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.096+0000 7ff5d9c00700 1 --2- 172.21.15.43:0/3365939375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5d410b890 0x7ff5d41131f0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00b0d0 tx=0x7ff5cc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.096+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3365939375 shutdown_connections 2024-05-27T07:22:35.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.096+0000 7ff5d9c00700 1 --2- 172.21.15.43:0/3365939375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5d410b890 0x7ff5d41131f0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.096+0000 7ff5d9c00700 1 --2- 172.21.15.43:0/3365939375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d4106610 0x7ff5d4106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.096+0000 7ff5d9c00700 1 --2- 172.21.15.43:0/3365939375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d410af40 0x7ff5d410b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.096+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3365939375 >> 172.21.15.43:0/3365939375 conn(0x7ff5d4102020 msgr2=0x7ff5d4104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:35.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.097+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3365939375 shutdown_connections 2024-05-27T07:22:35.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.097+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3365939375 wait complete. 2024-05-27T07:22:35.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.098+0000 7ff5d9c00700 1 Processor -- start 2024-05-27T07:22:35.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.098+0000 7ff5d9c00700 1 -- start start 2024-05-27T07:22:35.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.099+0000 7ff5d9c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d4106610 0x7ff5d419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.099+0000 7ff5d9c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5d410af40 0x7ff5d419d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.099+0000 7ff5d9c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d410b890 0x7ff5d41a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.099+0000 7ff5d9c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d41163c0 con 0x7ff5d410af40 2024-05-27T07:22:35.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.099+0000 7ff5d9c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d4116240 con 0x7ff5d410b890 2024-05-27T07:22:35.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.099+0000 7ff5d9c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d4116540 con 0x7ff5d4106610 2024-05-27T07:22:35.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.099+0000 7ff5cbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5d410af40 0x7ff5d419d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.099+0000 7ff5cbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5d410af40 0x7ff5d419d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53490/0 (socket says 172.21.15.43:53490) 2024-05-27T07:22:35.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.099+0000 7ff5cbe00700 1 -- 172.21.15.43:0/3536017627 learned_addr learned my addr 172.21.15.43:0/3536017627 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.100+0000 7ff5d3e00700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d4106610 0x7ff5d419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.100+0000 7ff5cbe00700 1 -- 172.21.15.43:0/3536017627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d4106610 msgr2=0x7ff5d419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.100+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d4106610 0x7ff5d419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.100+0000 7ff5cbe00700 1 -- 172.21.15.43:0/3536017627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d410b890 msgr2=0x7ff5d41a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.100+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d410b890 0x7ff5d41a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.100+0000 7ff5cbe00700 1 -- 172.21.15.43:0/3536017627 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d41a1be0 con 0x7ff5d410af40 2024-05-27T07:22:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.100+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5d410af40 0x7ff5d419d2b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00d980 tx=0x7ff5bc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.101+0000 7ff5d1600700 1 -- 172.21.15.43:0/3536017627 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc017070 con 0x7ff5d410af40 2024-05-27T07:22:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.101+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3536017627 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d41a1ed0 con 0x7ff5d410af40 2024-05-27T07:22:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.101+0000 7ff5d1600700 1 -- 172.21.15.43:0/3536017627 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5bc010460 con 0x7ff5d410af40 2024-05-27T07:22:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.101+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3536017627 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d41a97f0 con 0x7ff5d410af40 2024-05-27T07:22:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.101+0000 7ff5d1600700 1 -- 172.21.15.43:0/3536017627 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc020060 con 0x7ff5d410af40 2024-05-27T07:22:35.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.105+0000 7ff5d1600700 1 -- 172.21.15.43:0/3536017627 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff5bc010610 con 0x7ff5d410af40 2024-05-27T07:22:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.106+0000 7ff5e9d97b80 1 Processor -- start 2024-05-27T07:22:35.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.106+0000 7ff5e9d97b80 1 -- start start 2024-05-27T07:22:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad843690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 0x561dad843bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad857ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561dad770e40 con 0x561dad7af9f0 2024-05-27T07:22:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561dad7609e0 con 0x561dad6c3e40 2024-05-27T07:22:35.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561dad764a20 con 0x561dad855ad0 2024-05-27T07:22:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5d1600700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5b80b29f0 0x7ff5b80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5d1600700 1 -- 172.21.15.43:0/3536017627 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9408+0+0 (secure 0 0 0) 0x7ff5bc0e24f0 con 0x7ff5d410af40 2024-05-27T07:22:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5d9200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad843690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5d9200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad843690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53496/0 (socket says 172.21.15.43:53496) 2024-05-27T07:22:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5d9200700 1 -- 172.21.15.43:0/3029089384 learned_addr learned my addr 172.21.15.43:0/3029089384 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5d9200700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5b80b29f0 0x7ff5b80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/3029089384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad857ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.107+0000 7ff5d3e00700 1 --2- 172.21.15.43:0/3029089384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 0x561dad843bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.108+0000 7ff5d9200700 1 -- 172.21.15.43:0/3029089384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 msgr2=0x561dad857ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.108+0000 7ff5d9200700 1 --2- 172.21.15.43:0/3029089384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad857ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.108+0000 7ff5d9200700 1 -- 172.21.15.43:0/3029089384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 msgr2=0x561dad843bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.108+0000 7ff5d9200700 1 --2- 172.21.15.43:0/3029089384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 0x561dad843bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.108+0000 7ff5d9200700 1 -- 172.21.15.43:0/3029089384 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dad729420 con 0x561dad7af9f0 2024-05-27T07:22:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.110+0000 7ff5d9200700 1 --2- 172.21.15.43:0/3029089384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad843690 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009500 tx=0x7ff5cc009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a468840181c5a2c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.110+0000 7ff5c9600700 1 -- 172.21.15.43:0/3029089384 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc02e070 con 0x561dad7af9f0 2024-05-27T07:22:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.110+0000 7ff5d9200700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5b80b29f0 0x7ff5b80b4f00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc02b000 tx=0x7ff5cc029000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:35.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.110+0000 7ff5c9600700 1 -- 172.21.15.43:0/3029089384 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5cc007a50 con 0x561dad7af9f0 2024-05-27T07:22:35.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.110+0000 7ff5c9600700 1 -- 172.21.15.43:0/3029089384 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc0296b0 con 0x561dad7af9f0 2024-05-27T07:22:35.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.112+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/3029089384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 msgr2=0x561dad843690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:35.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.112+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/3029089384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad843690 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009500 tx=0x7ff5cc009530 comp rx=0 tx=0).stop 2024-05-27T07:22:35.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.112+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/3029089384 shutdown_connections 2024-05-27T07:22:35.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.112+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/3029089384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad857ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.112+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/3029089384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad843690 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.112+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/3029089384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 0x561dad843bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.112+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/3029089384 >> 172.21.15.43:0/3029089384 conn(0x561dad75d390 msgr2=0x561dad83e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:35.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.112+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/3029089384 shutdown_connections 2024-05-27T07:22:35.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.112+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/3029089384 wait complete. 2024-05-27T07:22:35.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.113+0000 7ff5e9d97b80 1 -- start start 2024-05-27T07:22:35.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.113+0000 7ff5e9d97b80 1 Processor -- start 2024-05-27T07:22:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.114+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 0x561dad852a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.114+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad852fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.114+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad853510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.114+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561dad666fa0 con 0x561dad7af9f0 2024-05-27T07:22:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.114+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561dad701760 con 0x561dad6c3e40 2024-05-27T07:22:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.114+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561dad858580 con 0x561dad855ad0 2024-05-27T07:22:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.114+0000 7ff5e9d97b80 10 client.0 _ll_register_callbacks cb 0x561dad70ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.115+0000 7ff5d3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 0x561dad852a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.115+0000 7ff5cbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad852fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.115+0000 7ff5d9200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad853510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.115+0000 7ff5cbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad852fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53512/0 (socket says 172.21.15.43:53512) 2024-05-27T07:22:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.115+0000 7ff5cbe00700 1 -- 172.21.15.43:0/1429445589 learned_addr learned my addr 172.21.15.43:0/1429445589 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.115+0000 7ff5d3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 0x561dad852a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:60030/0 (socket says 172.21.15.43:60030) 2024-05-27T07:22:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.116+0000 7ff5cbe00700 1 -- 172.21.15.43:0/1429445589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 msgr2=0x561dad853510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.116+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad853510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.116+0000 7ff5cbe00700 1 -- 172.21.15.43:0/1429445589 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 msgr2=0x561dad852a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:35.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.116+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 0x561dad852a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.116+0000 7ff5cbe00700 1 -- 172.21.15.43:0/1429445589 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dad6b08d0 con 0x561dad7af9f0 2024-05-27T07:22:35.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.116+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad852fd0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0df930 tx=0x7ff5bc010bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618e3faa7db45319 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:35.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.116+0000 7ff5c2000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:35.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.117+0000 7ff5c2000700 1 -- 172.21.15.43:0/1429445589 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0e2350 con 0x561dad7af9f0 2024-05-27T07:22:35.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.117+0000 7ff5c2000700 1 -- 172.21.15.43:0/1429445589 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9408+0+0 (secure 0 0 0) 0x7ff5bc0a96c0 con 0x561dad7af9f0 2024-05-27T07:22:35.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.117+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/1429445589 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5cc007a50 con 0x561dad7af9f0 2024-05-27T07:22:35.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.117+0000 7ff5c2000700 1 -- 172.21.15.43:0/1429445589 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7ff5bc0a9b60 con 0x561dad7af9f0 2024-05-27T07:22:35.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.117+0000 7ff5e9d97b80 10 client.16737 fetch_fsmap learned FSMap version 87 2024-05-27T07:22:35.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.117+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/1429445589 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x561dad731060 con 0x561dad7af9f0 2024-05-27T07:22:35.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.118+0000 7ff5c2000700 1 -- 172.21.15.43:0/1429445589 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5cc007a50 con 0x561dad7af9f0 2024-05-27T07:22:35.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.118+0000 7ff5c2000700 1 -- 172.21.15.43:0/1429445589 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 87) ==== 5966+0+0 (secure 0 0 0) 0x7ff5bc0b1ec0 con 0x561dad7af9f0 2024-05-27T07:22:35.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.118+0000 7ff5e9d97b80 10 client.16737 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:22:35.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.118+0000 7ff5e9d97b80 10 client.16737 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:22:35.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.118+0000 7ff5e9d97b80 20 client.16737 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:35.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.118+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561dad8658e0 0x561dad867d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.118+0000 7ff5e9d97b80 4 client.16737 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:35.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.118+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/1429445589 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dad772d90 con 0x561dad8658e0 2024-05-27T07:22:35.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.119+0000 7ff5d3e00700 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561dad8658e0 0x561dad867d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.120+0000 7ff5d3e00700 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561dad8658e0 0x561dad867d30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2cccf2e1d38b1d2 server_cookie=3317fcca8b4b69be in_seq=0 out_seq=0 2024-05-27T07:22:35.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.120+0000 7ff5c2000700 10 client.16737 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:22:35.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.122+0000 7ff5c2000700 1 -- 172.21.15.43:0/1429445589 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561dad772d90 con 0x561dad8658e0 2024-05-27T07:22:35.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.122+0000 7ff5c2000700 10 client.16737 handle_command_reply: tid=0 2024-05-27T07:22:35.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.122+0000 7ff5e9d97b80 1 client.16737 shutdown 2024-05-27T07:22:35.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.122+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/1429445589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 msgr2=0x561dad852fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:35.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.122+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad852fd0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0df930 tx=0x7ff5bc010bb0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.122+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/1429445589 shutdown_connections 2024-05-27T07:22:35.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.122+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561dad8658e0 0x561dad867d30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.123+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad853510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.123+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad6c3e40 0x561dad852a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.123+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/1429445589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad852fd0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.123+0000 7ff5c2000700 0 client.16737 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:22:35.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.123+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/1429445589 >> 172.21.15.43:0/1429445589 conn(0x561dad75d390 msgr2=0x561dad843710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:35.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.123+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/1429445589 shutdown_connections 2024-05-27T07:22:35.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.123+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/1429445589 wait complete. 2024-05-27T07:22:35.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.123+0000 7ff5e9d97b80 20 client.16737 trim_cache size 0 max 16384 2024-05-27T07:22:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.143+0000 7ff5e9d97b80 1 Processor -- start 2024-05-27T07:22:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.143+0000 7ff5e9d97b80 1 -- start start 2024-05-27T07:22:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.143+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad6c3e40 0x561dad8519f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.143+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad7af9f0 0x561dad83e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.143+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad842830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.143+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d41aec20 con 0x561dad6c3e40 2024-05-27T07:22:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d41af2f0 con 0x561dad7af9f0 2024-05-27T07:22:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d41aca10 con 0x561dad855ad0 2024-05-27T07:22:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5d3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad6c3e40 0x561dad8519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5d3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad6c3e40 0x561dad8519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53518/0 (socket says 172.21.15.43:53518) 2024-05-27T07:22:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5d3e00700 1 -- 172.21.15.43:0/614046426 learned_addr learned my addr 172.21.15.43:0/614046426 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5d9200700 1 --2- 172.21.15.43:0/614046426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad842830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/614046426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad7af9f0 0x561dad83e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5d3e00700 1 -- 172.21.15.43:0/614046426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 msgr2=0x561dad842830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5d3e00700 1 --2- 172.21.15.43:0/614046426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad842830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5d3e00700 1 -- 172.21.15.43:0/614046426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad7af9f0 msgr2=0x561dad83e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5d3e00700 1 --2- 172.21.15.43:0/614046426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad7af9f0 0x561dad83e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.144+0000 7ff5d3e00700 1 -- 172.21.15.43:0/614046426 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dad83f180 con 0x561dad6c3e40 2024-05-27T07:22:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.145+0000 7ff5d3e00700 1 --2- 172.21.15.43:0/614046426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad6c3e40 0x561dad8519f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4000c00 tx=0x7ff5c400b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2369068cfc0206 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.145+0000 7ff5c8c00700 1 -- 172.21.15.43:0/614046426 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4014c20 con 0x561dad6c3e40 2024-05-27T07:22:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.145+0000 7ff5c8c00700 1 -- 172.21.15.43:0/614046426 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5c40044c0 con 0x561dad6c3e40 2024-05-27T07:22:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.145+0000 7ff5c8c00700 1 -- 172.21.15.43:0/614046426 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4003e40 con 0x561dad6c3e40 2024-05-27T07:22:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.146+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/614046426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad6c3e40 msgr2=0x561dad8519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.146+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/614046426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad6c3e40 0x561dad8519f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4000c00 tx=0x7ff5c400b4d0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.147+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/614046426 shutdown_connections 2024-05-27T07:22:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.147+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/614046426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad855ad0 0x561dad842830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.147+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/614046426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad6c3e40 0x561dad8519f0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.147+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/614046426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad7af9f0 0x561dad83e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.147+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/614046426 >> 172.21.15.43:0/614046426 conn(0x561dad75d390 msgr2=0x561dad842120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:35.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.147+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/614046426 shutdown_connections 2024-05-27T07:22:35.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.147+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/614046426 wait complete. 2024-05-27T07:22:35.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.147+0000 7ff5e9d97b80 1 -- start start 2024-05-27T07:22:35.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.148+0000 7ff5e9d97b80 1 Processor -- start 2024-05-27T07:22:35.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.148+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad6c3e40 0x561dad85db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.148+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad85e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.148+0000 7ff5e9d97b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad855ad0 0x561dad858fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.148+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d41ad0e0 con 0x561dad7af9f0 2024-05-27T07:22:35.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.148+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d41ae550 con 0x561dad855ad0 2024-05-27T07:22:35.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.148+0000 7ff5e9d97b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d41ad7b0 con 0x561dad6c3e40 2024-05-27T07:22:35.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.148+0000 7ff5e9d97b80 10 client.0 _ll_register_callbacks cb 0x561dad70ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:35.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5cbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad85e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5cbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad85e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53528/0 (socket says 172.21.15.43:53528) 2024-05-27T07:22:35.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5cbe00700 1 -- 172.21.15.43:0/4256235538 learned_addr learned my addr 172.21.15.43:0/4256235538 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:35.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5d3e00700 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad6c3e40 0x561dad85db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5d9200700 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad855ad0 0x561dad858fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5cbe00700 1 -- 172.21.15.43:0/4256235538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad6c3e40 msgr2=0x561dad85db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad6c3e40 0x561dad85db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5cbe00700 1 -- 172.21.15.43:0/4256235538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad855ad0 msgr2=0x561dad858fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad855ad0 0x561dad858fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5cbe00700 1 -- 172.21.15.43:0/4256235538 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dad859750 con 0x561dad7af9f0 2024-05-27T07:22:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.149+0000 7ff5d9200700 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad855ad0 0x561dad858fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.150+0000 7ff5cbe00700 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad85e080 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0b2230 tx=0x7ff5bc06efa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c5687cad898cca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.150+0000 7ff5c2000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.150+0000 7ff5c2000700 1 -- 172.21.15.43:0/4256235538 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0a9460 con 0x561dad7af9f0 2024-05-27T07:22:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.150+0000 7ff5c2000700 1 -- 172.21.15.43:0/4256235538 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9408+0+0 (secure 0 0 0) 0x7ff5bc0a98d0 con 0x561dad7af9f0 2024-05-27T07:22:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.150+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/4256235538 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5c40044c0 con 0x561dad7af9f0 2024-05-27T07:22:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.150+0000 7ff5c2000700 1 -- 172.21.15.43:0/4256235538 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7ff5bc0b22c0 con 0x561dad7af9f0 2024-05-27T07:22:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.150+0000 7ff5e9d97b80 10 client.16743 fetch_fsmap learned FSMap version 87 2024-05-27T07:22:35.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.150+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/4256235538 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x561dad8cff00 con 0x561dad7af9f0 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.151+0000 7ff5c2000700 1 -- 172.21.15.43:0/4256235538 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff5c40044c0 con 0x561dad7af9f0 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.151+0000 7ff5c2000700 1 -- 172.21.15.43:0/4256235538 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 87) ==== 5966+0+0 (secure 0 0 0) 0x7ff5bc0a86c0 con 0x561dad7af9f0 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.151+0000 7ff5e9d97b80 10 client.16743 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.151+0000 7ff5e9d97b80 10 client.16743 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.151+0000 7ff5e9d97b80 20 client.16743 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.151+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561dad8d05b0 0x561dad8d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.151+0000 7ff5e9d97b80 4 client.16743 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.151+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/4256235538 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561dad76c570 con 0x561dad8d05b0 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.152+0000 7ff5d3e00700 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561dad8d05b0 0x561dad8d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.153+0000 7ff5d3e00700 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561dad8d05b0 0x561dad8d29d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dfae2fb1cac55562 server_cookie=9941d15dfb2c07eb in_seq=0 out_seq=0 2024-05-27T07:22:35.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.153+0000 7ff5c2000700 10 client.16743 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:22:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.153+0000 7ff5c2000700 1 -- 172.21.15.43:0/4256235538 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561dad76c570 con 0x561dad8d05b0 2024-05-27T07:22:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.153+0000 7ff5c2000700 10 client.16743 handle_command_reply: tid=0 2024-05-27T07:22:35.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.153+0000 7ff5e9d97b80 1 client.16743 shutdown 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/4256235538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 msgr2=0x561dad85e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad85e080 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0b2230 tx=0x7ff5bc06efa0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/4256235538 shutdown_connections 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x561dad8d05b0 0x561dad8d29d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dad855ad0 0x561dad858fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dad6c3e40 0x561dad85db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 1 --2- 172.21.15.43:0/4256235538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dad7af9f0 0x561dad85e080 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/4256235538 >> 172.21.15.43:0/4256235538 conn(0x561dad75d390 msgr2=0x561dad852870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5c2000700 0 client.16743 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/4256235538 shutdown_connections 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 1 -- 172.21.15.43:0/4256235538 wait complete. 2024-05-27T07:22:35.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.154+0000 7ff5e9d97b80 20 client.16743 trim_cache size 0 max 16384 2024-05-27T07:22:35.151 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:22:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.155+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3536017627 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5b80b29f0 msgr2=0x7ff5b80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.155+0000 7ff5d9c00700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5b80b29f0 0x7ff5b80b4f00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc02b000 tx=0x7ff5cc029000 comp rx=0 tx=0).stop 2024-05-27T07:22:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.155+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3536017627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5d410af40 msgr2=0x7ff5d419d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.155+0000 7ff5d9c00700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5d410af40 0x7ff5d419d2b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00d980 tx=0x7ff5bc00dd40 comp rx=0 tx=0).stop 2024-05-27T07:22:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.155+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3536017627 shutdown_connections 2024-05-27T07:22:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.155+0000 7ff5d9c00700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5b80b29f0 0x7ff5b80b4f00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.155+0000 7ff5d9c00700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5d410b890 0x7ff5d41a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.155+0000 7ff5d9c00700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d4106610 0x7ff5d419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.155+0000 7ff5d9c00700 1 --2- 172.21.15.43:0/3536017627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5d410af40 0x7ff5d419d2b0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.155+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3536017627 >> 172.21.15.43:0/3536017627 conn(0x7ff5d4102020 msgr2=0x7ff5d410a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.156+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3536017627 shutdown_connections 2024-05-27T07:22:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:35.156+0000 7ff5d9c00700 1 -- 172.21.15.43:0/3536017627 wait complete. 2024-05-27T07:22:35.214 INFO:tasks.workunit.client.0.smithi043.stdout: 1 47094 0.62 MB/sec execute 279 sec latency 52.676 ms 2024-05-27T07:22:36.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:22:36.215 INFO:tasks.workunit.client.0.smithi043.stdout: 1 47216 0.62 MB/sec execute 280 sec latency 50.587 ms 2024-05-27T07:22:36.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:36 smithi043 ceph-mon[31531]: pgmap v291: 129 pgs: 129 active+clean; 81 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 337 KiB/s wr, 32 op/s 2024-05-27T07:22:36.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:36 smithi132 ceph-mon[35283]: pgmap v291: 129 pgs: 129 active+clean; 81 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 337 KiB/s wr, 32 op/s 2024-05-27T07:22:36.662 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:36.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:36 smithi179 ceph-mon[35888]: pgmap v291: 129 pgs: 129 active+clean; 81 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 337 KiB/s wr, 32 op/s 2024-05-27T07:22:37.215 INFO:tasks.workunit.client.0.smithi043.stdout: 1 47331 0.62 MB/sec execute 281 sec latency 53.593 ms 2024-05-27T07:22:38.215 INFO:tasks.workunit.client.0.smithi043.stdout: 1 47530 0.63 MB/sec execute 282 sec latency 54.805 ms 2024-05-27T07:22:38.531 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:38 smithi043 ceph-mon[31531]: pgmap v292: 129 pgs: 129 active+clean; 80 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 471 KiB/s wr, 57 op/s 2024-05-27T07:22:38.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:38 smithi132 ceph-mon[35283]: pgmap v292: 129 pgs: 129 active+clean; 80 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 471 KiB/s wr, 57 op/s 2024-05-27T07:22:38.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:38 smithi179 ceph-mon[35888]: pgmap v292: 129 pgs: 129 active+clean; 80 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 471 KiB/s wr, 57 op/s 2024-05-27T07:22:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.944+0000 7f955c200700 1 -- 172.21.15.43:0/293773023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9554076440 msgr2=0x7f95541106b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.944+0000 7f955c200700 1 --2- 172.21.15.43:0/293773023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9554076440 0x7f95541106b0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9540009990 tx=0x7f9540009ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.944+0000 7f955c200700 1 -- 172.21.15.43:0/293773023 shutdown_connections 2024-05-27T07:22:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.944+0000 7f955c200700 1 --2- 172.21.15.43:0/293773023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9554110bf0 0x7f9554113010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.944+0000 7f955c200700 1 --2- 172.21.15.43:0/293773023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9554075af0 0x7f9554075f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.944+0000 7f955c200700 1 --2- 172.21.15.43:0/293773023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9554076440 0x7f95541106b0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.944+0000 7f955c200700 1 -- 172.21.15.43:0/293773023 >> 172.21.15.43:0/293773023 conn(0x7f9554100030 msgr2=0x7f9554102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.944+0000 7f955c200700 1 -- 172.21.15.43:0/293773023 shutdown_connections 2024-05-27T07:22:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.944+0000 7f955c200700 1 -- 172.21.15.43:0/293773023 wait complete. 2024-05-27T07:22:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.946+0000 7f955c200700 1 Processor -- start 2024-05-27T07:22:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.946+0000 7f955c200700 1 -- start start 2024-05-27T07:22:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.946+0000 7f955c200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9554075af0 0x7f955419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.946+0000 7f955c200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9554076440 0x7f955419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.946+0000 7f955c200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9554110bf0 0x7f95541a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.946+0000 7f955c200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f95541165d0 con 0x7f9554076440 2024-05-27T07:22:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.946+0000 7f955c200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9554116450 con 0x7f9554075af0 2024-05-27T07:22:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.946+0000 7f955c200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9554116750 con 0x7f9554110bf0 2024-05-27T07:22:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f9558e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9554076440 0x7f955419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f9558e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9554076440 0x7f955419d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53550/0 (socket says 172.21.15.43:53550) 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f9558e00700 1 -- 172.21.15.43:0/2019748283 learned_addr learned my addr 172.21.15.43:0/2019748283 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f9559800700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9554075af0 0x7f955419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f955a200700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9554110bf0 0x7f95541a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f9558e00700 1 -- 172.21.15.43:0/2019748283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9554110bf0 msgr2=0x7f95541a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f9558e00700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9554110bf0 0x7f95541a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f9558e00700 1 -- 172.21.15.43:0/2019748283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9554075af0 msgr2=0x7f955419d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f9558e00700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9554075af0 0x7f955419d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f9558e00700 1 -- 172.21.15.43:0/2019748283 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95541a1ed0 con 0x7f9554076440 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.947+0000 7f9559800700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9554075af0 0x7f955419d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.948+0000 7f9558e00700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9554076440 0x7f955419d5a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9540000c00 tx=0x7f95400057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.948+0000 7f9552000700 1 -- 172.21.15.43:0/2019748283 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9540027070 con 0x7f9554076440 2024-05-27T07:22:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.948+0000 7f9552000700 1 -- 172.21.15.43:0/2019748283 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f954000fb00 con 0x7f9554076440 2024-05-27T07:22:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.948+0000 7f9552000700 1 -- 172.21.15.43:0/2019748283 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9540022bc0 con 0x7f9554076440 2024-05-27T07:22:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.948+0000 7f955c200700 1 -- 172.21.15.43:0/2019748283 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95541a2160 con 0x7f9554076440 2024-05-27T07:22:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.949+0000 7f955c200700 1 -- 172.21.15.43:0/2019748283 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95541a25f0 con 0x7f9554076440 2024-05-27T07:22:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.953+0000 7f9552000700 1 -- 172.21.15.43:0/2019748283 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f954000f670 con 0x7f9554076440 2024-05-27T07:22:38.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.954+0000 7f956ad2cb80 1 Processor -- start 2024-05-27T07:22:38.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.954+0000 7f956ad2cb80 1 -- start start 2024-05-27T07:22:38.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.955+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 0x55bc5acf47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.955+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5acf4d30 0x55bc5acee870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.955+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acef1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.955+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55bc5ac1ce40 con 0x55bc5aceedb0 2024-05-27T07:22:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.955+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bc5ac0c9e0 con 0x55bc5ab6fe40 2024-05-27T07:22:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.955+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55bc5ac10a20 con 0x55bc5acf4d30 2024-05-27T07:22:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.956+0000 7f9558e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acef1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.956+0000 7f9552000700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f953c0b2a70 0x7f953c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.956+0000 7f955a200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 0x55bc5acf47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.956+0000 7f9558e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acef1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53556/0 (socket says 172.21.15.43:53556) 2024-05-27T07:22:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.956+0000 7f9558e00700 1 -- 172.21.15.43:0/962383242 learned_addr learned my addr 172.21.15.43:0/962383242 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.956+0000 7f9552000700 1 -- 172.21.15.43:0/2019748283 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9408+0+0 (secure 0 0 0) 0x7f95400e3810 con 0x7f9554076440 2024-05-27T07:22:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.956+0000 7f955a200700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f953c0b2a70 0x7f953c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.956+0000 7f9559800700 1 --2- 172.21.15.43:0/962383242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5acf4d30 0x55bc5acee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.956+0000 7f955a200700 1 -- 172.21.15.43:0/962383242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5acf4d30 msgr2=0x55bc5acee870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.956+0000 7f955a200700 1 --2- 172.21.15.43:0/962383242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5acf4d30 0x55bc5acee870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.957+0000 7f955a200700 1 -- 172.21.15.43:0/962383242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 msgr2=0x55bc5acef1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.957+0000 7f955a200700 1 --2- 172.21.15.43:0/962383242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acef1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.957+0000 7f955a200700 1 -- 172.21.15.43:0/962383242 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc5abd5420 con 0x55bc5ab6fe40 2024-05-27T07:22:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.957+0000 7f955a200700 1 --2- 172.21.15.43:0/962383242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 0x55bc5acf47f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9554076ad0 tx=0x7f954c004320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2946ef3f3b160d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.957+0000 7f955a200700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f953c0b2a70 0x7f953c0b4f80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f954c01b000 tx=0x7f954c01d000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.958+0000 7f9546a00700 1 -- 172.21.15.43:0/962383242 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c01e040 con 0x55bc5ab6fe40 2024-05-27T07:22:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.958+0000 7f9546a00700 1 -- 172.21.15.43:0/962383242 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f954c003a00 con 0x55bc5ab6fe40 2024-05-27T07:22:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.958+0000 7f9546a00700 1 -- 172.21.15.43:0/962383242 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c01a9c0 con 0x55bc5ab6fe40 2024-05-27T07:22:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.959+0000 7f956ad2cb80 1 -- 172.21.15.43:0/962383242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 msgr2=0x55bc5acf47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.959+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/962383242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 0x55bc5acf47f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9554076ad0 tx=0x7f954c004320 comp rx=0 tx=0).stop 2024-05-27T07:22:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.959+0000 7f956ad2cb80 1 -- 172.21.15.43:0/962383242 shutdown_connections 2024-05-27T07:22:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.959+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/962383242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acef1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.959+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/962383242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 0x55bc5acf47f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.959+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/962383242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5acf4d30 0x55bc5acee870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.959+0000 7f956ad2cb80 1 -- 172.21.15.43:0/962383242 >> 172.21.15.43:0/962383242 conn(0x55bc5abcffc0 msgr2=0x55bc5ace98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.959+0000 7f956ad2cb80 1 -- 172.21.15.43:0/962383242 shutdown_connections 2024-05-27T07:22:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.959+0000 7f956ad2cb80 1 -- 172.21.15.43:0/962383242 wait complete. 2024-05-27T07:22:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.960+0000 7f956ad2cb80 1 -- start start 2024-05-27T07:22:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.960+0000 7f956ad2cb80 1 Processor -- start 2024-05-27T07:22:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5ab6fe40 0x55bc5acf84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acf8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5acf4d30 0x55bc5acf8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55bc5ab12fa0 con 0x55bc5aceedb0 2024-05-27T07:22:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bc5abad760 con 0x55bc5acf4d30 2024-05-27T07:22:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55bc5acfb7a0 con 0x55bc5ab6fe40 2024-05-27T07:22:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f956ad2cb80 10 client.0 _ll_register_callbacks cb 0x55bc5abb6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f9558e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acf8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f9558e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acf8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53568/0 (socket says 172.21.15.43:53568) 2024-05-27T07:22:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f9558e00700 1 -- 172.21.15.43:0/2389888825 learned_addr learned my addr 172.21.15.43:0/2389888825 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f9559800700 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5ab6fe40 0x55bc5acf84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.961+0000 7f955a200700 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5acf4d30 0x55bc5acf8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.962+0000 7f9558e00700 1 -- 172.21.15.43:0/2389888825 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5ab6fe40 msgr2=0x55bc5acf84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.962+0000 7f9558e00700 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5ab6fe40 0x55bc5acf84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.962+0000 7f9558e00700 1 -- 172.21.15.43:0/2389888825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5acf4d30 msgr2=0x55bc5acf8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.962+0000 7f9558e00700 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5acf4d30 0x55bc5acf8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.962+0000 7f9558e00700 1 -- 172.21.15.43:0/2389888825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc5ab5c8d0 con 0x55bc5aceedb0 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.962+0000 7f9558e00700 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acf8a30 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f95400a6850 tx=0x7f95400e1110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730ed5bad9aa3752 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.962+0000 7f953b400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.962+0000 7f953b400700 1 -- 172.21.15.43:0/2389888825 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95400b35a0 con 0x55bc5aceedb0 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.962+0000 7f953b400700 1 -- 172.21.15.43:0/2389888825 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9408+0+0 (secure 0 0 0) 0x7f9540092110 con 0x55bc5aceedb0 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.962+0000 7f956ad2cb80 1 -- 172.21.15.43:0/2389888825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f954c003a00 con 0x55bc5aceedb0 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.963+0000 7f953b400700 1 -- 172.21.15.43:0/2389888825 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f95400aa2e0 con 0x55bc5aceedb0 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.963+0000 7f956ad2cb80 10 client.16755 fetch_fsmap learned FSMap version 87 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.963+0000 7f956ad2cb80 1 -- 172.21.15.43:0/2389888825 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x55bc5abdd060 con 0x55bc5aceedb0 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.963+0000 7f953b400700 1 -- 172.21.15.43:0/2389888825 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f954c003a00 con 0x55bc5aceedb0 2024-05-27T07:22:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.963+0000 7f953b400700 1 -- 172.21.15.43:0/2389888825 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 87) ==== 5966+0+0 (secure 0 0 0) 0x7f954002a030 con 0x55bc5aceedb0 2024-05-27T07:22:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.963+0000 7f956ad2cb80 10 client.16755 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:22:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.963+0000 7f956ad2cb80 10 client.16755 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:22:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.963+0000 7f956ad2cb80 20 client.16755 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.963+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bc5ad086b0 0x55bc5ad0ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.963+0000 7f956ad2cb80 4 client.16755 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.964+0000 7f956ad2cb80 1 -- 172.21.15.43:0/2389888825 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc5ac1ed90 con 0x55bc5ad086b0 2024-05-27T07:22:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.964+0000 7f9559800700 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bc5ad086b0 0x55bc5ad0ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.965+0000 7f9559800700 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bc5ad086b0 0x55bc5ad0ab00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ca2905336d883b3e server_cookie=78e0a3cee4c40b12 in_seq=0 out_seq=0 2024-05-27T07:22:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.965+0000 7f953b400700 10 client.16755 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:22:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f953b400700 1 -- 172.21.15.43:0/2389888825 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bc5ac1ed90 con 0x55bc5ad086b0 2024-05-27T07:22:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f953b400700 10 client.16755 handle_command_reply: tid=0 2024-05-27T07:22:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 client.16755 shutdown 2024-05-27T07:22:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 -- 172.21.15.43:0/2389888825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 msgr2=0x55bc5acf8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acf8a30 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f95400a6850 tx=0x7f95400e1110 comp rx=0 tx=0).stop 2024-05-27T07:22:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 -- 172.21.15.43:0/2389888825 shutdown_connections 2024-05-27T07:22:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bc5ad086b0 0x55bc5ad0ab00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5acf4d30 0x55bc5acf8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5ab6fe40 0x55bc5acf84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/2389888825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5aceedb0 0x55bc5acf8a30 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f953b400700 0 client.16755 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:22:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 -- 172.21.15.43:0/2389888825 >> 172.21.15.43:0/2389888825 conn(0x55bc5abcffc0 msgr2=0x55bc5ace9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 -- 172.21.15.43:0/2389888825 shutdown_connections 2024-05-27T07:22:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 1 -- 172.21.15.43:0/2389888825 wait complete. 2024-05-27T07:22:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.968+0000 7f956ad2cb80 20 client.16755 trim_cache size 0 max 16384 2024-05-27T07:22:38.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.985+0000 7f956ad2cb80 1 Processor -- start 2024-05-27T07:22:38.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.986+0000 7f956ad2cb80 1 -- start start 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.986+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5ab6fe40 0x55bc5acee870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.986+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5aceedb0 0x55bc5acf2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.986+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5acf4d30 0x55bc5acf3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.986+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f95541b3210 con 0x55bc5ab6fe40 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.986+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f95541b38e0 con 0x55bc5aceedb0 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.986+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f95541b1000 con 0x55bc5acf4d30 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.986+0000 7f9559800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5ab6fe40 0x55bc5acee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.986+0000 7f9559800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5ab6fe40 0x55bc5acee870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53580/0 (socket says 172.21.15.43:53580) 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.986+0000 7f9559800700 1 -- 172.21.15.43:0/623531655 learned_addr learned my addr 172.21.15.43:0/623531655 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.987+0000 7f9558e00700 1 --2- 172.21.15.43:0/623531655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5aceedb0 0x55bc5acf2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.987+0000 7f955a200700 1 --2- 172.21.15.43:0/623531655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5acf4d30 0x55bc5acf3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.987+0000 7f9559800700 1 -- 172.21.15.43:0/623531655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5acf4d30 msgr2=0x55bc5acf3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.987+0000 7f9559800700 1 --2- 172.21.15.43:0/623531655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5acf4d30 0x55bc5acf3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.987+0000 7f9559800700 1 -- 172.21.15.43:0/623531655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5aceedb0 msgr2=0x55bc5acf2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.987+0000 7f9559800700 1 --2- 172.21.15.43:0/623531655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5aceedb0 0x55bc5acf2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.987+0000 7f9559800700 1 -- 172.21.15.43:0/623531655 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc5acef800 con 0x55bc5ab6fe40 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.987+0000 7f9558e00700 1 --2- 172.21.15.43:0/623531655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5aceedb0 0x55bc5acf2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.987+0000 7f9559800700 1 --2- 172.21.15.43:0/623531655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5ab6fe40 0x55bc5acee870 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x55bc5ad0b240 tx=0x7f954800b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d814fdb7b3e9ed53 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.988+0000 7f9546000700 1 -- 172.21.15.43:0/623531655 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95480140d0 con 0x55bc5ab6fe40 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.988+0000 7f9546000700 1 -- 172.21.15.43:0/623531655 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9548014270 con 0x55bc5ab6fe40 2024-05-27T07:22:38.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.988+0000 7f9546000700 1 -- 172.21.15.43:0/623531655 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954801b330 con 0x55bc5ab6fe40 2024-05-27T07:22:38.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.989+0000 7f956ad2cb80 1 -- 172.21.15.43:0/623531655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5ab6fe40 msgr2=0x55bc5acee870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.989+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/623531655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5ab6fe40 0x55bc5acee870 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x55bc5ad0b240 tx=0x7f954800b770 comp rx=0 tx=0).stop 2024-05-27T07:22:38.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.989+0000 7f956ad2cb80 1 -- 172.21.15.43:0/623531655 shutdown_connections 2024-05-27T07:22:38.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.989+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/623531655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5acf4d30 0x55bc5acf3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.989+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/623531655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5ab6fe40 0x55bc5acee870 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.989+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/623531655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5aceedb0 0x55bc5acf2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.989+0000 7f956ad2cb80 1 -- 172.21.15.43:0/623531655 >> 172.21.15.43:0/623531655 conn(0x55bc5abcffc0 msgr2=0x55bc5acf82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:38.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.989+0000 7f956ad2cb80 1 -- 172.21.15.43:0/623531655 shutdown_connections 2024-05-27T07:22:38.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.989+0000 7f956ad2cb80 1 -- 172.21.15.43:0/623531655 wait complete. 2024-05-27T07:22:38.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.990+0000 7f956ad2cb80 1 -- start start 2024-05-27T07:22:38.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.990+0000 7f956ad2cb80 1 Processor -- start 2024-05-27T07:22:38.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 0x55bc5ad01280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5aceedb0 0x55bc5ad0fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f956ad2cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5acf4d30 0x55bc5ad10330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f95541b16d0 con 0x55bc5acf4d30 2024-05-27T07:22:38.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f95541b2b40 con 0x55bc5ab6fe40 2024-05-27T07:22:38.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f956ad2cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f95541b1da0 con 0x55bc5aceedb0 2024-05-27T07:22:38.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f956ad2cb80 10 client.0 _ll_register_callbacks cb 0x55bc5abb6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:38.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f955a200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5acf4d30 0x55bc5ad10330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f955a200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5acf4d30 0x55bc5ad10330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53582/0 (socket says 172.21.15.43:53582) 2024-05-27T07:22:38.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f955a200700 1 -- 172.21.15.43:0/541612811 learned_addr learned my addr 172.21.15.43:0/541612811 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:38.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f9558e00700 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5aceedb0 0x55bc5ad0fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.991+0000 7f9559800700 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 0x55bc5ad01280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.992+0000 7f955a200700 1 -- 172.21.15.43:0/541612811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5aceedb0 msgr2=0x55bc5ad0fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.992+0000 7f955a200700 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5aceedb0 0x55bc5ad0fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.992+0000 7f955a200700 1 -- 172.21.15.43:0/541612811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 msgr2=0x55bc5ad01280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.992+0000 7f955a200700 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 0x55bc5ad01280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.992+0000 7f955a200700 1 -- 172.21.15.43:0/541612811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc5ad10ab0 con 0x55bc5acf4d30 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.992+0000 7f9559800700 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 0x55bc5ad01280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.992+0000 7f955a200700 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5acf4d30 0x55bc5ad10330 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x55bc5acefb30 tx=0x7f954c007b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689f2fcfb37cb40a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.992+0000 7f953b400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.993+0000 7f953b400700 1 -- 172.21.15.43:0/541612811 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c01e040 con 0x55bc5acf4d30 2024-05-27T07:22:38.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.993+0000 7f956ad2cb80 1 -- 172.21.15.43:0/541612811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9548014270 con 0x55bc5acf4d30 2024-05-27T07:22:38.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.993+0000 7f953b400700 1 -- 172.21.15.43:0/541612811 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9408+0+0 (secure 0 0 0) 0x7f954c01a470 con 0x55bc5acf4d30 2024-05-27T07:22:38.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.993+0000 7f953b400700 1 -- 172.21.15.43:0/541612811 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f954c03b050 con 0x55bc5acf4d30 2024-05-27T07:22:38.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.993+0000 7f956ad2cb80 10 client.16761 fetch_fsmap learned FSMap version 87 2024-05-27T07:22:38.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.993+0000 7f956ad2cb80 1 -- 172.21.15.43:0/541612811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=4+}) -- 0x55bc5ad73200 con 0x55bc5acf4d30 2024-05-27T07:22:38.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.993+0000 7f953b400700 1 -- 172.21.15.43:0/541612811 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9548014270 con 0x55bc5acf4d30 2024-05-27T07:22:38.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.994+0000 7f953b400700 1 -- 172.21.15.43:0/541612811 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 87) ==== 5966+0+0 (secure 0 0 0) 0x7f954c02c040 con 0x55bc5acf4d30 2024-05-27T07:22:38.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.994+0000 7f956ad2cb80 10 client.16761 fetch_fsmap finished waiting for FSMap version 87 2024-05-27T07:22:38.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.994+0000 7f956ad2cb80 10 client.16761 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:22:38.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.994+0000 7f956ad2cb80 20 client.16761 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:38.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.994+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bc5ad73960 0x55bc5ad75d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:38.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.994+0000 7f956ad2cb80 4 client.16761 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:22:38.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.994+0000 7f956ad2cb80 1 -- 172.21.15.43:0/541612811 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc5ac18570 con 0x55bc5ad73960 2024-05-27T07:22:38.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.995+0000 7f9559800700 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bc5ad73960 0x55bc5ad75d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:38.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.995+0000 7f9559800700 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bc5ad73960 0x55bc5ad75d80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=34570998ab128891 server_cookie=2a940dd711e1b41f in_seq=0 out_seq=0 2024-05-27T07:22:38.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.995+0000 7f953b400700 10 client.16761 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:22:38.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.996+0000 7f953b400700 1 -- 172.21.15.43:0/541612811 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc5ac18570 con 0x55bc5ad73960 2024-05-27T07:22:38.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.996+0000 7f953b400700 10 client.16761 handle_command_reply: tid=0 2024-05-27T07:22:38.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.996+0000 7f956ad2cb80 1 client.16761 shutdown 2024-05-27T07:22:38.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.996+0000 7f956ad2cb80 1 -- 172.21.15.43:0/541612811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5acf4d30 msgr2=0x55bc5ad10330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.996+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5acf4d30 0x55bc5ad10330 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x55bc5acefb30 tx=0x7f954c007b70 comp rx=0 tx=0).stop 2024-05-27T07:22:38.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.996+0000 7f956ad2cb80 1 -- 172.21.15.43:0/541612811 shutdown_connections 2024-05-27T07:22:38.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.996+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bc5ad73960 0x55bc5ad75d80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.996+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bc5acf4d30 0x55bc5ad10330 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.997+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bc5ab6fe40 0x55bc5ad01280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.997+0000 7f956ad2cb80 1 --2- 172.21.15.43:0/541612811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bc5aceedb0 0x55bc5ad0fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.997+0000 7f956ad2cb80 1 -- 172.21.15.43:0/541612811 >> 172.21.15.43:0/541612811 conn(0x55bc5abcffc0 msgr2=0x55bc5acf29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:38.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.997+0000 7f953b400700 0 client.16761 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:22:38.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.997+0000 7f956ad2cb80 1 -- 172.21.15.43:0/541612811 shutdown_connections 2024-05-27T07:22:38.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.997+0000 7f956ad2cb80 1 -- 172.21.15.43:0/541612811 wait complete. 2024-05-27T07:22:38.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.997+0000 7f956ad2cb80 20 client.16761 trim_cache size 0 max 16384 2024-05-27T07:22:38.988 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:22:38.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 -- 172.21.15.43:0/2019748283 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f953c0b2a70 msgr2=0x7f953c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f953c0b2a70 0x7f953c0b4f80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f954c01b000 tx=0x7f954c01d000 comp rx=0 tx=0).stop 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 -- 172.21.15.43:0/2019748283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9554076440 msgr2=0x7f955419d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9554076440 0x7f955419d5a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9540000c00 tx=0x7f95400057a0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 -- 172.21.15.43:0/2019748283 shutdown_connections 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f953c0b2a70 0x7f953c0b4f80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9554110bf0 0x7f95541a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9554075af0 0x7f955419d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 --2- 172.21.15.43:0/2019748283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9554076440 0x7f955419d5a0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 -- 172.21.15.43:0/2019748283 >> 172.21.15.43:0/2019748283 conn(0x7f9554100030 msgr2=0x7f9554100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 -- 172.21.15.43:0/2019748283 shutdown_connections 2024-05-27T07:22:38.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:38.998+0000 7f955c200700 1 -- 172.21.15.43:0/2019748283 wait complete. 2024-05-27T07:22:39.215 INFO:tasks.workunit.client.0.smithi043.stdout: 1 47620 0.63 MB/sec execute 283 sec latency 68.558 ms 2024-05-27T07:22:39.600 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:39 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:22:39.601 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:39 smithi179 systemd[1]: Stopped Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:39.601 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:39 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.544s CPU time. 2024-05-27T07:22:39.601 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:39 smithi179 systemd[1]: Starting Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:22:39.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:22:39.956 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:39 smithi179 podman[73491]: 2024-05-27 07:22:39.517772502 +0000 UTC m=+0.036617531 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:22:39.956 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:39 smithi179 podman[73491]: 2024-05-27 07:22:39.645391598 +0000 UTC m=+0.164236659 container create e3a2a4712ad832f9927352a3470d7e7d7af22ce31ebbff940fbfd11d32a8da4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-27T07:22:40.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:39 smithi179 podman[73491]: 2024-05-27 07:22:39.966423406 +0000 UTC m=+0.485268474 container init e3a2a4712ad832f9927352a3470d7e7d7af22ce31ebbff940fbfd11d32a8da4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-27T07:22:40.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:39 smithi179 podman[73491]: 2024-05-27 07:22:39.977097556 +0000 UTC m=+0.495942627 container start e3a2a4712ad832f9927352a3470d7e7d7af22ce31ebbff940fbfd11d32a8da4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-27T07:22:40.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 bash[73491]: e3a2a4712ad832f9927352a3470d7e7d7af22ce31ebbff940fbfd11d32a8da4c 2024-05-27T07:22:40.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 systemd[1]: Started Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:40.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-mds[73511]: -- 172.21.15.179:0/163999676 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55731f39da40 con 0x55731f3e5180 2024-05-27T07:22:40.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: starting mds.f at 2024-05-27T07:22:40.215 INFO:tasks.workunit.client.0.smithi043.stdout: 1 47721 0.62 MB/sec execute 284 sec latency 68.288 ms 2024-05-27T07:22:40.436 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:40.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:40 smithi132 ceph-mon[35283]: pgmap v293: 129 pgs: 129 active+clean; 80 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 447 KiB/s wr, 54 op/s 2024-05-27T07:22:40.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:40 smithi132 ceph-mon[35283]: daemon mds.f restarted 2024-05-27T07:22:40.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:40 smithi043 ceph-mon[31531]: pgmap v293: 129 pgs: 129 active+clean; 80 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 447 KiB/s wr, 54 op/s 2024-05-27T07:22:40.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:40 smithi043 ceph-mon[31531]: daemon mds.f restarted 2024-05-27T07:22:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:40 smithi179 ceph-mon[35888]: pgmap v293: 129 pgs: 129 active+clean; 80 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 447 KiB/s wr, 54 op/s 2024-05-27T07:22:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:40 smithi179 ceph-mon[35888]: daemon mds.f restarted 2024-05-27T07:22:40.709 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb942000700 time 2024-05-27T07:22:40.428830+0000 2024-05-27T07:22:40.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:40.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:40.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb957a14ba6] 2024-05-27T07:22:40.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb957a14dd1] 2024-05-27T07:22:40.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5572dfecd23d] 2024-05-27T07:22:40.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5572dfeb8971] 2024-05-27T07:22:40.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 5: (SnapRealm::check_cache() const+0x13e) [0x5572dfeb6fa0] 2024-05-27T07:22:40.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5572dfeb7651] 2024-05-27T07:22:40.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5572dfe4c347] 2024-05-27T07:22:40.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5572dfb4887e] 2024-05-27T07:22:40.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5572dfb4de94] 2024-05-27T07:22:40.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5572dfb58969] 2024-05-27T07:22:40.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 11: (MDLog::_replay_thread()+0x1f38) [0x5572dfef148c] 2024-05-27T07:22:40.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5572dfefa9cb] 2024-05-27T07:22:40.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 13: (Thread::entry_wrapper()+0x43) [0x7fb9579e372d] 2024-05-27T07:22:40.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb9579e3749] 2024-05-27T07:22:40.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9565871ca] 2024-05-27T07:22:40.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 16: clone() 2024-05-27T07:22:40.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.431+0000 7fb942000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb942000700 time 2024-05-27T07:22:40.428830+0000 2024-05-27T07:22:40.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:40.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:40.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb957a14ba6] 2024-05-27T07:22:40.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb957a14dd1] 2024-05-27T07:22:40.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5572dfecd23d] 2024-05-27T07:22:40.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5572dfeb8971] 2024-05-27T07:22:40.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 5: (SnapRealm::check_cache() const+0x13e) [0x5572dfeb6fa0] 2024-05-27T07:22:40.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5572dfeb7651] 2024-05-27T07:22:40.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5572dfe4c347] 2024-05-27T07:22:40.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5572dfb4887e] 2024-05-27T07:22:40.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5572dfb4de94] 2024-05-27T07:22:40.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5572dfb58969] 2024-05-27T07:22:40.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 11: (MDLog::_replay_thread()+0x1f38) [0x5572dfef148c] 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5572dfefa9cb] 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 13: (Thread::entry_wrapper()+0x43) [0x7fb9579e372d] 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb9579e3749] 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9565871ca] 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 16: clone() 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: *** Caught signal (Aborted) ** 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: in thread 7fb942000700 thread_name:md_log_replay 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb956591d20] 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2: gsignal() 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 3: abort() 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb957a14ced] 2024-05-27T07:22:40.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb957a14dd1] 2024-05-27T07:22:40.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5572dfecd23d] 2024-05-27T07:22:40.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5572dfeb8971] 2024-05-27T07:22:40.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 8: (SnapRealm::check_cache() const+0x13e) [0x5572dfeb6fa0] 2024-05-27T07:22:40.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5572dfeb7651] 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5572dfe4c347] 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5572dfb4887e] 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5572dfb4de94] 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5572dfb58969] 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 14: (MDLog::_replay_thread()+0x1f38) [0x5572dfef148c] 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5572dfefa9cb] 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 16: (Thread::entry_wrapper()+0x43) [0x7fb9579e372d] 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb9579e3749] 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9565871ca] 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 19: clone() 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.435+0000 7fb942000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: in thread 7fb942000700 thread_name:md_log_replay 2024-05-27T07:22:40.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb956591d20] 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2: gsignal() 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 3: abort() 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb957a14ced] 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb957a14dd1] 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5572dfecd23d] 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5572dfeb8971] 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 8: (SnapRealm::check_cache() const+0x13e) [0x5572dfeb6fa0] 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5572dfeb7651] 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5572dfe4c347] 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5572dfb4887e] 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5572dfb4de94] 2024-05-27T07:22:40.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5572dfb58969] 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 14: (MDLog::_replay_thread()+0x1f38) [0x5572dfef148c] 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5572dfefa9cb] 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 16: (Thread::entry_wrapper()+0x43) [0x7fb9579e372d] 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb9579e3749] 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9565871ca] 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 19: clone() 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: -1> 2024-05-27T07:22:40.431+0000 7fb942000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb942000700 time 2024-05-27T07:22:40.428830+0000 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:40.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb957a14ba6] 2024-05-27T07:22:40.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb957a14dd1] 2024-05-27T07:22:40.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5572dfecd23d] 2024-05-27T07:22:40.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5572dfeb8971] 2024-05-27T07:22:40.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 5: (SnapRealm::check_cache() const+0x13e) [0x5572dfeb6fa0] 2024-05-27T07:22:40.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5572dfeb7651] 2024-05-27T07:22:40.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5572dfe4c347] 2024-05-27T07:22:40.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5572dfb4887e] 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5572dfb4de94] 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5572dfb58969] 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 11: (MDLog::_replay_thread()+0x1f38) [0x5572dfef148c] 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5572dfefa9cb] 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 13: (Thread::entry_wrapper()+0x43) [0x7fb9579e372d] 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb9579e3749] 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9565871ca] 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 16: clone() 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 0> 2024-05-27T07:22:40.435+0000 7fb942000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: in thread 7fb942000700 thread_name:md_log_replay 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb956591d20] 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2: gsignal() 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 3: abort() 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb957a14ced] 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb957a14dd1] 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5572dfecd23d] 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5572dfeb8971] 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 8: (SnapRealm::check_cache() const+0x13e) [0x5572dfeb6fa0] 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5572dfeb7651] 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5572dfe4c347] 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5572dfb4887e] 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5572dfb4de94] 2024-05-27T07:22:40.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5572dfb58969] 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 14: (MDLog::_replay_thread()+0x1f38) [0x5572dfef148c] 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5572dfefa9cb] 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 16: (Thread::entry_wrapper()+0x43) [0x7fb9579e372d] 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb9579e3749] 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9565871ca] 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 19: clone() 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: -1> 2024-05-27T07:22:40.431+0000 7fb942000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb942000700 time 2024-05-27T07:22:40.428830+0000 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb957a14ba6] 2024-05-27T07:22:40.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb957a14dd1] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5572dfecd23d] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5572dfeb8971] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 5: (SnapRealm::check_cache() const+0x13e) [0x5572dfeb6fa0] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5572dfeb7651] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5572dfe4c347] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5572dfb4887e] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5572dfb4de94] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5572dfb58969] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 11: (MDLog::_replay_thread()+0x1f38) [0x5572dfef148c] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5572dfefa9cb] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 13: (Thread::entry_wrapper()+0x43) [0x7fb9579e372d] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb9579e3749] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9565871ca] 2024-05-27T07:22:40.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 16: clone() 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 0> 2024-05-27T07:22:40.435+0000 7fb942000700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: in thread 7fb942000700 thread_name:md_log_replay 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb956591d20] 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2: gsignal() 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 3: abort() 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb957a14ced] 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb957a14dd1] 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5572dfecd23d] 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5572dfeb8971] 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 8: (SnapRealm::check_cache() const+0x13e) [0x5572dfeb6fa0] 2024-05-27T07:22:40.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5572dfeb7651] 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5572dfe4c347] 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5572dfb4887e] 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5572dfb4de94] 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5572dfb58969] 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 14: (MDLog::_replay_thread()+0x1f38) [0x5572dfef148c] 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5572dfefa9cb] 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 16: (Thread::entry_wrapper()+0x43) [0x7fb9579e372d] 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb9579e3749] 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9565871ca] 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 19: clone() 2024-05-27T07:22:40.725 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:40.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73507]: 2024-05-27T07:22:40.726 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:40 smithi179 podman[73542]: 2024-05-27 07:22:40.544501024 +0000 UTC m=+0.033227381 container died e3a2a4712ad832f9927352a3470d7e7d7af22ce31ebbff940fbfd11d32a8da4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:22:41.215 INFO:tasks.workunit.client.0.smithi043.stdout: 1 47842 0.62 MB/sec execute 285 sec latency 63.334 ms 2024-05-27T07:22:41.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:41 smithi132 ceph-mon[35283]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:22:41.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:41 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/1052759039,v1:172.21.15.179:6833/1052759039] up:boot 2024-05-27T07:22:41.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:41 smithi132 ceph-mon[35283]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:41.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:41 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:41.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:41 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:41.624 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:41 smithi043 ceph-mon[31531]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:22:41.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:41 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/1052759039,v1:172.21.15.179:6833/1052759039] up:boot 2024-05-27T07:22:41.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:41 smithi043 ceph-mon[31531]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:41.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:41 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:41.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:41 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:41.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:41 smithi179 ceph-mon[35888]: osdmap e120: 12 total, 12 up, 12 in 2024-05-27T07:22:41.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:41 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/1052759039,v1:172.21.15.179:6833/1052759039] up:boot 2024-05-27T07:22:41.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:41 smithi179 ceph-mon[35888]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:41.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:41 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:41.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:41 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:41.709 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:41 smithi179 podman[73542]: 2024-05-27 07:22:41.337900732 +0000 UTC m=+0.826627132 container remove e3a2a4712ad832f9927352a3470d7e7d7af22ce31ebbff940fbfd11d32a8da4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:22:41.709 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:41 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:22:42.097 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:41 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:22:42.098 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:41 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.565s CPU time. 2024-05-27T07:22:42.098 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:41 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:22:42.098 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:41 smithi179 systemd[1]: Stopped Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:42.098 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:41 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.551s CPU time. 2024-05-27T07:22:42.098 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:41 smithi179 systemd[1]: Starting Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:22:42.215 INFO:tasks.workunit.client.0.smithi043.stdout: 1 47993 0.62 MB/sec execute 286 sec latency 64.068 ms 2024-05-27T07:22:42.357 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:42 smithi179 ceph-mon[35888]: pgmap v295: 129 pgs: 129 active+clean; 80 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 372 KiB/s wr, 52 op/s 2024-05-27T07:22:42.357 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:42 smithi179 podman[73635]: 2024-05-27 07:22:42.012968804 +0000 UTC m=+0.037328675 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:22:42.357 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:42 smithi179 podman[73635]: 2024-05-27 07:22:42.197834287 +0000 UTC m=+0.222194179 container create 933dd87cee159d15860d4f1abf5a70bbe8e2a272e612cb6f82c036e973ffdc8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-27T07:22:42.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:42 smithi043 ceph-mon[31531]: pgmap v295: 129 pgs: 129 active+clean; 80 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 372 KiB/s wr, 52 op/s 2024-05-27T07:22:42.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:42 smithi132 ceph-mon[35283]: pgmap v295: 129 pgs: 129 active+clean; 80 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 372 KiB/s wr, 52 op/s 2024-05-27T07:22:42.640 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:42 smithi179 podman[73635]: 2024-05-27 07:22:42.544535801 +0000 UTC m=+0.568895750 container init 933dd87cee159d15860d4f1abf5a70bbe8e2a272e612cb6f82c036e973ffdc8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-27T07:22:42.641 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:42 smithi179 podman[73635]: 2024-05-27 07:22:42.55720263 +0000 UTC m=+0.581562490 container start 933dd87cee159d15860d4f1abf5a70bbe8e2a272e612cb6f82c036e973ffdc8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-27T07:22:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.838+0000 7fa037200700 1 -- 172.21.15.43:0/2534134289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa038115170 msgr2=0x7fa038117590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.838+0000 7fa037200700 1 --2- 172.21.15.43:0/2534134289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa038115170 0x7fa038117590 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa03000b0a0 tx=0x7fa03000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.838+0000 7fa037200700 1 -- 172.21.15.43:0/2534134289 shutdown_connections 2024-05-27T07:22:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.838+0000 7fa037200700 1 --2- 172.21.15.43:0/2534134289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa038115170 0x7fa038117590 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.838+0000 7fa037200700 1 --2- 172.21.15.43:0/2534134289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa038075b50 0x7fa038075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.838+0000 7fa037200700 1 --2- 172.21.15.43:0/2534134289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa0380764a0 0x7fa038114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.838+0000 7fa037200700 1 -- 172.21.15.43:0/2534134289 >> 172.21.15.43:0/2534134289 conn(0x7fa038079d00 msgr2=0x7fa03807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.839+0000 7fa037200700 1 -- 172.21.15.43:0/2534134289 shutdown_connections 2024-05-27T07:22:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.839+0000 7fa037200700 1 -- 172.21.15.43:0/2534134289 wait complete. 2024-05-27T07:22:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.840+0000 7fa037200700 1 Processor -- start 2024-05-27T07:22:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.840+0000 7fa037200700 1 -- start start 2024-05-27T07:22:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.841+0000 7fa037200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa038075b50 0x7fa0381a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.841+0000 7fa037200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0380764a0 0x7fa0381a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.841+0000 7fa037200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa038115170 0x7fa0381a5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.841+0000 7fa037200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa03811a720 con 0x7fa038075b50 2024-05-27T07:22:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.841+0000 7fa037200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa03811a5a0 con 0x7fa038115170 2024-05-27T07:22:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.841+0000 7fa037200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa03811a8a0 con 0x7fa0380764a0 2024-05-27T07:22:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.841+0000 7fa035e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa038075b50 0x7fa0381a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.841+0000 7fa035e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa038075b50 0x7fa0381a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53608/0 (socket says 172.21.15.43:53608) 2024-05-27T07:22:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.841+0000 7fa035e00700 1 -- 172.21.15.43:0/838534582 learned_addr learned my addr 172.21.15.43:0/838534582 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.841+0000 7fa02fe00700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0380764a0 0x7fa0381a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.842+0000 7fa036800700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa038115170 0x7fa0381a5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.842+0000 7fa02fe00700 1 -- 172.21.15.43:0/838534582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa038115170 msgr2=0x7fa0381a5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.842+0000 7fa02fe00700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa038115170 0x7fa0381a5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.842+0000 7fa02fe00700 1 -- 172.21.15.43:0/838534582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa038075b50 msgr2=0x7fa0381a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.842+0000 7fa02fe00700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa038075b50 0x7fa0381a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.842+0000 7fa02fe00700 1 -- 172.21.15.43:0/838534582 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0381a5fe0 con 0x7fa0380764a0 2024-05-27T07:22:42.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.842+0000 7fa035e00700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa038075b50 0x7fa0381a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:22:42.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.842+0000 7fa02fe00700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0380764a0 0x7fa0381a16b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa02000d950 tx=0x7fa02000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.843+0000 7fa02ea00700 1 -- 172.21.15.43:0/838534582 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020017070 con 0x7fa0380764a0 2024-05-27T07:22:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.843+0000 7fa037200700 1 -- 172.21.15.43:0/838534582 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0381a62d0 con 0x7fa0380764a0 2024-05-27T07:22:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.843+0000 7fa02ea00700 1 -- 172.21.15.43:0/838534582 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa020009dc0 con 0x7fa0380764a0 2024-05-27T07:22:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.843+0000 7fa02ea00700 1 -- 172.21.15.43:0/838534582 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020010cd0 con 0x7fa0380764a0 2024-05-27T07:22:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.843+0000 7fa037200700 1 -- 172.21.15.43:0/838534582 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0381adbf0 con 0x7fa0380764a0 2024-05-27T07:22:42.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.848+0000 7fa02ea00700 1 -- 172.21.15.43:0/838534582 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa02001a070 con 0x7fa0380764a0 2024-05-27T07:22:42.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.848+0000 7fa04b606b80 1 Processor -- start 2024-05-27T07:22:42.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.849+0000 7fa04b606b80 1 -- start start 2024-05-27T07:22:42.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.849+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b1361c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.849+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b1349ce40 0x555b1361cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.849+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b136262c0 0x555b136286e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.849+0000 7fa04b606b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555b13549e40 con 0x555b1349ce40 2024-05-27T07:22:42.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.849+0000 7fa04b606b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555b135399e0 con 0x555b136262c0 2024-05-27T07:22:42.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.849+0000 7fa04b606b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555b1353da20 con 0x555b135889f0 2024-05-27T07:22:42.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa035e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b1349ce40 0x555b1361cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa036800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b1361c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa035e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b1349ce40 0x555b1361cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53612/0 (socket says 172.21.15.43:53612) 2024-05-27T07:22:42.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa035e00700 1 -- 172.21.15.43:0/2578431694 learned_addr learned my addr 172.21.15.43:0/2578431694 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:42.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa02fe00700 1 --2- 172.21.15.43:0/2578431694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b136262c0 0x555b136286e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa036800700 1 -- 172.21.15.43:0/2578431694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b136262c0 msgr2=0x555b136286e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:42.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa036800700 1 --2- 172.21.15.43:0/2578431694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b136262c0 0x555b136286e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa036800700 1 -- 172.21.15.43:0/2578431694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b1349ce40 msgr2=0x555b1361cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:42.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa036800700 1 --2- 172.21.15.43:0/2578431694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b1349ce40 0x555b1361cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa036800700 1 -- 172.21.15.43:0/2578431694 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b13502420 con 0x555b135889f0 2024-05-27T07:22:42.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa02ea00700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa01c0b29f0 0x7fa01c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.850+0000 7fa035e00700 1 --2- 172.21.15.43:0/2578431694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b1349ce40 0x555b1361cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:22:42.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.851+0000 7fa02ea00700 1 -- 172.21.15.43:0/838534582 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9408+0+0 (secure 0 0 0) 0x7fa0200e2350 con 0x7fa0380764a0 2024-05-27T07:22:42.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.851+0000 7fa036800700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa01c0b29f0 0x7fa01c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.851+0000 7fa036800700 1 --2- 172.21.15.43:0/2578431694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b1361c750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa03000b780 tx=0x7fa030009db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df35f27da0b207df server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.851+0000 7fa027400700 1 -- 172.21.15.43:0/2578431694 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030004a50 con 0x555b135889f0 2024-05-27T07:22:42.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.852+0000 7fa027400700 1 -- 172.21.15.43:0/2578431694 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa030022040 con 0x555b135889f0 2024-05-27T07:22:42.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.852+0000 7fa027400700 1 -- 172.21.15.43:0/2578431694 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03002f050 con 0x555b135889f0 2024-05-27T07:22:42.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.853+0000 7fa04b606b80 1 -- 172.21.15.43:0/2578431694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 msgr2=0x555b1361c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.853+0000 7fa036800700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa01c0b29f0 0x7fa01c0b4f00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa030024000 tx=0x7fa03002b000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.853+0000 7fa04b606b80 1 --2- 172.21.15.43:0/2578431694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b1361c750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa03000b780 tx=0x7fa030009db0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.853+0000 7fa04b606b80 1 -- 172.21.15.43:0/2578431694 shutdown_connections 2024-05-27T07:22:42.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.853+0000 7fa04b606b80 1 --2- 172.21.15.43:0/2578431694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b136262c0 0x555b136286e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.853+0000 7fa04b606b80 1 --2- 172.21.15.43:0/2578431694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b1361c750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.853+0000 7fa04b606b80 1 --2- 172.21.15.43:0/2578431694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b1349ce40 0x555b1361cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.853+0000 7fa04b606b80 1 -- 172.21.15.43:0/2578431694 >> 172.21.15.43:0/2578431694 conn(0x555b13536390 msgr2=0x555b13617710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.853+0000 7fa04b606b80 1 -- 172.21.15.43:0/2578431694 shutdown_connections 2024-05-27T07:22:42.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.853+0000 7fa04b606b80 1 -- 172.21.15.43:0/2578431694 wait complete. 2024-05-27T07:22:42.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.854+0000 7fa04b606b80 1 -- start start 2024-05-27T07:22:42.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.854+0000 7fa04b606b80 1 Processor -- start 2024-05-27T07:22:42.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.855+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b1349ce40 0x555b13623250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.855+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b135889f0 0x555b13623790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.855+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13623cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.855+0000 7fa04b606b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555b1343ffa0 con 0x555b136262c0 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.855+0000 7fa04b606b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555b134da760 con 0x555b135889f0 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.855+0000 7fa04b606b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555b13628e00 con 0x555b1349ce40 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.855+0000 7fa04b606b80 10 client.0 _ll_register_callbacks cb 0x555b134e3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.855+0000 7fa036800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13623cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.856+0000 7fa036800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13623cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53616/0 (socket says 172.21.15.43:53616) 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.856+0000 7fa035e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b1349ce40 0x555b13623250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.856+0000 7fa036800700 1 -- 172.21.15.43:0/2940170099 learned_addr learned my addr 172.21.15.43:0/2940170099 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.856+0000 7fa02fe00700 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b135889f0 0x555b13623790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.856+0000 7fa036800700 1 -- 172.21.15.43:0/2940170099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b1349ce40 msgr2=0x555b13623250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.856+0000 7fa036800700 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b1349ce40 0x555b13623250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.856+0000 7fa036800700 1 -- 172.21.15.43:0/2940170099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b135889f0 msgr2=0x555b13623790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.856+0000 7fa036800700 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b135889f0 0x555b13623790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.856+0000 7fa036800700 1 -- 172.21.15.43:0/2940170099 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b134898d0 con 0x555b136262c0 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.856+0000 7fa035e00700 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b1349ce40 0x555b13623250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.857+0000 7fa036800700 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13623cd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fa03000b3f0 tx=0x7fa0300093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469ac56e9bc5ec76 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.857+0000 7fa01be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.857+0000 7fa01be00700 1 -- 172.21.15.43:0/2940170099 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0300296f0 con 0x555b136262c0 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.857+0000 7fa01be00700 1 -- 172.21.15.43:0/2940170099 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9408+0+0 (secure 0 0 0) 0x7fa030029960 con 0x555b136262c0 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.857+0000 7fa04b606b80 1 -- 172.21.15.43:0/2940170099 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa030022040 con 0x555b136262c0 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa01be00700 1 -- 172.21.15.43:0/2940170099 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fa030043050 con 0x555b136262c0 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa04b606b80 10 client.16776 fetch_fsmap learned FSMap version 89 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa04b606b80 1 -- 172.21.15.43:0/2940170099 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=4+}) -- 0x555b1350a060 con 0x555b136262c0 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa01be00700 1 -- 172.21.15.43:0/2940170099 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa030022040 con 0x555b136262c0 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa01be00700 1 -- 172.21.15.43:0/2940170099 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 89) ==== 5966+0+0 (secure 0 0 0) 0x7fa030011070 con 0x555b136262c0 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa04b606b80 10 client.16776 fetch_fsmap finished waiting for FSMap version 89 2024-05-27T07:22:42.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa04b606b80 10 client.16776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa04b606b80 20 client.16776 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa04b606b80 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555b13635cf0 0x555b136381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa04b606b80 4 client.16776 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.858+0000 7fa04b606b80 1 -- 172.21.15.43:0/2940170099 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b1354bd90 con 0x555b13635cf0 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.859+0000 7fa035e00700 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555b13635cf0 0x555b136381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.860+0000 7fa035e00700 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555b13635cf0 0x555b136381a0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c6d755bdab3b80f server_cookie=d54b74514458bf56 in_seq=0 out_seq=0 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.860+0000 7fa01be00700 10 client.16776 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.863+0000 7fa01be00700 1 -- 172.21.15.43:0/2940170099 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555b1354bd90 con 0x555b13635cf0 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.863+0000 7fa01be00700 10 client.16776 handle_command_reply: tid=0 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.863+0000 7fa04b606b80 1 client.16776 shutdown 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.863+0000 7fa04b606b80 1 -- 172.21.15.43:0/2940170099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 msgr2=0x555b13623cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.863+0000 7fa04b606b80 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13623cd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fa03000b3f0 tx=0x7fa0300093e0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.864+0000 7fa04b606b80 1 -- 172.21.15.43:0/2940170099 shutdown_connections 2024-05-27T07:22:42.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.864+0000 7fa04b606b80 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555b13635cf0 0x555b136381a0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.864+0000 7fa04b606b80 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13623cd0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.864+0000 7fa04b606b80 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b1349ce40 0x555b13623250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.864+0000 7fa04b606b80 1 --2- 172.21.15.43:0/2940170099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b135889f0 0x555b13623790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.864+0000 7fa04b606b80 1 -- 172.21.15.43:0/2940170099 >> 172.21.15.43:0/2940170099 conn(0x555b13536390 msgr2=0x555b13617260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.864+0000 7fa01be00700 0 client.16776 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:42.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.864+0000 7fa04b606b80 1 -- 172.21.15.43:0/2940170099 shutdown_connections 2024-05-27T07:22:42.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.864+0000 7fa04b606b80 1 -- 172.21.15.43:0/2940170099 wait complete. 2024-05-27T07:22:42.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.864+0000 7fa04b606b80 20 client.16776 trim_cache size 0 max 16384 2024-05-27T07:22:42.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.883+0000 7fa04b606b80 1 Processor -- start 2024-05-27T07:22:42.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.883+0000 7fa04b606b80 1 -- start start 2024-05-27T07:22:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.884+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 0x555b13622570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.884+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b13631f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.884+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b1363d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.884+0000 7fa04b606b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b7430 con 0x555b136262c0 2024-05-27T07:22:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.884+0000 7fa04b606b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b7b00 con 0x555b1349ce40 2024-05-27T07:22:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.884+0000 7fa04b606b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b5220 con 0x555b135889f0 2024-05-27T07:22:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.884+0000 7fa036800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b1363d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.884+0000 7fa036800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b1363d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53620/0 (socket says 172.21.15.43:53620) 2024-05-27T07:22:42.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.884+0000 7fa036800700 1 -- 172.21.15.43:0/1800496912 learned_addr learned my addr 172.21.15.43:0/1800496912 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:42.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.885+0000 7fa035e00700 1 --2- 172.21.15.43:0/1800496912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 0x555b13622570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.885+0000 7fa02fe00700 1 --2- 172.21.15.43:0/1800496912 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b13631f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.885+0000 7fa036800700 1 -- 172.21.15.43:0/1800496912 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 msgr2=0x555b13631f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.885+0000 7fa036800700 1 --2- 172.21.15.43:0/1800496912 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b13631f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.885+0000 7fa036800700 1 -- 172.21.15.43:0/1800496912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 msgr2=0x555b13622570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.885+0000 7fa036800700 1 --2- 172.21.15.43:0/1800496912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 0x555b13622570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.885+0000 7fa036800700 1 -- 172.21.15.43:0/1800496912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b13624040 con 0x555b136262c0 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.885+0000 7fa035e00700 1 --2- 172.21.15.43:0/1800496912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 0x555b13622570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.885+0000 7fa036800700 1 --2- 172.21.15.43:0/1800496912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b1363d960 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa030009e60 tx=0x7fa030006060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7324b0008eacdfff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.885+0000 7fa026a00700 1 -- 172.21.15.43:0/1800496912 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0300296f0 con 0x555b136262c0 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.886+0000 7fa026a00700 1 -- 172.21.15.43:0/1800496912 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa030022070 con 0x555b136262c0 2024-05-27T07:22:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.886+0000 7fa026a00700 1 -- 172.21.15.43:0/1800496912 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030005ea0 con 0x555b136262c0 2024-05-27T07:22:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.886+0000 7fa04b606b80 1 -- 172.21.15.43:0/1800496912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 msgr2=0x555b1363d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.886+0000 7fa04b606b80 1 --2- 172.21.15.43:0/1800496912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b1363d960 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa030009e60 tx=0x7fa030006060 comp rx=0 tx=0).stop 2024-05-27T07:22:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.887+0000 7fa04b606b80 1 -- 172.21.15.43:0/1800496912 shutdown_connections 2024-05-27T07:22:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.887+0000 7fa04b606b80 1 --2- 172.21.15.43:0/1800496912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b1363d960 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.887+0000 7fa04b606b80 1 --2- 172.21.15.43:0/1800496912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 0x555b13622570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.887+0000 7fa04b606b80 1 --2- 172.21.15.43:0/1800496912 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b13631f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.887+0000 7fa04b606b80 1 -- 172.21.15.43:0/1800496912 >> 172.21.15.43:0/1800496912 conn(0x555b13536390 msgr2=0x555b1361e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.887+0000 7fa04b606b80 1 -- 172.21.15.43:0/1800496912 shutdown_connections 2024-05-27T07:22:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.887+0000 7fa04b606b80 1 -- 172.21.15.43:0/1800496912 wait complete. 2024-05-27T07:22:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.887+0000 7fa04b606b80 1 -- start start 2024-05-27T07:22:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.888+0000 7fa04b606b80 1 Processor -- start 2024-05-27T07:22:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.888+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 0x555b136290d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.888+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b13629610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.888+0000 7fa04b606b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13629b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.888+0000 7fa04b606b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b58f0 con 0x555b136262c0 2024-05-27T07:22:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.888+0000 7fa04b606b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b6d60 con 0x555b1349ce40 2024-05-27T07:22:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.888+0000 7fa04b606b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b5fc0 con 0x555b135889f0 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.888+0000 7fa04b606b80 10 client.0 _ll_register_callbacks cb 0x555b134e3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.889+0000 7fa036800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13629b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.889+0000 7fa036800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13629b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53622/0 (socket says 172.21.15.43:53622) 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.889+0000 7fa036800700 1 -- 172.21.15.43:0/29388611 learned_addr learned my addr 172.21.15.43:0/29388611 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.889+0000 7fa02fe00700 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b13629610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.889+0000 7fa035e00700 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 0x555b136290d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.889+0000 7fa036800700 1 -- 172.21.15.43:0/29388611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 msgr2=0x555b13629610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.889+0000 7fa036800700 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b13629610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.889+0000 7fa036800700 1 -- 172.21.15.43:0/29388611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 msgr2=0x555b136290d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.889+0000 7fa036800700 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 0x555b136290d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.889+0000 7fa036800700 1 -- 172.21.15.43:0/29388611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b1369e8d0 con 0x555b136262c0 2024-05-27T07:22:42.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.890+0000 7fa036800700 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13629b50 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fa030009df0 tx=0x7fa030007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd4415790cec3fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:42.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.890+0000 7fa01be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:42.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.890+0000 7fa01be00700 1 -- 172.21.15.43:0/29388611 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0300325a0 con 0x555b136262c0 2024-05-27T07:22:42.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.890+0000 7fa01be00700 1 -- 172.21.15.43:0/29388611 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9408+0+0 (secure 0 0 0) 0x7fa030029960 con 0x555b136262c0 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.890+0000 7fa04b606b80 1 -- 172.21.15.43:0/29388611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa030022070 con 0x555b136262c0 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa01be00700 1 -- 172.21.15.43:0/29388611 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fa030032740 con 0x555b136262c0 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa04b606b80 10 client.16782 fetch_fsmap learned FSMap version 89 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa04b606b80 1 -- 172.21.15.43:0/29388611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=4+}) -- 0x555b1369fa80 con 0x555b136262c0 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa01be00700 1 -- 172.21.15.43:0/29388611 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa030022070 con 0x555b136262c0 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa01be00700 1 -- 172.21.15.43:0/29388611 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 89) ==== 5966+0+0 (secure 0 0 0) 0x7fa030011040 con 0x555b136262c0 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa04b606b80 10 client.16782 fetch_fsmap finished waiting for FSMap version 89 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa04b606b80 10 client.16782 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa04b606b80 20 client.16782 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa04b606b80 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555b1369fe40 0x555b136a2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa04b606b80 4 client.16782 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:22:42.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.891+0000 7fa04b606b80 1 -- 172.21.15.43:0/29388611 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555b13545570 con 0x555b1369fe40 2024-05-27T07:22:42.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.892+0000 7fa035e00700 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555b1369fe40 0x555b136a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:42.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.893+0000 7fa035e00700 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555b1369fe40 0x555b136a2260 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc1f2a039385a370 server_cookie=fa1fe4741f2379e1 in_seq=0 out_seq=0 2024-05-27T07:22:42.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.893+0000 7fa01be00700 10 client.16782 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:42.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.894+0000 7fa01be00700 1 -- 172.21.15.43:0/29388611 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555b13545570 con 0x555b1369fe40 2024-05-27T07:22:42.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.894+0000 7fa01be00700 10 client.16782 handle_command_reply: tid=0 2024-05-27T07:22:42.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.894+0000 7fa04b606b80 1 client.16782 shutdown 2024-05-27T07:22:42.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 1 -- 172.21.15.43:0/29388611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 msgr2=0x555b13629b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13629b50 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fa030009df0 tx=0x7fa030007e60 comp rx=0 tx=0).stop 2024-05-27T07:22:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 1 -- 172.21.15.43:0/29388611 shutdown_connections 2024-05-27T07:22:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555b1369fe40 0x555b136a2260 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555b136262c0 0x555b13629b50 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555b1349ce40 0x555b136290d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 1 --2- 172.21.15.43:0/29388611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555b135889f0 0x555b13629610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 1 -- 172.21.15.43:0/29388611 >> 172.21.15.43:0/29388611 conn(0x555b13536390 msgr2=0x555b1361e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa01be00700 0 client.16782 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 1 -- 172.21.15.43:0/29388611 shutdown_connections 2024-05-27T07:22:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 1 -- 172.21.15.43:0/29388611 wait complete. 2024-05-27T07:22:42.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.895+0000 7fa04b606b80 20 client.16782 trim_cache size 0 max 16384 2024-05-27T07:22:42.887 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:22:42.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.896+0000 7fa037200700 1 -- 172.21.15.43:0/838534582 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa01c0b29f0 msgr2=0x7fa01c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.896+0000 7fa037200700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa01c0b29f0 0x7fa01c0b4f00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa030024000 tx=0x7fa03002b000 comp rx=0 tx=0).stop 2024-05-27T07:22:42.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.896+0000 7fa037200700 1 -- 172.21.15.43:0/838534582 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0380764a0 msgr2=0x7fa0381a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:42.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.896+0000 7fa037200700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0380764a0 0x7fa0381a16b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa02000d950 tx=0x7fa02000dc60 comp rx=0 tx=0).stop 2024-05-27T07:22:42.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.896+0000 7fa037200700 1 -- 172.21.15.43:0/838534582 shutdown_connections 2024-05-27T07:22:42.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.896+0000 7fa037200700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa01c0b29f0 0x7fa01c0b4f00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.896+0000 7fa037200700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa038115170 0x7fa0381a5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.896+0000 7fa037200700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa038075b50 0x7fa0381a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.896+0000 7fa037200700 1 --2- 172.21.15.43:0/838534582 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0380764a0 0x7fa0381a16b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.896+0000 7fa037200700 1 -- 172.21.15.43:0/838534582 >> 172.21.15.43:0/838534582 conn(0x7fa038079d00 msgr2=0x7fa038100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.897+0000 7fa037200700 1 -- 172.21.15.43:0/838534582 shutdown_connections 2024-05-27T07:22:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:42.897+0000 7fa037200700 1 -- 172.21.15.43:0/838534582 wait complete. 2024-05-27T07:22:42.958 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:42 smithi179 bash[73635]: 933dd87cee159d15860d4f1abf5a70bbe8e2a272e612cb6f82c036e973ffdc8f 2024-05-27T07:22:42.958 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:42 smithi179 systemd[1]: Started Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:42.958 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:42 smithi179 ceph-mds[73655]: -- 172.21.15.179:0/815109791 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5649caebfa40 con 0x5649caf07600 2024-05-27T07:22:42.958 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:42 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: starting mds.l at 2024-05-27T07:22:43.215 INFO:tasks.workunit.client.0.smithi043.stdout: 1 48126 0.62 MB/sec execute 287 sec latency 60.781 ms 2024-05-27T07:22:43.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:43 smithi132 ceph-mon[35283]: daemon mds.l restarted 2024-05-27T07:22:43.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:43 smithi043 ceph-mon[31531]: daemon mds.l restarted 2024-05-27T07:22:43.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:43 smithi179 ceph-mon[35888]: daemon mds.l restarted 2024-05-27T07:22:44.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f22a3800700 time 2024-05-27T07:22:43.876711+0000 2024-05-27T07:22:44.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:44.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:44.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f22b93eaba6] 2024-05-27T07:22:44.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22b93eadd1] 2024-05-27T07:22:44.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5649bdaad23d] 2024-05-27T07:22:44.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5649bda98971] 2024-05-27T07:22:44.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 5: (SnapRealm::check_cache() const+0x13e) [0x5649bda96fa0] 2024-05-27T07:22:44.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5649bda97651] 2024-05-27T07:22:44.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5649bda2c347] 2024-05-27T07:22:44.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5649bd72887e] 2024-05-27T07:22:44.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5649bd72de94] 2024-05-27T07:22:44.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5649bd738969] 2024-05-27T07:22:44.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 11: (MDLog::_replay_thread()+0x1f38) [0x5649bdad148c] 2024-05-27T07:22:44.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5649bdada9cb] 2024-05-27T07:22:44.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 13: (Thread::entry_wrapper()+0x43) [0x7f22b93b972d] 2024-05-27T07:22:44.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 14: (Thread::_entry_func(void*)+0xd) [0x7f22b93b9749] 2024-05-27T07:22:44.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f22b7f5d1ca] 2024-05-27T07:22:44.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 16: clone() 2024-05-27T07:22:44.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: *** Caught signal (Aborted) ** 2024-05-27T07:22:44.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: in thread 7f22a3800700 thread_name:md_log_replay 2024-05-27T07:22:44.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:43.878+0000 7f22a3800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f22a3800700 time 2024-05-27T07:22:43.876711+0000 2024-05-27T07:22:44.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:44.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:44.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f22b93eaba6] 2024-05-27T07:22:44.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22b93eadd1] 2024-05-27T07:22:44.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5649bdaad23d] 2024-05-27T07:22:44.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5649bda98971] 2024-05-27T07:22:44.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 5: (SnapRealm::check_cache() const+0x13e) [0x5649bda96fa0] 2024-05-27T07:22:44.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5649bda97651] 2024-05-27T07:22:44.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5649bda2c347] 2024-05-27T07:22:44.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5649bd72887e] 2024-05-27T07:22:44.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5649bd72de94] 2024-05-27T07:22:44.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5649bd738969] 2024-05-27T07:22:44.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 11: (MDLog::_replay_thread()+0x1f38) [0x5649bdad148c] 2024-05-27T07:22:44.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5649bdada9cb] 2024-05-27T07:22:44.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 13: (Thread::entry_wrapper()+0x43) [0x7f22b93b972d] 2024-05-27T07:22:44.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 14: (Thread::_entry_func(void*)+0xd) [0x7f22b93b9749] 2024-05-27T07:22:44.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f22b7f5d1ca] 2024-05-27T07:22:44.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 16: clone() 2024-05-27T07:22:44.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:44.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22b7f67d20] 2024-05-27T07:22:44.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2: gsignal() 2024-05-27T07:22:44.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 3: abort() 2024-05-27T07:22:44.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f22b93eaced] 2024-05-27T07:22:44.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22b93eadd1] 2024-05-27T07:22:44.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5649bdaad23d] 2024-05-27T07:22:44.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5649bda98971] 2024-05-27T07:22:44.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 8: (SnapRealm::check_cache() const+0x13e) [0x5649bda96fa0] 2024-05-27T07:22:44.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5649bda97651] 2024-05-27T07:22:44.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5649bda2c347] 2024-05-27T07:22:44.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5649bd72887e] 2024-05-27T07:22:44.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5649bd72de94] 2024-05-27T07:22:44.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5649bd738969] 2024-05-27T07:22:44.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 14: (MDLog::_replay_thread()+0x1f38) [0x5649bdad148c] 2024-05-27T07:22:44.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5649bdada9cb] 2024-05-27T07:22:44.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 16: (Thread::entry_wrapper()+0x43) [0x7f22b93b972d] 2024-05-27T07:22:44.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 17: (Thread::_entry_func(void*)+0xd) [0x7f22b93b9749] 2024-05-27T07:22:44.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f22b7f5d1ca] 2024-05-27T07:22:44.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 19: clone() 2024-05-27T07:22:44.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:43.882+0000 7f22a3800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:44.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: in thread 7f22a3800700 thread_name:md_log_replay 2024-05-27T07:22:44.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:44.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22b7f67d20] 2024-05-27T07:22:44.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2: gsignal() 2024-05-27T07:22:44.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 3: abort() 2024-05-27T07:22:44.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f22b93eaced] 2024-05-27T07:22:44.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22b93eadd1] 2024-05-27T07:22:44.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5649bdaad23d] 2024-05-27T07:22:44.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5649bda98971] 2024-05-27T07:22:44.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 8: (SnapRealm::check_cache() const+0x13e) [0x5649bda96fa0] 2024-05-27T07:22:44.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5649bda97651] 2024-05-27T07:22:44.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5649bda2c347] 2024-05-27T07:22:44.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5649bd72887e] 2024-05-27T07:22:44.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5649bd72de94] 2024-05-27T07:22:44.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5649bd738969] 2024-05-27T07:22:44.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 14: (MDLog::_replay_thread()+0x1f38) [0x5649bdad148c] 2024-05-27T07:22:44.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5649bdada9cb] 2024-05-27T07:22:44.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 16: (Thread::entry_wrapper()+0x43) [0x7f22b93b972d] 2024-05-27T07:22:44.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 17: (Thread::_entry_func(void*)+0xd) [0x7f22b93b9749] 2024-05-27T07:22:44.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f22b7f5d1ca] 2024-05-27T07:22:44.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 19: clone() 2024-05-27T07:22:44.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:44.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: -1> 2024-05-27T07:22:43.878+0000 7f22a3800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f22a3800700 time 2024-05-27T07:22:43.876711+0000 2024-05-27T07:22:44.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:44.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:44.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f22b93eaba6] 2024-05-27T07:22:44.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22b93eadd1] 2024-05-27T07:22:44.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5649bdaad23d] 2024-05-27T07:22:44.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5649bda98971] 2024-05-27T07:22:44.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 5: (SnapRealm::check_cache() const+0x13e) [0x5649bda96fa0] 2024-05-27T07:22:44.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5649bda97651] 2024-05-27T07:22:44.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5649bda2c347] 2024-05-27T07:22:44.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5649bd72887e] 2024-05-27T07:22:44.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5649bd72de94] 2024-05-27T07:22:44.227 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5649bd738969] 2024-05-27T07:22:44.227 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 11: (MDLog::_replay_thread()+0x1f38) [0x5649bdad148c] 2024-05-27T07:22:44.227 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5649bdada9cb] 2024-05-27T07:22:44.227 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 13: (Thread::entry_wrapper()+0x43) [0x7f22b93b972d] 2024-05-27T07:22:44.228 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 14: (Thread::_entry_func(void*)+0xd) [0x7f22b93b9749] 2024-05-27T07:22:44.228 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f22b7f5d1ca] 2024-05-27T07:22:44.228 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 16: clone() 2024-05-27T07:22:44.228 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.228 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 0> 2024-05-27T07:22:43.882+0000 7f22a3800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:44.228 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: in thread 7f22a3800700 thread_name:md_log_replay 2024-05-27T07:22:44.228 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.229 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:44.229 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22b7f67d20] 2024-05-27T07:22:44.229 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2: gsignal() 2024-05-27T07:22:44.229 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 3: abort() 2024-05-27T07:22:44.229 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f22b93eaced] 2024-05-27T07:22:44.229 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22b93eadd1] 2024-05-27T07:22:44.230 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5649bdaad23d] 2024-05-27T07:22:44.230 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5649bda98971] 2024-05-27T07:22:44.230 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 8: (SnapRealm::check_cache() const+0x13e) [0x5649bda96fa0] 2024-05-27T07:22:44.230 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5649bda97651] 2024-05-27T07:22:44.230 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5649bda2c347] 2024-05-27T07:22:44.230 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5649bd72887e] 2024-05-27T07:22:44.230 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5649bd72de94] 2024-05-27T07:22:44.231 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5649bd738969] 2024-05-27T07:22:44.231 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 14: (MDLog::_replay_thread()+0x1f38) [0x5649bdad148c] 2024-05-27T07:22:44.231 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5649bdada9cb] 2024-05-27T07:22:44.231 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 16: (Thread::entry_wrapper()+0x43) [0x7f22b93b972d] 2024-05-27T07:22:44.231 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 17: (Thread::_entry_func(void*)+0xd) [0x7f22b93b9749] 2024-05-27T07:22:44.231 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f22b7f5d1ca] 2024-05-27T07:22:44.232 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 19: clone() 2024-05-27T07:22:44.232 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:44.232 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.232 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: -1> 2024-05-27T07:22:43.878+0000 7f22a3800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f22a3800700 time 2024-05-27T07:22:43.876711+0000 2024-05-27T07:22:44.232 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:44.232 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.233 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:44.233 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f22b93eaba6] 2024-05-27T07:22:44.233 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22b93eadd1] 2024-05-27T07:22:44.233 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5649bdaad23d] 2024-05-27T07:22:44.233 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5649bda98971] 2024-05-27T07:22:44.233 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 5: (SnapRealm::check_cache() const+0x13e) [0x5649bda96fa0] 2024-05-27T07:22:44.234 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5649bda97651] 2024-05-27T07:22:44.234 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5649bda2c347] 2024-05-27T07:22:44.234 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5649bd72887e] 2024-05-27T07:22:44.234 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5649bd72de94] 2024-05-27T07:22:44.234 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5649bd738969] 2024-05-27T07:22:44.234 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 11: (MDLog::_replay_thread()+0x1f38) [0x5649bdad148c] 2024-05-27T07:22:44.234 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5649bdada9cb] 2024-05-27T07:22:44.235 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 13: (Thread::entry_wrapper()+0x43) [0x7f22b93b972d] 2024-05-27T07:22:44.235 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 14: (Thread::_entry_func(void*)+0xd) [0x7f22b93b9749] 2024-05-27T07:22:44.235 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f22b7f5d1ca] 2024-05-27T07:22:44.235 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 16: clone() 2024-05-27T07:22:44.236 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.236 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 0> 2024-05-27T07:22:43.882+0000 7f22a3800700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:44.236 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: in thread 7f22a3800700 thread_name:md_log_replay 2024-05-27T07:22:44.236 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.236 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:44.236 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22b7f67d20] 2024-05-27T07:22:44.237 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2: gsignal() 2024-05-27T07:22:44.237 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 3: abort() 2024-05-27T07:22:44.237 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f22b93eaced] 2024-05-27T07:22:44.237 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f22b93eadd1] 2024-05-27T07:22:44.238 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5649bdaad23d] 2024-05-27T07:22:44.238 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5649bda98971] 2024-05-27T07:22:44.238 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 8: (SnapRealm::check_cache() const+0x13e) [0x5649bda96fa0] 2024-05-27T07:22:44.238 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5649bda97651] 2024-05-27T07:22:44.238 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5649bda2c347] 2024-05-27T07:22:44.238 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5649bd72887e] 2024-05-27T07:22:44.238 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5649bd72de94] 2024-05-27T07:22:44.239 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5649bd738969] 2024-05-27T07:22:44.239 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 14: (MDLog::_replay_thread()+0x1f38) [0x5649bdad148c] 2024-05-27T07:22:44.239 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5649bdada9cb] 2024-05-27T07:22:44.239 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 16: (Thread::entry_wrapper()+0x43) [0x7f22b93b972d] 2024-05-27T07:22:44.239 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 17: (Thread::_entry_func(void*)+0xd) [0x7f22b93b9749] 2024-05-27T07:22:44.239 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f22b7f5d1ca] 2024-05-27T07:22:44.240 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 19: clone() 2024-05-27T07:22:44.240 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:44.240 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73650]: 2024-05-27T07:22:44.240 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:43 smithi179 podman[73696]: 2024-05-27 07:22:43.996594223 +0000 UTC m=+0.031438618 container died 933dd87cee159d15860d4f1abf5a70bbe8e2a272e612cb6f82c036e973ffdc8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:22:44.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 48243 0.62 MB/sec execute 288 sec latency 64.621 ms 2024-05-27T07:22:44.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:44 smithi132 ceph-mon[35283]: pgmap v296: 129 pgs: 129 active+clean; 80 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 372 KiB/s wr, 52 op/s 2024-05-27T07:22:44.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:44 smithi132 ceph-mon[35283]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:22:44.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:44 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/1019742460,v1:172.21.15.179:6833/1019742460] up:boot 2024-05-27T07:22:44.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:44 smithi132 ceph-mon[35283]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:44.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:44 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:44.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:44 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:44.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:44 smithi043 ceph-mon[31531]: pgmap v296: 129 pgs: 129 active+clean; 80 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 372 KiB/s wr, 52 op/s 2024-05-27T07:22:44.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:44 smithi043 ceph-mon[31531]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:22:44.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:44 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/1019742460,v1:172.21.15.179:6833/1019742460] up:boot 2024-05-27T07:22:44.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:44 smithi043 ceph-mon[31531]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:44.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:44 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:44.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:44 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:44.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:44 smithi179 ceph-mon[35888]: pgmap v296: 129 pgs: 129 active+clean; 80 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 372 KiB/s wr, 52 op/s 2024-05-27T07:22:44.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:44 smithi179 ceph-mon[35888]: osdmap e121: 12 total, 12 up, 12 in 2024-05-27T07:22:44.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:44 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/1019742460,v1:172.21.15.179:6833/1019742460] up:boot 2024-05-27T07:22:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:44 smithi179 ceph-mon[35888]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:44 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:44.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:44 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:44.778 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:22:44.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:22:45.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:44 smithi179 podman[73696]: 2024-05-27 07:22:44.781874467 +0000 UTC m=+0.816718872 container remove 933dd87cee159d15860d4f1abf5a70bbe8e2a272e612cb6f82c036e973ffdc8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:22:45.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:44 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:22:45.215 INFO:tasks.workunit.client.0.smithi043.stdout: 1 48402 0.62 MB/sec execute 289 sec latency 65.826 ms 2024-05-27T07:22:45.278 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:45.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:45 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:22:45.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:45 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.445s CPU time. 2024-05-27T07:22:46.216 INFO:tasks.workunit.client.0.smithi043.stdout: 1 48519 0.62 MB/sec execute 290 sec latency 56.635 ms 2024-05-27T07:22:46.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:46 smithi132 ceph-mon[35283]: pgmap v298: 129 pgs: 129 active+clean; 80 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 244 KiB/s wr, 30 op/s 2024-05-27T07:22:46.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:46 smithi043 ceph-mon[31531]: pgmap v298: 129 pgs: 129 active+clean; 80 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 244 KiB/s wr, 30 op/s 2024-05-27T07:22:46.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:46 smithi179 ceph-mon[35888]: pgmap v298: 129 pgs: 129 active+clean; 80 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 244 KiB/s wr, 30 op/s 2024-05-27T07:22:47.216 INFO:tasks.workunit.client.0.smithi043.stdout: 1 48601 0.62 MB/sec execute 291 sec latency 50.747 ms 2024-05-27T07:22:47.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.915+0000 7f2927400700 1 -- 172.21.15.43:0/2269986969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2920105450 msgr2=0x7f2920105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:47.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.915+0000 7f2927400700 1 --2- 172.21.15.43:0/2269986969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2920105450 0x7f2920105860 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f2910009990 tx=0x7f2910009ca0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.915+0000 7f2927400700 1 -- 172.21.15.43:0/2269986969 shutdown_connections 2024-05-27T07:22:47.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.915+0000 7f2927400700 1 --2- 172.21.15.43:0/2269986969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f292010e8c0 0x7f2920110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.915+0000 7f2927400700 1 --2- 172.21.15.43:0/2269986969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2920105450 0x7f2920105860 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.915+0000 7f2927400700 1 --2- 172.21.15.43:0/2269986969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2920105da0 0x7f292010e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.915+0000 7f2927400700 1 -- 172.21.15.43:0/2269986969 >> 172.21.15.43:0/2269986969 conn(0x7f29200fedb0 msgr2=0x7f29201011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:47.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.916+0000 7f2927400700 1 -- 172.21.15.43:0/2269986969 shutdown_connections 2024-05-27T07:22:47.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.916+0000 7f2927400700 1 -- 172.21.15.43:0/2269986969 wait complete. 2024-05-27T07:22:47.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.918+0000 7f2927400700 1 Processor -- start 2024-05-27T07:22:47.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.918+0000 7f2927400700 1 -- start start 2024-05-27T07:22:47.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.918+0000 7f2927400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2920105450 0x7f292019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.918+0000 7f2927400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2920105da0 0x7f292019d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.919+0000 7f2927400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f292010e8c0 0x7f29201a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.919+0000 7f2927400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2920114310 con 0x7f292010e8c0 2024-05-27T07:22:47.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.919+0000 7f2927400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2920114190 con 0x7f2920105450 2024-05-27T07:22:47.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.919+0000 7f2927400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2920114490 con 0x7f2920105da0 2024-05-27T07:22:47.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.919+0000 7f2925400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f292010e8c0 0x7f29201a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.919+0000 7f2917e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2920105da0 0x7f292019d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.919+0000 7f2925400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f292010e8c0 0x7f29201a17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37746/0 (socket says 172.21.15.43:37746) 2024-05-27T07:22:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.919+0000 7f2924a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2920105450 0x7f292019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.919+0000 7f2925400700 1 -- 172.21.15.43:0/1767284429 learned_addr learned my addr 172.21.15.43:0/1767284429 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.920+0000 7f2925400700 1 -- 172.21.15.43:0/1767284429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2920105da0 msgr2=0x7f292019d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.920+0000 7f2925400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2920105da0 0x7f292019d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.920+0000 7f2925400700 1 -- 172.21.15.43:0/1767284429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2920105450 msgr2=0x7f292019cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.920+0000 7f2925400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2920105450 0x7f292019cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.920+0000 7f2925400700 1 -- 172.21.15.43:0/1767284429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29201a1de0 con 0x7f292010e8c0 2024-05-27T07:22:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.920+0000 7f2925400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f292010e8c0 0x7f29201a17f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f2918007d70 tx=0x7f291800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.921+0000 7f291e000700 1 -- 172.21.15.43:0/1767284429 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918016020 con 0x7f292010e8c0 2024-05-27T07:22:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.921+0000 7f2927400700 1 -- 172.21.15.43:0/1767284429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29201a20d0 con 0x7f292010e8c0 2024-05-27T07:22:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.921+0000 7f291e000700 1 -- 172.21.15.43:0/1767284429 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f291800ac90 con 0x7f292010e8c0 2024-05-27T07:22:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.921+0000 7f291e000700 1 -- 172.21.15.43:0/1767284429 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29180115e0 con 0x7f292010e8c0 2024-05-27T07:22:47.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.921+0000 7f2927400700 1 -- 172.21.15.43:0/1767284429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29201a2610 con 0x7f292010e8c0 2024-05-27T07:22:47.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.927+0000 7f291e000700 1 -- 172.21.15.43:0/1767284429 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f2918019020 con 0x7f292010e8c0 2024-05-27T07:22:47.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.928+0000 7f2935ee8b80 1 Processor -- start 2024-05-27T07:22:47.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.928+0000 7f2935ee8b80 1 -- start start 2024-05-27T07:22:47.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.928+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666bfc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.928+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666bfcdb0 0x555666bf68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.928+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 0x555666bf7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.928+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555666b24ec0 con 0x555666a77ec0 2024-05-27T07:22:47.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.928+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555666b14a60 con 0x555666bf6e30 2024-05-27T07:22:47.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.928+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555666b18aa0 con 0x555666bfcdb0 2024-05-27T07:22:47.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2917e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666bfc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2917e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666bfc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37756/0 (socket says 172.21.15.43:37756) 2024-05-27T07:22:47.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f291e000700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29040b2a70 0x7f29040b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2917e00700 1 -- 172.21.15.43:0/2853347301 learned_addr learned my addr 172.21.15.43:0/2853347301 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:47.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2924a00700 1 --2- 172.21.15.43:0/2853347301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666bfcdb0 0x555666bf68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f291e000700 1 -- 172.21.15.43:0/1767284429 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9408+0+0 (secure 0 0 0) 0x7f29180e2720 con 0x7f292010e8c0 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2917e00700 1 --2- 172.21.15.43:0/2853347301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 0x555666bf7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2925400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29040b2a70 0x7f29040b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2917e00700 1 -- 172.21.15.43:0/2853347301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666bfcdb0 msgr2=0x555666bf68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2917e00700 1 --2- 172.21.15.43:0/2853347301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666bfcdb0 0x555666bf68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2917e00700 1 -- 172.21.15.43:0/2853347301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 msgr2=0x555666bf7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2917e00700 1 --2- 172.21.15.43:0/2853347301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 0x555666bf7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.929+0000 7f2917e00700 1 -- 172.21.15.43:0/2853347301 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555666add4a0 con 0x555666a77ec0 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.930+0000 7f2925400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29040b2a70 0x7f29040b4f80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f29180dfc30 tx=0x7f2918012a70 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.930+0000 7f2917e00700 1 --2- 172.21.15.43:0/2853347301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666bfc870 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f290800bfb0 tx=0x7f290800b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d9e07cb8dd51d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.931+0000 7f2916000700 1 -- 172.21.15.43:0/2853347301 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908004030 con 0x555666a77ec0 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.931+0000 7f2916000700 1 -- 172.21.15.43:0/2853347301 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2908004830 con 0x555666a77ec0 2024-05-27T07:22:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.931+0000 7f2916000700 1 -- 172.21.15.43:0/2853347301 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908005360 con 0x555666a77ec0 2024-05-27T07:22:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.932+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2853347301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 msgr2=0x555666bfc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.932+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/2853347301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666bfc870 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f290800bfb0 tx=0x7f290800b9c0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.932+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2853347301 shutdown_connections 2024-05-27T07:22:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.932+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/2853347301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 0x555666bf7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.932+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/2853347301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666bfc870 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.932+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/2853347301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666bfcdb0 0x555666bf68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.932+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2853347301 >> 172.21.15.43:0/2853347301 conn(0x555666ad8040 msgr2=0x555666bf1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:47.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.932+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2853347301 shutdown_connections 2024-05-27T07:22:47.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.932+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2853347301 wait complete. 2024-05-27T07:22:47.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.933+0000 7f2935ee8b80 1 -- start start 2024-05-27T07:22:47.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.933+0000 7f2935ee8b80 1 Processor -- start 2024-05-27T07:22:47.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666a77ec0 0x555666c006b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bf6e30 0x555666c00bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bfcdb0 0x555666c01130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555666a1b020 con 0x555666bf6e30 2024-05-27T07:22:47.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555666ab57e0 con 0x555666bfcdb0 2024-05-27T07:22:47.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555666c03880 con 0x555666a77ec0 2024-05-27T07:22:47.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2935ee8b80 10 client.0 _ll_register_callbacks cb 0x555666abecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:47.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2924a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bf6e30 0x555666c00bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2924a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bf6e30 0x555666c00bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37772/0 (socket says 172.21.15.43:37772) 2024-05-27T07:22:47.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2924a00700 1 -- 172.21.15.43:0/2995256314 learned_addr learned my addr 172.21.15.43:0/2995256314 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:47.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2917e00700 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666a77ec0 0x555666c006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.934+0000 7f2917e00700 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bfcdb0 0x555666c01130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.935+0000 7f2924a00700 1 -- 172.21.15.43:0/2995256314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666a77ec0 msgr2=0x555666c006b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:47.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.935+0000 7f2924a00700 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666a77ec0 0x555666c006b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.935+0000 7f2924a00700 1 -- 172.21.15.43:0/2995256314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bfcdb0 msgr2=0x555666c01130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:47.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.935+0000 7f2924a00700 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bfcdb0 0x555666c01130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.935+0000 7f2924a00700 1 -- 172.21.15.43:0/2995256314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555666a64950 con 0x555666bf6e30 2024-05-27T07:22:47.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.935+0000 7f2924a00700 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bf6e30 0x555666c00bf0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f2910015000 tx=0x7f29100057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539b014b7c795eae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:47.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.935+0000 7f290ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:47.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.936+0000 7f290ea00700 1 -- 172.21.15.43:0/2995256314 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29100039c0 con 0x555666bf6e30 2024-05-27T07:22:47.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.936+0000 7f290ea00700 1 -- 172.21.15.43:0/2995256314 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9408+0+0 (secure 0 0 0) 0x7f2910003b60 con 0x555666bf6e30 2024-05-27T07:22:47.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.936+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2995256314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2908004830 con 0x555666bf6e30 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.936+0000 7f290ea00700 1 -- 172.21.15.43:0/2995256314 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f2910030050 con 0x555666bf6e30 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.936+0000 7f2935ee8b80 10 client.16794 fetch_fsmap learned FSMap version 91 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.936+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2995256314 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x555666ae50e0 con 0x555666bf6e30 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.937+0000 7f290ea00700 1 -- 172.21.15.43:0/2995256314 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2908004830 con 0x555666bf6e30 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.937+0000 7f290ea00700 1 -- 172.21.15.43:0/2995256314 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 91) ==== 5966+0+0 (secure 0 0 0) 0x7f291000f4f0 con 0x555666bf6e30 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.937+0000 7f2935ee8b80 10 client.16794 fetch_fsmap finished waiting for FSMap version 91 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.937+0000 7f2935ee8b80 10 client.16794 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.937+0000 7f2935ee8b80 20 client.16794 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.937+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555666c10bc0 0x555666c13010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.937+0000 7f2935ee8b80 4 client.16794 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.937+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2995256314 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555666b26e10 con 0x555666c10bc0 2024-05-27T07:22:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.938+0000 7f2917e00700 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555666c10bc0 0x555666c13010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.939+0000 7f2917e00700 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555666c10bc0 0x555666c13010 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df440fa4ddb16a33 server_cookie=c2bf7a1c46d8f858 in_seq=0 out_seq=0 2024-05-27T07:22:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.939+0000 7f290ea00700 10 client.16794 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.941+0000 7f290ea00700 1 -- 172.21.15.43:0/2995256314 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555666b26e10 con 0x555666c10bc0 2024-05-27T07:22:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.941+0000 7f290ea00700 10 client.16794 handle_command_reply: tid=0 2024-05-27T07:22:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.941+0000 7f2935ee8b80 1 client.16794 shutdown 2024-05-27T07:22:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.941+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2995256314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bf6e30 msgr2=0x555666c00bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.941+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bf6e30 0x555666c00bf0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f2910015000 tx=0x7f29100057e0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.941+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2995256314 shutdown_connections 2024-05-27T07:22:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.941+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bfcdb0 0x555666c01130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.941+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666a77ec0 0x555666c006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.941+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555666c10bc0 0x555666c13010 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.942+0000 7f290ea00700 0 client.16794 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.942+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/2995256314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bf6e30 0x555666c00bf0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.942+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2995256314 >> 172.21.15.43:0/2995256314 conn(0x555666ad8040 msgr2=0x555666bf1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.942+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2995256314 shutdown_connections 2024-05-27T07:22:47.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.942+0000 7f2935ee8b80 1 -- 172.21.15.43:0/2995256314 wait complete. 2024-05-27T07:22:47.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.942+0000 7f2935ee8b80 20 client.16794 trim_cache size 0 max 16384 2024-05-27T07:22:47.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.965+0000 7f2935ee8b80 1 Processor -- start 2024-05-27T07:22:47.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.966+0000 7f2935ee8b80 1 -- start start 2024-05-27T07:22:47.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.966+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666a77ec0 0x555666bf68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.966+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 0x555666c00ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.966+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bfcdb0 0x555666c010e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.966+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f29201b3180 con 0x555666bfcdb0 2024-05-27T07:22:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.966+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f29201b3850 con 0x555666bf6e30 2024-05-27T07:22:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.966+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f29201b0f70 con 0x555666a77ec0 2024-05-27T07:22:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.966+0000 7f2917e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bfcdb0 0x555666c010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.966+0000 7f2917e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bfcdb0 0x555666c010e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37778/0 (socket says 172.21.15.43:37778) 2024-05-27T07:22:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.967+0000 7f2917e00700 1 -- 172.21.15.43:0/1996862690 learned_addr learned my addr 172.21.15.43:0/1996862690 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.967+0000 7f2924a00700 1 --2- 172.21.15.43:0/1996862690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 0x555666c00ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.967+0000 7f2917e00700 1 --2- 172.21.15.43:0/1996862690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666a77ec0 0x555666bf68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.967+0000 7f2917e00700 1 -- 172.21.15.43:0/1996862690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666a77ec0 msgr2=0x555666bf68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.967+0000 7f2917e00700 1 --2- 172.21.15.43:0/1996862690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666a77ec0 0x555666bf68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.967+0000 7f2917e00700 1 -- 172.21.15.43:0/1996862690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 msgr2=0x555666c00ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.967+0000 7f2917e00700 1 --2- 172.21.15.43:0/1996862690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 0x555666c00ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.967+0000 7f2917e00700 1 -- 172.21.15.43:0/1996862690 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555666c02f50 con 0x555666bfcdb0 2024-05-27T07:22:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.967+0000 7f2924a00700 1 --2- 172.21.15.43:0/1996862690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 0x555666c00ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.968+0000 7f2917e00700 1 --2- 172.21.15.43:0/1996862690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bfcdb0 0x555666c010e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2908010a40 tx=0x7f290800b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8961d906c85967d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.968+0000 7f2915600700 1 -- 172.21.15.43:0/1996862690 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908032030 con 0x555666bfcdb0 2024-05-27T07:22:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.968+0000 7f2915600700 1 -- 172.21.15.43:0/1996862690 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2908015040 con 0x555666bfcdb0 2024-05-27T07:22:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.968+0000 7f2915600700 1 -- 172.21.15.43:0/1996862690 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908003f30 con 0x555666bfcdb0 2024-05-27T07:22:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.969+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1996862690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bfcdb0 msgr2=0x555666c010e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.969+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/1996862690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bfcdb0 0x555666c010e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2908010a40 tx=0x7f290800b9c0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.969+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1996862690 shutdown_connections 2024-05-27T07:22:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.969+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/1996862690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666bfcdb0 0x555666c010e0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.969+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/1996862690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666a77ec0 0x555666bf68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.969+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/1996862690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bf6e30 0x555666c00ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.969+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1996862690 >> 172.21.15.43:0/1996862690 conn(0x555666ad8040 msgr2=0x555666c00490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.969+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1996862690 shutdown_connections 2024-05-27T07:22:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.969+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1996862690 wait complete. 2024-05-27T07:22:47.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.970+0000 7f2935ee8b80 1 -- start start 2024-05-27T07:22:47.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.970+0000 7f2935ee8b80 1 Processor -- start 2024-05-27T07:22:47.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.971+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666c08d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.971+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666bf6e30 0x555666c09280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.971+0000 7f2935ee8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bfcdb0 0x555666c05ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.971+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f29201b1640 con 0x555666a77ec0 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.971+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f29201b2ab0 con 0x555666bfcdb0 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.971+0000 7f2935ee8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f29201b1d10 con 0x555666bf6e30 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.971+0000 7f2935ee8b80 10 client.0 _ll_register_callbacks cb 0x555666abecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.971+0000 7f2917e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666c08d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.971+0000 7f2917e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666c08d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37794/0 (socket says 172.21.15.43:37794) 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.971+0000 7f2917e00700 1 -- 172.21.15.43:0/1504607650 learned_addr learned my addr 172.21.15.43:0/1504607650 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.972+0000 7f2924a00700 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666bf6e30 0x555666c09280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.972+0000 7f2917e00700 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bfcdb0 0x555666c05ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.972+0000 7f2917e00700 1 -- 172.21.15.43:0/1504607650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666bf6e30 msgr2=0x555666c09280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.972+0000 7f2917e00700 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666bf6e30 0x555666c09280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.972+0000 7f2917e00700 1 -- 172.21.15.43:0/1504607650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bfcdb0 msgr2=0x555666c05ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.972+0000 7f2917e00700 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bfcdb0 0x555666c05ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.972+0000 7f2917e00700 1 -- 172.21.15.43:0/1504607650 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555666c06290 con 0x555666a77ec0 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.972+0000 7f2917e00700 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666c08d40 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f2908011000 tx=0x7f290800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f394d6aa92e318c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.972+0000 7f290ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.973+0000 7f290ea00700 1 -- 172.21.15.43:0/1504607650 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290803e060 con 0x555666a77ec0 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.973+0000 7f290ea00700 1 -- 172.21.15.43:0/1504607650 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9408+0+0 (secure 0 0 0) 0x7f2908004e00 con 0x555666a77ec0 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.973+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1504607650 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2908015040 con 0x555666a77ec0 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.973+0000 7f290ea00700 1 -- 172.21.15.43:0/1504607650 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f2908032040 con 0x555666a77ec0 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.973+0000 7f2935ee8b80 10 client.16800 fetch_fsmap learned FSMap version 91 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.973+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1504607650 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x555666c7c110 con 0x555666a77ec0 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.973+0000 7f290ea00700 1 -- 172.21.15.43:0/1504607650 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2908015040 con 0x555666a77ec0 2024-05-27T07:22:47.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.974+0000 7f290ea00700 1 -- 172.21.15.43:0/1504607650 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 91) ==== 5966+0+0 (secure 0 0 0) 0x7f2908003880 con 0x555666a77ec0 2024-05-27T07:22:47.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.974+0000 7f2935ee8b80 10 client.16800 fetch_fsmap finished waiting for FSMap version 91 2024-05-27T07:22:47.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.974+0000 7f2935ee8b80 10 client.16800 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:47.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.974+0000 7f2935ee8b80 20 client.16800 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:47.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.974+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555666c7c660 0x555666c7ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:47.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.974+0000 7f2935ee8b80 4 client.16800 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:22:47.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.974+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1504607650 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555666b205f0 con 0x555666c7c660 2024-05-27T07:22:47.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.974+0000 7f2924a00700 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555666c7c660 0x555666c7ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:47.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.975+0000 7f2924a00700 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555666c7c660 0x555666c7ea80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40d6e62bb5e8142e server_cookie=919bf91fe793c427 in_seq=0 out_seq=0 2024-05-27T07:22:47.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.975+0000 7f290ea00700 10 client.16800 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:47.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f290ea00700 1 -- 172.21.15.43:0/1504607650 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555666b205f0 con 0x555666c7c660 2024-05-27T07:22:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f290ea00700 10 client.16800 handle_command_reply: tid=0 2024-05-27T07:22:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 client.16800 shutdown 2024-05-27T07:22:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1504607650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 msgr2=0x555666c08d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666c08d40 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f2908011000 tx=0x7f290800bd50 comp rx=0 tx=0).stop 2024-05-27T07:22:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1504607650 shutdown_connections 2024-05-27T07:22:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555666bfcdb0 0x555666c05ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555666c7c660 0x555666c7ea80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555666a77ec0 0x555666c08d40 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 --2- 172.21.15.43:0/1504607650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555666bf6e30 0x555666c09280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1504607650 >> 172.21.15.43:0/1504607650 conn(0x555666ad8040 msgr2=0x555666c00ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:47.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f290ea00700 0 client.16800 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:47.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1504607650 shutdown_connections 2024-05-27T07:22:47.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.979+0000 7f2935ee8b80 1 -- 172.21.15.43:0/1504607650 wait complete. 2024-05-27T07:22:47.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.980+0000 7f2935ee8b80 20 client.16800 trim_cache size 0 max 16384 2024-05-27T07:22:47.971 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:22:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:22:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "ad97df14-d1ea-4167-930a-b80008641902", 2024-05-27T07:22:47.971 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:22:47.971 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:22:47.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.980+0000 7f2927400700 1 -- 172.21.15.43:0/1767284429 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29040b2a70 msgr2=0x7f29040b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:47.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.980+0000 7f2927400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29040b2a70 0x7f29040b4f80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f29180dfc30 tx=0x7f2918012a70 comp rx=0 tx=0).stop 2024-05-27T07:22:47.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.980+0000 7f2927400700 1 -- 172.21.15.43:0/1767284429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f292010e8c0 msgr2=0x7f29201a17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:47.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.980+0000 7f2927400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f292010e8c0 0x7f29201a17f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f2918007d70 tx=0x7f291800a340 comp rx=0 tx=0).stop 2024-05-27T07:22:47.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.981+0000 7f2927400700 1 -- 172.21.15.43:0/1767284429 shutdown_connections 2024-05-27T07:22:47.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.981+0000 7f2927400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29040b2a70 0x7f29040b4f80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.981+0000 7f2927400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f292010e8c0 0x7f29201a17f0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.981+0000 7f2927400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2920105450 0x7f292019cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.981+0000 7f2927400700 1 --2- 172.21.15.43:0/1767284429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2920105da0 0x7f292019d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.981+0000 7f2927400700 1 -- 172.21.15.43:0/1767284429 >> 172.21.15.43:0/1767284429 conn(0x7f29200fedb0 msgr2=0x7f2920103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.981+0000 7f2927400700 1 -- 172.21.15.43:0/1767284429 shutdown_connections 2024-05-27T07:22:47.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:47.981+0000 7f2927400700 1 -- 172.21.15.43:0/1767284429 wait complete. 2024-05-27T07:22:48.216 INFO:tasks.workunit.client.0.smithi043.stdout: 1 48726 0.62 MB/sec execute 292 sec latency 78.746 ms 2024-05-27T07:22:48.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:48 smithi132 ceph-mon[35283]: pgmap v299: 129 pgs: 129 active+clean; 81 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 348 KiB/s wr, 51 op/s 2024-05-27T07:22:48.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:48 smithi043 ceph-mon[31531]: pgmap v299: 129 pgs: 129 active+clean; 81 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 348 KiB/s wr, 51 op/s 2024-05-27T07:22:48.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:48 smithi179 ceph-mon[35888]: pgmap v299: 129 pgs: 129 active+clean; 81 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 348 KiB/s wr, 51 op/s 2024-05-27T07:22:48.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:22:49.216 INFO:tasks.workunit.client.0.smithi043.stdout: 1 48906 0.62 MB/sec execute 293 sec latency 56.941 ms 2024-05-27T07:22:49.446 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:22:49.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:22:49.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:49 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:22:49.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:22:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:49 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:22:49.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:22:49.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:49.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:49 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:22:50.216 INFO:tasks.workunit.client.0.smithi043.stdout: 1 49118 0.63 MB/sec execute 294 sec latency 49.897 ms 2024-05-27T07:22:50.624 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:50 smithi043 ceph-mon[31531]: pgmap v300: 129 pgs: 129 active+clean; 81 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 331 KiB/s wr, 49 op/s 2024-05-27T07:22:50.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:50 smithi179 ceph-mon[35888]: pgmap v300: 129 pgs: 129 active+clean; 81 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 331 KiB/s wr, 49 op/s 2024-05-27T07:22:50.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:50 smithi132 ceph-mon[35283]: pgmap v300: 129 pgs: 129 active+clean; 81 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 331 KiB/s wr, 49 op/s 2024-05-27T07:22:51.216 INFO:tasks.workunit.client.0.smithi043.stdout: 1 49345 0.63 MB/sec execute 295 sec latency 49.162 ms 2024-05-27T07:22:51.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.964+0000 7f3edd400700 1 -- 172.21.15.43:0/417781940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ed8109860 msgr2=0x7f3ed81111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:51.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.964+0000 7f3edd400700 1 --2- 172.21.15.43:0/417781940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ed8109860 0x7f3ed81111c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000b0d0 tx=0x7f3ed000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.965+0000 7f3edd400700 1 -- 172.21.15.43:0/417781940 shutdown_connections 2024-05-27T07:22:51.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.965+0000 7f3edd400700 1 --2- 172.21.15.43:0/417781940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ed8109860 0x7f3ed81111c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.965+0000 7f3edd400700 1 --2- 172.21.15.43:0/417781940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ed81045e0 0x7f3ed81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.965+0000 7f3edd400700 1 --2- 172.21.15.43:0/417781940 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed8108f10 0x7f3ed8109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.965+0000 7f3edd400700 1 -- 172.21.15.43:0/417781940 >> 172.21.15.43:0/417781940 conn(0x7f3ed8100030 msgr2=0x7f3ed8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:51.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.965+0000 7f3edd400700 1 -- 172.21.15.43:0/417781940 shutdown_connections 2024-05-27T07:22:51.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.965+0000 7f3edd400700 1 -- 172.21.15.43:0/417781940 wait complete. 2024-05-27T07:22:51.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.966+0000 7f3edd400700 1 Processor -- start 2024-05-27T07:22:51.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.966+0000 7f3edd400700 1 -- start start 2024-05-27T07:22:51.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.967+0000 7f3edd400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ed81045e0 0x7f3ed819cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.967+0000 7f3edd400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ed8108f10 0x7f3ed819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.967+0000 7f3edd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed8109860 0x7f3ed81a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.967+0000 7f3edd400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed8114340 con 0x7f3ed81045e0 2024-05-27T07:22:51.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.967+0000 7f3edd400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81141c0 con 0x7f3ed8109860 2024-05-27T07:22:51.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.967+0000 7f3edd400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81144c0 con 0x7f3ed8108f10 2024-05-27T07:22:51.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.967+0000 7f3ed6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ed81045e0 0x7f3ed819cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.967+0000 7f3ed6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ed81045e0 0x7f3ed819cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37812/0 (socket says 172.21.15.43:37812) 2024-05-27T07:22:51.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.967+0000 7f3ed6800700 1 -- 172.21.15.43:0/4015269144 learned_addr learned my addr 172.21.15.43:0/4015269144 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:51.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.967+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ed8108f10 0x7f3ed819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.968+0000 7f3ed7200700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed8109860 0x7f3ed81a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.968+0000 7f3ed6800700 1 -- 172.21.15.43:0/4015269144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ed8108f10 msgr2=0x7f3ed819d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:51.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.968+0000 7f3ed6800700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ed8108f10 0x7f3ed819d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.968+0000 7f3ed6800700 1 -- 172.21.15.43:0/4015269144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed8109860 msgr2=0x7f3ed81a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:51.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.968+0000 7f3ed6800700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed8109860 0x7f3ed81a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.968+0000 7f3ed6800700 1 -- 172.21.15.43:0/4015269144 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed81a1e00 con 0x7f3ed81045e0 2024-05-27T07:22:51.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.968+0000 7f3ed7200700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed8109860 0x7f3ed81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:22:51.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.968+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ed8108f10 0x7f3ed819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:51.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.968+0000 7f3ed6800700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ed81045e0 0x7f3ed819cf30 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800d950 tx=0x7f3ec800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:51.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.969+0000 7f3ecf400700 1 -- 172.21.15.43:0/4015269144 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8016070 con 0x7f3ed81045e0 2024-05-27T07:22:51.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.969+0000 7f3ecf400700 1 -- 172.21.15.43:0/4015269144 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ec8010460 con 0x7f3ed81045e0 2024-05-27T07:22:51.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.969+0000 7f3ecf400700 1 -- 172.21.15.43:0/4015269144 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec800f620 con 0x7f3ed81045e0 2024-05-27T07:22:51.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.969+0000 7f3edd400700 1 -- 172.21.15.43:0/4015269144 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed81a20f0 con 0x7f3ed81045e0 2024-05-27T07:22:51.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.969+0000 7f3edd400700 1 -- 172.21.15.43:0/4015269144 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed81a9a10 con 0x7f3ed81045e0 2024-05-27T07:22:51.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.974+0000 7f3ecf400700 1 -- 172.21.15.43:0/4015269144 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3ec8019070 con 0x7f3ed81045e0 2024-05-27T07:22:51.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.974+0000 7f3eec030b80 1 Processor -- start 2024-05-27T07:22:51.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.974+0000 7f3eec030b80 1 -- start start 2024-05-27T07:22:51.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.975+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 0x55e982755330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.975+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9825d6e40 0x55e982755870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.975+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e982755e70 0x55e982762110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.975+0000 7f3eec030b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e982683e40 con 0x55e982755e70 2024-05-27T07:22:51.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.975+0000 7f3eec030b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e9826739e0 con 0x55e9825d6e40 2024-05-27T07:22:51.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.975+0000 7f3eec030b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e982677a20 con 0x55e9826c29f0 2024-05-27T07:22:51.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.975+0000 7f3ed5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e982755e70 0x55e982762110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.975+0000 7f3ed5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e982755e70 0x55e982762110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37826/0 (socket says 172.21.15.43:37826) 2024-05-27T07:22:51.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.975+0000 7f3ed5e00700 1 -- 172.21.15.43:0/60068596 learned_addr learned my addr 172.21.15.43:0/60068596 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:51.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ed6800700 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9825d6e40 0x55e982755870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ed7200700 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 0x55e982755330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ecf400700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3ebc0b2a70 0x7f3ebc0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ecf400700 1 -- 172.21.15.43:0/4015269144 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9408+0+0 (secure 0 0 0) 0x7f3ec80e2480 con 0x7f3ed81045e0 2024-05-27T07:22:51.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ed5e00700 1 -- 172.21.15.43:0/60068596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 msgr2=0x55e982755330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:51.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 0x55e982755330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ed5e00700 1 -- 172.21.15.43:0/60068596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9825d6e40 msgr2=0x55e982755870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:51.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9825d6e40 0x55e982755870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ed5e00700 1 -- 172.21.15.43:0/60068596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e98263c420 con 0x55e982755e70 2024-05-27T07:22:51.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ed7200700 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 0x55e982755330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:51.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ed6800700 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9825d6e40 0x55e982755870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:22:51.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.976+0000 7f3ed7200700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3ebc0b2a70 0x7f3ebc0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.977+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e982755e70 0x55e982762110 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400dcb0 tx=0x7f3ec400f360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff5d9e0ab919e13 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:51.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.977+0000 7f3ec3e00700 1 -- 172.21.15.43:0/60068596 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec400fc90 con 0x55e982755e70 2024-05-27T07:22:51.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.978+0000 7f3ec3e00700 1 -- 172.21.15.43:0/60068596 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ec400fe30 con 0x55e982755e70 2024-05-27T07:22:51.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.978+0000 7f3ec3e00700 1 -- 172.21.15.43:0/60068596 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec40106c0 con 0x55e982755e70 2024-05-27T07:22:51.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.978+0000 7f3ed7200700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3ebc0b2a70 0x7f3ebc0b4f80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000b820 tx=0x7f3ed00099e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:51.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.979+0000 7f3eec030b80 1 -- 172.21.15.43:0/60068596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e982755e70 msgr2=0x55e982762110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:51.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.979+0000 7f3eec030b80 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e982755e70 0x55e982762110 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400dcb0 tx=0x7f3ec400f360 comp rx=0 tx=0).stop 2024-05-27T07:22:51.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.979+0000 7f3eec030b80 1 -- 172.21.15.43:0/60068596 shutdown_connections 2024-05-27T07:22:51.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.979+0000 7f3eec030b80 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e982755e70 0x55e982762110 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.979+0000 7f3eec030b80 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 0x55e982755330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.979+0000 7f3eec030b80 1 --2- 172.21.15.43:0/60068596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9825d6e40 0x55e982755870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.979+0000 7f3eec030b80 1 -- 172.21.15.43:0/60068596 >> 172.21.15.43:0/60068596 conn(0x55e982670390 msgr2=0x55e9827514c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:51.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.979+0000 7f3eec030b80 1 -- 172.21.15.43:0/60068596 shutdown_connections 2024-05-27T07:22:51.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.979+0000 7f3eec030b80 1 -- 172.21.15.43:0/60068596 wait complete. 2024-05-27T07:22:51.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.980+0000 7f3eec030b80 1 -- start start 2024-05-27T07:22:51.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.980+0000 7f3eec030b80 1 Processor -- start 2024-05-27T07:22:51.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.981+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9825d6e40 0x55e98276e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.981+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9826c29f0 0x55e98276e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.981+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e982755e70 0x55e982768670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.981+0000 7f3eec030b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e982579fa0 con 0x55e9826c29f0 2024-05-27T07:22:51.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.981+0000 7f3eec030b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e982614760 con 0x55e982755e70 2024-05-27T07:22:51.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.981+0000 7f3eec030b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e982762b80 con 0x55e9825d6e40 2024-05-27T07:22:51.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.981+0000 7f3eec030b80 10 client.0 _ll_register_callbacks cb 0x55e98261dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:51.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.981+0000 7f3ed5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9826c29f0 0x55e98276e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.981+0000 7f3ed5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9826c29f0 0x55e98276e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37832/0 (socket says 172.21.15.43:37832) 2024-05-27T07:22:51.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.981+0000 7f3ed5e00700 1 -- 172.21.15.43:0/2316658035 learned_addr learned my addr 172.21.15.43:0/2316658035 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:51.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.982+0000 7f3ed6800700 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9825d6e40 0x55e98276e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.982+0000 7f3ed7200700 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e982755e70 0x55e982768670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.982+0000 7f3ed5e00700 1 -- 172.21.15.43:0/2316658035 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9825d6e40 msgr2=0x55e98276e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:51.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.982+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9825d6e40 0x55e98276e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.982+0000 7f3ed5e00700 1 -- 172.21.15.43:0/2316658035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e982755e70 msgr2=0x55e982768670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:51.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.982+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e982755e70 0x55e982768670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.982+0000 7f3ed5e00700 1 -- 172.21.15.43:0/2316658035 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9825c38d0 con 0x55e9826c29f0 2024-05-27T07:22:51.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.983+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9826c29f0 0x55e98276e7b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005270 tx=0x7f3ec40052a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93810371a8107627 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:51.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.983+0000 7f3ec0c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:51.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.983+0000 7f3ec0c00700 1 -- 172.21.15.43:0/2316658035 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4011440 con 0x55e9826c29f0 2024-05-27T07:22:51.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.983+0000 7f3ec0c00700 1 -- 172.21.15.43:0/2316658035 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9408+0+0 (secure 0 0 0) 0x7f3ec40115e0 con 0x55e9826c29f0 2024-05-27T07:22:51.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.983+0000 7f3eec030b80 1 -- 172.21.15.43:0/2316658035 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ec400fe30 con 0x55e9826c29f0 2024-05-27T07:22:51.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.983+0000 7f3ec0c00700 1 -- 172.21.15.43:0/2316658035 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3ec4033050 con 0x55e9826c29f0 2024-05-27T07:22:51.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.983+0000 7f3eec030b80 10 client.16812 fetch_fsmap learned FSMap version 91 2024-05-27T07:22:51.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.983+0000 7f3eec030b80 1 -- 172.21.15.43:0/2316658035 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x55e982644060 con 0x55e9826c29f0 2024-05-27T07:22:51.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.984+0000 7f3ec0c00700 1 -- 172.21.15.43:0/2316658035 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ec400fe30 con 0x55e9826c29f0 2024-05-27T07:22:51.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.984+0000 7f3ec0c00700 1 -- 172.21.15.43:0/2316658035 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 91) ==== 5966+0+0 (secure 0 0 0) 0x7f3ec4024070 con 0x55e9826c29f0 2024-05-27T07:22:51.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.984+0000 7f3eec030b80 10 client.16812 fetch_fsmap finished waiting for FSMap version 91 2024-05-27T07:22:51.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.984+0000 7f3eec030b80 10 client.16812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:51.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.984+0000 7f3eec030b80 20 client.16812 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:51.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.984+0000 7f3eec030b80 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e9827781b0 0x55e98277a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:51.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.984+0000 7f3eec030b80 4 client.16812 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:22:51.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.984+0000 7f3eec030b80 1 -- 172.21.15.43:0/2316658035 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e982685d90 con 0x55e9827781b0 2024-05-27T07:22:51.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.985+0000 7f3ed6800700 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e9827781b0 0x55e98277a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:51.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.986+0000 7f3ed6800700 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e9827781b0 0x55e98277a600 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dd856c834f3cada server_cookie=dbabcf51eac45585 in_seq=0 out_seq=0 2024-05-27T07:22:51.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.986+0000 7f3ec0c00700 10 client.16812 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:51.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.988+0000 7f3ec0c00700 1 -- 172.21.15.43:0/2316658035 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e982685d90 con 0x55e9827781b0 2024-05-27T07:22:51.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.988+0000 7f3ec0c00700 10 client.16812 handle_command_reply: tid=0 2024-05-27T07:22:51.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.988+0000 7f3eec030b80 1 client.16812 shutdown 2024-05-27T07:22:51.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.989+0000 7f3eec030b80 1 -- 172.21.15.43:0/2316658035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9826c29f0 msgr2=0x55e98276e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:51.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.989+0000 7f3eec030b80 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9826c29f0 0x55e98276e7b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005270 tx=0x7f3ec40052a0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.989+0000 7f3eec030b80 1 -- 172.21.15.43:0/2316658035 shutdown_connections 2024-05-27T07:22:51.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.989+0000 7f3eec030b80 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e982755e70 0x55e982768670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.989+0000 7f3eec030b80 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9825d6e40 0x55e98276e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.989+0000 7f3eec030b80 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e9827781b0 0x55e98277a600 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.989+0000 7f3eec030b80 1 --2- 172.21.15.43:0/2316658035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9826c29f0 0x55e98276e7b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:51.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.989+0000 7f3eec030b80 1 -- 172.21.15.43:0/2316658035 >> 172.21.15.43:0/2316658035 conn(0x55e982670390 msgr2=0x55e982755310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:51.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.989+0000 7f3ec0c00700 0 client.16812 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:51.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.989+0000 7f3eec030b80 1 -- 172.21.15.43:0/2316658035 shutdown_connections 2024-05-27T07:22:51.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.990+0000 7f3eec030b80 1 -- 172.21.15.43:0/2316658035 wait complete. 2024-05-27T07:22:51.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:51.990+0000 7f3eec030b80 20 client.16812 trim_cache size 0 max 16384 2024-05-27T07:22:52.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.015+0000 7f3eec030b80 1 Processor -- start 2024-05-27T07:22:52.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.015+0000 7f3eec030b80 1 -- start start 2024-05-27T07:22:52.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.015+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e9827550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9826c29f0 0x55e9827555f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e982755e70 0x55e98277fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3eec030b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81aefb0 con 0x55e9825d6e40 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3eec030b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81af680 con 0x55e9826c29f0 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3eec030b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81acda0 con 0x55e982755e70 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3ed6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e9827550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3ed6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e9827550b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37842/0 (socket says 172.21.15.43:37842) 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3ed6800700 1 -- 172.21.15.43:0/1471180057 learned_addr learned my addr 172.21.15.43:0/1471180057 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3ed7200700 1 --2- 172.21.15.43:0/1471180057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e982755e70 0x55e98277fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/1471180057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9826c29f0 0x55e9827555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3ed6800700 1 -- 172.21.15.43:0/1471180057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e982755e70 msgr2=0x55e98277fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.016+0000 7f3ed6800700 1 --2- 172.21.15.43:0/1471180057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e982755e70 0x55e98277fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.017+0000 7f3ed6800700 1 -- 172.21.15.43:0/1471180057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9826c29f0 msgr2=0x55e9827555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.017+0000 7f3ed6800700 1 --2- 172.21.15.43:0/1471180057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9826c29f0 0x55e9827555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.017+0000 7f3ed6800700 1 -- 172.21.15.43:0/1471180057 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e982762670 con 0x55e9825d6e40 2024-05-27T07:22:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.017+0000 7f3ed6800700 1 --2- 172.21.15.43:0/1471180057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e9827550b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800b2f0 tx=0x7f3ec8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c1058985a3bcd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:52.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.017+0000 7f3ec3400700 1 -- 172.21.15.43:0/1471180057 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8020040 con 0x55e9825d6e40 2024-05-27T07:22:52.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.017+0000 7f3ec3400700 1 -- 172.21.15.43:0/1471180057 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ec80a8fa0 con 0x55e9825d6e40 2024-05-27T07:22:52.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.017+0000 7f3ec3400700 1 -- 172.21.15.43:0/1471180057 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec80a5720 con 0x55e9825d6e40 2024-05-27T07:22:52.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.018+0000 7f3eec030b80 1 -- 172.21.15.43:0/1471180057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 msgr2=0x55e9827550b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:52.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.018+0000 7f3eec030b80 1 --2- 172.21.15.43:0/1471180057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e9827550b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800b2f0 tx=0x7f3ec8000bc0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.018+0000 7f3eec030b80 1 -- 172.21.15.43:0/1471180057 shutdown_connections 2024-05-27T07:22:52.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.018+0000 7f3eec030b80 1 --2- 172.21.15.43:0/1471180057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e982755e70 0x55e98277fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.018+0000 7f3eec030b80 1 --2- 172.21.15.43:0/1471180057 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e9827550b0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.018+0000 7f3eec030b80 1 --2- 172.21.15.43:0/1471180057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9826c29f0 0x55e9827555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.018+0000 7f3eec030b80 1 -- 172.21.15.43:0/1471180057 >> 172.21.15.43:0/1471180057 conn(0x55e982670390 msgr2=0x55e982750e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:52.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.019+0000 7f3eec030b80 1 -- 172.21.15.43:0/1471180057 shutdown_connections 2024-05-27T07:22:52.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.019+0000 7f3eec030b80 1 -- 172.21.15.43:0/1471180057 wait complete. 2024-05-27T07:22:52.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.019+0000 7f3eec030b80 1 -- start start 2024-05-27T07:22:52.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.019+0000 7f3eec030b80 1 Processor -- start 2024-05-27T07:22:52.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.020+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e982774b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.020+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 0x55e98276feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.020+0000 7f3eec030b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e982755e70 0x55e9827703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.020+0000 7f3eec030b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81ad470 con 0x55e9825d6e40 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.020+0000 7f3eec030b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81ae8e0 con 0x55e982755e70 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.020+0000 7f3eec030b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81adb40 con 0x55e9826c29f0 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.020+0000 7f3eec030b80 10 client.0 _ll_register_callbacks cb 0x55e98261dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.021+0000 7f3ed6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e982774b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.021+0000 7f3ed6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e982774b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37856/0 (socket says 172.21.15.43:37856) 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.021+0000 7f3ed6800700 1 -- 172.21.15.43:0/1475654338 learned_addr learned my addr 172.21.15.43:0/1475654338 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.021+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 0x55e98276feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.021+0000 7f3ed7200700 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e982755e70 0x55e9827703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.021+0000 7f3ed6800700 1 -- 172.21.15.43:0/1475654338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 msgr2=0x55e98276feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.021+0000 7f3ed6800700 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 0x55e98276feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.021+0000 7f3ed6800700 1 -- 172.21.15.43:0/1475654338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e982755e70 msgr2=0x55e9827703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:22:52.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.021+0000 7f3ed6800700 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e982755e70 0x55e9827703f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.021+0000 7f3ed6800700 1 -- 172.21.15.43:0/1475654338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e982770ad0 con 0x55e9825d6e40 2024-05-27T07:22:52.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.022+0000 7f3ed6800700 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e982774b50 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80dfe80 tx=0x7f3ec80a9500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfe1b101f4da2a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:22:52.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.022+0000 7f3ec0c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:22:52.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.022+0000 7f3ec0c00700 1 -- 172.21.15.43:0/1475654338 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8020070 con 0x55e9825d6e40 2024-05-27T07:22:52.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.022+0000 7f3ec0c00700 1 -- 172.21.15.43:0/1475654338 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9408+0+0 (secure 0 0 0) 0x7f3ec80a96f0 con 0x55e9825d6e40 2024-05-27T07:22:52.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.022+0000 7f3eec030b80 1 -- 172.21.15.43:0/1475654338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ec80a8fa0 con 0x55e9825d6e40 2024-05-27T07:22:52.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.022+0000 7f3ec0c00700 1 -- 172.21.15.43:0/1475654338 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3ec80b22f0 con 0x55e9825d6e40 2024-05-27T07:22:52.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.022+0000 7f3eec030b80 10 client.16818 fetch_fsmap learned FSMap version 91 2024-05-27T07:22:52.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.022+0000 7f3eec030b80 1 -- 172.21.15.43:0/1475654338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=4+}) -- 0x55e9827e2b10 con 0x55e9825d6e40 2024-05-27T07:22:52.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.023+0000 7f3ec0c00700 1 -- 172.21.15.43:0/1475654338 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3ec80a8fa0 con 0x55e9825d6e40 2024-05-27T07:22:52.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.023+0000 7f3ec0c00700 1 -- 172.21.15.43:0/1475654338 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 91) ==== 5966+0+0 (secure 0 0 0) 0x7f3ec80b0bc0 con 0x55e9825d6e40 2024-05-27T07:22:52.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.023+0000 7f3eec030b80 10 client.16818 fetch_fsmap finished waiting for FSMap version 91 2024-05-27T07:22:52.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.023+0000 7f3eec030b80 10 client.16818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:22:52.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.023+0000 7f3eec030b80 20 client.16818 populate_metadata read hostname 'smithi043' 2024-05-27T07:22:52.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.023+0000 7f3eec030b80 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e9827e3160 0x55e9827e5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:22:52.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.023+0000 7f3eec030b80 4 client.16818 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:22:52.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.023+0000 7f3eec030b80 1 -- 172.21.15.43:0/1475654338 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e98267f570 con 0x55e9827e3160 2024-05-27T07:22:52.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.024+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e9827e3160 0x55e9827e5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:22:52.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.025+0000 7f3ed5e00700 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e9827e3160 0x55e9827e5580 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dac6744d4ae4b7cd server_cookie=611e488411de2e05 in_seq=0 out_seq=0 2024-05-27T07:22:52.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.025+0000 7f3ec0c00700 10 client.16818 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:52.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3ec0c00700 1 -- 172.21.15.43:0/1475654338 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e98267f570 con 0x55e9827e3160 2024-05-27T07:22:52.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3ec0c00700 10 client.16818 handle_command_reply: tid=0 2024-05-27T07:22:52.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3eec030b80 1 client.16818 shutdown 2024-05-27T07:22:52.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3eec030b80 1 -- 172.21.15.43:0/1475654338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 msgr2=0x55e982774b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:52.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3eec030b80 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e982774b50 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80dfe80 tx=0x7f3ec80a9500 comp rx=0 tx=0).stop 2024-05-27T07:22:52.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3eec030b80 1 -- 172.21.15.43:0/1475654338 shutdown_connections 2024-05-27T07:22:52.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3eec030b80 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e982755e70 0x55e9827703f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3eec030b80 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e9827e3160 0x55e9827e5580 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3eec030b80 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9825d6e40 0x55e982774b50 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3eec030b80 1 --2- 172.21.15.43:0/1475654338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9826c29f0 0x55e98276feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3eec030b80 1 -- 172.21.15.43:0/1475654338 >> 172.21.15.43:0/1475654338 conn(0x55e982670390 msgr2=0x55e982750e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:52.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.026+0000 7f3ec0c00700 0 client.16818 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:22:52.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.027+0000 7f3eec030b80 1 -- 172.21.15.43:0/1475654338 shutdown_connections 2024-05-27T07:22:52.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.027+0000 7f3eec030b80 1 -- 172.21.15.43:0/1475654338 wait complete. 2024-05-27T07:22:52.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.027+0000 7f3eec030b80 20 client.16818 trim_cache size 0 max 16384 2024-05-27T07:22:52.018 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:22:52.018 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:22:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:22:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: "ad97df14-d1ea-4167-930a-b80008641902": { 2024-05-27T07:22:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:22:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "ad97df14-d1ea-4167-930a-b80008641902", 2024-05-27T07:22:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:22:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:22:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:22:52.020 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:22:52.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.028+0000 7f3edd400700 1 -- 172.21.15.43:0/4015269144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3ebc0b2a70 msgr2=0x7f3ebc0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.028+0000 7f3edd400700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3ebc0b2a70 0x7f3ebc0b4f80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000b820 tx=0x7f3ed00099e0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.028+0000 7f3edd400700 1 -- 172.21.15.43:0/4015269144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ed81045e0 msgr2=0x7f3ed819cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.028+0000 7f3edd400700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ed81045e0 0x7f3ed819cf30 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800d950 tx=0x7f3ec800dd10 comp rx=0 tx=0).stop 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.028+0000 7f3edd400700 1 -- 172.21.15.43:0/4015269144 shutdown_connections 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.028+0000 7f3edd400700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3ebc0b2a70 0x7f3ebc0b4f80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.028+0000 7f3edd400700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed8109860 0x7f3ed81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.028+0000 7f3edd400700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ed81045e0 0x7f3ed819cf30 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.028+0000 7f3edd400700 1 --2- 172.21.15.43:0/4015269144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ed8108f10 0x7f3ed819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.028+0000 7f3edd400700 1 -- 172.21.15.43:0/4015269144 >> 172.21.15.43:0/4015269144 conn(0x7f3ed8100030 msgr2=0x7f3ed8110ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.029+0000 7f3edd400700 1 -- 172.21.15.43:0/4015269144 shutdown_connections 2024-05-27T07:22:52.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:22:52.029+0000 7f3edd400700 1 -- 172.21.15.43:0/4015269144 wait complete. 2024-05-27T07:22:52.216 INFO:tasks.workunit.client.0.smithi043.stdout: 1 49448 0.63 MB/sec execute 296 sec latency 53.097 ms 2024-05-27T07:22:52.350 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:51 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:22:52.350 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:51 smithi179 systemd[1]: Stopped Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:52.350 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:51 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.565s CPU time. 2024-05-27T07:22:52.350 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:51 smithi179 systemd[1]: Starting Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:22:52.643 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:52 smithi132 ceph-mon[35283]: pgmap v301: 129 pgs: 129 active+clean; 84 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 507 KiB/s wr, 39 op/s 2024-05-27T07:22:52.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:52 smithi043 ceph-mon[31531]: pgmap v301: 129 pgs: 129 active+clean; 84 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 507 KiB/s wr, 39 op/s 2024-05-27T07:22:52.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:52 smithi179 ceph-mon[35888]: pgmap v301: 129 pgs: 129 active+clean; 84 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 507 KiB/s wr, 39 op/s 2024-05-27T07:22:52.708 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:52 smithi179 podman[73786]: 2024-05-27 07:22:52.267297982 +0000 UTC m=+0.037102565 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:22:52.708 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:52 smithi179 podman[73786]: 2024-05-27 07:22:52.399897069 +0000 UTC m=+0.169701670 container create 613a3f6844f649ca06da3740403aa1834e9d15e3f9fc7c347e348b4391b43160 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:22:52.923 INFO:tasks.cephfs.filesystem:scrub status for tag:ad97df14-d1ea-4167-930a-b80008641902 - {'path': '/', 'tag': 'ad97df14-d1ea-4167-930a-b80008641902', 'options': 'recursive,force'} 2024-05-27T07:22:53.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:52 smithi179 podman[73786]: 2024-05-27 07:22:52.769191278 +0000 UTC m=+0.538995889 container init 613a3f6844f649ca06da3740403aa1834e9d15e3f9fc7c347e348b4391b43160 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:22:53.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:52 smithi179 podman[73786]: 2024-05-27 07:22:52.779318705 +0000 UTC m=+0.549123313 container start 613a3f6844f649ca06da3740403aa1834e9d15e3f9fc7c347e348b4391b43160 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-27T07:22:53.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:52 smithi179 ceph-mds[73804]: -- 172.21.15.179:0/948826814 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55599f771a40 con 0x55599f7b9a80 2024-05-27T07:22:53.210 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: starting mds.f at 2024-05-27T07:22:53.210 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:52 smithi179 bash[73786]: 613a3f6844f649ca06da3740403aa1834e9d15e3f9fc7c347e348b4391b43160 2024-05-27T07:22:53.210 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:52 smithi179 systemd[1]: Started Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:53.216 INFO:tasks.workunit.client.0.smithi043.stdout: 1 49473 0.63 MB/sec execute 297 sec latency 59.033 ms 2024-05-27T07:22:53.629 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:53 smithi179 ceph-mon[35888]: daemon mds.f restarted 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb98be00700 time 2024-05-27T07:22:53.524560+0000 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb9a18d5ba6] 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb9a18d5dd1] 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5559977f523d] 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5559977e0971] 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 5: (SnapRealm::check_cache() const+0x13e) [0x5559977defa0] 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5559977df651] 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555997774347] 2024-05-27T07:22:53.630 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55599747087e] 2024-05-27T07:22:53.631 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555997475e94] 2024-05-27T07:22:53.631 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555997480969] 2024-05-27T07:22:53.631 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 11: (MDLog::_replay_thread()+0x1f38) [0x55599781948c] 2024-05-27T07:22:53.631 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5559978229cb] 2024-05-27T07:22:53.631 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 13: (Thread::entry_wrapper()+0x43) [0x7fb9a18a472d] 2024-05-27T07:22:53.631 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb9a18a4749] 2024-05-27T07:22:53.631 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9a04481ca] 2024-05-27T07:22:53.631 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 16: clone() 2024-05-27T07:22:53.631 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.526+0000 7fb98be00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb98be00700 time 2024-05-27T07:22:53.524560+0000 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb9a18d5ba6] 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb9a18d5dd1] 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5559977f523d] 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5559977e0971] 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 5: (SnapRealm::check_cache() const+0x13e) [0x5559977defa0] 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5559977df651] 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555997774347] 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55599747087e] 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555997475e94] 2024-05-27T07:22:53.632 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555997480969] 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 11: (MDLog::_replay_thread()+0x1f38) [0x55599781948c] 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5559978229cb] 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 13: (Thread::entry_wrapper()+0x43) [0x7fb9a18a472d] 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb9a18a4749] 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9a04481ca] 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 16: clone() 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: *** Caught signal (Aborted) ** 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: in thread 7fb98be00700 thread_name:md_log_replay 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb9a0452d20] 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2: gsignal() 2024-05-27T07:22:53.633 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 3: abort() 2024-05-27T07:22:53.634 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb9a18d5ced] 2024-05-27T07:22:53.634 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb9a18d5dd1] 2024-05-27T07:22:53.634 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5559977f523d] 2024-05-27T07:22:53.634 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5559977e0971] 2024-05-27T07:22:53.634 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 8: (SnapRealm::check_cache() const+0x13e) [0x5559977defa0] 2024-05-27T07:22:53.634 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5559977df651] 2024-05-27T07:22:53.634 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555997774347] 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55599747087e] 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555997475e94] 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555997480969] 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 14: (MDLog::_replay_thread()+0x1f38) [0x55599781948c] 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5559978229cb] 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 16: (Thread::entry_wrapper()+0x43) [0x7fb9a18a472d] 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb9a18a4749] 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9a04481ca] 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 19: clone() 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.529+0000 7fb98be00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: in thread 7fb98be00700 thread_name:md_log_replay 2024-05-27T07:22:53.635 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb9a0452d20] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2: gsignal() 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 3: abort() 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb9a18d5ced] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb9a18d5dd1] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5559977f523d] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5559977e0971] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 8: (SnapRealm::check_cache() const+0x13e) [0x5559977defa0] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5559977df651] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555997774347] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55599747087e] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555997475e94] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555997480969] 2024-05-27T07:22:53.636 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 14: (MDLog::_replay_thread()+0x1f38) [0x55599781948c] 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5559978229cb] 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 16: (Thread::entry_wrapper()+0x43) [0x7fb9a18a472d] 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb9a18a4749] 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9a04481ca] 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 19: clone() 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: -1> 2024-05-27T07:22:53.526+0000 7fb98be00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb98be00700 time 2024-05-27T07:22:53.524560+0000 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb9a18d5ba6] 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb9a18d5dd1] 2024-05-27T07:22:53.637 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5559977f523d] 2024-05-27T07:22:53.638 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5559977e0971] 2024-05-27T07:22:53.638 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 5: (SnapRealm::check_cache() const+0x13e) [0x5559977defa0] 2024-05-27T07:22:53.638 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5559977df651] 2024-05-27T07:22:53.638 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555997774347] 2024-05-27T07:22:53.638 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55599747087e] 2024-05-27T07:22:53.638 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555997475e94] 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555997480969] 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 11: (MDLog::_replay_thread()+0x1f38) [0x55599781948c] 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5559978229cb] 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 13: (Thread::entry_wrapper()+0x43) [0x7fb9a18a472d] 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb9a18a4749] 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9a04481ca] 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 16: clone() 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 0> 2024-05-27T07:22:53.529+0000 7fb98be00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: in thread 7fb98be00700 thread_name:md_log_replay 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:53.639 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb9a0452d20] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2: gsignal() 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 3: abort() 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb9a18d5ced] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb9a18d5dd1] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5559977f523d] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5559977e0971] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 8: (SnapRealm::check_cache() const+0x13e) [0x5559977defa0] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5559977df651] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555997774347] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55599747087e] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555997475e94] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555997480969] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 14: (MDLog::_replay_thread()+0x1f38) [0x55599781948c] 2024-05-27T07:22:53.640 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5559978229cb] 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 16: (Thread::entry_wrapper()+0x43) [0x7fb9a18a472d] 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb9a18a4749] 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9a04481ca] 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 19: clone() 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: -2> 2024-05-27T07:22:53.526+0000 7fb98be00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fb98be00700 time 2024-05-27T07:22:53.524560+0000 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb9a18d5ba6] 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb9a18d5dd1] 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5559977f523d] 2024-05-27T07:22:53.641 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 4: (SnapRealm::build_snap_set() const+0x323) [0x5559977e0971] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 5: (SnapRealm::check_cache() const+0x13e) [0x5559977defa0] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5559977df651] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555997774347] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55599747087e] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555997475e94] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x555997480969] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 11: (MDLog::_replay_thread()+0x1f38) [0x55599781948c] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 12: (MDLog::ReplayThread::entry()+0x11) [0x5559978229cb] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 13: (Thread::entry_wrapper()+0x43) [0x7fb9a18a472d] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 14: (Thread::_entry_func(void*)+0xd) [0x7fb9a18a4749] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fb9a04481ca] 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 16: clone() 2024-05-27T07:22:53.642 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: -1> 2024-05-27T07:22:53.529+0000 7fb98be00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: in thread 7fb98be00700 thread_name:md_log_replay 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb9a0452d20] 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2: gsignal() 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 3: abort() 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb9a18d5ced] 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb9a18d5dd1] 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5559977f523d] 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 7: (SnapRealm::build_snap_set() const+0x323) [0x5559977e0971] 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 8: (SnapRealm::check_cache() const+0x13e) [0x5559977defa0] 2024-05-27T07:22:53.643 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x5559977df651] 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x555997774347] 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55599747087e] 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x555997475e94] 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x555997480969] 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 14: (MDLog::_replay_thread()+0x1f38) [0x55599781948c] 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 15: (MDLog::ReplayThread::entry()+0x11) [0x5559978229cb] 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 16: (Thread::entry_wrapper()+0x43) [0x7fb9a18a472d] 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 17: (Thread::_entry_func(void*)+0xd) [0x7fb9a18a4749] 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb9a04481ca] 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 19: clone() 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[73800]: 2024-05-27T07:22:53.644 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:53 smithi179 podman[73835]: 2024-05-27 07:22:53.638678744 +0000 UTC m=+0.036921747 container died 613a3f6844f649ca06da3740403aa1834e9d15e3f9fc7c347e348b4391b43160 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-27T07:22:53.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:53 smithi043 ceph-mon[31531]: daemon mds.f restarted 2024-05-27T07:22:53.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:53 smithi132 ceph-mon[35283]: daemon mds.f restarted 2024-05-27T07:22:54.217 INFO:tasks.workunit.client.0.smithi043.stdout: 1 49536 0.63 MB/sec execute 298 sec latency 56.441 ms 2024-05-27T07:22:54.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:54 smithi043 ceph-mon[31531]: pgmap v302: 129 pgs: 129 active+clean; 84 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 507 KiB/s wr, 39 op/s 2024-05-27T07:22:54.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:54 smithi043 ceph-mon[31531]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:22:54.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:54 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/1598053276,v1:172.21.15.179:6833/1598053276] up:boot 2024-05-27T07:22:54.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:54 smithi043 ceph-mon[31531]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:54.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:54.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:54 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:54.708 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:54 smithi179 podman[73835]: 2024-05-27 07:22:54.476080753 +0000 UTC m=+0.874323745 container remove 613a3f6844f649ca06da3740403aa1834e9d15e3f9fc7c347e348b4391b43160 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T07:22:54.708 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:54 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:22:54.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:54 smithi179 ceph-mon[35888]: pgmap v302: 129 pgs: 129 active+clean; 84 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 507 KiB/s wr, 39 op/s 2024-05-27T07:22:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:54 smithi179 ceph-mon[35888]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:22:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:54 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/1598053276,v1:172.21.15.179:6833/1598053276] up:boot 2024-05-27T07:22:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:54 smithi179 ceph-mon[35888]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:54 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:54.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:54 smithi132 ceph-mon[35283]: pgmap v302: 129 pgs: 129 active+clean; 84 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 507 KiB/s wr, 39 op/s 2024-05-27T07:22:54.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:54 smithi132 ceph-mon[35283]: osdmap e122: 12 total, 12 up, 12 in 2024-05-27T07:22:54.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:54 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/1598053276,v1:172.21.15.179:6833/1598053276] up:boot 2024-05-27T07:22:54.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:54 smithi132 ceph-mon[35283]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:54.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:22:54.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:54 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:55.208 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:54 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:22:55.208 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:22:54 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.620s CPU time. 2024-05-27T07:22:55.217 INFO:tasks.workunit.client.0.smithi043.stdout: 1 49718 0.63 MB/sec execute 299 sec latency 58.494 ms 2024-05-27T07:22:55.851 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:55 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:22:55.851 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:55 smithi179 systemd[1]: Stopped Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:55.851 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:55 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.445s CPU time. 2024-05-27T07:22:55.851 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:55 smithi179 systemd[1]: Starting Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:22:56.196 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:55 smithi179 podman[73925]: 2024-05-27 07:22:55.765170269 +0000 UTC m=+0.033301918 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:22:56.196 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:55 smithi179 podman[73925]: 2024-05-27 07:22:55.910568867 +0000 UTC m=+0.178700557 container create 8e8d82c265097526994c281c720ab5385d9068d2dd7e57b2362d3a484c27c26e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:22:56.196 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 podman[73925]: 2024-05-27 07:22:56.205965962 +0000 UTC m=+0.474097703 container init 8e8d82c265097526994c281c720ab5385d9068d2dd7e57b2362d3a484c27c26e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:22:56.217 INFO:tasks.workunit.client.0.smithi043.stdout: 1 49829 0.63 MB/sec execute 300 sec latency 50.866 ms 2024-05-27T07:22:56.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:56 smithi179 ceph-mon[35888]: pgmap v304: 129 pgs: 129 active+clean; 84 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 507 KiB/s wr, 39 op/s 2024-05-27T07:22:56.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:56 smithi179 ceph-mon[35888]: daemon mds.l restarted 2024-05-27T07:22:56.458 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 podman[73925]: 2024-05-27 07:22:56.215226491 +0000 UTC m=+0.483358184 container start 8e8d82c265097526994c281c720ab5385d9068d2dd7e57b2362d3a484c27c26e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0) 2024-05-27T07:22:56.458 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-mds[73944]: -- 172.21.15.179:0/2742722476 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ccb6da9a40 con 0x55ccb6df1a80 2024-05-27T07:22:56.458 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: starting mds.l at 2024-05-27T07:22:56.458 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 bash[73925]: 8e8d82c265097526994c281c720ab5385d9068d2dd7e57b2362d3a484c27c26e 2024-05-27T07:22:56.459 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 systemd[1]: Started Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:22:56.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:56 smithi043 ceph-mon[31531]: pgmap v304: 129 pgs: 129 active+clean; 84 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 507 KiB/s wr, 39 op/s 2024-05-27T07:22:56.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:56 smithi043 ceph-mon[31531]: daemon mds.l restarted 2024-05-27T07:22:56.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:56 smithi132 ceph-mon[35283]: pgmap v304: 129 pgs: 129 active+clean; 84 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 507 KiB/s wr, 39 op/s 2024-05-27T07:22:56.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:56 smithi132 ceph-mon[35283]: daemon mds.l restarted 2024-05-27T07:22:57.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f926fc00700 time 2024-05-27T07:22:56.914194+0000 2024-05-27T07:22:57.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:57.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f928562dba6] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f928562ddd1] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cc7924623d] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cc79231971] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cc7922ffa0] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cc79230651] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cc791c5347] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cc78ec187e] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cc78ec6e94] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cc78ed1969] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cc7926a48c] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cc792739cb] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 13: (Thread::entry_wrapper()+0x43) [0x7f92855fc72d] 2024-05-27T07:22:57.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 14: (Thread::_entry_func(void*)+0xd) [0x7f92855fc749] 2024-05-27T07:22:57.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92841a01ca] 2024-05-27T07:22:57.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 16: clone() 2024-05-27T07:22:57.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: *** Caught signal (Aborted) ** 2024-05-27T07:22:57.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: in thread 7f926fc00700 thread_name:md_log_replay 2024-05-27T07:22:57.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:56.915+0000 7f926fc00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f926fc00700 time 2024-05-27T07:22:56.914194+0000 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f928562dba6] 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f928562ddd1] 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cc7924623d] 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cc79231971] 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cc7922ffa0] 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cc79230651] 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cc791c5347] 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cc78ec187e] 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cc78ec6e94] 2024-05-27T07:22:57.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cc78ed1969] 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cc7926a48c] 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cc792739cb] 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 13: (Thread::entry_wrapper()+0x43) [0x7f92855fc72d] 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 14: (Thread::_entry_func(void*)+0xd) [0x7f92855fc749] 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92841a01ca] 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 16: clone() 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f92841aad20] 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2: gsignal() 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 3: abort() 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f928562dced] 2024-05-27T07:22:57.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f928562ddd1] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cc7924623d] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cc79231971] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cc7922ffa0] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cc79230651] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cc791c5347] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cc78ec187e] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cc78ec6e94] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cc78ed1969] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cc7926a48c] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cc792739cb] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 16: (Thread::entry_wrapper()+0x43) [0x7f92855fc72d] 2024-05-27T07:22:57.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 17: (Thread::_entry_func(void*)+0xd) [0x7f92855fc749] 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92841a01ca] 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 19: clone() 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:56.918+0000 7f926fc00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: in thread 7f926fc00700 thread_name:md_log_replay 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f92841aad20] 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2: gsignal() 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 3: abort() 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f928562dced] 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f928562ddd1] 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cc7924623d] 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cc79231971] 2024-05-27T07:22:57.216 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cc7922ffa0] 2024-05-27T07:22:57.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cc79230651] 2024-05-27T07:22:57.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cc791c5347] 2024-05-27T07:22:57.217 INFO:tasks.workunit.client.0.smithi043.stdout: 1 49928 0.63 MB/sec execute 301 sec latency 53.527 ms 2024-05-27T07:22:57.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cc78ec187e] 2024-05-27T07:22:57.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cc78ec6e94] 2024-05-27T07:22:57.217 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cc78ed1969] 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cc7926a48c] 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cc792739cb] 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 16: (Thread::entry_wrapper()+0x43) [0x7f92855fc72d] 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 17: (Thread::_entry_func(void*)+0xd) [0x7f92855fc749] 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92841a01ca] 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 19: clone() 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: -1> 2024-05-27T07:22:56.915+0000 7f926fc00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f926fc00700 time 2024-05-27T07:22:56.914194+0000 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f928562dba6] 2024-05-27T07:22:57.218 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f928562ddd1] 2024-05-27T07:22:57.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cc7924623d] 2024-05-27T07:22:57.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cc79231971] 2024-05-27T07:22:57.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cc7922ffa0] 2024-05-27T07:22:57.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cc79230651] 2024-05-27T07:22:57.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cc791c5347] 2024-05-27T07:22:57.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cc78ec187e] 2024-05-27T07:22:57.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cc78ec6e94] 2024-05-27T07:22:57.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cc78ed1969] 2024-05-27T07:22:57.219 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cc7926a48c] 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cc792739cb] 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 13: (Thread::entry_wrapper()+0x43) [0x7f92855fc72d] 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 14: (Thread::_entry_func(void*)+0xd) [0x7f92855fc749] 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92841a01ca] 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 16: clone() 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 0> 2024-05-27T07:22:56.918+0000 7f926fc00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: in thread 7f926fc00700 thread_name:md_log_replay 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f92841aad20] 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2: gsignal() 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 3: abort() 2024-05-27T07:22:57.220 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f928562dced] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f928562ddd1] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cc7924623d] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cc79231971] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cc7922ffa0] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cc79230651] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cc791c5347] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cc78ec187e] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cc78ec6e94] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cc78ed1969] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cc7926a48c] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cc792739cb] 2024-05-27T07:22:57.221 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 16: (Thread::entry_wrapper()+0x43) [0x7f92855fc72d] 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 17: (Thread::_entry_func(void*)+0xd) [0x7f92855fc749] 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92841a01ca] 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 19: clone() 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: -1> 2024-05-27T07:22:56.915+0000 7f926fc00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f926fc00700 time 2024-05-27T07:22:56.914194+0000 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f928562dba6] 2024-05-27T07:22:57.222 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f928562ddd1] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cc7924623d] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55cc79231971] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 5: (SnapRealm::check_cache() const+0x13e) [0x55cc7922ffa0] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cc79230651] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cc791c5347] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cc78ec187e] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cc78ec6e94] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55cc78ed1969] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 11: (MDLog::_replay_thread()+0x1f38) [0x55cc7926a48c] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55cc792739cb] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 13: (Thread::entry_wrapper()+0x43) [0x7f92855fc72d] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 14: (Thread::_entry_func(void*)+0xd) [0x7f92855fc749] 2024-05-27T07:22:57.223 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f92841a01ca] 2024-05-27T07:22:57.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 16: clone() 2024-05-27T07:22:57.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 0> 2024-05-27T07:22:56.918+0000 7f926fc00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:22:57.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: in thread 7f926fc00700 thread_name:md_log_replay 2024-05-27T07:22:57.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:22:57.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f92841aad20] 2024-05-27T07:22:57.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2: gsignal() 2024-05-27T07:22:57.224 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 3: abort() 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f928562dced] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f928562ddd1] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55cc7924623d] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55cc79231971] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 8: (SnapRealm::check_cache() const+0x13e) [0x55cc7922ffa0] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55cc79230651] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55cc791c5347] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55cc78ec187e] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55cc78ec6e94] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55cc78ed1969] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 14: (MDLog::_replay_thread()+0x1f38) [0x55cc7926a48c] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55cc792739cb] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 16: (Thread::entry_wrapper()+0x43) [0x7f92855fc72d] 2024-05-27T07:22:57.225 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 17: (Thread::_entry_func(void*)+0xd) [0x7f92855fc749] 2024-05-27T07:22:57.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f92841a01ca] 2024-05-27T07:22:57.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 19: clone() 2024-05-27T07:22:57.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:22:57.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[73940]: 2024-05-27T07:22:57.226 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:57 smithi179 podman[73975]: 2024-05-27 07:22:57.031925773 +0000 UTC m=+0.033203052 container died 8e8d82c265097526994c281c720ab5385d9068d2dd7e57b2362d3a484c27c26e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:22:57.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:57 smithi043 ceph-mon[31531]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:22:57.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:57 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/3464934955,v1:172.21.15.179:6833/3464934955] up:boot 2024-05-27T07:22:57.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:57 smithi043 ceph-mon[31531]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:57.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:57 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:57.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:57 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:57.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:57 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:22:57.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:57 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:22:57.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:57 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:22:57.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:57 smithi179 ceph-mon[35888]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:22:57.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:57 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/3464934955,v1:172.21.15.179:6833/3464934955] up:boot 2024-05-27T07:22:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:57 smithi179 ceph-mon[35888]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:57 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:57 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:57 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:22:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:57 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:22:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:57 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:22:57.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:57 smithi132 ceph-mon[35283]: osdmap e123: 12 total, 12 up, 12 in 2024-05-27T07:22:57.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:57 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/3464934955,v1:172.21.15.179:6833/3464934955] up:boot 2024-05-27T07:22:57.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:57 smithi132 ceph-mon[35283]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:22:57.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:57 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:22:57.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:57 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:22:57.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:57 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:22:57.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:57 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:22:57.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:57 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:22:58.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:57 smithi179 podman[73975]: 2024-05-27 07:22:57.838794021 +0000 UTC m=+0.840071359 container remove 8e8d82c265097526994c281c720ab5385d9068d2dd7e57b2362d3a484c27c26e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T07:22:58.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:57 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:22:58.217 INFO:tasks.workunit.client.0.smithi043.stdout: 1 49983 0.63 MB/sec execute 302 sec latency 47.637 ms 2024-05-27T07:22:58.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:58 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:22:58.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:22:58 smithi043 ceph-mon[31531]: pgmap v306: 129 pgs: 129 active+clean; 85 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 787 KiB/s wr, 50 op/s 2024-05-27T07:22:58.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:58 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:22:58.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:22:58 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.336s CPU time. 2024-05-27T07:22:58.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:58 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:22:58.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:22:58 smithi179 ceph-mon[35888]: pgmap v306: 129 pgs: 129 active+clean; 85 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 787 KiB/s wr, 50 op/s 2024-05-27T07:22:58.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:58 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:22:58.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:22:58 smithi132 ceph-mon[35283]: pgmap v306: 129 pgs: 129 active+clean; 85 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 787 KiB/s wr, 50 op/s 2024-05-27T07:22:59.217 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50050 0.63 MB/sec execute 303 sec latency 52.750 ms 2024-05-27T07:23:00.217 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50098 0.63 MB/sec execute 304 sec latency 56.938 ms 2024-05-27T07:23:00.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:00 smithi043 ceph-mon[31531]: pgmap v307: 129 pgs: 129 active+clean; 85 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 257 KiB/s wr, 22 op/s 2024-05-27T07:23:00.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:00 smithi179 ceph-mon[35888]: pgmap v307: 129 pgs: 129 active+clean; 85 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 257 KiB/s wr, 22 op/s 2024-05-27T07:23:00.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:00 smithi132 ceph-mon[35283]: pgmap v307: 129 pgs: 129 active+clean; 85 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 257 KiB/s wr, 22 op/s 2024-05-27T07:23:01.217 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50143 0.63 MB/sec execute 305 sec latency 208.537 ms 2024-05-27T07:23:02.218 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50143 0.62 MB/sec execute 306 sec latency 1208.703 ms 2024-05-27T07:23:02.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:02 smithi043 ceph-mon[31531]: pgmap v308: 129 pgs: 129 active+clean; 86 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 572 KiB/s wr, 50 op/s 2024-05-27T07:23:02.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:02 smithi179 ceph-mon[35888]: pgmap v308: 129 pgs: 129 active+clean; 86 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 572 KiB/s wr, 50 op/s 2024-05-27T07:23:02.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:02 smithi132 ceph-mon[35283]: pgmap v308: 129 pgs: 129 active+clean; 86 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 572 KiB/s wr, 50 op/s 2024-05-27T07:23:03.218 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50143 0.62 MB/sec execute 307 sec latency 2208.857 ms 2024-05-27T07:23:04.218 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50186 0.62 MB/sec execute 308 sec latency 2466.112 ms 2024-05-27T07:23:04.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:04 smithi043 ceph-mon[31531]: pgmap v309: 129 pgs: 129 active+clean; 86 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 489 KiB/s wr, 43 op/s 2024-05-27T07:23:04.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:04 smithi179 ceph-mon[35888]: pgmap v309: 129 pgs: 129 active+clean; 86 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 489 KiB/s wr, 43 op/s 2024-05-27T07:23:04.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:04 smithi132 ceph-mon[35283]: pgmap v309: 129 pgs: 129 active+clean; 86 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 489 KiB/s wr, 43 op/s 2024-05-27T07:23:05.218 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50260 0.62 MB/sec execute 309 sec latency 51.979 ms 2024-05-27T07:23:05.350 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:04 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:23:05.351 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:04 smithi179 systemd[1]: Stopped Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:23:05.351 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:04 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.620s CPU time. 2024-05-27T07:23:05.351 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:04 smithi179 systemd[1]: Starting Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:23:05.698 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:05 smithi179 podman[74065]: 2024-05-27 07:23:05.264359286 +0000 UTC m=+0.032282422 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:23:05.698 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:05 smithi179 podman[74065]: 2024-05-27 07:23:05.403909545 +0000 UTC m=+0.171832728 container create da3455954f5d5010ccc8492b67c8ef879690f4d9b301b9b1570679bf0b680b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T07:23:05.698 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:05 smithi179 podman[74065]: 2024-05-27 07:23:05.708515896 +0000 UTC m=+0.476439129 container init da3455954f5d5010ccc8492b67c8ef879690f4d9b301b9b1570679bf0b680b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:23:05.958 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:05 smithi179 podman[74065]: 2024-05-27 07:23:05.719863454 +0000 UTC m=+0.487786614 container start da3455954f5d5010ccc8492b67c8ef879690f4d9b301b9b1570679bf0b680b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-27T07:23:05.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:05 smithi179 bash[74065]: da3455954f5d5010ccc8492b67c8ef879690f4d9b301b9b1570679bf0b680b94 2024-05-27T07:23:05.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:05 smithi179 systemd[1]: Started Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:23:05.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:05 smithi179 ceph-mds[74083]: -- 172.21.15.179:0/71072451 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a9a0345a40 con 0x55a9a038d600 2024-05-27T07:23:05.959 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: starting mds.f at 2024-05-27T07:23:06.218 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50345 0.62 MB/sec execute 310 sec latency 62.499 ms 2024-05-27T07:23:06.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:06 smithi043 ceph-mon[31531]: pgmap v310: 129 pgs: 129 active+clean; 86 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 457 KiB/s wr, 40 op/s 2024-05-27T07:23:06.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:06 smithi043 ceph-mon[31531]: daemon mds.f restarted 2024-05-27T07:23:06.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:06 smithi043 ceph-mon[31531]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:23:06.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:06 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/3658053236,v1:172.21.15.179:6833/3658053236] up:boot 2024-05-27T07:23:06.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:06 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:06.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:23:06.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:06 smithi179 ceph-mon[35888]: pgmap v310: 129 pgs: 129 active+clean; 86 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 457 KiB/s wr, 40 op/s 2024-05-27T07:23:06.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:06 smithi179 ceph-mon[35888]: daemon mds.f restarted 2024-05-27T07:23:06.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:06 smithi179 ceph-mon[35888]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:23:06.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:06 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/3658053236,v1:172.21.15.179:6833/3658053236] up:boot 2024-05-27T07:23:06.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:06 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:06.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:23:06.709 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f994d200700 time 2024-05-27T07:23:06.308317+0000 2024-05-27T07:23:06.709 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:23:06.709 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:06.709 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9962c2aba6] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9962c2add1] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a983fc523d] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a983fb0971] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a983faefa0] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a983faf651] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a983f44347] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a983c4087e] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a983c45e94] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a983c50969] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a983fe948c] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a983ff29cb] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 13: (Thread::entry_wrapper()+0x43) [0x7f9962bf972d] 2024-05-27T07:23:06.710 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9962bf9749] 2024-05-27T07:23:06.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f996179d1ca] 2024-05-27T07:23:06.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 16: clone() 2024-05-27T07:23:06.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: *** Caught signal (Aborted) ** 2024-05-27T07:23:06.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: in thread 7f994d200700 thread_name:md_log_replay 2024-05-27T07:23:06.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.310+0000 7f994d200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f994d200700 time 2024-05-27T07:23:06.308317+0000 2024-05-27T07:23:06.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:23:06.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:06.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9962c2aba6] 2024-05-27T07:23:06.711 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9962c2add1] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a983fc523d] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a983fb0971] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a983faefa0] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a983faf651] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a983f44347] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a983c4087e] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a983c45e94] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a983c50969] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a983fe948c] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a983ff29cb] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 13: (Thread::entry_wrapper()+0x43) [0x7f9962bf972d] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9962bf9749] 2024-05-27T07:23:06.712 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f996179d1ca] 2024-05-27T07:23:06.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 16: clone() 2024-05-27T07:23:06.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:06.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f99617a7d20] 2024-05-27T07:23:06.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2: gsignal() 2024-05-27T07:23:06.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 3: abort() 2024-05-27T07:23:06.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9962c2aced] 2024-05-27T07:23:06.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9962c2add1] 2024-05-27T07:23:06.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a983fc523d] 2024-05-27T07:23:06.713 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a983fb0971] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a983faefa0] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a983faf651] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a983f44347] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a983c4087e] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a983c45e94] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a983c50969] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a983fe948c] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a983ff29cb] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 16: (Thread::entry_wrapper()+0x43) [0x7f9962bf972d] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9962bf9749] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f996179d1ca] 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 19: clone() 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.314+0000 7f994d200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:23:06.714 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: in thread 7f994d200700 thread_name:md_log_replay 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f99617a7d20] 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2: gsignal() 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 3: abort() 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9962c2aced] 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9962c2add1] 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a983fc523d] 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a983fb0971] 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a983faefa0] 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a983faf651] 2024-05-27T07:23:06.715 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a983f44347] 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a983c4087e] 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a983c45e94] 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a983c50969] 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a983fe948c] 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a983ff29cb] 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 16: (Thread::entry_wrapper()+0x43) [0x7f9962bf972d] 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9962bf9749] 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f996179d1ca] 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 19: clone() 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:23:06.716 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: -1> 2024-05-27T07:23:06.310+0000 7f994d200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f994d200700 time 2024-05-27T07:23:06.308317+0000 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9962c2aba6] 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9962c2add1] 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a983fc523d] 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a983fb0971] 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a983faefa0] 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a983faf651] 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a983f44347] 2024-05-27T07:23:06.717 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a983c4087e] 2024-05-27T07:23:06.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a983c45e94] 2024-05-27T07:23:06.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a983c50969] 2024-05-27T07:23:06.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a983fe948c] 2024-05-27T07:23:06.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a983ff29cb] 2024-05-27T07:23:06.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 13: (Thread::entry_wrapper()+0x43) [0x7f9962bf972d] 2024-05-27T07:23:06.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9962bf9749] 2024-05-27T07:23:06.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f996179d1ca] 2024-05-27T07:23:06.718 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 16: clone() 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 0> 2024-05-27T07:23:06.314+0000 7f994d200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: in thread 7f994d200700 thread_name:md_log_replay 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f99617a7d20] 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2: gsignal() 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 3: abort() 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9962c2aced] 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9962c2add1] 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a983fc523d] 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a983fb0971] 2024-05-27T07:23:06.719 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a983faefa0] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a983faf651] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a983f44347] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a983c4087e] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a983c45e94] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a983c50969] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a983fe948c] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a983ff29cb] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 16: (Thread::entry_wrapper()+0x43) [0x7f9962bf972d] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9962bf9749] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f996179d1ca] 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 19: clone() 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:23:06.720 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: -1> 2024-05-27T07:23:06.310+0000 7f994d200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f994d200700 time 2024-05-27T07:23:06.308317+0000 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9962c2aba6] 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9962c2add1] 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a983fc523d] 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a983fb0971] 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a983faefa0] 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a983faf651] 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a983f44347] 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a983c4087e] 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a983c45e94] 2024-05-27T07:23:06.721 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a983c50969] 2024-05-27T07:23:06.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a983fe948c] 2024-05-27T07:23:06.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a983ff29cb] 2024-05-27T07:23:06.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 13: (Thread::entry_wrapper()+0x43) [0x7f9962bf972d] 2024-05-27T07:23:06.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9962bf9749] 2024-05-27T07:23:06.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f996179d1ca] 2024-05-27T07:23:06.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 16: clone() 2024-05-27T07:23:06.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 0> 2024-05-27T07:23:06.314+0000 7f994d200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:23:06.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: in thread 7f994d200700 thread_name:md_log_replay 2024-05-27T07:23:06.722 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f99617a7d20] 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2: gsignal() 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 3: abort() 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9962c2aced] 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9962c2add1] 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a983fc523d] 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a983fb0971] 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a983faefa0] 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a983faf651] 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a983f44347] 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a983c4087e] 2024-05-27T07:23:06.723 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a983c45e94] 2024-05-27T07:23:06.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a983c50969] 2024-05-27T07:23:06.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a983fe948c] 2024-05-27T07:23:06.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a983ff29cb] 2024-05-27T07:23:06.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 16: (Thread::entry_wrapper()+0x43) [0x7f9962bf972d] 2024-05-27T07:23:06.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9962bf9749] 2024-05-27T07:23:06.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f996179d1ca] 2024-05-27T07:23:06.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 19: clone() 2024-05-27T07:23:06.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:23:06.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f[74079]: 2024-05-27T07:23:06.724 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:06 smithi179 podman[74114]: 2024-05-27 07:23:06.458418758 +0000 UTC m=+0.036323031 container died da3455954f5d5010ccc8492b67c8ef879690f4d9b301b9b1570679bf0b680b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:23:06.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:06 smithi132 ceph-mon[35283]: pgmap v310: 129 pgs: 129 active+clean; 86 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 457 KiB/s wr, 40 op/s 2024-05-27T07:23:06.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:06 smithi132 ceph-mon[35283]: daemon mds.f restarted 2024-05-27T07:23:06.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:06 smithi132 ceph-mon[35283]: osdmap e124: 12 total, 12 up, 12 in 2024-05-27T07:23:06.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:06 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/3658053236,v1:172.21.15.179:6833/3658053236] up:boot 2024-05-27T07:23:06.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:06 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:06.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-27T07:23:07.219 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50447 0.62 MB/sec execute 311 sec latency 66.545 ms 2024-05-27T07:23:07.679 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:07 smithi179 podman[74114]: 2024-05-27 07:23:07.254795423 +0000 UTC m=+0.832699696 container remove da3455954f5d5010ccc8492b67c8ef879690f4d9b301b9b1570679bf0b680b94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T07:23:07.679 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:07 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:23:07.958 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:07 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:23:07.958 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:07 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.566s CPU time. 2024-05-27T07:23:08.219 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50593 0.62 MB/sec execute 312 sec latency 57.674 ms 2024-05-27T07:23:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:08 smithi043 ceph-mon[31531]: pgmap v312: 129 pgs: 129 active+clean; 87 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 384 KiB/s wr, 53 op/s 2024-05-27T07:23:08.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:08 smithi043 ceph-mon[31531]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:23:08.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:08 smithi179 ceph-mon[35888]: pgmap v312: 129 pgs: 129 active+clean; 87 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 384 KiB/s wr, 53 op/s 2024-05-27T07:23:08.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:08 smithi179 ceph-mon[35888]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:23:08.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:08 smithi132 ceph-mon[35283]: pgmap v312: 129 pgs: 129 active+clean; 87 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 384 KiB/s wr, 53 op/s 2024-05-27T07:23:08.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:08 smithi132 ceph-mon[35283]: osdmap e125: 12 total, 12 up, 12 in 2024-05-27T07:23:08.844 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:08 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:23:08.844 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:08 smithi179 systemd[1]: Stopped Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:23:08.844 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:08 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.336s CPU time. 2024-05-27T07:23:08.845 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:08 smithi179 systemd[1]: Starting Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:23:09.195 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:08 smithi179 podman[74206]: 2024-05-27 07:23:08.759227749 +0000 UTC m=+0.033023555 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:23:09.195 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:09 smithi179 podman[74206]: 2024-05-27 07:23:09.023014643 +0000 UTC m=+0.296810535 container create b6b3f8348382dd96aaf9b44047c87f5e4bb2513c391e063323a84f6d13223be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-27T07:23:09.219 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50691 0.61 MB/sec execute 313 sec latency 51.460 ms 2024-05-27T07:23:09.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:09 smithi179 podman[74206]: 2024-05-27 07:23:09.318961671 +0000 UTC m=+0.592757536 container init b6b3f8348382dd96aaf9b44047c87f5e4bb2513c391e063323a84f6d13223be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:23:09.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:09 smithi179 podman[74206]: 2024-05-27 07:23:09.329496456 +0000 UTC m=+0.603292328 container start b6b3f8348382dd96aaf9b44047c87f5e4bb2513c391e063323a84f6d13223be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:23:09.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:09 smithi179 ceph-mds[74225]: -- 172.21.15.179:0/427108479 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dd6facba40 con 0x55dd6fb13600 2024-05-27T07:23:09.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: starting mds.l at 2024-05-27T07:23:09.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:09 smithi179 bash[74206]: b6b3f8348382dd96aaf9b44047c87f5e4bb2513c391e063323a84f6d13223be5 2024-05-27T07:23:09.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:09 smithi179 systemd[1]: Started Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:23:10.219 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50797 0.62 MB/sec execute 314 sec latency 55.876 ms 2024-05-27T07:23:10.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:10 smithi043 ceph-mon[31531]: pgmap v314: 129 pgs: 129 active+clean; 87 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 166 KiB/s wr, 39 op/s 2024-05-27T07:23:10.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:10 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:10.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:10 smithi043 ceph-mon[31531]: daemon mds.l restarted 2024-05-27T07:23:10.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:10 smithi179 ceph-mon[35888]: pgmap v314: 129 pgs: 129 active+clean; 87 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 166 KiB/s wr, 39 op/s 2024-05-27T07:23:10.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:10 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:10.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:10 smithi179 ceph-mon[35888]: daemon mds.l restarted 2024-05-27T07:23:10.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:10 smithi132 ceph-mon[35283]: pgmap v314: 129 pgs: 129 active+clean; 87 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 166 KiB/s wr, 39 op/s 2024-05-27T07:23:10.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:10 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:10.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:10 smithi132 ceph-mon[35283]: daemon mds.l restarted 2024-05-27T07:23:11.199 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3c2d200700 time 2024-05-27T07:23:10.755732+0000 2024-05-27T07:23:11.199 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3c42c67ba6] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3c42c67dd1] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dd55cc323d] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55dd55cae971] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 5: (SnapRealm::check_cache() const+0x13e) [0x55dd55cacfa0] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dd55cad651] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dd55c42347] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dd5593e87e] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dd55943e94] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55dd5594e969] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 11: (MDLog::_replay_thread()+0x1f38) [0x55dd55ce748c] 2024-05-27T07:23:11.200 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55dd55cf09cb] 2024-05-27T07:23:11.201 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 13: (Thread::entry_wrapper()+0x43) [0x7f3c42c3672d] 2024-05-27T07:23:11.201 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3c42c36749] 2024-05-27T07:23:11.201 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3c417da1ca] 2024-05-27T07:23:11.201 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 16: clone() 2024-05-27T07:23:11.201 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:10.756+0000 7f3c2d200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3c2d200700 time 2024-05-27T07:23:10.755732+0000 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3c42c67ba6] 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3c42c67dd1] 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dd55cc323d] 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55dd55cae971] 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 5: (SnapRealm::check_cache() const+0x13e) [0x55dd55cacfa0] 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dd55cad651] 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dd55c42347] 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dd5593e87e] 2024-05-27T07:23:11.202 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dd55943e94] 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55dd5594e969] 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 11: (MDLog::_replay_thread()+0x1f38) [0x55dd55ce748c] 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55dd55cf09cb] 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 13: (Thread::entry_wrapper()+0x43) [0x7f3c42c3672d] 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3c42c36749] 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3c417da1ca] 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 16: clone() 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: *** Caught signal (Aborted) ** 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: in thread 7f3c2d200700 thread_name:md_log_replay 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3c417e4d20] 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2: gsignal() 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 3: abort() 2024-05-27T07:23:11.203 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3c42c67ced] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3c42c67dd1] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dd55cc323d] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55dd55cae971] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 8: (SnapRealm::check_cache() const+0x13e) [0x55dd55cacfa0] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dd55cad651] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dd55c42347] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dd5593e87e] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dd55943e94] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55dd5594e969] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 14: (MDLog::_replay_thread()+0x1f38) [0x55dd55ce748c] 2024-05-27T07:23:11.204 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55dd55cf09cb] 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 16: (Thread::entry_wrapper()+0x43) [0x7f3c42c3672d] 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3c42c36749] 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3c417da1ca] 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 19: clone() 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:10.760+0000 7f3c2d200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: in thread 7f3c2d200700 thread_name:md_log_replay 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3c417e4d20] 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2: gsignal() 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 3: abort() 2024-05-27T07:23:11.205 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3c42c67ced] 2024-05-27T07:23:11.206 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3c42c67dd1] 2024-05-27T07:23:11.206 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dd55cc323d] 2024-05-27T07:23:11.206 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55dd55cae971] 2024-05-27T07:23:11.206 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 8: (SnapRealm::check_cache() const+0x13e) [0x55dd55cacfa0] 2024-05-27T07:23:11.206 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dd55cad651] 2024-05-27T07:23:11.206 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dd55c42347] 2024-05-27T07:23:11.206 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dd5593e87e] 2024-05-27T07:23:11.206 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dd55943e94] 2024-05-27T07:23:11.206 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55dd5594e969] 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 14: (MDLog::_replay_thread()+0x1f38) [0x55dd55ce748c] 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55dd55cf09cb] 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 16: (Thread::entry_wrapper()+0x43) [0x7f3c42c3672d] 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3c42c36749] 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3c417da1ca] 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 19: clone() 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: -1> 2024-05-27T07:23:10.756+0000 7f3c2d200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3c2d200700 time 2024-05-27T07:23:10.755732+0000 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:11.207 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3c42c67ba6] 2024-05-27T07:23:11.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3c42c67dd1] 2024-05-27T07:23:11.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dd55cc323d] 2024-05-27T07:23:11.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55dd55cae971] 2024-05-27T07:23:11.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 5: (SnapRealm::check_cache() const+0x13e) [0x55dd55cacfa0] 2024-05-27T07:23:11.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dd55cad651] 2024-05-27T07:23:11.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dd55c42347] 2024-05-27T07:23:11.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dd5593e87e] 2024-05-27T07:23:11.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dd55943e94] 2024-05-27T07:23:11.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55dd5594e969] 2024-05-27T07:23:11.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 11: (MDLog::_replay_thread()+0x1f38) [0x55dd55ce748c] 2024-05-27T07:23:11.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55dd55cf09cb] 2024-05-27T07:23:11.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 13: (Thread::entry_wrapper()+0x43) [0x7f3c42c3672d] 2024-05-27T07:23:11.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3c42c36749] 2024-05-27T07:23:11.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3c417da1ca] 2024-05-27T07:23:11.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 16: clone() 2024-05-27T07:23:11.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.209 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 0> 2024-05-27T07:23:10.760+0000 7f3c2d200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: in thread 7f3c2d200700 thread_name:md_log_replay 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3c417e4d20] 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2: gsignal() 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 3: abort() 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3c42c67ced] 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3c42c67dd1] 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dd55cc323d] 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55dd55cae971] 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 8: (SnapRealm::check_cache() const+0x13e) [0x55dd55cacfa0] 2024-05-27T07:23:11.210 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dd55cad651] 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dd55c42347] 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dd5593e87e] 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dd55943e94] 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55dd5594e969] 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 14: (MDLog::_replay_thread()+0x1f38) [0x55dd55ce748c] 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55dd55cf09cb] 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 16: (Thread::entry_wrapper()+0x43) [0x7f3c42c3672d] 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3c42c36749] 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3c417da1ca] 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 19: clone() 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.211 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: -1> 2024-05-27T07:23:10.756+0000 7f3c2d200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f3c2d200700 time 2024-05-27T07:23:10.755732+0000 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3c42c67ba6] 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3c42c67dd1] 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dd55cc323d] 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55dd55cae971] 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 5: (SnapRealm::check_cache() const+0x13e) [0x55dd55cacfa0] 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dd55cad651] 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dd55c42347] 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dd5593e87e] 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dd55943e94] 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55dd5594e969] 2024-05-27T07:23:11.212 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 11: (MDLog::_replay_thread()+0x1f38) [0x55dd55ce748c] 2024-05-27T07:23:11.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55dd55cf09cb] 2024-05-27T07:23:11.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 13: (Thread::entry_wrapper()+0x43) [0x7f3c42c3672d] 2024-05-27T07:23:11.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 14: (Thread::_entry_func(void*)+0xd) [0x7f3c42c36749] 2024-05-27T07:23:11.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f3c417da1ca] 2024-05-27T07:23:11.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 16: clone() 2024-05-27T07:23:11.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 0> 2024-05-27T07:23:10.760+0000 7f3c2d200700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:23:11.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: in thread 7f3c2d200700 thread_name:md_log_replay 2024-05-27T07:23:11.213 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3c417e4d20] 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2: gsignal() 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 3: abort() 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3c42c67ced] 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3c42c67dd1] 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55dd55cc323d] 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55dd55cae971] 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 8: (SnapRealm::check_cache() const+0x13e) [0x55dd55cacfa0] 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55dd55cad651] 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55dd55c42347] 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55dd5593e87e] 2024-05-27T07:23:11.214 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55dd55943e94] 2024-05-27T07:23:11.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55dd5594e969] 2024-05-27T07:23:11.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 14: (MDLog::_replay_thread()+0x1f38) [0x55dd55ce748c] 2024-05-27T07:23:11.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55dd55cf09cb] 2024-05-27T07:23:11.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 16: (Thread::entry_wrapper()+0x43) [0x7f3c42c3672d] 2024-05-27T07:23:11.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 17: (Thread::_entry_func(void*)+0xd) [0x7f3c42c36749] 2024-05-27T07:23:11.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3c417da1ca] 2024-05-27T07:23:11.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 19: clone() 2024-05-27T07:23:11.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:23:11.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l[74221]: 2024-05-27T07:23:11.215 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:10 smithi179 podman[74256]: 2024-05-27 07:23:10.876848334 +0000 UTC m=+0.033147393 container died b6b3f8348382dd96aaf9b44047c87f5e4bb2513c391e063323a84f6d13223be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:23:11.219 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50900 0.61 MB/sec execute 315 sec latency 53.929 ms 2024-05-27T07:23:11.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:11 smithi043 ceph-mon[31531]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:23:11.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:11 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.179:6832/137774809,v1:172.21.15.179:6833/137774809] up:boot 2024-05-27T07:23:11.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:11 smithi043 ceph-mon[31531]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:23:11.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:11 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:23:11.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:11 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:11.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:11 smithi179 ceph-mon[35888]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:23:11.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:11 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.179:6832/137774809,v1:172.21.15.179:6833/137774809] up:boot 2024-05-27T07:23:11.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:11 smithi179 ceph-mon[35888]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:23:11.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:11 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:23:11.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:11 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:11.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:11 smithi132 ceph-mon[35283]: osdmap e126: 12 total, 12 up, 12 in 2024-05-27T07:23:11.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:11 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.179:6832/137774809,v1:172.21.15.179:6833/137774809] up:boot 2024-05-27T07:23:11.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:11 smithi132 ceph-mon[35283]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-05-27T07:23:11.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:11 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-27T07:23:11.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:11 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:12.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:11 smithi179 podman[74256]: 2024-05-27 07:23:11.791036605 +0000 UTC m=+0.947335693 container remove b6b3f8348382dd96aaf9b44047c87f5e4bb2513c391e063323a84f6d13223be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-27T07:23:12.208 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:11 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:23:12.219 INFO:tasks.workunit.client.0.smithi043.stdout: 1 51075 0.62 MB/sec execute 316 sec latency 57.727 ms 2024-05-27T07:23:12.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:12 smithi132 ceph-mon[35283]: pgmap v316: 129 pgs: 129 active+clean; 88 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 675 KiB/s wr, 103 op/s 2024-05-27T07:23:12.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:12 smithi043 ceph-mon[31531]: pgmap v316: 129 pgs: 129 active+clean; 88 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 675 KiB/s wr, 103 op/s 2024-05-27T07:23:12.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:12 smithi179 ceph-mon[35888]: pgmap v316: 129 pgs: 129 active+clean; 88 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 675 KiB/s wr, 103 op/s 2024-05-27T07:23:12.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:12 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:23:12.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:12 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.339s CPU time. 2024-05-27T07:23:13.219 INFO:tasks.workunit.client.0.smithi043.stdout: 1 51160 0.62 MB/sec execute 317 sec latency 73.448 ms 2024-05-27T07:23:14.220 INFO:tasks.workunit.client.0.smithi043.stdout: 1 51249 0.62 MB/sec execute 318 sec latency 64.385 ms 2024-05-27T07:23:14.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:14 smithi132 ceph-mon[35283]: pgmap v317: 129 pgs: 129 active+clean; 88 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 416 KiB/s wr, 46 op/s 2024-05-27T07:23:14.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:14 smithi043 ceph-mon[31531]: pgmap v317: 129 pgs: 129 active+clean; 88 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 416 KiB/s wr, 46 op/s 2024-05-27T07:23:14.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:14 smithi179 ceph-mon[35888]: pgmap v317: 129 pgs: 129 active+clean; 88 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 416 KiB/s wr, 46 op/s 2024-05-27T07:23:15.220 INFO:tasks.workunit.client.0.smithi043.stdout: 1 51342 0.62 MB/sec execute 319 sec latency 65.024 ms 2024-05-27T07:23:15.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:15 smithi132 ceph-mon[35283]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:15.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:15 smithi043 ceph-mon[31531]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:15.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:15 smithi179 ceph-mon[35888]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-05-27T07:23:16.220 INFO:tasks.workunit.client.0.smithi043.stdout: 1 51473 0.61 MB/sec execute 320 sec latency 71.301 ms 2024-05-27T07:23:16.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:16 smithi132 ceph-mon[35283]: pgmap v318: 129 pgs: 129 active+clean; 88 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 340 KiB/s wr, 37 op/s 2024-05-27T07:23:16.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:16 smithi132 ceph-mon[35283]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:23:16.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:16 smithi043 ceph-mon[31531]: pgmap v318: 129 pgs: 129 active+clean; 88 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 340 KiB/s wr, 37 op/s 2024-05-27T07:23:16.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:16 smithi043 ceph-mon[31531]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:23:16.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:16 smithi179 ceph-mon[35888]: pgmap v318: 129 pgs: 129 active+clean; 88 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 340 KiB/s wr, 37 op/s 2024-05-27T07:23:16.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:16 smithi179 ceph-mon[35888]: osdmap e127: 12 total, 12 up, 12 in 2024-05-27T07:23:17.220 INFO:tasks.workunit.client.0.smithi043.stdout: 1 51568 0.61 MB/sec execute 321 sec latency 59.050 ms 2024-05-27T07:23:18.208 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:17 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:23:18.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:17 smithi179 systemd[1]: Stopped Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:23:18.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:17 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Consumed 1.566s CPU time. 2024-05-27T07:23:18.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:17 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Start request repeated too quickly. 2024-05-27T07:23:18.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:17 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-05-27T07:23:18.209 INFO:journalctl@ceph.mds.f.smithi179.stdout:May 27 07:23:17 smithi179 systemd[1]: Failed to start Ceph mds.f for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:23:18.220 INFO:tasks.workunit.client.0.smithi043.stdout: 1 51713 0.61 MB/sec execute 322 sec latency 69.971 ms 2024-05-27T07:23:18.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:18 smithi132 ceph-mon[35283]: pgmap v320: 129 pgs: 129 active+clean; 89 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 478 KiB/s wr, 63 op/s 2024-05-27T07:23:18.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:18 smithi043 ceph-mon[31531]: pgmap v320: 129 pgs: 129 active+clean; 89 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 478 KiB/s wr, 63 op/s 2024-05-27T07:23:18.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:18 smithi179 ceph-mon[35888]: pgmap v320: 129 pgs: 129 active+clean; 89 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 478 KiB/s wr, 63 op/s 2024-05-27T07:23:19.220 INFO:tasks.workunit.client.0.smithi043.stdout: 1 51832 0.61 MB/sec execute 323 sec latency 77.676 ms 2024-05-27T07:23:20.220 INFO:tasks.workunit.client.0.smithi043.stdout: 1 51985 0.61 MB/sec execute 324 sec latency 80.526 ms 2024-05-27T07:23:20.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:20 smithi132 ceph-mon[35283]: pgmap v321: 129 pgs: 129 active+clean; 89 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 448 KiB/s wr, 59 op/s 2024-05-27T07:23:20.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:20 smithi043 ceph-mon[31531]: pgmap v321: 129 pgs: 129 active+clean; 89 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 448 KiB/s wr, 59 op/s 2024-05-27T07:23:20.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:20 smithi179 ceph-mon[35888]: pgmap v321: 129 pgs: 129 active+clean; 89 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 448 KiB/s wr, 59 op/s 2024-05-27T07:23:21.220 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52060 0.61 MB/sec execute 325 sec latency 53.175 ms 2024-05-27T07:23:21.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:21 smithi132 ceph-mon[35283]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:23:21.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:21 smithi132 ceph-mon[35283]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:23:21.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:21 smithi043 ceph-mon[31531]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:23:21.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:21 smithi043 ceph-mon[31531]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:23:21.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:21 smithi179 ceph-mon[35888]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-05-27T07:23:21.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:21 smithi179 ceph-mon[35888]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2024-05-27T07:23:22.221 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52074 0.61 MB/sec execute 326 sec latency 885.871 ms 2024-05-27T07:23:22.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:22 smithi132 ceph-mon[35283]: pgmap v322: 129 pgs: 129 active+clean; 90 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 185 KiB/s wr, 33 op/s 2024-05-27T07:23:22.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:22 smithi043 ceph-mon[31531]: pgmap v322: 129 pgs: 129 active+clean; 90 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 185 KiB/s wr, 33 op/s 2024-05-27T07:23:22.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:22 smithi179 ceph-mon[35888]: pgmap v322: 129 pgs: 129 active+clean; 90 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 185 KiB/s wr, 33 op/s 2024-05-27T07:23:22.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:22 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-05-27T07:23:22.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:22 smithi179 systemd[1]: Stopped Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:23:22.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:22 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Consumed 2.339s CPU time. 2024-05-27T07:23:22.708 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:22 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Start request repeated too quickly. 2024-05-27T07:23:22.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:22 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-05-27T07:23:22.709 INFO:journalctl@ceph.mds.l.smithi179.stdout:May 27 07:23:22 smithi179 systemd[1]: Failed to start Ceph mds.l for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:23:22.925 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:23:23.221 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52104 0.61 MB/sec execute 327 sec latency 1445.358 ms 2024-05-27T07:23:23.419 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:24.221 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52175 0.61 MB/sec execute 328 sec latency 349.388 ms 2024-05-27T07:23:24.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:24 smithi132 ceph-mon[35283]: pgmap v323: 129 pgs: 129 active+clean; 90 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 185 KiB/s wr, 33 op/s 2024-05-27T07:23:24.612 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:24 smithi043 ceph-mon[31531]: pgmap v323: 129 pgs: 129 active+clean; 90 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 185 KiB/s wr, 33 op/s 2024-05-27T07:23:24.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:24 smithi179 ceph-mon[35888]: pgmap v323: 129 pgs: 129 active+clean; 90 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 185 KiB/s wr, 33 op/s 2024-05-27T07:23:25.221 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52175 0.60 MB/sec execute 329 sec latency 1349.483 ms 2024-05-27T07:23:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.803+0000 7efc25c00700 1 -- 172.21.15.43:0/2599611623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc2010a980 msgr2=0x7efc2010ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.803+0000 7efc25c00700 1 --2- 172.21.15.43:0/2599611623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc2010a980 0x7efc2010ad90 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7efc08009990 tx=0x7efc08009ca0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.803+0000 7efc25c00700 1 -- 172.21.15.43:0/2599611623 shutdown_connections 2024-05-27T07:23:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.803+0000 7efc25c00700 1 --2- 172.21.15.43:0/2599611623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc2010b2d0 0x7efc20117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.804+0000 7efc25c00700 1 --2- 172.21.15.43:0/2599611623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc20110960 0x7efc20110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.804+0000 7efc25c00700 1 --2- 172.21.15.43:0/2599611623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc2010a980 0x7efc2010ad90 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.804+0000 7efc25c00700 1 -- 172.21.15.43:0/2599611623 >> 172.21.15.43:0/2599611623 conn(0x7efc20077c70 msgr2=0x7efc200780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:25.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.804+0000 7efc25c00700 1 -- 172.21.15.43:0/2599611623 shutdown_connections 2024-05-27T07:23:25.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.804+0000 7efc25c00700 1 -- 172.21.15.43:0/2599611623 wait complete. 2024-05-27T07:23:25.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.805+0000 7efc25c00700 1 Processor -- start 2024-05-27T07:23:25.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.805+0000 7efc25c00700 1 -- start start 2024-05-27T07:23:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc25c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc2010a980 0x7efc201a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc25c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc2010b2d0 0x7efc201a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc25c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc20110960 0x7efc201a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc25c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efc2011a910 con 0x7efc2010a980 2024-05-27T07:23:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc25c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efc2011a790 con 0x7efc2010b2d0 2024-05-27T07:23:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc25c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efc2011aa90 con 0x7efc20110960 2024-05-27T07:23:25.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc1ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc2010a980 0x7efc201a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc1ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc2010a980 0x7efc201a1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59020/0 (socket says 172.21.15.43:59020) 2024-05-27T07:23:25.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc1ea00700 1 -- 172.21.15.43:0/2005804869 learned_addr learned my addr 172.21.15.43:0/2005804869 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:25.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc16000700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc2010b2d0 0x7efc201a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.806+0000 7efc1f400700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc20110960 0x7efc201a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.807+0000 7efc1ea00700 1 -- 172.21.15.43:0/2005804869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc20110960 msgr2=0x7efc201a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.807+0000 7efc1ea00700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc20110960 0x7efc201a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.807+0000 7efc1ea00700 1 -- 172.21.15.43:0/2005804869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc2010b2d0 msgr2=0x7efc201a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.807+0000 7efc1ea00700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc2010b2d0 0x7efc201a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.807+0000 7efc1ea00700 1 -- 172.21.15.43:0/2005804869 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc201a6210 con 0x7efc2010a980 2024-05-27T07:23:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.808+0000 7efc1ea00700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc2010a980 0x7efc201a1340 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efc1000d980 tx=0x7efc1000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.808+0000 7efc17e00700 1 -- 172.21.15.43:0/2005804869 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc10017070 con 0x7efc2010a980 2024-05-27T07:23:25.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.808+0000 7efc17e00700 1 -- 172.21.15.43:0/2005804869 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc1000fd20 con 0x7efc2010a980 2024-05-27T07:23:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.808+0000 7efc25c00700 1 -- 172.21.15.43:0/2005804869 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc201a6500 con 0x7efc2010a980 2024-05-27T07:23:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.808+0000 7efc17e00700 1 -- 172.21.15.43:0/2005804869 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc1000e3b0 con 0x7efc2010a980 2024-05-27T07:23:25.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.808+0000 7efc25c00700 1 -- 172.21.15.43:0/2005804869 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc201ade20 con 0x7efc2010a980 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.813+0000 7efc17e00700 1 -- 172.21.15.43:0/2005804869 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7efc1000f890 con 0x7efc2010a980 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.813+0000 7efc3469eb80 1 Processor -- start 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.813+0000 7efc3469eb80 1 -- start start 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.814+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7e8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.814+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a669e40 0x55a04a7e8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.814+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a7e8e70 0x55a04a7f5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.814+0000 7efc3469eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a04a716e40 con 0x55a04a7559f0 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.814+0000 7efc3469eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a04a7069e0 con 0x55a04a7e8e70 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.814+0000 7efc3469eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a04a70aa20 con 0x55a04a669e40 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.814+0000 7efc1f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7e8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.814+0000 7efc1f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7e8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59028/0 (socket says 172.21.15.43:59028) 2024-05-27T07:23:25.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.814+0000 7efc1f400700 1 -- 172.21.15.43:0/1585035050 learned_addr learned my addr 172.21.15.43:0/1585035050 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:25.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.815+0000 7efc16000700 1 --2- 172.21.15.43:0/1585035050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a7e8e70 0x55a04a7f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.815+0000 7efc1ea00700 1 --2- 172.21.15.43:0/1585035050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a669e40 0x55a04a7e8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.815+0000 7efc1f400700 1 -- 172.21.15.43:0/1585035050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a669e40 msgr2=0x55a04a7e8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:25.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.815+0000 7efc1f400700 1 --2- 172.21.15.43:0/1585035050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a669e40 0x55a04a7e8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.815+0000 7efc1f400700 1 -- 172.21.15.43:0/1585035050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a7e8e70 msgr2=0x55a04a7f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:25.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.815+0000 7efc1f400700 1 --2- 172.21.15.43:0/1585035050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a7e8e70 0x55a04a7f5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.815+0000 7efc17e00700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc040b29f0 0x7efc040b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.815+0000 7efc1f400700 1 -- 172.21.15.43:0/1585035050 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a04a6cf420 con 0x55a04a7559f0 2024-05-27T07:23:25.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.815+0000 7efc17e00700 1 -- 172.21.15.43:0/2005804869 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7efc100e2440 con 0x7efc2010a980 2024-05-27T07:23:25.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.815+0000 7efc1f400700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc040b29f0 0x7efc040b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.816+0000 7efc1f400700 1 --2- 172.21.15.43:0/1585035050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7e8330 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7efc180098f0 tx=0x7efc18009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184a4beb8a53cf24 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:25.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.816+0000 7efc0fe00700 1 -- 172.21.15.43:0/1585035050 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc18007700 con 0x55a04a7559f0 2024-05-27T07:23:25.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.816+0000 7efc0fe00700 1 -- 172.21.15.43:0/1585035050 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc18007db0 con 0x55a04a7559f0 2024-05-27T07:23:25.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.816+0000 7efc0fe00700 1 -- 172.21.15.43:0/1585035050 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc18015e50 con 0x55a04a7559f0 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.817+0000 7efc3469eb80 1 -- 172.21.15.43:0/1585035050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 msgr2=0x55a04a7e8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.817+0000 7efc3469eb80 1 --2- 172.21.15.43:0/1585035050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7e8330 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7efc180098f0 tx=0x7efc18009c00 comp rx=0 tx=0).stop 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.817+0000 7efc1f400700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc040b29f0 0x7efc040b4f00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efc1801f000 tx=0x7efc180146c0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.817+0000 7efc3469eb80 1 -- 172.21.15.43:0/1585035050 shutdown_connections 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.817+0000 7efc3469eb80 1 --2- 172.21.15.43:0/1585035050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a7e8e70 0x55a04a7f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.817+0000 7efc3469eb80 1 --2- 172.21.15.43:0/1585035050 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7e8330 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.817+0000 7efc3469eb80 1 --2- 172.21.15.43:0/1585035050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a669e40 0x55a04a7e8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.817+0000 7efc3469eb80 1 -- 172.21.15.43:0/1585035050 >> 172.21.15.43:0/1585035050 conn(0x55a04a703390 msgr2=0x55a04a7e44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.817+0000 7efc3469eb80 1 -- 172.21.15.43:0/1585035050 shutdown_connections 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.817+0000 7efc3469eb80 1 -- 172.21.15.43:0/1585035050 wait complete. 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.818+0000 7efc3469eb80 1 -- start start 2024-05-27T07:23:25.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.818+0000 7efc3469eb80 1 Processor -- start 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a669e40 0x55a04a7f2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 0x55a04a7f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc3469eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a04a60cfa0 con 0x55a04a7559f0 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc3469eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a04a6a7760 con 0x55a04a669e40 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc3469eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a04a7f5bd0 con 0x55a04a7e8e70 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc3469eb80 10 client.0 _ll_register_callbacks cb 0x55a04a6b0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc16000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc16000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59034/0 (socket says 172.21.15.43:59034) 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc16000700 1 -- 172.21.15.43:0/2912918439 learned_addr learned my addr 172.21.15.43:0/2912918439 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.819+0000 7efc1ea00700 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a669e40 0x55a04a7f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.820+0000 7efc1f400700 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 0x55a04a7f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.820+0000 7efc16000700 1 -- 172.21.15.43:0/2912918439 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 msgr2=0x55a04a7f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.820+0000 7efc16000700 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 0x55a04a7f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.820+0000 7efc16000700 1 -- 172.21.15.43:0/2912918439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a669e40 msgr2=0x55a04a7f2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.820+0000 7efc16000700 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a669e40 0x55a04a7f2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.820+0000 7efc16000700 1 -- 172.21.15.43:0/2912918439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a04a6568d0 con 0x55a04a7559f0 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.820+0000 7efc16000700 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7f2c90 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7efc2006d110 tx=0x7efc08005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b66dbe28e43e89 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.820+0000 7efc0cc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.821+0000 7efc0cc00700 1 -- 172.21.15.43:0/2912918439 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08003bd0 con 0x55a04a7559f0 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.821+0000 7efc0cc00700 1 -- 172.21.15.43:0/2912918439 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7efc08003d70 con 0x55a04a7559f0 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.821+0000 7efc3469eb80 1 -- 172.21.15.43:0/2912918439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc18007db0 con 0x55a04a7559f0 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.821+0000 7efc0cc00700 1 -- 172.21.15.43:0/2912918439 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7efc08031050 con 0x55a04a7559f0 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.821+0000 7efc3469eb80 10 client.16839 fetch_fsmap learned FSMap version 101 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.821+0000 7efc3469eb80 1 -- 172.21.15.43:0/2912918439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55a04a6d7060 con 0x55a04a7559f0 2024-05-27T07:23:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.821+0000 7efc0cc00700 1 -- 172.21.15.43:0/2912918439 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc18007db0 con 0x55a04a7559f0 2024-05-27T07:23:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.822+0000 7efc0cc00700 1 -- 172.21.15.43:0/2912918439 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 101) ==== 5381+0+0 (secure 0 0 0) 0x7efc080236a0 con 0x55a04a7559f0 2024-05-27T07:23:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.822+0000 7efc3469eb80 10 client.16839 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:23:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.822+0000 7efc3469eb80 10 client.16839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.822+0000 7efc3469eb80 20 client.16839 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.822+0000 7efc3469eb80 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a04a802e20 0x55a04a8052f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.822+0000 7efc3469eb80 4 client.16839 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.822+0000 7efc3469eb80 1 -- 172.21.15.43:0/2912918439 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a04a718d90 con 0x55a04a802e20 2024-05-27T07:23:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.822+0000 7efc1ea00700 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a04a802e20 0x55a04a8052f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.823+0000 7efc1ea00700 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a04a802e20 0x55a04a8052f0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a28a69b6d01da62b server_cookie=bced6ad31a4dea9f in_seq=0 out_seq=0 2024-05-27T07:23:25.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.823+0000 7efc0cc00700 10 client.16839 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:25.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc0cc00700 1 -- 172.21.15.43:0/2912918439 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a04a718d90 con 0x55a04a802e20 2024-05-27T07:23:25.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc0cc00700 10 client.16839 handle_command_reply: tid=0 2024-05-27T07:23:25.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc3469eb80 1 client.16839 shutdown 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc3469eb80 1 -- 172.21.15.43:0/2912918439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 msgr2=0x55a04a7f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc3469eb80 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7f2c90 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7efc2006d110 tx=0x7efc08005800 comp rx=0 tx=0).stop 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc3469eb80 1 -- 172.21.15.43:0/2912918439 shutdown_connections 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc3469eb80 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 0x55a04a7f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc3469eb80 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a669e40 0x55a04a7f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc3469eb80 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a04a802e20 0x55a04a8052f0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc3469eb80 1 --2- 172.21.15.43:0/2912918439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7f2c90 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc3469eb80 1 -- 172.21.15.43:0/2912918439 >> 172.21.15.43:0/2912918439 conn(0x55a04a703390 msgr2=0x55a04a7e8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.826+0000 7efc0cc00700 0 client.16839 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.827+0000 7efc3469eb80 1 -- 172.21.15.43:0/2912918439 shutdown_connections 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.827+0000 7efc3469eb80 1 -- 172.21.15.43:0/2912918439 wait complete. 2024-05-27T07:23:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.827+0000 7efc3469eb80 20 client.16839 trim_cache size 0 max 16384 2024-05-27T07:23:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.850+0000 7efc3469eb80 1 Processor -- start 2024-05-27T07:23:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.850+0000 7efc3469eb80 1 -- start start 2024-05-27T07:23:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.850+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a669e40 0x55a04a7e4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.850+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7fec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.850+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a7e8e70 0x55a04a80aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.850+0000 7efc3469eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efc201b7f50 con 0x55a04a7559f0 2024-05-27T07:23:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.850+0000 7efc3469eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efc201b5610 con 0x55a04a7e8e70 2024-05-27T07:23:25.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.850+0000 7efc3469eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efc201b7880 con 0x55a04a669e40 2024-05-27T07:23:25.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.851+0000 7efc16000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7fec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.851+0000 7efc16000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7fec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59050/0 (socket says 172.21.15.43:59050) 2024-05-27T07:23:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.851+0000 7efc16000700 1 -- 172.21.15.43:0/2827784352 learned_addr learned my addr 172.21.15.43:0/2827784352 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.851+0000 7efc1f400700 1 --2- 172.21.15.43:0/2827784352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a7e8e70 0x55a04a80aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.851+0000 7efc1ea00700 1 --2- 172.21.15.43:0/2827784352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a669e40 0x55a04a7e4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.851+0000 7efc16000700 1 -- 172.21.15.43:0/2827784352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a669e40 msgr2=0x55a04a7e4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.851+0000 7efc16000700 1 --2- 172.21.15.43:0/2827784352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a669e40 0x55a04a7e4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.851+0000 7efc16000700 1 -- 172.21.15.43:0/2827784352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a7e8e70 msgr2=0x55a04a80aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.851+0000 7efc16000700 1 --2- 172.21.15.43:0/2827784352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a7e8e70 0x55a04a80aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.851+0000 7efc16000700 1 -- 172.21.15.43:0/2827784352 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a04a802c00 con 0x55a04a7559f0 2024-05-27T07:23:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.852+0000 7efc16000700 1 --2- 172.21.15.43:0/2827784352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7fec70 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7efc08023b40 tx=0x7efc08009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a2cfafd95a9904 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.852+0000 7efc0f400700 1 -- 172.21.15.43:0/2827784352 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08023670 con 0x55a04a7559f0 2024-05-27T07:23:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.852+0000 7efc0f400700 1 -- 172.21.15.43:0/2827784352 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc08028070 con 0x55a04a7559f0 2024-05-27T07:23:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.852+0000 7efc0f400700 1 -- 172.21.15.43:0/2827784352 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0802c950 con 0x55a04a7559f0 2024-05-27T07:23:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.853+0000 7efc3469eb80 1 -- 172.21.15.43:0/2827784352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 msgr2=0x55a04a7fec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.853+0000 7efc3469eb80 1 --2- 172.21.15.43:0/2827784352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7fec70 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7efc08023b40 tx=0x7efc08009710 comp rx=0 tx=0).stop 2024-05-27T07:23:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.853+0000 7efc3469eb80 1 -- 172.21.15.43:0/2827784352 shutdown_connections 2024-05-27T07:23:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.853+0000 7efc3469eb80 1 --2- 172.21.15.43:0/2827784352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a7e8e70 0x55a04a80aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.853+0000 7efc3469eb80 1 --2- 172.21.15.43:0/2827784352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a669e40 0x55a04a7e4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.853+0000 7efc3469eb80 1 --2- 172.21.15.43:0/2827784352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7559f0 0x55a04a7fec70 secure :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7efc08023b40 tx=0x7efc08009710 comp rx=0 tx=0).stop 2024-05-27T07:23:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.853+0000 7efc3469eb80 1 -- 172.21.15.43:0/2827784352 >> 172.21.15.43:0/2827784352 conn(0x55a04a703390 msgr2=0x55a04a7e3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.853+0000 7efc3469eb80 1 -- 172.21.15.43:0/2827784352 shutdown_connections 2024-05-27T07:23:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.853+0000 7efc3469eb80 1 -- 172.21.15.43:0/2827784352 wait complete. 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.854+0000 7efc3469eb80 1 -- start start 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.854+0000 7efc3469eb80 1 Processor -- start 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a669e40 0x55a04a7fb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 0x55a04a7fb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc3469eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7ec780 0x55a04a7ecc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc3469eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efc201b71b0 con 0x55a04a7ec780 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc3469eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efc201b5ce0 con 0x55a04a669e40 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc3469eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efc201b63b0 con 0x55a04a7e8e70 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc3469eb80 10 client.0 _ll_register_callbacks cb 0x55a04a6b0c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc1f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7ec780 0x55a04a7ecc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc1f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7ec780 0x55a04a7ecc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59054/0 (socket says 172.21.15.43:59054) 2024-05-27T07:23:25.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc1f400700 1 -- 172.21.15.43:0/1899940466 learned_addr learned my addr 172.21.15.43:0/1899940466 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:25.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc16000700 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 0x55a04a7fb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.855+0000 7efc1ea00700 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a669e40 0x55a04a7fb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.856+0000 7efc1f400700 1 -- 172.21.15.43:0/1899940466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 msgr2=0x55a04a7fb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.856+0000 7efc1f400700 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 0x55a04a7fb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.856+0000 7efc1f400700 1 -- 172.21.15.43:0/1899940466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a669e40 msgr2=0x55a04a7fb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.856+0000 7efc1f400700 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a669e40 0x55a04a7fb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.856+0000 7efc1f400700 1 -- 172.21.15.43:0/1899940466 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a04a7ed3b0 con 0x55a04a7ec780 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.856+0000 7efc16000700 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 0x55a04a7fb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.856+0000 7efc1f400700 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7ec780 0x55a04a7ecc10 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7efc180098f0 tx=0x7efc18015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e34d6ae654948e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.856+0000 7efc0cc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.856+0000 7efc0cc00700 1 -- 172.21.15.43:0/1899940466 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc18015e50 con 0x55a04a7ec780 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.857+0000 7efc3469eb80 1 -- 172.21.15.43:0/1899940466 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc08028070 con 0x55a04a7ec780 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.857+0000 7efc0cc00700 1 -- 172.21.15.43:0/1899940466 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7efc18015b00 con 0x55a04a7ec780 2024-05-27T07:23:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.857+0000 7efc0cc00700 1 -- 172.21.15.43:0/1899940466 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7efc18041050 con 0x55a04a7ec780 2024-05-27T07:23:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.857+0000 7efc3469eb80 10 client.16845 fetch_fsmap learned FSMap version 101 2024-05-27T07:23:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.857+0000 7efc3469eb80 1 -- 172.21.15.43:0/1899940466 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x55a04a86d9c0 con 0x55a04a7ec780 2024-05-27T07:23:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.857+0000 7efc0cc00700 1 -- 172.21.15.43:0/1899940466 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc08028070 con 0x55a04a7ec780 2024-05-27T07:23:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.857+0000 7efc0cc00700 1 -- 172.21.15.43:0/1899940466 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 101) ==== 5381+0+0 (secure 0 0 0) 0x7efc1803a990 con 0x55a04a7ec780 2024-05-27T07:23:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.858+0000 7efc3469eb80 10 client.16845 fetch_fsmap finished waiting for FSMap version 101 2024-05-27T07:23:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.858+0000 7efc3469eb80 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.858+0000 7efc3469eb80 20 client.16845 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.858+0000 7efc3469eb80 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a04a86dee0 0x55a04a870300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.858+0000 7efc3469eb80 4 client.16845 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:23:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.858+0000 7efc3469eb80 1 -- 172.21.15.43:0/1899940466 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a04a7e3120 con 0x55a04a86dee0 2024-05-27T07:23:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.858+0000 7efc1ea00700 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a04a86dee0 0x55a04a870300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.859+0000 7efc1ea00700 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a04a86dee0 0x55a04a870300 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=922fac2e26e5b0ca server_cookie=27d6055b089bc977 in_seq=0 out_seq=0 2024-05-27T07:23:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.859+0000 7efc0cc00700 10 client.16845 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc0cc00700 1 -- 172.21.15.43:0/1899940466 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a04a7e3120 con 0x55a04a86dee0 2024-05-27T07:23:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc0cc00700 10 client.16845 handle_command_reply: tid=0 2024-05-27T07:23:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc3469eb80 1 client.16845 shutdown 2024-05-27T07:23:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc3469eb80 1 -- 172.21.15.43:0/1899940466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7ec780 msgr2=0x55a04a7ecc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc3469eb80 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7ec780 0x55a04a7ecc10 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7efc180098f0 tx=0x7efc18015000 comp rx=0 tx=0).stop 2024-05-27T07:23:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc3469eb80 1 -- 172.21.15.43:0/1899940466 shutdown_connections 2024-05-27T07:23:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc3469eb80 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55a04a86dee0 0x55a04a870300 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc3469eb80 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a04a7ec780 0x55a04a7ecc10 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc3469eb80 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a04a669e40 0x55a04a7fb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc3469eb80 1 --2- 172.21.15.43:0/1899940466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a04a7e8e70 0x55a04a7fb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc3469eb80 1 -- 172.21.15.43:0/1899940466 >> 172.21.15.43:0/1899940466 conn(0x55a04a703390 msgr2=0x55a04a7e9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.860+0000 7efc0cc00700 0 client.16845 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.861+0000 7efc3469eb80 1 -- 172.21.15.43:0/1899940466 shutdown_connections 2024-05-27T07:23:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.861+0000 7efc3469eb80 1 -- 172.21.15.43:0/1899940466 wait complete. 2024-05-27T07:23:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.861+0000 7efc3469eb80 20 client.16845 trim_cache size 0 max 16384 2024-05-27T07:23:25.853 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:23:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:23:25.853 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:23:25.853 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.861+0000 7efc25c00700 1 -- 172.21.15.43:0/2005804869 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc040b29f0 msgr2=0x7efc040b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.861+0000 7efc25c00700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc040b29f0 0x7efc040b4f00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efc1801f000 tx=0x7efc180146c0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.861+0000 7efc25c00700 1 -- 172.21.15.43:0/2005804869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc2010a980 msgr2=0x7efc201a1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.861+0000 7efc25c00700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc2010a980 0x7efc201a1340 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efc1000d980 tx=0x7efc1000dc90 comp rx=0 tx=0).stop 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.862+0000 7efc25c00700 1 -- 172.21.15.43:0/2005804869 shutdown_connections 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.862+0000 7efc25c00700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc040b29f0 0x7efc040b4f00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.862+0000 7efc25c00700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc20110960 0x7efc201a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.862+0000 7efc25c00700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc2010a980 0x7efc201a1340 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.862+0000 7efc25c00700 1 --2- 172.21.15.43:0/2005804869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc2010b2d0 0x7efc201a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.862+0000 7efc25c00700 1 -- 172.21.15.43:0/2005804869 >> 172.21.15.43:0/2005804869 conn(0x7efc20077c70 msgr2=0x7efc2007b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.862+0000 7efc25c00700 1 -- 172.21.15.43:0/2005804869 shutdown_connections 2024-05-27T07:23:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:25.862+0000 7efc25c00700 1 -- 172.21.15.43:0/2005804869 wait complete. 2024-05-27T07:23:26.221 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52175 0.60 MB/sec execute 330 sec latency 2349.579 ms 2024-05-27T07:23:26.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:26 smithi132 ceph-mon[35283]: pgmap v324: 129 pgs: 129 active+clean; 90 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 185 KiB/s wr, 33 op/s 2024-05-27T07:23:26.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:26 smithi132 ceph-mon[35283]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:23:26.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:26 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:26.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:26 smithi043 ceph-mon[31531]: pgmap v324: 129 pgs: 129 active+clean; 90 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 185 KiB/s wr, 33 op/s 2024-05-27T07:23:26.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:26 smithi043 ceph-mon[31531]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:23:26.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:26 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:26.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:26 smithi179 ceph-mon[35888]: pgmap v324: 129 pgs: 129 active+clean; 90 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 185 KiB/s wr, 33 op/s 2024-05-27T07:23:26.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:26 smithi179 ceph-mon[35888]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-05-27T07:23:26.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:26 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:26.792 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:23:26.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:23:27.221 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52175 0.60 MB/sec execute 331 sec latency 3349.698 ms 2024-05-27T07:23:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:28.221 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52241 0.60 MB/sec execute 332 sec latency 3908.575 ms 2024-05-27T07:23:28.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:28 smithi132 ceph-mon[35283]: pgmap v325: 129 pgs: 129 active+clean; 91 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 299 KiB/s wr, 23 op/s 2024-05-27T07:23:28.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:28 smithi043 ceph-mon[31531]: pgmap v325: 129 pgs: 129 active+clean; 91 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 299 KiB/s wr, 23 op/s 2024-05-27T07:23:28.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:28 smithi179 ceph-mon[35888]: pgmap v325: 129 pgs: 129 active+clean; 91 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 299 KiB/s wr, 23 op/s 2024-05-27T07:23:29.221 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52323 0.60 MB/sec execute 333 sec latency 52.406 ms 2024-05-27T07:23:30.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.140+0000 7fa5ab000700 1 -- 172.21.15.43:0/1526891525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5a41043f0 msgr2=0x7fa5a4104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:30.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.140+0000 7fa5ab000700 1 --2- 172.21.15.43:0/1526891525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5a41043f0 0x7fa5a4104800 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa5980099c0 tx=0x7fa598009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.140+0000 7fa5ab000700 1 -- 172.21.15.43:0/1526891525 shutdown_connections 2024-05-27T07:23:30.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.140+0000 7fa5ab000700 1 --2- 172.21.15.43:0/1526891525 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5a4109670 0x7fa5a4110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.140+0000 7fa5ab000700 1 --2- 172.21.15.43:0/1526891525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5a41043f0 0x7fa5a4104800 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.140+0000 7fa5ab000700 1 --2- 172.21.15.43:0/1526891525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5a4108d20 0x7fa5a4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.140+0000 7fa5ab000700 1 -- 172.21.15.43:0/1526891525 >> 172.21.15.43:0/1526891525 conn(0x7fa5a4079d00 msgr2=0x7fa5a407a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:30.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.140+0000 7fa5ab000700 1 -- 172.21.15.43:0/1526891525 shutdown_connections 2024-05-27T07:23:30.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.140+0000 7fa5ab000700 1 -- 172.21.15.43:0/1526891525 wait complete. 2024-05-27T07:23:30.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.141+0000 7fa5ab000700 1 Processor -- start 2024-05-27T07:23:30.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.141+0000 7fa5ab000700 1 -- start start 2024-05-27T07:23:30.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.142+0000 7fa5ab000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5a41043f0 0x7fa5a41a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.142+0000 7fa5ab000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5a4108d20 0x7fa5a41a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.142+0000 7fa5ab000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5a4109670 0x7fa5a419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.142+0000 7fa5ab000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a41141a0 con 0x7fa5a41043f0 2024-05-27T07:23:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.142+0000 7fa5ab000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a4114020 con 0x7fa5a4109670 2024-05-27T07:23:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.142+0000 7fa5ab000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a4114320 con 0x7fa5a4108d20 2024-05-27T07:23:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.142+0000 7fa5a9c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5a41043f0 0x7fa5a41a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.142+0000 7fa5a9c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5a41043f0 0x7fa5a41a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59072/0 (socket says 172.21.15.43:59072) 2024-05-27T07:23:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.142+0000 7fa5a9c00700 1 -- 172.21.15.43:0/4185199347 learned_addr learned my addr 172.21.15.43:0/4185199347 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.143+0000 7fa5a9200700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5a4108d20 0x7fa5a41a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.143+0000 7fa5aa600700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5a4109670 0x7fa5a419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.143+0000 7fa5a9c00700 1 -- 172.21.15.43:0/4185199347 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5a4108d20 msgr2=0x7fa5a41a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.143+0000 7fa5a9c00700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5a4108d20 0x7fa5a41a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.143+0000 7fa5a9c00700 1 -- 172.21.15.43:0/4185199347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5a4109670 msgr2=0x7fa5a419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.143+0000 7fa5a9c00700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5a4109670 0x7fa5a419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.143+0000 7fa5a9c00700 1 -- 172.21.15.43:0/4185199347 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a419b070 con 0x7fa5a41043f0 2024-05-27T07:23:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.143+0000 7fa5a9c00700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5a41043f0 0x7fa5a41a07a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa598000c00 tx=0x7fa59800f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.144+0000 7fa5a2000700 1 -- 172.21.15.43:0/4185199347 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598027070 con 0x7fa5a41043f0 2024-05-27T07:23:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.144+0000 7fa5a2000700 1 -- 172.21.15.43:0/4185199347 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa59800fe50 con 0x7fa5a41043f0 2024-05-27T07:23:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.144+0000 7fa5ab000700 1 -- 172.21.15.43:0/4185199347 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a419b300 con 0x7fa5a41043f0 2024-05-27T07:23:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.144+0000 7fa5a2000700 1 -- 172.21.15.43:0/4185199347 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5980173f0 con 0x7fa5a41043f0 2024-05-27T07:23:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.144+0000 7fa5ab000700 1 -- 172.21.15.43:0/4185199347 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a41a7630 con 0x7fa5a41043f0 2024-05-27T07:23:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.145+0000 7fa5ab000700 1 -- 172.21.15.43:0/4185199347 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a41058a0 con 0x7fa5a41043f0 2024-05-27T07:23:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.149+0000 7fa5a2000700 1 -- 172.21.15.43:0/4185199347 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa59802a070 con 0x7fa5a41043f0 2024-05-27T07:23:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.151+0000 7fa5a2000700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa58c0b2a70 0x7fa58c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.151+0000 7fa5a2000700 1 -- 172.21.15.43:0/4185199347 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7fa5980e3ba0 con 0x7fa5a41043f0 2024-05-27T07:23:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.151+0000 7fa5a9200700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa58c0b2a70 0x7fa58c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:30.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.152+0000 7fa5a9200700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa58c0b2a70 0x7fa58c0b4f80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa590005950 tx=0x7fa5900058e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.154+0000 7fa5a2000700 1 -- 172.21.15.43:0/4185199347 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa59805fdf0 con 0x7fa5a41043f0 2024-05-27T07:23:30.221 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52613 0.61 MB/sec execute 334 sec latency 59.129 ms 2024-05-27T07:23:30.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:30 smithi043 ceph-mon[31531]: pgmap v326: 129 pgs: 129 active+clean; 91 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 263 KiB/s wr, 21 op/s 2024-05-27T07:23:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.484+0000 7fa5ab000700 1 -- 172.21.15.43:0/4185199347 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5a419bef0 con 0x7fa5a41043f0 2024-05-27T07:23:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.486+0000 7fa5a2000700 1 -- 172.21.15.43:0/4185199347 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6956 (secure 0 0 0) 0x7fa598013070 con 0x7fa5a41043f0 2024-05-27T07:23:30.477 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:23:30.477 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:23:26.193343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34484,34448,34424,24484,34403]},"id":1}]} 2024-05-27T07:23:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.489+0000 7fa5ab000700 1 -- 172.21.15.43:0/4185199347 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa58c0b2a70 msgr2=0x7fa58c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.489+0000 7fa5ab000700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa58c0b2a70 0x7fa58c0b4f80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa590005950 tx=0x7fa5900058e0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.489+0000 7fa5ab000700 1 -- 172.21.15.43:0/4185199347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5a41043f0 msgr2=0x7fa5a41a07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.489+0000 7fa5ab000700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5a41043f0 0x7fa5a41a07a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa598000c00 tx=0x7fa59800f510 comp rx=0 tx=0).stop 2024-05-27T07:23:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.490+0000 7fa5ab000700 1 -- 172.21.15.43:0/4185199347 shutdown_connections 2024-05-27T07:23:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.490+0000 7fa5ab000700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa58c0b2a70 0x7fa58c0b4f80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.490+0000 7fa5ab000700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5a4109670 0x7fa5a419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.490+0000 7fa5ab000700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5a41043f0 0x7fa5a41a07a0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.490+0000 7fa5ab000700 1 --2- 172.21.15.43:0/4185199347 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5a4108d20 0x7fa5a41a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.490+0000 7fa5ab000700 1 -- 172.21.15.43:0/4185199347 >> 172.21.15.43:0/4185199347 conn(0x7fa5a4079d00 msgr2=0x7fa5a4101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.490+0000 7fa5ab000700 1 -- 172.21.15.43:0/4185199347 shutdown_connections 2024-05-27T07:23:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:30.490+0000 7fa5ab000700 1 -- 172.21.15.43:0/4185199347 wait complete. 2024-05-27T07:23:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 102 2024-05-27T07:23:30.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:30 smithi132 ceph-mon[35283]: pgmap v326: 129 pgs: 129 active+clean; 91 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 263 KiB/s wr, 21 op/s 2024-05-27T07:23:30.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:30 smithi179 ceph-mon[35888]: pgmap v326: 129 pgs: 129 active+clean; 91 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 263 KiB/s wr, 21 op/s 2024-05-27T07:23:31.221 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52875 0.61 MB/sec execute 335 sec latency 52.562 ms 2024-05-27T07:23:31.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:23:31.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:31 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:23:31.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:31 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4185199347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:23:31.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:31 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:23:31.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:31 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4185199347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:23:31.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:31 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:23:31.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:31 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4185199347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:23:31.889 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:32.222 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52990 0.61 MB/sec execute 336 sec latency 63.482 ms 2024-05-27T07:23:32.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:32 smithi043 ceph-mon[31531]: pgmap v327: 129 pgs: 129 active+clean; 93 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 436 KiB/s wr, 30 op/s 2024-05-27T07:23:32.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:32 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:23:32.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:32 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:23:32.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:32 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:23:32.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:32 smithi179 ceph-mon[35888]: pgmap v327: 129 pgs: 129 active+clean; 93 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 436 KiB/s wr, 30 op/s 2024-05-27T07:23:32.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:32 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:23:32.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:32 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:23:32.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:32 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:23:32.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:32 smithi132 ceph-mon[35283]: pgmap v327: 129 pgs: 129 active+clean; 93 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 436 KiB/s wr, 30 op/s 2024-05-27T07:23:32.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:32 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:23:32.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:32 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:23:32.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:32 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:23:33.222 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53014 0.61 MB/sec execute 337 sec latency 63.532 ms 2024-05-27T07:23:34.222 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53080 0.61 MB/sec execute 338 sec latency 57.882 ms 2024-05-27T07:23:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.266+0000 7f540be00700 1 -- 172.21.15.43:0/320956184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f540c106610 msgr2=0x7f540c106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.266+0000 7f540be00700 1 --2- 172.21.15.43:0/320956184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f540c106610 0x7f540c106a20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0099c0 tx=0x7f53fc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.266+0000 7f540be00700 1 -- 172.21.15.43:0/320956184 shutdown_connections 2024-05-27T07:23:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.266+0000 7f540be00700 1 --2- 172.21.15.43:0/320956184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f540c10b890 0x7f540c1131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.266+0000 7f540be00700 1 --2- 172.21.15.43:0/320956184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f540c106610 0x7f540c106a20 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.266+0000 7f540be00700 1 --2- 172.21.15.43:0/320956184 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c10af40 0x7f540c10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.266+0000 7f540be00700 1 -- 172.21.15.43:0/320956184 >> 172.21.15.43:0/320956184 conn(0x7f540c102020 msgr2=0x7f540c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.266+0000 7f540be00700 1 -- 172.21.15.43:0/320956184 shutdown_connections 2024-05-27T07:23:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.266+0000 7f540be00700 1 -- 172.21.15.43:0/320956184 wait complete. 2024-05-27T07:23:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.267+0000 7f540be00700 1 Processor -- start 2024-05-27T07:23:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.268+0000 7f540be00700 1 -- start start 2024-05-27T07:23:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.268+0000 7f540be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c106610 0x7f540c19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.268+0000 7f540be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f540c10af40 0x7f540c19d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.268+0000 7f540be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f540c10b890 0x7f540c1a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.268+0000 7f540be00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f540c1163c0 con 0x7f540c10af40 2024-05-27T07:23:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.268+0000 7f540be00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f540c116240 con 0x7f540c10b890 2024-05-27T07:23:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.268+0000 7f540be00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f540c116540 con 0x7f540c106610 2024-05-27T07:23:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.269+0000 7f540a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f540c10af40 0x7f540c19d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.269+0000 7f540a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f540c10af40 0x7f540c19d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59090/0 (socket says 172.21.15.43:59090) 2024-05-27T07:23:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.269+0000 7f540a000700 1 -- 172.21.15.43:0/2193354919 learned_addr learned my addr 172.21.15.43:0/2193354919 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.269+0000 7f540aa00700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c106610 0x7f540c19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.269+0000 7f540b400700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f540c10b890 0x7f540c1a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.269+0000 7f540a000700 1 -- 172.21.15.43:0/2193354919 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c106610 msgr2=0x7f540c19cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.269+0000 7f540a000700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c106610 0x7f540c19cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.269+0000 7f540a000700 1 -- 172.21.15.43:0/2193354919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f540c10b890 msgr2=0x7f540c1a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.269+0000 7f540a000700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f540c10b890 0x7f540c1a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.269+0000 7f540a000700 1 -- 172.21.15.43:0/2193354919 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540c1a1c00 con 0x7f540c10af40 2024-05-27T07:23:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.270+0000 7f540a000700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f540c10af40 0x7f540c19d2d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f53f800d980 tx=0x7f53f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.270+0000 7f5403400700 1 -- 172.21.15.43:0/2193354919 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8017070 con 0x7f540c10af40 2024-05-27T07:23:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.270+0000 7f5403400700 1 -- 172.21.15.43:0/2193354919 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53f800fd20 con 0x7f540c10af40 2024-05-27T07:23:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.270+0000 7f5403400700 1 -- 172.21.15.43:0/2193354919 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f800e3b0 con 0x7f540c10af40 2024-05-27T07:23:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.270+0000 7f540be00700 1 -- 172.21.15.43:0/2193354919 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f540c1a1ef0 con 0x7f540c10af40 2024-05-27T07:23:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.270+0000 7f540be00700 1 -- 172.21.15.43:0/2193354919 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f540c1a9810 con 0x7f540c10af40 2024-05-27T07:23:34.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.277+0000 7f5420375b80 1 Processor -- start 2024-05-27T07:23:34.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.278+0000 7f5420375b80 1 -- start start 2024-05-27T07:23:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.278+0000 7f5403400700 1 -- 172.21.15.43:0/2193354919 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f53f800f890 con 0x7f540c10af40 2024-05-27T07:23:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.278+0000 7f5420375b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad8ed5e40 0x55cad9053ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.278+0000 7f5420375b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad905ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.278+0000 7f5420375b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad9054820 0x55cad9061020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.278+0000 7f5420375b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cad8f82e40 con 0x55cad9054410 2024-05-27T07:23:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.278+0000 7f5420375b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cad8f729e0 con 0x55cad8ed5e40 2024-05-27T07:23:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.278+0000 7f5420375b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55cad8f76a20 con 0x55cad9054820 2024-05-27T07:23:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.279+0000 7f540aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad905ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.279+0000 7f540aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad905ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59106/0 (socket says 172.21.15.43:59106) 2024-05-27T07:23:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.279+0000 7f540aa00700 1 -- 172.21.15.43:0/1431105835 learned_addr learned my addr 172.21.15.43:0/1431105835 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.279+0000 7f540a000700 1 --2- 172.21.15.43:0/1431105835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad9054820 0x55cad9061020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.279+0000 7f540b400700 1 --2- 172.21.15.43:0/1431105835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad8ed5e40 0x55cad9053ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.279+0000 7f540aa00700 1 -- 172.21.15.43:0/1431105835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad9054820 msgr2=0x55cad9061020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:34.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.279+0000 7f540aa00700 1 --2- 172.21.15.43:0/1431105835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad9054820 0x55cad9061020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.279+0000 7f540aa00700 1 -- 172.21.15.43:0/1431105835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad8ed5e40 msgr2=0x55cad9053ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:34.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.279+0000 7f540aa00700 1 --2- 172.21.15.43:0/1431105835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad8ed5e40 0x55cad9053ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.279+0000 7f540aa00700 1 -- 172.21.15.43:0/1431105835 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cad8f3b420 con 0x55cad9054410 2024-05-27T07:23:34.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.280+0000 7f540aa00700 1 --2- 172.21.15.43:0/1431105835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad905ead0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f53fc015000 tx=0x7f53fc005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985f772c6596a034 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.280+0000 7f53f7e00700 1 -- 172.21.15.43:0/1431105835 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc027070 con 0x55cad9054410 2024-05-27T07:23:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.280+0000 7f53f7e00700 1 -- 172.21.15.43:0/1431105835 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53fc0037c0 con 0x55cad9054410 2024-05-27T07:23:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.280+0000 7f53f7e00700 1 -- 172.21.15.43:0/1431105835 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc00b9f0 con 0x55cad9054410 2024-05-27T07:23:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.281+0000 7f5403400700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f53f00b2a70 0x7f53f00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.281+0000 7f5403400700 1 -- 172.21.15.43:0/2193354919 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f53f80e2400 con 0x7f540c10af40 2024-05-27T07:23:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.281+0000 7f540b400700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f53f00b2a70 0x7f53f00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.281+0000 7f5420375b80 1 -- 172.21.15.43:0/1431105835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 msgr2=0x55cad905ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.281+0000 7f5420375b80 1 --2- 172.21.15.43:0/1431105835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad905ead0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f53fc015000 tx=0x7f53fc005040 comp rx=0 tx=0).stop 2024-05-27T07:23:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.282+0000 7f5420375b80 1 -- 172.21.15.43:0/1431105835 shutdown_connections 2024-05-27T07:23:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.282+0000 7f5420375b80 1 --2- 172.21.15.43:0/1431105835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad9054820 0x55cad9061020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.282+0000 7f5420375b80 1 --2- 172.21.15.43:0/1431105835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad8ed5e40 0x55cad9053ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.282+0000 7f5420375b80 1 --2- 172.21.15.43:0/1431105835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad905ead0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.282+0000 7f5420375b80 1 -- 172.21.15.43:0/1431105835 >> 172.21.15.43:0/1431105835 conn(0x55cad8f35fc0 msgr2=0x55cad904fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.282+0000 7f5420375b80 1 -- 172.21.15.43:0/1431105835 shutdown_connections 2024-05-27T07:23:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.282+0000 7f540b400700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f53f00b2a70 0x7f53f00b4f80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55cad9050690 tx=0x7f540400b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.282+0000 7f5420375b80 1 -- 172.21.15.43:0/1431105835 wait complete. 2024-05-27T07:23:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.283+0000 7f5420375b80 1 -- start start 2024-05-27T07:23:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.283+0000 7f5420375b80 1 Processor -- start 2024-05-27T07:23:34.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f5420375b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad8ed5e40 0x55cad905e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f5420375b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad9054410 0x55cad905ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f5420375b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054820 0x55cad905ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f5420375b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cad8e78fa0 con 0x55cad9054820 2024-05-27T07:23:34.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f5420375b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cad8f13760 con 0x55cad8ed5e40 2024-05-27T07:23:34.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f5420375b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55cad9061930 con 0x55cad9054410 2024-05-27T07:23:34.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f5420375b80 10 client.0 _ll_register_callbacks cb 0x55cad8f1cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:34.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f540b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054820 0x55cad905ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f540b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054820 0x55cad905ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59110/0 (socket says 172.21.15.43:59110) 2024-05-27T07:23:34.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f540b400700 1 -- 172.21.15.43:0/2660106294 learned_addr learned my addr 172.21.15.43:0/2660106294 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:34.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f540a000700 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad9054410 0x55cad905ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.284+0000 7f540aa00700 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad8ed5e40 0x55cad905e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.285+0000 7f540b400700 1 -- 172.21.15.43:0/2660106294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad9054410 msgr2=0x55cad905ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.285+0000 7f540b400700 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad9054410 0x55cad905ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.285+0000 7f540b400700 1 -- 172.21.15.43:0/2660106294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad8ed5e40 msgr2=0x55cad905e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.285+0000 7f540b400700 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad8ed5e40 0x55cad905e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.285+0000 7f540b400700 1 -- 172.21.15.43:0/2660106294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cad8ec28d0 con 0x55cad9054820 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.285+0000 7f540b400700 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054820 0x55cad905ef80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5404016df0 tx=0x7f5404004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91be725615916fe6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.285+0000 7f53f4c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.285+0000 7f53f4c00700 1 -- 172.21.15.43:0/2660106294 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5404026070 con 0x55cad9054820 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.286+0000 7f5420375b80 1 -- 172.21.15.43:0/2660106294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53fc0037c0 con 0x55cad9054820 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.286+0000 7f53f4c00700 1 -- 172.21.15.43:0/2660106294 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f5404009020 con 0x55cad9054820 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.286+0000 7f53f4c00700 1 -- 172.21.15.43:0/2660106294 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f5404033050 con 0x55cad9054820 2024-05-27T07:23:34.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.286+0000 7f5420375b80 10 client.16863 fetch_fsmap learned FSMap version 102 2024-05-27T07:23:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.286+0000 7f5420375b80 1 -- 172.21.15.43:0/2660106294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55cad8f43060 con 0x55cad9054820 2024-05-27T07:23:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.286+0000 7f53f4c00700 1 -- 172.21.15.43:0/2660106294 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53fc0037c0 con 0x55cad9054820 2024-05-27T07:23:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.287+0000 7f53f4c00700 1 -- 172.21.15.43:0/2660106294 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 102) ==== 4788+0+0 (secure 0 0 0) 0x7f5404008ca0 con 0x55cad9054820 2024-05-27T07:23:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.287+0000 7f5420375b80 10 client.16863 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:23:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.287+0000 7f5420375b80 10 client.16863 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:23:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.287+0000 7f5420375b80 20 client.16863 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.287+0000 7f5420375b80 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55cad906ec80 0x55cad9071110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.287+0000 7f5420375b80 4 client.16863 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.287+0000 7f5420375b80 1 -- 172.21.15.43:0/2660106294 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cad8f84d90 con 0x55cad906ec80 2024-05-27T07:23:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.287+0000 7f540aa00700 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55cad906ec80 0x55cad9071110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.288+0000 7f540aa00700 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55cad906ec80 0x55cad9071110 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=291087cf5446dc5 server_cookie=9d94bd0129b8d41e in_seq=0 out_seq=0 2024-05-27T07:23:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.288+0000 7f53f4c00700 10 client.16863 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:23:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.290+0000 7f53f4c00700 1 -- 172.21.15.43:0/2660106294 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cad8f84d90 con 0x55cad906ec80 2024-05-27T07:23:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.290+0000 7f53f4c00700 10 client.16863 handle_command_reply: tid=0 2024-05-27T07:23:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.290+0000 7f5420375b80 1 client.16863 shutdown 2024-05-27T07:23:34.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 1 -- 172.21.15.43:0/2660106294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054820 msgr2=0x55cad905ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054820 0x55cad905ef80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5404016df0 tx=0x7f5404004b40 comp rx=0 tx=0).stop 2024-05-27T07:23:34.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 1 -- 172.21.15.43:0/2660106294 shutdown_connections 2024-05-27T07:23:34.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55cad906ec80 0x55cad9071110 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054820 0x55cad905ef80 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad8ed5e40 0x55cad905e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f53f4c00700 0 client.16863 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:23:34.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 1 --2- 172.21.15.43:0/2660106294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad9054410 0x55cad905ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 1 -- 172.21.15.43:0/2660106294 >> 172.21.15.43:0/2660106294 conn(0x55cad8f35fc0 msgr2=0x55cad904eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 1 -- 172.21.15.43:0/2660106294 shutdown_connections 2024-05-27T07:23:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 1 -- 172.21.15.43:0/2660106294 wait complete. 2024-05-27T07:23:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.291+0000 7f5420375b80 20 client.16863 trim_cache size 0 max 16384 2024-05-27T07:23:34.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.310+0000 7f5420375b80 1 Processor -- start 2024-05-27T07:23:34.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.310+0000 7f5420375b80 1 -- start start 2024-05-27T07:23:34.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f5420375b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad8ed5e40 0x55cad9059cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f5420375b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad906aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f5420375b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad9054820 0x55cad9076910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f5420375b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f540c1b31c0 con 0x55cad9054410 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f5420375b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f540c1b0f50 con 0x55cad9054820 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f5420375b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f540c1b2af0 con 0x55cad8ed5e40 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f540a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad906aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f540a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad906aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59116/0 (socket says 172.21.15.43:59116) 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f540a000700 1 -- 172.21.15.43:0/39432731 learned_addr learned my addr 172.21.15.43:0/39432731 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f540aa00700 1 --2- 172.21.15.43:0/39432731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad8ed5e40 0x55cad9059cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.311+0000 7f540b400700 1 --2- 172.21.15.43:0/39432731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad9054820 0x55cad9076910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.312+0000 7f540a000700 1 -- 172.21.15.43:0/39432731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad8ed5e40 msgr2=0x55cad9059cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.312+0000 7f540a000700 1 --2- 172.21.15.43:0/39432731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad8ed5e40 0x55cad9059cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.312+0000 7f540a000700 1 -- 172.21.15.43:0/39432731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad9054820 msgr2=0x55cad9076910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.312+0000 7f540a000700 1 --2- 172.21.15.43:0/39432731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad9054820 0x55cad9076910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.312+0000 7f540a000700 1 -- 172.21.15.43:0/39432731 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cad9051350 con 0x55cad9054410 2024-05-27T07:23:34.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.312+0000 7f540a000700 1 --2- 172.21.15.43:0/39432731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad906aa30 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f53f80df7e0 tx=0x7f53f800b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ba93c84b911a15 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:34.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.312+0000 7f53f7400700 1 -- 172.21.15.43:0/39432731 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8020040 con 0x55cad9054410 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.312+0000 7f53f7400700 1 -- 172.21.15.43:0/39432731 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53f80a8e20 con 0x55cad9054410 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.312+0000 7f53f7400700 1 -- 172.21.15.43:0/39432731 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f80a9310 con 0x55cad9054410 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.313+0000 7f5420375b80 1 -- 172.21.15.43:0/39432731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 msgr2=0x55cad906aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.313+0000 7f5420375b80 1 --2- 172.21.15.43:0/39432731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad906aa30 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f53f80df7e0 tx=0x7f53f800b510 comp rx=0 tx=0).stop 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.313+0000 7f5420375b80 1 -- 172.21.15.43:0/39432731 shutdown_connections 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.313+0000 7f5420375b80 1 --2- 172.21.15.43:0/39432731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad9054820 0x55cad9076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.313+0000 7f5420375b80 1 --2- 172.21.15.43:0/39432731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad8ed5e40 0x55cad9059cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.313+0000 7f5420375b80 1 --2- 172.21.15.43:0/39432731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad906aa30 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.313+0000 7f5420375b80 1 -- 172.21.15.43:0/39432731 >> 172.21.15.43:0/39432731 conn(0x55cad8f35fc0 msgr2=0x55cad905cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.314+0000 7f5420375b80 1 -- 172.21.15.43:0/39432731 shutdown_connections 2024-05-27T07:23:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.314+0000 7f5420375b80 1 -- 172.21.15.43:0/39432731 wait complete. 2024-05-27T07:23:34.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.314+0000 7f5420375b80 1 -- start start 2024-05-27T07:23:34.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.314+0000 7f5420375b80 1 Processor -- start 2024-05-27T07:23:34.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.315+0000 7f5420375b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad8ed5e40 0x55cad90623e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.315+0000 7f5420375b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad9062980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.315+0000 7f5420375b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad9054820 0x55cad90d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.315+0000 7f5420375b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f540c1b1cf0 con 0x55cad9054410 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.315+0000 7f5420375b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f540c1b1620 con 0x55cad9054820 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.315+0000 7f5420375b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f540c1b2420 con 0x55cad8ed5e40 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.315+0000 7f5420375b80 10 client.0 _ll_register_callbacks cb 0x55cad8f1cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.315+0000 7f540a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad9062980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.315+0000 7f540a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad9062980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59124/0 (socket says 172.21.15.43:59124) 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.315+0000 7f540a000700 1 -- 172.21.15.43:0/3461393982 learned_addr learned my addr 172.21.15.43:0/3461393982 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.316+0000 7f540aa00700 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad8ed5e40 0x55cad90623e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.316+0000 7f540b400700 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad9054820 0x55cad90d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.316+0000 7f540a000700 1 -- 172.21.15.43:0/3461393982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad8ed5e40 msgr2=0x55cad90623e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.316+0000 7f540a000700 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad8ed5e40 0x55cad90623e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.316+0000 7f540a000700 1 -- 172.21.15.43:0/3461393982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad9054820 msgr2=0x55cad90d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.316+0000 7f540a000700 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad9054820 0x55cad90d17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.316+0000 7f540a000700 1 -- 172.21.15.43:0/3461393982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cad905f2f0 con 0x55cad9054410 2024-05-27T07:23:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.316+0000 7f540a000700 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad9062980 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f53f80df830 tx=0x7f53f8010820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814ba8c1173bdaf5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.316+0000 7f53f4c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.316+0000 7f53f4c00700 1 -- 172.21.15.43:0/3461393982 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8020070 con 0x55cad9054410 2024-05-27T07:23:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f5420375b80 1 -- 172.21.15.43:0/3461393982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53f80a8e20 con 0x55cad9054410 2024-05-27T07:23:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f53f4c00700 1 -- 172.21.15.43:0/3461393982 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f53f80a9780 con 0x55cad9054410 2024-05-27T07:23:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f53f4c00700 1 -- 172.21.15.43:0/3461393982 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f53f80a9a10 con 0x55cad9054410 2024-05-27T07:23:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f5420375b80 10 client.16869 fetch_fsmap learned FSMap version 102 2024-05-27T07:23:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f5420375b80 1 -- 172.21.15.43:0/3461393982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55cad90d8ef0 con 0x55cad9054410 2024-05-27T07:23:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f53f4c00700 1 -- 172.21.15.43:0/3461393982 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53f80a8e20 con 0x55cad9054410 2024-05-27T07:23:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f53f4c00700 1 -- 172.21.15.43:0/3461393982 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 102) ==== 4788+0+0 (secure 0 0 0) 0x7f53f80a4c80 con 0x55cad9054410 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f5420375b80 10 client.16869 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f5420375b80 10 client.16869 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f5420375b80 20 client.16869 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f5420375b80 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55cad90d9560 0x55cad90db980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f5420375b80 4 client.16869 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.317+0000 7f5420375b80 1 -- 172.21.15.43:0/3461393982 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cad8f7e570 con 0x55cad90d9560 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.318+0000 7f540aa00700 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55cad90d9560 0x55cad90db980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.319+0000 7f540aa00700 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55cad90d9560 0x55cad90db980 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d1efa31d4fc0a6dc server_cookie=f7b8426b654405cf in_seq=0 out_seq=0 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.319+0000 7f53f4c00700 10 client.16869 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.319+0000 7f53f4c00700 1 -- 172.21.15.43:0/3461393982 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cad8f7e570 con 0x55cad90d9560 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.319+0000 7f53f4c00700 10 client.16869 handle_command_reply: tid=0 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.319+0000 7f5420375b80 1 client.16869 shutdown 2024-05-27T07:23:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 1 -- 172.21.15.43:0/3461393982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 msgr2=0x55cad9062980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad9062980 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f53f80df830 tx=0x7f53f8010820 comp rx=0 tx=0).stop 2024-05-27T07:23:34.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 1 -- 172.21.15.43:0/3461393982 shutdown_connections 2024-05-27T07:23:34.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55cad90d9560 0x55cad90db980 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cad9054820 0x55cad90d17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cad8ed5e40 0x55cad90623e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 1 --2- 172.21.15.43:0/3461393982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cad9054410 0x55cad9062980 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f53f4c00700 0 client.16869 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:23:34.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 1 -- 172.21.15.43:0/3461393982 >> 172.21.15.43:0/3461393982 conn(0x55cad8f35fc0 msgr2=0x55cad905cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:34.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 1 -- 172.21.15.43:0/3461393982 shutdown_connections 2024-05-27T07:23:34.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 1 -- 172.21.15.43:0/3461393982 wait complete. 2024-05-27T07:23:34.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.320+0000 7f5420375b80 20 client.16869 trim_cache size 0 max 16384 2024-05-27T07:23:34.313 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:23:34.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 -- 172.21.15.43:0/2193354919 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f53f00b2a70 msgr2=0x7f53f00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f53f00b2a70 0x7f53f00b4f80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55cad9050690 tx=0x7f540400b040 comp rx=0 tx=0).stop 2024-05-27T07:23:34.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 -- 172.21.15.43:0/2193354919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f540c10af40 msgr2=0x7f540c19d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:34.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f540c10af40 0x7f540c19d2d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f53f800d980 tx=0x7f53f800dc90 comp rx=0 tx=0).stop 2024-05-27T07:23:34.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 -- 172.21.15.43:0/2193354919 shutdown_connections 2024-05-27T07:23:34.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f53f00b2a70 0x7f53f00b4f80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f540c10b890 0x7f540c1a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c106610 0x7f540c19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 --2- 172.21.15.43:0/2193354919 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f540c10af40 0x7f540c19d2d0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:34.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 -- 172.21.15.43:0/2193354919 >> 172.21.15.43:0/2193354919 conn(0x7f540c102020 msgr2=0x7f540c10a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:34.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.321+0000 7f540be00700 1 -- 172.21.15.43:0/2193354919 shutdown_connections 2024-05-27T07:23:34.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:34.322+0000 7f540be00700 1 -- 172.21.15.43:0/2193354919 wait complete. 2024-05-27T07:23:34.700 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:34 smithi043 ceph-mon[31531]: pgmap v328: 129 pgs: 129 active+clean; 93 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 374 KiB/s wr, 19 op/s 2024-05-27T07:23:34.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:34 smithi179 ceph-mon[35888]: pgmap v328: 129 pgs: 129 active+clean; 93 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 374 KiB/s wr, 19 op/s 2024-05-27T07:23:34.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:34 smithi132 ceph-mon[35283]: pgmap v328: 129 pgs: 129 active+clean; 93 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 374 KiB/s wr, 19 op/s 2024-05-27T07:23:35.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:23:35.222 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53268 0.62 MB/sec execute 339 sec latency 50.018 ms 2024-05-27T07:23:35.687 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:36.222 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53399 0.61 MB/sec execute 340 sec latency 48.533 ms 2024-05-27T07:23:36.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:36 smithi043 ceph-mon[31531]: pgmap v329: 129 pgs: 129 active+clean; 93 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 374 KiB/s wr, 19 op/s 2024-05-27T07:23:36.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:36 smithi179 ceph-mon[35888]: pgmap v329: 129 pgs: 129 active+clean; 93 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 374 KiB/s wr, 19 op/s 2024-05-27T07:23:36.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:36 smithi132 ceph-mon[35283]: pgmap v329: 129 pgs: 129 active+clean; 93 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 374 KiB/s wr, 19 op/s 2024-05-27T07:23:37.222 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53486 0.62 MB/sec execute 341 sec latency 51.662 ms 2024-05-27T07:23:38.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.016+0000 7f9016600700 1 -- 172.21.15.43:0/3595195840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f901010f6b0 msgr2=0x7f901010fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.016+0000 7f9016600700 1 --2- 172.21.15.43:0/3595195840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f901010f6b0 0x7f901010fac0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9004009990 tx=0x7f9004009ca0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.016+0000 7f9016600700 1 -- 172.21.15.43:0/3595195840 shutdown_connections 2024-05-27T07:23:38.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.016+0000 7f9016600700 1 --2- 172.21.15.43:0/3595195840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f901010b100 0x7f9010117610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.016+0000 7f9016600700 1 --2- 172.21.15.43:0/3595195840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f901010f6b0 0x7f901010fac0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.016+0000 7f9016600700 1 --2- 172.21.15.43:0/3595195840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f901010a7b0 0x7f901010abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.016+0000 7f9016600700 1 -- 172.21.15.43:0/3595195840 >> 172.21.15.43:0/3595195840 conn(0x7f9010076af0 msgr2=0x7f9010076f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:38.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.016+0000 7f9016600700 1 -- 172.21.15.43:0/3595195840 shutdown_connections 2024-05-27T07:23:38.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.017+0000 7f9016600700 1 -- 172.21.15.43:0/3595195840 wait complete. 2024-05-27T07:23:38.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.018+0000 7f9016600700 1 Processor -- start 2024-05-27T07:23:38.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.018+0000 7f9016600700 1 -- start start 2024-05-27T07:23:38.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.019+0000 7f9016600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f901010a7b0 0x7f90101a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.019+0000 7f9016600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f901010b100 0x7f90101a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.019+0000 7f9016600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90101a5a50 0x7f90101a5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.019+0000 7f9016600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f901011a750 con 0x7f901010b100 2024-05-27T07:23:38.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.019+0000 7f9016600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f901011a5d0 con 0x7f901010a7b0 2024-05-27T07:23:38.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.019+0000 7f9016600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f901011a8d0 con 0x7f90101a5a50 2024-05-27T07:23:38.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.019+0000 7f900fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f901010b100 0x7f90101a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.019+0000 7f900fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f901010b100 0x7f90101a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41592/0 (socket says 172.21.15.43:41592) 2024-05-27T07:23:38.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.019+0000 7f900fe00700 1 -- 172.21.15.43:0/32259158 learned_addr learned my addr 172.21.15.43:0/32259158 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:38.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.020+0000 7f9015200700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f901010a7b0 0x7f90101a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.020+0000 7f9015c00700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90101a5a50 0x7f90101a5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.020+0000 7f900fe00700 1 -- 172.21.15.43:0/32259158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90101a5a50 msgr2=0x7f90101a5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.020+0000 7f900fe00700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90101a5a50 0x7f90101a5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.020+0000 7f900fe00700 1 -- 172.21.15.43:0/32259158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f901010a7b0 msgr2=0x7f90101a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.020+0000 7f900fe00700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f901010a7b0 0x7f90101a12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.020+0000 7f900fe00700 1 -- 172.21.15.43:0/32259158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90101a6510 con 0x7f901010b100 2024-05-27T07:23:38.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.020+0000 7f900fe00700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f901010b100 0x7f90101a1880 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00c9b0 tx=0x7f8ffc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:38.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.021+0000 7f900d600700 1 -- 172.21.15.43:0/32259158 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc007b00 con 0x7f901010b100 2024-05-27T07:23:38.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.021+0000 7f900d600700 1 -- 172.21.15.43:0/32259158 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ffc007ca0 con 0x7f901010b100 2024-05-27T07:23:38.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.021+0000 7f9016600700 1 -- 172.21.15.43:0/32259158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90101a6800 con 0x7f901010b100 2024-05-27T07:23:38.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.021+0000 7f900d600700 1 -- 172.21.15.43:0/32259158 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc018660 con 0x7f901010b100 2024-05-27T07:23:38.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.021+0000 7f9016600700 1 -- 172.21.15.43:0/32259158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90101adfe0 con 0x7f901010b100 2024-05-27T07:23:38.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.025+0000 7f900d600700 1 -- 172.21.15.43:0/32259158 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f8ffc00f450 con 0x7f901010b100 2024-05-27T07:23:38.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.027+0000 7f9026683b80 1 Processor -- start 2024-05-27T07:23:38.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.027+0000 7f9026683b80 1 -- start start 2024-05-27T07:23:38.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f9026683b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 0x55921229d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f9026683b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55921211de40 0x55921229dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f900d600700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8ff80b29f0 0x7f8ff80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f9026683b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122a72c0 0x5592122a96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f9026683b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5592121cae40 con 0x55921211de40 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f9026683b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5592121ba9e0 con 0x5592122099f0 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f9026683b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5592121bea20 con 0x5592122a72c0 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f900d600700 1 -- 172.21.15.43:0/32259158 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f8ffc0e2a30 con 0x7f901010b100 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f900fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55921211de40 0x55921229dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f900fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55921211de40 0x55921229dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41602/0 (socket says 172.21.15.43:41602) 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f900fe00700 1 -- 172.21.15.43:0/251234194 learned_addr learned my addr 172.21.15.43:0/251234194 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f9015c00700 1 --2- 172.21.15.43:0/251234194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 0x55921229d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.028+0000 7f9015200700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8ff80b29f0 0x7f8ff80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.029+0000 7f9015c00700 1 --2- 172.21.15.43:0/251234194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122a72c0 0x5592122a96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.029+0000 7f900fe00700 1 -- 172.21.15.43:0/251234194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122a72c0 msgr2=0x5592122a96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:38.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.029+0000 7f900fe00700 1 --2- 172.21.15.43:0/251234194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122a72c0 0x5592122a96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.029+0000 7f900fe00700 1 -- 172.21.15.43:0/251234194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 msgr2=0x55921229d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:38.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.029+0000 7f900fe00700 1 --2- 172.21.15.43:0/251234194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 0x55921229d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.029+0000 7f900fe00700 1 -- 172.21.15.43:0/251234194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559212183420 con 0x55921211de40 2024-05-27T07:23:38.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.029+0000 7f9015c00700 1 --2- 172.21.15.43:0/251234194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122a72c0 0x5592122a96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:23:38.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.029+0000 7f900fe00700 1 --2- 172.21.15.43:0/251234194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55921211de40 0x55921229dc90 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0a5960 tx=0x7f8ffc0a9150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9fda11fecc9c2a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:38.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.029+0000 7f9015200700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8ff80b29f0 0x7f8ff80b4f00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f90040059d0 tx=0x7f9004005960 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:38.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.030+0000 7f9002000700 1 -- 172.21.15.43:0/251234194 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc0a5070 con 0x55921211de40 2024-05-27T07:23:38.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.030+0000 7f9002000700 1 -- 172.21.15.43:0/251234194 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ffc0b2080 con 0x55921211de40 2024-05-27T07:23:38.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.030+0000 7f9002000700 1 -- 172.21.15.43:0/251234194 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc0adc90 con 0x55921211de40 2024-05-27T07:23:38.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.031+0000 7f9026683b80 1 -- 172.21.15.43:0/251234194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55921211de40 msgr2=0x55921229dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.031+0000 7f9026683b80 1 --2- 172.21.15.43:0/251234194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55921211de40 0x55921229dc90 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0a5960 tx=0x7f8ffc0a9150 comp rx=0 tx=0).stop 2024-05-27T07:23:38.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.031+0000 7f9026683b80 1 -- 172.21.15.43:0/251234194 shutdown_connections 2024-05-27T07:23:38.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.031+0000 7f9026683b80 1 --2- 172.21.15.43:0/251234194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122a72c0 0x5592122a96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.031+0000 7f9026683b80 1 --2- 172.21.15.43:0/251234194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 0x55921229d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.031+0000 7f9026683b80 1 --2- 172.21.15.43:0/251234194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55921211de40 0x55921229dc90 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.031+0000 7f9026683b80 1 -- 172.21.15.43:0/251234194 >> 172.21.15.43:0/251234194 conn(0x5592121b7390 msgr2=0x559212298710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:38.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.031+0000 7f9026683b80 1 -- 172.21.15.43:0/251234194 shutdown_connections 2024-05-27T07:23:38.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.031+0000 7f9026683b80 1 -- 172.21.15.43:0/251234194 wait complete. 2024-05-27T07:23:38.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.032+0000 7f9026683b80 1 -- start start 2024-05-27T07:23:38.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.032+0000 7f9026683b80 1 Processor -- start 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.033+0000 7f9026683b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 0x5592122a4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.033+0000 7f9026683b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122099f0 0x5592122a47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.033+0000 7f9026683b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122a72c0 0x5592122a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.033+0000 7f9026683b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5592120c0fa0 con 0x5592122099f0 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.033+0000 7f9026683b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55921215b760 con 0x5592122a72c0 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.033+0000 7f9026683b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5592122a9da0 con 0x55921211de40 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.033+0000 7f9026683b80 10 client.0 _ll_register_callbacks cb 0x559212164c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.033+0000 7f900fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122099f0 0x5592122a47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.033+0000 7f900fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122099f0 0x5592122a47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41612/0 (socket says 172.21.15.43:41612) 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.033+0000 7f900fe00700 1 -- 172.21.15.43:0/1176233201 learned_addr learned my addr 172.21.15.43:0/1176233201 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f9015c00700 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122a72c0 0x5592122a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f9015200700 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 0x5592122a4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f900fe00700 1 -- 172.21.15.43:0/1176233201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 msgr2=0x5592122a4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f900fe00700 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 0x5592122a4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f900fe00700 1 -- 172.21.15.43:0/1176233201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122a72c0 msgr2=0x5592122a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f900fe00700 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122a72c0 0x5592122a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f900fe00700 1 -- 172.21.15.43:0/1176233201 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55921210a8d0 con 0x5592122099f0 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f9015200700 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 0x5592122a4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f900fe00700 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122099f0 0x5592122a47c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0a9870 tx=0x7f8ffc00fee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e48f6c9f77aa99 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f8ff6a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f8ff6a00700 1 -- 172.21.15.43:0/1176233201 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc0a5070 con 0x5592122099f0 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f9026683b80 1 -- 172.21.15.43:0/1176233201 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5592121ccd90 con 0x5592122099f0 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.034+0000 7f8ff6a00700 1 -- 172.21.15.43:0/1176233201 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f8ffc0a5ed0 con 0x5592122099f0 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.035+0000 7f8ff6a00700 1 -- 172.21.15.43:0/1176233201 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f8ffc0a9fe0 con 0x5592122099f0 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.035+0000 7f9026683b80 10 client.16881 fetch_fsmap learned FSMap version 102 2024-05-27T07:23:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.035+0000 7f9026683b80 1 -- 172.21.15.43:0/1176233201 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55921218b060 con 0x5592122099f0 2024-05-27T07:23:38.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.035+0000 7f8ff6a00700 1 -- 172.21.15.43:0/1176233201 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x5592121ccd90 con 0x5592122099f0 2024-05-27T07:23:38.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.035+0000 7f8ff6a00700 1 -- 172.21.15.43:0/1176233201 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 102) ==== 4788+0+0 (secure 0 0 0) 0x7f8ffc00f7d0 con 0x5592122099f0 2024-05-27T07:23:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.035+0000 7f9026683b80 10 client.16881 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:23:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.035+0000 7f9026683b80 10 client.16881 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:23:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.036+0000 7f9026683b80 20 client.16881 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.036+0000 7f9026683b80 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5592122b6ba0 0x5592122b9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.036+0000 7f9026683b80 4 client.16881 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.036+0000 7f9026683b80 1 -- 172.21.15.43:0/1176233201 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592121c6570 con 0x5592122b6ba0 2024-05-27T07:23:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.036+0000 7f9015200700 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5592122b6ba0 0x5592122b9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.038+0000 7f9015200700 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5592122b6ba0 0x5592122b9020 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5674c59d3ee2174f server_cookie=c027af5c5a381a2f in_seq=0 out_seq=0 2024-05-27T07:23:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.038+0000 7f8ff6a00700 10 client.16881 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:23:38.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.039+0000 7f8ff6a00700 1 -- 172.21.15.43:0/1176233201 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5592121c6570 con 0x5592122b6ba0 2024-05-27T07:23:38.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.039+0000 7f8ff6a00700 10 client.16881 handle_command_reply: tid=0 2024-05-27T07:23:38.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 client.16881 shutdown 2024-05-27T07:23:38.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 -- 172.21.15.43:0/1176233201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122099f0 msgr2=0x5592122a47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122099f0 0x5592122a47c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0a9870 tx=0x7f8ffc00fee0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 -- 172.21.15.43:0/1176233201 shutdown_connections 2024-05-27T07:23:38.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x5592122b6ba0 0x5592122b9020 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122a72c0 0x5592122a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 0x5592122a4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 --2- 172.21.15.43:0/1176233201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122099f0 0x5592122a47c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f8ff6a00700 0 client.16881 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:23:38.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 -- 172.21.15.43:0/1176233201 >> 172.21.15.43:0/1176233201 conn(0x5592121b7390 msgr2=0x559212298300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:38.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 -- 172.21.15.43:0/1176233201 shutdown_connections 2024-05-27T07:23:38.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 1 -- 172.21.15.43:0/1176233201 wait complete. 2024-05-27T07:23:38.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.040+0000 7f9026683b80 20 client.16881 trim_cache size 0 max 16384 2024-05-27T07:23:38.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.059+0000 7f9026683b80 1 Processor -- start 2024-05-27T07:23:38.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.059+0000 7f9026683b80 1 -- start start 2024-05-27T07:23:38.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.059+0000 7f9026683b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 0x559212298740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.059+0000 7f9026683b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 0x55921229f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.059+0000 7f9026683b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.059+0000 7f9026683b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f90101b7be0 con 0x5592122a72c0 2024-05-27T07:23:38.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.059+0000 7f9026683b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f90101b5970 con 0x5592122099f0 2024-05-27T07:23:38.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.059+0000 7f9026683b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f90101b7510 con 0x55921211de40 2024-05-27T07:23:38.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.060+0000 7f9015c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.060+0000 7f9015c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122b2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41618/0 (socket says 172.21.15.43:41618) 2024-05-27T07:23:38.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.060+0000 7f9015c00700 1 -- 172.21.15.43:0/405451681 learned_addr learned my addr 172.21.15.43:0/405451681 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:38.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.060+0000 7f9015200700 1 --2- 172.21.15.43:0/405451681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 0x559212298740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.060+0000 7f900fe00700 1 --2- 172.21.15.43:0/405451681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 0x55921229f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.061+0000 7f9015c00700 1 -- 172.21.15.43:0/405451681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 msgr2=0x559212298740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:38.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.061+0000 7f9015c00700 1 --2- 172.21.15.43:0/405451681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 0x559212298740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.061+0000 7f9015c00700 1 -- 172.21.15.43:0/405451681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 msgr2=0x55921229f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:38.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.061+0000 7f9015c00700 1 --2- 172.21.15.43:0/405451681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 0x55921229f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.061+0000 7f9015c00700 1 -- 172.21.15.43:0/405451681 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559212292930 con 0x5592122a72c0 2024-05-27T07:23:38.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.061+0000 7f900fe00700 1 --2- 172.21.15.43:0/405451681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 0x55921229f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:38.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.061+0000 7f9015c00700 1 --2- 172.21.15.43:0/405451681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122b2e40 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f900800a570 tx=0x7f9008004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57aaa136a7d31f6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:38.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.061+0000 7f9001600700 1 -- 172.21.15.43:0/405451681 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900801b020 con 0x5592122a72c0 2024-05-27T07:23:38.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.061+0000 7f9001600700 1 -- 172.21.15.43:0/405451681 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9008017af0 con 0x5592122a72c0 2024-05-27T07:23:38.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.062+0000 7f9001600700 1 -- 172.21.15.43:0/405451681 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9008008a10 con 0x5592122a72c0 2024-05-27T07:23:38.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.062+0000 7f9026683b80 1 -- 172.21.15.43:0/405451681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 msgr2=0x5592122b2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.062+0000 7f9026683b80 1 --2- 172.21.15.43:0/405451681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122b2e40 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f900800a570 tx=0x7f9008004b40 comp rx=0 tx=0).stop 2024-05-27T07:23:38.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.063+0000 7f9026683b80 1 -- 172.21.15.43:0/405451681 shutdown_connections 2024-05-27T07:23:38.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.063+0000 7f9026683b80 1 --2- 172.21.15.43:0/405451681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122b2e40 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.063+0000 7f9026683b80 1 --2- 172.21.15.43:0/405451681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55921211de40 0x559212298740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.063+0000 7f9026683b80 1 --2- 172.21.15.43:0/405451681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5592122099f0 0x55921229f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.063+0000 7f9026683b80 1 -- 172.21.15.43:0/405451681 >> 172.21.15.43:0/405451681 conn(0x5592121b7390 msgr2=0x55921229f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:38.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.063+0000 7f9026683b80 1 -- 172.21.15.43:0/405451681 shutdown_connections 2024-05-27T07:23:38.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.063+0000 7f9026683b80 1 -- 172.21.15.43:0/405451681 wait complete. 2024-05-27T07:23:38.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.063+0000 7f9026683b80 1 -- start start 2024-05-27T07:23:38.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.064+0000 7f9026683b80 1 Processor -- start 2024-05-27T07:23:38.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.064+0000 7f9026683b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55921211de40 0x5592122af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.064+0000 7f9026683b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122099f0 0x5592122a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.064+0000 7f9026683b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122aa590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.064+0000 7f9026683b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f90101b6710 con 0x5592122a72c0 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.064+0000 7f9026683b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f90101b6040 con 0x55921211de40 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.064+0000 7f9026683b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f90101b6e40 con 0x5592122099f0 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f9026683b80 10 client.0 _ll_register_callbacks cb 0x559212164c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f9015c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122aa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f9015c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122aa590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41628/0 (socket says 172.21.15.43:41628) 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f9015c00700 1 -- 172.21.15.43:0/3222233957 learned_addr learned my addr 172.21.15.43:0/3222233957 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f900fe00700 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122099f0 0x5592122a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f9015200700 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55921211de40 0x5592122af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f9015c00700 1 -- 172.21.15.43:0/3222233957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122099f0 msgr2=0x5592122a9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f9015c00700 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122099f0 0x5592122a9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f9015c00700 1 -- 172.21.15.43:0/3222233957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55921211de40 msgr2=0x5592122af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f9015c00700 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55921211de40 0x5592122af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.065+0000 7f9015c00700 1 -- 172.21.15.43:0/3222233957 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592122aad60 con 0x5592122a72c0 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.066+0000 7f9015c00700 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122aa590 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f900800a340 tx=0x7f90080084a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e120839cb6c2d9cc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.066+0000 7f8ff6a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.066+0000 7f8ff6a00700 1 -- 172.21.15.43:0/3222233957 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900801b030 con 0x5592122a72c0 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.066+0000 7f9026683b80 1 -- 172.21.15.43:0/3222233957 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9008017af0 con 0x5592122a72c0 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.066+0000 7f8ff6a00700 1 -- 172.21.15.43:0/3222233957 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f9008009b70 con 0x5592122a72c0 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f8ff6a00700 1 -- 172.21.15.43:0/3222233957 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f9008009d70 con 0x5592122a72c0 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f9026683b80 10 client.16887 fetch_fsmap learned FSMap version 102 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f9026683b80 1 -- 172.21.15.43:0/3222233957 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x5592123206a0 con 0x5592122a72c0 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f8ff6a00700 1 -- 172.21.15.43:0/3222233957 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9008017af0 con 0x5592122a72c0 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f8ff6a00700 1 -- 172.21.15.43:0/3222233957 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 102) ==== 4788+0+0 (secure 0 0 0) 0x7f9008029070 con 0x5592122a72c0 2024-05-27T07:23:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f9026683b80 10 client.16887 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:23:38.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f9026683b80 10 client.16887 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:23:38.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f9026683b80 20 client.16887 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:38.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f9026683b80 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559212320ca0 0x5592123230c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:38.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f9026683b80 4 client.16887 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:23:38.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.067+0000 7f9026683b80 1 -- 172.21.15.43:0/3222233957 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55921203d250 con 0x559212320ca0 2024-05-27T07:23:38.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.068+0000 7f9015200700 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559212320ca0 0x5592123230c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:38.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.070+0000 7f9015200700 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559212320ca0 0x5592123230c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=222f641521ea536f server_cookie=db0154f3fcdedbaa in_seq=0 out_seq=0 2024-05-27T07:23:38.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.070+0000 7f8ff6a00700 10 client.16887 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:23:38.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f8ff6a00700 1 -- 172.21.15.43:0/3222233957 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55921203d250 con 0x559212320ca0 2024-05-27T07:23:38.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f8ff6a00700 10 client.16887 handle_command_reply: tid=0 2024-05-27T07:23:38.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 client.16887 shutdown 2024-05-27T07:23:38.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 -- 172.21.15.43:0/3222233957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 msgr2=0x5592122aa590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122aa590 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f900800a340 tx=0x7f90080084a0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 -- 172.21.15.43:0/3222233957 shutdown_connections 2024-05-27T07:23:38.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592122a72c0 0x5592122aa590 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x559212320ca0 0x5592123230c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55921211de40 0x5592122af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 --2- 172.21.15.43:0/3222233957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5592122099f0 0x5592122a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f8ff6a00700 0 client.16887 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:23:38.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 -- 172.21.15.43:0/3222233957 >> 172.21.15.43:0/3222233957 conn(0x5592121b7390 msgr2=0x55921229c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:38.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 -- 172.21.15.43:0/3222233957 shutdown_connections 2024-05-27T07:23:38.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 1 -- 172.21.15.43:0/3222233957 wait complete. 2024-05-27T07:23:38.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.071+0000 7f9026683b80 20 client.16887 trim_cache size 0 max 16384 2024-05-27T07:23:38.064 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:23:38.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.072+0000 7f9016600700 1 -- 172.21.15.43:0/32259158 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8ff80b29f0 msgr2=0x7f8ff80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.072+0000 7f9016600700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8ff80b29f0 0x7f8ff80b4f00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f90040059d0 tx=0x7f9004005960 comp rx=0 tx=0).stop 2024-05-27T07:23:38.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.072+0000 7f9016600700 1 -- 172.21.15.43:0/32259158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f901010b100 msgr2=0x7f90101a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:38.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.072+0000 7f9016600700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f901010b100 0x7f90101a1880 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00c9b0 tx=0x7f8ffc00ccc0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.073+0000 7f9016600700 1 -- 172.21.15.43:0/32259158 shutdown_connections 2024-05-27T07:23:38.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.073+0000 7f9016600700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8ff80b29f0 0x7f8ff80b4f00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.073+0000 7f9016600700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90101a5a50 0x7f90101a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.073+0000 7f9016600700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f901010a7b0 0x7f90101a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.073+0000 7f9016600700 1 --2- 172.21.15.43:0/32259158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f901010b100 0x7f90101a1880 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:38.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.073+0000 7f9016600700 1 -- 172.21.15.43:0/32259158 >> 172.21.15.43:0/32259158 conn(0x7f9010076af0 msgr2=0x7f9010108a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:38.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.073+0000 7f9016600700 1 -- 172.21.15.43:0/32259158 shutdown_connections 2024-05-27T07:23:38.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:38.073+0000 7f9016600700 1 -- 172.21.15.43:0/32259158 wait complete. 2024-05-27T07:23:38.222 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53548 0.61 MB/sec execute 342 sec latency 42.998 ms 2024-05-27T07:23:38.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:38 smithi043 ceph-mon[31531]: pgmap v330: 129 pgs: 129 active+clean; 94 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 555 KiB/s wr, 34 op/s 2024-05-27T07:23:38.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:38 smithi179 ceph-mon[35888]: pgmap v330: 129 pgs: 129 active+clean; 94 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 555 KiB/s wr, 34 op/s 2024-05-27T07:23:38.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:38 smithi132 ceph-mon[35283]: pgmap v330: 129 pgs: 129 active+clean; 94 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 555 KiB/s wr, 34 op/s 2024-05-27T07:23:38.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:23:39.222 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53617 0.61 MB/sec execute 343 sec latency 55.785 ms 2024-05-27T07:23:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:40.222 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53676 0.61 MB/sec execute 344 sec latency 49.020 ms 2024-05-27T07:23:40.698 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:40 smithi043 ceph-mon[31531]: pgmap v331: 129 pgs: 129 active+clean; 94 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 355 KiB/s wr, 23 op/s 2024-05-27T07:23:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:40 smithi179 ceph-mon[35888]: pgmap v331: 129 pgs: 129 active+clean; 94 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 355 KiB/s wr, 23 op/s 2024-05-27T07:23:40.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:40 smithi132 ceph-mon[35283]: pgmap v331: 129 pgs: 129 active+clean; 94 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 355 KiB/s wr, 23 op/s 2024-05-27T07:23:41.223 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53736 0.61 MB/sec execute 345 sec latency 52.589 ms 2024-05-27T07:23:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.019+0000 7f30abe00700 1 -- 172.21.15.43:0/3129368211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac115170 msgr2=0x7f30ac117590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.019+0000 7f30abe00700 1 --2- 172.21.15.43:0/3129368211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac115170 0x7f30ac117590 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f30a400b0a0 tx=0x7f30a400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.019+0000 7f30abe00700 1 -- 172.21.15.43:0/3129368211 shutdown_connections 2024-05-27T07:23:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.019+0000 7f30abe00700 1 --2- 172.21.15.43:0/3129368211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac115170 0x7f30ac117590 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.019+0000 7f30abe00700 1 --2- 172.21.15.43:0/3129368211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30ac075b50 0x7f30ac075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.019+0000 7f30abe00700 1 --2- 172.21.15.43:0/3129368211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30ac0764a0 0x7f30ac114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.019+0000 7f30abe00700 1 -- 172.21.15.43:0/3129368211 >> 172.21.15.43:0/3129368211 conn(0x7f30ac079d00 msgr2=0x7f30ac07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.019+0000 7f30abe00700 1 -- 172.21.15.43:0/3129368211 shutdown_connections 2024-05-27T07:23:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.019+0000 7f30abe00700 1 -- 172.21.15.43:0/3129368211 wait complete. 2024-05-27T07:23:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.020+0000 7f30abe00700 1 Processor -- start 2024-05-27T07:23:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.021+0000 7f30abe00700 1 -- start start 2024-05-27T07:23:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.021+0000 7f30abe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac075b50 0x7f30ac1a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.021+0000 7f30abe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30ac0764a0 0x7f30ac1a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.021+0000 7f30abe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30ac115170 0x7f30ac1a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.021+0000 7f30abe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac11a780 con 0x7f30ac075b50 2024-05-27T07:23:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.021+0000 7f30abe00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac11a600 con 0x7f30ac115170 2024-05-27T07:23:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.021+0000 7f30abe00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac11a900 con 0x7f30ac0764a0 2024-05-27T07:23:42.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.021+0000 7f30aaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac075b50 0x7f30ac1a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.022+0000 7f30aaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac075b50 0x7f30ac1a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41642/0 (socket says 172.21.15.43:41642) 2024-05-27T07:23:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.022+0000 7f30aaa00700 1 -- 172.21.15.43:0/156194558 learned_addr learned my addr 172.21.15.43:0/156194558 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.022+0000 7f30ab400700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30ac115170 0x7f30ac1a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.022+0000 7f30a2000700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30ac0764a0 0x7f30ac1a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.022+0000 7f30aaa00700 1 -- 172.21.15.43:0/156194558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30ac0764a0 msgr2=0x7f30ac1a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.022+0000 7f30aaa00700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30ac0764a0 0x7f30ac1a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.022+0000 7f30aaa00700 1 -- 172.21.15.43:0/156194558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30ac115170 msgr2=0x7f30ac1a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.022+0000 7f30aaa00700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30ac115170 0x7f30ac1a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.022+0000 7f30aaa00700 1 -- 172.21.15.43:0/156194558 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ac1a5f70 con 0x7f30ac075b50 2024-05-27T07:23:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.023+0000 7f30aaa00700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac075b50 0x7f30ac1a1150 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f309c00c9b0 tx=0x7f309c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.023+0000 7f30a3e00700 1 -- 172.21.15.43:0/156194558 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c007b00 con 0x7f30ac075b50 2024-05-27T07:23:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.023+0000 7f30abe00700 1 -- 172.21.15.43:0/156194558 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ac1a6200 con 0x7f30ac075b50 2024-05-27T07:23:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.023+0000 7f30a3e00700 1 -- 172.21.15.43:0/156194558 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f309c007ca0 con 0x7f30ac075b50 2024-05-27T07:23:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.023+0000 7f30a3e00700 1 -- 172.21.15.43:0/156194558 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c018660 con 0x7f30ac075b50 2024-05-27T07:23:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.023+0000 7f30abe00700 1 -- 172.21.15.43:0/156194558 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ac1a6790 con 0x7f30ac075b50 2024-05-27T07:23:42.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.029+0000 7f30a3e00700 1 -- 172.21.15.43:0/156194558 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f309c004830 con 0x7f30ac075b50 2024-05-27T07:23:42.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.030+0000 7f30c05dcb80 1 Processor -- start 2024-05-27T07:23:42.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.031+0000 7f30c05dcb80 1 -- start start 2024-05-27T07:23:42.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.031+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5be56e40 0x55ee5bfd4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.031+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5410 0x55ee5bfdfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.031+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.031+0000 7f30a3e00700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30900b29f0 0x7f30900b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.031+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ee5bf03e40 con 0x55ee5bfd5410 2024-05-27T07:23:42.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.031+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ee5bef39e0 con 0x55ee5be56e40 2024-05-27T07:23:42.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.031+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ee5bef7a20 con 0x55ee5bfd5820 2024-05-27T07:23:42.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30a3e00700 1 -- 172.21.15.43:0/156194558 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f309c0e2a00 con 0x7f30ac075b50 2024-05-27T07:23:42.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30a2000700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30900b29f0 0x7f30900b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30ab400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30aaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5410 0x55ee5bfdfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30aaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5410 0x55ee5bfdfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41644/0 (socket says 172.21.15.43:41644) 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30aaa00700 1 -- 172.21.15.43:0/2134015160 learned_addr learned my addr 172.21.15.43:0/2134015160 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30ab400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:44212/0 (socket says 172.21.15.43:44212) 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30ab400700 1 --2- 172.21.15.43:0/2134015160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5be56e40 0x55ee5bfd4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30aaa00700 1 -- 172.21.15.43:0/2134015160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 msgr2=0x55ee5bfe2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30aaa00700 1 --2- 172.21.15.43:0/2134015160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.032+0000 7f30aaa00700 1 -- 172.21.15.43:0/2134015160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5be56e40 msgr2=0x55ee5bfd4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.033+0000 7f30aaa00700 1 --2- 172.21.15.43:0/2134015160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5be56e40 0x55ee5bfd4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.033+0000 7f30aaa00700 1 -- 172.21.15.43:0/2134015160 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5bebc420 con 0x55ee5bfd5410 2024-05-27T07:23:42.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.033+0000 7f30a2000700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30900b29f0 0x7f30900b4f00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3094005950 tx=0x7f30940058e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:42.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.033+0000 7f30aaa00700 1 --2- 172.21.15.43:0/2134015160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5410 0x55ee5bfdfad0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f309c0a98a0 tx=0x7f309c0a92e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae75d95abc63334 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:42.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.033+0000 7f309be00700 1 -- 172.21.15.43:0/2134015160 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c0a50a0 con 0x55ee5bfd5410 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.033+0000 7f309be00700 1 -- 172.21.15.43:0/2134015160 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f309c0b20c0 con 0x55ee5bfd5410 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.033+0000 7f309be00700 1 -- 172.21.15.43:0/2134015160 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c0b2770 con 0x55ee5bfd5410 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.034+0000 7f30c05dcb80 1 -- 172.21.15.43:0/2134015160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5410 msgr2=0x55ee5bfdfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.034+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/2134015160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5410 0x55ee5bfdfad0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f309c0a98a0 tx=0x7f309c0a92e0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.034+0000 7f30c05dcb80 1 -- 172.21.15.43:0/2134015160 shutdown_connections 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.034+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/2134015160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.034+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/2134015160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5be56e40 0x55ee5bfd4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.034+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/2134015160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5410 0x55ee5bfdfad0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.034+0000 7f30c05dcb80 1 -- 172.21.15.43:0/2134015160 >> 172.21.15.43:0/2134015160 conn(0x55ee5beb6fc0 msgr2=0x55ee5bfd0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.035+0000 7f30c05dcb80 1 -- 172.21.15.43:0/2134015160 shutdown_connections 2024-05-27T07:23:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.035+0000 7f30c05dcb80 1 -- 172.21.15.43:0/2134015160 wait complete. 2024-05-27T07:23:42.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.035+0000 7f30c05dcb80 1 -- start start 2024-05-27T07:23:42.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.036+0000 7f30c05dcb80 1 Processor -- start 2024-05-27T07:23:42.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.036+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5be56e40 0x55ee5bfdf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.036+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5410 0x55ee5bfdfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.036+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.036+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ee5bdf9fa0 con 0x55ee5bfd5820 2024-05-27T07:23:42.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.036+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ee5be94760 con 0x55ee5be56e40 2024-05-27T07:23:42.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.036+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ee5bfe2990 con 0x55ee5bfd5410 2024-05-27T07:23:42.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30ab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30c05dcb80 10 client.0 _ll_register_callbacks cb 0x55ee5be9dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30ab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41646/0 (socket says 172.21.15.43:41646) 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30ab400700 1 -- 172.21.15.43:0/412928607 learned_addr learned my addr 172.21.15.43:0/412928607 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30a2000700 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5410 0x55ee5bfdfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30aaa00700 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5be56e40 0x55ee5bfdf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30ab400700 1 -- 172.21.15.43:0/412928607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5410 msgr2=0x55ee5bfdfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30ab400700 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5410 0x55ee5bfdfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30ab400700 1 -- 172.21.15.43:0/412928607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5be56e40 msgr2=0x55ee5bfdf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30ab400700 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5be56e40 0x55ee5bfdf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.037+0000 7f30ab400700 1 -- 172.21.15.43:0/412928607 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5be438d0 con 0x55ee5bfd5820 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.038+0000 7f30ab400700 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe0030 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f30a400b0a0 tx=0x7f30a400cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb05246d913e7b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.038+0000 7f3098c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:42.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.038+0000 7f3098c00700 1 -- 172.21.15.43:0/412928607 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a4024020 con 0x55ee5bfd5820 2024-05-27T07:23:42.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.038+0000 7f30c05dcb80 1 -- 172.21.15.43:0/412928607 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f309c0b20c0 con 0x55ee5bfd5820 2024-05-27T07:23:42.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.038+0000 7f3098c00700 1 -- 172.21.15.43:0/412928607 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f30a40079d0 con 0x55ee5bfd5820 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f3098c00700 1 -- 172.21.15.43:0/412928607 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f30a4007bd0 con 0x55ee5bfd5820 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f30c05dcb80 10 client.16899 fetch_fsmap learned FSMap version 102 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f30c05dcb80 1 -- 172.21.15.43:0/412928607 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55ee5bec4060 con 0x55ee5bfd5820 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f3098c00700 1 -- 172.21.15.43:0/412928607 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f309c0b20c0 con 0x55ee5bfd5820 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f3098c00700 1 -- 172.21.15.43:0/412928607 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 102) ==== 4788+0+0 (secure 0 0 0) 0x7f30a4011070 con 0x55ee5bfd5820 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f30c05dcb80 10 client.16899 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f30c05dcb80 10 client.16899 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f30c05dcb80 20 client.16899 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55ee5bfefcf0 0x55ee5bff21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f30c05dcb80 4 client.16899 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:42.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.039+0000 7f30c05dcb80 1 -- 172.21.15.43:0/412928607 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee5bf05d90 con 0x55ee5bfefcf0 2024-05-27T07:23:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.040+0000 7f30aaa00700 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55ee5bfefcf0 0x55ee5bff21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.041+0000 7f30aaa00700 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55ee5bfefcf0 0x55ee5bff21a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d586d91dba21404 server_cookie=91f7c5b87bcef21a in_seq=0 out_seq=0 2024-05-27T07:23:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.041+0000 7f3098c00700 10 client.16899 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:23:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.043+0000 7f3098c00700 1 -- 172.21.15.43:0/412928607 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ee5bf05d90 con 0x55ee5bfefcf0 2024-05-27T07:23:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.043+0000 7f3098c00700 10 client.16899 handle_command_reply: tid=0 2024-05-27T07:23:42.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.043+0000 7f30c05dcb80 1 client.16899 shutdown 2024-05-27T07:23:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 1 -- 172.21.15.43:0/412928607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 msgr2=0x55ee5bfe0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe0030 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f30a400b0a0 tx=0x7f30a400cfd0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 1 -- 172.21.15.43:0/412928607 shutdown_connections 2024-05-27T07:23:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55ee5bfefcf0 0x55ee5bff21a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe0030 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5be56e40 0x55ee5bfdf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/412928607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5410 0x55ee5bfdfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f3098c00700 0 client.16899 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:23:42.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 1 -- 172.21.15.43:0/412928607 >> 172.21.15.43:0/412928607 conn(0x55ee5beb6fc0 msgr2=0x55ee5bfcfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:42.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 1 -- 172.21.15.43:0/412928607 shutdown_connections 2024-05-27T07:23:42.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 1 -- 172.21.15.43:0/412928607 wait complete. 2024-05-27T07:23:42.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.044+0000 7f30c05dcb80 20 client.16899 trim_cache size 0 max 16384 2024-05-27T07:23:42.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.062+0000 7f30c05dcb80 1 Processor -- start 2024-05-27T07:23:42.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.062+0000 7f30c05dcb80 1 -- start start 2024-05-27T07:23:42.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.062+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5be56e40 0x55ee5bfdad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.062+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5bfd5410 0x55ee5bfeba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.062+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 0x55ee5bfdfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.062+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac1b74d0 con 0x55ee5be56e40 2024-05-27T07:23:42.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.062+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac1b5260 con 0x55ee5bfd5410 2024-05-27T07:23:42.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.062+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac1b6e00 con 0x55ee5bfd5820 2024-05-27T07:23:42.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.062+0000 7f30aaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5be56e40 0x55ee5bfdad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.063+0000 7f30aaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5be56e40 0x55ee5bfdad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41658/0 (socket says 172.21.15.43:41658) 2024-05-27T07:23:42.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.063+0000 7f30aaa00700 1 -- 172.21.15.43:0/671973547 learned_addr learned my addr 172.21.15.43:0/671973547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:42.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.063+0000 7f30a2000700 1 --2- 172.21.15.43:0/671973547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5bfd5410 0x55ee5bfeba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.063+0000 7f30ab400700 1 --2- 172.21.15.43:0/671973547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 0x55ee5bfdfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.063+0000 7f30aaa00700 1 -- 172.21.15.43:0/671973547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 msgr2=0x55ee5bfdfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:42.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.063+0000 7f30aaa00700 1 --2- 172.21.15.43:0/671973547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 0x55ee5bfdfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.063+0000 7f30aaa00700 1 -- 172.21.15.43:0/671973547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5bfd5410 msgr2=0x55ee5bfeba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:42.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.063+0000 7f30aaa00700 1 --2- 172.21.15.43:0/671973547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5bfd5410 0x55ee5bfeba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.063+0000 7f30aaa00700 1 -- 172.21.15.43:0/671973547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5bfd2350 con 0x55ee5be56e40 2024-05-27T07:23:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.063+0000 7f30aaa00700 1 --2- 172.21.15.43:0/671973547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5be56e40 0x55ee5bfdad90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f309c0e0380 tx=0x7f309c00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c48e71a7b7fca3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.064+0000 7f309b400700 1 -- 172.21.15.43:0/671973547 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c0b20b0 con 0x55ee5be56e40 2024-05-27T07:23:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.064+0000 7f309b400700 1 -- 172.21.15.43:0/671973547 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f309c0a9420 con 0x55ee5be56e40 2024-05-27T07:23:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.064+0000 7f309b400700 1 -- 172.21.15.43:0/671973547 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c0a9b70 con 0x55ee5be56e40 2024-05-27T07:23:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.065+0000 7f30c05dcb80 1 -- 172.21.15.43:0/671973547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5be56e40 msgr2=0x55ee5bfdad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.065+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/671973547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5be56e40 0x55ee5bfdad90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f309c0e0380 tx=0x7f309c00cf00 comp rx=0 tx=0).stop 2024-05-27T07:23:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.065+0000 7f30c05dcb80 1 -- 172.21.15.43:0/671973547 shutdown_connections 2024-05-27T07:23:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.065+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/671973547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5bfd5820 0x55ee5bfdfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.065+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/671973547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5be56e40 0x55ee5bfdad90 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.065+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/671973547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5bfd5410 0x55ee5bfeba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.065+0000 7f30c05dcb80 1 -- 172.21.15.43:0/671973547 >> 172.21.15.43:0/671973547 conn(0x55ee5beb6fc0 msgr2=0x55ee5bfdf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:42.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.065+0000 7f30c05dcb80 1 -- 172.21.15.43:0/671973547 shutdown_connections 2024-05-27T07:23:42.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.065+0000 7f30c05dcb80 1 -- 172.21.15.43:0/671973547 wait complete. 2024-05-27T07:23:42.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.066+0000 7f30c05dcb80 1 -- start start 2024-05-27T07:23:42.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.066+0000 7f30c05dcb80 1 Processor -- start 2024-05-27T07:23:42.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5be56e40 0x55ee5bfe2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5bfd5410 0x55ee5bfe3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30c05dcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac1b6000 con 0x55ee5bfd5820 2024-05-27T07:23:42.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac1b5930 con 0x55ee5bfd5410 2024-05-27T07:23:42.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30c05dcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac1b6730 con 0x55ee5be56e40 2024-05-27T07:23:42.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30c05dcb80 10 client.0 _ll_register_callbacks cb 0x55ee5be9dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:42.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30ab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30ab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41670/0 (socket says 172.21.15.43:41670) 2024-05-27T07:23:42.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30ab400700 1 -- 172.21.15.43:0/4195844547 learned_addr learned my addr 172.21.15.43:0/4195844547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30aaa00700 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5be56e40 0x55ee5bfe2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.067+0000 7f30a2000700 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5bfd5410 0x55ee5bfe3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.068+0000 7f30ab400700 1 -- 172.21.15.43:0/4195844547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5be56e40 msgr2=0x55ee5bfe2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.068+0000 7f30ab400700 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5be56e40 0x55ee5bfe2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.068+0000 7f30ab400700 1 -- 172.21.15.43:0/4195844547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5bfd5410 msgr2=0x55ee5bfe3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.068+0000 7f30ab400700 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5bfd5410 0x55ee5bfe3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.068+0000 7f30ab400700 1 -- 172.21.15.43:0/4195844547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5c0588a0 con 0x55ee5bfd5820 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.068+0000 7f30ab400700 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe3660 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009cc0 tx=0x7f30a4007d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3493b32cc7a016e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.068+0000 7f3098c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.069+0000 7f3098c00700 1 -- 172.21.15.43:0/4195844547 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a4038060 con 0x55ee5bfd5820 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.069+0000 7f3098c00700 1 -- 172.21.15.43:0/4195844547 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f30a402a040 con 0x55ee5bfd5820 2024-05-27T07:23:42.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.069+0000 7f30c05dcb80 1 -- 172.21.15.43:0/4195844547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f309c0a9420 con 0x55ee5bfd5820 2024-05-27T07:23:42.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.069+0000 7f3098c00700 1 -- 172.21.15.43:0/4195844547 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f30a403f050 con 0x55ee5bfd5820 2024-05-27T07:23:42.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.069+0000 7f30c05dcb80 10 client.16905 fetch_fsmap learned FSMap version 102 2024-05-27T07:23:42.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.069+0000 7f30c05dcb80 1 -- 172.21.15.43:0/4195844547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55ee5c059b00 con 0x55ee5bfd5820 2024-05-27T07:23:42.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.070+0000 7f3098c00700 1 -- 172.21.15.43:0/4195844547 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f309c0a9420 con 0x55ee5bfd5820 2024-05-27T07:23:42.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.070+0000 7f3098c00700 1 -- 172.21.15.43:0/4195844547 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 102) ==== 4788+0+0 (secure 0 0 0) 0x7f30a4011030 con 0x55ee5bfd5820 2024-05-27T07:23:42.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.070+0000 7f30c05dcb80 10 client.16905 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:23:42.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.070+0000 7f30c05dcb80 10 client.16905 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:23:42.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.070+0000 7f30c05dcb80 20 client.16905 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:42.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.070+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55ee5c05a120 0x55ee5c05c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:42.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.070+0000 7f30c05dcb80 4 client.16905 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:23:42.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.070+0000 7f30c05dcb80 1 -- 172.21.15.43:0/4195844547 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee5beff570 con 0x55ee5c05a120 2024-05-27T07:23:42.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.071+0000 7f30aaa00700 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55ee5c05a120 0x55ee5c05c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:42.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.072+0000 7f30aaa00700 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55ee5c05a120 0x55ee5c05c540 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=18cc7157c9192687 server_cookie=fb9035fb83532160 in_seq=0 out_seq=0 2024-05-27T07:23:42.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.072+0000 7f3098c00700 10 client.16905 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:23:42.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.072+0000 7f3098c00700 1 -- 172.21.15.43:0/4195844547 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee5beff570 con 0x55ee5c05a120 2024-05-27T07:23:42.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.072+0000 7f3098c00700 10 client.16905 handle_command_reply: tid=0 2024-05-27T07:23:42.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.072+0000 7f30c05dcb80 1 client.16905 shutdown 2024-05-27T07:23:42.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 1 -- 172.21.15.43:0/4195844547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 msgr2=0x55ee5bfe3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:42.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe3660 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009cc0 tx=0x7f30a4007d70 comp rx=0 tx=0).stop 2024-05-27T07:23:42.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 1 -- 172.21.15.43:0/4195844547 shutdown_connections 2024-05-27T07:23:42.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55ee5c05a120 0x55ee5c05c540 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee5bfd5820 0x55ee5bfe3660 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5be56e40 0x55ee5bfe2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 1 --2- 172.21.15.43:0/4195844547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee5bfd5410 0x55ee5bfe3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 1 -- 172.21.15.43:0/4195844547 >> 172.21.15.43:0/4195844547 conn(0x55ee5beb6fc0 msgr2=0x55ee5bfe0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:42.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f3098c00700 0 client.16905 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:23:42.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 1 -- 172.21.15.43:0/4195844547 shutdown_connections 2024-05-27T07:23:42.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 1 -- 172.21.15.43:0/4195844547 wait complete. 2024-05-27T07:23:42.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.073+0000 7f30c05dcb80 20 client.16905 trim_cache size 0 max 16384 2024-05-27T07:23:42.065 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:23:42.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 -- 172.21.15.43:0/156194558 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30900b29f0 msgr2=0x7f30900b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:42.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30900b29f0 0x7f30900b4f00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3094005950 tx=0x7f30940058e0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 -- 172.21.15.43:0/156194558 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac075b50 msgr2=0x7f30ac1a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:42.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac075b50 0x7f30ac1a1150 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f309c00c9b0 tx=0x7f309c00ccc0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 -- 172.21.15.43:0/156194558 shutdown_connections 2024-05-27T07:23:42.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30900b29f0 0x7f30900b4f00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30ac115170 0x7f30ac1a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f30ac075b50 0x7f30ac1a1150 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 --2- 172.21.15.43:0/156194558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30ac0764a0 0x7f30ac1a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:42.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 -- 172.21.15.43:0/156194558 >> 172.21.15.43:0/156194558 conn(0x7f30ac079d00 msgr2=0x7f30ac100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:42.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 -- 172.21.15.43:0/156194558 shutdown_connections 2024-05-27T07:23:42.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:42.074+0000 7f30abe00700 1 -- 172.21.15.43:0/156194558 wait complete. 2024-05-27T07:23:42.222 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53813 0.61 MB/sec execute 346 sec latency 64.400 ms 2024-05-27T07:23:42.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:42 smithi043 ceph-mon[31531]: pgmap v332: 129 pgs: 129 active+clean; 96 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 607 KiB/s wr, 45 op/s 2024-05-27T07:23:42.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:42 smithi179 ceph-mon[35888]: pgmap v332: 129 pgs: 129 active+clean; 96 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 607 KiB/s wr, 45 op/s 2024-05-27T07:23:42.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:42 smithi132 ceph-mon[35283]: pgmap v332: 129 pgs: 129 active+clean; 96 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 607 KiB/s wr, 45 op/s 2024-05-27T07:23:42.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:23:43.223 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53900 0.61 MB/sec execute 347 sec latency 57.667 ms 2024-05-27T07:23:43.456 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:44.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 54046 0.61 MB/sec execute 348 sec latency 50.076 ms 2024-05-27T07:23:44.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:44 smithi043 ceph-mon[31531]: pgmap v333: 129 pgs: 129 active+clean; 96 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 433 KiB/s wr, 36 op/s 2024-05-27T07:23:44.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:44 smithi179 ceph-mon[35888]: pgmap v333: 129 pgs: 129 active+clean; 96 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 433 KiB/s wr, 36 op/s 2024-05-27T07:23:44.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:44 smithi132 ceph-mon[35283]: pgmap v333: 129 pgs: 129 active+clean; 96 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 433 KiB/s wr, 36 op/s 2024-05-27T07:23:45.223 INFO:tasks.workunit.client.0.smithi043.stdout: 1 54160 0.61 MB/sec execute 349 sec latency 56.482 ms 2024-05-27T07:23:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.867+0000 7f2776000700 1 -- 172.21.15.43:0/2480686826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27701045e0 msgr2=0x7f27701049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.867+0000 7f2776000700 1 --2- 172.21.15.43:0/2480686826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27701045e0 0x7f27701049f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f27600099c0 tx=0x7f2760009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.867+0000 7f2776000700 1 -- 172.21.15.43:0/2480686826 shutdown_connections 2024-05-27T07:23:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.867+0000 7f2776000700 1 --2- 172.21.15.43:0/2480686826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2770109860 0x7f27701111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.867+0000 7f2776000700 1 --2- 172.21.15.43:0/2480686826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27701045e0 0x7f27701049f0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.867+0000 7f2776000700 1 --2- 172.21.15.43:0/2480686826 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2770108f10 0x7f2770109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.867+0000 7f2776000700 1 -- 172.21.15.43:0/2480686826 >> 172.21.15.43:0/2480686826 conn(0x7f2770100030 msgr2=0x7f2770102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.867+0000 7f2776000700 1 -- 172.21.15.43:0/2480686826 shutdown_connections 2024-05-27T07:23:45.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.867+0000 7f2776000700 1 -- 172.21.15.43:0/2480686826 wait complete. 2024-05-27T07:23:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.869+0000 7f2776000700 1 Processor -- start 2024-05-27T07:23:45.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.869+0000 7f2776000700 1 -- start start 2024-05-27T07:23:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.870+0000 7f2776000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27701045e0 0x7f2770075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.870+0000 7f2776000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2770108f10 0x7f2770075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.870+0000 7f2776000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2770109860 0x7f2770076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.870+0000 7f2776000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2770114390 con 0x7f2770108f10 2024-05-27T07:23:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.870+0000 7f2776000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2770114210 con 0x7f2770109860 2024-05-27T07:23:45.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.870+0000 7f2776000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2770114510 con 0x7f27701045e0 2024-05-27T07:23:45.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.870+0000 7f276ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2770108f10 0x7f2770075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.870+0000 7f276ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2770108f10 0x7f2770075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53096/0 (socket says 172.21.15.43:53096) 2024-05-27T07:23:45.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.870+0000 7f276ea00700 1 -- 172.21.15.43:0/2390247628 learned_addr learned my addr 172.21.15.43:0/2390247628 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:45.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.871+0000 7f276f400700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27701045e0 0x7f2770075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.871+0000 7f276fe00700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2770109860 0x7f2770076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.871+0000 7f276ea00700 1 -- 172.21.15.43:0/2390247628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27701045e0 msgr2=0x7f2770075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.871+0000 7f276ea00700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27701045e0 0x7f2770075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.871+0000 7f276ea00700 1 -- 172.21.15.43:0/2390247628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2770109860 msgr2=0x7f2770076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.871+0000 7f276ea00700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2770109860 0x7f2770076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.871+0000 7f276ea00700 1 -- 172.21.15.43:0/2390247628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27701a9140 con 0x7f2770108f10 2024-05-27T07:23:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.871+0000 7f276ea00700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2770108f10 0x7f2770075ff0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f275c00d980 tx=0x7f275c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.872+0000 7f2767e00700 1 -- 172.21.15.43:0/2390247628 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c017070 con 0x7f2770108f10 2024-05-27T07:23:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.872+0000 7f2767e00700 1 -- 172.21.15.43:0/2390247628 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f275c00fd20 con 0x7f2770108f10 2024-05-27T07:23:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.872+0000 7f2776000700 1 -- 172.21.15.43:0/2390247628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27701a9430 con 0x7f2770108f10 2024-05-27T07:23:45.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.872+0000 7f2767e00700 1 -- 172.21.15.43:0/2390247628 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c00e3b0 con 0x7f2770108f10 2024-05-27T07:23:45.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.872+0000 7f2776000700 1 -- 172.21.15.43:0/2390247628 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27701a9970 con 0x7f2770108f10 2024-05-27T07:23:45.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.879+0000 7f2784acbb80 1 Processor -- start 2024-05-27T07:23:45.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.879+0000 7f2767e00700 1 -- 172.21.15.43:0/2390247628 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f275c00e550 con 0x7f2770108f10 2024-05-27T07:23:45.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.879+0000 7f2784acbb80 1 -- start start 2024-05-27T07:23:45.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.880+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd2dde40 0x55c2cd4627f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.880+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 0x55c2cd45c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.880+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd45cdb0 0x55c2cd45d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.880+0000 7f2784acbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c2cd38ae40 con 0x55c2cd45cdb0 2024-05-27T07:23:45.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.880+0000 7f2784acbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c2cd37a9e0 con 0x55c2cd462d30 2024-05-27T07:23:45.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.880+0000 7f2784acbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c2cd37ea20 con 0x55c2cd2dde40 2024-05-27T07:23:45.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.880+0000 7f276ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd45cdb0 0x55c2cd45d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.880+0000 7f276ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd45cdb0 0x55c2cd45d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53100/0 (socket says 172.21.15.43:53100) 2024-05-27T07:23:45.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.880+0000 7f276ea00700 1 -- 172.21.15.43:0/4116612365 learned_addr learned my addr 172.21.15.43:0/4116612365 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:45.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.880+0000 7f276fe00700 1 --2- 172.21.15.43:0/4116612365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd2dde40 0x55c2cd4627f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.881+0000 7f276f400700 1 --2- 172.21.15.43:0/4116612365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 0x55c2cd45c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.881+0000 7f276ea00700 1 -- 172.21.15.43:0/4116612365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd2dde40 msgr2=0x55c2cd4627f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:45.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.881+0000 7f276ea00700 1 --2- 172.21.15.43:0/4116612365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd2dde40 0x55c2cd4627f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.881+0000 7f276ea00700 1 -- 172.21.15.43:0/4116612365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 msgr2=0x55c2cd45c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:45.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.881+0000 7f276ea00700 1 --2- 172.21.15.43:0/4116612365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 0x55c2cd45c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.882+0000 7f276ea00700 1 -- 172.21.15.43:0/4116612365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2cd343420 con 0x55c2cd45cdb0 2024-05-27T07:23:45.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.882+0000 7f276f400700 1 --2- 172.21.15.43:0/4116612365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 0x55c2cd45c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:45.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.882+0000 7f2767e00700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27540b29f0 0x7f27540b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.882+0000 7f276ea00700 1 --2- 172.21.15.43:0/4116612365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd45cdb0 0x55c2cd45d1c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f275c0dfad0 tx=0x7f275c0e8c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0473e9569e135 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.882+0000 7f276fe00700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27540b29f0 0x7f27540b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.882+0000 7f2767e00700 1 -- 172.21.15.43:0/2390247628 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f275c0e27e0 con 0x7f2770108f10 2024-05-27T07:23:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.883+0000 7f2764c00700 1 -- 172.21.15.43:0/4116612365 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c0a55b0 con 0x55c2cd45cdb0 2024-05-27T07:23:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.883+0000 7f2764c00700 1 -- 172.21.15.43:0/4116612365 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f275c0a9200 con 0x55c2cd45cdb0 2024-05-27T07:23:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.883+0000 7f2764c00700 1 -- 172.21.15.43:0/4116612365 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c0a8e60 con 0x55c2cd45cdb0 2024-05-27T07:23:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.884+0000 7f2784acbb80 1 -- 172.21.15.43:0/4116612365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd45cdb0 msgr2=0x55c2cd45d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.884+0000 7f276fe00700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27540b29f0 0x7f27540b4f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55c2cd45db30 tx=0x7f2768009100 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.884+0000 7f2784acbb80 1 --2- 172.21.15.43:0/4116612365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd45cdb0 0x55c2cd45d1c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f275c0dfad0 tx=0x7f275c0e8c10 comp rx=0 tx=0).stop 2024-05-27T07:23:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.884+0000 7f2784acbb80 1 -- 172.21.15.43:0/4116612365 shutdown_connections 2024-05-27T07:23:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.884+0000 7f2784acbb80 1 --2- 172.21.15.43:0/4116612365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd45cdb0 0x55c2cd45d1c0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.884+0000 7f2784acbb80 1 --2- 172.21.15.43:0/4116612365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd2dde40 0x55c2cd4627f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.884+0000 7f2784acbb80 1 --2- 172.21.15.43:0/4116612365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 0x55c2cd45c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.884+0000 7f2784acbb80 1 -- 172.21.15.43:0/4116612365 >> 172.21.15.43:0/4116612365 conn(0x55c2cd33dfc0 msgr2=0x55c2cd4578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.884+0000 7f2784acbb80 1 -- 172.21.15.43:0/4116612365 shutdown_connections 2024-05-27T07:23:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.884+0000 7f2784acbb80 1 -- 172.21.15.43:0/4116612365 wait complete. 2024-05-27T07:23:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.885+0000 7f2784acbb80 1 -- start start 2024-05-27T07:23:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.885+0000 7f2784acbb80 1 Processor -- start 2024-05-27T07:23:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.886+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd466530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.886+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd45cdb0 0x55c2cd466a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.886+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 0x55c2cd466fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.886+0000 7f2784acbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c2cd280fa0 con 0x55c2cd2dde40 2024-05-27T07:23:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.886+0000 7f2784acbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c2cd31b760 con 0x55c2cd462d30 2024-05-27T07:23:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.886+0000 7f2784acbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c2cd469800 con 0x55c2cd45cdb0 2024-05-27T07:23:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.886+0000 7f2784acbb80 10 client.0 _ll_register_callbacks cb 0x55c2cd324c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.886+0000 7f276f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd466530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.886+0000 7f276f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd466530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53110/0 (socket says 172.21.15.43:53110) 2024-05-27T07:23:45.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.886+0000 7f276f400700 1 -- 172.21.15.43:0/683324083 learned_addr learned my addr 172.21.15.43:0/683324083 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.887+0000 7f276ea00700 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd45cdb0 0x55c2cd466a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.887+0000 7f276fe00700 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 0x55c2cd466fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.887+0000 7f276f400700 1 -- 172.21.15.43:0/683324083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd45cdb0 msgr2=0x55c2cd466a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.887+0000 7f276f400700 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd45cdb0 0x55c2cd466a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.887+0000 7f276f400700 1 -- 172.21.15.43:0/683324083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 msgr2=0x55c2cd466fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.887+0000 7f276f400700 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 0x55c2cd466fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.887+0000 7f276f400700 1 -- 172.21.15.43:0/683324083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2cd2ca8d0 con 0x55c2cd2dde40 2024-05-27T07:23:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.887+0000 7f276f400700 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd466530 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f27600099c0 tx=0x7f2760009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f24219d9da70a46 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.887+0000 7f2759600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.888+0000 7f2759600700 1 -- 172.21.15.43:0/683324083 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276000bc20 con 0x55c2cd2dde40 2024-05-27T07:23:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.888+0000 7f2759600700 1 -- 172.21.15.43:0/683324083 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f276000bdc0 con 0x55c2cd2dde40 2024-05-27T07:23:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.888+0000 7f2784acbb80 1 -- 172.21.15.43:0/683324083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f275c0a9200 con 0x55c2cd2dde40 2024-05-27T07:23:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.888+0000 7f2759600700 1 -- 172.21.15.43:0/683324083 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f2760030050 con 0x55c2cd2dde40 2024-05-27T07:23:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.888+0000 7f2784acbb80 10 client.16917 fetch_fsmap learned FSMap version 102 2024-05-27T07:23:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.888+0000 7f2784acbb80 1 -- 172.21.15.43:0/683324083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55c2cd34b060 con 0x55c2cd2dde40 2024-05-27T07:23:45.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.889+0000 7f2759600700 1 -- 172.21.15.43:0/683324083 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f275c0a9200 con 0x55c2cd2dde40 2024-05-27T07:23:45.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.889+0000 7f2759600700 1 -- 172.21.15.43:0/683324083 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 102) ==== 4788+0+0 (secure 0 0 0) 0x7f276000f720 con 0x55c2cd2dde40 2024-05-27T07:23:45.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.889+0000 7f2784acbb80 10 client.16917 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:23:45.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.889+0000 7f2784acbb80 10 client.16917 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:23:45.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.889+0000 7f2784acbb80 20 client.16917 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:45.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.889+0000 7f2784acbb80 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55c2cd476b80 0x55c2cd479010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.889+0000 7f2784acbb80 4 client.16917 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:45.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.889+0000 7f2784acbb80 1 -- 172.21.15.43:0/683324083 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2cd38cd90 con 0x55c2cd476b80 2024-05-27T07:23:45.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.890+0000 7f276ea00700 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55c2cd476b80 0x55c2cd479010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.891+0000 7f276ea00700 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55c2cd476b80 0x55c2cd479010 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef28b39d75b14b7b server_cookie=63d2482f75badade in_seq=0 out_seq=0 2024-05-27T07:23:45.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.891+0000 7f2759600700 10 client.16917 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:23:45.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.893+0000 7f2759600700 1 -- 172.21.15.43:0/683324083 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c2cd38cd90 con 0x55c2cd476b80 2024-05-27T07:23:45.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.893+0000 7f2759600700 10 client.16917 handle_command_reply: tid=0 2024-05-27T07:23:45.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.893+0000 7f2784acbb80 1 client.16917 shutdown 2024-05-27T07:23:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.893+0000 7f2784acbb80 1 -- 172.21.15.43:0/683324083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 msgr2=0x55c2cd466530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.893+0000 7f2784acbb80 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd466530 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f27600099c0 tx=0x7f2760009550 comp rx=0 tx=0).stop 2024-05-27T07:23:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.894+0000 7f2784acbb80 1 -- 172.21.15.43:0/683324083 shutdown_connections 2024-05-27T07:23:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.894+0000 7f2784acbb80 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55c2cd476b80 0x55c2cd479010 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.894+0000 7f2784acbb80 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd462d30 0x55c2cd466fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.894+0000 7f2784acbb80 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd466530 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.894+0000 7f2784acbb80 1 --2- 172.21.15.43:0/683324083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd45cdb0 0x55c2cd466a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.894+0000 7f2784acbb80 1 -- 172.21.15.43:0/683324083 >> 172.21.15.43:0/683324083 conn(0x55c2cd33dfc0 msgr2=0x55c2cd457580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.894+0000 7f2759600700 0 client.16917 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:23:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.894+0000 7f2784acbb80 1 -- 172.21.15.43:0/683324083 shutdown_connections 2024-05-27T07:23:45.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.894+0000 7f2784acbb80 1 -- 172.21.15.43:0/683324083 wait complete. 2024-05-27T07:23:45.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.894+0000 7f2784acbb80 20 client.16917 trim_cache size 0 max 16384 2024-05-27T07:23:45.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.916+0000 7f2784acbb80 1 Processor -- start 2024-05-27T07:23:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.916+0000 7f2784acbb80 1 -- start start 2024-05-27T07:23:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.916+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd45c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.916+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd45cdb0 0x55c2cd465490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.916+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd462d30 0x55c2cd468890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.916+0000 7f2784acbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f27701b33d0 con 0x55c2cd2dde40 2024-05-27T07:23:45.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.916+0000 7f2784acbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f27701b1160 con 0x55c2cd45cdb0 2024-05-27T07:23:45.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.916+0000 7f2784acbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f27701b2d00 con 0x55c2cd462d30 2024-05-27T07:23:45.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.917+0000 7f276f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd45c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.917+0000 7f276f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd45c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53118/0 (socket says 172.21.15.43:53118) 2024-05-27T07:23:45.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.917+0000 7f276f400700 1 -- 172.21.15.43:0/2931763356 learned_addr learned my addr 172.21.15.43:0/2931763356 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:45.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.917+0000 7f276ea00700 1 --2- 172.21.15.43:0/2931763356 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd45cdb0 0x55c2cd465490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.917+0000 7f276fe00700 1 --2- 172.21.15.43:0/2931763356 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd462d30 0x55c2cd468890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.917+0000 7f276f400700 1 -- 172.21.15.43:0/2931763356 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd462d30 msgr2=0x55c2cd468890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:45.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.917+0000 7f276f400700 1 --2- 172.21.15.43:0/2931763356 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd462d30 0x55c2cd468890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.917+0000 7f276f400700 1 -- 172.21.15.43:0/2931763356 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd45cdb0 msgr2=0x55c2cd465490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:45.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.917+0000 7f276f400700 1 --2- 172.21.15.43:0/2931763356 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd45cdb0 0x55c2cd465490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.917+0000 7f276f400700 1 -- 172.21.15.43:0/2931763356 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2cd467320 con 0x55c2cd2dde40 2024-05-27T07:23:45.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.918+0000 7f276f400700 1 --2- 172.21.15.43:0/2931763356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd45c870 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f27600058e0 tx=0x7f2760009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c69e63fc7a5b8c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:45.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.918+0000 7f275be00700 1 -- 172.21.15.43:0/2931763356 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276000ba20 con 0x55c2cd2dde40 2024-05-27T07:23:45.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.918+0000 7f275be00700 1 -- 172.21.15.43:0/2931763356 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2760027070 con 0x55c2cd2dde40 2024-05-27T07:23:45.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.918+0000 7f275be00700 1 -- 172.21.15.43:0/2931763356 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276000e9e0 con 0x55c2cd2dde40 2024-05-27T07:23:45.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.919+0000 7f2784acbb80 1 -- 172.21.15.43:0/2931763356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 msgr2=0x55c2cd45c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:45.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.919+0000 7f2784acbb80 1 --2- 172.21.15.43:0/2931763356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd45c870 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f27600058e0 tx=0x7f2760009cd0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.919+0000 7f2784acbb80 1 -- 172.21.15.43:0/2931763356 shutdown_connections 2024-05-27T07:23:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.919+0000 7f2784acbb80 1 --2- 172.21.15.43:0/2931763356 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd462d30 0x55c2cd468890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.919+0000 7f2784acbb80 1 --2- 172.21.15.43:0/2931763356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd2dde40 0x55c2cd45c870 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.919+0000 7f2784acbb80 1 --2- 172.21.15.43:0/2931763356 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd45cdb0 0x55c2cd465490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.919+0000 7f2784acbb80 1 -- 172.21.15.43:0/2931763356 >> 172.21.15.43:0/2931763356 conn(0x55c2cd33dfc0 msgr2=0x55c2cd460210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.919+0000 7f2784acbb80 1 -- 172.21.15.43:0/2931763356 shutdown_connections 2024-05-27T07:23:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.920+0000 7f2784acbb80 1 -- 172.21.15.43:0/2931763356 wait complete. 2024-05-27T07:23:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.920+0000 7f2784acbb80 1 -- start start 2024-05-27T07:23:45.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.920+0000 7f2784acbb80 1 Processor -- start 2024-05-27T07:23:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.921+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd2dde40 0x55c2cd47efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.921+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd45cdb0 0x55c2cd46ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.921+0000 7f2784acbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd462d30 0x55c2cd46f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.921+0000 7f2784acbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f27701b1f00 con 0x55c2cd462d30 2024-05-27T07:23:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.921+0000 7f2784acbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f27701b1830 con 0x55c2cd2dde40 2024-05-27T07:23:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.921+0000 7f2784acbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f27701b2630 con 0x55c2cd45cdb0 2024-05-27T07:23:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.921+0000 7f2784acbb80 10 client.0 _ll_register_callbacks cb 0x55c2cd324c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.921+0000 7f276fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd462d30 0x55c2cd46f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.921+0000 7f276fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd462d30 0x55c2cd46f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53132/0 (socket says 172.21.15.43:53132) 2024-05-27T07:23:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f276fe00700 1 -- 172.21.15.43:0/4076596894 learned_addr learned my addr 172.21.15.43:0/4076596894 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f276ea00700 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd45cdb0 0x55c2cd46ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f276f400700 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd2dde40 0x55c2cd47efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f276fe00700 1 -- 172.21.15.43:0/4076596894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd45cdb0 msgr2=0x55c2cd46ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f276fe00700 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd45cdb0 0x55c2cd46ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f276fe00700 1 -- 172.21.15.43:0/4076596894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd2dde40 msgr2=0x55c2cd47efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f276fe00700 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd2dde40 0x55c2cd47efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f276fe00700 1 -- 172.21.15.43:0/4076596894 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2cd47d830 con 0x55c2cd462d30 2024-05-27T07:23:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f276f400700 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd2dde40 0x55c2cd47efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f276fe00700 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd462d30 0x55c2cd46f0a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f27680077b0 tx=0x7f2768004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc956adef8e3d017 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:45.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.922+0000 7f2759600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:45.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.923+0000 7f2759600700 1 -- 172.21.15.43:0/4076596894 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2768028070 con 0x55c2cd462d30 2024-05-27T07:23:45.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.923+0000 7f2784acbb80 1 -- 172.21.15.43:0/4076596894 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2760027070 con 0x55c2cd462d30 2024-05-27T07:23:45.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.923+0000 7f2759600700 1 -- 172.21.15.43:0/4076596894 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7f2768016840 con 0x55c2cd462d30 2024-05-27T07:23:45.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.923+0000 7f2759600700 1 -- 172.21.15.43:0/4076596894 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f2768016a40 con 0x55c2cd462d30 2024-05-27T07:23:45.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.923+0000 7f2784acbb80 10 client.16923 fetch_fsmap learned FSMap version 102 2024-05-27T07:23:45.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.923+0000 7f2784acbb80 1 -- 172.21.15.43:0/4076596894 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=4+}) -- 0x55c2cd4e0950 con 0x55c2cd462d30 2024-05-27T07:23:45.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.924+0000 7f2759600700 1 -- 172.21.15.43:0/4076596894 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2760027070 con 0x55c2cd462d30 2024-05-27T07:23:45.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.924+0000 7f2759600700 1 -- 172.21.15.43:0/4076596894 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 102) ==== 4788+0+0 (secure 0 0 0) 0x7f276802dc90 con 0x55c2cd462d30 2024-05-27T07:23:45.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.924+0000 7f2784acbb80 10 client.16923 fetch_fsmap finished waiting for FSMap version 102 2024-05-27T07:23:45.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.924+0000 7f2784acbb80 10 client.16923 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:23:45.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.924+0000 7f2784acbb80 20 client.16923 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:45.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.924+0000 7f2784acbb80 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55c2cd4e10f0 0x55c2cd4e3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:45.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.924+0000 7f2784acbb80 4 client.16923 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:23:45.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.924+0000 7f2784acbb80 1 -- 172.21.15.43:0/4076596894 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2cd386570 con 0x55c2cd4e10f0 2024-05-27T07:23:45.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.925+0000 7f276f400700 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55c2cd4e10f0 0x55c2cd4e3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:45.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.926+0000 7f276f400700 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55c2cd4e10f0 0x55c2cd4e3510 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a59a5faf5d98eca4 server_cookie=8cebbdf25e5a5bb1 in_seq=0 out_seq=0 2024-05-27T07:23:45.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.926+0000 7f2759600700 10 client.16923 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:23:45.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.926+0000 7f2759600700 1 -- 172.21.15.43:0/4076596894 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2cd386570 con 0x55c2cd4e10f0 2024-05-27T07:23:45.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.926+0000 7f2759600700 10 client.16923 handle_command_reply: tid=0 2024-05-27T07:23:45.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.926+0000 7f2784acbb80 1 client.16923 shutdown 2024-05-27T07:23:45.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.926+0000 7f2784acbb80 1 -- 172.21.15.43:0/4076596894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd462d30 msgr2=0x55c2cd46f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:45.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.926+0000 7f2784acbb80 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd462d30 0x55c2cd46f0a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f27680077b0 tx=0x7f2768004ad0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.927+0000 7f2784acbb80 1 -- 172.21.15.43:0/4076596894 shutdown_connections 2024-05-27T07:23:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.927+0000 7f2784acbb80 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55c2cd4e10f0 0x55c2cd4e3510 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.927+0000 7f2784acbb80 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c2cd462d30 0x55c2cd46f0a0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.927+0000 7f2784acbb80 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2cd2dde40 0x55c2cd47efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.927+0000 7f2784acbb80 1 --2- 172.21.15.43:0/4076596894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c2cd45cdb0 0x55c2cd46ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.927+0000 7f2759600700 0 client.16923 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:23:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.927+0000 7f2784acbb80 1 -- 172.21.15.43:0/4076596894 >> 172.21.15.43:0/4076596894 conn(0x55c2cd33dfc0 msgr2=0x55c2cd466310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.927+0000 7f2784acbb80 1 -- 172.21.15.43:0/4076596894 shutdown_connections 2024-05-27T07:23:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.927+0000 7f2784acbb80 1 -- 172.21.15.43:0/4076596894 wait complete. 2024-05-27T07:23:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.927+0000 7f2784acbb80 20 client.16923 trim_cache size 0 max 16384 2024-05-27T07:23:45.919 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:23:45.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 -- 172.21.15.43:0/2390247628 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27540b29f0 msgr2=0x7f27540b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:45.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27540b29f0 0x7f27540b4f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55c2cd45db30 tx=0x7f2768009100 comp rx=0 tx=0).stop 2024-05-27T07:23:45.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 -- 172.21.15.43:0/2390247628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2770108f10 msgr2=0x7f2770075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:45.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2770108f10 0x7f2770075ff0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f275c00d980 tx=0x7f275c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:23:45.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 -- 172.21.15.43:0/2390247628 shutdown_connections 2024-05-27T07:23:45.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27540b29f0 0x7f27540b4f00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2770109860 0x7f2770076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27701045e0 0x7f2770075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 --2- 172.21.15.43:0/2390247628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2770108f10 0x7f2770075ff0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:45.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 -- 172.21.15.43:0/2390247628 >> 172.21.15.43:0/2390247628 conn(0x7f2770100030 msgr2=0x7f2770102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:45.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.928+0000 7f2776000700 1 -- 172.21.15.43:0/2390247628 shutdown_connections 2024-05-27T07:23:45.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:45.929+0000 7f2776000700 1 -- 172.21.15.43:0/2390247628 wait complete. 2024-05-27T07:23:46.223 INFO:tasks.workunit.client.0.smithi043.stdout: 1 54291 0.61 MB/sec execute 350 sec latency 53.644 ms 2024-05-27T07:23:46.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:46 smithi043 ceph-mon[31531]: pgmap v334: 129 pgs: 129 active+clean; 96 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 433 KiB/s wr, 36 op/s 2024-05-27T07:23:46.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:46 smithi179 ceph-mon[35888]: pgmap v334: 129 pgs: 129 active+clean; 96 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 433 KiB/s wr, 36 op/s 2024-05-27T07:23:46.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:23:46.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:46 smithi132 ceph-mon[35283]: pgmap v334: 129 pgs: 129 active+clean; 96 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 433 KiB/s wr, 36 op/s 2024-05-27T07:23:47.223 INFO:tasks.workunit.client.0.smithi043.stdout: 1 54411 0.61 MB/sec execute 351 sec latency 54.453 ms 2024-05-27T07:23:47.310 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:48.223 INFO:tasks.workunit.client.0.smithi043.stdout: 1 54520 0.60 MB/sec execute 352 sec latency 49.186 ms 2024-05-27T07:23:48.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:48 smithi043 ceph-mon[31531]: pgmap v335: 129 pgs: 129 active+clean; 94 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 583 KiB/s wr, 59 op/s 2024-05-27T07:23:48.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:48 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:48.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:48 smithi179 ceph-mon[35888]: pgmap v335: 129 pgs: 129 active+clean; 94 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 583 KiB/s wr, 59 op/s 2024-05-27T07:23:48.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:48 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:48.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:48 smithi132 ceph-mon[35283]: pgmap v335: 129 pgs: 129 active+clean; 94 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 583 KiB/s wr, 59 op/s 2024-05-27T07:23:48.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:48 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:49.224 INFO:tasks.workunit.client.0.smithi043.stdout: 1 54686 0.61 MB/sec execute 353 sec latency 59.652 ms 2024-05-27T07:23:49.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.774+0000 7efd57e00700 1 -- 172.21.15.43:0/1984791799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd5810b890 msgr2=0x7efd581131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.774+0000 7efd57e00700 1 --2- 172.21.15.43:0/1984791799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd5810b890 0x7efd581131f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7efd5000b0d0 tx=0x7efd5000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.775+0000 7efd57e00700 1 -- 172.21.15.43:0/1984791799 shutdown_connections 2024-05-27T07:23:49.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.775+0000 7efd57e00700 1 --2- 172.21.15.43:0/1984791799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd5810b890 0x7efd581131f0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.775+0000 7efd57e00700 1 --2- 172.21.15.43:0/1984791799 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd58106610 0x7efd58106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.775+0000 7efd57e00700 1 --2- 172.21.15.43:0/1984791799 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd5810af40 0x7efd5810b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.775+0000 7efd57e00700 1 -- 172.21.15.43:0/1984791799 >> 172.21.15.43:0/1984791799 conn(0x7efd58102020 msgr2=0x7efd58104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:49.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.775+0000 7efd57e00700 1 -- 172.21.15.43:0/1984791799 shutdown_connections 2024-05-27T07:23:49.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.775+0000 7efd57e00700 1 -- 172.21.15.43:0/1984791799 wait complete. 2024-05-27T07:23:49.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.776+0000 7efd57e00700 1 Processor -- start 2024-05-27T07:23:49.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.776+0000 7efd57e00700 1 -- start start 2024-05-27T07:23:49.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd57e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd58106610 0x7efd58110610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd57e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd5810af40 0x7efd58110b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd57e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd5810b890 0x7efd58111090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd57e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efd58116370 con 0x7efd5810b890 2024-05-27T07:23:49.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd57e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efd581161f0 con 0x7efd58106610 2024-05-27T07:23:49.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd57e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efd581164f0 con 0x7efd5810af40 2024-05-27T07:23:49.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd57400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd5810b890 0x7efd58111090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd57400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd5810b890 0x7efd58111090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53150/0 (socket says 172.21.15.43:53150) 2024-05-27T07:23:49.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd57400700 1 -- 172.21.15.43:0/943550194 learned_addr learned my addr 172.21.15.43:0/943550194 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:49.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd56000700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd5810af40 0x7efd58110b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.777+0000 7efd56a00700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd58106610 0x7efd58110610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.778+0000 7efd57400700 1 -- 172.21.15.43:0/943550194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd5810af40 msgr2=0x7efd58110b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.778+0000 7efd57400700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd5810af40 0x7efd58110b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.778+0000 7efd57400700 1 -- 172.21.15.43:0/943550194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd58106610 msgr2=0x7efd58110610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.778+0000 7efd57400700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd58106610 0x7efd58110610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.778+0000 7efd57400700 1 -- 172.21.15.43:0/943550194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd581ad240 con 0x7efd5810b890 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.778+0000 7efd57400700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd5810b890 0x7efd58111090 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efd50009850 tx=0x7efd50007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.778+0000 7efd4f400700 1 -- 172.21.15.43:0/943550194 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5001f030 con 0x7efd5810b890 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.778+0000 7efd4f400700 1 -- 172.21.15.43:0/943550194 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd5001bad0 con 0x7efd5810b890 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.778+0000 7efd57e00700 1 -- 172.21.15.43:0/943550194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd581ad4d0 con 0x7efd5810b890 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.779+0000 7efd4f400700 1 -- 172.21.15.43:0/943550194 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50012bb0 con 0x7efd5810b890 2024-05-27T07:23:49.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.779+0000 7efd57e00700 1 -- 172.21.15.43:0/943550194 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd581ad9d0 con 0x7efd5810b890 2024-05-27T07:23:49.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.784+0000 7efd4f400700 1 -- 172.21.15.43:0/943550194 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7efd50022030 con 0x7efd5810b890 2024-05-27T07:23:49.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.785+0000 7efd6c551b80 1 Processor -- start 2024-05-27T07:23:49.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.785+0000 7efd6c551b80 1 -- start start 2024-05-27T07:23:49.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.786+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562260f4de40 0x5622610cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.786+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc410 0x5622610d6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.786+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610d9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.786+0000 7efd6c551b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562260ffae40 con 0x5622610cc820 2024-05-27T07:23:49.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.786+0000 7efd6c551b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562260fea9e0 con 0x5622610cc410 2024-05-27T07:23:49.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.786+0000 7efd6c551b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562260feea20 con 0x562260f4de40 2024-05-27T07:23:49.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.786+0000 7efd4f400700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efd3c0b2a70 0x7efd3c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.786+0000 7efd56000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610d9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.786+0000 7efd56000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610d9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53160/0 (socket says 172.21.15.43:53160) 2024-05-27T07:23:49.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.786+0000 7efd56000700 1 -- 172.21.15.43:0/730236709 learned_addr learned my addr 172.21.15.43:0/730236709 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:49.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.787+0000 7efd56a00700 1 --2- 172.21.15.43:0/730236709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc410 0x5622610d6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.787+0000 7efd56000700 1 --2- 172.21.15.43:0/730236709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562260f4de40 0x5622610cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.787+0000 7efd4f400700 1 -- 172.21.15.43:0/943550194 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7efd5000f020 con 0x7efd5810b890 2024-05-27T07:23:49.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.787+0000 7efd57400700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efd3c0b2a70 0x7efd3c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.787+0000 7efd56000700 1 -- 172.21.15.43:0/730236709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562260f4de40 msgr2=0x5622610cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:49.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.787+0000 7efd56000700 1 --2- 172.21.15.43:0/730236709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562260f4de40 0x5622610cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.787+0000 7efd56000700 1 -- 172.21.15.43:0/730236709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc410 msgr2=0x5622610d6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:49.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.787+0000 7efd56000700 1 --2- 172.21.15.43:0/730236709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc410 0x5622610d6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.787+0000 7efd56000700 1 -- 172.21.15.43:0/730236709 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562260fb3420 con 0x5622610cc820 2024-05-27T07:23:49.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.788+0000 7efd56000700 1 --2- 172.21.15.43:0/730236709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610d9020 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efd44005e10 tx=0x7efd4400c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4f0a6dcd311380 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:49.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.788+0000 7efd57400700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efd3c0b2a70 0x7efd3c0b4f80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efd500e1050 tx=0x7efd50009060 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:49.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.788+0000 7efd43e00700 1 -- 172.21.15.43:0/730236709 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd44012450 con 0x5622610cc820 2024-05-27T07:23:49.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.788+0000 7efd43e00700 1 -- 172.21.15.43:0/730236709 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd44012a80 con 0x5622610cc820 2024-05-27T07:23:49.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.788+0000 7efd43e00700 1 -- 172.21.15.43:0/730236709 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd44011730 con 0x5622610cc820 2024-05-27T07:23:49.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.789+0000 7efd6c551b80 1 -- 172.21.15.43:0/730236709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 msgr2=0x5622610d9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.789+0000 7efd6c551b80 1 --2- 172.21.15.43:0/730236709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610d9020 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efd44005e10 tx=0x7efd4400c4b0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.789+0000 7efd6c551b80 1 -- 172.21.15.43:0/730236709 shutdown_connections 2024-05-27T07:23:49.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.789+0000 7efd6c551b80 1 --2- 172.21.15.43:0/730236709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610d9020 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.789+0000 7efd6c551b80 1 --2- 172.21.15.43:0/730236709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562260f4de40 0x5622610cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.789+0000 7efd6c551b80 1 --2- 172.21.15.43:0/730236709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc410 0x5622610d6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.789+0000 7efd6c551b80 1 -- 172.21.15.43:0/730236709 >> 172.21.15.43:0/730236709 conn(0x562260fadfc0 msgr2=0x5622610c7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:49.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.789+0000 7efd6c551b80 1 -- 172.21.15.43:0/730236709 shutdown_connections 2024-05-27T07:23:49.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.789+0000 7efd6c551b80 1 -- 172.21.15.43:0/730236709 wait complete. 2024-05-27T07:23:49.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.790+0000 7efd6c551b80 1 -- start start 2024-05-27T07:23:49.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.790+0000 7efd6c551b80 1 Processor -- start 2024-05-27T07:23:49.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562260f4de40 0x5622610d6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5622610cc410 0x5622610d6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc820 0x5622610d7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd6c551b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562260ef0fa0 con 0x562260f4de40 2024-05-27T07:23:49.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd6c551b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562260f8b760 con 0x5622610cc820 2024-05-27T07:23:49.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd6c551b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5622610d9990 con 0x5622610cc410 2024-05-27T07:23:49.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd6c551b80 10 client.0 _ll_register_callbacks cb 0x562260f94c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:49.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd56000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562260f4de40 0x5622610d6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd56000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562260f4de40 0x5622610d6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53174/0 (socket says 172.21.15.43:53174) 2024-05-27T07:23:49.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd56000700 1 -- 172.21.15.43:0/2109124910 learned_addr learned my addr 172.21.15.43:0/2109124910 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:49.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.791+0000 7efd56a00700 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5622610cc410 0x5622610d6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.792+0000 7efd56000700 1 -- 172.21.15.43:0/2109124910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5622610cc410 msgr2=0x5622610d6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:49.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.792+0000 7efd56000700 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5622610cc410 0x5622610d6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.792+0000 7efd56000700 1 -- 172.21.15.43:0/2109124910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc820 msgr2=0x5622610d7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:49.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.792+0000 7efd56000700 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc820 0x5622610d7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.792+0000 7efd56000700 1 -- 172.21.15.43:0/2109124910 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562260f3a8d0 con 0x562260f4de40 2024-05-27T07:23:49.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.792+0000 7efd56000700 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562260f4de40 0x5622610d6580 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7efd44001eb0 tx=0x7efd44014340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d674c6113f1fa207 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:49.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.792+0000 7efd40c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.793+0000 7efd40c00700 1 -- 172.21.15.43:0/2109124910 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd44004230 con 0x562260f4de40 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.793+0000 7efd40c00700 1 -- 172.21.15.43:0/2109124910 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7efd440043d0 con 0x562260f4de40 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.793+0000 7efd6c551b80 1 -- 172.21.15.43:0/2109124910 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd44012a80 con 0x562260f4de40 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.793+0000 7efd40c00700 1 -- 172.21.15.43:0/2109124910 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7efd44003680 con 0x562260f4de40 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.793+0000 7efd6c551b80 10 client.16935 fetch_fsmap learned FSMap version 103 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.793+0000 7efd6c551b80 1 -- 172.21.15.43:0/2109124910 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x562260fbb060 con 0x562260f4de40 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.794+0000 7efd40c00700 1 -- 172.21.15.43:0/2109124910 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd44012a80 con 0x562260f4de40 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.794+0000 7efd40c00700 1 -- 172.21.15.43:0/2109124910 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 103) ==== 4792+0+0 (secure 0 0 0) 0x7efd440277e0 con 0x562260f4de40 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.794+0000 7efd6c551b80 10 client.16935 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.794+0000 7efd6c551b80 10 client.16935 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:49.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.794+0000 7efd6c551b80 20 client.16935 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:49.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.794+0000 7efd6c551b80 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5622610e6ba0 0x5622610e9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.794+0000 7efd6c551b80 4 client.16935 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:49.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.794+0000 7efd6c551b80 1 -- 172.21.15.43:0/2109124910 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562260ffcd90 con 0x5622610e6ba0 2024-05-27T07:23:49.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.795+0000 7efd56a00700 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5622610e6ba0 0x5622610e9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.796+0000 7efd56a00700 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5622610e6ba0 0x5622610e9010 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce84b63b1ba04261 server_cookie=458a89b1fe853ea in_seq=0 out_seq=0 2024-05-27T07:23:49.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.796+0000 7efd40c00700 10 client.16935 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:49.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd40c00700 1 -- 172.21.15.43:0/2109124910 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562260ffcd90 con 0x5622610e6ba0 2024-05-27T07:23:49.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd40c00700 10 client.16935 handle_command_reply: tid=0 2024-05-27T07:23:49.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd6c551b80 1 client.16935 shutdown 2024-05-27T07:23:49.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd6c551b80 1 -- 172.21.15.43:0/2109124910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562260f4de40 msgr2=0x5622610d6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd6c551b80 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562260f4de40 0x5622610d6580 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7efd44001eb0 tx=0x7efd44014340 comp rx=0 tx=0).stop 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd6c551b80 1 -- 172.21.15.43:0/2109124910 shutdown_connections 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd6c551b80 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc820 0x5622610d7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd6c551b80 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5622610e6ba0 0x5622610e9010 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd6c551b80 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562260f4de40 0x5622610d6580 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd6c551b80 1 --2- 172.21.15.43:0/2109124910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5622610cc410 0x5622610d6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd6c551b80 1 -- 172.21.15.43:0/2109124910 >> 172.21.15.43:0/2109124910 conn(0x562260fadfc0 msgr2=0x5622610c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.799+0000 7efd40c00700 0 client.16935 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.800+0000 7efd6c551b80 1 -- 172.21.15.43:0/2109124910 shutdown_connections 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.800+0000 7efd6c551b80 1 -- 172.21.15.43:0/2109124910 wait complete. 2024-05-27T07:23:49.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.800+0000 7efd6c551b80 20 client.16935 trim_cache size 0 max 16384 2024-05-27T07:23:49.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.818+0000 7efd6c551b80 1 Processor -- start 2024-05-27T07:23:49.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.818+0000 7efd6c551b80 1 -- start start 2024-05-27T07:23:49.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.818+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562260f4de40 0x5622610d1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.818+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc410 0x5622610e2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.818+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5622610cc820 0x5622610d6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.818+0000 7efd6c551b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efd581b74f0 con 0x5622610cc410 2024-05-27T07:23:49.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.818+0000 7efd6c551b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efd581b5250 con 0x562260f4de40 2024-05-27T07:23:49.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.818+0000 7efd6c551b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efd581b6e20 con 0x5622610cc820 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc410 0x5622610e2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc410 0x5622610e2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53176/0 (socket says 172.21.15.43:53176) 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56a00700 1 -- 172.21.15.43:0/1414998056 learned_addr learned my addr 172.21.15.43:0/1414998056 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56000700 1 --2- 172.21.15.43:0/1414998056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562260f4de40 0x5622610d1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56a00700 1 -- 172.21.15.43:0/1414998056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5622610cc820 msgr2=0x5622610d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56a00700 1 --2- 172.21.15.43:0/1414998056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5622610cc820 0x5622610d6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56a00700 1 -- 172.21.15.43:0/1414998056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562260f4de40 msgr2=0x5622610d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56a00700 1 --2- 172.21.15.43:0/1414998056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562260f4de40 0x5622610d1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56a00700 1 -- 172.21.15.43:0/1414998056 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622610c9350 con 0x5622610cc410 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56000700 1 --2- 172.21.15.43:0/1414998056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562260f4de40 0x5622610d1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.819+0000 7efd56a00700 1 --2- 172.21.15.43:0/1414998056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc410 0x5622610e2a60 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x5622610cd430 tx=0x7efd48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d98f91135ad081b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:49.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.820+0000 7efd43400700 1 -- 172.21.15.43:0/1414998056 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48004030 con 0x5622610cc410 2024-05-27T07:23:49.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.820+0000 7efd43400700 1 -- 172.21.15.43:0/1414998056 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd48003680 con 0x5622610cc410 2024-05-27T07:23:49.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.820+0000 7efd43400700 1 -- 172.21.15.43:0/1414998056 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd480116a0 con 0x5622610cc410 2024-05-27T07:23:49.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.821+0000 7efd6c551b80 1 -- 172.21.15.43:0/1414998056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc410 msgr2=0x5622610e2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.821+0000 7efd6c551b80 1 --2- 172.21.15.43:0/1414998056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc410 0x5622610e2a60 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x5622610cd430 tx=0x7efd48004750 comp rx=0 tx=0).stop 2024-05-27T07:23:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.821+0000 7efd6c551b80 1 -- 172.21.15.43:0/1414998056 shutdown_connections 2024-05-27T07:23:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.821+0000 7efd6c551b80 1 --2- 172.21.15.43:0/1414998056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5622610cc820 0x5622610d6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.821+0000 7efd6c551b80 1 --2- 172.21.15.43:0/1414998056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562260f4de40 0x5622610d1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.821+0000 7efd6c551b80 1 --2- 172.21.15.43:0/1414998056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc410 0x5622610e2a60 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.821+0000 7efd6c551b80 1 -- 172.21.15.43:0/1414998056 >> 172.21.15.43:0/1414998056 conn(0x562260fadfc0 msgr2=0x5622610c7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.821+0000 7efd6c551b80 1 -- 172.21.15.43:0/1414998056 shutdown_connections 2024-05-27T07:23:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.821+0000 7efd6c551b80 1 -- 172.21.15.43:0/1414998056 wait complete. 2024-05-27T07:23:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.822+0000 7efd6c551b80 1 -- start start 2024-05-27T07:23:49.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.822+0000 7efd6c551b80 1 Processor -- start 2024-05-27T07:23:49.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.822+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562260f4de40 0x5622610d9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.822+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc410 0x5622610da150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.822+0000 7efd6c551b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610da690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.822+0000 7efd6c551b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efd581b5ff0 con 0x5622610cc820 2024-05-27T07:23:49.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd6c551b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efd581b5920 con 0x5622610cc410 2024-05-27T07:23:49.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd6c551b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efd581b6720 con 0x562260f4de40 2024-05-27T07:23:49.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd6c551b80 10 client.0 _ll_register_callbacks cb 0x562260f94c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:49.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd56000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610da690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd56000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610da690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53190/0 (socket says 172.21.15.43:53190) 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd56000700 1 -- 172.21.15.43:0/3693646996 learned_addr learned my addr 172.21.15.43:0/3693646996 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd56a00700 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc410 0x5622610da150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd56000700 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562260f4de40 0x5622610d9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd56000700 1 -- 172.21.15.43:0/3693646996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562260f4de40 msgr2=0x5622610d9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd56000700 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562260f4de40 0x5622610d9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd56000700 1 -- 172.21.15.43:0/3693646996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc410 msgr2=0x5622610da150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd56000700 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc410 0x5622610da150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.823+0000 7efd56000700 1 -- 172.21.15.43:0/3693646996 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56226114f690 con 0x5622610cc820 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.824+0000 7efd56000700 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610da690 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7efd44014970 tx=0x7efd44011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58a078c1bc6a077 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.824+0000 7efd40c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.824+0000 7efd40c00700 1 -- 172.21.15.43:0/3693646996 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd44004410 con 0x5622610cc820 2024-05-27T07:23:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.824+0000 7efd6c551b80 1 -- 172.21.15.43:0/3693646996 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd48003680 con 0x5622610cc820 2024-05-27T07:23:49.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.824+0000 7efd40c00700 1 -- 172.21.15.43:0/3693646996 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7efd44027440 con 0x5622610cc820 2024-05-27T07:23:49.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.824+0000 7efd40c00700 1 -- 172.21.15.43:0/3693646996 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7efd44012a80 con 0x5622610cc820 2024-05-27T07:23:49.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.824+0000 7efd6c551b80 10 client.16941 fetch_fsmap learned FSMap version 103 2024-05-27T07:23:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.824+0000 7efd6c551b80 1 -- 172.21.15.43:0/3693646996 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=4+}) -- 0x562261150780 con 0x5622610cc820 2024-05-27T07:23:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.825+0000 7efd40c00700 1 -- 172.21.15.43:0/3693646996 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd48003680 con 0x5622610cc820 2024-05-27T07:23:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.825+0000 7efd40c00700 1 -- 172.21.15.43:0/3693646996 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 103) ==== 4792+0+0 (secure 0 0 0) 0x7efd44003b20 con 0x5622610cc820 2024-05-27T07:23:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.825+0000 7efd6c551b80 10 client.16941 fetch_fsmap finished waiting for FSMap version 103 2024-05-27T07:23:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.825+0000 7efd6c551b80 10 client.16941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.825+0000 7efd6c551b80 20 client.16941 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.825+0000 7efd6c551b80 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562261150e30 0x562261153250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.825+0000 7efd6c551b80 4 client.16941 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:23:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.825+0000 7efd6c551b80 1 -- 172.21.15.43:0/3693646996 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562260ff6570 con 0x562261150e30 2024-05-27T07:23:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.826+0000 7efd56a00700 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562261150e30 0x562261153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:49.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.827+0000 7efd56a00700 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562261150e30 0x562261153250 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b53af96190abc1c5 server_cookie=5ae0a0e0b74307db in_seq=0 out_seq=0 2024-05-27T07:23:49.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.827+0000 7efd40c00700 10 client.16941 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd40c00700 1 -- 172.21.15.43:0/3693646996 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562260ff6570 con 0x562261150e30 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd40c00700 10 client.16941 handle_command_reply: tid=0 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 client.16941 shutdown 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 -- 172.21.15.43:0/3693646996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 msgr2=0x5622610da690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610da690 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7efd44014970 tx=0x7efd44011440 comp rx=0 tx=0).stop 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 -- 172.21.15.43:0/3693646996 shutdown_connections 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562261150e30 0x562261153250 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622610cc820 0x5622610da690 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562260f4de40 0x5622610d9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 --2- 172.21.15.43:0/3693646996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5622610cc410 0x5622610da150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 -- 172.21.15.43:0/3693646996 >> 172.21.15.43:0/3693646996 conn(0x562260fadfc0 msgr2=0x5622610c7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd40c00700 0 client.16941 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:49.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 -- 172.21.15.43:0/3693646996 shutdown_connections 2024-05-27T07:23:49.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 1 -- 172.21.15.43:0/3693646996 wait complete. 2024-05-27T07:23:49.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.828+0000 7efd6c551b80 20 client.16941 trim_cache size 0 max 16384 2024-05-27T07:23:49.820 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:23:49.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.829+0000 7efd57e00700 1 -- 172.21.15.43:0/943550194 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efd3c0b2a70 msgr2=0x7efd3c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.829+0000 7efd57e00700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efd3c0b2a70 0x7efd3c0b4f80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efd500e1050 tx=0x7efd50009060 comp rx=0 tx=0).stop 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.829+0000 7efd57e00700 1 -- 172.21.15.43:0/943550194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd5810b890 msgr2=0x7efd58111090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.829+0000 7efd57e00700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd5810b890 0x7efd58111090 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efd50009850 tx=0x7efd50007e40 comp rx=0 tx=0).stop 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.830+0000 7efd57e00700 1 -- 172.21.15.43:0/943550194 shutdown_connections 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.830+0000 7efd57e00700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efd3c0b2a70 0x7efd3c0b4f80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.830+0000 7efd57e00700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efd5810b890 0x7efd58111090 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.830+0000 7efd57e00700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd58106610 0x7efd58110610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.830+0000 7efd57e00700 1 --2- 172.21.15.43:0/943550194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd5810af40 0x7efd58110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.830+0000 7efd57e00700 1 -- 172.21.15.43:0/943550194 >> 172.21.15.43:0/943550194 conn(0x7efd58102020 msgr2=0x7efd5810a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.830+0000 7efd57e00700 1 -- 172.21.15.43:0/943550194 shutdown_connections 2024-05-27T07:23:49.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:49.830+0000 7efd57e00700 1 -- 172.21.15.43:0/943550194 wait complete. 2024-05-27T07:23:50.224 INFO:tasks.workunit.client.0.smithi043.stdout: 1 54775 0.61 MB/sec execute 354 sec latency 69.746 ms 2024-05-27T07:23:50.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:50 smithi043 ceph-mon[31531]: pgmap v336: 129 pgs: 129 active+clean; 94 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 402 KiB/s wr, 45 op/s 2024-05-27T07:23:50.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:50 smithi179 ceph-mon[35888]: pgmap v336: 129 pgs: 129 active+clean; 94 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 402 KiB/s wr, 45 op/s 2024-05-27T07:23:50.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:50 smithi132 ceph-mon[35283]: pgmap v336: 129 pgs: 129 active+clean; 94 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 402 KiB/s wr, 45 op/s 2024-05-27T07:23:51.224 INFO:tasks.workunit.client.0.smithi043.stdout: 1 54885 0.61 MB/sec execute 355 sec latency 72.702 ms 2024-05-27T07:23:51.650 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:23:51.650 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:23:51.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:51 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:51.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:51 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:51.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:51 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:52.143 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:52.224 INFO:tasks.workunit.client.0.smithi043.stdout: 1 55045 0.61 MB/sec execute 356 sec latency 64.283 ms 2024-05-27T07:23:52.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:52 smithi043 ceph-mon[31531]: pgmap v337: 129 pgs: 129 active+clean; 95 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 592 KiB/s wr, 64 op/s 2024-05-27T07:23:52.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:52 smithi179 ceph-mon[35888]: pgmap v337: 129 pgs: 129 active+clean; 95 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 592 KiB/s wr, 64 op/s 2024-05-27T07:23:52.741 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:52 smithi132 ceph-mon[35283]: pgmap v337: 129 pgs: 129 active+clean; 95 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 592 KiB/s wr, 64 op/s 2024-05-27T07:23:53.226 INFO:tasks.workunit.client.0.smithi043.stdout: 1 55151 0.60 MB/sec execute 357 sec latency 52.437 ms 2024-05-27T07:23:53.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:53 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:53.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:53 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:53.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:53 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:54.224 INFO:tasks.workunit.client.0.smithi043.stdout: 1 55293 0.60 MB/sec execute 358 sec latency 66.723 ms 2024-05-27T07:23:54.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:54 smithi043 ceph-mon[31531]: pgmap v338: 129 pgs: 129 active+clean; 95 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 340 KiB/s wr, 42 op/s 2024-05-27T07:23:54.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:54 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:54.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.816+0000 7fbae6600700 1 -- 172.21.15.43:0/1377178746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbae0110bf0 msgr2=0x7fbae0113010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.816+0000 7fbae6600700 1 --2- 172.21.15.43:0/1377178746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbae0110bf0 0x7fbae0113010 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fbad800b0a0 tx=0x7fbad800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.817+0000 7fbae6600700 1 -- 172.21.15.43:0/1377178746 shutdown_connections 2024-05-27T07:23:54.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.817+0000 7fbae6600700 1 --2- 172.21.15.43:0/1377178746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbae0110bf0 0x7fbae0113010 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.817+0000 7fbae6600700 1 --2- 172.21.15.43:0/1377178746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbae0075af0 0x7fbae0075f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.817+0000 7fbae6600700 1 --2- 172.21.15.43:0/1377178746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae0076440 0x7fbae01106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.817+0000 7fbae6600700 1 -- 172.21.15.43:0/1377178746 >> 172.21.15.43:0/1377178746 conn(0x7fbae0100030 msgr2=0x7fbae0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:54.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.817+0000 7fbae6600700 1 -- 172.21.15.43:0/1377178746 shutdown_connections 2024-05-27T07:23:54.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.817+0000 7fbae6600700 1 -- 172.21.15.43:0/1377178746 wait complete. 2024-05-27T07:23:54.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.818+0000 7fbae6600700 1 Processor -- start 2024-05-27T07:23:54.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.818+0000 7fbae6600700 1 -- start start 2024-05-27T07:23:54.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.819+0000 7fbae6600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae0075af0 0x7fbae019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.819+0000 7fbae6600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbae0076440 0x7fbae019d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.819+0000 7fbae6600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbae0110bf0 0x7fbae01a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.819+0000 7fbae6600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbae0116560 con 0x7fbae0110bf0 2024-05-27T07:23:54.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.819+0000 7fbae6600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbae01163e0 con 0x7fbae0076440 2024-05-27T07:23:54.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.819+0000 7fbae6600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbae01166e0 con 0x7fbae0075af0 2024-05-27T07:23:54.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.819+0000 7fbadfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbae0110bf0 0x7fbae01a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.819+0000 7fbadfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbae0110bf0 0x7fbae01a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56292/0 (socket says 172.21.15.43:56292) 2024-05-27T07:23:54.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.819+0000 7fbadfe00700 1 -- 172.21.15.43:0/1010249698 learned_addr learned my addr 172.21.15.43:0/1010249698 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:54.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.819+0000 7fbadf400700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae0075af0 0x7fbae019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.820+0000 7fbadea00700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbae0076440 0x7fbae019d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.820+0000 7fbadfe00700 1 -- 172.21.15.43:0/1010249698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae0075af0 msgr2=0x7fbae019ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.820+0000 7fbadfe00700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae0075af0 0x7fbae019ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.820+0000 7fbadfe00700 1 -- 172.21.15.43:0/1010249698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbae0076440 msgr2=0x7fbae019d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.820+0000 7fbadfe00700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbae0076440 0x7fbae019d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.820+0000 7fbadfe00700 1 -- 172.21.15.43:0/1010249698 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae01a1da0 con 0x7fbae0110bf0 2024-05-27T07:23:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.820+0000 7fbadfe00700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbae0110bf0 0x7fbae01a1700 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fbad800b830 tx=0x7fbad8004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.821+0000 7fbad7e00700 1 -- 172.21.15.43:0/1010249698 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad8021070 con 0x7fbae0110bf0 2024-05-27T07:23:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.821+0000 7fbad7e00700 1 -- 172.21.15.43:0/1010249698 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbad8007b80 con 0x7fbae0110bf0 2024-05-27T07:23:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.821+0000 7fbae6600700 1 -- 172.21.15.43:0/1010249698 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae01a2030 con 0x7fbae0110bf0 2024-05-27T07:23:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.821+0000 7fbad7e00700 1 -- 172.21.15.43:0/1010249698 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad80052a0 con 0x7fbae0110bf0 2024-05-27T07:23:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.821+0000 7fbae6600700 1 -- 172.21.15.43:0/1010249698 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae01a99b0 con 0x7fbae0110bf0 2024-05-27T07:23:54.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.826+0000 7fbad7e00700 1 -- 172.21.15.43:0/1010249698 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fbad8024070 con 0x7fbae0110bf0 2024-05-27T07:23:54.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.827+0000 7fbaf50e0b80 1 Processor -- start 2024-05-27T07:23:54.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.827+0000 7fbaf50e0b80 1 -- start start 2024-05-27T07:23:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.828+0000 7fbad7e00700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbac40b2a70 0x7fbac40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.828+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4180ec0 0x559ad4305870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.828+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad4305db0 0x559ad42ff8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.828+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad42ffe30 0x559ad4300240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.828+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559ad422dec0 con 0x559ad42ffe30 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.828+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559ad421da60 con 0x559ad4305db0 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.828+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x559ad4221aa0 con 0x559ad4180ec0 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.828+0000 7fbad7e00700 1 -- 172.21.15.43:0/1010249698 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7fbad800f020 con 0x7fbae0110bf0 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.828+0000 7fbadfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad42ffe30 0x559ad4300240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.829+0000 7fbadfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad42ffe30 0x559ad4300240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56298/0 (socket says 172.21.15.43:56298) 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.829+0000 7fbadfe00700 1 -- 172.21.15.43:0/1615278759 learned_addr learned my addr 172.21.15.43:0/1615278759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.829+0000 7fbadfe00700 1 -- 172.21.15.43:0/1615278759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4180ec0 msgr2=0x559ad4305870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.829+0000 7fbadfe00700 1 --2- 172.21.15.43:0/1615278759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4180ec0 0x559ad4305870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.829+0000 7fbadfe00700 1 -- 172.21.15.43:0/1615278759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad4305db0 msgr2=0x559ad42ff8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.829+0000 7fbadfe00700 1 --2- 172.21.15.43:0/1615278759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad4305db0 0x559ad42ff8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.829+0000 7fbadfe00700 1 -- 172.21.15.43:0/1615278759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ad41e64a0 con 0x559ad42ffe30 2024-05-27T07:23:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.829+0000 7fbadfe00700 1 --2- 172.21.15.43:0/1615278759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad42ffe30 0x559ad4300240 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fbad80aae30 tx=0x7fbad80aa6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ac5f5ad6a2900c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.830+0000 7fbad4c00700 1 -- 172.21.15.43:0/1615278759 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad80aa9b0 con 0x559ad42ffe30 2024-05-27T07:23:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.830+0000 7fbad4c00700 1 -- 172.21.15.43:0/1615278759 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbad80ab5a0 con 0x559ad42ffe30 2024-05-27T07:23:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.830+0000 7fbadea00700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbac40b2a70 0x7fbac40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.830+0000 7fbad4c00700 1 -- 172.21.15.43:0/1615278759 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad80e1500 con 0x559ad42ffe30 2024-05-27T07:23:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.831+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1615278759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad42ffe30 msgr2=0x559ad4300240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.831+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/1615278759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad42ffe30 0x559ad4300240 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fbad80aae30 tx=0x7fbad80aa6e0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.831+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1615278759 shutdown_connections 2024-05-27T07:23:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.831+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/1615278759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad42ffe30 0x559ad4300240 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.831+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/1615278759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4180ec0 0x559ad4305870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.831+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/1615278759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad4305db0 0x559ad42ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.831+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1615278759 >> 172.21.15.43:0/1615278759 conn(0x559ad41e1040 msgr2=0x559ad42fa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.831+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1615278759 shutdown_connections 2024-05-27T07:23:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.831+0000 7fbadea00700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbac40b2a70 0x7fbac40b4f80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00b6c0 tx=0x7fbacc00b410 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.831+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1615278759 wait complete. 2024-05-27T07:23:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.832+0000 7fbaf50e0b80 1 -- start start 2024-05-27T07:23:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.832+0000 7fbaf50e0b80 1 Processor -- start 2024-05-27T07:23:54.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.833+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad4309520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.833+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad4309a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.833+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 0x559ad4309fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.833+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559ad4124020 con 0x559ad4180ec0 2024-05-27T07:23:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.833+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559ad41be7e0 con 0x559ad42ffe30 2024-05-27T07:23:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.833+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x559ad430c640 con 0x559ad4305db0 2024-05-27T07:23:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.833+0000 7fbaf50e0b80 10 client.0 _ll_register_callbacks cb 0x559ad41c7cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.833+0000 7fbadf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad4309520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.833+0000 7fbadf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad4309520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56308/0 (socket says 172.21.15.43:56308) 2024-05-27T07:23:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.833+0000 7fbadf400700 1 -- 172.21.15.43:0/1278518338 learned_addr learned my addr 172.21.15.43:0/1278518338 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.834+0000 7fbadfe00700 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 0x559ad4309fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.834+0000 7fbadf400700 1 -- 172.21.15.43:0/1278518338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 msgr2=0x559ad4309fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.834+0000 7fbadf400700 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 0x559ad4309fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.834+0000 7fbadf400700 1 -- 172.21.15.43:0/1278518338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 msgr2=0x559ad4309a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:23:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.834+0000 7fbadf400700 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad4309a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.834+0000 7fbadf400700 1 -- 172.21.15.43:0/1278518338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ad416d950 con 0x559ad4180ec0 2024-05-27T07:23:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.834+0000 7fbadfe00700 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 0x559ad4309fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.835+0000 7fbadf400700 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad4309520 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fbad000bf00 tx=0x7fbad00106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487c0e25ef1ad9ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:54.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.835+0000 7fbac9600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:54.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.835+0000 7fbac9600700 1 -- 172.21.15.43:0/1278518338 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad001c680 con 0x559ad4180ec0 2024-05-27T07:23:54.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.835+0000 7fbac9600700 1 -- 172.21.15.43:0/1278518338 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7fbad0011440 con 0x559ad4180ec0 2024-05-27T07:23:54.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.835+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1278518338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbad80ab5a0 con 0x559ad4180ec0 2024-05-27T07:23:54.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.836+0000 7fbac9600700 1 -- 172.21.15.43:0/1278518338 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fbad001c820 con 0x559ad4180ec0 2024-05-27T07:23:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.836+0000 7fbaf50e0b80 10 client.16953 fetch_fsmap learned FSMap version 106 2024-05-27T07:23:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.836+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1278518338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x559ad41ee0e0 con 0x559ad4180ec0 2024-05-27T07:23:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.836+0000 7fbac9600700 1 -- 172.21.15.43:0/1278518338 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbad80ab5a0 con 0x559ad4180ec0 2024-05-27T07:23:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.837+0000 7fbac9600700 1 -- 172.21.15.43:0/1278518338 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 106) ==== 4804+0+0 (secure 0 0 0) 0x7fbad0011cd0 con 0x559ad4180ec0 2024-05-27T07:23:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.837+0000 7fbaf50e0b80 10 client.16953 fetch_fsmap finished waiting for FSMap version 106 2024-05-27T07:23:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.837+0000 7fbaf50e0b80 10 client.16953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.837+0000 7fbaf50e0b80 20 client.16953 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.837+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559ad4319a00 0x559ad431bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.837+0000 7fbaf50e0b80 4 client.16953 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.837+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1278518338 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ad422fe10 con 0x559ad4319a00 2024-05-27T07:23:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.837+0000 7fbadea00700 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559ad4319a00 0x559ad431bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.839+0000 7fbadea00700 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559ad4319a00 0x559ad431bed0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e11e278b8117d9a server_cookie=db70c4eca361a517 in_seq=0 out_seq=0 2024-05-27T07:23:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.839+0000 7fbac9600700 10 client.16953 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:54.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbac9600700 1 -- 172.21.15.43:0/1278518338 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559ad422fe10 con 0x559ad4319a00 2024-05-27T07:23:54.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbac9600700 10 client.16953 handle_command_reply: tid=0 2024-05-27T07:23:54.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbaf50e0b80 1 client.16953 shutdown 2024-05-27T07:23:54.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1278518338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 msgr2=0x559ad4309520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad4309520 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fbad000bf00 tx=0x7fbad00106b0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1278518338 shutdown_connections 2024-05-27T07:23:54.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 0x559ad4309fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559ad4319a00 0x559ad431bed0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad4309520 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/1278518338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad4309a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1278518338 >> 172.21.15.43:0/1278518338 conn(0x559ad41e1040 msgr2=0x559ad42fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.841+0000 7fbac9600700 0 client.16953 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.842+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1278518338 shutdown_connections 2024-05-27T07:23:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.842+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/1278518338 wait complete. 2024-05-27T07:23:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.842+0000 7fbaf50e0b80 20 client.16953 trim_cache size 0 max 16384 2024-05-27T07:23:54.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:54 smithi132 ceph-mon[35283]: pgmap v338: 129 pgs: 129 active+clean; 95 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 340 KiB/s wr, 42 op/s 2024-05-27T07:23:54.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:54 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.869+0000 7fbaf50e0b80 1 Processor -- start 2024-05-27T07:23:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.869+0000 7fbaf50e0b80 1 -- start start 2024-05-27T07:23:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.869+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad42ff8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.869+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad4308480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.869+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 0x559ad4315490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.869+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559ad4305080 con 0x559ad4180ec0 2024-05-27T07:23:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.869+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbae01b33d0 con 0x559ad42ffe30 2024-05-27T07:23:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.869+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbae01b10a0 con 0x559ad4305db0 2024-05-27T07:23:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.870+0000 7fbadf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad42ff8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.870+0000 7fbadf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad42ff8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56310/0 (socket says 172.21.15.43:56310) 2024-05-27T07:23:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.870+0000 7fbadf400700 1 -- 172.21.15.43:0/3113706183 learned_addr learned my addr 172.21.15.43:0/3113706183 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:54.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.870+0000 7fbadfe00700 1 --2- 172.21.15.43:0/3113706183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 0x559ad4315490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.870+0000 7fbadea00700 1 --2- 172.21.15.43:0/3113706183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad4308480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.870+0000 7fbadf400700 1 -- 172.21.15.43:0/3113706183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 msgr2=0x559ad4315490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:54.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.870+0000 7fbadf400700 1 --2- 172.21.15.43:0/3113706183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 0x559ad4315490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.870+0000 7fbadf400700 1 -- 172.21.15.43:0/3113706183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 msgr2=0x559ad4308480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:54.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.870+0000 7fbadf400700 1 --2- 172.21.15.43:0/3113706183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad4308480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.870+0000 7fbadf400700 1 -- 172.21.15.43:0/3113706183 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ad4305200 con 0x559ad4180ec0 2024-05-27T07:23:54.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.871+0000 7fbadf400700 1 --2- 172.21.15.43:0/3113706183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad42ff8f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbad001a5f0 tx=0x7fbad000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29280c65642355d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:54.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.871+0000 7fbacbe00700 1 -- 172.21.15.43:0/3113706183 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad002d070 con 0x559ad4180ec0 2024-05-27T07:23:54.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.871+0000 7fbacbe00700 1 -- 172.21.15.43:0/3113706183 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbad001a670 con 0x559ad4180ec0 2024-05-27T07:23:54.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.871+0000 7fbacbe00700 1 -- 172.21.15.43:0/3113706183 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad00052a0 con 0x559ad4180ec0 2024-05-27T07:23:54.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.872+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/3113706183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 msgr2=0x559ad42ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.872+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/3113706183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad42ff8f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbad001a5f0 tx=0x7fbad000b480 comp rx=0 tx=0).stop 2024-05-27T07:23:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.872+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/3113706183 shutdown_connections 2024-05-27T07:23:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.872+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/3113706183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4305db0 0x559ad4315490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.872+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/3113706183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4180ec0 0x559ad42ff8f0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.872+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/3113706183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad4308480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.872+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/3113706183 >> 172.21.15.43:0/3113706183 conn(0x559ad41e1040 msgr2=0x559ad4309d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.873+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/3113706183 shutdown_connections 2024-05-27T07:23:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.873+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/3113706183 wait complete. 2024-05-27T07:23:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.873+0000 7fbaf50e0b80 1 -- start start 2024-05-27T07:23:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.874+0000 7fbaf50e0b80 1 Processor -- start 2024-05-27T07:23:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.874+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4180ec0 0x559ad43212a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.874+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad43217e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.874+0000 7fbaf50e0b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4305db0 0x559ad4321d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.874+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbae01b1770 con 0x559ad4305db0 2024-05-27T07:23:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.874+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbae01b2d00 con 0x559ad42ffe30 2024-05-27T07:23:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.874+0000 7fbaf50e0b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbae01b1e70 con 0x559ad4180ec0 2024-05-27T07:23:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.874+0000 7fbaf50e0b80 10 client.0 _ll_register_callbacks cb 0x559ad431c490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.875+0000 7fbadfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4305db0 0x559ad4321d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.875+0000 7fbadfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4305db0 0x559ad4321d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56316/0 (socket says 172.21.15.43:56316) 2024-05-27T07:23:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.875+0000 7fbadfe00700 1 -- 172.21.15.43:0/2871496675 learned_addr learned my addr 172.21.15.43:0/2871496675 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.875+0000 7fbadea00700 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad43217e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.875+0000 7fbadf400700 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4180ec0 0x559ad43212a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.875+0000 7fbadfe00700 1 -- 172.21.15.43:0/2871496675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4180ec0 msgr2=0x559ad43212a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.875+0000 7fbadfe00700 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4180ec0 0x559ad43212a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.875+0000 7fbadfe00700 1 -- 172.21.15.43:0/2871496675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 msgr2=0x559ad43217e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.875+0000 7fbadfe00700 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad43217e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.875+0000 7fbadfe00700 1 -- 172.21.15.43:0/2871496675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ad43835b0 con 0x559ad4305db0 2024-05-27T07:23:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.876+0000 7fbadfe00700 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4305db0 0x559ad4321d20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbad80062a0 tx=0x7fbad8024610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e58869f327c0064 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.876+0000 7fbac9600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.876+0000 7fbac9600700 1 -- 172.21.15.43:0/2871496675 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad80aa9b0 con 0x559ad4305db0 2024-05-27T07:23:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.876+0000 7fbac9600700 1 -- 172.21.15.43:0/2871496675 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7fbad8007d30 con 0x559ad4305db0 2024-05-27T07:23:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.876+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/2871496675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbad001a670 con 0x559ad4305db0 2024-05-27T07:23:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.876+0000 7fbac9600700 1 -- 172.21.15.43:0/2871496675 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7fbad80a6d60 con 0x559ad4305db0 2024-05-27T07:23:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.876+0000 7fbaf50e0b80 10 client.16959 fetch_fsmap learned FSMap version 106 2024-05-27T07:23:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.877+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/2871496675 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=4+}) -- 0x559ad4384850 con 0x559ad4305db0 2024-05-27T07:23:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.877+0000 7fbac9600700 1 -- 172.21.15.43:0/2871496675 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbad001a670 con 0x559ad4305db0 2024-05-27T07:23:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.877+0000 7fbac9600700 1 -- 172.21.15.43:0/2871496675 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 106) ==== 4804+0+0 (secure 0 0 0) 0x7fbad8011070 con 0x559ad4305db0 2024-05-27T07:23:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.877+0000 7fbaf50e0b80 10 client.16959 fetch_fsmap finished waiting for FSMap version 106 2024-05-27T07:23:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.877+0000 7fbaf50e0b80 10 client.16959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.877+0000 7fbaf50e0b80 20 client.16959 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.877+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559ad4384c90 0x559ad43870b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.877+0000 7fbaf50e0b80 4 client.16959 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:23:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.877+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/2871496675 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559ad42295f0 con 0x559ad4384c90 2024-05-27T07:23:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.878+0000 7fbadf400700 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559ad4384c90 0x559ad43870b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.879+0000 7fbadf400700 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559ad4384c90 0x559ad43870b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79023ced9fd52b2b server_cookie=35b700642c1a44f3 in_seq=0 out_seq=0 2024-05-27T07:23:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.879+0000 7fbac9600700 10 client.16959 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbac9600700 1 -- 172.21.15.43:0/2871496675 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559ad42295f0 con 0x559ad4384c90 2024-05-27T07:23:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbac9600700 10 client.16959 handle_command_reply: tid=0 2024-05-27T07:23:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbaf50e0b80 1 client.16959 shutdown 2024-05-27T07:23:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/2871496675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4305db0 msgr2=0x559ad4321d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4305db0 0x559ad4321d20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbad80062a0 tx=0x7fbad8024610 comp rx=0 tx=0).stop 2024-05-27T07:23:54.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/2871496675 shutdown_connections 2024-05-27T07:23:54.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559ad4384c90 0x559ad43870b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ad4305db0 0x559ad4321d20 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559ad4180ec0 0x559ad43212a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbaf50e0b80 1 --2- 172.21.15.43:0/2871496675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559ad42ffe30 0x559ad43217e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbac9600700 0 client.16959 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:54.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.885+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/2871496675 >> 172.21.15.43:0/2871496675 conn(0x559ad41e1040 msgr2=0x559ad4309d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:54.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.886+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/2871496675 shutdown_connections 2024-05-27T07:23:54.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.886+0000 7fbaf50e0b80 1 -- 172.21.15.43:0/2871496675 wait complete. 2024-05-27T07:23:54.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.887+0000 7fbaf50e0b80 20 client.16959 trim_cache size 0 max 16384 2024-05-27T07:23:54.877 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:23:54.877 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:23:54.877 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "bc6bd47d-cc04-4d60-8abf-00c9d990f2ff", 2024-05-27T07:23:54.877 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:23:54.877 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:23:54.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.887+0000 7fbae6600700 1 -- 172.21.15.43:0/1010249698 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbac40b2a70 msgr2=0x7fbac40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.887+0000 7fbae6600700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbac40b2a70 0x7fbac40b4f80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbacc00b6c0 tx=0x7fbacc00b410 comp rx=0 tx=0).stop 2024-05-27T07:23:54.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.887+0000 7fbae6600700 1 -- 172.21.15.43:0/1010249698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbae0110bf0 msgr2=0x7fbae01a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:54.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.887+0000 7fbae6600700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbae0110bf0 0x7fbae01a1700 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fbad800b830 tx=0x7fbad8004d00 comp rx=0 tx=0).stop 2024-05-27T07:23:54.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.888+0000 7fbae6600700 1 -- 172.21.15.43:0/1010249698 shutdown_connections 2024-05-27T07:23:54.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.888+0000 7fbae6600700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbac40b2a70 0x7fbac40b4f80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.888+0000 7fbae6600700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbae0110bf0 0x7fbae01a1700 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.888+0000 7fbae6600700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae0075af0 0x7fbae019ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.888+0000 7fbae6600700 1 --2- 172.21.15.43:0/1010249698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbae0076440 0x7fbae019d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.888+0000 7fbae6600700 1 -- 172.21.15.43:0/1010249698 >> 172.21.15.43:0/1010249698 conn(0x7fbae0100030 msgr2=0x7fbae0100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.888+0000 7fbae6600700 1 -- 172.21.15.43:0/1010249698 shutdown_connections 2024-05-27T07:23:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:54.888+0000 7fbae6600700 1 -- 172.21.15.43:0/1010249698 wait complete. 2024-05-27T07:23:54.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:54 smithi179 ceph-mon[35888]: pgmap v338: 129 pgs: 129 active+clean; 95 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 340 KiB/s wr, 42 op/s 2024-05-27T07:23:54.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:54 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:55.224 INFO:tasks.workunit.client.0.smithi043.stdout: 1 55442 0.60 MB/sec execute 359 sec latency 62.025 ms 2024-05-27T07:23:55.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:55 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:23:55.645 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:55 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:55.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:23:55.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:55 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:23:55.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:55 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:55.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:55 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:23:55.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:55 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:23:56.224 INFO:tasks.workunit.client.0.smithi043.stdout: 1 55602 0.60 MB/sec execute 360 sec latency 49.454 ms 2024-05-27T07:23:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: pgmap v339: 129 pgs: 129 active+clean; 95 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 340 KiB/s wr, 42 op/s 2024-05-27T07:23:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:23:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:23:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:23:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:23:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:23:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.791 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:56 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: pgmap v339: 129 pgs: 129 active+clean; 95 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 340 KiB/s wr, 42 op/s 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:56 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:56.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: pgmap v339: 129 pgs: 129 active+clean; 95 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 340 KiB/s wr, 42 op/s 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:23:56.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:56 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:23:57.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 55679 0.60 MB/sec execute 361 sec latency 48.680 ms 2024-05-27T07:23:57.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:57 smithi132 ceph-mon[35283]: pgmap v340: 129 pgs: 129 active+clean; 96 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 428 KiB/s wr, 57 op/s 2024-05-27T07:23:57.928 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:57 smithi043 ceph-mon[31531]: pgmap v340: 129 pgs: 129 active+clean; 96 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 428 KiB/s wr, 57 op/s 2024-05-27T07:23:57.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:57 smithi179 ceph-mon[35888]: pgmap v340: 129 pgs: 129 active+clean; 96 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 428 KiB/s wr, 57 op/s 2024-05-27T07:23:58.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 55813 0.60 MB/sec execute 362 sec latency 49.361 ms 2024-05-27T07:23:58.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.560+0000 7fecbb200700 1 -- 172.21.15.43:0/1181386269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecb4105bf0 msgr2=0x7fecb410e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.560+0000 7fecbb200700 1 --2- 172.21.15.43:0/1181386269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecb4105bf0 0x7fecb410e140 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7feca4009990 tx=0x7feca4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.561+0000 7fecbb200700 1 -- 172.21.15.43:0/1181386269 shutdown_connections 2024-05-27T07:23:58.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.561+0000 7fecbb200700 1 --2- 172.21.15.43:0/1181386269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecb410e710 0x7fecb4110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.561+0000 7fecbb200700 1 --2- 172.21.15.43:0/1181386269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fecb41052a0 0x7fecb41056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.561+0000 7fecbb200700 1 --2- 172.21.15.43:0/1181386269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecb4105bf0 0x7fecb410e140 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.561+0000 7fecbb200700 1 -- 172.21.15.43:0/1181386269 >> 172.21.15.43:0/1181386269 conn(0x7fecb4078c30 msgr2=0x7fecb4079070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.561+0000 7fecbb200700 1 -- 172.21.15.43:0/1181386269 shutdown_connections 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.561+0000 7fecbb200700 1 -- 172.21.15.43:0/1181386269 wait complete. 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.562+0000 7fecbb200700 1 Processor -- start 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.563+0000 7fecbb200700 1 -- start start 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.563+0000 7fecbb200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecb41052a0 0x7fecb4102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.563+0000 7fecbb200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecb4105bf0 0x7fecb41034d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.563+0000 7fecbb200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fecb410e710 0x7fecb4103a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.563+0000 7fecbb200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fecb4114170 con 0x7fecb4105bf0 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.563+0000 7fecbb200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fecb4113ff0 con 0x7fecb41052a0 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.563+0000 7fecbb200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41142f0 con 0x7fecb410e710 2024-05-27T07:23:58.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.564+0000 7fecb9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecb4105bf0 0x7fecb41034d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.564+0000 7fecb9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecb4105bf0 0x7fecb41034d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56344/0 (socket says 172.21.15.43:56344) 2024-05-27T07:23:58.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.564+0000 7fecb9400700 1 -- 172.21.15.43:0/3835975614 learned_addr learned my addr 172.21.15.43:0/3835975614 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:58.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.564+0000 7fecb9e00700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecb41052a0 0x7fecb4102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.564+0000 7fecba800700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fecb410e710 0x7fecb4103a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.564+0000 7fecb9400700 1 -- 172.21.15.43:0/3835975614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fecb410e710 msgr2=0x7fecb4103a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.564+0000 7fecb9400700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fecb410e710 0x7fecb4103a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.564+0000 7fecb9400700 1 -- 172.21.15.43:0/3835975614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecb41052a0 msgr2=0x7fecb4102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.564+0000 7fecb9400700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecb41052a0 0x7fecb4102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.564+0000 7fecb9400700 1 -- 172.21.15.43:0/3835975614 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb41a8f40 con 0x7fecb4105bf0 2024-05-27T07:23:58.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.565+0000 7fecb9400700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecb4105bf0 0x7fecb41034d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7feca4005910 tx=0x7feca40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:58.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.565+0000 7fecaea00700 1 -- 172.21.15.43:0/3835975614 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca4027070 con 0x7fecb4105bf0 2024-05-27T07:23:58.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.565+0000 7fecaea00700 1 -- 172.21.15.43:0/3835975614 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feca400fb00 con 0x7fecb4105bf0 2024-05-27T07:23:58.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.565+0000 7fecaea00700 1 -- 172.21.15.43:0/3835975614 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca4022b90 con 0x7fecb4105bf0 2024-05-27T07:23:58.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.565+0000 7fecbb200700 1 -- 172.21.15.43:0/3835975614 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb41a91d0 con 0x7fecb4105bf0 2024-05-27T07:23:58.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.565+0000 7fecbb200700 1 -- 172.21.15.43:0/3835975614 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb41a9620 con 0x7fecb4105bf0 2024-05-27T07:23:58.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.569+0000 7fecaea00700 1 -- 172.21.15.43:0/3835975614 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7feca402a070 con 0x7fecb4105bf0 2024-05-27T07:23:58.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.571+0000 7fecaea00700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec9c0b29f0 0x7fec9c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.571+0000 7feccb2f1b80 1 Processor -- start 2024-05-27T07:23:58.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.571+0000 7fecaea00700 1 -- 172.21.15.43:0/3835975614 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7feca40e3a50 con 0x7fecb4105bf0 2024-05-27T07:23:58.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.571+0000 7feccb2f1b80 1 -- start start 2024-05-27T07:23:58.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.571+0000 7fecb9e00700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec9c0b29f0 0x7fec9c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.572+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb2f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.572+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0c9afe40 0x55ff0cb2fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.572+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0cb392c0 0x55ff0cb3b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.572+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ff0ca5ce40 con 0x55ff0ca9b9f0 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.572+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ff0ca4c9e0 con 0x55ff0c9afe40 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.572+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ff0ca50a20 con 0x55ff0cb392c0 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.572+0000 7fecb9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb2f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.572+0000 7fecb9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb2f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56358/0 (socket says 172.21.15.43:56358) 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.572+0000 7fecb9e00700 1 -- 172.21.15.43:0/1198770420 learned_addr learned my addr 172.21.15.43:0/1198770420 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.573+0000 7fecb9400700 1 --2- 172.21.15.43:0/1198770420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0c9afe40 0x55ff0cb2fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.573+0000 7fecb9e00700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec9c0b29f0 0x7fec9c0b4f00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feca8005950 tx=0x7feca80106b0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:58.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.573+0000 7fecba800700 1 --2- 172.21.15.43:0/1198770420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0cb392c0 0x55ff0cb3b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.573+0000 7fecb9e00700 1 -- 172.21.15.43:0/1198770420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0cb392c0 msgr2=0x55ff0cb3b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:58.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.573+0000 7fecb9e00700 1 --2- 172.21.15.43:0/1198770420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0cb392c0 0x55ff0cb3b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.573+0000 7fecb9e00700 1 -- 172.21.15.43:0/1198770420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0c9afe40 msgr2=0x55ff0cb2fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:58.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.573+0000 7fecb9e00700 1 --2- 172.21.15.43:0/1198770420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0c9afe40 0x55ff0cb2fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.573+0000 7fecb9e00700 1 -- 172.21.15.43:0/1198770420 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff0ca15420 con 0x55ff0ca9b9f0 2024-05-27T07:23:58.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.574+0000 7fecb9e00700 1 --2- 172.21.15.43:0/1198770420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb2f750 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7feca800bf10 tx=0x7feca801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6e048902222bbd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:58.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.574+0000 7feca3400700 1 -- 172.21.15.43:0/1198770420 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca80130d0 con 0x55ff0ca9b9f0 2024-05-27T07:23:58.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.574+0000 7feca3400700 1 -- 172.21.15.43:0/1198770420 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feca8013270 con 0x55ff0ca9b9f0 2024-05-27T07:23:58.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.574+0000 7feca3400700 1 -- 172.21.15.43:0/1198770420 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8005610 con 0x55ff0ca9b9f0 2024-05-27T07:23:58.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.575+0000 7feccb2f1b80 1 -- 172.21.15.43:0/1198770420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0ca9b9f0 msgr2=0x55ff0cb2f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.575+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/1198770420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb2f750 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7feca800bf10 tx=0x7feca801b000 comp rx=0 tx=0).stop 2024-05-27T07:23:58.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.575+0000 7feccb2f1b80 1 -- 172.21.15.43:0/1198770420 shutdown_connections 2024-05-27T07:23:58.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.575+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/1198770420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0cb392c0 0x55ff0cb3b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.575+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/1198770420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb2f750 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.575+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/1198770420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0c9afe40 0x55ff0cb2fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.575+0000 7feccb2f1b80 1 -- 172.21.15.43:0/1198770420 >> 172.21.15.43:0/1198770420 conn(0x55ff0ca49390 msgr2=0x55ff0cb2a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:58.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.575+0000 7feccb2f1b80 1 -- 172.21.15.43:0/1198770420 shutdown_connections 2024-05-27T07:23:58.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.576+0000 7feccb2f1b80 1 -- 172.21.15.43:0/1198770420 wait complete. 2024-05-27T07:23:58.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.576+0000 7feccb2f1b80 1 -- start start 2024-05-27T07:23:58.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.576+0000 7feccb2f1b80 1 Processor -- start 2024-05-27T07:23:58.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.577+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0c9afe40 0x55ff0cb36230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.577+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb36770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.577+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb36cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.577+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ff0c952fa0 con 0x55ff0cb392c0 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.577+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ff0c9ed760 con 0x55ff0ca9b9f0 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.577+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ff0cb3be00 con 0x55ff0c9afe40 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.577+0000 7feccb2f1b80 10 client.0 _ll_register_callbacks cb 0x55ff0c9f6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.578+0000 7fecba800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb36cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.578+0000 7fecba800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb36cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56372/0 (socket says 172.21.15.43:56372) 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.578+0000 7fecba800700 1 -- 172.21.15.43:0/4001837491 learned_addr learned my addr 172.21.15.43:0/4001837491 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.578+0000 7fecb9e00700 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0c9afe40 0x55ff0cb36230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.578+0000 7fecb9400700 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb36770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.578+0000 7fecba800700 1 -- 172.21.15.43:0/4001837491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0c9afe40 msgr2=0x55ff0cb36230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.578+0000 7fecba800700 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0c9afe40 0x55ff0cb36230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.578+0000 7fecba800700 1 -- 172.21.15.43:0/4001837491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0ca9b9f0 msgr2=0x55ff0cb36770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.578+0000 7fecba800700 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb36770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.578+0000 7fecba800700 1 -- 172.21.15.43:0/4001837491 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff0c99c8d0 con 0x55ff0cb392c0 2024-05-27T07:23:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.579+0000 7fecba800700 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb36cb0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fecb0003fd0 tx=0x7fecb0004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b69fe9369ead7f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.579+0000 7fec97e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.579+0000 7fec97e00700 1 -- 172.21.15.43:0/4001837491 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb0019030 con 0x55ff0cb392c0 2024-05-27T07:23:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.579+0000 7feccb2f1b80 1 -- 172.21.15.43:0/4001837491 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feca8013270 con 0x55ff0cb392c0 2024-05-27T07:23:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.579+0000 7fec97e00700 1 -- 172.21.15.43:0/4001837491 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7fecb0012040 con 0x55ff0cb392c0 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.579+0000 7fec97e00700 1 -- 172.21.15.43:0/4001837491 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fecb0023050 con 0x55ff0cb392c0 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.579+0000 7feccb2f1b80 10 client.16971 fetch_fsmap learned FSMap version 107 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.579+0000 7feccb2f1b80 1 -- 172.21.15.43:0/4001837491 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55ff0ca1d060 con 0x55ff0cb392c0 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.580+0000 7fec97e00700 1 -- 172.21.15.43:0/4001837491 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feca8013270 con 0x55ff0cb392c0 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.580+0000 7fec97e00700 1 -- 172.21.15.43:0/4001837491 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7fecb001e6e0 con 0x55ff0cb392c0 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.580+0000 7feccb2f1b80 10 client.16971 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.580+0000 7feccb2f1b80 10 client.16971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.580+0000 7feccb2f1b80 20 client.16971 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.580+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ff0cb49170 0x55ff0cb4b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.580+0000 7feccb2f1b80 4 client.16971 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:23:58.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.580+0000 7feccb2f1b80 1 -- 172.21.15.43:0/4001837491 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff0ca5ed90 con 0x55ff0cb49170 2024-05-27T07:23:58.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.581+0000 7fecb9e00700 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ff0cb49170 0x55ff0cb4b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.582+0000 7fecb9e00700 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ff0cb49170 0x55ff0cb4b620 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb8b1621c28b4ded server_cookie=3e42574ff1dbf049 in_seq=0 out_seq=0 2024-05-27T07:23:58.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.582+0000 7fec97e00700 10 client.16971 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:58.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7fec97e00700 1 -- 172.21.15.43:0/4001837491 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ff0ca5ed90 con 0x55ff0cb49170 2024-05-27T07:23:58.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7fec97e00700 10 client.16971 handle_command_reply: tid=0 2024-05-27T07:23:58.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7feccb2f1b80 1 client.16971 shutdown 2024-05-27T07:23:58.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7feccb2f1b80 1 -- 172.21.15.43:0/4001837491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 msgr2=0x55ff0cb36cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb36cb0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fecb0003fd0 tx=0x7fecb0004320 comp rx=0 tx=0).stop 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7feccb2f1b80 1 -- 172.21.15.43:0/4001837491 shutdown_connections 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ff0cb49170 0x55ff0cb4b620 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb36cb0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7fec97e00700 0 client.16971 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0c9afe40 0x55ff0cb36230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/4001837491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb36770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.586+0000 7feccb2f1b80 1 -- 172.21.15.43:0/4001837491 >> 172.21.15.43:0/4001837491 conn(0x55ff0ca49390 msgr2=0x55ff0cb2a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.587+0000 7feccb2f1b80 1 -- 172.21.15.43:0/4001837491 shutdown_connections 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.587+0000 7feccb2f1b80 1 -- 172.21.15.43:0/4001837491 wait complete. 2024-05-27T07:23:58.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.587+0000 7feccb2f1b80 20 client.16971 trim_cache size 0 max 16384 2024-05-27T07:23:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.611+0000 7feccb2f1b80 1 Processor -- start 2024-05-27T07:23:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.611+0000 7feccb2f1b80 1 -- start start 2024-05-27T07:23:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.611+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0c9afe40 0x55ff0cb2eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.611+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb44ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.611+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0cb392c0 0x55ff0cb50de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.611+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b31f0 con 0x55ff0c9afe40 2024-05-27T07:23:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.611+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b0e90 con 0x55ff0cb392c0 2024-05-27T07:23:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.611+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b2b20 con 0x55ff0ca9b9f0 2024-05-27T07:23:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.612+0000 7fecb9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0c9afe40 0x55ff0cb2eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.612+0000 7fecb9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0c9afe40 0x55ff0cb2eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56376/0 (socket says 172.21.15.43:56376) 2024-05-27T07:23:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.612+0000 7fecb9e00700 1 -- 172.21.15.43:0/2904045755 learned_addr learned my addr 172.21.15.43:0/2904045755 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.612+0000 7fecb9400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb44ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.612+0000 7fecba800700 1 --2- 172.21.15.43:0/2904045755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0cb392c0 0x55ff0cb50de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.613+0000 7fecb9e00700 1 -- 172.21.15.43:0/2904045755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0ca9b9f0 msgr2=0x55ff0cb44ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.613+0000 7fecb9e00700 1 --2- 172.21.15.43:0/2904045755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb44ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.613+0000 7fecb9e00700 1 -- 172.21.15.43:0/2904045755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0cb392c0 msgr2=0x55ff0cb50de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.613+0000 7fecb9e00700 1 --2- 172.21.15.43:0/2904045755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0cb392c0 0x55ff0cb50de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.613+0000 7fecb9e00700 1 -- 172.21.15.43:0/2904045755 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff0cb31e30 con 0x55ff0c9afe40 2024-05-27T07:23:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.613+0000 7fecba800700 1 --2- 172.21.15.43:0/2904045755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0cb392c0 0x55ff0cb50de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:23:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.613+0000 7fecb9e00700 1 --2- 172.21.15.43:0/2904045755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0c9afe40 0x55ff0cb2eb50 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7feca8024000 tx=0x7feca8013310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2038e6946e9eac9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.614+0000 7feca2a00700 1 -- 172.21.15.43:0/2904045755 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8005240 con 0x55ff0c9afe40 2024-05-27T07:23:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.614+0000 7feca2a00700 1 -- 172.21.15.43:0/2904045755 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feca8038070 con 0x55ff0c9afe40 2024-05-27T07:23:58.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.614+0000 7feca2a00700 1 -- 172.21.15.43:0/2904045755 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8004bb0 con 0x55ff0c9afe40 2024-05-27T07:23:58.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.615+0000 7feccb2f1b80 1 -- 172.21.15.43:0/2904045755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0c9afe40 msgr2=0x55ff0cb2eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.615+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/2904045755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0c9afe40 0x55ff0cb2eb50 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7feca8024000 tx=0x7feca8013310 comp rx=0 tx=0).stop 2024-05-27T07:23:58.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.615+0000 7feccb2f1b80 1 -- 172.21.15.43:0/2904045755 shutdown_connections 2024-05-27T07:23:58.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.615+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/2904045755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0cb392c0 0x55ff0cb50de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.615+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/2904045755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0c9afe40 0x55ff0cb2eb50 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.615+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/2904045755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb44ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.615+0000 7feccb2f1b80 1 -- 172.21.15.43:0/2904045755 >> 172.21.15.43:0/2904045755 conn(0x55ff0ca49390 msgr2=0x55ff0cb31720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:58.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.615+0000 7feccb2f1b80 1 -- 172.21.15.43:0/2904045755 shutdown_connections 2024-05-27T07:23:58.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.615+0000 7feccb2f1b80 1 -- 172.21.15.43:0/2904045755 wait complete. 2024-05-27T07:23:58.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.616+0000 7feccb2f1b80 1 -- start start 2024-05-27T07:23:58.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.616+0000 7feccb2f1b80 1 Processor -- start 2024-05-27T07:23:58.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0c9afe40 0x55ff0cb414f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb3c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7feccb2f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb3cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b1c90 con 0x55ff0cb392c0 2024-05-27T07:23:58.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b1560 con 0x55ff0c9afe40 2024-05-27T07:23:58.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7feccb2f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b23f0 con 0x55ff0ca9b9f0 2024-05-27T07:23:58.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7feccb2f1b80 10 client.0 _ll_register_callbacks cb 0x55ff0c9f6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:23:58.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7fecba800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb3cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7fecba800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb3cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56382/0 (socket says 172.21.15.43:56382) 2024-05-27T07:23:58.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7fecba800700 1 -- 172.21.15.43:0/710926511 learned_addr learned my addr 172.21.15.43:0/710926511 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:23:58.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7fecb9e00700 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0c9afe40 0x55ff0cb414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.617+0000 7fecb9400700 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb3c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.618+0000 7fecba800700 1 -- 172.21.15.43:0/710926511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0ca9b9f0 msgr2=0x55ff0cb3c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.618+0000 7fecba800700 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb3c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.618+0000 7fecba800700 1 -- 172.21.15.43:0/710926511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0c9afe40 msgr2=0x55ff0cb414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.618+0000 7fecba800700 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0c9afe40 0x55ff0cb414f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.618+0000 7fecba800700 1 -- 172.21.15.43:0/710926511 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff0cb3d2d0 con 0x55ff0cb392c0 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.618+0000 7fecba800700 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb3cd90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fecb0000f80 tx=0x7fecb0005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e113eb1e411609ae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.618+0000 7fec97e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.619+0000 7fec97e00700 1 -- 172.21.15.43:0/710926511 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb002c060 con 0x55ff0cb392c0 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.619+0000 7fec97e00700 1 -- 172.21.15.43:0/710926511 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9496+0+0 (secure 0 0 0) 0x7fecb001c610 con 0x55ff0cb392c0 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.619+0000 7feccb2f1b80 1 -- 172.21.15.43:0/710926511 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feca8038070 con 0x55ff0cb392c0 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.619+0000 7fec97e00700 1 -- 172.21.15.43:0/710926511 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fecb0019040 con 0x55ff0cb392c0 2024-05-27T07:23:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.619+0000 7feccb2f1b80 10 client.16977 fetch_fsmap learned FSMap version 107 2024-05-27T07:23:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.619+0000 7feccb2f1b80 1 -- 172.21.15.43:0/710926511 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55ff0cbbc140 con 0x55ff0cb392c0 2024-05-27T07:23:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.619+0000 7fec97e00700 1 -- 172.21.15.43:0/710926511 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feca8038070 con 0x55ff0cb392c0 2024-05-27T07:23:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.620+0000 7fec97e00700 1 -- 172.21.15.43:0/710926511 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7fecb001c920 con 0x55ff0cb392c0 2024-05-27T07:23:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.620+0000 7feccb2f1b80 10 client.16977 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:23:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.620+0000 7feccb2f1b80 10 client.16977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:23:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.620+0000 7feccb2f1b80 20 client.16977 populate_metadata read hostname 'smithi043' 2024-05-27T07:23:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.620+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ff0cbbc850 0x55ff0cbbec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:23:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.620+0000 7feccb2f1b80 4 client.16977 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:23:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.620+0000 7feccb2f1b80 1 -- 172.21.15.43:0/710926511 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ff0ca58570 con 0x55ff0cbbc850 2024-05-27T07:23:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.621+0000 7fecb9e00700 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ff0cbbc850 0x55ff0cbbec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:23:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.621+0000 7fecb9e00700 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ff0cbbc850 0x55ff0cbbec70 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af7de917a8af287e server_cookie=25c7ab2b984f6720 in_seq=0 out_seq=0 2024-05-27T07:23:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.621+0000 7fec97e00700 10 client.16977 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.622+0000 7fec97e00700 1 -- 172.21.15.43:0/710926511 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ff0ca58570 con 0x55ff0cbbc850 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.622+0000 7fec97e00700 10 client.16977 handle_command_reply: tid=0 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.622+0000 7feccb2f1b80 1 client.16977 shutdown 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.622+0000 7feccb2f1b80 1 -- 172.21.15.43:0/710926511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 msgr2=0x55ff0cb3cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.622+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb3cd90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fecb0000f80 tx=0x7fecb0005f10 comp rx=0 tx=0).stop 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.622+0000 7feccb2f1b80 1 -- 172.21.15.43:0/710926511 shutdown_connections 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.622+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ff0cbbc850 0x55ff0cbbec70 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.623+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ff0cb392c0 0x55ff0cb3cd90 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.623+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ff0c9afe40 0x55ff0cb414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.623+0000 7feccb2f1b80 1 --2- 172.21.15.43:0/710926511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ff0ca9b9f0 0x55ff0cb3c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.623+0000 7fec97e00700 0 client.16977 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.623+0000 7feccb2f1b80 1 -- 172.21.15.43:0/710926511 >> 172.21.15.43:0/710926511 conn(0x55ff0ca49390 msgr2=0x55ff0cb31720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.623+0000 7feccb2f1b80 1 -- 172.21.15.43:0/710926511 shutdown_connections 2024-05-27T07:23:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.623+0000 7feccb2f1b80 1 -- 172.21.15.43:0/710926511 wait complete. 2024-05-27T07:23:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.623+0000 7feccb2f1b80 20 client.16977 trim_cache size 0 max 16384 2024-05-27T07:23:58.614 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:23:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:23:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:23:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "bc6bd47d-cc04-4d60-8abf-00c9d990f2ff": { 2024-05-27T07:23:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:23:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "bc6bd47d-cc04-4d60-8abf-00c9d990f2ff", 2024-05-27T07:23:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:23:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:23:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:23:58.615 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 -- 172.21.15.43:0/3835975614 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec9c0b29f0 msgr2=0x7fec9c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec9c0b29f0 0x7fec9c0b4f00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feca8005950 tx=0x7feca80106b0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 -- 172.21.15.43:0/3835975614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecb4105bf0 msgr2=0x7fecb41034d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecb4105bf0 0x7fecb41034d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7feca4005910 tx=0x7feca40047b0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 -- 172.21.15.43:0/3835975614 shutdown_connections 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec9c0b29f0 0x7fec9c0b4f00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fecb410e710 0x7fecb4103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fecb41052a0 0x7fecb4102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 --2- 172.21.15.43:0/3835975614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecb4105bf0 0x7fecb41034d0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 -- 172.21.15.43:0/3835975614 >> 172.21.15.43:0/3835975614 conn(0x7fecb4078c30 msgr2=0x7fecb40ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 -- 172.21.15.43:0/3835975614 shutdown_connections 2024-05-27T07:23:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:23:58.624+0000 7fecbb200700 1 -- 172.21.15.43:0/3835975614 wait complete. 2024-05-27T07:23:59.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 55996 0.61 MB/sec execute 363 sec latency 55.262 ms 2024-05-27T07:24:00.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:23:59 smithi132 ceph-mon[35283]: pgmap v341: 129 pgs: 129 active+clean; 96 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 278 KiB/s wr, 34 op/s 2024-05-27T07:24:00.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:23:59 smithi043 ceph-mon[31531]: pgmap v341: 129 pgs: 129 active+clean; 96 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 278 KiB/s wr, 34 op/s 2024-05-27T07:24:00.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:23:59 smithi179 ceph-mon[35888]: pgmap v341: 129 pgs: 129 active+clean; 96 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 278 KiB/s wr, 34 op/s 2024-05-27T07:24:00.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56173 0.61 MB/sec execute 364 sec latency 62.002 ms 2024-05-27T07:24:00.290 INFO:tasks.cephfs.filesystem:scrub status for tag:bc6bd47d-cc04-4d60-8abf-00c9d990f2ff - {'path': '/', 'tag': 'bc6bd47d-cc04-4d60-8abf-00c9d990f2ff', 'options': 'recursive,force'} 2024-05-27T07:24:01.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56394 0.61 MB/sec execute 365 sec latency 56.937 ms 2024-05-27T07:24:02.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:01 smithi132 ceph-mon[35283]: pgmap v342: 129 pgs: 129 active+clean; 99 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 630 KiB/s wr, 50 op/s 2024-05-27T07:24:02.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:01 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:24:02.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:01 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:24:02.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:01 smithi043 ceph-mon[31531]: pgmap v342: 129 pgs: 129 active+clean; 99 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 630 KiB/s wr, 50 op/s 2024-05-27T07:24:02.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:01 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:24:02.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:01 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:24:02.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:01 smithi179 ceph-mon[35888]: pgmap v342: 129 pgs: 129 active+clean; 99 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 630 KiB/s wr, 50 op/s 2024-05-27T07:24:02.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:01 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:24:02.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:01 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:24:02.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56520 0.61 MB/sec execute 366 sec latency 70.166 ms 2024-05-27T07:24:03.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:02 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:24:03.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:02 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:24:03.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:02 smithi043 ceph-mon[31531]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:24:03.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:02 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:24:03.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:02 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:24:03.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:02 smithi179 ceph-mon[35888]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:24:03.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56547 0.61 MB/sec execute 367 sec latency 64.161 ms 2024-05-27T07:24:03.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:02 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:24:03.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:02 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:24:03.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:02 smithi132 ceph-mon[35283]: osdmap e128: 12 total, 12 up, 12 in 2024-05-27T07:24:04.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:03 smithi043 ceph-mon[31531]: pgmap v344: 129 pgs: 129 active+clean; 99 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 527 KiB/s wr, 37 op/s 2024-05-27T07:24:04.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:03 smithi179 ceph-mon[35888]: pgmap v344: 129 pgs: 129 active+clean; 99 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 527 KiB/s wr, 37 op/s 2024-05-27T07:24:04.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56571 0.61 MB/sec execute 368 sec latency 58.631 ms 2024-05-27T07:24:04.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:03 smithi132 ceph-mon[35283]: pgmap v344: 129 pgs: 129 active+clean; 99 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 527 KiB/s wr, 37 op/s 2024-05-27T07:24:05.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56736 0.62 MB/sec execute 369 sec latency 53.403 ms 2024-05-27T07:24:06.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:05 smithi043 ceph-mon[31531]: pgmap v345: 129 pgs: 129 active+clean; 99 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 527 KiB/s wr, 37 op/s 2024-05-27T07:24:06.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:05 smithi179 ceph-mon[35888]: pgmap v345: 129 pgs: 129 active+clean; 99 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 527 KiB/s wr, 37 op/s 2024-05-27T07:24:06.225 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56819 0.62 MB/sec execute 370 sec latency 63.988 ms 2024-05-27T07:24:06.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:05 smithi132 ceph-mon[35283]: pgmap v345: 129 pgs: 129 active+clean; 99 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 527 KiB/s wr, 37 op/s 2024-05-27T07:24:07.226 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56853 0.61 MB/sec execute 371 sec latency 931.702 ms 2024-05-27T07:24:08.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:07 smithi043 ceph-mon[31531]: pgmap v346: 129 pgs: 129 active+clean; 100 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 692 KiB/s wr, 56 op/s 2024-05-27T07:24:08.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:07 smithi179 ceph-mon[35888]: pgmap v346: 129 pgs: 129 active+clean; 100 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 692 KiB/s wr, 56 op/s 2024-05-27T07:24:08.226 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56853 0.61 MB/sec execute 372 sec latency 1931.836 ms 2024-05-27T07:24:08.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:07 smithi132 ceph-mon[35283]: pgmap v346: 129 pgs: 129 active+clean; 100 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 692 KiB/s wr, 56 op/s 2024-05-27T07:24:09.226 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56902 0.61 MB/sec execute 373 sec latency 2162.531 ms 2024-05-27T07:24:10.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:09 smithi043 ceph-mon[31531]: pgmap v347: 129 pgs: 129 active+clean; 100 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 692 KiB/s wr, 56 op/s 2024-05-27T07:24:10.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:09 smithi179 ceph-mon[35888]: pgmap v347: 129 pgs: 129 active+clean; 100 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 692 KiB/s wr, 56 op/s 2024-05-27T07:24:10.226 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56997 0.61 MB/sec execute 374 sec latency 68.627 ms 2024-05-27T07:24:10.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:09 smithi132 ceph-mon[35283]: pgmap v347: 129 pgs: 129 active+clean; 100 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 692 KiB/s wr, 56 op/s 2024-05-27T07:24:11.226 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57045 0.61 MB/sec execute 375 sec latency 52.222 ms 2024-05-27T07:24:12.226 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57098 0.61 MB/sec execute 376 sec latency 51.359 ms 2024-05-27T07:24:12.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:12 smithi132 ceph-mon[35283]: pgmap v348: 129 pgs: 129 active+clean; 101 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 461 KiB/s wr, 51 op/s 2024-05-27T07:24:12.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:12 smithi132 ceph-mon[35283]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:24:12.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:12 smithi043 ceph-mon[31531]: pgmap v348: 129 pgs: 129 active+clean; 101 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 461 KiB/s wr, 51 op/s 2024-05-27T07:24:12.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:12 smithi043 ceph-mon[31531]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:24:12.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:12 smithi179 ceph-mon[35888]: pgmap v348: 129 pgs: 129 active+clean; 101 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 461 KiB/s wr, 51 op/s 2024-05-27T07:24:12.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:12 smithi179 ceph-mon[35888]: osdmap e129: 12 total, 12 up, 12 in 2024-05-27T07:24:13.226 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57161 0.61 MB/sec execute 377 sec latency 51.851 ms 2024-05-27T07:24:14.226 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57213 0.61 MB/sec execute 378 sec latency 53.511 ms 2024-05-27T07:24:14.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:14 smithi132 ceph-mon[35283]: pgmap v350: 129 pgs: 129 active+clean; 101 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 461 KiB/s wr, 51 op/s 2024-05-27T07:24:14.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:14 smithi043 ceph-mon[31531]: pgmap v350: 129 pgs: 129 active+clean; 101 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 461 KiB/s wr, 51 op/s 2024-05-27T07:24:14.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:14 smithi179 ceph-mon[35888]: pgmap v350: 129 pgs: 129 active+clean; 101 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 461 KiB/s wr, 51 op/s 2024-05-27T07:24:15.226 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57263 0.61 MB/sec execute 379 sec latency 51.439 ms 2024-05-27T07:24:16.227 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57338 0.60 MB/sec execute 380 sec latency 51.230 ms 2024-05-27T07:24:16.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:16 smithi132 ceph-mon[35283]: pgmap v351: 129 pgs: 129 active+clean; 101 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 461 KiB/s wr, 51 op/s 2024-05-27T07:24:16.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:16 smithi043 ceph-mon[31531]: pgmap v351: 129 pgs: 129 active+clean; 101 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 461 KiB/s wr, 51 op/s 2024-05-27T07:24:16.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:16 smithi179 ceph-mon[35888]: pgmap v351: 129 pgs: 129 active+clean; 101 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 461 KiB/s wr, 51 op/s 2024-05-27T07:24:17.227 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57397 0.60 MB/sec execute 381 sec latency 66.993 ms 2024-05-27T07:24:18.227 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57510 0.60 MB/sec execute 382 sec latency 54.822 ms 2024-05-27T07:24:18.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:18 smithi132 ceph-mon[35283]: pgmap v352: 129 pgs: 129 active+clean; 103 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 371 KiB/s wr, 34 op/s 2024-05-27T07:24:18.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:18 smithi043 ceph-mon[31531]: pgmap v352: 129 pgs: 129 active+clean; 103 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 371 KiB/s wr, 34 op/s 2024-05-27T07:24:18.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:18 smithi179 ceph-mon[35888]: pgmap v352: 129 pgs: 129 active+clean; 103 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 371 KiB/s wr, 34 op/s 2024-05-27T07:24:19.230 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57642 0.60 MB/sec execute 383 sec latency 69.133 ms 2024-05-27T07:24:20.227 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57725 0.60 MB/sec execute 384 sec latency 57.279 ms 2024-05-27T07:24:20.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:20 smithi132 ceph-mon[35283]: pgmap v353: 129 pgs: 129 active+clean; 103 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 371 KiB/s wr, 34 op/s 2024-05-27T07:24:20.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:20 smithi043 ceph-mon[31531]: pgmap v353: 129 pgs: 129 active+clean; 103 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 371 KiB/s wr, 34 op/s 2024-05-27T07:24:20.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:20 smithi179 ceph-mon[35888]: pgmap v353: 129 pgs: 129 active+clean; 103 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 371 KiB/s wr, 34 op/s 2024-05-27T07:24:21.227 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57842 0.60 MB/sec execute 385 sec latency 56.785 ms 2024-05-27T07:24:22.227 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57955 0.60 MB/sec execute 386 sec latency 69.714 ms 2024-05-27T07:24:22.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:22 smithi132 ceph-mon[35283]: pgmap v354: 129 pgs: 129 active+clean; 103 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 371 KiB/s wr, 46 op/s 2024-05-27T07:24:22.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:22 smithi043 ceph-mon[31531]: pgmap v354: 129 pgs: 129 active+clean; 103 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 371 KiB/s wr, 46 op/s 2024-05-27T07:24:22.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:22 smithi179 ceph-mon[35888]: pgmap v354: 129 pgs: 129 active+clean; 103 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 371 KiB/s wr, 46 op/s 2024-05-27T07:24:23.227 INFO:tasks.workunit.client.0.smithi043.stdout: 1 58044 0.60 MB/sec execute 387 sec latency 55.265 ms 2024-05-27T07:24:24.228 INFO:tasks.workunit.client.0.smithi043.stdout: 1 58204 0.60 MB/sec execute 388 sec latency 61.455 ms 2024-05-27T07:24:24.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:24 smithi132 ceph-mon[35283]: pgmap v355: 129 pgs: 129 active+clean; 103 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 322 KiB/s wr, 40 op/s 2024-05-27T07:24:24.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:24 smithi043 ceph-mon[31531]: pgmap v355: 129 pgs: 129 active+clean; 103 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 322 KiB/s wr, 40 op/s 2024-05-27T07:24:24.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:24 smithi179 ceph-mon[35888]: pgmap v355: 129 pgs: 129 active+clean; 103 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 322 KiB/s wr, 40 op/s 2024-05-27T07:24:25.228 INFO:tasks.workunit.client.0.smithi043.stdout: 1 58293 0.60 MB/sec execute 389 sec latency 64.446 ms 2024-05-27T07:24:26.228 INFO:tasks.workunit.client.0.smithi043.stdout: 1 58401 0.60 MB/sec execute 390 sec latency 61.860 ms 2024-05-27T07:24:26.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:26 smithi132 ceph-mon[35283]: pgmap v356: 129 pgs: 129 active+clean; 103 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 309 KiB/s wr, 38 op/s 2024-05-27T07:24:26.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:26 smithi043 ceph-mon[31531]: pgmap v356: 129 pgs: 129 active+clean; 103 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 309 KiB/s wr, 38 op/s 2024-05-27T07:24:26.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:26 smithi179 ceph-mon[35888]: pgmap v356: 129 pgs: 129 active+clean; 103 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 309 KiB/s wr, 38 op/s 2024-05-27T07:24:27.228 INFO:tasks.workunit.client.0.smithi043.stdout: 1 58528 0.60 MB/sec execute 391 sec latency 57.326 ms 2024-05-27T07:24:28.228 INFO:tasks.workunit.client.0.smithi043.stdout: 1 58645 0.60 MB/sec execute 392 sec latency 62.057 ms 2024-05-27T07:24:28.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:28 smithi132 ceph-mon[35283]: pgmap v357: 129 pgs: 129 active+clean; 104 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 489 KiB/s wr, 55 op/s 2024-05-27T07:24:28.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:28 smithi043 ceph-mon[31531]: pgmap v357: 129 pgs: 129 active+clean; 104 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 489 KiB/s wr, 55 op/s 2024-05-27T07:24:28.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:28 smithi179 ceph-mon[35888]: pgmap v357: 129 pgs: 129 active+clean; 104 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 489 KiB/s wr, 55 op/s 2024-05-27T07:24:29.228 INFO:tasks.workunit.client.0.smithi043.stdout: 1 58770 0.60 MB/sec execute 393 sec latency 65.406 ms 2024-05-27T07:24:30.228 INFO:tasks.workunit.client.0.smithi043.stdout: 1 58879 0.60 MB/sec execute 394 sec latency 57.561 ms 2024-05-27T07:24:30.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:24:30.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:30 smithi132 ceph-mon[35283]: pgmap v358: 129 pgs: 129 active+clean; 104 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 340 KiB/s wr, 38 op/s 2024-05-27T07:24:30.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:30 smithi043 ceph-mon[31531]: pgmap v358: 129 pgs: 129 active+clean; 104 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 340 KiB/s wr, 38 op/s 2024-05-27T07:24:30.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:30 smithi179 ceph-mon[35888]: pgmap v358: 129 pgs: 129 active+clean; 104 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 340 KiB/s wr, 38 op/s 2024-05-27T07:24:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:31.228 INFO:tasks.workunit.client.0.smithi043.stdout: 1 59077 0.60 MB/sec execute 395 sec latency 63.388 ms 2024-05-27T07:24:32.229 INFO:tasks.workunit.client.0.smithi043.stdout: 1 59157 0.60 MB/sec execute 396 sec latency 57.020 ms 2024-05-27T07:24:32.553 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:32 smithi043 ceph-mon[31531]: pgmap v359: 129 pgs: 129 active+clean; 105 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 423 KiB/s wr, 53 op/s 2024-05-27T07:24:32.553 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:32 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:24:32.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:32 smithi132 ceph-mon[35283]: pgmap v359: 129 pgs: 129 active+clean; 105 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 423 KiB/s wr, 53 op/s 2024-05-27T07:24:32.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:32 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:24:32.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:32 smithi179 ceph-mon[35888]: pgmap v359: 129 pgs: 129 active+clean; 105 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 423 KiB/s wr, 53 op/s 2024-05-27T07:24:32.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:32 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:24:33.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.161+0000 7f7040800700 1 -- 172.21.15.43:0/3635152794 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7038106eb0 msgr2=0x7f7038113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.161+0000 7f7040800700 1 --2- 172.21.15.43:0/3635152794 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7038106eb0 0x7f7038113340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f703400b0a0 tx=0x7f703400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.161+0000 7f7040800700 1 -- 172.21.15.43:0/3635152794 shutdown_connections 2024-05-27T07:24:33.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.161+0000 7f7040800700 1 --2- 172.21.15.43:0/3635152794 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7038106eb0 0x7f7038113340 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.161+0000 7f7040800700 1 --2- 172.21.15.43:0/3635152794 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f703810c5a0 0x7f703810c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.161+0000 7f7040800700 1 --2- 172.21.15.43:0/3635152794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7038106560 0x7f7038106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.161+0000 7f7040800700 1 -- 172.21.15.43:0/3635152794 >> 172.21.15.43:0/3635152794 conn(0x7f7038102040 msgr2=0x7f7038104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:33.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.161+0000 7f7040800700 1 -- 172.21.15.43:0/3635152794 shutdown_connections 2024-05-27T07:24:33.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.162+0000 7f7040800700 1 -- 172.21.15.43:0/3635152794 wait complete. 2024-05-27T07:24:33.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.163+0000 7f7040800700 1 Processor -- start 2024-05-27T07:24:33.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.164+0000 7f7040800700 1 -- start start 2024-05-27T07:24:33.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.164+0000 7f7040800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7038106560 0x7f703819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.164+0000 7f7040800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7038106eb0 0x7f703819d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.164+0000 7f7040800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f703810c5a0 0x7f70381a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.164+0000 7f7040800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f70381165d0 con 0x7f703810c5a0 2024-05-27T07:24:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.164+0000 7f7040800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7038116450 con 0x7f7038106560 2024-05-27T07:24:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.164+0000 7f7040800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7038116750 con 0x7f7038106eb0 2024-05-27T07:24:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.165+0000 7f703de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7038106560 0x7f703819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.165+0000 7f703d400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7038106eb0 0x7f703819d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.165+0000 7f703de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7038106560 0x7f703819cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:35152/0 (socket says 172.21.15.43:35152) 2024-05-27T07:24:33.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.165+0000 7f703de00700 1 -- 172.21.15.43:0/3875059979 learned_addr learned my addr 172.21.15.43:0/3875059979 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:33.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.165+0000 7f703e800700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f703810c5a0 0x7f70381a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.165+0000 7f703d400700 1 -- 172.21.15.43:0/3875059979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7038106560 msgr2=0x7f703819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.165+0000 7f703d400700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7038106560 0x7f703819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.165+0000 7f703d400700 1 -- 172.21.15.43:0/3875059979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f703810c5a0 msgr2=0x7f70381a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.165+0000 7f703d400700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f703810c5a0 0x7f70381a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.165+0000 7f703d400700 1 -- 172.21.15.43:0/3875059979 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70381a1e30 con 0x7f7038106eb0 2024-05-27T07:24:33.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.166+0000 7f703d400700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7038106eb0 0x7f703819d500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7038075b70 tx=0x7f702800eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.166+0000 7f7032a00700 1 -- 172.21.15.43:0/3875059979 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7028010900 con 0x7f7038106eb0 2024-05-27T07:24:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.166+0000 7f7032a00700 1 -- 172.21.15.43:0/3875059979 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f702800c430 con 0x7f7038106eb0 2024-05-27T07:24:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.166+0000 7f7032a00700 1 -- 172.21.15.43:0/3875059979 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7028018680 con 0x7f7038106eb0 2024-05-27T07:24:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.166+0000 7f7040800700 1 -- 172.21.15.43:0/3875059979 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70381a20c0 con 0x7f7038106eb0 2024-05-27T07:24:33.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.166+0000 7f7040800700 1 -- 172.21.15.43:0/3875059979 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70381a9a40 con 0x7f7038106eb0 2024-05-27T07:24:33.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.173+0000 7f7032a00700 1 -- 172.21.15.43:0/3875059979 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f7028018820 con 0x7f7038106eb0 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.174+0000 7f704f3efb80 1 Processor -- start 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.174+0000 7f704f3efb80 1 -- start start 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.174+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c549f4fe40 0x55c54a0dd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.174+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 0x55c54a0d7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.174+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 0x55c54a0d79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.174+0000 7f704f3efb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c549ffce40 con 0x55c549f4fe40 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.174+0000 7f704f3efb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c549fec9e0 con 0x55c54a0dd540 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.175+0000 7f704f3efb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c549ff0a20 con 0x55c54a0d75c0 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.175+0000 7f703de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 0x55c54a0d7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.175+0000 7f703de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 0x55c54a0d7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:35168/0 (socket says 172.21.15.43:35168) 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.175+0000 7f703de00700 1 -- 172.21.15.43:0/4017350960 learned_addr learned my addr 172.21.15.43:0/4017350960 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.175+0000 7f703e800700 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c549f4fe40 0x55c54a0dd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f703de00700 1 -- 172.21.15.43:0/4017350960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 msgr2=0x55c54a0d79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f703d400700 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 0x55c54a0d79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f7032a00700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f70200b29f0 0x7f70200b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f703de00700 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 0x55c54a0d79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f703de00700 1 -- 172.21.15.43:0/4017350960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c549f4fe40 msgr2=0x55c54a0dd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f703de00700 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c549f4fe40 0x55c54a0dd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f703de00700 1 -- 172.21.15.43:0/4017350960 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c549fb5420 con 0x55c54a0dd540 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f7032a00700 1 -- 172.21.15.43:0/3875059979 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f70280e2ab0 con 0x7f7038106eb0 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f703d400700 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 0x55c54a0d79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f703e800700 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c549f4fe40 0x55c54a0dd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.176+0000 7f703de00700 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 0x55c54a0d7080 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f702c009df0 tx=0x7f702c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=225cc19be5e32ffe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.177+0000 7f7027400700 1 -- 172.21.15.43:0/4017350960 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c004220 con 0x55c54a0dd540 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.177+0000 7f703e800700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f70200b29f0 0x7f70200b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.177+0000 7f7027400700 1 -- 172.21.15.43:0/4017350960 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f702c0043c0 con 0x55c54a0dd540 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.177+0000 7f7027400700 1 -- 172.21.15.43:0/4017350960 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c019890 con 0x55c54a0dd540 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.178+0000 7f704f3efb80 1 -- 172.21.15.43:0/4017350960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 msgr2=0x55c54a0d7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.178+0000 7f704f3efb80 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 0x55c54a0d7080 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f702c009df0 tx=0x7f702c011720 comp rx=0 tx=0).stop 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.178+0000 7f703e800700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f70200b29f0 0x7f70200b4f00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55c54a0d8340 tx=0x7f70340099e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.178+0000 7f704f3efb80 1 -- 172.21.15.43:0/4017350960 shutdown_connections 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.178+0000 7f704f3efb80 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 0x55c54a0d79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.178+0000 7f704f3efb80 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c549f4fe40 0x55c54a0dd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.178+0000 7f704f3efb80 1 --2- 172.21.15.43:0/4017350960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 0x55c54a0d7080 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.178+0000 7f704f3efb80 1 -- 172.21.15.43:0/4017350960 >> 172.21.15.43:0/4017350960 conn(0x55c549faffc0 msgr2=0x55c54a0c98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:33.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.178+0000 7f704f3efb80 1 -- 172.21.15.43:0/4017350960 shutdown_connections 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.178+0000 7f704f3efb80 1 -- 172.21.15.43:0/4017350960 wait complete. 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.179+0000 7f704f3efb80 1 -- start start 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.179+0000 7f704f3efb80 1 Processor -- start 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.180+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 0x55c54a0e0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.180+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 0x55c54a0e1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.180+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0dd540 0x55c54a0e17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.180+0000 7f704f3efb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c549ef2fa0 con 0x55c54a0dd540 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.180+0000 7f704f3efb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c549f8d760 con 0x55c549f4fe40 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.180+0000 7f704f3efb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c54a0f5020 con 0x55c54a0d75c0 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.180+0000 7f704f3efb80 10 client.0 _ll_register_callbacks cb 0x55c549f96c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.180+0000 7f703e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0dd540 0x55c54a0e17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.181+0000 7f703e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0dd540 0x55c54a0e17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34174/0 (socket says 172.21.15.43:34174) 2024-05-27T07:24:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.181+0000 7f703d400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 0x55c54a0e1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.181+0000 7f703e800700 1 -- 172.21.15.43:0/3127774438 learned_addr learned my addr 172.21.15.43:0/3127774438 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.181+0000 7f703de00700 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 0x55c54a0e0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.181+0000 7f703d400700 1 -- 172.21.15.43:0/3127774438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 msgr2=0x55c54a0e0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.181+0000 7f703d400700 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 0x55c54a0e0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.181+0000 7f703d400700 1 -- 172.21.15.43:0/3127774438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0dd540 msgr2=0x55c54a0e17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.181+0000 7f703d400700 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0dd540 0x55c54a0e17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.181+0000 7f703d400700 1 -- 172.21.15.43:0/3127774438 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c549f3c8d0 con 0x55c54a0d75c0 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.182+0000 7f703d400700 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 0x55c54a0e1290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f70280dfe30 tx=0x7f70280a91b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56cbdedb69fa1d94 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.182+0000 7f701be00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.182+0000 7f701be00700 1 -- 172.21.15.43:0/3127774438 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70280a94d0 con 0x55c54a0d75c0 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.182+0000 7f701be00700 1 -- 172.21.15.43:0/3127774438 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f70280cc2c0 con 0x55c54a0d75c0 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.182+0000 7f704f3efb80 1 -- 172.21.15.43:0/3127774438 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f702c0043c0 con 0x55c54a0d75c0 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.182+0000 7f701be00700 1 -- 172.21.15.43:0/3127774438 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f70280aa0c0 con 0x55c54a0d75c0 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.183+0000 7f704f3efb80 10 client.26590 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.183+0000 7f704f3efb80 1 -- 172.21.15.43:0/3127774438 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55c549fbd060 con 0x55c54a0d75c0 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.183+0000 7f701be00700 1 -- 172.21.15.43:0/3127774438 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f702c0043c0 con 0x55c54a0d75c0 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.183+0000 7f701be00700 1 -- 172.21.15.43:0/3127774438 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7f70280b2420 con 0x55c54a0d75c0 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.183+0000 7f704f3efb80 10 client.26590 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.183+0000 7f704f3efb80 10 client.26590 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:33.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.183+0000 7f704f3efb80 20 client.26590 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:33.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.183+0000 7f704f3efb80 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c54a102220 0x55c54a1046b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.183+0000 7f704f3efb80 4 client.26590 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:33.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.183+0000 7f704f3efb80 1 -- 172.21.15.43:0/3127774438 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c549ffed90 con 0x55c54a102220 2024-05-27T07:24:33.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.184+0000 7f703de00700 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c54a102220 0x55c54a1046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.185+0000 7f703de00700 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c54a102220 0x55c54a1046b0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b628e87816cb62ba server_cookie=d69d758e9caf5352 in_seq=0 out_seq=0 2024-05-27T07:24:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.185+0000 7f701be00700 10 client.26590 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:24:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.186+0000 7f701be00700 1 -- 172.21.15.43:0/3127774438 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c549ffed90 con 0x55c54a102220 2024-05-27T07:24:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.186+0000 7f701be00700 10 client.26590 handle_command_reply: tid=0 2024-05-27T07:24:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.186+0000 7f704f3efb80 1 client.26590 shutdown 2024-05-27T07:24:33.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.186+0000 7f704f3efb80 1 -- 172.21.15.43:0/3127774438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 msgr2=0x55c54a0e1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.186+0000 7f704f3efb80 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 0x55c54a0e1290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f70280dfe30 tx=0x7f70280a91b0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.186+0000 7f704f3efb80 1 -- 172.21.15.43:0/3127774438 shutdown_connections 2024-05-27T07:24:33.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.186+0000 7f704f3efb80 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c54a102220 0x55c54a1046b0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.187+0000 7f704f3efb80 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0dd540 0x55c54a0e17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.187+0000 7f704f3efb80 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 0x55c54a0e0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.187+0000 7f704f3efb80 1 --2- 172.21.15.43:0/3127774438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0d75c0 0x55c54a0e1290 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.187+0000 7f704f3efb80 1 -- 172.21.15.43:0/3127774438 >> 172.21.15.43:0/3127774438 conn(0x55c549faffc0 msgr2=0x55c54a0c9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:33.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.187+0000 7f701be00700 0 client.26590 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:24:33.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.188+0000 7f704f3efb80 1 -- 172.21.15.43:0/3127774438 shutdown_connections 2024-05-27T07:24:33.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.188+0000 7f704f3efb80 1 -- 172.21.15.43:0/3127774438 wait complete. 2024-05-27T07:24:33.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.188+0000 7f704f3efb80 20 client.26590 trim_cache size 0 max 16384 2024-05-27T07:24:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.214+0000 7f704f3efb80 1 Processor -- start 2024-05-27T07:24:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.214+0000 7f704f3efb80 1 -- start start 2024-05-27T07:24:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 0x55c54a0d7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0e31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0dd540 0x55c54a0fe0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f704f3efb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7038072680 con 0x55c54a0d75c0 2024-05-27T07:24:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f704f3efb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7038070320 con 0x55c549f4fe40 2024-05-27T07:24:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f704f3efb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7038071fb0 con 0x55c54a0dd540 2024-05-27T07:24:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f703d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0e31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f703de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 0x55c54a0d7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f703de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 0x55c54a0d7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:35188/0 (socket says 172.21.15.43:35188) 2024-05-27T07:24:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f703de00700 1 -- 172.21.15.43:0/720525408 learned_addr learned my addr 172.21.15.43:0/720525408 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f703d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0e31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34176/0 (socket says 172.21.15.43:34176) 2024-05-27T07:24:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.215+0000 7f703e800700 1 --2- 172.21.15.43:0/720525408 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0dd540 0x55c54a0fe0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.216+0000 7f703d400700 1 -- 172.21.15.43:0/720525408 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0dd540 msgr2=0x55c54a0fe0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.216+0000 7f703d400700 1 --2- 172.21.15.43:0/720525408 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0dd540 0x55c54a0fe0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.216+0000 7f703d400700 1 -- 172.21.15.43:0/720525408 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 msgr2=0x55c54a0d7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.216+0000 7f703d400700 1 --2- 172.21.15.43:0/720525408 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 0x55c54a0d7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.216+0000 7f703d400700 1 -- 172.21.15.43:0/720525408 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c54a0e1b40 con 0x55c54a0d75c0 2024-05-27T07:24:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.216+0000 7f703d400700 1 --2- 172.21.15.43:0/720525408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0e31f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7028004990 tx=0x7f70280e2e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892cb2810645f48a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.217+0000 7f7026a00700 1 -- 172.21.15.43:0/720525408 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70280a5880 con 0x55c54a0d75c0 2024-05-27T07:24:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.217+0000 7f7026a00700 1 -- 172.21.15.43:0/720525408 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70280a94d0 con 0x55c54a0d75c0 2024-05-27T07:24:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.217+0000 7f7026a00700 1 -- 172.21.15.43:0/720525408 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70280a99c0 con 0x55c54a0d75c0 2024-05-27T07:24:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.218+0000 7f704f3efb80 1 -- 172.21.15.43:0/720525408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 msgr2=0x55c54a0e31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.218+0000 7f704f3efb80 1 --2- 172.21.15.43:0/720525408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0e31f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7028004990 tx=0x7f70280e2e40 comp rx=0 tx=0).stop 2024-05-27T07:24:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.218+0000 7f704f3efb80 1 -- 172.21.15.43:0/720525408 shutdown_connections 2024-05-27T07:24:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.218+0000 7f704f3efb80 1 --2- 172.21.15.43:0/720525408 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c54a0dd540 0x55c54a0fe0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.218+0000 7f704f3efb80 1 --2- 172.21.15.43:0/720525408 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c549f4fe40 0x55c54a0d7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.218+0000 7f704f3efb80 1 --2- 172.21.15.43:0/720525408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0e31f0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.218+0000 7f704f3efb80 1 -- 172.21.15.43:0/720525408 >> 172.21.15.43:0/720525408 conn(0x55c549faffc0 msgr2=0x55c54a0da950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.219+0000 7f704f3efb80 1 -- 172.21.15.43:0/720525408 shutdown_connections 2024-05-27T07:24:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.219+0000 7f704f3efb80 1 -- 172.21.15.43:0/720525408 wait complete. 2024-05-27T07:24:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.219+0000 7f704f3efb80 1 -- start start 2024-05-27T07:24:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.219+0000 7f704f3efb80 1 Processor -- start 2024-05-27T07:24:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.220+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c549f4fe40 0x55c54a0fa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.220+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0fa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.220+0000 7f704f3efb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 0x55c54a108e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.220+0000 7f704f3efb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7038071120 con 0x55c54a0d75c0 2024-05-27T07:24:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.220+0000 7f704f3efb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f70380709f0 con 0x55c54a0dd540 2024-05-27T07:24:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.220+0000 7f704f3efb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7038071880 con 0x55c549f4fe40 2024-05-27T07:24:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.220+0000 7f704f3efb80 10 client.0 _ll_register_callbacks cb 0x55c549f96c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.220+0000 7f703d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0fa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f703d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0fa7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34184/0 (socket says 172.21.15.43:34184) 2024-05-27T07:24:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f703d400700 1 -- 172.21.15.43:0/608857058 learned_addr learned my addr 172.21.15.43:0/608857058 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f703de00700 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c549f4fe40 0x55c54a0fa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f703e800700 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 0x55c54a108e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f703d400700 1 -- 172.21.15.43:0/608857058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c549f4fe40 msgr2=0x55c54a0fa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f703d400700 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c549f4fe40 0x55c54a0fa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f703d400700 1 -- 172.21.15.43:0/608857058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 msgr2=0x55c54a108e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f703d400700 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 0x55c54a108e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f703d400700 1 -- 172.21.15.43:0/608857058 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c54a1093a0 con 0x55c54a0d75c0 2024-05-27T07:24:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f703d400700 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0fa7f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f702800cc60 tx=0x7f70280a9fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6146c1fb53ba77e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.221+0000 7f701be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.222+0000 7f701be00700 1 -- 172.21.15.43:0/608857058 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70280add70 con 0x55c54a0d75c0 2024-05-27T07:24:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.222+0000 7f701be00700 1 -- 172.21.15.43:0/608857058 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f70280cc2c0 con 0x55c54a0d75c0 2024-05-27T07:24:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.222+0000 7f704f3efb80 1 -- 172.21.15.43:0/608857058 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70280a94d0 con 0x55c54a0d75c0 2024-05-27T07:24:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.222+0000 7f701be00700 1 -- 172.21.15.43:0/608857058 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f7028087050 con 0x55c54a0d75c0 2024-05-27T07:24:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.222+0000 7f704f3efb80 10 client.16995 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.222+0000 7f704f3efb80 1 -- 172.21.15.43:0/608857058 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55c54a16c370 con 0x55c54a0d75c0 2024-05-27T07:24:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.222+0000 7f701be00700 1 -- 172.21.15.43:0/608857058 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f70280a94d0 con 0x55c54a0d75c0 2024-05-27T07:24:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.223+0000 7f701be00700 1 -- 172.21.15.43:0/608857058 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7f702806f120 con 0x55c54a0d75c0 2024-05-27T07:24:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.223+0000 7f704f3efb80 10 client.16995 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.223+0000 7f704f3efb80 10 client.16995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.223+0000 7f704f3efb80 20 client.16995 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.223+0000 7f704f3efb80 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c54a16c7c0 0x55c54a16ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.223+0000 7f704f3efb80 4 client.16995 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:24:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.223+0000 7f704f3efb80 1 -- 172.21.15.43:0/608857058 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c549ff8570 con 0x55c54a16c7c0 2024-05-27T07:24:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.223+0000 7f703de00700 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c54a16c7c0 0x55c54a16ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.224+0000 7f703de00700 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c54a16c7c0 0x55c54a16ebe0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50b9ecfdfa05d4d3 server_cookie=7512532137d7ec84 in_seq=0 out_seq=0 2024-05-27T07:24:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.224+0000 7f701be00700 10 client.16995 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:24:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.224+0000 7f701be00700 1 -- 172.21.15.43:0/608857058 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c549ff8570 con 0x55c54a16c7c0 2024-05-27T07:24:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.224+0000 7f701be00700 10 client.16995 handle_command_reply: tid=0 2024-05-27T07:24:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.224+0000 7f704f3efb80 1 client.16995 shutdown 2024-05-27T07:24:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.224+0000 7f704f3efb80 1 -- 172.21.15.43:0/608857058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 msgr2=0x55c54a0fa7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.224+0000 7f704f3efb80 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0fa7f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f702800cc60 tx=0x7f70280a9fb0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.225+0000 7f704f3efb80 1 -- 172.21.15.43:0/608857058 shutdown_connections 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.225+0000 7f704f3efb80 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c54a0dd540 0x55c54a108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.225+0000 7f704f3efb80 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c549f4fe40 0x55c54a0fa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.225+0000 7f704f3efb80 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c54a16c7c0 0x55c54a16ebe0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.225+0000 7f704f3efb80 1 --2- 172.21.15.43:0/608857058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c54a0d75c0 0x55c54a0fa7f0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.225+0000 7f701be00700 0 client.16995 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.225+0000 7f704f3efb80 1 -- 172.21.15.43:0/608857058 >> 172.21.15.43:0/608857058 conn(0x55c549faffc0 msgr2=0x55c54a0dc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.225+0000 7f704f3efb80 1 -- 172.21.15.43:0/608857058 shutdown_connections 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.225+0000 7f704f3efb80 1 -- 172.21.15.43:0/608857058 wait complete. 2024-05-27T07:24:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.225+0000 7f704f3efb80 20 client.16995 trim_cache size 0 max 16384 2024-05-27T07:24:33.217 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:24:33.217 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:24:33.217 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:24:33.217 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:24:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.226+0000 7f7040800700 1 -- 172.21.15.43:0/3875059979 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f70200b29f0 msgr2=0x7f70200b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.226+0000 7f7040800700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f70200b29f0 0x7f70200b4f00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55c54a0d8340 tx=0x7f70340099e0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.226+0000 7f7040800700 1 -- 172.21.15.43:0/3875059979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7038106eb0 msgr2=0x7f703819d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.226+0000 7f7040800700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7038106eb0 0x7f703819d500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7038075b70 tx=0x7f702800eca0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.227+0000 7f7040800700 1 -- 172.21.15.43:0/3875059979 shutdown_connections 2024-05-27T07:24:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.227+0000 7f7040800700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f70200b29f0 0x7f70200b4f00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.227+0000 7f7040800700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f703810c5a0 0x7f70381a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.227+0000 7f7040800700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7038106560 0x7f703819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.227+0000 7f7040800700 1 --2- 172.21.15.43:0/3875059979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7038106eb0 0x7f703819d500 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.227+0000 7f7040800700 1 -- 172.21.15.43:0/3875059979 >> 172.21.15.43:0/3875059979 conn(0x7f7038102040 msgr2=0x7f7038103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.227+0000 7f7040800700 1 -- 172.21.15.43:0/3875059979 shutdown_connections 2024-05-27T07:24:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:33.227+0000 7f7040800700 1 -- 172.21.15.43:0/3875059979 wait complete. 2024-05-27T07:24:33.228 INFO:tasks.workunit.client.0.smithi043.stdout: 1 59328 0.60 MB/sec execute 397 sec latency 47.021 ms 2024-05-27T07:24:34.136 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:24:34.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:24:34.229 INFO:tasks.workunit.client.0.smithi043.stdout: 1 59523 0.60 MB/sec execute 398 sec latency 58.588 ms 2024-05-27T07:24:34.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:34 smithi043 ceph-mon[31531]: pgmap v360: 129 pgs: 129 active+clean; 105 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 264 KiB/s wr, 31 op/s 2024-05-27T07:24:34.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:34 smithi132 ceph-mon[35283]: pgmap v360: 129 pgs: 129 active+clean; 105 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 264 KiB/s wr, 31 op/s 2024-05-27T07:24:34.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:34 smithi179 ceph-mon[35888]: pgmap v360: 129 pgs: 129 active+clean; 105 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 264 KiB/s wr, 31 op/s 2024-05-27T07:24:35.100 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:35.229 INFO:tasks.workunit.client.0.smithi043.stdout: 1 59770 0.61 MB/sec execute 399 sec latency 47.059 ms 2024-05-27T07:24:36.229 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60059 0.61 MB/sec execute 400 sec latency 48.060 ms 2024-05-27T07:24:36.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:36 smithi043 ceph-mon[31531]: pgmap v361: 129 pgs: 129 active+clean; 105 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 264 KiB/s wr, 31 op/s 2024-05-27T07:24:36.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:36 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:36.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:36 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:36.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:36 smithi132 ceph-mon[35283]: pgmap v361: 129 pgs: 129 active+clean; 105 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 264 KiB/s wr, 31 op/s 2024-05-27T07:24:36.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:36 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:36.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:36 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:36.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:36 smithi179 ceph-mon[35888]: pgmap v361: 129 pgs: 129 active+clean; 105 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 264 KiB/s wr, 31 op/s 2024-05-27T07:24:36.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:36 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:36.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:36 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:37.229 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60094 0.61 MB/sec execute 401 sec latency 53.558 ms 2024-05-27T07:24:37.416 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:37 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:37.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.436+0000 7efbfe600700 1 -- 172.21.15.43:0/1955230575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbf8105470 msgr2=0x7efbf8105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:37.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.436+0000 7efbfe600700 1 --2- 172.21.15.43:0/1955230575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbf8105470 0x7efbf8105880 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009990 tx=0x7efbe8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.436+0000 7efbfe600700 1 -- 172.21.15.43:0/1955230575 shutdown_connections 2024-05-27T07:24:37.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.436+0000 7efbfe600700 1 --2- 172.21.15.43:0/1955230575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf810e8e0 0x7efbf8110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.436+0000 7efbfe600700 1 --2- 172.21.15.43:0/1955230575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbf8105470 0x7efbf8105880 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.436+0000 7efbfe600700 1 --2- 172.21.15.43:0/1955230575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efbf8105dc0 0x7efbf810e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.436+0000 7efbfe600700 1 -- 172.21.15.43:0/1955230575 >> 172.21.15.43:0/1955230575 conn(0x7efbf80fef20 msgr2=0x7efbf8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:37.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.436+0000 7efbfe600700 1 -- 172.21.15.43:0/1955230575 shutdown_connections 2024-05-27T07:24:37.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.437+0000 7efbfe600700 1 -- 172.21.15.43:0/1955230575 wait complete. 2024-05-27T07:24:37.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.438+0000 7efbfe600700 1 Processor -- start 2024-05-27T07:24:37.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.438+0000 7efbfe600700 1 -- start start 2024-05-27T07:24:37.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbfe600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf8105470 0x7efbf819cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:37.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbfe600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbf8105dc0 0x7efbf819d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:37.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbfe600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efbf810e8e0 0x7efbf81a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:37.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbfe600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efbf8104b40 con 0x7efbf8105dc0 2024-05-27T07:24:37.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbfe600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efbf81049c0 con 0x7efbf810e8e0 2024-05-27T07:24:37.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbfe600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efbf8104cc0 con 0x7efbf8105470 2024-05-27T07:24:37.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbf6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbf8105dc0 0x7efbf819d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:37.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbf6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbf8105dc0 0x7efbf819d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38794/0 (socket says 172.21.15.43:38794) 2024-05-27T07:24:37.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbf6a00700 1 -- 172.21.15.43:0/21433116 learned_addr learned my addr 172.21.15.43:0/21433116 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:37.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbf7400700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf8105470 0x7efbf819cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:37.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.439+0000 7efbf7e00700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efbf810e8e0 0x7efbf81a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:37.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.440+0000 7efbf6a00700 1 -- 172.21.15.43:0/21433116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf8105470 msgr2=0x7efbf819cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:37.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.440+0000 7efbf6a00700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf8105470 0x7efbf819cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.440+0000 7efbf6a00700 1 -- 172.21.15.43:0/21433116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efbf810e8e0 msgr2=0x7efbf81a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:37.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.440+0000 7efbf6a00700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efbf810e8e0 0x7efbf81a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.440+0000 7efbf6a00700 1 -- 172.21.15.43:0/21433116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf81a1e80 con 0x7efbf8105dc0 2024-05-27T07:24:37.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.441+0000 7efbf6a00700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbf8105dc0 0x7efbf819d550 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7efbe400d980 tx=0x7efbe400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:37.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.441+0000 7efbefe00700 1 -- 172.21.15.43:0/21433116 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe4017070 con 0x7efbf8105dc0 2024-05-27T07:24:37.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.442+0000 7efbefe00700 1 -- 172.21.15.43:0/21433116 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efbe4004830 con 0x7efbf8105dc0 2024-05-27T07:24:37.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.442+0000 7efbefe00700 1 -- 172.21.15.43:0/21433116 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe40053a0 con 0x7efbf8105dc0 2024-05-27T07:24:37.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.442+0000 7efbfe600700 1 -- 172.21.15.43:0/21433116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbf81a2170 con 0x7efbf8105dc0 2024-05-27T07:24:37.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.442+0000 7efbfe600700 1 -- 172.21.15.43:0/21433116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbf81a9a90 con 0x7efbf8105dc0 2024-05-27T07:24:37.433 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:37 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:37.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.446+0000 7efbfe600700 1 -- 172.21.15.43:0/21433116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbf80512b0 con 0x7efbf8105dc0 2024-05-27T07:24:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.450+0000 7efbefe00700 1 -- 172.21.15.43:0/21433116 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7efbe4005540 con 0x7efbf8105dc0 2024-05-27T07:24:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.452+0000 7efbefe00700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efbdc0b2b40 0x7efbdc0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.452+0000 7efbefe00700 1 -- 172.21.15.43:0/21433116 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7efbe40e28e0 con 0x7efbf8105dc0 2024-05-27T07:24:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.453+0000 7efbf7400700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efbdc0b2b40 0x7efbdc0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:37.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.454+0000 7efbf7400700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efbdc0b2b40 0x7efbdc0b5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efbe800b2e0 tx=0x7efbe8005fb0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:37.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.457+0000 7efbefe00700 1 -- 172.21.15.43:0/21433116 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efbe4020020 con 0x7efbf8105dc0 2024-05-27T07:24:37.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:37 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:37 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:37 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:37 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:37 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:37.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:37 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:37.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:37 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:37 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:37 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:37 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.815+0000 7efbfe600700 1 -- 172.21.15.43:0/21433116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efbf819e310 con 0x7efbf8105dc0 2024-05-27T07:24:37.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.815+0000 7efbefe00700 1 -- 172.21.15.43:0/21433116 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6964 (secure 0 0 0) 0x7efbe40a9e90 con 0x7efbf8105dc0 2024-05-27T07:24:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:24:37.806 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:23:55.398503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34403,24484,34424,34484,34448]},"id":1}]} 2024-05-27T07:24:37.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.819+0000 7efbfe600700 1 -- 172.21.15.43:0/21433116 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efbdc0b2b40 msgr2=0x7efbdc0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:37.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.819+0000 7efbfe600700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efbdc0b2b40 0x7efbdc0b5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efbe800b2e0 tx=0x7efbe8005fb0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.819+0000 7efbfe600700 1 -- 172.21.15.43:0/21433116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbf8105dc0 msgr2=0x7efbf819d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:37.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.819+0000 7efbfe600700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbf8105dc0 0x7efbf819d550 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7efbe400d980 tx=0x7efbe400dd40 comp rx=0 tx=0).stop 2024-05-27T07:24:37.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.820+0000 7efbfe600700 1 -- 172.21.15.43:0/21433116 shutdown_connections 2024-05-27T07:24:37.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.820+0000 7efbfe600700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efbdc0b2b40 0x7efbdc0b5050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.820+0000 7efbfe600700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efbf810e8e0 0x7efbf81a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.820+0000 7efbfe600700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf8105470 0x7efbf819cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.820+0000 7efbfe600700 1 --2- 172.21.15.43:0/21433116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efbf8105dc0 0x7efbf819d550 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:37.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.820+0000 7efbfe600700 1 -- 172.21.15.43:0/21433116 >> 172.21.15.43:0/21433116 conn(0x7efbf80fef20 msgr2=0x7efbf80ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:37.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.820+0000 7efbfe600700 1 -- 172.21.15.43:0/21433116 shutdown_connections 2024-05-27T07:24:37.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:37.820+0000 7efbfe600700 1 -- 172.21.15.43:0/21433116 wait complete. 2024-05-27T07:24:37.811 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 107 2024-05-27T07:24:37.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:37 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:37.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:37 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:37.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:37 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:37 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:37 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:37.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:37 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.229 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60155 0.61 MB/sec execute 402 sec latency 49.326 ms 2024-05-27T07:24:38.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:38 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 176.4M 2024-05-27T07:24:38.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:38 smithi043 ceph-mon[31531]: Unable to set osd_memory_target on smithi132 to 185069158: error parsing value: Value '185069158' is below minimum 939524096 2024-05-27T07:24:38.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:38 smithi043 ceph-mon[31531]: pgmap v362: 129 pgs: 129 active+clean; 108 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 654 KiB/s wr, 50 op/s 2024-05-27T07:24:38.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:38 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:38 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:38 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:38 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:38 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/21433116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:38.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:24:38.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:38 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi132 to 176.4M 2024-05-27T07:24:38.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:38 smithi179 ceph-mon[35888]: Unable to set osd_memory_target on smithi132 to 185069158: error parsing value: Value '185069158' is below minimum 939524096 2024-05-27T07:24:38.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:38 smithi179 ceph-mon[35888]: pgmap v362: 129 pgs: 129 active+clean; 108 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 654 KiB/s wr, 50 op/s 2024-05-27T07:24:38.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:38 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:38 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:38 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:38 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:38 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/21433116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:38.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:38 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi132 to 176.4M 2024-05-27T07:24:38.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:38 smithi132 ceph-mon[35283]: Unable to set osd_memory_target on smithi132 to 185069158: error parsing value: Value '185069158' is below minimum 939524096 2024-05-27T07:24:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:38 smithi132 ceph-mon[35283]: pgmap v362: 129 pgs: 129 active+clean; 108 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 654 KiB/s wr, 50 op/s 2024-05-27T07:24:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:38 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:38 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:38 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:38 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:38 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/21433116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:24:39.145 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:39.229 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60345 0.61 MB/sec execute 403 sec latency 50.630 ms 2024-05-27T07:24:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:39 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi179 to 176.4M 2024-05-27T07:24:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:39 smithi043 ceph-mon[31531]: Unable to set osd_memory_target on smithi179 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-05-27T07:24:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:39 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:39 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:39.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:39 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi179 to 176.4M 2024-05-27T07:24:39.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:39 smithi179 ceph-mon[35888]: Unable to set osd_memory_target on smithi179 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-05-27T07:24:39.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:39 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:39.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:39 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:39.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:39 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi179 to 176.4M 2024-05-27T07:24:39.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:39 smithi132 ceph-mon[35283]: Unable to set osd_memory_target on smithi179 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-05-27T07:24:39.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:39 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:39.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:39 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:40.229 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60395 0.61 MB/sec execute 404 sec latency 718.201 ms 2024-05-27T07:24:40.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:40 smithi043 ceph-mon[31531]: pgmap v363: 129 pgs: 129 active+clean; 108 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 473 KiB/s wr, 34 op/s 2024-05-27T07:24:40.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:40 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:40 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:40 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:40 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:40 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:24:40.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:40 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:24:40.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:40 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:40 smithi179 ceph-mon[35888]: pgmap v363: 129 pgs: 129 active+clean; 108 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 473 KiB/s wr, 34 op/s 2024-05-27T07:24:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:40 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:40 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:40 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:40 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:40 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:24:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:40 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:24:40.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:40 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:40.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:40 smithi132 ceph-mon[35283]: pgmap v363: 129 pgs: 129 active+clean; 108 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 473 KiB/s wr, 34 op/s 2024-05-27T07:24:40.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:40 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:40 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:40 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:40 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:24:40.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:40 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:24:40.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:40 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:24:40.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:40 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:24:41.230 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60395 0.61 MB/sec execute 405 sec latency 1718.312 ms 2024-05-27T07:24:41.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.348+0000 7f95efe00700 1 -- 172.21.15.43:0/1863981090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f0075b50 msgr2=0x7f95f0075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.348+0000 7f95efe00700 1 --2- 172.21.15.43:0/1863981090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f0075b50 0x7f95f0075f60 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009990 tx=0x7f95e0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.348+0000 7f95efe00700 1 -- 172.21.15.43:0/1863981090 shutdown_connections 2024-05-27T07:24:41.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.348+0000 7f95efe00700 1 --2- 172.21.15.43:0/1863981090 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95f0115170 0x7f95f0117590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.348+0000 7f95efe00700 1 --2- 172.21.15.43:0/1863981090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f0075b50 0x7f95f0075f60 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.348+0000 7f95efe00700 1 --2- 172.21.15.43:0/1863981090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95f00764a0 0x7f95f0114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.348+0000 7f95efe00700 1 -- 172.21.15.43:0/1863981090 >> 172.21.15.43:0/1863981090 conn(0x7f95f0079d00 msgr2=0x7f95f007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.348+0000 7f95efe00700 1 -- 172.21.15.43:0/1863981090 shutdown_connections 2024-05-27T07:24:41.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.348+0000 7f95efe00700 1 -- 172.21.15.43:0/1863981090 wait complete. 2024-05-27T07:24:41.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.350+0000 7f95efe00700 1 Processor -- start 2024-05-27T07:24:41.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.350+0000 7f95efe00700 1 -- start start 2024-05-27T07:24:41.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.350+0000 7f95efe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f0075b50 0x7f95f01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.350+0000 7f95efe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95f00764a0 0x7f95f01a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.350+0000 7f95efe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95f0115170 0x7f95f01a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.350+0000 7f95efe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f95f011a750 con 0x7f95f0075b50 2024-05-27T07:24:41.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.350+0000 7f95efe00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f95f011a5d0 con 0x7f95f0115170 2024-05-27T07:24:41.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.350+0000 7f95efe00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f95f011a8d0 con 0x7f95f00764a0 2024-05-27T07:24:41.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.351+0000 7f95eea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f0075b50 0x7f95f01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.351+0000 7f95eea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f0075b50 0x7f95f01a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38800/0 (socket says 172.21.15.43:38800) 2024-05-27T07:24:41.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.351+0000 7f95eea00700 1 -- 172.21.15.43:0/1066063967 learned_addr learned my addr 172.21.15.43:0/1066063967 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:41.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.351+0000 7f95e6000700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95f00764a0 0x7f95f01a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.351+0000 7f95ef400700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95f0115170 0x7f95f01a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.351+0000 7f95eea00700 1 -- 172.21.15.43:0/1066063967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95f00764a0 msgr2=0x7f95f01a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.351+0000 7f95eea00700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95f00764a0 0x7f95f01a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.351+0000 7f95eea00700 1 -- 172.21.15.43:0/1066063967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95f0115170 msgr2=0x7f95f01a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.351+0000 7f95eea00700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95f0115170 0x7f95f01a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.351+0000 7f95eea00700 1 -- 172.21.15.43:0/1066063967 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f01a6070 con 0x7f95f0075b50 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.352+0000 7f95eea00700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f0075b50 0x7f95f01a11a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f95e0000c00 tx=0x7f95e0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.352+0000 7f95e7e00700 1 -- 172.21.15.43:0/1066063967 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0027070 con 0x7f95f0075b50 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.352+0000 7f95efe00700 1 -- 172.21.15.43:0/1066063967 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f01a6300 con 0x7f95f0075b50 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.352+0000 7f95e7e00700 1 -- 172.21.15.43:0/1066063967 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95e000fc70 con 0x7f95f0075b50 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.352+0000 7f95e7e00700 1 -- 172.21.15.43:0/1066063967 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0022d40 con 0x7f95f0075b50 2024-05-27T07:24:41.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.352+0000 7f95efe00700 1 -- 172.21.15.43:0/1066063967 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f01a67e0 con 0x7f95f0075b50 2024-05-27T07:24:41.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.356+0000 7f95e7e00700 1 -- 172.21.15.43:0/1066063967 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f95e002a070 con 0x7f95f0075b50 2024-05-27T07:24:41.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.357+0000 7f95e7e00700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f95d40b29f0 0x7f95d40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.358+0000 7f95e7e00700 1 -- 172.21.15.43:0/1066063967 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f95e00e3810 con 0x7f95f0075b50 2024-05-27T07:24:41.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.358+0000 7f95e6000700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f95d40b29f0 0x7f95d40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.358+0000 7f96043abb80 1 Processor -- start 2024-05-27T07:24:41.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.358+0000 7f96043abb80 1 -- start start 2024-05-27T07:24:41.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.359+0000 7f95e6000700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f95d40b29f0 0x7f95d40b4f00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f95d8005950 tx=0x7f95d80058e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.359+0000 7f96043abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 0x55eee765fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.359+0000 7f96043abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee766aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.359+0000 7f96043abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 0x55eee766d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.359+0000 7f96043abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55eee758ee40 con 0x55eee7660410 2024-05-27T07:24:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.359+0000 7f96043abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55eee757e9e0 con 0x55eee74e1e40 2024-05-27T07:24:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.359+0000 7f96043abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55eee7582a20 con 0x55eee7660820 2024-05-27T07:24:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.359+0000 7f95e6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee766aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.359+0000 7f95e6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee766aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38806/0 (socket says 172.21.15.43:38806) 2024-05-27T07:24:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.359+0000 7f95e6000700 1 -- 172.21.15.43:0/2082664218 learned_addr learned my addr 172.21.15.43:0/2082664218 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.360+0000 7f95eea00700 1 --2- 172.21.15.43:0/2082664218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 0x55eee765fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.360+0000 7f95ef400700 1 --2- 172.21.15.43:0/2082664218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 0x55eee766d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.360+0000 7f95e6000700 1 -- 172.21.15.43:0/2082664218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 msgr2=0x55eee766d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.360+0000 7f95e6000700 1 --2- 172.21.15.43:0/2082664218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 0x55eee766d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.360+0000 7f95e6000700 1 -- 172.21.15.43:0/2082664218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 msgr2=0x55eee765fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.360+0000 7f95e6000700 1 --2- 172.21.15.43:0/2082664218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 0x55eee765fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.360+0000 7f95e6000700 1 -- 172.21.15.43:0/2082664218 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eee7547420 con 0x55eee7660410 2024-05-27T07:24:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.360+0000 7f95ef400700 1 --2- 172.21.15.43:0/2082664218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 0x55eee766d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:24:41.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.360+0000 7f95e6000700 1 --2- 172.21.15.43:0/2082664218 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee766aad0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f95d800fd40 tx=0x7f95d8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587b149e8f2272ac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.361+0000 7f95dfe00700 1 -- 172.21.15.43:0/2082664218 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8022070 con 0x55eee7660410 2024-05-27T07:24:41.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.361+0000 7f95dfe00700 1 -- 172.21.15.43:0/2082664218 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95d8011bb0 con 0x55eee7660410 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.361+0000 7f95dfe00700 1 -- 172.21.15.43:0/2082664218 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d801d5b0 con 0x55eee7660410 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.362+0000 7f96043abb80 1 -- 172.21.15.43:0/2082664218 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 msgr2=0x55eee766aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.362+0000 7f96043abb80 1 --2- 172.21.15.43:0/2082664218 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee766aad0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f95d800fd40 tx=0x7f95d8012000 comp rx=0 tx=0).stop 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.362+0000 7f96043abb80 1 -- 172.21.15.43:0/2082664218 shutdown_connections 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.362+0000 7f96043abb80 1 --2- 172.21.15.43:0/2082664218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 0x55eee766d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.362+0000 7f96043abb80 1 --2- 172.21.15.43:0/2082664218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 0x55eee765fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.362+0000 7f96043abb80 1 --2- 172.21.15.43:0/2082664218 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee766aad0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.362+0000 7f96043abb80 1 -- 172.21.15.43:0/2082664218 >> 172.21.15.43:0/2082664218 conn(0x55eee7541fc0 msgr2=0x55eee765bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.362+0000 7f96043abb80 1 -- 172.21.15.43:0/2082664218 shutdown_connections 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.362+0000 7f96043abb80 1 -- 172.21.15.43:0/2082664218 wait complete. 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.363+0000 7f96043abb80 1 -- start start 2024-05-27T07:24:41.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.363+0000 7f96043abb80 1 Processor -- start 2024-05-27T07:24:41.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.363+0000 7f96043abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee74e1e40 0x55eee766a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.363+0000 7f96043abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660410 0x55eee766aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.363+0000 7f96043abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 0x55eee766b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.363+0000 7f96043abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55eee7484fa0 con 0x55eee74e1e40 2024-05-27T07:24:41.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.363+0000 7f96043abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55eee751f760 con 0x55eee7660410 2024-05-27T07:24:41.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.363+0000 7f96043abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55eee766d990 con 0x55eee7660820 2024-05-27T07:24:41.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.363+0000 7f96043abb80 10 client.0 _ll_register_callbacks cb 0x55eee7528c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:41.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.364+0000 7f95eea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee74e1e40 0x55eee766a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.364+0000 7f95eea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee74e1e40 0x55eee766a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38816/0 (socket says 172.21.15.43:38816) 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.364+0000 7f95eea00700 1 -- 172.21.15.43:0/2535809344 learned_addr learned my addr 172.21.15.43:0/2535809344 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.364+0000 7f95e6000700 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660410 0x55eee766aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.364+0000 7f95ef400700 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 0x55eee766b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.364+0000 7f95eea00700 1 -- 172.21.15.43:0/2535809344 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 msgr2=0x55eee766b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.364+0000 7f95eea00700 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 0x55eee766b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.364+0000 7f95eea00700 1 -- 172.21.15.43:0/2535809344 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660410 msgr2=0x55eee766aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.364+0000 7f95eea00700 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660410 0x55eee766aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.364+0000 7f95eea00700 1 -- 172.21.15.43:0/2535809344 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eee74ce8d0 con 0x55eee74e1e40 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.365+0000 7f95eea00700 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee74e1e40 0x55eee766a5a0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f95e00a6050 tx=0x7f95e00227e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac53fb0749bde2fa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.365+0000 7f95dcc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.365+0000 7f95dcc00700 1 -- 172.21.15.43:0/2535809344 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0092300 con 0x55eee74e1e40 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.365+0000 7f95dcc00700 1 -- 172.21.15.43:0/2535809344 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f95e00e3380 con 0x55eee74e1e40 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.365+0000 7f96043abb80 1 -- 172.21.15.43:0/2535809344 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95d8011bb0 con 0x55eee74e1e40 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.365+0000 7f95dcc00700 1 -- 172.21.15.43:0/2535809344 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f95e006b050 con 0x55eee74e1e40 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.365+0000 7f96043abb80 10 client.17013 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.365+0000 7f96043abb80 1 -- 172.21.15.43:0/2535809344 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55eee754f060 con 0x55eee74e1e40 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.366+0000 7f95dcc00700 1 -- 172.21.15.43:0/2535809344 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95d8011bb0 con 0x55eee74e1e40 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.366+0000 7f95dcc00700 1 -- 172.21.15.43:0/2535809344 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7f95e00b3090 con 0x55eee74e1e40 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.366+0000 7f96043abb80 10 client.17013 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.366+0000 7f96043abb80 10 client.17013 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.366+0000 7f96043abb80 20 client.17013 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.366+0000 7f96043abb80 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55eee767a950 0x55eee767cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.366+0000 7f96043abb80 4 client.17013 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:41.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.366+0000 7f96043abb80 1 -- 172.21.15.43:0/2535809344 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eee7590d90 con 0x55eee767a950 2024-05-27T07:24:41.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.367+0000 7f95e6000700 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55eee767a950 0x55eee767cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.368+0000 7f95e6000700 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55eee767a950 0x55eee767cdb0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=23c718f739544465 server_cookie=e1c8e78a92928745 in_seq=0 out_seq=0 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.368+0000 7f95dcc00700 10 client.17013 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f95dcc00700 1 -- 172.21.15.43:0/2535809344 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55eee7590d90 con 0x55eee767a950 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f95dcc00700 10 client.17013 handle_command_reply: tid=0 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 client.17013 shutdown 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 -- 172.21.15.43:0/2535809344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee74e1e40 msgr2=0x55eee766a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee74e1e40 0x55eee766a5a0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f95e00a6050 tx=0x7f95e00227e0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 -- 172.21.15.43:0/2535809344 shutdown_connections 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55eee767a950 0x55eee767cdb0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660820 0x55eee766b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee74e1e40 0x55eee766a5a0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 --2- 172.21.15.43:0/2535809344 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660410 0x55eee766aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 -- 172.21.15.43:0/2535809344 >> 172.21.15.43:0/2535809344 conn(0x55eee7541fc0 msgr2=0x55eee765aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f95dcc00700 0 client.17013 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:24:41.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 -- 172.21.15.43:0/2535809344 shutdown_connections 2024-05-27T07:24:41.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.370+0000 7f96043abb80 1 -- 172.21.15.43:0/2535809344 wait complete. 2024-05-27T07:24:41.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.371+0000 7f96043abb80 20 client.17013 trim_cache size 0 max 16384 2024-05-27T07:24:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.387+0000 7f96043abb80 1 Processor -- start 2024-05-27T07:24:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.387+0000 7f96043abb80 1 -- start start 2024-05-27T07:24:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.388+0000 7f96043abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee74e1e40 0x55eee766ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.388+0000 7f96043abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee7676a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.388+0000 7f96043abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660820 0x55eee766ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.388+0000 7f96043abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f95f01b76c0 con 0x55eee7660410 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.388+0000 7f96043abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f95f01b5360 con 0x55eee7660820 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.388+0000 7f96043abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f95f01b6ff0 con 0x55eee74e1e40 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.388+0000 7f95e6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee7676a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.388+0000 7f95e6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee7676a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38828/0 (socket says 172.21.15.43:38828) 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.388+0000 7f95e6000700 1 -- 172.21.15.43:0/234201546 learned_addr learned my addr 172.21.15.43:0/234201546 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.388+0000 7f95eea00700 1 --2- 172.21.15.43:0/234201546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee74e1e40 0x55eee766ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95ef400700 1 --2- 172.21.15.43:0/234201546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660820 0x55eee766ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95e6000700 1 -- 172.21.15.43:0/234201546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee74e1e40 msgr2=0x55eee766ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95e6000700 1 --2- 172.21.15.43:0/234201546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee74e1e40 0x55eee766ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95e6000700 1 -- 172.21.15.43:0/234201546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660820 msgr2=0x55eee766ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95e6000700 1 --2- 172.21.15.43:0/234201546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660820 0x55eee766ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95e6000700 1 -- 172.21.15.43:0/234201546 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eee7668b10 con 0x55eee7660410 2024-05-27T07:24:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95ef400700 1 --2- 172.21.15.43:0/234201546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660820 0x55eee766ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:24:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95e6000700 1 --2- 172.21.15.43:0/234201546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee7676a30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f95d8011a00 tx=0x7f95d8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8d47cc96987e7a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95df400700 1 -- 172.21.15.43:0/234201546 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8022070 con 0x55eee7660410 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95df400700 1 -- 172.21.15.43:0/234201546 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95d8039070 con 0x55eee7660410 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.389+0000 7f95df400700 1 -- 172.21.15.43:0/234201546 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8004d30 con 0x55eee7660410 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.390+0000 7f96043abb80 1 -- 172.21.15.43:0/234201546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 msgr2=0x55eee7676a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.390+0000 7f96043abb80 1 --2- 172.21.15.43:0/234201546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee7676a30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f95d8011a00 tx=0x7f95d8015000 comp rx=0 tx=0).stop 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.390+0000 7f96043abb80 1 -- 172.21.15.43:0/234201546 shutdown_connections 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.390+0000 7f96043abb80 1 --2- 172.21.15.43:0/234201546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee7660820 0x55eee766ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.390+0000 7f96043abb80 1 --2- 172.21.15.43:0/234201546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee74e1e40 0x55eee766ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.390+0000 7f96043abb80 1 --2- 172.21.15.43:0/234201546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660410 0x55eee7676a30 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.390+0000 7f96043abb80 1 -- 172.21.15.43:0/234201546 >> 172.21.15.43:0/234201546 conn(0x55eee7541fc0 msgr2=0x55eee766a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.390+0000 7f96043abb80 1 -- 172.21.15.43:0/234201546 shutdown_connections 2024-05-27T07:24:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.390+0000 7f96043abb80 1 -- 172.21.15.43:0/234201546 wait complete. 2024-05-27T07:24:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.391+0000 7f96043abb80 1 -- start start 2024-05-27T07:24:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.391+0000 7f96043abb80 1 Processor -- start 2024-05-27T07:24:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f96043abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 0x55eee766e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f96043abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660410 0x55eee766ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f96043abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660820 0x55eee766f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f96043abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f95f01b6160 con 0x55eee7660820 2024-05-27T07:24:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f96043abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f95f01b5a30 con 0x55eee74e1e40 2024-05-27T07:24:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f96043abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f95f01b68c0 con 0x55eee7660410 2024-05-27T07:24:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f96043abb80 10 client.0 _ll_register_callbacks cb 0x55eee7528c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f95ef400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660820 0x55eee766f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f95ef400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660820 0x55eee766f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38834/0 (socket says 172.21.15.43:38834) 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f95ef400700 1 -- 172.21.15.43:0/242009075 learned_addr learned my addr 172.21.15.43:0/242009075 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f95e6000700 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660410 0x55eee766ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.392+0000 7f95eea00700 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 0x55eee766e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f95ef400700 1 -- 172.21.15.43:0/242009075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660410 msgr2=0x55eee766ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f95ef400700 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660410 0x55eee766ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f95ef400700 1 -- 172.21.15.43:0/242009075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 msgr2=0x55eee766e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f95ef400700 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 0x55eee766e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f95ef400700 1 -- 172.21.15.43:0/242009075 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eee76e4440 con 0x55eee7660820 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f95eea00700 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 0x55eee766e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f95ef400700 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660820 0x55eee766f3c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f95e8007af0 tx=0x7f95e8007eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163d47b7c96e147 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f95dcc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f95dcc00700 1 -- 172.21.15.43:0/242009075 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e801b070 con 0x55eee7660820 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f96043abb80 1 -- 172.21.15.43:0/242009075 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95d8039070 con 0x55eee7660820 2024-05-27T07:24:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.393+0000 7f95dcc00700 1 -- 172.21.15.43:0/242009075 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f95e8013b00 con 0x55eee7660820 2024-05-27T07:24:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f95dcc00700 1 -- 172.21.15.43:0/242009075 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f95e8027050 con 0x55eee7660820 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f96043abb80 10 client.17019 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f96043abb80 1 -- 172.21.15.43:0/242009075 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55eee76e56b0 con 0x55eee7660820 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f95dcc00700 1 -- 172.21.15.43:0/242009075 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95d8039070 con 0x55eee7660820 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f95dcc00700 1 -- 172.21.15.43:0/242009075 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7f95e8021770 con 0x55eee7660820 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f96043abb80 10 client.17019 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f96043abb80 10 client.17019 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f96043abb80 20 client.17019 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f96043abb80 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55eee76e5da0 0x55eee76e81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f96043abb80 4 client.17019 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.394+0000 7f96043abb80 1 -- 172.21.15.43:0/242009075 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eee758a570 con 0x55eee76e5da0 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.395+0000 7f95eea00700 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55eee76e5da0 0x55eee76e81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.396+0000 7f95eea00700 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55eee76e5da0 0x55eee76e81c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=96ee755d26ff2be4 server_cookie=116bb4dca6af70c2 in_seq=0 out_seq=0 2024-05-27T07:24:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.396+0000 7f95dcc00700 10 client.17019 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.396+0000 7f95dcc00700 1 -- 172.21.15.43:0/242009075 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eee758a570 con 0x55eee76e5da0 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.396+0000 7f95dcc00700 10 client.17019 handle_command_reply: tid=0 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.396+0000 7f96043abb80 1 client.17019 shutdown 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 1 -- 172.21.15.43:0/242009075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660820 msgr2=0x55eee766f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660820 0x55eee766f3c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f95e8007af0 tx=0x7f95e8007eb0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 1 -- 172.21.15.43:0/242009075 shutdown_connections 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55eee76e5da0 0x55eee76e81c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f95dcc00700 0 client.17019 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eee7660820 0x55eee766f3c0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eee74e1e40 0x55eee766e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 1 --2- 172.21.15.43:0/242009075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eee7660410 0x55eee766ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 1 -- 172.21.15.43:0/242009075 >> 172.21.15.43:0/242009075 conn(0x55eee7541fc0 msgr2=0x55eee766af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 1 -- 172.21.15.43:0/242009075 shutdown_connections 2024-05-27T07:24:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 1 -- 172.21.15.43:0/242009075 wait complete. 2024-05-27T07:24:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.397+0000 7f96043abb80 20 client.17019 trim_cache size 0 max 16384 2024-05-27T07:24:41.389 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:24:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 -- 172.21.15.43:0/1066063967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f95d40b29f0 msgr2=0x7f95d40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f95d40b29f0 0x7f95d40b4f00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f95d8005950 tx=0x7f95d80058e0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 -- 172.21.15.43:0/1066063967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f0075b50 msgr2=0x7f95f01a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f0075b50 0x7f95f01a11a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f95e0000c00 tx=0x7f95e0009f90 comp rx=0 tx=0).stop 2024-05-27T07:24:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 -- 172.21.15.43:0/1066063967 shutdown_connections 2024-05-27T07:24:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f95d40b29f0 0x7f95d40b4f00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f95f0115170 0x7f95f01a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95f0075b50 0x7f95f01a11a0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 --2- 172.21.15.43:0/1066063967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95f00764a0 0x7f95f01a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 -- 172.21.15.43:0/1066063967 >> 172.21.15.43:0/1066063967 conn(0x7f95f0079d00 msgr2=0x7f95f0100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 -- 172.21.15.43:0/1066063967 shutdown_connections 2024-05-27T07:24:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:41.398+0000 7f95efe00700 1 -- 172.21.15.43:0/1066063967 wait complete. 2024-05-27T07:24:41.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:41 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi043 to 177.5M 2024-05-27T07:24:41.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:41 smithi043 ceph-mon[31531]: Unable to set osd_memory_target on smithi043 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:24:41.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:41 smithi043 ceph-mon[31531]: pgmap v364: 129 pgs: 129 active+clean; 108 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 495 KiB/s wr, 35 op/s 2024-05-27T07:24:41.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:41 smithi043 ceph-mon[31531]: Health check failed: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:24:41.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:41 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi043 to 177.5M 2024-05-27T07:24:41.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:41 smithi179 ceph-mon[35888]: Unable to set osd_memory_target on smithi043 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:24:41.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:41 smithi179 ceph-mon[35888]: pgmap v364: 129 pgs: 129 active+clean; 108 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 495 KiB/s wr, 35 op/s 2024-05-27T07:24:41.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:41 smithi179 ceph-mon[35888]: Health check failed: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:24:41.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:41 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi043 to 177.5M 2024-05-27T07:24:41.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:41 smithi132 ceph-mon[35283]: Unable to set osd_memory_target on smithi043 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:24:41.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:41 smithi132 ceph-mon[35283]: pgmap v364: 129 pgs: 129 active+clean; 108 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 495 KiB/s wr, 35 op/s 2024-05-27T07:24:41.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:41 smithi132 ceph-mon[35283]: Health check failed: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:24:42.230 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60395 0.61 MB/sec execute 406 sec latency 2718.411 ms 2024-05-27T07:24:42.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:24:42.758 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:43.230 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60395 0.61 MB/sec execute 407 sec latency 3718.579 ms 2024-05-27T07:24:43.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:43 smithi043 ceph-mon[31531]: pgmap v365: 129 pgs: 129 active+clean; 109 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 580 KiB/s wr, 34 op/s 2024-05-27T07:24:43.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:43 smithi179 ceph-mon[35888]: pgmap v365: 129 pgs: 129 active+clean; 109 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 580 KiB/s wr, 34 op/s 2024-05-27T07:24:43.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:43 smithi132 ceph-mon[35283]: pgmap v365: 129 pgs: 129 active+clean; 109 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 580 KiB/s wr, 34 op/s 2024-05-27T07:24:44.230 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60450 0.61 MB/sec execute 408 sec latency 3944.804 ms 2024-05-27T07:24:45.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.064+0000 7fa029400700 1 -- 172.21.15.43:0/1064751712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa024104640 msgr2=0x7fa024104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.064+0000 7fa029400700 1 --2- 172.21.15.43:0/1064751712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa024104640 0x7fa024104a50 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fa0140099c0 tx=0x7fa014009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.064+0000 7fa029400700 1 -- 172.21.15.43:0/1064751712 shutdown_connections 2024-05-27T07:24:45.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.064+0000 7fa029400700 1 --2- 172.21.15.43:0/1064751712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa024109870 0x7fa0241111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.064+0000 7fa029400700 1 --2- 172.21.15.43:0/1064751712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa024104640 0x7fa024104a50 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.064+0000 7fa029400700 1 --2- 172.21.15.43:0/1064751712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa024108f20 0x7fa024109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.064+0000 7fa029400700 1 -- 172.21.15.43:0/1064751712 >> 172.21.15.43:0/1064751712 conn(0x7fa024079ef0 msgr2=0x7fa02407a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:45.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.064+0000 7fa029400700 1 -- 172.21.15.43:0/1064751712 shutdown_connections 2024-05-27T07:24:45.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.065+0000 7fa029400700 1 -- 172.21.15.43:0/1064751712 wait complete. 2024-05-27T07:24:45.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.066+0000 7fa029400700 1 Processor -- start 2024-05-27T07:24:45.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.066+0000 7fa029400700 1 -- start start 2024-05-27T07:24:45.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.066+0000 7fa029400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa024104640 0x7fa02419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.066+0000 7fa029400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa024108f20 0x7fa02419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.066+0000 7fa029400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa024109870 0x7fa02419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.066+0000 7fa029400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa024114350 con 0x7fa024109870 2024-05-27T07:24:45.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.066+0000 7fa029400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241141d0 con 0x7fa024104640 2024-05-27T07:24:45.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.066+0000 7fa029400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241144d0 con 0x7fa024108f20 2024-05-27T07:24:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.067+0000 7fa023200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa024109870 0x7fa02419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.067+0000 7fa023200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa024109870 0x7fa02419f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43458/0 (socket says 172.21.15.43:43458) 2024-05-27T07:24:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.067+0000 7fa023200700 1 -- 172.21.15.43:0/3442470682 learned_addr learned my addr 172.21.15.43:0/3442470682 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.067+0000 7fa021e00700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa024108f20 0x7fa02419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.067+0000 7fa022800700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa024104640 0x7fa02419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.067+0000 7fa023200700 1 -- 172.21.15.43:0/3442470682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa024108f20 msgr2=0x7fa02419b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.067+0000 7fa023200700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa024108f20 0x7fa02419b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.067+0000 7fa023200700 1 -- 172.21.15.43:0/3442470682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa024104640 msgr2=0x7fa02419ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.067+0000 7fa023200700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa024104640 0x7fa02419ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.067+0000 7fa023200700 1 -- 172.21.15.43:0/3442470682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02419fb80 con 0x7fa024109870 2024-05-27T07:24:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.068+0000 7fa023200700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa024109870 0x7fa02419f590 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa01c007d70 tx=0x7fa01c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.068+0000 7fa01b400700 1 -- 172.21.15.43:0/3442470682 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c016020 con 0x7fa024109870 2024-05-27T07:24:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.068+0000 7fa01b400700 1 -- 172.21.15.43:0/3442470682 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa01c00ad40 con 0x7fa024109870 2024-05-27T07:24:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.068+0000 7fa01b400700 1 -- 172.21.15.43:0/3442470682 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c0115e0 con 0x7fa024109870 2024-05-27T07:24:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.068+0000 7fa029400700 1 -- 172.21.15.43:0/3442470682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa02419fe70 con 0x7fa024109870 2024-05-27T07:24:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.068+0000 7fa029400700 1 -- 172.21.15.43:0/3442470682 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0241a0360 con 0x7fa024109870 2024-05-27T07:24:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.072+0000 7fa01b400700 1 -- 172.21.15.43:0/3442470682 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa01c019020 con 0x7fa024109870 2024-05-27T07:24:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.073+0000 7fa037f31b80 1 Processor -- start 2024-05-27T07:24:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.073+0000 7fa037f31b80 1 -- start start 2024-05-27T07:24:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.074+0000 7fa01b400700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0080b29f0 0x7fa0080b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.074+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.074+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066eec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.074+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066ef5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.074+0000 7fa037f31b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562d06618e40 con 0x562d066f50e0 2024-05-27T07:24:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.074+0000 7fa037f31b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562d066089e0 con 0x562d0656be40 2024-05-27T07:24:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.074+0000 7fa037f31b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562d0660ca20 con 0x562d066ef1c0 2024-05-27T07:24:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.074+0000 7fa01b400700 1 -- 172.21.15.43:0/3442470682 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7fa01c0e2870 con 0x7fa024109870 2024-05-27T07:24:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.074+0000 7fa021e00700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0080b29f0 0x7fa0080b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.074+0000 7fa021e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066eec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.075+0000 7fa021e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066eec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43460/0 (socket says 172.21.15.43:43460) 2024-05-27T07:24:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.075+0000 7fa021e00700 1 -- 172.21.15.43:0/1654897647 learned_addr learned my addr 172.21.15.43:0/1654897647 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.075+0000 7fa022800700 1 --2- 172.21.15.43:0/1654897647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.075+0000 7fa023200700 1 --2- 172.21.15.43:0/1654897647 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066ef5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.075+0000 7fa021e00700 1 -- 172.21.15.43:0/1654897647 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 msgr2=0x562d066ef5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.075+0000 7fa021e00700 1 --2- 172.21.15.43:0/1654897647 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066ef5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.075+0000 7fa021e00700 1 -- 172.21.15.43:0/1654897647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 msgr2=0x562d066f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.075+0000 7fa021e00700 1 --2- 172.21.15.43:0/1654897647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.075+0000 7fa021e00700 1 -- 172.21.15.43:0/1654897647 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d065d1420 con 0x562d066f50e0 2024-05-27T07:24:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.076+0000 7fa021e00700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0080b29f0 0x7fa0080b4f00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa010014a60 tx=0x7fa01000cc30 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.076+0000 7fa021e00700 1 --2- 172.21.15.43:0/1654897647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066eec80 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa010011a90 tx=0x7fa01001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71e2c099f450464 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.076+0000 7fa00fe00700 1 -- 172.21.15.43:0/1654897647 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa010022070 con 0x562d066f50e0 2024-05-27T07:24:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.076+0000 7fa00fe00700 1 -- 172.21.15.43:0/1654897647 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa010014070 con 0x562d066f50e0 2024-05-27T07:24:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.076+0000 7fa00fe00700 1 -- 172.21.15.43:0/1654897647 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa010012450 con 0x562d066f50e0 2024-05-27T07:24:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.077+0000 7fa037f31b80 1 -- 172.21.15.43:0/1654897647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 msgr2=0x562d066eec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.077+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1654897647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066eec80 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa010011a90 tx=0x7fa01001b000 comp rx=0 tx=0).stop 2024-05-27T07:24:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.077+0000 7fa037f31b80 1 -- 172.21.15.43:0/1654897647 shutdown_connections 2024-05-27T07:24:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.077+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1654897647 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066ef5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.077+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1654897647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.077+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1654897647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066eec80 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.077+0000 7fa037f31b80 1 -- 172.21.15.43:0/1654897647 >> 172.21.15.43:0/1654897647 conn(0x562d065cbfc0 msgr2=0x562d066e58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.077+0000 7fa037f31b80 1 -- 172.21.15.43:0/1654897647 shutdown_connections 2024-05-27T07:24:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.077+0000 7fa037f31b80 1 -- 172.21.15.43:0/1654897647 wait complete. 2024-05-27T07:24:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.078+0000 7fa037f31b80 1 -- start start 2024-05-27T07:24:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.078+0000 7fa037f31b80 1 Processor -- start 2024-05-27T07:24:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066f8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa037f31b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562d0650efa0 con 0x562d066f50e0 2024-05-27T07:24:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa037f31b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562d065a9760 con 0x562d0656be40 2024-05-27T07:24:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa037f31b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562d066fbbc0 con 0x562d066ef1c0 2024-05-27T07:24:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa037f31b80 10 client.0 _ll_register_callbacks cb 0x562d065b2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa023200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa023200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43464/0 (socket says 172.21.15.43:43464) 2024-05-27T07:24:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa023200700 1 -- 172.21.15.43:0/1061761869 learned_addr learned my addr 172.21.15.43:0/1061761869 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa021e00700 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066f8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.079+0000 7fa022800700 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.080+0000 7fa023200700 1 -- 172.21.15.43:0/1061761869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 msgr2=0x562d066f8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:45.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.080+0000 7fa023200700 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066f8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.080+0000 7fa023200700 1 -- 172.21.15.43:0/1061761869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 msgr2=0x562d066f89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.080+0000 7fa023200700 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.080+0000 7fa023200700 1 -- 172.21.15.43:0/1061761869 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d065588d0 con 0x562d066f50e0 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.080+0000 7fa023200700 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f9420 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa01c007d40 tx=0x7fa01c0a9230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e602978fd0548c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.080+0000 7fa00cc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.080+0000 7fa00cc00700 1 -- 172.21.15.43:0/1061761869 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c0a9340 con 0x562d066f50e0 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.080+0000 7fa00cc00700 1 -- 172.21.15.43:0/1061761869 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7fa01c0e25f0 con 0x562d066f50e0 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.080+0000 7fa037f31b80 1 -- 172.21.15.43:0/1061761869 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa010014070 con 0x562d066f50e0 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa00cc00700 1 -- 172.21.15.43:0/1061761869 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fa01c0a9f30 con 0x562d066f50e0 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa037f31b80 10 client.17031 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa037f31b80 1 -- 172.21.15.43:0/1061761869 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x562d065d9060 con 0x562d066f50e0 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa00cc00700 1 -- 172.21.15.43:0/1061761869 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa010014070 con 0x562d066f50e0 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa00cc00700 1 -- 172.21.15.43:0/1061761869 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7fa01c06b020 con 0x562d066f50e0 2024-05-27T07:24:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa037f31b80 10 client.17031 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa037f31b80 10 client.17031 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:24:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa037f31b80 20 client.17031 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562d06708f50 0x562d0670b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa037f31b80 4 client.17031 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.081+0000 7fa037f31b80 1 -- 172.21.15.43:0/1061761869 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d0661ad90 con 0x562d06708f50 2024-05-27T07:24:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.082+0000 7fa022800700 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562d06708f50 0x562d0670b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.083+0000 7fa022800700 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562d06708f50 0x562d0670b3e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f59ea1482c66868a server_cookie=ccb2966b0e9b9e63 in_seq=0 out_seq=0 2024-05-27T07:24:45.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.083+0000 7fa00cc00700 10 client.17031 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:24:45.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.085+0000 7fa00cc00700 1 -- 172.21.15.43:0/1061761869 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562d0661ad90 con 0x562d06708f50 2024-05-27T07:24:45.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.085+0000 7fa00cc00700 10 client.17031 handle_command_reply: tid=0 2024-05-27T07:24:45.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.085+0000 7fa037f31b80 1 client.17031 shutdown 2024-05-27T07:24:45.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 1 -- 172.21.15.43:0/1061761869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 msgr2=0x562d066f9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f9420 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa01c007d40 tx=0x7fa01c0a9230 comp rx=0 tx=0).stop 2024-05-27T07:24:45.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 1 -- 172.21.15.43:0/1061761869 shutdown_connections 2024-05-27T07:24:45.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f9420 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562d06708f50 0x562d0670b3e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1061761869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066f8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 1 -- 172.21.15.43:0/1061761869 >> 172.21.15.43:0/1061761869 conn(0x562d065cbfc0 msgr2=0x562d066e5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa00cc00700 0 client.17031 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:24:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 1 -- 172.21.15.43:0/1061761869 shutdown_connections 2024-05-27T07:24:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 1 -- 172.21.15.43:0/1061761869 wait complete. 2024-05-27T07:24:45.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.086+0000 7fa037f31b80 20 client.17031 trim_cache size 0 max 16384 2024-05-27T07:24:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.103+0000 7fa037f31b80 1 Processor -- start 2024-05-27T07:24:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.104+0000 7fa037f31b80 1 -- start start 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.104+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.104+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.104+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.104+0000 7fa037f31b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241aef80 con 0x562d066f50e0 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.104+0000 7fa037f31b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241acc20 con 0x562d0656be40 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.104+0000 7fa037f31b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241ae8b0 con 0x562d066ef1c0 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa023200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa023200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43474/0 (socket says 172.21.15.43:43474) 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa023200700 1 -- 172.21.15.43:0/1259528650 learned_addr learned my addr 172.21.15.43:0/1259528650 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa022800700 1 --2- 172.21.15.43:0/1259528650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa021e00700 1 --2- 172.21.15.43:0/1259528650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa023200700 1 -- 172.21.15.43:0/1259528650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 msgr2=0x562d066f2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:45.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa023200700 1 --2- 172.21.15.43:0/1259528650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066f2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa023200700 1 -- 172.21.15.43:0/1259528650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 msgr2=0x562d066f2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:45.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa023200700 1 --2- 172.21.15.43:0/1259528650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa023200700 1 -- 172.21.15.43:0/1259528650 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d066f9790 con 0x562d066f50e0 2024-05-27T07:24:45.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.105+0000 7fa021e00700 1 --2- 172.21.15.43:0/1259528650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.106+0000 7fa023200700 1 --2- 172.21.15.43:0/1259528650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f2fe0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0a58b0 tx=0x7fa01c007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f18764a41d509a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.106+0000 7fa00f400700 1 -- 172.21.15.43:0/1259528650 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c088070 con 0x562d066f50e0 2024-05-27T07:24:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.106+0000 7fa00f400700 1 -- 172.21.15.43:0/1259528650 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa01c0c6070 con 0x562d066f50e0 2024-05-27T07:24:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.106+0000 7fa00f400700 1 -- 172.21.15.43:0/1259528650 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c0b2690 con 0x562d066f50e0 2024-05-27T07:24:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.107+0000 7fa037f31b80 1 -- 172.21.15.43:0/1259528650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 msgr2=0x562d066f2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.107+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1259528650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f2fe0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0a58b0 tx=0x7fa01c007f50 comp rx=0 tx=0).stop 2024-05-27T07:24:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.107+0000 7fa037f31b80 1 -- 172.21.15.43:0/1259528650 shutdown_connections 2024-05-27T07:24:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.107+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1259528650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d066f2fe0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.107+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1259528650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d0656be40 0x562d066f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.107+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1259528650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d066ef1c0 0x562d066f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.107+0000 7fa037f31b80 1 -- 172.21.15.43:0/1259528650 >> 172.21.15.43:0/1259528650 conn(0x562d065cbfc0 msgr2=0x562d066f7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.107+0000 7fa037f31b80 1 -- 172.21.15.43:0/1259528650 shutdown_connections 2024-05-27T07:24:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.107+0000 7fa037f31b80 1 -- 172.21.15.43:0/1259528650 wait complete. 2024-05-27T07:24:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.107+0000 7fa037f31b80 1 -- start start 2024-05-27T07:24:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.108+0000 7fa037f31b80 1 Processor -- start 2024-05-27T07:24:45.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.108+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d0656be40 0x562d06711370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.108+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d066ef1c0 0x562d06700eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.108+0000 7fa037f31b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d06701410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.108+0000 7fa037f31b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241ada20 con 0x562d066f50e0 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.108+0000 7fa037f31b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241ad2f0 con 0x562d066ef1c0 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.108+0000 7fa037f31b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241ae180 con 0x562d0656be40 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.108+0000 7fa037f31b80 10 client.0 _ll_register_callbacks cb 0x562d065b2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa023200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d06701410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa023200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d06701410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43476/0 (socket says 172.21.15.43:43476) 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa023200700 1 -- 172.21.15.43:0/1088932826 learned_addr learned my addr 172.21.15.43:0/1088932826 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa021e00700 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d066ef1c0 0x562d06700eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa022800700 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d0656be40 0x562d06711370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa023200700 1 -- 172.21.15.43:0/1088932826 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d0656be40 msgr2=0x562d06711370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:45.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa023200700 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d0656be40 0x562d06711370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa023200700 1 -- 172.21.15.43:0/1088932826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d066ef1c0 msgr2=0x562d06700eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa023200700 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d066ef1c0 0x562d06700eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa023200700 1 -- 172.21.15.43:0/1088932826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d0670fbd0 con 0x562d066f50e0 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa023200700 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d06701410 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0a97c0 tx=0x7fa01c00af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aae645c33c61e53 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.109+0000 7fa00cc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.110+0000 7fa00cc00700 1 -- 172.21.15.43:0/1088932826 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c088070 con 0x562d066f50e0 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.110+0000 7fa00cc00700 1 -- 172.21.15.43:0/1088932826 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7fa01c0a5e20 con 0x562d066f50e0 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.110+0000 7fa037f31b80 1 -- 172.21.15.43:0/1088932826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa01c0c6070 con 0x562d066f50e0 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.110+0000 7fa00cc00700 1 -- 172.21.15.43:0/1088932826 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fa01c0adbe0 con 0x562d066f50e0 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.110+0000 7fa037f31b80 10 client.17037 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:45.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.110+0000 7fa037f31b80 1 -- 172.21.15.43:0/1088932826 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x562d06772c50 con 0x562d066f50e0 2024-05-27T07:24:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.110+0000 7fa00cc00700 1 -- 172.21.15.43:0/1088932826 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa01c0c6070 con 0x562d066f50e0 2024-05-27T07:24:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.110+0000 7fa00cc00700 1 -- 172.21.15.43:0/1088932826 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7fa01c06b050 con 0x562d066f50e0 2024-05-27T07:24:45.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.110+0000 7fa037f31b80 10 client.17037 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.110+0000 7fa037f31b80 10 client.17037 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:24:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.111+0000 7fa037f31b80 20 client.17037 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.111+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562d06772f20 0x562d06775340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.111+0000 7fa037f31b80 4 client.17037 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:24:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.111+0000 7fa037f31b80 1 -- 172.21.15.43:0/1088932826 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562d06614570 con 0x562d06772f20 2024-05-27T07:24:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.111+0000 7fa022800700 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562d06772f20 0x562d06775340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.113+0000 7fa022800700 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562d06772f20 0x562d06775340 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2c93debd8dba0c4d server_cookie=253071820a16ff06 in_seq=0 out_seq=0 2024-05-27T07:24:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.113+0000 7fa00cc00700 10 client.17037 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:24:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.113+0000 7fa00cc00700 1 -- 172.21.15.43:0/1088932826 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562d06614570 con 0x562d06772f20 2024-05-27T07:24:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.113+0000 7fa00cc00700 10 client.17037 handle_command_reply: tid=0 2024-05-27T07:24:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.113+0000 7fa037f31b80 1 client.17037 shutdown 2024-05-27T07:24:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 1 -- 172.21.15.43:0/1088932826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 msgr2=0x562d06701410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d06701410 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0a97c0 tx=0x7fa01c00af60 comp rx=0 tx=0).stop 2024-05-27T07:24:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 1 -- 172.21.15.43:0/1088932826 shutdown_connections 2024-05-27T07:24:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d066f50e0 0x562d06701410 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562d0656be40 0x562d06711370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x562d06772f20 0x562d06775340 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 1 --2- 172.21.15.43:0/1088932826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d066ef1c0 0x562d06700eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 1 -- 172.21.15.43:0/1088932826 >> 172.21.15.43:0/1088932826 conn(0x562d065cbfc0 msgr2=0x562d066f6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa00cc00700 0 client.17037 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:24:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 1 -- 172.21.15.43:0/1088932826 shutdown_connections 2024-05-27T07:24:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 1 -- 172.21.15.43:0/1088932826 wait complete. 2024-05-27T07:24:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.114+0000 7fa037f31b80 20 client.17037 trim_cache size 0 max 16384 2024-05-27T07:24:45.105 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:24:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 -- 172.21.15.43:0/3442470682 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0080b29f0 msgr2=0x7fa0080b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0080b29f0 0x7fa0080b4f00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa010014a60 tx=0x7fa01000cc30 comp rx=0 tx=0).stop 2024-05-27T07:24:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 -- 172.21.15.43:0/3442470682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa024109870 msgr2=0x7fa02419f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa024109870 0x7fa02419f590 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa01c007d70 tx=0x7fa01c00a340 comp rx=0 tx=0).stop 2024-05-27T07:24:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 -- 172.21.15.43:0/3442470682 shutdown_connections 2024-05-27T07:24:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa0080b29f0 0x7fa0080b4f00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa024109870 0x7fa02419f590 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa024104640 0x7fa02419ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 --2- 172.21.15.43:0/3442470682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa024108f20 0x7fa02419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 -- 172.21.15.43:0/3442470682 >> 172.21.15.43:0/3442470682 conn(0x7fa024079ef0 msgr2=0x7fa024102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 -- 172.21.15.43:0/3442470682 shutdown_connections 2024-05-27T07:24:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:45.115+0000 7fa029400700 1 -- 172.21.15.43:0/3442470682 wait complete. 2024-05-27T07:24:45.230 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60462 0.60 MB/sec execute 409 sec latency 882.056 ms 2024-05-27T07:24:45.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:45 smithi043 ceph-mon[31531]: pgmap v366: 129 pgs: 129 active+clean; 109 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 580 KiB/s wr, 34 op/s 2024-05-27T07:24:45.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:45 smithi179 ceph-mon[35888]: pgmap v366: 129 pgs: 129 active+clean; 109 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 580 KiB/s wr, 34 op/s 2024-05-27T07:24:45.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:45 smithi132 ceph-mon[35283]: pgmap v366: 129 pgs: 129 active+clean; 109 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 580 KiB/s wr, 34 op/s 2024-05-27T07:24:45.995 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:24:46.230 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60462 0.60 MB/sec execute 410 sec latency 1882.191 ms 2024-05-27T07:24:46.462 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:47.231 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60462 0.60 MB/sec execute 411 sec latency 2882.351 ms 2024-05-27T07:24:47.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:47 smithi043 ceph-mon[31531]: pgmap v367: 129 pgs: 129 active+clean; 110 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 632 KiB/s wr, 42 op/s 2024-05-27T07:24:47.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:47 smithi132 ceph-mon[35283]: pgmap v367: 129 pgs: 129 active+clean; 110 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 632 KiB/s wr, 42 op/s 2024-05-27T07:24:47.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:47 smithi179 ceph-mon[35888]: pgmap v367: 129 pgs: 129 active+clean; 110 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 632 KiB/s wr, 42 op/s 2024-05-27T07:24:48.230 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60462 0.60 MB/sec execute 412 sec latency 3882.443 ms 2024-05-27T07:24:48.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.857+0000 7f463c200700 1 -- 172.21.15.43:0/1167574338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4634108f10 msgr2=0x7f4634109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.857+0000 7f463c200700 1 --2- 172.21.15.43:0/1167574338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4634108f10 0x7f4634109320 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f46200099c0 tx=0x7f4620009cd0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.857+0000 7f463c200700 1 -- 172.21.15.43:0/1167574338 shutdown_connections 2024-05-27T07:24:48.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.857+0000 7f463c200700 1 --2- 172.21.15.43:0/1167574338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4634109860 0x7f46341111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.857+0000 7f463c200700 1 --2- 172.21.15.43:0/1167574338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46341045e0 0x7f46341049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.857+0000 7f463c200700 1 --2- 172.21.15.43:0/1167574338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4634108f10 0x7f4634109320 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.857+0000 7f463c200700 1 -- 172.21.15.43:0/1167574338 >> 172.21.15.43:0/1167574338 conn(0x7f4634100030 msgr2=0x7f4634102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:48.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.857+0000 7f463c200700 1 -- 172.21.15.43:0/1167574338 shutdown_connections 2024-05-27T07:24:48.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.857+0000 7f463c200700 1 -- 172.21.15.43:0/1167574338 wait complete. 2024-05-27T07:24:48.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.859+0000 7f463c200700 1 Processor -- start 2024-05-27T07:24:48.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.859+0000 7f463c200700 1 -- start start 2024-05-27T07:24:48.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.859+0000 7f463c200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46341045e0 0x7f463419d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.859+0000 7f463c200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4634109860 0x7f463419d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.860+0000 7f463c200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46341a18d0 0x7f46341a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.860+0000 7f463c200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4634114370 con 0x7f46341a18d0 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.860+0000 7f463c200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46341141f0 con 0x7f4634109860 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.860+0000 7f463c200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f46341144f0 con 0x7f46341045e0 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.860+0000 7f463a200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46341a18d0 0x7f46341a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.860+0000 7f463a200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46341a18d0 0x7f46341a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43492/0 (socket says 172.21.15.43:43492) 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.860+0000 7f463a200700 1 -- 172.21.15.43:0/3402315606 learned_addr learned my addr 172.21.15.43:0/3402315606 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.860+0000 7f4639800700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46341045e0 0x7f463419d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.860+0000 7f4638e00700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4634109860 0x7f463419d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.861+0000 7f463a200700 1 -- 172.21.15.43:0/3402315606 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46341045e0 msgr2=0x7f463419d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.861+0000 7f463a200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46341045e0 0x7f463419d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.861+0000 7f463a200700 1 -- 172.21.15.43:0/3402315606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4634109860 msgr2=0x7f463419d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.861+0000 7f463a200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4634109860 0x7f463419d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.861+0000 7f463a200700 1 -- 172.21.15.43:0/3402315606 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46341a2420 con 0x7f46341a18d0 2024-05-27T07:24:48.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.861+0000 7f463a200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46341a18d0 0x7f46341a1ce0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f462c00d980 tx=0x7f462c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:48.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.861+0000 7f4632000700 1 -- 172.21.15.43:0/3402315606 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c017070 con 0x7f46341a18d0 2024-05-27T07:24:48.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.862+0000 7f463c200700 1 -- 172.21.15.43:0/3402315606 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46341a2710 con 0x7f46341a18d0 2024-05-27T07:24:48.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.862+0000 7f4632000700 1 -- 172.21.15.43:0/3402315606 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f462c005020 con 0x7f46341a18d0 2024-05-27T07:24:48.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.862+0000 7f4632000700 1 -- 172.21.15.43:0/3402315606 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c00f980 con 0x7f46341a18d0 2024-05-27T07:24:48.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.862+0000 7f463c200700 1 -- 172.21.15.43:0/3402315606 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46341a9ec0 con 0x7f46341a18d0 2024-05-27T07:24:48.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.866+0000 7f4632000700 1 -- 172.21.15.43:0/3402315606 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f462c004a30 con 0x7f46341a18d0 2024-05-27T07:24:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.867+0000 7f464ae3fb80 1 Processor -- start 2024-05-27T07:24:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.867+0000 7f464ae3fb80 1 -- start start 2024-05-27T07:24:48.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd00be40 0x5594bd1907f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd190d30 0x5594bd18a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd18adb0 0x5594bd18b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5594bd0b8e40 con 0x5594bd18adb0 2024-05-27T07:24:48.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5594bd0a89e0 con 0x5594bd190d30 2024-05-27T07:24:48.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5594bd0aca20 con 0x5594bd00be40 2024-05-27T07:24:48.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f4632000700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f461c0b2a70 0x7f461c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f4638e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd18adb0 0x5594bd18b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f4639800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd190d30 0x5594bd18a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f4638e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd00be40 0x5594bd1907f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f4632000700 1 -- 172.21.15.43:0/3402315606 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f462c0e26f0 con 0x7f46341a18d0 2024-05-27T07:24:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f4638e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd00be40 0x5594bd1907f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:34162/0 (socket says 172.21.15.43:34162) 2024-05-27T07:24:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.868+0000 7f4638e00700 1 -- 172.21.15.43:0/2122449619 learned_addr learned my addr 172.21.15.43:0/2122449619 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:48.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.869+0000 7f463a200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f461c0b2a70 0x7f461c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.869+0000 7f4638e00700 1 -- 172.21.15.43:0/2122449619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd190d30 msgr2=0x5594bd18a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:48.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.869+0000 7f4638e00700 1 --2- 172.21.15.43:0/2122449619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd190d30 0x5594bd18a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.869+0000 7f4638e00700 1 -- 172.21.15.43:0/2122449619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd18adb0 msgr2=0x5594bd18b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:48.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.869+0000 7f4638e00700 1 --2- 172.21.15.43:0/2122449619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd18adb0 0x5594bd18b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.869+0000 7f4638e00700 1 -- 172.21.15.43:0/2122449619 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594bd071420 con 0x5594bd00be40 2024-05-27T07:24:48.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.869+0000 7f4639800700 1 --2- 172.21.15.43:0/2122449619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd190d30 0x5594bd18a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:24:48.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.869+0000 7f463a200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f461c0b2a70 0x7f461c0b4f80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f462c004800 tx=0x7f462c004750 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:48.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.870+0000 7f4638e00700 1 --2- 172.21.15.43:0/2122449619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd00be40 0x5594bd1907f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4620006010 tx=0x7f4620000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc2c29ed1e3bc76d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:48.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.870+0000 7f4626a00700 1 -- 172.21.15.43:0/2122449619 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620021040 con 0x5594bd00be40 2024-05-27T07:24:48.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.870+0000 7f4626a00700 1 -- 172.21.15.43:0/2122449619 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46200056d0 con 0x5594bd00be40 2024-05-27T07:24:48.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.870+0000 7f4626a00700 1 -- 172.21.15.43:0/2122449619 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620003ab0 con 0x5594bd00be40 2024-05-27T07:24:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.871+0000 7f464ae3fb80 1 -- 172.21.15.43:0/2122449619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd00be40 msgr2=0x5594bd1907f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.871+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/2122449619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd00be40 0x5594bd1907f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4620006010 tx=0x7f4620000bc0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.871+0000 7f464ae3fb80 1 -- 172.21.15.43:0/2122449619 shutdown_connections 2024-05-27T07:24:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.871+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/2122449619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd18adb0 0x5594bd18b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.871+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/2122449619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd00be40 0x5594bd1907f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.871+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/2122449619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd190d30 0x5594bd18a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.871+0000 7f464ae3fb80 1 -- 172.21.15.43:0/2122449619 >> 172.21.15.43:0/2122449619 conn(0x5594bd06bfc0 msgr2=0x5594bd1858d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.872+0000 7f464ae3fb80 1 -- 172.21.15.43:0/2122449619 shutdown_connections 2024-05-27T07:24:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.872+0000 7f464ae3fb80 1 -- 172.21.15.43:0/2122449619 wait complete. 2024-05-27T07:24:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.872+0000 7f464ae3fb80 1 -- start start 2024-05-27T07:24:48.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.873+0000 7f464ae3fb80 1 Processor -- start 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.873+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd00be40 0x5594bd194420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.873+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd18adb0 0x5594bd194960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.873+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd190d30 0x5594bd194ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.873+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5594bcfaefa0 con 0x5594bd190d30 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.873+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5594bd049760 con 0x5594bd00be40 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.873+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5594bd1977a0 con 0x5594bd18adb0 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.873+0000 7f464ae3fb80 10 client.0 _ll_register_callbacks cb 0x5594bd052c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.874+0000 7f4638e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd190d30 0x5594bd194ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.874+0000 7f4638e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd190d30 0x5594bd194ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43508/0 (socket says 172.21.15.43:43508) 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.874+0000 7f4638e00700 1 -- 172.21.15.43:0/1494669833 learned_addr learned my addr 172.21.15.43:0/1494669833 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.874+0000 7f4638e00700 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd00be40 0x5594bd194420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.874+0000 7f4639800700 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd18adb0 0x5594bd194960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.874+0000 7f4638e00700 1 -- 172.21.15.43:0/1494669833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd18adb0 msgr2=0x5594bd194960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:48.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.874+0000 7f4638e00700 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd18adb0 0x5594bd194960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.874+0000 7f4638e00700 1 -- 172.21.15.43:0/1494669833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd00be40 msgr2=0x5594bd194420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:48.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.874+0000 7f4638e00700 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd00be40 0x5594bd194420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.874+0000 7f4638e00700 1 -- 172.21.15.43:0/1494669833 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594bcff88d0 con 0x5594bd190d30 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.875+0000 7f4638e00700 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd190d30 0x5594bd194ea0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f46200037e0 tx=0x7f4620019040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dacfcc0995832e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.875+0000 7f461b400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.875+0000 7f461b400700 1 -- 172.21.15.43:0/1494669833 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620004db0 con 0x5594bd190d30 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.875+0000 7f461b400700 1 -- 172.21.15.43:0/1494669833 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f4620003850 con 0x5594bd190d30 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.875+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1494669833 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46200056d0 con 0x5594bd190d30 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.875+0000 7f461b400700 1 -- 172.21.15.43:0/1494669833 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f462003d050 con 0x5594bd190d30 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.875+0000 7f464ae3fb80 10 client.17049 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.875+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1494669833 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x5594bd079060 con 0x5594bd190d30 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.876+0000 7f461b400700 1 -- 172.21.15.43:0/1494669833 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46200056d0 con 0x5594bd190d30 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.876+0000 7f461b400700 1 -- 172.21.15.43:0/1494669833 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7f4620035020 con 0x5594bd190d30 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.876+0000 7f464ae3fb80 10 client.17049 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.876+0000 7f464ae3fb80 10 client.17049 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:24:48.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.876+0000 7f464ae3fb80 20 client.17049 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:48.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.876+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5594bd1accf0 0x5594bd1af110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.876+0000 7f464ae3fb80 4 client.17049 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:48.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.876+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1494669833 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594bd0bad90 con 0x5594bd1accf0 2024-05-27T07:24:48.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.877+0000 7f4639800700 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5594bd1accf0 0x5594bd1af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.878+0000 7f4639800700 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5594bd1accf0 0x5594bd1af110 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc87a2136728a94e server_cookie=5a529739df43c803 in_seq=0 out_seq=0 2024-05-27T07:24:48.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.878+0000 7f461b400700 10 client.17049 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:24:48.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f461b400700 1 -- 172.21.15.43:0/1494669833 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5594bd0bad90 con 0x5594bd1accf0 2024-05-27T07:24:48.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f461b400700 10 client.17049 handle_command_reply: tid=0 2024-05-27T07:24:48.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f464ae3fb80 1 client.17049 shutdown 2024-05-27T07:24:48.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1494669833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd190d30 msgr2=0x5594bd194ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd190d30 0x5594bd194ea0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f46200037e0 tx=0x7f4620019040 comp rx=0 tx=0).stop 2024-05-27T07:24:48.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1494669833 shutdown_connections 2024-05-27T07:24:48.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5594bd1accf0 0x5594bd1af110 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd190d30 0x5594bd194ea0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd00be40 0x5594bd194420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1494669833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd18adb0 0x5594bd194960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1494669833 >> 172.21.15.43:0/1494669833 conn(0x5594bd06bfc0 msgr2=0x5594bd185580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:48.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.880+0000 7f461b400700 0 client.17049 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:24:48.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.881+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1494669833 shutdown_connections 2024-05-27T07:24:48.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.881+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1494669833 wait complete. 2024-05-27T07:24:48.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.881+0000 7f464ae3fb80 20 client.17049 trim_cache size 0 max 16384 2024-05-27T07:24:48.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.899+0000 7f464ae3fb80 1 Processor -- start 2024-05-27T07:24:48.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.899+0000 7f464ae3fb80 1 -- start start 2024-05-27T07:24:48.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.899+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd18a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.899+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd18adb0 0x5594bd1968c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.899+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd190d30 0x5594bd1a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.899+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5594bd193ea0 con 0x5594bd00be40 2024-05-27T07:24:48.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.899+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46341af630 con 0x5594bd18adb0 2024-05-27T07:24:48.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.899+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f46341ad2d0 con 0x5594bd190d30 2024-05-27T07:24:48.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.900+0000 7f4638e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd18a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.900+0000 7f4638e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd18a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43516/0 (socket says 172.21.15.43:43516) 2024-05-27T07:24:48.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.900+0000 7f4638e00700 1 -- 172.21.15.43:0/345072708 learned_addr learned my addr 172.21.15.43:0/345072708 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:48.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.900+0000 7f4639800700 1 --2- 172.21.15.43:0/345072708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd18adb0 0x5594bd1968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.900+0000 7f4638e00700 1 -- 172.21.15.43:0/345072708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd190d30 msgr2=0x5594bd1a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:48.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.900+0000 7f4638e00700 1 --2- 172.21.15.43:0/345072708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd190d30 0x5594bd1a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.900+0000 7f4638e00700 1 -- 172.21.15.43:0/345072708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd18adb0 msgr2=0x5594bd1968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:48.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.900+0000 7f4638e00700 1 --2- 172.21.15.43:0/345072708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd18adb0 0x5594bd1968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.900+0000 7f4638e00700 1 -- 172.21.15.43:0/345072708 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594bd1971c0 con 0x5594bd00be40 2024-05-27T07:24:48.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.900+0000 7f4638e00700 1 --2- 172.21.15.43:0/345072708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd18a870 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f462000bcb0 tx=0x7f4620009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6aac4883004ad3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:48.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.901+0000 7f4626000700 1 -- 172.21.15.43:0/345072708 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620009510 con 0x5594bd00be40 2024-05-27T07:24:48.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.901+0000 7f4626000700 1 -- 172.21.15.43:0/345072708 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4620004db0 con 0x5594bd00be40 2024-05-27T07:24:48.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.901+0000 7f4626000700 1 -- 172.21.15.43:0/345072708 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620029c10 con 0x5594bd00be40 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.902+0000 7f464ae3fb80 1 -- 172.21.15.43:0/345072708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 msgr2=0x5594bd18a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.902+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/345072708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd18a870 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f462000bcb0 tx=0x7f4620009810 comp rx=0 tx=0).stop 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.902+0000 7f464ae3fb80 1 -- 172.21.15.43:0/345072708 shutdown_connections 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.902+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/345072708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd190d30 0x5594bd1a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.902+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/345072708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd18a870 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.902+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/345072708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd18adb0 0x5594bd1968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.902+0000 7f464ae3fb80 1 -- 172.21.15.43:0/345072708 >> 172.21.15.43:0/345072708 conn(0x5594bd06bfc0 msgr2=0x5594bd18e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.902+0000 7f464ae3fb80 1 -- 172.21.15.43:0/345072708 shutdown_connections 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.902+0000 7f464ae3fb80 1 -- 172.21.15.43:0/345072708 wait complete. 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.903+0000 7f464ae3fb80 1 -- start start 2024-05-27T07:24:48.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.903+0000 7f464ae3fb80 1 Processor -- start 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd1a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd18adb0 0x5594bd1a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f464ae3fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd190d30 0x5594bd1b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f46341ad9a0 con 0x5594bd00be40 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46341aef60 con 0x5594bd18adb0 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f464ae3fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f46341ae0d0 con 0x5594bd190d30 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f464ae3fb80 10 client.0 _ll_register_callbacks cb 0x5594bd052c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f4638e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd1a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f4638e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd1a54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43518/0 (socket says 172.21.15.43:43518) 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f4638e00700 1 -- 172.21.15.43:0/1280893818 learned_addr learned my addr 172.21.15.43:0/1280893818 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.904+0000 7f4639800700 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd18adb0 0x5594bd1a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.905+0000 7f4638e00700 1 -- 172.21.15.43:0/1280893818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd190d30 msgr2=0x5594bd1b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.905+0000 7f4638e00700 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd190d30 0x5594bd1b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.905+0000 7f4638e00700 1 -- 172.21.15.43:0/1280893818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd18adb0 msgr2=0x5594bd1a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:48.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.905+0000 7f4638e00700 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd18adb0 0x5594bd1a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.905+0000 7f4638e00700 1 -- 172.21.15.43:0/1280893818 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594bd1b4350 con 0x5594bd00be40 2024-05-27T07:24:48.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.905+0000 7f4638e00700 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd1a54e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f462000bc30 tx=0x7f462000bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7995021c655fd78 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:48.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.905+0000 7f461b400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:48.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.905+0000 7f461b400700 1 -- 172.21.15.43:0/1280893818 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462000b620 con 0x5594bd00be40 2024-05-27T07:24:48.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.905+0000 7f461b400700 1 -- 172.21.15.43:0/1280893818 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f462000b7c0 con 0x5594bd00be40 2024-05-27T07:24:48.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.905+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1280893818 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4620004db0 con 0x5594bd00be40 2024-05-27T07:24:48.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f461b400700 1 -- 172.21.15.43:0/1280893818 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f462000b9c0 con 0x5594bd00be40 2024-05-27T07:24:48.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f464ae3fb80 10 client.17055 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:48.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1280893818 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x5594bd217070 con 0x5594bd00be40 2024-05-27T07:24:48.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f461b400700 1 -- 172.21.15.43:0/1280893818 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4620004db0 con 0x5594bd00be40 2024-05-27T07:24:48.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f461b400700 1 -- 172.21.15.43:0/1280893818 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7f4620035040 con 0x5594bd00be40 2024-05-27T07:24:48.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f464ae3fb80 10 client.17055 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:48.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f464ae3fb80 10 client.17055 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:24:48.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f464ae3fb80 20 client.17055 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:48.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5594bd2174c0 0x5594bd2198e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:48.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f464ae3fb80 4 client.17055 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:24:48.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.906+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1280893818 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5594bd0b4570 con 0x5594bd2174c0 2024-05-27T07:24:48.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.907+0000 7f4639800700 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5594bd2174c0 0x5594bd2198e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:48.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.908+0000 7f4639800700 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5594bd2174c0 0x5594bd2198e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee904ba9cc71a3fc server_cookie=baa7a5e63e5db4c8 in_seq=0 out_seq=0 2024-05-27T07:24:48.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.908+0000 7f461b400700 10 client.17055 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:24:48.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f461b400700 1 -- 172.21.15.43:0/1280893818 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5594bd0b4570 con 0x5594bd2174c0 2024-05-27T07:24:48.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f461b400700 10 client.17055 handle_command_reply: tid=0 2024-05-27T07:24:48.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f464ae3fb80 1 client.17055 shutdown 2024-05-27T07:24:48.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1280893818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 msgr2=0x5594bd1a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd1a54e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f462000bc30 tx=0x7f462000bde0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1280893818 shutdown_connections 2024-05-27T07:24:48.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5594bd2174c0 0x5594bd2198e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5594bd190d30 0x5594bd1b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5594bd00be40 0x5594bd1a54e0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f464ae3fb80 1 --2- 172.21.15.43:0/1280893818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594bd18adb0 0x5594bd1a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f461b400700 0 client.17055 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:24:48.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.909+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1280893818 >> 172.21.15.43:0/1280893818 conn(0x5594bd06bfc0 msgr2=0x5594bd18fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:48.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.910+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1280893818 shutdown_connections 2024-05-27T07:24:48.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.910+0000 7f464ae3fb80 1 -- 172.21.15.43:0/1280893818 wait complete. 2024-05-27T07:24:48.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.910+0000 7f464ae3fb80 20 client.17055 trim_cache size 0 max 16384 2024-05-27T07:24:48.902 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:24:48.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 -- 172.21.15.43:0/3402315606 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f461c0b2a70 msgr2=0x7f461c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f461c0b2a70 0x7f461c0b4f80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f462c004800 tx=0x7f462c004750 comp rx=0 tx=0).stop 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 -- 172.21.15.43:0/3402315606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46341a18d0 msgr2=0x7f46341a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46341a18d0 0x7f46341a1ce0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f462c00d980 tx=0x7f462c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 -- 172.21.15.43:0/3402315606 shutdown_connections 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f461c0b2a70 0x7f461c0b4f80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46341a18d0 0x7f46341a1ce0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46341045e0 0x7f463419d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 --2- 172.21.15.43:0/3402315606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4634109860 0x7f463419d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 -- 172.21.15.43:0/3402315606 >> 172.21.15.43:0/3402315606 conn(0x7f4634100030 msgr2=0x7f463410a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 -- 172.21.15.43:0/3402315606 shutdown_connections 2024-05-27T07:24:48.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:48.911+0000 7f463c200700 1 -- 172.21.15.43:0/3402315606 wait complete. 2024-05-27T07:24:49.231 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60547 0.60 MB/sec execute 413 sec latency 4106.593 ms 2024-05-27T07:24:49.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:49 smithi132 ceph-mon[35283]: pgmap v368: 129 pgs: 129 active+clean; 110 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 225 KiB/s wr, 22 op/s 2024-05-27T07:24:49.683 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:49 smithi043 ceph-mon[31531]: pgmap v368: 129 pgs: 129 active+clean; 110 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 225 KiB/s wr, 22 op/s 2024-05-27T07:24:49.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:49 smithi179 ceph-mon[35888]: pgmap v368: 129 pgs: 129 active+clean; 110 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 225 KiB/s wr, 22 op/s 2024-05-27T07:24:49.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:24:50.230 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60606 0.60 MB/sec execute 414 sec latency 52.501 ms 2024-05-27T07:24:50.238 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:51.231 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60678 0.60 MB/sec execute 415 sec latency 52.598 ms 2024-05-27T07:24:51.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:51 smithi043 ceph-mon[31531]: pgmap v369: 129 pgs: 129 active+clean; 110 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 225 KiB/s wr, 22 op/s 2024-05-27T07:24:51.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:51 smithi132 ceph-mon[35283]: pgmap v369: 129 pgs: 129 active+clean; 110 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 225 KiB/s wr, 22 op/s 2024-05-27T07:24:51.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:51 smithi179 ceph-mon[35888]: pgmap v369: 129 pgs: 129 active+clean; 110 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 225 KiB/s wr, 22 op/s 2024-05-27T07:24:52.231 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60733 0.60 MB/sec execute 416 sec latency 50.356 ms 2024-05-27T07:24:52.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.531+0000 7f50f5000700 1 -- 172.21.15.43:0/1618222741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50f0108730 msgr2=0x7f50f0110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:52.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.531+0000 7f50f5000700 1 --2- 172.21.15.43:0/1618222741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50f0108730 0x7f50f0110090 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f50e800b0d0 tx=0x7f50e800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.531+0000 7f50f5000700 1 -- 172.21.15.43:0/1618222741 shutdown_connections 2024-05-27T07:24:52.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.531+0000 7f50f5000700 1 --2- 172.21.15.43:0/1618222741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50f0108730 0x7f50f0110090 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.531+0000 7f50f5000700 1 --2- 172.21.15.43:0/1618222741 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50f01034b0 0x7f50f01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.531+0000 7f50f5000700 1 --2- 172.21.15.43:0/1618222741 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50f0107de0 0x7f50f01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.531+0000 7f50f5000700 1 -- 172.21.15.43:0/1618222741 >> 172.21.15.43:0/1618222741 conn(0x7f50f00fedb0 msgr2=0x7f50f01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:52.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.532+0000 7f50f5000700 1 -- 172.21.15.43:0/1618222741 shutdown_connections 2024-05-27T07:24:52.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.532+0000 7f50f5000700 1 -- 172.21.15.43:0/1618222741 wait complete. 2024-05-27T07:24:52.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.533+0000 7f50f5000700 1 Processor -- start 2024-05-27T07:24:52.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.533+0000 7f50f5000700 1 -- start start 2024-05-27T07:24:52.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.534+0000 7f50f5000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50f01034b0 0x7f50f019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.534+0000 7f50f5000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50f0107de0 0x7f50f019d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.534+0000 7f50f5000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50f0108730 0x7f50f01a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.534+0000 7f50f5000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f50f0114400 con 0x7f50f0108730 2024-05-27T07:24:52.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.534+0000 7f50f5000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f50f0114280 con 0x7f50f01034b0 2024-05-27T07:24:52.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.534+0000 7f50f5000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f50f0114580 con 0x7f50f0107de0 2024-05-27T07:24:52.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.535+0000 7f50ee800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50f0108730 0x7f50f01a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.535+0000 7f50ee800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50f0108730 0x7f50f01a15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43538/0 (socket says 172.21.15.43:43538) 2024-05-27T07:24:52.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.535+0000 7f50ee800700 1 -- 172.21.15.43:0/660131226 learned_addr learned my addr 172.21.15.43:0/660131226 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.535+0000 7f50ed400700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50f0107de0 0x7f50f019d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.535+0000 7f50ee800700 1 -- 172.21.15.43:0/660131226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50f0107de0 msgr2=0x7f50f019d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.535+0000 7f50ee800700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50f0107de0 0x7f50f019d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.535+0000 7f50ee800700 1 -- 172.21.15.43:0/660131226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50f01034b0 msgr2=0x7f50f019cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.535+0000 7f50ee800700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50f01034b0 0x7f50f019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.535+0000 7f50ee800700 1 -- 172.21.15.43:0/660131226 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f01a1c60 con 0x7f50f0108730 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.536+0000 7f50ee800700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50f0108730 0x7f50f01a15c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f50e8013630 tx=0x7f50e8013710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.536+0000 7f50e6a00700 1 -- 172.21.15.43:0/660131226 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e8021020 con 0x7f50f0108730 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.536+0000 7f50e6a00700 1 -- 172.21.15.43:0/660131226 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f50e8009480 con 0x7f50f0108730 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.536+0000 7f50f5000700 1 -- 172.21.15.43:0/660131226 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50f01a1ef0 con 0x7f50f0108730 2024-05-27T07:24:52.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.536+0000 7f50e6a00700 1 -- 172.21.15.43:0/660131226 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e8005680 con 0x7f50f0108730 2024-05-27T07:24:52.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.536+0000 7f50f5000700 1 -- 172.21.15.43:0/660131226 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50f01a9870 con 0x7f50f0108730 2024-05-27T07:24:52.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.542+0000 7f50e6a00700 1 -- 172.21.15.43:0/660131226 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f50e8024020 con 0x7f50f0108730 2024-05-27T07:24:52.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.543+0000 7f5103a6bb80 1 Processor -- start 2024-05-27T07:24:52.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.544+0000 7f5103a6bb80 1 -- start start 2024-05-27T07:24:52.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.544+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b6c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.544+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef229ede40 0x55ef22b6c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.544+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 0x55ef22b79110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.544+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ef22a9ae40 con 0x55ef22ad99f0 2024-05-27T07:24:52.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.544+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ef22a8a9e0 con 0x55ef22b6ce70 2024-05-27T07:24:52.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.544+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ef22a8ea20 con 0x55ef229ede40 2024-05-27T07:24:52.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.545+0000 7f50ed400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b6c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.545+0000 7f50ed400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b6c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43540/0 (socket says 172.21.15.43:43540) 2024-05-27T07:24:52.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.545+0000 7f50ed400700 1 -- 172.21.15.43:0/2180406041 learned_addr learned my addr 172.21.15.43:0/2180406041 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:52.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.545+0000 7f50e6a00700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50d40b2a70 0x7f50d40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.545+0000 7f50ede00700 1 --2- 172.21.15.43:0/2180406041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef229ede40 0x55ef22b6c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.545+0000 7f50ed400700 1 --2- 172.21.15.43:0/2180406041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 0x55ef22b79110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.545+0000 7f50ee800700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50d40b2a70 0x7f50d40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.546+0000 7f50ed400700 1 -- 172.21.15.43:0/2180406041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef229ede40 msgr2=0x55ef22b6c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:52.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.546+0000 7f50ed400700 1 --2- 172.21.15.43:0/2180406041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef229ede40 0x55ef22b6c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.546+0000 7f50ed400700 1 -- 172.21.15.43:0/2180406041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 msgr2=0x55ef22b79110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:52.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.546+0000 7f50ed400700 1 --2- 172.21.15.43:0/2180406041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 0x55ef22b79110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.546+0000 7f50ed400700 1 -- 172.21.15.43:0/2180406041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef22a53420 con 0x55ef22ad99f0 2024-05-27T07:24:52.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.546+0000 7f50e6a00700 1 -- 172.21.15.43:0/660131226 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f50e800f020 con 0x7f50f0108730 2024-05-27T07:24:52.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.546+0000 7f50ed400700 1 --2- 172.21.15.43:0/2180406041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b6c330 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0135f0 tx=0x7f50dc00b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8351206e790e3daf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:52.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.546+0000 7f50db400700 1 -- 172.21.15.43:0/2180406041 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc019030 con 0x55ef22ad99f0 2024-05-27T07:24:52.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.546+0000 7f50db400700 1 -- 172.21.15.43:0/2180406041 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f50dc013dd0 con 0x55ef22ad99f0 2024-05-27T07:24:52.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.546+0000 7f50db400700 1 -- 172.21.15.43:0/2180406041 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc005580 con 0x55ef22ad99f0 2024-05-27T07:24:52.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.547+0000 7f5103a6bb80 1 -- 172.21.15.43:0/2180406041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 msgr2=0x55ef22b6c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:52.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.547+0000 7f50ee800700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50d40b2a70 0x7f50d40b4f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f50e80e11f0 tx=0x7f50e809f5e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:52.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.547+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/2180406041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b6c330 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0135f0 tx=0x7f50dc00b6d0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.548+0000 7f5103a6bb80 1 -- 172.21.15.43:0/2180406041 shutdown_connections 2024-05-27T07:24:52.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.548+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/2180406041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 0x55ef22b79110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.548+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/2180406041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b6c330 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.548+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/2180406041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef229ede40 0x55ef22b6c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.548+0000 7f5103a6bb80 1 -- 172.21.15.43:0/2180406041 >> 172.21.15.43:0/2180406041 conn(0x55ef22a87390 msgr2=0x55ef22b684c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:52.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.548+0000 7f5103a6bb80 1 -- 172.21.15.43:0/2180406041 shutdown_connections 2024-05-27T07:24:52.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.548+0000 7f5103a6bb80 1 -- 172.21.15.43:0/2180406041 wait complete. 2024-05-27T07:24:52.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.548+0000 7f5103a6bb80 1 -- start start 2024-05-27T07:24:52.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.549+0000 7f5103a6bb80 1 Processor -- start 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.549+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef229ede40 0x55ef22b76690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.549+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b76bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.549+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 0x55ef22b77110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.549+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ef22990fa0 con 0x55ef22ad99f0 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.549+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ef22a2b760 con 0x55ef22b6ce70 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.549+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ef22b79b80 con 0x55ef229ede40 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.549+0000 7f5103a6bb80 10 client.0 _ll_register_callbacks cb 0x55ef22a34c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.550+0000 7f50ede00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b76bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.550+0000 7f50ede00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b76bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43552/0 (socket says 172.21.15.43:43552) 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.550+0000 7f50ede00700 1 -- 172.21.15.43:0/3895843041 learned_addr learned my addr 172.21.15.43:0/3895843041 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.550+0000 7f50ed400700 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef229ede40 0x55ef22b76690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.550+0000 7f50ed400700 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 0x55ef22b77110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.550+0000 7f50ede00700 1 -- 172.21.15.43:0/3895843041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef229ede40 msgr2=0x55ef22b76690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.550+0000 7f50ede00700 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef229ede40 0x55ef22b76690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.550+0000 7f50ede00700 1 -- 172.21.15.43:0/3895843041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 msgr2=0x55ef22b77110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.550+0000 7f50ede00700 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 0x55ef22b77110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.550+0000 7f50ede00700 1 -- 172.21.15.43:0/3895843041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef229da8d0 con 0x55ef22ad99f0 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.551+0000 7f50ed400700 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 0x55ef22b77110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.551+0000 7f50ede00700 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b76bd0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f50e000e480 tx=0x7f50e000e790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211c00e3e68d5185 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.551+0000 7f50cfe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.551+0000 7f50cfe00700 1 -- 172.21.15.43:0/3895843041 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e0017070 con 0x55ef22ad99f0 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.551+0000 7f50cfe00700 1 -- 172.21.15.43:0/3895843041 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f50e0013470 con 0x55ef22ad99f0 2024-05-27T07:24:52.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.551+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3895843041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50dc013dd0 con 0x55ef22ad99f0 2024-05-27T07:24:52.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.552+0000 7f50cfe00700 1 -- 172.21.15.43:0/3895843041 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f50e0023050 con 0x55ef22ad99f0 2024-05-27T07:24:52.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.552+0000 7f5103a6bb80 10 client.17067 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:52.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.552+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3895843041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55ef22a5b060 con 0x55ef22ad99f0 2024-05-27T07:24:52.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.552+0000 7f50cfe00700 1 -- 172.21.15.43:0/3895843041 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f50dc013dd0 con 0x55ef22ad99f0 2024-05-27T07:24:52.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.552+0000 7f50cfe00700 1 -- 172.21.15.43:0/3895843041 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7f50e001d770 con 0x55ef22ad99f0 2024-05-27T07:24:52.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.552+0000 7f5103a6bb80 10 client.17067 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:52.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.552+0000 7f5103a6bb80 10 client.17067 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:24:52.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.553+0000 7f5103a6bb80 20 client.17067 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:52.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.553+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55ef22b87040 0x55ef22b89490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.553+0000 7f5103a6bb80 4 client.17067 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:52.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.553+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3895843041 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef22a9cd90 con 0x55ef22b87040 2024-05-27T07:24:52.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.553+0000 7f50ed400700 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55ef22b87040 0x55ef22b89490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.554+0000 7f50cfe00700 10 client.17067 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:24:52.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.554+0000 7f50ed400700 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55ef22b87040 0x55ef22b89490 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c46f3463aff7693e server_cookie=4cca53523b206111 in_seq=0 out_seq=0 2024-05-27T07:24:52.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.557+0000 7f50cfe00700 1 -- 172.21.15.43:0/3895843041 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ef22a9cd90 con 0x55ef22b87040 2024-05-27T07:24:52.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.557+0000 7f50cfe00700 10 client.17067 handle_command_reply: tid=0 2024-05-27T07:24:52.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 client.17067 shutdown 2024-05-27T07:24:52.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3895843041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 msgr2=0x55ef22b76bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:52.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b76bd0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f50e000e480 tx=0x7f50e000e790 comp rx=0 tx=0).stop 2024-05-27T07:24:52.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3895843041 shutdown_connections 2024-05-27T07:24:52.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55ef22b87040 0x55ef22b89490 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef22b6ce70 0x55ef22b77110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef229ede40 0x55ef22b76690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3895843041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b76bd0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3895843041 >> 172.21.15.43:0/3895843041 conn(0x55ef22a87390 msgr2=0x55ef22b6c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:52.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f50cfe00700 0 client.17067 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:24:52.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3895843041 shutdown_connections 2024-05-27T07:24:52.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.558+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3895843041 wait complete. 2024-05-27T07:24:52.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.559+0000 7f5103a6bb80 20 client.17067 trim_cache size 0 max 16384 2024-05-27T07:24:52.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.579+0000 7f5103a6bb80 1 Processor -- start 2024-05-27T07:24:52.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.579+0000 7f5103a6bb80 1 -- start start 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.579+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 0x55ef22b71e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.579+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef22ad99f0 0x55ef22b68590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.579+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22b6ce70 0x55ef22b82c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.579+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ef22b756b0 con 0x55ef22b6ce70 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.579+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f50f01b3190 con 0x55ef229ede40 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.579+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f50f01b0e30 con 0x55ef22ad99f0 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.580+0000 7f50ed400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22b6ce70 0x55ef22b82c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.580+0000 7f50ed400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22b6ce70 0x55ef22b82c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43558/0 (socket says 172.21.15.43:43558) 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.580+0000 7f50ed400700 1 -- 172.21.15.43:0/415885451 learned_addr learned my addr 172.21.15.43:0/415885451 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.580+0000 7f50ede00700 1 --2- 172.21.15.43:0/415885451 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef22ad99f0 0x55ef22b68590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.580+0000 7f50ed400700 1 --2- 172.21.15.43:0/415885451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 0x55ef22b71e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.580+0000 7f50ed400700 1 -- 172.21.15.43:0/415885451 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef22ad99f0 msgr2=0x55ef22b68590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:52.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.580+0000 7f50ed400700 1 --2- 172.21.15.43:0/415885451 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef22ad99f0 0x55ef22b68590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.580+0000 7f50ed400700 1 -- 172.21.15.43:0/415885451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 msgr2=0x55ef22b71e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:52.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.580+0000 7f50ed400700 1 --2- 172.21.15.43:0/415885451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 0x55ef22b71e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.580+0000 7f50ed400700 1 -- 172.21.15.43:0/415885451 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef22b74c00 con 0x55ef22b6ce70 2024-05-27T07:24:52.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.581+0000 7f50ed400700 1 --2- 172.21.15.43:0/415885451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22b6ce70 0x55ef22b82c80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f50dc005950 tx=0x7f50dc00b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf7c70a6a7dbe27 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:52.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.581+0000 7f50daa00700 1 -- 172.21.15.43:0/415885451 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc031070 con 0x55ef22b6ce70 2024-05-27T07:24:52.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.581+0000 7f50daa00700 1 -- 172.21.15.43:0/415885451 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f50dc024070 con 0x55ef22b6ce70 2024-05-27T07:24:52.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.581+0000 7f50daa00700 1 -- 172.21.15.43:0/415885451 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc02c530 con 0x55ef22b6ce70 2024-05-27T07:24:52.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.582+0000 7f5103a6bb80 1 -- 172.21.15.43:0/415885451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22b6ce70 msgr2=0x55ef22b82c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:52.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.582+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/415885451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22b6ce70 0x55ef22b82c80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f50dc005950 tx=0x7f50dc00b6d0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.582+0000 7f5103a6bb80 1 -- 172.21.15.43:0/415885451 shutdown_connections 2024-05-27T07:24:52.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.582+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/415885451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22b6ce70 0x55ef22b82c80 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.582+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/415885451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 0x55ef22b71e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.582+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/415885451 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef22ad99f0 0x55ef22b68590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.582+0000 7f5103a6bb80 1 -- 172.21.15.43:0/415885451 >> 172.21.15.43:0/415885451 conn(0x55ef22a87390 msgr2=0x55ef22b67e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:52.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.582+0000 7f5103a6bb80 1 -- 172.21.15.43:0/415885451 shutdown_connections 2024-05-27T07:24:52.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.582+0000 7f5103a6bb80 1 -- 172.21.15.43:0/415885451 wait complete. 2024-05-27T07:24:52.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.583+0000 7f5103a6bb80 1 -- start start 2024-05-27T07:24:52.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.583+0000 7f5103a6bb80 1 Processor -- start 2024-05-27T07:24:52.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.584+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 0x55ef22b71c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.584+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b70050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.584+0000 7f5103a6bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef22b6ce70 0x55ef22b70590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.584+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f50f01b1500 con 0x55ef22ad99f0 2024-05-27T07:24:52.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.584+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f50f01b2ac0 con 0x55ef229ede40 2024-05-27T07:24:52.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.584+0000 7f5103a6bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f50f01b1c30 con 0x55ef22b6ce70 2024-05-27T07:24:52.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.584+0000 7f5103a6bb80 10 client.0 _ll_register_callbacks cb 0x55ef22a34c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:52.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.584+0000 7f50ed400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 0x55ef22b71c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.584+0000 7f50ed400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 0x55ef22b71c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:36930/0 (socket says 172.21.15.43:36930) 2024-05-27T07:24:52.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.584+0000 7f50ed400700 1 -- 172.21.15.43:0/3099841935 learned_addr learned my addr 172.21.15.43:0/3099841935 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:52.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.585+0000 7f50ede00700 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b70050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.585+0000 7f50ed400700 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef22b6ce70 0x55ef22b70590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.585+0000 7f50ede00700 1 -- 172.21.15.43:0/3099841935 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef22b6ce70 msgr2=0x55ef22b70590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.585+0000 7f50ede00700 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef22b6ce70 0x55ef22b70590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.585+0000 7f50ede00700 1 -- 172.21.15.43:0/3099841935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 msgr2=0x55ef22b71c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.585+0000 7f50ede00700 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 0x55ef22b71c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.585+0000 7f50ede00700 1 -- 172.21.15.43:0/3099841935 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef22b70c00 con 0x55ef22ad99f0 2024-05-27T07:24:52.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.586+0000 7f50ede00700 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b70050 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f50e001ab50 tx=0x7f50e001ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e2d9d2d50c94f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:52.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.586+0000 7f50cfe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:52.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.586+0000 7f50cfe00700 1 -- 172.21.15.43:0/3099841935 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e0017040 con 0x55ef22ad99f0 2024-05-27T07:24:52.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.586+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3099841935 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50dc024070 con 0x55ef22ad99f0 2024-05-27T07:24:52.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.586+0000 7f50cfe00700 1 -- 172.21.15.43:0/3099841935 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7f50e001a360 con 0x55ef22ad99f0 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.587+0000 7f50cfe00700 1 -- 172.21.15.43:0/3099841935 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7f50e001a580 con 0x55ef22ad99f0 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.587+0000 7f5103a6bb80 10 client.17073 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.587+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3099841935 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55ef22b7ec70 con 0x55ef22ad99f0 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.587+0000 7f50cfe00700 1 -- 172.21.15.43:0/3099841935 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f50dc024070 con 0x55ef22ad99f0 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.587+0000 7f50cfe00700 1 -- 172.21.15.43:0/3099841935 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7f50e001d4d0 con 0x55ef22ad99f0 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.588+0000 7f5103a6bb80 10 client.17073 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.588+0000 7f5103a6bb80 10 client.17073 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.588+0000 7f5103a6bb80 20 client.17073 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.588+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55ef22b7f3c0 0x55ef22bf3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.588+0000 7f5103a6bb80 4 client.17073 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:24:52.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.588+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3099841935 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef22b67120 con 0x55ef22b7f3c0 2024-05-27T07:24:52.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.588+0000 7f50ed400700 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55ef22b7f3c0 0x55ef22bf3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:52.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f50ed400700 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55ef22b7f3c0 0x55ef22bf3b30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5db333dac9577504 server_cookie=528f01a60b2fc744 in_seq=0 out_seq=0 2024-05-27T07:24:52.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f50cfe00700 10 client.17073 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:24:52.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f50cfe00700 1 -- 172.21.15.43:0/3099841935 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef22b67120 con 0x55ef22b7f3c0 2024-05-27T07:24:52.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f50cfe00700 10 client.17073 handle_command_reply: tid=0 2024-05-27T07:24:52.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f5103a6bb80 1 client.17073 shutdown 2024-05-27T07:24:52.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3099841935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 msgr2=0x55ef22b70050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:52.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b70050 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f50e001ab50 tx=0x7f50e001ae70 comp rx=0 tx=0).stop 2024-05-27T07:24:52.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3099841935 shutdown_connections 2024-05-27T07:24:52.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55ef22b7f3c0 0x55ef22bf3b30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ef22b6ce70 0x55ef22b70590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.590+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ef229ede40 0x55ef22b71c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.591+0000 7f5103a6bb80 1 --2- 172.21.15.43:0/3099841935 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ef22ad99f0 0x55ef22b70050 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.591+0000 7f50cfe00700 0 client.17073 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:24:52.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.591+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3099841935 >> 172.21.15.43:0/3099841935 conn(0x55ef22a87390 msgr2=0x55ef22b6ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:52.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.591+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3099841935 shutdown_connections 2024-05-27T07:24:52.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.591+0000 7f5103a6bb80 1 -- 172.21.15.43:0/3099841935 wait complete. 2024-05-27T07:24:52.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.591+0000 7f5103a6bb80 20 client.17073 trim_cache size 0 max 16384 2024-05-27T07:24:52.583 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 -- 172.21.15.43:0/660131226 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50d40b2a70 msgr2=0x7f50d40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50d40b2a70 0x7f50d40b4f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f50e80e11f0 tx=0x7f50e809f5e0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 -- 172.21.15.43:0/660131226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50f0108730 msgr2=0x7f50f01a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50f0108730 0x7f50f01a15c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f50e8013630 tx=0x7f50e8013710 comp rx=0 tx=0).stop 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 -- 172.21.15.43:0/660131226 shutdown_connections 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f50d40b2a70 0x7f50d40b4f80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50f0108730 0x7f50f01a15c0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f50f01034b0 0x7f50f019cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 --2- 172.21.15.43:0/660131226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50f0107de0 0x7f50f019d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 -- 172.21.15.43:0/660131226 >> 172.21.15.43:0/660131226 conn(0x7f50f00fedb0 msgr2=0x7f50f010fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 -- 172.21.15.43:0/660131226 shutdown_connections 2024-05-27T07:24:52.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:52.592+0000 7f50f5000700 1 -- 172.21.15.43:0/660131226 wait complete. 2024-05-27T07:24:53.232 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60796 0.60 MB/sec execute 417 sec latency 47.796 ms 2024-05-27T07:24:53.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:24:53.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:53 smithi132 ceph-mon[35283]: pgmap v370: 129 pgs: 129 active+clean; 111 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 371 KiB/s wr, 31 op/s 2024-05-27T07:24:53.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:53 smithi043 ceph-mon[31531]: pgmap v370: 129 pgs: 129 active+clean; 111 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 371 KiB/s wr, 31 op/s 2024-05-27T07:24:53.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:53 smithi179 ceph-mon[35888]: pgmap v370: 129 pgs: 129 active+clean; 111 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 371 KiB/s wr, 31 op/s 2024-05-27T07:24:53.913 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:54.231 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60886 0.59 MB/sec execute 418 sec latency 55.328 ms 2024-05-27T07:24:55.231 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60955 0.59 MB/sec execute 419 sec latency 49.229 ms 2024-05-27T07:24:55.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:55 smithi132 ceph-mon[35283]: pgmap v371: 129 pgs: 129 active+clean; 111 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 206 KiB/s wr, 17 op/s 2024-05-27T07:24:55.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:55 smithi043 ceph-mon[31531]: pgmap v371: 129 pgs: 129 active+clean; 111 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 206 KiB/s wr, 17 op/s 2024-05-27T07:24:55.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:55 smithi179 ceph-mon[35888]: pgmap v371: 129 pgs: 129 active+clean; 111 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 206 KiB/s wr, 17 op/s 2024-05-27T07:24:56.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.219+0000 7fd0a6400700 1 -- 172.21.15.43:0/2307649195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a010e8e0 msgr2=0x7fd0a0110dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:56.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.219+0000 7fd0a6400700 1 --2- 172.21.15.43:0/2307649195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a010e8e0 0x7fd0a0110dc0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fd09800b0a0 tx=0x7fd09800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.220+0000 7fd0a6400700 1 -- 172.21.15.43:0/2307649195 shutdown_connections 2024-05-27T07:24:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.220+0000 7fd0a6400700 1 --2- 172.21.15.43:0/2307649195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a010e8e0 0x7fd0a0110dc0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.220+0000 7fd0a6400700 1 --2- 172.21.15.43:0/2307649195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0a01054b0 0x7fd0a01058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.220+0000 7fd0a6400700 1 --2- 172.21.15.43:0/2307649195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0a0105e00 0x7fd0a010e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.220+0000 7fd0a6400700 1 -- 172.21.15.43:0/2307649195 >> 172.21.15.43:0/2307649195 conn(0x7fd0a00fedb0 msgr2=0x7fd0a01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.220+0000 7fd0a6400700 1 -- 172.21.15.43:0/2307649195 shutdown_connections 2024-05-27T07:24:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.220+0000 7fd0a6400700 1 -- 172.21.15.43:0/2307649195 wait complete. 2024-05-27T07:24:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.221+0000 7fd0a6400700 1 Processor -- start 2024-05-27T07:24:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.222+0000 7fd0a6400700 1 -- start start 2024-05-27T07:24:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.222+0000 7fd0a6400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a01054b0 0x7fd0a019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.222+0000 7fd0a6400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0a0105e00 0x7fd0a019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.222+0000 7fd0a6400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0a010e8e0 0x7fd0a01a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.222+0000 7fd0a6400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a0114310 con 0x7fd0a01054b0 2024-05-27T07:24:56.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.222+0000 7fd0a6400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a0114190 con 0x7fd0a0105e00 2024-05-27T07:24:56.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.222+0000 7fd0a6400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a0114490 con 0x7fd0a010e8e0 2024-05-27T07:24:56.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.222+0000 7fd09f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a01054b0 0x7fd0a019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.222+0000 7fd09f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a01054b0 0x7fd0a019cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52244/0 (socket says 172.21.15.43:52244) 2024-05-27T07:24:56.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.223+0000 7fd09f400700 1 -- 172.21.15.43:0/358638847 learned_addr learned my addr 172.21.15.43:0/358638847 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.223+0000 7fd09ea00700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0a0105e00 0x7fd0a019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.223+0000 7fd09fe00700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0a010e8e0 0x7fd0a01a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.223+0000 7fd09f400700 1 -- 172.21.15.43:0/358638847 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0a010e8e0 msgr2=0x7fd0a01a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.223+0000 7fd09f400700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0a010e8e0 0x7fd0a01a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.223+0000 7fd09f400700 1 -- 172.21.15.43:0/358638847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0a0105e00 msgr2=0x7fd0a019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.223+0000 7fd09f400700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0a0105e00 0x7fd0a019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.223+0000 7fd09f400700 1 -- 172.21.15.43:0/358638847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a01a1d90 con 0x7fd0a01054b0 2024-05-27T07:24:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.223+0000 7fd09ea00700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0a0105e00 0x7fd0a019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.223+0000 7fd09f400700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a01054b0 0x7fd0a019cec0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fd09000d980 tx=0x7fd09000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.224+0000 7fd097e00700 1 -- 172.21.15.43:0/358638847 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd090017070 con 0x7fd0a01054b0 2024-05-27T07:24:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.224+0000 7fd097e00700 1 -- 172.21.15.43:0/358638847 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd090004830 con 0x7fd0a01054b0 2024-05-27T07:24:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.224+0000 7fd0a6400700 1 -- 172.21.15.43:0/358638847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a01a2080 con 0x7fd0a01054b0 2024-05-27T07:24:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.224+0000 7fd097e00700 1 -- 172.21.15.43:0/358638847 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900053a0 con 0x7fd0a01054b0 2024-05-27T07:24:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.224+0000 7fd0a6400700 1 -- 172.21.15.43:0/358638847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a01a99a0 con 0x7fd0a01054b0 2024-05-27T07:24:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.228+0000 7fd097e00700 1 -- 172.21.15.43:0/358638847 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd0900049e0 con 0x7fd0a01054b0 2024-05-27T07:24:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.231+0000 7fd0b4e5bb80 1 Processor -- start 2024-05-27T07:24:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.231+0000 7fd097e00700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd0840b29f0 0x7fd0840b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.231+0000 7fd0b4e5bb80 1 -- start start 2024-05-27T07:24:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.231+0000 7fd097e00700 1 -- 172.21.15.43:0/358638847 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7fd0900e29f0 con 0x7fd0a01054b0 2024-05-27T07:24:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.231+0000 7fd09fe00700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd0840b29f0 0x7fd0840b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.231+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3eff399f0 0x55c3effcc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.231+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3efe4de40 0x55c3effcc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.232+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3effcce70 0x55c3effd9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.232+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c3efefae40 con 0x55c3effcce70 2024-05-27T07:24:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.232+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c3efeea9e0 con 0x55c3eff399f0 2024-05-27T07:24:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.232+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c3efeeea20 con 0x55c3efe4de40 2024-05-27T07:24:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.232+0000 7fd09fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3effcce70 0x55c3effd9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.232+0000 7fd09fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3effcce70 0x55c3effd9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52248/0 (socket says 172.21.15.43:52248) 2024-05-27T07:24:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.232+0000 7fd09fe00700 1 -- 172.21.15.43:0/554731867 learned_addr learned my addr 172.21.15.43:0/554731867 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.232+0000 7fd09ea00700 1 --2- 172.21.15.43:0/554731867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3efe4de40 0x55c3effcc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.232+0000 7fd09f400700 1 --2- 172.21.15.43:0/554731867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3eff399f0 0x55c3effcc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.233+0000 7fd09fe00700 1 -- 172.21.15.43:0/554731867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3efe4de40 msgr2=0x55c3effcc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.233+0000 7fd09fe00700 1 --2- 172.21.15.43:0/554731867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3efe4de40 0x55c3effcc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.233+0000 7fd09fe00700 1 -- 172.21.15.43:0/554731867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3eff399f0 msgr2=0x55c3effcc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.233+0000 7fd09fe00700 1 --2- 172.21.15.43:0/554731867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3eff399f0 0x55c3effcc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.233+0000 7fd09fe00700 1 -- 172.21.15.43:0/554731867 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3efeb3420 con 0x55c3effcce70 2024-05-27T07:24:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.233+0000 7fd09fe00700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd0840b29f0 0x7fd0840b4f00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd0980093a0 tx=0x7fd0980093d0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.233+0000 7fd09fe00700 1 --2- 172.21.15.43:0/554731867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3effcce70 0x55c3effd9110 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd098009e60 tx=0x7fd098028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa2f30115766d19 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.233+0000 7fd094c00700 1 -- 172.21.15.43:0/554731867 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09802f070 con 0x55c3effcce70 2024-05-27T07:24:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.234+0000 7fd094c00700 1 -- 172.21.15.43:0/554731867 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd098007b20 con 0x55c3effcce70 2024-05-27T07:24:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.234+0000 7fd094c00700 1 -- 172.21.15.43:0/554731867 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098005040 con 0x55c3effcce70 2024-05-27T07:24:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.235+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/554731867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3effcce70 msgr2=0x55c3effd9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.235+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/554731867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3effcce70 0x55c3effd9110 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd098009e60 tx=0x7fd098028000 comp rx=0 tx=0).stop 2024-05-27T07:24:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.235+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/554731867 shutdown_connections 2024-05-27T07:24:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.235+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/554731867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3effcce70 0x55c3effd9110 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.235+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/554731867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3eff399f0 0x55c3effcc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.235+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/554731867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3efe4de40 0x55c3effcc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.235+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/554731867 >> 172.21.15.43:0/554731867 conn(0x55c3efee7390 msgr2=0x55c3effc84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.235+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/554731867 shutdown_connections 2024-05-27T07:24:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.235+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/554731867 wait complete. 2024-05-27T07:24:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.236+0000 7fd0b4e5bb80 1 -- start start 2024-05-27T07:24:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.236+0000 7fd0b4e5bb80 1 Processor -- start 2024-05-27T07:24:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 0x55c3effd65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 0x55c3effd7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c3efdf0fa0 con 0x55c3eff399f0 2024-05-27T07:24:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c3efe8b760 con 0x55c3efe4de40 2024-05-27T07:24:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c3effd9b20 con 0x55c3effcce70 2024-05-27T07:24:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd0b4e5bb80 10 client.0 _ll_register_callbacks cb 0x55c3efe94c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd09ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd09ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52262/0 (socket says 172.21.15.43:52262) 2024-05-27T07:24:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd09ea00700 1 -- 172.21.15.43:0/587375726 learned_addr learned my addr 172.21.15.43:0/587375726 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.237+0000 7fd09f400700 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 0x55c3effd65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.238+0000 7fd09fe00700 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 0x55c3effd7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.238+0000 7fd09ea00700 1 -- 172.21.15.43:0/587375726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 msgr2=0x55c3effd7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.238+0000 7fd09ea00700 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 0x55c3effd7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.238+0000 7fd09ea00700 1 -- 172.21.15.43:0/587375726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 msgr2=0x55c3effd65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.238+0000 7fd09ea00700 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 0x55c3effd65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.238+0000 7fd09ea00700 1 -- 172.21.15.43:0/587375726 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3efe3a8d0 con 0x55c3eff399f0 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.238+0000 7fd09fe00700 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 0x55c3effd7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.238+0000 7fd09ea00700 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd6b10 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00d9f0 tx=0x7fd08c00ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fb501b16a34ded server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.238+0000 7fd089600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.239+0000 7fd089600700 1 -- 172.21.15.43:0/587375726 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c01b440 con 0x55c3eff399f0 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.239+0000 7fd089600700 1 -- 172.21.15.43:0/587375726 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7fd08c011440 con 0x55c3eff399f0 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.239+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/587375726 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd098007b20 con 0x55c3eff399f0 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.239+0000 7fd089600700 1 -- 172.21.15.43:0/587375726 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fd08c011640 con 0x55c3eff399f0 2024-05-27T07:24:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.239+0000 7fd0b4e5bb80 10 client.17085 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.239+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/587375726 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55c3efebb060 con 0x55c3eff399f0 2024-05-27T07:24:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.239+0000 7fd089600700 1 -- 172.21.15.43:0/587375726 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd098007b20 con 0x55c3eff399f0 2024-05-27T07:24:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.240+0000 7fd089600700 1 -- 172.21.15.43:0/587375726 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7fd08c019020 con 0x55c3eff399f0 2024-05-27T07:24:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.240+0000 7fd0b4e5bb80 10 client.17085 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.240+0000 7fd0b4e5bb80 10 client.17085 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.240+0000 7fd0b4e5bb80 20 client.17085 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.240+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c3effe6eb0 0x55c3effe9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.240+0000 7fd0b4e5bb80 4 client.17085 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:24:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.240+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/587375726 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3efefcd90 con 0x55c3effe6eb0 2024-05-27T07:24:56.233 INFO:tasks.workunit.client.0.smithi043.stdout: 1 61087 0.59 MB/sec execute 420 sec latency 51.070 ms 2024-05-27T07:24:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.241+0000 7fd09f400700 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c3effe6eb0 0x55c3effe9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.241+0000 7fd09f400700 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c3effe6eb0 0x55c3effe9300 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2404fd78f729ed server_cookie=d39666231fc0d8a4 in_seq=0 out_seq=0 2024-05-27T07:24:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.241+0000 7fd089600700 10 client.17085 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:24:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.243+0000 7fd089600700 1 -- 172.21.15.43:0/587375726 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c3efefcd90 con 0x55c3effe6eb0 2024-05-27T07:24:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.243+0000 7fd089600700 10 client.17085 handle_command_reply: tid=0 2024-05-27T07:24:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.244+0000 7fd0b4e5bb80 1 client.17085 shutdown 2024-05-27T07:24:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.244+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/587375726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 msgr2=0x55c3effd6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:56.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.244+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd6b10 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00d9f0 tx=0x7fd08c00ddb0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.244+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/587375726 shutdown_connections 2024-05-27T07:24:56.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.244+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 0x55c3effd7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.244+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 0x55c3effd65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.244+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c3effe6eb0 0x55c3effe9300 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.244+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/587375726 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd6b10 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.244+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/587375726 >> 172.21.15.43:0/587375726 conn(0x55c3efee7390 msgr2=0x55c3effcc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:56.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.244+0000 7fd089600700 0 client.17085 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:24:56.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.245+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/587375726 shutdown_connections 2024-05-27T07:24:56.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.245+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/587375726 wait complete. 2024-05-27T07:24:56.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.245+0000 7fd0b4e5bb80 20 client.17085 trim_cache size 0 max 16384 2024-05-27T07:24:56.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.265+0000 7fd0b4e5bb80 1 Processor -- start 2024-05-27T07:24:56.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.265+0000 7fd0b4e5bb80 1 -- start start 2024-05-27T07:24:56.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.265+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3efe4de40 0x55c3effd1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.265+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effe2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.265+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3effcce70 0x55c3effc7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.265+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01b3410 con 0x55c3eff399f0 2024-05-27T07:24:56.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.265+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01b10b0 con 0x55c3effcce70 2024-05-27T07:24:56.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.265+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01b2d40 con 0x55c3efe4de40 2024-05-27T07:24:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effe2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effe2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52274/0 (socket says 172.21.15.43:52274) 2024-05-27T07:24:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09ea00700 1 -- 172.21.15.43:0/2968065093 learned_addr learned my addr 172.21.15.43:0/2968065093 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09f400700 1 --2- 172.21.15.43:0/2968065093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3efe4de40 0x55c3effd1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09fe00700 1 --2- 172.21.15.43:0/2968065093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3effcce70 0x55c3effc7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09ea00700 1 -- 172.21.15.43:0/2968065093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3efe4de40 msgr2=0x55c3effd1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09ea00700 1 --2- 172.21.15.43:0/2968065093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3efe4de40 0x55c3effd1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09ea00700 1 -- 172.21.15.43:0/2968065093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3effcce70 msgr2=0x55c3effc7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09ea00700 1 --2- 172.21.15.43:0/2968065093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3effcce70 0x55c3effc7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09ea00700 1 -- 172.21.15.43:0/2968065093 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3effd4b40 con 0x55c3eff399f0 2024-05-27T07:24:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.266+0000 7fd09fe00700 1 --2- 172.21.15.43:0/2968065093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3effcce70 0x55c3effc7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:24:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.267+0000 7fd09ea00700 1 --2- 172.21.15.43:0/2968065093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effe2bc0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00d9c0 tx=0x7fd08c004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da6c3af6859cc7c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.267+0000 7fd08be00700 1 -- 172.21.15.43:0/2968065093 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c02c070 con 0x55c3eff399f0 2024-05-27T07:24:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.267+0000 7fd08be00700 1 -- 172.21.15.43:0/2968065093 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd08c003d80 con 0x55c3eff399f0 2024-05-27T07:24:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.267+0000 7fd08be00700 1 -- 172.21.15.43:0/2968065093 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c019c50 con 0x55c3eff399f0 2024-05-27T07:24:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.268+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/2968065093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 msgr2=0x55c3effe2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.268+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/2968065093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effe2bc0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00d9c0 tx=0x7fd08c004530 comp rx=0 tx=0).stop 2024-05-27T07:24:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.268+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/2968065093 shutdown_connections 2024-05-27T07:24:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.268+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/2968065093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3effcce70 0x55c3effc7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.268+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/2968065093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3efe4de40 0x55c3effd1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.268+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/2968065093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effe2bc0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.268+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/2968065093 >> 172.21.15.43:0/2968065093 conn(0x55c3efee7390 msgr2=0x55c3effd4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.269+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/2968065093 shutdown_connections 2024-05-27T07:24:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.269+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/2968065093 wait complete. 2024-05-27T07:24:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.269+0000 7fd0b4e5bb80 1 -- start start 2024-05-27T07:24:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.270+0000 7fd0b4e5bb80 1 Processor -- start 2024-05-27T07:24:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.270+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 0x55c3effdf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.270+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.270+0000 7fd0b4e5bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 0x55c3effda2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.270+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01b1eb0 con 0x55c3eff399f0 2024-05-27T07:24:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.270+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01b1780 con 0x55c3efe4de40 2024-05-27T07:24:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.270+0000 7fd0b4e5bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01b2610 con 0x55c3effcce70 2024-05-27T07:24:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.270+0000 7fd0b4e5bb80 10 client.0 _ll_register_callbacks cb 0x55c3efe94c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:24:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52280/0 (socket says 172.21.15.43:52280) 2024-05-27T07:24:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09ea00700 1 -- 172.21.15.43:0/1342779211 learned_addr learned my addr 172.21.15.43:0/1342779211 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:24:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09f400700 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 0x55c3effdf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09fe00700 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 0x55c3effda2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09ea00700 1 -- 172.21.15.43:0/1342779211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 msgr2=0x55c3effda2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09ea00700 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 0x55c3effda2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09ea00700 1 -- 172.21.15.43:0/1342779211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 msgr2=0x55c3effdf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:24:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09ea00700 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 0x55c3effdf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09ea00700 1 -- 172.21.15.43:0/1342779211 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3effda7f0 con 0x55c3eff399f0 2024-05-27T07:24:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.271+0000 7fd09f400700 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 0x55c3effdf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:24:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.272+0000 7fd09ea00700 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd9d70 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00ddf0 tx=0x7fd08c01b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a63de93bbf2d9e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:24:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.272+0000 7fd089600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:24:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.272+0000 7fd089600700 1 -- 172.21.15.43:0/1342779211 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c03f060 con 0x55c3eff399f0 2024-05-27T07:24:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.272+0000 7fd089600700 1 -- 172.21.15.43:0/1342779211 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7fd08c019b70 con 0x55c3eff399f0 2024-05-27T07:24:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.272+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/1342779211 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd08c003d80 con 0x55c3eff399f0 2024-05-27T07:24:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.273+0000 7fd089600700 1 -- 172.21.15.43:0/1342779211 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fd08c02c040 con 0x55c3eff399f0 2024-05-27T07:24:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.273+0000 7fd0b4e5bb80 10 client.17091 fetch_fsmap learned FSMap version 107 2024-05-27T07:24:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.273+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/1342779211 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55c3f0050550 con 0x55c3eff399f0 2024-05-27T07:24:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.273+0000 7fd089600700 1 -- 172.21.15.43:0/1342779211 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd08c003d80 con 0x55c3eff399f0 2024-05-27T07:24:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.273+0000 7fd089600700 1 -- 172.21.15.43:0/1342779211 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7fd08c039030 con 0x55c3eff399f0 2024-05-27T07:24:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.273+0000 7fd0b4e5bb80 10 client.17091 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:24:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.273+0000 7fd0b4e5bb80 10 client.17091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:24:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.273+0000 7fd0b4e5bb80 20 client.17091 populate_metadata read hostname 'smithi043' 2024-05-27T07:24:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.274+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c3f0050b80 0x55c3f0052fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:24:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.274+0000 7fd0b4e5bb80 4 client.17091 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:24:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.274+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/1342779211 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c3efef6570 con 0x55c3f0050b80 2024-05-27T07:24:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.274+0000 7fd09f400700 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c3f0050b80 0x55c3f0052fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:24:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.275+0000 7fd09f400700 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c3f0050b80 0x55c3f0052fa0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7334444c0c106062 server_cookie=d97e345a734922f in_seq=0 out_seq=0 2024-05-27T07:24:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.275+0000 7fd089600700 10 client.17091 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:24:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd089600700 1 -- 172.21.15.43:0/1342779211 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c3efef6570 con 0x55c3f0050b80 2024-05-27T07:24:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd089600700 10 client.17091 handle_command_reply: tid=0 2024-05-27T07:24:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 client.17091 shutdown 2024-05-27T07:24:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/1342779211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 msgr2=0x55c3effd9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd9d70 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00ddf0 tx=0x7fd08c01b720 comp rx=0 tx=0).stop 2024-05-27T07:24:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/1342779211 shutdown_connections 2024-05-27T07:24:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3effcce70 0x55c3effda2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c3efe4de40 0x55c3effdf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c3f0050b80 0x55c3f0052fa0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 --2- 172.21.15.43:0/1342779211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3eff399f0 0x55c3effd9d70 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/1342779211 >> 172.21.15.43:0/1342779211 conn(0x55c3efee7390 msgr2=0x55c3effd4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd089600700 0 client.17091 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:24:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/1342779211 shutdown_connections 2024-05-27T07:24:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.276+0000 7fd0b4e5bb80 1 -- 172.21.15.43:0/1342779211 wait complete. 2024-05-27T07:24:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.277+0000 7fd0b4e5bb80 20 client.17091 trim_cache size 0 max 16384 2024-05-27T07:24:56.268 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:24:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.277+0000 7fd0a6400700 1 -- 172.21.15.43:0/358638847 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd0840b29f0 msgr2=0x7fd0840b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.277+0000 7fd0a6400700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd0840b29f0 0x7fd0840b4f00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd0980093a0 tx=0x7fd0980093d0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.277+0000 7fd0a6400700 1 -- 172.21.15.43:0/358638847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a01054b0 msgr2=0x7fd0a019cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:24:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.277+0000 7fd0a6400700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a01054b0 0x7fd0a019cec0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fd09000d980 tx=0x7fd09000dd40 comp rx=0 tx=0).stop 2024-05-27T07:24:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.278+0000 7fd0a6400700 1 -- 172.21.15.43:0/358638847 shutdown_connections 2024-05-27T07:24:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.278+0000 7fd0a6400700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd0840b29f0 0x7fd0840b4f00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.278+0000 7fd0a6400700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0a010e8e0 0x7fd0a01a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.278+0000 7fd0a6400700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0a01054b0 0x7fd0a019cec0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.278+0000 7fd0a6400700 1 --2- 172.21.15.43:0/358638847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0a0105e00 0x7fd0a019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:24:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.278+0000 7fd0a6400700 1 -- 172.21.15.43:0/358638847 >> 172.21.15.43:0/358638847 conn(0x7fd0a00fedb0 msgr2=0x7fd0a0103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:24:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.278+0000 7fd0a6400700 1 -- 172.21.15.43:0/358638847 shutdown_connections 2024-05-27T07:24:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:24:56.278+0000 7fd0a6400700 1 -- 172.21.15.43:0/358638847 wait complete. 2024-05-27T07:24:57.231 INFO:tasks.workunit.client.0.smithi043.stdout: 1 61219 0.59 MB/sec execute 421 sec latency 51.415 ms 2024-05-27T07:24:57.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:57 smithi043 ceph-mon[31531]: pgmap v372: 129 pgs: 129 active+clean; 113 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 367 KiB/s wr, 35 op/s 2024-05-27T07:24:57.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:57 smithi179 ceph-mon[35888]: pgmap v372: 129 pgs: 129 active+clean; 113 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 367 KiB/s wr, 35 op/s 2024-05-27T07:24:57.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:57 smithi132 ceph-mon[35283]: pgmap v372: 129 pgs: 129 active+clean; 113 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 367 KiB/s wr, 35 op/s 2024-05-27T07:24:58.108 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:24:58.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:24:58.231 INFO:tasks.workunit.client.0.smithi043.stdout: 1 61319 0.59 MB/sec execute 422 sec latency 52.903 ms 2024-05-27T07:24:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:24:59.232 INFO:tasks.workunit.client.0.smithi043.stdout: 1 61460 0.59 MB/sec execute 423 sec latency 50.547 ms 2024-05-27T07:24:59.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:24:59 smithi043 ceph-mon[31531]: pgmap v373: 129 pgs: 129 active+clean; 113 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 318 KiB/s wr, 28 op/s 2024-05-27T07:24:59.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:24:59 smithi179 ceph-mon[35888]: pgmap v373: 129 pgs: 129 active+clean; 113 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 318 KiB/s wr, 28 op/s 2024-05-27T07:24:59.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:24:59 smithi132 ceph-mon[35283]: pgmap v373: 129 pgs: 129 active+clean; 113 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 318 KiB/s wr, 28 op/s 2024-05-27T07:25:00.232 INFO:tasks.workunit.client.0.smithi043.stdout: 1 61568 0.59 MB/sec execute 424 sec latency 61.064 ms 2024-05-27T07:25:01.232 INFO:tasks.workunit.client.0.smithi043.stdout: 1 61608 0.59 MB/sec execute 425 sec latency 723.257 ms 2024-05-27T07:25:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.061+0000 7fb89b200700 1 -- 172.21.15.43:0/2482161722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb89c10af10 msgr2=0x7fb89c10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.061+0000 7fb89b200700 1 --2- 172.21.15.43:0/2482161722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb89c10af10 0x7fb89c10b320 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb8880099c0 tx=0x7fb888009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.061+0000 7fb89b200700 1 -- 172.21.15.43:0/2482161722 shutdown_connections 2024-05-27T07:25:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.061+0000 7fb89b200700 1 --2- 172.21.15.43:0/2482161722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb89c10b860 0x7fb89c1131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.061+0000 7fb89b200700 1 --2- 172.21.15.43:0/2482161722 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89c106630 0x7fb89c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.061+0000 7fb89b200700 1 --2- 172.21.15.43:0/2482161722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb89c10af10 0x7fb89c10b320 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.061+0000 7fb89b200700 1 -- 172.21.15.43:0/2482161722 >> 172.21.15.43:0/2482161722 conn(0x7fb89c102020 msgr2=0x7fb89c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.062+0000 7fb89b200700 1 -- 172.21.15.43:0/2482161722 shutdown_connections 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.062+0000 7fb89b200700 1 -- 172.21.15.43:0/2482161722 wait complete. 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.063+0000 7fb89b200700 1 Processor -- start 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.064+0000 7fb89b200700 1 -- start start 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.064+0000 7fb89b200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb89c106630 0x7fb89c19cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.064+0000 7fb89b200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89c10af10 0x7fb89c19d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.064+0000 7fb89b200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb89c10b860 0x7fb89c1a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.064+0000 7fb89b200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c116320 con 0x7fb89c106630 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.064+0000 7fb89b200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c1161a0 con 0x7fb89c10af10 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.064+0000 7fb89b200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c1164a0 con 0x7fb89c10b860 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.065+0000 7fb899e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb89c106630 0x7fb89c19cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.065+0000 7fb899e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb89c106630 0x7fb89c19cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52294/0 (socket says 172.21.15.43:52294) 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.065+0000 7fb899e00700 1 -- 172.21.15.43:0/3119529576 learned_addr learned my addr 172.21.15.43:0/3119529576 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.065+0000 7fb899400700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89c10af10 0x7fb89c19d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.065+0000 7fb89a800700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb89c10b860 0x7fb89c1a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.065+0000 7fb899e00700 1 -- 172.21.15.43:0/3119529576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb89c10b860 msgr2=0x7fb89c1a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.065+0000 7fb899e00700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb89c10b860 0x7fb89c1a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.065+0000 7fb899e00700 1 -- 172.21.15.43:0/3119529576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89c10af10 msgr2=0x7fb89c19d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.065+0000 7fb899e00700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89c10af10 0x7fb89c19d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.065+0000 7fb899e00700 1 -- 172.21.15.43:0/3119529576 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89c1a1c20 con 0x7fb89c106630 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.066+0000 7fb899e00700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb89c106630 0x7fb89c19cd50 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00d950 tx=0x7fb88c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.066+0000 7fb892a00700 1 -- 172.21.15.43:0/3119529576 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c016070 con 0x7fb89c106630 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.066+0000 7fb89b200700 1 -- 172.21.15.43:0/3119529576 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb89c1a1f10 con 0x7fb89c106630 2024-05-27T07:25:01.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.066+0000 7fb892a00700 1 -- 172.21.15.43:0/3119529576 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb88c010460 con 0x7fb89c106630 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.066+0000 7fb892a00700 1 -- 172.21.15.43:0/3119529576 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c00f620 con 0x7fb89c106630 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.066+0000 7fb89b200700 1 -- 172.21.15.43:0/3119529576 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb89c1a9830 con 0x7fb89c106630 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.070+0000 7fb892a00700 1 -- 172.21.15.43:0/3119529576 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb88c019070 con 0x7fb89c106630 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.071+0000 7fb8af4b1b80 1 Processor -- start 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.071+0000 7fb8af4b1b80 1 -- start start 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b2f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627019afec0 0x562701b2fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b3b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562701a5cec0 con 0x562701b39340 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562701a4ca60 con 0x5627019afec0 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562701a50aa0 con 0x562701a9ba70 2024-05-27T07:25:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb892a00700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8800b29f0 0x7fb8800b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb899400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b3b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb899400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b3b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52306/0 (socket says 172.21.15.43:52306) 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb899400700 1 -- 172.21.15.43:0/1568046152 learned_addr learned my addr 172.21.15.43:0/1568046152 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb892a00700 1 -- 172.21.15.43:0/3119529576 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7fb88c0e2690 con 0x7fb89c106630 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb89a800700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8800b29f0 0x7fb8800b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb899e00700 1 --2- 172.21.15.43:0/1568046152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627019afec0 0x562701b2fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.072+0000 7fb89a800700 1 --2- 172.21.15.43:0/1568046152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b2f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.073+0000 7fb899400700 1 -- 172.21.15.43:0/1568046152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 msgr2=0x562701b2f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.073+0000 7fb899400700 1 --2- 172.21.15.43:0/1568046152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b2f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.073+0000 7fb899400700 1 -- 172.21.15.43:0/1568046152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627019afec0 msgr2=0x562701b2fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.073+0000 7fb899400700 1 --2- 172.21.15.43:0/1568046152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627019afec0 0x562701b2fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.073+0000 7fb899400700 1 -- 172.21.15.43:0/1568046152 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562701a154a0 con 0x562701b39340 2024-05-27T07:25:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.073+0000 7fb899400700 1 --2- 172.21.15.43:0/1568046152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b3b760 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb888000c00 tx=0x7fb8880096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e241553ad5886da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.073+0000 7fb89a800700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8800b29f0 0x7fb8800b4f00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x562701b2bb10 tx=0x7fb8940090b0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.073+0000 7fb887400700 1 -- 172.21.15.43:0/1568046152 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888026070 con 0x562701b39340 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.074+0000 7fb887400700 1 -- 172.21.15.43:0/1568046152 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb888005680 con 0x562701b39340 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.074+0000 7fb887400700 1 -- 172.21.15.43:0/1568046152 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880038e0 con 0x562701b39340 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.074+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/1568046152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 msgr2=0x562701b3b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.074+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/1568046152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b3b760 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb888000c00 tx=0x7fb8880096a0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.075+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/1568046152 shutdown_connections 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.075+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/1568046152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b3b760 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.075+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/1568046152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b2f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.075+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/1568046152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627019afec0 0x562701b2fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.075+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/1568046152 >> 172.21.15.43:0/1568046152 conn(0x562701a49410 msgr2=0x562701b2a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.075+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/1568046152 shutdown_connections 2024-05-27T07:25:01.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.075+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/1568046152 wait complete. 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.075+0000 7fb8af4b1b80 1 -- start start 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.076+0000 7fb8af4b1b80 1 Processor -- start 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.076+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b36270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.076+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b367b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.076+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701b39340 0x562701b36cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.076+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562701953020 con 0x5627019afec0 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.076+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5627019ed7e0 con 0x562701b39340 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.076+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562701b3be20 con 0x562701a9ba70 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.076+0000 7fb8af4b1b80 10 client.0 _ll_register_callbacks cb 0x5627019f6cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b36270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b36270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52316/0 (socket says 172.21.15.43:52316) 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899e00700 1 -- 172.21.15.43:0/832002600 learned_addr learned my addr 172.21.15.43:0/832002600 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb89a800700 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701b39340 0x562701b36cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899400700 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899e00700 1 -- 172.21.15.43:0/832002600 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 msgr2=0x562701b367b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899e00700 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b367b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899e00700 1 -- 172.21.15.43:0/832002600 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701b39340 msgr2=0x562701b36cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899e00700 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701b39340 0x562701b36cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899e00700 1 -- 172.21.15.43:0/832002600 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56270199c950 con 0x5627019afec0 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899400700 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b367b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.077+0000 7fb899e00700 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b36270 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0a9240 tx=0x7fb88c0a9180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b43a5015f21eca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.078+0000 7fb87be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.078+0000 7fb87be00700 1 -- 172.21.15.43:0/832002600 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0c3060 con 0x5627019afec0 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.078+0000 7fb87be00700 1 -- 172.21.15.43:0/832002600 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7fb88c0dfc30 con 0x5627019afec0 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.078+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/832002600 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb888005680 con 0x5627019afec0 2024-05-27T07:25:01.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.078+0000 7fb87be00700 1 -- 172.21.15.43:0/832002600 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fb88c0e2070 con 0x5627019afec0 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.078+0000 7fb8af4b1b80 10 client.17103 fetch_fsmap learned FSMap version 107 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.078+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/832002600 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x562701a1d0e0 con 0x5627019afec0 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.078+0000 7fb87be00700 1 -- 172.21.15.43:0/832002600 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb888005680 con 0x5627019afec0 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.079+0000 7fb87be00700 1 -- 172.21.15.43:0/832002600 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7fb88c0a97a0 con 0x5627019afec0 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.079+0000 7fb8af4b1b80 10 client.17103 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.079+0000 7fb8af4b1b80 10 client.17103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.079+0000 7fb8af4b1b80 20 client.17103 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.079+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562701b49100 0x562701b4b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.079+0000 7fb8af4b1b80 4 client.17103 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.079+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/832002600 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562701a5ee10 con 0x562701b49100 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.080+0000 7fb899400700 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562701b49100 0x562701b4b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.080+0000 7fb899400700 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562701b49100 0x562701b4b5b0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d118aae39097373b server_cookie=53c21a4a0990d3e7 in_seq=0 out_seq=0 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.080+0000 7fb87be00700 10 client.17103 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:01.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.082+0000 7fb87be00700 1 -- 172.21.15.43:0/832002600 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562701a5ee10 con 0x562701b49100 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.082+0000 7fb87be00700 10 client.17103 handle_command_reply: tid=0 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.082+0000 7fb8af4b1b80 1 client.17103 shutdown 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/832002600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 msgr2=0x562701b36270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b36270 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0a9240 tx=0x7fb88c0a9180 comp rx=0 tx=0).stop 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/832002600 shutdown_connections 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701b39340 0x562701b36cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562701b49100 0x562701b4b5b0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b36270 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/832002600 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b367b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/832002600 >> 172.21.15.43:0/832002600 conn(0x562701a49410 msgr2=0x562701b2a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb87be00700 0 client.17103 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/832002600 shutdown_connections 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/832002600 wait complete. 2024-05-27T07:25:01.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.083+0000 7fb8af4b1b80 20 client.17103 trim_cache size 0 max 16384 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.106+0000 7fb8af4b1b80 1 Processor -- start 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.106+0000 7fb8af4b1b80 1 -- start start 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.106+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627019afec0 0x562701b2e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.106+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b44f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.106+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b50db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562701b353f0 con 0x562701b39340 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c1b3140 con 0x5627019afec0 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c1b0de0 con 0x562701a9ba70 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb89a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b50db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb89a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b50db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52328/0 (socket says 172.21.15.43:52328) 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb89a800700 1 -- 172.21.15.43:0/3556678728 learned_addr learned my addr 172.21.15.43:0/3556678728 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb899e00700 1 --2- 172.21.15.43:0/3556678728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627019afec0 0x562701b2e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb899400700 1 --2- 172.21.15.43:0/3556678728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b44f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb89a800700 1 -- 172.21.15.43:0/3556678728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 msgr2=0x562701b44f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb89a800700 1 --2- 172.21.15.43:0/3556678728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b44f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb89a800700 1 -- 172.21.15.43:0/3556678728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627019afec0 msgr2=0x562701b2e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb89a800700 1 --2- 172.21.15.43:0/3556678728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627019afec0 0x562701b2e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.107+0000 7fb89a800700 1 -- 172.21.15.43:0/3556678728 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562701b30de0 con 0x562701b39340 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.108+0000 7fb89a800700 1 --2- 172.21.15.43:0/3556678728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b50db0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb8940186f0 tx=0x7fb894009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f812cd06545d67fa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.108+0000 7fb886a00700 1 -- 172.21.15.43:0/3556678728 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894014570 con 0x562701b39340 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.108+0000 7fb886a00700 1 -- 172.21.15.43:0/3556678728 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb894028070 con 0x562701b39340 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.108+0000 7fb886a00700 1 -- 172.21.15.43:0/3556678728 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894023940 con 0x562701b39340 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.109+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/3556678728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 msgr2=0x562701b50db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.109+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/3556678728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b50db0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb8940186f0 tx=0x7fb894009680 comp rx=0 tx=0).stop 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.109+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/3556678728 shutdown_connections 2024-05-27T07:25:01.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.109+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/3556678728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562701b39340 0x562701b50db0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.109+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/3556678728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627019afec0 0x562701b2e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.109+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/3556678728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701a9ba70 0x562701b44f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.109+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/3556678728 >> 172.21.15.43:0/3556678728 conn(0x562701a49410 msgr2=0x562701b2e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.109+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/3556678728 shutdown_connections 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.109+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/3556678728 wait complete. 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.110+0000 7fb8af4b1b80 1 -- start start 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.110+0000 7fb8af4b1b80 1 Processor -- start 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b418e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701a9ba70 0x562701b3cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb8af4b1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701b39340 0x562701b3d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c1b14b0 con 0x5627019afec0 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c1b2a70 con 0x562701a9ba70 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb8af4b1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c1b1be0 con 0x562701b39340 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb8af4b1b80 10 client.0 _ll_register_callbacks cb 0x5627019f6cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb899e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb899e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b418e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52338/0 (socket says 172.21.15.43:52338) 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb899e00700 1 -- 172.21.15.43:0/268383473 learned_addr learned my addr 172.21.15.43:0/268383473 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb899400700 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701a9ba70 0x562701b3cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb89a800700 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701b39340 0x562701b3d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb899e00700 1 -- 172.21.15.43:0/268383473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701b39340 msgr2=0x562701b3d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb899e00700 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701b39340 0x562701b3d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb899e00700 1 -- 172.21.15.43:0/268383473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701a9ba70 msgr2=0x562701b3cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.111+0000 7fb899e00700 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701a9ba70 0x562701b3cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb899e00700 1 -- 172.21.15.43:0/268383473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562701b3d9c0 con 0x5627019afec0 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb89a800700 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701b39340 0x562701b3d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb899400700 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701a9ba70 0x562701b3cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb899e00700 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b418e0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0a91f0 tx=0x7fb88c0b22b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437c344110a5a99c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:01.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb87be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb87be00700 1 -- 172.21.15.43:0/268383473 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c020040 con 0x5627019afec0 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb87be00700 1 -- 172.21.15.43:0/268383473 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9496+0+0 (secure 0 0 0) 0x7fb88c0dfc30 con 0x5627019afec0 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/268383473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb894028070 con 0x5627019afec0 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb87be00700 1 -- 172.21.15.43:0/268383473 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fb88c0dfe30 con 0x5627019afec0 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb8af4b1b80 10 client.17109 fetch_fsmap learned FSMap version 107 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.112+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/268383473 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x562701bb3c90 con 0x5627019afec0 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.113+0000 7fb87be00700 1 -- 172.21.15.43:0/268383473 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb894028070 con 0x5627019afec0 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.113+0000 7fb87be00700 1 -- 172.21.15.43:0/268383473 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7fb88c0ca070 con 0x5627019afec0 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.113+0000 7fb8af4b1b80 10 client.17109 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.113+0000 7fb8af4b1b80 10 client.17109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.113+0000 7fb8af4b1b80 20 client.17109 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.113+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562701bb41b0 0x562701bb65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.113+0000 7fb8af4b1b80 4 client.17109 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:25:01.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.113+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/268383473 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562701a585f0 con 0x562701bb41b0 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.114+0000 7fb899400700 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562701bb41b0 0x562701bb65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.115+0000 7fb899400700 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562701bb41b0 0x562701bb65d0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caef20dd86f4392d server_cookie=b4d11ebc47301ea7 in_seq=0 out_seq=0 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.115+0000 7fb87be00700 10 client.17109 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.117+0000 7fb87be00700 1 -- 172.21.15.43:0/268383473 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562701a585f0 con 0x562701bb41b0 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.117+0000 7fb87be00700 10 client.17109 handle_command_reply: tid=0 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.117+0000 7fb8af4b1b80 1 client.17109 shutdown 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/268383473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 msgr2=0x562701b418e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b418e0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0a91f0 tx=0x7fb88c0b22b0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/268383473 shutdown_connections 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562701b39340 0x562701b3d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562701bb41b0 0x562701bb65d0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627019afec0 0x562701b418e0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 1 --2- 172.21.15.43:0/268383473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562701a9ba70 0x562701b3cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/268383473 >> 172.21.15.43:0/268383473 conn(0x562701a49410 msgr2=0x562701b36050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:01.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb87be00700 0 client.17109 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:01.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/268383473 shutdown_connections 2024-05-27T07:25:01.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 1 -- 172.21.15.43:0/268383473 wait complete. 2024-05-27T07:25:01.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.118+0000 7fb8af4b1b80 20 client.17109 trim_cache size 0 max 16384 2024-05-27T07:25:01.280 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:25:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:25:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "617dc255-efa5-43e4-931b-d081dc2e2a55", 2024-05-27T07:25:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:25:01.280 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 -- 172.21.15.43:0/3119529576 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8800b29f0 msgr2=0x7fb8800b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8800b29f0 0x7fb8800b4f00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x562701b2bb10 tx=0x7fb8940090b0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 -- 172.21.15.43:0/3119529576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb89c106630 msgr2=0x7fb89c19cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb89c106630 0x7fb89c19cd50 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00d950 tx=0x7fb88c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 -- 172.21.15.43:0/3119529576 shutdown_connections 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8800b29f0 0x7fb8800b4f00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb89c10b860 0x7fb89c1a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb89c106630 0x7fb89c19cd50 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 --2- 172.21.15.43:0/3119529576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb89c10af10 0x7fb89c19d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 -- 172.21.15.43:0/3119529576 >> 172.21.15.43:0/3119529576 conn(0x7fb89c102020 msgr2=0x7fb89c10c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:01.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.119+0000 7fb89b200700 1 -- 172.21.15.43:0/3119529576 shutdown_connections 2024-05-27T07:25:01.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:01.120+0000 7fb89b200700 1 -- 172.21.15.43:0/3119529576 wait complete. 2024-05-27T07:25:01.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:01 smithi043 ceph-mon[31531]: pgmap v374: 129 pgs: 129 active+clean; 113 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 318 KiB/s wr, 28 op/s 2024-05-27T07:25:01.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:01 smithi179 ceph-mon[35888]: pgmap v374: 129 pgs: 129 active+clean; 113 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 318 KiB/s wr, 28 op/s 2024-05-27T07:25:01.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:01 smithi132 ceph-mon[35283]: pgmap v374: 129 pgs: 129 active+clean; 113 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 318 KiB/s wr, 28 op/s 2024-05-27T07:25:02.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:25:02.232 INFO:tasks.workunit.client.0.smithi043.stdout: 1 61757 0.60 MB/sec execute 426 sec latency 79.006 ms 2024-05-27T07:25:02.582 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:02.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:25:02.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:25:02.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:25:02.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:25:02.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:25:02.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:25:02.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:25:02.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:02 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:25:02.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:02 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:25:02.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:25:02.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:25:02.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:25:02.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:25:02.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:25:02.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:25:02.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:02.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:02 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:25:03.232 INFO:tasks.workunit.client.0.smithi043.stdout: 1 61888 0.59 MB/sec execute 427 sec latency 74.562 ms 2024-05-27T07:25:03.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:03 smithi043 ceph-mon[31531]: pgmap v375: 129 pgs: 129 active+clean; 111 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 559 KiB/s wr, 67 op/s 2024-05-27T07:25:03.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:03 smithi043 ceph-mon[31531]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:25:03.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:03 smithi179 ceph-mon[35888]: pgmap v375: 129 pgs: 129 active+clean; 111 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 559 KiB/s wr, 67 op/s 2024-05-27T07:25:03.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:03 smithi179 ceph-mon[35888]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:25:03.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:03 smithi132 ceph-mon[35283]: pgmap v375: 129 pgs: 129 active+clean; 111 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 559 KiB/s wr, 67 op/s 2024-05-27T07:25:03.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:03 smithi132 ceph-mon[35283]: osdmap e130: 12 total, 12 up, 12 in 2024-05-27T07:25:04.232 INFO:tasks.workunit.client.0.smithi043.stdout: 1 61979 0.59 MB/sec execute 428 sec latency 79.244 ms 2024-05-27T07:25:05.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.089+0000 7fa513200700 1 -- 172.21.15.43:0/155778931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa50c1034b0 msgr2=0x7fa50c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.089+0000 7fa513200700 1 --2- 172.21.15.43:0/155778931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa50c1034b0 0x7fa50c1038c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0099c0 tx=0x7fa4fc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.089+0000 7fa513200700 1 -- 172.21.15.43:0/155778931 shutdown_connections 2024-05-27T07:25:05.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.089+0000 7fa513200700 1 --2- 172.21.15.43:0/155778931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa50c108730 0x7fa50c110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.089+0000 7fa513200700 1 --2- 172.21.15.43:0/155778931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa50c1034b0 0x7fa50c1038c0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.089+0000 7fa513200700 1 --2- 172.21.15.43:0/155778931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa50c107de0 0x7fa50c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.089+0000 7fa513200700 1 -- 172.21.15.43:0/155778931 >> 172.21.15.43:0/155778931 conn(0x7fa50c0fedb0 msgr2=0x7fa50c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.089+0000 7fa513200700 1 -- 172.21.15.43:0/155778931 shutdown_connections 2024-05-27T07:25:05.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.089+0000 7fa513200700 1 -- 172.21.15.43:0/155778931 wait complete. 2024-05-27T07:25:05.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.090+0000 7fa513200700 1 Processor -- start 2024-05-27T07:25:05.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.090+0000 7fa513200700 1 -- start start 2024-05-27T07:25:05.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.091+0000 7fa513200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa50c1034b0 0x7fa50c1a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.091+0000 7fa513200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa50c107de0 0x7fa50c1a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.091+0000 7fa513200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa50c108730 0x7fa50c19ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.091+0000 7fa513200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c114360 con 0x7fa50c107de0 2024-05-27T07:25:05.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.091+0000 7fa513200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c1141e0 con 0x7fa50c108730 2024-05-27T07:25:05.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.091+0000 7fa513200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c1144e0 con 0x7fa50c1034b0 2024-05-27T07:25:05.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.091+0000 7fa50b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa50c107de0 0x7fa50c1a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.091+0000 7fa50b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa50c107de0 0x7fa50c1a0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32776/0 (socket says 172.21.15.43:32776) 2024-05-27T07:25:05.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.091+0000 7fa50b400700 1 -- 172.21.15.43:0/3374612156 learned_addr learned my addr 172.21.15.43:0/3374612156 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:05.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.092+0000 7fa511200700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa50c108730 0x7fa50c19ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.092+0000 7fa50be00700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa50c1034b0 0x7fa50c1a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.092+0000 7fa50b400700 1 -- 172.21.15.43:0/3374612156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa50c1034b0 msgr2=0x7fa50c1a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.092+0000 7fa50b400700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa50c1034b0 0x7fa50c1a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.092+0000 7fa50b400700 1 -- 172.21.15.43:0/3374612156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa50c108730 msgr2=0x7fa50c19ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.092+0000 7fa50b400700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa50c108730 0x7fa50c19ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.092+0000 7fa50b400700 1 -- 172.21.15.43:0/3374612156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50c19b270 con 0x7fa50c107de0 2024-05-27T07:25:05.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.092+0000 7fa511200700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa50c108730 0x7fa50c19ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:05.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.092+0000 7fa50b400700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa50c107de0 0x7fa50c1a0ef0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800d950 tx=0x7fa4f800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.093+0000 7fa508c00700 1 -- 172.21.15.43:0/3374612156 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8016070 con 0x7fa50c107de0 2024-05-27T07:25:05.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.093+0000 7fa508c00700 1 -- 172.21.15.43:0/3374612156 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4f8004830 con 0x7fa50c107de0 2024-05-27T07:25:05.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.093+0000 7fa513200700 1 -- 172.21.15.43:0/3374612156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50c19b560 con 0x7fa50c107de0 2024-05-27T07:25:05.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.093+0000 7fa508c00700 1 -- 172.21.15.43:0/3374612156 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f80053e0 con 0x7fa50c107de0 2024-05-27T07:25:05.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.093+0000 7fa513200700 1 -- 172.21.15.43:0/3374612156 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50c1a7760 con 0x7fa50c107de0 2024-05-27T07:25:05.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.098+0000 7fa508c00700 1 -- 172.21.15.43:0/3374612156 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa4f8005580 con 0x7fa50c107de0 2024-05-27T07:25:05.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.098+0000 7fa521d30b80 1 Processor -- start 2024-05-27T07:25:05.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.098+0000 7fa521d30b80 1 -- start start 2024-05-27T07:25:05.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.099+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9f1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.099+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb872e40 0x55b9bb9f1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.099+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9fe110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.099+0000 7fa521d30b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b9bb91fe40 con 0x55b9bb9f1e70 2024-05-27T07:25:05.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.099+0000 7fa521d30b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b9bb90f9e0 con 0x55b9bb95e9f0 2024-05-27T07:25:05.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.099+0000 7fa521d30b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b9bb913a20 con 0x55b9bb872e40 2024-05-27T07:25:05.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.100+0000 7fa50b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9fe110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.100+0000 7fa50b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9fe110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32782/0 (socket says 172.21.15.43:32782) 2024-05-27T07:25:05.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.100+0000 7fa50b400700 1 -- 172.21.15.43:0/3804209664 learned_addr learned my addr 172.21.15.43:0/3804209664 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:05.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.100+0000 7fa511200700 1 --2- 172.21.15.43:0/3804209664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9f1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.100+0000 7fa50be00700 1 --2- 172.21.15.43:0/3804209664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb872e40 0x55b9bb9f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.100+0000 7fa508c00700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa4f00b2a70 0x7fa4f00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.100+0000 7fa50b400700 1 -- 172.21.15.43:0/3804209664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb872e40 msgr2=0x55b9bb9f1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.100+0000 7fa50b400700 1 --2- 172.21.15.43:0/3804209664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb872e40 0x55b9bb9f1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.100+0000 7fa50b400700 1 -- 172.21.15.43:0/3804209664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 msgr2=0x55b9bb9f1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.100+0000 7fa50b400700 1 --2- 172.21.15.43:0/3804209664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9f1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.101+0000 7fa50b400700 1 -- 172.21.15.43:0/3804209664 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9bb8d8420 con 0x55b9bb9f1e70 2024-05-27T07:25:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.101+0000 7fa511200700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa4f00b2a70 0x7fa4f00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.101+0000 7fa508c00700 1 -- 172.21.15.43:0/3374612156 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9584+0+0 (secure 0 0 0) 0x7fa4f80e2540 con 0x7fa50c107de0 2024-05-27T07:25:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.101+0000 7fa50b400700 1 --2- 172.21.15.43:0/3804209664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9fe110 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80df770 tx=0x7fa4f8008420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f342eecd91a91d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.101+0000 7fa501600700 1 -- 172.21.15.43:0/3804209664 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f80a8d10 con 0x55b9bb9f1e70 2024-05-27T07:25:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.102+0000 7fa501600700 1 -- 172.21.15.43:0/3804209664 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4f80187d0 con 0x55b9bb9f1e70 2024-05-27T07:25:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.102+0000 7fa501600700 1 -- 172.21.15.43:0/3804209664 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8018a20 con 0x55b9bb9f1e70 2024-05-27T07:25:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.103+0000 7fa521d30b80 1 -- 172.21.15.43:0/3804209664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 msgr2=0x55b9bb9fe110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.103+0000 7fa511200700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa4f00b2a70 0x7fa4f00b4f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55b9bb9f3ff0 tx=0x7fa504009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.103+0000 7fa521d30b80 1 --2- 172.21.15.43:0/3804209664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9fe110 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80df770 tx=0x7fa4f8008420 comp rx=0 tx=0).stop 2024-05-27T07:25:05.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.103+0000 7fa521d30b80 1 -- 172.21.15.43:0/3804209664 shutdown_connections 2024-05-27T07:25:05.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.103+0000 7fa521d30b80 1 --2- 172.21.15.43:0/3804209664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9fe110 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.103+0000 7fa521d30b80 1 --2- 172.21.15.43:0/3804209664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9f1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.103+0000 7fa521d30b80 1 --2- 172.21.15.43:0/3804209664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb872e40 0x55b9bb9f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.103+0000 7fa521d30b80 1 -- 172.21.15.43:0/3804209664 >> 172.21.15.43:0/3804209664 conn(0x55b9bb90c390 msgr2=0x55b9bb9ed4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.103+0000 7fa521d30b80 1 -- 172.21.15.43:0/3804209664 shutdown_connections 2024-05-27T07:25:05.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.103+0000 7fa521d30b80 1 -- 172.21.15.43:0/3804209664 wait complete. 2024-05-27T07:25:05.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.104+0000 7fa521d30b80 1 -- start start 2024-05-27T07:25:05.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.104+0000 7fa521d30b80 1 Processor -- start 2024-05-27T07:25:05.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.105+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9fb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.105+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9fbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.105+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9fc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.105+0000 7fa521d30b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b9bb815fa0 con 0x55b9bb872e40 2024-05-27T07:25:05.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.105+0000 7fa521d30b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b9bb8b0760 con 0x55b9bb95e9f0 2024-05-27T07:25:05.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.105+0000 7fa521d30b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b9bba02f30 con 0x55b9bb9f1e70 2024-05-27T07:25:05.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.105+0000 7fa521d30b80 10 client.0 _ll_register_callbacks cb 0x55b9bb8b9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:05.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.105+0000 7fa50be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9fb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.105+0000 7fa50be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9fb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32796/0 (socket says 172.21.15.43:32796) 2024-05-27T07:25:05.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.105+0000 7fa50be00700 1 -- 172.21.15.43:0/3906995433 learned_addr learned my addr 172.21.15.43:0/3906995433 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:05.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.106+0000 7fa50b400700 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9fbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.106+0000 7fa511200700 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9fc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.106+0000 7fa50be00700 1 -- 172.21.15.43:0/3906995433 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb9f1e70 msgr2=0x55b9bb9fc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.106+0000 7fa50be00700 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9fc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.106+0000 7fa50be00700 1 -- 172.21.15.43:0/3906995433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 msgr2=0x55b9bb9fbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.106+0000 7fa50be00700 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9fbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.106+0000 7fa50be00700 1 -- 172.21.15.43:0/3906995433 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9bb85f8d0 con 0x55b9bb872e40 2024-05-27T07:25:05.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.106+0000 7fa50be00700 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9fb620 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00b580 tx=0x7fa4fc003720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151c76bbaafc9c01 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.106+0000 7fa4f6000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:05.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.107+0000 7fa4f6000700 1 -- 172.21.15.43:0/3906995433 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc003bd0 con 0x55b9bb872e40 2024-05-27T07:25:05.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.107+0000 7fa4f6000700 1 -- 172.21.15.43:0/3906995433 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9584+0+0 (secure 0 0 0) 0x7fa4fc003d70 con 0x55b9bb872e40 2024-05-27T07:25:05.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.107+0000 7fa521d30b80 1 -- 172.21.15.43:0/3906995433 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b9bb921d90 con 0x55b9bb872e40 2024-05-27T07:25:05.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.107+0000 7fa4f6000700 1 -- 172.21.15.43:0/3906995433 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fa4fc032050 con 0x55b9bb872e40 2024-05-27T07:25:05.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.107+0000 7fa521d30b80 10 client.17121 fetch_fsmap learned FSMap version 107 2024-05-27T07:25:05.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.107+0000 7fa521d30b80 1 -- 172.21.15.43:0/3906995433 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55b9bb8e0060 con 0x55b9bb872e40 2024-05-27T07:25:05.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.107+0000 7fa4f6000700 1 -- 172.21.15.43:0/3906995433 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55b9bb921d90 con 0x55b9bb872e40 2024-05-27T07:25:05.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.108+0000 7fa4f6000700 1 -- 172.21.15.43:0/3906995433 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7fa4fc024540 con 0x55b9bb872e40 2024-05-27T07:25:05.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.108+0000 7fa521d30b80 10 client.17121 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:25:05.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.109+0000 7fa521d30b80 10 client.17121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:05.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.109+0000 7fa521d30b80 20 client.17121 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:05.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.109+0000 7fa521d30b80 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55b9bba102a0 0x55b9bba12750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.109+0000 7fa521d30b80 4 client.17121 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:05.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.109+0000 7fa521d30b80 1 -- 172.21.15.43:0/3906995433 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9bb91b570 con 0x55b9bba102a0 2024-05-27T07:25:05.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.109+0000 7fa50b400700 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55b9bba102a0 0x55b9bba12750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.110+0000 7fa50b400700 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55b9bba102a0 0x55b9bba12750 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4b2fcdfd3267f4b server_cookie=fa28355be2f15354 in_seq=0 out_seq=0 2024-05-27T07:25:05.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.110+0000 7fa4f6000700 10 client.17121 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:05.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa4f6000700 1 -- 172.21.15.43:0/3906995433 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b9bb91b570 con 0x55b9bba102a0 2024-05-27T07:25:05.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa4f6000700 10 client.17121 handle_command_reply: tid=0 2024-05-27T07:25:05.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa521d30b80 1 client.17121 shutdown 2024-05-27T07:25:05.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa521d30b80 1 -- 172.21.15.43:0/3906995433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 msgr2=0x55b9bb9fb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa521d30b80 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9fb620 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00b580 tx=0x7fa4fc003720 comp rx=0 tx=0).stop 2024-05-27T07:25:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa521d30b80 1 -- 172.21.15.43:0/3906995433 shutdown_connections 2024-05-27T07:25:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa521d30b80 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9fc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa521d30b80 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55b9bba102a0 0x55b9bba12750 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa521d30b80 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9fb620 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa521d30b80 1 --2- 172.21.15.43:0/3906995433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9fbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa521d30b80 1 -- 172.21.15.43:0/3906995433 >> 172.21.15.43:0/3906995433 conn(0x55b9bb90c390 msgr2=0x55b9bb9f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.113+0000 7fa4f6000700 0 client.17121 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.114+0000 7fa521d30b80 1 -- 172.21.15.43:0/3906995433 shutdown_connections 2024-05-27T07:25:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.114+0000 7fa521d30b80 1 -- 172.21.15.43:0/3906995433 wait complete. 2024-05-27T07:25:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.114+0000 7fa521d30b80 20 client.17121 trim_cache size 0 max 16384 2024-05-27T07:25:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.141+0000 7fa521d30b80 1 Processor -- start 2024-05-27T07:25:05.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.141+0000 7fa521d30b80 1 -- start start 2024-05-27T07:25:05.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bba0bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa521d30b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b9bba12b40 con 0x55b9bb872e40 2024-05-27T07:25:05.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa521d30b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c1af1c0 con 0x55b9bb95e9f0 2024-05-27T07:25:05.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa521d30b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c1ace60 con 0x55b9bb9f1e70 2024-05-27T07:25:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa50be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa50be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9f6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32806/0 (socket says 172.21.15.43:32806) 2024-05-27T07:25:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa50be00700 1 -- 172.21.15.43:0/4078553533 learned_addr learned my addr 172.21.15.43:0/4078553533 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa50b400700 1 --2- 172.21.15.43:0/4078553533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bba0bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.142+0000 7fa511200700 1 --2- 172.21.15.43:0/4078553533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.143+0000 7fa50be00700 1 -- 172.21.15.43:0/4078553533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb9f1e70 msgr2=0x55b9bb9f35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.143+0000 7fa50be00700 1 --2- 172.21.15.43:0/4078553533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9f35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.143+0000 7fa50be00700 1 -- 172.21.15.43:0/4078553533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 msgr2=0x55b9bba0bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.143+0000 7fa50be00700 1 --2- 172.21.15.43:0/4078553533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bba0bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.143+0000 7fa50be00700 1 -- 172.21.15.43:0/4078553533 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9bb9fc410 con 0x55b9bb872e40 2024-05-27T07:25:05.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.143+0000 7fa50be00700 1 --2- 172.21.15.43:0/4078553533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9f6e30 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc024980 tx=0x7fa4fc009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18603643479494ab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.143+0000 7fa500c00700 1 -- 172.21.15.43:0/4078553533 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc024670 con 0x55b9bb872e40 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.144+0000 7fa500c00700 1 -- 172.21.15.43:0/4078553533 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4fc029070 con 0x55b9bb872e40 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.144+0000 7fa500c00700 1 -- 172.21.15.43:0/4078553533 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc00b730 con 0x55b9bb872e40 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.144+0000 7fa521d30b80 1 -- 172.21.15.43:0/4078553533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 msgr2=0x55b9bb9f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.144+0000 7fa521d30b80 1 --2- 172.21.15.43:0/4078553533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9f6e30 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc024980 tx=0x7fa4fc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.145+0000 7fa521d30b80 1 -- 172.21.15.43:0/4078553533 shutdown_connections 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.145+0000 7fa521d30b80 1 --2- 172.21.15.43:0/4078553533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.145+0000 7fa521d30b80 1 --2- 172.21.15.43:0/4078553533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb872e40 0x55b9bb9f6e30 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.145+0000 7fa521d30b80 1 --2- 172.21.15.43:0/4078553533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb95e9f0 0x55b9bba0bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.145+0000 7fa521d30b80 1 -- 172.21.15.43:0/4078553533 >> 172.21.15.43:0/4078553533 conn(0x55b9bb90c390 msgr2=0x55b9bb9f2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.145+0000 7fa521d30b80 1 -- 172.21.15.43:0/4078553533 shutdown_connections 2024-05-27T07:25:05.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.145+0000 7fa521d30b80 1 -- 172.21.15.43:0/4078553533 wait complete. 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.146+0000 7fa521d30b80 1 -- start start 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.146+0000 7fa521d30b80 1 Processor -- start 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.146+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb872e40 0x55b9bb9f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.146+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.146+0000 7fa521d30b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9f5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.146+0000 7fa521d30b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c1ad530 con 0x55b9bb9f1e70 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.146+0000 7fa521d30b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c1aeaf0 con 0x55b9bb872e40 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.146+0000 7fa521d30b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c1adc60 con 0x55b9bb95e9f0 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.147+0000 7fa521d30b80 10 client.0 _ll_register_callbacks cb 0x55b9bb8b9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.147+0000 7fa511200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.147+0000 7fa511200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9f5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32820/0 (socket says 172.21.15.43:32820) 2024-05-27T07:25:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.147+0000 7fa511200700 1 -- 172.21.15.43:0/793999427 learned_addr learned my addr 172.21.15.43:0/793999427 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.147+0000 7fa50be00700 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb872e40 0x55b9bb9f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.147+0000 7fa50b400700 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.148+0000 7fa511200700 1 -- 172.21.15.43:0/793999427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb95e9f0 msgr2=0x55b9bb9f50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.148+0000 7fa511200700 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9f50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.148+0000 7fa511200700 1 -- 172.21.15.43:0/793999427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb872e40 msgr2=0x55b9bb9f6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.148+0000 7fa511200700 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb872e40 0x55b9bb9f6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.148+0000 7fa511200700 1 -- 172.21.15.43:0/793999427 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9bb9f5e00 con 0x55b9bb9f1e70 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.148+0000 7fa511200700 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9f5660 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa504016650 tx=0x7fa504004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f0d71601aec477 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.148+0000 7fa4f6000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.148+0000 7fa4f6000700 1 -- 172.21.15.43:0/793999427 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50402b600 con 0x55b9bb9f1e70 2024-05-27T07:25:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.148+0000 7fa4f6000700 1 -- 172.21.15.43:0/793999427 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9584+0+0 (secure 0 0 0) 0x7fa50402b7a0 con 0x55b9bb9f1e70 2024-05-27T07:25:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.148+0000 7fa521d30b80 1 -- 172.21.15.43:0/793999427 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4fc029070 con 0x55b9bb9f1e70 2024-05-27T07:25:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.149+0000 7fa4f6000700 1 -- 172.21.15.43:0/793999427 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fa504034050 con 0x55b9bb9f1e70 2024-05-27T07:25:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.149+0000 7fa521d30b80 10 client.17127 fetch_fsmap learned FSMap version 107 2024-05-27T07:25:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.149+0000 7fa521d30b80 1 -- 172.21.15.43:0/793999427 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=4+}) -- 0x55b9bba081e0 con 0x55b9bb9f1e70 2024-05-27T07:25:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.149+0000 7fa4f6000700 1 -- 172.21.15.43:0/793999427 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4fc029070 con 0x55b9bb9f1e70 2024-05-27T07:25:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.149+0000 7fa4f6000700 1 -- 172.21.15.43:0/793999427 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 107) ==== 4808+0+0 (secure 0 0 0) 0x7fa50402ccd0 con 0x55b9bb9f1e70 2024-05-27T07:25:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.149+0000 7fa521d30b80 10 client.17127 fetch_fsmap finished waiting for FSMap version 107 2024-05-27T07:25:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.149+0000 7fa521d30b80 10 client.17127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.150+0000 7fa521d30b80 20 client.17127 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.150+0000 7fa521d30b80 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55b9bba08980 0x55b9bba7cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:05.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.150+0000 7fa521d30b80 4 client.17127 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:25:05.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.150+0000 7fa521d30b80 1 -- 172.21.15.43:0/793999427 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b9bb792250 con 0x55b9bba08980 2024-05-27T07:25:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.150+0000 7fa50be00700 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55b9bba08980 0x55b9bba7cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.151+0000 7fa50be00700 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55b9bba08980 0x55b9bba7cdf0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f44ea723de478924 server_cookie=7f355e7d94c99527 in_seq=0 out_seq=0 2024-05-27T07:25:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.151+0000 7fa4f6000700 10 client.17127 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa4f6000700 1 -- 172.21.15.43:0/793999427 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b9bb792250 con 0x55b9bba08980 2024-05-27T07:25:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa4f6000700 10 client.17127 handle_command_reply: tid=0 2024-05-27T07:25:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa521d30b80 1 client.17127 shutdown 2024-05-27T07:25:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa521d30b80 1 -- 172.21.15.43:0/793999427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 msgr2=0x55b9bb9f5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa521d30b80 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9f5660 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa504016650 tx=0x7fa504004ad0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa521d30b80 1 -- 172.21.15.43:0/793999427 shutdown_connections 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa521d30b80 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55b9bba08980 0x55b9bba7cdf0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa521d30b80 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b9bb9f1e70 0x55b9bb9f5660 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa521d30b80 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b9bb872e40 0x55b9bb9f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa521d30b80 1 --2- 172.21.15.43:0/793999427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b9bb95e9f0 0x55b9bb9f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa4f6000700 0 client.17127 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.152+0000 7fa521d30b80 1 -- 172.21.15.43:0/793999427 >> 172.21.15.43:0/793999427 conn(0x55b9bb90c390 msgr2=0x55b9bb9f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.154+0000 7fa521d30b80 1 -- 172.21.15.43:0/793999427 shutdown_connections 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.154+0000 7fa521d30b80 1 -- 172.21.15.43:0/793999427 wait complete. 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.154+0000 7fa521d30b80 20 client.17127 trim_cache size 0 max 16384 2024-05-27T07:25:05.147 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:25:05.148 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:25:05.148 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:25:05.148 INFO:teuthology.orchestra.run.smithi043.stdout: "617dc255-efa5-43e4-931b-d081dc2e2a55": { 2024-05-27T07:25:05.148 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:25:05.148 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "617dc255-efa5-43e4-931b-d081dc2e2a55", 2024-05-27T07:25:05.148 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:25:05.148 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:25:05.148 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:25:05.148 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.155+0000 7fa513200700 1 -- 172.21.15.43:0/3374612156 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa4f00b2a70 msgr2=0x7fa4f00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.155+0000 7fa513200700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa4f00b2a70 0x7fa4f00b4f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55b9bb9f3ff0 tx=0x7fa504009040 comp rx=0 tx=0).stop 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.155+0000 7fa513200700 1 -- 172.21.15.43:0/3374612156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa50c107de0 msgr2=0x7fa50c1a0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.155+0000 7fa513200700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa50c107de0 0x7fa50c1a0ef0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800d950 tx=0x7fa4f800dc60 comp rx=0 tx=0).stop 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.155+0000 7fa513200700 1 -- 172.21.15.43:0/3374612156 shutdown_connections 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.155+0000 7fa513200700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa4f00b2a70 0x7fa4f00b4f80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.155+0000 7fa513200700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa50c108730 0x7fa50c19ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.155+0000 7fa513200700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa50c1034b0 0x7fa50c1a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.155+0000 7fa513200700 1 --2- 172.21.15.43:0/3374612156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa50c107de0 0x7fa50c1a0ef0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.155+0000 7fa513200700 1 -- 172.21.15.43:0/3374612156 >> 172.21.15.43:0/3374612156 conn(0x7fa50c0fedb0 msgr2=0x7fa50c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.156+0000 7fa513200700 1 -- 172.21.15.43:0/3374612156 shutdown_connections 2024-05-27T07:25:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:05.156+0000 7fa513200700 1 -- 172.21.15.43:0/3374612156 wait complete. 2024-05-27T07:25:05.232 INFO:tasks.workunit.client.0.smithi043.stdout: 1 62140 0.59 MB/sec execute 429 sec latency 73.884 ms 2024-05-27T07:25:05.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:05 smithi132 ceph-mon[35283]: pgmap v377: 129 pgs: 129 active+clean; 111 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 484 KiB/s wr, 67 op/s 2024-05-27T07:25:05.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:05 smithi043 ceph-mon[31531]: pgmap v377: 129 pgs: 129 active+clean; 111 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 484 KiB/s wr, 67 op/s 2024-05-27T07:25:05.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:05 smithi179 ceph-mon[35888]: pgmap v377: 129 pgs: 129 active+clean; 111 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 484 KiB/s wr, 67 op/s 2024-05-27T07:25:06.232 INFO:tasks.workunit.client.0.smithi043.stdout: 1 62238 0.59 MB/sec execute 430 sec latency 56.136 ms 2024-05-27T07:25:06.260 INFO:tasks.cephfs.filesystem:scrub status for tag:617dc255-efa5-43e4-931b-d081dc2e2a55 - {'path': '/', 'tag': '617dc255-efa5-43e4-931b-d081dc2e2a55', 'options': 'recursive,force'} 2024-05-27T07:25:07.232 INFO:tasks.workunit.client.0.smithi043.stdout: 1 62384 0.59 MB/sec execute 431 sec latency 58.022 ms 2024-05-27T07:25:07.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:07 smithi043 ceph-mon[31531]: pgmap v378: 129 pgs: 129 active+clean; 111 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 450 KiB/s wr, 67 op/s 2024-05-27T07:25:07.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:07 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:25:07.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:07 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:25:07.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:07 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:25:07.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:07 smithi179 ceph-mon[35888]: pgmap v378: 129 pgs: 129 active+clean; 111 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 450 KiB/s wr, 67 op/s 2024-05-27T07:25:07.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:07 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:25:07.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:07 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:25:07.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:07 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:25:07.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:07 smithi132 ceph-mon[35283]: pgmap v378: 129 pgs: 129 active+clean; 111 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 450 KiB/s wr, 67 op/s 2024-05-27T07:25:07.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:07 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:25:07.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:07 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:25:07.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:07 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:25:08.233 INFO:tasks.workunit.client.0.smithi043.stdout: 1 62499 0.59 MB/sec execute 432 sec latency 65.970 ms 2024-05-27T07:25:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:08 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:25:08.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:08 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:25:08.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:08 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:25:09.233 INFO:tasks.workunit.client.0.smithi043.stdout: 1 62637 0.59 MB/sec execute 433 sec latency 59.232 ms 2024-05-27T07:25:09.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:09 smithi043 ceph-mon[31531]: pgmap v379: 129 pgs: 129 active+clean; 111 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 450 KiB/s wr, 67 op/s 2024-05-27T07:25:09.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:09 smithi179 ceph-mon[35888]: pgmap v379: 129 pgs: 129 active+clean; 111 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 450 KiB/s wr, 67 op/s 2024-05-27T07:25:09.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:09 smithi132 ceph-mon[35283]: pgmap v379: 129 pgs: 129 active+clean; 111 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 450 KiB/s wr, 67 op/s 2024-05-27T07:25:10.233 INFO:tasks.workunit.client.0.smithi043.stdout: 1 62717 0.59 MB/sec execute 434 sec latency 61.440 ms 2024-05-27T07:25:11.233 INFO:tasks.workunit.client.0.smithi043.stdout: 1 62785 0.59 MB/sec execute 435 sec latency 53.417 ms 2024-05-27T07:25:11.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:11 smithi043 ceph-mon[31531]: pgmap v380: 129 pgs: 129 active+clean; 111 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 450 KiB/s wr, 67 op/s 2024-05-27T07:25:11.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:11 smithi043 ceph-mon[31531]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:25:11.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:11 smithi179 ceph-mon[35888]: pgmap v380: 129 pgs: 129 active+clean; 111 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 450 KiB/s wr, 67 op/s 2024-05-27T07:25:11.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:11 smithi179 ceph-mon[35888]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:25:11.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:11 smithi132 ceph-mon[35283]: pgmap v380: 129 pgs: 129 active+clean; 111 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 450 KiB/s wr, 67 op/s 2024-05-27T07:25:11.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:11 smithi132 ceph-mon[35283]: osdmap e131: 12 total, 12 up, 12 in 2024-05-27T07:25:12.233 INFO:tasks.workunit.client.0.smithi043.stdout: 1 62906 0.59 MB/sec execute 436 sec latency 62.783 ms 2024-05-27T07:25:13.233 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63086 0.59 MB/sec execute 437 sec latency 56.850 ms 2024-05-27T07:25:13.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:13 smithi043 ceph-mon[31531]: pgmap v382: 129 pgs: 129 active+clean; 113 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 365 KiB/s wr, 35 op/s 2024-05-27T07:25:13.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:13 smithi179 ceph-mon[35888]: pgmap v382: 129 pgs: 129 active+clean; 113 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 365 KiB/s wr, 35 op/s 2024-05-27T07:25:13.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:13 smithi132 ceph-mon[35283]: pgmap v382: 129 pgs: 129 active+clean; 113 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 365 KiB/s wr, 35 op/s 2024-05-27T07:25:14.233 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63258 0.60 MB/sec execute 438 sec latency 56.437 ms 2024-05-27T07:25:15.233 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63354 0.59 MB/sec execute 439 sec latency 61.798 ms 2024-05-27T07:25:15.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:15 smithi043 ceph-mon[31531]: pgmap v383: 129 pgs: 129 active+clean; 113 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 361 KiB/s wr, 35 op/s 2024-05-27T07:25:15.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:15 smithi179 ceph-mon[35888]: pgmap v383: 129 pgs: 129 active+clean; 113 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 361 KiB/s wr, 35 op/s 2024-05-27T07:25:15.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:15 smithi132 ceph-mon[35283]: pgmap v383: 129 pgs: 129 active+clean; 113 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 361 KiB/s wr, 35 op/s 2024-05-27T07:25:16.233 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63607 0.60 MB/sec execute 440 sec latency 57.853 ms 2024-05-27T07:25:17.234 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63636 0.60 MB/sec execute 441 sec latency 57.692 ms 2024-05-27T07:25:17.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:17 smithi043 ceph-mon[31531]: pgmap v384: 129 pgs: 129 active+clean; 116 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-05-27T07:25:17.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:17 smithi179 ceph-mon[35888]: pgmap v384: 129 pgs: 129 active+clean; 116 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-05-27T07:25:17.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:17 smithi132 ceph-mon[35283]: pgmap v384: 129 pgs: 129 active+clean; 116 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-05-27T07:25:18.234 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63659 0.60 MB/sec execute 442 sec latency 57.558 ms 2024-05-27T07:25:19.234 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63816 0.60 MB/sec execute 443 sec latency 51.804 ms 2024-05-27T07:25:19.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:19 smithi043 ceph-mon[31531]: pgmap v385: 129 pgs: 129 active+clean; 116 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-05-27T07:25:19.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:19 smithi179 ceph-mon[35888]: pgmap v385: 129 pgs: 129 active+clean; 116 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-05-27T07:25:19.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:19 smithi132 ceph-mon[35283]: pgmap v385: 129 pgs: 129 active+clean; 116 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-05-27T07:25:20.234 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63902 0.60 MB/sec execute 444 sec latency 60.799 ms 2024-05-27T07:25:21.234 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63943 0.60 MB/sec execute 445 sec latency 840.705 ms 2024-05-27T07:25:21.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:21 smithi043 ceph-mon[31531]: pgmap v386: 129 pgs: 129 active+clean; 116 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-05-27T07:25:21.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:21 smithi179 ceph-mon[35888]: pgmap v386: 129 pgs: 129 active+clean; 116 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-05-27T07:25:21.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:21 smithi132 ceph-mon[35283]: pgmap v386: 129 pgs: 129 active+clean; 116 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-05-27T07:25:22.234 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63943 0.60 MB/sec execute 446 sec latency 1840.850 ms 2024-05-27T07:25:23.234 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63943 0.60 MB/sec execute 447 sec latency 2840.946 ms 2024-05-27T07:25:23.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:23 smithi179 ceph-mon[35888]: pgmap v387: 129 pgs: 129 active+clean; 117 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 503 KiB/s wr, 32 op/s 2024-05-27T07:25:23.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:23 smithi132 ceph-mon[35283]: pgmap v387: 129 pgs: 129 active+clean; 117 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 503 KiB/s wr, 32 op/s 2024-05-27T07:25:23.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:23 smithi043 ceph-mon[31531]: pgmap v387: 129 pgs: 129 active+clean; 117 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 503 KiB/s wr, 32 op/s 2024-05-27T07:25:24.234 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63993 0.59 MB/sec execute 448 sec latency 3064.371 ms 2024-05-27T07:25:25.235 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64092 0.60 MB/sec execute 449 sec latency 68.702 ms 2024-05-27T07:25:25.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:25 smithi132 ceph-mon[35283]: pgmap v388: 129 pgs: 129 active+clean; 117 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 461 KiB/s wr, 29 op/s 2024-05-27T07:25:25.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:25 smithi043 ceph-mon[31531]: pgmap v388: 129 pgs: 129 active+clean; 117 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 461 KiB/s wr, 29 op/s 2024-05-27T07:25:25.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:25 smithi179 ceph-mon[35888]: pgmap v388: 129 pgs: 129 active+clean; 117 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 461 KiB/s wr, 29 op/s 2024-05-27T07:25:26.235 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64141 0.59 MB/sec execute 450 sec latency 55.955 ms 2024-05-27T07:25:27.235 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64196 0.59 MB/sec execute 451 sec latency 50.538 ms 2024-05-27T07:25:27.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:27 smithi132 ceph-mon[35283]: pgmap v389: 129 pgs: 129 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 620 KiB/s wr, 39 op/s 2024-05-27T07:25:27.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:27 smithi043 ceph-mon[31531]: pgmap v389: 129 pgs: 129 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 620 KiB/s wr, 39 op/s 2024-05-27T07:25:27.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:27 smithi179 ceph-mon[35888]: pgmap v389: 129 pgs: 129 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 620 KiB/s wr, 39 op/s 2024-05-27T07:25:28.235 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64257 0.59 MB/sec execute 452 sec latency 60.096 ms 2024-05-27T07:25:29.235 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64311 0.59 MB/sec execute 453 sec latency 51.641 ms 2024-05-27T07:25:29.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:29 smithi132 ceph-mon[35283]: pgmap v390: 129 pgs: 129 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 316 KiB/s wr, 21 op/s 2024-05-27T07:25:29.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:29 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:25:29.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:29 smithi043 ceph-mon[31531]: pgmap v390: 129 pgs: 129 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 316 KiB/s wr, 21 op/s 2024-05-27T07:25:29.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:29 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:25:29.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:29 smithi179 ceph-mon[35888]: pgmap v390: 129 pgs: 129 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 316 KiB/s wr, 21 op/s 2024-05-27T07:25:29.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:29 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:25:30.235 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64361 0.59 MB/sec execute 454 sec latency 57.121 ms 2024-05-27T07:25:31.235 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64435 0.59 MB/sec execute 455 sec latency 62.386 ms 2024-05-27T07:25:31.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:31 smithi132 ceph-mon[35283]: pgmap v391: 129 pgs: 129 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 316 KiB/s wr, 21 op/s 2024-05-27T07:25:31.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:31 smithi043 ceph-mon[31531]: pgmap v391: 129 pgs: 129 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 316 KiB/s wr, 21 op/s 2024-05-27T07:25:31.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:31 smithi179 ceph-mon[35888]: pgmap v391: 129 pgs: 129 active+clean; 118 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 316 KiB/s wr, 21 op/s 2024-05-27T07:25:32.235 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64496 0.59 MB/sec execute 456 sec latency 60.797 ms 2024-05-27T07:25:33.235 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64625 0.59 MB/sec execute 457 sec latency 64.693 ms 2024-05-27T07:25:33.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:33 smithi132 ceph-mon[35283]: pgmap v392: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 473 KiB/s wr, 37 op/s 2024-05-27T07:25:33.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:33 smithi043 ceph-mon[31531]: pgmap v392: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 473 KiB/s wr, 37 op/s 2024-05-27T07:25:33.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:33 smithi179 ceph-mon[35888]: pgmap v392: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 473 KiB/s wr, 37 op/s 2024-05-27T07:25:34.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64739 0.59 MB/sec execute 458 sec latency 58.337 ms 2024-05-27T07:25:35.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64828 0.59 MB/sec execute 459 sec latency 53.131 ms 2024-05-27T07:25:35.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:35 smithi132 ceph-mon[35283]: pgmap v393: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 316 KiB/s wr, 26 op/s 2024-05-27T07:25:35.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:35 smithi043 ceph-mon[31531]: pgmap v393: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 316 KiB/s wr, 26 op/s 2024-05-27T07:25:35.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:35 smithi179 ceph-mon[35888]: pgmap v393: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 316 KiB/s wr, 26 op/s 2024-05-27T07:25:36.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64941 0.59 MB/sec execute 460 sec latency 54.132 ms 2024-05-27T07:25:36.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:25:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:37.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 65069 0.59 MB/sec execute 461 sec latency 54.531 ms 2024-05-27T07:25:37.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:37 smithi132 ceph-mon[35283]: pgmap v394: 129 pgs: 129 active+clean; 119 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 484 KiB/s wr, 49 op/s 2024-05-27T07:25:37.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:37 smithi043 ceph-mon[31531]: pgmap v394: 129 pgs: 129 active+clean; 119 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 484 KiB/s wr, 49 op/s 2024-05-27T07:25:37.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:37 smithi179 ceph-mon[35888]: pgmap v394: 129 pgs: 129 active+clean; 119 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 484 KiB/s wr, 49 op/s 2024-05-27T07:25:38.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 65253 0.59 MB/sec execute 462 sec latency 55.064 ms 2024-05-27T07:25:39.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.152+0000 7f98cba00700 1 -- 172.21.15.43:0/1578531393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98c4108730 msgr2=0x7f98c4110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.152+0000 7f98cba00700 1 --2- 172.21.15.43:0/1578531393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98c4108730 0x7f98c4110090 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b0d0 tx=0x7f98bc00b3e0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.152+0000 7f98cba00700 1 -- 172.21.15.43:0/1578531393 shutdown_connections 2024-05-27T07:25:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.152+0000 7f98cba00700 1 --2- 172.21.15.43:0/1578531393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98c4108730 0x7f98c4110090 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.152+0000 7f98cba00700 1 --2- 172.21.15.43:0/1578531393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98c41034b0 0x7f98c41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.152+0000 7f98cba00700 1 --2- 172.21.15.43:0/1578531393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98c4107de0 0x7f98c41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.152+0000 7f98cba00700 1 -- 172.21.15.43:0/1578531393 >> 172.21.15.43:0/1578531393 conn(0x7f98c40fedb0 msgr2=0x7f98c41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.152+0000 7f98cba00700 1 -- 172.21.15.43:0/1578531393 shutdown_connections 2024-05-27T07:25:39.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.153+0000 7f98cba00700 1 -- 172.21.15.43:0/1578531393 wait complete. 2024-05-27T07:25:39.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.154+0000 7f98cba00700 1 Processor -- start 2024-05-27T07:25:39.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.154+0000 7f98cba00700 1 -- start start 2024-05-27T07:25:39.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.154+0000 7f98cba00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98c41034b0 0x7f98c41a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.154+0000 7f98cba00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98c4107de0 0x7f98c41a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.154+0000 7f98cba00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98c4108730 0x7f98c419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.155+0000 7f98cba00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41143b0 con 0x7f98c4108730 2024-05-27T07:25:39.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.155+0000 7f98cba00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4114230 con 0x7f98c41034b0 2024-05-27T07:25:39.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.155+0000 7f98cba00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4114530 con 0x7f98c4107de0 2024-05-27T07:25:39.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.155+0000 7f98c9a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98c4108730 0x7f98c419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.155+0000 7f98c9a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98c4108730 0x7f98c419ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48584/0 (socket says 172.21.15.43:48584) 2024-05-27T07:25:39.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.155+0000 7f98c9a00700 1 -- 172.21.15.43:0/481605651 learned_addr learned my addr 172.21.15.43:0/481605651 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:39.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.155+0000 7f98c9000700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98c41034b0 0x7f98c41a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.155+0000 7f98c3e00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98c4107de0 0x7f98c41a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.155+0000 7f98c9a00700 1 -- 172.21.15.43:0/481605651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98c4107de0 msgr2=0x7f98c41a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.155+0000 7f98c9a00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98c4107de0 0x7f98c41a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.156+0000 7f98c9a00700 1 -- 172.21.15.43:0/481605651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98c41034b0 msgr2=0x7f98c41a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.156+0000 7f98c9a00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98c41034b0 0x7f98c41a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.156+0000 7f98c9a00700 1 -- 172.21.15.43:0/481605651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c419b280 con 0x7f98c4108730 2024-05-27T07:25:39.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.156+0000 7f98c9a00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98c4108730 0x7f98c419ab50 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f98bc007d10 tx=0x7f98bc007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.156+0000 7f98c1600700 1 -- 172.21.15.43:0/481605651 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc01f070 con 0x7f98c4108730 2024-05-27T07:25:39.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.156+0000 7f98cba00700 1 -- 172.21.15.43:0/481605651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c419b510 con 0x7f98c4108730 2024-05-27T07:25:39.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.156+0000 7f98c1600700 1 -- 172.21.15.43:0/481605651 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98bc003e10 con 0x7f98c4108730 2024-05-27T07:25:39.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.156+0000 7f98c1600700 1 -- 172.21.15.43:0/481605651 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc01b740 con 0x7f98c4108730 2024-05-27T07:25:39.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.157+0000 7f98cba00700 1 -- 172.21.15.43:0/481605651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c41a7770 con 0x7f98c4108730 2024-05-27T07:25:39.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.160+0000 7f98c1600700 1 -- 172.21.15.43:0/481605651 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f98bc022070 con 0x7f98c4108730 2024-05-27T07:25:39.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.161+0000 7f98da62fb80 1 Processor -- start 2024-05-27T07:25:39.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.162+0000 7f98da62fb80 1 -- start start 2024-05-27T07:25:39.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.162+0000 7f98c1600700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98ac0b2a70 0x7f98ac0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.162+0000 7f98c1600700 1 -- 172.21.15.43:0/481605651 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f98bc00f020 con 0x7f98c4108730 2024-05-27T07:25:39.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.162+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b017f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.162+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2b01d30 0x5608b2afb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.162+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2afbdb0 0x5608b2afc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.162+0000 7f98c3e00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98ac0b2a70 0x7f98ac0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.162+0000 7f98da62fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5608b2a29e40 con 0x5608b297ce40 2024-05-27T07:25:39.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.162+0000 7f98da62fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5608b2a199e0 con 0x5608b2afbdb0 2024-05-27T07:25:39.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.162+0000 7f98da62fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5608b2a1da20 con 0x5608b2b01d30 2024-05-27T07:25:39.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c9000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c9000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b017f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48590/0 (socket says 172.21.15.43:48590) 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c9000700 1 -- 172.21.15.43:0/867568116 learned_addr learned my addr 172.21.15.43:0/867568116 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c3e00700 1 --2- 172.21.15.43:0/867568116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2b01d30 0x5608b2afb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c9a00700 1 --2- 172.21.15.43:0/867568116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2afbdb0 0x5608b2afc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c9000700 1 -- 172.21.15.43:0/867568116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2b01d30 msgr2=0x5608b2afb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c9000700 1 --2- 172.21.15.43:0/867568116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2b01d30 0x5608b2afb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c9000700 1 -- 172.21.15.43:0/867568116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2afbdb0 msgr2=0x5608b2afc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c9000700 1 --2- 172.21.15.43:0/867568116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2afbdb0 0x5608b2afc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c3e00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98ac0b2a70 0x7f98ac0b4f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f98c419c280 tx=0x7f98b000bf50 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c9000700 1 -- 172.21.15.43:0/867568116 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608b29e2420 con 0x5608b297ce40 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.163+0000 7f98c9a00700 1 --2- 172.21.15.43:0/867568116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2afbdb0 0x5608b2afc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:39.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.164+0000 7f98c9000700 1 --2- 172.21.15.43:0/867568116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b017f0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f98b800bd70 tx=0x7f98b80106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cc75670b306f85 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.164+0000 7f98b6000700 1 -- 172.21.15.43:0/867568116 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b801c480 con 0x5608b297ce40 2024-05-27T07:25:39.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.164+0000 7f98b6000700 1 -- 172.21.15.43:0/867568116 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98b801cab0 con 0x5608b297ce40 2024-05-27T07:25:39.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.164+0000 7f98b6000700 1 -- 172.21.15.43:0/867568116 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b80118e0 con 0x5608b297ce40 2024-05-27T07:25:39.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.165+0000 7f98da62fb80 1 -- 172.21.15.43:0/867568116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 msgr2=0x5608b2b017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.165+0000 7f98da62fb80 1 --2- 172.21.15.43:0/867568116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b017f0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f98b800bd70 tx=0x7f98b80106b0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.165+0000 7f98da62fb80 1 -- 172.21.15.43:0/867568116 shutdown_connections 2024-05-27T07:25:39.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.165+0000 7f98da62fb80 1 --2- 172.21.15.43:0/867568116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2afbdb0 0x5608b2afc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.165+0000 7f98da62fb80 1 --2- 172.21.15.43:0/867568116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b017f0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.165+0000 7f98da62fb80 1 --2- 172.21.15.43:0/867568116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2b01d30 0x5608b2afb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.165+0000 7f98da62fb80 1 -- 172.21.15.43:0/867568116 >> 172.21.15.43:0/867568116 conn(0x5608b29dcfc0 msgr2=0x5608b2af68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.166+0000 7f98da62fb80 1 -- 172.21.15.43:0/867568116 shutdown_connections 2024-05-27T07:25:39.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.166+0000 7f98da62fb80 1 -- 172.21.15.43:0/867568116 wait complete. 2024-05-27T07:25:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.166+0000 7f98da62fb80 1 -- start start 2024-05-27T07:25:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.167+0000 7f98da62fb80 1 Processor -- start 2024-05-27T07:25:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.167+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b05540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.167+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b05a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.167+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2b01d30 0x5608b2b05fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.167+0000 7f98da62fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5608b291ffa0 con 0x5608b297ce40 2024-05-27T07:25:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.167+0000 7f98da62fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5608b29ba760 con 0x5608b2b01d30 2024-05-27T07:25:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.167+0000 7f98da62fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5608b2b08800 con 0x5608b2afbdb0 2024-05-27T07:25:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.167+0000 7f98da62fb80 10 client.0 _ll_register_callbacks cb 0x5608b29c3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:39.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.168+0000 7f98c9000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b05540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.168+0000 7f98c9000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b05540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48604/0 (socket says 172.21.15.43:48604) 2024-05-27T07:25:39.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.168+0000 7f98c9000700 1 -- 172.21.15.43:0/4070411656 learned_addr learned my addr 172.21.15.43:0/4070411656 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:39.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.168+0000 7f98c9a00700 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2b01d30 0x5608b2b05fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.168+0000 7f98c3e00700 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b05a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.168+0000 7f98c9000700 1 -- 172.21.15.43:0/4070411656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 msgr2=0x5608b2b05a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.168+0000 7f98c9000700 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b05a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.168+0000 7f98c9000700 1 -- 172.21.15.43:0/4070411656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2b01d30 msgr2=0x5608b2b05fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.168+0000 7f98c9000700 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2b01d30 0x5608b2b05fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.168+0000 7f98c9000700 1 -- 172.21.15.43:0/4070411656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608b29698d0 con 0x5608b297ce40 2024-05-27T07:25:39.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.169+0000 7f98c9000700 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b05540 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f98b801cf10 tx=0x7f98b8010cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299bde84c9eccdc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.169+0000 7f98aaa00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:39.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.169+0000 7f98aaa00700 1 -- 172.21.15.43:0/4070411656 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8004030 con 0x5608b297ce40 2024-05-27T07:25:39.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.169+0000 7f98aaa00700 1 -- 172.21.15.43:0/4070411656 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f98b80041d0 con 0x5608b297ce40 2024-05-27T07:25:39.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.169+0000 7f98da62fb80 1 -- 172.21.15.43:0/4070411656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98b801cab0 con 0x5608b297ce40 2024-05-27T07:25:39.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.169+0000 7f98aaa00700 1 -- 172.21.15.43:0/4070411656 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f98b802f050 con 0x5608b297ce40 2024-05-27T07:25:39.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.169+0000 7f98da62fb80 10 client.17139 fetch_fsmap learned FSMap version 108 2024-05-27T07:25:39.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.169+0000 7f98da62fb80 1 -- 172.21.15.43:0/4070411656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5608b29ea060 con 0x5608b297ce40 2024-05-27T07:25:39.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.170+0000 7f98aaa00700 1 -- 172.21.15.43:0/4070411656 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98b801cab0 con 0x5608b297ce40 2024-05-27T07:25:39.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.170+0000 7f98aaa00700 1 -- 172.21.15.43:0/4070411656 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f98b8011580 con 0x5608b297ce40 2024-05-27T07:25:39.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.170+0000 7f98da62fb80 10 client.17139 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:25:39.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.170+0000 7f98da62fb80 10 client.17139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:39.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.170+0000 7f98da62fb80 20 client.17139 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:39.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.170+0000 7f98da62fb80 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5608b2b19de0 0x5608b2b1c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.170+0000 7f98da62fb80 4 client.17139 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:39.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.170+0000 7f98da62fb80 1 -- 172.21.15.43:0/4070411656 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608b2a2bd90 con 0x5608b2b19de0 2024-05-27T07:25:39.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.171+0000 7f98c3e00700 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5608b2b19de0 0x5608b2b1c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.172+0000 7f98aaa00700 10 client.17139 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.172+0000 7f98c3e00700 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5608b2b19de0 0x5608b2b1c2b0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a76c17cab58aaff server_cookie=cdb581db57ee3f72 in_seq=0 out_seq=0 2024-05-27T07:25:39.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98aaa00700 1 -- 172.21.15.43:0/4070411656 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5608b2a2bd90 con 0x5608b2b19de0 2024-05-27T07:25:39.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98aaa00700 10 client.17139 handle_command_reply: tid=0 2024-05-27T07:25:39.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 client.17139 shutdown 2024-05-27T07:25:39.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 -- 172.21.15.43:0/4070411656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 msgr2=0x5608b2b05540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b05540 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f98b801cf10 tx=0x7f98b8010cc0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 -- 172.21.15.43:0/4070411656 shutdown_connections 2024-05-27T07:25:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2b01d30 0x5608b2b05fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5608b2b19de0 0x5608b2b1c2b0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b05540 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 --2- 172.21.15.43:0/4070411656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b05a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 -- 172.21.15.43:0/4070411656 >> 172.21.15.43:0/4070411656 conn(0x5608b29dcfc0 msgr2=0x5608b2af6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98aaa00700 0 client.17139 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 -- 172.21.15.43:0/4070411656 shutdown_connections 2024-05-27T07:25:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.176+0000 7f98da62fb80 1 -- 172.21.15.43:0/4070411656 wait complete. 2024-05-27T07:25:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.177+0000 7f98da62fb80 20 client.17139 trim_cache size 0 max 16384 2024-05-27T07:25:39.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.200+0000 7f98da62fb80 1 Processor -- start 2024-05-27T07:25:39.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.200+0000 7f98da62fb80 1 -- start start 2024-05-27T07:25:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.201+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b297ce40 0x5608b2afb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.201+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b04520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.201+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b2b01d30 0x5608b2affc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.201+0000 7f98da62fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41af010 con 0x5608b2b01d30 2024-05-27T07:25:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.201+0000 7f98da62fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41acce0 con 0x5608b297ce40 2024-05-27T07:25:39.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.201+0000 7f98da62fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41ae940 con 0x5608b2afbdb0 2024-05-27T07:25:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.201+0000 7f98c9a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b2b01d30 0x5608b2affc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.201+0000 7f98c9a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b2b01d30 0x5608b2affc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48618/0 (socket says 172.21.15.43:48618) 2024-05-27T07:25:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.201+0000 7f98c9a00700 1 -- 172.21.15.43:0/1618034830 learned_addr learned my addr 172.21.15.43:0/1618034830 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.202+0000 7f98c9000700 1 --2- 172.21.15.43:0/1618034830 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b297ce40 0x5608b2afb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.202+0000 7f98c3e00700 1 --2- 172.21.15.43:0/1618034830 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b04520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.202+0000 7f98c9a00700 1 -- 172.21.15.43:0/1618034830 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 msgr2=0x5608b2b04520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.202+0000 7f98c9a00700 1 --2- 172.21.15.43:0/1618034830 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b04520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.202+0000 7f98c9a00700 1 -- 172.21.15.43:0/1618034830 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b297ce40 msgr2=0x5608b2afb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.202+0000 7f98c9a00700 1 --2- 172.21.15.43:0/1618034830 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b297ce40 0x5608b2afb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.202+0000 7f98c9a00700 1 -- 172.21.15.43:0/1618034830 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608b2b1c7e0 con 0x5608b2b01d30 2024-05-27T07:25:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.202+0000 7f98c9000700 1 --2- 172.21.15.43:0/1618034830 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b297ce40 0x5608b2afb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.202+0000 7f98c9a00700 1 --2- 172.21.15.43:0/1618034830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b2b01d30 0x5608b2affc40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0aae50 tx=0x7f98bc0aa730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c175e96f428c9e04 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.202+0000 7f98b5600700 1 -- 172.21.15.43:0/1618034830 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0af270 con 0x5608b2b01d30 2024-05-27T07:25:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.203+0000 7f98b5600700 1 -- 172.21.15.43:0/1618034830 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98bc0b3c90 con 0x5608b2b01d30 2024-05-27T07:25:39.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.203+0000 7f98b5600700 1 -- 172.21.15.43:0/1618034830 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0b3800 con 0x5608b2b01d30 2024-05-27T07:25:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.205+0000 7f98da62fb80 1 -- 172.21.15.43:0/1618034830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b2b01d30 msgr2=0x5608b2affc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.205+0000 7f98da62fb80 1 --2- 172.21.15.43:0/1618034830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b2b01d30 0x5608b2affc40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0aae50 tx=0x7f98bc0aa730 comp rx=0 tx=0).stop 2024-05-27T07:25:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.205+0000 7f98da62fb80 1 -- 172.21.15.43:0/1618034830 shutdown_connections 2024-05-27T07:25:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.205+0000 7f98da62fb80 1 --2- 172.21.15.43:0/1618034830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b2b01d30 0x5608b2affc40 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.205+0000 7f98da62fb80 1 --2- 172.21.15.43:0/1618034830 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b297ce40 0x5608b2afb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.205+0000 7f98da62fb80 1 --2- 172.21.15.43:0/1618034830 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b04520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.205+0000 7f98da62fb80 1 -- 172.21.15.43:0/1618034830 >> 172.21.15.43:0/1618034830 conn(0x5608b29dcfc0 msgr2=0x5608b2b05320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.205+0000 7f98da62fb80 1 -- 172.21.15.43:0/1618034830 shutdown_connections 2024-05-27T07:25:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.206+0000 7f98da62fb80 1 -- 172.21.15.43:0/1618034830 wait complete. 2024-05-27T07:25:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.206+0000 7f98da62fb80 1 -- start start 2024-05-27T07:25:39.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.206+0000 7f98da62fb80 1 Processor -- start 2024-05-27T07:25:39.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.207+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b11f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.207+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b12460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.207+0000 7f98da62fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2b01d30 0x5608b2b20a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.207+0000 7f98da62fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41adae0 con 0x5608b297ce40 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.207+0000 7f98da62fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41ad3b0 con 0x5608b2b01d30 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.207+0000 7f98da62fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41ae210 con 0x5608b2afbdb0 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.207+0000 7f98da62fb80 10 client.0 _ll_register_callbacks cb 0x5608b29c3c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.207+0000 7f98c9000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b11f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.207+0000 7f98c9000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b11f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48634/0 (socket says 172.21.15.43:48634) 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.207+0000 7f98c9000700 1 -- 172.21.15.43:0/3149201676 learned_addr learned my addr 172.21.15.43:0/3149201676 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.208+0000 7f98c3e00700 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b12460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.208+0000 7f98c9000700 1 -- 172.21.15.43:0/3149201676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 msgr2=0x5608b2b12460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.208+0000 7f98c9000700 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b12460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.208+0000 7f98c9000700 1 -- 172.21.15.43:0/3149201676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2b01d30 msgr2=0x5608b2b20a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.208+0000 7f98c9000700 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2b01d30 0x5608b2b20a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.208+0000 7f98c9000700 1 -- 172.21.15.43:0/3149201676 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608b2b12a70 con 0x5608b297ce40 2024-05-27T07:25:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.208+0000 7f98c3e00700 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b12460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.208+0000 7f98c9000700 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b11f20 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f98b8005d90 tx=0x7f98b800bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d780da5f3b016ba6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.208+0000 7f98aaa00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.209+0000 7f98aaa00700 1 -- 172.21.15.43:0/3149201676 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8003e60 con 0x5608b297ce40 2024-05-27T07:25:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.209+0000 7f98da62fb80 1 -- 172.21.15.43:0/3149201676 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98bc0b3c90 con 0x5608b297ce40 2024-05-27T07:25:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.209+0000 7f98aaa00700 1 -- 172.21.15.43:0/3149201676 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f98b801c750 con 0x5608b297ce40 2024-05-27T07:25:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.209+0000 7f98aaa00700 1 -- 172.21.15.43:0/3149201676 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f98b801c950 con 0x5608b297ce40 2024-05-27T07:25:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.209+0000 7f98da62fb80 10 client.17145 fetch_fsmap learned FSMap version 108 2024-05-27T07:25:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.209+0000 7f98da62fb80 1 -- 172.21.15.43:0/3149201676 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5608b2b83e50 con 0x5608b297ce40 2024-05-27T07:25:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.210+0000 7f98aaa00700 1 -- 172.21.15.43:0/3149201676 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98bc0b3c90 con 0x5608b297ce40 2024-05-27T07:25:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.210+0000 7f98aaa00700 1 -- 172.21.15.43:0/3149201676 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f98b8003710 con 0x5608b297ce40 2024-05-27T07:25:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.210+0000 7f98da62fb80 10 client.17145 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:25:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.210+0000 7f98da62fb80 10 client.17145 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:25:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.210+0000 7f98da62fb80 20 client.17145 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.210+0000 7f98da62fb80 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5608b2b84490 0x5608b2b868b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.210+0000 7f98da62fb80 4 client.17145 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:25:39.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.210+0000 7f98da62fb80 1 -- 172.21.15.43:0/3149201676 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5608b2a25570 con 0x5608b2b84490 2024-05-27T07:25:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.211+0000 7f98c3e00700 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5608b2b84490 0x5608b2b868b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.212+0000 7f98c3e00700 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5608b2b84490 0x5608b2b868b0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d468f0fd19c535f server_cookie=619442e41317ae3e in_seq=0 out_seq=0 2024-05-27T07:25:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.212+0000 7f98aaa00700 10 client.17145 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.212+0000 7f98aaa00700 1 -- 172.21.15.43:0/3149201676 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5608b2a25570 con 0x5608b2b84490 2024-05-27T07:25:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.212+0000 7f98aaa00700 10 client.17145 handle_command_reply: tid=0 2024-05-27T07:25:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 client.17145 shutdown 2024-05-27T07:25:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 -- 172.21.15.43:0/3149201676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 msgr2=0x5608b2b11f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b11f20 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f98b8005d90 tx=0x7f98b800bc70 comp rx=0 tx=0).stop 2024-05-27T07:25:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 -- 172.21.15.43:0/3149201676 shutdown_connections 2024-05-27T07:25:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5608b2b01d30 0x5608b2b20a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5608b2b84490 0x5608b2b868b0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5608b297ce40 0x5608b2b11f20 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 --2- 172.21.15.43:0/3149201676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5608b2afbdb0 0x5608b2b12460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 -- 172.21.15.43:0/3149201676 >> 172.21.15.43:0/3149201676 conn(0x5608b29dcfc0 msgr2=0x5608b2b05d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98aaa00700 0 client.17145 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:25:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 -- 172.21.15.43:0/3149201676 shutdown_connections 2024-05-27T07:25:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 1 -- 172.21.15.43:0/3149201676 wait complete. 2024-05-27T07:25:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.213+0000 7f98da62fb80 20 client.17145 trim_cache size 0 max 16384 2024-05-27T07:25:39.204 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:25:39.205 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:25:39.205 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:25:39.205 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:25:39.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 -- 172.21.15.43:0/481605651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98ac0b2a70 msgr2=0x7f98ac0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98ac0b2a70 0x7f98ac0b4f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f98c419c280 tx=0x7f98b000bf50 comp rx=0 tx=0).stop 2024-05-27T07:25:39.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 -- 172.21.15.43:0/481605651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98c4108730 msgr2=0x7f98c419ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98c4108730 0x7f98c419ab50 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f98bc007d10 tx=0x7f98bc007d40 comp rx=0 tx=0).stop 2024-05-27T07:25:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 -- 172.21.15.43:0/481605651 shutdown_connections 2024-05-27T07:25:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98ac0b2a70 0x7f98ac0b4f80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f98c4108730 0x7f98c419ab50 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f98c41034b0 0x7f98c41a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 --2- 172.21.15.43:0/481605651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98c4107de0 0x7f98c41a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 -- 172.21.15.43:0/481605651 >> 172.21.15.43:0/481605651 conn(0x7f98c40fedb0 msgr2=0x7f98c41075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 -- 172.21.15.43:0/481605651 shutdown_connections 2024-05-27T07:25:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:39.214+0000 7f98cba00700 1 -- 172.21.15.43:0/481605651 wait complete. 2024-05-27T07:25:39.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 65356 0.59 MB/sec execute 463 sec latency 61.203 ms 2024-05-27T07:25:39.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:39 smithi132 ceph-mon[35283]: pgmap v395: 129 pgs: 129 active+clean; 119 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 325 KiB/s wr, 39 op/s 2024-05-27T07:25:39.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:39 smithi043 ceph-mon[31531]: pgmap v395: 129 pgs: 129 active+clean; 119 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 325 KiB/s wr, 39 op/s 2024-05-27T07:25:39.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:39 smithi179 ceph-mon[35888]: pgmap v395: 129 pgs: 129 active+clean; 119 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 325 KiB/s wr, 39 op/s 2024-05-27T07:25:40.154 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:25:40.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:25:40.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 65462 0.59 MB/sec execute 464 sec latency 67.274 ms 2024-05-27T07:25:40.651 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:40.833 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:40 smithi043 ceph-mon[31531]: pgmap v396: 129 pgs: 129 active+clean; 119 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 325 KiB/s wr, 39 op/s 2024-05-27T07:25:40.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:40 smithi132 ceph-mon[35283]: pgmap v396: 129 pgs: 129 active+clean; 119 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 325 KiB/s wr, 39 op/s 2024-05-27T07:25:40.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:40 smithi179 ceph-mon[35888]: pgmap v396: 129 pgs: 129 active+clean; 119 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 325 KiB/s wr, 39 op/s 2024-05-27T07:25:41.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 65622 0.59 MB/sec execute 465 sec latency 52.124 ms 2024-05-27T07:25:41.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:41 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:41.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:41 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:41.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:41 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:25:42.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 65779 0.59 MB/sec execute 466 sec latency 56.017 ms 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.910+0000 7f2e4e600700 1 -- 172.21.15.43:0/2962054879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e481045e0 msgr2=0x7f2e481049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.910+0000 7f2e4e600700 1 --2- 172.21.15.43:0/2962054879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e481045e0 0x7f2e481049f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f2e380099c0 tx=0x7f2e38009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.910+0000 7f2e4e600700 1 -- 172.21.15.43:0/2962054879 shutdown_connections 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.911+0000 7f2e4e600700 1 --2- 172.21.15.43:0/2962054879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e48109860 0x7f2e481111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.911+0000 7f2e4e600700 1 --2- 172.21.15.43:0/2962054879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e481045e0 0x7f2e481049f0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.911+0000 7f2e4e600700 1 --2- 172.21.15.43:0/2962054879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e48108f10 0x7f2e48109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.911+0000 7f2e4e600700 1 -- 172.21.15.43:0/2962054879 >> 172.21.15.43:0/2962054879 conn(0x7f2e48100030 msgr2=0x7f2e48102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.911+0000 7f2e4e600700 1 -- 172.21.15.43:0/2962054879 shutdown_connections 2024-05-27T07:25:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.911+0000 7f2e4e600700 1 -- 172.21.15.43:0/2962054879 wait complete. 2024-05-27T07:25:42.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:42 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:25:42.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:42 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:25:42.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:42 smithi043 ceph-mon[31531]: pgmap v397: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 522 KiB/s wr, 58 op/s 2024-05-27T07:25:42.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:42 smithi043 ceph-mon[31531]: pgmap v398: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 437 KiB/s wr, 51 op/s 2024-05-27T07:25:42.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:42 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.912+0000 7f2e4e600700 1 Processor -- start 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.913+0000 7f2e4e600700 1 -- start start 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.913+0000 7f2e4e600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e481045e0 0x7f2e4819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.913+0000 7f2e4e600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e48108f10 0x7f2e4819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.913+0000 7f2e4e600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e48109860 0x7f2e481a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.913+0000 7f2e4e600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2e48114340 con 0x7f2e48109860 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.913+0000 7f2e4e600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2e481141c0 con 0x7f2e48108f10 2024-05-27T07:25:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.913+0000 7f2e4e600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2e481144c0 con 0x7f2e481045e0 2024-05-27T07:25:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e47e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e48109860 0x7f2e481a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e47e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e48109860 0x7f2e481a1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48654/0 (socket says 172.21.15.43:48654) 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e47e00700 1 -- 172.21.15.43:0/2445771915 learned_addr learned my addr 172.21.15.43:0/2445771915 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e47400700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e481045e0 0x7f2e4819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e46a00700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e48108f10 0x7f2e4819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e47e00700 1 -- 172.21.15.43:0/2445771915 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e481045e0 msgr2=0x7f2e4819cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e47e00700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e481045e0 0x7f2e4819cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e47e00700 1 -- 172.21.15.43:0/2445771915 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e48108f10 msgr2=0x7f2e4819d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e47e00700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e48108f10 0x7f2e4819d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e47e00700 1 -- 172.21.15.43:0/2445771915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e481a1ea0 con 0x7f2e48109860 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e47400700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e481045e0 0x7f2e4819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.914+0000 7f2e46a00700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e48108f10 0x7f2e4819d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.915+0000 7f2e47e00700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e48109860 0x7f2e481a1800 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f2e40007ee0 tx=0x7f2e4000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.915+0000 7f2e3fe00700 1 -- 172.21.15.43:0/2445771915 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4000dbd0 con 0x7f2e48109860 2024-05-27T07:25:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.915+0000 7f2e3fe00700 1 -- 172.21.15.43:0/2445771915 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e4000dd70 con 0x7f2e48109860 2024-05-27T07:25:42.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.915+0000 7f2e4e600700 1 -- 172.21.15.43:0/2445771915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e481a2190 con 0x7f2e48109860 2024-05-27T07:25:42.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.915+0000 7f2e3fe00700 1 -- 172.21.15.43:0/2445771915 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e400058d0 con 0x7f2e48109860 2024-05-27T07:25:42.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.915+0000 7f2e4e600700 1 -- 172.21.15.43:0/2445771915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e481a9ab0 con 0x7f2e48109860 2024-05-27T07:25:42.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.917+0000 7f2e4e600700 1 -- 172.21.15.43:0/2445771915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e480512b0 con 0x7f2e48109860 2024-05-27T07:25:42.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.919+0000 7f2e3fe00700 1 -- 172.21.15.43:0/2445771915 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f2e4001a020 con 0x7f2e48109860 2024-05-27T07:25:42.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.921+0000 7f2e3fe00700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2e2c0b2a70 0x7f2e2c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.921+0000 7f2e3fe00700 1 -- 172.21.15.43:0/2445771915 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f2e400e2820 con 0x7f2e48109860 2024-05-27T07:25:42.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.921+0000 7f2e47400700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2e2c0b2a70 0x7f2e2c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:42.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.922+0000 7f2e47400700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2e2c0b2a70 0x7f2e2c0b4f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e380058e0 tx=0x7f2e38005ab0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:42.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:42.925+0000 7f2e3fe00700 1 -- 172.21.15.43:0/2445771915 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2e40010070 con 0x7f2e48109860 2024-05-27T07:25:42.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:42 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:25:42.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:42 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:25:42.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:42 smithi179 ceph-mon[35888]: pgmap v397: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 522 KiB/s wr, 58 op/s 2024-05-27T07:25:42.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:42 smithi179 ceph-mon[35888]: pgmap v398: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 437 KiB/s wr, 51 op/s 2024-05-27T07:25:42.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:42 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:25:43.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:42 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:25:43.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:42 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:25:43.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:42 smithi132 ceph-mon[35283]: pgmap v397: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 522 KiB/s wr, 58 op/s 2024-05-27T07:25:43.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:42 smithi132 ceph-mon[35283]: pgmap v398: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 437 KiB/s wr, 51 op/s 2024-05-27T07:25:43.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:42 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:25:43.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.236+0000 7f2e4e600700 1 -- 172.21.15.43:0/2445771915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e481a9c80 con 0x7f2e48109860 2024-05-27T07:25:43.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.238+0000 7f2e3fe00700 1 -- 172.21.15.43:0/2445771915 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+6962 (secure 0 0 0) 0x7f2e400a9dd0 con 0x7f2e48109860 2024-05-27T07:25:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:25:43.228 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:25:28.442486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34448,34484,34424,34403,24484]},"id":1}]} 2024-05-27T07:25:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.241+0000 7f2e4e600700 1 -- 172.21.15.43:0/2445771915 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2e2c0b2a70 msgr2=0x7f2e2c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.241+0000 7f2e4e600700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2e2c0b2a70 0x7f2e2c0b4f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e380058e0 tx=0x7f2e38005ab0 comp rx=0 tx=0).stop 2024-05-27T07:25:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.241+0000 7f2e4e600700 1 -- 172.21.15.43:0/2445771915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e48109860 msgr2=0x7f2e481a1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:43.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.241+0000 7f2e4e600700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e48109860 0x7f2e481a1800 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f2e40007ee0 tx=0x7f2e4000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:25:43.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.241+0000 7f2e4e600700 1 -- 172.21.15.43:0/2445771915 shutdown_connections 2024-05-27T07:25:43.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.241+0000 7f2e4e600700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2e2c0b2a70 0x7f2e2c0b4f80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:43.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.241+0000 7f2e4e600700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e48109860 0x7f2e481a1800 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:43.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.241+0000 7f2e4e600700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e481045e0 0x7f2e4819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:43.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.241+0000 7f2e4e600700 1 --2- 172.21.15.43:0/2445771915 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e48108f10 0x7f2e4819d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:43.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.241+0000 7f2e4e600700 1 -- 172.21.15.43:0/2445771915 >> 172.21.15.43:0/2445771915 conn(0x7f2e48100030 msgr2=0x7f2e4810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:43.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.242+0000 7f2e4e600700 1 -- 172.21.15.43:0/2445771915 shutdown_connections 2024-05-27T07:25:43.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:43.242+0000 7f2e4e600700 1 -- 172.21.15.43:0/2445771915 wait complete. 2024-05-27T07:25:43.233 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 108 2024-05-27T07:25:43.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 65934 0.59 MB/sec execute 467 sec latency 70.589 ms 2024-05-27T07:25:43.859 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:43 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2445771915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:43.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:43 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2445771915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:44.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:25:44.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:43 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2445771915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:25:44.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 66070 0.58 MB/sec execute 468 sec latency 61.538 ms 2024-05-27T07:25:44.539 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:44.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:44 smithi043 ceph-mon[31531]: pgmap v399: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 437 KiB/s wr, 51 op/s 2024-05-27T07:25:44.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:44 smithi179 ceph-mon[35888]: pgmap v399: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 437 KiB/s wr, 51 op/s 2024-05-27T07:25:45.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:44 smithi132 ceph-mon[35283]: pgmap v399: 129 pgs: 129 active+clean; 120 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 437 KiB/s wr, 51 op/s 2024-05-27T07:25:45.237 INFO:tasks.workunit.client.0.smithi043.stdout: 1 66233 0.59 MB/sec execute 469 sec latency 55.746 ms 2024-05-27T07:25:46.236 INFO:tasks.workunit.client.0.smithi043.stdout: 1 66306 0.58 MB/sec execute 470 sec latency 52.231 ms 2024-05-27T07:25:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.914+0000 7f9459a00700 1 -- 172.21.15.43:0/1974876852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9454108f10 msgr2=0x7f9454109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.914+0000 7f9459a00700 1 --2- 172.21.15.43:0/1974876852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9454108f10 0x7f9454109320 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f943c0099c0 tx=0x7f943c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.914+0000 7f9459a00700 1 -- 172.21.15.43:0/1974876852 shutdown_connections 2024-05-27T07:25:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.914+0000 7f9459a00700 1 --2- 172.21.15.43:0/1974876852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9454109860 0x7f94541111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.914+0000 7f9459a00700 1 --2- 172.21.15.43:0/1974876852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94541045e0 0x7f94541049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.914+0000 7f9459a00700 1 --2- 172.21.15.43:0/1974876852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9454108f10 0x7f9454109320 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.914+0000 7f9459a00700 1 -- 172.21.15.43:0/1974876852 >> 172.21.15.43:0/1974876852 conn(0x7f9454100030 msgr2=0x7f9454102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.914+0000 7f9459a00700 1 -- 172.21.15.43:0/1974876852 shutdown_connections 2024-05-27T07:25:46.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.914+0000 7f9459a00700 1 -- 172.21.15.43:0/1974876852 wait complete. 2024-05-27T07:25:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.916+0000 7f9459a00700 1 Processor -- start 2024-05-27T07:25:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.916+0000 7f9459a00700 1 -- start start 2024-05-27T07:25:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f9459a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94541045e0 0x7f945419ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f9459a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9454108f10 0x7f945419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f9459a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9454109860 0x7f945419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f9459a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9454114370 con 0x7f9454109860 2024-05-27T07:25:46.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f9459a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f94541141f0 con 0x7f9454108f10 2024-05-27T07:25:46.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f9459a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f94541144f0 con 0x7f94541045e0 2024-05-27T07:25:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f9453400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9454109860 0x7f945419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f9453400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9454109860 0x7f945419f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51534/0 (socket says 172.21.15.43:51534) 2024-05-27T07:25:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f9453400700 1 -- 172.21.15.43:0/4025192338 learned_addr learned my addr 172.21.15.43:0/4025192338 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f9452a00700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94541045e0 0x7f945419ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.917+0000 7f944a000700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9454108f10 0x7f945419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.918+0000 7f9453400700 1 -- 172.21.15.43:0/4025192338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94541045e0 msgr2=0x7f945419ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:46.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.918+0000 7f9453400700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94541045e0 0x7f945419ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.918+0000 7f9453400700 1 -- 172.21.15.43:0/4025192338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9454108f10 msgr2=0x7f945419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:46.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.918+0000 7f9453400700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9454108f10 0x7f945419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.918+0000 7f9453400700 1 -- 172.21.15.43:0/4025192338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945419fc50 con 0x7f9454109860 2024-05-27T07:25:46.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.918+0000 7f9453400700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9454109860 0x7f945419f5b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f944c00d980 tx=0x7f944c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.919+0000 7f944be00700 1 -- 172.21.15.43:0/4025192338 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944c017070 con 0x7f9454109860 2024-05-27T07:25:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.919+0000 7f944be00700 1 -- 172.21.15.43:0/4025192338 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f944c005020 con 0x7f9454109860 2024-05-27T07:25:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.919+0000 7f9459a00700 1 -- 172.21.15.43:0/4025192338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f945419ff40 con 0x7f9454109860 2024-05-27T07:25:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.919+0000 7f9459a00700 1 -- 172.21.15.43:0/4025192338 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94541a7860 con 0x7f9454109860 2024-05-27T07:25:46.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.919+0000 7f944be00700 1 -- 172.21.15.43:0/4025192338 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944c00f980 con 0x7f9454109860 2024-05-27T07:25:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.923+0000 7f944be00700 1 -- 172.21.15.43:0/4025192338 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f944c00fb20 con 0x7f9454109860 2024-05-27T07:25:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.924+0000 7f94684d8b80 1 Processor -- start 2024-05-27T07:25:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.924+0000 7f94684d8b80 1 -- start start 2024-05-27T07:25:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ad439f0 0x56098adda750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ac57e40 0x56098addac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098addb290 0x56098ade7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f94684d8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56098ad04e40 con 0x56098ad439f0 2024-05-27T07:25:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f94684d8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56098acf49e0 con 0x56098addb290 2024-05-27T07:25:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f94684d8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56098acf8a20 con 0x56098ac57e40 2024-05-27T07:25:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f944be00700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94380b2a70 0x7f94380b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f944a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ad439f0 0x56098adda750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f944be00700 1 -- 172.21.15.43:0/4025192338 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f944c0e2610 con 0x7f9454109860 2024-05-27T07:25:46.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f9453400700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94380b2a70 0x7f94380b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f944a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ad439f0 0x56098adda750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51542/0 (socket says 172.21.15.43:51542) 2024-05-27T07:25:46.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.925+0000 7f944a000700 1 -- 172.21.15.43:0/4173993455 learned_addr learned my addr 172.21.15.43:0/4173993455 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:46.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.926+0000 7f9452a00700 1 --2- 172.21.15.43:0/4173993455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ac57e40 0x56098addac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.926+0000 7f944a000700 1 --2- 172.21.15.43:0/4173993455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098addb290 0x56098ade7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.926+0000 7f944a000700 1 -- 172.21.15.43:0/4173993455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ac57e40 msgr2=0x56098addac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:46.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.926+0000 7f944a000700 1 --2- 172.21.15.43:0/4173993455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ac57e40 0x56098addac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.926+0000 7f944a000700 1 -- 172.21.15.43:0/4173993455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098addb290 msgr2=0x56098ade7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:46.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.926+0000 7f944a000700 1 --2- 172.21.15.43:0/4173993455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098addb290 0x56098ade7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.926+0000 7f944a000700 1 -- 172.21.15.43:0/4173993455 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56098acbd420 con 0x56098ad439f0 2024-05-27T07:25:46.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.926+0000 7f944a000700 1 --2- 172.21.15.43:0/4173993455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ad439f0 0x56098adda750 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f945406c670 tx=0x7f943c009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391825595c18563c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:46.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.927+0000 7f9453400700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94380b2a70 0x7f94380b4f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f944c0a95a0 tx=0x7f944c0a54d0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:46.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.927+0000 7f9443e00700 1 -- 172.21.15.43:0/4173993455 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c005350 con 0x56098ad439f0 2024-05-27T07:25:46.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.927+0000 7f9443e00700 1 -- 172.21.15.43:0/4173993455 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f943c0054f0 con 0x56098ad439f0 2024-05-27T07:25:46.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.927+0000 7f9443e00700 1 -- 172.21.15.43:0/4173993455 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c003aa0 con 0x56098ad439f0 2024-05-27T07:25:46.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.928+0000 7f94684d8b80 1 -- 172.21.15.43:0/4173993455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ad439f0 msgr2=0x56098adda750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:46.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.928+0000 7f94684d8b80 1 --2- 172.21.15.43:0/4173993455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ad439f0 0x56098adda750 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f945406c670 tx=0x7f943c009f10 comp rx=0 tx=0).stop 2024-05-27T07:25:46.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.928+0000 7f94684d8b80 1 -- 172.21.15.43:0/4173993455 shutdown_connections 2024-05-27T07:25:46.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.928+0000 7f94684d8b80 1 --2- 172.21.15.43:0/4173993455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098addb290 0x56098ade7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.928+0000 7f94684d8b80 1 --2- 172.21.15.43:0/4173993455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ad439f0 0x56098adda750 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.928+0000 7f94684d8b80 1 --2- 172.21.15.43:0/4173993455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ac57e40 0x56098addac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.928+0000 7f94684d8b80 1 -- 172.21.15.43:0/4173993455 >> 172.21.15.43:0/4173993455 conn(0x56098acf1390 msgr2=0x56098add2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:46.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.928+0000 7f94684d8b80 1 -- 172.21.15.43:0/4173993455 shutdown_connections 2024-05-27T07:25:46.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.928+0000 7f94684d8b80 1 -- 172.21.15.43:0/4173993455 wait complete. 2024-05-27T07:25:46.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.929+0000 7f94684d8b80 1 -- start start 2024-05-27T07:25:46.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.929+0000 7f94684d8b80 1 Processor -- start 2024-05-27T07:25:46.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.930+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ac57e40 0x56098ade4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.930+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ad439f0 0x56098ade50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.930+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098addb290 0x56098ade5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.930+0000 7f94684d8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56098abfafa0 con 0x56098ac57e40 2024-05-27T07:25:46.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.930+0000 7f94684d8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56098ac95760 con 0x56098ad439f0 2024-05-27T07:25:46.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.930+0000 7f94684d8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56098ade7f40 con 0x56098addb290 2024-05-27T07:25:46.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.930+0000 7f94684d8b80 10 client.0 _ll_register_callbacks cb 0x56098ac9ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:46.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.930+0000 7f944a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ac57e40 0x56098ade4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.930+0000 7f944a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ac57e40 0x56098ade4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51554/0 (socket says 172.21.15.43:51554) 2024-05-27T07:25:46.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.930+0000 7f944a000700 1 -- 172.21.15.43:0/2825121621 learned_addr learned my addr 172.21.15.43:0/2825121621 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:46.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.931+0000 7f9452a00700 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ad439f0 0x56098ade50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.931+0000 7f944a000700 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098addb290 0x56098ade5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.931+0000 7f944a000700 1 -- 172.21.15.43:0/2825121621 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098addb290 msgr2=0x56098ade5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.931+0000 7f944a000700 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098addb290 0x56098ade5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.931+0000 7f944a000700 1 -- 172.21.15.43:0/2825121621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ad439f0 msgr2=0x56098ade50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.931+0000 7f944a000700 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ad439f0 0x56098ade50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.931+0000 7f944a000700 1 -- 172.21.15.43:0/2825121621 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56098ac448d0 con 0x56098ac57e40 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.932+0000 7f944a000700 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ac57e40 0x56098ade4bb0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f943c009990 tx=0x7f943c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4b969465e94d31 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.932+0000 7f9440c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.932+0000 7f9440c00700 1 -- 172.21.15.43:0/2825121621 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c003720 con 0x56098ac57e40 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.932+0000 7f9440c00700 1 -- 172.21.15.43:0/2825121621 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f943c021030 con 0x56098ac57e40 2024-05-27T07:25:46.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.932+0000 7f94684d8b80 1 -- 172.21.15.43:0/2825121621 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f943c0054f0 con 0x56098ac57e40 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.932+0000 7f9440c00700 1 -- 172.21.15.43:0/2825121621 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f943c0038c0 con 0x56098ac57e40 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.932+0000 7f94684d8b80 10 client.17163 fetch_fsmap learned FSMap version 108 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.932+0000 7f94684d8b80 1 -- 172.21.15.43:0/2825121621 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x56098acc5060 con 0x56098ac57e40 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.933+0000 7f9440c00700 1 -- 172.21.15.43:0/2825121621 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f943c0054f0 con 0x56098ac57e40 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.933+0000 7f9440c00700 1 -- 172.21.15.43:0/2825121621 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f943c00f990 con 0x56098ac57e40 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.933+0000 7f94684d8b80 10 client.17163 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.933+0000 7f94684d8b80 10 client.17163 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.933+0000 7f94684d8b80 20 client.17163 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.933+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56098adf4df0 0x56098adf7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.933+0000 7f94684d8b80 4 client.17163 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:46.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.933+0000 7f94684d8b80 1 -- 172.21.15.43:0/2825121621 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56098ad06d90 con 0x56098adf4df0 2024-05-27T07:25:46.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.934+0000 7f9452a00700 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56098adf4df0 0x56098adf7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.935+0000 7f9452a00700 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56098adf4df0 0x56098adf7230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=76f1816df70c760c server_cookie=58f5927ae0f2aa67 in_seq=0 out_seq=0 2024-05-27T07:25:46.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.935+0000 7f9440c00700 10 client.17163 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:25:46.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f9440c00700 1 -- 172.21.15.43:0/2825121621 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56098ad06d90 con 0x56098adf4df0 2024-05-27T07:25:46.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f9440c00700 10 client.17163 handle_command_reply: tid=0 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f94684d8b80 1 client.17163 shutdown 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f94684d8b80 1 -- 172.21.15.43:0/2825121621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ac57e40 msgr2=0x56098ade4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ac57e40 0x56098ade4bb0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f943c009990 tx=0x7f943c003680 comp rx=0 tx=0).stop 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f94684d8b80 1 -- 172.21.15.43:0/2825121621 shutdown_connections 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56098adf4df0 0x56098adf7230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098addb290 0x56098ade5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098ac57e40 0x56098ade4bb0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2825121621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ad439f0 0x56098ade50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f94684d8b80 1 -- 172.21.15.43:0/2825121621 >> 172.21.15.43:0/2825121621 conn(0x56098acf1390 msgr2=0x56098adda870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.937+0000 7f9440c00700 0 client.17163 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.938+0000 7f94684d8b80 1 -- 172.21.15.43:0/2825121621 shutdown_connections 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.938+0000 7f94684d8b80 1 -- 172.21.15.43:0/2825121621 wait complete. 2024-05-27T07:25:46.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.938+0000 7f94684d8b80 20 client.17163 trim_cache size 0 max 16384 2024-05-27T07:25:46.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.957+0000 7f94684d8b80 1 Processor -- start 2024-05-27T07:25:46.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.957+0000 7f94684d8b80 1 -- start start 2024-05-27T07:25:46.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ac57e40 0x56098add24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ad439f0 0x56098ade3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addb290 0x56098ade0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f94684d8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56098addacf0 con 0x56098addb290 2024-05-27T07:25:46.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f94684d8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f943000fa80 con 0x56098ac57e40 2024-05-27T07:25:46.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f94684d8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f943000d750 con 0x56098ad439f0 2024-05-27T07:25:46.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f944a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addb290 0x56098ade0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f944a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addb290 0x56098ade0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51564/0 (socket says 172.21.15.43:51564) 2024-05-27T07:25:46.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f944a000700 1 -- 172.21.15.43:0/2452915129 learned_addr learned my addr 172.21.15.43:0/2452915129 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:46.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f9452a00700 1 --2- 172.21.15.43:0/2452915129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ad439f0 0x56098ade3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.958+0000 7f944a000700 1 --2- 172.21.15.43:0/2452915129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ac57e40 0x56098add24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.959+0000 7f944a000700 1 -- 172.21.15.43:0/2452915129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ad439f0 msgr2=0x56098ade3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:46.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.959+0000 7f944a000700 1 --2- 172.21.15.43:0/2452915129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ad439f0 0x56098ade3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.959+0000 7f944a000700 1 -- 172.21.15.43:0/2452915129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ac57e40 msgr2=0x56098add24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:46.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.959+0000 7f944a000700 1 --2- 172.21.15.43:0/2452915129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ac57e40 0x56098add24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.959+0000 7f944a000700 1 -- 172.21.15.43:0/2452915129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56098add3840 con 0x56098addb290 2024-05-27T07:25:46.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.959+0000 7f944a000700 1 --2- 172.21.15.43:0/2452915129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addb290 0x56098ade0360 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f943c00fd00 tx=0x7f943c009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6cb4fde20d1aea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:46.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.959+0000 7f9443400700 1 -- 172.21.15.43:0/2452915129 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c0036f0 con 0x56098addb290 2024-05-27T07:25:46.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.959+0000 7f9443400700 1 -- 172.21.15.43:0/2452915129 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f943c027450 con 0x56098addb290 2024-05-27T07:25:46.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.959+0000 7f9443400700 1 -- 172.21.15.43:0/2452915129 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c003a60 con 0x56098addb290 2024-05-27T07:25:46.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.960+0000 7f94684d8b80 1 -- 172.21.15.43:0/2452915129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addb290 msgr2=0x56098ade0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:46.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.960+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2452915129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addb290 0x56098ade0360 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f943c00fd00 tx=0x7f943c009cd0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.961+0000 7f94684d8b80 1 -- 172.21.15.43:0/2452915129 shutdown_connections 2024-05-27T07:25:46.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.961+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2452915129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addb290 0x56098ade0360 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.961+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2452915129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ac57e40 0x56098add24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.961+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2452915129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ad439f0 0x56098ade3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.961+0000 7f94684d8b80 1 -- 172.21.15.43:0/2452915129 >> 172.21.15.43:0/2452915129 conn(0x56098acf1390 msgr2=0x56098ade4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:46.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.961+0000 7f94684d8b80 1 -- 172.21.15.43:0/2452915129 shutdown_connections 2024-05-27T07:25:46.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.961+0000 7f94684d8b80 1 -- 172.21.15.43:0/2452915129 wait complete. 2024-05-27T07:25:46.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.961+0000 7f94684d8b80 1 -- start start 2024-05-27T07:25:46.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.962+0000 7f94684d8b80 1 Processor -- start 2024-05-27T07:25:46.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.962+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ac57e40 0x56098adde870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.962+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ad439f0 0x56098addedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.962+0000 7f94684d8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addf2f0 0x56098adec260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.962+0000 7f94684d8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f943000de20 con 0x56098addf2f0 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.962+0000 7f94684d8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f943000f3b0 con 0x56098ac57e40 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.962+0000 7f94684d8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f943000e550 con 0x56098ad439f0 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.963+0000 7f94684d8b80 10 client.0 _ll_register_callbacks cb 0x56098addb1e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.963+0000 7f944a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addf2f0 0x56098adec260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.963+0000 7f944a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addf2f0 0x56098adec260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51578/0 (socket says 172.21.15.43:51578) 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.963+0000 7f944a000700 1 -- 172.21.15.43:0/2760682534 learned_addr learned my addr 172.21.15.43:0/2760682534 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.963+0000 7f944a000700 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ac57e40 0x56098adde870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.963+0000 7f9452a00700 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ad439f0 0x56098addedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.964+0000 7f944a000700 1 -- 172.21.15.43:0/2760682534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ad439f0 msgr2=0x56098addedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.964+0000 7f944a000700 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ad439f0 0x56098addedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.964+0000 7f944a000700 1 -- 172.21.15.43:0/2760682534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ac57e40 msgr2=0x56098adde870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.964+0000 7f944a000700 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ac57e40 0x56098adde870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.964+0000 7f944a000700 1 -- 172.21.15.43:0/2760682534 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56098adec7a0 con 0x56098addf2f0 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.964+0000 7f944a000700 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addf2f0 0x56098adec260 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f943c006010 tx=0x7f943c009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4ee3658b2e7372 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.964+0000 7f9440c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.964+0000 7f9440c00700 1 -- 172.21.15.43:0/2760682534 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c0036a0 con 0x56098addf2f0 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.964+0000 7f9440c00700 1 -- 172.21.15.43:0/2760682534 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f943c005460 con 0x56098addf2f0 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.964+0000 7f94684d8b80 1 -- 172.21.15.43:0/2760682534 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f943c027450 con 0x56098addf2f0 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f9440c00700 1 -- 172.21.15.43:0/2760682534 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f943c003ba0 con 0x56098addf2f0 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f94684d8b80 10 client.17169 fetch_fsmap learned FSMap version 108 2024-05-27T07:25:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f94684d8b80 1 -- 172.21.15.43:0/2760682534 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x56098adeda00 con 0x56098addf2f0 2024-05-27T07:25:46.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f9440c00700 1 -- 172.21.15.43:0/2760682534 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f943c027450 con 0x56098addf2f0 2024-05-27T07:25:46.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f9440c00700 1 -- 172.21.15.43:0/2760682534 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f943c0346a0 con 0x56098addf2f0 2024-05-27T07:25:46.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f94684d8b80 10 client.17169 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:25:46.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f94684d8b80 10 client.17169 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:25:46.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f94684d8b80 20 client.17169 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:46.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56098ae5f8c0 0x56098ae61ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:46.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f94684d8b80 4 client.17169 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:25:46.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.965+0000 7f94684d8b80 1 -- 172.21.15.43:0/2760682534 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56098ad00570 con 0x56098ae5f8c0 2024-05-27T07:25:46.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.966+0000 7f9452a00700 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56098ae5f8c0 0x56098ae61ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.967+0000 7f9452a00700 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56098ae5f8c0 0x56098ae61ce0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=268cfa70c8e8a0dc server_cookie=c127fd054c0c0910 in_seq=0 out_seq=0 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.967+0000 7f9440c00700 10 client.17169 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f9440c00700 1 -- 172.21.15.43:0/2760682534 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56098ad00570 con 0x56098ae5f8c0 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f9440c00700 10 client.17169 handle_command_reply: tid=0 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 client.17169 shutdown 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 -- 172.21.15.43:0/2760682534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addf2f0 msgr2=0x56098adec260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addf2f0 0x56098adec260 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f943c006010 tx=0x7f943c009390 comp rx=0 tx=0).stop 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 -- 172.21.15.43:0/2760682534 shutdown_connections 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56098ae5f8c0 0x56098ae61ce0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f9440c00700 0 client.17169 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56098addf2f0 0x56098adec260 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56098ac57e40 0x56098adde870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 --2- 172.21.15.43:0/2760682534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56098ad439f0 0x56098addedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 -- 172.21.15.43:0/2760682534 >> 172.21.15.43:0/2760682534 conn(0x56098acf1390 msgr2=0x56098adf0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 -- 172.21.15.43:0/2760682534 shutdown_connections 2024-05-27T07:25:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.968+0000 7f94684d8b80 1 -- 172.21.15.43:0/2760682534 wait complete. 2024-05-27T07:25:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f94684d8b80 20 client.17169 trim_cache size 0 max 16384 2024-05-27T07:25:46.960 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:25:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f9459a00700 1 -- 172.21.15.43:0/4025192338 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94380b2a70 msgr2=0x7f94380b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f9459a00700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94380b2a70 0x7f94380b4f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f944c0a95a0 tx=0x7f944c0a54d0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f9459a00700 1 -- 172.21.15.43:0/4025192338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9454109860 msgr2=0x7f945419f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f9459a00700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9454109860 0x7f945419f5b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f944c00d980 tx=0x7f944c00dc90 comp rx=0 tx=0).stop 2024-05-27T07:25:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f9459a00700 1 -- 172.21.15.43:0/4025192338 shutdown_connections 2024-05-27T07:25:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f9459a00700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f94380b2a70 0x7f94380b4f80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f9459a00700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9454109860 0x7f945419f5b0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f9459a00700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f94541045e0 0x7f945419ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f9459a00700 1 --2- 172.21.15.43:0/4025192338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9454108f10 0x7f945419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.969+0000 7f9459a00700 1 -- 172.21.15.43:0/4025192338 >> 172.21.15.43:0/4025192338 conn(0x7f9454100030 msgr2=0x7f945410a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:46.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.970+0000 7f9459a00700 1 -- 172.21.15.43:0/4025192338 shutdown_connections 2024-05-27T07:25:46.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:46.970+0000 7f9459a00700 1 -- 172.21.15.43:0/4025192338 wait complete. 2024-05-27T07:25:47.237 INFO:tasks.workunit.client.0.smithi043.stdout: 1 66430 0.59 MB/sec execute 471 sec latency 54.532 ms 2024-05-27T07:25:47.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:47 smithi132 ceph-mon[35283]: pgmap v400: 129 pgs: 129 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 448 KiB/s wr, 40 op/s 2024-05-27T07:25:47.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:47 smithi043 ceph-mon[31531]: pgmap v400: 129 pgs: 129 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 448 KiB/s wr, 40 op/s 2024-05-27T07:25:47.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:47 smithi179 ceph-mon[35888]: pgmap v400: 129 pgs: 129 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 448 KiB/s wr, 40 op/s 2024-05-27T07:25:47.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:25:48.237 INFO:tasks.workunit.client.0.smithi043.stdout: 1 66620 0.59 MB/sec execute 472 sec latency 55.569 ms 2024-05-27T07:25:48.311 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:49.237 INFO:tasks.workunit.client.0.smithi043.stdout: 1 66785 0.59 MB/sec execute 473 sec latency 292.650 ms 2024-05-27T07:25:49.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:49 smithi132 ceph-mon[35283]: pgmap v401: 129 pgs: 129 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 448 KiB/s wr, 40 op/s 2024-05-27T07:25:49.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:49 smithi043 ceph-mon[31531]: pgmap v401: 129 pgs: 129 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 448 KiB/s wr, 40 op/s 2024-05-27T07:25:49.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:49 smithi179 ceph-mon[35888]: pgmap v401: 129 pgs: 129 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 448 KiB/s wr, 40 op/s 2024-05-27T07:25:50.237 INFO:tasks.workunit.client.0.smithi043.stdout: 1 66891 0.59 MB/sec execute 474 sec latency 51.226 ms 2024-05-27T07:25:50.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.639+0000 7fb260800700 1 -- 172.21.15.43:0/854317088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb258111bc0 msgr2=0x7fb258111fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:50.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.639+0000 7fb260800700 1 --2- 172.21.15.43:0/854317088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb258111bc0 0x7fb258111fd0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009990 tx=0x7fb24c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.639+0000 7fb260800700 1 -- 172.21.15.43:0/854317088 shutdown_connections 2024-05-27T07:25:50.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.639+0000 7fb260800700 1 --2- 172.21.15.43:0/854317088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb25810b270 0x7fb2581198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.639+0000 7fb260800700 1 --2- 172.21.15.43:0/854317088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb258111bc0 0x7fb258111fd0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.639+0000 7fb260800700 1 --2- 172.21.15.43:0/854317088 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb25810a920 0x7fb25810ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.639+0000 7fb260800700 1 -- 172.21.15.43:0/854317088 >> 172.21.15.43:0/854317088 conn(0x7fb25806f4c0 msgr2=0x7fb25806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:50.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.639+0000 7fb260800700 1 -- 172.21.15.43:0/854317088 shutdown_connections 2024-05-27T07:25:50.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.639+0000 7fb260800700 1 -- 172.21.15.43:0/854317088 wait complete. 2024-05-27T07:25:50.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.640+0000 7fb260800700 1 Processor -- start 2024-05-27T07:25:50.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.641+0000 7fb260800700 1 -- start start 2024-05-27T07:25:50.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.641+0000 7fb260800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb25810a920 0x7fb2581a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.641+0000 7fb260800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25810b270 0x7fb2581a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.641+0000 7fb260800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb258111bc0 0x7fb2581a7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.641+0000 7fb260800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb25811cbe0 con 0x7fb25810b270 2024-05-27T07:25:50.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.641+0000 7fb260800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb25811ca60 con 0x7fb25810a920 2024-05-27T07:25:50.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.641+0000 7fb260800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb25811cd60 con 0x7fb258111bc0 2024-05-27T07:25:50.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25810b270 0x7fb2581a3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25810b270 0x7fb2581a3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51600/0 (socket says 172.21.15.43:51600) 2024-05-27T07:25:50.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25d400700 1 -- 172.21.15.43:0/3772991219 learned_addr learned my addr 172.21.15.43:0/3772991219 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:50.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25de00700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb25810a920 0x7fb2581a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25e800700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb258111bc0 0x7fb2581a7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25d400700 1 -- 172.21.15.43:0/3772991219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb258111bc0 msgr2=0x7fb2581a7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:50.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25d400700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb258111bc0 0x7fb2581a7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25d400700 1 -- 172.21.15.43:0/3772991219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb25810a920 msgr2=0x7fb2581a35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:50.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25d400700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb25810a920 0x7fb2581a35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25d400700 1 -- 172.21.15.43:0/3772991219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2581a8450 con 0x7fb25810b270 2024-05-27T07:25:50.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.642+0000 7fb25de00700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb25810a920 0x7fb2581a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:50.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.643+0000 7fb25d400700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25810b270 0x7fb2581a3b70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb24800d980 tx=0x7fb24800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:50.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.643+0000 7fb252a00700 1 -- 172.21.15.43:0/3772991219 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb248017070 con 0x7fb25810b270 2024-05-27T07:25:50.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.643+0000 7fb252a00700 1 -- 172.21.15.43:0/3772991219 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb248004830 con 0x7fb25810b270 2024-05-27T07:25:50.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.643+0000 7fb260800700 1 -- 172.21.15.43:0/3772991219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2581a8740 con 0x7fb25810b270 2024-05-27T07:25:50.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.643+0000 7fb252a00700 1 -- 172.21.15.43:0/3772991219 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2480053a0 con 0x7fb25810b270 2024-05-27T07:25:50.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.643+0000 7fb260800700 1 -- 172.21.15.43:0/3772991219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2581b0060 con 0x7fb25810b270 2024-05-27T07:25:50.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.650+0000 7fb26f2adb80 1 Processor -- start 2024-05-27T07:25:50.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.650+0000 7fb26f2adb80 1 -- start start 2024-05-27T07:25:50.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.650+0000 7fb252a00700 1 -- 172.21.15.43:0/3772991219 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb248005540 con 0x7fb25810b270 2024-05-27T07:25:50.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.650+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.651+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98a8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.651+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.651+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560aa97d6e40 con 0x560aa98159f0 2024-05-27T07:25:50.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.651+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560aa97c69e0 con 0x560aa9729e40 2024-05-27T07:25:50.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.651+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560aa97caa20 con 0x560aa98b7510 2024-05-27T07:25:50.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.651+0000 7fb25e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.651+0000 7fb25e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51606/0 (socket says 172.21.15.43:51606) 2024-05-27T07:25:50.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.651+0000 7fb25e800700 1 -- 172.21.15.43:0/2399738989 learned_addr learned my addr 172.21.15.43:0/2399738989 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:50.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.651+0000 7fb25d400700 1 --2- 172.21.15.43:0/2399738989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.651+0000 7fb25de00700 1 --2- 172.21.15.43:0/2399738989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98a8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.652+0000 7fb25e800700 1 -- 172.21.15.43:0/2399738989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 msgr2=0x560aa98b9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:50.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.652+0000 7fb25e800700 1 --2- 172.21.15.43:0/2399738989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.652+0000 7fb25e800700 1 -- 172.21.15.43:0/2399738989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 msgr2=0x560aa98a8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:50.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.652+0000 7fb25e800700 1 --2- 172.21.15.43:0/2399738989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98a8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.652+0000 7fb25e800700 1 -- 172.21.15.43:0/2399738989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560aa978f420 con 0x560aa98159f0 2024-05-27T07:25:50.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.652+0000 7fb25e800700 1 --2- 172.21.15.43:0/2399738989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a83f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb254009f20 tx=0x7fb254009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca1a75aff5e5053 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:50.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.652+0000 7fb247400700 1 -- 172.21.15.43:0/2399738989 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25401a070 con 0x560aa98159f0 2024-05-27T07:25:50.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.652+0000 7fb247400700 1 -- 172.21.15.43:0/2399738989 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb254015410 con 0x560aa98159f0 2024-05-27T07:25:50.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.653+0000 7fb247400700 1 -- 172.21.15.43:0/2399738989 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25400a660 con 0x560aa98159f0 2024-05-27T07:25:50.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.653+0000 7fb252a00700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb2400b29f0 0x7fb2400b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.653+0000 7fb252a00700 1 -- 172.21.15.43:0/3772991219 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7fb2480e29d0 con 0x7fb25810b270 2024-05-27T07:25:50.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.653+0000 7fb25de00700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb2400b29f0 0x7fb2400b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.654+0000 7fb26f2adb80 1 -- 172.21.15.43:0/2399738989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 msgr2=0x560aa98a83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:50.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.654+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/2399738989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a83f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb254009f20 tx=0x7fb254009f50 comp rx=0 tx=0).stop 2024-05-27T07:25:50.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.654+0000 7fb26f2adb80 1 -- 172.21.15.43:0/2399738989 shutdown_connections 2024-05-27T07:25:50.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.654+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/2399738989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.654+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/2399738989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a83f0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.654+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/2399738989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98a8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.654+0000 7fb26f2adb80 1 -- 172.21.15.43:0/2399738989 >> 172.21.15.43:0/2399738989 conn(0x560aa97c3390 msgr2=0x560aa98a44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:50.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.654+0000 7fb26f2adb80 1 -- 172.21.15.43:0/2399738989 shutdown_connections 2024-05-27T07:25:50.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.654+0000 7fb26f2adb80 1 -- 172.21.15.43:0/2399738989 wait complete. 2024-05-27T07:25:50.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.654+0000 7fb25de00700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb2400b29f0 0x7fb2400b4f00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x560aa98a4ec0 tx=0x7fb24c00b560 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:50.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.655+0000 7fb26f2adb80 1 -- start start 2024-05-27T07:25:50.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.655+0000 7fb26f2adb80 1 Processor -- start 2024-05-27T07:25:50.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.656+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa9729e40 0x560aa98b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.656+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa98159f0 0x560aa98b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.656+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.656+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560aa96ccfa0 con 0x560aa9729e40 2024-05-27T07:25:50.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.656+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560aa9767760 con 0x560aa98159f0 2024-05-27T07:25:50.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.656+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560aa98ba020 con 0x560aa98b7510 2024-05-27T07:25:50.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.656+0000 7fb26f2adb80 10 client.0 _ll_register_callbacks cb 0x560aa9770c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:50.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.656+0000 7fb25de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa9729e40 0x560aa98b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.656+0000 7fb25de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa9729e40 0x560aa98b6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51614/0 (socket says 172.21.15.43:51614) 2024-05-27T07:25:50.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.656+0000 7fb25de00700 1 -- 172.21.15.43:0/1456189430 learned_addr learned my addr 172.21.15.43:0/1456189430 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.657+0000 7fb25d400700 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa98159f0 0x560aa98b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.657+0000 7fb25e800700 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.657+0000 7fb25de00700 1 -- 172.21.15.43:0/1456189430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 msgr2=0x560aa98b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.657+0000 7fb25de00700 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.657+0000 7fb25de00700 1 -- 172.21.15.43:0/1456189430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa98159f0 msgr2=0x560aa98b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.657+0000 7fb25de00700 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa98159f0 0x560aa98b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.657+0000 7fb25de00700 1 -- 172.21.15.43:0/1456189430 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560aa97168d0 con 0x560aa9729e40 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.658+0000 7fb25de00700 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa9729e40 0x560aa98b6df0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb25810bc70 tx=0x7fb24c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed51dcc97da8b450 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.658+0000 7fb23be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.658+0000 7fb23be00700 1 -- 172.21.15.43:0/1456189430 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c033060 con 0x560aa9729e40 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.658+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1456189430 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb254015410 con 0x560aa9729e40 2024-05-27T07:25:50.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.658+0000 7fb23be00700 1 -- 172.21.15.43:0/1456189430 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7fb24c00bbe0 con 0x560aa9729e40 2024-05-27T07:25:50.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.658+0000 7fb23be00700 1 -- 172.21.15.43:0/1456189430 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fb24c03a050 con 0x560aa9729e40 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.658+0000 7fb26f2adb80 10 client.17181 fetch_fsmap learned FSMap version 108 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.658+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1456189430 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x560aa9797060 con 0x560aa9729e40 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.659+0000 7fb23be00700 1 -- 172.21.15.43:0/1456189430 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb254015410 con 0x560aa9729e40 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.659+0000 7fb23be00700 1 -- 172.21.15.43:0/1456189430 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7fb24c02d030 con 0x560aa9729e40 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.659+0000 7fb26f2adb80 10 client.17181 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.659+0000 7fb26f2adb80 10 client.17181 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.659+0000 7fb26f2adb80 20 client.17181 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.659+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560aa98cb7b0 0x560aa98cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.659+0000 7fb26f2adb80 4 client.17181 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.659+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1456189430 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560aa97d8d90 con 0x560aa98cb7b0 2024-05-27T07:25:50.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.660+0000 7fb25d400700 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560aa98cb7b0 0x560aa98cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.661+0000 7fb25d400700 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560aa98cb7b0 0x560aa98cdc80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b0dc3bf6525b3b5c server_cookie=bc56aaf5ca8c1b7d in_seq=0 out_seq=0 2024-05-27T07:25:50.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.661+0000 7fb23be00700 10 client.17181 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:25:50.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.664+0000 7fb23be00700 1 -- 172.21.15.43:0/1456189430 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560aa97d8d90 con 0x560aa98cb7b0 2024-05-27T07:25:50.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.664+0000 7fb23be00700 10 client.17181 handle_command_reply: tid=0 2024-05-27T07:25:50.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.664+0000 7fb26f2adb80 1 client.17181 shutdown 2024-05-27T07:25:50.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1456189430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa9729e40 msgr2=0x560aa98b6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:50.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa9729e40 0x560aa98b6df0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb25810bc70 tx=0x7fb24c009310 comp rx=0 tx=0).stop 2024-05-27T07:25:50.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1456189430 shutdown_connections 2024-05-27T07:25:50.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa9729e40 0x560aa98b6df0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560aa98cb7b0 0x560aa98cdc80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/1456189430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa98159f0 0x560aa98b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1456189430 >> 172.21.15.43:0/1456189430 conn(0x560aa97c3390 msgr2=0x560aa98a8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:50.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb23be00700 0 client.17181 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:25:50.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1456189430 shutdown_connections 2024-05-27T07:25:50.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1456189430 wait complete. 2024-05-27T07:25:50.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.665+0000 7fb26f2adb80 20 client.17181 trim_cache size 0 max 16384 2024-05-27T07:25:50.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.685+0000 7fb26f2adb80 1 Processor -- start 2024-05-27T07:25:50.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.685+0000 7fb26f2adb80 1 -- start start 2024-05-27T07:25:50.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.685+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.685+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.686+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.686+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581b78b0 con 0x560aa98159f0 2024-05-27T07:25:50.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.686+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581b5580 con 0x560aa9729e40 2024-05-27T07:25:50.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.686+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581b71e0 con 0x560aa98b7510 2024-05-27T07:25:50.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.686+0000 7fb25d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.686+0000 7fb25d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51630/0 (socket says 172.21.15.43:51630) 2024-05-27T07:25:50.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.686+0000 7fb25d400700 1 -- 172.21.15.43:0/1929556692 learned_addr learned my addr 172.21.15.43:0/1929556692 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:50.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.686+0000 7fb25de00700 1 --2- 172.21.15.43:0/1929556692 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.686+0000 7fb25e800700 1 --2- 172.21.15.43:0/1929556692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.687+0000 7fb25d400700 1 -- 172.21.15.43:0/1929556692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 msgr2=0x560aa98b8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:50.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.687+0000 7fb25d400700 1 --2- 172.21.15.43:0/1929556692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.687+0000 7fb25d400700 1 -- 172.21.15.43:0/1929556692 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 msgr2=0x560aa98a3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:50.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.687+0000 7fb25d400700 1 --2- 172.21.15.43:0/1929556692 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98a3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.687+0000 7fb25d400700 1 -- 172.21.15.43:0/1929556692 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560aa98ae6a0 con 0x560aa98159f0 2024-05-27T07:25:50.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.687+0000 7fb25d400700 1 --2- 172.21.15.43:0/1929556692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a4410 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb24800bfd0 tx=0x7fb2480cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbad26c3db0d48f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:50.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.687+0000 7fb246a00700 1 -- 172.21.15.43:0/1929556692 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2480a9f80 con 0x560aa98159f0 2024-05-27T07:25:50.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.687+0000 7fb246a00700 1 -- 172.21.15.43:0/1929556692 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb2480a9880 con 0x560aa98159f0 2024-05-27T07:25:50.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.687+0000 7fb246a00700 1 -- 172.21.15.43:0/1929556692 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2480dff70 con 0x560aa98159f0 2024-05-27T07:25:50.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.688+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1929556692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 msgr2=0x560aa98a4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:50.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.688+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/1929556692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a4410 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb24800bfd0 tx=0x7fb2480cc040 comp rx=0 tx=0).stop 2024-05-27T07:25:50.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.688+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1929556692 shutdown_connections 2024-05-27T07:25:50.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.689+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/1929556692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98b7510 0x560aa98b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.689+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/1929556692 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.689+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/1929556692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98159f0 0x560aa98a4410 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.689+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1929556692 >> 172.21.15.43:0/1929556692 conn(0x560aa97c3390 msgr2=0x560aa98aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:50.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.689+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1929556692 shutdown_connections 2024-05-27T07:25:50.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.689+0000 7fb26f2adb80 1 -- 172.21.15.43:0/1929556692 wait complete. 2024-05-27T07:25:50.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.689+0000 7fb26f2adb80 1 -- start start 2024-05-27T07:25:50.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.689+0000 7fb26f2adb80 1 Processor -- start 2024-05-27T07:25:50.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.690+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.690+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98159f0 0x560aa98c8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.690+0000 7fb26f2adb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98b7510 0x560aa98c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.690+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581b6380 con 0x560aa98b7510 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.690+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581b5c50 con 0x560aa9729e40 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.690+0000 7fb26f2adb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581b6ab0 con 0x560aa98159f0 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.690+0000 7fb26f2adb80 10 client.0 _ll_register_callbacks cb 0x560aa9770c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.690+0000 7fb25e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98b7510 0x560aa98c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.690+0000 7fb25e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98b7510 0x560aa98c31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51642/0 (socket says 172.21.15.43:51642) 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.690+0000 7fb25e800700 1 -- 172.21.15.43:0/898959336 learned_addr learned my addr 172.21.15.43:0/898959336 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.691+0000 7fb25de00700 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.691+0000 7fb25d400700 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98159f0 0x560aa98c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.691+0000 7fb25e800700 1 -- 172.21.15.43:0/898959336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98159f0 msgr2=0x560aa98c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:50.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.691+0000 7fb25e800700 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98159f0 0x560aa98c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.691+0000 7fb25e800700 1 -- 172.21.15.43:0/898959336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 msgr2=0x560aa98c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.691+0000 7fb25e800700 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.691+0000 7fb25e800700 1 -- 172.21.15.43:0/898959336 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560aa98c3850 con 0x560aa98b7510 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.691+0000 7fb25e800700 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98b7510 0x560aa98c31a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb254018040 tx=0x7fb25400c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1dbe2de2ec4be1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.691+0000 7fb23be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.692+0000 7fb23be00700 1 -- 172.21.15.43:0/898959336 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25401a040 con 0x560aa98b7510 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.692+0000 7fb26f2adb80 1 -- 172.21.15.43:0/898959336 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2480a9880 con 0x560aa98b7510 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.692+0000 7fb23be00700 1 -- 172.21.15.43:0/898959336 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7fb254015620 con 0x560aa98b7510 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.692+0000 7fb23be00700 1 -- 172.21.15.43:0/898959336 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fb254015820 con 0x560aa98b7510 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.692+0000 7fb26f2adb80 10 client.17187 fetch_fsmap learned FSMap version 108 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.692+0000 7fb26f2adb80 1 -- 172.21.15.43:0/898959336 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x560aa9935c50 con 0x560aa98b7510 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.692+0000 7fb23be00700 1 -- 172.21.15.43:0/898959336 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb2480a9880 con 0x560aa98b7510 2024-05-27T07:25:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.693+0000 7fb23be00700 1 -- 172.21.15.43:0/898959336 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7fb25401fcf0 con 0x560aa98b7510 2024-05-27T07:25:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.693+0000 7fb26f2adb80 10 client.17187 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:25:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.693+0000 7fb26f2adb80 10 client.17187 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:25:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.693+0000 7fb26f2adb80 20 client.17187 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.693+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560aa9936310 0x560aa9938730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.693+0000 7fb26f2adb80 4 client.17187 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:25:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.693+0000 7fb26f2adb80 1 -- 172.21.15.43:0/898959336 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560aa97d2570 con 0x560aa9936310 2024-05-27T07:25:50.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.694+0000 7fb25de00700 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560aa9936310 0x560aa9938730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:50.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb25de00700 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560aa9936310 0x560aa9938730 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=960a309115cea25c server_cookie=675c285f894e3c5f in_seq=0 out_seq=0 2024-05-27T07:25:50.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb23be00700 10 client.17187 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:25:50.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb23be00700 1 -- 172.21.15.43:0/898959336 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560aa97d2570 con 0x560aa9936310 2024-05-27T07:25:50.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb23be00700 10 client.17187 handle_command_reply: tid=0 2024-05-27T07:25:50.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb26f2adb80 1 client.17187 shutdown 2024-05-27T07:25:50.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb26f2adb80 1 -- 172.21.15.43:0/898959336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98b7510 msgr2=0x560aa98c31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:50.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98b7510 0x560aa98c31a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb254018040 tx=0x7fb25400c2e0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb26f2adb80 1 -- 172.21.15.43:0/898959336 shutdown_connections 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560aa98b7510 0x560aa98c31a0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560aa9936310 0x560aa9938730 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560aa9729e40 0x560aa98c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb26f2adb80 1 --2- 172.21.15.43:0/898959336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560aa98159f0 0x560aa98c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb23be00700 0 client.17187 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.695+0000 7fb26f2adb80 1 -- 172.21.15.43:0/898959336 >> 172.21.15.43:0/898959336 conn(0x560aa97c3390 msgr2=0x560aa98ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.696+0000 7fb26f2adb80 1 -- 172.21.15.43:0/898959336 shutdown_connections 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.696+0000 7fb26f2adb80 1 -- 172.21.15.43:0/898959336 wait complete. 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.696+0000 7fb26f2adb80 20 client.17187 trim_cache size 0 max 16384 2024-05-27T07:25:50.688 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:25:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 -- 172.21.15.43:0/3772991219 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb2400b29f0 msgr2=0x7fb2400b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb2400b29f0 0x7fb2400b4f00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x560aa98a4ec0 tx=0x7fb24c00b560 comp rx=0 tx=0).stop 2024-05-27T07:25:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 -- 172.21.15.43:0/3772991219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25810b270 msgr2=0x7fb2581a3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25810b270 0x7fb2581a3b70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb24800d980 tx=0x7fb24800dd40 comp rx=0 tx=0).stop 2024-05-27T07:25:50.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 -- 172.21.15.43:0/3772991219 shutdown_connections 2024-05-27T07:25:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb2400b29f0 0x7fb2400b4f00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb258111bc0 0x7fb2581a7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb25810a920 0x7fb2581a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 --2- 172.21.15.43:0/3772991219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb25810b270 0x7fb2581a3b70 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 -- 172.21.15.43:0/3772991219 >> 172.21.15.43:0/3772991219 conn(0x7fb25806f4c0 msgr2=0x7fb2581106c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.697+0000 7fb260800700 1 -- 172.21.15.43:0/3772991219 shutdown_connections 2024-05-27T07:25:50.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:50.698+0000 7fb260800700 1 -- 172.21.15.43:0/3772991219 wait complete. 2024-05-27T07:25:51.239 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67152 0.59 MB/sec execute 475 sec latency 55.857 ms 2024-05-27T07:25:51.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:51 smithi043 ceph-mon[31531]: pgmap v402: 129 pgs: 129 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 448 KiB/s wr, 40 op/s 2024-05-27T07:25:51.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:51 smithi132 ceph-mon[35283]: pgmap v402: 129 pgs: 129 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 448 KiB/s wr, 40 op/s 2024-05-27T07:25:51.594 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:25:51.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:51 smithi179 ceph-mon[35888]: pgmap v402: 129 pgs: 129 active+clean; 121 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 448 KiB/s wr, 40 op/s 2024-05-27T07:25:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:52.237 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67186 0.59 MB/sec execute 476 sec latency 63.897 ms 2024-05-27T07:25:53.237 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67227 0.59 MB/sec execute 477 sec latency 54.788 ms 2024-05-27T07:25:53.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:53 smithi043 ceph-mon[31531]: pgmap v403: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 599 KiB/s wr, 40 op/s 2024-05-27T07:25:53.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:53 smithi132 ceph-mon[35283]: pgmap v403: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 599 KiB/s wr, 40 op/s 2024-05-27T07:25:53.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:53 smithi179 ceph-mon[35888]: pgmap v403: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 599 KiB/s wr, 40 op/s 2024-05-27T07:25:54.237 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67400 0.60 MB/sec execute 478 sec latency 50.123 ms 2024-05-27T07:25:54.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.426+0000 7f681f200700 1 -- 172.21.15.43:0/83819947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6820109670 msgr2=0x7f6820110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.426+0000 7f681f200700 1 --2- 172.21.15.43:0/83819947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6820109670 0x7f6820110fd0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f681800b0d0 tx=0x7f681800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.426+0000 7f681f200700 1 -- 172.21.15.43:0/83819947 shutdown_connections 2024-05-27T07:25:54.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.427+0000 7f681f200700 1 --2- 172.21.15.43:0/83819947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6820109670 0x7f6820110fd0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.427+0000 7f681f200700 1 --2- 172.21.15.43:0/83819947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68201043f0 0x7f6820104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.427+0000 7f681f200700 1 --2- 172.21.15.43:0/83819947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6820108d20 0x7f6820109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.427+0000 7f681f200700 1 -- 172.21.15.43:0/83819947 >> 172.21.15.43:0/83819947 conn(0x7f6820079d00 msgr2=0x7f682007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:54.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.427+0000 7f681f200700 1 -- 172.21.15.43:0/83819947 shutdown_connections 2024-05-27T07:25:54.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.427+0000 7f681f200700 1 -- 172.21.15.43:0/83819947 wait complete. 2024-05-27T07:25:54.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.428+0000 7f681f200700 1 Processor -- start 2024-05-27T07:25:54.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.429+0000 7f681f200700 1 -- start start 2024-05-27T07:25:54.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.429+0000 7f681f200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68201043f0 0x7f682019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.429+0000 7f681f200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6820108d20 0x7f682019d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.429+0000 7f681f200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6820109670 0x7f68201a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.429+0000 7f681f200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f68201141a0 con 0x7f6820109670 2024-05-27T07:25:54.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.429+0000 7f681f200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6820114020 con 0x7f68201043f0 2024-05-27T07:25:54.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.429+0000 7f681f200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6820114320 con 0x7f6820108d20 2024-05-27T07:25:54.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.430+0000 7f681e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6820109670 0x7f68201a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.430+0000 7f681e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6820109670 0x7f68201a1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51664/0 (socket says 172.21.15.43:51664) 2024-05-27T07:25:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.430+0000 7f681e800700 1 -- 172.21.15.43:0/3180060573 learned_addr learned my addr 172.21.15.43:0/3180060573 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.430+0000 7f681de00700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68201043f0 0x7f682019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.430+0000 7f681d400700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6820108d20 0x7f682019d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.430+0000 7f681e800700 1 -- 172.21.15.43:0/3180060573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6820108d20 msgr2=0x7f682019d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.430+0000 7f681e800700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6820108d20 0x7f682019d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.430+0000 7f681e800700 1 -- 172.21.15.43:0/3180060573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68201043f0 msgr2=0x7f682019cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.430+0000 7f681e800700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68201043f0 0x7f682019cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.430+0000 7f681e800700 1 -- 172.21.15.43:0/3180060573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68201a1b70 con 0x7f6820109670 2024-05-27T07:25:54.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.431+0000 7f681e800700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6820109670 0x7f68201a1580 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6818007d10 tx=0x7f6818007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:54.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.431+0000 7f6816a00700 1 -- 172.21.15.43:0/3180060573 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681801f070 con 0x7f6820109670 2024-05-27T07:25:54.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.431+0000 7f6816a00700 1 -- 172.21.15.43:0/3180060573 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6818003e10 con 0x7f6820109670 2024-05-27T07:25:54.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.431+0000 7f681f200700 1 -- 172.21.15.43:0/3180060573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68201a1e00 con 0x7f6820109670 2024-05-27T07:25:54.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.431+0000 7f6816a00700 1 -- 172.21.15.43:0/3180060573 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681801b960 con 0x7f6820109670 2024-05-27T07:25:54.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.431+0000 7f681f200700 1 -- 172.21.15.43:0/3180060573 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68201a22e0 con 0x7f6820109670 2024-05-27T07:25:54.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.436+0000 7f6816a00700 1 -- 172.21.15.43:0/3180060573 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6818022070 con 0x7f6820109670 2024-05-27T07:25:54.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.437+0000 7f6833744b80 1 Processor -- start 2024-05-27T07:25:54.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.437+0000 7f6833744b80 1 -- start start 2024-05-27T07:25:54.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.437+0000 7f6833744b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x56489076c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.437+0000 7f6833744b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648905ece40 0x56489076cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.437+0000 7f6833744b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648907762c0 0x5648907786e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.437+0000 7f6833744b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564890699e40 con 0x5648906d89f0 2024-05-27T07:25:54.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.437+0000 7f6833744b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5648906899e0 con 0x5648905ece40 2024-05-27T07:25:54.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.437+0000 7f6833744b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56489068da20 con 0x5648907762c0 2024-05-27T07:25:54.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.438+0000 7f681d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x56489076c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.438+0000 7f681d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x56489076c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34860/0 (socket says 172.21.15.43:34860) 2024-05-27T07:25:54.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.438+0000 7f681d400700 1 -- 172.21.15.43:0/1737964212 learned_addr learned my addr 172.21.15.43:0/1737964212 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:54.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.438+0000 7f681de00700 1 --2- 172.21.15.43:0/1737964212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648905ece40 0x56489076cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.438+0000 7f681d400700 1 --2- 172.21.15.43:0/1737964212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648907762c0 0x5648907786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.439+0000 7f681d400700 1 -- 172.21.15.43:0/1737964212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648907762c0 msgr2=0x5648907786e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:54.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.439+0000 7f681d400700 1 --2- 172.21.15.43:0/1737964212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648907762c0 0x5648907786e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.439+0000 7f681d400700 1 -- 172.21.15.43:0/1737964212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648905ece40 msgr2=0x56489076cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:54.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.439+0000 7f681d400700 1 --2- 172.21.15.43:0/1737964212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648905ece40 0x56489076cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.439+0000 7f681d400700 1 -- 172.21.15.43:0/1737964212 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564890652420 con 0x5648906d89f0 2024-05-27T07:25:54.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.439+0000 7f6816a00700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f68040b29f0 0x7f68040b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.439+0000 7f6816a00700 1 -- 172.21.15.43:0/3180060573 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f681800f020 con 0x7f6820109670 2024-05-27T07:25:54.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.439+0000 7f681d400700 1 --2- 172.21.15.43:0/1737964212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x56489076c750 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f680c005fd0 tx=0x7f680c00c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae4d54adf4e79b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:54.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.440+0000 7f680b400700 1 -- 172.21.15.43:0/1737964212 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c018030 con 0x5648906d89f0 2024-05-27T07:25:54.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.440+0000 7f681de00700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f68040b29f0 0x7f68040b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.440+0000 7f680b400700 1 -- 172.21.15.43:0/1737964212 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f680c0044b0 con 0x5648906d89f0 2024-05-27T07:25:54.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.440+0000 7f680b400700 1 -- 172.21.15.43:0/1737964212 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c005020 con 0x5648906d89f0 2024-05-27T07:25:54.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.441+0000 7f6833744b80 1 -- 172.21.15.43:0/1737964212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 msgr2=0x56489076c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.441+0000 7f681de00700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f68040b29f0 0x7f68040b4f00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x564890768970 tx=0x7f681000a380 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:54.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.441+0000 7f6833744b80 1 --2- 172.21.15.43:0/1737964212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x56489076c750 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f680c005fd0 tx=0x7f680c00c5a0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.441+0000 7f6833744b80 1 -- 172.21.15.43:0/1737964212 shutdown_connections 2024-05-27T07:25:54.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.441+0000 7f6833744b80 1 --2- 172.21.15.43:0/1737964212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648907762c0 0x5648907786e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.441+0000 7f6833744b80 1 --2- 172.21.15.43:0/1737964212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x56489076c750 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.441+0000 7f6833744b80 1 --2- 172.21.15.43:0/1737964212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648905ece40 0x56489076cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.441+0000 7f6833744b80 1 -- 172.21.15.43:0/1737964212 >> 172.21.15.43:0/1737964212 conn(0x564890686390 msgr2=0x564890767710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:54.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.441+0000 7f6833744b80 1 -- 172.21.15.43:0/1737964212 shutdown_connections 2024-05-27T07:25:54.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.441+0000 7f6833744b80 1 -- 172.21.15.43:0/1737964212 wait complete. 2024-05-27T07:25:54.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.442+0000 7f6833744b80 1 -- start start 2024-05-27T07:25:54.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.442+0000 7f6833744b80 1 Processor -- start 2024-05-27T07:25:54.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.443+0000 7f6833744b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648905ece40 0x5648907732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.443+0000 7f6833744b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x564890773810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.443+0000 7f6833744b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 0x564890773d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.443+0000 7f6833744b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56489058ffa0 con 0x5648906d89f0 2024-05-27T07:25:54.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.443+0000 7f6833744b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56489062a760 con 0x5648907762c0 2024-05-27T07:25:54.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.443+0000 7f6833744b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564890778e00 con 0x5648905ece40 2024-05-27T07:25:54.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.443+0000 7f6833744b80 10 client.0 _ll_register_callbacks cb 0x564890633c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:54.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.443+0000 7f681d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x564890773810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.443+0000 7f681d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x564890773810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34868/0 (socket says 172.21.15.43:34868) 2024-05-27T07:25:54.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.443+0000 7f681d400700 1 -- 172.21.15.43:0/3850916056 learned_addr learned my addr 172.21.15.43:0/3850916056 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:54.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.444+0000 7f681de00700 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648905ece40 0x5648907732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.444+0000 7f681e800700 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 0x564890773d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.444+0000 7f681d400700 1 -- 172.21.15.43:0/3850916056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648905ece40 msgr2=0x5648907732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:54.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.444+0000 7f681d400700 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648905ece40 0x5648907732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.444+0000 7f681d400700 1 -- 172.21.15.43:0/3850916056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 msgr2=0x564890773d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:54.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.444+0000 7f681d400700 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 0x564890773d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.444+0000 7f681d400700 1 -- 172.21.15.43:0/3850916056 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648905d98d0 con 0x5648906d89f0 2024-05-27T07:25:54.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.444+0000 7f681d400700 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x564890773810 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f680c015040 tx=0x7f680c004a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=103a8e07279af552 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:54.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.444+0000 7f67ffe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:54.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.445+0000 7f67ffe00700 1 -- 172.21.15.43:0/3850916056 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c018040 con 0x5648906d89f0 2024-05-27T07:25:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.445+0000 7f6833744b80 1 -- 172.21.15.43:0/3850916056 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f680c0044b0 con 0x5648906d89f0 2024-05-27T07:25:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.445+0000 7f67ffe00700 1 -- 172.21.15.43:0/3850916056 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f680c005220 con 0x5648906d89f0 2024-05-27T07:25:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.445+0000 7f67ffe00700 1 -- 172.21.15.43:0/3850916056 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f680c02e050 con 0x5648906d89f0 2024-05-27T07:25:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.445+0000 7f6833744b80 10 client.17199 fetch_fsmap learned FSMap version 108 2024-05-27T07:25:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.445+0000 7f6833744b80 1 -- 172.21.15.43:0/3850916056 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x56489065a060 con 0x5648906d89f0 2024-05-27T07:25:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.446+0000 7f67ffe00700 1 -- 172.21.15.43:0/3850916056 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f680c0044b0 con 0x5648906d89f0 2024-05-27T07:25:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.446+0000 7f67ffe00700 1 -- 172.21.15.43:0/3850916056 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f680c023020 con 0x5648906d89f0 2024-05-27T07:25:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.446+0000 7f6833744b80 10 client.17199 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:25:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.446+0000 7f6833744b80 10 client.17199 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:25:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.446+0000 7f6833744b80 20 client.17199 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.446+0000 7f6833744b80 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564890786070 0x5648907884e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.446+0000 7f6833744b80 4 client.17199 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.446+0000 7f6833744b80 1 -- 172.21.15.43:0/3850916056 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56489069bd90 con 0x564890786070 2024-05-27T07:25:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.447+0000 7f681de00700 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564890786070 0x5648907884e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.448+0000 7f681de00700 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564890786070 0x5648907884e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fafffc3e8c16b92a server_cookie=b34045eeb4604925 in_seq=0 out_seq=0 2024-05-27T07:25:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.448+0000 7f67ffe00700 10 client.17199 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:25:54.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f67ffe00700 1 -- 172.21.15.43:0/3850916056 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56489069bd90 con 0x564890786070 2024-05-27T07:25:54.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f67ffe00700 10 client.17199 handle_command_reply: tid=0 2024-05-27T07:25:54.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 client.17199 shutdown 2024-05-27T07:25:54.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 -- 172.21.15.43:0/3850916056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 msgr2=0x564890773810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x564890773810 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f680c015040 tx=0x7f680c004a00 comp rx=0 tx=0).stop 2024-05-27T07:25:54.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 -- 172.21.15.43:0/3850916056 shutdown_connections 2024-05-27T07:25:54.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564890786070 0x5648907884e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 0x564890773d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648905ece40 0x5648907732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 --2- 172.21.15.43:0/3850916056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648906d89f0 0x564890773810 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f67ffe00700 0 client.17199 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:25:54.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 -- 172.21.15.43:0/3850916056 >> 172.21.15.43:0/3850916056 conn(0x564890686390 msgr2=0x5648907672a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:54.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 -- 172.21.15.43:0/3850916056 shutdown_connections 2024-05-27T07:25:54.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 1 -- 172.21.15.43:0/3850916056 wait complete. 2024-05-27T07:25:54.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.451+0000 7f6833744b80 20 client.17199 trim_cache size 0 max 16384 2024-05-27T07:25:54.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.469+0000 7f6833744b80 1 Processor -- start 2024-05-27T07:25:54.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.469+0000 7f6833744b80 1 -- start start 2024-05-27T07:25:54.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.470+0000 7f6833744b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890767740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.470+0000 7f6833744b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648906d89f0 0x564890781ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.470+0000 7f6833744b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648907762c0 0x56489076e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.470+0000 7f6833744b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f68201b3100 con 0x5648905ece40 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.470+0000 7f6833744b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f68201b0dd0 con 0x5648906d89f0 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.470+0000 7f6833744b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f68201b2a30 con 0x5648907762c0 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.470+0000 7f681de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890767740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.470+0000 7f681de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890767740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34878/0 (socket says 172.21.15.43:34878) 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.470+0000 7f681de00700 1 -- 172.21.15.43:0/1420414639 learned_addr learned my addr 172.21.15.43:0/1420414639 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.471+0000 7f681d400700 1 --2- 172.21.15.43:0/1420414639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648906d89f0 0x564890781ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.471+0000 7f681e800700 1 --2- 172.21.15.43:0/1420414639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648907762c0 0x56489076e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.471+0000 7f681de00700 1 -- 172.21.15.43:0/1420414639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648907762c0 msgr2=0x56489076e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.471+0000 7f681de00700 1 --2- 172.21.15.43:0/1420414639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648907762c0 0x56489076e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.471+0000 7f681de00700 1 -- 172.21.15.43:0/1420414639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648906d89f0 msgr2=0x564890781ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.471+0000 7f681de00700 1 --2- 172.21.15.43:0/1420414639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648906d89f0 0x564890781ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.471+0000 7f681de00700 1 -- 172.21.15.43:0/1420414639 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56489076d500 con 0x5648905ece40 2024-05-27T07:25:54.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.471+0000 7f681d400700 1 --2- 172.21.15.43:0/1420414639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648906d89f0 0x564890781ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.471+0000 7f681de00700 1 --2- 172.21.15.43:0/1420414639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890767740 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x564890788c40 tx=0x7f6810017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655c6d5018a93957 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.472+0000 7f680aa00700 1 -- 172.21.15.43:0/1420414639 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681001bdd0 con 0x5648905ece40 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.472+0000 7f680aa00700 1 -- 172.21.15.43:0/1420414639 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68100288a0 con 0x5648905ece40 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.472+0000 7f680aa00700 1 -- 172.21.15.43:0/1420414639 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681000f800 con 0x5648905ece40 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.473+0000 7f6833744b80 1 -- 172.21.15.43:0/1420414639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 msgr2=0x564890767740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.473+0000 7f6833744b80 1 --2- 172.21.15.43:0/1420414639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890767740 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x564890788c40 tx=0x7f6810017000 comp rx=0 tx=0).stop 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.473+0000 7f6833744b80 1 -- 172.21.15.43:0/1420414639 shutdown_connections 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.473+0000 7f6833744b80 1 --2- 172.21.15.43:0/1420414639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648907762c0 0x56489076e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.473+0000 7f6833744b80 1 --2- 172.21.15.43:0/1420414639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890767740 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.473+0000 7f6833744b80 1 --2- 172.21.15.43:0/1420414639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648906d89f0 0x564890781ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.473+0000 7f6833744b80 1 -- 172.21.15.43:0/1420414639 >> 172.21.15.43:0/1420414639 conn(0x564890686390 msgr2=0x56489076e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.473+0000 7f6833744b80 1 -- 172.21.15.43:0/1420414639 shutdown_connections 2024-05-27T07:25:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.473+0000 7f6833744b80 1 -- 172.21.15.43:0/1420414639 wait complete. 2024-05-27T07:25:54.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.473+0000 7f6833744b80 1 -- start start 2024-05-27T07:25:54.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.474+0000 7f6833744b80 1 Processor -- start 2024-05-27T07:25:54.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.474+0000 7f6833744b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890779050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.474+0000 7f6833744b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648906d89f0 0x564890779590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.474+0000 7f6833744b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 0x564890779ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.474+0000 7f6833744b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f68201b1bd0 con 0x5648905ece40 2024-05-27T07:25:54.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.474+0000 7f6833744b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f68201b14a0 con 0x5648907762c0 2024-05-27T07:25:54.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.474+0000 7f6833744b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f68201b2300 con 0x5648906d89f0 2024-05-27T07:25:54.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.474+0000 7f6833744b80 10 client.0 _ll_register_callbacks cb 0x564890633c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:54.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890779050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890779050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34884/0 (socket says 172.21.15.43:34884) 2024-05-27T07:25:54.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681de00700 1 -- 172.21.15.43:0/2521382457 learned_addr learned my addr 172.21.15.43:0/2521382457 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681d400700 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648906d89f0 0x564890779590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681e800700 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 0x564890779ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681de00700 1 -- 172.21.15.43:0/2521382457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648906d89f0 msgr2=0x564890779590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681de00700 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648906d89f0 0x564890779590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681de00700 1 -- 172.21.15.43:0/2521382457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 msgr2=0x564890779ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681de00700 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 0x564890779ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681de00700 1 -- 172.21.15.43:0/2521382457 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648907eeb60 con 0x5648905ece40 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.475+0000 7f681e800700 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 0x564890779ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.476+0000 7f681de00700 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890779050 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f681001bf60 tx=0x7f681001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9828570f2038809a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.476+0000 7f67ffe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.476+0000 7f67ffe00700 1 -- 172.21.15.43:0/2521382457 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681003f060 con 0x5648905ece40 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.476+0000 7f67ffe00700 1 -- 172.21.15.43:0/2521382457 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f6810005040 con 0x5648905ece40 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.476+0000 7f6833744b80 1 -- 172.21.15.43:0/2521382457 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68100288a0 con 0x5648905ece40 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.476+0000 7f67ffe00700 1 -- 172.21.15.43:0/2521382457 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f6810046050 con 0x5648905ece40 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.476+0000 7f6833744b80 10 client.17205 fetch_fsmap learned FSMap version 108 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.476+0000 7f6833744b80 1 -- 172.21.15.43:0/2521382457 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x5648907efdc0 con 0x5648905ece40 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.477+0000 7f67ffe00700 1 -- 172.21.15.43:0/2521382457 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f68100288a0 con 0x5648905ece40 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.477+0000 7f67ffe00700 1 -- 172.21.15.43:0/2521382457 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f681003b760 con 0x5648905ece40 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.477+0000 7f6833744b80 10 client.17205 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.477+0000 7f6833744b80 10 client.17205 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.477+0000 7f6833744b80 20 client.17205 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.477+0000 7f6833744b80 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5648907f04a0 0x5648907f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.477+0000 7f6833744b80 4 client.17205 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:25:54.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.477+0000 7f6833744b80 1 -- 172.21.15.43:0/2521382457 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564890695570 con 0x5648907f04a0 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.478+0000 7f681d400700 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5648907f04a0 0x5648907f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.479+0000 7f681d400700 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5648907f04a0 0x5648907f28c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f58656f2b767712 server_cookie=48a0fb61df2d216d in_seq=0 out_seq=0 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.479+0000 7f67ffe00700 10 client.17205 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.479+0000 7f67ffe00700 1 -- 172.21.15.43:0/2521382457 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564890695570 con 0x5648907f04a0 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.479+0000 7f67ffe00700 10 client.17205 handle_command_reply: tid=0 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.479+0000 7f6833744b80 1 client.17205 shutdown 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.479+0000 7f6833744b80 1 -- 172.21.15.43:0/2521382457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 msgr2=0x564890779050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.479+0000 7f6833744b80 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890779050 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f681001bf60 tx=0x7f681001bf90 comp rx=0 tx=0).stop 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.479+0000 7f6833744b80 1 -- 172.21.15.43:0/2521382457 shutdown_connections 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.479+0000 7f6833744b80 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5648907f04a0 0x5648907f28c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.480+0000 7f6833744b80 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5648907762c0 0x564890779ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.480+0000 7f6833744b80 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5648905ece40 0x564890779050 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.480+0000 7f6833744b80 1 --2- 172.21.15.43:0/2521382457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5648906d89f0 0x564890779590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.480+0000 7f6833744b80 1 -- 172.21.15.43:0/2521382457 >> 172.21.15.43:0/2521382457 conn(0x564890686390 msgr2=0x56489076e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:54.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.480+0000 7f67ffe00700 0 client.17205 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:25:54.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.480+0000 7f6833744b80 1 -- 172.21.15.43:0/2521382457 shutdown_connections 2024-05-27T07:25:54.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.480+0000 7f6833744b80 1 -- 172.21.15.43:0/2521382457 wait complete. 2024-05-27T07:25:54.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.480+0000 7f6833744b80 20 client.17205 trim_cache size 0 max 16384 2024-05-27T07:25:54.471 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:25:54.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 -- 172.21.15.43:0/3180060573 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f68040b29f0 msgr2=0x7f68040b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f68040b29f0 0x7f68040b4f00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x564890768970 tx=0x7f681000a380 comp rx=0 tx=0).stop 2024-05-27T07:25:54.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 -- 172.21.15.43:0/3180060573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6820109670 msgr2=0x7f68201a1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:54.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6820109670 0x7f68201a1580 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6818007d10 tx=0x7f6818007d40 comp rx=0 tx=0).stop 2024-05-27T07:25:54.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 -- 172.21.15.43:0/3180060573 shutdown_connections 2024-05-27T07:25:54.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f68040b29f0 0x7f68040b4f00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6820109670 0x7f68201a1580 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68201043f0 0x7f682019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 --2- 172.21.15.43:0/3180060573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6820108d20 0x7f682019d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:54.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 -- 172.21.15.43:0/3180060573 >> 172.21.15.43:0/3180060573 conn(0x7f6820079d00 msgr2=0x7f682010f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:54.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 -- 172.21.15.43:0/3180060573 shutdown_connections 2024-05-27T07:25:54.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:54.481+0000 7f681f200700 1 -- 172.21.15.43:0/3180060573 wait complete. 2024-05-27T07:25:55.237 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67488 0.60 MB/sec execute 479 sec latency 380.343 ms 2024-05-27T07:25:55.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:25:55.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:55 smithi132 ceph-mon[35283]: pgmap v404: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 499 KiB/s wr, 33 op/s 2024-05-27T07:25:55.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:55 smithi043 ceph-mon[31531]: pgmap v404: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 499 KiB/s wr, 33 op/s 2024-05-27T07:25:55.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:55 smithi179 ceph-mon[35888]: pgmap v404: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 499 KiB/s wr, 33 op/s 2024-05-27T07:25:55.790 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:56.237 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67488 0.59 MB/sec execute 480 sec latency 1380.479 ms 2024-05-27T07:25:57.238 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67488 0.59 MB/sec execute 481 sec latency 2380.612 ms 2024-05-27T07:25:57.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:57 smithi043 ceph-mon[31531]: pgmap v405: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 652 KiB/s wr, 46 op/s 2024-05-27T07:25:57.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:57 smithi132 ceph-mon[35283]: pgmap v405: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 652 KiB/s wr, 46 op/s 2024-05-27T07:25:57.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:57 smithi179 ceph-mon[35888]: pgmap v405: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 652 KiB/s wr, 46 op/s 2024-05-27T07:25:58.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.030+0000 7f8935600700 1 -- 172.21.15.43:0/604524556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f893010f6b0 msgr2=0x7f893010fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.030+0000 7f8935600700 1 --2- 172.21.15.43:0/604524556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f893010f6b0 0x7f893010fac0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f8920009990 tx=0x7f8920009ca0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.030+0000 7f8935600700 1 -- 172.21.15.43:0/604524556 shutdown_connections 2024-05-27T07:25:58.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.030+0000 7f8935600700 1 --2- 172.21.15.43:0/604524556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893010b100 0x7f8930117610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.030+0000 7f8935600700 1 --2- 172.21.15.43:0/604524556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f893010f6b0 0x7f893010fac0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.030+0000 7f8935600700 1 --2- 172.21.15.43:0/604524556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f893010a7b0 0x7f893010abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.030+0000 7f8935600700 1 -- 172.21.15.43:0/604524556 >> 172.21.15.43:0/604524556 conn(0x7f8930076af0 msgr2=0x7f8930076f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.030+0000 7f8935600700 1 -- 172.21.15.43:0/604524556 shutdown_connections 2024-05-27T07:25:58.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.031+0000 7f8935600700 1 -- 172.21.15.43:0/604524556 wait complete. 2024-05-27T07:25:58.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.032+0000 7f8935600700 1 Processor -- start 2024-05-27T07:25:58.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.032+0000 7f8935600700 1 -- start start 2024-05-27T07:25:58.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.032+0000 7f8935600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f893010a7b0 0x7f89301a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.032+0000 7f8935600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f893010b100 0x7f89301a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.032+0000 7f8935600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893010f6b0 0x7f89301a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.032+0000 7f8935600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f893011a7a0 con 0x7f893010a7b0 2024-05-27T07:25:58.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.032+0000 7f8935600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f893011a620 con 0x7f893010f6b0 2024-05-27T07:25:58.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.032+0000 7f8935600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f893011a920 con 0x7f893010b100 2024-05-27T07:25:58.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.033+0000 7f892fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f893010a7b0 0x7f89301a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.033+0000 7f892fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f893010a7b0 0x7f89301a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34904/0 (socket says 172.21.15.43:34904) 2024-05-27T07:25:58.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.033+0000 7f892fe00700 1 -- 172.21.15.43:0/3697889750 learned_addr learned my addr 172.21.15.43:0/3697889750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:58.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.033+0000 7f892f400700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f893010b100 0x7f89301a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.033+0000 7f8934c00700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893010f6b0 0x7f89301a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f892fe00700 1 -- 172.21.15.43:0/3697889750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f893010b100 msgr2=0x7f89301a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f892fe00700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f893010b100 0x7f89301a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f892fe00700 1 -- 172.21.15.43:0/3697889750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893010f6b0 msgr2=0x7f89301a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f892fe00700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893010f6b0 0x7f89301a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f892fe00700 1 -- 172.21.15.43:0/3697889750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89301a6040 con 0x7f893010a7b0 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f892f400700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f893010b100 0x7f89301a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f8934c00700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893010f6b0 0x7f89301a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f892fe00700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f893010a7b0 0x7f89301a1170 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f8920000c00 tx=0x7f8920005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f892cc00700 1 -- 172.21.15.43:0/3697889750 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920027070 con 0x7f893010a7b0 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f8935600700 1 -- 172.21.15.43:0/3697889750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89301a62d0 con 0x7f893010a7b0 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.034+0000 7f892cc00700 1 -- 172.21.15.43:0/3697889750 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f892000fb80 con 0x7f893010a7b0 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.035+0000 7f892cc00700 1 -- 172.21.15.43:0/3697889750 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920022c30 con 0x7f893010a7b0 2024-05-27T07:25:58.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.035+0000 7f8935600700 1 -- 172.21.15.43:0/3697889750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89301adc50 con 0x7f893010a7b0 2024-05-27T07:25:58.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.038+0000 7f892cc00700 1 -- 172.21.15.43:0/3697889750 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f892002a070 con 0x7f893010a7b0 2024-05-27T07:25:58.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.040+0000 7f8945658b80 1 Processor -- start 2024-05-27T07:25:58.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.040+0000 7f8945658b80 1 -- start start 2024-05-27T07:25:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.041+0000 7f8945658b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb04ce40 0x55e9cb1caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.041+0000 7f8945658b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.041+0000 7f8945658b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.041+0000 7f8945658b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb0f9e40 con 0x55e9cb1cb410 2024-05-27T07:25:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.041+0000 7f8945658b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb0e99e0 con 0x55e9cb1cb820 2024-05-27T07:25:58.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.041+0000 7f8945658b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb0eda20 con 0x55e9cb04ce40 2024-05-27T07:25:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.041+0000 7f892cc00700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f89140b29f0 0x7f89140b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.041+0000 7f892fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.041+0000 7f892fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34912/0 (socket says 172.21.15.43:34912) 2024-05-27T07:25:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.041+0000 7f892fe00700 1 -- 172.21.15.43:0/2829394925 learned_addr learned my addr 172.21.15.43:0/2829394925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.042+0000 7f892f400700 1 --2- 172.21.15.43:0/2829394925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.042+0000 7f8934c00700 1 --2- 172.21.15.43:0/2829394925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb04ce40 0x55e9cb1caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.042+0000 7f892cc00700 1 -- 172.21.15.43:0/3697889750 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f89200e39d0 con 0x7f893010a7b0 2024-05-27T07:25:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.042+0000 7f8934c00700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f89140b29f0 0x7f89140b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.042+0000 7f892fe00700 1 -- 172.21.15.43:0/2829394925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb04ce40 msgr2=0x55e9cb1caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:58.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.042+0000 7f892fe00700 1 --2- 172.21.15.43:0/2829394925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb04ce40 0x55e9cb1caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.042+0000 7f892fe00700 1 -- 172.21.15.43:0/2829394925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb820 msgr2=0x55e9cb1d8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:58.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.042+0000 7f892fe00700 1 --2- 172.21.15.43:0/2829394925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.042+0000 7f892fe00700 1 -- 172.21.15.43:0/2829394925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9cb0b2420 con 0x55e9cb1cb410 2024-05-27T07:25:58.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.042+0000 7f892f400700 1 --2- 172.21.15.43:0/2829394925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:25:58.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.043+0000 7f892fe00700 1 --2- 172.21.15.43:0/2829394925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5ad0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f89200aa610 tx=0x7f8920009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd1d62c6b421325 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.043+0000 7f8934c00700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f89140b29f0 0x7f89140b4f00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x55e9cb1c7690 tx=0x7f8928006040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.043+0000 7f8925600700 1 -- 172.21.15.43:0/2829394925 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892000fd30 con 0x55e9cb1cb410 2024-05-27T07:25:58.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.043+0000 7f8925600700 1 -- 172.21.15.43:0/2829394925 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89200afe50 con 0x55e9cb1cb410 2024-05-27T07:25:58.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.043+0000 7f8925600700 1 -- 172.21.15.43:0/2829394925 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89200aecc0 con 0x55e9cb1cb410 2024-05-27T07:25:58.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.044+0000 7f8945658b80 1 -- 172.21.15.43:0/2829394925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 msgr2=0x55e9cb1d5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.044+0000 7f8945658b80 1 --2- 172.21.15.43:0/2829394925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5ad0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f89200aa610 tx=0x7f8920009310 comp rx=0 tx=0).stop 2024-05-27T07:25:58.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.045+0000 7f8945658b80 1 -- 172.21.15.43:0/2829394925 shutdown_connections 2024-05-27T07:25:58.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.045+0000 7f8945658b80 1 --2- 172.21.15.43:0/2829394925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.045+0000 7f8945658b80 1 --2- 172.21.15.43:0/2829394925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb04ce40 0x55e9cb1caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.045+0000 7f8945658b80 1 --2- 172.21.15.43:0/2829394925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5ad0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.045+0000 7f8945658b80 1 -- 172.21.15.43:0/2829394925 >> 172.21.15.43:0/2829394925 conn(0x55e9cb0acfc0 msgr2=0x55e9cb1c6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.045+0000 7f8945658b80 1 -- 172.21.15.43:0/2829394925 shutdown_connections 2024-05-27T07:25:58.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.045+0000 7f8945658b80 1 -- 172.21.15.43:0/2829394925 wait complete. 2024-05-27T07:25:58.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.045+0000 7f8945658b80 1 -- start start 2024-05-27T07:25:58.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.046+0000 7f8945658b80 1 Processor -- start 2024-05-27T07:25:58.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.046+0000 7f8945658b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb04ce40 0x55e9cb1d5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.046+0000 7f8945658b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.046+0000 7f8945658b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.046+0000 7f8945658b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e9cafeffa0 con 0x55e9cb1cb820 2024-05-27T07:25:58.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.046+0000 7f8945658b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb08a760 con 0x55e9cb1cb410 2024-05-27T07:25:58.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.046+0000 7f8945658b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb1d8750 con 0x55e9cb04ce40 2024-05-27T07:25:58.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.046+0000 7f8945658b80 10 client.0 _ll_register_callbacks cb 0x55e9cb093c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:58.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.047+0000 7f8934c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.047+0000 7f8934c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34928/0 (socket says 172.21.15.43:34928) 2024-05-27T07:25:58.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.047+0000 7f8934c00700 1 -- 172.21.15.43:0/4272381839 learned_addr learned my addr 172.21.15.43:0/4272381839 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:58.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.047+0000 7f892fe00700 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb04ce40 0x55e9cb1d5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.047+0000 7f892f400700 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.047+0000 7f8934c00700 1 -- 172.21.15.43:0/4272381839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb04ce40 msgr2=0x55e9cb1d5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:58.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.048+0000 7f8934c00700 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb04ce40 0x55e9cb1d5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.048+0000 7f8934c00700 1 -- 172.21.15.43:0/4272381839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb410 msgr2=0x55e9cb1d5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.048+0000 7f8934c00700 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.048+0000 7f8934c00700 1 -- 172.21.15.43:0/4272381839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9cb0398d0 con 0x55e9cb1cb820 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.048+0000 7f892f400700 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.048+0000 7f8934c00700 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d60c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8928006130 tx=0x7f8928004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61889ce119ffdb4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.048+0000 7f891a000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.048+0000 7f891a000700 1 -- 172.21.15.43:0/4272381839 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928029060 con 0x55e9cb1cb820 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.048+0000 7f891a000700 1 -- 172.21.15.43:0/4272381839 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f892801bb90 con 0x55e9cb1cb820 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.048+0000 7f8945658b80 1 -- 172.21.15.43:0/4272381839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89200afe50 con 0x55e9cb1cb820 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.049+0000 7f891a000700 1 -- 172.21.15.43:0/4272381839 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f8928030050 con 0x55e9cb1cb820 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.049+0000 7f8945658b80 10 client.17217 fetch_fsmap learned FSMap version 108 2024-05-27T07:25:58.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.049+0000 7f8945658b80 1 -- 172.21.15.43:0/4272381839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55e9cb0ba060 con 0x55e9cb1cb820 2024-05-27T07:25:58.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.049+0000 7f891a000700 1 -- 172.21.15.43:0/4272381839 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89200afe50 con 0x55e9cb1cb820 2024-05-27T07:25:58.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.049+0000 7f891a000700 1 -- 172.21.15.43:0/4272381839 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f8928018040 con 0x55e9cb1cb820 2024-05-27T07:25:58.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.049+0000 7f8945658b80 10 client.17217 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:25:58.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.049+0000 7f8945658b80 10 client.17217 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:25:58.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.049+0000 7f8945658b80 20 client.17217 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:58.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.049+0000 7f8945658b80 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e9cb1e5ba0 0x55e9cb1e8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.050+0000 7f8945658b80 4 client.17217 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:25:58.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.050+0000 7f8945658b80 1 -- 172.21.15.43:0/4272381839 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9cb0fbd90 con 0x55e9cb1e5ba0 2024-05-27T07:25:58.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.050+0000 7f892fe00700 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e9cb1e5ba0 0x55e9cb1e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.051+0000 7f892fe00700 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e9cb1e5ba0 0x55e9cb1e8030 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2a7ba4dc0d1aea78 server_cookie=3dcf00954b236e6a in_seq=0 out_seq=0 2024-05-27T07:25:58.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.051+0000 7f891a000700 10 client.17217 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:25:58.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.053+0000 7f891a000700 1 -- 172.21.15.43:0/4272381839 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e9cb0fbd90 con 0x55e9cb1e5ba0 2024-05-27T07:25:58.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.053+0000 7f891a000700 10 client.17217 handle_command_reply: tid=0 2024-05-27T07:25:58.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.053+0000 7f8945658b80 1 client.17217 shutdown 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.053+0000 7f8945658b80 1 -- 172.21.15.43:0/4272381839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 msgr2=0x55e9cb1d60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.053+0000 7f8945658b80 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d60c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8928006130 tx=0x7f8928004ad0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.054+0000 7f8945658b80 1 -- 172.21.15.43:0/4272381839 shutdown_connections 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.054+0000 7f8945658b80 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e9cb1e5ba0 0x55e9cb1e8030 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.054+0000 7f8945658b80 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1d60c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.054+0000 7f8945658b80 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb04ce40 0x55e9cb1d5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.054+0000 7f8945658b80 1 --2- 172.21.15.43:0/4272381839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.054+0000 7f8945658b80 1 -- 172.21.15.43:0/4272381839 >> 172.21.15.43:0/4272381839 conn(0x55e9cb0acfc0 msgr2=0x55e9cb1c5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.054+0000 7f891a000700 0 client.17217 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.054+0000 7f8945658b80 1 -- 172.21.15.43:0/4272381839 shutdown_connections 2024-05-27T07:25:58.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.054+0000 7f8945658b80 1 -- 172.21.15.43:0/4272381839 wait complete. 2024-05-27T07:25:58.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.054+0000 7f8945658b80 20 client.17217 trim_cache size 0 max 16384 2024-05-27T07:25:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.071+0000 7f8945658b80 1 Processor -- start 2024-05-27T07:25:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.071+0000 7f8945658b80 1 -- start start 2024-05-27T07:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8945658b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb04ce40 0x55e9cb1e8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8945658b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8945658b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8945658b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb1d4660 con 0x55e9cb1cb820 2024-05-27T07:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8945658b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89301b7660 con 0x55e9cb04ce40 2024-05-27T07:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8945658b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f89301b5330 con 0x55e9cb1cb410 2024-05-27T07:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8934c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8934c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34942/0 (socket says 172.21.15.43:34942) 2024-05-27T07:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8934c00700 1 -- 172.21.15.43:0/3342399903 learned_addr learned my addr 172.21.15.43:0/3342399903 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f892f400700 1 --2- 172.21.15.43:0/3342399903 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f892fe00700 1 --2- 172.21.15.43:0/3342399903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb04ce40 0x55e9cb1e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8934c00700 1 -- 172.21.15.43:0/3342399903 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb410 msgr2=0x55e9cb1d7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8934c00700 1 --2- 172.21.15.43:0/3342399903 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8934c00700 1 -- 172.21.15.43:0/3342399903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb04ce40 msgr2=0x55e9cb1e8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8934c00700 1 --2- 172.21.15.43:0/3342399903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb04ce40 0x55e9cb1e8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.072+0000 7f8934c00700 1 -- 172.21.15.43:0/3342399903 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9cb1d6430 con 0x55e9cb1cb820 2024-05-27T07:25:58.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.073+0000 7f892f400700 1 --2- 172.21.15.43:0/3342399903 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:25:58.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.073+0000 7f8934c00700 1 --2- 172.21.15.43:0/3342399903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1cd000 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8928024db0 tx=0x7f89280078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc96e5ca7b387ecb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.073+0000 7f8924c00700 1 -- 172.21.15.43:0/3342399903 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928009a10 con 0x55e9cb1cb820 2024-05-27T07:25:58.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.073+0000 7f8924c00700 1 -- 172.21.15.43:0/3342399903 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8928005c00 con 0x55e9cb1cb820 2024-05-27T07:25:58.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.073+0000 7f8924c00700 1 -- 172.21.15.43:0/3342399903 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928032c90 con 0x55e9cb1cb820 2024-05-27T07:25:58.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.074+0000 7f8945658b80 1 -- 172.21.15.43:0/3342399903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 msgr2=0x55e9cb1cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.074+0000 7f8945658b80 1 --2- 172.21.15.43:0/3342399903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1cd000 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8928024db0 tx=0x7f89280078d0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.074+0000 7f8945658b80 1 -- 172.21.15.43:0/3342399903 shutdown_connections 2024-05-27T07:25:58.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.074+0000 7f8945658b80 1 --2- 172.21.15.43:0/3342399903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb820 0x55e9cb1cd000 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.074+0000 7f8945658b80 1 --2- 172.21.15.43:0/3342399903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb04ce40 0x55e9cb1e8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.074+0000 7f8945658b80 1 --2- 172.21.15.43:0/3342399903 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.074+0000 7f8945658b80 1 -- 172.21.15.43:0/3342399903 >> 172.21.15.43:0/3342399903 conn(0x55e9cb0acfc0 msgr2=0x55e9cb1cc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.074+0000 7f8945658b80 1 -- 172.21.15.43:0/3342399903 shutdown_connections 2024-05-27T07:25:58.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.074+0000 7f8945658b80 1 -- 172.21.15.43:0/3342399903 wait complete. 2024-05-27T07:25:58.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.075+0000 7f8945658b80 1 -- start start 2024-05-27T07:25:58.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.075+0000 7f8945658b80 1 Processor -- start 2024-05-27T07:25:58.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f8945658b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb04ce40 0x55e9cb1d8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f8945658b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f8945658b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb820 0x55e9cb24e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f8945658b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f89301b5a00 con 0x55e9cb1cb410 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f8945658b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f89301b6f90 con 0x55e9cb04ce40 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f8945658b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f89301b6130 con 0x55e9cb1cb820 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f8945658b80 10 client.0 _ll_register_callbacks cb 0x55e9cb093c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f892f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f892f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34948/0 (socket says 172.21.15.43:34948) 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f892f400700 1 -- 172.21.15.43:0/2886533358 learned_addr learned my addr 172.21.15.43:0/2886533358 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f892fe00700 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb04ce40 0x55e9cb1d8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.076+0000 7f8934c00700 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb820 0x55e9cb24e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.077+0000 7f892f400700 1 -- 172.21.15.43:0/2886533358 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb820 msgr2=0x55e9cb24e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:58.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.077+0000 7f892f400700 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb820 0x55e9cb24e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.077+0000 7f892f400700 1 -- 172.21.15.43:0/2886533358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb04ce40 msgr2=0x55e9cb1d8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:25:58.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.077+0000 7f892f400700 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb04ce40 0x55e9cb1d8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.077+0000 7f892f400700 1 -- 172.21.15.43:0/2886533358 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9cb24ec30 con 0x55e9cb1cb410 2024-05-27T07:25:58.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.077+0000 7f892f400700 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d94a0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f8930077ff0 tx=0x7f891c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658e369c210f4d82 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:25:58.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.077+0000 7f891a000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:25:58.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.077+0000 7f891a000700 1 -- 172.21.15.43:0/2886533358 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891c01c440 con 0x55e9cb1cb410 2024-05-27T07:25:58.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.077+0000 7f891a000700 1 -- 172.21.15.43:0/2886533358 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7f891c010420 con 0x55e9cb1cb410 2024-05-27T07:25:58.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.077+0000 7f8945658b80 1 -- 172.21.15.43:0/2886533358 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8928005c00 con 0x55e9cb1cb410 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f891a000700 1 -- 172.21.15.43:0/2886533358 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f891c010620 con 0x55e9cb1cb410 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f8945658b80 10 client.17223 fetch_fsmap learned FSMap version 108 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f8945658b80 1 -- 172.21.15.43:0/2886533358 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55e9cb24fe40 con 0x55e9cb1cb410 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f891a000700 1 -- 172.21.15.43:0/2886533358 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8928005c00 con 0x55e9cb1cb410 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f891a000700 1 -- 172.21.15.43:0/2886533358 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f891c01a070 con 0x55e9cb1cb410 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f8945658b80 10 client.17223 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f8945658b80 10 client.17223 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f8945658b80 20 client.17223 populate_metadata read hostname 'smithi043' 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f8945658b80 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e9cb250660 0x55e9cb252a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f8945658b80 4 client.17223 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.078+0000 7f8945658b80 1 -- 172.21.15.43:0/2886533358 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e9cb0f5570 con 0x55e9cb250660 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.079+0000 7f892fe00700 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e9cb250660 0x55e9cb252a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:25:58.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.080+0000 7f892fe00700 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e9cb250660 0x55e9cb252a80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=efa7ed17b4e15741 server_cookie=3260f6d62e58163 in_seq=0 out_seq=0 2024-05-27T07:25:58.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.080+0000 7f891a000700 10 client.17223 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.080+0000 7f891a000700 1 -- 172.21.15.43:0/2886533358 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e9cb0f5570 con 0x55e9cb250660 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.080+0000 7f891a000700 10 client.17223 handle_command_reply: tid=0 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.080+0000 7f8945658b80 1 client.17223 shutdown 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.080+0000 7f8945658b80 1 -- 172.21.15.43:0/2886533358 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 msgr2=0x55e9cb1d94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.080+0000 7f8945658b80 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d94a0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f8930077ff0 tx=0x7f891c003f90 comp rx=0 tx=0).stop 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.081+0000 7f8945658b80 1 -- 172.21.15.43:0/2886533358 shutdown_connections 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.081+0000 7f8945658b80 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e9cb250660 0x55e9cb252a80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.081+0000 7f8945658b80 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e9cb1cb820 0x55e9cb24e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.081+0000 7f8945658b80 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e9cb04ce40 0x55e9cb1d8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.081+0000 7f8945658b80 1 --2- 172.21.15.43:0/2886533358 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e9cb1cb410 0x55e9cb1d94a0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.081+0000 7f8945658b80 1 -- 172.21.15.43:0/2886533358 >> 172.21.15.43:0/2886533358 conn(0x55e9cb0acfc0 msgr2=0x55e9cb1d5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.081+0000 7f891a000700 0 client.17223 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:25:58.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.081+0000 7f8945658b80 1 -- 172.21.15.43:0/2886533358 shutdown_connections 2024-05-27T07:25:58.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.081+0000 7f8945658b80 1 -- 172.21.15.43:0/2886533358 wait complete. 2024-05-27T07:25:58.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.081+0000 7f8945658b80 20 client.17223 trim_cache size 0 max 16384 2024-05-27T07:25:58.073 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 -- 172.21.15.43:0/3697889750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f89140b29f0 msgr2=0x7f89140b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f89140b29f0 0x7f89140b4f00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x55e9cb1c7690 tx=0x7f8928006040 comp rx=0 tx=0).stop 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 -- 172.21.15.43:0/3697889750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f893010a7b0 msgr2=0x7f89301a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f893010a7b0 0x7f89301a1170 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f8920000c00 tx=0x7f8920005ef0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 -- 172.21.15.43:0/3697889750 shutdown_connections 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f89140b29f0 0x7f89140b4f00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893010f6b0 0x7f89301a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f893010a7b0 0x7f89301a1170 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 --2- 172.21.15.43:0/3697889750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f893010b100 0x7f89301a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 -- 172.21.15.43:0/3697889750 >> 172.21.15.43:0/3697889750 conn(0x7f8930076af0 msgr2=0x7f89301088b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 -- 172.21.15.43:0/3697889750 shutdown_connections 2024-05-27T07:25:58.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:25:58.082+0000 7f8935600700 1 -- 172.21.15.43:0/3697889750 wait complete. 2024-05-27T07:25:58.238 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67488 0.59 MB/sec execute 482 sec latency 3380.783 ms 2024-05-27T07:25:58.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:25:59.238 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67547 0.59 MB/sec execute 483 sec latency 3604.631 ms 2024-05-27T07:25:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:25:59.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:25:59 smithi132 ceph-mon[35283]: pgmap v406: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 474 KiB/s wr, 31 op/s 2024-05-27T07:25:59.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:25:59 smithi043 ceph-mon[31531]: pgmap v406: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 474 KiB/s wr, 31 op/s 2024-05-27T07:25:59.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:25:59 smithi179 ceph-mon[35888]: pgmap v406: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 474 KiB/s wr, 31 op/s 2024-05-27T07:26:00.238 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67649 0.59 MB/sec execute 484 sec latency 63.224 ms 2024-05-27T07:26:01.238 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67701 0.59 MB/sec execute 485 sec latency 61.984 ms 2024-05-27T07:26:01.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:01 smithi043 ceph-mon[31531]: pgmap v407: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 475 KiB/s wr, 31 op/s 2024-05-27T07:26:01.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:01 smithi132 ceph-mon[35283]: pgmap v407: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 475 KiB/s wr, 31 op/s 2024-05-27T07:26:01.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:01 smithi179 ceph-mon[35888]: pgmap v407: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 475 KiB/s wr, 31 op/s 2024-05-27T07:26:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.743+0000 7fa61d800700 1 -- 172.21.15.43:0/979483780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa618110970 msgr2=0x7fa618110d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.743+0000 7fa61d800700 1 --2- 172.21.15.43:0/979483780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa618110970 0x7fa618110d80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa60800b0a0 tx=0x7fa60800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.743+0000 7fa61d800700 1 -- 172.21.15.43:0/979483780 shutdown_connections 2024-05-27T07:26:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.743+0000 7fa61d800700 1 --2- 172.21.15.43:0/979483780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa61810b290 0x7fa618117810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.743+0000 7fa61d800700 1 --2- 172.21.15.43:0/979483780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa618110970 0x7fa618110d80 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.743+0000 7fa61d800700 1 --2- 172.21.15.43:0/979483780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa61810a940 0x7fa61810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.743+0000 7fa61d800700 1 -- 172.21.15.43:0/979483780 >> 172.21.15.43:0/979483780 conn(0x7fa618076d20 msgr2=0x7fa618077130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.744+0000 7fa61d800700 1 -- 172.21.15.43:0/979483780 shutdown_connections 2024-05-27T07:26:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.744+0000 7fa61d800700 1 -- 172.21.15.43:0/979483780 wait complete. 2024-05-27T07:26:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.745+0000 7fa61d800700 1 Processor -- start 2024-05-27T07:26:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.745+0000 7fa61d800700 1 -- start start 2024-05-27T07:26:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.745+0000 7fa61d800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa61810a940 0x7fa6181a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.745+0000 7fa61d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa61810b290 0x7fa6181a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.746+0000 7fa61d800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa618110970 0x7fa6181a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.746+0000 7fa61d800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa61811a930 con 0x7fa61810b290 2024-05-27T07:26:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.746+0000 7fa61d800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa61811a7b0 con 0x7fa61810a940 2024-05-27T07:26:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.746+0000 7fa61d800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa61811aab0 con 0x7fa618110970 2024-05-27T07:26:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.746+0000 7fa60e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa61810b290 0x7fa6181a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.746+0000 7fa60e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa61810b290 0x7fa6181a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34964/0 (socket says 172.21.15.43:34964) 2024-05-27T07:26:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.746+0000 7fa60e000700 1 -- 172.21.15.43:0/3994428031 learned_addr learned my addr 172.21.15.43:0/3994428031 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.746+0000 7fa616a00700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa61810a940 0x7fa6181a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.746+0000 7fa617400700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa618110970 0x7fa6181a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.747+0000 7fa60e000700 1 -- 172.21.15.43:0/3994428031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa618110970 msgr2=0x7fa6181a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.747+0000 7fa60e000700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa618110970 0x7fa6181a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.747+0000 7fa60e000700 1 -- 172.21.15.43:0/3994428031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa61810a940 msgr2=0x7fa6181a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.747+0000 7fa60e000700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa61810a940 0x7fa6181a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.747+0000 7fa60e000700 1 -- 172.21.15.43:0/3994428031 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6181a6230 con 0x7fa61810b290 2024-05-27T07:26:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.747+0000 7fa60e000700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa61810b290 0x7fa6181a1900 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fa60000d980 tx=0x7fa60000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:01.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.748+0000 7fa60fe00700 1 -- 172.21.15.43:0/3994428031 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa600017070 con 0x7fa61810b290 2024-05-27T07:26:01.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.748+0000 7fa61d800700 1 -- 172.21.15.43:0/3994428031 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6181a64c0 con 0x7fa61810b290 2024-05-27T07:26:01.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.748+0000 7fa61d800700 1 -- 172.21.15.43:0/3994428031 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6181ade40 con 0x7fa61810b290 2024-05-27T07:26:01.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.748+0000 7fa60fe00700 1 -- 172.21.15.43:0/3994428031 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa60000fd20 con 0x7fa61810b290 2024-05-27T07:26:01.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.748+0000 7fa60fe00700 1 -- 172.21.15.43:0/3994428031 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60000e3b0 con 0x7fa61810b290 2024-05-27T07:26:01.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.753+0000 7fa60fe00700 1 -- 172.21.15.43:0/3994428031 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa60000e550 con 0x7fa61810b290 2024-05-27T07:26:01.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.753+0000 7fa62c3f1b80 1 Processor -- start 2024-05-27T07:26:01.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.754+0000 7fa62c3f1b80 1 -- start start 2024-05-27T07:26:01.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.754+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48813e40 0x55da489987f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.754+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 0x55da48992870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.754+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da489931c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.754+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55da488c0e40 con 0x55da48813e40 2024-05-27T07:26:01.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.754+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55da488b09e0 con 0x55da48992db0 2024-05-27T07:26:01.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.754+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55da488b4a20 con 0x55da48998d30 2024-05-27T07:26:01.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.755+0000 7fa60fe00700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa5fc0b29f0 0x7fa5fc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.755+0000 7fa60e000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da489931c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.755+0000 7fa616a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 0x55da48992870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.755+0000 7fa616a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 0x55da48992870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:38734/0 (socket says 172.21.15.43:38734) 2024-05-27T07:26:01.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.755+0000 7fa616a00700 1 -- 172.21.15.43:0/1720111186 learned_addr learned my addr 172.21.15.43:0/1720111186 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:01.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.755+0000 7fa60fe00700 1 -- 172.21.15.43:0/3994428031 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7fa6000e23b0 con 0x7fa61810b290 2024-05-27T07:26:01.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.755+0000 7fa617400700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa5fc0b29f0 0x7fa5fc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.755+0000 7fa617400700 1 --2- 172.21.15.43:0/1720111186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48813e40 0x55da489987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.756+0000 7fa60e000700 1 -- 172.21.15.43:0/1720111186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 msgr2=0x55da48992870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:01.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.756+0000 7fa60e000700 1 --2- 172.21.15.43:0/1720111186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 0x55da48992870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.756+0000 7fa60e000700 1 -- 172.21.15.43:0/1720111186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48813e40 msgr2=0x55da489987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:01.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.756+0000 7fa60e000700 1 --2- 172.21.15.43:0/1720111186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48813e40 0x55da489987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.756+0000 7fa60e000700 1 -- 172.21.15.43:0/1720111186 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da48879420 con 0x55da48992db0 2024-05-27T07:26:01.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.756+0000 7fa617400700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa5fc0b29f0 0x7fa5fc0b4f00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55da48993b30 tx=0x7fa610004750 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:01.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.756+0000 7fa60e000700 1 --2- 172.21.15.43:0/1720111186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da489931c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa6000df6d0 tx=0x7fa600076e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16453f6bdf82f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:01.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.757+0000 7fa607e00700 1 -- 172.21.15.43:0/1720111186 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa600020070 con 0x55da48992db0 2024-05-27T07:26:01.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.757+0000 7fa607e00700 1 -- 172.21.15.43:0/1720111186 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa60006ec60 con 0x55da48992db0 2024-05-27T07:26:01.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.757+0000 7fa607e00700 1 -- 172.21.15.43:0/1720111186 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6000a94c0 con 0x55da48992db0 2024-05-27T07:26:01.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.758+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1720111186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 msgr2=0x55da489931c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.758+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/1720111186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da489931c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa6000df6d0 tx=0x7fa600076e30 comp rx=0 tx=0).stop 2024-05-27T07:26:01.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.758+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1720111186 shutdown_connections 2024-05-27T07:26:01.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.758+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/1720111186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da489931c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.758+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/1720111186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48813e40 0x55da489987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.758+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/1720111186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 0x55da48992870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.758+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1720111186 >> 172.21.15.43:0/1720111186 conn(0x55da48873fc0 msgr2=0x55da4898d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:01.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.758+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1720111186 shutdown_connections 2024-05-27T07:26:01.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.758+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1720111186 wait complete. 2024-05-27T07:26:01.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.759+0000 7fa62c3f1b80 1 -- start start 2024-05-27T07:26:01.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.759+0000 7fa62c3f1b80 1 Processor -- start 2024-05-27T07:26:01.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48813e40 0x55da4899c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48992db0 0x55da4899ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 0x55da4899cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55da487b6fa0 con 0x55da48992db0 2024-05-27T07:26:01.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55da48851760 con 0x55da48813e40 2024-05-27T07:26:01.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55da4899f7a0 con 0x55da48998d30 2024-05-27T07:26:01.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa62c3f1b80 10 client.0 _ll_register_callbacks cb 0x55da4885ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa60e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48992db0 0x55da4899ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa60e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48992db0 0x55da4899ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34972/0 (socket says 172.21.15.43:34972) 2024-05-27T07:26:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa60e000700 1 -- 172.21.15.43:0/1378673258 learned_addr learned my addr 172.21.15.43:0/1378673258 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.760+0000 7fa616a00700 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48813e40 0x55da4899c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.761+0000 7fa617400700 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 0x55da4899cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.761+0000 7fa60e000700 1 -- 172.21.15.43:0/1378673258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 msgr2=0x55da4899cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.761+0000 7fa60e000700 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 0x55da4899cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.761+0000 7fa60e000700 1 -- 172.21.15.43:0/1378673258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48813e40 msgr2=0x55da4899c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.761+0000 7fa60e000700 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48813e40 0x55da4899c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.761+0000 7fa60e000700 1 -- 172.21.15.43:0/1378673258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da488008d0 con 0x55da48992db0 2024-05-27T07:26:01.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.761+0000 7fa616a00700 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48813e40 0x55da4899c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:01.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.761+0000 7fa617400700 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 0x55da4899cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:01.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.761+0000 7fa60e000700 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48992db0 0x55da4899ca10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa600000c00 tx=0x7fa6000a9960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25648ee0442e9667 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:01.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.761+0000 7fa604c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.762+0000 7fa604c00700 1 -- 172.21.15.43:0/1378673258 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa600020030 con 0x55da48992db0 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.762+0000 7fa604c00700 1 -- 172.21.15.43:0/1378673258 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7fa6000a96d0 con 0x55da48992db0 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.762+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1378673258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa60006ec60 con 0x55da48992db0 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.762+0000 7fa604c00700 1 -- 172.21.15.43:0/1378673258 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fa6000e2170 con 0x55da48992db0 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.762+0000 7fa62c3f1b80 10 client.17235 fetch_fsmap learned FSMap version 108 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.762+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1378673258 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55da48881060 con 0x55da48992db0 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.762+0000 7fa604c00700 1 -- 172.21.15.43:0/1378673258 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa60006ec60 con 0x55da48992db0 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.763+0000 7fa604c00700 1 -- 172.21.15.43:0/1378673258 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7fa6000a84c0 con 0x55da48992db0 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.763+0000 7fa62c3f1b80 10 client.17235 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.763+0000 7fa62c3f1b80 10 client.17235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:01.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.763+0000 7fa62c3f1b80 20 client.17235 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.763+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55da489ac6f0 0x55da489aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.763+0000 7fa62c3f1b80 4 client.17235 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.763+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1378673258 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da488c2d90 con 0x55da489ac6f0 2024-05-27T07:26:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.763+0000 7fa616a00700 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55da489ac6f0 0x55da489aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.765+0000 7fa616a00700 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55da489ac6f0 0x55da489aeb40 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d96dc00c030316fc server_cookie=60204b494fddc8f2 in_seq=0 out_seq=0 2024-05-27T07:26:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.765+0000 7fa604c00700 10 client.17235 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:01.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.769+0000 7fa604c00700 1 -- 172.21.15.43:0/1378673258 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55da488c2d90 con 0x55da489ac6f0 2024-05-27T07:26:01.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.769+0000 7fa604c00700 10 client.17235 handle_command_reply: tid=0 2024-05-27T07:26:01.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.769+0000 7fa62c3f1b80 1 client.17235 shutdown 2024-05-27T07:26:01.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.769+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1378673258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48992db0 msgr2=0x55da4899ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.769+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48992db0 0x55da4899ca10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa600000c00 tx=0x7fa6000a9960 comp rx=0 tx=0).stop 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.769+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1378673258 shutdown_connections 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.770+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48998d30 0x55da4899cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.770+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48813e40 0x55da4899c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.770+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55da489ac6f0 0x55da489aeb40 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.770+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/1378673258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48992db0 0x55da4899ca10 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.770+0000 7fa604c00700 0 client.17235 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.770+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1378673258 >> 172.21.15.43:0/1378673258 conn(0x55da48873fc0 msgr2=0x55da4898d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.770+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1378673258 shutdown_connections 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.770+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/1378673258 wait complete. 2024-05-27T07:26:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.770+0000 7fa62c3f1b80 20 client.17235 trim_cache size 0 max 16384 2024-05-27T07:26:01.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.788+0000 7fa62c3f1b80 1 Processor -- start 2024-05-27T07:26:01.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.788+0000 7fa62c3f1b80 1 -- start start 2024-05-27T07:26:01.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.788+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48813e40 0x55da48992870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.788+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da48996c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.788+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.788+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f400fa30 con 0x55da48998d30 2024-05-27T07:26:01.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.788+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f400d700 con 0x55da48992db0 2024-05-27T07:26:01.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.788+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f400f360 con 0x55da48813e40 2024-05-27T07:26:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.789+0000 7fa617400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.789+0000 7fa617400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34986/0 (socket says 172.21.15.43:34986) 2024-05-27T07:26:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.789+0000 7fa617400700 1 -- 172.21.15.43:0/50527962 learned_addr learned my addr 172.21.15.43:0/50527962 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.789+0000 7fa60e000700 1 --2- 172.21.15.43:0/50527962 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da48996c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.789+0000 7fa616a00700 1 --2- 172.21.15.43:0/50527962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48813e40 0x55da48992870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.790+0000 7fa617400700 1 -- 172.21.15.43:0/50527962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48813e40 msgr2=0x55da48992870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.790+0000 7fa617400700 1 --2- 172.21.15.43:0/50527962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48813e40 0x55da48992870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.790+0000 7fa617400700 1 -- 172.21.15.43:0/50527962 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 msgr2=0x55da48996c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.790+0000 7fa617400700 1 --2- 172.21.15.43:0/50527962 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da48996c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.790+0000 7fa617400700 1 -- 172.21.15.43:0/50527962 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da4899d590 con 0x55da48998d30 2024-05-27T07:26:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.790+0000 7fa617400700 1 --2- 172.21.15.43:0/50527962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489aeb10 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fa610012e40 tx=0x7fa61001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3225f0c0e92b1187 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:01.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.790+0000 7fa607400700 1 -- 172.21.15.43:0/50527962 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61000dd30 con 0x55da48998d30 2024-05-27T07:26:01.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.791+0000 7fa607400700 1 -- 172.21.15.43:0/50527962 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa61002e070 con 0x55da48998d30 2024-05-27T07:26:01.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.791+0000 7fa607400700 1 -- 172.21.15.43:0/50527962 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610029660 con 0x55da48998d30 2024-05-27T07:26:01.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.792+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/50527962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 msgr2=0x55da489aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.792+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/50527962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489aeb10 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fa610012e40 tx=0x7fa61001b000 comp rx=0 tx=0).stop 2024-05-27T07:26:01.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.792+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/50527962 shutdown_connections 2024-05-27T07:26:01.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.792+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/50527962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489aeb10 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.792+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/50527962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48813e40 0x55da48992870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.792+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/50527962 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da48996c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.792+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/50527962 >> 172.21.15.43:0/50527962 conn(0x55da48873fc0 msgr2=0x55da4899c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:01.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.792+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/50527962 shutdown_connections 2024-05-27T07:26:01.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.792+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/50527962 wait complete. 2024-05-27T07:26:01.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.792+0000 7fa62c3f1b80 1 -- start start 2024-05-27T07:26:01.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.793+0000 7fa62c3f1b80 1 Processor -- start 2024-05-27T07:26:01.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.793+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48813e40 0x55da489a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.793+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da489a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.793+0000 7fa62c3f1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.793+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f400e500 con 0x55da48998d30 2024-05-27T07:26:01.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.793+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f400ddd0 con 0x55da48992db0 2024-05-27T07:26:01.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.793+0000 7fa62c3f1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f400ec30 con 0x55da48813e40 2024-05-27T07:26:01.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.793+0000 7fa62c3f1b80 10 client.0 _ll_register_callbacks cb 0x55da4885ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:01.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.794+0000 7fa617400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.794+0000 7fa617400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34988/0 (socket says 172.21.15.43:34988) 2024-05-27T07:26:01.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.794+0000 7fa617400700 1 -- 172.21.15.43:0/539583355 learned_addr learned my addr 172.21.15.43:0/539583355 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:01.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.794+0000 7fa616a00700 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48813e40 0x55da489a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.794+0000 7fa60e000700 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da489a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.794+0000 7fa617400700 1 -- 172.21.15.43:0/539583355 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48813e40 msgr2=0x55da489a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:01.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.794+0000 7fa617400700 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48813e40 0x55da489a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.794+0000 7fa617400700 1 -- 172.21.15.43:0/539583355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 msgr2=0x55da489a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:01.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.794+0000 7fa617400700 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da489a4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.794+0000 7fa617400700 1 -- 172.21.15.43:0/539583355 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da489b37d0 con 0x55da48998d30 2024-05-27T07:26:01.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.795+0000 7fa617400700 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489b3290 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa61000b3d0 tx=0x7fa61000dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b0ec054fed1a0b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:01.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.795+0000 7fa604c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:01.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.795+0000 7fa604c00700 1 -- 172.21.15.43:0/539583355 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610038070 con 0x55da48998d30 2024-05-27T07:26:01.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.795+0000 7fa604c00700 1 -- 172.21.15.43:0/539583355 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9496+0+0 (secure 0 0 0) 0x7fa610005450 con 0x55da48998d30 2024-05-27T07:26:01.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.795+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/539583355 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa61002e070 con 0x55da48998d30 2024-05-27T07:26:01.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.796+0000 7fa604c00700 1 -- 172.21.15.43:0/539583355 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7fa610044050 con 0x55da48998d30 2024-05-27T07:26:01.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.796+0000 7fa62c3f1b80 10 client.17241 fetch_fsmap learned FSMap version 108 2024-05-27T07:26:01.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.796+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/539583355 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55da48a16870 con 0x55da48998d30 2024-05-27T07:26:01.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.796+0000 7fa604c00700 1 -- 172.21.15.43:0/539583355 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa61002e070 con 0x55da48998d30 2024-05-27T07:26:01.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.796+0000 7fa604c00700 1 -- 172.21.15.43:0/539583355 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7fa61001e5b0 con 0x55da48998d30 2024-05-27T07:26:01.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.796+0000 7fa62c3f1b80 10 client.17241 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:26:01.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.796+0000 7fa62c3f1b80 10 client.17241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:01.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.796+0000 7fa62c3f1b80 20 client.17241 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:01.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.796+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55da48a16f50 0x55da48a19370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:01.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.797+0000 7fa62c3f1b80 4 client.17241 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:26:01.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.797+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/539583355 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da488bc570 con 0x55da48a16f50 2024-05-27T07:26:01.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.797+0000 7fa616a00700 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55da48a16f50 0x55da48a19370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:01.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.798+0000 7fa616a00700 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55da48a16f50 0x55da48a19370 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7d2371b14fa7ebb server_cookie=19913571b25324d5 in_seq=0 out_seq=0 2024-05-27T07:26:01.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.798+0000 7fa604c00700 10 client.17241 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:01.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.799+0000 7fa604c00700 1 -- 172.21.15.43:0/539583355 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da488bc570 con 0x55da48a16f50 2024-05-27T07:26:01.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa604c00700 10 client.17241 handle_command_reply: tid=0 2024-05-27T07:26:01.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 client.17241 shutdown 2024-05-27T07:26:01.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/539583355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 msgr2=0x55da489b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489b3290 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa61000b3d0 tx=0x7fa61000dd30 comp rx=0 tx=0).stop 2024-05-27T07:26:01.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/539583355 shutdown_connections 2024-05-27T07:26:01.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55da48a16f50 0x55da48a19370 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55da48998d30 0x55da489b3290 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da48813e40 0x55da489a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 --2- 172.21.15.43:0/539583355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55da48992db0 0x55da489a4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/539583355 >> 172.21.15.43:0/539583355 conn(0x55da48873fc0 msgr2=0x55da4899c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:01.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa604c00700 0 client.17241 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:01.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/539583355 shutdown_connections 2024-05-27T07:26:01.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 1 -- 172.21.15.43:0/539583355 wait complete. 2024-05-27T07:26:01.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.800+0000 7fa62c3f1b80 20 client.17241 trim_cache size 0 max 16384 2024-05-27T07:26:01.791 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.801+0000 7fa61d800700 1 -- 172.21.15.43:0/3994428031 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa5fc0b29f0 msgr2=0x7fa5fc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.801+0000 7fa61d800700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa5fc0b29f0 0x7fa5fc0b4f00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55da48993b30 tx=0x7fa610004750 comp rx=0 tx=0).stop 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.801+0000 7fa61d800700 1 -- 172.21.15.43:0/3994428031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa61810b290 msgr2=0x7fa6181a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.801+0000 7fa61d800700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa61810b290 0x7fa6181a1900 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fa60000d980 tx=0x7fa60000dc90 comp rx=0 tx=0).stop 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.801+0000 7fa61d800700 1 -- 172.21.15.43:0/3994428031 shutdown_connections 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.801+0000 7fa61d800700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa5fc0b29f0 0x7fa5fc0b4f00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.801+0000 7fa61d800700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa618110970 0x7fa6181a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.801+0000 7fa61d800700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa61810a940 0x7fa6181a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.801+0000 7fa61d800700 1 --2- 172.21.15.43:0/3994428031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa61810b290 0x7fa6181a1900 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.801+0000 7fa61d800700 1 -- 172.21.15.43:0/3994428031 >> 172.21.15.43:0/3994428031 conn(0x7fa618076d20 msgr2=0x7fa618109230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:01.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.802+0000 7fa61d800700 1 -- 172.21.15.43:0/3994428031 shutdown_connections 2024-05-27T07:26:01.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:01.802+0000 7fa61d800700 1 -- 172.21.15.43:0/3994428031 wait complete. 2024-05-27T07:26:02.238 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67768 0.59 MB/sec execute 486 sec latency 57.112 ms 2024-05-27T07:26:03.238 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67818 0.59 MB/sec execute 487 sec latency 138.358 ms 2024-05-27T07:26:03.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:03 smithi043 ceph-mon[31531]: pgmap v408: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 658 KiB/s wr, 47 op/s 2024-05-27T07:26:03.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:03 smithi043 ceph-mon[31531]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:26:03.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:03 smithi179 ceph-mon[35888]: pgmap v408: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 658 KiB/s wr, 47 op/s 2024-05-27T07:26:03.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:03 smithi179 ceph-mon[35888]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:26:03.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:03 smithi132 ceph-mon[35283]: pgmap v408: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 658 KiB/s wr, 47 op/s 2024-05-27T07:26:03.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:03 smithi132 ceph-mon[35283]: osdmap e132: 12 total, 12 up, 12 in 2024-05-27T07:26:04.238 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67871 0.59 MB/sec execute 488 sec latency 44.505 ms 2024-05-27T07:26:04.596 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:26:04.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:26:05.077 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:05.239 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67952 0.59 MB/sec execute 489 sec latency 61.639 ms 2024-05-27T07:26:05.562 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:05 smithi043 ceph-mon[31531]: pgmap v410: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 404 KiB/s wr, 35 op/s 2024-05-27T07:26:05.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:05 smithi179 ceph-mon[35888]: pgmap v410: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 404 KiB/s wr, 35 op/s 2024-05-27T07:26:05.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:05 smithi132 ceph-mon[35283]: pgmap v410: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 404 KiB/s wr, 35 op/s 2024-05-27T07:26:06.239 INFO:tasks.workunit.client.0.smithi043.stdout: 1 68016 0.59 MB/sec execute 490 sec latency 57.659 ms 2024-05-27T07:26:07.239 INFO:tasks.workunit.client.0.smithi043.stdout: 1 68115 0.58 MB/sec execute 491 sec latency 49.504 ms 2024-05-27T07:26:07.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.407+0000 7f0823600700 1 -- 172.21.15.43:0/2867721549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f081c1054b0 msgr2=0x7f081c1058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:07.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.407+0000 7f0823600700 1 --2- 172.21.15.43:0/2867721549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f081c1054b0 0x7f081c1058c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f0810009990 tx=0x7f0810009ca0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.407+0000 7f0823600700 1 -- 172.21.15.43:0/2867721549 shutdown_connections 2024-05-27T07:26:07.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.407+0000 7f0823600700 1 --2- 172.21.15.43:0/2867721549 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c10e8e0 0x7f081c110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.407+0000 7f0823600700 1 --2- 172.21.15.43:0/2867721549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f081c1054b0 0x7f081c1058c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.407+0000 7f0823600700 1 --2- 172.21.15.43:0/2867721549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f081c105e00 0x7f081c10e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.407+0000 7f0823600700 1 -- 172.21.15.43:0/2867721549 >> 172.21.15.43:0/2867721549 conn(0x7f081c0fedb0 msgr2=0x7f081c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:07.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.407+0000 7f0823600700 1 -- 172.21.15.43:0/2867721549 shutdown_connections 2024-05-27T07:26:07.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.407+0000 7f0823600700 1 -- 172.21.15.43:0/2867721549 wait complete. 2024-05-27T07:26:07.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.408+0000 7f0823600700 1 Processor -- start 2024-05-27T07:26:07.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.409+0000 7f0823600700 1 -- start start 2024-05-27T07:26:07.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.409+0000 7f0823600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c1054b0 0x7f081c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.409+0000 7f0823600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f081c105e00 0x7f081c19d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.409+0000 7f0823600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f081c10e8e0 0x7f081c1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.409+0000 7f0823600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f081c114300 con 0x7f081c10e8e0 2024-05-27T07:26:07.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.409+0000 7f0823600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f081c114180 con 0x7f081c105e00 2024-05-27T07:26:07.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.409+0000 7f0823600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f081c114480 con 0x7f081c1054b0 2024-05-27T07:26:07.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.410+0000 7f0821600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f081c10e8e0 0x7f081c1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.410+0000 7f0821600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f081c10e8e0 0x7f081c1a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38760/0 (socket says 172.21.15.43:38760) 2024-05-27T07:26:07.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.410+0000 7f0821600700 1 -- 172.21.15.43:0/1521594198 learned_addr learned my addr 172.21.15.43:0/1521594198 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:07.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.410+0000 7f081be00700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f081c105e00 0x7f081c19d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.410+0000 7f0820c00700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c1054b0 0x7f081c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.410+0000 7f0821600700 1 -- 172.21.15.43:0/1521594198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c1054b0 msgr2=0x7f081c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:07.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.410+0000 7f0821600700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c1054b0 0x7f081c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.410+0000 7f0821600700 1 -- 172.21.15.43:0/1521594198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f081c105e00 msgr2=0x7f081c19d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:07.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.410+0000 7f0821600700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f081c105e00 0x7f081c19d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.410+0000 7f0821600700 1 -- 172.21.15.43:0/1521594198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081c1a1dc0 con 0x7f081c10e8e0 2024-05-27T07:26:07.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.411+0000 7f0821600700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f081c10e8e0 0x7f081c1a1720 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f0814007ee0 tx=0x7f081400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:07.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.411+0000 7f0819600700 1 -- 172.21.15.43:0/1521594198 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081400dbd0 con 0x7f081c10e8e0 2024-05-27T07:26:07.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.411+0000 7f0823600700 1 -- 172.21.15.43:0/1521594198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f081c1a20b0 con 0x7f081c10e8e0 2024-05-27T07:26:07.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.411+0000 7f0819600700 1 -- 172.21.15.43:0/1521594198 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f081400dd70 con 0x7f081c10e8e0 2024-05-27T07:26:07.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.411+0000 7f0819600700 1 -- 172.21.15.43:0/1521594198 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08140058d0 con 0x7f081c10e8e0 2024-05-27T07:26:07.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.411+0000 7f0823600700 1 -- 172.21.15.43:0/1521594198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f081c1a99d0 con 0x7f081c10e8e0 2024-05-27T07:26:07.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.417+0000 7f0819600700 1 -- 172.21.15.43:0/1521594198 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f081401a020 con 0x7f081c10e8e0 2024-05-27T07:26:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.418+0000 7f08321abb80 1 Processor -- start 2024-05-27T07:26:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.418+0000 7f08321abb80 1 -- start start 2024-05-27T07:26:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.418+0000 7f0819600700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f08040b29f0 0x7f08040b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.418+0000 7f08321abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbac96a70 0x55dcbad293b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.418+0000 7f08321abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbabaaec0 0x55dcbad298f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.418+0000 7f08321abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad36190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.418+0000 7f08321abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dcbac57ec0 con 0x55dcbad29ef0 2024-05-27T07:26:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.419+0000 7f08321abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dcbac47a60 con 0x55dcbac96a70 2024-05-27T07:26:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.419+0000 7f08321abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55dcbac4baa0 con 0x55dcbabaaec0 2024-05-27T07:26:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.419+0000 7f0819600700 1 -- 172.21.15.43:0/1521594198 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9584+0+0 (secure 0 0 0) 0x7f08140e2120 con 0x7f081c10e8e0 2024-05-27T07:26:07.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.419+0000 7f081be00700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f08040b29f0 0x7f08040b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.419+0000 7f0821600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad36190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.419+0000 7f0821600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad36190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38774/0 (socket says 172.21.15.43:38774) 2024-05-27T07:26:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.419+0000 7f0821600700 1 -- 172.21.15.43:0/251532574 learned_addr learned my addr 172.21.15.43:0/251532574 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.419+0000 7f081be00700 1 --2- 172.21.15.43:0/251532574 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbabaaec0 0x55dcbad298f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.419+0000 7f0820c00700 1 --2- 172.21.15.43:0/251532574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbac96a70 0x55dcbad293b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.420+0000 7f0821600700 1 -- 172.21.15.43:0/251532574 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbabaaec0 msgr2=0x55dcbad298f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:07.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.420+0000 7f0821600700 1 --2- 172.21.15.43:0/251532574 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbabaaec0 0x55dcbad298f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.420+0000 7f0821600700 1 -- 172.21.15.43:0/251532574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbac96a70 msgr2=0x55dcbad293b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.420+0000 7f0821600700 1 --2- 172.21.15.43:0/251532574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbac96a70 0x55dcbad293b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.420+0000 7f0821600700 1 -- 172.21.15.43:0/251532574 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcbac104a0 con 0x55dcbad29ef0 2024-05-27T07:26:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.420+0000 7f081be00700 1 --2- 172.21.15.43:0/251532574 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbabaaec0 0x55dcbad298f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.420+0000 7f081be00700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f08040b29f0 0x7f08040b4f00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f080800b6c0 tx=0x7f0808012890 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.420+0000 7f0821600700 1 --2- 172.21.15.43:0/251532574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad36190 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f08140a8ce0 tx=0x7f08140a8ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cb762b584a7915 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.420+0000 7f080e000700 1 -- 172.21.15.43:0/251532574 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08140a8fe0 con 0x55dcbad29ef0 2024-05-27T07:26:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.421+0000 7f080e000700 1 -- 172.21.15.43:0/251532574 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08140ad390 con 0x55dcbad29ef0 2024-05-27T07:26:07.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.421+0000 7f080e000700 1 -- 172.21.15.43:0/251532574 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08140a9240 con 0x55dcbad29ef0 2024-05-27T07:26:07.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.421+0000 7f08321abb80 1 -- 172.21.15.43:0/251532574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 msgr2=0x55dcbad36190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:07.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.421+0000 7f08321abb80 1 --2- 172.21.15.43:0/251532574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad36190 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f08140a8ce0 tx=0x7f08140a8ee0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.422+0000 7f08321abb80 1 -- 172.21.15.43:0/251532574 shutdown_connections 2024-05-27T07:26:07.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.422+0000 7f08321abb80 1 --2- 172.21.15.43:0/251532574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad36190 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.422+0000 7f08321abb80 1 --2- 172.21.15.43:0/251532574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbac96a70 0x55dcbad293b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.422+0000 7f08321abb80 1 --2- 172.21.15.43:0/251532574 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbabaaec0 0x55dcbad298f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.422+0000 7f08321abb80 1 -- 172.21.15.43:0/251532574 >> 172.21.15.43:0/251532574 conn(0x55dcbac44410 msgr2=0x55dcbad25540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:07.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.422+0000 7f08321abb80 1 -- 172.21.15.43:0/251532574 shutdown_connections 2024-05-27T07:26:07.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.422+0000 7f08321abb80 1 -- 172.21.15.43:0/251532574 wait complete. 2024-05-27T07:26:07.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.422+0000 7f08321abb80 1 -- start start 2024-05-27T07:26:07.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.423+0000 7f08321abb80 1 Processor -- start 2024-05-27T07:26:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.423+0000 7f08321abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad33750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.423+0000 7f08321abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbac96a70 0x55dcbad33c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.423+0000 7f08321abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbad29ef0 0x55dcbad341d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.423+0000 7f08321abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dcbab4e020 con 0x55dcbabaaec0 2024-05-27T07:26:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.423+0000 7f08321abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dcbabe87e0 con 0x55dcbad29ef0 2024-05-27T07:26:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.423+0000 7f08321abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55dcbad36c00 con 0x55dcbac96a70 2024-05-27T07:26:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.423+0000 7f08321abb80 10 client.0 _ll_register_callbacks cb 0x55dcbabf1cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.424+0000 7f0820c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad33750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.424+0000 7f0820c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad33750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38782/0 (socket says 172.21.15.43:38782) 2024-05-27T07:26:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.424+0000 7f0820c00700 1 -- 172.21.15.43:0/3528697759 learned_addr learned my addr 172.21.15.43:0/3528697759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.424+0000 7f081be00700 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbac96a70 0x55dcbad33c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.424+0000 7f0821600700 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbad29ef0 0x55dcbad341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.424+0000 7f0820c00700 1 -- 172.21.15.43:0/3528697759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbac96a70 msgr2=0x55dcbad33c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.424+0000 7f0820c00700 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbac96a70 0x55dcbad33c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.424+0000 7f0820c00700 1 -- 172.21.15.43:0/3528697759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbad29ef0 msgr2=0x55dcbad341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.424+0000 7f0820c00700 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbad29ef0 0x55dcbad341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.424+0000 7f0820c00700 1 -- 172.21.15.43:0/3528697759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcbab97950 con 0x55dcbabaaec0 2024-05-27T07:26:07.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.425+0000 7f0820c00700 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad33750 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f0810009990 tx=0x7f081000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6ccdd8d8cbe95d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.425+0000 7f0802a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.425+0000 7f0802a00700 1 -- 172.21.15.43:0/3528697759 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0810003a50 con 0x55dcbabaaec0 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.425+0000 7f0802a00700 1 -- 172.21.15.43:0/3528697759 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9584+0+0 (secure 0 0 0) 0x7f0810003bf0 con 0x55dcbabaaec0 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.425+0000 7f08321abb80 1 -- 172.21.15.43:0/3528697759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08140ad390 con 0x55dcbabaaec0 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f0802a00700 1 -- 172.21.15.43:0/3528697759 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f0810003df0 con 0x55dcbabaaec0 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f08321abb80 10 client.17253 fetch_fsmap learned FSMap version 108 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f08321abb80 1 -- 172.21.15.43:0/3528697759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55dcbac180e0 con 0x55dcbabaaec0 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f0802a00700 1 -- 172.21.15.43:0/3528697759 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08140ad390 con 0x55dcbabaaec0 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f0802a00700 1 -- 172.21.15.43:0/3528697759 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f081000f740 con 0x55dcbabaaec0 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f08321abb80 10 client.17253 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f08321abb80 10 client.17253 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f08321abb80 20 client.17253 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f08321abb80 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dcbad440b0 0x55dcbad46580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f08321abb80 4 client.17253 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.426+0000 7f08321abb80 1 -- 172.21.15.43:0/3528697759 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcbac59e10 con 0x55dcbad440b0 2024-05-27T07:26:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.427+0000 7f081be00700 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dcbad440b0 0x55dcbad46580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.428+0000 7f081be00700 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dcbad440b0 0x55dcbad46580 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e3693e8e977bf77 server_cookie=742a5441bd5be07b in_seq=0 out_seq=0 2024-05-27T07:26:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.428+0000 7f0802a00700 10 client.17253 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f0802a00700 1 -- 172.21.15.43:0/3528697759 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dcbac59e10 con 0x55dcbad440b0 2024-05-27T07:26:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f0802a00700 10 client.17253 handle_command_reply: tid=0 2024-05-27T07:26:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f08321abb80 1 client.17253 shutdown 2024-05-27T07:26:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f08321abb80 1 -- 172.21.15.43:0/3528697759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 msgr2=0x55dcbad33750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f08321abb80 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad33750 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f0810009990 tx=0x7f081000bf40 comp rx=0 tx=0).stop 2024-05-27T07:26:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f08321abb80 1 -- 172.21.15.43:0/3528697759 shutdown_connections 2024-05-27T07:26:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f08321abb80 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbad29ef0 0x55dcbad341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f08321abb80 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dcbad440b0 0x55dcbad46580 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f08321abb80 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad33750 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f08321abb80 1 --2- 172.21.15.43:0/3528697759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbac96a70 0x55dcbad33c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f08321abb80 1 -- 172.21.15.43:0/3528697759 >> 172.21.15.43:0/3528697759 conn(0x55dcbac44410 msgr2=0x55dcbad29390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f0802a00700 0 client.17253 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.431+0000 7f08321abb80 1 -- 172.21.15.43:0/3528697759 shutdown_connections 2024-05-27T07:26:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.432+0000 7f08321abb80 1 -- 172.21.15.43:0/3528697759 wait complete. 2024-05-27T07:26:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.432+0000 7f08321abb80 20 client.17253 trim_cache size 0 max 16384 2024-05-27T07:26:07.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.462+0000 7f08321abb80 1 Processor -- start 2024-05-27T07:26:07.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.462+0000 7f08321abb80 1 -- start start 2024-05-27T07:26:07.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f08321abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbabaaec0 0x55dcbad2d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f08321abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbac96a70 0x55dcbad2ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f08321abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad4bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f08321abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1b3560 con 0x55dcbad29ef0 2024-05-27T07:26:07.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f08321abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1b1230 con 0x55dcbabaaec0 2024-05-27T07:26:07.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f08321abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1b2e90 con 0x55dcbac96a70 2024-05-27T07:26:07.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f0821600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad4bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f0821600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad4bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38798/0 (socket says 172.21.15.43:38798) 2024-05-27T07:26:07.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f0821600700 1 -- 172.21.15.43:0/1731791309 learned_addr learned my addr 172.21.15.43:0/1731791309 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:07.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f081be00700 1 --2- 172.21.15.43:0/1731791309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbac96a70 0x55dcbad2ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.463+0000 7f0820c00700 1 --2- 172.21.15.43:0/1731791309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbabaaec0 0x55dcbad2d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.464+0000 7f0821600700 1 -- 172.21.15.43:0/1731791309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbac96a70 msgr2=0x55dcbad2ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:07.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.464+0000 7f0821600700 1 --2- 172.21.15.43:0/1731791309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbac96a70 0x55dcbad2ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.464+0000 7f0821600700 1 -- 172.21.15.43:0/1731791309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbabaaec0 msgr2=0x55dcbad2d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:07.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.464+0000 7f0821600700 1 --2- 172.21.15.43:0/1731791309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbabaaec0 0x55dcbad2d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.464+0000 7f0821600700 1 -- 172.21.15.43:0/1731791309 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcbad2aad0 con 0x55dcbad29ef0 2024-05-27T07:26:07.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.464+0000 7f0821600700 1 --2- 172.21.15.43:0/1731791309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad4bd40 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f08140a8f70 tx=0x7f0814004e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e254d902d8f37f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:07.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.464+0000 7f080d600700 1 -- 172.21.15.43:0/1731791309 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0814005a70 con 0x55dcbad29ef0 2024-05-27T07:26:07.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.464+0000 7f080d600700 1 -- 172.21.15.43:0/1731791309 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08140a8fe0 con 0x55dcbad29ef0 2024-05-27T07:26:07.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.465+0000 7f080d600700 1 -- 172.21.15.43:0/1731791309 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08140a94a0 con 0x55dcbad29ef0 2024-05-27T07:26:07.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.466+0000 7f08321abb80 1 -- 172.21.15.43:0/1731791309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 msgr2=0x55dcbad4bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:07.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.466+0000 7f08321abb80 1 --2- 172.21.15.43:0/1731791309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad4bd40 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f08140a8f70 tx=0x7f0814004e10 comp rx=0 tx=0).stop 2024-05-27T07:26:07.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.466+0000 7f08321abb80 1 -- 172.21.15.43:0/1731791309 shutdown_connections 2024-05-27T07:26:07.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.466+0000 7f08321abb80 1 --2- 172.21.15.43:0/1731791309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbad29ef0 0x55dcbad4bd40 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.466+0000 7f08321abb80 1 --2- 172.21.15.43:0/1731791309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbabaaec0 0x55dcbad2d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.466+0000 7f08321abb80 1 --2- 172.21.15.43:0/1731791309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbac96a70 0x55dcbad2ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.466+0000 7f08321abb80 1 -- 172.21.15.43:0/1731791309 >> 172.21.15.43:0/1731791309 conn(0x55dcbac44410 msgr2=0x55dcbad297a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:07.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.466+0000 7f08321abb80 1 -- 172.21.15.43:0/1731791309 shutdown_connections 2024-05-27T07:26:07.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.466+0000 7f08321abb80 1 -- 172.21.15.43:0/1731791309 wait complete. 2024-05-27T07:26:07.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.466+0000 7f08321abb80 1 -- start start 2024-05-27T07:26:07.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.467+0000 7f08321abb80 1 Processor -- start 2024-05-27T07:26:07.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.467+0000 7f08321abb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad38340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.467+0000 7f08321abb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbac96a70 0x55dcbad38880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.467+0000 7f08321abb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbad29ef0 0x55dcbadb6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.467+0000 7f08321abb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1b2030 con 0x55dcbabaaec0 2024-05-27T07:26:07.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.467+0000 7f08321abb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1b1900 con 0x55dcbac96a70 2024-05-27T07:26:07.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.467+0000 7f08321abb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1b2760 con 0x55dcbad29ef0 2024-05-27T07:26:07.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.467+0000 7f08321abb80 10 client.0 _ll_register_callbacks cb 0x55dcbabf1cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:07.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0820c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad38340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0820c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad38340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38802/0 (socket says 172.21.15.43:38802) 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0820c00700 1 -- 172.21.15.43:0/1502905281 learned_addr learned my addr 172.21.15.43:0/1502905281 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0821600700 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbad29ef0 0x55dcbadb6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f081be00700 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbac96a70 0x55dcbad38880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0820c00700 1 -- 172.21.15.43:0/1502905281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbad29ef0 msgr2=0x55dcbadb6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0820c00700 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbad29ef0 0x55dcbadb6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0820c00700 1 -- 172.21.15.43:0/1502905281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbac96a70 msgr2=0x55dcbad38880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0820c00700 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbac96a70 0x55dcbad38880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0820c00700 1 -- 172.21.15.43:0/1502905281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcbadb6960 con 0x55dcbabaaec0 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0821600700 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbad29ef0 0x55dcbadb6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0820c00700 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad38340 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f08100098e0 tx=0x7f0810003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca906d0875b787a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.468+0000 7f0802a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.469+0000 7f0802a00700 1 -- 172.21.15.43:0/1502905281 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0810003680 con 0x55dcbabaaec0 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.469+0000 7f0802a00700 1 -- 172.21.15.43:0/1502905281 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9584+0+0 (secure 0 0 0) 0x7f081000f870 con 0x55dcbabaaec0 2024-05-27T07:26:07.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.469+0000 7f08321abb80 1 -- 172.21.15.43:0/1502905281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08140a8fe0 con 0x55dcbabaaec0 2024-05-27T07:26:07.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.469+0000 7f0802a00700 1 -- 172.21.15.43:0/1502905281 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f0810042050 con 0x55dcbabaaec0 2024-05-27T07:26:07.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.469+0000 7f08321abb80 10 client.17259 fetch_fsmap learned FSMap version 108 2024-05-27T07:26:07.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.469+0000 7f08321abb80 1 -- 172.21.15.43:0/1502905281 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x55dcbadb7bd0 con 0x55dcbabaaec0 2024-05-27T07:26:07.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.470+0000 7f0802a00700 1 -- 172.21.15.43:0/1502905281 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08140a8fe0 con 0x55dcbabaaec0 2024-05-27T07:26:07.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.470+0000 7f0802a00700 1 -- 172.21.15.43:0/1502905281 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f081000ea60 con 0x55dcbabaaec0 2024-05-27T07:26:07.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.470+0000 7f08321abb80 10 client.17259 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:26:07.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.470+0000 7f08321abb80 10 client.17259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:07.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.470+0000 7f08321abb80 20 client.17259 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:07.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.470+0000 7f08321abb80 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dcbadb82d0 0x55dcbadba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:07.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.470+0000 7f08321abb80 4 client.17259 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:26:07.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.470+0000 7f08321abb80 1 -- 172.21.15.43:0/1502905281 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dcbac535f0 con 0x55dcbadb82d0 2024-05-27T07:26:07.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.471+0000 7f081be00700 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dcbadb82d0 0x55dcbadba6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:07.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.472+0000 7f081be00700 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dcbadb82d0 0x55dcbadba6f0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e697dbe75d708470 server_cookie=90bbddc7f8cdd3d1 in_seq=0 out_seq=0 2024-05-27T07:26:07.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.472+0000 7f0802a00700 10 client.17259 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:07.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.475+0000 7f0802a00700 1 -- 172.21.15.43:0/1502905281 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dcbac535f0 con 0x55dcbadb82d0 2024-05-27T07:26:07.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.475+0000 7f0802a00700 10 client.17259 handle_command_reply: tid=0 2024-05-27T07:26:07.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.475+0000 7f08321abb80 1 client.17259 shutdown 2024-05-27T07:26:07.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.475+0000 7f08321abb80 1 -- 172.21.15.43:0/1502905281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 msgr2=0x55dcbad38340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:07.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.475+0000 7f08321abb80 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad38340 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f08100098e0 tx=0x7f0810003c10 comp rx=0 tx=0).stop 2024-05-27T07:26:07.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.475+0000 7f08321abb80 1 -- 172.21.15.43:0/1502905281 shutdown_connections 2024-05-27T07:26:07.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.475+0000 7f08321abb80 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dcbad29ef0 0x55dcbadb6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.475+0000 7f08321abb80 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dcbadb82d0 0x55dcbadba6f0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.476+0000 7f08321abb80 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dcbabaaec0 0x55dcbad38340 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.476+0000 7f08321abb80 1 --2- 172.21.15.43:0/1502905281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dcbac96a70 0x55dcbad38880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.476+0000 7f08321abb80 1 -- 172.21.15.43:0/1502905281 >> 172.21.15.43:0/1502905281 conn(0x55dcbac44410 msgr2=0x55dcbad297a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:07.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.476+0000 7f0802a00700 0 client.17259 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:07.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.476+0000 7f08321abb80 1 -- 172.21.15.43:0/1502905281 shutdown_connections 2024-05-27T07:26:07.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.476+0000 7f08321abb80 1 -- 172.21.15.43:0/1502905281 wait complete. 2024-05-27T07:26:07.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.476+0000 7f08321abb80 20 client.17259 trim_cache size 0 max 16384 2024-05-27T07:26:07.467 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:26:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:26:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "06695201-8a8c-4c7b-845d-79f273789ba1", 2024-05-27T07:26:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:26:07.467 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:26:07.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 -- 172.21.15.43:0/1521594198 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f08040b29f0 msgr2=0x7f08040b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f08040b29f0 0x7f08040b4f00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f080800b6c0 tx=0x7f0808012890 comp rx=0 tx=0).stop 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 -- 172.21.15.43:0/1521594198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f081c10e8e0 msgr2=0x7f081c1a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f081c10e8e0 0x7f081c1a1720 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f0814007ee0 tx=0x7f081400d3b0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 -- 172.21.15.43:0/1521594198 shutdown_connections 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f08040b29f0 0x7f08040b4f00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f081c10e8e0 0x7f081c1a1720 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c1054b0 0x7f081c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 --2- 172.21.15.43:0/1521594198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f081c105e00 0x7f081c19d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 -- 172.21.15.43:0/1521594198 >> 172.21.15.43:0/1521594198 conn(0x7f081c0fedb0 msgr2=0x7f081c1031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 -- 172.21.15.43:0/1521594198 shutdown_connections 2024-05-27T07:26:07.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:07.477+0000 7f0823600700 1 -- 172.21.15.43:0/1521594198 wait complete. 2024-05-27T07:26:07.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:07 smithi043 ceph-mon[31531]: pgmap v411: 129 pgs: 129 active+clean; 127 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 467 KiB/s wr, 53 op/s 2024-05-27T07:26:07.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:07 smithi179 ceph-mon[35888]: pgmap v411: 129 pgs: 129 active+clean; 127 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 467 KiB/s wr, 53 op/s 2024-05-27T07:26:07.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:07 smithi132 ceph-mon[35283]: pgmap v411: 129 pgs: 129 active+clean; 127 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 467 KiB/s wr, 53 op/s 2024-05-27T07:26:08.239 INFO:tasks.workunit.client.0.smithi043.stdout: 1 68245 0.58 MB/sec execute 492 sec latency 59.613 ms 2024-05-27T07:26:08.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:26:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:26:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:26:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:26:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:26:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:26:08.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:26:08.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:26:08.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:08 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:26:08.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:26:08.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:26:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:26:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:26:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:26:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:26:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:08 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:08.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:08.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:08 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:26:09.239 INFO:tasks.workunit.client.0.smithi043.stdout: 1 68362 0.58 MB/sec execute 493 sec latency 50.960 ms 2024-05-27T07:26:09.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:09 smithi043 ceph-mon[31531]: pgmap v412: 129 pgs: 129 active+clean; 127 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 467 KiB/s wr, 53 op/s 2024-05-27T07:26:09.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:09 smithi179 ceph-mon[35888]: pgmap v412: 129 pgs: 129 active+clean; 127 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 467 KiB/s wr, 53 op/s 2024-05-27T07:26:09.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:09 smithi132 ceph-mon[35283]: pgmap v412: 129 pgs: 129 active+clean; 127 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 467 KiB/s wr, 53 op/s 2024-05-27T07:26:10.239 INFO:tasks.workunit.client.0.smithi043.stdout: 1 68482 0.58 MB/sec execute 494 sec latency 56.088 ms 2024-05-27T07:26:11.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.032+0000 7f184f200700 1 -- 172.21.15.43:0/269385772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1848108d20 msgr2=0x7f1848109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.032+0000 7f184f200700 1 --2- 172.21.15.43:0/269385772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1848108d20 0x7f1848109130 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f18380099c0 tx=0x7f1838009cd0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.032+0000 7f184f200700 1 -- 172.21.15.43:0/269385772 shutdown_connections 2024-05-27T07:26:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.032+0000 7f184f200700 1 --2- 172.21.15.43:0/269385772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1848109670 0x7f1848110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.032+0000 7f184f200700 1 --2- 172.21.15.43:0/269385772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18481043f0 0x7f1848104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.032+0000 7f184f200700 1 --2- 172.21.15.43:0/269385772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1848108d20 0x7f1848109130 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.032+0000 7f184f200700 1 -- 172.21.15.43:0/269385772 >> 172.21.15.43:0/269385772 conn(0x7f1848079d00 msgr2=0x7f184807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.032+0000 7f184f200700 1 -- 172.21.15.43:0/269385772 shutdown_connections 2024-05-27T07:26:11.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.032+0000 7f184f200700 1 -- 172.21.15.43:0/269385772 wait complete. 2024-05-27T07:26:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.033+0000 7f184f200700 1 Processor -- start 2024-05-27T07:26:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.034+0000 7f184f200700 1 -- start start 2024-05-27T07:26:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.034+0000 7f184f200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18481043f0 0x7f184819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.034+0000 7f184f200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1848108d20 0x7f184819d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.034+0000 7f184f200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1848109670 0x7f18481a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.034+0000 7f184f200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1848114130 con 0x7f1848108d20 2024-05-27T07:26:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.034+0000 7f184f200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1848113fb0 con 0x7f18481043f0 2024-05-27T07:26:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.034+0000 7f184f200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f18481142b0 con 0x7f1848109670 2024-05-27T07:26:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1848108d20 0x7f184819d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1848108d20 0x7f184819d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38818/0 (socket says 172.21.15.43:38818) 2024-05-27T07:26:11.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184d400700 1 -- 172.21.15.43:0/1102058185 learned_addr learned my addr 172.21.15.43:0/1102058185 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184de00700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18481043f0 0x7f184819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184e800700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1848109670 0x7f18481a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184d400700 1 -- 172.21.15.43:0/1102058185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1848109670 msgr2=0x7f18481a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184d400700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1848109670 0x7f18481a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184d400700 1 -- 172.21.15.43:0/1102058185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18481043f0 msgr2=0x7f184819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184d400700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18481043f0 0x7f184819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184d400700 1 -- 172.21.15.43:0/1102058185 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18481a1c70 con 0x7f1848108d20 2024-05-27T07:26:11.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.035+0000 7f184e800700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1848109670 0x7f18481a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:11.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.036+0000 7f184d400700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1848108d20 0x7f184819d340 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f183800b580 tx=0x7f18380057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.036+0000 7f1842a00700 1 -- 172.21.15.43:0/1102058185 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838027070 con 0x7f1848108d20 2024-05-27T07:26:11.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.036+0000 7f184f200700 1 -- 172.21.15.43:0/1102058185 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18481a1f00 con 0x7f1848108d20 2024-05-27T07:26:11.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.036+0000 7f1842a00700 1 -- 172.21.15.43:0/1102058185 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f183800fd30 con 0x7f1848108d20 2024-05-27T07:26:11.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.036+0000 7f1842a00700 1 -- 172.21.15.43:0/1102058185 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18380173f0 con 0x7f1848108d20 2024-05-27T07:26:11.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.036+0000 7f184f200700 1 -- 172.21.15.43:0/1102058185 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18481a23e0 con 0x7f1848108d20 2024-05-27T07:26:11.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.041+0000 7f1842a00700 1 -- 172.21.15.43:0/1102058185 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f183802a070 con 0x7f1848108d20 2024-05-27T07:26:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.042+0000 7f185f30ab80 1 Processor -- start 2024-05-27T07:26:11.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.043+0000 7f185f30ab80 1 -- start start 2024-05-27T07:26:11.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.043+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5bded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.043+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.043+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5cb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.043+0000 7f185f30ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555ccc4ece40 con 0x555ccc43fe40 2024-05-27T07:26:11.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.043+0000 7f185f30ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555ccc4dc9e0 con 0x555ccc5be410 2024-05-27T07:26:11.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.043+0000 7f185f30ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555ccc4e0a20 con 0x555ccc5be820 2024-05-27T07:26:11.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.043+0000 7f1842a00700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18300b29f0 0x7f18300b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.043+0000 7f184e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5bded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f184e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5bded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38826/0 (socket says 172.21.15.43:38826) 2024-05-27T07:26:11.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f184e800700 1 -- 172.21.15.43:0/3345978536 learned_addr learned my addr 172.21.15.43:0/3345978536 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:11.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f184de00700 1 --2- 172.21.15.43:0/3345978536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f184e800700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18300b29f0 0x7f18300b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f1842a00700 1 -- 172.21.15.43:0/1102058185 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9584+0+0 (secure 0 0 0) 0x7f18380e3910 con 0x7f1848108d20 2024-05-27T07:26:11.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f184d400700 1 --2- 172.21.15.43:0/3345978536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f184e800700 1 -- 172.21.15.43:0/3345978536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 msgr2=0x555ccc5cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f184e800700 1 --2- 172.21.15.43:0/3345978536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f184e800700 1 -- 172.21.15.43:0/3345978536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 msgr2=0x555ccc5c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f184e800700 1 --2- 172.21.15.43:0/3345978536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.044+0000 7f184e800700 1 -- 172.21.15.43:0/3345978536 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ccc4a5420 con 0x555ccc43fe40 2024-05-27T07:26:11.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.045+0000 7f184e800700 1 --2- 172.21.15.43:0/3345978536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5bded0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f1844007f00 tx=0x7f1844018640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e935307da14d1fb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.045+0000 7f184e800700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18300b29f0 0x7f18300b4f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1844009710 tx=0x7f184400b000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.045+0000 7f1837400700 1 -- 172.21.15.43:0/3345978536 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184401e070 con 0x555ccc43fe40 2024-05-27T07:26:11.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.045+0000 7f1837400700 1 -- 172.21.15.43:0/3345978536 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1844009b90 con 0x555ccc43fe40 2024-05-27T07:26:11.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.045+0000 7f1837400700 1 -- 172.21.15.43:0/3345978536 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844005020 con 0x555ccc43fe40 2024-05-27T07:26:11.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.046+0000 7f185f30ab80 1 -- 172.21.15.43:0/3345978536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 msgr2=0x555ccc5bded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.046+0000 7f185f30ab80 1 --2- 172.21.15.43:0/3345978536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5bded0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f1844007f00 tx=0x7f1844018640 comp rx=0 tx=0).stop 2024-05-27T07:26:11.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.047+0000 7f185f30ab80 1 -- 172.21.15.43:0/3345978536 shutdown_connections 2024-05-27T07:26:11.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.047+0000 7f185f30ab80 1 --2- 172.21.15.43:0/3345978536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.047+0000 7f185f30ab80 1 --2- 172.21.15.43:0/3345978536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5bded0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.047+0000 7f185f30ab80 1 --2- 172.21.15.43:0/3345978536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.047+0000 7f185f30ab80 1 -- 172.21.15.43:0/3345978536 >> 172.21.15.43:0/3345978536 conn(0x555ccc49ffc0 msgr2=0x555ccc5b9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.047+0000 7f185f30ab80 1 -- 172.21.15.43:0/3345978536 shutdown_connections 2024-05-27T07:26:11.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.047+0000 7f185f30ab80 1 -- 172.21.15.43:0/3345978536 wait complete. 2024-05-27T07:26:11.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.048+0000 7f185f30ab80 1 -- start start 2024-05-27T07:26:11.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.048+0000 7f185f30ab80 1 Processor -- start 2024-05-27T07:26:11.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.048+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc43fe40 0x555ccc5c8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc5be410 0x555ccc5c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5c9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f185f30ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555ccc3e2fa0 con 0x555ccc5be410 2024-05-27T07:26:11.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f185f30ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555ccc47d760 con 0x555ccc43fe40 2024-05-27T07:26:11.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f185f30ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555ccc5cb8d0 con 0x555ccc5be820 2024-05-27T07:26:11.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f185f30ab80 10 client.0 _ll_register_callbacks cb 0x555ccc486c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:11.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f184d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc5be410 0x555ccc5c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f184d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc5be410 0x555ccc5c8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38830/0 (socket says 172.21.15.43:38830) 2024-05-27T07:26:11.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f184d400700 1 -- 172.21.15.43:0/3871905498 learned_addr learned my addr 172.21.15.43:0/3871905498 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:11.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f184de00700 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc43fe40 0x555ccc5c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f184e800700 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f184d400700 1 -- 172.21.15.43:0/3871905498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 msgr2=0x555ccc5c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f184d400700 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f184d400700 1 -- 172.21.15.43:0/3871905498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc43fe40 msgr2=0x555ccc5c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f184d400700 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc43fe40 0x555ccc5c8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.049+0000 7f184d400700 1 -- 172.21.15.43:0/3871905498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ccc42c8d0 con 0x555ccc5be410 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.050+0000 7f184e800700 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.050+0000 7f184d400700 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc5be410 0x555ccc5c8b50 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f18380a68a0 tx=0x7f18380e1160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8068cb433e5059a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.050+0000 7f182be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.050+0000 7f182be00700 1 -- 172.21.15.43:0/3871905498 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18380a9f90 con 0x555ccc5be410 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.050+0000 7f182be00700 1 -- 172.21.15.43:0/3871905498 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9584+0+0 (secure 0 0 0) 0x7f1838022470 con 0x555ccc5be410 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.050+0000 7f185f30ab80 1 -- 172.21.15.43:0/3871905498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1844009b90 con 0x555ccc5be410 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f182be00700 1 -- 172.21.15.43:0/3871905498 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f18380aa330 con 0x555ccc5be410 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f185f30ab80 10 client.17271 fetch_fsmap learned FSMap version 108 2024-05-27T07:26:11.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f185f30ab80 1 -- 172.21.15.43:0/3871905498 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x555ccc4ad060 con 0x555ccc5be410 2024-05-27T07:26:11.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f182be00700 1 -- 172.21.15.43:0/3871905498 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1844009b90 con 0x555ccc5be410 2024-05-27T07:26:11.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f182be00700 1 -- 172.21.15.43:0/3871905498 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f18380920e0 con 0x555ccc5be410 2024-05-27T07:26:11.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f185f30ab80 10 client.17271 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:26:11.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f185f30ab80 10 client.17271 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:11.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f185f30ab80 20 client.17271 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:11.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f185f30ab80 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555ccc5d8aa0 0x555ccc5daf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f185f30ab80 4 client.17271 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:11.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.051+0000 7f185f30ab80 1 -- 172.21.15.43:0/3871905498 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ccc4eed90 con 0x555ccc5d8aa0 2024-05-27T07:26:11.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.052+0000 7f184de00700 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555ccc5d8aa0 0x555ccc5daf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.054+0000 7f184de00700 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555ccc5d8aa0 0x555ccc5daf10 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69d44b5739ed00f2 server_cookie=3218d1e0ae934314 in_seq=0 out_seq=0 2024-05-27T07:26:11.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.054+0000 7f182be00700 10 client.17271 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f182be00700 1 -- 172.21.15.43:0/3871905498 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555ccc4eed90 con 0x555ccc5d8aa0 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f182be00700 10 client.17271 handle_command_reply: tid=0 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 client.17271 shutdown 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 -- 172.21.15.43:0/3871905498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc5be410 msgr2=0x555ccc5c8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc5be410 0x555ccc5c8b50 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f18380a68a0 tx=0x7f18380e1160 comp rx=0 tx=0).stop 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 -- 172.21.15.43:0/3871905498 shutdown_connections 2024-05-27T07:26:11.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc43fe40 0x555ccc5c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555ccc5d8aa0 0x555ccc5daf10 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 --2- 172.21.15.43:0/3871905498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc5be410 0x555ccc5c8b50 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 -- 172.21.15.43:0/3871905498 >> 172.21.15.43:0/3871905498 conn(0x555ccc49ffc0 msgr2=0x555ccc5b8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f182be00700 0 client.17271 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 -- 172.21.15.43:0/3871905498 shutdown_connections 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.057+0000 7f185f30ab80 1 -- 172.21.15.43:0/3871905498 wait complete. 2024-05-27T07:26:11.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.058+0000 7f185f30ab80 20 client.17271 trim_cache size 0 max 16384 2024-05-27T07:26:11.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.081+0000 7f185f30ab80 1 Processor -- start 2024-05-27T07:26:11.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.081+0000 7f185f30ab80 1 -- start start 2024-05-27T07:26:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.082+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.082+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5d43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.082+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5e06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.082+0000 7f185f30ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555ccc5b9aa0 con 0x555ccc43fe40 2024-05-27T07:26:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.082+0000 7f185f30ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f18481aee90 con 0x555ccc5be410 2024-05-27T07:26:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.082+0000 7f185f30ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f18481acb60 con 0x555ccc5be820 2024-05-27T07:26:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.082+0000 7f184de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.082+0000 7f184de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38832/0 (socket says 172.21.15.43:38832) 2024-05-27T07:26:11.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.082+0000 7f184de00700 1 -- 172.21.15.43:0/124971992 learned_addr learned my addr 172.21.15.43:0/124971992 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:11.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.082+0000 7f184e800700 1 --2- 172.21.15.43:0/124971992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5e06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.083+0000 7f184d400700 1 --2- 172.21.15.43:0/124971992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5d43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.083+0000 7f184de00700 1 -- 172.21.15.43:0/124971992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 msgr2=0x555ccc5e06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:11.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.083+0000 7f184de00700 1 --2- 172.21.15.43:0/124971992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5e06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.083+0000 7f184de00700 1 -- 172.21.15.43:0/124971992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 msgr2=0x555ccc5d43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:11.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.083+0000 7f184de00700 1 --2- 172.21.15.43:0/124971992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5d43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.083+0000 7f184de00700 1 -- 172.21.15.43:0/124971992 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ccc5b41d0 con 0x555ccc43fe40 2024-05-27T07:26:11.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.084+0000 7f184de00700 1 --2- 172.21.15.43:0/124971992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5c3e30 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x555ccc5bb050 tx=0x7f183c00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4a30375a21c7f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.084+0000 7f1836a00700 1 -- 172.21.15.43:0/124971992 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c003ac0 con 0x555ccc43fe40 2024-05-27T07:26:11.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.084+0000 7f1836a00700 1 -- 172.21.15.43:0/124971992 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f183c004cc0 con 0x555ccc43fe40 2024-05-27T07:26:11.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.084+0000 7f1836a00700 1 -- 172.21.15.43:0/124971992 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c0054b0 con 0x555ccc43fe40 2024-05-27T07:26:11.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.085+0000 7f185f30ab80 1 -- 172.21.15.43:0/124971992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 msgr2=0x555ccc5c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.085+0000 7f185f30ab80 1 --2- 172.21.15.43:0/124971992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5c3e30 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x555ccc5bb050 tx=0x7f183c00b410 comp rx=0 tx=0).stop 2024-05-27T07:26:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.086+0000 7f185f30ab80 1 -- 172.21.15.43:0/124971992 shutdown_connections 2024-05-27T07:26:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.086+0000 7f185f30ab80 1 --2- 172.21.15.43:0/124971992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5e06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.086+0000 7f185f30ab80 1 --2- 172.21.15.43:0/124971992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5c3e30 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.086+0000 7f185f30ab80 1 --2- 172.21.15.43:0/124971992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5d43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.086+0000 7f185f30ab80 1 -- 172.21.15.43:0/124971992 >> 172.21.15.43:0/124971992 conn(0x555ccc49ffc0 msgr2=0x555ccc5c8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.086+0000 7f185f30ab80 1 -- 172.21.15.43:0/124971992 shutdown_connections 2024-05-27T07:26:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.086+0000 7f185f30ab80 1 -- 172.21.15.43:0/124971992 wait complete. 2024-05-27T07:26:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.086+0000 7f185f30ab80 1 -- start start 2024-05-27T07:26:11.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.087+0000 7f185f30ab80 1 Processor -- start 2024-05-27T07:26:11.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.087+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.087+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.087+0000 7f185f30ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.087+0000 7f185f30ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f18481ad230 con 0x555ccc43fe40 2024-05-27T07:26:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.087+0000 7f185f30ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f18481ae7c0 con 0x555ccc5be410 2024-05-27T07:26:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.087+0000 7f185f30ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f18481ad960 con 0x555ccc5be820 2024-05-27T07:26:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.087+0000 7f185f30ab80 10 client.0 _ll_register_callbacks cb 0x555ccc486c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.088+0000 7f184de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.088+0000 7f184de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38848/0 (socket says 172.21.15.43:38848) 2024-05-27T07:26:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.088+0000 7f184de00700 1 -- 172.21.15.43:0/1555770074 learned_addr learned my addr 172.21.15.43:0/1555770074 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.088+0000 7f184d400700 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.088+0000 7f184e800700 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.088+0000 7f184de00700 1 -- 172.21.15.43:0/1555770074 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 msgr2=0x555ccc5cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.088+0000 7f184de00700 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.088+0000 7f184de00700 1 -- 172.21.15.43:0/1555770074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 msgr2=0x555ccc5ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.088+0000 7f184de00700 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.088+0000 7f184de00700 1 -- 172.21.15.43:0/1555770074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ccc5bb350 con 0x555ccc43fe40 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.089+0000 7f184de00700 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5cc8b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f183c0049f0 tx=0x7f183c00b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4ab8c5f42883ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.089+0000 7f182be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.089+0000 7f182be00700 1 -- 172.21.15.43:0/1555770074 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c003cd0 con 0x555ccc43fe40 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.089+0000 7f182be00700 1 -- 172.21.15.43:0/1555770074 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9584+0+0 (secure 0 0 0) 0x7f183c003e70 con 0x555ccc43fe40 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.089+0000 7f185f30ab80 1 -- 172.21.15.43:0/1555770074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f183c004cc0 con 0x555ccc43fe40 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.089+0000 7f182be00700 1 -- 172.21.15.43:0/1555770074 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (secure 0 0 0) 0x7f183c039050 con 0x555ccc43fe40 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.089+0000 7f185f30ab80 10 client.17277 fetch_fsmap learned FSMap version 108 2024-05-27T07:26:11.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.089+0000 7f185f30ab80 1 -- 172.21.15.43:0/1555770074 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=108,monmap=4+}) -- 0x555ccc5bf1d0 con 0x555ccc43fe40 2024-05-27T07:26:11.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.089+0000 7f182be00700 1 -- 172.21.15.43:0/1555770074 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f183c004cc0 con 0x555ccc43fe40 2024-05-27T07:26:11.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.090+0000 7f182be00700 1 -- 172.21.15.43:0/1555770074 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 108) ==== 4804+0+0 (secure 0 0 0) 0x7f183c027880 con 0x555ccc43fe40 2024-05-27T07:26:11.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.090+0000 7f185f30ab80 10 client.17277 fetch_fsmap finished waiting for FSMap version 108 2024-05-27T07:26:11.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.090+0000 7f185f30ab80 10 client.17277 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:11.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.090+0000 7f185f30ab80 20 client.17277 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:11.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.090+0000 7f185f30ab80 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555ccc643b10 0x555ccc645f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:11.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.090+0000 7f185f30ab80 4 client.17277 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:26:11.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.090+0000 7f185f30ab80 1 -- 172.21.15.43:0/1555770074 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555ccc4e8570 con 0x555ccc643b10 2024-05-27T07:26:11.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.090+0000 7f184d400700 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555ccc643b10 0x555ccc645f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:11.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.091+0000 7f184d400700 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555ccc643b10 0x555ccc645f30 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4beac6a1958363a7 server_cookie=a21a6aac603b8d2f in_seq=0 out_seq=0 2024-05-27T07:26:11.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.091+0000 7f182be00700 10 client.17277 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.096+0000 7f182be00700 1 -- 172.21.15.43:0/1555770074 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555ccc4e8570 con 0x555ccc643b10 2024-05-27T07:26:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.096+0000 7f182be00700 10 client.17277 handle_command_reply: tid=0 2024-05-27T07:26:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.096+0000 7f185f30ab80 1 client.17277 shutdown 2024-05-27T07:26:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.096+0000 7f185f30ab80 1 -- 172.21.15.43:0/1555770074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 msgr2=0x555ccc5cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.096+0000 7f185f30ab80 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5cc8b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f183c0049f0 tx=0x7f183c00b9e0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.096+0000 7f185f30ab80 1 -- 172.21.15.43:0/1555770074 shutdown_connections 2024-05-27T07:26:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.097+0000 7f185f30ab80 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ccc5be820 0x555ccc5cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.097+0000 7f185f30ab80 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x555ccc643b10 0x555ccc645f30 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.097+0000 7f185f30ab80 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555ccc43fe40 0x555ccc5cc8b0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.097+0000 7f185f30ab80 1 --2- 172.21.15.43:0/1555770074 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555ccc5be410 0x555ccc5ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.097+0000 7f185f30ab80 1 -- 172.21.15.43:0/1555770074 >> 172.21.15.43:0/1555770074 conn(0x555ccc49ffc0 msgr2=0x555ccc5c8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.097+0000 7f182be00700 0 client.17277 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.097+0000 7f185f30ab80 1 -- 172.21.15.43:0/1555770074 shutdown_connections 2024-05-27T07:26:11.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.097+0000 7f185f30ab80 1 -- 172.21.15.43:0/1555770074 wait complete. 2024-05-27T07:26:11.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.097+0000 7f185f30ab80 20 client.17277 trim_cache size 0 max 16384 2024-05-27T07:26:11.088 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:26:11.088 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:26:11.088 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:26:11.088 INFO:teuthology.orchestra.run.smithi043.stdout: "06695201-8a8c-4c7b-845d-79f273789ba1": { 2024-05-27T07:26:11.088 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:26:11.088 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "06695201-8a8c-4c7b-845d-79f273789ba1", 2024-05-27T07:26:11.088 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:26:11.088 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:26:11.089 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:26:11.089 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:26:11.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.098+0000 7f184f200700 1 -- 172.21.15.43:0/1102058185 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18300b29f0 msgr2=0x7f18300b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.098+0000 7f184f200700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18300b29f0 0x7f18300b4f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1844009710 tx=0x7f184400b000 comp rx=0 tx=0).stop 2024-05-27T07:26:11.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.098+0000 7f184f200700 1 -- 172.21.15.43:0/1102058185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1848108d20 msgr2=0x7f184819d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:11.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.098+0000 7f184f200700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1848108d20 0x7f184819d340 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f183800b580 tx=0x7f18380057c0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.098+0000 7f184f200700 1 -- 172.21.15.43:0/1102058185 shutdown_connections 2024-05-27T07:26:11.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.098+0000 7f184f200700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f18300b29f0 0x7f18300b4f00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.098+0000 7f184f200700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1848109670 0x7f18481a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.098+0000 7f184f200700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18481043f0 0x7f184819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.098+0000 7f184f200700 1 --2- 172.21.15.43:0/1102058185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1848108d20 0x7f184819d340 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:11.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.098+0000 7f184f200700 1 -- 172.21.15.43:0/1102058185 >> 172.21.15.43:0/1102058185 conn(0x7f1848079d00 msgr2=0x7f184810f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:11.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.099+0000 7f184f200700 1 -- 172.21.15.43:0/1102058185 shutdown_connections 2024-05-27T07:26:11.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:11.099+0000 7f184f200700 1 -- 172.21.15.43:0/1102058185 wait complete. 2024-05-27T07:26:11.239 INFO:tasks.workunit.client.0.smithi043.stdout: 1 68602 0.58 MB/sec execute 495 sec latency 46.645 ms 2024-05-27T07:26:11.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:11 smithi043 ceph-mon[31531]: pgmap v413: 129 pgs: 129 active+clean; 127 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 467 KiB/s wr, 53 op/s 2024-05-27T07:26:11.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:11 smithi043 ceph-mon[31531]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:26:11.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:11 smithi179 ceph-mon[35888]: pgmap v413: 129 pgs: 129 active+clean; 127 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 467 KiB/s wr, 53 op/s 2024-05-27T07:26:11.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:11 smithi179 ceph-mon[35888]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:26:11.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:11 smithi132 ceph-mon[35283]: pgmap v413: 129 pgs: 129 active+clean; 127 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 467 KiB/s wr, 53 op/s 2024-05-27T07:26:11.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:11 smithi132 ceph-mon[35283]: osdmap e133: 12 total, 12 up, 12 in 2024-05-27T07:26:12.025 INFO:tasks.cephfs.filesystem:scrub status for tag:06695201-8a8c-4c7b-845d-79f273789ba1 - {'path': '/', 'tag': '06695201-8a8c-4c7b-845d-79f273789ba1', 'options': 'recursive,force'} 2024-05-27T07:26:12.239 INFO:tasks.workunit.client.0.smithi043.stdout: 1 68707 0.58 MB/sec execute 496 sec latency 61.132 ms 2024-05-27T07:26:12.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:12 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:12.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:12 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:12.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:12 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:13.240 INFO:tasks.workunit.client.0.smithi043.stdout: 1 68868 0.59 MB/sec execute 497 sec latency 62.484 ms 2024-05-27T07:26:13.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:13 smithi043 ceph-mon[31531]: pgmap v415: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 574 KiB/s wr, 65 op/s 2024-05-27T07:26:13.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:13 smithi179 ceph-mon[35888]: pgmap v415: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 574 KiB/s wr, 65 op/s 2024-05-27T07:26:13.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:13 smithi132 ceph-mon[35283]: pgmap v415: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 574 KiB/s wr, 65 op/s 2024-05-27T07:26:14.240 INFO:tasks.workunit.client.0.smithi043.stdout: 1 68950 0.59 MB/sec execute 498 sec latency 66.079 ms 2024-05-27T07:26:15.240 INFO:tasks.workunit.client.0.smithi043.stdout: 1 69043 0.59 MB/sec execute 499 sec latency 74.114 ms 2024-05-27T07:26:15.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:15 smithi043 ceph-mon[31531]: pgmap v416: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 567 KiB/s wr, 64 op/s 2024-05-27T07:26:15.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:15 smithi179 ceph-mon[35888]: pgmap v416: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 567 KiB/s wr, 64 op/s 2024-05-27T07:26:15.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:15 smithi132 ceph-mon[35283]: pgmap v416: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 567 KiB/s wr, 64 op/s 2024-05-27T07:26:16.240 INFO:tasks.workunit.client.0.smithi043.stdout: 1 69149 0.58 MB/sec execute 500 sec latency 72.018 ms 2024-05-27T07:26:17.240 INFO:tasks.workunit.client.0.smithi043.stdout: 1 69281 0.58 MB/sec execute 501 sec latency 68.850 ms 2024-05-27T07:26:17.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:17 smithi043 ceph-mon[31531]: pgmap v417: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 457 KiB/s wr, 50 op/s 2024-05-27T07:26:17.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:17 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:26:17.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:17 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:26:17.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:17 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:26:17.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:17 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:26:17.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:17 smithi179 ceph-mon[35888]: pgmap v417: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 457 KiB/s wr, 50 op/s 2024-05-27T07:26:17.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:17 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:26:17.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:17 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:26:17.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:17 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:26:17.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:17 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:26:17.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:17 smithi132 ceph-mon[35283]: pgmap v417: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 457 KiB/s wr, 50 op/s 2024-05-27T07:26:17.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:17 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:26:17.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:17 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:26:17.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:17 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:26:17.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:17 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:26:18.240 INFO:tasks.workunit.client.0.smithi043.stdout: 1 69399 0.58 MB/sec execute 502 sec latency 60.227 ms 2024-05-27T07:26:19.240 INFO:tasks.workunit.client.0.smithi043.stdout: 1 69511 0.58 MB/sec execute 503 sec latency 68.667 ms 2024-05-27T07:26:19.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:19 smithi043 ceph-mon[31531]: pgmap v418: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 457 KiB/s wr, 50 op/s 2024-05-27T07:26:19.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:19 smithi179 ceph-mon[35888]: pgmap v418: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 457 KiB/s wr, 50 op/s 2024-05-27T07:26:19.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:19 smithi132 ceph-mon[35283]: pgmap v418: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 457 KiB/s wr, 50 op/s 2024-05-27T07:26:20.240 INFO:tasks.workunit.client.0.smithi043.stdout: 1 69632 0.58 MB/sec execute 504 sec latency 69.109 ms 2024-05-27T07:26:21.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 69771 0.58 MB/sec execute 505 sec latency 52.430 ms 2024-05-27T07:26:21.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:21 smithi132 ceph-mon[35283]: pgmap v419: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 457 KiB/s wr, 50 op/s 2024-05-27T07:26:21.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:21 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:21.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:21 smithi043 ceph-mon[31531]: pgmap v419: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 457 KiB/s wr, 50 op/s 2024-05-27T07:26:21.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:21 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:21.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:21 smithi179 ceph-mon[35888]: pgmap v419: 129 pgs: 129 active+clean; 130 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 457 KiB/s wr, 50 op/s 2024-05-27T07:26:21.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:21 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:22.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 69845 0.58 MB/sec execute 506 sec latency 59.431 ms 2024-05-27T07:26:23.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 69923 0.58 MB/sec execute 507 sec latency 51.255 ms 2024-05-27T07:26:23.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:23 smithi132 ceph-mon[35283]: pgmap v420: 129 pgs: 129 active+clean; 131 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 214 KiB/s wr, 29 op/s 2024-05-27T07:26:23.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:23 smithi043 ceph-mon[31531]: pgmap v420: 129 pgs: 129 active+clean; 131 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 214 KiB/s wr, 29 op/s 2024-05-27T07:26:23.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:23 smithi179 ceph-mon[35888]: pgmap v420: 129 pgs: 129 active+clean; 131 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 214 KiB/s wr, 29 op/s 2024-05-27T07:26:24.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 70027 0.58 MB/sec execute 508 sec latency 56.841 ms 2024-05-27T07:26:25.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 70291 0.59 MB/sec execute 509 sec latency 56.417 ms 2024-05-27T07:26:25.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:25 smithi132 ceph-mon[35283]: pgmap v421: 129 pgs: 129 active+clean; 131 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 195 KiB/s wr, 27 op/s 2024-05-27T07:26:25.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:25 smithi043 ceph-mon[31531]: pgmap v421: 129 pgs: 129 active+clean; 131 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 195 KiB/s wr, 27 op/s 2024-05-27T07:26:25.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:25 smithi179 ceph-mon[35888]: pgmap v421: 129 pgs: 129 active+clean; 131 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 195 KiB/s wr, 27 op/s 2024-05-27T07:26:26.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 70399 0.59 MB/sec execute 510 sec latency 57.998 ms 2024-05-27T07:26:27.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 70616 0.59 MB/sec execute 511 sec latency 53.396 ms 2024-05-27T07:26:27.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:27 smithi132 ceph-mon[35283]: pgmap v422: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 513 KiB/s wr, 42 op/s 2024-05-27T07:26:27.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:27 smithi043 ceph-mon[31531]: pgmap v422: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 513 KiB/s wr, 42 op/s 2024-05-27T07:26:27.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:27 smithi179 ceph-mon[35888]: pgmap v422: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 513 KiB/s wr, 42 op/s 2024-05-27T07:26:28.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 70709 0.59 MB/sec execute 512 sec latency 60.640 ms 2024-05-27T07:26:29.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 70736 0.59 MB/sec execute 513 sec latency 62.205 ms 2024-05-27T07:26:29.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:29 smithi132 ceph-mon[35283]: pgmap v423: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 399 KiB/s wr, 26 op/s 2024-05-27T07:26:29.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:29 smithi043 ceph-mon[31531]: pgmap v423: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 399 KiB/s wr, 26 op/s 2024-05-27T07:26:29.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:29 smithi179 ceph-mon[35888]: pgmap v423: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 399 KiB/s wr, 26 op/s 2024-05-27T07:26:30.241 INFO:tasks.workunit.client.0.smithi043.stdout: 1 70775 0.59 MB/sec execute 514 sec latency 60.528 ms 2024-05-27T07:26:31.242 INFO:tasks.workunit.client.0.smithi043.stdout: 1 70943 0.59 MB/sec execute 515 sec latency 57.995 ms 2024-05-27T07:26:31.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:31 smithi132 ceph-mon[35283]: pgmap v424: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 399 KiB/s wr, 26 op/s 2024-05-27T07:26:31.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:31 smithi043 ceph-mon[31531]: pgmap v424: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 399 KiB/s wr, 26 op/s 2024-05-27T07:26:31.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:31 smithi179 ceph-mon[35888]: pgmap v424: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 399 KiB/s wr, 26 op/s 2024-05-27T07:26:32.242 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71034 0.59 MB/sec execute 516 sec latency 165.377 ms 2024-05-27T07:26:33.242 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71034 0.59 MB/sec execute 517 sec latency 1165.511 ms 2024-05-27T07:26:33.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:33 smithi132 ceph-mon[35283]: pgmap v425: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 636 KiB/s wr, 39 op/s 2024-05-27T07:26:33.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:33 smithi043 ceph-mon[31531]: pgmap v425: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 636 KiB/s wr, 39 op/s 2024-05-27T07:26:33.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:33 smithi179 ceph-mon[35888]: pgmap v425: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 636 KiB/s wr, 39 op/s 2024-05-27T07:26:34.242 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71084 0.59 MB/sec execute 518 sec latency 1384.059 ms 2024-05-27T07:26:35.242 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71101 0.59 MB/sec execute 519 sec latency 823.662 ms 2024-05-27T07:26:35.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:35 smithi132 ceph-mon[35283]: pgmap v426: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 555 KiB/s wr, 28 op/s 2024-05-27T07:26:35.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:35 smithi043 ceph-mon[31531]: pgmap v426: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 555 KiB/s wr, 28 op/s 2024-05-27T07:26:35.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:35 smithi179 ceph-mon[35888]: pgmap v426: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 555 KiB/s wr, 28 op/s 2024-05-27T07:26:36.242 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71101 0.59 MB/sec execute 520 sec latency 1823.834 ms 2024-05-27T07:26:37.242 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71101 0.58 MB/sec execute 521 sec latency 2824.016 ms 2024-05-27T07:26:37.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:37 smithi132 ceph-mon[35283]: pgmap v427: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 641 KiB/s wr, 40 op/s 2024-05-27T07:26:37.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:37 smithi043 ceph-mon[31531]: pgmap v427: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 641 KiB/s wr, 40 op/s 2024-05-27T07:26:37.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:37 smithi179 ceph-mon[35888]: pgmap v427: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 641 KiB/s wr, 40 op/s 2024-05-27T07:26:38.243 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71101 0.58 MB/sec execute 522 sec latency 3824.164 ms 2024-05-27T07:26:39.243 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71180 0.58 MB/sec execute 523 sec latency 4037.919 ms 2024-05-27T07:26:39.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:39 smithi132 ceph-mon[35283]: pgmap v428: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 323 KiB/s wr, 24 op/s 2024-05-27T07:26:39.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:39 smithi043 ceph-mon[31531]: pgmap v428: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 323 KiB/s wr, 24 op/s 2024-05-27T07:26:39.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:39 smithi179 ceph-mon[35888]: pgmap v428: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 323 KiB/s wr, 24 op/s 2024-05-27T07:26:40.243 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71229 0.58 MB/sec execute 524 sec latency 52.659 ms 2024-05-27T07:26:41.243 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71283 0.58 MB/sec execute 525 sec latency 50.791 ms 2024-05-27T07:26:41.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:41 smithi132 ceph-mon[35283]: pgmap v429: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 323 KiB/s wr, 24 op/s 2024-05-27T07:26:41.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:41 smithi043 ceph-mon[31531]: pgmap v429: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 323 KiB/s wr, 24 op/s 2024-05-27T07:26:41.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:41 smithi179 ceph-mon[35888]: pgmap v429: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 323 KiB/s wr, 24 op/s 2024-05-27T07:26:42.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:26:42.243 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71348 0.58 MB/sec execute 526 sec latency 58.160 ms 2024-05-27T07:26:42.513 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:42.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:42 smithi132 ceph-mon[35283]: pgmap v430: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 488 KiB/s wr, 34 op/s 2024-05-27T07:26:42.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:42 smithi043 ceph-mon[31531]: pgmap v430: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 488 KiB/s wr, 34 op/s 2024-05-27T07:26:42.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:42 smithi179 ceph-mon[35888]: pgmap v430: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 488 KiB/s wr, 34 op/s 2024-05-27T07:26:43.243 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71427 0.58 MB/sec execute 527 sec latency 48.028 ms 2024-05-27T07:26:43.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:43 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:43.859 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:43 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:44.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:43 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:26:44.246 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71515 0.58 MB/sec execute 528 sec latency 53.805 ms 2024-05-27T07:26:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.796+0000 7ffb60a00700 1 -- 172.21.15.43:0/2289017802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb5c108d30 msgr2=0x7ffb5c109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.796+0000 7ffb60a00700 1 --2- 172.21.15.43:0/2289017802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb5c108d30 0x7ffb5c109140 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ffb480099c0 tx=0x7ffb48009cd0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.796+0000 7ffb60a00700 1 -- 172.21.15.43:0/2289017802 shutdown_connections 2024-05-27T07:26:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.796+0000 7ffb60a00700 1 --2- 172.21.15.43:0/2289017802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb5c109680 0x7ffb5c110fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.796+0000 7ffb60a00700 1 --2- 172.21.15.43:0/2289017802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb5c104400 0x7ffb5c104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.796+0000 7ffb60a00700 1 --2- 172.21.15.43:0/2289017802 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb5c108d30 0x7ffb5c109140 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.796+0000 7ffb60a00700 1 -- 172.21.15.43:0/2289017802 >> 172.21.15.43:0/2289017802 conn(0x7ffb5c079c60 msgr2=0x7ffb5c07a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.796+0000 7ffb60a00700 1 -- 172.21.15.43:0/2289017802 shutdown_connections 2024-05-27T07:26:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.796+0000 7ffb60a00700 1 -- 172.21.15.43:0/2289017802 wait complete. 2024-05-27T07:26:44.787 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:44 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:44.787 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:44 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:44.787 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:44 smithi043 ceph-mon[31531]: pgmap v431: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 257 KiB/s wr, 22 op/s 2024-05-27T07:26:44.788 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:44 smithi043 ceph-mon[31531]: pgmap v432: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 309 KiB/s wr, 26 op/s 2024-05-27T07:26:44.788 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:44 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:26:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.797+0000 7ffb60a00700 1 Processor -- start 2024-05-27T07:26:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.797+0000 7ffb60a00700 1 -- start start 2024-05-27T07:26:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb60a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb5c104400 0x7ffb5c1a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb60a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb5c108d30 0x7ffb5c1a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb60a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb5c109680 0x7ffb5c19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb60a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c114190 con 0x7ffb5c108d30 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb60a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c114010 con 0x7ffb5c104400 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb60a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c114310 con 0x7ffb5c109680 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb59e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb5c108d30 0x7ffb5c1a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb59e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb5c108d30 0x7ffb5c1a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59060/0 (socket says 172.21.15.43:59060) 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb59e00700 1 -- 172.21.15.43:0/2746988354 learned_addr learned my addr 172.21.15.43:0/2746988354 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb5a800700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb5c104400 0x7ffb5c1a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.798+0000 7ffb5b200700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb5c109680 0x7ffb5c19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.799+0000 7ffb59e00700 1 -- 172.21.15.43:0/2746988354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb5c109680 msgr2=0x7ffb5c19aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.799+0000 7ffb59e00700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb5c109680 0x7ffb5c19aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.799+0000 7ffb59e00700 1 -- 172.21.15.43:0/2746988354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb5c104400 msgr2=0x7ffb5c1a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.799+0000 7ffb59e00700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb5c104400 0x7ffb5c1a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.799+0000 7ffb59e00700 1 -- 172.21.15.43:0/2746988354 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb5c19b140 con 0x7ffb5c108d30 2024-05-27T07:26:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.799+0000 7ffb5b200700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb5c109680 0x7ffb5c19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.799+0000 7ffb59e00700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb5c108d30 0x7ffb5c1a0db0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ffb4800b580 tx=0x7ffb4800f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.799+0000 7ffb53400700 1 -- 172.21.15.43:0/2746988354 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb48027070 con 0x7ffb5c108d30 2024-05-27T07:26:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.800+0000 7ffb53400700 1 -- 172.21.15.43:0/2746988354 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb4800fda0 con 0x7ffb5c108d30 2024-05-27T07:26:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.800+0000 7ffb60a00700 1 -- 172.21.15.43:0/2746988354 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb5c19b3d0 con 0x7ffb5c108d30 2024-05-27T07:26:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.800+0000 7ffb53400700 1 -- 172.21.15.43:0/2746988354 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb480173f0 con 0x7ffb5c108d30 2024-05-27T07:26:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.800+0000 7ffb60a00700 1 -- 172.21.15.43:0/2746988354 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb5c19b8b0 con 0x7ffb5c108d30 2024-05-27T07:26:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.804+0000 7ffb53400700 1 -- 172.21.15.43:0/2746988354 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ffb4802a070 con 0x7ffb5c108d30 2024-05-27T07:26:44.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.805+0000 7ffb6fe73b80 1 Processor -- start 2024-05-27T07:26:44.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.805+0000 7ffb6fe73b80 1 -- start start 2024-05-27T07:26:44.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.806+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6b9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.806+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6ba4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.806+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf6c8ee0 0x5570bf6cb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.806+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5570bf5e8e40 con 0x5570bf53be40 2024-05-27T07:26:44.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.806+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5570bf5d89e0 con 0x5570bf6c8ee0 2024-05-27T07:26:44.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.806+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5570bf5dca20 con 0x5570bf6ba0e0 2024-05-27T07:26:44.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.806+0000 7ffb53400700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffb400b29f0 0x7ffb400b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.806+0000 7ffb5b200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6b9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.807+0000 7ffb5b200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6b9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59074/0 (socket says 172.21.15.43:59074) 2024-05-27T07:26:44.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.807+0000 7ffb5b200700 1 -- 172.21.15.43:0/2640442012 learned_addr learned my addr 172.21.15.43:0/2640442012 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:44.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.807+0000 7ffb53400700 1 -- 172.21.15.43:0/2746988354 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7ffb480e39d0 con 0x7ffb5c108d30 2024-05-27T07:26:44.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.807+0000 7ffb5b200700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffb400b29f0 0x7ffb400b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.807+0000 7ffb59e00700 1 --2- 172.21.15.43:0/2640442012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf6c8ee0 0x5570bf6cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.807+0000 7ffb5a800700 1 --2- 172.21.15.43:0/2640442012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6ba4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.807+0000 7ffb5b200700 1 -- 172.21.15.43:0/2640442012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 msgr2=0x5570bf6ba4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:44.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.808+0000 7ffb5b200700 1 --2- 172.21.15.43:0/2640442012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6ba4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.808+0000 7ffb5b200700 1 -- 172.21.15.43:0/2640442012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf6c8ee0 msgr2=0x5570bf6cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:44.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.808+0000 7ffb5b200700 1 --2- 172.21.15.43:0/2640442012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf6c8ee0 0x5570bf6cb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.808+0000 7ffb5b200700 1 -- 172.21.15.43:0/2640442012 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570bf5a1420 con 0x5570bf53be40 2024-05-27T07:26:44.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.808+0000 7ffb5b200700 1 --2- 172.21.15.43:0/2640442012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6b9ba0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007dd0 tx=0x7ffb54017570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ece5b9da08eeef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:44.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.808+0000 7ffb5b200700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffb400b29f0 0x7ffb400b4f00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffb540178c0 tx=0x7ffb5401b000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:44.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.809+0000 7ffb47e00700 1 -- 172.21.15.43:0/2640442012 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5401e070 con 0x5570bf53be40 2024-05-27T07:26:44.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.809+0000 7ffb47e00700 1 -- 172.21.15.43:0/2640442012 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb54013040 con 0x5570bf53be40 2024-05-27T07:26:44.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.809+0000 7ffb47e00700 1 -- 172.21.15.43:0/2640442012 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54012070 con 0x5570bf53be40 2024-05-27T07:26:44.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.810+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/2640442012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 msgr2=0x5570bf6b9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:44.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.810+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/2640442012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6b9ba0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007dd0 tx=0x7ffb54017570 comp rx=0 tx=0).stop 2024-05-27T07:26:44.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.810+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/2640442012 shutdown_connections 2024-05-27T07:26:44.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.810+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/2640442012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf6c8ee0 0x5570bf6cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.810+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/2640442012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6b9ba0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.810+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/2640442012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6ba4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.810+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/2640442012 >> 172.21.15.43:0/2640442012 conn(0x5570bf59bfc0 msgr2=0x5570bf6b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:44.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.810+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/2640442012 shutdown_connections 2024-05-27T07:26:44.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.810+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/2640442012 wait complete. 2024-05-27T07:26:44.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.811+0000 7ffb6fe73b80 1 -- start start 2024-05-27T07:26:44.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.811+0000 7ffb6fe73b80 1 Processor -- start 2024-05-27T07:26:44.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf6ba0e0 0x5570bf6c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6c8ee0 0x5570bf6c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5570bf4defa0 con 0x5570bf53be40 2024-05-27T07:26:44.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5570bf579760 con 0x5570bf6ba0e0 2024-05-27T07:26:44.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5570bf6cbcc0 con 0x5570bf6c8ee0 2024-05-27T07:26:44.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb6fe73b80 10 client.0 _ll_register_callbacks cb 0x5570bf582c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:44.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb5a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb5a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59078/0 (socket says 172.21.15.43:59078) 2024-05-27T07:26:44.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb5a800700 1 -- 172.21.15.43:0/4134602966 learned_addr learned my addr 172.21.15.43:0/4134602966 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:44.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb59e00700 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf6ba0e0 0x5570bf6c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.812+0000 7ffb5b200700 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6c8ee0 0x5570bf6c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.813+0000 7ffb5a800700 1 -- 172.21.15.43:0/4134602966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6c8ee0 msgr2=0x5570bf6c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:44.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.813+0000 7ffb5a800700 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6c8ee0 0x5570bf6c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.813+0000 7ffb5a800700 1 -- 172.21.15.43:0/4134602966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf6ba0e0 msgr2=0x5570bf6c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:44.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.813+0000 7ffb5a800700 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf6ba0e0 0x5570bf6c64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.813+0000 7ffb5a800700 1 -- 172.21.15.43:0/4134602966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570bf5288d0 con 0x5570bf53be40 2024-05-27T07:26:44.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.813+0000 7ffb5b200700 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6c8ee0 0x5570bf6c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:44.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.813+0000 7ffb5a800700 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c5f60 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00bf20 tx=0x7ffb4c0116c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81954cc95b755d90 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:44.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.813+0000 7ffb44c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:44.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.813+0000 7ffb44c00700 1 -- 172.21.15.43:0/4134602966 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c004030 con 0x5570bf53be40 2024-05-27T07:26:44.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.813+0000 7ffb44c00700 1 -- 172.21.15.43:0/4134602966 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7ffb4c0041d0 con 0x5570bf53be40 2024-05-27T07:26:44.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.814+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4134602966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb54013040 con 0x5570bf53be40 2024-05-27T07:26:44.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.814+0000 7ffb44c00700 1 -- 172.21.15.43:0/4134602966 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7ffb4c024050 con 0x5570bf53be40 2024-05-27T07:26:44.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.814+0000 7ffb6fe73b80 10 client.17289 fetch_fsmap learned FSMap version 110 2024-05-27T07:26:44.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.814+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4134602966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x5570bf5a9060 con 0x5570bf53be40 2024-05-27T07:26:44.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.814+0000 7ffb44c00700 1 -- 172.21.15.43:0/4134602966 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb54013040 con 0x5570bf53be40 2024-05-27T07:26:44.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.814+0000 7ffb44c00700 1 -- 172.21.15.43:0/4134602966 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 110) ==== 4796+0+0 (secure 0 0 0) 0x7ffb4c01b600 con 0x5570bf53be40 2024-05-27T07:26:44.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.814+0000 7ffb6fe73b80 10 client.17289 fetch_fsmap finished waiting for FSMap version 110 2024-05-27T07:26:44.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.814+0000 7ffb6fe73b80 10 client.17289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:44.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.814+0000 7ffb6fe73b80 20 client.17289 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:44.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.815+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5570bf6d9190 0x5570bf6db680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.815+0000 7ffb6fe73b80 4 client.17289 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:44.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.815+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4134602966 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570bf5ead90 con 0x5570bf6d9190 2024-05-27T07:26:44.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.815+0000 7ffb59e00700 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5570bf6d9190 0x5570bf6db680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.816+0000 7ffb59e00700 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5570bf6d9190 0x5570bf6db680 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1eff8cadbb0e174 server_cookie=a65f1bdac2c19481 in_seq=0 out_seq=0 2024-05-27T07:26:44.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.816+0000 7ffb44c00700 10 client.17289 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:44.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.818+0000 7ffb44c00700 1 -- 172.21.15.43:0/4134602966 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5570bf5ead90 con 0x5570bf6d9190 2024-05-27T07:26:44.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.818+0000 7ffb44c00700 10 client.17289 handle_command_reply: tid=0 2024-05-27T07:26:44.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.818+0000 7ffb6fe73b80 1 client.17289 shutdown 2024-05-27T07:26:44.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.818+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4134602966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 msgr2=0x5570bf6c5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.818+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c5f60 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00bf20 tx=0x7ffb4c0116c0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.819+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4134602966 shutdown_connections 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.819+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6c8ee0 0x5570bf6c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.819+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5570bf6d9190 0x5570bf6db680 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.819+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c5f60 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.819+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/4134602966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf6ba0e0 0x5570bf6c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.819+0000 7ffb44c00700 0 client.17289 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.819+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4134602966 >> 172.21.15.43:0/4134602966 conn(0x5570bf59bfc0 msgr2=0x5570bf6b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.819+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4134602966 shutdown_connections 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.819+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4134602966 wait complete. 2024-05-27T07:26:44.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.819+0000 7ffb6fe73b80 20 client.17289 trim_cache size 0 max 16384 2024-05-27T07:26:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.844+0000 7ffb6fe73b80 1 Processor -- start 2024-05-27T07:26:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.844+0000 7ffb6fe73b80 1 -- start start 2024-05-27T07:26:44.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.844+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf59bfc0 0x5570bf6c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.844+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.844+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6d4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.844+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c1aed10 con 0x5570bf53be40 2024-05-27T07:26:44.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.844+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c1aca40 con 0x5570bf59bfc0 2024-05-27T07:26:44.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.844+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c1ae640 con 0x5570bf6ba0e0 2024-05-27T07:26:44.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb59e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb59e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59090/0 (socket says 172.21.15.43:59090) 2024-05-27T07:26:44.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb59e00700 1 -- 172.21.15.43:0/4211884310 learned_addr learned my addr 172.21.15.43:0/4211884310 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:44.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb5a800700 1 --2- 172.21.15.43:0/4211884310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf59bfc0 0x5570bf6c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb5b200700 1 --2- 172.21.15.43:0/4211884310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6d4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb59e00700 1 -- 172.21.15.43:0/4211884310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 msgr2=0x5570bf6d4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:44.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb59e00700 1 --2- 172.21.15.43:0/4211884310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6d4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb59e00700 1 -- 172.21.15.43:0/4211884310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf59bfc0 msgr2=0x5570bf6c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:44.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb59e00700 1 --2- 172.21.15.43:0/4211884310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf59bfc0 0x5570bf6c4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb59e00700 1 -- 172.21.15.43:0/4211884310 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570bf6c44d0 con 0x5570bf53be40 2024-05-27T07:26:44.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.845+0000 7ffb5b200700 1 --2- 172.21.15.43:0/4211884310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6d4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:44.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.846+0000 7ffb59e00700 1 --2- 172.21.15.43:0/4211884310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c1590 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ffb480a7b80 tx=0x7ffb4800ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceaef63e46988e10 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:44.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.846+0000 7ffb47400700 1 -- 172.21.15.43:0/4211884310 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb480aa420 con 0x5570bf53be40 2024-05-27T07:26:44.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.846+0000 7ffb47400700 1 -- 172.21.15.43:0/4211884310 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb480afe50 con 0x5570bf53be40 2024-05-27T07:26:44.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.846+0000 7ffb47400700 1 -- 172.21.15.43:0/4211884310 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb480cd0b0 con 0x5570bf53be40 2024-05-27T07:26:44.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.847+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4211884310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 msgr2=0x5570bf6c1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:44.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.847+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/4211884310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c1590 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ffb480a7b80 tx=0x7ffb4800ff50 comp rx=0 tx=0).stop 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.847+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4211884310 shutdown_connections 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.847+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/4211884310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6d4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.847+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/4211884310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf59bfc0 0x5570bf6c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.847+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/4211884310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf53be40 0x5570bf6c1590 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.847+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4211884310 >> 172.21.15.43:0/4211884310 conn(0x5570bf6aef60 msgr2=0x5570bf6c0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.847+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4211884310 shutdown_connections 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.847+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/4211884310 wait complete. 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.848+0000 7ffb6fe73b80 1 -- start start 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.848+0000 7ffb6fe73b80 1 Processor -- start 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf53be40 0x5570bf6d15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf59bfc0 0x5570bf6c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb6fe73b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6ca180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c1ad7e0 con 0x5570bf59bfc0 2024-05-27T07:26:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c1ad110 con 0x5570bf53be40 2024-05-27T07:26:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb6fe73b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5c1adf10 con 0x5570bf6ba0e0 2024-05-27T07:26:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb6fe73b80 10 client.0 _ll_register_callbacks cb 0x5570bf582c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb59e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf59bfc0 0x5570bf6c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb59e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf59bfc0 0x5570bf6c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59094/0 (socket says 172.21.15.43:59094) 2024-05-27T07:26:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb59e00700 1 -- 172.21.15.43:0/1753458960 learned_addr learned my addr 172.21.15.43:0/1753458960 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb5a800700 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf53be40 0x5570bf6d15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb59e00700 1 -- 172.21.15.43:0/1753458960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 msgr2=0x5570bf6ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.849+0000 7ffb5b200700 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb59e00700 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb59e00700 1 -- 172.21.15.43:0/1753458960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf53be40 msgr2=0x5570bf6d15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb59e00700 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf53be40 0x5570bf6d15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb59e00700 1 -- 172.21.15.43:0/1753458960 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570bf6ca820 con 0x5570bf59bfc0 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb5a800700 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf53be40 0x5570bf6d15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb5b200700 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb59e00700 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf59bfc0 0x5570bf6c9c40 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ffb480aa610 tx=0x7ffb480e1230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c37130f96f1ab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb44c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb44c00700 1 -- 172.21.15.43:0/1753458960 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb480aa910 con 0x5570bf59bfc0 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb44c00700 1 -- 172.21.15.43:0/1753458960 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7ffb480a6f00 con 0x5570bf59bfc0 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.850+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/1753458960 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb480afe50 con 0x5570bf59bfc0 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb44c00700 1 -- 172.21.15.43:0/1753458960 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7ffb480ab010 con 0x5570bf59bfc0 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb6fe73b80 10 client.17295 fetch_fsmap learned FSMap version 110 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/1753458960 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x5570bf6cfea0 con 0x5570bf59bfc0 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb44c00700 1 -- 172.21.15.43:0/1753458960 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb480afe50 con 0x5570bf59bfc0 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb44c00700 1 -- 172.21.15.43:0/1753458960 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 110) ==== 4796+0+0 (secure 0 0 0) 0x7ffb480b3250 con 0x5570bf59bfc0 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb6fe73b80 10 client.17295 fetch_fsmap finished waiting for FSMap version 110 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb6fe73b80 10 client.17295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb6fe73b80 20 client.17295 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5570bf6c8ee0 0x5570bf745f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb6fe73b80 4 client.17295 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:26:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.851+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/1753458960 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5570bf5e4570 con 0x5570bf6c8ee0 2024-05-27T07:26:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.852+0000 7ffb5a800700 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5570bf6c8ee0 0x5570bf745f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.853+0000 7ffb5a800700 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5570bf6c8ee0 0x5570bf745f60 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f40eb5dc1379212 server_cookie=c171da332d9482a8 in_seq=0 out_seq=0 2024-05-27T07:26:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.853+0000 7ffb44c00700 10 client.17295 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb44c00700 1 -- 172.21.15.43:0/1753458960 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5570bf5e4570 con 0x5570bf6c8ee0 2024-05-27T07:26:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb44c00700 10 client.17295 handle_command_reply: tid=0 2024-05-27T07:26:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 client.17295 shutdown 2024-05-27T07:26:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/1753458960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf59bfc0 msgr2=0x5570bf6c9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf59bfc0 0x5570bf6c9c40 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ffb480aa610 tx=0x7ffb480e1230 comp rx=0 tx=0).stop 2024-05-27T07:26:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/1753458960 shutdown_connections 2024-05-27T07:26:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5570bf6ba0e0 0x5570bf6ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5570bf53be40 0x5570bf6d15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5570bf6c8ee0 0x5570bf745f60 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 --2- 172.21.15.43:0/1753458960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5570bf59bfc0 0x5570bf6c9c40 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/1753458960 >> 172.21.15.43:0/1753458960 conn(0x5570bf6aef60 msgr2=0x5570bf6c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb44c00700 0 client.17295 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:26:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/1753458960 shutdown_connections 2024-05-27T07:26:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 1 -- 172.21.15.43:0/1753458960 wait complete. 2024-05-27T07:26:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.854+0000 7ffb6fe73b80 20 client.17295 trim_cache size 0 max 16384 2024-05-27T07:26:44.846 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:26:44.847 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:26:44.847 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:26:44.847 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:26:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.855+0000 7ffb60a00700 1 -- 172.21.15.43:0/2746988354 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffb400b29f0 msgr2=0x7ffb400b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.855+0000 7ffb60a00700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffb400b29f0 0x7ffb400b4f00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffb540178c0 tx=0x7ffb5401b000 comp rx=0 tx=0).stop 2024-05-27T07:26:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.855+0000 7ffb60a00700 1 -- 172.21.15.43:0/2746988354 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb5c108d30 msgr2=0x7ffb5c1a0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.855+0000 7ffb60a00700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb5c108d30 0x7ffb5c1a0db0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ffb4800b580 tx=0x7ffb4800f460 comp rx=0 tx=0).stop 2024-05-27T07:26:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.855+0000 7ffb60a00700 1 -- 172.21.15.43:0/2746988354 shutdown_connections 2024-05-27T07:26:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.855+0000 7ffb60a00700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffb400b29f0 0x7ffb400b4f00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.855+0000 7ffb60a00700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb5c109680 0x7ffb5c19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.855+0000 7ffb60a00700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb5c104400 0x7ffb5c1a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.855+0000 7ffb60a00700 1 --2- 172.21.15.43:0/2746988354 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb5c108d30 0x7ffb5c1a0db0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.855+0000 7ffb60a00700 1 -- 172.21.15.43:0/2746988354 >> 172.21.15.43:0/2746988354 conn(0x7ffb5c079c60 msgr2=0x7ffb5c1019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.856+0000 7ffb60a00700 1 -- 172.21.15.43:0/2746988354 shutdown_connections 2024-05-27T07:26:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:44.856+0000 7ffb60a00700 1 -- 172.21.15.43:0/2746988354 wait complete. 2024-05-27T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:44 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:44 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:44 smithi179 ceph-mon[35888]: pgmap v431: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 257 KiB/s wr, 22 op/s 2024-05-27T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:44 smithi179 ceph-mon[35888]: pgmap v432: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 309 KiB/s wr, 26 op/s 2024-05-27T07:26:44.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:44 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:26:45.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:44 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:26:45.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:44 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:26:45.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:44 smithi132 ceph-mon[35283]: pgmap v431: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 257 KiB/s wr, 22 op/s 2024-05-27T07:26:45.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:44 smithi132 ceph-mon[35283]: pgmap v432: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 309 KiB/s wr, 26 op/s 2024-05-27T07:26:45.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:44 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:26:45.243 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71582 0.58 MB/sec execute 529 sec latency 50.204 ms 2024-05-27T07:26:45.734 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:26:45.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:26:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:46.244 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71717 0.58 MB/sec execute 530 sec latency 58.147 ms 2024-05-27T07:26:47.247 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71836 0.58 MB/sec execute 531 sec latency 59.958 ms 2024-05-27T07:26:47.340 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:47 smithi132 ceph-mon[35283]: pgmap v433: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 431 KiB/s wr, 35 op/s 2024-05-27T07:26:47.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:47 smithi043 ceph-mon[31531]: pgmap v433: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 431 KiB/s wr, 35 op/s 2024-05-27T07:26:47.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:47 smithi179 ceph-mon[35888]: pgmap v433: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 431 KiB/s wr, 35 op/s 2024-05-27T07:26:48.247 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71942 0.58 MB/sec execute 532 sec latency 49.965 ms 2024-05-27T07:26:48.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.546+0000 7fa7c6400700 1 -- 172.21.15.43:0/2278113827 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7c01054b0 msgr2=0x7fa7c01058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.546+0000 7fa7c6400700 1 --2- 172.21.15.43:0/2278113827 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7c01054b0 0x7fa7c01058c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009990 tx=0x7fa7b0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.547+0000 7fa7c6400700 1 -- 172.21.15.43:0/2278113827 shutdown_connections 2024-05-27T07:26:48.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.547+0000 7fa7c6400700 1 --2- 172.21.15.43:0/2278113827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7c010e8e0 0x7fa7c0110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.547+0000 7fa7c6400700 1 --2- 172.21.15.43:0/2278113827 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7c01054b0 0x7fa7c01058c0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.547+0000 7fa7c6400700 1 --2- 172.21.15.43:0/2278113827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7c0105e00 0x7fa7c010e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.547+0000 7fa7c6400700 1 -- 172.21.15.43:0/2278113827 >> 172.21.15.43:0/2278113827 conn(0x7fa7c00fede0 msgr2=0x7fa7c0101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.547+0000 7fa7c6400700 1 -- 172.21.15.43:0/2278113827 shutdown_connections 2024-05-27T07:26:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.547+0000 7fa7c6400700 1 -- 172.21.15.43:0/2278113827 wait complete. 2024-05-27T07:26:48.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.548+0000 7fa7c6400700 1 Processor -- start 2024-05-27T07:26:48.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.548+0000 7fa7c6400700 1 -- start start 2024-05-27T07:26:48.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7c6400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7c01054b0 0x7fa7c019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7c6400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7c0105e00 0x7fa7c019d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7c6400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7c010e8e0 0x7fa7c01a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7c6400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c01142e0 con 0x7fa7c01054b0 2024-05-27T07:26:48.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7c6400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c0114160 con 0x7fa7c0105e00 2024-05-27T07:26:48.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7c6400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c0114460 con 0x7fa7c010e8e0 2024-05-27T07:26:48.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7bf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7c01054b0 0x7fa7c019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7bf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7c01054b0 0x7fa7c019cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59116/0 (socket says 172.21.15.43:59116) 2024-05-27T07:26:48.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7bf400700 1 -- 172.21.15.43:0/4156748041 learned_addr learned my addr 172.21.15.43:0/4156748041 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:48.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7bfe00700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7c010e8e0 0x7fa7c01a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.549+0000 7fa7bea00700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7c0105e00 0x7fa7c019d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.550+0000 7fa7bf400700 1 -- 172.21.15.43:0/4156748041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7c010e8e0 msgr2=0x7fa7c01a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.550+0000 7fa7bf400700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7c010e8e0 0x7fa7c01a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.550+0000 7fa7bf400700 1 -- 172.21.15.43:0/4156748041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7c0105e00 msgr2=0x7fa7c019d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.550+0000 7fa7bf400700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7c0105e00 0x7fa7c019d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.550+0000 7fa7bf400700 1 -- 172.21.15.43:0/4156748041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c01a1e20 con 0x7fa7c01054b0 2024-05-27T07:26:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.550+0000 7fa7bf400700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7c01054b0 0x7fa7c019cf50 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0000c00 tx=0x7fa7b0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.550+0000 7fa7b7e00700 1 -- 172.21.15.43:0/4156748041 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0027070 con 0x7fa7c01054b0 2024-05-27T07:26:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.550+0000 7fa7b7e00700 1 -- 172.21.15.43:0/4156748041 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa7b000fc70 con 0x7fa7c01054b0 2024-05-27T07:26:48.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.550+0000 7fa7c6400700 1 -- 172.21.15.43:0/4156748041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c01a20b0 con 0x7fa7c01054b0 2024-05-27T07:26:48.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.551+0000 7fa7b7e00700 1 -- 172.21.15.43:0/4156748041 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0022d30 con 0x7fa7c01054b0 2024-05-27T07:26:48.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.551+0000 7fa7c6400700 1 -- 172.21.15.43:0/4156748041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c01a2590 con 0x7fa7c01054b0 2024-05-27T07:26:48.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.552+0000 7fa7c6400700 1 -- 172.21.15.43:0/4156748041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7c00512b0 con 0x7fa7c01054b0 2024-05-27T07:26:48.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.554+0000 7fa7b7e00700 1 -- 172.21.15.43:0/4156748041 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa7b002a070 con 0x7fa7c01054b0 2024-05-27T07:26:48.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.556+0000 7fa7b7e00700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa7a40b29f0 0x7fa7a40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:48.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.556+0000 7fa7b7e00700 1 -- 172.21.15.43:0/4156748041 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7fa7b00e3f50 con 0x7fa7c01054b0 2024-05-27T07:26:48.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.556+0000 7fa7bea00700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa7a40b29f0 0x7fa7a40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:48.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.557+0000 7fa7bea00700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa7a40b29f0 0x7fa7a40b4f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac005950 tx=0x7fa7ac0058e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:48.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.559+0000 7fa7b7e00700 1 -- 172.21.15.43:0/4156748041 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa7b0031020 con 0x7fa7c01054b0 2024-05-27T07:26:48.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.878+0000 7fa7c6400700 1 -- 172.21.15.43:0/4156748041 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7c01a9af0 con 0x7fa7c01054b0 2024-05-27T07:26:48.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.880+0000 7fa7b7e00700 1 -- 172.21.15.43:0/4156748041 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+6959 (secure 0 0 0) 0x7fa7b0013070 con 0x7fa7c01054b0 2024-05-27T07:26:48.870 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:26:48.870 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":110,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:26:20.402228+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34484,34448,34424,24484,34403]},"id":1}]} 2024-05-27T07:26:48.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.883+0000 7fa7c6400700 1 -- 172.21.15.43:0/4156748041 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa7a40b29f0 msgr2=0x7fa7a40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.883+0000 7fa7c6400700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa7a40b29f0 0x7fa7a40b4f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac005950 tx=0x7fa7ac0058e0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.883+0000 7fa7c6400700 1 -- 172.21.15.43:0/4156748041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7c01054b0 msgr2=0x7fa7c019cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:48.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.883+0000 7fa7c6400700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7c01054b0 0x7fa7c019cf50 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0000c00 tx=0x7fa7b0009f90 comp rx=0 tx=0).stop 2024-05-27T07:26:48.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.884+0000 7fa7c6400700 1 -- 172.21.15.43:0/4156748041 shutdown_connections 2024-05-27T07:26:48.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.884+0000 7fa7c6400700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa7a40b29f0 0x7fa7a40b4f00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.884+0000 7fa7c6400700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7c010e8e0 0x7fa7c01a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.884+0000 7fa7c6400700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7c01054b0 0x7fa7c019cf50 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.884+0000 7fa7c6400700 1 --2- 172.21.15.43:0/4156748041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7c0105e00 0x7fa7c019d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:48.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.884+0000 7fa7c6400700 1 -- 172.21.15.43:0/4156748041 >> 172.21.15.43:0/4156748041 conn(0x7fa7c00fede0 msgr2=0x7fa7c0100a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:48.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.884+0000 7fa7c6400700 1 -- 172.21.15.43:0/4156748041 shutdown_connections 2024-05-27T07:26:48.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:48.884+0000 7fa7c6400700 1 -- 172.21.15.43:0/4156748041 wait complete. 2024-05-27T07:26:48.875 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 110 2024-05-27T07:26:49.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:49 smithi043 ceph-mon[31531]: pgmap v434: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 431 KiB/s wr, 35 op/s 2024-05-27T07:26:49.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:49 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4156748041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:49.247 INFO:tasks.workunit.client.0.smithi043.stdout: 1 72083 0.58 MB/sec execute 533 sec latency 52.476 ms 2024-05-27T07:26:49.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:49 smithi179 ceph-mon[35888]: pgmap v434: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 431 KiB/s wr, 35 op/s 2024-05-27T07:26:49.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:49 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4156748041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:49.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:49 smithi132 ceph-mon[35283]: pgmap v434: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 431 KiB/s wr, 35 op/s 2024-05-27T07:26:49.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:49 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4156748041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:26:49.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:26:50.247 INFO:tasks.workunit.client.0.smithi043.stdout: 1 72189 0.58 MB/sec execute 534 sec latency 53.766 ms 2024-05-27T07:26:50.257 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:51.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 72367 0.58 MB/sec execute 535 sec latency 58.740 ms 2024-05-27T07:26:51.443 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:51 smithi043 ceph-mon[31531]: pgmap v435: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 431 KiB/s wr, 35 op/s 2024-05-27T07:26:51.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:51 smithi179 ceph-mon[35888]: pgmap v435: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 431 KiB/s wr, 35 op/s 2024-05-27T07:26:51.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:51 smithi132 ceph-mon[35283]: pgmap v435: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 431 KiB/s wr, 35 op/s 2024-05-27T07:26:52.247 INFO:tasks.workunit.client.0.smithi043.stdout: 1 72460 0.58 MB/sec execute 536 sec latency 64.376 ms 2024-05-27T07:26:52.334 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:52 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:52.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:52 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:52.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:52 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:52.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.613+0000 7f0274800700 1 -- 172.21.15.43:0/648124117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c108730 msgr2=0x7f026c110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.613+0000 7f0274800700 1 --2- 172.21.15.43:0/648124117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c108730 0x7f026c110090 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f026800b0d0 tx=0x7f026800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.613+0000 7f0274800700 1 -- 172.21.15.43:0/648124117 shutdown_connections 2024-05-27T07:26:52.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.613+0000 7f0274800700 1 --2- 172.21.15.43:0/648124117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c108730 0x7f026c110090 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.613+0000 7f0274800700 1 --2- 172.21.15.43:0/648124117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f026c1034b0 0x7f026c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.613+0000 7f0274800700 1 --2- 172.21.15.43:0/648124117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f026c107de0 0x7f026c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.613+0000 7f0274800700 1 -- 172.21.15.43:0/648124117 >> 172.21.15.43:0/648124117 conn(0x7f026c0fedb0 msgr2=0x7f026c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.613+0000 7f0274800700 1 -- 172.21.15.43:0/648124117 shutdown_connections 2024-05-27T07:26:52.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.613+0000 7f0274800700 1 -- 172.21.15.43:0/648124117 wait complete. 2024-05-27T07:26:52.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.615+0000 7f0274800700 1 Processor -- start 2024-05-27T07:26:52.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.615+0000 7f0274800700 1 -- start start 2024-05-27T07:26:52.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.616+0000 7f0274800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1034b0 0x7f026c0ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.616+0000 7f0274800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f026c107de0 0x7f026c0ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.616+0000 7f0274800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f026c108730 0x7f026c0ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.616+0000 7f0274800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f026c113260 con 0x7f026c1034b0 2024-05-27T07:26:52.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.616+0000 7f0274800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f026c1130e0 con 0x7f026c108730 2024-05-27T07:26:52.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.616+0000 7f0274800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f026c1133e0 con 0x7f026c107de0 2024-05-27T07:26:52.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.616+0000 7f0271e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1034b0 0x7f026c0ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.616+0000 7f0271e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1034b0 0x7f026c0ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59140/0 (socket says 172.21.15.43:59140) 2024-05-27T07:26:52.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.616+0000 7f0271e00700 1 -- 172.21.15.43:0/961446719 learned_addr learned my addr 172.21.15.43:0/961446719 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:52.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.616+0000 7f0271400700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f026c107de0 0x7f026c0ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.617+0000 7f0272800700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f026c108730 0x7f026c0ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.617+0000 7f0271e00700 1 -- 172.21.15.43:0/961446719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f026c107de0 msgr2=0x7f026c0ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.617+0000 7f0271e00700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f026c107de0 0x7f026c0ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.617+0000 7f0271e00700 1 -- 172.21.15.43:0/961446719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f026c108730 msgr2=0x7f026c0ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.617+0000 7f0271e00700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f026c108730 0x7f026c0ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.617+0000 7f0271e00700 1 -- 172.21.15.43:0/961446719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026c1a90a0 con 0x7f026c1034b0 2024-05-27T07:26:52.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.618+0000 7f0271e00700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1034b0 0x7f026c0ff290 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f026000d950 tx=0x7f026000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.618+0000 7f0266a00700 1 -- 172.21.15.43:0/961446719 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0260016070 con 0x7f026c1034b0 2024-05-27T07:26:52.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.618+0000 7f0266a00700 1 -- 172.21.15.43:0/961446719 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f026000fd20 con 0x7f026c1034b0 2024-05-27T07:26:52.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.618+0000 7f0266a00700 1 -- 172.21.15.43:0/961446719 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026000e3b0 con 0x7f026c1034b0 2024-05-27T07:26:52.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.618+0000 7f0274800700 1 -- 172.21.15.43:0/961446719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f026c1a9390 con 0x7f026c1034b0 2024-05-27T07:26:52.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.618+0000 7f0274800700 1 -- 172.21.15.43:0/961446719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f026c1a9880 con 0x7f026c1034b0 2024-05-27T07:26:52.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.623+0000 7f0266a00700 1 -- 172.21.15.43:0/961446719 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0260019070 con 0x7f026c1034b0 2024-05-27T07:26:52.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.623+0000 7f02832ceb80 1 Processor -- start 2024-05-27T07:26:52.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.624+0000 7f02832ceb80 1 -- start start 2024-05-27T07:26:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.624+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f98258fe40 0x55f9827147f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.624+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f982714d30 0x55f98270e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.624+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98270edb0 0x55f98270f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.624+0000 7f02832ceb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f98263ce40 con 0x55f982714d30 2024-05-27T07:26:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.624+0000 7f02832ceb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f98262c9e0 con 0x55f98258fe40 2024-05-27T07:26:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.624+0000 7f02832ceb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f982630a20 con 0x55f98270edb0 2024-05-27T07:26:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.624+0000 7f0271e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f982714d30 0x55f98270e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0271e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f982714d30 0x55f98270e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59148/0 (socket says 172.21.15.43:59148) 2024-05-27T07:26:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0271e00700 1 -- 172.21.15.43:0/2095013559 learned_addr learned my addr 172.21.15.43:0/2095013559 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0272800700 1 --2- 172.21.15.43:0/2095013559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f98258fe40 0x55f9827147f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0271400700 1 --2- 172.21.15.43:0/2095013559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98270edb0 0x55f98270f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0266a00700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02540b2a70 0x7f02540b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0266a00700 1 -- 172.21.15.43:0/961446719 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7f02600e24b0 con 0x7f026c1034b0 2024-05-27T07:26:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0271e00700 1 -- 172.21.15.43:0/2095013559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98270edb0 msgr2=0x55f98270f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0271e00700 1 --2- 172.21.15.43:0/2095013559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98270edb0 0x55f98270f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0271e00700 1 -- 172.21.15.43:0/2095013559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f98258fe40 msgr2=0x55f9827147f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0271e00700 1 --2- 172.21.15.43:0/2095013559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f98258fe40 0x55f9827147f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0271e00700 1 -- 172.21.15.43:0/2095013559 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9825f5420 con 0x55f982714d30 2024-05-27T07:26:52.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.625+0000 7f0272800700 1 --2- 172.21.15.43:0/2095013559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f98258fe40 0x55f9827147f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:26:52.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.626+0000 7f0272800700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02540b2a70 0x7f02540b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.626+0000 7f0271e00700 1 --2- 172.21.15.43:0/2095013559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f982714d30 0x55f98270e870 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f02600df740 tx=0x7f0260000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c408649cf725a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.626+0000 7f025b400700 1 -- 172.21.15.43:0/2095013559 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0260020070 con 0x55f982714d30 2024-05-27T07:26:52.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.626+0000 7f025b400700 1 -- 172.21.15.43:0/2095013559 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02600e5b70 con 0x55f982714d30 2024-05-27T07:26:52.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.626+0000 7f025b400700 1 -- 172.21.15.43:0/2095013559 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02600b21c0 con 0x55f982714d30 2024-05-27T07:26:52.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.627+0000 7f02832ceb80 1 -- 172.21.15.43:0/2095013559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f982714d30 msgr2=0x55f98270e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.627+0000 7f0272800700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02540b2a70 0x7f02540b4f80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0268009900 tx=0x7f02680099e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.627+0000 7f02832ceb80 1 --2- 172.21.15.43:0/2095013559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f982714d30 0x55f98270e870 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f02600df740 tx=0x7f0260000bc0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.627+0000 7f02832ceb80 1 -- 172.21.15.43:0/2095013559 shutdown_connections 2024-05-27T07:26:52.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.627+0000 7f02832ceb80 1 --2- 172.21.15.43:0/2095013559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98270edb0 0x55f98270f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.627+0000 7f02832ceb80 1 --2- 172.21.15.43:0/2095013559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f98258fe40 0x55f9827147f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.627+0000 7f02832ceb80 1 --2- 172.21.15.43:0/2095013559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f982714d30 0x55f98270e870 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.628+0000 7f02832ceb80 1 -- 172.21.15.43:0/2095013559 >> 172.21.15.43:0/2095013559 conn(0x55f9825effc0 msgr2=0x55f9827098d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.628+0000 7f02832ceb80 1 -- 172.21.15.43:0/2095013559 shutdown_connections 2024-05-27T07:26:52.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.628+0000 7f02832ceb80 1 -- 172.21.15.43:0/2095013559 wait complete. 2024-05-27T07:26:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.628+0000 7f02832ceb80 1 -- start start 2024-05-27T07:26:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.629+0000 7f02832ceb80 1 Processor -- start 2024-05-27T07:26:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.629+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98258fe40 0x55f9827184a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.629+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f9827189e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.629+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f982718f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.629+0000 7f02832ceb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f982532fa0 con 0x55f98270edb0 2024-05-27T07:26:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.629+0000 7f02832ceb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f9825cd760 con 0x55f982714d30 2024-05-27T07:26:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.629+0000 7f02832ceb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f98271b740 con 0x55f98258fe40 2024-05-27T07:26:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.629+0000 7f02832ceb80 10 client.0 _ll_register_callbacks cb 0x55f9825d6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:52.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0271400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f9827189e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0271400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f9827189e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59152/0 (socket says 172.21.15.43:59152) 2024-05-27T07:26:52.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0271400700 1 -- 172.21.15.43:0/1540876045 learned_addr learned my addr 172.21.15.43:0/1540876045 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:52.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0271e00700 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98258fe40 0x55f9827184a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0272800700 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f982718f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0271400700 1 -- 172.21.15.43:0/1540876045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98258fe40 msgr2=0x55f9827184a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0271400700 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98258fe40 0x55f9827184a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0271400700 1 -- 172.21.15.43:0/1540876045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 msgr2=0x55f982718f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0271400700 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f982718f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0271400700 1 -- 172.21.15.43:0/1540876045 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f98257c8d0 con 0x55f98270edb0 2024-05-27T07:26:52.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.630+0000 7f0272800700 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f982718f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:52.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.631+0000 7f0271400700 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f9827189e0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f025c00bf90 tx=0x7f025c011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195c512792370795 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.631+0000 7f024fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:52.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.631+0000 7f024fe00700 1 -- 172.21.15.43:0/1540876045 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c004830 con 0x55f98270edb0 2024-05-27T07:26:52.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.631+0000 7f024fe00700 1 -- 172.21.15.43:0/1540876045 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7f025c0049d0 con 0x55f98270edb0 2024-05-27T07:26:52.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.631+0000 7f02832ceb80 1 -- 172.21.15.43:0/1540876045 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f98263ed90 con 0x55f98270edb0 2024-05-27T07:26:52.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.631+0000 7f024fe00700 1 -- 172.21.15.43:0/1540876045 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f025c025050 con 0x55f98270edb0 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.632+0000 7f02832ceb80 10 client.17313 fetch_fsmap learned FSMap version 111 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.632+0000 7f02832ceb80 1 -- 172.21.15.43:0/1540876045 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x55f9825fd060 con 0x55f98270edb0 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.632+0000 7f024fe00700 1 -- 172.21.15.43:0/1540876045 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55f98263ed90 con 0x55f98270edb0 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.632+0000 7f024fe00700 1 -- 172.21.15.43:0/1540876045 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 111) ==== 4800+0+0 (secure 0 0 0) 0x7f025c01a710 con 0x55f98270edb0 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.632+0000 7f02832ceb80 10 client.17313 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.632+0000 7f02832ceb80 10 client.17313 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.632+0000 7f02832ceb80 20 client.17313 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.632+0000 7f02832ceb80 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55f982728a20 0x55f98272ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.632+0000 7f02832ceb80 4 client.17313 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.632+0000 7f02832ceb80 1 -- 172.21.15.43:0/1540876045 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f982638570 con 0x55f982728a20 2024-05-27T07:26:52.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.633+0000 7f0271e00700 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55f982728a20 0x55f98272ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.634+0000 7f0271e00700 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55f982728a20 0x55f98272ae70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=82cf9dc10dbfae83 server_cookie=8d015f3ef6252164 in_seq=0 out_seq=0 2024-05-27T07:26:52.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.634+0000 7f024fe00700 10 client.17313 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:26:52.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f024fe00700 1 -- 172.21.15.43:0/1540876045 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f982638570 con 0x55f982728a20 2024-05-27T07:26:52.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f024fe00700 10 client.17313 handle_command_reply: tid=0 2024-05-27T07:26:52.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f02832ceb80 1 client.17313 shutdown 2024-05-27T07:26:52.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f02832ceb80 1 -- 172.21.15.43:0/1540876045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 msgr2=0x55f9827189e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f02832ceb80 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f9827189e0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f025c00bf90 tx=0x7f025c011720 comp rx=0 tx=0).stop 2024-05-27T07:26:52.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f02832ceb80 1 -- 172.21.15.43:0/1540876045 shutdown_connections 2024-05-27T07:26:52.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f02832ceb80 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55f982728a20 0x55f98272ae70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f02832ceb80 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f982718f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f02832ceb80 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98258fe40 0x55f9827184a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f02832ceb80 1 --2- 172.21.15.43:0/1540876045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f9827189e0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f02832ceb80 1 -- 172.21.15.43:0/1540876045 >> 172.21.15.43:0/1540876045 conn(0x55f9825effc0 msgr2=0x55f982709580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f024fe00700 0 client.17313 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:26:52.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.636+0000 7f02832ceb80 1 -- 172.21.15.43:0/1540876045 shutdown_connections 2024-05-27T07:26:52.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.637+0000 7f02832ceb80 1 -- 172.21.15.43:0/1540876045 wait complete. 2024-05-27T07:26:52.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.637+0000 7f02832ceb80 20 client.17313 trim_cache size 0 max 16384 2024-05-27T07:26:52.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.656+0000 7f02832ceb80 1 Processor -- start 2024-05-27T07:26:52.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.656+0000 7f02832ceb80 1 -- start start 2024-05-27T07:26:52.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.656+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98258fe40 0x55f98270e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.656+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f982717400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.656+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f98271a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.656+0000 7f02832ceb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f026c1b34f0 con 0x55f98270edb0 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.656+0000 7f02832ceb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f026c1b11f0 con 0x55f982714d30 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.656+0000 7f02832ceb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f026c1b2e20 con 0x55f98258fe40 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.657+0000 7f0271400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f982717400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.657+0000 7f0271400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f982717400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59154/0 (socket says 172.21.15.43:59154) 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.657+0000 7f0271400700 1 -- 172.21.15.43:0/3205411238 learned_addr learned my addr 172.21.15.43:0/3205411238 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.657+0000 7f0271e00700 1 --2- 172.21.15.43:0/3205411238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98258fe40 0x55f98270e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.657+0000 7f0272800700 1 --2- 172.21.15.43:0/3205411238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f98271a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.657+0000 7f0271400700 1 -- 172.21.15.43:0/3205411238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98258fe40 msgr2=0x55f98270e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.657+0000 7f0271400700 1 --2- 172.21.15.43:0/3205411238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98258fe40 0x55f98270e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.657+0000 7f0271400700 1 -- 172.21.15.43:0/3205411238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 msgr2=0x55f98271a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.657+0000 7f0271400700 1 --2- 172.21.15.43:0/3205411238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f98271a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.657+0000 7f0271400700 1 -- 172.21.15.43:0/3205411238 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f98272b420 con 0x55f98270edb0 2024-05-27T07:26:52.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.658+0000 7f0271400700 1 --2- 172.21.15.43:0/3205411238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f982717400 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f025c00bf60 tx=0x7f025c01af50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b904a9dce01c3e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.658+0000 7f025aa00700 1 -- 172.21.15.43:0/3205411238 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c00eac0 con 0x55f98270edb0 2024-05-27T07:26:52.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.658+0000 7f025aa00700 1 -- 172.21.15.43:0/3205411238 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f025c00e7e0 con 0x55f98270edb0 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.658+0000 7f025aa00700 1 -- 172.21.15.43:0/3205411238 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c01a750 con 0x55f98270edb0 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.659+0000 7f02832ceb80 1 -- 172.21.15.43:0/3205411238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 msgr2=0x55f982717400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.659+0000 7f02832ceb80 1 --2- 172.21.15.43:0/3205411238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f982717400 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f025c00bf60 tx=0x7f025c01af50 comp rx=0 tx=0).stop 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.659+0000 7f02832ceb80 1 -- 172.21.15.43:0/3205411238 shutdown_connections 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.659+0000 7f02832ceb80 1 --2- 172.21.15.43:0/3205411238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f98271a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.659+0000 7f02832ceb80 1 --2- 172.21.15.43:0/3205411238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98258fe40 0x55f98270e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.659+0000 7f02832ceb80 1 --2- 172.21.15.43:0/3205411238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98270edb0 0x55f982717400 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.659+0000 7f02832ceb80 1 -- 172.21.15.43:0/3205411238 >> 172.21.15.43:0/3205411238 conn(0x55f9825effc0 msgr2=0x55f982712210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.659+0000 7f02832ceb80 1 -- 172.21.15.43:0/3205411238 shutdown_connections 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.659+0000 7f02832ceb80 1 -- 172.21.15.43:0/3205411238 wait complete. 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.660+0000 7f02832ceb80 1 -- start start 2024-05-27T07:26:52.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.660+0000 7f02832ceb80 1 Processor -- start 2024-05-27T07:26:52.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.661+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98258fe40 0x55f9827308c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.661+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98270edb0 0x55f982730e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.661+0000 7f02832ceb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f982720a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.661+0000 7f02832ceb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f026c1b1fc0 con 0x55f98258fe40 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.661+0000 7f02832ceb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f026c1b18c0 con 0x55f982714d30 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.661+0000 7f02832ceb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f026c1b26f0 con 0x55f98270edb0 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.661+0000 7f02832ceb80 10 client.0 _ll_register_callbacks cb 0x55f9825d6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.661+0000 7f0271e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98258fe40 0x55f9827308c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.661+0000 7f0271e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98258fe40 0x55f9827308c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59158/0 (socket says 172.21.15.43:59158) 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.661+0000 7f0271e00700 1 -- 172.21.15.43:0/4176666279 learned_addr learned my addr 172.21.15.43:0/4176666279 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.662+0000 7f0271400700 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98270edb0 0x55f982730e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.662+0000 7f0272800700 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f982720a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.662+0000 7f0271e00700 1 -- 172.21.15.43:0/4176666279 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98270edb0 msgr2=0x55f982730e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.662+0000 7f0271e00700 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98270edb0 0x55f982730e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.662+0000 7f0271e00700 1 -- 172.21.15.43:0/4176666279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 msgr2=0x55f982720a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.662+0000 7f0271e00700 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f982720a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.662+0000 7f0271e00700 1 -- 172.21.15.43:0/4176666279 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f982721180 con 0x55f98258fe40 2024-05-27T07:26:52.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.662+0000 7f0271e00700 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98258fe40 0x55f9827308c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f02600a53e0 tx=0x7f02600108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77aee25d1664c8e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.662+0000 7f024fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.663+0000 7f024fe00700 1 -- 172.21.15.43:0/4176666279 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0260020040 con 0x55f98258fe40 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.663+0000 7f024fe00700 1 -- 172.21.15.43:0/4176666279 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7f02600e2120 con 0x55f98258fe40 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.663+0000 7f02832ceb80 1 -- 172.21.15.43:0/4176666279 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f025c00e7e0 con 0x55f98258fe40 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.663+0000 7f024fe00700 1 -- 172.21.15.43:0/4176666279 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=111) ==== 24+0+0 (secure 0 0 0) 0x7f02600a9360 con 0x55f98258fe40 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.663+0000 7f02832ceb80 10 client.17319 fetch_fsmap learned FSMap version 111 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.663+0000 7f02832ceb80 1 -- 172.21.15.43:0/4176666279 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=111,monmap=4+}) -- 0x55f9827302c0 con 0x55f98258fe40 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.663+0000 7f024fe00700 1 -- 172.21.15.43:0/4176666279 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f025c00e7e0 con 0x55f98258fe40 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.663+0000 7f024fe00700 1 -- 172.21.15.43:0/4176666279 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 111) ==== 4800+0+0 (secure 0 0 0) 0x7f02600ca170 con 0x55f98258fe40 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.664+0000 7f02832ceb80 10 client.17319 fetch_fsmap finished waiting for FSMap version 111 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.664+0000 7f02832ceb80 10 client.17319 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.664+0000 7f02832ceb80 20 client.17319 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:52.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.664+0000 7f02832ceb80 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55f982792820 0x55f982794c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:52.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.664+0000 7f02832ceb80 4 client.17319 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:26:52.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.664+0000 7f02832ceb80 1 -- 172.21.15.43:0/4176666279 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f9824af250 con 0x55f982792820 2024-05-27T07:26:52.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.664+0000 7f0271400700 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55f982792820 0x55f982794c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:52.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.665+0000 7f0271400700 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55f982792820 0x55f982794c40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2041ca9d7db38cc4 server_cookie=e53f4b2927c57223 in_seq=0 out_seq=0 2024-05-27T07:26:52.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.665+0000 7f024fe00700 10 client.17319 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:26:52.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f024fe00700 1 -- 172.21.15.43:0/4176666279 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f9824af250 con 0x55f982792820 2024-05-27T07:26:52.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f024fe00700 10 client.17319 handle_command_reply: tid=0 2024-05-27T07:26:52.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f02832ceb80 1 client.17319 shutdown 2024-05-27T07:26:52.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f02832ceb80 1 -- 172.21.15.43:0/4176666279 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98258fe40 msgr2=0x55f9827308c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f02832ceb80 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98258fe40 0x55f9827308c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f02600a53e0 tx=0x7f02600108d0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f02832ceb80 1 -- 172.21.15.43:0/4176666279 shutdown_connections 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f02832ceb80 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55f982792820 0x55f982794c40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f02832ceb80 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f982714d30 0x55f982720a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f02832ceb80 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f98258fe40 0x55f9827308c0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f02832ceb80 1 --2- 172.21.15.43:0/4176666279 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f98270edb0 0x55f982730e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f02832ceb80 1 -- 172.21.15.43:0/4176666279 >> 172.21.15.43:0/4176666279 conn(0x55f9825effc0 msgr2=0x55f982713d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.666+0000 7f024fe00700 0 client.17319 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.667+0000 7f02832ceb80 1 -- 172.21.15.43:0/4176666279 shutdown_connections 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.667+0000 7f02832ceb80 1 -- 172.21.15.43:0/4176666279 wait complete. 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.667+0000 7f02832ceb80 20 client.17319 trim_cache size 0 max 16384 2024-05-27T07:26:52.657 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:26:52.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.667+0000 7f0274800700 1 -- 172.21.15.43:0/961446719 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02540b2a70 msgr2=0x7f02540b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02540b2a70 0x7f02540b4f80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0268009900 tx=0x7f02680099e0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 -- 172.21.15.43:0/961446719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1034b0 msgr2=0x7f026c0ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:52.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1034b0 0x7f026c0ff290 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f026000d950 tx=0x7f026000dc60 comp rx=0 tx=0).stop 2024-05-27T07:26:52.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 -- 172.21.15.43:0/961446719 shutdown_connections 2024-05-27T07:26:52.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02540b2a70 0x7f02540b4f80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f026c108730 0x7f026c0ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f026c1034b0 0x7f026c0ff290 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 --2- 172.21.15.43:0/961446719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f026c107de0 0x7f026c0ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:52.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 -- 172.21.15.43:0/961446719 >> 172.21.15.43:0/961446719 conn(0x7f026c0fedb0 msgr2=0x7f026c06b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:52.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 -- 172.21.15.43:0/961446719 shutdown_connections 2024-05-27T07:26:52.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:52.668+0000 7f0274800700 1 -- 172.21.15.43:0/961446719 wait complete. 2024-05-27T07:26:53.247 INFO:tasks.workunit.client.0.smithi043.stdout: 1 72579 0.58 MB/sec execute 537 sec latency 66.635 ms 2024-05-27T07:26:53.451 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:53 smithi043 ceph-mon[31531]: pgmap v436: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 544 KiB/s wr, 52 op/s 2024-05-27T07:26:53.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:53 smithi179 ceph-mon[35888]: pgmap v436: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 544 KiB/s wr, 52 op/s 2024-05-27T07:26:53.492 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:53 smithi132 ceph-mon[35283]: pgmap v436: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 544 KiB/s wr, 52 op/s 2024-05-27T07:26:53.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:26:54.118 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:54.247 INFO:tasks.workunit.client.0.smithi043.stdout: 1 72739 0.58 MB/sec execute 538 sec latency 58.931 ms 2024-05-27T07:26:55.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 72853 0.58 MB/sec execute 539 sec latency 60.485 ms 2024-05-27T07:26:55.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:55 smithi043 ceph-mon[31531]: pgmap v437: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 532 KiB/s wr, 51 op/s 2024-05-27T07:26:55.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:55 smithi179 ceph-mon[35888]: pgmap v437: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 532 KiB/s wr, 51 op/s 2024-05-27T07:26:55.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:55 smithi132 ceph-mon[35283]: pgmap v437: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 532 KiB/s wr, 51 op/s 2024-05-27T07:26:56.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 73010 0.58 MB/sec execute 540 sec latency 60.171 ms 2024-05-27T07:26:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.574+0000 7f56bbc00700 1 -- 172.21.15.43:0/121374619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56b4108f10 msgr2=0x7f56b4109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.574+0000 7f56bbc00700 1 --2- 172.21.15.43:0/121374619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56b4108f10 0x7f56b4109320 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f56a00099c0 tx=0x7f56a0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.574+0000 7f56bbc00700 1 -- 172.21.15.43:0/121374619 shutdown_connections 2024-05-27T07:26:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.574+0000 7f56bbc00700 1 --2- 172.21.15.43:0/121374619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f56b4109860 0x7f56b41111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.574+0000 7f56bbc00700 1 --2- 172.21.15.43:0/121374619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56b41045e0 0x7f56b41049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.574+0000 7f56bbc00700 1 --2- 172.21.15.43:0/121374619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56b4108f10 0x7f56b4109320 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.574+0000 7f56bbc00700 1 -- 172.21.15.43:0/121374619 >> 172.21.15.43:0/121374619 conn(0x7f56b4100030 msgr2=0x7f56b4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.574+0000 7f56bbc00700 1 -- 172.21.15.43:0/121374619 shutdown_connections 2024-05-27T07:26:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.574+0000 7f56bbc00700 1 -- 172.21.15.43:0/121374619 wait complete. 2024-05-27T07:26:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.576+0000 7f56bbc00700 1 Processor -- start 2024-05-27T07:26:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.576+0000 7f56bbc00700 1 -- start start 2024-05-27T07:26:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.576+0000 7f56bbc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56b41045e0 0x7f56b4075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.576+0000 7f56bbc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56b4108f10 0x7f56b4075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.576+0000 7f56bbc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f56b4109860 0x7f56b4076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.576+0000 7f56bbc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f56b4114340 con 0x7f56b4108f10 2024-05-27T07:26:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.576+0000 7f56bbc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41141c0 con 0x7f56b41045e0 2024-05-27T07:26:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56bbc00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41144c0 con 0x7f56b4109860 2024-05-27T07:26:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56b4108f10 0x7f56b4075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56b4108f10 0x7f56b4075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43986/0 (socket says 172.21.15.43:43986) 2024-05-27T07:26:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b3e00700 1 -- 172.21.15.43:0/1255384062 learned_addr learned my addr 172.21.15.43:0/1255384062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b9200700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56b41045e0 0x7f56b4075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b9c00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f56b4109860 0x7f56b4076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b3e00700 1 -- 172.21.15.43:0/1255384062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f56b4109860 msgr2=0x7f56b4076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b3e00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f56b4109860 0x7f56b4076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b3e00700 1 -- 172.21.15.43:0/1255384062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56b41045e0 msgr2=0x7f56b4075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b3e00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56b41045e0 0x7f56b4075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b3e00700 1 -- 172.21.15.43:0/1255384062 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b41a90f0 con 0x7f56b4108f10 2024-05-27T07:26:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.577+0000 7f56b9200700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56b41045e0 0x7f56b4075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.578+0000 7f56b3e00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56b4108f10 0x7f56b4075ff0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f56a000b580 tx=0x7f56a000f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.578+0000 7f56b1600700 1 -- 172.21.15.43:0/1255384062 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a0027070 con 0x7f56b4108f10 2024-05-27T07:26:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.578+0000 7f56b1600700 1 -- 172.21.15.43:0/1255384062 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56a000fda0 con 0x7f56b4108f10 2024-05-27T07:26:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.578+0000 7f56bbc00700 1 -- 172.21.15.43:0/1255384062 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b41a9380 con 0x7f56b4108f10 2024-05-27T07:26:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.578+0000 7f56b1600700 1 -- 172.21.15.43:0/1255384062 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a00173f0 con 0x7f56b4108f10 2024-05-27T07:26:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.579+0000 7f56bbc00700 1 -- 172.21.15.43:0/1255384062 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b41a9830 con 0x7f56b4108f10 2024-05-27T07:26:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.583+0000 7f56b1600700 1 -- 172.21.15.43:0/1255384062 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f56a002a070 con 0x7f56b4108f10 2024-05-27T07:26:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.584+0000 7f56ca7bcb80 1 Processor -- start 2024-05-27T07:26:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.584+0000 7f56b1600700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f569c0b2a70 0x7f569c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.584+0000 7f56ca7bcb80 1 -- start start 2024-05-27T07:26:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.584+0000 7f56b1600700 1 -- 172.21.15.43:0/1255384062 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7f56a00e3840 con 0x7f56b4108f10 2024-05-27T07:26:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.585+0000 7f56b9c00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f569c0b2a70 0x7f569c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.585+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a257f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.585+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 0x55d154a1f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.585+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a1fdb0 0x55d154a201c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.585+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d15494de40 con 0x55d1548a0e40 2024-05-27T07:26:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.585+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d15493d9e0 con 0x55d154a1fdb0 2024-05-27T07:26:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.585+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d154941a20 con 0x55d154a25d30 2024-05-27T07:26:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.585+0000 7f56b9200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a257f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.585+0000 7f56b9200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a257f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43990/0 (socket says 172.21.15.43:43990) 2024-05-27T07:26:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.585+0000 7f56b9200700 1 -- 172.21.15.43:0/2389829715 learned_addr learned my addr 172.21.15.43:0/2389829715 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.586+0000 7f56b9c00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f569c0b2a70 0x7f569c0b4f80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f56ac009c60 tx=0x7f56ac009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.586+0000 7f56b3e00700 1 --2- 172.21.15.43:0/2389829715 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 0x55d154a1f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.586+0000 7f56b9c00700 1 --2- 172.21.15.43:0/2389829715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a1fdb0 0x55d154a201c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.586+0000 7f56b9200700 1 -- 172.21.15.43:0/2389829715 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 msgr2=0x55d154a1f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.586+0000 7f56b9200700 1 --2- 172.21.15.43:0/2389829715 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 0x55d154a1f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.586+0000 7f56b9200700 1 -- 172.21.15.43:0/2389829715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a1fdb0 msgr2=0x55d154a201c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.586+0000 7f56b9200700 1 --2- 172.21.15.43:0/2389829715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a1fdb0 0x55d154a201c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.586+0000 7f56b9200700 1 -- 172.21.15.43:0/2389829715 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d154906420 con 0x55d1548a0e40 2024-05-27T07:26:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.587+0000 7f56b9200700 1 --2- 172.21.15.43:0/2389829715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a257f0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f56a800bd90 tx=0x7f56a800f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2dd3812c7a66118 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.587+0000 7f56a6000700 1 -- 172.21.15.43:0/2389829715 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a800fda0 con 0x55d1548a0e40 2024-05-27T07:26:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.587+0000 7f56a6000700 1 -- 172.21.15.43:0/2389829715 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56a80118d0 con 0x55d1548a0e40 2024-05-27T07:26:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.587+0000 7f56a6000700 1 -- 172.21.15.43:0/2389829715 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a80106c0 con 0x55d1548a0e40 2024-05-27T07:26:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.588+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/2389829715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 msgr2=0x55d154a257f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.588+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/2389829715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a257f0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f56a800bd90 tx=0x7f56a800f640 comp rx=0 tx=0).stop 2024-05-27T07:26:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.588+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/2389829715 shutdown_connections 2024-05-27T07:26:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.588+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/2389829715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a1fdb0 0x55d154a201c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.588+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/2389829715 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a257f0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.588+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/2389829715 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 0x55d154a1f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.588+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/2389829715 >> 172.21.15.43:0/2389829715 conn(0x55d154900fc0 msgr2=0x55d154a1a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.588+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/2389829715 shutdown_connections 2024-05-27T07:26:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.588+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/2389829715 wait complete. 2024-05-27T07:26:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.589+0000 7f56ca7bcb80 1 -- start start 2024-05-27T07:26:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.589+0000 7f56ca7bcb80 1 Processor -- start 2024-05-27T07:26:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.590+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1548a0e40 0x55d154a29540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.590+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a1fdb0 0x55d154a29a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.590+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a25d30 0x55d154a29fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.590+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d154843fa0 con 0x55d154a25d30 2024-05-27T07:26:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.590+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d1548de760 con 0x55d1548a0e40 2024-05-27T07:26:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.590+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d154a2c800 con 0x55d154a1fdb0 2024-05-27T07:26:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.590+0000 7f56ca7bcb80 10 client.0 _ll_register_callbacks cb 0x55d1548e7c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.590+0000 7f56b9c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a25d30 0x55d154a29fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.591+0000 7f56b9c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a25d30 0x55d154a29fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44000/0 (socket says 172.21.15.43:44000) 2024-05-27T07:26:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.591+0000 7f56b9200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1548a0e40 0x55d154a29540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.591+0000 7f56b9c00700 1 -- 172.21.15.43:0/3042320861 learned_addr learned my addr 172.21.15.43:0/3042320861 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.591+0000 7f56b3e00700 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a1fdb0 0x55d154a29a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.591+0000 7f56b9c00700 1 -- 172.21.15.43:0/3042320861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a1fdb0 msgr2=0x55d154a29a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.591+0000 7f56b9c00700 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a1fdb0 0x55d154a29a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.591+0000 7f56b9c00700 1 -- 172.21.15.43:0/3042320861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1548a0e40 msgr2=0x55d154a29540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.591+0000 7f56b9c00700 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1548a0e40 0x55d154a29540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.591+0000 7f56b9c00700 1 -- 172.21.15.43:0/3042320861 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15488d8d0 con 0x55d154a25d30 2024-05-27T07:26:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.592+0000 7f56b9c00700 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a25d30 0x55d154a29fc0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0169d0 tx=0x7f56ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bad706540a8d912 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.592+0000 7f569aa00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.592+0000 7f569aa00700 1 -- 172.21.15.43:0/3042320861 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac014660 con 0x55d154a25d30 2024-05-27T07:26:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.592+0000 7f569aa00700 1 -- 172.21.15.43:0/3042320861 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7f56ac014800 con 0x55d154a25d30 2024-05-27T07:26:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.592+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3042320861 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56a80118d0 con 0x55d154a25d30 2024-05-27T07:26:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.592+0000 7f569aa00700 1 -- 172.21.15.43:0/3042320861 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f56ac014a00 con 0x55d154a25d30 2024-05-27T07:26:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.593+0000 7f56ca7bcb80 10 client.17331 fetch_fsmap learned FSMap version 112 2024-05-27T07:26:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.593+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3042320861 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x55d15490e060 con 0x55d154a25d30 2024-05-27T07:26:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.593+0000 7f569aa00700 1 -- 172.21.15.43:0/3042320861 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56a80118d0 con 0x55d154a25d30 2024-05-27T07:26:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.593+0000 7f569aa00700 1 -- 172.21.15.43:0/3042320861 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7f56ac022c50 con 0x55d154a25d30 2024-05-27T07:26:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.593+0000 7f56ca7bcb80 10 client.17331 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:26:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.593+0000 7f56ca7bcb80 10 client.17331 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:26:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.593+0000 7f56ca7bcb80 20 client.17331 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.593+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d154a39ba0 0x55d154a3c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.593+0000 7f56ca7bcb80 4 client.17331 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:26:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.593+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3042320861 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d15494fd90 con 0x55d154a39ba0 2024-05-27T07:26:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.594+0000 7f56b9200700 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d154a39ba0 0x55d154a3c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.595+0000 7f56b9200700 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d154a39ba0 0x55d154a3c030 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=88c717926a9f1fb1 server_cookie=b46782ab9dcc4e50 in_seq=0 out_seq=0 2024-05-27T07:26:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.595+0000 7f569aa00700 10 client.17331 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:26:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.597+0000 7f569aa00700 1 -- 172.21.15.43:0/3042320861 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d15494fd90 con 0x55d154a39ba0 2024-05-27T07:26:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.597+0000 7f569aa00700 10 client.17331 handle_command_reply: tid=0 2024-05-27T07:26:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.597+0000 7f56ca7bcb80 1 client.17331 shutdown 2024-05-27T07:26:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.597+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3042320861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a25d30 msgr2=0x55d154a29fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.597+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a25d30 0x55d154a29fc0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0169d0 tx=0x7f56ac004ad0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.598+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3042320861 shutdown_connections 2024-05-27T07:26:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.598+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a25d30 0x55d154a29fc0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.598+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d154a39ba0 0x55d154a3c030 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.598+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1548a0e40 0x55d154a29540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.598+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3042320861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a1fdb0 0x55d154a29a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.598+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3042320861 >> 172.21.15.43:0/3042320861 conn(0x55d154900fc0 msgr2=0x55d154a1a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.598+0000 7f569aa00700 0 client.17331 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:26:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.598+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3042320861 shutdown_connections 2024-05-27T07:26:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.598+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3042320861 wait complete. 2024-05-27T07:26:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.598+0000 7f56ca7bcb80 20 client.17331 trim_cache size 0 max 16384 2024-05-27T07:26:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.616+0000 7f56ca7bcb80 1 Processor -- start 2024-05-27T07:26:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.616+0000 7f56ca7bcb80 1 -- start start 2024-05-27T07:26:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.617+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a1f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.617+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a1fdb0 0x55d154a29f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.617+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 0x55d154a2b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.617+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41b33b0 con 0x55d1548a0e40 2024-05-27T07:26:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.617+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41b1080 con 0x55d154a1fdb0 2024-05-27T07:26:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.617+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41b2ce0 con 0x55d154a25d30 2024-05-27T07:26:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.617+0000 7f56b9200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a1f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.617+0000 7f56b9200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a1f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44004/0 (socket says 172.21.15.43:44004) 2024-05-27T07:26:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.617+0000 7f56b9200700 1 -- 172.21.15.43:0/3384310679 learned_addr learned my addr 172.21.15.43:0/3384310679 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.618+0000 7f56b9c00700 1 --2- 172.21.15.43:0/3384310679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 0x55d154a2b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.618+0000 7f56b3e00700 1 --2- 172.21.15.43:0/3384310679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a1fdb0 0x55d154a29f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.618+0000 7f56b9200700 1 -- 172.21.15.43:0/3384310679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 msgr2=0x55d154a2b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.618+0000 7f56b9200700 1 --2- 172.21.15.43:0/3384310679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 0x55d154a2b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.618+0000 7f56b9200700 1 -- 172.21.15.43:0/3384310679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a1fdb0 msgr2=0x55d154a29f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.618+0000 7f56b9200700 1 --2- 172.21.15.43:0/3384310679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a1fdb0 0x55d154a29f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.618+0000 7f56b9200700 1 -- 172.21.15.43:0/3384310679 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d154a201c0 con 0x55d1548a0e40 2024-05-27T07:26:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.618+0000 7f56b9c00700 1 --2- 172.21.15.43:0/3384310679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 0x55d154a2b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:26:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.618+0000 7f56b9200700 1 --2- 172.21.15.43:0/3384310679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a1f870 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f56a800bd90 tx=0x7f56a800ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f917096f4a809a5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.618+0000 7f56a5600700 1 -- 172.21.15.43:0/3384310679 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a802b070 con 0x55d1548a0e40 2024-05-27T07:26:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.619+0000 7f56a5600700 1 -- 172.21.15.43:0/3384310679 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56a800fac0 con 0x55d1548a0e40 2024-05-27T07:26:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.619+0000 7f56a5600700 1 -- 172.21.15.43:0/3384310679 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a8004e00 con 0x55d1548a0e40 2024-05-27T07:26:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.620+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3384310679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 msgr2=0x55d154a1f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.620+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3384310679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a1f870 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f56a800bd90 tx=0x7f56a800ff70 comp rx=0 tx=0).stop 2024-05-27T07:26:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.620+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3384310679 shutdown_connections 2024-05-27T07:26:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.620+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3384310679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d154a25d30 0x55d154a2b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.620+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3384310679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1548a0e40 0x55d154a1f870 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.620+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3384310679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a1fdb0 0x55d154a29f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.620+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3384310679 >> 172.21.15.43:0/3384310679 conn(0x55d154900fc0 msgr2=0x55d154a23140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.620+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3384310679 shutdown_connections 2024-05-27T07:26:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.620+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3384310679 wait complete. 2024-05-27T07:26:56.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.620+0000 7f56ca7bcb80 1 -- start start 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.621+0000 7f56ca7bcb80 1 Processor -- start 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.621+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d1548a0e40 0x55d154a31e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.621+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a1fdb0 0x55d154a40e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.621+0000 7f56ca7bcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a25d30 0x55d154a41340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.621+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41b1e80 con 0x55d154a1fdb0 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.621+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41b1750 con 0x55d154a25d30 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.621+0000 7f56ca7bcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f56b41b25b0 con 0x55d1548a0e40 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.621+0000 7f56ca7bcb80 10 client.0 _ll_register_callbacks cb 0x55d1548e7c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a1fdb0 0x55d154a40e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a1fdb0 0x55d154a40e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44016/0 (socket says 172.21.15.43:44016) 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b3e00700 1 -- 172.21.15.43:0/3130673504 learned_addr learned my addr 172.21.15.43:0/3130673504 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b9200700 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d1548a0e40 0x55d154a31e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b9c00700 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a25d30 0x55d154a41340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b3e00700 1 -- 172.21.15.43:0/3130673504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d1548a0e40 msgr2=0x55d154a31e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:56.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b3e00700 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d1548a0e40 0x55d154a31e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b3e00700 1 -- 172.21.15.43:0/3130673504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a25d30 msgr2=0x55d154a41340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:26:56.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b3e00700 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a25d30 0x55d154a41340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b3e00700 1 -- 172.21.15.43:0/3130673504 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d154a41880 con 0x55d154a1fdb0 2024-05-27T07:26:56.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.622+0000 7f56b9200700 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d1548a0e40 0x55d154a31e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:26:56.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.623+0000 7f56b3e00700 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a1fdb0 0x55d154a40e00 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f56a00e0bc0 tx=0x7f56a0009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cffba5e5e4aec46 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:26:56.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.623+0000 7f569aa00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:26:56.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.623+0000 7f569aa00700 1 -- 172.21.15.43:0/3130673504 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a00a9ef0 con 0x55d154a1fdb0 2024-05-27T07:26:56.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.623+0000 7f569aa00700 1 -- 172.21.15.43:0/3130673504 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7f56a00b30c0 con 0x55d154a1fdb0 2024-05-27T07:26:56.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.623+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3130673504 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56a800fac0 con 0x55d154a1fdb0 2024-05-27T07:26:56.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.623+0000 7f569aa00700 1 -- 172.21.15.43:0/3130673504 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f56a00aa290 con 0x55d154a1fdb0 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.623+0000 7f56ca7bcb80 10 client.17337 fetch_fsmap learned FSMap version 112 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.623+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3130673504 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x55d154aa42c0 con 0x55d154a1fdb0 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.624+0000 7f569aa00700 1 -- 172.21.15.43:0/3130673504 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f56a800fac0 con 0x55d154a1fdb0 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.624+0000 7f569aa00700 1 -- 172.21.15.43:0/3130673504 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7f56a0092070 con 0x55d154a1fdb0 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.624+0000 7f56ca7bcb80 10 client.17337 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.624+0000 7f56ca7bcb80 10 client.17337 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.624+0000 7f56ca7bcb80 20 client.17337 populate_metadata read hostname 'smithi043' 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.624+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d154aa49b0 0x55d154aa6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.624+0000 7f56ca7bcb80 4 client.17337 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.624+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3130673504 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d154949570 con 0x55d154aa49b0 2024-05-27T07:26:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.625+0000 7f56b9200700 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d154aa49b0 0x55d154aa6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:26:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.626+0000 7f56b9200700 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d154aa49b0 0x55d154aa6dd0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7dbb568f35743d32 server_cookie=7b50b25e709bb18d in_seq=0 out_seq=0 2024-05-27T07:26:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.626+0000 7f569aa00700 10 client.17337 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:26:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.626+0000 7f569aa00700 1 -- 172.21.15.43:0/3130673504 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d154949570 con 0x55d154aa49b0 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.626+0000 7f569aa00700 10 client.17337 handle_command_reply: tid=0 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.626+0000 7f56ca7bcb80 1 client.17337 shutdown 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.626+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3130673504 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a1fdb0 msgr2=0x55d154a40e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.626+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a1fdb0 0x55d154a40e00 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f56a00e0bc0 tx=0x7f56a0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.626+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3130673504 shutdown_connections 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.627+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55d154aa49b0 0x55d154aa6dd0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.627+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d154a25d30 0x55d154a41340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.627+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d1548a0e40 0x55d154a31e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.627+0000 7f56ca7bcb80 1 --2- 172.21.15.43:0/3130673504 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d154a1fdb0 0x55d154a40e00 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.627+0000 7f569aa00700 0 client.17337 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.627+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3130673504 >> 172.21.15.43:0/3130673504 conn(0x55d154900fc0 msgr2=0x55d154a29320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.627+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3130673504 shutdown_connections 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.627+0000 7f56ca7bcb80 1 -- 172.21.15.43:0/3130673504 wait complete. 2024-05-27T07:26:56.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.627+0000 7f56ca7bcb80 20 client.17337 trim_cache size 0 max 16384 2024-05-27T07:26:56.618 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.628+0000 7f56bbc00700 1 -- 172.21.15.43:0/1255384062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f569c0b2a70 msgr2=0x7f569c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.628+0000 7f56bbc00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f569c0b2a70 0x7f569c0b4f80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f56ac009c60 tx=0x7f56ac009040 comp rx=0 tx=0).stop 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.628+0000 7f56bbc00700 1 -- 172.21.15.43:0/1255384062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56b4108f10 msgr2=0x7f56b4075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.628+0000 7f56bbc00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56b4108f10 0x7f56b4075ff0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f56a000b580 tx=0x7f56a000f460 comp rx=0 tx=0).stop 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.628+0000 7f56bbc00700 1 -- 172.21.15.43:0/1255384062 shutdown_connections 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.628+0000 7f56bbc00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f569c0b2a70 0x7f569c0b4f80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.628+0000 7f56bbc00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f56b4109860 0x7f56b4076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.628+0000 7f56bbc00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56b41045e0 0x7f56b4075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.628+0000 7f56bbc00700 1 --2- 172.21.15.43:0/1255384062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56b4108f10 0x7f56b4075ff0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.628+0000 7f56bbc00700 1 -- 172.21.15.43:0/1255384062 >> 172.21.15.43:0/1255384062 conn(0x7f56b4100030 msgr2=0x7f56b410a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.629+0000 7f56bbc00700 1 -- 172.21.15.43:0/1255384062 shutdown_connections 2024-05-27T07:26:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:26:56.629+0000 7f56bbc00700 1 -- 172.21.15.43:0/1255384062 wait complete. 2024-05-27T07:26:57.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 73173 0.58 MB/sec execute 541 sec latency 83.236 ms 2024-05-27T07:26:57.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:57 smithi043 ceph-mon[31531]: pgmap v438: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 547 KiB/s wr, 58 op/s 2024-05-27T07:26:57.384 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:57 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:57.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:57 smithi179 ceph-mon[35888]: pgmap v438: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 547 KiB/s wr, 58 op/s 2024-05-27T07:26:57.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:57 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:57.459 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:26:57.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:57 smithi132 ceph-mon[35283]: pgmap v438: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 547 KiB/s wr, 58 op/s 2024-05-27T07:26:57.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:57 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:26:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:26:58.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 73331 0.58 MB/sec execute 542 sec latency 49.742 ms 2024-05-27T07:26:59.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 73404 0.58 MB/sec execute 543 sec latency 50.683 ms 2024-05-27T07:26:59.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:26:59 smithi043 ceph-mon[31531]: pgmap v439: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 362 KiB/s wr, 40 op/s 2024-05-27T07:26:59.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:26:59 smithi179 ceph-mon[35888]: pgmap v439: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 362 KiB/s wr, 40 op/s 2024-05-27T07:26:59.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:26:59 smithi132 ceph-mon[35283]: pgmap v439: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 362 KiB/s wr, 40 op/s 2024-05-27T07:27:00.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 73492 0.58 MB/sec execute 544 sec latency 57.479 ms 2024-05-27T07:27:00.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.379+0000 7fc0e2200700 1 -- 172.21.15.43:0/977745915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0dc1043f0 msgr2=0x7fc0dc104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:00.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.379+0000 7fc0e2200700 1 --2- 172.21.15.43:0/977745915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0dc1043f0 0x7fc0dc104800 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00099c0 tx=0x7fc0d0009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.379+0000 7fc0e2200700 1 -- 172.21.15.43:0/977745915 shutdown_connections 2024-05-27T07:27:00.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.379+0000 7fc0e2200700 1 --2- 172.21.15.43:0/977745915 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0dc109670 0x7fc0dc110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.379+0000 7fc0e2200700 1 --2- 172.21.15.43:0/977745915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0dc1043f0 0x7fc0dc104800 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.379+0000 7fc0e2200700 1 --2- 172.21.15.43:0/977745915 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0dc108d20 0x7fc0dc109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.379+0000 7fc0e2200700 1 -- 172.21.15.43:0/977745915 >> 172.21.15.43:0/977745915 conn(0x7fc0dc079d00 msgr2=0x7fc0dc07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:00.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.379+0000 7fc0e2200700 1 -- 172.21.15.43:0/977745915 shutdown_connections 2024-05-27T07:27:00.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.379+0000 7fc0e2200700 1 -- 172.21.15.43:0/977745915 wait complete. 2024-05-27T07:27:00.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.380+0000 7fc0e2200700 1 Processor -- start 2024-05-27T07:27:00.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.381+0000 7fc0e2200700 1 -- start start 2024-05-27T07:27:00.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.381+0000 7fc0e2200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0dc1043f0 0x7fc0dc19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.381+0000 7fc0e2200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0dc108d20 0x7fc0dc19d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.381+0000 7fc0e2200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0dc109670 0x7fc0dc1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.381+0000 7fc0e2200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc1141a0 con 0x7fc0dc108d20 2024-05-27T07:27:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.381+0000 7fc0e2200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc114020 con 0x7fc0dc1043f0 2024-05-27T07:27:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.381+0000 7fc0e2200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc114320 con 0x7fc0dc109670 2024-05-27T07:27:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.381+0000 7fc0dbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0dc108d20 0x7fc0dc19d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.382+0000 7fc0dbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0dc108d20 0x7fc0dc19d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44034/0 (socket says 172.21.15.43:44034) 2024-05-27T07:27:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.382+0000 7fc0dbe00700 1 -- 172.21.15.43:0/2851868317 learned_addr learned my addr 172.21.15.43:0/2851868317 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.382+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0dc1043f0 0x7fc0dc19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.382+0000 7fc0e1800700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0dc109670 0x7fc0dc1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.382+0000 7fc0dbe00700 1 -- 172.21.15.43:0/2851868317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0dc109670 msgr2=0x7fc0dc1a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.382+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0dc109670 0x7fc0dc1a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.382+0000 7fc0dbe00700 1 -- 172.21.15.43:0/2851868317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0dc1043f0 msgr2=0x7fc0dc19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.382+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0dc1043f0 0x7fc0dc19cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.382+0000 7fc0dbe00700 1 -- 172.21.15.43:0/2851868317 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc1a1bb0 con 0x7fc0dc108d20 2024-05-27T07:27:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.382+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0dc1043f0 0x7fc0dc19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.383+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0dc108d20 0x7fc0dc19d330 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fc0c800c9e0 tx=0x7fc0c800cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.383+0000 7fc0d9600700 1 -- 172.21.15.43:0/2851868317 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8007aa0 con 0x7fc0dc108d20 2024-05-27T07:27:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.383+0000 7fc0d9600700 1 -- 172.21.15.43:0/2851868317 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0c8007c40 con 0x7fc0dc108d20 2024-05-27T07:27:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.383+0000 7fc0d9600700 1 -- 172.21.15.43:0/2851868317 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8018660 con 0x7fc0dc108d20 2024-05-27T07:27:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.383+0000 7fc0e2200700 1 -- 172.21.15.43:0/2851868317 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0dc1a1ea0 con 0x7fc0dc108d20 2024-05-27T07:27:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.383+0000 7fc0e2200700 1 -- 172.21.15.43:0/2851868317 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0dc1a2390 con 0x7fc0dc108d20 2024-05-27T07:27:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.388+0000 7fc0d9600700 1 -- 172.21.15.43:0/2851868317 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc0c8018800 con 0x7fc0dc108d20 2024-05-27T07:27:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.389+0000 7fc0f23f3b80 1 Processor -- start 2024-05-27T07:27:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.389+0000 7fc0f23f3b80 1 -- start start 2024-05-27T07:27:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.389+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfcffed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.389+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.389+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfd00820 0x55d0bfd0d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.389+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d0bfc2ee40 con 0x55d0bfb81e40 2024-05-27T07:27:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.389+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d0bfc1e9e0 con 0x55d0bfd00820 2024-05-27T07:27:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.389+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d0bfc22a20 con 0x55d0bfd00410 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.390+0000 7fc0e1800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfcffed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.390+0000 7fc0e1800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfcffed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44044/0 (socket says 172.21.15.43:44044) 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.390+0000 7fc0e1800700 1 -- 172.21.15.43:0/3375816387 learned_addr learned my addr 172.21.15.43:0/3375816387 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.390+0000 7fc0d9600700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc0c40b2a70 0x7fc0c40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.390+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.390+0000 7fc0d9600700 1 -- 172.21.15.43:0/2851868317 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7fc0c80e2940 con 0x7fc0dc108d20 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.390+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfd00820 0x55d0bfd0d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.390+0000 7fc0e1800700 1 -- 172.21.15.43:0/3375816387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 msgr2=0x55d0bfd0aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.391+0000 7fc0e1800700 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.391+0000 7fc0e1800700 1 -- 172.21.15.43:0/3375816387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfd00820 msgr2=0x55d0bfd0d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.391+0000 7fc0e1800700 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfd00820 0x55d0bfd0d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.391+0000 7fc0e1800700 1 -- 172.21.15.43:0/3375816387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0bfbe7420 con 0x55d0bfb81e40 2024-05-27T07:27:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.391+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.391+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfd00820 0x55d0bfd0d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.391+0000 7fc0e1800700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc0c40b2a70 0x7fc0c40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.391+0000 7fc0e1800700 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfcffed0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009d20 tx=0x7fc0d40077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7f76cfb4a6e537 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.392+0000 7fc0ce000700 1 -- 172.21.15.43:0/3375816387 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4021630 con 0x55d0bfb81e40 2024-05-27T07:27:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.392+0000 7fc0ce000700 1 -- 172.21.15.43:0/3375816387 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0d40145f0 con 0x55d0bfb81e40 2024-05-27T07:27:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.392+0000 7fc0ce000700 1 -- 172.21.15.43:0/3375816387 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d401fa30 con 0x55d0bfb81e40 2024-05-27T07:27:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.392+0000 7fc0e1800700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc0c40b2a70 0x7fc0c40b4f80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4007fd0 tx=0x7fc0d401f000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.393+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/3375816387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 msgr2=0x55d0bfcffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.393+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfcffed0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009d20 tx=0x7fc0d40077e0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.393+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/3375816387 shutdown_connections 2024-05-27T07:27:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.393+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfd00820 0x55d0bfd0d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.393+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfcffed0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.393+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/3375816387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.393+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/3375816387 >> 172.21.15.43:0/3375816387 conn(0x55d0bfbe1fc0 msgr2=0x55d0bfcfbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.393+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/3375816387 shutdown_connections 2024-05-27T07:27:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.393+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/3375816387 wait complete. 2024-05-27T07:27:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.394+0000 7fc0f23f3b80 1 -- start start 2024-05-27T07:27:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.394+0000 7fc0f23f3b80 1 Processor -- start 2024-05-27T07:27:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.395+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfb81e40 0x55d0bfd0a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.395+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.395+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00820 0x55d0bfd0b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.395+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d0bfb24fa0 con 0x55d0bfd00410 2024-05-27T07:27:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.395+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d0bfbbf760 con 0x55d0bfb81e40 2024-05-27T07:27:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.395+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d0bfd0d990 con 0x55d0bfd00820 2024-05-27T07:27:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.395+0000 7fc0f23f3b80 10 client.0 _ll_register_callbacks cb 0x55d0bfbc8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.395+0000 7fc0dbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.396+0000 7fc0dbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44054/0 (socket says 172.21.15.43:44054) 2024-05-27T07:27:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.396+0000 7fc0dbe00700 1 -- 172.21.15.43:0/985156922 learned_addr learned my addr 172.21.15.43:0/985156922 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.396+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfb81e40 0x55d0bfd0a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.396+0000 7fc0e1800700 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00820 0x55d0bfd0b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.396+0000 7fc0dbe00700 1 -- 172.21.15.43:0/985156922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00820 msgr2=0x55d0bfd0b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.396+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00820 0x55d0bfd0b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.396+0000 7fc0dbe00700 1 -- 172.21.15.43:0/985156922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfb81e40 msgr2=0x55d0bfd0a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.396+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfb81e40 0x55d0bfd0a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.396+0000 7fc0dbe00700 1 -- 172.21.15.43:0/985156922 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0bfb6e8d0 con 0x55d0bfd00410 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.397+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ab40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80a94f0 tx=0x7fc0c801f5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e859e01427a8c240 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.397+0000 7fc0c2a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.397+0000 7fc0c2a00700 1 -- 172.21.15.43:0/985156922 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80a9ef0 con 0x55d0bfd00410 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.397+0000 7fc0c2a00700 1 -- 172.21.15.43:0/985156922 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7fc0c80b21c0 con 0x55d0bfd00410 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.397+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/985156922 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d0bfc30d90 con 0x55d0bfd00410 2024-05-27T07:27:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.397+0000 7fc0c2a00700 1 -- 172.21.15.43:0/985156922 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fc0c8007df0 con 0x55d0bfd00410 2024-05-27T07:27:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.397+0000 7fc0f23f3b80 10 client.17349 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.397+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/985156922 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x55d0bfbef060 con 0x55d0bfd00410 2024-05-27T07:27:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.398+0000 7fc0c2a00700 1 -- 172.21.15.43:0/985156922 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55d0bfc30d90 con 0x55d0bfd00410 2024-05-27T07:27:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.398+0000 7fc0c2a00700 1 -- 172.21.15.43:0/985156922 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7fc0c8020030 con 0x55d0bfd00410 2024-05-27T07:27:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.398+0000 7fc0f23f3b80 10 client.17349 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.398+0000 7fc0f23f3b80 10 client.17349 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:27:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.398+0000 7fc0f23f3b80 20 client.17349 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.398+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d0bfd1acd0 0x55d0bfd1d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.398+0000 7fc0f23f3b80 4 client.17349 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.398+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/985156922 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0bfc2a570 con 0x55d0bfd1acd0 2024-05-27T07:27:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.399+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d0bfd1acd0 0x55d0bfd1d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.400+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d0bfd1acd0 0x55d0bfd1d120 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5175b7f8c961c6c server_cookie=24b93c8967864391 in_seq=0 out_seq=0 2024-05-27T07:27:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.400+0000 7fc0c2a00700 10 client.17349 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:27:00.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.403+0000 7fc0c2a00700 1 -- 172.21.15.43:0/985156922 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d0bfc2a570 con 0x55d0bfd1acd0 2024-05-27T07:27:00.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.403+0000 7fc0c2a00700 10 client.17349 handle_command_reply: tid=0 2024-05-27T07:27:00.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.403+0000 7fc0f23f3b80 1 client.17349 shutdown 2024-05-27T07:27:00.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.403+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/985156922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 msgr2=0x55d0bfd0ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:00.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.403+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ab40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80a94f0 tx=0x7fc0c801f5f0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.404+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/985156922 shutdown_connections 2024-05-27T07:27:00.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.404+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d0bfd1acd0 0x55d0bfd1d120 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.404+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00820 0x55d0bfd0b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.404+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfb81e40 0x55d0bfd0a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.404+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/985156922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ab40 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.404+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/985156922 >> 172.21.15.43:0/985156922 conn(0x55d0bfbe1fc0 msgr2=0x55d0bfcfaff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:00.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.404+0000 7fc0c2a00700 0 client.17349 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:27:00.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.404+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/985156922 shutdown_connections 2024-05-27T07:27:00.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.404+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/985156922 wait complete. 2024-05-27T07:27:00.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.404+0000 7fc0f23f3b80 20 client.17349 trim_cache size 0 max 16384 2024-05-27T07:27:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.423+0000 7fc0f23f3b80 1 Processor -- start 2024-05-27T07:27:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.423+0000 7fc0f23f3b80 1 -- start start 2024-05-27T07:27:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.423+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfbe1fc0 0x55d0bfd047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.424+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfb81e40 0x55d0bfd16a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.424+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.424+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d0bfcfbef0 con 0x55d0bfd00410 2024-05-27T07:27:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.424+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc1b3270 con 0x55d0bfb81e40 2024-05-27T07:27:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.424+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc1b0f40 con 0x55d0bfbe1fc0 2024-05-27T07:27:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.424+0000 7fc0e1800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.424+0000 7fc0e1800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44066/0 (socket says 172.21.15.43:44066) 2024-05-27T07:27:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.424+0000 7fc0e1800700 1 -- 172.21.15.43:0/4041570044 learned_addr learned my addr 172.21.15.43:0/4041570044 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.424+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/4041570044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfbe1fc0 0x55d0bfd047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.425+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/4041570044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfb81e40 0x55d0bfd16a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.425+0000 7fc0e1800700 1 -- 172.21.15.43:0/4041570044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfbe1fc0 msgr2=0x55d0bfd047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.425+0000 7fc0e1800700 1 --2- 172.21.15.43:0/4041570044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfbe1fc0 0x55d0bfd047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.425+0000 7fc0e1800700 1 -- 172.21.15.43:0/4041570044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfb81e40 msgr2=0x55d0bfd16a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.425+0000 7fc0e1800700 1 --2- 172.21.15.43:0/4041570044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfb81e40 0x55d0bfd16a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.425+0000 7fc0e1800700 1 -- 172.21.15.43:0/4041570044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0bfcfd350 con 0x55d0bfd00410 2024-05-27T07:27:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.425+0000 7fc0e1800700 1 --2- 172.21.15.43:0/4041570044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aaf0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4007be0 tx=0x7fc0d4024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632214121820b122 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.425+0000 7fc0cd600700 1 -- 172.21.15.43:0/4041570044 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d402f070 con 0x55d0bfd00410 2024-05-27T07:27:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.426+0000 7fc0cd600700 1 -- 172.21.15.43:0/4041570044 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0d401fe50 con 0x55d0bfd00410 2024-05-27T07:27:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.426+0000 7fc0cd600700 1 -- 172.21.15.43:0/4041570044 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d402bae0 con 0x55d0bfd00410 2024-05-27T07:27:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.427+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4041570044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 msgr2=0x55d0bfd0aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.427+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/4041570044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aaf0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4007be0 tx=0x7fc0d4024000 comp rx=0 tx=0).stop 2024-05-27T07:27:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.427+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4041570044 shutdown_connections 2024-05-27T07:27:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.427+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/4041570044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfd00410 0x55d0bfd0aaf0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.427+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/4041570044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfbe1fc0 0x55d0bfd047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.427+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/4041570044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfb81e40 0x55d0bfd16a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.427+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4041570044 >> 172.21.15.43:0/4041570044 conn(0x55d0bfcf4f60 msgr2=0x55d0bfd0a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.427+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4041570044 shutdown_connections 2024-05-27T07:27:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.427+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4041570044 wait complete. 2024-05-27T07:27:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.427+0000 7fc0f23f3b80 1 -- start start 2024-05-27T07:27:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.428+0000 7fc0f23f3b80 1 Processor -- start 2024-05-27T07:27:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.428+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfd13370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.428+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfbe1fc0 0x55d0bfd0e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.428+0000 7fc0f23f3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc1b1610 con 0x55d0bfb81e40 2024-05-27T07:27:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc1b2ba0 con 0x55d0bfbe1fc0 2024-05-27T07:27:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0f23f3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc1b1d40 con 0x55d0bfd00410 2024-05-27T07:27:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0f23f3b80 10 client.0 _ll_register_callbacks cb 0x55d0bfbc8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0e0e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfd13370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0e0e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfd13370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44078/0 (socket says 172.21.15.43:44078) 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0e0e00700 1 -- 172.21.15.43:0/4254545993 learned_addr learned my addr 172.21.15.43:0/4254545993 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0e1800700 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfbe1fc0 0x55d0bfd0e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0e0e00700 1 -- 172.21.15.43:0/4254545993 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 msgr2=0x55d0bfd0ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0e0e00700 1 -- 172.21.15.43:0/4254545993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfbe1fc0 msgr2=0x55d0bfd0e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfbe1fc0 0x55d0bfd0e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.429+0000 7fc0e0e00700 1 -- 172.21.15.43:0/4254545993 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0bfd0f420 con 0x55d0bfb81e40 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.430+0000 7fc0e1800700 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.430+0000 7fc0e0e00700 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfd13370 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0000c00 tx=0x7fc0d00095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff1d9d71e63b838 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.430+0000 7fc0c2a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.430+0000 7fc0c2a00700 1 -- 172.21.15.43:0/4254545993 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0033060 con 0x55d0bfb81e40 2024-05-27T07:27:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.430+0000 7fc0c2a00700 1 -- 172.21.15.43:0/4254545993 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9496+0+0 (secure 0 0 0) 0x7fc0d000b730 con 0x55d0bfb81e40 2024-05-27T07:27:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.430+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4254545993 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0d401fe50 con 0x55d0bfb81e40 2024-05-27T07:27:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0c2a00700 1 -- 172.21.15.43:0/4254545993 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fc0d003a050 con 0x55d0bfb81e40 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0f23f3b80 10 client.17355 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4254545993 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x55d0bfd85760 con 0x55d0bfb81e40 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0c2a00700 1 -- 172.21.15.43:0/4254545993 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0d401fe50 con 0x55d0bfb81e40 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0c2a00700 1 -- 172.21.15.43:0/4254545993 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7fc0d0003c60 con 0x55d0bfb81e40 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0f23f3b80 10 client.17355 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0f23f3b80 10 client.17355 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0f23f3b80 20 client.17355 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d0bfd00820 0x55d0bfd87ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0f23f3b80 4 client.17355 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.431+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4254545993 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0bfaa1250 con 0x55d0bfd00820 2024-05-27T07:27:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.432+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d0bfd00820 0x55d0bfd87ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.433+0000 7fc0dbe00700 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d0bfd00820 0x55d0bfd87ec0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8c85e548230a7a7f server_cookie=3816a8cbceedebcd in_seq=0 out_seq=0 2024-05-27T07:27:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.433+0000 7fc0c2a00700 10 client.17355 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:27:00.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.434+0000 7fc0c2a00700 1 -- 172.21.15.43:0/4254545993 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0bfaa1250 con 0x55d0bfd00820 2024-05-27T07:27:00.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.434+0000 7fc0c2a00700 10 client.17355 handle_command_reply: tid=0 2024-05-27T07:27:00.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.434+0000 7fc0f23f3b80 1 client.17355 shutdown 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4254545993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 msgr2=0x55d0bfd13370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfd13370 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0000c00 tx=0x7fc0d00095f0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4254545993 shutdown_connections 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55d0bfd00820 0x55d0bfd87ec0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d0bfd00410 0x55d0bfd0ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d0bfb81e40 0x55d0bfd13370 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 1 --2- 172.21.15.43:0/4254545993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0bfbe1fc0 0x55d0bfd0e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4254545993 >> 172.21.15.43:0/4254545993 conn(0x55d0bfcf4f60 msgr2=0x55d0bfd057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0c2a00700 0 client.17355 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4254545993 shutdown_connections 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 1 -- 172.21.15.43:0/4254545993 wait complete. 2024-05-27T07:27:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.435+0000 7fc0f23f3b80 20 client.17355 trim_cache size 0 max 16384 2024-05-27T07:27:00.426 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:27:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.436+0000 7fc0e2200700 1 -- 172.21.15.43:0/2851868317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc0c40b2a70 msgr2=0x7fc0c40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.436+0000 7fc0e2200700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc0c40b2a70 0x7fc0c40b4f80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4007fd0 tx=0x7fc0d401f000 comp rx=0 tx=0).stop 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.436+0000 7fc0e2200700 1 -- 172.21.15.43:0/2851868317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0dc108d20 msgr2=0x7fc0dc19d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.436+0000 7fc0e2200700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0dc108d20 0x7fc0dc19d330 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fc0c800c9e0 tx=0x7fc0c800cda0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.436+0000 7fc0e2200700 1 -- 172.21.15.43:0/2851868317 shutdown_connections 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.437+0000 7fc0e2200700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc0c40b2a70 0x7fc0c40b4f80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.437+0000 7fc0e2200700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0dc109670 0x7fc0dc1a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.437+0000 7fc0e2200700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0dc1043f0 0x7fc0dc19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.437+0000 7fc0e2200700 1 --2- 172.21.15.43:0/2851868317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0dc108d20 0x7fc0dc19d330 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.437+0000 7fc0e2200700 1 -- 172.21.15.43:0/2851868317 >> 172.21.15.43:0/2851868317 conn(0x7fc0dc079d00 msgr2=0x7fc0dc110cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.437+0000 7fc0e2200700 1 -- 172.21.15.43:0/2851868317 shutdown_connections 2024-05-27T07:27:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:00.437+0000 7fc0e2200700 1 -- 172.21.15.43:0/2851868317 wait complete. 2024-05-27T07:27:01.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 73605 0.58 MB/sec execute 545 sec latency 53.618 ms 2024-05-27T07:27:01.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:27:01.446 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:01 smithi043 ceph-mon[31531]: pgmap v440: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 362 KiB/s wr, 40 op/s 2024-05-27T07:27:01.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:01 smithi179 ceph-mon[35888]: pgmap v440: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 362 KiB/s wr, 40 op/s 2024-05-27T07:27:01.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:01 smithi132 ceph-mon[35283]: pgmap v440: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 362 KiB/s wr, 40 op/s 2024-05-27T07:27:01.868 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:02.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 73891 0.58 MB/sec execute 546 sec latency 56.525 ms 2024-05-27T07:27:03.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74014 0.58 MB/sec execute 547 sec latency 52.080 ms 2024-05-27T07:27:03.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:03 smithi043 ceph-mon[31531]: pgmap v441: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 613 KiB/s wr, 56 op/s 2024-05-27T07:27:03.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:03 smithi043 ceph-mon[31531]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:27:03.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:03 smithi179 ceph-mon[35888]: pgmap v441: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 613 KiB/s wr, 56 op/s 2024-05-27T07:27:03.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:03 smithi179 ceph-mon[35888]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:27:03.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:03 smithi132 ceph-mon[35283]: pgmap v441: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 613 KiB/s wr, 56 op/s 2024-05-27T07:27:03.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:03 smithi132 ceph-mon[35283]: osdmap e134: 12 total, 12 up, 12 in 2024-05-27T07:27:04.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.256+0000 7f6bdfc00700 1 -- 172.21.15.43:0/918581241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bd810b2d0 msgr2=0x7f6bd8117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:04.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.256+0000 7f6bdfc00700 1 --2- 172.21.15.43:0/918581241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bd810b2d0 0x7f6bd8117800 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000b0a0 tx=0x7f6bd000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.256+0000 7f6bdfc00700 1 -- 172.21.15.43:0/918581241 shutdown_connections 2024-05-27T07:27:04.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.256+0000 7f6bdfc00700 1 --2- 172.21.15.43:0/918581241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bd810b2d0 0x7f6bd8117800 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.256+0000 7f6bdfc00700 1 --2- 172.21.15.43:0/918581241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bd8110960 0x7f6bd8110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.256+0000 7f6bdfc00700 1 --2- 172.21.15.43:0/918581241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6bd810a980 0x7f6bd810ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.256+0000 7f6bdfc00700 1 -- 172.21.15.43:0/918581241 >> 172.21.15.43:0/918581241 conn(0x7f6bd8077c70 msgr2=0x7f6bd80780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:04.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.256+0000 7f6bdfc00700 1 -- 172.21.15.43:0/918581241 shutdown_connections 2024-05-27T07:27:04.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.256+0000 7f6bdfc00700 1 -- 172.21.15.43:0/918581241 wait complete. 2024-05-27T07:27:04.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.258+0000 7f6bdfc00700 1 Processor -- start 2024-05-27T07:27:04.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.258+0000 7f6bdfc00700 1 -- start start 2024-05-27T07:27:04.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74276 0.58 MB/sec execute 548 sec latency 58.982 ms 2024-05-27T07:27:04.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.259+0000 7f6bdfc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bd810a980 0x7f6bd81a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.259+0000 7f6bdfc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bd810b2d0 0x7f6bd81a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.259+0000 7f6bdfc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6bd8110960 0x7f6bd81a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.259+0000 7f6bdfc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd811a960 con 0x7f6bd810b2d0 2024-05-27T07:27:04.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.259+0000 7f6bdfc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd811a7e0 con 0x7f6bd8110960 2024-05-27T07:27:04.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.259+0000 7f6bdfc00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd811aae0 con 0x7f6bd810a980 2024-05-27T07:27:04.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.259+0000 7f6bd7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bd810b2d0 0x7f6bd81a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.260+0000 7f6bd7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bd810b2d0 0x7f6bd81a18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44098/0 (socket says 172.21.15.43:44098) 2024-05-27T07:27:04.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.260+0000 7f6bd7e00700 1 -- 172.21.15.43:0/25909920 learned_addr learned my addr 172.21.15.43:0/25909920 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:04.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.260+0000 7f6bdd200700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bd810a980 0x7f6bd81a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.260+0000 7f6bddc00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6bd8110960 0x7f6bd81a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.260+0000 7f6bd7e00700 1 -- 172.21.15.43:0/25909920 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bd810a980 msgr2=0x7f6bd81a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.260+0000 7f6bd7e00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bd810a980 0x7f6bd81a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.260+0000 7f6bd7e00700 1 -- 172.21.15.43:0/25909920 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6bd8110960 msgr2=0x7f6bd81a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.260+0000 7f6bd7e00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6bd8110960 0x7f6bd81a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.260+0000 7f6bd7e00700 1 -- 172.21.15.43:0/25909920 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd81a6130 con 0x7f6bd810b2d0 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.260+0000 7f6bddc00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6bd8110960 0x7f6bd81a5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.261+0000 7f6bd7e00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bd810b2d0 0x7f6bd81a18b0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400c9e0 tx=0x7f6bc400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.261+0000 7f6bd5600700 1 -- 172.21.15.43:0/25909920 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4007aa0 con 0x7f6bd810b2d0 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.261+0000 7f6bd5600700 1 -- 172.21.15.43:0/25909920 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6bc4007c40 con 0x7f6bd810b2d0 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.261+0000 7f6bdfc00700 1 -- 172.21.15.43:0/25909920 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd81a6420 con 0x7f6bd810b2d0 2024-05-27T07:27:04.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.261+0000 7f6bd5600700 1 -- 172.21.15.43:0/25909920 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4018660 con 0x7f6bd810b2d0 2024-05-27T07:27:04.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.261+0000 7f6bdfc00700 1 -- 172.21.15.43:0/25909920 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd81addf0 con 0x7f6bd810b2d0 2024-05-27T07:27:04.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.266+0000 7f6bd5600700 1 -- 172.21.15.43:0/25909920 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6bc4004830 con 0x7f6bd810b2d0 2024-05-27T07:27:04.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.267+0000 7f6bee6bcb80 1 Processor -- start 2024-05-27T07:27:04.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.268+0000 7f6bee6bcb80 1 -- start start 2024-05-27T07:27:04.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.268+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff1569330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.268+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff1569870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.268+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff1569e70 0x55eff1576110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.268+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55eff1497e40 con 0x55eff14d69f0 2024-05-27T07:27:04.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.268+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55eff14879e0 con 0x55eff1569e70 2024-05-27T07:27:04.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.268+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55eff148ba20 con 0x55eff13eae40 2024-05-27T07:27:04.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.269+0000 7f6bddc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff1569330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.269+0000 7f6bd5600700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6bc00b2af0 0x7f6bc00b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.269+0000 7f6bddc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff1569330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44114/0 (socket says 172.21.15.43:44114) 2024-05-27T07:27:04.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.269+0000 7f6bddc00700 1 -- 172.21.15.43:0/3422703966 learned_addr learned my addr 172.21.15.43:0/3422703966 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:04.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.269+0000 7f6bd5600700 1 -- 172.21.15.43:0/25909920 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9584+0+0 (secure 0 0 0) 0x7f6bc40e28d0 con 0x7f6bd810b2d0 2024-05-27T07:27:04.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.269+0000 7f6bdd200700 1 --2- 172.21.15.43:0/3422703966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff1569870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.269+0000 7f6bddc00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6bc00b2af0 0x7f6bc00b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.269+0000 7f6bd7e00700 1 --2- 172.21.15.43:0/3422703966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff1569e70 0x55eff1576110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.270+0000 7f6bddc00700 1 -- 172.21.15.43:0/3422703966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 msgr2=0x55eff1569870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:04.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.270+0000 7f6bddc00700 1 --2- 172.21.15.43:0/3422703966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff1569870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.270+0000 7f6bddc00700 1 -- 172.21.15.43:0/3422703966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff1569e70 msgr2=0x55eff1576110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:04.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.270+0000 7f6bddc00700 1 --2- 172.21.15.43:0/3422703966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff1569e70 0x55eff1576110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.270+0000 7f6bddc00700 1 -- 172.21.15.43:0/3422703966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eff1450420 con 0x55eff14d69f0 2024-05-27T07:27:04.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.270+0000 7f6bddc00700 1 --2- 172.21.15.43:0/3422703966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff1569330 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0004f00 tx=0x7f6bd0004f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b9201750f1092a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:04.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.270+0000 7f6bddc00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6bc00b2af0 0x7f6bc00b5000 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0004fa0 tx=0x7f6bd0013000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:04.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.271+0000 7f6bca000700 1 -- 172.21.15.43:0/3422703966 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd002d070 con 0x55eff14d69f0 2024-05-27T07:27:04.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.271+0000 7f6bca000700 1 -- 172.21.15.43:0/3422703966 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6bd001f8d0 con 0x55eff14d69f0 2024-05-27T07:27:04.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.271+0000 7f6bca000700 1 -- 172.21.15.43:0/3422703966 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0029670 con 0x55eff14d69f0 2024-05-27T07:27:04.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.272+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/3422703966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 msgr2=0x55eff1569330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:04.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.272+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/3422703966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff1569330 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0004f00 tx=0x7f6bd0004f30 comp rx=0 tx=0).stop 2024-05-27T07:27:04.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.272+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/3422703966 shutdown_connections 2024-05-27T07:27:04.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.272+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/3422703966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff1569e70 0x55eff1576110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.272+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/3422703966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff1569330 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.272+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/3422703966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff1569870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.272+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/3422703966 >> 172.21.15.43:0/3422703966 conn(0x55eff1484390 msgr2=0x55eff15654c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:04.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.272+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/3422703966 shutdown_connections 2024-05-27T07:27:04.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.272+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/3422703966 wait complete. 2024-05-27T07:27:04.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.273+0000 7f6bee6bcb80 1 -- start start 2024-05-27T07:27:04.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.273+0000 7f6bee6bcb80 1 Processor -- start 2024-05-27T07:27:04.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.274+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff13eae40 0x55eff1573670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.274+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff14d69f0 0x55eff1573bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.274+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff15740f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.274+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55eff138dfa0 con 0x55eff1569e70 2024-05-27T07:27:04.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.274+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55eff1428760 con 0x55eff13eae40 2024-05-27T07:27:04.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.274+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55eff1576b20 con 0x55eff14d69f0 2024-05-27T07:27:04.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.274+0000 7f6bee6bcb80 10 client.0 _ll_register_callbacks cb 0x55eff1431c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:04.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.274+0000 7f6bddc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff15740f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bddc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff15740f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44130/0 (socket says 172.21.15.43:44130) 2024-05-27T07:27:04.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bddc00700 1 -- 172.21.15.43:0/4166943324 learned_addr learned my addr 172.21.15.43:0/4166943324 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:04.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bdd200700 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff13eae40 0x55eff1573670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bd7e00700 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff14d69f0 0x55eff1573bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bddc00700 1 -- 172.21.15.43:0/4166943324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff14d69f0 msgr2=0x55eff1573bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:04.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bddc00700 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff14d69f0 0x55eff1573bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bddc00700 1 -- 172.21.15.43:0/4166943324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff13eae40 msgr2=0x55eff1573670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bddc00700 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff13eae40 0x55eff1573670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bddc00700 1 -- 172.21.15.43:0/4166943324 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eff13d78d0 con 0x55eff1569e70 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bddc00700 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff15740f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f6bd00062a0 tx=0x7f6bd001fdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a741c0a97e9b2640 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.275+0000 7f6bbea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.276+0000 7f6bbea00700 1 -- 172.21.15.43:0/4166943324 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd002d070 con 0x55eff1569e70 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.276+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4166943324 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bd001f8d0 con 0x55eff1569e70 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.276+0000 7f6bbea00700 1 -- 172.21.15.43:0/4166943324 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(134..134 src has 1..134) ==== 9584+0+0 (secure 0 0 0) 0x7f6bd0026040 con 0x55eff1569e70 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.276+0000 7f6bbea00700 1 -- 172.21.15.43:0/4166943324 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f6bd003f050 con 0x55eff1569e70 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.276+0000 7f6bee6bcb80 10 client.17367 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.276+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4166943324 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x55eff1458060 con 0x55eff1569e70 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.277+0000 7f6bbea00700 1 -- 172.21.15.43:0/4166943324 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6bd001f8d0 con 0x55eff1569e70 2024-05-27T07:27:04.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.277+0000 7f6bbea00700 1 -- 172.21.15.43:0/4166943324 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7f6bd0011070 con 0x55eff1569e70 2024-05-27T07:27:04.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.277+0000 7f6bee6bcb80 10 client.17367 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:04.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.277+0000 7f6bee6bcb80 10 client.17367 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:27:04.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.277+0000 7f6bee6bcb80 20 client.17367 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:04.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.277+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55eff1583bb0 0x55eff1586040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.277+0000 7f6bee6bcb80 4 client.17367 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:04.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.277+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4166943324 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eff1499d90 con 0x55eff1583bb0 2024-05-27T07:27:04.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.278+0000 7f6bdd200700 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55eff1583bb0 0x55eff1586040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.279+0000 7f6bdd200700 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55eff1583bb0 0x55eff1586040 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c9a3a56d0701333 server_cookie=5748360b2e88e1f4 in_seq=0 out_seq=0 2024-05-27T07:27:04.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.279+0000 7f6bbea00700 10 client.17367 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:27:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.281+0000 7f6bbea00700 1 -- 172.21.15.43:0/4166943324 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55eff1499d90 con 0x55eff1583bb0 2024-05-27T07:27:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.281+0000 7f6bbea00700 10 client.17367 handle_command_reply: tid=0 2024-05-27T07:27:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.281+0000 7f6bee6bcb80 1 client.17367 shutdown 2024-05-27T07:27:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.281+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4166943324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 msgr2=0x55eff15740f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.281+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff15740f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f6bd00062a0 tx=0x7f6bd001fdb0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.282+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4166943324 shutdown_connections 2024-05-27T07:27:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.282+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55eff1583bb0 0x55eff1586040 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.282+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff15740f0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.282+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff13eae40 0x55eff1573670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.282+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4166943324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff14d69f0 0x55eff1573bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.282+0000 7f6bbea00700 0 client.17367 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:27:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.282+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4166943324 >> 172.21.15.43:0/4166943324 conn(0x55eff1484390 msgr2=0x55eff1569350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.282+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4166943324 shutdown_connections 2024-05-27T07:27:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.282+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4166943324 wait complete. 2024-05-27T07:27:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.282+0000 7f6bee6bcb80 20 client.17367 trim_cache size 0 max 16384 2024-05-27T07:27:04.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.301+0000 7f6bee6bcb80 1 Processor -- start 2024-05-27T07:27:04.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.301+0000 7f6bee6bcb80 1 -- start start 2024-05-27T07:27:04.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.301+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff1569850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.301+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff14d69f0 0x55eff157fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff158b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55eff15730f0 con 0x55eff1569e70 2024-05-27T07:27:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb800fbc0 con 0x55eff14d69f0 2024-05-27T07:27:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb800d890 con 0x55eff13eae40 2024-05-27T07:27:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bddc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff158b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bddc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff158b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44134/0 (socket says 172.21.15.43:44134) 2024-05-27T07:27:04.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bddc00700 1 -- 172.21.15.43:0/3886467759 learned_addr learned my addr 172.21.15.43:0/3886467759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bdd200700 1 --2- 172.21.15.43:0/3886467759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff1569850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bddc00700 1 -- 172.21.15.43:0/3886467759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 msgr2=0x55eff1569850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bddc00700 1 --2- 172.21.15.43:0/3886467759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff1569850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bd7e00700 1 --2- 172.21.15.43:0/3886467759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff14d69f0 0x55eff157fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bddc00700 1 -- 172.21.15.43:0/3886467759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff14d69f0 msgr2=0x55eff157fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bddc00700 1 --2- 172.21.15.43:0/3886467759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff14d69f0 0x55eff157fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.302+0000 7f6bddc00700 1 -- 172.21.15.43:0/3886467759 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eff1574680 con 0x55eff1569e70 2024-05-27T07:27:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.303+0000 7f6bd7e00700 1 --2- 172.21.15.43:0/3886467759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff14d69f0 0x55eff157fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:04.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.303+0000 7f6bddc00700 1 --2- 172.21.15.43:0/3886467759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff158b800 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f6bd00317b0 tx=0x7f6bd00099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c229da838aed8a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:04.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.303+0000 7f6bc9600700 1 -- 172.21.15.43:0/3886467759 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0048070 con 0x55eff1569e70 2024-05-27T07:27:04.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.303+0000 7f6bc9600700 1 -- 172.21.15.43:0/3886467759 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6bd0036070 con 0x55eff1569e70 2024-05-27T07:27:04.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.303+0000 7f6bc9600700 1 -- 172.21.15.43:0/3886467759 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd002d070 con 0x55eff1569e70 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.304+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/3886467759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 msgr2=0x55eff158b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.304+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/3886467759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff158b800 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f6bd00317b0 tx=0x7f6bd00099e0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.304+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/3886467759 shutdown_connections 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.304+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/3886467759 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff1569e70 0x55eff158b800 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.304+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/3886467759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff1569850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.304+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/3886467759 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff14d69f0 0x55eff157fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.304+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/3886467759 >> 172.21.15.43:0/3886467759 conn(0x55eff1484390 msgr2=0x55eff1569140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.304+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/3886467759 shutdown_connections 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.305+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/3886467759 wait complete. 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.305+0000 7f6bee6bcb80 1 -- start start 2024-05-27T07:27:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.305+0000 7f6bee6bcb80 1 Processor -- start 2024-05-27T07:27:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff156d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff156d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bee6bcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff1569e70 0x55eff156db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb800df60 con 0x55eff14d69f0 2024-05-27T07:27:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb800f4f0 con 0x55eff1569e70 2024-05-27T07:27:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bee6bcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb800e690 con 0x55eff13eae40 2024-05-27T07:27:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bee6bcb80 10 client.0 _ll_register_callbacks cb 0x55eff1431c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bd7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff156d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bd7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff156d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44144/0 (socket says 172.21.15.43:44144) 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bd7e00700 1 -- 172.21.15.43:0/4277264242 learned_addr learned my addr 172.21.15.43:0/4277264242 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.306+0000 7f6bdd200700 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff156d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bddc00700 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff1569e70 0x55eff156db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bd7e00700 1 -- 172.21.15.43:0/4277264242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 msgr2=0x55eff156d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bd7e00700 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff156d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bd7e00700 1 -- 172.21.15.43:0/4277264242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff1569e70 msgr2=0x55eff156db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bd7e00700 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff1569e70 0x55eff156db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bd7e00700 1 -- 172.21.15.43:0/4277264242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eff1561240 con 0x55eff14d69f0 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bd7e00700 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff156d630 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40a69c0 tx=0x7f6bc4076080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7f7a22425c692a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bbea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bbea00700 1 -- 172.21.15.43:0/4277264242 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc40a9260 con 0x55eff14d69f0 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bbea00700 1 -- 172.21.15.43:0/4277264242 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(134..134 src has 1..134) ==== 9584+0+0 (secure 0 0 0) 0x7f6bc40a9bc0 con 0x55eff14d69f0 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.307+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4277264242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bd0036070 con 0x55eff14d69f0 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bbea00700 1 -- 172.21.15.43:0/4277264242 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f6bc40a9e50 con 0x55eff14d69f0 2024-05-27T07:27:04.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bee6bcb80 10 client.17373 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:04.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4277264242 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x55eff157bf50 con 0x55eff14d69f0 2024-05-27T07:27:04.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bbea00700 1 -- 172.21.15.43:0/4277264242 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6bd0036070 con 0x55eff14d69f0 2024-05-27T07:27:04.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bbea00700 1 -- 172.21.15.43:0/4277264242 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7f6bc406ecb0 con 0x55eff14d69f0 2024-05-27T07:27:04.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bee6bcb80 10 client.17373 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:04.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bee6bcb80 10 client.17373 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:27:04.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bee6bcb80 20 client.17373 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:04.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55eff157c670 0x55eff15f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:04.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bee6bcb80 4 client.17373 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:04.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.308+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4277264242 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eff1493570 con 0x55eff157c670 2024-05-27T07:27:04.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.309+0000 7f6bdd200700 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55eff157c670 0x55eff15f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:04.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.310+0000 7f6bdd200700 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55eff157c670 0x55eff15f06e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=48a1caa8c97a1f4e server_cookie=b620cb10ed672dec in_seq=0 out_seq=0 2024-05-27T07:27:04.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.310+0000 7f6bbea00700 10 client.17373 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.310+0000 7f6bbea00700 1 -- 172.21.15.43:0/4277264242 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eff1493570 con 0x55eff157c670 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.310+0000 7f6bbea00700 10 client.17373 handle_command_reply: tid=0 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.310+0000 7f6bee6bcb80 1 client.17373 shutdown 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4277264242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 msgr2=0x55eff156d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff156d630 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40a69c0 tx=0x7f6bc4076080 comp rx=0 tx=0).stop 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4277264242 shutdown_connections 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55eff157c670 0x55eff15f06e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eff1569e70 0x55eff156db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55eff13eae40 0x55eff156d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 1 --2- 172.21.15.43:0/4277264242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55eff14d69f0 0x55eff156d630 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4277264242 >> 172.21.15.43:0/4277264242 conn(0x55eff1484390 msgr2=0x55eff156aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bbea00700 0 client.17373 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:27:04.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4277264242 shutdown_connections 2024-05-27T07:27:04.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 1 -- 172.21.15.43:0/4277264242 wait complete. 2024-05-27T07:27:04.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.311+0000 7f6bee6bcb80 20 client.17373 trim_cache size 0 max 16384 2024-05-27T07:27:04.303 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:27:04.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.312+0000 7f6bdfc00700 1 -- 172.21.15.43:0/25909920 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6bc00b2af0 msgr2=0x7f6bc00b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:04.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.312+0000 7f6bdfc00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6bc00b2af0 0x7f6bc00b5000 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0004fa0 tx=0x7f6bd0013000 comp rx=0 tx=0).stop 2024-05-27T07:27:04.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.312+0000 7f6bdfc00700 1 -- 172.21.15.43:0/25909920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bd810b2d0 msgr2=0x7f6bd81a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:04.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.312+0000 7f6bdfc00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bd810b2d0 0x7f6bd81a18b0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400c9e0 tx=0x7f6bc400ccf0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.312+0000 7f6bdfc00700 1 -- 172.21.15.43:0/25909920 shutdown_connections 2024-05-27T07:27:04.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.312+0000 7f6bdfc00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6bc00b2af0 0x7f6bc00b5000 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.312+0000 7f6bdfc00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6bd8110960 0x7f6bd81a5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.312+0000 7f6bdfc00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bd810a980 0x7f6bd81a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.312+0000 7f6bdfc00700 1 --2- 172.21.15.43:0/25909920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6bd810b2d0 0x7f6bd81a18b0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:04.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.312+0000 7f6bdfc00700 1 -- 172.21.15.43:0/25909920 >> 172.21.15.43:0/25909920 conn(0x7f6bd8077c70 msgr2=0x7f6bd807b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:04.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.313+0000 7f6bdfc00700 1 -- 172.21.15.43:0/25909920 shutdown_connections 2024-05-27T07:27:04.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:04.313+0000 7f6bdfc00700 1 -- 172.21.15.43:0/25909920 wait complete. 2024-05-27T07:27:05.114 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:27:05.248 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74309 0.58 MB/sec execute 549 sec latency 60.475 ms 2024-05-27T07:27:05.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:05 smithi043 ceph-mon[31531]: pgmap v443: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 KiB/s wr, 38 op/s 2024-05-27T07:27:05.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:05 smithi179 ceph-mon[35888]: pgmap v443: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 KiB/s wr, 38 op/s 2024-05-27T07:27:05.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:05 smithi132 ceph-mon[35283]: pgmap v443: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 KiB/s wr, 38 op/s 2024-05-27T07:27:05.611 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:06.249 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74352 0.58 MB/sec execute 550 sec latency 52.449 ms 2024-05-27T07:27:07.249 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74531 0.59 MB/sec execute 551 sec latency 51.259 ms 2024-05-27T07:27:07.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:07 smithi043 ceph-mon[31531]: pgmap v444: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 661 KiB/s wr, 57 op/s 2024-05-27T07:27:07.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:07 smithi043 ceph-mon[31531]: osdmap e135: 12 total, 12 up, 12 in 2024-05-27T07:27:07.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:07 smithi132 ceph-mon[35283]: pgmap v444: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 661 KiB/s wr, 57 op/s 2024-05-27T07:27:07.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:07 smithi132 ceph-mon[35283]: osdmap e135: 12 total, 12 up, 12 in 2024-05-27T07:27:07.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:07 smithi179 ceph-mon[35888]: pgmap v444: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 661 KiB/s wr, 57 op/s 2024-05-27T07:27:07.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:07 smithi179 ceph-mon[35888]: osdmap e135: 12 total, 12 up, 12 in 2024-05-27T07:27:08.249 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74612 0.59 MB/sec execute 552 sec latency 414.837 ms 2024-05-27T07:27:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.394+0000 7f02a2a00700 1 -- 172.21.15.43:0/3645280227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f029c106570 msgr2=0x7f029c106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.394+0000 7f02a2a00700 1 --2- 172.21.15.43:0/3645280227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f029c106570 0x7f029c106980 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f028400b0a0 tx=0x7f028400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.394+0000 7f02a2a00700 1 -- 172.21.15.43:0/3645280227 shutdown_connections 2024-05-27T07:27:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.394+0000 7f02a2a00700 1 --2- 172.21.15.43:0/3645280227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f029c106ec0 0x7f029c113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.394+0000 7f02a2a00700 1 --2- 172.21.15.43:0/3645280227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f029c10c5b0 0x7f029c10c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.394+0000 7f02a2a00700 1 --2- 172.21.15.43:0/3645280227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f029c106570 0x7f029c106980 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.394+0000 7f02a2a00700 1 -- 172.21.15.43:0/3645280227 >> 172.21.15.43:0/3645280227 conn(0x7f029c102070 msgr2=0x7f029c1044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.395+0000 7f02a2a00700 1 -- 172.21.15.43:0/3645280227 shutdown_connections 2024-05-27T07:27:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.395+0000 7f02a2a00700 1 -- 172.21.15.43:0/3645280227 wait complete. 2024-05-27T07:27:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.396+0000 7f02a2a00700 1 Processor -- start 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.396+0000 7f02a2a00700 1 -- start start 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.396+0000 7f02a2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f029c106570 0x7f029c19d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.396+0000 7f02a2a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f029c106ec0 0x7f029c19d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f02a2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f029c10c5b0 0x7f029c1a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f02a2a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f029c116590 con 0x7f029c106570 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f02a2a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f029c116410 con 0x7f029c10c5b0 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f02a2a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f029c116710 con 0x7f029c106ec0 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f029be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f029c106570 0x7f029c19d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f029be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f029c106570 0x7f029c19d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57292/0 (socket says 172.21.15.43:57292) 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f029be00700 1 -- 172.21.15.43:0/315042079 learned_addr learned my addr 172.21.15.43:0/315042079 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:08.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f0293400700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f029c106ec0 0x7f029c19d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f02a0a00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f029c10c5b0 0x7f029c1a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f029be00700 1 -- 172.21.15.43:0/315042079 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f029c106ec0 msgr2=0x7f029c19d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f029be00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f029c106ec0 0x7f029c19d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f029be00700 1 -- 172.21.15.43:0/315042079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f029c10c5b0 msgr2=0x7f029c1a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f029be00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f029c10c5b0 0x7f029c1a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.397+0000 7f029be00700 1 -- 172.21.15.43:0/315042079 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f029c1a1f10 con 0x7f029c106570 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.398+0000 7f02a0a00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f029c10c5b0 0x7f029c1a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.398+0000 7f029be00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f029c106570 0x7f029c19d040 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f028c00d980 tx=0x7f028c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.398+0000 7f0299600700 1 -- 172.21.15.43:0/315042079 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c017070 con 0x7f029c106570 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.398+0000 7f0299600700 1 -- 172.21.15.43:0/315042079 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f028c010460 con 0x7f029c106570 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.398+0000 7f02a2a00700 1 -- 172.21.15.43:0/315042079 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f029c1a2200 con 0x7f029c106570 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.398+0000 7f0299600700 1 -- 172.21.15.43:0/315042079 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c00de60 con 0x7f029c106570 2024-05-27T07:27:08.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.398+0000 7f02a2a00700 1 -- 172.21.15.43:0/315042079 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f029c1a9b20 con 0x7f029c106570 2024-05-27T07:27:08.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.402+0000 7f0299600700 1 -- 172.21.15.43:0/315042079 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f028c010610 con 0x7f029c106570 2024-05-27T07:27:08.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.403+0000 7f02b155eb80 1 Processor -- start 2024-05-27T07:27:08.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.404+0000 7f02b155eb80 1 -- start start 2024-05-27T07:27:08.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.404+0000 7f0299600700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02800b2a70 0x7f02800b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.404+0000 7f0299600700 1 -- 172.21.15.43:0/315042079 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7f028c0e24a0 con 0x7f029c106570 2024-05-27T07:27:08.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.404+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693ead97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.404+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.404+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693ead41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.404+0000 7f02b155eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55693ea01e40 con 0x55693ead9d30 2024-05-27T07:27:08.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.404+0000 7f02b155eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55693e9f19e0 con 0x55693ead3db0 2024-05-27T07:27:08.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.404+0000 7f02b155eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55693e9f5a20 con 0x55693e954e40 2024-05-27T07:27:08.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.404+0000 7f02a0a00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02800b2a70 0x7f02800b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f0293400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f0293400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57298/0 (socket says 172.21.15.43:57298) 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f0293400700 1 -- 172.21.15.43:0/1606307534 learned_addr learned my addr 172.21.15.43:0/1606307534 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f029be00700 1 --2- 172.21.15.43:0/1606307534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693ead97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f02a0a00700 1 --2- 172.21.15.43:0/1606307534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693ead41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f0293400700 1 -- 172.21.15.43:0/1606307534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 msgr2=0x55693ead97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f0293400700 1 --2- 172.21.15.43:0/1606307534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693ead97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f0293400700 1 -- 172.21.15.43:0/1606307534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 msgr2=0x55693ead41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f0293400700 1 --2- 172.21.15.43:0/1606307534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693ead41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f0293400700 1 -- 172.21.15.43:0/1606307534 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55693e9ba420 con 0x55693ead9d30 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f02a0a00700 1 --2- 172.21.15.43:0/1606307534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693ead41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:08.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.405+0000 7f02a0a00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02800b2a70 0x7f02800b4f80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0294005fd0 tx=0x7f0294009f40 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:08.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.406+0000 7f0293400700 1 --2- 172.21.15.43:0/1606307534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead3870 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f029c075b70 tx=0x7f0284009d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c06f578e9a1a771 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:08.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.406+0000 7f0291600700 1 -- 172.21.15.43:0/1606307534 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284004a50 con 0x55693ead9d30 2024-05-27T07:27:08.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.406+0000 7f0291600700 1 -- 172.21.15.43:0/1606307534 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0284004bf0 con 0x55693ead9d30 2024-05-27T07:27:08.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.406+0000 7f0291600700 1 -- 172.21.15.43:0/1606307534 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028401c6a0 con 0x55693ead9d30 2024-05-27T07:27:08.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.407+0000 7f02b155eb80 1 -- 172.21.15.43:0/1606307534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 msgr2=0x55693ead3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.407+0000 7f02b155eb80 1 --2- 172.21.15.43:0/1606307534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead3870 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f029c075b70 tx=0x7f0284009d90 comp rx=0 tx=0).stop 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.407+0000 7f02b155eb80 1 -- 172.21.15.43:0/1606307534 shutdown_connections 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.407+0000 7f02b155eb80 1 --2- 172.21.15.43:0/1606307534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693ead41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.407+0000 7f02b155eb80 1 --2- 172.21.15.43:0/1606307534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693ead97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.407+0000 7f02b155eb80 1 --2- 172.21.15.43:0/1606307534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead3870 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.407+0000 7f02b155eb80 1 -- 172.21.15.43:0/1606307534 >> 172.21.15.43:0/1606307534 conn(0x55693e9b4fc0 msgr2=0x55693eace8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.407+0000 7f02b155eb80 1 -- 172.21.15.43:0/1606307534 shutdown_connections 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.408+0000 7f02b155eb80 1 -- 172.21.15.43:0/1606307534 wait complete. 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.408+0000 7f02b155eb80 1 -- start start 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.408+0000 7f02b155eb80 1 Processor -- start 2024-05-27T07:27:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.409+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693eadd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.409+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead3db0 0x55693eadd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.409+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead9d30 0x55693eaddec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.409+0000 7f02b155eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55693e8f7fa0 con 0x55693ead3db0 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.409+0000 7f02b155eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55693e992760 con 0x55693ead9d30 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.409+0000 7f02b155eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55693eae07a0 con 0x55693e954e40 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.409+0000 7f02b155eb80 10 client.0 _ll_register_callbacks cb 0x55693e99bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.410+0000 7f0293400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead3db0 0x55693eadd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.410+0000 7f0293400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead3db0 0x55693eadd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57304/0 (socket says 172.21.15.43:57304) 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.410+0000 7f0293400700 1 -- 172.21.15.43:0/554049952 learned_addr learned my addr 172.21.15.43:0/554049952 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.410+0000 7f02a0a00700 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead9d30 0x55693eaddec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.410+0000 7f029be00700 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693eadd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.410+0000 7f0293400700 1 -- 172.21.15.43:0/554049952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 msgr2=0x55693eadd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.410+0000 7f0293400700 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693eadd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.410+0000 7f0293400700 1 -- 172.21.15.43:0/554049952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead9d30 msgr2=0x55693eaddec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.410+0000 7f0293400700 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead9d30 0x55693eaddec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.410+0000 7f0293400700 1 -- 172.21.15.43:0/554049952 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55693e9418d0 con 0x55693ead3db0 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.411+0000 7f0293400700 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead3db0 0x55693eadd980 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f02840062a0 tx=0x7f0284007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30881d7f2e7138ca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.411+0000 7f028a000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.411+0000 7f028a000700 1 -- 172.21.15.43:0/554049952 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028402a070 con 0x55693ead3db0 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.411+0000 7f028a000700 1 -- 172.21.15.43:0/554049952 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7f028401c440 con 0x55693ead3db0 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.411+0000 7f02b155eb80 1 -- 172.21.15.43:0/554049952 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0284004bf0 con 0x55693ead3db0 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.411+0000 7f028a000700 1 -- 172.21.15.43:0/554049952 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f028401c640 con 0x55693ead3db0 2024-05-27T07:27:08.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.411+0000 7f02b155eb80 10 client.17385 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.411+0000 7f02b155eb80 1 -- 172.21.15.43:0/554049952 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x55693e9c2060 con 0x55693ead3db0 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.412+0000 7f028a000700 1 -- 172.21.15.43:0/554049952 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0284004bf0 con 0x55693ead3db0 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.412+0000 7f028a000700 1 -- 172.21.15.43:0/554049952 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7f0284011070 con 0x55693ead3db0 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.412+0000 7f02b155eb80 10 client.17385 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.412+0000 7f02b155eb80 10 client.17385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.412+0000 7f02b155eb80 20 client.17385 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.412+0000 7f02b155eb80 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55693eaed610 0x55693eaefa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.412+0000 7f02b155eb80 4 client.17385 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.412+0000 7f02b155eb80 1 -- 172.21.15.43:0/554049952 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55693ea03d90 con 0x55693eaed610 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.413+0000 7f029be00700 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55693eaed610 0x55693eaefa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.414+0000 7f029be00700 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55693eaed610 0x55693eaefa60 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b6a5a4df8f4b461 server_cookie=c852931ba928f66b in_seq=0 out_seq=0 2024-05-27T07:27:08.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.414+0000 7f028a000700 10 client.17385 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:08.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f028a000700 1 -- 172.21.15.43:0/554049952 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55693ea03d90 con 0x55693eaed610 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f028a000700 10 client.17385 handle_command_reply: tid=0 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 client.17385 shutdown 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 -- 172.21.15.43:0/554049952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead3db0 msgr2=0x55693eadd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead3db0 0x55693eadd980 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f02840062a0 tx=0x7f0284007930 comp rx=0 tx=0).stop 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 -- 172.21.15.43:0/554049952 shutdown_connections 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead9d30 0x55693eaddec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693eadd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55693eaed610 0x55693eaefa60 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 --2- 172.21.15.43:0/554049952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead3db0 0x55693eadd980 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 -- 172.21.15.43:0/554049952 >> 172.21.15.43:0/554049952 conn(0x55693e9b4fc0 msgr2=0x55693eace580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f028a000700 0 client.17385 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 -- 172.21.15.43:0/554049952 shutdown_connections 2024-05-27T07:27:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.416+0000 7f02b155eb80 1 -- 172.21.15.43:0/554049952 wait complete. 2024-05-27T07:27:08.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.417+0000 7f02b155eb80 20 client.17385 trim_cache size 0 max 16384 2024-05-27T07:27:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.433+0000 7f02b155eb80 1 Processor -- start 2024-05-27T07:27:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.434+0000 7f02b155eb80 1 -- start start 2024-05-27T07:27:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.434+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693ead3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.434+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693eadf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.434+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.434+0000 7f02b155eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f029c1b3350 con 0x55693ead9d30 2024-05-27T07:27:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.434+0000 7f02b155eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f029c1b1020 con 0x55693ead3db0 2024-05-27T07:27:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.434+0000 7f02b155eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f029c1b2c80 con 0x55693e954e40 2024-05-27T07:27:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f02a0a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f02a0a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57310/0 (socket says 172.21.15.43:57310) 2024-05-27T07:27:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f02a0a00700 1 -- 172.21.15.43:0/4111632730 learned_addr learned my addr 172.21.15.43:0/4111632730 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f029be00700 1 --2- 172.21.15.43:0/4111632730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693ead3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f0293400700 1 --2- 172.21.15.43:0/4111632730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693eadf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f02a0a00700 1 -- 172.21.15.43:0/4111632730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 msgr2=0x55693ead3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f02a0a00700 1 --2- 172.21.15.43:0/4111632730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693ead3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f02a0a00700 1 -- 172.21.15.43:0/4111632730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 msgr2=0x55693eadf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f02a0a00700 1 --2- 172.21.15.43:0/4111632730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693eadf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f02a0a00700 1 -- 172.21.15.43:0/4111632730 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55693eadc760 con 0x55693ead9d30 2024-05-27T07:27:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.435+0000 7f02a0a00700 1 --2- 172.21.15.43:0/4111632730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead8ca0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f029400ef60 tx=0x7f0294005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e4e703166300a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.436+0000 7f0290c00700 1 -- 172.21.15.43:0/4111632730 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0294010cf0 con 0x55693ead9d30 2024-05-27T07:27:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.436+0000 7f0290c00700 1 -- 172.21.15.43:0/4111632730 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0294028890 con 0x55693ead9d30 2024-05-27T07:27:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.436+0000 7f0290c00700 1 -- 172.21.15.43:0/4111632730 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02940206f0 con 0x55693ead9d30 2024-05-27T07:27:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.437+0000 7f02b155eb80 1 -- 172.21.15.43:0/4111632730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 msgr2=0x55693ead8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.437+0000 7f02b155eb80 1 --2- 172.21.15.43:0/4111632730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead8ca0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f029400ef60 tx=0x7f0294005ee0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.437+0000 7f02b155eb80 1 -- 172.21.15.43:0/4111632730 shutdown_connections 2024-05-27T07:27:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.437+0000 7f02b155eb80 1 --2- 172.21.15.43:0/4111632730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693ead9d30 0x55693ead8ca0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.437+0000 7f02b155eb80 1 --2- 172.21.15.43:0/4111632730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693e954e40 0x55693ead3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.437+0000 7f02b155eb80 1 --2- 172.21.15.43:0/4111632730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693eadf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.437+0000 7f02b155eb80 1 -- 172.21.15.43:0/4111632730 >> 172.21.15.43:0/4111632730 conn(0x55693e9b4fc0 msgr2=0x55693ead8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.437+0000 7f02b155eb80 1 -- 172.21.15.43:0/4111632730 shutdown_connections 2024-05-27T07:27:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.437+0000 7f02b155eb80 1 -- 172.21.15.43:0/4111632730 wait complete. 2024-05-27T07:27:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.437+0000 7f02b155eb80 1 -- start start 2024-05-27T07:27:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.438+0000 7f02b155eb80 1 Processor -- start 2024-05-27T07:27:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.438+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693e954e40 0x55693eae59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.438+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693eae5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.438+0000 7f02b155eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693ead9d30 0x55693eaf41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.438+0000 7f02b155eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f029c1b1e20 con 0x55693e954e40 2024-05-27T07:27:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.438+0000 7f02b155eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f029c1b16f0 con 0x55693ead3db0 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.438+0000 7f02b155eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f029c1b2550 con 0x55693ead9d30 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.438+0000 7f02b155eb80 10 client.0 _ll_register_callbacks cb 0x55693e99bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f029be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693e954e40 0x55693eae59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f029be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693e954e40 0x55693eae59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57312/0 (socket says 172.21.15.43:57312) 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f029be00700 1 -- 172.21.15.43:0/1309125166 learned_addr learned my addr 172.21.15.43:0/1309125166 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f0293400700 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693eae5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f02a0a00700 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693ead9d30 0x55693eaf41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f029be00700 1 -- 172.21.15.43:0/1309125166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693ead9d30 msgr2=0x55693eaf41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f029be00700 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693ead9d30 0x55693eaf41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f029be00700 1 -- 172.21.15.43:0/1309125166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 msgr2=0x55693eae5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f029be00700 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693eae5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f029be00700 1 -- 172.21.15.43:0/1309125166 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55693eaf4710 con 0x55693e954e40 2024-05-27T07:27:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f029be00700 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693e954e40 0x55693eae59f0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f028c0a65c0 tx=0x7f028c010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5144959997dbf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.439+0000 7f028a000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f028a000700 1 -- 172.21.15.43:0/1309125166 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c020040 con 0x55693e954e40 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f028a000700 1 -- 172.21.15.43:0/1309125166 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7f028c0a97b0 con 0x55693e954e40 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f02b155eb80 1 -- 172.21.15.43:0/1309125166 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0294028890 con 0x55693e954e40 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f028a000700 1 -- 172.21.15.43:0/1309125166 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f028c0b1d20 con 0x55693e954e40 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f02b155eb80 10 client.17391 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f02b155eb80 1 -- 172.21.15.43:0/1309125166 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x55693eaf5970 con 0x55693e954e40 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f028a000700 1 -- 172.21.15.43:0/1309125166 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0294028890 con 0x55693e954e40 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f028a000700 1 -- 172.21.15.43:0/1309125166 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7f028c0a85b0 con 0x55693e954e40 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f02b155eb80 10 client.17391 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f02b155eb80 10 client.17391 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.440+0000 7f02b155eb80 20 client.17391 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.441+0000 7f02b155eb80 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55693eb57150 0x55693eb59570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.441+0000 7f02b155eb80 4 client.17391 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:08.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.441+0000 7f02b155eb80 1 -- 172.21.15.43:0/1309125166 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55693e9fd570 con 0x55693eb57150 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.441+0000 7f0293400700 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55693eb57150 0x55693eb59570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.442+0000 7f0293400700 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55693eb57150 0x55693eb59570 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3489101f52df80cb server_cookie=fc716f6f84d419cf in_seq=0 out_seq=0 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.442+0000 7f028a000700 10 client.17391 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.443+0000 7f028a000700 1 -- 172.21.15.43:0/1309125166 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55693e9fd570 con 0x55693eb57150 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.443+0000 7f028a000700 10 client.17391 handle_command_reply: tid=0 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.443+0000 7f02b155eb80 1 client.17391 shutdown 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f02b155eb80 1 -- 172.21.15.43:0/1309125166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693e954e40 msgr2=0x55693eae59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f02b155eb80 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693e954e40 0x55693eae59f0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f028c0a65c0 tx=0x7f028c010f30 comp rx=0 tx=0).stop 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f02b155eb80 1 -- 172.21.15.43:0/1309125166 shutdown_connections 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f02b155eb80 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55693ead9d30 0x55693eaf41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f02b155eb80 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55693eb57150 0x55693eb59570 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f02b155eb80 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55693e954e40 0x55693eae59f0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f02b155eb80 1 --2- 172.21.15.43:0/1309125166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693ead3db0 0x55693eae5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f02b155eb80 1 -- 172.21.15.43:0/1309125166 >> 172.21.15.43:0/1309125166 conn(0x55693e9b4fc0 msgr2=0x55693ead7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f028a000700 0 client.17391 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f02b155eb80 1 -- 172.21.15.43:0/1309125166 shutdown_connections 2024-05-27T07:27:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.444+0000 7f02b155eb80 1 -- 172.21.15.43:0/1309125166 wait complete. 2024-05-27T07:27:08.435 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:27:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.445+0000 7f02b155eb80 20 client.17391 trim_cache size 0 max 16384 2024-05-27T07:27:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.445+0000 7f02a2a00700 1 -- 172.21.15.43:0/315042079 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02800b2a70 msgr2=0x7f02800b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.445+0000 7f02a2a00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02800b2a70 0x7f02800b4f80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0294005fd0 tx=0x7f0294009f40 comp rx=0 tx=0).stop 2024-05-27T07:27:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.446+0000 7f02a2a00700 1 -- 172.21.15.43:0/315042079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f029c106570 msgr2=0x7f029c19d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.446+0000 7f02a2a00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f029c106570 0x7f029c19d040 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f028c00d980 tx=0x7f028c00dd40 comp rx=0 tx=0).stop 2024-05-27T07:27:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.446+0000 7f02a2a00700 1 -- 172.21.15.43:0/315042079 shutdown_connections 2024-05-27T07:27:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.446+0000 7f02a2a00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02800b2a70 0x7f02800b4f80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.446+0000 7f02a2a00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f029c10c5b0 0x7f029c1a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.446+0000 7f02a2a00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f029c106570 0x7f029c19d040 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.446+0000 7f02a2a00700 1 --2- 172.21.15.43:0/315042079 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f029c106ec0 0x7f029c19d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.446+0000 7f02a2a00700 1 -- 172.21.15.43:0/315042079 >> 172.21.15.43:0/315042079 conn(0x7f029c102070 msgr2=0x7f029c1049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.447+0000 7f02a2a00700 1 -- 172.21.15.43:0/315042079 shutdown_connections 2024-05-27T07:27:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:08.447+0000 7f02a2a00700 1 -- 172.21.15.43:0/315042079 wait complete. 2024-05-27T07:27:09.249 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74667 0.59 MB/sec execute 553 sec latency 617.299 ms 2024-05-27T07:27:09.445 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:09 smithi043 ceph-mon[31531]: pgmap v446: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 826 KiB/s wr, 72 op/s 2024-05-27T07:27:09.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:09 smithi132 ceph-mon[35283]: pgmap v446: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 826 KiB/s wr, 72 op/s 2024-05-27T07:27:09.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:09 smithi179 ceph-mon[35888]: pgmap v446: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 826 KiB/s wr, 72 op/s 2024-05-27T07:27:10.249 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74679 0.58 MB/sec execute 554 sec latency 854.061 ms 2024-05-27T07:27:10.523 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:27:10.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:27:11.002 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:11.249 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74679 0.58 MB/sec execute 555 sec latency 1854.257 ms 2024-05-27T07:27:11.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:11 smithi043 ceph-mon[31531]: pgmap v447: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 450 KiB/s wr, 47 op/s 2024-05-27T07:27:11.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:11 smithi132 ceph-mon[35283]: pgmap v447: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 450 KiB/s wr, 47 op/s 2024-05-27T07:27:11.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:11 smithi179 ceph-mon[35888]: pgmap v447: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 450 KiB/s wr, 47 op/s 2024-05-27T07:27:12.249 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74679 0.58 MB/sec execute 556 sec latency 2854.352 ms 2024-05-27T07:27:13.249 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74679 0.58 MB/sec execute 557 sec latency 3854.433 ms 2024-05-27T07:27:13.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.345+0000 7fce3e400700 1 -- 172.21.15.43:0/587749547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce3810a980 msgr2=0x7fce3810ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:13.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.345+0000 7fce3e400700 1 --2- 172.21.15.43:0/587749547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce3810a980 0x7fce3810ad90 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fce20009990 tx=0x7fce20009ca0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.346+0000 7fce3e400700 1 -- 172.21.15.43:0/587749547 shutdown_connections 2024-05-27T07:27:13.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.346+0000 7fce3e400700 1 --2- 172.21.15.43:0/587749547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce3810b2d0 0x7fce38117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.346+0000 7fce3e400700 1 --2- 172.21.15.43:0/587749547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce38110960 0x7fce38110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.346+0000 7fce3e400700 1 --2- 172.21.15.43:0/587749547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce3810a980 0x7fce3810ad90 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.346+0000 7fce3e400700 1 -- 172.21.15.43:0/587749547 >> 172.21.15.43:0/587749547 conn(0x7fce38077c70 msgr2=0x7fce380780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:13.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.346+0000 7fce3e400700 1 -- 172.21.15.43:0/587749547 shutdown_connections 2024-05-27T07:27:13.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.346+0000 7fce3e400700 1 -- 172.21.15.43:0/587749547 wait complete. 2024-05-27T07:27:13.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.347+0000 7fce3e400700 1 Processor -- start 2024-05-27T07:27:13.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.347+0000 7fce3e400700 1 -- start start 2024-05-27T07:27:13.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.347+0000 7fce3e400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce3810a980 0x7fce381a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.347+0000 7fce3e400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce3810b2d0 0x7fce381a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.348+0000 7fce3e400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce38110960 0x7fce381a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.348+0000 7fce3e400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce3811a960 con 0x7fce3810b2d0 2024-05-27T07:27:13.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.348+0000 7fce3e400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce3811a7e0 con 0x7fce3810a980 2024-05-27T07:27:13.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.348+0000 7fce3e400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fce3811aae0 con 0x7fce38110960 2024-05-27T07:27:13.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.348+0000 7fce2ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce3810b2d0 0x7fce381a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.348+0000 7fce2ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce3810b2d0 0x7fce381a1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57326/0 (socket says 172.21.15.43:57326) 2024-05-27T07:27:13.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.348+0000 7fce2ea00700 1 -- 172.21.15.43:0/2212921610 learned_addr learned my addr 172.21.15.43:0/2212921610 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:13.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.348+0000 7fce37400700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce3810a980 0x7fce381a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.348+0000 7fce37e00700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce38110960 0x7fce381a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.349+0000 7fce2ea00700 1 -- 172.21.15.43:0/2212921610 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce38110960 msgr2=0x7fce381a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.349+0000 7fce2ea00700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce38110960 0x7fce381a5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.349+0000 7fce2ea00700 1 -- 172.21.15.43:0/2212921610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce3810a980 msgr2=0x7fce381a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.349+0000 7fce2ea00700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce3810a980 0x7fce381a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.349+0000 7fce2ea00700 1 -- 172.21.15.43:0/2212921610 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce381a62c0 con 0x7fce3810b2d0 2024-05-27T07:27:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.349+0000 7fce2ea00700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce3810b2d0 0x7fce381a1990 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fce20000c00 tx=0x7fce200057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.350+0000 7fce34c00700 1 -- 172.21.15.43:0/2212921610 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce20027070 con 0x7fce3810b2d0 2024-05-27T07:27:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.350+0000 7fce3e400700 1 -- 172.21.15.43:0/2212921610 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce381a6550 con 0x7fce3810b2d0 2024-05-27T07:27:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.350+0000 7fce34c00700 1 -- 172.21.15.43:0/2212921610 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce2000fb00 con 0x7fce3810b2d0 2024-05-27T07:27:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.350+0000 7fce3e400700 1 -- 172.21.15.43:0/2212921610 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce381a6a30 con 0x7fce3810b2d0 2024-05-27T07:27:13.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.351+0000 7fce34c00700 1 -- 172.21.15.43:0/2212921610 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce20022b50 con 0x7fce3810b2d0 2024-05-27T07:27:13.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.354+0000 7fce34c00700 1 -- 172.21.15.43:0/2212921610 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fce2002a070 con 0x7fce3810b2d0 2024-05-27T07:27:13.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.356+0000 7fce4cfa8b80 1 Processor -- start 2024-05-27T07:27:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.357+0000 7fce4cfa8b80 1 -- start start 2024-05-27T07:27:13.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.357+0000 7fce34c00700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce1c0b29f0 0x7fce1c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.357+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be81dec0 0x5640be9a2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.357+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be9a2db0 0x5640be99c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.357+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be99ce30 0x5640be99d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.357+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5640be8caec0 con 0x5640be81dec0 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.357+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5640be8baa60 con 0x5640be99ce30 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.357+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5640be8beaa0 con 0x5640be9a2db0 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.357+0000 7fce34c00700 1 -- 172.21.15.43:0/2212921610 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7fce200e39a0 con 0x7fce3810b2d0 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37e00700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce1c0b29f0 0x7fce1c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be81dec0 0x5640be9a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be81dec0 0x5640be9a2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57336/0 (socket says 172.21.15.43:57336) 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37400700 1 -- 172.21.15.43:0/3453322954 learned_addr learned my addr 172.21.15.43:0/3453322954 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37e00700 1 --2- 172.21.15.43:0/3453322954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be99ce30 0x5640be99d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce2ea00700 1 --2- 172.21.15.43:0/3453322954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be9a2db0 0x5640be99c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37400700 1 -- 172.21.15.43:0/3453322954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be9a2db0 msgr2=0x5640be99c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:13.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37400700 1 --2- 172.21.15.43:0/3453322954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be9a2db0 0x5640be99c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37400700 1 -- 172.21.15.43:0/3453322954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be99ce30 msgr2=0x5640be99d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:13.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37400700 1 --2- 172.21.15.43:0/3453322954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be99ce30 0x5640be99d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37400700 1 -- 172.21.15.43:0/3453322954 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640be8834a0 con 0x5640be81dec0 2024-05-27T07:27:13.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.358+0000 7fce37e00700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce1c0b29f0 0x7fce1c0b4f00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fce3000bf80 tx=0x7fce300127e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:13.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.359+0000 7fce37400700 1 --2- 172.21.15.43:0/3453322954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be81dec0 0x5640be9a2870 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fce28003fc0 tx=0x7fce28009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b80709eacc8b21 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:13.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.359+0000 7fce2cc00700 1 -- 172.21.15.43:0/3453322954 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce28004750 con 0x5640be81dec0 2024-05-27T07:27:13.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.359+0000 7fce2cc00700 1 -- 172.21.15.43:0/3453322954 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce2801a070 con 0x5640be81dec0 2024-05-27T07:27:13.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.359+0000 7fce2cc00700 1 -- 172.21.15.43:0/3453322954 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce280156e0 con 0x5640be81dec0 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.360+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3453322954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be81dec0 msgr2=0x5640be9a2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.360+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3453322954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be81dec0 0x5640be9a2870 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fce28003fc0 tx=0x7fce28009ba0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.360+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3453322954 shutdown_connections 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.360+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3453322954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be99ce30 0x5640be99d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.360+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3453322954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be81dec0 0x5640be9a2870 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.360+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3453322954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be9a2db0 0x5640be99c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.360+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3453322954 >> 172.21.15.43:0/3453322954 conn(0x5640be87e040 msgr2=0x5640be997950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.360+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3453322954 shutdown_connections 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.360+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3453322954 wait complete. 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.361+0000 7fce4cfa8b80 1 -- start start 2024-05-27T07:27:13.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.361+0000 7fce4cfa8b80 1 Processor -- start 2024-05-27T07:27:13.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.362+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.362+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be99ce30 0x5640be9a6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.362+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be9a2db0 0x5640be9a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.362+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5640be7c1020 con 0x5640be9a2db0 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.362+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5640be85b7e0 con 0x5640be81dec0 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.362+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5640be9a9880 con 0x5640be99ce30 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.362+0000 7fce4cfa8b80 10 client.0 _ll_register_callbacks cb 0x5640be864cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.362+0000 7fce37e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be9a2db0 0x5640be9a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.362+0000 7fce37e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be9a2db0 0x5640be9a6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57342/0 (socket says 172.21.15.43:57342) 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.362+0000 7fce37e00700 1 -- 172.21.15.43:0/3794693756 learned_addr learned my addr 172.21.15.43:0/3794693756 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.363+0000 7fce37400700 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.363+0000 7fce2ea00700 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be99ce30 0x5640be9a6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.363+0000 7fce37e00700 1 -- 172.21.15.43:0/3794693756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be99ce30 msgr2=0x5640be9a6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.363+0000 7fce37e00700 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be99ce30 0x5640be9a6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.363+0000 7fce37e00700 1 -- 172.21.15.43:0/3794693756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 msgr2=0x5640be9a64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:13.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.363+0000 7fce37e00700 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9a64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.363+0000 7fce37e00700 1 -- 172.21.15.43:0/3794693756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640be80a950 con 0x5640be9a2db0 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.363+0000 7fce37e00700 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be9a2db0 0x5640be9a6f50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fce3000ed30 tx=0x7fce30005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194d898f7dce8d96 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.363+0000 7fce25600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.364+0000 7fce25600700 1 -- 172.21.15.43:0/3794693756 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce30022030 con 0x5640be9a2db0 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.364+0000 7fce25600700 1 -- 172.21.15.43:0/3794693756 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7fce30010bf0 con 0x5640be9a2db0 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.364+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3794693756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce2801a070 con 0x5640be9a2db0 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.364+0000 7fce25600700 1 -- 172.21.15.43:0/3794693756 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fce3002e050 con 0x5640be9a2db0 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.364+0000 7fce4cfa8b80 10 client.17403 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.364+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3794693756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x5640be88b0e0 con 0x5640be9a2db0 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.364+0000 7fce25600700 1 -- 172.21.15.43:0/3794693756 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce2801a070 con 0x5640be9a2db0 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.365+0000 7fce25600700 1 -- 172.21.15.43:0/3794693756 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7fce30003880 con 0x5640be9a2db0 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.365+0000 7fce4cfa8b80 10 client.17403 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.365+0000 7fce4cfa8b80 10 client.17403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.365+0000 7fce4cfa8b80 20 client.17403 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:13.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.365+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5640be9b6ac0 0x5640be9b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.365+0000 7fce4cfa8b80 4 client.17403 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:13.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.365+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3794693756 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640be8cce10 con 0x5640be9b6ac0 2024-05-27T07:27:13.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.365+0000 7fce37400700 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5640be9b6ac0 0x5640be9b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.367+0000 7fce37400700 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5640be9b6ac0 0x5640be9b8f50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61757159c4f53c95 server_cookie=e2b23810ba3860f0 in_seq=0 out_seq=0 2024-05-27T07:27:13.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.367+0000 7fce25600700 10 client.17403 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:13.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce25600700 1 -- 172.21.15.43:0/3794693756 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5640be8cce10 con 0x5640be9b6ac0 2024-05-27T07:27:13.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce25600700 10 client.17403 handle_command_reply: tid=0 2024-05-27T07:27:13.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 client.17403 shutdown 2024-05-27T07:27:13.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3794693756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be9a2db0 msgr2=0x5640be9a6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:13.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be9a2db0 0x5640be9a6f50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fce3000ed30 tx=0x7fce30005d70 comp rx=0 tx=0).stop 2024-05-27T07:27:13.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3794693756 shutdown_connections 2024-05-27T07:27:13.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5640be9b6ac0 0x5640be9b8f50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be9a2db0 0x5640be9a6f50 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9a64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3794693756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be99ce30 0x5640be9a6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3794693756 >> 172.21.15.43:0/3794693756 conn(0x5640be87e040 msgr2=0x5640be99d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:13.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce25600700 0 client.17403 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:13.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3794693756 shutdown_connections 2024-05-27T07:27:13.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.369+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3794693756 wait complete. 2024-05-27T07:27:13.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.370+0000 7fce4cfa8b80 20 client.17403 trim_cache size 0 max 16384 2024-05-27T07:27:13.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.392+0000 7fce4cfa8b80 1 Processor -- start 2024-05-27T07:27:13.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.392+0000 7fce4cfa8b80 1 -- start start 2024-05-27T07:27:13.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be87e040 0x5640be99c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9a8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be99ce30 0x5640be9b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce381b77f0 con 0x5640be87e040 2024-05-27T07:27:13.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce381b54c0 con 0x5640be81dec0 2024-05-27T07:27:13.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fce381b7120 con 0x5640be99ce30 2024-05-27T07:27:13.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce37400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be87e040 0x5640be99c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce37400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be87e040 0x5640be99c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57348/0 (socket says 172.21.15.43:57348) 2024-05-27T07:27:13.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce37400700 1 -- 172.21.15.43:0/3913149553 learned_addr learned my addr 172.21.15.43:0/3913149553 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:13.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce37e00700 1 --2- 172.21.15.43:0/3913149553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be99ce30 0x5640be9b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce2ea00700 1 --2- 172.21.15.43:0/3913149553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9a8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce37400700 1 -- 172.21.15.43:0/3913149553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be99ce30 msgr2=0x5640be9b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:13.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce37400700 1 --2- 172.21.15.43:0/3913149553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be99ce30 0x5640be9b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce37400700 1 -- 172.21.15.43:0/3913149553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 msgr2=0x5640be9a8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:13.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce37400700 1 --2- 172.21.15.43:0/3913149553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9a8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.393+0000 7fce37400700 1 -- 172.21.15.43:0/3913149553 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640be9a72c0 con 0x5640be87e040 2024-05-27T07:27:13.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.394+0000 7fce37400700 1 --2- 172.21.15.43:0/3913149553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be87e040 0x5640be99c8f0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fce28009480 tx=0x7fce280091b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c7d0c03795918f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:13.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.394+0000 7fce27e00700 1 -- 172.21.15.43:0/3913149553 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce28004990 con 0x5640be87e040 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.394+0000 7fce27e00700 1 -- 172.21.15.43:0/3913149553 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce2801a030 con 0x5640be87e040 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.394+0000 7fce27e00700 1 -- 172.21.15.43:0/3913149553 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce28007760 con 0x5640be87e040 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.395+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3913149553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be87e040 msgr2=0x5640be99c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.395+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3913149553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be87e040 0x5640be99c8f0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fce28009480 tx=0x7fce280091b0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.395+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3913149553 shutdown_connections 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.395+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3913149553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be99ce30 0x5640be9b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.395+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3913149553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be87e040 0x5640be99c8f0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.395+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/3913149553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9a8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.395+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3913149553 >> 172.21.15.43:0/3913149553 conn(0x5640be9a2000 msgr2=0x5640be9a3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.395+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3913149553 shutdown_connections 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.395+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/3913149553 wait complete. 2024-05-27T07:27:13.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.396+0000 7fce4cfa8b80 1 -- start start 2024-05-27T07:27:13.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.396+0000 7fce4cfa8b80 1 Processor -- start 2024-05-27T07:27:13.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.397+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.397+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be87e040 0x5640be9abb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.397+0000 7fce4cfa8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be99ce30 0x5640be9ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.397+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce381b62c0 con 0x5640be99ce30 2024-05-27T07:27:13.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.397+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce381b5b90 con 0x5640be81dec0 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.397+0000 7fce4cfa8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fce381b69f0 con 0x5640be87e040 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.397+0000 7fce4cfa8b80 10 client.0 _ll_register_callbacks cb 0x5640be864cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.397+0000 7fce37e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be99ce30 0x5640be9ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.397+0000 7fce37e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be99ce30 0x5640be9ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57362/0 (socket says 172.21.15.43:57362) 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.397+0000 7fce37e00700 1 -- 172.21.15.43:0/1652475705 learned_addr learned my addr 172.21.15.43:0/1652475705 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.398+0000 7fce2ea00700 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be87e040 0x5640be9abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.398+0000 7fce37400700 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.398+0000 7fce37e00700 1 -- 172.21.15.43:0/1652475705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be87e040 msgr2=0x5640be9abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.398+0000 7fce37e00700 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be87e040 0x5640be9abb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.398+0000 7fce37e00700 1 -- 172.21.15.43:0/1652475705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 msgr2=0x5640be9af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.398+0000 7fce37e00700 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.398+0000 7fce37e00700 1 -- 172.21.15.43:0/1652475705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640be9ac5c0 con 0x5640be99ce30 2024-05-27T07:27:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.398+0000 7fce37400700 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:13.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.398+0000 7fce37e00700 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be99ce30 0x5640be9ac050 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fce30003c70 tx=0x7fce300039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa269732eebb495 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:13.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.398+0000 7fce25600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:13.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.399+0000 7fce25600700 1 -- 172.21.15.43:0/1652475705 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce30036070 con 0x5640be99ce30 2024-05-27T07:27:13.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.399+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/1652475705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce2801a030 con 0x5640be99ce30 2024-05-27T07:27:13.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.399+0000 7fce25600700 1 -- 172.21.15.43:0/1652475705 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7fce30003e60 con 0x5640be99ce30 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.399+0000 7fce25600700 1 -- 172.21.15.43:0/1652475705 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fce30042050 con 0x5640be99ce30 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.399+0000 7fce4cfa8b80 10 client.17409 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.399+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/1652475705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x5640bea22350 con 0x5640be99ce30 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.399+0000 7fce25600700 1 -- 172.21.15.43:0/1652475705 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce2801a030 con 0x5640be99ce30 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.399+0000 7fce25600700 1 -- 172.21.15.43:0/1652475705 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7fce30027570 con 0x5640be99ce30 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.400+0000 7fce4cfa8b80 10 client.17409 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.400+0000 7fce4cfa8b80 10 client.17409 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.400+0000 7fce4cfa8b80 20 client.17409 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.400+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5640be9a2db0 0x5640bea24a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.400+0000 7fce4cfa8b80 4 client.17409 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.400+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/1652475705 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5640be8c65f0 con 0x5640be9a2db0 2024-05-27T07:27:13.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.400+0000 7fce37400700 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5640be9a2db0 0x5640bea24a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:13.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.402+0000 7fce37400700 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5640be9a2db0 0x5640bea24a80 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9614a02aefeeb2eb server_cookie=e9c5499c87b61eaa in_seq=0 out_seq=0 2024-05-27T07:27:13.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.402+0000 7fce25600700 10 client.17409 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:13.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce25600700 1 -- 172.21.15.43:0/1652475705 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5640be8c65f0 con 0x5640be9a2db0 2024-05-27T07:27:13.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce25600700 10 client.17409 handle_command_reply: tid=0 2024-05-27T07:27:13.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce4cfa8b80 1 client.17409 shutdown 2024-05-27T07:27:13.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/1652475705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be99ce30 msgr2=0x5640be9ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:13.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be99ce30 0x5640be9ac050 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fce30003c70 tx=0x7fce300039e0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/1652475705 shutdown_connections 2024-05-27T07:27:13.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5640be9a2db0 0x5640bea24a80 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640be99ce30 0x5640be9ac050 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640be81dec0 0x5640be9af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce4cfa8b80 1 --2- 172.21.15.43:0/1652475705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640be87e040 0x5640be9abb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/1652475705 >> 172.21.15.43:0/1652475705 conn(0x5640be9a2000 msgr2=0x5640be9a34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:13.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.405+0000 7fce25600700 0 client.17409 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:13.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.406+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/1652475705 shutdown_connections 2024-05-27T07:27:13.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.406+0000 7fce4cfa8b80 1 -- 172.21.15.43:0/1652475705 wait complete. 2024-05-27T07:27:13.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.406+0000 7fce4cfa8b80 20 client.17409 trim_cache size 0 max 16384 2024-05-27T07:27:13.397 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:27:13.397 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:27:13.397 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "2edc5c44-570a-43bb-8d8c-eff625c8b07f", 2024-05-27T07:27:13.397 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:27:13.397 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:27:13.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 -- 172.21.15.43:0/2212921610 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce1c0b29f0 msgr2=0x7fce1c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce1c0b29f0 0x7fce1c0b4f00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fce3000bf80 tx=0x7fce300127e0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 -- 172.21.15.43:0/2212921610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce3810b2d0 msgr2=0x7fce381a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce3810b2d0 0x7fce381a1990 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fce20000c00 tx=0x7fce200057a0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 -- 172.21.15.43:0/2212921610 shutdown_connections 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce1c0b29f0 0x7fce1c0b4f00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce38110960 0x7fce381a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce3810a980 0x7fce381a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 --2- 172.21.15.43:0/2212921610 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce3810b2d0 0x7fce381a1990 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 -- 172.21.15.43:0/2212921610 >> 172.21.15.43:0/2212921610 conn(0x7fce38077c70 msgr2=0x7fce3807b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 -- 172.21.15.43:0/2212921610 shutdown_connections 2024-05-27T07:27:13.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:13.407+0000 7fce3e400700 1 -- 172.21.15.43:0/2212921610 wait complete. 2024-05-27T07:27:13.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:13 smithi132 ceph-mon[35283]: pgmap v448: 129 pgs: 129 active+clean; 145 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 584 KiB/s wr, 55 op/s 2024-05-27T07:27:13.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:13 smithi043 ceph-mon[31531]: pgmap v448: 129 pgs: 129 active+clean; 145 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 584 KiB/s wr, 55 op/s 2024-05-27T07:27:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:13 smithi179 ceph-mon[35888]: pgmap v448: 129 pgs: 129 active+clean; 145 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 584 KiB/s wr, 55 op/s 2024-05-27T07:27:14.249 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74762 0.58 MB/sec execute 558 sec latency 4056.131 ms 2024-05-27T07:27:14.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:27:14.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:27:14.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:27:14.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:27:14.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:27:14.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:27:14.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:27:14.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:27:14.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:27:14.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:14 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:27:14.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:27:14.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:14 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:27:14.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:14 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:27:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:15.250 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74820 0.58 MB/sec execute 559 sec latency 43.953 ms 2024-05-27T07:27:15.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:15 smithi043 ceph-mon[31531]: pgmap v449: 129 pgs: 129 active+clean; 145 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 575 KiB/s wr, 54 op/s 2024-05-27T07:27:15.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:15 smithi179 ceph-mon[35888]: pgmap v449: 129 pgs: 129 active+clean; 145 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 575 KiB/s wr, 54 op/s 2024-05-27T07:27:15.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:15 smithi132 ceph-mon[35283]: pgmap v449: 129 pgs: 129 active+clean; 145 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 575 KiB/s wr, 54 op/s 2024-05-27T07:27:16.249 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74882 0.58 MB/sec execute 560 sec latency 60.970 ms 2024-05-27T07:27:17.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.122+0000 7fce1e000700 1 -- 172.21.15.43:0/3698352322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce18105280 msgr2=0x7fce18105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:17.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.122+0000 7fce1e000700 1 --2- 172.21.15.43:0/3698352322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce18105280 0x7fce18105690 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009990 tx=0x7fce0c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.122+0000 7fce1e000700 1 -- 172.21.15.43:0/3698352322 shutdown_connections 2024-05-27T07:27:17.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.122+0000 7fce1e000700 1 --2- 172.21.15.43:0/3698352322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce1810e6f0 0x7fce18110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.122+0000 7fce1e000700 1 --2- 172.21.15.43:0/3698352322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce18105280 0x7fce18105690 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.122+0000 7fce1e000700 1 --2- 172.21.15.43:0/3698352322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce18105bd0 0x7fce1810e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.122+0000 7fce1e000700 1 -- 172.21.15.43:0/3698352322 >> 172.21.15.43:0/3698352322 conn(0x7fce18078af0 msgr2=0x7fce18078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:17.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.122+0000 7fce1e000700 1 -- 172.21.15.43:0/3698352322 shutdown_connections 2024-05-27T07:27:17.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.123+0000 7fce1e000700 1 -- 172.21.15.43:0/3698352322 wait complete. 2024-05-27T07:27:17.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.125+0000 7fce1e000700 1 Processor -- start 2024-05-27T07:27:17.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.125+0000 7fce1e000700 1 -- start start 2024-05-27T07:27:17.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.125+0000 7fce1e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce18105280 0x7fce181a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.125+0000 7fce1e000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce18105bd0 0x7fce181a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.125+0000 7fce1e000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce1810e6f0 0x7fce1819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1e000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce18114140 con 0x7fce18105280 2024-05-27T07:27:17.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1e000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce18113fc0 con 0x7fce1810e6f0 2024-05-27T07:27:17.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1e000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fce181142c0 con 0x7fce18105bd0 2024-05-27T07:27:17.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce18105280 0x7fce181a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce18105280 0x7fce181a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48624/0 (socket says 172.21.15.43:48624) 2024-05-27T07:27:17.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1cc00700 1 -- 172.21.15.43:0/1119868583 learned_addr learned my addr 172.21.15.43:0/1119868583 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:17.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce17e00700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce18105bd0 0x7fce181a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1d600700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce1810e6f0 0x7fce1819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1cc00700 1 -- 172.21.15.43:0/1119868583 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce18105bd0 msgr2=0x7fce181a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:17.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1cc00700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce18105bd0 0x7fce181a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1cc00700 1 -- 172.21.15.43:0/1119868583 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce1810e6f0 msgr2=0x7fce1819aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:17.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1cc00700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce1810e6f0 0x7fce1819aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1cc00700 1 -- 172.21.15.43:0/1119868583 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1819b040 con 0x7fce18105280 2024-05-27T07:27:17.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.126+0000 7fce1d600700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce1810e6f0 0x7fce1819aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:17.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.127+0000 7fce1cc00700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce18105280 0x7fce181a07e0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fce0c000c00 tx=0x7fce0c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:17.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.127+0000 7fce15600700 1 -- 172.21.15.43:0/1119868583 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c027070 con 0x7fce18105280 2024-05-27T07:27:17.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.127+0000 7fce15600700 1 -- 172.21.15.43:0/1119868583 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce0c00fb80 con 0x7fce18105280 2024-05-27T07:27:17.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.127+0000 7fce15600700 1 -- 172.21.15.43:0/1119868583 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c022c30 con 0x7fce18105280 2024-05-27T07:27:17.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.127+0000 7fce1e000700 1 -- 172.21.15.43:0/1119868583 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1819b2d0 con 0x7fce18105280 2024-05-27T07:27:17.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.127+0000 7fce1e000700 1 -- 172.21.15.43:0/1119868583 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce1819b7b0 con 0x7fce18105280 2024-05-27T07:27:17.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.132+0000 7fce15600700 1 -- 172.21.15.43:0/1119868583 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fce0c02a070 con 0x7fce18105280 2024-05-27T07:27:17.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.133+0000 7fce2e16cb80 1 Processor -- start 2024-05-27T07:27:17.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.133+0000 7fce2e16cb80 1 -- start start 2024-05-27T07:27:17.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.133+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4abd9f0 0x5629f4b51410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b51950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4b51f30 0x5629f4b616e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce15600700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce000b29f0 0x7fce000b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5629f4a7ee40 con 0x5629f4b51f30 2024-05-27T07:27:17.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5629f4a6e9e0 con 0x5629f49d1e40 2024-05-27T07:27:17.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5629f4a72a20 con 0x5629f4abd9f0 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce15600700 1 -- 172.21.15.43:0/1119868583 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7fce0c0e3b70 con 0x7fce18105280 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce17e00700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce000b29f0 0x7fce000b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce1d600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4b51f30 0x5629f4b616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce1d600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4b51f30 0x5629f4b616e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48640/0 (socket says 172.21.15.43:48640) 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce1d600700 1 -- 172.21.15.43:0/460964966 learned_addr learned my addr 172.21.15.43:0/460964966 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce1cc00700 1 --2- 172.21.15.43:0/460964966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b51950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.134+0000 7fce1d600700 1 --2- 172.21.15.43:0/460964966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4abd9f0 0x5629f4b51410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.135+0000 7fce1d600700 1 -- 172.21.15.43:0/460964966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4abd9f0 msgr2=0x5629f4b51410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.135+0000 7fce1d600700 1 --2- 172.21.15.43:0/460964966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4abd9f0 0x5629f4b51410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.135+0000 7fce1d600700 1 -- 172.21.15.43:0/460964966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 msgr2=0x5629f4b51950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.135+0000 7fce1d600700 1 --2- 172.21.15.43:0/460964966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b51950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.135+0000 7fce1d600700 1 -- 172.21.15.43:0/460964966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629f4a37420 con 0x5629f4b51f30 2024-05-27T07:27:17.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.135+0000 7fce17e00700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce000b29f0 0x7fce000b4f00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fce04005ea0 tx=0x7fce04005e30 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:17.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.135+0000 7fce1d600700 1 --2- 172.21.15.43:0/460964966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4b51f30 0x5629f4b616e0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fce10007970 tx=0x7fce10009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c8c271938c887 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:17.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.136+0000 7fce0a000700 1 -- 172.21.15.43:0/460964966 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10017560 con 0x5629f4b51f30 2024-05-27T07:27:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.136+0000 7fce0a000700 1 -- 172.21.15.43:0/460964966 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce10017b90 con 0x5629f4b51f30 2024-05-27T07:27:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.136+0000 7fce0a000700 1 -- 172.21.15.43:0/460964966 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10019a70 con 0x5629f4b51f30 2024-05-27T07:27:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.137+0000 7fce2e16cb80 1 -- 172.21.15.43:0/460964966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4b51f30 msgr2=0x5629f4b616e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.137+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/460964966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4b51f30 0x5629f4b616e0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fce10007970 tx=0x7fce10009380 comp rx=0 tx=0).stop 2024-05-27T07:27:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.137+0000 7fce2e16cb80 1 -- 172.21.15.43:0/460964966 shutdown_connections 2024-05-27T07:27:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.137+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/460964966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4b51f30 0x5629f4b616e0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.137+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/460964966 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4abd9f0 0x5629f4b51410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.137+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/460964966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b51950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.137+0000 7fce2e16cb80 1 -- 172.21.15.43:0/460964966 >> 172.21.15.43:0/460964966 conn(0x5629f4a6b390 msgr2=0x5629f4b4c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:17.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.137+0000 7fce2e16cb80 1 -- 172.21.15.43:0/460964966 shutdown_connections 2024-05-27T07:27:17.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.137+0000 7fce2e16cb80 1 -- 172.21.15.43:0/460964966 wait complete. 2024-05-27T07:27:17.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.138+0000 7fce2e16cb80 1 -- start start 2024-05-27T07:27:17.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.138+0000 7fce2e16cb80 1 Processor -- start 2024-05-27T07:27:17.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.139+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f49d1e40 0x5629f4b5edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.139+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4abd9f0 0x5629f4b5f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.139+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f4b51f30 0x5629f4b5f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.139+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5629f4974fa0 con 0x5629f49d1e40 2024-05-27T07:27:17.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.139+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5629f4a0f760 con 0x5629f4b51f30 2024-05-27T07:27:17.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.139+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5629f4b62130 con 0x5629f4abd9f0 2024-05-27T07:27:17.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.139+0000 7fce2e16cb80 10 client.0 _ll_register_callbacks cb 0x5629f4a18c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:17.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.139+0000 7fce1cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f49d1e40 0x5629f4b5edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.139+0000 7fce1cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f49d1e40 0x5629f4b5edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48644/0 (socket says 172.21.15.43:48644) 2024-05-27T07:27:17.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.139+0000 7fce1cc00700 1 -- 172.21.15.43:0/1787100142 learned_addr learned my addr 172.21.15.43:0/1787100142 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:17.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.140+0000 7fce17e00700 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4abd9f0 0x5629f4b5f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.140+0000 7fce1cc00700 1 -- 172.21.15.43:0/1787100142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4abd9f0 msgr2=0x5629f4b5f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:17.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.140+0000 7fce1cc00700 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4abd9f0 0x5629f4b5f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.140+0000 7fce1d600700 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f4b51f30 0x5629f4b5f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.140+0000 7fce1cc00700 1 -- 172.21.15.43:0/1787100142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f4b51f30 msgr2=0x5629f4b5f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:17.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.140+0000 7fce1cc00700 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f4b51f30 0x5629f4b5f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.140+0000 7fce1cc00700 1 -- 172.21.15.43:0/1787100142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629f49be8d0 con 0x5629f49d1e40 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.140+0000 7fce1d600700 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f4b51f30 0x5629f4b5f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.140+0000 7fce1cc00700 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f49d1e40 0x5629f4b5edc0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009750 tx=0x7fce0c0aa340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682d00e52479660 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.140+0000 7fcdfea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.141+0000 7fcdfea00700 1 -- 172.21.15.43:0/1787100142 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c0e1320 con 0x5629f49d1e40 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.141+0000 7fcdfea00700 1 -- 172.21.15.43:0/1787100142 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7fce0c0a70a0 con 0x5629f49d1e40 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.141+0000 7fce2e16cb80 1 -- 172.21.15.43:0/1787100142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce10017b90 con 0x5629f49d1e40 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.141+0000 7fcdfea00700 1 -- 172.21.15.43:0/1787100142 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fce0c0ab1b0 con 0x5629f49d1e40 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.141+0000 7fce2e16cb80 10 client.17421 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.141+0000 7fce2e16cb80 1 -- 172.21.15.43:0/1787100142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x5629f4a3f060 con 0x5629f49d1e40 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.141+0000 7fcdfea00700 1 -- 172.21.15.43:0/1787100142 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce10017b90 con 0x5629f49d1e40 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.142+0000 7fcdfea00700 1 -- 172.21.15.43:0/1787100142 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7fce0c070290 con 0x5629f49d1e40 2024-05-27T07:27:17.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.142+0000 7fce2e16cb80 10 client.17421 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.142+0000 7fce2e16cb80 10 client.17421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.142+0000 7fce2e16cb80 20 client.17421 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.142+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5629f4b6f240 0x5629f4b71710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.142+0000 7fce2e16cb80 4 client.17421 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.142+0000 7fce2e16cb80 1 -- 172.21.15.43:0/1787100142 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629f4a80d90 con 0x5629f4b6f240 2024-05-27T07:27:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.142+0000 7fce17e00700 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5629f4b6f240 0x5629f4b71710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.143+0000 7fce17e00700 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5629f4b6f240 0x5629f4b71710 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc9910ebd7eb7d75 server_cookie=4925ad46ec90bd27 in_seq=0 out_seq=0 2024-05-27T07:27:17.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.143+0000 7fcdfea00700 10 client.17421 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:17.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.146+0000 7fcdfea00700 1 -- 172.21.15.43:0/1787100142 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5629f4a80d90 con 0x5629f4b6f240 2024-05-27T07:27:17.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.146+0000 7fcdfea00700 10 client.17421 handle_command_reply: tid=0 2024-05-27T07:27:17.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.146+0000 7fce2e16cb80 1 client.17421 shutdown 2024-05-27T07:27:17.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.146+0000 7fce2e16cb80 1 -- 172.21.15.43:0/1787100142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f49d1e40 msgr2=0x5629f4b5edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.146+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f49d1e40 0x5629f4b5edc0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009750 tx=0x7fce0c0aa340 comp rx=0 tx=0).stop 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.147+0000 7fce2e16cb80 1 -- 172.21.15.43:0/1787100142 shutdown_connections 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.147+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f4b51f30 0x5629f4b5f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.147+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5629f4b6f240 0x5629f4b71710 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.147+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f49d1e40 0x5629f4b5edc0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.147+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/1787100142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4abd9f0 0x5629f4b5f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.147+0000 7fce2e16cb80 1 -- 172.21.15.43:0/1787100142 >> 172.21.15.43:0/1787100142 conn(0x5629f4a6b390 msgr2=0x5629f4b4c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.147+0000 7fcdfea00700 0 client.17421 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.147+0000 7fce2e16cb80 1 -- 172.21.15.43:0/1787100142 shutdown_connections 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.147+0000 7fce2e16cb80 1 -- 172.21.15.43:0/1787100142 wait complete. 2024-05-27T07:27:17.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.147+0000 7fce2e16cb80 20 client.17421 trim_cache size 0 max 16384 2024-05-27T07:27:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.171+0000 7fce2e16cb80 1 Processor -- start 2024-05-27T07:27:17.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.171+0000 7fce2e16cb80 1 -- start start 2024-05-27T07:27:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.172+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.172+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b5a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.172+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 0x5629f4b50830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.172+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce181aed20 con 0x5629f4abd9f0 2024-05-27T07:27:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.172+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce181ac9f0 con 0x5629f49d1e40 2024-05-27T07:27:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.172+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fce181ae650 con 0x5629f4b51f30 2024-05-27T07:27:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.172+0000 7fce17e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b5a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.172+0000 7fce17e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b5a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48656/0 (socket says 172.21.15.43:48656) 2024-05-27T07:27:17.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.172+0000 7fce17e00700 1 -- 172.21.15.43:0/74402608 learned_addr learned my addr 172.21.15.43:0/74402608 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:17.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.173+0000 7fce1d600700 1 --2- 172.21.15.43:0/74402608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 0x5629f4b50830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.173+0000 7fce1cc00700 1 --2- 172.21.15.43:0/74402608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.173+0000 7fce17e00700 1 -- 172.21.15.43:0/74402608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 msgr2=0x5629f4b50830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:17.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.173+0000 7fce17e00700 1 --2- 172.21.15.43:0/74402608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 0x5629f4b50830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.173+0000 7fce17e00700 1 -- 172.21.15.43:0/74402608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 msgr2=0x5629f4b718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.173+0000 7fce17e00700 1 --2- 172.21.15.43:0/74402608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.173+0000 7fce17e00700 1 -- 172.21.15.43:0/74402608 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629f4b5fbb0 con 0x5629f4abd9f0 2024-05-27T07:27:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.173+0000 7fce1d600700 1 --2- 172.21.15.43:0/74402608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 0x5629f4b50830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.173+0000 7fce17e00700 1 --2- 172.21.15.43:0/74402608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b5a590 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x5629f4b4bf00 tx=0x7fce0400af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fae647c86031b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.174+0000 7fce09600700 1 -- 172.21.15.43:0/74402608 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0401f820 con 0x5629f4abd9f0 2024-05-27T07:27:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.174+0000 7fce09600700 1 -- 172.21.15.43:0/74402608 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce0401fe50 con 0x5629f4abd9f0 2024-05-27T07:27:17.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.174+0000 7fce09600700 1 -- 172.21.15.43:0/74402608 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04010430 con 0x5629f4abd9f0 2024-05-27T07:27:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.175+0000 7fce2e16cb80 1 -- 172.21.15.43:0/74402608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 msgr2=0x5629f4b5a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.175+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/74402608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b5a590 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x5629f4b4bf00 tx=0x7fce0400af90 comp rx=0 tx=0).stop 2024-05-27T07:27:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.175+0000 7fce2e16cb80 1 -- 172.21.15.43:0/74402608 shutdown_connections 2024-05-27T07:27:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.175+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/74402608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 0x5629f4b50830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.175+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/74402608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.175+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/74402608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b5a590 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.175+0000 7fce2e16cb80 1 -- 172.21.15.43:0/74402608 >> 172.21.15.43:0/74402608 conn(0x5629f4a6b390 msgr2=0x5629f4b50120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.175+0000 7fce2e16cb80 1 -- 172.21.15.43:0/74402608 shutdown_connections 2024-05-27T07:27:17.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.175+0000 7fce2e16cb80 1 -- 172.21.15.43:0/74402608 wait complete. 2024-05-27T07:27:17.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.176+0000 7fce2e16cb80 1 -- start start 2024-05-27T07:27:17.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.176+0000 7fce2e16cb80 1 Processor -- start 2024-05-27T07:27:17.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b642f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b64830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce2e16cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 0x5629f4b64d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce181ad7f0 con 0x5629f4abd9f0 2024-05-27T07:27:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce181ad0c0 con 0x5629f49d1e40 2024-05-27T07:27:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce2e16cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fce181adf20 con 0x5629f4b51f30 2024-05-27T07:27:17.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce2e16cb80 10 client.0 _ll_register_callbacks cb 0x5629f4a18c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce17e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b64830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce17e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b64830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48658/0 (socket says 172.21.15.43:48658) 2024-05-27T07:27:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce17e00700 1 -- 172.21.15.43:0/3843014025 learned_addr learned my addr 172.21.15.43:0/3843014025 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.177+0000 7fce1cc00700 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b642f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fce1d600700 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 0x5629f4b64d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fce17e00700 1 -- 172.21.15.43:0/3843014025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 msgr2=0x5629f4b64d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:17.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fce17e00700 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 0x5629f4b64d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fce17e00700 1 -- 172.21.15.43:0/3843014025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 msgr2=0x5629f4b642f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fce17e00700 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b642f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fce17e00700 1 -- 172.21.15.43:0/3843014025 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629f4b59390 con 0x5629f4abd9f0 2024-05-27T07:27:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fce1cc00700 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b642f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fce1d600700 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 0x5629f4b64d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fce17e00700 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b64830 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fce0401fdc0 tx=0x7fce040050f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53d29478fd03bda server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fcdfea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.178+0000 7fcdfea00700 1 -- 172.21.15.43:0/3843014025 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04004750 con 0x5629f4abd9f0 2024-05-27T07:27:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.179+0000 7fcdfea00700 1 -- 172.21.15.43:0/3843014025 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7fce040048f0 con 0x5629f4abd9f0 2024-05-27T07:27:17.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.179+0000 7fce2e16cb80 1 -- 172.21.15.43:0/3843014025 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce0401fe50 con 0x5629f4abd9f0 2024-05-27T07:27:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.179+0000 7fcdfea00700 1 -- 172.21.15.43:0/3843014025 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fce04004af0 con 0x5629f4abd9f0 2024-05-27T07:27:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.179+0000 7fce2e16cb80 10 client.17427 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.179+0000 7fce2e16cb80 1 -- 172.21.15.43:0/3843014025 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x5629f4b5a600 con 0x5629f4abd9f0 2024-05-27T07:27:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.179+0000 7fcdfea00700 1 -- 172.21.15.43:0/3843014025 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce0401fe50 con 0x5629f4abd9f0 2024-05-27T07:27:17.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.180+0000 7fcdfea00700 1 -- 172.21.15.43:0/3843014025 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7fce04038740 con 0x5629f4abd9f0 2024-05-27T07:27:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.180+0000 7fce2e16cb80 10 client.17427 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.180+0000 7fce2e16cb80 10 client.17427 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.180+0000 7fce2e16cb80 20 client.17427 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.180+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5629f4bde800 0x5629f4be0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.180+0000 7fce2e16cb80 4 client.17427 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:27:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.180+0000 7fce2e16cb80 1 -- 172.21.15.43:0/3843014025 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5629f4a7a570 con 0x5629f4bde800 2024-05-27T07:27:17.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.180+0000 7fce1cc00700 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5629f4bde800 0x5629f4be0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.181+0000 7fce1cc00700 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5629f4bde800 0x5629f4be0c20 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57f50e22d3135ff0 server_cookie=87d1277849aa278b in_seq=0 out_seq=0 2024-05-27T07:27:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.181+0000 7fcdfea00700 10 client.17427 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fcdfea00700 1 -- 172.21.15.43:0/3843014025 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5629f4a7a570 con 0x5629f4bde800 2024-05-27T07:27:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fcdfea00700 10 client.17427 handle_command_reply: tid=0 2024-05-27T07:27:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fce2e16cb80 1 client.17427 shutdown 2024-05-27T07:27:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fce2e16cb80 1 -- 172.21.15.43:0/3843014025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 msgr2=0x5629f4b64830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:17.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b64830 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fce0401fdc0 tx=0x7fce040050f0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fce2e16cb80 1 -- 172.21.15.43:0/3843014025 shutdown_connections 2024-05-27T07:27:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629f4b51f30 0x5629f4b64d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5629f49d1e40 0x5629f4b642f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5629f4bde800 0x5629f4be0c20 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fce2e16cb80 1 --2- 172.21.15.43:0/3843014025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5629f4abd9f0 0x5629f4b64830 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fce2e16cb80 1 -- 172.21.15.43:0/3843014025 >> 172.21.15.43:0/3843014025 conn(0x5629f4a6b390 msgr2=0x5629f4b4c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fcdfea00700 0 client.17427 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.182+0000 7fce2e16cb80 1 -- 172.21.15.43:0/3843014025 shutdown_connections 2024-05-27T07:27:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.183+0000 7fce2e16cb80 1 -- 172.21.15.43:0/3843014025 wait complete. 2024-05-27T07:27:17.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.183+0000 7fce2e16cb80 20 client.17427 trim_cache size 0 max 16384 2024-05-27T07:27:17.174 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:27:17.174 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:27:17.174 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:27:17.174 INFO:teuthology.orchestra.run.smithi043.stdout: "2edc5c44-570a-43bb-8d8c-eff625c8b07f": { 2024-05-27T07:27:17.174 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:27:17.174 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "2edc5c44-570a-43bb-8d8c-eff625c8b07f", 2024-05-27T07:27:17.174 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:27:17.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:27:17.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:27:17.174 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:27:17.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 -- 172.21.15.43:0/1119868583 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce000b29f0 msgr2=0x7fce000b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:17.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce000b29f0 0x7fce000b4f00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fce04005ea0 tx=0x7fce04005e30 comp rx=0 tx=0).stop 2024-05-27T07:27:17.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 -- 172.21.15.43:0/1119868583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce18105280 msgr2=0x7fce181a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:17.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce18105280 0x7fce181a07e0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fce0c000c00 tx=0x7fce0c005ef0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 -- 172.21.15.43:0/1119868583 shutdown_connections 2024-05-27T07:27:17.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce000b29f0 0x7fce000b4f00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce1810e6f0 0x7fce1819aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce18105280 0x7fce181a07e0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 --2- 172.21.15.43:0/1119868583 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce18105bd0 0x7fce181a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:17.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 -- 172.21.15.43:0/1119868583 >> 172.21.15.43:0/1119868583 conn(0x7fce18078af0 msgr2=0x7fce181010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:17.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 -- 172.21.15.43:0/1119868583 shutdown_connections 2024-05-27T07:27:17.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:17.184+0000 7fce1e000700 1 -- 172.21.15.43:0/1119868583 wait complete. 2024-05-27T07:27:17.250 INFO:tasks.workunit.client.0.smithi043.stdout: 1 74947 0.58 MB/sec execute 561 sec latency 48.851 ms 2024-05-27T07:27:17.665 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:17 smithi043 ceph-mon[31531]: pgmap v450: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 431 KiB/s wr, 29 op/s 2024-05-27T07:27:17.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:17 smithi179 ceph-mon[35888]: pgmap v450: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 431 KiB/s wr, 29 op/s 2024-05-27T07:27:17.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:17 smithi132 ceph-mon[35283]: pgmap v450: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 431 KiB/s wr, 29 op/s 2024-05-27T07:27:18.193 INFO:tasks.cephfs.filesystem:scrub status for tag:2edc5c44-570a-43bb-8d8c-eff625c8b07f - {'path': '/', 'tag': '2edc5c44-570a-43bb-8d8c-eff625c8b07f', 'options': 'recursive,force'} 2024-05-27T07:27:18.250 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75006 0.58 MB/sec execute 562 sec latency 42.695 ms 2024-05-27T07:27:19.250 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75075 0.58 MB/sec execute 563 sec latency 60.424 ms 2024-05-27T07:27:19.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:19 smithi043 ceph-mon[31531]: pgmap v451: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 367 KiB/s wr, 25 op/s 2024-05-27T07:27:19.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:19 smithi179 ceph-mon[35888]: pgmap v451: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 367 KiB/s wr, 25 op/s 2024-05-27T07:27:19.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:19 smithi132 ceph-mon[35283]: pgmap v451: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 367 KiB/s wr, 25 op/s 2024-05-27T07:27:20.250 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75136 0.58 MB/sec execute 564 sec latency 54.186 ms 2024-05-27T07:27:21.250 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75234 0.58 MB/sec execute 565 sec latency 60.214 ms 2024-05-27T07:27:21.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:21 smithi043 ceph-mon[31531]: pgmap v452: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 359 KiB/s wr, 24 op/s 2024-05-27T07:27:21.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:21 smithi179 ceph-mon[35888]: pgmap v452: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 359 KiB/s wr, 24 op/s 2024-05-27T07:27:21.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:21 smithi132 ceph-mon[35283]: pgmap v452: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 359 KiB/s wr, 24 op/s 2024-05-27T07:27:22.250 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75358 0.58 MB/sec execute 566 sec latency 58.648 ms 2024-05-27T07:27:22.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:22 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:27:22.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:22 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:27:22.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:22 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:27:22.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:22 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:27:22.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:22 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:27:22.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:22 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:27:22.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:22 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:27:22.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:22 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:27:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:22 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:27:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:22 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:27:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:22 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:27:22.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:22 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:27:23.251 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75464 0.57 MB/sec execute 567 sec latency 59.275 ms 2024-05-27T07:27:23.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:23 smithi043 ceph-mon[31531]: pgmap v453: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 536 KiB/s wr, 42 op/s 2024-05-27T07:27:23.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:23 smithi179 ceph-mon[35888]: pgmap v453: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 536 KiB/s wr, 42 op/s 2024-05-27T07:27:23.742 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:23 smithi132 ceph-mon[35283]: pgmap v453: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 536 KiB/s wr, 42 op/s 2024-05-27T07:27:24.250 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75586 0.58 MB/sec execute 568 sec latency 61.049 ms 2024-05-27T07:27:25.251 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75699 0.57 MB/sec execute 569 sec latency 53.277 ms 2024-05-27T07:27:25.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:25 smithi043 ceph-mon[31531]: pgmap v454: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 356 KiB/s wr, 28 op/s 2024-05-27T07:27:25.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:25 smithi179 ceph-mon[35888]: pgmap v454: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 356 KiB/s wr, 28 op/s 2024-05-27T07:27:25.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:25 smithi132 ceph-mon[35283]: pgmap v454: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 356 KiB/s wr, 28 op/s 2024-05-27T07:27:26.251 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75796 0.57 MB/sec execute 570 sec latency 64.563 ms 2024-05-27T07:27:27.251 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75949 0.58 MB/sec execute 571 sec latency 63.985 ms 2024-05-27T07:27:27.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:27 smithi043 ceph-mon[31531]: pgmap v455: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 601 KiB/s wr, 51 op/s 2024-05-27T07:27:27.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:27 smithi179 ceph-mon[35888]: pgmap v455: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 601 KiB/s wr, 51 op/s 2024-05-27T07:27:27.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:27 smithi132 ceph-mon[35283]: pgmap v455: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 601 KiB/s wr, 51 op/s 2024-05-27T07:27:28.251 INFO:tasks.workunit.client.0.smithi043.stdout: 1 76040 0.58 MB/sec execute 572 sec latency 68.474 ms 2024-05-27T07:27:29.251 INFO:tasks.workunit.client.0.smithi043.stdout: 1 76138 0.58 MB/sec execute 573 sec latency 66.963 ms 2024-05-27T07:27:29.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:29 smithi043 ceph-mon[31531]: pgmap v456: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 421 KiB/s wr, 40 op/s 2024-05-27T07:27:29.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:29 smithi179 ceph-mon[35888]: pgmap v456: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 421 KiB/s wr, 40 op/s 2024-05-27T07:27:29.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:29 smithi132 ceph-mon[35283]: pgmap v456: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 421 KiB/s wr, 40 op/s 2024-05-27T07:27:30.251 INFO:tasks.workunit.client.0.smithi043.stdout: 1 76238 0.58 MB/sec execute 574 sec latency 73.128 ms 2024-05-27T07:27:31.252 INFO:tasks.workunit.client.0.smithi043.stdout: 1 76370 0.57 MB/sec execute 575 sec latency 65.945 ms 2024-05-27T07:27:31.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:31 smithi043 ceph-mon[31531]: pgmap v457: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 421 KiB/s wr, 40 op/s 2024-05-27T07:27:31.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:31 smithi179 ceph-mon[35888]: pgmap v457: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 421 KiB/s wr, 40 op/s 2024-05-27T07:27:31.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:31 smithi132 ceph-mon[35283]: pgmap v457: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 421 KiB/s wr, 40 op/s 2024-05-27T07:27:32.252 INFO:tasks.workunit.client.0.smithi043.stdout: 1 76461 0.57 MB/sec execute 576 sec latency 58.534 ms 2024-05-27T07:27:33.252 INFO:tasks.workunit.client.0.smithi043.stdout: 1 76600 0.57 MB/sec execute 577 sec latency 67.581 ms 2024-05-27T07:27:33.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:33 smithi043 ceph-mon[31531]: pgmap v458: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 546 KiB/s wr, 55 op/s 2024-05-27T07:27:33.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:33 smithi179 ceph-mon[35888]: pgmap v458: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 546 KiB/s wr, 55 op/s 2024-05-27T07:27:33.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:33 smithi132 ceph-mon[35283]: pgmap v458: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 546 KiB/s wr, 55 op/s 2024-05-27T07:27:34.252 INFO:tasks.workunit.client.0.smithi043.stdout: 1 76721 0.57 MB/sec execute 578 sec latency 72.510 ms 2024-05-27T07:27:35.252 INFO:tasks.workunit.client.0.smithi043.stdout: 1 76866 0.57 MB/sec execute 579 sec latency 54.580 ms 2024-05-27T07:27:35.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:35 smithi043 ceph-mon[31531]: pgmap v459: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 369 KiB/s wr, 38 op/s 2024-05-27T07:27:35.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:35 smithi179 ceph-mon[35888]: pgmap v459: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 369 KiB/s wr, 38 op/s 2024-05-27T07:27:35.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:35 smithi132 ceph-mon[35283]: pgmap v459: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 369 KiB/s wr, 38 op/s 2024-05-27T07:27:36.252 INFO:tasks.workunit.client.0.smithi043.stdout: 1 76936 0.57 MB/sec execute 580 sec latency 56.603 ms 2024-05-27T07:27:37.252 INFO:tasks.workunit.client.0.smithi043.stdout: 1 77015 0.57 MB/sec execute 581 sec latency 62.296 ms 2024-05-27T07:27:37.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:37 smithi043 ceph-mon[31531]: pgmap v460: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 532 KiB/s wr, 50 op/s 2024-05-27T07:27:37.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:37 smithi179 ceph-mon[35888]: pgmap v460: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 532 KiB/s wr, 50 op/s 2024-05-27T07:27:37.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:37 smithi132 ceph-mon[35283]: pgmap v460: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 532 KiB/s wr, 50 op/s 2024-05-27T07:27:38.252 INFO:tasks.workunit.client.0.smithi043.stdout: 1 77126 0.57 MB/sec execute 582 sec latency 59.910 ms 2024-05-27T07:27:39.252 INFO:tasks.workunit.client.0.smithi043.stdout: 1 77306 0.58 MB/sec execute 583 sec latency 60.052 ms 2024-05-27T07:27:39.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:39 smithi179 ceph-mon[35888]: pgmap v461: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 288 KiB/s wr, 27 op/s 2024-05-27T07:27:39.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:39 smithi132 ceph-mon[35283]: pgmap v461: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 288 KiB/s wr, 27 op/s 2024-05-27T07:27:39.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:39 smithi043 ceph-mon[31531]: pgmap v461: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 288 KiB/s wr, 27 op/s 2024-05-27T07:27:40.253 INFO:tasks.workunit.client.0.smithi043.stdout: 1 77508 0.58 MB/sec execute 584 sec latency 51.280 ms 2024-05-27T07:27:41.253 INFO:tasks.workunit.client.0.smithi043.stdout: 1 77712 0.58 MB/sec execute 585 sec latency 52.809 ms 2024-05-27T07:27:41.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:41 smithi132 ceph-mon[35283]: pgmap v462: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 288 KiB/s wr, 27 op/s 2024-05-27T07:27:41.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:41 smithi043 ceph-mon[31531]: pgmap v462: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 288 KiB/s wr, 27 op/s 2024-05-27T07:27:41.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:41 smithi179 ceph-mon[35888]: pgmap v462: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 288 KiB/s wr, 27 op/s 2024-05-27T07:27:42.253 INFO:tasks.workunit.client.0.smithi043.stdout: 1 77827 0.58 MB/sec execute 586 sec latency 56.365 ms 2024-05-27T07:27:43.253 INFO:tasks.workunit.client.0.smithi043.stdout: 1 77855 0.58 MB/sec execute 587 sec latency 60.796 ms 2024-05-27T07:27:43.760 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:43 smithi179 ceph-mon[35888]: pgmap v463: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 600 KiB/s wr, 44 op/s 2024-05-27T07:27:43.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:43 smithi132 ceph-mon[35283]: pgmap v463: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 600 KiB/s wr, 44 op/s 2024-05-27T07:27:43.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:43 smithi043 ceph-mon[31531]: pgmap v463: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 600 KiB/s wr, 44 op/s 2024-05-27T07:27:44.253 INFO:tasks.workunit.client.0.smithi043.stdout: 1 77878 0.58 MB/sec execute 588 sec latency 51.921 ms 2024-05-27T07:27:45.253 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78073 0.58 MB/sec execute 589 sec latency 48.535 ms 2024-05-27T07:27:45.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:45 smithi132 ceph-mon[35283]: pgmap v464: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 475 KiB/s wr, 28 op/s 2024-05-27T07:27:45.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:45 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:27:45.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:45 smithi043 ceph-mon[31531]: pgmap v464: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 475 KiB/s wr, 28 op/s 2024-05-27T07:27:45.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:45 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:27:45.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:45 smithi179 ceph-mon[35888]: pgmap v464: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 475 KiB/s wr, 28 op/s 2024-05-27T07:27:45.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:45 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:27:46.253 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78159 0.58 MB/sec execute 590 sec latency 423.278 ms 2024-05-27T07:27:46.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:46 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:27:46.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:46 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:27:46.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:46 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:27:46.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:46 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:27:46.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:46 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:27:46.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:46 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:27:46.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:46 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:27:46.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:46 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:27:46.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:46 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:27:47.254 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78159 0.58 MB/sec execute 591 sec latency 1423.419 ms 2024-05-27T07:27:47.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:47 smithi132 ceph-mon[35283]: pgmap v465: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 686 KiB/s wr, 42 op/s 2024-05-27T07:27:47.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:47 smithi132 ceph-mon[35283]: pgmap v466: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 627 KiB/s wr, 36 op/s 2024-05-27T07:27:47.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:47 smithi043 ceph-mon[31531]: pgmap v465: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 686 KiB/s wr, 42 op/s 2024-05-27T07:27:47.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:47 smithi043 ceph-mon[31531]: pgmap v466: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 627 KiB/s wr, 36 op/s 2024-05-27T07:27:47.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:47 smithi179 ceph-mon[35888]: pgmap v465: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 686 KiB/s wr, 42 op/s 2024-05-27T07:27:47.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:47 smithi179 ceph-mon[35888]: pgmap v466: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 627 KiB/s wr, 36 op/s 2024-05-27T07:27:48.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:27:48.253 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78159 0.58 MB/sec execute 592 sec latency 2423.530 ms 2024-05-27T07:27:48.674 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:49.253 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78220 0.58 MB/sec execute 593 sec latency 2627.812 ms 2024-05-27T07:27:49.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:49 smithi132 ceph-mon[35283]: pgmap v467: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 627 KiB/s wr, 36 op/s 2024-05-27T07:27:49.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:49 smithi043 ceph-mon[31531]: pgmap v467: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 627 KiB/s wr, 36 op/s 2024-05-27T07:27:49.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:49 smithi179 ceph-mon[35888]: pgmap v467: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 627 KiB/s wr, 36 op/s 2024-05-27T07:27:50.254 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78226 0.58 MB/sec execute 594 sec latency 983.101 ms 2024-05-27T07:27:50.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.929+0000 7fdd34c00700 1 -- 172.21.15.43:0/4052414939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd301039d0 msgr2=0x7fdd30107f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:50.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.929+0000 7fdd34c00700 1 --2- 172.21.15.43:0/4052414939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd301039d0 0x7fdd30107f00 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009990 tx=0x7fdd1c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.929+0000 7fdd34c00700 1 -- 172.21.15.43:0/4052414939 shutdown_connections 2024-05-27T07:27:50.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.929+0000 7fdd34c00700 1 --2- 172.21.15.43:0/4052414939 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd301084d0 0x7fdd301089a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.929+0000 7fdd34c00700 1 --2- 172.21.15.43:0/4052414939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd30103080 0x7fdd30103490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.929+0000 7fdd34c00700 1 --2- 172.21.15.43:0/4052414939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd301039d0 0x7fdd30107f00 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.929+0000 7fdd34c00700 1 -- 172.21.15.43:0/4052414939 >> 172.21.15.43:0/4052414939 conn(0x7fdd300fed60 msgr2=0x7fdd30101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:50.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.929+0000 7fdd34c00700 1 -- 172.21.15.43:0/4052414939 shutdown_connections 2024-05-27T07:27:50.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.929+0000 7fdd34c00700 1 -- 172.21.15.43:0/4052414939 wait complete. 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.930+0000 7fdd34c00700 1 Processor -- start 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.931+0000 7fdd34c00700 1 -- start start 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.931+0000 7fdd34c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd30103080 0x7fdd301a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.931+0000 7fdd34c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd301039d0 0x7fdd301a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.931+0000 7fdd34c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd301084d0 0x7fdd301a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.931+0000 7fdd34c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3011a760 con 0x7fdd30103080 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.931+0000 7fdd34c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3011a5e0 con 0x7fdd301039d0 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.931+0000 7fdd34c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3011a8e0 con 0x7fdd301084d0 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.931+0000 7fdd2f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd30103080 0x7fdd301a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.931+0000 7fdd2f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd30103080 0x7fdd301a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57240/0 (socket says 172.21.15.43:57240) 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.931+0000 7fdd2f400700 1 -- 172.21.15.43:0/3497476989 learned_addr learned my addr 172.21.15.43:0/3497476989 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.932+0000 7fdd2ea00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd301039d0 0x7fdd301a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.932+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd301084d0 0x7fdd301a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.932+0000 7fdd2f400700 1 -- 172.21.15.43:0/3497476989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd301084d0 msgr2=0x7fdd301a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:50.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.932+0000 7fdd2f400700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd301084d0 0x7fdd301a5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.932+0000 7fdd2f400700 1 -- 172.21.15.43:0/3497476989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd301039d0 msgr2=0x7fdd301a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:50.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.932+0000 7fdd2f400700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd301039d0 0x7fdd301a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.932+0000 7fdd2f400700 1 -- 172.21.15.43:0/3497476989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd301a6000 con 0x7fdd30103080 2024-05-27T07:27:50.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.932+0000 7fdd2f400700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd30103080 0x7fdd301a1130 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000d950 tx=0x7fdd2000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:50.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.933+0000 7fdd27e00700 1 -- 172.21.15.43:0/3497476989 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20016070 con 0x7fdd30103080 2024-05-27T07:27:50.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.933+0000 7fdd27e00700 1 -- 172.21.15.43:0/3497476989 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd20010460 con 0x7fdd30103080 2024-05-27T07:27:50.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.933+0000 7fdd27e00700 1 -- 172.21.15.43:0/3497476989 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2000f620 con 0x7fdd30103080 2024-05-27T07:27:50.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.933+0000 7fdd34c00700 1 -- 172.21.15.43:0/3497476989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd301a62f0 con 0x7fdd30103080 2024-05-27T07:27:50.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.933+0000 7fdd34c00700 1 -- 172.21.15.43:0/3497476989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd301adc10 con 0x7fdd30103080 2024-05-27T07:27:50.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.937+0000 7fdd27e00700 1 -- 172.21.15.43:0/3497476989 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fdd20019070 con 0x7fdd30103080 2024-05-27T07:27:50.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.938+0000 7fdd44cb8b80 1 Processor -- start 2024-05-27T07:27:50.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.938+0000 7fdd44cb8b80 1 -- start start 2024-05-27T07:27:50.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5820 0x556b140c2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556b13fe3e40 con 0x556b13f36e40 2024-05-27T07:27:50.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556b13fd39e0 con 0x556b140b5410 2024-05-27T07:27:50.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556b13fd7a20 con 0x556b140b5820 2024-05-27T07:27:50.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd27e00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd140b2a70 0x7fdd140b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd2fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd27e00700 1 -- 172.21.15.43:0/3497476989 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7fdd200e25e0 con 0x7fdd30103080 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd2fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57254/0 (socket says 172.21.15.43:57254) 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd2fe00700 1 -- 172.21.15.43:0/867922075 learned_addr learned my addr 172.21.15.43:0/867922075 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd140b2a70 0x7fdd140b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd2f400700 1 --2- 172.21.15.43:0/867922075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.939+0000 7fdd2ea00700 1 --2- 172.21.15.43:0/867922075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5820 0x556b140c2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.940+0000 7fdd2fe00700 1 -- 172.21.15.43:0/867922075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5820 msgr2=0x556b140c2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.940+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/867922075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5820 0x556b140c2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.940+0000 7fdd2fe00700 1 -- 172.21.15.43:0/867922075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 msgr2=0x556b140bfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:50.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.940+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/867922075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140bfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.940+0000 7fdd2fe00700 1 -- 172.21.15.43:0/867922075 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b13f9c420 con 0x556b13f36e40 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.940+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/867922075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140b4ed0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fdd28007bc0 tx=0x7fdd28007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8d908fd1e73138 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.940+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd140b2a70 0x7fdd140b4f80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdd280060b0 tx=0x7fdd2800b000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.941+0000 7fdd24c00700 1 -- 172.21.15.43:0/867922075 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2801e070 con 0x556b13f36e40 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.941+0000 7fdd24c00700 1 -- 172.21.15.43:0/867922075 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd280099a0 con 0x556b13f36e40 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.941+0000 7fdd24c00700 1 -- 172.21.15.43:0/867922075 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2801ad70 con 0x556b13f36e40 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.942+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/867922075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 msgr2=0x556b140b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.942+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/867922075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140b4ed0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fdd28007bc0 tx=0x7fdd28007ed0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.942+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/867922075 shutdown_connections 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.942+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/867922075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5820 0x556b140c2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.942+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/867922075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140b4ed0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.942+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/867922075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140bfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.942+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/867922075 >> 172.21.15.43:0/867922075 conn(0x556b13f96fc0 msgr2=0x556b140b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:50.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.942+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/867922075 shutdown_connections 2024-05-27T07:27:50.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.942+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/867922075 wait complete. 2024-05-27T07:27:50.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.943+0000 7fdd44cb8b80 1 -- start start 2024-05-27T07:27:50.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.943+0000 7fdd44cb8b80 1 Processor -- start 2024-05-27T07:27:50.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140bf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5820 0x556b140c0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556b13ed9fa0 con 0x556b13f36e40 2024-05-27T07:27:50.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556b13f74760 con 0x556b140b5410 2024-05-27T07:27:50.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556b140c2930 con 0x556b140b5820 2024-05-27T07:27:50.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd44cb8b80 10 client.0 _ll_register_callbacks cb 0x556b13f7dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:50.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140bf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140bf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57258/0 (socket says 172.21.15.43:57258) 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2f400700 1 -- 172.21.15.43:0/3932728763 learned_addr learned my addr 172.21.15.43:0/3932728763 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5820 0x556b140c0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2ea00700 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2f400700 1 -- 172.21.15.43:0/3932728763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5820 msgr2=0x556b140c0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2f400700 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5820 0x556b140c0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2f400700 1 -- 172.21.15.43:0/3932728763 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 msgr2=0x556b140bfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2f400700 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140bfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2f400700 1 -- 172.21.15.43:0/3932728763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b13f238d0 con 0x556b13f36e40 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.944+0000 7fdd2ea00700 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140bfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.945+0000 7fdd2f400700 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140bf680 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fdd200a9190 tx=0x7fdd20072d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d89cff7ef435bd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:50.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.945+0000 7fdd19600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.945+0000 7fdd19600700 1 -- 172.21.15.43:0/3932728763 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20020040 con 0x556b13f36e40 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.945+0000 7fdd19600700 1 -- 172.21.15.43:0/3932728763 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7fdd200a96f0 con 0x556b13f36e40 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.945+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3932728763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd280099a0 con 0x556b13f36e40 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.945+0000 7fdd19600700 1 -- 172.21.15.43:0/3932728763 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fdd200dfd90 con 0x556b13f36e40 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.945+0000 7fdd44cb8b80 10 client.17439 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.945+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3932728763 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x556b13fa4060 con 0x556b13f36e40 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.946+0000 7fdd19600700 1 -- 172.21.15.43:0/3932728763 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd280099a0 con 0x556b13f36e40 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.946+0000 7fdd19600700 1 -- 172.21.15.43:0/3932728763 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7fdd200bfc10 con 0x556b13f36e40 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.946+0000 7fdd44cb8b80 10 client.17439 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.946+0000 7fdd44cb8b80 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.946+0000 7fdd44cb8b80 20 client.17439 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.946+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556b140cfc60 0x556b140d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.946+0000 7fdd44cb8b80 4 client.17439 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:50.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.946+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3932728763 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b13fe5d90 con 0x556b140cfc60 2024-05-27T07:27:50.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.947+0000 7fdd2ea00700 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556b140cfc60 0x556b140d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.948+0000 7fdd2ea00700 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556b140cfc60 0x556b140d2110 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e93efc00272c9963 server_cookie=16b9b3ba18a67aba in_seq=0 out_seq=0 2024-05-27T07:27:50.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.948+0000 7fdd19600700 10 client.17439 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:50.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.950+0000 7fdd19600700 1 -- 172.21.15.43:0/3932728763 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556b13fe5d90 con 0x556b140cfc60 2024-05-27T07:27:50.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.950+0000 7fdd19600700 10 client.17439 handle_command_reply: tid=0 2024-05-27T07:27:50.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.950+0000 7fdd44cb8b80 1 client.17439 shutdown 2024-05-27T07:27:50.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.950+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3932728763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 msgr2=0x556b140bf680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:50.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.950+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140bf680 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fdd200a9190 tx=0x7fdd20072d30 comp rx=0 tx=0).stop 2024-05-27T07:27:50.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.951+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3932728763 shutdown_connections 2024-05-27T07:27:50.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.951+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5820 0x556b140c0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.951+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556b140cfc60 0x556b140d2110 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.951+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140bf680 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.951+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/3932728763 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140bfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.951+0000 7fdd19600700 0 client.17439 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:50.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.951+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3932728763 >> 172.21.15.43:0/3932728763 conn(0x556b13f96fc0 msgr2=0x556b140afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.951+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3932728763 shutdown_connections 2024-05-27T07:27:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.951+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3932728763 wait complete. 2024-05-27T07:27:50.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.951+0000 7fdd44cb8b80 20 client.17439 trim_cache size 0 max 16384 2024-05-27T07:27:50.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.974+0000 7fdd44cb8b80 1 Processor -- start 2024-05-27T07:27:50.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.974+0000 7fdd44cb8b80 1 -- start start 2024-05-27T07:27:50.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.975+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140c1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.975+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5410 0x556b140bae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.975+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5820 0x556b140bfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.975+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301b7740 con 0x556b13f36e40 2024-05-27T07:27:50.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.975+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301b5410 con 0x556b140b5820 2024-05-27T07:27:50.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.975+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301b7070 con 0x556b140b5410 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.975+0000 7fdd2f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140c1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.975+0000 7fdd2f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140c1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57272/0 (socket says 172.21.15.43:57272) 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.975+0000 7fdd2f400700 1 -- 172.21.15.43:0/3877224831 learned_addr learned my addr 172.21.15.43:0/3877224831 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd2ea00700 1 --2- 172.21.15.43:0/3877224831 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5410 0x556b140bae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/3877224831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5820 0x556b140bfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd2f400700 1 -- 172.21.15.43:0/3877224831 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5410 msgr2=0x556b140bae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd2f400700 1 --2- 172.21.15.43:0/3877224831 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5410 0x556b140bae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd2f400700 1 -- 172.21.15.43:0/3877224831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5820 msgr2=0x556b140bfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd2f400700 1 --2- 172.21.15.43:0/3877224831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5820 0x556b140bfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd2f400700 1 -- 172.21.15.43:0/3877224831 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b140b2350 con 0x556b13f36e40 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/3877224831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5820 0x556b140bfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd2f400700 1 --2- 172.21.15.43:0/3877224831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140c1b20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fdd200a5510 tx=0x7fdd20091190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42517a61dd2e8f4e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:50.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd1be00700 1 -- 172.21.15.43:0/3877224831 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20020070 con 0x556b13f36e40 2024-05-27T07:27:50.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd1be00700 1 -- 172.21.15.43:0/3877224831 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd200ad840 con 0x556b13f36e40 2024-05-27T07:27:50.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.976+0000 7fdd1be00700 1 -- 172.21.15.43:0/3877224831 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2008f9b0 con 0x556b13f36e40 2024-05-27T07:27:50.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.977+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3877224831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 msgr2=0x556b140c1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:50.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.977+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/3877224831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140c1b20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fdd200a5510 tx=0x7fdd20091190 comp rx=0 tx=0).stop 2024-05-27T07:27:50.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.977+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3877224831 shutdown_connections 2024-05-27T07:27:50.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.977+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/3877224831 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5820 0x556b140bfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.977+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/3877224831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b13f36e40 0x556b140c1b20 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.977+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/3877224831 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b140b5410 0x556b140bae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.977+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3877224831 >> 172.21.15.43:0/3877224831 conn(0x556b13f96fc0 msgr2=0x556b140bf640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.978+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3877224831 shutdown_connections 2024-05-27T07:27:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.978+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/3877224831 wait complete. 2024-05-27T07:27:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.978+0000 7fdd44cb8b80 1 -- start start 2024-05-27T07:27:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.978+0000 7fdd44cb8b80 1 Processor -- start 2024-05-27T07:27:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.979+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b13f36e40 0x556b140c3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.979+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.979+0000 7fdd44cb8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b140b5820 0x556b140c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.979+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301b6210 con 0x556b140b5820 2024-05-27T07:27:50.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.979+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301b5ae0 con 0x556b140b5410 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.979+0000 7fdd44cb8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301b6940 con 0x556b13f36e40 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.979+0000 7fdd44cb8b80 10 client.0 _ll_register_callbacks cb 0x556b13f7dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.979+0000 7fdd2fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b140b5820 0x556b140c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.979+0000 7fdd2fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b140b5820 0x556b140c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57286/0 (socket says 172.21.15.43:57286) 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.979+0000 7fdd2fe00700 1 -- 172.21.15.43:0/727309148 learned_addr learned my addr 172.21.15.43:0/727309148 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd2f400700 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b13f36e40 0x556b140c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd2ea00700 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd2fe00700 1 -- 172.21.15.43:0/727309148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b13f36e40 msgr2=0x556b140c3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b13f36e40 0x556b140c3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd2fe00700 1 -- 172.21.15.43:0/727309148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 msgr2=0x556b140c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:50.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd2fe00700 1 -- 172.21.15.43:0/727309148 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b141397e0 con 0x556b140b5820 2024-05-27T07:27:50.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd2ea00700 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:50.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd2fe00700 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b140b5820 0x556b140c4290 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fdd28007bc0 tx=0x7fdd280092b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce97b3cd5904321 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:50.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd19600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:50.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.980+0000 7fdd19600700 1 -- 172.21.15.43:0/727309148 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2801e030 con 0x556b140b5820 2024-05-27T07:27:50.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.981+0000 7fdd19600700 1 -- 172.21.15.43:0/727309148 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7fdd2801a880 con 0x556b140b5820 2024-05-27T07:27:50.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.981+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/727309148 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd200ad840 con 0x556b140b5820 2024-05-27T07:27:50.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.981+0000 7fdd19600700 1 -- 172.21.15.43:0/727309148 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fdd28038050 con 0x556b140b5820 2024-05-27T07:27:50.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.981+0000 7fdd44cb8b80 10 client.17445 fetch_fsmap learned FSMap version 112 2024-05-27T07:27:50.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.981+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/727309148 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=4+}) -- 0x556b1413a850 con 0x556b140b5820 2024-05-27T07:27:50.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.981+0000 7fdd19600700 1 -- 172.21.15.43:0/727309148 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd200ad840 con 0x556b140b5820 2024-05-27T07:27:50.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.981+0000 7fdd19600700 1 -- 172.21.15.43:0/727309148 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 112) ==== 4804+0+0 (secure 0 0 0) 0x7fdd28033030 con 0x556b140b5820 2024-05-27T07:27:50.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.981+0000 7fdd44cb8b80 10 client.17445 fetch_fsmap finished waiting for FSMap version 112 2024-05-27T07:27:50.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.981+0000 7fdd44cb8b80 10 client.17445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:27:50.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.981+0000 7fdd44cb8b80 20 client.17445 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:50.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.982+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556b1413af50 0x556b1413d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:50.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.982+0000 7fdd44cb8b80 4 client.17445 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:27:50.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.982+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/727309148 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556b13fdf570 con 0x556b1413af50 2024-05-27T07:27:50.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.982+0000 7fdd2f400700 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556b1413af50 0x556b1413d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:50.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.983+0000 7fdd2f400700 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556b1413af50 0x556b1413d370 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa3dd24b587e2824 server_cookie=83a80a0d0bb9e159 in_seq=0 out_seq=0 2024-05-27T07:27:50.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.983+0000 7fdd19600700 10 client.17445 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:50.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd19600700 1 -- 172.21.15.43:0/727309148 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556b13fdf570 con 0x556b1413af50 2024-05-27T07:27:50.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd19600700 10 client.17445 handle_command_reply: tid=0 2024-05-27T07:27:50.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 client.17445 shutdown 2024-05-27T07:27:50.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/727309148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b140b5820 msgr2=0x556b140c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:50.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b140b5820 0x556b140c4290 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fdd28007bc0 tx=0x7fdd280092b0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/727309148 shutdown_connections 2024-05-27T07:27:50.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556b1413af50 0x556b1413d370 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556b140b5820 0x556b140c4290 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556b13f36e40 0x556b140c3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd19600700 0 client.17445 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:27:50.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 --2- 172.21.15.43:0/727309148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b140b5410 0x556b140c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/727309148 >> 172.21.15.43:0/727309148 conn(0x556b13f96fc0 msgr2=0x556b140ca9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:50.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/727309148 shutdown_connections 2024-05-27T07:27:50.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 1 -- 172.21.15.43:0/727309148 wait complete. 2024-05-27T07:27:50.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.984+0000 7fdd44cb8b80 20 client.17445 trim_cache size 0 max 16384 2024-05-27T07:27:50.979 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:27:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:27:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:27:50.980 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:27:50.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.985+0000 7fdd34c00700 1 -- 172.21.15.43:0/3497476989 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd140b2a70 msgr2=0x7fdd140b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:50.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.985+0000 7fdd34c00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd140b2a70 0x7fdd140b4f80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdd280060b0 tx=0x7fdd2800b000 comp rx=0 tx=0).stop 2024-05-27T07:27:50.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.985+0000 7fdd34c00700 1 -- 172.21.15.43:0/3497476989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd30103080 msgr2=0x7fdd301a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:50.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.985+0000 7fdd34c00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd30103080 0x7fdd301a1130 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000d950 tx=0x7fdd2000dd10 comp rx=0 tx=0).stop 2024-05-27T07:27:50.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.986+0000 7fdd34c00700 1 -- 172.21.15.43:0/3497476989 shutdown_connections 2024-05-27T07:27:50.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.986+0000 7fdd34c00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd140b2a70 0x7fdd140b4f80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.986+0000 7fdd34c00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd301084d0 0x7fdd301a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.986+0000 7fdd34c00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd30103080 0x7fdd301a1130 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.986+0000 7fdd34c00700 1 --2- 172.21.15.43:0/3497476989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd301039d0 0x7fdd301a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:50.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.986+0000 7fdd34c00700 1 -- 172.21.15.43:0/3497476989 >> 172.21.15.43:0/3497476989 conn(0x7fdd300fed60 msgr2=0x7fdd30105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:50.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.986+0000 7fdd34c00700 1 -- 172.21.15.43:0/3497476989 shutdown_connections 2024-05-27T07:27:50.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:50.986+0000 7fdd34c00700 1 -- 172.21.15.43:0/3497476989 wait complete. 2024-05-27T07:27:51.254 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78226 0.58 MB/sec execute 595 sec latency 1983.210 ms 2024-05-27T07:27:51.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:51 smithi043 ceph-mon[31531]: pgmap v468: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 627 KiB/s wr, 36 op/s 2024-05-27T07:27:51.817 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:27:51.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:27:51.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:51 smithi132 ceph-mon[35283]: pgmap v468: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 627 KiB/s wr, 36 op/s 2024-05-27T07:27:51.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:51 smithi179 ceph-mon[35888]: pgmap v468: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 627 KiB/s wr, 36 op/s 2024-05-27T07:27:52.247 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:52.254 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78226 0.58 MB/sec execute 596 sec latency 2983.299 ms 2024-05-27T07:27:53.254 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78226 0.57 MB/sec execute 597 sec latency 3983.462 ms 2024-05-27T07:27:53.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:53 smithi043 ceph-mon[31531]: pgmap v469: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 302 KiB/s wr, 25 op/s 2024-05-27T07:27:53.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:53 smithi132 ceph-mon[35283]: pgmap v469: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 302 KiB/s wr, 25 op/s 2024-05-27T07:27:53.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:53 smithi179 ceph-mon[35888]: pgmap v469: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 302 KiB/s wr, 25 op/s 2024-05-27T07:27:54.254 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78308 0.58 MB/sec execute 598 sec latency 4187.759 ms 2024-05-27T07:27:54.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.762+0000 7f833dc00700 1 -- 172.21.15.43:0/1683504274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8338107de0 msgr2=0x7f83381081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:54.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.762+0000 7f833dc00700 1 --2- 172.21.15.43:0/1683504274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8338107de0 0x7f83381081f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f83240099c0 tx=0x7f8324009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:54.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.762+0000 7f833dc00700 1 -- 172.21.15.43:0/1683504274 shutdown_connections 2024-05-27T07:27:54.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.762+0000 7f833dc00700 1 --2- 172.21.15.43:0/1683504274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8338108730 0x7f8338110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:54.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.763+0000 7f833dc00700 1 --2- 172.21.15.43:0/1683504274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83381034b0 0x7f83381038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:54.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.763+0000 7f833dc00700 1 --2- 172.21.15.43:0/1683504274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8338107de0 0x7f83381081f0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:54.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.763+0000 7f833dc00700 1 -- 172.21.15.43:0/1683504274 >> 172.21.15.43:0/1683504274 conn(0x7f83380fedb0 msgr2=0x7f83381011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:54.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.763+0000 7f833dc00700 1 -- 172.21.15.43:0/1683504274 shutdown_connections 2024-05-27T07:27:54.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.763+0000 7f833dc00700 1 -- 172.21.15.43:0/1683504274 wait complete. 2024-05-27T07:27:54.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.765+0000 7f833dc00700 1 Processor -- start 2024-05-27T07:27:54.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.765+0000 7f833dc00700 1 -- start start 2024-05-27T07:27:54.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.765+0000 7f833dc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83381034b0 0x7f833819cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:54.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.765+0000 7f833dc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8338107de0 0x7f833819d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:54.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.765+0000 7f833dc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8338108730 0x7f83381a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:54.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.765+0000 7f833dc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f833806bf00 con 0x7f8338107de0 2024-05-27T07:27:54.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.765+0000 7f833dc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f833806bd80 con 0x7f8338108730 2024-05-27T07:27:54.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.765+0000 7f833dc00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f833806c080 con 0x7f83381034b0 2024-05-27T07:27:54.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.766+0000 7f8336000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8338107de0 0x7f833819d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:54.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.766+0000 7f8336000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8338107de0 0x7f833819d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60502/0 (socket says 172.21.15.43:60502) 2024-05-27T07:27:54.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.766+0000 7f8336000700 1 -- 172.21.15.43:0/3223778669 learned_addr learned my addr 172.21.15.43:0/3223778669 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:54.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.766+0000 7f8336a00700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83381034b0 0x7f833819cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:54.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.766+0000 7f8337400700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8338108730 0x7f83381a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:54.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.766+0000 7f8336000700 1 -- 172.21.15.43:0/3223778669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83381034b0 msgr2=0x7f833819cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:54.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.766+0000 7f8336000700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83381034b0 0x7f833819cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:54.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.766+0000 7f8336000700 1 -- 172.21.15.43:0/3223778669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8338108730 msgr2=0x7f83381a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:54.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.766+0000 7f8336000700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8338108730 0x7f83381a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:54.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.766+0000 7f8336000700 1 -- 172.21.15.43:0/3223778669 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83381a1de0 con 0x7f8338107de0 2024-05-27T07:27:54.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.767+0000 7f8336000700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8338107de0 0x7f833819d4b0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f832400b580 tx=0x7f83240057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:54.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.767+0000 7f832f400700 1 -- 172.21.15.43:0/3223778669 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8324027070 con 0x7f8338107de0 2024-05-27T07:27:54.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.767+0000 7f832f400700 1 -- 172.21.15.43:0/3223778669 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f832400fd30 con 0x7f8338107de0 2024-05-27T07:27:54.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.767+0000 7f833dc00700 1 -- 172.21.15.43:0/3223778669 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83381a2070 con 0x7f8338107de0 2024-05-27T07:27:54.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.767+0000 7f832f400700 1 -- 172.21.15.43:0/3223778669 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83240173f0 con 0x7f8338107de0 2024-05-27T07:27:54.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.767+0000 7f833dc00700 1 -- 172.21.15.43:0/3223778669 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83381a2550 con 0x7f8338107de0 2024-05-27T07:27:54.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.768+0000 7f833dc00700 1 -- 172.21.15.43:0/3223778669 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83380512b0 con 0x7f8338107de0 2024-05-27T07:27:54.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.773+0000 7f832f400700 1 -- 172.21.15.43:0/3223778669 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f832402a070 con 0x7f8338107de0 2024-05-27T07:27:54.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.774+0000 7f832f400700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f831c0b29f0 0x7f831c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:54.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.775+0000 7f832f400700 1 -- 172.21.15.43:0/3223778669 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7f83240e3b40 con 0x7f8338107de0 2024-05-27T07:27:54.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.775+0000 7f8336a00700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f831c0b29f0 0x7f831c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:54.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.776+0000 7f8336a00700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f831c0b29f0 0x7f831c0b4f00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8328005950 tx=0x7f832800b410 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:54.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:54.777+0000 7f832f400700 1 -- 172.21.15.43:0/3223778669 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f832405fdf0 con 0x7f8338107de0 2024-05-27T07:27:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.072+0000 7f833dc00700 1 -- 172.21.15.43:0/3223778669 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83381a9ab0 con 0x7f8338107de0 2024-05-27T07:27:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.074+0000 7f832f400700 1 -- 172.21.15.43:0/3223778669 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+6962 (secure 0 0 0) 0x7f8324013070 con 0x7f8338107de0 2024-05-27T07:27:55.066 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:27:55.066 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:26:55.403097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34448,34484,34424,34403,24484]},"id":1}]} 2024-05-27T07:27:55.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.077+0000 7f833dc00700 1 -- 172.21.15.43:0/3223778669 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f831c0b29f0 msgr2=0x7f831c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.077+0000 7f833dc00700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f831c0b29f0 0x7f831c0b4f00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8328005950 tx=0x7f832800b410 comp rx=0 tx=0).stop 2024-05-27T07:27:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.077+0000 7f833dc00700 1 -- 172.21.15.43:0/3223778669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8338107de0 msgr2=0x7f833819d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.077+0000 7f833dc00700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8338107de0 0x7f833819d4b0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f832400b580 tx=0x7f83240057c0 comp rx=0 tx=0).stop 2024-05-27T07:27:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.078+0000 7f833dc00700 1 -- 172.21.15.43:0/3223778669 shutdown_connections 2024-05-27T07:27:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.078+0000 7f833dc00700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f831c0b29f0 0x7f831c0b4f00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.078+0000 7f833dc00700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8338108730 0x7f83381a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.078+0000 7f833dc00700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83381034b0 0x7f833819cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.078+0000 7f833dc00700 1 --2- 172.21.15.43:0/3223778669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8338107de0 0x7f833819d4b0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.078+0000 7f833dc00700 1 -- 172.21.15.43:0/3223778669 >> 172.21.15.43:0/3223778669 conn(0x7f83380fedb0 msgr2=0x7f8338109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.078+0000 7f833dc00700 1 -- 172.21.15.43:0/3223778669 shutdown_connections 2024-05-27T07:27:55.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:55.078+0000 7f833dc00700 1 -- 172.21.15.43:0/3223778669 wait complete. 2024-05-27T07:27:55.070 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 112 2024-05-27T07:27:55.254 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78362 0.57 MB/sec execute 599 sec latency 48.738 ms 2024-05-27T07:27:55.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:55 smithi132 ceph-mon[35283]: pgmap v470: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 302 KiB/s wr, 25 op/s 2024-05-27T07:27:55.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:55 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3223778669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:27:55.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:55 smithi043 ceph-mon[31531]: pgmap v470: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 302 KiB/s wr, 25 op/s 2024-05-27T07:27:55.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:55 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3223778669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:27:55.932 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:27:55.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:55 smithi179 ceph-mon[35888]: pgmap v470: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 302 KiB/s wr, 25 op/s 2024-05-27T07:27:55.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:55 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3223778669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:27:56.264 INFO:tasks.workunit.client.0.smithi043.stdout: 1 cleanup 600 sec 2024-05-27T07:27:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:27:57.255 INFO:tasks.workunit.client.0.smithi043.stdout: 1 cleanup 601 sec 2024-05-27T07:27:57.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:57 smithi043 ceph-mon[31531]: pgmap v471: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 265 KiB/s wr, 22 op/s 2024-05-27T07:27:57.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:57 smithi132 ceph-mon[35283]: pgmap v471: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 265 KiB/s wr, 22 op/s 2024-05-27T07:27:57.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:57 smithi179 ceph-mon[35888]: pgmap v471: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 265 KiB/s wr, 22 op/s 2024-05-27T07:27:58.256 INFO:tasks.workunit.client.0.smithi043.stdout: 1 cleanup 602 sec 2024-05-27T07:27:58.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.771+0000 7f220a400700 1 -- 172.21.15.43:0/652987056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22041045e0 msgr2=0x7f22041049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.771+0000 7f220a400700 1 --2- 172.21.15.43:0/652987056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22041045e0 0x7f22041049f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f21f40099c0 tx=0x7f21f4009cd0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.771+0000 7f220a400700 1 -- 172.21.15.43:0/652987056 shutdown_connections 2024-05-27T07:27:58.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.771+0000 7f220a400700 1 --2- 172.21.15.43:0/652987056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2204109860 0x7f22041111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.771+0000 7f220a400700 1 --2- 172.21.15.43:0/652987056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22041045e0 0x7f22041049f0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.771+0000 7f220a400700 1 --2- 172.21.15.43:0/652987056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2204108f10 0x7f2204109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.771+0000 7f220a400700 1 -- 172.21.15.43:0/652987056 >> 172.21.15.43:0/652987056 conn(0x7f2204100030 msgr2=0x7f2204102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.771+0000 7f220a400700 1 -- 172.21.15.43:0/652987056 shutdown_connections 2024-05-27T07:27:58.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.771+0000 7f220a400700 1 -- 172.21.15.43:0/652987056 wait complete. 2024-05-27T07:27:58.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.773+0000 7f220a400700 1 Processor -- start 2024-05-27T07:27:58.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.773+0000 7f220a400700 1 -- start start 2024-05-27T07:27:58.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f220a400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22041045e0 0x7f220419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f220a400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2204108f10 0x7f220419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f220a400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2204109860 0x7f220419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f220a400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2204114390 con 0x7f22041045e0 2024-05-27T07:27:58.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f220a400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2204114210 con 0x7f2204108f10 2024-05-27T07:27:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f220a400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2204114510 con 0x7f2204109860 2024-05-27T07:27:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f2203400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22041045e0 0x7f220419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f2203400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22041045e0 0x7f220419ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60522/0 (socket says 172.21.15.43:60522) 2024-05-27T07:27:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f2203400700 1 -- 172.21.15.43:0/65682564 learned_addr learned my addr 172.21.15.43:0/65682564 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:58.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f2203e00700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2204109860 0x7f220419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.774+0000 7f2202a00700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2204108f10 0x7f220419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.775+0000 7f2203400700 1 -- 172.21.15.43:0/65682564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2204109860 msgr2=0x7f220419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.775+0000 7f2203400700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2204109860 0x7f220419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.775+0000 7f2203400700 1 -- 172.21.15.43:0/65682564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2204108f10 msgr2=0x7f220419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.775+0000 7f2203400700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2204108f10 0x7f220419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.775+0000 7f2203400700 1 -- 172.21.15.43:0/65682564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220419fc80 con 0x7f22041045e0 2024-05-27T07:27:58.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.775+0000 7f2203400700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22041045e0 0x7f220419ae60 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f21f4000c00 tx=0x7f21f400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.775+0000 7f21fbe00700 1 -- 172.21.15.43:0/65682564 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4027070 con 0x7f22041045e0 2024-05-27T07:27:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.775+0000 7f220a400700 1 -- 172.21.15.43:0/65682564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220419ff10 con 0x7f22041045e0 2024-05-27T07:27:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.775+0000 7f21fbe00700 1 -- 172.21.15.43:0/65682564 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21f40056d0 con 0x7f22041045e0 2024-05-27T07:27:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.775+0000 7f21fbe00700 1 -- 172.21.15.43:0/65682564 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f40173f0 con 0x7f22041045e0 2024-05-27T07:27:58.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.776+0000 7f220a400700 1 -- 172.21.15.43:0/65682564 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22041a0440 con 0x7f22041045e0 2024-05-27T07:27:58.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.780+0000 7f21fbe00700 1 -- 172.21.15.43:0/65682564 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f21f402a070 con 0x7f22041045e0 2024-05-27T07:27:58.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f2218f4ab80 1 Processor -- start 2024-05-27T07:27:58.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f2218f4ab80 1 -- start start 2024-05-27T07:27:58.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f21fbe00700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21e80b2a70 0x7f21e80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f21fbe00700 1 -- 172.21.15.43:0/65682564 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7f21f40e3900 con 0x7f22041045e0 2024-05-27T07:27:58.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c0386b09f0 0x55c0387433d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0385c4e40 0x55c038743910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f2203e00700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21e80b2a70 0x7f21e80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c038752510 0x55c038754930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c038671e40 con 0x55c0385c4e40 2024-05-27T07:27:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c0386619e0 con 0x55c038752510 2024-05-27T07:27:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.782+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c038665a20 con 0x55c0386b09f0 2024-05-27T07:27:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.783+0000 7f2202a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0385c4e40 0x55c038743910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.783+0000 7f2202a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0385c4e40 0x55c038743910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60536/0 (socket says 172.21.15.43:60536) 2024-05-27T07:27:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.783+0000 7f2202a00700 1 -- 172.21.15.43:0/4186694237 learned_addr learned my addr 172.21.15.43:0/4186694237 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.783+0000 7f2203e00700 1 --2- 172.21.15.43:0/4186694237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c038752510 0x55c038754930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.783+0000 7f2203400700 1 --2- 172.21.15.43:0/4186694237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c0386b09f0 0x55c0387433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.783+0000 7f2202a00700 1 -- 172.21.15.43:0/4186694237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c0386b09f0 msgr2=0x55c0387433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.783+0000 7f2202a00700 1 --2- 172.21.15.43:0/4186694237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c0386b09f0 0x55c0387433d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.783+0000 7f2202a00700 1 -- 172.21.15.43:0/4186694237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c038752510 msgr2=0x55c038754930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.783+0000 7f2202a00700 1 --2- 172.21.15.43:0/4186694237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c038752510 0x55c038754930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.783+0000 7f2202a00700 1 -- 172.21.15.43:0/4186694237 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c03862a420 con 0x55c0385c4e40 2024-05-27T07:27:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.784+0000 7f2203e00700 1 --2- 172.21.15.43:0/4186694237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c038752510 0x55c038754930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.784+0000 7f2203e00700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21e80b2a70 0x7f21e80b4f80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0077d0 tx=0x7f21fc00e4f0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.784+0000 7f2202a00700 1 --2- 172.21.15.43:0/4186694237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0385c4e40 0x55c038743910 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f21f000cc10 tx=0x7f21f000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbf912d36ab5354 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.784+0000 7f21f8c00700 1 -- 172.21.15.43:0/4186694237 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f000edc0 con 0x55c0385c4e40 2024-05-27T07:27:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.784+0000 7f21f8c00700 1 -- 172.21.15.43:0/4186694237 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21f0016070 con 0x55c0385c4e40 2024-05-27T07:27:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.784+0000 7f21f8c00700 1 -- 172.21.15.43:0/4186694237 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0010430 con 0x55c0385c4e40 2024-05-27T07:27:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.785+0000 7f2218f4ab80 1 -- 172.21.15.43:0/4186694237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0385c4e40 msgr2=0x55c038743910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.785+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/4186694237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0385c4e40 0x55c038743910 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f21f000cc10 tx=0x7f21f000cf20 comp rx=0 tx=0).stop 2024-05-27T07:27:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.785+0000 7f2218f4ab80 1 -- 172.21.15.43:0/4186694237 shutdown_connections 2024-05-27T07:27:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.785+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/4186694237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c038752510 0x55c038754930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.785+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/4186694237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c0386b09f0 0x55c0387433d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.785+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/4186694237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0385c4e40 0x55c038743910 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.785+0000 7f2218f4ab80 1 -- 172.21.15.43:0/4186694237 >> 172.21.15.43:0/4186694237 conn(0x55c03865e390 msgr2=0x55c03873f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.786+0000 7f2218f4ab80 1 -- 172.21.15.43:0/4186694237 shutdown_connections 2024-05-27T07:27:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.786+0000 7f2218f4ab80 1 -- 172.21.15.43:0/4186694237 wait complete. 2024-05-27T07:27:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.786+0000 7f2218f4ab80 1 -- start start 2024-05-27T07:27:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.786+0000 7f2218f4ab80 1 Processor -- start 2024-05-27T07:27:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.787+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0385c4e40 0x55c03874f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.787+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03874fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.787+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c038752510 0x55c03874ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.787+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c038567fa0 con 0x55c0386b09f0 2024-05-27T07:27:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.787+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c038602760 con 0x55c0385c4e40 2024-05-27T07:27:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.787+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c038754f60 con 0x55c038752510 2024-05-27T07:27:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.787+0000 7f2218f4ab80 10 client.0 _ll_register_callbacks cb 0x55c03860bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.787+0000 7f2202a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03874fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f2202a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03874fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60546/0 (socket says 172.21.15.43:60546) 2024-05-27T07:27:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f2202a00700 1 -- 172.21.15.43:0/1317820181 learned_addr learned my addr 172.21.15.43:0/1317820181 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:58.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f2203400700 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0385c4e40 0x55c03874f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f2203e00700 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c038752510 0x55c03874ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f2202a00700 1 -- 172.21.15.43:0/1317820181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c038752510 msgr2=0x55c03874ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f2202a00700 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c038752510 0x55c03874ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f2202a00700 1 -- 172.21.15.43:0/1317820181 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0385c4e40 msgr2=0x55c03874f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f2202a00700 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0385c4e40 0x55c03874f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f2202a00700 1 -- 172.21.15.43:0/1317820181 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0385b18d0 con 0x55c0386b09f0 2024-05-27T07:27:58.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f2202a00700 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03874fa80 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f21f00038e0 tx=0x7f21f0003950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76bbd81ff1e69b49 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.788+0000 7f21ed600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:58.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.789+0000 7f21ed600700 1 -- 172.21.15.43:0/1317820181 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0029060 con 0x55c0386b09f0 2024-05-27T07:27:58.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.789+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1317820181 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21f0016070 con 0x55c0386b09f0 2024-05-27T07:27:58.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.789+0000 7f21ed600700 1 -- 172.21.15.43:0/1317820181 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7f21f0005300 con 0x55c0386b09f0 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.789+0000 7f21ed600700 1 -- 172.21.15.43:0/1317820181 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7f21f0030050 con 0x55c0386b09f0 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.789+0000 7f2218f4ab80 10 client.17463 fetch_fsmap learned FSMap version 113 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.789+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1317820181 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x55c038632060 con 0x55c0386b09f0 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.790+0000 7f21ed600700 1 -- 172.21.15.43:0/1317820181 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21f0016070 con 0x55c0386b09f0 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.790+0000 7f21ed600700 1 -- 172.21.15.43:0/1317820181 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 113) ==== 4808+0+0 (secure 0 0 0) 0x7f21f0003db0 con 0x55c0386b09f0 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.790+0000 7f2218f4ab80 10 client.17463 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.790+0000 7f2218f4ab80 10 client.17463 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.790+0000 7f2218f4ab80 20 client.17463 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.790+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55c038762060 0x55c038764530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.790+0000 7f2218f4ab80 4 client.17463 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.790+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1317820181 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c038673d90 con 0x55c038762060 2024-05-27T07:27:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.791+0000 7f2203400700 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55c038762060 0x55c038764530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.792+0000 7f2203400700 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55c038762060 0x55c038764530 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a777a27520b98dca server_cookie=349fe6dc895c4a84 in_seq=0 out_seq=0 2024-05-27T07:27:58.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.792+0000 7f21ed600700 10 client.17463 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:27:58.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f21ed600700 1 -- 172.21.15.43:0/1317820181 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c038673d90 con 0x55c038762060 2024-05-27T07:27:58.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f21ed600700 10 client.17463 handle_command_reply: tid=0 2024-05-27T07:27:58.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f2218f4ab80 1 client.17463 shutdown 2024-05-27T07:27:58.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1317820181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 msgr2=0x55c03874fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03874fa80 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f21f00038e0 tx=0x7f21f0003950 comp rx=0 tx=0).stop 2024-05-27T07:27:58.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1317820181 shutdown_connections 2024-05-27T07:27:58.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55c038762060 0x55c038764530 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c038752510 0x55c03874ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0385c4e40 0x55c03874f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1317820181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03874fa80 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1317820181 >> 172.21.15.43:0/1317820181 conn(0x55c03865e390 msgr2=0x55c0387433d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:58.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.795+0000 7f21ed600700 0 client.17463 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:27:58.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.796+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1317820181 shutdown_connections 2024-05-27T07:27:58.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.796+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1317820181 wait complete. 2024-05-27T07:27:58.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.796+0000 7f2218f4ab80 20 client.17463 trim_cache size 0 max 16384 2024-05-27T07:27:58.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.815+0000 7f2218f4ab80 1 Processor -- start 2024-05-27T07:27:58.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.815+0000 7f2218f4ab80 1 -- start start 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.815+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0385c4e40 0x55c03873f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.815+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03875e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.815+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c038752510 0x55c038753930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.815+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f22041aeef0 con 0x55c0386b09f0 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.815+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f22041acb90 con 0x55c0385c4e40 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.815+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f22041ae820 con 0x55c038752510 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.816+0000 7f2202a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03875e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.816+0000 7f2202a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03875e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60548/0 (socket says 172.21.15.43:60548) 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.816+0000 7f2202a00700 1 -- 172.21.15.43:0/1892249841 learned_addr learned my addr 172.21.15.43:0/1892249841 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.816+0000 7f2203400700 1 --2- 172.21.15.43:0/1892249841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0385c4e40 0x55c03873f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.816+0000 7f2203e00700 1 --2- 172.21.15.43:0/1892249841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c038752510 0x55c038753930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.816+0000 7f2202a00700 1 -- 172.21.15.43:0/1892249841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c038752510 msgr2=0x55c038753930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.816+0000 7f2202a00700 1 --2- 172.21.15.43:0/1892249841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c038752510 0x55c038753930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.816+0000 7f2202a00700 1 -- 172.21.15.43:0/1892249841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0385c4e40 msgr2=0x55c03873f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.816+0000 7f2202a00700 1 --2- 172.21.15.43:0/1892249841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0385c4e40 0x55c03873f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.816+0000 7f2202a00700 1 -- 172.21.15.43:0/1892249841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c038752340 con 0x55c0386b09f0 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.817+0000 7f2202a00700 1 --2- 172.21.15.43:0/1892249841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03875e000 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f21f00038b0 tx=0x7f21f0004650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e770d34122bb1353 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.817+0000 7f21efe00700 1 -- 172.21.15.43:0/1892249841 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0029070 con 0x55c0386b09f0 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.817+0000 7f21efe00700 1 -- 172.21.15.43:0/1892249841 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21f0016070 con 0x55c0386b09f0 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.817+0000 7f21efe00700 1 -- 172.21.15.43:0/1892249841 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0003e90 con 0x55c0386b09f0 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.818+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1892249841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 msgr2=0x55c03875e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.818+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1892249841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03875e000 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f21f00038b0 tx=0x7f21f0004650 comp rx=0 tx=0).stop 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.818+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1892249841 shutdown_connections 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.818+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1892249841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c038752510 0x55c038753930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.818+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1892249841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0385c4e40 0x55c03873f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.818+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1892249841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c0386b09f0 0x55c03875e000 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.818+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1892249841 >> 172.21.15.43:0/1892249841 conn(0x55c03865e390 msgr2=0x55c03873ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:58.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.818+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1892249841 shutdown_connections 2024-05-27T07:27:58.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.818+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1892249841 wait complete. 2024-05-27T07:27:58.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.819+0000 7f2218f4ab80 1 -- start start 2024-05-27T07:27:58.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.819+0000 7f2218f4ab80 1 Processor -- start 2024-05-27T07:27:58.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c0385c4e40 0x55c038754480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0386b09f0 0x55c038752920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2218f4ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c038752510 0x55c038752e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f22041ad9c0 con 0x55c038752510 2024-05-27T07:27:58.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f22041ad290 con 0x55c0386b09f0 2024-05-27T07:27:58.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2218f4ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f22041ae0f0 con 0x55c0385c4e40 2024-05-27T07:27:58.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2218f4ab80 10 client.0 _ll_register_callbacks cb 0x55c03860bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:27:58.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2203e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c038752510 0x55c038752e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2203e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c038752510 0x55c038752e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60556/0 (socket says 172.21.15.43:60556) 2024-05-27T07:27:58.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2203e00700 1 -- 172.21.15.43:0/1058855780 learned_addr learned my addr 172.21.15.43:0/1058855780 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:27:58.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2203400700 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c0385c4e40 0x55c038754480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.820+0000 7f2202a00700 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0386b09f0 0x55c038752920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.821+0000 7f2203e00700 1 -- 172.21.15.43:0/1058855780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c0385c4e40 msgr2=0x55c038754480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.821+0000 7f2203e00700 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c0385c4e40 0x55c038754480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.821+0000 7f2203e00700 1 -- 172.21.15.43:0/1058855780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0386b09f0 msgr2=0x55c038752920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:27:58.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.821+0000 7f2203e00700 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0386b09f0 0x55c038752920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.821+0000 7f2203e00700 1 -- 172.21.15.43:0/1058855780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0387409c0 con 0x55c038752510 2024-05-27T07:27:58.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.821+0000 7f2202a00700 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0386b09f0 0x55c038752920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:27:58.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.821+0000 7f2203e00700 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c038752510 0x55c038752e60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009010 tx=0x7f21fc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23cf65757b111e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:27:58.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.821+0000 7f21ed600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:27:58.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.822+0000 7f21ed600700 1 -- 172.21.15.43:0/1058855780 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc026030 con 0x55c038752510 2024-05-27T07:27:58.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.822+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1058855780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21f0016070 con 0x55c038752510 2024-05-27T07:27:58.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.822+0000 7f21ed600700 1 -- 172.21.15.43:0/1058855780 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9496+0+0 (secure 0 0 0) 0x7f21fc010040 con 0x55c038752510 2024-05-27T07:27:58.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.822+0000 7f21ed600700 1 -- 172.21.15.43:0/1058855780 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7f21fc033050 con 0x55c038752510 2024-05-27T07:27:58.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.822+0000 7f2218f4ab80 10 client.17469 fetch_fsmap learned FSMap version 113 2024-05-27T07:27:58.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.822+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1058855780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x55c038759fc0 con 0x55c038752510 2024-05-27T07:27:58.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.822+0000 7f21ed600700 1 -- 172.21.15.43:0/1058855780 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21f0016070 con 0x55c038752510 2024-05-27T07:27:58.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.822+0000 7f21ed600700 1 -- 172.21.15.43:0/1058855780 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 113) ==== 4808+0+0 (secure 0 0 0) 0x7f21fc021040 con 0x55c038752510 2024-05-27T07:27:58.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.822+0000 7f2218f4ab80 10 client.17469 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T07:27:58.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.822+0000 7f2218f4ab80 10 client.17469 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:27:58.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.823+0000 7f2218f4ab80 20 client.17469 populate_metadata read hostname 'smithi043' 2024-05-27T07:27:58.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.823+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55c03875a780 0x55c0387cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:27:58.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.823+0000 7f2218f4ab80 4 client.17469 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:27:58.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.823+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1058855780 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c03866d570 con 0x55c03875a780 2024-05-27T07:27:58.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.823+0000 7f2203400700 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55c03875a780 0x55c0387cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:27:58.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.824+0000 7f2203400700 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55c03875a780 0x55c0387cec10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e6469a9f318bcd4d server_cookie=8ac1748f9a9379da in_seq=0 out_seq=0 2024-05-27T07:27:58.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.824+0000 7f21ed600700 10 client.17469 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:27:58.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f21ed600700 1 -- 172.21.15.43:0/1058855780 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c03866d570 con 0x55c03875a780 2024-05-27T07:27:58.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f21ed600700 10 client.17469 handle_command_reply: tid=0 2024-05-27T07:27:58.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f2218f4ab80 1 client.17469 shutdown 2024-05-27T07:27:58.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1058855780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c038752510 msgr2=0x55c038752e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c038752510 0x55c038752e60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009010 tx=0x7f21fc004b40 comp rx=0 tx=0).stop 2024-05-27T07:27:58.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1058855780 shutdown_connections 2024-05-27T07:27:58.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55c03875a780 0x55c0387cec10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c038752510 0x55c038752e60 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c0385c4e40 0x55c038754480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f2218f4ab80 1 --2- 172.21.15.43:0/1058855780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c0386b09f0 0x55c038752920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.825+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1058855780 >> 172.21.15.43:0/1058855780 conn(0x55c03865e390 msgr2=0x55c03874a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:58.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.826+0000 7f21ed600700 0 client.17469 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:27:58.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.826+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1058855780 shutdown_connections 2024-05-27T07:27:58.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.826+0000 7f2218f4ab80 1 -- 172.21.15.43:0/1058855780 wait complete. 2024-05-27T07:27:58.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.826+0000 7f2218f4ab80 20 client.17469 trim_cache size 0 max 16384 2024-05-27T07:27:58.818 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:27:58.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.826+0000 7f220a400700 1 -- 172.21.15.43:0/65682564 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21e80b2a70 msgr2=0x7f21e80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21e80b2a70 0x7f21e80b4f80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0077d0 tx=0x7f21fc00e4f0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 -- 172.21.15.43:0/65682564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22041045e0 msgr2=0x7f220419ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:27:58.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22041045e0 0x7f220419ae60 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f21f4000c00 tx=0x7f21f400f740 comp rx=0 tx=0).stop 2024-05-27T07:27:58.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 -- 172.21.15.43:0/65682564 shutdown_connections 2024-05-27T07:27:58.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f21e80b2a70 0x7f21e80b4f80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2204109860 0x7f220419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22041045e0 0x7f220419ae60 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 --2- 172.21.15.43:0/65682564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2204108f10 0x7f220419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:27:58.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 -- 172.21.15.43:0/65682564 >> 172.21.15.43:0/65682564 conn(0x7f2204100030 msgr2=0x7f2204102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:27:58.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 -- 172.21.15.43:0/65682564 shutdown_connections 2024-05-27T07:27:58.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:27:58.827+0000 7f220a400700 1 -- 172.21.15.43:0/65682564 wait complete. 2024-05-27T07:27:59.255 INFO:tasks.workunit.client.0.smithi043.stdout: 1 cleanup 603 sec 2024-05-27T07:27:59.653 INFO:tasks.workunit.client.0.smithi043.stdout: 0 cleanup 603 sec 2024-05-27T07:27:59.653 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-05-27T07:27:59.653 INFO:tasks.workunit.client.0.smithi043.stdout: Operation Count AvgLat MaxLat 2024-05-27T07:27:59.653 INFO:tasks.workunit.client.0.smithi043.stdout: ---------------------------------------- 2024-05-27T07:27:59.653 INFO:tasks.workunit.client.0.smithi043.stdout: NTCreateX 10973 16.170 85.366 2024-05-27T07:27:59.653 INFO:tasks.workunit.client.0.smithi043.stdout: Close 8092 0.215 4.075 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: Rename 468 104.387 4187.199 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: Unlink 2191 16.499 327.464 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: Qpathinfo 9943 13.293 3908.546 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: Qfileinfo 1746 0.470 2.083 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: Qfsinfo 1832 1.704 25.544 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: Sfileinfo 900 6.735 723.244 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: Find 3854 46.780 343.291 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: WriteX 5456 0.939 10.450 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: ReadX 17332 0.013 0.687 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: LockX 36 2.064 3.406 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: UnlockX 36 1.986 2.947 2024-05-27T07:27:59.654 INFO:tasks.workunit.client.0.smithi043.stdout: Flush 774 9.172 31.319 2024-05-27T07:27:59.655 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-05-27T07:27:59.655 INFO:tasks.workunit.client.0.smithi043.stdout:Throughput 0.574165 MB/sec 1 clients 1 procs max_latency=4187.759 ms 2024-05-27T07:27:59.655 INFO:tasks.workunit.client.0.smithi043.stderr:+ dbench 10 2024-05-27T07:27:59.658 INFO:tasks.workunit.client.0.smithi043.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-27T07:27:59.658 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-05-27T07:27:59.658 INFO:tasks.workunit.client.0.smithi043.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-27T07:27:59.662 INFO:tasks.workunit.client.0.smithi043.stdout:9 of 10 processes prepared for launch 0 sec 2024-05-27T07:27:59.679 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:27:59.763 INFO:tasks.workunit.client.0.smithi043.stdout:10 of 10 processes prepared for launch 0 sec 2024-05-27T07:27:59.763 INFO:tasks.workunit.client.0.smithi043.stdout:releasing clients 2024-05-27T07:27:59.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:59 smithi132 ceph-mon[35283]: pgmap v472: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 220 KiB/s wr, 18 op/s 2024-05-27T07:27:59.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:27:59 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:27:59.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:59 smithi043 ceph-mon[31531]: pgmap v472: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 220 KiB/s wr, 18 op/s 2024-05-27T07:27:59.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:27:59 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:27:59.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:59 smithi179 ceph-mon[35888]: pgmap v472: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 220 KiB/s wr, 18 op/s 2024-05-27T07:27:59.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:27:59 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:28:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:00.763 INFO:tasks.workunit.client.0.smithi043.stdout: 10 171 70.11 MB/sec warmup 1 sec latency 70.812 ms 2024-05-27T07:28:01.549 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:01 smithi043 ceph-mon[31531]: pgmap v473: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 220 KiB/s wr, 18 op/s 2024-05-27T07:28:01.763 INFO:tasks.workunit.client.0.smithi043.stdout: 10 289 53.55 MB/sec warmup 2 sec latency 217.654 ms 2024-05-27T07:28:01.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:01 smithi132 ceph-mon[35283]: pgmap v473: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 220 KiB/s wr, 18 op/s 2024-05-27T07:28:01.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:01 smithi179 ceph-mon[35888]: pgmap v473: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 220 KiB/s wr, 18 op/s 2024-05-27T07:28:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.573+0000 7f9aff400700 1 -- 172.21.15.43:0/2623558961 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae800bab0 con 0x7f9b00105bd0 2024-05-27T07:28:02.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.579+0000 7f9b05600700 1 -- 172.21.15.43:0/2623558961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b00105bd0 msgr2=0x7f9b0010e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:02.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.579+0000 7f9b05600700 1 --2- 172.21.15.43:0/2623558961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b00105bd0 0x7f9b0010e120 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8009990 tx=0x7f9ae8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.580+0000 7f9b05600700 1 -- 172.21.15.43:0/2623558961 shutdown_connections 2024-05-27T07:28:02.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.580+0000 7f9b05600700 1 --2- 172.21.15.43:0/2623558961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b0010e6f0 0x7f9b00110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.580+0000 7f9b05600700 1 --2- 172.21.15.43:0/2623558961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b00105280 0x7f9b00105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.580+0000 7f9b05600700 1 --2- 172.21.15.43:0/2623558961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b00105bd0 0x7f9b0010e120 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.580+0000 7f9b05600700 1 -- 172.21.15.43:0/2623558961 >> 172.21.15.43:0/2623558961 conn(0x7f9b00078b40 msgr2=0x7f9b00078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:02.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.581+0000 7f9b05600700 1 -- 172.21.15.43:0/2623558961 shutdown_connections 2024-05-27T07:28:02.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.582+0000 7f9b05600700 1 -- 172.21.15.43:0/2623558961 wait complete. 2024-05-27T07:28:02.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.584+0000 7f9b05600700 1 Processor -- start 2024-05-27T07:28:02.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.584+0000 7f9b05600700 1 -- start start 2024-05-27T07:28:02.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.585+0000 7f9b05600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b00105280 0x7f9b001a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.586+0000 7f9affe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b00105280 0x7f9b001a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.586+0000 7f9affe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b00105280 0x7f9b001a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60570/0 (socket says 172.21.15.43:60570) 2024-05-27T07:28:02.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.586+0000 7f9b05600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b00105bd0 0x7f9b001a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.586+0000 7f9b05600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b0010e6f0 0x7f9b0019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.587+0000 7f9affe00700 1 -- 172.21.15.43:0/1529895422 learned_addr learned my addr 172.21.15.43:0/1529895422 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:02.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.587+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00116330 con 0x7f9b00105280 2024-05-27T07:28:02.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.587+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001161b0 con 0x7f9b0010e6f0 2024-05-27T07:28:02.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.587+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001164b0 con 0x7f9b00105bd0 2024-05-27T07:28:02.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.588+0000 7f9b04c00700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b0010e6f0 0x7f9b0019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.588+0000 7f9affe00700 1 -- 172.21.15.43:0/1529895422 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b00105bd0 msgr2=0x7f9b001a2ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:28:02.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.588+0000 7f9affe00700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b00105bd0 0x7f9b001a2ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.588+0000 7f9affe00700 1 -- 172.21.15.43:0/1529895422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b0010e6f0 msgr2=0x7f9b0019cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:02.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.588+0000 7f9affe00700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b0010e6f0 0x7f9b0019cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.588+0000 7f9affe00700 1 -- 172.21.15.43:0/1529895422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0019d1d0 con 0x7f9b00105280 2024-05-27T07:28:02.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.590+0000 7f9affe00700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b00105280 0x7f9b001a29a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f9af0007e20 tx=0x7f9af000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:02.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.592+0000 7f9afd600700 1 -- 172.21.15.43:0/1529895422 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af0016020 con 0x7f9b00105280 2024-05-27T07:28:02.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.592+0000 7f9afd600700 1 -- 172.21.15.43:0/1529895422 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9af000ae00 con 0x7f9b00105280 2024-05-27T07:28:02.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.592+0000 7f9afd600700 1 -- 172.21.15.43:0/1529895422 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af0011670 con 0x7f9b00105280 2024-05-27T07:28:02.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.592+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b0019d4c0 con 0x7f9b00105280 2024-05-27T07:28:02.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.593+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b001a9750 con 0x7f9b00105280 2024-05-27T07:28:02.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.603+0000 7f9b156fcb80 1 Processor -- start 2024-05-27T07:28:02.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.603+0000 7f9b156fcb80 1 -- start start 2024-05-27T07:28:02.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.604+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7d8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.605+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7d94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.605+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.605+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ba6d6fee40 con 0x55ba6d651e40 2024-05-27T07:28:02.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.605+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ba6d6ee9e0 con 0x55ba6d7e3ac0 2024-05-27T07:28:02.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.606+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ba6d6f2a20 con 0x55ba6d73d9f0 2024-05-27T07:28:02.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.606+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7d8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.606+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7d8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:58826/0 (socket says 172.21.15.43:58826) 2024-05-27T07:28:02.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.606+0000 7f9b04c00700 1 -- 172.21.15.43:0/1831568232 learned_addr learned my addr 172.21.15.43:0/1831568232 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:02.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.606+0000 7f9affe00700 1 --2- 172.21.15.43:0/1831568232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7d94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.607+0000 7f9b04c00700 1 -- 172.21.15.43:0/1831568232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d7e3ac0 msgr2=0x55ba6d7e5ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:28:02.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.607+0000 7f9b04c00700 1 --2- 172.21.15.43:0/1831568232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e5ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.607+0000 7f9b04c00700 1 -- 172.21.15.43:0/1831568232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 msgr2=0x55ba6d7d94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:02.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.607+0000 7f9b04c00700 1 --2- 172.21.15.43:0/1831568232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7d94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.607+0000 7f9b04c00700 1 -- 172.21.15.43:0/1831568232 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba6d6b7420 con 0x55ba6d73d9f0 2024-05-27T07:28:02.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.607+0000 7f9affe00700 1 --2- 172.21.15.43:0/1831568232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7d94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:02.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.608+0000 7f9b04c00700 1 --2- 172.21.15.43:0/1831568232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7d8f70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009c80 tx=0x7f9af8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=182c7c1f40605a0e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:02.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.609+0000 7f9af5600700 1 -- 172.21.15.43:0/1831568232 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8010be0 con 0x55ba6d73d9f0 2024-05-27T07:28:02.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.609+0000 7f9af5600700 1 -- 172.21.15.43:0/1831568232 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9af8010d80 con 0x55ba6d73d9f0 2024-05-27T07:28:02.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.609+0000 7f9af5600700 1 -- 172.21.15.43:0/1831568232 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8011720 con 0x55ba6d73d9f0 2024-05-27T07:28:02.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.611+0000 7f9b156fcb80 1 -- 172.21.15.43:0/1831568232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 msgr2=0x55ba6d7d8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:02.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.611+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/1831568232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7d8f70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009c80 tx=0x7f9af8009f90 comp rx=0 tx=0).stop 2024-05-27T07:28:02.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.611+0000 7f9b156fcb80 1 -- 172.21.15.43:0/1831568232 shutdown_connections 2024-05-27T07:28:02.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.611+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/1831568232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.611+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/1831568232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7d8f70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.611+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/1831568232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7d94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.611+0000 7f9b156fcb80 1 -- 172.21.15.43:0/1831568232 >> 172.21.15.43:0/1831568232 conn(0x55ba6d6eb390 msgr2=0x55ba6d7cc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:02.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.615+0000 7f9afd600700 1 -- 172.21.15.43:0/1529895422 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9af0019020 con 0x7f9b00105280 2024-05-27T07:28:02.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.616+0000 7f9b156fcb80 1 -- 172.21.15.43:0/1831568232 shutdown_connections 2024-05-27T07:28:02.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.616+0000 7f9b156fcb80 1 -- 172.21.15.43:0/1831568232 wait complete. 2024-05-27T07:28:02.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.618+0000 7f9b156fcb80 1 -- start start 2024-05-27T07:28:02.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.618+0000 7f9b156fcb80 1 Processor -- start 2024-05-27T07:28:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.621+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7e0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.621+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7e1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.621+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.621+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ba6d5f4fa0 con 0x55ba6d651e40 2024-05-27T07:28:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.621+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ba6d68f760 con 0x55ba6d73d9f0 2024-05-27T07:28:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.621+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ba6d7e65d0 con 0x55ba6d7e3ac0 2024-05-27T07:28:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.621+0000 7f9b156fcb80 10 client.0 _ll_register_callbacks cb 0x55ba6d698c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:02.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.621+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7e0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.621+0000 7f9afd600700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9ae40b2980 0x7f9ae40b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.622+0000 7f9af7400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.622+0000 7f9afd600700 1 -- 172.21.15.43:0/1529895422 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 9584+0+0 (secure 0 0 0) 0x7f9af00e2af0 con 0x7f9b00105280 2024-05-27T07:28:02.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.622+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7e0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60586/0 (socket says 172.21.15.43:60586) 2024-05-27T07:28:02.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.622+0000 7f9b04c00700 1 -- 172.21.15.43:0/62269647 learned_addr learned my addr 172.21.15.43:0/62269647 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:02.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.623+0000 7f9affe00700 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7e1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.623+0000 7f9b04c00700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9ae40b2980 0x7f9ae40b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.623+0000 7f9b04c00700 1 -- 172.21.15.43:0/62269647 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d7e3ac0 msgr2=0x55ba6d7e1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:02.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.623+0000 7f9b04c00700 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.623+0000 7f9b04c00700 1 -- 172.21.15.43:0/62269647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d73d9f0 msgr2=0x55ba6d7e1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:02.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.623+0000 7f9b04c00700 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7e1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.623+0000 7f9b04c00700 1 -- 172.21.15.43:0/62269647 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba6d63e8d0 con 0x55ba6d651e40 2024-05-27T07:28:02.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.625+0000 7f9aee000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:28:02.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.625+0000 7f9b04c00700 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7e0b10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f9af8010c40 tx=0x7f9af8010c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27f3f88b64e3ce3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:02.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.626+0000 7f9b04c00700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9ae40b2980 0x7f9ae40b4e90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009c50 tx=0x7f9af8021000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:02.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.629+0000 7f9aee000700 1 -- 172.21.15.43:0/62269647 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8004e40 con 0x55ba6d651e40 2024-05-27T07:28:02.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.629+0000 7f9aee000700 1 -- 172.21.15.43:0/62269647 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 9584+0+0 (secure 0 0 0) 0x7f9af802b440 con 0x55ba6d651e40 2024-05-27T07:28:02.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.629+0000 7f9b156fcb80 1 -- 172.21.15.43:0/62269647 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9af8010d80 con 0x55ba6d651e40 2024-05-27T07:28:02.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.630+0000 7f9aee000700 1 -- 172.21.15.43:0/62269647 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7f9af802b640 con 0x55ba6d651e40 2024-05-27T07:28:02.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.630+0000 7f9b156fcb80 10 client.17478 fetch_fsmap learned FSMap version 113 2024-05-27T07:28:02.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.630+0000 7f9b156fcb80 1 -- 172.21.15.43:0/62269647 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x55ba6d6bf060 con 0x55ba6d651e40 2024-05-27T07:28:02.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.631+0000 7f9aee000700 1 -- 172.21.15.43:0/62269647 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9af8010d80 con 0x55ba6d651e40 2024-05-27T07:28:02.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.632+0000 7f9aee000700 1 -- 172.21.15.43:0/62269647 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 113) ==== 4808+0+0 (secure 0 0 0) 0x7f9af80357b0 con 0x55ba6d651e40 2024-05-27T07:28:02.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.632+0000 7f9b156fcb80 10 client.17478 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T07:28:02.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.632+0000 7f9b156fcb80 10 client.17478 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:28:02.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.632+0000 7f9b156fcb80 20 client.17478 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:02.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.632+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ba6d7f3880 0x55ba6d7f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.632+0000 7f9b156fcb80 4 client.17478 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:02.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.632+0000 7f9b156fcb80 1 -- 172.21.15.43:0/62269647 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba6d700d90 con 0x55ba6d7f3880 2024-05-27T07:28:02.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.633+0000 7f9affe00700 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ba6d7f3880 0x55ba6d7f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.635+0000 7f9affe00700 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ba6d7f3880 0x55ba6d7f5d30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8a7e232e947c646 server_cookie=bfc4d9e12a683195 in_seq=0 out_seq=0 2024-05-27T07:28:02.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.635+0000 7f9aee000700 10 client.17478 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:28:02.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.637+0000 7f9aee000700 1 -- 172.21.15.43:0/62269647 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ba6d700d90 con 0x55ba6d7f3880 2024-05-27T07:28:02.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.637+0000 7f9aee000700 10 client.17478 handle_command_reply: tid=0 2024-05-27T07:28:02.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.637+0000 7f9b156fcb80 1 client.17478 shutdown 2024-05-27T07:28:02.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.638+0000 7f9b156fcb80 1 -- 172.21.15.43:0/62269647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 msgr2=0x55ba6d7e0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:02.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.638+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7e0b10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f9af8010c40 tx=0x7f9af8010c70 comp rx=0 tx=0).stop 2024-05-27T07:28:02.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.638+0000 7f9b156fcb80 1 -- 172.21.15.43:0/62269647 shutdown_connections 2024-05-27T07:28:02.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.638+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.638+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d651e40 0x55ba6d7e0b10 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.638+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ba6d7f3880 0x55ba6d7f5d30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.638+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/62269647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7e1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.639+0000 7f9b156fcb80 1 -- 172.21.15.43:0/62269647 >> 172.21.15.43:0/62269647 conn(0x55ba6d6eb390 msgr2=0x55ba6d7cc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:02.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.639+0000 7f9aee000700 0 client.17478 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:28:02.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.639+0000 7f9b156fcb80 1 -- 172.21.15.43:0/62269647 shutdown_connections 2024-05-27T07:28:02.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.640+0000 7f9b156fcb80 1 -- 172.21.15.43:0/62269647 wait complete. 2024-05-27T07:28:02.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.640+0000 7f9b156fcb80 20 client.17478 trim_cache size 0 max 16384 2024-05-27T07:28:02.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.675+0000 7f9b156fcb80 1 Processor -- start 2024-05-27T07:28:02.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.675+0000 7f9b156fcb80 1 -- start start 2024-05-27T07:28:02.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.676+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d651e40 0x55ba6d7cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.676+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7ef670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.676+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.676+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001aefa0 con 0x55ba6d7e3ac0 2024-05-27T07:28:02.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.676+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001acc40 con 0x55ba6d73d9f0 2024-05-27T07:28:02.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.676+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001ae8d0 con 0x55ba6d651e40 2024-05-27T07:28:02.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.676+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.676+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60592/0 (socket says 172.21.15.43:60592) 2024-05-27T07:28:02.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.676+0000 7f9b04c00700 1 -- 172.21.15.43:0/25589069 learned_addr learned my addr 172.21.15.43:0/25589069 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:02.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.676+0000 7f9affe00700 1 --2- 172.21.15.43:0/25589069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d651e40 0x55ba6d7cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.677+0000 7f9af7400700 1 --2- 172.21.15.43:0/25589069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7ef670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.677+0000 7f9b04c00700 1 -- 172.21.15.43:0/25589069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d651e40 msgr2=0x55ba6d7cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:02.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.677+0000 7f9b04c00700 1 --2- 172.21.15.43:0/25589069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d651e40 0x55ba6d7cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.677+0000 7f9b04c00700 1 -- 172.21.15.43:0/25589069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d73d9f0 msgr2=0x55ba6d7ef670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:02.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.677+0000 7f9b04c00700 1 --2- 172.21.15.43:0/25589069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7ef670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.677+0000 7f9b04c00700 1 -- 172.21.15.43:0/25589069 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba6d7df080 con 0x55ba6d7e3ac0 2024-05-27T07:28:02.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.678+0000 7f9b04c00700 1 --2- 172.21.15.43:0/25589069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e4ee0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f9af801ce80 tx=0x7f9af80107e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5feb1d1a89dca3cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:02.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.678+0000 7f9af4c00700 1 -- 172.21.15.43:0/25589069 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af80057e0 con 0x55ba6d7e3ac0 2024-05-27T07:28:02.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.678+0000 7f9af4c00700 1 -- 172.21.15.43:0/25589069 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9af80109e0 con 0x55ba6d7e3ac0 2024-05-27T07:28:02.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.678+0000 7f9af4c00700 1 -- 172.21.15.43:0/25589069 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af800ea10 con 0x55ba6d7e3ac0 2024-05-27T07:28:02.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.680+0000 7f9b156fcb80 1 -- 172.21.15.43:0/25589069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 msgr2=0x55ba6d7e4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:02.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.680+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/25589069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e4ee0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f9af801ce80 tx=0x7f9af80107e0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.680+0000 7f9b156fcb80 1 -- 172.21.15.43:0/25589069 shutdown_connections 2024-05-27T07:28:02.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.680+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/25589069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7e4ee0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.680+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/25589069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d651e40 0x55ba6d7cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.680+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/25589069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7ef670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.680+0000 7f9b156fcb80 1 -- 172.21.15.43:0/25589069 >> 172.21.15.43:0/25589069 conn(0x55ba6d6eb390 msgr2=0x55ba6d7d8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:02.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.681+0000 7f9b156fcb80 1 -- 172.21.15.43:0/25589069 shutdown_connections 2024-05-27T07:28:02.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.681+0000 7f9b156fcb80 1 -- 172.21.15.43:0/25589069 wait complete. 2024-05-27T07:28:02.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.682+0000 7f9b156fcb80 1 -- start start 2024-05-27T07:28:02.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.683+0000 7f9b156fcb80 1 Processor -- start 2024-05-27T07:28:02.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.685+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d651e40 0x55ba6d7dbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.685+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7da960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.685+0000 7f9b156fcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7daea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.685+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001ada70 con 0x55ba6d7e3ac0 2024-05-27T07:28:02.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.685+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001ad340 con 0x55ba6d651e40 2024-05-27T07:28:02.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.685+0000 7f9b156fcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001ae1a0 con 0x55ba6d73d9f0 2024-05-27T07:28:02.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.685+0000 7f9b156fcb80 10 client.0 _ll_register_callbacks cb 0x55ba6d698c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:02.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.685+0000 7f9affe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d651e40 0x55ba6d7dbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.685+0000 7f9affe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d651e40 0x55ba6d7dbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41582/0 (socket says 172.21.15.43:41582) 2024-05-27T07:28:02.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.685+0000 7f9affe00700 1 -- 172.21.15.43:0/365851158 learned_addr learned my addr 172.21.15.43:0/365851158 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:02.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.686+0000 7f9af7400700 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7da960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.686+0000 7f9af7400700 1 -- 172.21.15.43:0/365851158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d651e40 msgr2=0x55ba6d7dbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:02.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.686+0000 7f9af7400700 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d651e40 0x55ba6d7dbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.686+0000 7f9af7400700 1 -- 172.21.15.43:0/365851158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 msgr2=0x55ba6d7daea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:02.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.686+0000 7f9af7400700 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7daea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.686+0000 7f9af7400700 1 -- 172.21.15.43:0/365851158 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba6d7db3e0 con 0x55ba6d73d9f0 2024-05-27T07:28:02.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.688+0000 7f9af7400700 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7da960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55ba6d7da930 tx=0x7f9ae8005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cb8a18a08091179 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:02.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.689+0000 7f9aee000700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:28:02.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.689+0000 7f9aee000700 1 -- 172.21.15.43:0/365851158 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae8003df0 con 0x55ba6d73d9f0 2024-05-27T07:28:02.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.689+0000 7f9aee000700 1 -- 172.21.15.43:0/365851158 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 9584+0+0 (secure 0 0 0) 0x7f9ae800b6b0 con 0x55ba6d73d9f0 2024-05-27T07:28:02.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.689+0000 7f9b156fcb80 1 -- 172.21.15.43:0/365851158 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9af80109e0 con 0x55ba6d73d9f0 2024-05-27T07:28:02.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.690+0000 7f9aee000700 1 -- 172.21.15.43:0/365851158 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7f9ae802f050 con 0x55ba6d73d9f0 2024-05-27T07:28:02.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.690+0000 7f9b156fcb80 10 client.27007 fetch_fsmap learned FSMap version 113 2024-05-27T07:28:02.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.690+0000 7f9b156fcb80 1 -- 172.21.15.43:0/365851158 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=4+}) -- 0x55ba6d7eb420 con 0x55ba6d73d9f0 2024-05-27T07:28:02.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.690+0000 7f9aee000700 1 -- 172.21.15.43:0/365851158 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9af80109e0 con 0x55ba6d73d9f0 2024-05-27T07:28:02.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.691+0000 7f9aee000700 1 -- 172.21.15.43:0/365851158 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 113) ==== 4808+0+0 (secure 0 0 0) 0x7f9ae800f6f0 con 0x55ba6d73d9f0 2024-05-27T07:28:02.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.691+0000 7f9b156fcb80 10 client.27007 fetch_fsmap finished waiting for FSMap version 113 2024-05-27T07:28:02.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.691+0000 7f9b156fcb80 10 client.27007 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:28:02.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.691+0000 7f9b156fcb80 20 client.27007 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:02.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.691+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ba6d7ebb30 0x55ba6d7ebf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:02.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.691+0000 7f9b156fcb80 4 client.27007 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:02.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.691+0000 7f9b156fcb80 1 -- 172.21.15.43:0/365851158 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ba6d6fa570 con 0x55ba6d7ebb30 2024-05-27T07:28:02.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.692+0000 7f9affe00700 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ba6d7ebb30 0x55ba6d7ebf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:02.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.694+0000 7f9affe00700 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ba6d7ebb30 0x55ba6d7ebf40 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a13bae5325778f91 server_cookie=6ad24513fa22e999 in_seq=0 out_seq=0 2024-05-27T07:28:02.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.694+0000 7f9aee000700 10 client.27007 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:28:02.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.695+0000 7f9aee000700 1 -- 172.21.15.43:0/365851158 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ba6d6fa570 con 0x55ba6d7ebb30 2024-05-27T07:28:02.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.695+0000 7f9aee000700 10 client.27007 handle_command_reply: tid=0 2024-05-27T07:28:02.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.695+0000 7f9b156fcb80 1 client.27007 shutdown 2024-05-27T07:28:02.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.696+0000 7f9b156fcb80 1 -- 172.21.15.43:0/365851158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 msgr2=0x55ba6d7da960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:02.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.696+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7da960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55ba6d7da930 tx=0x7f9ae8005ea0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.697+0000 7f9b156fcb80 1 -- 172.21.15.43:0/365851158 shutdown_connections 2024-05-27T07:28:02.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.697+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ba6d7e3ac0 0x55ba6d7daea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.697+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ba6d7ebb30 0x55ba6d7ebf40 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.697+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ba6d651e40 0x55ba6d7dbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.697+0000 7f9b156fcb80 1 --2- 172.21.15.43:0/365851158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ba6d73d9f0 0x55ba6d7da960 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.697+0000 7f9aee000700 0 client.27007 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:28:02.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.697+0000 7f9b156fcb80 1 -- 172.21.15.43:0/365851158 >> 172.21.15.43:0/365851158 conn(0x55ba6d6eb390 msgr2=0x55ba6d7e08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:02.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.698+0000 7f9b156fcb80 1 -- 172.21.15.43:0/365851158 shutdown_connections 2024-05-27T07:28:02.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.698+0000 7f9b156fcb80 1 -- 172.21.15.43:0/365851158 wait complete. 2024-05-27T07:28:02.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.698+0000 7f9b156fcb80 20 client.27007 trim_cache size 0 max 16384 2024-05-27T07:28:02.690 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:28:02.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.700+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9ae40b2980 msgr2=0x7f9ae40b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:02.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.700+0000 7f9b05600700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9ae40b2980 0x7f9ae40b4e90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009c50 tx=0x7f9af8021000 comp rx=0 tx=0).stop 2024-05-27T07:28:02.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.700+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b00105280 msgr2=0x7f9b001a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:02.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.700+0000 7f9b05600700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b00105280 0x7f9b001a29a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f9af0007e20 tx=0x7f9af000a340 comp rx=0 tx=0).stop 2024-05-27T07:28:02.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.701+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 shutdown_connections 2024-05-27T07:28:02.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.701+0000 7f9b05600700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9ae40b2980 0x7f9ae40b4e90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.701+0000 7f9b05600700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b0010e6f0 0x7f9b0019cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.701+0000 7f9b05600700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b00105280 0x7f9b001a29a0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.701+0000 7f9b05600700 1 --2- 172.21.15.43:0/1529895422 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b00105bd0 0x7f9b001a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:02.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.701+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 >> 172.21.15.43:0/1529895422 conn(0x7f9b00078b40 msgr2=0x7f9b000ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:02.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.701+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 shutdown_connections 2024-05-27T07:28:02.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:02.701+0000 7f9b05600700 1 -- 172.21.15.43:0/1529895422 wait complete. 2024-05-27T07:28:02.763 INFO:tasks.workunit.client.0.smithi043.stdout: 10 435 51.07 MB/sec warmup 3 sec latency 112.747 ms 2024-05-27T07:28:02.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:02 smithi043 ceph-mon[31531]: pgmap v474: 129 pgs: 129 active+clean; 216 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5.5 MiB/s wr, 57 op/s 2024-05-27T07:28:02.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:02 smithi043 ceph-mon[31531]: osdmap e136: 12 total, 12 up, 12 in 2024-05-27T07:28:02.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:02 smithi179 ceph-mon[35888]: pgmap v474: 129 pgs: 129 active+clean; 216 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5.5 MiB/s wr, 57 op/s 2024-05-27T07:28:02.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:02 smithi179 ceph-mon[35888]: osdmap e136: 12 total, 12 up, 12 in 2024-05-27T07:28:03.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:02 smithi132 ceph-mon[35283]: pgmap v474: 129 pgs: 129 active+clean; 216 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5.5 MiB/s wr, 57 op/s 2024-05-27T07:28:03.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:02 smithi132 ceph-mon[35283]: osdmap e136: 12 total, 12 up, 12 in 2024-05-27T07:28:03.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:28:03.763 INFO:tasks.workunit.client.0.smithi043.stdout: 10 609 51.32 MB/sec warmup 4 sec latency 124.576 ms 2024-05-27T07:28:04.174 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:04.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:04 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:28:04.626 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:04 smithi043 ceph-mon[31531]: pgmap v476: 129 pgs: 129 active+clean; 216 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6.6 MiB/s wr, 60 op/s 2024-05-27T07:28:04.763 INFO:tasks.workunit.client.0.smithi043.stdout: 10 713 42.49 MB/sec warmup 5 sec latency 201.609 ms 2024-05-27T07:28:04.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:04 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:28:04.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:04 smithi179 ceph-mon[35888]: pgmap v476: 129 pgs: 129 active+clean; 216 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6.6 MiB/s wr, 60 op/s 2024-05-27T07:28:05.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:04 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:28:05.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:04 smithi132 ceph-mon[35283]: pgmap v476: 129 pgs: 129 active+clean; 216 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6.6 MiB/s wr, 60 op/s 2024-05-27T07:28:05.763 INFO:tasks.workunit.client.0.smithi043.stdout: 10 806 35.77 MB/sec warmup 6 sec latency 180.702 ms 2024-05-27T07:28:06.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.433+0000 7f030e800700 1 -- 172.21.15.43:0/1156519223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0308109860 msgr2=0x7f03081111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:06.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.433+0000 7f030e800700 1 --2- 172.21.15.43:0/1156519223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0308109860 0x7f03081111c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f030000b0d0 tx=0x7f030000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.433+0000 7f030e800700 1 -- 172.21.15.43:0/1156519223 shutdown_connections 2024-05-27T07:28:06.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.433+0000 7f030e800700 1 --2- 172.21.15.43:0/1156519223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0308109860 0x7f03081111c0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.433+0000 7f030e800700 1 --2- 172.21.15.43:0/1156519223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03081045e0 0x7f03081049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.433+0000 7f030e800700 1 --2- 172.21.15.43:0/1156519223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0308108f10 0x7f0308109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.433+0000 7f030e800700 1 -- 172.21.15.43:0/1156519223 >> 172.21.15.43:0/1156519223 conn(0x7f0308100030 msgr2=0x7f0308102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:06.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.434+0000 7f030e800700 1 -- 172.21.15.43:0/1156519223 shutdown_connections 2024-05-27T07:28:06.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.434+0000 7f030e800700 1 -- 172.21.15.43:0/1156519223 wait complete. 2024-05-27T07:28:06.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.435+0000 7f030e800700 1 Processor -- start 2024-05-27T07:28:06.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.436+0000 7f030e800700 1 -- start start 2024-05-27T07:28:06.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.438+0000 7f030e800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03081045e0 0x7f03081a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.438+0000 7f030e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0308108f10 0x7f03081a2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.438+0000 7f030e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0308109860 0x7f03081ad170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.438+0000 7f0306a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0308108f10 0x7f03081a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.439+0000 7f0306a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0308108f10 0x7f03081a2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58402/0 (socket says 172.21.15.43:58402) 2024-05-27T07:28:06.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.440+0000 7f0307e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0308109860 0x7f03081ad170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.440+0000 7f0307e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0308109860 0x7f03081ad170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38432/0 (socket says 172.21.15.43:38432) 2024-05-27T07:28:06.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.440+0000 7f030e800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0308114410 con 0x7f0308109860 2024-05-27T07:28:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.440+0000 7f0307400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03081045e0 0x7f03081a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.441+0000 7f0307400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03081045e0 0x7f03081a22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:34626/0 (socket says 172.21.15.43:34626) 2024-05-27T07:28:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.441+0000 7f0307400700 1 -- 172.21.15.43:0/1210345967 learned_addr learned my addr 172.21.15.43:0/1210345967 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.441+0000 7f030e800700 1 -- 172.21.15.43:0/1210345967 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0308114290 con 0x7f0308108f10 2024-05-27T07:28:06.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.441+0000 7f030e800700 1 -- 172.21.15.43:0/1210345967 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0308114590 con 0x7f03081045e0 2024-05-27T07:28:06.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.443+0000 7f0306a00700 1 -- 172.21.15.43:0/1210345967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03081045e0 msgr2=0x7f03081a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:06.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.443+0000 7f0306a00700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03081045e0 0x7f03081a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.443+0000 7f0306a00700 1 -- 172.21.15.43:0/1210345967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0308109860 msgr2=0x7f03081ad170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:06.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.443+0000 7f0306a00700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0308109860 0x7f03081ad170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.443+0000 7f0306a00700 1 -- 172.21.15.43:0/1210345967 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03081ad6b0 con 0x7f0308108f10 2024-05-27T07:28:06.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.444+0000 7f0306a00700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0308108f10 0x7f03081a2800 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f02f400e9c0 tx=0x7f02f400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:06.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.444+0000 7f02ffe00700 1 -- 172.21.15.43:0/1210345967 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f400c9c0 con 0x7f0308108f10 2024-05-27T07:28:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.444+0000 7f02ffe00700 1 -- 172.21.15.43:0/1210345967 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02f4010430 con 0x7f0308108f10 2024-05-27T07:28:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.444+0000 7f02ffe00700 1 -- 172.21.15.43:0/1210345967 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4018680 con 0x7f0308108f10 2024-05-27T07:28:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.444+0000 7f030e800700 1 -- 172.21.15.43:0/1210345967 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03081ad940 con 0x7f0308108f10 2024-05-27T07:28:06.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.445+0000 7f030e800700 1 -- 172.21.15.43:0/1210345967 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03081ade30 con 0x7f0308108f10 2024-05-27T07:28:06.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.451+0000 7f031d3e8b80 1 Processor -- start 2024-05-27T07:28:06.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.451+0000 7f031d3e8b80 1 -- start start 2024-05-27T07:28:06.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f02ffe00700 1 -- 172.21.15.43:0/1210345967 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f02f40108e0 con 0x7f0308108f10 2024-05-27T07:28:06.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e271b9f0 0x55f2e27ae330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e262fe40 0x55f2e27ae870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e27aee70 0x55f2e27bb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f2e26dce40 con 0x55f2e271b9f0 2024-05-27T07:28:06.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f2e26cc9e0 con 0x55f2e262fe40 2024-05-27T07:28:06.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f2e26d0a20 con 0x55f2e27aee70 2024-05-27T07:28:06.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f0307400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e262fe40 0x55f2e27ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f0307400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e262fe40 0x55f2e27ae870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58404/0 (socket says 172.21.15.43:58404) 2024-05-27T07:28:06.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f0307400700 1 -- 172.21.15.43:0/2493012370 learned_addr learned my addr 172.21.15.43:0/2493012370 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:06.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f0306a00700 1 --2- 172.21.15.43:0/2493012370 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e27aee70 0x55f2e27bb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.452+0000 7f0307e00700 1 --2- 172.21.15.43:0/2493012370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e271b9f0 0x55f2e27ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.453+0000 7f0307400700 1 -- 172.21.15.43:0/2493012370 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e27aee70 msgr2=0x55f2e27bb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:06.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.453+0000 7f0307400700 1 --2- 172.21.15.43:0/2493012370 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e27aee70 0x55f2e27bb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.453+0000 7f0307400700 1 -- 172.21.15.43:0/2493012370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e271b9f0 msgr2=0x55f2e27ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:06.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.453+0000 7f0307400700 1 --2- 172.21.15.43:0/2493012370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e271b9f0 0x55f2e27ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.453+0000 7f0307400700 1 -- 172.21.15.43:0/2493012370 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2e2695420 con 0x55f2e262fe40 2024-05-27T07:28:06.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.454+0000 7f02ffe00700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02ec0b2b40 0x7f02ec0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.454+0000 7f02ffe00700 1 -- 172.21.15.43:0/1210345967 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f02f40e2a10 con 0x7f0308108f10 2024-05-27T07:28:06.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.454+0000 7f0307e00700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02ec0b2b40 0x7f02ec0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.454+0000 7f0307400700 1 --2- 172.21.15.43:0/2493012370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e262fe40 0x55f2e27ae870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009bb0 tx=0x7f02f8009ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e98c7dc6670f275 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:06.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.455+0000 7f0307e00700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02ec0b2b40 0x7f02ec0b5050 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0300009900 tx=0x7f030000b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:06.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.455+0000 7f02fcc00700 1 -- 172.21.15.43:0/2493012370 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8028070 con 0x55f2e262fe40 2024-05-27T07:28:06.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.455+0000 7f02fcc00700 1 -- 172.21.15.43:0/2493012370 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02f800ee20 con 0x55f2e262fe40 2024-05-27T07:28:06.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.455+0000 7f02fcc00700 1 -- 172.21.15.43:0/2493012370 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8028070 con 0x55f2e262fe40 2024-05-27T07:28:06.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.456+0000 7f031d3e8b80 1 -- 172.21.15.43:0/2493012370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e262fe40 msgr2=0x55f2e27ae870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:06.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.456+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/2493012370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e262fe40 0x55f2e27ae870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009bb0 tx=0x7f02f8009ec0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.457+0000 7f031d3e8b80 1 -- 172.21.15.43:0/2493012370 shutdown_connections 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.457+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/2493012370 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e27aee70 0x55f2e27bb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.457+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/2493012370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e271b9f0 0x55f2e27ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.457+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/2493012370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e262fe40 0x55f2e27ae870 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.457+0000 7f031d3e8b80 1 -- 172.21.15.43:0/2493012370 >> 172.21.15.43:0/2493012370 conn(0x55f2e26c9390 msgr2=0x55f2e27aa4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.457+0000 7f031d3e8b80 1 -- 172.21.15.43:0/2493012370 shutdown_connections 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.457+0000 7f031d3e8b80 1 -- 172.21.15.43:0/2493012370 wait complete. 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.458+0000 7f031d3e8b80 1 -- start start 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.458+0000 7f031d3e8b80 1 Processor -- start 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.458+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e262fe40 0x55f2e27b85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.458+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.458+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e27aee70 0x55f2e27b9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.458+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f2e25d2fa0 con 0x55f2e262fe40 2024-05-27T07:28:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.458+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f2e266d760 con 0x55f2e271b9f0 2024-05-27T07:28:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.458+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55f2e27bbb80 con 0x55f2e27aee70 2024-05-27T07:28:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.458+0000 7f031d3e8b80 10 client.0 _ll_register_callbacks cb 0x55f2e2676c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.459+0000 7f0306a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.459+0000 7f0306a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58420/0 (socket says 172.21.15.43:58420) 2024-05-27T07:28:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.459+0000 7f0306a00700 1 -- 172.21.15.43:0/1580491183 learned_addr learned my addr 172.21.15.43:0/1580491183 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.459+0000 7f0307e00700 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e27aee70 0x55f2e27b9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.460+0000 7f0306a00700 1 -- 172.21.15.43:0/1580491183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e27aee70 msgr2=0x55f2e27b9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.460+0000 7f0306a00700 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e27aee70 0x55f2e27b9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.460+0000 7f0306a00700 1 -- 172.21.15.43:0/1580491183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e262fe40 msgr2=0x55f2e27b85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.460+0000 7f0306a00700 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e262fe40 0x55f2e27b85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.460+0000 7f0306a00700 1 -- 172.21.15.43:0/1580491183 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2e261c8d0 con 0x55f2e271b9f0 2024-05-27T07:28:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.460+0000 7f0306a00700 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b8b10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f02f40a5940 tx=0x7f02f40a9e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=150f8024850e0f4e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.460+0000 7f02f1600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:28:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.461+0000 7f02f1600700 1 -- 172.21.15.43:0/1580491183 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f40a5780 con 0x55f2e271b9f0 2024-05-27T07:28:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.461+0000 7f02f1600700 1 -- 172.21.15.43:0/1580491183 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f02f40a5eb0 con 0x55f2e271b9f0 2024-05-27T07:28:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.461+0000 7f031d3e8b80 1 -- 172.21.15.43:0/1580491183 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02f800ee20 con 0x55f2e271b9f0 2024-05-27T07:28:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.461+0000 7f02f1600700 1 -- 172.21.15.43:0/1580491183 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) ==== 24+0+0 (secure 0 0 0) 0x7f02f40a9030 con 0x55f2e271b9f0 2024-05-27T07:28:06.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.461+0000 7f031d3e8b80 10 client.36896 fetch_fsmap learned FSMap version 114 2024-05-27T07:28:06.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.461+0000 7f031d3e8b80 1 -- 172.21.15.43:0/1580491183 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=4+}) -- 0x55f2e269d060 con 0x55f2e271b9f0 2024-05-27T07:28:06.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.461+0000 7f02f1600700 1 -- 172.21.15.43:0/1580491183 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02f800ee20 con 0x55f2e271b9f0 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.462+0000 7f02f1600700 1 -- 172.21.15.43:0/1580491183 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 114) ==== 4812+0+0 (secure 0 0 0) 0x7f02f400cb60 con 0x55f2e271b9f0 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.462+0000 7f031d3e8b80 10 client.36896 fetch_fsmap finished waiting for FSMap version 114 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.462+0000 7f031d3e8b80 10 client.36896 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.462+0000 7f031d3e8b80 20 client.36896 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.462+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f2e27c8f30 0x55f2e27cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.462+0000 7f031d3e8b80 4 client.36896 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.462+0000 7f031d3e8b80 1 -- 172.21.15.43:0/1580491183 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2e26ded90 con 0x55f2e27c8f30 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.462+0000 7f0307400700 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f2e27c8f30 0x55f2e27cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.463+0000 7f0307400700 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f2e27c8f30 0x55f2e27cb400 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=913bc80d92a91ceb server_cookie=b98b4c7c6a6a9d00 in_seq=0 out_seq=0 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.463+0000 7f02f1600700 10 client.36896 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.465+0000 7f02f1600700 1 -- 172.21.15.43:0/1580491183 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55f2e26ded90 con 0x55f2e27c8f30 2024-05-27T07:28:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.465+0000 7f02f1600700 10 client.36896 handle_command_reply: tid=0 2024-05-27T07:28:06.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.465+0000 7f031d3e8b80 1 client.36896 shutdown 2024-05-27T07:28:06.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.466+0000 7f031d3e8b80 1 -- 172.21.15.43:0/1580491183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 msgr2=0x55f2e27b8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:06.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.466+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b8b10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f02f40a5940 tx=0x7f02f40a9e00 comp rx=0 tx=0).stop 2024-05-27T07:28:06.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.466+0000 7f031d3e8b80 1 -- 172.21.15.43:0/1580491183 shutdown_connections 2024-05-27T07:28:06.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.466+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f2e27c8f30 0x55f2e27cb400 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.466+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e27aee70 0x55f2e27b9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.466+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e262fe40 0x55f2e27b85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.466+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/1580491183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b8b10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.466+0000 7f02f1600700 0 client.36896 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:28:06.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.466+0000 7f031d3e8b80 1 -- 172.21.15.43:0/1580491183 >> 172.21.15.43:0/1580491183 conn(0x55f2e26c9390 msgr2=0x55f2e27ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:06.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.466+0000 7f031d3e8b80 1 -- 172.21.15.43:0/1580491183 shutdown_connections 2024-05-27T07:28:06.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.467+0000 7f031d3e8b80 1 -- 172.21.15.43:0/1580491183 wait complete. 2024-05-27T07:28:06.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.467+0000 7f031d3e8b80 20 client.36896 trim_cache size 0 max 16384 2024-05-27T07:28:06.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.494+0000 7f031d3e8b80 1 Processor -- start 2024-05-27T07:28:06.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.494+0000 7f031d3e8b80 1 -- start start 2024-05-27T07:28:06.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.495+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 0x55f2e27b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.495+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.495+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 0x55f2e27baa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.495+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03081af190 con 0x55f2e27aee70 2024-05-27T07:28:06.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.495+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0308074720 con 0x55f2e271b9f0 2024-05-27T07:28:06.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.495+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f03081aeac0 con 0x55f2e262fe40 2024-05-27T07:28:06.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.496+0000 7f0306a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.496+0000 7f0307400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 0x55f2e27b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.496+0000 7f0307400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 0x55f2e27b3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:34646/0 (socket says 172.21.15.43:34646) 2024-05-27T07:28:06.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.496+0000 7f0307400700 1 -- 172.21.15.43:0/3772433678 learned_addr learned my addr 172.21.15.43:0/3772433678 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:06.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.496+0000 7f0307400700 1 -- 172.21.15.43:0/3772433678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 msgr2=0x55f2e27b7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:06.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.496+0000 7f0307400700 1 --2- 172.21.15.43:0/3772433678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.496+0000 7f0307400700 1 -- 172.21.15.43:0/3772433678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 msgr2=0x55f2e27baa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:06.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.496+0000 7f0307400700 1 --2- 172.21.15.43:0/3772433678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 0x55f2e27baa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.497+0000 7f0307400700 1 -- 172.21.15.43:0/3772433678 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2e27b9690 con 0x55f2e262fe40 2024-05-27T07:28:06.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.498+0000 7f0307400700 1 --2- 172.21.15.43:0/3772433678 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 0x55f2e27b3df0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x55f2e27aba20 tx=0x7f02f8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b45a213641227f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:06.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.498+0000 7f02f3e00700 1 -- 172.21.15.43:0/3772433678 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8028070 con 0x55f2e262fe40 2024-05-27T07:28:06.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.498+0000 7f02f3e00700 1 -- 172.21.15.43:0/3772433678 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02f803b070 con 0x55f2e262fe40 2024-05-27T07:28:06.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.498+0000 7f02f3e00700 1 -- 172.21.15.43:0/3772433678 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8024cd0 con 0x55f2e262fe40 2024-05-27T07:28:06.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.500+0000 7f031d3e8b80 1 -- 172.21.15.43:0/3772433678 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 msgr2=0x55f2e27b3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:06.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.500+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/3772433678 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 0x55f2e27b3df0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x55f2e27aba20 tx=0x7f02f8008040 comp rx=0 tx=0).stop 2024-05-27T07:28:06.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.500+0000 7f031d3e8b80 1 -- 172.21.15.43:0/3772433678 shutdown_connections 2024-05-27T07:28:06.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.500+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/3772433678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 0x55f2e27baa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.500+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/3772433678 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 0x55f2e27b3df0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.500+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/3772433678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.500+0000 7f031d3e8b80 1 -- 172.21.15.43:0/3772433678 >> 172.21.15.43:0/3772433678 conn(0x55f2e26c9390 msgr2=0x55f2e27ae0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:06.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.501+0000 7f031d3e8b80 1 -- 172.21.15.43:0/3772433678 shutdown_connections 2024-05-27T07:28:06.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.501+0000 7f031d3e8b80 1 -- 172.21.15.43:0/3772433678 wait complete. 2024-05-27T07:28:06.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.502+0000 7f031d3e8b80 1 -- start start 2024-05-27T07:28:06.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.502+0000 7f031d3e8b80 1 Processor -- start 2024-05-27T07:28:06.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.503+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 0x55f2e27b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.503+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.504+0000 7f031d3e8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 0x55f2e27b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.504+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0308075580 con 0x55f2e27aee70 2024-05-27T07:28:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.504+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0308074e50 con 0x55f2e271b9f0 2024-05-27T07:28:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.504+0000 7f031d3e8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f03081ae390 con 0x55f2e262fe40 2024-05-27T07:28:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.504+0000 7f031d3e8b80 10 client.0 _ll_register_callbacks cb 0x55f2e2676c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.504+0000 7f0307400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 0x55f2e27b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.505+0000 7f0306a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.505+0000 7f0307e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 0x55f2e27b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.505+0000 7f0307e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 0x55f2e27b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38470/0 (socket says 172.21.15.43:38470) 2024-05-27T07:28:06.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.505+0000 7f0306a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58444/0 (socket says 172.21.15.43:58444) 2024-05-27T07:28:06.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.505+0000 7f0306a00700 1 -- 172.21.15.43:0/983049866 learned_addr learned my addr 172.21.15.43:0/983049866 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:06.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.506+0000 7f0307e00700 1 -- 172.21.15.43:0/983049866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 msgr2=0x55f2e27b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:06.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.506+0000 7f0307e00700 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 0x55f2e27b21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.506+0000 7f0307e00700 1 -- 172.21.15.43:0/983049866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 msgr2=0x55f2e27b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:06.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.506+0000 7f0307e00700 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.506+0000 7f0307e00700 1 -- 172.21.15.43:0/983049866 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2e27bfe40 con 0x55f2e27aee70 2024-05-27T07:28:06.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.506+0000 7f0306a00700 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:06.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.507+0000 7f0307e00700 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 0x55f2e27b2c70 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x55f2e27b0ff0 tx=0x7f030000b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4dd24d2b1e8dbe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:06.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.507+0000 7f02f1600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:28:06.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.507+0000 7f02f1600700 1 -- 172.21.15.43:0/983049866 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0300038060 con 0x55f2e27aee70 2024-05-27T07:28:06.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.507+0000 7f031d3e8b80 1 -- 172.21.15.43:0/983049866 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02f803b070 con 0x55f2e27aee70 2024-05-27T07:28:06.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.507+0000 7f02f1600700 1 -- 172.21.15.43:0/983049866 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f03000077d0 con 0x55f2e27aee70 2024-05-27T07:28:06.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.508+0000 7f02f1600700 1 -- 172.21.15.43:0/983049866 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) ==== 24+0+0 (secure 0 0 0) 0x7f030003f050 con 0x55f2e27aee70 2024-05-27T07:28:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.508+0000 7f031d3e8b80 10 client.17493 fetch_fsmap learned FSMap version 114 2024-05-27T07:28:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.508+0000 7f031d3e8b80 1 -- 172.21.15.43:0/983049866 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=4+}) -- 0x55f2e27c10a0 con 0x55f2e27aee70 2024-05-27T07:28:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.509+0000 7f02f1600700 1 -- 172.21.15.43:0/983049866 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f02f803b070 con 0x55f2e27aee70 2024-05-27T07:28:06.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.509+0000 7f02f1600700 1 -- 172.21.15.43:0/983049866 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 114) ==== 4812+0+0 (secure 0 0 0) 0x7f0300011070 con 0x55f2e27aee70 2024-05-27T07:28:06.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.510+0000 7f031d3e8b80 10 client.17493 fetch_fsmap finished waiting for FSMap version 114 2024-05-27T07:28:06.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.510+0000 7f031d3e8b80 10 client.17493 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:28:06.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.510+0000 7f031d3e8b80 20 client.17493 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:06.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.510+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f2e27c17e0 0x55f2e2835ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:06.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.511+0000 7f031d3e8b80 4 client.17493 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:06.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.511+0000 7f031d3e8b80 1 -- 172.21.15.43:0/983049866 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f2e26d8570 con 0x55f2e27c17e0 2024-05-27T07:28:06.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.512+0000 7f0307400700 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f2e27c17e0 0x55f2e2835ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:06.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.513+0000 7f0307400700 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f2e27c17e0 0x55f2e2835ae0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=91693327dd98a243 server_cookie=4a88680058a7a4b2 in_seq=0 out_seq=0 2024-05-27T07:28:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.513+0000 7f02f1600700 10 client.17493 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:28:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.514+0000 7f02f1600700 1 -- 172.21.15.43:0/983049866 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f2e26d8570 con 0x55f2e27c17e0 2024-05-27T07:28:06.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.514+0000 7f02f1600700 10 client.17493 handle_command_reply: tid=0 2024-05-27T07:28:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.514+0000 7f031d3e8b80 1 client.17493 shutdown 2024-05-27T07:28:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.515+0000 7f031d3e8b80 1 -- 172.21.15.43:0/983049866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 msgr2=0x55f2e27b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:06.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.515+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 0x55f2e27b2c70 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x55f2e27b0ff0 tx=0x7f030000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.516+0000 7f031d3e8b80 1 -- 172.21.15.43:0/983049866 shutdown_connections 2024-05-27T07:28:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.516+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55f2e27c17e0 0x55f2e2835ae0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.516+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2e27aee70 0x55f2e27b2c70 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.516+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f2e262fe40 0x55f2e27b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.516+0000 7f031d3e8b80 1 --2- 172.21.15.43:0/983049866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f2e271b9f0 0x55f2e27b2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.516+0000 7f031d3e8b80 1 -- 172.21.15.43:0/983049866 >> 172.21.15.43:0/983049866 conn(0x55f2e26c9390 msgr2=0x55f2e27afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:06.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.516+0000 7f02f1600700 0 client.17493 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:28:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.517+0000 7f031d3e8b80 1 -- 172.21.15.43:0/983049866 shutdown_connections 2024-05-27T07:28:06.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.518+0000 7f031d3e8b80 1 -- 172.21.15.43:0/983049866 wait complete. 2024-05-27T07:28:06.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.518+0000 7f031d3e8b80 20 client.17493 trim_cache size 0 max 16384 2024-05-27T07:28:06.508 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:28:06.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.519+0000 7f030e800700 1 -- 172.21.15.43:0/1210345967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02ec0b2b40 msgr2=0x7f02ec0b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:06.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.519+0000 7f030e800700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02ec0b2b40 0x7f02ec0b5050 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0300009900 tx=0x7f030000b040 comp rx=0 tx=0).stop 2024-05-27T07:28:06.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.519+0000 7f030e800700 1 -- 172.21.15.43:0/1210345967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0308108f10 msgr2=0x7f03081a2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:06.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.519+0000 7f030e800700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0308108f10 0x7f03081a2800 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f02f400e9c0 tx=0x7f02f400ecd0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.520+0000 7f030e800700 1 -- 172.21.15.43:0/1210345967 shutdown_connections 2024-05-27T07:28:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.520+0000 7f030e800700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f02ec0b2b40 0x7f02ec0b5050 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.520+0000 7f030e800700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0308109860 0x7f03081ad170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.520+0000 7f030e800700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03081045e0 0x7f03081a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.520+0000 7f030e800700 1 --2- 172.21.15.43:0/1210345967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0308108f10 0x7f03081a2800 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.520+0000 7f030e800700 1 -- 172.21.15.43:0/1210345967 >> 172.21.15.43:0/1210345967 conn(0x7f0308100030 msgr2=0x7f0308108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:06.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.520+0000 7f030e800700 1 -- 172.21.15.43:0/1210345967 shutdown_connections 2024-05-27T07:28:06.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:06.520+0000 7f030e800700 1 -- 172.21.15.43:0/1210345967 wait complete. 2024-05-27T07:28:06.763 INFO:tasks.workunit.client.0.smithi043.stdout: 10 872 30.93 MB/sec warmup 7 sec latency 312.195 ms 2024-05-27T07:28:07.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:28:07.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:07 smithi132 ceph-mon[35283]: pgmap v477: 129 pgs: 129 active+clean; 359 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 21 MiB/s wr, 177 op/s 2024-05-27T07:28:07.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:07 smithi132 ceph-mon[35283]: osdmap e137: 12 total, 12 up, 12 in 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:07 smithi043 ceph-mon[31531]: pgmap v477: 129 pgs: 129 active+clean; 359 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 21 MiB/s wr, 177 op/s 2024-05-27T07:28:07.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:07 smithi043 ceph-mon[31531]: osdmap e137: 12 total, 12 up, 12 in 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:07 smithi179 ceph-mon[35888]: pgmap v477: 129 pgs: 129 active+clean; 359 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 21 MiB/s wr, 177 op/s 2024-05-27T07:28:07.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:07 smithi179 ceph-mon[35888]: osdmap e137: 12 total, 12 up, 12 in 2024-05-27T07:28:07.763 INFO:tasks.workunit.client.0.smithi043.stdout: 10 926 27.19 MB/sec warmup 8 sec latency 981.988 ms 2024-05-27T07:28:07.835 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:08.763 INFO:tasks.workunit.client.0.smithi043.stdout: 10 983 24.70 MB/sec warmup 9 sec latency 1982.035 ms 2024-05-27T07:28:09.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:09 smithi043 ceph-mon[31531]: pgmap v479: 129 pgs: 129 active+clean; 359 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 MiB/s wr, 221 op/s 2024-05-27T07:28:09.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:09 smithi179 ceph-mon[35888]: pgmap v479: 129 pgs: 129 active+clean; 359 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 MiB/s wr, 221 op/s 2024-05-27T07:28:09.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 1029 22.65 MB/sec warmup 10 sec latency 2982.117 ms 2024-05-27T07:28:09.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:09 smithi132 ceph-mon[35283]: pgmap v479: 129 pgs: 129 active+clean; 359 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 26 MiB/s wr, 221 op/s 2024-05-27T07:28:10.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 1116 20.68 MB/sec warmup 11 sec latency 3982.207 ms 2024-05-27T07:28:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:11 smithi043 ceph-mon[31531]: pgmap v480: 129 pgs: 129 active+clean; 359 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 MiB/s wr, 163 op/s 2024-05-27T07:28:11.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:11 smithi179 ceph-mon[35888]: pgmap v480: 129 pgs: 129 active+clean; 359 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 MiB/s wr, 163 op/s 2024-05-27T07:28:11.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 1195 19.26 MB/sec warmup 12 sec latency 4982.253 ms 2024-05-27T07:28:11.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:11 smithi132 ceph-mon[35283]: pgmap v480: 129 pgs: 129 active+clean; 359 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 MiB/s wr, 163 op/s 2024-05-27T07:28:12.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.268+0000 7f4281400700 1 -- 172.21.15.43:0/3399324552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c10e6f0 msgr2=0x7f427c110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:12.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.268+0000 7f4281400700 1 --2- 172.21.15.43:0/3399324552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c10e6f0 0x7f427c110bd0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f427400b0a0 tx=0x7f427400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.268+0000 7f4281400700 1 -- 172.21.15.43:0/3399324552 shutdown_connections 2024-05-27T07:28:12.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.268+0000 7f4281400700 1 --2- 172.21.15.43:0/3399324552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c10e6f0 0x7f427c110bd0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.268+0000 7f4281400700 1 --2- 172.21.15.43:0/3399324552 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f427c105280 0x7f427c105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.268+0000 7f4281400700 1 --2- 172.21.15.43:0/3399324552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f427c105bd0 0x7f427c10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.268+0000 7f4281400700 1 -- 172.21.15.43:0/3399324552 >> 172.21.15.43:0/3399324552 conn(0x7f427c078ab0 msgr2=0x7f427c078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.268+0000 7f4281400700 1 -- 172.21.15.43:0/3399324552 shutdown_connections 2024-05-27T07:28:12.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.268+0000 7f4281400700 1 -- 172.21.15.43:0/3399324552 wait complete. 2024-05-27T07:28:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.270+0000 7f4281400700 1 Processor -- start 2024-05-27T07:28:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.270+0000 7f4281400700 1 -- start start 2024-05-27T07:28:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.271+0000 7f4281400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c105280 0x7f427c1060f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.271+0000 7f4281400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f427c105bd0 0x7f427c106630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.271+0000 7f4281400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f427c10e6f0 0x7f427c106b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.271+0000 7f4281400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f427c114150 con 0x7f427c105280 2024-05-27T07:28:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.271+0000 7f4281400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f427c113fd0 con 0x7f427c10e6f0 2024-05-27T07:28:12.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.271+0000 7f4281400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f427c1142d0 con 0x7f427c105bd0 2024-05-27T07:28:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.271+0000 7f427be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c105280 0x7f427c1060f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.271+0000 7f427be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c105280 0x7f427c1060f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38482/0 (socket says 172.21.15.43:38482) 2024-05-27T07:28:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.271+0000 7f427be00700 1 -- 172.21.15.43:0/54475550 learned_addr learned my addr 172.21.15.43:0/54475550 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.272+0000 7f4280a00700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f427c10e6f0 0x7f427c106b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.272+0000 7f427b400700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f427c105bd0 0x7f427c106630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.272+0000 7f4280a00700 1 -- 172.21.15.43:0/54475550 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f427c105bd0 msgr2=0x7f427c106630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.272+0000 7f4280a00700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f427c105bd0 0x7f427c106630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.272+0000 7f4280a00700 1 -- 172.21.15.43:0/54475550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c105280 msgr2=0x7f427c1060f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:12.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.272+0000 7f4280a00700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c105280 0x7f427c1060f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.272+0000 7f4280a00700 1 -- 172.21.15.43:0/54475550 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c1ad2a0 con 0x7f427c10e6f0 2024-05-27T07:28:12.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.273+0000 7f4280a00700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f427c10e6f0 0x7f427c106b70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f427400b0a0 tx=0x7f4274004d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:12.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.273+0000 7f4278c00700 1 -- 172.21.15.43:0/54475550 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274021070 con 0x7f427c10e6f0 2024-05-27T07:28:12.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.273+0000 7f4281400700 1 -- 172.21.15.43:0/54475550 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427c1ad4d0 con 0x7f427c10e6f0 2024-05-27T07:28:12.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.273+0000 7f4281400700 1 -- 172.21.15.43:0/54475550 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427c1ada00 con 0x7f427c10e6f0 2024-05-27T07:28:12.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.274+0000 7f4278c00700 1 -- 172.21.15.43:0/54475550 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42740075c0 con 0x7f427c10e6f0 2024-05-27T07:28:12.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.274+0000 7f4278c00700 1 -- 172.21.15.43:0/54475550 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427401d8e0 con 0x7f427c10e6f0 2024-05-27T07:28:12.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.278+0000 7f4278c00700 1 -- 172.21.15.43:0/54475550 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4274024070 con 0x7f427c10e6f0 2024-05-27T07:28:12.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.278+0000 7f429148bb80 1 Processor -- start 2024-05-27T07:28:12.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.279+0000 7f429148bb80 1 -- start start 2024-05-27T07:28:12.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.279+0000 7f429148bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413740b9f0 0x56413749f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.279+0000 7f429148bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x56413749fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.279+0000 7f429148bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641374a92c0 0x5641374ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.279+0000 7f429148bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5641373cce40 con 0x5641374a92c0 2024-05-27T07:28:12.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.279+0000 7f429148bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5641373bc9e0 con 0x56413740b9f0 2024-05-27T07:28:12.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.279+0000 7f429148bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5641373c0a20 con 0x56413731fe40 2024-05-27T07:28:12.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.280+0000 7f427b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413740b9f0 0x56413749f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.280+0000 7f427b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413740b9f0 0x56413749f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58460/0 (socket says 172.21.15.43:58460) 2024-05-27T07:28:12.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.280+0000 7f427b400700 1 -- 172.21.15.43:0/2093190120 learned_addr learned my addr 172.21.15.43:0/2093190120 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.280+0000 7f4278c00700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42600b2b20 0x7f42600b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.280+0000 7f427be00700 1 --2- 172.21.15.43:0/2093190120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x56413749fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.280+0000 7f427b400700 1 --2- 172.21.15.43:0/2093190120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641374a92c0 0x5641374ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.281+0000 7f427b400700 1 -- 172.21.15.43:0/2093190120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 msgr2=0x56413749fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.281+0000 7f427b400700 1 --2- 172.21.15.43:0/2093190120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x56413749fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.281+0000 7f427b400700 1 -- 172.21.15.43:0/2093190120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641374a92c0 msgr2=0x5641374ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.281+0000 7f427b400700 1 --2- 172.21.15.43:0/2093190120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641374a92c0 0x5641374ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.281+0000 7f427b400700 1 -- 172.21.15.43:0/2093190120 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564137385420 con 0x56413740b9f0 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.281+0000 7f4278c00700 1 -- 172.21.15.43:0/54475550 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f42740e3530 con 0x7f427c10e6f0 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.281+0000 7f4280a00700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42600b2b20 0x7f42600b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.281+0000 7f427b400700 1 --2- 172.21.15.43:0/2093190120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413740b9f0 0x56413749f750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4268005c50 tx=0x7f4268007810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c44dcd578e947ba4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:12.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.282+0000 7f4280a00700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42600b2b20 0x7f42600b5030 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f427400d010 tx=0x7f4274009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:12.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.282+0000 7f4271600700 1 -- 172.21.15.43:0/2093190120 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268004a30 con 0x56413740b9f0 2024-05-27T07:28:12.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.283+0000 7f4271600700 1 -- 172.21.15.43:0/2093190120 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42680054b0 con 0x56413740b9f0 2024-05-27T07:28:12.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.283+0000 7f4271600700 1 -- 172.21.15.43:0/2093190120 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426800f790 con 0x56413740b9f0 2024-05-27T07:28:12.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.284+0000 7f429148bb80 1 -- 172.21.15.43:0/2093190120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413740b9f0 msgr2=0x56413749f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:12.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.284+0000 7f429148bb80 1 --2- 172.21.15.43:0/2093190120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413740b9f0 0x56413749f750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4268005c50 tx=0x7f4268007810 comp rx=0 tx=0).stop 2024-05-27T07:28:12.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.284+0000 7f429148bb80 1 -- 172.21.15.43:0/2093190120 shutdown_connections 2024-05-27T07:28:12.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.284+0000 7f429148bb80 1 --2- 172.21.15.43:0/2093190120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641374a92c0 0x5641374ab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.284+0000 7f429148bb80 1 --2- 172.21.15.43:0/2093190120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413740b9f0 0x56413749f750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.284+0000 7f429148bb80 1 --2- 172.21.15.43:0/2093190120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x56413749fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.284+0000 7f429148bb80 1 -- 172.21.15.43:0/2093190120 >> 172.21.15.43:0/2093190120 conn(0x5641373b9390 msgr2=0x56413749a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:12.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.284+0000 7f429148bb80 1 -- 172.21.15.43:0/2093190120 shutdown_connections 2024-05-27T07:28:12.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.284+0000 7f429148bb80 1 -- 172.21.15.43:0/2093190120 wait complete. 2024-05-27T07:28:12.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.285+0000 7f429148bb80 1 -- start start 2024-05-27T07:28:12.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.285+0000 7f429148bb80 1 Processor -- start 2024-05-27T07:28:12.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.286+0000 7f429148bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x5641374a6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.286+0000 7f429148bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413740b9f0 0x5641374a6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.286+0000 7f429148bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 0x5641374a6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.286+0000 7f429148bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5641372c2fa0 con 0x56413740b9f0 2024-05-27T07:28:12.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.286+0000 7f429148bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56413735d760 con 0x5641374a92c0 2024-05-27T07:28:12.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.286+0000 7f429148bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5641374abe00 con 0x56413731fe40 2024-05-27T07:28:12.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.286+0000 7f429148bb80 10 client.0 _ll_register_callbacks cb 0x564137366c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:12.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.287+0000 7f427be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413740b9f0 0x5641374a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.287+0000 7f427b400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x5641374a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.287+0000 7f427be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413740b9f0 0x5641374a6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38494/0 (socket says 172.21.15.43:38494) 2024-05-27T07:28:12.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.287+0000 7f427be00700 1 -- 172.21.15.43:0/3707419470 learned_addr learned my addr 172.21.15.43:0/3707419470 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:12.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.287+0000 7f427b400700 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 0x5641374a6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.287+0000 7f427be00700 1 -- 172.21.15.43:0/3707419470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 msgr2=0x5641374a6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:12.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.287+0000 7f427be00700 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x5641374a6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.287+0000 7f427be00700 1 -- 172.21.15.43:0/3707419470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 msgr2=0x5641374a6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:12.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.287+0000 7f427be00700 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 0x5641374a6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.287+0000 7f427be00700 1 -- 172.21.15.43:0/3707419470 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56413730c8d0 con 0x56413740b9f0 2024-05-27T07:28:12.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.288+0000 7f427b400700 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 0x5641374a6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:12.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.288+0000 7f427be00700 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413740b9f0 0x5641374a6790 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f426c012700 tx=0x7f426c012a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82eff64aa4a00d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:12.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.288+0000 7f4266000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:28:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.289+0000 7f4266000700 1 -- 172.21.15.43:0/3707419470 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c004460 con 0x56413740b9f0 2024-05-27T07:28:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.289+0000 7f4266000700 1 -- 172.21.15.43:0/3707419470 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f426c003680 con 0x56413740b9f0 2024-05-27T07:28:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.289+0000 7f429148bb80 1 -- 172.21.15.43:0/3707419470 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42680054b0 con 0x56413740b9f0 2024-05-27T07:28:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.290+0000 7f4266000700 1 -- 172.21.15.43:0/3707419470 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) ==== 24+0+0 (secure 0 0 0) 0x7f426c003880 con 0x56413740b9f0 2024-05-27T07:28:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.290+0000 7f429148bb80 10 client.17505 fetch_fsmap learned FSMap version 114 2024-05-27T07:28:12.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.290+0000 7f429148bb80 1 -- 172.21.15.43:0/3707419470 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=4+}) -- 0x56413738d060 con 0x56413740b9f0 2024-05-27T07:28:12.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.291+0000 7f4266000700 1 -- 172.21.15.43:0/3707419470 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42680054b0 con 0x56413740b9f0 2024-05-27T07:28:12.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.291+0000 7f4266000700 1 -- 172.21.15.43:0/3707419470 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 114) ==== 4812+0+0 (secure 0 0 0) 0x7f426c019070 con 0x56413740b9f0 2024-05-27T07:28:12.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.291+0000 7f429148bb80 10 client.17505 fetch_fsmap finished waiting for FSMap version 114 2024-05-27T07:28:12.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.291+0000 7f429148bb80 10 client.17505 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:28:12.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.291+0000 7f429148bb80 20 client.17505 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:12.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.291+0000 7f429148bb80 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5641374b9210 0x5641374bb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.291+0000 7f429148bb80 4 client.17505 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:12.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.291+0000 7f429148bb80 1 -- 172.21.15.43:0/3707419470 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641373ced90 con 0x5641374b9210 2024-05-27T07:28:12.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.291+0000 7f427b400700 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5641374b9210 0x5641374bb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.292+0000 7f427b400700 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5641374b9210 0x5641374bb660 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6c0ac62abebea7 server_cookie=7fdf31ed2d12c290 in_seq=0 out_seq=0 2024-05-27T07:28:12.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.292+0000 7f4266000700 10 client.17505 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:28:12.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.295+0000 7f4266000700 1 -- 172.21.15.43:0/3707419470 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5641373ced90 con 0x5641374b9210 2024-05-27T07:28:12.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.295+0000 7f4266000700 10 client.17505 handle_command_reply: tid=0 2024-05-27T07:28:12.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.295+0000 7f429148bb80 1 client.17505 shutdown 2024-05-27T07:28:12.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f429148bb80 1 -- 172.21.15.43:0/3707419470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413740b9f0 msgr2=0x5641374a6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:12.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f429148bb80 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413740b9f0 0x5641374a6790 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f426c012700 tx=0x7f426c012a10 comp rx=0 tx=0).stop 2024-05-27T07:28:12.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f429148bb80 1 -- 172.21.15.43:0/3707419470 shutdown_connections 2024-05-27T07:28:12.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f429148bb80 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x5641374b9210 0x5641374bb660 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f429148bb80 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 0x5641374a6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f429148bb80 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x5641374a6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f429148bb80 1 --2- 172.21.15.43:0/3707419470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413740b9f0 0x5641374a6790 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f4266000700 0 client.17505 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:28:12.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f429148bb80 1 -- 172.21.15.43:0/3707419470 >> 172.21.15.43:0/3707419470 conn(0x5641373b9390 msgr2=0x56413749a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:12.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f429148bb80 1 -- 172.21.15.43:0/3707419470 shutdown_connections 2024-05-27T07:28:12.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.296+0000 7f429148bb80 1 -- 172.21.15.43:0/3707419470 wait complete. 2024-05-27T07:28:12.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.297+0000 7f429148bb80 20 client.17505 trim_cache size 0 max 16384 2024-05-27T07:28:12.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.327+0000 7f429148bb80 1 Processor -- start 2024-05-27T07:28:12.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.327+0000 7f429148bb80 1 -- start start 2024-05-27T07:28:12.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.328+0000 7f429148bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413731fe40 0x5641374a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.328+0000 7f429148bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5641374a92c0 0x5641374b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.328+0000 7f429148bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413749a740 0x5641374c0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.328+0000 7f429148bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5641374a5cd0 con 0x56413731fe40 2024-05-27T07:28:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.328+0000 7f429148bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f425c00fae0 con 0x56413749a740 2024-05-27T07:28:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.328+0000 7f429148bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f425c00d750 con 0x5641374a92c0 2024-05-27T07:28:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.328+0000 7f427b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413731fe40 0x5641374a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.328+0000 7f427b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413731fe40 0x5641374a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38500/0 (socket says 172.21.15.43:38500) 2024-05-27T07:28:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.328+0000 7f427b400700 1 -- 172.21.15.43:0/207547320 learned_addr learned my addr 172.21.15.43:0/207547320 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.329+0000 7f427be00700 1 --2- 172.21.15.43:0/207547320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5641374a92c0 0x5641374b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.329+0000 7f427b400700 1 --2- 172.21.15.43:0/207547320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413749a740 0x5641374c0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.329+0000 7f427b400700 1 -- 172.21.15.43:0/207547320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5641374a92c0 msgr2=0x5641374b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:12.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.329+0000 7f427b400700 1 --2- 172.21.15.43:0/207547320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5641374a92c0 0x5641374b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.329+0000 7f427b400700 1 -- 172.21.15.43:0/207547320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413749a740 msgr2=0x5641374c0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:12.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.329+0000 7f427b400700 1 --2- 172.21.15.43:0/207547320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413749a740 0x5641374c0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.329+0000 7f427b400700 1 -- 172.21.15.43:0/207547320 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641374b67d0 con 0x56413731fe40 2024-05-27T07:28:12.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.330+0000 7f427b400700 1 --2- 172.21.15.43:0/207547320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413731fe40 0x5641374a18a0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4268005fd0 tx=0x7f4268004f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf40fc87aa05babf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:12.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.330+0000 7f4270c00700 1 -- 172.21.15.43:0/207547320 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268027040 con 0x56413731fe40 2024-05-27T07:28:12.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.330+0000 7f4270c00700 1 -- 172.21.15.43:0/207547320 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4268005390 con 0x56413731fe40 2024-05-27T07:28:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.330+0000 7f4270c00700 1 -- 172.21.15.43:0/207547320 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426800f8b0 con 0x56413731fe40 2024-05-27T07:28:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.332+0000 7f429148bb80 1 -- 172.21.15.43:0/207547320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413731fe40 msgr2=0x5641374a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:12.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.332+0000 7f429148bb80 1 --2- 172.21.15.43:0/207547320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413731fe40 0x5641374a18a0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4268005fd0 tx=0x7f4268004f40 comp rx=0 tx=0).stop 2024-05-27T07:28:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.332+0000 7f429148bb80 1 -- 172.21.15.43:0/207547320 shutdown_connections 2024-05-27T07:28:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.332+0000 7f429148bb80 1 --2- 172.21.15.43:0/207547320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56413749a740 0x5641374c0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.332+0000 7f429148bb80 1 --2- 172.21.15.43:0/207547320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413731fe40 0x5641374a18a0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.332+0000 7f429148bb80 1 --2- 172.21.15.43:0/207547320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5641374a92c0 0x5641374b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.332+0000 7f429148bb80 1 -- 172.21.15.43:0/207547320 >> 172.21.15.43:0/207547320 conn(0x5641373b9390 msgr2=0x5641374a1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.332+0000 7f429148bb80 1 -- 172.21.15.43:0/207547320 shutdown_connections 2024-05-27T07:28:12.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.332+0000 7f429148bb80 1 -- 172.21.15.43:0/207547320 wait complete. 2024-05-27T07:28:12.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.333+0000 7f429148bb80 1 -- start start 2024-05-27T07:28:12.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.334+0000 7f429148bb80 1 Processor -- start 2024-05-27T07:28:12.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.334+0000 7f429148bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x56413751c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.334+0000 7f429148bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413749a740 0x56413751c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.335+0000 7f429148bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 0x56413751cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.335+0000 7f429148bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f425c00de80 con 0x56413749a740 2024-05-27T07:28:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.335+0000 7f429148bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f425c00f410 con 0x5641374a92c0 2024-05-27T07:28:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.335+0000 7f429148bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f425c00e5b0 con 0x56413731fe40 2024-05-27T07:28:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.335+0000 7f429148bb80 10 client.0 _ll_register_callbacks cb 0x564137366c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.335+0000 7f427be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413749a740 0x56413751c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.335+0000 7f427b400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x56413751c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.335+0000 7f427be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413749a740 0x56413751c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38512/0 (socket says 172.21.15.43:38512) 2024-05-27T07:28:12.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.335+0000 7f427be00700 1 -- 172.21.15.43:0/1667406123 learned_addr learned my addr 172.21.15.43:0/1667406123 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.336+0000 7f427b400700 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 0x56413751cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.336+0000 7f427be00700 1 -- 172.21.15.43:0/1667406123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 msgr2=0x56413751c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.336+0000 7f427be00700 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x56413751c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.336+0000 7f427be00700 1 -- 172.21.15.43:0/1667406123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 msgr2=0x56413751cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.336+0000 7f427be00700 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 0x56413751cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.336+0000 7f427be00700 1 -- 172.21.15.43:0/1667406123 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641374b0350 con 0x56413749a740 2024-05-27T07:28:12.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.337+0000 7f427be00700 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413749a740 0x56413751c630 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f426c000c00 tx=0x7f426c026ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510db059ef3461c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:12.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.337+0000 7f4266000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:28:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.337+0000 7f4266000700 1 -- 172.21.15.43:0/1667406123 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c0054e0 con 0x56413749a740 2024-05-27T07:28:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.337+0000 7f4266000700 1 -- 172.21.15.43:0/1667406123 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f426c005680 con 0x56413749a740 2024-05-27T07:28:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.337+0000 7f429148bb80 1 -- 172.21.15.43:0/1667406123 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4268005390 con 0x56413749a740 2024-05-27T07:28:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.337+0000 7f4266000700 1 -- 172.21.15.43:0/1667406123 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) ==== 24+0+0 (secure 0 0 0) 0x7f426c003a20 con 0x56413749a740 2024-05-27T07:28:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.337+0000 7f429148bb80 10 client.17511 fetch_fsmap learned FSMap version 114 2024-05-27T07:28:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.337+0000 7f429148bb80 1 -- 172.21.15.43:0/1667406123 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=4+}) -- 0x5641374b15b0 con 0x56413749a740 2024-05-27T07:28:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.338+0000 7f4266000700 1 -- 172.21.15.43:0/1667406123 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4268005390 con 0x56413749a740 2024-05-27T07:28:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.338+0000 7f4266000700 1 -- 172.21.15.43:0/1667406123 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 114) ==== 4812+0+0 (secure 0 0 0) 0x7f426c019070 con 0x56413749a740 2024-05-27T07:28:12.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.338+0000 7f429148bb80 10 client.17511 fetch_fsmap finished waiting for FSMap version 114 2024-05-27T07:28:12.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.338+0000 7f429148bb80 10 client.17511 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:28:12.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.338+0000 7f429148bb80 20 client.17511 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:12.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.338+0000 7f429148bb80 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564137534d40 0x564137537160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:12.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.338+0000 7f429148bb80 4 client.17511 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:12.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.338+0000 7f429148bb80 1 -- 172.21.15.43:0/1667406123 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641373c8570 con 0x564137534d40 2024-05-27T07:28:12.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.339+0000 7f427b400700 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564137534d40 0x564137537160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:12.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.340+0000 7f427b400700 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564137534d40 0x564137537160 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3881434bface153a server_cookie=2ced77a980e104e1 in_seq=0 out_seq=0 2024-05-27T07:28:12.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.340+0000 7f4266000700 10 client.17511 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:28:12.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.340+0000 7f4266000700 1 -- 172.21.15.43:0/1667406123 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641373c8570 con 0x564137534d40 2024-05-27T07:28:12.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.340+0000 7f4266000700 10 client.17511 handle_command_reply: tid=0 2024-05-27T07:28:12.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.340+0000 7f429148bb80 1 client.17511 shutdown 2024-05-27T07:28:12.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 1 -- 172.21.15.43:0/1667406123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413749a740 msgr2=0x56413751c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:12.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413749a740 0x56413751c630 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f426c000c00 tx=0x7f426c026ed0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 1 -- 172.21.15.43:0/1667406123 shutdown_connections 2024-05-27T07:28:12.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564137534d40 0x564137537160 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5641374a92c0 0x56413751cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56413731fe40 0x56413751c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 1 --2- 172.21.15.43:0/1667406123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56413749a740 0x56413751c630 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 1 -- 172.21.15.43:0/1667406123 >> 172.21.15.43:0/1667406123 conn(0x5641373b9390 msgr2=0x5641374a1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f4266000700 0 client.17511 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:28:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 1 -- 172.21.15.43:0/1667406123 shutdown_connections 2024-05-27T07:28:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 1 -- 172.21.15.43:0/1667406123 wait complete. 2024-05-27T07:28:12.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.341+0000 7f429148bb80 20 client.17511 trim_cache size 0 max 16384 2024-05-27T07:28:12.333 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:28:12.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.342+0000 7f4281400700 1 -- 172.21.15.43:0/54475550 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42600b2b20 msgr2=0x7f42600b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:12.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.342+0000 7f4281400700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42600b2b20 0x7f42600b5030 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f427400d010 tx=0x7f4274009040 comp rx=0 tx=0).stop 2024-05-27T07:28:12.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.342+0000 7f4281400700 1 -- 172.21.15.43:0/54475550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f427c10e6f0 msgr2=0x7f427c106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:12.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.342+0000 7f4281400700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f427c10e6f0 0x7f427c106b70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f427400b0a0 tx=0x7f4274004d50 comp rx=0 tx=0).stop 2024-05-27T07:28:12.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.342+0000 7f4281400700 1 -- 172.21.15.43:0/54475550 shutdown_connections 2024-05-27T07:28:12.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.342+0000 7f4281400700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42600b2b20 0x7f42600b5030 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.342+0000 7f4281400700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f427c10e6f0 0x7f427c106b70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.342+0000 7f4281400700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c105280 0x7f427c1060f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.342+0000 7f4281400700 1 --2- 172.21.15.43:0/54475550 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f427c105bd0 0x7f427c106630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:12.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.343+0000 7f4281400700 1 -- 172.21.15.43:0/54475550 >> 172.21.15.43:0/54475550 conn(0x7f427c078ab0 msgr2=0x7f427c102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:12.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.343+0000 7f4281400700 1 -- 172.21.15.43:0/54475550 shutdown_connections 2024-05-27T07:28:12.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:12.343+0000 7f4281400700 1 -- 172.21.15.43:0/54475550 wait complete. 2024-05-27T07:28:12.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 1305 19.13 MB/sec warmup 13 sec latency 5982.323 ms 2024-05-27T07:28:13.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:28:13.694 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:13.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:13 smithi043 ceph-mon[31531]: pgmap v481: 129 pgs: 129 active+clean; 363 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 MiB/s wr, 191 op/s 2024-05-27T07:28:13.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:13 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:28:13.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:13 smithi179 ceph-mon[35888]: pgmap v481: 129 pgs: 129 active+clean; 363 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 MiB/s wr, 191 op/s 2024-05-27T07:28:13.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:13 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:28:13.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 1385 17.81 MB/sec warmup 14 sec latency 6982.378 ms 2024-05-27T07:28:13.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:13 smithi132 ceph-mon[35283]: pgmap v481: 129 pgs: 129 active+clean; 363 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 MiB/s wr, 191 op/s 2024-05-27T07:28:13.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:13 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:28:14.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 1432 16.65 MB/sec warmup 15 sec latency 7982.467 ms 2024-05-27T07:28:15.640 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:15 smithi043 ceph-mon[31531]: pgmap v482: 129 pgs: 129 active+clean; 363 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 180 op/s 2024-05-27T07:28:15.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:15 smithi179 ceph-mon[35888]: pgmap v482: 129 pgs: 129 active+clean; 363 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 180 op/s 2024-05-27T07:28:15.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 1528 15.83 MB/sec warmup 16 sec latency 8982.589 ms 2024-05-27T07:28:15.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:15 smithi132 ceph-mon[35283]: pgmap v482: 129 pgs: 129 active+clean; 363 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 180 op/s 2024-05-27T07:28:16.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.128+0000 7ffb10a00700 1 -- 172.21.15.43:0/2378079228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb0c10af40 msgr2=0x7ffb0c10b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:16.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.128+0000 7ffb10a00700 1 --2- 172.21.15.43:0/2378079228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb0c10af40 0x7ffb0c10b350 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800b0d0 tx=0x7ffaf800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.128+0000 7ffb10a00700 1 -- 172.21.15.43:0/2378079228 shutdown_connections 2024-05-27T07:28:16.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.128+0000 7ffb10a00700 1 --2- 172.21.15.43:0/2378079228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c10b890 0x7ffb0c1131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.128+0000 7ffb10a00700 1 --2- 172.21.15.43:0/2378079228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb0c106610 0x7ffb0c106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.128+0000 7ffb10a00700 1 --2- 172.21.15.43:0/2378079228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb0c10af40 0x7ffb0c10b350 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.128+0000 7ffb10a00700 1 -- 172.21.15.43:0/2378079228 >> 172.21.15.43:0/2378079228 conn(0x7ffb0c102020 msgr2=0x7ffb0c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.128+0000 7ffb10a00700 1 -- 172.21.15.43:0/2378079228 shutdown_connections 2024-05-27T07:28:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.129+0000 7ffb10a00700 1 -- 172.21.15.43:0/2378079228 wait complete. 2024-05-27T07:28:16.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.130+0000 7ffb10a00700 1 Processor -- start 2024-05-27T07:28:16.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.130+0000 7ffb10a00700 1 -- start start 2024-05-27T07:28:16.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.131+0000 7ffb10a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb0c106610 0x7ffb0c110690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.131+0000 7ffb10a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c10af40 0x7ffb0c110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.131+0000 7ffb10a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb0c10b890 0x7ffb0c111110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.131+0000 7ffb10a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c1163a0 con 0x7ffb0c106610 2024-05-27T07:28:16.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.131+0000 7ffb10a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c116220 con 0x7ffb0c10af40 2024-05-27T07:28:16.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.131+0000 7ffb10a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c116520 con 0x7ffb0c10b890 2024-05-27T07:28:16.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.131+0000 7ffb0b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb0c106610 0x7ffb0c110690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.131+0000 7ffb0b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb0c106610 0x7ffb0c110690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37360/0 (socket says 172.21.15.43:37360) 2024-05-27T07:28:16.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.131+0000 7ffb0b400700 1 -- 172.21.15.43:0/24567034 learned_addr learned my addr 172.21.15.43:0/24567034 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:16.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.132+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c10af40 0x7ffb0c110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.132+0000 7ffb0be00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb0c10b890 0x7ffb0c111110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.133+0000 7ffb0aa00700 1 -- 172.21.15.43:0/24567034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb0c10b890 msgr2=0x7ffb0c111110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:16.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.133+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb0c10b890 0x7ffb0c111110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.133+0000 7ffb0aa00700 1 -- 172.21.15.43:0/24567034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb0c106610 msgr2=0x7ffb0c110690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:16.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.133+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb0c106610 0x7ffb0c110690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.133+0000 7ffb0aa00700 1 -- 172.21.15.43:0/24567034 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0c1ad2f0 con 0x7ffb0c10af40 2024-05-27T07:28:16.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.133+0000 7ffb0b400700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb0c106610 0x7ffb0c110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:16.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.134+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c10af40 0x7ffb0c110bd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8000f80 tx=0x7ffaf8007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:16.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.135+0000 7ffb03e00700 1 -- 172.21.15.43:0/24567034 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf801f020 con 0x7ffb0c10af40 2024-05-27T07:28:16.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.135+0000 7ffb03e00700 1 -- 172.21.15.43:0/24567034 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffaf8003c00 con 0x7ffb0c10af40 2024-05-27T07:28:16.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.135+0000 7ffb03e00700 1 -- 172.21.15.43:0/24567034 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf801b6b0 con 0x7ffb0c10af40 2024-05-27T07:28:16.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.135+0000 7ffb10a00700 1 -- 172.21.15.43:0/24567034 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb0c1ad580 con 0x7ffb0c10af40 2024-05-27T07:28:16.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.135+0000 7ffb10a00700 1 -- 172.21.15.43:0/24567034 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb0c1ad980 con 0x7ffb0c10af40 2024-05-27T07:28:16.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.143+0000 7ffb03e00700 1 -- 172.21.15.43:0/24567034 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ffaf8022020 con 0x7ffb0c10af40 2024-05-27T07:28:16.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.144+0000 7ffb20b8cb80 1 Processor -- start 2024-05-27T07:28:16.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.145+0000 7ffb20b8cb80 1 -- start start 2024-05-27T07:28:16.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.145+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd380ae40 0x561dd3988ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.145+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989410 0x561dd3993ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.145+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3996020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.145+0000 7ffb20b8cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561dd38b7e40 con 0x561dd3989820 2024-05-27T07:28:16.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.145+0000 7ffb20b8cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561dd38a79e0 con 0x561dd3989410 2024-05-27T07:28:16.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.145+0000 7ffb20b8cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561dd38aba20 con 0x561dd380ae40 2024-05-27T07:28:16.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.145+0000 7ffb03e00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffaf00b2a70 0x7ffaf00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.146+0000 7ffb03e00700 1 -- 172.21.15.43:0/24567034 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7ffaf800f020 con 0x7ffb0c10af40 2024-05-27T07:28:16.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.147+0000 7ffb0aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3996020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.147+0000 7ffb0aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3996020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37374/0 (socket says 172.21.15.43:37374) 2024-05-27T07:28:16.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.147+0000 7ffb0aa00700 1 -- 172.21.15.43:0/1985818089 learned_addr learned my addr 172.21.15.43:0/1985818089 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:16.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.147+0000 7ffb0b400700 1 --2- 172.21.15.43:0/1985818089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989410 0x561dd3993ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.147+0000 7ffb0be00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffaf00b2a70 0x7ffaf00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.147+0000 7ffb0be00700 1 --2- 172.21.15.43:0/1985818089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd380ae40 0x561dd3988ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.148+0000 7ffb0b400700 1 -- 172.21.15.43:0/1985818089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd380ae40 msgr2=0x561dd3988ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:16.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.148+0000 7ffb0b400700 1 --2- 172.21.15.43:0/1985818089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd380ae40 0x561dd3988ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.148+0000 7ffb0b400700 1 -- 172.21.15.43:0/1985818089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 msgr2=0x561dd3996020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:16.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.148+0000 7ffb0b400700 1 --2- 172.21.15.43:0/1985818089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3996020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.148+0000 7ffb0b400700 1 -- 172.21.15.43:0/1985818089 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dd3870420 con 0x561dd3989410 2024-05-27T07:28:16.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.148+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/1985818089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3996020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:16.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.149+0000 7ffb0b400700 1 --2- 172.21.15.43:0/1985818089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989410 0x561dd3993ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffb040105e0 tx=0x7ffb0400bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4f355a1132b8e76 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:16.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.149+0000 7ffb0be00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffaf00b2a70 0x7ffaf00b4f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x561dd3985690 tx=0x7ffafc009990 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:16.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.150+0000 7ffb00c00700 1 -- 172.21.15.43:0/1985818089 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04011640 con 0x561dd3989410 2024-05-27T07:28:16.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.150+0000 7ffb00c00700 1 -- 172.21.15.43:0/1985818089 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb04011c70 con 0x561dd3989410 2024-05-27T07:28:16.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.150+0000 7ffb00c00700 1 -- 172.21.15.43:0/1985818089 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04019c60 con 0x561dd3989410 2024-05-27T07:28:16.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.152+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1985818089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989410 msgr2=0x561dd3993ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:16.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.152+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1985818089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989410 0x561dd3993ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffb040105e0 tx=0x7ffb0400bf90 comp rx=0 tx=0).stop 2024-05-27T07:28:16.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.153+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1985818089 shutdown_connections 2024-05-27T07:28:16.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.153+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1985818089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3996020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.153+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1985818089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd380ae40 0x561dd3988ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.153+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1985818089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989410 0x561dd3993ad0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.153+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1985818089 >> 172.21.15.43:0/1985818089 conn(0x561dd386afc0 msgr2=0x561dd3984d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:16.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.153+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1985818089 shutdown_connections 2024-05-27T07:28:16.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.153+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1985818089 wait complete. 2024-05-27T07:28:16.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.154+0000 7ffb20b8cb80 1 -- start start 2024-05-27T07:28:16.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.154+0000 7ffb20b8cb80 1 Processor -- start 2024-05-27T07:28:16.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd39936c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3993c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd399fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb20b8cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561dd37adfa0 con 0x561dd3989820 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb20b8cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561dd3848760 con 0x561dd380ae40 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb20b8cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561dd3996990 con 0x561dd3994140 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb20b8cb80 10 client.0 _ll_register_callbacks cb 0x561dd3851c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb0b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd39936c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb0b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd39936c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59568/0 (socket says 172.21.15.43:59568) 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb0aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3993c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.155+0000 7ffb0b400700 1 -- 172.21.15.43:0/1343230274 learned_addr learned my addr 172.21.15.43:0/1343230274 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.156+0000 7ffb0be00700 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd399fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.156+0000 7ffb0aa00700 1 -- 172.21.15.43:0/1343230274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 msgr2=0x561dd399fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:16.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.156+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd399fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.156+0000 7ffb0aa00700 1 -- 172.21.15.43:0/1343230274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 msgr2=0x561dd39936c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:16.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.156+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd39936c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.156+0000 7ffb0aa00700 1 -- 172.21.15.43:0/1343230274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dd37f78d0 con 0x561dd3989820 2024-05-27T07:28:16.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.156+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3993c00 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009fd0 tx=0x7ffaf80a6a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e0b8af49789c3a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.156+0000 7ffaf5600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:28:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.157+0000 7ffaf5600700 1 -- 172.21.15.43:0/1343230274 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80aac30 con 0x561dd3989820 2024-05-27T07:28:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.157+0000 7ffaf5600700 1 -- 172.21.15.43:0/1343230274 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7ffaf80e3380 con 0x561dd3989820 2024-05-27T07:28:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.157+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1343230274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb04011c70 con 0x561dd3989820 2024-05-27T07:28:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.157+0000 7ffaf5600700 1 -- 172.21.15.43:0/1343230274 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7ffaf8009cb0 con 0x561dd3989820 2024-05-27T07:28:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.157+0000 7ffb20b8cb80 10 client.17523 fetch_fsmap learned FSMap version 115 2024-05-27T07:28:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.157+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1343230274 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=4+}) -- 0x561dd3878060 con 0x561dd3989820 2024-05-27T07:28:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.158+0000 7ffaf5600700 1 -- 172.21.15.43:0/1343230274 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb04011c70 con 0x561dd3989820 2024-05-27T07:28:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.158+0000 7ffaf5600700 1 -- 172.21.15.43:0/1343230274 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 115) ==== 4820+0+0 (secure 0 0 0) 0x7ffaf802c060 con 0x561dd3989820 2024-05-27T07:28:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.158+0000 7ffb20b8cb80 10 client.17523 fetch_fsmap finished waiting for FSMap version 115 2024-05-27T07:28:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.158+0000 7ffb20b8cb80 10 client.17523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.158+0000 7ffb20b8cb80 20 client.17523 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.158+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561dd39a4090 0x561dd39a6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.158+0000 7ffb20b8cb80 4 client.17523 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.158+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1343230274 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dd38b9d90 con 0x561dd39a4090 2024-05-27T07:28:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.159+0000 7ffb0b400700 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561dd39a4090 0x561dd39a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.160+0000 7ffb0b400700 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561dd39a4090 0x561dd39a6540 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d8ee7ce4bee5f39 server_cookie=57a11908c464c8a in_seq=0 out_seq=0 2024-05-27T07:28:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.160+0000 7ffaf5600700 10 client.17523 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:16.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffaf5600700 1 -- 172.21.15.43:0/1343230274 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561dd38b9d90 con 0x561dd39a4090 2024-05-27T07:28:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffaf5600700 10 client.17523 handle_command_reply: tid=0 2024-05-27T07:28:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffb20b8cb80 1 client.17523 shutdown 2024-05-27T07:28:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1343230274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 msgr2=0x561dd3993c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3993c00 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009fd0 tx=0x7ffaf80a6a20 comp rx=0 tx=0).stop 2024-05-27T07:28:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1343230274 shutdown_connections 2024-05-27T07:28:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd399fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd39936c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561dd39a4090 0x561dd39a6540 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1343230274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd3993c00 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1343230274 >> 172.21.15.43:0/1343230274 conn(0x561dd386afc0 msgr2=0x561dd3983ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.163+0000 7ffaf5600700 0 client.17523 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.164+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1343230274 shutdown_connections 2024-05-27T07:28:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.164+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1343230274 wait complete. 2024-05-27T07:28:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.164+0000 7ffb20b8cb80 20 client.17523 trim_cache size 0 max 16384 2024-05-27T07:28:16.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.186+0000 7ffb20b8cb80 1 Processor -- start 2024-05-27T07:28:16.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.186+0000 7ffb20b8cb80 1 -- start start 2024-05-27T07:28:16.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.186+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd3995b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.186+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd398d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.186+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd399fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.186+0000 7ffb20b8cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c1b76f0 con 0x561dd3989820 2024-05-27T07:28:16.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.186+0000 7ffb20b8cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c1b5300 con 0x561dd380ae40 2024-05-27T07:28:16.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.186+0000 7ffb20b8cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c1b7020 con 0x561dd3994140 2024-05-27T07:28:16.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.187+0000 7ffb0aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd398d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.187+0000 7ffb0b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd3995b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.187+0000 7ffb0b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd3995b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59578/0 (socket says 172.21.15.43:59578) 2024-05-27T07:28:16.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.187+0000 7ffb0b400700 1 -- 172.21.15.43:0/1457401615 learned_addr learned my addr 172.21.15.43:0/1457401615 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:16.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.187+0000 7ffb0be00700 1 --2- 172.21.15.43:0/1457401615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd399fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.187+0000 7ffb0b400700 1 -- 172.21.15.43:0/1457401615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 msgr2=0x561dd399fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:16.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.187+0000 7ffb0b400700 1 --2- 172.21.15.43:0/1457401615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd399fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.187+0000 7ffb0b400700 1 -- 172.21.15.43:0/1457401615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 msgr2=0x561dd398d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:16.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.187+0000 7ffb0b400700 1 --2- 172.21.15.43:0/1457401615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd398d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.187+0000 7ffb0b400700 1 -- 172.21.15.43:0/1457401615 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dd3996870 con 0x561dd380ae40 2024-05-27T07:28:16.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.188+0000 7ffb0b400700 1 --2- 172.21.15.43:0/1457401615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd3995b30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffb040198b0 tx=0x7ffb04019840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3e2805d672bb93c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:16.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.188+0000 7ffaf7e00700 1 -- 172.21.15.43:0/1457401615 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0402b040 con 0x561dd380ae40 2024-05-27T07:28:16.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.188+0000 7ffaf7e00700 1 -- 172.21.15.43:0/1457401615 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb04019df0 con 0x561dd380ae40 2024-05-27T07:28:16.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.188+0000 7ffaf7e00700 1 -- 172.21.15.43:0/1457401615 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0401f040 con 0x561dd380ae40 2024-05-27T07:28:16.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.189+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1457401615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 msgr2=0x561dd3995b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:16.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.189+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1457401615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd3995b30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffb040198b0 tx=0x7ffb04019840 comp rx=0 tx=0).stop 2024-05-27T07:28:16.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.190+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1457401615 shutdown_connections 2024-05-27T07:28:16.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.190+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1457401615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd399fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.190+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1457401615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd380ae40 0x561dd3995b30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.190+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/1457401615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd3989820 0x561dd398d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.190+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1457401615 >> 172.21.15.43:0/1457401615 conn(0x561dd386afc0 msgr2=0x561dd39847c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:16.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.190+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1457401615 shutdown_connections 2024-05-27T07:28:16.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.190+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/1457401615 wait complete. 2024-05-27T07:28:16.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.191+0000 7ffb20b8cb80 1 -- start start 2024-05-27T07:28:16.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.191+0000 7ffb20b8cb80 1 Processor -- start 2024-05-27T07:28:16.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.192+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd380ae40 0x561dd3996c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.192+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989820 0x561dd39971d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.192+0000 7ffb0b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd380ae40 0x561dd3996c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.193+0000 7ffb20b8cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd3997710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.193+0000 7ffb20b8cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c1b6160 con 0x561dd380ae40 2024-05-27T07:28:16.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.193+0000 7ffb0b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd380ae40 0x561dd3996c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37392/0 (socket says 172.21.15.43:37392) 2024-05-27T07:28:16.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.193+0000 7ffb0b400700 1 -- 172.21.15.43:0/673415547 learned_addr learned my addr 172.21.15.43:0/673415547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:16.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.193+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/673415547 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c1b5a30 con 0x561dd3989820 2024-05-27T07:28:16.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.193+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/673415547 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c1b6890 con 0x561dd3994140 2024-05-27T07:28:16.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.193+0000 7ffb20b8cb80 10 client.0 _ll_register_callbacks cb 0x561dd3851c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:16.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.193+0000 7ffb0be00700 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd3997710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.194+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989820 0x561dd39971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.194+0000 7ffb0b400700 1 -- 172.21.15.43:0/673415547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 msgr2=0x561dd3997710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:16.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.194+0000 7ffb0b400700 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd3997710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.194+0000 7ffb0b400700 1 -- 172.21.15.43:0/673415547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989820 msgr2=0x561dd39971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:16.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.194+0000 7ffb0b400700 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989820 0x561dd39971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.194+0000 7ffb0b400700 1 -- 172.21.15.43:0/673415547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dd3a0cbc0 con 0x561dd380ae40 2024-05-27T07:28:16.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.195+0000 7ffb0b400700 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd380ae40 0x561dd3996c90 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ffb04019580 tx=0x7ffb040195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf381510392a472 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:16.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.195+0000 7ffaf5600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:28:16.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.196+0000 7ffaf5600700 1 -- 172.21.15.43:0/673415547 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0402b030 con 0x561dd380ae40 2024-05-27T07:28:16.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.196+0000 7ffaf5600700 1 -- 172.21.15.43:0/673415547 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7ffb040050b0 con 0x561dd380ae40 2024-05-27T07:28:16.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.197+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/673415547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb04019df0 con 0x561dd380ae40 2024-05-27T07:28:16.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.197+0000 7ffaf5600700 1 -- 172.21.15.43:0/673415547 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7ffb04005540 con 0x561dd380ae40 2024-05-27T07:28:16.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.197+0000 7ffb20b8cb80 10 client.17529 fetch_fsmap learned FSMap version 115 2024-05-27T07:28:16.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.198+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/673415547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=4+}) -- 0x561dd3a0dd00 con 0x561dd380ae40 2024-05-27T07:28:16.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.198+0000 7ffaf5600700 1 -- 172.21.15.43:0/673415547 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb04019df0 con 0x561dd380ae40 2024-05-27T07:28:16.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.198+0000 7ffaf5600700 1 -- 172.21.15.43:0/673415547 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 115) ==== 4820+0+0 (secure 0 0 0) 0x7ffb04003c00 con 0x561dd380ae40 2024-05-27T07:28:16.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.198+0000 7ffb20b8cb80 10 client.17529 fetch_fsmap finished waiting for FSMap version 115 2024-05-27T07:28:16.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.198+0000 7ffb20b8cb80 10 client.17529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:16.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.198+0000 7ffb20b8cb80 20 client.17529 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:16.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.198+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561dd3a0e340 0x561dd3a10760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:16.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.198+0000 7ffb20b8cb80 4 client.17529 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:28:16.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.198+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/673415547 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561dd38b3570 con 0x561dd3a0e340 2024-05-27T07:28:16.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.199+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561dd3a0e340 0x561dd3a10760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:16.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.200+0000 7ffb0aa00700 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561dd3a0e340 0x561dd3a10760 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93b25dc9f78fdd6f server_cookie=5b10b8fd618095c9 in_seq=0 out_seq=0 2024-05-27T07:28:16.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.200+0000 7ffaf5600700 10 client.17529 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:16.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.200+0000 7ffaf5600700 1 -- 172.21.15.43:0/673415547 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561dd38b3570 con 0x561dd3a0e340 2024-05-27T07:28:16.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.200+0000 7ffaf5600700 10 client.17529 handle_command_reply: tid=0 2024-05-27T07:28:16.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.200+0000 7ffb20b8cb80 1 client.17529 shutdown 2024-05-27T07:28:16.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.200+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/673415547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd380ae40 msgr2=0x561dd3996c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:16.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.200+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd380ae40 0x561dd3996c90 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ffb04019580 tx=0x7ffb040195b0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.201+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/673415547 shutdown_connections 2024-05-27T07:28:16.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.201+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561dd3994140 0x561dd3997710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.201+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561dd3a0e340 0x561dd3a10760 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.201+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561dd380ae40 0x561dd3996c90 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.201+0000 7ffb20b8cb80 1 --2- 172.21.15.43:0/673415547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561dd3989820 0x561dd39971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.201+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/673415547 >> 172.21.15.43:0/673415547 conn(0x561dd386afc0 msgr2=0x561dd3991e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:16.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.201+0000 7ffaf5600700 0 client.17529 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:16.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.202+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/673415547 shutdown_connections 2024-05-27T07:28:16.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.202+0000 7ffb20b8cb80 1 -- 172.21.15.43:0/673415547 wait complete. 2024-05-27T07:28:16.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.202+0000 7ffb20b8cb80 20 client.17529 trim_cache size 0 max 16384 2024-05-27T07:28:16.193 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:28:16.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.203+0000 7ffb10a00700 1 -- 172.21.15.43:0/24567034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffaf00b2a70 msgr2=0x7ffaf00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:16.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.203+0000 7ffb10a00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffaf00b2a70 0x7ffaf00b4f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x561dd3985690 tx=0x7ffafc009990 comp rx=0 tx=0).stop 2024-05-27T07:28:16.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.203+0000 7ffb10a00700 1 -- 172.21.15.43:0/24567034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c10af40 msgr2=0x7ffb0c110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:16.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.203+0000 7ffb10a00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c10af40 0x7ffb0c110bd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8000f80 tx=0x7ffaf8007a20 comp rx=0 tx=0).stop 2024-05-27T07:28:16.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.204+0000 7ffb10a00700 1 -- 172.21.15.43:0/24567034 shutdown_connections 2024-05-27T07:28:16.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.204+0000 7ffb10a00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ffaf00b2a70 0x7ffaf00b4f80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.204+0000 7ffb10a00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb0c10b890 0x7ffb0c111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.204+0000 7ffb10a00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb0c106610 0x7ffb0c110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.204+0000 7ffb10a00700 1 --2- 172.21.15.43:0/24567034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c10af40 0x7ffb0c110bd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:16.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.204+0000 7ffb10a00700 1 -- 172.21.15.43:0/24567034 >> 172.21.15.43:0/24567034 conn(0x7ffb0c102020 msgr2=0x7ffb0c10a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:16.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.204+0000 7ffb10a00700 1 -- 172.21.15.43:0/24567034 shutdown_connections 2024-05-27T07:28:16.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:16.205+0000 7ffb10a00700 1 -- 172.21.15.43:0/24567034 wait complete. 2024-05-27T07:28:16.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 1645 14.96 MB/sec warmup 17 sec latency 9881.904 ms 2024-05-27T07:28:17.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:17 smithi043 ceph-mon[31531]: pgmap v483: 129 pgs: 129 active+clean; 368 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 2.0 MiB/s wr, 106 op/s 2024-05-27T07:28:17.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:17 smithi179 ceph-mon[35888]: pgmap v483: 129 pgs: 129 active+clean; 368 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 2.0 MiB/s wr, 106 op/s 2024-05-27T07:28:17.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 1777 14.32 MB/sec warmup 18 sec latency 74.436 ms 2024-05-27T07:28:17.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:17 smithi132 ceph-mon[35283]: pgmap v483: 129 pgs: 129 active+clean; 368 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 2.0 MiB/s wr, 106 op/s 2024-05-27T07:28:17.969 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:28:17.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:28:18.545 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:18.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 1900 13.64 MB/sec warmup 19 sec latency 81.622 ms 2024-05-27T07:28:19.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:19 smithi043 ceph-mon[31531]: pgmap v484: 129 pgs: 129 active+clean; 368 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 MiB/s wr, 91 op/s 2024-05-27T07:28:19.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:19 smithi179 ceph-mon[35888]: pgmap v484: 129 pgs: 129 active+clean; 368 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 MiB/s wr, 91 op/s 2024-05-27T07:28:19.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 2043 13.37 MB/sec warmup 20 sec latency 83.623 ms 2024-05-27T07:28:19.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:19 smithi132 ceph-mon[35283]: pgmap v484: 129 pgs: 129 active+clean; 368 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 MiB/s wr, 91 op/s 2024-05-27T07:28:20.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 2159 13.05 MB/sec warmup 21 sec latency 85.470 ms 2024-05-27T07:28:20.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.872+0000 7f1b5fc00700 1 -- 172.21.15.43:0/670195572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b581054d0 msgr2=0x7f1b581058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.872+0000 7f1b5fc00700 1 --2- 172.21.15.43:0/670195572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b581054d0 0x7f1b581058e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009990 tx=0x7f1b4c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.873+0000 7f1b5fc00700 1 -- 172.21.15.43:0/670195572 shutdown_connections 2024-05-27T07:28:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.873+0000 7f1b5fc00700 1 --2- 172.21.15.43:0/670195572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b5810e900 0x7f1b58110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.873+0000 7f1b5fc00700 1 --2- 172.21.15.43:0/670195572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b581054d0 0x7f1b581058e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.873+0000 7f1b5fc00700 1 --2- 172.21.15.43:0/670195572 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b58105e20 0x7f1b5810e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.873+0000 7f1b5fc00700 1 -- 172.21.15.43:0/670195572 >> 172.21.15.43:0/670195572 conn(0x7f1b580fedb0 msgr2=0x7f1b581011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:20.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.874+0000 7f1b5fc00700 1 -- 172.21.15.43:0/670195572 shutdown_connections 2024-05-27T07:28:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.874+0000 7f1b5fc00700 1 -- 172.21.15.43:0/670195572 wait complete. 2024-05-27T07:28:20.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.875+0000 7f1b5fc00700 1 Processor -- start 2024-05-27T07:28:20.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.875+0000 7f1b5fc00700 1 -- start start 2024-05-27T07:28:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.876+0000 7f1b5fc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b581054d0 0x7f1b5810df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.876+0000 7f1b5fc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b58105e20 0x7f1b5810b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.876+0000 7f1b5fc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b5810e900 0x7f1b5810b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.876+0000 7f1b5fc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1b58104300 con 0x7f1b581054d0 2024-05-27T07:28:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.876+0000 7f1b5fc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1b58104180 con 0x7f1b58105e20 2024-05-27T07:28:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.876+0000 7f1b5fc00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1b58104480 con 0x7f1b5810e900 2024-05-27T07:28:20.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.877+0000 7f1b57e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b58105e20 0x7f1b5810b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.877+0000 7f1b57e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b58105e20 0x7f1b5810b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59614/0 (socket says 172.21.15.43:59614) 2024-05-27T07:28:20.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.877+0000 7f1b57e00700 1 -- 172.21.15.43:0/3872097803 learned_addr learned my addr 172.21.15.43:0/3872097803 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:20.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.877+0000 7f1b5d200700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b581054d0 0x7f1b5810df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.877+0000 7f1b5dc00700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b5810e900 0x7f1b5810b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.878+0000 7f1b5d200700 1 -- 172.21.15.43:0/3872097803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b5810e900 msgr2=0x7f1b5810b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.878+0000 7f1b5d200700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b5810e900 0x7f1b5810b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.878+0000 7f1b5d200700 1 -- 172.21.15.43:0/3872097803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b58105e20 msgr2=0x7f1b5810b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.878+0000 7f1b5d200700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b58105e20 0x7f1b5810b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.878+0000 7f1b5d200700 1 -- 172.21.15.43:0/3872097803 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5810bb90 con 0x7f1b581054d0 2024-05-27T07:28:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.878+0000 7f1b57e00700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b58105e20 0x7f1b5810b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.878+0000 7f1b5d200700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b581054d0 0x7f1b5810df90 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c000c00 tx=0x7f1b4c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.879+0000 7f1b55600700 1 -- 172.21.15.43:0/3872097803 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c026070 con 0x7f1b581054d0 2024-05-27T07:28:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.879+0000 7f1b55600700 1 -- 172.21.15.43:0/3872097803 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b4c021410 con 0x7f1b581054d0 2024-05-27T07:28:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.879+0000 7f1b5fc00700 1 -- 172.21.15.43:0/3872097803 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5810be20 con 0x7f1b581054d0 2024-05-27T07:28:20.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.879+0000 7f1b55600700 1 -- 172.21.15.43:0/3872097803 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c017650 con 0x7f1b581054d0 2024-05-27T07:28:20.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.880+0000 7f1b5fc00700 1 -- 172.21.15.43:0/3872097803 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b581adcc0 con 0x7f1b581054d0 2024-05-27T07:28:20.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.888+0000 7f1b6e759b80 1 Processor -- start 2024-05-27T07:28:20.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.889+0000 7f1b6e759b80 1 -- start start 2024-05-27T07:28:20.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.890+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485c03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.890+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe48441ec0 0x55fe485c08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.890+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe485c0ef0 0x55fe485cd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.890+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fe484eeec0 con 0x55fe4852da70 2024-05-27T07:28:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.890+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fe484dea60 con 0x55fe48441ec0 2024-05-27T07:28:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.890+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55fe484e2aa0 con 0x55fe485c0ef0 2024-05-27T07:28:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.890+0000 7f1b5dc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485c03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.890+0000 7f1b5dc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485c03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37412/0 (socket says 172.21.15.43:37412) 2024-05-27T07:28:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.890+0000 7f1b5dc00700 1 -- 172.21.15.43:0/1503689738 learned_addr learned my addr 172.21.15.43:0/1503689738 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.891+0000 7f1b57e00700 1 --2- 172.21.15.43:0/1503689738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe485c0ef0 0x55fe485cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.892+0000 7f1b57e00700 1 -- 172.21.15.43:0/1503689738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe48441ec0 msgr2=0x55fe485c08f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:28:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.892+0000 7f1b57e00700 1 --2- 172.21.15.43:0/1503689738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe48441ec0 0x55fe485c08f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.892+0000 7f1b57e00700 1 -- 172.21.15.43:0/1503689738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 msgr2=0x55fe485c03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:20.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.892+0000 7f1b57e00700 1 --2- 172.21.15.43:0/1503689738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485c03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.892+0000 7f1b57e00700 1 -- 172.21.15.43:0/1503689738 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe484a74a0 con 0x55fe485c0ef0 2024-05-27T07:28:20.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.892+0000 7f1b57e00700 1 --2- 172.21.15.43:0/1503689738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe485c0ef0 0x55fe485cd190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1b44005950 tx=0x7f1b4400f640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57262b8a3951d587 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:20.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.892+0000 7f1b55600700 1 -- 172.21.15.43:0/3872097803 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1b4c009d10 con 0x7f1b581054d0 2024-05-27T07:28:20.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.893+0000 7f1b4a000700 1 -- 172.21.15.43:0/1503689738 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44015070 con 0x55fe485c0ef0 2024-05-27T07:28:20.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.893+0000 7f1b4a000700 1 -- 172.21.15.43:0/1503689738 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b4400fda0 con 0x55fe485c0ef0 2024-05-27T07:28:20.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.893+0000 7f1b4a000700 1 -- 172.21.15.43:0/1503689738 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44010630 con 0x55fe485c0ef0 2024-05-27T07:28:20.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.895+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1503689738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe485c0ef0 msgr2=0x55fe485cd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.895+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/1503689738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe485c0ef0 0x55fe485cd190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1b44005950 tx=0x7f1b4400f640 comp rx=0 tx=0).stop 2024-05-27T07:28:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.895+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1503689738 shutdown_connections 2024-05-27T07:28:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.895+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/1503689738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe485c0ef0 0x55fe485cd190 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.895+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/1503689738 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485c03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.895+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/1503689738 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe48441ec0 0x55fe485c08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.895+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1503689738 >> 172.21.15.43:0/1503689738 conn(0x55fe484db410 msgr2=0x55fe485bc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:20.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.895+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1503689738 shutdown_connections 2024-05-27T07:28:20.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.895+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1503689738 wait complete. 2024-05-27T07:28:20.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.896+0000 7f1b55600700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1b400b2990 0x7f1b400b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.896+0000 7f1b6e759b80 1 -- start start 2024-05-27T07:28:20.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.896+0000 7f1b55600700 1 -- 172.21.15.43:0/3872097803 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f1b4c0e32f0 con 0x7f1b581054d0 2024-05-27T07:28:20.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.897+0000 7f1b6e759b80 1 Processor -- start 2024-05-27T07:28:20.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.897+0000 7f1b57e00700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1b400b2990 0x7f1b400b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.897+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.897+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485cac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.897+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 0x55fe485cb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.897+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fe483e5020 con 0x55fe4852da70 2024-05-27T07:28:20.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.897+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fe4847f7e0 con 0x55fe485c0ef0 2024-05-27T07:28:20.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.897+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55fe485cdc00 con 0x55fe48441ec0 2024-05-27T07:28:20.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.898+0000 7f1b6e759b80 10 client.0 _ll_register_callbacks cb 0x55fe48488cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:20.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.898+0000 7f1b5dc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 0x55fe485cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.898+0000 7f1b5dc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 0x55fe485cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59626/0 (socket says 172.21.15.43:59626) 2024-05-27T07:28:20.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.898+0000 7f1b5dc00700 1 -- 172.21.15.43:0/300617459 learned_addr learned my addr 172.21.15.43:0/300617459 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:20.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.899+0000 7f1b5dc00700 1 -- 172.21.15.43:0/300617459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 msgr2=0x55fe485ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:20.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.899+0000 7f1b5dc00700 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.899+0000 7f1b5dc00700 1 -- 172.21.15.43:0/300617459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 msgr2=0x55fe485cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:20.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.899+0000 7f1b5dc00700 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.899+0000 7f1b5dc00700 1 -- 172.21.15.43:0/300617459 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe4842e950 con 0x55fe485c0ef0 2024-05-27T07:28:20.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.899+0000 7f1b5dc00700 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 0x55fe485cb180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1b50011f30 tx=0x7f1b50014600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76431b3a86568ff6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:20.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.899+0000 7f1b3ea00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:28:20.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.899+0000 7f1b3ea00700 1 -- 172.21.15.43:0/300617459 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b500092d0 con 0x55fe485c0ef0 2024-05-27T07:28:20.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.899+0000 7f1b3ea00700 1 -- 172.21.15.43:0/300617459 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f1b50003a00 con 0x55fe485c0ef0 2024-05-27T07:28:20.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.900+0000 7f1b6e759b80 1 -- 172.21.15.43:0/300617459 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b4400fda0 con 0x55fe485c0ef0 2024-05-27T07:28:20.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.900+0000 7f1b57e00700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1b400b2990 0x7f1b400b4ea0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1b44013000 tx=0x7f1b44012e50 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:20.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.900+0000 7f1b3ea00700 1 -- 172.21.15.43:0/300617459 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f1b50003c00 con 0x55fe485c0ef0 2024-05-27T07:28:20.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.900+0000 7f1b6e759b80 10 client.36947 fetch_fsmap learned FSMap version 115 2024-05-27T07:28:20.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.900+0000 7f1b6e759b80 1 -- 172.21.15.43:0/300617459 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=4+}) -- 0x55fe484af0e0 con 0x55fe485c0ef0 2024-05-27T07:28:20.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.901+0000 7f1b3ea00700 1 -- 172.21.15.43:0/300617459 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b4400fda0 con 0x55fe485c0ef0 2024-05-27T07:28:20.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.901+0000 7f1b3ea00700 1 -- 172.21.15.43:0/300617459 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 115) ==== 4820+0+0 (secure 0 0 0) 0x7f1b50007500 con 0x55fe485c0ef0 2024-05-27T07:28:20.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.902+0000 7f1b6e759b80 10 client.36947 fetch_fsmap finished waiting for FSMap version 115 2024-05-27T07:28:20.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.902+0000 7f1b6e759b80 10 client.36947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:20.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.902+0000 7f1b6e759b80 20 client.36947 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:20.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.902+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fe485db020 0x55fe485dd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.902+0000 7f1b6e759b80 4 client.36947 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:20.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.902+0000 7f1b6e759b80 1 -- 172.21.15.43:0/300617459 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe484f0e10 con 0x55fe485db020 2024-05-27T07:28:20.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.902+0000 7f1b5d200700 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fe485db020 0x55fe485dd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.903+0000 7f1b5d200700 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fe485db020 0x55fe485dd4d0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=772de86ef137bf94 server_cookie=c0ec322ea1f7a352 in_seq=0 out_seq=0 2024-05-27T07:28:20.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.903+0000 7f1b3ea00700 10 client.36947 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:20.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.904+0000 7f1b3ea00700 1 -- 172.21.15.43:0/300617459 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55fe484f0e10 con 0x55fe485db020 2024-05-27T07:28:20.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.904+0000 7f1b3ea00700 10 client.36947 handle_command_reply: tid=0 2024-05-27T07:28:20.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.904+0000 7f1b6e759b80 1 client.36947 shutdown 2024-05-27T07:28:20.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.904+0000 7f1b6e759b80 1 -- 172.21.15.43:0/300617459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 msgr2=0x55fe485cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.904+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 0x55fe485cb180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1b50011f30 tx=0x7f1b50014600 comp rx=0 tx=0).stop 2024-05-27T07:28:20.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.905+0000 7f1b6e759b80 1 -- 172.21.15.43:0/300617459 shutdown_connections 2024-05-27T07:28:20.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.905+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 0x55fe485cb180 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.905+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.905+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fe485db020 0x55fe485dd4d0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.905+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/300617459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.905+0000 7f1b6e759b80 1 -- 172.21.15.43:0/300617459 >> 172.21.15.43:0/300617459 conn(0x55fe484db410 msgr2=0x55fe485c03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:20.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.905+0000 7f1b3ea00700 0 client.36947 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:20.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.905+0000 7f1b6e759b80 1 -- 172.21.15.43:0/300617459 shutdown_connections 2024-05-27T07:28:20.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.905+0000 7f1b6e759b80 1 -- 172.21.15.43:0/300617459 wait complete. 2024-05-27T07:28:20.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.905+0000 7f1b6e759b80 20 client.36947 trim_cache size 0 max 16384 2024-05-27T07:28:20.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.949+0000 7f1b6e759b80 1 Processor -- start 2024-05-27T07:28:20.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.949+0000 7f1b6e759b80 1 -- start start 2024-05-27T07:28:20.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.949+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.949+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe4852da70 0x55fe485cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.949+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe485c0ef0 0x55fe485cb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.949+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1b581b7940 con 0x55fe485c0ef0 2024-05-27T07:28:20.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.949+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1b581b5550 con 0x55fe4852da70 2024-05-27T07:28:20.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.950+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1b581b7270 con 0x55fe48441ec0 2024-05-27T07:28:20.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.950+0000 7f1b5d200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.950+0000 7f1b5d200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:34444/0 (socket says 172.21.15.43:34444) 2024-05-27T07:28:20.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.950+0000 7f1b5d200700 1 -- 172.21.15.43:0/3557772038 learned_addr learned my addr 172.21.15.43:0/3557772038 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:20.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.950+0000 7f1b57e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe4852da70 0x55fe485cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.951+0000 7f1b5dc00700 1 --2- 172.21.15.43:0/3557772038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe485c0ef0 0x55fe485cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.952+0000 7f1b5d200700 1 -- 172.21.15.43:0/3557772038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe4852da70 msgr2=0x55fe485cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:20.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.952+0000 7f1b5d200700 1 --2- 172.21.15.43:0/3557772038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe4852da70 0x55fe485cabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.952+0000 7f1b5d200700 1 -- 172.21.15.43:0/3557772038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe485c0ef0 msgr2=0x55fe485cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:20.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.952+0000 7f1b5d200700 1 --2- 172.21.15.43:0/3557772038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe485c0ef0 0x55fe485cb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.952+0000 7f1b5d200700 1 -- 172.21.15.43:0/3557772038 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe485b82c0 con 0x55fe48441ec0 2024-05-27T07:28:20.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.952+0000 7f1b5d200700 1 --2- 172.21.15.43:0/3557772038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d6ca0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c005b20 tx=0x7f1b4c00f880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac9a202b43060baa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:20.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.953+0000 7f1b49600700 1 -- 172.21.15.43:0/3557772038 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c0c4070 con 0x55fe48441ec0 2024-05-27T07:28:20.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.953+0000 7f1b49600700 1 -- 172.21.15.43:0/3557772038 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b4c0aa290 con 0x55fe48441ec0 2024-05-27T07:28:20.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.953+0000 7f1b49600700 1 -- 172.21.15.43:0/3557772038 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c021900 con 0x55fe48441ec0 2024-05-27T07:28:20.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.955+0000 7f1b6e759b80 1 -- 172.21.15.43:0/3557772038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 msgr2=0x55fe485d6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.955+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/3557772038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d6ca0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c005b20 tx=0x7f1b4c00f880 comp rx=0 tx=0).stop 2024-05-27T07:28:20.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.955+0000 7f1b6e759b80 1 -- 172.21.15.43:0/3557772038 shutdown_connections 2024-05-27T07:28:20.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.955+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/3557772038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe485c0ef0 0x55fe485cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.955+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/3557772038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d6ca0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.955+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/3557772038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe4852da70 0x55fe485cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.955+0000 7f1b6e759b80 1 -- 172.21.15.43:0/3557772038 >> 172.21.15.43:0/3557772038 conn(0x55fe484db410 msgr2=0x55fe485ca4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:20.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.955+0000 7f1b6e759b80 1 -- 172.21.15.43:0/3557772038 shutdown_connections 2024-05-27T07:28:20.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.955+0000 7f1b6e759b80 1 -- 172.21.15.43:0/3557772038 wait complete. 2024-05-27T07:28:20.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.956+0000 7f1b6e759b80 1 -- start start 2024-05-27T07:28:20.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.956+0000 7f1b6e759b80 1 Processor -- start 2024-05-27T07:28:20.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.957+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.957+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.957+0000 7f1b6e759b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 0x55fe4863ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.957+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1b581b63b0 con 0x55fe4852da70 2024-05-27T07:28:20.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.957+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1b581b5c80 con 0x55fe485c0ef0 2024-05-27T07:28:20.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.957+0000 7f1b6e759b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1b581b6ae0 con 0x55fe48441ec0 2024-05-27T07:28:20.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.957+0000 7f1b6e759b80 10 client.0 _ll_register_callbacks cb 0x55fe48488cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:20.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.958+0000 7f1b57e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.958+0000 7f1b5d200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.958+0000 7f1b5d200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:34460/0 (socket says 172.21.15.43:34460) 2024-05-27T07:28:20.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.958+0000 7f1b5d200700 1 -- 172.21.15.43:0/1675537061 learned_addr learned my addr 172.21.15.43:0/1675537061 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:20.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.958+0000 7f1b5dc00700 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 0x55fe4863ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.959+0000 7f1b5d200700 1 -- 172.21.15.43:0/1675537061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 msgr2=0x55fe4863ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:20.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.959+0000 7f1b5d200700 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 0x55fe4863ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.959+0000 7f1b5d200700 1 -- 172.21.15.43:0/1675537061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 msgr2=0x55fe485d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:20.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.959+0000 7f1b5d200700 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.959+0000 7f1b5d200700 1 -- 172.21.15.43:0/1675537061 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe4863f3a0 con 0x55fe48441ec0 2024-05-27T07:28:20.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.959+0000 7f1b57e00700 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:20.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.959+0000 7f1b5d200700 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d2e90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0e0bf0 tx=0x7f1b4c00fd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62b61ffd3b0563a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:20.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.959+0000 7f1b3ea00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:28:20.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.960+0000 7f1b3ea00700 1 -- 172.21.15.43:0/1675537061 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c0c4070 con 0x55fe48441ec0 2024-05-27T07:28:20.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.960+0000 7f1b3ea00700 1 -- 172.21.15.43:0/1675537061 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f1b4c021900 con 0x55fe48441ec0 2024-05-27T07:28:20.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.960+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1675537061 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b4c0aa290 con 0x55fe48441ec0 2024-05-27T07:28:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.960+0000 7f1b3ea00700 1 -- 172.21.15.43:0/1675537061 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f1b4c0ae640 con 0x55fe48441ec0 2024-05-27T07:28:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.960+0000 7f1b6e759b80 10 client.27070 fetch_fsmap learned FSMap version 115 2024-05-27T07:28:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.960+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1675537061 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=4+}) -- 0x55fe48657370 con 0x55fe48441ec0 2024-05-27T07:28:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.961+0000 7f1b3ea00700 1 -- 172.21.15.43:0/1675537061 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1b4c0aa290 con 0x55fe48441ec0 2024-05-27T07:28:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.961+0000 7f1b3ea00700 1 -- 172.21.15.43:0/1675537061 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 115) ==== 4820+0+0 (secure 0 0 0) 0x7f1b4c029030 con 0x55fe48441ec0 2024-05-27T07:28:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.961+0000 7f1b6e759b80 10 client.27070 fetch_fsmap finished waiting for FSMap version 115 2024-05-27T07:28:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.961+0000 7f1b6e759b80 10 client.27070 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:20.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.961+0000 7f1b6e759b80 20 client.27070 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.961+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fe48657a80 0x55fe48659ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.961+0000 7f1b6e759b80 4 client.27070 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.961+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1675537061 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fe484ea5f0 con 0x55fe48657a80 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.962+0000 7f1b57e00700 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fe48657a80 0x55fe48659ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.963+0000 7f1b57e00700 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fe48657a80 0x55fe48659ea0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d47af05854ac48d server_cookie=fe87a37bfbe18d10 in_seq=0 out_seq=0 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.963+0000 7f1b3ea00700 10 client.27070 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b3ea00700 1 -- 172.21.15.43:0/1675537061 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fe484ea5f0 con 0x55fe48657a80 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b3ea00700 10 client.27070 handle_command_reply: tid=0 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b6e759b80 1 client.27070 shutdown 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1675537061 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 msgr2=0x55fe485d2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d2e90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0e0bf0 tx=0x7f1b4c00fd40 comp rx=0 tx=0).stop 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1675537061 shutdown_connections 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe485c0ef0 0x55fe4863ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fe48441ec0 0x55fe485d2e90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55fe48657a80 0x55fe48659ea0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b6e759b80 1 --2- 172.21.15.43:0/1675537061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fe4852da70 0x55fe485d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1675537061 >> 172.21.15.43:0/1675537061 conn(0x55fe484db410 msgr2=0x55fe485cbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:20.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.965+0000 7f1b3ea00700 0 client.27070 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:20.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.966+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1675537061 shutdown_connections 2024-05-27T07:28:20.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.966+0000 7f1b6e759b80 1 -- 172.21.15.43:0/1675537061 wait complete. 2024-05-27T07:28:20.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.966+0000 7f1b6e759b80 20 client.27070 trim_cache size 0 max 16384 2024-05-27T07:28:20.958 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:28:20.958 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:28:20.958 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "d3c676aa-3b06-424f-8141-ceae7f3c8a24", 2024-05-27T07:28:20.958 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:28:20.958 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:28:20.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.967+0000 7f1b5fc00700 1 -- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1b400b2990 msgr2=0x7f1b400b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.967+0000 7f1b5fc00700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1b400b2990 0x7f1b400b4ea0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1b44013000 tx=0x7f1b44012e50 comp rx=0 tx=0).stop 2024-05-27T07:28:20.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.968+0000 7f1b5fc00700 1 -- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b581054d0 msgr2=0x7f1b5810df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:20.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.968+0000 7f1b5fc00700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b581054d0 0x7f1b5810df90 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c000c00 tx=0x7f1b4c0057c0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.968+0000 7f1b5fc00700 1 -- 172.21.15.43:0/3872097803 shutdown_connections 2024-05-27T07:28:20.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.968+0000 7f1b5fc00700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1b400b2990 0x7f1b400b4ea0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.968+0000 7f1b5fc00700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b5810e900 0x7f1b5810b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.968+0000 7f1b5fc00700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b581054d0 0x7f1b5810df90 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.968+0000 7f1b5fc00700 1 --2- 172.21.15.43:0/3872097803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b58105e20 0x7f1b5810b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:20.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.968+0000 7f1b5fc00700 1 -- 172.21.15.43:0/3872097803 >> 172.21.15.43:0/3872097803 conn(0x7f1b580fedb0 msgr2=0x7f1b58101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:20.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.969+0000 7f1b5fc00700 1 -- 172.21.15.43:0/3872097803 shutdown_connections 2024-05-27T07:28:20.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:20.969+0000 7f1b5fc00700 1 -- 172.21.15.43:0/3872097803 wait complete. 2024-05-27T07:28:21.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:21 smithi043 ceph-mon[31531]: pgmap v485: 129 pgs: 129 active+clean; 368 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-05-27T07:28:21.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:21 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:28:21.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:21 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:28:21.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:21 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:21.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:21 smithi179 ceph-mon[35888]: pgmap v485: 129 pgs: 129 active+clean; 368 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-05-27T07:28:21.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:21 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:28:21.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:21 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:28:21.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:21 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:21.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 2250 12.60 MB/sec warmup 22 sec latency 159.512 ms 2024-05-27T07:28:21.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:28:21.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:21 smithi132 ceph-mon[35283]: pgmap v485: 129 pgs: 129 active+clean; 368 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-05-27T07:28:21.861 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:21 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:28:21.861 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:21 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:28:21.861 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:21 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.405 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:22.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:22 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:28:22.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:22 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:28:22.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:22 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:28:22.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:22 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:28:22.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:22 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:28:22.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:22 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:28:22.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:22 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:28:22.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:22 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:28:22.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 2347 12.26 MB/sec warmup 23 sec latency 107.489 ms 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:22 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:22 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:22 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:22 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:28:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:23 smithi043 ceph-mon[31531]: pgmap v486: 129 pgs: 129 active+clean; 372 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 2.6 MiB/s wr, 153 op/s 2024-05-27T07:28:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:23 smithi179 ceph-mon[35888]: pgmap v486: 129 pgs: 129 active+clean; 372 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 2.6 MiB/s wr, 153 op/s 2024-05-27T07:28:23.740 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:23 smithi132 ceph-mon[35283]: pgmap v486: 129 pgs: 129 active+clean; 372 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 2.6 MiB/s wr, 153 op/s 2024-05-27T07:28:23.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 2506 12.23 MB/sec warmup 24 sec latency 90.825 ms 2024-05-27T07:28:24.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 2672 12.43 MB/sec warmup 25 sec latency 80.418 ms 2024-05-27T07:28:24.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.867+0000 7f46ebe00700 1 -- 172.21.15.43:0/1447459758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec1043f0 msgr2=0x7f46ec104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:24.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.867+0000 7f46ebe00700 1 --2- 172.21.15.43:0/1447459758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec1043f0 0x7f46ec104800 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0099c0 tx=0x7f46dc009cd0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.868+0000 7f46ebe00700 1 -- 172.21.15.43:0/1447459758 shutdown_connections 2024-05-27T07:28:24.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.868+0000 7f46ebe00700 1 --2- 172.21.15.43:0/1447459758 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46ec109670 0x7f46ec110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.868+0000 7f46ebe00700 1 --2- 172.21.15.43:0/1447459758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec1043f0 0x7f46ec104800 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.868+0000 7f46ebe00700 1 --2- 172.21.15.43:0/1447459758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec108d20 0x7f46ec109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.868+0000 7f46ebe00700 1 -- 172.21.15.43:0/1447459758 >> 172.21.15.43:0/1447459758 conn(0x7f46ec079d00 msgr2=0x7f46ec07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:24.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.868+0000 7f46ebe00700 1 -- 172.21.15.43:0/1447459758 shutdown_connections 2024-05-27T07:28:24.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.868+0000 7f46ebe00700 1 -- 172.21.15.43:0/1447459758 wait complete. 2024-05-27T07:28:24.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.870+0000 7f46ebe00700 1 Processor -- start 2024-05-27T07:28:24.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.870+0000 7f46ebe00700 1 -- start start 2024-05-27T07:28:24.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46ebe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec1043f0 0x7f46ec1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46ebe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46ec108d20 0x7f46ec199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46ebe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec109670 0x7f46ec1a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46ebe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec1141f0 con 0x7f46ec1043f0 2024-05-27T07:28:24.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46ebe00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec114070 con 0x7f46ec109670 2024-05-27T07:28:24.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46ebe00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec114370 con 0x7f46ec108d20 2024-05-27T07:28:24.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46eb400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec109670 0x7f46ec1a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46eb400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec109670 0x7f46ec1a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:39568/0 (socket says 172.21.15.43:39568) 2024-05-27T07:28:24.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46eb400700 1 -- 172.21.15.43:0/2211711051 learned_addr learned my addr 172.21.15.43:0/2211711051 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:24.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46ea000700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46ec108d20 0x7f46ec199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.871+0000 7f46eaa00700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec1043f0 0x7f46ec1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.872+0000 7f46eb400700 1 -- 172.21.15.43:0/2211711051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46ec108d20 msgr2=0x7f46ec199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:24.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.872+0000 7f46eb400700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46ec108d20 0x7f46ec199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.872+0000 7f46eb400700 1 -- 172.21.15.43:0/2211711051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec1043f0 msgr2=0x7f46ec1996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:24.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.872+0000 7f46eb400700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec1043f0 0x7f46ec1996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.872+0000 7f46eb400700 1 -- 172.21.15.43:0/2211711051 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ec1a0c60 con 0x7f46ec109670 2024-05-27T07:28:24.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.872+0000 7f46ea000700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46ec108d20 0x7f46ec199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:24.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.873+0000 7f46eb400700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec109670 0x7f46ec1a0720 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f46e400ef30 tx=0x7f46e400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:24.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.873+0000 7f46e3400700 1 -- 172.21.15.43:0/2211711051 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e40092e0 con 0x7f46ec109670 2024-05-27T07:28:24.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.873+0000 7f46e3400700 1 -- 172.21.15.43:0/2211711051 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46e4004bb0 con 0x7f46ec109670 2024-05-27T07:28:24.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.873+0000 7f46e3400700 1 -- 172.21.15.43:0/2211711051 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e4004020 con 0x7f46ec109670 2024-05-27T07:28:24.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.874+0000 7f46ebe00700 1 -- 172.21.15.43:0/2211711051 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46ec1a0ef0 con 0x7f46ec109670 2024-05-27T07:28:24.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.874+0000 7f46ebe00700 1 -- 172.21.15.43:0/2211711051 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46ec1a13e0 con 0x7f46ec109670 2024-05-27T07:28:24.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.880+0000 7f46e3400700 1 -- 172.21.15.43:0/2211711051 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f46e401b030 con 0x7f46ec109670 2024-05-27T07:28:24.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.881+0000 7f47003a5b80 1 Processor -- start 2024-05-27T07:28:24.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.882+0000 7f47003a5b80 1 -- start start 2024-05-27T07:28:24.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.882+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba4f29f0 0x5588ba586750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.882+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba406e40 0x5588ba586c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.882+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba5902c0 0x5588ba5926e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.882+0000 7f47003a5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5588ba4b3e40 con 0x5588ba406e40 2024-05-27T07:28:24.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.882+0000 7f47003a5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5588ba4a39e0 con 0x5588ba5902c0 2024-05-27T07:28:24.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.882+0000 7f47003a5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5588ba4a7a20 con 0x5588ba4f29f0 2024-05-27T07:28:24.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.883+0000 7f46ea000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba5902c0 0x5588ba5926e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.883+0000 7f46ea000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba5902c0 0x5588ba5926e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:39582/0 (socket says 172.21.15.43:39582) 2024-05-27T07:28:24.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.883+0000 7f46ea000700 1 -- 172.21.15.43:0/1260681536 learned_addr learned my addr 172.21.15.43:0/1260681536 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:24.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.883+0000 7f46ea000700 1 --2- 172.21.15.43:0/1260681536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba4f29f0 0x5588ba586750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.883+0000 7f46eaa00700 1 --2- 172.21.15.43:0/1260681536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba406e40 0x5588ba586c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.883+0000 7f46e3400700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46d00b2aa0 0x7f46d00b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.884+0000 7f46ea000700 1 -- 172.21.15.43:0/1260681536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba4f29f0 msgr2=0x5588ba586750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:24.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.884+0000 7f46ea000700 1 --2- 172.21.15.43:0/1260681536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba4f29f0 0x5588ba586750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.884+0000 7f46ea000700 1 -- 172.21.15.43:0/1260681536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba406e40 msgr2=0x5588ba586c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:24.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.884+0000 7f46ea000700 1 --2- 172.21.15.43:0/1260681536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba406e40 0x5588ba586c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.884+0000 7f46ea000700 1 -- 172.21.15.43:0/1260681536 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588ba46c420 con 0x5588ba5902c0 2024-05-27T07:28:24.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.884+0000 7f46e3400700 1 -- 172.21.15.43:0/2211711051 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f46e40e24a0 con 0x7f46ec109670 2024-05-27T07:28:24.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.884+0000 7f46eb400700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46d00b2aa0 0x7f46d00b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.884+0000 7f46ea000700 1 --2- 172.21.15.43:0/1260681536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba5902c0 0x5588ba5926e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x5588ba582a90 tx=0x7f46d8009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9da5ae6d5a5fc23 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:24.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.885+0000 7f46d7e00700 1 -- 172.21.15.43:0/1260681536 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8012e60 con 0x5588ba5902c0 2024-05-27T07:28:24.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.885+0000 7f46d7e00700 1 -- 172.21.15.43:0/1260681536 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46d800f8e0 con 0x5588ba5902c0 2024-05-27T07:28:24.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.885+0000 7f46d7e00700 1 -- 172.21.15.43:0/1260681536 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d801e7d0 con 0x5588ba5902c0 2024-05-27T07:28:24.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.886+0000 7f47003a5b80 1 -- 172.21.15.43:0/1260681536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba5902c0 msgr2=0x5588ba5926e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:24.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.886+0000 7f47003a5b80 1 --2- 172.21.15.43:0/1260681536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba5902c0 0x5588ba5926e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x5588ba582a90 tx=0x7f46d8009790 comp rx=0 tx=0).stop 2024-05-27T07:28:24.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.887+0000 7f46eb400700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46d00b2aa0 0x7f46d00b4fb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f46e40a54c0 tx=0x7f46e4009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:24.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.887+0000 7f47003a5b80 1 -- 172.21.15.43:0/1260681536 shutdown_connections 2024-05-27T07:28:24.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.887+0000 7f47003a5b80 1 --2- 172.21.15.43:0/1260681536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba5902c0 0x5588ba5926e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.887+0000 7f47003a5b80 1 --2- 172.21.15.43:0/1260681536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba4f29f0 0x5588ba586750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.887+0000 7f47003a5b80 1 --2- 172.21.15.43:0/1260681536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba406e40 0x5588ba586c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.887+0000 7f47003a5b80 1 -- 172.21.15.43:0/1260681536 >> 172.21.15.43:0/1260681536 conn(0x5588ba4a0390 msgr2=0x5588ba581710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:24.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.887+0000 7f47003a5b80 1 -- 172.21.15.43:0/1260681536 shutdown_connections 2024-05-27T07:28:24.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.887+0000 7f47003a5b80 1 -- 172.21.15.43:0/1260681536 wait complete. 2024-05-27T07:28:24.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.888+0000 7f47003a5b80 1 -- start start 2024-05-27T07:28:24.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.889+0000 7f47003a5b80 1 Processor -- start 2024-05-27T07:28:24.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 0x5588ba58d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 0x5588ba58d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba5902c0 0x5588ba58de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f47003a5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5588ba3a9fa0 con 0x5588ba4f29f0 2024-05-27T07:28:24.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f47003a5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5588ba444760 con 0x5588ba406e40 2024-05-27T07:28:24.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f47003a5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5588ba592e00 con 0x5588ba5902c0 2024-05-27T07:28:24.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f47003a5b80 10 client.0 _ll_register_callbacks cb 0x5588ba44dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:24.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f46ea000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 0x5588ba58d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f46ea000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 0x5588ba58d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:39584/0 (socket says 172.21.15.43:39584) 2024-05-27T07:28:24.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f46ea000700 1 -- 172.21.15.43:0/269972380 learned_addr learned my addr 172.21.15.43:0/269972380 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:24.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.890+0000 7f46ea000700 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba5902c0 0x5588ba58de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.891+0000 7f46eaa00700 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 0x5588ba58d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.891+0000 7f46ea000700 1 -- 172.21.15.43:0/269972380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba5902c0 msgr2=0x5588ba58de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.891+0000 7f46ea000700 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba5902c0 0x5588ba58de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.891+0000 7f46ea000700 1 -- 172.21.15.43:0/269972380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 msgr2=0x5588ba58d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.891+0000 7f46ea000700 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 0x5588ba58d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.891+0000 7f46ea000700 1 -- 172.21.15.43:0/269972380 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588ba3f38d0 con 0x5588ba406e40 2024-05-27T07:28:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.892+0000 7f46ea000700 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 0x5588ba58d3b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f46d801e470 tx=0x7f46d8009c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5598fb3d06ca3508 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.892+0000 7f46d4c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:28:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.892+0000 7f46d4c00700 1 -- 172.21.15.43:0/269972380 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8005030 con 0x5588ba406e40 2024-05-27T07:28:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.892+0000 7f47003a5b80 1 -- 172.21.15.43:0/269972380 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46d800f8e0 con 0x5588ba406e40 2024-05-27T07:28:24.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.892+0000 7f46d4c00700 1 -- 172.21.15.43:0/269972380 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f46d801b040 con 0x5588ba406e40 2024-05-27T07:28:24.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.892+0000 7f46d4c00700 1 -- 172.21.15.43:0/269972380 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f46d8033050 con 0x5588ba406e40 2024-05-27T07:28:24.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.892+0000 7f47003a5b80 10 client.36965 fetch_fsmap learned FSMap version 115 2024-05-27T07:28:24.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.893+0000 7f47003a5b80 1 -- 172.21.15.43:0/269972380 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=4+}) -- 0x5588ba474060 con 0x5588ba406e40 2024-05-27T07:28:24.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.893+0000 7f46d4c00700 1 -- 172.21.15.43:0/269972380 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46d800f8e0 con 0x5588ba406e40 2024-05-27T07:28:24.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.893+0000 7f46d4c00700 1 -- 172.21.15.43:0/269972380 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 115) ==== 4820+0+0 (secure 0 0 0) 0x7f46d802b020 con 0x5588ba406e40 2024-05-27T07:28:24.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.893+0000 7f47003a5b80 10 client.36965 fetch_fsmap finished waiting for FSMap version 115 2024-05-27T07:28:24.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.893+0000 7f47003a5b80 10 client.36965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:24.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.893+0000 7f47003a5b80 20 client.36965 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:24.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.893+0000 7f47003a5b80 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5588ba59ff70 0x5588ba5a2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.893+0000 7f47003a5b80 4 client.36965 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:24.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.893+0000 7f47003a5b80 1 -- 172.21.15.43:0/269972380 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588ba4b5d90 con 0x5588ba59ff70 2024-05-27T07:28:24.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.894+0000 7f46eaa00700 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5588ba59ff70 0x5588ba5a2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.895+0000 7f46eaa00700 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5588ba59ff70 0x5588ba5a2400 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e42a9a54c14d961a server_cookie=14ba1308b5877284 in_seq=0 out_seq=0 2024-05-27T07:28:24.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.895+0000 7f46d4c00700 10 client.36965 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:24.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.897+0000 7f46d4c00700 1 -- 172.21.15.43:0/269972380 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5588ba4b5d90 con 0x5588ba59ff70 2024-05-27T07:28:24.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.897+0000 7f46d4c00700 10 client.36965 handle_command_reply: tid=0 2024-05-27T07:28:24.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.897+0000 7f47003a5b80 1 client.36965 shutdown 2024-05-27T07:28:24.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 1 -- 172.21.15.43:0/269972380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 msgr2=0x5588ba58d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:24.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 0x5588ba58d3b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f46d801e470 tx=0x7f46d8009c70 comp rx=0 tx=0).stop 2024-05-27T07:28:24.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 1 -- 172.21.15.43:0/269972380 shutdown_connections 2024-05-27T07:28:24.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba5902c0 0x5588ba58de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 0x5588ba58d3b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5588ba59ff70 0x5588ba5a2400 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 1 --2- 172.21.15.43:0/269972380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 0x5588ba58d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 1 -- 172.21.15.43:0/269972380 >> 172.21.15.43:0/269972380 conn(0x5588ba4a0390 msgr2=0x5588ba5868c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:24.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f46d4c00700 0 client.36965 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:24.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 1 -- 172.21.15.43:0/269972380 shutdown_connections 2024-05-27T07:28:24.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 1 -- 172.21.15.43:0/269972380 wait complete. 2024-05-27T07:28:24.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.898+0000 7f47003a5b80 20 client.36965 trim_cache size 0 max 16384 2024-05-27T07:28:24.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.936+0000 7f47003a5b80 1 Processor -- start 2024-05-27T07:28:24.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.936+0000 7f47003a5b80 1 -- start start 2024-05-27T07:28:24.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.937+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba406e40 0x5588ba58c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.937+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba4f29f0 0x5588ba585b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.937+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba5902c0 0x5588ba581740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.937+0000 7f47003a5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec1c0020 con 0x5588ba5902c0 2024-05-27T07:28:24.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.937+0000 7f47003a5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec1bdc30 con 0x5588ba4f29f0 2024-05-27T07:28:24.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.937+0000 7f47003a5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec1bf950 con 0x5588ba406e40 2024-05-27T07:28:24.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.938+0000 7f46eaa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba4f29f0 0x5588ba585b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.938+0000 7f46eaa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba4f29f0 0x5588ba585b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:39598/0 (socket says 172.21.15.43:39598) 2024-05-27T07:28:24.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.938+0000 7f46eaa00700 1 -- 172.21.15.43:0/1374449689 learned_addr learned my addr 172.21.15.43:0/1374449689 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:24.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.938+0000 7f46ea000700 1 --2- 172.21.15.43:0/1374449689 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba5902c0 0x5588ba581740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.939+0000 7f46eaa00700 1 -- 172.21.15.43:0/1374449689 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba406e40 msgr2=0x5588ba58c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:24.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.939+0000 7f46eaa00700 1 --2- 172.21.15.43:0/1374449689 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba406e40 0x5588ba58c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.939+0000 7f46eaa00700 1 -- 172.21.15.43:0/1374449689 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba5902c0 msgr2=0x5588ba581740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:24.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.939+0000 7f46eaa00700 1 --2- 172.21.15.43:0/1374449689 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba5902c0 0x5588ba581740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.939+0000 7f46eaa00700 1 -- 172.21.15.43:0/1374449689 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588ba58b920 con 0x5588ba4f29f0 2024-05-27T07:28:24.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.939+0000 7f46ea000700 1 --2- 172.21.15.43:0/1374449689 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba5902c0 0x5588ba581740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:24.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.940+0000 7f46eaa00700 1 --2- 172.21.15.43:0/1374449689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba4f29f0 0x5588ba585b00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0099c0 tx=0x7f46dc005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b1935071b6d3ca7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:24.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.941+0000 7f46d7400700 1 -- 172.21.15.43:0/1374449689 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc0053c0 con 0x5588ba4f29f0 2024-05-27T07:28:24.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.941+0000 7f46d7400700 1 -- 172.21.15.43:0/1374449689 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46dc005560 con 0x5588ba4f29f0 2024-05-27T07:28:24.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.941+0000 7f46d7400700 1 -- 172.21.15.43:0/1374449689 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc00baa0 con 0x5588ba4f29f0 2024-05-27T07:28:24.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.942+0000 7f47003a5b80 1 -- 172.21.15.43:0/1374449689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba4f29f0 msgr2=0x5588ba585b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:24.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.942+0000 7f47003a5b80 1 --2- 172.21.15.43:0/1374449689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba4f29f0 0x5588ba585b00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0099c0 tx=0x7f46dc005040 comp rx=0 tx=0).stop 2024-05-27T07:28:24.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.943+0000 7f47003a5b80 1 -- 172.21.15.43:0/1374449689 shutdown_connections 2024-05-27T07:28:24.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.943+0000 7f47003a5b80 1 --2- 172.21.15.43:0/1374449689 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba5902c0 0x5588ba581740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.943+0000 7f47003a5b80 1 --2- 172.21.15.43:0/1374449689 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba406e40 0x5588ba58c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.943+0000 7f47003a5b80 1 --2- 172.21.15.43:0/1374449689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba4f29f0 0x5588ba585b00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.943+0000 7f47003a5b80 1 -- 172.21.15.43:0/1374449689 >> 172.21.15.43:0/1374449689 conn(0x5588ba4a0390 msgr2=0x5588ba5881d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:24.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.943+0000 7f47003a5b80 1 -- 172.21.15.43:0/1374449689 shutdown_connections 2024-05-27T07:28:24.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.944+0000 7f47003a5b80 1 -- 172.21.15.43:0/1374449689 wait complete. 2024-05-27T07:28:24.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.944+0000 7f47003a5b80 1 -- start start 2024-05-27T07:28:24.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.944+0000 7f47003a5b80 1 Processor -- start 2024-05-27T07:28:24.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.946+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 0x5588ba602d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.946+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 0x5588ba6032e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.946+0000 7f47003a5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba5902c0 0x5588ba593630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.946+0000 7f46ea000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 0x5588ba602d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.946+0000 7f47003a5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec1bea90 con 0x5588ba4f29f0 2024-05-27T07:28:24.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.947+0000 7f47003a5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec1be360 con 0x5588ba406e40 2024-05-27T07:28:24.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.947+0000 7f46eaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 0x5588ba6032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.947+0000 7f46eaa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 0x5588ba6032e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53750/0 (socket says 172.21.15.43:53750) 2024-05-27T07:28:24.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.947+0000 7f46eaa00700 1 -- 172.21.15.43:0/510624184 learned_addr learned my addr 172.21.15.43:0/510624184 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:24.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.947+0000 7f47003a5b80 1 -- 172.21.15.43:0/510624184 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec1bf1c0 con 0x5588ba5902c0 2024-05-27T07:28:24.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.947+0000 7f47003a5b80 10 client.0 _ll_register_callbacks cb 0x5588ba44dc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:24.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.948+0000 7f46ea000700 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba5902c0 0x5588ba593630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.949+0000 7f46eaa00700 1 -- 172.21.15.43:0/510624184 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba5902c0 msgr2=0x5588ba593630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:24.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.949+0000 7f46eaa00700 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba5902c0 0x5588ba593630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.949+0000 7f46eaa00700 1 -- 172.21.15.43:0/510624184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 msgr2=0x5588ba602d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:24.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.949+0000 7f46eaa00700 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 0x5588ba602d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.949+0000 7f46eaa00700 1 -- 172.21.15.43:0/510624184 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588ba593cd0 con 0x5588ba4f29f0 2024-05-27T07:28:24.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.952+0000 7f46eaa00700 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 0x5588ba6032e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f46dc005130 tx=0x7f46dc005160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad82a22780bfd516 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:24.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.952+0000 7f46d4c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:28:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.952+0000 7f46d4c00700 1 -- 172.21.15.43:0/510624184 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc039060 con 0x5588ba4f29f0 2024-05-27T07:28:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.952+0000 7f47003a5b80 1 -- 172.21.15.43:0/510624184 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46dc005560 con 0x5588ba4f29f0 2024-05-27T07:28:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.952+0000 7f46d4c00700 1 -- 172.21.15.43:0/510624184 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f46dc0039c0 con 0x5588ba4f29f0 2024-05-27T07:28:24.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.953+0000 7f46d4c00700 1 -- 172.21.15.43:0/510624184 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f46dc040050 con 0x5588ba4f29f0 2024-05-27T07:28:24.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.953+0000 7f47003a5b80 10 client.17562 fetch_fsmap learned FSMap version 115 2024-05-27T07:28:24.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.953+0000 7f47003a5b80 1 -- 172.21.15.43:0/510624184 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=4+}) -- 0x5588ba60a460 con 0x5588ba4f29f0 2024-05-27T07:28:24.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.954+0000 7f46d4c00700 1 -- 172.21.15.43:0/510624184 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46dc005560 con 0x5588ba4f29f0 2024-05-27T07:28:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.954+0000 7f46d4c00700 1 -- 172.21.15.43:0/510624184 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 115) ==== 4820+0+0 (secure 0 0 0) 0x7f46dc00f660 con 0x5588ba4f29f0 2024-05-27T07:28:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.954+0000 7f47003a5b80 10 client.17562 fetch_fsmap finished waiting for FSMap version 115 2024-05-27T07:28:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.954+0000 7f47003a5b80 10 client.17562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.954+0000 7f47003a5b80 20 client.17562 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.954+0000 7f47003a5b80 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5588ba60ab40 0x5588ba60cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.954+0000 7f47003a5b80 4 client.17562 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:28:24.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.954+0000 7f47003a5b80 1 -- 172.21.15.43:0/510624184 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5588ba4af570 con 0x5588ba60ab40 2024-05-27T07:28:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.955+0000 7f46ea000700 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5588ba60ab40 0x5588ba60cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.956+0000 7f46ea000700 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5588ba60ab40 0x5588ba60cf60 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b457c20d09ad84ef server_cookie=d1f8515243682137 in_seq=0 out_seq=0 2024-05-27T07:28:24.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.956+0000 7f46d4c00700 10 client.17562 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:24.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.956+0000 7f46d4c00700 1 -- 172.21.15.43:0/510624184 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5588ba4af570 con 0x5588ba60ab40 2024-05-27T07:28:24.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.956+0000 7f46d4c00700 10 client.17562 handle_command_reply: tid=0 2024-05-27T07:28:24.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.956+0000 7f47003a5b80 1 client.17562 shutdown 2024-05-27T07:28:24.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 1 -- 172.21.15.43:0/510624184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 msgr2=0x5588ba6032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 0x5588ba6032e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f46dc005130 tx=0x7f46dc005160 comp rx=0 tx=0).stop 2024-05-27T07:28:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 1 -- 172.21.15.43:0/510624184 shutdown_connections 2024-05-27T07:28:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5588ba5902c0 0x5588ba593630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5588ba406e40 0x5588ba602d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5588ba60ab40 0x5588ba60cf60 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 1 --2- 172.21.15.43:0/510624184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5588ba4f29f0 0x5588ba6032e0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 1 -- 172.21.15.43:0/510624184 >> 172.21.15.43:0/510624184 conn(0x5588ba4a0390 msgr2=0x5588ba58c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f46d4c00700 0 client.17562 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 1 -- 172.21.15.43:0/510624184 shutdown_connections 2024-05-27T07:28:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 1 -- 172.21.15.43:0/510624184 wait complete. 2024-05-27T07:28:24.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.957+0000 7f47003a5b80 20 client.17562 trim_cache size 0 max 16384 2024-05-27T07:28:24.948 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:28:24.948 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:28:24.948 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:28:24.948 INFO:teuthology.orchestra.run.smithi043.stdout: "d3c676aa-3b06-424f-8141-ceae7f3c8a24": { 2024-05-27T07:28:24.948 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:28:24.948 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "d3c676aa-3b06-424f-8141-ceae7f3c8a24", 2024-05-27T07:28:24.948 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:28:24.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:28:24.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:28:24.949 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:28:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 -- 172.21.15.43:0/2211711051 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46d00b2aa0 msgr2=0x7f46d00b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:24.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46d00b2aa0 0x7f46d00b4fb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f46e40a54c0 tx=0x7f46e4009040 comp rx=0 tx=0).stop 2024-05-27T07:28:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 -- 172.21.15.43:0/2211711051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec109670 msgr2=0x7f46ec1a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec109670 0x7f46ec1a0720 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f46e400ef30 tx=0x7f46e400c3a0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 -- 172.21.15.43:0/2211711051 shutdown_connections 2024-05-27T07:28:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f46d00b2aa0 0x7f46d00b4fb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46ec109670 0x7f46ec1a0720 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46ec1043f0 0x7f46ec1996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 --2- 172.21.15.43:0/2211711051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46ec108d20 0x7f46ec199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 -- 172.21.15.43:0/2211711051 >> 172.21.15.43:0/2211711051 conn(0x7f46ec079d00 msgr2=0x7f46ec102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.959+0000 7f46ebe00700 1 -- 172.21.15.43:0/2211711051 shutdown_connections 2024-05-27T07:28:24.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:24.960+0000 7f46ebe00700 1 -- 172.21.15.43:0/2211711051 wait complete. 2024-05-27T07:28:25.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:25 smithi043 ceph-mon[31531]: pgmap v487: 129 pgs: 129 active+clean; 372 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-05-27T07:28:25.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:25 smithi179 ceph-mon[35888]: pgmap v487: 129 pgs: 129 active+clean; 372 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-05-27T07:28:25.764 INFO:tasks.workunit.client.0.smithi043.stdout: 10 2884 12.55 MB/sec warmup 26 sec latency 87.941 ms 2024-05-27T07:28:25.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:25 smithi132 ceph-mon[35283]: pgmap v487: 129 pgs: 129 active+clean; 372 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-05-27T07:28:26.114 INFO:tasks.cephfs.filesystem:scrub status for tag:d3c676aa-3b06-424f-8141-ceae7f3c8a24 - {'path': '/', 'tag': 'd3c676aa-3b06-424f-8141-ceae7f3c8a24', 'options': 'recursive,force'} 2024-05-27T07:28:26.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 3046 12.32 MB/sec warmup 27 sec latency 72.598 ms 2024-05-27T07:28:27.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:27 smithi043 ceph-mon[31531]: pgmap v488: 129 pgs: 129 active+clean; 392 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 3.5 MiB/s wr, 191 op/s 2024-05-27T07:28:27.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:28:27.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:28:27.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:28:27.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:28:27.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:28:27.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:27 smithi179 ceph-mon[35888]: pgmap v488: 129 pgs: 129 active+clean; 392 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 3.5 MiB/s wr, 191 op/s 2024-05-27T07:28:27.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:28:27.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:28:27.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:28:27.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:28:27.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:28:27.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 3130 12.10 MB/sec warmup 28 sec latency 69.566 ms 2024-05-27T07:28:27.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:27 smithi132 ceph-mon[35283]: pgmap v488: 129 pgs: 129 active+clean; 392 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 3.5 MiB/s wr, 191 op/s 2024-05-27T07:28:27.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:28:27.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:28:27.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:28:27.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:28:27.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:28:28.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 3233 12.06 MB/sec warmup 29 sec latency 77.469 ms 2024-05-27T07:28:29.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:29 smithi043 ceph-mon[31531]: pgmap v489: 129 pgs: 129 active+clean; 392 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-05-27T07:28:29.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:29 smithi179 ceph-mon[35888]: pgmap v489: 129 pgs: 129 active+clean; 392 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-05-27T07:28:29.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 3354 11.98 MB/sec warmup 30 sec latency 119.028 ms 2024-05-27T07:28:29.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:29 smithi132 ceph-mon[35283]: pgmap v489: 129 pgs: 129 active+clean; 392 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-05-27T07:28:30.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 3530 12.18 MB/sec warmup 31 sec latency 129.795 ms 2024-05-27T07:28:31.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:31 smithi043 ceph-mon[31531]: pgmap v490: 129 pgs: 129 active+clean; 392 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-05-27T07:28:31.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:31 smithi179 ceph-mon[35888]: pgmap v490: 129 pgs: 129 active+clean; 392 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-05-27T07:28:31.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 3665 12.01 MB/sec warmup 32 sec latency 75.694 ms 2024-05-27T07:28:31.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:31 smithi132 ceph-mon[35283]: pgmap v490: 129 pgs: 129 active+clean; 392 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-05-27T07:28:32.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 3772 11.79 MB/sec warmup 33 sec latency 62.188 ms 2024-05-27T07:28:33.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:33 smithi043 ceph-mon[31531]: pgmap v491: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 4.9 MiB/s wr, 213 op/s 2024-05-27T07:28:33.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:33 smithi179 ceph-mon[35888]: pgmap v491: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 4.9 MiB/s wr, 213 op/s 2024-05-27T07:28:33.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 3846 11.48 MB/sec warmup 34 sec latency 77.777 ms 2024-05-27T07:28:33.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:33 smithi132 ceph-mon[35283]: pgmap v491: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 4.9 MiB/s wr, 213 op/s 2024-05-27T07:28:34.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 3947 11.27 MB/sec warmup 35 sec latency 73.754 ms 2024-05-27T07:28:35.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:35 smithi043 ceph-mon[31531]: pgmap v492: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.0 MiB/s wr, 148 op/s 2024-05-27T07:28:35.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:35 smithi179 ceph-mon[35888]: pgmap v492: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.0 MiB/s wr, 148 op/s 2024-05-27T07:28:35.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 4064 11.11 MB/sec warmup 36 sec latency 89.916 ms 2024-05-27T07:28:35.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:35 smithi132 ceph-mon[35283]: pgmap v492: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.0 MiB/s wr, 148 op/s 2024-05-27T07:28:36.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 4140 10.85 MB/sec warmup 37 sec latency 327.210 ms 2024-05-27T07:28:37.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:37 smithi043 ceph-mon[31531]: pgmap v493: 129 pgs: 129 active+clean; 411 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 5.4 MiB/s wr, 221 op/s 2024-05-27T07:28:37.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:37 smithi179 ceph-mon[35888]: pgmap v493: 129 pgs: 129 active+clean; 411 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 5.4 MiB/s wr, 221 op/s 2024-05-27T07:28:37.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 4247 10.66 MB/sec warmup 38 sec latency 80.861 ms 2024-05-27T07:28:37.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:37 smithi132 ceph-mon[35283]: pgmap v493: 129 pgs: 129 active+clean; 411 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 5.4 MiB/s wr, 221 op/s 2024-05-27T07:28:38.765 INFO:tasks.workunit.client.0.smithi043.stdout: 10 4359 10.51 MB/sec warmup 39 sec latency 100.277 ms 2024-05-27T07:28:39.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:39 smithi043 ceph-mon[31531]: pgmap v494: 129 pgs: 129 active+clean; 411 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 3.4 MiB/s wr, 142 op/s 2024-05-27T07:28:39.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:39 smithi179 ceph-mon[35888]: pgmap v494: 129 pgs: 129 active+clean; 411 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 3.4 MiB/s wr, 142 op/s 2024-05-27T07:28:39.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 4467 10.30 MB/sec warmup 40 sec latency 85.616 ms 2024-05-27T07:28:39.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:39 smithi132 ceph-mon[35283]: pgmap v494: 129 pgs: 129 active+clean; 411 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 3.4 MiB/s wr, 142 op/s 2024-05-27T07:28:40.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 4598 10.29 MB/sec warmup 41 sec latency 109.590 ms 2024-05-27T07:28:41.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:41 smithi043 ceph-mon[31531]: pgmap v495: 129 pgs: 129 active+clean; 411 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 3.4 MiB/s wr, 142 op/s 2024-05-27T07:28:41.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:41 smithi179 ceph-mon[35888]: pgmap v495: 129 pgs: 129 active+clean; 411 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 3.4 MiB/s wr, 142 op/s 2024-05-27T07:28:41.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 4698 10.19 MB/sec warmup 42 sec latency 223.305 ms 2024-05-27T07:28:41.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:41 smithi132 ceph-mon[35283]: pgmap v495: 129 pgs: 129 active+clean; 411 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 3.4 MiB/s wr, 142 op/s 2024-05-27T07:28:42.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 4825 10.05 MB/sec warmup 43 sec latency 78.005 ms 2024-05-27T07:28:43.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:43 smithi043 ceph-mon[31531]: pgmap v496: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 4.8 MiB/s wr, 243 op/s 2024-05-27T07:28:43.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:43 smithi179 ceph-mon[35888]: pgmap v496: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 4.8 MiB/s wr, 243 op/s 2024-05-27T07:28:43.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 4951 9.85 MB/sec warmup 44 sec latency 65.402 ms 2024-05-27T07:28:43.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:43 smithi132 ceph-mon[35283]: pgmap v496: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 4.8 MiB/s wr, 243 op/s 2024-05-27T07:28:44.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 5079 9.83 MB/sec warmup 45 sec latency 148.575 ms 2024-05-27T07:28:45.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:45 smithi043 ceph-mon[31531]: pgmap v497: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-05-27T07:28:45.707 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:45 smithi179 ceph-mon[35888]: pgmap v497: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-05-27T07:28:45.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 5190 9.65 MB/sec warmup 46 sec latency 245.780 ms 2024-05-27T07:28:45.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:45 smithi132 ceph-mon[35283]: pgmap v497: 129 pgs: 129 active+clean; 401 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-05-27T07:28:46.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 5276 9.46 MB/sec warmup 47 sec latency 272.044 ms 2024-05-27T07:28:46.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:46 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:28:46.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:46 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:28:46.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:46 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:28:47.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 5418 9.47 MB/sec warmup 48 sec latency 82.315 ms 2024-05-27T07:28:47.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:47 smithi132 ceph-mon[35283]: pgmap v498: 129 pgs: 129 active+clean; 397 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 3.8 MiB/s wr, 268 op/s 2024-05-27T07:28:47.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:47 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:47.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:47 smithi043 ceph-mon[31531]: pgmap v498: 129 pgs: 129 active+clean; 397 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 3.8 MiB/s wr, 268 op/s 2024-05-27T07:28:47.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:47 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:47.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:47 smithi179 ceph-mon[35888]: pgmap v498: 129 pgs: 129 active+clean; 397 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 3.8 MiB/s wr, 268 op/s 2024-05-27T07:28:47.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:47 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:28:48.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 5521 9.34 MB/sec warmup 49 sec latency 75.498 ms 2024-05-27T07:28:48.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:48 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:28:48.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:48 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:28:48.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:48 smithi043 ceph-mon[31531]: pgmap v499: 129 pgs: 129 active+clean; 397 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.3 MiB/s wr, 195 op/s 2024-05-27T07:28:48.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:48 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:28:48.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:48 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:28:48.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:48 smithi179 ceph-mon[35888]: pgmap v499: 129 pgs: 129 active+clean; 397 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.3 MiB/s wr, 195 op/s 2024-05-27T07:28:49.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:48 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:28:49.089 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:48 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:28:49.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:48 smithi132 ceph-mon[35283]: pgmap v499: 129 pgs: 129 active+clean; 397 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 2.3 MiB/s wr, 195 op/s 2024-05-27T07:28:49.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 5638 9.24 MB/sec warmup 50 sec latency 81.675 ms 2024-05-27T07:28:50.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:49 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:28:50.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:49 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:28:50.090 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:49 smithi132 ceph-mon[35283]: pgmap v500: 129 pgs: 129 active+clean; 397 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-05-27T07:28:50.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:49 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:28:50.201 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:49 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:28:50.201 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:49 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:28:50.201 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:49 smithi043 ceph-mon[31531]: pgmap v500: 129 pgs: 129 active+clean; 397 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-05-27T07:28:50.201 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:49 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:28:50.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:49 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:28:50.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:49 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:28:50.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:49 smithi179 ceph-mon[35888]: pgmap v500: 129 pgs: 129 active+clean; 397 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-05-27T07:28:50.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:49 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:28:50.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 5742 9.15 MB/sec warmup 51 sec latency 301.716 ms 2024-05-27T07:28:51.767 INFO:tasks.workunit.client.0.smithi043.stdout: 10 5912 9.25 MB/sec warmup 52 sec latency 86.294 ms 2024-05-27T07:28:52.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 6085 9.25 MB/sec warmup 53 sec latency 71.140 ms 2024-05-27T07:28:52.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:52 smithi132 ceph-mon[35283]: pgmap v501: 129 pgs: 129 active+clean; 404 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 3.6 MiB/s wr, 281 op/s 2024-05-27T07:28:52.906 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:52 smithi043 ceph-mon[31531]: pgmap v501: 129 pgs: 129 active+clean; 404 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 3.6 MiB/s wr, 281 op/s 2024-05-27T07:28:52.957 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:52 smithi179 ceph-mon[35888]: pgmap v501: 129 pgs: 129 active+clean; 404 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 3.6 MiB/s wr, 281 op/s 2024-05-27T07:28:53.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 6230 9.19 MB/sec warmup 54 sec latency 88.046 ms 2024-05-27T07:28:54.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 6329 9.17 MB/sec warmup 55 sec latency 72.325 ms 2024-05-27T07:28:54.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:54 smithi132 ceph-mon[35283]: pgmap v502: 129 pgs: 129 active+clean; 404 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-05-27T07:28:54.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:54 smithi043 ceph-mon[31531]: pgmap v502: 129 pgs: 129 active+clean; 404 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-05-27T07:28:54.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:54 smithi179 ceph-mon[35888]: pgmap v502: 129 pgs: 129 active+clean; 404 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-05-27T07:28:55.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 6434 9.17 MB/sec warmup 56 sec latency 85.948 ms 2024-05-27T07:28:56.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:28:56.698 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:28:56.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 6597 9.26 MB/sec warmup 57 sec latency 77.869 ms 2024-05-27T07:28:56.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:56 smithi132 ceph-mon[35283]: pgmap v503: 129 pgs: 129 active+clean; 404 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-05-27T07:28:56.874 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:56 smithi043 ceph-mon[31531]: pgmap v503: 129 pgs: 129 active+clean; 404 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-05-27T07:28:56.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:56 smithi179 ceph-mon[35888]: pgmap v503: 129 pgs: 129 active+clean; 404 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.1 MiB/s wr, 175 op/s 2024-05-27T07:28:57.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 6764 9.30 MB/sec warmup 58 sec latency 118.507 ms 2024-05-27T07:28:58.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 6879 9.31 MB/sec warmup 59 sec latency 274.791 ms 2024-05-27T07:28:58.825 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:28:58 smithi043 ceph-mon[31531]: pgmap v504: 129 pgs: 129 active+clean; 417 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-05-27T07:28:58.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:28:58 smithi132 ceph-mon[35283]: pgmap v504: 129 pgs: 129 active+clean; 417 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-05-27T07:28:58.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:28:58 smithi179 ceph-mon[35888]: pgmap v504: 129 pgs: 129 active+clean; 417 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-05-27T07:28:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.169+0000 7f40cd200700 1 -- 172.21.15.43:0/2552269066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40c8110c20 msgr2=0x7f40c8113040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.169+0000 7f40cd200700 1 --2- 172.21.15.43:0/2552269066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40c8110c20 0x7f40c8113040 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f40c000b0a0 tx=0x7f40c000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.169+0000 7f40cd200700 1 -- 172.21.15.43:0/2552269066 shutdown_connections 2024-05-27T07:28:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.169+0000 7f40cd200700 1 --2- 172.21.15.43:0/2552269066 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40c8110c20 0x7f40c8113040 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.169+0000 7f40cd200700 1 --2- 172.21.15.43:0/2552269066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40c8103070 0x7f40c8103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.169+0000 7f40cd200700 1 --2- 172.21.15.43:0/2552269066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40c8103a50 0x7f40c81106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.169+0000 7f40cd200700 1 -- 172.21.15.43:0/2552269066 >> 172.21.15.43:0/2552269066 conn(0x7f40c80fedb0 msgr2=0x7f40c81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.170+0000 7f40cd200700 1 -- 172.21.15.43:0/2552269066 shutdown_connections 2024-05-27T07:28:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.170+0000 7f40cd200700 1 -- 172.21.15.43:0/2552269066 wait complete. 2024-05-27T07:28:59.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.171+0000 7f40cd200700 1 Processor -- start 2024-05-27T07:28:59.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.172+0000 7f40cd200700 1 -- start start 2024-05-27T07:28:59.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.172+0000 7f40cd200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40c8103070 0x7f40c810a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.172+0000 7f40cd200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40c8103a50 0x7f40c810a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.172+0000 7f40cd200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40c8110c20 0x7f40c810fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.172+0000 7f40cd200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f40c8116580 con 0x7f40c8103a50 2024-05-27T07:28:59.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.172+0000 7f40cd200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f40c8116400 con 0x7f40c8103070 2024-05-27T07:28:59.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.172+0000 7f40cd200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f40c8116700 con 0x7f40c8110c20 2024-05-27T07:28:59.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.173+0000 7f40c5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40c8103a50 0x7f40c810a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.173+0000 7f40c5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40c8103a50 0x7f40c810a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57162/0 (socket says 172.21.15.43:57162) 2024-05-27T07:28:59.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.173+0000 7f40c5400700 1 -- 172.21.15.43:0/3075488386 learned_addr learned my addr 172.21.15.43:0/3075488386 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:59.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.173+0000 7f40c5e00700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40c8103070 0x7f40c810a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.173+0000 7f40c6800700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40c8110c20 0x7f40c810fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.173+0000 7f40c5400700 1 -- 172.21.15.43:0/3075488386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40c8110c20 msgr2=0x7f40c810fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.173+0000 7f40c5400700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40c8110c20 0x7f40c810fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.173+0000 7f40c5400700 1 -- 172.21.15.43:0/3075488386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40c8103070 msgr2=0x7f40c810a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.173+0000 7f40c5400700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40c8103070 0x7f40c810a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.174+0000 7f40c5400700 1 -- 172.21.15.43:0/3075488386 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c8110620 con 0x7f40c8103a50 2024-05-27T07:28:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.174+0000 7f40c5400700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40c8103a50 0x7f40c810a630 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f40b400e940 tx=0x7f40b400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.175+0000 7f40bea00700 1 -- 172.21.15.43:0/3075488386 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4004220 con 0x7f40c8103a50 2024-05-27T07:28:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.175+0000 7f40bea00700 1 -- 172.21.15.43:0/3075488386 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f40b40043c0 con 0x7f40c8103a50 2024-05-27T07:28:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.175+0000 7f40bea00700 1 -- 172.21.15.43:0/3075488386 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4004220 con 0x7f40c8103a50 2024-05-27T07:28:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.175+0000 7f40cd200700 1 -- 172.21.15.43:0/3075488386 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c81108b0 con 0x7f40c8103a50 2024-05-27T07:28:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.176+0000 7f40cd200700 1 -- 172.21.15.43:0/3075488386 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c8105490 con 0x7f40c8103a50 2024-05-27T07:28:59.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.183+0000 7f40bea00700 1 -- 172.21.15.43:0/3075488386 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f40b40039b0 con 0x7f40c8103a50 2024-05-27T07:28:59.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.186+0000 7f40dbde8b80 1 Processor -- start 2024-05-27T07:28:59.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.186+0000 7f40bea00700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f40ac0b2c70 0x7f40ac0b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.187+0000 7f40dbde8b80 1 -- start start 2024-05-27T07:28:59.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.187+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c3673b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.187+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c3678f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.187+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 0x560f2c385530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.187+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560f2c295e40 con 0x560f2c383110 2024-05-27T07:28:59.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.187+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560f2c2859e0 con 0x560f2c2d49f0 2024-05-27T07:28:59.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.187+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560f2c289a20 con 0x560f2c1e8e40 2024-05-27T07:28:59.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.187+0000 7f40bea00700 1 -- 172.21.15.43:0/3075488386 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f40b40e2720 con 0x7f40c8103a50 2024-05-27T07:28:59.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.188+0000 7f40c5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c3673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.188+0000 7f40c6800700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f40ac0b2c70 0x7f40ac0b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.188+0000 7f40c5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c3673b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41580/0 (socket says 172.21.15.43:41580) 2024-05-27T07:28:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.188+0000 7f40c5e00700 1 -- 172.21.15.43:0/2967684056 learned_addr learned my addr 172.21.15.43:0/2967684056 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.188+0000 7f40c5400700 1 --2- 172.21.15.43:0/2967684056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c3678f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.188+0000 7f40c6800700 1 --2- 172.21.15.43:0/2967684056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 0x560f2c385530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.189+0000 7f40c5e00700 1 -- 172.21.15.43:0/2967684056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 msgr2=0x560f2c3678f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.189+0000 7f40c5e00700 1 --2- 172.21.15.43:0/2967684056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c3678f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.189+0000 7f40c5e00700 1 -- 172.21.15.43:0/2967684056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 msgr2=0x560f2c385530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.189+0000 7f40c5e00700 1 --2- 172.21.15.43:0/2967684056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 0x560f2c385530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.189+0000 7f40c5e00700 1 -- 172.21.15.43:0/2967684056 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f2c24e420 con 0x560f2c2d49f0 2024-05-27T07:28:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.189+0000 7f40c5e00700 1 --2- 172.21.15.43:0/2967684056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c3673b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009d80 tx=0x7f40b8011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bdb2cd2154bbf1c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.190+0000 7f40b3400700 1 -- 172.21.15.43:0/2967684056 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8004230 con 0x560f2c2d49f0 2024-05-27T07:28:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.190+0000 7f40b3400700 1 -- 172.21.15.43:0/2967684056 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f40b80043d0 con 0x560f2c2d49f0 2024-05-27T07:28:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.190+0000 7f40b3400700 1 -- 172.21.15.43:0/2967684056 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8019820 con 0x560f2c2d49f0 2024-05-27T07:28:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.191+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2967684056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 msgr2=0x560f2c3673b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.191+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/2967684056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c3673b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009d80 tx=0x7f40b8011720 comp rx=0 tx=0).stop 2024-05-27T07:28:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.191+0000 7f40c6800700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f40ac0b2c70 0x7f40ac0b5180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f40c810ad00 tx=0x7f40c000bb90 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.191+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2967684056 shutdown_connections 2024-05-27T07:28:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.191+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/2967684056 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 0x560f2c385530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.191+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/2967684056 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c3673b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.191+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/2967684056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c3678f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.191+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2967684056 >> 172.21.15.43:0/2967684056 conn(0x560f2c282390 msgr2=0x560f2c3634c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.191+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2967684056 shutdown_connections 2024-05-27T07:28:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.192+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2967684056 wait complete. 2024-05-27T07:28:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.193+0000 7f40dbde8b80 1 -- start start 2024-05-27T07:28:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.193+0000 7f40dbde8b80 1 Processor -- start 2024-05-27T07:28:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.194+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c1e8e40 0x560f2c380060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.194+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c3805a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.194+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c383110 0x560f2c380ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.194+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560f2c18bfa0 con 0x560f2c2d49f0 2024-05-27T07:28:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.194+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560f2c226760 con 0x560f2c1e8e40 2024-05-27T07:28:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.194+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560f2c385b90 con 0x560f2c383110 2024-05-27T07:28:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.194+0000 7f40dbde8b80 10 client.0 _ll_register_callbacks cb 0x560f2c22fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.195+0000 7f40c5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c3805a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.195+0000 7f40c5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c3805a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57182/0 (socket says 172.21.15.43:57182) 2024-05-27T07:28:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.195+0000 7f40c5400700 1 -- 172.21.15.43:0/38022868 learned_addr learned my addr 172.21.15.43:0/38022868 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.195+0000 7f40c5e00700 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c1e8e40 0x560f2c380060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.195+0000 7f40c6800700 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c383110 0x560f2c380ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.195+0000 7f40c5e00700 1 -- 172.21.15.43:0/38022868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c383110 msgr2=0x560f2c380ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.195+0000 7f40c5e00700 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c383110 0x560f2c380ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.195+0000 7f40c5e00700 1 -- 172.21.15.43:0/38022868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 msgr2=0x560f2c3805a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.195+0000 7f40c5e00700 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c3805a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.196+0000 7f40c5e00700 1 -- 172.21.15.43:0/38022868 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f2c1d58d0 con 0x560f2c1e8e40 2024-05-27T07:28:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.196+0000 7f40c5400700 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c3805a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:28:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.196+0000 7f40c5e00700 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c1e8e40 0x560f2c380060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009d80 tx=0x7f40b8003940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd2cec792c025d74 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.196+0000 7f40a7e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:28:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.197+0000 7f40a7e00700 1 -- 172.21.15.43:0/38022868 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8027070 con 0x560f2c1e8e40 2024-05-27T07:28:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.197+0000 7f40a7e00700 1 -- 172.21.15.43:0/38022868 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f40b8005450 con 0x560f2c1e8e40 2024-05-27T07:28:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.197+0000 7f40dbde8b80 1 -- 172.21.15.43:0/38022868 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40b80043d0 con 0x560f2c1e8e40 2024-05-27T07:28:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.198+0000 7f40a7e00700 1 -- 172.21.15.43:0/38022868 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f40b8033050 con 0x560f2c1e8e40 2024-05-27T07:28:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.198+0000 7f40dbde8b80 10 client.36983 fetch_fsmap learned FSMap version 116 2024-05-27T07:28:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.198+0000 7f40dbde8b80 1 -- 172.21.15.43:0/38022868 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=4+}) -- 0x560f2c256060 con 0x560f2c1e8e40 2024-05-27T07:28:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.198+0000 7f40a7e00700 1 -- 172.21.15.43:0/38022868 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f40b80043d0 con 0x560f2c1e8e40 2024-05-27T07:28:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.198+0000 7f40a7e00700 1 -- 172.21.15.43:0/38022868 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 116) ==== 4816+0+0 (secure 0 0 0) 0x7f40b802d770 con 0x560f2c1e8e40 2024-05-27T07:28:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.198+0000 7f40dbde8b80 10 client.36983 fetch_fsmap finished waiting for FSMap version 116 2024-05-27T07:28:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.198+0000 7f40dbde8b80 10 client.36983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.198+0000 7f40dbde8b80 20 client.36983 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.198+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560f2c392da0 0x560f2c395270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.199+0000 7f40dbde8b80 4 client.36983 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:28:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.199+0000 7f40dbde8b80 1 -- 172.21.15.43:0/38022868 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f2c297d90 con 0x560f2c392da0 2024-05-27T07:28:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.199+0000 7f40c5400700 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560f2c392da0 0x560f2c395270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.200+0000 7f40a7e00700 10 client.36983 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.200+0000 7f40c5400700 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560f2c392da0 0x560f2c395270 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80ea4633a5a6f52a server_cookie=21dbd43ac725ee70 in_seq=0 out_seq=0 2024-05-27T07:28:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.202+0000 7f40a7e00700 1 -- 172.21.15.43:0/38022868 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560f2c297d90 con 0x560f2c392da0 2024-05-27T07:28:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.202+0000 7f40a7e00700 10 client.36983 handle_command_reply: tid=0 2024-05-27T07:28:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.202+0000 7f40dbde8b80 1 client.36983 shutdown 2024-05-27T07:28:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 1 -- 172.21.15.43:0/38022868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c1e8e40 msgr2=0x560f2c380060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c1e8e40 0x560f2c380060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009d80 tx=0x7f40b8003940 comp rx=0 tx=0).stop 2024-05-27T07:28:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 1 -- 172.21.15.43:0/38022868 shutdown_connections 2024-05-27T07:28:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c383110 0x560f2c380ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c1e8e40 0x560f2c380060 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560f2c392da0 0x560f2c395270 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/38022868 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c3805a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 1 -- 172.21.15.43:0/38022868 >> 172.21.15.43:0/38022868 conn(0x560f2c282390 msgr2=0x560f2c3674d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40a7e00700 0 client.36983 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 1 -- 172.21.15.43:0/38022868 shutdown_connections 2024-05-27T07:28:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 1 -- 172.21.15.43:0/38022868 wait complete. 2024-05-27T07:28:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.203+0000 7f40dbde8b80 20 client.36983 trim_cache size 0 max 16384 2024-05-27T07:28:59.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.239+0000 7f40dbde8b80 1 Processor -- start 2024-05-27T07:28:59.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.239+0000 7f40dbde8b80 1 -- start start 2024-05-27T07:28:59.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.240+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c395420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.240+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c363590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.240+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c383110 0x560f2c38ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.240+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81b7950 con 0x560f2c2d49f0 2024-05-27T07:28:59.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.240+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f40c8073cf0 con 0x560f2c383110 2024-05-27T07:28:59.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.240+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81b7280 con 0x560f2c1e8e40 2024-05-27T07:28:59.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.240+0000 7f40c5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c363590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.240+0000 7f40c5400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c363590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57186/0 (socket says 172.21.15.43:57186) 2024-05-27T07:28:59.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.240+0000 7f40c5400700 1 -- 172.21.15.43:0/1391338795 learned_addr learned my addr 172.21.15.43:0/1391338795 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:59.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.240+0000 7f40c5e00700 1 --2- 172.21.15.43:0/1391338795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c395420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.241+0000 7f40c5400700 1 -- 172.21.15.43:0/1391338795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 msgr2=0x560f2c395420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:59.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.241+0000 7f40c5400700 1 --2- 172.21.15.43:0/1391338795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c395420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.241+0000 7f40c5400700 1 -- 172.21.15.43:0/1391338795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c383110 msgr2=0x560f2c38ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:59.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.241+0000 7f40c5400700 1 --2- 172.21.15.43:0/1391338795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c383110 0x560f2c38ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.241+0000 7f40c5400700 1 -- 172.21.15.43:0/1391338795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f2c380e50 con 0x560f2c2d49f0 2024-05-27T07:28:59.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.241+0000 7f40c5e00700 1 --2- 172.21.15.43:0/1391338795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c395420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:59.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.241+0000 7f40c5400700 1 --2- 172.21.15.43:0/1391338795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c363590 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f40b400e910 tx=0x7f40b4003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c55d7a65e2b4ad1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:59.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.242+0000 7f40b2a00700 1 -- 172.21.15.43:0/1391338795 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b40a95d0 con 0x560f2c2d49f0 2024-05-27T07:28:59.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.242+0000 7f40b2a00700 1 -- 172.21.15.43:0/1391338795 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f40b40c8070 con 0x560f2c2d49f0 2024-05-27T07:28:59.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.242+0000 7f40b2a00700 1 -- 172.21.15.43:0/1391338795 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4003ca0 con 0x560f2c2d49f0 2024-05-27T07:28:59.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.243+0000 7f40dbde8b80 1 -- 172.21.15.43:0/1391338795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 msgr2=0x560f2c363590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:59.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.243+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/1391338795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c363590 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f40b400e910 tx=0x7f40b4003680 comp rx=0 tx=0).stop 2024-05-27T07:28:59.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.244+0000 7f40dbde8b80 1 -- 172.21.15.43:0/1391338795 shutdown_connections 2024-05-27T07:28:59.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.244+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/1391338795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c383110 0x560f2c38ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.244+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/1391338795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c395420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.244+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/1391338795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c2d49f0 0x560f2c363590 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.244+0000 7f40dbde8b80 1 -- 172.21.15.43:0/1391338795 >> 172.21.15.43:0/1391338795 conn(0x560f2c282390 msgr2=0x560f2c362e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:59.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.244+0000 7f40dbde8b80 1 -- 172.21.15.43:0/1391338795 shutdown_connections 2024-05-27T07:28:59.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.244+0000 7f40dbde8b80 1 -- 172.21.15.43:0/1391338795 wait complete. 2024-05-27T07:28:59.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.244+0000 7f40dbde8b80 1 -- start start 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.245+0000 7f40dbde8b80 1 Processor -- start 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.246+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c399a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.246+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c399f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.246+0000 7f40dbde8b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 0x560f2c39a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.246+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f40c8074b20 con 0x560f2c383110 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.246+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f40c8074420 con 0x560f2c2d49f0 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.246+0000 7f40dbde8b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f40c8075250 con 0x560f2c1e8e40 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.246+0000 7f40dbde8b80 10 client.0 _ll_register_callbacks cb 0x560f2c22fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c399f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c399f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41602/0 (socket says 172.21.15.43:41602) 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c5400700 1 -- 172.21.15.43:0/2457461004 learned_addr learned my addr 172.21.15.43:0/2457461004 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:28:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c6800700 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 0x560f2c39a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c5e00700 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c399a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c5400700 1 -- 172.21.15.43:0/2457461004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 msgr2=0x560f2c399a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c5400700 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c399a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c5400700 1 -- 172.21.15.43:0/2457461004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 msgr2=0x560f2c39a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c5400700 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 0x560f2c39a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c5400700 1 -- 172.21.15.43:0/2457461004 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f2c389fa0 con 0x560f2c2d49f0 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.247+0000 7f40c5e00700 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c399a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.248+0000 7f40c6800700 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 0x560f2c39a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.248+0000 7f40c5400700 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c399f60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f40b40a9510 tx=0x7f40b40b24e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9d8202a30f1858c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.248+0000 7f40a7e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.248+0000 7f40a7e00700 1 -- 172.21.15.43:0/2457461004 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4003ca0 con 0x560f2c2d49f0 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.248+0000 7f40a7e00700 1 -- 172.21.15.43:0/2457461004 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9496+0+0 (secure 0 0 0) 0x7f40b40a9830 con 0x560f2c2d49f0 2024-05-27T07:28:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.248+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2457461004 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40b40c8070 con 0x560f2c2d49f0 2024-05-27T07:28:59.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.249+0000 7f40a7e00700 1 -- 172.21.15.43:0/2457461004 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f40b4003e40 con 0x560f2c2d49f0 2024-05-27T07:28:59.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.249+0000 7f40dbde8b80 10 client.36986 fetch_fsmap learned FSMap version 116 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.249+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2457461004 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=4+}) -- 0x560f2c38b210 con 0x560f2c2d49f0 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.250+0000 7f40a7e00700 1 -- 172.21.15.43:0/2457461004 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f40b40c8070 con 0x560f2c2d49f0 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.250+0000 7f40a7e00700 1 -- 172.21.15.43:0/2457461004 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 116) ==== 4816+0+0 (secure 0 0 0) 0x7f40b40e2270 con 0x560f2c2d49f0 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.250+0000 7f40dbde8b80 10 client.36986 fetch_fsmap finished waiting for FSMap version 116 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.250+0000 7f40dbde8b80 10 client.36986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.250+0000 7f40dbde8b80 20 client.36986 populate_metadata read hostname 'smithi043' 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.250+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560f2c3fd900 0x560f2c3ffd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.251+0000 7f40dbde8b80 4 client.36986 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.251+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2457461004 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560f2c291570 con 0x560f2c3fd900 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.251+0000 7f40c5e00700 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560f2c3fd900 0x560f2c3ffd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.252+0000 7f40a7e00700 10 client.36986 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.252+0000 7f40c5e00700 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560f2c3fd900 0x560f2c3ffd20 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=380fc921c87d46a2 server_cookie=460cd59c5105e18b in_seq=0 out_seq=0 2024-05-27T07:28:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.252+0000 7f40a7e00700 1 -- 172.21.15.43:0/2457461004 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560f2c291570 con 0x560f2c3fd900 2024-05-27T07:28:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.252+0000 7f40a7e00700 10 client.36986 handle_command_reply: tid=0 2024-05-27T07:28:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40dbde8b80 1 client.36986 shutdown 2024-05-27T07:28:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2457461004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 msgr2=0x560f2c399f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c399f60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f40b40a9510 tx=0x7f40b40b24e0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2457461004 shutdown_connections 2024-05-27T07:28:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x560f2c3fd900 0x560f2c3ffd20 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f2c383110 0x560f2c39a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f2c1e8e40 0x560f2c399a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40dbde8b80 1 --2- 172.21.15.43:0/2457461004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2c2d49f0 0x560f2c399f60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40a7e00700 0 client.36986 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:28:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2457461004 >> 172.21.15.43:0/2457461004 conn(0x560f2c282390 msgr2=0x560f2c37e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.253+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2457461004 shutdown_connections 2024-05-27T07:28:59.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.254+0000 7f40dbde8b80 1 -- 172.21.15.43:0/2457461004 wait complete. 2024-05-27T07:28:59.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.254+0000 7f40dbde8b80 20 client.36986 trim_cache size 0 max 16384 2024-05-27T07:28:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:28:59.246 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:28:59.246 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:28:59.246 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:28:59.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.255+0000 7f40cd200700 1 -- 172.21.15.43:0/3075488386 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f40ac0b2c70 msgr2=0x7f40ac0b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:59.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.255+0000 7f40cd200700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f40ac0b2c70 0x7f40ac0b5180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f40c810ad00 tx=0x7f40c000bb90 comp rx=0 tx=0).stop 2024-05-27T07:28:59.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.255+0000 7f40cd200700 1 -- 172.21.15.43:0/3075488386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40c8103a50 msgr2=0x7f40c810a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:28:59.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.255+0000 7f40cd200700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40c8103a50 0x7f40c810a630 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f40b400e940 tx=0x7f40b400ec50 comp rx=0 tx=0).stop 2024-05-27T07:28:59.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.255+0000 7f40cd200700 1 -- 172.21.15.43:0/3075488386 shutdown_connections 2024-05-27T07:28:59.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.255+0000 7f40cd200700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f40ac0b2c70 0x7f40ac0b5180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.255+0000 7f40cd200700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40c8110c20 0x7f40c810fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.255+0000 7f40cd200700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40c8103070 0x7f40c810a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.256+0000 7f40cd200700 1 --2- 172.21.15.43:0/3075488386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40c8103a50 0x7f40c810a630 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:28:59.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.256+0000 7f40cd200700 1 -- 172.21.15.43:0/3075488386 >> 172.21.15.43:0/3075488386 conn(0x7f40c80fedb0 msgr2=0x7f40c81127a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:28:59.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.256+0000 7f40cd200700 1 -- 172.21.15.43:0/3075488386 shutdown_connections 2024-05-27T07:28:59.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:28:59.256+0000 7f40cd200700 1 -- 172.21.15.43:0/3075488386 wait complete. 2024-05-27T07:28:59.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 6981 9.28 MB/sec warmup 60 sec latency 88.436 ms 2024-05-27T07:29:00.198 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:29:00.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:29:00.632 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:00.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7010 9.15 MB/sec warmup 61 sec latency 674.001 ms 2024-05-27T07:29:00.788 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:00 smithi043 ceph-mon[31531]: pgmap v505: 129 pgs: 129 active+clean; 417 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-05-27T07:29:00.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:00 smithi132 ceph-mon[35283]: pgmap v505: 129 pgs: 129 active+clean; 417 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-05-27T07:29:00.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:00 smithi179 ceph-mon[35888]: pgmap v505: 129 pgs: 129 active+clean; 417 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-05-27T07:29:01.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7061 9.05 MB/sec warmup 62 sec latency 1032.472 ms 2024-05-27T07:29:02.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:02 smithi043 ceph-mon[31531]: pgmap v506: 129 pgs: 129 active+clean; 430 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 4.7 MiB/s wr, 213 op/s 2024-05-27T07:29:02.769 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7159 8.99 MB/sec warmup 63 sec latency 122.658 ms 2024-05-27T07:29:02.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:02 smithi132 ceph-mon[35283]: pgmap v506: 129 pgs: 129 active+clean; 430 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 4.7 MiB/s wr, 213 op/s 2024-05-27T07:29:02.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:02 smithi179 ceph-mon[35888]: pgmap v506: 129 pgs: 129 active+clean; 430 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 4.7 MiB/s wr, 213 op/s 2024-05-27T07:29:03.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.062+0000 7f30a4000700 1 -- 172.21.15.43:0/4239403088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f309c115360 msgr2=0x7f309c117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:03.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.062+0000 7f30a4000700 1 --2- 172.21.15.43:0/4239403088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f309c115360 0x7f309c117780 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f308c0098f0 tx=0x7f308c009c00 comp rx=0 tx=0).stop 2024-05-27T07:29:03.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.063+0000 7f30a4000700 1 -- 172.21.15.43:0/4239403088 shutdown_connections 2024-05-27T07:29:03.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.063+0000 7f30a4000700 1 --2- 172.21.15.43:0/4239403088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f309c115360 0x7f309c117780 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:03.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.063+0000 7f30a4000700 1 --2- 172.21.15.43:0/4239403088 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f309c075b50 0x7f309c075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:03.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.063+0000 7f30a4000700 1 --2- 172.21.15.43:0/4239403088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f309c0764a0 0x7f309c114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:03.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.063+0000 7f30a4000700 1 -- 172.21.15.43:0/4239403088 >> 172.21.15.43:0/4239403088 conn(0x7f309c100030 msgr2=0x7f309c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:03.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.064+0000 7f30a4000700 1 -- 172.21.15.43:0/4239403088 shutdown_connections 2024-05-27T07:29:03.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.064+0000 7f30a4000700 1 -- 172.21.15.43:0/4239403088 wait complete. 2024-05-27T07:29:03.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.065+0000 7f30a4000700 1 Processor -- start 2024-05-27T07:29:03.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.065+0000 7f30a4000700 1 -- start start 2024-05-27T07:29:03.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.066+0000 7f30a4000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f309c075b50 0x7f309c114c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:03.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.066+0000 7f30a4000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f309c0764a0 0x7f309c10edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:03.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.066+0000 7f30a4000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f309c115360 0x7f309c10f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:03.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.066+0000 7f30a4000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f309c108d50 con 0x7f309c115360 2024-05-27T07:29:03.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.066+0000 7f30a4000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f309c108bd0 con 0x7f309c0764a0 2024-05-27T07:29:03.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.066+0000 7f30a4000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f309c108ed0 con 0x7f309c075b50 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.066+0000 7f30a0c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f309c0764a0 0x7f309c10edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.066+0000 7f30a0c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f309c0764a0 0x7f309c10edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41622/0 (socket says 172.21.15.43:41622) 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.066+0000 7f30a0c00700 1 -- 172.21.15.43:0/4200570012 learned_addr learned my addr 172.21.15.43:0/4200570012 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.066+0000 7f30a1600700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f309c075b50 0x7f309c114c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.067+0000 7f30a2000700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f309c115360 0x7f309c10f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.067+0000 7f30a0c00700 1 -- 172.21.15.43:0/4200570012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f309c075b50 msgr2=0x7f309c114c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.067+0000 7f30a0c00700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f309c075b50 0x7f309c114c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.067+0000 7f30a0c00700 1 -- 172.21.15.43:0/4200570012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f309c115360 msgr2=0x7f309c10f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.067+0000 7f30a0c00700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f309c115360 0x7f309c10f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.067+0000 7f30a0c00700 1 -- 172.21.15.43:0/4200570012 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309c10fb60 con 0x7f309c0764a0 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.068+0000 7f30a0c00700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f309c0764a0 0x7f309c10edb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f309000c0c0 tx=0x7f309000c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:03.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.068+0000 7f309a000700 1 -- 172.21.15.43:0/4200570012 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309000cd10 con 0x7f309c0764a0 2024-05-27T07:29:03.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.068+0000 7f309a000700 1 -- 172.21.15.43:0/4200570012 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30900075c0 con 0x7f309c0764a0 2024-05-27T07:29:03.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.068+0000 7f30a4000700 1 -- 172.21.15.43:0/4200570012 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f309c1b5fd0 con 0x7f309c0764a0 2024-05-27T07:29:03.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.068+0000 7f30a4000700 1 -- 172.21.15.43:0/4200570012 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f309c1b6480 con 0x7f309c0764a0 2024-05-27T07:29:03.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.069+0000 7f309a000700 1 -- 172.21.15.43:0/4200570012 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30900146e0 con 0x7f309c0764a0 2024-05-27T07:29:03.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.071+0000 7f30a4000700 1 -- 172.21.15.43:0/4200570012 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3080005330 con 0x7f309c0764a0 2024-05-27T07:29:03.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.075+0000 7f309a000700 1 -- 172.21.15.43:0/4200570012 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f309001b020 con 0x7f309c0764a0 2024-05-27T07:29:03.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.076+0000 7f309a000700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30840b2b40 0x7f30840b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:03.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.077+0000 7f309a000700 1 -- 172.21.15.43:0/4200570012 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 9584+0+0 (secure 0 0 0) 0x7f30900e2660 con 0x7f309c0764a0 2024-05-27T07:29:03.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.077+0000 7f30a1600700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30840b2b40 0x7f30840b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:03.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.078+0000 7f30a1600700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30840b2b40 0x7f30840b5050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3094009c60 tx=0x7f3094009260 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:03.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.085+0000 7f309a000700 1 -- 172.21.15.43:0/4200570012 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3090010040 con 0x7f309c0764a0 2024-05-27T07:29:03.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.580+0000 7f30a4000700 1 -- 172.21.15.43:0/4200570012 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30800058f0 con 0x7f309c0764a0 2024-05-27T07:29:03.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.582+0000 7f309a000700 1 -- 172.21.15.43:0/4200570012 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+6967 (secure 0 0 0) 0x7f30900a9c20 con 0x7f309c0764a0 2024-05-27T07:29:03.573 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:29:03.573 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:28:45.404907+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6841/2697504369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2697504369},{"type":"v1","addr":"172.21.15.43:6841","nonce":2697504369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34448,34484,34424,34403,24484]},"id":1}]} 2024-05-27T07:29:03.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 -- 172.21.15.43:0/4200570012 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30840b2b40 msgr2=0x7f30840b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:03.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30840b2b40 0x7f30840b5050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3094009c60 tx=0x7f3094009260 comp rx=0 tx=0).stop 2024-05-27T07:29:03.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 -- 172.21.15.43:0/4200570012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f309c0764a0 msgr2=0x7f309c10edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:03.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f309c0764a0 0x7f309c10edb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f309000c0c0 tx=0x7f309000c3d0 comp rx=0 tx=0).stop 2024-05-27T07:29:03.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 -- 172.21.15.43:0/4200570012 shutdown_connections 2024-05-27T07:29:03.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f30840b2b40 0x7f30840b5050 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:03.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f309c115360 0x7f309c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:03.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f309c075b50 0x7f309c114c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:03.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 --2- 172.21.15.43:0/4200570012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f309c0764a0 0x7f309c10edb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:03.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 -- 172.21.15.43:0/4200570012 >> 172.21.15.43:0/4200570012 conn(0x7f309c100030 msgr2=0x7f309c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:03.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 -- 172.21.15.43:0/4200570012 shutdown_connections 2024-05-27T07:29:03.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:03.586+0000 7f30a4000700 1 -- 172.21.15.43:0/4200570012 wait complete. 2024-05-27T07:29:03.578 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 116 2024-05-27T07:29:03.769 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7284 9.00 MB/sec warmup 64 sec latency 81.168 ms 2024-05-27T07:29:03.839 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:03 smithi132 ceph-mon[35283]: osdmap e138: 12 total, 12 up, 12 in 2024-05-27T07:29:03.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:03 smithi043 ceph-mon[31531]: osdmap e138: 12 total, 12 up, 12 in 2024-05-27T07:29:03.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:03 smithi179 ceph-mon[35888]: osdmap e138: 12 total, 12 up, 12 in 2024-05-27T07:29:04.410 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:29:04.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-mon[31531]: pgmap v508: 129 pgs: 129 active+clean; 430 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 4.3 MiB/s wr, 167 op/s 2024-05-27T07:29:04.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/4200570012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:29:04.768 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7385 8.91 MB/sec warmup 65 sec latency 101.037 ms 2024-05-27T07:29:04.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:04 smithi132 ceph-mon[35283]: pgmap v508: 129 pgs: 129 active+clean; 430 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 4.3 MiB/s wr, 167 op/s 2024-05-27T07:29:04.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:04 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/4200570012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:29:04.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:04 smithi179 ceph-mon[35888]: pgmap v508: 129 pgs: 129 active+clean; 430 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 4.3 MiB/s wr, 167 op/s 2024-05-27T07:29:04.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:04 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/4200570012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:29:05.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcc60400700 time 2024-05-27T07:29:04.716639+0000 2024-05-27T07:29:05.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:05.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:05.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcc75ff2ba6] 2024-05-27T07:29:05.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcc75ff2dd1] 2024-05-27T07:29:05.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851976e23d] 2024-05-27T07:29:05.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558519759971] 2024-05-27T07:29:05.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 5: (SnapRealm::check_cache() const+0x13e) [0x558519757fa0] 2024-05-27T07:29:05.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558519758651] 2024-05-27T07:29:05.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5585196ed347] 2024-05-27T07:29:05.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5585193e987e] 2024-05-27T07:29:05.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5585193eee94] 2024-05-27T07:29:05.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5585193f9969] 2024-05-27T07:29:05.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 11: (MDLog::_replay_thread()+0x1f38) [0x55851979248c] 2024-05-27T07:29:05.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55851979b9cb] 2024-05-27T07:29:05.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 13: (Thread::entry_wrapper()+0x43) [0x7fcc75fc172d] 2024-05-27T07:29:05.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcc75fc1749] 2024-05-27T07:29:05.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcc74b651ca] 2024-05-27T07:29:05.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 16: clone() 2024-05-27T07:29:05.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:04.718+0000 7fcc60400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcc60400700 time 2024-05-27T07:29:04.716639+0000 2024-05-27T07:29:05.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:05.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:05.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcc75ff2ba6] 2024-05-27T07:29:05.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcc75ff2dd1] 2024-05-27T07:29:05.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851976e23d] 2024-05-27T07:29:05.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558519759971] 2024-05-27T07:29:05.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 5: (SnapRealm::check_cache() const+0x13e) [0x558519757fa0] 2024-05-27T07:29:05.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558519758651] 2024-05-27T07:29:05.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5585196ed347] 2024-05-27T07:29:05.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5585193e987e] 2024-05-27T07:29:05.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5585193eee94] 2024-05-27T07:29:05.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5585193f9969] 2024-05-27T07:29:05.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 11: (MDLog::_replay_thread()+0x1f38) [0x55851979248c] 2024-05-27T07:29:05.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55851979b9cb] 2024-05-27T07:29:05.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 13: (Thread::entry_wrapper()+0x43) [0x7fcc75fc172d] 2024-05-27T07:29:05.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcc75fc1749] 2024-05-27T07:29:05.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcc74b651ca] 2024-05-27T07:29:05.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 16: clone() 2024-05-27T07:29:05.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: *** Caught signal (Aborted) ** 2024-05-27T07:29:05.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: in thread 7fcc60400700 thread_name:md_log_replay 2024-05-27T07:29:05.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:05.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcc74b6fd20] 2024-05-27T07:29:05.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2: gsignal() 2024-05-27T07:29:05.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 3: abort() 2024-05-27T07:29:05.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcc75ff2ced] 2024-05-27T07:29:05.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcc75ff2dd1] 2024-05-27T07:29:05.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851976e23d] 2024-05-27T07:29:05.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558519759971] 2024-05-27T07:29:05.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 8: (SnapRealm::check_cache() const+0x13e) [0x558519757fa0] 2024-05-27T07:29:05.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558519758651] 2024-05-27T07:29:05.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5585196ed347] 2024-05-27T07:29:05.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5585193e987e] 2024-05-27T07:29:05.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5585193eee94] 2024-05-27T07:29:05.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5585193f9969] 2024-05-27T07:29:05.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 14: (MDLog::_replay_thread()+0x1f38) [0x55851979248c] 2024-05-27T07:29:05.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55851979b9cb] 2024-05-27T07:29:05.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 16: (Thread::entry_wrapper()+0x43) [0x7fcc75fc172d] 2024-05-27T07:29:05.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcc75fc1749] 2024-05-27T07:29:05.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcc74b651ca] 2024-05-27T07:29:05.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 19: clone() 2024-05-27T07:29:05.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:04.721+0000 7fcc60400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:05.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: in thread 7fcc60400700 thread_name:md_log_replay 2024-05-27T07:29:05.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:05.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcc74b6fd20] 2024-05-27T07:29:05.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2: gsignal() 2024-05-27T07:29:05.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 3: abort() 2024-05-27T07:29:05.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcc75ff2ced] 2024-05-27T07:29:05.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcc75ff2dd1] 2024-05-27T07:29:05.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851976e23d] 2024-05-27T07:29:05.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558519759971] 2024-05-27T07:29:05.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 8: (SnapRealm::check_cache() const+0x13e) [0x558519757fa0] 2024-05-27T07:29:05.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558519758651] 2024-05-27T07:29:05.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5585196ed347] 2024-05-27T07:29:05.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5585193e987e] 2024-05-27T07:29:05.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5585193eee94] 2024-05-27T07:29:05.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5585193f9969] 2024-05-27T07:29:05.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 14: (MDLog::_replay_thread()+0x1f38) [0x55851979248c] 2024-05-27T07:29:05.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55851979b9cb] 2024-05-27T07:29:05.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 16: (Thread::entry_wrapper()+0x43) [0x7fcc75fc172d] 2024-05-27T07:29:05.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcc75fc1749] 2024-05-27T07:29:05.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcc74b651ca] 2024-05-27T07:29:05.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 19: clone() 2024-05-27T07:29:05.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: -1> 2024-05-27T07:29:04.718+0000 7fcc60400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcc60400700 time 2024-05-27T07:29:04.716639+0000 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcc75ff2ba6] 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcc75ff2dd1] 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851976e23d] 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558519759971] 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 5: (SnapRealm::check_cache() const+0x13e) [0x558519757fa0] 2024-05-27T07:29:05.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558519758651] 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5585196ed347] 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5585193e987e] 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5585193eee94] 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5585193f9969] 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 11: (MDLog::_replay_thread()+0x1f38) [0x55851979248c] 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55851979b9cb] 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 13: (Thread::entry_wrapper()+0x43) [0x7fcc75fc172d] 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcc75fc1749] 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcc74b651ca] 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 16: clone() 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 0> 2024-05-27T07:29:04.721+0000 7fcc60400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: in thread 7fcc60400700 thread_name:md_log_replay 2024-05-27T07:29:05.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcc74b6fd20] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2: gsignal() 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 3: abort() 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcc75ff2ced] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcc75ff2dd1] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851976e23d] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558519759971] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 8: (SnapRealm::check_cache() const+0x13e) [0x558519757fa0] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558519758651] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5585196ed347] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5585193e987e] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5585193eee94] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5585193f9969] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 14: (MDLog::_replay_thread()+0x1f38) [0x55851979248c] 2024-05-27T07:29:05.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55851979b9cb] 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 16: (Thread::entry_wrapper()+0x43) [0x7fcc75fc172d] 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcc75fc1749] 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcc74b651ca] 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 19: clone() 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: -1> 2024-05-27T07:29:04.718+0000 7fcc60400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fcc60400700 time 2024-05-27T07:29:04.716639+0000 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fcc75ff2ba6] 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcc75ff2dd1] 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851976e23d] 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 4: (SnapRealm::build_snap_set() const+0x323) [0x558519759971] 2024-05-27T07:29:05.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 5: (SnapRealm::check_cache() const+0x13e) [0x558519757fa0] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558519758651] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5585196ed347] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5585193e987e] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5585193eee94] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x5585193f9969] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 11: (MDLog::_replay_thread()+0x1f38) [0x55851979248c] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55851979b9cb] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 13: (Thread::entry_wrapper()+0x43) [0x7fcc75fc172d] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 14: (Thread::_entry_func(void*)+0xd) [0x7fcc75fc1749] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcc74b651ca] 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 16: clone() 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 0> 2024-05-27T07:29:04.721+0000 7fcc60400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: in thread 7fcc60400700 thread_name:md_log_replay 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcc74b6fd20] 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2: gsignal() 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 3: abort() 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fcc75ff2ced] 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fcc75ff2dd1] 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55851976e23d] 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 7: (SnapRealm::build_snap_set() const+0x323) [0x558519759971] 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 8: (SnapRealm::check_cache() const+0x13e) [0x558519757fa0] 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x558519758651] 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x5585196ed347] 2024-05-27T07:29:05.219 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x5585193e987e] 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x5585193eee94] 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x5585193f9969] 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 14: (MDLog::_replay_thread()+0x1f38) [0x55851979248c] 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55851979b9cb] 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 16: (Thread::entry_wrapper()+0x43) [0x7fcc75fc172d] 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 17: (Thread::_entry_func(void*)+0xd) [0x7fcc75fc1749] 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcc74b651ca] 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 19: clone() 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[70839]: 2024-05-27T07:29:05.220 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:04 smithi043 podman[118302]: 2024-05-27 07:29:04.907130755 +0000 UTC m=+0.055055149 container died 840760a352fe09043f334b550867b03c9a81526b36d2918c680af6bd450ed588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2) 2024-05-27T07:29:05.770 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7488 8.84 MB/sec warmup 66 sec latency 102.301 ms 2024-05-27T07:29:05.772 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:05.944 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:05 smithi043 podman[118302]: 2024-05-27 07:29:05.606073646 +0000 UTC m=+0.753998044 container remove 840760a352fe09043f334b550867b03c9a81526b36d2918c680af6bd450ed588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-27T07:29:05.944 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:05 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:06.538 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:06 smithi043 ceph-mon[31531]: pgmap v509: 129 pgs: 129 active+clean; 430 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 4.3 MiB/s wr, 167 op/s 2024-05-27T07:29:06.769 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7580 8.77 MB/sec warmup 67 sec latency 102.053 ms 2024-05-27T07:29:06.788 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:06 smithi043 podman[118373]: 2024-05-27 07:29:06.569884928 +0000 UTC m=+0.663515079 container died 00526791b0c2588d8ab1922a1d43df0002616454566c44de821590083fa6d99b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=stoic_elbakyan, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:29:06.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:06 smithi132 ceph-mon[35283]: pgmap v509: 129 pgs: 129 active+clean; 430 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 4.3 MiB/s wr, 167 op/s 2024-05-27T07:29:06.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:06 smithi179 ceph-mon[35888]: pgmap v509: 129 pgs: 129 active+clean; 430 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 4.3 MiB/s wr, 167 op/s 2024-05-27T07:29:07.201 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:06 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:29:07.201 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:06 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Consumed 7.507s CPU time. 2024-05-27T07:29:07.769 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7691 8.76 MB/sec warmup 68 sec latency 82.900 ms 2024-05-27T07:29:08.769 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7786 8.66 MB/sec warmup 69 sec latency 77.717 ms 2024-05-27T07:29:08.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:08 smithi132 ceph-mon[35283]: pgmap v510: 129 pgs: 129 active+clean; 441 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 4.2 MiB/s wr, 178 op/s 2024-05-27T07:29:08.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:08 smithi043 ceph-mon[31531]: pgmap v510: 129 pgs: 129 active+clean; 441 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 4.2 MiB/s wr, 178 op/s 2024-05-27T07:29:08.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:08 smithi179 ceph-mon[35888]: pgmap v510: 129 pgs: 129 active+clean; 441 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 4.2 MiB/s wr, 178 op/s 2024-05-27T07:29:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.250+0000 7fe819e00700 1 -- 172.21.15.43:0/3776964664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe814110980 msgr2=0x7fe814110d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.250+0000 7fe819e00700 1 --2- 172.21.15.43:0/3776964664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe814110980 0x7fe814110d90 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fe804009990 tx=0x7fe804009ca0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.251+0000 7fe819e00700 1 -- 172.21.15.43:0/3776964664 shutdown_connections 2024-05-27T07:29:09.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.251+0000 7fe819e00700 1 --2- 172.21.15.43:0/3776964664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe81410b2a0 0x7fe814117820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.251+0000 7fe819e00700 1 --2- 172.21.15.43:0/3776964664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe814110980 0x7fe814110d90 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.251+0000 7fe819e00700 1 --2- 172.21.15.43:0/3776964664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe81410a950 0x7fe81410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.251+0000 7fe819e00700 1 -- 172.21.15.43:0/3776964664 >> 172.21.15.43:0/3776964664 conn(0x7fe814076d20 msgr2=0x7fe814077130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.251+0000 7fe819e00700 1 -- 172.21.15.43:0/3776964664 shutdown_connections 2024-05-27T07:29:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.251+0000 7fe819e00700 1 -- 172.21.15.43:0/3776964664 wait complete. 2024-05-27T07:29:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.253+0000 7fe819e00700 1 Processor -- start 2024-05-27T07:29:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.253+0000 7fe819e00700 1 -- start start 2024-05-27T07:29:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.253+0000 7fe819e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe81410a950 0x7fe8141a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.253+0000 7fe819e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81410b2a0 0x7fe8141a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.254+0000 7fe819e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe814110980 0x7fe8141a5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.254+0000 7fe819e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe81411a990 con 0x7fe81410b2a0 2024-05-27T07:29:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.254+0000 7fe819e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe81411a810 con 0x7fe81410a950 2024-05-27T07:29:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.254+0000 7fe819e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe81411ab10 con 0x7fe814110980 2024-05-27T07:29:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.254+0000 7fe812000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81410b2a0 0x7fe8141a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.254+0000 7fe812000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81410b2a0 0x7fe8141a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42026/0 (socket says 172.21.15.43:42026) 2024-05-27T07:29:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.254+0000 7fe812000700 1 -- 172.21.15.43:0/1752631938 learned_addr learned my addr 172.21.15.43:0/1752631938 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.254+0000 7fe812a00700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe81410a950 0x7fe8141a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.255+0000 7fe813400700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe814110980 0x7fe8141a5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.255+0000 7fe812000700 1 -- 172.21.15.43:0/1752631938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe814110980 msgr2=0x7fe8141a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.255+0000 7fe812000700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe814110980 0x7fe8141a5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.255+0000 7fe812000700 1 -- 172.21.15.43:0/1752631938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe81410a950 msgr2=0x7fe8141a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.255+0000 7fe812000700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe81410a950 0x7fe8141a1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.255+0000 7fe812000700 1 -- 172.21.15.43:0/1752631938 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8141a62f0 con 0x7fe81410b2a0 2024-05-27T07:29:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.255+0000 7fe812000700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81410b2a0 0x7fe8141a19c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe80000d980 tx=0x7fe80000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.256+0000 7fe818a00700 1 -- 172.21.15.43:0/1752631938 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800017070 con 0x7fe81410b2a0 2024-05-27T07:29:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.256+0000 7fe818a00700 1 -- 172.21.15.43:0/1752631938 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe800009dc0 con 0x7fe81410b2a0 2024-05-27T07:29:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.256+0000 7fe818a00700 1 -- 172.21.15.43:0/1752631938 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80000f3c0 con 0x7fe81410b2a0 2024-05-27T07:29:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.256+0000 7fe819e00700 1 -- 172.21.15.43:0/1752631938 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8141a65e0 con 0x7fe81410b2a0 2024-05-27T07:29:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.256+0000 7fe819e00700 1 -- 172.21.15.43:0/1752631938 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8141adf00 con 0x7fe81410b2a0 2024-05-27T07:29:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.262+0000 7fe818a00700 1 -- 172.21.15.43:0/1752631938 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe800009930 con 0x7fe81410b2a0 2024-05-27T07:29:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.264+0000 7fe8289ccb80 1 Processor -- start 2024-05-27T07:29:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.264+0000 7fe8289ccb80 1 -- start start 2024-05-27T07:29:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.265+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aea7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.265+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6ae4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.265+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6ae51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.265+0000 7fe8289ccb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d9f6a12e40 con 0x55d9f6ae4db0 2024-05-27T07:29:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.265+0000 7fe8289ccb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d9f6a029e0 con 0x55d9f6965e40 2024-05-27T07:29:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.265+0000 7fe8289ccb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d9f6a06a20 con 0x55d9f6aead30 2024-05-27T07:29:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.265+0000 7fe812000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6ae51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.266+0000 7fe813400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aea7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.266+0000 7fe812000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6ae51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42042/0 (socket says 172.21.15.43:42042) 2024-05-27T07:29:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.266+0000 7fe812000700 1 -- 172.21.15.43:0/1027478714 learned_addr learned my addr 172.21.15.43:0/1027478714 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.266+0000 7fe818a00700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe7f80b2bc0 0x7fe7f80b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.266+0000 7fe812a00700 1 --2- 172.21.15.43:0/1027478714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6ae4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.266+0000 7fe818a00700 1 -- 172.21.15.43:0/1752631938 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 9584+0+0 (secure 0 0 0) 0x7fe8000e2110 con 0x7fe81410b2a0 2024-05-27T07:29:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.266+0000 7fe812000700 1 -- 172.21.15.43:0/1027478714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 msgr2=0x55d9f6ae4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.266+0000 7fe812000700 1 --2- 172.21.15.43:0/1027478714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6ae4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.266+0000 7fe812000700 1 -- 172.21.15.43:0/1027478714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 msgr2=0x55d9f6aea7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.267+0000 7fe812000700 1 --2- 172.21.15.43:0/1027478714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aea7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.267+0000 7fe812000700 1 -- 172.21.15.43:0/1027478714 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9f69cb420 con 0x55d9f6ae4db0 2024-05-27T07:29:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.267+0000 7fe813400700 1 --2- 172.21.15.43:0/1027478714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aea7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.267+0000 7fe813400700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe7f80b2bc0 0x7fe7f80b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.267+0000 7fe812000700 1 --2- 172.21.15.43:0/1027478714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6ae51c0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fe8000a6200 tx=0x7fe800010ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5883e1d692b16ce5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.268+0000 7fe808c00700 1 -- 172.21.15.43:0/1027478714 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800020030 con 0x55d9f6ae4db0 2024-05-27T07:29:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.268+0000 7fe808c00700 1 -- 172.21.15.43:0/1027478714 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe800010460 con 0x55d9f6ae4db0 2024-05-27T07:29:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.268+0000 7fe808c00700 1 -- 172.21.15.43:0/1027478714 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800020030 con 0x55d9f6ae4db0 2024-05-27T07:29:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.269+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1027478714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 msgr2=0x55d9f6ae51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.269+0000 7fe813400700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe7f80b2bc0 0x7fe7f80b50d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x55d9f6ae5b30 tx=0x7fe80c009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.269+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/1027478714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6ae51c0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fe8000a6200 tx=0x7fe800010ed0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.270+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1027478714 shutdown_connections 2024-05-27T07:29:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.270+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/1027478714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6ae51c0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.270+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/1027478714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aea7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.270+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/1027478714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6ae4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.270+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1027478714 >> 172.21.15.43:0/1027478714 conn(0x55d9f69c5fc0 msgr2=0x55d9f6adf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:09.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.271+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1027478714 shutdown_connections 2024-05-27T07:29:09.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.271+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1027478714 wait complete. 2024-05-27T07:29:09.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.272+0000 7fe8289ccb80 1 -- start start 2024-05-27T07:29:09.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.272+0000 7fe8289ccb80 1 Processor -- start 2024-05-27T07:29:09.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.273+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.273+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6aeeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.273+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6aeefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.273+0000 7fe8289ccb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d9f6908fa0 con 0x55d9f6ae4db0 2024-05-27T07:29:09.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.273+0000 7fe8289ccb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d9f69a3760 con 0x55d9f6965e40 2024-05-27T07:29:09.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.273+0000 7fe8289ccb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d9f6af1740 con 0x55d9f6aead30 2024-05-27T07:29:09.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.273+0000 7fe8289ccb80 10 client.0 _ll_register_callbacks cb 0x55d9f69acc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:09.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.273+0000 7fe812a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.273+0000 7fe812a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:56020/0 (socket says 172.21.15.43:56020) 2024-05-27T07:29:09.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.273+0000 7fe812a00700 1 -- 172.21.15.43:0/396114479 learned_addr learned my addr 172.21.15.43:0/396114479 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:09.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.274+0000 7fe813400700 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6aeefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.274+0000 7fe812000700 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6aeeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.274+0000 7fe812a00700 1 -- 172.21.15.43:0/396114479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 msgr2=0x55d9f6aeefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:09.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.274+0000 7fe812a00700 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6aeefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.274+0000 7fe812a00700 1 -- 172.21.15.43:0/396114479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 msgr2=0x55d9f6aeeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:09.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.274+0000 7fe812a00700 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6aeeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.274+0000 7fe812a00700 1 -- 172.21.15.43:0/396114479 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9f69528d0 con 0x55d9f6965e40 2024-05-27T07:29:09.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.275+0000 7fe812000700 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6aeeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:09.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.275+0000 7fe812a00700 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aee560 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe804009990 tx=0x7fe8040057e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51f64b97b9e1fc9c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.275+0000 7fe7fd600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:29:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.275+0000 7fe7fd600700 1 -- 172.21.15.43:0/396114479 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804028060 con 0x55d9f6965e40 2024-05-27T07:29:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.275+0000 7fe8289ccb80 1 -- 172.21.15.43:0/396114479 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d9f6a14d90 con 0x55d9f6965e40 2024-05-27T07:29:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.276+0000 7fe7fd600700 1 -- 172.21.15.43:0/396114479 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 9584+0+0 (secure 0 0 0) 0x7fe80400b6b0 con 0x55d9f6965e40 2024-05-27T07:29:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.276+0000 7fe7fd600700 1 -- 172.21.15.43:0/396114479 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fe80402f050 con 0x55d9f6965e40 2024-05-27T07:29:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.276+0000 7fe8289ccb80 10 client.37004 fetch_fsmap learned FSMap version 116 2024-05-27T07:29:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.276+0000 7fe8289ccb80 1 -- 172.21.15.43:0/396114479 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=4+}) -- 0x55d9f69d3060 con 0x55d9f6965e40 2024-05-27T07:29:09.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.277+0000 7fe7fd600700 1 -- 172.21.15.43:0/396114479 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55d9f6a14d90 con 0x55d9f6965e40 2024-05-27T07:29:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.277+0000 7fe7fd600700 1 -- 172.21.15.43:0/396114479 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 116) ==== 4816+0+0 (secure 0 0 0) 0x7fe80400f7d0 con 0x55d9f6965e40 2024-05-27T07:29:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.277+0000 7fe8289ccb80 10 client.37004 fetch_fsmap finished waiting for FSMap version 116 2024-05-27T07:29:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.277+0000 7fe8289ccb80 10 client.37004 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:29:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.277+0000 7fe8289ccb80 20 client.37004 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.277+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55d9f6b0fa70 0x55d9f6b11f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.277+0000 7fe8289ccb80 4 client.37004 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.277+0000 7fe8289ccb80 1 -- 172.21.15.43:0/396114479 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9f6a0e570 con 0x55d9f6b0fa70 2024-05-27T07:29:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.279+0000 7fe812000700 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55d9f6b0fa70 0x55d9f6b11f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.280+0000 7fe812000700 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55d9f6b0fa70 0x55d9f6b11f20 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=482d53f90f14019c server_cookie=f63bc7f6449ebf77 in_seq=0 out_seq=0 2024-05-27T07:29:09.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.280+0000 7fe7fd600700 10 client.37004 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:29:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.282+0000 7fe7fd600700 1 -- 172.21.15.43:0/396114479 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d9f6a0e570 con 0x55d9f6b0fa70 2024-05-27T07:29:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.282+0000 7fe7fd600700 10 client.37004 handle_command_reply: tid=0 2024-05-27T07:29:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.282+0000 7fe8289ccb80 1 client.37004 shutdown 2024-05-27T07:29:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.283+0000 7fe8289ccb80 1 -- 172.21.15.43:0/396114479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 msgr2=0x55d9f6aee560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.283+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aee560 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe804009990 tx=0x7fe8040057e0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.283+0000 7fe8289ccb80 1 -- 172.21.15.43:0/396114479 shutdown_connections 2024-05-27T07:29:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.283+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55d9f6b0fa70 0x55d9f6b11f20 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.283+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6aeefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.283+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6965e40 0x55d9f6aee560 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.283+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/396114479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6ae4db0 0x55d9f6aeeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.283+0000 7fe7fd600700 0 client.37004 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:29:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.283+0000 7fe8289ccb80 1 -- 172.21.15.43:0/396114479 >> 172.21.15.43:0/396114479 conn(0x55d9f69c5fc0 msgr2=0x55d9f6adf580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:09.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.284+0000 7fe8289ccb80 1 -- 172.21.15.43:0/396114479 shutdown_connections 2024-05-27T07:29:09.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.284+0000 7fe8289ccb80 1 -- 172.21.15.43:0/396114479 wait complete. 2024-05-27T07:29:09.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.284+0000 7fe8289ccb80 20 client.37004 trim_cache size 0 max 16384 2024-05-27T07:29:09.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.309+0000 7fe8289ccb80 1 Processor -- start 2024-05-27T07:29:09.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.310+0000 7fe8289ccb80 1 -- start start 2024-05-27T07:29:09.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.310+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6965e40 0x55d9f6ae4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.310+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 0x55d9f6ae9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.310+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6aee340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.310+0000 7fe8289ccb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe8141af2c0 con 0x55d9f6965e40 2024-05-27T07:29:09.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.310+0000 7fe8289ccb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe814083640 con 0x55d9f6ae4db0 2024-05-27T07:29:09.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.310+0000 7fe8289ccb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe8141aebf0 con 0x55d9f6aead30 2024-05-27T07:29:09.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.310+0000 7fe812a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6965e40 0x55d9f6ae4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.311+0000 7fe812a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6965e40 0x55d9f6ae4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42060/0 (socket says 172.21.15.43:42060) 2024-05-27T07:29:09.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.311+0000 7fe812a00700 1 -- 172.21.15.43:0/2141644199 learned_addr learned my addr 172.21.15.43:0/2141644199 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:09.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.311+0000 7fe812000700 1 --2- 172.21.15.43:0/2141644199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 0x55d9f6ae9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.311+0000 7fe812a00700 1 -- 172.21.15.43:0/2141644199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 msgr2=0x55d9f6aee340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:09.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.311+0000 7fe812a00700 1 --2- 172.21.15.43:0/2141644199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6aee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.311+0000 7fe812a00700 1 -- 172.21.15.43:0/2141644199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 msgr2=0x55d9f6ae9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:09.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.311+0000 7fe812a00700 1 --2- 172.21.15.43:0/2141644199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 0x55d9f6ae9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.311+0000 7fe812a00700 1 -- 172.21.15.43:0/2141644199 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9f6aeab60 con 0x55d9f6965e40 2024-05-27T07:29:09.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.311+0000 7fe812a00700 1 --2- 172.21.15.43:0/2141644199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6965e40 0x55d9f6ae4870 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fe80400fb70 tx=0x7fe804009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6d909c86ae4167 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:09.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.312+0000 7fe7ffe00700 1 -- 172.21.15.43:0/2141644199 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804038030 con 0x55d9f6965e40 2024-05-27T07:29:09.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.312+0000 7fe7ffe00700 1 -- 172.21.15.43:0/2141644199 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe80400b2c0 con 0x55d9f6965e40 2024-05-27T07:29:09.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.312+0000 7fe7ffe00700 1 -- 172.21.15.43:0/2141644199 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804028040 con 0x55d9f6965e40 2024-05-27T07:29:09.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.313+0000 7fe8289ccb80 1 -- 172.21.15.43:0/2141644199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6965e40 msgr2=0x55d9f6ae4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:09.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.313+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/2141644199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6965e40 0x55d9f6ae4870 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fe80400fb70 tx=0x7fe804009ca0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.315+0000 7fe8289ccb80 1 -- 172.21.15.43:0/2141644199 shutdown_connections 2024-05-27T07:29:09.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.315+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/2141644199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6aead30 0x55d9f6aee340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.315+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/2141644199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6965e40 0x55d9f6ae4870 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.315+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/2141644199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 0x55d9f6ae9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.315+0000 7fe8289ccb80 1 -- 172.21.15.43:0/2141644199 >> 172.21.15.43:0/2141644199 conn(0x55d9f69c5fc0 msgr2=0x55d9f6ae96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:09.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.315+0000 7fe8289ccb80 1 -- 172.21.15.43:0/2141644199 shutdown_connections 2024-05-27T07:29:09.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.315+0000 7fe8289ccb80 1 -- 172.21.15.43:0/2141644199 wait complete. 2024-05-27T07:29:09.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.316+0000 7fe8289ccb80 1 -- start start 2024-05-27T07:29:09.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.316+0000 7fe8289ccb80 1 Processor -- start 2024-05-27T07:29:09.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.317+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6965e40 0x55d9f6b17ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.318+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 0x55d9f6af6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.318+0000 7fe8289ccb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6aead30 0x55d9f6af6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.318+0000 7fe812a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6965e40 0x55d9f6b17ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.319+0000 7fe812a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6965e40 0x55d9f6b17ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:43888/0 (socket says 172.21.15.43:43888) 2024-05-27T07:29:09.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.319+0000 7fe812a00700 1 -- 172.21.15.43:0/1505086069 learned_addr learned my addr 172.21.15.43:0/1505086069 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:09.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.319+0000 7fe812000700 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 0x55d9f6af6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.319+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe814084470 con 0x55d9f6aead30 2024-05-27T07:29:09.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.319+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe814083d70 con 0x55d9f6ae4db0 2024-05-27T07:29:09.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.319+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe8141ae460 con 0x55d9f6965e40 2024-05-27T07:29:09.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.319+0000 7fe8289ccb80 10 client.0 _ll_register_callbacks cb 0x55d9f69acc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:09.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.320+0000 7fe813400700 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6aead30 0x55d9f6af6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.321+0000 7fe812000700 1 -- 172.21.15.43:0/1505086069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6965e40 msgr2=0x55d9f6b17ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:09.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.321+0000 7fe812000700 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6965e40 0x55d9f6b17ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.321+0000 7fe812000700 1 -- 172.21.15.43:0/1505086069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6aead30 msgr2=0x55d9f6af6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.321+0000 7fe812000700 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6aead30 0x55d9f6af6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.321+0000 7fe812000700 1 -- 172.21.15.43:0/1505086069 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9f6af7410 con 0x55d9f6ae4db0 2024-05-27T07:29:09.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.321+0000 7fe813400700 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6aead30 0x55d9f6af6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:09.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.322+0000 7fe812000700 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 0x55d9f6af6990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe8000a9840 tx=0x7fe8000a8b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e306fcfd337e5324 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:09.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.322+0000 7fe7fd600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:29:09.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.323+0000 7fe7fd600700 1 -- 172.21.15.43:0/1505086069 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800020040 con 0x55d9f6ae4db0 2024-05-27T07:29:09.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.323+0000 7fe7fd600700 1 -- 172.21.15.43:0/1505086069 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 9584+0+0 (secure 0 0 0) 0x7fe8000a91f0 con 0x55d9f6ae4db0 2024-05-27T07:29:09.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.323+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe80400b2c0 con 0x55d9f6ae4db0 2024-05-27T07:29:09.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.324+0000 7fe7fd600700 1 -- 172.21.15.43:0/1505086069 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fe8000a4e50 con 0x55d9f6ae4db0 2024-05-27T07:29:09.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.324+0000 7fe8289ccb80 10 client.37010 fetch_fsmap learned FSMap version 116 2024-05-27T07:29:09.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.324+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=4+}) -- 0x55d9f6b17540 con 0x55d9f6ae4db0 2024-05-27T07:29:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.325+0000 7fe7fd600700 1 -- 172.21.15.43:0/1505086069 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe80400b2c0 con 0x55d9f6ae4db0 2024-05-27T07:29:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.325+0000 7fe7fd600700 1 -- 172.21.15.43:0/1505086069 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 116) ==== 4816+0+0 (secure 0 0 0) 0x7fe800023060 con 0x55d9f6ae4db0 2024-05-27T07:29:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.325+0000 7fe8289ccb80 10 client.37010 fetch_fsmap finished waiting for FSMap version 116 2024-05-27T07:29:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.325+0000 7fe8289ccb80 10 client.37010 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:29:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.325+0000 7fe8289ccb80 20 client.37010 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.325+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55d9f6b79b30 0x55d9f6b7bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.325+0000 7fe8289ccb80 4 client.37010 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.325+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9f6885250 con 0x55d9f6b79b30 2024-05-27T07:29:09.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.326+0000 7fe812a00700 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55d9f6b79b30 0x55d9f6b7bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.327+0000 7fe812a00700 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55d9f6b79b30 0x55d9f6b7bf50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9de8e2e7d170bc95 server_cookie=d23c008d79a3f98a in_seq=0 out_seq=0 2024-05-27T07:29:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.327+0000 7fe7fd600700 10 client.37010 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:29:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.328+0000 7fe7fd600700 1 -- 172.21.15.43:0/1505086069 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9f6885250 con 0x55d9f6b79b30 2024-05-27T07:29:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.328+0000 7fe7fd600700 10 client.37010 handle_command_reply: tid=0 2024-05-27T07:29:09.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.328+0000 7fe8289ccb80 1 client.37010 shutdown 2024-05-27T07:29:09.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.328+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 msgr2=0x55d9f6af6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:09.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.328+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 0x55d9f6af6990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe8000a9840 tx=0x7fe8000a8b70 comp rx=0 tx=0).stop 2024-05-27T07:29:09.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.329+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 shutdown_connections 2024-05-27T07:29:09.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.329+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55d9f6b79b30 0x55d9f6b7bf50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.329+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d9f6aead30 0x55d9f6af6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.329+0000 7fe7fd600700 0 client.37010 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:29:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.329+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d9f6965e40 0x55d9f6b17ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.329+0000 7fe8289ccb80 1 --2- 172.21.15.43:0/1505086069 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9f6ae4db0 0x55d9f6af6990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.329+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 >> 172.21.15.43:0/1505086069 conn(0x55d9f69c5fc0 msgr2=0x55d9f6ae8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.329+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 shutdown_connections 2024-05-27T07:29:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.329+0000 7fe8289ccb80 1 -- 172.21.15.43:0/1505086069 wait complete. 2024-05-27T07:29:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.329+0000 7fe8289ccb80 20 client.37010 trim_cache size 0 max 16384 2024-05-27T07:29:09.321 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:29:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.330+0000 7fe819e00700 1 -- 172.21.15.43:0/1752631938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe7f80b2bc0 msgr2=0x7fe7f80b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.330+0000 7fe819e00700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe7f80b2bc0 0x7fe7f80b50d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x55d9f6ae5b30 tx=0x7fe80c009040 comp rx=0 tx=0).stop 2024-05-27T07:29:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.331+0000 7fe819e00700 1 -- 172.21.15.43:0/1752631938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81410b2a0 msgr2=0x7fe8141a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.331+0000 7fe819e00700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81410b2a0 0x7fe8141a19c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe80000d980 tx=0x7fe80000dc90 comp rx=0 tx=0).stop 2024-05-27T07:29:09.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.331+0000 7fe819e00700 1 -- 172.21.15.43:0/1752631938 shutdown_connections 2024-05-27T07:29:09.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.331+0000 7fe819e00700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe7f80b2bc0 0x7fe7f80b50d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.331+0000 7fe819e00700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe814110980 0x7fe8141a5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.331+0000 7fe819e00700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe81410a950 0x7fe8141a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.331+0000 7fe819e00700 1 --2- 172.21.15.43:0/1752631938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe81410b2a0 0x7fe8141a19c0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:09.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.331+0000 7fe819e00700 1 -- 172.21.15.43:0/1752631938 >> 172.21.15.43:0/1752631938 conn(0x7fe814076d20 msgr2=0x7fe8141092e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:09.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.332+0000 7fe819e00700 1 -- 172.21.15.43:0/1752631938 shutdown_connections 2024-05-27T07:29:09.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:09.332+0000 7fe819e00700 1 -- 172.21.15.43:0/1752631938 wait complete. 2024-05-27T07:29:09.769 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7878 8.57 MB/sec warmup 70 sec latency 201.827 ms 2024-05-27T07:29:10.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:29:10.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:10 smithi043 ceph-mon[31531]: pgmap v511: 129 pgs: 129 active+clean; 441 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 4.2 MiB/s wr, 178 op/s 2024-05-27T07:29:10.769 INFO:tasks.workunit.client.0.smithi043.stdout: 10 7999 8.59 MB/sec warmup 71 sec latency 620.133 ms 2024-05-27T07:29:10.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:10 smithi132 ceph-mon[35283]: pgmap v511: 129 pgs: 129 active+clean; 441 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 4.2 MiB/s wr, 178 op/s 2024-05-27T07:29:10.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:10 smithi179 ceph-mon[35888]: pgmap v511: 129 pgs: 129 active+clean; 441 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 4.2 MiB/s wr, 178 op/s 2024-05-27T07:29:10.974 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:11.769 INFO:tasks.workunit.client.0.smithi043.stdout: 10 8105 8.54 MB/sec warmup 72 sec latency 170.243 ms 2024-05-27T07:29:12.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:12 smithi043 ceph-mon[31531]: osdmap e139: 12 total, 12 up, 12 in 2024-05-27T07:29:12.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:12 smithi043 ceph-mon[31531]: pgmap v513: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 3.8 MiB/s wr, 224 op/s 2024-05-27T07:29:12.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:12 smithi179 ceph-mon[35888]: osdmap e139: 12 total, 12 up, 12 in 2024-05-27T07:29:12.708 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:12 smithi179 ceph-mon[35888]: pgmap v513: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 3.8 MiB/s wr, 224 op/s 2024-05-27T07:29:12.769 INFO:tasks.workunit.client.0.smithi043.stdout: 10 8214 8.45 MB/sec warmup 73 sec latency 87.265 ms 2024-05-27T07:29:12.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:12 smithi132 ceph-mon[35283]: osdmap e139: 12 total, 12 up, 12 in 2024-05-27T07:29:12.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:12 smithi132 ceph-mon[35283]: pgmap v513: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 3.8 MiB/s wr, 224 op/s 2024-05-27T07:29:13.770 INFO:tasks.workunit.client.0.smithi043.stdout: 10 8335 8.40 MB/sec warmup 74 sec latency 80.896 ms 2024-05-27T07:29:14.770 INFO:tasks.workunit.client.0.smithi043.stdout: 10 8445 8.30 MB/sec warmup 75 sec latency 99.919 ms 2024-05-27T07:29:14.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:14 smithi132 ceph-mon[35283]: pgmap v514: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 3.4 MiB/s wr, 200 op/s 2024-05-27T07:29:14.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:14 smithi043 ceph-mon[31531]: pgmap v514: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 3.4 MiB/s wr, 200 op/s 2024-05-27T07:29:14.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:14 smithi179 ceph-mon[35888]: pgmap v514: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 3.4 MiB/s wr, 200 op/s 2024-05-27T07:29:15.770 INFO:tasks.workunit.client.0.smithi043.stdout: 10 8555 8.24 MB/sec warmup 76 sec latency 259.222 ms 2024-05-27T07:29:16.770 INFO:tasks.workunit.client.0.smithi043.stdout: 10 8681 8.18 MB/sec warmup 77 sec latency 98.169 ms 2024-05-27T07:29:16.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:16 smithi132 ceph-mon[35283]: pgmap v515: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 3.4 MiB/s wr, 200 op/s 2024-05-27T07:29:16.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:16 smithi132 ceph-mon[35283]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:16.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:16 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:29:16.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:16 smithi043 ceph-mon[31531]: pgmap v515: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 3.4 MiB/s wr, 200 op/s 2024-05-27T07:29:16.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:16 smithi043 ceph-mon[31531]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:16.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:16 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:29:16.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:16 smithi179 ceph-mon[35888]: pgmap v515: 129 pgs: 129 active+clean; 453 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 3.4 MiB/s wr, 200 op/s 2024-05-27T07:29:16.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:16 smithi179 ceph-mon[35888]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:16.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:16 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:29:17.452 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:16 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-05-27T07:29:17.452 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:16 smithi043 systemd[1]: Stopped Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:29:17.452 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:16 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Consumed 7.507s CPU time. 2024-05-27T07:29:17.452 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:16 smithi043 systemd[1]: Starting Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:29:17.770 INFO:tasks.workunit.client.0.smithi043.stdout: 10 8766 8.13 MB/sec warmup 78 sec latency 363.920 ms 2024-05-27T07:29:18.770 INFO:tasks.workunit.client.0.smithi043.stdout: 10 8850 8.11 MB/sec warmup 79 sec latency 385.127 ms 2024-05-27T07:29:18.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:18 smithi132 ceph-mon[35283]: pgmap v516: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-05-27T07:29:18.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:18 smithi043 ceph-mon[31531]: pgmap v516: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-05-27T07:29:18.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:18 smithi179 ceph-mon[35888]: pgmap v516: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-05-27T07:29:19.770 INFO:tasks.workunit.client.0.smithi043.stdout: 10 8961 8.06 MB/sec warmup 80 sec latency 88.019 ms 2024-05-27T07:29:20.663 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:20 smithi043 ceph-mon[31531]: pgmap v517: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-05-27T07:29:20.770 INFO:tasks.workunit.client.0.smithi043.stdout: 10 9074 8.01 MB/sec warmup 81 sec latency 86.192 ms 2024-05-27T07:29:20.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:20 smithi132 ceph-mon[35283]: pgmap v517: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-05-27T07:29:20.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:20 smithi179 ceph-mon[35888]: pgmap v517: 129 pgs: 129 active+clean; 458 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-05-27T07:29:21.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.325+0000 7fd955400700 1 -- 172.21.15.43:0/304137867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9501043f0 msgr2=0x7fd950104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:21.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.325+0000 7fd955400700 1 --2- 172.21.15.43:0/304137867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9501043f0 0x7fd950104800 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9400098f0 tx=0x7fd940009c00 comp rx=0 tx=0).stop 2024-05-27T07:29:21.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.325+0000 7fd955400700 1 -- 172.21.15.43:0/304137867 shutdown_connections 2024-05-27T07:29:21.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.325+0000 7fd955400700 1 --2- 172.21.15.43:0/304137867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd950109670 0x7fd950110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.325+0000 7fd955400700 1 --2- 172.21.15.43:0/304137867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9501043f0 0x7fd950104800 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.325+0000 7fd955400700 1 --2- 172.21.15.43:0/304137867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd950108d20 0x7fd950109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.325+0000 7fd955400700 1 -- 172.21.15.43:0/304137867 >> 172.21.15.43:0/304137867 conn(0x7fd950079d00 msgr2=0x7fd95007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:21.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.326+0000 7fd955400700 1 -- 172.21.15.43:0/304137867 shutdown_connections 2024-05-27T07:29:21.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.326+0000 7fd955400700 1 -- 172.21.15.43:0/304137867 wait complete. 2024-05-27T07:29:21.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.327+0000 7fd955400700 1 Processor -- start 2024-05-27T07:29:21.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.327+0000 7fd955400700 1 -- start start 2024-05-27T07:29:21.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.328+0000 7fd955400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9501043f0 0x7fd9501a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.328+0000 7fd955400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd950108d20 0x7fd9501a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.328+0000 7fd955400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd950109670 0x7fd95019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.328+0000 7fd955400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd9501141a0 con 0x7fd9501043f0 2024-05-27T07:29:21.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.328+0000 7fd947400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd950108d20 0x7fd9501a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.329+0000 7fd947400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd950108d20 0x7fd9501a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:56694/0 (socket says 172.21.15.43:56694) 2024-05-27T07:29:21.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.329+0000 7fd947400700 1 -- 172.21.15.43:0/447145951 learned_addr learned my addr 172.21.15.43:0/447145951 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:21.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.329+0000 7fd94fe00700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9501043f0 0x7fd9501a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.329+0000 7fd955400700 1 -- 172.21.15.43:0/447145951 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd950114020 con 0x7fd950108d20 2024-05-27T07:29:21.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.329+0000 7fd955400700 1 -- 172.21.15.43:0/447145951 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd950114320 con 0x7fd950109670 2024-05-27T07:29:21.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.330+0000 7fd94fe00700 1 -- 172.21.15.43:0/447145951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd950109670 msgr2=0x7fd95019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:21.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.330+0000 7fd94fe00700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd950109670 0x7fd95019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.330+0000 7fd94fe00700 1 -- 172.21.15.43:0/447145951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd950108d20 msgr2=0x7fd9501a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:21.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.330+0000 7fd94fe00700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd950108d20 0x7fd9501a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.330+0000 7fd94fe00700 1 -- 172.21.15.43:0/447145951 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd95019b110 con 0x7fd9501043f0 2024-05-27T07:29:21.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.330+0000 7fd94fe00700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9501043f0 0x7fd9501a0840 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fd940009fd0 tx=0x7fd940009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:21.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.331+0000 7fd94d600700 1 -- 172.21.15.43:0/447145951 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94000b790 con 0x7fd9501043f0 2024-05-27T07:29:21.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.331+0000 7fd94d600700 1 -- 172.21.15.43:0/447145951 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd9400054d0 con 0x7fd9501043f0 2024-05-27T07:29:21.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.331+0000 7fd94d600700 1 -- 172.21.15.43:0/447145951 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9400039b0 con 0x7fd9501043f0 2024-05-27T07:29:21.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.331+0000 7fd955400700 1 -- 172.21.15.43:0/447145951 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd95019b3a0 con 0x7fd9501043f0 2024-05-27T07:29:21.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.331+0000 7fd955400700 1 -- 172.21.15.43:0/447145951 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd95019b830 con 0x7fd9501043f0 2024-05-27T07:29:21.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.338+0000 7fd94d600700 1 -- 172.21.15.43:0/447145951 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd940005040 con 0x7fd9501043f0 2024-05-27T07:29:21.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.339+0000 7fd9654d2b80 1 Processor -- start 2024-05-27T07:29:21.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.340+0000 7fd9654d2b80 1 -- start start 2024-05-27T07:29:21.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.340+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e26981e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.340+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26969ee40 0x55e26981ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.340+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 0x55e26982a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.340+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e26974be40 con 0x55e2698282c0 2024-05-27T07:29:21.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.340+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e26973b9e0 con 0x55e26978a9f0 2024-05-27T07:29:21.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.340+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e26973fa20 con 0x55e26969ee40 2024-05-27T07:29:21.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.341+0000 7fd954a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e26981e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.341+0000 7fd954a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e26981e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:56696/0 (socket says 172.21.15.43:56696) 2024-05-27T07:29:21.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.341+0000 7fd954a00700 1 -- 172.21.15.43:0/3479663101 learned_addr learned my addr 172.21.15.43:0/3479663101 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:21.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.342+0000 7fd947400700 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 0x55e26982a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.342+0000 7fd954a00700 1 -- 172.21.15.43:0/3479663101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26969ee40 msgr2=0x55e26981ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:21.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.342+0000 7fd94fe00700 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26969ee40 0x55e26981ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.343+0000 7fd94d600700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9340b2a70 0x7fd9340b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.343+0000 7fd954a00700 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26969ee40 0x55e26981ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.343+0000 7fd954a00700 1 -- 172.21.15.43:0/3479663101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 msgr2=0x55e26982a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:21.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.343+0000 7fd954a00700 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 0x55e26982a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.343+0000 7fd954a00700 1 -- 172.21.15.43:0/3479663101 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e269704420 con 0x55e26978a9f0 2024-05-27T07:29:21.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.343+0000 7fd94fe00700 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26969ee40 0x55e26981ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:21.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.343+0000 7fd947400700 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 0x55e26982a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:21.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.343+0000 7fd954a00700 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e26981e750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd948009be0 tx=0x7fd948009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c6948f996c76510 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:21.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.344+0000 7fd945600700 1 -- 172.21.15.43:0/3479663101 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd948004bb0 con 0x55e26978a9f0 2024-05-27T07:29:21.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.344+0000 7fd945600700 1 -- 172.21.15.43:0/3479663101 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd948004d50 con 0x55e26978a9f0 2024-05-27T07:29:21.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.344+0000 7fd945600700 1 -- 172.21.15.43:0/3479663101 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd948013650 con 0x55e26978a9f0 2024-05-27T07:29:21.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.344+0000 7fd94d600700 1 -- 172.21.15.43:0/447145951 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7fd9400e38e0 con 0x7fd9501043f0 2024-05-27T07:29:21.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.345+0000 7fd954a00700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9340b2a70 0x7fd9340b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.346+0000 7fd9654d2b80 1 -- 172.21.15.43:0/3479663101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 msgr2=0x55e26981e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:21.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.346+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e26981e750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd948009be0 tx=0x7fd948009ef0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.346+0000 7fd9654d2b80 1 -- 172.21.15.43:0/3479663101 shutdown_connections 2024-05-27T07:29:21.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.346+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 0x55e26982a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.346+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e26981e750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.346+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/3479663101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26969ee40 0x55e26981ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.346+0000 7fd9654d2b80 1 -- 172.21.15.43:0/3479663101 >> 172.21.15.43:0/3479663101 conn(0x55e269738390 msgr2=0x55e269819710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:21.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.346+0000 7fd954a00700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9340b2a70 0x7fd9340b4f80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd948015040 tx=0x7fd9480139d0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:21.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.346+0000 7fd9654d2b80 1 -- 172.21.15.43:0/3479663101 shutdown_connections 2024-05-27T07:29:21.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.346+0000 7fd9654d2b80 1 -- 172.21.15.43:0/3479663101 wait complete. 2024-05-27T07:29:21.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.347+0000 7fd9654d2b80 1 -- start start 2024-05-27T07:29:21.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.347+0000 7fd9654d2b80 1 Processor -- start 2024-05-27T07:29:21.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.348+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26969ee40 0x55e269827d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.348+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e2698250f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.349+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e2698282c0 0x55e2698257b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.349+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e269641fa0 con 0x55e26969ee40 2024-05-27T07:29:21.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.349+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e2696dc760 con 0x55e26978a9f0 2024-05-27T07:29:21.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.349+0000 7fd947400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e2698250f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.349+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e26982ae60 con 0x55e2698282c0 2024-05-27T07:29:21.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.349+0000 7fd9654d2b80 10 client.0 _ll_register_callbacks cb 0x55e2696e5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:21.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.349+0000 7fd947400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e2698250f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:56702/0 (socket says 172.21.15.43:56702) 2024-05-27T07:29:21.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.349+0000 7fd947400700 1 -- 172.21.15.43:0/2683527575 learned_addr learned my addr 172.21.15.43:0/2683527575 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:21.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.349+0000 7fd954a00700 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e2698282c0 0x55e2698257b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.350+0000 7fd94fe00700 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26969ee40 0x55e269827d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.350+0000 7fd947400700 1 -- 172.21.15.43:0/2683527575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e2698282c0 msgr2=0x55e2698257b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:21.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.350+0000 7fd947400700 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e2698282c0 0x55e2698257b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.350+0000 7fd947400700 1 -- 172.21.15.43:0/2683527575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26969ee40 msgr2=0x55e269827d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:21.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.350+0000 7fd947400700 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26969ee40 0x55e269827d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.350+0000 7fd947400700 1 -- 172.21.15.43:0/2683527575 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e26968b8d0 con 0x55e26978a9f0 2024-05-27T07:29:21.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.350+0000 7fd94fe00700 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26969ee40 0x55e269827d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:21.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.351+0000 7fd947400700 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e2698250f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd938013780 tx=0x7fd938013b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71355e25889c23b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:21.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.351+0000 7fd93e000700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:29:21.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.352+0000 7fd93e000700 1 -- 172.21.15.43:0/2683527575 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938020060 con 0x55e26978a9f0 2024-05-27T07:29:21.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.352+0000 7fd93e000700 1 -- 172.21.15.43:0/2683527575 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7fd938004460 con 0x55e26978a9f0 2024-05-27T07:29:21.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.352+0000 7fd9654d2b80 1 -- 172.21.15.43:0/2683527575 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd948004d50 con 0x55e26978a9f0 2024-05-27T07:29:21.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.353+0000 7fd93e000700 1 -- 172.21.15.43:0/2683527575 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) ==== 24+0+0 (secure 0 0 0) 0x7fd938027050 con 0x55e26978a9f0 2024-05-27T07:29:21.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.353+0000 7fd9654d2b80 10 client.37022 fetch_fsmap learned FSMap version 117 2024-05-27T07:29:21.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.353+0000 7fd9654d2b80 1 -- 172.21.15.43:0/2683527575 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=4+}) -- 0x55e26970c060 con 0x55e26978a9f0 2024-05-27T07:29:21.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.353+0000 7fd93e000700 1 -- 172.21.15.43:0/2683527575 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd948004d50 con 0x55e26978a9f0 2024-05-27T07:29:21.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.353+0000 7fd93e000700 1 -- 172.21.15.43:0/2683527575 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 117) ==== 4223+0+0 (secure 0 0 0) 0x7fd9380119c0 con 0x55e26978a9f0 2024-05-27T07:29:21.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.354+0000 7fd9654d2b80 10 client.37022 fetch_fsmap finished waiting for FSMap version 117 2024-05-27T07:29:21.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.354+0000 7fd9654d2b80 10 client.37022 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:29:21.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.354+0000 7fd9654d2b80 20 client.37022 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:21.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.354+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55e269837fa0 0x55e26983a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.354+0000 7fd9654d2b80 4 client.37022 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:21.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.354+0000 7fd9654d2b80 1 -- 172.21.15.43:0/2683527575 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e26974dd90 con 0x55e269837fa0 2024-05-27T07:29:21.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.354+0000 7fd94fe00700 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55e269837fa0 0x55e26983a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.355+0000 7fd93e000700 10 client.37022 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:29:21.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.355+0000 7fd94fe00700 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55e269837fa0 0x55e26983a470 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6a660fb4fb8bdeb4 server_cookie=ab4c74fe23fa8852 in_seq=0 out_seq=0 2024-05-27T07:29:21.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.356+0000 7fd93e000700 1 -- 172.21.15.43:0/2683527575 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e26974dd90 con 0x55e269837fa0 2024-05-27T07:29:21.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.356+0000 7fd93e000700 10 client.37022 handle_command_reply: tid=0 2024-05-27T07:29:21.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd9654d2b80 1 client.37022 shutdown 2024-05-27T07:29:21.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd9654d2b80 1 -- 172.21.15.43:0/2683527575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 msgr2=0x55e2698250f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:21.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e2698250f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd938013780 tx=0x7fd938013b40 comp rx=0 tx=0).stop 2024-05-27T07:29:21.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd9654d2b80 1 -- 172.21.15.43:0/2683527575 shutdown_connections 2024-05-27T07:29:21.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e2698282c0 0x55e2698257b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26969ee40 0x55e269827d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55e269837fa0 0x55e26983a470 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/2683527575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26978a9f0 0x55e2698250f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd93e000700 0 client.37022 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:29:21.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd9654d2b80 1 -- 172.21.15.43:0/2683527575 >> 172.21.15.43:0/2683527575 conn(0x55e269738390 msgr2=0x55e269819260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:21.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.357+0000 7fd9654d2b80 1 -- 172.21.15.43:0/2683527575 shutdown_connections 2024-05-27T07:29:21.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.358+0000 7fd9654d2b80 1 -- 172.21.15.43:0/2683527575 wait complete. 2024-05-27T07:29:21.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.358+0000 7fd9654d2b80 20 client.37022 trim_cache size 0 max 16384 2024-05-27T07:29:21.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.386+0000 7fd9654d2b80 1 Processor -- start 2024-05-27T07:29:21.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.386+0000 7fd9654d2b80 1 -- start start 2024-05-27T07:29:21.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.386+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e269833f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.386+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26978a9f0 0x55e269839fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.386+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 0x55e26983a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.386+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd9501b2ad0 con 0x55e2698282c0 2024-05-27T07:29:21.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.386+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9501b0e40 con 0x55e26969ee40 2024-05-27T07:29:21.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.386+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd9501b23d0 con 0x55e26978a9f0 2024-05-27T07:29:21.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.387+0000 7fd94fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e269833f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.387+0000 7fd94fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e269833f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:56710/0 (socket says 172.21.15.43:56710) 2024-05-27T07:29:21.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.387+0000 7fd94fe00700 1 -- 172.21.15.43:0/1894700380 learned_addr learned my addr 172.21.15.43:0/1894700380 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:21.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.387+0000 7fd947400700 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26978a9f0 0x55e269839fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.387+0000 7fd954a00700 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 0x55e26983a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.387+0000 7fd94fe00700 1 -- 172.21.15.43:0/1894700380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26978a9f0 msgr2=0x55e269839fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:21.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.387+0000 7fd94fe00700 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26978a9f0 0x55e269839fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.387+0000 7fd94fe00700 1 -- 172.21.15.43:0/1894700380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 msgr2=0x55e26983a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:21.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.388+0000 7fd94fe00700 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 0x55e26983a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.388+0000 7fd94fe00700 1 -- 172.21.15.43:0/1894700380 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e26981a0e0 con 0x55e26969ee40 2024-05-27T07:29:21.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.388+0000 7fd947400700 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26978a9f0 0x55e269839fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:21.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.388+0000 7fd954a00700 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 0x55e26983a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:21.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.388+0000 7fd94fe00700 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e269833f80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd94000fad0 tx=0x7fd9400e5df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=243181064b560628 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:21.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.388+0000 7fd944c00700 1 -- 172.21.15.43:0/1894700380 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9400e5e60 con 0x55e26969ee40 2024-05-27T07:29:21.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.389+0000 7fd944c00700 1 -- 172.21.15.43:0/1894700380 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd9400aa390 con 0x55e26969ee40 2024-05-27T07:29:21.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.389+0000 7fd944c00700 1 -- 172.21.15.43:0/1894700380 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9400aa880 con 0x55e26969ee40 2024-05-27T07:29:21.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.390+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1894700380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 msgr2=0x55e269833f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:21.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.390+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e269833f80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd94000fad0 tx=0x7fd9400e5df0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.390+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1894700380 shutdown_connections 2024-05-27T07:29:21.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.390+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e2698282c0 0x55e26983a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.390+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e269833f80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.390+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/1894700380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e26978a9f0 0x55e269839fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.390+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1894700380 >> 172.21.15.43:0/1894700380 conn(0x55e269738390 msgr2=0x55e2698239e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:21.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.390+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1894700380 shutdown_connections 2024-05-27T07:29:21.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.390+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1894700380 wait complete. 2024-05-27T07:29:21.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.391+0000 7fd9654d2b80 1 -- start start 2024-05-27T07:29:21.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.391+0000 7fd9654d2b80 1 Processor -- start 2024-05-27T07:29:21.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.393+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e26982b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.393+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26978a9f0 0x55e26982b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.393+0000 7fd9654d2b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e2698282c0 0x55e26982bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.393+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd9501b1c40 con 0x55e26978a9f0 2024-05-27T07:29:21.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.393+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9501b1510 con 0x55e26969ee40 2024-05-27T07:29:21.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.393+0000 7fd9654d2b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e26982a6e0 con 0x55e2698282c0 2024-05-27T07:29:21.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.393+0000 7fd9654d2b80 10 client.0 _ll_register_callbacks cb 0x55e2696e5c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:21.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.394+0000 7fd94fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e26982b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.394+0000 7fd94fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e26982b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:56724/0 (socket says 172.21.15.43:56724) 2024-05-27T07:29:21.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.394+0000 7fd94fe00700 1 -- 172.21.15.43:0/1201130475 learned_addr learned my addr 172.21.15.43:0/1201130475 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:21.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.394+0000 7fd954a00700 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e2698282c0 0x55e26982bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.394+0000 7fd947400700 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26978a9f0 0x55e26982b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.395+0000 7fd94fe00700 1 -- 172.21.15.43:0/1201130475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e2698282c0 msgr2=0x55e26982bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:21.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.395+0000 7fd94fe00700 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e2698282c0 0x55e26982bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.395+0000 7fd94fe00700 1 -- 172.21.15.43:0/1201130475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26978a9f0 msgr2=0x55e26982b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:21.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.395+0000 7fd94fe00700 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26978a9f0 0x55e26982b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.395+0000 7fd94fe00700 1 -- 172.21.15.43:0/1201130475 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2698a0c00 con 0x55e26969ee40 2024-05-27T07:29:21.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.395+0000 7fd94fe00700 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e26982b130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd9400aa810 tx=0x7fd940009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80a387dc5dcd2916 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:21.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.395+0000 7fd93e000700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:29:21.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.395+0000 7fd93e000700 1 -- 172.21.15.43:0/1201130475 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940005680 con 0x55e26969ee40 2024-05-27T07:29:21.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.395+0000 7fd93e000700 1 -- 172.21.15.43:0/1201130475 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7fd9400e0ee0 con 0x55e26969ee40 2024-05-27T07:29:21.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.395+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1201130475 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9400aa390 con 0x55e26969ee40 2024-05-27T07:29:21.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.396+0000 7fd93e000700 1 -- 172.21.15.43:0/1201130475 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) ==== 24+0+0 (secure 0 0 0) 0x7fd9400aaf80 con 0x55e26969ee40 2024-05-27T07:29:21.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.396+0000 7fd9654d2b80 10 client.37028 fetch_fsmap learned FSMap version 117 2024-05-27T07:29:21.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.396+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1201130475 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=4+}) -- 0x55e2698a1e60 con 0x55e26969ee40 2024-05-27T07:29:21.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.396+0000 7fd93e000700 1 -- 172.21.15.43:0/1201130475 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd9400c7020 con 0x55e26969ee40 2024-05-27T07:29:21.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.397+0000 7fd93e000700 1 -- 172.21.15.43:0/1201130475 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 117) ==== 4223+0+0 (secure 0 0 0) 0x7fd9400ca020 con 0x55e26969ee40 2024-05-27T07:29:21.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.397+0000 7fd9654d2b80 10 client.37028 fetch_fsmap finished waiting for FSMap version 117 2024-05-27T07:29:21.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.397+0000 7fd9654d2b80 10 client.37028 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:29:21.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.397+0000 7fd9654d2b80 20 client.37028 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:21.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.397+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55e2698a2460 0x55e2698a4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:21.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.397+0000 7fd9654d2b80 4 client.37028 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:21.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.397+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1201130475 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fd9400aa390 con 0x55e2698a2460 2024-05-27T07:29:21.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.397+0000 7fd947400700 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55e2698a2460 0x55e2698a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:21.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.399+0000 7fd947400700 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55e2698a2460 0x55e2698a4880 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=11995bd89054a9c9 server_cookie=b849ad0f78ddbf34 in_seq=0 out_seq=0 2024-05-27T07:29:21.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.399+0000 7fd93e000700 10 client.37028 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:29:21.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.400+0000 7fd93e000700 1 -- 172.21.15.43:0/1201130475 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fd9400aa390 con 0x55e2698a2460 2024-05-27T07:29:21.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.400+0000 7fd93e000700 10 client.37028 handle_command_reply: tid=0 2024-05-27T07:29:21.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.400+0000 7fd9654d2b80 1 client.37028 shutdown 2024-05-27T07:29:21.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.400+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1201130475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 msgr2=0x55e26982b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:21.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.400+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e26982b130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd9400aa810 tx=0x7fd940009c00 comp rx=0 tx=0).stop 2024-05-27T07:29:21.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.401+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1201130475 shutdown_connections 2024-05-27T07:29:21.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.401+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e2698282c0 0x55e26982bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.401+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55e2698a2460 0x55e2698a4880 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.401+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e26969ee40 0x55e26982b130 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.401+0000 7fd9654d2b80 1 --2- 172.21.15.43:0/1201130475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e26978a9f0 0x55e26982b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.401+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1201130475 >> 172.21.15.43:0/1201130475 conn(0x55e269738390 msgr2=0x55e26981e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:21.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.401+0000 7fd93e000700 0 client.37028 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:29:21.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.401+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1201130475 shutdown_connections 2024-05-27T07:29:21.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.401+0000 7fd9654d2b80 1 -- 172.21.15.43:0/1201130475 wait complete. 2024-05-27T07:29:21.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.402+0000 7fd9654d2b80 20 client.37028 trim_cache size 0 max 16384 2024-05-27T07:29:21.393 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:29:21.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.403+0000 7fd955400700 1 -- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9340b2a70 msgr2=0x7fd9340b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:21.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.403+0000 7fd955400700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9340b2a70 0x7fd9340b4f80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd948015040 tx=0x7fd9480139d0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.403+0000 7fd955400700 1 -- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9501043f0 msgr2=0x7fd9501a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:21.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.403+0000 7fd955400700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9501043f0 0x7fd9501a0840 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fd940009fd0 tx=0x7fd940009490 comp rx=0 tx=0).stop 2024-05-27T07:29:21.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.403+0000 7fd955400700 1 -- 172.21.15.43:0/447145951 shutdown_connections 2024-05-27T07:29:21.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.403+0000 7fd955400700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd9340b2a70 0x7fd9340b4f80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.403+0000 7fd955400700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd950109670 0x7fd95019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.403+0000 7fd955400700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd9501043f0 0x7fd9501a0840 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.403+0000 7fd955400700 1 --2- 172.21.15.43:0/447145951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd950108d20 0x7fd9501a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:21.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.403+0000 7fd955400700 1 -- 172.21.15.43:0/447145951 >> 172.21.15.43:0/447145951 conn(0x7fd950079d00 msgr2=0x7fd950102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:21.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.404+0000 7fd955400700 1 -- 172.21.15.43:0/447145951 shutdown_connections 2024-05-27T07:29:21.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:21.404+0000 7fd955400700 1 -- 172.21.15.43:0/447145951 wait complete. 2024-05-27T07:29:21.581 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:21 smithi043 podman[118735]: 2024-05-27 07:29:21.199218184 +0000 UTC m=+0.336393930 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:21.582 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:21 smithi043 podman[118735]: 2024-05-27 07:29:21.329744254 +0000 UTC m=+0.466919977 container create 9ba9d67a837991cb8359b285955f36e728e603a5aa0f99a4fa20f45431744437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=) 2024-05-27T07:29:21.770 INFO:tasks.workunit.client.0.smithi043.stdout: 10 9224 8.05 MB/sec warmup 82 sec latency 90.529 ms 2024-05-27T07:29:21.952 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:21 smithi043 podman[118735]: 2024-05-27 07:29:21.590666082 +0000 UTC m=+0.727841823 container init 9ba9d67a837991cb8359b285955f36e728e603a5aa0f99a4fa20f45431744437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-05-27T07:29:21.952 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:21 smithi043 podman[118735]: 2024-05-27 07:29:21.602257919 +0000 UTC m=+0.739433643 container start 9ba9d67a837991cb8359b285955f36e728e603a5aa0f99a4fa20f45431744437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-27T07:29:21.952 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:21 smithi043 bash[118735]: 9ba9d67a837991cb8359b285955f36e728e603a5aa0f99a4fa20f45431744437 2024-05-27T07:29:21.952 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:21 smithi043 systemd[1]: Started Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:29:21.952 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: starting mds.a at 2024-05-27T07:29:22.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:29:22.771 INFO:tasks.workunit.client.0.smithi043.stdout: 10 9354 8.05 MB/sec warmup 83 sec latency 98.128 ms 2024-05-27T07:29:22.817 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8d23400700 time 2024-05-27T07:29:22.642233+0000 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d38f4eba6] 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d38f4edd1] 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559179cfb23d] 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559179ce6971] 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 5: (SnapRealm::check_cache() const+0x13e) [0x559179ce4fa0] 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559179ce5651] 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559179c7a347] 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55917997687e] 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55917997be94] 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559179986969] 2024-05-27T07:29:22.818 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 11: (MDLog::_replay_thread()+0x1f38) [0x559179d1f48c] 2024-05-27T07:29:22.819 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559179d289cb] 2024-05-27T07:29:22.819 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 13: (Thread::entry_wrapper()+0x43) [0x7f8d38f1d72d] 2024-05-27T07:29:22.819 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8d38f1d749] 2024-05-27T07:29:22.819 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8d37ac11ca] 2024-05-27T07:29:22.819 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 16: clone() 2024-05-27T07:29:22.819 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.643+0000 7f8d23400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8d23400700 time 2024-05-27T07:29:22.642233+0000 2024-05-27T07:29:22.819 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-mon[31531]: pgmap v518: 129 pgs: 129 active+clean; 463 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 2.7 MiB/s wr, 205 op/s 2024-05-27T07:29:22.820 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:22.820 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.820 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:22.820 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d38f4eba6] 2024-05-27T07:29:22.820 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d38f4edd1] 2024-05-27T07:29:22.820 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559179cfb23d] 2024-05-27T07:29:22.820 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559179ce6971] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 5: (SnapRealm::check_cache() const+0x13e) [0x559179ce4fa0] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559179ce5651] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559179c7a347] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55917997687e] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55917997be94] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559179986969] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 11: (MDLog::_replay_thread()+0x1f38) [0x559179d1f48c] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559179d289cb] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 13: (Thread::entry_wrapper()+0x43) [0x7f8d38f1d72d] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8d38f1d749] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8d37ac11ca] 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 16: clone() 2024-05-27T07:29:22.821 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: *** Caught signal (Aborted) ** 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: in thread 7f8d23400700 thread_name:md_log_replay 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d37acbd20] 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2: gsignal() 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 3: abort() 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d38f4eced] 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d38f4edd1] 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559179cfb23d] 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559179ce6971] 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 8: (SnapRealm::check_cache() const+0x13e) [0x559179ce4fa0] 2024-05-27T07:29:22.822 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559179ce5651] 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559179c7a347] 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55917997687e] 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55917997be94] 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559179986969] 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 14: (MDLog::_replay_thread()+0x1f38) [0x559179d1f48c] 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559179d289cb] 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 16: (Thread::entry_wrapper()+0x43) [0x7f8d38f1d72d] 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8d38f1d749] 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8d37ac11ca] 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 19: clone() 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.648+0000 7f8d23400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: in thread 7f8d23400700 thread_name:md_log_replay 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.823 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d37acbd20] 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2: gsignal() 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 3: abort() 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d38f4eced] 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d38f4edd1] 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559179cfb23d] 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559179ce6971] 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 8: (SnapRealm::check_cache() const+0x13e) [0x559179ce4fa0] 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559179ce5651] 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559179c7a347] 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55917997687e] 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55917997be94] 2024-05-27T07:29:22.824 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559179986969] 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 14: (MDLog::_replay_thread()+0x1f38) [0x559179d1f48c] 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559179d289cb] 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 16: (Thread::entry_wrapper()+0x43) [0x7f8d38f1d72d] 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8d38f1d749] 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8d37ac11ca] 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 19: clone() 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: -1> 2024-05-27T07:29:22.643+0000 7f8d23400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8d23400700 time 2024-05-27T07:29:22.642233+0000 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d38f4eba6] 2024-05-27T07:29:22.825 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d38f4edd1] 2024-05-27T07:29:22.826 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559179cfb23d] 2024-05-27T07:29:22.826 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559179ce6971] 2024-05-27T07:29:22.826 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 5: (SnapRealm::check_cache() const+0x13e) [0x559179ce4fa0] 2024-05-27T07:29:22.826 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559179ce5651] 2024-05-27T07:29:22.826 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559179c7a347] 2024-05-27T07:29:22.826 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55917997687e] 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55917997be94] 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559179986969] 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 11: (MDLog::_replay_thread()+0x1f38) [0x559179d1f48c] 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559179d289cb] 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 13: (Thread::entry_wrapper()+0x43) [0x7f8d38f1d72d] 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8d38f1d749] 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8d37ac11ca] 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 16: clone() 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 0> 2024-05-27T07:29:22.648+0000 7f8d23400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: in thread 7f8d23400700 thread_name:md_log_replay 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.827 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d37acbd20] 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2: gsignal() 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 3: abort() 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d38f4eced] 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d38f4edd1] 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559179cfb23d] 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559179ce6971] 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 8: (SnapRealm::check_cache() const+0x13e) [0x559179ce4fa0] 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559179ce5651] 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559179c7a347] 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55917997687e] 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55917997be94] 2024-05-27T07:29:22.828 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559179986969] 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 14: (MDLog::_replay_thread()+0x1f38) [0x559179d1f48c] 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559179d289cb] 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 16: (Thread::entry_wrapper()+0x43) [0x7f8d38f1d72d] 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8d38f1d749] 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8d37ac11ca] 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 19: clone() 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: -1> 2024-05-27T07:29:22.643+0000 7f8d23400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8d23400700 time 2024-05-27T07:29:22.642233+0000 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:22.829 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d38f4eba6] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d38f4edd1] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559179cfb23d] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 4: (SnapRealm::build_snap_set() const+0x323) [0x559179ce6971] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 5: (SnapRealm::check_cache() const+0x13e) [0x559179ce4fa0] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559179ce5651] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559179c7a347] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55917997687e] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55917997be94] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x559179986969] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 11: (MDLog::_replay_thread()+0x1f38) [0x559179d1f48c] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 12: (MDLog::ReplayThread::entry()+0x11) [0x559179d289cb] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 13: (Thread::entry_wrapper()+0x43) [0x7f8d38f1d72d] 2024-05-27T07:29:22.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 14: (Thread::_entry_func(void*)+0xd) [0x7f8d38f1d749] 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8d37ac11ca] 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 16: clone() 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 0> 2024-05-27T07:29:22.648+0000 7f8d23400700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: in thread 7f8d23400700 thread_name:md_log_replay 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d37acbd20] 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2: gsignal() 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 3: abort() 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d38f4eced] 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d38f4edd1] 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x559179cfb23d] 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 7: (SnapRealm::build_snap_set() const+0x323) [0x559179ce6971] 2024-05-27T07:29:22.831 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 8: (SnapRealm::check_cache() const+0x13e) [0x559179ce4fa0] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x559179ce5651] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x559179c7a347] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55917997687e] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55917997be94] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x559179986969] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 14: (MDLog::_replay_thread()+0x1f38) [0x559179d1f48c] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 15: (MDLog::ReplayThread::entry()+0x11) [0x559179d289cb] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 16: (Thread::entry_wrapper()+0x43) [0x7f8d38f1d72d] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 17: (Thread::_entry_func(void*)+0xd) [0x7f8d38f1d749] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8d37ac11ca] 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 19: clone() 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[118800]: 2024-05-27T07:29:22.832 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:22 smithi043 podman[118872]: 2024-05-27 07:29:22.827272813 +0000 UTC m=+0.067771098 container died 9ba9d67a837991cb8359b285955f36e728e603a5aa0f99a4fa20f45431744437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0) 2024-05-27T07:29:22.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:22 smithi132 ceph-mon[35283]: pgmap v518: 129 pgs: 129 active+clean; 463 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 2.7 MiB/s wr, 205 op/s 2024-05-27T07:29:22.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:22 smithi179 ceph-mon[35888]: pgmap v518: 129 pgs: 129 active+clean; 463 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 2.7 MiB/s wr, 205 op/s 2024-05-27T07:29:23.756 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:23.810 INFO:tasks.workunit.client.0.smithi043.stdout: 10 9532 8.12 MB/sec warmup 84 sec latency 107.994 ms 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:23 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/204955060,v1:172.21.15.43:6835/204955060] up:boot 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:23 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:23 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:23.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:23 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:23.943 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:23 smithi043 podman[118872]: 2024-05-27 07:29:23.562335462 +0000 UTC m=+0.802833755 container remove 9ba9d67a837991cb8359b285955f36e728e603a5aa0f99a4fa20f45431744437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-27T07:29:23.943 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:23 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:23.946 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:23 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/204955060,v1:172.21.15.43:6835/204955060] up:boot 2024-05-27T07:29:23.946 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:23 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:29:23.946 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:23 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:23.947 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:23 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:23.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:23 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/204955060,v1:172.21.15.43:6835/204955060] up:boot 2024-05-27T07:29:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:23 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:29:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:23 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:23.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:23 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:24.771 INFO:tasks.workunit.client.0.smithi043.stdout: 10 9611 8.05 MB/sec warmup 85 sec latency 107.160 ms 2024-05-27T07:29:24.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:24 smithi132 ceph-mon[35283]: pgmap v519: 129 pgs: 129 active+clean; 463 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.3 MiB/s wr, 172 op/s 2024-05-27T07:29:24.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:24 smithi043 ceph-mon[31531]: pgmap v519: 129 pgs: 129 active+clean; 463 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.3 MiB/s wr, 172 op/s 2024-05-27T07:29:24.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:24 smithi179 ceph-mon[35888]: pgmap v519: 129 pgs: 129 active+clean; 463 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.3 MiB/s wr, 172 op/s 2024-05-27T07:29:25.202 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:24 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:29:25.202 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:24 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.747s CPU time. 2024-05-27T07:29:25.771 INFO:tasks.workunit.client.0.smithi043.stdout: 10 9713 8.02 MB/sec warmup 86 sec latency 98.457 ms 2024-05-27T07:29:26.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.541+0000 7efc4aa00700 1 -- 172.21.15.43:0/3911354882 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c017400 con 0x7efc4c1107d0 2024-05-27T07:29:26.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.542+0000 7efc51a00700 1 -- 172.21.15.43:0/3911354882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c1107d0 msgr2=0x7efc4c110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.542+0000 7efc51a00700 1 --2- 172.21.15.43:0/3911354882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c1107d0 0x7efc4c110be0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7efc3c0098f0 tx=0x7efc3c009c00 comp rx=0 tx=0).stop 2024-05-27T07:29:26.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.542+0000 7efc51a00700 1 -- 172.21.15.43:0/3911354882 shutdown_connections 2024-05-27T07:29:26.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.542+0000 7efc51a00700 1 --2- 172.21.15.43:0/3911354882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc4c10b140 0x7efc4c117670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.542+0000 7efc51a00700 1 --2- 172.21.15.43:0/3911354882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c1107d0 0x7efc4c110be0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.542+0000 7efc51a00700 1 --2- 172.21.15.43:0/3911354882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc4c10a7f0 0x7efc4c10ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.542+0000 7efc51a00700 1 -- 172.21.15.43:0/3911354882 >> 172.21.15.43:0/3911354882 conn(0x7efc4c076af0 msgr2=0x7efc4c076f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.543+0000 7efc51a00700 1 -- 172.21.15.43:0/3911354882 shutdown_connections 2024-05-27T07:29:26.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.543+0000 7efc51a00700 1 -- 172.21.15.43:0/3911354882 wait complete. 2024-05-27T07:29:26.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.547+0000 7efc51a00700 1 Processor -- start 2024-05-27T07:29:26.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.547+0000 7efc51a00700 1 -- start start 2024-05-27T07:29:26.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.548+0000 7efc51a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc4c10a7f0 0x7efc4c1a1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.548+0000 7efc51a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc4c10b140 0x7efc4c1a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.548+0000 7efc51a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c1107d0 0x7efc4c1a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.549+0000 7efc51a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c11a840 con 0x7efc4c1107d0 2024-05-27T07:29:26.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.549+0000 7efc51a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c11a6c0 con 0x7efc4c10b140 2024-05-27T07:29:26.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.549+0000 7efc51a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c11a9c0 con 0x7efc4c10a7f0 2024-05-27T07:29:26.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.549+0000 7efc4b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc4c10b140 0x7efc4c1a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.549+0000 7efc4b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc4c10b140 0x7efc4c1a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52110/0 (socket says 172.21.15.43:52110) 2024-05-27T07:29:26.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.549+0000 7efc4b400700 1 -- 172.21.15.43:0/3731921303 learned_addr learned my addr 172.21.15.43:0/3731921303 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:26.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.549+0000 7efc4be00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc4c10a7f0 0x7efc4c1a1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.549+0000 7efc51000700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c1107d0 0x7efc4c1a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.550+0000 7efc4be00700 1 -- 172.21.15.43:0/3731921303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc4c10b140 msgr2=0x7efc4c1a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.550+0000 7efc4be00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc4c10b140 0x7efc4c1a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.550+0000 7efc4be00700 1 -- 172.21.15.43:0/3731921303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c1107d0 msgr2=0x7efc4c1a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.550+0000 7efc4be00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c1107d0 0x7efc4c1a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.550+0000 7efc4be00700 1 -- 172.21.15.43:0/3731921303 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4c1a60d0 con 0x7efc4c10a7f0 2024-05-27T07:29:26.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.551+0000 7efc4be00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc4c10a7f0 0x7efc4c1a1200 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efc3c000c00 tx=0x7efc3c00fa40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.554+0000 7efc48c00700 1 -- 172.21.15.43:0/3731921303 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c00b6e0 con 0x7efc4c10a7f0 2024-05-27T07:29:26.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.555+0000 7efc48c00700 1 -- 172.21.15.43:0/3731921303 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc3c005040 con 0x7efc4c10a7f0 2024-05-27T07:29:26.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.555+0000 7efc51a00700 1 -- 172.21.15.43:0/3731921303 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc4c1a6360 con 0x7efc4c10a7f0 2024-05-27T07:29:26.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.555+0000 7efc48c00700 1 -- 172.21.15.43:0/3731921303 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c003a40 con 0x7efc4c10a7f0 2024-05-27T07:29:26.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.555+0000 7efc51a00700 1 -- 172.21.15.43:0/3731921303 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc4c1a6810 con 0x7efc4c10a7f0 2024-05-27T07:29:26.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.561+0000 7efc61acbb80 1 Processor -- start 2024-05-27T07:29:26.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.561+0000 7efc61acbb80 1 -- start start 2024-05-27T07:29:26.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.562+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.562+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 0x562e694daad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.562+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694dd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.562+0000 7efc61acbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562e693fee40 con 0x562e69351e40 2024-05-27T07:29:26.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.562+0000 7efc61acbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562e693ee9e0 con 0x562e694d0820 2024-05-27T07:29:26.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.562+0000 7efc61acbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562e693f2a20 con 0x562e694d0410 2024-05-27T07:29:26.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.562+0000 7efc51000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.562+0000 7efc51000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36236/0 (socket says 172.21.15.43:36236) 2024-05-27T07:29:26.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.562+0000 7efc51000700 1 -- 172.21.15.43:0/2410734512 learned_addr learned my addr 172.21.15.43:0/2410734512 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:26.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.563+0000 7efc4be00700 1 --2- 172.21.15.43:0/2410734512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 0x562e694daad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.563+0000 7efc51000700 1 -- 172.21.15.43:0/2410734512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 msgr2=0x562e694daad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.563+0000 7efc51000700 1 --2- 172.21.15.43:0/2410734512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 0x562e694daad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.563+0000 7efc51000700 1 -- 172.21.15.43:0/2410734512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 msgr2=0x562e694dd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.563+0000 7efc4b400700 1 --2- 172.21.15.43:0/2410734512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694dd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.563+0000 7efc51000700 1 --2- 172.21.15.43:0/2410734512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694dd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.563+0000 7efc51000700 1 -- 172.21.15.43:0/2410734512 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e693b7420 con 0x562e69351e40 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.563+0000 7efc4b400700 1 --2- 172.21.15.43:0/2410734512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694dd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.564+0000 7efc51000700 1 --2- 172.21.15.43:0/2410734512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694cfed0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7efc44009440 tx=0x7efc44009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920eb163bd07d445 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.564+0000 7efc41600700 1 -- 172.21.15.43:0/2410734512 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44013810 con 0x562e69351e40 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.564+0000 7efc41600700 1 -- 172.21.15.43:0/2410734512 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc44013e40 con 0x562e69351e40 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.564+0000 7efc41600700 1 -- 172.21.15.43:0/2410734512 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44015de0 con 0x562e69351e40 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.565+0000 7efc61acbb80 1 -- 172.21.15.43:0/2410734512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 msgr2=0x562e694cfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.565+0000 7efc61acbb80 1 --2- 172.21.15.43:0/2410734512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694cfed0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7efc44009440 tx=0x7efc44009920 comp rx=0 tx=0).stop 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.565+0000 7efc61acbb80 1 -- 172.21.15.43:0/2410734512 shutdown_connections 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.565+0000 7efc61acbb80 1 --2- 172.21.15.43:0/2410734512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694dd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.565+0000 7efc61acbb80 1 --2- 172.21.15.43:0/2410734512 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694cfed0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.565+0000 7efc61acbb80 1 --2- 172.21.15.43:0/2410734512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 0x562e694daad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.565+0000 7efc61acbb80 1 -- 172.21.15.43:0/2410734512 >> 172.21.15.43:0/2410734512 conn(0x562e693b1fc0 msgr2=0x562e694cbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.566+0000 7efc61acbb80 1 -- 172.21.15.43:0/2410734512 shutdown_connections 2024-05-27T07:29:26.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.566+0000 7efc61acbb80 1 -- 172.21.15.43:0/2410734512 wait complete. 2024-05-27T07:29:26.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.567+0000 7efc61acbb80 1 -- start start 2024-05-27T07:29:26.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.567+0000 7efc61acbb80 1 Processor -- start 2024-05-27T07:29:26.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.569+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694da650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.569+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e694d0410 0x562e694dab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.569+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694db0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.569+0000 7efc61acbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562e692f4fa0 con 0x562e694d0410 2024-05-27T07:29:26.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.569+0000 7efc61acbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562e6938f760 con 0x562e694d0820 2024-05-27T07:29:26.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.569+0000 7efc61acbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562e694dd990 con 0x562e69351e40 2024-05-27T07:29:26.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.569+0000 7efc61acbb80 10 client.0 _ll_register_callbacks cb 0x562e69398c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:26.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.570+0000 7efc4b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694db0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.570+0000 7efc4b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694db0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52124/0 (socket says 172.21.15.43:52124) 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.570+0000 7efc4b400700 1 -- 172.21.15.43:0/2861234987 learned_addr learned my addr 172.21.15.43:0/2861234987 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.570+0000 7efc51000700 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694da650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.570+0000 7efc51000700 1 -- 172.21.15.43:0/2861234987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 msgr2=0x562e694db0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.570+0000 7efc51000700 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694db0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.571+0000 7efc51000700 1 -- 172.21.15.43:0/2861234987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e694d0410 msgr2=0x562e694dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.571+0000 7efc51000700 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e694d0410 0x562e694dab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.571+0000 7efc51000700 1 -- 172.21.15.43:0/2861234987 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e6933e8d0 con 0x562e69351e40 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.572+0000 7efc51000700 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694da650 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc44000f80 tx=0x7efc44013840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8f4c4f8b7902c7b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.572+0000 7efc36000700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.572+0000 7efc36000700 1 -- 172.21.15.43:0/2861234987 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44015640 con 0x562e69351e40 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.572+0000 7efc36000700 1 -- 172.21.15.43:0/2861234987 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7efc440157e0 con 0x562e69351e40 2024-05-27T07:29:26.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.573+0000 7efc61acbb80 1 -- 172.21.15.43:0/2861234987 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562e69400d90 con 0x562e69351e40 2024-05-27T07:29:26.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.573+0000 7efc36000700 1 -- 172.21.15.43:0/2861234987 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7efc4402c050 con 0x562e69351e40 2024-05-27T07:29:26.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.573+0000 7efc61acbb80 10 client.27148 fetch_fsmap learned FSMap version 119 2024-05-27T07:29:26.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.573+0000 7efc61acbb80 1 -- 172.21.15.43:0/2861234987 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x562e693bf060 con 0x562e69351e40 2024-05-27T07:29:26.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.574+0000 7efc36000700 1 -- 172.21.15.43:0/2861234987 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x562e69400d90 con 0x562e69351e40 2024-05-27T07:29:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.575+0000 7efc36000700 1 -- 172.21.15.43:0/2861234987 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 119) ==== 4816+0+0 (secure 0 0 0) 0x7efc440055f0 con 0x562e69351e40 2024-05-27T07:29:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.575+0000 7efc61acbb80 10 client.27148 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:29:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.575+0000 7efc61acbb80 10 client.27148 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:29:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.575+0000 7efc61acbb80 20 client.27148 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:26.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.575+0000 7efc61acbb80 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x562e694eac30 0x562e694ed0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.575+0000 7efc61acbb80 4 client.27148 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:26.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.575+0000 7efc61acbb80 1 -- 172.21.15.43:0/2861234987 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e693fa570 con 0x562e694eac30 2024-05-27T07:29:26.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.576+0000 7efc48c00700 1 -- 172.21.15.43:0/3731921303 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7efc3c0054d0 con 0x7efc4c10a7f0 2024-05-27T07:29:26.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.577+0000 7efc4be00700 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x562e694eac30 0x562e694ed0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.581+0000 7efc36000700 10 client.27148 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:29:26.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.581+0000 7efc48c00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc300b2a50 0x7efc300b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.581+0000 7efc4be00700 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x562e694eac30 0x562e694ed0c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6144afa643be6400 server_cookie=95ce85147bce5aa2 in_seq=0 out_seq=0 2024-05-27T07:29:26.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.582+0000 7efc48c00700 1 -- 172.21.15.43:0/3731921303 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7efc3c0e5d50 con 0x7efc4c10a7f0 2024-05-27T07:29:26.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.582+0000 7efc4b400700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc300b2a50 0x7efc300b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.583+0000 7efc36000700 1 -- 172.21.15.43:0/2861234987 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562e693fa570 con 0x562e694eac30 2024-05-27T07:29:26.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.583+0000 7efc36000700 10 client.27148 handle_command_reply: tid=0 2024-05-27T07:29:26.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.583+0000 7efc61acbb80 1 client.27148 shutdown 2024-05-27T07:29:26.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.584+0000 7efc61acbb80 1 -- 172.21.15.43:0/2861234987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 msgr2=0x562e694da650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.584+0000 7efc61acbb80 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694da650 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc44000f80 tx=0x7efc44013840 comp rx=0 tx=0).stop 2024-05-27T07:29:26.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.584+0000 7efc4b400700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc300b2a50 0x7efc300b4f60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efc38009f20 tx=0x7efc38009310 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.584+0000 7efc61acbb80 1 -- 172.21.15.43:0/2861234987 shutdown_connections 2024-05-27T07:29:26.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.584+0000 7efc61acbb80 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x562e694eac30 0x562e694ed0c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.584+0000 7efc61acbb80 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694db0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.584+0000 7efc61acbb80 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694da650 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.584+0000 7efc61acbb80 1 --2- 172.21.15.43:0/2861234987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e694d0410 0x562e694dab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.584+0000 7efc36000700 0 client.27148 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:29:26.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.584+0000 7efc61acbb80 1 -- 172.21.15.43:0/2861234987 >> 172.21.15.43:0/2861234987 conn(0x562e693b1fc0 msgr2=0x562e694caff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.585+0000 7efc61acbb80 1 -- 172.21.15.43:0/2861234987 shutdown_connections 2024-05-27T07:29:26.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.585+0000 7efc61acbb80 1 -- 172.21.15.43:0/2861234987 wait complete. 2024-05-27T07:29:26.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.585+0000 7efc61acbb80 20 client.27148 trim_cache size 0 max 16384 2024-05-27T07:29:26.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.610+0000 7efc61acbb80 1 Processor -- start 2024-05-27T07:29:26.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.610+0000 7efc61acbb80 1 -- start start 2024-05-27T07:29:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.611+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694d5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.611+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 0x562e694d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.611+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.611+0000 7efc61acbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c1bff80 con 0x562e69351e40 2024-05-27T07:29:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.611+0000 7efc61acbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c0747f0 con 0x562e694d0820 2024-05-27T07:29:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.611+0000 7efc61acbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c1bf8b0 con 0x562e694d0410 2024-05-27T07:29:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.612+0000 7efc4be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694d5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.612+0000 7efc4be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694d5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36256/0 (socket says 172.21.15.43:36256) 2024-05-27T07:29:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.612+0000 7efc4be00700 1 -- 172.21.15.43:0/4114961282 learned_addr learned my addr 172.21.15.43:0/4114961282 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.613+0000 7efc51000700 1 --2- 172.21.15.43:0/4114961282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.613+0000 7efc4b400700 1 --2- 172.21.15.43:0/4114961282 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 0x562e694d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.613+0000 7efc4be00700 1 -- 172.21.15.43:0/4114961282 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 msgr2=0x562e694d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.613+0000 7efc4be00700 1 --2- 172.21.15.43:0/4114961282 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 0x562e694d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.613+0000 7efc4be00700 1 -- 172.21.15.43:0/4114961282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 msgr2=0x562e694f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.613+0000 7efc4be00700 1 --2- 172.21.15.43:0/4114961282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.613+0000 7efc4be00700 1 -- 172.21.15.43:0/4114961282 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e694c61d0 con 0x562e69351e40 2024-05-27T07:29:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.614+0000 7efc4b400700 1 --2- 172.21.15.43:0/4114961282 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 0x562e694d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:26.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.615+0000 7efc4be00700 1 --2- 172.21.15.43:0/4114961282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694d5e70 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7efc3c0ac990 tx=0x7efc3c0e7340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7df487308b6b24 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.615+0000 7efc40c00700 1 -- 172.21.15.43:0/4114961282 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0aca60 con 0x562e69351e40 2024-05-27T07:29:26.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.615+0000 7efc40c00700 1 -- 172.21.15.43:0/4114961282 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc3c0b53a0 con 0x562e69351e40 2024-05-27T07:29:26.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.615+0000 7efc40c00700 1 -- 172.21.15.43:0/4114961282 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0a8420 con 0x562e69351e40 2024-05-27T07:29:26.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.617+0000 7efc61acbb80 1 -- 172.21.15.43:0/4114961282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 msgr2=0x562e694d5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.617+0000 7efc61acbb80 1 --2- 172.21.15.43:0/4114961282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694d5e70 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7efc3c0ac990 tx=0x7efc3c0e7340 comp rx=0 tx=0).stop 2024-05-27T07:29:26.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.617+0000 7efc61acbb80 1 -- 172.21.15.43:0/4114961282 shutdown_connections 2024-05-27T07:29:26.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.617+0000 7efc61acbb80 1 --2- 172.21.15.43:0/4114961282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0820 0x562e694f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.617+0000 7efc61acbb80 1 --2- 172.21.15.43:0/4114961282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e69351e40 0x562e694d5e70 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.617+0000 7efc61acbb80 1 --2- 172.21.15.43:0/4114961282 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e694d0410 0x562e694d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.617+0000 7efc61acbb80 1 -- 172.21.15.43:0/4114961282 >> 172.21.15.43:0/4114961282 conn(0x562e693b1fc0 msgr2=0x562e694d8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.617+0000 7efc61acbb80 1 -- 172.21.15.43:0/4114961282 shutdown_connections 2024-05-27T07:29:26.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.618+0000 7efc61acbb80 1 -- 172.21.15.43:0/4114961282 wait complete. 2024-05-27T07:29:26.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.618+0000 7efc61acbb80 1 -- start start 2024-05-27T07:29:26.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.619+0000 7efc61acbb80 1 Processor -- start 2024-05-27T07:29:26.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.620+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694e2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.620+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0410 0x562e694e31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.620+0000 7efc61acbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e694d0820 0x562e694d6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.620+0000 7efc61acbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c075620 con 0x562e694d0820 2024-05-27T07:29:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.620+0000 7efc61acbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c074f20 con 0x562e694d0410 2024-05-27T07:29:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.620+0000 7efc61acbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c1bf120 con 0x562e69351e40 2024-05-27T07:29:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.620+0000 7efc61acbb80 10 client.0 _ll_register_callbacks cb 0x562e69398c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.621+0000 7efc4b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0410 0x562e694e31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.621+0000 7efc4be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694e2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.621+0000 7efc4be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694e2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:33266/0 (socket says 172.21.15.43:33266) 2024-05-27T07:29:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.621+0000 7efc4be00700 1 -- 172.21.15.43:0/1881134197 learned_addr learned my addr 172.21.15.43:0/1881134197 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.621+0000 7efc4b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0410 0x562e694e31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52152/0 (socket says 172.21.15.43:52152) 2024-05-27T07:29:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.621+0000 7efc51000700 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e694d0820 0x562e694d6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.622+0000 7efc4b400700 1 -- 172.21.15.43:0/1881134197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 msgr2=0x562e694e2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.622+0000 7efc4b400700 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694e2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.622+0000 7efc4b400700 1 -- 172.21.15.43:0/1881134197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e694d0820 msgr2=0x562e694d6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.622+0000 7efc4b400700 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e694d0820 0x562e694d6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.622+0000 7efc4b400700 1 -- 172.21.15.43:0/1881134197 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e694cc620 con 0x562e694d0410 2024-05-27T07:29:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.624+0000 7efc4be00700 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694e2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.624+0000 7efc4b400700 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0410 0x562e694e31f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efc380096a0 tx=0x7efc38004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62e192fb036c9b3a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.624+0000 7efc36000700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:29:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.625+0000 7efc36000700 1 -- 172.21.15.43:0/1881134197 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3801d410 con 0x562e694d0410 2024-05-27T07:29:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.625+0000 7efc61acbb80 1 -- 172.21.15.43:0/1881134197 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562e69271250 con 0x562e694d0410 2024-05-27T07:29:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.625+0000 7efc36000700 1 -- 172.21.15.43:0/1881134197 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7efc3801d5b0 con 0x562e694d0410 2024-05-27T07:29:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.625+0000 7efc36000700 1 -- 172.21.15.43:0/1881134197 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7efc38035050 con 0x562e694d0410 2024-05-27T07:29:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.625+0000 7efc61acbb80 10 client.37049 fetch_fsmap learned FSMap version 119 2024-05-27T07:29:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.626+0000 7efc61acbb80 1 -- 172.21.15.43:0/1881134197 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x562e694cd350 con 0x562e694d0410 2024-05-27T07:29:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.626+0000 7efc36000700 1 -- 172.21.15.43:0/1881134197 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x562e69271250 con 0x562e694d0410 2024-05-27T07:29:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.626+0000 7efc36000700 1 -- 172.21.15.43:0/1881134197 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 119) ==== 4816+0+0 (secure 0 0 0) 0x7efc38027810 con 0x562e694d0410 2024-05-27T07:29:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.626+0000 7efc61acbb80 10 client.37049 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:29:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.626+0000 7efc61acbb80 10 client.37049 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:29:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.626+0000 7efc61acbb80 20 client.37049 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.627+0000 7efc61acbb80 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x562e695551e0 0x562e69557600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.627+0000 7efc61acbb80 4 client.37049 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.627+0000 7efc61acbb80 1 -- 172.21.15.43:0/1881134197 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e693c2b90 con 0x562e695551e0 2024-05-27T07:29:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.627+0000 7efc4be00700 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x562e695551e0 0x562e69557600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.628+0000 7efc4be00700 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x562e695551e0 0x562e69557600 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1e1da3e7df71799b server_cookie=d2d4ccb1d645aa34 in_seq=0 out_seq=0 2024-05-27T07:29:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.628+0000 7efc36000700 10 client.37049 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:29:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.628+0000 7efc36000700 1 -- 172.21.15.43:0/1881134197 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e693c2b90 con 0x562e695551e0 2024-05-27T07:29:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.628+0000 7efc36000700 10 client.37049 handle_command_reply: tid=0 2024-05-27T07:29:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.628+0000 7efc61acbb80 1 client.37049 shutdown 2024-05-27T07:29:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc61acbb80 1 -- 172.21.15.43:0/1881134197 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0410 msgr2=0x562e694e31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc61acbb80 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0410 0x562e694e31f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efc380096a0 tx=0x7efc38004750 comp rx=0 tx=0).stop 2024-05-27T07:29:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc61acbb80 1 -- 172.21.15.43:0/1881134197 shutdown_connections 2024-05-27T07:29:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc61acbb80 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x562e695551e0 0x562e69557600 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc61acbb80 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562e694d0820 0x562e694d6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc61acbb80 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562e69351e40 0x562e694e2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc61acbb80 1 --2- 172.21.15.43:0/1881134197 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562e694d0410 0x562e694e31f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc61acbb80 1 -- 172.21.15.43:0/1881134197 >> 172.21.15.43:0/1881134197 conn(0x562e693b1fc0 msgr2=0x562e694da430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc36000700 0 client.37049 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:29:26.620 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:29:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc61acbb80 1 -- 172.21.15.43:0/1881134197 shutdown_connections 2024-05-27T07:29:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.629+0000 7efc61acbb80 1 -- 172.21.15.43:0/1881134197 wait complete. 2024-05-27T07:29:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.630+0000 7efc61acbb80 20 client.37049 trim_cache size 0 max 16384 2024-05-27T07:29:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 -- 172.21.15.43:0/3731921303 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc300b2a50 msgr2=0x7efc300b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc300b2a50 0x7efc300b4f60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efc38009f20 tx=0x7efc38009310 comp rx=0 tx=0).stop 2024-05-27T07:29:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 -- 172.21.15.43:0/3731921303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc4c10a7f0 msgr2=0x7efc4c1a1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc4c10a7f0 0x7efc4c1a1200 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efc3c000c00 tx=0x7efc3c00fa40 comp rx=0 tx=0).stop 2024-05-27T07:29:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 -- 172.21.15.43:0/3731921303 shutdown_connections 2024-05-27T07:29:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7efc300b2a50 0x7efc300b4f60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc4c1107d0 0x7efc4c1a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc4c10a7f0 0x7efc4c1a1200 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 --2- 172.21.15.43:0/3731921303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc4c10b140 0x7efc4c1a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 -- 172.21.15.43:0/3731921303 >> 172.21.15.43:0/3731921303 conn(0x7efc4c076af0 msgr2=0x7efc4c1093d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.631+0000 7efc51a00700 1 -- 172.21.15.43:0/3731921303 shutdown_connections 2024-05-27T07:29:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:26.632+0000 7efc51a00700 1 -- 172.21.15.43:0/3731921303 wait complete. 2024-05-27T07:29:26.771 INFO:tasks.workunit.client.0.smithi043.stdout: 10 9824 7.99 MB/sec warmup 87 sec latency 97.385 ms 2024-05-27T07:29:26.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:26 smithi132 ceph-mon[35283]: pgmap v520: 129 pgs: 129 active+clean; 463 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.3 MiB/s wr, 172 op/s 2024-05-27T07:29:26.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:26 smithi043 ceph-mon[31531]: pgmap v520: 129 pgs: 129 active+clean; 463 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.3 MiB/s wr, 172 op/s 2024-05-27T07:29:26.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:26 smithi179 ceph-mon[35888]: pgmap v520: 129 pgs: 129 active+clean; 463 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.3 MiB/s wr, 172 op/s 2024-05-27T07:29:27.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:29:27.771 INFO:tasks.workunit.client.0.smithi043.stdout: 10 9980 8.05 MB/sec warmup 88 sec latency 85.341 ms 2024-05-27T07:29:28.285 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:28.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:28 smithi043 ceph-mon[31531]: pgmap v521: 129 pgs: 129 active+clean; 474 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 243 op/s 2024-05-27T07:29:28.771 INFO:tasks.workunit.client.0.smithi043.stdout: 10 10109 8.08 MB/sec warmup 89 sec latency 88.804 ms 2024-05-27T07:29:28.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:28 smithi132 ceph-mon[35283]: pgmap v521: 129 pgs: 129 active+clean; 474 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 243 op/s 2024-05-27T07:29:28.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:28 smithi179 ceph-mon[35888]: pgmap v521: 129 pgs: 129 active+clean; 474 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 243 op/s 2024-05-27T07:29:29.771 INFO:tasks.workunit.client.0.smithi043.stdout: 10 10224 8.03 MB/sec warmup 90 sec latency 87.501 ms 2024-05-27T07:29:30.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.519+0000 7fac6aa00700 1 -- 172.21.15.43:0/2142384506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac64107de0 msgr2=0x7fac641081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.519+0000 7fac6aa00700 1 --2- 172.21.15.43:0/2142384506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac64107de0 0x7fac641081f0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fac500099c0 tx=0x7fac50009cd0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.519+0000 7fac6aa00700 1 -- 172.21.15.43:0/2142384506 shutdown_connections 2024-05-27T07:29:30.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.519+0000 7fac6aa00700 1 --2- 172.21.15.43:0/2142384506 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac64108730 0x7fac64110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.519+0000 7fac6aa00700 1 --2- 172.21.15.43:0/2142384506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac641034b0 0x7fac641038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.519+0000 7fac6aa00700 1 --2- 172.21.15.43:0/2142384506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac64107de0 0x7fac641081f0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.519+0000 7fac6aa00700 1 -- 172.21.15.43:0/2142384506 >> 172.21.15.43:0/2142384506 conn(0x7fac640fedb0 msgr2=0x7fac641011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:30.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.520+0000 7fac6aa00700 1 -- 172.21.15.43:0/2142384506 shutdown_connections 2024-05-27T07:29:30.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.522+0000 7fac6aa00700 1 -- 172.21.15.43:0/2142384506 wait complete. 2024-05-27T07:29:30.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.523+0000 7fac6aa00700 1 Processor -- start 2024-05-27T07:29:30.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.523+0000 7fac6aa00700 1 -- start start 2024-05-27T07:29:30.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.523+0000 7fac6aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac641034b0 0x7fac64110870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.524+0000 7fac6aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac64107de0 0x7fac64110db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.524+0000 7fac6aa00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac64108730 0x7fac641112f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.524+0000 7fac6aa00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fac64114430 con 0x7fac64107de0 2024-05-27T07:29:30.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.524+0000 7fac6aa00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fac641142b0 con 0x7fac641034b0 2024-05-27T07:29:30.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.524+0000 7fac6aa00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fac641145b0 con 0x7fac64108730 2024-05-27T07:29:30.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.524+0000 7fac63400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac64107de0 0x7fac64110db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.524+0000 7fac63400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac64107de0 0x7fac64110db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36282/0 (socket says 172.21.15.43:36282) 2024-05-27T07:29:30.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.524+0000 7fac63400700 1 -- 172.21.15.43:0/2981019048 learned_addr learned my addr 172.21.15.43:0/2981019048 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:30.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.524+0000 7fac68a00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac64108730 0x7fac641112f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.524+0000 7fac63e00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac641034b0 0x7fac64110870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.525+0000 7fac63400700 1 -- 172.21.15.43:0/2981019048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac64108730 msgr2=0x7fac641112f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.525+0000 7fac63400700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac64108730 0x7fac641112f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.525+0000 7fac63400700 1 -- 172.21.15.43:0/2981019048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac641034b0 msgr2=0x7fac64110870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.525+0000 7fac63400700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac641034b0 0x7fac64110870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.525+0000 7fac63400700 1 -- 172.21.15.43:0/2981019048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac641a91e0 con 0x7fac64107de0 2024-05-27T07:29:30.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.525+0000 7fac63e00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac641034b0 0x7fac64110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:30.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.525+0000 7fac63400700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac64107de0 0x7fac64110db0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fac5000b580 tx=0x7fac5000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:30.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.526+0000 7fac60c00700 1 -- 172.21.15.43:0/2981019048 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50027070 con 0x7fac64107de0 2024-05-27T07:29:30.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.526+0000 7fac60c00700 1 -- 172.21.15.43:0/2981019048 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac500056d0 con 0x7fac64107de0 2024-05-27T07:29:30.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.526+0000 7fac60c00700 1 -- 172.21.15.43:0/2981019048 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac500173f0 con 0x7fac64107de0 2024-05-27T07:29:30.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.526+0000 7fac6aa00700 1 -- 172.21.15.43:0/2981019048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac641a9470 con 0x7fac64107de0 2024-05-27T07:29:30.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.526+0000 7fac6aa00700 1 -- 172.21.15.43:0/2981019048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac641a9870 con 0x7fac64107de0 2024-05-27T07:29:30.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.534+0000 7fac60c00700 1 -- 172.21.15.43:0/2981019048 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fac500050d0 con 0x7fac64107de0 2024-05-27T07:29:30.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.535+0000 7fac79510b80 1 Processor -- start 2024-05-27T07:29:30.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.535+0000 7fac79510b80 1 -- start start 2024-05-27T07:29:30.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.536+0000 7fac79510b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x5597813fe330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.536+0000 7fac79510b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 0x5597813fe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.536+0000 7fac79510b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x55978140b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.536+0000 7fac79510b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55978132ce40 con 0x55978136b9f0 2024-05-27T07:29:30.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.536+0000 7fac79510b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55978131c9e0 con 0x5597813fee70 2024-05-27T07:29:30.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.536+0000 7fac79510b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x559781320a20 con 0x55978127fe40 2024-05-27T07:29:30.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.537+0000 7fac68a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x5597813fe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.537+0000 7fac68a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x5597813fe330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36290/0 (socket says 172.21.15.43:36290) 2024-05-27T07:29:30.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.537+0000 7fac63e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 0x5597813fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.537+0000 7fac68a00700 1 -- 172.21.15.43:0/3256495703 learned_addr learned my addr 172.21.15.43:0/3256495703 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:30.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.538+0000 7fac63400700 1 --2- 172.21.15.43:0/3256495703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x55978140b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.538+0000 7fac63e00700 1 -- 172.21.15.43:0/3256495703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 msgr2=0x55978140b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:30.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.538+0000 7fac63e00700 1 --2- 172.21.15.43:0/3256495703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x55978140b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.538+0000 7fac63e00700 1 -- 172.21.15.43:0/3256495703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 msgr2=0x5597813fe330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:30.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.538+0000 7fac63e00700 1 --2- 172.21.15.43:0/3256495703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x5597813fe330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.538+0000 7fac63e00700 1 -- 172.21.15.43:0/3256495703 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597812e5420 con 0x55978127fe40 2024-05-27T07:29:30.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.538+0000 7fac63400700 1 --2- 172.21.15.43:0/3256495703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x55978140b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:30.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.538+0000 7fac68a00700 1 --2- 172.21.15.43:0/3256495703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x5597813fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:30.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.539+0000 7fac63e00700 1 --2- 172.21.15.43:0/3256495703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 0x5597813fe870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac5400a9d0 tx=0x7fac5400ace0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cef916ad4ab1f72 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:30.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.539+0000 7fac60c00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac480b2a70 0x7fac480b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.540+0000 7fac60c00700 1 -- 172.21.15.43:0/2981019048 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7fac500e37d0 con 0x7fac64107de0 2024-05-27T07:29:30.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.540+0000 7fac59600700 1 -- 172.21.15.43:0/3256495703 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54015070 con 0x55978127fe40 2024-05-27T07:29:30.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.540+0000 7fac59600700 1 -- 172.21.15.43:0/3256495703 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac540113f0 con 0x55978127fe40 2024-05-27T07:29:30.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.540+0000 7fac59600700 1 -- 172.21.15.43:0/3256495703 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac540185c0 con 0x55978127fe40 2024-05-27T07:29:30.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.540+0000 7fac68a00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac480b2a70 0x7fac480b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.541+0000 7fac79510b80 1 -- 172.21.15.43:0/3256495703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 msgr2=0x5597813fe870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.541+0000 7fac79510b80 1 --2- 172.21.15.43:0/3256495703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 0x5597813fe870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac5400a9d0 tx=0x7fac5400ace0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.541+0000 7fac79510b80 1 -- 172.21.15.43:0/3256495703 shutdown_connections 2024-05-27T07:29:30.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.541+0000 7fac79510b80 1 --2- 172.21.15.43:0/3256495703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x55978140b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.541+0000 7fac79510b80 1 --2- 172.21.15.43:0/3256495703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x5597813fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.541+0000 7fac79510b80 1 --2- 172.21.15.43:0/3256495703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 0x5597813fe870 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.541+0000 7fac79510b80 1 -- 172.21.15.43:0/3256495703 >> 172.21.15.43:0/3256495703 conn(0x559781319390 msgr2=0x5597813fa4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:30.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.542+0000 7fac68a00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac480b2a70 0x7fac480b4f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x559781400ff0 tx=0x7fac5c00e040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:30.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.542+0000 7fac79510b80 1 -- 172.21.15.43:0/3256495703 shutdown_connections 2024-05-27T07:29:30.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.542+0000 7fac79510b80 1 -- 172.21.15.43:0/3256495703 wait complete. 2024-05-27T07:29:30.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.543+0000 7fac79510b80 1 -- start start 2024-05-27T07:29:30.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.543+0000 7fac79510b80 1 Processor -- start 2024-05-27T07:29:30.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac79510b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 0x559781408650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac79510b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x559781408b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac79510b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x5597814090d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac79510b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559781222fa0 con 0x55978136b9f0 2024-05-27T07:29:30.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac79510b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5597812bd760 con 0x5597813fee70 2024-05-27T07:29:30.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac79510b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55978140bb80 con 0x55978127fe40 2024-05-27T07:29:30.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac79510b80 10 client.0 _ll_register_callbacks cb 0x5597812c6c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:30.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac63400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x559781408b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac63400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x559781408b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36292/0 (socket says 172.21.15.43:36292) 2024-05-27T07:29:30.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac63400700 1 -- 172.21.15.43:0/1294543906 learned_addr learned my addr 172.21.15.43:0/1294543906 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:30.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac68a00700 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x5597814090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.544+0000 7fac63e00700 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 0x559781408650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.545+0000 7fac63400700 1 -- 172.21.15.43:0/1294543906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 msgr2=0x559781408650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:30.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.545+0000 7fac63400700 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 0x559781408650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.545+0000 7fac63400700 1 -- 172.21.15.43:0/1294543906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 msgr2=0x5597814090d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:30.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.545+0000 7fac63400700 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x5597814090d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.545+0000 7fac63400700 1 -- 172.21.15.43:0/1294543906 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55978126c8d0 con 0x55978136b9f0 2024-05-27T07:29:30.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.545+0000 7fac68a00700 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x5597814090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:30.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.546+0000 7fac63400700 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x559781408b90 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fac500aa470 tx=0x7fac50022750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1a554b4f42a7c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:30.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.547+0000 7fac4e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:29:30.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.547+0000 7fac4e000700 1 -- 172.21.15.43:0/1294543906 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac500aa0c0 con 0x55978136b9f0 2024-05-27T07:29:30.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.547+0000 7fac4e000700 1 -- 172.21.15.43:0/1294543906 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7fac500922e0 con 0x55978136b9f0 2024-05-27T07:29:30.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.548+0000 7fac79510b80 1 -- 172.21.15.43:0/1294543906 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac540113f0 con 0x55978136b9f0 2024-05-27T07:29:30.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.548+0000 7fac4e000700 1 -- 172.21.15.43:0/1294543906 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fac500a9ee0 con 0x55978136b9f0 2024-05-27T07:29:30.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.548+0000 7fac79510b80 10 client.17649 fetch_fsmap learned FSMap version 119 2024-05-27T07:29:30.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.548+0000 7fac79510b80 1 -- 172.21.15.43:0/1294543906 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x5597812ed060 con 0x55978136b9f0 2024-05-27T07:29:30.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.549+0000 7fac4e000700 1 -- 172.21.15.43:0/1294543906 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac540113f0 con 0x55978136b9f0 2024-05-27T07:29:30.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.549+0000 7fac4e000700 1 -- 172.21.15.43:0/1294543906 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 119) ==== 4816+0+0 (secure 0 0 0) 0x7fac500e3060 con 0x55978136b9f0 2024-05-27T07:29:30.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.549+0000 7fac79510b80 10 client.17649 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:29:30.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.549+0000 7fac79510b80 10 client.17649 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:29:30.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.549+0000 7fac79510b80 20 client.17649 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:30.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.549+0000 7fac79510b80 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x559781421220 0x559781423670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.549+0000 7fac79510b80 4 client.17649 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:30.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.549+0000 7fac79510b80 1 -- 172.21.15.43:0/1294543906 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55978132ed90 con 0x559781421220 2024-05-27T07:29:30.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.550+0000 7fac63e00700 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x559781421220 0x559781423670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.551+0000 7fac63e00700 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x559781421220 0x559781423670 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=266f5b2e17da136 server_cookie=7e850f6f295e4ec1 in_seq=0 out_seq=0 2024-05-27T07:29:30.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.551+0000 7fac4e000700 10 client.17649 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:29:30.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.552+0000 7fac4e000700 1 -- 172.21.15.43:0/1294543906 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55978132ed90 con 0x559781421220 2024-05-27T07:29:30.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.552+0000 7fac4e000700 10 client.17649 handle_command_reply: tid=0 2024-05-27T07:29:30.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.552+0000 7fac79510b80 1 client.17649 shutdown 2024-05-27T07:29:30.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.553+0000 7fac79510b80 1 -- 172.21.15.43:0/1294543906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 msgr2=0x559781408b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.553+0000 7fac79510b80 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x559781408b90 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fac500aa470 tx=0x7fac50022750 comp rx=0 tx=0).stop 2024-05-27T07:29:30.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.554+0000 7fac79510b80 1 -- 172.21.15.43:0/1294543906 shutdown_connections 2024-05-27T07:29:30.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.554+0000 7fac79510b80 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x559781421220 0x559781423670 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.554+0000 7fac79510b80 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x5597814090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.554+0000 7fac79510b80 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978127fe40 0x559781408650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.554+0000 7fac79510b80 1 --2- 172.21.15.43:0/1294543906 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978136b9f0 0x559781408b90 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.554+0000 7fac79510b80 1 -- 172.21.15.43:0/1294543906 >> 172.21.15.43:0/1294543906 conn(0x559781319390 msgr2=0x5597813fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:30.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.554+0000 7fac4e000700 0 client.17649 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:29:30.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.554+0000 7fac79510b80 1 -- 172.21.15.43:0/1294543906 shutdown_connections 2024-05-27T07:29:30.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.554+0000 7fac79510b80 1 -- 172.21.15.43:0/1294543906 wait complete. 2024-05-27T07:29:30.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.554+0000 7fac79510b80 20 client.17649 trim_cache size 0 max 16384 2024-05-27T07:29:30.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.586+0000 7fac79510b80 1 Processor -- start 2024-05-27T07:29:30.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.586+0000 7fac79510b80 1 -- start start 2024-05-27T07:29:30.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.587+0000 7fac79510b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978127fe40 0x5597814238a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.587+0000 7fac79510b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978136b9f0 0x559781403e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.587+0000 7fac79510b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x559781407650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.587+0000 7fac79510b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5597813fa680 con 0x55978127fe40 2024-05-27T07:29:30.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.587+0000 7fac79510b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fac641af050 con 0x5597813fee70 2024-05-27T07:29:30.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.587+0000 7fac79510b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fac641acc90 con 0x55978136b9f0 2024-05-27T07:29:30.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.587+0000 7fac63e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978127fe40 0x5597814238a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.587+0000 7fac63e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978127fe40 0x5597814238a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36304/0 (socket says 172.21.15.43:36304) 2024-05-27T07:29:30.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.587+0000 7fac63e00700 1 -- 172.21.15.43:0/4031016366 learned_addr learned my addr 172.21.15.43:0/4031016366 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:30.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.588+0000 7fac68a00700 1 --2- 172.21.15.43:0/4031016366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x559781407650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.588+0000 7fac63e00700 1 -- 172.21.15.43:0/4031016366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978136b9f0 msgr2=0x559781403e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:30.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.588+0000 7fac63e00700 1 --2- 172.21.15.43:0/4031016366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978136b9f0 0x559781403e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.588+0000 7fac63e00700 1 -- 172.21.15.43:0/4031016366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 msgr2=0x559781407650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:30.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.588+0000 7fac63e00700 1 --2- 172.21.15.43:0/4031016366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x559781407650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.588+0000 7fac63e00700 1 -- 172.21.15.43:0/4031016366 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597813ffa50 con 0x55978127fe40 2024-05-27T07:29:30.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.588+0000 7fac68a00700 1 --2- 172.21.15.43:0/4031016366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x559781407650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:30.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.588+0000 7fac63e00700 1 --2- 172.21.15.43:0/4031016366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978127fe40 0x5597814238a0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fac5400a8f0 tx=0x7fac5400a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e463087eb1fcec78 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:30.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.589+0000 7fac58c00700 1 -- 172.21.15.43:0/4031016366 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54015040 con 0x55978127fe40 2024-05-27T07:29:30.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.589+0000 7fac58c00700 1 -- 172.21.15.43:0/4031016366 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac540113f0 con 0x55978127fe40 2024-05-27T07:29:30.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.589+0000 7fac58c00700 1 -- 172.21.15.43:0/4031016366 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5401a840 con 0x55978127fe40 2024-05-27T07:29:30.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.590+0000 7fac79510b80 1 -- 172.21.15.43:0/4031016366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978127fe40 msgr2=0x5597814238a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.590+0000 7fac79510b80 1 --2- 172.21.15.43:0/4031016366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978127fe40 0x5597814238a0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fac5400a8f0 tx=0x7fac5400a1d0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.590+0000 7fac79510b80 1 -- 172.21.15.43:0/4031016366 shutdown_connections 2024-05-27T07:29:30.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.590+0000 7fac79510b80 1 --2- 172.21.15.43:0/4031016366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597813fee70 0x559781407650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.590+0000 7fac79510b80 1 --2- 172.21.15.43:0/4031016366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55978127fe40 0x5597814238a0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.590+0000 7fac79510b80 1 --2- 172.21.15.43:0/4031016366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978136b9f0 0x559781403e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.590+0000 7fac79510b80 1 -- 172.21.15.43:0/4031016366 >> 172.21.15.43:0/4031016366 conn(0x559781319390 msgr2=0x559781406f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:30.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.591+0000 7fac79510b80 1 -- 172.21.15.43:0/4031016366 shutdown_connections 2024-05-27T07:29:30.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.591+0000 7fac79510b80 1 -- 172.21.15.43:0/4031016366 wait complete. 2024-05-27T07:29:30.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.591+0000 7fac79510b80 1 -- start start 2024-05-27T07:29:30.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.592+0000 7fac79510b80 1 Processor -- start 2024-05-27T07:29:30.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.592+0000 7fac79510b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55978127fe40 0x559781403db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.592+0000 7fac79510b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978136b9f0 0x559781402140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.592+0000 7fac79510b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597813fee70 0x559781402740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.592+0000 7fac79510b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fac641ad3c0 con 0x5597813fee70 2024-05-27T07:29:30.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.592+0000 7fac79510b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fac641ae980 con 0x55978127fe40 2024-05-27T07:29:30.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.592+0000 7fac79510b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fac641adac0 con 0x55978136b9f0 2024-05-27T07:29:30.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.593+0000 7fac79510b80 10 client.0 _ll_register_callbacks cb 0x5597813f9220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:30.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.593+0000 7fac63e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55978127fe40 0x559781403db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.593+0000 7fac63e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55978127fe40 0x559781403db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52202/0 (socket says 172.21.15.43:52202) 2024-05-27T07:29:30.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.593+0000 7fac63e00700 1 -- 172.21.15.43:0/1616595243 learned_addr learned my addr 172.21.15.43:0/1616595243 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:30.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.593+0000 7fac68a00700 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597813fee70 0x559781402740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.593+0000 7fac63400700 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978136b9f0 0x559781402140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.594+0000 7fac68a00700 1 -- 172.21.15.43:0/1616595243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978136b9f0 msgr2=0x559781402140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:30.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.594+0000 7fac68a00700 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978136b9f0 0x559781402140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.594+0000 7fac68a00700 1 -- 172.21.15.43:0/1616595243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55978127fe40 msgr2=0x559781403db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:30.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.594+0000 7fac68a00700 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55978127fe40 0x559781403db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.594+0000 7fac68a00700 1 -- 172.21.15.43:0/1616595243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559781402f30 con 0x5597813fee70 2024-05-27T07:29:30.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.594+0000 7fac63e00700 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55978127fe40 0x559781403db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:30.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.594+0000 7fac68a00700 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597813fee70 0x559781402740 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009660 tx=0x7fac5c004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb31407c5076492 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:30.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.594+0000 7fac4e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:29:30.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.595+0000 7fac4e000700 1 -- 172.21.15.43:0/1616595243 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5c0169e0 con 0x5597813fee70 2024-05-27T07:29:30.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.595+0000 7fac4e000700 1 -- 172.21.15.43:0/1616595243 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7fac5c016b80 con 0x5597813fee70 2024-05-27T07:29:30.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.595+0000 7fac79510b80 1 -- 172.21.15.43:0/1616595243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac540113f0 con 0x5597813fee70 2024-05-27T07:29:30.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.596+0000 7fac4e000700 1 -- 172.21.15.43:0/1616595243 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) ==== 24+0+0 (secure 0 0 0) 0x7fac5c031050 con 0x5597813fee70 2024-05-27T07:29:30.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.596+0000 7fac79510b80 10 client.17655 fetch_fsmap learned FSMap version 119 2024-05-27T07:29:30.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.596+0000 7fac79510b80 1 -- 172.21.15.43:0/1616595243 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=4+}) -- 0x559781410dc0 con 0x5597813fee70 2024-05-27T07:29:30.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.596+0000 7fac4e000700 1 -- 172.21.15.43:0/1616595243 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac540113f0 con 0x5597813fee70 2024-05-27T07:29:30.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.597+0000 7fac4e000700 1 -- 172.21.15.43:0/1616595243 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 119) ==== 4816+0+0 (secure 0 0 0) 0x7fac5c02c3b0 con 0x5597813fee70 2024-05-27T07:29:30.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.597+0000 7fac79510b80 10 client.17655 fetch_fsmap finished waiting for FSMap version 119 2024-05-27T07:29:30.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.597+0000 7fac79510b80 10 client.17655 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:29:30.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.597+0000 7fac79510b80 20 client.17655 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:30.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.597+0000 7fac79510b80 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x559781411590 0x55978148dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:30.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.597+0000 7fac79510b80 4 client.17655 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:30.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.597+0000 7fac79510b80 1 -- 172.21.15.43:0/1616595243 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559781328570 con 0x559781411590 2024-05-27T07:29:30.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.597+0000 7fac63e00700 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x559781411590 0x55978148dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:30.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.598+0000 7fac63e00700 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x559781411590 0x55978148dd50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d7b8f3f880521d19 server_cookie=8f42084933561e97 in_seq=0 out_seq=0 2024-05-27T07:29:30.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.598+0000 7fac4e000700 10 client.17655 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:29:30.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.598+0000 7fac4e000700 1 -- 172.21.15.43:0/1616595243 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559781328570 con 0x559781411590 2024-05-27T07:29:30.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.598+0000 7fac4e000700 10 client.17655 handle_command_reply: tid=0 2024-05-27T07:29:30.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.598+0000 7fac79510b80 1 client.17655 shutdown 2024-05-27T07:29:30.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac79510b80 1 -- 172.21.15.43:0/1616595243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597813fee70 msgr2=0x559781402740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac79510b80 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597813fee70 0x559781402740 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009660 tx=0x7fac5c004be0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac79510b80 1 -- 172.21.15.43:0/1616595243 shutdown_connections 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac79510b80 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x559781411590 0x55978148dd50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac79510b80 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597813fee70 0x559781402740 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac79510b80 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55978127fe40 0x559781403db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac79510b80 1 --2- 172.21.15.43:0/1616595243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55978136b9f0 0x559781402140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac79510b80 1 -- 172.21.15.43:0/1616595243 >> 172.21.15.43:0/1616595243 conn(0x559781319390 msgr2=0x5597813ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac4e000700 0 client.17655 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac79510b80 1 -- 172.21.15.43:0/1616595243 shutdown_connections 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.599+0000 7fac79510b80 1 -- 172.21.15.43:0/1616595243 wait complete. 2024-05-27T07:29:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.600+0000 7fac79510b80 20 client.17655 trim_cache size 0 max 16384 2024-05-27T07:29:30.591 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:29:30.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.600+0000 7fac6aa00700 1 -- 172.21.15.43:0/2981019048 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac480b2a70 msgr2=0x7fac480b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.600+0000 7fac6aa00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac480b2a70 0x7fac480b4f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x559781400ff0 tx=0x7fac5c00e040 comp rx=0 tx=0).stop 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.600+0000 7fac6aa00700 1 -- 172.21.15.43:0/2981019048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac64107de0 msgr2=0x7fac64110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.600+0000 7fac6aa00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac64107de0 0x7fac64110db0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fac5000b580 tx=0x7fac5000f740 comp rx=0 tx=0).stop 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.601+0000 7fac6aa00700 1 -- 172.21.15.43:0/2981019048 shutdown_connections 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.601+0000 7fac6aa00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac480b2a70 0x7fac480b4f80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.601+0000 7fac6aa00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac64108730 0x7fac641112f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.601+0000 7fac6aa00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac641034b0 0x7fac64110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.601+0000 7fac6aa00700 1 --2- 172.21.15.43:0/2981019048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac64107de0 0x7fac64110db0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.601+0000 7fac6aa00700 1 -- 172.21.15.43:0/2981019048 >> 172.21.15.43:0/2981019048 conn(0x7fac640fedb0 msgr2=0x7fac64109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.601+0000 7fac6aa00700 1 -- 172.21.15.43:0/2981019048 shutdown_connections 2024-05-27T07:29:30.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:30.601+0000 7fac6aa00700 1 -- 172.21.15.43:0/2981019048 wait complete. 2024-05-27T07:29:30.771 INFO:tasks.workunit.client.0.smithi043.stdout: 10 10300 8.02 MB/sec warmup 91 sec latency 441.223 ms 2024-05-27T07:29:30.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:30 smithi132 ceph-mon[35283]: pgmap v522: 129 pgs: 129 active+clean; 474 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 151 op/s 2024-05-27T07:29:30.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:30 smithi043 ceph-mon[31531]: pgmap v522: 129 pgs: 129 active+clean; 474 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 151 op/s 2024-05-27T07:29:30.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:30 smithi179 ceph-mon[35888]: pgmap v522: 129 pgs: 129 active+clean; 474 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 151 op/s 2024-05-27T07:29:31.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:29:31.771 INFO:tasks.workunit.client.0.smithi043.stdout: 10 10416 8.03 MB/sec warmup 92 sec latency 102.424 ms 2024-05-27T07:29:32.141 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:32.771 INFO:tasks.workunit.client.0.smithi043.stdout: 10 10528 7.97 MB/sec warmup 93 sec latency 105.931 ms 2024-05-27T07:29:32.810 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:32 smithi043 ceph-mon[31531]: pgmap v523: 129 pgs: 129 active+clean; 484 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 217 op/s 2024-05-27T07:29:32.811 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:32 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:32.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:32 smithi132 ceph-mon[35283]: pgmap v523: 129 pgs: 129 active+clean; 484 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 217 op/s 2024-05-27T07:29:32.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:32 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:32.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:32 smithi179 ceph-mon[35888]: pgmap v523: 129 pgs: 129 active+clean; 484 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 217 op/s 2024-05-27T07:29:32.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:32 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:33.772 INFO:tasks.workunit.client.0.smithi043.stdout: 10 10655 7.97 MB/sec warmup 94 sec latency 79.836 ms 2024-05-27T07:29:34.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:34 smithi043 ceph-mon[31531]: pgmap v524: 129 pgs: 129 active+clean; 484 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-05-27T07:29:34.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.658+0000 7f3d0f600700 1 -- 172.21.15.43:0/938069339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d08105e40 msgr2=0x7f3d0810e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:34.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.658+0000 7f3d0f600700 1 --2- 172.21.15.43:0/938069339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d08105e40 0x7f3d0810e390 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009990 tx=0x7f3cf4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.658+0000 7f3d0f600700 1 -- 172.21.15.43:0/938069339 shutdown_connections 2024-05-27T07:29:34.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.658+0000 7f3d0f600700 1 --2- 172.21.15.43:0/938069339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d0810e8d0 0x7f3d08110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.658+0000 7f3d0f600700 1 --2- 172.21.15.43:0/938069339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d081054f0 0x7f3d08105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.658+0000 7f3d0f600700 1 --2- 172.21.15.43:0/938069339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d08105e40 0x7f3d0810e390 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.658+0000 7f3d0f600700 1 -- 172.21.15.43:0/938069339 >> 172.21.15.43:0/938069339 conn(0x7f3d080fedb0 msgr2=0x7f3d081011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:34.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.660+0000 7f3d0f600700 1 -- 172.21.15.43:0/938069339 shutdown_connections 2024-05-27T07:29:34.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.660+0000 7f3d0f600700 1 -- 172.21.15.43:0/938069339 wait complete. 2024-05-27T07:29:34.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.661+0000 7f3d0f600700 1 Processor -- start 2024-05-27T07:29:34.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.661+0000 7f3d0f600700 1 -- start start 2024-05-27T07:29:34.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.662+0000 7f3d0f600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d081054f0 0x7f3d081997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.662+0000 7f3d0f600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d08105e40 0x7f3d08199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.662+0000 7f3d0f600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d0810e8d0 0x7f3d081a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.662+0000 7f3d0f600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3d08114300 con 0x7f3d081054f0 2024-05-27T07:29:34.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.662+0000 7f3d0f600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d08114180 con 0x7f3d0810e8d0 2024-05-27T07:29:34.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.662+0000 7f3d0f600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3d08114480 con 0x7f3d08105e40 2024-05-27T07:29:34.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.662+0000 7f3d0cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d081054f0 0x7f3d081997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.662+0000 7f3d0cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d081054f0 0x7f3d081997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59808/0 (socket says 172.21.15.43:59808) 2024-05-27T07:29:34.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.662+0000 7f3d0cc00700 1 -- 172.21.15.43:0/1867367777 learned_addr learned my addr 172.21.15.43:0/1867367777 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:34.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.662+0000 7f3d0d600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d0810e8d0 0x7f3d081a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.663+0000 7f3d07e00700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d08105e40 0x7f3d08199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.663+0000 7f3d0d600700 1 -- 172.21.15.43:0/1867367777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d08105e40 msgr2=0x7f3d08199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:34.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.663+0000 7f3d0d600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d08105e40 0x7f3d08199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.663+0000 7f3d0d600700 1 -- 172.21.15.43:0/1867367777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d081054f0 msgr2=0x7f3d081997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:34.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.663+0000 7f3d0d600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d081054f0 0x7f3d081997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.663+0000 7f3d0d600700 1 -- 172.21.15.43:0/1867367777 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d081a0d70 con 0x7f3d0810e8d0 2024-05-27T07:29:34.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.663+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d081054f0 0x7f3d081997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:34.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.663+0000 7f3d07e00700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d08105e40 0x7f3d08199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:34.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.664+0000 7f3d0d600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d0810e8d0 0x7f3d081a0830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000efb0 tx=0x7f3d0000c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:34.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.664+0000 7f3d05600700 1 -- 172.21.15.43:0/1867367777 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0000cb00 con 0x7f3d0810e8d0 2024-05-27T07:29:34.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.664+0000 7f3d0f600700 1 -- 172.21.15.43:0/1867367777 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d081a1060 con 0x7f3d0810e8d0 2024-05-27T07:29:34.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.664+0000 7f3d05600700 1 -- 172.21.15.43:0/1867367777 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3d0000cca0 con 0x7f3d0810e8d0 2024-05-27T07:29:34.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.664+0000 7f3d05600700 1 -- 172.21.15.43:0/1867367777 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00004870 con 0x7f3d0810e8d0 2024-05-27T07:29:34.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.665+0000 7f3d0f600700 1 -- 172.21.15.43:0/1867367777 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d081a1550 con 0x7f3d0810e8d0 2024-05-27T07:29:34.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.669+0000 7f3d05600700 1 -- 172.21.15.43:0/1867367777 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3d00012050 con 0x7f3d0810e8d0 2024-05-27T07:29:34.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.672+0000 7f3d1e139b80 1 Processor -- start 2024-05-27T07:29:34.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.671+0000 7f3d05600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3cf00b29f0 0x7f3cf00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.672+0000 7f3d05600700 1 -- 172.21.15.43:0/1867367777 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7f3d000e2470 con 0x7f3d0810e8d0 2024-05-27T07:29:34.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.672+0000 7f3d1e139b80 1 -- start start 2024-05-27T07:29:34.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.672+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3cf00b29f0 0x7f3cf00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.673+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3149f0 0x556c0a3a7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.673+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a228e40 0x556c0a3a7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.673+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 0x556c0a3b4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.673+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556c0a2d5e40 con 0x556c0a228e40 2024-05-27T07:29:34.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.673+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556c0a2c59e0 con 0x556c0a3a7e70 2024-05-27T07:29:34.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.673+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556c0a2c9a20 con 0x556c0a3149f0 2024-05-27T07:29:34.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.673+0000 7f3d0d600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 0x556c0a3b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.673+0000 7f3d0d600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 0x556c0a3b4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:44552/0 (socket says 172.21.15.43:44552) 2024-05-27T07:29:34.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.673+0000 7f3d0d600700 1 -- 172.21.15.43:0/3205043000 learned_addr learned my addr 172.21.15.43:0/3205043000 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:34.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.674+0000 7f3d0d600700 1 -- 172.21.15.43:0/3205043000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3149f0 msgr2=0x556c0a3a7330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:29:34.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.674+0000 7f3d0d600700 1 --2- 172.21.15.43:0/3205043000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3149f0 0x556c0a3a7330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.674+0000 7f3d0d600700 1 -- 172.21.15.43:0/3205043000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a228e40 msgr2=0x556c0a3a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:34.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.674+0000 7f3d07e00700 1 --2- 172.21.15.43:0/3205043000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a228e40 0x556c0a3a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.674+0000 7f3d0d600700 1 --2- 172.21.15.43:0/3205043000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a228e40 0x556c0a3a7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.674+0000 7f3d0d600700 1 -- 172.21.15.43:0/3205043000 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c0a28e420 con 0x556c0a3a7e70 2024-05-27T07:29:34.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.674+0000 7f3d07e00700 1 --2- 172.21.15.43:0/3205043000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a228e40 0x556c0a3a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.675+0000 7f3d0d600700 1 --2- 172.21.15.43:0/3205043000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 0x556c0a3b4110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3d000a5430 tx=0x7f3d0000cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e96fd6e2b54dc68e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.675+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3cf00b29f0 0x7f3cf00b4f00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00bd50 tx=0x7f3cfc00b320 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.675+0000 7f3cfa000700 1 -- 172.21.15.43:0/3205043000 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d000a9ab0 con 0x556c0a3a7e70 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.675+0000 7f3cfa000700 1 -- 172.21.15.43:0/3205043000 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3d000a93b0 con 0x556c0a3a7e70 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.675+0000 7f3cfa000700 1 -- 172.21.15.43:0/3205043000 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d000a9610 con 0x556c0a3a7e70 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.676+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3205043000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 msgr2=0x556c0a3b4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.676+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/3205043000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 0x556c0a3b4110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3d000a5430 tx=0x7f3d0000cf70 comp rx=0 tx=0).stop 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.676+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3205043000 shutdown_connections 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.676+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/3205043000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 0x556c0a3b4110 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.676+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/3205043000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3149f0 0x556c0a3a7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.676+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/3205043000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a228e40 0x556c0a3a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.676+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3205043000 >> 172.21.15.43:0/3205043000 conn(0x556c0a2c2390 msgr2=0x556c0a3a34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:34.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.677+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3205043000 shutdown_connections 2024-05-27T07:29:34.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.677+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3205043000 wait complete. 2024-05-27T07:29:34.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.677+0000 7f3d1e139b80 1 -- start start 2024-05-27T07:29:34.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.678+0000 7f3d1e139b80 1 Processor -- start 2024-05-27T07:29:34.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.678+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.678+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3149f0 0x556c0a3b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.678+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3a7e70 0x556c0a3b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.678+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556c0a1cbfa0 con 0x556c0a3a7e70 2024-05-27T07:29:34.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.678+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556c0a266760 con 0x556c0a228e40 2024-05-27T07:29:34.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.678+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556c0a3b4b80 con 0x556c0a3149f0 2024-05-27T07:29:34.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d1e139b80 10 client.0 _ll_register_callbacks cb 0x556c0a26fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:34.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d0cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d0cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:44566/0 (socket says 172.21.15.43:44566) 2024-05-27T07:29:34.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d0cc00700 1 -- 172.21.15.43:0/804659182 learned_addr learned my addr 172.21.15.43:0/804659182 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:34.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d0d600700 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3a7e70 0x556c0a3b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d07e00700 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3149f0 0x556c0a3b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d0cc00700 1 -- 172.21.15.43:0/804659182 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3149f0 msgr2=0x556c0a3b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:34.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3149f0 0x556c0a3b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d0cc00700 1 -- 172.21.15.43:0/804659182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3a7e70 msgr2=0x556c0a3b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:34.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3a7e70 0x556c0a3b21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.679+0000 7f3d0cc00700 1 -- 172.21.15.43:0/804659182 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c0a2158d0 con 0x556c0a228e40 2024-05-27T07:29:34.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.680+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b1760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00ffb0 tx=0x7f3cfc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=151f3d3a975cf9ba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:34.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.680+0000 7f3ceea00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:29:34.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.680+0000 7f3ceea00700 1 -- 172.21.15.43:0/804659182 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc022070 con 0x556c0a228e40 2024-05-27T07:29:34.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.680+0000 7f3ceea00700 1 -- 172.21.15.43:0/804659182 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7f3cfc004a30 con 0x556c0a228e40 2024-05-27T07:29:34.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.680+0000 7f3d1e139b80 1 -- 172.21.15.43:0/804659182 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d000a93b0 con 0x556c0a228e40 2024-05-27T07:29:34.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.681+0000 7f3ceea00700 1 -- 172.21.15.43:0/804659182 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) ==== 24+0+0 (secure 0 0 0) 0x7f3cfc02d050 con 0x556c0a228e40 2024-05-27T07:29:34.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.681+0000 7f3d1e139b80 10 client.37067 fetch_fsmap learned FSMap version 121 2024-05-27T07:29:34.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.681+0000 7f3d1e139b80 1 -- 172.21.15.43:0/804659182 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=4+}) -- 0x556c0a296060 con 0x556c0a228e40 2024-05-27T07:29:34.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.681+0000 7f3ceea00700 1 -- 172.21.15.43:0/804659182 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3d000a93b0 con 0x556c0a228e40 2024-05-27T07:29:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.682+0000 7f3ceea00700 1 -- 172.21.15.43:0/804659182 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 121) ==== 4816+0+0 (secure 0 0 0) 0x7f3cfc027720 con 0x556c0a228e40 2024-05-27T07:29:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.682+0000 7f3d1e139b80 10 client.37067 fetch_fsmap finished waiting for FSMap version 121 2024-05-27T07:29:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.682+0000 7f3d1e139b80 10 client.37067 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.682+0000 7f3d1e139b80 20 client.37067 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.682+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556c0a3c1f10 0x556c0a3c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.682+0000 7f3d1e139b80 4 client.37067 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.682+0000 7f3d1e139b80 1 -- 172.21.15.43:0/804659182 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c0a2d7d90 con 0x556c0a3c1f10 2024-05-27T07:29:34.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.682+0000 7f3d07e00700 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556c0a3c1f10 0x556c0a3c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.683+0000 7f3d07e00700 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556c0a3c1f10 0x556c0a3c43c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be64e06fa80500dd server_cookie=69476012f4cd32ba in_seq=0 out_seq=0 2024-05-27T07:29:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.683+0000 7f3ceea00700 10 client.37067 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.684+0000 7f3ceea00700 1 -- 172.21.15.43:0/804659182 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556c0a2d7d90 con 0x556c0a3c1f10 2024-05-27T07:29:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.684+0000 7f3ceea00700 10 client.37067 handle_command_reply: tid=0 2024-05-27T07:29:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.684+0000 7f3d1e139b80 1 client.37067 shutdown 2024-05-27T07:29:34.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3d1e139b80 1 -- 172.21.15.43:0/804659182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 msgr2=0x556c0a3b1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:34.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b1760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00ffb0 tx=0x7f3cfc004750 comp rx=0 tx=0).stop 2024-05-27T07:29:34.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3d1e139b80 1 -- 172.21.15.43:0/804659182 shutdown_connections 2024-05-27T07:29:34.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556c0a3c1f10 0x556c0a3c43c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3a7e70 0x556c0a3b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b1760 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/804659182 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3149f0 0x556c0a3b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3ceea00700 0 client.37067 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:34.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3d1e139b80 1 -- 172.21.15.43:0/804659182 >> 172.21.15.43:0/804659182 conn(0x556c0a2c2390 msgr2=0x556c0a3a73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:34.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3d1e139b80 1 -- 172.21.15.43:0/804659182 shutdown_connections 2024-05-27T07:29:34.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.685+0000 7f3d1e139b80 1 -- 172.21.15.43:0/804659182 wait complete. 2024-05-27T07:29:34.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.686+0000 7f3d1e139b80 20 client.37067 trim_cache size 0 max 16384 2024-05-27T07:29:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.715+0000 7f3d1e139b80 1 Processor -- start 2024-05-27T07:29:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.715+0000 7f3d1e139b80 1 -- start start 2024-05-27T07:29:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.716+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.716+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a3acf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.716+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3a7e70 0x556c0a3b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.716+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556c0a3a7850 con 0x556c0a3149f0 2024-05-27T07:29:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.716+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d081c0160 con 0x556c0a228e40 2024-05-27T07:29:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.716+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3d081bdda0 con 0x556c0a3a7e70 2024-05-27T07:29:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.717+0000 7f3d07e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a3acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.717+0000 7f3d0d600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3a7e70 0x556c0a3b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.717+0000 7f3d0cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.717+0000 7f3d07e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a3acf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59842/0 (socket says 172.21.15.43:59842) 2024-05-27T07:29:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.717+0000 7f3d07e00700 1 -- 172.21.15.43:0/1919479671 learned_addr learned my addr 172.21.15.43:0/1919479671 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.717+0000 7f3d07e00700 1 -- 172.21.15.43:0/1919479671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3a7e70 msgr2=0x556c0a3b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:34.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.718+0000 7f3d07e00700 1 --2- 172.21.15.43:0/1919479671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3a7e70 0x556c0a3b1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.718+0000 7f3d07e00700 1 -- 172.21.15.43:0/1919479671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 msgr2=0x556c0a3b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:34.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.718+0000 7f3d07e00700 1 --2- 172.21.15.43:0/1919479671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.718+0000 7f3d07e00700 1 -- 172.21.15.43:0/1919479671 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c0a3afcd0 con 0x556c0a3149f0 2024-05-27T07:29:34.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.718+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/1919479671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:34.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.719+0000 7f3d0d600700 1 --2- 172.21.15.43:0/1919479671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3a7e70 0x556c0a3b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.719+0000 7f3d07e00700 1 --2- 172.21.15.43:0/1919479671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a3acf30 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400b2e0 tx=0x7f3cf4005360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801f13326fd0716a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.720+0000 7f3cf9600700 1 -- 172.21.15.43:0/1919479671 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf40054d0 con 0x556c0a3149f0 2024-05-27T07:29:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.720+0000 7f3cf9600700 1 -- 172.21.15.43:0/1919479671 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3cf4005670 con 0x556c0a3149f0 2024-05-27T07:29:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.720+0000 7f3cf9600700 1 -- 172.21.15.43:0/1919479671 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf400ba70 con 0x556c0a3149f0 2024-05-27T07:29:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.721+0000 7f3d1e139b80 1 -- 172.21.15.43:0/1919479671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 msgr2=0x556c0a3acf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.721+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/1919479671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a3acf30 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400b2e0 tx=0x7f3cf4005360 comp rx=0 tx=0).stop 2024-05-27T07:29:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.722+0000 7f3d1e139b80 1 -- 172.21.15.43:0/1919479671 shutdown_connections 2024-05-27T07:29:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.722+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/1919479671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a3a7e70 0x556c0a3b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.722+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/1919479671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a228e40 0x556c0a3b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.722+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/1919479671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a3acf30 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.722+0000 7f3d1e139b80 1 -- 172.21.15.43:0/1919479671 >> 172.21.15.43:0/1919479671 conn(0x556c0a2c2390 msgr2=0x556c0a3b1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.723+0000 7f3d1e139b80 1 -- 172.21.15.43:0/1919479671 shutdown_connections 2024-05-27T07:29:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.723+0000 7f3d1e139b80 1 -- 172.21.15.43:0/1919479671 wait complete. 2024-05-27T07:29:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.723+0000 7f3d1e139b80 1 -- start start 2024-05-27T07:29:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.724+0000 7f3d1e139b80 1 Processor -- start 2024-05-27T07:29:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.724+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a228e40 0x556c0a429c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.724+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a42a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.725+0000 7f3d1e139b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 0x556c0a3b9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.725+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3d081be4d0 con 0x556c0a3149f0 2024-05-27T07:29:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.725+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d081bfa60 con 0x556c0a3a7e70 2024-05-27T07:29:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.725+0000 7f3d1e139b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3d081bebd0 con 0x556c0a228e40 2024-05-27T07:29:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.725+0000 7f3d1e139b80 10 client.0 _ll_register_callbacks cb 0x556c0a26fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.725+0000 7f3d07e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a42a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.725+0000 7f3d07e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a42a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59852/0 (socket says 172.21.15.43:59852) 2024-05-27T07:29:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.725+0000 7f3d07e00700 1 -- 172.21.15.43:0/3643674503 learned_addr learned my addr 172.21.15.43:0/3643674503 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.726+0000 7f3d0d600700 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 0x556c0a3b9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.726+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a228e40 0x556c0a429c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.726+0000 7f3d07e00700 1 -- 172.21.15.43:0/3643674503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a228e40 msgr2=0x556c0a429c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.726+0000 7f3d07e00700 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a228e40 0x556c0a429c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.726+0000 7f3d07e00700 1 -- 172.21.15.43:0/3643674503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 msgr2=0x556c0a3b9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.726+0000 7f3d07e00700 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 0x556c0a3b9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.726+0000 7f3d07e00700 1 -- 172.21.15.43:0/3643674503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c0a3ba340 con 0x556c0a3149f0 2024-05-27T07:29:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.726+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a228e40 0x556c0a429c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.726+0000 7f3ceea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:29:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.726+0000 7f3d07e00700 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a42a1d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009ce0 tx=0x7f3cf40051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f1396a9c1fe8bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.727+0000 7f3ceea00700 1 -- 172.21.15.43:0/3643674503 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4003950 con 0x556c0a3149f0 2024-05-27T07:29:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.727+0000 7f3ceea00700 1 -- 172.21.15.43:0/3643674503 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 9496+0+0 (secure 0 0 0) 0x7f3cf4003af0 con 0x556c0a3149f0 2024-05-27T07:29:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.727+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3643674503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cf4005670 con 0x556c0a3149f0 2024-05-27T07:29:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.728+0000 7f3ceea00700 1 -- 172.21.15.43:0/3643674503 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) ==== 24+0+0 (secure 0 0 0) 0x7f3cf4003cf0 con 0x556c0a3149f0 2024-05-27T07:29:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.728+0000 7f3d1e139b80 10 client.17670 fetch_fsmap learned FSMap version 121 2024-05-27T07:29:34.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.728+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3643674503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=4+}) -- 0x556c0a3ac2b0 con 0x556c0a3149f0 2024-05-27T07:29:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.729+0000 7f3ceea00700 1 -- 172.21.15.43:0/3643674503 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3cf4005670 con 0x556c0a3149f0 2024-05-27T07:29:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.729+0000 7f3ceea00700 1 -- 172.21.15.43:0/3643674503 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 121) ==== 4816+0+0 (secure 0 0 0) 0x7f3cf4025740 con 0x556c0a3149f0 2024-05-27T07:29:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.729+0000 7f3d1e139b80 10 client.17670 fetch_fsmap finished waiting for FSMap version 121 2024-05-27T07:29:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.729+0000 7f3d1e139b80 10 client.17670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.729+0000 7f3d1e139b80 20 client.17670 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:34.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.729+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556c0a43db70 0x556c0a43ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.729+0000 7f3d1e139b80 4 client.17670 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:29:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.729+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3643674503 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c0a2d1570 con 0x556c0a43db70 2024-05-27T07:29:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.730+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556c0a43db70 0x556c0a43ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.730+0000 7f3ceea00700 10 client.17670 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.730+0000 7f3d0cc00700 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556c0a43db70 0x556c0a43ff90 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11e4726792945c6c server_cookie=8e2278b439bd5d5e in_seq=0 out_seq=0 2024-05-27T07:29:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.731+0000 7f3ceea00700 1 -- 172.21.15.43:0/3643674503 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c0a2d1570 con 0x556c0a43db70 2024-05-27T07:29:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.731+0000 7f3ceea00700 10 client.17670 handle_command_reply: tid=0 2024-05-27T07:29:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.731+0000 7f3d1e139b80 1 client.17670 shutdown 2024-05-27T07:29:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.731+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3643674503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 msgr2=0x556c0a42a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.731+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a42a1d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4009ce0 tx=0x7f3cf40051d0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.732+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3643674503 shutdown_connections 2024-05-27T07:29:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.732+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556c0a3a7e70 0x556c0a3b9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.732+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556c0a228e40 0x556c0a429c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.732+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x556c0a43db70 0x556c0a43ff90 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.732+0000 7f3d1e139b80 1 --2- 172.21.15.43:0/3643674503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c0a3149f0 0x556c0a42a1d0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.732+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3643674503 >> 172.21.15.43:0/3643674503 conn(0x556c0a2c2390 msgr2=0x556c0a3a8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.732+0000 7f3ceea00700 0 client.17670 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.732+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3643674503 shutdown_connections 2024-05-27T07:29:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.732+0000 7f3d1e139b80 1 -- 172.21.15.43:0/3643674503 wait complete. 2024-05-27T07:29:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.733+0000 7f3d1e139b80 20 client.17670 trim_cache size 0 max 16384 2024-05-27T07:29:34.724 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:29:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.733+0000 7f3d0f600700 1 -- 172.21.15.43:0/1867367777 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3cf00b29f0 msgr2=0x7f3cf00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.734+0000 7f3d0f600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3cf00b29f0 0x7f3cf00b4f00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00bd50 tx=0x7f3cfc00b320 comp rx=0 tx=0).stop 2024-05-27T07:29:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.734+0000 7f3d0f600700 1 -- 172.21.15.43:0/1867367777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d0810e8d0 msgr2=0x7f3d081a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.734+0000 7f3d0f600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d0810e8d0 0x7f3d081a0830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000efb0 tx=0x7f3d0000c450 comp rx=0 tx=0).stop 2024-05-27T07:29:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.734+0000 7f3d0f600700 1 -- 172.21.15.43:0/1867367777 shutdown_connections 2024-05-27T07:29:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.734+0000 7f3d0f600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3cf00b29f0 0x7f3cf00b4f00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.734+0000 7f3d0f600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d0810e8d0 0x7f3d081a0830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.734+0000 7f3d0f600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d081054f0 0x7f3d081997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.734+0000 7f3d0f600700 1 --2- 172.21.15.43:0/1867367777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d08105e40 0x7f3d08199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.734+0000 7f3d0f600700 1 -- 172.21.15.43:0/1867367777 >> 172.21.15.43:0/1867367777 conn(0x7f3d080fedb0 msgr2=0x7f3d08103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.734+0000 7f3d0f600700 1 -- 172.21.15.43:0/1867367777 shutdown_connections 2024-05-27T07:29:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:34.735+0000 7f3d0f600700 1 -- 172.21.15.43:0/1867367777 wait complete. 2024-05-27T07:29:34.775 INFO:tasks.workunit.client.0.smithi043.stdout: 10 10754 7.95 MB/sec warmup 95 sec latency 85.418 ms 2024-05-27T07:29:34.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:34 smithi132 ceph-mon[35283]: pgmap v524: 129 pgs: 129 active+clean; 484 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-05-27T07:29:34.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:34 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:34.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:34 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:34.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:34 smithi179 ceph-mon[35888]: pgmap v524: 129 pgs: 129 active+clean; 484 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-05-27T07:29:34.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:34 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:34 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:34 smithi043 systemd[1]: Stopped Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:34 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.747s CPU time. 2024-05-27T07:29:35.452 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:34 smithi043 systemd[1]: Starting Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:29:35.772 INFO:tasks.workunit.client.0.smithi043.stdout: 10 10872 7.93 MB/sec warmup 96 sec latency 117.121 ms 2024-05-27T07:29:36.384 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:36 smithi043 podman[119502]: 2024-05-27 07:29:35.925948319 +0000 UTC m=+0.046196117 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:36.385 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:36 smithi043 podman[119502]: 2024-05-27 07:29:36.053676338 +0000 UTC m=+0.173924131 container create 21936dae9afc390b4f533cf1be34ce5d6e97a8f353df6683041cb829b161bbf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-05-27T07:29:36.653 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:29:36.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:29:36.702 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:36 smithi043 podman[119502]: 2024-05-27 07:29:36.394853736 +0000 UTC m=+0.515101535 container init 21936dae9afc390b4f533cf1be34ce5d6e97a8f353df6683041cb829b161bbf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-27T07:29:36.702 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:36 smithi043 podman[119502]: 2024-05-27 07:29:36.405309036 +0000 UTC m=+0.525556828 container start 21936dae9afc390b4f533cf1be34ce5d6e97a8f353df6683041cb829b161bbf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-27T07:29:36.702 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:36 smithi043 bash[119502]: 21936dae9afc390b4f533cf1be34ce5d6e97a8f353df6683041cb829b161bbf4 2024-05-27T07:29:36.702 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:36 smithi043 systemd[1]: Started Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:29:36.702 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:36 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: starting mds.a at 2024-05-27T07:29:36.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:36 smithi043 ceph-mon[31531]: pgmap v525: 129 pgs: 129 active+clean; 484 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-05-27T07:29:36.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 10981 7.88 MB/sec warmup 97 sec latency 97.448 ms 2024-05-27T07:29:36.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:36 smithi132 ceph-mon[35283]: pgmap v525: 129 pgs: 129 active+clean; 484 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-05-27T07:29:36.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:36 smithi179 ceph-mon[35888]: pgmap v525: 129 pgs: 129 active+clean; 484 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-05-27T07:29:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:37.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-mon[31531]: daemon mds.a restarted 2024-05-27T07:29:37.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-mon[31531]: pgmap v526: 129 pgs: 129 active+clean; 499 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.8 MiB/s wr, 208 op/s 2024-05-27T07:29:37.772 INFO:tasks.workunit.client.0.smithi043.stdout: 10 11115 7.93 MB/sec warmup 98 sec latency 116.957 ms 2024-05-27T07:29:37.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:37 smithi132 ceph-mon[35283]: daemon mds.a restarted 2024-05-27T07:29:37.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:37 smithi132 ceph-mon[35283]: pgmap v526: 129 pgs: 129 active+clean; 499 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.8 MiB/s wr, 208 op/s 2024-05-27T07:29:37.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:37 smithi179 ceph-mon[35888]: daemon mds.a restarted 2024-05-27T07:29:37.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:37 smithi179 ceph-mon[35888]: pgmap v526: 129 pgs: 129 active+clean; 499 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.8 MiB/s wr, 208 op/s 2024-05-27T07:29:38.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd809c00700 time 2024-05-27T07:29:37.749793+0000 2024-05-27T07:29:38.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:38.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd81f6e8ba6] 2024-05-27T07:29:38.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd81f6e8dd1] 2024-05-27T07:29:38.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a55af9123d] 2024-05-27T07:29:38.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a55af7c971] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a55af7afa0] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a55af7b651] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a55af10347] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a55ac0c87e] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a55ac11e94] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a55ac1c969] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a55afb548c] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a55afbe9cb] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 13: (Thread::entry_wrapper()+0x43) [0x7fd81f6b772d] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd81f6b7749] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd81e25b1ca] 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 16: clone() 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: *** Caught signal (Aborted) ** 2024-05-27T07:29:38.204 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: in thread 7fd809c00700 thread_name:md_log_replay 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:37.752+0000 7fd809c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd809c00700 time 2024-05-27T07:29:37.749793+0000 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd81f6e8ba6] 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd81f6e8dd1] 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a55af9123d] 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a55af7c971] 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a55af7afa0] 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a55af7b651] 2024-05-27T07:29:38.205 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a55af10347] 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a55ac0c87e] 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a55ac11e94] 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a55ac1c969] 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a55afb548c] 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a55afbe9cb] 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 13: (Thread::entry_wrapper()+0x43) [0x7fd81f6b772d] 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd81f6b7749] 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd81e25b1ca] 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 16: clone() 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd81e265d20] 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2: gsignal() 2024-05-27T07:29:38.206 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 3: abort() 2024-05-27T07:29:38.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd81f6e8ced] 2024-05-27T07:29:38.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd81f6e8dd1] 2024-05-27T07:29:38.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a55af9123d] 2024-05-27T07:29:38.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a55af7c971] 2024-05-27T07:29:38.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a55af7afa0] 2024-05-27T07:29:38.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a55af7b651] 2024-05-27T07:29:38.207 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a55af10347] 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a55ac0c87e] 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a55ac11e94] 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a55ac1c969] 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a55afb548c] 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a55afbe9cb] 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 16: (Thread::entry_wrapper()+0x43) [0x7fd81f6b772d] 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd81f6b7749] 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd81e25b1ca] 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 19: clone() 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:37.755+0000 7fd809c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: in thread 7fd809c00700 thread_name:md_log_replay 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.208 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd81e265d20] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2: gsignal() 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 3: abort() 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd81f6e8ced] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd81f6e8dd1] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a55af9123d] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a55af7c971] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a55af7afa0] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a55af7b651] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a55af10347] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a55ac0c87e] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a55ac11e94] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a55ac1c969] 2024-05-27T07:29:38.209 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a55afb548c] 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a55afbe9cb] 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 16: (Thread::entry_wrapper()+0x43) [0x7fd81f6b772d] 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd81f6b7749] 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd81e25b1ca] 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 19: clone() 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: -1> 2024-05-27T07:29:37.752+0000 7fd809c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd809c00700 time 2024-05-27T07:29:37.749793+0000 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd81f6e8ba6] 2024-05-27T07:29:38.210 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd81f6e8dd1] 2024-05-27T07:29:38.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a55af9123d] 2024-05-27T07:29:38.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a55af7c971] 2024-05-27T07:29:38.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a55af7afa0] 2024-05-27T07:29:38.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a55af7b651] 2024-05-27T07:29:38.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a55af10347] 2024-05-27T07:29:38.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a55ac0c87e] 2024-05-27T07:29:38.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a55ac11e94] 2024-05-27T07:29:38.211 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a55ac1c969] 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a55afb548c] 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a55afbe9cb] 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 13: (Thread::entry_wrapper()+0x43) [0x7fd81f6b772d] 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd81f6b7749] 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd81e25b1ca] 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 16: clone() 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 0> 2024-05-27T07:29:37.755+0000 7fd809c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: in thread 7fd809c00700 thread_name:md_log_replay 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd81e265d20] 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2: gsignal() 2024-05-27T07:29:38.212 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 3: abort() 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd81f6e8ced] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd81f6e8dd1] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a55af9123d] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a55af7c971] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a55af7afa0] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a55af7b651] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a55af10347] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a55ac0c87e] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a55ac11e94] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a55ac1c969] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a55afb548c] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a55afbe9cb] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 16: (Thread::entry_wrapper()+0x43) [0x7fd81f6b772d] 2024-05-27T07:29:38.213 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd81f6b7749] 2024-05-27T07:29:38.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd81e25b1ca] 2024-05-27T07:29:38.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 19: clone() 2024-05-27T07:29:38.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:38.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: -1> 2024-05-27T07:29:37.752+0000 7fd809c00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7fd809c00700 time 2024-05-27T07:29:37.749793+0000 2024-05-27T07:29:38.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:38.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.214 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd81f6e8ba6] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd81f6e8dd1] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a55af9123d] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 4: (SnapRealm::build_snap_set() const+0x323) [0x55a55af7c971] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 5: (SnapRealm::check_cache() const+0x13e) [0x55a55af7afa0] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a55af7b651] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a55af10347] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a55ac0c87e] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a55ac11e94] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x55a55ac1c969] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 11: (MDLog::_replay_thread()+0x1f38) [0x55a55afb548c] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 12: (MDLog::ReplayThread::entry()+0x11) [0x55a55afbe9cb] 2024-05-27T07:29:38.215 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 13: (Thread::entry_wrapper()+0x43) [0x7fd81f6b772d] 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 14: (Thread::_entry_func(void*)+0xd) [0x7fd81f6b7749] 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd81e25b1ca] 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 16: clone() 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 0> 2024-05-27T07:29:37.755+0000 7fd809c00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: in thread 7fd809c00700 thread_name:md_log_replay 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd81e265d20] 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2: gsignal() 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 3: abort() 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd81f6e8ced] 2024-05-27T07:29:38.216 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd81f6e8dd1] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x55a55af9123d] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 7: (SnapRealm::build_snap_set() const+0x323) [0x55a55af7c971] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 8: (SnapRealm::check_cache() const+0x13e) [0x55a55af7afa0] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x55a55af7b651] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x55a55af10347] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x55a55ac0c87e] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x55a55ac11e94] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x55a55ac1c969] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 14: (MDLog::_replay_thread()+0x1f38) [0x55a55afb548c] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 15: (MDLog::ReplayThread::entry()+0x11) [0x55a55afbe9cb] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 16: (Thread::entry_wrapper()+0x43) [0x7fd81f6b772d] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 17: (Thread::_entry_func(void*)+0xd) [0x7fd81f6b7749] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd81e25b1ca] 2024-05-27T07:29:38.217 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 19: clone() 2024-05-27T07:29:38.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:38.218 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[119517]: 2024-05-27T07:29:38.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:38 smithi043 ceph-mon[31531]: osdmap e140: 12 total, 12 up, 12 in 2024-05-27T07:29:38.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:38 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/1852588013,v1:172.21.15.43:6835/1852588013] up:boot 2024-05-27T07:29:38.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:38 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:29:38.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:38 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:38.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 11210 7.86 MB/sec warmup 99 sec latency 99.585 ms 2024-05-27T07:29:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:38 smithi132 ceph-mon[35283]: osdmap e140: 12 total, 12 up, 12 in 2024-05-27T07:29:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:38 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/1852588013,v1:172.21.15.43:6835/1852588013] up:boot 2024-05-27T07:29:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:38 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:29:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:38 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:38.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:38 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:38.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:38 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:38.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:38 smithi179 ceph-mon[35888]: osdmap e140: 12 total, 12 up, 12 in 2024-05-27T07:29:38.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:38 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/1852588013,v1:172.21.15.43:6835/1852588013] up:boot 2024-05-27T07:29:38.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:38 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:29:38.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:38 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:29:38.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:38 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:29:39.195 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:38 smithi043 podman[119632]: 2024-05-27 07:29:38.892752679 +0000 UTC m=+1.030567119 container died 21936dae9afc390b4f533cf1be34ce5d6e97a8f353df6683041cb829b161bbf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-05-27T07:29:39.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 11328 7.91 MB/sec warmup 100 sec latency 93.464 ms 2024-05-27T07:29:39.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:39 smithi132 ceph-mon[35283]: pgmap v528: 129 pgs: 129 active+clean; 499 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 4.1 MiB/s wr, 164 op/s 2024-05-27T07:29:39.895 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:39 smithi043 podman[119632]: 2024-05-27 07:29:39.703914032 +0000 UTC m=+1.841728437 container remove 21936dae9afc390b4f533cf1be34ce5d6e97a8f353df6683041cb829b161bbf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-05-27T07:29:39.895 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:39 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:29:39.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:39 smithi043 ceph-mon[31531]: pgmap v528: 129 pgs: 129 active+clean; 499 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 4.1 MiB/s wr, 164 op/s 2024-05-27T07:29:39.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:39 smithi179 ceph-mon[35888]: pgmap v528: 129 pgs: 129 active+clean; 499 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 4.1 MiB/s wr, 164 op/s 2024-05-27T07:29:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.540+0000 7f42c0400700 1 -- 172.21.15.43:0/3493015148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42b8103240 msgr2=0x7f42b8103650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.540+0000 7f42c0400700 1 --2- 172.21.15.43:0/3493015148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42b8103240 0x7f42b8103650 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00b0a0 tx=0x7f42ac00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.540+0000 7f42c0400700 1 -- 172.21.15.43:0/3493015148 shutdown_connections 2024-05-27T07:29:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.540+0000 7f42c0400700 1 --2- 172.21.15.43:0/3493015148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42b8108690 0x7f42b8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.540+0000 7f42c0400700 1 --2- 172.21.15.43:0/3493015148 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42b8103240 0x7f42b8103650 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.540+0000 7f42c0400700 1 --2- 172.21.15.43:0/3493015148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b8103b90 0x7f42b81080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.540+0000 7f42c0400700 1 -- 172.21.15.43:0/3493015148 >> 172.21.15.43:0/3493015148 conn(0x7f42b80fedb0 msgr2=0x7f42b81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.541+0000 7f42c0400700 1 -- 172.21.15.43:0/3493015148 shutdown_connections 2024-05-27T07:29:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.541+0000 7f42c0400700 1 -- 172.21.15.43:0/3493015148 wait complete. 2024-05-27T07:29:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.542+0000 7f42c0400700 1 Processor -- start 2024-05-27T07:29:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.543+0000 7f42c0400700 1 -- start start 2024-05-27T07:29:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.543+0000 7f42c0400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42b8103240 0x7f42b8075230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.543+0000 7f42c0400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42b8103b90 0x7f42b806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.543+0000 7f42c0400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b8108690 0x7f42b806d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.543+0000 7f42c0400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f42b811a910 con 0x7f42b8103240 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.543+0000 7f42c0400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f42b811a790 con 0x7f42b8103b90 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.543+0000 7f42c0400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f42b811aa90 con 0x7f42b8108690 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.544+0000 7f42bda00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42b8103240 0x7f42b8075230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.544+0000 7f42bda00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42b8103240 0x7f42b8075230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59872/0 (socket says 172.21.15.43:59872) 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.544+0000 7f42bda00700 1 -- 172.21.15.43:0/2297497644 learned_addr learned my addr 172.21.15.43:0/2297497644 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.544+0000 7f42bd000700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42b8103b90 0x7f42b806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.544+0000 7f42be400700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b8108690 0x7f42b806d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.544+0000 7f42bda00700 1 -- 172.21.15.43:0/2297497644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b8108690 msgr2=0x7f42b806d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.544+0000 7f42bda00700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b8108690 0x7f42b806d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.544+0000 7f42bda00700 1 -- 172.21.15.43:0/2297497644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42b8103b90 msgr2=0x7f42b806d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.544+0000 7f42bda00700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42b8103b90 0x7f42b806d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.544+0000 7f42bda00700 1 -- 172.21.15.43:0/2297497644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b806de50 con 0x7f42b8103240 2024-05-27T07:29:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.545+0000 7f42bda00700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42b8103240 0x7f42b8075230 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00b070 tx=0x7f42ac004e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.546+0000 7f42b6000700 1 -- 172.21.15.43:0/2297497644 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac021070 con 0x7f42b8103240 2024-05-27T07:29:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.546+0000 7f42b6000700 1 -- 172.21.15.43:0/2297497644 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42ac007c80 con 0x7f42b8103240 2024-05-27T07:29:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.546+0000 7f42c0400700 1 -- 172.21.15.43:0/2297497644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42b806e0e0 con 0x7f42b8103240 2024-05-27T07:29:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.546+0000 7f42b6000700 1 -- 172.21.15.43:0/2297497644 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac0047d0 con 0x7f42b8103240 2024-05-27T07:29:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.546+0000 7f42c0400700 1 -- 172.21.15.43:0/2297497644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42b81b63b0 con 0x7f42b8103240 2024-05-27T07:29:40.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.552+0000 7f42ceef3b80 1 Processor -- start 2024-05-27T07:29:40.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.552+0000 7f42ceef3b80 1 -- start start 2024-05-27T07:29:40.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.552+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.552+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02ab7ec0 0x55cf02c368f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.552+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02c36ef0 0x55cf02c43190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.552+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cf02b64ec0 con 0x55cf02ba3a70 2024-05-27T07:29:40.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.552+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cf02b54a60 con 0x55cf02ab7ec0 2024-05-27T07:29:40.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.553+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55cf02b58aa0 con 0x55cf02c36ef0 2024-05-27T07:29:40.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.553+0000 7f42bd000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02c36ef0 0x55cf02c43190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.553+0000 7f42be400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.553+0000 7f42be400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c363b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59886/0 (socket says 172.21.15.43:59886) 2024-05-27T07:29:40.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.553+0000 7f42be400700 1 -- 172.21.15.43:0/3447493474 learned_addr learned my addr 172.21.15.43:0/3447493474 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:40.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.554+0000 7f42b6000700 1 -- 172.21.15.43:0/2297497644 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f42ac003a00 con 0x7f42b8103240 2024-05-27T07:29:40.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.554+0000 7f42be400700 1 -- 172.21.15.43:0/3447493474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02c36ef0 msgr2=0x55cf02c43190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.554+0000 7f42be400700 1 --2- 172.21.15.43:0/3447493474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02c36ef0 0x55cf02c43190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.554+0000 7f42be400700 1 -- 172.21.15.43:0/3447493474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02ab7ec0 msgr2=0x55cf02c368f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.554+0000 7f42be400700 1 --2- 172.21.15.43:0/3447493474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02ab7ec0 0x55cf02c368f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.554+0000 7f42be400700 1 -- 172.21.15.43:0/3447493474 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf02b1d4a0 con 0x55cf02ba3a70 2024-05-27T07:29:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.555+0000 7f42be400700 1 --2- 172.21.15.43:0/3447493474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c363b0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f42b000cc00 tx=0x7f42b000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42548c087019dcf2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.555+0000 7f42b6000700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42a00b2a70 0x7f42a00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.556+0000 7f42aaa00700 1 -- 172.21.15.43:0/3447493474 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b000ecf0 con 0x55cf02ba3a70 2024-05-27T07:29:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.556+0000 7f42aaa00700 1 -- 172.21.15.43:0/3447493474 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42b000ee90 con 0x55cf02ba3a70 2024-05-27T07:29:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.556+0000 7f42aaa00700 1 -- 172.21.15.43:0/3447493474 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0010430 con 0x55cf02ba3a70 2024-05-27T07:29:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.556+0000 7f42b6000700 1 -- 172.21.15.43:0/2297497644 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 9408+0+0 (secure 0 0 0) 0x7f42ac00f020 con 0x7f42b8103240 2024-05-27T07:29:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.556+0000 7f42bda00700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42a00b2a70 0x7f42a00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.557+0000 7f42ceef3b80 1 -- 172.21.15.43:0/3447493474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 msgr2=0x55cf02c363b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.557+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/3447493474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c363b0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f42b000cc00 tx=0x7f42b000cf10 comp rx=0 tx=0).stop 2024-05-27T07:29:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.557+0000 7f42ceef3b80 1 -- 172.21.15.43:0/3447493474 shutdown_connections 2024-05-27T07:29:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.557+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/3447493474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02c36ef0 0x55cf02c43190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.557+0000 7f42bda00700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42a00b2a70 0x7f42a00b4f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x55cf02c33aa0 tx=0x7f42ac00b3b0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.557+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/3447493474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c363b0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.557+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/3447493474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02ab7ec0 0x55cf02c368f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.557+0000 7f42ceef3b80 1 -- 172.21.15.43:0/3447493474 >> 172.21.15.43:0/3447493474 conn(0x55cf02b51410 msgr2=0x55cf02c32540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.557+0000 7f42ceef3b80 1 -- 172.21.15.43:0/3447493474 shutdown_connections 2024-05-27T07:29:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.557+0000 7f42ceef3b80 1 -- 172.21.15.43:0/3447493474 wait complete. 2024-05-27T07:29:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.558+0000 7f42ceef3b80 1 -- start start 2024-05-27T07:29:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.558+0000 7f42ceef3b80 1 Processor -- start 2024-05-27T07:29:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.559+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02ab7ec0 0x55cf02c407e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.559+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ba3a70 0x55cf02c40d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.559+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c36ef0 0x55cf02c41260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.559+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cf02a5b020 con 0x55cf02c36ef0 2024-05-27T07:29:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.559+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cf02af57e0 con 0x55cf02ab7ec0 2024-05-27T07:29:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.559+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55cf02c54c10 con 0x55cf02ba3a70 2024-05-27T07:29:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.559+0000 7f42ceef3b80 10 client.0 _ll_register_callbacks cb 0x55cf02afecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:40.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.559+0000 7f42be400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c36ef0 0x55cf02c41260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.560+0000 7f42be400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c36ef0 0x55cf02c41260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59892/0 (socket says 172.21.15.43:59892) 2024-05-27T07:29:40.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.560+0000 7f42be400700 1 -- 172.21.15.43:0/4123659176 learned_addr learned my addr 172.21.15.43:0/4123659176 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:40.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.560+0000 7f42bd000700 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ba3a70 0x55cf02c40d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.560+0000 7f42be400700 1 -- 172.21.15.43:0/4123659176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ba3a70 msgr2=0x55cf02c40d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.560+0000 7f42be400700 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ba3a70 0x55cf02c40d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.560+0000 7f42be400700 1 -- 172.21.15.43:0/4123659176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02ab7ec0 msgr2=0x55cf02c407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.560+0000 7f42be400700 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02ab7ec0 0x55cf02c407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.560+0000 7f42be400700 1 -- 172.21.15.43:0/4123659176 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf02aa4950 con 0x55cf02c36ef0 2024-05-27T07:29:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.560+0000 7f42bd000700 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ba3a70 0x55cf02c40d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.561+0000 7f42be400700 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c36ef0 0x55cf02c41260 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f42b00059d0 tx=0x7f42b0005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c8ca7055a375a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.561+0000 7f429f400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:29:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.561+0000 7f429f400700 1 -- 172.21.15.43:0/4123659176 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0005290 con 0x55cf02c36ef0 2024-05-27T07:29:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.561+0000 7f429f400700 1 -- 172.21.15.43:0/4123659176 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 9408+0+0 (secure 0 0 0) 0x7f42b0004030 con 0x55cf02c36ef0 2024-05-27T07:29:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.561+0000 7f42ceef3b80 1 -- 172.21.15.43:0/4123659176 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42b000ee90 con 0x55cf02c36ef0 2024-05-27T07:29:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.562+0000 7f429f400700 1 -- 172.21.15.43:0/4123659176 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=123) ==== 24+0+0 (secure 0 0 0) 0x7f42b0004230 con 0x55cf02c36ef0 2024-05-27T07:29:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.562+0000 7f42ceef3b80 10 client.17688 fetch_fsmap learned FSMap version 123 2024-05-27T07:29:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.562+0000 7f42ceef3b80 1 -- 172.21.15.43:0/4123659176 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=123,monmap=4+}) -- 0x55cf02b250e0 con 0x55cf02c36ef0 2024-05-27T07:29:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.562+0000 7f429f400700 1 -- 172.21.15.43:0/4123659176 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42b000ee90 con 0x55cf02c36ef0 2024-05-27T07:29:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.562+0000 7f429f400700 1 -- 172.21.15.43:0/4123659176 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 123) ==== 4816+0+0 (secure 0 0 0) 0x7f42b00038b0 con 0x55cf02c36ef0 2024-05-27T07:29:40.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.563+0000 7f42ceef3b80 10 client.17688 fetch_fsmap finished waiting for FSMap version 123 2024-05-27T07:29:40.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.563+0000 7f42ceef3b80 10 client.17688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:40.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.563+0000 7f42ceef3b80 20 client.17688 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:40.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.563+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cf02c61f20 0x55cf02c643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.563+0000 7f42ceef3b80 4 client.17688 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:40.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.563+0000 7f42ceef3b80 1 -- 172.21.15.43:0/4123659176 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf02b66e10 con 0x55cf02c61f20 2024-05-27T07:29:40.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 11411 7.87 MB/sec warmup 101 sec latency 290.939 ms 2024-05-27T07:29:40.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.839+0000 7f42bd000700 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cf02c61f20 0x55cf02c643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:40 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:29:40.830 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:40 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.743s CPU time. 2024-05-27T07:29:40.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.843+0000 7f42bd000700 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cf02c61f20 0x55cf02c643b0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc6fcc864dfd12ec server_cookie=6ae5ed4a8ee56cee in_seq=0 out_seq=0 2024-05-27T07:29:40.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.843+0000 7f429f400700 10 client.17688 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:40.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.853+0000 7f429f400700 1 -- 172.21.15.43:0/4123659176 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55cf02b66e10 con 0x55cf02c61f20 2024-05-27T07:29:40.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.853+0000 7f429f400700 10 client.17688 handle_command_reply: tid=0 2024-05-27T07:29:40.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.853+0000 7f42ceef3b80 1 client.17688 shutdown 2024-05-27T07:29:40.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.854+0000 7f42ceef3b80 1 -- 172.21.15.43:0/4123659176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c36ef0 msgr2=0x55cf02c41260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.854+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c36ef0 0x55cf02c41260 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f42b00059d0 tx=0x7f42b0005690 comp rx=0 tx=0).stop 2024-05-27T07:29:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.854+0000 7f42ceef3b80 1 -- 172.21.15.43:0/4123659176 shutdown_connections 2024-05-27T07:29:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.854+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cf02c61f20 0x55cf02c643b0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.854+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c36ef0 0x55cf02c41260 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.854+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02ab7ec0 0x55cf02c407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.854+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/4123659176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ba3a70 0x55cf02c40d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.854+0000 7f42ceef3b80 1 -- 172.21.15.43:0/4123659176 >> 172.21.15.43:0/4123659176 conn(0x55cf02b51410 msgr2=0x55cf02c364d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.854+0000 7f429f400700 0 client.17688 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:40.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.855+0000 7f42ceef3b80 1 -- 172.21.15.43:0/4123659176 shutdown_connections 2024-05-27T07:29:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.855+0000 7f42ceef3b80 1 -- 172.21.15.43:0/4123659176 wait complete. 2024-05-27T07:29:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.856+0000 7f42ceef3b80 20 client.17688 trim_cache size 0 max 16384 2024-05-27T07:29:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.911+0000 7f42ceef3b80 1 Processor -- start 2024-05-27T07:29:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.912+0000 7f42ceef3b80 1 -- start start 2024-05-27T07:29:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.912+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ab7ec0 0x55cf02c3bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.912+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c32570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.912+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 0x55cf02c40cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.912+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cf02c38230 con 0x55cf02ba3a70 2024-05-27T07:29:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.912+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81b76d0 con 0x55cf02c36ef0 2024-05-27T07:29:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.912+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f42b8112310 con 0x55cf02ab7ec0 2024-05-27T07:29:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42bd000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c32570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42bd000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c32570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59896/0 (socket says 172.21.15.43:59896) 2024-05-27T07:29:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42bd000700 1 -- 172.21.15.43:0/2752137615 learned_addr learned my addr 172.21.15.43:0/2752137615 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42be400700 1 --2- 172.21.15.43:0/2752137615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ab7ec0 0x55cf02c3bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42be400700 1 --2- 172.21.15.43:0/2752137615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 0x55cf02c40cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42bd000700 1 -- 172.21.15.43:0/2752137615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ab7ec0 msgr2=0x55cf02c3bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42bd000700 1 --2- 172.21.15.43:0/2752137615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ab7ec0 0x55cf02c3bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42bd000700 1 -- 172.21.15.43:0/2752137615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 msgr2=0x55cf02c40cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42bd000700 1 --2- 172.21.15.43:0/2752137615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 0x55cf02c40cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42bd000700 1 -- 172.21.15.43:0/2752137615 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf02c338d0 con 0x55cf02ba3a70 2024-05-27T07:29:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.913+0000 7f42be400700 1 --2- 172.21.15.43:0/2752137615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ab7ec0 0x55cf02c3bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.914+0000 7f42bd000700 1 --2- 172.21.15.43:0/2752137615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c32570 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f42a4000c00 tx=0x7f42a4011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3dc250704a1bcf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.915+0000 7f42aa000700 1 -- 172.21.15.43:0/2752137615 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a4022070 con 0x55cf02ba3a70 2024-05-27T07:29:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.915+0000 7f42aa000700 1 -- 172.21.15.43:0/2752137615 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42a4003680 con 0x55cf02ba3a70 2024-05-27T07:29:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.915+0000 7f42aa000700 1 -- 172.21.15.43:0/2752137615 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a401d690 con 0x55cf02ba3a70 2024-05-27T07:29:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.916+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2752137615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 msgr2=0x55cf02c32570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.916+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/2752137615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c32570 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f42a4000c00 tx=0x7f42a4011820 comp rx=0 tx=0).stop 2024-05-27T07:29:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.917+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2752137615 shutdown_connections 2024-05-27T07:29:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.917+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/2752137615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 0x55cf02c40cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.917+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/2752137615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ab7ec0 0x55cf02c3bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.917+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/2752137615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02ba3a70 0x55cf02c32570 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.917+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2752137615 >> 172.21.15.43:0/2752137615 conn(0x55cf02b51410 msgr2=0x55cf02c405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.917+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2752137615 shutdown_connections 2024-05-27T07:29:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.917+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2752137615 wait complete. 2024-05-27T07:29:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.917+0000 7f42ceef3b80 1 -- start start 2024-05-27T07:29:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.918+0000 7f42ceef3b80 1 Processor -- start 2024-05-27T07:29:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.919+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ab7ec0 0x55cf02c3a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.919+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 0x55cf02c3ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.919+0000 7f42ceef3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c3b400 0x55cf02c58ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.919+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f42b8112a40 con 0x55cf02c3b400 2024-05-27T07:29:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.919+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81b6fd0 con 0x55cf02c36ef0 2024-05-27T07:29:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.919+0000 7f42ceef3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81b6840 con 0x55cf02ab7ec0 2024-05-27T07:29:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.919+0000 7f42ceef3b80 10 client.0 _ll_register_callbacks cb 0x55cf02afecf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.919+0000 7f42bd000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 0x55cf02c3ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.920+0000 7f42bd000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 0x55cf02c3ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:44634/0 (socket says 172.21.15.43:44634) 2024-05-27T07:29:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.920+0000 7f42bd000700 1 -- 172.21.15.43:0/2137791559 learned_addr learned my addr 172.21.15.43:0/2137791559 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.920+0000 7f42be400700 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c3b400 0x55cf02c58ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.920+0000 7f42bd000700 1 -- 172.21.15.43:0/2137791559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ab7ec0 msgr2=0x55cf02c3a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.920+0000 7f42bd000700 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ab7ec0 0x55cf02c3a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.920+0000 7f42bd000700 1 -- 172.21.15.43:0/2137791559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c3b400 msgr2=0x55cf02c58ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.920+0000 7f42bd000700 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c3b400 0x55cf02c58ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.920+0000 7f42bd000700 1 -- 172.21.15.43:0/2137791559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf02c2e2c0 con 0x55cf02c36ef0 2024-05-27T07:29:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.921+0000 7f42be400700 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c3b400 0x55cf02c58ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.921+0000 7f42bd000700 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 0x55cf02c3ae20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55cf02b61140 tx=0x7f42a4003fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc62d79848e8ce22 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.921+0000 7f429f400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:29:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.922+0000 7f429f400700 1 -- 172.21.15.43:0/2137791559 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a402b020 con 0x55cf02c36ef0 2024-05-27T07:29:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.922+0000 7f429f400700 1 -- 172.21.15.43:0/2137791559 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 9408+0+0 (secure 0 0 0) 0x7f42a4003830 con 0x55cf02c36ef0 2024-05-27T07:29:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.922+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2137791559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42a4003680 con 0x55cf02c36ef0 2024-05-27T07:29:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.924+0000 7f429f400700 1 -- 172.21.15.43:0/2137791559 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=123) ==== 24+0+0 (secure 0 0 0) 0x7f42a4022040 con 0x55cf02c36ef0 2024-05-27T07:29:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.924+0000 7f42ceef3b80 10 client.37082 fetch_fsmap learned FSMap version 123 2024-05-27T07:29:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.924+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2137791559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=123,monmap=4+}) -- 0x55cf02c5a5d0 con 0x55cf02c36ef0 2024-05-27T07:29:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.925+0000 7f429f400700 1 -- 172.21.15.43:0/2137791559 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f42a4003680 con 0x55cf02c36ef0 2024-05-27T07:29:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.925+0000 7f429f400700 1 -- 172.21.15.43:0/2137791559 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 123) ==== 4816+0+0 (secure 0 0 0) 0x7f42a4038080 con 0x55cf02c36ef0 2024-05-27T07:29:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.925+0000 7f42ceef3b80 10 client.37082 fetch_fsmap finished waiting for FSMap version 123 2024-05-27T07:29:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.926+0000 7f42ceef3b80 10 client.37082 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.926+0000 7f42ceef3b80 20 client.37082 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.926+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cf02ccca90 0x55cf02cceeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.926+0000 7f42ceef3b80 4 client.37082 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:29:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.926+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2137791559 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cf02b605f0 con 0x55cf02ccca90 2024-05-27T07:29:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.926+0000 7f42be400700 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cf02ccca90 0x55cf02cceeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.928+0000 7f42be400700 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cf02ccca90 0x55cf02cceeb0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1a529223dccfcd3 server_cookie=b1295a58df326ba4 in_seq=0 out_seq=0 2024-05-27T07:29:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.928+0000 7f429f400700 10 client.37082 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.932+0000 7f429f400700 1 -- 172.21.15.43:0/2137791559 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cf02b605f0 con 0x55cf02ccca90 2024-05-27T07:29:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.932+0000 7f429f400700 10 client.37082 handle_command_reply: tid=0 2024-05-27T07:29:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.932+0000 7f42ceef3b80 1 client.37082 shutdown 2024-05-27T07:29:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.932+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2137791559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 msgr2=0x55cf02c3ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:40.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.932+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 0x55cf02c3ae20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55cf02b61140 tx=0x7f42a4003fb0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.933+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2137791559 shutdown_connections 2024-05-27T07:29:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.933+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55cf02ccca90 0x55cf02cceeb0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.933+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf02c3b400 0x55cf02c58ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.933+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cf02ab7ec0 0x55cf02c3a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.933+0000 7f42ceef3b80 1 --2- 172.21.15.43:0/2137791559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cf02c36ef0 0x55cf02c3ae20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.933+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2137791559 >> 172.21.15.43:0/2137791559 conn(0x55cf02b51410 msgr2=0x55cf02c40ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.933+0000 7f429f400700 0 client.37082 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.933+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2137791559 shutdown_connections 2024-05-27T07:29:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.933+0000 7f42ceef3b80 1 -- 172.21.15.43:0/2137791559 wait complete. 2024-05-27T07:29:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.934+0000 7f42ceef3b80 20 client.37082 trim_cache size 0 max 16384 2024-05-27T07:29:40.925 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:29:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:29:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "c6af25f3-304f-4fe9-afab-8c56cecaa67c", 2024-05-27T07:29:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:29:40.925 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:29:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.935+0000 7f42c0400700 1 -- 172.21.15.43:0/2297497644 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42a00b2a70 msgr2=0x7f42a00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.935+0000 7f42c0400700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42a00b2a70 0x7f42a00b4f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x55cf02c33aa0 tx=0x7f42ac00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.935+0000 7f42c0400700 1 -- 172.21.15.43:0/2297497644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42b8103240 msgr2=0x7f42b8075230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.935+0000 7f42c0400700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42b8103240 0x7f42b8075230 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00b070 tx=0x7f42ac004e00 comp rx=0 tx=0).stop 2024-05-27T07:29:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.935+0000 7f42c0400700 1 -- 172.21.15.43:0/2297497644 shutdown_connections 2024-05-27T07:29:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.935+0000 7f42c0400700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f42a00b2a70 0x7f42a00b4f80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.935+0000 7f42c0400700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b8108690 0x7f42b806d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.935+0000 7f42c0400700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42b8103240 0x7f42b8075230 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.935+0000 7f42c0400700 1 --2- 172.21.15.43:0/2297497644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42b8103b90 0x7f42b806d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.935+0000 7f42c0400700 1 -- 172.21.15.43:0/2297497644 >> 172.21.15.43:0/2297497644 conn(0x7f42b80fedb0 msgr2=0x7f42b8106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.936+0000 7f42c0400700 1 -- 172.21.15.43:0/2297497644 shutdown_connections 2024-05-27T07:29:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:40.936+0000 7f42c0400700 1 -- 172.21.15.43:0/2297497644 wait complete. 2024-05-27T07:29:41.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:41 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:29:41.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:41 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:41.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:41 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:29:41.458 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:41 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:41.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:41 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:29:41.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:41 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:41.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 11512 7.83 MB/sec warmup 102 sec latency 398.748 ms 2024-05-27T07:29:41.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:29:42.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:29:42.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:29:42.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:29:42.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:29:42.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:29:42.126 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.127 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:42 smithi043 ceph-mon[31531]: pgmap v529: 129 pgs: 129 active+clean; 493 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-27T07:29:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:29:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:29:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:29:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:29:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:29:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.460 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:42 smithi179 ceph-mon[35888]: pgmap v529: 129 pgs: 129 active+clean; 493 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-27T07:29:42.482 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:29:42.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:29:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:29:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:29:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:29:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:29:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:29:42.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:42 smithi132 ceph-mon[35283]: pgmap v529: 129 pgs: 129 active+clean; 493 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-27T07:29:42.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 11622 7.81 MB/sec warmup 103 sec latency 92.613 ms 2024-05-27T07:29:43.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 11739 7.77 MB/sec warmup 104 sec latency 111.774 ms 2024-05-27T07:29:44.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 11853 7.73 MB/sec warmup 105 sec latency 86.465 ms 2024-05-27T07:29:44.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:44 smithi132 ceph-mon[35283]: pgmap v530: 129 pgs: 129 active+clean; 493 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-27T07:29:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.855+0000 7f2016000700 1 -- 172.21.15.43:0/2843784768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20101034b0 msgr2=0x7f20101038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.855+0000 7f2016000700 1 --2- 172.21.15.43:0/2843784768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20101034b0 0x7f20101038c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f200000b0d0 tx=0x7f200000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:44 smithi043 ceph-mon[31531]: pgmap v530: 129 pgs: 129 active+clean; 493 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-27T07:29:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.856+0000 7f2016000700 1 -- 172.21.15.43:0/2843784768 shutdown_connections 2024-05-27T07:29:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.856+0000 7f2016000700 1 --2- 172.21.15.43:0/2843784768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2010108730 0x7f2010110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.856+0000 7f2016000700 1 --2- 172.21.15.43:0/2843784768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20101034b0 0x7f20101038c0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.856+0000 7f2016000700 1 --2- 172.21.15.43:0/2843784768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2010107de0 0x7f20101081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.856+0000 7f2016000700 1 -- 172.21.15.43:0/2843784768 >> 172.21.15.43:0/2843784768 conn(0x7f20100fedb0 msgr2=0x7f20101011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.857+0000 7f2016000700 1 -- 172.21.15.43:0/2843784768 shutdown_connections 2024-05-27T07:29:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.857+0000 7f2016000700 1 -- 172.21.15.43:0/2843784768 wait complete. 2024-05-27T07:29:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.859+0000 7f2016000700 1 Processor -- start 2024-05-27T07:29:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.859+0000 7f2016000700 1 -- start start 2024-05-27T07:29:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.860+0000 7f2016000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20101034b0 0x7f201019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.860+0000 7f2016000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2010107de0 0x7f201019d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.860+0000 7f2016000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2010108730 0x7f20101a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.860+0000 7f2016000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2010112150 con 0x7f20101034b0 2024-05-27T07:29:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.860+0000 7f2016000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2010111fd0 con 0x7f2010107de0 2024-05-27T07:29:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.860+0000 7f2016000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f20101122d0 con 0x7f2010108730 2024-05-27T07:29:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.860+0000 7f200f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20101034b0 0x7f201019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.860+0000 7f200ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2010107de0 0x7f201019d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.860+0000 7f200ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2010107de0 0x7f201019d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58768/0 (socket says 172.21.15.43:58768) 2024-05-27T07:29:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.860+0000 7f200ea00700 1 -- 172.21.15.43:0/3459922258 learned_addr learned my addr 172.21.15.43:0/3459922258 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.861+0000 7f200fe00700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2010108730 0x7f20101a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.861+0000 7f200ea00700 1 -- 172.21.15.43:0/3459922258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2010108730 msgr2=0x7f20101a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.861+0000 7f200ea00700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2010108730 0x7f20101a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.861+0000 7f200ea00700 1 -- 172.21.15.43:0/3459922258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20101034b0 msgr2=0x7f201019cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.861+0000 7f200ea00700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20101034b0 0x7f201019cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.861+0000 7f200ea00700 1 -- 172.21.15.43:0/3459922258 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20101a1e80 con 0x7f2010107de0 2024-05-27T07:29:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.861+0000 7f200f400700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20101034b0 0x7f201019cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.862+0000 7f200fe00700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2010108730 0x7f20101a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:44.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.864+0000 7f200ea00700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2010107de0 0x7f201019d550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00e970 tx=0x7f1ffc00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.864+0000 7f2007e00700 1 -- 172.21.15.43:0/3459922258 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc00cbd0 con 0x7f2010107de0 2024-05-27T07:29:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.864+0000 7f2007e00700 1 -- 172.21.15.43:0/3459922258 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ffc00cd70 con 0x7f2010107de0 2024-05-27T07:29:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.864+0000 7f2007e00700 1 -- 172.21.15.43:0/3459922258 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc005610 con 0x7f2010107de0 2024-05-27T07:29:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.865+0000 7f2016000700 1 -- 172.21.15.43:0/3459922258 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20101a2170 con 0x7f2010107de0 2024-05-27T07:29:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.865+0000 7f2016000700 1 -- 172.21.15.43:0/3459922258 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20101a9a90 con 0x7f2010107de0 2024-05-27T07:29:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.871+0000 7f2024be1b80 1 Processor -- start 2024-05-27T07:29:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.871+0000 7f2024be1b80 1 -- start start 2024-05-27T07:29:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.871+0000 7f2007e00700 1 -- 172.21.15.43:0/3459922258 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1ffc004750 con 0x7f2010107de0 2024-05-27T07:29:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.872+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbacb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.872+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.872+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 0x5561ffbb9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.872+0000 7f2024be1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5561ffad2e40 con 0x5561ffb119f0 2024-05-27T07:29:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.872+0000 7f2024be1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5561ffac29e0 con 0x5561ffa25e40 2024-05-27T07:29:44.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.872+0000 7f2024be1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5561ffac6a20 con 0x5561ffbad690 2024-05-27T07:29:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.873+0000 7f200ea00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 0x5561ffbb9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.873+0000 7f200fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbacb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.873+0000 7f200fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbacb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60906/0 (socket says 172.21.15.43:60906) 2024-05-27T07:29:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.873+0000 7f200ea00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 0x5561ffbb9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:52410/0 (socket says 172.21.15.43:52410) 2024-05-27T07:29:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.873+0000 7f200fe00700 1 -- 172.21.15.43:0/3195411608 learned_addr learned my addr 172.21.15.43:0/3195411608 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:44.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.873+0000 7f200f400700 1 --2- 172.21.15.43:0/3195411608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.874+0000 7f200fe00700 1 -- 172.21.15.43:0/3195411608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 msgr2=0x5561ffbb9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.874+0000 7f200fe00700 1 --2- 172.21.15.43:0/3195411608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 0x5561ffbb9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.874+0000 7f200fe00700 1 -- 172.21.15.43:0/3195411608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 msgr2=0x5561ffbad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.874+0000 7f200fe00700 1 --2- 172.21.15.43:0/3195411608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.874+0000 7f200fe00700 1 -- 172.21.15.43:0/3195411608 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561ffa8b420 con 0x5561ffb119f0 2024-05-27T07:29:44.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.874+0000 7f200f400700 1 --2- 172.21.15.43:0/3195411608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:29:44.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.874+0000 7f2007e00700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1ff40b2a70 0x7f1ff40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.875+0000 7f2007e00700 1 -- 172.21.15.43:0/3459922258 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 9408+0+0 (secure 0 0 0) 0x7f1ffc0e2c80 con 0x7f2010107de0 2024-05-27T07:29:44.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.875+0000 7f200f400700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1ff40b2a70 0x7f1ff40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.876+0000 7f200fe00700 1 --2- 172.21.15.43:0/3195411608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbacb50 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f2008009c20 tx=0x7f2008009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6b3ef4163ad640 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.876+0000 7f2004c00700 1 -- 172.21.15.43:0/3195411608 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008016070 con 0x5561ffb119f0 2024-05-27T07:29:44.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.876+0000 7f2004c00700 1 -- 172.21.15.43:0/3195411608 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2008011470 con 0x5561ffb119f0 2024-05-27T07:29:44.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.876+0000 7f2004c00700 1 -- 172.21.15.43:0/3195411608 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008010630 con 0x5561ffb119f0 2024-05-27T07:29:44.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.876+0000 7f200f400700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1ff40b2a70 0x7f1ff40b4f80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x5561ffba17a0 tx=0x7f200000b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.878+0000 7f2024be1b80 1 -- 172.21.15.43:0/3195411608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 msgr2=0x5561ffbacb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.878+0000 7f2024be1b80 1 --2- 172.21.15.43:0/3195411608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbacb50 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f2008009c20 tx=0x7f2008009f30 comp rx=0 tx=0).stop 2024-05-27T07:29:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.879+0000 7f2024be1b80 1 -- 172.21.15.43:0/3195411608 shutdown_connections 2024-05-27T07:29:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.879+0000 7f2024be1b80 1 --2- 172.21.15.43:0/3195411608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 0x5561ffbb9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.879+0000 7f2024be1b80 1 --2- 172.21.15.43:0/3195411608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbacb50 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.879+0000 7f2024be1b80 1 --2- 172.21.15.43:0/3195411608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.879+0000 7f2024be1b80 1 -- 172.21.15.43:0/3195411608 >> 172.21.15.43:0/3195411608 conn(0x5561ffabf390 msgr2=0x5561ffba0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.879+0000 7f2024be1b80 1 -- 172.21.15.43:0/3195411608 shutdown_connections 2024-05-27T07:29:44.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.879+0000 7f2024be1b80 1 -- 172.21.15.43:0/3195411608 wait complete. 2024-05-27T07:29:44.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.880+0000 7f2024be1b80 1 -- start start 2024-05-27T07:29:44.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.880+0000 7f2024be1b80 1 Processor -- start 2024-05-27T07:29:44.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.881+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbb6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.881+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbb74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.882+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 0x5561ffbb7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.882+0000 7f2024be1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5561ff9c8fa0 con 0x5561ffb119f0 2024-05-27T07:29:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.882+0000 7f2024be1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5561ffa63760 con 0x5561ffa25e40 2024-05-27T07:29:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.882+0000 7f2024be1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5561ffbba3a0 con 0x5561ffbad690 2024-05-27T07:29:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.882+0000 7f2024be1b80 10 client.0 _ll_register_callbacks cb 0x5561ffa6cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.882+0000 7f200ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbb74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.883+0000 7f200f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbb6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.883+0000 7f200f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbb6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58776/0 (socket says 172.21.15.43:58776) 2024-05-27T07:29:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.883+0000 7f200fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 0x5561ffbb7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.883+0000 7f200f400700 1 -- 172.21.15.43:0/1272027544 learned_addr learned my addr 172.21.15.43:0/1272027544 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:44.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.883+0000 7f200f400700 1 -- 172.21.15.43:0/1272027544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 msgr2=0x5561ffbb7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.883+0000 7f200f400700 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 0x5561ffbb7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.883+0000 7f200f400700 1 -- 172.21.15.43:0/1272027544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 msgr2=0x5561ffbb74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.883+0000 7f200f400700 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbb74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.883+0000 7f200f400700 1 -- 172.21.15.43:0/1272027544 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561ffa128d0 con 0x5561ffa25e40 2024-05-27T07:29:44.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.883+0000 7f200ea00700 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbb74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:29:44.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.884+0000 7f200f400700 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbb6fa0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2000003c80 tx=0x7f20000099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbc9cddd0cb066b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.884+0000 7f1ff9600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:29:44.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.884+0000 7f1ff9600700 1 -- 172.21.15.43:0/1272027544 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200002e060 con 0x5561ffa25e40 2024-05-27T07:29:44.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.884+0000 7f1ff9600700 1 -- 172.21.15.43:0/1272027544 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 9408+0+0 (secure 0 0 0) 0x7f2000007d60 con 0x5561ffa25e40 2024-05-27T07:29:44.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.885+0000 7f2024be1b80 1 -- 172.21.15.43:0/1272027544 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2008011470 con 0x5561ffa25e40 2024-05-27T07:29:44.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.885+0000 7f1ff9600700 1 -- 172.21.15.43:0/1272027544 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=123) ==== 24+0+0 (secure 0 0 0) 0x7f2000003e20 con 0x5561ffa25e40 2024-05-27T07:29:44.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.886+0000 7f2024be1b80 10 client.37094 fetch_fsmap learned FSMap version 123 2024-05-27T07:29:44.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.886+0000 7f2024be1b80 1 -- 172.21.15.43:0/1272027544 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=123,monmap=4+}) -- 0x5561ffa93060 con 0x5561ffa25e40 2024-05-27T07:29:44.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.886+0000 7f1ff9600700 1 -- 172.21.15.43:0/1272027544 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2008011470 con 0x5561ffa25e40 2024-05-27T07:29:44.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.887+0000 7f1ff9600700 1 -- 172.21.15.43:0/1272027544 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 123) ==== 4816+0+0 (secure 0 0 0) 0x7f2000011070 con 0x5561ffa25e40 2024-05-27T07:29:44.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.887+0000 7f2024be1b80 10 client.37094 fetch_fsmap finished waiting for FSMap version 123 2024-05-27T07:29:44.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.887+0000 7f2024be1b80 10 client.37094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:44.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.887+0000 7f2024be1b80 20 client.37094 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:44.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.887+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5561ffbc77e0 0x5561ffbc9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.887+0000 7f2024be1b80 4 client.37094 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:29:44.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.887+0000 7f2024be1b80 1 -- 172.21.15.43:0/1272027544 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561ffad4d90 con 0x5561ffbc77e0 2024-05-27T07:29:44.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.888+0000 7f200ea00700 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5561ffbc77e0 0x5561ffbc9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.889+0000 7f1ff9600700 10 client.37094 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:44.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.891+0000 7f200ea00700 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5561ffbc77e0 0x5561ffbc9cb0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfb7404230de319 server_cookie=731c50f3439dbf35 in_seq=0 out_seq=0 2024-05-27T07:29:44.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.892+0000 7f1ff9600700 1 -- 172.21.15.43:0/1272027544 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5561ffad4d90 con 0x5561ffbc77e0 2024-05-27T07:29:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.892+0000 7f1ff9600700 10 client.37094 handle_command_reply: tid=0 2024-05-27T07:29:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.892+0000 7f2024be1b80 1 client.37094 shutdown 2024-05-27T07:29:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.893+0000 7f2024be1b80 1 -- 172.21.15.43:0/1272027544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 msgr2=0x5561ffbb6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.893+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbb6fa0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2000003c80 tx=0x7f20000099e0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.893+0000 7f2024be1b80 1 -- 172.21.15.43:0/1272027544 shutdown_connections 2024-05-27T07:29:44.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.893+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffbad690 0x5561ffbb7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.893+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffa25e40 0x5561ffbb6fa0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.893+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5561ffbc77e0 0x5561ffbc9cb0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.893+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1272027544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffb119f0 0x5561ffbb74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.893+0000 7f2024be1b80 1 -- 172.21.15.43:0/1272027544 >> 172.21.15.43:0/1272027544 conn(0x5561ffabf390 msgr2=0x5561ffbacd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.893+0000 7f1ff9600700 0 client.37094 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.894+0000 7f2024be1b80 1 -- 172.21.15.43:0/1272027544 shutdown_connections 2024-05-27T07:29:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.894+0000 7f2024be1b80 1 -- 172.21.15.43:0/1272027544 wait complete. 2024-05-27T07:29:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.894+0000 7f2024be1b80 20 client.37094 trim_cache size 0 max 16384 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.936+0000 7f2024be1b80 1 Processor -- start 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.936+0000 7f2024be1b80 1 -- start start 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.936+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffa25e40 0x5561ffbb62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.936+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffb119f0 0x5561ffbc34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.936+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffbad690 0x5561ffbaee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.936+0000 7f2024be1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5561ffbad2f0 con 0x5561ffa25e40 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.936+0000 7f2024be1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f20101aeff0 con 0x5561ffbad690 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.936+0000 7f2024be1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f20101acc30 con 0x5561ffb119f0 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.937+0000 7f200f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffa25e40 0x5561ffbb62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.937+0000 7f200f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffa25e40 0x5561ffbb62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60930/0 (socket says 172.21.15.43:60930) 2024-05-27T07:29:44.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.937+0000 7f200f400700 1 -- 172.21.15.43:0/1008182334 learned_addr learned my addr 172.21.15.43:0/1008182334 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.938+0000 7f200ea00700 1 --2- 172.21.15.43:0/1008182334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffb119f0 0x5561ffbc34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.938+0000 7f200f400700 1 -- 172.21.15.43:0/1008182334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffb119f0 msgr2=0x5561ffbc34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.938+0000 7f200f400700 1 --2- 172.21.15.43:0/1008182334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffb119f0 0x5561ffbc34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.938+0000 7f200f400700 1 -- 172.21.15.43:0/1008182334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffbad690 msgr2=0x5561ffbaee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.938+0000 7f200f400700 1 --2- 172.21.15.43:0/1008182334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffbad690 0x5561ffbaee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.938+0000 7f200f400700 1 -- 172.21.15.43:0/1008182334 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561ffbad470 con 0x5561ffa25e40 2024-05-27T07:29:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.939+0000 7f200f400700 1 --2- 172.21.15.43:0/1008182334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffa25e40 0x5561ffbb62c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f2000000f80 tx=0x7f200000baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75ea5394febacf0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.940+0000 7f1ffbe00700 1 -- 172.21.15.43:0/1008182334 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200002e040 con 0x5561ffa25e40 2024-05-27T07:29:44.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.940+0000 7f1ffbe00700 1 -- 172.21.15.43:0/1008182334 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2000032460 con 0x5561ffa25e40 2024-05-27T07:29:44.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.940+0000 7f1ffbe00700 1 -- 172.21.15.43:0/1008182334 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000038080 con 0x5561ffa25e40 2024-05-27T07:29:44.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.941+0000 7f2024be1b80 1 -- 172.21.15.43:0/1008182334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffa25e40 msgr2=0x5561ffbb62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.941+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1008182334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffa25e40 0x5561ffbb62c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f2000000f80 tx=0x7f200000baa0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.942+0000 7f2024be1b80 1 -- 172.21.15.43:0/1008182334 shutdown_connections 2024-05-27T07:29:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.942+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1008182334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffbad690 0x5561ffbaee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.942+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1008182334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffa25e40 0x5561ffbb62c0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.942+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1008182334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffb119f0 0x5561ffbc34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.942+0000 7f2024be1b80 1 -- 172.21.15.43:0/1008182334 >> 172.21.15.43:0/1008182334 conn(0x5561ffabf390 msgr2=0x5561ffbae710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.943+0000 7f2024be1b80 1 -- 172.21.15.43:0/1008182334 shutdown_connections 2024-05-27T07:29:44.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.943+0000 7f2024be1b80 1 -- 172.21.15.43:0/1008182334 wait complete. 2024-05-27T07:29:44.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.944+0000 7f2024be1b80 1 -- start start 2024-05-27T07:29:44.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.944+0000 7f2024be1b80 1 Processor -- start 2024-05-27T07:29:44.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.946+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffa25e40 0x5561ffc2f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.946+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffb119f0 0x5561ffc2f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.946+0000 7f2024be1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffbad690 0x5561ffc2fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.946+0000 7f2024be1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f20101ad360 con 0x5561ffbad690 2024-05-27T07:29:44.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.946+0000 7f2024be1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f20101ae8f0 con 0x5561ffb119f0 2024-05-27T07:29:44.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.946+0000 7f2024be1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f20101ada60 con 0x5561ffa25e40 2024-05-27T07:29:44.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.946+0000 7f2024be1b80 10 client.0 _ll_register_callbacks cb 0x5561ffa6cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:29:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.947+0000 7f200ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffb119f0 0x5561ffc2f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.947+0000 7f200ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffb119f0 0x5561ffc2f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58806/0 (socket says 172.21.15.43:58806) 2024-05-27T07:29:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.947+0000 7f200ea00700 1 -- 172.21.15.43:0/1722711297 learned_addr learned my addr 172.21.15.43:0/1722711297 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:29:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.947+0000 7f200fe00700 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffbad690 0x5561ffc2fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.947+0000 7f200f400700 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffa25e40 0x5561ffc2f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.947+0000 7f200ea00700 1 -- 172.21.15.43:0/1722711297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffa25e40 msgr2=0x5561ffc2f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.947+0000 7f200ea00700 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffa25e40 0x5561ffc2f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.947+0000 7f200ea00700 1 -- 172.21.15.43:0/1722711297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffbad690 msgr2=0x5561ffc2fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:29:44.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.948+0000 7f200ea00700 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffbad690 0x5561ffc2fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.948+0000 7f200ea00700 1 -- 172.21.15.43:0/1722711297 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561ffbbf6f0 con 0x5561ffb119f0 2024-05-27T07:29:44.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.948+0000 7f200fe00700 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffbad690 0x5561ffc2fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:29:44.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.949+0000 7f200ea00700 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffb119f0 0x5561ffc2f880 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0a9b40 tx=0x7f1ffc0a95b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35f91e281471ea39 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:29:44.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.949+0000 7f1ff9600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:29:44.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.949+0000 7f1ff9600700 1 -- 172.21.15.43:0/1722711297 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc0cae60 con 0x5561ffb119f0 2024-05-27T07:29:44.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.949+0000 7f1ff9600700 1 -- 172.21.15.43:0/1722711297 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(140..140 src has 1..140) ==== 9408+0+0 (secure 0 0 0) 0x7f1ffc0e02a0 con 0x5561ffb119f0 2024-05-27T07:29:44.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.950+0000 7f2024be1b80 1 -- 172.21.15.43:0/1722711297 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2000032460 con 0x5561ffb119f0 2024-05-27T07:29:44.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.950+0000 7f1ff9600700 1 -- 172.21.15.43:0/1722711297 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=123) ==== 24+0+0 (secure 0 0 0) 0x7f1ffc0e6220 con 0x5561ffb119f0 2024-05-27T07:29:44.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.951+0000 7f2024be1b80 10 client.37097 fetch_fsmap learned FSMap version 123 2024-05-27T07:29:44.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.951+0000 7f2024be1b80 1 -- 172.21.15.43:0/1722711297 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=123,monmap=4+}) -- 0x5561ffbb1320 con 0x5561ffb119f0 2024-05-27T07:29:44.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.951+0000 7f1ff9600700 1 -- 172.21.15.43:0/1722711297 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2000032460 con 0x5561ffb119f0 2024-05-27T07:29:44.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.951+0000 7f1ff9600700 1 -- 172.21.15.43:0/1722711297 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 123) ==== 4816+0+0 (secure 0 0 0) 0x7f1ffc0ea040 con 0x5561ffb119f0 2024-05-27T07:29:44.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.951+0000 7f2024be1b80 10 client.37097 fetch_fsmap finished waiting for FSMap version 123 2024-05-27T07:29:44.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.951+0000 7f2024be1b80 10 client.37097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:29:44.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.951+0000 7f2024be1b80 20 client.37097 populate_metadata read hostname 'smithi043' 2024-05-27T07:29:44.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.951+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5561ffc43350 0x5561ffc45770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:29:44.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.951+0000 7f2024be1b80 4 client.37097 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:29:44.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.951+0000 7f2024be1b80 1 -- 172.21.15.43:0/1722711297 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5561fface570 con 0x5561ffc43350 2024-05-27T07:29:44.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.952+0000 7f200f400700 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5561ffc43350 0x5561ffc45770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:29:44.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.952+0000 7f1ff9600700 10 client.37097 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:44.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.953+0000 7f200f400700 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5561ffc43350 0x5561ffc45770 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4c08da2c827274f server_cookie=903a5327896b9f67 in_seq=0 out_seq=0 2024-05-27T07:29:44.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.954+0000 7f1ff9600700 1 -- 172.21.15.43:0/1722711297 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5561fface570 con 0x5561ffc43350 2024-05-27T07:29:44.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.954+0000 7f1ff9600700 10 client.37097 handle_command_reply: tid=0 2024-05-27T07:29:44.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.954+0000 7f2024be1b80 1 client.37097 shutdown 2024-05-27T07:29:44.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.954+0000 7f2024be1b80 1 -- 172.21.15.43:0/1722711297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffb119f0 msgr2=0x5561ffc2f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.954+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffb119f0 0x5561ffc2f880 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0a9b40 tx=0x7f1ffc0a95b0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.955+0000 7f2024be1b80 1 -- 172.21.15.43:0/1722711297 shutdown_connections 2024-05-27T07:29:44.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.955+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5561ffc43350 0x5561ffc45770 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.955+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561ffbad690 0x5561ffc2fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.955+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5561ffa25e40 0x5561ffc2f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.955+0000 7f2024be1b80 1 --2- 172.21.15.43:0/1722711297 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5561ffb119f0 0x5561ffc2f880 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.955+0000 7f1ff9600700 0 client.37097 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:29:44.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.955+0000 7f2024be1b80 1 -- 172.21.15.43:0/1722711297 >> 172.21.15.43:0/1722711297 conn(0x5561ffabf390 msgr2=0x5561ffbb6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.955+0000 7f2024be1b80 1 -- 172.21.15.43:0/1722711297 shutdown_connections 2024-05-27T07:29:44.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.955+0000 7f2024be1b80 1 -- 172.21.15.43:0/1722711297 wait complete. 2024-05-27T07:29:44.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.955+0000 7f2024be1b80 20 client.37097 trim_cache size 0 max 16384 2024-05-27T07:29:44.946 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:29:44.946 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:29:44.947 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:29:44.947 INFO:teuthology.orchestra.run.smithi043.stdout: "c6af25f3-304f-4fe9-afab-8c56cecaa67c": { 2024-05-27T07:29:44.947 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:29:44.947 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "c6af25f3-304f-4fe9-afab-8c56cecaa67c", 2024-05-27T07:29:44.947 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:29:44.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:29:44.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:29:44.947 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.956+0000 7f2016000700 1 -- 172.21.15.43:0/3459922258 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1ff40b2a70 msgr2=0x7f1ff40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.956+0000 7f2016000700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1ff40b2a70 0x7f1ff40b4f80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x5561ffba17a0 tx=0x7f200000b040 comp rx=0 tx=0).stop 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.956+0000 7f2016000700 1 -- 172.21.15.43:0/3459922258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2010107de0 msgr2=0x7f201019d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.956+0000 7f2016000700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2010107de0 0x7f201019d550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00e970 tx=0x7f1ffc00ed30 comp rx=0 tx=0).stop 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.956+0000 7f2016000700 1 -- 172.21.15.43:0/3459922258 shutdown_connections 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.956+0000 7f2016000700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1ff40b2a70 0x7f1ff40b4f80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.956+0000 7f2016000700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2010108730 0x7f20101a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.956+0000 7f2016000700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20101034b0 0x7f201019cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.957+0000 7f2016000700 1 --2- 172.21.15.43:0/3459922258 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2010107de0 0x7f201019d550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.957+0000 7f2016000700 1 -- 172.21.15.43:0/3459922258 >> 172.21.15.43:0/3459922258 conn(0x7f20100fedb0 msgr2=0x7f20101011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.957+0000 7f2016000700 1 -- 172.21.15.43:0/3459922258 shutdown_connections 2024-05-27T07:29:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:29:44.957+0000 7f2016000700 1 -- 172.21.15.43:0/3459922258 wait complete. 2024-05-27T07:29:44.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:44 smithi179 ceph-mon[35888]: pgmap v530: 129 pgs: 129 active+clean; 493 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-27T07:29:45.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 11970 7.70 MB/sec warmup 106 sec latency 102.463 ms 2024-05-27T07:29:46.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 12065 7.65 MB/sec warmup 107 sec latency 209.861 ms 2024-05-27T07:29:46.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:46 smithi132 ceph-mon[35283]: pgmap v531: 129 pgs: 129 active+clean; 493 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-27T07:29:46.950 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:46 smithi043 ceph-mon[31531]: pgmap v531: 129 pgs: 129 active+clean; 493 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-27T07:29:46.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:46 smithi179 ceph-mon[35888]: pgmap v531: 129 pgs: 129 active+clean; 493 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 189 op/s 2024-05-27T07:29:47.006 INFO:tasks.cephfs.filesystem:scrub status for tag:c6af25f3-304f-4fe9-afab-8c56cecaa67c - {'path': '/', 'tag': 'c6af25f3-304f-4fe9-afab-8c56cecaa67c', 'options': 'recursive,force'} 2024-05-27T07:29:47.773 INFO:tasks.workunit.client.0.smithi043.stdout: 10 12178 7.61 MB/sec warmup 108 sec latency 86.991 ms 2024-05-27T07:29:48.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 12289 7.56 MB/sec warmup 109 sec latency 175.641 ms 2024-05-27T07:29:48.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:48 smithi132 ceph-mon[35283]: pgmap v532: 129 pgs: 129 active+clean; 497 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.2 MiB/s wr, 198 op/s 2024-05-27T07:29:48.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:48 smithi043 ceph-mon[31531]: pgmap v532: 129 pgs: 129 active+clean; 497 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.2 MiB/s wr, 198 op/s 2024-05-27T07:29:48.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:48 smithi179 ceph-mon[35888]: pgmap v532: 129 pgs: 129 active+clean; 497 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.2 MiB/s wr, 198 op/s 2024-05-27T07:29:49.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 12418 7.54 MB/sec warmup 110 sec latency 257.600 ms 2024-05-27T07:29:50.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 12553 7.55 MB/sec warmup 111 sec latency 87.191 ms 2024-05-27T07:29:50.840 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:50 smithi132 ceph-mon[35283]: pgmap v533: 129 pgs: 129 active+clean; 497 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.7 MiB/s wr, 167 op/s 2024-05-27T07:29:50.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:50 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:50.952 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:50 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:50 smithi043 systemd[1]: Stopped Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:50 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.743s CPU time. 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:50 smithi043 systemd[1]: Starting Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:50 smithi043 ceph-mon[31531]: pgmap v533: 129 pgs: 129 active+clean; 497 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.7 MiB/s wr, 167 op/s 2024-05-27T07:29:50.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:50 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:50.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:50 smithi179 ceph-mon[35888]: pgmap v533: 129 pgs: 129 active+clean; 497 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.7 MiB/s wr, 167 op/s 2024-05-27T07:29:50.958 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:50 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:29:51.702 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:51 smithi043 podman[120148]: 2024-05-27 07:29:51.134743755 +0000 UTC m=+0.057830254 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:29:51.702 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:51 smithi043 podman[120148]: 2024-05-27 07:29:51.26751099 +0000 UTC m=+0.190597476 container create 9d5bbb721eae1bc22cc52c126fc2bed4cb84696203ef339a407a495745be618d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-27T07:29:51.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 12678 7.59 MB/sec warmup 112 sec latency 104.378 ms 2024-05-27T07:29:52.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:51 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:51 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:51 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:51 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:51 smithi132 ceph-mon[35283]: pgmap v534: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-05-27T07:29:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:51 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:51 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:51 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:51 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:51 smithi043 ceph-mon[31531]: pgmap v534: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-05-27T07:29:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:51 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:51 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:51 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:51 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:51 smithi179 ceph-mon[35888]: pgmap v534: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-05-27T07:29:52.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 12839 7.61 MB/sec warmup 113 sec latency 85.275 ms 2024-05-27T07:29:52.836 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:52 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:29:52.836 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:52 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:29:52.836 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:52 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:29:52.837 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:52 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:29:52.837 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:52 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:29:53.091 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:52 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:53.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:52 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:53.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:52 smithi132 ceph-mon[35283]: pgmap v535: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.7 MiB/s wr, 185 op/s 2024-05-27T07:29:53.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:52 smithi132 ceph-mon[35283]: pgmap v536: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.3 MiB/s wr, 227 op/s 2024-05-27T07:29:53.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:52 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:53.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:52 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:29:53.202 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:52 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:29:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:52 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:29:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:52 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:29:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:52 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:29:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:52 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:52 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:52 smithi043 ceph-mon[31531]: pgmap v535: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.7 MiB/s wr, 185 op/s 2024-05-27T07:29:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:52 smithi043 ceph-mon[31531]: pgmap v536: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.3 MiB/s wr, 227 op/s 2024-05-27T07:29:53.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:52 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:52 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:29:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:52 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:29:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:52 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:29:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:52 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:29:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:52 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:29:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:52 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:29:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:52 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:29:53.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:52 smithi179 ceph-mon[35888]: pgmap v535: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.7 MiB/s wr, 185 op/s 2024-05-27T07:29:53.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:52 smithi179 ceph-mon[35888]: pgmap v536: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.3 MiB/s wr, 227 op/s 2024-05-27T07:29:53.210 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:52 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:29:53.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 12959 7.60 MB/sec warmup 114 sec latency 77.256 ms 2024-05-27T07:29:54.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 13047 7.58 MB/sec warmup 115 sec latency 89.618 ms 2024-05-27T07:29:55.459 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:55 smithi179 ceph-mon[35888]: pgmap v537: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.3 MiB/s wr, 227 op/s 2024-05-27T07:29:55.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:55 smithi043 ceph-mon[31531]: pgmap v537: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.3 MiB/s wr, 227 op/s 2024-05-27T07:29:55.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:55 smithi132 ceph-mon[35283]: pgmap v537: 129 pgs: 129 active+clean; 498 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.3 MiB/s wr, 227 op/s 2024-05-27T07:29:55.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 13176 7.59 MB/sec warmup 116 sec latency 80.979 ms 2024-05-27T07:29:56.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 13275 7.60 MB/sec warmup 117 sec latency 310.436 ms 2024-05-27T07:29:57.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:57 smithi132 ceph-mon[35283]: pgmap v538: 129 pgs: 129 active+clean; 507 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 225 op/s 2024-05-27T07:29:57.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:57 smithi132 ceph-mon[35283]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:57.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:57 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:29:57.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:57 smithi043 ceph-mon[31531]: pgmap v538: 129 pgs: 129 active+clean; 507 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 225 op/s 2024-05-27T07:29:57.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:57 smithi043 ceph-mon[31531]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:57.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:57 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:29:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:57 smithi179 ceph-mon[35888]: pgmap v538: 129 pgs: 129 active+clean; 507 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 225 op/s 2024-05-27T07:29:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:57 smithi179 ceph-mon[35888]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:29:57.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:57 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:29:57.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 13371 7.59 MB/sec warmup 118 sec latency 425.936 ms 2024-05-27T07:29:58.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 13496 7.57 MB/sec warmup 119 sec latency 71.266 ms 2024-05-27T07:29:59.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:58 smithi043 podman[120148]: 2024-05-27 07:29:58.898383133 +0000 UTC m=+7.821469616 container init 9d5bbb721eae1bc22cc52c126fc2bed4cb84696203ef339a407a495745be618d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:29:59.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:58 smithi043 podman[120148]: 2024-05-27 07:29:58.912635398 +0000 UTC m=+7.835721882 container start 9d5bbb721eae1bc22cc52c126fc2bed4cb84696203ef339a407a495745be618d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:29:59.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 bash[120148]: 9d5bbb721eae1bc22cc52c126fc2bed4cb84696203ef339a407a495745be618d 2024-05-27T07:29:59.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 systemd[1]: Started Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:29:59.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-mds[120253]: -- 172.21.15.43:0/369690748 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560018737a40 con 0x56001877f600 2024-05-27T07:29:59.203 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: starting mds.a at 2024-05-27T07:29:59.590 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:29:59 smithi132 ceph-mon[35283]: pgmap v539: 129 pgs: 129 active+clean; 507 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 225 op/s 2024-05-27T07:29:59.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-mon[31531]: pgmap v539: 129 pgs: 129 active+clean; 507 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 225 op/s 2024-05-27T07:29:59.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8696e00700 time 2024-05-27T07:29:59.331040+0000 2024-05-27T07:29:59.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:59.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:59.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f86ac820ba6] 2024-05-27T07:29:59.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86ac820dd1] 2024-05-27T07:29:59.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56000cf8123d] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56000cf6c971] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 5: (SnapRealm::check_cache() const+0x13e) [0x56000cf6afa0] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56000cf6b651] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56000cf00347] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56000cbfc87e] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56000cc01e94] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56000cc0c969] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 11: (MDLog::_replay_thread()+0x1f38) [0x56000cfa548c] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56000cfae9cb] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 13: (Thread::entry_wrapper()+0x43) [0x7f86ac7ef72d] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 14: (Thread::_entry_func(void*)+0xd) [0x7f86ac7ef749] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86ab3931ca] 2024-05-27T07:29:59.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 16: clone() 2024-05-27T07:29:59.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.333+0000 7f8696e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8696e00700 time 2024-05-27T07:29:59.331040+0000 2024-05-27T07:29:59.708 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f86ac820ba6] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86ac820dd1] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56000cf8123d] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56000cf6c971] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 5: (SnapRealm::check_cache() const+0x13e) [0x56000cf6afa0] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56000cf6b651] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56000cf00347] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56000cbfc87e] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56000cc01e94] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56000cc0c969] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 11: (MDLog::_replay_thread()+0x1f38) [0x56000cfa548c] 2024-05-27T07:29:59.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56000cfae9cb] 2024-05-27T07:29:59.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 13: (Thread::entry_wrapper()+0x43) [0x7f86ac7ef72d] 2024-05-27T07:29:59.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 14: (Thread::_entry_func(void*)+0xd) [0x7f86ac7ef749] 2024-05-27T07:29:59.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86ab3931ca] 2024-05-27T07:29:59.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 16: clone() 2024-05-27T07:29:59.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: *** Caught signal (Aborted) ** 2024-05-27T07:29:59.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: in thread 7f8696e00700 thread_name:md_log_replay 2024-05-27T07:29:59.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:59.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86ab39dd20] 2024-05-27T07:29:59.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2: gsignal() 2024-05-27T07:29:59.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 3: abort() 2024-05-27T07:29:59.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f86ac820ced] 2024-05-27T07:29:59.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:29:59 smithi179 ceph-mon[35888]: pgmap v539: 129 pgs: 129 active+clean; 507 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 225 op/s 2024-05-27T07:29:59.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86ac820dd1] 2024-05-27T07:29:59.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56000cf8123d] 2024-05-27T07:29:59.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56000cf6c971] 2024-05-27T07:29:59.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 8: (SnapRealm::check_cache() const+0x13e) [0x56000cf6afa0] 2024-05-27T07:29:59.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56000cf6b651] 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56000cf00347] 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56000cbfc87e] 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56000cc01e94] 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56000cc0c969] 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 14: (MDLog::_replay_thread()+0x1f38) [0x56000cfa548c] 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56000cfae9cb] 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 16: (Thread::entry_wrapper()+0x43) [0x7f86ac7ef72d] 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 17: (Thread::_entry_func(void*)+0xd) [0x7f86ac7ef749] 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f86ab3931ca] 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 19: clone() 2024-05-27T07:29:59.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.336+0000 7f8696e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: in thread 7f8696e00700 thread_name:md_log_replay 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86ab39dd20] 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2: gsignal() 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 3: abort() 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f86ac820ced] 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86ac820dd1] 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56000cf8123d] 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56000cf6c971] 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 8: (SnapRealm::check_cache() const+0x13e) [0x56000cf6afa0] 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56000cf6b651] 2024-05-27T07:29:59.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56000cf00347] 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56000cbfc87e] 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56000cc01e94] 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56000cc0c969] 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 14: (MDLog::_replay_thread()+0x1f38) [0x56000cfa548c] 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56000cfae9cb] 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 16: (Thread::entry_wrapper()+0x43) [0x7f86ac7ef72d] 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 17: (Thread::_entry_func(void*)+0xd) [0x7f86ac7ef749] 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f86ab3931ca] 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 19: clone() 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: -1> 2024-05-27T07:29:59.333+0000 7f8696e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8696e00700 time 2024-05-27T07:29:59.331040+0000 2024-05-27T07:29:59.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f86ac820ba6] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86ac820dd1] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56000cf8123d] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56000cf6c971] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 5: (SnapRealm::check_cache() const+0x13e) [0x56000cf6afa0] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56000cf6b651] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56000cf00347] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56000cbfc87e] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56000cc01e94] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56000cc0c969] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 11: (MDLog::_replay_thread()+0x1f38) [0x56000cfa548c] 2024-05-27T07:29:59.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56000cfae9cb] 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 13: (Thread::entry_wrapper()+0x43) [0x7f86ac7ef72d] 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 14: (Thread::_entry_func(void*)+0xd) [0x7f86ac7ef749] 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86ab3931ca] 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 16: clone() 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 0> 2024-05-27T07:29:59.336+0000 7f8696e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: in thread 7f8696e00700 thread_name:md_log_replay 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86ab39dd20] 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2: gsignal() 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 3: abort() 2024-05-27T07:29:59.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f86ac820ced] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86ac820dd1] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56000cf8123d] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56000cf6c971] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 8: (SnapRealm::check_cache() const+0x13e) [0x56000cf6afa0] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56000cf6b651] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56000cf00347] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56000cbfc87e] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56000cc01e94] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56000cc0c969] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 14: (MDLog::_replay_thread()+0x1f38) [0x56000cfa548c] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56000cfae9cb] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 16: (Thread::entry_wrapper()+0x43) [0x7f86ac7ef72d] 2024-05-27T07:29:59.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 17: (Thread::_entry_func(void*)+0xd) [0x7f86ac7ef749] 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f86ab3931ca] 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 19: clone() 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: -1> 2024-05-27T07:29:59.333+0000 7f8696e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f8696e00700 time 2024-05-27T07:29:59.331040+0000 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f86ac820ba6] 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86ac820dd1] 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56000cf8123d] 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56000cf6c971] 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 5: (SnapRealm::check_cache() const+0x13e) [0x56000cf6afa0] 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56000cf6b651] 2024-05-27T07:29:59.720 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56000cf00347] 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56000cbfc87e] 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56000cc01e94] 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56000cc0c969] 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 11: (MDLog::_replay_thread()+0x1f38) [0x56000cfa548c] 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56000cfae9cb] 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 13: (Thread::entry_wrapper()+0x43) [0x7f86ac7ef72d] 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 14: (Thread::_entry_func(void*)+0xd) [0x7f86ac7ef749] 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f86ab3931ca] 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 16: clone() 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.721 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 0> 2024-05-27T07:29:59.336+0000 7f8696e00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: in thread 7f8696e00700 thread_name:md_log_replay 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f86ab39dd20] 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2: gsignal() 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 3: abort() 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f86ac820ced] 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f86ac820dd1] 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56000cf8123d] 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56000cf6c971] 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 8: (SnapRealm::check_cache() const+0x13e) [0x56000cf6afa0] 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56000cf6b651] 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56000cf00347] 2024-05-27T07:29:59.722 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56000cbfc87e] 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56000cc01e94] 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56000cc0c969] 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 14: (MDLog::_replay_thread()+0x1f38) [0x56000cfa548c] 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56000cfae9cb] 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 16: (Thread::entry_wrapper()+0x43) [0x7f86ac7ef72d] 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 17: (Thread::_entry_func(void*)+0xd) [0x7f86ac7ef749] 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f86ab3931ca] 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 19: clone() 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120176]: 2024-05-27T07:29:59.723 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:29:59 smithi043 podman[120291]: 2024-05-27 07:29:59.47614479 +0000 UTC m=+0.045679288 container died 9d5bbb721eae1bc22cc52c126fc2bed4cb84696203ef339a407a495745be618d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:30:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/4197503627,v1:172.21.15.43:6835/4197503627] up:boot 2024-05-27T07:30:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:30:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: Health detail: HEALTH_WARN 6 failed cephadm daemon(s) 2024-05-27T07:30:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: [WRN] CEPHADM_FAILED_DAEMON: 6 failed cephadm daemon(s) 2024-05-27T07:30:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: daemon mds.j on smithi043 is in error state 2024-05-27T07:30:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: daemon mds.d on smithi043 is in error state 2024-05-27T07:30:00.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: daemon mds.g on smithi043 is in error state 2024-05-27T07:30:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: daemon mds.l on smithi179 is in error state 2024-05-27T07:30:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: daemon mds.f on smithi179 is in error state 2024-05-27T07:30:00.592 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:00 smithi132 ceph-mon[35283]: daemon mds.c on smithi179 is in error state 2024-05-27T07:30:00.702 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:00 smithi043 podman[120291]: 2024-05-27 07:30:00.311858583 +0000 UTC m=+0.881393070 container remove 9d5bbb721eae1bc22cc52c126fc2bed4cb84696203ef339a407a495745be618d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-27T07:30:00.702 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:00 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/4197503627,v1:172.21.15.43:6835/4197503627] up:boot 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: Health detail: HEALTH_WARN 6 failed cephadm daemon(s) 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: [WRN] CEPHADM_FAILED_DAEMON: 6 failed cephadm daemon(s) 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: daemon mds.j on smithi043 is in error state 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: daemon mds.d on smithi043 is in error state 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: daemon mds.g on smithi043 is in error state 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: daemon mds.l on smithi179 is in error state 2024-05-27T07:30:00.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: daemon mds.f on smithi179 is in error state 2024-05-27T07:30:00.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:00 smithi043 ceph-mon[31531]: daemon mds.c on smithi179 is in error state 2024-05-27T07:30:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/4197503627,v1:172.21.15.43:6835/4197503627] up:boot 2024-05-27T07:30:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:30:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: Health detail: HEALTH_WARN 6 failed cephadm daemon(s) 2024-05-27T07:30:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: [WRN] CEPHADM_FAILED_DAEMON: 6 failed cephadm daemon(s) 2024-05-27T07:30:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: daemon mds.j on smithi043 is in error state 2024-05-27T07:30:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: daemon mds.d on smithi043 is in error state 2024-05-27T07:30:00.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: daemon mds.g on smithi043 is in error state 2024-05-27T07:30:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: daemon mds.l on smithi179 is in error state 2024-05-27T07:30:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: daemon mds.f on smithi179 is in error state 2024-05-27T07:30:00.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:00 smithi179 ceph-mon[35888]: daemon mds.c on smithi179 is in error state 2024-05-27T07:30:00.774 INFO:tasks.workunit.client.0.smithi043.stdout: 10 13712 3.14 MB/sec execute 1 sec latency 112.964 ms 2024-05-27T07:30:01.202 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:00 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:30:01.202 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:00 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.699s CPU time. 2024-05-27T07:30:01.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:01 smithi132 ceph-mon[35283]: pgmap v540: 129 pgs: 129 active+clean; 507 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-05-27T07:30:01.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:01 smithi043 ceph-mon[31531]: pgmap v540: 129 pgs: 129 active+clean; 507 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-05-27T07:30:01.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:01 smithi179 ceph-mon[35888]: pgmap v540: 129 pgs: 129 active+clean; 507 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-05-27T07:30:01.775 INFO:tasks.workunit.client.0.smithi043.stdout: 10 13802 2.57 MB/sec execute 2 sec latency 80.319 ms 2024-05-27T07:30:02.775 INFO:tasks.workunit.client.0.smithi043.stdout: 10 13921 4.59 MB/sec execute 3 sec latency 108.923 ms 2024-05-27T07:30:03.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:03 smithi132 ceph-mon[35283]: pgmap v541: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 191 op/s 2024-05-27T07:30:03.591 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:03 smithi132 ceph-mon[35283]: osdmap e141: 12 total, 12 up, 12 in 2024-05-27T07:30:03.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:03 smithi043 ceph-mon[31531]: pgmap v541: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 191 op/s 2024-05-27T07:30:03.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:03 smithi043 ceph-mon[31531]: osdmap e141: 12 total, 12 up, 12 in 2024-05-27T07:30:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:03 smithi179 ceph-mon[35888]: pgmap v541: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 191 op/s 2024-05-27T07:30:03.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:03 smithi179 ceph-mon[35888]: osdmap e141: 12 total, 12 up, 12 in 2024-05-27T07:30:03.775 INFO:tasks.workunit.client.0.smithi043.stdout: 10 14046 4.57 MB/sec execute 4 sec latency 196.621 ms 2024-05-27T07:30:04.775 INFO:tasks.workunit.client.0.smithi043.stdout: 10 14188 5.52 MB/sec execute 5 sec latency 172.255 ms 2024-05-27T07:30:05.596 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:05 smithi132 ceph-mon[35283]: pgmap v543: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 191 op/s 2024-05-27T07:30:05.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:05 smithi043 ceph-mon[31531]: pgmap v543: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 191 op/s 2024-05-27T07:30:05.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:05 smithi179 ceph-mon[35888]: pgmap v543: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 191 op/s 2024-05-27T07:30:05.775 INFO:tasks.workunit.client.0.smithi043.stdout: 10 14296 5.64 MB/sec execute 6 sec latency 99.005 ms 2024-05-27T07:30:06.775 INFO:tasks.workunit.client.0.smithi043.stdout: 10 14398 5.65 MB/sec execute 7 sec latency 417.616 ms 2024-05-27T07:30:07.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:07 smithi043 ceph-mon[31531]: pgmap v544: 129 pgs: 129 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.9 MiB/s wr, 200 op/s 2024-05-27T07:30:07.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:07 smithi043 ceph-mon[31531]: osdmap e142: 12 total, 12 up, 12 in 2024-05-27T07:30:07.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:07 smithi179 ceph-mon[35888]: pgmap v544: 129 pgs: 129 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.9 MiB/s wr, 200 op/s 2024-05-27T07:30:07.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:07 smithi179 ceph-mon[35888]: osdmap e142: 12 total, 12 up, 12 in 2024-05-27T07:30:07.775 INFO:tasks.workunit.client.0.smithi043.stdout: 10 14523 5.67 MB/sec execute 8 sec latency 87.460 ms 2024-05-27T07:30:07.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:07 smithi132 ceph-mon[35283]: pgmap v544: 129 pgs: 129 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.9 MiB/s wr, 200 op/s 2024-05-27T07:30:07.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:07 smithi132 ceph-mon[35283]: osdmap e142: 12 total, 12 up, 12 in 2024-05-27T07:30:08.776 INFO:tasks.workunit.client.0.smithi043.stdout: 10 14668 6.33 MB/sec execute 9 sec latency 68.018 ms 2024-05-27T07:30:09.702 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:09 smithi043 ceph-mon[31531]: pgmap v546: 129 pgs: 129 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.8 MiB/s wr, 250 op/s 2024-05-27T07:30:09.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:09 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:09.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:09 smithi179 ceph-mon[35888]: pgmap v546: 129 pgs: 129 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.8 MiB/s wr, 250 op/s 2024-05-27T07:30:09.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:09 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:09.776 INFO:tasks.workunit.client.0.smithi043.stdout: 10 14790 6.34 MB/sec execute 10 sec latency 97.431 ms 2024-05-27T07:30:09.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:09 smithi132 ceph-mon[35283]: pgmap v546: 129 pgs: 129 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.8 MiB/s wr, 250 op/s 2024-05-27T07:30:09.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:09 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:10.776 INFO:tasks.workunit.client.0.smithi043.stdout: 10 14922 6.53 MB/sec execute 11 sec latency 83.840 ms 2024-05-27T07:30:11.407 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:10 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-05-27T07:30:11.407 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:10 smithi043 systemd[1]: Stopped Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:30:11.407 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:10 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.699s CPU time. 2024-05-27T07:30:11.407 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:10 smithi043 systemd[1]: Starting Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:30:11.407 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:11 smithi043 ceph-mon[31531]: pgmap v547: 129 pgs: 129 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.6 MiB/s wr, 126 op/s 2024-05-27T07:30:11.676 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:11 smithi043 podman[120434]: 2024-05-27 07:30:11.351792488 +0000 UTC m=+0.049380343 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:30:11.676 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:11 smithi043 podman[120434]: 2024-05-27 07:30:11.496301068 +0000 UTC m=+0.193888929 container create 862cf9ab7b930fe3f40d82c91b9a3be5b415b2969bd46edbfc35838237dea97f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:30:11.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:11 smithi179 ceph-mon[35888]: pgmap v547: 129 pgs: 129 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.6 MiB/s wr, 126 op/s 2024-05-27T07:30:11.776 INFO:tasks.workunit.client.0.smithi043.stdout: 10 15028 6.14 MB/sec execute 12 sec latency 104.458 ms 2024-05-27T07:30:11.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:11 smithi132 ceph-mon[35283]: pgmap v547: 129 pgs: 129 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.6 MiB/s wr, 126 op/s 2024-05-27T07:30:12.031 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:11 smithi043 podman[120434]: 2024-05-27 07:30:11.783221801 +0000 UTC m=+0.480809677 container init 862cf9ab7b930fe3f40d82c91b9a3be5b415b2969bd46edbfc35838237dea97f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS) 2024-05-27T07:30:12.032 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:11 smithi043 podman[120434]: 2024-05-27 07:30:11.79353995 +0000 UTC m=+0.491127807 container start 862cf9ab7b930fe3f40d82c91b9a3be5b415b2969bd46edbfc35838237dea97f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:30:12.032 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:11 smithi043 ceph-mds[120456]: -- 172.21.15.43:0/3853713240 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627d98e3a40 con 0x5627d992b600 2024-05-27T07:30:12.032 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:11 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: starting mds.a at 2024-05-27T07:30:12.409 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 bash[120434]: 862cf9ab7b930fe3f40d82c91b9a3be5b415b2969bd46edbfc35838237dea97f 2024-05-27T07:30:12.409 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 systemd[1]: Started Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:30:12.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-mon[31531]: daemon mds.a restarted 2024-05-27T07:30:12.704 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9ed8a00700 time 2024-05-27T07:30:12.554440+0000 2024-05-27T07:30:12.704 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:12.704 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:12.704 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9eee423ba6] 2024-05-27T07:30:12.704 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9eee423dd1] 2024-05-27T07:30:12.704 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56279ecb023d] 2024-05-27T07:30:12.704 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56279ec9b971] 2024-05-27T07:30:12.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 5: (SnapRealm::check_cache() const+0x13e) [0x56279ec99fa0] 2024-05-27T07:30:12.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56279ec9a651] 2024-05-27T07:30:12.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56279ec2f347] 2024-05-27T07:30:12.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56279e92b87e] 2024-05-27T07:30:12.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56279e930e94] 2024-05-27T07:30:12.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56279e93b969] 2024-05-27T07:30:12.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 11: (MDLog::_replay_thread()+0x1f38) [0x56279ecd448c] 2024-05-27T07:30:12.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56279ecdd9cb] 2024-05-27T07:30:12.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 13: (Thread::entry_wrapper()+0x43) [0x7f9eee3f272d] 2024-05-27T07:30:12.705 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9eee3f2749] 2024-05-27T07:30:12.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9eecf961ca] 2024-05-27T07:30:12.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 16: clone() 2024-05-27T07:30:12.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.556+0000 7f9ed8a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9ed8a00700 time 2024-05-27T07:30:12.554440+0000 2024-05-27T07:30:12.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:12.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:12.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9eee423ba6] 2024-05-27T07:30:12.706 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9eee423dd1] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56279ecb023d] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56279ec9b971] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 5: (SnapRealm::check_cache() const+0x13e) [0x56279ec99fa0] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56279ec9a651] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56279ec2f347] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56279e92b87e] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56279e930e94] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56279e93b969] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 11: (MDLog::_replay_thread()+0x1f38) [0x56279ecd448c] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56279ecdd9cb] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 13: (Thread::entry_wrapper()+0x43) [0x7f9eee3f272d] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9eee3f2749] 2024-05-27T07:30:12.707 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9eecf961ca] 2024-05-27T07:30:12.708 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 16: clone() 2024-05-27T07:30:12.708 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.708 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: *** Caught signal (Aborted) ** 2024-05-27T07:30:12.708 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: in thread 7f9ed8a00700 thread_name:md_log_replay 2024-05-27T07:30:12.708 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:12.708 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9eecfa0d20] 2024-05-27T07:30:12.708 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2: gsignal() 2024-05-27T07:30:12.708 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 3: abort() 2024-05-27T07:30:12.708 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9eee423ced] 2024-05-27T07:30:12.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9eee423dd1] 2024-05-27T07:30:12.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56279ecb023d] 2024-05-27T07:30:12.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56279ec9b971] 2024-05-27T07:30:12.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 8: (SnapRealm::check_cache() const+0x13e) [0x56279ec99fa0] 2024-05-27T07:30:12.709 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56279ec9a651] 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56279ec2f347] 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56279e92b87e] 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56279e930e94] 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56279e93b969] 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 14: (MDLog::_replay_thread()+0x1f38) [0x56279ecd448c] 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56279ecdd9cb] 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 16: (Thread::entry_wrapper()+0x43) [0x7f9eee3f272d] 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9eee3f2749] 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9eecf961ca] 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 19: clone() 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.559+0000 7f9ed8a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:12.710 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: in thread 7f9ed8a00700 thread_name:md_log_replay 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9eecfa0d20] 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2: gsignal() 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 3: abort() 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9eee423ced] 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9eee423dd1] 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56279ecb023d] 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56279ec9b971] 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 8: (SnapRealm::check_cache() const+0x13e) [0x56279ec99fa0] 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56279ec9a651] 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56279ec2f347] 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56279e92b87e] 2024-05-27T07:30:12.711 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56279e930e94] 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56279e93b969] 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 14: (MDLog::_replay_thread()+0x1f38) [0x56279ecd448c] 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56279ecdd9cb] 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 16: (Thread::entry_wrapper()+0x43) [0x7f9eee3f272d] 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9eee3f2749] 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9eecf961ca] 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 19: clone() 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: -1> 2024-05-27T07:30:12.556+0000 7f9ed8a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9ed8a00700 time 2024-05-27T07:30:12.554440+0000 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.712 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9eee423ba6] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9eee423dd1] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56279ecb023d] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56279ec9b971] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 5: (SnapRealm::check_cache() const+0x13e) [0x56279ec99fa0] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56279ec9a651] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56279ec2f347] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56279e92b87e] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56279e930e94] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56279e93b969] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 11: (MDLog::_replay_thread()+0x1f38) [0x56279ecd448c] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56279ecdd9cb] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 13: (Thread::entry_wrapper()+0x43) [0x7f9eee3f272d] 2024-05-27T07:30:12.713 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9eee3f2749] 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9eecf961ca] 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 16: clone() 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 0> 2024-05-27T07:30:12.559+0000 7f9ed8a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: in thread 7f9ed8a00700 thread_name:md_log_replay 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9eecfa0d20] 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2: gsignal() 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 3: abort() 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9eee423ced] 2024-05-27T07:30:12.714 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9eee423dd1] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56279ecb023d] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56279ec9b971] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 8: (SnapRealm::check_cache() const+0x13e) [0x56279ec99fa0] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56279ec9a651] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56279ec2f347] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56279e92b87e] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56279e930e94] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56279e93b969] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 14: (MDLog::_replay_thread()+0x1f38) [0x56279ecd448c] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56279ecdd9cb] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 16: (Thread::entry_wrapper()+0x43) [0x7f9eee3f272d] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9eee3f2749] 2024-05-27T07:30:12.715 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9eecf961ca] 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 19: clone() 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: -1> 2024-05-27T07:30:12.556+0000 7f9ed8a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f9ed8a00700 time 2024-05-27T07:30:12.554440+0000 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3872-g38f9d900/rpm/el8/BUILD/ceph-19.0.0-3872-g38f9d900/src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9eee423ba6] 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9eee423dd1] 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56279ecb023d] 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 4: (SnapRealm::build_snap_set() const+0x323) [0x56279ec9b971] 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 5: (SnapRealm::check_cache() const+0x13e) [0x56279ec99fa0] 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56279ec9a651] 2024-05-27T07:30:12.716 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56279ec2f347] 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56279e92b87e] 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56279e930e94] 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 10: (EUpdate::replay(MDSRank*)+0x63) [0x56279e93b969] 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 11: (MDLog::_replay_thread()+0x1f38) [0x56279ecd448c] 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 12: (MDLog::ReplayThread::entry()+0x11) [0x56279ecdd9cb] 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 13: (Thread::entry_wrapper()+0x43) [0x7f9eee3f272d] 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 14: (Thread::_entry_func(void*)+0xd) [0x7f9eee3f2749] 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9eecf961ca] 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 16: clone() 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 0> 2024-05-27T07:30:12.559+0000 7f9ed8a00700 -1 *** Caught signal (Aborted) ** 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: in thread 7f9ed8a00700 thread_name:md_log_replay 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.717 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9eecfa0d20] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2: gsignal() 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 3: abort() 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9eee423ced] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9eee423dd1] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 6: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x56279ecb023d] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 7: (SnapRealm::build_snap_set() const+0x323) [0x56279ec9b971] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 8: (SnapRealm::check_cache() const+0x13e) [0x56279ec99fa0] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 9: (operator<<(std::ostream&, SnapRealm const&)+0x1fd) [0x56279ec9a651] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 10: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x271) [0x56279ec2f347] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 11: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x22c) [0x56279e92b87e] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 12: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x2702) [0x56279e930e94] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 13: (EUpdate::replay(MDSRank*)+0x63) [0x56279e93b969] 2024-05-27T07:30:12.718 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 14: (MDLog::_replay_thread()+0x1f38) [0x56279ecd448c] 2024-05-27T07:30:12.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 15: (MDLog::ReplayThread::entry()+0x11) [0x56279ecdd9cb] 2024-05-27T07:30:12.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 16: (Thread::entry_wrapper()+0x43) [0x7f9eee3f272d] 2024-05-27T07:30:12.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 17: (Thread::_entry_func(void*)+0xd) [0x7f9eee3f2749] 2024-05-27T07:30:12.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9eecf961ca] 2024-05-27T07:30:12.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 19: clone() 2024-05-27T07:30:12.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T07:30:12.719 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a[120450]: 2024-05-27T07:30:12.719 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:12 smithi179 ceph-mon[35888]: daemon mds.a restarted 2024-05-27T07:30:12.776 INFO:tasks.workunit.client.0.smithi043.stdout: 10 15144 6.41 MB/sec execute 13 sec latency 98.737 ms 2024-05-27T07:30:12.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:12 smithi132 ceph-mon[35283]: daemon mds.a restarted 2024-05-27T07:30:13.098 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:12 smithi043 podman[120491]: 2024-05-27 07:30:12.724273178 +0000 UTC m=+0.048715437 container died 862cf9ab7b930fe3f40d82c91b9a3be5b415b2969bd46edbfc35838237dea97f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:30:13.703 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:13 smithi043 podman[120491]: 2024-05-27 07:30:13.573367048 +0000 UTC m=+0.897809305 container remove 862cf9ab7b930fe3f40d82c91b9a3be5b415b2969bd46edbfc35838237dea97f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-a, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:30:13.703 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:13 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-05-27T07:30:13.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:13 smithi043 ceph-mon[31531]: pgmap v548: 129 pgs: 129 active+clean; 547 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.0 MiB/s wr, 200 op/s 2024-05-27T07:30:13.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:13 smithi043 ceph-mon[31531]: osdmap e143: 12 total, 12 up, 12 in 2024-05-27T07:30:13.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:13 smithi043 ceph-mon[31531]: mds.? [v2:172.21.15.43:6834/2818836081,v1:172.21.15.43:6835/2818836081] up:boot 2024-05-27T07:30:13.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:13 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:30:13.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:13 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:13.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:13 smithi043 ceph-mon[31531]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:13 smithi179 ceph-mon[35888]: pgmap v548: 129 pgs: 129 active+clean; 547 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.0 MiB/s wr, 200 op/s 2024-05-27T07:30:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:13 smithi179 ceph-mon[35888]: osdmap e143: 12 total, 12 up, 12 in 2024-05-27T07:30:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:13 smithi179 ceph-mon[35888]: mds.? [v2:172.21.15.43:6834/2818836081,v1:172.21.15.43:6835/2818836081] up:boot 2024-05-27T07:30:13.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:13 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:30:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:13 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:13.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:13 smithi179 ceph-mon[35888]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:13.776 INFO:tasks.workunit.client.0.smithi043.stdout: 10 15259 6.30 MB/sec execute 14 sec latency 79.708 ms 2024-05-27T07:30:13.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:13 smithi132 ceph-mon[35283]: pgmap v548: 129 pgs: 129 active+clean; 547 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.0 MiB/s wr, 200 op/s 2024-05-27T07:30:13.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:13 smithi132 ceph-mon[35283]: osdmap e143: 12 total, 12 up, 12 in 2024-05-27T07:30:13.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:13 smithi132 ceph-mon[35283]: mds.? [v2:172.21.15.43:6834/2818836081,v1:172.21.15.43:6835/2818836081] up:boot 2024-05-27T07:30:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:13 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby 5 up:active 2024-05-27T07:30:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:13 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-27T07:30:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:13 smithi132 ceph-mon[35283]: fsmap cephfs:5 1 up:standby-replay 5 up:active 2024-05-27T07:30:14.453 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:14 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:30:14.453 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:14 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.666s CPU time. 2024-05-27T07:30:14.776 INFO:tasks.workunit.client.0.smithi043.stdout: 10 15376 6.30 MB/sec execute 15 sec latency 77.818 ms 2024-05-27T07:30:15.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:15 smithi043 ceph-mon[31531]: pgmap v550: 129 pgs: 129 active+clean; 547 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-05-27T07:30:15.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:15 smithi179 ceph-mon[35888]: pgmap v550: 129 pgs: 129 active+clean; 547 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-05-27T07:30:15.776 INFO:tasks.workunit.client.0.smithi043.stdout: 10 15489 6.10 MB/sec execute 16 sec latency 65.683 ms 2024-05-27T07:30:15.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:15 smithi132 ceph-mon[35283]: pgmap v550: 129 pgs: 129 active+clean; 547 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-05-27T07:30:16.776 INFO:tasks.workunit.client.0.smithi043.stdout: 10 15590 6.03 MB/sec execute 17 sec latency 170.065 ms 2024-05-27T07:30:17.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:30:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:17.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:17 smithi043 ceph-mon[31531]: pgmap v551: 129 pgs: 129 active+clean; 552 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 202 op/s 2024-05-27T07:30:17.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:17 smithi179 ceph-mon[35888]: pgmap v551: 129 pgs: 129 active+clean; 552 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 202 op/s 2024-05-27T07:30:17.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 15696 5.84 MB/sec execute 18 sec latency 84.037 ms 2024-05-27T07:30:17.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:17 smithi132 ceph-mon[35283]: pgmap v551: 129 pgs: 129 active+clean; 552 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 202 op/s 2024-05-27T07:30:18.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 15839 5.99 MB/sec execute 19 sec latency 119.020 ms 2024-05-27T07:30:19.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:19 smithi043 ceph-mon[31531]: pgmap v552: 129 pgs: 129 active+clean; 552 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 197 op/s 2024-05-27T07:30:19.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:19 smithi179 ceph-mon[35888]: pgmap v552: 129 pgs: 129 active+clean; 552 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 197 op/s 2024-05-27T07:30:19.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 15950 5.89 MB/sec execute 20 sec latency 90.703 ms 2024-05-27T07:30:19.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:19 smithi132 ceph-mon[35283]: pgmap v552: 129 pgs: 129 active+clean; 552 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 197 op/s 2024-05-27T07:30:20.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.098+0000 7f3dab600700 1 -- 172.21.15.43:0/2391424145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3da4110960 msgr2=0x7f3da4110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.098+0000 7f3dab600700 1 --2- 172.21.15.43:0/2391424145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3da4110960 0x7f3da4110d70 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009990 tx=0x7f3d94009ca0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.098+0000 7f3dab600700 1 -- 172.21.15.43:0/2391424145 shutdown_connections 2024-05-27T07:30:20.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.098+0000 7f3dab600700 1 --2- 172.21.15.43:0/2391424145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da410b2d0 0x7f3da4117800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.098+0000 7f3dab600700 1 --2- 172.21.15.43:0/2391424145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3da4110960 0x7f3da4110d70 secure :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009990 tx=0x7f3d94009ca0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.098+0000 7f3dab600700 1 --2- 172.21.15.43:0/2391424145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da410a980 0x7f3da410ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.098+0000 7f3dab600700 1 -- 172.21.15.43:0/2391424145 >> 172.21.15.43:0/2391424145 conn(0x7f3da4077c70 msgr2=0x7f3da40780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:20.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.098+0000 7f3dab600700 1 -- 172.21.15.43:0/2391424145 shutdown_connections 2024-05-27T07:30:20.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.098+0000 7f3dab600700 1 -- 172.21.15.43:0/2391424145 wait complete. 2024-05-27T07:30:20.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.100+0000 7f3dab600700 1 Processor -- start 2024-05-27T07:30:20.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.100+0000 7f3dab600700 1 -- start start 2024-05-27T07:30:20.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.101+0000 7f3dab600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da410a980 0x7f3da41a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.101+0000 7f3dab600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3da410b2d0 0x7f3da41a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.101+0000 7f3dab600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da41a5ce0 0x7f3da41a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.101+0000 7f3dab600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3da411a960 con 0x7f3da410b2d0 2024-05-27T07:30:20.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.101+0000 7f3dab600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3da411a7e0 con 0x7f3da410a980 2024-05-27T07:30:20.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.101+0000 7f3dab600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3da411aae0 con 0x7f3da41a5ce0 2024-05-27T07:30:20.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.101+0000 7f3da8c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da410a980 0x7f3da41a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.101+0000 7f3da8c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da410a980 0x7f3da41a1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:35372/0 (socket says 172.21.15.43:35372) 2024-05-27T07:30:20.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.101+0000 7f3da8c00700 1 -- 172.21.15.43:0/689107952 learned_addr learned my addr 172.21.15.43:0/689107952 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:20.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.101+0000 7f3da9600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da41a5ce0 0x7f3da41a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.102+0000 7f3da3e00700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3da410b2d0 0x7f3da41a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.102+0000 7f3da9600700 1 -- 172.21.15.43:0/689107952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da410a980 msgr2=0x7f3da41a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.102+0000 7f3da9600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da410a980 0x7f3da41a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.102+0000 7f3da9600700 1 -- 172.21.15.43:0/689107952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3da410b2d0 msgr2=0x7f3da41a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.102+0000 7f3da9600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3da410b2d0 0x7f3da41a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.102+0000 7f3da9600700 1 -- 172.21.15.43:0/689107952 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da41a6830 con 0x7f3da41a5ce0 2024-05-27T07:30:20.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.103+0000 7f3da8c00700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da410a980 0x7f3da41a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:20.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.103+0000 7f3da9600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da41a5ce0 0x7f3da41a60f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00efc0 tx=0x7f3d9c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:20.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.103+0000 7f3da1600700 1 -- 172.21.15.43:0/689107952 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c0092e0 con 0x7f3da41a5ce0 2024-05-27T07:30:20.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.103+0000 7f3dab600700 1 -- 172.21.15.43:0/689107952 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da41a6ac0 con 0x7f3da41a5ce0 2024-05-27T07:30:20.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.104+0000 7f3dab600700 1 -- 172.21.15.43:0/689107952 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da41ae260 con 0x7f3da41a5ce0 2024-05-27T07:30:20.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.104+0000 7f3da1600700 1 -- 172.21.15.43:0/689107952 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3d9c004bb0 con 0x7f3da41a5ce0 2024-05-27T07:30:20.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.104+0000 7f3da1600700 1 -- 172.21.15.43:0/689107952 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c004020 con 0x7f3da41a5ce0 2024-05-27T07:30:20.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.112+0000 7f3da1600700 1 -- 172.21.15.43:0/689107952 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3d9c01b030 con 0x7f3da41a5ce0 2024-05-27T07:30:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.113+0000 7f3dba0e5b80 1 Processor -- start 2024-05-27T07:30:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.113+0000 7f3dba0e5b80 1 -- start start 2024-05-27T07:30:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.113+0000 7f3da1600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3d8c0b2a70 0x7f3d8c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.113+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddb0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.114+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.114+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddce120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.114+0000 7f3dba0e5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56425dcdee40 con 0x56425dd1d9f0 2024-05-27T07:30:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.114+0000 7f3dba0e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56425dcce9e0 con 0x56425ddb0e70 2024-05-27T07:30:20.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.114+0000 7f3dba0e5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56425dcd2a20 con 0x56425dc31e40 2024-05-27T07:30:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.114+0000 7f3da1600700 1 -- 172.21.15.43:0/689107952 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f3d9c0e2290 con 0x7f3da41a5ce0 2024-05-27T07:30:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.114+0000 7f3da8c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddb0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.114+0000 7f3da8c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddb0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36396/0 (socket says 172.21.15.43:36396) 2024-05-27T07:30:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.114+0000 7f3da8c00700 1 -- 172.21.15.43:0/565271733 learned_addr learned my addr 172.21.15.43:0/565271733 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.115+0000 7f3da3e00700 1 --2- 172.21.15.43:0/565271733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.115+0000 7f3da9600700 1 --2- 172.21.15.43:0/565271733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.115+0000 7f3da8c00700 1 -- 172.21.15.43:0/565271733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 msgr2=0x56425ddb0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:20.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.115+0000 7f3da8c00700 1 --2- 172.21.15.43:0/565271733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.115+0000 7f3da8c00700 1 -- 172.21.15.43:0/565271733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 msgr2=0x56425ddce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:20.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.115+0000 7f3da8c00700 1 --2- 172.21.15.43:0/565271733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddce120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.115+0000 7f3da8c00700 1 -- 172.21.15.43:0/565271733 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56425dc97420 con 0x56425dd1d9f0 2024-05-27T07:30:20.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.116+0000 7f3da3e00700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3d8c0b2a70 0x7f3d8c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.116+0000 7f3da8c00700 1 --2- 172.21.15.43:0/565271733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddb0330 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f3d94015000 tx=0x7f3d9400ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4fb9674eeba133e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:20.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.117+0000 7f3d92000700 1 -- 172.21.15.43:0/565271733 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94005310 con 0x56425dd1d9f0 2024-05-27T07:30:20.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.117+0000 7f3d92000700 1 -- 172.21.15.43:0/565271733 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3d940054b0 con 0x56425dd1d9f0 2024-05-27T07:30:20.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.117+0000 7f3da3e00700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3d8c0b2a70 0x7f3d8c0b4f80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x56425ddad790 tx=0x7f3d98014040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:20.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.117+0000 7f3d92000700 1 -- 172.21.15.43:0/565271733 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94005310 con 0x56425dd1d9f0 2024-05-27T07:30:20.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.120+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/565271733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 msgr2=0x56425ddb0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.120+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/565271733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddb0330 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f3d94015000 tx=0x7f3d9400ba60 comp rx=0 tx=0).stop 2024-05-27T07:30:20.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.120+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/565271733 shutdown_connections 2024-05-27T07:30:20.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.120+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/565271733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.120+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/565271733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddb0330 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.120+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/565271733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.120+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/565271733 >> 172.21.15.43:0/565271733 conn(0x56425dccb390 msgr2=0x56425ddac4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.121+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/565271733 shutdown_connections 2024-05-27T07:30:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.121+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/565271733 wait complete. 2024-05-27T07:30:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.122+0000 7f3dba0e5b80 1 -- start start 2024-05-27T07:30:20.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.122+0000 7f3dba0e5b80 1 Processor -- start 2024-05-27T07:30:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.123+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.123+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddb85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.123+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddb8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.123+0000 7f3dba0e5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56425dbd4fa0 con 0x56425dd1d9f0 2024-05-27T07:30:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.123+0000 7f3da9600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddb8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.124+0000 7f3da9600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddb8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:35388/0 (socket says 172.21.15.43:35388) 2024-05-27T07:30:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.124+0000 7f3da9600700 1 -- 172.21.15.43:0/2546066226 learned_addr learned my addr 172.21.15.43:0/2546066226 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.124+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/2546066226 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56425dc6f760 con 0x56425ddb0e70 2024-05-27T07:30:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.124+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/2546066226 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56425ddceb30 con 0x56425dc31e40 2024-05-27T07:30:20.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.124+0000 7f3da8c00700 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.124+0000 7f3da3e00700 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddb85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.124+0000 7f3dba0e5b80 10 client.0 _ll_register_callbacks cb 0x56425dc78c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:20.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.125+0000 7f3da9600700 1 -- 172.21.15.43:0/2546066226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 msgr2=0x56425ddb8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:20.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.125+0000 7f3da9600700 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.125+0000 7f3da9600700 1 -- 172.21.15.43:0/2546066226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 msgr2=0x56425ddb85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.125+0000 7f3da9600700 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddb85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.125+0000 7f3da9600700 1 -- 172.21.15.43:0/2546066226 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56425dc1e8d0 con 0x56425ddb0e70 2024-05-27T07:30:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.125+0000 7f3da8c00700 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.125+0000 7f3da9600700 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddb8b00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00ef90 tx=0x7f3d9c0dfb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20f85c231ef1303c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.125+0000 7f3d86a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:30:20.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.126+0000 7f3d86a00700 1 -- 172.21.15.43:0/2546066226 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c04e3b0 con 0x56425ddb0e70 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.126+0000 7f3d86a00700 1 -- 172.21.15.43:0/2546066226 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f3d9c0b1b90 con 0x56425ddb0e70 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.126+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/2546066226 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d940054b0 con 0x56425ddb0e70 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.126+0000 7f3d86a00700 1 -- 172.21.15.43:0/2546066226 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f3d9c0a8d60 con 0x56425ddb0e70 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.126+0000 7f3dba0e5b80 10 client.37121 fetch_fsmap learned FSMap version 129 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.126+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/2546066226 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=4+}) -- 0x56425dc9f060 con 0x56425ddb0e70 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.127+0000 7f3d86a00700 1 -- 172.21.15.43:0/2546066226 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3d940054b0 con 0x56425ddb0e70 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.127+0000 7f3d86a00700 1 -- 172.21.15.43:0/2546066226 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 129) ==== 4812+0+0 (secure 0 0 0) 0x7f3d9c0a4bc0 con 0x56425ddb0e70 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.127+0000 7f3dba0e5b80 10 client.37121 fetch_fsmap finished waiting for FSMap version 129 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.127+0000 7f3dba0e5b80 10 client.37121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.127+0000 7f3dba0e5b80 20 client.37121 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:20.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.127+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56425dddbef0 0x56425ddde3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.127+0000 7f3dba0e5b80 4 client.37121 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:20.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.127+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/2546066226 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56425dce0d90 con 0x56425dddbef0 2024-05-27T07:30:20.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.128+0000 7f3da8c00700 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56425dddbef0 0x56425ddde3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.131+0000 7f3da8c00700 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56425dddbef0 0x56425ddde3e0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fde0916b4d575ee server_cookie=320917f89de5cdf4 in_seq=0 out_seq=0 2024-05-27T07:30:20.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.131+0000 7f3d86a00700 10 client.37121 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:20.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.132+0000 7f3d86a00700 1 -- 172.21.15.43:0/2546066226 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56425dce0d90 con 0x56425dddbef0 2024-05-27T07:30:20.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.132+0000 7f3d86a00700 10 client.37121 handle_command_reply: tid=0 2024-05-27T07:30:20.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.132+0000 7f3dba0e5b80 1 client.37121 shutdown 2024-05-27T07:30:20.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/2546066226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 msgr2=0x56425ddb8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddb8b00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00ef90 tx=0x7f3d9c0dfb90 comp rx=0 tx=0).stop 2024-05-27T07:30:20.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/2546066226 shutdown_connections 2024-05-27T07:30:20.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddb8b00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56425dddbef0 0x56425ddde3e0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/2546066226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddb85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/2546066226 >> 172.21.15.43:0/2546066226 conn(0x56425dccb390 msgr2=0x56425ddb0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:20.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3d86a00700 0 client.37121 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:20.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/2546066226 shutdown_connections 2024-05-27T07:30:20.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/2546066226 wait complete. 2024-05-27T07:30:20.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.133+0000 7f3dba0e5b80 20 client.37121 trim_cache size 0 max 16384 2024-05-27T07:30:20.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.167+0000 7f3dba0e5b80 1 Processor -- start 2024-05-27T07:30:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.167+0000 7f3dba0e5b80 1 -- start start 2024-05-27T07:30:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.167+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.167+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddd7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.167+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddcd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.167+0000 7f3dba0e5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56425ddde630 con 0x56425dd1d9f0 2024-05-27T07:30:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.167+0000 7f3dba0e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8800f9e0 con 0x56425ddb0e70 2024-05-27T07:30:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.167+0000 7f3dba0e5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8800d650 con 0x56425dc31e40 2024-05-27T07:30:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.168+0000 7f3da8c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.168+0000 7f3da9600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddcd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.168+0000 7f3da8c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:44056/0 (socket says 172.21.15.43:44056) 2024-05-27T07:30:20.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.168+0000 7f3da8c00700 1 -- 172.21.15.43:0/379175617 learned_addr learned my addr 172.21.15.43:0/379175617 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:20.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.168+0000 7f3da3e00700 1 --2- 172.21.15.43:0/379175617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddd7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.169+0000 7f3da9600700 1 -- 172.21.15.43:0/379175617 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 msgr2=0x56425ddb07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:20.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.169+0000 7f3da9600700 1 --2- 172.21.15.43:0/379175617 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.169+0000 7f3da9600700 1 -- 172.21.15.43:0/379175617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 msgr2=0x56425ddd7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:20.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.169+0000 7f3da9600700 1 --2- 172.21.15.43:0/379175617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddd7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.169+0000 7f3da9600700 1 -- 172.21.15.43:0/379175617 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56425dda8240 con 0x56425ddb0e70 2024-05-27T07:30:20.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.169+0000 7f3da3e00700 1 --2- 172.21.15.43:0/379175617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddd7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.170+0000 7f3da9600700 1 --2- 172.21.15.43:0/379175617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddcd120 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0c38c0 tx=0x7f3d9c0a5110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=208ebc45f39aa148 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.170+0000 7f3d91600700 1 -- 172.21.15.43:0/379175617 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c0b2020 con 0x56425ddb0e70 2024-05-27T07:30:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.170+0000 7f3d91600700 1 -- 172.21.15.43:0/379175617 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3d9c0a9250 con 0x56425ddb0e70 2024-05-27T07:30:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.170+0000 7f3d91600700 1 -- 172.21.15.43:0/379175617 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c0b1b90 con 0x56425ddb0e70 2024-05-27T07:30:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.171+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/379175617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 msgr2=0x56425ddcd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.171+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/379175617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddcd120 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0c38c0 tx=0x7f3d9c0a5110 comp rx=0 tx=0).stop 2024-05-27T07:30:20.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.171+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/379175617 shutdown_connections 2024-05-27T07:30:20.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.171+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/379175617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddcd120 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.171+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/379175617 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddb07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.171+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/379175617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddd7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.171+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/379175617 >> 172.21.15.43:0/379175617 conn(0x56425dccb390 msgr2=0x56425ddb00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:20.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.172+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/379175617 shutdown_connections 2024-05-27T07:30:20.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.172+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/379175617 wait complete. 2024-05-27T07:30:20.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.172+0000 7f3dba0e5b80 1 -- start start 2024-05-27T07:30:20.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.173+0000 7f3dba0e5b80 1 Processor -- start 2024-05-27T07:30:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.173+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddd4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.173+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddcc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.173+0000 7f3dba0e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.173+0000 7f3dba0e5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8800dd50 con 0x56425dd1d9f0 2024-05-27T07:30:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.173+0000 7f3dba0e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8800f2e0 con 0x56425ddb0e70 2024-05-27T07:30:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.173+0000 7f3dba0e5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8800e450 con 0x56425dc31e40 2024-05-27T07:30:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.173+0000 7f3dba0e5b80 10 client.0 _ll_register_callbacks cb 0x56425dc78c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.174+0000 7f3da3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddcc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.174+0000 7f3da3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddcc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36424/0 (socket says 172.21.15.43:36424) 2024-05-27T07:30:20.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.174+0000 7f3da9600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.174+0000 7f3da3e00700 1 -- 172.21.15.43:0/921090810 learned_addr learned my addr 172.21.15.43:0/921090810 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:20.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.174+0000 7f3da8c00700 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddd4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.175+0000 7f3da3e00700 1 -- 172.21.15.43:0/921090810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 msgr2=0x56425ddd4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.175+0000 7f3da3e00700 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddd4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.175+0000 7f3da3e00700 1 -- 172.21.15.43:0/921090810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 msgr2=0x56425ddccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.175+0000 7f3da3e00700 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.175+0000 7f3da3e00700 1 -- 172.21.15.43:0/921090810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56425ddb8e70 con 0x56425dd1d9f0 2024-05-27T07:30:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.175+0000 7f3da3e00700 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddcc6f0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f3da406d110 tx=0x7f3d98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925fd8fbdbc00e88 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.175+0000 7f3d86a00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:30:20.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.176+0000 7f3d86a00700 1 -- 172.21.15.43:0/921090810 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9802d060 con 0x56425dd1d9f0 2024-05-27T07:30:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.176+0000 7f3d86a00700 1 -- 172.21.15.43:0/921090810 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f3d9801c450 con 0x56425dd1d9f0 2024-05-27T07:30:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.176+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/921090810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d9c0a9250 con 0x56425dd1d9f0 2024-05-27T07:30:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.177+0000 7f3d86a00700 1 -- 172.21.15.43:0/921090810 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f3d9801c650 con 0x56425dd1d9f0 2024-05-27T07:30:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.177+0000 7f3dba0e5b80 10 client.17733 fetch_fsmap learned FSMap version 129 2024-05-27T07:30:20.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.177+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/921090810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=4+}) -- 0x56425ddd3920 con 0x56425dd1d9f0 2024-05-27T07:30:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.178+0000 7f3d86a00700 1 -- 172.21.15.43:0/921090810 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3d9c0a9250 con 0x56425dd1d9f0 2024-05-27T07:30:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.178+0000 7f3d86a00700 1 -- 172.21.15.43:0/921090810 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 129) ==== 4812+0+0 (secure 0 0 0) 0x7f3d98029730 con 0x56425dd1d9f0 2024-05-27T07:30:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.178+0000 7f3dba0e5b80 10 client.17733 fetch_fsmap finished waiting for FSMap version 129 2024-05-27T07:30:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.178+0000 7f3dba0e5b80 10 client.17733 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.178+0000 7f3dba0e5b80 20 client.17733 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.178+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56425de46b50 0x56425de48f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.178+0000 7f3dba0e5b80 4 client.17733 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:30:20.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.178+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/921090810 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56425dcda570 con 0x56425de46b50 2024-05-27T07:30:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.178+0000 7f3da8c00700 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56425de46b50 0x56425de48f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:20.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.180+0000 7f3da8c00700 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56425de46b50 0x56425de48f70 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d063ed7adf6bfccd server_cookie=5efa13076c07beb0 in_seq=0 out_seq=0 2024-05-27T07:30:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.180+0000 7f3d86a00700 10 client.17733 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.180+0000 7f3d86a00700 1 -- 172.21.15.43:0/921090810 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56425dcda570 con 0x56425de46b50 2024-05-27T07:30:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.180+0000 7f3d86a00700 10 client.17733 handle_command_reply: tid=0 2024-05-27T07:30:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.180+0000 7f3dba0e5b80 1 client.17733 shutdown 2024-05-27T07:30:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.180+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/921090810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 msgr2=0x56425ddcc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.180+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddcc6f0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f3da406d110 tx=0x7f3d98004750 comp rx=0 tx=0).stop 2024-05-27T07:30:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.181+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/921090810 shutdown_connections 2024-05-27T07:30:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.181+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56425ddb0e70 0x56425ddccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.181+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56425dc31e40 0x56425ddd4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.181+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56425de46b50 0x56425de48f70 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.181+0000 7f3d86a00700 0 client.17733 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.181+0000 7f3dba0e5b80 1 --2- 172.21.15.43:0/921090810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56425dd1d9f0 0x56425ddcc6f0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.181+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/921090810 >> 172.21.15.43:0/921090810 conn(0x56425dccb390 msgr2=0x56425ddb3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.181+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/921090810 shutdown_connections 2024-05-27T07:30:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.181+0000 7f3dba0e5b80 1 -- 172.21.15.43:0/921090810 wait complete. 2024-05-27T07:30:20.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.181+0000 7f3dba0e5b80 20 client.17733 trim_cache size 0 max 16384 2024-05-27T07:30:20.173 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:30:20.173 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:30:20.173 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:30:20.173 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:30:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.183+0000 7f3dab600700 1 -- 172.21.15.43:0/689107952 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3d8c0b2a70 msgr2=0x7f3d8c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.183+0000 7f3dab600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3d8c0b2a70 0x7f3d8c0b4f80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x56425ddad790 tx=0x7f3d98014040 comp rx=0 tx=0).stop 2024-05-27T07:30:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.183+0000 7f3dab600700 1 -- 172.21.15.43:0/689107952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da41a5ce0 msgr2=0x7f3da41a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.183+0000 7f3dab600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da41a5ce0 0x7f3da41a60f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00efc0 tx=0x7f3d9c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.183+0000 7f3dab600700 1 -- 172.21.15.43:0/689107952 shutdown_connections 2024-05-27T07:30:20.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.183+0000 7f3dab600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f3d8c0b2a70 0x7f3d8c0b4f80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.183+0000 7f3dab600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da41a5ce0 0x7f3da41a60f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.183+0000 7f3dab600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da410a980 0x7f3da41a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.183+0000 7f3dab600700 1 --2- 172.21.15.43:0/689107952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3da410b2d0 0x7f3da41a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:20.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.183+0000 7f3dab600700 1 -- 172.21.15.43:0/689107952 >> 172.21.15.43:0/689107952 conn(0x7f3da4077c70 msgr2=0x7f3da407be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:20.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.184+0000 7f3dab600700 1 -- 172.21.15.43:0/689107952 shutdown_connections 2024-05-27T07:30:20.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:20.184+0000 7f3dab600700 1 -- 172.21.15.43:0/689107952 wait complete. 2024-05-27T07:30:20.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 16097 6.00 MB/sec execute 21 sec latency 88.035 ms 2024-05-27T07:30:20.970 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:30:20.971 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:30:21.457 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:21.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:21 smithi043 ceph-mon[31531]: pgmap v553: 129 pgs: 129 active+clean; 552 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 197 op/s 2024-05-27T07:30:21.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:21 smithi179 ceph-mon[35888]: pgmap v553: 129 pgs: 129 active+clean; 552 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 197 op/s 2024-05-27T07:30:21.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 16173 5.87 MB/sec execute 22 sec latency 401.578 ms 2024-05-27T07:30:21.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:21 smithi132 ceph-mon[35283]: pgmap v553: 129 pgs: 129 active+clean; 552 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 197 op/s 2024-05-27T07:30:22.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 16280 5.93 MB/sec execute 23 sec latency 93.724 ms 2024-05-27T07:30:23.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:23 smithi043 ceph-mon[31531]: pgmap v554: 129 pgs: 129 active+clean; 564 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.3 MiB/s wr, 192 op/s 2024-05-27T07:30:23.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:23 smithi179 ceph-mon[35888]: pgmap v554: 129 pgs: 129 active+clean; 564 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.3 MiB/s wr, 192 op/s 2024-05-27T07:30:23.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 16414 5.94 MB/sec execute 24 sec latency 89.280 ms 2024-05-27T07:30:23.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.846+0000 7fac54c00700 1 -- 172.21.15.43:0/703917636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac501043f0 msgr2=0x7fac50104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:23.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.846+0000 7fac54c00700 1 --2- 172.21.15.43:0/703917636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac501043f0 0x7fac50104800 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fac400098f0 tx=0x7fac40009c00 comp rx=0 tx=0).stop 2024-05-27T07:30:23.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.847+0000 7fac54c00700 1 -- 172.21.15.43:0/703917636 shutdown_connections 2024-05-27T07:30:23.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.847+0000 7fac54c00700 1 --2- 172.21.15.43:0/703917636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac50109670 0x7fac50110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:23.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.847+0000 7fac54c00700 1 --2- 172.21.15.43:0/703917636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac501043f0 0x7fac50104800 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:23.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.847+0000 7fac54c00700 1 --2- 172.21.15.43:0/703917636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac50108d20 0x7fac50109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:23.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.847+0000 7fac54c00700 1 -- 172.21.15.43:0/703917636 >> 172.21.15.43:0/703917636 conn(0x7fac50079d00 msgr2=0x7fac5007a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:23.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.847+0000 7fac54c00700 1 -- 172.21.15.43:0/703917636 shutdown_connections 2024-05-27T07:30:23.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.847+0000 7fac54c00700 1 -- 172.21.15.43:0/703917636 wait complete. 2024-05-27T07:30:23.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:23 smithi132 ceph-mon[35283]: pgmap v554: 129 pgs: 129 active+clean; 564 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.3 MiB/s wr, 192 op/s 2024-05-27T07:30:23.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.849+0000 7fac54c00700 1 Processor -- start 2024-05-27T07:30:23.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.849+0000 7fac54c00700 1 -- start start 2024-05-27T07:30:23.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.850+0000 7fac54c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac501043f0 0x7fac5019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:23.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.850+0000 7fac54c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac50108d20 0x7fac5019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:23.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.850+0000 7fac54c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac50109670 0x7fac501a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:23.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.850+0000 7fac54c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fac50114210 con 0x7fac501043f0 2024-05-27T07:30:23.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.850+0000 7fac54c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fac50114090 con 0x7fac50108d20 2024-05-27T07:30:23.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.850+0000 7fac54c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fac50114390 con 0x7fac50109670 2024-05-27T07:30:23.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.851+0000 7fac4f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac501043f0 0x7fac5019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:23.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.851+0000 7fac4f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac501043f0 0x7fac5019cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36440/0 (socket says 172.21.15.43:36440) 2024-05-27T07:30:23.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.851+0000 7fac4f400700 1 -- 172.21.15.43:0/1845351749 learned_addr learned my addr 172.21.15.43:0/1845351749 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:23.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.852+0000 7fac4ea00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac50108d20 0x7fac5019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:23.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.852+0000 7fac4fe00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac50109670 0x7fac501a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.852+0000 7fac4f400700 1 -- 172.21.15.43:0/1845351749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac50109670 msgr2=0x7fac501a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.852+0000 7fac4f400700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac50109670 0x7fac501a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.852+0000 7fac4f400700 1 -- 172.21.15.43:0/1845351749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac50108d20 msgr2=0x7fac5019d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.852+0000 7fac4f400700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac50108d20 0x7fac5019d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.852+0000 7fac4f400700 1 -- 172.21.15.43:0/1845351749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac501a1d80 con 0x7fac501043f0 2024-05-27T07:30:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.853+0000 7fac4fe00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac50109670 0x7fac501a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.853+0000 7fac4f400700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac501043f0 0x7fac5019cf00 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fac40009fd0 tx=0x7fac4000fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.854+0000 7fac47e00700 1 -- 172.21.15.43:0/1845351749 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4000b710 con 0x7fac501043f0 2024-05-27T07:30:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.854+0000 7fac47e00700 1 -- 172.21.15.43:0/1845351749 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac40005040 con 0x7fac501043f0 2024-05-27T07:30:23.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.854+0000 7fac47e00700 1 -- 172.21.15.43:0/1845351749 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4000b710 con 0x7fac501043f0 2024-05-27T07:30:23.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.854+0000 7fac54c00700 1 -- 172.21.15.43:0/1845351749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac501a2010 con 0x7fac501043f0 2024-05-27T07:30:23.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.854+0000 7fac54c00700 1 -- 172.21.15.43:0/1845351749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac501a23e0 con 0x7fac501043f0 2024-05-27T07:30:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.857+0000 7fac54c00700 1 -- 172.21.15.43:0/1845351749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac501058a0 con 0x7fac501043f0 2024-05-27T07:30:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.859+0000 7fac47e00700 1 -- 172.21.15.43:0/1845351749 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fac4000b8b0 con 0x7fac501043f0 2024-05-27T07:30:23.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.862+0000 7fac47e00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac340b2c20 0x7fac340b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:23.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.862+0000 7fac47e00700 1 -- 172.21.15.43:0/1845351749 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7fac400e3750 con 0x7fac501043f0 2024-05-27T07:30:23.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.866+0000 7fac4ea00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac340b2c20 0x7fac340b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.867+0000 7fac4ea00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac340b2c20 0x7fac340b5130 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fac50108b90 tx=0x7fac3c0093e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:23.870+0000 7fac47e00700 1 -- 172.21.15.43:0/1845351749 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fac40013030 con 0x7fac501043f0 2024-05-27T07:30:24.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.342+0000 7fac54c00700 1 -- 172.21.15.43:0/1845351749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac501a9990 con 0x7fac501043f0 2024-05-27T07:30:24.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.343+0000 7fac47e00700 1 -- 172.21.15.43:0/1845351749 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+6964 (secure 0 0 0) 0x7fac400aae50 con 0x7fac501043f0 2024-05-27T07:30:24.336 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:30:24.336 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:30:12.440962+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_17718":{"gid":17718,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/2818836081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":2818836081},{"type":"v1","addr":"172.21.15.43:6835","nonce":2818836081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[24484,34403,34424,34448,34484]},"id":1}]} 2024-05-27T07:30:24.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 -- 172.21.15.43:0/1845351749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac340b2c20 msgr2=0x7fac340b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:24.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac340b2c20 0x7fac340b5130 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fac50108b90 tx=0x7fac3c0093e0 comp rx=0 tx=0).stop 2024-05-27T07:30:24.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 -- 172.21.15.43:0/1845351749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac501043f0 msgr2=0x7fac5019cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:24.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac501043f0 0x7fac5019cf00 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fac40009fd0 tx=0x7fac4000fa50 comp rx=0 tx=0).stop 2024-05-27T07:30:24.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 -- 172.21.15.43:0/1845351749 shutdown_connections 2024-05-27T07:30:24.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fac340b2c20 0x7fac340b5130 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:24.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac50109670 0x7fac501a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:24.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac501043f0 0x7fac5019cf00 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:24.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 --2- 172.21.15.43:0/1845351749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac50108d20 0x7fac5019d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:24.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 -- 172.21.15.43:0/1845351749 >> 172.21.15.43:0/1845351749 conn(0x7fac50079d00 msgr2=0x7fac501027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:24.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 -- 172.21.15.43:0/1845351749 shutdown_connections 2024-05-27T07:30:24.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:24.347+0000 7fac54c00700 1 -- 172.21.15.43:0/1845351749 wait complete. 2024-05-27T07:30:24.341 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 129 2024-05-27T07:30:24.703 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:24 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-05-27T07:30:24.703 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:24 smithi043 systemd[1]: Stopped Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:30:24.703 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:24 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.666s CPU time. 2024-05-27T07:30:24.703 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:24 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-05-27T07:30:24.703 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:24 smithi043 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-05-27T07:30:24.703 INFO:journalctl@ceph.mds.a.smithi043.stdout:May 27 07:30:24 smithi043 systemd[1]: Failed to start Ceph mds.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:30:24.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:24 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1845351749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:30:24.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:24 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1845351749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:30:24.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 16556 6.23 MB/sec execute 25 sec latency 80.107 ms 2024-05-27T07:30:24.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:24 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1845351749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:30:25.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:25 smithi179 ceph-mon[35888]: pgmap v555: 129 pgs: 129 active+clean; 564 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-05-27T07:30:25.745 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:25 smithi132 ceph-mon[35283]: pgmap v555: 129 pgs: 129 active+clean; 564 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-05-27T07:30:25.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 16680 6.40 MB/sec execute 26 sec latency 75.922 ms 2024-05-27T07:30:25.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:25 smithi043 ceph-mon[31531]: pgmap v555: 129 pgs: 129 active+clean; 564 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-05-27T07:30:26.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 16803 6.48 MB/sec execute 27 sec latency 80.121 ms 2024-05-27T07:30:27.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 16873 6.40 MB/sec execute 28 sec latency 364.071 ms 2024-05-27T07:30:27.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:27 smithi132 ceph-mon[35283]: pgmap v556: 129 pgs: 129 active+clean; 567 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 243 op/s 2024-05-27T07:30:27.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:27 smithi043 ceph-mon[31531]: pgmap v556: 129 pgs: 129 active+clean; 567 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 243 op/s 2024-05-27T07:30:27.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:27 smithi179 ceph-mon[35888]: pgmap v556: 129 pgs: 129 active+clean; 567 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 243 op/s 2024-05-27T07:30:28.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 16935 6.29 MB/sec execute 29 sec latency 457.772 ms 2024-05-27T07:30:29.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 17053 6.28 MB/sec execute 30 sec latency 90.255 ms 2024-05-27T07:30:29.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:29 smithi132 ceph-mon[35283]: pgmap v557: 129 pgs: 129 active+clean; 567 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.1 MiB/s wr, 159 op/s 2024-05-27T07:30:29.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:29 smithi043 ceph-mon[31531]: pgmap v557: 129 pgs: 129 active+clean; 567 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.1 MiB/s wr, 159 op/s 2024-05-27T07:30:29.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:29 smithi179 ceph-mon[35888]: pgmap v557: 129 pgs: 129 active+clean; 567 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.1 MiB/s wr, 159 op/s 2024-05-27T07:30:30.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 17176 6.23 MB/sec execute 31 sec latency 80.975 ms 2024-05-27T07:30:31.757 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:31 smithi043 ceph-mon[31531]: pgmap v558: 129 pgs: 129 active+clean; 567 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.1 MiB/s wr, 159 op/s 2024-05-27T07:30:31.757 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:31 smithi043 ceph-mon[31531]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:31.757 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:31 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:31.777 INFO:tasks.workunit.client.0.smithi043.stdout: 10 17291 6.14 MB/sec execute 32 sec latency 116.188 ms 2024-05-27T07:30:31.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:31 smithi132 ceph-mon[35283]: pgmap v558: 129 pgs: 129 active+clean; 567 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.1 MiB/s wr, 159 op/s 2024-05-27T07:30:31.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:31 smithi132 ceph-mon[35283]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:31.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:31 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:31 smithi179 ceph-mon[35888]: pgmap v558: 129 pgs: 129 active+clean; 567 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.1 MiB/s wr, 159 op/s 2024-05-27T07:30:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:31 smithi179 ceph-mon[35888]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-05-27T07:30:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:31 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:32.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:30:32.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 17382 6.02 MB/sec execute 33 sec latency 104.176 ms 2024-05-27T07:30:32.870 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:33.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:33 smithi043 ceph-mon[31531]: pgmap v559: 129 pgs: 129 active+clean; 575 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 4.4 MiB/s wr, 228 op/s 2024-05-27T07:30:33.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 17463 5.88 MB/sec execute 34 sec latency 105.006 ms 2024-05-27T07:30:33.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:33 smithi132 ceph-mon[35283]: pgmap v559: 129 pgs: 129 active+clean; 575 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 4.4 MiB/s wr, 228 op/s 2024-05-27T07:30:33.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:33 smithi179 ceph-mon[35888]: pgmap v559: 129 pgs: 129 active+clean; 575 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 4.4 MiB/s wr, 228 op/s 2024-05-27T07:30:34.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 17559 5.83 MB/sec execute 35 sec latency 102.567 ms 2024-05-27T07:30:35.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.236+0000 7f5d97e00700 1 -- 172.21.15.43:0/218762819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d98102b40 msgr2=0x7f5d98110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:35.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.236+0000 7f5d97e00700 1 --2- 172.21.15.43:0/218762819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d98102b40 0x7f5d98110e90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009f90 tx=0x7f5d9000b330 comp rx=0 tx=0).stop 2024-05-27T07:30:35.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.237+0000 7f5d97e00700 1 -- 172.21.15.43:0/218762819 shutdown_connections 2024-05-27T07:30:35.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.237+0000 7f5d97e00700 1 --2- 172.21.15.43:0/218762819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d98102b40 0x7f5d98110e90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.237+0000 7f5d97e00700 1 --2- 172.21.15.43:0/218762819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d981091b0 0x7f5d981095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.237+0000 7f5d97e00700 1 --2- 172.21.15.43:0/218762819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d981021f0 0x7f5d98102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.237+0000 7f5d97e00700 1 -- 172.21.15.43:0/218762819 >> 172.21.15.43:0/218762819 conn(0x7f5d98077ee0 msgr2=0x7f5d98078320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:35.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.237+0000 7f5d97e00700 1 -- 172.21.15.43:0/218762819 shutdown_connections 2024-05-27T07:30:35.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.237+0000 7f5d97e00700 1 -- 172.21.15.43:0/218762819 wait complete. 2024-05-27T07:30:35.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.239+0000 7f5d97e00700 1 Processor -- start 2024-05-27T07:30:35.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.239+0000 7f5d97e00700 1 -- start start 2024-05-27T07:30:35.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.239+0000 7f5d97e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d981021f0 0x7f5d981a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.239+0000 7f5d97e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d98102b40 0x7f5d981a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.239+0000 7f5d97e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d981091b0 0x7f5d981acf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.239+0000 7f5d97e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5d981141e0 con 0x7f5d98102b40 2024-05-27T07:30:35.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.239+0000 7f5d97e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5d98114060 con 0x7f5d981021f0 2024-05-27T07:30:35.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.239+0000 7f5d97e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5d98114360 con 0x7f5d981091b0 2024-05-27T07:30:35.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.240+0000 7f5d96000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d98102b40 0x7f5d981a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.240+0000 7f5d96000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d98102b40 0x7f5d981a2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46420/0 (socket says 172.21.15.43:46420) 2024-05-27T07:30:35.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.240+0000 7f5d96000700 1 -- 172.21.15.43:0/1768620500 learned_addr learned my addr 172.21.15.43:0/1768620500 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:35.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.240+0000 7f5d97400700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d981091b0 0x7f5d981acf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.240+0000 7f5d96a00700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d981021f0 0x7f5d981a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.241+0000 7f5d96000700 1 -- 172.21.15.43:0/1768620500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d981091b0 msgr2=0x7f5d981acf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:35.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.241+0000 7f5d96000700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d981091b0 0x7f5d981acf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.241+0000 7f5d96000700 1 -- 172.21.15.43:0/1768620500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d981021f0 msgr2=0x7f5d981a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:35.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.241+0000 7f5d96000700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d981021f0 0x7f5d981a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.241+0000 7f5d96000700 1 -- 172.21.15.43:0/1768620500 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d981ad480 con 0x7f5d98102b40 2024-05-27T07:30:35.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.241+0000 7f5d96000700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d98102b40 0x7f5d981a2460 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400ec10 tx=0x7f5d8400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:35.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.242+0000 7f5d8f400700 1 -- 172.21.15.43:0/1768620500 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8400cd40 con 0x7f5d98102b40 2024-05-27T07:30:35.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.242+0000 7f5d8f400700 1 -- 172.21.15.43:0/1768620500 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d840044b0 con 0x7f5d98102b40 2024-05-27T07:30:35.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.242+0000 7f5d8f400700 1 -- 172.21.15.43:0/1768620500 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d84010430 con 0x7f5d98102b40 2024-05-27T07:30:35.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.242+0000 7f5d97e00700 1 -- 172.21.15.43:0/1768620500 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d981ad710 con 0x7f5d98102b40 2024-05-27T07:30:35.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.242+0000 7f5d97e00700 1 -- 172.21.15.43:0/1768620500 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d981adbd0 con 0x7f5d98102b40 2024-05-27T07:30:35.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.250+0000 7f5d8f400700 1 -- 172.21.15.43:0/1768620500 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5d840037a0 con 0x7f5d98102b40 2024-05-27T07:30:35.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.250+0000 7f5dac15ab80 1 Processor -- start 2024-05-27T07:30:35.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.251+0000 7f5dac15ab80 1 -- start start 2024-05-27T07:30:35.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.251+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbd059f0 0x55debbd99750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.251+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbc19e40 0x55debbd99c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.252+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbda32c0 0x55debbda56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.252+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55debbcc6e40 con 0x55debbda32c0 2024-05-27T07:30:35.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.252+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55debbcb69e0 con 0x55debbd059f0 2024-05-27T07:30:35.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.252+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55debbcbaa20 con 0x55debbc19e40 2024-05-27T07:30:35.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.252+0000 7f5d96000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbda32c0 0x55debbda56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.252+0000 7f5d96000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbda32c0 0x55debbda56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46430/0 (socket says 172.21.15.43:46430) 2024-05-27T07:30:35.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.252+0000 7f5d96000700 1 -- 172.21.15.43:0/3101965875 learned_addr learned my addr 172.21.15.43:0/3101965875 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:35.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.253+0000 7f5d96000700 1 -- 172.21.15.43:0/3101965875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbc19e40 msgr2=0x55debbd99c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:30:35.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.253+0000 7f5d96000700 1 --2- 172.21.15.43:0/3101965875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbc19e40 0x55debbd99c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.253+0000 7f5d96000700 1 -- 172.21.15.43:0/3101965875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbd059f0 msgr2=0x55debbd99750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:35.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.253+0000 7f5d96000700 1 --2- 172.21.15.43:0/3101965875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbd059f0 0x55debbd99750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.253+0000 7f5d96000700 1 -- 172.21.15.43:0/3101965875 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55debbc7f420 con 0x55debbda32c0 2024-05-27T07:30:35.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.253+0000 7f5d8f400700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d7c0b2a70 0x7f5d7c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.253+0000 7f5d8f400700 1 -- 172.21.15.43:0/1768620500 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f5d840e2d30 con 0x7f5d98102b40 2024-05-27T07:30:35.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.253+0000 7f5d96000700 1 --2- 172.21.15.43:0/3101965875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbda32c0 0x55debbda56e0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5d840e0010 tx=0x7f5d840e6d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76ab56ddfb32940 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:35.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.254+0000 7f5d97400700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d7c0b2a70 0x7f5d7c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.254+0000 7f5d83e00700 1 -- 172.21.15.43:0/3101965875 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d840a5b20 con 0x55debbda32c0 2024-05-27T07:30:35.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.254+0000 7f5d83e00700 1 -- 172.21.15.43:0/3101965875 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d84091070 con 0x55debbda32c0 2024-05-27T07:30:35.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.254+0000 7f5d83e00700 1 -- 172.21.15.43:0/3101965875 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d840a5b20 con 0x55debbda32c0 2024-05-27T07:30:35.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.256+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3101965875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbda32c0 msgr2=0x55debbda56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:35.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.256+0000 7f5d97400700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d7c0b2a70 0x7f5d7c0b4f80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009840 tx=0x7f5d90009cc0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:35.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.256+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/3101965875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbda32c0 0x55debbda56e0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5d840e0010 tx=0x7f5d840e6d10 comp rx=0 tx=0).stop 2024-05-27T07:30:35.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.256+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3101965875 shutdown_connections 2024-05-27T07:30:35.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.256+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/3101965875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbda32c0 0x55debbda56e0 secure :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5d840e0010 tx=0x7f5d840e6d10 comp rx=0 tx=0).stop 2024-05-27T07:30:35.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.256+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/3101965875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbd059f0 0x55debbd99750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.256+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/3101965875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbc19e40 0x55debbd99c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.256+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3101965875 >> 172.21.15.43:0/3101965875 conn(0x55debbcb3390 msgr2=0x55debbd94710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:35.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.256+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3101965875 shutdown_connections 2024-05-27T07:30:35.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.256+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3101965875 wait complete. 2024-05-27T07:30:35.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.257+0000 7f5dac15ab80 1 -- start start 2024-05-27T07:30:35.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.257+0000 7f5dac15ab80 1 Processor -- start 2024-05-27T07:30:35.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.258+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbc19e40 0x55debbda03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.258+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.258+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 0x55debbdaf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.258+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55debbbbcfa0 con 0x55debbd059f0 2024-05-27T07:30:35.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.258+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55debbc57760 con 0x55debbda0e70 2024-05-27T07:30:35.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.258+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55debbda5e00 con 0x55debbc19e40 2024-05-27T07:30:35.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.258+0000 7f5dac15ab80 10 client.0 _ll_register_callbacks cb 0x55debbc60c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:35.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.258+0000 7f5d96000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.259+0000 7f5d96a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbc19e40 0x55debbda03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.259+0000 7f5d96000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46438/0 (socket says 172.21.15.43:46438) 2024-05-27T07:30:35.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.259+0000 7f5d96000700 1 -- 172.21.15.43:0/88479703 learned_addr learned my addr 172.21.15.43:0/88479703 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:35.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.259+0000 7f5d97400700 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 0x55debbdaf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.259+0000 7f5d96000700 1 -- 172.21.15.43:0/88479703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbc19e40 msgr2=0x55debbda03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.259+0000 7f5d96000700 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbc19e40 0x55debbda03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.259+0000 7f5d96000700 1 -- 172.21.15.43:0/88479703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 msgr2=0x55debbdaf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.259+0000 7f5d96000700 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 0x55debbdaf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.259+0000 7f5d96000700 1 -- 172.21.15.43:0/88479703 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55debbc068d0 con 0x55debbd059f0 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.260+0000 7f5d96000700 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda0930 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009b10 tx=0x7f5d840165e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbe90b73d26394e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.260+0000 7f5d80c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.260+0000 7f5d80c00700 1 -- 172.21.15.43:0/88479703 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d840e0560 con 0x55debbd059f0 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.260+0000 7f5dac15ab80 1 -- 172.21.15.43:0/88479703 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d84091070 con 0x55debbd059f0 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.260+0000 7f5d80c00700 1 -- 172.21.15.43:0/88479703 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f5d840aa0d0 con 0x55debbd059f0 2024-05-27T07:30:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.261+0000 7f5d80c00700 1 -- 172.21.15.43:0/88479703 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) ==== 24+0+0 (secure 0 0 0) 0x7f5d840a93d0 con 0x55debbd059f0 2024-05-27T07:30:35.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.261+0000 7f5dac15ab80 10 client.17751 fetch_fsmap learned FSMap version 130 2024-05-27T07:30:35.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.261+0000 7f5dac15ab80 1 -- 172.21.15.43:0/88479703 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=4+}) -- 0x55debbc87060 con 0x55debbd059f0 2024-05-27T07:30:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.261+0000 7f5d80c00700 1 -- 172.21.15.43:0/88479703 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d84091070 con 0x55debbd059f0 2024-05-27T07:30:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.261+0000 7f5d80c00700 1 -- 172.21.15.43:0/88479703 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 130) ==== 4219+0+0 (secure 0 0 0) 0x7f5d840b2630 con 0x55debbd059f0 2024-05-27T07:30:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.262+0000 7f5dac15ab80 10 client.17751 fetch_fsmap finished waiting for FSMap version 130 2024-05-27T07:30:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.262+0000 7f5dac15ab80 10 client.17751 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:30:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.262+0000 7f5dac15ab80 20 client.17751 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.262+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55debbdc4380 0x55debbdc67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.262+0000 7f5dac15ab80 4 client.17751 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.262+0000 7f5dac15ab80 1 -- 172.21.15.43:0/88479703 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55debbcc8d90 con 0x55debbdc4380 2024-05-27T07:30:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.262+0000 7f5d96a00700 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55debbdc4380 0x55debbdc67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.264+0000 7f5d96a00700 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55debbdc4380 0x55debbdc67c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a8aa8fe588f2c30a server_cookie=3cd318499a73a515 in_seq=0 out_seq=0 2024-05-27T07:30:35.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.264+0000 7f5d80c00700 10 client.17751 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:30:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.267+0000 7f5d80c00700 1 -- 172.21.15.43:0/88479703 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55debbcc8d90 con 0x55debbdc4380 2024-05-27T07:30:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.267+0000 7f5d80c00700 10 client.17751 handle_command_reply: tid=0 2024-05-27T07:30:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.267+0000 7f5dac15ab80 1 client.17751 shutdown 2024-05-27T07:30:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 1 -- 172.21.15.43:0/88479703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 msgr2=0x55debbda0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda0930 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009b10 tx=0x7f5d840165e0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 1 -- 172.21.15.43:0/88479703 shutdown_connections 2024-05-27T07:30:35.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55debbdc4380 0x55debbdc67c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 0x55debbdaf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbc19e40 0x55debbda03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/88479703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda0930 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 1 -- 172.21.15.43:0/88479703 >> 172.21.15.43:0/88479703 conn(0x55debbcb3390 msgr2=0x55debbd997f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5d80c00700 0 client.17751 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:30:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 1 -- 172.21.15.43:0/88479703 shutdown_connections 2024-05-27T07:30:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 1 -- 172.21.15.43:0/88479703 wait complete. 2024-05-27T07:30:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.268+0000 7f5dac15ab80 20 client.17751 trim_cache size 0 max 16384 2024-05-27T07:30:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.300+0000 7f5dac15ab80 1 Processor -- start 2024-05-27T07:30:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.301+0000 7f5dac15ab80 1 -- start start 2024-05-27T07:30:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.301+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbc19e40 0x55debbda08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.301+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbd059f0 0x55debbdaef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.301+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 0x55debbd99750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.301+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5d981ae7b0 con 0x55debbc19e40 2024-05-27T07:30:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.301+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5d98074600 con 0x55debbda0e70 2024-05-27T07:30:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.301+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5d981ae0b0 con 0x55debbd059f0 2024-05-27T07:30:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.302+0000 7f5d96a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbc19e40 0x55debbda08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.302+0000 7f5d96a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbc19e40 0x55debbda08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46452/0 (socket says 172.21.15.43:46452) 2024-05-27T07:30:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.302+0000 7f5d96000700 1 --2- 172.21.15.43:0/1535470650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbd059f0 0x55debbdaef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.302+0000 7f5d97400700 1 --2- 172.21.15.43:0/1535470650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 0x55debbd99750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.302+0000 7f5d96a00700 1 -- 172.21.15.43:0/1535470650 learned_addr learned my addr 172.21.15.43:0/1535470650 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.302+0000 7f5d97400700 1 -- 172.21.15.43:0/1535470650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbd059f0 msgr2=0x55debbdaef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.302+0000 7f5d97400700 1 --2- 172.21.15.43:0/1535470650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbd059f0 0x55debbdaef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.302+0000 7f5d97400700 1 -- 172.21.15.43:0/1535470650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbc19e40 msgr2=0x55debbda08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.303+0000 7f5d97400700 1 --2- 172.21.15.43:0/1535470650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbc19e40 0x55debbda08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.303+0000 7f5d97400700 1 -- 172.21.15.43:0/1535470650 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55debbd9f710 con 0x55debbda0e70 2024-05-27T07:30:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.303+0000 7f5d97400700 1 --2- 172.21.15.43:0/1535470650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 0x55debbd99750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009790 tx=0x7f5d9001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b252fbfb02f10ba9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.304+0000 7f5d83400700 1 -- 172.21.15.43:0/1535470650 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d90007700 con 0x55debbda0e70 2024-05-27T07:30:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.304+0000 7f5d83400700 1 -- 172.21.15.43:0/1535470650 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d9000be50 con 0x55debbda0e70 2024-05-27T07:30:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.304+0000 7f5d83400700 1 -- 172.21.15.43:0/1535470650 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d90005760 con 0x55debbda0e70 2024-05-27T07:30:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.305+0000 7f5dac15ab80 1 -- 172.21.15.43:0/1535470650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 msgr2=0x55debbd99750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.305+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/1535470650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 0x55debbd99750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009790 tx=0x7f5d9001d000 comp rx=0 tx=0).stop 2024-05-27T07:30:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.305+0000 7f5dac15ab80 1 -- 172.21.15.43:0/1535470650 shutdown_connections 2024-05-27T07:30:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.305+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/1535470650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbda0e70 0x55debbd99750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.305+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/1535470650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbc19e40 0x55debbda08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.305+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/1535470650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbd059f0 0x55debbdaef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.305+0000 7f5dac15ab80 1 -- 172.21.15.43:0/1535470650 >> 172.21.15.43:0/1535470650 conn(0x55debbcb3390 msgr2=0x55debbda01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.306+0000 7f5dac15ab80 1 -- 172.21.15.43:0/1535470650 shutdown_connections 2024-05-27T07:30:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.306+0000 7f5dac15ab80 1 -- 172.21.15.43:0/1535470650 wait complete. 2024-05-27T07:30:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.306+0000 7f5dac15ab80 1 -- start start 2024-05-27T07:30:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.307+0000 7f5dac15ab80 1 Processor -- start 2024-05-27T07:30:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.307+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbc19e40 0x55debbda6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.307+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.307+0000 7f5dac15ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbda0e70 0x55debbe2ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.307+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5d98075400 con 0x55debbd059f0 2024-05-27T07:30:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.307+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5d98074d00 con 0x55debbc19e40 2024-05-27T07:30:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.307+0000 7f5dac15ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55debbd933a0 con 0x55debbda0e70 2024-05-27T07:30:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.308+0000 7f5dac15ab80 10 client.0 _ll_register_callbacks cb 0x55debbc60c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.309+0000 7f5d96a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbc19e40 0x55debbda6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.309+0000 7f5d96a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbc19e40 0x55debbda6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52708/0 (socket says 172.21.15.43:52708) 2024-05-27T07:30:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.309+0000 7f5d96a00700 1 -- 172.21.15.43:0/3609136544 learned_addr learned my addr 172.21.15.43:0/3609136544 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.309+0000 7f5d96000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.310+0000 7f5d97400700 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbda0e70 0x55debbe2ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.310+0000 7f5d96a00700 1 -- 172.21.15.43:0/3609136544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbda0e70 msgr2=0x55debbe2ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.310+0000 7f5d96a00700 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbda0e70 0x55debbe2ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.310+0000 7f5d96a00700 1 -- 172.21.15.43:0/3609136544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 msgr2=0x55debbda6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.311+0000 7f5d96a00700 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.311+0000 7f5d96a00700 1 -- 172.21.15.43:0/3609136544 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55debbda6f20 con 0x55debbc19e40 2024-05-27T07:30:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.312+0000 7f5d96000700 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.312+0000 7f5d96a00700 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbc19e40 0x55debbda6150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5d88007ca0 tx=0x7f5d88007fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1642542060754d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.312+0000 7f5d80c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:30:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.313+0000 7f5d80c00700 1 -- 172.21.15.43:0/3609136544 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d88014c60 con 0x55debbc19e40 2024-05-27T07:30:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.313+0000 7f5d80c00700 1 -- 172.21.15.43:0/3609136544 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f5d8800f040 con 0x55debbc19e40 2024-05-27T07:30:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.313+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3609136544 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d9000be50 con 0x55debbc19e40 2024-05-27T07:30:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.314+0000 7f5d80c00700 1 -- 172.21.15.43:0/3609136544 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) ==== 24+0+0 (secure 0 0 0) 0x7f5d88014e00 con 0x55debbc19e40 2024-05-27T07:30:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.314+0000 7f5dac15ab80 10 client.37148 fetch_fsmap learned FSMap version 130 2024-05-27T07:30:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.314+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3609136544 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=4+}) -- 0x55debbe3f400 con 0x55debbc19e40 2024-05-27T07:30:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.314+0000 7f5d80c00700 1 -- 172.21.15.43:0/3609136544 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d9000be50 con 0x55debbc19e40 2024-05-27T07:30:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.314+0000 7f5d80c00700 1 -- 172.21.15.43:0/3609136544 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 130) ==== 4219+0+0 (secure 0 0 0) 0x7f5d88006760 con 0x55debbc19e40 2024-05-27T07:30:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.315+0000 7f5dac15ab80 10 client.37148 fetch_fsmap finished waiting for FSMap version 130 2024-05-27T07:30:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.315+0000 7f5dac15ab80 10 client.37148 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:30:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.315+0000 7f5dac15ab80 20 client.37148 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.315+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55debbe3fb20 0x55debbe41f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.315+0000 7f5dac15ab80 4 client.37148 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:30:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.315+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3609136544 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55debbcc2570 con 0x55debbe3fb20 2024-05-27T07:30:35.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.317+0000 7f5d96000700 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55debbe3fb20 0x55debbe41f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.323+0000 7f5d80c00700 10 client.37148 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:30:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.323+0000 7f5d96000700 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55debbe3fb20 0x55debbe41f40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5ef9571a0ba7252e server_cookie=8ed495b69212084e in_seq=0 out_seq=0 2024-05-27T07:30:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.325+0000 7f5d80c00700 1 -- 172.21.15.43:0/3609136544 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55debbcc2570 con 0x55debbe3fb20 2024-05-27T07:30:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.325+0000 7f5d80c00700 10 client.37148 handle_command_reply: tid=0 2024-05-27T07:30:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.325+0000 7f5dac15ab80 1 client.37148 shutdown 2024-05-27T07:30:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.325+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3609136544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbc19e40 msgr2=0x55debbda6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.325+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbc19e40 0x55debbda6150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5d88007ca0 tx=0x7f5d88007fb0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.326+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3609136544 shutdown_connections 2024-05-27T07:30:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.326+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55debbe3fb20 0x55debbe41f40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.326+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55debbda0e70 0x55debbe2ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.326+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55debbc19e40 0x55debbda6150 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.326+0000 7f5dac15ab80 1 --2- 172.21.15.43:0/3609136544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55debbd059f0 0x55debbda6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.326+0000 7f5d80c00700 0 client.37148 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:30:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.326+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3609136544 >> 172.21.15.43:0/3609136544 conn(0x55debbcb3390 msgr2=0x55debbd9e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.327+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3609136544 shutdown_connections 2024-05-27T07:30:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.327+0000 7f5dac15ab80 1 -- 172.21.15.43:0/3609136544 wait complete. 2024-05-27T07:30:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.328+0000 7f5dac15ab80 20 client.37148 trim_cache size 0 max 16384 2024-05-27T07:30:35.321 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:30:35.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.329+0000 7f5d97e00700 1 -- 172.21.15.43:0/1768620500 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d7c0b2a70 msgr2=0x7f5d7c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:35.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.329+0000 7f5d97e00700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d7c0b2a70 0x7f5d7c0b4f80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009840 tx=0x7f5d90009cc0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.330+0000 7f5d97e00700 1 -- 172.21.15.43:0/1768620500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d98102b40 msgr2=0x7f5d981a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.330+0000 7f5d97e00700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d98102b40 0x7f5d981a2460 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400ec10 tx=0x7f5d8400ef20 comp rx=0 tx=0).stop 2024-05-27T07:30:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.330+0000 7f5d97e00700 1 -- 172.21.15.43:0/1768620500 shutdown_connections 2024-05-27T07:30:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.330+0000 7f5d97e00700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5d7c0b2a70 0x7f5d7c0b4f80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.330+0000 7f5d97e00700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d981091b0 0x7f5d981acf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.330+0000 7f5d97e00700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d981021f0 0x7f5d981a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.330+0000 7f5d97e00700 1 --2- 172.21.15.43:0/1768620500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d98102b40 0x7f5d981a2460 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.330+0000 7f5d97e00700 1 -- 172.21.15.43:0/1768620500 >> 172.21.15.43:0/1768620500 conn(0x7f5d98077ee0 msgr2=0x7f5d98100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.331+0000 7f5d97e00700 1 -- 172.21.15.43:0/1768620500 shutdown_connections 2024-05-27T07:30:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:35.331+0000 7f5d97e00700 1 -- 172.21.15.43:0/1768620500 wait complete. 2024-05-27T07:30:35.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:35 smithi043 ceph-mon[31531]: pgmap v560: 129 pgs: 129 active+clean; 575 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.0 MiB/s wr, 151 op/s 2024-05-27T07:30:35.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 17669 5.84 MB/sec execute 36 sec latency 74.492 ms 2024-05-27T07:30:35.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:35 smithi132 ceph-mon[35283]: pgmap v560: 129 pgs: 129 active+clean; 575 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.0 MiB/s wr, 151 op/s 2024-05-27T07:30:35.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:35 smithi179 ceph-mon[35888]: pgmap v560: 129 pgs: 129 active+clean; 575 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.0 MiB/s wr, 151 op/s 2024-05-27T07:30:36.294 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:30:36.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 17776 5.80 MB/sec execute 37 sec latency 78.733 ms 2024-05-27T07:30:36.883 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:37.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:37 smithi043 ceph-mon[31531]: pgmap v561: 129 pgs: 129 active+clean; 577 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-05-27T07:30:37.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 17904 5.90 MB/sec execute 38 sec latency 94.151 ms 2024-05-27T07:30:37.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:37 smithi132 ceph-mon[35283]: pgmap v561: 129 pgs: 129 active+clean; 577 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-05-27T07:30:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:37 smithi179 ceph-mon[35888]: pgmap v561: 129 pgs: 129 active+clean; 577 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.4 MiB/s wr, 224 op/s 2024-05-27T07:30:38.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 18039 6.02 MB/sec execute 39 sec latency 99.212 ms 2024-05-27T07:30:39.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.262+0000 7f577ec00700 1 -- 172.21.15.43:0/4285370577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f577810e8d0 msgr2=0x7f5778110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:39.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.262+0000 7f577ec00700 1 --2- 172.21.15.43:0/4285370577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f577810e8d0 0x7f5778110db0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f57680098f0 tx=0x7f5768009c00 comp rx=0 tx=0).stop 2024-05-27T07:30:39.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.262+0000 7f577ec00700 1 -- 172.21.15.43:0/4285370577 shutdown_connections 2024-05-27T07:30:39.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.263+0000 7f577ec00700 1 --2- 172.21.15.43:0/4285370577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f577810e8d0 0x7f5778110db0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.263+0000 7f577ec00700 1 --2- 172.21.15.43:0/4285370577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57781054f0 0x7f5778105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.263+0000 7f577ec00700 1 --2- 172.21.15.43:0/4285370577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5778105e40 0x7f577810e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.263+0000 7f577ec00700 1 -- 172.21.15.43:0/4285370577 >> 172.21.15.43:0/4285370577 conn(0x7f57780fedb0 msgr2=0x7f57781011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:39.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.263+0000 7f577ec00700 1 -- 172.21.15.43:0/4285370577 shutdown_connections 2024-05-27T07:30:39.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.263+0000 7f577ec00700 1 -- 172.21.15.43:0/4285370577 wait complete. 2024-05-27T07:30:39.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.264+0000 7f577ec00700 1 Processor -- start 2024-05-27T07:30:39.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.265+0000 7f577ec00700 1 -- start start 2024-05-27T07:30:39.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.265+0000 7f577ec00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57781054f0 0x7f577819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.265+0000 7f577ec00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5778105e40 0x7f577819d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.265+0000 7f577ec00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f577810e8d0 0x7f57781a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.265+0000 7f577ec00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f57781165a0 con 0x7f57781054f0 2024-05-27T07:30:39.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.265+0000 7f577ec00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5778116420 con 0x7f577810e8d0 2024-05-27T07:30:39.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.265+0000 7f577ec00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5778116720 con 0x7f5778105e40 2024-05-27T07:30:39.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.266+0000 7f5777e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57781054f0 0x7f577819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.266+0000 7f577cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f577810e8d0 0x7f57781a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.266+0000 7f5777e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57781054f0 0x7f577819cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46480/0 (socket says 172.21.15.43:46480) 2024-05-27T07:30:39.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.266+0000 7f5777e00700 1 -- 172.21.15.43:0/2863767591 learned_addr learned my addr 172.21.15.43:0/2863767591 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:39.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.266+0000 7f5777400700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5778105e40 0x7f577819d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.266+0000 7f577cc00700 1 -- 172.21.15.43:0/2863767591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5778105e40 msgr2=0x7f577819d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:39.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.266+0000 7f577cc00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5778105e40 0x7f577819d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.266+0000 7f577cc00700 1 -- 172.21.15.43:0/2863767591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57781054f0 msgr2=0x7f577819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:39.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.266+0000 7f577cc00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57781054f0 0x7f577819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.266+0000 7f577cc00700 1 -- 172.21.15.43:0/2863767591 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57781a1c20 con 0x7f577810e8d0 2024-05-27T07:30:39.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.267+0000 7f577cc00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f577810e8d0 0x7f57781a1580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5768009810 tx=0x7f576800b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:39.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.267+0000 7f5774c00700 1 -- 172.21.15.43:0/2863767591 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768025070 con 0x7f577810e8d0 2024-05-27T07:30:39.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.267+0000 7f5774c00700 1 -- 172.21.15.43:0/2863767591 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f576800ba90 con 0x7f577810e8d0 2024-05-27T07:30:39.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.267+0000 7f577ec00700 1 -- 172.21.15.43:0/2863767591 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57781a1eb0 con 0x7f577810e8d0 2024-05-27T07:30:39.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.268+0000 7f577ec00700 1 -- 172.21.15.43:0/2863767591 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57781a2390 con 0x7f577810e8d0 2024-05-27T07:30:39.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.268+0000 7f5774c00700 1 -- 172.21.15.43:0/2863767591 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768004ad0 con 0x7f577810e8d0 2024-05-27T07:30:39.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.273+0000 7f5774c00700 1 -- 172.21.15.43:0/2863767591 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5768028070 con 0x7f577810e8d0 2024-05-27T07:30:39.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.273+0000 7f578d6dfb80 1 Processor -- start 2024-05-27T07:30:39.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.274+0000 7f578d6dfb80 1 -- start start 2024-05-27T07:30:39.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.275+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf226629f0 0x55bf226fdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.275+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf226fe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.275+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf2270a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.275+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55bf22623e40 con 0x55bf22576e40 2024-05-27T07:30:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.275+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bf226139e0 con 0x55bf226fe690 2024-05-27T07:30:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.275+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55bf22617a20 con 0x55bf226629f0 2024-05-27T07:30:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.275+0000 7f5774c00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f575c0b2a70 0x7f575c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.276+0000 7f5774c00700 1 -- 172.21.15.43:0/2863767591 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f57680e3580 con 0x7f577810e8d0 2024-05-27T07:30:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.276+0000 7f5777e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf226fe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.276+0000 7f5777e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf226fe090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46496/0 (socket says 172.21.15.43:46496) 2024-05-27T07:30:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.276+0000 7f5777e00700 1 -- 172.21.15.43:0/2533697486 learned_addr learned my addr 172.21.15.43:0/2533697486 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.276+0000 7f577cc00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f575c0b2a70 0x7f575c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.277+0000 7f5777e00700 1 -- 172.21.15.43:0/2533697486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf226629f0 msgr2=0x55bf226fdb50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:30:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.277+0000 7f5777e00700 1 --2- 172.21.15.43:0/2533697486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf226629f0 0x55bf226fdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.277+0000 7f5777e00700 1 -- 172.21.15.43:0/2533697486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 msgr2=0x55bf2270a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.277+0000 7f5777e00700 1 --2- 172.21.15.43:0/2533697486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf2270a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.277+0000 7f5777e00700 1 -- 172.21.15.43:0/2533697486 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf225dc420 con 0x55bf22576e40 2024-05-27T07:30:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.277+0000 7f5777e00700 1 --2- 172.21.15.43:0/2533697486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf226fe090 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f5770003f00 tx=0x7f57700099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811985a060c3d037 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.277+0000 7f577cc00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f575c0b2a70 0x7f575c0b4f80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f57680a6680 tx=0x7f576800b3e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.278+0000 7f576d600700 1 -- 172.21.15.43:0/2533697486 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5770019030 con 0x55bf22576e40 2024-05-27T07:30:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.278+0000 7f576d600700 1 -- 172.21.15.43:0/2533697486 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57700092e0 con 0x55bf22576e40 2024-05-27T07:30:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.278+0000 7f576d600700 1 -- 172.21.15.43:0/2533697486 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5770015450 con 0x55bf22576e40 2024-05-27T07:30:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.279+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2533697486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 msgr2=0x55bf226fe090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.279+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/2533697486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf226fe090 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f5770003f00 tx=0x7f57700099e0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.279+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2533697486 shutdown_connections 2024-05-27T07:30:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.279+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/2533697486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf2270a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.279+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/2533697486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf226629f0 0x55bf226fdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.279+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/2533697486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf226fe090 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.279+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2533697486 >> 172.21.15.43:0/2533697486 conn(0x55bf22610390 msgr2=0x55bf226f1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.279+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2533697486 shutdown_connections 2024-05-27T07:30:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.280+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2533697486 wait complete. 2024-05-27T07:30:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.280+0000 7f578d6dfb80 1 -- start start 2024-05-27T07:30:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.280+0000 7f578d6dfb80 1 Processor -- start 2024-05-27T07:30:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.281+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf22576e40 0x55bf22708000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.281+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 0x55bf22708540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.281+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22708a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.281+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55bf22519fa0 con 0x55bf226629f0 2024-05-27T07:30:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.281+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bf225b4760 con 0x55bf226fe690 2024-05-27T07:30:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.281+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55bf2270b3a0 con 0x55bf22576e40 2024-05-27T07:30:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.281+0000 7f578d6dfb80 10 client.0 _ll_register_callbacks cb 0x55bf225bdc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.282+0000 7f5777400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22708a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.282+0000 7f5777400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22708a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52768/0 (socket says 172.21.15.43:52768) 2024-05-27T07:30:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.282+0000 7f5777e00700 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 0x55bf22708540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.282+0000 7f5777400700 1 -- 172.21.15.43:0/2071330521 learned_addr learned my addr 172.21.15.43:0/2071330521 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.282+0000 7f5777400700 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf22576e40 0x55bf22708000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.283+0000 7f5777e00700 1 -- 172.21.15.43:0/2071330521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf22576e40 msgr2=0x55bf22708000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.283+0000 7f5777e00700 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf22576e40 0x55bf22708000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.283+0000 7f5777e00700 1 -- 172.21.15.43:0/2071330521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 msgr2=0x55bf22708a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.283+0000 7f5777e00700 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22708a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.283+0000 7f5777e00700 1 -- 172.21.15.43:0/2071330521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf225638d0 con 0x55bf226629f0 2024-05-27T07:30:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.283+0000 7f5777e00700 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 0x55bf22708540 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f5770003ed0 tx=0x7f57700077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d721cfba7ef94c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.283+0000 7f5762000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:30:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.284+0000 7f5762000700 1 -- 172.21.15.43:0/2071330521 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5770023070 con 0x55bf226629f0 2024-05-27T07:30:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.284+0000 7f5762000700 1 -- 172.21.15.43:0/2071330521 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f5770015450 con 0x55bf226629f0 2024-05-27T07:30:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.284+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2071330521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57700092e0 con 0x55bf226629f0 2024-05-27T07:30:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f5762000700 1 -- 172.21.15.43:0/2071330521 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) ==== 24+0+0 (secure 0 0 0) 0x7f5770015690 con 0x55bf226629f0 2024-05-27T07:30:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f578d6dfb80 10 client.17769 fetch_fsmap learned FSMap version 131 2024-05-27T07:30:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2071330521 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=4+}) -- 0x55bf225e4060 con 0x55bf226629f0 2024-05-27T07:30:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f5762000700 1 -- 172.21.15.43:0/2071330521 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f57700092e0 con 0x55bf226629f0 2024-05-27T07:30:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f5762000700 1 -- 172.21.15.43:0/2071330521 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 131) ==== 4215+0+0 (secure 0 0 0) 0x7f577001f680 con 0x55bf226629f0 2024-05-27T07:30:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f578d6dfb80 10 client.17769 fetch_fsmap finished waiting for FSMap version 131 2024-05-27T07:30:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f578d6dfb80 10 client.17769 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:30:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f578d6dfb80 20 client.17769 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55bf227186a0 0x55bf2271aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f578d6dfb80 4 client.17769 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.285+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2071330521 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf22625d90 con 0x55bf227186a0 2024-05-27T07:30:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.286+0000 7f5777400700 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55bf227186a0 0x55bf2271aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.287+0000 7f5762000700 10 client.17769 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:30:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.287+0000 7f5777400700 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55bf227186a0 0x55bf2271aaf0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=db7761d80797420b server_cookie=caed03817151a893 in_seq=0 out_seq=0 2024-05-27T07:30:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.288+0000 7f5762000700 1 -- 172.21.15.43:0/2071330521 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bf22625d90 con 0x55bf227186a0 2024-05-27T07:30:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.288+0000 7f5762000700 10 client.17769 handle_command_reply: tid=0 2024-05-27T07:30:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.289+0000 7f578d6dfb80 1 client.17769 shutdown 2024-05-27T07:30:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.289+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2071330521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 msgr2=0x55bf22708540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.289+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 0x55bf22708540 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f5770003ed0 tx=0x7f57700077f0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.289+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2071330521 shutdown_connections 2024-05-27T07:30:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.289+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55bf227186a0 0x55bf2271aaf0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.289+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22708a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.289+0000 7f5762000700 0 client.17769 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:30:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.289+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf22576e40 0x55bf22708000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.289+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/2071330521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 0x55bf22708540 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.289+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2071330521 >> 172.21.15.43:0/2071330521 conn(0x55bf22610390 msgr2=0x55bf226fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:39.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.291+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2071330521 shutdown_connections 2024-05-27T07:30:39.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.291+0000 7f578d6dfb80 1 -- 172.21.15.43:0/2071330521 wait complete. 2024-05-27T07:30:39.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.291+0000 7f578d6dfb80 20 client.17769 trim_cache size 0 max 16384 2024-05-27T07:30:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.321+0000 7f578d6dfb80 1 Processor -- start 2024-05-27T07:30:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.321+0000 7f578d6dfb80 1 -- start start 2024-05-27T07:30:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.321+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf22576e40 0x55bf226f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.322+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 0x55bf22703840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.322+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22702120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.322+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f575800f3c0 con 0x55bf226629f0 2024-05-27T07:30:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.322+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f575800d730 con 0x55bf226fe690 2024-05-27T07:30:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.322+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f575800ecc0 con 0x55bf22576e40 2024-05-27T07:30:39.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.323+0000 7f5777400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22702120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.323+0000 7f5777400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22702120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52784/0 (socket says 172.21.15.43:52784) 2024-05-27T07:30:39.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.323+0000 7f5777400700 1 -- 172.21.15.43:0/3675007038 learned_addr learned my addr 172.21.15.43:0/3675007038 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:39.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.324+0000 7f5777e00700 1 --2- 172.21.15.43:0/3675007038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 0x55bf22703840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.325+0000 7f5777400700 1 --2- 172.21.15.43:0/3675007038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf22576e40 0x55bf226f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.325+0000 7f5777400700 1 -- 172.21.15.43:0/3675007038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf22576e40 msgr2=0x55bf226f14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:39.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.325+0000 7f5777400700 1 --2- 172.21.15.43:0/3675007038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf22576e40 0x55bf226f14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.325+0000 7f5777400700 1 -- 172.21.15.43:0/3675007038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 msgr2=0x55bf22703840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.326+0000 7f5777400700 1 --2- 172.21.15.43:0/3675007038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 0x55bf22703840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.326+0000 7f5777400700 1 -- 172.21.15.43:0/3675007038 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf22706570 con 0x55bf226fe690 2024-05-27T07:30:39.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.328+0000 7f5777400700 1 --2- 172.21.15.43:0/3675007038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22702120 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55bf2271b1e0 tx=0x7f5764015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5d684c0f5f47e36 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:39.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.330+0000 7f576cc00700 1 -- 172.21.15.43:0/3675007038 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764003ae0 con 0x55bf226fe690 2024-05-27T07:30:39.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.330+0000 7f576cc00700 1 -- 172.21.15.43:0/3675007038 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5764028070 con 0x55bf226fe690 2024-05-27T07:30:39.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.330+0000 7f576cc00700 1 -- 172.21.15.43:0/3675007038 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764003ae0 con 0x55bf226fe690 2024-05-27T07:30:39.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.331+0000 7f578d6dfb80 1 -- 172.21.15.43:0/3675007038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 msgr2=0x55bf22702120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:39.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.332+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/3675007038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22702120 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55bf2271b1e0 tx=0x7f5764015040 comp rx=0 tx=0).stop 2024-05-27T07:30:39.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.332+0000 7f578d6dfb80 1 -- 172.21.15.43:0/3675007038 shutdown_connections 2024-05-27T07:30:39.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.332+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/3675007038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf22702120 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.332+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/3675007038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf22576e40 0x55bf226f14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.332+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/3675007038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf226629f0 0x55bf22703840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.333+0000 7f578d6dfb80 1 -- 172.21.15.43:0/3675007038 >> 172.21.15.43:0/3675007038 conn(0x55bf22610390 msgr2=0x55bf22709560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:39.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.333+0000 7f578d6dfb80 1 -- 172.21.15.43:0/3675007038 shutdown_connections 2024-05-27T07:30:39.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.333+0000 7f578d6dfb80 1 -- 172.21.15.43:0/3675007038 wait complete. 2024-05-27T07:30:39.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.334+0000 7f578d6dfb80 1 -- start start 2024-05-27T07:30:39.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.335+0000 7f578d6dfb80 1 Processor -- start 2024-05-27T07:30:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.335+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf22710af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.335+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf226629f0 0x55bf2277b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.335+0000 7f578d6dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf2277bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.335+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f575800e530 con 0x55bf22576e40 2024-05-27T07:30:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.335+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f575800de30 con 0x55bf226fe690 2024-05-27T07:30:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.336+0000 7f578d6dfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55bf226f0150 con 0x55bf226629f0 2024-05-27T07:30:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.336+0000 7f578d6dfb80 10 client.0 _ll_register_callbacks cb 0x55bf225bdc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.337+0000 7f5777400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf2277bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.337+0000 7f5777400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf22710af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.337+0000 7f5777400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf22710af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46530/0 (socket says 172.21.15.43:46530) 2024-05-27T07:30:39.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.337+0000 7f5777400700 1 -- 172.21.15.43:0/1991403605 learned_addr learned my addr 172.21.15.43:0/1991403605 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:39.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.338+0000 7f5777400700 1 -- 172.21.15.43:0/1991403605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf226629f0 msgr2=0x55bf2277b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:39.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.338+0000 7f5777400700 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf226629f0 0x55bf2277b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.339+0000 7f5777400700 1 -- 172.21.15.43:0/1991403605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 msgr2=0x55bf2277bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:39.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.339+0000 7f5777400700 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf2277bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.339+0000 7f5777400700 1 -- 172.21.15.43:0/1991403605 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf2277c260 con 0x55bf22576e40 2024-05-27T07:30:39.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.339+0000 7f5777400700 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf22710af0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x55bf226fe610 tx=0x7f57640096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1177115afa87a26 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:39.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.339+0000 7f5762000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:30:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.340+0000 7f5762000700 1 -- 172.21.15.43:0/1991403605 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764021610 con 0x55bf22576e40 2024-05-27T07:30:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.340+0000 7f578d6dfb80 1 -- 172.21.15.43:0/1991403605 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5764028070 con 0x55bf22576e40 2024-05-27T07:30:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.341+0000 7f5762000700 1 -- 172.21.15.43:0/1991403605 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f5764009040 con 0x55bf22576e40 2024-05-27T07:30:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.341+0000 7f5762000700 1 -- 172.21.15.43:0/1991403605 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) ==== 24+0+0 (secure 0 0 0) 0x7f576403f050 con 0x55bf22576e40 2024-05-27T07:30:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.341+0000 7f578d6dfb80 10 client.17775 fetch_fsmap learned FSMap version 131 2024-05-27T07:30:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.341+0000 7f578d6dfb80 1 -- 172.21.15.43:0/1991403605 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=4+}) -- 0x55bf2270e210 con 0x55bf22576e40 2024-05-27T07:30:39.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.342+0000 7f5762000700 1 -- 172.21.15.43:0/1991403605 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5764028070 con 0x55bf22576e40 2024-05-27T07:30:39.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.342+0000 7f5762000700 1 -- 172.21.15.43:0/1991403605 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 131) ==== 4215+0+0 (secure 0 0 0) 0x7f576402f740 con 0x55bf22576e40 2024-05-27T07:30:39.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.342+0000 7f578d6dfb80 10 client.17775 fetch_fsmap finished waiting for FSMap version 131 2024-05-27T07:30:39.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.342+0000 7f578d6dfb80 10 client.17775 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:30:39.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.342+0000 7f578d6dfb80 20 client.17775 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:39.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.342+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55bf226f30b0 0x55bf226f54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:39.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.342+0000 7f578d6dfb80 4 client.17775 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:30:39.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.342+0000 7f578d6dfb80 1 -- 172.21.15.43:0/1991403605 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bf2261f570 con 0x55bf226f30b0 2024-05-27T07:30:39.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.343+0000 7f5777e00700 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55bf226f30b0 0x55bf226f54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:39.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.348+0000 7f5777e00700 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55bf226f30b0 0x55bf226f54d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3e0c27c3a7dd8d9a server_cookie=cfc0ce83b76c621a in_seq=0 out_seq=0 2024-05-27T07:30:39.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.348+0000 7f5762000700 10 client.17775 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:30:39.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.349+0000 7f5762000700 1 -- 172.21.15.43:0/1991403605 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bf2261f570 con 0x55bf226f30b0 2024-05-27T07:30:39.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.349+0000 7f5762000700 10 client.17775 handle_command_reply: tid=0 2024-05-27T07:30:39.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.349+0000 7f578d6dfb80 1 client.17775 shutdown 2024-05-27T07:30:39.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.349+0000 7f578d6dfb80 1 -- 172.21.15.43:0/1991403605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 msgr2=0x55bf22710af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.349+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf22710af0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x55bf226fe610 tx=0x7f57640096d0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.350+0000 7f578d6dfb80 1 -- 172.21.15.43:0/1991403605 shutdown_connections 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.350+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55bf226f30b0 0x55bf226f54d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.350+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bf226fe690 0x55bf2277bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.350+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bf22576e40 0x55bf22710af0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.350+0000 7f578d6dfb80 1 --2- 172.21.15.43:0/1991403605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bf226629f0 0x55bf2277b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.350+0000 7f578d6dfb80 1 -- 172.21.15.43:0/1991403605 >> 172.21.15.43:0/1991403605 conn(0x55bf22610390 msgr2=0x55bf22701e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.350+0000 7f5762000700 0 client.17775 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.350+0000 7f578d6dfb80 1 -- 172.21.15.43:0/1991403605 shutdown_connections 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.350+0000 7f578d6dfb80 1 -- 172.21.15.43:0/1991403605 wait complete. 2024-05-27T07:30:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.351+0000 7f578d6dfb80 20 client.17775 trim_cache size 0 max 16384 2024-05-27T07:30:39.345 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:30:39.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.352+0000 7f577ec00700 1 -- 172.21.15.43:0/2863767591 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f575c0b2a70 msgr2=0x7f575c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:39.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.352+0000 7f577ec00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f575c0b2a70 0x7f575c0b4f80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f57680a6680 tx=0x7f576800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.352+0000 7f577ec00700 1 -- 172.21.15.43:0/2863767591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f577810e8d0 msgr2=0x7f57781a1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:39.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.352+0000 7f577ec00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f577810e8d0 0x7f57781a1580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5768009810 tx=0x7f576800b790 comp rx=0 tx=0).stop 2024-05-27T07:30:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.353+0000 7f577ec00700 1 -- 172.21.15.43:0/2863767591 shutdown_connections 2024-05-27T07:30:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.353+0000 7f577ec00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f575c0b2a70 0x7f575c0b4f80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.353+0000 7f577ec00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f577810e8d0 0x7f57781a1580 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.353+0000 7f577ec00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f57781054f0 0x7f577819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.353+0000 7f577ec00700 1 --2- 172.21.15.43:0/2863767591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5778105e40 0x7f577819d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.353+0000 7f577ec00700 1 -- 172.21.15.43:0/2863767591 >> 172.21.15.43:0/2863767591 conn(0x7f57780fedb0 msgr2=0x7f57780ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.354+0000 7f577ec00700 1 -- 172.21.15.43:0/2863767591 shutdown_connections 2024-05-27T07:30:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:39.354+0000 7f577ec00700 1 -- 172.21.15.43:0/2863767591 wait complete. 2024-05-27T07:30:39.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:39 smithi043 ceph-mon[31531]: pgmap v562: 129 pgs: 129 active+clean; 577 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-05-27T07:30:39.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:39 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:39.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 18163 6.01 MB/sec execute 40 sec latency 97.239 ms 2024-05-27T07:30:39.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:39 smithi132 ceph-mon[35283]: pgmap v562: 129 pgs: 129 active+clean; 577 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-05-27T07:30:39.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:39 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:39 smithi179 ceph-mon[35888]: pgmap v562: 129 pgs: 129 active+clean; 577 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-05-27T07:30:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:39 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:40.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:30:40.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 18333 6.15 MB/sec execute 41 sec latency 83.166 ms 2024-05-27T07:30:40.963 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:41.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:41 smithi043 ceph-mon[31531]: pgmap v563: 129 pgs: 129 active+clean; 577 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-05-27T07:30:41.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 18436 6.11 MB/sec execute 42 sec latency 173.491 ms 2024-05-27T07:30:41.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:41 smithi132 ceph-mon[35283]: pgmap v563: 129 pgs: 129 active+clean; 577 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-05-27T07:30:41.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:41 smithi179 ceph-mon[35888]: pgmap v563: 129 pgs: 129 active+clean; 577 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-05-27T07:30:42.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 18555 6.09 MB/sec execute 43 sec latency 84.192 ms 2024-05-27T07:30:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.399+0000 7f4e9c000700 1 -- 172.21.15.43:0/256500255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e94109860 msgr2=0x7f4e941111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.399+0000 7f4e9c000700 1 --2- 172.21.15.43:0/256500255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e94109860 0x7f4e941111c0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800b0d0 tx=0x7f4e8800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.399+0000 7f4e9c000700 1 -- 172.21.15.43:0/256500255 shutdown_connections 2024-05-27T07:30:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.399+0000 7f4e9c000700 1 --2- 172.21.15.43:0/256500255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e94109860 0x7f4e941111c0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.399+0000 7f4e9c000700 1 --2- 172.21.15.43:0/256500255 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e941045e0 0x7f4e941049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.399+0000 7f4e9c000700 1 --2- 172.21.15.43:0/256500255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e94108f10 0x7f4e94109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.399+0000 7f4e9c000700 1 -- 172.21.15.43:0/256500255 >> 172.21.15.43:0/256500255 conn(0x7f4e94100030 msgr2=0x7f4e94102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.400+0000 7f4e9c000700 1 -- 172.21.15.43:0/256500255 shutdown_connections 2024-05-27T07:30:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.400+0000 7f4e9c000700 1 -- 172.21.15.43:0/256500255 wait complete. 2024-05-27T07:30:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.401+0000 7f4e9c000700 1 Processor -- start 2024-05-27T07:30:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.402+0000 7f4e9c000700 1 -- start start 2024-05-27T07:30:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.402+0000 7f4e9c000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e941045e0 0x7f4e9419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.402+0000 7f4e9c000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e94108f10 0x7f4e9419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.402+0000 7f4e9c000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e94109860 0x7f4e9419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.402+0000 7f4e9c000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4e94114410 con 0x7f4e94109860 2024-05-27T07:30:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.402+0000 7f4e9c000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4e94114290 con 0x7f4e941045e0 2024-05-27T07:30:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.402+0000 7f4e9c000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4e94114590 con 0x7f4e94108f10 2024-05-27T07:30:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.403+0000 7f4e99600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e941045e0 0x7f4e9419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.403+0000 7f4e99600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e941045e0 0x7f4e9419ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52802/0 (socket says 172.21.15.43:52802) 2024-05-27T07:30:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.403+0000 7f4e99600700 1 -- 172.21.15.43:0/2671273538 learned_addr learned my addr 172.21.15.43:0/2671273538 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.403+0000 7f4e98c00700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e94108f10 0x7f4e9419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.404+0000 7f4e9a000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e94109860 0x7f4e9419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.404+0000 7f4e99600700 1 -- 172.21.15.43:0/2671273538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e94108f10 msgr2=0x7f4e9419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.404+0000 7f4e99600700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e94108f10 0x7f4e9419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.404+0000 7f4e99600700 1 -- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e94109860 msgr2=0x7f4e9419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.404+0000 7f4e99600700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e94109860 0x7f4e9419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.404+0000 7f4e99600700 1 -- 172.21.15.43:0/2671273538 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9419fd30 con 0x7f4e941045e0 2024-05-27T07:30:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.404+0000 7f4e9a000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e94109860 0x7f4e9419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:30:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.405+0000 7f4e99600700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e941045e0 0x7f4e9419ae60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00b530 tx=0x7f4e8c00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.405+0000 7f4e92000700 1 -- 172.21.15.43:0/2671273538 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c015020 con 0x7f4e941045e0 2024-05-27T07:30:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.405+0000 7f4e92000700 1 -- 172.21.15.43:0/2671273538 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e8c011cc0 con 0x7f4e941045e0 2024-05-27T07:30:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.405+0000 7f4e92000700 1 -- 172.21.15.43:0/2671273538 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c015020 con 0x7f4e941045e0 2024-05-27T07:30:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.406+0000 7f4e9c000700 1 -- 172.21.15.43:0/2671273538 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e941a0020 con 0x7f4e941045e0 2024-05-27T07:30:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.406+0000 7f4e9c000700 1 -- 172.21.15.43:0/2671273538 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e941a7940 con 0x7f4e941045e0 2024-05-27T07:30:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.412+0000 7f4e92000700 1 -- 172.21.15.43:0/2671273538 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4e8c018020 con 0x7f4e941045e0 2024-05-27T07:30:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.414+0000 7f4eaaa5ab80 1 Processor -- start 2024-05-27T07:30:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.416+0000 7f4eaaa5ab80 1 -- start start 2024-05-27T07:30:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.418+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 0x5569df3738b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.418+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df373e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.418+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df364f30 0x5569df3653e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.418+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5569df292e40 con 0x5569df1e5e40 2024-05-27T07:30:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.418+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5569df2829e0 con 0x5569df364f30 2024-05-27T07:30:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.418+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5569df286a20 con 0x5569df2d19f0 2024-05-27T07:30:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.418+0000 7f4e9a000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 0x5569df3738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.418+0000 7f4e9a000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 0x5569df3738b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:57574/0 (socket says 172.21.15.43:57574) 2024-05-27T07:30:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.418+0000 7f4e9a000700 1 -- 172.21.15.43:0/2231335709 learned_addr learned my addr 172.21.15.43:0/2231335709 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.418+0000 7f4e98c00700 1 --2- 172.21.15.43:0/2231335709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df364f30 0x5569df3653e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.419+0000 7f4e92000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e7c0b2c70 0x7f4e7c0b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.419+0000 7f4e92000700 1 -- 172.21.15.43:0/2671273538 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f4e8c0e1c40 con 0x7f4e941045e0 2024-05-27T07:30:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.420+0000 7f4e98c00700 1 -- 172.21.15.43:0/2231335709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 msgr2=0x5569df3738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.420+0000 7f4e98c00700 1 --2- 172.21.15.43:0/2231335709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 0x5569df3738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.420+0000 7f4e98c00700 1 -- 172.21.15.43:0/2231335709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 msgr2=0x5569df373e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.420+0000 7f4e98c00700 1 --2- 172.21.15.43:0/2231335709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df373e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.420+0000 7f4e98c00700 1 -- 172.21.15.43:0/2231335709 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569df24b420 con 0x5569df364f30 2024-05-27T07:30:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.420+0000 7f4e9a000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e7c0b2c70 0x7f4e7c0b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.420+0000 7f4e98c00700 1 --2- 172.21.15.43:0/2231335709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df364f30 0x5569df3653e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4e9406c670 tx=0x7f4e80009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72cfbeae7ea227f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.421+0000 7f4e86a00700 1 -- 172.21.15.43:0/2231335709 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80011c80 con 0x5569df364f30 2024-05-27T07:30:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.421+0000 7f4e86a00700 1 -- 172.21.15.43:0/2231335709 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e80011e20 con 0x5569df364f30 2024-05-27T07:30:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.421+0000 7f4e86a00700 1 -- 172.21.15.43:0/2231335709 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e800193e0 con 0x5569df364f30 2024-05-27T07:30:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.422+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2231335709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df364f30 msgr2=0x5569df3653e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.422+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/2231335709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df364f30 0x5569df3653e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4e9406c670 tx=0x7f4e80009f60 comp rx=0 tx=0).stop 2024-05-27T07:30:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.422+0000 7f4e9a000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e7c0b2c70 0x7f4e7c0b5180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5569df373330 tx=0x7f4e880099e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.422+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2231335709 shutdown_connections 2024-05-27T07:30:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.422+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/2231335709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df364f30 0x5569df3653e0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4e9406c670 tx=0x7f4e80009f60 comp rx=0 tx=0).stop 2024-05-27T07:30:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.422+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/2231335709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 0x5569df3738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.422+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/2231335709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df373e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.422+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2231335709 >> 172.21.15.43:0/2231335709 conn(0x5569df27f390 msgr2=0x5569df3600f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.423+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2231335709 shutdown_connections 2024-05-27T07:30:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.423+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2231335709 wait complete. 2024-05-27T07:30:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.423+0000 7f4eaaa5ab80 1 -- start start 2024-05-27T07:30:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.424+0000 7f4eaaa5ab80 1 Processor -- start 2024-05-27T07:30:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.424+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df3776b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.424+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 0x5569df377bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.424+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 0x5569df383830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.424+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5569df188fa0 con 0x5569df1e5e40 2024-05-27T07:30:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.424+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5569df223760 con 0x5569df378130 2024-05-27T07:30:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.425+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5569df37a500 con 0x5569df2d19f0 2024-05-27T07:30:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.425+0000 7f4eaaa5ab80 10 client.0 _ll_register_callbacks cb 0x5569df22cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.425+0000 7f4e99600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df3776b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.425+0000 7f4e99600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df3776b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46562/0 (socket says 172.21.15.43:46562) 2024-05-27T07:30:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.425+0000 7f4e99600700 1 -- 172.21.15.43:0/2070845570 learned_addr learned my addr 172.21.15.43:0/2070845570 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.425+0000 7f4e98c00700 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 0x5569df377bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.426+0000 7f4e99600700 1 -- 172.21.15.43:0/2070845570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 msgr2=0x5569df377bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.426+0000 7f4e9a000700 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 0x5569df383830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.426+0000 7f4e99600700 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 0x5569df377bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.426+0000 7f4e99600700 1 -- 172.21.15.43:0/2070845570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 msgr2=0x5569df383830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.426+0000 7f4e99600700 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 0x5569df383830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.426+0000 7f4e99600700 1 -- 172.21.15.43:0/2070845570 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569df1d28d0 con 0x5569df1e5e40 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.426+0000 7f4e9a000700 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 0x5569df383830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.426+0000 7f4e98c00700 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 0x5569df377bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.427+0000 7f4e99600700 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df3776b0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0a89c0 tx=0x7f4e8c06e650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a31efc6e02d672 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.427+0000 7f4e7b400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.427+0000 7f4e7b400700 1 -- 172.21.15.43:0/2070845570 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c0a4450 con 0x5569df1e5e40 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.427+0000 7f4e7b400700 1 -- 172.21.15.43:0/2070845570 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f4e8c0172c0 con 0x5569df1e5e40 2024-05-27T07:30:43.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.428+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2070845570 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e80011e20 con 0x5569df1e5e40 2024-05-27T07:30:43.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.428+0000 7f4e7b400700 1 -- 172.21.15.43:0/2070845570 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) ==== 24+0+0 (secure 0 0 0) 0x7f4e8c0174c0 con 0x5569df1e5e40 2024-05-27T07:30:43.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.428+0000 7f4eaaa5ab80 10 client.17784 fetch_fsmap learned FSMap version 131 2024-05-27T07:30:43.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.428+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2070845570 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=4+}) -- 0x5569df253060 con 0x5569df1e5e40 2024-05-27T07:30:43.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.429+0000 7f4e7b400700 1 -- 172.21.15.43:0/2070845570 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e80011e20 con 0x5569df1e5e40 2024-05-27T07:30:43.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.429+0000 7f4e7b400700 1 -- 172.21.15.43:0/2070845570 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 131) ==== 4215+0+0 (secure 0 0 0) 0x7f4e8c0df2a0 con 0x5569df1e5e40 2024-05-27T07:30:43.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.429+0000 7f4eaaa5ab80 10 client.17784 fetch_fsmap finished waiting for FSMap version 131 2024-05-27T07:30:43.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.429+0000 7f4eaaa5ab80 10 client.17784 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:30:43.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.429+0000 7f4eaaa5ab80 20 client.17784 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:43.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.429+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5569df387e50 0x5569df38a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.429+0000 7f4eaaa5ab80 4 client.17784 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:43.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.429+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2070845570 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569df294d90 con 0x5569df387e50 2024-05-27T07:30:43.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.430+0000 7f4e98c00700 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5569df387e50 0x5569df38a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.432+0000 7f4e7b400700 10 client.17784 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:30:43.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.432+0000 7f4e98c00700 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5569df387e50 0x5569df38a2b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=47f94bf6dff0692d server_cookie=860b9d953101211c in_seq=0 out_seq=0 2024-05-27T07:30:43.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.433+0000 7f4e7b400700 1 -- 172.21.15.43:0/2070845570 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5569df294d90 con 0x5569df387e50 2024-05-27T07:30:43.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.433+0000 7f4e7b400700 10 client.17784 handle_command_reply: tid=0 2024-05-27T07:30:43.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.433+0000 7f4eaaa5ab80 1 client.17784 shutdown 2024-05-27T07:30:43.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2070845570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 msgr2=0x5569df3776b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:43.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df3776b0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0a89c0 tx=0x7f4e8c06e650 comp rx=0 tx=0).stop 2024-05-27T07:30:43.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2070845570 shutdown_connections 2024-05-27T07:30:43.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5569df387e50 0x5569df38a2b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 0x5569df383830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df3776b0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/2070845570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df2d19f0 0x5569df377bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2070845570 >> 172.21.15.43:0/2070845570 conn(0x5569df27f390 msgr2=0x5569df35ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:43.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4e7b400700 0 client.17784 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:30:43.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2070845570 shutdown_connections 2024-05-27T07:30:43.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.434+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/2070845570 wait complete. 2024-05-27T07:30:43.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.435+0000 7f4eaaa5ab80 20 client.17784 trim_cache size 0 max 16384 2024-05-27T07:30:43.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.461+0000 7f4eaaa5ab80 1 Processor -- start 2024-05-27T07:30:43.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.461+0000 7f4eaaa5ab80 1 -- start start 2024-05-27T07:30:43.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.461+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df1e5e40 0x5569df372d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.461+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df2d19f0 0x5569df365340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.462+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 0x5569df383560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.462+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941ae950 con 0x5569df2d19f0 2024-05-27T07:30:43.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.462+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941accc0 con 0x5569df378130 2024-05-27T07:30:43.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.462+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941ae250 con 0x5569df1e5e40 2024-05-27T07:30:43.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.462+0000 7f4e98c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df2d19f0 0x5569df365340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.462+0000 7f4e98c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df2d19f0 0x5569df365340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46574/0 (socket says 172.21.15.43:46574) 2024-05-27T07:30:43.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.462+0000 7f4e98c00700 1 -- 172.21.15.43:0/3383766975 learned_addr learned my addr 172.21.15.43:0/3383766975 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:43.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.462+0000 7f4e9a000700 1 --2- 172.21.15.43:0/3383766975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 0x5569df383560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.463+0000 7f4e98c00700 1 -- 172.21.15.43:0/3383766975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df1e5e40 msgr2=0x5569df372d10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:30:43.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.463+0000 7f4e98c00700 1 --2- 172.21.15.43:0/3383766975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df1e5e40 0x5569df372d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.463+0000 7f4e98c00700 1 -- 172.21.15.43:0/3383766975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 msgr2=0x5569df383560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:43.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.463+0000 7f4e98c00700 1 --2- 172.21.15.43:0/3383766975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 0x5569df383560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.463+0000 7f4e98c00700 1 -- 172.21.15.43:0/3383766975 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569df3769d0 con 0x5569df2d19f0 2024-05-27T07:30:43.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.463+0000 7f4e9a000700 1 --2- 172.21.15.43:0/3383766975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 0x5569df383560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:43.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.464+0000 7f4e98c00700 1 --2- 172.21.15.43:0/3383766975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df2d19f0 0x5569df365340 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4e80003b30 tx=0x7f4e80009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2be5b80f3ee5f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:43.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.464+0000 7f4e86000700 1 -- 172.21.15.43:0/3383766975 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80003c60 con 0x5569df2d19f0 2024-05-27T07:30:43.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.464+0000 7f4e86000700 1 -- 172.21.15.43:0/3383766975 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e80003e00 con 0x5569df2d19f0 2024-05-27T07:30:43.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.464+0000 7f4e86000700 1 -- 172.21.15.43:0/3383766975 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8001a730 con 0x5569df2d19f0 2024-05-27T07:30:43.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.465+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/3383766975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df2d19f0 msgr2=0x5569df365340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:43.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.465+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/3383766975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df2d19f0 0x5569df365340 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4e80003b30 tx=0x7f4e80009710 comp rx=0 tx=0).stop 2024-05-27T07:30:43.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.465+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/3383766975 shutdown_connections 2024-05-27T07:30:43.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.465+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/3383766975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df378130 0x5569df383560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.465+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/3383766975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df1e5e40 0x5569df372d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.465+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/3383766975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df2d19f0 0x5569df365340 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.465+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/3383766975 >> 172.21.15.43:0/3383766975 conn(0x5569df27f390 msgr2=0x5569df372600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.466+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/3383766975 shutdown_connections 2024-05-27T07:30:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.466+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/3383766975 wait complete. 2024-05-27T07:30:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.466+0000 7f4eaaa5ab80 1 -- start start 2024-05-27T07:30:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.467+0000 7f4eaaa5ab80 1 Processor -- start 2024-05-27T07:30:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.467+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df38ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.467+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df2d19f0 0x5569df37f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.467+0000 7f4eaaa5ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df378130 0x5569df37fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.467+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941adac0 con 0x5569df1e5e40 2024-05-27T07:30:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.467+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4e941ad3c0 con 0x5569df2d19f0 2024-05-27T07:30:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.467+0000 7f4eaaa5ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5569df188fa0 con 0x5569df378130 2024-05-27T07:30:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.467+0000 7f4eaaa5ab80 10 client.0 _ll_register_callbacks cb 0x5569df22cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e98c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df2d19f0 0x5569df37f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e98c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df2d19f0 0x5569df37f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52844/0 (socket says 172.21.15.43:52844) 2024-05-27T07:30:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e98c00700 1 -- 172.21.15.43:0/190161163 learned_addr learned my addr 172.21.15.43:0/190161163 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e99600700 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df38ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e99600700 1 -- 172.21.15.43:0/190161163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df378130 msgr2=0x5569df37fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e9a000700 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df378130 0x5569df37fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e99600700 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df378130 0x5569df37fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e99600700 1 -- 172.21.15.43:0/190161163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df2d19f0 msgr2=0x5569df37f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e99600700 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df2d19f0 0x5569df37f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e99600700 1 -- 172.21.15.43:0/190161163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569df380280 con 0x5569df1e5e40 2024-05-27T07:30:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.468+0000 7f4e98c00700 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df2d19f0 0x5569df37f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.469+0000 7f4e9a000700 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df378130 0x5569df37fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.469+0000 7f4e99600700 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df38ffe0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c010b30 tx=0x7f4e8c010380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c72e336037bb509 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.469+0000 7f4e7b400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:30:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.470+0000 7f4e7b400700 1 -- 172.21.15.43:0/190161163 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c0a4450 con 0x5569df1e5e40 2024-05-27T07:30:43.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.470+0000 7f4e7b400700 1 -- 172.21.15.43:0/190161163 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f4e8c0a52b0 con 0x5569df1e5e40 2024-05-27T07:30:43.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.470+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/190161163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e80003e00 con 0x5569df1e5e40 2024-05-27T07:30:43.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.470+0000 7f4e7b400700 1 -- 172.21.15.43:0/190161163 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=131) ==== 24+0+0 (secure 0 0 0) 0x7f4e8c0ad0c0 con 0x5569df1e5e40 2024-05-27T07:30:43.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.470+0000 7f4eaaa5ab80 10 client.17790 fetch_fsmap learned FSMap version 131 2024-05-27T07:30:43.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.470+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/190161163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=131,monmap=4+}) -- 0x5569df3f1980 con 0x5569df1e5e40 2024-05-27T07:30:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.470+0000 7f4e7b400700 1 -- 172.21.15.43:0/190161163 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e80003e00 con 0x5569df1e5e40 2024-05-27T07:30:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.471+0000 7f4e7b400700 1 -- 172.21.15.43:0/190161163 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 131) ==== 4215+0+0 (secure 0 0 0) 0x7f4e8c0a8f20 con 0x5569df1e5e40 2024-05-27T07:30:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.471+0000 7f4eaaa5ab80 10 client.17790 fetch_fsmap finished waiting for FSMap version 131 2024-05-27T07:30:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.471+0000 7f4eaaa5ab80 10 client.17790 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:30:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.471+0000 7f4eaaa5ab80 20 client.17790 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.471+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5569df3f1fa0 0x5569df3f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.471+0000 7f4eaaa5ab80 4 client.17790 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:30:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.471+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/190161163 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5569df28e570 con 0x5569df3f1fa0 2024-05-27T07:30:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.471+0000 7f4e98c00700 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5569df3f1fa0 0x5569df3f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.472+0000 7f4e98c00700 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5569df3f1fa0 0x5569df3f43c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cb14d781de513261 server_cookie=eb079c0e55124ae9 in_seq=0 out_seq=0 2024-05-27T07:30:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.472+0000 7f4e7b400700 10 client.17790 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:30:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.472+0000 7f4e7b400700 1 -- 172.21.15.43:0/190161163 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5569df28e570 con 0x5569df3f1fa0 2024-05-27T07:30:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.472+0000 7f4e7b400700 10 client.17790 handle_command_reply: tid=0 2024-05-27T07:30:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.472+0000 7f4eaaa5ab80 1 client.17790 shutdown 2024-05-27T07:30:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.472+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/190161163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 msgr2=0x5569df38ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.472+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df38ffe0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c010b30 tx=0x7f4e8c010380 comp rx=0 tx=0).stop 2024-05-27T07:30:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.473+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/190161163 shutdown_connections 2024-05-27T07:30:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.473+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x5569df3f1fa0 0x5569df3f43c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.473+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5569df378130 0x5569df37fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.473+0000 7f4e7b400700 0 client.17790 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:30:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.473+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5569df1e5e40 0x5569df38ffe0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.473+0000 7f4eaaa5ab80 1 --2- 172.21.15.43:0/190161163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569df2d19f0 0x5569df37f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.473+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/190161163 >> 172.21.15.43:0/190161163 conn(0x5569df27f390 msgr2=0x5569df375b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.473+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/190161163 shutdown_connections 2024-05-27T07:30:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.473+0000 7f4eaaa5ab80 1 -- 172.21.15.43:0/190161163 wait complete. 2024-05-27T07:30:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.474+0000 7f4eaaa5ab80 20 client.17790 trim_cache size 0 max 16384 2024-05-27T07:30:43.467 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:30:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.476+0000 7f4e9c000700 1 -- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e7c0b2c70 msgr2=0x7f4e7c0b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.476+0000 7f4e9c000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e7c0b2c70 0x7f4e7c0b5180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5569df373330 tx=0x7f4e880099e0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.476+0000 7f4e9c000700 1 -- 172.21.15.43:0/2671273538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e941045e0 msgr2=0x7f4e9419ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.476+0000 7f4e9c000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e941045e0 0x7f4e9419ae60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00b530 tx=0x7f4e8c00b840 comp rx=0 tx=0).stop 2024-05-27T07:30:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.477+0000 7f4e9c000700 1 -- 172.21.15.43:0/2671273538 shutdown_connections 2024-05-27T07:30:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.477+0000 7f4e9c000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f4e7c0b2c70 0x7f4e7c0b5180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.477+0000 7f4e9c000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e94109860 0x7f4e9419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.477+0000 7f4e9c000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e941045e0 0x7f4e9419ae60 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00b530 tx=0x7f4e8c00b840 comp rx=0 tx=0).stop 2024-05-27T07:30:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.477+0000 7f4e9c000700 1 --2- 172.21.15.43:0/2671273538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e94108f10 0x7f4e9419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.477+0000 7f4e9c000700 1 -- 172.21.15.43:0/2671273538 >> 172.21.15.43:0/2671273538 conn(0x7f4e94100030 msgr2=0x7f4e94108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.477+0000 7f4e9c000700 1 -- 172.21.15.43:0/2671273538 shutdown_connections 2024-05-27T07:30:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:43.478+0000 7f4e9c000700 1 -- 172.21.15.43:0/2671273538 wait complete. 2024-05-27T07:30:43.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:43 smithi043 ceph-mon[31531]: pgmap v564: 129 pgs: 129 active+clean; 583 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 4.3 MiB/s wr, 230 op/s 2024-05-27T07:30:43.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 18655 6.05 MB/sec execute 44 sec latency 94.594 ms 2024-05-27T07:30:43.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:43 smithi132 ceph-mon[35283]: pgmap v564: 129 pgs: 129 active+clean; 583 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 4.3 MiB/s wr, 230 op/s 2024-05-27T07:30:43.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:43 smithi179 ceph-mon[35888]: pgmap v564: 129 pgs: 129 active+clean; 583 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 4.3 MiB/s wr, 230 op/s 2024-05-27T07:30:44.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:30:44.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:44 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:44.778 INFO:tasks.workunit.client.0.smithi043.stdout: 10 18771 6.05 MB/sec execute 45 sec latency 85.812 ms 2024-05-27T07:30:44.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:44 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:44.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:44 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:44.995 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:45.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 18889 6.09 MB/sec execute 46 sec latency 94.407 ms 2024-05-27T07:30:45.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:45 smithi132 ceph-mon[35283]: pgmap v565: 129 pgs: 129 active+clean; 583 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 3.1 MiB/s wr, 161 op/s 2024-05-27T07:30:45.862 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:45 smithi179 ceph-mon[35888]: pgmap v565: 129 pgs: 129 active+clean; 583 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 3.1 MiB/s wr, 161 op/s 2024-05-27T07:30:45.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:45 smithi043 ceph-mon[31531]: pgmap v565: 129 pgs: 129 active+clean; 583 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 3.1 MiB/s wr, 161 op/s 2024-05-27T07:30:46.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 18988 6.09 MB/sec execute 47 sec latency 334.879 ms 2024-05-27T07:30:46.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:46 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:46.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:46 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:46.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:46 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:30:47.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.436+0000 7fba60400700 1 -- 172.21.15.43:0/1545319867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba58107de0 msgr2=0x7fba581081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:47.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.436+0000 7fba60400700 1 --2- 172.21.15.43:0/1545319867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba58107de0 0x7fba581081f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fba440098f0 tx=0x7fba44009c00 comp rx=0 tx=0).stop 2024-05-27T07:30:47.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.436+0000 7fba60400700 1 -- 172.21.15.43:0/1545319867 shutdown_connections 2024-05-27T07:30:47.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.436+0000 7fba60400700 1 --2- 172.21.15.43:0/1545319867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba58108730 0x7fba5806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.436+0000 7fba60400700 1 --2- 172.21.15.43:0/1545319867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba581034b0 0x7fba581038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.436+0000 7fba60400700 1 --2- 172.21.15.43:0/1545319867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba58107de0 0x7fba581081f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.436+0000 7fba60400700 1 -- 172.21.15.43:0/1545319867 >> 172.21.15.43:0/1545319867 conn(0x7fba580fedb0 msgr2=0x7fba581011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:47.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.437+0000 7fba60400700 1 -- 172.21.15.43:0/1545319867 shutdown_connections 2024-05-27T07:30:47.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.437+0000 7fba60400700 1 -- 172.21.15.43:0/1545319867 wait complete. 2024-05-27T07:30:47.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.438+0000 7fba60400700 1 Processor -- start 2024-05-27T07:30:47.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.439+0000 7fba60400700 1 -- start start 2024-05-27T07:30:47.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.439+0000 7fba60400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba581034b0 0x7fba5819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.439+0000 7fba60400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba58107de0 0x7fba5819d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.440+0000 7fba60400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba58108730 0x7fba581a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.440+0000 7fba5da00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba581034b0 0x7fba5819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.440+0000 7fba60400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fba581166b0 con 0x7fba58107de0 2024-05-27T07:30:47.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.440+0000 7fba5da00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba581034b0 0x7fba5819cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59610/0 (socket says 172.21.15.43:59610) 2024-05-27T07:30:47.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.440+0000 7fba60400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fba5810ffb0 con 0x7fba581034b0 2024-05-27T07:30:47.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.440+0000 7fba5da00700 1 -- 172.21.15.43:0/3623368308 learned_addr learned my addr 172.21.15.43:0/3623368308 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:47.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.440+0000 7fba60400700 1 -- 172.21.15.43:0/3623368308 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fba58116830 con 0x7fba58108730 2024-05-27T07:30:47.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.440+0000 7fba5d000700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba58107de0 0x7fba5819d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.441+0000 7fba5e400700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba58108730 0x7fba581a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.441+0000 7fba5da00700 1 -- 172.21.15.43:0/3623368308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba58108730 msgr2=0x7fba581a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:47.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.441+0000 7fba5da00700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba58108730 0x7fba581a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.441+0000 7fba5da00700 1 -- 172.21.15.43:0/3623368308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba58107de0 msgr2=0x7fba5819d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:47.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.441+0000 7fba5da00700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba58107de0 0x7fba5819d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.441+0000 7fba5da00700 1 -- 172.21.15.43:0/3623368308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba581a1e30 con 0x7fba581034b0 2024-05-27T07:30:47.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.441+0000 7fba5e400700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba58108730 0x7fba581a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:30:47.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.441+0000 7fba5da00700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba581034b0 0x7fba5819cf60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00ef10 tx=0x7fba4c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:47.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.442+0000 7fba56000700 1 -- 172.21.15.43:0/3623368308 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00cad0 con 0x7fba581034b0 2024-05-27T07:30:47.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.442+0000 7fba60400700 1 -- 172.21.15.43:0/3623368308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba581a20c0 con 0x7fba581034b0 2024-05-27T07:30:47.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.442+0000 7fba56000700 1 -- 172.21.15.43:0/3623368308 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba4c019070 con 0x7fba581034b0 2024-05-27T07:30:47.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.442+0000 7fba56000700 1 -- 172.21.15.43:0/3623368308 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c0146e0 con 0x7fba581034b0 2024-05-27T07:30:47.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.442+0000 7fba60400700 1 -- 172.21.15.43:0/3623368308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba581a9a40 con 0x7fba581034b0 2024-05-27T07:30:47.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.447+0000 7fba56000700 1 -- 172.21.15.43:0/3623368308 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fba4c012050 con 0x7fba581034b0 2024-05-27T07:30:47.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.448+0000 7fba6ef91b80 1 Processor -- start 2024-05-27T07:30:47.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.448+0000 7fba6ef91b80 1 -- start start 2024-05-27T07:30:47.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.449+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dba0e40 0x55b06dd257f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.449+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dd25d30 0x55b06dd1f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.449+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd201c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.449+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b06dc4de40 con 0x55b06dd1fdb0 2024-05-27T07:30:47.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.449+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b06dc3d9e0 con 0x55b06dd25d30 2024-05-27T07:30:47.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.449+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b06dc41a20 con 0x55b06dba0e40 2024-05-27T07:30:47.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.449+0000 7fba56000700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba400b2a70 0x7fba400b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba5d000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd201c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba5d000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd201c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58082/0 (socket says 172.21.15.43:58082) 2024-05-27T07:30:47.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba5d000700 1 -- 172.21.15.43:0/1597894734 learned_addr learned my addr 172.21.15.43:0/1597894734 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:47.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba5da00700 1 --2- 172.21.15.43:0/1597894734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dd25d30 0x55b06dd1f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba5d000700 1 -- 172.21.15.43:0/1597894734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dba0e40 msgr2=0x55b06dd257f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:30:47.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba5d000700 1 --2- 172.21.15.43:0/1597894734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dba0e40 0x55b06dd257f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba5e400700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba400b2a70 0x7fba400b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba56000700 1 -- 172.21.15.43:0/3623368308 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7fba4c0e29d0 con 0x7fba581034b0 2024-05-27T07:30:47.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba5d000700 1 -- 172.21.15.43:0/1597894734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dd25d30 msgr2=0x55b06dd1f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:47.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba5d000700 1 --2- 172.21.15.43:0/1597894734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dd25d30 0x55b06dd1f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.450+0000 7fba5d000700 1 -- 172.21.15.43:0/1597894734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b06dc06420 con 0x55b06dd1fdb0 2024-05-27T07:30:47.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.451+0000 7fba5d000700 1 --2- 172.21.15.43:0/1597894734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd201c0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fba58076aa0 tx=0x7fba4400fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7214876f7cfc20 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:47.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.451+0000 7fba5e400700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba400b2a70 0x7fba400b4f80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55b06dd20b30 tx=0x7fba50009310 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:47.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.451+0000 7fba4aa00700 1 -- 172.21.15.43:0/1597894734 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba44027070 con 0x55b06dd1fdb0 2024-05-27T07:30:47.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.452+0000 7fba4aa00700 1 -- 172.21.15.43:0/1597894734 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba4400f8f0 con 0x55b06dd1fdb0 2024-05-27T07:30:47.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.452+0000 7fba4aa00700 1 -- 172.21.15.43:0/1597894734 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba44004930 con 0x55b06dd1fdb0 2024-05-27T07:30:47.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.453+0000 7fba6ef91b80 1 -- 172.21.15.43:0/1597894734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 msgr2=0x55b06dd201c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:47.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.453+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/1597894734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd201c0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fba58076aa0 tx=0x7fba4400fb00 comp rx=0 tx=0).stop 2024-05-27T07:30:47.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.453+0000 7fba6ef91b80 1 -- 172.21.15.43:0/1597894734 shutdown_connections 2024-05-27T07:30:47.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.453+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/1597894734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd201c0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.453+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/1597894734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dba0e40 0x55b06dd257f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.453+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/1597894734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dd25d30 0x55b06dd1f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.453+0000 7fba6ef91b80 1 -- 172.21.15.43:0/1597894734 >> 172.21.15.43:0/1597894734 conn(0x55b06dc00fc0 msgr2=0x55b06dd1a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:47.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.454+0000 7fba6ef91b80 1 -- 172.21.15.43:0/1597894734 shutdown_connections 2024-05-27T07:30:47.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.454+0000 7fba6ef91b80 1 -- 172.21.15.43:0/1597894734 wait complete. 2024-05-27T07:30:47.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.454+0000 7fba6ef91b80 1 -- start start 2024-05-27T07:30:47.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.455+0000 7fba6ef91b80 1 Processor -- start 2024-05-27T07:30:47.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.455+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dba0e40 0x55b06dd29650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.455+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dd1fdb0 0x55b06dd29b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.455+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd25d30 0x55b06dd2a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.455+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b06db43fa0 con 0x55b06dd25d30 2024-05-27T07:30:47.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.455+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b06dbde760 con 0x55b06dd1fdb0 2024-05-27T07:30:47.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.455+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b06dd2c7a0 con 0x55b06dba0e40 2024-05-27T07:30:47.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.456+0000 7fba6ef91b80 10 client.0 _ll_register_callbacks cb 0x55b06dbe7c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:47.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.456+0000 7fba5e400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd25d30 0x55b06dd2a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.456+0000 7fba5e400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd25d30 0x55b06dd2a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58090/0 (socket says 172.21.15.43:58090) 2024-05-27T07:30:47.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.456+0000 7fba5e400700 1 -- 172.21.15.43:0/3879657429 learned_addr learned my addr 172.21.15.43:0/3879657429 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.456+0000 7fba5d000700 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dd1fdb0 0x55b06dd29b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.457+0000 7fba5da00700 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dba0e40 0x55b06dd29650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.457+0000 7fba5e400700 1 -- 172.21.15.43:0/3879657429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dba0e40 msgr2=0x55b06dd29650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:47.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.457+0000 7fba5e400700 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dba0e40 0x55b06dd29650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.457+0000 7fba5e400700 1 -- 172.21.15.43:0/3879657429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dd1fdb0 msgr2=0x55b06dd29b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:47.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.457+0000 7fba5e400700 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dd1fdb0 0x55b06dd29b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.457+0000 7fba5e400700 1 -- 172.21.15.43:0/3879657429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b06db8d8d0 con 0x55b06dd25d30 2024-05-27T07:30:47.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.458+0000 7fba5e400700 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd25d30 0x55b06dd2a0d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fba50009fd0 tx=0x7fba50004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef644681831b188 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:47.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.458+0000 7fba3f400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:30:47.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.459+0000 7fba3f400700 1 -- 172.21.15.43:0/3879657429 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50023070 con 0x55b06dd25d30 2024-05-27T07:30:47.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.459+0000 7fba3f400700 1 -- 172.21.15.43:0/3879657429 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7fba5001f450 con 0x55b06dd25d30 2024-05-27T07:30:47.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.459+0000 7fba6ef91b80 1 -- 172.21.15.43:0/3879657429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba4400f8f0 con 0x55b06dd25d30 2024-05-27T07:30:47.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.460+0000 7fba3f400700 1 -- 172.21.15.43:0/3879657429 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) ==== 24+0+0 (secure 0 0 0) 0x7fba5002f050 con 0x55b06dd25d30 2024-05-27T07:30:47.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.460+0000 7fba6ef91b80 10 client.17802 fetch_fsmap learned FSMap version 133 2024-05-27T07:30:47.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.460+0000 7fba6ef91b80 1 -- 172.21.15.43:0/3879657429 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=4+}) -- 0x55b06dc0e060 con 0x55b06dd25d30 2024-05-27T07:30:47.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.461+0000 7fba3f400700 1 -- 172.21.15.43:0/3879657429 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba4400f8f0 con 0x55b06dd25d30 2024-05-27T07:30:47.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.461+0000 7fba3f400700 1 -- 172.21.15.43:0/3879657429 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 133) ==== 4207+0+0 (secure 0 0 0) 0x7fba5000f040 con 0x55b06dd25d30 2024-05-27T07:30:47.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.461+0000 7fba6ef91b80 10 client.17802 fetch_fsmap finished waiting for FSMap version 133 2024-05-27T07:30:47.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.461+0000 7fba6ef91b80 10 client.17802 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:30:47.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.461+0000 7fba6ef91b80 20 client.17802 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:47.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.461+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55b06dd39c40 0x55b06dd3c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.461+0000 7fba6ef91b80 4 client.17802 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:47.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.461+0000 7fba6ef91b80 1 -- 172.21.15.43:0/3879657429 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b06dc4fd90 con 0x55b06dd39c40 2024-05-27T07:30:47.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.462+0000 7fba5da00700 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55b06dd39c40 0x55b06dd3c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.463+0000 7fba5da00700 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55b06dd39c40 0x55b06dd3c0d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ff8caca3db05abac server_cookie=977b06e93e1be2ec in_seq=0 out_seq=0 2024-05-27T07:30:47.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.463+0000 7fba3f400700 10 client.17802 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:30:47.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.465+0000 7fba3f400700 1 -- 172.21.15.43:0/3879657429 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b06dc4fd90 con 0x55b06dd39c40 2024-05-27T07:30:47.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.465+0000 7fba3f400700 10 client.17802 handle_command_reply: tid=0 2024-05-27T07:30:47.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.465+0000 7fba6ef91b80 1 client.17802 shutdown 2024-05-27T07:30:47.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.466+0000 7fba6ef91b80 1 -- 172.21.15.43:0/3879657429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd25d30 msgr2=0x55b06dd2a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:47.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.466+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd25d30 0x55b06dd2a0d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fba50009fd0 tx=0x7fba50004750 comp rx=0 tx=0).stop 2024-05-27T07:30:47.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.466+0000 7fba6ef91b80 1 -- 172.21.15.43:0/3879657429 shutdown_connections 2024-05-27T07:30:47.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.466+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55b06dd39c40 0x55b06dd3c0d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.466+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd25d30 0x55b06dd2a0d0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.466+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dba0e40 0x55b06dd29650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.466+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/3879657429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dd1fdb0 0x55b06dd29b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.466+0000 7fba6ef91b80 1 -- 172.21.15.43:0/3879657429 >> 172.21.15.43:0/3879657429 conn(0x55b06dc00fc0 msgr2=0x55b06dd1a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:47.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.466+0000 7fba3f400700 0 client.17802 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:30:47.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.467+0000 7fba6ef91b80 1 -- 172.21.15.43:0/3879657429 shutdown_connections 2024-05-27T07:30:47.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.467+0000 7fba6ef91b80 1 -- 172.21.15.43:0/3879657429 wait complete. 2024-05-27T07:30:47.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.467+0000 7fba6ef91b80 20 client.17802 trim_cache size 0 max 16384 2024-05-27T07:30:47.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.499+0000 7fba6ef91b80 1 Processor -- start 2024-05-27T07:30:47.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.499+0000 7fba6ef91b80 1 -- start start 2024-05-27T07:30:47.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.499+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd1f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.500+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd24d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.500+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 0x55b06dd23b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.500+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fba581b2b40 con 0x55b06dd1fdb0 2024-05-27T07:30:47.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.500+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fba581b0f10 con 0x55b06dba0e40 2024-05-27T07:30:47.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.500+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fba581b2470 con 0x55b06dd25d30 2024-05-27T07:30:47.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.500+0000 7fba5da00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd1f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.500+0000 7fba5da00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd1f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59634/0 (socket says 172.21.15.43:59634) 2024-05-27T07:30:47.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.500+0000 7fba5da00700 1 -- 172.21.15.43:0/2913409443 learned_addr learned my addr 172.21.15.43:0/2913409443 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:47.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.500+0000 7fba5e400700 1 --2- 172.21.15.43:0/2913409443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 0x55b06dd23b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.501+0000 7fba5da00700 1 -- 172.21.15.43:0/2913409443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 msgr2=0x55b06dd23b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:47.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.501+0000 7fba5da00700 1 --2- 172.21.15.43:0/2913409443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 0x55b06dd23b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.501+0000 7fba5da00700 1 -- 172.21.15.43:0/2913409443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 msgr2=0x55b06dd24d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:47.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.501+0000 7fba5da00700 1 --2- 172.21.15.43:0/2913409443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd24d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.501+0000 7fba5da00700 1 -- 172.21.15.43:0/2913409443 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b06dd39a20 con 0x55b06dba0e40 2024-05-27T07:30:47.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.502+0000 7fba5da00700 1 --2- 172.21.15.43:0/2913409443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd1f870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0a5980 tx=0x7fba4c091040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fedcb08c9a270752 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:47.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.503+0000 7fba4a000700 1 -- 172.21.15.43:0/2913409443 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00cc70 con 0x55b06dba0e40 2024-05-27T07:30:47.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.503+0000 7fba4a000700 1 -- 172.21.15.43:0/2913409443 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba4c0b20a0 con 0x55b06dba0e40 2024-05-27T07:30:47.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.503+0000 7fba4a000700 1 -- 172.21.15.43:0/2913409443 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c0b2760 con 0x55b06dba0e40 2024-05-27T07:30:47.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.505+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2913409443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 msgr2=0x55b06dd1f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:47.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.505+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/2913409443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd1f870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0a5980 tx=0x7fba4c091040 comp rx=0 tx=0).stop 2024-05-27T07:30:47.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.505+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2913409443 shutdown_connections 2024-05-27T07:30:47.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.505+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/2913409443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 0x55b06dd23b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.505+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/2913409443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd1f870 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.505+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/2913409443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd24d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.505+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2913409443 >> 172.21.15.43:0/2913409443 conn(0x55b06dc00fc0 msgr2=0x55b06dd29430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:47.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.505+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2913409443 shutdown_connections 2024-05-27T07:30:47.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.506+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2913409443 wait complete. 2024-05-27T07:30:47.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.506+0000 7fba6ef91b80 1 -- start start 2024-05-27T07:30:47.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.506+0000 7fba6ef91b80 1 Processor -- start 2024-05-27T07:30:47.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.508+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd31f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.508+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd40f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.508+0000 7fba6ef91b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 0x55b06dd414c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.508+0000 7fba5da00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd31f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.508+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fba581b1ce0 con 0x55b06dd1fdb0 2024-05-27T07:30:47.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.508+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fba581b1610 con 0x55b06dba0e40 2024-05-27T07:30:47.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.508+0000 7fba6ef91b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b06dd240b0 con 0x55b06dd25d30 2024-05-27T07:30:47.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.509+0000 7fba6ef91b80 10 client.0 _ll_register_callbacks cb 0x55b06dbe7c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:47.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.509+0000 7fba5da00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd31f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59638/0 (socket says 172.21.15.43:59638) 2024-05-27T07:30:47.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.509+0000 7fba5da00700 1 -- 172.21.15.43:0/2860667739 learned_addr learned my addr 172.21.15.43:0/2860667739 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:47.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.509+0000 7fba5d000700 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd40f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.509+0000 7fba5e400700 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 0x55b06dd414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.510+0000 7fba5e400700 1 -- 172.21.15.43:0/2860667739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 msgr2=0x55b06dd31f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:47.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.510+0000 7fba5e400700 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd31f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.510+0000 7fba5e400700 1 -- 172.21.15.43:0/2860667739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 msgr2=0x55b06dd40f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:47.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.510+0000 7fba5e400700 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd40f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.510+0000 7fba5e400700 1 -- 172.21.15.43:0/2860667739 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b06dd41a80 con 0x55b06dd25d30 2024-05-27T07:30:47.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.510+0000 7fba5d000700 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd40f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:30:47.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.510+0000 7fba5e400700 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 0x55b06dd414c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fba50011ca0 tx=0x7fba50011fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26df86d71e9daa52 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:47.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.511+0000 7fba3f400700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:30:47.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.511+0000 7fba3f400700 1 -- 172.21.15.43:0/2860667739 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50023040 con 0x55b06dd25d30 2024-05-27T07:30:47.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.511+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2860667739 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b06dc49570 con 0x55b06dd25d30 2024-05-27T07:30:47.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.511+0000 7fba3f400700 1 -- 172.21.15.43:0/2860667739 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7fba50027d40 con 0x55b06dd25d30 2024-05-27T07:30:47.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.511+0000 7fba3f400700 1 -- 172.21.15.43:0/2860667739 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) ==== 24+0+0 (secure 0 0 0) 0x7fba50043050 con 0x55b06dd25d30 2024-05-27T07:30:47.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.511+0000 7fba6ef91b80 10 client.27280 fetch_fsmap learned FSMap version 133 2024-05-27T07:30:47.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.511+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2860667739 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=4+}) -- 0x55b06dda44f0 con 0x55b06dd25d30 2024-05-27T07:30:47.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.512+0000 7fba3f400700 1 -- 172.21.15.43:0/2860667739 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55b06dc49570 con 0x55b06dd25d30 2024-05-27T07:30:47.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.512+0000 7fba3f400700 1 -- 172.21.15.43:0/2860667739 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 133) ==== 4207+0+0 (secure 0 0 0) 0x7fba50032040 con 0x55b06dd25d30 2024-05-27T07:30:47.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.512+0000 7fba6ef91b80 10 client.27280 fetch_fsmap finished waiting for FSMap version 133 2024-05-27T07:30:47.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.512+0000 7fba6ef91b80 10 client.27280 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:30:47.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.512+0000 7fba6ef91b80 20 client.27280 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:47.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.512+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55b06dda4b40 0x55b06dda6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:47.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.512+0000 7fba6ef91b80 4 client.27280 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:30:47.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.512+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2860667739 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b06dac0250 con 0x55b06dda4b40 2024-05-27T07:30:47.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.513+0000 7fba5da00700 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55b06dda4b40 0x55b06dda6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:47.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.515+0000 7fba5da00700 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55b06dda4b40 0x55b06dda6f60 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ec03a00393f1cfbd server_cookie=d5d0be831fd786ad in_seq=0 out_seq=0 2024-05-27T07:30:47.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.515+0000 7fba3f400700 10 client.27280 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:30:47.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.515+0000 7fba3f400700 1 -- 172.21.15.43:0/2860667739 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b06dac0250 con 0x55b06dda4b40 2024-05-27T07:30:47.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.515+0000 7fba3f400700 10 client.27280 handle_command_reply: tid=0 2024-05-27T07:30:47.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.515+0000 7fba6ef91b80 1 client.27280 shutdown 2024-05-27T07:30:47.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2860667739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 msgr2=0x55b06dd414c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:47.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 0x55b06dd414c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fba50011ca0 tx=0x7fba50011fc0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2860667739 shutdown_connections 2024-05-27T07:30:47.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55b06dda4b40 0x55b06dda6f60 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b06dd25d30 0x55b06dd414c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b06dba0e40 0x55b06dd31f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 1 --2- 172.21.15.43:0/2860667739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b06dd1fdb0 0x55b06dd40f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba3f400700 0 client.27280 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:30:47.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2860667739 >> 172.21.15.43:0/2860667739 conn(0x55b06dc00fc0 msgr2=0x55b06dd238d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:47.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2860667739 shutdown_connections 2024-05-27T07:30:47.510 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:30:47.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 1 -- 172.21.15.43:0/2860667739 wait complete. 2024-05-27T07:30:47.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.516+0000 7fba6ef91b80 20 client.27280 trim_cache size 0 max 16384 2024-05-27T07:30:47.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.517+0000 7fba60400700 1 -- 172.21.15.43:0/3623368308 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba400b2a70 msgr2=0x7fba400b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:47.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.517+0000 7fba60400700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba400b2a70 0x7fba400b4f80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55b06dd20b30 tx=0x7fba50009310 comp rx=0 tx=0).stop 2024-05-27T07:30:47.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.518+0000 7fba60400700 1 -- 172.21.15.43:0/3623368308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba581034b0 msgr2=0x7fba5819cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:47.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.518+0000 7fba60400700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba581034b0 0x7fba5819cf60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00ef10 tx=0x7fba4c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.519+0000 7fba60400700 1 -- 172.21.15.43:0/3623368308 shutdown_connections 2024-05-27T07:30:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.519+0000 7fba60400700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fba400b2a70 0x7fba400b4f80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.519+0000 7fba60400700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba58108730 0x7fba581a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.519+0000 7fba60400700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fba581034b0 0x7fba5819cf60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.519+0000 7fba60400700 1 --2- 172.21.15.43:0/3623368308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba58107de0 0x7fba5819d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:47.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.519+0000 7fba60400700 1 -- 172.21.15.43:0/3623368308 >> 172.21.15.43:0/3623368308 conn(0x7fba580fedb0 msgr2=0x7fba581011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:47.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.521+0000 7fba60400700 1 -- 172.21.15.43:0/3623368308 shutdown_connections 2024-05-27T07:30:47.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:47.521+0000 7fba60400700 1 -- 172.21.15.43:0/3623368308 wait complete. 2024-05-27T07:30:47.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 19103 6.04 MB/sec execute 48 sec latency 341.048 ms 2024-05-27T07:30:47.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:47 smithi132 ceph-mon[35283]: pgmap v566: 129 pgs: 129 active+clean; 587 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 4.3 MiB/s wr, 240 op/s 2024-05-27T07:30:47.909 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:47 smithi043 ceph-mon[31531]: pgmap v566: 129 pgs: 129 active+clean; 587 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 4.3 MiB/s wr, 240 op/s 2024-05-27T07:30:47.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:47 smithi179 ceph-mon[35888]: pgmap v566: 129 pgs: 129 active+clean; 587 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 4.3 MiB/s wr, 240 op/s 2024-05-27T07:30:48.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:30:48.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 19245 6.13 MB/sec execute 49 sec latency 88.071 ms 2024-05-27T07:30:48.998 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:49.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 19373 6.15 MB/sec execute 50 sec latency 81.473 ms 2024-05-27T07:30:49.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:49 smithi132 ceph-mon[35283]: pgmap v567: 129 pgs: 129 active+clean; 587 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-05-27T07:30:49.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:49 smithi043 ceph-mon[31531]: pgmap v567: 129 pgs: 129 active+clean; 587 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-05-27T07:30:49.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:49 smithi179 ceph-mon[35888]: pgmap v567: 129 pgs: 129 active+clean; 587 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-05-27T07:30:50.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:50 smithi043 ceph-mon[31531]: pgmap v568: 129 pgs: 129 active+clean; 587 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-05-27T07:30:50.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 19493 6.14 MB/sec execute 51 sec latency 79.128 ms 2024-05-27T07:30:50.959 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:50 smithi179 ceph-mon[35888]: pgmap v568: 129 pgs: 129 active+clean; 587 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-05-27T07:30:51.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:50 smithi132 ceph-mon[35283]: pgmap v568: 129 pgs: 129 active+clean; 587 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-05-27T07:30:51.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.498+0000 7f140f800700 1 -- 172.21.15.43:0/3154378565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f140810b540 msgr2=0x7f1408119890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:51.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.498+0000 7f140f800700 1 --2- 172.21.15.43:0/3154378565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f140810b540 0x7f1408119890 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f1400009fc0 tx=0x7f140000b330 comp rx=0 tx=0).stop 2024-05-27T07:30:51.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.499+0000 7f140f800700 1 -- 172.21.15.43:0/3154378565 shutdown_connections 2024-05-27T07:30:51.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.499+0000 7f140f800700 1 --2- 172.21.15.43:0/3154378565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f140810b540 0x7f1408119890 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.499+0000 7f140f800700 1 --2- 172.21.15.43:0/3154378565 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1408111bb0 0x7f1408111fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.499+0000 7f140f800700 1 --2- 172.21.15.43:0/3154378565 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f140810abf0 0x7f140810b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.499+0000 7f140f800700 1 -- 172.21.15.43:0/3154378565 >> 172.21.15.43:0/3154378565 conn(0x7f140806f980 msgr2=0x7f140806fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:51.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.499+0000 7f140f800700 1 -- 172.21.15.43:0/3154378565 shutdown_connections 2024-05-27T07:30:51.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.499+0000 7f140f800700 1 -- 172.21.15.43:0/3154378565 wait complete. 2024-05-27T07:30:51.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.500+0000 7f140f800700 1 Processor -- start 2024-05-27T07:30:51.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.501+0000 7f140f800700 1 -- start start 2024-05-27T07:30:51.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.501+0000 7f140f800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f140810abf0 0x7f14081a5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.501+0000 7f140f800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f140810b540 0x7f14081a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.501+0000 7f140f800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1408111bb0 0x7f14081a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.501+0000 7f140f800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f140811cb70 con 0x7f140810b540 2024-05-27T07:30:51.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.501+0000 7f140f800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f140811c9f0 con 0x7f140810abf0 2024-05-27T07:30:51.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.501+0000 7f140f800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f140811ccf0 con 0x7f1408111bb0 2024-05-27T07:30:51.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.502+0000 7f140ce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f140810abf0 0x7f14081a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.502+0000 7f140ce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f140810abf0 0x7f14081a5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59656/0 (socket says 172.21.15.43:59656) 2024-05-27T07:30:51.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.502+0000 7f140ce00700 1 -- 172.21.15.43:0/2076129317 learned_addr learned my addr 172.21.15.43:0/2076129317 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:51.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.502+0000 7f1407e00700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f140810b540 0x7f14081a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.502+0000 7f140d800700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1408111bb0 0x7f14081a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.502+0000 7f140ce00700 1 -- 172.21.15.43:0/2076129317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1408111bb0 msgr2=0x7f14081a9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:51.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.502+0000 7f140ce00700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1408111bb0 0x7f14081a9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.502+0000 7f140ce00700 1 -- 172.21.15.43:0/2076129317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f140810b540 msgr2=0x7f14081a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:51.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.502+0000 7f140ce00700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f140810b540 0x7f14081a5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.502+0000 7f140ce00700 1 -- 172.21.15.43:0/2076129317 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14081aa630 con 0x7f140810abf0 2024-05-27T07:30:51.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.503+0000 7f140ce00700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f140810abf0 0x7f14081a5760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00e9c0 tx=0x7f13fc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:51.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.503+0000 7f1405600700 1 -- 172.21.15.43:0/2076129317 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13fc00cbd0 con 0x7f140810abf0 2024-05-27T07:30:51.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.503+0000 7f1405600700 1 -- 172.21.15.43:0/2076129317 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f13fc00cd70 con 0x7f140810abf0 2024-05-27T07:30:51.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.503+0000 7f1405600700 1 -- 172.21.15.43:0/2076129317 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13fc0054e0 con 0x7f140810abf0 2024-05-27T07:30:51.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.503+0000 7f140f800700 1 -- 172.21.15.43:0/2076129317 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14081aa920 con 0x7f140810abf0 2024-05-27T07:30:51.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.504+0000 7f140f800700 1 -- 172.21.15.43:0/2076129317 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14081b2240 con 0x7f140810abf0 2024-05-27T07:30:51.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.509+0000 7f1405600700 1 -- 172.21.15.43:0/2076129317 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f13fc004750 con 0x7f140810abf0 2024-05-27T07:30:51.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.510+0000 7f141e42fb80 1 Processor -- start 2024-05-27T07:30:51.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.510+0000 7f141e42fb80 1 -- start start 2024-05-27T07:30:51.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.510+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 0x561123c4c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.510+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c4c540 0x561123c46080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.510+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c465c0 0x561123c469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.510+0000 7f141e42fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561123b6be40 con 0x561123c465c0 2024-05-27T07:30:51.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f141e42fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561123b5b9e0 con 0x561123abee40 2024-05-27T07:30:51.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f141e42fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561123b5fa20 con 0x561123c4c540 2024-05-27T07:30:51.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f140d800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 0x561123c4c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f140d800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 0x561123c4c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59664/0 (socket says 172.21.15.43:59664) 2024-05-27T07:30:51.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f140d800700 1 -- 172.21.15.43:0/2522096823 learned_addr learned my addr 172.21.15.43:0/2522096823 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:51.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f140ce00700 1 --2- 172.21.15.43:0/2522096823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c4c540 0x561123c46080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f140d800700 1 -- 172.21.15.43:0/2522096823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c4c540 msgr2=0x561123c46080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f140d800700 1 --2- 172.21.15.43:0/2522096823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c4c540 0x561123c46080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f140d800700 1 -- 172.21.15.43:0/2522096823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c465c0 msgr2=0x561123c469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f140d800700 1 --2- 172.21.15.43:0/2522096823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c465c0 0x561123c469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.511+0000 7f140d800700 1 -- 172.21.15.43:0/2522096823 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561123b24420 con 0x561123abee40 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.512+0000 7f140d800700 1 --2- 172.21.15.43:0/2522096823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 0x561123c4c000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f14000094b0 tx=0x7f14000094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=338e987a5f3f7b18 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.512+0000 7f1405600700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13f00b2a70 0x7f13f00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.512+0000 7f1405600700 1 -- 172.21.15.43:0/2076129317 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f13fc0e2a50 con 0x7f140810abf0 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.513+0000 7f13fa000700 1 -- 172.21.15.43:0/2522096823 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400021070 con 0x561123abee40 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.513+0000 7f13fa000700 1 -- 172.21.15.43:0/2522096823 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1400004c50 con 0x561123abee40 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.513+0000 7f13fa000700 1 -- 172.21.15.43:0/2522096823 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400007bb0 con 0x561123abee40 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.514+0000 7f140ce00700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13f00b2a70 0x7f13f00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.515+0000 7f140ce00700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13f00b2a70 0x7f13f00b4f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0a5a00 tx=0x7f13fc00ef40 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:51.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.515+0000 7f141e42fb80 1 -- 172.21.15.43:0/2522096823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 msgr2=0x561123c4c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:51.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.515+0000 7f141e42fb80 1 --2- 172.21.15.43:0/2522096823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 0x561123c4c000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f14000094b0 tx=0x7f14000094e0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.515+0000 7f141e42fb80 1 -- 172.21.15.43:0/2522096823 shutdown_connections 2024-05-27T07:30:51.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.515+0000 7f141e42fb80 1 --2- 172.21.15.43:0/2522096823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c465c0 0x561123c469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.515+0000 7f141e42fb80 1 --2- 172.21.15.43:0/2522096823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 0x561123c4c000 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.515+0000 7f141e42fb80 1 --2- 172.21.15.43:0/2522096823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c4c540 0x561123c46080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.515+0000 7f141e42fb80 1 -- 172.21.15.43:0/2522096823 >> 172.21.15.43:0/2522096823 conn(0x561123b1efc0 msgr2=0x561123c388d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:51.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.515+0000 7f141e42fb80 1 -- 172.21.15.43:0/2522096823 shutdown_connections 2024-05-27T07:30:51.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.516+0000 7f141e42fb80 1 -- 172.21.15.43:0/2522096823 wait complete. 2024-05-27T07:30:51.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.516+0000 7f141e42fb80 1 -- start start 2024-05-27T07:30:51.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.517+0000 7f141e42fb80 1 Processor -- start 2024-05-27T07:30:51.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.518+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123abee40 0x561123c4fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.518+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c465c0 0x561123c502e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.518+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123c4c540 0x561123c50820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.518+0000 7f141e42fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561123a61fa0 con 0x561123c465c0 2024-05-27T07:30:51.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.518+0000 7f140d800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123abee40 0x561123c4fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.518+0000 7f141e42fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561123afc760 con 0x561123c4c540 2024-05-27T07:30:51.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.518+0000 7f141e42fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561123c52fb0 con 0x561123abee40 2024-05-27T07:30:51.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.519+0000 7f141e42fb80 10 client.0 _ll_register_callbacks cb 0x561123b05c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:51.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.519+0000 7f1407e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c465c0 0x561123c502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.519+0000 7f140d800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123abee40 0x561123c4fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:50728/0 (socket says 172.21.15.43:50728) 2024-05-27T07:30:51.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.519+0000 7f140d800700 1 -- 172.21.15.43:0/608100207 learned_addr learned my addr 172.21.15.43:0/608100207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:51.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.519+0000 7f140d800700 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123c4c540 0x561123c50820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.520+0000 7f1407e00700 1 -- 172.21.15.43:0/608100207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123abee40 msgr2=0x561123c4fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:51.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.520+0000 7f1407e00700 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123abee40 0x561123c4fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.520+0000 7f1407e00700 1 -- 172.21.15.43:0/608100207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123c4c540 msgr2=0x561123c50820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:51.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.520+0000 7f1407e00700 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123c4c540 0x561123c50820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.520+0000 7f1407e00700 1 -- 172.21.15.43:0/608100207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561123aab8d0 con 0x561123c465c0 2024-05-27T07:30:51.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.521+0000 7f1407e00700 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c465c0 0x561123c502e0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f13f40116f0 tx=0x7f13f4011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b802e2a6ab13e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:51.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.521+0000 7f13eea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:30:51.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.521+0000 7f13eea00700 1 -- 172.21.15.43:0/608100207 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f401e060 con 0x561123c465c0 2024-05-27T07:30:51.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.522+0000 7f13eea00700 1 -- 172.21.15.43:0/608100207 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f13f4004300 con 0x561123c465c0 2024-05-27T07:30:51.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.522+0000 7f141e42fb80 1 -- 172.21.15.43:0/608100207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1400004c50 con 0x561123c465c0 2024-05-27T07:30:51.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.523+0000 7f13eea00700 1 -- 172.21.15.43:0/608100207 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) ==== 24+0+0 (secure 0 0 0) 0x7f13f4025050 con 0x561123c465c0 2024-05-27T07:30:51.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.523+0000 7f141e42fb80 10 client.17814 fetch_fsmap learned FSMap version 133 2024-05-27T07:30:51.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.523+0000 7f141e42fb80 1 -- 172.21.15.43:0/608100207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=4+}) -- 0x561123b2c060 con 0x561123c465c0 2024-05-27T07:30:51.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.524+0000 7f13eea00700 1 -- 172.21.15.43:0/608100207 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1400004c50 con 0x561123c465c0 2024-05-27T07:30:51.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.524+0000 7f13eea00700 1 -- 172.21.15.43:0/608100207 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 133) ==== 4207+0+0 (secure 0 0 0) 0x7f13f40195a0 con 0x561123c465c0 2024-05-27T07:30:51.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.524+0000 7f141e42fb80 10 client.17814 fetch_fsmap finished waiting for FSMap version 133 2024-05-27T07:30:51.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.524+0000 7f141e42fb80 10 client.17814 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:51.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.524+0000 7f141e42fb80 20 client.17814 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:51.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.524+0000 7f141e42fb80 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561123c60360 0x561123c627d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.524+0000 7f141e42fb80 4 client.17814 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:51.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.524+0000 7f141e42fb80 1 -- 172.21.15.43:0/608100207 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561123b6dd90 con 0x561123c60360 2024-05-27T07:30:51.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.525+0000 7f140d800700 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561123c60360 0x561123c627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.527+0000 7f140d800700 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561123c60360 0x561123c627d0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4146d5721db65fc server_cookie=ba52dd17c319fd40 in_seq=0 out_seq=0 2024-05-27T07:30:51.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.527+0000 7f13eea00700 10 client.17814 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:51.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.528+0000 7f13eea00700 1 -- 172.21.15.43:0/608100207 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561123b6dd90 con 0x561123c60360 2024-05-27T07:30:51.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.528+0000 7f13eea00700 10 client.17814 handle_command_reply: tid=0 2024-05-27T07:30:51.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.529+0000 7f141e42fb80 1 client.17814 shutdown 2024-05-27T07:30:51.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.529+0000 7f141e42fb80 1 -- 172.21.15.43:0/608100207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c465c0 msgr2=0x561123c502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:51.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.529+0000 7f141e42fb80 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c465c0 0x561123c502e0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f13f40116f0 tx=0x7f13f4011a00 comp rx=0 tx=0).stop 2024-05-27T07:30:51.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.529+0000 7f141e42fb80 1 -- 172.21.15.43:0/608100207 shutdown_connections 2024-05-27T07:30:51.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.529+0000 7f141e42fb80 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123c4c540 0x561123c50820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.529+0000 7f141e42fb80 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123abee40 0x561123c4fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.529+0000 7f141e42fb80 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561123c60360 0x561123c627d0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.529+0000 7f141e42fb80 1 --2- 172.21.15.43:0/608100207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c465c0 0x561123c502e0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.529+0000 7f141e42fb80 1 -- 172.21.15.43:0/608100207 >> 172.21.15.43:0/608100207 conn(0x561123b1efc0 msgr2=0x561123c38580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:51.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.529+0000 7f13eea00700 0 client.17814 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:51.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.530+0000 7f141e42fb80 1 -- 172.21.15.43:0/608100207 shutdown_connections 2024-05-27T07:30:51.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.530+0000 7f141e42fb80 1 -- 172.21.15.43:0/608100207 wait complete. 2024-05-27T07:30:51.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.530+0000 7f141e42fb80 20 client.17814 trim_cache size 0 max 16384 2024-05-27T07:30:51.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.563+0000 7f141e42fb80 1 Processor -- start 2024-05-27T07:30:51.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.563+0000 7f141e42fb80 1 -- start start 2024-05-27T07:30:51.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.564+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123abee40 0x561123c4b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.564+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123c465c0 0x561123c4fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.564+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c4c540 0x561123c500c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.564+0000 7f141e42fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b7150 con 0x561123abee40 2024-05-27T07:30:51.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.564+0000 7f141e42fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b5520 con 0x561123c465c0 2024-05-27T07:30:51.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.564+0000 7f141e42fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b6a80 con 0x561123c4c540 2024-05-27T07:30:51.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.564+0000 7f140d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123abee40 0x561123c4b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.565+0000 7f140d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123abee40 0x561123c4b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58160/0 (socket says 172.21.15.43:58160) 2024-05-27T07:30:51.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.565+0000 7f140d800700 1 -- 172.21.15.43:0/1176529298 learned_addr learned my addr 172.21.15.43:0/1176529298 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:51.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.565+0000 7f140d800700 1 --2- 172.21.15.43:0/1176529298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c4c540 0x561123c500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.565+0000 7f1407e00700 1 --2- 172.21.15.43:0/1176529298 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123c465c0 0x561123c4fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.566+0000 7f140d800700 1 -- 172.21.15.43:0/1176529298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c4c540 msgr2=0x561123c500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:51.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.566+0000 7f140d800700 1 --2- 172.21.15.43:0/1176529298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c4c540 0x561123c500c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.566+0000 7f140d800700 1 -- 172.21.15.43:0/1176529298 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123c465c0 msgr2=0x561123c4fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:51.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.566+0000 7f140d800700 1 --2- 172.21.15.43:0/1176529298 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123c465c0 0x561123c4fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.566+0000 7f140d800700 1 -- 172.21.15.43:0/1176529298 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561123c4e310 con 0x561123abee40 2024-05-27T07:30:51.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.567+0000 7f140d800700 1 --2- 172.21.15.43:0/1176529298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123abee40 0x561123c4b610 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f1400000f80 tx=0x7f1400027040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8039f300b845dd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:51.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.568+0000 7f13f9600700 1 -- 172.21.15.43:0/1176529298 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400021040 con 0x561123abee40 2024-05-27T07:30:51.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.568+0000 7f13f9600700 1 -- 172.21.15.43:0/1176529298 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f14000076e0 con 0x561123abee40 2024-05-27T07:30:51.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.568+0000 7f13f9600700 1 -- 172.21.15.43:0/1176529298 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400021040 con 0x561123abee40 2024-05-27T07:30:51.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.569+0000 7f141e42fb80 1 -- 172.21.15.43:0/1176529298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123abee40 msgr2=0x561123c4b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:51.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.569+0000 7f141e42fb80 1 --2- 172.21.15.43:0/1176529298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123abee40 0x561123c4b610 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f1400000f80 tx=0x7f1400027040 comp rx=0 tx=0).stop 2024-05-27T07:30:51.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.570+0000 7f141e42fb80 1 -- 172.21.15.43:0/1176529298 shutdown_connections 2024-05-27T07:30:51.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.570+0000 7f141e42fb80 1 --2- 172.21.15.43:0/1176529298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c4c540 0x561123c500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.570+0000 7f141e42fb80 1 --2- 172.21.15.43:0/1176529298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123abee40 0x561123c4b610 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.570+0000 7f141e42fb80 1 --2- 172.21.15.43:0/1176529298 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123c465c0 0x561123c4fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.570+0000 7f141e42fb80 1 -- 172.21.15.43:0/1176529298 >> 172.21.15.43:0/1176529298 conn(0x561123b1efc0 msgr2=0x561123c4af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:51.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.571+0000 7f141e42fb80 1 -- 172.21.15.43:0/1176529298 shutdown_connections 2024-05-27T07:30:51.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.571+0000 7f141e42fb80 1 -- 172.21.15.43:0/1176529298 wait complete. 2024-05-27T07:30:51.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.572+0000 7f141e42fb80 1 -- start start 2024-05-27T07:30:51.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.572+0000 7f141e42fb80 1 Processor -- start 2024-05-27T07:30:51.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.573+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 0x561123c58200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.573+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c465c0 0x561123c58740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.573+0000 7f141e42fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c4c540 0x561123c45070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.573+0000 7f141e42fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b62f0 con 0x561123c4c540 2024-05-27T07:30:51.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.573+0000 7f141e42fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b5c20 con 0x561123abee40 2024-05-27T07:30:51.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.573+0000 7f141e42fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561123c52ba0 con 0x561123c465c0 2024-05-27T07:30:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.573+0000 7f141e42fb80 10 client.0 _ll_register_callbacks cb 0x561123c5c850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.574+0000 7f1407e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c465c0 0x561123c58740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.575+0000 7f1407e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c465c0 0x561123c58740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:50738/0 (socket says 172.21.15.43:50738) 2024-05-27T07:30:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.575+0000 7f1407e00700 1 -- 172.21.15.43:0/2143079178 learned_addr learned my addr 172.21.15.43:0/2143079178 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:51.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.575+0000 7f140d800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c4c540 0x561123c45070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.576+0000 7f140d800700 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 0x561123c58200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.576+0000 7f140d800700 1 -- 172.21.15.43:0/2143079178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c465c0 msgr2=0x561123c58740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:51.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.576+0000 7f140d800700 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c465c0 0x561123c58740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.576+0000 7f140d800700 1 -- 172.21.15.43:0/2143079178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 msgr2=0x561123c58200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:51.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.576+0000 7f140d800700 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 0x561123c58200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.576+0000 7f140d800700 1 -- 172.21.15.43:0/2143079178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561123c38c80 con 0x561123c4c540 2024-05-27T07:30:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.577+0000 7f140d800700 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c4c540 0x561123c45070 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f1400009550 tx=0x7f1400007500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a07d6382908b35 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.577+0000 7f13eea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:30:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.578+0000 7f13eea00700 1 -- 172.21.15.43:0/2143079178 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400021070 con 0x561123c4c540 2024-05-27T07:30:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.578+0000 7f13eea00700 1 -- 172.21.15.43:0/2143079178 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f1400007a90 con 0x561123c4c540 2024-05-27T07:30:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.578+0000 7f141e42fb80 1 -- 172.21.15.43:0/2143079178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14000076e0 con 0x561123c4c540 2024-05-27T07:30:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.578+0000 7f13eea00700 1 -- 172.21.15.43:0/2143079178 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) ==== 24+0+0 (secure 0 0 0) 0x7f1400007c90 con 0x561123c4c540 2024-05-27T07:30:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.578+0000 7f141e42fb80 10 client.17820 fetch_fsmap learned FSMap version 133 2024-05-27T07:30:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.578+0000 7f141e42fb80 1 -- 172.21.15.43:0/2143079178 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=4+}) -- 0x561123cca2a0 con 0x561123c4c540 2024-05-27T07:30:51.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.579+0000 7f13eea00700 1 -- 172.21.15.43:0/2143079178 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f14000076e0 con 0x561123c4c540 2024-05-27T07:30:51.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.579+0000 7f13eea00700 1 -- 172.21.15.43:0/2143079178 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 133) ==== 4207+0+0 (secure 0 0 0) 0x7f1400011070 con 0x561123c4c540 2024-05-27T07:30:51.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.579+0000 7f141e42fb80 10 client.17820 fetch_fsmap finished waiting for FSMap version 133 2024-05-27T07:30:51.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.579+0000 7f141e42fb80 10 client.17820 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:51.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.579+0000 7f141e42fb80 20 client.17820 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:51.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.579+0000 7f141e42fb80 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561123cca700 0x561123cccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:51.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.579+0000 7f141e42fb80 4 client.17820 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:30:51.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.579+0000 7f141e42fb80 1 -- 172.21.15.43:0/2143079178 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561123b67570 con 0x561123cca700 2024-05-27T07:30:51.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.580+0000 7f1407e00700 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561123cca700 0x561123cccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:51.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.582+0000 7f13eea00700 10 client.17820 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:51.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.582+0000 7f1407e00700 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561123cca700 0x561123cccb20 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebcc8cf65546a6c3 server_cookie=b899c0a00a0658cb in_seq=0 out_seq=0 2024-05-27T07:30:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.583+0000 7f13eea00700 1 -- 172.21.15.43:0/2143079178 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561123b67570 con 0x561123cca700 2024-05-27T07:30:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.583+0000 7f13eea00700 10 client.17820 handle_command_reply: tid=0 2024-05-27T07:30:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.583+0000 7f141e42fb80 1 client.17820 shutdown 2024-05-27T07:30:51.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.583+0000 7f141e42fb80 1 -- 172.21.15.43:0/2143079178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c4c540 msgr2=0x561123c45070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:51.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.583+0000 7f141e42fb80 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c4c540 0x561123c45070 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f1400009550 tx=0x7f1400007500 comp rx=0 tx=0).stop 2024-05-27T07:30:51.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.584+0000 7f141e42fb80 1 -- 172.21.15.43:0/2143079178 shutdown_connections 2024-05-27T07:30:51.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.584+0000 7f141e42fb80 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561123cca700 0x561123cccb20 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.584+0000 7f141e42fb80 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561123c4c540 0x561123c45070 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.584+0000 7f141e42fb80 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561123abee40 0x561123c58200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.584+0000 7f13eea00700 0 client.17820 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:51.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.584+0000 7f141e42fb80 1 --2- 172.21.15.43:0/2143079178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561123c465c0 0x561123c58740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.584+0000 7f141e42fb80 1 -- 172.21.15.43:0/2143079178 >> 172.21.15.43:0/2143079178 conn(0x561123b1efc0 msgr2=0x561123c49950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:51.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.585+0000 7f141e42fb80 1 -- 172.21.15.43:0/2143079178 shutdown_connections 2024-05-27T07:30:51.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.585+0000 7f141e42fb80 1 -- 172.21.15.43:0/2143079178 wait complete. 2024-05-27T07:30:51.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.586+0000 7f141e42fb80 20 client.17820 trim_cache size 0 max 16384 2024-05-27T07:30:51.577 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:30:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 -- 172.21.15.43:0/2076129317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13f00b2a70 msgr2=0x7f13f00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13f00b2a70 0x7f13f00b4f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0a5a00 tx=0x7f13fc00ef40 comp rx=0 tx=0).stop 2024-05-27T07:30:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 -- 172.21.15.43:0/2076129317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f140810abf0 msgr2=0x7f14081a5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f140810abf0 0x7f14081a5760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00e9c0 tx=0x7f13fc00ecd0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 -- 172.21.15.43:0/2076129317 shutdown_connections 2024-05-27T07:30:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f13f00b2a70 0x7f13f00b4f80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1408111bb0 0x7f14081a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f140810abf0 0x7f14081a5760 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 --2- 172.21.15.43:0/2076129317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f140810b540 0x7f14081a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 -- 172.21.15.43:0/2076129317 >> 172.21.15.43:0/2076129317 conn(0x7f140806f980 msgr2=0x7f1408110200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.587+0000 7f140f800700 1 -- 172.21.15.43:0/2076129317 shutdown_connections 2024-05-27T07:30:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:51.588+0000 7f140f800700 1 -- 172.21.15.43:0/2076129317 wait complete. 2024-05-27T07:30:51.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 19578 6.11 MB/sec execute 52 sec latency 112.583 ms 2024-05-27T07:30:52.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 19696 6.11 MB/sec execute 53 sec latency 84.273 ms 2024-05-27T07:30:53.630 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:30:53.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:30:53.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:53 smithi043 ceph-mon[31531]: pgmap v569: 129 pgs: 129 active+clean; 600 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 4.7 MiB/s wr, 233 op/s 2024-05-27T07:30:53.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:53 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:53.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:53 smithi179 ceph-mon[35888]: pgmap v569: 129 pgs: 129 active+clean; 600 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 4.7 MiB/s wr, 233 op/s 2024-05-27T07:30:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:53 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:53.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 19829 6.16 MB/sec execute 54 sec latency 106.231 ms 2024-05-27T07:30:53.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:53 smithi132 ceph-mon[35283]: pgmap v569: 129 pgs: 129 active+clean; 600 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 4.7 MiB/s wr, 233 op/s 2024-05-27T07:30:53.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:53 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:30:54.330 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:54.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:54.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:54.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:30:54.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:54.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:54.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:30:54.779 INFO:tasks.workunit.client.0.smithi043.stdout: 10 19945 6.18 MB/sec execute 55 sec latency 118.085 ms 2024-05-27T07:30:54.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:30:54.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:30:54.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:30:55.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:55 smithi043 ceph-mon[31531]: pgmap v570: 129 pgs: 129 active+clean; 600 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-05-27T07:30:55.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:55 smithi043 ceph-mon[31531]: pgmap v571: 129 pgs: 129 active+clean; 600 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 3.5 MiB/s wr, 171 op/s 2024-05-27T07:30:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:55 smithi179 ceph-mon[35888]: pgmap v570: 129 pgs: 129 active+clean; 600 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-05-27T07:30:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:55 smithi179 ceph-mon[35888]: pgmap v571: 129 pgs: 129 active+clean; 600 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 3.5 MiB/s wr, 171 op/s 2024-05-27T07:30:55.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:55 smithi132 ceph-mon[35283]: pgmap v570: 129 pgs: 129 active+clean; 600 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-05-27T07:30:55.743 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:55 smithi132 ceph-mon[35283]: pgmap v571: 129 pgs: 129 active+clean; 600 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 3.5 MiB/s wr, 171 op/s 2024-05-27T07:30:55.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20071 6.18 MB/sec execute 56 sec latency 101.206 ms 2024-05-27T07:30:56.780 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20168 6.15 MB/sec execute 57 sec latency 94.927 ms 2024-05-27T07:30:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.374+0000 7f113a000700 1 -- 172.21.15.43:0/3856854051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f113410e8d0 msgr2=0x7f1134110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.374+0000 7f113a000700 1 --2- 172.21.15.43:0/3856854051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f113410e8d0 0x7f1134110db0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f112c009f90 tx=0x7f112c00b330 comp rx=0 tx=0).stop 2024-05-27T07:30:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.374+0000 7f113a000700 1 -- 172.21.15.43:0/3856854051 shutdown_connections 2024-05-27T07:30:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.374+0000 7f113a000700 1 --2- 172.21.15.43:0/3856854051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f113410e8d0 0x7f1134110db0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.374+0000 7f113a000700 1 --2- 172.21.15.43:0/3856854051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11341054f0 0x7f1134105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.374+0000 7f113a000700 1 --2- 172.21.15.43:0/3856854051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1134105e40 0x7f113410e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.374+0000 7f113a000700 1 -- 172.21.15.43:0/3856854051 >> 172.21.15.43:0/3856854051 conn(0x7f11340fedb0 msgr2=0x7f11341011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.374+0000 7f113a000700 1 -- 172.21.15.43:0/3856854051 shutdown_connections 2024-05-27T07:30:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.375+0000 7f113a000700 1 -- 172.21.15.43:0/3856854051 wait complete. 2024-05-27T07:30:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.376+0000 7f113a000700 1 Processor -- start 2024-05-27T07:30:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.377+0000 7f113a000700 1 -- start start 2024-05-27T07:30:57.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.377+0000 7f113a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11341054f0 0x7f11341a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.377+0000 7f113a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1134105e40 0x7f11341a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.377+0000 7f113a000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f113410e8d0 0x7f113419aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.377+0000 7f113a000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11341142e0 con 0x7f1134105e40 2024-05-27T07:30:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.377+0000 7f113a000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1134114160 con 0x7f11341054f0 2024-05-27T07:30:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.378+0000 7f113a000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1134114460 con 0x7f113410e8d0 2024-05-27T07:30:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.378+0000 7f1133400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11341054f0 0x7f11341a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.378+0000 7f1133400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11341054f0 0x7f11341a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:50558/0 (socket says 172.21.15.43:50558) 2024-05-27T07:30:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.378+0000 7f1133400700 1 -- 172.21.15.43:0/3432233240 learned_addr learned my addr 172.21.15.43:0/3432233240 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.378+0000 7f1132a00700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1134105e40 0x7f11341a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.378+0000 7f1133e00700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f113410e8d0 0x7f113419aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.379+0000 7f1132a00700 1 -- 172.21.15.43:0/3432233240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f113410e8d0 msgr2=0x7f113419aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.379+0000 7f1132a00700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f113410e8d0 0x7f113419aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.379+0000 7f1132a00700 1 -- 172.21.15.43:0/3432233240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11341054f0 msgr2=0x7f11341a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.379+0000 7f1132a00700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11341054f0 0x7f11341a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.379+0000 7f1132a00700 1 -- 172.21.15.43:0/3432233240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113419b150 con 0x7f1134105e40 2024-05-27T07:30:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.380+0000 7f1132a00700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1134105e40 0x7f11341a0da0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f112000d750 tx=0x7f112000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:57.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.380+0000 7f112be00700 1 -- 172.21.15.43:0/3432233240 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120014030 con 0x7f1134105e40 2024-05-27T07:30:57.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.380+0000 7f112be00700 1 -- 172.21.15.43:0/3432233240 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11200044b0 con 0x7f1134105e40 2024-05-27T07:30:57.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.380+0000 7f113a000700 1 -- 172.21.15.43:0/3432233240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f113419b440 con 0x7f1134105e40 2024-05-27T07:30:57.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.380+0000 7f112be00700 1 -- 172.21.15.43:0/3432233240 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120005260 con 0x7f1134105e40 2024-05-27T07:30:57.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.381+0000 7f113a000700 1 -- 172.21.15.43:0/3432233240 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1134103110 con 0x7f1134105e40 2024-05-27T07:30:57.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.387+0000 7f112be00700 1 -- 172.21.15.43:0/3432233240 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f11200049e0 con 0x7f1134105e40 2024-05-27T07:30:57.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.388+0000 7f1148bdfb80 1 Processor -- start 2024-05-27T07:30:57.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.388+0000 7f1148bdfb80 1 -- start start 2024-05-27T07:30:57.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.389+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 0x55c80c99e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.389+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c99e5c0 0x55c80c998100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.390+0000 7f112be00700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f11180b29f0 0x7f11180b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.390+0000 7f1133e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 0x55c80c99e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.390+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c998a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.390+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c80c8bdec0 con 0x55c80c810ec0 2024-05-27T07:30:57.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.390+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c80c8ada60 con 0x55c80c998640 2024-05-27T07:30:57.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.390+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c80c8b1aa0 con 0x55c80c99e5c0 2024-05-27T07:30:57.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.390+0000 7f1133e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 0x55c80c99e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35918/0 (socket says 172.21.15.43:35918) 2024-05-27T07:30:57.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.390+0000 7f1133e00700 1 -- 172.21.15.43:0/3296657129 learned_addr learned my addr 172.21.15.43:0/3296657129 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:57.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.390+0000 7f1133400700 1 --2- 172.21.15.43:0/3296657129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c99e5c0 0x55c80c998100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.390+0000 7f1133e00700 1 --2- 172.21.15.43:0/3296657129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c998a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.391+0000 7f112be00700 1 -- 172.21.15.43:0/3432233240 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f11200e2540 con 0x7f1134105e40 2024-05-27T07:30:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.391+0000 7f1132a00700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f11180b29f0 0x7f11180b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.391+0000 7f1133e00700 1 -- 172.21.15.43:0/3296657129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c99e5c0 msgr2=0x55c80c998100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.391+0000 7f1133e00700 1 --2- 172.21.15.43:0/3296657129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c99e5c0 0x55c80c998100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.391+0000 7f1133e00700 1 -- 172.21.15.43:0/3296657129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 msgr2=0x55c80c998a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.391+0000 7f1133e00700 1 --2- 172.21.15.43:0/3296657129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c998a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.391+0000 7f1133e00700 1 -- 172.21.15.43:0/3296657129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c80c8764a0 con 0x55c80c810ec0 2024-05-27T07:30:57.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.392+0000 7f1133e00700 1 --2- 172.21.15.43:0/3296657129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 0x55c80c99e080 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f112c0094f0 tx=0x7f112c0097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4597dd9dd44ef602 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:57.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.392+0000 7f1128c00700 1 -- 172.21.15.43:0/3296657129 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c00bbf0 con 0x55c80c810ec0 2024-05-27T07:30:57.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.393+0000 7f1132a00700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f11180b29f0 0x7f11180b4f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f11200a6720 tx=0x7f11200058e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:57.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.393+0000 7f1128c00700 1 -- 172.21.15.43:0/3296657129 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f112c00bd90 con 0x55c80c810ec0 2024-05-27T07:30:57.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.393+0000 7f1128c00700 1 -- 172.21.15.43:0/3296657129 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c0207d0 con 0x55c80c810ec0 2024-05-27T07:30:57.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.394+0000 7f1148bdfb80 1 -- 172.21.15.43:0/3296657129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 msgr2=0x55c80c99e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:57.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.394+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/3296657129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 0x55c80c99e080 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f112c0094f0 tx=0x7f112c0097e0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.394+0000 7f1148bdfb80 1 -- 172.21.15.43:0/3296657129 shutdown_connections 2024-05-27T07:30:57.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.394+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/3296657129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c998a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.394+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/3296657129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 0x55c80c99e080 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.394+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/3296657129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c99e5c0 0x55c80c998100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.394+0000 7f1148bdfb80 1 -- 172.21.15.43:0/3296657129 >> 172.21.15.43:0/3296657129 conn(0x55c80c871040 msgr2=0x55c80c98a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:57.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.395+0000 7f1148bdfb80 1 -- 172.21.15.43:0/3296657129 shutdown_connections 2024-05-27T07:30:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.395+0000 7f1148bdfb80 1 -- 172.21.15.43:0/3296657129 wait complete. 2024-05-27T07:30:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.396+0000 7f1148bdfb80 1 -- start start 2024-05-27T07:30:57.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.396+0000 7f1148bdfb80 1 Processor -- start 2024-05-27T07:30:57.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.396+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c810ec0 0x55c80c9a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.396+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9a2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.396+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 0x55c80c9a2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.396+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c80c7b4020 con 0x55c80c99e5c0 2024-05-27T07:30:57.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.396+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c80c84e7e0 con 0x55c80c998640 2024-05-27T07:30:57.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.396+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c80c9a5030 con 0x55c80c810ec0 2024-05-27T07:30:57.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.397+0000 7f1148bdfb80 10 client.0 _ll_register_callbacks cb 0x55c80c857cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:57.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.397+0000 7f1133400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.397+0000 7f1133e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 0x55c80c9a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.397+0000 7f1133400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9a2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:50574/0 (socket says 172.21.15.43:50574) 2024-05-27T07:30:57.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.397+0000 7f1133400700 1 -- 172.21.15.43:0/1697625313 learned_addr learned my addr 172.21.15.43:0/1697625313 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:57.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.398+0000 7f1133e00700 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c810ec0 0x55c80c9a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.398+0000 7f1133e00700 1 -- 172.21.15.43:0/1697625313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c810ec0 msgr2=0x55c80c9a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:57.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.398+0000 7f1133e00700 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c810ec0 0x55c80c9a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.398+0000 7f1133e00700 1 -- 172.21.15.43:0/1697625313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 msgr2=0x55c80c9a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.398+0000 7f1133e00700 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.398+0000 7f1133e00700 1 -- 172.21.15.43:0/1697625313 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c80c7fd950 con 0x55c80c99e5c0 2024-05-27T07:30:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.399+0000 7f1133e00700 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 0x55c80c9a2870 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x55c80c9a1670 tx=0x7f112c0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1c0a1f6f4cdf51 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.399+0000 7f111d600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:30:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.399+0000 7f111d600700 1 -- 172.21.15.43:0/1697625313 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c007940 con 0x55c80c99e5c0 2024-05-27T07:30:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.399+0000 7f111d600700 1 -- 172.21.15.43:0/1697625313 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f112c030580 con 0x55c80c99e5c0 2024-05-27T07:30:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.399+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1697625313 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f112c00bd90 con 0x55c80c99e5c0 2024-05-27T07:30:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.399+0000 7f111d600700 1 -- 172.21.15.43:0/1697625313 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) ==== 24+0+0 (secure 0 0 0) 0x7f112c030050 con 0x55c80c99e5c0 2024-05-27T07:30:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.399+0000 7f1148bdfb80 10 client.17832 fetch_fsmap learned FSMap version 133 2024-05-27T07:30:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.399+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1697625313 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=4+}) -- 0x55c80c87e0e0 con 0x55c80c99e5c0 2024-05-27T07:30:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.400+0000 7f111d600700 1 -- 172.21.15.43:0/1697625313 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f112c00bd90 con 0x55c80c99e5c0 2024-05-27T07:30:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.400+0000 7f111d600700 1 -- 172.21.15.43:0/1697625313 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 133) ==== 4207+0+0 (secure 0 0 0) 0x7f112c011070 con 0x55c80c99e5c0 2024-05-27T07:30:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.400+0000 7f1148bdfb80 10 client.17832 fetch_fsmap finished waiting for FSMap version 133 2024-05-27T07:30:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.400+0000 7f1148bdfb80 10 client.17832 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.400+0000 7f1148bdfb80 20 client.17832 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.400+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c80c9b20b0 0x55c80c9b4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.400+0000 7f1148bdfb80 4 client.17832 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:30:57.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.400+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1697625313 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c80c8bfe10 con 0x55c80c9b20b0 2024-05-27T07:30:57.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.401+0000 7f1133400700 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c80c9b20b0 0x55c80c9b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.402+0000 7f1133400700 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c80c9b20b0 0x55c80c9b4500 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ba461823b6a65a4 server_cookie=f886711b22c19583 in_seq=0 out_seq=0 2024-05-27T07:30:57.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.402+0000 7f111d600700 10 client.17832 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.403+0000 7f111d600700 1 -- 172.21.15.43:0/1697625313 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c80c8bfe10 con 0x55c80c9b20b0 2024-05-27T07:30:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.403+0000 7f111d600700 10 client.17832 handle_command_reply: tid=0 2024-05-27T07:30:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f1148bdfb80 1 client.17832 shutdown 2024-05-27T07:30:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1697625313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 msgr2=0x55c80c9a2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 0x55c80c9a2870 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x55c80c9a1670 tx=0x7f112c0096c0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1697625313 shutdown_connections 2024-05-27T07:30:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c80c9b20b0 0x55c80c9b4500 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 0x55c80c9a2870 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c810ec0 0x55c80c9a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1697625313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f111d600700 0 client.17832 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:57.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1697625313 >> 172.21.15.43:0/1697625313 conn(0x55c80c871040 msgr2=0x55c80c98a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:57.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.404+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1697625313 shutdown_connections 2024-05-27T07:30:57.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.405+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1697625313 wait complete. 2024-05-27T07:30:57.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.405+0000 7f1148bdfb80 20 client.17832 trim_cache size 0 max 16384 2024-05-27T07:30:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.440+0000 7f1148bdfb80 1 Processor -- start 2024-05-27T07:30:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.440+0000 7f1148bdfb80 1 -- start start 2024-05-27T07:30:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.441+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 0x55c80c998100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.441+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c998640 0x55c80c9a4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.441+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c99e5c0 0x55c80c9ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.441+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11341ae740 con 0x55c80c810ec0 2024-05-27T07:30:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.441+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f11341acb10 con 0x55c80c99e5c0 2024-05-27T07:30:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.441+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f11341ae070 con 0x55c80c998640 2024-05-27T07:30:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.442+0000 7f1133e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c99e5c0 0x55c80c9ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.442+0000 7f1133e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c99e5c0 0x55c80c9ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:50588/0 (socket says 172.21.15.43:50588) 2024-05-27T07:30:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.442+0000 7f1133e00700 1 -- 172.21.15.43:0/2757692388 learned_addr learned my addr 172.21.15.43:0/2757692388 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.443+0000 7f1133e00700 1 -- 172.21.15.43:0/2757692388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c998640 msgr2=0x55c80c9a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.443+0000 7f1133e00700 1 --2- 172.21.15.43:0/2757692388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c998640 0x55c80c9a4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.443+0000 7f1133e00700 1 -- 172.21.15.43:0/2757692388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 msgr2=0x55c80c998100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.443+0000 7f1133e00700 1 --2- 172.21.15.43:0/2757692388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 0x55c80c998100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.443+0000 7f1133e00700 1 -- 172.21.15.43:0/2757692388 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c80c9a1120 con 0x55c80c99e5c0 2024-05-27T07:30:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.443+0000 7f1133e00700 1 --2- 172.21.15.43:0/2757692388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c99e5c0 0x55c80c9ae090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f112c009f90 tx=0x7f112c0092e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e6503a05a108b80 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.444+0000 7f111fe00700 1 -- 172.21.15.43:0/2757692388 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c00bd90 con 0x55c80c99e5c0 2024-05-27T07:30:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.444+0000 7f111fe00700 1 -- 172.21.15.43:0/2757692388 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f112c003d40 con 0x55c80c99e5c0 2024-05-27T07:30:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.444+0000 7f111fe00700 1 -- 172.21.15.43:0/2757692388 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c007940 con 0x55c80c99e5c0 2024-05-27T07:30:57.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.445+0000 7f1148bdfb80 1 -- 172.21.15.43:0/2757692388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c99e5c0 msgr2=0x55c80c9ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:57.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.445+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/2757692388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c99e5c0 0x55c80c9ae090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f112c009f90 tx=0x7f112c0092e0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.445+0000 7f1148bdfb80 1 -- 172.21.15.43:0/2757692388 shutdown_connections 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.445+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/2757692388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c99e5c0 0x55c80c9ae090 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.445+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/2757692388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c810ec0 0x55c80c998100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.445+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/2757692388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c998640 0x55c80c9a4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.445+0000 7f1148bdfb80 1 -- 172.21.15.43:0/2757692388 >> 172.21.15.43:0/2757692388 conn(0x55c80c871040 msgr2=0x55c80c9a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.445+0000 7f1148bdfb80 1 -- 172.21.15.43:0/2757692388 shutdown_connections 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.445+0000 7f1148bdfb80 1 -- 172.21.15.43:0/2757692388 wait complete. 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.446+0000 7f1148bdfb80 1 -- start start 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.446+0000 7f1148bdfb80 1 Processor -- start 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.447+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c810ec0 0x55c80c9aa280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.447+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9aa820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.447+0000 7f1148bdfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 0x55c80c9a7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.447+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11341ad8e0 con 0x55c80c99e5c0 2024-05-27T07:30:57.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.447+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f11341ad210 con 0x55c80c998640 2024-05-27T07:30:57.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.447+0000 7f1148bdfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c80c986dc0 con 0x55c80c810ec0 2024-05-27T07:30:57.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.447+0000 7f1148bdfb80 10 client.0 _ll_register_callbacks cb 0x55c80c857cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:30:57.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.448+0000 7f1133400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9aa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.448+0000 7f1133400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9aa820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:50594/0 (socket says 172.21.15.43:50594) 2024-05-27T07:30:57.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.448+0000 7f1133400700 1 -- 172.21.15.43:0/1752129021 learned_addr learned my addr 172.21.15.43:0/1752129021 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:30:57.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.448+0000 7f1133e00700 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 0x55c80c9a7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.448+0000 7f1133400700 1 -- 172.21.15.43:0/1752129021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c810ec0 msgr2=0x55c80c9aa280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:57.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.448+0000 7f1133400700 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c810ec0 0x55c80c9aa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.448+0000 7f1133400700 1 -- 172.21.15.43:0/1752129021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 msgr2=0x55c80c9a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:30:57.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.448+0000 7f1133400700 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 0x55c80c9a7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.448+0000 7f1133400700 1 -- 172.21.15.43:0/1752129021 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c80c9a7860 con 0x55c80c998640 2024-05-27T07:30:57.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.449+0000 7f1133400700 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9aa820 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1124009260 tx=0x7f1124005600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37ae1f21b244223e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:30:57.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.449+0000 7f111d600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:30:57.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.449+0000 7f111d600700 1 -- 172.21.15.43:0/1752129021 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1124010630 con 0x55c80c998640 2024-05-27T07:30:57.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.450+0000 7f111d600700 1 -- 172.21.15.43:0/1752129021 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f11240107d0 con 0x55c80c998640 2024-05-27T07:30:57.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.450+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1752129021 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f112c003d40 con 0x55c80c998640 2024-05-27T07:30:57.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.450+0000 7f111d600700 1 -- 172.21.15.43:0/1752129021 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) ==== 24+0+0 (secure 0 0 0) 0x7f1124004e50 con 0x55c80c998640 2024-05-27T07:30:57.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.450+0000 7f1148bdfb80 10 client.37229 fetch_fsmap learned FSMap version 133 2024-05-27T07:30:57.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.450+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1752129021 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=4+}) -- 0x55c80ca1d530 con 0x55c80c998640 2024-05-27T07:30:57.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.451+0000 7f111d600700 1 -- 172.21.15.43:0/1752129021 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f112c003d40 con 0x55c80c998640 2024-05-27T07:30:57.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.451+0000 7f111d600700 1 -- 172.21.15.43:0/1752129021 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 133) ==== 4207+0+0 (secure 0 0 0) 0x7f112401eae0 con 0x55c80c998640 2024-05-27T07:30:57.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.451+0000 7f1148bdfb80 10 client.37229 fetch_fsmap finished waiting for FSMap version 133 2024-05-27T07:30:57.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.451+0000 7f1148bdfb80 10 client.37229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:30:57.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.451+0000 7f1148bdfb80 20 client.37229 populate_metadata read hostname 'smithi043' 2024-05-27T07:30:57.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.451+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c80ca1dd30 0x55c80ca20150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:30:57.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.451+0000 7f1148bdfb80 4 client.37229 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:30:57.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.451+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1752129021 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c80c8b95f0 con 0x55c80ca1dd30 2024-05-27T07:30:57.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.451+0000 7f1133e00700 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c80ca1dd30 0x55c80ca20150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:30:57.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.452+0000 7f1133e00700 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c80ca1dd30 0x55c80ca20150 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57e65d714a697812 server_cookie=f858859b95f90253 in_seq=0 out_seq=0 2024-05-27T07:30:57.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.452+0000 7f111d600700 10 client.37229 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:57.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f111d600700 1 -- 172.21.15.43:0/1752129021 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c80c8b95f0 con 0x55c80ca1dd30 2024-05-27T07:30:57.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f111d600700 10 client.37229 handle_command_reply: tid=0 2024-05-27T07:30:57.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f1148bdfb80 1 client.37229 shutdown 2024-05-27T07:30:57.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1752129021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 msgr2=0x55c80c9aa820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:57.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9aa820 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1124009260 tx=0x7f1124005600 comp rx=0 tx=0).stop 2024-05-27T07:30:57.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1752129021 shutdown_connections 2024-05-27T07:30:57.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c80ca1dd30 0x55c80ca20150 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c80c99e5c0 0x55c80c9a7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c80c810ec0 0x55c80c9aa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f1148bdfb80 1 --2- 172.21.15.43:0/1752129021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c80c998640 0x55c80c9aa820 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1752129021 >> 172.21.15.43:0/1752129021 conn(0x55c80c871040 msgr2=0x55c80c9a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:57.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.455+0000 7f111d600700 0 client.37229 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:30:57.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.456+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1752129021 shutdown_connections 2024-05-27T07:30:57.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.456+0000 7f1148bdfb80 1 -- 172.21.15.43:0/1752129021 wait complete. 2024-05-27T07:30:57.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.456+0000 7f1148bdfb80 20 client.37229 trim_cache size 0 max 16384 2024-05-27T07:30:57.449 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:30:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:30:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "dc435ed1-40c1-40a9-a1ec-79d26684405a", 2024-05-27T07:30:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:30:57.449 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.457+0000 7f113a000700 1 -- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f11180b29f0 msgr2=0x7f11180b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.457+0000 7f113a000700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f11180b29f0 0x7f11180b4f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f11200a6720 tx=0x7f11200058e0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.457+0000 7f113a000700 1 -- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1134105e40 msgr2=0x7f11341a0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.457+0000 7f113a000700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1134105e40 0x7f11341a0da0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f112000d750 tx=0x7f112000da60 comp rx=0 tx=0).stop 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.457+0000 7f113a000700 1 -- 172.21.15.43:0/3432233240 shutdown_connections 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.457+0000 7f113a000700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f11180b29f0 0x7f11180b4f00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.457+0000 7f113a000700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f113410e8d0 0x7f113419aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.457+0000 7f113a000700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f11341054f0 0x7f11341a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.457+0000 7f113a000700 1 --2- 172.21.15.43:0/3432233240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1134105e40 0x7f11341a0da0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.457+0000 7f113a000700 1 -- 172.21.15.43:0/3432233240 >> 172.21.15.43:0/3432233240 conn(0x7f11340fedb0 msgr2=0x7f11340ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.458+0000 7f113a000700 1 -- 172.21.15.43:0/3432233240 shutdown_connections 2024-05-27T07:30:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:30:57.458+0000 7f113a000700 1 -- 172.21.15.43:0/3432233240 wait complete. 2024-05-27T07:30:57.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:57 smithi043 ceph-mon[31531]: pgmap v572: 129 pgs: 129 active+clean; 602 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-27T07:30:57.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:57 smithi179 ceph-mon[35888]: pgmap v572: 129 pgs: 129 active+clean; 602 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-27T07:30:57.780 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20257 6.15 MB/sec execute 58 sec latency 126.567 ms 2024-05-27T07:30:57.841 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:57 smithi132 ceph-mon[35283]: pgmap v572: 129 pgs: 129 active+clean; 602 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-27T07:30:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:30:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:30:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:30:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:30:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:30:58.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:30:58.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:30:58.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:58 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:30:58.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:58 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.780 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20364 6.13 MB/sec execute 59 sec latency 74.526 ms 2024-05-27T07:30:58.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:30:58.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:30:58.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:30:58.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:30:58.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:30:58.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:30:58.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:30:58.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:58.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:58 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:59.078 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:30:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:30:59.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:59 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:59.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:30:59 smithi043 ceph-mon[31531]: pgmap v573: 129 pgs: 129 active+clean; 602 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-27T07:30:59.709 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:59 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:59.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:30:59 smithi179 ceph-mon[35888]: pgmap v573: 129 pgs: 129 active+clean; 602 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-27T07:30:59.780 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20487 6.24 MB/sec execute 60 sec latency 87.554 ms 2024-05-27T07:30:59.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:59 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:30:59.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:30:59 smithi132 ceph-mon[35283]: pgmap v573: 129 pgs: 129 active+clean; 602 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-27T07:31:00.780 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20567 6.20 MB/sec execute 61 sec latency 381.036 ms 2024-05-27T07:31:01.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:01 smithi043 ceph-mon[31531]: pgmap v574: 129 pgs: 129 active+clean; 602 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-27T07:31:01.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:01 smithi179 ceph-mon[35888]: pgmap v574: 129 pgs: 129 active+clean; 602 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-27T07:31:01.780 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20567 6.10 MB/sec execute 62 sec latency 1381.217 ms 2024-05-27T07:31:01.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:01 smithi132 ceph-mon[35283]: pgmap v574: 129 pgs: 129 active+clean; 602 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.9 MiB/s wr, 176 op/s 2024-05-27T07:31:02.780 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20567 6.01 MB/sec execute 63 sec latency 2381.356 ms 2024-05-27T07:31:03.781 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20580 5.91 MB/sec execute 64 sec latency 2580.691 ms 2024-05-27T07:31:03.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:03 smithi132 ceph-mon[35283]: pgmap v575: 129 pgs: 129 active+clean; 610 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-05-27T07:31:03.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:03 smithi043 ceph-mon[31531]: pgmap v575: 129 pgs: 129 active+clean; 610 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-05-27T07:31:03.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:03 smithi179 ceph-mon[35888]: pgmap v575: 129 pgs: 129 active+clean; 610 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-05-27T07:31:04.781 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20580 5.82 MB/sec execute 65 sec latency 1737.987 ms 2024-05-27T07:31:05.781 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20580 5.73 MB/sec execute 66 sec latency 2738.140 ms 2024-05-27T07:31:05.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:05 smithi132 ceph-mon[35283]: pgmap v576: 129 pgs: 129 active+clean; 610 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-05-27T07:31:05.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:05 smithi043 ceph-mon[31531]: pgmap v576: 129 pgs: 129 active+clean; 610 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-05-27T07:31:05.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:05 smithi179 ceph-mon[35888]: pgmap v576: 129 pgs: 129 active+clean; 610 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-05-27T07:31:06.781 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20580 5.65 MB/sec execute 67 sec latency 3738.301 ms 2024-05-27T07:31:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.628+0000 7f3270800700 1 -- 172.21.15.43:0/1178154209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3268109880 msgr2=0x7f32681111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.628+0000 7f3270800700 1 --2- 172.21.15.43:0/1178154209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3268109880 0x7f32681111e0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f326400b0d0 tx=0x7f326400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:07 smithi043 ceph-mon[31531]: pgmap v577: 129 pgs: 129 active+clean; 607 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-05-27T07:31:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.628+0000 7f3270800700 1 -- 172.21.15.43:0/1178154209 shutdown_connections 2024-05-27T07:31:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.628+0000 7f3270800700 1 --2- 172.21.15.43:0/1178154209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3268109880 0x7f32681111e0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.628+0000 7f3270800700 1 --2- 172.21.15.43:0/1178154209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3268104600 0x7f3268104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.628+0000 7f3270800700 1 --2- 172.21.15.43:0/1178154209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3268108f30 0x7f3268109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.628+0000 7f3270800700 1 -- 172.21.15.43:0/1178154209 >> 172.21.15.43:0/1178154209 conn(0x7f326807af60 msgr2=0x7f326807b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.629+0000 7f3270800700 1 -- 172.21.15.43:0/1178154209 shutdown_connections 2024-05-27T07:31:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.629+0000 7f3270800700 1 -- 172.21.15.43:0/1178154209 wait complete. 2024-05-27T07:31:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.630+0000 7f3270800700 1 Processor -- start 2024-05-27T07:31:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.630+0000 7f3270800700 1 -- start start 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.631+0000 7f3270800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3268104600 0x7f326819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.631+0000 7f3270800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3268108f30 0x7f326819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.631+0000 7f3270800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3268109880 0x7f32681a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.631+0000 7f3270800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3268114420 con 0x7f3268109880 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.631+0000 7f3270800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32681142a0 con 0x7f3268104600 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.631+0000 7f3270800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f32681145a0 con 0x7f3268108f30 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.631+0000 7f326e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3268109880 0x7f32681a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.631+0000 7f326e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3268109880 0x7f32681a1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44800/0 (socket says 172.21.15.43:44800) 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.631+0000 7f326e800700 1 -- 172.21.15.43:0/4002931711 learned_addr learned my addr 172.21.15.43:0/4002931711 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.632+0000 7f326e800700 1 -- 172.21.15.43:0/4002931711 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3268108f30 msgr2=0x7f326819d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.632+0000 7f326e800700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3268108f30 0x7f326819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.632+0000 7f326e800700 1 -- 172.21.15.43:0/4002931711 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3268104600 msgr2=0x7f326819cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:07.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.632+0000 7f326e800700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3268104600 0x7f326819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.632+0000 7f326e800700 1 -- 172.21.15.43:0/4002931711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32681a1ea0 con 0x7f3268109880 2024-05-27T07:31:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.632+0000 7f326e800700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3268109880 0x7f32681a1800 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3264007d10 tx=0x7f3264007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.632+0000 7f3262a00700 1 -- 172.21.15.43:0/4002931711 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326401f070 con 0x7f3268109880 2024-05-27T07:31:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.633+0000 7f3262a00700 1 -- 172.21.15.43:0/4002931711 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3264003e10 con 0x7f3268109880 2024-05-27T07:31:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.633+0000 7f3270800700 1 -- 172.21.15.43:0/4002931711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32681a2130 con 0x7f3268109880 2024-05-27T07:31:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.633+0000 7f3262a00700 1 -- 172.21.15.43:0/4002931711 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326401b820 con 0x7f3268109880 2024-05-27T07:31:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.633+0000 7f3270800700 1 -- 172.21.15.43:0/4002931711 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32681a2610 con 0x7f3268109880 2024-05-27T07:31:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.637+0000 7f3262a00700 1 -- 172.21.15.43:0/4002931711 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3264022070 con 0x7f3268109880 2024-05-27T07:31:07.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.638+0000 7f327f32eb80 1 Processor -- start 2024-05-27T07:31:07.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.638+0000 7f327f32eb80 1 -- start start 2024-05-27T07:31:07.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.639+0000 7f3262a00700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32500b29f0 0x7f32500b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.639+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd82330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.639+0000 7f3262a00700 1 -- 172.21.15.43:0/4002931711 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f326400f020 con 0x7f3268109880 2024-05-27T07:31:07.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.639+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd82870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.639+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.639+0000 7f327f32eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5560ffcb0e40 con 0x5560ffd82e70 2024-05-27T07:31:07.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.639+0000 7f327f32eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5560ffca09e0 con 0x5560ffc03e40 2024-05-27T07:31:07.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.639+0000 7f327f32eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5560ffca4a20 con 0x5560ffcef9f0 2024-05-27T07:31:07.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.639+0000 7f326d400700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32500b29f0 0x7f32500b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44802/0 (socket says 172.21.15.43:44802) 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326e800700 1 -- 172.21.15.43:0/1140338349 learned_addr learned my addr 172.21.15.43:0/1140338349 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326d400700 1 --2- 172.21.15.43:0/1140338349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd82870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326de00700 1 --2- 172.21.15.43:0/1140338349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd82330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326e800700 1 -- 172.21.15.43:0/1140338349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 msgr2=0x5560ffd82330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326e800700 1 --2- 172.21.15.43:0/1140338349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd82330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326e800700 1 -- 172.21.15.43:0/1140338349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 msgr2=0x5560ffd82870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326e800700 1 --2- 172.21.15.43:0/1140338349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd82870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326e800700 1 -- 172.21.15.43:0/1140338349 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560ffc69420 con 0x5560ffd82e70 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326d400700 1 --2- 172.21.15.43:0/1140338349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd82870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.640+0000 7f326d400700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32500b29f0 0x7f32500b4f00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3258006fd0 tx=0x7f32580125b0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:07.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.641+0000 7f326e800700 1 --2- 172.21.15.43:0/1140338349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f110 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f32640a6fb0 tx=0x7f32640aaa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53396fb4f2c54549 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:07.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.641+0000 7f3257400700 1 -- 172.21.15.43:0/1140338349 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32640af2e0 con 0x5560ffd82e70 2024-05-27T07:31:07.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.641+0000 7f3257400700 1 -- 172.21.15.43:0/1140338349 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32640b3d00 con 0x5560ffd82e70 2024-05-27T07:31:07.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.641+0000 7f3257400700 1 -- 172.21.15.43:0/1140338349 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32640ab190 con 0x5560ffd82e70 2024-05-27T07:31:07.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.642+0000 7f327f32eb80 1 -- 172.21.15.43:0/1140338349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 msgr2=0x5560ffd8f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:07.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.642+0000 7f327f32eb80 1 --2- 172.21.15.43:0/1140338349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f110 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f32640a6fb0 tx=0x7f32640aaa40 comp rx=0 tx=0).stop 2024-05-27T07:31:07.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.642+0000 7f327f32eb80 1 -- 172.21.15.43:0/1140338349 shutdown_connections 2024-05-27T07:31:07.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.642+0000 7f327f32eb80 1 --2- 172.21.15.43:0/1140338349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f110 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.642+0000 7f327f32eb80 1 --2- 172.21.15.43:0/1140338349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd82330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.642+0000 7f327f32eb80 1 --2- 172.21.15.43:0/1140338349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd82870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.642+0000 7f327f32eb80 1 -- 172.21.15.43:0/1140338349 >> 172.21.15.43:0/1140338349 conn(0x5560ffc9d390 msgr2=0x5560ffd7e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:07.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.642+0000 7f327f32eb80 1 -- 172.21.15.43:0/1140338349 shutdown_connections 2024-05-27T07:31:07.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.642+0000 7f327f32eb80 1 -- 172.21.15.43:0/1140338349 wait complete. 2024-05-27T07:31:07.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.643+0000 7f327f32eb80 1 -- start start 2024-05-27T07:31:07.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.643+0000 7f327f32eb80 1 Processor -- start 2024-05-27T07:31:07.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd8c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd8cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f327f32eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5560ffba6fa0 con 0x5560ffd82e70 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f327f32eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5560ffc41760 con 0x5560ffc03e40 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f327f32eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5560ffcb0e40 con 0x5560ffcef9f0 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f327f32eb80 10 client.0 _ll_register_callbacks cb 0x5560ffc4ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f326e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f326e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44810/0 (socket says 172.21.15.43:44810) 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f326e800700 1 -- 172.21.15.43:0/1733125284 learned_addr learned my addr 172.21.15.43:0/1733125284 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f326d400700 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd8cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.644+0000 7f326de00700 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd8c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f326e800700 1 -- 172.21.15.43:0/1733125284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 msgr2=0x5560ffd8cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:07.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f326e800700 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd8cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f326e800700 1 -- 172.21.15.43:0/1733125284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 msgr2=0x5560ffd8c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:07.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f326e800700 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd8c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f326e800700 1 -- 172.21.15.43:0/1733125284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560ffbf08d0 con 0x5560ffd82e70 2024-05-27T07:31:07.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f326de00700 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd8c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:31:07.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f326e800700 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8d1e0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f32640e1890 tx=0x7f32640e1320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791cc8debd1df1f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:07.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f324be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:31:07.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f324be00700 1 -- 172.21.15.43:0/1733125284 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326406e020 con 0x5560ffd82e70 2024-05-27T07:31:07.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f324be00700 1 -- 172.21.15.43:0/1733125284 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f32640ab390 con 0x5560ffd82e70 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.645+0000 7f327f32eb80 1 -- 172.21.15.43:0/1733125284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32640b3d00 con 0x5560ffd82e70 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f324be00700 1 -- 172.21.15.43:0/1733125284 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) ==== 24+0+0 (secure 0 0 0) 0x7f3264023c40 con 0x5560ffd82e70 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f327f32eb80 10 client.17844 fetch_fsmap learned FSMap version 134 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f327f32eb80 1 -- 172.21.15.43:0/1733125284 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=4+}) -- 0x5560ffc71060 con 0x5560ffd82e70 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f324be00700 1 -- 172.21.15.43:0/1733125284 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32640b3d00 con 0x5560ffd82e70 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f324be00700 1 -- 172.21.15.43:0/1733125284 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 134) ==== 4199+0+0 (secure 0 0 0) 0x7f32640c1030 con 0x5560ffd82e70 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f327f32eb80 10 client.17844 fetch_fsmap finished waiting for FSMap version 134 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f327f32eb80 10 client.17844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f327f32eb80 20 client.17844 populate_metadata read hostname 'smithi043' 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f327f32eb80 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5560ffd9c600 0x5560ffd9ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f327f32eb80 4 client.17844 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.646+0000 7f327f32eb80 1 -- 172.21.15.43:0/1733125284 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560ffcb2d90 con 0x5560ffd9c600 2024-05-27T07:31:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.647+0000 7f326de00700 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5560ffd9c600 0x5560ffd9ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.648+0000 7f326de00700 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5560ffd9c600 0x5560ffd9ea50 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90d8ba93c6220ae7 server_cookie=9a6fae33ca9491 in_seq=0 out_seq=0 2024-05-27T07:31:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.648+0000 7f324be00700 10 client.17844 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:31:07.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f324be00700 1 -- 172.21.15.43:0/1733125284 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5560ffcb2d90 con 0x5560ffd9c600 2024-05-27T07:31:07.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f324be00700 10 client.17844 handle_command_reply: tid=0 2024-05-27T07:31:07.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 client.17844 shutdown 2024-05-27T07:31:07.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 -- 172.21.15.43:0/1733125284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 msgr2=0x5560ffd8d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:07.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8d1e0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f32640e1890 tx=0x7f32640e1320 comp rx=0 tx=0).stop 2024-05-27T07:31:07.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 -- 172.21.15.43:0/1733125284 shutdown_connections 2024-05-27T07:31:07.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5560ffd9c600 0x5560ffd9ea50 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8d1e0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd8c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 --2- 172.21.15.43:0/1733125284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd8cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 -- 172.21.15.43:0/1733125284 >> 172.21.15.43:0/1733125284 conn(0x5560ffc9d390 msgr2=0x5560ffd823b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:07.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f324be00700 0 client.17844 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:31:07.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 -- 172.21.15.43:0/1733125284 shutdown_connections 2024-05-27T07:31:07.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.651+0000 7f327f32eb80 1 -- 172.21.15.43:0/1733125284 wait complete. 2024-05-27T07:31:07.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.652+0000 7f327f32eb80 20 client.17844 trim_cache size 0 max 16384 2024-05-27T07:31:07.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.674+0000 7f327f32eb80 1 Processor -- start 2024-05-27T07:31:07.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.674+0000 7f327f32eb80 1 -- start start 2024-05-27T07:31:07.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.675+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffc03e40 0x5560ffd87f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.675+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffcef9f0 0x5560ffd848b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.675+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffd82e70 0x5560ffd8e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.675+0000 7f327f32eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5560ffd7e7e0 con 0x5560ffcef9f0 2024-05-27T07:31:07.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.675+0000 7f327f32eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32681b2bd0 con 0x5560ffd82e70 2024-05-27T07:31:07.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.675+0000 7f327f32eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f32681b1000 con 0x5560ffc03e40 2024-05-27T07:31:07.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.675+0000 7f326d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffcef9f0 0x5560ffd848b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.675+0000 7f326d400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffcef9f0 0x5560ffd848b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44822/0 (socket says 172.21.15.43:44822) 2024-05-27T07:31:07.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.675+0000 7f326d400700 1 -- 172.21.15.43:0/2287339049 learned_addr learned my addr 172.21.15.43:0/2287339049 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:07.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.676+0000 7f326de00700 1 --2- 172.21.15.43:0/2287339049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffc03e40 0x5560ffd87f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.676+0000 7f326e800700 1 --2- 172.21.15.43:0/2287339049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffd82e70 0x5560ffd8e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.676+0000 7f326d400700 1 -- 172.21.15.43:0/2287339049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffc03e40 msgr2=0x5560ffd87f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:07.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.676+0000 7f326d400700 1 --2- 172.21.15.43:0/2287339049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffc03e40 0x5560ffd87f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.676+0000 7f326d400700 1 -- 172.21.15.43:0/2287339049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffd82e70 msgr2=0x5560ffd8e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:07.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.676+0000 7f326d400700 1 --2- 172.21.15.43:0/2287339049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffd82e70 0x5560ffd8e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.676+0000 7f326d400700 1 -- 172.21.15.43:0/2287339049 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560ffd7f000 con 0x5560ffcef9f0 2024-05-27T07:31:07.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.676+0000 7f326d400700 1 --2- 172.21.15.43:0/2287339049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffcef9f0 0x5560ffd848b0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f326806b4f0 tx=0x7f3258019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8ea0099320d5d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:07.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.677+0000 7f3256a00700 1 -- 172.21.15.43:0/2287339049 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258027070 con 0x5560ffcef9f0 2024-05-27T07:31:07.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.677+0000 7f3256a00700 1 -- 172.21.15.43:0/2287339049 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3258012e60 con 0x5560ffcef9f0 2024-05-27T07:31:07.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.677+0000 7f3256a00700 1 -- 172.21.15.43:0/2287339049 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258015740 con 0x5560ffcef9f0 2024-05-27T07:31:07.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.678+0000 7f327f32eb80 1 -- 172.21.15.43:0/2287339049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffcef9f0 msgr2=0x5560ffd848b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:07.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.678+0000 7f327f32eb80 1 --2- 172.21.15.43:0/2287339049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffcef9f0 0x5560ffd848b0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f326806b4f0 tx=0x7f3258019000 comp rx=0 tx=0).stop 2024-05-27T07:31:07.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.678+0000 7f327f32eb80 1 -- 172.21.15.43:0/2287339049 shutdown_connections 2024-05-27T07:31:07.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.678+0000 7f327f32eb80 1 --2- 172.21.15.43:0/2287339049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffd82e70 0x5560ffd8e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.678+0000 7f327f32eb80 1 --2- 172.21.15.43:0/2287339049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffc03e40 0x5560ffd87f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.678+0000 7f327f32eb80 1 --2- 172.21.15.43:0/2287339049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffcef9f0 0x5560ffd848b0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.678+0000 7f327f32eb80 1 -- 172.21.15.43:0/2287339049 >> 172.21.15.43:0/2287339049 conn(0x5560ffc9d390 msgr2=0x5560ffd8c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:07.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.678+0000 7f327f32eb80 1 -- 172.21.15.43:0/2287339049 shutdown_connections 2024-05-27T07:31:07.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.678+0000 7f327f32eb80 1 -- 172.21.15.43:0/2287339049 wait complete. 2024-05-27T07:31:07.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.678+0000 7f327f32eb80 1 -- start start 2024-05-27T07:31:07.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.679+0000 7f327f32eb80 1 Processor -- start 2024-05-27T07:31:07.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.679+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd94a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.679+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd94f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.679+0000 7f327f32eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.679+0000 7f327f32eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32681b1700 con 0x5560ffd82e70 2024-05-27T07:31:07.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.679+0000 7f327f32eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32681b2500 con 0x5560ffc03e40 2024-05-27T07:31:07.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.679+0000 7f327f32eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f32681b1dd0 con 0x5560ffcef9f0 2024-05-27T07:31:07.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.679+0000 7f327f32eb80 10 client.0 _ll_register_callbacks cb 0x5560ffc4ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:07.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.680+0000 7f326e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.680+0000 7f326e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44824/0 (socket says 172.21.15.43:44824) 2024-05-27T07:31:07.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.680+0000 7f326e800700 1 -- 172.21.15.43:0/3864801930 learned_addr learned my addr 172.21.15.43:0/3864801930 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:07.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.680+0000 7f326e800700 1 -- 172.21.15.43:0/3864801930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 msgr2=0x5560ffd94f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:07.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.680+0000 7f326e800700 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd94f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.680+0000 7f326e800700 1 -- 172.21.15.43:0/3864801930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 msgr2=0x5560ffd94a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:07.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.680+0000 7f326e800700 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd94a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.680+0000 7f326e800700 1 -- 172.21.15.43:0/3864801930 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560ffd8ff10 con 0x5560ffd82e70 2024-05-27T07:31:07.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.680+0000 7f326e800700 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f9d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f32640aac30 tx=0x7f32640aa750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5abf7258366017 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:07.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.680+0000 7f324be00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:31:07.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.681+0000 7f324be00700 1 -- 172.21.15.43:0/3864801930 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326406e070 con 0x5560ffd82e70 2024-05-27T07:31:07.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.681+0000 7f324be00700 1 -- 172.21.15.43:0/3864801930 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 9408+0+0 (secure 0 0 0) 0x7f3264005bc0 con 0x5560ffd82e70 2024-05-27T07:31:07.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.681+0000 7f327f32eb80 1 -- 172.21.15.43:0/3864801930 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3258012e60 con 0x5560ffd82e70 2024-05-27T07:31:07.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.681+0000 7f324be00700 1 -- 172.21.15.43:0/3864801930 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) ==== 24+0+0 (secure 0 0 0) 0x7f32640a9f70 con 0x5560ffd82e70 2024-05-27T07:31:07.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.681+0000 7f327f32eb80 10 client.17850 fetch_fsmap learned FSMap version 134 2024-05-27T07:31:07.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.681+0000 7f327f32eb80 1 -- 172.21.15.43:0/3864801930 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=4+}) -- 0x5560ffd8ff10 con 0x5560ffd82e70 2024-05-27T07:31:07.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.681+0000 7f324be00700 1 -- 172.21.15.43:0/3864801930 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3258012e60 con 0x5560ffd82e70 2024-05-27T07:31:07.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.681+0000 7f324be00700 1 -- 172.21.15.43:0/3864801930 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 134) ==== 4199+0+0 (secure 0 0 0) 0x7f32640c1070 con 0x5560ffd82e70 2024-05-27T07:31:07.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.682+0000 7f327f32eb80 10 client.17850 fetch_fsmap finished waiting for FSMap version 134 2024-05-27T07:31:07.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.682+0000 7f327f32eb80 10 client.17850 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:07.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.682+0000 7f327f32eb80 20 client.17850 populate_metadata read hostname 'smithi043' 2024-05-27T07:31:07.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.682+0000 7f327f32eb80 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5560ffe10300 0x5560ffe12720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:07.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.682+0000 7f327f32eb80 4 client.17850 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:31:07.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.682+0000 7f327f32eb80 1 -- 172.21.15.43:0/3864801930 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5560ffcac570 con 0x5560ffe10300 2024-05-27T07:31:07.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.684+0000 7f326de00700 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5560ffe10300 0x5560ffe12720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:07.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.685+0000 7f326de00700 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5560ffe10300 0x5560ffe12720 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c672c2359690e003 server_cookie=4c241a46fbe34239 in_seq=0 out_seq=0 2024-05-27T07:31:07.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.685+0000 7f324be00700 10 client.17850 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:31:07.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.685+0000 7f324be00700 1 -- 172.21.15.43:0/3864801930 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5560ffcac570 con 0x5560ffe10300 2024-05-27T07:31:07.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.685+0000 7f324be00700 10 client.17850 handle_command_reply: tid=0 2024-05-27T07:31:07.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.685+0000 7f327f32eb80 1 client.17850 shutdown 2024-05-27T07:31:07.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 1 -- 172.21.15.43:0/3864801930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 msgr2=0x5560ffd8f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:07.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f9d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f32640aac30 tx=0x7f32640aa750 comp rx=0 tx=0).stop 2024-05-27T07:31:07.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 1 -- 172.21.15.43:0/3864801930 shutdown_connections 2024-05-27T07:31:07.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5560ffe10300 0x5560ffe12720 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5560ffd82e70 0x5560ffd8f9d0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560ffc03e40 0x5560ffd94a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 1 --2- 172.21.15.43:0/3864801930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5560ffcef9f0 0x5560ffd94f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 1 -- 172.21.15.43:0/3864801930 >> 172.21.15.43:0/3864801930 conn(0x5560ffc9d390 msgr2=0x5560ffd8cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:07.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f324be00700 0 client.17850 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:31:07.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 1 -- 172.21.15.43:0/3864801930 shutdown_connections 2024-05-27T07:31:07.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 1 -- 172.21.15.43:0/3864801930 wait complete. 2024-05-27T07:31:07.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.686+0000 7f327f32eb80 20 client.17850 trim_cache size 0 max 16384 2024-05-27T07:31:07.679 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:31:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:31:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:31:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "dc435ed1-40c1-40a9-a1ec-79d26684405a": { 2024-05-27T07:31:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:31:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "dc435ed1-40c1-40a9-a1ec-79d26684405a", 2024-05-27T07:31:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:31:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:31:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:31:07.680 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:31:07.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 -- 172.21.15.43:0/4002931711 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32500b29f0 msgr2=0x7f32500b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:07.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32500b29f0 0x7f32500b4f00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3258006fd0 tx=0x7f32580125b0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 -- 172.21.15.43:0/4002931711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3268109880 msgr2=0x7f32681a1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:07.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3268109880 0x7f32681a1800 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3264007d10 tx=0x7f3264007df0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 -- 172.21.15.43:0/4002931711 shutdown_connections 2024-05-27T07:31:07.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32500b29f0 0x7f32500b4f00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3268109880 0x7f32681a1800 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3268104600 0x7f326819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 --2- 172.21.15.43:0/4002931711 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3268108f30 0x7f326819d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:07.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 -- 172.21.15.43:0/4002931711 >> 172.21.15.43:0/4002931711 conn(0x7f326807af60 msgr2=0x7f3268110f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:07.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 -- 172.21.15.43:0/4002931711 shutdown_connections 2024-05-27T07:31:07.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:07.687+0000 7f3270800700 1 -- 172.21.15.43:0/4002931711 wait complete. 2024-05-27T07:31:07.781 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20580 5.57 MB/sec execute 68 sec latency 4738.441 ms 2024-05-27T07:31:07.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:07 smithi132 ceph-mon[35283]: pgmap v577: 129 pgs: 129 active+clean; 607 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-05-27T07:31:07.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:07 smithi179 ceph-mon[35888]: pgmap v577: 129 pgs: 129 active+clean; 607 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-05-27T07:31:08.707 INFO:tasks.cephfs.filesystem:scrub status for tag:dc435ed1-40c1-40a9-a1ec-79d26684405a - {'path': '/', 'tag': 'dc435ed1-40c1-40a9-a1ec-79d26684405a', 'options': 'recursive,force'} 2024-05-27T07:31:08.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20684 5.58 MB/sec execute 69 sec latency 4864.930 ms 2024-05-27T07:31:08.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:08 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:31:08.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:08 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:31:08.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:08 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:31:09.781 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20804 5.53 MB/sec execute 70 sec latency 77.320 ms 2024-05-27T07:31:09.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:09 smithi132 ceph-mon[35283]: pgmap v578: 129 pgs: 129 active+clean; 607 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 106 op/s 2024-05-27T07:31:09.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:09 smithi043 ceph-mon[31531]: pgmap v578: 129 pgs: 129 active+clean; 607 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 106 op/s 2024-05-27T07:31:09.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:09 smithi179 ceph-mon[35888]: pgmap v578: 129 pgs: 129 active+clean; 607 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 106 op/s 2024-05-27T07:31:10.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 20921 5.49 MB/sec execute 71 sec latency 89.589 ms 2024-05-27T07:31:11.781 INFO:tasks.workunit.client.0.smithi043.stdout: 10 21026 5.46 MB/sec execute 72 sec latency 89.524 ms 2024-05-27T07:31:11.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:11 smithi132 ceph-mon[35283]: pgmap v579: 129 pgs: 129 active+clean; 607 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 106 op/s 2024-05-27T07:31:11.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:11 smithi043 ceph-mon[31531]: pgmap v579: 129 pgs: 129 active+clean; 607 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 106 op/s 2024-05-27T07:31:11.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:11 smithi179 ceph-mon[35888]: pgmap v579: 129 pgs: 129 active+clean; 607 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 106 op/s 2024-05-27T07:31:12.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 21133 5.42 MB/sec execute 73 sec latency 105.337 ms 2024-05-27T07:31:12.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:12 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:31:12.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:12 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:31:12.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:12 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:31:12.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:12 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:31:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:12 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:31:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:12 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:31:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:12 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:31:12.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:12 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:12 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:12 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:12 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:31:12.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:12 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:31:13.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 21255 5.44 MB/sec execute 74 sec latency 159.164 ms 2024-05-27T07:31:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:13 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:31:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:13 smithi132 ceph-mon[35283]: pgmap v580: 129 pgs: 129 active+clean; 617 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.6 MiB/s wr, 155 op/s 2024-05-27T07:31:13.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:13 smithi132 ceph-mon[35283]: osdmap e144: 12 total, 12 up, 12 in 2024-05-27T07:31:13.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:13 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:31:13.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:13 smithi043 ceph-mon[31531]: pgmap v580: 129 pgs: 129 active+clean; 617 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.6 MiB/s wr, 155 op/s 2024-05-27T07:31:13.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:13 smithi043 ceph-mon[31531]: osdmap e144: 12 total, 12 up, 12 in 2024-05-27T07:31:13.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:13 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:31:13.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:13 smithi179 ceph-mon[35888]: pgmap v580: 129 pgs: 129 active+clean; 617 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.6 MiB/s wr, 155 op/s 2024-05-27T07:31:13.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:13 smithi179 ceph-mon[35888]: osdmap e144: 12 total, 12 up, 12 in 2024-05-27T07:31:14.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 21393 5.46 MB/sec execute 75 sec latency 169.510 ms 2024-05-27T07:31:15.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 21508 5.44 MB/sec execute 76 sec latency 78.817 ms 2024-05-27T07:31:15.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:15 smithi132 ceph-mon[35283]: pgmap v582: 129 pgs: 129 active+clean; 617 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.2 MiB/s wr, 95 op/s 2024-05-27T07:31:15.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:15 smithi043 ceph-mon[31531]: pgmap v582: 129 pgs: 129 active+clean; 617 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.2 MiB/s wr, 95 op/s 2024-05-27T07:31:15.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:15 smithi179 ceph-mon[35888]: pgmap v582: 129 pgs: 129 active+clean; 617 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.2 MiB/s wr, 95 op/s 2024-05-27T07:31:16.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 21643 5.47 MB/sec execute 77 sec latency 75.752 ms 2024-05-27T07:31:17.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 21782 5.50 MB/sec execute 78 sec latency 72.612 ms 2024-05-27T07:31:17.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:17 smithi132 ceph-mon[35283]: pgmap v583: 129 pgs: 129 active+clean; 629 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-05-27T07:31:17.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:17 smithi043 ceph-mon[31531]: pgmap v583: 129 pgs: 129 active+clean; 629 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-05-27T07:31:17.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:17 smithi179 ceph-mon[35888]: pgmap v583: 129 pgs: 129 active+clean; 629 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-05-27T07:31:18.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 21921 5.57 MB/sec execute 79 sec latency 86.354 ms 2024-05-27T07:31:19.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 22062 5.62 MB/sec execute 80 sec latency 79.782 ms 2024-05-27T07:31:19.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:19 smithi132 ceph-mon[35283]: pgmap v584: 129 pgs: 129 active+clean; 629 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-05-27T07:31:19.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:19 smithi043 ceph-mon[31531]: pgmap v584: 129 pgs: 129 active+clean; 629 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-05-27T07:31:19.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:19 smithi179 ceph-mon[35888]: pgmap v584: 129 pgs: 129 active+clean; 629 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-05-27T07:31:20.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 22205 5.62 MB/sec execute 81 sec latency 67.475 ms 2024-05-27T07:31:21.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 22348 5.66 MB/sec execute 82 sec latency 157.386 ms 2024-05-27T07:31:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:21 smithi132 ceph-mon[35283]: pgmap v585: 129 pgs: 129 active+clean; 629 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-05-27T07:31:21.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:21 smithi132 ceph-mon[35283]: osdmap e145: 12 total, 12 up, 12 in 2024-05-27T07:31:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:21 smithi043 ceph-mon[31531]: pgmap v585: 129 pgs: 129 active+clean; 629 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-05-27T07:31:21.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:21 smithi043 ceph-mon[31531]: osdmap e145: 12 total, 12 up, 12 in 2024-05-27T07:31:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:21 smithi179 ceph-mon[35888]: pgmap v585: 129 pgs: 129 active+clean; 629 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-05-27T07:31:21.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:21 smithi179 ceph-mon[35888]: osdmap e145: 12 total, 12 up, 12 in 2024-05-27T07:31:22.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 22451 5.62 MB/sec execute 83 sec latency 221.482 ms 2024-05-27T07:31:23.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 22576 5.66 MB/sec execute 84 sec latency 362.994 ms 2024-05-27T07:31:23.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:23 smithi132 ceph-mon[35283]: pgmap v587: 129 pgs: 129 active+clean; 644 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.9 MiB/s wr, 194 op/s 2024-05-27T07:31:23.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:23 smithi043 ceph-mon[31531]: pgmap v587: 129 pgs: 129 active+clean; 644 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.9 MiB/s wr, 194 op/s 2024-05-27T07:31:23.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:23 smithi179 ceph-mon[35888]: pgmap v587: 129 pgs: 129 active+clean; 644 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.9 MiB/s wr, 194 op/s 2024-05-27T07:31:24.782 INFO:tasks.workunit.client.0.smithi043.stdout: 10 22709 5.69 MB/sec execute 85 sec latency 85.092 ms 2024-05-27T07:31:25.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 22861 5.76 MB/sec execute 86 sec latency 66.283 ms 2024-05-27T07:31:25.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:25 smithi132 ceph-mon[35283]: pgmap v588: 129 pgs: 129 active+clean; 644 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.8 MiB/s wr, 190 op/s 2024-05-27T07:31:25.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:25 smithi043 ceph-mon[31531]: pgmap v588: 129 pgs: 129 active+clean; 644 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.8 MiB/s wr, 190 op/s 2024-05-27T07:31:25.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:25 smithi179 ceph-mon[35888]: pgmap v588: 129 pgs: 129 active+clean; 644 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.8 MiB/s wr, 190 op/s 2024-05-27T07:31:26.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 22959 5.74 MB/sec execute 87 sec latency 73.475 ms 2024-05-27T07:31:27.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 23061 5.75 MB/sec execute 88 sec latency 65.349 ms 2024-05-27T07:31:27.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:27 smithi132 ceph-mon[35283]: pgmap v589: 129 pgs: 129 active+clean; 655 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-27T07:31:27.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:27 smithi043 ceph-mon[31531]: pgmap v589: 129 pgs: 129 active+clean; 655 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-27T07:31:27.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:27 smithi179 ceph-mon[35888]: pgmap v589: 129 pgs: 129 active+clean; 655 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-27T07:31:28.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 23227 5.88 MB/sec execute 89 sec latency 71.580 ms 2024-05-27T07:31:29.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 23366 5.88 MB/sec execute 90 sec latency 79.854 ms 2024-05-27T07:31:29.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:29 smithi132 ceph-mon[35283]: pgmap v590: 129 pgs: 129 active+clean; 655 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-27T07:31:29.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:29 smithi043 ceph-mon[31531]: pgmap v590: 129 pgs: 129 active+clean; 655 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-27T07:31:29.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:29 smithi179 ceph-mon[35888]: pgmap v590: 129 pgs: 129 active+clean; 655 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-27T07:31:30.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 23501 5.89 MB/sec execute 91 sec latency 106.039 ms 2024-05-27T07:31:31.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 23581 5.85 MB/sec execute 92 sec latency 277.470 ms 2024-05-27T07:31:31.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:31 smithi132 ceph-mon[35283]: pgmap v591: 129 pgs: 129 active+clean; 655 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-27T07:31:31.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:31 smithi043 ceph-mon[31531]: pgmap v591: 129 pgs: 129 active+clean; 655 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-27T07:31:31.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:31 smithi179 ceph-mon[35888]: pgmap v591: 129 pgs: 129 active+clean; 655 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 188 op/s 2024-05-27T07:31:32.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 23666 5.79 MB/sec execute 93 sec latency 82.127 ms 2024-05-27T07:31:33.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 23735 5.80 MB/sec execute 94 sec latency 447.324 ms 2024-05-27T07:31:33.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:33 smithi132 ceph-mon[35283]: pgmap v592: 129 pgs: 129 active+clean; 668 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 4.2 MiB/s wr, 177 op/s 2024-05-27T07:31:33.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:33 smithi043 ceph-mon[31531]: pgmap v592: 129 pgs: 129 active+clean; 668 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 4.2 MiB/s wr, 177 op/s 2024-05-27T07:31:33.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:33 smithi179 ceph-mon[35888]: pgmap v592: 129 pgs: 129 active+clean; 668 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 4.2 MiB/s wr, 177 op/s 2024-05-27T07:31:34.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 23841 5.82 MB/sec execute 95 sec latency 451.603 ms 2024-05-27T07:31:35.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 23960 5.81 MB/sec execute 96 sec latency 73.083 ms 2024-05-27T07:31:35.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:35 smithi132 ceph-mon[35283]: pgmap v593: 129 pgs: 129 active+clean; 668 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 3.9 MiB/s wr, 161 op/s 2024-05-27T07:31:35.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:35 smithi043 ceph-mon[31531]: pgmap v593: 129 pgs: 129 active+clean; 668 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 3.9 MiB/s wr, 161 op/s 2024-05-27T07:31:35.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:35 smithi179 ceph-mon[35888]: pgmap v593: 129 pgs: 129 active+clean; 668 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 3.9 MiB/s wr, 161 op/s 2024-05-27T07:31:36.783 INFO:tasks.workunit.client.0.smithi043.stdout: 10 24064 5.77 MB/sec execute 97 sec latency 78.064 ms 2024-05-27T07:31:37.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 24170 5.77 MB/sec execute 98 sec latency 80.227 ms 2024-05-27T07:31:37.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:37 smithi132 ceph-mon[35283]: pgmap v594: 129 pgs: 129 active+clean; 669 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 5.2 MiB/s wr, 237 op/s 2024-05-27T07:31:37.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:37 smithi043 ceph-mon[31531]: pgmap v594: 129 pgs: 129 active+clean; 669 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 5.2 MiB/s wr, 237 op/s 2024-05-27T07:31:37.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:37 smithi179 ceph-mon[35888]: pgmap v594: 129 pgs: 129 active+clean; 669 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 5.2 MiB/s wr, 237 op/s 2024-05-27T07:31:38.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:31:38.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 24259 5.77 MB/sec execute 99 sec latency 230.786 ms 2024-05-27T07:31:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:39.677 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:39 smithi043 ceph-mon[31531]: pgmap v595: 129 pgs: 129 active+clean; 669 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.5 MiB/s wr, 156 op/s 2024-05-27T07:31:39.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 24353 5.74 MB/sec execute 100 sec latency 414.846 ms 2024-05-27T07:31:39.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:39 smithi132 ceph-mon[35283]: pgmap v595: 129 pgs: 129 active+clean; 669 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.5 MiB/s wr, 156 op/s 2024-05-27T07:31:39.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:39 smithi179 ceph-mon[35888]: pgmap v595: 129 pgs: 129 active+clean; 669 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.5 MiB/s wr, 156 op/s 2024-05-27T07:31:40.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 24459 5.76 MB/sec execute 101 sec latency 272.655 ms 2024-05-27T07:31:41.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:41 smithi043 ceph-mon[31531]: pgmap v596: 129 pgs: 129 active+clean; 669 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.5 MiB/s wr, 156 op/s 2024-05-27T07:31:41.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 24594 5.79 MB/sec execute 102 sec latency 85.610 ms 2024-05-27T07:31:41.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:41 smithi179 ceph-mon[35888]: pgmap v596: 129 pgs: 129 active+clean; 669 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.5 MiB/s wr, 156 op/s 2024-05-27T07:31:42.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:41 smithi132 ceph-mon[35283]: pgmap v596: 129 pgs: 129 active+clean; 669 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.5 MiB/s wr, 156 op/s 2024-05-27T07:31:42.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.132+0000 7f12ad400700 1 -- 172.21.15.43:0/456423933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12a00965d0 msgr2=0x7f12a00969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.133+0000 7f12ad400700 1 --2- 172.21.15.43:0/456423933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12a00965d0 0x7f12a00969e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1294009990 tx=0x7f1294009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.133+0000 7f12ad400700 1 -- 172.21.15.43:0/456423933 shutdown_connections 2024-05-27T07:31:42.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.133+0000 7f12ad400700 1 --2- 172.21.15.43:0/456423933 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12a0096f20 0x7f12a00a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.133+0000 7f12ad400700 1 --2- 172.21.15.43:0/456423933 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12a009c5f0 0x7f12a009ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.133+0000 7f12ad400700 1 --2- 172.21.15.43:0/456423933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12a00965d0 0x7f12a00969e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.133+0000 7f12ad400700 1 -- 172.21.15.43:0/456423933 >> 172.21.15.43:0/456423933 conn(0x7f12a000d280 msgr2=0x7f12a000d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.133+0000 7f12ad400700 1 -- 172.21.15.43:0/456423933 shutdown_connections 2024-05-27T07:31:42.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.133+0000 7f12ad400700 1 -- 172.21.15.43:0/456423933 wait complete. 2024-05-27T07:31:42.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.135+0000 7f12ad400700 1 Processor -- start 2024-05-27T07:31:42.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.135+0000 7f12ad400700 1 -- start start 2024-05-27T07:31:42.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.136+0000 7f12ad400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12a00965d0 0x7f12a00a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.137+0000 7f12ad400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12a0096f20 0x7f12a00a3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.137+0000 7f12a7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12a00965d0 0x7f12a00a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.137+0000 7f12a7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12a00965d0 0x7f12a00a2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44698/0 (socket says 172.21.15.43:44698) 2024-05-27T07:31:42.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.137+0000 7f12ad400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12a009c5f0 0x7f12a00a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.138+0000 7f12ad400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12a0008220 con 0x7f12a00965d0 2024-05-27T07:31:42.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.138+0000 7f12ad400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f12a00080a0 con 0x7f12a0096f20 2024-05-27T07:31:42.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.138+0000 7f12ad400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f12a00083a0 con 0x7f12a009c5f0 2024-05-27T07:31:42.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.138+0000 7f12a7e00700 1 -- 172.21.15.43:0/2034167237 learned_addr learned my addr 172.21.15.43:0/2034167237 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:42.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.138+0000 7f12a7400700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12a0096f20 0x7f12a00a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.139+0000 7f12aca00700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12a009c5f0 0x7f12a00a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.139+0000 7f12a7400700 1 -- 172.21.15.43:0/2034167237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12a009c5f0 msgr2=0x7f12a00a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.139+0000 7f12a7400700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12a009c5f0 0x7f12a00a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.139+0000 7f12a7400700 1 -- 172.21.15.43:0/2034167237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12a00965d0 msgr2=0x7f12a00a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.139+0000 7f12a7400700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12a00965d0 0x7f12a00a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.139+0000 7f12a7400700 1 -- 172.21.15.43:0/2034167237 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a00a0990 con 0x7f12a0096f20 2024-05-27T07:31:42.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.140+0000 7f12a7400700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12a0096f20 0x7f12a00a3410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1294000c00 tx=0x7f12940057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.140+0000 7f12a4c00700 1 -- 172.21.15.43:0/2034167237 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1294026070 con 0x7f12a0096f20 2024-05-27T07:31:42.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.140+0000 7f12a4c00700 1 -- 172.21.15.43:0/2034167237 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1294003ef0 con 0x7f12a0096f20 2024-05-27T07:31:42.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.141+0000 7f12ad400700 1 -- 172.21.15.43:0/2034167237 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a00a0c20 con 0x7f12a0096f20 2024-05-27T07:31:42.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.141+0000 7f12ad400700 1 -- 172.21.15.43:0/2034167237 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a0140020 con 0x7f12a0096f20 2024-05-27T07:31:42.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.141+0000 7f12a4c00700 1 -- 172.21.15.43:0/2034167237 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129400fc00 con 0x7f12a0096f20 2024-05-27T07:31:42.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.146+0000 7f12a4c00700 1 -- 172.21.15.43:0/2034167237 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1294029070 con 0x7f12a0096f20 2024-05-27T07:31:42.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.147+0000 7f12bd542b80 1 Processor -- start 2024-05-27T07:31:42.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.148+0000 7f12bd542b80 1 -- start start 2024-05-27T07:31:42.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.149+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee08f2ce40 0x55ee090aae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.149+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.149+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee090ab7b0 0x55ee090b8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.149+0000 7f12bd542b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ee08fd9e40 con 0x55ee090ab3a0 2024-05-27T07:31:42.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.149+0000 7f12bd542b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ee08fc99e0 con 0x55ee08f2ce40 2024-05-27T07:31:42.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.149+0000 7f12bd542b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ee08fcda20 con 0x55ee090ab7b0 2024-05-27T07:31:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.149+0000 7f12a4c00700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f12900b2b20 0x7f12900b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.150+0000 7f12a4c00700 1 -- 172.21.15.43:0/2034167237 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7f12940e3250 con 0x7f12a0096f20 2024-05-27T07:31:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.150+0000 7f12aca00700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f12900b2b20 0x7f12900b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.150+0000 7f12a7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.150+0000 7f12a7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44708/0 (socket says 172.21.15.43:44708) 2024-05-27T07:31:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.150+0000 7f12a7e00700 1 -- 172.21.15.43:0/1020129062 learned_addr learned my addr 172.21.15.43:0/1020129062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:42.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.151+0000 7f12a7400700 1 --2- 172.21.15.43:0/1020129062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee090ab7b0 0x55ee090b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.151+0000 7f12a7e00700 1 -- 172.21.15.43:0/1020129062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee090ab7b0 msgr2=0x55ee090b8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.151+0000 7f12a7e00700 1 --2- 172.21.15.43:0/1020129062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee090ab7b0 0x55ee090b8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.151+0000 7f12a7e00700 1 -- 172.21.15.43:0/1020129062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee08f2ce40 msgr2=0x55ee090aae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.151+0000 7f12a7e00700 1 --2- 172.21.15.43:0/1020129062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee08f2ce40 0x55ee090aae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.151+0000 7f12a7e00700 1 -- 172.21.15.43:0/1020129062 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee08f92420 con 0x55ee090ab3a0 2024-05-27T07:31:42.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.151+0000 7f12a7e00700 1 --2- 172.21.15.43:0/1020129062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5b10 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f129c009b50 tx=0x7f129c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72292c0a11abcc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.152+0000 7f1299600700 1 -- 172.21.15.43:0/1020129062 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c017070 con 0x55ee090ab3a0 2024-05-27T07:31:42.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.152+0000 7f1299600700 1 -- 172.21.15.43:0/1020129062 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f129c00ed60 con 0x55ee090ab3a0 2024-05-27T07:31:42.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.152+0000 7f1299600700 1 -- 172.21.15.43:0/1020129062 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c013e30 con 0x55ee090ab3a0 2024-05-27T07:31:42.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.154+0000 7f12aca00700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f12900b2b20 0x7f12900b5030 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f12a8051580 tx=0x7f12a806d060 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.154+0000 7f12bd542b80 1 -- 172.21.15.43:0/1020129062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 msgr2=0x55ee090b5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.154+0000 7f12bd542b80 1 --2- 172.21.15.43:0/1020129062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5b10 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f129c009b50 tx=0x7f129c009e60 comp rx=0 tx=0).stop 2024-05-27T07:31:42.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.154+0000 7f12bd542b80 1 -- 172.21.15.43:0/1020129062 shutdown_connections 2024-05-27T07:31:42.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.154+0000 7f12bd542b80 1 --2- 172.21.15.43:0/1020129062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee090ab7b0 0x55ee090b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.154+0000 7f12bd542b80 1 --2- 172.21.15.43:0/1020129062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee08f2ce40 0x55ee090aae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.154+0000 7f12bd542b80 1 --2- 172.21.15.43:0/1020129062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5b10 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.154+0000 7f12bd542b80 1 -- 172.21.15.43:0/1020129062 >> 172.21.15.43:0/1020129062 conn(0x55ee08f8cfc0 msgr2=0x55ee090a6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.155+0000 7f12bd542b80 1 -- 172.21.15.43:0/1020129062 shutdown_connections 2024-05-27T07:31:42.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.155+0000 7f12bd542b80 1 -- 172.21.15.43:0/1020129062 wait complete. 2024-05-27T07:31:42.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.156+0000 7f12bd542b80 1 -- start start 2024-05-27T07:31:42.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.156+0000 7f12bd542b80 1 Processor -- start 2024-05-27T07:31:42.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.156+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee08f2ce40 0x55ee090b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.156+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.156+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee090ab7b0 0x55ee090b5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.156+0000 7f12bd542b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ee08ecffa0 con 0x55ee090ab3a0 2024-05-27T07:31:42.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.156+0000 7f12bd542b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ee08f6a760 con 0x55ee08f2ce40 2024-05-27T07:31:42.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.156+0000 7f12bd542b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ee090b8a20 con 0x55ee090ab7b0 2024-05-27T07:31:42.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.157+0000 7f12bd542b80 10 client.0 _ll_register_callbacks cb 0x55ee08f73c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:42.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.157+0000 7f12a7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.157+0000 7f12a7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44714/0 (socket says 172.21.15.43:44714) 2024-05-27T07:31:42.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.157+0000 7f12a7400700 1 -- 172.21.15.43:0/3084295127 learned_addr learned my addr 172.21.15.43:0/3084295127 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:42.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.158+0000 7f12a7400700 1 -- 172.21.15.43:0/3084295127 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee090ab7b0 msgr2=0x55ee090b5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.158+0000 7f12a7400700 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee090ab7b0 0x55ee090b5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.158+0000 7f12a7400700 1 -- 172.21.15.43:0/3084295127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee08f2ce40 msgr2=0x55ee090b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.158+0000 7f12a7400700 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee08f2ce40 0x55ee090b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.158+0000 7f12a7400700 1 -- 172.21.15.43:0/3084295127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee08f198d0 con 0x55ee090ab3a0 2024-05-27T07:31:42.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.158+0000 7f12a7400700 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5a20 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f12940a6210 tx=0x7f129400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef0c45ea616c872 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.159+0000 7f128e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:31:42.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.159+0000 7f128e000700 1 -- 172.21.15.43:0/3084295127 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12940aa190 con 0x55ee090ab3a0 2024-05-27T07:31:42.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.159+0000 7f128e000700 1 -- 172.21.15.43:0/3084295127 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7f12940aa600 con 0x55ee090ab3a0 2024-05-27T07:31:42.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.159+0000 7f12bd542b80 1 -- 172.21.15.43:0/3084295127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f129c00ed60 con 0x55ee090ab3a0 2024-05-27T07:31:42.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.160+0000 7f128e000700 1 -- 172.21.15.43:0/3084295127 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) ==== 24+0+0 (secure 0 0 0) 0x7f12940ae540 con 0x55ee090ab3a0 2024-05-27T07:31:42.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.160+0000 7f12bd542b80 10 client.17862 fetch_fsmap learned FSMap version 134 2024-05-27T07:31:42.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.160+0000 7f12bd542b80 1 -- 172.21.15.43:0/3084295127 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=4+}) -- 0x55ee08f9a060 con 0x55ee090ab3a0 2024-05-27T07:31:42.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.160+0000 7f128e000700 1 -- 172.21.15.43:0/3084295127 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f129c00ed60 con 0x55ee090ab3a0 2024-05-27T07:31:42.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.161+0000 7f128e000700 1 -- 172.21.15.43:0/3084295127 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 134) ==== 4199+0+0 (secure 0 0 0) 0x7f12940a93f0 con 0x55ee090ab3a0 2024-05-27T07:31:42.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.161+0000 7f12bd542b80 10 client.17862 fetch_fsmap finished waiting for FSMap version 134 2024-05-27T07:31:42.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.161+0000 7f12bd542b80 10 client.17862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:42.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.161+0000 7f12bd542b80 20 client.17862 populate_metadata read hostname 'smithi043' 2024-05-27T07:31:42.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.161+0000 7f12bd542b80 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ee090c5b30 0x55ee090c8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.161+0000 7f12bd542b80 4 client.17862 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:42.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.161+0000 7f12bd542b80 1 -- 172.21.15.43:0/3084295127 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee08fdbd90 con 0x55ee090c5b30 2024-05-27T07:31:42.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.162+0000 7f12a7e00700 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ee090c5b30 0x55ee090c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.163+0000 7f128e000700 10 client.17862 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:31:42.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.163+0000 7f12a7e00700 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ee090c5b30 0x55ee090c8020 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd32010c8701267e server_cookie=effaaa002376ec4b in_seq=0 out_seq=0 2024-05-27T07:31:42.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.165+0000 7f128e000700 1 -- 172.21.15.43:0/3084295127 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ee08fdbd90 con 0x55ee090c5b30 2024-05-27T07:31:42.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.165+0000 7f128e000700 10 client.17862 handle_command_reply: tid=0 2024-05-27T07:31:42.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.165+0000 7f12bd542b80 1 client.17862 shutdown 2024-05-27T07:31:42.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.165+0000 7f12bd542b80 1 -- 172.21.15.43:0/3084295127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 msgr2=0x55ee090b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.165+0000 7f12bd542b80 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5a20 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f12940a6210 tx=0x7f129400f690 comp rx=0 tx=0).stop 2024-05-27T07:31:42.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.166+0000 7f12bd542b80 1 -- 172.21.15.43:0/3084295127 shutdown_connections 2024-05-27T07:31:42.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.166+0000 7f12bd542b80 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee090ab7b0 0x55ee090b5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.166+0000 7f12bd542b80 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee08f2ce40 0x55ee090b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.166+0000 7f12bd542b80 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ee090c5b30 0x55ee090c8020 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.166+0000 7f128e000700 0 client.17862 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:31:42.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.166+0000 7f12bd542b80 1 --2- 172.21.15.43:0/3084295127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090b5a20 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.166+0000 7f12bd542b80 1 -- 172.21.15.43:0/3084295127 >> 172.21.15.43:0/3084295127 conn(0x55ee08f8cfc0 msgr2=0x55ee090a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.166+0000 7f12bd542b80 1 -- 172.21.15.43:0/3084295127 shutdown_connections 2024-05-27T07:31:42.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.166+0000 7f12bd542b80 1 -- 172.21.15.43:0/3084295127 wait complete. 2024-05-27T07:31:42.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.166+0000 7f12bd542b80 20 client.17862 trim_cache size 0 max 16384 2024-05-27T07:31:42.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.207+0000 7f12bd542b80 1 Processor -- start 2024-05-27T07:31:42.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.207+0000 7f12bd542b80 1 -- start start 2024-05-27T07:31:42.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.208+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee08f2ce40 0x55ee090b7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.208+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.208+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 0x55ee090aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.208+0000 7f12bd542b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12a0149280 con 0x55ee090ab3a0 2024-05-27T07:31:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.208+0000 7f12bd542b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f12a01476b0 con 0x55ee090ab7b0 2024-05-27T07:31:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.208+0000 7f12bd542b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f12a0148bb0 con 0x55ee08f2ce40 2024-05-27T07:31:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.208+0000 7f12aca00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 0x55ee090aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.208+0000 7f12aca00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 0x55ee090aceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:42534/0 (socket says 172.21.15.43:42534) 2024-05-27T07:31:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.209+0000 7f12aca00700 1 -- 172.21.15.43:0/196968869 learned_addr learned my addr 172.21.15.43:0/196968869 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.209+0000 7f12a7400700 1 --2- 172.21.15.43:0/196968869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.209+0000 7f12a7e00700 1 --2- 172.21.15.43:0/196968869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee08f2ce40 0x55ee090b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.209+0000 7f12aca00700 1 -- 172.21.15.43:0/196968869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee08f2ce40 msgr2=0x55ee090b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.209+0000 7f12aca00700 1 --2- 172.21.15.43:0/196968869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee08f2ce40 0x55ee090b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.209+0000 7f12aca00700 1 -- 172.21.15.43:0/196968869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 msgr2=0x55ee090c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.209+0000 7f12aca00700 1 --2- 172.21.15.43:0/196968869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090c1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.209+0000 7f12aca00700 1 -- 172.21.15.43:0/196968869 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee090b65a0 con 0x55ee090ab7b0 2024-05-27T07:31:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.209+0000 7f12a7400700 1 --2- 172.21.15.43:0/196968869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:31:42.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.210+0000 7f12aca00700 1 --2- 172.21.15.43:0/196968869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 0x55ee090aceb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f12a8067060 tx=0x7f12a8082000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22b6e30f9f0f37e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.211+0000 7f1298c00700 1 -- 172.21.15.43:0/196968869 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a80816a0 con 0x55ee090ab7b0 2024-05-27T07:31:42.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.211+0000 7f1298c00700 1 -- 172.21.15.43:0/196968869 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f12a8092070 con 0x55ee090ab7b0 2024-05-27T07:31:42.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.211+0000 7f1298c00700 1 -- 172.21.15.43:0/196968869 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a808db60 con 0x55ee090ab7b0 2024-05-27T07:31:42.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.213+0000 7f12bd542b80 1 -- 172.21.15.43:0/196968869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 msgr2=0x55ee090aceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.213+0000 7f12bd542b80 1 --2- 172.21.15.43:0/196968869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 0x55ee090aceb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f12a8067060 tx=0x7f12a8082000 comp rx=0 tx=0).stop 2024-05-27T07:31:42.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.213+0000 7f12bd542b80 1 -- 172.21.15.43:0/196968869 shutdown_connections 2024-05-27T07:31:42.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.213+0000 7f12bd542b80 1 --2- 172.21.15.43:0/196968869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 0x55ee090aceb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.213+0000 7f12bd542b80 1 --2- 172.21.15.43:0/196968869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee08f2ce40 0x55ee090b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.213+0000 7f12bd542b80 1 --2- 172.21.15.43:0/196968869 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.213+0000 7f12bd542b80 1 -- 172.21.15.43:0/196968869 >> 172.21.15.43:0/196968869 conn(0x55ee08f8cfc0 msgr2=0x55ee090ac7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.214+0000 7f12bd542b80 1 -- 172.21.15.43:0/196968869 shutdown_connections 2024-05-27T07:31:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.214+0000 7f12bd542b80 1 -- 172.21.15.43:0/196968869 wait complete. 2024-05-27T07:31:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.215+0000 7f12bd542b80 1 -- start start 2024-05-27T07:31:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.215+0000 7f12bd542b80 1 Processor -- start 2024-05-27T07:31:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.216+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee08f2ce40 0x55ee090ba1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.216+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090ba700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.216+0000 7f12bd542b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 0x55ee0912f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.216+0000 7f12bd542b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12a0148480 con 0x55ee090ab3a0 2024-05-27T07:31:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.216+0000 7f12bd542b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f12a0147db0 con 0x55ee090ab7b0 2024-05-27T07:31:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.216+0000 7f12bd542b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ee090c2150 con 0x55ee08f2ce40 2024-05-27T07:31:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.216+0000 7f12bd542b80 10 client.0 _ll_register_callbacks cb 0x55ee08f73c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.217+0000 7f12a7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090ba700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.217+0000 7f12a7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090ba700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44740/0 (socket says 172.21.15.43:44740) 2024-05-27T07:31:42.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.217+0000 7f12a7400700 1 -- 172.21.15.43:0/1933308886 learned_addr learned my addr 172.21.15.43:0/1933308886 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:42.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.218+0000 7f12a7400700 1 -- 172.21.15.43:0/1933308886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee08f2ce40 msgr2=0x55ee090ba1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.218+0000 7f12a7400700 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee08f2ce40 0x55ee090ba1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.218+0000 7f12a7400700 1 -- 172.21.15.43:0/1933308886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 msgr2=0x55ee0912f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.218+0000 7f12a7400700 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 0x55ee0912f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.218+0000 7f12a7400700 1 -- 172.21.15.43:0/1933308886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee0912fc30 con 0x55ee090ab3a0 2024-05-27T07:31:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.219+0000 7f12a7400700 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090ba700 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f1294006010 tx=0x7f12940a6050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be63c6471750b7a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.219+0000 7f128e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:31:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.220+0000 7f128e000700 1 -- 172.21.15.43:0/1933308886 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12940b2f60 con 0x55ee090ab3a0 2024-05-27T07:31:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.220+0000 7f128e000700 1 -- 172.21.15.43:0/1933308886 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7f12940a6780 con 0x55ee090ab3a0 2024-05-27T07:31:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.220+0000 7f12bd542b80 1 -- 172.21.15.43:0/1933308886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12a8092070 con 0x55ee090ab3a0 2024-05-27T07:31:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.221+0000 7f128e000700 1 -- 172.21.15.43:0/1933308886 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) ==== 24+0+0 (secure 0 0 0) 0x7f12940b1830 con 0x55ee090ab3a0 2024-05-27T07:31:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.221+0000 7f12bd542b80 10 client.17865 fetch_fsmap learned FSMap version 134 2024-05-27T07:31:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.221+0000 7f12bd542b80 1 -- 172.21.15.43:0/1933308886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=4+}) -- 0x55ee09130d60 con 0x55ee090ab3a0 2024-05-27T07:31:42.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.221+0000 7f128e000700 1 -- 172.21.15.43:0/1933308886 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f12a8092070 con 0x55ee090ab3a0 2024-05-27T07:31:42.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.221+0000 7f128e000700 1 -- 172.21.15.43:0/1933308886 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 134) ==== 4199+0+0 (secure 0 0 0) 0x7f12940a8e60 con 0x55ee090ab3a0 2024-05-27T07:31:42.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.221+0000 7f12bd542b80 10 client.17865 fetch_fsmap finished waiting for FSMap version 134 2024-05-27T07:31:42.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.221+0000 7f12bd542b80 10 client.17865 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:31:42.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.221+0000 7f12bd542b80 20 client.17865 populate_metadata read hostname 'smithi043' 2024-05-27T07:31:42.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.222+0000 7f12bd542b80 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ee090c5b30 0x55ee091332b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:42.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.222+0000 7f12bd542b80 4 client.17865 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:31:42.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.222+0000 7f12bd542b80 1 -- 172.21.15.43:0/1933308886 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ee090ac470 con 0x55ee090c5b30 2024-05-27T07:31:42.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.223+0000 7f12a7e00700 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ee090c5b30 0x55ee091332b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:42.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.224+0000 7f128e000700 10 client.17865 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:31:42.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.225+0000 7f12a7e00700 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ee090c5b30 0x55ee091332b0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2c012ccb527beed server_cookie=64f8ac9a7036a3fa in_seq=0 out_seq=0 2024-05-27T07:31:42.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.226+0000 7f128e000700 1 -- 172.21.15.43:0/1933308886 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ee090ac470 con 0x55ee090c5b30 2024-05-27T07:31:42.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.226+0000 7f128e000700 10 client.17865 handle_command_reply: tid=0 2024-05-27T07:31:42.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.226+0000 7f12bd542b80 1 client.17865 shutdown 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.226+0000 7f12bd542b80 1 -- 172.21.15.43:0/1933308886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 msgr2=0x55ee090ba700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.226+0000 7f12bd542b80 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090ba700 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f1294006010 tx=0x7f12940a6050 comp rx=0 tx=0).stop 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.227+0000 7f12bd542b80 1 -- 172.21.15.43:0/1933308886 shutdown_connections 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.227+0000 7f12bd542b80 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ee090ab7b0 0x55ee0912f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.227+0000 7f12bd542b80 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee08f2ce40 0x55ee090ba1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.227+0000 7f12bd542b80 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ee090c5b30 0x55ee091332b0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.227+0000 7f12bd542b80 1 --2- 172.21.15.43:0/1933308886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ee090ab3a0 0x55ee090ba700 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.227+0000 7f12bd542b80 1 -- 172.21.15.43:0/1933308886 >> 172.21.15.43:0/1933308886 conn(0x55ee08f8cfc0 msgr2=0x55ee090ad1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.227+0000 7f128e000700 0 client.17865 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.227+0000 7f12bd542b80 1 -- 172.21.15.43:0/1933308886 shutdown_connections 2024-05-27T07:31:42.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.227+0000 7f12bd542b80 1 -- 172.21.15.43:0/1933308886 wait complete. 2024-05-27T07:31:42.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.228+0000 7f12bd542b80 20 client.17865 trim_cache size 0 max 16384 2024-05-27T07:31:42.221 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:31:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:31:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:31:42.221 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:31:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.229+0000 7f12ad400700 1 -- 172.21.15.43:0/2034167237 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f12900b2b20 msgr2=0x7f12900b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.229+0000 7f12ad400700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f12900b2b20 0x7f12900b5030 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f12a8051580 tx=0x7f12a806d060 comp rx=0 tx=0).stop 2024-05-27T07:31:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.229+0000 7f12ad400700 1 -- 172.21.15.43:0/2034167237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12a0096f20 msgr2=0x7f12a00a3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.229+0000 7f12ad400700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12a0096f20 0x7f12a00a3410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1294000c00 tx=0x7f12940057c0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.230+0000 7f12ad400700 1 -- 172.21.15.43:0/2034167237 shutdown_connections 2024-05-27T07:31:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.230+0000 7f12ad400700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f12900b2b20 0x7f12900b5030 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.230+0000 7f12ad400700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12a009c5f0 0x7f12a00a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.230+0000 7f12ad400700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12a00965d0 0x7f12a00a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.230+0000 7f12ad400700 1 --2- 172.21.15.43:0/2034167237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f12a0096f20 0x7f12a00a3410 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:42.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.230+0000 7f12ad400700 1 -- 172.21.15.43:0/2034167237 >> 172.21.15.43:0/2034167237 conn(0x7f12a000d280 msgr2=0x7f12a0094240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:42.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.230+0000 7f12ad400700 1 -- 172.21.15.43:0/2034167237 shutdown_connections 2024-05-27T07:31:42.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:42.231+0000 7f12ad400700 1 -- 172.21.15.43:0/2034167237 wait complete. 2024-05-27T07:31:42.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 24691 5.75 MB/sec execute 103 sec latency 114.624 ms 2024-05-27T07:31:42.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:42 smithi043 ceph-mon[31531]: pgmap v597: 129 pgs: 129 active+clean; 673 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.0 MiB/s wr, 235 op/s 2024-05-27T07:31:42.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:42 smithi179 ceph-mon[35888]: pgmap v597: 129 pgs: 129 active+clean; 673 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.0 MiB/s wr, 235 op/s 2024-05-27T07:31:43.024 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:31:43.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:31:43.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:42 smithi132 ceph-mon[35283]: pgmap v597: 129 pgs: 129 active+clean; 673 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.0 MiB/s wr, 235 op/s 2024-05-27T07:31:43.721 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:43.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 24768 5.71 MB/sec execute 104 sec latency 129.479 ms 2024-05-27T07:31:44.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 24874 5.68 MB/sec execute 105 sec latency 98.687 ms 2024-05-27T07:31:45.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:45 smithi179 ceph-mon[35888]: pgmap v598: 129 pgs: 129 active+clean; 673 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.8 MiB/s wr, 154 op/s 2024-05-27T07:31:45.685 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:45 smithi043 ceph-mon[31531]: pgmap v598: 129 pgs: 129 active+clean; 673 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.8 MiB/s wr, 154 op/s 2024-05-27T07:31:45.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 24994 5.68 MB/sec execute 106 sec latency 92.379 ms 2024-05-27T07:31:45.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:45 smithi132 ceph-mon[35283]: pgmap v598: 129 pgs: 129 active+clean; 673 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.8 MiB/s wr, 154 op/s 2024-05-27T07:31:46.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.377+0000 7f8d78c00700 1 -- 172.21.15.43:0/1774276317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d74107de0 msgr2=0x7f8d741081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:46.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.377+0000 7f8d78c00700 1 --2- 172.21.15.43:0/1774276317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d74107de0 0x7f8d741081f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009990 tx=0x7f8d60009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.377+0000 7f8d78c00700 1 -- 172.21.15.43:0/1774276317 shutdown_connections 2024-05-27T07:31:46.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.377+0000 7f8d78c00700 1 --2- 172.21.15.43:0/1774276317 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d74108730 0x7f8d74110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.377+0000 7f8d78c00700 1 --2- 172.21.15.43:0/1774276317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d741034b0 0x7f8d741038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.377+0000 7f8d78c00700 1 --2- 172.21.15.43:0/1774276317 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d74107de0 0x7f8d741081f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.377+0000 7f8d78c00700 1 -- 172.21.15.43:0/1774276317 >> 172.21.15.43:0/1774276317 conn(0x7f8d740fedb0 msgr2=0x7f8d741011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:46.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.377+0000 7f8d78c00700 1 -- 172.21.15.43:0/1774276317 shutdown_connections 2024-05-27T07:31:46.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.377+0000 7f8d78c00700 1 -- 172.21.15.43:0/1774276317 wait complete. 2024-05-27T07:31:46.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.379+0000 7f8d78c00700 1 Processor -- start 2024-05-27T07:31:46.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.381+0000 7f8d78c00700 1 -- start start 2024-05-27T07:31:46.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.382+0000 7f8d78c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d741034b0 0x7f8d74110870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:46.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.382+0000 7f8d78c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d74107de0 0x7f8d74110db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:46.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.382+0000 7f8d78c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d74108730 0x7f8d741112f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:46.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.382+0000 7f8d78c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8d741143c0 con 0x7f8d74108730 2024-05-27T07:31:46.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.382+0000 7f8d78c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74114240 con 0x7f8d741034b0 2024-05-27T07:31:46.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.382+0000 7f8d78c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74114540 con 0x7f8d74107de0 2024-05-27T07:31:46.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.382+0000 7f8d71e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d741034b0 0x7f8d74110870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.382+0000 7f8d71e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d741034b0 0x7f8d74110870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:53480/0 (socket says 172.21.15.43:53480) 2024-05-27T07:31:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.382+0000 7f8d71e00700 1 -- 172.21.15.43:0/3513648280 learned_addr learned my addr 172.21.15.43:0/3513648280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.383+0000 7f8d72800700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d74108730 0x7f8d741112f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.383+0000 7f8d71400700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d74107de0 0x7f8d74110db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.383+0000 7f8d72800700 1 -- 172.21.15.43:0/3513648280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d74107de0 msgr2=0x7f8d74110db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.383+0000 7f8d72800700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d74107de0 0x7f8d74110db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.383+0000 7f8d72800700 1 -- 172.21.15.43:0/3513648280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d741034b0 msgr2=0x7f8d74110870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.383+0000 7f8d72800700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d741034b0 0x7f8d74110870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.383+0000 7f8d72800700 1 -- 172.21.15.43:0/3513648280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d741b18b0 con 0x7f8d74108730 2024-05-27T07:31:46.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.384+0000 7f8d72800700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d74108730 0x7f8d741112f0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00efc0 tx=0x7f8d6c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:46.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.384+0000 7f8d6aa00700 1 -- 172.21.15.43:0/3513648280 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c0092e0 con 0x7f8d74108730 2024-05-27T07:31:46.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.384+0000 7f8d6aa00700 1 -- 172.21.15.43:0/3513648280 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8d6c004bb0 con 0x7f8d74108730 2024-05-27T07:31:46.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.384+0000 7f8d78c00700 1 -- 172.21.15.43:0/3513648280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d741b1b40 con 0x7f8d74108730 2024-05-27T07:31:46.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.384+0000 7f8d6aa00700 1 -- 172.21.15.43:0/3513648280 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c004020 con 0x7f8d74108730 2024-05-27T07:31:46.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.385+0000 7f8d78c00700 1 -- 172.21.15.43:0/3513648280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d741b20d0 con 0x7f8d74108730 2024-05-27T07:31:46.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.387+0000 7f8d5fe00700 1 -- 172.21.15.43:0/3513648280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d74104960 con 0x7f8d74108730 2024-05-27T07:31:46.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.391+0000 7f8d6aa00700 1 -- 172.21.15.43:0/3513648280 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f8d6c01b030 con 0x7f8d74108730 2024-05-27T07:31:46.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.393+0000 7f8d6aa00700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8d580b2a70 0x7f8d580b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:46.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.393+0000 7f8d71e00700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8d580b2a70 0x7f8d580b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:46.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.393+0000 7f8d6aa00700 1 -- 172.21.15.43:0/3513648280 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7f8d6c0e2400 con 0x7f8d74108730 2024-05-27T07:31:46.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.395+0000 7f8d71e00700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8d580b2a70 0x7f8d580b4f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8d741119b0 tx=0x7f8d6400b500 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:46.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.403+0000 7f8d6aa00700 1 -- 172.21.15.43:0/3513648280 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8d6c022020 con 0x7f8d74108730 2024-05-27T07:31:46.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 25108 5.65 MB/sec execute 107 sec latency 93.106 ms 2024-05-27T07:31:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.915+0000 7f8d5fe00700 1 -- 172.21.15.43:0/3513648280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d74111da0 con 0x7f8d74108730 2024-05-27T07:31:46.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.916+0000 7f8d6aa00700 1 -- 172.21.15.43:0/3513648280 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+6124 (secure 0 0 0) 0x7f8d6c010040 con 0x7f8d74108730 2024-05-27T07:31:46.909 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:31:46.909 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":134,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:31:06.582926+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34484,34448,34424,24484,34403]},"id":1}]} 2024-05-27T07:31:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.920+0000 7f8d5fe00700 1 -- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8d580b2a70 msgr2=0x7f8d580b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.920+0000 7f8d5fe00700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8d580b2a70 0x7f8d580b4f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8d741119b0 tx=0x7f8d6400b500 comp rx=0 tx=0).stop 2024-05-27T07:31:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.920+0000 7f8d5fe00700 1 -- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d74108730 msgr2=0x7f8d741112f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.920+0000 7f8d5fe00700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d74108730 0x7f8d741112f0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00efc0 tx=0x7f8d6c00c3a0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.921+0000 7f8d5fe00700 1 -- 172.21.15.43:0/3513648280 shutdown_connections 2024-05-27T07:31:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.921+0000 7f8d5fe00700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f8d580b2a70 0x7f8d580b4f80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.921+0000 7f8d5fe00700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d74108730 0x7f8d741112f0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.921+0000 7f8d5fe00700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d741034b0 0x7f8d74110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.921+0000 7f8d5fe00700 1 --2- 172.21.15.43:0/3513648280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d74107de0 0x7f8d74110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.921+0000 7f8d5fe00700 1 -- 172.21.15.43:0/3513648280 >> 172.21.15.43:0/3513648280 conn(0x7f8d740fedb0 msgr2=0x7f8d74101da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.921+0000 7f8d5fe00700 1 -- 172.21.15.43:0/3513648280 shutdown_connections 2024-05-27T07:31:46.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:46.921+0000 7f8d5fe00700 1 -- 172.21.15.43:0/3513648280 wait complete. 2024-05-27T07:31:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 134 2024-05-27T07:31:47.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:47 smithi043 ceph-mon[31531]: pgmap v599: 129 pgs: 129 active+clean; 672 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 237 op/s 2024-05-27T07:31:47.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:47 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3513648280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:31:47.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:47 smithi179 ceph-mon[35888]: pgmap v599: 129 pgs: 129 active+clean; 672 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 237 op/s 2024-05-27T07:31:47.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:47 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3513648280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:31:47.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 25253 5.71 MB/sec execute 108 sec latency 90.849 ms 2024-05-27T07:31:47.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:47 smithi132 ceph-mon[35283]: pgmap v599: 129 pgs: 129 active+clean; 672 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 237 op/s 2024-05-27T07:31:47.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:47 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3513648280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:31:47.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:31:48.402 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:48 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:31:48.549 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:48.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:48 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:31:48.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 25344 5.68 MB/sec execute 109 sec latency 208.915 ms 2024-05-27T07:31:48.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:48 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:31:49.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:49 smithi043 ceph-mon[31531]: pgmap v600: 129 pgs: 129 active+clean; 672 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 2.6 MiB/s wr, 161 op/s 2024-05-27T07:31:49.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:49 smithi179 ceph-mon[35888]: pgmap v600: 129 pgs: 129 active+clean; 672 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 2.6 MiB/s wr, 161 op/s 2024-05-27T07:31:49.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 25479 5.72 MB/sec execute 110 sec latency 84.399 ms 2024-05-27T07:31:49.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:49 smithi132 ceph-mon[35283]: pgmap v600: 129 pgs: 129 active+clean; 672 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 2.6 MiB/s wr, 161 op/s 2024-05-27T07:31:50.784 INFO:tasks.workunit.client.0.smithi043.stdout: 10 25566 5.68 MB/sec execute 111 sec latency 96.074 ms 2024-05-27T07:31:51.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.225+0000 7f54da400700 1 -- 172.21.15.43:0/1282082964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d4075ad0 msgr2=0x7f54d4075ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:51.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.225+0000 7f54da400700 1 --2- 172.21.15.43:0/1282082964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d4075ad0 0x7f54d4075ee0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f54c4009990 tx=0x7f54c4009ca0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.225+0000 7f54da400700 1 -- 172.21.15.43:0/1282082964 shutdown_connections 2024-05-27T07:31:51.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.225+0000 7f54da400700 1 --2- 172.21.15.43:0/1282082964 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54d4110c10 0x7f54d4113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.225+0000 7f54da400700 1 --2- 172.21.15.43:0/1282082964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d4075ad0 0x7f54d4075ee0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.225+0000 7f54da400700 1 --2- 172.21.15.43:0/1282082964 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54d4076420 0x7f54d41106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.225+0000 7f54da400700 1 -- 172.21.15.43:0/1282082964 >> 172.21.15.43:0/1282082964 conn(0x7f54d4100030 msgr2=0x7f54d4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:51.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.225+0000 7f54da400700 1 -- 172.21.15.43:0/1282082964 shutdown_connections 2024-05-27T07:31:51.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.226+0000 7f54da400700 1 -- 172.21.15.43:0/1282082964 wait complete. 2024-05-27T07:31:51.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.227+0000 7f54da400700 1 Processor -- start 2024-05-27T07:31:51.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.228+0000 7f54da400700 1 -- start start 2024-05-27T07:31:51.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.228+0000 7f54da400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d4075ad0 0x7f54d4104330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.228+0000 7f54da400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54d4076420 0x7f54d4104870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.228+0000 7f54da400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54d4110c10 0x7f54d4104db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.228+0000 7f54da400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4116590 con 0x7f54d4075ad0 2024-05-27T07:31:51.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.228+0000 7f54da400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4116410 con 0x7f54d4076420 2024-05-27T07:31:51.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.228+0000 7f54da400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4116710 con 0x7f54d4110c10 2024-05-27T07:31:51.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.229+0000 7f54d2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54d4076420 0x7f54d4104870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.229+0000 7f54d3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d4075ad0 0x7f54d4104330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.229+0000 7f54d3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d4075ad0 0x7f54d4104330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58180/0 (socket says 172.21.15.43:58180) 2024-05-27T07:31:51.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.229+0000 7f54d3400700 1 -- 172.21.15.43:0/3298133808 learned_addr learned my addr 172.21.15.43:0/3298133808 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:51.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.229+0000 7f54d2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54d4076420 0x7f54d4104870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:53498/0 (socket says 172.21.15.43:53498) 2024-05-27T07:31:51.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.229+0000 7f54d3e00700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54d4110c10 0x7f54d4104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.230+0000 7f54d3400700 1 -- 172.21.15.43:0/3298133808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54d4110c10 msgr2=0x7f54d4104db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:51.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.230+0000 7f54d3400700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54d4110c10 0x7f54d4104db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.230+0000 7f54d3400700 1 -- 172.21.15.43:0/3298133808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54d4076420 msgr2=0x7f54d4104870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:51.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.230+0000 7f54d3400700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54d4076420 0x7f54d4104870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.230+0000 7f54d3400700 1 -- 172.21.15.43:0/3298133808 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d4074530 con 0x7f54d4075ad0 2024-05-27T07:31:51.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.231+0000 7f54d3400700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d4075ad0 0x7f54d4104330 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f54c400b650 tx=0x7f54c400bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.231+0000 7f54cbe00700 1 -- 172.21.15.43:0/3298133808 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4027070 con 0x7f54d4075ad0 2024-05-27T07:31:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.231+0000 7f54cbe00700 1 -- 172.21.15.43:0/3298133808 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f54c4022470 con 0x7f54d4075ad0 2024-05-27T07:31:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.231+0000 7f54da400700 1 -- 172.21.15.43:0/3298133808 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d40747c0 con 0x7f54d4075ad0 2024-05-27T07:31:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.232+0000 7f54cbe00700 1 -- 172.21.15.43:0/3298133808 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4017650 con 0x7f54d4075ad0 2024-05-27T07:31:51.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.233+0000 7f54da400700 1 -- 172.21.15.43:0/3298133808 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d4074ca0 con 0x7f54d4075ad0 2024-05-27T07:31:51.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.238+0000 7f54cbe00700 1 -- 172.21.15.43:0/3298133808 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f54c4009d10 con 0x7f54d4075ad0 2024-05-27T07:31:51.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.239+0000 7f54e9024b80 1 Processor -- start 2024-05-27T07:31:51.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.239+0000 7f54e9024b80 1 -- start start 2024-05-27T07:31:51.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.240+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf457e40 0x55efaf5dc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.240+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 0x55efaf5d6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.240+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf5d6db0 0x55efaf5d71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.240+0000 7f54e9024b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55efaf504e40 con 0x55efaf5d6db0 2024-05-27T07:31:51.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.240+0000 7f54e9024b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55efaf4f49e0 con 0x55efaf5dcd30 2024-05-27T07:31:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.240+0000 7f54e9024b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55efaf4f8a20 con 0x55efaf457e40 2024-05-27T07:31:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.240+0000 7f54cbe00700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f54b80b2b40 0x7f54b80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.241+0000 7f54d3e00700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f54b80b2b40 0x7f54b80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.241+0000 7f54d3e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf457e40 0x55efaf5dc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.241+0000 7f54d3e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf457e40 0x55efaf5dc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:45304/0 (socket says 172.21.15.43:45304) 2024-05-27T07:31:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.241+0000 7f54d3e00700 1 -- 172.21.15.43:0/1219921195 learned_addr learned my addr 172.21.15.43:0/1219921195 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.241+0000 7f54d3400700 1 --2- 172.21.15.43:0/1219921195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 0x55efaf5d6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.242+0000 7f54cbe00700 1 -- 172.21.15.43:0/3298133808 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7f54c40e3550 con 0x7f54d4075ad0 2024-05-27T07:31:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.242+0000 7f54d2a00700 1 --2- 172.21.15.43:0/1219921195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf5d6db0 0x55efaf5d71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.243+0000 7f54d3e00700 1 -- 172.21.15.43:0/1219921195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 msgr2=0x55efaf5d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.243+0000 7f54d3e00700 1 --2- 172.21.15.43:0/1219921195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 0x55efaf5d6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.243+0000 7f54d3e00700 1 -- 172.21.15.43:0/1219921195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf5d6db0 msgr2=0x55efaf5d71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.243+0000 7f54d3e00700 1 --2- 172.21.15.43:0/1219921195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf5d6db0 0x55efaf5d71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.244+0000 7f54d3e00700 1 -- 172.21.15.43:0/1219921195 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efaf4bd420 con 0x55efaf457e40 2024-05-27T07:31:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.244+0000 7f54d2a00700 1 --2- 172.21.15.43:0/1219921195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf5d6db0 0x55efaf5d71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:31:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.244+0000 7f54d3e00700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f54b80b2b40 0x7f54b80b5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f54cc007890 tx=0x7f54cc009cb0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.244+0000 7f54d3e00700 1 --2- 172.21.15.43:0/1219921195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf457e40 0x55efaf5dc7f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0172f0 tx=0x7f54cc01d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88960a7dc2bf986c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.245+0000 7f54c8c00700 1 -- 172.21.15.43:0/1219921195 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc017af0 con 0x55efaf457e40 2024-05-27T07:31:51.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.245+0000 7f54c8c00700 1 -- 172.21.15.43:0/1219921195 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f54cc017c90 con 0x55efaf457e40 2024-05-27T07:31:51.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.245+0000 7f54c8c00700 1 -- 172.21.15.43:0/1219921195 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc022650 con 0x55efaf457e40 2024-05-27T07:31:51.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.248+0000 7f54e9024b80 1 -- 172.21.15.43:0/1219921195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf457e40 msgr2=0x55efaf5dc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:51.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.248+0000 7f54e9024b80 1 --2- 172.21.15.43:0/1219921195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf457e40 0x55efaf5dc7f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0172f0 tx=0x7f54cc01d000 comp rx=0 tx=0).stop 2024-05-27T07:31:51.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.248+0000 7f54e9024b80 1 -- 172.21.15.43:0/1219921195 shutdown_connections 2024-05-27T07:31:51.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.248+0000 7f54e9024b80 1 --2- 172.21.15.43:0/1219921195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf5d6db0 0x55efaf5d71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.248+0000 7f54e9024b80 1 --2- 172.21.15.43:0/1219921195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf457e40 0x55efaf5dc7f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.248+0000 7f54e9024b80 1 --2- 172.21.15.43:0/1219921195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 0x55efaf5d6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.248+0000 7f54e9024b80 1 -- 172.21.15.43:0/1219921195 >> 172.21.15.43:0/1219921195 conn(0x55efaf4b7fc0 msgr2=0x55efaf5d18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:51.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.249+0000 7f54e9024b80 1 -- 172.21.15.43:0/1219921195 shutdown_connections 2024-05-27T07:31:51.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.249+0000 7f54e9024b80 1 -- 172.21.15.43:0/1219921195 wait complete. 2024-05-27T07:31:51.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.251+0000 7f54e9024b80 1 -- start start 2024-05-27T07:31:51.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.251+0000 7f54e9024b80 1 Processor -- start 2024-05-27T07:31:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.253+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.253+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5d6db0 0x55efaf5e0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.254+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5dcd30 0x55efaf5e0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.254+0000 7f54e9024b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55efaf3fafa0 con 0x55efaf457e40 2024-05-27T07:31:51.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.254+0000 7f54e9024b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55efaf495760 con 0x55efaf5d6db0 2024-05-27T07:31:51.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.254+0000 7f54e9024b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55efaf5e3740 con 0x55efaf5dcd30 2024-05-27T07:31:51.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.255+0000 7f54e9024b80 10 client.0 _ll_register_callbacks cb 0x55efaf49ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.255+0000 7f54d2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5d6db0 0x55efaf5e0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.255+0000 7f54d3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.255+0000 7f54d3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58200/0 (socket says 172.21.15.43:58200) 2024-05-27T07:31:51.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.255+0000 7f54d3400700 1 -- 172.21.15.43:0/2815968901 learned_addr learned my addr 172.21.15.43:0/2815968901 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:51.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.256+0000 7f54d3400700 1 -- 172.21.15.43:0/2815968901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5dcd30 msgr2=0x55efaf5e0f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:31:51.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.256+0000 7f54d3e00700 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5dcd30 0x55efaf5e0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.256+0000 7f54d3400700 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5dcd30 0x55efaf5e0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.256+0000 7f54d3400700 1 -- 172.21.15.43:0/2815968901 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5d6db0 msgr2=0x55efaf5e0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:51.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.256+0000 7f54d3400700 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5d6db0 0x55efaf5e0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.256+0000 7f54d3400700 1 -- 172.21.15.43:0/2815968901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efaf4448d0 con 0x55efaf457e40 2024-05-27T07:31:51.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.257+0000 7f54d3400700 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e04f0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f54c40aa190 tx=0x7f54c40e0dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8dfedc36799fda server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:51.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.257+0000 7f54bd600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:31:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.258+0000 7f54bd600700 1 -- 172.21.15.43:0/2815968901 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c40c4060 con 0x55efaf457e40 2024-05-27T07:31:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.259+0000 7f54bd600700 1 -- 172.21.15.43:0/2815968901 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7f54c40b2fe0 con 0x55efaf457e40 2024-05-27T07:31:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.259+0000 7f54e9024b80 1 -- 172.21.15.43:0/2815968901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54cc017c90 con 0x55efaf457e40 2024-05-27T07:31:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.260+0000 7f54bd600700 1 -- 172.21.15.43:0/2815968901 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7f54c40c9050 con 0x55efaf457e40 2024-05-27T07:31:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.260+0000 7f54e9024b80 10 client.17883 fetch_fsmap learned FSMap version 135 2024-05-27T07:31:51.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.260+0000 7f54e9024b80 1 -- 172.21.15.43:0/2815968901 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=4+}) -- 0x55efaf4c5060 con 0x55efaf457e40 2024-05-27T07:31:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.260+0000 7f54bd600700 1 -- 172.21.15.43:0/2815968901 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f54cc017c90 con 0x55efaf457e40 2024-05-27T07:31:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.260+0000 7f54bd600700 1 -- 172.21.15.43:0/2815968901 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 135) ==== 4195+0+0 (secure 0 0 0) 0x7f54c40e3170 con 0x55efaf457e40 2024-05-27T07:31:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.260+0000 7f54e9024b80 10 client.17883 fetch_fsmap finished waiting for FSMap version 135 2024-05-27T07:31:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.260+0000 7f54e9024b80 10 client.17883 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:31:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.261+0000 7f54e9024b80 20 client.17883 populate_metadata read hostname 'smithi043' 2024-05-27T07:31:51.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.261+0000 7f54e9024b80 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55efaf5f06f0 0x55efaf5f2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.261+0000 7f54e9024b80 4 client.17883 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:51.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.261+0000 7f54e9024b80 1 -- 172.21.15.43:0/2815968901 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efaf506d90 con 0x55efaf5f06f0 2024-05-27T07:31:51.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.261+0000 7f54d2a00700 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55efaf5f06f0 0x55efaf5f2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.263+0000 7f54d2a00700 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55efaf5f06f0 0x55efaf5f2b40 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3f06139853c97787 server_cookie=dbae681ebfb567ca in_seq=0 out_seq=0 2024-05-27T07:31:51.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.263+0000 7f54bd600700 10 client.17883 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:31:51.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.265+0000 7f54bd600700 1 -- 172.21.15.43:0/2815968901 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55efaf506d90 con 0x55efaf5f06f0 2024-05-27T07:31:51.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.265+0000 7f54bd600700 10 client.17883 handle_command_reply: tid=0 2024-05-27T07:31:51.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.265+0000 7f54e9024b80 1 client.17883 shutdown 2024-05-27T07:31:51.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54e9024b80 1 -- 172.21.15.43:0/2815968901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 msgr2=0x55efaf5e04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:51.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54e9024b80 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e04f0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f54c40aa190 tx=0x7f54c40e0dd0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54e9024b80 1 -- 172.21.15.43:0/2815968901 shutdown_connections 2024-05-27T07:31:51.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54e9024b80 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55efaf5f06f0 0x55efaf5f2b40 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54e9024b80 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5dcd30 0x55efaf5e0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54e9024b80 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e04f0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54e9024b80 1 --2- 172.21.15.43:0/2815968901 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5d6db0 0x55efaf5e0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54bd600700 0 client.17883 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:31:51.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54e9024b80 1 -- 172.21.15.43:0/2815968901 >> 172.21.15.43:0/2815968901 conn(0x55efaf4b7fc0 msgr2=0x55efaf5d1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:51.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54e9024b80 1 -- 172.21.15.43:0/2815968901 shutdown_connections 2024-05-27T07:31:51.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.266+0000 7f54e9024b80 1 -- 172.21.15.43:0/2815968901 wait complete. 2024-05-27T07:31:51.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.267+0000 7f54e9024b80 20 client.17883 trim_cache size 0 max 16384 2024-05-27T07:31:51.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.294+0000 7f54e9024b80 1 Processor -- start 2024-05-27T07:31:51.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.294+0000 7f54e9024b80 1 -- start start 2024-05-27T07:31:51.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.294+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5d6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.295+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5d6db0 0x55efaf5dac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.295+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 0x55efaf5dbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.295+0000 7f54e9024b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41bf8f0 con 0x55efaf457e40 2024-05-27T07:31:51.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.295+0000 7f54e9024b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41bdd50 con 0x55efaf5dcd30 2024-05-27T07:31:51.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.295+0000 7f54e9024b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41bf220 con 0x55efaf5d6db0 2024-05-27T07:31:51.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.295+0000 7f54d3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 0x55efaf5dbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.295+0000 7f54d3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5d6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.295+0000 7f54d3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 0x55efaf5dbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:53532/0 (socket says 172.21.15.43:53532) 2024-05-27T07:31:51.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.295+0000 7f54d3e00700 1 -- 172.21.15.43:0/1522302493 learned_addr learned my addr 172.21.15.43:0/1522302493 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:51.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.295+0000 7f54d2a00700 1 --2- 172.21.15.43:0/1522302493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5d6db0 0x55efaf5dac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.296+0000 7f54d3e00700 1 -- 172.21.15.43:0/1522302493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5d6db0 msgr2=0x55efaf5dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:51.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.296+0000 7f54d3e00700 1 --2- 172.21.15.43:0/1522302493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5d6db0 0x55efaf5dac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.296+0000 7f54d3e00700 1 -- 172.21.15.43:0/1522302493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 msgr2=0x55efaf5d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:51.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.296+0000 7f54d3e00700 1 --2- 172.21.15.43:0/1522302493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5d6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.296+0000 7f54d3e00700 1 -- 172.21.15.43:0/1522302493 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efaf5dea60 con 0x55efaf5dcd30 2024-05-27T07:31:51.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.296+0000 7f54d3e00700 1 --2- 172.21.15.43:0/1522302493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 0x55efaf5dbcc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0225f0 tx=0x7f54cc022620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6967a69a8271da8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:51.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.297+0000 7f54bfe00700 1 -- 172.21.15.43:0/1522302493 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc030070 con 0x55efaf5dcd30 2024-05-27T07:31:51.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.297+0000 7f54bfe00700 1 -- 172.21.15.43:0/1522302493 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f54cc009460 con 0x55efaf5dcd30 2024-05-27T07:31:51.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.297+0000 7f54bfe00700 1 -- 172.21.15.43:0/1522302493 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc022e10 con 0x55efaf5dcd30 2024-05-27T07:31:51.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.298+0000 7f54e9024b80 1 -- 172.21.15.43:0/1522302493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 msgr2=0x55efaf5dbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:51.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.298+0000 7f54e9024b80 1 --2- 172.21.15.43:0/1522302493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 0x55efaf5dbcc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0225f0 tx=0x7f54cc022620 comp rx=0 tx=0).stop 2024-05-27T07:31:51.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.298+0000 7f54e9024b80 1 -- 172.21.15.43:0/1522302493 shutdown_connections 2024-05-27T07:31:51.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.298+0000 7f54e9024b80 1 --2- 172.21.15.43:0/1522302493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5dcd30 0x55efaf5dbcc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.298+0000 7f54e9024b80 1 --2- 172.21.15.43:0/1522302493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5d6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.298+0000 7f54e9024b80 1 --2- 172.21.15.43:0/1522302493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5d6db0 0x55efaf5dac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.298+0000 7f54e9024b80 1 -- 172.21.15.43:0/1522302493 >> 172.21.15.43:0/1522302493 conn(0x55efaf4b7fc0 msgr2=0x55efaf5e04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:51.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.299+0000 7f54e9024b80 1 -- 172.21.15.43:0/1522302493 shutdown_connections 2024-05-27T07:31:51.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.299+0000 7f54e9024b80 1 -- 172.21.15.43:0/1522302493 wait complete. 2024-05-27T07:31:51.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.299+0000 7f54e9024b80 1 -- start start 2024-05-27T07:31:51.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.300+0000 7f54e9024b80 1 Processor -- start 2024-05-27T07:31:51.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.300+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.300+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5d6db0 0x55efaf5e90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.300+0000 7f54e9024b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5dcd30 0x55efaf5f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.300+0000 7f54e9024b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41beb20 con 0x55efaf457e40 2024-05-27T07:31:51.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.300+0000 7f54e9024b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41be450 con 0x55efaf5d6db0 2024-05-27T07:31:51.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.300+0000 7f54e9024b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55efaf5f7db0 con 0x55efaf5dcd30 2024-05-27T07:31:51.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.300+0000 7f54e9024b80 10 client.0 _ll_register_callbacks cb 0x55efaf49ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:51.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.301+0000 7f54d3e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5dcd30 0x55efaf5f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.301+0000 7f54d3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.301+0000 7f54d3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58210/0 (socket says 172.21.15.43:58210) 2024-05-27T07:31:51.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.301+0000 7f54d3400700 1 -- 172.21.15.43:0/885472064 learned_addr learned my addr 172.21.15.43:0/885472064 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:51.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.301+0000 7f54d2a00700 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5d6db0 0x55efaf5e90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.302+0000 7f54d3400700 1 -- 172.21.15.43:0/885472064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5dcd30 msgr2=0x55efaf5f76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:51.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.302+0000 7f54d3400700 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5dcd30 0x55efaf5f76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.302+0000 7f54d3400700 1 -- 172.21.15.43:0/885472064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5d6db0 msgr2=0x55efaf5e90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.302+0000 7f54d3400700 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5d6db0 0x55efaf5e90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.302+0000 7f54d3400700 1 -- 172.21.15.43:0/885472064 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efaf5f7fb0 con 0x55efaf457e40 2024-05-27T07:31:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.303+0000 7f54d3400700 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e8b90 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f54c4005f50 tx=0x7f54c40cb2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64203f7bf1d07ccb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:51.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.303+0000 7f54bd600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:31:51.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.303+0000 7f54bd600700 1 -- 172.21.15.43:0/885472064 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c40a6350 con 0x55efaf457e40 2024-05-27T07:31:51.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.303+0000 7f54bd600700 1 -- 172.21.15.43:0/885472064 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7f54c40e3170 con 0x55efaf457e40 2024-05-27T07:31:51.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.303+0000 7f54e9024b80 1 -- 172.21.15.43:0/885472064 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55efaf500570 con 0x55efaf457e40 2024-05-27T07:31:51.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.304+0000 7f54bd600700 1 -- 172.21.15.43:0/885472064 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7f54c40b7030 con 0x55efaf457e40 2024-05-27T07:31:51.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.304+0000 7f54e9024b80 10 client.17889 fetch_fsmap learned FSMap version 135 2024-05-27T07:31:51.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.304+0000 7f54e9024b80 1 -- 172.21.15.43:0/885472064 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=4+}) -- 0x55efaf65ad10 con 0x55efaf457e40 2024-05-27T07:31:51.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.304+0000 7f54bd600700 1 -- 172.21.15.43:0/885472064 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55efaf500570 con 0x55efaf457e40 2024-05-27T07:31:51.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.305+0000 7f54bd600700 1 -- 172.21.15.43:0/885472064 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 135) ==== 4195+0+0 (secure 0 0 0) 0x7f54c40ba020 con 0x55efaf457e40 2024-05-27T07:31:51.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.305+0000 7f54e9024b80 10 client.17889 fetch_fsmap finished waiting for FSMap version 135 2024-05-27T07:31:51.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.305+0000 7f54e9024b80 10 client.17889 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:31:51.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.305+0000 7f54e9024b80 20 client.17889 populate_metadata read hostname 'smithi043' 2024-05-27T07:31:51.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.306+0000 7f54e9024b80 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55efaf65b190 0x55efaf65d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:51.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.306+0000 7f54e9024b80 4 client.17889 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:31:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.306+0000 7f54e9024b80 1 -- 172.21.15.43:0/885472064 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55efaf377250 con 0x55efaf65b190 2024-05-27T07:31:51.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.306+0000 7f54d2a00700 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55efaf65b190 0x55efaf65d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.307+0000 7f54d2a00700 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55efaf65b190 0x55efaf65d5b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f01b3acad6f8a81b server_cookie=aef0994d300177e4 in_seq=0 out_seq=0 2024-05-27T07:31:51.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.308+0000 7f54bd600700 10 client.17889 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:31:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.313+0000 7f54bd600700 1 -- 172.21.15.43:0/885472064 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55efaf377250 con 0x55efaf65b190 2024-05-27T07:31:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.313+0000 7f54bd600700 10 client.17889 handle_command_reply: tid=0 2024-05-27T07:31:51.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.313+0000 7f54e9024b80 1 client.17889 shutdown 2024-05-27T07:31:51.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.313+0000 7f54e9024b80 1 -- 172.21.15.43:0/885472064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 msgr2=0x55efaf5e8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:51.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.313+0000 7f54e9024b80 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e8b90 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f54c4005f50 tx=0x7f54c40cb2a0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.313+0000 7f54e9024b80 1 -- 172.21.15.43:0/885472064 shutdown_connections 2024-05-27T07:31:51.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.313+0000 7f54e9024b80 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55efaf65b190 0x55efaf65d5b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.314+0000 7f54e9024b80 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55efaf5dcd30 0x55efaf5f76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.314+0000 7f54bd600700 0 client.17889 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:31:51.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.314+0000 7f54e9024b80 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55efaf457e40 0x55efaf5e8b90 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.314+0000 7f54e9024b80 1 --2- 172.21.15.43:0/885472064 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55efaf5d6db0 0x55efaf5e90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.314+0000 7f54e9024b80 1 -- 172.21.15.43:0/885472064 >> 172.21.15.43:0/885472064 conn(0x55efaf4b7fc0 msgr2=0x55efaf5e0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:51.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.314+0000 7f54e9024b80 1 -- 172.21.15.43:0/885472064 shutdown_connections 2024-05-27T07:31:51.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.314+0000 7f54e9024b80 1 -- 172.21.15.43:0/885472064 wait complete. 2024-05-27T07:31:51.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.314+0000 7f54e9024b80 20 client.17889 trim_cache size 0 max 16384 2024-05-27T07:31:51.309 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:31:51.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 -- 172.21.15.43:0/3298133808 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f54b80b2b40 msgr2=0x7f54b80b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:51.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f54b80b2b40 0x7f54b80b5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f54cc007890 tx=0x7f54cc009cb0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 -- 172.21.15.43:0/3298133808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d4075ad0 msgr2=0x7f54d4104330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:51.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d4075ad0 0x7f54d4104330 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f54c400b650 tx=0x7f54c400bfc0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 -- 172.21.15.43:0/3298133808 shutdown_connections 2024-05-27T07:31:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f54b80b2b40 0x7f54b80b5050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54d4110c10 0x7f54d4104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d4075ad0 0x7f54d4104330 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 --2- 172.21.15.43:0/3298133808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54d4076420 0x7f54d4104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 -- 172.21.15.43:0/3298133808 >> 172.21.15.43:0/3298133808 conn(0x7f54d4100030 msgr2=0x7f54d4101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.316+0000 7f54da400700 1 -- 172.21.15.43:0/3298133808 shutdown_connections 2024-05-27T07:31:51.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:51.317+0000 7f54da400700 1 -- 172.21.15.43:0/3298133808 wait complete. 2024-05-27T07:31:51.704 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:51 smithi043 ceph-mon[31531]: pgmap v601: 129 pgs: 129 active+clean; 672 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 2.6 MiB/s wr, 161 op/s 2024-05-27T07:31:51.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:51 smithi179 ceph-mon[35888]: pgmap v601: 129 pgs: 129 active+clean; 672 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 2.6 MiB/s wr, 161 op/s 2024-05-27T07:31:51.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 25677 5.68 MB/sec execute 112 sec latency 88.966 ms 2024-05-27T07:31:51.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:51 smithi132 ceph-mon[35283]: pgmap v601: 129 pgs: 129 active+clean; 672 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 2.6 MiB/s wr, 161 op/s 2024-05-27T07:31:52.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:31:52.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 25817 5.71 MB/sec execute 113 sec latency 86.577 ms 2024-05-27T07:31:52.845 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:53.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:53 smithi043 ceph-mon[31531]: pgmap v602: 129 pgs: 129 active+clean; 681 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-05-27T07:31:53.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:53 smithi179 ceph-mon[35888]: pgmap v602: 129 pgs: 129 active+clean; 681 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-05-27T07:31:53.786 INFO:tasks.workunit.client.0.smithi043.stdout: 10 25962 5.77 MB/sec execute 114 sec latency 234.951 ms 2024-05-27T07:31:53.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:53 smithi132 ceph-mon[35283]: pgmap v602: 129 pgs: 129 active+clean; 681 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-05-27T07:31:54.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 26130 5.80 MB/sec execute 115 sec latency 85.548 ms 2024-05-27T07:31:55.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.419+0000 7f0efae00700 1 -- 172.21.15.43:0/1037724073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ef4105510 msgr2=0x7f0ef4105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.419+0000 7f0efae00700 1 --2- 172.21.15.43:0/1037724073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ef4105510 0x7f0ef4105920 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0098f0 tx=0x7f0eec009c00 comp rx=0 tx=0).stop 2024-05-27T07:31:55.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.420+0000 7f0efae00700 1 -- 172.21.15.43:0/1037724073 shutdown_connections 2024-05-27T07:31:55.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.420+0000 7f0efae00700 1 --2- 172.21.15.43:0/1037724073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef4110cf0 0x7f0ef4113110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.420+0000 7f0efae00700 1 --2- 172.21.15.43:0/1037724073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ef4105510 0x7f0ef4105920 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.420+0000 7f0efae00700 1 --2- 172.21.15.43:0/1037724073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ef4105e60 0x7f0ef41107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.420+0000 7f0efae00700 1 -- 172.21.15.43:0/1037724073 >> 172.21.15.43:0/1037724073 conn(0x7f0ef40fedb0 msgr2=0x7f0ef41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:55.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.422+0000 7f0efae00700 1 -- 172.21.15.43:0/1037724073 shutdown_connections 2024-05-27T07:31:55.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.422+0000 7f0efae00700 1 -- 172.21.15.43:0/1037724073 wait complete. 2024-05-27T07:31:55.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.423+0000 7f0efae00700 1 Processor -- start 2024-05-27T07:31:55.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.423+0000 7f0efae00700 1 -- start start 2024-05-27T07:31:55.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.424+0000 7f0efae00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef4105510 0x7f0ef41a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.425+0000 7f0efae00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ef4105e60 0x7f0ef41a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.425+0000 7f0efae00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ef4110cf0 0x7f0ef41a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.425+0000 7f0efae00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef410cc70 con 0x7f0ef4105510 2024-05-27T07:31:55.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.425+0000 7f0efae00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef410caf0 con 0x7f0ef4110cf0 2024-05-27T07:31:55.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.425+0000 7f0efae00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef410cdf0 con 0x7f0ef4105e60 2024-05-27T07:31:55.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.425+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef4105510 0x7f0ef41a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.425+0000 7f0ef3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef4105510 0x7f0ef41a1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53886/0 (socket says 172.21.15.43:53886) 2024-05-27T07:31:55.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.425+0000 7f0ef3e00700 1 -- 172.21.15.43:0/2709890758 learned_addr learned my addr 172.21.15.43:0/2709890758 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:55.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.426+0000 7f0ef3400700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ef4105e60 0x7f0ef41a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.426+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ef4110cf0 0x7f0ef41a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.426+0000 7f0ef3e00700 1 -- 172.21.15.43:0/2709890758 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ef4105e60 msgr2=0x7f0ef41a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.426+0000 7f0ef3e00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ef4105e60 0x7f0ef41a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.426+0000 7f0ef3e00700 1 -- 172.21.15.43:0/2709890758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ef4110cf0 msgr2=0x7f0ef41a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.427+0000 7f0ef3e00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ef4110cf0 0x7f0ef41a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.427+0000 7f0ef3e00700 1 -- 172.21.15.43:0/2709890758 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef41a6350 con 0x7f0ef4105510 2024-05-27T07:31:55.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.427+0000 7f0ef3400700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ef4105e60 0x7f0ef41a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:31:55.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.427+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ef4110cf0 0x7f0ef41a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:31:55.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.427+0000 7f0ef3e00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef4105510 0x7f0ef41a1480 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f0eec011040 tx=0x7f0eec00fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:55.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.428+0000 7f0ef0c00700 1 -- 172.21.15.43:0/2709890758 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec027070 con 0x7f0ef4105510 2024-05-27T07:31:55.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.429+0000 7f0ef0c00700 1 -- 172.21.15.43:0/2709890758 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0eec005040 con 0x7f0ef4105510 2024-05-27T07:31:55.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.429+0000 7f0ef0c00700 1 -- 172.21.15.43:0/2709890758 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec004a50 con 0x7f0ef4105510 2024-05-27T07:31:55.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.429+0000 7f0efae00700 1 -- 172.21.15.43:0/2709890758 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef41a65e0 con 0x7f0ef4105510 2024-05-27T07:31:55.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.429+0000 7f0efae00700 1 -- 172.21.15.43:0/2709890758 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef41a6a70 con 0x7f0ef4105510 2024-05-27T07:31:55.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.438+0000 7f0f0992ab80 1 Processor -- start 2024-05-27T07:31:55.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.439+0000 7f0f0992ab80 1 -- start start 2024-05-27T07:31:55.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.439+0000 7f0ef0c00700 1 -- 172.21.15.43:0/2709890758 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0eec02a070 con 0x7f0ef4105510 2024-05-27T07:31:55.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.439+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb162f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.440+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162fd30 0x55ccb1629870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.440+0000 7f0ef8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb162f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.440+0000 7f0ef8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb162f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53892/0 (socket says 172.21.15.43:53892) 2024-05-27T07:31:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.440+0000 7f0ef8e00700 1 -- 172.21.15.43:0/2478810346 learned_addr learned my addr 172.21.15.43:0/2478810346 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.440+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/2478810346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb1629db0 0x55ccb162a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.440+0000 7f0f0992ab80 1 -- 172.21.15.43:0/2478810346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ccb1557e40 con 0x55ccb14aae40 2024-05-27T07:31:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.440+0000 7f0f0992ab80 1 -- 172.21.15.43:0/2478810346 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ccb15479e0 con 0x55ccb1629db0 2024-05-27T07:31:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.440+0000 7f0f0992ab80 1 -- 172.21.15.43:0/2478810346 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ccb154ba20 con 0x55ccb162fd30 2024-05-27T07:31:55.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.441+0000 7f0ef8e00700 1 -- 172.21.15.43:0/2478810346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162fd30 msgr2=0x55ccb1629870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:31:55.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.441+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/2478810346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162fd30 0x55ccb1629870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.441+0000 7f0ef8e00700 1 -- 172.21.15.43:0/2478810346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb1629db0 msgr2=0x55ccb162a1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:31:55.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.441+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/2478810346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb1629db0 0x55ccb162a1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.441+0000 7f0ef8e00700 1 -- 172.21.15.43:0/2478810346 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccb1510420 con 0x55ccb14aae40 2024-05-27T07:31:55.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.442+0000 7f0ef0c00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0ed80b2bc0 0x7f0ed80b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.442+0000 7f0ef0c00700 1 -- 172.21.15.43:0/2709890758 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7f0eec0e3570 con 0x7f0ef4105510 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.442+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/2478810346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb162f7f0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009440 tx=0x7f0ee40099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef939221c4ab117 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.443+0000 7f0ee9600700 1 -- 172.21.15.43:0/2478810346 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4013810 con 0x55ccb14aae40 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.443+0000 7f0ee9600700 1 -- 172.21.15.43:0/2478810346 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0ee4013e40 con 0x55ccb14aae40 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.443+0000 7f0ee9600700 1 -- 172.21.15.43:0/2478810346 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40143e0 con 0x55ccb14aae40 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.444+0000 7f0f0992ab80 1 -- 172.21.15.43:0/2478810346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 msgr2=0x55ccb162f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.444+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/2478810346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb162f7f0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009440 tx=0x7f0ee40099e0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.445+0000 7f0f0992ab80 1 -- 172.21.15.43:0/2478810346 shutdown_connections 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.445+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/2478810346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb1629db0 0x55ccb162a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.445+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/2478810346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb162f7f0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.445+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/2478810346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162fd30 0x55ccb1629870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.445+0000 7f0f0992ab80 1 -- 172.21.15.43:0/2478810346 >> 172.21.15.43:0/2478810346 conn(0x55ccb150afc0 msgr2=0x55ccb16248d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.445+0000 7f0ef3e00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0ed80b2bc0 0x7f0ed80b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.445+0000 7f0f0992ab80 1 -- 172.21.15.43:0/2478810346 shutdown_connections 2024-05-27T07:31:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.445+0000 7f0f0992ab80 1 -- 172.21.15.43:0/2478810346 wait complete. 2024-05-27T07:31:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.447+0000 7f0f0992ab80 1 -- start start 2024-05-27T07:31:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.447+0000 7f0f0992ab80 1 Processor -- start 2024-05-27T07:31:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.448+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb14aae40 0x55ccb1633550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.449+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb1629db0 0x55ccb1633a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.449+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162fd30 0x55ccb1633fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.449+0000 7f0f0992ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ccb144dfa0 con 0x55ccb1629db0 2024-05-27T07:31:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.449+0000 7f0f0992ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ccb14e8760 con 0x55ccb14aae40 2024-05-27T07:31:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.449+0000 7f0f0992ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ccb1636800 con 0x55ccb162fd30 2024-05-27T07:31:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.449+0000 7f0f0992ab80 10 client.0 _ll_register_callbacks cb 0x55ccb14f1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.450+0000 7f0ef8e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb14aae40 0x55ccb1633550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.450+0000 7f0ef3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb1629db0 0x55ccb1633a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.450+0000 7f0ef3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb1629db0 0x55ccb1633a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53904/0 (socket says 172.21.15.43:53904) 2024-05-27T07:31:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.450+0000 7f0ef3400700 1 -- 172.21.15.43:0/1583646658 learned_addr learned my addr 172.21.15.43:0/1583646658 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.452+0000 7f0ef3400700 1 -- 172.21.15.43:0/1583646658 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162fd30 msgr2=0x55ccb1633fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.452+0000 7f0ef3400700 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162fd30 0x55ccb1633fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.453+0000 7f0ef3400700 1 -- 172.21.15.43:0/1583646658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb14aae40 msgr2=0x55ccb1633550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.453+0000 7f0ef3400700 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb14aae40 0x55ccb1633550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.453+0000 7f0ef3400700 1 -- 172.21.15.43:0/1583646658 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccb14978d0 con 0x55ccb1629db0 2024-05-27T07:31:55.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.454+0000 7f0ef3e00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0ed80b2bc0 0x7f0ed80b50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0aa1b0 tx=0x7f0eec005ec0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:55.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.454+0000 7f0ef3400700 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb1629db0 0x55ccb1633a90 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00d1e0 tx=0x7f0edc00d5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8037d50ea22c50c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:55.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.455+0000 7f0ee2000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:31:55.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.455+0000 7f0ee2000700 1 -- 172.21.15.43:0/1583646658 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc013480 con 0x55ccb1629db0 2024-05-27T07:31:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.455+0000 7f0ee2000700 1 -- 172.21.15.43:0/1583646658 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7f0edc013620 con 0x55ccb1629db0 2024-05-27T07:31:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.455+0000 7f0f0992ab80 1 -- 172.21.15.43:0/1583646658 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ccb1559d90 con 0x55ccb1629db0 2024-05-27T07:31:55.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.456+0000 7f0ee2000700 1 -- 172.21.15.43:0/1583646658 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7f0edc021050 con 0x55ccb1629db0 2024-05-27T07:31:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.456+0000 7f0f0992ab80 10 client.17901 fetch_fsmap learned FSMap version 135 2024-05-27T07:31:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.456+0000 7f0f0992ab80 1 -- 172.21.15.43:0/1583646658 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=4+}) -- 0x55ccb1518060 con 0x55ccb1629db0 2024-05-27T07:31:55.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.456+0000 7f0ee2000700 1 -- 172.21.15.43:0/1583646658 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55ccb1559d90 con 0x55ccb1629db0 2024-05-27T07:31:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.457+0000 7f0ee2000700 1 -- 172.21.15.43:0/1583646658 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 135) ==== 4195+0+0 (secure 0 0 0) 0x7f0edc012780 con 0x55ccb1629db0 2024-05-27T07:31:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.457+0000 7f0f0992ab80 10 client.17901 fetch_fsmap finished waiting for FSMap version 135 2024-05-27T07:31:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.457+0000 7f0f0992ab80 10 client.17901 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:31:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.457+0000 7f0f0992ab80 20 client.17901 populate_metadata read hostname 'smithi043' 2024-05-27T07:31:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.457+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ccb1643cd0 0x55ccb16461c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.457+0000 7f0f0992ab80 4 client.17901 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:31:55.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.457+0000 7f0f0992ab80 1 -- 172.21.15.43:0/1583646658 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccb1553570 con 0x55ccb1643cd0 2024-05-27T07:31:55.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.460+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ccb1643cd0 0x55ccb16461c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.461+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ccb1643cd0 0x55ccb16461c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6376525b4c5a0d92 server_cookie=15c100ce0d33fd83 in_seq=0 out_seq=0 2024-05-27T07:31:55.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.461+0000 7f0ee2000700 10 client.17901 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:31:55.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.462+0000 7f0ee2000700 1 -- 172.21.15.43:0/1583646658 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ccb1553570 con 0x55ccb1643cd0 2024-05-27T07:31:55.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.462+0000 7f0ee2000700 10 client.17901 handle_command_reply: tid=0 2024-05-27T07:31:55.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.462+0000 7f0f0992ab80 1 client.17901 shutdown 2024-05-27T07:31:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.463+0000 7f0f0992ab80 1 -- 172.21.15.43:0/1583646658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb1629db0 msgr2=0x55ccb1633a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.463+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb1629db0 0x55ccb1633a90 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00d1e0 tx=0x7f0edc00d5a0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.464+0000 7f0f0992ab80 1 -- 172.21.15.43:0/1583646658 shutdown_connections 2024-05-27T07:31:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.464+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162fd30 0x55ccb1633fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.464+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ccb1643cd0 0x55ccb16461c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.464+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb14aae40 0x55ccb1633550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.464+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/1583646658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb1629db0 0x55ccb1633a90 secure :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00d1e0 tx=0x7f0edc00d5a0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.464+0000 7f0f0992ab80 1 -- 172.21.15.43:0/1583646658 >> 172.21.15.43:0/1583646658 conn(0x55ccb150afc0 msgr2=0x55ccb162a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.464+0000 7f0ee2000700 0 client.17901 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:31:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.464+0000 7f0f0992ab80 1 -- 172.21.15.43:0/1583646658 shutdown_connections 2024-05-27T07:31:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.464+0000 7f0f0992ab80 1 -- 172.21.15.43:0/1583646658 wait complete. 2024-05-27T07:31:55.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.465+0000 7f0f0992ab80 20 client.17901 trim_cache size 0 max 16384 2024-05-27T07:31:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.493+0000 7f0f0992ab80 1 Processor -- start 2024-05-27T07:31:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.493+0000 7f0f0992ab80 1 -- start start 2024-05-27T07:31:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.494+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb1629870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.494+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 0x55ccb1632890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.494+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 0x55ccb16359f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.494+0000 7f0f0992ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ccb1646320 con 0x55ccb14aae40 2024-05-27T07:31:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.494+0000 7f0f0992ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b7090 con 0x55ccb162fd30 2024-05-27T07:31:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.494+0000 7f0f0992ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b54f0 con 0x55ccb162f0e0 2024-05-27T07:31:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.494+0000 7f0ef8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb1629870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.495+0000 7f0ef3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 0x55ccb1632890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.495+0000 7f0ef8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb1629870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53908/0 (socket says 172.21.15.43:53908) 2024-05-27T07:31:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.495+0000 7f0ef8e00700 1 -- 172.21.15.43:0/3661726709 learned_addr learned my addr 172.21.15.43:0/3661726709 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.495+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/3661726709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 0x55ccb16359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.495+0000 7f0ef8e00700 1 -- 172.21.15.43:0/3661726709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 msgr2=0x55ccb16359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.495+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/3661726709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 0x55ccb16359f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.495+0000 7f0ef8e00700 1 -- 172.21.15.43:0/3661726709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 msgr2=0x55ccb1632890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.495+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/3661726709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 0x55ccb1632890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.495+0000 7f0ef8e00700 1 -- 172.21.15.43:0/3661726709 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccb1626030 con 0x55ccb14aae40 2024-05-27T07:31:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.496+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/3661726709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb1629870 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x55ccb162ab30 tx=0x7f0ee4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48c2d97ebdcbc53 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.497+0000 7f0ee8c00700 1 -- 172.21.15.43:0/3661726709 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4004c10 con 0x55ccb14aae40 2024-05-27T07:31:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.497+0000 7f0ee8c00700 1 -- 172.21.15.43:0/3661726709 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0ee4004db0 con 0x55ccb14aae40 2024-05-27T07:31:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.497+0000 7f0ee8c00700 1 -- 172.21.15.43:0/3661726709 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4013610 con 0x55ccb14aae40 2024-05-27T07:31:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.499+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3661726709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 msgr2=0x55ccb1629870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.499+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/3661726709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb1629870 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x55ccb162ab30 tx=0x7f0ee4009060 comp rx=0 tx=0).stop 2024-05-27T07:31:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.499+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3661726709 shutdown_connections 2024-05-27T07:31:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.499+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/3661726709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 0x55ccb16359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.499+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/3661726709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb1629870 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.499+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/3661726709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 0x55ccb1632890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.499+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3661726709 >> 172.21.15.43:0/3661726709 conn(0x55ccb150afc0 msgr2=0x55ccb162d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.500+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3661726709 shutdown_connections 2024-05-27T07:31:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.500+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3661726709 wait complete. 2024-05-27T07:31:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.501+0000 7f0f0992ab80 1 -- start start 2024-05-27T07:31:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.501+0000 7f0f0992ab80 1 Processor -- start 2024-05-27T07:31:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.502+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb163bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.502+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 0x55ccb163c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.503+0000 7f0f0992ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 0x55ccb164a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.503+0000 7f0f0992ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b5bf0 con 0x55ccb14aae40 2024-05-27T07:31:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.503+0000 7f0f0992ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b69c0 con 0x55ccb162fd30 2024-05-27T07:31:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.503+0000 7f0f0992ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41b62c0 con 0x55ccb162f0e0 2024-05-27T07:31:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.503+0000 7f0f0992ab80 10 client.0 _ll_register_callbacks cb 0x55ccb14f1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:31:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.503+0000 7f0ef8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb163bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.503+0000 7f0ef8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb163bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53914/0 (socket says 172.21.15.43:53914) 2024-05-27T07:31:55.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.503+0000 7f0ef8e00700 1 -- 172.21.15.43:0/3172451212 learned_addr learned my addr 172.21.15.43:0/3172451212 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:31:55.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.503+0000 7f0ef3400700 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 0x55ccb163c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.504+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 0x55ccb164a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.504+0000 7f0ef3400700 1 -- 172.21.15.43:0/3172451212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 msgr2=0x55ccb164a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:55.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.504+0000 7f0ef3400700 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 0x55ccb164a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.504+0000 7f0ef3400700 1 -- 172.21.15.43:0/3172451212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 msgr2=0x55ccb163bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:31:55.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.504+0000 7f0ef3400700 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb163bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.504+0000 7f0ef3400700 1 -- 172.21.15.43:0/3172451212 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccb164af30 con 0x55ccb162f0e0 2024-05-27T07:31:55.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.504+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 0x55ccb164a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:31:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.505+0000 7f0ef3400700 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 0x55ccb163c000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0edc01bec0 tx=0x7f0edc00da10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1ea2e2d43566bef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:31:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.505+0000 7f0ee2000700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:31:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.506+0000 7f0ee2000700 1 -- 172.21.15.43:0/3172451212 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc0125f0 con 0x55ccb162f0e0 2024-05-27T07:31:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.506+0000 7f0ee2000700 1 -- 172.21.15.43:0/3172451212 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7f0edc012790 con 0x55ccb162f0e0 2024-05-27T07:31:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.507+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3172451212 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ee4004db0 con 0x55ccb162f0e0 2024-05-27T07:31:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.507+0000 7f0ee2000700 1 -- 172.21.15.43:0/3172451212 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7f0edc012990 con 0x55ccb162f0e0 2024-05-27T07:31:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.507+0000 7f0f0992ab80 10 client.27355 fetch_fsmap learned FSMap version 135 2024-05-27T07:31:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.507+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3172451212 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=4+}) -- 0x55ccb16adfd0 con 0x55ccb162f0e0 2024-05-27T07:31:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.508+0000 7f0ee2000700 1 -- 172.21.15.43:0/3172451212 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0ee4004db0 con 0x55ccb162f0e0 2024-05-27T07:31:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.508+0000 7f0ee2000700 1 -- 172.21.15.43:0/3172451212 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 135) ==== 4195+0+0 (secure 0 0 0) 0x7f0edc003a20 con 0x55ccb162f0e0 2024-05-27T07:31:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.508+0000 7f0f0992ab80 10 client.27355 fetch_fsmap finished waiting for FSMap version 135 2024-05-27T07:31:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.508+0000 7f0f0992ab80 10 client.27355 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:31:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.508+0000 7f0f0992ab80 20 client.27355 populate_metadata read hostname 'smithi043' 2024-05-27T07:31:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.508+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ccb16ae660 0x55ccb16b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:31:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.508+0000 7f0f0992ab80 4 client.27355 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:31:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.508+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3172451212 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ccb13ca250 con 0x55ccb16ae660 2024-05-27T07:31:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.509+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ccb16ae660 0x55ccb16b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:31:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.510+0000 7f0ef8e00700 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ccb16ae660 0x55ccb16b0a80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f58acefe020f9d6c server_cookie=3131e4f340957ea6 in_seq=0 out_seq=0 2024-05-27T07:31:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.510+0000 7f0ee2000700 10 client.27355 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:31:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.511+0000 7f0ee2000700 1 -- 172.21.15.43:0/3172451212 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ccb13ca250 con 0x55ccb16ae660 2024-05-27T07:31:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.511+0000 7f0ee2000700 10 client.27355 handle_command_reply: tid=0 2024-05-27T07:31:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.511+0000 7f0f0992ab80 1 client.27355 shutdown 2024-05-27T07:31:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.511+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3172451212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 msgr2=0x55ccb163c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.511+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 0x55ccb163c000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0edc01bec0 tx=0x7f0edc00da10 comp rx=0 tx=0).stop 2024-05-27T07:31:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.512+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3172451212 shutdown_connections 2024-05-27T07:31:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.512+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ccb16ae660 0x55ccb16b0a80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.512+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ccb162fd30 0x55ccb164a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.512+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb14aae40 0x55ccb163bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.512+0000 7f0ee2000700 0 client.27355 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:31:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.512+0000 7f0f0992ab80 1 --2- 172.21.15.43:0/3172451212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ccb162f0e0 0x55ccb163c000 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.512+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3172451212 >> 172.21.15.43:0/3172451212 conn(0x55ccb150afc0 msgr2=0x55ccb16325f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.513+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3172451212 shutdown_connections 2024-05-27T07:31:55.509 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:31:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.513+0000 7f0f0992ab80 1 -- 172.21.15.43:0/3172451212 wait complete. 2024-05-27T07:31:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.513+0000 7f0f0992ab80 20 client.27355 trim_cache size 0 max 16384 2024-05-27T07:31:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.514+0000 7f0efae00700 1 -- 172.21.15.43:0/2709890758 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0ed80b2bc0 msgr2=0x7f0ed80b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.514+0000 7f0efae00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0ed80b2bc0 0x7f0ed80b50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0aa1b0 tx=0x7f0eec005ec0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.514+0000 7f0efae00700 1 -- 172.21.15.43:0/2709890758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef4105510 msgr2=0x7f0ef41a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:31:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.514+0000 7f0efae00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef4105510 0x7f0ef41a1480 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f0eec011040 tx=0x7f0eec00fa40 comp rx=0 tx=0).stop 2024-05-27T07:31:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.516+0000 7f0efae00700 1 -- 172.21.15.43:0/2709890758 shutdown_connections 2024-05-27T07:31:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.516+0000 7f0efae00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0ed80b2bc0 0x7f0ed80b50d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.516+0000 7f0efae00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ef4110cf0 0x7f0ef41a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.516+0000 7f0efae00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ef4105510 0x7f0ef41a1480 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.516+0000 7f0efae00700 1 --2- 172.21.15.43:0/2709890758 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ef4105e60 0x7f0ef41a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:31:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.516+0000 7f0efae00700 1 -- 172.21.15.43:0/2709890758 >> 172.21.15.43:0/2709890758 conn(0x7f0ef40fedb0 msgr2=0x7f0ef41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:31:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.516+0000 7f0efae00700 1 -- 172.21.15.43:0/2709890758 shutdown_connections 2024-05-27T07:31:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:31:55.516+0000 7f0efae00700 1 -- 172.21.15.43:0/2709890758 wait complete. 2024-05-27T07:31:55.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:55 smithi043 ceph-mon[31531]: pgmap v603: 129 pgs: 129 active+clean; 681 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 2.4 MiB/s wr, 157 op/s 2024-05-27T07:31:55.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:55 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:55 smithi179 ceph-mon[35888]: pgmap v603: 129 pgs: 129 active+clean; 681 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 2.4 MiB/s wr, 157 op/s 2024-05-27T07:31:55.710 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:55 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:55.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 26236 5.79 MB/sec execute 116 sec latency 101.347 ms 2024-05-27T07:31:55.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:55 smithi132 ceph-mon[35283]: pgmap v603: 129 pgs: 129 active+clean; 681 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 2.4 MiB/s wr, 157 op/s 2024-05-27T07:31:55.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:55 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:31:56.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:31:56.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 26326 5.80 MB/sec execute 117 sec latency 90.190 ms 2024-05-27T07:31:57.354 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:31:57.723 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:57 smithi043 ceph-mon[31531]: pgmap v604: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 4.1 MiB/s wr, 232 op/s 2024-05-27T07:31:57.723 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:57 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:57.723 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:57 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:57.723 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:57 smithi043 ceph-mon[31531]: pgmap v605: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 3.6 MiB/s wr, 177 op/s 2024-05-27T07:31:57.723 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:57 smithi043 ceph-mon[31531]: pgmap v606: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 4.4 MiB/s wr, 220 op/s 2024-05-27T07:31:57.723 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:57 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:31:57.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 26414 5.81 MB/sec execute 118 sec latency 80.377 ms 2024-05-27T07:31:57.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:57 smithi132 ceph-mon[35283]: pgmap v604: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 4.1 MiB/s wr, 232 op/s 2024-05-27T07:31:57.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:57 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:57.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:57 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:57.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:57 smithi132 ceph-mon[35283]: pgmap v605: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 3.6 MiB/s wr, 177 op/s 2024-05-27T07:31:57.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:57 smithi132 ceph-mon[35283]: pgmap v606: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 4.4 MiB/s wr, 220 op/s 2024-05-27T07:31:57.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:57 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:31:57.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:57 smithi179 ceph-mon[35888]: pgmap v604: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 4.1 MiB/s wr, 232 op/s 2024-05-27T07:31:57.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:57 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:31:57.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:57 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:31:57.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:57 smithi179 ceph-mon[35888]: pgmap v605: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 3.6 MiB/s wr, 177 op/s 2024-05-27T07:31:57.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:57 smithi179 ceph-mon[35888]: pgmap v606: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 4.4 MiB/s wr, 220 op/s 2024-05-27T07:31:57.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:57 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:31:58.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 26503 5.83 MB/sec execute 119 sec latency 442.884 ms 2024-05-27T07:31:59.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 26635 5.85 MB/sec execute 120 sec latency 78.276 ms 2024-05-27T07:31:59.842 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:31:59 smithi132 ceph-mon[35283]: pgmap v607: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 4.4 MiB/s wr, 220 op/s 2024-05-27T07:31:59.917 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:31:59 smithi043 ceph-mon[31531]: pgmap v607: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 4.4 MiB/s wr, 220 op/s 2024-05-27T07:31:59.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:31:59 smithi179 ceph-mon[35888]: pgmap v607: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 4.4 MiB/s wr, 220 op/s 2024-05-27T07:32:00.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.138+0000 7fdc05200700 1 -- 172.21.15.43:0/534352031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc001034b0 msgr2=0x7fdc001038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.138+0000 7fdc05200700 1 --2- 172.21.15.43:0/534352031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc001034b0 0x7fdc001038c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800b0a0 tx=0x7fdbf800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.140+0000 7fdc05200700 1 -- 172.21.15.43:0/534352031 shutdown_connections 2024-05-27T07:32:00.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.140+0000 7fdc05200700 1 --2- 172.21.15.43:0/534352031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc00108730 0x7fdc0006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.140+0000 7fdc05200700 1 --2- 172.21.15.43:0/534352031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc001034b0 0x7fdc001038c0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.140+0000 7fdc05200700 1 --2- 172.21.15.43:0/534352031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc00107de0 0x7fdc001081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.140+0000 7fdc05200700 1 -- 172.21.15.43:0/534352031 >> 172.21.15.43:0/534352031 conn(0x7fdc000fedb0 msgr2=0x7fdc001011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.141+0000 7fdc05200700 1 -- 172.21.15.43:0/534352031 shutdown_connections 2024-05-27T07:32:00.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.141+0000 7fdc05200700 1 -- 172.21.15.43:0/534352031 wait complete. 2024-05-27T07:32:00.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.142+0000 7fdc05200700 1 Processor -- start 2024-05-27T07:32:00.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.143+0000 7fdc05200700 1 -- start start 2024-05-27T07:32:00.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.143+0000 7fdc05200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc001034b0 0x7fdc00111b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.143+0000 7fdc05200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc00107de0 0x7fdc0010eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.143+0000 7fdc05200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc00108730 0x7fdc0010f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.143+0000 7fdc05200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0010d5b0 con 0x7fdc00108730 2024-05-27T07:32:00.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.143+0000 7fdc05200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0010d430 con 0x7fdc001034b0 2024-05-27T07:32:00.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.143+0000 7fdc05200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0010d730 con 0x7fdc00107de0 2024-05-27T07:32:00.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.144+0000 7fdbfd400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc00107de0 0x7fdc0010eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.144+0000 7fdbfe800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc00108730 0x7fdc0010f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.144+0000 7fdbfd400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc00107de0 0x7fdc0010eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:57128/0 (socket says 172.21.15.43:57128) 2024-05-27T07:32:00.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.144+0000 7fdbfd400700 1 -- 172.21.15.43:0/3803835163 learned_addr learned my addr 172.21.15.43:0/3803835163 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:00.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.144+0000 7fdbfde00700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc001034b0 0x7fdc00111b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.144+0000 7fdbfe800700 1 -- 172.21.15.43:0/3803835163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc00107de0 msgr2=0x7fdc0010eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.144+0000 7fdbfe800700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc00107de0 0x7fdc0010eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.144+0000 7fdbfe800700 1 -- 172.21.15.43:0/3803835163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc001034b0 msgr2=0x7fdc00111b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.144+0000 7fdbfe800700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc001034b0 0x7fdc00111b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.144+0000 7fdbfe800700 1 -- 172.21.15.43:0/3803835163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0010f970 con 0x7fdc00108730 2024-05-27T07:32:00.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.145+0000 7fdbfe800700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc00108730 0x7fdc0010f1e0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000d720 tx=0x7fdbf000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.145+0000 7fdbf6a00700 1 -- 172.21.15.43:0/3803835163 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0016070 con 0x7fdc00108730 2024-05-27T07:32:00.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.146+0000 7fdbf6a00700 1 -- 172.21.15.43:0/3803835163 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbf0004830 con 0x7fdc00108730 2024-05-27T07:32:00.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.146+0000 7fdbf6a00700 1 -- 172.21.15.43:0/3803835163 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00054e0 con 0x7fdc00108730 2024-05-27T07:32:00.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.146+0000 7fdc05200700 1 -- 172.21.15.43:0/3803835163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc001ad800 con 0x7fdc00108730 2024-05-27T07:32:00.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.146+0000 7fdc05200700 1 -- 172.21.15.43:0/3803835163 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc001adc60 con 0x7fdc00108730 2024-05-27T07:32:00.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.151+0000 7fdbf6a00700 1 -- 172.21.15.43:0/3803835163 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fdbf0004220 con 0x7fdc00108730 2024-05-27T07:32:00.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.153+0000 7fdc13d0db80 1 Processor -- start 2024-05-27T07:32:00.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.153+0000 7fdc13d0db80 1 -- start start 2024-05-27T07:32:00.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.153+0000 7fdbf6a00700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdbe40b2af0 0x7fdbe40b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.154+0000 7fdbf6a00700 1 -- 172.21.15.43:0/3803835163 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7fdbf00e2660 con 0x7fdc00108730 2024-05-27T07:32:00.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.154+0000 7fdbfd400700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdbe40b2af0 0x7fdbe40b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.154+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ec027f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.154+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 0x55c30ebfc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.154+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ebfd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.154+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c30eb2ae40 con 0x55c30ea7de40 2024-05-27T07:32:00.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.154+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c30eb1a9e0 con 0x55c30ebfcdb0 2024-05-27T07:32:00.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.154+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c30eb1ea20 con 0x55c30ec02d30 2024-05-27T07:32:00.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.155+0000 7fdbfde00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ec027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.155+0000 7fdbfde00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ec027f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53942/0 (socket says 172.21.15.43:53942) 2024-05-27T07:32:00.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.155+0000 7fdbfde00700 1 -- 172.21.15.43:0/1201415085 learned_addr learned my addr 172.21.15.43:0/1201415085 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:00.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.156+0000 7fdbfd400700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdbe40b2af0 0x7fdbe40b5000 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdc0006b8d0 tx=0x7fdbec009260 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.156+0000 7fdbfd400700 1 --2- 172.21.15.43:0/1201415085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 0x55c30ebfc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.156+0000 7fdbfe800700 1 --2- 172.21.15.43:0/1201415085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ebfd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.156+0000 7fdbfde00700 1 -- 172.21.15.43:0/1201415085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 msgr2=0x55c30ebfc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.156+0000 7fdbfde00700 1 --2- 172.21.15.43:0/1201415085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 0x55c30ebfc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.156+0000 7fdbfde00700 1 -- 172.21.15.43:0/1201415085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 msgr2=0x55c30ebfd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.156+0000 7fdbfde00700 1 --2- 172.21.15.43:0/1201415085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ebfd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.156+0000 7fdbfde00700 1 -- 172.21.15.43:0/1201415085 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c30eae3420 con 0x55c30ea7de40 2024-05-27T07:32:00.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.158+0000 7fdbfde00700 1 --2- 172.21.15.43:0/1201415085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ec027f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8006070 tx=0x7fdbf8009d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ccd612dcc1a386 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.159+0000 7fdbeb400700 1 -- 172.21.15.43:0/1201415085 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf800bcb0 con 0x55c30ea7de40 2024-05-27T07:32:00.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.159+0000 7fdbeb400700 1 -- 172.21.15.43:0/1201415085 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbf800be50 con 0x55c30ea7de40 2024-05-27T07:32:00.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.159+0000 7fdbeb400700 1 -- 172.21.15.43:0/1201415085 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8005040 con 0x55c30ea7de40 2024-05-27T07:32:00.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.160+0000 7fdc13d0db80 1 -- 172.21.15.43:0/1201415085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 msgr2=0x55c30ec027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.160+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/1201415085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ec027f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8006070 tx=0x7fdbf8009d90 comp rx=0 tx=0).stop 2024-05-27T07:32:00.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.160+0000 7fdc13d0db80 1 -- 172.21.15.43:0/1201415085 shutdown_connections 2024-05-27T07:32:00.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.160+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/1201415085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ebfd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.160+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/1201415085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ec027f0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.160+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/1201415085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 0x55c30ebfc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.160+0000 7fdc13d0db80 1 -- 172.21.15.43:0/1201415085 >> 172.21.15.43:0/1201415085 conn(0x55c30eaddfc0 msgr2=0x55c30ebf78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.161+0000 7fdc13d0db80 1 -- 172.21.15.43:0/1201415085 shutdown_connections 2024-05-27T07:32:00.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.161+0000 7fdc13d0db80 1 -- 172.21.15.43:0/1201415085 wait complete. 2024-05-27T07:32:00.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.161+0000 7fdc13d0db80 1 -- start start 2024-05-27T07:32:00.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.162+0000 7fdc13d0db80 1 Processor -- start 2024-05-27T07:32:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.162+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ea7de40 0x55c30ec06510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.162+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ebfcdb0 0x55c30ec06a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.162+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec06f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.162+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c30ea20fa0 con 0x55c30ec02d30 2024-05-27T07:32:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.162+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c30eabb760 con 0x55c30ea7de40 2024-05-27T07:32:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.162+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c30ec097a0 con 0x55c30ebfcdb0 2024-05-27T07:32:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.163+0000 7fdc13d0db80 10 client.0 _ll_register_callbacks cb 0x55c30eac4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:00.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.163+0000 7fdbfe800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec06f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.163+0000 7fdbfd400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ebfcdb0 0x55c30ec06a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.163+0000 7fdbfe800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec06f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53958/0 (socket says 172.21.15.43:53958) 2024-05-27T07:32:00.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.163+0000 7fdbfe800700 1 -- 172.21.15.43:0/4069014187 learned_addr learned my addr 172.21.15.43:0/4069014187 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:00.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.164+0000 7fdbfde00700 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ea7de40 0x55c30ec06510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.164+0000 7fdbfe800700 1 -- 172.21.15.43:0/4069014187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ebfcdb0 msgr2=0x55c30ec06a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.164+0000 7fdbfe800700 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ebfcdb0 0x55c30ec06a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.164+0000 7fdbfe800700 1 -- 172.21.15.43:0/4069014187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ea7de40 msgr2=0x55c30ec06510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.164+0000 7fdbfe800700 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ea7de40 0x55c30ec06510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.164+0000 7fdbfe800700 1 -- 172.21.15.43:0/4069014187 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c30ea6a8d0 con 0x55c30ec02d30 2024-05-27T07:32:00.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.164+0000 7fdbfe800700 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec06f90 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fdbf00a6780 tx=0x7fdbf00a9cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7f1a99515c05d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.164+0000 7fdbdfe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:32:00.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.165+0000 7fdbdfe00700 1 -- 172.21.15.43:0/4069014187 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00b2370 con 0x55c30ec02d30 2024-05-27T07:32:00.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.165+0000 7fdbdfe00700 1 -- 172.21.15.43:0/4069014187 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7fdbf00a5b00 con 0x55c30ec02d30 2024-05-27T07:32:00.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.165+0000 7fdc13d0db80 1 -- 172.21.15.43:0/4069014187 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbf800be50 con 0x55c30ec02d30 2024-05-27T07:32:00.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.166+0000 7fdbdfe00700 1 -- 172.21.15.43:0/4069014187 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7fdbf00e23f0 con 0x55c30ec02d30 2024-05-27T07:32:00.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.166+0000 7fdc13d0db80 10 client.17919 fetch_fsmap learned FSMap version 135 2024-05-27T07:32:00.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.166+0000 7fdc13d0db80 1 -- 172.21.15.43:0/4069014187 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=4+}) -- 0x55c30eaeb060 con 0x55c30ec02d30 2024-05-27T07:32:00.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.167+0000 7fdbdfe00700 1 -- 172.21.15.43:0/4069014187 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbf800be50 con 0x55c30ec02d30 2024-05-27T07:32:00.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.167+0000 7fdbdfe00700 1 -- 172.21.15.43:0/4069014187 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 135) ==== 4195+0+0 (secure 0 0 0) 0x7fdbf00b1ee0 con 0x55c30ec02d30 2024-05-27T07:32:00.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.167+0000 7fdc13d0db80 10 client.17919 fetch_fsmap finished waiting for FSMap version 135 2024-05-27T07:32:00.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.167+0000 7fdc13d0db80 10 client.17919 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:32:00.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.167+0000 7fdc13d0db80 20 client.17919 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:00.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.167+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c30ec16b10 0x55c30ec18f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.167+0000 7fdc13d0db80 4 client.17919 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:00.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.167+0000 7fdc13d0db80 1 -- 172.21.15.43:0/4069014187 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c30eb2cd90 con 0x55c30ec16b10 2024-05-27T07:32:00.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.167+0000 7fdbfde00700 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c30ec16b10 0x55c30ec18f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.169+0000 7fdbfde00700 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c30ec16b10 0x55c30ec18f80 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=43b17b6363729fa6 server_cookie=5a33b44388320922 in_seq=0 out_seq=0 2024-05-27T07:32:00.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.169+0000 7fdbdfe00700 10 client.17919 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:32:00.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.170+0000 7fdbdfe00700 1 -- 172.21.15.43:0/4069014187 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c30eb2cd90 con 0x55c30ec16b10 2024-05-27T07:32:00.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.170+0000 7fdbdfe00700 10 client.17919 handle_command_reply: tid=0 2024-05-27T07:32:00.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.170+0000 7fdc13d0db80 1 client.17919 shutdown 2024-05-27T07:32:00.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.171+0000 7fdc13d0db80 1 -- 172.21.15.43:0/4069014187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 msgr2=0x55c30ec06f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.171+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec06f90 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fdbf00a6780 tx=0x7fdbf00a9cc0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.171+0000 7fdc13d0db80 1 -- 172.21.15.43:0/4069014187 shutdown_connections 2024-05-27T07:32:00.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.171+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c30ec16b10 0x55c30ec18f80 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.171+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec06f90 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.171+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ea7de40 0x55c30ec06510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.171+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/4069014187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ebfcdb0 0x55c30ec06a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.171+0000 7fdbdfe00700 0 client.17919 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:32:00.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.171+0000 7fdc13d0db80 1 -- 172.21.15.43:0/4069014187 >> 172.21.15.43:0/4069014187 conn(0x55c30eaddfc0 msgr2=0x55c30ebf7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.172+0000 7fdc13d0db80 1 -- 172.21.15.43:0/4069014187 shutdown_connections 2024-05-27T07:32:00.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.172+0000 7fdc13d0db80 1 -- 172.21.15.43:0/4069014187 wait complete. 2024-05-27T07:32:00.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.172+0000 7fdc13d0db80 20 client.17919 trim_cache size 0 max 16384 2024-05-27T07:32:00.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.199+0000 7fdc13d0db80 1 Processor -- start 2024-05-27T07:32:00.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.199+0000 7fdc13d0db80 1 -- start start 2024-05-27T07:32:00.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.200+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ebfc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.200+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ec12890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.200+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 0x55c30ec00850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.200+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001b70b0 con 0x55c30ea7de40 2024-05-27T07:32:00.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.200+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001b5510 con 0x55c30ebfcdb0 2024-05-27T07:32:00.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.200+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001b69e0 con 0x55c30ec02d30 2024-05-27T07:32:00.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.200+0000 7fdbfd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ec12890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.201+0000 7fdbfde00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ebfc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.201+0000 7fdbfe800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 0x55c30ec00850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.201+0000 7fdbfd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ec12890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:49726/0 (socket says 172.21.15.43:49726) 2024-05-27T07:32:00.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.201+0000 7fdbfd400700 1 -- 172.21.15.43:0/1491665836 learned_addr learned my addr 172.21.15.43:0/1491665836 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:00.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.202+0000 7fdbfe800700 1 -- 172.21.15.43:0/1491665836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 msgr2=0x55c30ec12890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.202+0000 7fdbfe800700 1 --2- 172.21.15.43:0/1491665836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ec12890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.202+0000 7fdbfe800700 1 -- 172.21.15.43:0/1491665836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 msgr2=0x55c30ebfc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.202+0000 7fdbfe800700 1 --2- 172.21.15.43:0/1491665836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ebfc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.202+0000 7fdbfe800700 1 -- 172.21.15.43:0/1491665836 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c30ec06f00 con 0x55c30ec02d30 2024-05-27T07:32:00.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.202+0000 7fdbfd400700 1 --2- 172.21.15.43:0/1491665836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ec12890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:00.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.202+0000 7fdbfde00700 1 --2- 172.21.15.43:0/1491665836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ebfc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.203+0000 7fdbfe800700 1 --2- 172.21.15.43:0/1491665836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 0x55c30ec00850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdbf00beb20 tx=0x7fdbf00e2590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46c9d7d8bfcd7d4a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.203+0000 7fdbeaa00700 1 -- 172.21.15.43:0/1491665836 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00be960 con 0x55c30ec02d30 2024-05-27T07:32:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.203+0000 7fdbeaa00700 1 -- 172.21.15.43:0/1491665836 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbf00c3070 con 0x55c30ec02d30 2024-05-27T07:32:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.203+0000 7fdbeaa00700 1 -- 172.21.15.43:0/1491665836 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00b1ee0 con 0x55c30ec02d30 2024-05-27T07:32:00.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.204+0000 7fdc13d0db80 1 -- 172.21.15.43:0/1491665836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 msgr2=0x55c30ec00850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.204+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/1491665836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 0x55c30ec00850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdbf00beb20 tx=0x7fdbf00e2590 comp rx=0 tx=0).stop 2024-05-27T07:32:00.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.205+0000 7fdc13d0db80 1 -- 172.21.15.43:0/1491665836 shutdown_connections 2024-05-27T07:32:00.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.205+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/1491665836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ec02d30 0x55c30ec00850 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.205+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/1491665836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ea7de40 0x55c30ebfc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.205+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/1491665836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ec12890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.205+0000 7fdc13d0db80 1 -- 172.21.15.43:0/1491665836 >> 172.21.15.43:0/1491665836 conn(0x55c30eaddfc0 msgr2=0x55c30ec00140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.206+0000 7fdc13d0db80 1 -- 172.21.15.43:0/1491665836 shutdown_connections 2024-05-27T07:32:00.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.206+0000 7fdc13d0db80 1 -- 172.21.15.43:0/1491665836 wait complete. 2024-05-27T07:32:00.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.206+0000 7fdc13d0db80 1 -- start start 2024-05-27T07:32:00.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.206+0000 7fdc13d0db80 1 Processor -- start 2024-05-27T07:32:00.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.207+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ea7de40 0x55c30ec0ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.207+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ec0efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.207+0000 7fdc13d0db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec1d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.207+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001b62e0 con 0x55c30ec02d30 2024-05-27T07:32:00.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.207+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001b5c10 con 0x55c30ebfcdb0 2024-05-27T07:32:00.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.207+0000 7fdc13d0db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c30ec09270 con 0x55c30ea7de40 2024-05-27T07:32:00.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.207+0000 7fdc13d0db80 10 client.0 _ll_register_callbacks cb 0x55c30eac4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:00.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.208+0000 7fdbfe800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec1d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.208+0000 7fdbfde00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ea7de40 0x55c30ec0ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.208+0000 7fdbfde00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ea7de40 0x55c30ec0ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:57166/0 (socket says 172.21.15.43:57166) 2024-05-27T07:32:00.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.208+0000 7fdbfde00700 1 -- 172.21.15.43:0/2232586462 learned_addr learned my addr 172.21.15.43:0/2232586462 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.208+0000 7fdbfd400700 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ec0efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.209+0000 7fdbfde00700 1 -- 172.21.15.43:0/2232586462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 msgr2=0x55c30ec0efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.209+0000 7fdbfde00700 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ec0efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.209+0000 7fdbfde00700 1 -- 172.21.15.43:0/2232586462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 msgr2=0x55c30ec1d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.209+0000 7fdbfde00700 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec1d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.209+0000 7fdbfde00700 1 -- 172.21.15.43:0/2232586462 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c30ec1dea0 con 0x55c30ea7de40 2024-05-27T07:32:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.209+0000 7fdbfe800700 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec1d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.211+0000 7fdbfde00700 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ea7de40 0x55c30ec0ea80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800b010 tx=0x7fdbf8009dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89f5b1419abc1a7d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.211+0000 7fdbdfe00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:32:00.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.212+0000 7fdbdfe00700 1 -- 172.21.15.43:0/2232586462 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8004d20 con 0x55c30ea7de40 2024-05-27T07:32:00.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.212+0000 7fdbdfe00700 1 -- 172.21.15.43:0/2232586462 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 9496+0+0 (secure 0 0 0) 0x7fdbf800f040 con 0x55c30ea7de40 2024-05-27T07:32:00.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.212+0000 7fdc13d0db80 1 -- 172.21.15.43:0/2232586462 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbf00c3070 con 0x55c30ea7de40 2024-05-27T07:32:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.213+0000 7fdbdfe00700 1 -- 172.21.15.43:0/2232586462 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7fdbf80075c0 con 0x55c30ea7de40 2024-05-27T07:32:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.213+0000 7fdc13d0db80 10 client.27373 fetch_fsmap learned FSMap version 135 2024-05-27T07:32:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.213+0000 7fdc13d0db80 1 -- 172.21.15.43:0/2232586462 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=4+}) -- 0x55c30ec80e90 con 0x55c30ea7de40 2024-05-27T07:32:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.214+0000 7fdbdfe00700 1 -- 172.21.15.43:0/2232586462 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbf00c3070 con 0x55c30ea7de40 2024-05-27T07:32:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.214+0000 7fdbdfe00700 1 -- 172.21.15.43:0/2232586462 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 135) ==== 4195+0+0 (secure 0 0 0) 0x7fdbf8011070 con 0x55c30ea7de40 2024-05-27T07:32:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.214+0000 7fdc13d0db80 10 client.27373 fetch_fsmap finished waiting for FSMap version 135 2024-05-27T07:32:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.214+0000 7fdc13d0db80 10 client.27373 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:32:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.214+0000 7fdc13d0db80 20 client.27373 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.214+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c30ec812e0 0x55c30ec83700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.215+0000 7fdc13d0db80 4 client.27373 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:32:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.215+0000 7fdc13d0db80 1 -- 172.21.15.43:0/2232586462 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c30eb26570 con 0x55c30ec812e0 2024-05-27T07:32:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.215+0000 7fdbfd400700 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c30ec812e0 0x55c30ec83700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.216+0000 7fdbfd400700 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c30ec812e0 0x55c30ec83700 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dddcb21010ed5e56 server_cookie=6b7b3aa9a0a567fd in_seq=0 out_seq=0 2024-05-27T07:32:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.217+0000 7fdbdfe00700 10 client.27373 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:32:00.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.217+0000 7fdbdfe00700 1 -- 172.21.15.43:0/2232586462 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c30eb26570 con 0x55c30ec812e0 2024-05-27T07:32:00.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.217+0000 7fdbdfe00700 10 client.27373 handle_command_reply: tid=0 2024-05-27T07:32:00.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.217+0000 7fdc13d0db80 1 client.27373 shutdown 2024-05-27T07:32:00.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.218+0000 7fdc13d0db80 1 -- 172.21.15.43:0/2232586462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ea7de40 msgr2=0x55c30ec0ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.218+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ea7de40 0x55c30ec0ea80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800b010 tx=0x7fdbf8009dc0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.219+0000 7fdc13d0db80 1 -- 172.21.15.43:0/2232586462 shutdown_connections 2024-05-27T07:32:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.219+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c30ec812e0 0x55c30ec83700 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.219+0000 7fdbdfe00700 0 client.27373 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:32:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.219+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c30ec02d30 0x55c30ec1d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.219+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c30ea7de40 0x55c30ec0ea80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.219+0000 7fdc13d0db80 1 --2- 172.21.15.43:0/2232586462 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c30ebfcdb0 0x55c30ec0efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.219+0000 7fdc13d0db80 1 -- 172.21.15.43:0/2232586462 >> 172.21.15.43:0/2232586462 conn(0x55c30eaddfc0 msgr2=0x55c30ec03430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.219+0000 7fdc13d0db80 1 -- 172.21.15.43:0/2232586462 shutdown_connections 2024-05-27T07:32:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.219+0000 7fdc13d0db80 1 -- 172.21.15.43:0/2232586462 wait complete. 2024-05-27T07:32:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.220+0000 7fdc13d0db80 20 client.27373 trim_cache size 0 max 16384 2024-05-27T07:32:00.214 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:32:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 -- 172.21.15.43:0/3803835163 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdbe40b2af0 msgr2=0x7fdbe40b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdbe40b2af0 0x7fdbe40b5000 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdc0006b8d0 tx=0x7fdbec009260 comp rx=0 tx=0).stop 2024-05-27T07:32:00.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 -- 172.21.15.43:0/3803835163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc00108730 msgr2=0x7fdc0010f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:00.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc00108730 0x7fdc0010f1e0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000d720 tx=0x7fdbf000da30 comp rx=0 tx=0).stop 2024-05-27T07:32:00.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 -- 172.21.15.43:0/3803835163 shutdown_connections 2024-05-27T07:32:00.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdbe40b2af0 0x7fdbe40b5000 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdc00108730 0x7fdc0010f1e0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdc001034b0 0x7fdc00111b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 --2- 172.21.15.43:0/3803835163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc00107de0 0x7fdc0010eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 -- 172.21.15.43:0/3803835163 >> 172.21.15.43:0/3803835163 conn(0x7fdc000fedb0 msgr2=0x7fdc0010b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 -- 172.21.15.43:0/3803835163 shutdown_connections 2024-05-27T07:32:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:00.221+0000 7fdc05200700 1 -- 172.21.15.43:0/3803835163 wait complete. 2024-05-27T07:32:00.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 26777 5.94 MB/sec execute 121 sec latency 125.854 ms 2024-05-27T07:32:01.162 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:32:01.785 INFO:tasks.workunit.client.0.smithi043.stdout: 10 26890 5.91 MB/sec execute 122 sec latency 110.593 ms 2024-05-27T07:32:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:01.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:01 smithi043 ceph-mon[31531]: pgmap v608: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 2.4 MiB/s wr, 110 op/s 2024-05-27T07:32:01.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:01 smithi179 ceph-mon[35888]: pgmap v608: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 2.4 MiB/s wr, 110 op/s 2024-05-27T07:32:02.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:01 smithi132 ceph-mon[35283]: pgmap v608: 129 pgs: 129 active+clean; 694 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 2.4 MiB/s wr, 110 op/s 2024-05-27T07:32:02.706 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:02 smithi043 ceph-mon[31531]: osdmap e146: 12 total, 12 up, 12 in 2024-05-27T07:32:02.786 INFO:tasks.workunit.client.0.smithi043.stdout: 10 26984 5.91 MB/sec execute 123 sec latency 99.437 ms 2024-05-27T07:32:02.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:02 smithi179 ceph-mon[35888]: osdmap e146: 12 total, 12 up, 12 in 2024-05-27T07:32:03.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:02 smithi132 ceph-mon[35283]: osdmap e146: 12 total, 12 up, 12 in 2024-05-27T07:32:03.670 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:03 smithi043 ceph-mon[31531]: pgmap v610: 129 pgs: 129 active+clean; 708 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.4 MiB/s wr, 159 op/s 2024-05-27T07:32:03.786 INFO:tasks.workunit.client.0.smithi043.stdout: 10 27066 5.89 MB/sec execute 124 sec latency 181.437 ms 2024-05-27T07:32:03.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:03 smithi179 ceph-mon[35888]: pgmap v610: 129 pgs: 129 active+clean; 708 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.4 MiB/s wr, 159 op/s 2024-05-27T07:32:04.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:03 smithi132 ceph-mon[35283]: pgmap v610: 129 pgs: 129 active+clean; 708 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 4.4 MiB/s wr, 159 op/s 2024-05-27T07:32:04.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.317+0000 7f9918e00700 1 -- 172.21.15.43:0/3778494035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9914103a00 msgr2=0x7f9914110720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.317+0000 7f9918e00700 1 --2- 172.21.15.43:0/3778494035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9914103a00 0x7f9914110720 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009990 tx=0x7f98fc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.318+0000 7f9918e00700 1 -- 172.21.15.43:0/3778494035 shutdown_connections 2024-05-27T07:32:04.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.318+0000 7f9918e00700 1 --2- 172.21.15.43:0/3778494035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9914110c60 0x7f9914113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.318+0000 7f9918e00700 1 --2- 172.21.15.43:0/3778494035 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99141030b0 0x7f99141034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.318+0000 7f9918e00700 1 --2- 172.21.15.43:0/3778494035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9914103a00 0x7f9914110720 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.318+0000 7f9918e00700 1 -- 172.21.15.43:0/3778494035 >> 172.21.15.43:0/3778494035 conn(0x7f99140fedb0 msgr2=0x7f99141011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.318+0000 7f9918e00700 1 -- 172.21.15.43:0/3778494035 shutdown_connections 2024-05-27T07:32:04.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.318+0000 7f9918e00700 1 -- 172.21.15.43:0/3778494035 wait complete. 2024-05-27T07:32:04.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.320+0000 7f9918e00700 1 Processor -- start 2024-05-27T07:32:04.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.320+0000 7f9918e00700 1 -- start start 2024-05-27T07:32:04.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.321+0000 7f9918e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99141030b0 0x7f991419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.321+0000 7f9918e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9914103a00 0x7f991419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.321+0000 7f9918e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9914110c60 0x7f9914198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.322+0000 7f9918e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9914107940 con 0x7f9914103a00 2024-05-27T07:32:04.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.322+0000 7f9918e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f99141077c0 con 0x7f9914110c60 2024-05-27T07:32:04.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.322+0000 7f9918e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9914107ac0 con 0x7f99141030b0 2024-05-27T07:32:04.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.322+0000 7f9911e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99141030b0 0x7f991419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.322+0000 7f9911400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9914103a00 0x7f991419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.322+0000 7f9911e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99141030b0 0x7f991419e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:57186/0 (socket says 172.21.15.43:57186) 2024-05-27T07:32:04.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.322+0000 7f9911e00700 1 -- 172.21.15.43:0/2535672926 learned_addr learned my addr 172.21.15.43:0/2535672926 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:04.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.323+0000 7f9911400700 1 -- 172.21.15.43:0/2535672926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99141030b0 msgr2=0x7f991419e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.323+0000 7f9911400700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99141030b0 0x7f991419e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.323+0000 7f9911400700 1 -- 172.21.15.43:0/2535672926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9914110c60 msgr2=0x7f9914198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.323+0000 7f9911400700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9914110c60 0x7f9914198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.323+0000 7f9911400700 1 -- 172.21.15.43:0/2535672926 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9914199360 con 0x7f9914103a00 2024-05-27T07:32:04.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.323+0000 7f9911400700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9914103a00 0x7f991419ed80 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f98fc000c00 tx=0x7f98fc0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.324+0000 7f990aa00700 1 -- 172.21.15.43:0/2535672926 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc026070 con 0x7f9914103a00 2024-05-27T07:32:04.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.324+0000 7f990aa00700 1 -- 172.21.15.43:0/2535672926 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98fc021410 con 0x7f9914103a00 2024-05-27T07:32:04.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.324+0000 7f990aa00700 1 -- 172.21.15.43:0/2535672926 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc017650 con 0x7f9914103a00 2024-05-27T07:32:04.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.324+0000 7f9918e00700 1 -- 172.21.15.43:0/2535672926 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99141995f0 con 0x7f9914103a00 2024-05-27T07:32:04.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.324+0000 7f9918e00700 1 -- 172.21.15.43:0/2535672926 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9914199a80 con 0x7f9914103a00 2024-05-27T07:32:04.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.331+0000 7f990aa00700 1 -- 172.21.15.43:0/2535672926 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f98fc009d10 con 0x7f9914103a00 2024-05-27T07:32:04.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.334+0000 7f9927a42b80 1 Processor -- start 2024-05-27T07:32:04.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.334+0000 7f990aa00700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98f80b2a70 0x7f98f80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.335+0000 7f990aa00700 1 -- 172.21.15.43:0/2535672926 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 9584+0+0 (secure 0 0 0) 0x7f98fc0e3730 con 0x7f9914103a00 2024-05-27T07:32:04.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.335+0000 7f9912800700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98f80b2a70 0x7f98f80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.335+0000 7f9927a42b80 1 -- start start 2024-05-27T07:32:04.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.336+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.336+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf8629e40 0x55bbf87a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.336+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.336+0000 7f9927a42b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55bbf86d6e40 con 0x55bbf87159f0 2024-05-27T07:32:04.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.336+0000 7f9927a42b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bbf86c69e0 con 0x55bbf87a8e70 2024-05-27T07:32:04.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.336+0000 7f9927a42b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55bbf86caa20 con 0x55bbf8629e40 2024-05-27T07:32:04.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.337+0000 7f9911400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf8629e40 0x55bbf87a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.337+0000 7f9911400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf8629e40 0x55bbf87a8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:57190/0 (socket says 172.21.15.43:57190) 2024-05-27T07:32:04.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.337+0000 7f9911400700 1 -- 172.21.15.43:0/1098749483 learned_addr learned my addr 172.21.15.43:0/1098749483 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:04.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.337+0000 7f9911e00700 1 --2- 172.21.15.43:0/1098749483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.337+0000 7f9911400700 1 -- 172.21.15.43:0/1098749483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 msgr2=0x55bbf87b5110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:32:04.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.337+0000 7f9911400700 1 --2- 172.21.15.43:0/1098749483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b5110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.337+0000 7f9911400700 1 -- 172.21.15.43:0/1098749483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 msgr2=0x55bbf87a8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.337+0000 7f9911400700 1 --2- 172.21.15.43:0/1098749483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.338+0000 7f9911400700 1 -- 172.21.15.43:0/1098749483 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbf868f420 con 0x55bbf8629e40 2024-05-27T07:32:04.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.338+0000 7f9912800700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98f80b2a70 0x7f98f80b4f80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f99040060b0 tx=0x7f9904015040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.338+0000 7f9911e00700 1 --2- 172.21.15.43:0/1098749483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:04.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.338+0000 7f9911400700 1 --2- 172.21.15.43:0/1098749483 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf8629e40 0x55bbf87a8870 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0aa610 tx=0x7f98fc0a9ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a3bb5a00334b2fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.339+0000 7f9903400700 1 -- 172.21.15.43:0/1098749483 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0aea30 con 0x55bbf8629e40 2024-05-27T07:32:04.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.339+0000 7f9903400700 1 -- 172.21.15.43:0/1098749483 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98fc0b3450 con 0x55bbf8629e40 2024-05-27T07:32:04.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.339+0000 7f9903400700 1 -- 172.21.15.43:0/1098749483 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0aa8e0 con 0x55bbf8629e40 2024-05-27T07:32:04.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.340+0000 7f9927a42b80 1 -- 172.21.15.43:0/1098749483 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf8629e40 msgr2=0x55bbf87a8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.340+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1098749483 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf8629e40 0x55bbf87a8870 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0aa610 tx=0x7f98fc0a9ef0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.340+0000 7f9927a42b80 1 -- 172.21.15.43:0/1098749483 shutdown_connections 2024-05-27T07:32:04.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.341+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1098749483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.341+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1098749483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.341+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1098749483 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf8629e40 0x55bbf87a8870 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.341+0000 7f9927a42b80 1 -- 172.21.15.43:0/1098749483 >> 172.21.15.43:0/1098749483 conn(0x55bbf86c3390 msgr2=0x55bbf87a44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.341+0000 7f9927a42b80 1 -- 172.21.15.43:0/1098749483 shutdown_connections 2024-05-27T07:32:04.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.341+0000 7f9927a42b80 1 -- 172.21.15.43:0/1098749483 wait complete. 2024-05-27T07:32:04.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.342+0000 7f9927a42b80 1 -- start start 2024-05-27T07:32:04.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.343+0000 7f9927a42b80 1 Processor -- start 2024-05-27T07:32:04.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.344+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf8629e40 0x55bbf87b2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.344+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87159f0 0x55bbf87b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.345+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.345+0000 7f9927a42b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55bbf85ccfa0 con 0x55bbf8629e40 2024-05-27T07:32:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.345+0000 7f9927a42b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55bbf8667760 con 0x55bbf87a8e70 2024-05-27T07:32:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.345+0000 7f9927a42b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55bbf87b5940 con 0x55bbf87159f0 2024-05-27T07:32:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.345+0000 7f9911400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87159f0 0x55bbf87b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.345+0000 7f9911e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf8629e40 0x55bbf87b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.345+0000 7f9927a42b80 10 client.0 _ll_register_callbacks cb 0x55bbf8670c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.345+0000 7f9911e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf8629e40 0x55bbf87b2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54002/0 (socket says 172.21.15.43:54002) 2024-05-27T07:32:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.345+0000 7f9911e00700 1 -- 172.21.15.43:0/1770823457 learned_addr learned my addr 172.21.15.43:0/1770823457 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.346+0000 7f9912800700 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.346+0000 7f9911e00700 1 -- 172.21.15.43:0/1770823457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87159f0 msgr2=0x55bbf87b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.347+0000 7f9911e00700 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87159f0 0x55bbf87b2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.347+0000 7f9911e00700 1 -- 172.21.15.43:0/1770823457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 msgr2=0x55bbf87b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.347+0000 7f9911e00700 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.347+0000 7f9911e00700 1 -- 172.21.15.43:0/1770823457 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbf86168d0 con 0x55bbf8629e40 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.347+0000 7f9911400700 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87159f0 0x55bbf87b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.347+0000 7f9911e00700 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf8629e40 0x55bbf87b2770 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f990c011410 tx=0x7f990c0117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fdab4f23fd0d856 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.347+0000 7f98f7e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.348+0000 7f98f7e00700 1 -- 172.21.15.43:0/1770823457 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c004230 con 0x55bbf8629e40 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.348+0000 7f98f7e00700 1 -- 172.21.15.43:0/1770823457 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(146..146 src has 1..146) ==== 9584+0+0 (secure 0 0 0) 0x7f990c0043d0 con 0x55bbf8629e40 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.348+0000 7f9927a42b80 1 -- 172.21.15.43:0/1770823457 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98fc0b3450 con 0x55bbf8629e40 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.348+0000 7f98f7e00700 1 -- 172.21.15.43:0/1770823457 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7f990c024050 con 0x55bbf8629e40 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.348+0000 7f9927a42b80 10 client.17934 fetch_fsmap learned FSMap version 135 2024-05-27T07:32:04.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.348+0000 7f9927a42b80 1 -- 172.21.15.43:0/1770823457 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=4+}) -- 0x55bbf8697060 con 0x55bbf8629e40 2024-05-27T07:32:04.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.349+0000 7f98f7e00700 1 -- 172.21.15.43:0/1770823457 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f98fc0b3450 con 0x55bbf8629e40 2024-05-27T07:32:04.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.349+0000 7f98f7e00700 1 -- 172.21.15.43:0/1770823457 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 135) ==== 4195+0+0 (secure 0 0 0) 0x7f990c01b720 con 0x55bbf8629e40 2024-05-27T07:32:04.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.349+0000 7f9927a42b80 10 client.17934 fetch_fsmap finished waiting for FSMap version 135 2024-05-27T07:32:04.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.349+0000 7f9927a42b80 10 client.17934 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:32:04.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.349+0000 7f9927a42b80 20 client.17934 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:04.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.349+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bbf87c2c90 0x55bbf87c5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.349+0000 7f9927a42b80 4 client.17934 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:04.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.349+0000 7f9927a42b80 1 -- 172.21.15.43:0/1770823457 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbf86d8d90 con 0x55bbf87c2c90 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.350+0000 7f9911400700 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bbf87c2c90 0x55bbf87c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.351+0000 7f98f7e00700 10 client.17934 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.351+0000 7f9911400700 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bbf87c2c90 0x55bbf87c5140 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d98b1c21119e742c server_cookie=1625eabace13e36d in_seq=0 out_seq=0 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f98f7e00700 1 -- 172.21.15.43:0/1770823457 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55bbf86d8d90 con 0x55bbf87c2c90 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f98f7e00700 10 client.17934 handle_command_reply: tid=0 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f9927a42b80 1 client.17934 shutdown 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f9927a42b80 1 -- 172.21.15.43:0/1770823457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf8629e40 msgr2=0x55bbf87b2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf8629e40 0x55bbf87b2770 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f990c011410 tx=0x7f990c0117d0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f9927a42b80 1 -- 172.21.15.43:0/1770823457 shutdown_connections 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bbf87c2c90 0x55bbf87c5140 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf8629e40 0x55bbf87b2770 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1770823457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87159f0 0x55bbf87b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f9927a42b80 1 -- 172.21.15.43:0/1770823457 >> 172.21.15.43:0/1770823457 conn(0x55bbf86c3390 msgr2=0x55bbf87a8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.352+0000 7f98f7e00700 0 client.17934 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:32:04.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.353+0000 7f9927a42b80 1 -- 172.21.15.43:0/1770823457 shutdown_connections 2024-05-27T07:32:04.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.353+0000 7f9927a42b80 1 -- 172.21.15.43:0/1770823457 wait complete. 2024-05-27T07:32:04.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.353+0000 7f9927a42b80 20 client.17934 trim_cache size 0 max 16384 2024-05-27T07:32:04.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.378+0000 7f9927a42b80 1 Processor -- start 2024-05-27T07:32:04.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.378+0000 7f9927a42b80 1 -- start start 2024-05-27T07:32:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.379+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.379+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87c5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.379+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87be9a0 0x55bbf87bedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.379+0000 7f9927a42b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55bbf87b18f0 con 0x55bbf87159f0 2024-05-27T07:32:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.379+0000 7f9927a42b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f99141ae930 con 0x55bbf87a8e70 2024-05-27T07:32:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.379+0000 7f9927a42b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f99141acd90 con 0x55bbf87be9a0 2024-05-27T07:32:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.379+0000 7f9911e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.380+0000 7f9911e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54006/0 (socket says 172.21.15.43:54006) 2024-05-27T07:32:04.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.380+0000 7f9911e00700 1 -- 172.21.15.43:0/2005650537 learned_addr learned my addr 172.21.15.43:0/2005650537 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:04.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.380+0000 7f9911e00700 1 -- 172.21.15.43:0/2005650537 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87be9a0 msgr2=0x55bbf87bedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.380+0000 7f9911e00700 1 --2- 172.21.15.43:0/2005650537 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87be9a0 0x55bbf87bedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.380+0000 7f9911e00700 1 -- 172.21.15.43:0/2005650537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 msgr2=0x55bbf87c5310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:32:04.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.380+0000 7f9911e00700 1 --2- 172.21.15.43:0/2005650537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87c5310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.380+0000 7f9911e00700 1 -- 172.21.15.43:0/2005650537 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbf87a0240 con 0x55bbf87159f0 2024-05-27T07:32:04.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.381+0000 7f9911e00700 1 --2- 172.21.15.43:0/2005650537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a88c0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f990c01a3a0 tx=0x7f990c00b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd20b371674ac917 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.381+0000 7f9902a00700 1 -- 172.21.15.43:0/2005650537 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c005270 con 0x55bbf87159f0 2024-05-27T07:32:04.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.381+0000 7f9902a00700 1 -- 172.21.15.43:0/2005650537 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f990c02c070 con 0x55bbf87159f0 2024-05-27T07:32:04.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.381+0000 7f9902a00700 1 -- 172.21.15.43:0/2005650537 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c0039f0 con 0x55bbf87159f0 2024-05-27T07:32:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.382+0000 7f9927a42b80 1 -- 172.21.15.43:0/2005650537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 msgr2=0x55bbf87a88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.382+0000 7f9927a42b80 1 --2- 172.21.15.43:0/2005650537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a88c0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f990c01a3a0 tx=0x7f990c00b740 comp rx=0 tx=0).stop 2024-05-27T07:32:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.382+0000 7f9927a42b80 1 -- 172.21.15.43:0/2005650537 shutdown_connections 2024-05-27T07:32:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.382+0000 7f9927a42b80 1 --2- 172.21.15.43:0/2005650537 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87be9a0 0x55bbf87bedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.383+0000 7f9927a42b80 1 --2- 172.21.15.43:0/2005650537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87a88c0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.383+0000 7f9927a42b80 1 --2- 172.21.15.43:0/2005650537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.383+0000 7f9927a42b80 1 -- 172.21.15.43:0/2005650537 >> 172.21.15.43:0/2005650537 conn(0x55bbf86c3390 msgr2=0x55bbf87a81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.383+0000 7f9927a42b80 1 -- 172.21.15.43:0/2005650537 shutdown_connections 2024-05-27T07:32:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.383+0000 7f9927a42b80 1 -- 172.21.15.43:0/2005650537 wait complete. 2024-05-27T07:32:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.383+0000 7f9927a42b80 1 -- start start 2024-05-27T07:32:04.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.384+0000 7f9927a42b80 1 Processor -- start 2024-05-27T07:32:04.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.384+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.384+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9927a42b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87be9a0 0x55bbf87b6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9927a42b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f99141ad490 con 0x55bbf87159f0 2024-05-27T07:32:04.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9927a42b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f99141ae260 con 0x55bbf87a8e70 2024-05-27T07:32:04.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9927a42b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f99141adb60 con 0x55bbf87be9a0 2024-05-27T07:32:04.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9911e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9927a42b80 10 client.0 _ll_register_callbacks cb 0x55bbf8670c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9911e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87b5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54020/0 (socket says 172.21.15.43:54020) 2024-05-27T07:32:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9911e00700 1 -- 172.21.15.43:0/1475094661 learned_addr learned my addr 172.21.15.43:0/1475094661 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9911400700 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9911e00700 1 -- 172.21.15.43:0/1475094661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87be9a0 msgr2=0x55bbf87b6610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:32:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9911e00700 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87be9a0 0x55bbf87b6610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9911e00700 1 -- 172.21.15.43:0/1475094661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 msgr2=0x55bbf87b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9911e00700 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9911e00700 1 -- 172.21.15.43:0/1475094661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbf87b3560 con 0x55bbf87159f0 2024-05-27T07:32:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.385+0000 7f9911400700 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:04.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.386+0000 7f9911e00700 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87b5b90 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f990c00b370 tx=0x7f990c005310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9640637264e78e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:04.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.386+0000 7f98f7e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:32:04.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.386+0000 7f98f7e00700 1 -- 172.21.15.43:0/1475094661 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c01b880 con 0x55bbf87159f0 2024-05-27T07:32:04.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.386+0000 7f98f7e00700 1 -- 172.21.15.43:0/1475094661 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(146..146 src has 1..146) ==== 9584+0+0 (secure 0 0 0) 0x7f990c003680 con 0x55bbf87159f0 2024-05-27T07:32:04.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.387+0000 7f9927a42b80 1 -- 172.21.15.43:0/1475094661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f990c02c070 con 0x55bbf87159f0 2024-05-27T07:32:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.387+0000 7f98f7e00700 1 -- 172.21.15.43:0/1475094661 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7f990c003880 con 0x55bbf87159f0 2024-05-27T07:32:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.387+0000 7f9927a42b80 10 client.17940 fetch_fsmap learned FSMap version 135 2024-05-27T07:32:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.387+0000 7f9927a42b80 1 -- 172.21.15.43:0/1475094661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=4+}) -- 0x55bbf882caa0 con 0x55bbf87159f0 2024-05-27T07:32:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.388+0000 7f98f7e00700 1 -- 172.21.15.43:0/1475094661 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f990c02c070 con 0x55bbf87159f0 2024-05-27T07:32:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.388+0000 7f98f7e00700 1 -- 172.21.15.43:0/1475094661 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 135) ==== 4195+0+0 (secure 0 0 0) 0x7f990c004af0 con 0x55bbf87159f0 2024-05-27T07:32:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.388+0000 7f9927a42b80 10 client.17940 fetch_fsmap finished waiting for FSMap version 135 2024-05-27T07:32:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.388+0000 7f9927a42b80 10 client.17940 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:32:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.388+0000 7f9927a42b80 20 client.17940 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.388+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bbf882d070 0x55bbf882f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.388+0000 7f9927a42b80 4 client.17940 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:32:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.388+0000 7f9927a42b80 1 -- 172.21.15.43:0/1475094661 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bbf86d2570 con 0x55bbf882d070 2024-05-27T07:32:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.389+0000 7f9911400700 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bbf882d070 0x55bbf882f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.390+0000 7f9911400700 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bbf882d070 0x55bbf882f490 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=615ee3f6b416b4e0 server_cookie=ac9eab15176bf153 in_seq=0 out_seq=0 2024-05-27T07:32:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.390+0000 7f98f7e00700 10 client.17940 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:32:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.390+0000 7f98f7e00700 1 -- 172.21.15.43:0/1475094661 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bbf86d2570 con 0x55bbf882d070 2024-05-27T07:32:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.390+0000 7f98f7e00700 10 client.17940 handle_command_reply: tid=0 2024-05-27T07:32:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.390+0000 7f9927a42b80 1 client.17940 shutdown 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f9927a42b80 1 -- 172.21.15.43:0/1475094661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 msgr2=0x55bbf87b5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87b5b90 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f990c00b370 tx=0x7f990c005310 comp rx=0 tx=0).stop 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f9927a42b80 1 -- 172.21.15.43:0/1475094661 shutdown_connections 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55bbf882d070 0x55bbf882f490 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bbf87be9a0 0x55bbf87b6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bbf87159f0 0x55bbf87b5b90 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f98f7e00700 0 client.17940 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f9927a42b80 1 --2- 172.21.15.43:0/1475094661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55bbf87a8e70 0x55bbf87b60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f9927a42b80 1 -- 172.21.15.43:0/1475094661 >> 172.21.15.43:0/1475094661 conn(0x55bbf86c3390 msgr2=0x55bbf87bd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f9927a42b80 1 -- 172.21.15.43:0/1475094661 shutdown_connections 2024-05-27T07:32:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.391+0000 7f9927a42b80 1 -- 172.21.15.43:0/1475094661 wait complete. 2024-05-27T07:32:04.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.392+0000 7f9927a42b80 20 client.17940 trim_cache size 0 max 16384 2024-05-27T07:32:04.386 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 -- 172.21.15.43:0/2535672926 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98f80b2a70 msgr2=0x7f98f80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98f80b2a70 0x7f98f80b4f80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f99040060b0 tx=0x7f9904015040 comp rx=0 tx=0).stop 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 -- 172.21.15.43:0/2535672926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9914103a00 msgr2=0x7f991419ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9914103a00 0x7f991419ed80 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f98fc000c00 tx=0x7f98fc0057c0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 -- 172.21.15.43:0/2535672926 shutdown_connections 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f98f80b2a70 0x7f98f80b4f80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9914110c60 0x7f9914198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99141030b0 0x7f991419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 --2- 172.21.15.43:0/2535672926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9914103a00 0x7f991419ed80 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 -- 172.21.15.43:0/2535672926 >> 172.21.15.43:0/2535672926 conn(0x7f99140fedb0 msgr2=0x7f9914112930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:04.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 -- 172.21.15.43:0/2535672926 shutdown_connections 2024-05-27T07:32:04.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:04.393+0000 7f9918e00700 1 -- 172.21.15.43:0/2535672926 wait complete. 2024-05-27T07:32:04.786 INFO:tasks.workunit.client.0.smithi043.stdout: 10 27126 5.85 MB/sec execute 125 sec latency 274.055 ms 2024-05-27T07:32:04.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:04 smithi043 ceph-mon[31531]: pgmap v611: 129 pgs: 129 active+clean; 708 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 3.4 MiB/s wr, 123 op/s 2024-05-27T07:32:04.960 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:04 smithi179 ceph-mon[35888]: pgmap v611: 129 pgs: 129 active+clean; 708 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 3.4 MiB/s wr, 123 op/s 2024-05-27T07:32:05.092 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:04 smithi132 ceph-mon[35283]: pgmap v611: 129 pgs: 129 active+clean; 708 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 3.4 MiB/s wr, 123 op/s 2024-05-27T07:32:05.786 INFO:tasks.workunit.client.0.smithi043.stdout: 10 27234 5.85 MB/sec execute 126 sec latency 81.709 ms 2024-05-27T07:32:06.786 INFO:tasks.workunit.client.0.smithi043.stdout: 10 27346 5.84 MB/sec execute 127 sec latency 95.013 ms 2024-05-27T07:32:07.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:07 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:32:07.593 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:07 smithi132 ceph-mon[35283]: pgmap v612: 129 pgs: 129 active+clean; 721 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-05-27T07:32:07.706 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:07 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:32:07.706 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:07 smithi043 ceph-mon[31531]: pgmap v612: 129 pgs: 129 active+clean; 721 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-05-27T07:32:07.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:07 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:32:07.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:07 smithi179 ceph-mon[35888]: pgmap v612: 129 pgs: 129 active+clean; 721 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-05-27T07:32:07.786 INFO:tasks.workunit.client.0.smithi043.stdout: 10 27460 5.84 MB/sec execute 128 sec latency 411.527 ms 2024-05-27T07:32:08.786 INFO:tasks.workunit.client.0.smithi043.stdout: 10 27563 5.83 MB/sec execute 129 sec latency 1411.693 ms 2024-05-27T07:32:09.786 INFO:tasks.workunit.client.0.smithi043.stdout: 10 27614 5.79 MB/sec execute 130 sec latency 2411.824 ms 2024-05-27T07:32:09.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:09 smithi132 ceph-mon[35283]: pgmap v613: 129 pgs: 129 active+clean; 721 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-05-27T07:32:09.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:09 smithi043 ceph-mon[31531]: pgmap v613: 129 pgs: 129 active+clean; 721 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-05-27T07:32:09.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:09 smithi179 ceph-mon[35888]: pgmap v613: 129 pgs: 129 active+clean; 721 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-05-27T07:32:10.786 INFO:tasks.workunit.client.0.smithi043.stdout: 10 27710 5.76 MB/sec execute 131 sec latency 3411.902 ms 2024-05-27T07:32:11.787 INFO:tasks.workunit.client.0.smithi043.stdout: 10 27823 5.77 MB/sec execute 132 sec latency 4411.956 ms 2024-05-27T07:32:11.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:11 smithi132 ceph-mon[35283]: pgmap v614: 129 pgs: 129 active+clean; 721 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-05-27T07:32:11.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:11 smithi132 ceph-mon[35283]: osdmap e147: 12 total, 12 up, 12 in 2024-05-27T07:32:11.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:11 smithi043 ceph-mon[31531]: pgmap v614: 129 pgs: 129 active+clean; 721 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-05-27T07:32:11.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:11 smithi043 ceph-mon[31531]: osdmap e147: 12 total, 12 up, 12 in 2024-05-27T07:32:11.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:11 smithi179 ceph-mon[35888]: pgmap v614: 129 pgs: 129 active+clean; 721 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-05-27T07:32:11.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:11 smithi179 ceph-mon[35888]: osdmap e147: 12 total, 12 up, 12 in 2024-05-27T07:32:12.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:32:12.787 INFO:tasks.workunit.client.0.smithi043.stdout: 10 27925 5.77 MB/sec execute 133 sec latency 5222.368 ms 2024-05-27T07:32:12.962 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:13.755 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:13 smithi043 ceph-mon[31531]: pgmap v616: 129 pgs: 129 active+clean; 731 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.4 MiB/s wr, 188 op/s 2024-05-27T07:32:13.787 INFO:tasks.workunit.client.0.smithi043.stdout: 10 28023 5.74 MB/sec execute 134 sec latency 172.596 ms 2024-05-27T07:32:13.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:13 smithi132 ceph-mon[35283]: pgmap v616: 129 pgs: 129 active+clean; 731 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.4 MiB/s wr, 188 op/s 2024-05-27T07:32:13.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:13 smithi179 ceph-mon[35888]: pgmap v616: 129 pgs: 129 active+clean; 731 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.4 MiB/s wr, 188 op/s 2024-05-27T07:32:14.787 INFO:tasks.workunit.client.0.smithi043.stdout: 10 28129 5.71 MB/sec execute 135 sec latency 85.984 ms 2024-05-27T07:32:15.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.682+0000 7fec35e00700 1 -- 172.21.15.43:0/2242686119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec301092a0 msgr2=0x7fec30117480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.682+0000 7fec35e00700 1 --2- 172.21.15.43:0/2242686119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec301092a0 0x7fec30117480 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fec28009990 tx=0x7fec28009ca0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.682+0000 7fec35e00700 1 -- 172.21.15.43:0/2242686119 shutdown_connections 2024-05-27T07:32:15.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.682+0000 7fec35e00700 1 --2- 172.21.15.43:0/2242686119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec301092a0 0x7fec30117480 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.682+0000 7fec35e00700 1 --2- 172.21.15.43:0/2242686119 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec301030d0 0x7fec301034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.682+0000 7fec35e00700 1 --2- 172.21.15.43:0/2242686119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec30103a20 0x7fec30114f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.682+0000 7fec35e00700 1 -- 172.21.15.43:0/2242686119 >> 172.21.15.43:0/2242686119 conn(0x7fec300fedb0 msgr2=0x7fec301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:15.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.683+0000 7fec35e00700 1 -- 172.21.15.43:0/2242686119 shutdown_connections 2024-05-27T07:32:15.677 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:15 smithi043 ceph-mon[31531]: pgmap v617: 129 pgs: 129 active+clean; 731 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.4 MiB/s wr, 188 op/s 2024-05-27T07:32:15.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.683+0000 7fec35e00700 1 -- 172.21.15.43:0/2242686119 wait complete. 2024-05-27T07:32:15.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.684+0000 7fec35e00700 1 Processor -- start 2024-05-27T07:32:15.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.685+0000 7fec35e00700 1 -- start start 2024-05-27T07:32:15.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.685+0000 7fec35e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec301030d0 0x7fec301a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.685+0000 7fec35e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec30103a20 0x7fec301a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.685+0000 7fec35e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec301092a0 0x7fec301a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.685+0000 7fec35e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fec3011a9d0 con 0x7fec301030d0 2024-05-27T07:32:15.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.685+0000 7fec35e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fec3011a850 con 0x7fec301092a0 2024-05-27T07:32:15.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.685+0000 7fec35e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fec3011ab50 con 0x7fec30103a20 2024-05-27T07:32:15.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.686+0000 7fec2ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec301030d0 0x7fec301a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.686+0000 7fec2ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec301030d0 0x7fec301a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53360/0 (socket says 172.21.15.43:53360) 2024-05-27T07:32:15.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.686+0000 7fec2ea00700 1 -- 172.21.15.43:0/38897345 learned_addr learned my addr 172.21.15.43:0/38897345 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:15.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.686+0000 7fec2f400700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec301092a0 0x7fec301a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.687+0000 7fec26000700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec30103a20 0x7fec301a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.687+0000 7fec2ea00700 1 -- 172.21.15.43:0/38897345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec30103a20 msgr2=0x7fec301a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.687+0000 7fec2ea00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec30103a20 0x7fec301a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.687+0000 7fec2ea00700 1 -- 172.21.15.43:0/38897345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec301092a0 msgr2=0x7fec301a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.687+0000 7fec2ea00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec301092a0 0x7fec301a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.687+0000 7fec2ea00700 1 -- 172.21.15.43:0/38897345 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec301a61f0 con 0x7fec301030d0 2024-05-27T07:32:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.687+0000 7fec2f400700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec301092a0 0x7fec301a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.687+0000 7fec26000700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec30103a20 0x7fec301a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.687+0000 7fec2ea00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec301030d0 0x7fec301a1320 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fec2000ef10 tx=0x7fec2000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.688+0000 7fec34a00700 1 -- 172.21.15.43:0/38897345 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2000cad0 con 0x7fec301030d0 2024-05-27T07:32:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.688+0000 7fec34a00700 1 -- 172.21.15.43:0/38897345 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec20019070 con 0x7fec301030d0 2024-05-27T07:32:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.688+0000 7fec34a00700 1 -- 172.21.15.43:0/38897345 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec20014480 con 0x7fec301030d0 2024-05-27T07:32:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.688+0000 7fec35e00700 1 -- 172.21.15.43:0/38897345 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec301a6480 con 0x7fec301030d0 2024-05-27T07:32:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.688+0000 7fec35e00700 1 -- 172.21.15.43:0/38897345 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec301a6970 con 0x7fec301030d0 2024-05-27T07:32:15.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.695+0000 7fec44998b80 1 Processor -- start 2024-05-27T07:32:15.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.696+0000 7fec44998b80 1 -- start start 2024-05-27T07:32:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.696+0000 7fec34a00700 1 -- 172.21.15.43:0/38897345 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fec20012050 con 0x7fec301030d0 2024-05-27T07:32:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.696+0000 7fec44998b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 0x5626d7b233d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.696+0000 7fec44998b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d79a4e40 0x5626d7b23910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.696+0000 7fec44998b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7b3f110 0x5626d7b41530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.696+0000 7fec44998b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5626d7a51e40 con 0x5626d7b3f110 2024-05-27T07:32:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.696+0000 7fec44998b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5626d7a419e0 con 0x5626d79a4e40 2024-05-27T07:32:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.696+0000 7fec44998b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5626d7a45a20 con 0x5626d7a909f0 2024-05-27T07:32:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.697+0000 7fec26000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7b3f110 0x5626d7b41530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.697+0000 7fec26000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7b3f110 0x5626d7b41530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53362/0 (socket says 172.21.15.43:53362) 2024-05-27T07:32:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.697+0000 7fec26000700 1 -- 172.21.15.43:0/1844387789 learned_addr learned my addr 172.21.15.43:0/1844387789 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.697+0000 7fec2f400700 1 --2- 172.21.15.43:0/1844387789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 0x5626d7b233d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.698+0000 7fec2ea00700 1 --2- 172.21.15.43:0/1844387789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d79a4e40 0x5626d7b23910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.698+0000 7fec2f400700 1 -- 172.21.15.43:0/1844387789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d79a4e40 msgr2=0x5626d7b23910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.698+0000 7fec2f400700 1 --2- 172.21.15.43:0/1844387789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d79a4e40 0x5626d7b23910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.698+0000 7fec2f400700 1 -- 172.21.15.43:0/1844387789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7b3f110 msgr2=0x5626d7b41530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.698+0000 7fec2f400700 1 --2- 172.21.15.43:0/1844387789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7b3f110 0x5626d7b41530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.698+0000 7fec2f400700 1 -- 172.21.15.43:0/1844387789 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626d7a0a420 con 0x5626d7a909f0 2024-05-27T07:32:15.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.698+0000 7fec2ea00700 1 --2- 172.21.15.43:0/1844387789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d79a4e40 0x5626d7b23910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:15.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.698+0000 7fec2f400700 1 --2- 172.21.15.43:0/1844387789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 0x5626d7b233d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fec280098e0 tx=0x7fec28005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5c8e74e9a7373f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.699+0000 7fec34a00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec140b2a70 0x7fec140b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.699+0000 7fec34a00700 1 -- 172.21.15.43:0/38897345 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7fec200e2aa0 con 0x7fec301030d0 2024-05-27T07:32:15.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.699+0000 7fec24c00700 1 -- 172.21.15.43:0/1844387789 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec280055c0 con 0x5626d7a909f0 2024-05-27T07:32:15.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.699+0000 7fec24c00700 1 -- 172.21.15.43:0/1844387789 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec28003b10 con 0x5626d7a909f0 2024-05-27T07:32:15.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.699+0000 7fec24c00700 1 -- 172.21.15.43:0/1844387789 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec280055c0 con 0x5626d7a909f0 2024-05-27T07:32:15.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.700+0000 7fec2ea00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec140b2a70 0x7fec140b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.702+0000 7fec44998b80 1 -- 172.21.15.43:0/1844387789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 msgr2=0x5626d7b233d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.702+0000 7fec44998b80 1 --2- 172.21.15.43:0/1844387789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 0x5626d7b233d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fec280098e0 tx=0x7fec28005040 comp rx=0 tx=0).stop 2024-05-27T07:32:15.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.702+0000 7fec2ea00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec140b2a70 0x7fec140b4f80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x5626d7b1fec0 tx=0x7fec20009eb0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.702+0000 7fec44998b80 1 -- 172.21.15.43:0/1844387789 shutdown_connections 2024-05-27T07:32:15.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.702+0000 7fec44998b80 1 --2- 172.21.15.43:0/1844387789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7b3f110 0x5626d7b41530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.702+0000 7fec44998b80 1 --2- 172.21.15.43:0/1844387789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 0x5626d7b233d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.702+0000 7fec44998b80 1 --2- 172.21.15.43:0/1844387789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d79a4e40 0x5626d7b23910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.702+0000 7fec44998b80 1 -- 172.21.15.43:0/1844387789 >> 172.21.15.43:0/1844387789 conn(0x5626d7a3e390 msgr2=0x5626d7b1f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:15.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.703+0000 7fec44998b80 1 -- 172.21.15.43:0/1844387789 shutdown_connections 2024-05-27T07:32:15.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.703+0000 7fec44998b80 1 -- 172.21.15.43:0/1844387789 wait complete. 2024-05-27T07:32:15.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.703+0000 7fec44998b80 1 -- start start 2024-05-27T07:32:15.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.704+0000 7fec44998b80 1 Processor -- start 2024-05-27T07:32:15.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.704+0000 7fec44998b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d79a4e40 0x5626d7b3c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.704+0000 7fec44998b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7a909f0 0x5626d7b3c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.705+0000 7fec44998b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b3cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.705+0000 7fec44998b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5626d7947fa0 con 0x5626d7a909f0 2024-05-27T07:32:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.705+0000 7fec44998b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5626d79e2760 con 0x5626d7b3f110 2024-05-27T07:32:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.705+0000 7fec44998b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5626d7b41b00 con 0x5626d79a4e40 2024-05-27T07:32:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.705+0000 7fec44998b80 10 client.0 _ll_register_callbacks cb 0x5626d79ebc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.705+0000 7fec2f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b3cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.705+0000 7fec2f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b3cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48334/0 (socket says 172.21.15.43:48334) 2024-05-27T07:32:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.705+0000 7fec2f400700 1 -- 172.21.15.43:0/629457522 learned_addr learned my addr 172.21.15.43:0/629457522 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.705+0000 7fec26000700 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7a909f0 0x5626d7b3c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.705+0000 7fec2f400700 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d79a4e40 0x5626d7b3c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.706+0000 7fec2f400700 1 -- 172.21.15.43:0/629457522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d79a4e40 msgr2=0x5626d7b3c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.706+0000 7fec2f400700 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d79a4e40 0x5626d7b3c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.706+0000 7fec2f400700 1 -- 172.21.15.43:0/629457522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7a909f0 msgr2=0x5626d7b3c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.706+0000 7fec2f400700 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7a909f0 0x5626d7b3c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.706+0000 7fec2f400700 1 -- 172.21.15.43:0/629457522 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626d79918d0 con 0x5626d7b3f110 2024-05-27T07:32:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.707+0000 7fec2f400700 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b3cc10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x5626d7b3ba10 tx=0x7fec2801a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=581db32bab46afb4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.707+0000 7fec1d600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:32:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.707+0000 7fec1d600700 1 -- 172.21.15.43:0/629457522 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec28035060 con 0x5626d7b3f110 2024-05-27T07:32:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.707+0000 7fec1d600700 1 -- 172.21.15.43:0/629457522 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7fec28027040 con 0x5626d7b3f110 2024-05-27T07:32:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.707+0000 7fec44998b80 1 -- 172.21.15.43:0/629457522 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec28003b10 con 0x5626d7b3f110 2024-05-27T07:32:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.708+0000 7fec1d600700 1 -- 172.21.15.43:0/629457522 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=136) ==== 24+0+0 (secure 0 0 0) 0x7fec2800bad0 con 0x5626d7b3f110 2024-05-27T07:32:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.708+0000 7fec44998b80 10 client.37313 fetch_fsmap learned FSMap version 136 2024-05-27T07:32:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.708+0000 7fec44998b80 1 -- 172.21.15.43:0/629457522 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=136,monmap=4+}) -- 0x5626d7a12060 con 0x5626d7b3f110 2024-05-27T07:32:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.708+0000 7fec1d600700 1 -- 172.21.15.43:0/629457522 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec28003b10 con 0x5626d7b3f110 2024-05-27T07:32:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.708+0000 7fec1d600700 1 -- 172.21.15.43:0/629457522 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 136) ==== 4191+0+0 (secure 0 0 0) 0x7fec2800fbb0 con 0x5626d7b3f110 2024-05-27T07:32:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.708+0000 7fec44998b80 10 client.37313 fetch_fsmap finished waiting for FSMap version 136 2024-05-27T07:32:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.708+0000 7fec44998b80 10 client.37313 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.708+0000 7fec44998b80 20 client.37313 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.709+0000 7fec44998b80 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5626d7b4e9c0 0x5626d7b50e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.709+0000 7fec44998b80 4 client.37313 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.709+0000 7fec44998b80 1 -- 172.21.15.43:0/629457522 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626d7a53d90 con 0x5626d7b4e9c0 2024-05-27T07:32:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.709+0000 7fec26000700 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5626d7b4e9c0 0x5626d7b50e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.710+0000 7fec26000700 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5626d7b4e9c0 0x5626d7b50e20 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1e950ebf1dd3df7 server_cookie=7b4a32679b3f90c6 in_seq=0 out_seq=0 2024-05-27T07:32:15.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.710+0000 7fec1d600700 10 client.37313 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:15.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.711+0000 7fec1d600700 1 -- 172.21.15.43:0/629457522 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5626d7a53d90 con 0x5626d7b4e9c0 2024-05-27T07:32:15.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.711+0000 7fec1d600700 10 client.37313 handle_command_reply: tid=0 2024-05-27T07:32:15.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.711+0000 7fec44998b80 1 client.37313 shutdown 2024-05-27T07:32:15.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec44998b80 1 -- 172.21.15.43:0/629457522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 msgr2=0x5626d7b3cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec44998b80 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b3cc10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x5626d7b3ba10 tx=0x7fec2801a040 comp rx=0 tx=0).stop 2024-05-27T07:32:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec44998b80 1 -- 172.21.15.43:0/629457522 shutdown_connections 2024-05-27T07:32:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec44998b80 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b3cc10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec44998b80 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d79a4e40 0x5626d7b3c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec44998b80 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5626d7b4e9c0 0x5626d7b50e20 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec1d600700 0 client.37313 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec44998b80 1 --2- 172.21.15.43:0/629457522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d7a909f0 0x5626d7b3c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec44998b80 1 -- 172.21.15.43:0/629457522 >> 172.21.15.43:0/629457522 conn(0x5626d7a3e390 msgr2=0x5626d7b233d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec44998b80 1 -- 172.21.15.43:0/629457522 shutdown_connections 2024-05-27T07:32:15.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.712+0000 7fec44998b80 1 -- 172.21.15.43:0/629457522 wait complete. 2024-05-27T07:32:15.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.713+0000 7fec44998b80 20 client.37313 trim_cache size 0 max 16384 2024-05-27T07:32:15.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.745+0000 7fec44998b80 1 Processor -- start 2024-05-27T07:32:15.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.746+0000 7fec44998b80 1 -- start start 2024-05-27T07:32:15.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.747+0000 7fec44998b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 0x5626d7b3e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.747+0000 7fec44998b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 0x5626d7b377c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.747+0000 7fec44998b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b4aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.747+0000 7fec44998b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fec301ae7e0 con 0x5626d79a4e40 2024-05-27T07:32:15.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.747+0000 7fec44998b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fec30113c40 con 0x5626d7b3f110 2024-05-27T07:32:15.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.747+0000 7fec44998b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fec301ae110 con 0x5626d7a909f0 2024-05-27T07:32:15.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.748+0000 7fec2f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 0x5626d7b3e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.748+0000 7fec2f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 0x5626d7b3e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53378/0 (socket says 172.21.15.43:53378) 2024-05-27T07:32:15.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.748+0000 7fec2f400700 1 -- 172.21.15.43:0/4071689189 learned_addr learned my addr 172.21.15.43:0/4071689189 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:15.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.748+0000 7fec2f400700 1 --2- 172.21.15.43:0/4071689189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b4aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.749+0000 7fec26000700 1 --2- 172.21.15.43:0/4071689189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 0x5626d7b377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.750+0000 7fec2f400700 1 -- 172.21.15.43:0/4071689189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 msgr2=0x5626d7b377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.750+0000 7fec2f400700 1 --2- 172.21.15.43:0/4071689189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 0x5626d7b377c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.750+0000 7fec2f400700 1 -- 172.21.15.43:0/4071689189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 msgr2=0x5626d7b3e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.750+0000 7fec2f400700 1 --2- 172.21.15.43:0/4071689189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 0x5626d7b3e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.750+0000 7fec2f400700 1 -- 172.21.15.43:0/4071689189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626d7b418a0 con 0x5626d7b3f110 2024-05-27T07:32:15.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.750+0000 7fec26000700 1 --2- 172.21.15.43:0/4071689189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 0x5626d7b377c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:15.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.752+0000 7fec2f400700 1 --2- 172.21.15.43:0/4071689189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b4aba0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec28000c00 tx=0x7fec28009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c830b4524a5d7389 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.754+0000 7fec1fe00700 1 -- 172.21.15.43:0/4071689189 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec28035030 con 0x5626d7b3f110 2024-05-27T07:32:15.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.754+0000 7fec1fe00700 1 -- 172.21.15.43:0/4071689189 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec28003ac0 con 0x5626d7b3f110 2024-05-27T07:32:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.754+0000 7fec1fe00700 1 -- 172.21.15.43:0/4071689189 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec28003e60 con 0x5626d7b3f110 2024-05-27T07:32:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.755+0000 7fec44998b80 1 -- 172.21.15.43:0/4071689189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 msgr2=0x5626d7b4aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.755+0000 7fec44998b80 1 --2- 172.21.15.43:0/4071689189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b4aba0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec28000c00 tx=0x7fec28009510 comp rx=0 tx=0).stop 2024-05-27T07:32:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.757+0000 7fec44998b80 1 -- 172.21.15.43:0/4071689189 shutdown_connections 2024-05-27T07:32:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.757+0000 7fec44998b80 1 --2- 172.21.15.43:0/4071689189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7b3f110 0x5626d7b4aba0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.757+0000 7fec44998b80 1 --2- 172.21.15.43:0/4071689189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 0x5626d7b3e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.757+0000 7fec44998b80 1 --2- 172.21.15.43:0/4071689189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7a909f0 0x5626d7b377c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.757+0000 7fec44998b80 1 -- 172.21.15.43:0/4071689189 >> 172.21.15.43:0/4071689189 conn(0x5626d7a3e390 msgr2=0x5626d7b370b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:15.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.758+0000 7fec44998b80 1 -- 172.21.15.43:0/4071689189 shutdown_connections 2024-05-27T07:32:15.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.758+0000 7fec44998b80 1 -- 172.21.15.43:0/4071689189 wait complete. 2024-05-27T07:32:15.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.760+0000 7fec44998b80 1 -- start start 2024-05-27T07:32:15.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.760+0000 7fec44998b80 1 Processor -- start 2024-05-27T07:32:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.761+0000 7fec44998b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 0x5626d7b3fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.761+0000 7fec44998b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7a909f0 0x5626d7b40060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.761+0000 7fec44998b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7b3f110 0x5626d7b45d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.761+0000 7fec44998b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fec301149e0 con 0x5626d79a4e40 2024-05-27T07:32:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.761+0000 7fec44998b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fec30114310 con 0x5626d7a909f0 2024-05-27T07:32:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.761+0000 7fec44998b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5626d7b1ece0 con 0x5626d7b3f110 2024-05-27T07:32:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.761+0000 7fec44998b80 10 client.0 _ll_register_callbacks cb 0x5626d79ebc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.762+0000 7fec26000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7a909f0 0x5626d7b40060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.762+0000 7fec26000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7a909f0 0x5626d7b40060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48346/0 (socket says 172.21.15.43:48346) 2024-05-27T07:32:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.762+0000 7fec26000700 1 -- 172.21.15.43:0/4292275662 learned_addr learned my addr 172.21.15.43:0/4292275662 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.763+0000 7fec2f400700 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7b3f110 0x5626d7b45d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.763+0000 7fec2f400700 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 0x5626d7b3fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.764+0000 7fec26000700 1 -- 172.21.15.43:0/4292275662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7b3f110 msgr2=0x5626d7b45d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.764+0000 7fec26000700 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7b3f110 0x5626d7b45d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.764+0000 7fec26000700 1 -- 172.21.15.43:0/4292275662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 msgr2=0x5626d7b3fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:15.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.764+0000 7fec26000700 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 0x5626d7b3fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.764+0000 7fec26000700 1 -- 172.21.15.43:0/4292275662 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626d7b462d0 con 0x5626d7a909f0 2024-05-27T07:32:15.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.764+0000 7fec2f400700 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 0x5626d7b3fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:15.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.765+0000 7fec26000700 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7a909f0 0x5626d7b40060 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x5626d7b515c0 tx=0x7fec18016710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1d000f356a027d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:15.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.766+0000 7fec1d600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:32:15.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.766+0000 7fec1d600700 1 -- 172.21.15.43:0/4292275662 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1800aea0 con 0x5626d7a909f0 2024-05-27T07:32:15.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.767+0000 7fec44998b80 1 -- 172.21.15.43:0/4292275662 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec28003ac0 con 0x5626d7a909f0 2024-05-27T07:32:15.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.767+0000 7fec1d600700 1 -- 172.21.15.43:0/4292275662 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7fec1801c240 con 0x5626d7a909f0 2024-05-27T07:32:15.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.768+0000 7fec1d600700 1 -- 172.21.15.43:0/4292275662 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=136) ==== 24+0+0 (secure 0 0 0) 0x7fec1802e050 con 0x5626d7a909f0 2024-05-27T07:32:15.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.768+0000 7fec44998b80 10 client.37319 fetch_fsmap learned FSMap version 136 2024-05-27T07:32:15.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.768+0000 7fec44998b80 1 -- 172.21.15.43:0/4292275662 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=136,monmap=4+}) -- 0x5626d7b47590 con 0x5626d7a909f0 2024-05-27T07:32:15.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.769+0000 7fec1d600700 1 -- 172.21.15.43:0/4292275662 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec28003ac0 con 0x5626d7a909f0 2024-05-27T07:32:15.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.769+0000 7fec1d600700 1 -- 172.21.15.43:0/4292275662 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 136) ==== 4191+0+0 (secure 0 0 0) 0x7fec18025710 con 0x5626d7a909f0 2024-05-27T07:32:15.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.769+0000 7fec44998b80 10 client.37319 fetch_fsmap finished waiting for FSMap version 136 2024-05-27T07:32:15.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.769+0000 7fec44998b80 10 client.37319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:15.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.769+0000 7fec44998b80 20 client.37319 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:15.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.769+0000 7fec44998b80 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5626d7bb9590 0x5626d7bbb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:15.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.769+0000 7fec44998b80 4 client.37319 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:32:15.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.769+0000 7fec44998b80 1 -- 172.21.15.43:0/4292275662 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5626d7a4d570 con 0x5626d7bb9590 2024-05-27T07:32:15.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.771+0000 7fec2f400700 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5626d7bb9590 0x5626d7bbb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:15.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.772+0000 7fec2f400700 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5626d7bb9590 0x5626d7bbb9b0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97e59dee68362da server_cookie=ef72f1ae08e9ae0c in_seq=0 out_seq=0 2024-05-27T07:32:15.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.772+0000 7fec1d600700 10 client.37319 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:15.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.774+0000 7fec1d600700 1 -- 172.21.15.43:0/4292275662 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5626d7a4d570 con 0x5626d7bb9590 2024-05-27T07:32:15.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.774+0000 7fec1d600700 10 client.37319 handle_command_reply: tid=0 2024-05-27T07:32:15.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.774+0000 7fec44998b80 1 client.37319 shutdown 2024-05-27T07:32:15.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.775+0000 7fec44998b80 1 -- 172.21.15.43:0/4292275662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7a909f0 msgr2=0x5626d7b40060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.775+0000 7fec44998b80 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7a909f0 0x5626d7b40060 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x5626d7b515c0 tx=0x7fec18016710 comp rx=0 tx=0).stop 2024-05-27T07:32:15.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.775+0000 7fec44998b80 1 -- 172.21.15.43:0/4292275662 shutdown_connections 2024-05-27T07:32:15.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.775+0000 7fec44998b80 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5626d7b3f110 0x5626d7b45d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.775+0000 7fec44998b80 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5626d7bb9590 0x5626d7bbb9b0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.775+0000 7fec44998b80 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5626d79a4e40 0x5626d7b3fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.775+0000 7fec44998b80 1 --2- 172.21.15.43:0/4292275662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5626d7a909f0 0x5626d7b40060 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.775+0000 7fec44998b80 1 -- 172.21.15.43:0/4292275662 >> 172.21.15.43:0/4292275662 conn(0x5626d7a3e390 msgr2=0x5626d7b3aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:15.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.775+0000 7fec1d600700 0 client.37319 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:15.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.776+0000 7fec44998b80 1 -- 172.21.15.43:0/4292275662 shutdown_connections 2024-05-27T07:32:15.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.777+0000 7fec44998b80 1 -- 172.21.15.43:0/4292275662 wait complete. 2024-05-27T07:32:15.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.777+0000 7fec44998b80 20 client.37319 trim_cache size 0 max 16384 2024-05-27T07:32:15.771 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:32:15.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.779+0000 7fec35e00700 1 -- 172.21.15.43:0/38897345 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec140b2a70 msgr2=0x7fec140b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.779+0000 7fec35e00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec140b2a70 0x7fec140b4f80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x5626d7b1fec0 tx=0x7fec20009eb0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.779+0000 7fec35e00700 1 -- 172.21.15.43:0/38897345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec301030d0 msgr2=0x7fec301a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:15.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.779+0000 7fec35e00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec301030d0 0x7fec301a1320 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fec2000ef10 tx=0x7fec2000c3a0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.779+0000 7fec35e00700 1 -- 172.21.15.43:0/38897345 shutdown_connections 2024-05-27T07:32:15.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.779+0000 7fec35e00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fec140b2a70 0x7fec140b4f80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.779+0000 7fec35e00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec301092a0 0x7fec301a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.779+0000 7fec35e00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec301030d0 0x7fec301a1320 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.779+0000 7fec35e00700 1 --2- 172.21.15.43:0/38897345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec30103a20 0x7fec301a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:15.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.779+0000 7fec35e00700 1 -- 172.21.15.43:0/38897345 >> 172.21.15.43:0/38897345 conn(0x7fec300fedb0 msgr2=0x7fec30116470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:15.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.780+0000 7fec35e00700 1 -- 172.21.15.43:0/38897345 shutdown_connections 2024-05-27T07:32:15.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:15.780+0000 7fec35e00700 1 -- 172.21.15.43:0/38897345 wait complete. 2024-05-27T07:32:15.789 INFO:tasks.workunit.client.0.smithi043.stdout: 10 28247 5.73 MB/sec execute 136 sec latency 110.187 ms 2024-05-27T07:32:15.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:15 smithi179 ceph-mon[35888]: pgmap v617: 129 pgs: 129 active+clean; 731 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.4 MiB/s wr, 188 op/s 2024-05-27T07:32:16.000 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:15 smithi132 ceph-mon[35283]: pgmap v617: 129 pgs: 129 active+clean; 731 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.4 MiB/s wr, 188 op/s 2024-05-27T07:32:16.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 28371 5.73 MB/sec execute 137 sec latency 104.494 ms 2024-05-27T07:32:17.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 28468 5.70 MB/sec execute 138 sec latency 329.565 ms 2024-05-27T07:32:17.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:17 smithi043 ceph-mon[31531]: pgmap v618: 129 pgs: 129 active+clean; 738 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-27T07:32:17.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:17 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:32:17.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:17 smithi179 ceph-mon[35888]: pgmap v618: 129 pgs: 129 active+clean; 738 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-27T07:32:17.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:17 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:32:18.063 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:32:18.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:32:18.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:17 smithi132 ceph-mon[35283]: pgmap v618: 129 pgs: 129 active+clean; 738 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-27T07:32:18.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:17 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:32:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:18.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 28590 5.71 MB/sec execute 139 sec latency 87.848 ms 2024-05-27T07:32:19.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 28723 5.73 MB/sec execute 140 sec latency 97.246 ms 2024-05-27T07:32:19.866 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:19 smithi043 ceph-mon[31531]: pgmap v619: 129 pgs: 129 active+clean; 738 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-27T07:32:19.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:19 smithi179 ceph-mon[35888]: pgmap v619: 129 pgs: 129 active+clean; 738 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-27T07:32:20.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:19 smithi132 ceph-mon[35283]: pgmap v619: 129 pgs: 129 active+clean; 738 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-27T07:32:20.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 28863 5.75 MB/sec execute 141 sec latency 206.319 ms 2024-05-27T07:32:20.865 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:20 smithi043 ceph-mon[31531]: pgmap v620: 129 pgs: 129 active+clean; 738 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-27T07:32:20.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:20 smithi179 ceph-mon[35888]: pgmap v620: 129 pgs: 129 active+clean; 738 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-27T07:32:21.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.194+0000 7fb3f9a00700 1 -- 172.21.15.43:0/4107646534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3f4105de0 msgr2=0x7fb3f410e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.194+0000 7fb3f9a00700 1 --2- 172.21.15.43:0/4107646534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3f4105de0 0x7fb3f410e330 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0098f0 tx=0x7fb3dc009c00 comp rx=0 tx=0).stop 2024-05-27T07:32:21.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.194+0000 7fb3f9a00700 1 -- 172.21.15.43:0/4107646534 shutdown_connections 2024-05-27T07:32:21.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.194+0000 7fb3f9a00700 1 --2- 172.21.15.43:0/4107646534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3f410e900 0x7fb3f4110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.194+0000 7fb3f9a00700 1 --2- 172.21.15.43:0/4107646534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3f4105490 0x7fb3f41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.194+0000 7fb3f9a00700 1 --2- 172.21.15.43:0/4107646534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3f4105de0 0x7fb3f410e330 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.194+0000 7fb3f9a00700 1 -- 172.21.15.43:0/4107646534 >> 172.21.15.43:0/4107646534 conn(0x7fb3f40fedb0 msgr2=0x7fb3f41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:21.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.194+0000 7fb3f9a00700 1 -- 172.21.15.43:0/4107646534 shutdown_connections 2024-05-27T07:32:21.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.195+0000 7fb3f9a00700 1 -- 172.21.15.43:0/4107646534 wait complete. 2024-05-27T07:32:21.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.196+0000 7fb3f9a00700 1 Processor -- start 2024-05-27T07:32:21.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.196+0000 7fb3f9a00700 1 -- start start 2024-05-27T07:32:21.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.197+0000 7fb3f9a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3f4105490 0x7fb3f419ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.197+0000 7fb3f9a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3f4105de0 0x7fb3f419b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.197+0000 7fb3f9a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3f410e900 0x7fb3f419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.197+0000 7fb3f9a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f4114340 con 0x7fb3f4105490 2024-05-27T07:32:21.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.197+0000 7fb3f9a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f41141c0 con 0x7fb3f410e900 2024-05-27T07:32:21.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.197+0000 7fb3f9a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f41144c0 con 0x7fb3f4105de0 2024-05-27T07:32:21.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.198+0000 7fb3f2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3f4105490 0x7fb3f419ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.198+0000 7fb3f2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3f4105490 0x7fb3f419ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53394/0 (socket says 172.21.15.43:53394) 2024-05-27T07:32:21.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.198+0000 7fb3f2a00700 1 -- 172.21.15.43:0/3691777034 learned_addr learned my addr 172.21.15.43:0/3691777034 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:21.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.198+0000 7fb3ea000700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3f4105de0 0x7fb3f419b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.198+0000 7fb3f3400700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3f410e900 0x7fb3f419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.198+0000 7fb3f2a00700 1 -- 172.21.15.43:0/3691777034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3f4105de0 msgr2=0x7fb3f419b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.198+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3f4105de0 0x7fb3f419b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.198+0000 7fb3f2a00700 1 -- 172.21.15.43:0/3691777034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3f410e900 msgr2=0x7fb3f419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.198+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3f410e900 0x7fb3f419f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.199+0000 7fb3f2a00700 1 -- 172.21.15.43:0/3691777034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f419fd20 con 0x7fb3f4105490 2024-05-27T07:32:21.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.199+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3f4105490 0x7fb3f419ae50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400ef10 tx=0x7fb3e400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:21.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.200+0000 7fb3ebe00700 1 -- 172.21.15.43:0/3691777034 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e400cad0 con 0x7fb3f4105490 2024-05-27T07:32:21.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.200+0000 7fb3ebe00700 1 -- 172.21.15.43:0/3691777034 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3e4019070 con 0x7fb3f4105490 2024-05-27T07:32:21.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.200+0000 7fb3ebe00700 1 -- 172.21.15.43:0/3691777034 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4014480 con 0x7fb3f4105490 2024-05-27T07:32:21.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.200+0000 7fb3f9a00700 1 -- 172.21.15.43:0/3691777034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3f419ffb0 con 0x7fb3f4105490 2024-05-27T07:32:21.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.200+0000 7fb3f9a00700 1 -- 172.21.15.43:0/3691777034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3f41a0470 con 0x7fb3f4105490 2024-05-27T07:32:21.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.206+0000 7fb3ebe00700 1 -- 172.21.15.43:0/3691777034 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb3e4012050 con 0x7fb3f4105490 2024-05-27T07:32:21.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.206+0000 7fb4084b5b80 1 Processor -- start 2024-05-27T07:32:21.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.207+0000 7fb4084b5b80 1 -- start start 2024-05-27T07:32:21.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.207+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.207+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.207+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435d45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.207+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559a434edec0 con 0x559a435d2190 2024-05-27T07:32:21.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.207+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559a434dda60 con 0x559a435c9660 2024-05-27T07:32:21.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.207+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x559a434e1aa0 con 0x559a43440ec0 2024-05-27T07:32:21.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.208+0000 7fb3f2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.208+0000 7fb3f3400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.208+0000 7fb3ea000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435d45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.208+0000 7fb3f3400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435c9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:38914/0 (socket says 172.21.15.43:38914) 2024-05-27T07:32:21.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.208+0000 7fb3f3400700 1 -- 172.21.15.43:0/4247760350 learned_addr learned my addr 172.21.15.43:0/4247760350 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:21.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.209+0000 7fb3ea000700 1 -- 172.21.15.43:0/4247760350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 msgr2=0x559a435c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:21.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.209+0000 7fb3ea000700 1 --2- 172.21.15.43:0/4247760350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.209+0000 7fb3ebe00700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3d80b2b40 0x7fb3d80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.209+0000 7fb3ea000700 1 -- 172.21.15.43:0/4247760350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 msgr2=0x559a435c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:21.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.209+0000 7fb3ea000700 1 --2- 172.21.15.43:0/4247760350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435c9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.209+0000 7fb3ea000700 1 -- 172.21.15.43:0/4247760350 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a434a64a0 con 0x559a435d2190 2024-05-27T07:32:21.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.209+0000 7fb3f3400700 1 --2- 172.21.15.43:0/4247760350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:21.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.209+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/4247760350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:21.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.209+0000 7fb3ebe00700 1 -- 172.21.15.43:0/3691777034 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7fb3e40e29e0 con 0x7fb3f4105490 2024-05-27T07:32:21.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.210+0000 7fb3f3400700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3d80b2b40 0x7fb3d80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.210+0000 7fb3ea000700 1 --2- 172.21.15.43:0/4247760350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435d45b0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc000c00 tx=0x7fb3dc009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2195a1c5473554 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:21.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.211+0000 7fb3e3e00700 1 -- 172.21.15.43:0/4247760350 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc027070 con 0x559a435d2190 2024-05-27T07:32:21.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.211+0000 7fb3e3e00700 1 -- 172.21.15.43:0/4247760350 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3dc00f8f0 con 0x559a435d2190 2024-05-27T07:32:21.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.211+0000 7fb3e3e00700 1 -- 172.21.15.43:0/4247760350 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc0048d0 con 0x559a435d2190 2024-05-27T07:32:21.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.212+0000 7fb4084b5b80 1 -- 172.21.15.43:0/4247760350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 msgr2=0x559a435d45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.212+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/4247760350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435d45b0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc000c00 tx=0x7fb3dc009630 comp rx=0 tx=0).stop 2024-05-27T07:32:21.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.212+0000 7fb3f3400700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3d80b2b40 0x7fb3d80b5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009750 tx=0x7fb3ec009310 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:21.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.212+0000 7fb4084b5b80 1 -- 172.21.15.43:0/4247760350 shutdown_connections 2024-05-27T07:32:21.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.212+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/4247760350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435d45b0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.212+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/4247760350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.212+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/4247760350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.212+0000 7fb4084b5b80 1 -- 172.21.15.43:0/4247760350 >> 172.21.15.43:0/4247760350 conn(0x559a434a1040 msgr2=0x559a435baaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:21.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.212+0000 7fb4084b5b80 1 -- 172.21.15.43:0/4247760350 shutdown_connections 2024-05-27T07:32:21.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.212+0000 7fb4084b5b80 1 -- 172.21.15.43:0/4247760350 wait complete. 2024-05-27T07:32:21.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.213+0000 7fb4084b5b80 1 -- start start 2024-05-27T07:32:21.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.213+0000 7fb4084b5b80 1 Processor -- start 2024-05-27T07:32:21.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559a433e4020 con 0x559a435d2190 2024-05-27T07:32:21.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559a4347e7e0 con 0x559a435c9660 2024-05-27T07:32:21.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x559a435d5040 con 0x559a43440ec0 2024-05-27T07:32:21.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb4084b5b80 10 client.0 _ll_register_callbacks cb 0x559a43487cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:21.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb3ea000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb3f3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb3f3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435cfbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53416/0 (socket says 172.21.15.43:53416) 2024-05-27T07:32:21.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb3f3400700 1 -- 172.21.15.43:0/1148812012 learned_addr learned my addr 172.21.15.43:0/1148812012 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:21.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb3ea000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48376/0 (socket says 172.21.15.43:48376) 2024-05-27T07:32:21.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.214+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.215+0000 7fb3ea000700 1 -- 172.21.15.43:0/1148812012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 msgr2=0x559a435cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:21.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.215+0000 7fb3ea000700 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.215+0000 7fb3ea000700 1 -- 172.21.15.43:0/1148812012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 msgr2=0x559a435cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:21.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.215+0000 7fb3ea000700 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.215+0000 7fb3ea000700 1 -- 172.21.15.43:0/1148812012 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a4342d950 con 0x559a435c9660 2024-05-27T07:32:21.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.215+0000 7fb3f3400700 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:21.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.216+0000 7fb3ea000700 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435cf690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009810 tx=0x7fb3dc00fed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a08cf5008a20c585 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:21.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.216+0000 7fb3e0c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:32:21.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.216+0000 7fb3e0c00700 1 -- 172.21.15.43:0/1148812012 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc027040 con 0x559a435c9660 2024-05-27T07:32:21.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.216+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1148812012 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3dc00f8f0 con 0x559a435c9660 2024-05-27T07:32:21.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.217+0000 7fb3e0c00700 1 -- 172.21.15.43:0/1148812012 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7fb3dc0053c0 con 0x559a435c9660 2024-05-27T07:32:21.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.217+0000 7fb3e0c00700 1 -- 172.21.15.43:0/1148812012 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7fb3dc0055c0 con 0x559a435c9660 2024-05-27T07:32:21.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.217+0000 7fb4084b5b80 10 client.37331 fetch_fsmap learned FSMap version 137 2024-05-27T07:32:21.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.217+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1148812012 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=4+}) -- 0x559a434ae0e0 con 0x559a435c9660 2024-05-27T07:32:21.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.218+0000 7fb3e0c00700 1 -- 172.21.15.43:0/1148812012 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3dc00f8f0 con 0x559a435c9660 2024-05-27T07:32:21.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.218+0000 7fb3e0c00700 1 -- 172.21.15.43:0/1148812012 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 137) ==== 4187+0+0 (secure 0 0 0) 0x7fb3dc02a050 con 0x559a435c9660 2024-05-27T07:32:21.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.218+0000 7fb4084b5b80 10 client.37331 fetch_fsmap finished waiting for FSMap version 137 2024-05-27T07:32:21.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.218+0000 7fb4084b5b80 10 client.37331 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:21.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.218+0000 7fb4084b5b80 20 client.37331 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:21.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.218+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559a435e2040 0x559a435e4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.218+0000 7fb4084b5b80 4 client.37331 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:21.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.218+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1148812012 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a434efe10 con 0x559a435e2040 2024-05-27T07:32:21.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.219+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559a435e2040 0x559a435e4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.220+0000 7fb3e0c00700 10 client.37331 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:21.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.220+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559a435e2040 0x559a435e4490 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=222a815620790b34 server_cookie=7f11ca19d591841a in_seq=0 out_seq=0 2024-05-27T07:32:21.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.221+0000 7fb3e0c00700 1 -- 172.21.15.43:0/1148812012 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x559a434efe10 con 0x559a435e2040 2024-05-27T07:32:21.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.221+0000 7fb3e0c00700 10 client.37331 handle_command_reply: tid=0 2024-05-27T07:32:21.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.221+0000 7fb4084b5b80 1 client.37331 shutdown 2024-05-27T07:32:21.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.221+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1148812012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 msgr2=0x559a435cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.221+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435cf690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009810 tx=0x7fb3dc00fed0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.221+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1148812012 shutdown_connections 2024-05-27T07:32:21.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.221+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559a435e2040 0x559a435e4490 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.221+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.221+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.221+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/1148812012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435cf690 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.222+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1148812012 >> 172.21.15.43:0/1148812012 conn(0x559a434a1040 msgr2=0x559a435ba680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:21.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.222+0000 7fb3e0c00700 0 client.37331 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:21.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.222+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1148812012 shutdown_connections 2024-05-27T07:32:21.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.222+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1148812012 wait complete. 2024-05-27T07:32:21.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.222+0000 7fb4084b5b80 20 client.37331 trim_cache size 0 max 16384 2024-05-27T07:32:21.224 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:20 smithi132 ceph-mon[35283]: pgmap v620: 129 pgs: 129 active+clean; 738 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-05-27T07:32:21.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.254+0000 7fb4084b5b80 1 Processor -- start 2024-05-27T07:32:21.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.255+0000 7fb4084b5b80 1 -- start start 2024-05-27T07:32:21.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.255+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435d15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.255+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435de0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.255+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435e9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.255+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f41ae680 con 0x559a435d2190 2024-05-27T07:32:21.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.255+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f41acb40 con 0x559a435c9660 2024-05-27T07:32:21.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.255+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f41adfb0 con 0x559a43440ec0 2024-05-27T07:32:21.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.256+0000 7fb3ea000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435de0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.256+0000 7fb3ea000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435de0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48384/0 (socket says 172.21.15.43:48384) 2024-05-27T07:32:21.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.256+0000 7fb3ea000700 1 -- 172.21.15.43:0/1435241669 learned_addr learned my addr 172.21.15.43:0/1435241669 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:21.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.256+0000 7fb3f3400700 1 --2- 172.21.15.43:0/1435241669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435e9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.256+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/1435241669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435d15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.256+0000 7fb3ea000700 1 -- 172.21.15.43:0/1435241669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 msgr2=0x559a435d15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.256+0000 7fb3ea000700 1 --2- 172.21.15.43:0/1435241669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435d15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.256+0000 7fb3ea000700 1 -- 172.21.15.43:0/1435241669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 msgr2=0x559a435e9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.256+0000 7fb3ea000700 1 --2- 172.21.15.43:0/1435241669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435e9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.256+0000 7fb3ea000700 1 -- 172.21.15.43:0/1435241669 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a435d0210 con 0x559a435c9660 2024-05-27T07:32:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.257+0000 7fb3ea000700 1 --2- 172.21.15.43:0/1435241669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435de0e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00b830 tx=0x7fb3dc00bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d7cc04adc775d2c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.257+0000 7fb3e3400700 1 -- 172.21.15.43:0/1435241669 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc027040 con 0x559a435c9660 2024-05-27T07:32:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.257+0000 7fb3e3400700 1 -- 172.21.15.43:0/1435241669 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3dc00baa0 con 0x559a435c9660 2024-05-27T07:32:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.257+0000 7fb3e3400700 1 -- 172.21.15.43:0/1435241669 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc02b780 con 0x559a435c9660 2024-05-27T07:32:21.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.258+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1435241669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 msgr2=0x559a435de0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.258+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/1435241669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435de0e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00b830 tx=0x7fb3dc00bf40 comp rx=0 tx=0).stop 2024-05-27T07:32:21.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.258+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1435241669 shutdown_connections 2024-05-27T07:32:21.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.258+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/1435241669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a435d2190 0x559a435e9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.258+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/1435241669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a43440ec0 0x559a435d15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.258+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/1435241669 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435c9660 0x559a435de0e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.259+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1435241669 >> 172.21.15.43:0/1435241669 conn(0x559a434a1040 msgr2=0x559a435cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:21.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.259+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1435241669 shutdown_connections 2024-05-27T07:32:21.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.259+0000 7fb4084b5b80 1 -- 172.21.15.43:0/1435241669 wait complete. 2024-05-27T07:32:21.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.260+0000 7fb4084b5b80 1 -- start start 2024-05-27T07:32:21.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.260+0000 7fb4084b5b80 1 Processor -- start 2024-05-27T07:32:21.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.261+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a43440ec0 0x559a43645e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.261+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a435c9660 0x559a43646380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.261+0000 7fb4084b5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435d2190 0x559a435d67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.261+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f41ad8e0 con 0x559a43440ec0 2024-05-27T07:32:21.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.261+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f41ad210 con 0x559a435d2190 2024-05-27T07:32:21.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.261+0000 7fb4084b5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x559a435bbe80 con 0x559a435c9660 2024-05-27T07:32:21.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.261+0000 7fb4084b5b80 10 client.0 _ll_register_callbacks cb 0x559a43487cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:21.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.261+0000 7fb3f3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435d2190 0x559a435d67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.262+0000 7fb3f3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435d2190 0x559a435d67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48392/0 (socket says 172.21.15.43:48392) 2024-05-27T07:32:21.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.262+0000 7fb3f3400700 1 -- 172.21.15.43:0/325389702 learned_addr learned my addr 172.21.15.43:0/325389702 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:21.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.262+0000 7fb3ea000700 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a435c9660 0x559a43646380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.263+0000 7fb3f3400700 1 -- 172.21.15.43:0/325389702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a435c9660 msgr2=0x559a43646380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:21.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.263+0000 7fb3f3400700 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a435c9660 0x559a43646380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.263+0000 7fb3f3400700 1 -- 172.21.15.43:0/325389702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a43440ec0 msgr2=0x559a43645e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:21.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.262+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a43440ec0 0x559a43645e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.263+0000 7fb3f3400700 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a43440ec0 0x559a43645e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.263+0000 7fb3f3400700 1 -- 172.21.15.43:0/325389702 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a435d6e30 con 0x559a435d2190 2024-05-27T07:32:21.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.263+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a43440ec0 0x559a43645e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:21.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.263+0000 7fb3f3400700 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435d2190 0x559a435d67f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec020d30 tx=0x7fb3ec004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f511dffe7ccdc895 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:21.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.263+0000 7fb3e0c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:32:21.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.264+0000 7fb3e0c00700 1 -- 172.21.15.43:0/325389702 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec02f060 con 0x559a435d2190 2024-05-27T07:32:21.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.264+0000 7fb4084b5b80 1 -- 172.21.15.43:0/325389702 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3dc00baa0 con 0x559a435d2190 2024-05-27T07:32:21.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.264+0000 7fb3e0c00700 1 -- 172.21.15.43:0/325389702 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7fb3ec02a440 con 0x559a435d2190 2024-05-27T07:32:21.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.265+0000 7fb3e0c00700 1 -- 172.21.15.43:0/325389702 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7fb3ec01ee40 con 0x559a435d2190 2024-05-27T07:32:21.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.265+0000 7fb4084b5b80 10 client.37337 fetch_fsmap learned FSMap version 137 2024-05-27T07:32:21.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.265+0000 7fb4084b5b80 1 -- 172.21.15.43:0/325389702 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=4+}) -- 0x559a4364d600 con 0x559a435d2190 2024-05-27T07:32:21.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.266+0000 7fb3e0c00700 1 -- 172.21.15.43:0/325389702 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb3dc00baa0 con 0x559a435d2190 2024-05-27T07:32:21.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.266+0000 7fb3e0c00700 1 -- 172.21.15.43:0/325389702 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 137) ==== 4187+0+0 (secure 0 0 0) 0x7fb3ec0053b0 con 0x559a435d2190 2024-05-27T07:32:21.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.266+0000 7fb4084b5b80 10 client.37337 fetch_fsmap finished waiting for FSMap version 137 2024-05-27T07:32:21.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.266+0000 7fb4084b5b80 10 client.37337 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:21.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.266+0000 7fb4084b5b80 20 client.37337 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:21.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.266+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559a4364dcf0 0x559a43650110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:21.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.267+0000 7fb4084b5b80 4 client.37337 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:32:21.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.267+0000 7fb4084b5b80 1 -- 172.21.15.43:0/325389702 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559a434e95f0 con 0x559a4364dcf0 2024-05-27T07:32:21.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.267+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559a4364dcf0 0x559a43650110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:21.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.270+0000 7fb3f2a00700 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559a4364dcf0 0x559a43650110 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6de429f985aaff7 server_cookie=3aa5054365679b1 in_seq=0 out_seq=0 2024-05-27T07:32:21.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.270+0000 7fb3e0c00700 10 client.37337 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:21.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb3e0c00700 1 -- 172.21.15.43:0/325389702 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559a434e95f0 con 0x559a4364dcf0 2024-05-27T07:32:21.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb3e0c00700 10 client.37337 handle_command_reply: tid=0 2024-05-27T07:32:21.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb4084b5b80 1 client.37337 shutdown 2024-05-27T07:32:21.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb4084b5b80 1 -- 172.21.15.43:0/325389702 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435d2190 msgr2=0x559a435d67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435d2190 0x559a435d67f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec020d30 tx=0x7fb3ec004750 comp rx=0 tx=0).stop 2024-05-27T07:32:21.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb4084b5b80 1 -- 172.21.15.43:0/325389702 shutdown_connections 2024-05-27T07:32:21.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559a435d2190 0x559a435d67f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x559a4364dcf0 0x559a43650110 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a43440ec0 0x559a43645e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb4084b5b80 1 --2- 172.21.15.43:0/325389702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559a435c9660 0x559a43646380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.273+0000 7fb4084b5b80 1 -- 172.21.15.43:0/325389702 >> 172.21.15.43:0/325389702 conn(0x559a434a1040 msgr2=0x559a435c9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:21.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.274+0000 7fb3e0c00700 0 client.37337 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:21.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.274+0000 7fb4084b5b80 1 -- 172.21.15.43:0/325389702 shutdown_connections 2024-05-27T07:32:21.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.274+0000 7fb4084b5b80 1 -- 172.21.15.43:0/325389702 wait complete. 2024-05-27T07:32:21.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.274+0000 7fb4084b5b80 20 client.37337 trim_cache size 0 max 16384 2024-05-27T07:32:21.268 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:32:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:32:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "839c093e-ddcb-420b-9e0d-ec0a33b0fd49", 2024-05-27T07:32:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:32:21.269 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:32:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.275+0000 7fb3f9a00700 1 -- 172.21.15.43:0/3691777034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3d80b2b40 msgr2=0x7fb3d80b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.275+0000 7fb3f9a00700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3d80b2b40 0x7fb3d80b5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009750 tx=0x7fb3ec009310 comp rx=0 tx=0).stop 2024-05-27T07:32:21.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.275+0000 7fb3f9a00700 1 -- 172.21.15.43:0/3691777034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3f4105490 msgr2=0x7fb3f419ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:21.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.275+0000 7fb3f9a00700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3f4105490 0x7fb3f419ae50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400ef10 tx=0x7fb3e400c3a0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.275+0000 7fb3f9a00700 1 -- 172.21.15.43:0/3691777034 shutdown_connections 2024-05-27T07:32:21.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.275+0000 7fb3f9a00700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb3d80b2b40 0x7fb3d80b5050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.276+0000 7fb3f9a00700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3f410e900 0x7fb3f419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.276+0000 7fb3f9a00700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3f4105490 0x7fb3f419ae50 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.276+0000 7fb3f9a00700 1 --2- 172.21.15.43:0/3691777034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3f4105de0 0x7fb3f419b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:21.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.276+0000 7fb3f9a00700 1 -- 172.21.15.43:0/3691777034 >> 172.21.15.43:0/3691777034 conn(0x7fb3f40fedb0 msgr2=0x7fb3f4100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:21.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.276+0000 7fb3f9a00700 1 -- 172.21.15.43:0/3691777034 shutdown_connections 2024-05-27T07:32:21.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:21.276+0000 7fb3f9a00700 1 -- 172.21.15.43:0/3691777034 wait complete. 2024-05-27T07:32:21.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 28978 5.73 MB/sec execute 142 sec latency 219.672 ms 2024-05-27T07:32:21.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:21 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:32:21.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:21 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:21 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:32:21.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:21 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:32:22.463 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:21 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:32:22.464 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:21 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:22.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 29089 5.75 MB/sec execute 143 sec latency 386.496 ms 2024-05-27T07:32:22.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:32:22.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:32:22.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:32:22.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:32:22.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:32:22.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:32:22.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:22 smithi043 ceph-mon[31531]: pgmap v621: 129 pgs: 129 active+clean; 747 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-05-27T07:32:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:32:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:32:22.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:32:22.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:32:22.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:32:22.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:32:22.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:22.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:22 smithi179 ceph-mon[35888]: pgmap v621: 129 pgs: 129 active+clean; 747 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-05-27T07:32:23.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:32:23.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:32:23.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:22 smithi132 ceph-mon[35283]: pgmap v621: 129 pgs: 129 active+clean; 747 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-05-27T07:32:23.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 29206 5.76 MB/sec execute 144 sec latency 82.816 ms 2024-05-27T07:32:24.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 29357 5.79 MB/sec execute 145 sec latency 86.222 ms 2024-05-27T07:32:25.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.150+0000 7fb4d1200700 1 -- 172.21.15.43:0/3480753727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4cc109860 msgr2=0x7fb4cc1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:25.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.150+0000 7fb4d1200700 1 --2- 172.21.15.43:0/3480753727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4cc109860 0x7fb4cc1111c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400b0d0 tx=0x7fb4c400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.150+0000 7fb4d1200700 1 -- 172.21.15.43:0/3480753727 shutdown_connections 2024-05-27T07:32:25.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.150+0000 7fb4d1200700 1 --2- 172.21.15.43:0/3480753727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4cc109860 0x7fb4cc1111c0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.150+0000 7fb4d1200700 1 --2- 172.21.15.43:0/3480753727 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4cc1045e0 0x7fb4cc1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.150+0000 7fb4d1200700 1 --2- 172.21.15.43:0/3480753727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc108f10 0x7fb4cc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.150+0000 7fb4d1200700 1 -- 172.21.15.43:0/3480753727 >> 172.21.15.43:0/3480753727 conn(0x7fb4cc100030 msgr2=0x7fb4cc102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.150+0000 7fb4d1200700 1 -- 172.21.15.43:0/3480753727 shutdown_connections 2024-05-27T07:32:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.150+0000 7fb4d1200700 1 -- 172.21.15.43:0/3480753727 wait complete. 2024-05-27T07:32:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.152+0000 7fb4d1200700 1 Processor -- start 2024-05-27T07:32:25.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.152+0000 7fb4d1200700 1 -- start start 2024-05-27T07:32:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.152+0000 7fb4d1200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4cc1045e0 0x7fb4cc0774e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.153+0000 7fb4d1200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4cc108f10 0x7fb4cc06e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.153+0000 7fb4d1200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc109860 0x7fb4cc06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.153+0000 7fb4d1200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc114340 con 0x7fb4cc1045e0 2024-05-27T07:32:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.153+0000 7fb4d1200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1141c0 con 0x7fb4cc109860 2024-05-27T07:32:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.153+0000 7fb4d1200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1144c0 con 0x7fb4cc108f10 2024-05-27T07:32:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.153+0000 7fb4ca800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc109860 0x7fb4cc06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.153+0000 7fb4ca800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc109860 0x7fb4cc06ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52458/0 (socket says 172.21.15.43:52458) 2024-05-27T07:32:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.153+0000 7fb4ca800700 1 -- 172.21.15.43:0/2800606144 learned_addr learned my addr 172.21.15.43:0/2800606144 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:25.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.153+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4cc1045e0 0x7fb4cc0774e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.154+0000 7fb4c9400700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4cc108f10 0x7fb4cc06e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.154+0000 7fb4c9e00700 1 -- 172.21.15.43:0/2800606144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4cc108f10 msgr2=0x7fb4cc06e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.154+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4cc108f10 0x7fb4cc06e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.154+0000 7fb4c9e00700 1 -- 172.21.15.43:0/2800606144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc109860 msgr2=0x7fb4cc06ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.154+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc109860 0x7fb4cc06ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.154+0000 7fb4c9e00700 1 -- 172.21.15.43:0/2800606144 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4cc06f2b0 con 0x7fb4cc1045e0 2024-05-27T07:32:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.154+0000 7fb4c9400700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4cc108f10 0x7fb4cc06e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.154+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4cc1045e0 0x7fb4cc0774e0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00e9f0 tx=0x7fb4bc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.155+0000 7fb4c2a00700 1 -- 172.21.15.43:0/2800606144 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc00cbd0 con 0x7fb4cc1045e0 2024-05-27T07:32:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.155+0000 7fb4c2a00700 1 -- 172.21.15.43:0/2800606144 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4bc00cd70 con 0x7fb4cc1045e0 2024-05-27T07:32:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.155+0000 7fb4c2a00700 1 -- 172.21.15.43:0/2800606144 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc018680 con 0x7fb4cc1045e0 2024-05-27T07:32:25.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.155+0000 7fb4d1200700 1 -- 172.21.15.43:0/2800606144 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4cc072f90 con 0x7fb4cc1045e0 2024-05-27T07:32:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.155+0000 7fb4d1200700 1 -- 172.21.15.43:0/2800606144 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4cc073480 con 0x7fb4cc1045e0 2024-05-27T07:32:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.160+0000 7fb4c2a00700 1 -- 172.21.15.43:0/2800606144 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb4bc010430 con 0x7fb4cc1045e0 2024-05-27T07:32:25.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.162+0000 7fb4dfe0bb80 1 Processor -- start 2024-05-27T07:32:25.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.162+0000 7fb4dfe0bb80 1 -- start start 2024-05-27T07:32:25.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.163+0000 7fb4c2a00700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4b00b2a70 0x7fb4b00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.163+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e40679e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.163+0000 7fb4c2a00700 1 -- 172.21.15.43:0/2800606144 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7fb4bc0e2b40 con 0x7fb4cc1045e0 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.163+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e406798870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.163+0000 7fb4ca800700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4b00b2a70 0x7fb4b00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.164+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 0x55e4067991c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.164+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e4066c6e40 con 0x55e406798db0 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.164+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e4066b69e0 con 0x55e40679ed30 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.164+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e4066baa20 con 0x55e406619e40 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.164+0000 7fb4c9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e40679e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.164+0000 7fb4c9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e40679e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:55814/0 (socket says 172.21.15.43:55814) 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.164+0000 7fb4c9e00700 1 -- 172.21.15.43:0/2502689079 learned_addr learned my addr 172.21.15.43:0/2502689079 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:25.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.164+0000 7fb4ca800700 1 --2- 172.21.15.43:0/2502689079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 0x55e4067991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.164+0000 7fb4c9400700 1 --2- 172.21.15.43:0/2502689079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e406798870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.165+0000 7fb4ca800700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4b00b2a70 0x7fb4b00b4f80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009fd0 tx=0x7fb4c4009c90 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.165+0000 7fb4ca800700 1 -- 172.21.15.43:0/2502689079 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 msgr2=0x55e40679e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.165+0000 7fb4ca800700 1 --2- 172.21.15.43:0/2502689079 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e40679e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.165+0000 7fb4ca800700 1 -- 172.21.15.43:0/2502689079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 msgr2=0x55e406798870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.165+0000 7fb4ca800700 1 --2- 172.21.15.43:0/2502689079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e406798870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.165+0000 7fb4ca800700 1 -- 172.21.15.43:0/2502689079 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e40667f420 con 0x55e406798db0 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.165+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/2502689079 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e40679e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.166+0000 7fb4ca800700 1 --2- 172.21.15.43:0/2502689079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 0x55e4067991c0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4006070 tx=0x7fb4c4024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ae4be7033a7f21 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.166+0000 7fb4b7400700 1 -- 172.21.15.43:0/2502689079 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4024070 con 0x55e406798db0 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.166+0000 7fb4b7400700 1 -- 172.21.15.43:0/2502689079 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4c4007c50 con 0x55e406798db0 2024-05-27T07:32:25.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.166+0000 7fb4b7400700 1 -- 172.21.15.43:0/2502689079 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c401cce0 con 0x55e406798db0 2024-05-27T07:32:25.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.167+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/2502689079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 msgr2=0x55e4067991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:25.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.167+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/2502689079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 0x55e4067991c0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4006070 tx=0x7fb4c4024000 comp rx=0 tx=0).stop 2024-05-27T07:32:25.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.168+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/2502689079 shutdown_connections 2024-05-27T07:32:25.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.168+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/2502689079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 0x55e4067991c0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.168+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/2502689079 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e40679e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.168+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/2502689079 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e406798870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.168+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/2502689079 >> 172.21.15.43:0/2502689079 conn(0x55e406679fc0 msgr2=0x55e4067938d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:25.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.168+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/2502689079 shutdown_connections 2024-05-27T07:32:25.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.168+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/2502689079 wait complete. 2024-05-27T07:32:25.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.169+0000 7fb4dfe0bb80 1 -- start start 2024-05-27T07:32:25.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.169+0000 7fb4dfe0bb80 1 Processor -- start 2024-05-27T07:32:25.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.170+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e4067a24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.170+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 0x55e4067a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.170+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.170+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e4065bcfa0 con 0x55e406798db0 2024-05-27T07:32:25.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.170+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e406657760 con 0x55e40679ed30 2024-05-27T07:32:25.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.170+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e4067a57a0 con 0x55e406619e40 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.170+0000 7fb4dfe0bb80 10 client.0 _ll_register_callbacks cb 0x55e406660c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4ca800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4ca800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52472/0 (socket says 172.21.15.43:52472) 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4ca800700 1 -- 172.21.15.43:0/3553874248 learned_addr learned my addr 172.21.15.43:0/3553874248 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4c9400700 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 0x55e4067a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e4067a24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4ca800700 1 -- 172.21.15.43:0/3553874248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 msgr2=0x55e4067a24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4ca800700 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e4067a24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4ca800700 1 -- 172.21.15.43:0/3553874248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 msgr2=0x55e4067a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4ca800700 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 0x55e4067a2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4ca800700 1 -- 172.21.15.43:0/3553874248 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4066068d0 con 0x55e40679ed30 2024-05-27T07:32:25.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.171+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e4067a24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:32:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.172+0000 7fb4ca800700 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a2f50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400d010 tx=0x7fb4c4007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9f02aaf45474336 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.172+0000 7fb4abe00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:32:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.173+0000 7fb4abe00700 1 -- 172.21.15.43:0/3553874248 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4036070 con 0x55e40679ed30 2024-05-27T07:32:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.173+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3553874248 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4c4007c50 con 0x55e40679ed30 2024-05-27T07:32:25.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.173+0000 7fb4abe00700 1 -- 172.21.15.43:0/3553874248 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7fb4c4028400 con 0x55e40679ed30 2024-05-27T07:32:25.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.174+0000 7fb4abe00700 1 -- 172.21.15.43:0/3553874248 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7fb4c4042050 con 0x55e40679ed30 2024-05-27T07:32:25.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.174+0000 7fb4dfe0bb80 10 client.37349 fetch_fsmap learned FSMap version 137 2024-05-27T07:32:25.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.174+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3553874248 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=4+}) -- 0x55e406687060 con 0x55e40679ed30 2024-05-27T07:32:25.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.175+0000 7fb4abe00700 1 -- 172.21.15.43:0/3553874248 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4c4007c50 con 0x55e40679ed30 2024-05-27T07:32:25.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.175+0000 7fb4abe00700 1 -- 172.21.15.43:0/3553874248 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 137) ==== 4187+0+0 (secure 0 0 0) 0x7fb4c402f030 con 0x55e40679ed30 2024-05-27T07:32:25.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.175+0000 7fb4dfe0bb80 10 client.37349 fetch_fsmap finished waiting for FSMap version 137 2024-05-27T07:32:25.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.175+0000 7fb4dfe0bb80 10 client.37349 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:25.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.176+0000 7fb4dfe0bb80 20 client.37349 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:25.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.176+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e4067b2380 0x55e4067b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.176+0000 7fb4dfe0bb80 4 client.37349 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:25.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.176+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3553874248 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4066c8d90 con 0x55e4067b2380 2024-05-27T07:32:25.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.176+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e4067b2380 0x55e4067b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.177+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e4067b2380 0x55e4067b47a0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31b1beddd20d7c72 server_cookie=300999884c57ef88 in_seq=0 out_seq=0 2024-05-27T07:32:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.177+0000 7fb4abe00700 10 client.37349 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.178+0000 7fb4abe00700 1 -- 172.21.15.43:0/3553874248 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e4066c8d90 con 0x55e4067b2380 2024-05-27T07:32:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.178+0000 7fb4abe00700 10 client.37349 handle_command_reply: tid=0 2024-05-27T07:32:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.178+0000 7fb4dfe0bb80 1 client.37349 shutdown 2024-05-27T07:32:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3553874248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 msgr2=0x55e4067a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a2f50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400d010 tx=0x7fb4c4007ea0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3553874248 shutdown_connections 2024-05-27T07:32:25.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a2f50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406619e40 0x55e4067a24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e4067b2380 0x55e4067b47a0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/3553874248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406798db0 0x55e4067a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4abe00700 0 client.37349 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:25.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3553874248 >> 172.21.15.43:0/3553874248 conn(0x55e406679fc0 msgr2=0x55e406793580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:25.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3553874248 shutdown_connections 2024-05-27T07:32:25.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.179+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3553874248 wait complete. 2024-05-27T07:32:25.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.180+0000 7fb4dfe0bb80 20 client.37349 trim_cache size 0 max 16384 2024-05-27T07:32:25.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.218+0000 7fb4dfe0bb80 1 Processor -- start 2024-05-27T07:32:25.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.219+0000 7fb4dfe0bb80 1 -- start start 2024-05-27T07:32:25.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.219+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 0x55e406798870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.219+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406798db0 0x55e4067a4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.219+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.219+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e40679c940 con 0x55e406619e40 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.219+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1bf800 con 0x55e40679ed30 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.219+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1bdcc0 con 0x55e406798db0 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.220+0000 7fb4c9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 0x55e406798870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.220+0000 7fb4c9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 0x55e406798870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59584/0 (socket says 172.21.15.43:59584) 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.220+0000 7fb4c9e00700 1 -- 172.21.15.43:0/3579940046 learned_addr learned my addr 172.21.15.43:0/3579940046 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.220+0000 7fb4ca800700 1 --2- 172.21.15.43:0/3579940046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.220+0000 7fb4c9400700 1 --2- 172.21.15.43:0/3579940046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406798db0 0x55e4067a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.220+0000 7fb4c9e00700 1 -- 172.21.15.43:0/3579940046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406798db0 msgr2=0x55e4067a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.220+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/3579940046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406798db0 0x55e4067a4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.220+0000 7fb4c9e00700 1 -- 172.21.15.43:0/3579940046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 msgr2=0x55e4067ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.220+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/3579940046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067ae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.220+0000 7fb4c9e00700 1 -- 172.21.15.43:0/3579940046 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4067a5230 con 0x55e406619e40 2024-05-27T07:32:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.221+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/3579940046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 0x55e406798870 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00efb0 tx=0x7fb4bc009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7828f379b0a3a17 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.221+0000 7fb4b6a00700 1 -- 172.21.15.43:0/3579940046 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc08f070 con 0x55e406619e40 2024-05-27T07:32:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.221+0000 7fb4b6a00700 1 -- 172.21.15.43:0/3579940046 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4bc0a99f0 con 0x55e406619e40 2024-05-27T07:32:25.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.221+0000 7fb4b6a00700 1 -- 172.21.15.43:0/3579940046 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0b2620 con 0x55e406619e40 2024-05-27T07:32:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.223+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3579940046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 msgr2=0x55e406798870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.223+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/3579940046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 0x55e406798870 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00efb0 tx=0x7fb4bc009790 comp rx=0 tx=0).stop 2024-05-27T07:32:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.223+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3579940046 shutdown_connections 2024-05-27T07:32:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.223+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/3579940046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.223+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/3579940046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 0x55e406798870 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.223+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/3579940046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406798db0 0x55e4067a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.223+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3579940046 >> 172.21.15.43:0/3579940046 conn(0x55e406679fc0 msgr2=0x55e4067a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.223+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3579940046 shutdown_connections 2024-05-27T07:32:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.223+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/3579940046 wait complete. 2024-05-27T07:32:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.224+0000 7fb4dfe0bb80 1 -- start start 2024-05-27T07:32:25.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.224+0000 7fb4dfe0bb80 1 Processor -- start 2024-05-27T07:32:25.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.225+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 0x55e4067aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.225+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406798db0 0x55e4067ab130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.225+0000 7fb4dfe0bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.225+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1be390 con 0x55e406619e40 2024-05-27T07:32:25.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.225+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1bf130 con 0x55e40679ed30 2024-05-27T07:32:25.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.225+0000 7fb4dfe0bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1bea60 con 0x55e406798db0 2024-05-27T07:32:25.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.225+0000 7fb4dfe0bb80 10 client.0 _ll_register_callbacks cb 0x55e406660c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:25.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.226+0000 7fb4c9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 0x55e4067aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.226+0000 7fb4ca800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.226+0000 7fb4ca800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52486/0 (socket says 172.21.15.43:52486) 2024-05-27T07:32:25.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.226+0000 7fb4ca800700 1 -- 172.21.15.43:0/731602273 learned_addr learned my addr 172.21.15.43:0/731602273 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:25.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.226+0000 7fb4c9400700 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406798db0 0x55e4067ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.226+0000 7fb4ca800700 1 -- 172.21.15.43:0/731602273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406798db0 msgr2=0x55e4067ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:25.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.226+0000 7fb4ca800700 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406798db0 0x55e4067ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.226+0000 7fb4ca800700 1 -- 172.21.15.43:0/731602273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 msgr2=0x55e4067aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:25.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.226+0000 7fb4ca800700 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 0x55e4067aabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.226+0000 7fb4ca800700 1 -- 172.21.15.43:0/731602273 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4067a81b0 con 0x55e40679ed30 2024-05-27T07:32:25.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.227+0000 7fb4ca800700 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a79e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400b8a0 tx=0x7fb4c4030a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a46dfe5d50f7d3bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:25.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.227+0000 7fb4abe00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:32:25.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.227+0000 7fb4abe00700 1 -- 172.21.15.43:0/731602273 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4024070 con 0x55e40679ed30 2024-05-27T07:32:25.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.227+0000 7fb4abe00700 1 -- 172.21.15.43:0/731602273 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7fb4c401b3a0 con 0x55e40679ed30 2024-05-27T07:32:25.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.227+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/731602273 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4bc0a99f0 con 0x55e40679ed30 2024-05-27T07:32:25.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.228+0000 7fb4abe00700 1 -- 172.21.15.43:0/731602273 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7fb4c4036040 con 0x55e40679ed30 2024-05-27T07:32:25.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.228+0000 7fb4dfe0bb80 10 client.37355 fetch_fsmap learned FSMap version 137 2024-05-27T07:32:25.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.228+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/731602273 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=4+}) -- 0x55e40681d870 con 0x55e40679ed30 2024-05-27T07:32:25.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.228+0000 7fb4abe00700 1 -- 172.21.15.43:0/731602273 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4bc0a99f0 con 0x55e40679ed30 2024-05-27T07:32:25.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.228+0000 7fb4abe00700 1 -- 172.21.15.43:0/731602273 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 137) ==== 4187+0+0 (secure 0 0 0) 0x7fb4c402f040 con 0x55e40679ed30 2024-05-27T07:32:25.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.229+0000 7fb4dfe0bb80 10 client.37355 fetch_fsmap finished waiting for FSMap version 137 2024-05-27T07:32:25.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.229+0000 7fb4dfe0bb80 10 client.37355 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:25.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.229+0000 7fb4dfe0bb80 20 client.37355 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:25.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.229+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e40681deb0 0x55e4068202d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:25.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.229+0000 7fb4dfe0bb80 4 client.37355 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:32:25.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.229+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/731602273 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e4066c2570 con 0x55e40681deb0 2024-05-27T07:32:25.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.230+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e40681deb0 0x55e4068202d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:25.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.231+0000 7fb4c9e00700 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e40681deb0 0x55e4068202d0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30ff172ab2982757 server_cookie=a0e22947bc823762 in_seq=0 out_seq=0 2024-05-27T07:32:25.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.231+0000 7fb4abe00700 10 client.37355 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:25.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4abe00700 1 -- 172.21.15.43:0/731602273 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e4066c2570 con 0x55e40681deb0 2024-05-27T07:32:25.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4abe00700 10 client.37355 handle_command_reply: tid=0 2024-05-27T07:32:25.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4dfe0bb80 1 client.37355 shutdown 2024-05-27T07:32:25.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/731602273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 msgr2=0x55e4067a79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:25.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a79e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400b8a0 tx=0x7fb4c4030a10 comp rx=0 tx=0).stop 2024-05-27T07:32:25.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/731602273 shutdown_connections 2024-05-27T07:32:25.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e40679ed30 0x55e4067a79e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e40681deb0 0x55e4068202d0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e406619e40 0x55e4067aabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4dfe0bb80 1 --2- 172.21.15.43:0/731602273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e406798db0 0x55e4067ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.232+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/731602273 >> 172.21.15.43:0/731602273 conn(0x55e406679fc0 msgr2=0x55e4067a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:25.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.233+0000 7fb4abe00700 0 client.37355 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:25.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.233+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/731602273 shutdown_connections 2024-05-27T07:32:25.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.233+0000 7fb4dfe0bb80 1 -- 172.21.15.43:0/731602273 wait complete. 2024-05-27T07:32:25.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.233+0000 7fb4dfe0bb80 20 client.37355 trim_cache size 0 max 16384 2024-05-27T07:32:25.227 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:32:25.227 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:32:25.227 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:32:25.227 INFO:teuthology.orchestra.run.smithi043.stdout: "839c093e-ddcb-420b-9e0d-ec0a33b0fd49": { 2024-05-27T07:32:25.227 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:32:25.228 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "839c093e-ddcb-420b-9e0d-ec0a33b0fd49", 2024-05-27T07:32:25.228 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:32:25.228 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:32:25.228 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:32:25.228 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.234+0000 7fb4d1200700 1 -- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4b00b2a70 msgr2=0x7fb4b00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.234+0000 7fb4d1200700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4b00b2a70 0x7fb4b00b4f80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009fd0 tx=0x7fb4c4009c90 comp rx=0 tx=0).stop 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.234+0000 7fb4d1200700 1 -- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4cc1045e0 msgr2=0x7fb4cc0774e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.234+0000 7fb4d1200700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4cc1045e0 0x7fb4cc0774e0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00e9f0 tx=0x7fb4bc00ed00 comp rx=0 tx=0).stop 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.234+0000 7fb4d1200700 1 -- 172.21.15.43:0/2800606144 shutdown_connections 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.234+0000 7fb4d1200700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4b00b2a70 0x7fb4b00b4f80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.234+0000 7fb4d1200700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4cc109860 0x7fb4cc06ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.234+0000 7fb4d1200700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4cc1045e0 0x7fb4cc0774e0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.234+0000 7fb4d1200700 1 --2- 172.21.15.43:0/2800606144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4cc108f10 0x7fb4cc06e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.234+0000 7fb4d1200700 1 -- 172.21.15.43:0/2800606144 >> 172.21.15.43:0/2800606144 conn(0x7fb4cc100030 msgr2=0x7fb4cc1086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.235+0000 7fb4d1200700 1 -- 172.21.15.43:0/2800606144 shutdown_connections 2024-05-27T07:32:25.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:25.235+0000 7fb4d1200700 1 -- 172.21.15.43:0/2800606144 wait complete. 2024-05-27T07:32:25.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 29465 5.80 MB/sec execute 146 sec latency 86.897 ms 2024-05-27T07:32:25.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:25 smithi132 ceph-mon[35283]: pgmap v622: 129 pgs: 129 active+clean; 747 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.7 MiB/s wr, 165 op/s 2024-05-27T07:32:25.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:25 smithi043 ceph-mon[31531]: pgmap v622: 129 pgs: 129 active+clean; 747 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.7 MiB/s wr, 165 op/s 2024-05-27T07:32:25.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:25 smithi179 ceph-mon[35888]: pgmap v622: 129 pgs: 129 active+clean; 747 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.7 MiB/s wr, 165 op/s 2024-05-27T07:32:26.026 INFO:tasks.cephfs.filesystem:scrub status for tag:839c093e-ddcb-420b-9e0d-ec0a33b0fd49 - {'path': '/', 'tag': '839c093e-ddcb-420b-9e0d-ec0a33b0fd49', 'options': 'recursive,force'} 2024-05-27T07:32:26.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 29586 5.80 MB/sec execute 147 sec latency 73.348 ms 2024-05-27T07:32:27.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 29714 5.83 MB/sec execute 148 sec latency 78.344 ms 2024-05-27T07:32:27.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:27 smithi132 ceph-mon[35283]: pgmap v623: 129 pgs: 129 active+clean; 760 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 4.4 MiB/s wr, 242 op/s 2024-05-27T07:32:27.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:27 smithi043 ceph-mon[31531]: pgmap v623: 129 pgs: 129 active+clean; 760 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 4.4 MiB/s wr, 242 op/s 2024-05-27T07:32:27.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:27 smithi179 ceph-mon[35888]: pgmap v623: 129 pgs: 129 active+clean; 760 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 4.4 MiB/s wr, 242 op/s 2024-05-27T07:32:28.788 INFO:tasks.workunit.client.0.smithi043.stdout: 10 29875 5.88 MB/sec execute 149 sec latency 70.712 ms 2024-05-27T07:32:29.789 INFO:tasks.workunit.client.0.smithi043.stdout: 10 29969 5.87 MB/sec execute 150 sec latency 71.061 ms 2024-05-27T07:32:29.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:29 smithi132 ceph-mon[35283]: pgmap v624: 129 pgs: 129 active+clean; 760 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-05-27T07:32:29.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:29 smithi043 ceph-mon[31531]: pgmap v624: 129 pgs: 129 active+clean; 760 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-05-27T07:32:29.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:29 smithi179 ceph-mon[35888]: pgmap v624: 129 pgs: 129 active+clean; 760 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-05-27T07:32:30.789 INFO:tasks.workunit.client.0.smithi043.stdout: 10 30110 5.89 MB/sec execute 151 sec latency 69.683 ms 2024-05-27T07:32:31.789 INFO:tasks.workunit.client.0.smithi043.stdout: 10 30234 5.92 MB/sec execute 152 sec latency 85.435 ms 2024-05-27T07:32:31.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:31 smithi043 ceph-mon[31531]: pgmap v625: 129 pgs: 129 active+clean; 760 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-05-27T07:32:31.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:31 smithi179 ceph-mon[35888]: pgmap v625: 129 pgs: 129 active+clean; 760 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-05-27T07:32:32.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:31 smithi132 ceph-mon[35283]: pgmap v625: 129 pgs: 129 active+clean; 760 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-05-27T07:32:32.789 INFO:tasks.workunit.client.0.smithi043.stdout: 10 30333 5.91 MB/sec execute 153 sec latency 95.100 ms 2024-05-27T07:32:32.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:32 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:32:32.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:32 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:32:32.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:32 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:32:32.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:32 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:32:32.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:32 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:32:32.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:32 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:32:32.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:32 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:32:32.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:32 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:32:32.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:32 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:32:32.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:32 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:32:33.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:32 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:32:33.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:32 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:32:33.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:32 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:32:33.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:32 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:32:33.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:32 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:32:33.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 30461 5.90 MB/sec execute 154 sec latency 130.574 ms 2024-05-27T07:32:33.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:33 smithi043 ceph-mon[31531]: pgmap v626: 129 pgs: 129 active+clean; 777 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 5.4 MiB/s wr, 229 op/s 2024-05-27T07:32:33.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:33 smithi179 ceph-mon[35888]: pgmap v626: 129 pgs: 129 active+clean; 777 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 5.4 MiB/s wr, 229 op/s 2024-05-27T07:32:34.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:33 smithi132 ceph-mon[35283]: pgmap v626: 129 pgs: 129 active+clean; 777 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 5.4 MiB/s wr, 229 op/s 2024-05-27T07:32:34.789 INFO:tasks.workunit.client.0.smithi043.stdout: 10 30550 5.91 MB/sec execute 155 sec latency 491.946 ms 2024-05-27T07:32:34.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:34 smithi043 ceph-mon[31531]: pgmap v627: 129 pgs: 129 active+clean; 777 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.8 MiB/s wr, 147 op/s 2024-05-27T07:32:34.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:34 smithi179 ceph-mon[35888]: pgmap v627: 129 pgs: 129 active+clean; 777 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.8 MiB/s wr, 147 op/s 2024-05-27T07:32:35.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:34 smithi132 ceph-mon[35283]: pgmap v627: 129 pgs: 129 active+clean; 777 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.8 MiB/s wr, 147 op/s 2024-05-27T07:32:35.789 INFO:tasks.workunit.client.0.smithi043.stdout: 10 30641 5.89 MB/sec execute 156 sec latency 85.661 ms 2024-05-27T07:32:36.789 INFO:tasks.workunit.client.0.smithi043.stdout: 10 30747 5.90 MB/sec execute 157 sec latency 96.474 ms 2024-05-27T07:32:37.789 INFO:tasks.workunit.client.0.smithi043.stdout: 10 30867 5.91 MB/sec execute 158 sec latency 87.868 ms 2024-05-27T07:32:37.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:37 smithi132 ceph-mon[35283]: pgmap v628: 129 pgs: 129 active+clean; 785 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 5.5 MiB/s wr, 221 op/s 2024-05-27T07:32:37.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:37 smithi043 ceph-mon[31531]: pgmap v628: 129 pgs: 129 active+clean; 785 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 5.5 MiB/s wr, 221 op/s 2024-05-27T07:32:37.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:37 smithi179 ceph-mon[35888]: pgmap v628: 129 pgs: 129 active+clean; 785 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 5.5 MiB/s wr, 221 op/s 2024-05-27T07:32:38.789 INFO:tasks.workunit.client.0.smithi043.stdout: 10 30982 5.91 MB/sec execute 159 sec latency 100.727 ms 2024-05-27T07:32:39.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 31098 5.92 MB/sec execute 160 sec latency 95.360 ms 2024-05-27T07:32:39.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:39 smithi132 ceph-mon[35283]: pgmap v629: 129 pgs: 129 active+clean; 785 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 3.9 MiB/s wr, 145 op/s 2024-05-27T07:32:39.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:39 smithi043 ceph-mon[31531]: pgmap v629: 129 pgs: 129 active+clean; 785 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 3.9 MiB/s wr, 145 op/s 2024-05-27T07:32:39.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:39 smithi179 ceph-mon[35888]: pgmap v629: 129 pgs: 129 active+clean; 785 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 3.9 MiB/s wr, 145 op/s 2024-05-27T07:32:40.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 31190 5.90 MB/sec execute 161 sec latency 79.576 ms 2024-05-27T07:32:41.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 31262 5.87 MB/sec execute 162 sec latency 376.701 ms 2024-05-27T07:32:41.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:41 smithi043 ceph-mon[31531]: pgmap v630: 129 pgs: 129 active+clean; 785 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 3.9 MiB/s wr, 145 op/s 2024-05-27T07:32:41.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:41 smithi179 ceph-mon[35888]: pgmap v630: 129 pgs: 129 active+clean; 785 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 3.9 MiB/s wr, 145 op/s 2024-05-27T07:32:42.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:41 smithi132 ceph-mon[35283]: pgmap v630: 129 pgs: 129 active+clean; 785 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 3.9 MiB/s wr, 145 op/s 2024-05-27T07:32:42.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 31362 5.84 MB/sec execute 163 sec latency 102.522 ms 2024-05-27T07:32:43.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 31484 5.83 MB/sec execute 164 sec latency 81.527 ms 2024-05-27T07:32:43.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:43 smithi043 ceph-mon[31531]: pgmap v631: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.5 MiB/s wr, 230 op/s 2024-05-27T07:32:43.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:43 smithi179 ceph-mon[35888]: pgmap v631: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.5 MiB/s wr, 230 op/s 2024-05-27T07:32:44.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:43 smithi132 ceph-mon[35283]: pgmap v631: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.5 MiB/s wr, 230 op/s 2024-05-27T07:32:44.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 31551 5.82 MB/sec execute 165 sec latency 546.276 ms 2024-05-27T07:32:45.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 31665 5.79 MB/sec execute 166 sec latency 85.060 ms 2024-05-27T07:32:45.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:45 smithi043 ceph-mon[31531]: pgmap v632: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.4 MiB/s wr, 160 op/s 2024-05-27T07:32:45.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:45 smithi179 ceph-mon[35888]: pgmap v632: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.4 MiB/s wr, 160 op/s 2024-05-27T07:32:46.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:45 smithi132 ceph-mon[35283]: pgmap v632: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.4 MiB/s wr, 160 op/s 2024-05-27T07:32:46.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 31782 5.79 MB/sec execute 167 sec latency 94.707 ms 2024-05-27T07:32:47.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 31894 5.77 MB/sec execute 168 sec latency 87.551 ms 2024-05-27T07:32:47.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:47 smithi132 ceph-mon[35283]: pgmap v633: 129 pgs: 129 active+clean; 786 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.3 MiB/s wr, 233 op/s 2024-05-27T07:32:47.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:47 smithi043 ceph-mon[31531]: pgmap v633: 129 pgs: 129 active+clean; 786 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.3 MiB/s wr, 233 op/s 2024-05-27T07:32:47.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:47 smithi179 ceph-mon[35888]: pgmap v633: 129 pgs: 129 active+clean; 786 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.3 MiB/s wr, 233 op/s 2024-05-27T07:32:48.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 32015 5.77 MB/sec execute 169 sec latency 73.735 ms 2024-05-27T07:32:49.790 INFO:tasks.workunit.client.0.smithi043.stdout: 10 32119 5.78 MB/sec execute 170 sec latency 298.782 ms 2024-05-27T07:32:49.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:49 smithi043 ceph-mon[31531]: pgmap v634: 129 pgs: 129 active+clean; 786 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-05-27T07:32:49.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:49 smithi179 ceph-mon[35888]: pgmap v634: 129 pgs: 129 active+clean; 786 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-05-27T07:32:50.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:49 smithi132 ceph-mon[35283]: pgmap v634: 129 pgs: 129 active+clean; 786 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-05-27T07:32:50.791 INFO:tasks.workunit.client.0.smithi043.stdout: 10 32271 5.83 MB/sec execute 171 sec latency 73.389 ms 2024-05-27T07:32:51.791 INFO:tasks.workunit.client.0.smithi043.stdout: 10 32417 5.85 MB/sec execute 172 sec latency 96.330 ms 2024-05-27T07:32:51.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:51 smithi043 ceph-mon[31531]: pgmap v635: 129 pgs: 129 active+clean; 786 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-05-27T07:32:51.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:51 smithi179 ceph-mon[35888]: pgmap v635: 129 pgs: 129 active+clean; 786 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-05-27T07:32:52.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:51 smithi132 ceph-mon[35283]: pgmap v635: 129 pgs: 129 active+clean; 786 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-05-27T07:32:52.791 INFO:tasks.workunit.client.0.smithi043.stdout: 10 32572 5.87 MB/sec execute 173 sec latency 87.202 ms 2024-05-27T07:32:53.791 INFO:tasks.workunit.client.0.smithi043.stdout: 10 32701 5.87 MB/sec execute 174 sec latency 106.601 ms 2024-05-27T07:32:53.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:53 smithi043 ceph-mon[31531]: pgmap v636: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 4.2 MiB/s wr, 243 op/s 2024-05-27T07:32:53.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:53 smithi179 ceph-mon[35888]: pgmap v636: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 4.2 MiB/s wr, 243 op/s 2024-05-27T07:32:54.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:53 smithi132 ceph-mon[35283]: pgmap v636: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 4.2 MiB/s wr, 243 op/s 2024-05-27T07:32:54.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 32760 5.84 MB/sec execute 175 sec latency 404.121 ms 2024-05-27T07:32:55.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 32861 5.83 MB/sec execute 176 sec latency 83.074 ms 2024-05-27T07:32:55.895 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:55 smithi132 ceph-mon[35283]: pgmap v637: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 2.5 MiB/s wr, 157 op/s 2024-05-27T07:32:55.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:55 smithi043 ceph-mon[31531]: pgmap v637: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 2.5 MiB/s wr, 157 op/s 2024-05-27T07:32:55.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:55 smithi179 ceph-mon[35888]: pgmap v637: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 2.5 MiB/s wr, 157 op/s 2024-05-27T07:32:56.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:32:56.612 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:32:56.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 32992 5.86 MB/sec execute 177 sec latency 102.920 ms 2024-05-27T07:32:57.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 33143 5.89 MB/sec execute 178 sec latency 102.920 ms 2024-05-27T07:32:57.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:57 smithi043 ceph-mon[31531]: pgmap v638: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.2 MiB/s wr, 230 op/s 2024-05-27T07:32:57.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:57 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:57.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:57 smithi179 ceph-mon[35888]: pgmap v638: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.2 MiB/s wr, 230 op/s 2024-05-27T07:32:57.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:57 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:58.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:57 smithi132 ceph-mon[35283]: pgmap v638: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.2 MiB/s wr, 230 op/s 2024-05-27T07:32:58.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:57 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:32:58.706 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:58 smithi043 ceph-mon[31531]: pgmap v639: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 3.3 MiB/s wr, 156 op/s 2024-05-27T07:32:58.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 33257 5.89 MB/sec execute 179 sec latency 103.768 ms 2024-05-27T07:32:58.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:58 smithi179 ceph-mon[35888]: pgmap v639: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 3.3 MiB/s wr, 156 op/s 2024-05-27T07:32:59.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:58 smithi132 ceph-mon[35283]: pgmap v639: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 3.3 MiB/s wr, 156 op/s 2024-05-27T07:32:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.400+0000 7f2500200700 1 -- 172.21.15.43:0/918210487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24f81054f0 msgr2=0x7f24f8105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.400+0000 7f2500200700 1 --2- 172.21.15.43:0/918210487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24f81054f0 0x7f24f8105900 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f24ec009990 tx=0x7f24ec009ca0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.400+0000 7f2500200700 1 -- 172.21.15.43:0/918210487 shutdown_connections 2024-05-27T07:32:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.400+0000 7f2500200700 1 --2- 172.21.15.43:0/918210487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24f810e8d0 0x7f24f8110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.400+0000 7f2500200700 1 --2- 172.21.15.43:0/918210487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24f81054f0 0x7f24f8105900 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.400+0000 7f2500200700 1 --2- 172.21.15.43:0/918210487 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24f8105e40 0x7f24f810e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.400+0000 7f2500200700 1 -- 172.21.15.43:0/918210487 >> 172.21.15.43:0/918210487 conn(0x7f24f80fedb0 msgr2=0x7f24f81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.401+0000 7f2500200700 1 -- 172.21.15.43:0/918210487 shutdown_connections 2024-05-27T07:32:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.401+0000 7f2500200700 1 -- 172.21.15.43:0/918210487 wait complete. 2024-05-27T07:32:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.402+0000 7f2500200700 1 Processor -- start 2024-05-27T07:32:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.403+0000 7f2500200700 1 -- start start 2024-05-27T07:32:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.403+0000 7f2500200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24f81054f0 0x7f24f8075390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.403+0000 7f2500200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24f8105e40 0x7f24f806e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.404+0000 7f24fd800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24f81054f0 0x7f24f8075390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.404+0000 7f24fd800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24f81054f0 0x7f24f8075390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41404/0 (socket says 172.21.15.43:41404) 2024-05-27T07:32:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.406+0000 7f2500200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24f810e8d0 0x7f24f806eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.406+0000 7f24fd800700 1 -- 172.21.15.43:0/658003545 learned_addr learned my addr 172.21.15.43:0/658003545 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.406+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24f8114340 con 0x7f24f8105e40 2024-05-27T07:32:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.406+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f24f81141c0 con 0x7f24f81054f0 2024-05-27T07:32:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.406+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f24f81144c0 con 0x7f24f810e8d0 2024-05-27T07:32:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.407+0000 7f24fe200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24f810e8d0 0x7f24f806eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.407+0000 7f24fce00700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24f8105e40 0x7f24f806e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.407+0000 7f24fe200700 1 -- 172.21.15.43:0/658003545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24f81054f0 msgr2=0x7f24f8075390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.407+0000 7f24fe200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24f81054f0 0x7f24f8075390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.407+0000 7f24fe200700 1 -- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24f8105e40 msgr2=0x7f24f806e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.407+0000 7f24fe200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24f8105e40 0x7f24f806e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.407+0000 7f24fe200700 1 -- 172.21.15.43:0/658003545 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f8072ec0 con 0x7f24f810e8d0 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.408+0000 7f24fce00700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24f8105e40 0x7f24f806e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.408+0000 7f24fe200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24f810e8d0 0x7f24f806eb10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f24f0007830 tx=0x7f24f0007bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.408+0000 7f24f6000700 1 -- 172.21.15.43:0/658003545 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f0004db0 con 0x7f24f810e8d0 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.408+0000 7f24f6000700 1 -- 172.21.15.43:0/658003545 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24f001a8f0 con 0x7f24f810e8d0 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.408+0000 7f24f6000700 1 -- 172.21.15.43:0/658003545 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f00055b0 con 0x7f24f810e8d0 2024-05-27T07:32:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.409+0000 7f24fd800700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24f81054f0 0x7f24f8075390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.409+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f80731b0 con 0x7f24f810e8d0 2024-05-27T07:32:59.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.409+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f80736a0 con 0x7f24f810e8d0 2024-05-27T07:32:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.415+0000 7f24f6000700 1 -- 172.21.15.43:0/658003545 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f24f001a460 con 0x7f24f810e8d0 2024-05-27T07:32:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.415+0000 7f250ec5db80 1 Processor -- start 2024-05-27T07:32:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.416+0000 7f250ec5db80 1 -- start start 2024-05-27T07:32:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.416+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a6397f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.416+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a639d30 0x561d3a633870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.416+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a6341c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.416+0000 7f250ec5db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561d3a561e40 con 0x561d3a639d30 2024-05-27T07:32:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.416+0000 7f250ec5db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561d3a5519e0 con 0x561d3a633db0 2024-05-27T07:32:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.416+0000 7f250ec5db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561d3a555a20 con 0x561d3a4b4e40 2024-05-27T07:32:59.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.417+0000 7f24fce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a6341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.417+0000 7f24fce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a6341c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41412/0 (socket says 172.21.15.43:41412) 2024-05-27T07:32:59.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.417+0000 7f24fce00700 1 -- 172.21.15.43:0/3085364931 learned_addr learned my addr 172.21.15.43:0/3085364931 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:59.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.417+0000 7f24fce00700 1 --2- 172.21.15.43:0/3085364931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a6397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.418+0000 7f24f6000700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24e00b29f0 0x7f24e00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.418+0000 7f24fd800700 1 --2- 172.21.15.43:0/3085364931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a639d30 0x561d3a633870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.418+0000 7f24fce00700 1 -- 172.21.15.43:0/3085364931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 msgr2=0x561d3a6397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.418+0000 7f24fce00700 1 --2- 172.21.15.43:0/3085364931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a6397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.418+0000 7f24fce00700 1 -- 172.21.15.43:0/3085364931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a639d30 msgr2=0x561d3a633870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.418+0000 7f24fce00700 1 --2- 172.21.15.43:0/3085364931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a639d30 0x561d3a633870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.418+0000 7f24f6000700 1 -- 172.21.15.43:0/658003545 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7f24f00eaac0 con 0x7f24f810e8d0 2024-05-27T07:32:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.418+0000 7f24fce00700 1 -- 172.21.15.43:0/3085364931 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d3a51a420 con 0x561d3a633db0 2024-05-27T07:32:59.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.419+0000 7f24fce00700 1 --2- 172.21.15.43:0/3085364931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a6341c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f24e40094f0 tx=0x7f24e4009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b96be6d1d050fa17 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:59.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.419+0000 7f24fe200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24e00b29f0 0x7f24e00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.419+0000 7f24eaa00700 1 -- 172.21.15.43:0/3085364931 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4004030 con 0x561d3a633db0 2024-05-27T07:32:59.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.419+0000 7f24eaa00700 1 -- 172.21.15.43:0/3085364931 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24e4003680 con 0x561d3a633db0 2024-05-27T07:32:59.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.419+0000 7f24eaa00700 1 -- 172.21.15.43:0/3085364931 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e40126b0 con 0x561d3a633db0 2024-05-27T07:32:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.421+0000 7f250ec5db80 1 -- 172.21.15.43:0/3085364931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 msgr2=0x561d3a6341c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.421+0000 7f250ec5db80 1 --2- 172.21.15.43:0/3085364931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a6341c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f24e40094f0 tx=0x7f24e4009700 comp rx=0 tx=0).stop 2024-05-27T07:32:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.421+0000 7f24fe200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24e00b29f0 0x7f24e00b4f00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f24f00e7e40 tx=0x7f24f0004f50 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.421+0000 7f250ec5db80 1 -- 172.21.15.43:0/3085364931 shutdown_connections 2024-05-27T07:32:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.421+0000 7f250ec5db80 1 --2- 172.21.15.43:0/3085364931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a6341c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.421+0000 7f250ec5db80 1 --2- 172.21.15.43:0/3085364931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a6397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.421+0000 7f250ec5db80 1 --2- 172.21.15.43:0/3085364931 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a639d30 0x561d3a633870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.421+0000 7f250ec5db80 1 -- 172.21.15.43:0/3085364931 >> 172.21.15.43:0/3085364931 conn(0x561d3a514fc0 msgr2=0x561d3a62e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.422+0000 7f250ec5db80 1 -- 172.21.15.43:0/3085364931 shutdown_connections 2024-05-27T07:32:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.422+0000 7f250ec5db80 1 -- 172.21.15.43:0/3085364931 wait complete. 2024-05-27T07:32:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.422+0000 7f250ec5db80 1 -- start start 2024-05-27T07:32:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.423+0000 7f250ec5db80 1 Processor -- start 2024-05-27T07:32:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.423+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a4b4e40 0x561d3a63d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.423+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a633db0 0x561d3a63db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.423+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a639d30 0x561d3a63e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.423+0000 7f250ec5db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561d3a457fa0 con 0x561d3a4b4e40 2024-05-27T07:32:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.423+0000 7f250ec5db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561d3a4f2760 con 0x561d3a639d30 2024-05-27T07:32:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.423+0000 7f250ec5db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561d3a640800 con 0x561d3a633db0 2024-05-27T07:32:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.424+0000 7f250ec5db80 10 client.0 _ll_register_callbacks cb 0x561d3a4fbc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.424+0000 7f24fd800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a633db0 0x561d3a63db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.424+0000 7f24fd800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a633db0 0x561d3a63db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:57356/0 (socket says 172.21.15.43:57356) 2024-05-27T07:32:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.424+0000 7f24fd800700 1 -- 172.21.15.43:0/523727310 learned_addr learned my addr 172.21.15.43:0/523727310 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.424+0000 7f24fce00700 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a639d30 0x561d3a63e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.424+0000 7f24fce00700 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a4b4e40 0x561d3a63d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.425+0000 7f24fce00700 1 -- 172.21.15.43:0/523727310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a633db0 msgr2=0x561d3a63db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.425+0000 7f24fce00700 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a633db0 0x561d3a63db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.425+0000 7f24fce00700 1 -- 172.21.15.43:0/523727310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a4b4e40 msgr2=0x561d3a63d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.425+0000 7f24fce00700 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a4b4e40 0x561d3a63d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.425+0000 7f24fce00700 1 -- 172.21.15.43:0/523727310 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d3a4a18d0 con 0x561d3a639d30 2024-05-27T07:32:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.425+0000 7f24fce00700 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a639d30 0x561d3a63e040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f24e40094c0 tx=0x7f24e4009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d3f55ce7d382ae7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.426+0000 7f24df400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:32:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.426+0000 7f24df400700 1 -- 172.21.15.43:0/523727310 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e40043d0 con 0x561d3a639d30 2024-05-27T07:32:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.426+0000 7f24df400700 1 -- 172.21.15.43:0/523727310 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7f24e4005050 con 0x561d3a639d30 2024-05-27T07:32:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.426+0000 7f250ec5db80 1 -- 172.21.15.43:0/523727310 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24e4003680 con 0x561d3a639d30 2024-05-27T07:32:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.426+0000 7f24df400700 1 -- 172.21.15.43:0/523727310 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) ==== 24+0+0 (secure 0 0 0) 0x7f24e4033050 con 0x561d3a639d30 2024-05-27T07:32:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.427+0000 7f250ec5db80 10 client.37364 fetch_fsmap learned FSMap version 138 2024-05-27T07:32:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.427+0000 7f250ec5db80 1 -- 172.21.15.43:0/523727310 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=4+}) -- 0x561d3a522060 con 0x561d3a639d30 2024-05-27T07:32:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.427+0000 7f24df400700 1 -- 172.21.15.43:0/523727310 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24e4003680 con 0x561d3a639d30 2024-05-27T07:32:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.427+0000 7f24df400700 1 -- 172.21.15.43:0/523727310 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 138) ==== 4191+0+0 (secure 0 0 0) 0x7f24e4004b70 con 0x561d3a639d30 2024-05-27T07:32:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.427+0000 7f250ec5db80 10 client.37364 fetch_fsmap finished waiting for FSMap version 138 2024-05-27T07:32:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.427+0000 7f250ec5db80 10 client.37364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.427+0000 7f250ec5db80 20 client.37364 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.427+0000 7f250ec5db80 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561d3a64d7b0 0x561d3a64fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.427+0000 7f250ec5db80 4 client.37364 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:32:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.427+0000 7f250ec5db80 1 -- 172.21.15.43:0/523727310 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d3a563d90 con 0x561d3a64d7b0 2024-05-27T07:32:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.428+0000 7f24fd800700 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561d3a64d7b0 0x561d3a64fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.429+0000 7f24fd800700 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561d3a64d7b0 0x561d3a64fc20 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bef7860fdd2bcef server_cookie=21fcb68fa478da24 in_seq=0 out_seq=0 2024-05-27T07:32:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.429+0000 7f24df400700 10 client.37364 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.430+0000 7f24df400700 1 -- 172.21.15.43:0/523727310 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561d3a563d90 con 0x561d3a64d7b0 2024-05-27T07:32:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.430+0000 7f24df400700 10 client.37364 handle_command_reply: tid=0 2024-05-27T07:32:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.431+0000 7f250ec5db80 1 client.37364 shutdown 2024-05-27T07:32:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.431+0000 7f250ec5db80 1 -- 172.21.15.43:0/523727310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a639d30 msgr2=0x561d3a63e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.431+0000 7f250ec5db80 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a639d30 0x561d3a63e040 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f24e40094c0 tx=0x7f24e4009700 comp rx=0 tx=0).stop 2024-05-27T07:32:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.431+0000 7f250ec5db80 1 -- 172.21.15.43:0/523727310 shutdown_connections 2024-05-27T07:32:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.431+0000 7f250ec5db80 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a639d30 0x561d3a63e040 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.431+0000 7f250ec5db80 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561d3a64d7b0 0x561d3a64fc20 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.431+0000 7f250ec5db80 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a4b4e40 0x561d3a63d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.431+0000 7f250ec5db80 1 --2- 172.21.15.43:0/523727310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a633db0 0x561d3a63db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.431+0000 7f250ec5db80 1 -- 172.21.15.43:0/523727310 >> 172.21.15.43:0/523727310 conn(0x561d3a514fc0 msgr2=0x561d3a62e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.432+0000 7f24df400700 0 client.37364 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.432+0000 7f250ec5db80 1 -- 172.21.15.43:0/523727310 shutdown_connections 2024-05-27T07:32:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.432+0000 7f250ec5db80 1 -- 172.21.15.43:0/523727310 wait complete. 2024-05-27T07:32:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.432+0000 7f250ec5db80 20 client.37364 trim_cache size 0 max 16384 2024-05-27T07:32:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.467+0000 7f250ec5db80 1 Processor -- start 2024-05-27T07:32:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.467+0000 7f250ec5db80 1 -- start start 2024-05-27T07:32:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.468+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a633870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.468+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a637880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.468+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a638e20 0x561d3a63dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.468+0000 7f250ec5db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24f81b7150 con 0x561d3a638e20 2024-05-27T07:32:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.468+0000 7f250ec5db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f24f81b55e0 con 0x561d3a633db0 2024-05-27T07:32:59.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.468+0000 7f250ec5db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f24f81b6a80 con 0x561d3a4b4e40 2024-05-27T07:32:59.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.469+0000 7f24fce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a638e20 0x561d3a63dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.469+0000 7f24fd800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a637880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.469+0000 7f24fd800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a637880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41432/0 (socket says 172.21.15.43:41432) 2024-05-27T07:32:59.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.469+0000 7f24fd800700 1 -- 172.21.15.43:0/3228160779 learned_addr learned my addr 172.21.15.43:0/3228160779 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:59.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.469+0000 7f24fce00700 1 --2- 172.21.15.43:0/3228160779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a633870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.469+0000 7f24fd800700 1 -- 172.21.15.43:0/3228160779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 msgr2=0x561d3a633870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:59.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.469+0000 7f24fd800700 1 --2- 172.21.15.43:0/3228160779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a633870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.469+0000 7f24fd800700 1 -- 172.21.15.43:0/3228160779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a638e20 msgr2=0x561d3a63dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:59.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.469+0000 7f24fd800700 1 --2- 172.21.15.43:0/3228160779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a638e20 0x561d3a63dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.469+0000 7f24fd800700 1 -- 172.21.15.43:0/3228160779 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d3a63bb30 con 0x561d3a633db0 2024-05-27T07:32:59.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.470+0000 7f24fce00700 1 --2- 172.21.15.43:0/3228160779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a633870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:32:59.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.470+0000 7f24fd800700 1 --2- 172.21.15.43:0/3228160779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a637880 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00b2e0 tx=0x7f24ec005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b269e78dc4279f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:59.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.471+0000 7f24ea000700 1 -- 172.21.15.43:0/3228160779 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec003880 con 0x561d3a633db0 2024-05-27T07:32:59.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.471+0000 7f24ea000700 1 -- 172.21.15.43:0/3228160779 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24ec00b6b0 con 0x561d3a633db0 2024-05-27T07:32:59.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.471+0000 7f24ea000700 1 -- 172.21.15.43:0/3228160779 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec00f6c0 con 0x561d3a633db0 2024-05-27T07:32:59.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.472+0000 7f250ec5db80 1 -- 172.21.15.43:0/3228160779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 msgr2=0x561d3a637880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.472+0000 7f250ec5db80 1 --2- 172.21.15.43:0/3228160779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a637880 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00b2e0 tx=0x7f24ec005040 comp rx=0 tx=0).stop 2024-05-27T07:32:59.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.472+0000 7f250ec5db80 1 -- 172.21.15.43:0/3228160779 shutdown_connections 2024-05-27T07:32:59.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.472+0000 7f250ec5db80 1 --2- 172.21.15.43:0/3228160779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a638e20 0x561d3a63dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.472+0000 7f250ec5db80 1 --2- 172.21.15.43:0/3228160779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a633870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.472+0000 7f250ec5db80 1 --2- 172.21.15.43:0/3228160779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a633db0 0x561d3a637880 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.472+0000 7f250ec5db80 1 -- 172.21.15.43:0/3228160779 >> 172.21.15.43:0/3228160779 conn(0x561d3a514fc0 msgr2=0x561d3a62db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:59.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.472+0000 7f250ec5db80 1 -- 172.21.15.43:0/3228160779 shutdown_connections 2024-05-27T07:32:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.472+0000 7f250ec5db80 1 -- 172.21.15.43:0/3228160779 wait complete. 2024-05-27T07:32:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.473+0000 7f250ec5db80 1 -- start start 2024-05-27T07:32:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.473+0000 7f250ec5db80 1 Processor -- start 2024-05-27T07:32:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.474+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a645c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.474+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a633db0 0x561d3a646230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.474+0000 7f250ec5db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a638e20 0x561d3a642ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.474+0000 7f250ec5db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24f81b63b0 con 0x561d3a633db0 2024-05-27T07:32:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.474+0000 7f250ec5db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f24f81b5cb0 con 0x561d3a638e20 2024-05-27T07:32:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.474+0000 7f250ec5db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561d3a563d90 con 0x561d3a4b4e40 2024-05-27T07:32:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.474+0000 7f250ec5db80 10 client.0 _ll_register_callbacks cb 0x561d3a4fbc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:32:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.475+0000 7f24fd800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a633db0 0x561d3a646230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.475+0000 7f24fd800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a633db0 0x561d3a646230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47288/0 (socket says 172.21.15.43:47288) 2024-05-27T07:32:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.475+0000 7f24fd800700 1 -- 172.21.15.43:0/2677586749 learned_addr learned my addr 172.21.15.43:0/2677586749 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:32:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.476+0000 7f24fce00700 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a638e20 0x561d3a642ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.476+0000 7f24fce00700 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a645c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.477+0000 7f24fd800700 1 -- 172.21.15.43:0/2677586749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 msgr2=0x561d3a645c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.477+0000 7f24fd800700 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a645c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.477+0000 7f24fd800700 1 -- 172.21.15.43:0/2677586749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a638e20 msgr2=0x561d3a642ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:32:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.477+0000 7f24fd800700 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a638e20 0x561d3a642ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.477+0000 7f24fd800700 1 -- 172.21.15.43:0/2677586749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d3a643280 con 0x561d3a633db0 2024-05-27T07:32:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.478+0000 7f24fd800700 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a633db0 0x561d3a646230 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f24ec005910 tx=0x7f24ec005940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ef50e2c3924437 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:32:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.478+0000 7f24df400700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:32:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.478+0000 7f24df400700 1 -- 172.21.15.43:0/2677586749 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec00baf0 con 0x561d3a633db0 2024-05-27T07:32:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.479+0000 7f250ec5db80 1 -- 172.21.15.43:0/2677586749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24ec00b6b0 con 0x561d3a633db0 2024-05-27T07:32:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.479+0000 7f24df400700 1 -- 172.21.15.43:0/2677586749 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(147..147 src has 1..147) ==== 9496+0+0 (secure 0 0 0) 0x7f24ec00bc90 con 0x561d3a633db0 2024-05-27T07:32:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.479+0000 7f24df400700 1 -- 172.21.15.43:0/2677586749 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) ==== 24+0+0 (secure 0 0 0) 0x7f24ec003680 con 0x561d3a633db0 2024-05-27T07:32:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.479+0000 7f250ec5db80 10 client.18006 fetch_fsmap learned FSMap version 138 2024-05-27T07:32:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.479+0000 7f250ec5db80 1 -- 172.21.15.43:0/2677586749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=4+}) -- 0x561d3a6b8cc0 con 0x561d3a633db0 2024-05-27T07:32:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.480+0000 7f24df400700 1 -- 172.21.15.43:0/2677586749 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24ec00b6b0 con 0x561d3a633db0 2024-05-27T07:32:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.480+0000 7f24df400700 1 -- 172.21.15.43:0/2677586749 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 138) ==== 4191+0+0 (secure 0 0 0) 0x7f24ec00f990 con 0x561d3a633db0 2024-05-27T07:32:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.481+0000 7f250ec5db80 10 client.18006 fetch_fsmap finished waiting for FSMap version 138 2024-05-27T07:32:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.481+0000 7f250ec5db80 10 client.18006 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:32:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.481+0000 7f250ec5db80 20 client.18006 populate_metadata read hostname 'smithi043' 2024-05-27T07:32:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.481+0000 7f250ec5db80 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561d3a6b9320 0x561d3a6bb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:32:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.481+0000 7f250ec5db80 4 client.18006 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:32:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.481+0000 7f250ec5db80 1 -- 172.21.15.43:0/2677586749 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561d3a55d570 con 0x561d3a6b9320 2024-05-27T07:32:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.481+0000 7f24fce00700 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561d3a6b9320 0x561d3a6bb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:32:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.482+0000 7f24fce00700 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561d3a6b9320 0x561d3a6bb740 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e13d62da0557da31 server_cookie=e4d9a49406250ba9 in_seq=0 out_seq=0 2024-05-27T07:32:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.482+0000 7f24df400700 10 client.18006 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.482+0000 7f24df400700 1 -- 172.21.15.43:0/2677586749 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561d3a55d570 con 0x561d3a6b9320 2024-05-27T07:32:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.482+0000 7f24df400700 10 client.18006 handle_command_reply: tid=0 2024-05-27T07:32:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.482+0000 7f250ec5db80 1 client.18006 shutdown 2024-05-27T07:32:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.482+0000 7f250ec5db80 1 -- 172.21.15.43:0/2677586749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a633db0 msgr2=0x561d3a646230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.482+0000 7f250ec5db80 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a633db0 0x561d3a646230 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f24ec005910 tx=0x7f24ec005940 comp rx=0 tx=0).stop 2024-05-27T07:32:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.483+0000 7f250ec5db80 1 -- 172.21.15.43:0/2677586749 shutdown_connections 2024-05-27T07:32:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.483+0000 7f250ec5db80 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561d3a638e20 0x561d3a642ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.483+0000 7f250ec5db80 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561d3a4b4e40 0x561d3a645c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.483+0000 7f250ec5db80 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x561d3a6b9320 0x561d3a6bb740 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.483+0000 7f250ec5db80 1 --2- 172.21.15.43:0/2677586749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561d3a633db0 0x561d3a646230 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.483+0000 7f250ec5db80 1 -- 172.21.15.43:0/2677586749 >> 172.21.15.43:0/2677586749 conn(0x561d3a514fc0 msgr2=0x561d3a63db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.483+0000 7f24df400700 0 client.18006 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:32:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.483+0000 7f250ec5db80 1 -- 172.21.15.43:0/2677586749 shutdown_connections 2024-05-27T07:32:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.484+0000 7f250ec5db80 1 -- 172.21.15.43:0/2677586749 wait complete. 2024-05-27T07:32:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.484+0000 7f250ec5db80 20 client.18006 trim_cache size 0 max 16384 2024-05-27T07:32:59.478 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:32:59.478 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:32:59.478 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:32:59.478 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:32:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24e00b29f0 msgr2=0x7f24e00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24e00b29f0 0x7f24e00b4f00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f24f00e7e40 tx=0x7f24f0004f50 comp rx=0 tx=0).stop 2024-05-27T07:32:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24f810e8d0 msgr2=0x7f24f806eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:32:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24f810e8d0 0x7f24f806eb10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f24f0007830 tx=0x7f24f0007bf0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 shutdown_connections 2024-05-27T07:32:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f24e00b29f0 0x7f24e00b4f00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24f810e8d0 0x7f24f806eb10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24f81054f0 0x7f24f8075390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 --2- 172.21.15.43:0/658003545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24f8105e40 0x7f24f806e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:32:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 >> 172.21.15.43:0/658003545 conn(0x7f24f80fedb0 msgr2=0x7f24f80ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:32:59.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 shutdown_connections 2024-05-27T07:32:59.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:32:59.485+0000 7f2500200700 1 -- 172.21.15.43:0/658003545 wait complete. 2024-05-27T07:32:59.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 33334 5.88 MB/sec execute 180 sec latency 108.650 ms 2024-05-27T07:32:59.948 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:59 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:32:59.948 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:59 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:59.948 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:59 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:59.948 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:59 smithi043 ceph-mon[31531]: pgmap v640: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 3.8 MiB/s wr, 180 op/s 2024-05-27T07:32:59.948 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:59 smithi043 ceph-mon[31531]: pgmap v641: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.7 MiB/s wr, 223 op/s 2024-05-27T07:32:59.948 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:32:59 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:32:59.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:59 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:32:59.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:59 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:32:59.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:59 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:32:59.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:59 smithi179 ceph-mon[35888]: pgmap v640: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 3.8 MiB/s wr, 180 op/s 2024-05-27T07:32:59.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:59 smithi179 ceph-mon[35888]: pgmap v641: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.7 MiB/s wr, 223 op/s 2024-05-27T07:32:59.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:32:59 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:33:00.093 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:59 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:00.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:59 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:33:00.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:59 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:33:00.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:59 smithi132 ceph-mon[35283]: pgmap v640: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 3.8 MiB/s wr, 180 op/s 2024-05-27T07:33:00.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:59 smithi132 ceph-mon[35283]: pgmap v641: 129 pgs: 129 active+clean; 803 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.7 MiB/s wr, 223 op/s 2024-05-27T07:33:00.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:32:59 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:33:00.539 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:33:00.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:33:00.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 33397 5.87 MB/sec execute 181 sec latency 130.383 ms 2024-05-27T07:33:01.152 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:01.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 33487 5.86 MB/sec execute 182 sec latency 99.016 ms 2024-05-27T07:33:02.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:02 smithi043 ceph-mon[31531]: pgmap v642: 129 pgs: 129 active+clean; 817 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 5.3 MiB/s wr, 224 op/s 2024-05-27T07:33:02.211 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:02 smithi179 ceph-mon[35888]: pgmap v642: 129 pgs: 129 active+clean; 817 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 5.3 MiB/s wr, 224 op/s 2024-05-27T07:33:02.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:02 smithi132 ceph-mon[35283]: pgmap v642: 129 pgs: 129 active+clean; 817 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 5.3 MiB/s wr, 224 op/s 2024-05-27T07:33:02.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 33611 5.88 MB/sec execute 183 sec latency 107.853 ms 2024-05-27T07:33:03.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:03 smithi043 ceph-mon[31531]: osdmap e148: 12 total, 12 up, 12 in 2024-05-27T07:33:03.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:03 smithi132 ceph-mon[35283]: osdmap e148: 12 total, 12 up, 12 in 2024-05-27T07:33:03.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:03 smithi179 ceph-mon[35888]: osdmap e148: 12 total, 12 up, 12 in 2024-05-27T07:33:03.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 33724 5.91 MB/sec execute 184 sec latency 288.349 ms 2024-05-27T07:33:04.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:04 smithi132 ceph-mon[35283]: pgmap v644: 129 pgs: 129 active+clean; 817 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 3.8 MiB/s wr, 158 op/s 2024-05-27T07:33:04.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:04 smithi043 ceph-mon[31531]: pgmap v644: 129 pgs: 129 active+clean; 817 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 3.8 MiB/s wr, 158 op/s 2024-05-27T07:33:04.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.361+0000 7f58ba400700 1 -- 172.21.15.43:0/1823430398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58b4106eb0 msgr2=0x7f58b4113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:04.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.361+0000 7f58ba400700 1 --2- 172.21.15.43:0/1823430398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58b4106eb0 0x7f58b4113340 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00b0a0 tx=0x7f58ac00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.362+0000 7f58ba400700 1 -- 172.21.15.43:0/1823430398 shutdown_connections 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.362+0000 7f58ba400700 1 --2- 172.21.15.43:0/1823430398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58b4106eb0 0x7f58b4113340 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.362+0000 7f58ba400700 1 --2- 172.21.15.43:0/1823430398 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b410c5a0 0x7f58b410c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.362+0000 7f58ba400700 1 --2- 172.21.15.43:0/1823430398 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b4106560 0x7f58b4106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.362+0000 7f58ba400700 1 -- 172.21.15.43:0/1823430398 >> 172.21.15.43:0/1823430398 conn(0x7f58b4102040 msgr2=0x7f58b4104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.363+0000 7f58ba400700 1 -- 172.21.15.43:0/1823430398 shutdown_connections 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.363+0000 7f58ba400700 1 -- 172.21.15.43:0/1823430398 wait complete. 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.364+0000 7f58ba400700 1 Processor -- start 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.364+0000 7f58ba400700 1 -- start start 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.365+0000 7f58ba400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58b4106560 0x7f58b4074f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.365+0000 7f58ba400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b4106eb0 0x7f58b40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.365+0000 7f58ba400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b410c5a0 0x7f58b4112850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.365+0000 7f58ba400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f58b40793f0 con 0x7f58b4106eb0 2024-05-27T07:33:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.365+0000 7f58ba400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f58b4079270 con 0x7f58b4106560 2024-05-27T07:33:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.365+0000 7f58ba400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f58b4079570 con 0x7f58b410c5a0 2024-05-27T07:33:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.366+0000 7f58b3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58b4106560 0x7f58b4074f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.366+0000 7f58b3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58b4106560 0x7f58b4074f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:41458/0 (socket says 172.21.15.43:41458) 2024-05-27T07:33:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.366+0000 7f58b3400700 1 -- 172.21.15.43:0/1819725551 learned_addr learned my addr 172.21.15.43:0/1819725551 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.366+0000 7f58b3e00700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b410c5a0 0x7f58b4112850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.366+0000 7f58b3400700 1 -- 172.21.15.43:0/1819725551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b410c5a0 msgr2=0x7f58b4112850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.366+0000 7f58b3400700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b410c5a0 0x7f58b4112850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.366+0000 7f58b3400700 1 -- 172.21.15.43:0/1819725551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b4106eb0 msgr2=0x7f58b40754a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.366+0000 7f58b3400700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b4106eb0 0x7f58b40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.366+0000 7f58b3400700 1 -- 172.21.15.43:0/1819725551 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b4112ff0 con 0x7f58b4106560 2024-05-27T07:33:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.367+0000 7f58b3400700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58b4106560 0x7f58b4074f40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f58a400d980 tx=0x7f58a400dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.367+0000 7f58abe00700 1 -- 172.21.15.43:0/1819725551 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a4017070 con 0x7f58b4106560 2024-05-27T07:33:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.368+0000 7f58abe00700 1 -- 172.21.15.43:0/1819725551 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58a4010460 con 0x7f58b4106560 2024-05-27T07:33:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.368+0000 7f58ba400700 1 -- 172.21.15.43:0/1819725551 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b4113280 con 0x7f58b4106560 2024-05-27T07:33:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.368+0000 7f58abe00700 1 -- 172.21.15.43:0/1819725551 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a400de60 con 0x7f58b4106560 2024-05-27T07:33:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.368+0000 7f58ba400700 1 -- 172.21.15.43:0/1819725551 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b4070b60 con 0x7f58b4106560 2024-05-27T07:33:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.370+0000 7f58ba400700 1 -- 172.21.15.43:0/1819725551 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58b4107c80 con 0x7f58b4106560 2024-05-27T07:33:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.375+0000 7f58abe00700 1 -- 172.21.15.43:0/1819725551 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f58a4010610 con 0x7f58b4106560 2024-05-27T07:33:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.377+0000 7f58abe00700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f58980b2b20 0x7f58980b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.378+0000 7f58abe00700 1 -- 172.21.15.43:0/1819725551 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 9584+0+0 (secure 0 0 0) 0x7f58a40e24d0 con 0x7f58b4106560 2024-05-27T07:33:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.378+0000 7f58b2a00700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f58980b2b20 0x7f58980b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.379+0000 7f58abe00700 1 -- 172.21.15.43:0/1819725551 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f58a40a9ad0 con 0x7f58b4106560 2024-05-27T07:33:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.380+0000 7f58b2a00700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f58980b2b20 0x7f58980b5030 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f589c005fd0 tx=0x7f589c005d00 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:04.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:04 smithi179 ceph-mon[35888]: pgmap v644: 129 pgs: 129 active+clean; 817 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 3.8 MiB/s wr, 158 op/s 2024-05-27T07:33:04.792 INFO:tasks.workunit.client.0.smithi043.stdout: 10 33857 5.91 MB/sec execute 185 sec latency 111.079 ms 2024-05-27T07:33:04.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.861+0000 7f58ba400700 1 -- 172.21.15.43:0/1819725551 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58b4071650 con 0x7f58b4106560 2024-05-27T07:33:04.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.862+0000 7f58abe00700 1 -- 172.21.15.43:0/1819725551 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+6123 (secure 0 0 0) 0x7f58a40a93d0 con 0x7f58b4106560 2024-05-27T07:33:04.857 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:33:04.857 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":139,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:33:03.596758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34403,24484,34424,34484,34448]},"id":1}]} 2024-05-27T07:33:04.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.866+0000 7f58ba400700 1 -- 172.21.15.43:0/1819725551 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f58980b2b20 msgr2=0x7f58980b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:04.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.866+0000 7f58ba400700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f58980b2b20 0x7f58980b5030 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f589c005fd0 tx=0x7f589c005d00 comp rx=0 tx=0).stop 2024-05-27T07:33:04.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.866+0000 7f58ba400700 1 -- 172.21.15.43:0/1819725551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58b4106560 msgr2=0x7f58b4074f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:04.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.866+0000 7f58ba400700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58b4106560 0x7f58b4074f40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f58a400d980 tx=0x7f58a400dd40 comp rx=0 tx=0).stop 2024-05-27T07:33:04.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.867+0000 7f58ba400700 1 -- 172.21.15.43:0/1819725551 shutdown_connections 2024-05-27T07:33:04.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.867+0000 7f58ba400700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f58980b2b20 0x7f58980b5030 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:04.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.867+0000 7f58ba400700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b410c5a0 0x7f58b4112850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:04.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.867+0000 7f58ba400700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f58b4106560 0x7f58b4074f40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:04.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.867+0000 7f58ba400700 1 --2- 172.21.15.43:0/1819725551 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58b4106eb0 0x7f58b40754a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:04.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.867+0000 7f58ba400700 1 -- 172.21.15.43:0/1819725551 >> 172.21.15.43:0/1819725551 conn(0x7f58b4102040 msgr2=0x7f58b4108060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:04.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.867+0000 7f58ba400700 1 -- 172.21.15.43:0/1819725551 shutdown_connections 2024-05-27T07:33:04.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:04.867+0000 7f58ba400700 1 -- 172.21.15.43:0/1819725551 wait complete. 2024-05-27T07:33:04.864 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 139 2024-05-27T07:33:05.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:05 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:05.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:05 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1819725551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:33:05.343 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:05 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:05.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:05 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1819725551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:33:05.461 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:05 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:33:05.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:05 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1819725551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:33:05.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34001 5.91 MB/sec execute 186 sec latency 163.794 ms 2024-05-27T07:33:06.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:33:06.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:06 smithi043 ceph-mon[31531]: pgmap v645: 129 pgs: 129 active+clean; 817 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 3.8 MiB/s wr, 158 op/s 2024-05-27T07:33:06.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:06 smithi132 ceph-mon[35283]: pgmap v645: 129 pgs: 129 active+clean; 817 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 3.8 MiB/s wr, 158 op/s 2024-05-27T07:33:06.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:06 smithi179 ceph-mon[35888]: pgmap v645: 129 pgs: 129 active+clean; 817 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 3.8 MiB/s wr, 158 op/s 2024-05-27T07:33:06.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34077 5.89 MB/sec execute 187 sec latency 108.056 ms 2024-05-27T07:33:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:07.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34144 5.88 MB/sec execute 188 sec latency 426.732 ms 2024-05-27T07:33:08.455 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:08 smithi043 ceph-mon[31531]: pgmap v646: 129 pgs: 129 active+clean; 817 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 5.8 MiB/s wr, 223 op/s 2024-05-27T07:33:08.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:08 smithi132 ceph-mon[35283]: pgmap v646: 129 pgs: 129 active+clean; 817 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 5.8 MiB/s wr, 223 op/s 2024-05-27T07:33:08.711 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:08 smithi179 ceph-mon[35888]: pgmap v646: 129 pgs: 129 active+clean; 817 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 5.8 MiB/s wr, 223 op/s 2024-05-27T07:33:08.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34234 5.87 MB/sec execute 189 sec latency 83.846 ms 2024-05-27T07:33:09.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.205+0000 7f3256e00700 1 -- 172.21.15.43:0/91639407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3250105e40 msgr2=0x7f325010e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.205+0000 7f3256e00700 1 --2- 172.21.15.43:0/91639407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3250105e40 0x7f325010e390 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f323c009990 tx=0x7f323c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.206+0000 7f3256e00700 1 -- 172.21.15.43:0/91639407 shutdown_connections 2024-05-27T07:33:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.206+0000 7f3256e00700 1 --2- 172.21.15.43:0/91639407 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f325010e8d0 0x7f3250110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.206+0000 7f3256e00700 1 --2- 172.21.15.43:0/91639407 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32501054f0 0x7f3250105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.206+0000 7f3256e00700 1 --2- 172.21.15.43:0/91639407 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3250105e40 0x7f325010e390 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.206+0000 7f3256e00700 1 -- 172.21.15.43:0/91639407 >> 172.21.15.43:0/91639407 conn(0x7f32500fedb0 msgr2=0x7f32501011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:09.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.206+0000 7f3256e00700 1 -- 172.21.15.43:0/91639407 shutdown_connections 2024-05-27T07:33:09.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.206+0000 7f3256e00700 1 -- 172.21.15.43:0/91639407 wait complete. 2024-05-27T07:33:09.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.208+0000 7f3256e00700 1 Processor -- start 2024-05-27T07:33:09.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.208+0000 7f3256e00700 1 -- start start 2024-05-27T07:33:09.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f3256e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32501054f0 0x7f325010de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f3256e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3250105e40 0x7f325010afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f3256e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f325010e8d0 0x7f325010b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f3256e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3250104320 con 0x7f3250105e40 2024-05-27T07:33:09.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f3256e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32501041a0 con 0x7f32501054f0 2024-05-27T07:33:09.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f3256e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f32501044a0 con 0x7f325010e8d0 2024-05-27T07:33:09.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f324fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32501054f0 0x7f325010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f324fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32501054f0 0x7f325010de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40678/0 (socket says 172.21.15.43:40678) 2024-05-27T07:33:09.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f324fe00700 1 -- 172.21.15.43:0/663650938 learned_addr learned my addr 172.21.15.43:0/663650938 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:09.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f3254e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f325010e8d0 0x7f325010b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.209+0000 7f324f400700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3250105e40 0x7f325010afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.210+0000 7f3254e00700 1 -- 172.21.15.43:0/663650938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32501054f0 msgr2=0x7f325010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:09.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.210+0000 7f3254e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32501054f0 0x7f325010de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.210+0000 7f3254e00700 1 -- 172.21.15.43:0/663650938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3250105e40 msgr2=0x7f325010afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.210+0000 7f3254e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3250105e40 0x7f325010afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.210+0000 7f3254e00700 1 -- 172.21.15.43:0/663650938 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325010ba50 con 0x7f325010e8d0 2024-05-27T07:33:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.211+0000 7f3254e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f325010e8d0 0x7f325010b510 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3248007f10 tx=0x7f324800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.212+0000 7f324cc00700 1 -- 172.21.15.43:0/663650938 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3248004db0 con 0x7f325010e8d0 2024-05-27T07:33:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.212+0000 7f324cc00700 1 -- 172.21.15.43:0/663650938 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3248009e20 con 0x7f325010e8d0 2024-05-27T07:33:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.212+0000 7f324cc00700 1 -- 172.21.15.43:0/663650938 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3248015400 con 0x7f325010e8d0 2024-05-27T07:33:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.212+0000 7f3256e00700 1 -- 172.21.15.43:0/663650938 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f325010bd40 con 0x7f325010e8d0 2024-05-27T07:33:09.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.212+0000 7f3256e00700 1 -- 172.21.15.43:0/663650938 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32501adc80 con 0x7f325010e8d0 2024-05-27T07:33:09.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.218+0000 7f324cc00700 1 -- 172.21.15.43:0/663650938 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f3248005240 con 0x7f325010e8d0 2024-05-27T07:33:09.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.219+0000 7f326585bb80 1 Processor -- start 2024-05-27T07:33:09.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.219+0000 7f326585bb80 1 -- start start 2024-05-27T07:33:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.219+0000 7f326585bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc1fcde40 0x55ecc21527f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.220+0000 7f326585bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 0x55ecc214c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.220+0000 7f326585bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 0x55ecc214d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.220+0000 7f326585bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ecc207ae40 con 0x55ecc214cdb0 2024-05-27T07:33:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.220+0000 7f326585bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ecc206a9e0 con 0x55ecc1fcde40 2024-05-27T07:33:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.220+0000 7f326585bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ecc206ea20 con 0x55ecc2152d30 2024-05-27T07:33:09.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.220+0000 7f324fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 0x55ecc214c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.221+0000 7f324fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 0x55ecc214c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:42388/0 (socket says 172.21.15.43:42388) 2024-05-27T07:33:09.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.221+0000 7f324fe00700 1 -- 172.21.15.43:0/2076291010 learned_addr learned my addr 172.21.15.43:0/2076291010 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:09.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.221+0000 7f324f400700 1 --2- 172.21.15.43:0/2076291010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc1fcde40 0x55ecc21527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.221+0000 7f324f400700 1 --2- 172.21.15.43:0/2076291010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 0x55ecc214d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.221+0000 7f324cc00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32340b2a70 0x7f32340b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.221+0000 7f324fe00700 1 -- 172.21.15.43:0/2076291010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc1fcde40 msgr2=0x55ecc21527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:09.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.221+0000 7f324fe00700 1 --2- 172.21.15.43:0/2076291010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc1fcde40 0x55ecc21527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.221+0000 7f324fe00700 1 -- 172.21.15.43:0/2076291010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 msgr2=0x55ecc214d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:09.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.221+0000 7f324fe00700 1 --2- 172.21.15.43:0/2076291010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 0x55ecc214d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.221+0000 7f324fe00700 1 -- 172.21.15.43:0/2076291010 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecc2033420 con 0x55ecc2152d30 2024-05-27T07:33:09.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.222+0000 7f324cc00700 1 -- 172.21.15.43:0/663650938 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 9584+0+0 (secure 0 0 0) 0x7f32480e2300 con 0x7f325010e8d0 2024-05-27T07:33:09.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.222+0000 7f324fe00700 1 --2- 172.21.15.43:0/2076291010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 0x55ecc214c870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3240009ea0 tx=0x7f3240011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=318bf18eef10b675 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:09.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.222+0000 7f3254e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32340b2a70 0x7f32340b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.223+0000 7f3245600700 1 -- 172.21.15.43:0/2076291010 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324001e070 con 0x55ecc2152d30 2024-05-27T07:33:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.223+0000 7f3245600700 1 -- 172.21.15.43:0/2076291010 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f324000fb10 con 0x55ecc2152d30 2024-05-27T07:33:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.223+0000 7f3245600700 1 -- 172.21.15.43:0/2076291010 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240019bb0 con 0x55ecc2152d30 2024-05-27T07:33:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.224+0000 7f326585bb80 1 -- 172.21.15.43:0/2076291010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 msgr2=0x55ecc214c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.224+0000 7f326585bb80 1 --2- 172.21.15.43:0/2076291010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 0x55ecc214c870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3240009ea0 tx=0x7f3240011720 comp rx=0 tx=0).stop 2024-05-27T07:33:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.224+0000 7f3254e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32340b2a70 0x7f32340b4f80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3248004b80 tx=0x7f3248004ad0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.224+0000 7f326585bb80 1 -- 172.21.15.43:0/2076291010 shutdown_connections 2024-05-27T07:33:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.224+0000 7f326585bb80 1 --2- 172.21.15.43:0/2076291010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 0x55ecc214d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.224+0000 7f326585bb80 1 --2- 172.21.15.43:0/2076291010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc1fcde40 0x55ecc21527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.224+0000 7f326585bb80 1 --2- 172.21.15.43:0/2076291010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 0x55ecc214c870 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.224+0000 7f326585bb80 1 -- 172.21.15.43:0/2076291010 >> 172.21.15.43:0/2076291010 conn(0x55ecc202dfc0 msgr2=0x55ecc21478d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:09.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.227+0000 7f326585bb80 1 -- 172.21.15.43:0/2076291010 shutdown_connections 2024-05-27T07:33:09.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.227+0000 7f326585bb80 1 -- 172.21.15.43:0/2076291010 wait complete. 2024-05-27T07:33:09.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.228+0000 7f326585bb80 1 -- start start 2024-05-27T07:33:09.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.228+0000 7f326585bb80 1 Processor -- start 2024-05-27T07:33:09.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.229+0000 7f326585bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc1fcde40 0x55ecc2156510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.229+0000 7f326585bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc214cdb0 0x55ecc2156a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.229+0000 7f326585bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc2152d30 0x55ecc2156f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.229+0000 7f326585bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ecc1f70fa0 con 0x55ecc2152d30 2024-05-27T07:33:09.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.229+0000 7f326585bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ecc200b760 con 0x55ecc1fcde40 2024-05-27T07:33:09.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.229+0000 7f326585bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ecc2159800 con 0x55ecc214cdb0 2024-05-27T07:33:09.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.229+0000 7f326585bb80 10 client.0 _ll_register_callbacks cb 0x55ecc2014c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:09.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.230+0000 7f324fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc214cdb0 0x55ecc2156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.230+0000 7f324fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc214cdb0 0x55ecc2156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:42398/0 (socket says 172.21.15.43:42398) 2024-05-27T07:33:09.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.230+0000 7f324fe00700 1 -- 172.21.15.43:0/568694891 learned_addr learned my addr 172.21.15.43:0/568694891 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:09.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.230+0000 7f324f400700 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc1fcde40 0x55ecc2156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.231+0000 7f324f400700 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc2152d30 0x55ecc2156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.232+0000 7f324fe00700 1 -- 172.21.15.43:0/568694891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc1fcde40 msgr2=0x55ecc2156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:09.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.232+0000 7f324fe00700 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc1fcde40 0x55ecc2156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.232+0000 7f324fe00700 1 -- 172.21.15.43:0/568694891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc2152d30 msgr2=0x55ecc2156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:09.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.232+0000 7f324fe00700 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc2152d30 0x55ecc2156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.232+0000 7f324fe00700 1 -- 172.21.15.43:0/568694891 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecc1fba8d0 con 0x55ecc214cdb0 2024-05-27T07:33:09.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.232+0000 7f324f400700 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc2152d30 0x55ecc2156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:09.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.233+0000 7f324fe00700 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc214cdb0 0x55ecc2156a50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3240000c00 tx=0x7f324000f6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15429c95bb625cd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:09.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.233+0000 7f323a000700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:33:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.234+0000 7f323a000700 1 -- 172.21.15.43:0/568694891 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324001e030 con 0x55ecc214cdb0 2024-05-27T07:33:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.234+0000 7f323a000700 1 -- 172.21.15.43:0/568694891 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9584+0+0 (secure 0 0 0) 0x7f3240004260 con 0x55ecc214cdb0 2024-05-27T07:33:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.235+0000 7f326585bb80 1 -- 172.21.15.43:0/568694891 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f324000fb10 con 0x55ecc214cdb0 2024-05-27T07:33:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.241+0000 7f323a000700 1 -- 172.21.15.43:0/568694891 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f324000fe20 con 0x55ecc214cdb0 2024-05-27T07:33:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.241+0000 7f326585bb80 10 client.27463 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.241+0000 7f326585bb80 1 -- 172.21.15.43:0/568694891 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x55ecc203b060 con 0x55ecc214cdb0 2024-05-27T07:33:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.242+0000 7f323a000700 1 -- 172.21.15.43:0/568694891 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f324000fb10 con 0x55ecc214cdb0 2024-05-27T07:33:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.242+0000 7f323a000700 1 -- 172.21.15.43:0/568694891 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f32400198d0 con 0x55ecc214cdb0 2024-05-27T07:33:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.243+0000 7f326585bb80 10 client.27463 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.243+0000 7f326585bb80 10 client.27463 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:33:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.243+0000 7f326585bb80 20 client.27463 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.243+0000 7f326585bb80 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ecc2166a00 0x55ecc2168e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.243+0000 7f326585bb80 4 client.27463 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.243+0000 7f326585bb80 1 -- 172.21.15.43:0/568694891 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecc207cd90 con 0x55ecc2166a00 2024-05-27T07:33:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.244+0000 7f324f400700 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ecc2166a00 0x55ecc2168e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.245+0000 7f323a000700 10 client.27463 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:33:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.245+0000 7f324f400700 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ecc2166a00 0x55ecc2168e50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=244f33a38652404e server_cookie=2e0adcdc5e295fc4 in_seq=0 out_seq=0 2024-05-27T07:33:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.246+0000 7f323a000700 1 -- 172.21.15.43:0/568694891 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ecc207cd90 con 0x55ecc2166a00 2024-05-27T07:33:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.246+0000 7f323a000700 10 client.27463 handle_command_reply: tid=0 2024-05-27T07:33:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.246+0000 7f326585bb80 1 client.27463 shutdown 2024-05-27T07:33:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.247+0000 7f326585bb80 1 -- 172.21.15.43:0/568694891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc214cdb0 msgr2=0x55ecc2156a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.247+0000 7f326585bb80 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc214cdb0 0x55ecc2156a50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3240000c00 tx=0x7f324000f6b0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.248+0000 7f326585bb80 1 -- 172.21.15.43:0/568694891 shutdown_connections 2024-05-27T07:33:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.248+0000 7f326585bb80 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ecc2166a00 0x55ecc2168e50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.248+0000 7f326585bb80 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc2152d30 0x55ecc2156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.248+0000 7f326585bb80 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc1fcde40 0x55ecc2156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.248+0000 7f326585bb80 1 --2- 172.21.15.43:0/568694891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc214cdb0 0x55ecc2156a50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.248+0000 7f326585bb80 1 -- 172.21.15.43:0/568694891 >> 172.21.15.43:0/568694891 conn(0x55ecc202dfc0 msgr2=0x55ecc2147580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.248+0000 7f323a000700 0 client.27463 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:33:09.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.248+0000 7f326585bb80 1 -- 172.21.15.43:0/568694891 shutdown_connections 2024-05-27T07:33:09.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.248+0000 7f326585bb80 1 -- 172.21.15.43:0/568694891 wait complete. 2024-05-27T07:33:09.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.249+0000 7f326585bb80 20 client.27463 trim_cache size 0 max 16384 2024-05-27T07:33:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.275+0000 7f326585bb80 1 Processor -- start 2024-05-27T07:33:09.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.276+0000 7f326585bb80 1 -- start start 2024-05-27T07:33:09.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.276+0000 7f326585bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc1fcde40 0x55ecc214c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.276+0000 7f326585bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc214cdb0 0x55ecc2155850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.276+0000 7f326585bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 0x55ecc21589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.276+0000 7f326585bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32501b7230 con 0x55ecc1fcde40 2024-05-27T07:33:09.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.276+0000 7f326585bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32501b5690 con 0x55ecc214cdb0 2024-05-27T07:33:09.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.276+0000 7f326585bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f32501b6b60 con 0x55ecc2152d30 2024-05-27T07:33:09.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.277+0000 7f324fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc214cdb0 0x55ecc2155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.277+0000 7f324fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc214cdb0 0x55ecc2155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40698/0 (socket says 172.21.15.43:40698) 2024-05-27T07:33:09.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.277+0000 7f324fe00700 1 -- 172.21.15.43:0/3364457687 learned_addr learned my addr 172.21.15.43:0/3364457687 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:09.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.277+0000 7f324f400700 1 --2- 172.21.15.43:0/3364457687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc1fcde40 0x55ecc214c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.277+0000 7f324fe00700 1 -- 172.21.15.43:0/3364457687 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 msgr2=0x55ecc21589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.277+0000 7f324fe00700 1 --2- 172.21.15.43:0/3364457687 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 0x55ecc21589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.277+0000 7f324fe00700 1 -- 172.21.15.43:0/3364457687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc1fcde40 msgr2=0x55ecc214c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.278+0000 7f324fe00700 1 --2- 172.21.15.43:0/3364457687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc1fcde40 0x55ecc214c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.278+0000 7f324fe00700 1 -- 172.21.15.43:0/3364457687 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecc2157300 con 0x55ecc214cdb0 2024-05-27T07:33:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.278+0000 7f324f400700 1 --2- 172.21.15.43:0/3364457687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc1fcde40 0x55ecc214c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.278+0000 7f324fe00700 1 --2- 172.21.15.43:0/3364457687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc214cdb0 0x55ecc2155850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3240005e90 tx=0x7f3240009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb91344385bb7a31 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.278+0000 7f3244c00700 1 -- 172.21.15.43:0/3364457687 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324001e030 con 0x55ecc214cdb0 2024-05-27T07:33:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.279+0000 7f3244c00700 1 -- 172.21.15.43:0/3364457687 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f324001f040 con 0x55ecc214cdb0 2024-05-27T07:33:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.279+0000 7f3244c00700 1 -- 172.21.15.43:0/3364457687 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240019740 con 0x55ecc214cdb0 2024-05-27T07:33:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.280+0000 7f326585bb80 1 -- 172.21.15.43:0/3364457687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc214cdb0 msgr2=0x55ecc2155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.280+0000 7f326585bb80 1 --2- 172.21.15.43:0/3364457687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc214cdb0 0x55ecc2155850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3240005e90 tx=0x7f3240009900 comp rx=0 tx=0).stop 2024-05-27T07:33:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.280+0000 7f326585bb80 1 -- 172.21.15.43:0/3364457687 shutdown_connections 2024-05-27T07:33:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.280+0000 7f326585bb80 1 --2- 172.21.15.43:0/3364457687 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc2152d30 0x55ecc21589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.280+0000 7f326585bb80 1 --2- 172.21.15.43:0/3364457687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc1fcde40 0x55ecc214c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.280+0000 7f326585bb80 1 --2- 172.21.15.43:0/3364457687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc214cdb0 0x55ecc2155850 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.280+0000 7f326585bb80 1 -- 172.21.15.43:0/3364457687 >> 172.21.15.43:0/3364457687 conn(0x55ecc202dfc0 msgr2=0x55ecc2150140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.280+0000 7f326585bb80 1 -- 172.21.15.43:0/3364457687 shutdown_connections 2024-05-27T07:33:09.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.280+0000 7f326585bb80 1 -- 172.21.15.43:0/3364457687 wait complete. 2024-05-27T07:33:09.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.281+0000 7f326585bb80 1 -- start start 2024-05-27T07:33:09.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.281+0000 7f326585bb80 1 Processor -- start 2024-05-27T07:33:09.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.282+0000 7f326585bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc1fcde40 0x55ecc215ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.282+0000 7f326585bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 0x55ecc215ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.282+0000 7f326585bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc2152d30 0x55ecc216d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.282+0000 7f326585bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32501b6490 con 0x55ecc214cdb0 2024-05-27T07:33:09.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.282+0000 7f326585bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32501b5d60 con 0x55ecc2152d30 2024-05-27T07:33:09.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.282+0000 7f326585bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ecc207cd90 con 0x55ecc1fcde40 2024-05-27T07:33:09.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.282+0000 7f326585bb80 10 client.0 _ll_register_callbacks cb 0x55ecc2014c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:09.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.283+0000 7f324fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 0x55ecc215ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.283+0000 7f324fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 0x55ecc215ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55818/0 (socket says 172.21.15.43:55818) 2024-05-27T07:33:09.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.283+0000 7f324fe00700 1 -- 172.21.15.43:0/3237344706 learned_addr learned my addr 172.21.15.43:0/3237344706 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:09.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.283+0000 7f324f400700 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc1fcde40 0x55ecc215ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.283+0000 7f324f400700 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc2152d30 0x55ecc216d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.284+0000 7f324fe00700 1 -- 172.21.15.43:0/3237344706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc1fcde40 msgr2=0x55ecc215ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:09.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.284+0000 7f324fe00700 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc1fcde40 0x55ecc215ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.284+0000 7f324fe00700 1 -- 172.21.15.43:0/3237344706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc2152d30 msgr2=0x55ecc216d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:09.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.284+0000 7f324fe00700 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc2152d30 0x55ecc216d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.284+0000 7f324fe00700 1 -- 172.21.15.43:0/3237344706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecc216dbc0 con 0x55ecc214cdb0 2024-05-27T07:33:09.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.284+0000 7f324fe00700 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 0x55ecc215ef90 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x55ecc213fc20 tx=0x7f3240004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afd25f7f79c97ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:09.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.284+0000 7f323a000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:33:09.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.285+0000 7f323a000700 1 -- 172.21.15.43:0/3237344706 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324003d060 con 0x55ecc214cdb0 2024-05-27T07:33:09.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.285+0000 7f323a000700 1 -- 172.21.15.43:0/3237344706 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(148..148 src has 1..148) ==== 9584+0+0 (secure 0 0 0) 0x7f324000f9e0 con 0x55ecc214cdb0 2024-05-27T07:33:09.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.286+0000 7f326585bb80 1 -- 172.21.15.43:0/3237344706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f324001f040 con 0x55ecc214cdb0 2024-05-27T07:33:09.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.286+0000 7f323a000700 1 -- 172.21.15.43:0/3237344706 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f3240044050 con 0x55ecc214cdb0 2024-05-27T07:33:09.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.286+0000 7f326585bb80 10 client.18021 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:09.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.286+0000 7f326585bb80 1 -- 172.21.15.43:0/3237344706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x55ecc21d0c60 con 0x55ecc214cdb0 2024-05-27T07:33:09.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.287+0000 7f323a000700 1 -- 172.21.15.43:0/3237344706 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f324001f040 con 0x55ecc214cdb0 2024-05-27T07:33:09.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.287+0000 7f323a000700 1 -- 172.21.15.43:0/3237344706 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f3240034030 con 0x55ecc214cdb0 2024-05-27T07:33:09.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.287+0000 7f326585bb80 10 client.18021 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:09.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.287+0000 7f326585bb80 10 client.18021 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:33:09.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.287+0000 7f326585bb80 20 client.18021 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:09.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.287+0000 7f326585bb80 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ecc21d1180 0x55ecc21d35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:09.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.287+0000 7f326585bb80 4 client.18021 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:09.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.287+0000 7f326585bb80 1 -- 172.21.15.43:0/3237344706 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ecc2076570 con 0x55ecc21d1180 2024-05-27T07:33:09.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.288+0000 7f324f400700 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ecc21d1180 0x55ecc21d35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:09.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.289+0000 7f324f400700 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ecc21d1180 0x55ecc21d35a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dd32b4eae3400050 server_cookie=68bac11956cd4fe5 in_seq=0 out_seq=0 2024-05-27T07:33:09.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.289+0000 7f323a000700 10 client.18021 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:33:09.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f323a000700 1 -- 172.21.15.43:0/3237344706 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ecc2076570 con 0x55ecc21d1180 2024-05-27T07:33:09.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f323a000700 10 client.18021 handle_command_reply: tid=0 2024-05-27T07:33:09.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f326585bb80 1 client.18021 shutdown 2024-05-27T07:33:09.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f326585bb80 1 -- 172.21.15.43:0/3237344706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 msgr2=0x55ecc215ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:09.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f326585bb80 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 0x55ecc215ef90 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x55ecc213fc20 tx=0x7f3240004510 comp rx=0 tx=0).stop 2024-05-27T07:33:09.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f326585bb80 1 -- 172.21.15.43:0/3237344706 shutdown_connections 2024-05-27T07:33:09.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f326585bb80 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55ecc21d1180 0x55ecc21d35a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f326585bb80 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ecc2152d30 0x55ecc216d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f326585bb80 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ecc1fcde40 0x55ecc215ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f326585bb80 1 --2- 172.21.15.43:0/3237344706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecc214cdb0 0x55ecc215ef90 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f323a000700 0 client.18021 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:33:09.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.291+0000 7f326585bb80 1 -- 172.21.15.43:0/3237344706 >> 172.21.15.43:0/3237344706 conn(0x55ecc202dfc0 msgr2=0x55ecc21555b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:09.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.292+0000 7f326585bb80 1 -- 172.21.15.43:0/3237344706 shutdown_connections 2024-05-27T07:33:09.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.292+0000 7f326585bb80 1 -- 172.21.15.43:0/3237344706 wait complete. 2024-05-27T07:33:09.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.292+0000 7f326585bb80 20 client.18021 trim_cache size 0 max 16384 2024-05-27T07:33:09.286 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:33:09.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 -- 172.21.15.43:0/663650938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32340b2a70 msgr2=0x7f32340b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:09.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32340b2a70 0x7f32340b4f80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3248004b80 tx=0x7f3248004ad0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 -- 172.21.15.43:0/663650938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f325010e8d0 msgr2=0x7f325010b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:09.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f325010e8d0 0x7f325010b510 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3248007f10 tx=0x7f324800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 -- 172.21.15.43:0/663650938 shutdown_connections 2024-05-27T07:33:09.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f32340b2a70 0x7f32340b4f80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f325010e8d0 0x7f325010b510 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32501054f0 0x7f325010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 --2- 172.21.15.43:0/663650938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3250105e40 0x7f325010afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:09.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 -- 172.21.15.43:0/663650938 >> 172.21.15.43:0/663650938 conn(0x7f32500fedb0 msgr2=0x7f32501019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:09.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 -- 172.21.15.43:0/663650938 shutdown_connections 2024-05-27T07:33:09.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:09.293+0000 7f3256e00700 1 -- 172.21.15.43:0/663650938 wait complete. 2024-05-27T07:33:09.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34350 5.88 MB/sec execute 190 sec latency 86.305 ms 2024-05-27T07:33:10.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:10 smithi132 ceph-mon[35283]: pgmap v647: 129 pgs: 129 active+clean; 817 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 4.6 MiB/s wr, 178 op/s 2024-05-27T07:33:10.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:10 smithi043 ceph-mon[31531]: pgmap v647: 129 pgs: 129 active+clean; 817 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 4.6 MiB/s wr, 178 op/s 2024-05-27T07:33:10.712 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:10 smithi179 ceph-mon[35888]: pgmap v647: 129 pgs: 129 active+clean; 817 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 4.6 MiB/s wr, 178 op/s 2024-05-27T07:33:10.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34493 5.92 MB/sec execute 191 sec latency 84.464 ms 2024-05-27T07:33:11.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34595 5.92 MB/sec execute 192 sec latency 93.130 ms 2024-05-27T07:33:12.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34662 5.91 MB/sec execute 193 sec latency 426.251 ms 2024-05-27T07:33:12.843 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:12 smithi132 ceph-mon[35283]: pgmap v648: 129 pgs: 129 active+clean; 833 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 178 op/s 2024-05-27T07:33:12.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:12 smithi132 ceph-mon[35283]: osdmap e149: 12 total, 12 up, 12 in 2024-05-27T07:33:12.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:12 smithi043 ceph-mon[31531]: pgmap v648: 129 pgs: 129 active+clean; 833 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 178 op/s 2024-05-27T07:33:12.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:12 smithi043 ceph-mon[31531]: osdmap e149: 12 total, 12 up, 12 in 2024-05-27T07:33:12.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:12 smithi179 ceph-mon[35888]: pgmap v648: 129 pgs: 129 active+clean; 833 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 178 op/s 2024-05-27T07:33:12.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:12 smithi179 ceph-mon[35888]: osdmap e149: 12 total, 12 up, 12 in 2024-05-27T07:33:13.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34761 5.88 MB/sec execute 194 sec latency 91.751 ms 2024-05-27T07:33:14.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34826 5.86 MB/sec execute 195 sec latency 262.316 ms 2024-05-27T07:33:14.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:14 smithi132 ceph-mon[35283]: pgmap v650: 129 pgs: 129 active+clean; 833 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 178 op/s 2024-05-27T07:33:14.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:14 smithi043 ceph-mon[31531]: pgmap v650: 129 pgs: 129 active+clean; 833 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 178 op/s 2024-05-27T07:33:14.961 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:14 smithi179 ceph-mon[35888]: pgmap v650: 129 pgs: 129 active+clean; 833 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 178 op/s 2024-05-27T07:33:15.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34869 5.85 MB/sec execute 196 sec latency 476.653 ms 2024-05-27T07:33:16.706 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:16 smithi043 ceph-mon[31531]: pgmap v651: 129 pgs: 129 active+clean; 833 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 178 op/s 2024-05-27T07:33:16.764 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:16 smithi179 ceph-mon[35888]: pgmap v651: 129 pgs: 129 active+clean; 833 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 178 op/s 2024-05-27T07:33:16.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:33:16.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 34970 5.82 MB/sec execute 197 sec latency 77.446 ms 2024-05-27T07:33:16.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:16 smithi132 ceph-mon[35283]: pgmap v651: 129 pgs: 129 active+clean; 833 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 4.1 MiB/s wr, 178 op/s 2024-05-27T07:33:17.332 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:17.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 35058 5.80 MB/sec execute 198 sec latency 141.578 ms 2024-05-27T07:33:18.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:18 smithi043 ceph-mon[31531]: pgmap v652: 129 pgs: 129 active+clean; 840 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 3.2 MiB/s wr, 178 op/s 2024-05-27T07:33:18.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 35183 5.79 MB/sec execute 199 sec latency 86.977 ms 2024-05-27T07:33:18.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:18 smithi132 ceph-mon[35283]: pgmap v652: 129 pgs: 129 active+clean; 840 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 3.2 MiB/s wr, 178 op/s 2024-05-27T07:33:18.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:18 smithi179 ceph-mon[35888]: pgmap v652: 129 pgs: 129 active+clean; 840 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 3.2 MiB/s wr, 178 op/s 2024-05-27T07:33:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.659+0000 7ff5a6800700 1 -- 172.21.15.43:0/2932768554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5a0115360 msgr2=0x7ff5a0117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.659+0000 7ff5a6800700 1 --2- 172.21.15.43:0/2932768554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5a0115360 0x7ff5a0117780 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7ff59800b0a0 tx=0x7ff59800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.659+0000 7ff5a6800700 1 -- 172.21.15.43:0/2932768554 shutdown_connections 2024-05-27T07:33:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.659+0000 7ff5a6800700 1 --2- 172.21.15.43:0/2932768554 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5a0115360 0x7ff5a0117780 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.659+0000 7ff5a6800700 1 --2- 172.21.15.43:0/2932768554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a0075b50 0x7ff5a0075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.659+0000 7ff5a6800700 1 --2- 172.21.15.43:0/2932768554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5a00764a0 0x7ff5a0114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.659+0000 7ff5a6800700 1 -- 172.21.15.43:0/2932768554 >> 172.21.15.43:0/2932768554 conn(0x7ff5a0100030 msgr2=0x7ff5a0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.659+0000 7ff5a6800700 1 -- 172.21.15.43:0/2932768554 shutdown_connections 2024-05-27T07:33:19.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.659+0000 7ff5a6800700 1 -- 172.21.15.43:0/2932768554 wait complete. 2024-05-27T07:33:19.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.660+0000 7ff5a6800700 1 Processor -- start 2024-05-27T07:33:19.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.661+0000 7ff5a6800700 1 -- start start 2024-05-27T07:33:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.661+0000 7ff5a6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5a0075b50 0x7ff5a01a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.661+0000 7ff5a6800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a00764a0 0x7ff5a01a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.662+0000 7ff5a6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5a0115360 0x7ff5a01a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.662+0000 7ff5a6800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a011a910 con 0x7ff5a0115360 2024-05-27T07:33:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.662+0000 7ff5a6800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a011a790 con 0x7ff5a0075b50 2024-05-27T07:33:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.662+0000 7ff5a6800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a011aa90 con 0x7ff5a00764a0 2024-05-27T07:33:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.662+0000 7ff59f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5a0075b50 0x7ff5a01a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.662+0000 7ff59f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5a0075b50 0x7ff5a01a12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:34094/0 (socket says 172.21.15.43:34094) 2024-05-27T07:33:19.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.662+0000 7ff59f400700 1 -- 172.21.15.43:0/1888485208 learned_addr learned my addr 172.21.15.43:0/1888485208 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:19.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.663+0000 7ff59fe00700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5a0115360 0x7ff5a01a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.663+0000 7ff59ea00700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a00764a0 0x7ff5a01a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.663+0000 7ff59fe00700 1 -- 172.21.15.43:0/1888485208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a00764a0 msgr2=0x7ff5a01a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.663+0000 7ff59fe00700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a00764a0 0x7ff5a01a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.664+0000 7ff59fe00700 1 -- 172.21.15.43:0/1888485208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5a0075b50 msgr2=0x7ff5a01a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.664+0000 7ff59fe00700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5a0075b50 0x7ff5a01a12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.664+0000 7ff59fe00700 1 -- 172.21.15.43:0/1888485208 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a01a61b0 con 0x7ff5a0115360 2024-05-27T07:33:19.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.664+0000 7ff59f400700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5a0075b50 0x7ff5a01a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:19.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.664+0000 7ff59fe00700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5a0115360 0x7ff5a01a5b10 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7ff59800b0a0 tx=0x7ff598004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:19.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.665+0000 7ff597e00700 1 -- 172.21.15.43:0/1888485208 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598021070 con 0x7ff5a0115360 2024-05-27T07:33:19.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.665+0000 7ff597e00700 1 -- 172.21.15.43:0/1888485208 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff598007b80 con 0x7ff5a0115360 2024-05-27T07:33:19.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.665+0000 7ff597e00700 1 -- 172.21.15.43:0/1888485208 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598005040 con 0x7ff5a0115360 2024-05-27T07:33:19.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.665+0000 7ff5a6800700 1 -- 172.21.15.43:0/1888485208 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a01a6440 con 0x7ff5a0115360 2024-05-27T07:33:19.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.665+0000 7ff5a6800700 1 -- 172.21.15.43:0/1888485208 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a01a68d0 con 0x7ff5a0115360 2024-05-27T07:33:19.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.672+0000 7ff597e00700 1 -- 172.21.15.43:0/1888485208 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff598024070 con 0x7ff5a0115360 2024-05-27T07:33:19.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.672+0000 7ff5b53a4b80 1 Processor -- start 2024-05-27T07:33:19.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.673+0000 7ff5b53a4b80 1 -- start start 2024-05-27T07:33:19.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.676+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa73878e40 0x55aa739fd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.676+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 0x55aa739f7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.676+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 0x55aa739f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.676+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55aa73925e40 con 0x55aa73878e40 2024-05-27T07:33:19.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.676+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55aa739159e0 con 0x55aa739fdd30 2024-05-27T07:33:19.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.676+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55aa73919a20 con 0x55aa739f7db0 2024-05-27T07:33:19.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.676+0000 7ff597e00700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5840b2a70 0x7ff5840b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.676+0000 7ff597e00700 1 -- 172.21.15.43:0/1888485208 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7ff59800f020 con 0x7ff5a0115360 2024-05-27T07:33:19.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.677+0000 7ff59f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 0x55aa739f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.677+0000 7ff59f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 0x55aa739f7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:34096/0 (socket says 172.21.15.43:34096) 2024-05-27T07:33:19.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.677+0000 7ff59f400700 1 -- 172.21.15.43:0/879236580 learned_addr learned my addr 172.21.15.43:0/879236580 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:19.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.677+0000 7ff59ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa73878e40 0x55aa739fd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.677+0000 7ff59f400700 1 -- 172.21.15.43:0/879236580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 msgr2=0x55aa739f81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:19.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.677+0000 7ff59f400700 1 --2- 172.21.15.43:0/879236580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 0x55aa739f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.677+0000 7ff59f400700 1 -- 172.21.15.43:0/879236580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa73878e40 msgr2=0x55aa739fd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:19.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.677+0000 7ff59f400700 1 --2- 172.21.15.43:0/879236580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa73878e40 0x55aa739fd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.677+0000 7ff59f400700 1 -- 172.21.15.43:0/879236580 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa738de420 con 0x55aa739fdd30 2024-05-27T07:33:19.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.677+0000 7ff59fe00700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5840b2a70 0x7ff5840b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.678+0000 7ff59f400700 1 --2- 172.21.15.43:0/879236580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 0x55aa739f7870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff5900107a0 tx=0x7ff590010ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b3cd07c9e565af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:19.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.678+0000 7ff594c00700 1 -- 172.21.15.43:0/879236580 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff590011700 con 0x55aa739fdd30 2024-05-27T07:33:19.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.678+0000 7ff594c00700 1 -- 172.21.15.43:0/879236580 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff590011d30 con 0x55aa739fdd30 2024-05-27T07:33:19.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.678+0000 7ff594c00700 1 -- 172.21.15.43:0/879236580 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff590019ca0 con 0x55aa739fdd30 2024-05-27T07:33:19.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.679+0000 7ff59fe00700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5840b2a70 0x7ff5840b4f80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff598009900 tx=0x7ff5980099e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:19.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.681+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/879236580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 msgr2=0x55aa739f7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.681+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/879236580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 0x55aa739f7870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff5900107a0 tx=0x7ff590010ab0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.682+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/879236580 shutdown_connections 2024-05-27T07:33:19.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.682+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/879236580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 0x55aa739f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.682+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/879236580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa73878e40 0x55aa739fd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.682+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/879236580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 0x55aa739f7870 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.682+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/879236580 >> 172.21.15.43:0/879236580 conn(0x55aa738d8fc0 msgr2=0x55aa739f28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:19.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.682+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/879236580 shutdown_connections 2024-05-27T07:33:19.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.683+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/879236580 wait complete. 2024-05-27T07:33:19.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.684+0000 7ff5b53a4b80 1 -- start start 2024-05-27T07:33:19.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.684+0000 7ff5b53a4b80 1 Processor -- start 2024-05-27T07:33:19.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.687+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa73878e40 0x55aa73a01510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.687+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 0x55aa73a01a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.687+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 0x55aa73a01f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.687+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55aa7381bfa0 con 0x55aa73878e40 2024-05-27T07:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.687+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55aa738b6760 con 0x55aa739fdd30 2024-05-27T07:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.687+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55aa73a04800 con 0x55aa739f7db0 2024-05-27T07:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.687+0000 7ff5b53a4b80 10 client.0 _ll_register_callbacks cb 0x55aa738bfc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:19.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.688+0000 7ff59f400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 0x55aa73a01a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.688+0000 7ff59f400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 0x55aa73a01a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:56274/0 (socket says 172.21.15.43:56274) 2024-05-27T07:33:19.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.688+0000 7ff59f400700 1 -- 172.21.15.43:0/3303973380 learned_addr learned my addr 172.21.15.43:0/3303973380 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:19.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.688+0000 7ff59ea00700 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 0x55aa73a01f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.689+0000 7ff59f400700 1 -- 172.21.15.43:0/3303973380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 msgr2=0x55aa73a01f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:19.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.689+0000 7ff59f400700 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 0x55aa73a01f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.689+0000 7ff59f400700 1 -- 172.21.15.43:0/3303973380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa73878e40 msgr2=0x55aa73a01510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:19.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.689+0000 7ff59f400700 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa73878e40 0x55aa73a01510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.689+0000 7ff59f400700 1 -- 172.21.15.43:0/3303973380 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa738658d0 con 0x55aa739f7db0 2024-05-27T07:33:19.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.691+0000 7ff59f400700 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 0x55aa73a01a50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff5900046e0 tx=0x7ff590019880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fe62db301ed795b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:19.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.691+0000 7ff589600700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:33:19.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.692+0000 7ff589600700 1 -- 172.21.15.43:0/3303973380 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff590027020 con 0x55aa739f7db0 2024-05-27T07:33:19.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.692+0000 7ff589600700 1 -- 172.21.15.43:0/3303973380 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7ff590019b50 con 0x55aa739f7db0 2024-05-27T07:33:19.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.693+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/3303973380 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff590011d30 con 0x55aa739f7db0 2024-05-27T07:33:19.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.694+0000 7ff589600700 1 -- 172.21.15.43:0/3303973380 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7ff590027020 con 0x55aa739f7db0 2024-05-27T07:33:19.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.694+0000 7ff5b53a4b80 10 client.27475 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:19.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.694+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/3303973380 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x55aa738e6060 con 0x55aa739f7db0 2024-05-27T07:33:19.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.695+0000 7ff589600700 1 -- 172.21.15.43:0/3303973380 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff590019d50 con 0x55aa739f7db0 2024-05-27T07:33:19.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.695+0000 7ff589600700 1 -- 172.21.15.43:0/3303973380 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7ff590022030 con 0x55aa739f7db0 2024-05-27T07:33:19.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.697+0000 7ff5b53a4b80 10 client.27475 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:19.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.697+0000 7ff5b53a4b80 10 client.27475 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:33:19.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.697+0000 7ff5b53a4b80 20 client.27475 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:19.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.697+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55aa73a11770 0x55aa73a13c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.697+0000 7ff5b53a4b80 4 client.27475 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:19.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.697+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/3303973380 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff590011d30 con 0x55aa73a11770 2024-05-27T07:33:19.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.698+0000 7ff59ea00700 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55aa73a11770 0x55aa73a13c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.701+0000 7ff589600700 10 client.27475 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:33:19.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.701+0000 7ff59ea00700 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55aa73a11770 0x55aa73a13c00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=795875fd57f4987e server_cookie=7912dcb82667fb45 in_seq=0 out_seq=0 2024-05-27T07:33:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.704+0000 7ff589600700 1 -- 172.21.15.43:0/3303973380 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7ff590011d30 con 0x55aa73a11770 2024-05-27T07:33:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.704+0000 7ff589600700 10 client.27475 handle_command_reply: tid=0 2024-05-27T07:33:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.704+0000 7ff5b53a4b80 1 client.27475 shutdown 2024-05-27T07:33:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.705+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/3303973380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 msgr2=0x55aa73a01a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.705+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 0x55aa73a01a50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff5900046e0 tx=0x7ff590019880 comp rx=0 tx=0).stop 2024-05-27T07:33:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.707+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/3303973380 shutdown_connections 2024-05-27T07:33:19.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.707+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55aa73a11770 0x55aa73a13c00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.707+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa739fdd30 0x55aa73a01f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.707+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa73878e40 0x55aa73a01510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.707+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/3303973380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739f7db0 0x55aa73a01a50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.707+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/3303973380 >> 172.21.15.43:0/3303973380 conn(0x55aa738d8fc0 msgr2=0x55aa739f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:19.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.707+0000 7ff589600700 0 client.27475 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:33:19.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.708+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/3303973380 shutdown_connections 2024-05-27T07:33:19.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.708+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/3303973380 wait complete. 2024-05-27T07:33:19.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.708+0000 7ff5b53a4b80 20 client.27475 trim_cache size 0 max 16384 2024-05-27T07:33:19.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.745+0000 7ff5b53a4b80 1 Processor -- start 2024-05-27T07:33:19.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.745+0000 7ff5b53a4b80 1 -- start start 2024-05-27T07:33:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.746+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 0x55aa739ff150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.746+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 0x55aa73a00510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.746+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 0x55aa73a014d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.746+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a01b7030 con 0x55aa739f7db0 2024-05-27T07:33:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.746+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a01b5490 con 0x55aa73878e40 2024-05-27T07:33:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.746+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a01b6960 con 0x55aa739fdd30 2024-05-27T07:33:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.747+0000 7ff59ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 0x55aa739ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.747+0000 7ff59ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 0x55aa739ff150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:34112/0 (socket says 172.21.15.43:34112) 2024-05-27T07:33:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.747+0000 7ff59ea00700 1 -- 172.21.15.43:0/2258931579 learned_addr learned my addr 172.21.15.43:0/2258931579 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.747+0000 7ff59ea00700 1 --2- 172.21.15.43:0/2258931579 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 0x55aa73a014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.747+0000 7ff59f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 0x55aa73a00510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.748+0000 7ff59ea00700 1 -- 172.21.15.43:0/2258931579 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 msgr2=0x55aa73a014d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.748+0000 7ff59ea00700 1 --2- 172.21.15.43:0/2258931579 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 0x55aa73a014d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.748+0000 7ff59ea00700 1 -- 172.21.15.43:0/2258931579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 msgr2=0x55aa73a00510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:19.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.748+0000 7ff59ea00700 1 --2- 172.21.15.43:0/2258931579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 0x55aa73a00510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.749+0000 7ff59ea00700 1 -- 172.21.15.43:0/2258931579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa739ffa80 con 0x55aa73878e40 2024-05-27T07:33:19.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.749+0000 7ff59f400700 1 --2- 172.21.15.43:0/2258931579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 0x55aa73a00510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.752+0000 7ff59ea00700 1 --2- 172.21.15.43:0/2258931579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 0x55aa739ff150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff58c0092a0 tx=0x7ff58c012040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6adf391c9ec1cd9b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.752+0000 7ff58be00700 1 -- 172.21.15.43:0/2258931579 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c004a30 con 0x55aa73878e40 2024-05-27T07:33:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.753+0000 7ff58be00700 1 -- 172.21.15.43:0/2258931579 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff58c0054b0 con 0x55aa73878e40 2024-05-27T07:33:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.753+0000 7ff58be00700 1 -- 172.21.15.43:0/2258931579 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c028720 con 0x55aa73878e40 2024-05-27T07:33:19.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.755+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/2258931579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 msgr2=0x55aa739ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.755+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/2258931579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 0x55aa739ff150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff58c0092a0 tx=0x7ff58c012040 comp rx=0 tx=0).stop 2024-05-27T07:33:19.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.756+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/2258931579 shutdown_connections 2024-05-27T07:33:19.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.756+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/2258931579 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 0x55aa73a014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.756+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/2258931579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 0x55aa739ff150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.756+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/2258931579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 0x55aa73a00510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.757+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/2258931579 >> 172.21.15.43:0/2258931579 conn(0x55aa738d8fc0 msgr2=0x55aa739ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:19.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.758+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/2258931579 shutdown_connections 2024-05-27T07:33:19.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.758+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/2258931579 wait complete. 2024-05-27T07:33:19.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.759+0000 7ff5b53a4b80 1 -- start start 2024-05-27T07:33:19.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.760+0000 7ff5b53a4b80 1 Processor -- start 2024-05-27T07:33:19.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.761+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 0x55aa73a09ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.761+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 0x55aa73a0a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.761+0000 7ff5b53a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 0x55aa73a183f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.761+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a01b6290 con 0x55aa739f7db0 2024-05-27T07:33:19.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.761+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a01b5b60 con 0x55aa73878e40 2024-05-27T07:33:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.761+0000 7ff5b53a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55aa739fd070 con 0x55aa739fdd30 2024-05-27T07:33:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.763+0000 7ff5b53a4b80 10 client.0 _ll_register_callbacks cb 0x55aa738bfc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.763+0000 7ff59ea00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 0x55aa73a183f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.763+0000 7ff59ea00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 0x55aa73a183f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:56288/0 (socket says 172.21.15.43:56288) 2024-05-27T07:33:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.763+0000 7ff59ea00700 1 -- 172.21.15.43:0/596126212 learned_addr learned my addr 172.21.15.43:0/596126212 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.763+0000 7ff59ea00700 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 0x55aa73a09ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.764+0000 7ff59f400700 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 0x55aa73a0a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.766+0000 7ff59ea00700 1 -- 172.21.15.43:0/596126212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 msgr2=0x55aa73a09ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:19.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.767+0000 7ff59ea00700 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 0x55aa73a09ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.767+0000 7ff59ea00700 1 -- 172.21.15.43:0/596126212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 msgr2=0x55aa73a0a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:19.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.767+0000 7ff59ea00700 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 0x55aa73a0a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.767+0000 7ff59ea00700 1 -- 172.21.15.43:0/596126212 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa73a19d40 con 0x55aa739fdd30 2024-05-27T07:33:19.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.767+0000 7ff59ea00700 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 0x55aa73a183f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff58c004fd0 tx=0x7ff58c0098e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f1aa40a5fa58e90 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:19.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.767+0000 7ff589600700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:33:19.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.768+0000 7ff59f400700 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 0x55aa73a0a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:19.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.768+0000 7ff589600700 1 -- 172.21.15.43:0/596126212 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c034040 con 0x55aa739fdd30 2024-05-27T07:33:19.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.769+0000 7ff589600700 1 -- 172.21.15.43:0/596126212 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7ff58c004950 con 0x55aa739fdd30 2024-05-27T07:33:19.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.769+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/596126212 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff58c0054b0 con 0x55aa739fdd30 2024-05-27T07:33:19.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.769+0000 7ff589600700 1 -- 172.21.15.43:0/596126212 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7ff58c0052c0 con 0x55aa739fdd30 2024-05-27T07:33:19.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.769+0000 7ff5b53a4b80 10 client.27481 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:19.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.769+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/596126212 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x55aa73a8c590 con 0x55aa739fdd30 2024-05-27T07:33:19.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.770+0000 7ff589600700 1 -- 172.21.15.43:0/596126212 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff58c0054b0 con 0x55aa739fdd30 2024-05-27T07:33:19.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.770+0000 7ff589600700 1 -- 172.21.15.43:0/596126212 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7ff58c010b70 con 0x55aa739fdd30 2024-05-27T07:33:19.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.770+0000 7ff5b53a4b80 10 client.27481 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:19.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.770+0000 7ff5b53a4b80 10 client.27481 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:33:19.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.770+0000 7ff5b53a4b80 20 client.27481 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:19.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.770+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55aa73a8c940 0x55aa73a8ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:19.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.770+0000 7ff5b53a4b80 4 client.27481 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:19.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.770+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/596126212 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aa73927d90 con 0x55aa73a8c940 2024-05-27T07:33:19.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.771+0000 7ff59f400700 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55aa73a8c940 0x55aa73a8ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:19.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.772+0000 7ff59f400700 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55aa73a8c940 0x55aa73a8ed60 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b000c17a433c5ea3 server_cookie=1273a08580910bab in_seq=0 out_seq=0 2024-05-27T07:33:19.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.772+0000 7ff589600700 10 client.27481 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:33:19.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.772+0000 7ff589600700 1 -- 172.21.15.43:0/596126212 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aa73927d90 con 0x55aa73a8c940 2024-05-27T07:33:19.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.772+0000 7ff589600700 10 client.27481 handle_command_reply: tid=0 2024-05-27T07:33:19.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.772+0000 7ff5b53a4b80 1 client.27481 shutdown 2024-05-27T07:33:19.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/596126212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 msgr2=0x55aa73a183f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 0x55aa73a183f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff58c004fd0 tx=0x7ff58c0098e0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/596126212 shutdown_connections 2024-05-27T07:33:19.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa739fdd30 0x55aa73a183f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55aa73a8c940 0x55aa73a8ed60 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aa73878e40 0x55aa73a09ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff5b53a4b80 1 --2- 172.21.15.43:0/596126212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa739f7db0 0x55aa73a0a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/596126212 >> 172.21.15.43:0/596126212 conn(0x55aa738d8fc0 msgr2=0x55aa739f6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:19.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff589600700 0 client.27481 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:33:19.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/596126212 shutdown_connections 2024-05-27T07:33:19.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.773+0000 7ff5b53a4b80 1 -- 172.21.15.43:0/596126212 wait complete. 2024-05-27T07:33:19.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.774+0000 7ff5b53a4b80 20 client.27481 trim_cache size 0 max 16384 2024-05-27T07:33:19.769 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:33:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.774+0000 7ff5a6800700 1 -- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5840b2a70 msgr2=0x7ff5840b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.774+0000 7ff5a6800700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5840b2a70 0x7ff5840b4f80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff598009900 tx=0x7ff5980099e0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.774+0000 7ff5a6800700 1 -- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5a0115360 msgr2=0x7ff5a01a5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.774+0000 7ff5a6800700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5a0115360 0x7ff5a01a5b10 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7ff59800b0a0 tx=0x7ff598004d00 comp rx=0 tx=0).stop 2024-05-27T07:33:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.775+0000 7ff5a6800700 1 -- 172.21.15.43:0/1888485208 shutdown_connections 2024-05-27T07:33:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.775+0000 7ff5a6800700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff5840b2a70 0x7ff5840b4f80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.775+0000 7ff5a6800700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5a0115360 0x7ff5a01a5b10 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.775+0000 7ff5a6800700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff5a0075b50 0x7ff5a01a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.775+0000 7ff5a6800700 1 --2- 172.21.15.43:0/1888485208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a00764a0 0x7ff5a01a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.775+0000 7ff5a6800700 1 -- 172.21.15.43:0/1888485208 >> 172.21.15.43:0/1888485208 conn(0x7ff5a0100030 msgr2=0x7ff5a0100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.775+0000 7ff5a6800700 1 -- 172.21.15.43:0/1888485208 shutdown_connections 2024-05-27T07:33:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:19.775+0000 7ff5a6800700 1 -- 172.21.15.43:0/1888485208 wait complete. 2024-05-27T07:33:19.793 INFO:tasks.workunit.client.0.smithi043.stdout: 10 35292 5.78 MB/sec execute 200 sec latency 136.399 ms 2024-05-27T07:33:20.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:20 smithi043 ceph-mon[31531]: pgmap v653: 129 pgs: 129 active+clean; 840 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 3.2 MiB/s wr, 178 op/s 2024-05-27T07:33:20.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 35409 5.77 MB/sec execute 201 sec latency 90.477 ms 2024-05-27T07:33:20.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:20 smithi132 ceph-mon[35283]: pgmap v653: 129 pgs: 129 active+clean; 840 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 3.2 MiB/s wr, 178 op/s 2024-05-27T07:33:20.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:20 smithi179 ceph-mon[35888]: pgmap v653: 129 pgs: 129 active+clean; 840 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 3.2 MiB/s wr, 178 op/s 2024-05-27T07:33:21.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:33:21.636 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:21.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 35518 5.76 MB/sec execute 202 sec latency 127.023 ms 2024-05-27T07:33:22.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 35617 5.77 MB/sec execute 203 sec latency 314.464 ms 2024-05-27T07:33:22.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:22 smithi132 ceph-mon[35283]: pgmap v654: 129 pgs: 129 active+clean; 844 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-05-27T07:33:22.878 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:22 smithi043 ceph-mon[31531]: pgmap v654: 129 pgs: 129 active+clean; 844 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-05-27T07:33:22.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:22 smithi179 ceph-mon[35888]: pgmap v654: 129 pgs: 129 active+clean; 844 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-05-27T07:33:23.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 35774 5.82 MB/sec execute 204 sec latency 80.869 ms 2024-05-27T07:33:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.051+0000 7f526b200700 1 -- 172.21.15.43:0/2289339560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5264105e20 msgr2=0x7f526410e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.051+0000 7f526b200700 1 --2- 172.21.15.43:0/2289339560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5264105e20 0x7f526410e370 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5250009990 tx=0x7f5250009ca0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.051+0000 7f526b200700 1 -- 172.21.15.43:0/2289339560 shutdown_connections 2024-05-27T07:33:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.051+0000 7f526b200700 1 --2- 172.21.15.43:0/2289339560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f526410e900 0x7f5264110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.051+0000 7f526b200700 1 --2- 172.21.15.43:0/2289339560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52641054d0 0x7f52641058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.051+0000 7f526b200700 1 --2- 172.21.15.43:0/2289339560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5264105e20 0x7f526410e370 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.051+0000 7f526b200700 1 -- 172.21.15.43:0/2289339560 >> 172.21.15.43:0/2289339560 conn(0x7f52640fedb0 msgr2=0x7f52641011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.052+0000 7f526b200700 1 -- 172.21.15.43:0/2289339560 shutdown_connections 2024-05-27T07:33:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.052+0000 7f526b200700 1 -- 172.21.15.43:0/2289339560 wait complete. 2024-05-27T07:33:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.053+0000 7f526b200700 1 Processor -- start 2024-05-27T07:33:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.053+0000 7f526b200700 1 -- start start 2024-05-27T07:33:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.054+0000 7f526b200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52641054d0 0x7f526410e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.054+0000 7f526b200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5264105e20 0x7f526410b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.054+0000 7f526b200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526410e900 0x7f526410b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.054+0000 7f526b200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f52641046b0 con 0x7f5264105e20 2024-05-27T07:33:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.054+0000 7f526b200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5264104530 con 0x7f52641054d0 2024-05-27T07:33:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.054+0000 7f5263e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52641054d0 0x7f526410e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.054+0000 7f526b200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5264104830 con 0x7f526410e900 2024-05-27T07:33:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.054+0000 7f5263e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52641054d0 0x7f526410e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:34128/0 (socket says 172.21.15.43:34128) 2024-05-27T07:33:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.054+0000 7f5263e00700 1 -- 172.21.15.43:0/2725502705 learned_addr learned my addr 172.21.15.43:0/2725502705 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.055+0000 7f5263400700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5264105e20 0x7f526410b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.055+0000 7f5263e00700 1 -- 172.21.15.43:0/2725502705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526410e900 msgr2=0x7f526410b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.055+0000 7f5263e00700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526410e900 0x7f526410b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.055+0000 7f5263e00700 1 -- 172.21.15.43:0/2725502705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5264105e20 msgr2=0x7f526410b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.055+0000 7f5263e00700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5264105e20 0x7f526410b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.055+0000 7f5263e00700 1 -- 172.21.15.43:0/2725502705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526410c200 con 0x7f52641054d0 2024-05-27T07:33:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.056+0000 7f5263e00700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52641054d0 0x7f526410e2c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f525400e9a0 tx=0x7f525400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.056+0000 7f5260c00700 1 -- 172.21.15.43:0/2725502705 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525400cbd0 con 0x7f52641054d0 2024-05-27T07:33:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.056+0000 7f526b200700 1 -- 172.21.15.43:0/2725502705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52641ad820 con 0x7f52641054d0 2024-05-27T07:33:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.056+0000 7f526b200700 1 -- 172.21.15.43:0/2725502705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52641add60 con 0x7f52641054d0 2024-05-27T07:33:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.056+0000 7f5260c00700 1 -- 172.21.15.43:0/2725502705 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f525400cd70 con 0x7f52641054d0 2024-05-27T07:33:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.056+0000 7f5260c00700 1 -- 172.21.15.43:0/2725502705 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254018680 con 0x7f52641054d0 2024-05-27T07:33:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.063+0000 7f5279dcab80 1 Processor -- start 2024-05-27T07:33:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.063+0000 7f5279dcab80 1 -- start start 2024-05-27T07:33:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.063+0000 7f5260c00700 1 -- 172.21.15.43:0/2725502705 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5254018820 con 0x7f52641054d0 2024-05-27T07:33:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.064+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 0x55a4a517d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.064+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a5177870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.064+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a51781c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.064+0000 7f5279dcab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a4a50a5e40 con 0x55a4a517dd30 2024-05-27T07:33:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.064+0000 7f5279dcab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a4a50959e0 con 0x55a4a5177db0 2024-05-27T07:33:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.064+0000 7f5279dcab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a4a5099a20 con 0x55a4a4ff8e40 2024-05-27T07:33:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.064+0000 7f5263400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a51781c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.064+0000 7f5263400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a51781c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:34132/0 (socket says 172.21.15.43:34132) 2024-05-27T07:33:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.064+0000 7f5263400700 1 -- 172.21.15.43:0/7845674 learned_addr learned my addr 172.21.15.43:0/7845674 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.065+0000 7f5263e00700 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a5177870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.065+0000 7f5263400700 1 -- 172.21.15.43:0/7845674 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 msgr2=0x55a4a517d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.065+0000 7f5269200700 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 0x55a4a517d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.065+0000 7f5263400700 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 0x55a4a517d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.065+0000 7f5263400700 1 -- 172.21.15.43:0/7845674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 msgr2=0x55a4a5177870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.065+0000 7f5263400700 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a5177870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.065+0000 7f5263400700 1 -- 172.21.15.43:0/7845674 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4a505e420 con 0x55a4a5177db0 2024-05-27T07:33:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.065+0000 7f5263e00700 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a5177870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.066+0000 7f5269200700 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 0x55a4a517d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.066+0000 7f5263400700 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a51781c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5250015000 tx=0x7f525000b280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8839b00265c07f80 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.066+0000 7f5260c00700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52480b2af0 0x7f52480b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.066+0000 7f5260c00700 1 -- 172.21.15.43:0/2725502705 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7f52540e2bf0 con 0x7f52641054d0 2024-05-27T07:33:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.067+0000 7f5259600700 1 -- 172.21.15.43:0/7845674 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525000b950 con 0x55a4a5177db0 2024-05-27T07:33:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.067+0000 7f5259600700 1 -- 172.21.15.43:0/7845674 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f525000f460 con 0x55a4a5177db0 2024-05-27T07:33:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.067+0000 7f5259600700 1 -- 172.21.15.43:0/7845674 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250017660 con 0x55a4a5177db0 2024-05-27T07:33:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.067+0000 7f5269200700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52480b2af0 0x7f52480b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.068+0000 7f5269200700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52480b2af0 0x7f52480b5000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f525c009550 tx=0x7f525c009040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.068+0000 7f5279dcab80 1 -- 172.21.15.43:0/7845674 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 msgr2=0x55a4a51781c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.068+0000 7f5279dcab80 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a51781c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5250015000 tx=0x7f525000b280 comp rx=0 tx=0).stop 2024-05-27T07:33:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.069+0000 7f5279dcab80 1 -- 172.21.15.43:0/7845674 shutdown_connections 2024-05-27T07:33:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.069+0000 7f5279dcab80 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a51781c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.069+0000 7f5279dcab80 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 0x55a4a517d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.069+0000 7f5279dcab80 1 --2- 172.21.15.43:0/7845674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a5177870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.069+0000 7f5279dcab80 1 -- 172.21.15.43:0/7845674 >> 172.21.15.43:0/7845674 conn(0x55a4a5058fc0 msgr2=0x55a4a51728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.069+0000 7f5279dcab80 1 -- 172.21.15.43:0/7845674 shutdown_connections 2024-05-27T07:33:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.069+0000 7f5279dcab80 1 -- 172.21.15.43:0/7845674 wait complete. 2024-05-27T07:33:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.070+0000 7f5279dcab80 1 -- start start 2024-05-27T07:33:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.070+0000 7f5279dcab80 1 Processor -- start 2024-05-27T07:33:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.071+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a4ff8e40 0x55a4a5181450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.071+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a5181990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.071+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a517dd30 0x55a4a5181ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.071+0000 7f5279dcab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a4a4f9bfa0 con 0x55a4a4ff8e40 2024-05-27T07:33:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.071+0000 7f5279dcab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a4a5036760 con 0x55a4a5177db0 2024-05-27T07:33:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.071+0000 7f5279dcab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a4a51847a0 con 0x55a4a517dd30 2024-05-27T07:33:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.071+0000 7f5263e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a4ff8e40 0x55a4a5181450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.071+0000 7f5263e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a4ff8e40 0x55a4a5181450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41536/0 (socket says 172.21.15.43:41536) 2024-05-27T07:33:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.071+0000 7f5263e00700 1 -- 172.21.15.43:0/3118431207 learned_addr learned my addr 172.21.15.43:0/3118431207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.072+0000 7f5279dcab80 10 client.0 _ll_register_callbacks cb 0x55a4a503fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.072+0000 7f5263400700 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a5181990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.072+0000 7f5263e00700 1 -- 172.21.15.43:0/3118431207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a517dd30 msgr2=0x55a4a5181ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.072+0000 7f5263e00700 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a517dd30 0x55a4a5181ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.072+0000 7f5263e00700 1 -- 172.21.15.43:0/3118431207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 msgr2=0x55a4a5181990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.072+0000 7f5263e00700 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a5181990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.072+0000 7f5263e00700 1 -- 172.21.15.43:0/3118431207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4a4fe58d0 con 0x55a4a4ff8e40 2024-05-27T07:33:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.073+0000 7f5263e00700 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a4ff8e40 0x55a4a5181450 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f52540a6a00 tx=0x7f5254010af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b1fe8010b27ba9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.073+0000 7f524e000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:33:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.073+0000 7f524e000700 1 -- 172.21.15.43:0/3118431207 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52540aa490 con 0x55a4a4ff8e40 2024-05-27T07:33:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.073+0000 7f524e000700 1 -- 172.21.15.43:0/3118431207 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7f5254010640 con 0x55a4a4ff8e40 2024-05-27T07:33:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.073+0000 7f5279dcab80 1 -- 172.21.15.43:0/3118431207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f525000f460 con 0x55a4a4ff8e40 2024-05-27T07:33:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f524e000700 1 -- 172.21.15.43:0/3118431207 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f52540ab080 con 0x55a4a4ff8e40 2024-05-27T07:33:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f5279dcab80 10 client.18045 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f5279dcab80 1 -- 172.21.15.43:0/3118431207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x55a4a5066060 con 0x55a4a4ff8e40 2024-05-27T07:33:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f524e000700 1 -- 172.21.15.43:0/3118431207 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f525000f460 con 0x55a4a4ff8e40 2024-05-27T07:33:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f524e000700 1 -- 172.21.15.43:0/3118431207 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f52540b3350 con 0x55a4a4ff8e40 2024-05-27T07:33:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f5279dcab80 10 client.18045 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f5279dcab80 10 client.18045 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:33:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f5279dcab80 20 client.18045 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a4a51918f0 0x55a4a5193dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f5279dcab80 4 client.18045 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.074+0000 7f5279dcab80 1 -- 172.21.15.43:0/3118431207 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4a50a7d90 con 0x55a4a51918f0 2024-05-27T07:33:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.075+0000 7f5263400700 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a4a51918f0 0x55a4a5193dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.076+0000 7f5263400700 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a4a51918f0 0x55a4a5193dc0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=645600fd035fb10 server_cookie=4fb3f5c520ca324c in_seq=0 out_seq=0 2024-05-27T07:33:24.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.076+0000 7f524e000700 10 client.18045 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:33:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.078+0000 7f524e000700 1 -- 172.21.15.43:0/3118431207 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a4a50a7d90 con 0x55a4a51918f0 2024-05-27T07:33:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.078+0000 7f524e000700 10 client.18045 handle_command_reply: tid=0 2024-05-27T07:33:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.078+0000 7f5279dcab80 1 client.18045 shutdown 2024-05-27T07:33:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f5279dcab80 1 -- 172.21.15.43:0/3118431207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a4ff8e40 msgr2=0x55a4a5181450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a4ff8e40 0x55a4a5181450 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f52540a6a00 tx=0x7f5254010af0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f5279dcab80 1 -- 172.21.15.43:0/3118431207 shutdown_connections 2024-05-27T07:33:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a4a51918f0 0x55a4a5193dc0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a517dd30 0x55a4a5181ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a4ff8e40 0x55a4a5181450 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3118431207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a5181990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f5279dcab80 1 -- 172.21.15.43:0/3118431207 >> 172.21.15.43:0/3118431207 conn(0x55a4a5058fc0 msgr2=0x55a4a5172580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f524e000700 0 client.18045 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:33:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f5279dcab80 1 -- 172.21.15.43:0/3118431207 shutdown_connections 2024-05-27T07:33:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.079+0000 7f5279dcab80 1 -- 172.21.15.43:0/3118431207 wait complete. 2024-05-27T07:33:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.080+0000 7f5279dcab80 20 client.18045 trim_cache size 0 max 16384 2024-05-27T07:33:24.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.108+0000 7f5279dcab80 1 Processor -- start 2024-05-27T07:33:24.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.108+0000 7f5279dcab80 1 -- start start 2024-05-27T07:33:24.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.109+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 0x55a4a5177870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.109+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a517d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.109+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a51838f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.109+0000 7f5279dcab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f524400f3d0 con 0x55a4a517dd30 2024-05-27T07:33:24.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.109+0000 7f5279dcab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f524400d830 con 0x55a4a5177db0 2024-05-27T07:33:24.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.109+0000 7f5279dcab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f524400ed00 con 0x55a4a4ff8e40 2024-05-27T07:33:24.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.110+0000 7f5269200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a51838f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.110+0000 7f5263400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a517d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.110+0000 7f5269200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a51838f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41546/0 (socket says 172.21.15.43:41546) 2024-05-27T07:33:24.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.110+0000 7f5269200700 1 -- 172.21.15.43:0/708254809 learned_addr learned my addr 172.21.15.43:0/708254809 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:24.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.111+0000 7f5263e00700 1 --2- 172.21.15.43:0/708254809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 0x55a4a5177870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.111+0000 7f5263400700 1 -- 172.21.15.43:0/708254809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 msgr2=0x55a4a5177870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:24.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.111+0000 7f5263400700 1 --2- 172.21.15.43:0/708254809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 0x55a4a5177870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.111+0000 7f5263400700 1 -- 172.21.15.43:0/708254809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 msgr2=0x55a4a51838f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:24.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.111+0000 7f5263400700 1 --2- 172.21.15.43:0/708254809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a51838f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.111+0000 7f5263400700 1 -- 172.21.15.43:0/708254809 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4a5174320 con 0x55a4a5177db0 2024-05-27T07:33:24.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.112+0000 7f5263400700 1 --2- 172.21.15.43:0/708254809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a517d000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5250005f50 tx=0x7f525000fd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc02f741dda298c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:24.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.112+0000 7f5258c00700 1 -- 172.21.15.43:0/708254809 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250035070 con 0x55a4a5177db0 2024-05-27T07:33:24.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.113+0000 7f5258c00700 1 -- 172.21.15.43:0/708254809 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f525000f930 con 0x55a4a5177db0 2024-05-27T07:33:24.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.113+0000 7f5258c00700 1 -- 172.21.15.43:0/708254809 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525000be80 con 0x55a4a5177db0 2024-05-27T07:33:24.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.114+0000 7f5279dcab80 1 -- 172.21.15.43:0/708254809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 msgr2=0x55a4a517d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:24.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.114+0000 7f5279dcab80 1 --2- 172.21.15.43:0/708254809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a517d000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5250005f50 tx=0x7f525000fd80 comp rx=0 tx=0).stop 2024-05-27T07:33:24.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.114+0000 7f5279dcab80 1 -- 172.21.15.43:0/708254809 shutdown_connections 2024-05-27T07:33:24.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.114+0000 7f5279dcab80 1 --2- 172.21.15.43:0/708254809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a51838f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.114+0000 7f5279dcab80 1 --2- 172.21.15.43:0/708254809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a4ff8e40 0x55a4a5177870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.114+0000 7f5279dcab80 1 --2- 172.21.15.43:0/708254809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a5177db0 0x55a4a517d000 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.114+0000 7f5279dcab80 1 -- 172.21.15.43:0/708254809 >> 172.21.15.43:0/708254809 conn(0x55a4a5058fc0 msgr2=0x55a4a5181410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:24.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.114+0000 7f5279dcab80 1 -- 172.21.15.43:0/708254809 shutdown_connections 2024-05-27T07:33:24.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.114+0000 7f5279dcab80 1 -- 172.21.15.43:0/708254809 wait complete. 2024-05-27T07:33:24.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.115+0000 7f5279dcab80 1 -- start start 2024-05-27T07:33:24.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.115+0000 7f5279dcab80 1 Processor -- start 2024-05-27T07:33:24.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a4ff8e40 0x55a4a5189a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a5177db0 0x55a4a5189fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5279dcab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a51985b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5279dcab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f524400e630 con 0x55a4a517dd30 2024-05-27T07:33:24.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5279dcab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f524400df00 con 0x55a4a4ff8e40 2024-05-27T07:33:24.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5279dcab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a4a5173720 con 0x55a4a5177db0 2024-05-27T07:33:24.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5279dcab80 10 client.0 _ll_register_callbacks cb 0x55a4a503fc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:24.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5263e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a4ff8e40 0x55a4a5189a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5263e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a4ff8e40 0x55a4a5189a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:34154/0 (socket says 172.21.15.43:34154) 2024-05-27T07:33:24.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5263e00700 1 -- 172.21.15.43:0/3520619223 learned_addr learned my addr 172.21.15.43:0/3520619223 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:24.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.116+0000 7f5269200700 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a51985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.117+0000 7f5263400700 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a5177db0 0x55a4a5189fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.117+0000 7f5263e00700 1 -- 172.21.15.43:0/3520619223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a5177db0 msgr2=0x55a4a5189fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:24.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.117+0000 7f5263e00700 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a5177db0 0x55a4a5189fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.118+0000 7f5263e00700 1 -- 172.21.15.43:0/3520619223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 msgr2=0x55a4a51985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:24.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.118+0000 7f5263e00700 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a51985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.118+0000 7f5263e00700 1 -- 172.21.15.43:0/3520619223 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4a5198af0 con 0x55a4a4ff8e40 2024-05-27T07:33:24.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.118+0000 7f5269200700 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a51985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:24.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.119+0000 7f5263e00700 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a4ff8e40 0x55a4a5189a90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f525400cfd0 tx=0x7f52540aa1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e885a7ce9b84edd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:24.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.119+0000 7f524e000700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:33:24.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.119+0000 7f524e000700 1 -- 172.21.15.43:0/3520619223 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52540aabe0 con 0x55a4a4ff8e40 2024-05-27T07:33:24.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.119+0000 7f524e000700 1 -- 172.21.15.43:0/3520619223 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7f52540a9be0 con 0x55a4a4ff8e40 2024-05-27T07:33:24.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.119+0000 7f5279dcab80 1 -- 172.21.15.43:0/3520619223 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f525000f930 con 0x55a4a4ff8e40 2024-05-27T07:33:24.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f524e000700 1 -- 172.21.15.43:0/3520619223 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f52540aa490 con 0x55a4a4ff8e40 2024-05-27T07:33:24.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f5279dcab80 10 client.37424 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:24.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f5279dcab80 1 -- 172.21.15.43:0/3520619223 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x55a4a51fbb30 con 0x55a4a4ff8e40 2024-05-27T07:33:24.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f524e000700 1 -- 172.21.15.43:0/3520619223 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f525000f930 con 0x55a4a4ff8e40 2024-05-27T07:33:24.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f524e000700 1 -- 172.21.15.43:0/3520619223 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f52540a62f0 con 0x55a4a4ff8e40 2024-05-27T07:33:24.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f5279dcab80 10 client.37424 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:24.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f5279dcab80 10 client.37424 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:33:24.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f5279dcab80 20 client.37424 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:24.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a4a51fc0d0 0x55a4a51fe4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:24.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f5279dcab80 4 client.37424 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:24.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.120+0000 7f5279dcab80 1 -- 172.21.15.43:0/3520619223 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a4a50a1570 con 0x55a4a51fc0d0 2024-05-27T07:33:24.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.121+0000 7f5263400700 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a4a51fc0d0 0x55a4a51fe4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:24.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.122+0000 7f524e000700 10 client.37424 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:33:24.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.123+0000 7f5263400700 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a4a51fc0d0 0x55a4a51fe4f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85037285369b7d6d server_cookie=2fa19fc6f97bb8ae in_seq=0 out_seq=0 2024-05-27T07:33:24.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f524e000700 1 -- 172.21.15.43:0/3520619223 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a4a50a1570 con 0x55a4a51fc0d0 2024-05-27T07:33:24.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f524e000700 10 client.37424 handle_command_reply: tid=0 2024-05-27T07:33:24.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f5279dcab80 1 client.37424 shutdown 2024-05-27T07:33:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f5279dcab80 1 -- 172.21.15.43:0/3520619223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a4ff8e40 msgr2=0x55a4a5189a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a4ff8e40 0x55a4a5189a90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f525400cfd0 tx=0x7f52540aa1f0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f5279dcab80 1 -- 172.21.15.43:0/3520619223 shutdown_connections 2024-05-27T07:33:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a4a51fc0d0 0x55a4a51fe4f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a4a517dd30 0x55a4a51985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a4a4ff8e40 0x55a4a5189a90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f5279dcab80 1 --2- 172.21.15.43:0/3520619223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a4a5177db0 0x55a4a5189fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.124+0000 7f5279dcab80 1 -- 172.21.15.43:0/3520619223 >> 172.21.15.43:0/3520619223 conn(0x55a4a5058fc0 msgr2=0x55a4a517cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.125+0000 7f524e000700 0 client.37424 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:33:24.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.125+0000 7f5279dcab80 1 -- 172.21.15.43:0/3520619223 shutdown_connections 2024-05-27T07:33:24.124 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:33:24.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.125+0000 7f5279dcab80 1 -- 172.21.15.43:0/3520619223 wait complete. 2024-05-27T07:33:24.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.125+0000 7f5279dcab80 20 client.37424 trim_cache size 0 max 16384 2024-05-27T07:33:24.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.126+0000 7f526b200700 1 -- 172.21.15.43:0/2725502705 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52480b2af0 msgr2=0x7f52480b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:24.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.126+0000 7f526b200700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52480b2af0 0x7f52480b5000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f525c009550 tx=0x7f525c009040 comp rx=0 tx=0).stop 2024-05-27T07:33:24.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.126+0000 7f526b200700 1 -- 172.21.15.43:0/2725502705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52641054d0 msgr2=0x7f526410e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:24.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.126+0000 7f526b200700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52641054d0 0x7f526410e2c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f525400e9a0 tx=0x7f525400ecb0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.126+0000 7f526b200700 1 -- 172.21.15.43:0/2725502705 shutdown_connections 2024-05-27T07:33:24.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.127+0000 7f526b200700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f52480b2af0 0x7f52480b5000 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.127+0000 7f526b200700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526410e900 0x7f526410b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.127+0000 7f526b200700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52641054d0 0x7f526410e2c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.127+0000 7f526b200700 1 --2- 172.21.15.43:0/2725502705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5264105e20 0x7f526410b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:24.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.127+0000 7f526b200700 1 -- 172.21.15.43:0/2725502705 >> 172.21.15.43:0/2725502705 conn(0x7f52640fedb0 msgr2=0x7f52640ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:24.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.127+0000 7f526b200700 1 -- 172.21.15.43:0/2725502705 shutdown_connections 2024-05-27T07:33:24.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:24.127+0000 7f526b200700 1 -- 172.21.15.43:0/2725502705 wait complete. 2024-05-27T07:33:24.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 35901 5.81 MB/sec execute 205 sec latency 112.460 ms 2024-05-27T07:33:24.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:24 smithi132 ceph-mon[35283]: pgmap v655: 129 pgs: 129 active+clean; 844 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 2.4 MiB/s wr, 148 op/s 2024-05-27T07:33:24.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:24 smithi043 ceph-mon[31531]: pgmap v655: 129 pgs: 129 active+clean; 844 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 2.4 MiB/s wr, 148 op/s 2024-05-27T07:33:24.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:24 smithi179 ceph-mon[35888]: pgmap v655: 129 pgs: 129 active+clean; 844 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 2.4 MiB/s wr, 148 op/s 2024-05-27T07:33:25.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:33:25.675 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:25.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 36016 5.82 MB/sec execute 206 sec latency 120.957 ms 2024-05-27T07:33:26.731 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:26 smithi043 ceph-mon[31531]: pgmap v656: 129 pgs: 129 active+clean; 844 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 2.3 MiB/s wr, 142 op/s 2024-05-27T07:33:26.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 36138 5.85 MB/sec execute 207 sec latency 94.585 ms 2024-05-27T07:33:26.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:26 smithi132 ceph-mon[35283]: pgmap v656: 129 pgs: 129 active+clean; 844 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 2.3 MiB/s wr, 142 op/s 2024-05-27T07:33:26.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:26 smithi179 ceph-mon[35888]: pgmap v656: 129 pgs: 129 active+clean; 844 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 2.3 MiB/s wr, 142 op/s 2024-05-27T07:33:27.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 36255 5.85 MB/sec execute 208 sec latency 223.013 ms 2024-05-27T07:33:28.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.241+0000 7f74ad000700 1 -- 172.21.15.43:0/1134471041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a8105bd0 msgr2=0x7f74a810e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.241+0000 7f74ad000700 1 --2- 172.21.15.43:0/1134471041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a8105bd0 0x7f74a810e120 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f7494009990 tx=0x7f7494009ca0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.242+0000 7f74ad000700 1 -- 172.21.15.43:0/1134471041 shutdown_connections 2024-05-27T07:33:28.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.242+0000 7f74ad000700 1 --2- 172.21.15.43:0/1134471041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74a810e6f0 0x7f74a8110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.242+0000 7f74ad000700 1 --2- 172.21.15.43:0/1134471041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74a8105280 0x7f74a8105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.242+0000 7f74ad000700 1 --2- 172.21.15.43:0/1134471041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a8105bd0 0x7f74a810e120 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.242+0000 7f74ad000700 1 -- 172.21.15.43:0/1134471041 >> 172.21.15.43:0/1134471041 conn(0x7f74a8078ab0 msgr2=0x7f74a8078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:28.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.242+0000 7f74ad000700 1 -- 172.21.15.43:0/1134471041 shutdown_connections 2024-05-27T07:33:28.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.242+0000 7f74ad000700 1 -- 172.21.15.43:0/1134471041 wait complete. 2024-05-27T07:33:28.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.244+0000 7f74ad000700 1 Processor -- start 2024-05-27T07:33:28.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.244+0000 7f74ad000700 1 -- start start 2024-05-27T07:33:28.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.245+0000 7f74ad000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74a8105280 0x7f74a819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.245+0000 7f74ad000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a8105bd0 0x7f74a819d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.245+0000 7f74ad000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74a810e6f0 0x7f74a81a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.245+0000 7f74ad000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f74a8114150 con 0x7f74a8105bd0 2024-05-27T07:33:28.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.245+0000 7f74ad000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f74a8113fd0 con 0x7f74a810e6f0 2024-05-27T07:33:28.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.245+0000 7f74ad000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f74a81142d0 con 0x7f74a8105280 2024-05-27T07:33:28.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.245+0000 7f74a7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74a810e6f0 0x7f74a81a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.245+0000 7f74a7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74a810e6f0 0x7f74a81a15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:36074/0 (socket says 172.21.15.43:36074) 2024-05-27T07:33:28.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.246+0000 7f74a7e00700 1 -- 172.21.15.43:0/1979861923 learned_addr learned my addr 172.21.15.43:0/1979861923 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:28.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.246+0000 7f74a6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a8105bd0 0x7f74a819d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.246+0000 7f74a7e00700 1 -- 172.21.15.43:0/1979861923 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74a8105280 msgr2=0x7f74a819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.246+0000 7f74a7e00700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74a8105280 0x7f74a819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.246+0000 7f74a7e00700 1 -- 172.21.15.43:0/1979861923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a8105bd0 msgr2=0x7f74a819d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.246+0000 7f74a7e00700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a8105bd0 0x7f74a819d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.246+0000 7f74a7e00700 1 -- 172.21.15.43:0/1979861923 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a81a1c70 con 0x7f74a810e6f0 2024-05-27T07:33:28.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.247+0000 7f74a7e00700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74a810e6f0 0x7f74a81a15d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f74a000ef30 tx=0x7f74a000c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:28.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.247+0000 7f749fe00700 1 -- 172.21.15.43:0/1979861923 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a00092e0 con 0x7f74a810e6f0 2024-05-27T07:33:28.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.247+0000 7f749fe00700 1 -- 172.21.15.43:0/1979861923 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f74a0004bb0 con 0x7f74a810e6f0 2024-05-27T07:33:28.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.247+0000 7f74ad000700 1 -- 172.21.15.43:0/1979861923 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a81a1f00 con 0x7f74a810e6f0 2024-05-27T07:33:28.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.247+0000 7f74ad000700 1 -- 172.21.15.43:0/1979861923 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a81a9880 con 0x7f74a810e6f0 2024-05-27T07:33:28.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.248+0000 7f749fe00700 1 -- 172.21.15.43:0/1979861923 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0004280 con 0x7f74a810e6f0 2024-05-27T07:33:28.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.253+0000 7f749fe00700 1 -- 172.21.15.43:0/1979861923 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f74a001b030 con 0x7f74a810e6f0 2024-05-27T07:33:28.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.255+0000 7f74bd16fb80 1 Processor -- start 2024-05-27T07:33:28.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.255+0000 7f74bd16fb80 1 -- start start 2024-05-27T07:33:28.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.256+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 0x55e65bf1a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.256+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bd9ae40 0x55e65bf1ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.256+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bf242c0 0x55e65bf266e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.256+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e65be47e40 con 0x55e65be869f0 2024-05-27T07:33:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.256+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e65be379e0 con 0x55e65bd9ae40 2024-05-27T07:33:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.256+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e65be3ba20 con 0x55e65bf242c0 2024-05-27T07:33:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.256+0000 7f74a6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 0x55e65bf1a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.257+0000 7f74a6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 0x55e65bf1a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39220/0 (socket says 172.21.15.43:39220) 2024-05-27T07:33:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.257+0000 7f74a6a00700 1 -- 172.21.15.43:0/1117448656 learned_addr learned my addr 172.21.15.43:0/1117448656 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.257+0000 7f74a6a00700 1 --2- 172.21.15.43:0/1117448656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bf242c0 0x55e65bf266e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.257+0000 7f74a6a00700 1 -- 172.21.15.43:0/1117448656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bf242c0 msgr2=0x55e65bf266e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.257+0000 7f74a6a00700 1 --2- 172.21.15.43:0/1117448656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bf242c0 0x55e65bf266e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.257+0000 7f74a6a00700 1 -- 172.21.15.43:0/1117448656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bd9ae40 msgr2=0x55e65bf1ac90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:33:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.257+0000 7f74a6a00700 1 --2- 172.21.15.43:0/1117448656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bd9ae40 0x55e65bf1ac90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.257+0000 7f74a6a00700 1 -- 172.21.15.43:0/1117448656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e65be00420 con 0x55e65be869f0 2024-05-27T07:33:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.258+0000 7f74a6a00700 1 --2- 172.21.15.43:0/1117448656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 0x55e65bf1a750 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f7494009750 tx=0x7f7494005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65020f5c40763472 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.258+0000 7f749cc00700 1 -- 172.21.15.43:0/1117448656 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494004990 con 0x55e65be869f0 2024-05-27T07:33:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.258+0000 7f749cc00700 1 -- 172.21.15.43:0/1117448656 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f749401c070 con 0x55e65be869f0 2024-05-27T07:33:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.259+0000 7f749cc00700 1 -- 172.21.15.43:0/1117448656 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749400bea0 con 0x55e65be869f0 2024-05-27T07:33:28.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.259+0000 7f749fe00700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f748c0b2a70 0x7f748c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.259+0000 7f749fe00700 1 -- 172.21.15.43:0/1979861923 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7f74a00e24d0 con 0x7f74a810e6f0 2024-05-27T07:33:28.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.259+0000 7f74a7400700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f748c0b2a70 0x7f748c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.260+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1117448656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 msgr2=0x55e65bf1a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.260+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/1117448656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 0x55e65bf1a750 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f7494009750 tx=0x7f7494005ea0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.260+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1117448656 shutdown_connections 2024-05-27T07:33:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.260+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/1117448656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bf242c0 0x55e65bf266e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.260+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/1117448656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 0x55e65bf1a750 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.260+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/1117448656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bd9ae40 0x55e65bf1ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.260+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1117448656 >> 172.21.15.43:0/1117448656 conn(0x55e65be34390 msgr2=0x55e65bf15710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.261+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1117448656 shutdown_connections 2024-05-27T07:33:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.261+0000 7f74a7400700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f748c0b2a70 0x7f748c0b4f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55e65bf16970 tx=0x7f7498005c80 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.261+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1117448656 wait complete. 2024-05-27T07:33:28.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.261+0000 7f74bd16fb80 1 -- start start 2024-05-27T07:33:28.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.262+0000 7f74bd16fb80 1 Processor -- start 2024-05-27T07:33:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.263+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bd9ae40 0x55e65bf21250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.263+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 0x55e65bf21790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.263+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 0x55e65bf21cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.263+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e65bd3dfa0 con 0x55e65be869f0 2024-05-27T07:33:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.263+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e65bdd8760 con 0x55e65bf242c0 2024-05-27T07:33:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.263+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e65bf26da0 con 0x55e65bd9ae40 2024-05-27T07:33:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.263+0000 7f74bd16fb80 10 client.0 _ll_register_callbacks cb 0x55e65bde1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.264+0000 7f74a7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 0x55e65bf21cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.264+0000 7f74a7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 0x55e65bf21cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:36094/0 (socket says 172.21.15.43:36094) 2024-05-27T07:33:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.264+0000 7f74a7e00700 1 -- 172.21.15.43:0/375406644 learned_addr learned my addr 172.21.15.43:0/375406644 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:28.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.265+0000 7f74a7400700 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bd9ae40 0x55e65bf21250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.265+0000 7f74a6a00700 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 0x55e65bf21790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.265+0000 7f74a7e00700 1 -- 172.21.15.43:0/375406644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bd9ae40 msgr2=0x55e65bf21250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:28.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.265+0000 7f74a7e00700 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bd9ae40 0x55e65bf21250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.265+0000 7f74a7e00700 1 -- 172.21.15.43:0/375406644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 msgr2=0x55e65bf21790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:28.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.265+0000 7f74a7e00700 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 0x55e65bf21790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.265+0000 7f74a7e00700 1 -- 172.21.15.43:0/375406644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e65bd878d0 con 0x55e65bf242c0 2024-05-27T07:33:28.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.267+0000 7f74a7e00700 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 0x55e65bf21cd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009c10 tx=0x7f74a00a8f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fde9a0e1f2e5609 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:28.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.267+0000 7f7491600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:33:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.268+0000 7f7491600700 1 -- 172.21.15.43:0/375406644 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a00a9410 con 0x55e65bf242c0 2024-05-27T07:33:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.268+0000 7f7491600700 1 -- 172.21.15.43:0/375406644 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7f74a00b1d50 con 0x55e65bf242c0 2024-05-27T07:33:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.268+0000 7f74bd16fb80 1 -- 172.21.15.43:0/375406644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f749401c070 con 0x55e65bf242c0 2024-05-27T07:33:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.268+0000 7f7491600700 1 -- 172.21.15.43:0/375406644 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f74a00ad7c0 con 0x55e65bf242c0 2024-05-27T07:33:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.269+0000 7f74bd16fb80 10 client.37433 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:28.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.269+0000 7f74bd16fb80 1 -- 172.21.15.43:0/375406644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x55e65be08060 con 0x55e65bf242c0 2024-05-27T07:33:28.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.269+0000 7f7491600700 1 -- 172.21.15.43:0/375406644 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f749401c070 con 0x55e65bf242c0 2024-05-27T07:33:28.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.269+0000 7f7491600700 1 -- 172.21.15.43:0/375406644 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f74a00a8790 con 0x55e65bf242c0 2024-05-27T07:33:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.270+0000 7f74bd16fb80 10 client.37433 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.270+0000 7f74bd16fb80 10 client.37433 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:33:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.270+0000 7f74bd16fb80 20 client.37433 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:28.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.270+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e65bf3c910 0x55e65bf3edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.270+0000 7f74bd16fb80 4 client.37433 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:28.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.270+0000 7f74bd16fb80 1 -- 172.21.15.43:0/375406644 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e65be49d90 con 0x55e65bf3c910 2024-05-27T07:33:28.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.270+0000 7f74a7400700 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e65bf3c910 0x55e65bf3edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.271+0000 7f74a7400700 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e65bf3c910 0x55e65bf3edc0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a1d9867918b5072 server_cookie=9ec188726cc676fc in_seq=0 out_seq=0 2024-05-27T07:33:28.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.271+0000 7f7491600700 10 client.37433 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:33:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.272+0000 7f7491600700 1 -- 172.21.15.43:0/375406644 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e65be49d90 con 0x55e65bf3c910 2024-05-27T07:33:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.272+0000 7f7491600700 10 client.37433 handle_command_reply: tid=0 2024-05-27T07:33:28.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.272+0000 7f74bd16fb80 1 client.37433 shutdown 2024-05-27T07:33:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.273+0000 7f74bd16fb80 1 -- 172.21.15.43:0/375406644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 msgr2=0x55e65bf21cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.273+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 0x55e65bf21cd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009c10 tx=0x7f74a00a8f20 comp rx=0 tx=0).stop 2024-05-27T07:33:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.273+0000 7f74bd16fb80 1 -- 172.21.15.43:0/375406644 shutdown_connections 2024-05-27T07:33:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.273+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e65bf3c910 0x55e65bf3edc0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.273+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 0x55e65bf21cd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.273+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bd9ae40 0x55e65bf21250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.273+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/375406644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65be869f0 0x55e65bf21790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.273+0000 7f7491600700 0 client.37433 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:33:28.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.273+0000 7f74bd16fb80 1 -- 172.21.15.43:0/375406644 >> 172.21.15.43:0/375406644 conn(0x55e65be34390 msgr2=0x55e65bf15260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.274+0000 7f74bd16fb80 1 -- 172.21.15.43:0/375406644 shutdown_connections 2024-05-27T07:33:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.274+0000 7f74bd16fb80 1 -- 172.21.15.43:0/375406644 wait complete. 2024-05-27T07:33:28.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.275+0000 7f74bd16fb80 20 client.37433 trim_cache size 0 max 16384 2024-05-27T07:33:28.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.305+0000 7f74bd16fb80 1 Processor -- start 2024-05-27T07:33:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.305+0000 7f74bd16fb80 1 -- start start 2024-05-27T07:33:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.306+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bd9ae40 0x55e65bf1adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.306+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65be869f0 0x55e65bf38760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.306+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 0x55e65bf19830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.306+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f748800f210 con 0x55e65bd9ae40 2024-05-27T07:33:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.306+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f748800d670 con 0x55e65bf242c0 2024-05-27T07:33:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.306+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f748800eb40 con 0x55e65be869f0 2024-05-27T07:33:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.307+0000 7f74a6a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65be869f0 0x55e65bf38760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.307+0000 7f74a7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bd9ae40 0x55e65bf1adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.307+0000 7f74a7400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bd9ae40 0x55e65bf1adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39232/0 (socket says 172.21.15.43:39232) 2024-05-27T07:33:28.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.307+0000 7f74a7400700 1 -- 172.21.15.43:0/4222419945 learned_addr learned my addr 172.21.15.43:0/4222419945 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:28.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.308+0000 7f74a7400700 1 -- 172.21.15.43:0/4222419945 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65be869f0 msgr2=0x55e65bf38760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:28.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.308+0000 7f74a7400700 1 --2- 172.21.15.43:0/4222419945 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65be869f0 0x55e65bf38760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.308+0000 7f74a7400700 1 -- 172.21.15.43:0/4222419945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 msgr2=0x55e65bf19830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:28.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.308+0000 7f74a7400700 1 --2- 172.21.15.43:0/4222419945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 0x55e65bf19830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.308+0000 7f74a7400700 1 -- 172.21.15.43:0/4222419945 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e65bf0f930 con 0x55e65bd9ae40 2024-05-27T07:33:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.309+0000 7f74a7400700 1 --2- 172.21.15.43:0/4222419945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bd9ae40 0x55e65bf1adc0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x55e65bf20ad0 tx=0x7f749800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8ea53f7dcda1a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:28.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.309+0000 7f7493e00700 1 -- 172.21.15.43:0/4222419945 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749801fb10 con 0x55e65bd9ae40 2024-05-27T07:33:28.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.310+0000 7f7493e00700 1 -- 172.21.15.43:0/4222419945 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7498028030 con 0x55e65bd9ae40 2024-05-27T07:33:28.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.310+0000 7f7493e00700 1 -- 172.21.15.43:0/4222419945 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498003aa0 con 0x55e65bd9ae40 2024-05-27T07:33:28.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.312+0000 7f74bd16fb80 1 -- 172.21.15.43:0/4222419945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bd9ae40 msgr2=0x55e65bf1adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.312+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/4222419945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bd9ae40 0x55e65bf1adc0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x55e65bf20ad0 tx=0x7f749800f000 comp rx=0 tx=0).stop 2024-05-27T07:33:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.312+0000 7f74bd16fb80 1 -- 172.21.15.43:0/4222419945 shutdown_connections 2024-05-27T07:33:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.312+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/4222419945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65bf242c0 0x55e65bf19830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.312+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/4222419945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bd9ae40 0x55e65bf1adc0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.312+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/4222419945 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65be869f0 0x55e65bf38760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.312+0000 7f74bd16fb80 1 -- 172.21.15.43:0/4222419945 >> 172.21.15.43:0/4222419945 conn(0x55e65be34390 msgr2=0x55e65bf19120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.312+0000 7f74bd16fb80 1 -- 172.21.15.43:0/4222419945 shutdown_connections 2024-05-27T07:33:28.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.312+0000 7f74bd16fb80 1 -- 172.21.15.43:0/4222419945 wait complete. 2024-05-27T07:33:28.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.313+0000 7f74bd16fb80 1 -- start start 2024-05-27T07:33:28.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.313+0000 7f74bd16fb80 1 Processor -- start 2024-05-27T07:33:28.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.314+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65be869f0 0x55e65bfa2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.314+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bf242c0 0x55e65bf34950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.314+0000 7f74bd16fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bf34e90 0x55e65bf9fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.314+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f748800e470 con 0x55e65bf34e90 2024-05-27T07:33:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.314+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f748800dd40 con 0x55e65be869f0 2024-05-27T07:33:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.314+0000 7f74bd16fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e65bdd8760 con 0x55e65bf242c0 2024-05-27T07:33:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.314+0000 7f74bd16fb80 10 client.0 _ll_register_callbacks cb 0x55e65bde1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.315+0000 7f74a7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bf34e90 0x55e65bf9fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.315+0000 7f74a7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bf34e90 0x55e65bf9fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39240/0 (socket says 172.21.15.43:39240) 2024-05-27T07:33:28.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.315+0000 7f74a7e00700 1 -- 172.21.15.43:0/1492978365 learned_addr learned my addr 172.21.15.43:0/1492978365 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.315+0000 7f74a7e00700 1 -- 172.21.15.43:0/1492978365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bf242c0 msgr2=0x55e65bf34950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.315+0000 7f74a7e00700 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bf242c0 0x55e65bf34950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.315+0000 7f74a7e00700 1 -- 172.21.15.43:0/1492978365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65be869f0 msgr2=0x55e65bfa2a80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:33:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.315+0000 7f74a7e00700 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65be869f0 0x55e65bfa2a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.315+0000 7f74a7e00700 1 -- 172.21.15.43:0/1492978365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e65bf352a0 con 0x55e65bf34e90 2024-05-27T07:33:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.316+0000 7f74a7e00700 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bf34e90 0x55e65bf9fe00 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f74a00a6680 tx=0x7f74a00a9950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c6d8850046c1f00 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:28.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.316+0000 7f7491600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:33:28.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.316+0000 7f7491600700 1 -- 172.21.15.43:0/1492978365 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a00ad5c0 con 0x55e65bf34e90 2024-05-27T07:33:28.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.316+0000 7f7491600700 1 -- 172.21.15.43:0/1492978365 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7f74a00b1f60 con 0x55e65bf34e90 2024-05-27T07:33:28.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.316+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1492978365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7498028030 con 0x55e65bf34e90 2024-05-27T07:33:28.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.317+0000 7f7491600700 1 -- 172.21.15.43:0/1492978365 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f74a00a9410 con 0x55e65bf34e90 2024-05-27T07:33:28.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.317+0000 7f74bd16fb80 10 client.18066 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:28.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.317+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1492978365 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x55e65bfb8720 con 0x55e65bf34e90 2024-05-27T07:33:28.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.317+0000 7f7491600700 1 -- 172.21.15.43:0/1492978365 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7498028030 con 0x55e65bf34e90 2024-05-27T07:33:28.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.318+0000 7f7491600700 1 -- 172.21.15.43:0/1492978365 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f74a00a8670 con 0x55e65bf34e90 2024-05-27T07:33:28.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.318+0000 7f74bd16fb80 10 client.18066 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:28.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.318+0000 7f74bd16fb80 10 client.18066 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:33:28.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.318+0000 7f74bd16fb80 20 client.18066 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:28.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.318+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e65bfb8e80 0x55e65bfbb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:28.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.318+0000 7f74bd16fb80 4 client.18066 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:28.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.318+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1492978365 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e65be43570 con 0x55e65bfb8e80 2024-05-27T07:33:28.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.318+0000 7f74a7400700 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e65bfb8e80 0x55e65bfbb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:28.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f7491600700 10 client.18066 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:33:28.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f74a7400700 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e65bfb8e80 0x55e65bfbb2a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b6511583b1758c5c server_cookie=5e0ae96ec49cae26 in_seq=0 out_seq=0 2024-05-27T07:33:28.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f7491600700 1 -- 172.21.15.43:0/1492978365 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e65be43570 con 0x55e65bfb8e80 2024-05-27T07:33:28.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f7491600700 10 client.18066 handle_command_reply: tid=0 2024-05-27T07:33:28.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f74bd16fb80 1 client.18066 shutdown 2024-05-27T07:33:28.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1492978365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bf34e90 msgr2=0x55e65bf9fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bf34e90 0x55e65bf9fe00 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f74a00a6680 tx=0x7f74a00a9950 comp rx=0 tx=0).stop 2024-05-27T07:33:28.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1492978365 shutdown_connections 2024-05-27T07:33:28.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55e65bfb8e80 0x55e65bfbb2a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e65bf34e90 0x55e65bf9fe00 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e65be869f0 0x55e65bfa2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f74bd16fb80 1 --2- 172.21.15.43:0/1492978365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e65bf242c0 0x55e65bf34950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.320+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1492978365 >> 172.21.15.43:0/1492978365 conn(0x55e65be34390 msgr2=0x55e65bf21030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:28.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.321+0000 7f7491600700 0 client.18066 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:33:28.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.321+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1492978365 shutdown_connections 2024-05-27T07:33:28.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.321+0000 7f74bd16fb80 1 -- 172.21.15.43:0/1492978365 wait complete. 2024-05-27T07:33:28.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.321+0000 7f74bd16fb80 20 client.18066 trim_cache size 0 max 16384 2024-05-27T07:33:28.317 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:33:28.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.322+0000 7f74ad000700 1 -- 172.21.15.43:0/1979861923 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f748c0b2a70 msgr2=0x7f748c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.322+0000 7f74ad000700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f748c0b2a70 0x7f748c0b4f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55e65bf16970 tx=0x7f7498005c80 comp rx=0 tx=0).stop 2024-05-27T07:33:28.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.322+0000 7f74ad000700 1 -- 172.21.15.43:0/1979861923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74a810e6f0 msgr2=0x7f74a81a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:28.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.322+0000 7f74ad000700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74a810e6f0 0x7f74a81a15d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f74a000ef30 tx=0x7f74a000c450 comp rx=0 tx=0).stop 2024-05-27T07:33:28.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.323+0000 7f74ad000700 1 -- 172.21.15.43:0/1979861923 shutdown_connections 2024-05-27T07:33:28.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.323+0000 7f74ad000700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f748c0b2a70 0x7f748c0b4f80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.323+0000 7f74ad000700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f74a810e6f0 0x7f74a81a15d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.323+0000 7f74ad000700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74a8105280 0x7f74a819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.323+0000 7f74ad000700 1 --2- 172.21.15.43:0/1979861923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74a8105bd0 0x7f74a819d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:28.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.323+0000 7f74ad000700 1 -- 172.21.15.43:0/1979861923 >> 172.21.15.43:0/1979861923 conn(0x7f74a8078ab0 msgr2=0x7f74a8102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:28.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.323+0000 7f74ad000700 1 -- 172.21.15.43:0/1979861923 shutdown_connections 2024-05-27T07:33:28.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:28.323+0000 7f74ad000700 1 -- 172.21.15.43:0/1979861923 wait complete. 2024-05-27T07:33:28.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:28 smithi043 ceph-mon[31531]: pgmap v657: 129 pgs: 129 active+clean; 853 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 4.2 MiB/s wr, 221 op/s 2024-05-27T07:33:28.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 36378 5.86 MB/sec execute 209 sec latency 226.967 ms 2024-05-27T07:33:28.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:28 smithi132 ceph-mon[35283]: pgmap v657: 129 pgs: 129 active+clean; 853 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 4.2 MiB/s wr, 221 op/s 2024-05-27T07:33:28.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:28 smithi179 ceph-mon[35888]: pgmap v657: 129 pgs: 129 active+clean; 853 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 4.2 MiB/s wr, 221 op/s 2024-05-27T07:33:29.258 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:33:29.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 36467 5.84 MB/sec execute 210 sec latency 88.827 ms 2024-05-27T07:33:29.860 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:30.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:30 smithi043 ceph-mon[31531]: pgmap v658: 129 pgs: 129 active+clean; 853 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.0 MiB/s wr, 156 op/s 2024-05-27T07:33:30.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 36596 5.86 MB/sec execute 211 sec latency 82.307 ms 2024-05-27T07:33:30.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:30 smithi132 ceph-mon[35283]: pgmap v658: 129 pgs: 129 active+clean; 853 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.0 MiB/s wr, 156 op/s 2024-05-27T07:33:30.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:30 smithi179 ceph-mon[35888]: pgmap v658: 129 pgs: 129 active+clean; 853 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.0 MiB/s wr, 156 op/s 2024-05-27T07:33:31.795 INFO:tasks.workunit.client.0.smithi043.stdout: 10 36697 5.85 MB/sec execute 212 sec latency 80.715 ms 2024-05-27T07:33:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.210+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2301411383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f0109840 msgr2=0x7fd2f01111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.210+0000 7fd2f4c00700 1 --2- 172.21.15.43:0/2301411383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f0109840 0x7fd2f01111a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800b0d0 tx=0x7fd2e800b3e0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.210+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2301411383 shutdown_connections 2024-05-27T07:33:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.210+0000 7fd2f4c00700 1 --2- 172.21.15.43:0/2301411383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f0109840 0x7fd2f01111a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.210+0000 7fd2f4c00700 1 --2- 172.21.15.43:0/2301411383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2f01045c0 0x7fd2f01049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.210+0000 7fd2f4c00700 1 --2- 172.21.15.43:0/2301411383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2f0108ef0 0x7fd2f0109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.210+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2301411383 >> 172.21.15.43:0/2301411383 conn(0x7fd2f00ffe90 msgr2=0x7fd2f01022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.213+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2301411383 shutdown_connections 2024-05-27T07:33:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.213+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2301411383 wait complete. 2024-05-27T07:33:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.215+0000 7fd2f4c00700 1 Processor -- start 2024-05-27T07:33:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.215+0000 7fd2f4c00700 1 -- start start 2024-05-27T07:33:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.216+0000 7fd2f4c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2f01045c0 0x7fd2f01a2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.216+0000 7fd2f4c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f0108ef0 0x7fd2f01a2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.216+0000 7fd2f4c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2f0109840 0x7fd2f01ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.216+0000 7fd2f4c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f0114370 con 0x7fd2f0108ef0 2024-05-27T07:33:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.216+0000 7fd2f4c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01141f0 con 0x7fd2f01045c0 2024-05-27T07:33:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.216+0000 7fd2f4c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01144f0 con 0x7fd2f0109840 2024-05-27T07:33:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.217+0000 7fd2ede00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2f01045c0 0x7fd2f01a2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.217+0000 7fd2ede00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2f01045c0 0x7fd2f01a2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:36122/0 (socket says 172.21.15.43:36122) 2024-05-27T07:33:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.217+0000 7fd2ede00700 1 -- 172.21.15.43:0/2621974856 learned_addr learned my addr 172.21.15.43:0/2621974856 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.217+0000 7fd2e7e00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f0108ef0 0x7fd2f01a2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.217+0000 7fd2ede00700 1 -- 172.21.15.43:0/2621974856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2f0109840 msgr2=0x7fd2f01ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.217+0000 7fd2ede00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2f0109840 0x7fd2f01ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.217+0000 7fd2ede00700 1 -- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f0108ef0 msgr2=0x7fd2f01a2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.217+0000 7fd2ede00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f0108ef0 0x7fd2f01a2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.217+0000 7fd2ede00700 1 -- 172.21.15.43:0/2621974856 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f01ad610 con 0x7fd2f01045c0 2024-05-27T07:33:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.218+0000 7fd2e7e00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f0108ef0 0x7fd2f01a2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:33:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.218+0000 7fd2ede00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2f01045c0 0x7fd2f01a2050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000e9a0 tx=0x7fd2e000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.219+0000 7fd2e6a00700 1 -- 172.21.15.43:0/2621974856 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e000cbd0 con 0x7fd2f01045c0 2024-05-27T07:33:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.219+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2621974856 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f01ad840 con 0x7fd2f01045c0 2024-05-27T07:33:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.219+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2621974856 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f01add80 con 0x7fd2f01045c0 2024-05-27T07:33:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.219+0000 7fd2e6a00700 1 -- 172.21.15.43:0/2621974856 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2e000cd70 con 0x7fd2f01045c0 2024-05-27T07:33:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.219+0000 7fd2e6a00700 1 -- 172.21.15.43:0/2621974856 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0018680 con 0x7fd2f01045c0 2024-05-27T07:33:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.225+0000 7fd2e6a00700 1 -- 172.21.15.43:0/2621974856 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd2e0018820 con 0x7fd2f01045c0 2024-05-27T07:33:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.227+0000 7fd30368eb80 1 Processor -- start 2024-05-27T07:33:32.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.228+0000 7fd2e6a00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd2d40b2a70 0x7fd2d40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.228+0000 7fd2e6a00700 1 -- 172.21.15.43:0/2621974856 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7fd2e00e2c90 con 0x7fd2f01045c0 2024-05-27T07:33:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.228+0000 7fd2ee800700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd2d40b2a70 0x7fd2d40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.229+0000 7fd30368eb80 1 -- start start 2024-05-27T07:33:32.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.229+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e7680330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.229+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e7680870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.229+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7680e70 0x5628e768d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.229+0000 7fd30368eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5628e75aee40 con 0x5628e7680e70 2024-05-27T07:33:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.229+0000 7fd30368eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5628e759e9e0 con 0x5628e7501e40 2024-05-27T07:33:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.229+0000 7fd30368eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5628e75a2a20 con 0x5628e75ed9f0 2024-05-27T07:33:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.230+0000 7fd2ee800700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd2d40b2a70 0x7fd2d40b4f80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009900 tx=0x7fd2e800b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.230+0000 7fd2e7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e7680870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.230+0000 7fd2ee800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7680e70 0x5628e768d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.230+0000 7fd2ee800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7680e70 0x5628e768d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39258/0 (socket says 172.21.15.43:39258) 2024-05-27T07:33:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.230+0000 7fd2ee800700 1 -- 172.21.15.43:0/2100040478 learned_addr learned my addr 172.21.15.43:0/2100040478 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.230+0000 7fd2ede00700 1 --2- 172.21.15.43:0/2100040478 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e7680330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.231+0000 7fd2e7e00700 1 -- 172.21.15.43:0/2100040478 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 msgr2=0x5628e7680330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.231+0000 7fd2e7e00700 1 --2- 172.21.15.43:0/2100040478 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e7680330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.231+0000 7fd2e7e00700 1 -- 172.21.15.43:0/2100040478 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7680e70 msgr2=0x5628e768d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.231+0000 7fd2e7e00700 1 --2- 172.21.15.43:0/2100040478 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7680e70 0x5628e768d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.231+0000 7fd2e7e00700 1 -- 172.21.15.43:0/2100040478 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628e7567420 con 0x5628e7501e40 2024-05-27T07:33:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.231+0000 7fd2ee800700 1 --2- 172.21.15.43:0/2100040478 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7680e70 0x5628e768d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.232+0000 7fd2e7e00700 1 --2- 172.21.15.43:0/2100040478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e7680870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd2f006c160 tx=0x7fd2d8009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81afad86e2964cf2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.232+0000 7fd2df400700 1 -- 172.21.15.43:0/2100040478 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8010be0 con 0x5628e7501e40 2024-05-27T07:33:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.232+0000 7fd2df400700 1 -- 172.21.15.43:0/2100040478 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2d80044b0 con 0x5628e7501e40 2024-05-27T07:33:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.232+0000 7fd2df400700 1 -- 172.21.15.43:0/2100040478 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8003e00 con 0x5628e7501e40 2024-05-27T07:33:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.234+0000 7fd30368eb80 1 -- 172.21.15.43:0/2100040478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 msgr2=0x5628e7680870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.234+0000 7fd30368eb80 1 --2- 172.21.15.43:0/2100040478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e7680870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd2f006c160 tx=0x7fd2d8009f50 comp rx=0 tx=0).stop 2024-05-27T07:33:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.234+0000 7fd30368eb80 1 -- 172.21.15.43:0/2100040478 shutdown_connections 2024-05-27T07:33:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.234+0000 7fd30368eb80 1 --2- 172.21.15.43:0/2100040478 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7680e70 0x5628e768d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.234+0000 7fd30368eb80 1 --2- 172.21.15.43:0/2100040478 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e7680330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.234+0000 7fd30368eb80 1 --2- 172.21.15.43:0/2100040478 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e7680870 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.234+0000 7fd30368eb80 1 -- 172.21.15.43:0/2100040478 >> 172.21.15.43:0/2100040478 conn(0x5628e759b390 msgr2=0x5628e767c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.235+0000 7fd30368eb80 1 -- 172.21.15.43:0/2100040478 shutdown_connections 2024-05-27T07:33:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.235+0000 7fd30368eb80 1 -- 172.21.15.43:0/2100040478 wait complete. 2024-05-27T07:33:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.236+0000 7fd30368eb80 1 -- start start 2024-05-27T07:33:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.236+0000 7fd30368eb80 1 Processor -- start 2024-05-27T07:33:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.237+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7501e40 0x5628e768a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.238+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e768ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.238+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.238+0000 7fd30368eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5628e74a4fa0 con 0x5628e7501e40 2024-05-27T07:33:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.238+0000 7fd30368eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5628e753f760 con 0x5628e7680e70 2024-05-27T07:33:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.238+0000 7fd2e7e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e768ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.238+0000 7fd30368eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5628e768db80 con 0x5628e75ed9f0 2024-05-27T07:33:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.238+0000 7fd2ee800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.238+0000 7fd2ede00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7501e40 0x5628e768a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.238+0000 7fd2e7e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e768ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:47376/0 (socket says 172.21.15.43:47376) 2024-05-27T07:33:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.238+0000 7fd2e7e00700 1 -- 172.21.15.43:0/3056571625 learned_addr learned my addr 172.21.15.43:0/3056571625 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.239+0000 7fd30368eb80 10 client.0 _ll_register_callbacks cb 0x5628e7548c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.239+0000 7fd2ee800700 1 -- 172.21.15.43:0/3056571625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 msgr2=0x5628e768ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.239+0000 7fd2ee800700 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e768ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.239+0000 7fd2ee800700 1 -- 172.21.15.43:0/3056571625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7501e40 msgr2=0x5628e768a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.239+0000 7fd2ee800700 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7501e40 0x5628e768a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.239+0000 7fd2ee800700 1 -- 172.21.15.43:0/3056571625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628e74ee8d0 con 0x5628e7680e70 2024-05-27T07:33:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.240+0000 7fd2e7e00700 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e768ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:33:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.240+0000 7fd2ee800700 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768b0d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009e60 tx=0x7fd2e80099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77fbee3899bb8926 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.240+0000 7fd2d3e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:33:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.241+0000 7fd2d3e00700 1 -- 172.21.15.43:0/3056571625 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e802d030 con 0x5628e7680e70 2024-05-27T07:33:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.241+0000 7fd2d3e00700 1 -- 172.21.15.43:0/3056571625 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7fd2e80077d0 con 0x5628e7680e70 2024-05-27T07:33:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.241+0000 7fd30368eb80 1 -- 172.21.15.43:0/3056571625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2d80044b0 con 0x5628e7680e70 2024-05-27T07:33:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.241+0000 7fd2d3e00700 1 -- 172.21.15.43:0/3056571625 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7fd2e803a050 con 0x5628e7680e70 2024-05-27T07:33:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.241+0000 7fd30368eb80 10 client.37445 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.241+0000 7fd30368eb80 1 -- 172.21.15.43:0/3056571625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x5628e756f060 con 0x5628e7680e70 2024-05-27T07:33:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.242+0000 7fd2d3e00700 1 -- 172.21.15.43:0/3056571625 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2d80044b0 con 0x5628e7680e70 2024-05-27T07:33:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.244+0000 7fd2d3e00700 1 -- 172.21.15.43:0/3056571625 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7fd2e8011070 con 0x5628e7680e70 2024-05-27T07:33:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.244+0000 7fd30368eb80 10 client.37445 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.244+0000 7fd30368eb80 10 client.37445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.244+0000 7fd30368eb80 20 client.37445 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.244+0000 7fd30368eb80 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5628e769aee0 0x5628e769d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.244+0000 7fd30368eb80 4 client.37445 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.244+0000 7fd30368eb80 1 -- 172.21.15.43:0/3056571625 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628e75b0d90 con 0x5628e769aee0 2024-05-27T07:33:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.244+0000 7fd2ede00700 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5628e769aee0 0x5628e769d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.245+0000 7fd2ede00700 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5628e769aee0 0x5628e769d370 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7455f6a59e38b68 server_cookie=5db9c7c0a8ece787 in_seq=0 out_seq=0 2024-05-27T07:33:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.245+0000 7fd2d3e00700 10 client.37445 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.246+0000 7fd2d3e00700 1 -- 172.21.15.43:0/3056571625 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5628e75b0d90 con 0x5628e769aee0 2024-05-27T07:33:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.246+0000 7fd2d3e00700 10 client.37445 handle_command_reply: tid=0 2024-05-27T07:33:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.247+0000 7fd30368eb80 1 client.37445 shutdown 2024-05-27T07:33:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.247+0000 7fd30368eb80 1 -- 172.21.15.43:0/3056571625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 msgr2=0x5628e768b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.247+0000 7fd30368eb80 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768b0d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009e60 tx=0x7fd2e80099e0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.247+0000 7fd30368eb80 1 -- 172.21.15.43:0/3056571625 shutdown_connections 2024-05-27T07:33:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.247+0000 7fd30368eb80 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768b0d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.247+0000 7fd30368eb80 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5628e769aee0 0x5628e769d370 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.247+0000 7fd30368eb80 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e7501e40 0x5628e768a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.247+0000 7fd30368eb80 1 --2- 172.21.15.43:0/3056571625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e768ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.247+0000 7fd30368eb80 1 -- 172.21.15.43:0/3056571625 >> 172.21.15.43:0/3056571625 conn(0x5628e759b390 msgr2=0x5628e7680310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:32.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.248+0000 7fd2d3e00700 0 client.37445 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:32.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.248+0000 7fd30368eb80 1 -- 172.21.15.43:0/3056571625 shutdown_connections 2024-05-27T07:33:32.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.248+0000 7fd30368eb80 1 -- 172.21.15.43:0/3056571625 wait complete. 2024-05-27T07:33:32.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.248+0000 7fd30368eb80 20 client.37445 trim_cache size 0 max 16384 2024-05-27T07:33:32.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.278+0000 7fd30368eb80 1 Processor -- start 2024-05-27T07:33:32.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.278+0000 7fd30368eb80 1 -- start start 2024-05-27T07:33:32.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.278+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e769d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.278+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e75ed9f0 0x5628e76807c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.278+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e7680e70 0x5628e7696c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.278+0000 7fd30368eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc00f390 con 0x5628e75ed9f0 2024-05-27T07:33:32.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.278+0000 7fd30368eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc00d7f0 con 0x5628e7501e40 2024-05-27T07:33:32.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.278+0000 7fd30368eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc00ecc0 con 0x5628e7680e70 2024-05-27T07:33:32.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.279+0000 7fd2ede00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e769d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.279+0000 7fd2ede00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e769d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:36158/0 (socket says 172.21.15.43:36158) 2024-05-27T07:33:32.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.279+0000 7fd2ede00700 1 -- 172.21.15.43:0/2691882173 learned_addr learned my addr 172.21.15.43:0/2691882173 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:32.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.279+0000 7fd2e7e00700 1 --2- 172.21.15.43:0/2691882173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e75ed9f0 0x5628e76807c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.279+0000 7fd2ee800700 1 --2- 172.21.15.43:0/2691882173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e7680e70 0x5628e7696c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.280+0000 7fd2ede00700 1 -- 172.21.15.43:0/2691882173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e7680e70 msgr2=0x5628e7696c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:32.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.280+0000 7fd2ede00700 1 --2- 172.21.15.43:0/2691882173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e7680e70 0x5628e7696c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.280+0000 7fd2ede00700 1 -- 172.21.15.43:0/2691882173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e75ed9f0 msgr2=0x5628e76807c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:32.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.280+0000 7fd2ede00700 1 --2- 172.21.15.43:0/2691882173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e75ed9f0 0x5628e76807c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.280+0000 7fd2ede00700 1 -- 172.21.15.43:0/2691882173 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628e768b630 con 0x5628e7501e40 2024-05-27T07:33:32.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.281+0000 7fd2ede00700 1 --2- 172.21.15.43:0/2691882173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e769d560 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00a9840 tx=0x7fd2e001f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaaa5bf3024ea97e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:32.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.281+0000 7fd2dea00700 1 -- 172.21.15.43:0/2691882173 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e00a9650 con 0x5628e7501e40 2024-05-27T07:33:32.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.281+0000 7fd2dea00700 1 -- 172.21.15.43:0/2691882173 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2e00aa240 con 0x5628e7501e40 2024-05-27T07:33:32.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.281+0000 7fd2dea00700 1 -- 172.21.15.43:0/2691882173 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0010b20 con 0x5628e7501e40 2024-05-27T07:33:32.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.282+0000 7fd30368eb80 1 -- 172.21.15.43:0/2691882173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 msgr2=0x5628e769d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:32.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.282+0000 7fd30368eb80 1 --2- 172.21.15.43:0/2691882173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e769d560 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00a9840 tx=0x7fd2e001f000 comp rx=0 tx=0).stop 2024-05-27T07:33:32.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.283+0000 7fd30368eb80 1 -- 172.21.15.43:0/2691882173 shutdown_connections 2024-05-27T07:33:32.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.283+0000 7fd30368eb80 1 --2- 172.21.15.43:0/2691882173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e7680e70 0x5628e7696c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.283+0000 7fd30368eb80 1 --2- 172.21.15.43:0/2691882173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7501e40 0x5628e769d560 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.283+0000 7fd30368eb80 1 --2- 172.21.15.43:0/2691882173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e75ed9f0 0x5628e76807c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.283+0000 7fd30368eb80 1 -- 172.21.15.43:0/2691882173 >> 172.21.15.43:0/2691882173 conn(0x5628e759b390 msgr2=0x5628e76800b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:32.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.283+0000 7fd30368eb80 1 -- 172.21.15.43:0/2691882173 shutdown_connections 2024-05-27T07:33:32.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.283+0000 7fd30368eb80 1 -- 172.21.15.43:0/2691882173 wait complete. 2024-05-27T07:33:32.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.284+0000 7fd30368eb80 1 -- start start 2024-05-27T07:33:32.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.284+0000 7fd30368eb80 1 Processor -- start 2024-05-27T07:33:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e7693660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd30368eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e768e370 0x5628e768e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd30368eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc00e5f0 con 0x5628e768e370 2024-05-27T07:33:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd30368eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc00dec0 con 0x5628e7680e70 2024-05-27T07:33:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd30368eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5628e767bb90 con 0x5628e75ed9f0 2024-05-27T07:33:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd30368eb80 10 client.0 _ll_register_callbacks cb 0x5628e7548c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd2e7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd2ee800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e768e370 0x5628e768e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd2e7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:36174/0 (socket says 172.21.15.43:36174) 2024-05-27T07:33:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.285+0000 7fd2e7e00700 1 -- 172.21.15.43:0/1857795367 learned_addr learned my addr 172.21.15.43:0/1857795367 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.286+0000 7fd2ede00700 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e7693660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.286+0000 7fd2e7e00700 1 -- 172.21.15.43:0/1857795367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 msgr2=0x5628e7693660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.287+0000 7fd2e7e00700 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e7693660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.287+0000 7fd2e7e00700 1 -- 172.21.15.43:0/1857795367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e768e370 msgr2=0x5628e768e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.287+0000 7fd2e7e00700 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e768e370 0x5628e768e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.287+0000 7fd2e7e00700 1 -- 172.21.15.43:0/1857795367 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628e7714be0 con 0x5628e7680e70 2024-05-27T07:33:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.287+0000 7fd2e7e00700 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768de30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80046a0 tx=0x7fd2d80053b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62356b9b398318c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.287+0000 7fd2d3e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:33:32.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.288+0000 7fd2d3e00700 1 -- 172.21.15.43:0/1857795367 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d802b060 con 0x5628e7680e70 2024-05-27T07:33:32.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.288+0000 7fd2d3e00700 1 -- 172.21.15.43:0/1857795367 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7fd2d80048d0 con 0x5628e7680e70 2024-05-27T07:33:32.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.288+0000 7fd30368eb80 1 -- 172.21.15.43:0/1857795367 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2e00aa240 con 0x5628e7680e70 2024-05-27T07:33:32.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.289+0000 7fd2d3e00700 1 -- 172.21.15.43:0/1857795367 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7fd2d8032050 con 0x5628e7680e70 2024-05-27T07:33:32.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.289+0000 7fd30368eb80 10 client.37451 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:32.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.289+0000 7fd30368eb80 1 -- 172.21.15.43:0/1857795367 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x5628e7715e40 con 0x5628e7680e70 2024-05-27T07:33:32.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.290+0000 7fd2d3e00700 1 -- 172.21.15.43:0/1857795367 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd2e00aa240 con 0x5628e7680e70 2024-05-27T07:33:32.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.290+0000 7fd2d3e00700 1 -- 172.21.15.43:0/1857795367 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7fd2d8026b30 con 0x5628e7680e70 2024-05-27T07:33:32.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.290+0000 7fd30368eb80 10 client.37451 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:32.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.290+0000 7fd30368eb80 10 client.37451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:32.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.290+0000 7fd30368eb80 20 client.37451 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:32.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.290+0000 7fd30368eb80 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5628e7716440 0x5628e7718860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:32.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.290+0000 7fd30368eb80 4 client.37451 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:33:32.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.290+0000 7fd30368eb80 1 -- 172.21.15.43:0/1857795367 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628e75aa570 con 0x5628e7716440 2024-05-27T07:33:32.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.290+0000 7fd2ede00700 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5628e7716440 0x5628e7718860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:32.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.291+0000 7fd2ede00700 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5628e7716440 0x5628e7718860 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=389096c5d6f3a48e server_cookie=37e82c4877c6a31a in_seq=0 out_seq=0 2024-05-27T07:33:32.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.291+0000 7fd2d3e00700 10 client.37451 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:32.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.293+0000 7fd2d3e00700 1 -- 172.21.15.43:0/1857795367 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628e75aa570 con 0x5628e7716440 2024-05-27T07:33:32.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.293+0000 7fd2d3e00700 10 client.37451 handle_command_reply: tid=0 2024-05-27T07:33:32.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.293+0000 7fd30368eb80 1 client.37451 shutdown 2024-05-27T07:33:32.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd30368eb80 1 -- 172.21.15.43:0/1857795367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 msgr2=0x5628e768de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:32.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd30368eb80 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768de30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80046a0 tx=0x7fd2d80053b0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd30368eb80 1 -- 172.21.15.43:0/1857795367 shutdown_connections 2024-05-27T07:33:32.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd30368eb80 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5628e7716440 0x5628e7718860 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd30368eb80 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5628e768e370 0x5628e768e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd30368eb80 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628e75ed9f0 0x5628e7693660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd2d3e00700 0 client.37451 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:32.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd30368eb80 1 --2- 172.21.15.43:0/1857795367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5628e7680e70 0x5628e768de30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd30368eb80 1 -- 172.21.15.43:0/1857795367 >> 172.21.15.43:0/1857795367 conn(0x5628e759b390 msgr2=0x5628e768a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:32.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd30368eb80 1 -- 172.21.15.43:0/1857795367 shutdown_connections 2024-05-27T07:33:32.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.294+0000 7fd30368eb80 1 -- 172.21.15.43:0/1857795367 wait complete. 2024-05-27T07:33:32.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.295+0000 7fd30368eb80 20 client.37451 trim_cache size 0 max 16384 2024-05-27T07:33:32.291 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:33:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.295+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd2d40b2a70 msgr2=0x7fd2d40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:32.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.295+0000 7fd2f4c00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd2d40b2a70 0x7fd2d40b4f80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009900 tx=0x7fd2e800b040 comp rx=0 tx=0).stop 2024-05-27T07:33:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.295+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2621974856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2f01045c0 msgr2=0x7fd2f01a2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.295+0000 7fd2f4c00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2f01045c0 0x7fd2f01a2050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000e9a0 tx=0x7fd2e000ed60 comp rx=0 tx=0).stop 2024-05-27T07:33:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.296+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2621974856 shutdown_connections 2024-05-27T07:33:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.296+0000 7fd2f4c00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd2d40b2a70 0x7fd2d40b4f80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.296+0000 7fd2f4c00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2f0109840 0x7fd2f01ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.296+0000 7fd2f4c00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2f01045c0 0x7fd2f01a2050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.296+0000 7fd2f4c00700 1 --2- 172.21.15.43:0/2621974856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f0108ef0 0x7fd2f01a2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:32.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.296+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2621974856 >> 172.21.15.43:0/2621974856 conn(0x7fd2f00ffe90 msgr2=0x7fd2f010a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:32.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.297+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2621974856 shutdown_connections 2024-05-27T07:33:32.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:32.297+0000 7fd2f4c00700 1 -- 172.21.15.43:0/2621974856 wait complete. 2024-05-27T07:33:32.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 36829 5.87 MB/sec execute 213 sec latency 107.098 ms 2024-05-27T07:33:32.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:32 smithi132 ceph-mon[35283]: pgmap v659: 129 pgs: 129 active+clean; 864 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-05-27T07:33:32.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:32 smithi043 ceph-mon[31531]: pgmap v659: 129 pgs: 129 active+clean; 864 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-05-27T07:33:32.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:32 smithi179 ceph-mon[35888]: pgmap v659: 129 pgs: 129 active+clean; 864 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.6 MiB/s wr, 229 op/s 2024-05-27T07:33:33.794 INFO:tasks.workunit.client.0.smithi043.stdout: 10 36944 5.86 MB/sec execute 214 sec latency 79.622 ms 2024-05-27T07:33:34.134 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:33:34.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:33:34.530 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:34 smithi043 ceph-mon[31531]: pgmap v660: 129 pgs: 129 active+clean; 864 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-05-27T07:33:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:34.795 INFO:tasks.workunit.client.0.smithi043.stdout: 10 37060 5.86 MB/sec execute 215 sec latency 99.930 ms 2024-05-27T07:33:34.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:34 smithi132 ceph-mon[35283]: pgmap v660: 129 pgs: 129 active+clean; 864 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-05-27T07:33:34.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:34 smithi179 ceph-mon[35888]: pgmap v660: 129 pgs: 129 active+clean; 864 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-05-27T07:33:35.795 INFO:tasks.workunit.client.0.smithi043.stdout: 10 37150 5.85 MB/sec execute 216 sec latency 81.139 ms 2024-05-27T07:33:36.798 INFO:tasks.workunit.client.0.smithi043.stdout: 10 37274 5.85 MB/sec execute 217 sec latency 97.258 ms 2024-05-27T07:33:36.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:36 smithi132 ceph-mon[35283]: pgmap v661: 129 pgs: 129 active+clean; 864 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-05-27T07:33:36.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:36 smithi043 ceph-mon[31531]: pgmap v661: 129 pgs: 129 active+clean; 864 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-05-27T07:33:36.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:36 smithi179 ceph-mon[35888]: pgmap v661: 129 pgs: 129 active+clean; 864 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-05-27T07:33:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.531+0000 7f91c5600700 1 -- 172.21.15.43:0/2590420563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91c0105bd0 msgr2=0x7f91c010e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.531+0000 7f91c5600700 1 --2- 172.21.15.43:0/2590420563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91c0105bd0 0x7f91c010e120 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f91a8009990 tx=0x7f91a8009ca0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.531+0000 7f91c5600700 1 -- 172.21.15.43:0/2590420563 shutdown_connections 2024-05-27T07:33:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.531+0000 7f91c5600700 1 --2- 172.21.15.43:0/2590420563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91c010e6f0 0x7f91c0110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.531+0000 7f91c5600700 1 --2- 172.21.15.43:0/2590420563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91c0105280 0x7f91c0105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.531+0000 7f91c5600700 1 --2- 172.21.15.43:0/2590420563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91c0105bd0 0x7f91c010e120 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.531+0000 7f91c5600700 1 -- 172.21.15.43:0/2590420563 >> 172.21.15.43:0/2590420563 conn(0x7f91c0078ab0 msgr2=0x7f91c0078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:37.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.533+0000 7f91c5600700 1 -- 172.21.15.43:0/2590420563 shutdown_connections 2024-05-27T07:33:37.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.533+0000 7f91c5600700 1 -- 172.21.15.43:0/2590420563 wait complete. 2024-05-27T07:33:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.535+0000 7f91c5600700 1 Processor -- start 2024-05-27T07:33:37.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.535+0000 7f91c5600700 1 -- start start 2024-05-27T07:33:37.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.536+0000 7f91c5600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91c0105280 0x7f91c0102f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.536+0000 7f91c5600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91c0105bd0 0x7f91c01034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.536+0000 7f91c5600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91c010e6f0 0x7f91c01039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.536+0000 7f91c5600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f91c0114100 con 0x7f91c0105bd0 2024-05-27T07:33:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.536+0000 7f91c5600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f91c0113f80 con 0x7f91c010e6f0 2024-05-27T07:33:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.536+0000 7f91c5600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f91c0114280 con 0x7f91c0105280 2024-05-27T07:33:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.536+0000 7f91c4c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91c010e6f0 0x7f91c01039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.536+0000 7f91c4c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91c010e6f0 0x7f91c01039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:36968/0 (socket says 172.21.15.43:36968) 2024-05-27T07:33:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.536+0000 7f91c4c00700 1 -- 172.21.15.43:0/3896814873 learned_addr learned my addr 172.21.15.43:0/3896814873 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.537+0000 7f91bfe00700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91c0105280 0x7f91c0102f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.537+0000 7f91c4c00700 1 -- 172.21.15.43:0/3896814873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91c0105280 msgr2=0x7f91c0102f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.537+0000 7f91c4c00700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91c0105280 0x7f91c0102f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.537+0000 7f91bf400700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91c0105bd0 0x7f91c01034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.537+0000 7f91c4c00700 1 -- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91c0105bd0 msgr2=0x7f91c01034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.537+0000 7f91c4c00700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91c0105bd0 0x7f91c01034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.537+0000 7f91c4c00700 1 -- 172.21.15.43:0/3896814873 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c01a8f50 con 0x7f91c010e6f0 2024-05-27T07:33:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.537+0000 7f91bf400700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91c0105bd0 0x7f91c01034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.538+0000 7f91c4c00700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91c010e6f0 0x7f91c01039f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f91b8007ee0 tx=0x7f91b800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.538+0000 7f91bcc00700 1 -- 172.21.15.43:0/3896814873 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b800dbd0 con 0x7f91c010e6f0 2024-05-27T07:33:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.538+0000 7f91bcc00700 1 -- 172.21.15.43:0/3896814873 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91b800dd70 con 0x7f91c010e6f0 2024-05-27T07:33:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.538+0000 7f91bcc00700 1 -- 172.21.15.43:0/3896814873 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b80058d0 con 0x7f91c010e6f0 2024-05-27T07:33:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.538+0000 7f91c5600700 1 -- 172.21.15.43:0/3896814873 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c019a970 con 0x7f91c010e6f0 2024-05-27T07:33:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.539+0000 7f91c5600700 1 -- 172.21.15.43:0/3896814873 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c019ae60 con 0x7f91c010e6f0 2024-05-27T07:33:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.544+0000 7f91bcc00700 1 -- 172.21.15.43:0/3896814873 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f91b801a020 con 0x7f91c010e6f0 2024-05-27T07:33:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.546+0000 7f91d56a4b80 1 Processor -- start 2024-05-27T07:33:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.546+0000 7f91d56a4b80 1 -- start start 2024-05-27T07:33:37.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.547+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.547+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.547+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 0x562a8e9d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.547+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562a8e8f1ec0 con 0x562a8e844ec0 2024-05-27T07:33:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.547+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562a8e8e1a60 con 0x562a8e9c3490 2024-05-27T07:33:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.547+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562a8e8e5aa0 con 0x562a8e9c38a0 2024-05-27T07:33:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.547+0000 7f91bfe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.547+0000 7f91bfe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:36980/0 (socket says 172.21.15.43:36980) 2024-05-27T07:33:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.547+0000 7f91bfe00700 1 -- 172.21.15.43:0/424701814 learned_addr learned my addr 172.21.15.43:0/424701814 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.548+0000 7f91bcc00700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91a40b2bc0 0x7f91a40b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.548+0000 7f91bfe00700 1 -- 172.21.15.43:0/424701814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 msgr2=0x562a8e9d00a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:33:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.548+0000 7f91bfe00700 1 --2- 172.21.15.43:0/424701814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 0x562a8e9d00a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.548+0000 7f91bfe00700 1 -- 172.21.15.43:0/424701814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 msgr2=0x562a8e9c2f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:33:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.548+0000 7f91bcc00700 1 -- 172.21.15.43:0/3896814873 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7f91b80e2140 con 0x7f91c010e6f0 2024-05-27T07:33:37.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.549+0000 7f91c4c00700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91a40b2bc0 0x7f91a40b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.550+0000 7f91bf400700 1 --2- 172.21.15.43:0/424701814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.550+0000 7f91c4c00700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91a40b2bc0 0x7f91a40b50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f91b800dfd0 tx=0x7f91b801e040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:37.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.550+0000 7f91bfe00700 1 --2- 172.21.15.43:0/424701814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.550+0000 7f91bfe00700 1 -- 172.21.15.43:0/424701814 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a8e8aa4a0 con 0x562a8e9c3490 2024-05-27T07:33:37.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.550+0000 7f91bf400700 1 --2- 172.21.15.43:0/424701814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:37.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.551+0000 7f91bfe00700 1 --2- 172.21.15.43:0/424701814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cdb50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f91b000bd70 tx=0x7f91b000f640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f36178ed3b1248f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:37.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.551+0000 7f91b5600700 1 -- 172.21.15.43:0/424701814 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0015070 con 0x562a8e9c3490 2024-05-27T07:33:37.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.551+0000 7f91b5600700 1 -- 172.21.15.43:0/424701814 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91b000fe50 con 0x562a8e9c3490 2024-05-27T07:33:37.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.551+0000 7f91b5600700 1 -- 172.21.15.43:0/424701814 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0011730 con 0x562a8e9c3490 2024-05-27T07:33:37.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.552+0000 7f91d56a4b80 1 -- 172.21.15.43:0/424701814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 msgr2=0x562a8e9cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.552+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/424701814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cdb50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f91b000bd70 tx=0x7f91b000f640 comp rx=0 tx=0).stop 2024-05-27T07:33:37.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.553+0000 7f91d56a4b80 1 -- 172.21.15.43:0/424701814 shutdown_connections 2024-05-27T07:33:37.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.553+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/424701814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 0x562a8e9d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.553+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/424701814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.553+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/424701814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cdb50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.553+0000 7f91d56a4b80 1 -- 172.21.15.43:0/424701814 >> 172.21.15.43:0/424701814 conn(0x562a8e8a5040 msgr2=0x562a8e9bedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:37.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.553+0000 7f91d56a4b80 1 -- 172.21.15.43:0/424701814 shutdown_connections 2024-05-27T07:33:37.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.553+0000 7f91d56a4b80 1 -- 172.21.15.43:0/424701814 wait complete. 2024-05-27T07:33:37.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.554+0000 7f91d56a4b80 1 -- start start 2024-05-27T07:33:37.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.554+0000 7f91d56a4b80 1 Processor -- start 2024-05-27T07:33:37.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.555+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.555+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c3490 0x562a8e9cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.555+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c38a0 0x562a8e9ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.555+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562a8e7e8020 con 0x562a8e844ec0 2024-05-27T07:33:37.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.555+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x562a8e8827e0 con 0x562a8e9c38a0 2024-05-27T07:33:37.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.555+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562a8e9d0a10 con 0x562a8e9c3490 2024-05-27T07:33:37.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.555+0000 7f91d56a4b80 10 client.0 _ll_register_callbacks cb 0x562a8e88bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:37.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.557+0000 7f91bf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.557+0000 7f91bfe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c3490 0x562a8e9cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.557+0000 7f91bf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60174/0 (socket says 172.21.15.43:60174) 2024-05-27T07:33:37.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.557+0000 7f91bf400700 1 -- 172.21.15.43:0/1010719927 learned_addr learned my addr 172.21.15.43:0/1010719927 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:37.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.558+0000 7f91bf400700 1 -- 172.21.15.43:0/1010719927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c3490 msgr2=0x562a8e9cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.558+0000 7f91bf400700 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c3490 0x562a8e9cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.558+0000 7f91bf400700 1 -- 172.21.15.43:0/1010719927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c38a0 msgr2=0x562a8e9ce0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:33:37.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.558+0000 7f91bf400700 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c38a0 0x562a8e9ce0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.558+0000 7f91bf400700 1 -- 172.21.15.43:0/1010719927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a8e831950 con 0x562a8e844ec0 2024-05-27T07:33:37.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.558+0000 7f91bf400700 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9cd650 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f91a8015000 tx=0x7f91a80057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7920230e757125d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:37.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.558+0000 7f91ae000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:33:37.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.559+0000 7f91ae000700 1 -- 172.21.15.43:0/1010719927 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a802e060 con 0x562a8e844ec0 2024-05-27T07:33:37.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.559+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1010719927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91b000fe50 con 0x562a8e844ec0 2024-05-27T07:33:37.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.559+0000 7f91ae000700 1 -- 172.21.15.43:0/1010719927 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7f91a80037c0 con 0x562a8e844ec0 2024-05-27T07:33:37.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.560+0000 7f91ae000700 1 -- 172.21.15.43:0/1010719927 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f91a80039c0 con 0x562a8e844ec0 2024-05-27T07:33:37.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.560+0000 7f91d56a4b80 10 client.18090 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:37.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.560+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1010719927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x562a8e8b20e0 con 0x562a8e844ec0 2024-05-27T07:33:37.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.561+0000 7f91ae000700 1 -- 172.21.15.43:0/1010719927 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91b000fe50 con 0x562a8e844ec0 2024-05-27T07:33:37.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.561+0000 7f91ae000700 1 -- 172.21.15.43:0/1010719927 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f91a80344f0 con 0x562a8e844ec0 2024-05-27T07:33:37.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.561+0000 7f91d56a4b80 10 client.18090 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:37.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.561+0000 7f91d56a4b80 10 client.18090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:37.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.561+0000 7f91d56a4b80 20 client.18090 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:37.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.561+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562a8e9ddc60 0x562a8e9e00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.561+0000 7f91d56a4b80 4 client.18090 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:37.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.561+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1010719927 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a8e8f3e10 con 0x562a8e9ddc60 2024-05-27T07:33:37.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.562+0000 7f91bfe00700 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562a8e9ddc60 0x562a8e9e00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.563+0000 7f91ae000700 10 client.18090 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:37.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.563+0000 7f91bfe00700 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562a8e9ddc60 0x562a8e9e00b0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=208e88bfd92e2a8c server_cookie=8457c6eeac2ddfbc in_seq=0 out_seq=0 2024-05-27T07:33:37.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.564+0000 7f91ae000700 1 -- 172.21.15.43:0/1010719927 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x562a8e8f3e10 con 0x562a8e9ddc60 2024-05-27T07:33:37.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.564+0000 7f91ae000700 10 client.18090 handle_command_reply: tid=0 2024-05-27T07:33:37.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.564+0000 7f91d56a4b80 1 client.18090 shutdown 2024-05-27T07:33:37.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1010719927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 msgr2=0x562a8e9cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9cd650 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f91a8015000 tx=0x7f91a80057d0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1010719927 shutdown_connections 2024-05-27T07:33:37.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c38a0 0x562a8e9ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562a8e9ddc60 0x562a8e9e00b0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9cd650 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1010719927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c3490 0x562a8e9cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1010719927 >> 172.21.15.43:0/1010719927 conn(0x562a8e8a5040 msgr2=0x562a8e9be070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:37.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91ae000700 0 client.18090 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:37.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1010719927 shutdown_connections 2024-05-27T07:33:37.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.565+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1010719927 wait complete. 2024-05-27T07:33:37.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.566+0000 7f91d56a4b80 20 client.18090 trim_cache size 0 max 16384 2024-05-27T07:33:37.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.602+0000 7f91d56a4b80 1 Processor -- start 2024-05-27T07:33:37.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.602+0000 7f91d56a4b80 1 -- start start 2024-05-27T07:33:37.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.603+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.603+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.603+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 0x562a8e9cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.603+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01ae770 con 0x562a8e844ec0 2024-05-27T07:33:37.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.603+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01acbd0 con 0x562a8e9c3490 2024-05-27T07:33:37.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.603+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01ae0a0 con 0x562a8e9c38a0 2024-05-27T07:33:37.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.603+0000 7f91bf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.603+0000 7f91bf400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60176/0 (socket says 172.21.15.43:60176) 2024-05-27T07:33:37.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.603+0000 7f91bf400700 1 -- 172.21.15.43:0/669475576 learned_addr learned my addr 172.21.15.43:0/669475576 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:37.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.604+0000 7f91bf400700 1 -- 172.21.15.43:0/669475576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 msgr2=0x562a8e9cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.604+0000 7f91bf400700 1 --2- 172.21.15.43:0/669475576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 0x562a8e9cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.604+0000 7f91bf400700 1 -- 172.21.15.43:0/669475576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 msgr2=0x562a8e9cf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.604+0000 7f91bfe00700 1 --2- 172.21.15.43:0/669475576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.604+0000 7f91bf400700 1 --2- 172.21.15.43:0/669475576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.604+0000 7f91bf400700 1 -- 172.21.15.43:0/669475576 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a8e9c03d0 con 0x562a8e844ec0 2024-05-27T07:33:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.604+0000 7f91bf400700 1 --2- 172.21.15.43:0/669475576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c8e50 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f91a8034a30 tx=0x7f91a8034fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ea569bee114db2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.604+0000 7f91bfe00700 1 --2- 172.21.15.43:0/669475576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:33:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.605+0000 7f91b4c00700 1 -- 172.21.15.43:0/669475576 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a8003b60 con 0x562a8e844ec0 2024-05-27T07:33:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.605+0000 7f91b4c00700 1 -- 172.21.15.43:0/669475576 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91a8003d00 con 0x562a8e844ec0 2024-05-27T07:33:37.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.605+0000 7f91b4c00700 1 -- 172.21.15.43:0/669475576 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a800ecf0 con 0x562a8e844ec0 2024-05-27T07:33:37.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.606+0000 7f91d56a4b80 1 -- 172.21.15.43:0/669475576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 msgr2=0x562a8e9c8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.606+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/669475576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c8e50 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f91a8034a30 tx=0x7f91a8034fd0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.607+0000 7f91d56a4b80 1 -- 172.21.15.43:0/669475576 shutdown_connections 2024-05-27T07:33:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.607+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/669475576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 0x562a8e9cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.607+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/669475576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e844ec0 0x562a8e9c8e50 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.607+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/669475576 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e9c3490 0x562a8e9cf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.607+0000 7f91d56a4b80 1 -- 172.21.15.43:0/669475576 >> 172.21.15.43:0/669475576 conn(0x562a8e8a5040 msgr2=0x562a8e9cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.607+0000 7f91d56a4b80 1 -- 172.21.15.43:0/669475576 shutdown_connections 2024-05-27T07:33:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.607+0000 7f91d56a4b80 1 -- 172.21.15.43:0/669475576 wait complete. 2024-05-27T07:33:37.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.608+0000 7f91d56a4b80 1 -- start start 2024-05-27T07:33:37.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.608+0000 7f91d56a4b80 1 Processor -- start 2024-05-27T07:33:37.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.609+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e844ec0 0x562a8e9d6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.609+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e9c3490 0x562a8e9d18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.609+0000 7f91d56a4b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 0x562a8e9d1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.609+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01ad9d0 con 0x562a8e9c3490 2024-05-27T07:33:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.609+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f91c01ad2a0 con 0x562a8e844ec0 2024-05-27T07:33:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.609+0000 7f91d56a4b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562a8e9cfef0 con 0x562a8e9c38a0 2024-05-27T07:33:37.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.609+0000 7f91d56a4b80 10 client.0 _ll_register_callbacks cb 0x562a8e88bcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.609+0000 7f91bf400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e844ec0 0x562a8e9d6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.610+0000 7f91bfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e9c3490 0x562a8e9d18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.610+0000 7f91bfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e9c3490 0x562a8e9d18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60178/0 (socket says 172.21.15.43:60178) 2024-05-27T07:33:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.610+0000 7f91bfe00700 1 -- 172.21.15.43:0/1560599882 learned_addr learned my addr 172.21.15.43:0/1560599882 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:37.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.610+0000 7f91bf400700 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 0x562a8e9d1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.610+0000 7f91bfe00700 1 -- 172.21.15.43:0/1560599882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 msgr2=0x562a8e9d1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.610+0000 7f91bfe00700 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 0x562a8e9d1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.610+0000 7f91bfe00700 1 -- 172.21.15.43:0/1560599882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e844ec0 msgr2=0x562a8e9d6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.610+0000 7f91bfe00700 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e844ec0 0x562a8e9d6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.610+0000 7f91bfe00700 1 -- 172.21.15.43:0/1560599882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a8e9d26e0 con 0x562a8e9c3490 2024-05-27T07:33:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.611+0000 7f91bfe00700 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e9c3490 0x562a8e9d18c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x562a8e9c00d0 tx=0x7f91b000b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1972afa18b06c4f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.611+0000 7f91ae000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:33:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.611+0000 7f91ae000700 1 -- 172.21.15.43:0/1560599882 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b002b070 con 0x562a8e9c3490 2024-05-27T07:33:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.611+0000 7f91ae000700 1 -- 172.21.15.43:0/1560599882 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7f91b0004240 con 0x562a8e9c3490 2024-05-27T07:33:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.611+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1560599882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91a8003d00 con 0x562a8e9c3490 2024-05-27T07:33:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.612+0000 7f91ae000700 1 -- 172.21.15.43:0/1560599882 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f91b0015040 con 0x562a8e9c3490 2024-05-27T07:33:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.612+0000 7f91d56a4b80 10 client.18096 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.612+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1560599882 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x562a8ea487c0 con 0x562a8e9c3490 2024-05-27T07:33:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.613+0000 7f91ae000700 1 -- 172.21.15.43:0/1560599882 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91a8003d00 con 0x562a8e9c3490 2024-05-27T07:33:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.613+0000 7f91ae000700 1 -- 172.21.15.43:0/1560599882 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f91b001a460 con 0x562a8e9c3490 2024-05-27T07:33:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.613+0000 7f91d56a4b80 10 client.18096 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.613+0000 7f91d56a4b80 10 client.18096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.613+0000 7f91d56a4b80 20 client.18096 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.613+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562a8ea48dd0 0x562a8ea4b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.613+0000 7f91d56a4b80 4 client.18096 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:33:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.613+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1560599882 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562a8e8ed5f0 con 0x562a8ea48dd0 2024-05-27T07:33:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.613+0000 7f91bf400700 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562a8ea48dd0 0x562a8ea4b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:37.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.614+0000 7f91bf400700 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562a8ea48dd0 0x562a8ea4b1f0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21d06094e7a88e68 server_cookie=569e0de86acf5e27 in_seq=0 out_seq=0 2024-05-27T07:33:37.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.614+0000 7f91ae000700 10 client.18096 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:37.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.616+0000 7f91ae000700 1 -- 172.21.15.43:0/1560599882 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562a8e8ed5f0 con 0x562a8ea48dd0 2024-05-27T07:33:37.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.616+0000 7f91ae000700 10 client.18096 handle_command_reply: tid=0 2024-05-27T07:33:37.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.616+0000 7f91d56a4b80 1 client.18096 shutdown 2024-05-27T07:33:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1560599882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e9c3490 msgr2=0x562a8e9d18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e9c3490 0x562a8e9d18c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x562a8e9c00d0 tx=0x7f91b000b620 comp rx=0 tx=0).stop 2024-05-27T07:33:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1560599882 shutdown_connections 2024-05-27T07:33:37.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562a8e9c38a0 0x562a8e9d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562a8e844ec0 0x562a8e9d6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x562a8ea48dd0 0x562a8ea4b1f0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91d56a4b80 1 --2- 172.21.15.43:0/1560599882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562a8e9c3490 0x562a8e9d18c0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1560599882 >> 172.21.15.43:0/1560599882 conn(0x562a8e8a5040 msgr2=0x562a8e9d8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91ae000700 0 client.18096 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1560599882 shutdown_connections 2024-05-27T07:33:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.617+0000 7f91d56a4b80 1 -- 172.21.15.43:0/1560599882 wait complete. 2024-05-27T07:33:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.618+0000 7f91d56a4b80 20 client.18096 trim_cache size 0 max 16384 2024-05-27T07:33:37.614 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:33:37.614 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:33:37.614 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "183420b6-df04-4a75-bef0-23f1cdb11ee3", 2024-05-27T07:33:37.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:33:37.614 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:33:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 -- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91a40b2bc0 msgr2=0x7f91a40b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91a40b2bc0 0x7f91a40b50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f91b800dfd0 tx=0x7f91b801e040 comp rx=0 tx=0).stop 2024-05-27T07:33:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 -- 172.21.15.43:0/3896814873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91c010e6f0 msgr2=0x7f91c01039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91c010e6f0 0x7f91c01039f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f91b8007ee0 tx=0x7f91b800d3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 -- 172.21.15.43:0/3896814873 shutdown_connections 2024-05-27T07:33:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f91a40b2bc0 0x7f91a40b50d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91c010e6f0 0x7f91c01039f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91c0105280 0x7f91c0102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 --2- 172.21.15.43:0/3896814873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f91c0105bd0 0x7f91c01034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 -- 172.21.15.43:0/3896814873 >> 172.21.15.43:0/3896814873 conn(0x7f91c0078ab0 msgr2=0x7f91c0101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:37.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.619+0000 7f91c5600700 1 -- 172.21.15.43:0/3896814873 shutdown_connections 2024-05-27T07:33:37.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:37.620+0000 7f91c5600700 1 -- 172.21.15.43:0/3896814873 wait complete. 2024-05-27T07:33:37.798 INFO:tasks.workunit.client.0.smithi043.stdout: 10 37380 5.85 MB/sec execute 218 sec latency 168.080 ms 2024-05-27T07:33:38.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:33:38.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: pgmap v662: 129 pgs: 129 active+clean; 880 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 5.5 MiB/s wr, 220 op/s 2024-05-27T07:33:38.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:33:38.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:33:38.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:33:38.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:33:38.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:33:38.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:33:38.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.711 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:33:38.711 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.711 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.711 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:33:38.711 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.711 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.712 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:38 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.798 INFO:tasks.workunit.client.0.smithi043.stdout: 10 37515 5.88 MB/sec execute 219 sec latency 167.311 ms 2024-05-27T07:33:38.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: pgmap v662: 129 pgs: 129 active+clean; 880 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 5.5 MiB/s wr, 220 op/s 2024-05-27T07:33:38.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:33:38.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:33:38.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:33:38.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:33:38.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:33:38.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:33:38.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:33:38.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:33:38.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:38 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: pgmap v662: 129 pgs: 129 active+clean; 880 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 5.5 MiB/s wr, 220 op/s 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:33:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:38.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:38 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:39.037 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:33:39.798 INFO:tasks.workunit.client.0.smithi043.stdout: 10 37634 5.87 MB/sec execute 220 sec latency 98.248 ms 2024-05-27T07:33:39.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:39 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:39.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:39 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:39.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:39 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:33:40.798 INFO:tasks.workunit.client.0.smithi043.stdout: 10 37784 5.90 MB/sec execute 221 sec latency 106.397 ms 2024-05-27T07:33:40.813 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:40 smithi043 ceph-mon[31531]: pgmap v663: 129 pgs: 129 active+clean; 880 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 3.6 MiB/s wr, 141 op/s 2024-05-27T07:33:40.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:40 smithi132 ceph-mon[35283]: pgmap v663: 129 pgs: 129 active+clean; 880 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 3.6 MiB/s wr, 141 op/s 2024-05-27T07:33:40.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:40 smithi179 ceph-mon[35888]: pgmap v663: 129 pgs: 129 active+clean; 880 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 3.6 MiB/s wr, 141 op/s 2024-05-27T07:33:41.799 INFO:tasks.workunit.client.0.smithi043.stdout: 10 37895 5.91 MB/sec execute 222 sec latency 80.281 ms 2024-05-27T07:33:42.798 INFO:tasks.workunit.client.0.smithi043.stdout: 10 37958 5.89 MB/sec execute 223 sec latency 391.796 ms 2024-05-27T07:33:42.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:42 smithi132 ceph-mon[35283]: pgmap v664: 129 pgs: 129 active+clean; 885 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 5.5 MiB/s wr, 229 op/s 2024-05-27T07:33:42.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:42 smithi043 ceph-mon[31531]: pgmap v664: 129 pgs: 129 active+clean; 885 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 5.5 MiB/s wr, 229 op/s 2024-05-27T07:33:42.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:42 smithi179 ceph-mon[35888]: pgmap v664: 129 pgs: 129 active+clean; 885 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 5.5 MiB/s wr, 229 op/s 2024-05-27T07:33:43.798 INFO:tasks.workunit.client.0.smithi043.stdout: 10 38051 5.87 MB/sec execute 224 sec latency 71.224 ms 2024-05-27T07:33:44.799 INFO:tasks.workunit.client.0.smithi043.stdout: 10 38133 5.89 MB/sec execute 225 sec latency 387.822 ms 2024-05-27T07:33:44.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:44 smithi132 ceph-mon[35283]: pgmap v665: 129 pgs: 129 active+clean; 885 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.9 MiB/s wr, 157 op/s 2024-05-27T07:33:44.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:44 smithi043 ceph-mon[31531]: pgmap v665: 129 pgs: 129 active+clean; 885 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.9 MiB/s wr, 157 op/s 2024-05-27T07:33:44.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:44 smithi179 ceph-mon[35888]: pgmap v665: 129 pgs: 129 active+clean; 885 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.9 MiB/s wr, 157 op/s 2024-05-27T07:33:45.798 INFO:tasks.workunit.client.0.smithi043.stdout: 10 38241 5.89 MB/sec execute 226 sec latency 472.609 ms 2024-05-27T07:33:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.032+0000 7fbe0b000700 1 -- 172.21.15.43:0/1507963447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe04106eb0 msgr2=0x7fbe04113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.032+0000 7fbe0b000700 1 --2- 172.21.15.43:0/1507963447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe04106eb0 0x7fbe04113340 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00b0a0 tx=0x7fbdfc00b3b0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.033+0000 7fbe0b000700 1 -- 172.21.15.43:0/1507963447 shutdown_connections 2024-05-27T07:33:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.033+0000 7fbe0b000700 1 --2- 172.21.15.43:0/1507963447 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe04106eb0 0x7fbe04113340 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.033+0000 7fbe0b000700 1 --2- 172.21.15.43:0/1507963447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe0410c5a0 0x7fbe0410c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.033+0000 7fbe0b000700 1 --2- 172.21.15.43:0/1507963447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe04106560 0x7fbe04106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.033+0000 7fbe0b000700 1 -- 172.21.15.43:0/1507963447 >> 172.21.15.43:0/1507963447 conn(0x7fbe04102040 msgr2=0x7fbe04104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.033+0000 7fbe0b000700 1 -- 172.21.15.43:0/1507963447 shutdown_connections 2024-05-27T07:33:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.033+0000 7fbe0b000700 1 -- 172.21.15.43:0/1507963447 wait complete. 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.034+0000 7fbe0b000700 1 Processor -- start 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.034+0000 7fbe0b000700 1 -- start start 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.034+0000 7fbe0b000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe04106560 0x7fbe0419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.034+0000 7fbe0b000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe04106eb0 0x7fbe0419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.034+0000 7fbe0b000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe0410c5a0 0x7fbe041a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.034+0000 7fbe0b000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbe04116580 con 0x7fbe04106560 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.034+0000 7fbe0b000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbe04116400 con 0x7fbe0410c5a0 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.034+0000 7fbe0b000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbe04116700 con 0x7fbe04106eb0 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.035+0000 7fbe03e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe04106560 0x7fbe0419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.035+0000 7fbe03e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe04106560 0x7fbe0419ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60200/0 (socket says 172.21.15.43:60200) 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.035+0000 7fbe03e00700 1 -- 172.21.15.43:0/2837890633 learned_addr learned my addr 172.21.15.43:0/2837890633 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.035+0000 7fbe09000700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe0410c5a0 0x7fbe041a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.035+0000 7fbe03400700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe04106eb0 0x7fbe0419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.035+0000 7fbe03e00700 1 -- 172.21.15.43:0/2837890633 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe04106eb0 msgr2=0x7fbe0419d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.035+0000 7fbe03e00700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe04106eb0 0x7fbe0419d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.035+0000 7fbe03e00700 1 -- 172.21.15.43:0/2837890633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe0410c5a0 msgr2=0x7fbe041a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.035+0000 7fbe03e00700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe0410c5a0 0x7fbe041a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.035+0000 7fbe03e00700 1 -- 172.21.15.43:0/2837890633 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe041a1c90 con 0x7fbe04106560 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.036+0000 7fbe03e00700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe04106560 0x7fbe0419ce70 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400c9e0 tx=0x7fbdf400cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.036+0000 7fbe00c00700 1 -- 172.21.15.43:0/2837890633 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4007aa0 con 0x7fbe04106560 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.036+0000 7fbe00c00700 1 -- 172.21.15.43:0/2837890633 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdf4007c40 con 0x7fbe04106560 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.036+0000 7fbe00c00700 1 -- 172.21.15.43:0/2837890633 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4018660 con 0x7fbe04106560 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.036+0000 7fbe0b000700 1 -- 172.21.15.43:0/2837890633 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe041a1f80 con 0x7fbe04106560 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.036+0000 7fbe0b000700 1 -- 172.21.15.43:0/2837890633 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe041a9950 con 0x7fbe04106560 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.040+0000 7fbe00c00700 1 -- 172.21.15.43:0/2837890633 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fbdf4018800 con 0x7fbe04106560 2024-05-27T07:33:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.040+0000 7fbe19adbb80 1 Processor -- start 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe19adbb80 1 -- start start 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d7d9e40 0x55800d95e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d95ed30 0x55800d958870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d9591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55800d886e40 con 0x55800d95ed30 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55800d8769e0 con 0x55800d7d9e40 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55800d87aa20 con 0x55800d958db0 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe00c00700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbde80b2a70 0x7fbde80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe03e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d95ed30 0x55800d958870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe03e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d95ed30 0x55800d958870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60202/0 (socket says 172.21.15.43:60202) 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.041+0000 7fbe03e00700 1 -- 172.21.15.43:0/1756817656 learned_addr learned my addr 172.21.15.43:0/1756817656 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.042+0000 7fbe00c00700 1 -- 172.21.15.43:0/2837890633 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7fbdf40e2960 con 0x7fbe04106560 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.042+0000 7fbe09000700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbde80b2a70 0x7fbde80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.042+0000 7fbe03400700 1 --2- 172.21.15.43:0/1756817656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d9591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.042+0000 7fbe09000700 1 --2- 172.21.15.43:0/1756817656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d7d9e40 0x55800d95e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.042+0000 7fbe03e00700 1 -- 172.21.15.43:0/1756817656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 msgr2=0x55800d9591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.042+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1756817656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d9591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.042+0000 7fbe03e00700 1 -- 172.21.15.43:0/1756817656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d7d9e40 msgr2=0x55800d95e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.042+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1756817656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d7d9e40 0x55800d95e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.042+0000 7fbe03e00700 1 -- 172.21.15.43:0/1756817656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55800d83f420 con 0x55800d95ed30 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.042+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1756817656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d95ed30 0x55800d958870 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400c9b0 tx=0x7fbdf40a8f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b23ee745b90b8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.043+0000 7fbdf9600700 1 -- 172.21.15.43:0/1756817656 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf40e0110 con 0x55800d95ed30 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.043+0000 7fbe09000700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbde80b2a70 0x7fbde80b4f80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55800d959b30 tx=0x7fbdfc0060a0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.043+0000 7fbdf9600700 1 -- 172.21.15.43:0/1756817656 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdf40a9320 con 0x55800d95ed30 2024-05-27T07:33:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.043+0000 7fbdf9600700 1 -- 172.21.15.43:0/1756817656 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf40a9cb0 con 0x55800d95ed30 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.043+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1756817656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d95ed30 msgr2=0x55800d958870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.043+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1756817656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d95ed30 0x55800d958870 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400c9b0 tx=0x7fbdf40a8f80 comp rx=0 tx=0).stop 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.044+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1756817656 shutdown_connections 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.044+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1756817656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d9591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.044+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1756817656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d7d9e40 0x55800d95e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.044+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1756817656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d95ed30 0x55800d958870 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.044+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1756817656 >> 172.21.15.43:0/1756817656 conn(0x55800d839fc0 msgr2=0x55800d9538d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.044+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1756817656 shutdown_connections 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.044+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1756817656 wait complete. 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.044+0000 7fbe19adbb80 1 -- start start 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe19adbb80 1 Processor -- start 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d7d9e40 0x55800d9624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d958db0 0x55800d9629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d95ed30 0x55800d962f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55800d77cfa0 con 0x55800d958db0 2024-05-27T07:33:46.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55800d817760 con 0x55800d7d9e40 2024-05-27T07:33:46.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55800d965800 con 0x55800d95ed30 2024-05-27T07:33:46.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe19adbb80 10 client.0 _ll_register_callbacks cb 0x55800d820c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:46.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe03400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d958db0 0x55800d9629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe03400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d958db0 0x55800d9629f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60210/0 (socket says 172.21.15.43:60210) 2024-05-27T07:33:46.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.045+0000 7fbe03400700 1 -- 172.21.15.43:0/1574077780 learned_addr learned my addr 172.21.15.43:0/1574077780 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:46.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.046+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d7d9e40 0x55800d9624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.046+0000 7fbe09000700 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d95ed30 0x55800d962f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.046+0000 7fbe03400700 1 -- 172.21.15.43:0/1574077780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d95ed30 msgr2=0x55800d962f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.046+0000 7fbe03400700 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d95ed30 0x55800d962f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.046+0000 7fbe03400700 1 -- 172.21.15.43:0/1574077780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d7d9e40 msgr2=0x55800d9624b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.046+0000 7fbe03400700 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d7d9e40 0x55800d9624b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.046+0000 7fbe03400700 1 -- 172.21.15.43:0/1574077780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55800d7c68d0 con 0x55800d958db0 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.046+0000 7fbe03400700 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d958db0 0x55800d9629f0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0010790 tx=0x7fbdf0010b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79a7c9c806e924 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.046+0000 7fbdee000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.046+0000 7fbdee000700 1 -- 172.21.15.43:0/1574077780 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf000e9e0 con 0x55800d958db0 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbdee000700 1 -- 172.21.15.43:0/1574077780 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7fbdf000eb80 con 0x55800d958db0 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1574077780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdf40a9320 con 0x55800d958db0 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbdee000700 1 -- 172.21.15.43:0/1574077780 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7fbdf000ed80 con 0x55800d958db0 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbe19adbb80 10 client.18108 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:46.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1574077780 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x55800d847060 con 0x55800d958db0 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbdee000700 1 -- 172.21.15.43:0/1574077780 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdf40a9320 con 0x55800d958db0 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbdee000700 1 -- 172.21.15.43:0/1574077780 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7fbdf00114d0 con 0x55800d958db0 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbe19adbb80 10 client.18108 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbe19adbb80 10 client.18108 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbe19adbb80 20 client.18108 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55800d972a80 0x55800d974f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbe19adbb80 4 client.18108 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.047+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1574077780 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55800d888d90 con 0x55800d972a80 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.048+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55800d972a80 0x55800d974f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.049+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55800d972a80 0x55800d974f30 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b3826b47a0be806 server_cookie=c167af1ff699147a in_seq=0 out_seq=0 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.049+0000 7fbdee000700 10 client.18108 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbdee000700 1 -- 172.21.15.43:0/1574077780 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55800d888d90 con 0x55800d972a80 2024-05-27T07:33:46.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbdee000700 10 client.18108 handle_command_reply: tid=0 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbe19adbb80 1 client.18108 shutdown 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1574077780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d958db0 msgr2=0x55800d9629f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d958db0 0x55800d9629f0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0010790 tx=0x7fbdf0010b50 comp rx=0 tx=0).stop 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1574077780 shutdown_connections 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d95ed30 0x55800d962f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d7d9e40 0x55800d9624b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55800d972a80 0x55800d974f30 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1574077780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d958db0 0x55800d9629f0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbdee000700 0 client.18108 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1574077780 >> 172.21.15.43:0/1574077780 conn(0x55800d839fc0 msgr2=0x55800d952b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.051+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1574077780 shutdown_connections 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.052+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1574077780 wait complete. 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.052+0000 7fbe19adbb80 20 client.18108 trim_cache size 0 max 16384 2024-05-27T07:33:46.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.074+0000 7fbe19adbb80 1 Processor -- start 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.074+0000 7fbe19adbb80 1 -- start start 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d958870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d96e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d95ed30 0x55800d962b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbe041b2ba0 con 0x55800d7d9e40 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbe041b1000 con 0x55800d95ed30 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbe041b24d0 con 0x55800d958db0 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe03e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d958870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe03e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d958870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60226/0 (socket says 172.21.15.43:60226) 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe03e00700 1 -- 172.21.15.43:0/1777483768 learned_addr learned my addr 172.21.15.43:0/1777483768 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe03e00700 1 -- 172.21.15.43:0/1777483768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 msgr2=0x55800d96e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe03400700 1 --2- 172.21.15.43:0/1777483768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d96e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1777483768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d96e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe03e00700 1 -- 172.21.15.43:0/1777483768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d95ed30 msgr2=0x55800d962b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1777483768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d95ed30 0x55800d962b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.075+0000 7fbe03e00700 1 -- 172.21.15.43:0/1777483768 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55800d960440 con 0x55800d7d9e40 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.076+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1777483768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d958870 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40a9510 tx=0x7fbdf40cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87f1df021142795 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.076+0000 7fbdf8c00700 1 -- 172.21.15.43:0/1777483768 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf40a3070 con 0x55800d7d9e40 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.076+0000 7fbdf8c00700 1 -- 172.21.15.43:0/1777483768 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdf40a9320 con 0x55800d7d9e40 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.076+0000 7fbdf8c00700 1 -- 172.21.15.43:0/1777483768 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf40adbc0 con 0x55800d7d9e40 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.077+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1777483768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 msgr2=0x55800d958870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.077+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1777483768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d958870 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40a9510 tx=0x7fbdf40cf040 comp rx=0 tx=0).stop 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.077+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1777483768 shutdown_connections 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.077+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1777483768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d95ed30 0x55800d962b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.077+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1777483768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d958870 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.077+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1777483768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d96e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.077+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1777483768 >> 172.21.15.43:0/1777483768 conn(0x55800d839fc0 msgr2=0x55800d962470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.077+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1777483768 shutdown_connections 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.077+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1777483768 wait complete. 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.077+0000 7fbe19adbb80 1 -- start start 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe19adbb80 1 Processor -- start 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d96aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d979d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe19adbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d95ed30 0x55800d97a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbe041b1e00 con 0x55800d7d9e40 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbe041b16d0 con 0x55800d95ed30 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe19adbb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55800d962290 con 0x55800d958db0 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe19adbb80 10 client.0 _ll_register_callbacks cb 0x55800d820c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe03e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d96aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe03e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d96aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60240/0 (socket says 172.21.15.43:60240) 2024-05-27T07:33:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.078+0000 7fbe03e00700 1 -- 172.21.15.43:0/1495419638 learned_addr learned my addr 172.21.15.43:0/1495419638 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbe09000700 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d95ed30 0x55800d97a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbe03400700 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d979d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbe03e00700 1 -- 172.21.15.43:0/1495419638 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 msgr2=0x55800d979d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d979d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbe03e00700 1 -- 172.21.15.43:0/1495419638 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d95ed30 msgr2=0x55800d97a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d95ed30 0x55800d97a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbe03e00700 1 -- 172.21.15.43:0/1495419638 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55800d97a7a0 con 0x55800d7d9e40 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbe03e00700 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d96aeb0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40a97a0 tx=0x7fbdf409ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec1a7867cb5a82c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbdee000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbdee000700 1 -- 172.21.15.43:0/1495419638 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf40c2070 con 0x55800d7d9e40 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbdee000700 1 -- 172.21.15.43:0/1495419638 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(149..149 src has 1..149) ==== 9496+0+0 (secure 0 0 0) 0x7fbdf40a9a70 con 0x55800d7d9e40 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.079+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1495419638 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdf40a9320 con 0x55800d7d9e40 2024-05-27T07:33:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.080+0000 7fbdee000700 1 -- 172.21.15.43:0/1495419638 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7fbdf40a3050 con 0x55800d7d9e40 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.080+0000 7fbe19adbb80 10 client.18114 fetch_fsmap learned FSMap version 139 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.080+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1495419638 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x55800d9dd1f0 con 0x55800d7d9e40 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.080+0000 7fbdee000700 1 -- 172.21.15.43:0/1495419638 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbdf40a9320 con 0x55800d7d9e40 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.080+0000 7fbdee000700 1 -- 172.21.15.43:0/1495419638 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7fbdf4020030 con 0x55800d7d9e40 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.080+0000 7fbe19adbb80 10 client.18114 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.080+0000 7fbe19adbb80 10 client.18114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.080+0000 7fbe19adbb80 20 client.18114 populate_metadata read hostname 'smithi043' 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.080+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55800d9dd420 0x55800d9df840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.081+0000 7fbe19adbb80 4 client.18114 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.081+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1495419638 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55800d882570 con 0x55800d9dd420 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.081+0000 7fbe03400700 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55800d9dd420 0x55800d9df840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.082+0000 7fbe03400700 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55800d9dd420 0x55800d9df840 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=572b3821d038ffd7 server_cookie=8fcac6afd44bddea in_seq=0 out_seq=0 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.082+0000 7fbdee000700 10 client.18114 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbdee000700 1 -- 172.21.15.43:0/1495419638 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55800d882570 con 0x55800d9dd420 2024-05-27T07:33:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbdee000700 10 client.18114 handle_command_reply: tid=0 2024-05-27T07:33:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 client.18114 shutdown 2024-05-27T07:33:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1495419638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 msgr2=0x55800d96aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d96aeb0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40a97a0 tx=0x7fbdf409ee20 comp rx=0 tx=0).stop 2024-05-27T07:33:46.708 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:33:46.708 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:33:46.708 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:33:46.708 INFO:teuthology.orchestra.run.smithi043.stdout: "183420b6-df04-4a75-bef0-23f1cdb11ee3": { 2024-05-27T07:33:46.708 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "183420b6-df04-4a75-bef0-23f1cdb11ee3", 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1495419638 shutdown_connections 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55800d95ed30 0x55800d97a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55800d9dd420 0x55800d9df840 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55800d7d9e40 0x55800d96aeb0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 --2- 172.21.15.43:0/1495419638 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55800d958db0 0x55800d979d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbdee000700 0 client.18114 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1495419638 >> 172.21.15.43:0/1495419638 conn(0x55800d839fc0 msgr2=0x55800d962470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1495419638 shutdown_connections 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 1 -- 172.21.15.43:0/1495419638 wait complete. 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.083+0000 7fbe19adbb80 20 client.18114 trim_cache size 0 max 16384 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 -- 172.21.15.43:0/2837890633 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbde80b2a70 msgr2=0x7fbde80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbde80b2a70 0x7fbde80b4f80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55800d959b30 tx=0x7fbdfc0060a0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 -- 172.21.15.43:0/2837890633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe04106560 msgr2=0x7fbe0419ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe04106560 0x7fbe0419ce70 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400c9e0 tx=0x7fbdf400cda0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 -- 172.21.15.43:0/2837890633 shutdown_connections 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fbde80b2a70 0x7fbde80b4f80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbe0410c5a0 0x7fbe041a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbe04106560 0x7fbe0419ce70 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 --2- 172.21.15.43:0/2837890633 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe04106eb0 0x7fbe0419d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:33:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 -- 172.21.15.43:0/2837890633 >> 172.21.15.43:0/2837890633 conn(0x7fbe04102040 msgr2=0x7fbe041037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:33:46.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 -- 172.21.15.43:0/2837890633 shutdown_connections 2024-05-27T07:33:46.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:33:42.084+0000 7fbe0b000700 1 -- 172.21.15.43:0/2837890633 wait complete. 2024-05-27T07:33:46.798 INFO:tasks.workunit.client.0.smithi043.stdout: 10 38317 5.87 MB/sec execute 227 sec latency 314.454 ms 2024-05-27T07:33:46.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:46 smithi132 ceph-mon[35283]: pgmap v666: 129 pgs: 129 active+clean; 885 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.9 MiB/s wr, 157 op/s 2024-05-27T07:33:46.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:46 smithi043 ceph-mon[31531]: pgmap v666: 129 pgs: 129 active+clean; 885 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.9 MiB/s wr, 157 op/s 2024-05-27T07:33:46.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:46 smithi179 ceph-mon[35888]: pgmap v666: 129 pgs: 129 active+clean; 885 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 3.9 MiB/s wr, 157 op/s 2024-05-27T07:33:47.799 INFO:tasks.workunit.client.0.smithi043.stdout: 10 38402 5.85 MB/sec execute 228 sec latency 302.333 ms 2024-05-27T07:33:48.799 INFO:tasks.workunit.client.0.smithi043.stdout: 10 38470 5.84 MB/sec execute 229 sec latency 438.468 ms 2024-05-27T07:33:48.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:48 smithi132 ceph-mon[35283]: pgmap v667: 129 pgs: 129 active+clean; 891 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.4 MiB/s wr, 226 op/s 2024-05-27T07:33:48.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:48 smithi043 ceph-mon[31531]: pgmap v667: 129 pgs: 129 active+clean; 891 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.4 MiB/s wr, 226 op/s 2024-05-27T07:33:48.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:48 smithi179 ceph-mon[35888]: pgmap v667: 129 pgs: 129 active+clean; 891 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.4 MiB/s wr, 226 op/s 2024-05-27T07:33:49.480 INFO:tasks.cephfs.filesystem:scrub status for tag:183420b6-df04-4a75-bef0-23f1cdb11ee3 - {'path': '/', 'tag': '183420b6-df04-4a75-bef0-23f1cdb11ee3', 'options': 'recursive,force'} 2024-05-27T07:33:49.799 INFO:tasks.workunit.client.0.smithi043.stdout: 10 38591 5.82 MB/sec execute 230 sec latency 83.718 ms 2024-05-27T07:33:50.799 INFO:tasks.workunit.client.0.smithi043.stdout: 10 38699 5.81 MB/sec execute 231 sec latency 70.017 ms 2024-05-27T07:33:50.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:50 smithi132 ceph-mon[35283]: pgmap v668: 129 pgs: 129 active+clean; 891 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 157 op/s 2024-05-27T07:33:50.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:50 smithi043 ceph-mon[31531]: pgmap v668: 129 pgs: 129 active+clean; 891 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 157 op/s 2024-05-27T07:33:50.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:50 smithi179 ceph-mon[35888]: pgmap v668: 129 pgs: 129 active+clean; 891 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 157 op/s 2024-05-27T07:33:51.801 INFO:tasks.workunit.client.0.smithi043.stdout: 10 38816 5.80 MB/sec execute 232 sec latency 82.963 ms 2024-05-27T07:33:52.801 INFO:tasks.workunit.client.0.smithi043.stdout: 10 38902 5.80 MB/sec execute 233 sec latency 272.858 ms 2024-05-27T07:33:52.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:52 smithi132 ceph-mon[35283]: pgmap v669: 129 pgs: 129 active+clean; 896 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-05-27T07:33:52.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:52 smithi043 ceph-mon[31531]: pgmap v669: 129 pgs: 129 active+clean; 896 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-05-27T07:33:52.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:52 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:33:52.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:52 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:33:52.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:52 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:33:52.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:52 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:33:52.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:52 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:33:52.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:52 smithi179 ceph-mon[35888]: pgmap v669: 129 pgs: 129 active+clean; 896 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-05-27T07:33:52.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:52 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:33:52.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:52 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:33:52.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:52 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:33:52.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:52 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:33:52.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:52 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:33:53.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:52 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:33:53.344 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:52 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:33:53.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:52 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:33:53.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:52 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:33:53.345 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:52 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:33:53.803 INFO:tasks.workunit.client.0.smithi043.stdout: 10 39038 5.82 MB/sec execute 234 sec latency 448.698 ms 2024-05-27T07:33:54.802 INFO:tasks.workunit.client.0.smithi043.stdout: 10 39174 5.82 MB/sec execute 235 sec latency 94.066 ms 2024-05-27T07:33:54.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:54 smithi132 ceph-mon[35283]: pgmap v670: 129 pgs: 129 active+clean; 896 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-05-27T07:33:54.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:54 smithi043 ceph-mon[31531]: pgmap v670: 129 pgs: 129 active+clean; 896 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-05-27T07:33:54.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:54 smithi179 ceph-mon[35888]: pgmap v670: 129 pgs: 129 active+clean; 896 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-05-27T07:33:55.802 INFO:tasks.workunit.client.0.smithi043.stdout: 10 39317 5.84 MB/sec execute 236 sec latency 70.265 ms 2024-05-27T07:33:56.802 INFO:tasks.workunit.client.0.smithi043.stdout: 10 39422 5.84 MB/sec execute 237 sec latency 104.648 ms 2024-05-27T07:33:56.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:56 smithi043 ceph-mon[31531]: pgmap v671: 129 pgs: 129 active+clean; 896 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-05-27T07:33:56.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:56 smithi179 ceph-mon[35888]: pgmap v671: 129 pgs: 129 active+clean; 896 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-05-27T07:33:57.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:56 smithi132 ceph-mon[35283]: pgmap v671: 129 pgs: 129 active+clean; 896 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-05-27T07:33:57.802 INFO:tasks.workunit.client.0.smithi043.stdout: 10 39529 5.84 MB/sec execute 238 sec latency 78.377 ms 2024-05-27T07:33:58.802 INFO:tasks.workunit.client.0.smithi043.stdout: 10 39678 5.86 MB/sec execute 239 sec latency 79.800 ms 2024-05-27T07:33:58.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:58 smithi043 ceph-mon[31531]: pgmap v672: 129 pgs: 129 active+clean; 904 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.4 MiB/s wr, 219 op/s 2024-05-27T07:33:58.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:58 smithi179 ceph-mon[35888]: pgmap v672: 129 pgs: 129 active+clean; 904 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.4 MiB/s wr, 219 op/s 2024-05-27T07:33:59.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:58 smithi132 ceph-mon[35283]: pgmap v672: 129 pgs: 129 active+clean; 904 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.4 MiB/s wr, 219 op/s 2024-05-27T07:33:59.805 INFO:tasks.workunit.client.0.smithi043.stdout: 10 39817 5.86 MB/sec execute 240 sec latency 87.684 ms 2024-05-27T07:33:59.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:33:59 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:33:59.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:33:59 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:00.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:33:59 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:34:00.803 INFO:tasks.workunit.client.0.smithi043.stdout: 10 39934 5.88 MB/sec execute 241 sec latency 133.354 ms 2024-05-27T07:34:00.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:00 smithi043 ceph-mon[31531]: pgmap v673: 129 pgs: 129 active+clean; 904 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-05-27T07:34:00.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:00 smithi179 ceph-mon[35888]: pgmap v673: 129 pgs: 129 active+clean; 904 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-05-27T07:34:01.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:00 smithi132 ceph-mon[35283]: pgmap v673: 129 pgs: 129 active+clean; 904 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.9 MiB/s wr, 149 op/s 2024-05-27T07:34:01.803 INFO:tasks.workunit.client.0.smithi043.stdout: 10 40017 5.86 MB/sec execute 242 sec latency 100.312 ms 2024-05-27T07:34:01.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:01.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:01.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:34:01.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:01.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:01.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:34:02.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:34:02.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:34:02.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:34:02.804 INFO:tasks.workunit.client.0.smithi043.stdout: 10 40111 5.86 MB/sec execute 243 sec latency 106.470 ms 2024-05-27T07:34:02.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:02 smithi043 ceph-mon[31531]: pgmap v674: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 4.9 MiB/s wr, 238 op/s 2024-05-27T07:34:02.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:02 smithi043 ceph-mon[31531]: pgmap v675: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.5 MiB/s wr, 198 op/s 2024-05-27T07:34:02.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:02 smithi043 ceph-mon[31531]: pgmap v676: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 5.6 MiB/s wr, 248 op/s 2024-05-27T07:34:02.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:02 smithi043 ceph-mon[31531]: osdmap e150: 12 total, 12 up, 12 in 2024-05-27T07:34:02.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:02 smithi179 ceph-mon[35888]: pgmap v674: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 4.9 MiB/s wr, 238 op/s 2024-05-27T07:34:02.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:02 smithi179 ceph-mon[35888]: pgmap v675: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.5 MiB/s wr, 198 op/s 2024-05-27T07:34:02.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:02 smithi179 ceph-mon[35888]: pgmap v676: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 5.6 MiB/s wr, 248 op/s 2024-05-27T07:34:02.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:02 smithi179 ceph-mon[35888]: osdmap e150: 12 total, 12 up, 12 in 2024-05-27T07:34:03.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:02 smithi132 ceph-mon[35283]: pgmap v674: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 4.9 MiB/s wr, 238 op/s 2024-05-27T07:34:03.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:02 smithi132 ceph-mon[35283]: pgmap v675: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.5 MiB/s wr, 198 op/s 2024-05-27T07:34:03.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:02 smithi132 ceph-mon[35283]: pgmap v676: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 5.6 MiB/s wr, 248 op/s 2024-05-27T07:34:03.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:02 smithi132 ceph-mon[35283]: osdmap e150: 12 total, 12 up, 12 in 2024-05-27T07:34:03.804 INFO:tasks.workunit.client.0.smithi043.stdout: 10 40217 5.87 MB/sec execute 244 sec latency 182.304 ms 2024-05-27T07:34:04.804 INFO:tasks.workunit.client.0.smithi043.stdout: 10 40344 5.87 MB/sec execute 245 sec latency 86.301 ms 2024-05-27T07:34:04.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:04 smithi043 ceph-mon[31531]: pgmap v678: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-05-27T07:34:04.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:04 smithi179 ceph-mon[35888]: pgmap v678: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-05-27T07:34:05.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:04 smithi132 ceph-mon[35283]: pgmap v678: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-05-27T07:34:05.804 INFO:tasks.workunit.client.0.smithi043.stdout: 10 40467 5.88 MB/sec execute 246 sec latency 69.877 ms 2024-05-27T07:34:05.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:05 smithi043 ceph-mon[31531]: pgmap v679: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-05-27T07:34:05.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:05 smithi179 ceph-mon[35888]: pgmap v679: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-05-27T07:34:06.094 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:05 smithi132 ceph-mon[35283]: pgmap v679: 129 pgs: 129 active+clean; 913 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-05-27T07:34:06.804 INFO:tasks.workunit.client.0.smithi043.stdout: 10 40596 5.87 MB/sec execute 247 sec latency 80.768 ms 2024-05-27T07:34:07.804 INFO:tasks.workunit.client.0.smithi043.stdout: 10 40718 5.88 MB/sec execute 248 sec latency 91.678 ms 2024-05-27T07:34:08.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:08 smithi043 ceph-mon[31531]: pgmap v680: 129 pgs: 129 active+clean; 931 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-05-27T07:34:08.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:08 smithi179 ceph-mon[35888]: pgmap v680: 129 pgs: 129 active+clean; 931 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-05-27T07:34:08.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:08 smithi132 ceph-mon[35283]: pgmap v680: 129 pgs: 129 active+clean; 931 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-05-27T07:34:08.804 INFO:tasks.workunit.client.0.smithi043.stdout: 10 40800 5.87 MB/sec execute 249 sec latency 402.766 ms 2024-05-27T07:34:09.804 INFO:tasks.workunit.client.0.smithi043.stdout: 10 40917 5.86 MB/sec execute 250 sec latency 82.349 ms 2024-05-27T07:34:10.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:10 smithi043 ceph-mon[31531]: pgmap v681: 129 pgs: 129 active+clean; 931 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 2.5 MiB/s wr, 122 op/s 2024-05-27T07:34:10.462 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:10 smithi179 ceph-mon[35888]: pgmap v681: 129 pgs: 129 active+clean; 931 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 2.5 MiB/s wr, 122 op/s 2024-05-27T07:34:10.594 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:10 smithi132 ceph-mon[35283]: pgmap v681: 129 pgs: 129 active+clean; 931 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 2.5 MiB/s wr, 122 op/s 2024-05-27T07:34:10.804 INFO:tasks.workunit.client.0.smithi043.stdout: 10 41042 5.88 MB/sec execute 251 sec latency 91.493 ms 2024-05-27T07:34:11.805 INFO:tasks.workunit.client.0.smithi043.stdout: 10 41148 5.88 MB/sec execute 252 sec latency 84.975 ms 2024-05-27T07:34:12.805 INFO:tasks.workunit.client.0.smithi043.stdout: 10 41258 5.87 MB/sec execute 253 sec latency 81.215 ms 2024-05-27T07:34:12.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:12 smithi132 ceph-mon[35283]: pgmap v682: 129 pgs: 129 active+clean; 949 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-27T07:34:12.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:12 smithi132 ceph-mon[35283]: osdmap e151: 12 total, 12 up, 12 in 2024-05-27T07:34:12.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:12 smithi043 ceph-mon[31531]: pgmap v682: 129 pgs: 129 active+clean; 949 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-27T07:34:12.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:12 smithi043 ceph-mon[31531]: osdmap e151: 12 total, 12 up, 12 in 2024-05-27T07:34:12.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:12 smithi179 ceph-mon[35888]: pgmap v682: 129 pgs: 129 active+clean; 949 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-27T07:34:12.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:12 smithi179 ceph-mon[35888]: osdmap e151: 12 total, 12 up, 12 in 2024-05-27T07:34:13.805 INFO:tasks.workunit.client.0.smithi043.stdout: 10 41411 5.90 MB/sec execute 254 sec latency 94.946 ms 2024-05-27T07:34:14.805 INFO:tasks.workunit.client.0.smithi043.stdout: 10 41526 5.90 MB/sec execute 255 sec latency 95.792 ms 2024-05-27T07:34:14.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:14 smithi132 ceph-mon[35283]: pgmap v684: 129 pgs: 129 active+clean; 949 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-27T07:34:14.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:14 smithi043 ceph-mon[31531]: pgmap v684: 129 pgs: 129 active+clean; 949 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-27T07:34:14.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:14 smithi179 ceph-mon[35888]: pgmap v684: 129 pgs: 129 active+clean; 949 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-27T07:34:15.805 INFO:tasks.workunit.client.0.smithi043.stdout: 10 41663 5.90 MB/sec execute 256 sec latency 76.535 ms 2024-05-27T07:34:16.765 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:16 smithi179 ceph-mon[35888]: pgmap v685: 129 pgs: 129 active+clean; 949 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-27T07:34:16.805 INFO:tasks.workunit.client.0.smithi043.stdout: 10 41716 5.88 MB/sec execute 257 sec latency 467.263 ms 2024-05-27T07:34:16.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:16 smithi132 ceph-mon[35283]: pgmap v685: 129 pgs: 129 active+clean; 949 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-27T07:34:16.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:16 smithi043 ceph-mon[31531]: pgmap v685: 129 pgs: 129 active+clean; 949 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.1 MiB/s wr, 186 op/s 2024-05-27T07:34:17.805 INFO:tasks.workunit.client.0.smithi043.stdout: 10 41811 5.88 MB/sec execute 258 sec latency 359.103 ms 2024-05-27T07:34:18.805 INFO:tasks.workunit.client.0.smithi043.stdout: 10 41910 5.87 MB/sec execute 259 sec latency 71.247 ms 2024-05-27T07:34:18.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:18 smithi132 ceph-mon[35283]: pgmap v686: 129 pgs: 129 active+clean; 963 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-05-27T07:34:18.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:18 smithi043 ceph-mon[31531]: pgmap v686: 129 pgs: 129 active+clean; 963 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-05-27T07:34:18.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:18 smithi179 ceph-mon[35888]: pgmap v686: 129 pgs: 129 active+clean; 963 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-05-27T07:34:19.481 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:34:19.805 INFO:tasks.workunit.client.0.smithi043.stdout: 10 42002 5.85 MB/sec execute 260 sec latency 107.779 ms 2024-05-27T07:34:20.085 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:20.505 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:20 smithi043 ceph-mon[31531]: pgmap v687: 129 pgs: 129 active+clean; 963 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-05-27T07:34:20.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 42113 5.86 MB/sec execute 261 sec latency 105.823 ms 2024-05-27T07:34:20.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:20 smithi132 ceph-mon[35283]: pgmap v687: 129 pgs: 129 active+clean; 963 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-05-27T07:34:20.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:20 smithi179 ceph-mon[35888]: pgmap v687: 129 pgs: 129 active+clean; 963 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-05-27T07:34:21.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 42236 5.85 MB/sec execute 262 sec latency 88.255 ms 2024-05-27T07:34:22.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:22 smithi043 ceph-mon[31531]: pgmap v688: 129 pgs: 129 active+clean; 968 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.8 MiB/s wr, 191 op/s 2024-05-27T07:34:22.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 42336 5.87 MB/sec execute 263 sec latency 300.100 ms 2024-05-27T07:34:22.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:22 smithi132 ceph-mon[35283]: pgmap v688: 129 pgs: 129 active+clean; 968 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.8 MiB/s wr, 191 op/s 2024-05-27T07:34:22.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:22 smithi179 ceph-mon[35888]: pgmap v688: 129 pgs: 129 active+clean; 968 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.8 MiB/s wr, 191 op/s 2024-05-27T07:34:23.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.232+0000 7f9371600700 1 -- 172.21.15.43:0/3036755740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f936c111bc0 msgr2=0x7f936c111fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:23.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.232+0000 7f9371600700 1 --2- 172.21.15.43:0/3036755740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f936c111bc0 0x7f936c111fd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f935c009990 tx=0x7f935c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.232+0000 7f9371600700 1 -- 172.21.15.43:0/3036755740 shutdown_connections 2024-05-27T07:34:23.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.232+0000 7f9371600700 1 --2- 172.21.15.43:0/3036755740 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f936c10b270 0x7f936c1198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.232+0000 7f9371600700 1 --2- 172.21.15.43:0/3036755740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f936c111bc0 0x7f936c111fd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.232+0000 7f9371600700 1 --2- 172.21.15.43:0/3036755740 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f936c10a920 0x7f936c10ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.232+0000 7f9371600700 1 -- 172.21.15.43:0/3036755740 >> 172.21.15.43:0/3036755740 conn(0x7f936c06f4c0 msgr2=0x7f936c06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:23.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.233+0000 7f9371600700 1 -- 172.21.15.43:0/3036755740 shutdown_connections 2024-05-27T07:34:23.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.233+0000 7f9371600700 1 -- 172.21.15.43:0/3036755740 wait complete. 2024-05-27T07:34:23.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.234+0000 7f9371600700 1 Processor -- start 2024-05-27T07:34:23.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.235+0000 7f9371600700 1 -- start start 2024-05-27T07:34:23.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.235+0000 7f9371600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f936c10a920 0x7f936c116d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.235+0000 7f9371600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f936c10b270 0x7f936c117250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.235+0000 7f9371600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f936c111bc0 0x7f936c117790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.235+0000 7f9371600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f936c10dfb0 con 0x7f936c10a920 2024-05-27T07:34:23.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.235+0000 7f9371600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f936c10de30 con 0x7f936c10b270 2024-05-27T07:34:23.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.235+0000 7f9371600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f936c10e130 con 0x7f936c111bc0 2024-05-27T07:34:23.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.236+0000 7f936aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f936c10a920 0x7f936c116d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.237+0000 7f936a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f936c10b270 0x7f936c117250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.237+0000 7f936a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f936c10b270 0x7f936c117250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:45046/0 (socket says 172.21.15.43:45046) 2024-05-27T07:34:23.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.237+0000 7f936a000700 1 -- 172.21.15.43:0/2218801000 learned_addr learned my addr 172.21.15.43:0/2218801000 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:23.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.237+0000 7f936b400700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f936c111bc0 0x7f936c117790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.238+0000 7f936a000700 1 -- 172.21.15.43:0/2218801000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f936c111bc0 msgr2=0x7f936c117790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:23.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.238+0000 7f936a000700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f936c111bc0 0x7f936c117790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.238+0000 7f936a000700 1 -- 172.21.15.43:0/2218801000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f936c10a920 msgr2=0x7f936c116d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:23.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.238+0000 7f936a000700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f936c10a920 0x7f936c116d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.238+0000 7f936a000700 1 -- 172.21.15.43:0/2218801000 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936c1b5c50 con 0x7f936c10b270 2024-05-27T07:34:23.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.238+0000 7f936a000700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f936c10b270 0x7f936c117250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f935800eb30 tx=0x7f935800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:23.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.239+0000 7f9363400700 1 -- 172.21.15.43:0/2218801000 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935800cbd0 con 0x7f936c10b270 2024-05-27T07:34:23.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.239+0000 7f9363400700 1 -- 172.21.15.43:0/2218801000 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f93580044b0 con 0x7f936c10b270 2024-05-27T07:34:23.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.239+0000 7f9363400700 1 -- 172.21.15.43:0/2218801000 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9358003dc0 con 0x7f936c10b270 2024-05-27T07:34:23.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.240+0000 7f9371600700 1 -- 172.21.15.43:0/2218801000 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f936c1b5f40 con 0x7f936c10b270 2024-05-27T07:34:23.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.240+0000 7f9371600700 1 -- 172.21.15.43:0/2218801000 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f936c1b6340 con 0x7f936c10b270 2024-05-27T07:34:23.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.248+0000 7f938009ab80 1 Processor -- start 2024-05-27T07:34:23.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.249+0000 7f938009ab80 1 -- start start 2024-05-27T07:34:23.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.250+0000 7f938009ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0eda330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.250+0000 7f938009ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0d5be40 0x5627d0eda870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.250+0000 7f938009ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0edae70 0x5627d0ee7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.250+0000 7f938009ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5627d0e08e40 con 0x5627d0edae70 2024-05-27T07:34:23.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.250+0000 7f938009ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5627d0df89e0 con 0x5627d0e479f0 2024-05-27T07:34:23.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.250+0000 7f938009ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5627d0dfca20 con 0x5627d0d5be40 2024-05-27T07:34:23.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.250+0000 7f936aa00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0d5be40 0x5627d0eda870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.250+0000 7f936aa00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0d5be40 0x5627d0eda870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:45526/0 (socket says 172.21.15.43:45526) 2024-05-27T07:34:23.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.250+0000 7f936b400700 1 --2- 172.21.15.43:0/3917640062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0eda330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.251+0000 7f936aa00700 1 -- 172.21.15.43:0/3917640062 learned_addr learned my addr 172.21.15.43:0/3917640062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:23.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.251+0000 7f936b400700 1 -- 172.21.15.43:0/3917640062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0d5be40 msgr2=0x5627d0eda870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:23.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.252+0000 7f936b400700 1 --2- 172.21.15.43:0/3917640062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0d5be40 0x5627d0eda870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.252+0000 7f936b400700 1 -- 172.21.15.43:0/3917640062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0edae70 msgr2=0x5627d0ee7110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:34:23.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.252+0000 7f936b400700 1 --2- 172.21.15.43:0/3917640062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0edae70 0x5627d0ee7110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.252+0000 7f936b400700 1 -- 172.21.15.43:0/3917640062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627d0dc1420 con 0x5627d0e479f0 2024-05-27T07:34:23.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.252+0000 7f9363400700 1 -- 172.21.15.43:0/2218801000 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9358003f60 con 0x7f936c10b270 2024-05-27T07:34:23.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.252+0000 7f936b400700 1 --2- 172.21.15.43:0/3917640062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0eda330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9364011580 tx=0x7f9364009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d2b897555ee94b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:23.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.253+0000 7f9357e00700 1 -- 172.21.15.43:0/3917640062 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364011e80 con 0x5627d0e479f0 2024-05-27T07:34:23.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.253+0000 7f9357e00700 1 -- 172.21.15.43:0/3917640062 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9364004bb0 con 0x5627d0e479f0 2024-05-27T07:34:23.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.253+0000 7f9357e00700 1 -- 172.21.15.43:0/3917640062 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93640045d0 con 0x5627d0e479f0 2024-05-27T07:34:23.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.255+0000 7f938009ab80 1 -- 172.21.15.43:0/3917640062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 msgr2=0x5627d0eda330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:23.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.255+0000 7f938009ab80 1 --2- 172.21.15.43:0/3917640062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0eda330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9364011580 tx=0x7f9364009920 comp rx=0 tx=0).stop 2024-05-27T07:34:23.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.257+0000 7f9363400700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f93500b2a70 0x7f93500b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.258+0000 7f9363400700 1 -- 172.21.15.43:0/2218801000 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f93580e2c10 con 0x7f936c10b270 2024-05-27T07:34:23.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.258+0000 7f936b400700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f93500b2a70 0x7f93500b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.256+0000 7f938009ab80 1 -- 172.21.15.43:0/3917640062 shutdown_connections 2024-05-27T07:34:23.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.259+0000 7f938009ab80 1 --2- 172.21.15.43:0/3917640062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0edae70 0x5627d0ee7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.259+0000 7f938009ab80 1 --2- 172.21.15.43:0/3917640062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0eda330 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.259+0000 7f938009ab80 1 --2- 172.21.15.43:0/3917640062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0d5be40 0x5627d0eda870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.259+0000 7f936b400700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f93500b2a70 0x7f93500b4f80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9364009520 tx=0x7f93640041b0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.260+0000 7f938009ab80 1 -- 172.21.15.43:0/3917640062 >> 172.21.15.43:0/3917640062 conn(0x5627d0df5390 msgr2=0x5627d0ed64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:23.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.260+0000 7f938009ab80 1 -- 172.21.15.43:0/3917640062 shutdown_connections 2024-05-27T07:34:23.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.260+0000 7f938009ab80 1 -- 172.21.15.43:0/3917640062 wait complete. 2024-05-27T07:34:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.261+0000 7f938009ab80 1 -- start start 2024-05-27T07:34:23.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.262+0000 7f938009ab80 1 Processor -- start 2024-05-27T07:34:23.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.264+0000 7f938009ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0d5be40 0x5627d0ee4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.264+0000 7f938009ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0ee4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.265+0000 7f938009ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0ee50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.265+0000 7f936aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0d5be40 0x5627d0ee4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.265+0000 7f936aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0d5be40 0x5627d0ee4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43772/0 (socket says 172.21.15.43:43772) 2024-05-27T07:34:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.265+0000 7f936aa00700 1 -- 172.21.15.43:0/1599534837 learned_addr learned my addr 172.21.15.43:0/1599534837 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.265+0000 7f936a000700 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0ee4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.266+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5627d0cfefa0 con 0x5627d0d5be40 2024-05-27T07:34:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.266+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5627d0d99760 con 0x5627d0e479f0 2024-05-27T07:34:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.266+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5627d0ef8b90 con 0x5627d0edae70 2024-05-27T07:34:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.266+0000 7f938009ab80 10 client.0 _ll_register_callbacks cb 0x5627d0da2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.266+0000 7f936b400700 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0ee50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.267+0000 7f936b400700 1 -- 172.21.15.43:0/1599534837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 msgr2=0x5627d0ee4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.267+0000 7f936b400700 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0ee4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.267+0000 7f936b400700 1 -- 172.21.15.43:0/1599534837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0d5be40 msgr2=0x5627d0ee4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:23.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.267+0000 7f936b400700 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0d5be40 0x5627d0ee4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.267+0000 7f936b400700 1 -- 172.21.15.43:0/1599534837 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627d0d488d0 con 0x5627d0edae70 2024-05-27T07:34:23.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.268+0000 7f936a000700 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0ee4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:23.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.268+0000 7f936b400700 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0ee50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9364009ec0 tx=0x7f936400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b7c35d5a174801e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:23.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.268+0000 7f9354c00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:34:23.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.269+0000 7f9354c00700 1 -- 172.21.15.43:0/1599534837 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364007710 con 0x5627d0edae70 2024-05-27T07:34:23.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.269+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9364004bb0 con 0x5627d0edae70 2024-05-27T07:34:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.270+0000 7f9354c00700 1 -- 172.21.15.43:0/1599534837 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f93640078b0 con 0x5627d0edae70 2024-05-27T07:34:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.271+0000 7f9354c00700 1 -- 172.21.15.43:0/1599534837 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f936403c050 con 0x5627d0edae70 2024-05-27T07:34:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.271+0000 7f938009ab80 10 client.27547 fetch_fsmap learned FSMap version 139 2024-05-27T07:34:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.271+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x5627d0dc9060 con 0x5627d0edae70 2024-05-27T07:34:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.272+0000 7f9354c00700 1 -- 172.21.15.43:0/1599534837 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9364004bb0 con 0x5627d0edae70 2024-05-27T07:34:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.272+0000 7f9354c00700 1 -- 172.21.15.43:0/1599534837 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f936401fa60 con 0x5627d0edae70 2024-05-27T07:34:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.272+0000 7f938009ab80 10 client.27547 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:34:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.272+0000 7f938009ab80 10 client.27547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.273+0000 7f938009ab80 20 client.27547 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.273+0000 7f938009ab80 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5627d0f05de0 0x5627d0f082b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.273+0000 7f938009ab80 4 client.27547 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.273+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627d0e0ad90 con 0x5627d0f05de0 2024-05-27T07:34:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.273+0000 7f936aa00700 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5627d0f05de0 0x5627d0f082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.276+0000 7f936aa00700 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5627d0f05de0 0x5627d0f082b0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc685524f9bdeddb server_cookie=3bfa4afb6ab92005 in_seq=0 out_seq=0 2024-05-27T07:34:23.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.276+0000 7f9354c00700 10 client.27547 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:23.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.278+0000 7f9354c00700 1 -- 172.21.15.43:0/1599534837 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5627d0e0ad90 con 0x5627d0f05de0 2024-05-27T07:34:23.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.279+0000 7f9354c00700 10 client.27547 handle_command_reply: tid=0 2024-05-27T07:34:23.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.279+0000 7f938009ab80 1 client.27547 shutdown 2024-05-27T07:34:23.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.281+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 msgr2=0x5627d0ee50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:23.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.281+0000 7f938009ab80 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0ee50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9364009ec0 tx=0x7f936400b040 comp rx=0 tx=0).stop 2024-05-27T07:34:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.282+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 shutdown_connections 2024-05-27T07:34:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.282+0000 7f938009ab80 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0ee50d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.282+0000 7f938009ab80 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5627d0f05de0 0x5627d0f082b0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.282+0000 7f938009ab80 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0d5be40 0x5627d0ee4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.282+0000 7f938009ab80 1 --2- 172.21.15.43:0/1599534837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0ee4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.282+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 >> 172.21.15.43:0/1599534837 conn(0x5627d0df5390 msgr2=0x5627d0eda3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.282+0000 7f9354c00700 0 client.27547 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.282+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 shutdown_connections 2024-05-27T07:34:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.282+0000 7f938009ab80 1 -- 172.21.15.43:0/1599534837 wait complete. 2024-05-27T07:34:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.283+0000 7f938009ab80 20 client.27547 trim_cache size 0 max 16384 2024-05-27T07:34:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.317+0000 7f938009ab80 1 Processor -- start 2024-05-27T07:34:23.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.320+0000 7f938009ab80 1 -- start start 2024-05-27T07:34:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.320+0000 7f938009ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0d5be40 0x5627d0ee35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.320+0000 7f938009ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0e479f0 0x5627d0edfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.320+0000 7f938009ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0f01c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.320+0000 7f938009ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f936c1b70b0 con 0x5627d0e479f0 2024-05-27T07:34:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.320+0000 7f938009ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f936c126c30 con 0x5627d0d5be40 2024-05-27T07:34:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.320+0000 7f938009ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f936c1b69e0 con 0x5627d0edae70 2024-05-27T07:34:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.321+0000 7f936aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0d5be40 0x5627d0ee35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.321+0000 7f936aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0d5be40 0x5627d0ee35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:45068/0 (socket says 172.21.15.43:45068) 2024-05-27T07:34:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.321+0000 7f936aa00700 1 -- 172.21.15.43:0/2256351988 learned_addr learned my addr 172.21.15.43:0/2256351988 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.322+0000 7f936a000700 1 --2- 172.21.15.43:0/2256351988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0e479f0 0x5627d0edfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.322+0000 7f936b400700 1 --2- 172.21.15.43:0/2256351988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0f01c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.323+0000 7f936b400700 1 -- 172.21.15.43:0/2256351988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0d5be40 msgr2=0x5627d0ee35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.323+0000 7f936b400700 1 --2- 172.21.15.43:0/2256351988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0d5be40 0x5627d0ee35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.323+0000 7f936b400700 1 -- 172.21.15.43:0/2256351988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0e479f0 msgr2=0x5627d0edfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.323+0000 7f936b400700 1 --2- 172.21.15.43:0/2256351988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0e479f0 0x5627d0edfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.323+0000 7f936b400700 1 -- 172.21.15.43:0/2256351988 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627d0ed6620 con 0x5627d0edae70 2024-05-27T07:34:23.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.324+0000 7f936b400700 1 --2- 172.21.15.43:0/2256351988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0f01c60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f936402b4d0 tx=0x7f9364003a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d887b9160af530f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:23.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.325+0000 7f9357400700 1 -- 172.21.15.43:0/2256351988 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364011e80 con 0x5627d0edae70 2024-05-27T07:34:23.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.325+0000 7f9357400700 1 -- 172.21.15.43:0/2256351988 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9364005c60 con 0x5627d0edae70 2024-05-27T07:34:23.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.326+0000 7f9357400700 1 -- 172.21.15.43:0/2256351988 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364007d50 con 0x5627d0edae70 2024-05-27T07:34:23.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.327+0000 7f938009ab80 1 -- 172.21.15.43:0/2256351988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 msgr2=0x5627d0f01c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:23.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.327+0000 7f938009ab80 1 --2- 172.21.15.43:0/2256351988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0f01c60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f936402b4d0 tx=0x7f9364003a00 comp rx=0 tx=0).stop 2024-05-27T07:34:23.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.327+0000 7f938009ab80 1 -- 172.21.15.43:0/2256351988 shutdown_connections 2024-05-27T07:34:23.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.327+0000 7f938009ab80 1 --2- 172.21.15.43:0/2256351988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0f01c60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.327+0000 7f938009ab80 1 --2- 172.21.15.43:0/2256351988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0d5be40 0x5627d0ee35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.327+0000 7f938009ab80 1 --2- 172.21.15.43:0/2256351988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0e479f0 0x5627d0edfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.327+0000 7f938009ab80 1 -- 172.21.15.43:0/2256351988 >> 172.21.15.43:0/2256351988 conn(0x5627d0df5390 msgr2=0x5627d0ee5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:23.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.327+0000 7f938009ab80 1 -- 172.21.15.43:0/2256351988 shutdown_connections 2024-05-27T07:34:23.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.327+0000 7f938009ab80 1 -- 172.21.15.43:0/2256351988 wait complete. 2024-05-27T07:34:23.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.328+0000 7f938009ab80 1 -- start start 2024-05-27T07:34:23.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.328+0000 7f938009ab80 1 Processor -- start 2024-05-27T07:34:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.329+0000 7f938009ab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0d5be40 0x5627d0edea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.329+0000 7f938009ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0edef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.329+0000 7f938009ab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0efce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.329+0000 7f938009ab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f936c127a30 con 0x5627d0d5be40 2024-05-27T07:34:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.329+0000 7f938009ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f936c127300 con 0x5627d0e479f0 2024-05-27T07:34:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.329+0000 7f938009ab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5627d0efd390 con 0x5627d0edae70 2024-05-27T07:34:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.329+0000 7f938009ab80 10 client.0 _ll_register_callbacks cb 0x5627d0da2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.330+0000 7f936a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0edef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.330+0000 7f936a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0edef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:45076/0 (socket says 172.21.15.43:45076) 2024-05-27T07:34:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.330+0000 7f936a000700 1 -- 172.21.15.43:0/3364001357 learned_addr learned my addr 172.21.15.43:0/3364001357 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.330+0000 7f936a000700 1 -- 172.21.15.43:0/3364001357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 msgr2=0x5627d0efce50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:34:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.330+0000 7f936a000700 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0efce50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.330+0000 7f936a000700 1 -- 172.21.15.43:0/3364001357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0d5be40 msgr2=0x5627d0edea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.330+0000 7f936a000700 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0d5be40 0x5627d0edea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.330+0000 7f936a000700 1 -- 172.21.15.43:0/3364001357 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627d0efd590 con 0x5627d0e479f0 2024-05-27T07:34:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.331+0000 7f936a000700 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0edef70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f93580e0040 tx=0x7f93580a9a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6a56c7e125932ac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.331+0000 7f9354c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:34:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.331+0000 7f9354c00700 1 -- 172.21.15.43:0/3364001357 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93580abe60 con 0x5627d0e479f0 2024-05-27T07:34:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.332+0000 7f9354c00700 1 -- 172.21.15.43:0/3364001357 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f93580b2700 con 0x5627d0e479f0 2024-05-27T07:34:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.332+0000 7f938009ab80 1 -- 172.21.15.43:0/3364001357 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9364005c60 con 0x5627d0e479f0 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.332+0000 7f9354c00700 1 -- 172.21.15.43:0/3364001357 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f935800ce90 con 0x5627d0e479f0 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.332+0000 7f938009ab80 10 client.37487 fetch_fsmap learned FSMap version 139 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.332+0000 7f938009ab80 1 -- 172.21.15.43:0/3364001357 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x5627d0efe860 con 0x5627d0e479f0 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.333+0000 7f9354c00700 1 -- 172.21.15.43:0/3364001357 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9364005c60 con 0x5627d0e479f0 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.333+0000 7f9354c00700 1 -- 172.21.15.43:0/3364001357 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f93580a5490 con 0x5627d0e479f0 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.333+0000 7f938009ab80 10 client.37487 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.333+0000 7f938009ab80 10 client.37487 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.333+0000 7f938009ab80 20 client.37487 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.333+0000 7f938009ab80 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5627d0f70bd0 0x5627d0f72ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.333+0000 7f938009ab80 4 client.37487 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.333+0000 7f938009ab80 1 -- 172.21.15.43:0/3364001357 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5627d0e04570 con 0x5627d0f70bd0 2024-05-27T07:34:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.333+0000 7f936aa00700 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5627d0f70bd0 0x5627d0f72ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.334+0000 7f936aa00700 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5627d0f70bd0 0x5627d0f72ff0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd8d58209ef8f7c server_cookie=f95c7f13e9c7c9b3 in_seq=0 out_seq=0 2024-05-27T07:34:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.334+0000 7f9354c00700 10 client.37487 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f9354c00700 1 -- 172.21.15.43:0/3364001357 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5627d0e04570 con 0x5627d0f70bd0 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f9354c00700 10 client.37487 handle_command_reply: tid=0 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f938009ab80 1 client.37487 shutdown 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f938009ab80 1 -- 172.21.15.43:0/3364001357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 msgr2=0x5627d0edef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f938009ab80 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0edef70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f93580e0040 tx=0x7f93580a9a00 comp rx=0 tx=0).stop 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f938009ab80 1 -- 172.21.15.43:0/3364001357 shutdown_connections 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f938009ab80 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5627d0edae70 0x5627d0efce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f938009ab80 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5627d0f70bd0 0x5627d0f72ff0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f938009ab80 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5627d0d5be40 0x5627d0edea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f938009ab80 1 --2- 172.21.15.43:0/3364001357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5627d0e479f0 0x5627d0edef70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.335+0000 7f938009ab80 1 -- 172.21.15.43:0/3364001357 >> 172.21.15.43:0/3364001357 conn(0x5627d0df5390 msgr2=0x5627d0ee5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:23.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.336+0000 7f9354c00700 0 client.37487 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:23.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.337+0000 7f938009ab80 1 -- 172.21.15.43:0/3364001357 shutdown_connections 2024-05-27T07:34:23.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.337+0000 7f938009ab80 1 -- 172.21.15.43:0/3364001357 wait complete. 2024-05-27T07:34:23.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.337+0000 7f938009ab80 20 client.37487 trim_cache size 0 max 16384 2024-05-27T07:34:23.332 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:34:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:34:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:34:23.333 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:34:23.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.338+0000 7f9371600700 1 -- 172.21.15.43:0/2218801000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f93500b2a70 msgr2=0x7f93500b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:23.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.338+0000 7f9371600700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f93500b2a70 0x7f93500b4f80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9364009520 tx=0x7f93640041b0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.338+0000 7f9371600700 1 -- 172.21.15.43:0/2218801000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f936c10b270 msgr2=0x7f936c117250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:23.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.338+0000 7f9371600700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f936c10b270 0x7f936c117250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f935800eb30 tx=0x7f935800ee40 comp rx=0 tx=0).stop 2024-05-27T07:34:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.338+0000 7f9371600700 1 -- 172.21.15.43:0/2218801000 shutdown_connections 2024-05-27T07:34:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.338+0000 7f9371600700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f93500b2a70 0x7f93500b4f80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.338+0000 7f9371600700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f936c111bc0 0x7f936c117790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.338+0000 7f9371600700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f936c10a920 0x7f936c116d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.338+0000 7f9371600700 1 --2- 172.21.15.43:0/2218801000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f936c10b270 0x7f936c117250 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.338+0000 7f9371600700 1 -- 172.21.15.43:0/2218801000 >> 172.21.15.43:0/2218801000 conn(0x7f936c06f4c0 msgr2=0x7f936c110760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.339+0000 7f9371600700 1 -- 172.21.15.43:0/2218801000 shutdown_connections 2024-05-27T07:34:23.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:23.339+0000 7f9371600700 1 -- 172.21.15.43:0/2218801000 wait complete. 2024-05-27T07:34:23.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 42439 5.86 MB/sec execute 264 sec latency 103.525 ms 2024-05-27T07:34:24.213 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:34:24.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:34:24.750 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:24 smithi043 ceph-mon[31531]: pgmap v689: 129 pgs: 129 active+clean; 968 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 165 op/s 2024-05-27T07:34:24.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 42554 5.86 MB/sec execute 265 sec latency 96.427 ms 2024-05-27T07:34:24.812 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:24.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:24 smithi132 ceph-mon[35283]: pgmap v689: 129 pgs: 129 active+clean; 968 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 165 op/s 2024-05-27T07:34:24.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:24 smithi179 ceph-mon[35888]: pgmap v689: 129 pgs: 129 active+clean; 968 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 165 op/s 2024-05-27T07:34:25.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 42688 5.88 MB/sec execute 266 sec latency 81.055 ms 2024-05-27T07:34:26.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 42784 5.87 MB/sec execute 267 sec latency 87.129 ms 2024-05-27T07:34:26.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:26 smithi132 ceph-mon[35283]: pgmap v690: 129 pgs: 129 active+clean; 968 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-05-27T07:34:26.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:26 smithi043 ceph-mon[31531]: pgmap v690: 129 pgs: 129 active+clean; 968 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-05-27T07:34:26.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:26 smithi179 ceph-mon[35888]: pgmap v690: 129 pgs: 129 active+clean; 968 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-05-27T07:34:27.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.174+0000 7fc5c5200700 1 -- 172.21.15.43:0/3502802457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c01054d0 msgr2=0x7fc5c01058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:27.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.174+0000 7fc5c5200700 1 --2- 172.21.15.43:0/3502802457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c01054d0 0x7fc5c01058e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009990 tx=0x7fc5b0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:34:27.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.174+0000 7fc5c5200700 1 -- 172.21.15.43:0/3502802457 shutdown_connections 2024-05-27T07:34:27.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.174+0000 7fc5c5200700 1 --2- 172.21.15.43:0/3502802457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5c010e900 0x7fc5c0110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:27.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.174+0000 7fc5c5200700 1 --2- 172.21.15.43:0/3502802457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c01054d0 0x7fc5c01058e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:27.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.174+0000 7fc5c5200700 1 --2- 172.21.15.43:0/3502802457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c0105e20 0x7fc5c010e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:27.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.174+0000 7fc5c5200700 1 -- 172.21.15.43:0/3502802457 >> 172.21.15.43:0/3502802457 conn(0x7fc5c00fedb0 msgr2=0x7fc5c01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:27.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.175+0000 7fc5c5200700 1 -- 172.21.15.43:0/3502802457 shutdown_connections 2024-05-27T07:34:27.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.175+0000 7fc5c5200700 1 -- 172.21.15.43:0/3502802457 wait complete. 2024-05-27T07:34:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.177+0000 7fc5c5200700 1 Processor -- start 2024-05-27T07:34:27.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.177+0000 7fc5c5200700 1 -- start start 2024-05-27T07:34:27.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.177+0000 7fc5c5200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5c01054d0 0x7fc5c019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:27.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.177+0000 7fc5c5200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c0105e20 0x7fc5c0198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:27.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.177+0000 7fc5c5200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c010e900 0x7fc5c0198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:27.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.177+0000 7fc5c5200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0112160 con 0x7fc5c0105e20 2024-05-27T07:34:27.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.177+0000 7fc5c5200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0111fe0 con 0x7fc5c010e900 2024-05-27T07:34:27.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.177+0000 7fc5c5200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c01122e0 con 0x7fc5c01054d0 2024-05-27T07:34:27.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.178+0000 7fc5bd400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c0105e20 0x7fc5c0198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:27.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.178+0000 7fc5be800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c010e900 0x7fc5c0198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:27.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.178+0000 7fc5be800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c010e900 0x7fc5c0198ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38772/0 (socket says 172.21.15.43:38772) 2024-05-27T07:34:27.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.178+0000 7fc5be800700 1 -- 172.21.15.43:0/3903699570 learned_addr learned my addr 172.21.15.43:0/3903699570 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:27.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.178+0000 7fc5be800700 1 -- 172.21.15.43:0/3903699570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5c01054d0 msgr2=0x7fc5c019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:27.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.178+0000 7fc5bde00700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5c01054d0 0x7fc5c019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:27.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.178+0000 7fc5be800700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5c01054d0 0x7fc5c019f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:27.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.178+0000 7fc5be800700 1 -- 172.21.15.43:0/3903699570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c0105e20 msgr2=0x7fc5c0198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:27.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.179+0000 7fc5be800700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c0105e20 0x7fc5c0198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:27.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.180+0000 7fc5be800700 1 -- 172.21.15.43:0/3903699570 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c019d330 con 0x7fc5c010e900 2024-05-27T07:34:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.180+0000 7fc5bd400700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c0105e20 0x7fc5c0198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.180+0000 7fc5be800700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c010e900 0x7fc5c0198ff0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c370 tx=0x7fc5b800c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.180+0000 7fc5b6a00700 1 -- 172.21.15.43:0/3903699570 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b80090d0 con 0x7fc5c010e900 2024-05-27T07:34:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.180+0000 7fc5c5200700 1 -- 172.21.15.43:0/3903699570 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c019d5c0 con 0x7fc5c010e900 2024-05-27T07:34:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.181+0000 7fc5c5200700 1 -- 172.21.15.43:0/3903699570 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c019da40 con 0x7fc5c010e900 2024-05-27T07:34:27.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.181+0000 7fc5b6a00700 1 -- 172.21.15.43:0/3903699570 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc5b8009270 con 0x7fc5c010e900 2024-05-27T07:34:27.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.181+0000 7fc5b6a00700 1 -- 172.21.15.43:0/3903699570 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8004790 con 0x7fc5c010e900 2024-05-27T07:34:27.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.183+0000 7fc5c5200700 1 -- 172.21.15.43:0/3903699570 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc59c005330 con 0x7fc5c010e900 2024-05-27T07:34:27.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.185+0000 7fc5b6a00700 1 -- 172.21.15.43:0/3903699570 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc5b801a020 con 0x7fc5c010e900 2024-05-27T07:34:27.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.187+0000 7fc5b6a00700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc5a40b2a70 0x7fc5a40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:27.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.187+0000 7fc5b6a00700 1 -- 172.21.15.43:0/3903699570 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7fc5b80e2220 con 0x7fc5c010e900 2024-05-27T07:34:27.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.188+0000 7fc5bde00700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc5a40b2a70 0x7fc5a40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:27.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.189+0000 7fc5bde00700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc5a40b2a70 0x7fc5a40b4f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006010 tx=0x7fc5b0009630 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:27.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.193+0000 7fc5b6a00700 1 -- 172.21.15.43:0/3903699570 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc5b8010040 con 0x7fc5c010e900 2024-05-27T07:34:27.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.694+0000 7fc5c5200700 1 -- 172.21.15.43:0/3903699570 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc59c0058f0 con 0x7fc5c010e900 2024-05-27T07:34:27.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.695+0000 7fc5b6a00700 1 -- 172.21.15.43:0/3903699570 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+6123 (secure 0 0 0) 0x7fc5b80a9830 con 0x7fc5c010e900 2024-05-27T07:34:27.690 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:34:27.690 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":139,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:33:03.596758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34403,24484,34424,34484,34448]},"id":1}]} 2024-05-27T07:34:27.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.699+0000 7fc5c5200700 1 -- 172.21.15.43:0/3903699570 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc5a40b2a70 msgr2=0x7fc5a40b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:27.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.699+0000 7fc5c5200700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc5a40b2a70 0x7fc5a40b4f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006010 tx=0x7fc5b0009630 comp rx=0 tx=0).stop 2024-05-27T07:34:27.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.699+0000 7fc5c5200700 1 -- 172.21.15.43:0/3903699570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c010e900 msgr2=0x7fc5c0198ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:27.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.699+0000 7fc5c5200700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c010e900 0x7fc5c0198ff0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c370 tx=0x7fc5b800c730 comp rx=0 tx=0).stop 2024-05-27T07:34:27.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.699+0000 7fc5c5200700 1 -- 172.21.15.43:0/3903699570 shutdown_connections 2024-05-27T07:34:27.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.699+0000 7fc5c5200700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc5a40b2a70 0x7fc5a40b4f80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:27.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.699+0000 7fc5c5200700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c010e900 0x7fc5c0198ff0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:27.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.699+0000 7fc5c5200700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5c01054d0 0x7fc5c019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:27.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.699+0000 7fc5c5200700 1 --2- 172.21.15.43:0/3903699570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5c0105e20 0x7fc5c0198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:27.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.699+0000 7fc5c5200700 1 -- 172.21.15.43:0/3903699570 >> 172.21.15.43:0/3903699570 conn(0x7fc5c00fedb0 msgr2=0x7fc5c00ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:27.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.700+0000 7fc5c5200700 1 -- 172.21.15.43:0/3903699570 shutdown_connections 2024-05-27T07:34:27.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:27.700+0000 7fc5c5200700 1 -- 172.21.15.43:0/3903699570 wait complete. 2024-05-27T07:34:27.696 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 139 2024-05-27T07:34:27.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 42889 5.87 MB/sec execute 268 sec latency 75.484 ms 2024-05-27T07:34:28.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 42997 5.88 MB/sec execute 269 sec latency 232.829 ms 2024-05-27T07:34:28.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:28 smithi132 ceph-mon[35283]: pgmap v691: 129 pgs: 129 active+clean; 977 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.8 MiB/s wr, 232 op/s 2024-05-27T07:34:28.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:28 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3903699570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:34:28.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:28 smithi043 ceph-mon[31531]: pgmap v691: 129 pgs: 129 active+clean; 977 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.8 MiB/s wr, 232 op/s 2024-05-27T07:34:28.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:28 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3903699570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:34:28.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:28 smithi179 ceph-mon[35888]: pgmap v691: 129 pgs: 129 active+clean; 977 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.8 MiB/s wr, 232 op/s 2024-05-27T07:34:28.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:28 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3903699570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:34:29.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:34:29.666 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:29.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 43112 5.87 MB/sec execute 270 sec latency 85.842 ms 2024-05-27T07:34:30.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 43219 5.87 MB/sec execute 271 sec latency 96.999 ms 2024-05-27T07:34:30.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:30 smithi132 ceph-mon[35283]: pgmap v692: 129 pgs: 129 active+clean; 977 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.9 MiB/s wr, 144 op/s 2024-05-27T07:34:30.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:30 smithi043 ceph-mon[31531]: pgmap v692: 129 pgs: 129 active+clean; 977 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.9 MiB/s wr, 144 op/s 2024-05-27T07:34:30.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:30 smithi179 ceph-mon[35888]: pgmap v692: 129 pgs: 129 active+clean; 977 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.9 MiB/s wr, 144 op/s 2024-05-27T07:34:31.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 43355 5.88 MB/sec execute 272 sec latency 139.573 ms 2024-05-27T07:34:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.201+0000 7f3208e00700 1 -- 172.21.15.43:0/3214896744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3204103b90 msgr2=0x7f32041080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.201+0000 7f3208e00700 1 --2- 172.21.15.43:0/3214896744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3204103b90 0x7f32041080c0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009990 tx=0x7f31f0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.201+0000 7f3208e00700 1 -- 172.21.15.43:0/3214896744 shutdown_connections 2024-05-27T07:34:32.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.201+0000 7f3208e00700 1 --2- 172.21.15.43:0/3214896744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3204108690 0x7f3204108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.201+0000 7f3208e00700 1 --2- 172.21.15.43:0/3214896744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3204103240 0x7f3204103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.201+0000 7f3208e00700 1 --2- 172.21.15.43:0/3214896744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3204103b90 0x7f32041080c0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.201+0000 7f3208e00700 1 -- 172.21.15.43:0/3214896744 >> 172.21.15.43:0/3214896744 conn(0x7f32040fedb0 msgr2=0x7f32041011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.202+0000 7f3208e00700 1 -- 172.21.15.43:0/3214896744 shutdown_connections 2024-05-27T07:34:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.202+0000 7f3208e00700 1 -- 172.21.15.43:0/3214896744 wait complete. 2024-05-27T07:34:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.205+0000 7f3208e00700 1 Processor -- start 2024-05-27T07:34:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.205+0000 7f3208e00700 1 -- start start 2024-05-27T07:34:32.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.206+0000 7f3208e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3204103240 0x7f32041a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.206+0000 7f3208e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3204103b90 0x7f32041a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.206+0000 7f3208e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3204108690 0x7f32041a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.206+0000 7f3208e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f320411a910 con 0x7f3204108690 2024-05-27T07:34:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.206+0000 7f3208e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f320411a790 con 0x7f3204103240 2024-05-27T07:34:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.206+0000 7f3208e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f320411aa90 con 0x7f3204103b90 2024-05-27T07:34:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.206+0000 7f3201e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3204103240 0x7f32041a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.206+0000 7f3201e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3204103240 0x7f32041a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38790/0 (socket says 172.21.15.43:38790) 2024-05-27T07:34:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.206+0000 7f3201e00700 1 -- 172.21.15.43:0/62364383 learned_addr learned my addr 172.21.15.43:0/62364383 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.206+0000 7f3201400700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3204103b90 0x7f32041a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.207+0000 7f3202800700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3204108690 0x7f32041a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.207+0000 7f3201e00700 1 -- 172.21.15.43:0/62364383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3204103b90 msgr2=0x7f32041a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.207+0000 7f3201e00700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3204103b90 0x7f32041a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.207+0000 7f3201e00700 1 -- 172.21.15.43:0/62364383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3204108690 msgr2=0x7f32041a5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.207+0000 7f3201e00700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3204108690 0x7f32041a5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.207+0000 7f3201e00700 1 -- 172.21.15.43:0/62364383 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32041a61d0 con 0x7f3204103240 2024-05-27T07:34:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.208+0000 7f3201e00700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3204103240 0x7f32041a1300 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31f400d950 tx=0x7f31f400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.208+0000 7f31faa00700 1 -- 172.21.15.43:0/62364383 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f4016070 con 0x7f3204103240 2024-05-27T07:34:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.209+0000 7f3208e00700 1 -- 172.21.15.43:0/62364383 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32041a64c0 con 0x7f3204103240 2024-05-27T07:34:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.209+0000 7f31faa00700 1 -- 172.21.15.43:0/62364383 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31f400fd20 con 0x7f3204103240 2024-05-27T07:34:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.209+0000 7f31faa00700 1 -- 172.21.15.43:0/62364383 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f4020060 con 0x7f3204103240 2024-05-27T07:34:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.209+0000 7f3208e00700 1 -- 172.21.15.43:0/62364383 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32041adde0 con 0x7f3204103240 2024-05-27T07:34:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.217+0000 7f31faa00700 1 -- 172.21.15.43:0/62364383 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f31f4019070 con 0x7f3204103240 2024-05-27T07:34:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.219+0000 7f32178dcb80 1 Processor -- start 2024-05-27T07:34:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.220+0000 7f32178dcb80 1 -- start start 2024-05-27T07:34:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.220+0000 7f31faa00700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f31e80b2a70 0x7f31e80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.220+0000 7f31faa00700 1 -- 172.21.15.43:0/62364383 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f31f40e2480 con 0x7f3204103240 2024-05-27T07:34:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.220+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561414871d30 0x56141486b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56141486bdb0 0x56141486c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f32178dcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561414799e40 con 0x561414871d30 2024-05-27T07:34:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f32178dcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5614147899e0 con 0x56141486bdb0 2024-05-27T07:34:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f32178dcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56141478da20 con 0x5614146ece40 2024-05-27T07:34:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f3202800700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f31e80b2a70 0x7f31e80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f3201e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f3201e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:46312/0 (socket says 172.21.15.43:46312) 2024-05-27T07:34:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f3201e00700 1 -- 172.21.15.43:0/2358536067 learned_addr learned my addr 172.21.15.43:0/2358536067 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f3202800700 1 --2- 172.21.15.43:0/2358536067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56141486bdb0 0x56141486c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.221+0000 7f3201400700 1 --2- 172.21.15.43:0/2358536067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561414871d30 0x56141486b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.222+0000 7f3201e00700 1 -- 172.21.15.43:0/2358536067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56141486bdb0 msgr2=0x56141486c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.222+0000 7f3201e00700 1 --2- 172.21.15.43:0/2358536067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56141486bdb0 0x56141486c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.222+0000 7f3201e00700 1 -- 172.21.15.43:0/2358536067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561414871d30 msgr2=0x56141486b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.222+0000 7f3201e00700 1 --2- 172.21.15.43:0/2358536067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561414871d30 0x56141486b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.222+0000 7f3201e00700 1 -- 172.21.15.43:0/2358536067 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561414752420 con 0x5614146ece40 2024-05-27T07:34:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.222+0000 7f3201400700 1 --2- 172.21.15.43:0/2358536067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561414871d30 0x56141486b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:34:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.223+0000 7f3201e00700 1 --2- 172.21.15.43:0/2358536067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148717f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f31f40a53b0 tx=0x7f31f40a8ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2421e7b94b614ed6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.223+0000 7f3202800700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f31e80b2a70 0x7f31e80b4f80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f31fc003fd0 tx=0x7f31fc003a20 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.223+0000 7f31ef400700 1 -- 172.21.15.43:0/2358536067 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f40a9a30 con 0x5614146ece40 2024-05-27T07:34:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.223+0000 7f31ef400700 1 -- 172.21.15.43:0/2358536067 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31f40a9330 con 0x5614146ece40 2024-05-27T07:34:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.223+0000 7f31ef400700 1 -- 172.21.15.43:0/2358536067 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f40a97d0 con 0x5614146ece40 2024-05-27T07:34:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.225+0000 7f32178dcb80 1 -- 172.21.15.43:0/2358536067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 msgr2=0x5614148717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.225+0000 7f32178dcb80 1 --2- 172.21.15.43:0/2358536067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148717f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f31f40a53b0 tx=0x7f31f40a8ec0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.225+0000 7f32178dcb80 1 -- 172.21.15.43:0/2358536067 shutdown_connections 2024-05-27T07:34:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.225+0000 7f32178dcb80 1 --2- 172.21.15.43:0/2358536067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56141486bdb0 0x56141486c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.225+0000 7f32178dcb80 1 --2- 172.21.15.43:0/2358536067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148717f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.225+0000 7f32178dcb80 1 --2- 172.21.15.43:0/2358536067 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561414871d30 0x56141486b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.225+0000 7f32178dcb80 1 -- 172.21.15.43:0/2358536067 >> 172.21.15.43:0/2358536067 conn(0x56141474cfc0 msgr2=0x5614148668d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.225+0000 7f32178dcb80 1 -- 172.21.15.43:0/2358536067 shutdown_connections 2024-05-27T07:34:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.225+0000 7f32178dcb80 1 -- 172.21.15.43:0/2358536067 wait complete. 2024-05-27T07:34:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.226+0000 7f32178dcb80 1 -- start start 2024-05-27T07:34:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.226+0000 7f32178dcb80 1 Processor -- start 2024-05-27T07:34:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.227+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148755c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.227+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x561414875b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.227+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561414871d30 0x561414876040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.227+0000 7f32178dcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56141468ffa0 con 0x56141486bdb0 2024-05-27T07:34:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.227+0000 7f32178dcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56141472a760 con 0x561414871d30 2024-05-27T07:34:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.227+0000 7f32178dcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5614148897b0 con 0x5614146ece40 2024-05-27T07:34:32.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.227+0000 7f32178dcb80 10 client.0 _ll_register_callbacks cb 0x561414733c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.228+0000 7f3201e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.228+0000 7f3201e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148755c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:46314/0 (socket says 172.21.15.43:46314) 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.228+0000 7f3201e00700 1 -- 172.21.15.43:0/3200641748 learned_addr learned my addr 172.21.15.43:0/3200641748 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.228+0000 7f3201400700 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x561414875b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.228+0000 7f3202800700 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561414871d30 0x561414876040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.229+0000 7f3201400700 1 -- 172.21.15.43:0/3200641748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 msgr2=0x5614148755c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.229+0000 7f3201400700 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148755c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.229+0000 7f3201400700 1 -- 172.21.15.43:0/3200641748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561414871d30 msgr2=0x561414876040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.229+0000 7f3201400700 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561414871d30 0x561414876040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.229+0000 7f3201400700 1 -- 172.21.15.43:0/3200641748 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614146d98d0 con 0x56141486bdb0 2024-05-27T07:34:32.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.229+0000 7f3201e00700 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148755c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:34:32.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.230+0000 7f3201400700 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x561414875b00 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f31f00098e0 tx=0x7f31f0003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9babdce3f1b8f121 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.230+0000 7f31e3e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:34:32.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.230+0000 7f31e3e00700 1 -- 172.21.15.43:0/3200641748 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f0003bd0 con 0x56141486bdb0 2024-05-27T07:34:32.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.231+0000 7f31e3e00700 1 -- 172.21.15.43:0/3200641748 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f31f0003d70 con 0x56141486bdb0 2024-05-27T07:34:32.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.231+0000 7f32178dcb80 1 -- 172.21.15.43:0/3200641748 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31f40a9330 con 0x56141486bdb0 2024-05-27T07:34:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.231+0000 7f31e3e00700 1 -- 172.21.15.43:0/3200641748 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f31f0031050 con 0x56141486bdb0 2024-05-27T07:34:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.231+0000 7f32178dcb80 10 client.18141 fetch_fsmap learned FSMap version 139 2024-05-27T07:34:32.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.231+0000 7f32178dcb80 1 -- 172.21.15.43:0/3200641748 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x56141475a060 con 0x56141486bdb0 2024-05-27T07:34:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.232+0000 7f31e3e00700 1 -- 172.21.15.43:0/3200641748 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31f40a9330 con 0x56141486bdb0 2024-05-27T07:34:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.232+0000 7f31e3e00700 1 -- 172.21.15.43:0/3200641748 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f31f0023860 con 0x56141486bdb0 2024-05-27T07:34:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.232+0000 7f32178dcb80 10 client.18141 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:34:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.232+0000 7f32178dcb80 10 client.18141 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:34:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.232+0000 7f32178dcb80 20 client.18141 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:32.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.232+0000 7f32178dcb80 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561414896b70 0x561414899000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.232+0000 7f32178dcb80 4 client.18141 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.232+0000 7f32178dcb80 1 -- 172.21.15.43:0/3200641748 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56141479bd90 con 0x561414896b70 2024-05-27T07:34:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.233+0000 7f3201e00700 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561414896b70 0x561414899000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.234+0000 7f3201e00700 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561414896b70 0x561414899000 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e816c752be98958c server_cookie=a9d7038c1efa65df in_seq=0 out_seq=0 2024-05-27T07:34:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.234+0000 7f31e3e00700 10 client.18141 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:34:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.235+0000 7f31e3e00700 1 -- 172.21.15.43:0/3200641748 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56141479bd90 con 0x561414896b70 2024-05-27T07:34:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.235+0000 7f31e3e00700 10 client.18141 handle_command_reply: tid=0 2024-05-27T07:34:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.236+0000 7f32178dcb80 1 client.18141 shutdown 2024-05-27T07:34:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.236+0000 7f32178dcb80 1 -- 172.21.15.43:0/3200641748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 msgr2=0x561414875b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.236+0000 7f32178dcb80 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x561414875b00 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f31f00098e0 tx=0x7f31f0003680 comp rx=0 tx=0).stop 2024-05-27T07:34:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.237+0000 7f32178dcb80 1 -- 172.21.15.43:0/3200641748 shutdown_connections 2024-05-27T07:34:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.237+0000 7f32178dcb80 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561414896b70 0x561414899000 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.237+0000 7f32178dcb80 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561414871d30 0x561414876040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.237+0000 7f32178dcb80 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5614146ece40 0x5614148755c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.237+0000 7f32178dcb80 1 --2- 172.21.15.43:0/3200641748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x561414875b00 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.237+0000 7f32178dcb80 1 -- 172.21.15.43:0/3200641748 >> 172.21.15.43:0/3200641748 conn(0x56141474cfc0 msgr2=0x561414866580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.237+0000 7f31e3e00700 0 client.18141 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:34:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.237+0000 7f32178dcb80 1 -- 172.21.15.43:0/3200641748 shutdown_connections 2024-05-27T07:34:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.237+0000 7f32178dcb80 1 -- 172.21.15.43:0/3200641748 wait complete. 2024-05-27T07:34:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.238+0000 7f32178dcb80 20 client.18141 trim_cache size 0 max 16384 2024-05-27T07:34:32.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.265+0000 7f32178dcb80 1 Processor -- start 2024-05-27T07:34:32.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.265+0000 7f32178dcb80 1 -- start start 2024-05-27T07:34:32.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.265+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 0x56141486b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.265+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x5614148753a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.265+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 0x5614148758e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.266+0000 7f32178dcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32041ae7d0 con 0x56141486bdb0 2024-05-27T07:34:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.266+0000 7f32178dcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f32041122b0 con 0x5614146ece40 2024-05-27T07:34:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.266+0000 7f32178dcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f32041ae120 con 0x561414871d30 2024-05-27T07:34:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.266+0000 7f3201e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 0x56141486b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.266+0000 7f3202800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 0x5614148758e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.266+0000 7f3201400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x5614148753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.266+0000 7f3202800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 0x5614148758e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:46322/0 (socket says 172.21.15.43:46322) 2024-05-27T07:34:32.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.266+0000 7f3202800700 1 -- 172.21.15.43:0/4116201750 learned_addr learned my addr 172.21.15.43:0/4116201750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:32.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.267+0000 7f3201400700 1 -- 172.21.15.43:0/4116201750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 msgr2=0x5614148758e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.267+0000 7f3201400700 1 --2- 172.21.15.43:0/4116201750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 0x5614148758e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.267+0000 7f3201400700 1 -- 172.21.15.43:0/4116201750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 msgr2=0x56141486b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.267+0000 7f3201400700 1 --2- 172.21.15.43:0/4116201750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 0x56141486b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.267+0000 7f3201400700 1 -- 172.21.15.43:0/4116201750 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561414862d00 con 0x56141486bdb0 2024-05-27T07:34:32.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.267+0000 7f3201400700 1 --2- 172.21.15.43:0/4116201750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x5614148753a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f31f000ee50 tx=0x7f31f0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e4a6507d33a0fb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.268+0000 7f31eea00700 1 -- 172.21.15.43:0/4116201750 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f0039070 con 0x56141486bdb0 2024-05-27T07:34:32.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.268+0000 7f31eea00700 1 -- 172.21.15.43:0/4116201750 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31f0028070 con 0x56141486bdb0 2024-05-27T07:34:32.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.268+0000 7f31eea00700 1 -- 172.21.15.43:0/4116201750 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f000b9d0 con 0x56141486bdb0 2024-05-27T07:34:32.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.269+0000 7f32178dcb80 1 -- 172.21.15.43:0/4116201750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 msgr2=0x5614148753a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.269+0000 7f32178dcb80 1 --2- 172.21.15.43:0/4116201750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x5614148753a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f31f000ee50 tx=0x7f31f0005fd0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.270+0000 7f32178dcb80 1 -- 172.21.15.43:0/4116201750 shutdown_connections 2024-05-27T07:34:32.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.270+0000 7f32178dcb80 1 --2- 172.21.15.43:0/4116201750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 0x5614148758e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.270+0000 7f32178dcb80 1 --2- 172.21.15.43:0/4116201750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 0x56141486b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.270+0000 7f32178dcb80 1 --2- 172.21.15.43:0/4116201750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x5614148753a0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.270+0000 7f32178dcb80 1 -- 172.21.15.43:0/4116201750 >> 172.21.15.43:0/4116201750 conn(0x56141474cfc0 msgr2=0x561414873eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.270+0000 7f32178dcb80 1 -- 172.21.15.43:0/4116201750 shutdown_connections 2024-05-27T07:34:32.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.271+0000 7f32178dcb80 1 -- 172.21.15.43:0/4116201750 wait complete. 2024-05-27T07:34:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.271+0000 7f32178dcb80 1 -- start start 2024-05-27T07:34:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.271+0000 7f32178dcb80 1 Processor -- start 2024-05-27T07:34:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.272+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 0x56141489ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.272+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x56141488ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.272+0000 7f32178dcb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 0x56141488ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.272+0000 7f32178dcb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32040512b0 con 0x56141486bdb0 2024-05-27T07:34:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.272+0000 7f32178dcb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3204112980 con 0x5614146ece40 2024-05-27T07:34:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.272+0000 7f32178dcb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5614148745c0 con 0x561414871d30 2024-05-27T07:34:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.272+0000 7f32178dcb80 10 client.0 _ll_register_callbacks cb 0x561414733c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.273+0000 7f3202800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 0x56141488ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.273+0000 7f3202800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 0x56141488ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:46332/0 (socket says 172.21.15.43:46332) 2024-05-27T07:34:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.273+0000 7f3202800700 1 -- 172.21.15.43:0/990517696 learned_addr learned my addr 172.21.15.43:0/990517696 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.273+0000 7f3201e00700 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 0x56141489ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.273+0000 7f3201400700 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x56141488ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.274+0000 7f3201e00700 1 -- 172.21.15.43:0/990517696 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 msgr2=0x56141488ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.274+0000 7f3201e00700 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 0x56141488ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.274+0000 7f3201e00700 1 -- 172.21.15.43:0/990517696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 msgr2=0x56141488ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:32.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.274+0000 7f3201e00700 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x56141488ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.274+0000 7f3201e00700 1 -- 172.21.15.43:0/990517696 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56141489d7f0 con 0x5614146ece40 2024-05-27T07:34:32.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.274+0000 7f3201400700 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x56141488ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:34:32.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.275+0000 7f3201e00700 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 0x56141489ef30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x561414874e40 tx=0x7f31f4005910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f66e16171123f68f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:32.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.275+0000 7f31e3e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:34:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.275+0000 7f31e3e00700 1 -- 172.21.15.43:0/990517696 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f40a9330 con 0x5614146ece40 2024-05-27T07:34:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.276+0000 7f31e3e00700 1 -- 172.21.15.43:0/990517696 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f31f40a9590 con 0x5614146ece40 2024-05-27T07:34:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.276+0000 7f32178dcb80 1 -- 172.21.15.43:0/990517696 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31f0028070 con 0x5614146ece40 2024-05-27T07:34:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.276+0000 7f31e3e00700 1 -- 172.21.15.43:0/990517696 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f31f40a9790 con 0x5614146ece40 2024-05-27T07:34:32.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.276+0000 7f32178dcb80 10 client.37511 fetch_fsmap learned FSMap version 139 2024-05-27T07:34:32.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.276+0000 7f32178dcb80 1 -- 172.21.15.43:0/990517696 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x561414900870 con 0x5614146ece40 2024-05-27T07:34:32.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.277+0000 7f31e3e00700 1 -- 172.21.15.43:0/990517696 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31f0028070 con 0x5614146ece40 2024-05-27T07:34:32.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.277+0000 7f31e3e00700 1 -- 172.21.15.43:0/990517696 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f31f40b1d00 con 0x5614146ece40 2024-05-27T07:34:32.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.277+0000 7f32178dcb80 10 client.37511 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:34:32.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.277+0000 7f32178dcb80 10 client.37511 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:34:32.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.277+0000 7f32178dcb80 20 client.37511 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:32.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.277+0000 7f32178dcb80 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561414900f60 0x561414903380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:32.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.277+0000 7f32178dcb80 4 client.37511 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:34:32.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.277+0000 7f32178dcb80 1 -- 172.21.15.43:0/990517696 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561414795570 con 0x561414900f60 2024-05-27T07:34:32.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.278+0000 7f3201400700 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561414900f60 0x561414903380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:32.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.280+0000 7f3201400700 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561414900f60 0x561414903380 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=37d450a1c8a2f165 server_cookie=f9ebbe75609f00f5 in_seq=0 out_seq=0 2024-05-27T07:34:32.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.280+0000 7f31e3e00700 10 client.37511 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:34:32.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.281+0000 7f31e3e00700 1 -- 172.21.15.43:0/990517696 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561414795570 con 0x561414900f60 2024-05-27T07:34:32.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.281+0000 7f31e3e00700 10 client.37511 handle_command_reply: tid=0 2024-05-27T07:34:32.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.281+0000 7f32178dcb80 1 client.37511 shutdown 2024-05-27T07:34:32.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.282+0000 7f32178dcb80 1 -- 172.21.15.43:0/990517696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 msgr2=0x56141489ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.282+0000 7f32178dcb80 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 0x56141489ef30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x561414874e40 tx=0x7f31f4005910 comp rx=0 tx=0).stop 2024-05-27T07:34:32.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.283+0000 7f32178dcb80 1 -- 172.21.15.43:0/990517696 shutdown_connections 2024-05-27T07:34:32.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.283+0000 7f32178dcb80 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x561414900f60 0x561414903380 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.283+0000 7f32178dcb80 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561414871d30 0x56141488ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.283+0000 7f32178dcb80 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614146ece40 0x56141489ef30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.283+0000 7f32178dcb80 1 --2- 172.21.15.43:0/990517696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56141486bdb0 0x56141488ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.283+0000 7f32178dcb80 1 -- 172.21.15.43:0/990517696 >> 172.21.15.43:0/990517696 conn(0x56141474cfc0 msgr2=0x56141486f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.284+0000 7f31e3e00700 0 client.37511 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:34:32.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.284+0000 7f32178dcb80 1 -- 172.21.15.43:0/990517696 shutdown_connections 2024-05-27T07:34:32.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.285+0000 7f32178dcb80 1 -- 172.21.15.43:0/990517696 wait complete. 2024-05-27T07:34:32.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.285+0000 7f32178dcb80 20 client.37511 trim_cache size 0 max 16384 2024-05-27T07:34:32.281 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:34:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.288+0000 7f3208e00700 1 -- 172.21.15.43:0/62364383 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f31e80b2a70 msgr2=0x7f31e80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.288+0000 7f3208e00700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f31e80b2a70 0x7f31e80b4f80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f31fc003fd0 tx=0x7f31fc003a20 comp rx=0 tx=0).stop 2024-05-27T07:34:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.288+0000 7f3208e00700 1 -- 172.21.15.43:0/62364383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3204103240 msgr2=0x7f32041a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.288+0000 7f3208e00700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3204103240 0x7f32041a1300 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31f400d950 tx=0x7f31f400dc60 comp rx=0 tx=0).stop 2024-05-27T07:34:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.288+0000 7f3208e00700 1 -- 172.21.15.43:0/62364383 shutdown_connections 2024-05-27T07:34:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.288+0000 7f3208e00700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f31e80b2a70 0x7f31e80b4f80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.288+0000 7f3208e00700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3204108690 0x7f32041a5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.288+0000 7f3208e00700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3204103240 0x7f32041a1300 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.288+0000 7f3208e00700 1 --2- 172.21.15.43:0/62364383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3204103b90 0x7f32041a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.288+0000 7f3208e00700 1 -- 172.21.15.43:0/62364383 >> 172.21.15.43:0/62364383 conn(0x7f32040fedb0 msgr2=0x7f3204106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.289+0000 7f3208e00700 1 -- 172.21.15.43:0/62364383 shutdown_connections 2024-05-27T07:34:32.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:32.289+0000 7f3208e00700 1 -- 172.21.15.43:0/62364383 wait complete. 2024-05-27T07:34:32.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 43474 5.88 MB/sec execute 273 sec latency 80.550 ms 2024-05-27T07:34:32.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:32 smithi132 ceph-mon[35283]: pgmap v693: 129 pgs: 129 active+clean; 986 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 4.6 MiB/s wr, 222 op/s 2024-05-27T07:34:32.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:32 smithi043 ceph-mon[31531]: pgmap v693: 129 pgs: 129 active+clean; 986 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 4.6 MiB/s wr, 222 op/s 2024-05-27T07:34:32.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:32 smithi179 ceph-mon[35888]: pgmap v693: 129 pgs: 129 active+clean; 986 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 4.6 MiB/s wr, 222 op/s 2024-05-27T07:34:33.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 43588 5.87 MB/sec execute 274 sec latency 98.873 ms 2024-05-27T07:34:33.888 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:34:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:34.806 INFO:tasks.workunit.client.0.smithi043.stdout: 10 43685 5.88 MB/sec execute 275 sec latency 82.132 ms 2024-05-27T07:34:34.844 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:34 smithi043 ceph-mon[31531]: pgmap v694: 129 pgs: 129 active+clean; 986 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 150 op/s 2024-05-27T07:34:34.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:34 smithi132 ceph-mon[35283]: pgmap v694: 129 pgs: 129 active+clean; 986 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 150 op/s 2024-05-27T07:34:34.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:34 smithi179 ceph-mon[35888]: pgmap v694: 129 pgs: 129 active+clean; 986 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 150 op/s 2024-05-27T07:34:35.807 INFO:tasks.workunit.client.0.smithi043.stdout: 10 43780 5.87 MB/sec execute 276 sec latency 99.744 ms 2024-05-27T07:34:36.807 INFO:tasks.workunit.client.0.smithi043.stdout: 10 43911 5.89 MB/sec execute 277 sec latency 84.362 ms 2024-05-27T07:34:36.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:36 smithi132 ceph-mon[35283]: pgmap v695: 129 pgs: 129 active+clean; 986 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 150 op/s 2024-05-27T07:34:36.877 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:36 smithi043 ceph-mon[31531]: pgmap v695: 129 pgs: 129 active+clean; 986 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 150 op/s 2024-05-27T07:34:36.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:36 smithi179 ceph-mon[35888]: pgmap v695: 129 pgs: 129 active+clean; 986 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 150 op/s 2024-05-27T07:34:37.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.239+0000 7f1e9e000700 1 -- 172.21.15.43:0/3539895783 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e98110c20 msgr2=0x7f1e98113040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:37.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.239+0000 7f1e9e000700 1 --2- 172.21.15.43:0/3539895783 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e98110c20 0x7f1e98113040 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000b0a0 tx=0x7f1e9000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.239+0000 7f1e9e000700 1 -- 172.21.15.43:0/3539895783 shutdown_connections 2024-05-27T07:34:37.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.239+0000 7f1e9e000700 1 --2- 172.21.15.43:0/3539895783 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e98110c20 0x7f1e98113040 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.239+0000 7f1e9e000700 1 --2- 172.21.15.43:0/3539895783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e98103070 0x7f1e98103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.239+0000 7f1e9e000700 1 --2- 172.21.15.43:0/3539895783 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e98103a50 0x7f1e981106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.239+0000 7f1e9e000700 1 -- 172.21.15.43:0/3539895783 >> 172.21.15.43:0/3539895783 conn(0x7f1e980fedb0 msgr2=0x7f1e981011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:37.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.240+0000 7f1e9e000700 1 -- 172.21.15.43:0/3539895783 shutdown_connections 2024-05-27T07:34:37.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.240+0000 7f1e9e000700 1 -- 172.21.15.43:0/3539895783 wait complete. 2024-05-27T07:34:37.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.241+0000 7f1e9e000700 1 Processor -- start 2024-05-27T07:34:37.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.241+0000 7f1e9e000700 1 -- start start 2024-05-27T07:34:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.242+0000 7f1e9e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e98103070 0x7f1e9819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.242+0000 7f1e9e000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e98103a50 0x7f1e9819d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.242+0000 7f1e9e000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e98110c20 0x7f1e981a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.242+0000 7f1e9e000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1e98116580 con 0x7f1e98103070 2024-05-27T07:34:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.242+0000 7f1e9e000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1e98116400 con 0x7f1e98103a50 2024-05-27T07:34:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.242+0000 7f1e9e000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1e98116700 con 0x7f1e98110c20 2024-05-27T07:34:37.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.243+0000 7f1e97400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e98103070 0x7f1e9819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.243+0000 7f1e97e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e98110c20 0x7f1e981a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.243+0000 7f1e97400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e98103070 0x7f1e9819ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50202/0 (socket says 172.21.15.43:50202) 2024-05-27T07:34:37.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.243+0000 7f1e97400700 1 -- 172.21.15.43:0/1204457586 learned_addr learned my addr 172.21.15.43:0/1204457586 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:37.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.243+0000 7f1e96a00700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e98103a50 0x7f1e9819d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.243+0000 7f1e97400700 1 -- 172.21.15.43:0/1204457586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e98110c20 msgr2=0x7f1e981a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:37.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.243+0000 7f1e97400700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e98110c20 0x7f1e981a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.243+0000 7f1e97400700 1 -- 172.21.15.43:0/1204457586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e98103a50 msgr2=0x7f1e9819d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:37.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.244+0000 7f1e97400700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e98103a50 0x7f1e9819d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.244+0000 7f1e97400700 1 -- 172.21.15.43:0/1204457586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e981a1da0 con 0x7f1e98103070 2024-05-27T07:34:37.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.244+0000 7f1e97400700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e98103070 0x7f1e9819ced0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f1e8800e9a0 tx=0x7f1e8800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:37.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.245+0000 7f1e8fe00700 1 -- 172.21.15.43:0/1204457586 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8800ca10 con 0x7f1e98103070 2024-05-27T07:34:37.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.245+0000 7f1e9e000700 1 -- 172.21.15.43:0/1204457586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e981a2030 con 0x7f1e98103070 2024-05-27T07:34:37.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.245+0000 7f1e9e000700 1 -- 172.21.15.43:0/1204457586 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e981a99b0 con 0x7f1e98103070 2024-05-27T07:34:37.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.245+0000 7f1e8fe00700 1 -- 172.21.15.43:0/1204457586 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e8800cbb0 con 0x7f1e98103070 2024-05-27T07:34:37.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.245+0000 7f1e8fe00700 1 -- 172.21.15.43:0/1204457586 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e88026060 con 0x7f1e98103070 2024-05-27T07:34:37.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.252+0000 7f1e8fe00700 1 -- 172.21.15.43:0/1204457586 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1e88010430 con 0x7f1e98103070 2024-05-27T07:34:37.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.254+0000 7f1e8fe00700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e7c0b2bf0 0x7f1e7c0b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.255+0000 7f1e8fe00700 1 -- 172.21.15.43:0/1204457586 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f1e88018420 con 0x7f1e98103070 2024-05-27T07:34:37.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.255+0000 7f1e96a00700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e7c0b2bf0 0x7f1e7c0b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.256+0000 7f1e96a00700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e7c0b2bf0 0x7f1e7c0b5100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1e98112330 tx=0x7f1e84009260 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:37.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.257+0000 7f1eacc1fb80 1 Processor -- start 2024-05-27T07:34:37.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.257+0000 7f1eacc1fb80 1 -- start start 2024-05-27T07:34:37.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.258+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546c639f0 0x561546cf6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.258+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546cf6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.258+0000 7f1e97400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546c639f0 0x561546cf6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.259+0000 7f1e97400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546c639f0 0x561546cf6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:41144/0 (socket says 172.21.15.43:41144) 2024-05-27T07:34:37.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.259+0000 7f1e97400700 1 -- 172.21.15.43:0/2295078309 learned_addr learned my addr 172.21.15.43:0/2295078309 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:37.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.259+0000 7f1e96a00700 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546cf6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.259+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546cf6e70 0x561546d03110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.259+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/2295078309 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561546c24e40 con 0x561546b77e40 2024-05-27T07:34:37.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.259+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/2295078309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561546c149e0 con 0x561546cf6e70 2024-05-27T07:34:37.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.259+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/2295078309 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561546c18a20 con 0x561546c639f0 2024-05-27T07:34:37.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.259+0000 7f1e97e00700 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546cf6e70 0x561546d03110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.260+0000 7f1e96a00700 1 -- 172.21.15.43:0/2295078309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546c639f0 msgr2=0x561546cf6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:37.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.260+0000 7f1e96a00700 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546c639f0 0x561546cf6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.260+0000 7f1e96a00700 1 -- 172.21.15.43:0/2295078309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546cf6e70 msgr2=0x561546d03110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:37.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.260+0000 7f1e96a00700 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546cf6e70 0x561546d03110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.260+0000 7f1e96a00700 1 -- 172.21.15.43:0/2295078309 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561546bdd420 con 0x561546b77e40 2024-05-27T07:34:37.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.260+0000 7f1e97e00700 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546cf6e70 0x561546d03110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:34:37.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.261+0000 7f1e96a00700 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546cf6870 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f1e9806c6e0 tx=0x7f1e84014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af8749f6f80b884 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:37.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.262+0000 7f1e8cc00700 1 -- 172.21.15.43:0/2295078309 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400ea10 con 0x561546b77e40 2024-05-27T07:34:37.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.262+0000 7f1e8cc00700 1 -- 172.21.15.43:0/2295078309 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e8400ebb0 con 0x561546b77e40 2024-05-27T07:34:37.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.262+0000 7f1e8cc00700 1 -- 172.21.15.43:0/2295078309 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8401d410 con 0x561546b77e40 2024-05-27T07:34:37.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.263+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/2295078309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 msgr2=0x561546cf6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:37.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.263+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546cf6870 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f1e9806c6e0 tx=0x7f1e84014000 comp rx=0 tx=0).stop 2024-05-27T07:34:37.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.264+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/2295078309 shutdown_connections 2024-05-27T07:34:37.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.264+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546cf6e70 0x561546d03110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.264+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546c639f0 0x561546cf6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.264+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/2295078309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546cf6870 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.264+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/2295078309 >> 172.21.15.43:0/2295078309 conn(0x561546c11390 msgr2=0x561546cf24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:37.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.264+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/2295078309 shutdown_connections 2024-05-27T07:34:37.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.264+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/2295078309 wait complete. 2024-05-27T07:34:37.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.265+0000 7f1eacc1fb80 1 -- start start 2024-05-27T07:34:37.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.265+0000 7f1eacc1fb80 1 Processor -- start 2024-05-27T07:34:37.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546d0f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546c639f0 0x561546d0f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546cf6e70 0x561546d09740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1eacc1fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561546b1afa0 con 0x561546c639f0 2024-05-27T07:34:37.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1eacc1fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x561546bb5760 con 0x561546b77e40 2024-05-27T07:34:37.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1eacc1fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561546d03b80 con 0x561546cf6e70 2024-05-27T07:34:37.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1eacc1fb80 10 client.0 _ll_register_callbacks cb 0x561546bbec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:37.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1e97400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546d0f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1e97400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546d0f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:35870/0 (socket says 172.21.15.43:35870) 2024-05-27T07:34:37.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1e97400700 1 -- 172.21.15.43:0/1767633496 learned_addr learned my addr 172.21.15.43:0/1767633496 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:37.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.266+0000 7f1e97e00700 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546cf6e70 0x561546d09740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.267+0000 7f1e97400700 1 -- 172.21.15.43:0/1767633496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546cf6e70 msgr2=0x561546d09740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:37.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.267+0000 7f1e97400700 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546cf6e70 0x561546d09740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.267+0000 7f1e97400700 1 -- 172.21.15.43:0/1767633496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546c639f0 msgr2=0x561546d0f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:37.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.267+0000 7f1e96a00700 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546c639f0 0x561546d0f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.267+0000 7f1e97400700 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546c639f0 0x561546d0f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.267+0000 7f1e97400700 1 -- 172.21.15.43:0/1767633496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561546b648d0 con 0x561546b77e40 2024-05-27T07:34:37.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.267+0000 7f1e96a00700 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546c639f0 0x561546d0f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:37.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.268+0000 7f1e97400700 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546d0f340 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1e88000c00 tx=0x7f1e880aecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=256666f7364ad20f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:37.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.268+0000 7f1e81600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:34:37.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.268+0000 7f1e81600700 1 -- 172.21.15.43:0/1767633496 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e880a5ff0 con 0x561546b77e40 2024-05-27T07:34:37.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.268+0000 7f1e81600700 1 -- 172.21.15.43:0/1767633496 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f1e880a54f0 con 0x561546b77e40 2024-05-27T07:34:37.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.268+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/1767633496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e8400ebb0 con 0x561546b77e40 2024-05-27T07:34:37.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.269+0000 7f1e81600700 1 -- 172.21.15.43:0/1767633496 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f1e880a6190 con 0x561546b77e40 2024-05-27T07:34:37.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.269+0000 7f1eacc1fb80 10 client.37523 fetch_fsmap learned FSMap version 139 2024-05-27T07:34:37.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.269+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/1767633496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x561546be5060 con 0x561546b77e40 2024-05-27T07:34:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.269+0000 7f1e81600700 1 -- 172.21.15.43:0/1767633496 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e8400ebb0 con 0x561546b77e40 2024-05-27T07:34:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.269+0000 7f1e81600700 1 -- 172.21.15.43:0/1767633496 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f1e880a9a60 con 0x561546b77e40 2024-05-27T07:34:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.269+0000 7f1eacc1fb80 10 client.37523 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:34:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.269+0000 7f1eacc1fb80 10 client.37523 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:34:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.269+0000 7f1eacc1fb80 20 client.37523 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.269+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x561546d19780 0x561546d1bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.270+0000 7f1eacc1fb80 4 client.37523 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:37.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.270+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/1767633496 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561546c26d90 con 0x561546d19780 2024-05-27T07:34:37.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.270+0000 7f1e96a00700 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x561546d19780 0x561546d1bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.271+0000 7f1e96a00700 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x561546d19780 0x561546d1bc10 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7f26c06a15abe8f5 server_cookie=a1984c2fa505072d in_seq=0 out_seq=0 2024-05-27T07:34:37.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.271+0000 7f1e81600700 10 client.37523 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:34:37.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1e81600700 1 -- 172.21.15.43:0/1767633496 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561546c26d90 con 0x561546d19780 2024-05-27T07:34:37.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1e81600700 10 client.37523 handle_command_reply: tid=0 2024-05-27T07:34:37.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1eacc1fb80 1 client.37523 shutdown 2024-05-27T07:34:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/1767633496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 msgr2=0x561546d0f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546d0f340 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1e88000c00 tx=0x7f1e880aecd0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/1767633496 shutdown_connections 2024-05-27T07:34:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546cf6e70 0x561546d09740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x561546d19780 0x561546d1bc10 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546d0f340 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/1767633496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546c639f0 0x561546d0f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/1767633496 >> 172.21.15.43:0/1767633496 conn(0x561546c11390 msgr2=0x561546cf6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.274+0000 7f1e81600700 0 client.37523 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:34:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.275+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/1767633496 shutdown_connections 2024-05-27T07:34:37.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.275+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/1767633496 wait complete. 2024-05-27T07:34:37.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.275+0000 7f1eacc1fb80 20 client.37523 trim_cache size 0 max 16384 2024-05-27T07:34:37.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.297+0000 7f1eacc1fb80 1 Processor -- start 2024-05-27T07:34:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.297+0000 7f1eacc1fb80 1 -- start start 2024-05-27T07:34:37.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.297+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546cf6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.297+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546c639f0 0x561546d213d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.298+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546cf6e70 0x561546d21910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.298+0000 7f1eacc1fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7800f240 con 0x561546cf6e70 2024-05-27T07:34:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.298+0000 7f1eacc1fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7800d6a0 con 0x561546b77e40 2024-05-27T07:34:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.298+0000 7f1eacc1fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7800eb70 con 0x561546c639f0 2024-05-27T07:34:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.298+0000 7f1e97400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546cf6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.298+0000 7f1e97400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546cf6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:35876/0 (socket says 172.21.15.43:35876) 2024-05-27T07:34:37.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.299+0000 7f1e97400700 1 -- 172.21.15.43:0/3627826745 learned_addr learned my addr 172.21.15.43:0/3627826745 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:37.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.299+0000 7f1e97400700 1 -- 172.21.15.43:0/3627826745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546c639f0 msgr2=0x561546d213d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:34:37.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.299+0000 7f1e97400700 1 --2- 172.21.15.43:0/3627826745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546c639f0 0x561546d213d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.299+0000 7f1e97400700 1 -- 172.21.15.43:0/3627826745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546cf6e70 msgr2=0x561546d21910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.300+0000 7f1e97400700 1 --2- 172.21.15.43:0/3627826745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546cf6e70 0x561546d21910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.300+0000 7f1e97400700 1 -- 172.21.15.43:0/3627826745 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561546cf3810 con 0x561546b77e40 2024-05-27T07:34:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.300+0000 7f1e97400700 1 --2- 172.21.15.43:0/3627826745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546cf6830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1e880a66a0 tx=0x7f1e88009350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdb008fa697d97ac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:37.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.301+0000 7f1e83e00700 1 -- 172.21.15.43:0/3627826745 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e880a6360 con 0x561546b77e40 2024-05-27T07:34:37.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.302+0000 7f1e83e00700 1 -- 172.21.15.43:0/3627826745 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e88010e50 con 0x561546b77e40 2024-05-27T07:34:37.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.302+0000 7f1e83e00700 1 -- 172.21.15.43:0/3627826745 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e880a6500 con 0x561546b77e40 2024-05-27T07:34:37.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.304+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/3627826745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 msgr2=0x561546cf6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:37.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.304+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/3627826745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546cf6830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1e880a66a0 tx=0x7f1e88009350 comp rx=0 tx=0).stop 2024-05-27T07:34:37.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.304+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/3627826745 shutdown_connections 2024-05-27T07:34:37.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.304+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/3627826745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546cf6e70 0x561546d21910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.304+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/3627826745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546b77e40 0x561546cf6830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.304+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/3627826745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546c639f0 0x561546d213d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.304+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/3627826745 >> 172.21.15.43:0/3627826745 conn(0x561546c11390 msgr2=0x561546cf6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:37.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.304+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/3627826745 shutdown_connections 2024-05-27T07:34:37.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.305+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/3627826745 wait complete. 2024-05-27T07:34:37.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.306+0000 7f1eacc1fb80 1 -- start start 2024-05-27T07:34:37.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.306+0000 7f1eacc1fb80 1 Processor -- start 2024-05-27T07:34:37.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.308+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546d10830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.308+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546c639f0 0x561546d10d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.308+0000 7f1eacc1fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546cf6e70 0x561546d82290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.308+0000 7f1eacc1fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7800e4a0 con 0x561546b77e40 2024-05-27T07:34:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.308+0000 7f1eacc1fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7800dd70 con 0x561546c639f0 2024-05-27T07:34:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.308+0000 7f1eacc1fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561546d115f0 con 0x561546cf6e70 2024-05-27T07:34:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.308+0000 7f1eacc1fb80 10 client.0 _ll_register_callbacks cb 0x561546bbec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.308+0000 7f1e97400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546d10830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.308+0000 7f1e97400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546d10830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50236/0 (socket says 172.21.15.43:50236) 2024-05-27T07:34:37.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.308+0000 7f1e97400700 1 -- 172.21.15.43:0/4176875318 learned_addr learned my addr 172.21.15.43:0/4176875318 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:37.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.309+0000 7f1e96a00700 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546c639f0 0x561546d10d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.310+0000 7f1e97400700 1 -- 172.21.15.43:0/4176875318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546cf6e70 msgr2=0x561546d82290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.310+0000 7f1e97400700 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546cf6e70 0x561546d82290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.310+0000 7f1e97400700 1 -- 172.21.15.43:0/4176875318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546c639f0 msgr2=0x561546d10d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.310+0000 7f1e97400700 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546c639f0 0x561546d10d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.310+0000 7f1e97400700 1 -- 172.21.15.43:0/4176875318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561546d827d0 con 0x561546b77e40 2024-05-27T07:34:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.310+0000 7f1e96a00700 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546c639f0 0x561546d10d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.311+0000 7f1e97400700 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546d10830 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f1e880a9120 tx=0x7f1e880056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed66d5163f0a8f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.311+0000 7f1e81600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:34:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.311+0000 7f1e81600700 1 -- 172.21.15.43:0/4176875318 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e880a66d0 con 0x561546b77e40 2024-05-27T07:34:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.312+0000 7f1e81600700 1 -- 172.21.15.43:0/4176875318 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f1e88088020 con 0x561546b77e40 2024-05-27T07:34:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.312+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/4176875318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e88010e50 con 0x561546b77e40 2024-05-27T07:34:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.312+0000 7f1e81600700 1 -- 172.21.15.43:0/4176875318 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f1e880a66d0 con 0x561546b77e40 2024-05-27T07:34:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.312+0000 7f1eacc1fb80 10 client.18159 fetch_fsmap learned FSMap version 139 2024-05-27T07:34:37.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.312+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/4176875318 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x561546d83a00 con 0x561546b77e40 2024-05-27T07:34:37.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.312+0000 7f1e81600700 1 -- 172.21.15.43:0/4176875318 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e88010e50 con 0x561546b77e40 2024-05-27T07:34:37.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.312+0000 7f1e81600700 1 -- 172.21.15.43:0/4176875318 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 139) ==== 4195+0+0 (secure 0 0 0) 0x7f1e880c9020 con 0x561546b77e40 2024-05-27T07:34:37.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.314+0000 7f1eacc1fb80 10 client.18159 fetch_fsmap finished waiting for FSMap version 139 2024-05-27T07:34:37.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.314+0000 7f1eacc1fb80 10 client.18159 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:34:37.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.314+0000 7f1eacc1fb80 20 client.18159 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:37.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.314+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x561546d83ec0 0x561546d862e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:37.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.314+0000 7f1eacc1fb80 4 client.18159 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:34:37.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.314+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/4176875318 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561546c20570 con 0x561546d83ec0 2024-05-27T07:34:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.317+0000 7f1e96a00700 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x561546d83ec0 0x561546d862e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:37.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.319+0000 7f1e96a00700 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x561546d83ec0 0x561546d862e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d1d69badf9b67207 server_cookie=9035ccbe3754c863 in_seq=0 out_seq=0 2024-05-27T07:34:37.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.319+0000 7f1e81600700 10 client.18159 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:34:37.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.321+0000 7f1e81600700 1 -- 172.21.15.43:0/4176875318 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561546c20570 con 0x561546d83ec0 2024-05-27T07:34:37.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.321+0000 7f1e81600700 10 client.18159 handle_command_reply: tid=0 2024-05-27T07:34:37.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.321+0000 7f1eacc1fb80 1 client.18159 shutdown 2024-05-27T07:34:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.322+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/4176875318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 msgr2=0x561546d10830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.322+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546d10830 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f1e880a9120 tx=0x7f1e880056b0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.322+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/4176875318 shutdown_connections 2024-05-27T07:34:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.322+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561546cf6e70 0x561546d82290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.322+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561546b77e40 0x561546d10830 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.322+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x561546d83ec0 0x561546d862e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.322+0000 7f1eacc1fb80 1 --2- 172.21.15.43:0/4176875318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561546c639f0 0x561546d10d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.322+0000 7f1e81600700 0 client.18159 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:34:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.322+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/4176875318 >> 172.21.15.43:0/4176875318 conn(0x561546c11390 msgr2=0x561546cf7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.323+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/4176875318 shutdown_connections 2024-05-27T07:34:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.324+0000 7f1eacc1fb80 1 -- 172.21.15.43:0/4176875318 wait complete. 2024-05-27T07:34:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.324+0000 7f1eacc1fb80 20 client.18159 trim_cache size 0 max 16384 2024-05-27T07:34:37.320 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:34:37.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.326+0000 7f1e9e000700 1 -- 172.21.15.43:0/1204457586 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e7c0b2bf0 msgr2=0x7f1e7c0b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:37.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.326+0000 7f1e9e000700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e7c0b2bf0 0x7f1e7c0b5100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1e98112330 tx=0x7f1e84009260 comp rx=0 tx=0).stop 2024-05-27T07:34:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.326+0000 7f1e9e000700 1 -- 172.21.15.43:0/1204457586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e98103070 msgr2=0x7f1e9819ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.326+0000 7f1e9e000700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e98103070 0x7f1e9819ced0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f1e8800e9a0 tx=0x7f1e8800ecb0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.326+0000 7f1e9e000700 1 -- 172.21.15.43:0/1204457586 shutdown_connections 2024-05-27T07:34:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.326+0000 7f1e9e000700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e7c0b2bf0 0x7f1e7c0b5100 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.326+0000 7f1e9e000700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e98110c20 0x7f1e981a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.326+0000 7f1e9e000700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e98103070 0x7f1e9819ced0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.326+0000 7f1e9e000700 1 --2- 172.21.15.43:0/1204457586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e98103a50 0x7f1e9819d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.326+0000 7f1e9e000700 1 -- 172.21.15.43:0/1204457586 >> 172.21.15.43:0/1204457586 conn(0x7f1e980fedb0 msgr2=0x7f1e98112780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.327+0000 7f1e9e000700 1 -- 172.21.15.43:0/1204457586 shutdown_connections 2024-05-27T07:34:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:37.327+0000 7f1e9e000700 1 -- 172.21.15.43:0/1204457586 wait complete. 2024-05-27T07:34:37.807 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44014 5.88 MB/sec execute 278 sec latency 110.130 ms 2024-05-27T07:34:38.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:34:38.553 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:38 smithi043 ceph-mon[31531]: pgmap v696: 129 pgs: 129 active+clean; 999 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.1 MiB/s wr, 222 op/s 2024-05-27T07:34:38.807 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44144 5.89 MB/sec execute 279 sec latency 101.490 ms 2024-05-27T07:34:38.836 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:38.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:38 smithi132 ceph-mon[35283]: pgmap v696: 129 pgs: 129 active+clean; 999 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.1 MiB/s wr, 222 op/s 2024-05-27T07:34:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:38 smithi179 ceph-mon[35888]: pgmap v696: 129 pgs: 129 active+clean; 999 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.1 MiB/s wr, 222 op/s 2024-05-27T07:34:39.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44251 5.89 MB/sec execute 280 sec latency 87.869 ms 2024-05-27T07:34:40.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44379 5.89 MB/sec execute 281 sec latency 90.801 ms 2024-05-27T07:34:40.844 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:40 smithi132 ceph-mon[35283]: pgmap v697: 129 pgs: 129 active+clean; 999 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 150 op/s 2024-05-27T07:34:40.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:40 smithi043 ceph-mon[31531]: pgmap v697: 129 pgs: 129 active+clean; 999 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 150 op/s 2024-05-27T07:34:40.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:40 smithi179 ceph-mon[35888]: pgmap v697: 129 pgs: 129 active+clean; 999 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 150 op/s 2024-05-27T07:34:41.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44508 5.90 MB/sec execute 282 sec latency 100.392 ms 2024-05-27T07:34:42.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44553 5.89 MB/sec execute 283 sec latency 459.930 ms 2024-05-27T07:34:42.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:42 smithi132 ceph-mon[35283]: pgmap v698: 129 pgs: 129 active+clean; 1006 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 5.5 MiB/s wr, 237 op/s 2024-05-27T07:34:42.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:42 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:42.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:42 smithi043 ceph-mon[31531]: pgmap v698: 129 pgs: 129 active+clean; 1006 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 5.5 MiB/s wr, 237 op/s 2024-05-27T07:34:42.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:42 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:42.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:42 smithi179 ceph-mon[35888]: pgmap v698: 129 pgs: 129 active+clean; 1006 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 5.5 MiB/s wr, 237 op/s 2024-05-27T07:34:42.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:42 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:43.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44590 5.87 MB/sec execute 284 sec latency 523.777 ms 2024-05-27T07:34:44.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44644 5.85 MB/sec execute 285 sec latency 423.776 ms 2024-05-27T07:34:44.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:44 smithi132 ceph-mon[35283]: pgmap v699: 129 pgs: 129 active+clean; 1006 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-05-27T07:34:44.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:44 smithi043 ceph-mon[31531]: pgmap v699: 129 pgs: 129 active+clean; 1006 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-05-27T07:34:44.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:44 smithi179 ceph-mon[35888]: pgmap v699: 129 pgs: 129 active+clean; 1006 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-05-27T07:34:45.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44763 5.86 MB/sec execute 286 sec latency 93.871 ms 2024-05-27T07:34:46.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.109+0000 7f1517200700 1 -- 172.21.15.43:0/3636538602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15101034b0 msgr2=0x7f15101038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.109+0000 7f1517200700 1 --2- 172.21.15.43:0/3636538602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15101034b0 0x7f15101038c0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f15000099c0 tx=0x7f1500009cd0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.109+0000 7f1517200700 1 -- 172.21.15.43:0/3636538602 shutdown_connections 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.109+0000 7f1517200700 1 --2- 172.21.15.43:0/3636538602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1510108730 0x7f1510110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.109+0000 7f1517200700 1 --2- 172.21.15.43:0/3636538602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f15101034b0 0x7f15101038c0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.109+0000 7f1517200700 1 --2- 172.21.15.43:0/3636538602 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1510107de0 0x7f15101081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.109+0000 7f1517200700 1 -- 172.21.15.43:0/3636538602 >> 172.21.15.43:0/3636538602 conn(0x7f15100fedb0 msgr2=0x7f15101011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.110+0000 7f1517200700 1 -- 172.21.15.43:0/3636538602 shutdown_connections 2024-05-27T07:34:46.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.110+0000 7f1517200700 1 -- 172.21.15.43:0/3636538602 wait complete. 2024-05-27T07:34:46.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.112+0000 7f1517200700 1 Processor -- start 2024-05-27T07:34:46.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.112+0000 7f1517200700 1 -- start start 2024-05-27T07:34:46.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.113+0000 7f1517200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15101034b0 0x7f151019ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.113+0000 7f1517200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1510107de0 0x7f151019b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.113+0000 7f1517200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1510108730 0x7f151019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.113+0000 7f1517200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1510113260 con 0x7f1510107de0 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.113+0000 7f1517200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f15101130e0 con 0x7f1510108730 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.113+0000 7f1517200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f15101133e0 con 0x7f15101034b0 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.114+0000 7f150fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15101034b0 0x7f151019ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.114+0000 7f150f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1510107de0 0x7f151019b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.114+0000 7f150f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1510107de0 0x7f151019b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59402/0 (socket says 172.21.15.43:59402) 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.114+0000 7f150f400700 1 -- 172.21.15.43:0/583064655 learned_addr learned my addr 172.21.15.43:0/583064655 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.114+0000 7f150fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15101034b0 0x7f151019ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:38428/0 (socket says 172.21.15.43:38428) 2024-05-27T07:34:46.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.114+0000 7f1515200700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1510108730 0x7f151019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.115+0000 7f150fe00700 1 -- 172.21.15.43:0/583064655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1510108730 msgr2=0x7f151019f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.115+0000 7f150fe00700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1510108730 0x7f151019f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.115+0000 7f150fe00700 1 -- 172.21.15.43:0/583064655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1510107de0 msgr2=0x7f151019b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.115+0000 7f150fe00700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1510107de0 0x7f151019b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.115+0000 7f150fe00700 1 -- 172.21.15.43:0/583064655 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151019fc50 con 0x7f15101034b0 2024-05-27T07:34:46.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.115+0000 7f150fe00700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15101034b0 0x7f151019ad80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f15000098e0 tx=0x7f15000047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.116+0000 7f150cc00700 1 -- 172.21.15.43:0/583064655 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1500027070 con 0x7f15101034b0 2024-05-27T07:34:46.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.116+0000 7f150cc00700 1 -- 172.21.15.43:0/583064655 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f150000fb00 con 0x7f15101034b0 2024-05-27T07:34:46.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.116+0000 7f150cc00700 1 -- 172.21.15.43:0/583064655 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1500022b60 con 0x7f15101034b0 2024-05-27T07:34:46.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.116+0000 7f1517200700 1 -- 172.21.15.43:0/583064655 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151019fee0 con 0x7f15101034b0 2024-05-27T07:34:46.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.116+0000 7f1517200700 1 -- 172.21.15.43:0/583064655 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15101a0340 con 0x7f15101034b0 2024-05-27T07:34:46.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.130+0000 7f150cc00700 1 -- 172.21.15.43:0/583064655 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f150000fcb0 con 0x7f15101034b0 2024-05-27T07:34:46.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.130+0000 7f1525cdfb80 1 Processor -- start 2024-05-27T07:34:46.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.130+0000 7f1525cdfb80 1 -- start start 2024-05-27T07:34:46.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.131+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8a0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.131+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 0x55809d8a0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.131+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 0x55809d8ad110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.131+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55809d7cee40 con 0x55809d80d9f0 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.131+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55809d7be9e0 con 0x55809d721e40 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.131+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55809d7c2a20 con 0x55809d8a0e70 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.132+0000 7f1515200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.132+0000 7f1515200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8a0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59412/0 (socket says 172.21.15.43:59412) 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.132+0000 7f150fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 0x55809d8a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.133+0000 7f1515200700 1 -- 172.21.15.43:0/4108616293 learned_addr learned my addr 172.21.15.43:0/4108616293 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:46.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.133+0000 7f150f400700 1 --2- 172.21.15.43:0/4108616293 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 0x55809d8ad110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.133+0000 7f1515200700 1 -- 172.21.15.43:0/4108616293 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 msgr2=0x55809d8ad110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.134+0000 7f1515200700 1 --2- 172.21.15.43:0/4108616293 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 0x55809d8ad110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.134+0000 7f1515200700 1 -- 172.21.15.43:0/4108616293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 msgr2=0x55809d8a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.134+0000 7f1515200700 1 --2- 172.21.15.43:0/4108616293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 0x55809d8a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.134+0000 7f1515200700 1 -- 172.21.15.43:0/4108616293 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55809d787420 con 0x55809d80d9f0 2024-05-27T07:34:46.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.135+0000 7f1515200700 1 --2- 172.21.15.43:0/4108616293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8a0330 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1508003fa0 tx=0x7f1508009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51951ecc63dee6e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.135+0000 7f1505600700 1 -- 172.21.15.43:0/4108616293 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508004bb0 con 0x55809d80d9f0 2024-05-27T07:34:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.136+0000 7f1505600700 1 -- 172.21.15.43:0/4108616293 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1508004820 con 0x55809d80d9f0 2024-05-27T07:34:46.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.136+0000 7f1505600700 1 -- 172.21.15.43:0/4108616293 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508007cb0 con 0x55809d80d9f0 2024-05-27T07:34:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.137+0000 7f1525cdfb80 1 -- 172.21.15.43:0/4108616293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 msgr2=0x55809d8a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.137+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/4108616293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8a0330 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1508003fa0 tx=0x7f1508009ba0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.138+0000 7f1525cdfb80 1 -- 172.21.15.43:0/4108616293 shutdown_connections 2024-05-27T07:34:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.138+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/4108616293 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 0x55809d8ad110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.138+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/4108616293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8a0330 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.138+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/4108616293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 0x55809d8a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.138+0000 7f1525cdfb80 1 -- 172.21.15.43:0/4108616293 >> 172.21.15.43:0/4108616293 conn(0x55809d7bb390 msgr2=0x55809d89c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.138+0000 7f1525cdfb80 1 -- 172.21.15.43:0/4108616293 shutdown_connections 2024-05-27T07:34:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.138+0000 7f150cc00700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f14f40b2aa0 0x7f14f40b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.139+0000 7f150cc00700 1 -- 172.21.15.43:0/583064655 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f15000e3ba0 con 0x7f15101034b0 2024-05-27T07:34:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.140+0000 7f150f400700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f14f40b2aa0 0x7f14f40b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.141+0000 7f1525cdfb80 1 -- 172.21.15.43:0/4108616293 wait complete. 2024-05-27T07:34:46.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.141+0000 7f150f400700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f14f40b2aa0 0x7f14f40b4fb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f15101099d0 tx=0x7f14fc007480 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.141+0000 7f1525cdfb80 1 -- start start 2024-05-27T07:34:46.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.142+0000 7f1525cdfb80 1 Processor -- start 2024-05-27T07:34:46.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.143+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 0x55809d8aa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.143+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.143+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 0x55809d8ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.143+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55809d6c4fa0 con 0x55809d80d9f0 2024-05-27T07:34:46.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.143+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55809d75f760 con 0x55809d721e40 2024-05-27T07:34:46.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.144+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55809d8adb80 con 0x55809d8a0e70 2024-05-27T07:34:46.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.144+0000 7f1525cdfb80 10 client.0 _ll_register_callbacks cb 0x55809d768c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.144+0000 7f1515200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 0x55809d8ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.144+0000 7f1515200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 0x55809d8ab1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:38452/0 (socket says 172.21.15.43:38452) 2024-05-27T07:34:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.144+0000 7f1515200700 1 -- 172.21.15.43:0/598862227 learned_addr learned my addr 172.21.15.43:0/598862227 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:46.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.145+0000 7f150fe00700 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 0x55809d8aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.145+0000 7f1515200700 1 -- 172.21.15.43:0/598862227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 msgr2=0x55809d8aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.145+0000 7f1515200700 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 0x55809d8aa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.145+0000 7f1515200700 1 -- 172.21.15.43:0/598862227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 msgr2=0x55809d8aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.145+0000 7f1515200700 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.145+0000 7f1515200700 1 -- 172.21.15.43:0/598862227 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55809d70e8d0 con 0x55809d8a0e70 2024-05-27T07:34:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.145+0000 7f150fe00700 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 0x55809d8aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.146+0000 7f1515200700 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 0x55809d8ab1e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1508007b60 tx=0x7f1508007b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d96fe7791dcf8526 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.146+0000 7f14fa000700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:34:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.146+0000 7f14fa000700 1 -- 172.21.15.43:0/598862227 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508007e60 con 0x55809d8a0e70 2024-05-27T07:34:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.146+0000 7f1525cdfb80 1 -- 172.21.15.43:0/598862227 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1508004820 con 0x55809d8a0e70 2024-05-27T07:34:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.146+0000 7f14fa000700 1 -- 172.21.15.43:0/598862227 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f1508004ce0 con 0x55809d8a0e70 2024-05-27T07:34:46.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.147+0000 7f14fa000700 1 -- 172.21.15.43:0/598862227 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=140) ==== 24+0+0 (secure 0 0 0) 0x7f150802c050 con 0x55809d8a0e70 2024-05-27T07:34:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.147+0000 7f1525cdfb80 10 client.27595 fetch_fsmap learned FSMap version 140 2024-05-27T07:34:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.147+0000 7f1525cdfb80 1 -- 172.21.15.43:0/598862227 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=140,monmap=4+}) -- 0x55809d78f060 con 0x55809d8a0e70 2024-05-27T07:34:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.148+0000 7f14fa000700 1 -- 172.21.15.43:0/598862227 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1508004820 con 0x55809d8a0e70 2024-05-27T07:34:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.148+0000 7f14fa000700 1 -- 172.21.15.43:0/598862227 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 140) ==== 4199+0+0 (secure 0 0 0) 0x7f1508015bb0 con 0x55809d8a0e70 2024-05-27T07:34:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.148+0000 7f1525cdfb80 10 client.27595 fetch_fsmap finished waiting for FSMap version 140 2024-05-27T07:34:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.148+0000 7f1525cdfb80 10 client.27595 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:34:46.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.148+0000 7f1525cdfb80 20 client.27595 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:46.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.148+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55809d8badd0 0x55809d8bd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.148+0000 7f1525cdfb80 4 client.27595 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:46.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.148+0000 7f1525cdfb80 1 -- 172.21.15.43:0/598862227 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55809d7d0d90 con 0x55809d8badd0 2024-05-27T07:34:46.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.150+0000 7f150fe00700 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55809d8badd0 0x55809d8bd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.151+0000 7f14fa000700 10 client.27595 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:34:46.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.151+0000 7f150fe00700 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55809d8badd0 0x55809d8bd280 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9256a433a3143309 server_cookie=535061bfb742dd8f in_seq=0 out_seq=0 2024-05-27T07:34:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f14fa000700 1 -- 172.21.15.43:0/598862227 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55809d7d0d90 con 0x55809d8badd0 2024-05-27T07:34:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f14fa000700 10 client.27595 handle_command_reply: tid=0 2024-05-27T07:34:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f1525cdfb80 1 client.27595 shutdown 2024-05-27T07:34:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f1525cdfb80 1 -- 172.21.15.43:0/598862227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 msgr2=0x55809d8ab1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 0x55809d8ab1e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1508007b60 tx=0x7f1508007b90 comp rx=0 tx=0).stop 2024-05-27T07:34:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f1525cdfb80 1 -- 172.21.15.43:0/598862227 shutdown_connections 2024-05-27T07:34:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55809d8badd0 0x55809d8bd280 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d8a0e70 0x55809d8ab1e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d721e40 0x55809d8aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/598862227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f1525cdfb80 1 -- 172.21.15.43:0/598862227 >> 172.21.15.43:0/598862227 conn(0x55809d7bb390 msgr2=0x55809d8a0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.153+0000 7f14fa000700 0 client.27595 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:34:46.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.154+0000 7f1525cdfb80 1 -- 172.21.15.43:0/598862227 shutdown_connections 2024-05-27T07:34:46.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.154+0000 7f1525cdfb80 1 -- 172.21.15.43:0/598862227 wait complete. 2024-05-27T07:34:46.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.154+0000 7f1525cdfb80 20 client.27595 trim_cache size 0 max 16384 2024-05-27T07:34:46.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.180+0000 7f1525cdfb80 1 Processor -- start 2024-05-27T07:34:46.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.180+0000 7f1525cdfb80 1 -- start start 2024-05-27T07:34:46.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.181+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 0x55809d8a0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.181+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d80d9f0 0x55809d89c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.181+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d8a0e70 0x55809d8a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.181+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55809d8a9780 con 0x55809d8a0e70 2024-05-27T07:34:46.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.181+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f15101b2c40 con 0x55809d80d9f0 2024-05-27T07:34:46.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.181+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f15101b1070 con 0x55809d721e40 2024-05-27T07:34:46.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.182+0000 7f150fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 0x55809d8a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.182+0000 7f1515200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d8a0e70 0x55809d8a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.182+0000 7f1515200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d8a0e70 0x55809d8a5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59420/0 (socket says 172.21.15.43:59420) 2024-05-27T07:34:46.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.182+0000 7f1515200700 1 -- 172.21.15.43:0/1537499853 learned_addr learned my addr 172.21.15.43:0/1537499853 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:46.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.182+0000 7f150f400700 1 --2- 172.21.15.43:0/1537499853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d80d9f0 0x55809d89c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.183+0000 7f1515200700 1 -- 172.21.15.43:0/1537499853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 msgr2=0x55809d8a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.183+0000 7f1515200700 1 --2- 172.21.15.43:0/1537499853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 0x55809d8a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.183+0000 7f1515200700 1 -- 172.21.15.43:0/1537499853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d80d9f0 msgr2=0x55809d89c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.183+0000 7f1515200700 1 --2- 172.21.15.43:0/1537499853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d80d9f0 0x55809d89c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.183+0000 7f1515200700 1 -- 172.21.15.43:0/1537499853 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55809d8a8cd0 con 0x55809d8a0e70 2024-05-27T07:34:46.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.183+0000 7f150f400700 1 --2- 172.21.15.43:0/1537499853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d80d9f0 0x55809d89c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:46.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.183+0000 7f1515200700 1 --2- 172.21.15.43:0/1537499853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d8a0e70 0x55809d8a5ef0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f150801e5b0 tx=0x7f1508009270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d21b35c2a868dc6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.184+0000 7f1504c00700 1 -- 172.21.15.43:0/1537499853 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508034070 con 0x55809d8a0e70 2024-05-27T07:34:46.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.184+0000 7f1504c00700 1 -- 172.21.15.43:0/1537499853 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1508023070 con 0x55809d8a0e70 2024-05-27T07:34:46.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.184+0000 7f1504c00700 1 -- 172.21.15.43:0/1537499853 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508005c60 con 0x55809d8a0e70 2024-05-27T07:34:46.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.185+0000 7f1525cdfb80 1 -- 172.21.15.43:0/1537499853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d8a0e70 msgr2=0x55809d8a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.185+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/1537499853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d8a0e70 0x55809d8a5ef0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f150801e5b0 tx=0x7f1508009270 comp rx=0 tx=0).stop 2024-05-27T07:34:46.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.186+0000 7f1525cdfb80 1 -- 172.21.15.43:0/1537499853 shutdown_connections 2024-05-27T07:34:46.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.186+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/1537499853 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d8a0e70 0x55809d8a5ef0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.186+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/1537499853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 0x55809d8a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.186+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/1537499853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d80d9f0 0x55809d89c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.186+0000 7f1525cdfb80 1 -- 172.21.15.43:0/1537499853 >> 172.21.15.43:0/1537499853 conn(0x55809d7bb390 msgr2=0x55809d89bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.186+0000 7f1525cdfb80 1 -- 172.21.15.43:0/1537499853 shutdown_connections 2024-05-27T07:34:46.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.186+0000 7f1525cdfb80 1 -- 172.21.15.43:0/1537499853 wait complete. 2024-05-27T07:34:46.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.187+0000 7f1525cdfb80 1 -- start start 2024-05-27T07:34:46.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.187+0000 7f1525cdfb80 1 Processor -- start 2024-05-27T07:34:46.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.188+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 0x55809d8addd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.188+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8ae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.188+0000 7f1525cdfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d8a0e70 0x55809d8ae850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.188+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f15101b1770 con 0x55809d80d9f0 2024-05-27T07:34:46.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.188+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f15101b2570 con 0x55809d8a0e70 2024-05-27T07:34:46.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.188+0000 7f1525cdfb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f15101b1ea0 con 0x55809d721e40 2024-05-27T07:34:46.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.188+0000 7f1525cdfb80 10 client.0 _ll_register_callbacks cb 0x55809d768c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:46.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.188+0000 7f150fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 0x55809d8addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.188+0000 7f150fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 0x55809d8addd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:38478/0 (socket says 172.21.15.43:38478) 2024-05-27T07:34:46.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.188+0000 7f150fe00700 1 -- 172.21.15.43:0/902210729 learned_addr learned my addr 172.21.15.43:0/902210729 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:46.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.189+0000 7f150f400700 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.189+0000 7f150fe00700 1 -- 172.21.15.43:0/902210729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d8a0e70 msgr2=0x55809d8ae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.189+0000 7f1515200700 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d8a0e70 0x55809d8ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.189+0000 7f150fe00700 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d8a0e70 0x55809d8ae850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.189+0000 7f150fe00700 1 -- 172.21.15.43:0/902210729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 msgr2=0x55809d8ae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:46.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.189+0000 7f150fe00700 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8ae310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.189+0000 7f150fe00700 1 -- 172.21.15.43:0/902210729 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55809d92c220 con 0x55809d721e40 2024-05-27T07:34:46.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.190+0000 7f1515200700 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d8a0e70 0x55809d8ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:46.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.190+0000 7f150f400700 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8ae310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:34:46.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.190+0000 7f150fe00700 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 0x55809d8addd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15000aa7e0 tx=0x7f15000e5f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3270503d8d876150 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:46.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.190+0000 7f14fa000700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:34:46.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.190+0000 7f14fa000700 1 -- 172.21.15.43:0/902210729 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15000e9070 con 0x55809d721e40 2024-05-27T07:34:46.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.191+0000 7f1525cdfb80 1 -- 172.21.15.43:0/902210729 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1508023070 con 0x55809d721e40 2024-05-27T07:34:46.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.191+0000 7f14fa000700 1 -- 172.21.15.43:0/902210729 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f15000aaf50 con 0x55809d721e40 2024-05-27T07:34:46.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.191+0000 7f14fa000700 1 -- 172.21.15.43:0/902210729 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=140) ==== 24+0+0 (secure 0 0 0) 0x7f15000aa8b0 con 0x55809d721e40 2024-05-27T07:34:46.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.191+0000 7f1525cdfb80 10 client.27601 fetch_fsmap learned FSMap version 140 2024-05-27T07:34:46.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.191+0000 7f1525cdfb80 1 -- 172.21.15.43:0/902210729 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=140,monmap=4+}) -- 0x55809d92d510 con 0x55809d721e40 2024-05-27T07:34:46.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.192+0000 7f14fa000700 1 -- 172.21.15.43:0/902210729 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1508023070 con 0x55809d721e40 2024-05-27T07:34:46.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.192+0000 7f14fa000700 1 -- 172.21.15.43:0/902210729 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 140) ==== 4199+0+0 (secure 0 0 0) 0x7f15000b3420 con 0x55809d721e40 2024-05-27T07:34:46.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.192+0000 7f1525cdfb80 10 client.27601 fetch_fsmap finished waiting for FSMap version 140 2024-05-27T07:34:46.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.192+0000 7f1525cdfb80 10 client.27601 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:34:46.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.192+0000 7f1525cdfb80 20 client.27601 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:46.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.193+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55809d92dc00 0x55809d930020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:46.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.193+0000 7f1525cdfb80 4 client.27601 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:34:46.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.193+0000 7f1525cdfb80 1 -- 172.21.15.43:0/902210729 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55809d89b120 con 0x55809d92dc00 2024-05-27T07:34:46.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.194+0000 7f150f400700 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55809d92dc00 0x55809d930020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:46.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.195+0000 7f150f400700 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55809d92dc00 0x55809d930020 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16b493ba77cb7c5a server_cookie=1730c30ca7e9aa8 in_seq=0 out_seq=0 2024-05-27T07:34:46.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.195+0000 7f14fa000700 10 client.27601 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:34:46.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.195+0000 7f14fa000700 1 -- 172.21.15.43:0/902210729 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55809d89b120 con 0x55809d92dc00 2024-05-27T07:34:46.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.195+0000 7f14fa000700 10 client.27601 handle_command_reply: tid=0 2024-05-27T07:34:46.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.195+0000 7f1525cdfb80 1 client.27601 shutdown 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 1 -- 172.21.15.43:0/902210729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 msgr2=0x55809d8addd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 0x55809d8addd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15000aa7e0 tx=0x7f15000e5f70 comp rx=0 tx=0).stop 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 1 -- 172.21.15.43:0/902210729 shutdown_connections 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55809d92dc00 0x55809d930020 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55809d8a0e70 0x55809d8ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55809d721e40 0x55809d8addd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 1 --2- 172.21.15.43:0/902210729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55809d80d9f0 0x55809d8ae310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 1 -- 172.21.15.43:0/902210729 >> 172.21.15.43:0/902210729 conn(0x55809d7bb390 msgr2=0x55809d8aa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f14fa000700 0 client.27601 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 1 -- 172.21.15.43:0/902210729 shutdown_connections 2024-05-27T07:34:46.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 1 -- 172.21.15.43:0/902210729 wait complete. 2024-05-27T07:34:46.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.196+0000 7f1525cdfb80 20 client.27601 trim_cache size 0 max 16384 2024-05-27T07:34:46.193 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:34:46.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 -- 172.21.15.43:0/583064655 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f14f40b2aa0 msgr2=0x7f14f40b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f14f40b2aa0 0x7f14f40b4fb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f15101099d0 tx=0x7f14fc007480 comp rx=0 tx=0).stop 2024-05-27T07:34:46.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 -- 172.21.15.43:0/583064655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15101034b0 msgr2=0x7f151019ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:46.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15101034b0 0x7f151019ad80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f15000098e0 tx=0x7f15000047b0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 -- 172.21.15.43:0/583064655 shutdown_connections 2024-05-27T07:34:46.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f14f40b2aa0 0x7f14f40b4fb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1510108730 0x7f151019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15101034b0 0x7f151019ad80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 --2- 172.21.15.43:0/583064655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1510107de0 0x7f151019b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:46.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 -- 172.21.15.43:0/583064655 >> 172.21.15.43:0/583064655 conn(0x7f15100fedb0 msgr2=0x7f151006baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:46.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.198+0000 7f1517200700 1 -- 172.21.15.43:0/583064655 shutdown_connections 2024-05-27T07:34:46.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:46.199+0000 7f1517200700 1 -- 172.21.15.43:0/583064655 wait complete. 2024-05-27T07:34:46.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44859 5.85 MB/sec execute 287 sec latency 103.634 ms 2024-05-27T07:34:46.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:46 smithi132 ceph-mon[35283]: pgmap v700: 129 pgs: 129 active+clean; 1006 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-05-27T07:34:46.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:46 smithi043 ceph-mon[31531]: pgmap v700: 129 pgs: 129 active+clean; 1006 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-05-27T07:34:46.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:46 smithi179 ceph-mon[35888]: pgmap v700: 129 pgs: 129 active+clean; 1006 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-05-27T07:34:47.182 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:34:47.757 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:47.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 44964 5.85 MB/sec execute 288 sec latency 85.945 ms 2024-05-27T07:34:47.925 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:47 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:47.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:47 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:48.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:47 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:34:48.652 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:48 smithi043 ceph-mon[31531]: pgmap v701: 129 pgs: 129 active+clean; 1013 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.7 MiB/s wr, 214 op/s 2024-05-27T07:34:48.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 45082 5.84 MB/sec execute 289 sec latency 92.882 ms 2024-05-27T07:34:48.962 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:48 smithi179 ceph-mon[35888]: pgmap v701: 129 pgs: 129 active+clean; 1013 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.7 MiB/s wr, 214 op/s 2024-05-27T07:34:49.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:48 smithi132 ceph-mon[35283]: pgmap v701: 129 pgs: 129 active+clean; 1013 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.7 MiB/s wr, 214 op/s 2024-05-27T07:34:49.809 INFO:tasks.workunit.client.0.smithi043.stdout: 10 45221 5.85 MB/sec execute 290 sec latency 118.074 ms 2024-05-27T07:34:50.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.787+0000 7ff6cb400700 1 -- 172.21.15.43:0/4208719295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c4108730 msgr2=0x7ff6c4110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.787+0000 7ff6cb400700 1 --2- 172.21.15.43:0/4208719295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c4108730 0x7ff6c4110090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009fc0 tx=0x7ff6bc00b330 comp rx=0 tx=0).stop 2024-05-27T07:34:50.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.787+0000 7ff6cb400700 1 -- 172.21.15.43:0/4208719295 shutdown_connections 2024-05-27T07:34:50.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.787+0000 7ff6cb400700 1 --2- 172.21.15.43:0/4208719295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c4108730 0x7ff6c4110090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.787+0000 7ff6cb400700 1 --2- 172.21.15.43:0/4208719295 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c41034b0 0x7ff6c41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.787+0000 7ff6cb400700 1 --2- 172.21.15.43:0/4208719295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6c4107de0 0x7ff6c41081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.788+0000 7ff6cb400700 1 -- 172.21.15.43:0/4208719295 >> 172.21.15.43:0/4208719295 conn(0x7ff6c40fedb0 msgr2=0x7ff6c41011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:50.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.788+0000 7ff6cb400700 1 -- 172.21.15.43:0/4208719295 shutdown_connections 2024-05-27T07:34:50.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.789+0000 7ff6cb400700 1 -- 172.21.15.43:0/4208719295 wait complete. 2024-05-27T07:34:50.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.790+0000 7ff6cb400700 1 Processor -- start 2024-05-27T07:34:50.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.791+0000 7ff6cb400700 1 -- start start 2024-05-27T07:34:50.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.792+0000 7ff6cb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c41034b0 0x7ff6c419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.792+0000 7ff6cb400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6c4107de0 0x7ff6c419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.792+0000 7ff6cb400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c4108730 0x7ff6c41a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.792+0000 7ff6cb400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c4114310 con 0x7ff6c41034b0 2024-05-27T07:34:50.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.792+0000 7ff6cb400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c4114190 con 0x7ff6c4107de0 2024-05-27T07:34:50.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.792+0000 7ff6cb400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c4114490 con 0x7ff6c4108730 2024-05-27T07:34:50.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.792+0000 7ff6c3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6c4107de0 0x7ff6c419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.792+0000 7ff6c3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6c4107de0 0x7ff6c419d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48574/0 (socket says 172.21.15.43:48574) 2024-05-27T07:34:50.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.792+0000 7ff6c3e00700 1 -- 172.21.15.43:0/1273131965 learned_addr learned my addr 172.21.15.43:0/1273131965 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:50.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.793+0000 7ff6c9400700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c4108730 0x7ff6c41a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.793+0000 7ff6c3e00700 1 -- 172.21.15.43:0/1273131965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c4108730 msgr2=0x7ff6c41a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.793+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c41034b0 0x7ff6c419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.794+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c4108730 0x7ff6c41a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.794+0000 7ff6c3e00700 1 -- 172.21.15.43:0/1273131965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c41034b0 msgr2=0x7ff6c419cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.794+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c41034b0 0x7ff6c419cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.794+0000 7ff6c3e00700 1 -- 172.21.15.43:0/1273131965 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c41a1d90 con 0x7ff6c4107de0 2024-05-27T07:34:50.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.794+0000 7ff6c9400700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c4108730 0x7ff6c41a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:50.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.795+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6c4107de0 0x7ff6c419d460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000d980 tx=0x7ff6b000dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:50.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.795+0000 7ff6c1600700 1 -- 172.21.15.43:0/1273131965 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0017070 con 0x7ff6c4107de0 2024-05-27T07:34:50.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.795+0000 7ff6cb400700 1 -- 172.21.15.43:0/1273131965 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c41a2080 con 0x7ff6c4107de0 2024-05-27T07:34:50.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.795+0000 7ff6cb400700 1 -- 172.21.15.43:0/1273131965 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c41a99a0 con 0x7ff6c4107de0 2024-05-27T07:34:50.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.796+0000 7ff6c1600700 1 -- 172.21.15.43:0/1273131965 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff6b0010460 con 0x7ff6c4107de0 2024-05-27T07:34:50.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.796+0000 7ff6c1600700 1 -- 172.21.15.43:0/1273131965 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0020060 con 0x7ff6c4107de0 2024-05-27T07:34:50.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.800+0000 7ff6c1600700 1 -- 172.21.15.43:0/1273131965 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff6b0009b40 con 0x7ff6c4107de0 2024-05-27T07:34:50.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.804+0000 7ff6c1600700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff6ac0b2a70 0x7ff6ac0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.804+0000 7ff6c1600700 1 -- 172.21.15.43:0/1273131965 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7ff6b00e23d0 con 0x7ff6c4107de0 2024-05-27T07:34:50.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.804+0000 7ff6d9f1db80 1 Processor -- start 2024-05-27T07:34:50.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.804+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff6ac0b2a70 0x7ff6ac0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.804+0000 7ff6d9f1db80 1 -- start start 2024-05-27T07:34:50.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.805+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a80abe40 0x55d4a82307f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.805+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 0x55d4a822a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.806+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a822adb0 0x55d4a822b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.806+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d4a8158e40 con 0x55d4a822adb0 2024-05-27T07:34:50.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.806+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d4a81489e0 con 0x55d4a8230d30 2024-05-27T07:34:50.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.806+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d4a814ca20 con 0x55d4a80abe40 2024-05-27T07:34:50.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.806+0000 7ff6c3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 0x55d4a822a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.806+0000 7ff6c3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 0x55d4a822a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48578/0 (socket says 172.21.15.43:48578) 2024-05-27T07:34:50.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.806+0000 7ff6c3e00700 1 -- 172.21.15.43:0/2617196855 learned_addr learned my addr 172.21.15.43:0/2617196855 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:50.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.806+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff6ac0b2a70 0x7ff6ac0b4f80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009b10 tx=0x7ff6b80094f0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:50.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.807+0000 7ff6c9400700 1 --2- 172.21.15.43:0/2617196855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a822adb0 0x55d4a822b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.807+0000 7ff6c3e00700 1 -- 172.21.15.43:0/2617196855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a80abe40 msgr2=0x55d4a82307f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:34:50.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.807+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/2617196855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a80abe40 0x55d4a82307f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.807+0000 7ff6c3e00700 1 -- 172.21.15.43:0/2617196855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a822adb0 msgr2=0x55d4a822b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:50.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.807+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/2617196855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a822adb0 0x55d4a822b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.807+0000 7ff6c3e00700 1 -- 172.21.15.43:0/2617196855 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4a8111420 con 0x55d4a8230d30 2024-05-27T07:34:50.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.807+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/2617196855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 0x55d4a822a870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00a8fe0 tx=0x7ff6b00a8e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8b38aa343db8f01 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:50.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.808+0000 7ff6b6000700 1 -- 172.21.15.43:0/2617196855 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b00ad690 con 0x55d4a8230d30 2024-05-27T07:34:50.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.808+0000 7ff6b6000700 1 -- 172.21.15.43:0/2617196855 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff6b00b2140 con 0x55d4a8230d30 2024-05-27T07:34:50.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.808+0000 7ff6b6000700 1 -- 172.21.15.43:0/2617196855 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b00ad120 con 0x55d4a8230d30 2024-05-27T07:34:50.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.809+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/2617196855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 msgr2=0x55d4a822a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.809+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/2617196855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 0x55d4a822a870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00a8fe0 tx=0x7ff6b00a8e70 comp rx=0 tx=0).stop 2024-05-27T07:34:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.810+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/2617196855 shutdown_connections 2024-05-27T07:34:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.810+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/2617196855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a822adb0 0x55d4a822b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.810+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/2617196855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a80abe40 0x55d4a82307f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.810+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/2617196855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 0x55d4a822a870 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.810+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/2617196855 >> 172.21.15.43:0/2617196855 conn(0x55d4a810bfc0 msgr2=0x55d4a82258d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:50.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.810+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/2617196855 shutdown_connections 2024-05-27T07:34:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.810+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/2617196855 wait complete. 2024-05-27T07:34:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.811+0000 7ff6d9f1db80 1 -- start start 2024-05-27T07:34:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.811+0000 7ff6d9f1db80 1 Processor -- start 2024-05-27T07:34:50.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.811+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a80abe40 0x55d4a82345a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.812+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a822adb0 0x55d4a8234ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.812+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a8230d30 0x55d4a8235020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.812+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d4a804efa0 con 0x55d4a8230d30 2024-05-27T07:34:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.812+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d4a80e9760 con 0x55d4a80abe40 2024-05-27T07:34:50.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.812+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d4a823ff50 con 0x55d4a822adb0 2024-05-27T07:34:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.812+0000 7ff6d9f1db80 10 client.0 _ll_register_callbacks cb 0x55d4a80f2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.812+0000 7ff6c8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a80abe40 0x55d4a82345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.812+0000 7ff6c8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a80abe40 0x55d4a82345a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48588/0 (socket says 172.21.15.43:48588) 2024-05-27T07:34:50.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.812+0000 7ff6c8a00700 1 -- 172.21.15.43:0/11949031 learned_addr learned my addr 172.21.15.43:0/11949031 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:50.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.812+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a822adb0 0x55d4a8234ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.813+0000 7ff6c9400700 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a8230d30 0x55d4a8235020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.813+0000 7ff6c8a00700 1 -- 172.21.15.43:0/11949031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a822adb0 msgr2=0x55d4a8234ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:50.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.813+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a822adb0 0x55d4a8234ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.813+0000 7ff6c8a00700 1 -- 172.21.15.43:0/11949031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a8230d30 msgr2=0x55d4a8235020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:50.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.813+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a8230d30 0x55d4a8235020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.813+0000 7ff6c8a00700 1 -- 172.21.15.43:0/11949031 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4a80988d0 con 0x55d4a80abe40 2024-05-27T07:34:50.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.813+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a80abe40 0x55d4a82345a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8011720 tx=0x7ff6b8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d219121a0f85f183 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:50.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.814+0000 7ff6aaa00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:34:50.812 INFO:tasks.workunit.client.0.smithi043.stdout: 10 45325 5.85 MB/sec execute 291 sec latency 85.739 ms 2024-05-27T07:34:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.814+0000 7ff6aaa00700 1 -- 172.21.15.43:0/11949031 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8022020 con 0x55d4a80abe40 2024-05-27T07:34:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.814+0000 7ff6aaa00700 1 -- 172.21.15.43:0/11949031 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7ff6b801e640 con 0x55d4a80abe40 2024-05-27T07:34:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.814+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/11949031 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6b00b2140 con 0x55d4a80abe40 2024-05-27T07:34:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.814+0000 7ff6aaa00700 1 -- 172.21.15.43:0/11949031 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) ==== 24+0+0 (secure 0 0 0) 0x7ff6b802d050 con 0x55d4a80abe40 2024-05-27T07:34:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.815+0000 7ff6d9f1db80 10 client.37553 fetch_fsmap learned FSMap version 141 2024-05-27T07:34:50.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.815+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/11949031 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=4+}) -- 0x55d4a8119060 con 0x55d4a80abe40 2024-05-27T07:34:50.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.816+0000 7ff6aaa00700 1 -- 172.21.15.43:0/11949031 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff6b00b2140 con 0x55d4a80abe40 2024-05-27T07:34:50.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.816+0000 7ff6aaa00700 1 -- 172.21.15.43:0/11949031 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 141) ==== 4203+0+0 (secure 0 0 0) 0x7ff6b800f040 con 0x55d4a80abe40 2024-05-27T07:34:50.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.816+0000 7ff6d9f1db80 10 client.37553 fetch_fsmap finished waiting for FSMap version 141 2024-05-27T07:34:50.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.816+0000 7ff6d9f1db80 10 client.37553 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:34:50.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.817+0000 7ff6d9f1db80 20 client.37553 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:50.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.817+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55d4a824d320 0x55d4a824f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.817+0000 7ff6d9f1db80 4 client.37553 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:50.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.817+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/11949031 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4a815ad90 con 0x55d4a824d320 2024-05-27T07:34:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.818+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55d4a824d320 0x55d4a824f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.818+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55d4a824d320 0x55d4a824f790 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ec149dc14ab32115 server_cookie=12edee7539cd6389 in_seq=0 out_seq=0 2024-05-27T07:34:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.818+0000 7ff6aaa00700 10 client.37553 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:34:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.819+0000 7ff6aaa00700 1 -- 172.21.15.43:0/11949031 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d4a815ad90 con 0x55d4a824d320 2024-05-27T07:34:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.819+0000 7ff6aaa00700 10 client.37553 handle_command_reply: tid=0 2024-05-27T07:34:50.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.819+0000 7ff6d9f1db80 1 client.37553 shutdown 2024-05-27T07:34:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.820+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/11949031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a80abe40 msgr2=0x55d4a82345a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.820+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a80abe40 0x55d4a82345a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8011720 tx=0x7ff6b8004750 comp rx=0 tx=0).stop 2024-05-27T07:34:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.820+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/11949031 shutdown_connections 2024-05-27T07:34:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.820+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55d4a824d320 0x55d4a824f790 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.820+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a8230d30 0x55d4a8235020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.820+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a80abe40 0x55d4a82345a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.820+0000 7ff6aaa00700 0 client.37553 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:34:50.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.820+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/11949031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a822adb0 0x55d4a8234ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.820+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/11949031 >> 172.21.15.43:0/11949031 conn(0x55d4a810bfc0 msgr2=0x55d4a8225580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:50.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.821+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/11949031 shutdown_connections 2024-05-27T07:34:50.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.821+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/11949031 wait complete. 2024-05-27T07:34:50.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.821+0000 7ff6d9f1db80 20 client.37553 trim_cache size 0 max 16384 2024-05-27T07:34:50.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.844+0000 7ff6d9f1db80 1 Processor -- start 2024-05-27T07:34:50.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.845+0000 7ff6d9f1db80 1 -- start start 2024-05-27T07:34:50.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.845+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a822a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.845+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a822adb0 0x55d4a824f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.845+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a8230d30 0x55d4a8236900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.845+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a400f3e0 con 0x55d4a80abe40 2024-05-27T07:34:50.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.845+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a400d7e0 con 0x55d4a822adb0 2024-05-27T07:34:50.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.845+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a400ed10 con 0x55d4a8230d30 2024-05-27T07:34:50.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.846+0000 7ff6c3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a822adb0 0x55d4a824f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.846+0000 7ff6c8a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a822a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.846+0000 7ff6c3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a822adb0 0x55d4a824f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48598/0 (socket says 172.21.15.43:48598) 2024-05-27T07:34:50.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.846+0000 7ff6c3e00700 1 -- 172.21.15.43:0/2231276196 learned_addr learned my addr 172.21.15.43:0/2231276196 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:50.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.846+0000 7ff6c8a00700 1 -- 172.21.15.43:0/2231276196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a8230d30 msgr2=0x55d4a8236900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:50.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.846+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/2231276196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a8230d30 0x55d4a8236900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.846+0000 7ff6c8a00700 1 -- 172.21.15.43:0/2231276196 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a822adb0 msgr2=0x55d4a824f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:50.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.846+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/2231276196 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a822adb0 0x55d4a824f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.846+0000 7ff6c8a00700 1 -- 172.21.15.43:0/2231276196 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4a82263a0 con 0x55d4a80abe40 2024-05-27T07:34:50.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.847+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/2231276196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a822a870 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8026d10 tx=0x7ff6b8011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbf4a752866670d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:50.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.848+0000 7ff6b5600700 1 -- 172.21.15.43:0/2231276196 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8022040 con 0x55d4a80abe40 2024-05-27T07:34:50.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.848+0000 7ff6b5600700 1 -- 172.21.15.43:0/2231276196 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff6b8017030 con 0x55d4a80abe40 2024-05-27T07:34:50.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.848+0000 7ff6b5600700 1 -- 172.21.15.43:0/2231276196 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b80103d0 con 0x55d4a80abe40 2024-05-27T07:34:50.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.849+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/2231276196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 msgr2=0x55d4a822a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.849+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/2231276196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a822a870 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8026d10 tx=0x7ff6b8011ef0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.850+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/2231276196 shutdown_connections 2024-05-27T07:34:50.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.850+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/2231276196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a8230d30 0x55d4a8236900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.850+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/2231276196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a822a870 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.850+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/2231276196 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a822adb0 0x55d4a824f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.850+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/2231276196 >> 172.21.15.43:0/2231276196 conn(0x55d4a810bfc0 msgr2=0x55d4a822e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:50.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.850+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/2231276196 shutdown_connections 2024-05-27T07:34:50.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.850+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/2231276196 wait complete. 2024-05-27T07:34:50.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.851+0000 7ff6d9f1db80 1 -- start start 2024-05-27T07:34:50.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.851+0000 7ff6d9f1db80 1 Processor -- start 2024-05-27T07:34:50.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.852+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a8245210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.852+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a822adb0 0x55d4a8245750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.852+0000 7ff6d9f1db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 0x55d4a8253f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.852+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a400e640 con 0x55d4a80abe40 2024-05-27T07:34:50.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.852+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a400df10 con 0x55d4a8230d30 2024-05-27T07:34:50.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.852+0000 7ff6d9f1db80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d4a82373a0 con 0x55d4a822adb0 2024-05-27T07:34:50.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.852+0000 7ff6d9f1db80 10 client.0 _ll_register_callbacks cb 0x55d4a8249820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:50.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.853+0000 7ff6c8a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a8245210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.853+0000 7ff6c8a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a8245210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59494/0 (socket says 172.21.15.43:59494) 2024-05-27T07:34:50.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.853+0000 7ff6c8a00700 1 -- 172.21.15.43:0/3824495183 learned_addr learned my addr 172.21.15.43:0/3824495183 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:50.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.853+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a822adb0 0x55d4a8245750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.854+0000 7ff6c8a00700 1 -- 172.21.15.43:0/3824495183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a822adb0 msgr2=0x55d4a8245750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:50.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.854+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a822adb0 0x55d4a8245750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.854+0000 7ff6c8a00700 1 -- 172.21.15.43:0/3824495183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 msgr2=0x55d4a8253f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:34:50.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.854+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 0x55d4a8253f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.854+0000 7ff6c8a00700 1 -- 172.21.15.43:0/3824495183 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4a8245e30 con 0x55d4a80abe40 2024-05-27T07:34:50.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.854+0000 7ff6c8a00700 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a8245210 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7ff6b80108c0 tx=0x7ff6b8010480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b720edb2ccb97b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:50.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.855+0000 7ff6aaa00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:34:50.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.855+0000 7ff6aaa00700 1 -- 172.21.15.43:0/3824495183 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8022040 con 0x55d4a80abe40 2024-05-27T07:34:50.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.855+0000 7ff6aaa00700 1 -- 172.21.15.43:0/3824495183 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7ff6b8005020 con 0x55d4a80abe40 2024-05-27T07:34:50.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.855+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/3824495183 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6b8017030 con 0x55d4a80abe40 2024-05-27T07:34:50.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.856+0000 7ff6aaa00700 1 -- 172.21.15.43:0/3824495183 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) ==== 24+0+0 (secure 0 0 0) 0x7ff6b8010c90 con 0x55d4a80abe40 2024-05-27T07:34:50.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.856+0000 7ff6d9f1db80 10 client.18186 fetch_fsmap learned FSMap version 141 2024-05-27T07:34:50.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.856+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/3824495183 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=4+}) -- 0x55d4a82b73a0 con 0x55d4a80abe40 2024-05-27T07:34:50.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.858+0000 7ff6aaa00700 1 -- 172.21.15.43:0/3824495183 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff6b8017030 con 0x55d4a80abe40 2024-05-27T07:34:50.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.858+0000 7ff6aaa00700 1 -- 172.21.15.43:0/3824495183 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 141) ==== 4203+0+0 (secure 0 0 0) 0x7ff6b8004cd0 con 0x55d4a80abe40 2024-05-27T07:34:50.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.858+0000 7ff6d9f1db80 10 client.18186 fetch_fsmap finished waiting for FSMap version 141 2024-05-27T07:34:50.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.858+0000 7ff6d9f1db80 10 client.18186 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:34:50.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.858+0000 7ff6d9f1db80 20 client.18186 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:50.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.858+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55d4a82b7990 0x55d4a82b9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:50.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.858+0000 7ff6d9f1db80 4 client.18186 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:34:50.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.858+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/3824495183 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d4a8154570 con 0x55d4a82b7990 2024-05-27T07:34:50.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.859+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55d4a82b7990 0x55d4a82b9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:50.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.860+0000 7ff6c3e00700 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55d4a82b7990 0x55d4a82b9db0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ace7312649017ff server_cookie=ec8e06920e6e60fc in_seq=0 out_seq=0 2024-05-27T07:34:50.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.860+0000 7ff6aaa00700 10 client.18186 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:34:50.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.860+0000 7ff6aaa00700 1 -- 172.21.15.43:0/3824495183 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d4a8154570 con 0x55d4a82b7990 2024-05-27T07:34:50.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.860+0000 7ff6aaa00700 10 client.18186 handle_command_reply: tid=0 2024-05-27T07:34:50.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.860+0000 7ff6d9f1db80 1 client.18186 shutdown 2024-05-27T07:34:50.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.861+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/3824495183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 msgr2=0x55d4a8245210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.861+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a8245210 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7ff6b80108c0 tx=0x7ff6b8010480 comp rx=0 tx=0).stop 2024-05-27T07:34:50.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.861+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/3824495183 shutdown_connections 2024-05-27T07:34:50.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.861+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55d4a82b7990 0x55d4a82b9db0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.861+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d4a8230d30 0x55d4a8253f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.861+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4a80abe40 0x55d4a8245210 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.861+0000 7ff6d9f1db80 1 --2- 172.21.15.43:0/3824495183 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d4a822adb0 0x55d4a8245750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.861+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/3824495183 >> 172.21.15.43:0/3824495183 conn(0x55d4a810bfc0 msgr2=0x55d4a8234380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:50.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.861+0000 7ff6aaa00700 0 client.18186 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:34:50.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.862+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/3824495183 shutdown_connections 2024-05-27T07:34:50.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.862+0000 7ff6d9f1db80 1 -- 172.21.15.43:0/3824495183 wait complete. 2024-05-27T07:34:50.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.862+0000 7ff6d9f1db80 20 client.18186 trim_cache size 0 max 16384 2024-05-27T07:34:50.858 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:34:50.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.863+0000 7ff6cb400700 1 -- 172.21.15.43:0/1273131965 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff6ac0b2a70 msgr2=0x7ff6ac0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.863+0000 7ff6cb400700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff6ac0b2a70 0x7ff6ac0b4f80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009b10 tx=0x7ff6b80094f0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.863+0000 7ff6cb400700 1 -- 172.21.15.43:0/1273131965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6c4107de0 msgr2=0x7ff6c419d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:50.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.863+0000 7ff6cb400700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6c4107de0 0x7ff6c419d460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000d980 tx=0x7ff6b000dd40 comp rx=0 tx=0).stop 2024-05-27T07:34:50.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.863+0000 7ff6cb400700 1 -- 172.21.15.43:0/1273131965 shutdown_connections 2024-05-27T07:34:50.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.863+0000 7ff6cb400700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff6ac0b2a70 0x7ff6ac0b4f80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.863+0000 7ff6cb400700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c4108730 0x7ff6c41a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.863+0000 7ff6cb400700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c41034b0 0x7ff6c419cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.863+0000 7ff6cb400700 1 --2- 172.21.15.43:0/1273131965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6c4107de0 0x7ff6c419d460 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:50.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.863+0000 7ff6cb400700 1 -- 172.21.15.43:0/1273131965 >> 172.21.15.43:0/1273131965 conn(0x7ff6c40fedb0 msgr2=0x7ff6c4109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:50.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.864+0000 7ff6cb400700 1 -- 172.21.15.43:0/1273131965 shutdown_connections 2024-05-27T07:34:50.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:50.864+0000 7ff6cb400700 1 -- 172.21.15.43:0/1273131965 wait complete. 2024-05-27T07:34:50.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:50 smithi043 ceph-mon[31531]: pgmap v702: 129 pgs: 129 active+clean; 1013 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 3.1 MiB/s wr, 142 op/s 2024-05-27T07:34:50.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:50 smithi179 ceph-mon[35888]: pgmap v702: 129 pgs: 129 active+clean; 1013 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 3.1 MiB/s wr, 142 op/s 2024-05-27T07:34:51.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:50 smithi132 ceph-mon[35283]: pgmap v702: 129 pgs: 129 active+clean; 1013 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 3.1 MiB/s wr, 142 op/s 2024-05-27T07:34:51.810 INFO:tasks.workunit.client.0.smithi043.stdout: 10 45468 5.86 MB/sec execute 292 sec latency 80.043 ms 2024-05-27T07:34:51.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:34:52.397 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:52.810 INFO:tasks.workunit.client.0.smithi043.stdout: 10 45566 5.86 MB/sec execute 293 sec latency 102.064 ms 2024-05-27T07:34:52.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:52 smithi043 ceph-mon[31531]: pgmap v703: 129 pgs: 129 active+clean; 1022 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 5.0 MiB/s wr, 219 op/s 2024-05-27T07:34:52.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:52 smithi179 ceph-mon[35888]: pgmap v703: 129 pgs: 129 active+clean; 1022 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 5.0 MiB/s wr, 219 op/s 2024-05-27T07:34:53.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:52 smithi132 ceph-mon[35283]: pgmap v703: 129 pgs: 129 active+clean; 1022 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 5.0 MiB/s wr, 219 op/s 2024-05-27T07:34:53.810 INFO:tasks.workunit.client.0.smithi043.stdout: 10 45646 5.86 MB/sec execute 294 sec latency 286.228 ms 2024-05-27T07:34:54.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.796+0000 7f7218c00700 1 -- 172.21.15.43:0/4157681880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7214105bd0 msgr2=0x7f721410e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:54.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.796+0000 7f7218c00700 1 --2- 172.21.15.43:0/4157681880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7214105bd0 0x7f721410e120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f71fc009990 tx=0x7f71fc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.796+0000 7f7218c00700 1 -- 172.21.15.43:0/4157681880 shutdown_connections 2024-05-27T07:34:54.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.796+0000 7f7218c00700 1 --2- 172.21.15.43:0/4157681880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f721410e6f0 0x7f7214110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.796+0000 7f7218c00700 1 --2- 172.21.15.43:0/4157681880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7214105280 0x7f7214105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.797+0000 7f7218c00700 1 --2- 172.21.15.43:0/4157681880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7214105bd0 0x7f721410e120 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.797+0000 7f7218c00700 1 -- 172.21.15.43:0/4157681880 >> 172.21.15.43:0/4157681880 conn(0x7f7214078ab0 msgr2=0x7f7214078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:54.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.797+0000 7f7218c00700 1 -- 172.21.15.43:0/4157681880 shutdown_connections 2024-05-27T07:34:54.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.797+0000 7f7218c00700 1 -- 172.21.15.43:0/4157681880 wait complete. 2024-05-27T07:34:54.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.798+0000 7f7218c00700 1 Processor -- start 2024-05-27T07:34:54.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.798+0000 7f7218c00700 1 -- start start 2024-05-27T07:34:54.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.799+0000 7f7218c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7214105280 0x7f7214102f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.799+0000 7f7218c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7214105bd0 0x7f72141034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.799+0000 7f7218c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721410e6f0 0x7f72141039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.799+0000 7f7218c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7214114100 con 0x7f721410e6f0 2024-05-27T07:34:54.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.799+0000 7f7218c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7214113f80 con 0x7f7214105bd0 2024-05-27T07:34:54.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.799+0000 7f7218c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7214114280 con 0x7f7214105280 2024-05-27T07:34:54.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f720be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7214105bd0 0x7f72141034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f7213e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721410e6f0 0x7f72141039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f720be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7214105bd0 0x7f72141034b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37736/0 (socket says 172.21.15.43:37736) 2024-05-27T07:34:54.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f720be00700 1 -- 172.21.15.43:0/499535241 learned_addr learned my addr 172.21.15.43:0/499535241 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:54.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f7213400700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7214105280 0x7f7214102f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f720be00700 1 -- 172.21.15.43:0/499535241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7214105280 msgr2=0x7f7214102f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:54.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f720be00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7214105280 0x7f7214102f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f720be00700 1 -- 172.21.15.43:0/499535241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721410e6f0 msgr2=0x7f72141039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:54.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f720be00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721410e6f0 0x7f72141039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f720be00700 1 -- 172.21.15.43:0/499535241 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72141a8ed0 con 0x7f7214105bd0 2024-05-27T07:34:54.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.800+0000 7f7213e00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721410e6f0 0x7f72141039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:54.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.801+0000 7f720be00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7214105bd0 0x7f72141034b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f71fc006010 tx=0x7f71fc00bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:54.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.801+0000 7f7210c00700 1 -- 172.21.15.43:0/499535241 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc027070 con 0x7f7214105bd0 2024-05-27T07:34:54.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.802+0000 7f7210c00700 1 -- 172.21.15.43:0/499535241 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71fc022470 con 0x7f7214105bd0 2024-05-27T07:34:54.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.802+0000 7f7210c00700 1 -- 172.21.15.43:0/499535241 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc017650 con 0x7f7214105bd0 2024-05-27T07:34:54.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.802+0000 7f7218c00700 1 -- 172.21.15.43:0/499535241 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72141a9160 con 0x7f7214105bd0 2024-05-27T07:34:54.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.802+0000 7f7218c00700 1 -- 172.21.15.43:0/499535241 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72141a95f0 con 0x7f7214105bd0 2024-05-27T07:34:54.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.810+0000 7f7228cecb80 1 Processor -- start 2024-05-27T07:34:54.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.812+0000 7f7210c00700 1 -- 172.21.15.43:0/499535241 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f71fc02a070 con 0x7f7214105bd0 2024-05-27T07:34:54.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.812+0000 7f7228cecb80 1 -- start start 2024-05-27T07:34:54.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 45741 5.85 MB/sec execute 295 sec latency 364.920 ms 2024-05-27T07:34:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.813+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87ddeb9f0 0x55e87de7f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.813+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87dcffe40 0x55e87de7fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.813+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 0x55e87de8b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.813+0000 7f7228cecb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e87ddace40 con 0x55e87de892c0 2024-05-27T07:34:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.813+0000 7f7228cecb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e87dd9c9e0 con 0x55e87ddeb9f0 2024-05-27T07:34:54.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.813+0000 7f7228cecb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e87dda0a20 con 0x55e87dcffe40 2024-05-27T07:34:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.814+0000 7f7213e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87ddeb9f0 0x55e87de7f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.814+0000 7f7213e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87ddeb9f0 0x55e87de7f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37752/0 (socket says 172.21.15.43:37752) 2024-05-27T07:34:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.814+0000 7f7213e00700 1 -- 172.21.15.43:0/2565505496 learned_addr learned my addr 172.21.15.43:0/2565505496 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.814+0000 7f7213400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87dcffe40 0x55e87de7fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.814+0000 7f720be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 0x55e87de8b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.815+0000 7f7210c00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f71f80b2a70 0x7f71f80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.815+0000 7f7210c00700 1 -- 172.21.15.43:0/499535241 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f71fc0e39f0 con 0x7f7214105bd0 2024-05-27T07:34:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.816+0000 7f720be00700 1 -- 172.21.15.43:0/2565505496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87dcffe40 msgr2=0x55e87de7fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.816+0000 7f720be00700 1 --2- 172.21.15.43:0/2565505496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87dcffe40 0x55e87de7fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.816+0000 7f720be00700 1 -- 172.21.15.43:0/2565505496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87ddeb9f0 msgr2=0x55e87de7f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:54.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.816+0000 7f720be00700 1 --2- 172.21.15.43:0/2565505496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87ddeb9f0 0x55e87de7f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.816+0000 7f720be00700 1 -- 172.21.15.43:0/2565505496 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e87dd65420 con 0x55e87de892c0 2024-05-27T07:34:54.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.816+0000 7f7213e00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f71f80b2a70 0x7f71f80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.816+0000 7f720be00700 1 --2- 172.21.15.43:0/2565505496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 0x55e87de8b6e0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0e0d10 tx=0x7f71fc009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726e20e8eeae7bf9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:54.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.817+0000 7f7208c00700 1 -- 172.21.15.43:0/2565505496 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc0aa0a0 con 0x55e87de892c0 2024-05-27T07:34:54.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.817+0000 7f7208c00700 1 -- 172.21.15.43:0/2565505496 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71fc090e50 con 0x55e87de892c0 2024-05-27T07:34:54.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.817+0000 7f7208c00700 1 -- 172.21.15.43:0/2565505496 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc0b3700 con 0x55e87de892c0 2024-05-27T07:34:54.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.820+0000 7f7228cecb80 1 -- 172.21.15.43:0/2565505496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 msgr2=0x55e87de8b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:54.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.820+0000 7f7213e00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f71f80b2a70 0x7f71f80b4f80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x55e87de7ba90 tx=0x7f720c00b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:54.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.820+0000 7f7228cecb80 1 --2- 172.21.15.43:0/2565505496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 0x55e87de8b6e0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0e0d10 tx=0x7f71fc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.820+0000 7f7228cecb80 1 -- 172.21.15.43:0/2565505496 shutdown_connections 2024-05-27T07:34:54.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.820+0000 7f7228cecb80 1 --2- 172.21.15.43:0/2565505496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 0x55e87de8b6e0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.820+0000 7f7228cecb80 1 --2- 172.21.15.43:0/2565505496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87ddeb9f0 0x55e87de7f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.820+0000 7f7228cecb80 1 --2- 172.21.15.43:0/2565505496 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87dcffe40 0x55e87de7fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.820+0000 7f7228cecb80 1 -- 172.21.15.43:0/2565505496 >> 172.21.15.43:0/2565505496 conn(0x55e87dd99390 msgr2=0x55e87de7a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:54.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.821+0000 7f7228cecb80 1 -- 172.21.15.43:0/2565505496 shutdown_connections 2024-05-27T07:34:54.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.821+0000 7f7228cecb80 1 -- 172.21.15.43:0/2565505496 wait complete. 2024-05-27T07:34:54.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.822+0000 7f7228cecb80 1 -- start start 2024-05-27T07:34:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.822+0000 7f7228cecb80 1 Processor -- start 2024-05-27T07:34:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.823+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 0x55e87de861f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.823+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de86730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.823+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de86c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.823+0000 7f7228cecb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e87dca2fa0 con 0x55e87dcffe40 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.823+0000 7f7228cecb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e87dd3d760 con 0x55e87de892c0 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.823+0000 7f7228cecb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e87ddace40 con 0x55e87ddeb9f0 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.823+0000 7f7228cecb80 10 client.0 _ll_register_callbacks cb 0x55e87dd46c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.824+0000 7f7213e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de86c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.824+0000 7f7213e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de86c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37756/0 (socket says 172.21.15.43:37756) 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.824+0000 7f7213e00700 1 -- 172.21.15.43:0/3358940801 learned_addr learned my addr 172.21.15.43:0/3358940801 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.824+0000 7f720be00700 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de86730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.824+0000 7f7213400700 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 0x55e87de861f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.825+0000 7f7213e00700 1 -- 172.21.15.43:0/3358940801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 msgr2=0x55e87de86730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.825+0000 7f7213e00700 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de86730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.825+0000 7f7213e00700 1 -- 172.21.15.43:0/3358940801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 msgr2=0x55e87de861f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.825+0000 7f7213e00700 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 0x55e87de861f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.825+0000 7f7213e00700 1 -- 172.21.15.43:0/3358940801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e87dcec8d0 con 0x55e87de892c0 2024-05-27T07:34:54.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.825+0000 7f7213400700 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 0x55e87de861f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:34:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.825+0000 7f7213e00700 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de86c70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f720c016c00 tx=0x7f720c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=819834de590f32a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.825+0000 7f7201600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:34:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.826+0000 7f7201600700 1 -- 172.21.15.43:0/3358940801 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720c026060 con 0x55e87de892c0 2024-05-27T07:34:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.826+0000 7f7228cecb80 1 -- 172.21.15.43:0/3358940801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71fc090e50 con 0x55e87de892c0 2024-05-27T07:34:54.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.826+0000 7f7201600700 1 -- 172.21.15.43:0/3358940801 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f720c015a30 con 0x55e87de892c0 2024-05-27T07:34:54.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.827+0000 7f7201600700 1 -- 172.21.15.43:0/3358940801 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) ==== 24+0+0 (secure 0 0 0) 0x7f720c01c070 con 0x55e87de892c0 2024-05-27T07:34:54.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.827+0000 7f7228cecb80 10 client.37568 fetch_fsmap learned FSMap version 141 2024-05-27T07:34:54.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.827+0000 7f7228cecb80 1 -- 172.21.15.43:0/3358940801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=4+}) -- 0x55e87dd6d060 con 0x55e87de892c0 2024-05-27T07:34:54.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.827+0000 7f7201600700 1 -- 172.21.15.43:0/3358940801 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71fc090e50 con 0x55e87de892c0 2024-05-27T07:34:54.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.827+0000 7f7201600700 1 -- 172.21.15.43:0/3358940801 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 141) ==== 4203+0+0 (secure 0 0 0) 0x7f720c02abc0 con 0x55e87de892c0 2024-05-27T07:34:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.828+0000 7f7228cecb80 10 client.37568 fetch_fsmap finished waiting for FSMap version 141 2024-05-27T07:34:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.828+0000 7f7228cecb80 10 client.37568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.828+0000 7f7228cecb80 20 client.37568 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.828+0000 7f7228cecb80 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e87dea9c30 0x55e87deac120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.828+0000 7f7228cecb80 4 client.37568 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.828+0000 7f7228cecb80 1 -- 172.21.15.43:0/3358940801 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e87ddaed90 con 0x55e87dea9c30 2024-05-27T07:34:54.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.828+0000 7f7213400700 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e87dea9c30 0x55e87deac120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.829+0000 7f7213400700 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e87dea9c30 0x55e87deac120 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6777fbffcea448bb server_cookie=ec7290e3794610ef in_seq=0 out_seq=0 2024-05-27T07:34:54.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.829+0000 7f7201600700 10 client.37568 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:54.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.831+0000 7f7201600700 1 -- 172.21.15.43:0/3358940801 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e87ddaed90 con 0x55e87dea9c30 2024-05-27T07:34:54.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.831+0000 7f7201600700 10 client.37568 handle_command_reply: tid=0 2024-05-27T07:34:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.831+0000 7f7228cecb80 1 client.37568 shutdown 2024-05-27T07:34:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.831+0000 7f7228cecb80 1 -- 172.21.15.43:0/3358940801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 msgr2=0x55e87de86c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.831+0000 7f7228cecb80 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de86c70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f720c016c00 tx=0x7f720c009040 comp rx=0 tx=0).stop 2024-05-27T07:34:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.832+0000 7f7228cecb80 1 -- 172.21.15.43:0/3358940801 shutdown_connections 2024-05-27T07:34:54.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.832+0000 7f7228cecb80 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de86c70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.832+0000 7f7228cecb80 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e87dea9c30 0x55e87deac120 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.832+0000 7f7228cecb80 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 0x55e87de861f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.832+0000 7f7228cecb80 1 --2- 172.21.15.43:0/3358940801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de86730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.832+0000 7f7228cecb80 1 -- 172.21.15.43:0/3358940801 >> 172.21.15.43:0/3358940801 conn(0x55e87dd99390 msgr2=0x55e87de7f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.832+0000 7f7201600700 0 client.37568 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.832+0000 7f7228cecb80 1 -- 172.21.15.43:0/3358940801 shutdown_connections 2024-05-27T07:34:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.832+0000 7f7228cecb80 1 -- 172.21.15.43:0/3358940801 wait complete. 2024-05-27T07:34:54.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.832+0000 7f7228cecb80 20 client.37568 trim_cache size 0 max 16384 2024-05-27T07:34:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.863+0000 7f7228cecb80 1 Processor -- start 2024-05-27T07:34:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.863+0000 7f7228cecb80 1 -- start start 2024-05-27T07:34:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.863+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 0x55e87dea5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.864+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de7f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.864+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de7fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.864+0000 7f7228cecb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e87de85c70 con 0x55e87dcffe40 2024-05-27T07:34:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.864+0000 7f7228cecb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f72141aa350 con 0x55e87de892c0 2024-05-27T07:34:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.864+0000 7f7228cecb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7214074820 con 0x55e87ddeb9f0 2024-05-27T07:34:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.864+0000 7f7213400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 0x55e87dea5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.865+0000 7f7213e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de7fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.865+0000 7f7213e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de7fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37760/0 (socket says 172.21.15.43:37760) 2024-05-27T07:34:54.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.865+0000 7f7213e00700 1 -- 172.21.15.43:0/3410109383 learned_addr learned my addr 172.21.15.43:0/3410109383 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:54.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.865+0000 7f720be00700 1 --2- 172.21.15.43:0/3410109383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de7f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.867+0000 7f7213e00700 1 -- 172.21.15.43:0/3410109383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 msgr2=0x55e87de7f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.867+0000 7f7213e00700 1 --2- 172.21.15.43:0/3410109383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de7f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.867+0000 7f7213e00700 1 -- 172.21.15.43:0/3410109383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 msgr2=0x55e87dea5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:54.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.867+0000 7f7213e00700 1 --2- 172.21.15.43:0/3410109383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 0x55e87dea5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.867+0000 7f7213e00700 1 -- 172.21.15.43:0/3410109383 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e87de801d0 con 0x55e87de892c0 2024-05-27T07:34:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.868+0000 7f7213e00700 1 --2- 172.21.15.43:0/3410109383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de7fc90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f720c009b00 tx=0x7f720c0158c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df9a935abc199f28 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.869+0000 7f7203e00700 1 -- 172.21.15.43:0/3410109383 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720c037020 con 0x55e87de892c0 2024-05-27T07:34:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.869+0000 7f7203e00700 1 -- 172.21.15.43:0/3410109383 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f720c01d040 con 0x55e87de892c0 2024-05-27T07:34:54.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.869+0000 7f7203e00700 1 -- 172.21.15.43:0/3410109383 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f720c01c070 con 0x55e87de892c0 2024-05-27T07:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.871+0000 7f7228cecb80 1 -- 172.21.15.43:0/3410109383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 msgr2=0x55e87de7fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.871+0000 7f7228cecb80 1 --2- 172.21.15.43:0/3410109383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de7fc90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f720c009b00 tx=0x7f720c0158c0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.872+0000 7f7228cecb80 1 -- 172.21.15.43:0/3410109383 shutdown_connections 2024-05-27T07:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.872+0000 7f7228cecb80 1 --2- 172.21.15.43:0/3410109383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87de892c0 0x55e87de7fc90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.872+0000 7f7228cecb80 1 --2- 172.21.15.43:0/3410109383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87dcffe40 0x55e87dea5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.872+0000 7f7228cecb80 1 --2- 172.21.15.43:0/3410109383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de7f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.872+0000 7f7228cecb80 1 -- 172.21.15.43:0/3410109383 >> 172.21.15.43:0/3410109383 conn(0x55e87dd99390 msgr2=0x55e87de84ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.873+0000 7f7228cecb80 1 -- 172.21.15.43:0/3410109383 shutdown_connections 2024-05-27T07:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.873+0000 7f7228cecb80 1 -- 172.21.15.43:0/3410109383 wait complete. 2024-05-27T07:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.874+0000 7f7228cecb80 1 -- start start 2024-05-27T07:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.874+0000 7f7228cecb80 1 Processor -- start 2024-05-27T07:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.876+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87dcffe40 0x55e87de9ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.876+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de9d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.876+0000 7f7228cecb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 0x55e87de9d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.876+0000 7f7228cecb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7214074f50 con 0x55e87de892c0 2024-05-27T07:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.876+0000 7f7228cecb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f72141a9c80 con 0x55e87dcffe40 2024-05-27T07:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.876+0000 7f7228cecb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7214075680 con 0x55e87ddeb9f0 2024-05-27T07:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.876+0000 7f7228cecb80 10 client.0 _ll_register_callbacks cb 0x55e87dd46c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.877+0000 7f7213400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87dcffe40 0x55e87de9ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.877+0000 7f720be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de9d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.877+0000 7f720be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de9d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:43036/0 (socket says 172.21.15.43:43036) 2024-05-27T07:34:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.877+0000 7f720be00700 1 -- 172.21.15.43:0/4235558907 learned_addr learned my addr 172.21.15.43:0/4235558907 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.877+0000 7f7213e00700 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 0x55e87de9d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.879+0000 7f720be00700 1 -- 172.21.15.43:0/4235558907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87dcffe40 msgr2=0x55e87de9ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:54.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.879+0000 7f720be00700 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87dcffe40 0x55e87de9ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.879+0000 7f720be00700 1 -- 172.21.15.43:0/4235558907 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 msgr2=0x55e87de9d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.879+0000 7f720be00700 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 0x55e87de9d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.879+0000 7f720be00700 1 -- 172.21.15.43:0/4235558907 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e87df0c770 con 0x55e87ddeb9f0 2024-05-27T07:34:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.880+0000 7f720be00700 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de9d200 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0aa870 tx=0x7f71fc0b3140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a88ebbc1c3590485 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:54.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.880+0000 7f7201600700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:34:54.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.880+0000 7f7201600700 1 -- 172.21.15.43:0/4235558907 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc0aa440 con 0x55e87ddeb9f0 2024-05-27T07:34:54.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.880+0000 7f7201600700 1 -- 172.21.15.43:0/4235558907 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f71fc0cd0c0 con 0x55e87ddeb9f0 2024-05-27T07:34:54.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.881+0000 7f7228cecb80 1 -- 172.21.15.43:0/4235558907 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f720c01d040 con 0x55e87ddeb9f0 2024-05-27T07:34:54.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.881+0000 7f7201600700 1 -- 172.21.15.43:0/4235558907 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) ==== 24+0+0 (secure 0 0 0) 0x7f71fc0ab030 con 0x55e87ddeb9f0 2024-05-27T07:34:54.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.882+0000 7f7228cecb80 10 client.27625 fetch_fsmap learned FSMap version 141 2024-05-27T07:34:54.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.882+0000 7f7228cecb80 1 -- 172.21.15.43:0/4235558907 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=4+}) -- 0x55e87df0d9d0 con 0x55e87ddeb9f0 2024-05-27T07:34:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.882+0000 7f7201600700 1 -- 172.21.15.43:0/4235558907 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f720c01d040 con 0x55e87ddeb9f0 2024-05-27T07:34:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.883+0000 7f7201600700 1 -- 172.21.15.43:0/4235558907 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 141) ==== 4203+0+0 (secure 0 0 0) 0x7f71fc022920 con 0x55e87ddeb9f0 2024-05-27T07:34:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.883+0000 7f7228cecb80 10 client.27625 fetch_fsmap finished waiting for FSMap version 141 2024-05-27T07:34:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.883+0000 7f7228cecb80 10 client.27625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.883+0000 7f7228cecb80 20 client.27625 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:54.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.883+0000 7f7228cecb80 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e87df0e010 0x55e87df10430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:54.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.883+0000 7f7228cecb80 4 client.27625 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:34:54.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.883+0000 7f7228cecb80 1 -- 172.21.15.43:0/4235558907 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e87dda8570 con 0x55e87df0e010 2024-05-27T07:34:54.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:54 smithi043 ceph-mon[31531]: pgmap v704: 129 pgs: 129 active+clean; 1022 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 131 op/s 2024-05-27T07:34:54.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.883+0000 7f7213400700 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e87df0e010 0x55e87df10430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:54.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.884+0000 7f7201600700 10 client.27625 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:54.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.885+0000 7f7213400700 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e87df0e010 0x55e87df10430 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da83d46e9fdce2 server_cookie=8fb199426ac5b3a9 in_seq=0 out_seq=0 2024-05-27T07:34:54.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.885+0000 7f7201600700 1 -- 172.21.15.43:0/4235558907 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e87dda8570 con 0x55e87df0e010 2024-05-27T07:34:54.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.885+0000 7f7201600700 10 client.27625 handle_command_reply: tid=0 2024-05-27T07:34:54.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.885+0000 7f7228cecb80 1 client.27625 shutdown 2024-05-27T07:34:54.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.886+0000 7f7228cecb80 1 -- 172.21.15.43:0/4235558907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 msgr2=0x55e87de9d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:54.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.886+0000 7f7228cecb80 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de9d200 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0aa870 tx=0x7f71fc0b3140 comp rx=0 tx=0).stop 2024-05-27T07:34:54.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.888+0000 7f7228cecb80 1 -- 172.21.15.43:0/4235558907 shutdown_connections 2024-05-27T07:34:54.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.888+0000 7f7228cecb80 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e87df0e010 0x55e87df10430 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.888+0000 7f7228cecb80 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e87de892c0 0x55e87de9d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.888+0000 7f7228cecb80 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87dcffe40 0x55e87de9ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.888+0000 7f7228cecb80 1 --2- 172.21.15.43:0/4235558907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e87ddeb9f0 0x55e87de9d200 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.888+0000 7f7228cecb80 1 -- 172.21.15.43:0/4235558907 >> 172.21.15.43:0/4235558907 conn(0x55e87dd99390 msgr2=0x55e87deabc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:54.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.888+0000 7f7201600700 0 client.27625 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:54.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.888+0000 7f7228cecb80 1 -- 172.21.15.43:0/4235558907 shutdown_connections 2024-05-27T07:34:54.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.888+0000 7f7228cecb80 1 -- 172.21.15.43:0/4235558907 wait complete. 2024-05-27T07:34:54.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.888+0000 7f7228cecb80 20 client.27625 trim_cache size 0 max 16384 2024-05-27T07:34:54.886 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:34:54.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.890+0000 7f7218c00700 1 -- 172.21.15.43:0/499535241 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f71f80b2a70 msgr2=0x7f71f80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:54.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.890+0000 7f7218c00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f71f80b2a70 0x7f71f80b4f80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x55e87de7ba90 tx=0x7f720c00b040 comp rx=0 tx=0).stop 2024-05-27T07:34:54.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.890+0000 7f7218c00700 1 -- 172.21.15.43:0/499535241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7214105bd0 msgr2=0x7f72141034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:54.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.890+0000 7f7218c00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7214105bd0 0x7f72141034b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f71fc006010 tx=0x7f71fc00bfc0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.891+0000 7f7218c00700 1 -- 172.21.15.43:0/499535241 shutdown_connections 2024-05-27T07:34:54.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.891+0000 7f7218c00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f71f80b2a70 0x7f71f80b4f80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.891+0000 7f7218c00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f721410e6f0 0x7f72141039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.891+0000 7f7218c00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7214105280 0x7f7214102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.891+0000 7f7218c00700 1 --2- 172.21.15.43:0/499535241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7214105bd0 0x7f72141034b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:54.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.891+0000 7f7218c00700 1 -- 172.21.15.43:0/499535241 >> 172.21.15.43:0/499535241 conn(0x7f7214078ab0 msgr2=0x7f72140ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:54.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.891+0000 7f7218c00700 1 -- 172.21.15.43:0/499535241 shutdown_connections 2024-05-27T07:34:54.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:54.892+0000 7f7218c00700 1 -- 172.21.15.43:0/499535241 wait complete. 2024-05-27T07:34:54.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:54 smithi179 ceph-mon[35888]: pgmap v704: 129 pgs: 129 active+clean; 1022 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 131 op/s 2024-05-27T07:34:55.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:54 smithi132 ceph-mon[35283]: pgmap v704: 129 pgs: 129 active+clean; 1022 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 131 op/s 2024-05-27T07:34:55.810 INFO:tasks.workunit.client.0.smithi043.stdout: 10 45855 5.84 MB/sec execute 296 sec latency 105.809 ms 2024-05-27T07:34:56.736 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:34:56.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:34:56.810 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46018 5.88 MB/sec execute 297 sec latency 73.084 ms 2024-05-27T07:34:56.897 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:56 smithi132 ceph-mon[35283]: pgmap v705: 129 pgs: 129 active+clean; 1022 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 131 op/s 2024-05-27T07:34:56.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:56 smithi043 ceph-mon[31531]: pgmap v705: 129 pgs: 129 active+clean; 1022 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 131 op/s 2024-05-27T07:34:56.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:56 smithi179 ceph-mon[35888]: pgmap v705: 129 pgs: 129 active+clean; 1022 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 131 op/s 2024-05-27T07:34:57.330 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:34:57.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46123 5.87 MB/sec execute 298 sec latency 75.679 ms 2024-05-27T07:34:58.810 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46234 5.87 MB/sec execute 299 sec latency 84.667 ms 2024-05-27T07:34:58.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:58 smithi043 ceph-mon[31531]: pgmap v706: 129 pgs: 129 active+clean; 1.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 4.5 MiB/s wr, 210 op/s 2024-05-27T07:34:58.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:58 smithi179 ceph-mon[35888]: pgmap v706: 129 pgs: 129 active+clean; 1.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 4.5 MiB/s wr, 210 op/s 2024-05-27T07:34:59.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:58 smithi132 ceph-mon[35283]: pgmap v706: 129 pgs: 129 active+clean; 1.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 4.5 MiB/s wr, 210 op/s 2024-05-27T07:34:59.693 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:34:59 smithi043 ceph-mon[31531]: pgmap v707: 129 pgs: 129 active+clean; 1.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-05-27T07:34:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.749+0000 7f970e800700 1 -- 172.21.15.43:0/789396787 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9708105c90 msgr2=0x7f970810e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.749+0000 7f970e800700 1 --2- 172.21.15.43:0/789396787 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9708105c90 0x7f970810e1e0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f96f40098f0 tx=0x7f96f4009c00 comp rx=0 tx=0).stop 2024-05-27T07:34:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.750+0000 7f970e800700 1 -- 172.21.15.43:0/789396787 shutdown_connections 2024-05-27T07:34:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.750+0000 7f970e800700 1 --2- 172.21.15.43:0/789396787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f970810e720 0x7f9708110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.750+0000 7f970e800700 1 --2- 172.21.15.43:0/789396787 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9708105340 0x7f9708105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.750+0000 7f970e800700 1 --2- 172.21.15.43:0/789396787 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9708105c90 0x7f970810e1e0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.750+0000 7f970e800700 1 -- 172.21.15.43:0/789396787 >> 172.21.15.43:0/789396787 conn(0x7f9708078ab0 msgr2=0x7f9708078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.750+0000 7f970e800700 1 -- 172.21.15.43:0/789396787 shutdown_connections 2024-05-27T07:34:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.750+0000 7f970e800700 1 -- 172.21.15.43:0/789396787 wait complete. 2024-05-27T07:34:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.751+0000 7f970e800700 1 Processor -- start 2024-05-27T07:34:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.751+0000 7f970e800700 1 -- start start 2024-05-27T07:34:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.752+0000 7f970e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9708105340 0x7f97081a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.752+0000 7f970e800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9708105c90 0x7f97081a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.752+0000 7f970e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f970810e720 0x7f970819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.752+0000 7f970e800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9708114150 con 0x7f9708105340 2024-05-27T07:34:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.752+0000 7f970e800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9708113fd0 con 0x7f970810e720 2024-05-27T07:34:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.752+0000 7f970e800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f97081142d0 con 0x7f9708105c90 2024-05-27T07:34:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.753+0000 7f970ca00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9708105c90 0x7f97081a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.753+0000 7f970ca00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9708105c90 0x7f97081a0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:43066/0 (socket says 172.21.15.43:43066) 2024-05-27T07:34:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.753+0000 7f970ca00700 1 -- 172.21.15.43:0/1675920127 learned_addr learned my addr 172.21.15.43:0/1675920127 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.753+0000 7f970de00700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f970810e720 0x7f970819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.754+0000 7f970de00700 1 -- 172.21.15.43:0/1675920127 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9708105c90 msgr2=0x7f97081a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.754+0000 7f970de00700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9708105c90 0x7f97081a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.754+0000 7f970de00700 1 -- 172.21.15.43:0/1675920127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9708105340 msgr2=0x7f97081a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.754+0000 7f970de00700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9708105340 0x7f97081a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.754+0000 7f970de00700 1 -- 172.21.15.43:0/1675920127 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f970819b0e0 con 0x7f970810e720 2024-05-27T07:34:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.754+0000 7f970de00700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f970810e720 0x7f970819aa10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f970000efc0 tx=0x7f970000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.754+0000 7f9706000700 1 -- 172.21.15.43:0/1675920127 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97000092e0 con 0x7f970810e720 2024-05-27T07:34:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.755+0000 7f9706000700 1 -- 172.21.15.43:0/1675920127 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9700004bb0 con 0x7f970810e720 2024-05-27T07:34:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.755+0000 7f9706000700 1 -- 172.21.15.43:0/1675920127 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700004020 con 0x7f970810e720 2024-05-27T07:34:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.755+0000 7f970e800700 1 -- 172.21.15.43:0/1675920127 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f970819b370 con 0x7f970810e720 2024-05-27T07:34:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.755+0000 7f970e800700 1 -- 172.21.15.43:0/1675920127 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f970819b770 con 0x7f970810e720 2024-05-27T07:34:59.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.760+0000 7f9706000700 1 -- 172.21.15.43:0/1675920127 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f970001b030 con 0x7f970810e720 2024-05-27T07:34:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.764+0000 7f971e91bb80 1 Processor -- start 2024-05-27T07:34:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.764+0000 7f9706000700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f96f00b2a70 0x7f96f00b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.765+0000 7f9706000700 1 -- 172.21.15.43:0/1675920127 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f97000e2370 con 0x7f970810e720 2024-05-27T07:34:59.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.765+0000 7f970ca00700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f96f00b2a70 0x7f96f00b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.766+0000 7f971e91bb80 1 -- start start 2024-05-27T07:34:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.766+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496b2cec0 0x557496caaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.766+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cab490 0x557496cb5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.766+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496cb80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.766+0000 7f971e91bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557496bd9ec0 con 0x557496cab8a0 2024-05-27T07:34:59.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.766+0000 7f971e91bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557496bc9a60 con 0x557496cab490 2024-05-27T07:34:59.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.766+0000 7f971e91bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557496bcdaa0 con 0x557496b2cec0 2024-05-27T07:34:59.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.767+0000 7f970ca00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cab490 0x557496cb5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.767+0000 7f970ca00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cab490 0x557496cb5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37792/0 (socket says 172.21.15.43:37792) 2024-05-27T07:34:59.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.767+0000 7f970ca00700 1 -- 172.21.15.43:0/324757861 learned_addr learned my addr 172.21.15.43:0/324757861 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:59.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.768+0000 7f970d400700 1 --2- 172.21.15.43:0/324757861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496b2cec0 0x557496caaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.768+0000 7f970ca00700 1 -- 172.21.15.43:0/324757861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496b2cec0 msgr2=0x557496caaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.768+0000 7f970ca00700 1 --2- 172.21.15.43:0/324757861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496b2cec0 0x557496caaf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.768+0000 7f970ca00700 1 -- 172.21.15.43:0/324757861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 msgr2=0x557496cb80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.768+0000 7f970ca00700 1 --2- 172.21.15.43:0/324757861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496cb80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.768+0000 7f970ca00700 1 -- 172.21.15.43:0/324757861 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557496b924a0 con 0x557496cab490 2024-05-27T07:34:59.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.769+0000 7f970ca00700 1 --2- 172.21.15.43:0/324757861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cab490 0x557496cb5b50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f96f400c010 tx=0x7f96f4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81d4cf156db302 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.770+0000 7f970ca00700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f96f00b2a70 0x7f96f00b4f80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9708102800 tx=0x7f96f4019040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.770+0000 7f96faa00700 1 -- 172.21.15.43:0/324757861 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4005240 con 0x557496cab490 2024-05-27T07:34:59.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.771+0000 7f96faa00700 1 -- 172.21.15.43:0/324757861 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96f4029070 con 0x557496cab490 2024-05-27T07:34:59.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.771+0000 7f96faa00700 1 -- 172.21.15.43:0/324757861 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f400b7c0 con 0x557496cab490 2024-05-27T07:34:59.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.772+0000 7f971e91bb80 1 -- 172.21.15.43:0/324757861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cab490 msgr2=0x557496cb5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.772+0000 7f971e91bb80 1 --2- 172.21.15.43:0/324757861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cab490 0x557496cb5b50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f96f400c010 tx=0x7f96f4009260 comp rx=0 tx=0).stop 2024-05-27T07:34:59.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.773+0000 7f971e91bb80 1 -- 172.21.15.43:0/324757861 shutdown_connections 2024-05-27T07:34:59.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.773+0000 7f971e91bb80 1 --2- 172.21.15.43:0/324757861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496cb80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.773+0000 7f971e91bb80 1 --2- 172.21.15.43:0/324757861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496b2cec0 0x557496caaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.773+0000 7f971e91bb80 1 --2- 172.21.15.43:0/324757861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cab490 0x557496cb5b50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.773+0000 7f971e91bb80 1 -- 172.21.15.43:0/324757861 >> 172.21.15.43:0/324757861 conn(0x557496b8d040 msgr2=0x557496ca6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.775+0000 7f971e91bb80 1 -- 172.21.15.43:0/324757861 shutdown_connections 2024-05-27T07:34:59.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.775+0000 7f971e91bb80 1 -- 172.21.15.43:0/324757861 wait complete. 2024-05-27T07:34:59.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.776+0000 7f971e91bb80 1 -- start start 2024-05-27T07:34:59.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.776+0000 7f971e91bb80 1 Processor -- start 2024-05-27T07:34:59.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.777+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496cb5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.777+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496cb5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.778+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cb61a0 0x557496cd2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.778+0000 7f971e91bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557496ad0020 con 0x557496cab8a0 2024-05-27T07:34:59.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.778+0000 7f971e91bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557496b6a7e0 con 0x557496b2cec0 2024-05-27T07:34:59.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.778+0000 7f971e91bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557496cc99c0 con 0x557496cb61a0 2024-05-27T07:34:59.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.778+0000 7f971e91bb80 10 client.0 _ll_register_callbacks cb 0x557496b73cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:59.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.778+0000 7f970d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496cb5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.778+0000 7f970d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496cb5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37796/0 (socket says 172.21.15.43:37796) 2024-05-27T07:34:59.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.778+0000 7f970d400700 1 -- 172.21.15.43:0/2949366132 learned_addr learned my addr 172.21.15.43:0/2949366132 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:59.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.778+0000 7f970ca00700 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496cb5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.778+0000 7f970de00700 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cb61a0 0x557496cd2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.779+0000 7f970d400700 1 -- 172.21.15.43:0/2949366132 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cb61a0 msgr2=0x557496cd2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.779+0000 7f970d400700 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cb61a0 0x557496cd2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.779+0000 7f970d400700 1 -- 172.21.15.43:0/2949366132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 msgr2=0x557496cb5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.779+0000 7f970d400700 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496cb5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.779+0000 7f970d400700 1 -- 172.21.15.43:0/2949366132 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557496b19950 con 0x557496b2cec0 2024-05-27T07:34:59.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.779+0000 7f970ca00700 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496cb5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:34:59.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.781+0000 7f970d400700 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496cb5720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00e370 tx=0x7f96fc00e730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46f142daf6f4b3ac server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.781+0000 7f96ef400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:34:59.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.782+0000 7f96ef400700 1 -- 172.21.15.43:0/2949366132 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc016070 con 0x557496b2cec0 2024-05-27T07:34:59.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.782+0000 7f971e91bb80 1 -- 172.21.15.43:0/2949366132 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96f4029070 con 0x557496b2cec0 2024-05-27T07:34:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.783+0000 7f96ef400700 1 -- 172.21.15.43:0/2949366132 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f96fc004a30 con 0x557496b2cec0 2024-05-27T07:34:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.783+0000 7f96ef400700 1 -- 172.21.15.43:0/2949366132 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) ==== 24+0+0 (secure 0 0 0) 0x7f96fc021050 con 0x557496b2cec0 2024-05-27T07:34:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.783+0000 7f971e91bb80 10 client.37586 fetch_fsmap learned FSMap version 141 2024-05-27T07:34:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.783+0000 7f971e91bb80 1 -- 172.21.15.43:0/2949366132 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=4+}) -- 0x557496b9a0e0 con 0x557496b2cec0 2024-05-27T07:34:59.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.784+0000 7f96ef400700 1 -- 172.21.15.43:0/2949366132 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f96f4029070 con 0x557496b2cec0 2024-05-27T07:34:59.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.784+0000 7f96ef400700 1 -- 172.21.15.43:0/2949366132 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 141) ==== 4203+0+0 (secure 0 0 0) 0x7f96fc00f040 con 0x557496b2cec0 2024-05-27T07:34:59.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.784+0000 7f971e91bb80 10 client.37586 fetch_fsmap finished waiting for FSMap version 141 2024-05-27T07:34:59.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.784+0000 7f971e91bb80 10 client.37586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:59.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.784+0000 7f971e91bb80 20 client.37586 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:59.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.784+0000 7f971e91bb80 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x557496cd7320 0x557496cd97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.784+0000 7f971e91bb80 4 client.37586 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:34:59.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.784+0000 7f971e91bb80 1 -- 172.21.15.43:0/2949366132 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557496bdbe10 con 0x557496cd7320 2024-05-27T07:34:59.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.785+0000 7f970ca00700 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x557496cd7320 0x557496cd97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.786+0000 7f96ef400700 10 client.37586 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:59.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.786+0000 7f970ca00700 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x557496cd7320 0x557496cd97f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19bf67cbe371c108 server_cookie=5e23593bcce2e681 in_seq=0 out_seq=0 2024-05-27T07:34:59.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.788+0000 7f96ef400700 1 -- 172.21.15.43:0/2949366132 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x557496bdbe10 con 0x557496cd7320 2024-05-27T07:34:59.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.788+0000 7f96ef400700 10 client.37586 handle_command_reply: tid=0 2024-05-27T07:34:59.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.788+0000 7f971e91bb80 1 client.37586 shutdown 2024-05-27T07:34:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.788+0000 7f971e91bb80 1 -- 172.21.15.43:0/2949366132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 msgr2=0x557496cb5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.788+0000 7f971e91bb80 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496cb5720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00e370 tx=0x7f96fc00e730 comp rx=0 tx=0).stop 2024-05-27T07:34:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.789+0000 7f971e91bb80 1 -- 172.21.15.43:0/2949366132 shutdown_connections 2024-05-27T07:34:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.789+0000 7f971e91bb80 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cb61a0 0x557496cd2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.789+0000 7f971e91bb80 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496cb5720 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.789+0000 7f971e91bb80 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x557496cd7320 0x557496cd97f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.789+0000 7f971e91bb80 1 --2- 172.21.15.43:0/2949366132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496cb5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.789+0000 7f971e91bb80 1 -- 172.21.15.43:0/2949366132 >> 172.21.15.43:0/2949366132 conn(0x557496b8d040 msgr2=0x557496ca6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.789+0000 7f96ef400700 0 client.37586 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.789+0000 7f971e91bb80 1 -- 172.21.15.43:0/2949366132 shutdown_connections 2024-05-27T07:34:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.789+0000 7f971e91bb80 1 -- 172.21.15.43:0/2949366132 wait complete. 2024-05-27T07:34:59.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.790+0000 7f971e91bb80 20 client.37586 trim_cache size 0 max 16384 2024-05-27T07:34:59.810 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46359 5.88 MB/sec execute 300 sec latency 98.494 ms 2024-05-27T07:34:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.822+0000 7f971e91bb80 1 Processor -- start 2024-05-27T07:34:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.822+0000 7f971e91bb80 1 -- start start 2024-05-27T07:34:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.823+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496b2cec0 0x557496ca6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.823+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cab8a0 0x557496caf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.823+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cb61a0 0x557496caf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.823+0000 7f971e91bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557496cb4a20 con 0x557496b2cec0 2024-05-27T07:34:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.823+0000 7f971e91bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f97080701b0 con 0x557496cb61a0 2024-05-27T07:34:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.823+0000 7f971e91bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f970806e5b0 con 0x557496cab8a0 2024-05-27T07:34:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.824+0000 7f970de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cb61a0 0x557496caf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.824+0000 7f970ca00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cab8a0 0x557496caf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.824+0000 7f970de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cb61a0 0x557496caf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37800/0 (socket says 172.21.15.43:37800) 2024-05-27T07:34:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.824+0000 7f970de00700 1 -- 172.21.15.43:0/2473346351 learned_addr learned my addr 172.21.15.43:0/2473346351 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.824+0000 7f970d400700 1 --2- 172.21.15.43:0/2473346351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496b2cec0 0x557496ca6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.824+0000 7f970de00700 1 -- 172.21.15.43:0/2473346351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cab8a0 msgr2=0x557496caf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.824+0000 7f970de00700 1 --2- 172.21.15.43:0/2473346351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cab8a0 0x557496caf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.824+0000 7f970de00700 1 -- 172.21.15.43:0/2473346351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496b2cec0 msgr2=0x557496ca6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.824+0000 7f970de00700 1 --2- 172.21.15.43:0/2473346351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496b2cec0 0x557496ca6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.824+0000 7f970de00700 1 -- 172.21.15.43:0/2473346351 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557496caaac0 con 0x557496cb61a0 2024-05-27T07:34:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.825+0000 7f970de00700 1 --2- 172.21.15.43:0/2473346351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cb61a0 0x557496caf610 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f97000dfcf0 tx=0x7f97000a8a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3a837a3031c4a2f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.825+0000 7f96fa000700 1 -- 172.21.15.43:0/2473346351 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97000c3070 con 0x557496cb61a0 2024-05-27T07:34:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.825+0000 7f96fa000700 1 -- 172.21.15.43:0/2473346351 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97000e0e50 con 0x557496cb61a0 2024-05-27T07:34:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.825+0000 7f96fa000700 1 -- 172.21.15.43:0/2473346351 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97000b20e0 con 0x557496cb61a0 2024-05-27T07:34:59.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.827+0000 7f971e91bb80 1 -- 172.21.15.43:0/2473346351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cb61a0 msgr2=0x557496caf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.827+0000 7f971e91bb80 1 --2- 172.21.15.43:0/2473346351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cb61a0 0x557496caf610 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f97000dfcf0 tx=0x7f97000a8a80 comp rx=0 tx=0).stop 2024-05-27T07:34:59.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.827+0000 7f971e91bb80 1 -- 172.21.15.43:0/2473346351 shutdown_connections 2024-05-27T07:34:59.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.827+0000 7f971e91bb80 1 --2- 172.21.15.43:0/2473346351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496cb61a0 0x557496caf610 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.827+0000 7f971e91bb80 1 --2- 172.21.15.43:0/2473346351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496b2cec0 0x557496ca6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.827+0000 7f971e91bb80 1 --2- 172.21.15.43:0/2473346351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cab8a0 0x557496caf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.827+0000 7f971e91bb80 1 -- 172.21.15.43:0/2473346351 >> 172.21.15.43:0/2473346351 conn(0x557496b8d040 msgr2=0x557496cae9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.827+0000 7f971e91bb80 1 -- 172.21.15.43:0/2473346351 shutdown_connections 2024-05-27T07:34:59.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.828+0000 7f971e91bb80 1 -- 172.21.15.43:0/2473346351 wait complete. 2024-05-27T07:34:59.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.828+0000 7f971e91bb80 1 -- start start 2024-05-27T07:34:59.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.828+0000 7f971e91bb80 1 Processor -- start 2024-05-27T07:34:59.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.830+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496ccab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.830+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496ccb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.830+0000 7f971e91bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cb61a0 0x557496ccb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.830+0000 7f970d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496ccab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.830+0000 7f970d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496ccab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37812/0 (socket says 172.21.15.43:37812) 2024-05-27T07:34:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f971e91bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f970806ece0 con 0x557496cab8a0 2024-05-27T07:34:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f971e91bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f970806fae0 con 0x557496b2cec0 2024-05-27T07:34:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f971e91bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f970806f410 con 0x557496cb61a0 2024-05-27T07:34:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f970d400700 1 -- 172.21.15.43:0/4109495169 learned_addr learned my addr 172.21.15.43:0/4109495169 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:34:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f971e91bb80 10 client.0 _ll_register_callbacks cb 0x557496b73cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:34:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f970ca00700 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496ccb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f970de00700 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cb61a0 0x557496ccb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f970d400700 1 -- 172.21.15.43:0/4109495169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cb61a0 msgr2=0x557496ccb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f970d400700 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cb61a0 0x557496ccb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f970d400700 1 -- 172.21.15.43:0/4109495169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 msgr2=0x557496ccb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:34:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f970d400700 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496ccb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.831+0000 7f970d400700 1 -- 172.21.15.43:0/4109495169 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557496d40f70 con 0x557496b2cec0 2024-05-27T07:34:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.832+0000 7f970d400700 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496ccab50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f96fc010680 tx=0x7f96fc019d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac2301d42c10c10a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:34:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.832+0000 7f96ef400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:34:59.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.833+0000 7f96ef400700 1 -- 172.21.15.43:0/4109495169 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc016070 con 0x557496b2cec0 2024-05-27T07:34:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.833+0000 7f971e91bb80 1 -- 172.21.15.43:0/4109495169 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97000e0e50 con 0x557496b2cec0 2024-05-27T07:34:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.833+0000 7f96ef400700 1 -- 172.21.15.43:0/4109495169 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7f96fc004510 con 0x557496b2cec0 2024-05-27T07:34:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.834+0000 7f96ef400700 1 -- 172.21.15.43:0/4109495169 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) ==== 24+0+0 (secure 0 0 0) 0x7f96fc033050 con 0x557496b2cec0 2024-05-27T07:34:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.834+0000 7f971e91bb80 10 client.37592 fetch_fsmap learned FSMap version 141 2024-05-27T07:34:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.834+0000 7f971e91bb80 1 -- 172.21.15.43:0/4109495169 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=4+}) -- 0x557496d421d0 con 0x557496b2cec0 2024-05-27T07:34:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.834+0000 7f96ef400700 1 -- 172.21.15.43:0/4109495169 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f97000e0e50 con 0x557496b2cec0 2024-05-27T07:34:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.835+0000 7f96ef400700 1 -- 172.21.15.43:0/4109495169 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 141) ==== 4203+0+0 (secure 0 0 0) 0x7f96fc010a80 con 0x557496b2cec0 2024-05-27T07:34:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.835+0000 7f971e91bb80 10 client.37592 fetch_fsmap finished waiting for FSMap version 141 2024-05-27T07:34:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.835+0000 7f971e91bb80 10 client.37592 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:34:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.835+0000 7f971e91bb80 20 client.37592 populate_metadata read hostname 'smithi043' 2024-05-27T07:34:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.835+0000 7f971e91bb80 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x557496d428d0 0x557496d44cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:34:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.835+0000 7f971e91bb80 4 client.37592 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:34:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.835+0000 7f971e91bb80 1 -- 172.21.15.43:0/4109495169 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557496bd55f0 con 0x557496d428d0 2024-05-27T07:34:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.835+0000 7f970ca00700 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x557496d428d0 0x557496d44cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:34:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.836+0000 7f970ca00700 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x557496d428d0 0x557496d44cf0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae5b7c6eb721437a server_cookie=de5fe61eda7dd68d in_seq=0 out_seq=0 2024-05-27T07:34:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.836+0000 7f96ef400700 10 client.37592 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f96ef400700 1 -- 172.21.15.43:0/4109495169 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557496bd55f0 con 0x557496d428d0 2024-05-27T07:34:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f96ef400700 10 client.37592 handle_command_reply: tid=0 2024-05-27T07:34:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f971e91bb80 1 client.37592 shutdown 2024-05-27T07:34:59.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f971e91bb80 1 -- 172.21.15.43:0/4109495169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 msgr2=0x557496ccab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f971e91bb80 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496ccab50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f96fc010680 tx=0x7f96fc019d70 comp rx=0 tx=0).stop 2024-05-27T07:34:59.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f971e91bb80 1 -- 172.21.15.43:0/4109495169 shutdown_connections 2024-05-27T07:34:59.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f971e91bb80 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557496cb61a0 0x557496ccb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f971e91bb80 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557496b2cec0 0x557496ccab50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f971e91bb80 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x557496d428d0 0x557496d44cf0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f971e91bb80 1 --2- 172.21.15.43:0/4109495169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557496cab8a0 0x557496ccb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f971e91bb80 1 -- 172.21.15.43:0/4109495169 >> 172.21.15.43:0/4109495169 conn(0x557496b8d040 msgr2=0x557496cb09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.839+0000 7f96ef400700 0 client.37592 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:34:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.840+0000 7f971e91bb80 1 -- 172.21.15.43:0/4109495169 shutdown_connections 2024-05-27T07:34:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.840+0000 7f971e91bb80 1 -- 172.21.15.43:0/4109495169 wait complete. 2024-05-27T07:34:59.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.840+0000 7f971e91bb80 20 client.37592 trim_cache size 0 max 16384 2024-05-27T07:34:59.838 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:34:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:34:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "f78b2626-afbf-440a-89b0-3bc5ec2a2fea", 2024-05-27T07:34:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:34:59.838 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:34:59.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.842+0000 7f970e800700 1 -- 172.21.15.43:0/1675920127 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f96f00b2a70 msgr2=0x7f96f00b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.842+0000 7f970e800700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f96f00b2a70 0x7f96f00b4f80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9708102800 tx=0x7f96f4019040 comp rx=0 tx=0).stop 2024-05-27T07:34:59.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.842+0000 7f970e800700 1 -- 172.21.15.43:0/1675920127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f970810e720 msgr2=0x7f970819aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:34:59.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.842+0000 7f970e800700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f970810e720 0x7f970819aa10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f970000efc0 tx=0x7f970000c3a0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.842+0000 7f970e800700 1 -- 172.21.15.43:0/1675920127 shutdown_connections 2024-05-27T07:34:59.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.842+0000 7f970e800700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f96f00b2a70 0x7f96f00b4f80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.842+0000 7f970e800700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f970810e720 0x7f970819aa10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.842+0000 7f970e800700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9708105340 0x7f97081a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.842+0000 7f970e800700 1 --2- 172.21.15.43:0/1675920127 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9708105c90 0x7f97081a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:34:59.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.842+0000 7f970e800700 1 -- 172.21.15.43:0/1675920127 >> 172.21.15.43:0/1675920127 conn(0x7f9708078ab0 msgr2=0x7f97080fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:34:59.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.843+0000 7f970e800700 1 -- 172.21.15.43:0/1675920127 shutdown_connections 2024-05-27T07:34:59.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:34:59.843+0000 7f970e800700 1 -- 172.21.15.43:0/1675920127 wait complete. 2024-05-27T07:34:59.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:34:59 smithi179 ceph-mon[35888]: pgmap v707: 129 pgs: 129 active+clean; 1.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-05-27T07:35:00.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:34:59 smithi132 ceph-mon[35283]: pgmap v707: 129 pgs: 129 active+clean; 1.0 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.696 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.697 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:00 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.810 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46457 5.88 MB/sec execute 301 sec latency 137.877 ms 2024-05-27T07:35:00.949 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:00.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:00 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:01.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:35:01.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:01.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:35:01.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:35:01.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:35:01.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:35:01.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:35:01.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:01.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:35:01.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:01.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:01.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:01.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:00 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:01.433 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:01.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46468 5.86 MB/sec execute 302 sec latency 883.261 ms 2024-05-27T07:35:01.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:01 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:01.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:01 smithi043 ceph-mon[31531]: pgmap v708: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-05-27T07:35:01.863 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:01.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:01 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:01.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:01 smithi179 ceph-mon[35888]: pgmap v708: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-05-27T07:35:01.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:02.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:01 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:35:02.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:01 smithi132 ceph-mon[35283]: pgmap v708: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 5.1 MiB/s wr, 229 op/s 2024-05-27T07:35:02.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:35:02.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46468 5.84 MB/sec execute 303 sec latency 1883.401 ms 2024-05-27T07:35:03.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46545 5.83 MB/sec execute 304 sec latency 1950.732 ms 2024-05-27T07:35:04.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.183+0000 7fc77c600700 1 -- 172.21.15.43:0/1654749595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc774108690 msgr2=0x7fc774108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.183+0000 7fc77c600700 1 --2- 172.21.15.43:0/1654749595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc774108690 0x7fc774108b60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009f90 tx=0x7fc76c00b330 comp rx=0 tx=0).stop 2024-05-27T07:35:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.183+0000 7fc77c600700 1 -- 172.21.15.43:0/1654749595 shutdown_connections 2024-05-27T07:35:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.183+0000 7fc77c600700 1 --2- 172.21.15.43:0/1654749595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc774108690 0x7fc774108b60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.183+0000 7fc77c600700 1 --2- 172.21.15.43:0/1654749595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc774103240 0x7fc774103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.183+0000 7fc77c600700 1 --2- 172.21.15.43:0/1654749595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc774103b90 0x7fc7741080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.183+0000 7fc77c600700 1 -- 172.21.15.43:0/1654749595 >> 172.21.15.43:0/1654749595 conn(0x7fc7740fedb0 msgr2=0x7fc7741011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.183+0000 7fc77c600700 1 -- 172.21.15.43:0/1654749595 shutdown_connections 2024-05-27T07:35:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.183+0000 7fc77c600700 1 -- 172.21.15.43:0/1654749595 wait complete. 2024-05-27T07:35:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.185+0000 7fc77c600700 1 Processor -- start 2024-05-27T07:35:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.185+0000 7fc77c600700 1 -- start start 2024-05-27T07:35:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.186+0000 7fc77c600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc774103240 0x7fc774075390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.186+0000 7fc77c600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc774103b90 0x7fc77406e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.186+0000 7fc77c600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc774108690 0x7fc77406eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.186+0000 7fc77c600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc77411a930 con 0x7fc774103240 2024-05-27T07:35:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.186+0000 7fc77c600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc77411a7b0 con 0x7fc774108690 2024-05-27T07:35:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.186+0000 7fc77c600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc77411aab0 con 0x7fc774103b90 2024-05-27T07:35:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.186+0000 7fc779c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc774103240 0x7fc774075390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.186+0000 7fc779c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc774103240 0x7fc774075390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51556/0 (socket says 172.21.15.43:51556) 2024-05-27T07:35:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.186+0000 7fc779c00700 1 -- 172.21.15.43:0/3171009269 learned_addr learned my addr 172.21.15.43:0/3171009269 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.187+0000 7fc779c00700 1 -- 172.21.15.43:0/3171009269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc774103b90 msgr2=0x7fc77406e4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:35:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.187+0000 7fc779c00700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc774103b90 0x7fc77406e4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.187+0000 7fc779c00700 1 -- 172.21.15.43:0/3171009269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc774108690 msgr2=0x7fc77406eb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:35:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.187+0000 7fc779c00700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc774108690 0x7fc77406eb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.187+0000 7fc779c00700 1 -- 172.21.15.43:0/3171009269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc774072ec0 con 0x7fc774103240 2024-05-27T07:35:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.188+0000 7fc779c00700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc774103240 0x7fc774075390 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fc76400e9c0 tx=0x7fc76400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:04.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.188+0000 7fc772000700 1 -- 172.21.15.43:0/3171009269 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76400cbd0 con 0x7fc774103240 2024-05-27T07:35:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.188+0000 7fc772000700 1 -- 172.21.15.43:0/3171009269 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc76400cd70 con 0x7fc774103240 2024-05-27T07:35:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.188+0000 7fc772000700 1 -- 172.21.15.43:0/3171009269 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7640055f0 con 0x7fc774103240 2024-05-27T07:35:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.189+0000 7fc77c600700 1 -- 172.21.15.43:0/3171009269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7740731b0 con 0x7fc774103240 2024-05-27T07:35:04.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.189+0000 7fc77c600700 1 -- 172.21.15.43:0/3171009269 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc774073640 con 0x7fc774103240 2024-05-27T07:35:04.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.193+0000 7fc772000700 1 -- 172.21.15.43:0/3171009269 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fc764004750 con 0x7fc774103240 2024-05-27T07:35:04.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.195+0000 7fc772000700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc75c0b2a20 0x7fc75c0b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.195+0000 7fc772000700 1 -- 172.21.15.43:0/3171009269 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7fc7640e2e00 con 0x7fc774103240 2024-05-27T07:35:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.195+0000 7fc779200700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc75c0b2a20 0x7fc75c0b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.195+0000 7fc78b178b80 1 Processor -- start 2024-05-27T07:35:04.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.195+0000 7fc78b178b80 1 -- start start 2024-05-27T07:35:04.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc779200700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc75c0b2a20 0x7fc75c0b4f30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc774108ae0 tx=0x7fc768009310 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:04.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fdf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0fd9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 0x55dae0fda1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc779c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fdf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc78b178b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dae0f07e40 con 0x55dae0fd9db0 2024-05-27T07:35:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc78b178b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dae0ef79e0 con 0x55dae0e5ae40 2024-05-27T07:35:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc78b178b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55dae0efba20 con 0x55dae0fdfd30 2024-05-27T07:35:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc779c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fdf7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37846/0 (socket says 172.21.15.43:37846) 2024-05-27T07:35:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc779c00700 1 -- 172.21.15.43:0/3353378772 learned_addr learned my addr 172.21.15.43:0/3353378772 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.196+0000 7fc77a600700 1 --2- 172.21.15.43:0/3353378772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 0x55dae0fda1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc779200700 1 --2- 172.21.15.43:0/3353378772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0fd9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc779c00700 1 -- 172.21.15.43:0/3353378772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 msgr2=0x55dae0fd9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc779c00700 1 --2- 172.21.15.43:0/3353378772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0fd9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc779c00700 1 -- 172.21.15.43:0/3353378772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 msgr2=0x55dae0fda1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc779c00700 1 --2- 172.21.15.43:0/3353378772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 0x55dae0fda1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc779c00700 1 -- 172.21.15.43:0/3353378772 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dae0ec0420 con 0x55dae0e5ae40 2024-05-27T07:35:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc779200700 1 --2- 172.21.15.43:0/3353378772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0fd9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc779c00700 1 --2- 172.21.15.43:0/3353378772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fdf7f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc7640a5d30 tx=0x7fc7640a94a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4cb86e8e8e20796 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc762a00700 1 -- 172.21.15.43:0/3353378772 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7640b2460 con 0x55dae0e5ae40 2024-05-27T07:35:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc762a00700 1 -- 172.21.15.43:0/3353378772 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7640aa3b0 con 0x55dae0e5ae40 2024-05-27T07:35:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.197+0000 7fc762a00700 1 -- 172.21.15.43:0/3353378772 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7640a9f10 con 0x55dae0e5ae40 2024-05-27T07:35:04.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.198+0000 7fc78b178b80 1 -- 172.21.15.43:0/3353378772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 msgr2=0x55dae0fdf7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.199+0000 7fc78b178b80 1 --2- 172.21.15.43:0/3353378772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fdf7f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc7640a5d30 tx=0x7fc7640a94a0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.199+0000 7fc78b178b80 1 -- 172.21.15.43:0/3353378772 shutdown_connections 2024-05-27T07:35:04.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.199+0000 7fc78b178b80 1 --2- 172.21.15.43:0/3353378772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 0x55dae0fda1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.199+0000 7fc78b178b80 1 --2- 172.21.15.43:0/3353378772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fdf7f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.199+0000 7fc78b178b80 1 --2- 172.21.15.43:0/3353378772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0fd9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.199+0000 7fc78b178b80 1 -- 172.21.15.43:0/3353378772 >> 172.21.15.43:0/3353378772 conn(0x55dae0ebafc0 msgr2=0x55dae0fd48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:04.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.199+0000 7fc78b178b80 1 -- 172.21.15.43:0/3353378772 shutdown_connections 2024-05-27T07:35:04.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.199+0000 7fc78b178b80 1 -- 172.21.15.43:0/3353378772 wait complete. 2024-05-27T07:35:04.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.199+0000 7fc78b178b80 1 -- start start 2024-05-27T07:35:04.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.200+0000 7fc78b178b80 1 Processor -- start 2024-05-27T07:35:04.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.200+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fe34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.200+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fd9db0 0x55dae0fe3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.200+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fdfd30 0x55dae0fe3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.200+0000 7fc78b178b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dae0dfdfa0 con 0x55dae0fdfd30 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.200+0000 7fc78b178b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dae0e98760 con 0x55dae0e5ae40 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.200+0000 7fc78b178b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55dae0fe6800 con 0x55dae0fd9db0 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.200+0000 7fc78b178b80 10 client.0 _ll_register_callbacks cb 0x55dae0ea1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc779c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fe34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc779c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fe34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37848/0 (socket says 172.21.15.43:37848) 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc779c00700 1 -- 172.21.15.43:0/1987309150 learned_addr learned my addr 172.21.15.43:0/1987309150 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc77a600700 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fdfd30 0x55dae0fe3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc779200700 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fd9db0 0x55dae0fe3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc779c00700 1 -- 172.21.15.43:0/1987309150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fd9db0 msgr2=0x55dae0fe3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc779c00700 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fd9db0 0x55dae0fe3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc779c00700 1 -- 172.21.15.43:0/1987309150 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fdfd30 msgr2=0x55dae0fe3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:04.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc779c00700 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fdfd30 0x55dae0fe3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc779c00700 1 -- 172.21.15.43:0/1987309150 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dae0e478d0 con 0x55dae0e5ae40 2024-05-27T07:35:04.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc779c00700 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fe34d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc7640a9960 tx=0x7fc7640b2960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26c342e11fae36f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:04.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.201+0000 7fc757400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:35:04.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.202+0000 7fc757400700 1 -- 172.21.15.43:0/1987309150 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7640c9060 con 0x55dae0e5ae40 2024-05-27T07:35:04.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.202+0000 7fc78b178b80 1 -- 172.21.15.43:0/1987309150 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7640aa3b0 con 0x55dae0e5ae40 2024-05-27T07:35:04.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.202+0000 7fc757400700 1 -- 172.21.15.43:0/1987309150 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7fc7640a62a0 con 0x55dae0e5ae40 2024-05-27T07:35:04.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.202+0000 7fc757400700 1 -- 172.21.15.43:0/1987309150 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) ==== 24+0+0 (secure 0 0 0) 0x7fc76406f070 con 0x55dae0e5ae40 2024-05-27T07:35:04.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.202+0000 7fc78b178b80 10 client.37601 fetch_fsmap learned FSMap version 141 2024-05-27T07:35:04.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.202+0000 7fc78b178b80 1 -- 172.21.15.43:0/1987309150 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=4+}) -- 0x55dae0ec8060 con 0x55dae0e5ae40 2024-05-27T07:35:04.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.202+0000 7fc757400700 1 -- 172.21.15.43:0/1987309150 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7640aa3b0 con 0x55dae0e5ae40 2024-05-27T07:35:04.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.202+0000 7fc757400700 1 -- 172.21.15.43:0/1987309150 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 141) ==== 4203+0+0 (secure 0 0 0) 0x7fc7640c4040 con 0x55dae0e5ae40 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.202+0000 7fc78b178b80 10 client.37601 fetch_fsmap finished waiting for FSMap version 141 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.202+0000 7fc78b178b80 10 client.37601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.203+0000 7fc78b178b80 20 client.37601 populate_metadata read hostname 'smithi043' 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.203+0000 7fc78b178b80 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dae0ff3640 0x55dae0ff5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.203+0000 7fc78b178b80 4 client.37601 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.203+0000 7fc78b178b80 1 -- 172.21.15.43:0/1987309150 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dae0f09d90 con 0x55dae0ff3640 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.203+0000 7fc779200700 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dae0ff3640 0x55dae0ff5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.204+0000 7fc779200700 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dae0ff3640 0x55dae0ff5ab0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47cbba7e8153f44f server_cookie=59ad4a829939b3ce in_seq=0 out_seq=0 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.204+0000 7fc757400700 10 client.37601 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc757400700 1 -- 172.21.15.43:0/1987309150 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dae0f09d90 con 0x55dae0ff3640 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc757400700 10 client.37601 handle_command_reply: tid=0 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 client.37601 shutdown 2024-05-27T07:35:04.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 -- 172.21.15.43:0/1987309150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 msgr2=0x55dae0fe34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fe34d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc7640a9960 tx=0x7fc7640b2960 comp rx=0 tx=0).stop 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 -- 172.21.15.43:0/1987309150 shutdown_connections 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dae0ff3640 0x55dae0ff5ab0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fdfd30 0x55dae0fe3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0fe34d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc757400700 0 client.37601 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 --2- 172.21.15.43:0/1987309150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fd9db0 0x55dae0fe3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 -- 172.21.15.43:0/1987309150 >> 172.21.15.43:0/1987309150 conn(0x55dae0ebafc0 msgr2=0x55dae0fd4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 -- 172.21.15.43:0/1987309150 shutdown_connections 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.205+0000 7fc78b178b80 1 -- 172.21.15.43:0/1987309150 wait complete. 2024-05-27T07:35:04.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.206+0000 7fc78b178b80 20 client.37601 trim_cache size 0 max 16384 2024-05-27T07:35:04.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.227+0000 7fc78b178b80 1 Processor -- start 2024-05-27T07:35:04.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.227+0000 7fc78b178b80 1 -- start start 2024-05-27T07:35:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0e5ae40 0x55dae0fd9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0fd9db0 0x55dae0fe2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0fe3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc78b178b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc7741bfa30 con 0x55dae0e5ae40 2024-05-27T07:35:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc78b178b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc7741bde30 con 0x55dae0fd9db0 2024-05-27T07:35:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc78b178b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc7741bf360 con 0x55dae0fdfd30 2024-05-27T07:35:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc779c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0e5ae40 0x55dae0fd9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc779c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0e5ae40 0x55dae0fd9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51572/0 (socket says 172.21.15.43:51572) 2024-05-27T07:35:04.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc779c00700 1 -- 172.21.15.43:0/2261849367 learned_addr learned my addr 172.21.15.43:0/2261849367 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:04.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc779200700 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0fd9db0 0x55dae0fe2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc77a600700 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0fe3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc779c00700 1 -- 172.21.15.43:0/2261849367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 msgr2=0x55dae0fe3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:04.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc779c00700 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0fe3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc779c00700 1 -- 172.21.15.43:0/2261849367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0fd9db0 msgr2=0x55dae0fe2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:04.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc779c00700 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0fd9db0 0x55dae0fe2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc779c00700 1 -- 172.21.15.43:0/2261849367 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dae0fda800 con 0x55dae0e5ae40 2024-05-27T07:35:04.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc779200700 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0fd9db0 0x55dae0fe2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:04.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.228+0000 7fc77a600700 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0fe3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:35:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.229+0000 7fc779c00700 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0e5ae40 0x55dae0fd9870 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fc7640e0180 tx=0x7fc7640a9320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89729edf62a5774 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.229+0000 7fc762000700 1 -- 172.21.15.43:0/2261849367 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7640c9070 con 0x55dae0e5ae40 2024-05-27T07:35:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.229+0000 7fc762000700 1 -- 172.21.15.43:0/2261849367 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7640b2460 con 0x55dae0e5ae40 2024-05-27T07:35:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.229+0000 7fc762000700 1 -- 172.21.15.43:0/2261849367 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7640aa3b0 con 0x55dae0e5ae40 2024-05-27T07:35:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.230+0000 7fc78b178b80 1 -- 172.21.15.43:0/2261849367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0e5ae40 msgr2=0x55dae0fd9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.230+0000 7fc78b178b80 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0e5ae40 0x55dae0fd9870 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fc7640e0180 tx=0x7fc7640a9320 comp rx=0 tx=0).stop 2024-05-27T07:35:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.230+0000 7fc78b178b80 1 -- 172.21.15.43:0/2261849367 shutdown_connections 2024-05-27T07:35:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.230+0000 7fc78b178b80 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0fe3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.230+0000 7fc78b178b80 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0e5ae40 0x55dae0fd9870 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.230+0000 7fc78b178b80 1 --2- 172.21.15.43:0/2261849367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0fd9db0 0x55dae0fe2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.231+0000 7fc78b178b80 1 -- 172.21.15.43:0/2261849367 >> 172.21.15.43:0/2261849367 conn(0x55dae0ebafc0 msgr2=0x55dae0fdd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.231+0000 7fc78b178b80 1 -- 172.21.15.43:0/2261849367 shutdown_connections 2024-05-27T07:35:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.231+0000 7fc78b178b80 1 -- 172.21.15.43:0/2261849367 wait complete. 2024-05-27T07:35:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.231+0000 7fc78b178b80 1 -- start start 2024-05-27T07:35:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.231+0000 7fc78b178b80 1 Processor -- start 2024-05-27T07:35:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.232+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0febfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.232+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 0x55dae0ffa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.232+0000 7fc78b178b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0ffae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.232+0000 7fc78b178b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc7741bec90 con 0x55dae0fd9db0 2024-05-27T07:35:04.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.232+0000 7fc78b178b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc7741be560 con 0x55dae0e5ae40 2024-05-27T07:35:04.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.232+0000 7fc78b178b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55dae0f09d90 con 0x55dae0fdfd30 2024-05-27T07:35:04.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.232+0000 7fc78b178b80 10 client.0 _ll_register_callbacks cb 0x55dae0ea1c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:04.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc779c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0febfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc779200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 0x55dae0ffa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc779c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0febfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:37878/0 (socket says 172.21.15.43:37878) 2024-05-27T07:35:04.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc779c00700 1 -- 172.21.15.43:0/2475219152 learned_addr learned my addr 172.21.15.43:0/2475219152 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:04.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc77a600700 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0ffae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc779c00700 1 -- 172.21.15.43:0/2475219152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 msgr2=0x55dae0ffae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:04.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc779c00700 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0ffae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc779c00700 1 -- 172.21.15.43:0/2475219152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 msgr2=0x55dae0ffa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc779c00700 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 0x55dae0ffa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc779c00700 1 -- 172.21.15.43:0/2475219152 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dae0ffb340 con 0x55dae0e5ae40 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.233+0000 7fc779200700 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 0x55dae0ffa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.234+0000 7fc779c00700 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0febfb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc7640a6f20 tx=0x7fc7640a5ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ef05e4c11bde002 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.234+0000 7fc757400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.234+0000 7fc757400700 1 -- 172.21.15.43:0/2475219152 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7640c9040 con 0x55dae0e5ae40 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.234+0000 7fc757400700 1 -- 172.21.15.43:0/2475219152 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(151..151 src has 1..151) ==== 9496+0+0 (secure 0 0 0) 0x7fc7640a62a0 con 0x55dae0e5ae40 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.234+0000 7fc78b178b80 1 -- 172.21.15.43:0/2475219152 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7640b2460 con 0x55dae0e5ae40 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.234+0000 7fc757400700 1 -- 172.21.15.43:0/2475219152 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) ==== 24+0+0 (secure 0 0 0) 0x7fc7640a9150 con 0x55dae0e5ae40 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.234+0000 7fc78b178b80 10 client.37607 fetch_fsmap learned FSMap version 141 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.234+0000 7fc78b178b80 1 -- 172.21.15.43:0/2475219152 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=4+}) -- 0x55dae0ffb340 con 0x55dae0e5ae40 2024-05-27T07:35:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.235+0000 7fc757400700 1 -- 172.21.15.43:0/2475219152 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7640b2460 con 0x55dae0e5ae40 2024-05-27T07:35:04.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.235+0000 7fc757400700 1 -- 172.21.15.43:0/2475219152 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 141) ==== 4203+0+0 (secure 0 0 0) 0x7fc7640c4040 con 0x55dae0e5ae40 2024-05-27T07:35:04.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.235+0000 7fc78b178b80 10 client.37607 fetch_fsmap finished waiting for FSMap version 141 2024-05-27T07:35:04.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.235+0000 7fc78b178b80 10 client.37607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:04.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.235+0000 7fc78b178b80 20 client.37607 populate_metadata read hostname 'smithi043' 2024-05-27T07:35:04.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.235+0000 7fc78b178b80 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dae0ff3640 0x55dae1060030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:04.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.235+0000 7fc78b178b80 4 client.37607 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:35:04.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.235+0000 7fc78b178b80 1 -- 172.21.15.43:0/2475219152 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dae0f03570 con 0x55dae0ff3640 2024-05-27T07:35:04.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.236+0000 7fc779200700 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dae0ff3640 0x55dae1060030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:04.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.237+0000 7fc757400700 10 client.37607 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:35:04.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.237+0000 7fc779200700 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dae0ff3640 0x55dae1060030 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=166169635116b9df server_cookie=6a6f570abdbb3df4 in_seq=0 out_seq=0 2024-05-27T07:35:04.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.238+0000 7fc757400700 1 -- 172.21.15.43:0/2475219152 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dae0f03570 con 0x55dae0ff3640 2024-05-27T07:35:04.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.238+0000 7fc757400700 10 client.37607 handle_command_reply: tid=0 2024-05-27T07:35:04.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.238+0000 7fc78b178b80 1 client.37607 shutdown 2024-05-27T07:35:04.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.238+0000 7fc78b178b80 1 -- 172.21.15.43:0/2475219152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 msgr2=0x55dae0febfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.238+0000 7fc78b178b80 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0febfb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc7640a6f20 tx=0x7fc7640a5ac0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.239+0000 7fc78b178b80 1 -- 172.21.15.43:0/2475219152 shutdown_connections 2024-05-27T07:35:04.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.239+0000 7fc78b178b80 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dae0fdfd30 0x55dae0ffae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.239+0000 7fc78b178b80 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dae0e5ae40 0x55dae0febfb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.239+0000 7fc78b178b80 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dae0ff3640 0x55dae1060030 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.239+0000 7fc78b178b80 1 --2- 172.21.15.43:0/2475219152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dae0fd9db0 0x55dae0ffa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.239+0000 7fc78b178b80 1 -- 172.21.15.43:0/2475219152 >> 172.21.15.43:0/2475219152 conn(0x55dae0ebafc0 msgr2=0x55dae0fe0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:04.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.239+0000 7fc757400700 0 client.37607 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:35:04.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.239+0000 7fc78b178b80 1 -- 172.21.15.43:0/2475219152 shutdown_connections 2024-05-27T07:35:04.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.239+0000 7fc78b178b80 1 -- 172.21.15.43:0/2475219152 wait complete. 2024-05-27T07:35:04.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.239+0000 7fc78b178b80 20 client.37607 trim_cache size 0 max 16384 2024-05-27T07:35:04.235 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:35:04.235 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:35:04.235 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:35:04.235 INFO:teuthology.orchestra.run.smithi043.stdout: "f78b2626-afbf-440a-89b0-3bc5ec2a2fea": { 2024-05-27T07:35:04.235 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:35:04.235 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "f78b2626-afbf-440a-89b0-3bc5ec2a2fea", 2024-05-27T07:35:04.236 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:35:04.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:35:04.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:35:04.236 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:35:04.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 -- 172.21.15.43:0/3171009269 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc75c0b2a20 msgr2=0x7fc75c0b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc75c0b2a20 0x7fc75c0b4f30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc774108ae0 tx=0x7fc768009310 comp rx=0 tx=0).stop 2024-05-27T07:35:04.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 -- 172.21.15.43:0/3171009269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc774103240 msgr2=0x7fc774075390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:04.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc774103240 0x7fc774075390 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fc76400e9c0 tx=0x7fc76400ed80 comp rx=0 tx=0).stop 2024-05-27T07:35:04.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 -- 172.21.15.43:0/3171009269 shutdown_connections 2024-05-27T07:35:04.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fc75c0b2a20 0x7fc75c0b4f30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc774108690 0x7fc77406eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc774103240 0x7fc774075390 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 --2- 172.21.15.43:0/3171009269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc774103b90 0x7fc77406e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:04.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 -- 172.21.15.43:0/3171009269 >> 172.21.15.43:0/3171009269 conn(0x7fc7740fedb0 msgr2=0x7fc774106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:04.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 -- 172.21.15.43:0/3171009269 shutdown_connections 2024-05-27T07:35:04.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:04.240+0000 7fc77c600700 1 -- 172.21.15.43:0/3171009269 wait complete. 2024-05-27T07:35:04.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:04 smithi043 ceph-mon[31531]: pgmap v709: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 3.2 MiB/s wr, 152 op/s 2024-05-27T07:35:04.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:04 smithi179 ceph-mon[35888]: pgmap v709: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 3.2 MiB/s wr, 152 op/s 2024-05-27T07:35:04.595 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:04 smithi132 ceph-mon[35283]: pgmap v709: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 3.2 MiB/s wr, 152 op/s 2024-05-27T07:35:04.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46626 5.83 MB/sec execute 305 sec latency 286.099 ms 2024-05-27T07:35:05.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46704 5.81 MB/sec execute 306 sec latency 267.453 ms 2024-05-27T07:35:05.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:05.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:05.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:05 smithi043 ceph-mon[31531]: pgmap v710: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 3.2 MiB/s wr, 152 op/s 2024-05-27T07:35:05.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:05.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:05.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:05.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:05.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:05 smithi179 ceph-mon[35888]: pgmap v710: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 3.2 MiB/s wr, 152 op/s 2024-05-27T07:35:05.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:05.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:06.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:06.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:06.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:05 smithi132 ceph-mon[35283]: pgmap v710: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 3.2 MiB/s wr, 152 op/s 2024-05-27T07:35:06.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:06.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:06.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46822 5.82 MB/sec execute 307 sec latency 102.251 ms 2024-05-27T07:35:06.953 INFO:tasks.cephfs.filesystem:scrub status for tag:f78b2626-afbf-440a-89b0-3bc5ec2a2fea - {'path': '/', 'tag': 'f78b2626-afbf-440a-89b0-3bc5ec2a2fea', 'options': 'recursive,force'} 2024-05-27T07:35:06.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:06 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi132 to 176.4M 2024-05-27T07:35:06.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:06 smithi179 ceph-mon[35888]: Unable to set osd_memory_target on smithi132 to 185069158: error parsing value: Value '185069158' is below minimum 939524096 2024-05-27T07:35:06.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:06.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:06 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi132 to 176.4M 2024-05-27T07:35:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:06 smithi132 ceph-mon[35283]: Unable to set osd_memory_target on smithi132 to 185069158: error parsing value: Value '185069158' is below minimum 939524096 2024-05-27T07:35:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:06 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi132 to 176.4M 2024-05-27T07:35:07.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:06 smithi043 ceph-mon[31531]: Unable to set osd_memory_target on smithi132 to 185069158: error parsing value: Value '185069158' is below minimum 939524096 2024-05-27T07:35:07.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:07.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 46899 5.81 MB/sec execute 308 sec latency 194.344 ms 2024-05-27T07:35:08.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:07 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi179 to 176.4M 2024-05-27T07:35:08.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:07 smithi132 ceph-mon[35283]: Unable to set osd_memory_target on smithi179 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-05-27T07:35:08.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:07 smithi132 ceph-mon[35283]: osdmap e152: 12 total, 12 up, 12 in 2024-05-27T07:35:08.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:07 smithi132 ceph-mon[35283]: pgmap v712: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 3.2 MiB/s wr, 161 op/s 2024-05-27T07:35:08.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:07 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi179 to 176.4M 2024-05-27T07:35:08.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:07 smithi043 ceph-mon[31531]: Unable to set osd_memory_target on smithi179 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-05-27T07:35:08.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:07 smithi043 ceph-mon[31531]: osdmap e152: 12 total, 12 up, 12 in 2024-05-27T07:35:08.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:07 smithi043 ceph-mon[31531]: pgmap v712: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 3.2 MiB/s wr, 161 op/s 2024-05-27T07:35:08.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:07 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi179 to 176.4M 2024-05-27T07:35:08.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:07 smithi179 ceph-mon[35888]: Unable to set osd_memory_target on smithi179 to 185047654: error parsing value: Value '185047654' is below minimum 939524096 2024-05-27T07:35:08.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:07 smithi179 ceph-mon[35888]: osdmap e152: 12 total, 12 up, 12 in 2024-05-27T07:35:08.213 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:07 smithi179 ceph-mon[35888]: pgmap v712: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 3.2 MiB/s wr, 161 op/s 2024-05-27T07:35:08.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 47005 5.81 MB/sec execute 309 sec latency 153.904 ms 2024-05-27T07:35:09.811 INFO:tasks.workunit.client.0.smithi043.stdout: 10 47129 5.80 MB/sec execute 310 sec latency 106.757 ms 2024-05-27T07:35:10.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:10 smithi043 ceph-mon[31531]: pgmap v713: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 3.2 MiB/s wr, 161 op/s 2024-05-27T07:35:10.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:10 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:10.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:10 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:10.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:10 smithi179 ceph-mon[35888]: pgmap v713: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 3.2 MiB/s wr, 161 op/s 2024-05-27T07:35:10.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:10 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:10.463 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:10 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:10.595 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:10 smithi132 ceph-mon[35283]: pgmap v713: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 3.2 MiB/s wr, 161 op/s 2024-05-27T07:35:10.595 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:10 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:10.595 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:10 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:10.812 INFO:tasks.workunit.client.0.smithi043.stdout: 10 47261 5.81 MB/sec execute 311 sec latency 104.650 ms 2024-05-27T07:35:11.812 INFO:tasks.workunit.client.0.smithi043.stdout: 10 47353 5.80 MB/sec execute 312 sec latency 99.019 ms 2024-05-27T07:35:12.812 INFO:tasks.workunit.client.0.smithi043.stdout: 10 47445 5.80 MB/sec execute 313 sec latency 316.773 ms 2024-05-27T07:35:12.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:12 smithi132 ceph-mon[35283]: pgmap v714: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 3.2 MiB/s wr, 166 op/s 2024-05-27T07:35:12.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:12 smithi132 ceph-mon[35283]: osdmap e153: 12 total, 12 up, 12 in 2024-05-27T07:35:12.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:12 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:35:12.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:12 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:35:12.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:12 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:35:12.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:12 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:35:12.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:12 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:35:12.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:12 smithi043 ceph-mon[31531]: pgmap v714: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 3.2 MiB/s wr, 166 op/s 2024-05-27T07:35:12.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:12 smithi043 ceph-mon[31531]: osdmap e153: 12 total, 12 up, 12 in 2024-05-27T07:35:12.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:12 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:35:12.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:12 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:35:12.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:12 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:35:12.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:12 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:35:12.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:12 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:35:12.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:12 smithi179 ceph-mon[35888]: pgmap v714: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 3.2 MiB/s wr, 166 op/s 2024-05-27T07:35:12.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:12 smithi179 ceph-mon[35888]: osdmap e153: 12 total, 12 up, 12 in 2024-05-27T07:35:12.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:12 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:35:12.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:12 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:35:12.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:12 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:35:12.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:12 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:35:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:12 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:35:13.812 INFO:tasks.workunit.client.0.smithi043.stdout: 10 47572 5.82 MB/sec execute 314 sec latency 129.386 ms 2024-05-27T07:35:14.812 INFO:tasks.workunit.client.0.smithi043.stdout: 10 47683 5.82 MB/sec execute 315 sec latency 90.791 ms 2024-05-27T07:35:14.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:14 smithi132 ceph-mon[35283]: pgmap v716: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 4.0 MiB/s wr, 207 op/s 2024-05-27T07:35:14.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:14 smithi043 ceph-mon[31531]: pgmap v716: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 4.0 MiB/s wr, 207 op/s 2024-05-27T07:35:14.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:14 smithi179 ceph-mon[35888]: pgmap v716: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 4.0 MiB/s wr, 207 op/s 2024-05-27T07:35:15.812 INFO:tasks.workunit.client.0.smithi043.stdout: 10 47832 5.85 MB/sec execute 316 sec latency 82.705 ms 2024-05-27T07:35:16.814 INFO:tasks.workunit.client.0.smithi043.stdout: 10 47934 5.84 MB/sec execute 317 sec latency 88.454 ms 2024-05-27T07:35:16.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:16 smithi132 ceph-mon[35283]: pgmap v717: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-05-27T07:35:16.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:16 smithi043 ceph-mon[31531]: pgmap v717: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-05-27T07:35:16.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:16 smithi179 ceph-mon[35888]: pgmap v717: 129 pgs: 129 active+clean; 1.0 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-05-27T07:35:17.812 INFO:tasks.workunit.client.0.smithi043.stdout: 10 48081 5.85 MB/sec execute 318 sec latency 100.018 ms 2024-05-27T07:35:17.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:17.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:17.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:17 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:17.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:17.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:17.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:17 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:17.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-27T07:35:17.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:35:17.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:35:17.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:17 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:35:18.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:18 smithi179 ceph-mon[35888]: pgmap v718: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 4.2 MiB/s wr, 190 op/s 2024-05-27T07:35:18.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:18 smithi179 ceph-mon[35888]: Adjusting osd_memory_target on smithi043 to 177.5M 2024-05-27T07:35:18.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:18 smithi179 ceph-mon[35888]: Unable to set osd_memory_target on smithi043 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:35:18.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:18 smithi179 ceph-mon[35888]: pgmap v719: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 5.1 MiB/s wr, 230 op/s 2024-05-27T07:35:18.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:18 smithi179 ceph-mon[35888]: pgmap v720: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 3.9 MiB/s wr, 155 op/s 2024-05-27T07:35:18.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:18 smithi179 ceph-mon[35888]: Health check update: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:35:18.812 INFO:tasks.workunit.client.0.smithi043.stdout: 10 48175 5.84 MB/sec execute 319 sec latency 100.799 ms 2024-05-27T07:35:18.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:18 smithi132 ceph-mon[35283]: pgmap v718: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 4.2 MiB/s wr, 190 op/s 2024-05-27T07:35:18.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:18 smithi132 ceph-mon[35283]: Adjusting osd_memory_target on smithi043 to 177.5M 2024-05-27T07:35:18.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:18 smithi132 ceph-mon[35283]: Unable to set osd_memory_target on smithi043 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:35:18.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:18 smithi132 ceph-mon[35283]: pgmap v719: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 5.1 MiB/s wr, 230 op/s 2024-05-27T07:35:18.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:18 smithi132 ceph-mon[35283]: pgmap v720: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 3.9 MiB/s wr, 155 op/s 2024-05-27T07:35:18.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:18 smithi132 ceph-mon[35283]: Health check update: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:35:18.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:18 smithi043 ceph-mon[31531]: pgmap v718: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 4.2 MiB/s wr, 190 op/s 2024-05-27T07:35:18.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:18 smithi043 ceph-mon[31531]: Adjusting osd_memory_target on smithi043 to 177.5M 2024-05-27T07:35:18.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:18 smithi043 ceph-mon[31531]: Unable to set osd_memory_target on smithi043 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-27T07:35:18.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:18 smithi043 ceph-mon[31531]: pgmap v719: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 5.1 MiB/s wr, 230 op/s 2024-05-27T07:35:18.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:18 smithi043 ceph-mon[31531]: pgmap v720: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 3.9 MiB/s wr, 155 op/s 2024-05-27T07:35:18.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:18 smithi043 ceph-mon[31531]: Health check update: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:35:19.812 INFO:tasks.workunit.client.0.smithi043.stdout: 10 48269 5.82 MB/sec execute 320 sec latency 98.329 ms 2024-05-27T07:35:20.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 48398 5.82 MB/sec execute 321 sec latency 77.006 ms 2024-05-27T07:35:20.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:20 smithi132 ceph-mon[35283]: pgmap v721: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 3.1 MiB/s wr, 124 op/s 2024-05-27T07:35:20.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:20 smithi043 ceph-mon[31531]: pgmap v721: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 3.1 MiB/s wr, 124 op/s 2024-05-27T07:35:20.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:20 smithi179 ceph-mon[35888]: pgmap v721: 129 pgs: 129 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 3.1 MiB/s wr, 124 op/s 2024-05-27T07:35:21.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 48519 5.81 MB/sec execute 322 sec latency 84.337 ms 2024-05-27T07:35:22.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 48638 5.82 MB/sec execute 323 sec latency 131.183 ms 2024-05-27T07:35:22.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:22 smithi132 ceph-mon[35283]: pgmap v722: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 5.1 MiB/s wr, 221 op/s 2024-05-27T07:35:22.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:22 smithi043 ceph-mon[31531]: pgmap v722: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 5.1 MiB/s wr, 221 op/s 2024-05-27T07:35:22.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:22 smithi179 ceph-mon[35888]: pgmap v722: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 5.1 MiB/s wr, 221 op/s 2024-05-27T07:35:23.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 48739 5.81 MB/sec execute 324 sec latency 77.513 ms 2024-05-27T07:35:24.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 48860 5.81 MB/sec execute 325 sec latency 80.801 ms 2024-05-27T07:35:24.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:24 smithi132 ceph-mon[35283]: pgmap v723: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 5.1 MiB/s wr, 221 op/s 2024-05-27T07:35:24.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:24 smithi043 ceph-mon[31531]: pgmap v723: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 5.1 MiB/s wr, 221 op/s 2024-05-27T07:35:24.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:24 smithi179 ceph-mon[35888]: pgmap v723: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 5.1 MiB/s wr, 221 op/s 2024-05-27T07:35:25.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 49002 5.83 MB/sec execute 326 sec latency 86.196 ms 2024-05-27T07:35:25.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:25 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:25.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:25 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:25.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:25 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:26.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 49148 5.85 MB/sec execute 327 sec latency 87.523 ms 2024-05-27T07:35:26.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:26 smithi132 ceph-mon[35283]: pgmap v724: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 2.2 MiB/s wr, 103 op/s 2024-05-27T07:35:26.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:26 smithi043 ceph-mon[31531]: pgmap v724: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 2.2 MiB/s wr, 103 op/s 2024-05-27T07:35:26.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:26 smithi179 ceph-mon[35888]: pgmap v724: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 2.2 MiB/s wr, 103 op/s 2024-05-27T07:35:27.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 49300 5.85 MB/sec execute 328 sec latency 111.373 ms 2024-05-27T07:35:28.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 49418 5.86 MB/sec execute 329 sec latency 145.336 ms 2024-05-27T07:35:28.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:28 smithi132 ceph-mon[35283]: pgmap v725: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 190 op/s 2024-05-27T07:35:28.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:28 smithi179 ceph-mon[35888]: pgmap v725: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 190 op/s 2024-05-27T07:35:28.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:28 smithi043 ceph-mon[31531]: pgmap v725: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 190 op/s 2024-05-27T07:35:29.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 49506 5.85 MB/sec execute 330 sec latency 79.404 ms 2024-05-27T07:35:29.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:29 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:29.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:29 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:29.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:29 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:30.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 49624 5.86 MB/sec execute 331 sec latency 80.446 ms 2024-05-27T07:35:30.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:30 smithi043 ceph-mon[31531]: pgmap v726: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-05-27T07:35:30.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:30 smithi179 ceph-mon[35888]: pgmap v726: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-05-27T07:35:31.103 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:30 smithi132 ceph-mon[35283]: pgmap v726: 129 pgs: 129 active+clean; 1.0 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-05-27T07:35:31.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 49749 5.87 MB/sec execute 332 sec latency 74.219 ms 2024-05-27T07:35:32.813 INFO:tasks.workunit.client.0.smithi043.stdout: 10 49870 5.87 MB/sec execute 333 sec latency 71.872 ms 2024-05-27T07:35:32.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:32 smithi043 ceph-mon[31531]: pgmap v727: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-05-27T07:35:32.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:32 smithi179 ceph-mon[35888]: pgmap v727: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-05-27T07:35:33.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:32 smithi132 ceph-mon[35283]: pgmap v727: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-05-27T07:35:33.814 INFO:tasks.workunit.client.0.smithi043.stdout: 10 49989 5.87 MB/sec execute 334 sec latency 144.331 ms 2024-05-27T07:35:34.814 INFO:tasks.workunit.client.0.smithi043.stdout: 10 50070 5.87 MB/sec execute 335 sec latency 471.980 ms 2024-05-27T07:35:34.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:34 smithi043 ceph-mon[31531]: pgmap v728: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 170 op/s 2024-05-27T07:35:34.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:34 smithi179 ceph-mon[35888]: pgmap v728: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 170 op/s 2024-05-27T07:35:35.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:34 smithi132 ceph-mon[35283]: pgmap v728: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 170 op/s 2024-05-27T07:35:35.814 INFO:tasks.workunit.client.0.smithi043.stdout: 10 50166 5.86 MB/sec execute 336 sec latency 90.947 ms 2024-05-27T07:35:36.814 INFO:tasks.workunit.client.0.smithi043.stdout: 10 50267 5.85 MB/sec execute 337 sec latency 74.004 ms 2024-05-27T07:35:36.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:35:36.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:36 smithi043 ceph-mon[31531]: pgmap v729: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 170 op/s 2024-05-27T07:35:36.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:36 smithi179 ceph-mon[35888]: pgmap v729: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 170 op/s 2024-05-27T07:35:37.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:36 smithi132 ceph-mon[35283]: pgmap v729: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 170 op/s 2024-05-27T07:35:37.727 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:37.814 INFO:tasks.workunit.client.0.smithi043.stdout: 10 50359 5.84 MB/sec execute 338 sec latency 88.335 ms 2024-05-27T07:35:38.669 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:38 smithi043 ceph-mon[31531]: pgmap v730: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.9 MiB/s wr, 242 op/s 2024-05-27T07:35:38.815 INFO:tasks.workunit.client.0.smithi043.stdout: 10 50437 5.84 MB/sec execute 339 sec latency 390.586 ms 2024-05-27T07:35:38.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:38 smithi179 ceph-mon[35888]: pgmap v730: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.9 MiB/s wr, 242 op/s 2024-05-27T07:35:39.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:38 smithi132 ceph-mon[35283]: pgmap v730: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.9 MiB/s wr, 242 op/s 2024-05-27T07:35:39.815 INFO:tasks.workunit.client.0.smithi043.stdout: 10 50518 5.83 MB/sec execute 340 sec latency 550.889 ms 2024-05-27T07:35:40.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.263+0000 7f872be00700 1 -- 172.21.15.43:0/4035871307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f872c109670 msgr2=0x7f872c110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.263+0000 7f872be00700 1 --2- 172.21.15.43:0/4035871307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f872c109670 0x7f872c110fd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f87240098f0 tx=0x7f8724009c00 comp rx=0 tx=0).stop 2024-05-27T07:35:40.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.263+0000 7f872be00700 1 -- 172.21.15.43:0/4035871307 shutdown_connections 2024-05-27T07:35:40.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.263+0000 7f872be00700 1 --2- 172.21.15.43:0/4035871307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f872c109670 0x7f872c110fd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.263+0000 7f872be00700 1 --2- 172.21.15.43:0/4035871307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f872c1043f0 0x7f872c104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.263+0000 7f872be00700 1 --2- 172.21.15.43:0/4035871307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f872c108d20 0x7f872c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.263+0000 7f872be00700 1 -- 172.21.15.43:0/4035871307 >> 172.21.15.43:0/4035871307 conn(0x7f872c079d00 msgr2=0x7f872c07a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:40.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.263+0000 7f872be00700 1 -- 172.21.15.43:0/4035871307 shutdown_connections 2024-05-27T07:35:40.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.263+0000 7f872be00700 1 -- 172.21.15.43:0/4035871307 wait complete. 2024-05-27T07:35:40.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.265+0000 7f872be00700 1 Processor -- start 2024-05-27T07:35:40.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.265+0000 7f872be00700 1 -- start start 2024-05-27T07:35:40.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f872be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f872c1043f0 0x7f872c19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f872be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f872c108d20 0x7f872c19d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f872be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f872c109670 0x7f872c1a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f872be00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f872c1141a0 con 0x7f872c109670 2024-05-27T07:35:40.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f872be00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f872c114020 con 0x7f872c108d20 2024-05-27T07:35:40.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f872be00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f872c114320 con 0x7f872c1043f0 2024-05-27T07:35:40.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f872aa00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f872c1043f0 0x7f872c19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f8722000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f872c108d20 0x7f872c19d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f8722000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f872c108d20 0x7f872c19d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:47004/0 (socket says 172.21.15.43:47004) 2024-05-27T07:35:40.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f8722000700 1 -- 172.21.15.43:0/4239535607 learned_addr learned my addr 172.21.15.43:0/4239535607 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:40.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.266+0000 7f872b400700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f872c109670 0x7f872c1a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.267+0000 7f872b400700 1 -- 172.21.15.43:0/4239535607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f872c1043f0 msgr2=0x7f872c19cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.267+0000 7f872b400700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f872c1043f0 0x7f872c19cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.267+0000 7f872b400700 1 -- 172.21.15.43:0/4239535607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f872c108d20 msgr2=0x7f872c19d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.267+0000 7f872b400700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f872c108d20 0x7f872c19d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.267+0000 7f872b400700 1 -- 172.21.15.43:0/4239535607 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872c1a1cc0 con 0x7f872c109670 2024-05-27T07:35:40.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.268+0000 7f872b400700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f872c109670 0x7f872c1a1620 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f8724009810 tx=0x7f872400fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.268+0000 7f8723e00700 1 -- 172.21.15.43:0/4239535607 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872400b5e0 con 0x7f872c109670 2024-05-27T07:35:40.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.268+0000 7f8723e00700 1 -- 172.21.15.43:0/4239535607 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8724005040 con 0x7f872c109670 2024-05-27T07:35:40.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.268+0000 7f8723e00700 1 -- 172.21.15.43:0/4239535607 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240038e0 con 0x7f872c109670 2024-05-27T07:35:40.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.269+0000 7f872be00700 1 -- 172.21.15.43:0/4239535607 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872c1a1f50 con 0x7f872c109670 2024-05-27T07:35:40.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.269+0000 7f872be00700 1 -- 172.21.15.43:0/4239535607 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f872c1a2380 con 0x7f872c109670 2024-05-27T07:35:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.275+0000 7f8723e00700 1 -- 172.21.15.43:0/4239535607 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f8724029060 con 0x7f872c109670 2024-05-27T07:35:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.275+0000 7f8740583b80 1 Processor -- start 2024-05-27T07:35:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.276+0000 7f8740583b80 1 -- start start 2024-05-27T07:35:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.276+0000 7f8740583b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d22750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.276+0000 7f8740583b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d22c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.276+0000 7f8740583b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18d2c2c0 0x55ed18d2e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.276+0000 7f8740583b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ed18c4fe40 con 0x55ed18c8e9f0 2024-05-27T07:35:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.276+0000 7f8740583b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ed18c3f9e0 con 0x55ed18ba2e40 2024-05-27T07:35:40.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.276+0000 7f8740583b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ed18c43a20 con 0x55ed18d2c2c0 2024-05-27T07:35:40.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.277+0000 7f8722000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d22750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.277+0000 7f8722000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d22750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56720/0 (socket says 172.21.15.43:56720) 2024-05-27T07:35:40.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.277+0000 7f8722000700 1 -- 172.21.15.43:0/3476528023 learned_addr learned my addr 172.21.15.43:0/3476528023 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:40.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.277+0000 7f872aa00700 1 --2- 172.21.15.43:0/3476528023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d22c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.277+0000 7f8722000700 1 --2- 172.21.15.43:0/3476528023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18d2c2c0 0x55ed18d2e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.278+0000 7f8723e00700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f87100b2aa0 0x7f87100b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.278+0000 7f8722000700 1 -- 172.21.15.43:0/3476528023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18d2c2c0 msgr2=0x55ed18d2e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.278+0000 7f8722000700 1 --2- 172.21.15.43:0/3476528023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18d2c2c0 0x55ed18d2e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.278+0000 7f8722000700 1 -- 172.21.15.43:0/3476528023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 msgr2=0x55ed18d22c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.278+0000 7f8722000700 1 --2- 172.21.15.43:0/3476528023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d22c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.278+0000 7f8722000700 1 -- 172.21.15.43:0/3476528023 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed18c08420 con 0x55ed18c8e9f0 2024-05-27T07:35:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.278+0000 7f872aa00700 1 --2- 172.21.15.43:0/3476528023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d22c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:35:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.278+0000 7f8723e00700 1 -- 172.21.15.43:0/4239535607 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9496+0+0 (secure 0 0 0) 0x7f87240e3170 con 0x7f872c109670 2024-05-27T07:35:40.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.279+0000 7f872b400700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f87100b2aa0 0x7f87100b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.279+0000 7f8722000700 1 --2- 172.21.15.43:0/3476528023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d22750 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8714009f80 tx=0x7f87140099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cdb0796cc4914f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.279+0000 7f871be00700 1 -- 172.21.15.43:0/3476528023 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871401f070 con 0x55ed18c8e9f0 2024-05-27T07:35:40.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.279+0000 7f871be00700 1 -- 172.21.15.43:0/3476528023 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8714003e10 con 0x55ed18c8e9f0 2024-05-27T07:35:40.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.279+0000 7f871be00700 1 -- 172.21.15.43:0/3476528023 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87140075c0 con 0x55ed18c8e9f0 2024-05-27T07:35:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.280+0000 7f8740583b80 1 -- 172.21.15.43:0/3476528023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 msgr2=0x55ed18d22750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.280+0000 7f872b400700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f87100b2aa0 0x7f87100b4fb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f87240aa0a0 tx=0x7f8724005ec0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.280+0000 7f8740583b80 1 --2- 172.21.15.43:0/3476528023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d22750 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8714009f80 tx=0x7f87140099e0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.281+0000 7f8740583b80 1 -- 172.21.15.43:0/3476528023 shutdown_connections 2024-05-27T07:35:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.281+0000 7f8740583b80 1 --2- 172.21.15.43:0/3476528023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18d2c2c0 0x55ed18d2e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.281+0000 7f8740583b80 1 --2- 172.21.15.43:0/3476528023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d22750 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.281+0000 7f8740583b80 1 --2- 172.21.15.43:0/3476528023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d22c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.281+0000 7f8740583b80 1 -- 172.21.15.43:0/3476528023 >> 172.21.15.43:0/3476528023 conn(0x55ed18c3c390 msgr2=0x55ed18d1d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:40.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.281+0000 7f8740583b80 1 -- 172.21.15.43:0/3476528023 shutdown_connections 2024-05-27T07:35:40.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.281+0000 7f8740583b80 1 -- 172.21.15.43:0/3476528023 wait complete. 2024-05-27T07:35:40.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.282+0000 7f8740583b80 1 -- start start 2024-05-27T07:35:40.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.283+0000 7f8740583b80 1 Processor -- start 2024-05-27T07:35:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.283+0000 7f8740583b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18ba2e40 0x55ed18d292f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.283+0000 7f8740583b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 0x55ed18d29830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.283+0000 7f8740583b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18d2c2c0 0x55ed18d29d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.283+0000 7f8740583b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ed18b45fa0 con 0x55ed18ba2e40 2024-05-27T07:35:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.283+0000 7f8740583b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ed18be0760 con 0x55ed18d2c2c0 2024-05-27T07:35:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.283+0000 7f8740583b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ed18d2ee00 con 0x55ed18c8e9f0 2024-05-27T07:35:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.283+0000 7f8740583b80 10 client.0 _ll_register_callbacks cb 0x55ed18be9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.284+0000 7f8722000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18d2c2c0 0x55ed18d29d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.284+0000 7f8722000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18ba2e40 0x55ed18d292f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.284+0000 7f872aa00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 0x55ed18d29830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.284+0000 7f872aa00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 0x55ed18d29830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:51542/0 (socket says 172.21.15.43:51542) 2024-05-27T07:35:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.284+0000 7f872aa00700 1 -- 172.21.15.43:0/3446490180 learned_addr learned my addr 172.21.15.43:0/3446490180 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.285+0000 7f872aa00700 1 -- 172.21.15.43:0/3446490180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18d2c2c0 msgr2=0x55ed18d29d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.285+0000 7f872aa00700 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18d2c2c0 0x55ed18d29d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.285+0000 7f872aa00700 1 -- 172.21.15.43:0/3446490180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18ba2e40 msgr2=0x55ed18d292f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.286+0000 7f872aa00700 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18ba2e40 0x55ed18d292f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.286+0000 7f872aa00700 1 -- 172.21.15.43:0/3446490180 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed18b8f8d0 con 0x55ed18c8e9f0 2024-05-27T07:35:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.286+0000 7f8722000700 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18ba2e40 0x55ed18d292f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:35:40.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.286+0000 7f872aa00700 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 0x55ed18d29830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f871c0128a0 tx=0x7f871c012bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=208809840431f0d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.287+0000 7f8718c00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:35:40.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.287+0000 7f8718c00700 1 -- 172.21.15.43:0/3446490180 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0108e0 con 0x55ed18c8e9f0 2024-05-27T07:35:40.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.287+0000 7f8740583b80 1 -- 172.21.15.43:0/3446490180 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8714003e10 con 0x55ed18c8e9f0 2024-05-27T07:35:40.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.287+0000 7f8718c00700 1 -- 172.21.15.43:0/3446490180 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9496+0+0 (secure 0 0 0) 0x7f871c010a80 con 0x55ed18c8e9f0 2024-05-27T07:35:40.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.288+0000 7f8718c00700 1 -- 172.21.15.43:0/3446490180 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f871c010ca0 con 0x55ed18c8e9f0 2024-05-27T07:35:40.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.288+0000 7f8740583b80 10 client.27661 fetch_fsmap learned FSMap version 144 2024-05-27T07:35:40.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.288+0000 7f8740583b80 1 -- 172.21.15.43:0/3446490180 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55ed18c10060 con 0x55ed18c8e9f0 2024-05-27T07:35:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.289+0000 7f8718c00700 1 -- 172.21.15.43:0/3446490180 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8714003e10 con 0x55ed18c8e9f0 2024-05-27T07:35:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.289+0000 7f8718c00700 1 -- 172.21.15.43:0/3446490180 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f871c0254f0 con 0x55ed18c8e9f0 2024-05-27T07:35:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.289+0000 7f8740583b80 10 client.27661 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:35:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.289+0000 7f8740583b80 10 client.27661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.289+0000 7f8740583b80 20 client.27661 populate_metadata read hostname 'smithi043' 2024-05-27T07:35:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.289+0000 7f8740583b80 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ed18d44ad0 0x55ed18d46f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.289+0000 7f8740583b80 4 client.27661 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.289+0000 7f8740583b80 1 -- 172.21.15.43:0/3446490180 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed18c51d90 con 0x55ed18d44ad0 2024-05-27T07:35:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.290+0000 7f8722000700 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ed18d44ad0 0x55ed18d46f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.291+0000 7f8722000700 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ed18d44ad0 0x55ed18d46f20 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af0c0b83c3625f41 server_cookie=7194c2347462bd6d in_seq=0 out_seq=0 2024-05-27T07:35:40.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.291+0000 7f8718c00700 10 client.27661 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:35:40.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.293+0000 7f8718c00700 1 -- 172.21.15.43:0/3446490180 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ed18c51d90 con 0x55ed18d44ad0 2024-05-27T07:35:40.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.293+0000 7f8718c00700 10 client.27661 handle_command_reply: tid=0 2024-05-27T07:35:40.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.294+0000 7f8740583b80 1 client.27661 shutdown 2024-05-27T07:35:40.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.294+0000 7f8740583b80 1 -- 172.21.15.43:0/3446490180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 msgr2=0x55ed18d29830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.294+0000 7f8740583b80 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 0x55ed18d29830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f871c0128a0 tx=0x7f871c012bb0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.294+0000 7f8740583b80 1 -- 172.21.15.43:0/3446490180 shutdown_connections 2024-05-27T07:35:40.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.294+0000 7f8740583b80 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18d2c2c0 0x55ed18d29d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.294+0000 7f8740583b80 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ed18d44ad0 0x55ed18d46f20 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.294+0000 7f8740583b80 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18ba2e40 0x55ed18d292f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.294+0000 7f8740583b80 1 --2- 172.21.15.43:0/3446490180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 0x55ed18d29830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.294+0000 7f8718c00700 0 client.27661 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:35:40.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.294+0000 7f8740583b80 1 -- 172.21.15.43:0/3446490180 >> 172.21.15.43:0/3446490180 conn(0x55ed18c3c390 msgr2=0x55ed18d1d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:40.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.295+0000 7f8740583b80 1 -- 172.21.15.43:0/3446490180 shutdown_connections 2024-05-27T07:35:40.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.295+0000 7f8740583b80 1 -- 172.21.15.43:0/3446490180 wait complete. 2024-05-27T07:35:40.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.295+0000 7f8740583b80 20 client.27661 trim_cache size 0 max 16384 2024-05-27T07:35:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.328+0000 7f8740583b80 1 Processor -- start 2024-05-27T07:35:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.328+0000 7f8740583b80 1 -- start start 2024-05-27T07:35:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.329+0000 7f8740583b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d24b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.329+0000 7f8740583b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d1d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.329+0000 7f8740583b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18d2c2c0 0x55ed18d40760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.329+0000 7f8740583b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f872c1ae580 con 0x55ed18c8e9f0 2024-05-27T07:35:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.329+0000 7f8740583b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f872c1ac9e0 con 0x55ed18ba2e40 2024-05-27T07:35:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.329+0000 7f8740583b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f872c1adeb0 con 0x55ed18d2c2c0 2024-05-27T07:35:40.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.329+0000 7f8722000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d24b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.329+0000 7f8722000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d24b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:47048/0 (socket says 172.21.15.43:47048) 2024-05-27T07:35:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.329+0000 7f8722000700 1 -- 172.21.15.43:0/1886557470 learned_addr learned my addr 172.21.15.43:0/1886557470 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.330+0000 7f8722000700 1 --2- 172.21.15.43:0/1886557470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18d2c2c0 0x55ed18d40760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.330+0000 7f872aa00700 1 --2- 172.21.15.43:0/1886557470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d1d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.330+0000 7f8722000700 1 -- 172.21.15.43:0/1886557470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18d2c2c0 msgr2=0x55ed18d40760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.330+0000 7f8722000700 1 --2- 172.21.15.43:0/1886557470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18d2c2c0 0x55ed18d40760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.330+0000 7f8722000700 1 -- 172.21.15.43:0/1886557470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 msgr2=0x55ed18d1d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.330+0000 7f8722000700 1 --2- 172.21.15.43:0/1886557470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d1d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.330+0000 7f8722000700 1 -- 172.21.15.43:0/1886557470 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed18d2c0f0 con 0x55ed18ba2e40 2024-05-27T07:35:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.330+0000 7f872aa00700 1 --2- 172.21.15.43:0/1886557470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d1d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:35:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.331+0000 7f8722000700 1 --2- 172.21.15.43:0/1886557470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d24b80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8714013fd0 tx=0x7f8714013360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=666604e4ce0008d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.331+0000 7f871b400700 1 -- 172.21.15.43:0/1886557470 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871401f040 con 0x55ed18ba2e40 2024-05-27T07:35:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.331+0000 7f871b400700 1 -- 172.21.15.43:0/1886557470 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8714009260 con 0x55ed18ba2e40 2024-05-27T07:35:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.331+0000 7f871b400700 1 -- 172.21.15.43:0/1886557470 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714005110 con 0x55ed18ba2e40 2024-05-27T07:35:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.332+0000 7f8740583b80 1 -- 172.21.15.43:0/1886557470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 msgr2=0x55ed18d24b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.332+0000 7f8740583b80 1 --2- 172.21.15.43:0/1886557470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d24b80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8714013fd0 tx=0x7f8714013360 comp rx=0 tx=0).stop 2024-05-27T07:35:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.332+0000 7f8740583b80 1 -- 172.21.15.43:0/1886557470 shutdown_connections 2024-05-27T07:35:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.332+0000 7f8740583b80 1 --2- 172.21.15.43:0/1886557470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18d2c2c0 0x55ed18d40760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.332+0000 7f8740583b80 1 --2- 172.21.15.43:0/1886557470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d24b80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.332+0000 7f8740583b80 1 --2- 172.21.15.43:0/1886557470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18c8e9f0 0x55ed18d1d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.332+0000 7f8740583b80 1 -- 172.21.15.43:0/1886557470 >> 172.21.15.43:0/1886557470 conn(0x55ed18c3c390 msgr2=0x55ed18d21790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.333+0000 7f8740583b80 1 -- 172.21.15.43:0/1886557470 shutdown_connections 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.333+0000 7f8740583b80 1 -- 172.21.15.43:0/1886557470 wait complete. 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.333+0000 7f8740583b80 1 -- start start 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.334+0000 7f8740583b80 1 Processor -- start 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.335+0000 7f8740583b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d3ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.335+0000 7f8740583b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 0x55ed18d381c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.335+0000 7f8740583b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18d2c2c0 0x55ed18d38700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.335+0000 7f8740583b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f872c1ad7e0 con 0x55ed18d2c2c0 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.335+0000 7f8740583b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f872c1ad110 con 0x55ed18ba2e40 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.335+0000 7f8740583b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ed18c51d90 con 0x55ed18c8e9f0 2024-05-27T07:35:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.335+0000 7f8740583b80 10 client.0 _ll_register_callbacks cb 0x55ed18be9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.336+0000 7f8722000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18d2c2c0 0x55ed18d38700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.336+0000 7f8722000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18d2c2c0 0x55ed18d38700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56750/0 (socket says 172.21.15.43:56750) 2024-05-27T07:35:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.336+0000 7f8722000700 1 -- 172.21.15.43:0/1341569334 learned_addr learned my addr 172.21.15.43:0/1341569334 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.336+0000 7f8722000700 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d3ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.336+0000 7f872aa00700 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 0x55ed18d381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.337+0000 7f8722000700 1 -- 172.21.15.43:0/1341569334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 msgr2=0x55ed18d381c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.337+0000 7f8722000700 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 0x55ed18d381c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.337+0000 7f8722000700 1 -- 172.21.15.43:0/1341569334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 msgr2=0x55ed18d3ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.337+0000 7f8722000700 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d3ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.337+0000 7f8722000700 1 -- 172.21.15.43:0/1341569334 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed18d38c40 con 0x55ed18d2c2c0 2024-05-27T07:35:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.337+0000 7f8722000700 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18d2c2c0 0x55ed18d38700 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f8714009f80 tx=0x7f87140155b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b65751a967a176 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.337+0000 7f8718c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:35:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.339+0000 7f8718c00700 1 -- 172.21.15.43:0/1341569334 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871401f070 con 0x55ed18d2c2c0 2024-05-27T07:35:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.339+0000 7f8718c00700 1 -- 172.21.15.43:0/1341569334 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9496+0+0 (secure 0 0 0) 0x7f87140077c0 con 0x55ed18d2c2c0 2024-05-27T07:35:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.339+0000 7f8740583b80 1 -- 172.21.15.43:0/1341569334 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8714009260 con 0x55ed18d2c2c0 2024-05-27T07:35:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.339+0000 7f8718c00700 1 -- 172.21.15.43:0/1341569334 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f8714007e20 con 0x55ed18d2c2c0 2024-05-27T07:35:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.340+0000 7f8740583b80 10 client.18240 fetch_fsmap learned FSMap version 144 2024-05-27T07:35:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.340+0000 7f8740583b80 1 -- 172.21.15.43:0/1341569334 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55ed18daf240 con 0x55ed18d2c2c0 2024-05-27T07:35:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.340+0000 7f8718c00700 1 -- 172.21.15.43:0/1341569334 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8714009260 con 0x55ed18d2c2c0 2024-05-27T07:35:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.340+0000 7f8718c00700 1 -- 172.21.15.43:0/1341569334 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f8714041520 con 0x55ed18d2c2c0 2024-05-27T07:35:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.340+0000 7f8740583b80 10 client.18240 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:35:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.340+0000 7f8740583b80 10 client.18240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:35:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.340+0000 7f8740583b80 20 client.18240 populate_metadata read hostname 'smithi043' 2024-05-27T07:35:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.340+0000 7f8740583b80 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ed18daf730 0x55ed18db1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.341+0000 7f8740583b80 4 client.18240 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:35:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.341+0000 7f8740583b80 1 -- 172.21.15.43:0/1341569334 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ed18c4b570 con 0x55ed18daf730 2024-05-27T07:35:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.341+0000 7f872aa00700 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ed18daf730 0x55ed18db1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.342+0000 7f872aa00700 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ed18daf730 0x55ed18db1b50 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbacd04a2e8d22bc server_cookie=496a15e181de0996 in_seq=0 out_seq=0 2024-05-27T07:35:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.342+0000 7f8718c00700 10 client.18240 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:35:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.342+0000 7f8718c00700 1 -- 172.21.15.43:0/1341569334 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ed18c4b570 con 0x55ed18daf730 2024-05-27T07:35:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.342+0000 7f8718c00700 10 client.18240 handle_command_reply: tid=0 2024-05-27T07:35:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.342+0000 7f8740583b80 1 client.18240 shutdown 2024-05-27T07:35:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.343+0000 7f8740583b80 1 -- 172.21.15.43:0/1341569334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18d2c2c0 msgr2=0x55ed18d38700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.343+0000 7f8740583b80 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18d2c2c0 0x55ed18d38700 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f8714009f80 tx=0x7f87140155b0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.343+0000 7f8740583b80 1 -- 172.21.15.43:0/1341569334 shutdown_connections 2024-05-27T07:35:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.343+0000 7f8740583b80 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55ed18daf730 0x55ed18db1b50 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.343+0000 7f8718c00700 0 client.18240 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:35:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.343+0000 7f8740583b80 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ed18d2c2c0 0x55ed18d38700 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.343+0000 7f8740583b80 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed18ba2e40 0x55ed18d3ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.343+0000 7f8740583b80 1 --2- 172.21.15.43:0/1341569334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed18c8e9f0 0x55ed18d381c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.343+0000 7f8740583b80 1 -- 172.21.15.43:0/1341569334 >> 172.21.15.43:0/1341569334 conn(0x55ed18c3c390 msgr2=0x55ed18d277e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.344+0000 7f8740583b80 1 -- 172.21.15.43:0/1341569334 shutdown_connections 2024-05-27T07:35:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.344+0000 7f8740583b80 1 -- 172.21.15.43:0/1341569334 wait complete. 2024-05-27T07:35:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.345+0000 7f8740583b80 20 client.18240 trim_cache size 0 max 16384 2024-05-27T07:35:40.341 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:35:40.341 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:35:40.341 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:35:40.341 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 -- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f87100b2aa0 msgr2=0x7f87100b4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f87100b2aa0 0x7f87100b4fb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f87240aa0a0 tx=0x7f8724005ec0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 -- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f872c109670 msgr2=0x7f872c1a1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f872c109670 0x7f872c1a1620 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f8724009810 tx=0x7f872400fa70 comp rx=0 tx=0).stop 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 -- 172.21.15.43:0/4239535607 shutdown_connections 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f87100b2aa0 0x7f87100b4fb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f872c109670 0x7f872c1a1620 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f872c1043f0 0x7f872c19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 --2- 172.21.15.43:0/4239535607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f872c108d20 0x7f872c19d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 -- 172.21.15.43:0/4239535607 >> 172.21.15.43:0/4239535607 conn(0x7f872c079d00 msgr2=0x7f872c101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.346+0000 7f872be00700 1 -- 172.21.15.43:0/4239535607 shutdown_connections 2024-05-27T07:35:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:40.347+0000 7f872be00700 1 -- 172.21.15.43:0/4239535607 wait complete. 2024-05-27T07:35:40.815 INFO:tasks.workunit.client.0.smithi043.stdout: 10 50630 5.83 MB/sec execute 341 sec latency 319.817 ms 2024-05-27T07:35:40.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:40 smithi043 ceph-mon[31531]: pgmap v731: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.7 MiB/s wr, 154 op/s 2024-05-27T07:35:40.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:40 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:40.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:40 smithi179 ceph-mon[35888]: pgmap v731: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.7 MiB/s wr, 154 op/s 2024-05-27T07:35:40.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:40 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:40.969 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:40 smithi132 ceph-mon[35283]: pgmap v731: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.7 MiB/s wr, 154 op/s 2024-05-27T07:35:40.969 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:40 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:35:41.815 INFO:tasks.workunit.client.0.smithi043.stdout: 10 50720 5.83 MB/sec execute 342 sec latency 365.624 ms 2024-05-27T07:35:41.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:41 smithi043 ceph-mon[31531]: pgmap v732: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 5.3 MiB/s wr, 222 op/s 2024-05-27T07:35:41.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:41 smithi179 ceph-mon[35888]: pgmap v732: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 5.3 MiB/s wr, 222 op/s 2024-05-27T07:35:42.095 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:41 smithi132 ceph-mon[35283]: pgmap v732: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 5.3 MiB/s wr, 222 op/s 2024-05-27T07:35:42.815 INFO:tasks.workunit.client.0.smithi043.stdout: 10 50848 5.82 MB/sec execute 343 sec latency 330.181 ms 2024-05-27T07:35:43.815 INFO:tasks.workunit.client.0.smithi043.stdout: 10 50929 5.82 MB/sec execute 344 sec latency 366.742 ms 2024-05-27T07:35:44.187 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:35:44.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:35:44.416 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:44 smithi043 ceph-mon[31531]: pgmap v733: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 3.3 MiB/s wr, 139 op/s 2024-05-27T07:35:44.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:44 smithi179 ceph-mon[35888]: pgmap v733: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 3.3 MiB/s wr, 139 op/s 2024-05-27T07:35:44.815 INFO:tasks.workunit.client.0.smithi043.stdout: 10 51021 5.82 MB/sec execute 345 sec latency 88.787 ms 2024-05-27T07:35:44.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:44 smithi132 ceph-mon[35283]: pgmap v733: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 3.3 MiB/s wr, 139 op/s 2024-05-27T07:35:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:45.815 INFO:tasks.workunit.client.0.smithi043.stdout: 10 51111 5.82 MB/sec execute 346 sec latency 99.112 ms 2024-05-27T07:35:46.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:46 smithi043 ceph-mon[31531]: pgmap v734: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 3.3 MiB/s wr, 139 op/s 2024-05-27T07:35:46.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:46 smithi179 ceph-mon[35888]: pgmap v734: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 3.3 MiB/s wr, 139 op/s 2024-05-27T07:35:46.815 INFO:tasks.workunit.client.0.smithi043.stdout: 10 51204 5.83 MB/sec execute 347 sec latency 142.386 ms 2024-05-27T07:35:46.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:46 smithi132 ceph-mon[35283]: pgmap v734: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 3.3 MiB/s wr, 139 op/s 2024-05-27T07:35:47.815 INFO:tasks.workunit.client.0.smithi043.stdout: 10 51345 5.84 MB/sec execute 348 sec latency 94.109 ms 2024-05-27T07:35:48.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:48 smithi043 ceph-mon[31531]: pgmap v735: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 4.9 MiB/s wr, 214 op/s 2024-05-27T07:35:48.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:48 smithi179 ceph-mon[35888]: pgmap v735: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 4.9 MiB/s wr, 214 op/s 2024-05-27T07:35:48.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 51490 5.86 MB/sec execute 349 sec latency 111.117 ms 2024-05-27T07:35:48.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:48 smithi132 ceph-mon[35283]: pgmap v735: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 4.9 MiB/s wr, 214 op/s 2024-05-27T07:35:49.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.470+0000 7fe173200700 1 -- 172.21.15.43:0/501849960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe17410b890 msgr2=0x7fe1741131f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.470+0000 7fe173200700 1 --2- 172.21.15.43:0/501849960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe17410b890 0x7fe1741131f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009990 tx=0x7fe16c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.471+0000 7fe173200700 1 -- 172.21.15.43:0/501849960 shutdown_connections 2024-05-27T07:35:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.471+0000 7fe173200700 1 --2- 172.21.15.43:0/501849960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe17410b890 0x7fe1741131f0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.471+0000 7fe173200700 1 --2- 172.21.15.43:0/501849960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe174106610 0x7fe174106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.471+0000 7fe173200700 1 --2- 172.21.15.43:0/501849960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe17410af40 0x7fe17410b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.471+0000 7fe173200700 1 -- 172.21.15.43:0/501849960 >> 172.21.15.43:0/501849960 conn(0x7fe174102020 msgr2=0x7fe174104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:49.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.471+0000 7fe173200700 1 -- 172.21.15.43:0/501849960 shutdown_connections 2024-05-27T07:35:49.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.471+0000 7fe173200700 1 -- 172.21.15.43:0/501849960 wait complete. 2024-05-27T07:35:49.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.473+0000 7fe173200700 1 Processor -- start 2024-05-27T07:35:49.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.473+0000 7fe173200700 1 -- start start 2024-05-27T07:35:49.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.474+0000 7fe173200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe174106610 0x7fe1741a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:49.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.474+0000 7fe173200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe17410af40 0x7fe1741a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.474+0000 7fe173200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe17410b890 0x7fe1741a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.474+0000 7fe173200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe17407a8c0 con 0x7fe17410af40 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.474+0000 7fe173200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe17407a740 con 0x7fe174106610 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.474+0000 7fe173200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe17407aa40 con 0x7fe17410b890 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.474+0000 7fe16be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe17410af40 0x7fe1741a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.474+0000 7fe16be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe17410af40 0x7fe1741a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40296/0 (socket says 172.21.15.43:40296) 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.474+0000 7fe16be00700 1 -- 172.21.15.43:0/350708381 learned_addr learned my addr 172.21.15.43:0/350708381 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.474+0000 7fe171e00700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe174106610 0x7fe1741a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.475+0000 7fe16be00700 1 -- 172.21.15.43:0/350708381 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe17410b890 msgr2=0x7fe1741a5980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.475+0000 7fe16be00700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe17410b890 0x7fe1741a5980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.475+0000 7fe16be00700 1 -- 172.21.15.43:0/350708381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe174106610 msgr2=0x7fe1741a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.475+0000 7fe16be00700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe174106610 0x7fe1741a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.475+0000 7fe16be00700 1 -- 172.21.15.43:0/350708381 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1741a6020 con 0x7fe17410af40 2024-05-27T07:35:49.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.475+0000 7fe16be00700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe17410af40 0x7fe1741a16f0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00e970 tx=0x7fe15c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:49.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.476+0000 7fe16aa00700 1 -- 172.21.15.43:0/350708381 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c00cbd0 con 0x7fe17410af40 2024-05-27T07:35:49.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.476+0000 7fe173200700 1 -- 172.21.15.43:0/350708381 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1741a6310 con 0x7fe17410af40 2024-05-27T07:35:49.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.476+0000 7fe173200700 1 -- 172.21.15.43:0/350708381 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17406d1d0 con 0x7fe17410af40 2024-05-27T07:35:49.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.476+0000 7fe16aa00700 1 -- 172.21.15.43:0/350708381 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe15c00cd70 con 0x7fe17410af40 2024-05-27T07:35:49.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.476+0000 7fe16aa00700 1 -- 172.21.15.43:0/350708381 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15c005490 con 0x7fe17410af40 2024-05-27T07:35:49.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.478+0000 7fe173200700 1 -- 172.21.15.43:0/350708381 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe150005330 con 0x7fe17410af40 2024-05-27T07:35:49.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.481+0000 7fe16aa00700 1 -- 172.21.15.43:0/350708381 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe15c021440 con 0x7fe17410af40 2024-05-27T07:35:49.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.483+0000 7fe16aa00700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe1580b2bf0 0x7fe1580b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:49.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.484+0000 7fe16aa00700 1 -- 172.21.15.43:0/350708381 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9496+0+0 (secure 0 0 0) 0x7fe15c005230 con 0x7fe17410af40 2024-05-27T07:35:49.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.484+0000 7fe171e00700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe1580b2bf0 0x7fe1580b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:49.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.485+0000 7fe171e00700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe1580b2bf0 0x7fe1580b5100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe164009550 tx=0x7fe1640091b0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:49.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.489+0000 7fe16aa00700 1 -- 172.21.15.43:0/350708381 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe15c0a9fa0 con 0x7fe17410af40 2024-05-27T07:35:49.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 51593 5.85 MB/sec execute 350 sec latency 349.164 ms 2024-05-27T07:35:49.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.959+0000 7fe173200700 1 -- 172.21.15.43:0/350708381 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1500058f0 con 0x7fe17410af40 2024-05-27T07:35:49.960 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:35:49.960 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":144,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:35:39.002310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34448,34484,34424,34403,24484]},"id":1}]} 2024-05-27T07:35:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.964+0000 7fe16aa00700 1 -- 172.21.15.43:0/350708381 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+6124 (secure 0 0 0) 0x7fe15c0a98a0 con 0x7fe17410af40 2024-05-27T07:35:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.967+0000 7fe173200700 1 -- 172.21.15.43:0/350708381 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe1580b2bf0 msgr2=0x7fe1580b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.967+0000 7fe173200700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe1580b2bf0 0x7fe1580b5100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe164009550 tx=0x7fe1640091b0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.967+0000 7fe173200700 1 -- 172.21.15.43:0/350708381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe17410af40 msgr2=0x7fe1741a16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.967+0000 7fe173200700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe17410af40 0x7fe1741a16f0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00e970 tx=0x7fe15c00ec80 comp rx=0 tx=0).stop 2024-05-27T07:35:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.968+0000 7fe173200700 1 -- 172.21.15.43:0/350708381 shutdown_connections 2024-05-27T07:35:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.968+0000 7fe173200700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe1580b2bf0 0x7fe1580b5100 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.968+0000 7fe173200700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe17410b890 0x7fe1741a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.968+0000 7fe173200700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe174106610 0x7fe1741a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.968+0000 7fe173200700 1 --2- 172.21.15.43:0/350708381 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe17410af40 0x7fe1741a16f0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.968+0000 7fe173200700 1 -- 172.21.15.43:0/350708381 >> 172.21.15.43:0/350708381 conn(0x7fe174102020 msgr2=0x7fe174104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.968+0000 7fe173200700 1 -- 172.21.15.43:0/350708381 shutdown_connections 2024-05-27T07:35:49.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:49.969+0000 7fe173200700 1 -- 172.21.15.43:0/350708381 wait complete. 2024-05-27T07:35:49.965 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 144 2024-05-27T07:35:50.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:50 smithi043 ceph-mon[31531]: pgmap v736: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 3.1 MiB/s wr, 143 op/s 2024-05-27T07:35:50.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:50 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/350708381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:50.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:50 smithi179 ceph-mon[35888]: pgmap v736: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 3.1 MiB/s wr, 143 op/s 2024-05-27T07:35:50.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:50 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/350708381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:50.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 51706 5.84 MB/sec execute 351 sec latency 98.291 ms 2024-05-27T07:35:50.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:50 smithi132 ceph-mon[35283]: pgmap v736: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 3.1 MiB/s wr, 143 op/s 2024-05-27T07:35:50.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:50 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/350708381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:35:50.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:35:51.590 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:51.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 51794 5.83 MB/sec execute 352 sec latency 101.876 ms 2024-05-27T07:35:52.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:52 smithi043 ceph-mon[31531]: pgmap v737: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 4.8 MiB/s wr, 219 op/s 2024-05-27T07:35:52.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:52 smithi179 ceph-mon[35888]: pgmap v737: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 4.8 MiB/s wr, 219 op/s 2024-05-27T07:35:52.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 51906 5.83 MB/sec execute 353 sec latency 85.413 ms 2024-05-27T07:35:52.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:52 smithi132 ceph-mon[35283]: pgmap v737: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 4.8 MiB/s wr, 219 op/s 2024-05-27T07:35:53.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 51979 5.82 MB/sec execute 354 sec latency 261.929 ms 2024-05-27T07:35:54.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:54 smithi043 ceph-mon[31531]: pgmap v738: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 151 op/s 2024-05-27T07:35:54.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:54 smithi179 ceph-mon[35888]: pgmap v738: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 151 op/s 2024-05-27T07:35:54.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 52102 5.82 MB/sec execute 355 sec latency 113.541 ms 2024-05-27T07:35:54.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:54 smithi132 ceph-mon[35283]: pgmap v738: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 151 op/s 2024-05-27T07:35:55.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 52213 5.82 MB/sec execute 356 sec latency 91.101 ms 2024-05-27T07:35:56.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:56 smithi043 ceph-mon[31531]: pgmap v739: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 151 op/s 2024-05-27T07:35:56.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:56 smithi179 ceph-mon[35888]: pgmap v739: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 151 op/s 2024-05-27T07:35:56.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 52325 5.82 MB/sec execute 357 sec latency 204.859 ms 2024-05-27T07:35:56.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:56 smithi132 ceph-mon[35283]: pgmap v739: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 151 op/s 2024-05-27T07:35:57.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.058+0000 7f1f82c00700 1 -- 172.21.15.43:0/335979116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f7c0764a0 msgr2=0x7f1f7c114e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:57.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.058+0000 7f1f82c00700 1 --2- 172.21.15.43:0/335979116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f7c0764a0 0x7f1f7c114e20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009990 tx=0x7f1f68009ca0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.060+0000 7f1f82c00700 1 -- 172.21.15.43:0/335979116 shutdown_connections 2024-05-27T07:35:57.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.060+0000 7f1f82c00700 1 --2- 172.21.15.43:0/335979116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f7c115360 0x7f1f7c117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.060+0000 7f1f82c00700 1 --2- 172.21.15.43:0/335979116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f7c075b50 0x7f1f7c075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.060+0000 7f1f82c00700 1 --2- 172.21.15.43:0/335979116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f7c0764a0 0x7f1f7c114e20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.060+0000 7f1f82c00700 1 -- 172.21.15.43:0/335979116 >> 172.21.15.43:0/335979116 conn(0x7f1f7c100030 msgr2=0x7f1f7c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.060+0000 7f1f82c00700 1 -- 172.21.15.43:0/335979116 shutdown_connections 2024-05-27T07:35:57.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.061+0000 7f1f82c00700 1 -- 172.21.15.43:0/335979116 wait complete. 2024-05-27T07:35:57.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.062+0000 7f1f82c00700 1 Processor -- start 2024-05-27T07:35:57.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.062+0000 7f1f82c00700 1 -- start start 2024-05-27T07:35:57.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.063+0000 7f1f82c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f7c075b50 0x7f1f7c1a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.063+0000 7f1f82c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f7c0764a0 0x7f1f7c1a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.063+0000 7f1f82c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f7c115360 0x7f1f7c1a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f82c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c11a960 con 0x7f1f7c075b50 2024-05-27T07:35:57.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f82c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c11a7e0 con 0x7f1f7c115360 2024-05-27T07:35:57.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f82c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c11aae0 con 0x7f1f7c0764a0 2024-05-27T07:35:57.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f7be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f7c075b50 0x7f1f7c1a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f7be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f7c075b50 0x7f1f7c1a1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49622/0 (socket says 172.21.15.43:49622) 2024-05-27T07:35:57.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f7be00700 1 -- 172.21.15.43:0/3473571836 learned_addr learned my addr 172.21.15.43:0/3473571836 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:57.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f7b400700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f7c0764a0 0x7f1f7c1a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f7be00700 1 -- 172.21.15.43:0/3473571836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f7c0764a0 msgr2=0x7f1f7c1a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f7be00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f7c0764a0 0x7f1f7c1a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f80c00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f7c115360 0x7f1f7c1a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f7be00700 1 -- 172.21.15.43:0/3473571836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f7c115360 msgr2=0x7f1f7c1a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f7be00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f7c115360 0x7f1f7c1a5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.064+0000 7f1f7be00700 1 -- 172.21.15.43:0/3473571836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c1a6220 con 0x7f1f7c075b50 2024-05-27T07:35:57.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.065+0000 7f1f7be00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f7c075b50 0x7f1f7c1a1350 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00d950 tx=0x7f1f6c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.066+0000 7f1f78c00700 1 -- 172.21.15.43:0/3473571836 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c016070 con 0x7f1f7c075b50 2024-05-27T07:35:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.066+0000 7f1f82c00700 1 -- 172.21.15.43:0/3473571836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7c1a6510 con 0x7f1f7c075b50 2024-05-27T07:35:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.066+0000 7f1f78c00700 1 -- 172.21.15.43:0/3473571836 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f6c010460 con 0x7f1f7c075b50 2024-05-27T07:35:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.066+0000 7f1f78c00700 1 -- 172.21.15.43:0/3473571836 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c00f620 con 0x7f1f7c075b50 2024-05-27T07:35:57.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.066+0000 7f1f82c00700 1 -- 172.21.15.43:0/3473571836 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7c1ade30 con 0x7f1f7c075b50 2024-05-27T07:35:57.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.072+0000 7f1f78c00700 1 -- 172.21.15.43:0/3473571836 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1f6c019070 con 0x7f1f7c075b50 2024-05-27T07:35:57.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.074+0000 7f1f91718b80 1 Processor -- start 2024-05-27T07:35:57.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.075+0000 7f1f78c00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1f600b2af0 0x7f1f600b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.075+0000 7f1f78c00700 1 -- 172.21.15.43:0/3473571836 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 9496+0+0 (secure 0 0 0) 0x7f1f6c0e1ff0 con 0x7f1f7c075b50 2024-05-27T07:35:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.076+0000 7f1f91718b80 1 -- start start 2024-05-27T07:35:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.076+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c0776330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.076+0000 7f1f80c00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1f600b2af0 0x7f1f600b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.076+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 0x55e3c0776870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.076+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0783110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.076+0000 7f1f91718b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e3c06a4e40 con 0x55e3c0776e70 2024-05-27T07:35:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.076+0000 7f1f91718b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e3c06949e0 con 0x55e3c05f7e40 2024-05-27T07:35:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.076+0000 7f1f91718b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e3c0698a20 con 0x55e3c06e39f0 2024-05-27T07:35:57.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.077+0000 7f1f80c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0783110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.077+0000 7f1f80c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0783110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49638/0 (socket says 172.21.15.43:49638) 2024-05-27T07:35:57.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.077+0000 7f1f80c00700 1 -- 172.21.15.43:0/1656528525 learned_addr learned my addr 172.21.15.43:0/1656528525 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:57.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.077+0000 7f1f7b400700 1 --2- 172.21.15.43:0/1656528525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 0x55e3c0776870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.078+0000 7f1f7be00700 1 --2- 172.21.15.43:0/1656528525 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c0776330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.078+0000 7f1f7b400700 1 -- 172.21.15.43:0/1656528525 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 msgr2=0x55e3c0776330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:57.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.078+0000 7f1f7b400700 1 --2- 172.21.15.43:0/1656528525 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c0776330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.078+0000 7f1f7b400700 1 -- 172.21.15.43:0/1656528525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 msgr2=0x55e3c0783110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:57.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.078+0000 7f1f7b400700 1 --2- 172.21.15.43:0/1656528525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0783110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.078+0000 7f1f7b400700 1 -- 172.21.15.43:0/1656528525 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3c065d420 con 0x55e3c05f7e40 2024-05-27T07:35:57.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.079+0000 7f1f80c00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1f600b2af0 0x7f1f600b5000 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1f740076f0 tx=0x7f1f74009510 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:57.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.079+0000 7f1f7b400700 1 --2- 172.21.15.43:0/1656528525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 0x55e3c0776870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1f68000c00 tx=0x7f1f680096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4896a5a01279264 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:57.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.080+0000 7f1f71600700 1 -- 172.21.15.43:0/1656528525 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f680054a0 con 0x55e3c05f7e40 2024-05-27T07:35:57.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.080+0000 7f1f71600700 1 -- 172.21.15.43:0/1656528525 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f68005640 con 0x55e3c05f7e40 2024-05-27T07:35:57.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.080+0000 7f1f71600700 1 -- 172.21.15.43:0/1656528525 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800ba50 con 0x55e3c05f7e40 2024-05-27T07:35:57.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.081+0000 7f1f91718b80 1 -- 172.21.15.43:0/1656528525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 msgr2=0x55e3c0776870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:57.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.081+0000 7f1f91718b80 1 --2- 172.21.15.43:0/1656528525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 0x55e3c0776870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1f68000c00 tx=0x7f1f680096a0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.082+0000 7f1f91718b80 1 -- 172.21.15.43:0/1656528525 shutdown_connections 2024-05-27T07:35:57.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.082+0000 7f1f91718b80 1 --2- 172.21.15.43:0/1656528525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0783110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.082+0000 7f1f91718b80 1 --2- 172.21.15.43:0/1656528525 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c0776330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.082+0000 7f1f91718b80 1 --2- 172.21.15.43:0/1656528525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 0x55e3c0776870 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.082+0000 7f1f91718b80 1 -- 172.21.15.43:0/1656528525 >> 172.21.15.43:0/1656528525 conn(0x55e3c0691390 msgr2=0x55e3c07724c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:57.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.082+0000 7f1f91718b80 1 -- 172.21.15.43:0/1656528525 shutdown_connections 2024-05-27T07:35:57.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.083+0000 7f1f91718b80 1 -- 172.21.15.43:0/1656528525 wait complete. 2024-05-27T07:35:57.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.083+0000 7f1f91718b80 1 -- start start 2024-05-27T07:35:57.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.084+0000 7f1f91718b80 1 Processor -- start 2024-05-27T07:35:57.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.084+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c05f7e40 0x55e3c07805e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.084+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c0780b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.085+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c0776e70 0x55e3c0781060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.085+0000 7f1f91718b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e3c059afa0 con 0x55e3c05f7e40 2024-05-27T07:35:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.085+0000 7f1f91718b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e3c0635760 con 0x55e3c0776e70 2024-05-27T07:35:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.085+0000 7f1f91718b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e3c0794b30 con 0x55e3c06e39f0 2024-05-27T07:35:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.085+0000 7f1f91718b80 10 client.0 _ll_register_callbacks cb 0x55e3c063ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.085+0000 7f1f7be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c05f7e40 0x55e3c07805e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.085+0000 7f1f7be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c05f7e40 0x55e3c07805e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49652/0 (socket says 172.21.15.43:49652) 2024-05-27T07:35:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.085+0000 7f1f7be00700 1 -- 172.21.15.43:0/3416801516 learned_addr learned my addr 172.21.15.43:0/3416801516 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.085+0000 7f1f80c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c0776e70 0x55e3c0781060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.086+0000 7f1f7b400700 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c0780b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.086+0000 7f1f80c00700 1 -- 172.21.15.43:0/3416801516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 msgr2=0x55e3c0780b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.086+0000 7f1f80c00700 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c0780b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.086+0000 7f1f80c00700 1 -- 172.21.15.43:0/3416801516 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c05f7e40 msgr2=0x55e3c07805e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.086+0000 7f1f80c00700 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c05f7e40 0x55e3c07805e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.086+0000 7f1f80c00700 1 -- 172.21.15.43:0/3416801516 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3c05e48d0 con 0x55e3c0776e70 2024-05-27T07:35:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.087+0000 7f1f80c00700 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c0776e70 0x55e3c0781060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1f74009fd0 tx=0x7f1f7400d050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc1d2c1b0e040313 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:57.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.088+0000 7f1f66000700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:35:57.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.088+0000 7f1f66000700 1 -- 172.21.15.43:0/3416801516 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f74028060 con 0x55e3c0776e70 2024-05-27T07:35:57.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.088+0000 7f1f91718b80 1 -- 172.21.15.43:0/3416801516 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f68005640 con 0x55e3c0776e70 2024-05-27T07:35:57.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.089+0000 7f1f66000700 1 -- 172.21.15.43:0/3416801516 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9496+0+0 (secure 0 0 0) 0x7f1f74003c10 con 0x55e3c0776e70 2024-05-27T07:35:57.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.090+0000 7f1f66000700 1 -- 172.21.15.43:0/3416801516 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f1f7402f050 con 0x55e3c0776e70 2024-05-27T07:35:57.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.090+0000 7f1f91718b80 10 client.37640 fetch_fsmap learned FSMap version 144 2024-05-27T07:35:57.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.090+0000 7f1f91718b80 1 -- 172.21.15.43:0/3416801516 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55e3c0665060 con 0x55e3c0776e70 2024-05-27T07:35:57.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.091+0000 7f1f66000700 1 -- 172.21.15.43:0/3416801516 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f68005640 con 0x55e3c0776e70 2024-05-27T07:35:57.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.091+0000 7f1f66000700 1 -- 172.21.15.43:0/3416801516 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f1f74022030 con 0x55e3c0776e70 2024-05-27T07:35:57.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.092+0000 7f1f91718b80 10 client.37640 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:35:57.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.092+0000 7f1f91718b80 10 client.37640 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:35:57.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.092+0000 7f1f91718b80 20 client.37640 populate_metadata read hostname 'smithi043' 2024-05-27T07:35:57.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.092+0000 7f1f91718b80 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55e3c07a1b80 0x55e3c07a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.092+0000 7f1f91718b80 4 client.37640 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:35:57.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.092+0000 7f1f91718b80 1 -- 172.21.15.43:0/3416801516 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3c06a6d90 con 0x55e3c07a1b80 2024-05-27T07:35:57.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.093+0000 7f1f7be00700 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55e3c07a1b80 0x55e3c07a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.096+0000 7f1f7be00700 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55e3c07a1b80 0x55e3c07a3ff0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1a47aa17c4c1cf8c server_cookie=1591f7336db1272a in_seq=0 out_seq=0 2024-05-27T07:35:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.096+0000 7f1f66000700 10 client.37640 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:35:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f66000700 1 -- 172.21.15.43:0/3416801516 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e3c06a6d90 con 0x55e3c07a1b80 2024-05-27T07:35:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f66000700 10 client.37640 handle_command_reply: tid=0 2024-05-27T07:35:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f91718b80 1 client.37640 shutdown 2024-05-27T07:35:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f91718b80 1 -- 172.21.15.43:0/3416801516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c0776e70 msgr2=0x55e3c0781060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f91718b80 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c0776e70 0x55e3c0781060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1f74009fd0 tx=0x7f1f7400d050 comp rx=0 tx=0).stop 2024-05-27T07:35:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f91718b80 1 -- 172.21.15.43:0/3416801516 shutdown_connections 2024-05-27T07:35:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f91718b80 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55e3c07a1b80 0x55e3c07a3ff0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f91718b80 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c0776e70 0x55e3c0781060 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f91718b80 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c05f7e40 0x55e3c07805e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f66000700 0 client.37640 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:35:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f91718b80 1 --2- 172.21.15.43:0/3416801516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c0780b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.098+0000 7f1f91718b80 1 -- 172.21.15.43:0/3416801516 >> 172.21.15.43:0/3416801516 conn(0x55e3c0691390 msgr2=0x55e3c0776310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.099+0000 7f1f91718b80 1 -- 172.21.15.43:0/3416801516 shutdown_connections 2024-05-27T07:35:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.099+0000 7f1f91718b80 1 -- 172.21.15.43:0/3416801516 wait complete. 2024-05-27T07:35:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.100+0000 7f1f91718b80 20 client.37640 trim_cache size 0 max 16384 2024-05-27T07:35:57.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.133+0000 7f1f91718b80 1 Processor -- start 2024-05-27T07:35:57.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.133+0000 7f1f91718b80 1 -- start start 2024-05-27T07:35:57.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c05f7e40 0x55e3c077f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c06e39f0 0x55e3c077bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c077a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f91718b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e3c0772680 con 0x55e3c0776e70 2024-05-27T07:35:57.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f91718b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c1aea20 con 0x55e3c06e39f0 2024-05-27T07:35:57.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f91718b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c074860 con 0x55e3c05f7e40 2024-05-27T07:35:57.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f7b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c06e39f0 0x55e3c077bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f7b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c06e39f0 0x55e3c077bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:49610/0 (socket says 172.21.15.43:49610) 2024-05-27T07:35:57.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f7b400700 1 -- 172.21.15.43:0/2805460007 learned_addr learned my addr 172.21.15.43:0/2805460007 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:57.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f7be00700 1 --2- 172.21.15.43:0/2805460007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c05f7e40 0x55e3c077f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.134+0000 7f1f80c00700 1 --2- 172.21.15.43:0/2805460007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c077a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.135+0000 7f1f7be00700 1 -- 172.21.15.43:0/2805460007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c06e39f0 msgr2=0x55e3c077bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:57.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.135+0000 7f1f7be00700 1 --2- 172.21.15.43:0/2805460007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c06e39f0 0x55e3c077bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.135+0000 7f1f7be00700 1 -- 172.21.15.43:0/2805460007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 msgr2=0x55e3c077a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:57.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.136+0000 7f1f7be00700 1 --2- 172.21.15.43:0/2805460007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c077a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.136+0000 7f1f7be00700 1 -- 172.21.15.43:0/2805460007 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3c0777a50 con 0x55e3c05f7e40 2024-05-27T07:35:57.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.136+0000 7f1f80c00700 1 --2- 172.21.15.43:0/2805460007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c077a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:35:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.136+0000 7f1f7be00700 1 --2- 172.21.15.43:0/2805460007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c05f7e40 0x55e3c077f540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0a4f20 tx=0x7f1f6c0a8c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cd74d5862d1fa80 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.137+0000 7f1f70c00700 1 -- 172.21.15.43:0/2805460007 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c0c8070 con 0x55e3c05f7e40 2024-05-27T07:35:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.138+0000 7f1f70c00700 1 -- 172.21.15.43:0/2805460007 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f6c0b1d00 con 0x55e3c05f7e40 2024-05-27T07:35:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.138+0000 7f1f70c00700 1 -- 172.21.15.43:0/2805460007 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c020040 con 0x55e3c05f7e40 2024-05-27T07:35:57.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.139+0000 7f1f91718b80 1 -- 172.21.15.43:0/2805460007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c05f7e40 msgr2=0x55e3c077f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:57.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.139+0000 7f1f91718b80 1 --2- 172.21.15.43:0/2805460007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c05f7e40 0x55e3c077f540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0a4f20 tx=0x7f1f6c0a8c70 comp rx=0 tx=0).stop 2024-05-27T07:35:57.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.140+0000 7f1f91718b80 1 -- 172.21.15.43:0/2805460007 shutdown_connections 2024-05-27T07:35:57.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.140+0000 7f1f91718b80 1 --2- 172.21.15.43:0/2805460007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c077a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.140+0000 7f1f91718b80 1 --2- 172.21.15.43:0/2805460007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c05f7e40 0x55e3c077f540 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.141+0000 7f1f91718b80 1 --2- 172.21.15.43:0/2805460007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c06e39f0 0x55e3c077bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.141+0000 7f1f91718b80 1 -- 172.21.15.43:0/2805460007 >> 172.21.15.43:0/2805460007 conn(0x55e3c0691390 msgr2=0x55e3c0779ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.141+0000 7f1f91718b80 1 -- 172.21.15.43:0/2805460007 shutdown_connections 2024-05-27T07:35:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.141+0000 7f1f91718b80 1 -- 172.21.15.43:0/2805460007 wait complete. 2024-05-27T07:35:57.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.142+0000 7f1f91718b80 1 -- start start 2024-05-27T07:35:57.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.142+0000 7f1f91718b80 1 Processor -- start 2024-05-27T07:35:57.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.144+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 0x55e3c0799dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.144+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c079a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.144+0000 7f1f91718b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0795360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.145+0000 7f1f91718b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c074f90 con 0x55e3c0776e70 2024-05-27T07:35:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.145+0000 7f1f91718b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c1ae350 con 0x55e3c05f7e40 2024-05-27T07:35:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.145+0000 7f1f91718b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c075660 con 0x55e3c06e39f0 2024-05-27T07:35:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.145+0000 7f1f91718b80 10 client.0 _ll_register_callbacks cb 0x55e3c063ec70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:35:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.145+0000 7f1f80c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0795360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.145+0000 7f1f80c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0795360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49660/0 (socket says 172.21.15.43:49660) 2024-05-27T07:35:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.145+0000 7f1f80c00700 1 -- 172.21.15.43:0/344532282 learned_addr learned my addr 172.21.15.43:0/344532282 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:35:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.145+0000 7f1f7be00700 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 0x55e3c0799dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.146+0000 7f1f7b400700 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c079a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.147+0000 7f1f80c00700 1 -- 172.21.15.43:0/344532282 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 msgr2=0x55e3c079a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.147+0000 7f1f80c00700 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c079a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.147+0000 7f1f80c00700 1 -- 172.21.15.43:0/344532282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 msgr2=0x55e3c0799dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:35:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.147+0000 7f1f80c00700 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 0x55e3c0799dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.147+0000 7f1f80c00700 1 -- 172.21.15.43:0/344532282 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3c0795a10 con 0x55e3c0776e70 2024-05-27T07:35:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.149+0000 7f1f80c00700 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0795360 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1f7401c040 tx=0x7f1f7401a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82af394c70fac58f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:35:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.149+0000 7f1f66000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:35:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.150+0000 7f1f66000700 1 -- 172.21.15.43:0/344532282 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7402f050 con 0x55e3c0776e70 2024-05-27T07:35:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.150+0000 7f1f66000700 1 -- 172.21.15.43:0/344532282 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(153..153 src has 1..153) ==== 9496+0+0 (secure 0 0 0) 0x7f1f7402a300 con 0x55e3c0776e70 2024-05-27T07:35:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.151+0000 7f1f91718b80 1 -- 172.21.15.43:0/344532282 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f6c0b1d00 con 0x55e3c0776e70 2024-05-27T07:35:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.152+0000 7f1f66000700 1 -- 172.21.15.43:0/344532282 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f1f74028060 con 0x55e3c0776e70 2024-05-27T07:35:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.152+0000 7f1f91718b80 10 client.18264 fetch_fsmap learned FSMap version 144 2024-05-27T07:35:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.152+0000 7f1f91718b80 1 -- 172.21.15.43:0/344532282 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55e3c080bfc0 con 0x55e3c0776e70 2024-05-27T07:35:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.153+0000 7f1f66000700 1 -- 172.21.15.43:0/344532282 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f6c0b1d00 con 0x55e3c0776e70 2024-05-27T07:35:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.153+0000 7f1f66000700 1 -- 172.21.15.43:0/344532282 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f1f74032070 con 0x55e3c0776e70 2024-05-27T07:35:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.154+0000 7f1f91718b80 10 client.18264 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:35:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.154+0000 7f1f91718b80 10 client.18264 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:35:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.154+0000 7f1f91718b80 20 client.18264 populate_metadata read hostname 'smithi043' 2024-05-27T07:35:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.154+0000 7f1f91718b80 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55e3c080c6f0 0x55e3c080eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:35:57.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.154+0000 7f1f91718b80 4 client.18264 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:35:57.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.154+0000 7f1f91718b80 1 -- 172.21.15.43:0/344532282 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e3c06a0570 con 0x55e3c080c6f0 2024-05-27T07:35:57.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.154+0000 7f1f7be00700 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55e3c080c6f0 0x55e3c080eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:35:57.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.155+0000 7f1f66000700 10 client.18264 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:35:57.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.156+0000 7f1f7be00700 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55e3c080c6f0 0x55e3c080eb10 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dc75956d6577acc7 server_cookie=2149e84a713ba8d5 in_seq=0 out_seq=0 2024-05-27T07:35:57.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.157+0000 7f1f66000700 1 -- 172.21.15.43:0/344532282 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e3c06a0570 con 0x55e3c080c6f0 2024-05-27T07:35:57.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.157+0000 7f1f66000700 10 client.18264 handle_command_reply: tid=0 2024-05-27T07:35:57.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.157+0000 7f1f91718b80 1 client.18264 shutdown 2024-05-27T07:35:57.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.158+0000 7f1f91718b80 1 -- 172.21.15.43:0/344532282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 msgr2=0x55e3c0795360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:57.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.158+0000 7f1f91718b80 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0795360 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1f7401c040 tx=0x7f1f7401a970 comp rx=0 tx=0).stop 2024-05-27T07:35:57.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.159+0000 7f1f91718b80 1 -- 172.21.15.43:0/344532282 shutdown_connections 2024-05-27T07:35:57.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.159+0000 7f1f91718b80 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55e3c080c6f0 0x55e3c080eb10 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.159+0000 7f1f66000700 0 client.18264 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:35:57.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.159+0000 7f1f91718b80 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e3c0776e70 0x55e3c0795360 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.159+0000 7f1f91718b80 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e3c05f7e40 0x55e3c0799dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.159+0000 7f1f91718b80 1 --2- 172.21.15.43:0/344532282 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e3c06e39f0 0x55e3c079a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.159+0000 7f1f91718b80 1 -- 172.21.15.43:0/344532282 >> 172.21.15.43:0/344532282 conn(0x55e3c0691390 msgr2=0x55e3c0780df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.160+0000 7f1f91718b80 1 -- 172.21.15.43:0/344532282 shutdown_connections 2024-05-27T07:35:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.160+0000 7f1f91718b80 1 -- 172.21.15.43:0/344532282 wait complete. 2024-05-27T07:35:57.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.161+0000 7f1f91718b80 20 client.18264 trim_cache size 0 max 16384 2024-05-27T07:35:57.157 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:35:57.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.162+0000 7f1f82c00700 1 -- 172.21.15.43:0/3473571836 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1f600b2af0 msgr2=0x7f1f600b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:57.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.162+0000 7f1f82c00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1f600b2af0 0x7f1f600b5000 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1f740076f0 tx=0x7f1f74009510 comp rx=0 tx=0).stop 2024-05-27T07:35:57.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.162+0000 7f1f82c00700 1 -- 172.21.15.43:0/3473571836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f7c075b50 msgr2=0x7f1f7c1a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:35:57.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.162+0000 7f1f82c00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f7c075b50 0x7f1f7c1a1350 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00d950 tx=0x7f1f6c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:35:57.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.162+0000 7f1f82c00700 1 -- 172.21.15.43:0/3473571836 shutdown_connections 2024-05-27T07:35:57.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.162+0000 7f1f82c00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1f600b2af0 0x7f1f600b5000 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.162+0000 7f1f82c00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f7c115360 0x7f1f7c1a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.162+0000 7f1f82c00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f7c075b50 0x7f1f7c1a1350 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.162+0000 7f1f82c00700 1 --2- 172.21.15.43:0/3473571836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f7c0764a0 0x7f1f7c1a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:35:57.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.162+0000 7f1f82c00700 1 -- 172.21.15.43:0/3473571836 >> 172.21.15.43:0/3473571836 conn(0x7f1f7c100030 msgr2=0x7f1f7c1006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:35:57.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.163+0000 7f1f82c00700 1 -- 172.21.15.43:0/3473571836 shutdown_connections 2024-05-27T07:35:57.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:35:57.163+0000 7f1f82c00700 1 -- 172.21.15.43:0/3473571836 wait complete. 2024-05-27T07:35:57.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 52458 5.83 MB/sec execute 358 sec latency 108.892 ms 2024-05-27T07:35:58.167 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:35:58.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:35:58 smithi043 ceph-mon[31531]: pgmap v740: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 4.8 MiB/s wr, 216 op/s 2024-05-27T07:35:58.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:35:58 smithi179 ceph-mon[35888]: pgmap v740: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 4.8 MiB/s wr, 216 op/s 2024-05-27T07:35:58.794 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:35:58.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 52582 5.84 MB/sec execute 359 sec latency 89.786 ms 2024-05-27T07:35:58.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:35:58 smithi132 ceph-mon[35283]: pgmap v740: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 4.8 MiB/s wr, 216 op/s 2024-05-27T07:35:59.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 52678 5.83 MB/sec execute 360 sec latency 110.781 ms 2024-05-27T07:36:00.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:00 smithi043 ceph-mon[31531]: pgmap v741: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 141 op/s 2024-05-27T07:36:00.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:00 smithi179 ceph-mon[35888]: pgmap v741: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 141 op/s 2024-05-27T07:36:00.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 52795 5.83 MB/sec execute 361 sec latency 172.778 ms 2024-05-27T07:36:00.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:00 smithi132 ceph-mon[35283]: pgmap v741: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 141 op/s 2024-05-27T07:36:01.817 INFO:tasks.workunit.client.0.smithi043.stdout: 10 52924 5.86 MB/sec execute 362 sec latency 150.433 ms 2024-05-27T07:36:02.713 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:02 smithi179 ceph-mon[35888]: pgmap v742: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.4 MiB/s wr, 221 op/s 2024-05-27T07:36:02.816 INFO:tasks.workunit.client.0.smithi043.stdout: 10 53029 5.86 MB/sec execute 363 sec latency 138.604 ms 2024-05-27T07:36:02.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:02 smithi132 ceph-mon[35283]: pgmap v742: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.4 MiB/s wr, 221 op/s 2024-05-27T07:36:02.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:02 smithi043 ceph-mon[31531]: pgmap v742: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.4 MiB/s wr, 221 op/s 2024-05-27T07:36:03.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:03 smithi043 ceph-mon[31531]: osdmap e154: 12 total, 12 up, 12 in 2024-05-27T07:36:03.817 INFO:tasks.workunit.client.0.smithi043.stdout: 10 53145 5.86 MB/sec execute 364 sec latency 153.386 ms 2024-05-27T07:36:03.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:03 smithi132 ceph-mon[35283]: osdmap e154: 12 total, 12 up, 12 in 2024-05-27T07:36:03.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:03 smithi179 ceph-mon[35888]: osdmap e154: 12 total, 12 up, 12 in 2024-05-27T07:36:04.817 INFO:tasks.workunit.client.0.smithi043.stdout: 10 53246 5.86 MB/sec execute 365 sec latency 131.228 ms 2024-05-27T07:36:04.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:04 smithi132 ceph-mon[35283]: pgmap v744: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 174 op/s 2024-05-27T07:36:04.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:04 smithi043 ceph-mon[31531]: pgmap v744: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 174 op/s 2024-05-27T07:36:04.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:04 smithi179 ceph-mon[35888]: pgmap v744: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 174 op/s 2024-05-27T07:36:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.130+0000 7fa23ea00700 1 -- 172.21.15.43:0/3912864644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa238108730 msgr2=0x7fa23806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.130+0000 7fa23ea00700 1 --2- 172.21.15.43:0/3912864644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa238108730 0x7fa23806b600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa2300098f0 tx=0x7fa230009c00 comp rx=0 tx=0).stop 2024-05-27T07:36:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.131+0000 7fa23ea00700 1 -- 172.21.15.43:0/3912864644 shutdown_connections 2024-05-27T07:36:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.131+0000 7fa23ea00700 1 --2- 172.21.15.43:0/3912864644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa238108730 0x7fa23806b600 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.131+0000 7fa23ea00700 1 --2- 172.21.15.43:0/3912864644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa2381034b0 0x7fa2381038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.131+0000 7fa23ea00700 1 --2- 172.21.15.43:0/3912864644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa238107de0 0x7fa2381081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.131+0000 7fa23ea00700 1 -- 172.21.15.43:0/3912864644 >> 172.21.15.43:0/3912864644 conn(0x7fa2380fedb0 msgr2=0x7fa2381011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.131+0000 7fa23ea00700 1 -- 172.21.15.43:0/3912864644 shutdown_connections 2024-05-27T07:36:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.131+0000 7fa23ea00700 1 -- 172.21.15.43:0/3912864644 wait complete. 2024-05-27T07:36:05.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.133+0000 7fa23ea00700 1 Processor -- start 2024-05-27T07:36:05.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.133+0000 7fa23ea00700 1 -- start start 2024-05-27T07:36:05.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.135+0000 7fa23ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2381034b0 0x7fa23810fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.136+0000 7fa23ea00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa238107de0 0x7fa2381101f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.136+0000 7fa23ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa238108730 0x7fa2381159b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.136+0000 7fa237e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2381034b0 0x7fa23810fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.136+0000 7fa23ea00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa2381166b0 con 0x7fa2381034b0 2024-05-27T07:36:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.137+0000 7fa23ea00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa23810bfe0 con 0x7fa238108730 2024-05-27T07:36:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.137+0000 7fa23ea00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa238116830 con 0x7fa238107de0 2024-05-27T07:36:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.137+0000 7fa237e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2381034b0 0x7fa23810fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43574/0 (socket says 172.21.15.43:43574) 2024-05-27T07:36:05.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.137+0000 7fa237e00700 1 -- 172.21.15.43:0/4226054810 learned_addr learned my addr 172.21.15.43:0/4226054810 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:05.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.137+0000 7fa23ca00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa238108730 0x7fa2381159b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.137+0000 7fa22f400700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa238107de0 0x7fa2381101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.138+0000 7fa23ca00700 1 -- 172.21.15.43:0/4226054810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa238107de0 msgr2=0x7fa2381101f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.138+0000 7fa23ca00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa238107de0 0x7fa2381101f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.138+0000 7fa23ca00700 1 -- 172.21.15.43:0/4226054810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2381034b0 msgr2=0x7fa23810fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.138+0000 7fa23ca00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2381034b0 0x7fa23810fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.138+0000 7fa23ca00700 1 -- 172.21.15.43:0/4226054810 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2381161e0 con 0x7fa238108730 2024-05-27T07:36:05.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.139+0000 7fa22f400700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa238107de0 0x7fa2381101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:05.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.140+0000 7fa23ca00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa238108730 0x7fa2381159b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa2300098f0 tx=0x7fa23000b560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:05.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.140+0000 7fa235600700 1 -- 172.21.15.43:0/4226054810 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23000bb90 con 0x7fa238108730 2024-05-27T07:36:05.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.140+0000 7fa235600700 1 -- 172.21.15.43:0/4226054810 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa2300054d0 con 0x7fa238108730 2024-05-27T07:36:05.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.140+0000 7fa23ea00700 1 -- 172.21.15.43:0/4226054810 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa238116470 con 0x7fa238108730 2024-05-27T07:36:05.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.140+0000 7fa235600700 1 -- 172.21.15.43:0/4226054810 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23000bb90 con 0x7fa238108730 2024-05-27T07:36:05.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.141+0000 7fa23ea00700 1 -- 172.21.15.43:0/4226054810 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2381a3100 con 0x7fa238108730 2024-05-27T07:36:05.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.148+0000 7fa235600700 1 -- 172.21.15.43:0/4226054810 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa230003cb0 con 0x7fa238108730 2024-05-27T07:36:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.151+0000 7fa235600700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa21c0b2ec0 0x7fa21c0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.152+0000 7fa235600700 1 -- 172.21.15.43:0/4226054810 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 9584+0+0 (secure 0 0 0) 0x7fa23000b990 con 0x7fa238108730 2024-05-27T07:36:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.152+0000 7fa237e00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa21c0b2ec0 0x7fa21c0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.153+0000 7fa237e00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa21c0b2ec0 0x7fa21c0b53d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa2280062a0 tx=0x7fa22800b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:05.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.170+0000 7fa24d460b80 1 Processor -- start 2024-05-27T07:36:05.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.172+0000 7fa24d460b80 1 -- start start 2024-05-27T07:36:05.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.174+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec505787f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.174+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50578d30 0x55ec50572870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.175+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 0x55ec505731c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.175+0000 7fa24d460b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ec504a0e40 con 0x55ec503f3e40 2024-05-27T07:36:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.175+0000 7fa237e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec505787f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.175+0000 7fa237e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec505787f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43586/0 (socket says 172.21.15.43:43586) 2024-05-27T07:36:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.175+0000 7fa237e00700 1 -- 172.21.15.43:0/1820419589 learned_addr learned my addr 172.21.15.43:0/1820419589 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.176+0000 7fa22f400700 1 --2- 172.21.15.43:0/1820419589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50578d30 0x55ec50572870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.176+0000 7fa24d460b80 1 -- 172.21.15.43:0/1820419589 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ec504909e0 con 0x55ec50572db0 2024-05-27T07:36:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.176+0000 7fa24d460b80 1 -- 172.21.15.43:0/1820419589 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ec50494a20 con 0x55ec50578d30 2024-05-27T07:36:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.176+0000 7fa23ca00700 1 --2- 172.21.15.43:0/1820419589 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 0x55ec505731c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.177+0000 7fa237e00700 1 -- 172.21.15.43:0/1820419589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50578d30 msgr2=0x55ec50572870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.177+0000 7fa237e00700 1 --2- 172.21.15.43:0/1820419589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50578d30 0x55ec50572870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.177+0000 7fa237e00700 1 -- 172.21.15.43:0/1820419589 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 msgr2=0x55ec505731c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.177+0000 7fa237e00700 1 --2- 172.21.15.43:0/1820419589 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 0x55ec505731c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.177+0000 7fa237e00700 1 -- 172.21.15.43:0/1820419589 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec50459420 con 0x55ec503f3e40 2024-05-27T07:36:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.178+0000 7fa237e00700 1 --2- 172.21.15.43:0/1820419589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec505787f0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa228015760 tx=0x7fa228013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a686bf0916ba1c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.179+0000 7fa22d600700 1 -- 172.21.15.43:0/1820419589 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22801c050 con 0x55ec503f3e40 2024-05-27T07:36:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.180+0000 7fa22d600700 1 -- 172.21.15.43:0/1820419589 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa228026070 con 0x55ec503f3e40 2024-05-27T07:36:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.180+0000 7fa22d600700 1 -- 172.21.15.43:0/1820419589 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2280168b0 con 0x55ec503f3e40 2024-05-27T07:36:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.181+0000 7fa24d460b80 1 -- 172.21.15.43:0/1820419589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 msgr2=0x55ec505787f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.181+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1820419589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec505787f0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa228015760 tx=0x7fa228013000 comp rx=0 tx=0).stop 2024-05-27T07:36:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.181+0000 7fa24d460b80 1 -- 172.21.15.43:0/1820419589 shutdown_connections 2024-05-27T07:36:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.181+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1820419589 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 0x55ec505731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.181+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1820419589 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec505787f0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.181+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1820419589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50578d30 0x55ec50572870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.182+0000 7fa24d460b80 1 -- 172.21.15.43:0/1820419589 >> 172.21.15.43:0/1820419589 conn(0x55ec50453fc0 msgr2=0x55ec5056d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.182+0000 7fa24d460b80 1 -- 172.21.15.43:0/1820419589 shutdown_connections 2024-05-27T07:36:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.183+0000 7fa24d460b80 1 -- 172.21.15.43:0/1820419589 wait complete. 2024-05-27T07:36:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.183+0000 7fa24d460b80 1 -- start start 2024-05-27T07:36:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.184+0000 7fa24d460b80 1 Processor -- start 2024-05-27T07:36:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.185+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec503f3e40 0x55ec5057c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.185+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec50572db0 0x55ec5057c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.185+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50578d30 0x55ec5057cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.185+0000 7fa24d460b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ec50396fa0 con 0x55ec50572db0 2024-05-27T07:36:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.185+0000 7fa24d460b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ec50431760 con 0x55ec50578d30 2024-05-27T07:36:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.185+0000 7fa24d460b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ec5057f800 con 0x55ec503f3e40 2024-05-27T07:36:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.185+0000 7fa24d460b80 10 client.0 _ll_register_callbacks cb 0x55ec5043ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.186+0000 7fa22f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec50572db0 0x55ec5057c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.186+0000 7fa22f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec50572db0 0x55ec5057c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43600/0 (socket says 172.21.15.43:43600) 2024-05-27T07:36:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.186+0000 7fa22f400700 1 -- 172.21.15.43:0/1069490785 learned_addr learned my addr 172.21.15.43:0/1069490785 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.187+0000 7fa22f400700 1 -- 172.21.15.43:0/1069490785 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec503f3e40 msgr2=0x55ec5057c4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:36:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.187+0000 7fa22f400700 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec503f3e40 0x55ec5057c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.187+0000 7fa22f400700 1 -- 172.21.15.43:0/1069490785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50578d30 msgr2=0x55ec5057cf20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:36:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.187+0000 7fa22f400700 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50578d30 0x55ec5057cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.187+0000 7fa22f400700 1 -- 172.21.15.43:0/1069490785 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec503e08d0 con 0x55ec50572db0 2024-05-27T07:36:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.188+0000 7fa22f400700 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec50572db0 0x55ec5057c9e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fa22000df00 tx=0x7fa2200116c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=176b28fd1af3a9ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.188+0000 7fa226000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:36:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.189+0000 7fa226000700 1 -- 172.21.15.43:0/1069490785 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220012450 con 0x55ec50572db0 2024-05-27T07:36:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.189+0000 7fa24d460b80 1 -- 172.21.15.43:0/1069490785 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa228026070 con 0x55ec50572db0 2024-05-27T07:36:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.189+0000 7fa226000700 1 -- 172.21.15.43:0/1069490785 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(154..154 src has 1..154) ==== 9584+0+0 (secure 0 0 0) 0x7fa2200125f0 con 0x55ec50572db0 2024-05-27T07:36:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.191+0000 7fa226000700 1 -- 172.21.15.43:0/1069490785 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7fa2200127f0 con 0x55ec50572db0 2024-05-27T07:36:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.191+0000 7fa24d460b80 10 client.18276 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.191+0000 7fa24d460b80 1 -- 172.21.15.43:0/1069490785 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55ec50461060 con 0x55ec50572db0 2024-05-27T07:36:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.193+0000 7fa226000700 1 -- 172.21.15.43:0/1069490785 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa228026070 con 0x55ec50572db0 2024-05-27T07:36:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.193+0000 7fa226000700 1 -- 172.21.15.43:0/1069490785 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7fa220025a90 con 0x55ec50572db0 2024-05-27T07:36:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.193+0000 7fa24d460b80 10 client.18276 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.193+0000 7fa24d460b80 10 client.18276 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:36:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.193+0000 7fa24d460b80 20 client.18276 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.193+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ec5058cbd0 0x55ec5058f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.193+0000 7fa24d460b80 4 client.18276 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:05.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.193+0000 7fa24d460b80 1 -- 172.21.15.43:0/1069490785 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec504a2d90 con 0x55ec5058cbd0 2024-05-27T07:36:05.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.194+0000 7fa237e00700 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ec5058cbd0 0x55ec5058f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.196+0000 7fa226000700 10 client.18276 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:36:05.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.196+0000 7fa237e00700 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ec5058cbd0 0x55ec5058f0c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6844a12e4252d47e server_cookie=3e765b3d1ee79ba6 in_seq=0 out_seq=0 2024-05-27T07:36:05.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.197+0000 7fa226000700 1 -- 172.21.15.43:0/1069490785 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ec504a2d90 con 0x55ec5058cbd0 2024-05-27T07:36:05.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.197+0000 7fa226000700 10 client.18276 handle_command_reply: tid=0 2024-05-27T07:36:05.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.197+0000 7fa24d460b80 1 client.18276 shutdown 2024-05-27T07:36:05.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.198+0000 7fa24d460b80 1 -- 172.21.15.43:0/1069490785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec50572db0 msgr2=0x55ec5057c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:05.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.198+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec50572db0 0x55ec5057c9e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fa22000df00 tx=0x7fa2200116c0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.199+0000 7fa24d460b80 1 -- 172.21.15.43:0/1069490785 shutdown_connections 2024-05-27T07:36:05.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.199+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ec5058cbd0 0x55ec5058f0c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.199+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50578d30 0x55ec5057cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.199+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec503f3e40 0x55ec5057c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.199+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1069490785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec50572db0 0x55ec5057c9e0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.199+0000 7fa24d460b80 1 -- 172.21.15.43:0/1069490785 >> 172.21.15.43:0/1069490785 conn(0x55ec50453fc0 msgr2=0x55ec5056d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:05.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.199+0000 7fa226000700 0 client.18276 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:36:05.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.199+0000 7fa24d460b80 1 -- 172.21.15.43:0/1069490785 shutdown_connections 2024-05-27T07:36:05.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.199+0000 7fa24d460b80 1 -- 172.21.15.43:0/1069490785 wait complete. 2024-05-27T07:36:05.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.200+0000 7fa24d460b80 20 client.18276 trim_cache size 0 max 16384 2024-05-27T07:36:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.224+0000 7fa24d460b80 1 Processor -- start 2024-05-27T07:36:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.225+0000 7fa24d460b80 1 -- start start 2024-05-27T07:36:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.225+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50572870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.225+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50572db0 0x55ec5058f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.225+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50578d30 0x55ec50588900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.225+0000 7fa24d460b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa21400f4c0 con 0x55ec503f3e40 2024-05-27T07:36:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.225+0000 7fa24d460b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa21400d920 con 0x55ec50578d30 2024-05-27T07:36:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.225+0000 7fa24d460b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa21400edf0 con 0x55ec50572db0 2024-05-27T07:36:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.226+0000 7fa237e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50572870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.226+0000 7fa237e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50572870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43616/0 (socket says 172.21.15.43:43616) 2024-05-27T07:36:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.226+0000 7fa237e00700 1 -- 172.21.15.43:0/1970805970 learned_addr learned my addr 172.21.15.43:0/1970805970 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.226+0000 7fa22f400700 1 --2- 172.21.15.43:0/1970805970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50572db0 0x55ec5058f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.226+0000 7fa23ca00700 1 --2- 172.21.15.43:0/1970805970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50578d30 0x55ec50588900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.227+0000 7fa237e00700 1 -- 172.21.15.43:0/1970805970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50572db0 msgr2=0x55ec5058f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.227+0000 7fa237e00700 1 --2- 172.21.15.43:0/1970805970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50572db0 0x55ec5058f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.227+0000 7fa237e00700 1 -- 172.21.15.43:0/1970805970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50578d30 msgr2=0x55ec50588900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.227+0000 7fa237e00700 1 --2- 172.21.15.43:0/1970805970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50578d30 0x55ec50588900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.227+0000 7fa237e00700 1 -- 172.21.15.43:0/1970805970 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec5058a140 con 0x55ec503f3e40 2024-05-27T07:36:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.228+0000 7fa237e00700 1 --2- 172.21.15.43:0/1970805970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50572870 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa228015d50 tx=0x7fa228007fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fedd1d56d205136 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.229+0000 7fa22cc00700 1 -- 172.21.15.43:0/1970805970 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22801c040 con 0x55ec503f3e40 2024-05-27T07:36:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.229+0000 7fa22cc00700 1 -- 172.21.15.43:0/1970805970 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa228026030 con 0x55ec503f3e40 2024-05-27T07:36:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.229+0000 7fa22cc00700 1 -- 172.21.15.43:0/1970805970 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2280218e0 con 0x55ec503f3e40 2024-05-27T07:36:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.231+0000 7fa24d460b80 1 -- 172.21.15.43:0/1970805970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 msgr2=0x55ec50572870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.231+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1970805970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50572870 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa228015d50 tx=0x7fa228007fd0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.232+0000 7fa24d460b80 1 -- 172.21.15.43:0/1970805970 shutdown_connections 2024-05-27T07:36:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.232+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1970805970 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50578d30 0x55ec50588900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.232+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1970805970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50572870 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.232+0000 7fa24d460b80 1 --2- 172.21.15.43:0/1970805970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50572db0 0x55ec5058f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.232+0000 7fa24d460b80 1 -- 172.21.15.43:0/1970805970 >> 172.21.15.43:0/1970805970 conn(0x55ec50453fc0 msgr2=0x55ec50576210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.232+0000 7fa24d460b80 1 -- 172.21.15.43:0/1970805970 shutdown_connections 2024-05-27T07:36:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.232+0000 7fa24d460b80 1 -- 172.21.15.43:0/1970805970 wait complete. 2024-05-27T07:36:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.233+0000 7fa24d460b80 1 -- start start 2024-05-27T07:36:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.233+0000 7fa24d460b80 1 Processor -- start 2024-05-27T07:36:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.234+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50584db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.234+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 0x55ec505852f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.234+0000 7fa24d460b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50578d30 0x55ec505879b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.234+0000 7fa24d460b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa21400e720 con 0x55ec503f3e40 2024-05-27T07:36:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.234+0000 7fa24d460b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa21400e050 con 0x55ec50572db0 2024-05-27T07:36:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.234+0000 7fa24d460b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55ec50588f00 con 0x55ec50578d30 2024-05-27T07:36:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.234+0000 7fa24d460b80 10 client.0 _ll_register_callbacks cb 0x55ec5043ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.235+0000 7fa237e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50584db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.235+0000 7fa237e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50584db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43618/0 (socket says 172.21.15.43:43618) 2024-05-27T07:36:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.235+0000 7fa237e00700 1 -- 172.21.15.43:0/4113949527 learned_addr learned my addr 172.21.15.43:0/4113949527 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.235+0000 7fa22f400700 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 0x55ec505852f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.236+0000 7fa23ca00700 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50578d30 0x55ec505879b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.236+0000 7fa237e00700 1 -- 172.21.15.43:0/4113949527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50578d30 msgr2=0x55ec505879b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.236+0000 7fa237e00700 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50578d30 0x55ec505879b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.236+0000 7fa237e00700 1 -- 172.21.15.43:0/4113949527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 msgr2=0x55ec505852f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.236+0000 7fa237e00700 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 0x55ec505852f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.236+0000 7fa237e00700 1 -- 172.21.15.43:0/4113949527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec505881a0 con 0x55ec503f3e40 2024-05-27T07:36:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.236+0000 7fa22f400700 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 0x55ec505852f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.236+0000 7fa237e00700 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50584db0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fa2280061e0 tx=0x7fa228004ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488b9c5d69605c8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.237+0000 7fa226000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:36:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.237+0000 7fa226000700 1 -- 172.21.15.43:0/4113949527 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22801c050 con 0x55ec503f3e40 2024-05-27T07:36:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.237+0000 7fa24d460b80 1 -- 172.21.15.43:0/4113949527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa228026030 con 0x55ec503f3e40 2024-05-27T07:36:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.238+0000 7fa226000700 1 -- 172.21.15.43:0/4113949527 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(154..154 src has 1..154) ==== 9584+0+0 (secure 0 0 0) 0x7fa228035740 con 0x55ec503f3e40 2024-05-27T07:36:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.238+0000 7fa226000700 1 -- 172.21.15.43:0/4113949527 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7fa228035980 con 0x55ec503f3e40 2024-05-27T07:36:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.238+0000 7fa24d460b80 10 client.18282 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.238+0000 7fa24d460b80 1 -- 172.21.15.43:0/4113949527 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55ec505f7230 con 0x55ec503f3e40 2024-05-27T07:36:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.239+0000 7fa226000700 1 -- 172.21.15.43:0/4113949527 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa228026030 con 0x55ec503f3e40 2024-05-27T07:36:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.240+0000 7fa226000700 1 -- 172.21.15.43:0/4113949527 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7fa228016490 con 0x55ec503f3e40 2024-05-27T07:36:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.241+0000 7fa24d460b80 10 client.18282 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.241+0000 7fa24d460b80 10 client.18282 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:36:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.241+0000 7fa24d460b80 20 client.18282 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.241+0000 7fa24d460b80 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ec505f7810 0x55ec505f9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.241+0000 7fa24d460b80 4 client.18282 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.241+0000 7fa24d460b80 1 -- 172.21.15.43:0/4113949527 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec5049c570 con 0x55ec505f7810 2024-05-27T07:36:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.242+0000 7fa22f400700 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ec505f7810 0x55ec505f9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:05.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.245+0000 7fa22f400700 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ec505f7810 0x55ec505f9c30 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e46cfc13cc829494 server_cookie=50813bf6670bd1ee in_seq=0 out_seq=0 2024-05-27T07:36:05.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.245+0000 7fa226000700 10 client.18282 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:36:05.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.247+0000 7fa226000700 1 -- 172.21.15.43:0/4113949527 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec5049c570 con 0x55ec505f7810 2024-05-27T07:36:05.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.247+0000 7fa226000700 10 client.18282 handle_command_reply: tid=0 2024-05-27T07:36:05.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.247+0000 7fa24d460b80 1 client.18282 shutdown 2024-05-27T07:36:05.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa24d460b80 1 -- 172.21.15.43:0/4113949527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 msgr2=0x55ec50584db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:05.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa24d460b80 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50584db0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fa2280061e0 tx=0x7fa228004ae0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa24d460b80 1 -- 172.21.15.43:0/4113949527 shutdown_connections 2024-05-27T07:36:05.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa24d460b80 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec50578d30 0x55ec505879b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa24d460b80 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ec503f3e40 0x55ec50584db0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa24d460b80 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55ec505f7810 0x55ec505f9c30 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa24d460b80 1 --2- 172.21.15.43:0/4113949527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec50572db0 0x55ec505852f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa24d460b80 1 -- 172.21.15.43:0/4113949527 >> 172.21.15.43:0/4113949527 conn(0x55ec50453fc0 msgr2=0x55ec50577de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:05.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa226000700 0 client.18282 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:36:05.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa24d460b80 1 -- 172.21.15.43:0/4113949527 shutdown_connections 2024-05-27T07:36:05.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.248+0000 7fa24d460b80 1 -- 172.21.15.43:0/4113949527 wait complete. 2024-05-27T07:36:05.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.249+0000 7fa24d460b80 20 client.18282 trim_cache size 0 max 16384 2024-05-27T07:36:05.246 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:36:05.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.250+0000 7fa23ea00700 1 -- 172.21.15.43:0/4226054810 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa21c0b2ec0 msgr2=0x7fa21c0b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:05.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.250+0000 7fa23ea00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa21c0b2ec0 0x7fa21c0b53d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa2280062a0 tx=0x7fa22800b040 comp rx=0 tx=0).stop 2024-05-27T07:36:05.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.250+0000 7fa23ea00700 1 -- 172.21.15.43:0/4226054810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa238108730 msgr2=0x7fa2381159b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:05.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.250+0000 7fa23ea00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa238108730 0x7fa2381159b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa2300098f0 tx=0x7fa23000b560 comp rx=0 tx=0).stop 2024-05-27T07:36:05.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.252+0000 7fa23ea00700 1 -- 172.21.15.43:0/4226054810 shutdown_connections 2024-05-27T07:36:05.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.252+0000 7fa23ea00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa21c0b2ec0 0x7fa21c0b53d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.252+0000 7fa23ea00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa238108730 0x7fa2381159b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.252+0000 7fa23ea00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2381034b0 0x7fa23810fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.252+0000 7fa23ea00700 1 --2- 172.21.15.43:0/4226054810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa238107de0 0x7fa2381101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:05.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.252+0000 7fa23ea00700 1 -- 172.21.15.43:0/4226054810 >> 172.21.15.43:0/4226054810 conn(0x7fa2380fedb0 msgr2=0x7fa2381011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:05.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.252+0000 7fa23ea00700 1 -- 172.21.15.43:0/4226054810 shutdown_connections 2024-05-27T07:36:05.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:05.253+0000 7fa23ea00700 1 -- 172.21.15.43:0/4226054810 wait complete. 2024-05-27T07:36:05.817 INFO:tasks.workunit.client.0.smithi043.stdout: 10 53349 5.86 MB/sec execute 366 sec latency 129.879 ms 2024-05-27T07:36:06.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:36:06.696 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:06.817 INFO:tasks.workunit.client.0.smithi043.stdout: 10 53434 5.84 MB/sec execute 367 sec latency 81.639 ms 2024-05-27T07:36:06.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:06 smithi132 ceph-mon[35283]: pgmap v745: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 174 op/s 2024-05-27T07:36:06.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:06 smithi043 ceph-mon[31531]: pgmap v745: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 174 op/s 2024-05-27T07:36:06.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:06 smithi179 ceph-mon[35888]: pgmap v745: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.5 MiB/s wr, 174 op/s 2024-05-27T07:36:07.817 INFO:tasks.workunit.client.0.smithi043.stdout: 10 53541 5.84 MB/sec execute 368 sec latency 85.092 ms 2024-05-27T07:36:08.817 INFO:tasks.workunit.client.0.smithi043.stdout: 10 53661 5.84 MB/sec execute 369 sec latency 100.207 ms 2024-05-27T07:36:08.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:08 smithi132 ceph-mon[35283]: pgmap v746: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 4.8 MiB/s wr, 211 op/s 2024-05-27T07:36:08.886 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:08 smithi043 ceph-mon[31531]: pgmap v746: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 4.8 MiB/s wr, 211 op/s 2024-05-27T07:36:08.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:08 smithi179 ceph-mon[35888]: pgmap v746: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 4.8 MiB/s wr, 211 op/s 2024-05-27T07:36:09.818 INFO:tasks.workunit.client.0.smithi043.stdout: 10 53745 5.84 MB/sec execute 370 sec latency 388.185 ms 2024-05-27T07:36:10.818 INFO:tasks.workunit.client.0.smithi043.stdout: 10 53845 5.83 MB/sec execute 371 sec latency 114.961 ms 2024-05-27T07:36:10.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:10 smithi132 ceph-mon[35283]: pgmap v747: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 4.8 MiB/s wr, 211 op/s 2024-05-27T07:36:10.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:10 smithi043 ceph-mon[31531]: pgmap v747: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 4.8 MiB/s wr, 211 op/s 2024-05-27T07:36:10.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:10 smithi179 ceph-mon[35888]: pgmap v747: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 4.8 MiB/s wr, 211 op/s 2024-05-27T07:36:11.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.125+0000 7f868ca00700 1 -- 172.21.15.43:0/1252660081 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c016610 con 0x7f8690108f30 2024-05-27T07:36:11.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.126+0000 7f8695200700 1 -- 172.21.15.43:0/1252660081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8690108f30 msgr2=0x7f8690109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:11.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.126+0000 7f8695200700 1 --2- 172.21.15.43:0/1252660081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8690108f30 0x7f8690109340 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f867c00fb90 tx=0x7f867c00fea0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.128+0000 7f8695200700 1 -- 172.21.15.43:0/1252660081 shutdown_connections 2024-05-27T07:36:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.128+0000 7f8695200700 1 --2- 172.21.15.43:0/1252660081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8690109880 0x7f86901111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.128+0000 7f8695200700 1 --2- 172.21.15.43:0/1252660081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8690104600 0x7f8690104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.128+0000 7f8695200700 1 --2- 172.21.15.43:0/1252660081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8690108f30 0x7f8690109340 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.128+0000 7f8695200700 1 -- 172.21.15.43:0/1252660081 >> 172.21.15.43:0/1252660081 conn(0x7f869007af60 msgr2=0x7f869007b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.128+0000 7f8695200700 1 -- 172.21.15.43:0/1252660081 shutdown_connections 2024-05-27T07:36:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.128+0000 7f8695200700 1 -- 172.21.15.43:0/1252660081 wait complete. 2024-05-27T07:36:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.130+0000 7f8695200700 1 Processor -- start 2024-05-27T07:36:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.131+0000 7f8695200700 1 -- start start 2024-05-27T07:36:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.131+0000 7f8695200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8690104600 0x7f86901a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.131+0000 7f8695200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8690108f30 0x7f86901a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.131+0000 7f8695200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8690109880 0x7f86901a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.131+0000 7f8695200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f869011aa90 con 0x7f8690108f30 2024-05-27T07:36:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.131+0000 7f8695200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f869011a910 con 0x7f8690109880 2024-05-27T07:36:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.131+0000 7f8695200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f869011ac10 con 0x7f8690104600 2024-05-27T07:36:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.132+0000 7f868e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8690109880 0x7f86901a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.132+0000 7f868e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8690109880 0x7f86901a5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:54496/0 (socket says 172.21.15.43:54496) 2024-05-27T07:36:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.132+0000 7f868e800700 1 -- 172.21.15.43:0/2801206714 learned_addr learned my addr 172.21.15.43:0/2801206714 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.133+0000 7f868de00700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8690104600 0x7f86901a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.133+0000 7f868e800700 1 -- 172.21.15.43:0/2801206714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8690104600 msgr2=0x7f86901a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.133+0000 7f868e800700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8690104600 0x7f86901a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.133+0000 7f868e800700 1 -- 172.21.15.43:0/2801206714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8690108f30 msgr2=0x7f86901a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.133+0000 7f868e800700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8690108f30 0x7f86901a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.133+0000 7f868e800700 1 -- 172.21.15.43:0/2801206714 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86901a6310 con 0x7f8690109880 2024-05-27T07:36:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.134+0000 7f868e800700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8690109880 0x7f86901a5c70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f868800c370 tx=0x7f868800c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.135+0000 7f8686a00700 1 -- 172.21.15.43:0/2801206714 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86880090d0 con 0x7f8690109880 2024-05-27T07:36:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.135+0000 7f8686a00700 1 -- 172.21.15.43:0/2801206714 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8688009270 con 0x7f8690109880 2024-05-27T07:36:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.135+0000 7f8686a00700 1 -- 172.21.15.43:0/2801206714 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688004790 con 0x7f8690109880 2024-05-27T07:36:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.136+0000 7f8695200700 1 -- 172.21.15.43:0/2801206714 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86901a65a0 con 0x7f8690109880 2024-05-27T07:36:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.136+0000 7f8695200700 1 -- 172.21.15.43:0/2801206714 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86901a6a90 con 0x7f8690109880 2024-05-27T07:36:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.142+0000 7f8686a00700 1 -- 172.21.15.43:0/2801206714 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f868801a020 con 0x7f8690109880 2024-05-27T07:36:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.145+0000 7f8686a00700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f86740b2b40 0x7f86740b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.145+0000 7f8686a00700 1 -- 172.21.15.43:0/2801206714 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 9584+0+0 (secure 0 0 0) 0x7f86880e2310 con 0x7f8690109880 2024-05-27T07:36:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.145+0000 7f868de00700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f86740b2b40 0x7f86740b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.147+0000 7f868de00700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f86740b2b40 0x7f86740b5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8680005fd0 tx=0x7f8680009310 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:11.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.162+0000 7f86a3dafb80 1 Processor -- start 2024-05-27T07:36:11.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.162+0000 7f86a3dafb80 1 -- start start 2024-05-27T07:36:11.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.163+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5558269c3e40 0x555826b487f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.163+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b42870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.163+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b42db0 0x555826b431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.163+0000 7f86a3dafb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555826a70e40 con 0x555826b42db0 2024-05-27T07:36:11.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.163+0000 7f86a3dafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555826a609e0 con 0x555826b48d30 2024-05-27T07:36:11.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.163+0000 7f86a3dafb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555826a64a20 con 0x5558269c3e40 2024-05-27T07:36:11.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.164+0000 7f868d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b42870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.164+0000 7f868d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b42870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:54498/0 (socket says 172.21.15.43:54498) 2024-05-27T07:36:11.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.164+0000 7f868d400700 1 -- 172.21.15.43:0/3259184927 learned_addr learned my addr 172.21.15.43:0/3259184927 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:11.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.164+0000 7f868de00700 1 --2- 172.21.15.43:0/3259184927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5558269c3e40 0x555826b487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.164+0000 7f868e800700 1 --2- 172.21.15.43:0/3259184927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b42db0 0x555826b431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.164+0000 7f868d400700 1 -- 172.21.15.43:0/3259184927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5558269c3e40 msgr2=0x555826b487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:11.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.164+0000 7f868d400700 1 --2- 172.21.15.43:0/3259184927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5558269c3e40 0x555826b487f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.164+0000 7f868d400700 1 -- 172.21.15.43:0/3259184927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b42db0 msgr2=0x555826b431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:11.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.164+0000 7f868d400700 1 --2- 172.21.15.43:0/3259184927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b42db0 0x555826b431c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.164+0000 7f868d400700 1 -- 172.21.15.43:0/3259184927 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555826a29420 con 0x555826b48d30 2024-05-27T07:36:11.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.165+0000 7f868d400700 1 --2- 172.21.15.43:0/3259184927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b42870 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f867c0059c0 tx=0x7f867c0099f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c807b82145777a35 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:11.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.165+0000 7f867b400700 1 -- 172.21.15.43:0/3259184927 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c01c070 con 0x555826b48d30 2024-05-27T07:36:11.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.165+0000 7f867b400700 1 -- 172.21.15.43:0/3259184927 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f867c0054d0 con 0x555826b48d30 2024-05-27T07:36:11.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.165+0000 7f867b400700 1 -- 172.21.15.43:0/3259184927 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c00f540 con 0x555826b48d30 2024-05-27T07:36:11.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.167+0000 7f86a3dafb80 1 -- 172.21.15.43:0/3259184927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 msgr2=0x555826b42870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:11.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.167+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/3259184927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b42870 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f867c0059c0 tx=0x7f867c0099f0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.167+0000 7f86a3dafb80 1 -- 172.21.15.43:0/3259184927 shutdown_connections 2024-05-27T07:36:11.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.167+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/3259184927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b42db0 0x555826b431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.167+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/3259184927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5558269c3e40 0x555826b487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.167+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/3259184927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b42870 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.167+0000 7f86a3dafb80 1 -- 172.21.15.43:0/3259184927 >> 172.21.15.43:0/3259184927 conn(0x555826a23fc0 msgr2=0x555826b3d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:11.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.167+0000 7f86a3dafb80 1 -- 172.21.15.43:0/3259184927 shutdown_connections 2024-05-27T07:36:11.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.167+0000 7f86a3dafb80 1 -- 172.21.15.43:0/3259184927 wait complete. 2024-05-27T07:36:11.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.168+0000 7f86a3dafb80 1 -- start start 2024-05-27T07:36:11.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.168+0000 7f86a3dafb80 1 Processor -- start 2024-05-27T07:36:11.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.169+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5558269c3e40 0x555826b4c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.169+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b42db0 0x555826b4ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.169+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b48d30 0x555826b4cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.169+0000 7f86a3dafb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555826966fa0 con 0x555826b48d30 2024-05-27T07:36:11.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.169+0000 7f86a3dafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555826a01760 con 0x555826b42db0 2024-05-27T07:36:11.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.169+0000 7f86a3dafb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555826b4f7a0 con 0x5558269c3e40 2024-05-27T07:36:11.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.169+0000 7f86a3dafb80 10 client.0 _ll_register_callbacks cb 0x555826a0ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:11.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.170+0000 7f868de00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5558269c3e40 0x555826b4c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.170+0000 7f868d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b42db0 0x555826b4ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.170+0000 7f868d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b42db0 0x555826b4ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:54512/0 (socket says 172.21.15.43:54512) 2024-05-27T07:36:11.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.170+0000 7f868d400700 1 -- 172.21.15.43:0/4107562418 learned_addr learned my addr 172.21.15.43:0/4107562418 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:11.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.170+0000 7f868e800700 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b48d30 0x555826b4cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.171+0000 7f868e800700 1 -- 172.21.15.43:0/4107562418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5558269c3e40 msgr2=0x555826b4c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:11.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.171+0000 7f868e800700 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5558269c3e40 0x555826b4c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.171+0000 7f868e800700 1 -- 172.21.15.43:0/4107562418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b42db0 msgr2=0x555826b4ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:11.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.171+0000 7f868e800700 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b42db0 0x555826b4ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.171+0000 7f868e800700 1 -- 172.21.15.43:0/4107562418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558269b08d0 con 0x555826b48d30 2024-05-27T07:36:11.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.171+0000 7f868e800700 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b48d30 0x555826b4cfa0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f86880df640 tx=0x7f86880a8b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe2671132303254 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.171+0000 7f866fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:36:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.172+0000 7f866fe00700 1 -- 172.21.15.43:0/4107562418 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86880c2070 con 0x555826b48d30 2024-05-27T07:36:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.172+0000 7f866fe00700 1 -- 172.21.15.43:0/4107562418 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(154..154 src has 1..154) ==== 9584+0+0 (secure 0 0 0) 0x7f86880ad170 con 0x555826b48d30 2024-05-27T07:36:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.172+0000 7f86a3dafb80 1 -- 172.21.15.43:0/4107562418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f867c0054d0 con 0x555826b48d30 2024-05-27T07:36:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.173+0000 7f866fe00700 1 -- 172.21.15.43:0/4107562418 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f86880ad370 con 0x555826b48d30 2024-05-27T07:36:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.173+0000 7f86a3dafb80 10 client.18291 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.173+0000 7f86a3dafb80 1 -- 172.21.15.43:0/4107562418 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x555826a31060 con 0x555826b48d30 2024-05-27T07:36:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.173+0000 7f866fe00700 1 -- 172.21.15.43:0/4107562418 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f867c0054d0 con 0x555826b48d30 2024-05-27T07:36:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.173+0000 7f866fe00700 1 -- 172.21.15.43:0/4107562418 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f86880b1ba0 con 0x555826b48d30 2024-05-27T07:36:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.174+0000 7f86a3dafb80 10 client.18291 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.174+0000 7f86a3dafb80 10 client.18291 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:36:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.174+0000 7f86a3dafb80 20 client.18291 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.174+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555826b5c9d0 0x555826b5eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.174+0000 7f86a3dafb80 4 client.18291 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:11.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.174+0000 7f86a3dafb80 1 -- 172.21.15.43:0/4107562418 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555826a72d90 con 0x555826b5c9d0 2024-05-27T07:36:11.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.174+0000 7f868de00700 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555826b5c9d0 0x555826b5eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.175+0000 7f866fe00700 10 client.18291 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:36:11.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.175+0000 7f868de00700 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555826b5c9d0 0x555826b5eea0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7364a8dddd83e73 server_cookie=ded59f173a2c785b in_seq=0 out_seq=0 2024-05-27T07:36:11.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.176+0000 7f866fe00700 1 -- 172.21.15.43:0/4107562418 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x555826a72d90 con 0x555826b5c9d0 2024-05-27T07:36:11.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.176+0000 7f866fe00700 10 client.18291 handle_command_reply: tid=0 2024-05-27T07:36:11.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.177+0000 7f86a3dafb80 1 client.18291 shutdown 2024-05-27T07:36:11.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.177+0000 7f86a3dafb80 1 -- 172.21.15.43:0/4107562418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b48d30 msgr2=0x555826b4cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:11.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.177+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b48d30 0x555826b4cfa0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f86880df640 tx=0x7f86880a8b20 comp rx=0 tx=0).stop 2024-05-27T07:36:11.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.178+0000 7f86a3dafb80 1 -- 172.21.15.43:0/4107562418 shutdown_connections 2024-05-27T07:36:11.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.178+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555826b5c9d0 0x555826b5eea0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.178+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b48d30 0x555826b4cfa0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.178+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5558269c3e40 0x555826b4c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.178+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/4107562418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b42db0 0x555826b4ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.178+0000 7f866fe00700 0 client.18291 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:36:11.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.178+0000 7f86a3dafb80 1 -- 172.21.15.43:0/4107562418 >> 172.21.15.43:0/4107562418 conn(0x555826a23fc0 msgr2=0x555826b3d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:11.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.179+0000 7f86a3dafb80 1 -- 172.21.15.43:0/4107562418 shutdown_connections 2024-05-27T07:36:11.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.179+0000 7f86a3dafb80 1 -- 172.21.15.43:0/4107562418 wait complete. 2024-05-27T07:36:11.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.179+0000 7f86a3dafb80 20 client.18291 trim_cache size 0 max 16384 2024-05-27T07:36:11.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.206+0000 7f86a3dafb80 1 Processor -- start 2024-05-27T07:36:11.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.206+0000 7f86a3dafb80 1 -- start start 2024-05-27T07:36:11.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.206+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5558269c3e40 0x555826b42870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.206+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b42db0 0x555826b46c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.207+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b47ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.207+0000 7f86a3dafb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86901b7120 con 0x5558269c3e40 2024-05-27T07:36:11.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.207+0000 7f86a3dafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f86901b5580 con 0x555826b48d30 2024-05-27T07:36:11.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.207+0000 7f86a3dafb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f86901b6a50 con 0x555826b42db0 2024-05-27T07:36:11.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.207+0000 7f868e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b47ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.207+0000 7f868e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b47ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:54526/0 (socket says 172.21.15.43:54526) 2024-05-27T07:36:11.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.207+0000 7f868e800700 1 -- 172.21.15.43:0/476388947 learned_addr learned my addr 172.21.15.43:0/476388947 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:11.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.207+0000 7f868de00700 1 --2- 172.21.15.43:0/476388947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5558269c3e40 0x555826b42870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.207+0000 7f868d400700 1 --2- 172.21.15.43:0/476388947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b42db0 0x555826b46c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.208+0000 7f868e800700 1 -- 172.21.15.43:0/476388947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b42db0 msgr2=0x555826b46c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:11.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.208+0000 7f868e800700 1 --2- 172.21.15.43:0/476388947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b42db0 0x555826b46c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.208+0000 7f868e800700 1 -- 172.21.15.43:0/476388947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5558269c3e40 msgr2=0x555826b42870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:11.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.208+0000 7f868e800700 1 --2- 172.21.15.43:0/476388947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5558269c3e40 0x555826b42870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.208+0000 7f868e800700 1 -- 172.21.15.43:0/476388947 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555826b43740 con 0x555826b48d30 2024-05-27T07:36:11.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.208+0000 7f868de00700 1 --2- 172.21.15.43:0/476388947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5558269c3e40 0x555826b42870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:11.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.208+0000 7f868e800700 1 --2- 172.21.15.43:0/476388947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b47ca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f86880a8ed0 tx=0x7f86880e2050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=218df11055638457 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:11.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.209+0000 7f867aa00700 1 -- 172.21.15.43:0/476388947 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86880b9070 con 0x555826b48d30 2024-05-27T07:36:11.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.209+0000 7f867aa00700 1 -- 172.21.15.43:0/476388947 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f86880a8ce0 con 0x555826b48d30 2024-05-27T07:36:11.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.209+0000 7f867aa00700 1 -- 172.21.15.43:0/476388947 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86880a91d0 con 0x555826b48d30 2024-05-27T07:36:11.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.210+0000 7f86a3dafb80 1 -- 172.21.15.43:0/476388947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 msgr2=0x555826b47ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:11.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.210+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/476388947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b47ca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f86880a8ed0 tx=0x7f86880e2050 comp rx=0 tx=0).stop 2024-05-27T07:36:11.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.211+0000 7f86a3dafb80 1 -- 172.21.15.43:0/476388947 shutdown_connections 2024-05-27T07:36:11.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.211+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/476388947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555826b48d30 0x555826b47ca0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.211+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/476388947 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5558269c3e40 0x555826b42870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.211+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/476388947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b42db0 0x555826b46c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.211+0000 7f86a3dafb80 1 -- 172.21.15.43:0/476388947 >> 172.21.15.43:0/476388947 conn(0x555826a23fc0 msgr2=0x555826b4c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:11.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.211+0000 7f86a3dafb80 1 -- 172.21.15.43:0/476388947 shutdown_connections 2024-05-27T07:36:11.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.211+0000 7f86a3dafb80 1 -- 172.21.15.43:0/476388947 wait complete. 2024-05-27T07:36:11.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.212+0000 7f86a3dafb80 1 -- start start 2024-05-27T07:36:11.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.212+0000 7f86a3dafb80 1 Processor -- start 2024-05-27T07:36:11.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.213+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5558269c3e40 0x555826b551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.213+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b42db0 0x555826b63ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.213+0000 7f86a3dafb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b48d30 0x555826b64410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.213+0000 7f868de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5558269c3e40 0x555826b551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.213+0000 7f868de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5558269c3e40 0x555826b551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:54540/0 (socket says 172.21.15.43:54540) 2024-05-27T07:36:11.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.213+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86901b6380 con 0x555826b42db0 2024-05-27T07:36:11.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.214+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f86901b5cb0 con 0x5558269c3e40 2024-05-27T07:36:11.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.214+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x555826a72d90 con 0x555826b48d30 2024-05-27T07:36:11.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.213+0000 7f868de00700 1 -- 172.21.15.43:0/450140705 learned_addr learned my addr 172.21.15.43:0/450140705 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:11.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.214+0000 7f86a3dafb80 10 client.0 _ll_register_callbacks cb 0x555826a0ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:11.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.214+0000 7f868e800700 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b48d30 0x555826b64410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.214+0000 7f868de00700 1 -- 172.21.15.43:0/450140705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b48d30 msgr2=0x555826b64410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:11.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.214+0000 7f868de00700 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b48d30 0x555826b64410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.214+0000 7f868de00700 1 -- 172.21.15.43:0/450140705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b42db0 msgr2=0x555826b63ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:11.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.214+0000 7f868de00700 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b42db0 0x555826b63ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.214+0000 7f868de00700 1 -- 172.21.15.43:0/450140705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555826b64b30 con 0x5558269c3e40 2024-05-27T07:36:11.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.214+0000 7f868e800700 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b48d30 0x555826b64410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:11.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.215+0000 7f868de00700 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5558269c3e40 0x555826b551e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x555826b43b30 tx=0x7f86800047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50345bd0fa1fd648 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:11.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.215+0000 7f866fe00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:36:11.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.215+0000 7f866fe00700 1 -- 172.21.15.43:0/450140705 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868002b580 con 0x5558269c3e40 2024-05-27T07:36:11.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.215+0000 7f866fe00700 1 -- 172.21.15.43:0/450140705 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(154..154 src has 1..154) ==== 9584+0+0 (secure 0 0 0) 0x7f868002b720 con 0x5558269c3e40 2024-05-27T07:36:11.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.215+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86880a8ce0 con 0x5558269c3e40 2024-05-27T07:36:11.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.216+0000 7f866fe00700 1 -- 172.21.15.43:0/450140705 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f868002b920 con 0x5558269c3e40 2024-05-27T07:36:11.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.216+0000 7f86a3dafb80 10 client.37676 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:11.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.216+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x555826bc74a0 con 0x5558269c3e40 2024-05-27T07:36:11.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.217+0000 7f866fe00700 1 -- 172.21.15.43:0/450140705 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f86880a8ce0 con 0x5558269c3e40 2024-05-27T07:36:11.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.217+0000 7f866fe00700 1 -- 172.21.15.43:0/450140705 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f8680033400 con 0x5558269c3e40 2024-05-27T07:36:11.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.217+0000 7f86a3dafb80 10 client.37676 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:11.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.217+0000 7f86a3dafb80 10 client.37676 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:36:11.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.217+0000 7f86a3dafb80 20 client.37676 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:11.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.217+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555826bc7b90 0x555826bc9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:11.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.217+0000 7f86a3dafb80 4 client.37676 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:11.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.217+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555826a6c570 con 0x555826bc7b90 2024-05-27T07:36:11.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.218+0000 7f868d400700 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555826bc7b90 0x555826bc9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:11.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.218+0000 7f868d400700 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555826bc7b90 0x555826bc9fb0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=42eb28d92de224f7 server_cookie=7fba7ecddacddccf in_seq=0 out_seq=0 2024-05-27T07:36:11.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.218+0000 7f866fe00700 10 client.37676 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:36:11.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.219+0000 7f866fe00700 1 -- 172.21.15.43:0/450140705 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555826a6c570 con 0x555826bc7b90 2024-05-27T07:36:11.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.219+0000 7f866fe00700 10 client.37676 handle_command_reply: tid=0 2024-05-27T07:36:11.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.219+0000 7f86a3dafb80 1 client.37676 shutdown 2024-05-27T07:36:11.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.219+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5558269c3e40 msgr2=0x555826b551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:11.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.219+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5558269c3e40 0x555826b551e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x555826b43b30 tx=0x7f86800047c0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.220+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 shutdown_connections 2024-05-27T07:36:11.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.220+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x555826bc7b90 0x555826bc9fb0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.220+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555826b48d30 0x555826b64410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.220+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5558269c3e40 0x555826b551e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.220+0000 7f86a3dafb80 1 --2- 172.21.15.43:0/450140705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555826b42db0 0x555826b63ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.220+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 >> 172.21.15.43:0/450140705 conn(0x555826a23fc0 msgr2=0x555826b47ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:11.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.220+0000 7f866fe00700 0 client.37676 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:36:11.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.221+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 shutdown_connections 2024-05-27T07:36:11.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.221+0000 7f86a3dafb80 1 -- 172.21.15.43:0/450140705 wait complete. 2024-05-27T07:36:11.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.221+0000 7f86a3dafb80 20 client.37676 trim_cache size 0 max 16384 2024-05-27T07:36:11.218 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:36:11.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.222+0000 7f8695200700 1 -- 172.21.15.43:0/2801206714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f86740b2b40 msgr2=0x7f86740b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:11.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.222+0000 7f8695200700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f86740b2b40 0x7f86740b5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8680005fd0 tx=0x7f8680009310 comp rx=0 tx=0).stop 2024-05-27T07:36:11.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.222+0000 7f8695200700 1 -- 172.21.15.43:0/2801206714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8690109880 msgr2=0x7f86901a5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:11.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.222+0000 7f8695200700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8690109880 0x7f86901a5c70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f868800c370 tx=0x7f868800c730 comp rx=0 tx=0).stop 2024-05-27T07:36:11.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.223+0000 7f8695200700 1 -- 172.21.15.43:0/2801206714 shutdown_connections 2024-05-27T07:36:11.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.223+0000 7f8695200700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f86740b2b40 0x7f86740b5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.223+0000 7f8695200700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8690109880 0x7f86901a5c70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.223+0000 7f8695200700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8690104600 0x7f86901a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.223+0000 7f8695200700 1 --2- 172.21.15.43:0/2801206714 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8690108f30 0x7f86901a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:11.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.223+0000 7f8695200700 1 -- 172.21.15.43:0/2801206714 >> 172.21.15.43:0/2801206714 conn(0x7f869007af60 msgr2=0x7f8690102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:11.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.223+0000 7f8695200700 1 -- 172.21.15.43:0/2801206714 shutdown_connections 2024-05-27T07:36:11.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:11.223+0000 7f8695200700 1 -- 172.21.15.43:0/2801206714 wait complete. 2024-05-27T07:36:11.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 53930 5.82 MB/sec execute 372 sec latency 83.821 ms 2024-05-27T07:36:12.461 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:36:12.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:12 smithi043 ceph-mon[31531]: pgmap v748: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-05-27T07:36:12.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:12 smithi043 ceph-mon[31531]: osdmap e155: 12 total, 12 up, 12 in 2024-05-27T07:36:12.818 INFO:tasks.workunit.client.0.smithi043.stdout: 10 54035 5.82 MB/sec execute 373 sec latency 113.115 ms 2024-05-27T07:36:12.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:12 smithi132 ceph-mon[35283]: pgmap v748: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-05-27T07:36:12.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:12 smithi132 ceph-mon[35283]: osdmap e155: 12 total, 12 up, 12 in 2024-05-27T07:36:12.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:12 smithi179 ceph-mon[35888]: pgmap v748: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-05-27T07:36:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:12 smithi179 ceph-mon[35888]: osdmap e155: 12 total, 12 up, 12 in 2024-05-27T07:36:13.154 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:13.818 INFO:tasks.workunit.client.0.smithi043.stdout: 10 54144 5.81 MB/sec execute 374 sec latency 101.629 ms 2024-05-27T07:36:14.818 INFO:tasks.workunit.client.0.smithi043.stdout: 10 54267 5.83 MB/sec execute 375 sec latency 126.074 ms 2024-05-27T07:36:14.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:14 smithi132 ceph-mon[35283]: pgmap v750: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-05-27T07:36:14.867 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:14 smithi043 ceph-mon[31531]: pgmap v750: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-05-27T07:36:14.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:14 smithi179 ceph-mon[35888]: pgmap v750: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-05-27T07:36:15.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 54416 5.84 MB/sec execute 376 sec latency 159.387 ms 2024-05-27T07:36:16.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 54527 5.83 MB/sec execute 377 sec latency 140.396 ms 2024-05-27T07:36:16.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:16 smithi132 ceph-mon[35283]: pgmap v751: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-05-27T07:36:16.877 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:16 smithi043 ceph-mon[31531]: pgmap v751: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-05-27T07:36:16.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:16 smithi179 ceph-mon[35888]: pgmap v751: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-05-27T07:36:17.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 54643 5.83 MB/sec execute 378 sec latency 92.707 ms 2024-05-27T07:36:18.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.425+0000 7f99c3800700 1 -- 172.21.15.43:0/2687906256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99bc109840 msgr2=0x7f99bc1111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:18.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.425+0000 7f99c3800700 1 --2- 172.21.15.43:0/2687906256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99bc109840 0x7f99bc1111a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f99b400b0a0 tx=0x7f99b400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.425+0000 7f99c3800700 1 -- 172.21.15.43:0/2687906256 shutdown_connections 2024-05-27T07:36:18.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.425+0000 7f99c3800700 1 --2- 172.21.15.43:0/2687906256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99bc109840 0x7f99bc1111a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.425+0000 7f99c3800700 1 --2- 172.21.15.43:0/2687906256 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99bc1045c0 0x7f99bc1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.425+0000 7f99c3800700 1 --2- 172.21.15.43:0/2687906256 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99bc108ef0 0x7f99bc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.425+0000 7f99c3800700 1 -- 172.21.15.43:0/2687906256 >> 172.21.15.43:0/2687906256 conn(0x7f99bc0ffe90 msgr2=0x7f99bc1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:18.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.425+0000 7f99c3800700 1 -- 172.21.15.43:0/2687906256 shutdown_connections 2024-05-27T07:36:18.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.425+0000 7f99c3800700 1 -- 172.21.15.43:0/2687906256 wait complete. 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.427+0000 7f99c3800700 1 Processor -- start 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.427+0000 7f99c3800700 1 -- start start 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.427+0000 7f99c3800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99bc1045c0 0x7f99bc1a2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.427+0000 7f99c3800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99bc108ef0 0x7f99bc1a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.427+0000 7f99c3800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99bc109840 0x7f99bc1ad0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.427+0000 7f99c3800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc114390 con 0x7f99bc1045c0 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.427+0000 7f99c3800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc114210 con 0x7f99bc108ef0 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.427+0000 7f99c3800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc114510 con 0x7f99bc109840 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.428+0000 7f99c0e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99bc1045c0 0x7f99bc1a2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.428+0000 7f99bbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99bc108ef0 0x7f99bc1a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.428+0000 7f99c0e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99bc1045c0 0x7f99bc1a2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44118/0 (socket says 172.21.15.43:44118) 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.428+0000 7f99c0e00700 1 -- 172.21.15.43:0/2707645384 learned_addr learned my addr 172.21.15.43:0/2707645384 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.428+0000 7f99c1800700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99bc109840 0x7f99bc1ad0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.429+0000 7f99bbe00700 1 -- 172.21.15.43:0/2707645384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99bc109840 msgr2=0x7f99bc1ad0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.429+0000 7f99bbe00700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99bc109840 0x7f99bc1ad0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.429+0000 7f99bbe00700 1 -- 172.21.15.43:0/2707645384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99bc1045c0 msgr2=0x7f99bc1a2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.429+0000 7f99bbe00700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99bc1045c0 0x7f99bc1a2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.429+0000 7f99bbe00700 1 -- 172.21.15.43:0/2707645384 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99bc1ad630 con 0x7f99bc108ef0 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.429+0000 7f99c0e00700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99bc1045c0 0x7f99bc1a2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.429+0000 7f99bbe00700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99bc108ef0 0x7f99bc1a25b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f99a800e9c0 tx=0x7f99a800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.429+0000 7f99b9600700 1 -- 172.21.15.43:0/2707645384 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8004220 con 0x7f99bc108ef0 2024-05-27T07:36:18.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.429+0000 7f99c3800700 1 -- 172.21.15.43:0/2707645384 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99bc1ad860 con 0x7f99bc108ef0 2024-05-27T07:36:18.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.429+0000 7f99b9600700 1 -- 172.21.15.43:0/2707645384 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99a80043c0 con 0x7f99bc108ef0 2024-05-27T07:36:18.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.430+0000 7f99b9600700 1 -- 172.21.15.43:0/2707645384 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a800ce60 con 0x7f99bc108ef0 2024-05-27T07:36:18.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.430+0000 7f99c3800700 1 -- 172.21.15.43:0/2707645384 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99bc1add70 con 0x7f99bc108ef0 2024-05-27T07:36:18.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.434+0000 7f99b9600700 1 -- 172.21.15.43:0/2707645384 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f99a8004750 con 0x7f99bc108ef0 2024-05-27T07:36:18.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.435+0000 7f99b9600700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f99a40b2a20 0x7f99a40b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.436+0000 7f99b9600700 1 -- 172.21.15.43:0/2707645384 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7f99a80e2970 con 0x7f99bc108ef0 2024-05-27T07:36:18.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.436+0000 7f99c0e00700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f99a40b2a20 0x7f99a40b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.437+0000 7f99c0e00700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f99a40b2a20 0x7f99a40b4f30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f99b0005950 tx=0x7f99b000b500 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:18.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.466+0000 7f99d2321b80 1 Processor -- start 2024-05-27T07:36:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.466+0000 7f99d2321b80 1 -- start start 2024-05-27T07:36:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.467+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41cb3e40 0x556a41e387f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.467+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e38d30 0x556a41e32870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.467+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 0x556a41e331c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.467+0000 7f99d2321b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556a41d60e40 con 0x556a41e32db0 2024-05-27T07:36:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.467+0000 7f99d2321b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556a41d509e0 con 0x556a41cb3e40 2024-05-27T07:36:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.467+0000 7f99d2321b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556a41d54a20 con 0x556a41e38d30 2024-05-27T07:36:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.468+0000 7f99c0e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41cb3e40 0x556a41e387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.468+0000 7f99c0e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41cb3e40 0x556a41e387f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:43852/0 (socket says 172.21.15.43:43852) 2024-05-27T07:36:18.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.468+0000 7f99c0e00700 1 -- 172.21.15.43:0/4046281954 learned_addr learned my addr 172.21.15.43:0/4046281954 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:18.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.468+0000 7f99c1800700 1 --2- 172.21.15.43:0/4046281954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 0x556a41e331c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.469+0000 7f99bbe00700 1 --2- 172.21.15.43:0/4046281954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e38d30 0x556a41e32870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.469+0000 7f99c1800700 1 -- 172.21.15.43:0/4046281954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e38d30 msgr2=0x556a41e32870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.469+0000 7f99c1800700 1 --2- 172.21.15.43:0/4046281954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e38d30 0x556a41e32870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.469+0000 7f99c1800700 1 -- 172.21.15.43:0/4046281954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41cb3e40 msgr2=0x556a41e387f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.469+0000 7f99c1800700 1 --2- 172.21.15.43:0/4046281954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41cb3e40 0x556a41e387f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.469+0000 7f99c1800700 1 -- 172.21.15.43:0/4046281954 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a41d19420 con 0x556a41e32db0 2024-05-27T07:36:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.469+0000 7f99bbe00700 1 --2- 172.21.15.43:0/4046281954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e38d30 0x556a41e32870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.470+0000 7f99c1800700 1 --2- 172.21.15.43:0/4046281954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 0x556a41e331c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009fd0 tx=0x7f99b4009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd8253d1b8c5e44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.470+0000 7f99ae000700 1 -- 172.21.15.43:0/4046281954 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b400bcb0 con 0x556a41e32db0 2024-05-27T07:36:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.470+0000 7f99ae000700 1 -- 172.21.15.43:0/4046281954 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99b400be50 con 0x556a41e32db0 2024-05-27T07:36:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.471+0000 7f99ae000700 1 -- 172.21.15.43:0/4046281954 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4007cd0 con 0x556a41e32db0 2024-05-27T07:36:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.472+0000 7f99d2321b80 1 -- 172.21.15.43:0/4046281954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 msgr2=0x556a41e331c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.472+0000 7f99d2321b80 1 --2- 172.21.15.43:0/4046281954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 0x556a41e331c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009fd0 tx=0x7f99b4009390 comp rx=0 tx=0).stop 2024-05-27T07:36:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.472+0000 7f99d2321b80 1 -- 172.21.15.43:0/4046281954 shutdown_connections 2024-05-27T07:36:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.472+0000 7f99d2321b80 1 --2- 172.21.15.43:0/4046281954 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 0x556a41e331c0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.472+0000 7f99d2321b80 1 --2- 172.21.15.43:0/4046281954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41cb3e40 0x556a41e387f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.472+0000 7f99d2321b80 1 --2- 172.21.15.43:0/4046281954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e38d30 0x556a41e32870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.472+0000 7f99d2321b80 1 -- 172.21.15.43:0/4046281954 >> 172.21.15.43:0/4046281954 conn(0x556a41d13fc0 msgr2=0x556a41e2d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.474+0000 7f99d2321b80 1 -- 172.21.15.43:0/4046281954 shutdown_connections 2024-05-27T07:36:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.474+0000 7f99d2321b80 1 -- 172.21.15.43:0/4046281954 wait complete. 2024-05-27T07:36:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.475+0000 7f99d2321b80 1 -- start start 2024-05-27T07:36:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.475+0000 7f99d2321b80 1 Processor -- start 2024-05-27T07:36:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.476+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e32db0 0x556a41e3c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.476+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 0x556a41e3cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.476+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e3d040 0x556a41e48ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.476+0000 7f99d2321b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556a41c56fa0 con 0x556a41e38d30 2024-05-27T07:36:18.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.476+0000 7f99d2321b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556a41cf1760 con 0x556a41e3d040 2024-05-27T07:36:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.476+0000 7f99d2321b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556a41e3f7a0 con 0x556a41e32db0 2024-05-27T07:36:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.476+0000 7f99d2321b80 10 client.0 _ll_register_callbacks cb 0x556a41cfac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.477+0000 7f99c0e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e32db0 0x556a41e3c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.477+0000 7f99c1800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e3d040 0x556a41e48ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.477+0000 7f99bbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 0x556a41e3cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.477+0000 7f99c1800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e3d040 0x556a41e48ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:43858/0 (socket says 172.21.15.43:43858) 2024-05-27T07:36:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.477+0000 7f99c1800700 1 -- 172.21.15.43:0/2742986565 learned_addr learned my addr 172.21.15.43:0/2742986565 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.478+0000 7f99c1800700 1 -- 172.21.15.43:0/2742986565 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e32db0 msgr2=0x556a41e3c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.478+0000 7f99c1800700 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e32db0 0x556a41e3c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.478+0000 7f99c1800700 1 -- 172.21.15.43:0/2742986565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 msgr2=0x556a41e3cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.478+0000 7f99c1800700 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 0x556a41e3cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.478+0000 7f99c1800700 1 -- 172.21.15.43:0/2742986565 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a41ca08d0 con 0x556a41e3d040 2024-05-27T07:36:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.478+0000 7f99c1800700 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e3d040 0x556a41e48ad0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f99b400d010 tx=0x7f99b4003cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5359da9312fc481 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.478+0000 7f99a2a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:36:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.479+0000 7f99a2a00700 1 -- 172.21.15.43:0/2742986565 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4007ce0 con 0x556a41e3d040 2024-05-27T07:36:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.479+0000 7f99a2a00700 1 -- 172.21.15.43:0/2742986565 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7f99b402f480 con 0x556a41e3d040 2024-05-27T07:36:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.479+0000 7f99d2321b80 1 -- 172.21.15.43:0/2742986565 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99b400be50 con 0x556a41e3d040 2024-05-27T07:36:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.480+0000 7f99a2a00700 1 -- 172.21.15.43:0/2742986565 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f99b40076e0 con 0x556a41e3d040 2024-05-27T07:36:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.480+0000 7f99d2321b80 10 client.37688 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.480+0000 7f99d2321b80 1 -- 172.21.15.43:0/2742986565 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x556a41d21060 con 0x556a41e3d040 2024-05-27T07:36:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.480+0000 7f99a2a00700 1 -- 172.21.15.43:0/2742986565 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99b400be50 con 0x556a41e3d040 2024-05-27T07:36:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.480+0000 7f99a2a00700 1 -- 172.21.15.43:0/2742986565 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f99b4011070 con 0x556a41e3d040 2024-05-27T07:36:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.481+0000 7f99d2321b80 10 client.37688 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.481+0000 7f99d2321b80 10 client.37688 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:36:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.481+0000 7f99d2321b80 20 client.37688 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.481+0000 7f99d2321b80 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x556a41e4ca00 0x556a41e4eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.481+0000 7f99d2321b80 4 client.37688 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.481+0000 7f99d2321b80 1 -- 172.21.15.43:0/2742986565 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a41d62d90 con 0x556a41e4ca00 2024-05-27T07:36:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.482+0000 7f99c0e00700 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x556a41e4ca00 0x556a41e4eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.483+0000 7f99c0e00700 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x556a41e4ca00 0x556a41e4eea0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e67ca9716ac1cdde server_cookie=a5dc7a6d4463e1d4 in_seq=0 out_seq=0 2024-05-27T07:36:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.483+0000 7f99a2a00700 10 client.37688 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:36:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.484+0000 7f99a2a00700 1 -- 172.21.15.43:0/2742986565 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x556a41d62d90 con 0x556a41e4ca00 2024-05-27T07:36:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.484+0000 7f99a2a00700 10 client.37688 handle_command_reply: tid=0 2024-05-27T07:36:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.484+0000 7f99d2321b80 1 client.37688 shutdown 2024-05-27T07:36:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99d2321b80 1 -- 172.21.15.43:0/2742986565 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e3d040 msgr2=0x556a41e48ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99d2321b80 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e3d040 0x556a41e48ad0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f99b400d010 tx=0x7f99b4003cb0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99d2321b80 1 -- 172.21.15.43:0/2742986565 shutdown_connections 2024-05-27T07:36:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99d2321b80 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x556a41e4ca00 0x556a41e4eea0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99a2a00700 0 client.37688 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:36:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99d2321b80 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e3d040 0x556a41e48ad0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99d2321b80 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e32db0 0x556a41e3c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99d2321b80 1 --2- 172.21.15.43:0/2742986565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 0x556a41e3cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99d2321b80 1 -- 172.21.15.43:0/2742986565 >> 172.21.15.43:0/2742986565 conn(0x556a41d13fc0 msgr2=0x556a41e2d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99d2321b80 1 -- 172.21.15.43:0/2742986565 shutdown_connections 2024-05-27T07:36:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.485+0000 7f99d2321b80 1 -- 172.21.15.43:0/2742986565 wait complete. 2024-05-27T07:36:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.486+0000 7f99d2321b80 20 client.37688 trim_cache size 0 max 16384 2024-05-27T07:36:18.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.516+0000 7f99d2321b80 1 Processor -- start 2024-05-27T07:36:18.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.517+0000 7f99d2321b80 1 -- start start 2024-05-27T07:36:18.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.517+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 0x556a41e32870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.517+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e38d30 0x556a41e3ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.517+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e3b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.517+0000 7f99d2321b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc1b6f90 con 0x556a41e32db0 2024-05-27T07:36:18.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.517+0000 7f99d2321b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc1b53f0 con 0x556a41e38d30 2024-05-27T07:36:18.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.517+0000 7f99d2321b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc1b68c0 con 0x556a41e3d040 2024-05-27T07:36:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.518+0000 7f99c1800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e3b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.518+0000 7f99c1800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e3b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:47342/0 (socket says 172.21.15.43:47342) 2024-05-27T07:36:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.518+0000 7f99c1800700 1 -- 172.21.15.43:0/1385122542 learned_addr learned my addr 172.21.15.43:0/1385122542 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:18.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.518+0000 7f99c1800700 1 -- 172.21.15.43:0/1385122542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e38d30 msgr2=0x556a41e3ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.519+0000 7f99c0e00700 1 --2- 172.21.15.43:0/1385122542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 0x556a41e32870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.519+0000 7f99bbe00700 1 --2- 172.21.15.43:0/1385122542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e38d30 0x556a41e3ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.519+0000 7f99c1800700 1 --2- 172.21.15.43:0/1385122542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e38d30 0x556a41e3ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.519+0000 7f99c1800700 1 -- 172.21.15.43:0/1385122542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 msgr2=0x556a41e32870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.519+0000 7f99c1800700 1 --2- 172.21.15.43:0/1385122542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 0x556a41e32870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.519+0000 7f99c1800700 1 -- 172.21.15.43:0/1385122542 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a41e3f3f0 con 0x556a41e3d040 2024-05-27T07:36:18.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.519+0000 7f99bbe00700 1 --2- 172.21.15.43:0/1385122542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e38d30 0x556a41e3ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.520+0000 7f99c1800700 1 --2- 172.21.15.43:0/1385122542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e3b5c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f99b4005430 tx=0x7f99b4009390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc63148328270d89 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.520+0000 7f99ad600700 1 -- 172.21.15.43:0/1385122542 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b40054b0 con 0x556a41e3d040 2024-05-27T07:36:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.520+0000 7f99ad600700 1 -- 172.21.15.43:0/1385122542 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99b403d070 con 0x556a41e3d040 2024-05-27T07:36:18.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.520+0000 7f99ad600700 1 -- 172.21.15.43:0/1385122542 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4012650 con 0x556a41e3d040 2024-05-27T07:36:18.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.521+0000 7f99d2321b80 1 -- 172.21.15.43:0/1385122542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 msgr2=0x556a41e3b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:18.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.521+0000 7f99d2321b80 1 --2- 172.21.15.43:0/1385122542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e3b5c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f99b4005430 tx=0x7f99b4009390 comp rx=0 tx=0).stop 2024-05-27T07:36:18.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.522+0000 7f99d2321b80 1 -- 172.21.15.43:0/1385122542 shutdown_connections 2024-05-27T07:36:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.522+0000 7f99d2321b80 1 --2- 172.21.15.43:0/1385122542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e3b5c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.522+0000 7f99d2321b80 1 --2- 172.21.15.43:0/1385122542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e32db0 0x556a41e32870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.522+0000 7f99d2321b80 1 --2- 172.21.15.43:0/1385122542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e38d30 0x556a41e3ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.522+0000 7f99d2321b80 1 -- 172.21.15.43:0/1385122542 >> 172.21.15.43:0/1385122542 conn(0x556a41d13fc0 msgr2=0x556a41e3aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.522+0000 7f99d2321b80 1 -- 172.21.15.43:0/1385122542 shutdown_connections 2024-05-27T07:36:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.522+0000 7f99d2321b80 1 -- 172.21.15.43:0/1385122542 wait complete. 2024-05-27T07:36:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.523+0000 7f99d2321b80 1 -- start start 2024-05-27T07:36:18.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.523+0000 7f99d2321b80 1 Processor -- start 2024-05-27T07:36:18.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.524+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e32db0 0x556a41e45100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.524+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 0x556a41e48240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.524+0000 7f99d2321b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e48780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.525+0000 7f99d2321b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc1b61f0 con 0x556a41e38d30 2024-05-27T07:36:18.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.525+0000 7f99d2321b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc1b5b20 con 0x556a41e32db0 2024-05-27T07:36:18.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.525+0000 7f99d2321b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x556a41e3c3a0 con 0x556a41e3d040 2024-05-27T07:36:18.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.525+0000 7f99d2321b80 10 client.0 _ll_register_callbacks cb 0x556a41cfac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:18.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.525+0000 7f99c0e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e32db0 0x556a41e45100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.525+0000 7f99c0e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e32db0 0x556a41e45100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:43862/0 (socket says 172.21.15.43:43862) 2024-05-27T07:36:18.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.525+0000 7f99c0e00700 1 -- 172.21.15.43:0/678293232 learned_addr learned my addr 172.21.15.43:0/678293232 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:18.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.525+0000 7f99c1800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e48780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.525+0000 7f99bbe00700 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 0x556a41e48240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.526+0000 7f99c0e00700 1 -- 172.21.15.43:0/678293232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 msgr2=0x556a41e48780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:18.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.526+0000 7f99c0e00700 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e48780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.526+0000 7f99c0e00700 1 -- 172.21.15.43:0/678293232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 msgr2=0x556a41e48240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:18.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.526+0000 7f99c0e00700 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 0x556a41e48240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.526+0000 7f99c0e00700 1 -- 172.21.15.43:0/678293232 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a41e48f00 con 0x556a41e32db0 2024-05-27T07:36:18.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.527+0000 7f99bbe00700 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 0x556a41e48240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:36:18.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.527+0000 7f99c1800700 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e48780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:18.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.527+0000 7f99c0e00700 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e32db0 0x556a41e45100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x556a41e33c70 tx=0x7f99b00047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8f557cf62de5193 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:18.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.527+0000 7f99a2a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:36:18.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.528+0000 7f99a2a00700 1 -- 172.21.15.43:0/678293232 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b002d060 con 0x556a41e32db0 2024-05-27T07:36:18.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.528+0000 7f99a2a00700 1 -- 172.21.15.43:0/678293232 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7f99b00282d0 con 0x556a41e32db0 2024-05-27T07:36:18.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.529+0000 7f99d2321b80 1 -- 172.21.15.43:0/678293232 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99b403d070 con 0x556a41e32db0 2024-05-27T07:36:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.530+0000 7f99a2a00700 1 -- 172.21.15.43:0/678293232 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f99b00284d0 con 0x556a41e32db0 2024-05-27T07:36:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.531+0000 7f99d2321b80 10 client.37691 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.531+0000 7f99d2321b80 1 -- 172.21.15.43:0/678293232 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x556a41eb7410 con 0x556a41e32db0 2024-05-27T07:36:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.531+0000 7f99a2a00700 1 -- 172.21.15.43:0/678293232 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f99b403d070 con 0x556a41e32db0 2024-05-27T07:36:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.532+0000 7f99a2a00700 1 -- 172.21.15.43:0/678293232 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f99b001f920 con 0x556a41e32db0 2024-05-27T07:36:18.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.532+0000 7f99d2321b80 10 client.37691 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:18.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.532+0000 7f99d2321b80 10 client.37691 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:36:18.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.532+0000 7f99d2321b80 20 client.37691 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:18.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.532+0000 7f99d2321b80 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x556a41eb79e0 0x556a41eb9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:18.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.532+0000 7f99d2321b80 4 client.37691 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:18.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.532+0000 7f99d2321b80 1 -- 172.21.15.43:0/678293232 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556a41d5c570 con 0x556a41eb79e0 2024-05-27T07:36:18.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.533+0000 7f99bbe00700 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x556a41eb79e0 0x556a41eb9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:18.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.534+0000 7f99bbe00700 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x556a41eb79e0 0x556a41eb9e00 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=39d09d15f6b068f6 server_cookie=5e4212cc0eb97be6 in_seq=0 out_seq=0 2024-05-27T07:36:18.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.534+0000 7f99a2a00700 10 client.37691 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:36:18.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.535+0000 7f99a2a00700 1 -- 172.21.15.43:0/678293232 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556a41d5c570 con 0x556a41eb79e0 2024-05-27T07:36:18.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.535+0000 7f99a2a00700 10 client.37691 handle_command_reply: tid=0 2024-05-27T07:36:18.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.535+0000 7f99d2321b80 1 client.37691 shutdown 2024-05-27T07:36:18.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.535+0000 7f99d2321b80 1 -- 172.21.15.43:0/678293232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e32db0 msgr2=0x556a41e45100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:18.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.535+0000 7f99d2321b80 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e32db0 0x556a41e45100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x556a41e33c70 tx=0x7f99b00047c0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.538+0000 7f99d2321b80 1 -- 172.21.15.43:0/678293232 shutdown_connections 2024-05-27T07:36:18.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.538+0000 7f99d2321b80 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x556a41eb79e0 0x556a41eb9e00 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.538+0000 7f99d2321b80 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556a41e3d040 0x556a41e48780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.538+0000 7f99d2321b80 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556a41e32db0 0x556a41e45100 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.538+0000 7f99d2321b80 1 --2- 172.21.15.43:0/678293232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556a41e38d30 0x556a41e48240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.538+0000 7f99d2321b80 1 -- 172.21.15.43:0/678293232 >> 172.21.15.43:0/678293232 conn(0x556a41d13fc0 msgr2=0x556a41e3b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:18.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.538+0000 7f99a2a00700 0 client.37691 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:36:18.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.540+0000 7f99d2321b80 1 -- 172.21.15.43:0/678293232 shutdown_connections 2024-05-27T07:36:18.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.541+0000 7f99d2321b80 1 -- 172.21.15.43:0/678293232 wait complete. 2024-05-27T07:36:18.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.542+0000 7f99d2321b80 20 client.37691 trim_cache size 0 max 16384 2024-05-27T07:36:18.540 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:36:18.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.544+0000 7f99c3800700 1 -- 172.21.15.43:0/2707645384 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f99a40b2a20 msgr2=0x7f99a40b4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:18.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.544+0000 7f99c3800700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f99a40b2a20 0x7f99a40b4f30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f99b0005950 tx=0x7f99b000b500 comp rx=0 tx=0).stop 2024-05-27T07:36:18.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.544+0000 7f99c3800700 1 -- 172.21.15.43:0/2707645384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99bc108ef0 msgr2=0x7f99bc1a25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:18.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.544+0000 7f99c3800700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99bc108ef0 0x7f99bc1a25b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f99a800e9c0 tx=0x7f99a800ecd0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.545+0000 7f99c3800700 1 -- 172.21.15.43:0/2707645384 shutdown_connections 2024-05-27T07:36:18.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.545+0000 7f99c3800700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f99a40b2a20 0x7f99a40b4f30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.545+0000 7f99c3800700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99bc109840 0x7f99bc1ad0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.545+0000 7f99c3800700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99bc1045c0 0x7f99bc1a2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.545+0000 7f99c3800700 1 --2- 172.21.15.43:0/2707645384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99bc108ef0 0x7f99bc1a25b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:18.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.545+0000 7f99c3800700 1 -- 172.21.15.43:0/2707645384 >> 172.21.15.43:0/2707645384 conn(0x7f99bc0ffe90 msgr2=0x7f99bc1086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:18.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.545+0000 7f99c3800700 1 -- 172.21.15.43:0/2707645384 shutdown_connections 2024-05-27T07:36:18.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:18.546+0000 7f99c3800700 1 -- 172.21.15.43:0/2707645384 wait complete. 2024-05-27T07:36:18.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:18 smithi043 ceph-mon[31531]: pgmap v752: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.2 MiB/s wr, 186 op/s 2024-05-27T07:36:18.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:18 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:36:18.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 54735 5.83 MB/sec execute 379 sec latency 129.252 ms 2024-05-27T07:36:18.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:18 smithi132 ceph-mon[35283]: pgmap v752: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.2 MiB/s wr, 186 op/s 2024-05-27T07:36:18.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:18 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:36:18.963 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:18 smithi179 ceph-mon[35888]: pgmap v752: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.2 MiB/s wr, 186 op/s 2024-05-27T07:36:18.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:18 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:36:19.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:36:19.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 54839 5.83 MB/sec execute 380 sec latency 87.336 ms 2024-05-27T07:36:20.121 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:20.554 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:20 smithi043 ceph-mon[31531]: pgmap v753: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.2 MiB/s wr, 186 op/s 2024-05-27T07:36:20.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 54941 5.83 MB/sec execute 381 sec latency 144.082 ms 2024-05-27T07:36:20.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:20 smithi132 ceph-mon[35283]: pgmap v753: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.2 MiB/s wr, 186 op/s 2024-05-27T07:36:20.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:20 smithi179 ceph-mon[35888]: pgmap v753: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.2 MiB/s wr, 186 op/s 2024-05-27T07:36:21.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 55082 5.84 MB/sec execute 382 sec latency 96.436 ms 2024-05-27T07:36:22.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 55215 5.85 MB/sec execute 383 sec latency 86.764 ms 2024-05-27T07:36:22.845 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:22 smithi132 ceph-mon[35283]: pgmap v754: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 4.2 MiB/s wr, 197 op/s 2024-05-27T07:36:22.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:22 smithi043 ceph-mon[31531]: pgmap v754: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 4.2 MiB/s wr, 197 op/s 2024-05-27T07:36:22.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:22 smithi179 ceph-mon[35888]: pgmap v754: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 4.2 MiB/s wr, 197 op/s 2024-05-27T07:36:23.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 55313 5.85 MB/sec execute 384 sec latency 95.972 ms 2024-05-27T07:36:24.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:24 smithi043 ceph-mon[31531]: pgmap v755: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 3.6 MiB/s wr, 166 op/s 2024-05-27T07:36:24.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 55410 5.85 MB/sec execute 385 sec latency 107.393 ms 2024-05-27T07:36:24.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:24 smithi132 ceph-mon[35283]: pgmap v755: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 3.6 MiB/s wr, 166 op/s 2024-05-27T07:36:24.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:24 smithi179 ceph-mon[35888]: pgmap v755: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 3.6 MiB/s wr, 166 op/s 2024-05-27T07:36:25.819 INFO:tasks.workunit.client.0.smithi043.stdout: 10 55488 5.84 MB/sec execute 386 sec latency 175.010 ms 2024-05-27T07:36:26.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.157+0000 7fdf18000700 1 -- 172.21.15.43:0/2559818728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf10108f10 msgr2=0x7fdf10109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:26.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.157+0000 7fdf18000700 1 --2- 172.21.15.43:0/2559818728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf10108f10 0x7fdf10109320 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009920 tx=0x7fdefc009c30 comp rx=0 tx=0).stop 2024-05-27T07:36:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.157+0000 7fdf18000700 1 -- 172.21.15.43:0/2559818728 shutdown_connections 2024-05-27T07:36:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.157+0000 7fdf18000700 1 --2- 172.21.15.43:0/2559818728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf10109860 0x7fdf101111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.157+0000 7fdf18000700 1 --2- 172.21.15.43:0/2559818728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf101045e0 0x7fdf101049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.158+0000 7fdf18000700 1 --2- 172.21.15.43:0/2559818728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf10108f10 0x7fdf10109320 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.158+0000 7fdf18000700 1 -- 172.21.15.43:0/2559818728 >> 172.21.15.43:0/2559818728 conn(0x7fdf10100030 msgr2=0x7fdf10102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:26.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.159+0000 7fdf18000700 1 -- 172.21.15.43:0/2559818728 shutdown_connections 2024-05-27T07:36:26.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.159+0000 7fdf18000700 1 -- 172.21.15.43:0/2559818728 wait complete. 2024-05-27T07:36:26.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.161+0000 7fdf18000700 1 Processor -- start 2024-05-27T07:36:26.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.161+0000 7fdf18000700 1 -- start start 2024-05-27T07:36:26.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.161+0000 7fdf18000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf101045e0 0x7fdf1019d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.161+0000 7fdf18000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf10108f10 0x7fdf1019d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.162+0000 7fdf18000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf10109860 0x7fdf101a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.162+0000 7fdf18000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdf10114370 con 0x7fdf10108f10 2024-05-27T07:36:26.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.162+0000 7fdf18000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101141f0 con 0x7fdf10109860 2024-05-27T07:36:26.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.162+0000 7fdf18000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101144f0 con 0x7fdf101045e0 2024-05-27T07:36:26.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.162+0000 7fdf15600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf101045e0 0x7fdf1019d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.162+0000 7fdf14c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf10108f10 0x7fdf1019d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.162+0000 7fdf15600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf101045e0 0x7fdf1019d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:37820/0 (socket says 172.21.15.43:37820) 2024-05-27T07:36:26.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.162+0000 7fdf15600700 1 -- 172.21.15.43:0/376382814 learned_addr learned my addr 172.21.15.43:0/376382814 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:26.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.163+0000 7fdf16000700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf10109860 0x7fdf101a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.163+0000 7fdf14c00700 1 -- 172.21.15.43:0/376382814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf101045e0 msgr2=0x7fdf1019d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:26.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.163+0000 7fdf14c00700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf101045e0 0x7fdf1019d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.163+0000 7fdf14c00700 1 -- 172.21.15.43:0/376382814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf10109860 msgr2=0x7fdf101a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:26.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.163+0000 7fdf14c00700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf10109860 0x7fdf101a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.163+0000 7fdf14c00700 1 -- 172.21.15.43:0/376382814 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf101a1e40 con 0x7fdf10108f10 2024-05-27T07:36:26.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.164+0000 7fdf14c00700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf10108f10 0x7fdf1019d5c0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fdefc000c00 tx=0x7fdefc00f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:26.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.164+0000 7fdf0e000700 1 -- 172.21.15.43:0/376382814 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc00b5e0 con 0x7fdf10108f10 2024-05-27T07:36:26.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.164+0000 7fdf0e000700 1 -- 172.21.15.43:0/376382814 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdefc005040 con 0x7fdf10108f10 2024-05-27T07:36:26.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.164+0000 7fdf0e000700 1 -- 172.21.15.43:0/376382814 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0038e0 con 0x7fdf10108f10 2024-05-27T07:36:26.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.164+0000 7fdf18000700 1 -- 172.21.15.43:0/376382814 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf101a20d0 con 0x7fdf10108f10 2024-05-27T07:36:26.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.165+0000 7fdf18000700 1 -- 172.21.15.43:0/376382814 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf101a2560 con 0x7fdf10108f10 2024-05-27T07:36:26.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.170+0000 7fdf0e000700 1 -- 172.21.15.43:0/376382814 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fdefc0051f0 con 0x7fdf10108f10 2024-05-27T07:36:26.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.171+0000 7fdf26b0fb80 1 Processor -- start 2024-05-27T07:36:26.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.171+0000 7fdf26b0fb80 1 -- start start 2024-05-27T07:36:26.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.172+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cd9a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.172+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cd9a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.172+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cd9ae70 0x55e24cda7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.172+0000 7fdf26b0fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e24ccc8e40 con 0x55e24cd079f0 2024-05-27T07:36:26.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.172+0000 7fdf26b0fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e24ccb89e0 con 0x55e24cd9ae70 2024-05-27T07:36:26.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.172+0000 7fdf26b0fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e24ccbca20 con 0x55e24cc1be40 2024-05-27T07:36:26.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.173+0000 7fdf15600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cd9a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.173+0000 7fdf0e000700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdef80b2af0 0x7fdef80b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.173+0000 7fdf15600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cd9a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:37822/0 (socket says 172.21.15.43:37822) 2024-05-27T07:36:26.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.173+0000 7fdf15600700 1 -- 172.21.15.43:0/268766984 learned_addr learned my addr 172.21.15.43:0/268766984 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:26.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.173+0000 7fdf0e000700 1 -- 172.21.15.43:0/376382814 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7fdefc0e3830 con 0x7fdf10108f10 2024-05-27T07:36:26.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.174+0000 7fdf15600700 1 -- 172.21.15.43:0/268766984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cd9ae70 msgr2=0x55e24cda7110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:36:26.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.174+0000 7fdf15600700 1 --2- 172.21.15.43:0/268766984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cd9ae70 0x55e24cda7110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.174+0000 7fdf15600700 1 -- 172.21.15.43:0/268766984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 msgr2=0x55e24cd9a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.174+0000 7fdf16000700 1 --2- 172.21.15.43:0/268766984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cd9a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.174+0000 7fdf15600700 1 --2- 172.21.15.43:0/268766984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cd9a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.175+0000 7fdf15600700 1 -- 172.21.15.43:0/268766984 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e24cc81420 con 0x55e24cc1be40 2024-05-27T07:36:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.175+0000 7fdf16000700 1 --2- 172.21.15.43:0/268766984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cd9a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:36:26.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.176+0000 7fdf16000700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdef80b2af0 0x7fdef80b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.177+0000 7fdf15600700 1 --2- 172.21.15.43:0/268766984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cd9a870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400ae40 tx=0x7fdf0400f6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2caac95a4e9cc6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:26.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.178+0000 7fdf02a00700 1 -- 172.21.15.43:0/268766984 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0401d070 con 0x55e24cc1be40 2024-05-27T07:36:26.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.178+0000 7fdf02a00700 1 -- 172.21.15.43:0/268766984 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdf04018ad0 con 0x55e24cc1be40 2024-05-27T07:36:26.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.178+0000 7fdf02a00700 1 -- 172.21.15.43:0/268766984 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf04010850 con 0x55e24cc1be40 2024-05-27T07:36:26.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.178+0000 7fdf16000700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdef80b2af0 0x7fdef80b5000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x55e24cd9cff0 tx=0x7fdf08004ad0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:26.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.180+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/268766984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 msgr2=0x55e24cd9a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:26.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.180+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/268766984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cd9a870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400ae40 tx=0x7fdf0400f6a0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.180+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/268766984 shutdown_connections 2024-05-27T07:36:26.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.180+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/268766984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cd9ae70 0x55e24cda7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.180+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/268766984 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cd9a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.180+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/268766984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cd9a870 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.180+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/268766984 >> 172.21.15.43:0/268766984 conn(0x55e24ccb5390 msgr2=0x55e24cd964c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:26.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.180+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/268766984 shutdown_connections 2024-05-27T07:36:26.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.181+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/268766984 wait complete. 2024-05-27T07:36:26.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.182+0000 7fdf26b0fb80 1 -- start start 2024-05-27T07:36:26.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.182+0000 7fdf26b0fb80 1 Processor -- start 2024-05-27T07:36:26.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.184+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cda46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.184+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cda4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.184+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cd9ae70 0x55e24cda5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.184+0000 7fdf26b0fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e24cbbefa0 con 0x55e24cd079f0 2024-05-27T07:36:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.184+0000 7fdf26b0fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e24cc59760 con 0x55e24cc1be40 2024-05-27T07:36:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.184+0000 7fdf26b0fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e24cda7b80 con 0x55e24cd9ae70 2024-05-27T07:36:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.184+0000 7fdf26b0fb80 10 client.0 _ll_register_callbacks cb 0x55e24cc62c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.185+0000 7fdf15600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cda46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.185+0000 7fdf15600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cda46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59904/0 (socket says 172.21.15.43:59904) 2024-05-27T07:36:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.185+0000 7fdf15600700 1 -- 172.21.15.43:0/4100500841 learned_addr learned my addr 172.21.15.43:0/4100500841 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:26.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.185+0000 7fdf14c00700 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cda4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.185+0000 7fdf15600700 1 -- 172.21.15.43:0/4100500841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cd9ae70 msgr2=0x55e24cda5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:26.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.185+0000 7fdf15600700 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cd9ae70 0x55e24cda5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.185+0000 7fdf15600700 1 -- 172.21.15.43:0/4100500841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 msgr2=0x55e24cda4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.185+0000 7fdf15600700 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cda4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.185+0000 7fdf15600700 1 -- 172.21.15.43:0/4100500841 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e24cc088d0 con 0x55e24cc1be40 2024-05-27T07:36:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.186+0000 7fdf15600700 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cda46e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400a320 tx=0x7fdf04018720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e519093988f8d441 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.186+0000 7fdef7400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:36:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.186+0000 7fdef7400700 1 -- 172.21.15.43:0/4100500841 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0401d040 con 0x55e24cc1be40 2024-05-27T07:36:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.187+0000 7fdef7400700 1 -- 172.21.15.43:0/4100500841 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7fdf04005230 con 0x55e24cc1be40 2024-05-27T07:36:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.187+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/4100500841 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf04018ad0 con 0x55e24cc1be40 2024-05-27T07:36:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.188+0000 7fdef7400700 1 -- 172.21.15.43:0/4100500841 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7fdf04005430 con 0x55e24cc1be40 2024-05-27T07:36:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.188+0000 7fdf26b0fb80 10 client.37700 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:26.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.188+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/4100500841 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55e24cc89060 con 0x55e24cc1be40 2024-05-27T07:36:26.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.188+0000 7fdef7400700 1 -- 172.21.15.43:0/4100500841 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdf04018ad0 con 0x55e24cc1be40 2024-05-27T07:36:26.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.188+0000 7fdef7400700 1 -- 172.21.15.43:0/4100500841 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7fdf04020070 con 0x55e24cc1be40 2024-05-27T07:36:26.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.189+0000 7fdf26b0fb80 10 client.37700 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:26.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.189+0000 7fdf26b0fb80 10 client.37700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:26.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.189+0000 7fdf26b0fb80 20 client.37700 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:26.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.189+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e24cdb4c90 0x55e24cdb7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.189+0000 7fdf26b0fb80 4 client.37700 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:26.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.189+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/4100500841 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e24cccad90 con 0x55e24cdb4c90 2024-05-27T07:36:26.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.190+0000 7fdf14c00700 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e24cdb4c90 0x55e24cdb7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.193+0000 7fdef7400700 10 client.37700 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:26.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.193+0000 7fdf14c00700 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e24cdb4c90 0x55e24cdb7100 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f5c71705ec7cb07 server_cookie=e32d1d7dc12579f0 in_seq=0 out_seq=0 2024-05-27T07:36:26.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.194+0000 7fdef7400700 1 -- 172.21.15.43:0/4100500841 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e24cccad90 con 0x55e24cdb4c90 2024-05-27T07:36:26.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.194+0000 7fdef7400700 10 client.37700 handle_command_reply: tid=0 2024-05-27T07:36:26.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.194+0000 7fdf26b0fb80 1 client.37700 shutdown 2024-05-27T07:36:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.195+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/4100500841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 msgr2=0x55e24cda46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.195+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cda46e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400a320 tx=0x7fdf04018720 comp rx=0 tx=0).stop 2024-05-27T07:36:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.195+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/4100500841 shutdown_connections 2024-05-27T07:36:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.195+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cd9ae70 0x55e24cda5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.195+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cda46e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.195+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e24cdb4c90 0x55e24cdb7100 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.195+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/4100500841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cda4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.195+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/4100500841 >> 172.21.15.43:0/4100500841 conn(0x55e24ccb5390 msgr2=0x55e24cd9a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:26.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.195+0000 7fdef7400700 0 client.37700 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:26.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.196+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/4100500841 shutdown_connections 2024-05-27T07:36:26.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.196+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/4100500841 wait complete. 2024-05-27T07:36:26.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.197+0000 7fdf26b0fb80 20 client.37700 trim_cache size 0 max 16384 2024-05-27T07:36:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.219+0000 7fdf26b0fb80 1 Processor -- start 2024-05-27T07:36:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.220+0000 7fdf26b0fb80 1 -- start start 2024-05-27T07:36:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.220+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cda6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.220+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cd9a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.220+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cd9ae70 0x55e24cdb0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.220+0000 7fdf26b0fb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b2cb0 con 0x55e24cd079f0 2024-05-27T07:36:26.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.220+0000 7fdf26b0fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b1110 con 0x55e24cd9ae70 2024-05-27T07:36:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.220+0000 7fdf26b0fb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b25e0 con 0x55e24cc1be40 2024-05-27T07:36:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.221+0000 7fdf15600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cda6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.221+0000 7fdf15600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cda6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:37850/0 (socket says 172.21.15.43:37850) 2024-05-27T07:36:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.221+0000 7fdf15600700 1 -- 172.21.15.43:0/2367361126 learned_addr learned my addr 172.21.15.43:0/2367361126 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.221+0000 7fdf14c00700 1 --2- 172.21.15.43:0/2367361126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cd9a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.221+0000 7fdf15600700 1 -- 172.21.15.43:0/2367361126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cd9ae70 msgr2=0x55e24cdb0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.222+0000 7fdf15600700 1 --2- 172.21.15.43:0/2367361126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cd9ae70 0x55e24cdb0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.222+0000 7fdf15600700 1 -- 172.21.15.43:0/2367361126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 msgr2=0x55e24cd9a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.222+0000 7fdf15600700 1 --2- 172.21.15.43:0/2367361126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cd9a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.222+0000 7fdf15600700 1 -- 172.21.15.43:0/2367361126 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e24cd92240 con 0x55e24cc1be40 2024-05-27T07:36:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.222+0000 7fdf15600700 1 --2- 172.21.15.43:0/2367361126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cda6b80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400ae40 tx=0x7fdf0400faa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb0c2660c9298818 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.223+0000 7fdf02000700 1 -- 172.21.15.43:0/2367361126 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0401d040 con 0x55e24cc1be40 2024-05-27T07:36:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.223+0000 7fdf02000700 1 -- 172.21.15.43:0/2367361126 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdf04026070 con 0x55e24cc1be40 2024-05-27T07:36:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.223+0000 7fdf02000700 1 -- 172.21.15.43:0/2367361126 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0401d040 con 0x55e24cc1be40 2024-05-27T07:36:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.224+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/2367361126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 msgr2=0x55e24cda6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.224+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/2367361126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cda6b80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400ae40 tx=0x7fdf0400faa0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.224+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/2367361126 shutdown_connections 2024-05-27T07:36:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.224+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/2367361126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cd9ae70 0x55e24cdb0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.224+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/2367361126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cc1be40 0x55e24cda6b80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.224+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/2367361126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd079f0 0x55e24cd9a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.224+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/2367361126 >> 172.21.15.43:0/2367361126 conn(0x55e24ccb5390 msgr2=0x55e24cd9a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.224+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/2367361126 shutdown_connections 2024-05-27T07:36:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.224+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/2367361126 wait complete. 2024-05-27T07:36:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.225+0000 7fdf26b0fb80 1 -- start start 2024-05-27T07:36:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.225+0000 7fdf26b0fb80 1 Processor -- start 2024-05-27T07:36:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.226+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cd9e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.226+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cd079f0 0x55e24cd9e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.226+0000 7fdf26b0fb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd9ae70 0x55e24cd9ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.227+0000 7fdf15600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cd9e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.227+0000 7fdf15600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cd9e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:59920/0 (socket says 172.21.15.43:59920) 2024-05-27T07:36:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.227+0000 7fdf15600700 1 -- 172.21.15.43:0/1733391743 learned_addr learned my addr 172.21.15.43:0/1733391743 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.227+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b1f10 con 0x55e24cd9ae70 2024-05-27T07:36:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.227+0000 7fdf14c00700 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cd079f0 0x55e24cd9e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.227+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b1840 con 0x55e24cc1be40 2024-05-27T07:36:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.227+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e24cd95150 con 0x55e24cd079f0 2024-05-27T07:36:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.227+0000 7fdf16000700 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd9ae70 0x55e24cd9ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.227+0000 7fdf26b0fb80 10 client.0 _ll_register_callbacks cb 0x55e24cc62c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.228+0000 7fdf14c00700 1 -- 172.21.15.43:0/1733391743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 msgr2=0x55e24cd9e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.228+0000 7fdf14c00700 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cd9e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.228+0000 7fdf14c00700 1 -- 172.21.15.43:0/1733391743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd9ae70 msgr2=0x55e24cd9ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.228+0000 7fdf14c00700 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd9ae70 0x55e24cd9ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.228+0000 7fdf14c00700 1 -- 172.21.15.43:0/1733391743 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e24cdabe10 con 0x55e24cd079f0 2024-05-27T07:36:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.228+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf04026070 con 0x55e24cd079f0 2024-05-27T07:36:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.228+0000 7fdf15600700 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cd9e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.229+0000 7fdf14c00700 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cd079f0 0x55e24cd9e690 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0e0b50 tx=0x7fdefc0a9f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4722c241599f4b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.229+0000 7fdef7400700 10 client.27742 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:36:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.229+0000 7fdef7400700 1 -- 172.21.15.43:0/1733391743 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0aa770 con 0x55e24cd079f0 2024-05-27T07:36:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.229+0000 7fdef7400700 1 -- 172.21.15.43:0/1733391743 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7fdefc0e0dd0 con 0x55e24cd079f0 2024-05-27T07:36:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.229+0000 7fdef7400700 1 -- 172.21.15.43:0/1733391743 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7fdefc0aeb20 con 0x55e24cd079f0 2024-05-27T07:36:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.229+0000 7fdf26b0fb80 10 client.27742 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.229+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({fsmap=144}) -- 0x55e24cdacfb0 con 0x55e24cd079f0 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.230+0000 7fdef7400700 1 -- 172.21.15.43:0/1733391743 <== mon.1 v2:172.21.15.179:3300/0 4 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7fdefc005510 con 0x55e24cd079f0 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.230+0000 7fdf26b0fb80 10 client.27742 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.230+0000 7fdf26b0fb80 10 client.27742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.230+0000 7fdf26b0fb80 20 client.27742 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.230+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e24cdad5e0 0x55e24ce217e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.230+0000 7fdf26b0fb80 4 client.27742 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.230+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e24ccc4570 con 0x55e24cdad5e0 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.231+0000 7fdf15600700 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e24cdad5e0 0x55e24ce217e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.231+0000 7fdf15600700 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e24cdad5e0 0x55e24ce217e0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56a0ec5b63788468 server_cookie=455edb7217b812b5 in_seq=0 out_seq=0 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.231+0000 7fdef7400700 10 client.27742 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.232+0000 7fdef7400700 1 -- 172.21.15.43:0/1733391743 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e24ccc4570 con 0x55e24cdad5e0 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.232+0000 7fdef7400700 10 client.27742 handle_command_reply: tid=0 2024-05-27T07:36:26.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.232+0000 7fdf26b0fb80 1 client.27742 shutdown 2024-05-27T07:36:26.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.232+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cd079f0 msgr2=0x55e24cd9e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:26.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.232+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cd079f0 0x55e24cd9e690 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0e0b50 tx=0x7fdefc0a9f90 comp rx=0 tx=0).stop 2024-05-27T07:36:26.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.233+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 shutdown_connections 2024-05-27T07:36:26.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.233+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e24cdad5e0 0x55e24ce217e0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.233+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e24cd9ae70 0x55e24cd9ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.233+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e24cc1be40 0x55e24cd9e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.233+0000 7fdf26b0fb80 1 --2- 172.21.15.43:0/1733391743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e24cd079f0 0x55e24cd9e690 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.233+0000 7fdef7400700 0 client.27742 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:26.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.233+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 >> 172.21.15.43:0/1733391743 conn(0x55e24ccb5390 msgr2=0x55e24cd9bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:26.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.233+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 shutdown_connections 2024-05-27T07:36:26.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.233+0000 7fdf26b0fb80 1 -- 172.21.15.43:0/1733391743 wait complete. 2024-05-27T07:36:26.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.233+0000 7fdf26b0fb80 20 client.27742 trim_cache size 0 max 16384 2024-05-27T07:36:26.232 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.234+0000 7fdf18000700 1 -- 172.21.15.43:0/376382814 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdef80b2af0 msgr2=0x7fdef80b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.234+0000 7fdf18000700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdef80b2af0 0x7fdef80b5000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x55e24cd9cff0 tx=0x7fdf08004ad0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.235+0000 7fdf18000700 1 -- 172.21.15.43:0/376382814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf10108f10 msgr2=0x7fdf1019d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.235+0000 7fdf18000700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf10108f10 0x7fdf1019d5c0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fdefc000c00 tx=0x7fdefc00f950 comp rx=0 tx=0).stop 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.235+0000 7fdf18000700 1 -- 172.21.15.43:0/376382814 shutdown_connections 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.235+0000 7fdf18000700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdef80b2af0 0x7fdef80b5000 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.235+0000 7fdf18000700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf10109860 0x7fdf101a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.235+0000 7fdf18000700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf101045e0 0x7fdf1019d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.235+0000 7fdf18000700 1 --2- 172.21.15.43:0/376382814 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf10108f10 0x7fdf1019d5c0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.235+0000 7fdf18000700 1 -- 172.21.15.43:0/376382814 >> 172.21.15.43:0/376382814 conn(0x7fdf10100030 msgr2=0x7fdf1010a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:26.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.235+0000 7fdf18000700 1 -- 172.21.15.43:0/376382814 shutdown_connections 2024-05-27T07:36:26.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:26.235+0000 7fdf18000700 1 -- 172.21.15.43:0/376382814 wait complete. 2024-05-27T07:36:26.820 INFO:tasks.workunit.client.0.smithi043.stdout: 10 55599 5.84 MB/sec execute 387 sec latency 110.958 ms 2024-05-27T07:36:26.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:26 smithi132 ceph-mon[35283]: pgmap v756: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 3.5 MiB/s wr, 164 op/s 2024-05-27T07:36:26.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:26 smithi043 ceph-mon[31531]: pgmap v756: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 3.5 MiB/s wr, 164 op/s 2024-05-27T07:36:26.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:26 smithi179 ceph-mon[35888]: pgmap v756: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 3.5 MiB/s wr, 164 op/s 2024-05-27T07:36:27.820 INFO:tasks.workunit.client.0.smithi043.stdout: 10 55726 5.85 MB/sec execute 388 sec latency 84.087 ms 2024-05-27T07:36:28.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:28 smithi043 ceph-mon[31531]: pgmap v757: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 5.4 MiB/s wr, 236 op/s 2024-05-27T07:36:28.820 INFO:tasks.workunit.client.0.smithi043.stdout: 10 55867 5.85 MB/sec execute 389 sec latency 103.949 ms 2024-05-27T07:36:28.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:28 smithi132 ceph-mon[35283]: pgmap v757: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 5.4 MiB/s wr, 236 op/s 2024-05-27T07:36:28.849 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:36:28.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:36:28.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:28 smithi179 ceph-mon[35888]: pgmap v757: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 5.4 MiB/s wr, 236 op/s 2024-05-27T07:36:29.520 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:29.820 INFO:tasks.workunit.client.0.smithi043.stdout: 10 55965 5.85 MB/sec execute 390 sec latency 172.049 ms 2024-05-27T07:36:30.820 INFO:tasks.workunit.client.0.smithi043.stdout: 10 56050 5.84 MB/sec execute 391 sec latency 110.101 ms 2024-05-27T07:36:30.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:30 smithi132 ceph-mon[35283]: pgmap v758: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-05-27T07:36:30.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:30 smithi043 ceph-mon[31531]: pgmap v758: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-05-27T07:36:30.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:30 smithi179 ceph-mon[35888]: pgmap v758: 129 pgs: 129 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-05-27T07:36:31.820 INFO:tasks.workunit.client.0.smithi043.stdout: 10 56131 5.83 MB/sec execute 392 sec latency 229.228 ms 2024-05-27T07:36:32.820 INFO:tasks.workunit.client.0.smithi043.stdout: 10 56235 5.83 MB/sec execute 393 sec latency 75.462 ms 2024-05-27T07:36:32.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:32 smithi132 ceph-mon[35283]: pgmap v759: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 5.3 MiB/s wr, 228 op/s 2024-05-27T07:36:32.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:32 smithi043 ceph-mon[31531]: pgmap v759: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 5.3 MiB/s wr, 228 op/s 2024-05-27T07:36:32.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:32 smithi179 ceph-mon[35888]: pgmap v759: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 5.3 MiB/s wr, 228 op/s 2024-05-27T07:36:33.820 INFO:tasks.workunit.client.0.smithi043.stdout: 10 56349 5.84 MB/sec execute 394 sec latency 70.299 ms 2024-05-27T07:36:34.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:34 smithi043 ceph-mon[31531]: pgmap v760: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.6 MiB/s wr, 146 op/s 2024-05-27T07:36:34.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:34 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:36:34.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:34 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:36:34.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:34 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:34.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:34 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:34.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:34 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:36:34.821 INFO:tasks.workunit.client.0.smithi043.stdout: 10 56466 5.85 MB/sec execute 395 sec latency 111.728 ms 2024-05-27T07:36:34.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:34 smithi179 ceph-mon[35888]: pgmap v760: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.6 MiB/s wr, 146 op/s 2024-05-27T07:36:34.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:34 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:36:34.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:34 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:36:34.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:34 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:34.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:34 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:34.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:34 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:36:35.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:34 smithi132 ceph-mon[35283]: pgmap v760: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.6 MiB/s wr, 146 op/s 2024-05-27T07:36:35.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:34 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:36:35.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:34 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:36:35.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:34 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:36:35.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:34 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:36:35.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:34 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:36:35.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.504+0000 7f6d66200700 1 -- 172.21.15.43:0/219216442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d60109680 msgr2=0x7f6d60110fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.504+0000 7f6d66200700 1 --2- 172.21.15.43:0/219216442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d60109680 0x7f6d60110fe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000b0a0 tx=0x7f6d5000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.504+0000 7f6d66200700 1 -- 172.21.15.43:0/219216442 shutdown_connections 2024-05-27T07:36:35.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.504+0000 7f6d66200700 1 --2- 172.21.15.43:0/219216442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d60109680 0x7f6d60110fe0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.504+0000 7f6d66200700 1 --2- 172.21.15.43:0/219216442 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d60104400 0x7f6d60104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.504+0000 7f6d66200700 1 --2- 172.21.15.43:0/219216442 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d60108d30 0x7f6d60109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.504+0000 7f6d66200700 1 -- 172.21.15.43:0/219216442 >> 172.21.15.43:0/219216442 conn(0x7f6d60079d40 msgr2=0x7f6d6007a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.505+0000 7f6d66200700 1 -- 172.21.15.43:0/219216442 shutdown_connections 2024-05-27T07:36:35.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.505+0000 7f6d66200700 1 -- 172.21.15.43:0/219216442 wait complete. 2024-05-27T07:36:35.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.506+0000 7f6d66200700 1 Processor -- start 2024-05-27T07:36:35.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.507+0000 7f6d66200700 1 -- start start 2024-05-27T07:36:35.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.507+0000 7f6d66200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d60104400 0x7f6d601a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.507+0000 7f6d66200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d60108d30 0x7f6d601a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.507+0000 7f6d66200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d60109680 0x7f6d601a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.508+0000 7f6d66200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6d60111fc0 con 0x7f6d60104400 2024-05-27T07:36:35.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.508+0000 7f6d66200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6d60111e40 con 0x7f6d60109680 2024-05-27T07:36:35.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.508+0000 7f6d66200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6d60112140 con 0x7f6d60108d30 2024-05-27T07:36:35.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.508+0000 7f6d65800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d60109680 0x7f6d601a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.508+0000 7f6d65800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d60109680 0x7f6d601a59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:44660/0 (socket says 172.21.15.43:44660) 2024-05-27T07:36:35.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.508+0000 7f6d65800700 1 -- 172.21.15.43:0/3290489324 learned_addr learned my addr 172.21.15.43:0/3290489324 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:35.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.508+0000 7f6d5fe00700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d60108d30 0x7f6d601a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.509+0000 7f6d64e00700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d60104400 0x7f6d601a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.509+0000 7f6d65800700 1 -- 172.21.15.43:0/3290489324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d60108d30 msgr2=0x7f6d601a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.509+0000 7f6d65800700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d60108d30 0x7f6d601a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.509+0000 7f6d65800700 1 -- 172.21.15.43:0/3290489324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d60104400 msgr2=0x7f6d601a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.509+0000 7f6d65800700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d60104400 0x7f6d601a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.509+0000 7f6d65800700 1 -- 172.21.15.43:0/3290489324 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d601a6050 con 0x7f6d60109680 2024-05-27T07:36:35.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.510+0000 7f6d65800700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d60109680 0x7f6d601a59b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000b070 tx=0x7f6d50004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.510+0000 7f6d5d600700 1 -- 172.21.15.43:0/3290489324 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50021070 con 0x7f6d60109680 2024-05-27T07:36:35.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.510+0000 7f6d5d600700 1 -- 172.21.15.43:0/3290489324 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d50003c00 con 0x7f6d60109680 2024-05-27T07:36:35.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.510+0000 7f6d66200700 1 -- 172.21.15.43:0/3290489324 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d601a62e0 con 0x7f6d60109680 2024-05-27T07:36:35.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.510+0000 7f6d5d600700 1 -- 172.21.15.43:0/3290489324 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50004560 con 0x7f6d60109680 2024-05-27T07:36:35.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.510+0000 7f6d66200700 1 -- 172.21.15.43:0/3290489324 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d601adc60 con 0x7f6d60109680 2024-05-27T07:36:35.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.516+0000 7f6d5d600700 1 -- 172.21.15.43:0/3290489324 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6d50004700 con 0x7f6d60109680 2024-05-27T07:36:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.518+0000 7f6d5d600700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6d440b2b20 0x7f6d440b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.519+0000 7f6d5d600700 1 -- 172.21.15.43:0/3290489324 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7f6d5000f030 con 0x7f6d60109680 2024-05-27T07:36:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.519+0000 7f6d64e00700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6d440b2b20 0x7f6d440b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.559+0000 7f6d64e00700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6d440b2b20 0x7f6d440b5030 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6d58005950 tx=0x7f6d5800b410 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.562+0000 7f6d762b1b80 1 Processor -- start 2024-05-27T07:36:35.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.563+0000 7f6d762b1b80 1 -- start start 2024-05-27T07:36:35.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.563+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3bfe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.563+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3a7eec0 0x5639b3bfed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.563+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3c08340 0x5639b3c0a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.563+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5639b3b2bec0 con 0x5639b3a7eec0 2024-05-27T07:36:35.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.563+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5639b3b1ba60 con 0x5639b3c08340 2024-05-27T07:36:35.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.563+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5639b3b1faa0 con 0x5639b3b6aa70 2024-05-27T07:36:35.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.564+0000 7f6d65800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3c08340 0x5639b3c0a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.564+0000 7f6d65800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3c08340 0x5639b3c0a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:44672/0 (socket says 172.21.15.43:44672) 2024-05-27T07:36:35.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.564+0000 7f6d65800700 1 -- 172.21.15.43:0/57133268 learned_addr learned my addr 172.21.15.43:0/57133268 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:35.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.564+0000 7f6d64e00700 1 --2- 172.21.15.43:0/57133268 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3bfe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.564+0000 7f6d5fe00700 1 --2- 172.21.15.43:0/57133268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3a7eec0 0x5639b3bfed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.565+0000 7f6d65800700 1 -- 172.21.15.43:0/57133268 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 msgr2=0x5639b3bfe7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.565+0000 7f6d65800700 1 --2- 172.21.15.43:0/57133268 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3bfe7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.565+0000 7f6d65800700 1 -- 172.21.15.43:0/57133268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3a7eec0 msgr2=0x5639b3bfed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.565+0000 7f6d65800700 1 --2- 172.21.15.43:0/57133268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3a7eec0 0x5639b3bfed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.565+0000 7f6d65800700 1 -- 172.21.15.43:0/57133268 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639b3ae44a0 con 0x5639b3c08340 2024-05-27T07:36:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.565+0000 7f6d65800700 1 --2- 172.21.15.43:0/57133268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3c08340 0x5639b3c0a760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6d500a77b0 tx=0x7f6d5000b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f268e2bfbe5d964a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.566+0000 7f6d56000700 1 -- 172.21.15.43:0/57133268 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500ab240 con 0x5639b3c08340 2024-05-27T07:36:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.566+0000 7f6d56000700 1 -- 172.21.15.43:0/57133268 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d500a5e50 con 0x5639b3c08340 2024-05-27T07:36:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.566+0000 7f6d56000700 1 -- 172.21.15.43:0/57133268 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500ab240 con 0x5639b3c08340 2024-05-27T07:36:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.567+0000 7f6d762b1b80 1 -- 172.21.15.43:0/57133268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3c08340 msgr2=0x5639b3c0a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.567+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/57133268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3c08340 0x5639b3c0a760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6d500a77b0 tx=0x7f6d5000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.567+0000 7f6d762b1b80 1 -- 172.21.15.43:0/57133268 shutdown_connections 2024-05-27T07:36:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.567+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/57133268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3c08340 0x5639b3c0a760 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.567+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/57133268 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3bfe7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.567+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/57133268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3a7eec0 0x5639b3bfed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.567+0000 7f6d762b1b80 1 -- 172.21.15.43:0/57133268 >> 172.21.15.43:0/57133268 conn(0x5639b3b18410 msgr2=0x5639b3bf9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.568+0000 7f6d762b1b80 1 -- 172.21.15.43:0/57133268 shutdown_connections 2024-05-27T07:36:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.568+0000 7f6d762b1b80 1 -- 172.21.15.43:0/57133268 wait complete. 2024-05-27T07:36:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.568+0000 7f6d762b1b80 1 -- start start 2024-05-27T07:36:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.569+0000 7f6d762b1b80 1 Processor -- start 2024-05-27T07:36:35.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.569+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.569+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3c05810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.569+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c05d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.569+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5639b3a22020 con 0x5639b3c08340 2024-05-27T07:36:35.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.569+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5639b3abc7e0 con 0x5639b3a7eec0 2024-05-27T07:36:35.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.569+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5639b3c0aca0 con 0x5639b3b6aa70 2024-05-27T07:36:35.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.569+0000 7f6d762b1b80 10 client.0 _ll_register_callbacks cb 0x5639b3ac5cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:35.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.570+0000 7f6d64e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.570+0000 7f6d64e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c052d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:44686/0 (socket says 172.21.15.43:44686) 2024-05-27T07:36:35.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.570+0000 7f6d64e00700 1 -- 172.21.15.43:0/3072437082 learned_addr learned my addr 172.21.15.43:0/3072437082 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:35.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.570+0000 7f6d65800700 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c05d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.570+0000 7f6d5fe00700 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3c05810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.571+0000 7f6d65800700 1 -- 172.21.15.43:0/3072437082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 msgr2=0x5639b3c05810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.571+0000 7f6d65800700 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3c05810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.571+0000 7f6d65800700 1 -- 172.21.15.43:0/3072437082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 msgr2=0x5639b3c052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.571+0000 7f6d65800700 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.571+0000 7f6d65800700 1 -- 172.21.15.43:0/3072437082 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639b3a6b950 con 0x5639b3c08340 2024-05-27T07:36:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.571+0000 7f6d64e00700 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:36:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.572+0000 7f6d65800700 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c05d50 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f6d50004fd0 tx=0x7f6d50004020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c24620d60e9137 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.572+0000 7f6d4ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:36:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.572+0000 7f6d4ea00700 1 -- 172.21.15.43:0/3072437082 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500ab240 con 0x5639b3c08340 2024-05-27T07:36:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.572+0000 7f6d4ea00700 1 -- 172.21.15.43:0/3072437082 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7f6d500e45b0 con 0x5639b3c08340 2024-05-27T07:36:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.573+0000 7f6d762b1b80 1 -- 172.21.15.43:0/3072437082 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d500a5e50 con 0x5639b3c08340 2024-05-27T07:36:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.573+0000 7f6d4ea00700 1 -- 172.21.15.43:0/3072437082 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f6d500afae0 con 0x5639b3c08340 2024-05-27T07:36:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.573+0000 7f6d762b1b80 10 client.18333 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.573+0000 7f6d762b1b80 1 -- 172.21.15.43:0/3072437082 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x5639b3aec0e0 con 0x5639b3c08340 2024-05-27T07:36:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.574+0000 7f6d4ea00700 1 -- 172.21.15.43:0/3072437082 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d500a5e50 con 0x5639b3c08340 2024-05-27T07:36:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.575+0000 7f6d4ea00700 1 -- 172.21.15.43:0/3072437082 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f6d5002d060 con 0x5639b3c08340 2024-05-27T07:36:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.575+0000 7f6d762b1b80 10 client.18333 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.575+0000 7f6d762b1b80 10 client.18333 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.575+0000 7f6d762b1b80 20 client.18333 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.575+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5639b3c17910 0x5639b3c19d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.575+0000 7f6d762b1b80 4 client.18333 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.575+0000 7f6d762b1b80 1 -- 172.21.15.43:0/3072437082 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639b3b2de10 con 0x5639b3c17910 2024-05-27T07:36:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.576+0000 7f6d64e00700 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5639b3c17910 0x5639b3c19d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.577+0000 7f6d4ea00700 10 client.18333 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:35.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.577+0000 7f6d64e00700 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5639b3c17910 0x5639b3c19d30 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=604c71fa746d8b49 server_cookie=1d30a535549a818f in_seq=0 out_seq=0 2024-05-27T07:36:35.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.579+0000 7f6d4ea00700 1 -- 172.21.15.43:0/3072437082 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5639b3b2de10 con 0x5639b3c17910 2024-05-27T07:36:35.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.579+0000 7f6d4ea00700 10 client.18333 handle_command_reply: tid=0 2024-05-27T07:36:35.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.579+0000 7f6d762b1b80 1 client.18333 shutdown 2024-05-27T07:36:35.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.579+0000 7f6d762b1b80 1 -- 172.21.15.43:0/3072437082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 msgr2=0x5639b3c05d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.579+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c05d50 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f6d50004fd0 tx=0x7f6d50004020 comp rx=0 tx=0).stop 2024-05-27T07:36:35.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.580+0000 7f6d762b1b80 1 -- 172.21.15.43:0/3072437082 shutdown_connections 2024-05-27T07:36:35.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.580+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5639b3c17910 0x5639b3c19d30 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.580+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c05d50 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.580+0000 7f6d4ea00700 0 client.18333 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:35.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.580+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.580+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/3072437082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3c05810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.580+0000 7f6d762b1b80 1 -- 172.21.15.43:0/3072437082 >> 172.21.15.43:0/3072437082 conn(0x5639b3b18410 msgr2=0x5639b3bf9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.580+0000 7f6d762b1b80 1 -- 172.21.15.43:0/3072437082 shutdown_connections 2024-05-27T07:36:35.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.580+0000 7f6d762b1b80 1 -- 172.21.15.43:0/3072437082 wait complete. 2024-05-27T07:36:35.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.581+0000 7f6d762b1b80 20 client.18333 trim_cache size 0 max 16384 2024-05-27T07:36:35.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.620+0000 7f6d762b1b80 1 Processor -- start 2024-05-27T07:36:35.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.620+0000 7f6d762b1b80 1 -- start start 2024-05-27T07:36:35.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.621+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3a7eec0 0x5639b3bf97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.621+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3b6aa70 0x5639b3c02530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.621+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c02a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.621+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5639b3bfeab0 con 0x5639b3c08340 2024-05-27T07:36:35.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.621+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5639b3c04450 con 0x5639b3b6aa70 2024-05-27T07:36:35.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.621+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6d601b6da0 con 0x5639b3a7eec0 2024-05-27T07:36:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.621+0000 7f6d65800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c02a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.621+0000 7f6d5fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3b6aa70 0x5639b3c02530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.621+0000 7f6d65800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c02a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60900/0 (socket says 172.21.15.43:60900) 2024-05-27T07:36:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.621+0000 7f6d65800700 1 -- 172.21.15.43:0/4250601302 learned_addr learned my addr 172.21.15.43:0/4250601302 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.622+0000 7f6d64e00700 1 --2- 172.21.15.43:0/4250601302 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3a7eec0 0x5639b3bf97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.622+0000 7f6d65800700 1 -- 172.21.15.43:0/4250601302 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3a7eec0 msgr2=0x5639b3bf97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.622+0000 7f6d65800700 1 --2- 172.21.15.43:0/4250601302 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3a7eec0 0x5639b3bf97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.622+0000 7f6d65800700 1 -- 172.21.15.43:0/4250601302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3b6aa70 msgr2=0x5639b3c02530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.622+0000 7f6d65800700 1 --2- 172.21.15.43:0/4250601302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3b6aa70 0x5639b3c02530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.622+0000 7f6d65800700 1 -- 172.21.15.43:0/4250601302 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639b3c03840 con 0x5639b3c08340 2024-05-27T07:36:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.622+0000 7f6d65800700 1 --2- 172.21.15.43:0/4250601302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c02a70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f6d500a89a0 tx=0x7f6d5000b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928cf25b7cc542d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.623+0000 7f6d55600700 1 -- 172.21.15.43:0/4250601302 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500abe30 con 0x5639b3c08340 2024-05-27T07:36:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.623+0000 7f6d55600700 1 -- 172.21.15.43:0/4250601302 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d500a5e50 con 0x5639b3c08340 2024-05-27T07:36:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.623+0000 7f6d55600700 1 -- 172.21.15.43:0/4250601302 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500b4070 con 0x5639b3c08340 2024-05-27T07:36:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.625+0000 7f6d762b1b80 1 -- 172.21.15.43:0/4250601302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 msgr2=0x5639b3c02a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.625+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/4250601302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c02a70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f6d500a89a0 tx=0x7f6d5000b3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.625+0000 7f6d762b1b80 1 -- 172.21.15.43:0/4250601302 shutdown_connections 2024-05-27T07:36:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.625+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/4250601302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c02a70 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.625+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/4250601302 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3a7eec0 0x5639b3bf97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.625+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/4250601302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3b6aa70 0x5639b3c02530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.625+0000 7f6d762b1b80 1 -- 172.21.15.43:0/4250601302 >> 172.21.15.43:0/4250601302 conn(0x5639b3b18410 msgr2=0x5639b3c00860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.626+0000 7f6d762b1b80 1 -- 172.21.15.43:0/4250601302 shutdown_connections 2024-05-27T07:36:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.627+0000 7f6d762b1b80 1 -- 172.21.15.43:0/4250601302 wait complete. 2024-05-27T07:36:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.628+0000 7f6d762b1b80 1 -- start start 2024-05-27T07:36:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.628+0000 7f6d762b1b80 1 Processor -- start 2024-05-27T07:36:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.629+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c1abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.629+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3c1b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.629+0000 7f6d762b1b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c1b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.629+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6d601b66d0 con 0x5639b3c08340 2024-05-27T07:36:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.629+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6d601b5200 con 0x5639b3a7eec0 2024-05-27T07:36:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.629+0000 7f6d762b1b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6d601b5930 con 0x5639b3b6aa70 2024-05-27T07:36:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.629+0000 7f6d762b1b80 10 client.0 _ll_register_callbacks cb 0x5639b3ac5cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.630+0000 7f6d64e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c1abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.630+0000 7f6d64e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c1abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:44696/0 (socket says 172.21.15.43:44696) 2024-05-27T07:36:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.630+0000 7f6d64e00700 1 -- 172.21.15.43:0/489015960 learned_addr learned my addr 172.21.15.43:0/489015960 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.630+0000 7f6d65800700 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c1b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.630+0000 7f6d5fe00700 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3c1b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.631+0000 7f6d65800700 1 -- 172.21.15.43:0/489015960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 msgr2=0x5639b3c1b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.631+0000 7f6d65800700 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3c1b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.631+0000 7f6d65800700 1 -- 172.21.15.43:0/489015960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 msgr2=0x5639b3c1abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.631+0000 7f6d65800700 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c1abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.631+0000 7f6d65800700 1 -- 172.21.15.43:0/489015960 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639b3c81230 con 0x5639b3c08340 2024-05-27T07:36:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.631+0000 7f6d64e00700 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c1abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.632+0000 7f6d65800700 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c1b670 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f6d500ab8f0 tx=0x7f6d500adf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d2ee4fc8ee9bd9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.633+0000 7f6d4ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:36:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.633+0000 7f6d4ea00700 1 -- 172.21.15.43:0/489015960 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500ab730 con 0x5639b3c08340 2024-05-27T07:36:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.633+0000 7f6d762b1b80 1 -- 172.21.15.43:0/489015960 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d500a5e50 con 0x5639b3c08340 2024-05-27T07:36:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.634+0000 7f6d4ea00700 1 -- 172.21.15.43:0/489015960 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7f6d500a7d20 con 0x5639b3c08340 2024-05-27T07:36:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.634+0000 7f6d4ea00700 1 -- 172.21.15.43:0/489015960 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f6d500ab730 con 0x5639b3c08340 2024-05-27T07:36:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.634+0000 7f6d762b1b80 10 client.18339 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.634+0000 7f6d762b1b80 1 -- 172.21.15.43:0/489015960 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x5639b3c81230 con 0x5639b3c08340 2024-05-27T07:36:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.635+0000 7f6d4ea00700 1 -- 172.21.15.43:0/489015960 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d500a5e50 con 0x5639b3c08340 2024-05-27T07:36:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.635+0000 7f6d4ea00700 1 -- 172.21.15.43:0/489015960 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f6d50011070 con 0x5639b3c08340 2024-05-27T07:36:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.635+0000 7f6d762b1b80 10 client.18339 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.635+0000 7f6d762b1b80 10 client.18339 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.635+0000 7f6d762b1b80 20 client.18339 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.635+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5639b3c82760 0x5639b3c84b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.635+0000 7f6d762b1b80 4 client.18339 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.635+0000 7f6d762b1b80 1 -- 172.21.15.43:0/489015960 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5639b3b275f0 con 0x5639b3c82760 2024-05-27T07:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.636+0000 7f6d64e00700 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5639b3c82760 0x5639b3c84b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.637+0000 7f6d64e00700 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5639b3c82760 0x5639b3c84b80 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71a549d7ee929d41 server_cookie=dfb0ada211efce6a in_seq=0 out_seq=0 2024-05-27T07:36:35.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.637+0000 7f6d4ea00700 10 client.18339 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.639+0000 7f6d4ea00700 1 -- 172.21.15.43:0/489015960 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5639b3b275f0 con 0x5639b3c82760 2024-05-27T07:36:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.639+0000 7f6d4ea00700 10 client.18339 handle_command_reply: tid=0 2024-05-27T07:36:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.639+0000 7f6d762b1b80 1 client.18339 shutdown 2024-05-27T07:36:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d762b1b80 1 -- 172.21.15.43:0/489015960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 msgr2=0x5639b3c1b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c1b670 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f6d500ab8f0 tx=0x7f6d500adf60 comp rx=0 tx=0).stop 2024-05-27T07:36:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d762b1b80 1 -- 172.21.15.43:0/489015960 shutdown_connections 2024-05-27T07:36:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5639b3c82760 0x5639b3c84b80 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639b3c08340 0x5639b3c1b670 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5639b3a7eec0 0x5639b3c1abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d762b1b80 1 --2- 172.21.15.43:0/489015960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5639b3b6aa70 0x5639b3c1b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d762b1b80 1 -- 172.21.15.43:0/489015960 >> 172.21.15.43:0/489015960 conn(0x5639b3b18410 msgr2=0x5639b3c00860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d4ea00700 0 client.18339 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d762b1b80 1 -- 172.21.15.43:0/489015960 shutdown_connections 2024-05-27T07:36:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.640+0000 7f6d762b1b80 1 -- 172.21.15.43:0/489015960 wait complete. 2024-05-27T07:36:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.641+0000 7f6d762b1b80 20 client.18339 trim_cache size 0 max 16384 2024-05-27T07:36:35.638 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:36:35.638 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:36:35.638 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "c6ba073b-1f8b-42ce-a3bc-d0e506326d24", 2024-05-27T07:36:35.638 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:36:35.638 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:36:35.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.642+0000 7f6d66200700 1 -- 172.21.15.43:0/3290489324 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6d440b2b20 msgr2=0x7f6d440b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.642+0000 7f6d66200700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6d440b2b20 0x7f6d440b5030 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6d58005950 tx=0x7f6d5800b410 comp rx=0 tx=0).stop 2024-05-27T07:36:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.642+0000 7f6d66200700 1 -- 172.21.15.43:0/3290489324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d60109680 msgr2=0x7f6d601a59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.642+0000 7f6d66200700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d60109680 0x7f6d601a59b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000b070 tx=0x7f6d50004d00 comp rx=0 tx=0).stop 2024-05-27T07:36:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.643+0000 7f6d66200700 1 -- 172.21.15.43:0/3290489324 shutdown_connections 2024-05-27T07:36:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.643+0000 7f6d66200700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6d440b2b20 0x7f6d440b5030 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.643+0000 7f6d66200700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d60109680 0x7f6d601a59b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.643+0000 7f6d66200700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6d60104400 0x7f6d601a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.643+0000 7f6d66200700 1 --2- 172.21.15.43:0/3290489324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d60108d30 0x7f6d601a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.643+0000 7f6d66200700 1 -- 172.21.15.43:0/3290489324 >> 172.21.15.43:0/3290489324 conn(0x7f6d60079d40 msgr2=0x7f6d60102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.644+0000 7f6d66200700 1 -- 172.21.15.43:0/3290489324 shutdown_connections 2024-05-27T07:36:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:35.644+0000 7f6d66200700 1 -- 172.21.15.43:0/3290489324 wait complete. 2024-05-27T07:36:35.820 INFO:tasks.workunit.client.0.smithi043.stdout: 10 56574 5.85 MB/sec execute 396 sec latency 164.930 ms 2024-05-27T07:36:35.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:35 smithi043 ceph-mon[31531]: pgmap v761: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 4.0 MiB/s wr, 160 op/s 2024-05-27T07:36:35.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:35 smithi043 ceph-mon[31531]: pgmap v762: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.9 MiB/s wr, 197 op/s 2024-05-27T07:36:35.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:35 smithi179 ceph-mon[35888]: pgmap v761: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 4.0 MiB/s wr, 160 op/s 2024-05-27T07:36:35.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:35 smithi179 ceph-mon[35888]: pgmap v762: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.9 MiB/s wr, 197 op/s 2024-05-27T07:36:36.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:35 smithi132 ceph-mon[35283]: pgmap v761: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 4.0 MiB/s wr, 160 op/s 2024-05-27T07:36:36.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:35 smithi132 ceph-mon[35283]: pgmap v762: 129 pgs: 129 active+clean; 1.2 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 4.9 MiB/s wr, 197 op/s 2024-05-27T07:36:36.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:36:36.821 INFO:tasks.workunit.client.0.smithi043.stdout: 10 56702 5.86 MB/sec execute 397 sec latency 101.672 ms 2024-05-27T07:36:36.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:36:36.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:36:36.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:36:36.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:36:36.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:36:36.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:36:36.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:36:36.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:36 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:36:36.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:36:36.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:36:36.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:36:36.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:36:36.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:36:36.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:36:36.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:36.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:36 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:37.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:36:37.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:37.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:36:37.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:36:37.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:36:37.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:36:37.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:36:37.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:37.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:36:37.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:37.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:37.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:37.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:37.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:36 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:36:37.439 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:36:37.821 INFO:tasks.workunit.client.0.smithi043.stdout: 10 56796 5.85 MB/sec execute 398 sec latency 106.203 ms 2024-05-27T07:36:37.941 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:37 smithi043 ceph-mon[31531]: pgmap v763: 129 pgs: 129 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.2 MiB/s wr, 204 op/s 2024-05-27T07:36:37.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:37 smithi179 ceph-mon[35888]: pgmap v763: 129 pgs: 129 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.2 MiB/s wr, 204 op/s 2024-05-27T07:36:38.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:37 smithi132 ceph-mon[35283]: pgmap v763: 129 pgs: 129 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.2 MiB/s wr, 204 op/s 2024-05-27T07:36:38.821 INFO:tasks.workunit.client.0.smithi043.stdout: 10 56926 5.85 MB/sec execute 399 sec latency 183.568 ms 2024-05-27T07:36:39.821 INFO:tasks.workunit.client.0.smithi043.stdout: 10 57008 5.84 MB/sec execute 400 sec latency 108.592 ms 2024-05-27T07:36:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.865+0000 7f1e7e200700 1 -- 172.21.15.43:0/323398728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e78076440 msgr2=0x7f1e781106b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.865+0000 7f1e7e200700 1 --2- 172.21.15.43:0/323398728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e78076440 0x7f1e781106b0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400b0a0 tx=0x7f1e6400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.866+0000 7f1e7e200700 1 -- 172.21.15.43:0/323398728 shutdown_connections 2024-05-27T07:36:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.866+0000 7f1e7e200700 1 --2- 172.21.15.43:0/323398728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e78110bf0 0x7f1e78113010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.866+0000 7f1e7e200700 1 --2- 172.21.15.43:0/323398728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e78075af0 0x7f1e78075f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.866+0000 7f1e7e200700 1 --2- 172.21.15.43:0/323398728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e78076440 0x7f1e781106b0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.866+0000 7f1e7e200700 1 -- 172.21.15.43:0/323398728 >> 172.21.15.43:0/323398728 conn(0x7f1e78100030 msgr2=0x7f1e78102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.866+0000 7f1e7e200700 1 -- 172.21.15.43:0/323398728 shutdown_connections 2024-05-27T07:36:39.863 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:39 smithi043 ceph-mon[31531]: pgmap v764: 129 pgs: 129 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.2 MiB/s wr, 204 op/s 2024-05-27T07:36:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.866+0000 7f1e7e200700 1 -- 172.21.15.43:0/323398728 wait complete. 2024-05-27T07:36:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.867+0000 7f1e7e200700 1 Processor -- start 2024-05-27T07:36:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.868+0000 7f1e7e200700 1 -- start start 2024-05-27T07:36:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.868+0000 7f1e7e200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e78075af0 0x7f1e7819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.868+0000 7f1e7e200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e78076440 0x7f1e7819d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.868+0000 7f1e7e200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e78110bf0 0x7f1e781a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.868+0000 7f1e7e200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1e781165b0 con 0x7f1e78076440 2024-05-27T07:36:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.868+0000 7f1e7e200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1e78116430 con 0x7f1e78075af0 2024-05-27T07:36:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.868+0000 7f1e7e200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1e78116730 con 0x7f1e78110bf0 2024-05-27T07:36:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.869+0000 7f1e76a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e78076440 0x7f1e7819d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.869+0000 7f1e76a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e78076440 0x7f1e7819d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60928/0 (socket says 172.21.15.43:60928) 2024-05-27T07:36:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.869+0000 7f1e76a00700 1 -- 172.21.15.43:0/3342973349 learned_addr learned my addr 172.21.15.43:0/3342973349 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.869+0000 7f1e77400700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e78075af0 0x7f1e7819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.869+0000 7f1e77e00700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e78110bf0 0x7f1e781a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.870+0000 7f1e76a00700 1 -- 172.21.15.43:0/3342973349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e78110bf0 msgr2=0x7f1e781a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.870+0000 7f1e76a00700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e78110bf0 0x7f1e781a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.870+0000 7f1e76a00700 1 -- 172.21.15.43:0/3342973349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e78075af0 msgr2=0x7f1e7819cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.870+0000 7f1e76a00700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e78075af0 0x7f1e7819cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.870+0000 7f1e76a00700 1 -- 172.21.15.43:0/3342973349 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e781a1e90 con 0x7f1e78076440 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.870+0000 7f1e76a00700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e78076440 0x7f1e7819d560 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400b780 tx=0x7f1e64004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.871+0000 7f1e6fe00700 1 -- 172.21.15.43:0/3342973349 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64026070 con 0x7f1e78076440 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.871+0000 7f1e6fe00700 1 -- 172.21.15.43:0/3342973349 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e64007c00 con 0x7f1e78076440 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.871+0000 7f1e6fe00700 1 -- 172.21.15.43:0/3342973349 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e640058f0 con 0x7f1e78076440 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.871+0000 7f1e7e200700 1 -- 172.21.15.43:0/3342973349 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e781a2120 con 0x7f1e78076440 2024-05-27T07:36:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.871+0000 7f1e7e200700 1 -- 172.21.15.43:0/3342973349 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e781a2550 con 0x7f1e78076440 2024-05-27T07:36:39.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.877+0000 7f1e6fe00700 1 -- 172.21.15.43:0/3342973349 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f1e64031020 con 0x7f1e78076440 2024-05-27T07:36:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.877+0000 7f1e8cd7eb80 1 Processor -- start 2024-05-27T07:36:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.878+0000 7f1e8cd7eb80 1 -- start start 2024-05-27T07:36:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.878+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd143b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.878+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd12bce40 0x55dfd143b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.878+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd143be70 0x55dfd1448110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.878+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dfd1369e40 con 0x55dfd143be70 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.878+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dfd13599e0 con 0x55dfd13a89f0 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.878+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55dfd135da20 con 0x55dfd12bce40 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.879+0000 7f1e77e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd143b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.879+0000 7f1e77e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd143b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:44732/0 (socket says 172.21.15.43:44732) 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.879+0000 7f1e77e00700 1 -- 172.21.15.43:0/1853351841 learned_addr learned my addr 172.21.15.43:0/1853351841 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.879+0000 7f1e76a00700 1 --2- 172.21.15.43:0/1853351841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd143be70 0x55dfd1448110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.879+0000 7f1e77400700 1 --2- 172.21.15.43:0/1853351841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd12bce40 0x55dfd143b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.879+0000 7f1e6fe00700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e5c0b2a70 0x7f1e5c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.879+0000 7f1e77e00700 1 -- 172.21.15.43:0/1853351841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd12bce40 msgr2=0x55dfd143b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.879+0000 7f1e77e00700 1 --2- 172.21.15.43:0/1853351841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd12bce40 0x55dfd143b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.880+0000 7f1e77e00700 1 -- 172.21.15.43:0/1853351841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd143be70 msgr2=0x55dfd1448110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.880+0000 7f1e77e00700 1 --2- 172.21.15.43:0/1853351841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd143be70 0x55dfd1448110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.880+0000 7f1e77e00700 1 -- 172.21.15.43:0/1853351841 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfd1322420 con 0x55dfd13a89f0 2024-05-27T07:36:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.880+0000 7f1e76a00700 1 --2- 172.21.15.43:0/1853351841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd143be70 0x55dfd1448110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.880+0000 7f1e6fe00700 1 -- 172.21.15.43:0/3342973349 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7f1e6400f020 con 0x7f1e78076440 2024-05-27T07:36:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.880+0000 7f1e77e00700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e5c0b2a70 0x7f1e5c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.880+0000 7f1e77e00700 1 --2- 172.21.15.43:0/1853351841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd143b330 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1e68005c50 tx=0x7f1e680106b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8765c0c5b5112004 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:39.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.881+0000 7f1e77e00700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e5c0b2a70 0x7f1e5c0b4f80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1e68021000 tx=0x7f1e6801e000 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.881+0000 7f1e6cc00700 1 -- 172.21.15.43:0/1853351841 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68009e40 con 0x55dfd13a89f0 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.881+0000 7f1e6cc00700 1 -- 172.21.15.43:0/1853351841 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e68011440 con 0x55dfd13a89f0 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.881+0000 7f1e6cc00700 1 -- 172.21.15.43:0/1853351841 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6801d600 con 0x55dfd13a89f0 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.882+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1853351841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 msgr2=0x55dfd143b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.882+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1853351841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd143b330 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1e68005c50 tx=0x7f1e680106b0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.882+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1853351841 shutdown_connections 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.882+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1853351841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd143be70 0x55dfd1448110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.882+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1853351841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd143b330 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.882+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1853351841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd12bce40 0x55dfd143b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.882+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1853351841 >> 172.21.15.43:0/1853351841 conn(0x55dfd1356390 msgr2=0x55dfd14374c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.883+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1853351841 shutdown_connections 2024-05-27T07:36:39.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.883+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1853351841 wait complete. 2024-05-27T07:36:39.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.883+0000 7f1e8cd7eb80 1 -- start start 2024-05-27T07:36:39.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.884+0000 7f1e8cd7eb80 1 Processor -- start 2024-05-27T07:36:39.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.884+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 0x55dfd1445680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.884+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd1445bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.884+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 0x55dfd1446100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.884+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dfd125ffa0 con 0x55dfd12bce40 2024-05-27T07:36:39.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.884+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dfd12fa760 con 0x55dfd13a89f0 2024-05-27T07:36:39.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.884+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55dfd1448b80 con 0x55dfd143be70 2024-05-27T07:36:39.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.884+0000 7f1e8cd7eb80 10 client.0 _ll_register_callbacks cb 0x55dfd1303c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.885+0000 7f1e77400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 0x55dfd1445680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.885+0000 7f1e77e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 0x55dfd1446100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.885+0000 7f1e77e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 0x55dfd1446100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:46948/0 (socket says 172.21.15.43:46948) 2024-05-27T07:36:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.885+0000 7f1e77e00700 1 -- 172.21.15.43:0/1391364223 learned_addr learned my addr 172.21.15.43:0/1391364223 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.885+0000 7f1e76a00700 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd1445bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.886+0000 7f1e77e00700 1 -- 172.21.15.43:0/1391364223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 msgr2=0x55dfd1445bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.886+0000 7f1e77e00700 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd1445bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.886+0000 7f1e77e00700 1 -- 172.21.15.43:0/1391364223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 msgr2=0x55dfd1445680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.886+0000 7f1e77e00700 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 0x55dfd1445680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.886+0000 7f1e77e00700 1 -- 172.21.15.43:0/1391364223 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfd12a98d0 con 0x55dfd143be70 2024-05-27T07:36:39.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.886+0000 7f1e76a00700 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd1445bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:36:39.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.886+0000 7f1e77e00700 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 0x55dfd1446100 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1e68011f80 tx=0x7f1e68010d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c054963ae1675d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:39.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.886+0000 7f1e61600700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:36:39.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.887+0000 7f1e61600700 1 -- 172.21.15.43:0/1391364223 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e680115f0 con 0x55dfd143be70 2024-05-27T07:36:39.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.887+0000 7f1e61600700 1 -- 172.21.15.43:0/1391364223 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7f1e68011790 con 0x55dfd143be70 2024-05-27T07:36:39.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.887+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1391364223 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e68011440 con 0x55dfd143be70 2024-05-27T07:36:39.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.888+0000 7f1e61600700 1 -- 172.21.15.43:0/1391364223 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f1e68037050 con 0x55dfd143be70 2024-05-27T07:36:39.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.888+0000 7f1e8cd7eb80 10 client.27769 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:39.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.888+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1391364223 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55dfd132a060 con 0x55dfd143be70 2024-05-27T07:36:39.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.889+0000 7f1e61600700 1 -- 172.21.15.43:0/1391364223 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e68011440 con 0x55dfd143be70 2024-05-27T07:36:39.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.889+0000 7f1e61600700 1 -- 172.21.15.43:0/1391364223 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f1e680324c0 con 0x55dfd143be70 2024-05-27T07:36:39.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.889+0000 7f1e8cd7eb80 10 client.27769 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:39.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.889+0000 7f1e8cd7eb80 10 client.27769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:39.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.889+0000 7f1e8cd7eb80 20 client.27769 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:39.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.889+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dfd1455c70 0x55dfd1458140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.889+0000 7f1e8cd7eb80 4 client.27769 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:36:39.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.889+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1391364223 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfd136bd90 con 0x55dfd1455c70 2024-05-27T07:36:39.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.890+0000 7f1e77400700 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dfd1455c70 0x55dfd1458140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.891+0000 7f1e77400700 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dfd1455c70 0x55dfd1458140 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b0944e2eeef662f server_cookie=ad31bbde0385a704 in_seq=0 out_seq=0 2024-05-27T07:36:39.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.891+0000 7f1e61600700 10 client.27769 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.892+0000 7f1e61600700 1 -- 172.21.15.43:0/1391364223 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55dfd136bd90 con 0x55dfd1455c70 2024-05-27T07:36:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.892+0000 7f1e61600700 10 client.27769 handle_command_reply: tid=0 2024-05-27T07:36:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.892+0000 7f1e8cd7eb80 1 client.27769 shutdown 2024-05-27T07:36:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.893+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1391364223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 msgr2=0x55dfd1446100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.893+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 0x55dfd1446100 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1e68011f80 tx=0x7f1e68010d30 comp rx=0 tx=0).stop 2024-05-27T07:36:39.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.893+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1391364223 shutdown_connections 2024-05-27T07:36:39.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.893+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 0x55dfd1446100 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.893+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dfd1455c70 0x55dfd1458140 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.893+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 0x55dfd1445680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.893+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1391364223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd13a89f0 0x55dfd1445bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.893+0000 7f1e61600700 0 client.27769 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:39.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.893+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1391364223 >> 172.21.15.43:0/1391364223 conn(0x55dfd1356390 msgr2=0x55dfd143b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:39.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.894+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1391364223 shutdown_connections 2024-05-27T07:36:39.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.894+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1391364223 wait complete. 2024-05-27T07:36:39.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.894+0000 7f1e8cd7eb80 20 client.27769 trim_cache size 0 max 16384 2024-05-27T07:36:39.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.932+0000 7f1e8cd7eb80 1 Processor -- start 2024-05-27T07:36:39.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.932+0000 7f1e8cd7eb80 1 -- start start 2024-05-27T07:36:39.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.932+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd12bce40 0x55dfd143f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.932+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd13a89f0 0x55dfd1451c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.932+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 0x55dfd145d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.932+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dfd1444800 con 0x55dfd13a89f0 2024-05-27T07:36:39.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.932+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55dfd1445100 con 0x55dfd12bce40 2024-05-27T07:36:39.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.932+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1e781ae9b0 con 0x55dfd143be70 2024-05-27T07:36:39.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.933+0000 7f1e76a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd13a89f0 0x55dfd1451c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.933+0000 7f1e76a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd13a89f0 0x55dfd1451c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60958/0 (socket says 172.21.15.43:60958) 2024-05-27T07:36:39.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.933+0000 7f1e76a00700 1 -- 172.21.15.43:0/1452723789 learned_addr learned my addr 172.21.15.43:0/1452723789 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:39.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.933+0000 7f1e77400700 1 --2- 172.21.15.43:0/1452723789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd12bce40 0x55dfd143f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.933+0000 7f1e76a00700 1 -- 172.21.15.43:0/1452723789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 msgr2=0x55dfd145d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:39.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.934+0000 7f1e76a00700 1 --2- 172.21.15.43:0/1452723789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 0x55dfd145d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.934+0000 7f1e76a00700 1 -- 172.21.15.43:0/1452723789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd12bce40 msgr2=0x55dfd143f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:36:39.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.934+0000 7f1e76a00700 1 --2- 172.21.15.43:0/1452723789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd12bce40 0x55dfd143f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.934+0000 7f1e76a00700 1 -- 172.21.15.43:0/1452723789 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfd1433240 con 0x55dfd13a89f0 2024-05-27T07:36:39.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.934+0000 7f1e76a00700 1 --2- 172.21.15.43:0/1452723789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd13a89f0 0x55dfd1451c20 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f1e640e8f00 tx=0x7f1e64007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c58c4c4be61d09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:39.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.935+0000 7f1e63e00700 1 -- 172.21.15.43:0/1452723789 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e640b2410 con 0x55dfd13a89f0 2024-05-27T07:36:39.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.935+0000 7f1e63e00700 1 -- 172.21.15.43:0/1452723789 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e640eee50 con 0x55dfd13a89f0 2024-05-27T07:36:39.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.935+0000 7f1e63e00700 1 -- 172.21.15.43:0/1452723789 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e640ae250 con 0x55dfd13a89f0 2024-05-27T07:36:39.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.937+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1452723789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd13a89f0 msgr2=0x55dfd1451c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:39.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.937+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1452723789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd13a89f0 0x55dfd1451c20 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f1e640e8f00 tx=0x7f1e64007e80 comp rx=0 tx=0).stop 2024-05-27T07:36:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.937+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1452723789 shutdown_connections 2024-05-27T07:36:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.937+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1452723789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd143be70 0x55dfd145d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.937+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1452723789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd12bce40 0x55dfd143f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.937+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/1452723789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd13a89f0 0x55dfd1451c20 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.937+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1452723789 >> 172.21.15.43:0/1452723789 conn(0x55dfd1356390 msgr2=0x55dfd143eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.937+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1452723789 shutdown_connections 2024-05-27T07:36:39.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.937+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/1452723789 wait complete. 2024-05-27T07:36:39.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.938+0000 7f1e8cd7eb80 1 -- start start 2024-05-27T07:36:39.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.938+0000 7f1e8cd7eb80 1 Processor -- start 2024-05-27T07:36:39.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.939+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 0x55dfd14b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.939+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd13a89f0 0x55dfd14b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.939+0000 7f1e8cd7eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd143be70 0x55dfd14493b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.939+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1e781ae2e0 con 0x55dfd12bce40 2024-05-27T07:36:39.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.939+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1e781ace10 con 0x55dfd143be70 2024-05-27T07:36:39.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.939+0000 7f1e8cd7eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1e781ad540 con 0x55dfd13a89f0 2024-05-27T07:36:39.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.939+0000 7f1e8cd7eb80 10 client.0 _ll_register_callbacks cb 0x55dfd1303c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:36:39.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.940+0000 7f1e77400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 0x55dfd14b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.940+0000 7f1e77400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 0x55dfd14b8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60972/0 (socket says 172.21.15.43:60972) 2024-05-27T07:36:39.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.940+0000 7f1e77400700 1 -- 172.21.15.43:0/4015167048 learned_addr learned my addr 172.21.15.43:0/4015167048 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:36:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.940+0000 7f1e77400700 1 -- 172.21.15.43:0/4015167048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd13a89f0 msgr2=0x55dfd14b9060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:36:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.940+0000 7f1e77400700 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd13a89f0 0x55dfd14b9060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.940+0000 7f1e77400700 1 -- 172.21.15.43:0/4015167048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd143be70 msgr2=0x55dfd14493b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:36:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.940+0000 7f1e77400700 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd143be70 0x55dfd14493b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.940+0000 7f1e77400700 1 -- 172.21.15.43:0/4015167048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfd14499f0 con 0x55dfd12bce40 2024-05-27T07:36:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.941+0000 7f1e77400700 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 0x55dfd14b8b20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f1e70000c00 tx=0x7f1e70012850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb4e0a2e5402e7e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:36:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.941+0000 7f1e61600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:36:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.944+0000 7f1e61600700 1 -- 172.21.15.43:0/4015167048 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70004440 con 0x55dfd12bce40 2024-05-27T07:36:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.944+0000 7f1e61600700 1 -- 172.21.15.43:0/4015167048 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(155..155 src has 1..155) ==== 9496+0+0 (secure 0 0 0) 0x7f1e70003680 con 0x55dfd12bce40 2024-05-27T07:36:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.944+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/4015167048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e640eee50 con 0x55dfd12bce40 2024-05-27T07:36:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.945+0000 7f1e61600700 1 -- 172.21.15.43:0/4015167048 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f1e70003880 con 0x55dfd12bce40 2024-05-27T07:36:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.945+0000 7f1e8cd7eb80 10 client.18357 fetch_fsmap learned FSMap version 144 2024-05-27T07:36:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.945+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/4015167048 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=4+}) -- 0x55dfd14c0260 con 0x55dfd12bce40 2024-05-27T07:36:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.946+0000 7f1e61600700 1 -- 172.21.15.43:0/4015167048 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1e640eee50 con 0x55dfd12bce40 2024-05-27T07:36:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.946+0000 7f1e61600700 1 -- 172.21.15.43:0/4015167048 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 144) ==== 4199+0+0 (secure 0 0 0) 0x7f1e70025020 con 0x55dfd12bce40 2024-05-27T07:36:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.946+0000 7f1e8cd7eb80 10 client.18357 fetch_fsmap finished waiting for FSMap version 144 2024-05-27T07:36:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.946+0000 7f1e8cd7eb80 10 client.18357 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:36:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.946+0000 7f1e8cd7eb80 20 client.18357 populate_metadata read hostname 'smithi043' 2024-05-27T07:36:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.946+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dfd14c08b0 0x55dfd14c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:36:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.946+0000 7f1e8cd7eb80 4 client.18357 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:36:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.946+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/4015167048 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dfd1365570 con 0x55dfd14c08b0 2024-05-27T07:36:39.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.949+0000 7f1e76a00700 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dfd14c08b0 0x55dfd14c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:36:39.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.952+0000 7f1e76a00700 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dfd14c08b0 0x55dfd14c2cd0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a00e7543aa322161 server_cookie=4acfee7507222028 in_seq=0 out_seq=0 2024-05-27T07:36:39.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.952+0000 7f1e61600700 10 client.18357 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:39.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.952+0000 7f1e61600700 1 -- 172.21.15.43:0/4015167048 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dfd1365570 con 0x55dfd14c08b0 2024-05-27T07:36:39.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.952+0000 7f1e61600700 10 client.18357 handle_command_reply: tid=0 2024-05-27T07:36:39.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.952+0000 7f1e8cd7eb80 1 client.18357 shutdown 2024-05-27T07:36:39.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.953+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/4015167048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 msgr2=0x55dfd14b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:39.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.953+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 0x55dfd14b8b20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f1e70000c00 tx=0x7f1e70012850 comp rx=0 tx=0).stop 2024-05-27T07:36:39.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.953+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/4015167048 shutdown_connections 2024-05-27T07:36:39.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.953+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dfd143be70 0x55dfd14493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.953+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55dfd14c08b0 0x55dfd14c2cd0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.953+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfd12bce40 0x55dfd14b8b20 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.953+0000 7f1e8cd7eb80 1 --2- 172.21.15.43:0/4015167048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55dfd13a89f0 0x55dfd14b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.953+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/4015167048 >> 172.21.15.43:0/4015167048 conn(0x55dfd1356390 msgr2=0x55dfd1443b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:39.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.953+0000 7f1e61600700 0 client.18357 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:36:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.955+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/4015167048 shutdown_connections 2024-05-27T07:36:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.955+0000 7f1e8cd7eb80 1 -- 172.21.15.43:0/4015167048 wait complete. 2024-05-27T07:36:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.956+0000 7f1e8cd7eb80 20 client.18357 trim_cache size 0 max 16384 2024-05-27T07:36:39.953 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:36:39.953 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:36:39.953 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:36:39.953 INFO:teuthology.orchestra.run.smithi043.stdout: "c6ba073b-1f8b-42ce-a3bc-d0e506326d24": { 2024-05-27T07:36:39.953 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:36:39.953 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "c6ba073b-1f8b-42ce-a3bc-d0e506326d24", 2024-05-27T07:36:39.953 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:36:39.953 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:36:39.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:36:39.954 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:36:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.958+0000 7f1e7e200700 1 -- 172.21.15.43:0/3342973349 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e5c0b2a70 msgr2=0x7f1e5c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.958+0000 7f1e7e200700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e5c0b2a70 0x7f1e5c0b4f80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1e68021000 tx=0x7f1e6801e000 comp rx=0 tx=0).stop 2024-05-27T07:36:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.958+0000 7f1e7e200700 1 -- 172.21.15.43:0/3342973349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e78076440 msgr2=0x7f1e7819d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:36:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.958+0000 7f1e7e200700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e78076440 0x7f1e7819d560 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400b780 tx=0x7f1e64004d00 comp rx=0 tx=0).stop 2024-05-27T07:36:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.958+0000 7f1e7e200700 1 -- 172.21.15.43:0/3342973349 shutdown_connections 2024-05-27T07:36:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.958+0000 7f1e7e200700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f1e5c0b2a70 0x7f1e5c0b4f80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.958+0000 7f1e7e200700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e78110bf0 0x7f1e781a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.958+0000 7f1e7e200700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1e78075af0 0x7f1e7819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:36:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.958+0000 7f1e7e200700 1 --2- 172.21.15.43:0/3342973349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e78076440 0x7f1e7819d560 secure :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400b780 tx=0x7f1e64004d00 comp rx=0 tx=0).stop 2024-05-27T07:36:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.958+0000 7f1e7e200700 1 -- 172.21.15.43:0/3342973349 >> 172.21.15.43:0/3342973349 conn(0x7f1e78100030 msgr2=0x7f1e78101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:36:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.959+0000 7f1e7e200700 1 -- 172.21.15.43:0/3342973349 shutdown_connections 2024-05-27T07:36:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:36:39.960+0000 7f1e7e200700 1 -- 172.21.15.43:0/3342973349 wait complete. 2024-05-27T07:36:39.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:39 smithi179 ceph-mon[35888]: pgmap v764: 129 pgs: 129 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.2 MiB/s wr, 204 op/s 2024-05-27T07:36:40.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:39 smithi132 ceph-mon[35283]: pgmap v764: 129 pgs: 129 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.2 MiB/s wr, 204 op/s 2024-05-27T07:36:40.821 INFO:tasks.workunit.client.0.smithi043.stdout: 10 57096 5.83 MB/sec execute 401 sec latency 139.133 ms 2024-05-27T07:36:41.821 INFO:tasks.workunit.client.0.smithi043.stdout: 10 57182 5.82 MB/sec execute 402 sec latency 135.846 ms 2024-05-27T07:36:41.910 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:41 smithi043 ceph-mon[31531]: pgmap v765: 129 pgs: 129 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.8 MiB/s wr, 105 op/s 2024-05-27T07:36:41.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:41 smithi179 ceph-mon[35888]: pgmap v765: 129 pgs: 129 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.8 MiB/s wr, 105 op/s 2024-05-27T07:36:42.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:41 smithi132 ceph-mon[35283]: pgmap v765: 129 pgs: 129 active+clean; 1.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.8 MiB/s wr, 105 op/s 2024-05-27T07:36:42.822 INFO:tasks.workunit.client.0.smithi043.stdout: 10 57307 5.83 MB/sec execute 403 sec latency 105.826 ms 2024-05-27T07:36:43.474 INFO:tasks.cephfs.filesystem:scrub status for tag:c6ba073b-1f8b-42ce-a3bc-d0e506326d24 - {'path': '/', 'tag': 'c6ba073b-1f8b-42ce-a3bc-d0e506326d24', 'options': 'recursive,force'} 2024-05-27T07:36:43.707 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:43 smithi043 ceph-mon[31531]: pgmap v766: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 4.9 MiB/s wr, 204 op/s 2024-05-27T07:36:43.822 INFO:tasks.workunit.client.0.smithi043.stdout: 10 57380 5.83 MB/sec execute 404 sec latency 607.169 ms 2024-05-27T07:36:43.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:43 smithi179 ceph-mon[35888]: pgmap v766: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 4.9 MiB/s wr, 204 op/s 2024-05-27T07:36:44.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:43 smithi132 ceph-mon[35283]: pgmap v766: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 4.9 MiB/s wr, 204 op/s 2024-05-27T07:36:44.822 INFO:tasks.workunit.client.0.smithi043.stdout: 10 57480 5.82 MB/sec execute 405 sec latency 672.175 ms 2024-05-27T07:36:45.822 INFO:tasks.workunit.client.0.smithi043.stdout: 10 57616 5.83 MB/sec execute 406 sec latency 101.582 ms 2024-05-27T07:36:45.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:45 smithi043 ceph-mon[31531]: pgmap v767: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.3 MiB/s wr, 181 op/s 2024-05-27T07:36:45.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:45 smithi179 ceph-mon[35888]: pgmap v767: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.3 MiB/s wr, 181 op/s 2024-05-27T07:36:46.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:45 smithi132 ceph-mon[35283]: pgmap v767: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.3 MiB/s wr, 181 op/s 2024-05-27T07:36:46.822 INFO:tasks.workunit.client.0.smithi043.stdout: 10 57730 5.83 MB/sec execute 407 sec latency 76.657 ms 2024-05-27T07:36:47.822 INFO:tasks.workunit.client.0.smithi043.stdout: 10 57860 5.83 MB/sec execute 408 sec latency 76.714 ms 2024-05-27T07:36:47.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:47 smithi132 ceph-mon[35283]: pgmap v768: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-05-27T07:36:47.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:47 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:36:47.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:47 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:36:47.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:47 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:36:47.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:47 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:36:47.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:47 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:36:47.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:47 smithi043 ceph-mon[31531]: pgmap v768: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-05-27T07:36:47.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:47 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:36:47.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:47 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:36:47.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:47 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:36:47.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:47 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:36:47.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:47 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:36:47.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:47 smithi179 ceph-mon[35888]: pgmap v768: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.3 MiB/s wr, 231 op/s 2024-05-27T07:36:47.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:47 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:36:47.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:47 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:36:47.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:47 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:36:47.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:47 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:36:47.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:47 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:36:48.822 INFO:tasks.workunit.client.0.smithi043.stdout: 10 57981 5.84 MB/sec execute 409 sec latency 75.920 ms 2024-05-27T07:36:49.822 INFO:tasks.workunit.client.0.smithi043.stdout: 10 58106 5.84 MB/sec execute 410 sec latency 135.030 ms 2024-05-27T07:36:49.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:49 smithi132 ceph-mon[35283]: pgmap v769: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.2 MiB/s wr, 153 op/s 2024-05-27T07:36:49.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:49 smithi043 ceph-mon[31531]: pgmap v769: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.2 MiB/s wr, 153 op/s 2024-05-27T07:36:49.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:49 smithi179 ceph-mon[35888]: pgmap v769: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.2 MiB/s wr, 153 op/s 2024-05-27T07:36:50.822 INFO:tasks.workunit.client.0.smithi043.stdout: 10 58217 5.83 MB/sec execute 411 sec latency 227.797 ms 2024-05-27T07:36:51.823 INFO:tasks.workunit.client.0.smithi043.stdout: 10 58346 5.84 MB/sec execute 412 sec latency 129.367 ms 2024-05-27T07:36:51.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:51 smithi132 ceph-mon[35283]: pgmap v770: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.2 MiB/s wr, 153 op/s 2024-05-27T07:36:51.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:51 smithi043 ceph-mon[31531]: pgmap v770: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.2 MiB/s wr, 153 op/s 2024-05-27T07:36:51.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:51 smithi179 ceph-mon[35888]: pgmap v770: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.2 MiB/s wr, 153 op/s 2024-05-27T07:36:52.823 INFO:tasks.workunit.client.0.smithi043.stdout: 10 58470 5.83 MB/sec execute 413 sec latency 97.828 ms 2024-05-27T07:36:53.823 INFO:tasks.workunit.client.0.smithi043.stdout: 10 58588 5.84 MB/sec execute 414 sec latency 98.396 ms 2024-05-27T07:36:53.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:53 smithi132 ceph-mon[35283]: pgmap v771: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.0 MiB/s wr, 225 op/s 2024-05-27T07:36:53.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:53 smithi043 ceph-mon[31531]: pgmap v771: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.0 MiB/s wr, 225 op/s 2024-05-27T07:36:53.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:53 smithi179 ceph-mon[35888]: pgmap v771: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.0 MiB/s wr, 225 op/s 2024-05-27T07:36:54.823 INFO:tasks.workunit.client.0.smithi043.stdout: 10 58686 5.84 MB/sec execute 415 sec latency 69.958 ms 2024-05-27T07:36:55.823 INFO:tasks.workunit.client.0.smithi043.stdout: 10 58810 5.84 MB/sec execute 416 sec latency 84.587 ms 2024-05-27T07:36:55.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:55 smithi132 ceph-mon[35283]: pgmap v772: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-05-27T07:36:55.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:55 smithi043 ceph-mon[31531]: pgmap v772: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-05-27T07:36:55.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:55 smithi179 ceph-mon[35888]: pgmap v772: 129 pgs: 129 active+clean; 1.2 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-05-27T07:36:56.823 INFO:tasks.workunit.client.0.smithi043.stdout: 10 58967 5.86 MB/sec execute 417 sec latency 85.476 ms 2024-05-27T07:36:57.823 INFO:tasks.workunit.client.0.smithi043.stdout: 10 59100 5.87 MB/sec execute 418 sec latency 88.857 ms 2024-05-27T07:36:57.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:57 smithi132 ceph-mon[35283]: pgmap v773: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 5.6 MiB/s wr, 238 op/s 2024-05-27T07:36:57.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:57 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:36:57.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:57 smithi043 ceph-mon[31531]: pgmap v773: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 5.6 MiB/s wr, 238 op/s 2024-05-27T07:36:57.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:57 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:36:57.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:57 smithi179 ceph-mon[35888]: pgmap v773: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 5.6 MiB/s wr, 238 op/s 2024-05-27T07:36:57.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:57 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:36:58.824 INFO:tasks.workunit.client.0.smithi043.stdout: 10 59215 5.87 MB/sec execute 419 sec latency 81.457 ms 2024-05-27T07:36:59.824 INFO:tasks.workunit.client.0.smithi043.stdout: 10 59312 5.87 MB/sec execute 420 sec latency 106.079 ms 2024-05-27T07:36:59.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:36:59 smithi132 ceph-mon[35283]: pgmap v774: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.0 MiB/s wr, 159 op/s 2024-05-27T07:36:59.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:36:59 smithi043 ceph-mon[31531]: pgmap v774: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.0 MiB/s wr, 159 op/s 2024-05-27T07:36:59.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:36:59 smithi179 ceph-mon[35888]: pgmap v774: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.0 MiB/s wr, 159 op/s 2024-05-27T07:37:00.825 INFO:tasks.workunit.client.0.smithi043.stdout: 10 59405 5.87 MB/sec execute 421 sec latency 124.546 ms 2024-05-27T07:37:01.825 INFO:tasks.workunit.client.0.smithi043.stdout: 10 59546 5.87 MB/sec execute 422 sec latency 111.737 ms 2024-05-27T07:37:01.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:01 smithi132 ceph-mon[35283]: pgmap v775: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.0 MiB/s wr, 159 op/s 2024-05-27T07:37:01.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:01 smithi043 ceph-mon[31531]: pgmap v775: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.0 MiB/s wr, 159 op/s 2024-05-27T07:37:01.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:01 smithi179 ceph-mon[35888]: pgmap v775: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.0 MiB/s wr, 159 op/s 2024-05-27T07:37:02.825 INFO:tasks.workunit.client.0.smithi043.stdout: 10 59635 5.87 MB/sec execute 423 sec latency 98.731 ms 2024-05-27T07:37:03.825 INFO:tasks.workunit.client.0.smithi043.stdout: 10 59740 5.87 MB/sec execute 424 sec latency 152.467 ms 2024-05-27T07:37:03.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:03 smithi043 ceph-mon[31531]: pgmap v776: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 6.2 MiB/s wr, 249 op/s 2024-05-27T07:37:03.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:03 smithi043 ceph-mon[31531]: osdmap e156: 12 total, 12 up, 12 in 2024-05-27T07:37:03.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:03 smithi179 ceph-mon[35888]: pgmap v776: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 6.2 MiB/s wr, 249 op/s 2024-05-27T07:37:03.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:03 smithi179 ceph-mon[35888]: osdmap e156: 12 total, 12 up, 12 in 2024-05-27T07:37:04.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:03 smithi132 ceph-mon[35283]: pgmap v776: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 6.2 MiB/s wr, 249 op/s 2024-05-27T07:37:04.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:03 smithi132 ceph-mon[35283]: osdmap e156: 12 total, 12 up, 12 in 2024-05-27T07:37:04.826 INFO:tasks.workunit.client.0.smithi043.stdout: 10 59840 5.86 MB/sec execute 425 sec latency 102.055 ms 2024-05-27T07:37:05.826 INFO:tasks.workunit.client.0.smithi043.stdout: 10 59952 5.86 MB/sec execute 426 sec latency 95.963 ms 2024-05-27T07:37:05.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:05 smithi043 ceph-mon[31531]: pgmap v778: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 5.2 MiB/s wr, 213 op/s 2024-05-27T07:37:05.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:05 smithi179 ceph-mon[35888]: pgmap v778: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 5.2 MiB/s wr, 213 op/s 2024-05-27T07:37:06.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:05 smithi132 ceph-mon[35283]: pgmap v778: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 5.2 MiB/s wr, 213 op/s 2024-05-27T07:37:06.826 INFO:tasks.workunit.client.0.smithi043.stdout: 10 60058 5.86 MB/sec execute 427 sec latency 108.091 ms 2024-05-27T07:37:07.826 INFO:tasks.workunit.client.0.smithi043.stdout: 10 60161 5.85 MB/sec execute 428 sec latency 143.138 ms 2024-05-27T07:37:07.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:07 smithi043 ceph-mon[31531]: pgmap v779: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 4.9 MiB/s wr, 205 op/s 2024-05-27T07:37:07.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:07 smithi179 ceph-mon[35888]: pgmap v779: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 4.9 MiB/s wr, 205 op/s 2024-05-27T07:37:08.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:07 smithi132 ceph-mon[35283]: pgmap v779: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 4.9 MiB/s wr, 205 op/s 2024-05-27T07:37:08.826 INFO:tasks.workunit.client.0.smithi043.stdout: 10 60222 5.84 MB/sec execute 429 sec latency 456.641 ms 2024-05-27T07:37:09.826 INFO:tasks.workunit.client.0.smithi043.stdout: 10 60337 5.84 MB/sec execute 430 sec latency 149.541 ms 2024-05-27T07:37:09.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:09 smithi043 ceph-mon[31531]: pgmap v780: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 4.9 MiB/s wr, 205 op/s 2024-05-27T07:37:09.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:09 smithi179 ceph-mon[35888]: pgmap v780: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 4.9 MiB/s wr, 205 op/s 2024-05-27T07:37:10.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:09 smithi132 ceph-mon[35283]: pgmap v780: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 4.9 MiB/s wr, 205 op/s 2024-05-27T07:37:10.826 INFO:tasks.workunit.client.0.smithi043.stdout: 10 60472 5.85 MB/sec execute 431 sec latency 101.546 ms 2024-05-27T07:37:11.826 INFO:tasks.workunit.client.0.smithi043.stdout: 10 60593 5.85 MB/sec execute 432 sec latency 110.720 ms 2024-05-27T07:37:11.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:11 smithi043 ceph-mon[31531]: pgmap v781: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 4.9 MiB/s wr, 205 op/s 2024-05-27T07:37:11.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:11 smithi043 ceph-mon[31531]: osdmap e157: 12 total, 12 up, 12 in 2024-05-27T07:37:11.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:11 smithi179 ceph-mon[35888]: pgmap v781: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 4.9 MiB/s wr, 205 op/s 2024-05-27T07:37:11.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:11 smithi179 ceph-mon[35888]: osdmap e157: 12 total, 12 up, 12 in 2024-05-27T07:37:12.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:11 smithi132 ceph-mon[35283]: pgmap v781: 129 pgs: 129 active+clean; 1.2 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 4.9 MiB/s wr, 205 op/s 2024-05-27T07:37:12.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:11 smithi132 ceph-mon[35283]: osdmap e157: 12 total, 12 up, 12 in 2024-05-27T07:37:12.826 INFO:tasks.workunit.client.0.smithi043.stdout: 10 60734 5.85 MB/sec execute 433 sec latency 100.251 ms 2024-05-27T07:37:12.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:12 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:37:12.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:12 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:37:13.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:12 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:37:13.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:37:13.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 60799 5.84 MB/sec execute 434 sec latency 365.749 ms 2024-05-27T07:37:13.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:13 smithi043 ceph-mon[31531]: pgmap v783: 129 pgs: 129 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 4.5 MiB/s wr, 196 op/s 2024-05-27T07:37:13.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:13 smithi179 ceph-mon[35888]: pgmap v783: 129 pgs: 129 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 4.5 MiB/s wr, 196 op/s 2024-05-27T07:37:14.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:13 smithi132 ceph-mon[35283]: pgmap v783: 129 pgs: 129 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 4.5 MiB/s wr, 196 op/s 2024-05-27T07:37:14.827 INFO:tasks.workunit.client.0.smithi043.stdout: 10 60876 5.84 MB/sec execute 435 sec latency 491.443 ms 2024-05-27T07:37:15.827 INFO:tasks.workunit.client.0.smithi043.stdout: 10 60987 5.84 MB/sec execute 436 sec latency 131.927 ms 2024-05-27T07:37:15.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:15 smithi043 ceph-mon[31531]: pgmap v784: 129 pgs: 129 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.3 MiB/s wr, 190 op/s 2024-05-27T07:37:15.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:15 smithi179 ceph-mon[35888]: pgmap v784: 129 pgs: 129 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.3 MiB/s wr, 190 op/s 2024-05-27T07:37:16.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:15 smithi132 ceph-mon[35283]: pgmap v784: 129 pgs: 129 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.3 MiB/s wr, 190 op/s 2024-05-27T07:37:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:16.827 INFO:tasks.workunit.client.0.smithi043.stdout: 10 61084 5.84 MB/sec execute 437 sec latency 217.932 ms 2024-05-27T07:37:17.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:17 smithi043 ceph-mon[31531]: pgmap v785: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-27T07:37:17.827 INFO:tasks.workunit.client.0.smithi043.stdout: 10 61216 5.84 MB/sec execute 438 sec latency 96.371 ms 2024-05-27T07:37:17.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:17 smithi179 ceph-mon[35888]: pgmap v785: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-27T07:37:18.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:17 smithi132 ceph-mon[35283]: pgmap v785: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-27T07:37:18.827 INFO:tasks.workunit.client.0.smithi043.stdout: 10 61329 5.85 MB/sec execute 439 sec latency 148.938 ms 2024-05-27T07:37:19.827 INFO:tasks.workunit.client.0.smithi043.stdout: 10 61410 5.84 MB/sec execute 440 sec latency 236.771 ms 2024-05-27T07:37:19.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:19 smithi043 ceph-mon[31531]: pgmap v786: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-27T07:37:19.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:19 smithi179 ceph-mon[35888]: pgmap v786: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-27T07:37:20.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:19 smithi132 ceph-mon[35283]: pgmap v786: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-27T07:37:20.827 INFO:tasks.workunit.client.0.smithi043.stdout: 10 61513 5.84 MB/sec execute 441 sec latency 126.137 ms 2024-05-27T07:37:21.827 INFO:tasks.workunit.client.0.smithi043.stdout: 10 61632 5.84 MB/sec execute 442 sec latency 78.334 ms 2024-05-27T07:37:21.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:21 smithi043 ceph-mon[31531]: pgmap v787: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-27T07:37:21.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:21 smithi179 ceph-mon[35888]: pgmap v787: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-27T07:37:22.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:21 smithi132 ceph-mon[35283]: pgmap v787: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.9 MiB/s wr, 181 op/s 2024-05-27T07:37:22.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.365+0000 7f2779e00700 1 -- 172.21.15.43:0/1219815710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27741045c0 msgr2=0x7f27741049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:22.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.365+0000 7f2779e00700 1 --2- 172.21.15.43:0/1219815710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27741045c0 0x7f27741049d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f2764009920 tx=0x7f2764009c30 comp rx=0 tx=0).stop 2024-05-27T07:37:22.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.366+0000 7f2779e00700 1 -- 172.21.15.43:0/1219815710 shutdown_connections 2024-05-27T07:37:22.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.366+0000 7f2779e00700 1 --2- 172.21.15.43:0/1219815710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2774109840 0x7f27741111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.366+0000 7f2779e00700 1 --2- 172.21.15.43:0/1219815710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27741045c0 0x7f27741049d0 secure :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f2764009920 tx=0x7f2764009c30 comp rx=0 tx=0).stop 2024-05-27T07:37:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.366+0000 7f2779e00700 1 --2- 172.21.15.43:0/1219815710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2774108ef0 0x7f2774109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.366+0000 7f2779e00700 1 -- 172.21.15.43:0/1219815710 >> 172.21.15.43:0/1219815710 conn(0x7f27740ffe90 msgr2=0x7f27741022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:22.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.367+0000 7f2779e00700 1 -- 172.21.15.43:0/1219815710 shutdown_connections 2024-05-27T07:37:22.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.368+0000 7f2779e00700 1 -- 172.21.15.43:0/1219815710 wait complete. 2024-05-27T07:37:22.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.370+0000 7f2779e00700 1 Processor -- start 2024-05-27T07:37:22.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.371+0000 7f2779e00700 1 -- start start 2024-05-27T07:37:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.372+0000 7f2779e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2774108ef0 0x7f277410dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.372+0000 7f2779e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2774109840 0x7f277410e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.372+0000 7f2779e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27741ab180 0x7f27741ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.372+0000 7f2779e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2774077650 con 0x7f2774108ef0 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.372+0000 7f2779e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f27740774d0 con 0x7f27741ab180 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.372+0000 7f2779e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f27740777d0 con 0x7f2774109840 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2773400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27741ab180 0x7f27741ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2773400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27741ab180 0x7f27741ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38342/0 (socket says 172.21.15.43:38342) 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2773400700 1 -- 172.21.15.43:0/4283454009 learned_addr learned my addr 172.21.15.43:0/4283454009 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2772000700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2774109840 0x7f277410e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2772a00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2774108ef0 0x7f277410dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2773400700 1 -- 172.21.15.43:0/4283454009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2774109840 msgr2=0x7f277410e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2773400700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2774109840 0x7f277410e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2773400700 1 -- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2774108ef0 msgr2=0x7f277410dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:22.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2773400700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2774108ef0 0x7f277410dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2773400700 1 -- 172.21.15.43:0/4283454009 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27741adb60 con 0x7f27741ab180 2024-05-27T07:37:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.373+0000 7f2772a00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2774108ef0 0x7f277410dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.374+0000 7f2773400700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27741ab180 0x7f27741ad5a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f276c00e3c0 tx=0x7f276c00e6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.374+0000 7f2778a00700 1 -- 172.21.15.43:0/4283454009 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c0092d0 con 0x7f27741ab180 2024-05-27T07:37:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.374+0000 7f2778a00700 1 -- 172.21.15.43:0/4283454009 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f276c00ee70 con 0x7f27741ab180 2024-05-27T07:37:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.374+0000 7f2778a00700 1 -- 172.21.15.43:0/4283454009 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c007d80 con 0x7f27741ab180 2024-05-27T07:37:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.375+0000 7f2779e00700 1 -- 172.21.15.43:0/4283454009 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27741ade50 con 0x7f27741ab180 2024-05-27T07:37:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.375+0000 7f2779e00700 1 -- 172.21.15.43:0/4283454009 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27741ae2b0 con 0x7f27741ab180 2024-05-27T07:37:22.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.380+0000 7f2778a00700 1 -- 172.21.15.43:0/4283454009 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f276c012050 con 0x7f27741ab180 2024-05-27T07:37:22.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.383+0000 7f2778a00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27580b29f0 0x7f27580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.383+0000 7f2778a00700 1 -- 172.21.15.43:0/4283454009 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f276c0e3390 con 0x7f27741ab180 2024-05-27T07:37:22.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.383+0000 7f2772a00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27580b29f0 0x7f27580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.419+0000 7f2772a00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27580b29f0 0x7f27580b4f00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2764009810 tx=0x7f2764005dc0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:22.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.422+0000 7f2788873b80 1 Processor -- start 2024-05-27T07:37:22.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.422+0000 7f2788873b80 1 -- start start 2024-05-27T07:37:22.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.423+0000 7f2788873b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 0x56050eb32330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.423+0000 7f2788873b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 0x56050eb32870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.423+0000 7f2788873b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.423+0000 7f2788873b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56050ea60e40 con 0x56050eb32e70 2024-05-27T07:37:22.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.423+0000 7f2788873b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56050ea509e0 con 0x56050e9b3e40 2024-05-27T07:37:22.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.423+0000 7f2788873b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56050ea54a20 con 0x56050ea9f9f0 2024-05-27T07:37:22.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.423+0000 7f2772000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 0x56050eb32870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.423+0000 7f2772000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 0x56050eb32870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38346/0 (socket says 172.21.15.43:38346) 2024-05-27T07:37:22.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.423+0000 7f2772000700 1 -- 172.21.15.43:0/2840546301 learned_addr learned my addr 172.21.15.43:0/2840546301 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:22.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.424+0000 7f2772a00700 1 --2- 172.21.15.43:0/2840546301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 0x56050eb32330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.424+0000 7f2773400700 1 --2- 172.21.15.43:0/2840546301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.425+0000 7f2772000700 1 -- 172.21.15.43:0/2840546301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 msgr2=0x56050eb32330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:22.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.425+0000 7f2772000700 1 --2- 172.21.15.43:0/2840546301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 0x56050eb32330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.425+0000 7f2772000700 1 -- 172.21.15.43:0/2840546301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 msgr2=0x56050eb3f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:22.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.425+0000 7f2772000700 1 --2- 172.21.15.43:0/2840546301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.425+0000 7f2772000700 1 -- 172.21.15.43:0/2840546301 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56050ea19420 con 0x56050e9b3e40 2024-05-27T07:37:22.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.425+0000 7f2773400700 1 --2- 172.21.15.43:0/2840546301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:37:22.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.426+0000 7f2772000700 1 --2- 172.21.15.43:0/2840546301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 0x56050eb32870 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2760009cb0 tx=0x7f2760011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3eb6484567ad3837 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:22.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.426+0000 7f2768c00700 1 -- 172.21.15.43:0/2840546301 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2760011d80 con 0x56050e9b3e40 2024-05-27T07:37:22.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.426+0000 7f2768c00700 1 -- 172.21.15.43:0/2840546301 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27600044e0 con 0x56050e9b3e40 2024-05-27T07:37:22.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.426+0000 7f2768c00700 1 -- 172.21.15.43:0/2840546301 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27600193e0 con 0x56050e9b3e40 2024-05-27T07:37:22.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.428+0000 7f2788873b80 1 -- 172.21.15.43:0/2840546301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 msgr2=0x56050eb32870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.428+0000 7f2788873b80 1 --2- 172.21.15.43:0/2840546301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 0x56050eb32870 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2760009cb0 tx=0x7f2760011440 comp rx=0 tx=0).stop 2024-05-27T07:37:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.428+0000 7f2788873b80 1 -- 172.21.15.43:0/2840546301 shutdown_connections 2024-05-27T07:37:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.428+0000 7f2788873b80 1 --2- 172.21.15.43:0/2840546301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.428+0000 7f2788873b80 1 --2- 172.21.15.43:0/2840546301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 0x56050eb32330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.428+0000 7f2788873b80 1 --2- 172.21.15.43:0/2840546301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 0x56050eb32870 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.428+0000 7f2788873b80 1 -- 172.21.15.43:0/2840546301 >> 172.21.15.43:0/2840546301 conn(0x56050ea4d390 msgr2=0x56050eb2e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.428+0000 7f2788873b80 1 -- 172.21.15.43:0/2840546301 shutdown_connections 2024-05-27T07:37:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.428+0000 7f2788873b80 1 -- 172.21.15.43:0/2840546301 wait complete. 2024-05-27T07:37:22.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.430+0000 7f2788873b80 1 -- start start 2024-05-27T07:37:22.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.430+0000 7f2788873b80 1 Processor -- start 2024-05-27T07:37:22.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.432+0000 7f2788873b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 0x56050eb3c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.432+0000 7f2788873b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 0x56050eb3cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.432+0000 7f2788873b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.433+0000 7f2788873b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56050e956fa0 con 0x56050eb32e70 2024-05-27T07:37:22.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.433+0000 7f2788873b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56050e9f1760 con 0x56050e9b3e40 2024-05-27T07:37:22.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.433+0000 7f2788873b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56050eb3fb80 con 0x56050ea9f9f0 2024-05-27T07:37:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.433+0000 7f2788873b80 10 client.0 _ll_register_callbacks cb 0x56050e9fac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.433+0000 7f2773400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.433+0000 7f2773400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45488/0 (socket says 172.21.15.43:45488) 2024-05-27T07:37:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.433+0000 7f2772a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 0x56050eb3c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.433+0000 7f2773400700 1 -- 172.21.15.43:0/1380692411 learned_addr learned my addr 172.21.15.43:0/1380692411 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.433+0000 7f2772000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 0x56050eb3cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.434+0000 7f2773400700 1 -- 172.21.15.43:0/1380692411 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 msgr2=0x56050eb3cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.434+0000 7f2773400700 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 0x56050eb3cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.434+0000 7f2773400700 1 -- 172.21.15.43:0/1380692411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 msgr2=0x56050eb3c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:22.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.434+0000 7f2773400700 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 0x56050eb3c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.434+0000 7f2773400700 1 -- 172.21.15.43:0/1380692411 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56050e9a08d0 con 0x56050eb32e70 2024-05-27T07:37:22.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.434+0000 7f2772000700 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 0x56050eb3cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:22.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.435+0000 7f2773400700 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3d110 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f276c0e0710 tx=0x7f276c0a67e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba69f2ee1a1cdd9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:22.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.435+0000 7f275d600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:37:22.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.436+0000 7f275d600700 1 -- 172.21.15.43:0/1380692411 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c0e3160 con 0x56050eb32e70 2024-05-27T07:37:22.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.436+0000 7f275d600700 1 -- 172.21.15.43:0/1380692411 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f276c0a68c0 con 0x56050eb32e70 2024-05-27T07:37:22.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.436+0000 7f2788873b80 1 -- 172.21.15.43:0/1380692411 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27600044e0 con 0x56050eb32e70 2024-05-27T07:37:22.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.437+0000 7f275d600700 1 -- 172.21.15.43:0/1380692411 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=146) ==== 24+0+0 (secure 0 0 0) 0x7f276c0aa9d0 con 0x56050eb32e70 2024-05-27T07:37:22.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.437+0000 7f2788873b80 10 client.18366 fetch_fsmap learned FSMap version 146 2024-05-27T07:37:22.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.437+0000 7f2788873b80 1 -- 172.21.15.43:0/1380692411 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=146,monmap=4+}) -- 0x56050ea21060 con 0x56050eb32e70 2024-05-27T07:37:22.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.437+0000 7f275d600700 1 -- 172.21.15.43:0/1380692411 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27600044e0 con 0x56050eb32e70 2024-05-27T07:37:22.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.439+0000 7f275d600700 1 -- 172.21.15.43:0/1380692411 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 146) ==== 4191+0+0 (secure 0 0 0) 0x7f276c0b08d0 con 0x56050eb32e70 2024-05-27T07:37:22.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.439+0000 7f2788873b80 10 client.18366 fetch_fsmap finished waiting for FSMap version 146 2024-05-27T07:37:22.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.439+0000 7f2788873b80 10 client.18366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:37:22.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.439+0000 7f2788873b80 20 client.18366 populate_metadata read hostname 'smithi043' 2024-05-27T07:37:22.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.439+0000 7f2788873b80 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56050eb4ce20 0x56050eb4f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.439+0000 7f2788873b80 4 client.18366 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:22.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.439+0000 7f2788873b80 1 -- 172.21.15.43:0/1380692411 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56050ea62d90 con 0x56050eb4ce20 2024-05-27T07:37:22.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.439+0000 7f2772a00700 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56050eb4ce20 0x56050eb4f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.440+0000 7f2772a00700 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56050eb4ce20 0x56050eb4f2f0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9e285c905f44925 server_cookie=ae116400b735e962 in_seq=0 out_seq=0 2024-05-27T07:37:22.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.440+0000 7f275d600700 10 client.18366 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:37:22.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.441+0000 7f275d600700 1 -- 172.21.15.43:0/1380692411 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56050ea62d90 con 0x56050eb4ce20 2024-05-27T07:37:22.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.441+0000 7f275d600700 10 client.18366 handle_command_reply: tid=0 2024-05-27T07:37:22.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.441+0000 7f2788873b80 1 client.18366 shutdown 2024-05-27T07:37:22.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.442+0000 7f2788873b80 1 -- 172.21.15.43:0/1380692411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 msgr2=0x56050eb3d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:22.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.442+0000 7f2788873b80 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3d110 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f276c0e0710 tx=0x7f276c0a67e0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.442+0000 7f2788873b80 1 -- 172.21.15.43:0/1380692411 shutdown_connections 2024-05-27T07:37:22.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.442+0000 7f2788873b80 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56050eb4ce20 0x56050eb4f2f0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.442+0000 7f2788873b80 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050eb3d110 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.442+0000 7f2788873b80 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050e9b3e40 0x56050eb3c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.442+0000 7f2788873b80 1 --2- 172.21.15.43:0/1380692411 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050ea9f9f0 0x56050eb3cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.442+0000 7f2788873b80 1 -- 172.21.15.43:0/1380692411 >> 172.21.15.43:0/1380692411 conn(0x56050ea4d390 msgr2=0x56050eb32310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:22.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.442+0000 7f275d600700 0 client.18366 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:37:22.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.443+0000 7f2788873b80 1 -- 172.21.15.43:0/1380692411 shutdown_connections 2024-05-27T07:37:22.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.443+0000 7f2788873b80 1 -- 172.21.15.43:0/1380692411 wait complete. 2024-05-27T07:37:22.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.443+0000 7f2788873b80 20 client.18366 trim_cache size 0 max 16384 2024-05-27T07:37:22.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.477+0000 7f2788873b80 1 Processor -- start 2024-05-27T07:37:22.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.477+0000 7f2788873b80 1 -- start start 2024-05-27T07:37:22.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.478+0000 7f2788873b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050e9b3e40 0x56050eb4f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.478+0000 7f2788873b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050ea9f9f0 0x56050eb365f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.478+0000 7f2788873b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050eb32e70 0x56050eb48c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.478+0000 7f2788873b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f27741b78d0 con 0x56050ea9f9f0 2024-05-27T07:37:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.478+0000 7f2788873b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f27741b5d90 con 0x56050eb32e70 2024-05-27T07:37:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.478+0000 7f2788873b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f27741b7200 con 0x56050e9b3e40 2024-05-27T07:37:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.478+0000 7f2772000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050ea9f9f0 0x56050eb365f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.478+0000 7f2772000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050ea9f9f0 0x56050eb365f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45492/0 (socket says 172.21.15.43:45492) 2024-05-27T07:37:22.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.478+0000 7f2772000700 1 -- 172.21.15.43:0/3242354614 learned_addr learned my addr 172.21.15.43:0/3242354614 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:22.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.479+0000 7f2773400700 1 --2- 172.21.15.43:0/3242354614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050eb32e70 0x56050eb48c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.479+0000 7f2772a00700 1 --2- 172.21.15.43:0/3242354614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050e9b3e40 0x56050eb4f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.479+0000 7f2772000700 1 -- 172.21.15.43:0/3242354614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050e9b3e40 msgr2=0x56050eb4f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:22.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.479+0000 7f2772000700 1 --2- 172.21.15.43:0/3242354614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050e9b3e40 0x56050eb4f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.479+0000 7f2772000700 1 -- 172.21.15.43:0/3242354614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050eb32e70 msgr2=0x56050eb48c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:22.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.479+0000 7f2772000700 1 --2- 172.21.15.43:0/3242354614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050eb32e70 0x56050eb48c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.479+0000 7f2772000700 1 -- 172.21.15.43:0/3242354614 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56050eb2a240 con 0x56050ea9f9f0 2024-05-27T07:37:22.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.480+0000 7f2772000700 1 --2- 172.21.15.43:0/3242354614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050ea9f9f0 0x56050eb365f0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2760009cb0 tx=0x7f2760005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d56bb9bfd14c77e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:22.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.480+0000 7f275fe00700 1 -- 172.21.15.43:0/3242354614 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2760027020 con 0x56050ea9f9f0 2024-05-27T07:37:22.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.481+0000 7f275fe00700 1 -- 172.21.15.43:0/3242354614 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27600054e0 con 0x56050ea9f9f0 2024-05-27T07:37:22.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.481+0000 7f275fe00700 1 -- 172.21.15.43:0/3242354614 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2760003b80 con 0x56050ea9f9f0 2024-05-27T07:37:22.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.482+0000 7f2788873b80 1 -- 172.21.15.43:0/3242354614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050ea9f9f0 msgr2=0x56050eb365f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:22.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.482+0000 7f2788873b80 1 --2- 172.21.15.43:0/3242354614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050ea9f9f0 0x56050eb365f0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2760009cb0 tx=0x7f2760005e00 comp rx=0 tx=0).stop 2024-05-27T07:37:22.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.482+0000 7f2788873b80 1 -- 172.21.15.43:0/3242354614 shutdown_connections 2024-05-27T07:37:22.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.482+0000 7f2788873b80 1 --2- 172.21.15.43:0/3242354614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050eb32e70 0x56050eb48c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.482+0000 7f2788873b80 1 --2- 172.21.15.43:0/3242354614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050e9b3e40 0x56050eb4f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.482+0000 7f2788873b80 1 --2- 172.21.15.43:0/3242354614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050ea9f9f0 0x56050eb365f0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.482+0000 7f2788873b80 1 -- 172.21.15.43:0/3242354614 >> 172.21.15.43:0/3242354614 conn(0x56050ea4d390 msgr2=0x56050eb35ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.482+0000 7f2788873b80 1 -- 172.21.15.43:0/3242354614 shutdown_connections 2024-05-27T07:37:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.482+0000 7f2788873b80 1 -- 172.21.15.43:0/3242354614 wait complete. 2024-05-27T07:37:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.483+0000 7f2788873b80 1 -- start start 2024-05-27T07:37:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.483+0000 7f2788873b80 1 Processor -- start 2024-05-27T07:37:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.484+0000 7f2788873b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050e9b3e40 0x56050eb403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.484+0000 7f2788873b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050ea9f9f0 0x56050eb408f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.484+0000 7f2788873b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050ebb59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.484+0000 7f2788873b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f27741b6b30 con 0x56050eb32e70 2024-05-27T07:37:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.484+0000 7f2788873b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f27741b6460 con 0x56050ea9f9f0 2024-05-27T07:37:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.484+0000 7f2788873b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56050ea62d90 con 0x56050e9b3e40 2024-05-27T07:37:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.484+0000 7f2788873b80 10 client.0 _ll_register_callbacks cb 0x56050e9fac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.485+0000 7f2772000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050ea9f9f0 0x56050eb408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.485+0000 7f2772000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050ea9f9f0 0x56050eb408f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38370/0 (socket says 172.21.15.43:38370) 2024-05-27T07:37:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.485+0000 7f2772000700 1 -- 172.21.15.43:0/3634266979 learned_addr learned my addr 172.21.15.43:0/3634266979 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.485+0000 7f2772a00700 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050e9b3e40 0x56050eb403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.485+0000 7f2772000700 1 -- 172.21.15.43:0/3634266979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050e9b3e40 msgr2=0x56050eb403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.486+0000 7f2772000700 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050e9b3e40 0x56050eb403b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.486+0000 7f2772000700 1 -- 172.21.15.43:0/3634266979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 msgr2=0x56050ebb59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.486+0000 7f2772000700 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050ebb59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.486+0000 7f2772000700 1 -- 172.21.15.43:0/3634266979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56050ebb5f20 con 0x56050ea9f9f0 2024-05-27T07:37:22.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.487+0000 7f2772000700 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050ea9f9f0 0x56050eb408f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2760003770 tx=0x7f2760005690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b2f60d11a3d8e99 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:22.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.487+0000 7f275d600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:37:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.488+0000 7f275d600700 1 -- 172.21.15.43:0/3634266979 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2760032070 con 0x56050ea9f9f0 2024-05-27T07:37:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.488+0000 7f275d600700 1 -- 172.21.15.43:0/3634266979 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f2760003bd0 con 0x56050ea9f9f0 2024-05-27T07:37:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.488+0000 7f2788873b80 1 -- 172.21.15.43:0/3634266979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27600054e0 con 0x56050ea9f9f0 2024-05-27T07:37:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.489+0000 7f275d600700 1 -- 172.21.15.43:0/3634266979 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=146) ==== 24+0+0 (secure 0 0 0) 0x7f276003e050 con 0x56050ea9f9f0 2024-05-27T07:37:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.489+0000 7f2788873b80 10 client.37754 fetch_fsmap learned FSMap version 146 2024-05-27T07:37:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.489+0000 7f2788873b80 1 -- 172.21.15.43:0/3634266979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=146,monmap=4+}) -- 0x56050ebb7170 con 0x56050ea9f9f0 2024-05-27T07:37:22.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.489+0000 7f275d600700 1 -- 172.21.15.43:0/3634266979 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f27600054e0 con 0x56050ea9f9f0 2024-05-27T07:37:22.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.490+0000 7f275d600700 1 -- 172.21.15.43:0/3634266979 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 146) ==== 4191+0+0 (secure 0 0 0) 0x7f2760018070 con 0x56050ea9f9f0 2024-05-27T07:37:22.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.490+0000 7f2788873b80 10 client.37754 fetch_fsmap finished waiting for FSMap version 146 2024-05-27T07:37:22.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.490+0000 7f2788873b80 10 client.37754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:37:22.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.490+0000 7f2788873b80 20 client.37754 populate_metadata read hostname 'smithi043' 2024-05-27T07:37:22.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.490+0000 7f2788873b80 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56050ebb75f0 0x56050ebb9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:22.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.490+0000 7f2788873b80 4 client.37754 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:37:22.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.490+0000 7f2788873b80 1 -- 172.21.15.43:0/3634266979 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56050ea5c570 con 0x56050ebb75f0 2024-05-27T07:37:22.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.491+0000 7f2772a00700 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56050ebb75f0 0x56050ebb9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:22.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.492+0000 7f2772a00700 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56050ebb75f0 0x56050ebb9a10 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8af775f6cce796d7 server_cookie=2eaf20bf1c61e765 in_seq=0 out_seq=0 2024-05-27T07:37:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.493+0000 7f275d600700 10 client.37754 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:37:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.493+0000 7f275d600700 1 -- 172.21.15.43:0/3634266979 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56050ea5c570 con 0x56050ebb75f0 2024-05-27T07:37:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.493+0000 7f275d600700 10 client.37754 handle_command_reply: tid=0 2024-05-27T07:37:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 client.37754 shutdown 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 -- 172.21.15.43:0/3634266979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050ea9f9f0 msgr2=0x56050eb408f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050ea9f9f0 0x56050eb408f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2760003770 tx=0x7f2760005690 comp rx=0 tx=0).stop 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 -- 172.21.15.43:0/3634266979 shutdown_connections 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56050ebb75f0 0x56050ebb9a10 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56050eb32e70 0x56050ebb59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56050e9b3e40 0x56050eb403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 --2- 172.21.15.43:0/3634266979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56050ea9f9f0 0x56050eb408f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 -- 172.21.15.43:0/3634266979 >> 172.21.15.43:0/3634266979 conn(0x56050ea4d390 msgr2=0x56050eb3e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f275d600700 0 client.37754 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 -- 172.21.15.43:0/3634266979 shutdown_connections 2024-05-27T07:37:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.494+0000 7f2788873b80 1 -- 172.21.15.43:0/3634266979 wait complete. 2024-05-27T07:37:22.492 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:37:22.492 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:37:22.492 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:37:22.492 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:37:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.495+0000 7f2788873b80 20 client.37754 trim_cache size 0 max 16384 2024-05-27T07:37:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 -- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27580b29f0 msgr2=0x7f27580b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27580b29f0 0x7f27580b4f00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2764009810 tx=0x7f2764005dc0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 -- 172.21.15.43:0/4283454009 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27741ab180 msgr2=0x7f27741ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27741ab180 0x7f27741ad5a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f276c00e3c0 tx=0x7f276c00e6d0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 -- 172.21.15.43:0/4283454009 shutdown_connections 2024-05-27T07:37:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f27580b29f0 0x7f27580b4f00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27741ab180 0x7f27741ad5a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2774108ef0 0x7f277410dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 --2- 172.21.15.43:0/4283454009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2774109840 0x7f277410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 -- 172.21.15.43:0/4283454009 >> 172.21.15.43:0/4283454009 conn(0x7f27740ffe90 msgr2=0x7f27741061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.496+0000 7f2779e00700 1 -- 172.21.15.43:0/4283454009 shutdown_connections 2024-05-27T07:37:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:22.497+0000 7f2779e00700 1 -- 172.21.15.43:0/4283454009 wait complete. 2024-05-27T07:37:22.828 INFO:tasks.workunit.client.0.smithi043.stdout: 10 61766 5.84 MB/sec execute 443 sec latency 107.830 ms 2024-05-27T07:37:23.096 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:22 smithi132 ceph-mon[35283]: pgmap v788: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.6 MiB/s wr, 160 op/s 2024-05-27T07:37:23.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:22 smithi043 ceph-mon[31531]: pgmap v788: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.6 MiB/s wr, 160 op/s 2024-05-27T07:37:23.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:22 smithi179 ceph-mon[35888]: pgmap v788: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.6 MiB/s wr, 160 op/s 2024-05-27T07:37:23.827 INFO:tasks.workunit.client.0.smithi043.stdout: 10 61874 5.85 MB/sec execute 444 sec latency 100.912 ms 2024-05-27T07:37:24.828 INFO:tasks.workunit.client.0.smithi043.stdout: 10 61984 5.85 MB/sec execute 445 sec latency 161.734 ms 2024-05-27T07:37:25.055 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:37:25.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:37:25.596 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:25 smithi132 ceph-mon[35283]: pgmap v789: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.2 MiB/s wr, 143 op/s 2024-05-27T07:37:25.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:25 smithi043 ceph-mon[31531]: pgmap v789: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.2 MiB/s wr, 143 op/s 2024-05-27T07:37:25.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:25 smithi179 ceph-mon[35888]: pgmap v789: 129 pgs: 129 active+clean; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.2 MiB/s wr, 143 op/s 2024-05-27T07:37:25.828 INFO:tasks.workunit.client.0.smithi043.stdout: 10 62062 5.84 MB/sec execute 446 sec latency 109.306 ms 2024-05-27T07:37:25.899 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:26.828 INFO:tasks.workunit.client.0.smithi043.stdout: 10 62185 5.85 MB/sec execute 447 sec latency 162.860 ms 2024-05-27T07:37:27.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:27 smithi043 ceph-mon[31531]: pgmap v790: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 5.2 MiB/s wr, 214 op/s 2024-05-27T07:37:27.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:27 smithi179 ceph-mon[35888]: pgmap v790: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 5.2 MiB/s wr, 214 op/s 2024-05-27T07:37:27.828 INFO:tasks.workunit.client.0.smithi043.stdout: 10 62307 5.86 MB/sec execute 448 sec latency 182.455 ms 2024-05-27T07:37:27.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:27 smithi132 ceph-mon[35283]: pgmap v790: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 5.2 MiB/s wr, 214 op/s 2024-05-27T07:37:28.828 INFO:tasks.workunit.client.0.smithi043.stdout: 10 62438 5.86 MB/sec execute 449 sec latency 113.262 ms 2024-05-27T07:37:29.649 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:29 smithi132 ceph-mon[35283]: pgmap v791: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 3.6 MiB/s wr, 140 op/s 2024-05-27T07:37:29.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:29 smithi043 ceph-mon[31531]: pgmap v791: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 3.6 MiB/s wr, 140 op/s 2024-05-27T07:37:29.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:29 smithi179 ceph-mon[35888]: pgmap v791: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 3.6 MiB/s wr, 140 op/s 2024-05-27T07:37:29.828 INFO:tasks.workunit.client.0.smithi043.stdout: 10 62539 5.86 MB/sec execute 450 sec latency 133.326 ms 2024-05-27T07:37:30.829 INFO:tasks.workunit.client.0.smithi043.stdout: 10 62656 5.87 MB/sec execute 451 sec latency 85.058 ms 2024-05-27T07:37:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.126+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3577580665 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4f01032d0 msgr2=0x7fe4f01036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.126+0000 7fe4f5a00700 1 --2- 172.21.15.43:0/3577580665 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4f01032d0 0x7fe4f01036e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00098f0 tx=0x7fe4e0009c00 comp rx=0 tx=0).stop 2024-05-27T07:37:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.126+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3577580665 shutdown_connections 2024-05-27T07:37:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.126+0000 7fe4f5a00700 1 --2- 172.21.15.43:0/3577580665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f0108550 0x7fe4f006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.126+0000 7fe4f5a00700 1 --2- 172.21.15.43:0/3577580665 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4f01032d0 0x7fe4f01036e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.126+0000 7fe4f5a00700 1 --2- 172.21.15.43:0/3577580665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4f0107c00 0x7fe4f0108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.126+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3577580665 >> 172.21.15.43:0/3577580665 conn(0x7fe4f0078ab0 msgr2=0x7fe4f0078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.126+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3577580665 shutdown_connections 2024-05-27T07:37:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.127+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3577580665 wait complete. 2024-05-27T07:37:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.128+0000 7fe4f5a00700 1 Processor -- start 2024-05-27T07:37:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.129+0000 7fe4f5a00700 1 -- start start 2024-05-27T07:37:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.130+0000 7fe4f5a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f01032d0 0x7fe4f01134d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.130+0000 7fe4f5a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4f0107c00 0x7fe4f0113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.130+0000 7fe4f5a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4f0108550 0x7fe4f0113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.130+0000 7fe4efe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f01032d0 0x7fe4f01134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.130+0000 7fe4efe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f01032d0 0x7fe4f01134d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51936/0 (socket says 172.21.15.43:51936) 2024-05-27T07:37:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.131+0000 7fe4efe00700 1 -- 172.21.15.43:0/3897483503 learned_addr learned my addr 172.21.15.43:0/3897483503 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.131+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f01164d0 con 0x7fe4f01032d0 2024-05-27T07:37:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.131+0000 7fe4ef400700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4f0107c00 0x7fe4f0113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.131+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f010fe20 con 0x7fe4f0108550 2024-05-27T07:37:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.131+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f0116650 con 0x7fe4f0107c00 2024-05-27T07:37:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.131+0000 7fe4f5000700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4f0108550 0x7fe4f0113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.132+0000 7fe4efe00700 1 -- 172.21.15.43:0/3897483503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4f0107c00 msgr2=0x7fe4f0113a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.132+0000 7fe4efe00700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4f0107c00 0x7fe4f0113a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.132+0000 7fe4efe00700 1 -- 172.21.15.43:0/3897483503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4f0108550 msgr2=0x7fe4f0113f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.132+0000 7fe4efe00700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4f0108550 0x7fe4f0113f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.132+0000 7fe4efe00700 1 -- 172.21.15.43:0/3897483503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f01ad3f0 con 0x7fe4f01032d0 2024-05-27T07:37:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.132+0000 7fe4f5000700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4f0108550 0x7fe4f0113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:37:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.133+0000 7fe4efe00700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f01032d0 0x7fe4f01134d0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fe4e000c010 tx=0x7fe4e000fa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.133+0000 7fe4ecc00700 1 -- 172.21.15.43:0/3897483503 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0027070 con 0x7fe4f01032d0 2024-05-27T07:37:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.133+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4f01ad680 con 0x7fe4f01032d0 2024-05-27T07:37:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.133+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4f01adb30 con 0x7fe4f01032d0 2024-05-27T07:37:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.133+0000 7fe4ecc00700 1 -- 172.21.15.43:0/3897483503 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe4e00054d0 con 0x7fe4f01032d0 2024-05-27T07:37:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.133+0000 7fe4ecc00700 1 -- 172.21.15.43:0/3897483503 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e000bd80 con 0x7fe4f01032d0 2024-05-27T07:37:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.135+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4d0005330 con 0x7fe4f01032d0 2024-05-27T07:37:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.138+0000 7fe4ecc00700 1 -- 172.21.15.43:0/3897483503 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe4e002a070 con 0x7fe4f01032d0 2024-05-27T07:37:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.142+0000 7fe4ecc00700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe4d40b29f0 0x7fe4d40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.142+0000 7fe4ef400700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe4d40b29f0 0x7fe4d40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.144+0000 7fe4ecc00700 1 -- 172.21.15.43:0/3897483503 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7fe4e00e3400 con 0x7fe4f01032d0 2024-05-27T07:37:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.144+0000 7fe4ef400700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe4d40b29f0 0x7fe4d40b4f00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe4f006b7b0 tx=0x7fe4dc00b410 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.148+0000 7fe4ecc00700 1 -- 172.21.15.43:0/3897483503 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe4e0013030 con 0x7fe4f01032d0 2024-05-27T07:37:31.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:31 smithi043 ceph-mon[31531]: pgmap v792: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 3.6 MiB/s wr, 140 op/s 2024-05-27T07:37:31.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.623+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4d00058f0 con 0x7fe4f01032d0 2024-05-27T07:37:31.622 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:37:31.622 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":146,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:37:11.295449+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34484,34448,34424,24484,34403]},"id":1}]} 2024-05-27T07:37:31.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.624+0000 7fe4ecc00700 1 -- 172.21.15.43:0/3897483503 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6121 (secure 0 0 0) 0x7fe4e00aaa40 con 0x7fe4f01032d0 2024-05-27T07:37:31.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.628+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe4d40b29f0 msgr2=0x7fe4d40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:31.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.628+0000 7fe4f5a00700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe4d40b29f0 0x7fe4d40b4f00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe4f006b7b0 tx=0x7fe4dc00b410 comp rx=0 tx=0).stop 2024-05-27T07:37:31.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.628+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f01032d0 msgr2=0x7fe4f01134d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:31.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.628+0000 7fe4f5a00700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f01032d0 0x7fe4f01134d0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fe4e000c010 tx=0x7fe4e000fa90 comp rx=0 tx=0).stop 2024-05-27T07:37:31.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.629+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 shutdown_connections 2024-05-27T07:37:31.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.629+0000 7fe4f5a00700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe4d40b29f0 0x7fe4d40b4f00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:31.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.629+0000 7fe4f5a00700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4f0108550 0x7fe4f0113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:31.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.629+0000 7fe4f5a00700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4f01032d0 0x7fe4f01134d0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:31.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.629+0000 7fe4f5a00700 1 --2- 172.21.15.43:0/3897483503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4f0107c00 0x7fe4f0113a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:31.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.629+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 >> 172.21.15.43:0/3897483503 conn(0x7fe4f0078ab0 msgr2=0x7fe4f0101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:31.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.629+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 shutdown_connections 2024-05-27T07:37:31.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:31.629+0000 7fe4f5a00700 1 -- 172.21.15.43:0/3897483503 wait complete. 2024-05-27T07:37:31.629 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 146 2024-05-27T07:37:31.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:31 smithi179 ceph-mon[35888]: pgmap v792: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 3.6 MiB/s wr, 140 op/s 2024-05-27T07:37:31.828 INFO:tasks.workunit.client.0.smithi043.stdout: 10 62734 5.86 MB/sec execute 452 sec latency 101.113 ms 2024-05-27T07:37:31.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:31 smithi132 ceph-mon[35283]: pgmap v792: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 3.6 MiB/s wr, 140 op/s 2024-05-27T07:37:32.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:37:32.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:32 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3897483503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:32.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:32 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3897483503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:32.828 INFO:tasks.workunit.client.0.smithi043.stdout: 10 62854 5.87 MB/sec execute 453 sec latency 86.899 ms 2024-05-27T07:37:32.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:32 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3897483503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:37:33.165 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:33.662 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:33 smithi043 ceph-mon[31531]: pgmap v793: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 5.7 MiB/s wr, 214 op/s 2024-05-27T07:37:33.717 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:33 smithi179 ceph-mon[35888]: pgmap v793: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 5.7 MiB/s wr, 214 op/s 2024-05-27T07:37:33.828 INFO:tasks.workunit.client.0.smithi043.stdout: 10 62964 5.88 MB/sec execute 454 sec latency 87.011 ms 2024-05-27T07:37:33.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:33 smithi132 ceph-mon[35283]: pgmap v793: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 5.7 MiB/s wr, 214 op/s 2024-05-27T07:37:34.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:34 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:37:34.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:34 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:37:34.829 INFO:tasks.workunit.client.0.smithi043.stdout: 10 63049 5.87 MB/sec execute 455 sec latency 136.885 ms 2024-05-27T07:37:34.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:34 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:37:35.716 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:35 smithi043 ceph-mon[31531]: pgmap v794: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 4.0 MiB/s wr, 144 op/s 2024-05-27T07:37:35.716 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:35 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:37:35.725 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:35 smithi179 ceph-mon[35888]: pgmap v794: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 4.0 MiB/s wr, 144 op/s 2024-05-27T07:37:35.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:35 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:37:35.829 INFO:tasks.workunit.client.0.smithi043.stdout: 10 63128 5.86 MB/sec execute 456 sec latency 128.271 ms 2024-05-27T07:37:35.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:35 smithi132 ceph-mon[35283]: pgmap v794: 129 pgs: 129 active+clean; 1.3 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 4.0 MiB/s wr, 144 op/s 2024-05-27T07:37:35.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:35 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:37:36.829 INFO:tasks.workunit.client.0.smithi043.stdout: 10 63231 5.87 MB/sec execute 457 sec latency 135.075 ms 2024-05-27T07:37:37.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.441+0000 7f766ca00700 1 -- 172.21.15.43:0/646518331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7668105300 msgr2=0x7f7668105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.441+0000 7f766ca00700 1 --2- 172.21.15.43:0/646518331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7668105300 0x7f7668105710 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f7658009990 tx=0x7f7658009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.441+0000 7f766ca00700 1 -- 172.21.15.43:0/646518331 shutdown_connections 2024-05-27T07:37:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.441+0000 7f766ca00700 1 --2- 172.21.15.43:0/646518331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766810e730 0x7f7668110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.441+0000 7f766ca00700 1 --2- 172.21.15.43:0/646518331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7668105300 0x7f7668105710 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.441+0000 7f766ca00700 1 --2- 172.21.15.43:0/646518331 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7668105c50 0x7f766810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.441+0000 7f766ca00700 1 -- 172.21.15.43:0/646518331 >> 172.21.15.43:0/646518331 conn(0x7f7668078ab0 msgr2=0x7f7668078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.441+0000 7f766ca00700 1 -- 172.21.15.43:0/646518331 shutdown_connections 2024-05-27T07:37:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.441+0000 7f766ca00700 1 -- 172.21.15.43:0/646518331 wait complete. 2024-05-27T07:37:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.443+0000 7f766ca00700 1 Processor -- start 2024-05-27T07:37:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.443+0000 7f766ca00700 1 -- start start 2024-05-27T07:37:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f766ca00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7668105300 0x7f76681a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f766ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7668105c50 0x7f76681a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f766ca00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766810e730 0x7f76681acea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f766ca00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7668114140 con 0x7f7668105c50 2024-05-27T07:37:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f766ca00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7668113fc0 con 0x7f7668105300 2024-05-27T07:37:37.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f766ca00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f76681142c0 con 0x7f766810e730 2024-05-27T07:37:37.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f7666800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7668105300 0x7f76681a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f7666800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7668105300 0x7f76681a1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:51780/0 (socket says 172.21.15.43:51780) 2024-05-27T07:37:37.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f7666800700 1 -- 172.21.15.43:0/476259536 learned_addr learned my addr 172.21.15.43:0/476259536 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:37.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f7667200700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766810e730 0x7f76681acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.444+0000 7f7665e00700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7668105c50 0x7f76681a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.445+0000 7f7666800700 1 -- 172.21.15.43:0/476259536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766810e730 msgr2=0x7f76681acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:37.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.445+0000 7f7666800700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766810e730 0x7f76681acea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.445+0000 7f7666800700 1 -- 172.21.15.43:0/476259536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7668105c50 msgr2=0x7f76681a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:37.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.445+0000 7f7666800700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7668105c50 0x7f76681a2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.445+0000 7f7666800700 1 -- 172.21.15.43:0/476259536 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76681ad3e0 con 0x7f7668105300 2024-05-27T07:37:37.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.446+0000 7f7666800700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7668105300 0x7f76681a1e20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f76580098e0 tx=0x7f76580057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:37.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.446+0000 7f765f400700 1 -- 172.21.15.43:0/476259536 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658027070 con 0x7f7668105300 2024-05-27T07:37:37.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.447+0000 7f766ca00700 1 -- 172.21.15.43:0/476259536 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76681ad5b0 con 0x7f7668105300 2024-05-27T07:37:37.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.447+0000 7f766ca00700 1 -- 172.21.15.43:0/476259536 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76681ada90 con 0x7f7668105300 2024-05-27T07:37:37.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.447+0000 7f765f400700 1 -- 172.21.15.43:0/476259536 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7658022470 con 0x7f7668105300 2024-05-27T07:37:37.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.447+0000 7f765f400700 1 -- 172.21.15.43:0/476259536 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658017650 con 0x7f7668105300 2024-05-27T07:37:37.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.452+0000 7f765f400700 1 -- 172.21.15.43:0/476259536 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f765802a070 con 0x7f7668105300 2024-05-27T07:37:37.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.453+0000 7f765f400700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f764c0b2a70 0x7f764c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.454+0000 7f765f400700 1 -- 172.21.15.43:0/476259536 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f76580e3850 con 0x7f7668105300 2024-05-27T07:37:37.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.454+0000 7f7665e00700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f764c0b2a70 0x7f764c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.455+0000 7f7665e00700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f764c0b2a70 0x7f764c0b4f80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7668103750 tx=0x7f7654009260 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:37.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.484+0000 7f767bfa5b80 1 Processor -- start 2024-05-27T07:37:37.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.484+0000 7f767bfa5b80 1 -- start start 2024-05-27T07:37:37.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.485+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x5650731ffed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.485+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200410 0x56507320aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.485+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200820 0x56507320d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.485+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56507312ee40 con 0x565073081e40 2024-05-27T07:37:37.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.485+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56507311e9e0 con 0x565073200410 2024-05-27T07:37:37.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.485+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x565073122a20 con 0x565073200820 2024-05-27T07:37:37.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.485+0000 7f7666800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x5650731ffed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.485+0000 7f7666800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x5650731ffed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34534/0 (socket says 172.21.15.43:34534) 2024-05-27T07:37:37.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.485+0000 7f7666800700 1 -- 172.21.15.43:0/258480186 learned_addr learned my addr 172.21.15.43:0/258480186 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:37.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.486+0000 7f7665e00700 1 --2- 172.21.15.43:0/258480186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200410 0x56507320aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.486+0000 7f7667200700 1 --2- 172.21.15.43:0/258480186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200820 0x56507320d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.486+0000 7f7666800700 1 -- 172.21.15.43:0/258480186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200820 msgr2=0x56507320d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.486+0000 7f7666800700 1 --2- 172.21.15.43:0/258480186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200820 0x56507320d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.486+0000 7f7666800700 1 -- 172.21.15.43:0/258480186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200410 msgr2=0x56507320aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.486+0000 7f7666800700 1 --2- 172.21.15.43:0/258480186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200410 0x56507320aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.486+0000 7f7666800700 1 -- 172.21.15.43:0/258480186 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650730e7420 con 0x565073081e40 2024-05-27T07:37:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.487+0000 7f7666800700 1 --2- 172.21.15.43:0/258480186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x5650731ffed0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f76580a6870 tx=0x7f76580aa030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600f200201c1f15b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:37.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.487+0000 7f7653e00700 1 -- 172.21.15.43:0/258480186 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76580aa7f0 con 0x565073081e40 2024-05-27T07:37:37.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.487+0000 7f7653e00700 1 -- 172.21.15.43:0/258480186 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f76580aeba0 con 0x565073081e40 2024-05-27T07:37:37.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.487+0000 7f7653e00700 1 -- 172.21.15.43:0/258480186 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76580aacb0 con 0x565073081e40 2024-05-27T07:37:37.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.488+0000 7f767bfa5b80 1 -- 172.21.15.43:0/258480186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 msgr2=0x5650731ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:37.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.488+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/258480186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x5650731ffed0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f76580a6870 tx=0x7f76580aa030 comp rx=0 tx=0).stop 2024-05-27T07:37:37.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.489+0000 7f767bfa5b80 1 -- 172.21.15.43:0/258480186 shutdown_connections 2024-05-27T07:37:37.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.489+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/258480186 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200820 0x56507320d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.489+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/258480186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x5650731ffed0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.489+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/258480186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200410 0x56507320aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.489+0000 7f767bfa5b80 1 -- 172.21.15.43:0/258480186 >> 172.21.15.43:0/258480186 conn(0x5650730e1fc0 msgr2=0x5650731fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:37.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.489+0000 7f767bfa5b80 1 -- 172.21.15.43:0/258480186 shutdown_connections 2024-05-27T07:37:37.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.489+0000 7f767bfa5b80 1 -- 172.21.15.43:0/258480186 wait complete. 2024-05-27T07:37:37.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.489+0000 7f767bfa5b80 1 -- start start 2024-05-27T07:37:37.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.490+0000 7f767bfa5b80 1 Processor -- start 2024-05-27T07:37:37.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.490+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x56507320a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 0x56507320ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200820 0x56507320b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x565073024fa0 con 0x565073081e40 2024-05-27T07:37:37.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5650730bf760 con 0x565073200820 2024-05-27T07:37:37.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56507320d750 con 0x565073200410 2024-05-27T07:37:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f767bfa5b80 10 client.0 _ll_register_callbacks cb 0x5650730c8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f7665e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 0x56507320ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f7665e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 0x56507320ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:42040/0 (socket says 172.21.15.43:42040) 2024-05-27T07:37:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f7665e00700 1 -- 172.21.15.43:0/1385315100 learned_addr learned my addr 172.21.15.43:0/1385315100 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:37.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f7666800700 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x56507320a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.491+0000 7f7667200700 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200820 0x56507320b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.492+0000 7f7665e00700 1 -- 172.21.15.43:0/1385315100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200820 msgr2=0x56507320b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:37.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.492+0000 7f7665e00700 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200820 0x56507320b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.492+0000 7f7665e00700 1 -- 172.21.15.43:0/1385315100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 msgr2=0x56507320a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:37.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.492+0000 7f7665e00700 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x56507320a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.492+0000 7f7665e00700 1 -- 172.21.15.43:0/1385315100 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56507306e8d0 con 0x565073200410 2024-05-27T07:37:37.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.492+0000 7f7666800700 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x56507320a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:37:37.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.492+0000 7f7667200700 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200820 0x56507320b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:37.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.492+0000 7f7665e00700 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 0x56507320ab00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7654004580 tx=0x7f76540047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84575a32746a3d64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:37.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.492+0000 7f7650c00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:37:37.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.493+0000 7f7650c00700 1 -- 172.21.15.43:0/1385315100 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654027420 con 0x565073200410 2024-05-27T07:37:37.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.493+0000 7f7650c00700 1 -- 172.21.15.43:0/1385315100 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f765401d410 con 0x565073200410 2024-05-27T07:37:37.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.493+0000 7f767bfa5b80 1 -- 172.21.15.43:0/1385315100 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76580aeba0 con 0x565073200410 2024-05-27T07:37:37.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.494+0000 7f7650c00700 1 -- 172.21.15.43:0/1385315100 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f765401d610 con 0x565073200410 2024-05-27T07:37:37.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.494+0000 7f767bfa5b80 10 client.27799 fetch_fsmap learned FSMap version 147 2024-05-27T07:37:37.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.494+0000 7f767bfa5b80 1 -- 172.21.15.43:0/1385315100 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x5650730ef060 con 0x565073200410 2024-05-27T07:37:37.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.494+0000 7f7650c00700 1 -- 172.21.15.43:0/1385315100 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f76580aeba0 con 0x565073200410 2024-05-27T07:37:37.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.495+0000 7f7650c00700 1 -- 172.21.15.43:0/1385315100 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f7654027be0 con 0x565073200410 2024-05-27T07:37:37.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.495+0000 7f767bfa5b80 10 client.27799 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:37:37.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.495+0000 7f767bfa5b80 10 client.27799 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:37:37.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.495+0000 7f767bfa5b80 20 client.27799 populate_metadata read hostname 'smithi043' 2024-05-27T07:37:37.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.495+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56507321ab50 0x56507321cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.495+0000 7f767bfa5b80 4 client.27799 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:37.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.495+0000 7f767bfa5b80 1 -- 172.21.15.43:0/1385315100 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565073130d90 con 0x56507321ab50 2024-05-27T07:37:37.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.496+0000 7f7666800700 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56507321ab50 0x56507321cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.496+0000 7f7666800700 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56507321ab50 0x56507321cfe0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ff5e7901a93ed8a9 server_cookie=a23d822fdf68a3d6 in_seq=0 out_seq=0 2024-05-27T07:37:37.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.496+0000 7f7650c00700 10 client.27799 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:37:37.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.497+0000 7f7650c00700 1 -- 172.21.15.43:0/1385315100 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x565073130d90 con 0x56507321ab50 2024-05-27T07:37:37.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.497+0000 7f7650c00700 10 client.27799 handle_command_reply: tid=0 2024-05-27T07:37:37.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.497+0000 7f767bfa5b80 1 client.27799 shutdown 2024-05-27T07:37:37.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f767bfa5b80 1 -- 172.21.15.43:0/1385315100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 msgr2=0x56507320ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:37.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 0x56507320ab00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7654004580 tx=0x7f76540047c0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f767bfa5b80 1 -- 172.21.15.43:0/1385315100 shutdown_connections 2024-05-27T07:37:37.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56507321ab50 0x56507321cfe0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200820 0x56507320b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565073081e40 0x56507320a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/1385315100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 0x56507320ab00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f767bfa5b80 1 -- 172.21.15.43:0/1385315100 >> 172.21.15.43:0/1385315100 conn(0x5650730e1fc0 msgr2=0x5650731faff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:37.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f7650c00700 0 client.27799 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:37:37.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f767bfa5b80 1 -- 172.21.15.43:0/1385315100 shutdown_connections 2024-05-27T07:37:37.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.498+0000 7f767bfa5b80 1 -- 172.21.15.43:0/1385315100 wait complete. 2024-05-27T07:37:37.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.499+0000 7f767bfa5b80 20 client.27799 trim_cache size 0 max 16384 2024-05-27T07:37:37.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.527+0000 7f767bfa5b80 1 Processor -- start 2024-05-27T07:37:37.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.527+0000 7f767bfa5b80 1 -- start start 2024-05-27T07:37:37.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.527+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073205e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.527+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073081e40 0x56507320abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.527+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 0x5650732047a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.527+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f764800f250 con 0x5650730e1fc0 2024-05-27T07:37:37.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.527+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f764800d710 con 0x565073081e40 2024-05-27T07:37:37.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.527+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f764800eb80 con 0x565073200410 2024-05-27T07:37:37.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.528+0000 7f7666800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073205e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.528+0000 7f7666800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073205e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34558/0 (socket says 172.21.15.43:34558) 2024-05-27T07:37:37.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.528+0000 7f7666800700 1 -- 172.21.15.43:0/307229409 learned_addr learned my addr 172.21.15.43:0/307229409 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:37.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.528+0000 7f7667200700 1 --2- 172.21.15.43:0/307229409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 0x5650732047a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.528+0000 7f7665e00700 1 --2- 172.21.15.43:0/307229409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073081e40 0x56507320abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.528+0000 7f7666800700 1 -- 172.21.15.43:0/307229409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 msgr2=0x5650732047a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:37.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.528+0000 7f7666800700 1 --2- 172.21.15.43:0/307229409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 0x5650732047a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.528+0000 7f7666800700 1 -- 172.21.15.43:0/307229409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073081e40 msgr2=0x56507320abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:37.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.528+0000 7f7666800700 1 --2- 172.21.15.43:0/307229409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073081e40 0x56507320abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.529+0000 7f7666800700 1 -- 172.21.15.43:0/307229409 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650731fd350 con 0x5650730e1fc0 2024-05-27T07:37:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.529+0000 7f7666800700 1 --2- 172.21.15.43:0/307229409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073205e30 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f765800fe20 tx=0x7f76580b3050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f6483916297669 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.530+0000 7f7653400700 1 -- 172.21.15.43:0/307229409 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76580a9f60 con 0x5650730e1fc0 2024-05-27T07:37:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.530+0000 7f7653400700 1 -- 172.21.15.43:0/307229409 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f76580b10b0 con 0x5650730e1fc0 2024-05-27T07:37:37.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.530+0000 7f7653400700 1 -- 172.21.15.43:0/307229409 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76580aacb0 con 0x5650730e1fc0 2024-05-27T07:37:37.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.531+0000 7f767bfa5b80 1 -- 172.21.15.43:0/307229409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 msgr2=0x565073205e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.531+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/307229409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073205e30 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f765800fe20 tx=0x7f76580b3050 comp rx=0 tx=0).stop 2024-05-27T07:37:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.531+0000 7f767bfa5b80 1 -- 172.21.15.43:0/307229409 shutdown_connections 2024-05-27T07:37:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.531+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/307229409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073200410 0x5650732047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.531+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/307229409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073205e30 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.531+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/307229409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073081e40 0x56507320abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.531+0000 7f767bfa5b80 1 -- 172.21.15.43:0/307229409 >> 172.21.15.43:0/307229409 conn(0x5650731f4f60 msgr2=0x5650731fba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.531+0000 7f767bfa5b80 1 -- 172.21.15.43:0/307229409 shutdown_connections 2024-05-27T07:37:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.531+0000 7f767bfa5b80 1 -- 172.21.15.43:0/307229409 wait complete. 2024-05-27T07:37:37.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.532+0000 7f767bfa5b80 1 -- start start 2024-05-27T07:37:37.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.532+0000 7f767bfa5b80 1 Processor -- start 2024-05-27T07:37:37.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.533+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073081e40 0x565073212c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.533+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073213160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.533+0000 7f767bfa5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200410 0x56507320e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.533+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f764800e4b0 con 0x5650730e1fc0 2024-05-27T07:37:37.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.533+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f764800dde0 con 0x565073200410 2024-05-27T07:37:37.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.533+0000 7f767bfa5b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56507320d3c0 con 0x565073081e40 2024-05-27T07:37:37.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.533+0000 7f767bfa5b80 10 client.0 _ll_register_callbacks cb 0x5650730c8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:37.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.534+0000 7f7665e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073213160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.534+0000 7f7665e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073213160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34560/0 (socket says 172.21.15.43:34560) 2024-05-27T07:37:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.534+0000 7f7665e00700 1 -- 172.21.15.43:0/2044267441 learned_addr learned my addr 172.21.15.43:0/2044267441 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.534+0000 7f7666800700 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073081e40 0x565073212c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.534+0000 7f7667200700 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200410 0x56507320e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.534+0000 7f7665e00700 1 -- 172.21.15.43:0/2044267441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073081e40 msgr2=0x565073212c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.534+0000 7f7665e00700 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073081e40 0x565073212c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.534+0000 7f7665e00700 1 -- 172.21.15.43:0/2044267441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200410 msgr2=0x56507320e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.534+0000 7f7665e00700 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200410 0x56507320e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.535+0000 7f7665e00700 1 -- 172.21.15.43:0/2044267441 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56507320e7b0 con 0x5650730e1fc0 2024-05-27T07:37:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.535+0000 7f7665e00700 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073213160 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f7654016040 tx=0x7f765401dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad26d5faecc7d63 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.535+0000 7f7650c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:37:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.536+0000 7f7650c00700 1 -- 172.21.15.43:0/2044267441 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765401d410 con 0x5650730e1fc0 2024-05-27T07:37:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.536+0000 7f7650c00700 1 -- 172.21.15.43:0/2044267441 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f7654023040 con 0x5650730e1fc0 2024-05-27T07:37:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.536+0000 7f767bfa5b80 1 -- 172.21.15.43:0/2044267441 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56507312a570 con 0x5650730e1fc0 2024-05-27T07:37:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.536+0000 7f7650c00700 1 -- 172.21.15.43:0/2044267441 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f7654031c60 con 0x5650730e1fc0 2024-05-27T07:37:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.537+0000 7f767bfa5b80 10 client.18393 fetch_fsmap learned FSMap version 147 2024-05-27T07:37:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.537+0000 7f767bfa5b80 1 -- 172.21.15.43:0/2044267441 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x565073285160 con 0x5650730e1fc0 2024-05-27T07:37:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.537+0000 7f7650c00700 1 -- 172.21.15.43:0/2044267441 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x56507312a570 con 0x5650730e1fc0 2024-05-27T07:37:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.537+0000 7f7650c00700 1 -- 172.21.15.43:0/2044267441 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f765401ac70 con 0x5650730e1fc0 2024-05-27T07:37:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.538+0000 7f767bfa5b80 10 client.18393 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:37:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.538+0000 7f767bfa5b80 10 client.18393 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:37:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.538+0000 7f767bfa5b80 20 client.18393 populate_metadata read hostname 'smithi043' 2024-05-27T07:37:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.538+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x565073200820 0x565073287880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.538+0000 7f767bfa5b80 4 client.18393 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:37:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.538+0000 7f767bfa5b80 1 -- 172.21.15.43:0/2044267441 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565072fa1250 con 0x565073200820 2024-05-27T07:37:37.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.539+0000 7f7666800700 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x565073200820 0x565073287880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:37.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.541+0000 7f7666800700 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x565073200820 0x565073287880 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c2893266f0765d29 server_cookie=770568512bf39e64 in_seq=0 out_seq=0 2024-05-27T07:37:37.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.541+0000 7f7650c00700 10 client.18393 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:37:37.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.542+0000 7f7650c00700 1 -- 172.21.15.43:0/2044267441 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565072fa1250 con 0x565073200820 2024-05-27T07:37:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.542+0000 7f7650c00700 10 client.18393 handle_command_reply: tid=0 2024-05-27T07:37:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.542+0000 7f767bfa5b80 1 client.18393 shutdown 2024-05-27T07:37:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.542+0000 7f767bfa5b80 1 -- 172.21.15.43:0/2044267441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 msgr2=0x565073213160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.542+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073213160 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f7654016040 tx=0x7f765401dd30 comp rx=0 tx=0).stop 2024-05-27T07:37:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.543+0000 7f767bfa5b80 1 -- 172.21.15.43:0/2044267441 shutdown_connections 2024-05-27T07:37:37.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.543+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x565073200820 0x565073287880 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.543+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565073200410 0x56507320e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.543+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565073081e40 0x565073212c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.543+0000 7f767bfa5b80 1 --2- 172.21.15.43:0/2044267441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5650730e1fc0 0x565073213160 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.543+0000 7f767bfa5b80 1 -- 172.21.15.43:0/2044267441 >> 172.21.15.43:0/2044267441 conn(0x5650731f4f60 msgr2=0x56507320ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.543+0000 7f7650c00700 0 client.18393 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:37:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.543+0000 7f767bfa5b80 1 -- 172.21.15.43:0/2044267441 shutdown_connections 2024-05-27T07:37:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.543+0000 7f767bfa5b80 1 -- 172.21.15.43:0/2044267441 wait complete. 2024-05-27T07:37:37.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.544+0000 7f767bfa5b80 20 client.18393 trim_cache size 0 max 16384 2024-05-27T07:37:37.542 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:37:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.545+0000 7f766ca00700 1 -- 172.21.15.43:0/476259536 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f764c0b2a70 msgr2=0x7f764c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.545+0000 7f766ca00700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f764c0b2a70 0x7f764c0b4f80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7668103750 tx=0x7f7654009260 comp rx=0 tx=0).stop 2024-05-27T07:37:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.545+0000 7f766ca00700 1 -- 172.21.15.43:0/476259536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7668105300 msgr2=0x7f76681a1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.545+0000 7f766ca00700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7668105300 0x7f76681a1e20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f76580098e0 tx=0x7f76580057c0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.546+0000 7f766ca00700 1 -- 172.21.15.43:0/476259536 shutdown_connections 2024-05-27T07:37:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.546+0000 7f766ca00700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f764c0b2a70 0x7f764c0b4f80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.546+0000 7f766ca00700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766810e730 0x7f76681acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.546+0000 7f766ca00700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7668105300 0x7f76681a1e20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.546+0000 7f766ca00700 1 --2- 172.21.15.43:0/476259536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7668105c50 0x7f76681a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.546+0000 7f766ca00700 1 -- 172.21.15.43:0/476259536 >> 172.21.15.43:0/476259536 conn(0x7f7668078ab0 msgr2=0x7f7668104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.546+0000 7f766ca00700 1 -- 172.21.15.43:0/476259536 shutdown_connections 2024-05-27T07:37:37.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:37.546+0000 7f766ca00700 1 -- 172.21.15.43:0/476259536 wait complete. 2024-05-27T07:37:37.596 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:37 smithi132 ceph-mon[35283]: pgmap v795: 129 pgs: 129 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 6.1 MiB/s wr, 219 op/s 2024-05-27T07:37:37.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:37 smithi043 ceph-mon[31531]: pgmap v795: 129 pgs: 129 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 6.1 MiB/s wr, 219 op/s 2024-05-27T07:37:37.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:37 smithi179 ceph-mon[35888]: pgmap v795: 129 pgs: 129 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 6.1 MiB/s wr, 219 op/s 2024-05-27T07:37:37.829 INFO:tasks.workunit.client.0.smithi043.stdout: 10 63320 5.86 MB/sec execute 458 sec latency 112.283 ms 2024-05-27T07:37:38.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:37:38.829 INFO:tasks.workunit.client.0.smithi043.stdout: 10 63417 5.85 MB/sec execute 459 sec latency 97.123 ms 2024-05-27T07:37:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:39.586 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:39 smithi132 ceph-mon[35283]: pgmap v796: 129 pgs: 129 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 4.1 MiB/s wr, 148 op/s 2024-05-27T07:37:39.662 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:39 smithi043 ceph-mon[31531]: pgmap v796: 129 pgs: 129 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 4.1 MiB/s wr, 148 op/s 2024-05-27T07:37:39.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:39 smithi179 ceph-mon[35888]: pgmap v796: 129 pgs: 129 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 4.1 MiB/s wr, 148 op/s 2024-05-27T07:37:39.829 INFO:tasks.workunit.client.0.smithi043.stdout: 10 63509 5.84 MB/sec execute 460 sec latency 144.382 ms 2024-05-27T07:37:40.829 INFO:tasks.workunit.client.0.smithi043.stdout: 10 63620 5.84 MB/sec execute 461 sec latency 105.237 ms 2024-05-27T07:37:41.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:41 smithi043 ceph-mon[31531]: pgmap v797: 129 pgs: 129 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 4.1 MiB/s wr, 148 op/s 2024-05-27T07:37:41.714 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:41 smithi179 ceph-mon[35888]: pgmap v797: 129 pgs: 129 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 4.1 MiB/s wr, 148 op/s 2024-05-27T07:37:41.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 63731 5.83 MB/sec execute 462 sec latency 96.220 ms 2024-05-27T07:37:41.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:41 smithi132 ceph-mon[35283]: pgmap v797: 129 pgs: 129 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 4.1 MiB/s wr, 148 op/s 2024-05-27T07:37:42.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 63839 5.83 MB/sec execute 463 sec latency 104.954 ms 2024-05-27T07:37:43.818 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:43 smithi043 ceph-mon[31531]: pgmap v798: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 5.7 MiB/s wr, 220 op/s 2024-05-27T07:37:43.818 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:43 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:43.818 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:43 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.818+0000 7f7945200700 1 -- 172.21.15.43:0/3317073544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7940109520 msgr2=0x7f79401099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.818+0000 7f7945200700 1 --2- 172.21.15.43:0/3317073544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7940109520 0x7f79401099f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7938009990 tx=0x7f7938009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.821+0000 7f7945200700 1 -- 172.21.15.43:0/3317073544 shutdown_connections 2024-05-27T07:37:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.821+0000 7f7945200700 1 --2- 172.21.15.43:0/3317073544 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7940109520 0x7f79401099f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.822+0000 7f7945200700 1 --2- 172.21.15.43:0/3317073544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7940104520 0x7f7940104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.822+0000 7f7945200700 1 --2- 172.21.15.43:0/3317073544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7940104e70 0x7f7940108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.822+0000 7f7945200700 1 -- 172.21.15.43:0/3317073544 >> 172.21.15.43:0/3317073544 conn(0x7f7940100030 msgr2=0x7f7940102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.822+0000 7f7945200700 1 -- 172.21.15.43:0/3317073544 shutdown_connections 2024-05-27T07:37:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.822+0000 7f7945200700 1 -- 172.21.15.43:0/3317073544 wait complete. 2024-05-27T07:37:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.825+0000 7f7945200700 1 Processor -- start 2024-05-27T07:37:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.826+0000 7f7945200700 1 -- start start 2024-05-27T07:37:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.826+0000 7f7945200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7940104520 0x7f79401a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.827+0000 7f7945200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7940104e70 0x7f79401a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.827+0000 7f793de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7940104520 0x7f79401a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.827+0000 7f7945200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7940109520 0x7f79401a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.827+0000 7f793de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7940104520 0x7f79401a1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:51848/0 (socket says 172.21.15.43:51848) 2024-05-27T07:37:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.827+0000 7f7935400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7940104e70 0x7f79401a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.828+0000 7f7935400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7940104e70 0x7f79401a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:42072/0 (socket says 172.21.15.43:42072) 2024-05-27T07:37:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.828+0000 7f793de00700 1 -- 172.21.15.43:0/3537082062 learned_addr learned my addr 172.21.15.43:0/3537082062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.828+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f794010e030 con 0x7f7940109520 2024-05-27T07:37:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.828+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f794010deb0 con 0x7f7940104520 2024-05-27T07:37:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.828+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f794010e1b0 con 0x7f7940104e70 2024-05-27T07:37:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.828+0000 7f793e800700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7940109520 0x7f79401a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.828+0000 7f793de00700 1 -- 172.21.15.43:0/3537082062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7940104e70 msgr2=0x7f79401a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.829+0000 7f793de00700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7940104e70 0x7f79401a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.829+0000 7f793de00700 1 -- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7940109520 msgr2=0x7f79401a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.829+0000 7f793de00700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7940109520 0x7f79401a5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.829+0000 7f793de00700 1 -- 172.21.15.43:0/3537082062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79401a6240 con 0x7f7940104520 2024-05-27T07:37:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.829+0000 7f793e800700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7940109520 0x7f79401a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.829+0000 7f793de00700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7940104520 0x7f79401a1370 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f793000ef30 tx=0x7f793000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.830+0000 7f7937400700 1 -- 172.21.15.43:0/3537082062 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79300092e0 con 0x7f7940104520 2024-05-27T07:37:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.830+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79401a64d0 con 0x7f7940104520 2024-05-27T07:37:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.830+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79401ade50 con 0x7f7940104520 2024-05-27T07:37:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.830+0000 7f7937400700 1 -- 172.21.15.43:0/3537082062 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7930004bb0 con 0x7f7940104520 2024-05-27T07:37:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.830+0000 7f7937400700 1 -- 172.21.15.43:0/3537082062 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7930004020 con 0x7f7940104520 2024-05-27T07:37:43.829 INFO:tasks.workunit.client.0.smithi043.stdout: 10 63939 5.83 MB/sec execute 464 sec latency 88.647 ms 2024-05-27T07:37:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.837+0000 7f7937400700 1 -- 172.21.15.43:0/3537082062 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f793001b030 con 0x7f7940104520 2024-05-27T07:37:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.839+0000 7f7937400700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79240b2af0 0x7f79240b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.839+0000 7f7937400700 1 -- 172.21.15.43:0/3537082062 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f79300e33d0 con 0x7f7940104520 2024-05-27T07:37:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.840+0000 7f7935400700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79240b2af0 0x7f79240b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:43 smithi132 ceph-mon[35283]: pgmap v798: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 5.7 MiB/s wr, 220 op/s 2024-05-27T07:37:43.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:43 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:43.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:43 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:43.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.872+0000 7f7935400700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79240b2af0 0x7f79240b5000 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f79400775b0 tx=0x7f79280073d0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:43.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.875+0000 7f7953c4bb80 1 Processor -- start 2024-05-27T07:37:43.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.876+0000 7f7953c4bb80 1 -- start start 2024-05-27T07:37:43.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.877+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00acda9f0 0x55b00ad6d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.877+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 0x55b00ad6d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.877+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00ad6de70 0x55b00ad7a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.877+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b00ac9be40 con 0x55b00abeee40 2024-05-27T07:37:43.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.877+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b00ac8b9e0 con 0x55b00acda9f0 2024-05-27T07:37:43.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.877+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b00ac8fa20 con 0x55b00ad6de70 2024-05-27T07:37:43.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.878+0000 7f793de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00acda9f0 0x55b00ad6d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.878+0000 7f793de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00acda9f0 0x55b00ad6d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:51856/0 (socket says 172.21.15.43:51856) 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.878+0000 7f793de00700 1 -- 172.21.15.43:0/3486791616 learned_addr learned my addr 172.21.15.43:0/3486791616 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.878+0000 7f7935400700 1 --2- 172.21.15.43:0/3486791616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 0x55b00ad6d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.878+0000 7f793de00700 1 -- 172.21.15.43:0/3486791616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00ad6de70 msgr2=0x55b00ad7a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.878+0000 7f793de00700 1 --2- 172.21.15.43:0/3486791616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00ad6de70 0x55b00ad7a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.878+0000 7f793de00700 1 -- 172.21.15.43:0/3486791616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 msgr2=0x55b00ad6d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.878+0000 7f793de00700 1 --2- 172.21.15.43:0/3486791616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 0x55b00ad6d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.878+0000 7f793de00700 1 -- 172.21.15.43:0/3486791616 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b00ac54420 con 0x55b00acda9f0 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.878+0000 7f7935400700 1 --2- 172.21.15.43:0/3486791616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 0x55b00ad6d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.879+0000 7f793de00700 1 --2- 172.21.15.43:0/3486791616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00acda9f0 0x55b00ad6d330 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f793000c7a0 tx=0x7f79300a9b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21844a84625fbbd5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.879+0000 7f792f400700 1 -- 172.21.15.43:0/3486791616 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79300aa310 con 0x55b00acda9f0 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.879+0000 7f792f400700 1 -- 172.21.15.43:0/3486791616 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79300ae6c0 con 0x55b00acda9f0 2024-05-27T07:37:43.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.879+0000 7f792f400700 1 -- 172.21.15.43:0/3486791616 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79300b2eb0 con 0x55b00acda9f0 2024-05-27T07:37:43.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.881+0000 7f7953c4bb80 1 -- 172.21.15.43:0/3486791616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00acda9f0 msgr2=0x55b00ad6d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:43.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.881+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/3486791616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00acda9f0 0x55b00ad6d330 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f793000c7a0 tx=0x7f79300a9b70 comp rx=0 tx=0).stop 2024-05-27T07:37:43.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.881+0000 7f7953c4bb80 1 -- 172.21.15.43:0/3486791616 shutdown_connections 2024-05-27T07:37:43.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.881+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/3486791616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00ad6de70 0x55b00ad7a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.881+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/3486791616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00acda9f0 0x55b00ad6d330 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.881+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/3486791616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 0x55b00ad6d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.881+0000 7f7953c4bb80 1 -- 172.21.15.43:0/3486791616 >> 172.21.15.43:0/3486791616 conn(0x55b00ac88390 msgr2=0x55b00ad694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:43.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.881+0000 7f7953c4bb80 1 -- 172.21.15.43:0/3486791616 shutdown_connections 2024-05-27T07:37:43.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.881+0000 7f7953c4bb80 1 -- 172.21.15.43:0/3486791616 wait complete. 2024-05-27T07:37:43.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.882+0000 7f7953c4bb80 1 -- start start 2024-05-27T07:37:43.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.882+0000 7f7953c4bb80 1 Processor -- start 2024-05-27T07:37:43.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.883+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 0x55b00ad77650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.883+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00acda9f0 0x55b00ad77b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.883+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 0x55b00ad780d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.883+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b00ab91fa0 con 0x55b00abeee40 2024-05-27T07:37:43.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.883+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b00ac2c760 con 0x55b00ad6de70 2024-05-27T07:37:43.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.883+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b00ad7ab80 con 0x55b00acda9f0 2024-05-27T07:37:43.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.883+0000 7f7953c4bb80 10 client.0 _ll_register_callbacks cb 0x55b00ac35c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:43.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.884+0000 7f7935400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00acda9f0 0x55b00ad77b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.884+0000 7f793de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 0x55b00ad77650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.884+0000 7f793de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 0x55b00ad77650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34586/0 (socket says 172.21.15.43:34586) 2024-05-27T07:37:43.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.884+0000 7f793de00700 1 -- 172.21.15.43:0/2363393249 learned_addr learned my addr 172.21.15.43:0/2363393249 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:43.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.884+0000 7f7935400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00acda9f0 0x55b00ad77b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:42092/0 (socket says 172.21.15.43:42092) 2024-05-27T07:37:43.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.884+0000 7f793e800700 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 0x55b00ad780d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.885+0000 7f793e800700 1 -- 172.21.15.43:0/2363393249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00acda9f0 msgr2=0x55b00ad77b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:43.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.885+0000 7f793e800700 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00acda9f0 0x55b00ad77b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.885+0000 7f793e800700 1 -- 172.21.15.43:0/2363393249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 msgr2=0x55b00ad77650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:43.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.885+0000 7f793e800700 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 0x55b00ad77650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.885+0000 7f793e800700 1 -- 172.21.15.43:0/2363393249 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b00abdb8d0 con 0x55b00ad6de70 2024-05-27T07:37:43.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.885+0000 7f793e800700 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 0x55b00ad780d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f793800b650 tx=0x7f793800bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d67100364fc95fb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:43.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.885+0000 7f7923e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:37:43.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.886+0000 7f7923e00700 1 -- 172.21.15.43:0/2363393249 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793800be50 con 0x55b00ad6de70 2024-05-27T07:37:43.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.886+0000 7f7923e00700 1 -- 172.21.15.43:0/2363393249 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f79380050d0 con 0x55b00ad6de70 2024-05-27T07:37:43.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.886+0000 7f7953c4bb80 1 -- 172.21.15.43:0/2363393249 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79300ae6c0 con 0x55b00ad6de70 2024-05-27T07:37:43.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.886+0000 7f7923e00700 1 -- 172.21.15.43:0/2363393249 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f79380052d0 con 0x55b00ad6de70 2024-05-27T07:37:43.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.887+0000 7f7953c4bb80 10 client.37790 fetch_fsmap learned FSMap version 147 2024-05-27T07:37:43.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.887+0000 7f7953c4bb80 1 -- 172.21.15.43:0/2363393249 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55b00ac5c060 con 0x55b00ad6de70 2024-05-27T07:37:43.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.887+0000 7f7923e00700 1 -- 172.21.15.43:0/2363393249 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79300ae6c0 con 0x55b00ad6de70 2024-05-27T07:37:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.887+0000 7f7923e00700 1 -- 172.21.15.43:0/2363393249 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f7938003680 con 0x55b00ad6de70 2024-05-27T07:37:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.887+0000 7f7953c4bb80 10 client.37790 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:37:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.887+0000 7f7953c4bb80 10 client.37790 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:37:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.887+0000 7f7953c4bb80 20 client.37790 populate_metadata read hostname 'smithi043' 2024-05-27T07:37:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.887+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55b00ad87f80 0x55b00ad8a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.887+0000 7f7953c4bb80 4 client.37790 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.887+0000 7f7953c4bb80 1 -- 172.21.15.43:0/2363393249 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b00ac9dd90 con 0x55b00ad87f80 2024-05-27T07:37:43.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.888+0000 7f793de00700 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55b00ad87f80 0x55b00ad8a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.888+0000 7f793de00700 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55b00ad87f80 0x55b00ad8a3f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=557c6de5e235d4c5 server_cookie=1d4ebe184409dc4c in_seq=0 out_seq=0 2024-05-27T07:37:43.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.888+0000 7f7923e00700 10 client.37790 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:37:43.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.889+0000 7f7923e00700 1 -- 172.21.15.43:0/2363393249 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b00ac9dd90 con 0x55b00ad87f80 2024-05-27T07:37:43.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.889+0000 7f7923e00700 10 client.37790 handle_command_reply: tid=0 2024-05-27T07:37:43.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.889+0000 7f7953c4bb80 1 client.37790 shutdown 2024-05-27T07:37:43.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.890+0000 7f7953c4bb80 1 -- 172.21.15.43:0/2363393249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 msgr2=0x55b00ad780d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:43.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.890+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 0x55b00ad780d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f793800b650 tx=0x7f793800bd30 comp rx=0 tx=0).stop 2024-05-27T07:37:43.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.890+0000 7f7953c4bb80 1 -- 172.21.15.43:0/2363393249 shutdown_connections 2024-05-27T07:37:43.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.890+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55b00ad87f80 0x55b00ad8a3f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.890+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 0x55b00ad780d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.890+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00abeee40 0x55b00ad77650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.890+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/2363393249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00acda9f0 0x55b00ad77b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.890+0000 7f7953c4bb80 1 -- 172.21.15.43:0/2363393249 >> 172.21.15.43:0/2363393249 conn(0x55b00ac88390 msgr2=0x55b00ad6d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:43.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.890+0000 7f7923e00700 0 client.37790 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:37:43.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.890+0000 7f7953c4bb80 1 -- 172.21.15.43:0/2363393249 shutdown_connections 2024-05-27T07:37:43.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.891+0000 7f7953c4bb80 1 -- 172.21.15.43:0/2363393249 wait complete. 2024-05-27T07:37:43.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.891+0000 7f7953c4bb80 20 client.37790 trim_cache size 0 max 16384 2024-05-27T07:37:43.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.918+0000 7f7953c4bb80 1 Processor -- start 2024-05-27T07:37:43.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.918+0000 7f7953c4bb80 1 -- start start 2024-05-27T07:37:43.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.919+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00abeee40 0x55b00ad72e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.919+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 0x55b00ad765b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.919+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 0x55b00ad715f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.919+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b00ad69680 con 0x55b00acda9f0 2024-05-27T07:37:43.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.919+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f791c00f250 con 0x55b00ad6de70 2024-05-27T07:37:43.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.919+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f791c00d710 con 0x55b00abeee40 2024-05-27T07:37:43.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.920+0000 7f793e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 0x55b00ad715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.920+0000 7f793de00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00abeee40 0x55b00ad72e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.920+0000 7f793de00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00abeee40 0x55b00ad72e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:42096/0 (socket says 172.21.15.43:42096) 2024-05-27T07:37:43.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.920+0000 7f793de00700 1 -- 172.21.15.43:0/3974089133 learned_addr learned my addr 172.21.15.43:0/3974089133 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:43.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.920+0000 7f7935400700 1 --2- 172.21.15.43:0/3974089133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 0x55b00ad765b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.921+0000 7f7935400700 1 -- 172.21.15.43:0/3974089133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00abeee40 msgr2=0x55b00ad72e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:43.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.921+0000 7f7935400700 1 --2- 172.21.15.43:0/3974089133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00abeee40 0x55b00ad72e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.921+0000 7f7935400700 1 -- 172.21.15.43:0/3974089133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 msgr2=0x55b00ad715f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:43.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.921+0000 7f7935400700 1 --2- 172.21.15.43:0/3974089133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 0x55b00ad715f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.921+0000 7f7935400700 1 -- 172.21.15.43:0/3974089133 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b00ad75bc0 con 0x55b00acda9f0 2024-05-27T07:37:43.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.922+0000 7f7935400700 1 --2- 172.21.15.43:0/3974089133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 0x55b00ad765b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x55b00ad68d90 tx=0x7f7928013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e981f276feafb6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:43.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.923+0000 7f792ea00700 1 -- 172.21.15.43:0/3974089133 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928011d80 con 0x55b00acda9f0 2024-05-27T07:37:43.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.923+0000 7f792ea00700 1 -- 172.21.15.43:0/3974089133 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79280044e0 con 0x55b00acda9f0 2024-05-27T07:37:43.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.923+0000 7f792ea00700 1 -- 172.21.15.43:0/3974089133 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792801d440 con 0x55b00acda9f0 2024-05-27T07:37:43.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.924+0000 7f7953c4bb80 1 -- 172.21.15.43:0/3974089133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 msgr2=0x55b00ad765b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:43.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.924+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/3974089133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 0x55b00ad765b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x55b00ad68d90 tx=0x7f7928013000 comp rx=0 tx=0).stop 2024-05-27T07:37:43.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.924+0000 7f7953c4bb80 1 -- 172.21.15.43:0/3974089133 shutdown_connections 2024-05-27T07:37:43.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.924+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/3974089133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ad6de70 0x55b00ad715f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.924+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/3974089133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00abeee40 0x55b00ad72e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.924+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/3974089133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 0x55b00ad765b0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.924+0000 7f7953c4bb80 1 -- 172.21.15.43:0/3974089133 >> 172.21.15.43:0/3974089133 conn(0x55b00ac88390 msgr2=0x55b00ad70ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:43.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.924+0000 7f7953c4bb80 1 -- 172.21.15.43:0/3974089133 shutdown_connections 2024-05-27T07:37:43.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.924+0000 7f7953c4bb80 1 -- 172.21.15.43:0/3974089133 wait complete. 2024-05-27T07:37:43.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.925+0000 7f7953c4bb80 1 -- start start 2024-05-27T07:37:43.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.925+0000 7f7953c4bb80 1 Processor -- start 2024-05-27T07:37:43.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.926+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00abeee40 0x55b00adeac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.926+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 0x55b00adeb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.926+0000 7f7953c4bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00ad6de70 0x55b00ad7fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.926+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f791c00dde0 con 0x55b00acda9f0 2024-05-27T07:37:43.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.926+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f791c00eb80 con 0x55b00abeee40 2024-05-27T07:37:43.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.926+0000 7f7953c4bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f791c00e4b0 con 0x55b00ad6de70 2024-05-27T07:37:43.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.926+0000 7f7953c4bb80 10 client.0 _ll_register_callbacks cb 0x55b00ac35c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:43.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.926+0000 7f793de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00abeee40 0x55b00adeac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.926+0000 7f793de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00abeee40 0x55b00adeac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:51882/0 (socket says 172.21.15.43:51882) 2024-05-27T07:37:43.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.926+0000 7f793de00700 1 -- 172.21.15.43:0/989102440 learned_addr learned my addr 172.21.15.43:0/989102440 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:43.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.927+0000 7f793e800700 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00ad6de70 0x55b00ad7fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.927+0000 7f7935400700 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 0x55b00adeb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.927+0000 7f793de00700 1 -- 172.21.15.43:0/989102440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00ad6de70 msgr2=0x55b00ad7fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:43.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.927+0000 7f793de00700 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00ad6de70 0x55b00ad7fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.927+0000 7f793de00700 1 -- 172.21.15.43:0/989102440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 msgr2=0x55b00adeb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:43.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.927+0000 7f793de00700 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 0x55b00adeb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.927+0000 7f793de00700 1 -- 172.21.15.43:0/989102440 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b00ad80150 con 0x55b00abeee40 2024-05-27T07:37:43.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.929+0000 7f7935400700 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 0x55b00adeb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:43.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.929+0000 7f793de00700 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00abeee40 0x55b00adeac40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f793000c7a0 tx=0x7f7930003ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5085f521d36512e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:43.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.930+0000 7f7923e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:37:43.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.930+0000 7f7923e00700 1 -- 172.21.15.43:0/989102440 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79300aa310 con 0x55b00abeee40 2024-05-27T07:37:43.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.930+0000 7f7923e00700 1 -- 172.21.15.43:0/989102440 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f79300aa780 con 0x55b00abeee40 2024-05-27T07:37:43.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.930+0000 7f7953c4bb80 1 -- 172.21.15.43:0/989102440 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79280044e0 con 0x55b00abeee40 2024-05-27T07:37:43.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.931+0000 7f7923e00700 1 -- 172.21.15.43:0/989102440 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f79300ca050 con 0x55b00abeee40 2024-05-27T07:37:43.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.931+0000 7f7953c4bb80 10 client.37796 fetch_fsmap learned FSMap version 147 2024-05-27T07:37:43.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.931+0000 7f7953c4bb80 1 -- 172.21.15.43:0/989102440 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55b00adf23e0 con 0x55b00abeee40 2024-05-27T07:37:43.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.931+0000 7f7923e00700 1 -- 172.21.15.43:0/989102440 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79280044e0 con 0x55b00abeee40 2024-05-27T07:37:43.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.932+0000 7f7923e00700 1 -- 172.21.15.43:0/989102440 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f79300a5e80 con 0x55b00abeee40 2024-05-27T07:37:43.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.932+0000 7f7953c4bb80 10 client.37796 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:37:43.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.932+0000 7f7953c4bb80 10 client.37796 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:37:43.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.932+0000 7f7953c4bb80 20 client.37796 populate_metadata read hostname 'smithi043' 2024-05-27T07:37:43.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.933+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55b00adf2a10 0x55b00adf4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:43.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.933+0000 7f7953c4bb80 4 client.37796 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:37:43.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.933+0000 7f7953c4bb80 1 -- 172.21.15.43:0/989102440 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b00ac97570 con 0x55b00adf2a10 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.933+0000 7f7935400700 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55b00adf2a10 0x55b00adf4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.934+0000 7f7935400700 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55b00adf2a10 0x55b00adf4e30 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=292108b54b01d0ce server_cookie=d97c0be457f4cd20 in_seq=0 out_seq=0 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.934+0000 7f7923e00700 10 client.37796 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.935+0000 7f7923e00700 1 -- 172.21.15.43:0/989102440 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b00ac97570 con 0x55b00adf2a10 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.935+0000 7f7923e00700 10 client.37796 handle_command_reply: tid=0 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.935+0000 7f7953c4bb80 1 client.37796 shutdown 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7953c4bb80 1 -- 172.21.15.43:0/989102440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00abeee40 msgr2=0x55b00adeac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00abeee40 0x55b00adeac40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f793000c7a0 tx=0x7f7930003ab0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7953c4bb80 1 -- 172.21.15.43:0/989102440 shutdown_connections 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b00ad6de70 0x55b00ad7fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55b00adf2a10 0x55b00adf4e30 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00abeee40 0x55b00adeac40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7953c4bb80 1 --2- 172.21.15.43:0/989102440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b00acda9f0 0x55b00adeb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7953c4bb80 1 -- 172.21.15.43:0/989102440 >> 172.21.15.43:0/989102440 conn(0x55b00ac88390 msgr2=0x55b00ad78ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:43.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7923e00700 0 client.37796 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:37:43.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7953c4bb80 1 -- 172.21.15.43:0/989102440 shutdown_connections 2024-05-27T07:37:43.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.936+0000 7f7953c4bb80 1 -- 172.21.15.43:0/989102440 wait complete. 2024-05-27T07:37:43.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.937+0000 7f7953c4bb80 20 client.37796 trim_cache size 0 max 16384 2024-05-27T07:37:43.936 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:37:43.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.939+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79240b2af0 msgr2=0x7f79240b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:43.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.939+0000 7f7945200700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79240b2af0 0x7f79240b5000 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f79400775b0 tx=0x7f79280073d0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.940+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7940104520 msgr2=0x7f79401a1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:43.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.940+0000 7f7945200700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7940104520 0x7f79401a1370 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f793000ef30 tx=0x7f793000c3a0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.940+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 shutdown_connections 2024-05-27T07:37:43.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.940+0000 7f7945200700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f79240b2af0 0x7f79240b5000 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.940+0000 7f7945200700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7940109520 0x7f79401a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.940+0000 7f7945200700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7940104520 0x7f79401a1370 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.940+0000 7f7945200700 1 --2- 172.21.15.43:0/3537082062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7940104e70 0x7f79401a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:43.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.940+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 >> 172.21.15.43:0/3537082062 conn(0x7f7940100030 msgr2=0x7f7940100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:43.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.941+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 shutdown_connections 2024-05-27T07:37:43.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:43.941+0000 7f7945200700 1 -- 172.21.15.43:0/3537082062 wait complete. 2024-05-27T07:37:43.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:43 smithi179 ceph-mon[35888]: pgmap v798: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 5.7 MiB/s wr, 220 op/s 2024-05-27T07:37:43.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:43 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:43.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:43 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:44.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:37:44.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 64041 5.82 MB/sec execute 465 sec latency 106.407 ms 2024-05-27T07:37:45.502 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:45.822 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:45 smithi043 ceph-mon[31531]: pgmap v799: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 3.7 MiB/s wr, 146 op/s 2024-05-27T07:37:45.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 64148 5.81 MB/sec execute 466 sec latency 102.877 ms 2024-05-27T07:37:45.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:45 smithi132 ceph-mon[35283]: pgmap v799: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 3.7 MiB/s wr, 146 op/s 2024-05-27T07:37:45.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:45 smithi179 ceph-mon[35888]: pgmap v799: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 3.7 MiB/s wr, 146 op/s 2024-05-27T07:37:46.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 64254 5.84 MB/sec execute 467 sec latency 349.502 ms 2024-05-27T07:37:47.796 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:47 smithi179 ceph-mon[35888]: pgmap v800: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-05-27T07:37:47.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 64377 5.83 MB/sec execute 468 sec latency 174.386 ms 2024-05-27T07:37:47.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:47 smithi132 ceph-mon[35283]: pgmap v800: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-05-27T07:37:47.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:47 smithi043 ceph-mon[31531]: pgmap v800: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 5.6 MiB/s wr, 242 op/s 2024-05-27T07:37:48.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 64479 5.84 MB/sec execute 469 sec latency 334.402 ms 2024-05-27T07:37:49.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 64615 5.83 MB/sec execute 470 sec latency 88.378 ms 2024-05-27T07:37:49.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:49 smithi132 ceph-mon[35283]: pgmap v801: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 3.5 MiB/s wr, 167 op/s 2024-05-27T07:37:49.942 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:49 smithi043 ceph-mon[31531]: pgmap v801: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 3.5 MiB/s wr, 167 op/s 2024-05-27T07:37:49.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:49 smithi179 ceph-mon[35888]: pgmap v801: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 3.5 MiB/s wr, 167 op/s 2024-05-27T07:37:50.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.520+0000 7fce7e600700 1 -- 172.21.15.43:0/3839869599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce781043f0 msgr2=0x7fce78104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.520+0000 7fce7e600700 1 --2- 172.21.15.43:0/3839869599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce781043f0 0x7fce78104800 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fce680099c0 tx=0x7fce68009cd0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.521+0000 7fce7e600700 1 -- 172.21.15.43:0/3839869599 shutdown_connections 2024-05-27T07:37:50.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.521+0000 7fce7e600700 1 --2- 172.21.15.43:0/3839869599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce78109670 0x7fce78110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.521+0000 7fce7e600700 1 --2- 172.21.15.43:0/3839869599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce781043f0 0x7fce78104800 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.521+0000 7fce7e600700 1 --2- 172.21.15.43:0/3839869599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce78108d20 0x7fce78109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.521+0000 7fce7e600700 1 -- 172.21.15.43:0/3839869599 >> 172.21.15.43:0/3839869599 conn(0x7fce78079d00 msgr2=0x7fce7807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.521+0000 7fce7e600700 1 -- 172.21.15.43:0/3839869599 shutdown_connections 2024-05-27T07:37:50.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.521+0000 7fce7e600700 1 -- 172.21.15.43:0/3839869599 wait complete. 2024-05-27T07:37:50.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.522+0000 7fce7e600700 1 Processor -- start 2024-05-27T07:37:50.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.523+0000 7fce7e600700 1 -- start start 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.523+0000 7fce7e600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce781043f0 0x7fce781a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.523+0000 7fce7e600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce78108d20 0x7fce781a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.523+0000 7fce7e600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce78109670 0x7fce7819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.523+0000 7fce7e600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce78114210 con 0x7fce781043f0 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.523+0000 7fce7e600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce78114090 con 0x7fce78109670 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.523+0000 7fce7e600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fce78114390 con 0x7fce78108d20 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.524+0000 7fce7d200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce781043f0 0x7fce781a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.524+0000 7fce7d200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce781043f0 0x7fce781a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36574/0 (socket says 172.21.15.43:36574) 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.524+0000 7fce7d200700 1 -- 172.21.15.43:0/3257489499 learned_addr learned my addr 172.21.15.43:0/3257489499 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.524+0000 7fce7dc00700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce78109670 0x7fce7819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.524+0000 7fce7d200700 1 -- 172.21.15.43:0/3257489499 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce78108d20 msgr2=0x7fce781a0d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T07:37:50.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.524+0000 7fce7d200700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce78108d20 0x7fce781a0d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.524+0000 7fce7d200700 1 -- 172.21.15.43:0/3257489499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce78109670 msgr2=0x7fce7819aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.524+0000 7fce7d200700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce78109670 0x7fce7819aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.524+0000 7fce7d200700 1 -- 172.21.15.43:0/3257489499 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7819b0b0 con 0x7fce781043f0 2024-05-27T07:37:50.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.524+0000 7fce7dc00700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce78109670 0x7fce7819aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:37:50.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.525+0000 7fce7d200700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce781043f0 0x7fce781a0850 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fce68000c00 tx=0x7fce6800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.525+0000 7fce76000700 1 -- 172.21.15.43:0/3257489499 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce68027070 con 0x7fce781043f0 2024-05-27T07:37:50.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.525+0000 7fce7e600700 1 -- 172.21.15.43:0/3257489499 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7819b340 con 0x7fce781043f0 2024-05-27T07:37:50.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.526+0000 7fce7e600700 1 -- 172.21.15.43:0/3257489499 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce7819b870 con 0x7fce781043f0 2024-05-27T07:37:50.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.526+0000 7fce76000700 1 -- 172.21.15.43:0/3257489499 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce680056d0 con 0x7fce781043f0 2024-05-27T07:37:50.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.526+0000 7fce76000700 1 -- 172.21.15.43:0/3257489499 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce680173f0 con 0x7fce781043f0 2024-05-27T07:37:50.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.530+0000 7fce76000700 1 -- 172.21.15.43:0/3257489499 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fce6802a070 con 0x7fce781043f0 2024-05-27T07:37:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.532+0000 7fce76000700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce5c0b2a70 0x7fce5c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.532+0000 7fce76000700 1 -- 172.21.15.43:0/3257489499 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7fce680e4720 con 0x7fce781043f0 2024-05-27T07:37:50.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.532+0000 7fce6fe00700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce5c0b2a70 0x7fce5c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.555+0000 7fce6fe00700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce5c0b2a70 0x7fce5c0b4f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fce7807a230 tx=0x7fce64006cb0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.558+0000 7fce8e809b80 1 Processor -- start 2024-05-27T07:37:50.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.558+0000 7fce8e809b80 1 -- start start 2024-05-27T07:37:50.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.559+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.559+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf410 0x55a8393d9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.559+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf820 0x55a8393e0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.559+0000 7fce8e809b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a8392fde40 con 0x55a8393cf410 2024-05-27T07:37:50.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.559+0000 7fce8e809b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a8392ed9e0 con 0x55a839250e40 2024-05-27T07:37:50.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.559+0000 7fce8e809b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a8392f1a20 con 0x55a8393cf820 2024-05-27T07:37:50.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.560+0000 7fce6fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf410 0x55a8393d9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.560+0000 7fce6fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf410 0x55a8393d9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36588/0 (socket says 172.21.15.43:36588) 2024-05-27T07:37:50.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.560+0000 7fce6fe00700 1 -- 172.21.15.43:0/1022946084 learned_addr learned my addr 172.21.15.43:0/1022946084 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:50.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.560+0000 7fce7dc00700 1 --2- 172.21.15.43:0/1022946084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf820 0x55a8393e0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.560+0000 7fce6fe00700 1 -- 172.21.15.43:0/1022946084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf820 msgr2=0x55a8393e0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.560+0000 7fce6fe00700 1 --2- 172.21.15.43:0/1022946084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf820 0x55a8393e0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.560+0000 7fce6fe00700 1 -- 172.21.15.43:0/1022946084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 msgr2=0x55a8393ceed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:37:50.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.560+0000 7fce6fe00700 1 --2- 172.21.15.43:0/1022946084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.560+0000 7fce6fe00700 1 -- 172.21.15.43:0/1022946084 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8392b6420 con 0x55a8393cf410 2024-05-27T07:37:50.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.561+0000 7fce7dc00700 1 --2- 172.21.15.43:0/1022946084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf820 0x55a8393e0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:50.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.562+0000 7fce6fe00700 1 --2- 172.21.15.43:0/1022946084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf410 0x55a8393d9ad0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fce64015910 tx=0x7fce6400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342b6400c8b8241a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.563+0000 7fce6e000700 1 -- 172.21.15.43:0/1022946084 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64023070 con 0x55a8393cf410 2024-05-27T07:37:50.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.563+0000 7fce6e000700 1 -- 172.21.15.43:0/1022946084 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce64012a00 con 0x55a8393cf410 2024-05-27T07:37:50.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.563+0000 7fce6e000700 1 -- 172.21.15.43:0/1022946084 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce640138e0 con 0x55a8393cf410 2024-05-27T07:37:50.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.564+0000 7fce8e809b80 1 -- 172.21.15.43:0/1022946084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf410 msgr2=0x55a8393d9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.564+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1022946084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf410 0x55a8393d9ad0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fce64015910 tx=0x7fce6400e000 comp rx=0 tx=0).stop 2024-05-27T07:37:50.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.565+0000 7fce8e809b80 1 -- 172.21.15.43:0/1022946084 shutdown_connections 2024-05-27T07:37:50.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.565+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1022946084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf820 0x55a8393e0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.565+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1022946084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.565+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1022946084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf410 0x55a8393d9ad0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.565+0000 7fce8e809b80 1 -- 172.21.15.43:0/1022946084 >> 172.21.15.43:0/1022946084 conn(0x55a8392b0fc0 msgr2=0x55a8393cad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.566+0000 7fce8e809b80 1 -- 172.21.15.43:0/1022946084 shutdown_connections 2024-05-27T07:37:50.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.566+0000 7fce8e809b80 1 -- 172.21.15.43:0/1022946084 wait complete. 2024-05-27T07:37:50.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.567+0000 7fce8e809b80 1 -- start start 2024-05-27T07:37:50.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.567+0000 7fce8e809b80 1 Processor -- start 2024-05-27T07:37:50.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.568+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393d3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.568+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf410 0x55a8393d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.568+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393d7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.568+0000 7fce7d200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393d3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.568+0000 7fce7d200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393d3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58056/0 (socket says 172.21.15.43:58056) 2024-05-27T07:37:50.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.569+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a8391f3fa0 con 0x55a8393cf820 2024-05-27T07:37:50.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.569+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a83928e760 con 0x55a839250e40 2024-05-27T07:37:50.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.569+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a8393da390 con 0x55a8393cf410 2024-05-27T07:37:50.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.569+0000 7fce8e809b80 10 client.0 _ll_register_callbacks cb 0x55a839297c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:50.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.569+0000 7fce7d200700 1 -- 172.21.15.43:0/214169131 learned_addr learned my addr 172.21.15.43:0/214169131 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:50.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.569+0000 7fce6fe00700 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf410 0x55a8393d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.569+0000 7fce7dc00700 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393d7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.570+0000 7fce7d200700 1 -- 172.21.15.43:0/214169131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf410 msgr2=0x55a8393d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.570+0000 7fce7d200700 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf410 0x55a8393d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.570+0000 7fce7d200700 1 -- 172.21.15.43:0/214169131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 msgr2=0x55a8393d7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.570+0000 7fce7d200700 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393d7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.570+0000 7fce7d200700 1 -- 172.21.15.43:0/214169131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a83923d8d0 con 0x55a839250e40 2024-05-27T07:37:50.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.571+0000 7fce7d200700 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393d3630 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fce6800f7b0 tx=0x7fce680abba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0f780d7ce962e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.571+0000 7fce62a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:37:50.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.571+0000 7fce62a00700 1 -- 172.21.15.43:0/214169131 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce680ab170 con 0x55a839250e40 2024-05-27T07:37:50.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.571+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce64012a00 con 0x55a839250e40 2024-05-27T07:37:50.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.571+0000 7fce62a00700 1 -- 172.21.15.43:0/214169131 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7fce680e4290 con 0x55a839250e40 2024-05-27T07:37:50.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.571+0000 7fce62a00700 1 -- 172.21.15.43:0/214169131 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fce680abd60 con 0x55a839250e40 2024-05-27T07:37:50.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.572+0000 7fce8e809b80 10 client.37802 fetch_fsmap learned FSMap version 147 2024-05-27T07:37:50.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.572+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55a8392be060 con 0x55a839250e40 2024-05-27T07:37:50.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.572+0000 7fce62a00700 1 -- 172.21.15.43:0/214169131 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce64012a00 con 0x55a839250e40 2024-05-27T07:37:50.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.572+0000 7fce62a00700 1 -- 172.21.15.43:0/214169131 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fce680b0ba0 con 0x55a839250e40 2024-05-27T07:37:50.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.572+0000 7fce8e809b80 10 client.37802 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:37:50.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.572+0000 7fce8e809b80 10 client.37802 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:37:50.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.573+0000 7fce8e809b80 20 client.37802 populate_metadata read hostname 'smithi043' 2024-05-27T07:37:50.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.573+0000 7fce8e809b80 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a8393f2370 0x55a8393f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.573+0000 7fce8e809b80 4 client.37802 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:50.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.573+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8392ffd90 con 0x55a8393f2370 2024-05-27T07:37:50.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.573+0000 7fce6fe00700 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a8393f2370 0x55a8393f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.574+0000 7fce6fe00700 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a8393f2370 0x55a8393f47c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=10881ea3ef071a04 server_cookie=89d83cf17fb38b3f in_seq=0 out_seq=0 2024-05-27T07:37:50.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.574+0000 7fce62a00700 10 client.37802 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:37:50.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.575+0000 7fce62a00700 1 -- 172.21.15.43:0/214169131 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a8392ffd90 con 0x55a8393f2370 2024-05-27T07:37:50.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.575+0000 7fce62a00700 10 client.37802 handle_command_reply: tid=0 2024-05-27T07:37:50.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.575+0000 7fce8e809b80 1 client.37802 shutdown 2024-05-27T07:37:50.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.576+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 msgr2=0x55a8393d3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.576+0000 7fce8e809b80 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393d3630 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fce6800f7b0 tx=0x7fce680abba0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.576+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 shutdown_connections 2024-05-27T07:37:50.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.576+0000 7fce8e809b80 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a8393f2370 0x55a8393f47c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.576+0000 7fce8e809b80 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393d7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.576+0000 7fce8e809b80 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393d3630 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.576+0000 7fce8e809b80 1 --2- 172.21.15.43:0/214169131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf410 0x55a8393d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.576+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 >> 172.21.15.43:0/214169131 conn(0x55a8392b0fc0 msgr2=0x55a8393c9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.576+0000 7fce62a00700 0 client.37802 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:37:50.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.577+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 shutdown_connections 2024-05-27T07:37:50.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.577+0000 7fce8e809b80 1 -- 172.21.15.43:0/214169131 wait complete. 2024-05-27T07:37:50.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.577+0000 7fce8e809b80 20 client.37802 trim_cache size 0 max 16384 2024-05-27T07:37:50.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.604+0000 7fce8e809b80 1 Processor -- start 2024-05-27T07:37:50.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.604+0000 7fce8e809b80 1 -- start start 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.605+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a839250e40 0x55a8393d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.605+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8393cf410 0x55a8393f9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.605+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393fa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.605+0000 7fce8e809b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce5800f1c0 con 0x55a8393cf820 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.605+0000 7fce8e809b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce5800d680 con 0x55a8393cf410 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.605+0000 7fce8e809b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fce5800eaf0 con 0x55a839250e40 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.605+0000 7fce7dc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393fa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.605+0000 7fce7dc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393fa4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36596/0 (socket says 172.21.15.43:36596) 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.606+0000 7fce7dc00700 1 -- 172.21.15.43:0/1454774274 learned_addr learned my addr 172.21.15.43:0/1454774274 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.606+0000 7fce6fe00700 1 --2- 172.21.15.43:0/1454774274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8393cf410 0x55a8393f9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.606+0000 7fce7d200700 1 --2- 172.21.15.43:0/1454774274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a839250e40 0x55a8393d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.606+0000 7fce6fe00700 1 -- 172.21.15.43:0/1454774274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a839250e40 msgr2=0x55a8393d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.606+0000 7fce6fe00700 1 --2- 172.21.15.43:0/1454774274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a839250e40 0x55a8393d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.606+0000 7fce6fe00700 1 -- 172.21.15.43:0/1454774274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 msgr2=0x55a8393fa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.606+0000 7fce6fe00700 1 --2- 172.21.15.43:0/1454774274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393fa4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.606+0000 7fce6fe00700 1 -- 172.21.15.43:0/1454774274 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8393cc350 con 0x55a8393cf410 2024-05-27T07:37:50.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.607+0000 7fce6fe00700 1 --2- 172.21.15.43:0/1454774274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8393cf410 0x55a8393f9f60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fce64009fd0 tx=0x7fce64013d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de9aeda9a06aa95 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.607+0000 7fce6d600700 1 -- 172.21.15.43:0/1454774274 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64023040 con 0x55a8393cf410 2024-05-27T07:37:50.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.607+0000 7fce6d600700 1 -- 172.21.15.43:0/1454774274 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce64030070 con 0x55a8393cf410 2024-05-27T07:37:50.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.607+0000 7fce6d600700 1 -- 172.21.15.43:0/1454774274 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64012c60 con 0x55a8393cf410 2024-05-27T07:37:50.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.608+0000 7fce8e809b80 1 -- 172.21.15.43:0/1454774274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8393cf410 msgr2=0x55a8393f9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.608+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1454774274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8393cf410 0x55a8393f9f60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fce64009fd0 tx=0x7fce64013d70 comp rx=0 tx=0).stop 2024-05-27T07:37:50.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.608+0000 7fce8e809b80 1 -- 172.21.15.43:0/1454774274 shutdown_connections 2024-05-27T07:37:50.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.608+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1454774274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393fa4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.608+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1454774274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a839250e40 0x55a8393d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.608+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1454774274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a8393cf410 0x55a8393f9f60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.608+0000 7fce8e809b80 1 -- 172.21.15.43:0/1454774274 >> 172.21.15.43:0/1454774274 conn(0x55a8392b0fc0 msgr2=0x55a8393d1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.609+0000 7fce8e809b80 1 -- 172.21.15.43:0/1454774274 shutdown_connections 2024-05-27T07:37:50.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.609+0000 7fce8e809b80 1 -- 172.21.15.43:0/1454774274 wait complete. 2024-05-27T07:37:50.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.609+0000 7fce8e809b80 1 -- start start 2024-05-27T07:37:50.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.610+0000 7fce8e809b80 1 Processor -- start 2024-05-27T07:37:50.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.610+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393ee930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.610+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf410 0x55a8393e9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.610+0000 7fce8e809b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393ea1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.610+0000 7fce8e809b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce5800e420 con 0x55a8393cf820 2024-05-27T07:37:50.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.610+0000 7fce8e809b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce5800dd50 con 0x55a839250e40 2024-05-27T07:37:50.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.610+0000 7fce8e809b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a8392ffd90 con 0x55a8393cf410 2024-05-27T07:37:50.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.610+0000 7fce8e809b80 10 client.0 _ll_register_callbacks cb 0x55a839297c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.611+0000 7fce7d200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.611+0000 7fce7d200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393ee930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:58070/0 (socket says 172.21.15.43:58070) 2024-05-27T07:37:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.611+0000 7fce7d200700 1 -- 172.21.15.43:0/1480908467 learned_addr learned my addr 172.21.15.43:0/1480908467 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.611+0000 7fce6fe00700 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf410 0x55a8393e9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.611+0000 7fce7d200700 1 -- 172.21.15.43:0/1480908467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf410 msgr2=0x55a8393e9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.611+0000 7fce7d200700 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf410 0x55a8393e9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.611+0000 7fce7d200700 1 -- 172.21.15.43:0/1480908467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 msgr2=0x55a8393ea1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.611+0000 7fce7d200700 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393ea1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.611+0000 7fce7d200700 1 -- 172.21.15.43:0/1480908467 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8393ea710 con 0x55a839250e40 2024-05-27T07:37:50.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.616+0000 7fce7d200700 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393ee930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fce680a88d0 tx=0x7fce680a7650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=510a6ed0c3ba5524 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:50.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.616+0000 7fce62a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:37:50.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.617+0000 7fce62a00700 1 -- 172.21.15.43:0/1480908467 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce680ab170 con 0x55a839250e40 2024-05-27T07:37:50.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.617+0000 7fce62a00700 1 -- 172.21.15.43:0/1480908467 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7fce680e44a0 con 0x55a839250e40 2024-05-27T07:37:50.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.617+0000 7fce8e809b80 1 -- 172.21.15.43:0/1480908467 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce64030070 con 0x55a839250e40 2024-05-27T07:37:50.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.617+0000 7fce62a00700 1 -- 172.21.15.43:0/1480908467 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fce680ab660 con 0x55a839250e40 2024-05-27T07:37:50.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.617+0000 7fce8e809b80 10 client.37808 fetch_fsmap learned FSMap version 147 2024-05-27T07:37:50.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.617+0000 7fce8e809b80 1 -- 172.21.15.43:0/1480908467 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55a8393ea710 con 0x55a839250e40 2024-05-27T07:37:50.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.618+0000 7fce62a00700 1 -- 172.21.15.43:0/1480908467 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fce64030070 con 0x55a839250e40 2024-05-27T07:37:50.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.619+0000 7fce62a00700 1 -- 172.21.15.43:0/1480908467 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fce680cc330 con 0x55a839250e40 2024-05-27T07:37:50.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.619+0000 7fce8e809b80 10 client.37808 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:37:50.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.619+0000 7fce8e809b80 10 client.37808 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:37:50.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.619+0000 7fce8e809b80 20 client.37808 populate_metadata read hostname 'smithi043' 2024-05-27T07:37:50.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.619+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a8393e89c0 0x55a839456e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:50.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.619+0000 7fce8e809b80 4 client.37808 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:37:50.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.619+0000 7fce8e809b80 1 -- 172.21.15.43:0/1480908467 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a8392f9570 con 0x55a8393e89c0 2024-05-27T07:37:50.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.619+0000 7fce6fe00700 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a8393e89c0 0x55a839456e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:50.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.620+0000 7fce6fe00700 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a8393e89c0 0x55a839456e00 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f5a095a12d00b346 server_cookie=7c8b42ad4361d6a6 in_seq=0 out_seq=0 2024-05-27T07:37:50.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.620+0000 7fce62a00700 10 client.37808 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:37:50.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce62a00700 1 -- 172.21.15.43:0/1480908467 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a8392f9570 con 0x55a8393e89c0 2024-05-27T07:37:50.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce62a00700 10 client.37808 handle_command_reply: tid=0 2024-05-27T07:37:50.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce8e809b80 1 client.37808 shutdown 2024-05-27T07:37:50.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce8e809b80 1 -- 172.21.15.43:0/1480908467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 msgr2=0x55a8393ee930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393ee930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fce680a88d0 tx=0x7fce680a7650 comp rx=0 tx=0).stop 2024-05-27T07:37:50.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce8e809b80 1 -- 172.21.15.43:0/1480908467 shutdown_connections 2024-05-27T07:37:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55a8393e89c0 0x55a839456e00 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a8393cf820 0x55a8393ea1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a839250e40 0x55a8393ee930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce8e809b80 1 --2- 172.21.15.43:0/1480908467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a8393cf410 0x55a8393e9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce8e809b80 1 -- 172.21.15.43:0/1480908467 >> 172.21.15.43:0/1480908467 conn(0x55a8392b0fc0 msgr2=0x55a8393d8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.621+0000 7fce62a00700 0 client.37808 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:37:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.622+0000 7fce8e809b80 1 -- 172.21.15.43:0/1480908467 shutdown_connections 2024-05-27T07:37:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.622+0000 7fce8e809b80 1 -- 172.21.15.43:0/1480908467 wait complete. 2024-05-27T07:37:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.622+0000 7fce8e809b80 20 client.37808 trim_cache size 0 max 16384 2024-05-27T07:37:50.621 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:37:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 -- 172.21.15.43:0/3257489499 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce5c0b2a70 msgr2=0x7fce5c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce5c0b2a70 0x7fce5c0b4f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fce7807a230 tx=0x7fce64006cb0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 -- 172.21.15.43:0/3257489499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce781043f0 msgr2=0x7fce781a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce781043f0 0x7fce781a0850 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fce68000c00 tx=0x7fce6800f690 comp rx=0 tx=0).stop 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 -- 172.21.15.43:0/3257489499 shutdown_connections 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce5c0b2a70 0x7fce5c0b4f80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce78109670 0x7fce7819aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce781043f0 0x7fce781a0850 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 --2- 172.21.15.43:0/3257489499 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce78108d20 0x7fce781a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 -- 172.21.15.43:0/3257489499 >> 172.21.15.43:0/3257489499 conn(0x7fce78079d00 msgr2=0x7fce78101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.623+0000 7fce7e600700 1 -- 172.21.15.43:0/3257489499 shutdown_connections 2024-05-27T07:37:50.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:50.624+0000 7fce7e600700 1 -- 172.21.15.43:0/3257489499 wait complete. 2024-05-27T07:37:50.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 64738 5.84 MB/sec execute 471 sec latency 104.085 ms 2024-05-27T07:37:51.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:37:51.728 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:51 smithi043 ceph-mon[31531]: pgmap v802: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 3.5 MiB/s wr, 167 op/s 2024-05-27T07:37:51.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 64817 5.83 MB/sec execute 472 sec latency 117.478 ms 2024-05-27T07:37:51.831 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:51 smithi132 ceph-mon[35283]: pgmap v802: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 3.5 MiB/s wr, 167 op/s 2024-05-27T07:37:51.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:51 smithi179 ceph-mon[35888]: pgmap v802: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 3.5 MiB/s wr, 167 op/s 2024-05-27T07:37:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:37:52.830 INFO:tasks.workunit.client.0.smithi043.stdout: 10 64904 5.82 MB/sec execute 473 sec latency 95.817 ms 2024-05-27T07:37:53.597 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:53 smithi132 ceph-mon[35283]: pgmap v803: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.8 MiB/s wr, 234 op/s 2024-05-27T07:37:53.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 65009 5.82 MB/sec execute 474 sec latency 235.690 ms 2024-05-27T07:37:53.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:53 smithi043 ceph-mon[31531]: pgmap v803: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.8 MiB/s wr, 234 op/s 2024-05-27T07:37:53.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:53 smithi179 ceph-mon[35888]: pgmap v803: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.8 MiB/s wr, 234 op/s 2024-05-27T07:37:54.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 65112 5.83 MB/sec execute 475 sec latency 336.798 ms 2024-05-27T07:37:55.209 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:55.209 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:55.209 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:37:55.209 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:37:55.209 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:54 smithi043 ceph-mon[31531]: pgmap v804: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 168 op/s 2024-05-27T07:37:55.209 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:54 smithi043 ceph-mon[31531]: pgmap v805: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-05-27T07:37:55.209 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:55.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:55.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:55.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:37:55.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:37:55.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:54 smithi179 ceph-mon[35888]: pgmap v804: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 168 op/s 2024-05-27T07:37:55.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:54 smithi179 ceph-mon[35888]: pgmap v805: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-05-27T07:37:55.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:55.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:55.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:55.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:37:55.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:37:55.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:54 smithi132 ceph-mon[35283]: pgmap v804: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 3.3 MiB/s wr, 168 op/s 2024-05-27T07:37:55.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:54 smithi132 ceph-mon[35283]: pgmap v805: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.0 MiB/s wr, 203 op/s 2024-05-27T07:37:55.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:37:55.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 65252 5.83 MB/sec execute 476 sec latency 87.444 ms 2024-05-27T07:37:56.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 65369 5.83 MB/sec execute 477 sec latency 131.266 ms 2024-05-27T07:37:57.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:56 smithi043 ceph-mon[31531]: pgmap v806: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 3.0 MiB/s wr, 147 op/s 2024-05-27T07:37:57.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:56 smithi179 ceph-mon[35888]: pgmap v806: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 3.0 MiB/s wr, 147 op/s 2024-05-27T07:37:57.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:56 smithi132 ceph-mon[35283]: pgmap v806: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 3.0 MiB/s wr, 147 op/s 2024-05-27T07:37:57.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 65499 5.84 MB/sec execute 478 sec latency 94.638 ms 2024-05-27T07:37:58.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.186+0000 7f6ce4200700 1 -- 172.21.15.43:0/3428497296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cdc1034b0 msgr2=0x7f6cdc1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.186+0000 7f6ce4200700 1 --2- 172.21.15.43:0/3428497296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cdc1034b0 0x7f6cdc1038c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009990 tx=0x7f6cd0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.186+0000 7f6ce4200700 1 -- 172.21.15.43:0/3428497296 shutdown_connections 2024-05-27T07:37:58.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.186+0000 7f6ce4200700 1 --2- 172.21.15.43:0/3428497296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cdc108730 0x7f6cdc110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.186+0000 7f6ce4200700 1 --2- 172.21.15.43:0/3428497296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cdc1034b0 0x7f6cdc1038c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.186+0000 7f6ce4200700 1 --2- 172.21.15.43:0/3428497296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cdc107de0 0x7f6cdc1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.186+0000 7f6ce4200700 1 -- 172.21.15.43:0/3428497296 >> 172.21.15.43:0/3428497296 conn(0x7f6cdc0fedb0 msgr2=0x7f6cdc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:58.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.186+0000 7f6ce4200700 1 -- 172.21.15.43:0/3428497296 shutdown_connections 2024-05-27T07:37:58.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.187+0000 7f6ce4200700 1 -- 172.21.15.43:0/3428497296 wait complete. 2024-05-27T07:37:58.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.188+0000 7f6ce4200700 1 Processor -- start 2024-05-27T07:37:58.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.189+0000 7f6ce4200700 1 -- start start 2024-05-27T07:37:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.189+0000 7f6ce4200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cdc1034b0 0x7f6cdc199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.189+0000 7f6ce4200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cdc107de0 0x7f6cdc199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.189+0000 7f6ce4200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cdc108730 0x7f6cdc19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.190+0000 7f6ce4200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc113210 con 0x7f6cdc1034b0 2024-05-27T07:37:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.190+0000 7f6ce4200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc113090 con 0x7f6cdc108730 2024-05-27T07:37:58.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.190+0000 7f6ce4200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc113390 con 0x7f6cdc107de0 2024-05-27T07:37:58.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.190+0000 7f6ce0e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cdc107de0 0x7f6cdc199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.190+0000 7f6ce1800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cdc1034b0 0x7f6cdc199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.190+0000 7f6ce1800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cdc1034b0 0x7f6cdc199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45372/0 (socket says 172.21.15.43:45372) 2024-05-27T07:37:58.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.190+0000 7f6ce1800700 1 -- 172.21.15.43:0/4222938367 learned_addr learned my addr 172.21.15.43:0/4222938367 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:58.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.190+0000 7f6ce2200700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cdc108730 0x7f6cdc19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.191+0000 7f6ce1800700 1 -- 172.21.15.43:0/4222938367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cdc107de0 msgr2=0x7f6cdc199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.191+0000 7f6ce1800700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cdc107de0 0x7f6cdc199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.191+0000 7f6ce1800700 1 -- 172.21.15.43:0/4222938367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cdc108730 msgr2=0x7f6cdc19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.191+0000 7f6ce1800700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cdc108730 0x7f6cdc19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.191+0000 7f6ce1800700 1 -- 172.21.15.43:0/4222938367 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc1a2c20 con 0x7f6cdc1034b0 2024-05-27T07:37:58.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.191+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cdc107de0 0x7f6cdc199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:37:58.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.192+0000 7f6ce1800700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cdc1034b0 0x7f6cdc199960 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f6cd000b5c0 tx=0x7f6cd00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:58.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.193+0000 7f6cda000700 1 -- 172.21.15.43:0/4222938367 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0027070 con 0x7f6cdc1034b0 2024-05-27T07:37:58.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.193+0000 7f6ce4200700 1 -- 172.21.15.43:0/4222938367 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cdc1a2eb0 con 0x7f6cdc1034b0 2024-05-27T07:37:58.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.193+0000 7f6cda000700 1 -- 172.21.15.43:0/4222938367 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6cd000fb80 con 0x7f6cdc1034b0 2024-05-27T07:37:58.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.193+0000 7f6ce4200700 1 -- 172.21.15.43:0/4222938367 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cdc1a3390 con 0x7f6cdc1034b0 2024-05-27T07:37:58.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.194+0000 7f6cda000700 1 -- 172.21.15.43:0/4222938367 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0022ba0 con 0x7f6cdc1034b0 2024-05-27T07:37:58.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.198+0000 7f6cda000700 1 -- 172.21.15.43:0/4222938367 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6cd0004830 con 0x7f6cdc1034b0 2024-05-27T07:37:58.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.201+0000 7f6cda000700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6cc40b29f0 0x7f6cc40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.201+0000 7f6cda000700 1 -- 172.21.15.43:0/4222938367 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f6cd00e4570 con 0x7f6cdc1034b0 2024-05-27T07:37:58.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.202+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6cc40b29f0 0x7f6cc40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.221+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6cc40b29f0 0x7f6cc40b4f00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8005d90 tx=0x7f6cc8005d00 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:58.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.225+0000 7f6cf2caab80 1 Processor -- start 2024-05-27T07:37:58.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.225+0000 7f6cf2caab80 1 -- start start 2024-05-27T07:37:58.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.226+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 0x55a856bc57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.226+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 0x55a856bbf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.226+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856bbfdb0 0x55a856bc01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.226+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a856aede40 con 0x55a856a40e40 2024-05-27T07:37:58.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.226+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a856add9e0 con 0x55a856bbfdb0 2024-05-27T07:37:58.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.226+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a856ae1a20 con 0x55a856bc5d30 2024-05-27T07:37:58.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.227+0000 7f6ce2200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856bbfdb0 0x55a856bc01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.227+0000 7f6ce1800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 0x55a856bc57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.227+0000 7f6ce2200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856bbfdb0 0x55a856bc01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:49538/0 (socket says 172.21.15.43:49538) 2024-05-27T07:37:58.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.227+0000 7f6ce0e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 0x55a856bbf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.227+0000 7f6ce1800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 0x55a856bc57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45374/0 (socket says 172.21.15.43:45374) 2024-05-27T07:37:58.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.227+0000 7f6ce2200700 1 -- 172.21.15.43:0/1031589840 learned_addr learned my addr 172.21.15.43:0/1031589840 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:58.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.228+0000 7f6ce0e00700 1 -- 172.21.15.43:0/1031589840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856bbfdb0 msgr2=0x55a856bc01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:58.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.228+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/1031589840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856bbfdb0 0x55a856bc01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.228+0000 7f6ce0e00700 1 -- 172.21.15.43:0/1031589840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 msgr2=0x55a856bc57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:58.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.228+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/1031589840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 0x55a856bc57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.228+0000 7f6ce0e00700 1 -- 172.21.15.43:0/1031589840 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a856aa6420 con 0x55a856bc5d30 2024-05-27T07:37:58.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.228+0000 7f6ce1800700 1 --2- 172.21.15.43:0/1031589840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 0x55a856bc57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:37:58.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.229+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/1031589840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 0x55a856bbf870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc076aa0 tx=0x7f6cc8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d806a2c05736fb39 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:58.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.230+0000 7f6ccea00700 1 -- 172.21.15.43:0/1031589840 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8011b00 con 0x55a856bc5d30 2024-05-27T07:37:58.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.230+0000 7f6ccea00700 1 -- 172.21.15.43:0/1031589840 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6cc8023070 con 0x55a856bc5d30 2024-05-27T07:37:58.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.230+0000 7f6ccea00700 1 -- 172.21.15.43:0/1031589840 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc80103d0 con 0x55a856bc5d30 2024-05-27T07:37:58.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.232+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1031589840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 msgr2=0x55a856bbf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.232+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1031589840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 0x55a856bbf870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc076aa0 tx=0x7f6cc8009f90 comp rx=0 tx=0).stop 2024-05-27T07:37:58.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.232+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1031589840 shutdown_connections 2024-05-27T07:37:58.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.232+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1031589840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856bbfdb0 0x55a856bc01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.232+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1031589840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 0x55a856bc57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.232+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1031589840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 0x55a856bbf870 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.233+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1031589840 >> 172.21.15.43:0/1031589840 conn(0x55a856aa0fc0 msgr2=0x55a856bba8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:58.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.233+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1031589840 shutdown_connections 2024-05-27T07:37:58.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.234+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1031589840 wait complete. 2024-05-27T07:37:58.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.234+0000 7f6cf2caab80 1 -- start start 2024-05-27T07:37:58.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.235+0000 7f6cf2caab80 1 Processor -- start 2024-05-27T07:37:58.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.236+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856a40e40 0x55a856bc94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.236+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 0x55a856bc99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.236+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 0x55a856bc9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.236+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a8569e3fa0 con 0x55a856bbfdb0 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.236+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a856a7e760 con 0x55a856a40e40 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.236+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a856bcc800 con 0x55a856bc5d30 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.236+0000 7f6cf2caab80 10 client.0 _ll_register_callbacks cb 0x55a856a87c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.237+0000 7f6ce2200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 0x55a856bc9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.237+0000 7f6ce2200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 0x55a856bc9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:48352/0 (socket says 172.21.15.43:48352) 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.237+0000 7f6ce2200700 1 -- 172.21.15.43:0/1684165858 learned_addr learned my addr 172.21.15.43:0/1684165858 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.237+0000 7f6ce1800700 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856a40e40 0x55a856bc94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.237+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 0x55a856bc99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.238+0000 7f6ce1800700 1 -- 172.21.15.43:0/1684165858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 msgr2=0x55a856bc9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.238+0000 7f6ce1800700 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 0x55a856bc9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.238+0000 7f6ce1800700 1 -- 172.21.15.43:0/1684165858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 msgr2=0x55a856bc99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:58.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.238+0000 7f6ce1800700 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 0x55a856bc99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.238+0000 7f6ce1800700 1 -- 172.21.15.43:0/1684165858 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a856a2d8d0 con 0x55a856a40e40 2024-05-27T07:37:58.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.238+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 0x55a856bc99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:37:58.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.239+0000 7f6cc3400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:37:58.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.239+0000 7f6ce1800700 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856a40e40 0x55a856bc94a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00e1890 tx=0x7f6cd0022f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a36121836e3c6da4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:58.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.239+0000 7f6cc3400700 1 -- 172.21.15.43:0/1684165858 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00e4370 con 0x55a856a40e40 2024-05-27T07:37:58.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.239+0000 7f6cc3400700 1 -- 172.21.15.43:0/1684165858 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f6cd00a7aa0 con 0x55a856a40e40 2024-05-27T07:37:58.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.239+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1684165858 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cc8023070 con 0x55a856a40e40 2024-05-27T07:37:58.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.240+0000 7f6cc3400700 1 -- 172.21.15.43:0/1684165858 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f6cd00e4370 con 0x55a856a40e40 2024-05-27T07:37:58.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.240+0000 7f6cf2caab80 10 client.37820 fetch_fsmap learned FSMap version 147 2024-05-27T07:37:58.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.240+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1684165858 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55a856aae060 con 0x55a856a40e40 2024-05-27T07:37:58.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.240+0000 7f6cc3400700 1 -- 172.21.15.43:0/1684165858 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6cc8023070 con 0x55a856a40e40 2024-05-27T07:37:58.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.240+0000 7f6cc3400700 1 -- 172.21.15.43:0/1684165858 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f6cd00a5c60 con 0x55a856a40e40 2024-05-27T07:37:58.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.241+0000 7f6cf2caab80 10 client.37820 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:37:58.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.241+0000 7f6cf2caab80 10 client.37820 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:37:58.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.241+0000 7f6cf2caab80 20 client.37820 populate_metadata read hostname 'smithi043' 2024-05-27T07:37:58.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.241+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a856bd9800 0x55a856bdbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.241+0000 7f6cf2caab80 4 client.37820 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:37:58.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.241+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1684165858 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a856aefd90 con 0x55a856bd9800 2024-05-27T07:37:58.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.241+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a856bd9800 0x55a856bdbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.243+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a856bd9800 0x55a856bdbcb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a16622adeb478165 server_cookie=b37afedfd1fd120b in_seq=0 out_seq=0 2024-05-27T07:37:58.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.243+0000 7f6cc3400700 10 client.37820 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:37:58.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.244+0000 7f6cc3400700 1 -- 172.21.15.43:0/1684165858 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55a856aefd90 con 0x55a856bd9800 2024-05-27T07:37:58.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.244+0000 7f6cc3400700 10 client.37820 handle_command_reply: tid=0 2024-05-27T07:37:58.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.244+0000 7f6cf2caab80 1 client.37820 shutdown 2024-05-27T07:37:58.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.245+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1684165858 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856a40e40 msgr2=0x55a856bc94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.245+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856a40e40 0x55a856bc94a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00e1890 tx=0x7f6cd0022f70 comp rx=0 tx=0).stop 2024-05-27T07:37:58.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.245+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1684165858 shutdown_connections 2024-05-27T07:37:58.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.245+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a856bd9800 0x55a856bdbcb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.245+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bc5d30 0x55a856bc9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.245+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856a40e40 0x55a856bc94a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.245+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1684165858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 0x55a856bc99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.245+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1684165858 >> 172.21.15.43:0/1684165858 conn(0x55a856aa0fc0 msgr2=0x55a856bba580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.245+0000 7f6cc3400700 0 client.37820 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:37:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.246+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1684165858 shutdown_connections 2024-05-27T07:37:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.246+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1684165858 wait complete. 2024-05-27T07:37:58.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.246+0000 7f6cf2caab80 20 client.37820 trim_cache size 0 max 16384 2024-05-27T07:37:58.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.275+0000 7f6cf2caab80 1 Processor -- start 2024-05-27T07:37:58.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.275+0000 7f6cf2caab80 1 -- start start 2024-05-27T07:37:58.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.276+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bbf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.276+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856a40e40 0x55a856bcb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.277+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 0x55a856bd5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.277+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc1bf6f0 con 0x55a856bbfdb0 2024-05-27T07:37:58.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.277+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc1aab80 con 0x55a856aa0fc0 2024-05-27T07:37:58.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.277+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc1bf020 con 0x55a856a40e40 2024-05-27T07:37:58.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.277+0000 7f6ce1800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bbf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.277+0000 7f6ce1800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bbf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:49554/0 (socket says 172.21.15.43:49554) 2024-05-27T07:37:58.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.277+0000 7f6ce1800700 1 -- 172.21.15.43:0/3070654115 learned_addr learned my addr 172.21.15.43:0/3070654115 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:58.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.277+0000 7f6ce2200700 1 --2- 172.21.15.43:0/3070654115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 0x55a856bd5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.277+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/3070654115 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856a40e40 0x55a856bcb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.278+0000 7f6ce1800700 1 -- 172.21.15.43:0/3070654115 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856a40e40 msgr2=0x55a856bcb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:58.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.278+0000 7f6ce1800700 1 --2- 172.21.15.43:0/3070654115 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856a40e40 0x55a856bcb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.278+0000 7f6ce1800700 1 -- 172.21.15.43:0/3070654115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 msgr2=0x55a856bd5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:58.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.278+0000 7f6ce1800700 1 --2- 172.21.15.43:0/3070654115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 0x55a856bd5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.278+0000 7f6ce1800700 1 -- 172.21.15.43:0/3070654115 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a856bc0b30 con 0x55a856aa0fc0 2024-05-27T07:37:58.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.279+0000 7f6ce1800700 1 --2- 172.21.15.43:0/3070654115 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bbf870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00a8720 tx=0x7f6cd00a74a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5f269b7a4308c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:58.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.280+0000 7f6cce000700 1 -- 172.21.15.43:0/3070654115 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd006e070 con 0x55a856aa0fc0 2024-05-27T07:37:58.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.280+0000 7f6cce000700 1 -- 172.21.15.43:0/3070654115 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6cd00a5bd0 con 0x55a856aa0fc0 2024-05-27T07:37:58.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.280+0000 7f6cce000700 1 -- 172.21.15.43:0/3070654115 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00ab970 con 0x55a856aa0fc0 2024-05-27T07:37:58.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.281+0000 7f6cf2caab80 1 -- 172.21.15.43:0/3070654115 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 msgr2=0x55a856bbf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.281+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/3070654115 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bbf870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00a8720 tx=0x7f6cd00a74a0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.282+0000 7f6cf2caab80 1 -- 172.21.15.43:0/3070654115 shutdown_connections 2024-05-27T07:37:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.282+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/3070654115 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856bbfdb0 0x55a856bd5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.282+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/3070654115 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bbf870 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.282+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/3070654115 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856a40e40 0x55a856bcb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.282+0000 7f6cf2caab80 1 -- 172.21.15.43:0/3070654115 >> 172.21.15.43:0/3070654115 conn(0x55a856bdbfd0 msgr2=0x55a856bc3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.282+0000 7f6cf2caab80 1 -- 172.21.15.43:0/3070654115 shutdown_connections 2024-05-27T07:37:58.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.282+0000 7f6cf2caab80 1 -- 172.21.15.43:0/3070654115 wait complete. 2024-05-27T07:37:58.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.283+0000 7f6cf2caab80 1 -- start start 2024-05-27T07:37:58.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.284+0000 7f6cf2caab80 1 Processor -- start 2024-05-27T07:37:58.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.286+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 0x55a856bd1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.287+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bd1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.287+0000 7f6cf2caab80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bbfdb0 0x55a856be0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.287+0000 7f6ce0e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bd1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.287+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc1be950 con 0x55a856a40e40 2024-05-27T07:37:58.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.287+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc1ab250 con 0x55a856aa0fc0 2024-05-27T07:37:58.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.287+0000 7f6cf2caab80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a856bc7a10 con 0x55a856bbfdb0 2024-05-27T07:37:58.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.288+0000 7f6ce1800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 0x55a856bd1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.288+0000 7f6ce0e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bd1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:49564/0 (socket says 172.21.15.43:49564) 2024-05-27T07:37:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.288+0000 7f6ce0e00700 1 -- 172.21.15.43:0/1341838752 learned_addr learned my addr 172.21.15.43:0/1341838752 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:37:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.288+0000 7f6cf2caab80 10 client.0 _ll_register_callbacks cb 0x55a856a87c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:37:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.288+0000 7f6ce2200700 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bbfdb0 0x55a856be0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.289+0000 7f6ce0e00700 1 -- 172.21.15.43:0/1341838752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bbfdb0 msgr2=0x55a856be0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.289+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bbfdb0 0x55a856be0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.289+0000 7f6ce0e00700 1 -- 172.21.15.43:0/1341838752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 msgr2=0x55a856bd1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:37:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.289+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 0x55a856bd1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.289+0000 7f6ce0e00700 1 -- 172.21.15.43:0/1341838752 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a856be09a0 con 0x55a856aa0fc0 2024-05-27T07:37:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.289+0000 7f6ce0e00700 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bd1f90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800ecc0 tx=0x7f6cc8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed63de7ab9d155a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:37:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.289+0000 7f6cc3400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:37:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.290+0000 7f6cc3400700 1 -- 172.21.15.43:0/1341838752 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc800e680 con 0x55a856aa0fc0 2024-05-27T07:37:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.290+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1341838752 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cd00a5bd0 con 0x55a856aa0fc0 2024-05-27T07:37:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.290+0000 7f6cc3400700 1 -- 172.21.15.43:0/1341838752 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 9496+0+0 (secure 0 0 0) 0x7f6cc8004870 con 0x55a856aa0fc0 2024-05-27T07:37:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.290+0000 7f6cc3400700 1 -- 172.21.15.43:0/1341838752 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f6cc8004a70 con 0x55a856aa0fc0 2024-05-27T07:37:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.290+0000 7f6cf2caab80 10 client.37826 fetch_fsmap learned FSMap version 147 2024-05-27T07:37:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.291+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1341838752 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55a856c43a60 con 0x55a856aa0fc0 2024-05-27T07:37:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.291+0000 7f6cc3400700 1 -- 172.21.15.43:0/1341838752 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6cd00a5bd0 con 0x55a856aa0fc0 2024-05-27T07:37:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.291+0000 7f6cc3400700 1 -- 172.21.15.43:0/1341838752 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f6cc800f040 con 0x55a856aa0fc0 2024-05-27T07:37:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.291+0000 7f6cf2caab80 10 client.37826 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:37:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.291+0000 7f6cf2caab80 10 client.37826 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:37:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.291+0000 7f6cf2caab80 20 client.37826 populate_metadata read hostname 'smithi043' 2024-05-27T07:37:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.291+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a856bc5d30 0x55a856c46190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:37:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.291+0000 7f6cf2caab80 4 client.37826 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:37:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.291+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1341838752 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a856ae9570 con 0x55a856bc5d30 2024-05-27T07:37:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.292+0000 7f6ce1800700 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a856bc5d30 0x55a856c46190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:37:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.293+0000 7f6cc3400700 10 client.37826 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:37:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.293+0000 7f6ce1800700 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a856bc5d30 0x55a856c46190 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a35c6e32851e66db server_cookie=c9ed7990e3ddb688 in_seq=0 out_seq=0 2024-05-27T07:37:58.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.294+0000 7f6cc3400700 1 -- 172.21.15.43:0/1341838752 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a856ae9570 con 0x55a856bc5d30 2024-05-27T07:37:58.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.294+0000 7f6cc3400700 10 client.37826 handle_command_reply: tid=0 2024-05-27T07:37:58.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.294+0000 7f6cf2caab80 1 client.37826 shutdown 2024-05-27T07:37:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.294+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1341838752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 msgr2=0x55a856bd1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.294+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bd1f90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800ecc0 tx=0x7f6cc8005fd0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.294+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1341838752 shutdown_connections 2024-05-27T07:37:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.295+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x55a856bc5d30 0x55a856c46190 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.295+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a856bbfdb0 0x55a856be0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.295+0000 7f6cc3400700 0 client.37826 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:37:58.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.295+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a856a40e40 0x55a856bd1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.295+0000 7f6cf2caab80 1 --2- 172.21.15.43:0/1341838752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a856aa0fc0 0x55a856bd1f90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.295+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1341838752 >> 172.21.15.43:0/1341838752 conn(0x55a856bdbfd0 msgr2=0x55a856bc4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:58.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.295+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1341838752 shutdown_connections 2024-05-27T07:37:58.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.295+0000 7f6cf2caab80 1 -- 172.21.15.43:0/1341838752 wait complete. 2024-05-27T07:37:58.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.295+0000 7f6cf2caab80 20 client.37826 trim_cache size 0 max 16384 2024-05-27T07:37:58.292 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:37:58.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.296+0000 7f6ce4200700 1 -- 172.21.15.43:0/4222938367 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6cc40b29f0 msgr2=0x7f6cc40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.296+0000 7f6ce4200700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6cc40b29f0 0x7f6cc40b4f00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8005d90 tx=0x7f6cc8005d00 comp rx=0 tx=0).stop 2024-05-27T07:37:58.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.296+0000 7f6ce4200700 1 -- 172.21.15.43:0/4222938367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cdc1034b0 msgr2=0x7f6cdc199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:37:58.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.296+0000 7f6ce4200700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cdc1034b0 0x7f6cdc199960 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f6cd000b5c0 tx=0x7f6cd00047b0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.296+0000 7f6ce4200700 1 -- 172.21.15.43:0/4222938367 shutdown_connections 2024-05-27T07:37:58.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.296+0000 7f6ce4200700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6cc40b29f0 0x7f6cc40b4f00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.296+0000 7f6ce4200700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6cdc108730 0x7f6cdc19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.296+0000 7f6ce4200700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cdc1034b0 0x7f6cdc199960 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.296+0000 7f6ce4200700 1 --2- 172.21.15.43:0/4222938367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cdc107de0 0x7f6cdc199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:37:58.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.296+0000 7f6ce4200700 1 -- 172.21.15.43:0/4222938367 >> 172.21.15.43:0/4222938367 conn(0x7f6cdc0fedb0 msgr2=0x7f6cdc101d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:37:58.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.297+0000 7f6ce4200700 1 -- 172.21.15.43:0/4222938367 shutdown_connections 2024-05-27T07:37:58.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:37:58.297+0000 7f6ce4200700 1 -- 172.21.15.43:0/4222938367 wait complete. 2024-05-27T07:37:58.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 65593 5.84 MB/sec execute 479 sec latency 159.753 ms 2024-05-27T07:37:59.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:37:58 smithi043 ceph-mon[31531]: pgmap v807: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.5 MiB/s wr, 170 op/s 2024-05-27T07:37:59.214 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:37:58 smithi179 ceph-mon[35888]: pgmap v807: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.5 MiB/s wr, 170 op/s 2024-05-27T07:37:59.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:37:59.346 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:37:58 smithi132 ceph-mon[35283]: pgmap v807: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.5 MiB/s wr, 170 op/s 2024-05-27T07:37:59.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 65653 5.83 MB/sec execute 480 sec latency 362.772 ms 2024-05-27T07:37:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:00.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 65771 5.84 MB/sec execute 481 sec latency 126.103 ms 2024-05-27T07:38:01.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:00 smithi043 ceph-mon[31531]: pgmap v808: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.5 MiB/s wr, 169 op/s 2024-05-27T07:38:01.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:00 smithi179 ceph-mon[35888]: pgmap v808: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.5 MiB/s wr, 169 op/s 2024-05-27T07:38:01.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:00 smithi132 ceph-mon[35283]: pgmap v808: 129 pgs: 129 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.5 MiB/s wr, 169 op/s 2024-05-27T07:38:01.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 65889 5.85 MB/sec execute 482 sec latency 230.430 ms 2024-05-27T07:38:02.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 65958 5.84 MB/sec execute 483 sec latency 296.381 ms 2024-05-27T07:38:03.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:02 smithi043 ceph-mon[31531]: pgmap v809: 129 pgs: 129 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 4.9 MiB/s wr, 187 op/s 2024-05-27T07:38:03.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:02 smithi043 ceph-mon[31531]: osdmap e158: 12 total, 12 up, 12 in 2024-05-27T07:38:03.346 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:02 smithi132 ceph-mon[35283]: pgmap v809: 129 pgs: 129 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 4.9 MiB/s wr, 187 op/s 2024-05-27T07:38:03.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:02 smithi132 ceph-mon[35283]: osdmap e158: 12 total, 12 up, 12 in 2024-05-27T07:38:03.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:02 smithi179 ceph-mon[35888]: pgmap v809: 129 pgs: 129 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 4.9 MiB/s wr, 187 op/s 2024-05-27T07:38:03.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:02 smithi179 ceph-mon[35888]: osdmap e158: 12 total, 12 up, 12 in 2024-05-27T07:38:03.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 66074 5.85 MB/sec execute 484 sec latency 96.551 ms 2024-05-27T07:38:04.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.438+0000 7f31b7200700 1 -- 172.21.15.43:0/2053727219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31b81043f0 msgr2=0x7f31b8104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.438+0000 7f31b7200700 1 --2- 172.21.15.43:0/2053727219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31b81043f0 0x7f31b8104800 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f31a80099c0 tx=0x7f31a8009cd0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.438+0000 7f31b7200700 1 -- 172.21.15.43:0/2053727219 shutdown_connections 2024-05-27T07:38:04.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.438+0000 7f31b7200700 1 --2- 172.21.15.43:0/2053727219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b8109670 0x7f31b8110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.438+0000 7f31b7200700 1 --2- 172.21.15.43:0/2053727219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31b81043f0 0x7f31b8104800 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.438+0000 7f31b7200700 1 --2- 172.21.15.43:0/2053727219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b8108d20 0x7f31b8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.438+0000 7f31b7200700 1 -- 172.21.15.43:0/2053727219 >> 172.21.15.43:0/2053727219 conn(0x7f31b8079d00 msgr2=0x7f31b807a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.439+0000 7f31b7200700 1 -- 172.21.15.43:0/2053727219 shutdown_connections 2024-05-27T07:38:04.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.439+0000 7f31b7200700 1 -- 172.21.15.43:0/2053727219 wait complete. 2024-05-27T07:38:04.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.440+0000 7f31b7200700 1 Processor -- start 2024-05-27T07:38:04.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.441+0000 7f31b7200700 1 -- start start 2024-05-27T07:38:04.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.441+0000 7f31b7200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b81043f0 0x7f31b819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.441+0000 7f31b7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b8108d20 0x7f31b819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.441+0000 7f31b7200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31b8109670 0x7f31b8198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.441+0000 7f31b7200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f31b8111fb0 con 0x7f31b8109670 2024-05-27T07:38:04.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.441+0000 7f31b7200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f31b8111e30 con 0x7f31b8108d20 2024-05-27T07:38:04.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.441+0000 7f31b7200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f31b8112130 con 0x7f31b81043f0 2024-05-27T07:38:04.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.442+0000 7f31b5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b8108d20 0x7f31b819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.442+0000 7f31b5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b8108d20 0x7f31b819eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40326/0 (socket says 172.21.15.43:40326) 2024-05-27T07:38:04.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.442+0000 7f31b5400700 1 -- 172.21.15.43:0/1602857316 learned_addr learned my addr 172.21.15.43:0/1602857316 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:04.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.442+0000 7f31b5e00700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b81043f0 0x7f31b819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.442+0000 7f31b6800700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31b8109670 0x7f31b8198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.442+0000 7f31b5400700 1 -- 172.21.15.43:0/1602857316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b81043f0 msgr2=0x7f31b819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.442+0000 7f31b5400700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b81043f0 0x7f31b819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.443+0000 7f31b5400700 1 -- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31b8109670 msgr2=0x7f31b8198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.443+0000 7f31b5400700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31b8109670 0x7f31b8198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.443+0000 7f31b5400700 1 -- 172.21.15.43:0/1602857316 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b8198f20 con 0x7f31b8108d20 2024-05-27T07:38:04.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.443+0000 7f31b6800700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31b8109670 0x7f31b8198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:04.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.443+0000 7f31b5400700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b8108d20 0x7f31b819eb90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f31a4012fc0 tx=0x7f31a40100c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.444+0000 7f31aea00700 1 -- 172.21.15.43:0/1602857316 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a400d2e0 con 0x7f31b8108d20 2024-05-27T07:38:04.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.444+0000 7f31aea00700 1 -- 172.21.15.43:0/1602857316 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31a4004cc0 con 0x7f31b8108d20 2024-05-27T07:38:04.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.444+0000 7f31aea00700 1 -- 172.21.15.43:0/1602857316 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4005020 con 0x7f31b8108d20 2024-05-27T07:38:04.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.444+0000 7f31b7200700 1 -- 172.21.15.43:0/1602857316 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31b81991b0 con 0x7f31b8108d20 2024-05-27T07:38:04.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.444+0000 7f31b7200700 1 -- 172.21.15.43:0/1602857316 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31b81996a0 con 0x7f31b8108d20 2024-05-27T07:38:04.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.450+0000 7f31cb72cb80 1 Processor -- start 2024-05-27T07:38:04.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.451+0000 7f31cb72cb80 1 -- start start 2024-05-27T07:38:04.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.451+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf64f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.451+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf64fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.451+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf65b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.451+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5621cf57ce40 con 0x5621cf4cfe40 2024-05-27T07:38:04.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.451+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5621cf56c9e0 con 0x5621cf5bb9f0 2024-05-27T07:38:04.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.451+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5621cf570a20 con 0x5621cf6592c0 2024-05-27T07:38:04.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.452+0000 7f31b6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf64f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.452+0000 7f31b6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf64f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40330/0 (socket says 172.21.15.43:40330) 2024-05-27T07:38:04.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.452+0000 7f31b6800700 1 -- 172.21.15.43:0/1531286877 learned_addr learned my addr 172.21.15.43:0/1531286877 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:04.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.452+0000 7f31b6800700 1 -- 172.21.15.43:0/1531286877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 msgr2=0x5621cf65b6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:38:04.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.452+0000 7f31b6800700 1 --2- 172.21.15.43:0/1531286877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf65b6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.452+0000 7f31b6800700 1 -- 172.21.15.43:0/1531286877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 msgr2=0x5621cf64fc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:38:04.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.453+0000 7f31b5e00700 1 --2- 172.21.15.43:0/1531286877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf64fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.453+0000 7f31b6800700 1 --2- 172.21.15.43:0/1531286877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf64fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.453+0000 7f31b6800700 1 -- 172.21.15.43:0/1531286877 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621cf535420 con 0x5621cf5bb9f0 2024-05-27T07:38:04.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.453+0000 7f31aea00700 1 -- 172.21.15.43:0/1602857316 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f31a401f020 con 0x7f31b8108d20 2024-05-27T07:38:04.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.453+0000 7f31b6800700 1 --2- 172.21.15.43:0/1531286877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf64f750 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009400 tx=0x7f31b0009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de99ebc09c255d9d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.454+0000 7f31a3400700 1 -- 172.21.15.43:0/1531286877 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b001a070 con 0x5621cf5bb9f0 2024-05-27T07:38:04.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.454+0000 7f31a3400700 1 -- 172.21.15.43:0/1531286877 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31b0015a80 con 0x5621cf5bb9f0 2024-05-27T07:38:04.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.454+0000 7f31a3400700 1 -- 172.21.15.43:0/1531286877 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b00148c0 con 0x5621cf5bb9f0 2024-05-27T07:38:04.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.455+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1531286877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 msgr2=0x5621cf64f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.455+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/1531286877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf64f750 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009400 tx=0x7f31b0009920 comp rx=0 tx=0).stop 2024-05-27T07:38:04.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.456+0000 7f31aea00700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f319c0b2a70 0x7f319c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.456+0000 7f31aea00700 1 -- 172.21.15.43:0/1602857316 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 9584+0+0 (secure 0 0 0) 0x7f31a40e6420 con 0x7f31b8108d20 2024-05-27T07:38:04.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.456+0000 7f31b6800700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f319c0b2a70 0x7f319c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.456+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1531286877 shutdown_connections 2024-05-27T07:38:04.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.456+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/1531286877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf65b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.456+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/1531286877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf64f750 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.456+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/1531286877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf64fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.456+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1531286877 >> 172.21.15.43:0/1531286877 conn(0x5621cf569390 msgr2=0x5621cf64a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.458+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1531286877 shutdown_connections 2024-05-27T07:38:04.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.458+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1531286877 wait complete. 2024-05-27T07:38:04.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.458+0000 7f31b6800700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f319c0b2a70 0x7f319c0b4f80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f31b0007fd0 tx=0x7f31b000d040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.459+0000 7f31cb72cb80 1 -- start start 2024-05-27T07:38:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.459+0000 7f31cb72cb80 1 Processor -- start 2024-05-27T07:38:04.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.460+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.460+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf656820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.460+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf656d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.460+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5621cf472fa0 con 0x5621cf4cfe40 2024-05-27T07:38:04.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.460+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5621cf50d760 con 0x5621cf5bb9f0 2024-05-27T07:38:04.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.460+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5621cf65be00 con 0x5621cf6592c0 2024-05-27T07:38:04.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.460+0000 7f31cb72cb80 10 client.0 _ll_register_callbacks cb 0x5621cf516c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:04.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.460+0000 7f31b5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.460+0000 7f31b5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52682/0 (socket says 172.21.15.43:52682) 2024-05-27T07:38:04.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.460+0000 7f31b5e00700 1 -- 172.21.15.43:0/3248193570 learned_addr learned my addr 172.21.15.43:0/3248193570 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:04.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.461+0000 7f31b6800700 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf656d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.461+0000 7f31b5400700 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf656820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.461+0000 7f31b5e00700 1 -- 172.21.15.43:0/3248193570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 msgr2=0x5621cf656d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.461+0000 7f31b5e00700 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf656d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.461+0000 7f31b5e00700 1 -- 172.21.15.43:0/3248193570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 msgr2=0x5621cf656820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.461+0000 7f31b5e00700 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf656820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.461+0000 7f31b5e00700 1 -- 172.21.15.43:0/3248193570 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621cf4bc8d0 con 0x5621cf4cfe40 2024-05-27T07:38:04.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.462+0000 7f31b5e00700 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6562e0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f31a80099c0 tx=0x7f31a8005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91954405771d524 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.462+0000 7f3197e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:38:04.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.462+0000 7f3197e00700 1 -- 172.21.15.43:0/3248193570 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a80039c0 con 0x5621cf4cfe40 2024-05-27T07:38:04.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.463+0000 7f3197e00700 1 -- 172.21.15.43:0/3248193570 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9584+0+0 (secure 0 0 0) 0x7f31a8003b60 con 0x5621cf4cfe40 2024-05-27T07:38:04.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.463+0000 7f31cb72cb80 1 -- 172.21.15.43:0/3248193570 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b0015a80 con 0x5621cf4cfe40 2024-05-27T07:38:04.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.464+0000 7f3197e00700 1 -- 172.21.15.43:0/3248193570 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f31a8003d60 con 0x5621cf4cfe40 2024-05-27T07:38:04.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.464+0000 7f31cb72cb80 10 client.18447 fetch_fsmap learned FSMap version 147 2024-05-27T07:38:04.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.464+0000 7f31cb72cb80 1 -- 172.21.15.43:0/3248193570 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x5621cf53d060 con 0x5621cf4cfe40 2024-05-27T07:38:04.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.464+0000 7f3197e00700 1 -- 172.21.15.43:0/3248193570 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31b0015a80 con 0x5621cf4cfe40 2024-05-27T07:38:04.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.464+0000 7f3197e00700 1 -- 172.21.15.43:0/3248193570 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f31a802d550 con 0x5621cf4cfe40 2024-05-27T07:38:04.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.465+0000 7f31cb72cb80 10 client.18447 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:38:04.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.465+0000 7f31cb72cb80 10 client.18447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:04.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.465+0000 7f31cb72cb80 20 client.18447 populate_metadata read hostname 'smithi043' 2024-05-27T07:38:04.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.465+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5621cf669230 0x5621cf66b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.465+0000 7f31cb72cb80 4 client.18447 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:38:04.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.465+0000 7f31cb72cb80 1 -- 172.21.15.43:0/3248193570 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621cf57ed90 con 0x5621cf669230 2024-05-27T07:38:04.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.466+0000 7f31b5400700 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5621cf669230 0x5621cf66b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.467+0000 7f31b5400700 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5621cf669230 0x5621cf66b6c0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21f4434fcf0dc99b server_cookie=8fa18bb12f321ea in_seq=0 out_seq=0 2024-05-27T07:38:04.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.467+0000 7f3197e00700 10 client.18447 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:04.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.468+0000 7f3197e00700 1 -- 172.21.15.43:0/3248193570 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5621cf57ed90 con 0x5621cf669230 2024-05-27T07:38:04.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.468+0000 7f3197e00700 10 client.18447 handle_command_reply: tid=0 2024-05-27T07:38:04.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.469+0000 7f31cb72cb80 1 client.18447 shutdown 2024-05-27T07:38:04.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.471+0000 7f31cb72cb80 1 -- 172.21.15.43:0/3248193570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 msgr2=0x5621cf6562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.471+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6562e0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f31a80099c0 tx=0x7f31a8005760 comp rx=0 tx=0).stop 2024-05-27T07:38:04.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.471+0000 7f31cb72cb80 1 -- 172.21.15.43:0/3248193570 shutdown_connections 2024-05-27T07:38:04.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.471+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf656d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.471+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5621cf669230 0x5621cf66b6c0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.471+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6562e0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.471+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/3248193570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf656820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.471+0000 7f3197e00700 0 client.18447 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:04.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.471+0000 7f31cb72cb80 1 -- 172.21.15.43:0/3248193570 >> 172.21.15.43:0/3248193570 conn(0x5621cf569390 msgr2=0x5621cf64a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.473+0000 7f31cb72cb80 1 -- 172.21.15.43:0/3248193570 shutdown_connections 2024-05-27T07:38:04.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.473+0000 7f31cb72cb80 1 -- 172.21.15.43:0/3248193570 wait complete. 2024-05-27T07:38:04.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.474+0000 7f31cb72cb80 20 client.18447 trim_cache size 0 max 16384 2024-05-27T07:38:04.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.510+0000 7f31cb72cb80 1 Processor -- start 2024-05-27T07:38:04.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.510+0000 7f31cb72cb80 1 -- start start 2024-05-27T07:38:04.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.511+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf4cfe40 0x5621cf64a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.511+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf655240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.511+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf6592c0 0x5621cf664f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.511+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5621cf64fe80 con 0x5621cf6592c0 2024-05-27T07:38:04.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.511+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f31b81ae7c0 con 0x5621cf5bb9f0 2024-05-27T07:38:04.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.511+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f31b81acc80 con 0x5621cf4cfe40 2024-05-27T07:38:04.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.511+0000 7f31b5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf655240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.512+0000 7f31b5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf655240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40350/0 (socket says 172.21.15.43:40350) 2024-05-27T07:38:04.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.512+0000 7f31b5400700 1 -- 172.21.15.43:0/880747865 learned_addr learned my addr 172.21.15.43:0/880747865 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:04.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.512+0000 7f31b5e00700 1 --2- 172.21.15.43:0/880747865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf4cfe40 0x5621cf64a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.512+0000 7f31b5400700 1 -- 172.21.15.43:0/880747865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf4cfe40 msgr2=0x5621cf64a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.513+0000 7f31b5400700 1 --2- 172.21.15.43:0/880747865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf4cfe40 0x5621cf64a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.513+0000 7f31b5400700 1 -- 172.21.15.43:0/880747865 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf6592c0 msgr2=0x5621cf664f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.513+0000 7f31b5400700 1 --2- 172.21.15.43:0/880747865 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf6592c0 0x5621cf664f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.513+0000 7f31b5400700 1 -- 172.21.15.43:0/880747865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621cf654850 con 0x5621cf5bb9f0 2024-05-27T07:38:04.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.513+0000 7f31b5e00700 1 --2- 172.21.15.43:0/880747865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf4cfe40 0x5621cf64a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:04.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.515+0000 7f31b5400700 1 --2- 172.21.15.43:0/880747865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf655240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f31a40ad230 tx=0x7f31a400d480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f17964126b502ead server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.516+0000 7f31a2a00700 1 -- 172.21.15.43:0/880747865 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a40e3e60 con 0x5621cf5bb9f0 2024-05-27T07:38:04.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.516+0000 7f31a2a00700 1 -- 172.21.15.43:0/880747865 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31a40ad9a0 con 0x5621cf5bb9f0 2024-05-27T07:38:04.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.516+0000 7f31a2a00700 1 -- 172.21.15.43:0/880747865 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a40ad500 con 0x5621cf5bb9f0 2024-05-27T07:38:04.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.519+0000 7f31cb72cb80 1 -- 172.21.15.43:0/880747865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 msgr2=0x5621cf655240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.519+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/880747865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf655240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f31a40ad230 tx=0x7f31a400d480 comp rx=0 tx=0).stop 2024-05-27T07:38:04.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.520+0000 7f31cb72cb80 1 -- 172.21.15.43:0/880747865 shutdown_connections 2024-05-27T07:38:04.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.520+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/880747865 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf6592c0 0x5621cf664f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.520+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/880747865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf4cfe40 0x5621cf64a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.520+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/880747865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf655240 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.520+0000 7f31cb72cb80 1 -- 172.21.15.43:0/880747865 >> 172.21.15.43:0/880747865 conn(0x5621cf569390 msgr2=0x5621cf64e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.521+0000 7f31cb72cb80 1 -- 172.21.15.43:0/880747865 shutdown_connections 2024-05-27T07:38:04.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.521+0000 7f31cb72cb80 1 -- 172.21.15.43:0/880747865 wait complete. 2024-05-27T07:38:04.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.523+0000 7f31cb72cb80 1 -- start start 2024-05-27T07:38:04.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.524+0000 7f31cb72cb80 1 Processor -- start 2024-05-27T07:38:04.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.525+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6d4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.525+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf6d5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.525+0000 7f31cb72cb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf6600e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.525+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f31b81ad350 con 0x5621cf4cfe40 2024-05-27T07:38:04.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.525+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f31b81ae0f0 con 0x5621cf5bb9f0 2024-05-27T07:38:04.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.525+0000 7f31cb72cb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f31b81ada20 con 0x5621cf6592c0 2024-05-27T07:38:04.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.525+0000 7f31cb72cb80 10 client.0 _ll_register_callbacks cb 0x5621cf516c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:04.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.528+0000 7f31b5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6d4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.528+0000 7f31b5e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6d4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52688/0 (socket says 172.21.15.43:52688) 2024-05-27T07:38:04.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.528+0000 7f31b5e00700 1 -- 172.21.15.43:0/1562161476 learned_addr learned my addr 172.21.15.43:0/1562161476 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:04.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.529+0000 7f31b6800700 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf6600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.530+0000 7f31b5400700 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf6d5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.531+0000 7f31b5e00700 1 -- 172.21.15.43:0/1562161476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 msgr2=0x5621cf6600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.531+0000 7f31b5e00700 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf6600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.531+0000 7f31b5e00700 1 -- 172.21.15.43:0/1562161476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 msgr2=0x5621cf6d5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:04.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.531+0000 7f31b5e00700 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf6d5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.531+0000 7f31b5e00700 1 -- 172.21.15.43:0/1562161476 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621cf660620 con 0x5621cf4cfe40 2024-05-27T07:38:04.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.532+0000 7f31b5400700 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf6d5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:04.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.532+0000 7f31b6800700 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf6600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:04.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.533+0000 7f31b5e00700 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6d4d00 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009d10 tx=0x7f31a8009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fb2a82615c7004 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:04.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.534+0000 7f3197e00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:38:04.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.535+0000 7f3197e00700 1 -- 172.21.15.43:0/1562161476 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a80173f0 con 0x5621cf4cfe40 2024-05-27T07:38:04.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.535+0000 7f3197e00700 1 -- 172.21.15.43:0/1562161476 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(158..158 src has 1..158) ==== 9584+0+0 (secure 0 0 0) 0x7f31a8017590 con 0x5621cf4cfe40 2024-05-27T07:38:04.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.536+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1562161476 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31a40ad9a0 con 0x5621cf4cfe40 2024-05-27T07:38:04.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.536+0000 7f3197e00700 1 -- 172.21.15.43:0/1562161476 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f31a8017790 con 0x5621cf4cfe40 2024-05-27T07:38:04.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.537+0000 7f31cb72cb80 10 client.18450 fetch_fsmap learned FSMap version 147 2024-05-27T07:38:04.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.537+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1562161476 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x5621cf661750 con 0x5621cf4cfe40 2024-05-27T07:38:04.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.539+0000 7f3197e00700 1 -- 172.21.15.43:0/1562161476 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31a40ad9a0 con 0x5621cf4cfe40 2024-05-27T07:38:04.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.539+0000 7f3197e00700 1 -- 172.21.15.43:0/1562161476 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f31a800b730 con 0x5621cf4cfe40 2024-05-27T07:38:04.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.540+0000 7f31cb72cb80 10 client.18450 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:38:04.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.540+0000 7f31cb72cb80 10 client.18450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:04.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.540+0000 7f31cb72cb80 20 client.18450 populate_metadata read hostname 'smithi043' 2024-05-27T07:38:04.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.540+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5621cf6cbd10 0x5621cf6ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:04.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.540+0000 7f31cb72cb80 4 client.18450 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:38:04.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.540+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1562161476 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621cf578570 con 0x5621cf6cbd10 2024-05-27T07:38:04.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.541+0000 7f31b5400700 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5621cf6cbd10 0x5621cf6ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:04.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.542+0000 7f31b5400700 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5621cf6cbd10 0x5621cf6ce130 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82c38508d99a1fa1 server_cookie=296de529b609284a in_seq=0 out_seq=0 2024-05-27T07:38:04.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.542+0000 7f3197e00700 10 client.18450 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:04.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.544+0000 7f3197e00700 1 -- 172.21.15.43:0/1562161476 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621cf578570 con 0x5621cf6cbd10 2024-05-27T07:38:04.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.544+0000 7f3197e00700 10 client.18450 handle_command_reply: tid=0 2024-05-27T07:38:04.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.544+0000 7f31cb72cb80 1 client.18450 shutdown 2024-05-27T07:38:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.545+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1562161476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 msgr2=0x5621cf6d4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.545+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6d4d00 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009d10 tx=0x7f31a8009fd0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.545+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1562161476 shutdown_connections 2024-05-27T07:38:04.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.545+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5621cf6592c0 0x5621cf6600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.545+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5621cf6cbd10 0x5621cf6ce130 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.545+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5621cf4cfe40 0x5621cf6d4d00 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.545+0000 7f31cb72cb80 1 --2- 172.21.15.43:0/1562161476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5621cf5bb9f0 0x5621cf6d5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.545+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1562161476 >> 172.21.15.43:0/1562161476 conn(0x5621cf569390 msgr2=0x5621cf6518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.545+0000 7f3197e00700 0 client.18450 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:04.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.546+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1562161476 shutdown_connections 2024-05-27T07:38:04.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.546+0000 7f31cb72cb80 1 -- 172.21.15.43:0/1562161476 wait complete. 2024-05-27T07:38:04.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.546+0000 7f31cb72cb80 20 client.18450 trim_cache size 0 max 16384 2024-05-27T07:38:04.545 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:38:04.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.550+0000 7f31b7200700 1 -- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f319c0b2a70 msgr2=0x7f319c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.550+0000 7f31b7200700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f319c0b2a70 0x7f319c0b4f80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f31b0007fd0 tx=0x7f31b000d040 comp rx=0 tx=0).stop 2024-05-27T07:38:04.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.550+0000 7f31b7200700 1 -- 172.21.15.43:0/1602857316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b8108d20 msgr2=0x7f31b819eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:04.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.550+0000 7f31b7200700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b8108d20 0x7f31b819eb90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f31a4012fc0 tx=0x7f31a40100c0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.550+0000 7f31b7200700 1 -- 172.21.15.43:0/1602857316 shutdown_connections 2024-05-27T07:38:04.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.550+0000 7f31b7200700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f319c0b2a70 0x7f319c0b4f80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.550+0000 7f31b7200700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f31b8109670 0x7f31b8198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.550+0000 7f31b7200700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b81043f0 0x7f31b819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.550+0000 7f31b7200700 1 --2- 172.21.15.43:0/1602857316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b8108d20 0x7f31b819eb90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:04.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.550+0000 7f31b7200700 1 -- 172.21.15.43:0/1602857316 >> 172.21.15.43:0/1602857316 conn(0x7f31b8079d00 msgr2=0x7f31b8102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:04.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.551+0000 7f31b7200700 1 -- 172.21.15.43:0/1602857316 shutdown_connections 2024-05-27T07:38:04.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:04.551+0000 7f31b7200700 1 -- 172.21.15.43:0/1602857316 wait complete. 2024-05-27T07:38:04.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 66221 5.86 MB/sec execute 485 sec latency 111.349 ms 2024-05-27T07:38:05.243 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:04 smithi043 ceph-mon[31531]: pgmap v811: 129 pgs: 129 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 4.7 MiB/s wr, 179 op/s 2024-05-27T07:38:05.346 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:04 smithi132 ceph-mon[35283]: pgmap v811: 129 pgs: 129 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 4.7 MiB/s wr, 179 op/s 2024-05-27T07:38:05.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:04 smithi179 ceph-mon[35888]: pgmap v811: 129 pgs: 129 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 4.7 MiB/s wr, 179 op/s 2024-05-27T07:38:05.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 66305 5.85 MB/sec execute 486 sec latency 99.164 ms 2024-05-27T07:38:06.722 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:38:06.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:38:06.831 INFO:tasks.workunit.client.0.smithi043.stdout: 10 66393 5.85 MB/sec execute 487 sec latency 81.754 ms 2024-05-27T07:38:07.209 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:06 smithi043 ceph-mon[31531]: pgmap v812: 129 pgs: 129 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 5.4 MiB/s wr, 201 op/s 2024-05-27T07:38:07.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:06 smithi132 ceph-mon[35283]: pgmap v812: 129 pgs: 129 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 5.4 MiB/s wr, 201 op/s 2024-05-27T07:38:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:07.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:06 smithi179 ceph-mon[35888]: pgmap v812: 129 pgs: 129 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 5.4 MiB/s wr, 201 op/s 2024-05-27T07:38:07.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 66473 5.84 MB/sec execute 488 sec latency 389.578 ms 2024-05-27T07:38:08.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 66595 5.85 MB/sec execute 489 sec latency 429.758 ms 2024-05-27T07:38:09.208 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:08 smithi043 ceph-mon[31531]: pgmap v813: 129 pgs: 129 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.2 MiB/s wr, 187 op/s 2024-05-27T07:38:09.241 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:08 smithi132 ceph-mon[35283]: pgmap v813: 129 pgs: 129 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.2 MiB/s wr, 187 op/s 2024-05-27T07:38:09.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:08 smithi179 ceph-mon[35888]: pgmap v813: 129 pgs: 129 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.2 MiB/s wr, 187 op/s 2024-05-27T07:38:09.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 66710 5.86 MB/sec execute 490 sec latency 97.591 ms 2024-05-27T07:38:10.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 66805 5.85 MB/sec execute 491 sec latency 99.359 ms 2024-05-27T07:38:10.998 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:10 smithi043 ceph-mon[31531]: pgmap v814: 129 pgs: 129 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.2 MiB/s wr, 187 op/s 2024-05-27T07:38:11.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:11 smithi132 ceph-mon[35283]: pgmap v814: 129 pgs: 129 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.2 MiB/s wr, 187 op/s 2024-05-27T07:38:11.464 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:10 smithi179 ceph-mon[35888]: pgmap v814: 129 pgs: 129 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.2 MiB/s wr, 187 op/s 2024-05-27T07:38:11.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.584+0000 7f0dc4e00700 1 -- 172.21.15.43:0/1414882948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0dc0105490 msgr2=0x7f0dc01058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.584+0000 7f0dc4e00700 1 --2- 172.21.15.43:0/1414882948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0dc0105490 0x7f0dc01058a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0db0009990 tx=0x7f0db0009ca0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.584+0000 7f0dc4e00700 1 -- 172.21.15.43:0/1414882948 shutdown_connections 2024-05-27T07:38:11.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.584+0000 7f0dc4e00700 1 --2- 172.21.15.43:0/1414882948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc010e900 0x7f0dc0110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.584+0000 7f0dc4e00700 1 --2- 172.21.15.43:0/1414882948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0dc0105490 0x7f0dc01058a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.584+0000 7f0dc4e00700 1 --2- 172.21.15.43:0/1414882948 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dc0105de0 0x7f0dc010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.584+0000 7f0dc4e00700 1 -- 172.21.15.43:0/1414882948 >> 172.21.15.43:0/1414882948 conn(0x7f0dc00fedb0 msgr2=0x7f0dc01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:11.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.584+0000 7f0dc4e00700 1 -- 172.21.15.43:0/1414882948 shutdown_connections 2024-05-27T07:38:11.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.585+0000 7f0dc4e00700 1 -- 172.21.15.43:0/1414882948 wait complete. 2024-05-27T07:38:11.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.586+0000 7f0dc4e00700 1 Processor -- start 2024-05-27T07:38:11.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.587+0000 7f0dc4e00700 1 -- start start 2024-05-27T07:38:11.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.587+0000 7f0dc4e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc0105490 0x7f0dc019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.587+0000 7f0dc4e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0dc0105de0 0x7f0dc019d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.587+0000 7f0dc4e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dc010e900 0x7f0dc01a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.587+0000 7f0dc4e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc0114380 con 0x7f0dc0105490 2024-05-27T07:38:11.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.587+0000 7f0dc4e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc0114200 con 0x7f0dc0105de0 2024-05-27T07:38:11.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.587+0000 7f0dc4e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc0114500 con 0x7f0dc010e900 2024-05-27T07:38:11.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.588+0000 7f0dbde00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc0105490 0x7f0dc019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.588+0000 7f0dbde00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc0105490 0x7f0dc019cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:52714/0 (socket says 172.21.15.43:52714) 2024-05-27T07:38:11.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.588+0000 7f0dbde00700 1 -- 172.21.15.43:0/3130426889 learned_addr learned my addr 172.21.15.43:0/3130426889 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:11.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.588+0000 7f0dbd400700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0dc0105de0 0x7f0dc019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.588+0000 7f0dbe800700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dc010e900 0x7f0dc01a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.588+0000 7f0dbde00700 1 -- 172.21.15.43:0/3130426889 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dc010e900 msgr2=0x7f0dc01a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.588+0000 7f0dbde00700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dc010e900 0x7f0dc01a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.588+0000 7f0dbde00700 1 -- 172.21.15.43:0/3130426889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0dc0105de0 msgr2=0x7f0dc019d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.588+0000 7f0dbde00700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0dc0105de0 0x7f0dc019d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.588+0000 7f0dbde00700 1 -- 172.21.15.43:0/3130426889 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc01a1ea0 con 0x7f0dc0105490 2024-05-27T07:38:11.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.589+0000 7f0dbe800700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dc010e900 0x7f0dc01a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:11.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.589+0000 7f0dbd400700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0dc0105de0 0x7f0dc019d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:11.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.589+0000 7f0dbde00700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc0105490 0x7f0dc019cfd0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0db0006010 tx=0x7f0db000bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:11.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.590+0000 7f0db6a00700 1 -- 172.21.15.43:0/3130426889 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0027070 con 0x7f0dc0105490 2024-05-27T07:38:11.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.590+0000 7f0db6a00700 1 -- 172.21.15.43:0/3130426889 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0db000fb90 con 0x7f0dc0105490 2024-05-27T07:38:11.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.590+0000 7f0db6a00700 1 -- 172.21.15.43:0/3130426889 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db00173f0 con 0x7f0dc0105490 2024-05-27T07:38:11.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.590+0000 7f0dc4e00700 1 -- 172.21.15.43:0/3130426889 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc01a2130 con 0x7f0dc0105490 2024-05-27T07:38:11.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.590+0000 7f0dc4e00700 1 -- 172.21.15.43:0/3130426889 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc01a25c0 con 0x7f0dc0105490 2024-05-27T07:38:11.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.597+0000 7f0db6a00700 1 -- 172.21.15.43:0/3130426889 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0db002a070 con 0x7f0dc0105490 2024-05-27T07:38:11.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.600+0000 7f0db6a00700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0da40b2b40 0x7f0da40b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.600+0000 7f0db6a00700 1 -- 172.21.15.43:0/3130426889 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 9496+0+0 (secure 0 0 0) 0x7f0db00e4730 con 0x7f0dc0105490 2024-05-27T07:38:11.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.600+0000 7f0dbd400700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0da40b2b40 0x7f0da40b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.618+0000 7f0dbd400700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0da40b2b40 0x7f0da40b5050 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0104bb0 tx=0x7f0dac008040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.622+0000 7f0dd3858b80 1 Processor -- start 2024-05-27T07:38:11.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.622+0000 7f0dd3858b80 1 -- start start 2024-05-27T07:38:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d7a5db0 0x56539d79f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 0x56539d7a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56539d6cdec0 con 0x56539d7a5db0 2024-05-27T07:38:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56539d6bda60 con 0x56539d620ec0 2024-05-27T07:38:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56539d6c1aa0 con 0x56539d79fe30 2024-05-27T07:38:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dbde00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dbde00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40388/0 (socket says 172.21.15.43:40388) 2024-05-27T07:38:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dbde00700 1 -- 172.21.15.43:0/1249718366 learned_addr learned my addr 172.21.15.43:0/1249718366 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dbd400700 1 --2- 172.21.15.43:0/1249718366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d7a5db0 0x56539d79f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.623+0000 7f0dbe800700 1 --2- 172.21.15.43:0/1249718366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 0x56539d7a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.624+0000 7f0dbde00700 1 -- 172.21.15.43:0/1249718366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 msgr2=0x56539d7a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.624+0000 7f0dbde00700 1 --2- 172.21.15.43:0/1249718366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 0x56539d7a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.624+0000 7f0dbde00700 1 -- 172.21.15.43:0/1249718366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d7a5db0 msgr2=0x56539d79f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.624+0000 7f0dbde00700 1 --2- 172.21.15.43:0/1249718366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d7a5db0 0x56539d79f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.624+0000 7f0dbde00700 1 -- 172.21.15.43:0/1249718366 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56539d6864a0 con 0x56539d620ec0 2024-05-27T07:38:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.624+0000 7f0dbd400700 1 --2- 172.21.15.43:0/1249718366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d7a5db0 0x56539d79f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:38:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.625+0000 7f0dbde00700 1 --2- 172.21.15.43:0/1249718366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a5870 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0db000b2e0 tx=0x7f0db00ab570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9b4fadfcab3f06e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.625+0000 7f0dab400700 1 -- 172.21.15.43:0/1249718366 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db00afa20 con 0x56539d620ec0 2024-05-27T07:38:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.625+0000 7f0dab400700 1 -- 172.21.15.43:0/1249718366 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0db00b4440 con 0x56539d620ec0 2024-05-27T07:38:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.625+0000 7f0dab400700 1 -- 172.21.15.43:0/1249718366 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db00b4240 con 0x56539d620ec0 2024-05-27T07:38:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.627+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1249718366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 msgr2=0x56539d7a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.627+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/1249718366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a5870 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0db000b2e0 tx=0x7f0db00ab570 comp rx=0 tx=0).stop 2024-05-27T07:38:11.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.627+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1249718366 shutdown_connections 2024-05-27T07:38:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.627+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/1249718366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 0x56539d7a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.627+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/1249718366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a5870 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.627+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/1249718366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d7a5db0 0x56539d79f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.627+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1249718366 >> 172.21.15.43:0/1249718366 conn(0x56539d681040 msgr2=0x56539d79a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.627+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1249718366 shutdown_connections 2024-05-27T07:38:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.627+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1249718366 wait complete. 2024-05-27T07:38:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.628+0000 7f0dd3858b80 1 -- start start 2024-05-27T07:38:11.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.628+0000 7f0dd3858b80 1 Processor -- start 2024-05-27T07:38:11.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.629+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.629+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d79fe30 0x56539d7a9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.629+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 0x56539d7aa010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.629+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56539d5c4020 con 0x56539d79fe30 2024-05-27T07:38:11.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.629+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56539d65e7e0 con 0x56539d620ec0 2024-05-27T07:38:11.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.629+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56539d7ac640 con 0x56539d7a5db0 2024-05-27T07:38:11.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.629+0000 7f0dd3858b80 10 client.0 _ll_register_callbacks cb 0x56539d667cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:11.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.630+0000 7f0dbde00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.630+0000 7f0dbde00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40400/0 (socket says 172.21.15.43:40400) 2024-05-27T07:38:11.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.630+0000 7f0dbde00700 1 -- 172.21.15.43:0/82521866 learned_addr learned my addr 172.21.15.43:0/82521866 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:11.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.630+0000 7f0dbe800700 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 0x56539d7aa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.630+0000 7f0dbd400700 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d79fe30 0x56539d7a9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.631+0000 7f0dbde00700 1 -- 172.21.15.43:0/82521866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 msgr2=0x56539d7aa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:11.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.631+0000 7f0dbde00700 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 0x56539d7aa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.631+0000 7f0dbde00700 1 -- 172.21.15.43:0/82521866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d79fe30 msgr2=0x56539d7a9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:11.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.631+0000 7f0dbde00700 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d79fe30 0x56539d7a9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.631+0000 7f0dbde00700 1 -- 172.21.15.43:0/82521866 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56539d60d950 con 0x56539d620ec0 2024-05-27T07:38:11.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.631+0000 7f0dbd400700 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d79fe30 0x56539d7a9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:11.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.631+0000 7f0dbde00700 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a9590 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0db00ab320 tx=0x7f0db00ab7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad0bb2e301ee9d4f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:11.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.631+0000 7f0d9fe00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:38:11.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.631+0000 7f0d9fe00700 1 -- 172.21.15.43:0/82521866 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db00c5060 con 0x56539d620ec0 2024-05-27T07:38:11.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.632+0000 7f0d9fe00700 1 -- 172.21.15.43:0/82521866 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 9496+0+0 (secure 0 0 0) 0x7f0db00e44a0 con 0x56539d620ec0 2024-05-27T07:38:11.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.632+0000 7f0dd3858b80 1 -- 172.21.15.43:0/82521866 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0db00b4440 con 0x56539d620ec0 2024-05-27T07:38:11.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.632+0000 7f0d9fe00700 1 -- 172.21.15.43:0/82521866 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f0db00afa20 con 0x56539d620ec0 2024-05-27T07:38:11.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.632+0000 7f0dd3858b80 10 client.37853 fetch_fsmap learned FSMap version 147 2024-05-27T07:38:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.632+0000 7f0dd3858b80 1 -- 172.21.15.43:0/82521866 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x56539d68e0e0 con 0x56539d620ec0 2024-05-27T07:38:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.633+0000 7f0d9fe00700 1 -- 172.21.15.43:0/82521866 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0db00b4440 con 0x56539d620ec0 2024-05-27T07:38:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.633+0000 7f0d9fe00700 1 -- 172.21.15.43:0/82521866 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f0db00a6fe0 con 0x56539d620ec0 2024-05-27T07:38:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.633+0000 7f0dd3858b80 10 client.37853 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:38:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.633+0000 7f0dd3858b80 10 client.37853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.633+0000 7f0dd3858b80 20 client.37853 populate_metadata read hostname 'smithi043' 2024-05-27T07:38:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.633+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56539d7b9350 0x56539d7bb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.633+0000 7f0dd3858b80 4 client.37853 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:38:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.633+0000 7f0dd3858b80 1 -- 172.21.15.43:0/82521866 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56539d6cfe10 con 0x56539d7b9350 2024-05-27T07:38:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.634+0000 7f0dbd400700 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56539d7b9350 0x56539d7bb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.635+0000 7f0dbd400700 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56539d7b9350 0x56539d7bb770 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7433a2bd8335c2b7 server_cookie=94cf2df18868407e in_seq=0 out_seq=0 2024-05-27T07:38:11.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.635+0000 7f0d9fe00700 10 client.37853 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:11.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0d9fe00700 1 -- 172.21.15.43:0/82521866 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56539d6cfe10 con 0x56539d7b9350 2024-05-27T07:38:11.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0d9fe00700 10 client.37853 handle_command_reply: tid=0 2024-05-27T07:38:11.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0dd3858b80 1 client.37853 shutdown 2024-05-27T07:38:11.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0dd3858b80 1 -- 172.21.15.43:0/82521866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 msgr2=0x56539d7a9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a9590 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0db00ab320 tx=0x7f0db00ab7f0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0dd3858b80 1 -- 172.21.15.43:0/82521866 shutdown_connections 2024-05-27T07:38:11.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 0x56539d7aa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d620ec0 0x56539d7a9590 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56539d7b9350 0x56539d7bb770 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/82521866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d79fe30 0x56539d7a9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.636+0000 7f0dd3858b80 1 -- 172.21.15.43:0/82521866 >> 172.21.15.43:0/82521866 conn(0x56539d681040 msgr2=0x56539d7a0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:11.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.637+0000 7f0d9fe00700 0 client.37853 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:11.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.637+0000 7f0dd3858b80 1 -- 172.21.15.43:0/82521866 shutdown_connections 2024-05-27T07:38:11.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.637+0000 7f0dd3858b80 1 -- 172.21.15.43:0/82521866 wait complete. 2024-05-27T07:38:11.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.637+0000 7f0dd3858b80 20 client.37853 trim_cache size 0 max 16384 2024-05-27T07:38:11.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.675+0000 7f0dd3858b80 1 Processor -- start 2024-05-27T07:38:11.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.675+0000 7f0dd3858b80 1 -- start start 2024-05-27T07:38:11.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.676+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 0x56539d7a71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.676+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 0x56539d7b5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.676+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d7a5db0 0x56539d79a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.676+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc01b2be0 con 0x56539d620ec0 2024-05-27T07:38:11.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.676+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc01b10a0 con 0x56539d7a5db0 2024-05-27T07:38:11.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.676+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc01b2510 con 0x56539d79fe30 2024-05-27T07:38:11.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.677+0000 7f0dbe800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d7a5db0 0x56539d79a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.677+0000 7f0dbe800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d7a5db0 0x56539d79a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40404/0 (socket says 172.21.15.43:40404) 2024-05-27T07:38:11.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.677+0000 7f0dbe800700 1 -- 172.21.15.43:0/792504683 learned_addr learned my addr 172.21.15.43:0/792504683 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:11.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.677+0000 7f0dbde00700 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 0x56539d7a71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.678+0000 7f0dbd400700 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 0x56539d7b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.678+0000 7f0dbe800700 1 -- 172.21.15.43:0/792504683 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 msgr2=0x56539d7b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:11.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.678+0000 7f0dbe800700 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 0x56539d7b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.678+0000 7f0dbe800700 1 -- 172.21.15.43:0/792504683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 msgr2=0x56539d7a71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:11.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.678+0000 7f0dbe800700 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 0x56539d7a71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.678+0000 7f0dbe800700 1 -- 172.21.15.43:0/792504683 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56539d7a0ab0 con 0x56539d7a5db0 2024-05-27T07:38:11.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.679+0000 7f0dbde00700 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 0x56539d7a71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:38:11.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.679+0000 7f0dbd400700 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 0x56539d7b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:38:11.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.679+0000 7f0dbe800700 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d7a5db0 0x56539d79a600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0db8007630 tx=0x7f0db800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fefd6f6865488566 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:11.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.680+0000 7f0daaa00700 1 -- 172.21.15.43:0/792504683 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db80165e0 con 0x56539d7a5db0 2024-05-27T07:38:11.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.680+0000 7f0daaa00700 1 -- 172.21.15.43:0/792504683 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0db801c070 con 0x56539d7a5db0 2024-05-27T07:38:11.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.680+0000 7f0daaa00700 1 -- 172.21.15.43:0/792504683 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db8017b20 con 0x56539d7a5db0 2024-05-27T07:38:11.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.682+0000 7f0dd3858b80 1 -- 172.21.15.43:0/792504683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d7a5db0 msgr2=0x56539d79a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.682+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d7a5db0 0x56539d79a600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0db8007630 tx=0x7f0db800b040 comp rx=0 tx=0).stop 2024-05-27T07:38:11.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.682+0000 7f0dd3858b80 1 -- 172.21.15.43:0/792504683 shutdown_connections 2024-05-27T07:38:11.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.682+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d7a5db0 0x56539d79a600 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.682+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 0x56539d7a71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.682+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/792504683 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d79fe30 0x56539d7b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.682+0000 7f0dd3858b80 1 -- 172.21.15.43:0/792504683 >> 172.21.15.43:0/792504683 conn(0x56539d681040 msgr2=0x56539d7a9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:11.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.683+0000 7f0dd3858b80 1 -- 172.21.15.43:0/792504683 shutdown_connections 2024-05-27T07:38:11.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.683+0000 7f0dd3858b80 1 -- 172.21.15.43:0/792504683 wait complete. 2024-05-27T07:38:11.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.683+0000 7f0dd3858b80 1 -- start start 2024-05-27T07:38:11.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.684+0000 7f0dd3858b80 1 Processor -- start 2024-05-27T07:38:11.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.685+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 0x56539d7aead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.685+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d79fe30 0x56539d7af010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.686+0000 7f0dd3858b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 0x56539d82c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.686+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc01b1e40 con 0x56539d620ec0 2024-05-27T07:38:11.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.686+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc01b1770 con 0x56539d79fe30 2024-05-27T07:38:11.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.686+0000 7f0dd3858b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56539d7a9370 con 0x56539d7a5db0 2024-05-27T07:38:11.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.686+0000 7f0dd3858b80 10 client.0 _ll_register_callbacks cb 0x56539d667cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:11.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.686+0000 7f0dbd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d79fe30 0x56539d7af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.686+0000 7f0dbde00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 0x56539d7aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.687+0000 7f0dbd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d79fe30 0x56539d7af010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40418/0 (socket says 172.21.15.43:40418) 2024-05-27T07:38:11.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.687+0000 7f0dbd400700 1 -- 172.21.15.43:0/1157149601 learned_addr learned my addr 172.21.15.43:0/1157149601 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:11.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.687+0000 7f0dbe800700 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 0x56539d82c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.688+0000 7f0dbde00700 1 -- 172.21.15.43:0/1157149601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 msgr2=0x56539d82c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:11.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.688+0000 7f0dbde00700 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 0x56539d82c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.688+0000 7f0dbde00700 1 -- 172.21.15.43:0/1157149601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d79fe30 msgr2=0x56539d7af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:11.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.688+0000 7f0dbde00700 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d79fe30 0x56539d7af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.688+0000 7f0dbde00700 1 -- 172.21.15.43:0/1157149601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56539d7941b0 con 0x56539d620ec0 2024-05-27T07:38:11.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.688+0000 7f0dbe800700 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 0x56539d82c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:38:11.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.688+0000 7f0dbde00700 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 0x56539d7aead0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f0db00cccf0 tx=0x7f0db0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6267ee5ba36fef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:11.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.688+0000 7f0d9fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:38:11.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.689+0000 7f0d9fe00700 1 -- 172.21.15.43:0/1157149601 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db00c5070 con 0x56539d620ec0 2024-05-27T07:38:11.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.689+0000 7f0d9fe00700 1 -- 172.21.15.43:0/1157149601 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 9496+0+0 (secure 0 0 0) 0x7f0db00b0dc0 con 0x56539d620ec0 2024-05-27T07:38:11.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.689+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1157149601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0db801c070 con 0x56539d620ec0 2024-05-27T07:38:11.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.690+0000 7f0d9fe00700 1 -- 172.21.15.43:0/1157149601 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f0db00b8050 con 0x56539d620ec0 2024-05-27T07:38:11.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.690+0000 7f0dd3858b80 10 client.18468 fetch_fsmap learned FSMap version 147 2024-05-27T07:38:11.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.690+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1157149601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x56539d7941b0 con 0x56539d620ec0 2024-05-27T07:38:11.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.691+0000 7f0d9fe00700 1 -- 172.21.15.43:0/1157149601 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0db801c070 con 0x56539d620ec0 2024-05-27T07:38:11.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.691+0000 7f0d9fe00700 1 -- 172.21.15.43:0/1157149601 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f0db00bb080 con 0x56539d620ec0 2024-05-27T07:38:11.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.691+0000 7f0dd3858b80 10 client.18468 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:38:11.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.691+0000 7f0dd3858b80 10 client.18468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:11.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.691+0000 7f0dd3858b80 20 client.18468 populate_metadata read hostname 'smithi043' 2024-05-27T07:38:11.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.691+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56539d82dab0 0x56539d82fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:11.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.691+0000 7f0dd3858b80 4 client.18468 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:38:11.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.691+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1157149601 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56539d6c95f0 con 0x56539d82dab0 2024-05-27T07:38:11.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.692+0000 7f0dbd400700 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56539d82dab0 0x56539d82fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:11.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.693+0000 7f0d9fe00700 10 client.18468 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:11.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.693+0000 7f0dbd400700 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56539d82dab0 0x56539d82fed0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8096d948dccd37e6 server_cookie=41127c49c4f2d317 in_seq=0 out_seq=0 2024-05-27T07:38:11.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0d9fe00700 1 -- 172.21.15.43:0/1157149601 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56539d6c95f0 con 0x56539d82dab0 2024-05-27T07:38:11.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0d9fe00700 10 client.18468 handle_command_reply: tid=0 2024-05-27T07:38:11.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0dd3858b80 1 client.18468 shutdown 2024-05-27T07:38:11.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1157149601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 msgr2=0x56539d7aead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 0x56539d7aead0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f0db00cccf0 tx=0x7f0db0009310 comp rx=0 tx=0).stop 2024-05-27T07:38:11.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1157149601 shutdown_connections 2024-05-27T07:38:11.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56539d7a5db0 0x56539d82c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56539d82dab0 0x56539d82fed0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56539d620ec0 0x56539d7aead0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0dd3858b80 1 --2- 172.21.15.43:0/1157149601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56539d79fe30 0x56539d7af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1157149601 >> 172.21.15.43:0/1157149601 conn(0x56539d681040 msgr2=0x56539d79a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:11.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.697+0000 7f0d9fe00700 0 client.18468 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:11.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.698+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1157149601 shutdown_connections 2024-05-27T07:38:11.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.698+0000 7f0dd3858b80 1 -- 172.21.15.43:0/1157149601 wait complete. 2024-05-27T07:38:11.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.698+0000 7f0dd3858b80 20 client.18468 trim_cache size 0 max 16384 2024-05-27T07:38:11.695 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:38:11.696 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:38:11.696 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "4b36deb4-ce3b-4a0c-88f4-e7b5489066c2", 2024-05-27T07:38:11.696 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:38:11.696 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:38:11.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.699+0000 7f0dc4e00700 1 -- 172.21.15.43:0/3130426889 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0da40b2b40 msgr2=0x7f0da40b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.699+0000 7f0dc4e00700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0da40b2b40 0x7f0da40b5050 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0104bb0 tx=0x7f0dac008040 comp rx=0 tx=0).stop 2024-05-27T07:38:11.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.700+0000 7f0dc4e00700 1 -- 172.21.15.43:0/3130426889 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc0105490 msgr2=0x7f0dc019cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:11.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.700+0000 7f0dc4e00700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc0105490 0x7f0dc019cfd0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0db0006010 tx=0x7f0db000bfa0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.700+0000 7f0dc4e00700 1 -- 172.21.15.43:0/3130426889 shutdown_connections 2024-05-27T07:38:11.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.700+0000 7f0dc4e00700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f0da40b2b40 0x7f0da40b5050 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.700+0000 7f0dc4e00700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dc010e900 0x7f0dc01a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.700+0000 7f0dc4e00700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0dc0105490 0x7f0dc019cfd0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.700+0000 7f0dc4e00700 1 --2- 172.21.15.43:0/3130426889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0dc0105de0 0x7f0dc019d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:11.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.700+0000 7f0dc4e00700 1 -- 172.21.15.43:0/3130426889 >> 172.21.15.43:0/3130426889 conn(0x7f0dc00fedb0 msgr2=0x7f0dc0103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:11.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.701+0000 7f0dc4e00700 1 -- 172.21.15.43:0/3130426889 shutdown_connections 2024-05-27T07:38:11.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:11.701+0000 7f0dc4e00700 1 -- 172.21.15.43:0/3130426889 wait complete. 2024-05-27T07:38:11.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 66884 5.84 MB/sec execute 492 sec latency 112.617 ms 2024-05-27T07:38:12.686 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: osdmap e159: 12 total, 12 up, 12 in 2024-05-27T07:38:12.686 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:38:12.686 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.687 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:38:12.687 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:38:12.687 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:38:12.687 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:38:12.687 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:38:12.687 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.688 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.688 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:38:12.688 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.688 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.688 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:38:12.688 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.689 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.689 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:12 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:38:12.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 66984 5.84 MB/sec execute 493 sec latency 118.930 ms 2024-05-27T07:38:12.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: osdmap e159: 12 total, 12 up, 12 in 2024-05-27T07:38:12.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:38:12.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:38:12.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:38:12.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:38:12.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:38:12.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:38:12.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:38:12.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:38:12.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:12 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: osdmap e159: 12 total, 12 up, 12 in 2024-05-27T07:38:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:38:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:38:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:38:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:38:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:38:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:38:12.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:38:12.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:38:12.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:12.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:12 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:38:13.528 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:13.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 67078 5.83 MB/sec execute 494 sec latency 109.906 ms 2024-05-27T07:38:13.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:13 smithi132 ceph-mon[35283]: pgmap v816: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 4.9 MiB/s wr, 178 op/s 2024-05-27T07:38:13.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:13 smithi043 ceph-mon[31531]: pgmap v816: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 4.9 MiB/s wr, 178 op/s 2024-05-27T07:38:13.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:13 smithi179 ceph-mon[35888]: pgmap v816: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 4.9 MiB/s wr, 178 op/s 2024-05-27T07:38:14.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 67185 5.82 MB/sec execute 495 sec latency 105.651 ms 2024-05-27T07:38:15.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 67295 5.82 MB/sec execute 496 sec latency 99.835 ms 2024-05-27T07:38:15.846 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:15 smithi132 ceph-mon[35283]: pgmap v817: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 4.8 MiB/s wr, 176 op/s 2024-05-27T07:38:15.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:15 smithi043 ceph-mon[31531]: pgmap v817: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 4.8 MiB/s wr, 176 op/s 2024-05-27T07:38:15.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:15 smithi179 ceph-mon[35888]: pgmap v817: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 4.8 MiB/s wr, 176 op/s 2024-05-27T07:38:16.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 67410 5.82 MB/sec execute 497 sec latency 111.915 ms 2024-05-27T07:38:17.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 67506 5.82 MB/sec execute 498 sec latency 222.059 ms 2024-05-27T07:38:17.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:17 smithi132 ceph-mon[35283]: pgmap v818: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 4.5 MiB/s wr, 173 op/s 2024-05-27T07:38:17.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:17 smithi043 ceph-mon[31531]: pgmap v818: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 4.5 MiB/s wr, 173 op/s 2024-05-27T07:38:17.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:17 smithi179 ceph-mon[35888]: pgmap v818: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 4.5 MiB/s wr, 173 op/s 2024-05-27T07:38:18.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 67609 5.82 MB/sec execute 499 sec latency 110.212 ms 2024-05-27T07:38:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.429+0000 7f5b7a800700 1 -- 172.21.15.43:0/98573774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b7410c5a0 msgr2=0x7f5b7410c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.429+0000 7f5b7a800700 1 --2- 172.21.15.43:0/98573774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b7410c5a0 0x7f5b7410c9b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009990 tx=0x7f5b64009ca0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.429+0000 7f5b7a800700 1 -- 172.21.15.43:0/98573774 shutdown_connections 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.429+0000 7f5b7a800700 1 --2- 172.21.15.43:0/98573774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b74106eb0 0x7f5b74113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.429+0000 7f5b7a800700 1 --2- 172.21.15.43:0/98573774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b7410c5a0 0x7f5b7410c9b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.429+0000 7f5b7a800700 1 --2- 172.21.15.43:0/98573774 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b74106560 0x7f5b74106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.429+0000 7f5b7a800700 1 -- 172.21.15.43:0/98573774 >> 172.21.15.43:0/98573774 conn(0x7f5b74102040 msgr2=0x7f5b74104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:19.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.431+0000 7f5b7a800700 1 -- 172.21.15.43:0/98573774 shutdown_connections 2024-05-27T07:38:19.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.431+0000 7f5b7a800700 1 -- 172.21.15.43:0/98573774 wait complete. 2024-05-27T07:38:19.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.432+0000 7f5b7a800700 1 Processor -- start 2024-05-27T07:38:19.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.433+0000 7f5b7a800700 1 -- start start 2024-05-27T07:38:19.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.433+0000 7f5b7a800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b74106560 0x7f5b741126f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.433+0000 7f5b7a800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b74106eb0 0x7f5b74112c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.433+0000 7f5b7a800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b7410c5a0 0x7f5b741a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.433+0000 7f5b7a800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5b74079180 con 0x7f5b74106560 2024-05-27T07:38:19.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.433+0000 7f5b7a800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5b74079000 con 0x7f5b7410c5a0 2024-05-27T07:38:19.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.433+0000 7f5b7a800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5b74079300 con 0x7f5b74106eb0 2024-05-27T07:38:19.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.434+0000 7f5b72a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b74106eb0 0x7f5b74112c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.434+0000 7f5b73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b74106560 0x7f5b741126f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.434+0000 7f5b73400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b74106560 0x7f5b741126f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46458/0 (socket says 172.21.15.43:46458) 2024-05-27T07:38:19.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.434+0000 7f5b73400700 1 -- 172.21.15.43:0/945711430 learned_addr learned my addr 172.21.15.43:0/945711430 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:19.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.434+0000 7f5b73e00700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b7410c5a0 0x7f5b741a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.435+0000 7f5b73e00700 1 -- 172.21.15.43:0/945711430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b74106eb0 msgr2=0x7f5b74112c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.435+0000 7f5b73e00700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b74106eb0 0x7f5b74112c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.435+0000 7f5b73e00700 1 -- 172.21.15.43:0/945711430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b74106560 msgr2=0x7f5b741126f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.435+0000 7f5b73e00700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b74106560 0x7f5b741126f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.435+0000 7f5b73e00700 1 -- 172.21.15.43:0/945711430 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b741a6210 con 0x7f5b7410c5a0 2024-05-27T07:38:19.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.436+0000 7f5b73e00700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b7410c5a0 0x7f5b741a5b70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c007f10 tx=0x7f5b6c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:19.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.436+0000 7f5b6be00700 1 -- 172.21.15.43:0/945711430 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c00ddd0 con 0x7f5b7410c5a0 2024-05-27T07:38:19.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.437+0000 7f5b6be00700 1 -- 172.21.15.43:0/945711430 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5b6c004bb0 con 0x7f5b7410c5a0 2024-05-27T07:38:19.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.437+0000 7f5b6be00700 1 -- 172.21.15.43:0/945711430 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c015400 con 0x7f5b7410c5a0 2024-05-27T07:38:19.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.437+0000 7f5b7a800700 1 -- 172.21.15.43:0/945711430 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b741a6500 con 0x7f5b7410c5a0 2024-05-27T07:38:19.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.437+0000 7f5b7a800700 1 -- 172.21.15.43:0/945711430 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b741ade20 con 0x7f5b7410c5a0 2024-05-27T07:38:19.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.442+0000 7f5b6be00700 1 -- 172.21.15.43:0/945711430 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f5b6c005020 con 0x7f5b7410c5a0 2024-05-27T07:38:19.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.444+0000 7f5b6be00700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5b580b2a70 0x7f5b580b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.445+0000 7f5b6be00700 1 -- 172.21.15.43:0/945711430 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 9496+0+0 (secure 0 0 0) 0x7f5b6c0e24f0 con 0x7f5b7410c5a0 2024-05-27T07:38:19.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.445+0000 7f5b73400700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5b580b2a70 0x7f5b580b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.447+0000 7f5b73400700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5b580b2a70 0x7f5b580b4f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009750 tx=0x7f5b6400b260 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:19.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.453+0000 7f5b89295b80 1 Processor -- start 2024-05-27T07:38:19.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.453+0000 7f5b89295b80 1 -- start start 2024-05-27T07:38:19.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.454+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x5603151367f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.454+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315136d30 0x560315130870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.454+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x5603151311c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.454+0000 7f5b89295b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56031505ee40 con 0x560315136d30 2024-05-27T07:38:19.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.454+0000 7f5b89295b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56031504e9e0 con 0x560315130db0 2024-05-27T07:38:19.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.454+0000 7f5b89295b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560315052a20 con 0x560314fb1e40 2024-05-27T07:38:19.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.455+0000 7f5b73400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x5603151367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.455+0000 7f5b73400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x5603151367f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:59664/0 (socket says 172.21.15.43:59664) 2024-05-27T07:38:19.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.455+0000 7f5b73400700 1 -- 172.21.15.43:0/2000947242 learned_addr learned my addr 172.21.15.43:0/2000947242 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:19.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.455+0000 7f5b73400700 1 -- 172.21.15.43:0/2000947242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 msgr2=0x5603151311c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:19.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.455+0000 7f5b73400700 1 --2- 172.21.15.43:0/2000947242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x5603151311c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.455+0000 7f5b73400700 1 -- 172.21.15.43:0/2000947242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315136d30 msgr2=0x560315130870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:19.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.456+0000 7f5b72a00700 1 --2- 172.21.15.43:0/2000947242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315136d30 0x560315130870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.456+0000 7f5b73400700 1 --2- 172.21.15.43:0/2000947242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315136d30 0x560315130870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.456+0000 7f5b73400700 1 -- 172.21.15.43:0/2000947242 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560315017420 con 0x560314fb1e40 2024-05-27T07:38:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.457+0000 7f5b73400700 1 --2- 172.21.15.43:0/2000947242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x5603151367f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5b640059d0 tx=0x7f5b6401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2961d0b00411109f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.457+0000 7f5b68c00700 1 -- 172.21.15.43:0/2000947242 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6400b6b0 con 0x560314fb1e40 2024-05-27T07:38:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.457+0000 7f5b68c00700 1 -- 172.21.15.43:0/2000947242 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5b6400bce0 con 0x560314fb1e40 2024-05-27T07:38:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.457+0000 7f5b68c00700 1 -- 172.21.15.43:0/2000947242 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6402fb90 con 0x560314fb1e40 2024-05-27T07:38:19.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.459+0000 7f5b89295b80 1 -- 172.21.15.43:0/2000947242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 msgr2=0x5603151367f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.459+0000 7f5b89295b80 1 --2- 172.21.15.43:0/2000947242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x5603151367f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5b640059d0 tx=0x7f5b6401d000 comp rx=0 tx=0).stop 2024-05-27T07:38:19.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.459+0000 7f5b89295b80 1 -- 172.21.15.43:0/2000947242 shutdown_connections 2024-05-27T07:38:19.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.459+0000 7f5b89295b80 1 --2- 172.21.15.43:0/2000947242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x5603151311c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.459+0000 7f5b89295b80 1 --2- 172.21.15.43:0/2000947242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x5603151367f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.459+0000 7f5b89295b80 1 --2- 172.21.15.43:0/2000947242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315136d30 0x560315130870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.459+0000 7f5b89295b80 1 -- 172.21.15.43:0/2000947242 >> 172.21.15.43:0/2000947242 conn(0x560315011fc0 msgr2=0x56031512b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:19.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.459+0000 7f5b89295b80 1 -- 172.21.15.43:0/2000947242 shutdown_connections 2024-05-27T07:38:19.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.459+0000 7f5b89295b80 1 -- 172.21.15.43:0/2000947242 wait complete. 2024-05-27T07:38:19.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.460+0000 7f5b89295b80 1 -- start start 2024-05-27T07:38:19.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.460+0000 7f5b89295b80 1 Processor -- start 2024-05-27T07:38:19.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.461+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x56031513a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.461+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315130db0 0x56031513aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.461+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315136d30 0x56031513aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.461+0000 7f5b89295b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560314f54fa0 con 0x560315130db0 2024-05-27T07:38:19.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.461+0000 7f5b89295b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560314fef760 con 0x560315136d30 2024-05-27T07:38:19.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.461+0000 7f5b89295b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56031513d7a0 con 0x560314fb1e40 2024-05-27T07:38:19.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.461+0000 7f5b89295b80 10 client.0 _ll_register_callbacks cb 0x560314ff8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:19.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.462+0000 7f5b73e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315136d30 0x56031513aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.462+0000 7f5b73e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315136d30 0x56031513aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:45388/0 (socket says 172.21.15.43:45388) 2024-05-27T07:38:19.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.462+0000 7f5b73e00700 1 -- 172.21.15.43:0/4097825327 learned_addr learned my addr 172.21.15.43:0/4097825327 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:19.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.462+0000 7f5b73400700 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x56031513a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.462+0000 7f5b72a00700 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315130db0 0x56031513aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.462+0000 7f5b73e00700 1 -- 172.21.15.43:0/4097825327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 msgr2=0x56031513a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:19.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.462+0000 7f5b73e00700 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x56031513a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.462+0000 7f5b73e00700 1 -- 172.21.15.43:0/4097825327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315130db0 msgr2=0x56031513aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:19.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.463+0000 7f5b73e00700 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315130db0 0x56031513aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.463+0000 7f5b73e00700 1 -- 172.21.15.43:0/4097825327 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560314f9e8d0 con 0x560315136d30 2024-05-27T07:38:19.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.463+0000 7f5b72a00700 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315130db0 0x56031513aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:38:19.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.463+0000 7f5b73e00700 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315136d30 0x56031513aff0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0a5390 tx=0x7f5b6c0a8fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=803094a7ea1efdf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:19.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.463+0000 7f5b5d600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:38:19.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.463+0000 7f5b5d600700 1 -- 172.21.15.43:0/4097825327 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c0ad750 con 0x560315136d30 2024-05-27T07:38:19.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.463+0000 7f5b5d600700 1 -- 172.21.15.43:0/4097825327 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 9496+0+0 (secure 0 0 0) 0x7f5b6c009440 con 0x560315136d30 2024-05-27T07:38:19.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.464+0000 7f5b89295b80 1 -- 172.21.15.43:0/4097825327 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b6400bce0 con 0x560315136d30 2024-05-27T07:38:19.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.464+0000 7f5b5d600700 1 -- 172.21.15.43:0/4097825327 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f5b6c0b2200 con 0x560315136d30 2024-05-27T07:38:19.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.464+0000 7f5b89295b80 10 client.37868 fetch_fsmap learned FSMap version 147 2024-05-27T07:38:19.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.464+0000 7f5b89295b80 1 -- 172.21.15.43:0/4097825327 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x56031501f060 con 0x560315136d30 2024-05-27T07:38:19.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.465+0000 7f5b5d600700 1 -- 172.21.15.43:0/4097825327 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5b6c005310 con 0x560315136d30 2024-05-27T07:38:19.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.465+0000 7f5b5d600700 1 -- 172.21.15.43:0/4097825327 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f5b6c0a72f0 con 0x560315136d30 2024-05-27T07:38:19.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.467+0000 7f5b89295b80 10 client.37868 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:38:19.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.467+0000 7f5b89295b80 10 client.37868 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:19.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.467+0000 7f5b89295b80 20 client.37868 populate_metadata read hostname 'smithi043' 2024-05-27T07:38:19.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.467+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56031514aba0 0x56031514cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.467+0000 7f5b89295b80 4 client.37868 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:38:19.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.467+0000 7f5b89295b80 1 -- 172.21.15.43:0/4097825327 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5b6400bce0 con 0x56031514aba0 2024-05-27T07:38:19.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.467+0000 7f5b73400700 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56031514aba0 0x56031514cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.468+0000 7f5b73400700 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56031514aba0 0x56031514cff0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9ec90404c3686a4 server_cookie=af51d3800ce46406 in_seq=0 out_seq=0 2024-05-27T07:38:19.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.468+0000 7f5b5d600700 10 client.37868 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:19.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.469+0000 7f5b5d600700 1 -- 172.21.15.43:0/4097825327 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f5b6400bce0 con 0x56031514aba0 2024-05-27T07:38:19.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.469+0000 7f5b5d600700 10 client.37868 handle_command_reply: tid=0 2024-05-27T07:38:19.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.470+0000 7f5b89295b80 1 client.37868 shutdown 2024-05-27T07:38:19.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.470+0000 7f5b89295b80 1 -- 172.21.15.43:0/4097825327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315136d30 msgr2=0x56031513aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.470+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315136d30 0x56031513aff0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0a5390 tx=0x7f5b6c0a8fa0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.471+0000 7f5b89295b80 1 -- 172.21.15.43:0/4097825327 shutdown_connections 2024-05-27T07:38:19.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.471+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315136d30 0x56031513aff0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.471+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x56031513a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.471+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x56031514aba0 0x56031514cff0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.471+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4097825327 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315130db0 0x56031513aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.471+0000 7f5b89295b80 1 -- 172.21.15.43:0/4097825327 >> 172.21.15.43:0/4097825327 conn(0x560315011fc0 msgr2=0x56031512b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:19.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.471+0000 7f5b5d600700 0 client.37868 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:19.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.472+0000 7f5b89295b80 1 -- 172.21.15.43:0/4097825327 shutdown_connections 2024-05-27T07:38:19.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.472+0000 7f5b89295b80 1 -- 172.21.15.43:0/4097825327 wait complete. 2024-05-27T07:38:19.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.472+0000 7f5b89295b80 20 client.37868 trim_cache size 0 max 16384 2024-05-27T07:38:19.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.510+0000 7f5b89295b80 1 Processor -- start 2024-05-27T07:38:19.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.510+0000 7f5b89295b80 1 -- start start 2024-05-27T07:38:19.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.510+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560314fb1e40 0x560315130870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.510+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x560315134c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.510+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560315136d30 0x560315135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.510+0000 7f5b89295b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5b741b70a0 con 0x560314fb1e40 2024-05-27T07:38:19.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.510+0000 7f5b89295b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5b741b5560 con 0x560315130db0 2024-05-27T07:38:19.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.510+0000 7f5b89295b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5b741b69d0 con 0x560315136d30 2024-05-27T07:38:19.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.511+0000 7f5b72a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x560315134c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.511+0000 7f5b72a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x560315134c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:45404/0 (socket says 172.21.15.43:45404) 2024-05-27T07:38:19.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.511+0000 7f5b72a00700 1 -- 172.21.15.43:0/869017158 learned_addr learned my addr 172.21.15.43:0/869017158 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:19.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.511+0000 7f5b73e00700 1 --2- 172.21.15.43:0/869017158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560315136d30 0x560315135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.511+0000 7f5b73400700 1 --2- 172.21.15.43:0/869017158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560314fb1e40 0x560315130870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.511+0000 7f5b72a00700 1 -- 172.21.15.43:0/869017158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560315136d30 msgr2=0x560315135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:19.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.511+0000 7f5b72a00700 1 --2- 172.21.15.43:0/869017158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560315136d30 0x560315135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.511+0000 7f5b72a00700 1 -- 172.21.15.43:0/869017158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560314fb1e40 msgr2=0x560315130870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:19.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.512+0000 7f5b72a00700 1 --2- 172.21.15.43:0/869017158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560314fb1e40 0x560315130870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.512+0000 7f5b72a00700 1 -- 172.21.15.43:0/869017158 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603151398a0 con 0x560315130db0 2024-05-27T07:38:19.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.512+0000 7f5b73400700 1 --2- 172.21.15.43:0/869017158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560314fb1e40 0x560315130870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:38:19.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.512+0000 7f5b72a00700 1 --2- 172.21.15.43:0/869017158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x560315134c40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009fb0 tx=0x7f5b60012730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dbe79c811633acf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:19.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.512+0000 7f5b5fe00700 1 -- 172.21.15.43:0/869017158 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b60004020 con 0x560315130db0 2024-05-27T07:38:19.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.513+0000 7f5b5fe00700 1 -- 172.21.15.43:0/869017158 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5b60003680 con 0x560315130db0 2024-05-27T07:38:19.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.513+0000 7f5b5fe00700 1 -- 172.21.15.43:0/869017158 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6000f680 con 0x560315130db0 2024-05-27T07:38:19.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.514+0000 7f5b89295b80 1 -- 172.21.15.43:0/869017158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 msgr2=0x560315134c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.514+0000 7f5b89295b80 1 --2- 172.21.15.43:0/869017158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x560315134c40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009fb0 tx=0x7f5b60012730 comp rx=0 tx=0).stop 2024-05-27T07:38:19.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.514+0000 7f5b89295b80 1 -- 172.21.15.43:0/869017158 shutdown_connections 2024-05-27T07:38:19.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.514+0000 7f5b89295b80 1 --2- 172.21.15.43:0/869017158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560315136d30 0x560315135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.514+0000 7f5b89295b80 1 --2- 172.21.15.43:0/869017158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560314fb1e40 0x560315130870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.514+0000 7f5b89295b80 1 --2- 172.21.15.43:0/869017158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x560315134c40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.514+0000 7f5b89295b80 1 -- 172.21.15.43:0/869017158 >> 172.21.15.43:0/869017158 conn(0x560315011fc0 msgr2=0x56031513a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:19.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.514+0000 7f5b89295b80 1 -- 172.21.15.43:0/869017158 shutdown_connections 2024-05-27T07:38:19.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.516+0000 7f5b89295b80 1 -- 172.21.15.43:0/869017158 wait complete. 2024-05-27T07:38:19.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.516+0000 7f5b89295b80 1 -- start start 2024-05-27T07:38:19.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.517+0000 7f5b89295b80 1 Processor -- start 2024-05-27T07:38:19.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.517+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x56031512f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.517+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x56031512fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.517+0000 7f5b89295b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315136d30 0x5603151303e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.517+0000 7f5b89295b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5b741b6300 con 0x560315136d30 2024-05-27T07:38:19.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.517+0000 7f5b89295b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5b741b5c30 con 0x560315130db0 2024-05-27T07:38:19.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.517+0000 7f5b89295b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56031513b630 con 0x560314fb1e40 2024-05-27T07:38:19.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.517+0000 7f5b89295b80 10 client.0 _ll_register_callbacks cb 0x560314ff8c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:38:19.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.518+0000 7f5b73400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x56031512f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.518+0000 7f5b73e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315136d30 0x5603151303e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.518+0000 7f5b72a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x56031512fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.519+0000 7f5b72a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x56031512fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:45408/0 (socket says 172.21.15.43:45408) 2024-05-27T07:38:19.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.519+0000 7f5b72a00700 1 -- 172.21.15.43:0/4049242537 learned_addr learned my addr 172.21.15.43:0/4049242537 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:38:19.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.519+0000 7f5b72a00700 1 -- 172.21.15.43:0/4049242537 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 msgr2=0x56031512f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:19.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.519+0000 7f5b72a00700 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x56031512f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.519+0000 7f5b72a00700 1 -- 172.21.15.43:0/4049242537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315136d30 msgr2=0x5603151303e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:38:19.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.519+0000 7f5b72a00700 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315136d30 0x5603151303e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.519+0000 7f5b72a00700 1 -- 172.21.15.43:0/4049242537 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603151435f0 con 0x560315130db0 2024-05-27T07:38:19.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.520+0000 7f5b72a00700 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x56031512fea0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5b60012a80 tx=0x7f5b600042d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dd8fc8de9f77461 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:38:19.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.520+0000 7f5b5d600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:38:19.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.521+0000 7f5b5d600700 1 -- 172.21.15.43:0/4049242537 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b60029020 con 0x560315130db0 2024-05-27T07:38:19.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.521+0000 7f5b5d600700 1 -- 172.21.15.43:0/4049242537 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 9496+0+0 (secure 0 0 0) 0x7f5b60024650 con 0x560315130db0 2024-05-27T07:38:19.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.521+0000 7f5b89295b80 1 -- 172.21.15.43:0/4049242537 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b60003680 con 0x560315130db0 2024-05-27T07:38:19.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.522+0000 7f5b5d600700 1 -- 172.21.15.43:0/4049242537 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f5b60024850 con 0x560315130db0 2024-05-27T07:38:19.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.522+0000 7f5b89295b80 10 client.37874 fetch_fsmap learned FSMap version 147 2024-05-27T07:38:19.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.522+0000 7f5b89295b80 1 -- 172.21.15.43:0/4049242537 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x5603151b5680 con 0x560315130db0 2024-05-27T07:38:19.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.522+0000 7f5b5d600700 1 -- 172.21.15.43:0/4049242537 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5b60003680 con 0x560315130db0 2024-05-27T07:38:19.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.523+0000 7f5b5d600700 1 -- 172.21.15.43:0/4049242537 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f5b6000f790 con 0x560315130db0 2024-05-27T07:38:19.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.523+0000 7f5b89295b80 10 client.37874 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:38:19.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.523+0000 7f5b89295b80 10 client.37874 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:38:19.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.523+0000 7f5b89295b80 20 client.37874 populate_metadata read hostname 'smithi043' 2024-05-27T07:38:19.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.523+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5603151b5d50 0x5603151b8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:38:19.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.523+0000 7f5b89295b80 4 client.37874 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:38:19.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.523+0000 7f5b89295b80 1 -- 172.21.15.43:0/4049242537 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560315060d90 con 0x5603151b5d50 2024-05-27T07:38:19.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.523+0000 7f5b73400700 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5603151b5d50 0x5603151b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:38:19.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.525+0000 7f5b73400700 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5603151b5d50 0x5603151b8170 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4611c11ed6f5bcc9 server_cookie=af6c560da55c8988 in_seq=0 out_seq=0 2024-05-27T07:38:19.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.525+0000 7f5b5d600700 10 client.37874 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:19.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.525+0000 7f5b5d600700 1 -- 172.21.15.43:0/4049242537 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560315060d90 con 0x5603151b5d50 2024-05-27T07:38:19.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.525+0000 7f5b5d600700 10 client.37874 handle_command_reply: tid=0 2024-05-27T07:38:19.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.525+0000 7f5b89295b80 1 client.37874 shutdown 2024-05-27T07:38:19.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 1 -- 172.21.15.43:0/4049242537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 msgr2=0x56031512fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x56031512fea0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5b60012a80 tx=0x7f5b600042d0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 1 -- 172.21.15.43:0/4049242537 shutdown_connections 2024-05-27T07:38:19.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5603151b5d50 0x5603151b8170 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560315136d30 0x5603151303e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560314fb1e40 0x56031512f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 1 --2- 172.21.15.43:0/4049242537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560315130db0 0x56031512fea0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 1 -- 172.21.15.43:0/4049242537 >> 172.21.15.43:0/4049242537 conn(0x560315011fc0 msgr2=0x5603151349a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:19.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b5d600700 0 client.37874 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:38:19.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 1 -- 172.21.15.43:0/4049242537 shutdown_connections 2024-05-27T07:38:19.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 1 -- 172.21.15.43:0/4049242537 wait complete. 2024-05-27T07:38:19.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.526+0000 7f5b89295b80 20 client.37874 trim_cache size 0 max 16384 2024-05-27T07:38:19.524 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:38:19.524 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:38:19.524 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:38:19.524 INFO:teuthology.orchestra.run.smithi043.stdout: "4b36deb4-ce3b-4a0c-88f4-e7b5489066c2": { 2024-05-27T07:38:19.525 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:38:19.525 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "4b36deb4-ce3b-4a0c-88f4-e7b5489066c2", 2024-05-27T07:38:19.525 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:38:19.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:38:19.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:38:19.525 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:38:19.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 -- 172.21.15.43:0/945711430 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5b580b2a70 msgr2=0x7f5b580b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5b580b2a70 0x7f5b580b4f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009750 tx=0x7f5b6400b260 comp rx=0 tx=0).stop 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 -- 172.21.15.43:0/945711430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b7410c5a0 msgr2=0x7f5b741a5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b7410c5a0 0x7f5b741a5b70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c007f10 tx=0x7f5b6c00d3b0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 -- 172.21.15.43:0/945711430 shutdown_connections 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f5b580b2a70 0x7f5b580b4f80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b7410c5a0 0x7f5b741a5b70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b74106560 0x7f5b741126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 --2- 172.21.15.43:0/945711430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b74106eb0 0x7f5b74112c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 -- 172.21.15.43:0/945711430 >> 172.21.15.43:0/945711430 conn(0x7f5b74102040 msgr2=0x7f5b74108060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 -- 172.21.15.43:0/945711430 shutdown_connections 2024-05-27T07:38:19.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:38:19.528+0000 7f5b7a800700 1 -- 172.21.15.43:0/945711430 wait complete. 2024-05-27T07:38:19.708 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:19 smithi043 ceph-mon[31531]: pgmap v819: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 4.6 MiB/s wr, 184 op/s 2024-05-27T07:38:19.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 67670 5.81 MB/sec execute 500 sec latency 385.574 ms 2024-05-27T07:38:19.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:19 smithi179 ceph-mon[35888]: pgmap v819: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 4.6 MiB/s wr, 184 op/s 2024-05-27T07:38:20.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:19 smithi132 ceph-mon[35283]: pgmap v819: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 4.6 MiB/s wr, 184 op/s 2024-05-27T07:38:20.652 INFO:tasks.cephfs.filesystem:scrub status for tag:4b36deb4-ce3b-4a0c-88f4-e7b5489066c2 - {'path': '/', 'tag': '4b36deb4-ce3b-4a0c-88f4-e7b5489066c2', 'options': 'recursive,force'} 2024-05-27T07:38:20.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 67778 5.80 MB/sec execute 501 sec latency 91.282 ms 2024-05-27T07:38:21.832 INFO:tasks.workunit.client.0.smithi043.stdout: 10 67914 5.81 MB/sec execute 502 sec latency 83.600 ms 2024-05-27T07:38:21.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:21 smithi043 ceph-mon[31531]: pgmap v820: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 4.6 MiB/s wr, 184 op/s 2024-05-27T07:38:21.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:21 smithi179 ceph-mon[35888]: pgmap v820: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 4.6 MiB/s wr, 184 op/s 2024-05-27T07:38:22.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:21 smithi132 ceph-mon[35283]: pgmap v820: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 4.6 MiB/s wr, 184 op/s 2024-05-27T07:38:22.833 INFO:tasks.workunit.client.0.smithi043.stdout: 10 68037 5.80 MB/sec execute 503 sec latency 88.093 ms 2024-05-27T07:38:23.833 INFO:tasks.workunit.client.0.smithi043.stdout: 10 68196 5.82 MB/sec execute 504 sec latency 83.234 ms 2024-05-27T07:38:23.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:23 smithi043 ceph-mon[31531]: pgmap v821: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.7 MiB/s wr, 193 op/s 2024-05-27T07:38:23.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:23 smithi179 ceph-mon[35888]: pgmap v821: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.7 MiB/s wr, 193 op/s 2024-05-27T07:38:24.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:23 smithi132 ceph-mon[35283]: pgmap v821: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.7 MiB/s wr, 193 op/s 2024-05-27T07:38:24.833 INFO:tasks.workunit.client.0.smithi043.stdout: 10 68270 5.82 MB/sec execute 505 sec latency 454.284 ms 2024-05-27T07:38:25.833 INFO:tasks.workunit.client.0.smithi043.stdout: 10 68405 5.81 MB/sec execute 506 sec latency 96.682 ms 2024-05-27T07:38:25.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:25 smithi043 ceph-mon[31531]: pgmap v822: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.2 MiB/s wr, 166 op/s 2024-05-27T07:38:25.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:25 smithi179 ceph-mon[35888]: pgmap v822: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.2 MiB/s wr, 166 op/s 2024-05-27T07:38:26.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:25 smithi132 ceph-mon[35283]: pgmap v822: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.2 MiB/s wr, 166 op/s 2024-05-27T07:38:26.834 INFO:tasks.workunit.client.0.smithi043.stdout: 10 68522 5.82 MB/sec execute 507 sec latency 83.030 ms 2024-05-27T07:38:27.834 INFO:tasks.workunit.client.0.smithi043.stdout: 10 68624 5.82 MB/sec execute 508 sec latency 121.193 ms 2024-05-27T07:38:27.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:27 smithi043 ceph-mon[31531]: pgmap v823: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 4.4 MiB/s wr, 216 op/s 2024-05-27T07:38:27.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:38:27.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:38:27.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:38:27.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:38:27.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:27 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:38:27.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:27 smithi179 ceph-mon[35888]: pgmap v823: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 4.4 MiB/s wr, 216 op/s 2024-05-27T07:38:27.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:38:27.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:38:27.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:38:27.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:38:27.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:27 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:38:28.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:27 smithi132 ceph-mon[35283]: pgmap v823: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 4.4 MiB/s wr, 216 op/s 2024-05-27T07:38:28.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:38:28.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:38:28.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:38:28.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:38:28.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:27 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:38:28.834 INFO:tasks.workunit.client.0.smithi043.stdout: 10 68720 5.81 MB/sec execute 509 sec latency 245.098 ms 2024-05-27T07:38:29.834 INFO:tasks.workunit.client.0.smithi043.stdout: 10 68894 5.83 MB/sec execute 510 sec latency 131.430 ms 2024-05-27T07:38:29.899 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:29 smithi132 ceph-mon[35283]: pgmap v824: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-05-27T07:38:29.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:29 smithi043 ceph-mon[31531]: pgmap v824: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-05-27T07:38:29.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:29 smithi179 ceph-mon[35888]: pgmap v824: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-05-27T07:38:30.834 INFO:tasks.workunit.client.0.smithi043.stdout: 10 68997 5.84 MB/sec execute 511 sec latency 253.318 ms 2024-05-27T07:38:31.834 INFO:tasks.workunit.client.0.smithi043.stdout: 10 69147 5.84 MB/sec execute 512 sec latency 103.641 ms 2024-05-27T07:38:31.888 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:31 smithi043 ceph-mon[31531]: pgmap v825: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.1 MiB/s wr, 154 op/s 2024-05-27T07:38:31.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:31 smithi179 ceph-mon[35888]: pgmap v825: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.1 MiB/s wr, 154 op/s 2024-05-27T07:38:32.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:31 smithi132 ceph-mon[35283]: pgmap v825: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 3.1 MiB/s wr, 154 op/s 2024-05-27T07:38:32.834 INFO:tasks.workunit.client.0.smithi043.stdout: 10 69232 5.84 MB/sec execute 513 sec latency 105.982 ms 2024-05-27T07:38:33.834 INFO:tasks.workunit.client.0.smithi043.stdout: 10 69321 5.84 MB/sec execute 514 sec latency 75.098 ms 2024-05-27T07:38:33.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:33 smithi043 ceph-mon[31531]: pgmap v826: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.2 MiB/s wr, 222 op/s 2024-05-27T07:38:33.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:33 smithi179 ceph-mon[35888]: pgmap v826: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.2 MiB/s wr, 222 op/s 2024-05-27T07:38:34.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:33 smithi132 ceph-mon[35283]: pgmap v826: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.2 MiB/s wr, 222 op/s 2024-05-27T07:38:34.834 INFO:tasks.workunit.client.0.smithi043.stdout: 10 69434 5.85 MB/sec execute 515 sec latency 261.804 ms 2024-05-27T07:38:35.834 INFO:tasks.workunit.client.0.smithi043.stdout: 10 69529 5.85 MB/sec execute 516 sec latency 388.843 ms 2024-05-27T07:38:35.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:35 smithi043 ceph-mon[31531]: pgmap v827: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.8 MiB/s wr, 138 op/s 2024-05-27T07:38:35.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:35 smithi179 ceph-mon[35888]: pgmap v827: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.8 MiB/s wr, 138 op/s 2024-05-27T07:38:36.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:35 smithi132 ceph-mon[35283]: pgmap v827: 129 pgs: 129 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.8 MiB/s wr, 138 op/s 2024-05-27T07:38:36.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 69642 5.85 MB/sec execute 517 sec latency 89.295 ms 2024-05-27T07:38:37.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 69733 5.85 MB/sec execute 518 sec latency 328.482 ms 2024-05-27T07:38:37.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:37 smithi043 ceph-mon[31531]: pgmap v828: 129 pgs: 129 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 5.7 MiB/s wr, 197 op/s 2024-05-27T07:38:37.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:37 smithi179 ceph-mon[35888]: pgmap v828: 129 pgs: 129 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 5.7 MiB/s wr, 197 op/s 2024-05-27T07:38:38.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:37 smithi132 ceph-mon[35283]: pgmap v828: 129 pgs: 129 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 5.7 MiB/s wr, 197 op/s 2024-05-27T07:38:38.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 69849 5.85 MB/sec execute 519 sec latency 106.205 ms 2024-05-27T07:38:39.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 69956 5.85 MB/sec execute 520 sec latency 103.311 ms 2024-05-27T07:38:39.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:39 smithi043 ceph-mon[31531]: pgmap v829: 129 pgs: 129 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.7 MiB/s wr, 153 op/s 2024-05-27T07:38:39.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:39 smithi179 ceph-mon[35888]: pgmap v829: 129 pgs: 129 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.7 MiB/s wr, 153 op/s 2024-05-27T07:38:40.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:39 smithi132 ceph-mon[35283]: pgmap v829: 129 pgs: 129 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.7 MiB/s wr, 153 op/s 2024-05-27T07:38:40.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 70041 5.85 MB/sec execute 521 sec latency 96.754 ms 2024-05-27T07:38:41.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 70136 5.84 MB/sec execute 522 sec latency 71.293 ms 2024-05-27T07:38:41.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:41 smithi043 ceph-mon[31531]: pgmap v830: 129 pgs: 129 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.4 MiB/s wr, 133 op/s 2024-05-27T07:38:41.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:41 smithi179 ceph-mon[35888]: pgmap v830: 129 pgs: 129 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.4 MiB/s wr, 133 op/s 2024-05-27T07:38:42.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:41 smithi132 ceph-mon[35283]: pgmap v830: 129 pgs: 129 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 4.4 MiB/s wr, 133 op/s 2024-05-27T07:38:42.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 70242 5.84 MB/sec execute 523 sec latency 112.832 ms 2024-05-27T07:38:43.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 70364 5.84 MB/sec execute 524 sec latency 90.808 ms 2024-05-27T07:38:43.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:43 smithi043 ceph-mon[31531]: pgmap v831: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.5 MiB/s wr, 207 op/s 2024-05-27T07:38:43.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:43 smithi179 ceph-mon[35888]: pgmap v831: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.5 MiB/s wr, 207 op/s 2024-05-27T07:38:44.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:43 smithi132 ceph-mon[35283]: pgmap v831: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.5 MiB/s wr, 207 op/s 2024-05-27T07:38:44.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 70473 5.83 MB/sec execute 525 sec latency 103.548 ms 2024-05-27T07:38:45.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 70600 5.84 MB/sec execute 526 sec latency 617.502 ms 2024-05-27T07:38:45.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:45 smithi043 ceph-mon[31531]: pgmap v832: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.3 MiB/s wr, 139 op/s 2024-05-27T07:38:45.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:45 smithi179 ceph-mon[35888]: pgmap v832: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.3 MiB/s wr, 139 op/s 2024-05-27T07:38:46.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:45 smithi132 ceph-mon[35283]: pgmap v832: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.3 MiB/s wr, 139 op/s 2024-05-27T07:38:46.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 70717 5.84 MB/sec execute 527 sec latency 83.674 ms 2024-05-27T07:38:47.835 INFO:tasks.workunit.client.0.smithi043.stdout: 10 70827 5.84 MB/sec execute 528 sec latency 96.878 ms 2024-05-27T07:38:47.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:47 smithi043 ceph-mon[31531]: pgmap v833: 129 pgs: 129 active+clean; 1.5 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 6.0 MiB/s wr, 209 op/s 2024-05-27T07:38:47.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:47 smithi179 ceph-mon[35888]: pgmap v833: 129 pgs: 129 active+clean; 1.5 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 6.0 MiB/s wr, 209 op/s 2024-05-27T07:38:48.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:47 smithi132 ceph-mon[35283]: pgmap v833: 129 pgs: 129 active+clean; 1.5 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 6.0 MiB/s wr, 209 op/s 2024-05-27T07:38:48.836 INFO:tasks.workunit.client.0.smithi043.stdout: 10 70936 5.84 MB/sec execute 529 sec latency 112.016 ms 2024-05-27T07:38:49.836 INFO:tasks.workunit.client.0.smithi043.stdout: 10 71037 5.83 MB/sec execute 530 sec latency 108.014 ms 2024-05-27T07:38:49.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:49 smithi043 ceph-mon[31531]: pgmap v834: 129 pgs: 129 active+clean; 1.5 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 4.4 MiB/s wr, 168 op/s 2024-05-27T07:38:49.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:49 smithi179 ceph-mon[35888]: pgmap v834: 129 pgs: 129 active+clean; 1.5 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 4.4 MiB/s wr, 168 op/s 2024-05-27T07:38:50.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:49 smithi132 ceph-mon[35283]: pgmap v834: 129 pgs: 129 active+clean; 1.5 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 4.4 MiB/s wr, 168 op/s 2024-05-27T07:38:50.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:38:50.836 INFO:tasks.workunit.client.0.smithi043.stdout: 10 71155 5.83 MB/sec execute 531 sec latency 105.192 ms 2024-05-27T07:38:50.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:50 smithi043 ceph-mon[31531]: pgmap v835: 129 pgs: 129 active+clean; 1.5 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 4.2 MiB/s wr, 161 op/s 2024-05-27T07:38:50.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:50 smithi179 ceph-mon[35888]: pgmap v835: 129 pgs: 129 active+clean; 1.5 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 4.2 MiB/s wr, 161 op/s 2024-05-27T07:38:51.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:50 smithi132 ceph-mon[35283]: pgmap v835: 129 pgs: 129 active+clean; 1.5 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 4.2 MiB/s wr, 161 op/s 2024-05-27T07:38:51.542 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:38:51.837 INFO:tasks.workunit.client.0.smithi043.stdout: 10 71255 5.82 MB/sec execute 532 sec latency 123.098 ms 2024-05-27T07:38:52.837 INFO:tasks.workunit.client.0.smithi043.stdout: 10 71354 5.82 MB/sec execute 533 sec latency 296.090 ms 2024-05-27T07:38:53.209 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:52 smithi043 ceph-mon[31531]: pgmap v836: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 5.7 MiB/s wr, 247 op/s 2024-05-27T07:38:53.217 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:52 smithi179 ceph-mon[35888]: pgmap v836: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 5.7 MiB/s wr, 247 op/s 2024-05-27T07:38:53.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:52 smithi132 ceph-mon[35283]: pgmap v836: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 5.7 MiB/s wr, 247 op/s 2024-05-27T07:38:53.836 INFO:tasks.workunit.client.0.smithi043.stdout: 10 71476 5.81 MB/sec execute 534 sec latency 130.551 ms 2024-05-27T07:38:54.837 INFO:tasks.workunit.client.0.smithi043.stdout: 10 71578 5.81 MB/sec execute 535 sec latency 109.159 ms 2024-05-27T07:38:55.213 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:54 smithi043 ceph-mon[31531]: pgmap v837: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 3.6 MiB/s wr, 173 op/s 2024-05-27T07:38:55.480 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:54 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:38:55.481 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:54 smithi132 ceph-mon[35283]: pgmap v837: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 3.6 MiB/s wr, 173 op/s 2024-05-27T07:38:55.482 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:54 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:38:55.482 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:54 smithi179 ceph-mon[35888]: pgmap v837: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 3.6 MiB/s wr, 173 op/s 2024-05-27T07:38:55.482 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:54 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:38:55.837 INFO:tasks.workunit.client.0.smithi043.stdout: 10 71676 5.81 MB/sec execute 536 sec latency 125.630 ms 2024-05-27T07:38:56.837 INFO:tasks.workunit.client.0.smithi043.stdout: 10 71751 5.80 MB/sec execute 537 sec latency 265.695 ms 2024-05-27T07:38:56.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:56 smithi043 ceph-mon[31531]: pgmap v838: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 4.7 MiB/s wr, 234 op/s 2024-05-27T07:38:57.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:56 smithi179 ceph-mon[35888]: pgmap v838: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 4.7 MiB/s wr, 234 op/s 2024-05-27T07:38:57.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:56 smithi132 ceph-mon[35283]: pgmap v838: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 4.7 MiB/s wr, 234 op/s 2024-05-27T07:38:57.837 INFO:tasks.workunit.client.0.smithi043.stdout: 10 71844 5.80 MB/sec execute 538 sec latency 127.149 ms 2024-05-27T07:38:58.837 INFO:tasks.workunit.client.0.smithi043.stdout: 10 71904 5.80 MB/sec execute 539 sec latency 412.411 ms 2024-05-27T07:38:59.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:38:58 smithi043 ceph-mon[31531]: pgmap v839: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 3.1 MiB/s wr, 177 op/s 2024-05-27T07:38:59.347 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:38:58 smithi132 ceph-mon[35283]: pgmap v839: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 3.1 MiB/s wr, 177 op/s 2024-05-27T07:38:59.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:38:58 smithi179 ceph-mon[35888]: pgmap v839: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 3.1 MiB/s wr, 177 op/s 2024-05-27T07:38:59.837 INFO:tasks.workunit.client.0.smithi043.stdout: 10 72033 5.82 MB/sec execute 540 sec latency 98.594 ms 2024-05-27T07:39:00.837 INFO:tasks.workunit.client.0.smithi043.stdout: 10 72173 5.82 MB/sec execute 541 sec latency 147.660 ms 2024-05-27T07:39:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.902+0000 7fe67de00700 1 -- 172.21.15.43:0/1458925531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe678103b90 msgr2=0x7fe6781080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:00.903 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:39:00.903 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:00 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:39:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.903+0000 7fe67de00700 1 --2- 172.21.15.43:0/1458925531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe678103b90 0x7fe6781080c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe6640098f0 tx=0x7fe664009c00 comp rx=0 tx=0).stop 2024-05-27T07:39:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.905+0000 7fe67de00700 1 -- 172.21.15.43:0/1458925531 shutdown_connections 2024-05-27T07:39:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.905+0000 7fe67de00700 1 --2- 172.21.15.43:0/1458925531 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe678108690 0x7fe678108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.905+0000 7fe67de00700 1 --2- 172.21.15.43:0/1458925531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe678103240 0x7fe678103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.905+0000 7fe67de00700 1 --2- 172.21.15.43:0/1458925531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe678103b90 0x7fe6781080c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.905+0000 7fe67de00700 1 -- 172.21.15.43:0/1458925531 >> 172.21.15.43:0/1458925531 conn(0x7fe6780fedb0 msgr2=0x7fe6781011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.905+0000 7fe67de00700 1 -- 172.21.15.43:0/1458925531 shutdown_connections 2024-05-27T07:39:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.905+0000 7fe67de00700 1 -- 172.21.15.43:0/1458925531 wait complete. 2024-05-27T07:39:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.907+0000 7fe67de00700 1 Processor -- start 2024-05-27T07:39:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.907+0000 7fe67de00700 1 -- start start 2024-05-27T07:39:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.907+0000 7fe67de00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe678103240 0x7fe67806e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.907+0000 7fe67de00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe678103b90 0x7fe67806ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.907+0000 7fe67de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe678108690 0x7fe67806f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.907+0000 7fe67de00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe67811a910 con 0x7fe678103b90 2024-05-27T07:39:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.907+0000 7fe67de00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe67811a790 con 0x7fe678108690 2024-05-27T07:39:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.907+0000 7fe67de00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe67811aa90 con 0x7fe678103240 2024-05-27T07:39:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.908+0000 7fe677400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe678108690 0x7fe67806f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.908+0000 7fe677400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe678108690 0x7fe67806f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:44862/0 (socket says 172.21.15.43:44862) 2024-05-27T07:39:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.908+0000 7fe677400700 1 -- 172.21.15.43:0/2907181151 learned_addr learned my addr 172.21.15.43:0/2907181151 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.908+0000 7fe676a00700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe678103240 0x7fe67806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.908+0000 7fe676000700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe678103b90 0x7fe67806ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.909+0000 7fe677400700 1 -- 172.21.15.43:0/2907181151 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe678103240 msgr2=0x7fe67806e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.909+0000 7fe677400700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe678103240 0x7fe67806e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.910+0000 7fe677400700 1 -- 172.21.15.43:0/2907181151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe678103b90 msgr2=0x7fe67806ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.910+0000 7fe677400700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe678103b90 0x7fe67806ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.910+0000 7fe677400700 1 -- 172.21.15.43:0/2907181151 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6780730a0 con 0x7fe678108690 2024-05-27T07:39:00.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.910+0000 7fe677400700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe678108690 0x7fe67806f0f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe67000ef30 tx=0x7fe67000c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:00.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.912+0000 7fe67ca00700 1 -- 172.21.15.43:0/2907181151 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6700092e0 con 0x7fe678108690 2024-05-27T07:39:00.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.912+0000 7fe67ca00700 1 -- 172.21.15.43:0/2907181151 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe670004bb0 con 0x7fe678108690 2024-05-27T07:39:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.913+0000 7fe67de00700 1 -- 172.21.15.43:0/2907181151 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe678073330 con 0x7fe678108690 2024-05-27T07:39:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.913+0000 7fe67ca00700 1 -- 172.21.15.43:0/2907181151 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670004020 con 0x7fe678108690 2024-05-27T07:39:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.914+0000 7fe67de00700 1 -- 172.21.15.43:0/2907181151 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe678073870 con 0x7fe678108690 2024-05-27T07:39:00.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.919+0000 7fe67ca00700 1 -- 172.21.15.43:0/2907181151 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fe67001b030 con 0x7fe678108690 2024-05-27T07:39:00.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.921+0000 7fe67ca00700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe65c0b2bc0 0x7fe65c0b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.921+0000 7fe67ca00700 1 -- 172.21.15.43:0/2907181151 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 9496+0+0 (secure 0 0 0) 0x7fe6700e20c0 con 0x7fe678108690 2024-05-27T07:39:00.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.923+0000 7fe676a00700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe65c0b2bc0 0x7fe65c0b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:00.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.924+0000 7fe676a00700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe65c0b2bc0 0x7fe65c0b50d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe67806fec0 tx=0x7fe668009310 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:00.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:39:00.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:00 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:39:00.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.989+0000 7fe68c876b80 1 Processor -- start 2024-05-27T07:39:00.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.990+0000 7fe68c876b80 1 -- start start 2024-05-27T07:39:00.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.991+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dc9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.991+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dc9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.991+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dd6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:00.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.991+0000 7fe68c876b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558505cf7e40 con 0x558505d369f0 2024-05-27T07:39:00.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.991+0000 7fe68c876b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558505ce79e0 con 0x558505dc9e70 2024-05-27T07:39:00.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.991+0000 7fe68c876b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x558505ceba20 con 0x558505c4ae40 2024-05-27T07:39:00.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.992+0000 7fe676a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dc9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:00.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.992+0000 7fe676a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dc9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43904/0 (socket says 172.21.15.43:43904) 2024-05-27T07:39:00.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.992+0000 7fe676a00700 1 -- 172.21.15.43:0/1240150651 learned_addr learned my addr 172.21.15.43:0/1240150651 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:00.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.993+0000 7fe676000700 1 --2- 172.21.15.43:0/1240150651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dc9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:00.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.993+0000 7fe676a00700 1 -- 172.21.15.43:0/1240150651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 msgr2=0x558505dc9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:00.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.993+0000 7fe676a00700 1 --2- 172.21.15.43:0/1240150651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dc9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.993+0000 7fe676a00700 1 -- 172.21.15.43:0/1240150651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 msgr2=0x558505dd6110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:39:00.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.993+0000 7fe676a00700 1 --2- 172.21.15.43:0/1240150651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dd6110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:00.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.993+0000 7fe676a00700 1 -- 172.21.15.43:0/1240150651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558505cb0420 con 0x558505d369f0 2024-05-27T07:39:00.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.994+0000 7fe676a00700 1 --2- 172.21.15.43:0/1240150651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dc9330 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fe66800e460 tx=0x7fe668014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e22ece495848e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:00.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.997+0000 7fe66cc00700 1 -- 172.21.15.43:0/1240150651 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe668004830 con 0x558505d369f0 2024-05-27T07:39:00.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.997+0000 7fe66cc00700 1 -- 172.21.15.43:0/1240150651 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe668005020 con 0x558505d369f0 2024-05-27T07:39:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:00.999+0000 7fe68c876b80 1 -- 172.21.15.43:0/1240150651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 msgr2=0x558505dc9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.001+0000 7fe66cc00700 1 -- 172.21.15.43:0/1240150651 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6680266e0 con 0x558505d369f0 2024-05-27T07:39:00.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.001+0000 7fe68c876b80 1 --2- 172.21.15.43:0/1240150651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dc9330 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fe66800e460 tx=0x7fe668014000 comp rx=0 tx=0).stop 2024-05-27T07:39:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.002+0000 7fe68c876b80 1 -- 172.21.15.43:0/1240150651 shutdown_connections 2024-05-27T07:39:01.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.002+0000 7fe68c876b80 1 --2- 172.21.15.43:0/1240150651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dd6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.002+0000 7fe68c876b80 1 --2- 172.21.15.43:0/1240150651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dc9330 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.002+0000 7fe68c876b80 1 --2- 172.21.15.43:0/1240150651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dc9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.002+0000 7fe68c876b80 1 -- 172.21.15.43:0/1240150651 >> 172.21.15.43:0/1240150651 conn(0x558505ce4390 msgr2=0x558505dc54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.003+0000 7fe68c876b80 1 -- 172.21.15.43:0/1240150651 shutdown_connections 2024-05-27T07:39:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.003+0000 7fe68c876b80 1 -- 172.21.15.43:0/1240150651 wait complete. 2024-05-27T07:39:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.004+0000 7fe68c876b80 1 -- start start 2024-05-27T07:39:01.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.005+0000 7fe68c876b80 1 Processor -- start 2024-05-27T07:39:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.008+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.008+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dd3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.008+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dd4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.008+0000 7fe68c876b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558505bedfa0 con 0x558505d369f0 2024-05-27T07:39:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.009+0000 7fe68c876b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558505c88760 con 0x558505dc9e70 2024-05-27T07:39:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.009+0000 7fe68c876b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x558505de7be0 con 0x558505c4ae40 2024-05-27T07:39:01.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.009+0000 7fe68c876b80 10 client.0 _ll_register_callbacks cb 0x558505c91c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.010+0000 7fe676000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dd3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.010+0000 7fe676a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.010+0000 7fe676a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:32998/0 (socket says 172.21.15.43:32998) 2024-05-27T07:39:01.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.010+0000 7fe676a00700 1 -- 172.21.15.43:0/4048340351 learned_addr learned my addr 172.21.15.43:0/4048340351 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:01.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.012+0000 7fe677400700 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dd4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.012+0000 7fe676000700 1 -- 172.21.15.43:0/4048340351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 msgr2=0x558505dd3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.012+0000 7fe676000700 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.012+0000 7fe676000700 1 -- 172.21.15.43:0/4048340351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 msgr2=0x558505dd4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.012+0000 7fe676000700 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dd4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.012+0000 7fe676000700 1 -- 172.21.15.43:0/4048340351 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558505c378d0 con 0x558505d369f0 2024-05-27T07:39:01.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.012+0000 7fe677400700 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dd4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:01.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.013+0000 7fe661600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:39:01.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.013+0000 7fe676000700 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dd3b50 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fe6640098c0 tx=0x7fe66400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ad43d42bce759d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.014+0000 7fe661600700 1 -- 172.21.15.43:0/4048340351 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe664005240 con 0x558505d369f0 2024-05-27T07:39:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.015+0000 7fe661600700 1 -- 172.21.15.43:0/4048340351 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 9496+0+0 (secure 0 0 0) 0x7fe6640053e0 con 0x558505d369f0 2024-05-27T07:39:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.015+0000 7fe68c876b80 1 -- 172.21.15.43:0/4048340351 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe668005020 con 0x558505d369f0 2024-05-27T07:39:01.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.016+0000 7fe661600700 1 -- 172.21.15.43:0/4048340351 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fe6640055e0 con 0x558505d369f0 2024-05-27T07:39:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.016+0000 7fe68c876b80 10 client.18492 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.017+0000 7fe68c876b80 1 -- 172.21.15.43:0/4048340351 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x558505cb8060 con 0x558505d369f0 2024-05-27T07:39:01.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.017+0000 7fe661600700 1 -- 172.21.15.43:0/4048340351 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe668005020 con 0x558505d369f0 2024-05-27T07:39:01.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.018+0000 7fe661600700 1 -- 172.21.15.43:0/4048340351 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fe66402a030 con 0x558505d369f0 2024-05-27T07:39:01.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.018+0000 7fe68c876b80 10 client.18492 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:01.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.018+0000 7fe68c876b80 10 client.18492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:01.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.019+0000 7fe68c876b80 20 client.18492 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:01.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.019+0000 7fe68c876b80 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558505df4dc0 0x558505df7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.019+0000 7fe68c876b80 4 client.18492 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:01.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.019+0000 7fe68c876b80 1 -- 172.21.15.43:0/4048340351 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558505cf9d90 con 0x558505df4dc0 2024-05-27T07:39:01.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.020+0000 7fe676a00700 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558505df4dc0 0x558505df7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.022+0000 7fe661600700 10 client.18492 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:01.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.022+0000 7fe676a00700 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558505df4dc0 0x558505df7210 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bfa1f11b434a68f server_cookie=57b831271c668ad9 in_seq=0 out_seq=0 2024-05-27T07:39:01.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.024+0000 7fe661600700 1 -- 172.21.15.43:0/4048340351 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558505cf9d90 con 0x558505df4dc0 2024-05-27T07:39:01.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.024+0000 7fe661600700 10 client.18492 handle_command_reply: tid=0 2024-05-27T07:39:01.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.024+0000 7fe68c876b80 1 client.18492 shutdown 2024-05-27T07:39:01.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.025+0000 7fe68c876b80 1 -- 172.21.15.43:0/4048340351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 msgr2=0x558505dd3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.025+0000 7fe68c876b80 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dd3b50 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fe6640098c0 tx=0x7fe66400f690 comp rx=0 tx=0).stop 2024-05-27T07:39:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.025+0000 7fe68c876b80 1 -- 172.21.15.43:0/4048340351 shutdown_connections 2024-05-27T07:39:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.025+0000 7fe68c876b80 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dd4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.025+0000 7fe68c876b80 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.025+0000 7fe68c876b80 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558505df4dc0 0x558505df7210 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.025+0000 7fe68c876b80 1 --2- 172.21.15.43:0/4048340351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dd3b50 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.025+0000 7fe68c876b80 1 -- 172.21.15.43:0/4048340351 >> 172.21.15.43:0/4048340351 conn(0x558505ce4390 msgr2=0x558505dc93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.026+0000 7fe661600700 0 client.18492 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.026+0000 7fe68c876b80 1 -- 172.21.15.43:0/4048340351 shutdown_connections 2024-05-27T07:39:01.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.026+0000 7fe68c876b80 1 -- 172.21.15.43:0/4048340351 wait complete. 2024-05-27T07:39:01.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.026+0000 7fe68c876b80 20 client.18492 trim_cache size 0 max 16384 2024-05-27T07:39:01.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.076+0000 7fe68c876b80 1 Processor -- start 2024-05-27T07:39:01.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.076+0000 7fe68c876b80 1 -- start start 2024-05-27T07:39:01.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.077+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.077+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dcedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.077+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dcd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.077+0000 7fe68c876b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558505dc3e50 con 0x558505d369f0 2024-05-27T07:39:01.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.077+0000 7fe68c876b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781b6f40 con 0x558505dc9e70 2024-05-27T07:39:01.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.077+0000 7fe68c876b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe678111f70 con 0x558505c4ae40 2024-05-27T07:39:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.077+0000 7fe676a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.077+0000 7fe676a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:33010/0 (socket says 172.21.15.43:33010) 2024-05-27T07:39:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.077+0000 7fe676a00700 1 -- 172.21.15.43:0/2540094663 learned_addr learned my addr 172.21.15.43:0/2540094663 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.078+0000 7fe676000700 1 --2- 172.21.15.43:0/2540094663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dcedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.078+0000 7fe677400700 1 --2- 172.21.15.43:0/2540094663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dcd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.079+0000 7fe676000700 1 -- 172.21.15.43:0/2540094663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 msgr2=0x558505dd2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.079+0000 7fe676000700 1 --2- 172.21.15.43:0/2540094663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.079+0000 7fe676000700 1 -- 172.21.15.43:0/2540094663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 msgr2=0x558505dcd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.079+0000 7fe676000700 1 --2- 172.21.15.43:0/2540094663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dcd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.079+0000 7fe676000700 1 -- 172.21.15.43:0/2540094663 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558505dc1240 con 0x558505d369f0 2024-05-27T07:39:01.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.080+0000 7fe676a00700 1 --2- 172.21.15.43:0/2540094663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:01.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.080+0000 7fe676000700 1 --2- 172.21.15.43:0/2540094663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dcedf0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fe66402b870 tx=0x7fe664009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=522a717cf8592003 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:01.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.081+0000 7fe663e00700 1 -- 172.21.15.43:0/2540094663 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe664039070 con 0x558505d369f0 2024-05-27T07:39:01.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.082+0000 7fe663e00700 1 -- 172.21.15.43:0/2540094663 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe66402bdf0 con 0x558505d369f0 2024-05-27T07:39:01.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.082+0000 7fe663e00700 1 -- 172.21.15.43:0/2540094663 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe664017400 con 0x558505d369f0 2024-05-27T07:39:01.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.083+0000 7fe68c876b80 1 -- 172.21.15.43:0/2540094663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 msgr2=0x558505dcedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.083+0000 7fe68c876b80 1 --2- 172.21.15.43:0/2540094663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dcedf0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fe66402b870 tx=0x7fe664009230 comp rx=0 tx=0).stop 2024-05-27T07:39:01.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.084+0000 7fe68c876b80 1 -- 172.21.15.43:0/2540094663 shutdown_connections 2024-05-27T07:39:01.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.084+0000 7fe68c876b80 1 --2- 172.21.15.43:0/2540094663 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505dc9e70 0x558505dcd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.084+0000 7fe68c876b80 1 --2- 172.21.15.43:0/2540094663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505c4ae40 0x558505dd2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.084+0000 7fe68c876b80 1 --2- 172.21.15.43:0/2540094663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505dcedf0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.084+0000 7fe68c876b80 1 -- 172.21.15.43:0/2540094663 >> 172.21.15.43:0/2540094663 conn(0x558505ce4390 msgr2=0x558505dccee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.085+0000 7fe68c876b80 1 -- 172.21.15.43:0/2540094663 shutdown_connections 2024-05-27T07:39:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.085+0000 7fe68c876b80 1 -- 172.21.15.43:0/2540094663 wait complete. 2024-05-27T07:39:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.086+0000 7fe68c876b80 1 -- start start 2024-05-27T07:39:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.086+0000 7fe68c876b80 1 Processor -- start 2024-05-27T07:39:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.087+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505c4ae40 0x558505decf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.088+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505ded470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.088+0000 7fe68c876b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505dc9e70 0x558505de8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.088+0000 7fe68c876b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe678112640 con 0x558505d369f0 2024-05-27T07:39:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.088+0000 7fe68c876b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe67806bf30 con 0x558505c4ae40 2024-05-27T07:39:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.088+0000 7fe68c876b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe678112d10 con 0x558505dc9e70 2024-05-27T07:39:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.088+0000 7fe68c876b80 10 client.0 _ll_register_callbacks cb 0x558505c91c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:01.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.089+0000 7fe676a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505c4ae40 0x558505decf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.089+0000 7fe676a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505c4ae40 0x558505decf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:44900/0 (socket says 172.21.15.43:44900) 2024-05-27T07:39:01.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.089+0000 7fe676a00700 1 -- 172.21.15.43:0/1569356800 learned_addr learned my addr 172.21.15.43:0/1569356800 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:01.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.089+0000 7fe677400700 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505dc9e70 0x558505de8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.089+0000 7fe676000700 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505ded470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.090+0000 7fe676a00700 1 -- 172.21.15.43:0/1569356800 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505dc9e70 msgr2=0x558505de8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.090+0000 7fe676a00700 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505dc9e70 0x558505de8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.090+0000 7fe676a00700 1 -- 172.21.15.43:0/1569356800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 msgr2=0x558505ded470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:01.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.090+0000 7fe676a00700 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505ded470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.091+0000 7fe676a00700 1 -- 172.21.15.43:0/1569356800 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558505dd4400 con 0x558505c4ae40 2024-05-27T07:39:01.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.091+0000 7fe676000700 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505ded470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:01.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.091+0000 7fe661600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:39:01.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.091+0000 7fe676a00700 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505c4ae40 0x558505decf30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x558505dc5ed0 tx=0x7fe668026630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cef60c06bb4a3228 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:01.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.092+0000 7fe661600700 1 -- 172.21.15.43:0/1569356800 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe668026be0 con 0x558505c4ae40 2024-05-27T07:39:01.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.092+0000 7fe68c876b80 1 -- 172.21.15.43:0/1569356800 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe66402bdf0 con 0x558505c4ae40 2024-05-27T07:39:01.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.092+0000 7fe661600700 1 -- 172.21.15.43:0/1569356800 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(159..159 src has 1..159) ==== 9496+0+0 (secure 0 0 0) 0x7fe6680269e0 con 0x558505c4ae40 2024-05-27T07:39:01.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.093+0000 7fe661600700 1 -- 172.21.15.43:0/1569356800 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fe668041050 con 0x558505c4ae40 2024-05-27T07:39:01.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.094+0000 7fe68c876b80 10 client.37886 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:01.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.094+0000 7fe68c876b80 1 -- 172.21.15.43:0/1569356800 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x558505e5f090 con 0x558505c4ae40 2024-05-27T07:39:01.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.094+0000 7fe661600700 1 -- 172.21.15.43:0/1569356800 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe66402bdf0 con 0x558505c4ae40 2024-05-27T07:39:01.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.094+0000 7fe661600700 1 -- 172.21.15.43:0/1569356800 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fe668003820 con 0x558505c4ae40 2024-05-27T07:39:01.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.095+0000 7fe68c876b80 10 client.37886 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:01.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.095+0000 7fe68c876b80 10 client.37886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:01.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.095+0000 7fe68c876b80 20 client.37886 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:01.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.095+0000 7fe68c876b80 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558505e5f740 0x558505e61b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:01.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.095+0000 7fe68c876b80 4 client.37886 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:39:01.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.095+0000 7fe68c876b80 1 -- 172.21.15.43:0/1569356800 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558505cf3570 con 0x558505e5f740 2024-05-27T07:39:01.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.095+0000 7fe676000700 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558505e5f740 0x558505e61b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.097+0000 7fe676000700 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558505e5f740 0x558505e61b60 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4c68072551f85d0 server_cookie=22c03c69ea7c19f0 in_seq=0 out_seq=0 2024-05-27T07:39:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.097+0000 7fe661600700 10 client.37886 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.097+0000 7fe661600700 1 -- 172.21.15.43:0/1569356800 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558505cf3570 con 0x558505e5f740 2024-05-27T07:39:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.097+0000 7fe661600700 10 client.37886 handle_command_reply: tid=0 2024-05-27T07:39:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.097+0000 7fe68c876b80 1 client.37886 shutdown 2024-05-27T07:39:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.098+0000 7fe68c876b80 1 -- 172.21.15.43:0/1569356800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505c4ae40 msgr2=0x558505decf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.098+0000 7fe68c876b80 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505c4ae40 0x558505decf30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x558505dc5ed0 tx=0x7fe668026630 comp rx=0 tx=0).stop 2024-05-27T07:39:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.098+0000 7fe68c876b80 1 -- 172.21.15.43:0/1569356800 shutdown_connections 2024-05-27T07:39:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.098+0000 7fe68c876b80 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558505dc9e70 0x558505de8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.098+0000 7fe68c876b80 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558505c4ae40 0x558505decf30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.098+0000 7fe68c876b80 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x558505e5f740 0x558505e61b60 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.098+0000 7fe661600700 0 client.37886 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:01.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.098+0000 7fe68c876b80 1 --2- 172.21.15.43:0/1569356800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558505d369f0 0x558505ded470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.098+0000 7fe68c876b80 1 -- 172.21.15.43:0/1569356800 >> 172.21.15.43:0/1569356800 conn(0x558505ce4390 msgr2=0x558505dd4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:39:01.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:00 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:39:01.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.100+0000 7fe68c876b80 1 -- 172.21.15.43:0/1569356800 shutdown_connections 2024-05-27T07:39:01.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.100+0000 7fe68c876b80 1 -- 172.21.15.43:0/1569356800 wait complete. 2024-05-27T07:39:01.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.100+0000 7fe68c876b80 20 client.37886 trim_cache size 0 max 16384 2024-05-27T07:39:01.100 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:39:01.100 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:39:01.100 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:39:01.100 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.102+0000 7fe67de00700 1 -- 172.21.15.43:0/2907181151 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe65c0b2bc0 msgr2=0x7fe65c0b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.102+0000 7fe67de00700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe65c0b2bc0 0x7fe65c0b50d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe67806fec0 tx=0x7fe668009310 comp rx=0 tx=0).stop 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.102+0000 7fe67de00700 1 -- 172.21.15.43:0/2907181151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe678108690 msgr2=0x7fe67806f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.102+0000 7fe67de00700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe678108690 0x7fe67806f0f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe67000ef30 tx=0x7fe67000c450 comp rx=0 tx=0).stop 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.102+0000 7fe67de00700 1 -- 172.21.15.43:0/2907181151 shutdown_connections 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.102+0000 7fe67de00700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fe65c0b2bc0 0x7fe65c0b50d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.102+0000 7fe67de00700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe678108690 0x7fe67806f0f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.102+0000 7fe67de00700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe678103240 0x7fe67806e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.102+0000 7fe67de00700 1 --2- 172.21.15.43:0/2907181151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe678103b90 0x7fe67806ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.103+0000 7fe67de00700 1 -- 172.21.15.43:0/2907181151 >> 172.21.15.43:0/2907181151 conn(0x7fe6780fedb0 msgr2=0x7fe678106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.103+0000 7fe67de00700 1 -- 172.21.15.43:0/2907181151 shutdown_connections 2024-05-27T07:39:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:01.103+0000 7fe67de00700 1 -- 172.21.15.43:0/2907181151 wait complete. 2024-05-27T07:39:01.837 INFO:tasks.workunit.client.0.smithi043.stdout: 10 72260 5.82 MB/sec execute 542 sec latency 171.306 ms 2024-05-27T07:39:01.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:01 smithi043 ceph-mon[31531]: pgmap v840: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.8 MiB/s wr, 159 op/s 2024-05-27T07:39:01.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:01.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:01.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:01 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:39:01.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:01 smithi179 ceph-mon[35888]: pgmap v840: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.8 MiB/s wr, 159 op/s 2024-05-27T07:39:01.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:01.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:01.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:01 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:39:02.093 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:39:02.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:39:02.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:01 smithi132 ceph-mon[35283]: pgmap v840: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.8 MiB/s wr, 159 op/s 2024-05-27T07:39:02.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:39:02.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:39:02.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:01 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:39:02.826 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:02.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 72411 5.83 MB/sec execute 543 sec latency 151.751 ms 2024-05-27T07:39:02.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:02 smithi043 ceph-mon[31531]: pgmap v841: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 4.9 MiB/s wr, 229 op/s 2024-05-27T07:39:02.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:02 smithi043 ceph-mon[31531]: pgmap v842: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 4.0 MiB/s wr, 169 op/s 2024-05-27T07:39:02.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:02 smithi043 ceph-mon[31531]: osdmap e160: 12 total, 12 up, 12 in 2024-05-27T07:39:02.964 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:02 smithi179 ceph-mon[35888]: pgmap v841: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 4.9 MiB/s wr, 229 op/s 2024-05-27T07:39:02.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:02 smithi179 ceph-mon[35888]: pgmap v842: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 4.0 MiB/s wr, 169 op/s 2024-05-27T07:39:02.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:02 smithi179 ceph-mon[35888]: osdmap e160: 12 total, 12 up, 12 in 2024-05-27T07:39:03.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:02 smithi132 ceph-mon[35283]: pgmap v841: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 4.9 MiB/s wr, 229 op/s 2024-05-27T07:39:03.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:02 smithi132 ceph-mon[35283]: pgmap v842: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 4.0 MiB/s wr, 169 op/s 2024-05-27T07:39:03.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:02 smithi132 ceph-mon[35283]: osdmap e160: 12 total, 12 up, 12 in 2024-05-27T07:39:03.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 72494 5.83 MB/sec execute 544 sec latency 178.292 ms 2024-05-27T07:39:03.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:03 smithi043 ceph-mon[31531]: pgmap v844: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.3 MiB/s wr, 118 op/s 2024-05-27T07:39:03.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:03 smithi179 ceph-mon[35888]: pgmap v844: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.3 MiB/s wr, 118 op/s 2024-05-27T07:39:04.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:03 smithi132 ceph-mon[35283]: pgmap v844: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.3 MiB/s wr, 118 op/s 2024-05-27T07:39:04.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 72591 5.83 MB/sec execute 545 sec latency 153.601 ms 2024-05-27T07:39:05.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 72684 5.82 MB/sec execute 546 sec latency 76.987 ms 2024-05-27T07:39:06.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 72828 5.84 MB/sec execute 547 sec latency 137.487 ms 2024-05-27T07:39:06.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:06 smithi043 ceph-mon[31531]: pgmap v845: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.0 MiB/s wr, 98 op/s 2024-05-27T07:39:06.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:06 smithi179 ceph-mon[35888]: pgmap v845: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.0 MiB/s wr, 98 op/s 2024-05-27T07:39:07.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:06 smithi132 ceph-mon[35283]: pgmap v845: 129 pgs: 129 active+clean; 1.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.0 MiB/s wr, 98 op/s 2024-05-27T07:39:07.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.212+0000 7f2beac00700 1 -- 172.21.15.43:0/2522312760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2be4106570 msgr2=0x7f2be4106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:07.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.212+0000 7f2beac00700 1 --2- 172.21.15.43:0/2522312760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2be4106570 0x7f2be4106980 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc009990 tx=0x7f2bcc009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:07.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.212+0000 7f2beac00700 1 -- 172.21.15.43:0/2522312760 shutdown_connections 2024-05-27T07:39:07.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.212+0000 7f2beac00700 1 --2- 172.21.15.43:0/2522312760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2be4106ec0 0x7f2be4113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:07.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.212+0000 7f2beac00700 1 --2- 172.21.15.43:0/2522312760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2be410c5b0 0x7f2be410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:07.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.212+0000 7f2beac00700 1 --2- 172.21.15.43:0/2522312760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2be4106570 0x7f2be4106980 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:07.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.212+0000 7f2beac00700 1 -- 172.21.15.43:0/2522312760 >> 172.21.15.43:0/2522312760 conn(0x7f2be4102070 msgr2=0x7f2be41044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:07.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.212+0000 7f2beac00700 1 -- 172.21.15.43:0/2522312760 shutdown_connections 2024-05-27T07:39:07.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.212+0000 7f2beac00700 1 -- 172.21.15.43:0/2522312760 wait complete. 2024-05-27T07:39:07.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.214+0000 7f2beac00700 1 Processor -- start 2024-05-27T07:39:07.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.214+0000 7f2beac00700 1 -- start start 2024-05-27T07:39:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.214+0000 7f2beac00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2be4106570 0x7f2be41121a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.214+0000 7f2beac00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2be4106ec0 0x7f2be4112700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.214+0000 7f2beac00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2be410c5b0 0x7f2be41a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.214+0000 7f2beac00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2be4079d50 con 0x7f2be4106570 2024-05-27T07:39:07.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.214+0000 7f2beac00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2be4079bd0 con 0x7f2be4106ec0 2024-05-27T07:39:07.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.214+0000 7f2beac00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2be4079ed0 con 0x7f2be410c5b0 2024-05-27T07:39:07.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.215+0000 7f2be3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2be4106570 0x7f2be41121a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:07.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.215+0000 7f2be3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2be4106ec0 0x7f2be4112700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:07.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.215+0000 7f2be3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2be4106570 0x7f2be41121a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56562/0 (socket says 172.21.15.43:56562) 2024-05-27T07:39:07.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.215+0000 7f2be3e00700 1 -- 172.21.15.43:0/2925653749 learned_addr learned my addr 172.21.15.43:0/2925653749 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:07.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.215+0000 7f2be8c00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2be410c5b0 0x7f2be41a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:07.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.216+0000 7f2be3e00700 1 -- 172.21.15.43:0/2925653749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2be410c5b0 msgr2=0x7f2be41a5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:07.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.216+0000 7f2be3e00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2be410c5b0 0x7f2be41a5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:07.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.216+0000 7f2be3e00700 1 -- 172.21.15.43:0/2925653749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2be4106ec0 msgr2=0x7f2be4112700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:07.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.216+0000 7f2be3e00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2be4106ec0 0x7f2be4112700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:07.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.216+0000 7f2be3e00700 1 -- 172.21.15.43:0/2925653749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be41a6250 con 0x7f2be4106570 2024-05-27T07:39:07.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.216+0000 7f2be3400700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2be4106ec0 0x7f2be4112700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:07.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.217+0000 7f2be3e00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2be4106570 0x7f2be41121a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00d980 tx=0x7f2bdc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:07.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.217+0000 7f2be0c00700 1 -- 172.21.15.43:0/2925653749 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc017070 con 0x7f2be4106570 2024-05-27T07:39:07.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.217+0000 7f2be0c00700 1 -- 172.21.15.43:0/2925653749 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2bdc009dc0 con 0x7f2be4106570 2024-05-27T07:39:07.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.217+0000 7f2beac00700 1 -- 172.21.15.43:0/2925653749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be41a6540 con 0x7f2be4106570 2024-05-27T07:39:07.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.217+0000 7f2beac00700 1 -- 172.21.15.43:0/2925653749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be41ade90 con 0x7f2be4106570 2024-05-27T07:39:07.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.218+0000 7f2be0c00700 1 -- 172.21.15.43:0/2925653749 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc00f3c0 con 0x7f2be4106570 2024-05-27T07:39:07.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.219+0000 7f2beac00700 1 -- 172.21.15.43:0/2925653749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be40512b0 con 0x7f2be4106570 2024-05-27T07:39:07.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.226+0000 7f2be0c00700 1 -- 172.21.15.43:0/2925653749 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f2bdc048d10 con 0x7f2be4106570 2024-05-27T07:39:07.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.229+0000 7f2be0c00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bc80b2a70 0x7f2bc80b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:07.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.229+0000 7f2be0c00700 1 -- 172.21.15.43:0/2925653749 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 9584+0+0 (secure 0 0 0) 0x7f2bdc0e2640 con 0x7f2be4106570 2024-05-27T07:39:07.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.229+0000 7f2be3400700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bc80b2a70 0x7f2bc80b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:07.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.231+0000 7f2be3400700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bc80b2a70 0x7f2bc80b4f80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2be407c0a0 tx=0x7f2bcc00b540 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:07.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.231+0000 7f2be0c00700 1 -- 172.21.15.43:0/2925653749 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2bdc0a9c00 con 0x7f2be4106570 2024-05-27T07:39:07.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.719+0000 7f2beac00700 1 -- 172.21.15.43:0/2925653749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2be4113500 con 0x7f2be4106570 2024-05-27T07:39:07.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.720+0000 7f2be0c00700 1 -- 172.21.15.43:0/2925653749 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+6120 (secure 0 0 0) 0x7f2bdc0a9500 con 0x7f2be4106570 2024-05-27T07:39:07.718 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:39:07.718 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":147,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:37:32.859028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34403,24484,34424,34484,34448]},"id":1}]} 2024-05-27T07:39:07.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 -- 172.21.15.43:0/2925653749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bc80b2a70 msgr2=0x7f2bc80b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bc80b2a70 0x7f2bc80b4f80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2be407c0a0 tx=0x7f2bcc00b540 comp rx=0 tx=0).stop 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 -- 172.21.15.43:0/2925653749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2be4106570 msgr2=0x7f2be41121a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2be4106570 0x7f2be41121a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00d980 tx=0x7f2bdc00dc90 comp rx=0 tx=0).stop 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 -- 172.21.15.43:0/2925653749 shutdown_connections 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f2bc80b2a70 0x7f2bc80b4f80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2be410c5b0 0x7f2be41a5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2be4106570 0x7f2be41121a0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 --2- 172.21.15.43:0/2925653749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2be4106ec0 0x7f2be4112700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 -- 172.21.15.43:0/2925653749 >> 172.21.15.43:0/2925653749 conn(0x7f2be4102070 msgr2=0x7f2be4105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 -- 172.21.15.43:0/2925653749 shutdown_connections 2024-05-27T07:39:07.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:07.724+0000 7f2beac00700 1 -- 172.21.15.43:0/2925653749 wait complete. 2024-05-27T07:39:07.724 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 147 2024-05-27T07:39:07.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 72919 5.84 MB/sec execute 548 sec latency 90.887 ms 2024-05-27T07:39:08.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:39:08.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 73020 5.84 MB/sec execute 549 sec latency 88.806 ms 2024-05-27T07:39:08.963 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:08 smithi043 ceph-mon[31531]: pgmap v846: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.0 MiB/s wr, 211 op/s 2024-05-27T07:39:08.963 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:08 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2925653749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:08.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:08 smithi179 ceph-mon[35888]: pgmap v846: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.0 MiB/s wr, 211 op/s 2024-05-27T07:39:08.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:08 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2925653749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:09.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:08 smithi132 ceph-mon[35283]: pgmap v846: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.0 MiB/s wr, 211 op/s 2024-05-27T07:39:09.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:08 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2925653749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:39:09.197 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:09.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 73132 5.85 MB/sec execute 550 sec latency 97.271 ms 2024-05-27T07:39:10.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 73233 5.84 MB/sec execute 551 sec latency 84.949 ms 2024-05-27T07:39:10.958 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:10 smithi043 ceph-mon[31531]: pgmap v847: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.9 MiB/s wr, 108 op/s 2024-05-27T07:39:10.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:10 smithi179 ceph-mon[35888]: pgmap v847: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.9 MiB/s wr, 108 op/s 2024-05-27T07:39:11.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:10 smithi132 ceph-mon[35283]: pgmap v847: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.9 MiB/s wr, 108 op/s 2024-05-27T07:39:11.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 73331 5.84 MB/sec execute 552 sec latency 88.528 ms 2024-05-27T07:39:12.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 73414 5.83 MB/sec execute 553 sec latency 274.823 ms 2024-05-27T07:39:12.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:12 smithi132 ceph-mon[35283]: pgmap v848: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:12.847 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:12 smithi132 ceph-mon[35283]: osdmap e161: 12 total, 12 up, 12 in 2024-05-27T07:39:12.908 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:12 smithi043 ceph-mon[31531]: pgmap v848: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:12.908 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:12 smithi043 ceph-mon[31531]: osdmap e161: 12 total, 12 up, 12 in 2024-05-27T07:39:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:12 smithi179 ceph-mon[35888]: pgmap v848: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:12.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:12 smithi179 ceph-mon[35888]: osdmap e161: 12 total, 12 up, 12 in 2024-05-27T07:39:13.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.730+0000 7fa9c3400700 1 -- 172.21.15.43:0/2755049712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9bc108690 msgr2=0x7fa9bc108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:13.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.730+0000 7fa9c3400700 1 --2- 172.21.15.43:0/2755049712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9bc108690 0x7fa9bc108b60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400b0a0 tx=0x7fa9b400b3b0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.730+0000 7fa9c3400700 1 -- 172.21.15.43:0/2755049712 shutdown_connections 2024-05-27T07:39:13.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.730+0000 7fa9c3400700 1 --2- 172.21.15.43:0/2755049712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9bc108690 0x7fa9bc108b60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.730+0000 7fa9c3400700 1 --2- 172.21.15.43:0/2755049712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9bc103240 0x7fa9bc103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.730+0000 7fa9c3400700 1 --2- 172.21.15.43:0/2755049712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9bc103b90 0x7fa9bc1080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.730+0000 7fa9c3400700 1 -- 172.21.15.43:0/2755049712 >> 172.21.15.43:0/2755049712 conn(0x7fa9bc0fedb0 msgr2=0x7fa9bc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:13.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.730+0000 7fa9c3400700 1 -- 172.21.15.43:0/2755049712 shutdown_connections 2024-05-27T07:39:13.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.730+0000 7fa9c3400700 1 -- 172.21.15.43:0/2755049712 wait complete. 2024-05-27T07:39:13.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.732+0000 7fa9c3400700 1 Processor -- start 2024-05-27T07:39:13.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.732+0000 7fa9c3400700 1 -- start start 2024-05-27T07:39:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.733+0000 7fa9c3400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9bc103240 0x7fa9bc1a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.733+0000 7fa9c3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9bc103b90 0x7fa9bc1a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.733+0000 7fa9c3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9bc108690 0x7fa9bc1a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.733+0000 7fa9c3400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc11a910 con 0x7fa9bc103b90 2024-05-27T07:39:13.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.733+0000 7fa9c3400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc11a790 con 0x7fa9bc108690 2024-05-27T07:39:13.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.733+0000 7fa9c3400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc11aa90 con 0x7fa9bc103240 2024-05-27T07:39:13.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.733+0000 7fa9bbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9bc103b90 0x7fa9bc1a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.733+0000 7fa9bbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9bc103b90 0x7fa9bc1a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56582/0 (socket says 172.21.15.43:56582) 2024-05-27T07:39:13.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.733+0000 7fa9bbe00700 1 -- 172.21.15.43:0/2168649778 learned_addr learned my addr 172.21.15.43:0/2168649778 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.734+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9bc103240 0x7fa9bc1a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.734+0000 7fa9c1400700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9bc108690 0x7fa9bc1a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.734+0000 7fa9bbe00700 1 -- 172.21.15.43:0/2168649778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9bc103240 msgr2=0x7fa9bc1a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.734+0000 7fa9bbe00700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9bc103240 0x7fa9bc1a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.734+0000 7fa9bbe00700 1 -- 172.21.15.43:0/2168649778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9bc108690 msgr2=0x7fa9bc1a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.734+0000 7fa9bbe00700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9bc108690 0x7fa9bc1a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.734+0000 7fa9bbe00700 1 -- 172.21.15.43:0/2168649778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc1a6170 con 0x7fa9bc103b90 2024-05-27T07:39:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.734+0000 7fa9c1400700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9bc108690 0x7fa9bc1a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.734+0000 7fa9bbe00700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9bc103b90 0x7fa9bc1a1840 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000d950 tx=0x7fa9b000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:13.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.735+0000 7fa9b9600700 1 -- 172.21.15.43:0/2168649778 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0016070 con 0x7fa9bc103b90 2024-05-27T07:39:13.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.735+0000 7fa9b9600700 1 -- 172.21.15.43:0/2168649778 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9b0010460 con 0x7fa9bc103b90 2024-05-27T07:39:13.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.735+0000 7fa9c3400700 1 -- 172.21.15.43:0/2168649778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9bc1a6460 con 0x7fa9bc103b90 2024-05-27T07:39:13.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.735+0000 7fa9b9600700 1 -- 172.21.15.43:0/2168649778 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b000f620 con 0x7fa9bc103b90 2024-05-27T07:39:13.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.735+0000 7fa9c3400700 1 -- 172.21.15.43:0/2168649778 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9bc1add80 con 0x7fa9bc103b90 2024-05-27T07:39:13.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.741+0000 7fa9b9600700 1 -- 172.21.15.43:0/2168649778 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fa9b0019070 con 0x7fa9bc103b90 2024-05-27T07:39:13.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.742+0000 7fa9b9600700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa9a40b2af0 0x7fa9a40b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.743+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa9a40b2af0 0x7fa9a40b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.743+0000 7fa9b9600700 1 -- 172.21.15.43:0/2168649778 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fa9b00e2590 con 0x7fa9bc103b90 2024-05-27T07:39:13.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.744+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa9a40b2af0 0x7fa9a40b5000 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009bf0 tx=0x7fa9ac009260 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:13.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.760+0000 7fa9d1eafb80 1 Processor -- start 2024-05-27T07:39:13.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.760+0000 7fa9d1eafb80 1 -- start start 2024-05-27T07:39:13.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.761+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478d89be40 0x56478da207f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.761+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478da20d30 0x56478da1a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.761+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da1b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.761+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56478d948e40 con 0x56478da1adb0 2024-05-27T07:39:13.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.761+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56478d9389e0 con 0x56478da20d30 2024-05-27T07:39:13.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.761+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56478d93ca20 con 0x56478d89be40 2024-05-27T07:39:13.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.762+0000 7fa9c1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da1b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.762+0000 7fa9c1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da1b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56590/0 (socket says 172.21.15.43:56590) 2024-05-27T07:39:13.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.762+0000 7fa9c1400700 1 -- 172.21.15.43:0/3332847208 learned_addr learned my addr 172.21.15.43:0/3332847208 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:13.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.763+0000 7fa9c1400700 1 -- 172.21.15.43:0/3332847208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478d89be40 msgr2=0x56478da207f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:13.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.763+0000 7fa9c1400700 1 --2- 172.21.15.43:0/3332847208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478d89be40 0x56478da207f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.763+0000 7fa9c1400700 1 -- 172.21.15.43:0/3332847208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478da20d30 msgr2=0x56478da1a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:13.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.763+0000 7fa9c1400700 1 --2- 172.21.15.43:0/3332847208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478da20d30 0x56478da1a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.763+0000 7fa9c1400700 1 -- 172.21.15.43:0/3332847208 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56478d901420 con 0x56478da1adb0 2024-05-27T07:39:13.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.764+0000 7fa9c1400700 1 --2- 172.21.15.43:0/3332847208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da1b1c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009d90 tx=0x7fa9b4009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78df873952d1475a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:13.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.764+0000 7fa9aa000700 1 -- 172.21.15.43:0/3332847208 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b400bcb0 con 0x56478da1adb0 2024-05-27T07:39:13.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.765+0000 7fa9aa000700 1 -- 172.21.15.43:0/3332847208 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9b400be50 con 0x56478da1adb0 2024-05-27T07:39:13.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.765+0000 7fa9aa000700 1 -- 172.21.15.43:0/3332847208 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b401c440 con 0x56478da1adb0 2024-05-27T07:39:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.766+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/3332847208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 msgr2=0x56478da1b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.766+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/3332847208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da1b1c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009d90 tx=0x7fa9b4009dc0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.767+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/3332847208 shutdown_connections 2024-05-27T07:39:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.767+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/3332847208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da1b1c0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.767+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/3332847208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478d89be40 0x56478da207f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.767+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/3332847208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478da20d30 0x56478da1a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.767+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/3332847208 >> 172.21.15.43:0/3332847208 conn(0x56478d8fbfc0 msgr2=0x56478da158d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.767+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/3332847208 shutdown_connections 2024-05-27T07:39:13.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.767+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/3332847208 wait complete. 2024-05-27T07:39:13.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.768+0000 7fa9d1eafb80 1 -- start start 2024-05-27T07:39:13.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.768+0000 7fa9d1eafb80 1 Processor -- start 2024-05-27T07:39:13.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.769+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478d89be40 0x56478da24620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.769+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da24b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.769+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478da20d30 0x56478da250a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.769+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56478d83efa0 con 0x56478da1adb0 2024-05-27T07:39:13.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.769+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56478d8d9760 con 0x56478da20d30 2024-05-27T07:39:13.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.769+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56478da277a0 con 0x56478d89be40 2024-05-27T07:39:13.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.769+0000 7fa9d1eafb80 10 client.0 _ll_register_callbacks cb 0x56478d8e2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:13.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.769+0000 7fa9bbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da24b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.770+0000 7fa9c1400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478da20d30 0x56478da250a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.770+0000 7fa9bbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da24b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56604/0 (socket says 172.21.15.43:56604) 2024-05-27T07:39:13.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.770+0000 7fa9bbe00700 1 -- 172.21.15.43:0/297253978 learned_addr learned my addr 172.21.15.43:0/297253978 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:13.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.770+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478d89be40 0x56478da24620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.771+0000 7fa9c1400700 1 -- 172.21.15.43:0/297253978 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478d89be40 msgr2=0x56478da24620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:13.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.771+0000 7fa9c1400700 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478d89be40 0x56478da24620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.771+0000 7fa9c1400700 1 -- 172.21.15.43:0/297253978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 msgr2=0x56478da24b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:13.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.771+0000 7fa9c1400700 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da24b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.771+0000 7fa9c1400700 1 -- 172.21.15.43:0/297253978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56478d8888d0 con 0x56478da20d30 2024-05-27T07:39:13.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.771+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478d89be40 0x56478da24620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:13.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.771+0000 7fa9c1400700 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478da20d30 0x56478da250a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400b3f0 tx=0x7fa9b40077f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c43c1ba97b9722c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:13.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.771+0000 7fa99ea00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:39:13.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.772+0000 7fa99ea00700 1 -- 172.21.15.43:0/297253978 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b402a070 con 0x56478da20d30 2024-05-27T07:39:13.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.772+0000 7fa99ea00700 1 -- 172.21.15.43:0/297253978 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fa9b401c440 con 0x56478da20d30 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.772+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/297253978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9b400be50 con 0x56478da20d30 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.772+0000 7fa99ea00700 1 -- 172.21.15.43:0/297253978 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fa9b4033050 con 0x56478da20d30 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.773+0000 7fa9d1eafb80 10 client.37901 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.773+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/297253978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x56478d909060 con 0x56478da20d30 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.773+0000 7fa99ea00700 1 -- 172.21.15.43:0/297253978 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9b400be50 con 0x56478da20d30 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.773+0000 7fa99ea00700 1 -- 172.21.15.43:0/297253978 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fa9b4011070 con 0x56478da20d30 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.773+0000 7fa9d1eafb80 10 client.37901 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.773+0000 7fa9d1eafb80 10 client.37901 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.773+0000 7fa9d1eafb80 20 client.37901 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.773+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56478da347d0 0x56478da36c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.774+0000 7fa9d1eafb80 4 client.37901 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:13.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.774+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/297253978 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56478d94ad90 con 0x56478da347d0 2024-05-27T07:39:13.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.774+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56478da347d0 0x56478da36c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.775+0000 7fa99ea00700 10 client.37901 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:39:13.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.776+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56478da347d0 0x56478da36c30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c3cf00f987594df6 server_cookie=33a8b7e644e0d46b in_seq=0 out_seq=0 2024-05-27T07:39:13.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.781+0000 7fa99ea00700 1 -- 172.21.15.43:0/297253978 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x56478d94ad90 con 0x56478da347d0 2024-05-27T07:39:13.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.781+0000 7fa99ea00700 10 client.37901 handle_command_reply: tid=0 2024-05-27T07:39:13.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.781+0000 7fa9d1eafb80 1 client.37901 shutdown 2024-05-27T07:39:13.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/297253978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478da20d30 msgr2=0x56478da250a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:13.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478da20d30 0x56478da250a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400b3f0 tx=0x7fa9b40077f0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/297253978 shutdown_connections 2024-05-27T07:39:13.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56478da347d0 0x56478da36c30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478da20d30 0x56478da250a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478d89be40 0x56478da24620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/297253978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da1adb0 0x56478da24b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/297253978 >> 172.21.15.43:0/297253978 conn(0x56478d8fbfc0 msgr2=0x56478da15580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:13.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa99ea00700 0 client.37901 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:39:13.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/297253978 shutdown_connections 2024-05-27T07:39:13.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.782+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/297253978 wait complete. 2024-05-27T07:39:13.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.783+0000 7fa9d1eafb80 20 client.37901 trim_cache size 0 max 16384 2024-05-27T07:39:13.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.812+0000 7fa9d1eafb80 1 Processor -- start 2024-05-27T07:39:13.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.813+0000 7fa9d1eafb80 1 -- start start 2024-05-27T07:39:13.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.813+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da1a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.813+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478da1adb0 0x56478da1e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.813+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da22f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.813+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1b6eb0 con 0x56478da20d30 2024-05-27T07:39:13.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.813+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1b5370 con 0x56478d89be40 2024-05-27T07:39:13.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.813+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1b67e0 con 0x56478da1adb0 2024-05-27T07:39:13.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.814+0000 7fa9c0a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da1a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.814+0000 7fa9c0a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da1a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38724/0 (socket says 172.21.15.43:38724) 2024-05-27T07:39:13.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.814+0000 7fa9c0a00700 1 -- 172.21.15.43:0/1014115454 learned_addr learned my addr 172.21.15.43:0/1014115454 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:13.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.814+0000 7fa9bbe00700 1 --2- 172.21.15.43:0/1014115454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478da1adb0 0x56478da1e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.814+0000 7fa9c1400700 1 --2- 172.21.15.43:0/1014115454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da22f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.815+0000 7fa9c1400700 1 -- 172.21.15.43:0/1014115454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478da1adb0 msgr2=0x56478da1e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:13.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.815+0000 7fa9c1400700 1 --2- 172.21.15.43:0/1014115454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478da1adb0 0x56478da1e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.815+0000 7fa9c1400700 1 -- 172.21.15.43:0/1014115454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 msgr2=0x56478da1a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:13.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.815+0000 7fa9c1400700 1 --2- 172.21.15.43:0/1014115454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da1a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.815+0000 7fa9c1400700 1 -- 172.21.15.43:0/1014115454 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56478da22b90 con 0x56478da20d30 2024-05-27T07:39:13.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.815+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/1014115454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da1a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.815+0000 7fa9c1400700 1 --2- 172.21.15.43:0/1014115454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da22f10 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40126d0 tx=0x7fa9b40098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef837f6886391586 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.815+0000 7fa9a9600700 1 -- 172.21.15.43:0/1014115454 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b402a040 con 0x56478da20d30 2024-05-27T07:39:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.816+0000 7fa9a9600700 1 -- 172.21.15.43:0/1014115454 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9b4007da0 con 0x56478da20d30 2024-05-27T07:39:13.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.816+0000 7fa9a9600700 1 -- 172.21.15.43:0/1014115454 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4005300 con 0x56478da20d30 2024-05-27T07:39:13.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.818+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/1014115454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 msgr2=0x56478da22f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:13.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.818+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/1014115454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da22f10 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40126d0 tx=0x7fa9b40098a0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.819+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/1014115454 shutdown_connections 2024-05-27T07:39:13.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.819+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/1014115454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da22f10 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.819+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/1014115454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da1a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.819+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/1014115454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478da1adb0 0x56478da1e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.819+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/1014115454 >> 172.21.15.43:0/1014115454 conn(0x56478d8fbfc0 msgr2=0x56478da1e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:13.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.819+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/1014115454 shutdown_connections 2024-05-27T07:39:13.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.819+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/1014115454 wait complete. 2024-05-27T07:39:13.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.819+0000 7fa9d1eafb80 1 -- start start 2024-05-27T07:39:13.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.820+0000 7fa9d1eafb80 1 Processor -- start 2024-05-27T07:39:13.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.821+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da3c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.821+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478da1adb0 0x56478da3cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.821+0000 7fa9d1eafb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da2cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.821+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1b6110 con 0x56478da20d30 2024-05-27T07:39:13.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.821+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1b5a40 con 0x56478d89be40 2024-05-27T07:39:13.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.821+0000 7fa9d1eafb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x56478da147c0 con 0x56478da1adb0 2024-05-27T07:39:13.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.821+0000 7fa9d1eafb80 10 client.0 _ll_register_callbacks cb 0x56478d8e2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:13.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.822+0000 7fa9c0a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da3c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.822+0000 7fa9c0a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da3c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38732/0 (socket says 172.21.15.43:38732) 2024-05-27T07:39:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.822+0000 7fa9c1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da2cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.822+0000 7fa9c1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da2cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56620/0 (socket says 172.21.15.43:56620) 2024-05-27T07:39:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.822+0000 7fa9c1400700 1 -- 172.21.15.43:0/4120148577 learned_addr learned my addr 172.21.15.43:0/4120148577 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.822+0000 7fa9bbe00700 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478da1adb0 0x56478da3cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.822+0000 7fa9c1400700 1 -- 172.21.15.43:0/4120148577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478da1adb0 msgr2=0x56478da3cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:13.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.822+0000 7fa9c1400700 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478da1adb0 0x56478da3cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.822+0000 7fa9c1400700 1 -- 172.21.15.43:0/4120148577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 msgr2=0x56478da3c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.823+0000 7fa9c1400700 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da3c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.823+0000 7fa9c1400700 1 -- 172.21.15.43:0/4120148577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56478da2d250 con 0x56478da20d30 2024-05-27T07:39:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.823+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da3c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.823+0000 7fa9c1400700 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da2cab0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4007a80 tx=0x7fa9b4000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f23ab2e010f825 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:13.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.823+0000 7fa99ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:39:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.824+0000 7fa99ea00700 1 -- 172.21.15.43:0/4120148577 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b402a040 con 0x56478da20d30 2024-05-27T07:39:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.824+0000 7fa99ea00700 1 -- 172.21.15.43:0/4120148577 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fa9b40129c0 con 0x56478da20d30 2024-05-27T07:39:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.824+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/4120148577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9b4007da0 con 0x56478da20d30 2024-05-27T07:39:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.825+0000 7fa99ea00700 1 -- 172.21.15.43:0/4120148577 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fa9b4051050 con 0x56478da20d30 2024-05-27T07:39:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.825+0000 7fa9d1eafb80 10 client.18522 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.825+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/4120148577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x56478da3c050 con 0x56478da20d30 2024-05-27T07:39:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.825+0000 7fa99ea00700 1 -- 172.21.15.43:0/4120148577 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa9b4007da0 con 0x56478da20d30 2024-05-27T07:39:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.826+0000 7fa99ea00700 1 -- 172.21.15.43:0/4120148577 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fa9b4011040 con 0x56478da20d30 2024-05-27T07:39:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.826+0000 7fa9d1eafb80 10 client.18522 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.826+0000 7fa9d1eafb80 10 client.18522 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:39:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.826+0000 7fa9d1eafb80 20 client.18522 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.826+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56478daaf4b0 0x56478dab18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.826+0000 7fa9d1eafb80 4 client.18522 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:39:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.826+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/4120148577 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56478d944570 con 0x56478daaf4b0 2024-05-27T07:39:13.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.828+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56478daaf4b0 0x56478dab18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.829+0000 7fa9c0a00700 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56478daaf4b0 0x56478dab18d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=87684469ff285b14 server_cookie=531898201af5b515 in_seq=0 out_seq=0 2024-05-27T07:39:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.829+0000 7fa99ea00700 10 client.18522 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:39:13.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.832+0000 7fa99ea00700 1 -- 172.21.15.43:0/4120148577 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56478d944570 con 0x56478daaf4b0 2024-05-27T07:39:13.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.832+0000 7fa99ea00700 10 client.18522 handle_command_reply: tid=0 2024-05-27T07:39:13.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.832+0000 7fa9d1eafb80 1 client.18522 shutdown 2024-05-27T07:39:13.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.832+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/4120148577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 msgr2=0x56478da2cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:13.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.832+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da2cab0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4007a80 tx=0x7fa9b4000f40 comp rx=0 tx=0).stop 2024-05-27T07:39:13.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.833+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/4120148577 shutdown_connections 2024-05-27T07:39:13.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.833+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x56478daaf4b0 0x56478dab18d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.833+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56478da20d30 0x56478da2cab0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.833+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56478d89be40 0x56478da3c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.833+0000 7fa9d1eafb80 1 --2- 172.21.15.43:0/4120148577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56478da1adb0 0x56478da3cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.833+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/4120148577 >> 172.21.15.43:0/4120148577 conn(0x56478d8fbfc0 msgr2=0x56478da24400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:13.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.833+0000 7fa99ea00700 0 client.18522 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:39:13.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.834+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/4120148577 shutdown_connections 2024-05-27T07:39:13.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.834+0000 7fa9d1eafb80 1 -- 172.21.15.43:0/4120148577 wait complete. 2024-05-27T07:39:13.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.834+0000 7fa9d1eafb80 20 client.18522 trim_cache size 0 max 16384 2024-05-27T07:39:13.833 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:39:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.835+0000 7fa9c3400700 1 -- 172.21.15.43:0/2168649778 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa9a40b2af0 msgr2=0x7fa9a40b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.835+0000 7fa9c3400700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa9a40b2af0 0x7fa9a40b5000 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009bf0 tx=0x7fa9ac009260 comp rx=0 tx=0).stop 2024-05-27T07:39:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.835+0000 7fa9c3400700 1 -- 172.21.15.43:0/2168649778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9bc103b90 msgr2=0x7fa9bc1a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:13.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.835+0000 7fa9c3400700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9bc103b90 0x7fa9bc1a1840 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000d950 tx=0x7fa9b000dd10 comp rx=0 tx=0).stop 2024-05-27T07:39:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.835+0000 7fa9c3400700 1 -- 172.21.15.43:0/2168649778 shutdown_connections 2024-05-27T07:39:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.835+0000 7fa9c3400700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fa9a40b2af0 0x7fa9a40b5000 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.835+0000 7fa9c3400700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9bc108690 0x7fa9bc1a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.835+0000 7fa9c3400700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9bc103240 0x7fa9bc1a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.835+0000 7fa9c3400700 1 --2- 172.21.15.43:0/2168649778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa9bc103b90 0x7fa9bc1a1840 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.835+0000 7fa9c3400700 1 -- 172.21.15.43:0/2168649778 >> 172.21.15.43:0/2168649778 conn(0x7fa9bc0fedb0 msgr2=0x7fa9bc105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.836+0000 7fa9c3400700 1 -- 172.21.15.43:0/2168649778 shutdown_connections 2024-05-27T07:39:13.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:13.836+0000 7fa9c3400700 1 -- 172.21.15.43:0/2168649778 wait complete. 2024-05-27T07:39:13.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 73512 5.84 MB/sec execute 554 sec latency 238.723 ms 2024-05-27T07:39:14.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:14 smithi043 ceph-mon[31531]: pgmap v850: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:14.940 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:14 smithi132 ceph-mon[35283]: pgmap v850: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:14.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:39:14.942 INFO:tasks.workunit.client.0.smithi043.stdout: 10 73604 5.83 MB/sec execute 555 sec latency 267.068 ms 2024-05-27T07:39:14.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:14 smithi179 ceph-mon[35888]: pgmap v850: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:15.554 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:15.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 73701 5.82 MB/sec execute 556 sec latency 124.466 ms 2024-05-27T07:39:16.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 73834 5.83 MB/sec execute 557 sec latency 86.413 ms 2024-05-27T07:39:16.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:16 smithi043 ceph-mon[31531]: pgmap v851: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:16.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:16 smithi179 ceph-mon[35888]: pgmap v851: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:17.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:16 smithi132 ceph-mon[35283]: pgmap v851: 129 pgs: 129 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:17.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 73935 5.82 MB/sec execute 558 sec latency 147.865 ms 2024-05-27T07:39:18.838 INFO:tasks.workunit.client.0.smithi043.stdout: 10 74034 5.81 MB/sec execute 559 sec latency 141.527 ms 2024-05-27T07:39:18.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:18 smithi043 ceph-mon[31531]: pgmap v852: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:18.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:18 smithi179 ceph-mon[35888]: pgmap v852: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:19.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:18 smithi132 ceph-mon[35283]: pgmap v852: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:19.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 74154 5.82 MB/sec execute 560 sec latency 96.442 ms 2024-05-27T07:39:20.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 74266 5.83 MB/sec execute 561 sec latency 80.227 ms 2024-05-27T07:39:20.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.923+0000 7f4947a00700 1 -- 172.21.15.43:0/3082383846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4940103a40 msgr2=0x7f4940110760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:20.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.923+0000 7f4947a00700 1 --2- 172.21.15.43:0/3082383846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4940103a40 0x7f4940110760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4928009990 tx=0x7f4928009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.923+0000 7f4947a00700 1 -- 172.21.15.43:0/3082383846 shutdown_connections 2024-05-27T07:39:20.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.923+0000 7f4947a00700 1 --2- 172.21.15.43:0/3082383846 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4940110ca0 0x7f49401130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.923+0000 7f4947a00700 1 --2- 172.21.15.43:0/3082383846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49401030f0 0x7f4940103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.923+0000 7f4947a00700 1 --2- 172.21.15.43:0/3082383846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4940103a40 0x7f4940110760 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4928009990 tx=0x7f4928009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.923+0000 7f4947a00700 1 -- 172.21.15.43:0/3082383846 >> 172.21.15.43:0/3082383846 conn(0x7f49400fedb0 msgr2=0x7f49401011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:20.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.924+0000 7f4947a00700 1 -- 172.21.15.43:0/3082383846 shutdown_connections 2024-05-27T07:39:20.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.924+0000 7f4947a00700 1 -- 172.21.15.43:0/3082383846 wait complete. 2024-05-27T07:39:20.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:20 smithi043 ceph-mon[31531]: pgmap v853: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:20.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.926+0000 7f4947a00700 1 Processor -- start 2024-05-27T07:39:20.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.926+0000 7f4947a00700 1 -- start start 2024-05-27T07:39:20.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.926+0000 7f4947a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49401030f0 0x7f494006f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.927+0000 7f4947a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4940110ca0 0x7f494006fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.927+0000 7f4947a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49400755c0 0x7f49401b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.927+0000 7f4947a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f494006d930 con 0x7f49400755c0 2024-05-27T07:39:20.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.927+0000 7f4947a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f494006d7b0 con 0x7f49401030f0 2024-05-27T07:39:20.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.927+0000 7f4947a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f494006dab0 con 0x7f4940110ca0 2024-05-27T07:39:20.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.928+0000 7f4945000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49401030f0 0x7f494006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.928+0000 7f4945000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49401030f0 0x7f494006f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38888/0 (socket says 172.21.15.43:38888) 2024-05-27T07:39:20.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.928+0000 7f493fe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4940110ca0 0x7f494006fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.928+0000 7f4945000700 1 -- 172.21.15.43:0/3761351652 learned_addr learned my addr 172.21.15.43:0/3761351652 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:20.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.928+0000 7f4945a00700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49400755c0 0x7f49401b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.929+0000 7f493fe00700 1 -- 172.21.15.43:0/3761351652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49401030f0 msgr2=0x7f494006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:20.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.929+0000 7f493fe00700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49401030f0 0x7f494006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.929+0000 7f493fe00700 1 -- 172.21.15.43:0/3761351652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49400755c0 msgr2=0x7f49401b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:20.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.929+0000 7f493fe00700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49400755c0 0x7f49401b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.929+0000 7f493fe00700 1 -- 172.21.15.43:0/3761351652 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49401b20d0 con 0x7f4940110ca0 2024-05-27T07:39:20.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.930+0000 7f4945000700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49401030f0 0x7f494006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:20.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.930+0000 7f493fe00700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4940110ca0 0x7f494006fa80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f494006b890 tx=0x7f49280057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:20.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.930+0000 7f493d600700 1 -- 172.21.15.43:0/3761351652 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4928026070 con 0x7f4940110ca0 2024-05-27T07:39:20.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.930+0000 7f4947a00700 1 -- 172.21.15.43:0/3761351652 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4940105300 con 0x7f4940110ca0 2024-05-27T07:39:20.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.930+0000 7f4947a00700 1 -- 172.21.15.43:0/3761351652 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4940105760 con 0x7f4940110ca0 2024-05-27T07:39:20.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.931+0000 7f493d600700 1 -- 172.21.15.43:0/3761351652 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f49280095c0 con 0x7f4940110ca0 2024-05-27T07:39:20.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.931+0000 7f493d600700 1 -- 172.21.15.43:0/3761351652 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4928021d60 con 0x7f4940110ca0 2024-05-27T07:39:20.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.937+0000 7f493d600700 1 -- 172.21.15.43:0/3761351652 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f4928009d10 con 0x7f4940110ca0 2024-05-27T07:39:20.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.940+0000 7f493d600700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f49240b2b20 0x7f49240b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.940+0000 7f493d600700 1 -- 172.21.15.43:0/3761351652 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7f49280e3370 con 0x7f4940110ca0 2024-05-27T07:39:20.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.940+0000 7f4945000700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f49240b2b20 0x7f49240b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.941+0000 7f4945000700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f49240b2b20 0x7f49240b5030 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4938009480 tx=0x7f493800b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:20.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:20 smithi179 ceph-mon[35888]: pgmap v853: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:20.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.969+0000 7f4956529b80 1 Processor -- start 2024-05-27T07:39:20.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.969+0000 7f4956529b80 1 -- start start 2024-05-27T07:39:20.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.969+0000 7f4956529b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3bb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.969+0000 7f4956529b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3bb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.969+0000 7f4956529b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3c8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.969+0000 7f4956529b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c13c2e9e40 con 0x55c13c3bbe70 2024-05-27T07:39:20.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.969+0000 7f4956529b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c13c2d99e0 con 0x55c13c23ce40 2024-05-27T07:39:20.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.969+0000 7f4956529b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c13c2dda20 con 0x55c13c3289f0 2024-05-27T07:39:20.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.970+0000 7f4945a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.970+0000 7f4945a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3c8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49390/0 (socket says 172.21.15.43:49390) 2024-05-27T07:39:20.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.970+0000 7f4945a00700 1 -- 172.21.15.43:0/477757917 learned_addr learned my addr 172.21.15.43:0/477757917 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:20.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.970+0000 7f4945000700 1 --2- 172.21.15.43:0/477757917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3bb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.970+0000 7f4945a00700 1 -- 172.21.15.43:0/477757917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 msgr2=0x55c13c3bb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:20.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.970+0000 7f4945a00700 1 --2- 172.21.15.43:0/477757917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3bb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.970+0000 7f4945a00700 1 -- 172.21.15.43:0/477757917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 msgr2=0x55c13c3bb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:20.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.970+0000 7f4945a00700 1 --2- 172.21.15.43:0/477757917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3bb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.970+0000 7f4945a00700 1 -- 172.21.15.43:0/477757917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c13c2a2420 con 0x55c13c3bbe70 2024-05-27T07:39:20.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.971+0000 7f4945a00700 1 --2- 172.21.15.43:0/477757917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3c8110 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f49300046a0 tx=0x7f493000b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c81f6e03ee7a3e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:20.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.972+0000 7f4936000700 1 -- 172.21.15.43:0/477757917 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930004a30 con 0x55c13c3bbe70 2024-05-27T07:39:20.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.972+0000 7f4936000700 1 -- 172.21.15.43:0/477757917 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4930015070 con 0x55c13c3bbe70 2024-05-27T07:39:20.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.972+0000 7f4936000700 1 -- 172.21.15.43:0/477757917 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930003ac0 con 0x55c13c3bbe70 2024-05-27T07:39:20.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.973+0000 7f4956529b80 1 -- 172.21.15.43:0/477757917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 msgr2=0x55c13c3c8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:20.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.973+0000 7f4956529b80 1 --2- 172.21.15.43:0/477757917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3c8110 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f49300046a0 tx=0x7f493000b970 comp rx=0 tx=0).stop 2024-05-27T07:39:20.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.973+0000 7f4956529b80 1 -- 172.21.15.43:0/477757917 shutdown_connections 2024-05-27T07:39:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.973+0000 7f4956529b80 1 --2- 172.21.15.43:0/477757917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3c8110 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.973+0000 7f4956529b80 1 --2- 172.21.15.43:0/477757917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3bb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.973+0000 7f4956529b80 1 --2- 172.21.15.43:0/477757917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3bb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.973+0000 7f4956529b80 1 -- 172.21.15.43:0/477757917 >> 172.21.15.43:0/477757917 conn(0x55c13c2d6390 msgr2=0x55c13c3b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.974+0000 7f4956529b80 1 -- 172.21.15.43:0/477757917 shutdown_connections 2024-05-27T07:39:20.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.974+0000 7f4956529b80 1 -- 172.21.15.43:0/477757917 wait complete. 2024-05-27T07:39:20.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.975+0000 7f4956529b80 1 -- start start 2024-05-27T07:39:20.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.975+0000 7f4956529b80 1 Processor -- start 2024-05-27T07:39:20.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.976+0000 7f4956529b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c23ce40 0x55c13c3c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.976+0000 7f4956529b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3289f0 0x55c13c3c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.976+0000 7f4956529b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c3bbe70 0x55c13c3c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.976+0000 7f4956529b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c13c1dffa0 con 0x55c13c3289f0 2024-05-27T07:39:20.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.976+0000 7f4956529b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c13c27a760 con 0x55c13c3bbe70 2024-05-27T07:39:20.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.976+0000 7f4956529b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c13c3c8b80 con 0x55c13c23ce40 2024-05-27T07:39:20.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.976+0000 7f4956529b80 10 client.0 _ll_register_callbacks cb 0x55c13c283c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:20.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.977+0000 7f4945a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c3bbe70 0x55c13c3c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.977+0000 7f4945a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c3bbe70 0x55c13c3c60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38910/0 (socket says 172.21.15.43:38910) 2024-05-27T07:39:20.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.977+0000 7f4945a00700 1 -- 172.21.15.43:0/633083123 learned_addr learned my addr 172.21.15.43:0/633083123 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:20.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.977+0000 7f4945000700 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c23ce40 0x55c13c3c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.977+0000 7f493fe00700 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3289f0 0x55c13c3c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.978+0000 7f4945a00700 1 -- 172.21.15.43:0/633083123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c23ce40 msgr2=0x55c13c3c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:20.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.978+0000 7f4945a00700 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c23ce40 0x55c13c3c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.978+0000 7f4945a00700 1 -- 172.21.15.43:0/633083123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3289f0 msgr2=0x55c13c3c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:20.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.978+0000 7f4945a00700 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3289f0 0x55c13c3c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.978+0000 7f4945a00700 1 -- 172.21.15.43:0/633083123 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c13c2298d0 con 0x55c13c3bbe70 2024-05-27T07:39:20.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.979+0000 7f4945a00700 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c3bbe70 0x55c13c3c60e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4930004c10 tx=0x7f4930004c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90a44d75e3277b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:20.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.979+0000 7f492ea00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:39:20.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.979+0000 7f492ea00700 1 -- 172.21.15.43:0/633083123 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930028060 con 0x55c13c3bbe70 2024-05-27T07:39:20.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.979+0000 7f4956529b80 1 -- 172.21.15.43:0/633083123 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4930015070 con 0x55c13c3bbe70 2024-05-27T07:39:20.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.979+0000 7f492ea00700 1 -- 172.21.15.43:0/633083123 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7f4930003c50 con 0x55c13c3bbe70 2024-05-27T07:39:20.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.980+0000 7f492ea00700 1 -- 172.21.15.43:0/633083123 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f493002f050 con 0x55c13c3bbe70 2024-05-27T07:39:20.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.980+0000 7f4956529b80 10 client.37913 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:20.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.980+0000 7f4956529b80 1 -- 172.21.15.43:0/633083123 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55c13c2aa060 con 0x55c13c3bbe70 2024-05-27T07:39:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.980+0000 7f492ea00700 1 -- 172.21.15.43:0/633083123 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4930015070 con 0x55c13c3bbe70 2024-05-27T07:39:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.981+0000 7f492ea00700 1 -- 172.21.15.43:0/633083123 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f4930023690 con 0x55c13c3bbe70 2024-05-27T07:39:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.981+0000 7f4956529b80 10 client.37913 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.981+0000 7f4956529b80 10 client.37913 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:39:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.981+0000 7f4956529b80 20 client.37913 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.981+0000 7f4956529b80 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c13c3d5dd0 0x55c13c3d8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.981+0000 7f4956529b80 4 client.37913 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:20.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.981+0000 7f4956529b80 1 -- 172.21.15.43:0/633083123 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c13c2ebd90 con 0x55c13c3d5dd0 2024-05-27T07:39:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.981+0000 7f4945000700 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c13c3d5dd0 0x55c13c3d8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.982+0000 7f4945000700 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c13c3d5dd0 0x55c13c3d8280 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b8216253cefa3258 server_cookie=2a6698e541e60ffc in_seq=0 out_seq=0 2024-05-27T07:39:20.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.982+0000 7f492ea00700 10 client.37913 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:39:20.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.984+0000 7f492ea00700 1 -- 172.21.15.43:0/633083123 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c13c2ebd90 con 0x55c13c3d5dd0 2024-05-27T07:39:20.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.984+0000 7f492ea00700 10 client.37913 handle_command_reply: tid=0 2024-05-27T07:39:20.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.984+0000 7f4956529b80 1 client.37913 shutdown 2024-05-27T07:39:20.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f4956529b80 1 -- 172.21.15.43:0/633083123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c3bbe70 msgr2=0x55c13c3c60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:20.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f4956529b80 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c3bbe70 0x55c13c3c60e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4930004c10 tx=0x7f4930004c40 comp rx=0 tx=0).stop 2024-05-27T07:39:20.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f4956529b80 1 -- 172.21.15.43:0/633083123 shutdown_connections 2024-05-27T07:39:20.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f4956529b80 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c13c3d5dd0 0x55c13c3d8280 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f4956529b80 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c3bbe70 0x55c13c3c60e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f4956529b80 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c23ce40 0x55c13c3c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f4956529b80 1 --2- 172.21.15.43:0/633083123 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3289f0 0x55c13c3c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:20.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f4956529b80 1 -- 172.21.15.43:0/633083123 >> 172.21.15.43:0/633083123 conn(0x55c13c2d6390 msgr2=0x55c13c3bb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:20.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f492ea00700 0 client.37913 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:39:20.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f4956529b80 1 -- 172.21.15.43:0/633083123 shutdown_connections 2024-05-27T07:39:20.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.985+0000 7f4956529b80 1 -- 172.21.15.43:0/633083123 wait complete. 2024-05-27T07:39:20.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:20.986+0000 7f4956529b80 20 client.37913 trim_cache size 0 max 16384 2024-05-27T07:39:21.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.013+0000 7f4956529b80 1 Processor -- start 2024-05-27T07:39:21.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.013+0000 7f4956529b80 1 -- start start 2024-05-27T07:39:21.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.013+0000 7f4956529b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:21.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.013+0000 7f4956529b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3c0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:21.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.013+0000 7f4956529b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:21.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.013+0000 7f4956529b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f491c00f250 con 0x55c13c3bbe70 2024-05-27T07:39:21.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.013+0000 7f4956529b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f491c00d710 con 0x55c13c23ce40 2024-05-27T07:39:21.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.013+0000 7f4956529b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f491c00eb80 con 0x55c13c3289f0 2024-05-27T07:39:21.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.014+0000 7f4945000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:21.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.014+0000 7f4945000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3c45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38926/0 (socket says 172.21.15.43:38926) 2024-05-27T07:39:21.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.014+0000 7f4945000700 1 -- 172.21.15.43:0/2856593383 learned_addr learned my addr 172.21.15.43:0/2856593383 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:21.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.014+0000 7f4945000700 1 -- 172.21.15.43:0/2856593383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 msgr2=0x55c13c3c0e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T07:39:21.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.014+0000 7f4945000700 1 --2- 172.21.15.43:0/2856593383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3c0e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.014+0000 7f4945000700 1 -- 172.21.15.43:0/2856593383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 msgr2=0x55c13c3d1c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:39:21.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.015+0000 7f4945a00700 1 --2- 172.21.15.43:0/2856593383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:21.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.015+0000 7f4945000700 1 --2- 172.21.15.43:0/2856593383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.015+0000 7f4945000700 1 -- 172.21.15.43:0/2856593383 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c13c3b3240 con 0x55c13c23ce40 2024-05-27T07:39:21.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.016+0000 7f4945000700 1 --2- 172.21.15.43:0/2856593383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3c45c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4938009fd0 tx=0x7f4938013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a34c9378e39bff06 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:21.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.016+0000 7f4935600700 1 -- 172.21.15.43:0/2856593383 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938030070 con 0x55c13c23ce40 2024-05-27T07:39:21.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.016+0000 7f4935600700 1 -- 172.21.15.43:0/2856593383 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f493801d040 con 0x55c13c23ce40 2024-05-27T07:39:21.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.016+0000 7f4935600700 1 -- 172.21.15.43:0/2856593383 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493801c040 con 0x55c13c23ce40 2024-05-27T07:39:21.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.017+0000 7f4956529b80 1 -- 172.21.15.43:0/2856593383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 msgr2=0x55c13c3c45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:21.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.017+0000 7f4956529b80 1 --2- 172.21.15.43:0/2856593383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3c45c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4938009fd0 tx=0x7f4938013000 comp rx=0 tx=0).stop 2024-05-27T07:39:21.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.018+0000 7f4956529b80 1 -- 172.21.15.43:0/2856593383 shutdown_connections 2024-05-27T07:39:21.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.018+0000 7f4956529b80 1 --2- 172.21.15.43:0/2856593383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.018+0000 7f4956529b80 1 --2- 172.21.15.43:0/2856593383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3c45c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.018+0000 7f4956529b80 1 --2- 172.21.15.43:0/2856593383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3c0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.018+0000 7f4956529b80 1 -- 172.21.15.43:0/2856593383 >> 172.21.15.43:0/2856593383 conn(0x55c13c2d6390 msgr2=0x55c13c3c5620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:21.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.018+0000 7f4956529b80 1 -- 172.21.15.43:0/2856593383 shutdown_connections 2024-05-27T07:39:21.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.018+0000 7f4956529b80 1 -- 172.21.15.43:0/2856593383 wait complete. 2024-05-27T07:39:21.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.019+0000 7f4956529b80 1 -- start start 2024-05-27T07:39:21.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.019+0000 7f4956529b80 1 Processor -- start 2024-05-27T07:39:21.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4956529b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3ce140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:21.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4956529b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:21.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4956529b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:21.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4956529b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f491c00e4b0 con 0x55c13c3bbe70 2024-05-27T07:39:21.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4956529b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f491c00dde0 con 0x55c13c23ce40 2024-05-27T07:39:21.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4956529b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c13c3c01f0 con 0x55c13c3289f0 2024-05-27T07:39:21.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4956529b80 10 client.0 _ll_register_callbacks cb 0x55c13c283c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:21.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4945000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:21.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4945000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3ce140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:38932/0 (socket says 172.21.15.43:38932) 2024-05-27T07:39:21.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4945000700 1 -- 172.21.15.43:0/4165524911 learned_addr learned my addr 172.21.15.43:0/4165524911 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:21.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f493fe00700 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:21.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.020+0000 7f4945a00700 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:21.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.021+0000 7f4945000700 1 -- 172.21.15.43:0/4165524911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 msgr2=0x55c13c3ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:21.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.021+0000 7f4945000700 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3ce680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.021+0000 7f4945000700 1 -- 172.21.15.43:0/4165524911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 msgr2=0x55c13c3bfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:21.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.021+0000 7f4945000700 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3bfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.021+0000 7f4945000700 1 -- 172.21.15.43:0/4165524911 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c13c3c03f0 con 0x55c13c23ce40 2024-05-27T07:39:21.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.021+0000 7f4945a00700 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3bfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:21.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.021+0000 7f492ea00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:39:21.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.021+0000 7f4945000700 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3ce140 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4938003fd0 tx=0x7f4938015d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6763dcdb7ce3dc3f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:21.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.021+0000 7f492ea00700 1 -- 172.21.15.43:0/4165524911 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493801c070 con 0x55c13c23ce40 2024-05-27T07:39:21.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.022+0000 7f4956529b80 1 -- 172.21.15.43:0/4165524911 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f493801d040 con 0x55c13c23ce40 2024-05-27T07:39:21.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.022+0000 7f492ea00700 1 -- 172.21.15.43:0/4165524911 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7f4938004ce0 con 0x55c13c23ce40 2024-05-27T07:39:21.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.022+0000 7f492ea00700 1 -- 172.21.15.43:0/4165524911 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f4938030070 con 0x55c13c23ce40 2024-05-27T07:39:21.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.022+0000 7f4956529b80 10 client.37919 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:21.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.022+0000 7f4956529b80 1 -- 172.21.15.43:0/4165524911 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55c13c3cdb60 con 0x55c13c23ce40 2024-05-27T07:39:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.022+0000 7f492ea00700 1 -- 172.21.15.43:0/4165524911 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f493801d040 con 0x55c13c23ce40 2024-05-27T07:39:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.023+0000 7f492ea00700 1 -- 172.21.15.43:0/4165524911 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f4938022040 con 0x55c13c23ce40 2024-05-27T07:39:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.023+0000 7f4956529b80 10 client.37919 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.023+0000 7f4956529b80 10 client.37919 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:39:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.023+0000 7f4956529b80 20 client.37919 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:21.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.023+0000 7f4956529b80 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c13c440bd0 0x55c13c442ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.023+0000 7f4956529b80 4 client.37919 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:39:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.023+0000 7f4956529b80 1 -- 172.21.15.43:0/4165524911 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c13c2e5570 con 0x55c13c440bd0 2024-05-27T07:39:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.023+0000 7f493fe00700 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c13c440bd0 0x55c13c442ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.024+0000 7f493fe00700 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c13c440bd0 0x55c13c442ff0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b3fda6d0d3b1f13b server_cookie=8c25ed8e6b88f1fe in_seq=0 out_seq=0 2024-05-27T07:39:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.024+0000 7f492ea00700 10 client.37919 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:39:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.025+0000 7f492ea00700 1 -- 172.21.15.43:0/4165524911 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c13c2e5570 con 0x55c13c440bd0 2024-05-27T07:39:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.025+0000 7f492ea00700 10 client.37919 handle_command_reply: tid=0 2024-05-27T07:39:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.025+0000 7f4956529b80 1 client.37919 shutdown 2024-05-27T07:39:21.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.025+0000 7f4956529b80 1 -- 172.21.15.43:0/4165524911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 msgr2=0x55c13c3ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:21.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.025+0000 7f4956529b80 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3ce140 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4938003fd0 tx=0x7f4938015d10 comp rx=0 tx=0).stop 2024-05-27T07:39:21.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.026+0000 7f4956529b80 1 -- 172.21.15.43:0/4165524911 shutdown_connections 2024-05-27T07:39:21.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.026+0000 7f4956529b80 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c13c3bbe70 0x55c13c3bfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.026+0000 7f4956529b80 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x55c13c440bd0 0x55c13c442ff0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.026+0000 7f4956529b80 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c13c23ce40 0x55c13c3ce140 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.026+0000 7f4956529b80 1 --2- 172.21.15.43:0/4165524911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c13c3289f0 0x55c13c3ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.026+0000 7f492ea00700 0 client.37919 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:39:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.026+0000 7f4956529b80 1 -- 172.21.15.43:0/4165524911 >> 172.21.15.43:0/4165524911 conn(0x55c13c2d6390 msgr2=0x55c13c3c69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.026+0000 7f4956529b80 1 -- 172.21.15.43:0/4165524911 shutdown_connections 2024-05-27T07:39:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.027+0000 7f4956529b80 1 -- 172.21.15.43:0/4165524911 wait complete. 2024-05-27T07:39:21.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.027+0000 7f4956529b80 20 client.37919 trim_cache size 0 max 16384 2024-05-27T07:39:21.026 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.029+0000 7f4947a00700 1 -- 172.21.15.43:0/3761351652 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f49240b2b20 msgr2=0x7f49240b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.029+0000 7f4947a00700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f49240b2b20 0x7f49240b5030 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4938009480 tx=0x7f493800b040 comp rx=0 tx=0).stop 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.029+0000 7f4947a00700 1 -- 172.21.15.43:0/3761351652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4940110ca0 msgr2=0x7f494006fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.029+0000 7f4947a00700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4940110ca0 0x7f494006fa80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f494006b890 tx=0x7f49280057c0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.030+0000 7f4947a00700 1 -- 172.21.15.43:0/3761351652 shutdown_connections 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.030+0000 7f4947a00700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f49240b2b20 0x7f49240b5030 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.030+0000 7f4947a00700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49400755c0 0x7f49401b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.030+0000 7f4947a00700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49401030f0 0x7f494006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.030+0000 7f4947a00700 1 --2- 172.21.15.43:0/3761351652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4940110ca0 0x7f494006fa80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.030+0000 7f4947a00700 1 -- 172.21.15.43:0/3761351652 >> 172.21.15.43:0/3761351652 conn(0x7f49400fedb0 msgr2=0x7f4940101d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.030+0000 7f4947a00700 1 -- 172.21.15.43:0/3761351652 shutdown_connections 2024-05-27T07:39:21.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:21.031+0000 7f4947a00700 1 -- 172.21.15.43:0/3761351652 wait complete. 2024-05-27T07:39:21.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:20 smithi132 ceph-mon[35283]: pgmap v853: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 175 op/s 2024-05-27T07:39:21.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 74365 5.82 MB/sec execute 562 sec latency 168.771 ms 2024-05-27T07:39:22.135 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:39:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:22.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 74478 5.82 MB/sec execute 563 sec latency 99.710 ms 2024-05-27T07:39:22.914 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:22 smithi043 ceph-mon[31531]: pgmap v854: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.7 MiB/s wr, 195 op/s 2024-05-27T07:39:22.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:22 smithi179 ceph-mon[35888]: pgmap v854: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.7 MiB/s wr, 195 op/s 2024-05-27T07:39:23.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:22 smithi132 ceph-mon[35283]: pgmap v854: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 4.7 MiB/s wr, 195 op/s 2024-05-27T07:39:23.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 74581 5.82 MB/sec execute 564 sec latency 95.411 ms 2024-05-27T07:39:24.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 74681 5.82 MB/sec execute 565 sec latency 340.667 ms 2024-05-27T07:39:24.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:24 smithi043 ceph-mon[31531]: pgmap v855: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 163 op/s 2024-05-27T07:39:24.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:24 smithi179 ceph-mon[35888]: pgmap v855: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 163 op/s 2024-05-27T07:39:25.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:24 smithi132 ceph-mon[35283]: pgmap v855: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 163 op/s 2024-05-27T07:39:25.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 74767 5.81 MB/sec execute 566 sec latency 227.059 ms 2024-05-27T07:39:26.677 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:26 smithi043 ceph-mon[31531]: pgmap v856: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 163 op/s 2024-05-27T07:39:26.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 74879 5.81 MB/sec execute 567 sec latency 110.471 ms 2024-05-27T07:39:26.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:26 smithi179 ceph-mon[35888]: pgmap v856: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 163 op/s 2024-05-27T07:39:27.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:26 smithi132 ceph-mon[35283]: pgmap v856: 129 pgs: 129 active+clean; 1.5 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.9 MiB/s wr, 163 op/s 2024-05-27T07:39:27.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 75000 5.81 MB/sec execute 568 sec latency 89.254 ms 2024-05-27T07:39:27.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.906+0000 7fb47f600700 1 -- 172.21.15.43:0/1200553303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb478103240 msgr2=0x7fb478103650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:27.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.906+0000 7fb47f600700 1 --2- 172.21.15.43:0/1200553303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb478103240 0x7fb478103650 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0098f0 tx=0x7fb46c009c00 comp rx=0 tx=0).stop 2024-05-27T07:39:27.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.906+0000 7fb47f600700 1 -- 172.21.15.43:0/1200553303 shutdown_connections 2024-05-27T07:39:27.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.906+0000 7fb47f600700 1 --2- 172.21.15.43:0/1200553303 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb478108690 0x7fb478108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.906+0000 7fb47f600700 1 --2- 172.21.15.43:0/1200553303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb478103240 0x7fb478103650 secure :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0098f0 tx=0x7fb46c009c00 comp rx=0 tx=0).stop 2024-05-27T07:39:27.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.906+0000 7fb47f600700 1 --2- 172.21.15.43:0/1200553303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb478103b90 0x7fb4781080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.906+0000 7fb47f600700 1 -- 172.21.15.43:0/1200553303 >> 172.21.15.43:0/1200553303 conn(0x7fb4780fedb0 msgr2=0x7fb4781011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:27.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.906+0000 7fb47f600700 1 -- 172.21.15.43:0/1200553303 shutdown_connections 2024-05-27T07:39:27.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.906+0000 7fb47f600700 1 -- 172.21.15.43:0/1200553303 wait complete. 2024-05-27T07:39:27.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.907+0000 7fb47f600700 1 Processor -- start 2024-05-27T07:39:27.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.908+0000 7fb47f600700 1 -- start start 2024-05-27T07:39:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.908+0000 7fb47f600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb478103b90 0x7fb4781a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.908+0000 7fb47f600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb478108690 0x7fb4781a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.908+0000 7fb47f600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4781a5cc0 0x7fb4781a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.908+0000 7fb47f600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb47811a910 con 0x7fb478108690 2024-05-27T07:39:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.908+0000 7fb47f600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb47811a790 con 0x7fb4781a5cc0 2024-05-27T07:39:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.908+0000 7fb47f600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb47811aa90 con 0x7fb478103b90 2024-05-27T07:39:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.909+0000 7fb477e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb478108690 0x7fb4781a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.909+0000 7fb47cc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb478103b90 0x7fb4781a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.909+0000 7fb477e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb478108690 0x7fb4781a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33358/0 (socket says 172.21.15.43:33358) 2024-05-27T07:39:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.909+0000 7fb47cc00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb478103b90 0x7fb4781a1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:38972/0 (socket says 172.21.15.43:38972) 2024-05-27T07:39:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.909+0000 7fb477e00700 1 -- 172.21.15.43:0/1700795063 learned_addr learned my addr 172.21.15.43:0/1700795063 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.909+0000 7fb47d600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4781a5cc0 0x7fb4781a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.910+0000 7fb47d600700 1 -- 172.21.15.43:0/1700795063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb478103b90 msgr2=0x7fb4781a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:27.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.910+0000 7fb47d600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb478103b90 0x7fb4781a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.910+0000 7fb47d600700 1 -- 172.21.15.43:0/1700795063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb478108690 msgr2=0x7fb4781a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:27.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.910+0000 7fb47d600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb478108690 0x7fb4781a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.910+0000 7fb47d600700 1 -- 172.21.15.43:0/1700795063 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4781a6810 con 0x7fb4781a5cc0 2024-05-27T07:39:27.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.910+0000 7fb47cc00700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb478103b90 0x7fb4781a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:27.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.911+0000 7fb47d600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4781a5cc0 0x7fb4781a60d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb470009cd0 tx=0x7fb47000c620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:27.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.911+0000 7fb475600700 1 -- 172.21.15.43:0/1700795063 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47000cdc0 con 0x7fb4781a5cc0 2024-05-27T07:39:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.911+0000 7fb47f600700 1 -- 172.21.15.43:0/1700795063 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4781a6aa0 con 0x7fb4781a5cc0 2024-05-27T07:39:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.911+0000 7fb475600700 1 -- 172.21.15.43:0/1700795063 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb470003a00 con 0x7fb4781a5cc0 2024-05-27T07:39:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.911+0000 7fb475600700 1 -- 172.21.15.43:0/1700795063 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4700044c0 con 0x7fb4781a5cc0 2024-05-27T07:39:27.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.912+0000 7fb47f600700 1 -- 172.21.15.43:0/1700795063 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4781ae220 con 0x7fb4781a5cc0 2024-05-27T07:39:27.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.918+0000 7fb475600700 1 -- 172.21.15.43:0/1700795063 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb47001a020 con 0x7fb4781a5cc0 2024-05-27T07:39:27.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.920+0000 7fb475600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4600b29f0 0x7fb4600b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.920+0000 7fb475600700 1 -- 172.21.15.43:0/1700795063 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fb4700e23b0 con 0x7fb4781a5cc0 2024-05-27T07:39:27.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.921+0000 7fb47cc00700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4600b29f0 0x7fb4600b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.922+0000 7fb47cc00700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4600b29f0 0x7fb4600b4f00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb46c000c00 tx=0x7fb46c019040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:27.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.922+0000 7fb48e0c3b80 1 Processor -- start 2024-05-27T07:39:27.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.923+0000 7fb48e0c3b80 1 -- start start 2024-05-27T07:39:27.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.924+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.924+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.924+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d730a1e70 0x564d730ae110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.924+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564d72fcfe40 con 0x564d72f22e40 2024-05-27T07:39:27.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.924+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564d72fbf9e0 con 0x564d7300e9f0 2024-05-27T07:39:27.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.924+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564d72fc3a20 con 0x564d730a1e70 2024-05-27T07:39:27.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.924+0000 7fb477e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.925+0000 7fb47cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.925+0000 7fb47cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40970/0 (socket says 172.21.15.43:40970) 2024-05-27T07:39:27.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.925+0000 7fb47cc00700 1 -- 172.21.15.43:0/2557938336 learned_addr learned my addr 172.21.15.43:0/2557938336 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:27.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.925+0000 7fb477e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33364/0 (socket says 172.21.15.43:33364) 2024-05-27T07:39:27.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.925+0000 7fb47d600700 1 --2- 172.21.15.43:0/2557938336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d730a1e70 0x564d730ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.925+0000 7fb47cc00700 1 -- 172.21.15.43:0/2557938336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d730a1e70 msgr2=0x564d730ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:27.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.925+0000 7fb47cc00700 1 --2- 172.21.15.43:0/2557938336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d730a1e70 0x564d730ae110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.926+0000 7fb47cc00700 1 -- 172.21.15.43:0/2557938336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 msgr2=0x564d730a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:27.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.926+0000 7fb47cc00700 1 --2- 172.21.15.43:0/2557938336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.926+0000 7fb47cc00700 1 -- 172.21.15.43:0/2557938336 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d72f88420 con 0x564d7300e9f0 2024-05-27T07:39:27.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.926+0000 7fb47cc00700 1 --2- 172.21.15.43:0/2557938336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a1330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb46c016000 tx=0x7fb46c01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=489e36f5f3dd1632 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:27.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.927+0000 7fb46a000700 1 -- 172.21.15.43:0/2557938336 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c031070 con 0x564d7300e9f0 2024-05-27T07:39:27.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.927+0000 7fb46a000700 1 -- 172.21.15.43:0/2557938336 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb46c0054d0 con 0x564d7300e9f0 2024-05-27T07:39:27.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.927+0000 7fb46a000700 1 -- 172.21.15.43:0/2557938336 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c00b560 con 0x564d7300e9f0 2024-05-27T07:39:27.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.929+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/2557938336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 msgr2=0x564d730a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:27.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.929+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/2557938336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a1330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb46c016000 tx=0x7fb46c01d000 comp rx=0 tx=0).stop 2024-05-27T07:39:27.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.929+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/2557938336 shutdown_connections 2024-05-27T07:39:27.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.929+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/2557938336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d730a1e70 0x564d730ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.929+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/2557938336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a1330 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.929+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/2557938336 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.929+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/2557938336 >> 172.21.15.43:0/2557938336 conn(0x564d72fbc390 msgr2=0x564d7309d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:27.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.930+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/2557938336 shutdown_connections 2024-05-27T07:39:27.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.930+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/2557938336 wait complete. 2024-05-27T07:39:27.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.931+0000 7fb48e0c3b80 1 -- start start 2024-05-27T07:39:27.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.931+0000 7fb48e0c3b80 1 Processor -- start 2024-05-27T07:39:27.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.932+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d72f22e40 0x564d730ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.932+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 0x564d730abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.932+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 0x564d730ac070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.933+0000 7fb477e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 0x564d730abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.933+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564d72ec5fa0 con 0x564d730a1e70 2024-05-27T07:39:27.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.933+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564d72f60760 con 0x564d72f22e40 2024-05-27T07:39:27.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.933+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564d730aeb20 con 0x564d7300e9f0 2024-05-27T07:39:27.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.933+0000 7fb47cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d72f22e40 0x564d730ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.933+0000 7fb477e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 0x564d730abb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:38990/0 (socket says 172.21.15.43:38990) 2024-05-27T07:39:27.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.933+0000 7fb477e00700 1 -- 172.21.15.43:0/3317427712 learned_addr learned my addr 172.21.15.43:0/3317427712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:27.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.933+0000 7fb48e0c3b80 10 client.0 _ll_register_callbacks cb 0x564d72f69c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:27.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.933+0000 7fb47d600700 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 0x564d730ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.934+0000 7fb47d600700 1 -- 172.21.15.43:0/3317427712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 msgr2=0x564d730abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:27.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.934+0000 7fb47d600700 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 0x564d730abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.934+0000 7fb47d600700 1 -- 172.21.15.43:0/3317427712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d72f22e40 msgr2=0x564d730ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:27.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.934+0000 7fb47d600700 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d72f22e40 0x564d730ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.934+0000 7fb47d600700 1 -- 172.21.15.43:0/3317427712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d72f0f8d0 con 0x564d730a1e70 2024-05-27T07:39:27.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.935+0000 7fb47cc00700 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d72f22e40 0x564d730ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:27.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.935+0000 7fb47d600700 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 0x564d730ac070 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb4700df6d0 tx=0x7fb4700a8d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbed03831b63095 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:27.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.936+0000 7fb45ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:39:27.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.936+0000 7fb45ea00700 1 -- 172.21.15.43:0/3317427712 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4700c3060 con 0x564d730a1e70 2024-05-27T07:39:27.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.936+0000 7fb45ea00700 1 -- 172.21.15.43:0/3317427712 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fb4700b1c30 con 0x564d730a1e70 2024-05-27T07:39:27.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.936+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3317427712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb46c0054d0 con 0x564d730a1e70 2024-05-27T07:39:27.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.937+0000 7fb45ea00700 1 -- 172.21.15.43:0/3317427712 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fb4700a9960 con 0x564d730a1e70 2024-05-27T07:39:27.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.937+0000 7fb48e0c3b80 10 client.18549 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:27.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.937+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3317427712 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x564d72f90060 con 0x564d730a1e70 2024-05-27T07:39:27.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.937+0000 7fb45ea00700 1 -- 172.21.15.43:0/3317427712 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb46c0054d0 con 0x564d730a1e70 2024-05-27T07:39:27.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.939+0000 7fb45ea00700 1 -- 172.21.15.43:0/3317427712 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fb4700c4c10 con 0x564d730a1e70 2024-05-27T07:39:27.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.939+0000 7fb48e0c3b80 10 client.18549 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:27.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.939+0000 7fb48e0c3b80 10 client.18549 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:39:27.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.939+0000 7fb48e0c3b80 20 client.18549 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:27.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.939+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564d730bbe10 0x564d730be280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.939+0000 7fb48e0c3b80 4 client.18549 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:27.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.939+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3317427712 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d72fd1d90 con 0x564d730bbe10 2024-05-27T07:39:27.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.940+0000 7fb47cc00700 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564d730bbe10 0x564d730be280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.941+0000 7fb45ea00700 10 client.18549 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:39:27.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.941+0000 7fb47cc00700 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564d730bbe10 0x564d730be280 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dd3215ad9195c1ae server_cookie=acc4f07737fd25ce in_seq=0 out_seq=0 2024-05-27T07:39:27.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.943+0000 7fb45ea00700 1 -- 172.21.15.43:0/3317427712 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564d72fd1d90 con 0x564d730bbe10 2024-05-27T07:39:27.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.943+0000 7fb45ea00700 10 client.18549 handle_command_reply: tid=0 2024-05-27T07:39:27.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.943+0000 7fb48e0c3b80 1 client.18549 shutdown 2024-05-27T07:39:27.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3317427712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 msgr2=0x564d730ac070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:27.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 0x564d730ac070 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb4700df6d0 tx=0x7fb4700a8d70 comp rx=0 tx=0).stop 2024-05-27T07:39:27.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3317427712 shutdown_connections 2024-05-27T07:39:27.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564d730bbe10 0x564d730be280 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 0x564d730ac070 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d72f22e40 0x564d730ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb45ea00700 0 client.18549 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:39:27.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3317427712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 0x564d730abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3317427712 >> 172.21.15.43:0/3317427712 conn(0x564d72fbc390 msgr2=0x564d730a1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:27.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3317427712 shutdown_connections 2024-05-27T07:39:27.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3317427712 wait complete. 2024-05-27T07:39:27.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.944+0000 7fb48e0c3b80 20 client.18549 trim_cache size 0 max 16384 2024-05-27T07:39:27.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.970+0000 7fb48e0c3b80 1 Processor -- start 2024-05-27T07:39:27.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.970+0000 7fb48e0c3b80 1 -- start start 2024-05-27T07:39:27.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.970+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.970+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 0x564d730a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.970+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d730a1e70 0x564d730aa550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.970+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb4781b77e0 con 0x564d72f22e40 2024-05-27T07:39:27.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.970+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb4781b5ca0 con 0x564d730a1e70 2024-05-27T07:39:27.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.970+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb4781b7110 con 0x564d7300e9f0 2024-05-27T07:39:27.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.971+0000 7fb47cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.971+0000 7fb47cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33380/0 (socket says 172.21.15.43:33380) 2024-05-27T07:39:27.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.971+0000 7fb47cc00700 1 -- 172.21.15.43:0/2804064737 learned_addr learned my addr 172.21.15.43:0/2804064737 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:27.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.971+0000 7fb477e00700 1 --2- 172.21.15.43:0/2804064737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 0x564d730a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.971+0000 7fb47d600700 1 --2- 172.21.15.43:0/2804064737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d730a1e70 0x564d730aa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.971+0000 7fb47cc00700 1 -- 172.21.15.43:0/2804064737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 msgr2=0x564d730a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:27.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.971+0000 7fb47cc00700 1 --2- 172.21.15.43:0/2804064737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 0x564d730a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.972+0000 7fb47cc00700 1 -- 172.21.15.43:0/2804064737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d730a1e70 msgr2=0x564d730aa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:27.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.972+0000 7fb47cc00700 1 --2- 172.21.15.43:0/2804064737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d730a1e70 0x564d730aa550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.972+0000 7fb47cc00700 1 -- 172.21.15.43:0/2804064737 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d730ac3e0 con 0x564d72f22e40 2024-05-27T07:39:27.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.972+0000 7fb47cc00700 1 --2- 172.21.15.43:0/2804064737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a6e30 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fb46c016000 tx=0x7fb46c0041d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3c2f01b5745daa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:27.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.973+0000 7fb469600700 1 -- 172.21.15.43:0/2804064737 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c03d070 con 0x564d72f22e40 2024-05-27T07:39:27.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.973+0000 7fb469600700 1 -- 172.21.15.43:0/2804064737 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb46c004750 con 0x564d72f22e40 2024-05-27T07:39:27.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.973+0000 7fb469600700 1 -- 172.21.15.43:0/2804064737 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c031040 con 0x564d72f22e40 2024-05-27T07:39:27.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.974+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/2804064737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 msgr2=0x564d730a6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:27.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.974+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/2804064737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a6e30 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fb46c016000 tx=0x7fb46c0041d0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.975+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/2804064737 shutdown_connections 2024-05-27T07:39:27.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.975+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/2804064737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d730a1e70 0x564d730aa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.975+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/2804064737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d72f22e40 0x564d730a6e30 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.975+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/2804064737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d7300e9f0 0x564d730a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.975+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/2804064737 >> 172.21.15.43:0/2804064737 conn(0x564d72fbc390 msgr2=0x564d730a11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:27.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.975+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/2804064737 shutdown_connections 2024-05-27T07:39:27.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.976+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/2804064737 wait complete. 2024-05-27T07:39:27.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.976+0000 7fb48e0c3b80 1 -- start start 2024-05-27T07:39:27.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.977+0000 7fb48e0c3b80 1 Processor -- start 2024-05-27T07:39:27.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.977+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d72f22e40 0x564d730a5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.977+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.977+0000 7fb48e0c3b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 0x564d730b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.978+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb4781b6a40 con 0x564d730a1e70 2024-05-27T07:39:27.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.978+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb4781b6370 con 0x564d7300e9f0 2024-05-27T07:39:27.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.978+0000 7fb48e0c3b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564d730aca80 con 0x564d72f22e40 2024-05-27T07:39:27.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.978+0000 7fb48e0c3b80 10 client.0 _ll_register_callbacks cb 0x564d72f69c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:27.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.978+0000 7fb477e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.978+0000 7fb477e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:40994/0 (socket says 172.21.15.43:40994) 2024-05-27T07:39:27.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.978+0000 7fb477e00700 1 -- 172.21.15.43:0/3699118500 learned_addr learned my addr 172.21.15.43:0/3699118500 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:27.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.978+0000 7fb47cc00700 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d72f22e40 0x564d730a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.978+0000 7fb47d600700 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 0x564d730b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.979+0000 7fb477e00700 1 -- 172.21.15.43:0/3699118500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d72f22e40 msgr2=0x564d730a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:27.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.979+0000 7fb477e00700 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d72f22e40 0x564d730a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.979+0000 7fb477e00700 1 -- 172.21.15.43:0/3699118500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 msgr2=0x564d730b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:27.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.979+0000 7fb477e00700 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 0x564d730b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.979+0000 7fb477e00700 1 -- 172.21.15.43:0/3699118500 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d730a6360 con 0x564d7300e9f0 2024-05-27T07:39:27.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.980+0000 7fb47d600700 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 0x564d730b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:27.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.980+0000 7fb477e00700 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a5b60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x564d730a1180 tx=0x7fb464012b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f87bcacb2c929258 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:27.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.980+0000 7fb45ea00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:39:27.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.981+0000 7fb45ea00700 1 -- 172.21.15.43:0/3699118500 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464004440 con 0x564d7300e9f0 2024-05-27T07:39:27.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.981+0000 7fb45ea00700 1 -- 172.21.15.43:0/3699118500 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fb464003680 con 0x564d7300e9f0 2024-05-27T07:39:27.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.981+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3699118500 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb46c004750 con 0x564d7300e9f0 2024-05-27T07:39:27.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.982+0000 7fb45ea00700 1 -- 172.21.15.43:0/3699118500 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fb464003880 con 0x564d7300e9f0 2024-05-27T07:39:27.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.982+0000 7fb48e0c3b80 10 client.37937 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:27.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.982+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3699118500 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x564d730b4340 con 0x564d7300e9f0 2024-05-27T07:39:27.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.983+0000 7fb45ea00700 1 -- 172.21.15.43:0/3699118500 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb46c004750 con 0x564d7300e9f0 2024-05-27T07:39:27.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.983+0000 7fb45ea00700 1 -- 172.21.15.43:0/3699118500 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fb464025030 con 0x564d7300e9f0 2024-05-27T07:39:27.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.983+0000 7fb48e0c3b80 10 client.37937 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:27.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.983+0000 7fb48e0c3b80 10 client.37937 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:39:27.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.983+0000 7fb48e0c3b80 20 client.37937 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:27.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.983+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564d73126950 0x564d73128d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:27.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.983+0000 7fb48e0c3b80 4 client.37937 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:39:27.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.983+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3699118500 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d72fcb570 con 0x564d73126950 2024-05-27T07:39:27.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.984+0000 7fb47cc00700 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564d73126950 0x564d73128d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:27.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.985+0000 7fb47cc00700 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564d73126950 0x564d73128d70 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a4fac674040d700 server_cookie=4efb1b7ad3a73a2b in_seq=0 out_seq=0 2024-05-27T07:39:27.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb45ea00700 10 client.37937 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:39:27.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb45ea00700 1 -- 172.21.15.43:0/3699118500 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d72fcb570 con 0x564d73126950 2024-05-27T07:39:27.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb45ea00700 10 client.37937 handle_command_reply: tid=0 2024-05-27T07:39:27.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb48e0c3b80 1 client.37937 shutdown 2024-05-27T07:39:27.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3699118500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 msgr2=0x564d730a5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:27.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a5b60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x564d730a1180 tx=0x7fb464012b80 comp rx=0 tx=0).stop 2024-05-27T07:39:27.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3699118500 shutdown_connections 2024-05-27T07:39:27.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x564d73126950 0x564d73128d70 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564d730a1e70 0x564d730b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564d72f22e40 0x564d730a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb48e0c3b80 1 --2- 172.21.15.43:0/3699118500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564d7300e9f0 0x564d730a5b60 secure :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0x564d730a1180 tx=0x7fb464012b80 comp rx=0 tx=0).stop 2024-05-27T07:39:27.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.986+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3699118500 >> 172.21.15.43:0/3699118500 conn(0x564d72fbc390 msgr2=0x564d730ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:27.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.987+0000 7fb45ea00700 0 client.37937 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:39:27.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.987+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3699118500 shutdown_connections 2024-05-27T07:39:27.985 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:39:27.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.988+0000 7fb48e0c3b80 1 -- 172.21.15.43:0/3699118500 wait complete. 2024-05-27T07:39:27.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.988+0000 7fb48e0c3b80 20 client.37937 trim_cache size 0 max 16384 2024-05-27T07:39:27.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.989+0000 7fb47f600700 1 -- 172.21.15.43:0/1700795063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4600b29f0 msgr2=0x7fb4600b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:27.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.989+0000 7fb47f600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4600b29f0 0x7fb4600b4f00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb46c000c00 tx=0x7fb46c019040 comp rx=0 tx=0).stop 2024-05-27T07:39:27.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.989+0000 7fb47f600700 1 -- 172.21.15.43:0/1700795063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4781a5cc0 msgr2=0x7fb4781a60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:27.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.989+0000 7fb47f600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4781a5cc0 0x7fb4781a60d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb470009cd0 tx=0x7fb47000c620 comp rx=0 tx=0).stop 2024-05-27T07:39:27.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.989+0000 7fb47f600700 1 -- 172.21.15.43:0/1700795063 shutdown_connections 2024-05-27T07:39:27.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.989+0000 7fb47f600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb4600b29f0 0x7fb4600b4f00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.989+0000 7fb47f600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4781a5cc0 0x7fb4781a60d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.989+0000 7fb47f600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb478103b90 0x7fb4781a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.990+0000 7fb47f600700 1 --2- 172.21.15.43:0/1700795063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb478108690 0x7fb4781a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:27.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.990+0000 7fb47f600700 1 -- 172.21.15.43:0/1700795063 >> 172.21.15.43:0/1700795063 conn(0x7fb4780fedb0 msgr2=0x7fb478106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:27.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.990+0000 7fb47f600700 1 -- 172.21.15.43:0/1700795063 shutdown_connections 2024-05-27T07:39:27.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:27.990+0000 7fb47f600700 1 -- 172.21.15.43:0/1700795063 wait complete. 2024-05-27T07:39:28.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 75097 5.80 MB/sec execute 569 sec latency 103.583 ms 2024-05-27T07:39:28.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:28 smithi043 ceph-mon[31531]: pgmap v857: 129 pgs: 129 active+clean; 1.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.3 MiB/s wr, 242 op/s 2024-05-27T07:39:28.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:28 smithi179 ceph-mon[35888]: pgmap v857: 129 pgs: 129 active+clean; 1.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.3 MiB/s wr, 242 op/s 2024-05-27T07:39:29.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:28 smithi132 ceph-mon[35283]: pgmap v857: 129 pgs: 129 active+clean; 1.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.3 MiB/s wr, 242 op/s 2024-05-27T07:39:29.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:39:29.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 75192 5.79 MB/sec execute 570 sec latency 108.545 ms 2024-05-27T07:39:30.471 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:30.839 INFO:tasks.workunit.client.0.smithi043.stdout: 10 75312 5.80 MB/sec execute 571 sec latency 146.747 ms 2024-05-27T07:39:30.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:30 smithi043 ceph-mon[31531]: pgmap v858: 129 pgs: 129 active+clean; 1.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-05-27T07:39:30.900 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:30 smithi132 ceph-mon[35283]: pgmap v858: 129 pgs: 129 active+clean; 1.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-05-27T07:39:30.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:30 smithi179 ceph-mon[35888]: pgmap v858: 129 pgs: 129 active+clean; 1.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-05-27T07:39:31.840 INFO:tasks.workunit.client.0.smithi043.stdout: 10 75412 5.80 MB/sec execute 572 sec latency 303.883 ms 2024-05-27T07:39:32.840 INFO:tasks.workunit.client.0.smithi043.stdout: 10 75503 5.80 MB/sec execute 573 sec latency 330.481 ms 2024-05-27T07:39:32.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:32 smithi043 ceph-mon[31531]: pgmap v859: 129 pgs: 129 active+clean; 1.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 5.0 MiB/s wr, 241 op/s 2024-05-27T07:39:32.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:32 smithi179 ceph-mon[35888]: pgmap v859: 129 pgs: 129 active+clean; 1.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 5.0 MiB/s wr, 241 op/s 2024-05-27T07:39:33.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:32 smithi132 ceph-mon[35283]: pgmap v859: 129 pgs: 129 active+clean; 1.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 5.0 MiB/s wr, 241 op/s 2024-05-27T07:39:33.840 INFO:tasks.workunit.client.0.smithi043.stdout: 10 75612 5.81 MB/sec execute 574 sec latency 248.919 ms 2024-05-27T07:39:34.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 75740 5.81 MB/sec execute 575 sec latency 146.754 ms 2024-05-27T07:39:34.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:34 smithi043 ceph-mon[31531]: pgmap v860: 129 pgs: 129 active+clean; 1.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 3.0 MiB/s wr, 150 op/s 2024-05-27T07:39:34.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:34 smithi179 ceph-mon[35888]: pgmap v860: 129 pgs: 129 active+clean; 1.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 3.0 MiB/s wr, 150 op/s 2024-05-27T07:39:35.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:34 smithi132 ceph-mon[35283]: pgmap v860: 129 pgs: 129 active+clean; 1.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 3.0 MiB/s wr, 150 op/s 2024-05-27T07:39:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.262+0000 7f9c27e00700 1 -- 172.21.15.43:0/2560026120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c28102f20 msgr2=0x7f9c28103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.262+0000 7f9c27e00700 1 --2- 172.21.15.43:0/2560026120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c28102f20 0x7f9c28103330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009990 tx=0x7f9c18009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.262+0000 7f9c27e00700 1 -- 172.21.15.43:0/2560026120 shutdown_connections 2024-05-27T07:39:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.262+0000 7f9c27e00700 1 --2- 172.21.15.43:0/2560026120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c28110ad0 0x7f9c28112ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.263+0000 7f9c27e00700 1 --2- 172.21.15.43:0/2560026120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c28102f20 0x7f9c28103330 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009990 tx=0x7f9c18009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.263+0000 7f9c27e00700 1 --2- 172.21.15.43:0/2560026120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c28103870 0x7f9c28110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.263+0000 7f9c27e00700 1 -- 172.21.15.43:0/2560026120 >> 172.21.15.43:0/2560026120 conn(0x7f9c28078ab0 msgr2=0x7f9c28078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.263+0000 7f9c27e00700 1 -- 172.21.15.43:0/2560026120 shutdown_connections 2024-05-27T07:39:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.263+0000 7f9c27e00700 1 -- 172.21.15.43:0/2560026120 wait complete. 2024-05-27T07:39:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.265+0000 7f9c27e00700 1 Processor -- start 2024-05-27T07:39:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.265+0000 7f9c27e00700 1 -- start start 2024-05-27T07:39:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.266+0000 7f9c27e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c28103870 0x7f9c2810f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.266+0000 7f9c27e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c28110ad0 0x7f9c2810f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.266+0000 7f9c27e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c281a6da0 0x7f9c281a71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.266+0000 7f9c27e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9c28105c90 con 0x7f9c28103870 2024-05-27T07:39:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.266+0000 7f9c27e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9c28105b10 con 0x7f9c28110ad0 2024-05-27T07:39:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.266+0000 7f9c27e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9c28105e10 con 0x7f9c281a6da0 2024-05-27T07:39:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.266+0000 7f9c26a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c28103870 0x7f9c2810f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.267+0000 7f9c26a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c28103870 0x7f9c2810f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54730/0 (socket says 172.21.15.43:54730) 2024-05-27T07:39:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.267+0000 7f9c26a00700 1 -- 172.21.15.43:0/4216551577 learned_addr learned my addr 172.21.15.43:0/4216551577 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.267+0000 7f9c27400700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c281a6da0 0x7f9c281a71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.267+0000 7f9c26000700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c28110ad0 0x7f9c2810f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.268+0000 7f9c26a00700 1 -- 172.21.15.43:0/4216551577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c281a6da0 msgr2=0x7f9c281a71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.268+0000 7f9c26a00700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c281a6da0 0x7f9c281a71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.268+0000 7f9c26a00700 1 -- 172.21.15.43:0/4216551577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c28110ad0 msgr2=0x7f9c2810f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.268+0000 7f9c26a00700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c28110ad0 0x7f9c2810f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.268+0000 7f9c26a00700 1 -- 172.21.15.43:0/4216551577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c281a78b0 con 0x7f9c28103870 2024-05-27T07:39:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.268+0000 7f9c26000700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c28110ad0 0x7f9c2810f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.269+0000 7f9c26a00700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c28103870 0x7f9c2810f140 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9c180098e0 tx=0x7f9c180047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.269+0000 7f9c1f400700 1 -- 172.21.15.43:0/4216551577 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c18026070 con 0x7f9c28103870 2024-05-27T07:39:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.269+0000 7f9c1f400700 1 -- 172.21.15.43:0/4216551577 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c18021410 con 0x7f9c28103870 2024-05-27T07:39:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.269+0000 7f9c27e00700 1 -- 172.21.15.43:0/4216551577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c281a7b40 con 0x7f9c28103870 2024-05-27T07:39:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.269+0000 7f9c1f400700 1 -- 172.21.15.43:0/4216551577 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c18017650 con 0x7f9c28103870 2024-05-27T07:39:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.269+0000 7f9c27e00700 1 -- 172.21.15.43:0/4216551577 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c281ae020 con 0x7f9c28103870 2024-05-27T07:39:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.274+0000 7f9c1f400700 1 -- 172.21.15.43:0/4216551577 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9c18029070 con 0x7f9c28103870 2024-05-27T07:39:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.277+0000 7f9c1f400700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c0c0b2a70 0x7f9c0c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.277+0000 7f9c1f400700 1 -- 172.21.15.43:0/4216551577 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7f9c180e4650 con 0x7f9c28103870 2024-05-27T07:39:35.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.277+0000 7f9c26000700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c0c0b2a70 0x7f9c0c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.290+0000 7f9c26000700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c0c0b2a70 0x7f9c0c0b4f80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9c281027b0 tx=0x7f9c14009260 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.294+0000 7f9c3c620b80 1 Processor -- start 2024-05-27T07:39:35.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.294+0000 7f9c3c620b80 1 -- start start 2024-05-27T07:39:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.295+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492a00e40 0x564492b7eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.295+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.295+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 0x564492b8c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.295+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564492aade40 con 0x564492b7f820 2024-05-27T07:39:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.295+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564492a9d9e0 con 0x564492b7f410 2024-05-27T07:39:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.295+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564492aa1a20 con 0x564492a00e40 2024-05-27T07:39:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.295+0000 7f9c26000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.295+0000 7f9c26000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48500/0 (socket says 172.21.15.43:48500) 2024-05-27T07:39:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.295+0000 7f9c26000700 1 -- 172.21.15.43:0/1524843455 learned_addr learned my addr 172.21.15.43:0/1524843455 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.295+0000 7f9c27400700 1 --2- 172.21.15.43:0/1524843455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 0x564492b8c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.296+0000 7f9c26a00700 1 --2- 172.21.15.43:0/1524843455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492a00e40 0x564492b7eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.296+0000 7f9c26000700 1 -- 172.21.15.43:0/1524843455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492a00e40 msgr2=0x564492b7eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.296+0000 7f9c26000700 1 --2- 172.21.15.43:0/1524843455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492a00e40 0x564492b7eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.296+0000 7f9c26000700 1 -- 172.21.15.43:0/1524843455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 msgr2=0x564492b8c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.296+0000 7f9c26000700 1 --2- 172.21.15.43:0/1524843455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 0x564492b8c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.296+0000 7f9c26000700 1 -- 172.21.15.43:0/1524843455 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564492a66420 con 0x564492b7f410 2024-05-27T07:39:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.296+0000 7f9c26000700 1 --2- 172.21.15.43:0/1524843455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89ad0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9c14004370 tx=0x7f9c14014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a47e36c722866071 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.297+0000 7f9c13e00700 1 -- 172.21.15.43:0/1524843455 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14003d10 con 0x564492b7f410 2024-05-27T07:39:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.297+0000 7f9c13e00700 1 -- 172.21.15.43:0/1524843455 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c1401e8b0 con 0x564492b7f410 2024-05-27T07:39:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.297+0000 7f9c13e00700 1 -- 172.21.15.43:0/1524843455 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c140266a0 con 0x564492b7f410 2024-05-27T07:39:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.298+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1524843455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 msgr2=0x564492b89ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.298+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1524843455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89ad0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9c14004370 tx=0x7f9c14014000 comp rx=0 tx=0).stop 2024-05-27T07:39:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.298+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1524843455 shutdown_connections 2024-05-27T07:39:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.298+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1524843455 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 0x564492b8c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.298+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1524843455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492a00e40 0x564492b7eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.298+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1524843455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89ad0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.298+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1524843455 >> 172.21.15.43:0/1524843455 conn(0x564492a60fc0 msgr2=0x564492b7ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.299+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1524843455 shutdown_connections 2024-05-27T07:39:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.299+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1524843455 wait complete. 2024-05-27T07:39:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.300+0000 7f9c3c620b80 1 -- start start 2024-05-27T07:39:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.300+0000 7f9c3c620b80 1 Processor -- start 2024-05-27T07:39:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 0x564492b89510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 0x564492b89f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5644929a3fa0 con 0x564492a00e40 2024-05-27T07:39:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564492a3e760 con 0x564492b7f410 2024-05-27T07:39:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564492b8c990 con 0x564492b7f820 2024-05-27T07:39:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c3c620b80 10 client.0 _ll_register_callbacks cb 0x564492a47c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c27400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 0x564492b89f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c26000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c26000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48506/0 (socket says 172.21.15.43:48506) 2024-05-27T07:39:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.301+0000 7f9c26000700 1 -- 172.21.15.43:0/1719284228 learned_addr learned my addr 172.21.15.43:0/1719284228 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.302+0000 7f9c26a00700 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 0x564492b89510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.302+0000 7f9c26000700 1 -- 172.21.15.43:0/1719284228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 msgr2=0x564492b89f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.302+0000 7f9c26000700 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 0x564492b89f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.302+0000 7f9c26000700 1 -- 172.21.15.43:0/1719284228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 msgr2=0x564492b89510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.302+0000 7f9c26000700 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 0x564492b89510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.302+0000 7f9c26000700 1 -- 172.21.15.43:0/1719284228 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644929ed8d0 con 0x564492b7f410 2024-05-27T07:39:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.302+0000 7f9c26000700 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89a50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9c14004340 tx=0x7f9c14003d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e0a17a659056a18 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.302+0000 7f9c10c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:39:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.303+0000 7f9c10c00700 1 -- 172.21.15.43:0/1719284228 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1401ec70 con 0x564492b7f410 2024-05-27T07:39:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.303+0000 7f9c10c00700 1 -- 172.21.15.43:0/1719284228 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7f9c14005040 con 0x564492b7f410 2024-05-27T07:39:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.303+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1719284228 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c1401e8b0 con 0x564492b7f410 2024-05-27T07:39:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.303+0000 7f9c10c00700 1 -- 172.21.15.43:0/1719284228 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f9c14005240 con 0x564492b7f410 2024-05-27T07:39:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.303+0000 7f9c3c620b80 10 client.37949 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.303+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1719284228 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x564492a6e060 con 0x564492b7f410 2024-05-27T07:39:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.304+0000 7f9c10c00700 1 -- 172.21.15.43:0/1719284228 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c14004680 con 0x564492b7f410 2024-05-27T07:39:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.304+0000 7f9c10c00700 1 -- 172.21.15.43:0/1719284228 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f9c14031420 con 0x564492b7f410 2024-05-27T07:39:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.304+0000 7f9c3c620b80 10 client.37949 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.304+0000 7f9c3c620b80 10 client.37949 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:39:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.304+0000 7f9c3c620b80 20 client.37949 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.304+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564492b99ae0 0x564492b9bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.304+0000 7f9c3c620b80 4 client.37949 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.305+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1719284228 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9c1401e8b0 con 0x564492b99ae0 2024-05-27T07:39:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.305+0000 7f9c26a00700 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564492b99ae0 0x564492b9bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.306+0000 7f9c26a00700 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564492b99ae0 0x564492b9bf50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=337fbce1c094f774 server_cookie=3acab587397d8f46 in_seq=0 out_seq=0 2024-05-27T07:39:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.306+0000 7f9c10c00700 10 client.37949 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:39:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c10c00700 1 -- 172.21.15.43:0/1719284228 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x7f9c1401e8b0 con 0x564492b99ae0 2024-05-27T07:39:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c10c00700 10 client.37949 handle_command_reply: tid=0 2024-05-27T07:39:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c3c620b80 1 client.37949 shutdown 2024-05-27T07:39:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1719284228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 msgr2=0x564492b89a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89a50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9c14004340 tx=0x7f9c14003d00 comp rx=0 tx=0).stop 2024-05-27T07:39:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1719284228 shutdown_connections 2024-05-27T07:39:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564492b99ae0 0x564492b9bf50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 0x564492b89f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 0x564492b89510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1719284228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b89a50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c10c00700 0 client.37949 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:39:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.307+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1719284228 >> 172.21.15.43:0/1719284228 conn(0x564492a60fc0 msgr2=0x564492b79ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.308+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1719284228 shutdown_connections 2024-05-27T07:39:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.308+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1719284228 wait complete. 2024-05-27T07:39:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.308+0000 7f9c3c620b80 20 client.37949 trim_cache size 0 max 16384 2024-05-27T07:39:35.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.337+0000 7f9c3c620b80 1 Processor -- start 2024-05-27T07:39:35.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.338+0000 7f9c3c620b80 1 -- start start 2024-05-27T07:39:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.338+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492a00e40 0x564492b88470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.339+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b84cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.339+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 0x564492b95a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.339+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9c281b7350 con 0x564492b7f820 2024-05-27T07:39:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.339+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9c281b5810 con 0x564492b7f410 2024-05-27T07:39:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.339+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9c281b6c80 con 0x564492a00e40 2024-05-27T07:39:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.340+0000 7f9c26000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b84cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.340+0000 7f9c26000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b84cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48514/0 (socket says 172.21.15.43:48514) 2024-05-27T07:39:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.340+0000 7f9c26000700 1 -- 172.21.15.43:0/3813851505 learned_addr learned my addr 172.21.15.43:0/3813851505 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.340+0000 7f9c26a00700 1 --2- 172.21.15.43:0/3813851505 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492a00e40 0x564492b88470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.340+0000 7f9c27400700 1 --2- 172.21.15.43:0/3813851505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 0x564492b95a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.341+0000 7f9c26000700 1 -- 172.21.15.43:0/3813851505 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492a00e40 msgr2=0x564492b88470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.341+0000 7f9c26000700 1 --2- 172.21.15.43:0/3813851505 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492a00e40 0x564492b88470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.341+0000 7f9c26000700 1 -- 172.21.15.43:0/3813851505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 msgr2=0x564492b95a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.341+0000 7f9c26000700 1 --2- 172.21.15.43:0/3813851505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 0x564492b95a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.341+0000 7f9c26000700 1 -- 172.21.15.43:0/3813851505 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564492b7b1f0 con 0x564492b7f410 2024-05-27T07:39:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.341+0000 7f9c27400700 1 --2- 172.21.15.43:0/3813851505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 0x564492b95a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.341+0000 7f9c26000700 1 --2- 172.21.15.43:0/3813851505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b84cf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9c140317c0 tx=0x7f9c140058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e884816e848cb615 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.342+0000 7f9c13400700 1 -- 172.21.15.43:0/3813851505 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14043070 con 0x564492b7f410 2024-05-27T07:39:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.342+0000 7f9c13400700 1 -- 172.21.15.43:0/3813851505 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c14003ee0 con 0x564492b7f410 2024-05-27T07:39:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.342+0000 7f9c13400700 1 -- 172.21.15.43:0/3813851505 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1401a3a0 con 0x564492b7f410 2024-05-27T07:39:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.343+0000 7f9c3c620b80 1 -- 172.21.15.43:0/3813851505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 msgr2=0x564492b84cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.343+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/3813851505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b84cf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9c140317c0 tx=0x7f9c140058e0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.343+0000 7f9c3c620b80 1 -- 172.21.15.43:0/3813851505 shutdown_connections 2024-05-27T07:39:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.343+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/3813851505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492b7f820 0x564492b95a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.343+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/3813851505 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492a00e40 0x564492b88470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.343+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/3813851505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b84cf0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.343+0000 7f9c3c620b80 1 -- 172.21.15.43:0/3813851505 >> 172.21.15.43:0/3813851505 conn(0x564492a60fc0 msgr2=0x564492b894d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.344+0000 7f9c3c620b80 1 -- 172.21.15.43:0/3813851505 shutdown_connections 2024-05-27T07:39:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.344+0000 7f9c3c620b80 1 -- 172.21.15.43:0/3813851505 wait complete. 2024-05-27T07:39:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.344+0000 7f9c3c620b80 1 -- start start 2024-05-27T07:39:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.345+0000 7f9c3c620b80 1 Processor -- start 2024-05-27T07:39:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.345+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 0x564492b91fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.345+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b8d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.345+0000 7f9c3c620b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 0x564492b8d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.345+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9c281b65b0 con 0x564492a00e40 2024-05-27T07:39:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.345+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9c281b5ee0 con 0x564492b7f410 2024-05-27T07:39:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.345+0000 7f9c3c620b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564492b95fa0 con 0x564492b7f820 2024-05-27T07:39:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.345+0000 7f9c3c620b80 10 client.0 _ll_register_callbacks cb 0x564492a47c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.347+0000 7f9c26000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b8d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.347+0000 7f9c26a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 0x564492b91fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.348+0000 7f9c27400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 0x564492b8d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.348+0000 7f9c26a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 0x564492b91fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54754/0 (socket says 172.21.15.43:54754) 2024-05-27T07:39:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.348+0000 7f9c26a00700 1 -- 172.21.15.43:0/1333141049 learned_addr learned my addr 172.21.15.43:0/1333141049 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.349+0000 7f9c27400700 1 -- 172.21.15.43:0/1333141049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 msgr2=0x564492b8d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.349+0000 7f9c27400700 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b8d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.349+0000 7f9c27400700 1 -- 172.21.15.43:0/1333141049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 msgr2=0x564492b91fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.349+0000 7f9c27400700 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 0x564492b91fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.349+0000 7f9c27400700 1 -- 172.21.15.43:0/1333141049 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564492b8ddb0 con 0x564492b7f820 2024-05-27T07:39:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.349+0000 7f9c26a00700 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 0x564492b91fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.350+0000 7f9c27400700 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 0x564492b8d870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x564492b7fcb0 tx=0x7f9c20011f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8071859a3830bbcf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.350+0000 7f9c10c00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:39:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.351+0000 7f9c10c00700 1 -- 172.21.15.43:0/1333141049 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20007c80 con 0x564492b7f820 2024-05-27T07:39:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.351+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1333141049 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c14003ee0 con 0x564492b7f820 2024-05-27T07:39:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.351+0000 7f9c10c00700 1 -- 172.21.15.43:0/1333141049 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7f9c20003a00 con 0x564492b7f820 2024-05-27T07:39:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.352+0000 7f9c10c00700 1 -- 172.21.15.43:0/1333141049 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f9c2002a050 con 0x564492b7f820 2024-05-27T07:39:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.352+0000 7f9c3c620b80 10 client.27967 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.352+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1333141049 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x564492c04250 con 0x564492b7f820 2024-05-27T07:39:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.352+0000 7f9c10c00700 1 -- 172.21.15.43:0/1333141049 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9c14003ee0 con 0x564492b7f820 2024-05-27T07:39:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.354+0000 7f9c10c00700 1 -- 172.21.15.43:0/1333141049 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7f9c20003c20 con 0x564492b7f820 2024-05-27T07:39:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.354+0000 7f9c3c620b80 10 client.27967 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.354+0000 7f9c3c620b80 10 client.27967 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:39:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.354+0000 7f9c3c620b80 20 client.27967 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.354+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564492c048b0 0x564492c06cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.354+0000 7f9c3c620b80 4 client.27967 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:39:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.354+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1333141049 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564492aafd90 con 0x564492c048b0 2024-05-27T07:39:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.356+0000 7f9c26a00700 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564492c048b0 0x564492c06cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.358+0000 7f9c26a00700 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564492c048b0 0x564492c06cd0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=de90c6cf6b991b44 server_cookie=68cb7c0b6b4d7190 in_seq=0 out_seq=0 2024-05-27T07:39:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.358+0000 7f9c10c00700 10 client.27967 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:39:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.359+0000 7f9c10c00700 1 -- 172.21.15.43:0/1333141049 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564492aafd90 con 0x564492c048b0 2024-05-27T07:39:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.359+0000 7f9c10c00700 10 client.27967 handle_command_reply: tid=0 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.359+0000 7f9c3c620b80 1 client.27967 shutdown 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.359+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1333141049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 msgr2=0x564492b8d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.359+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 0x564492b8d870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x564492b7fcb0 tx=0x7f9c20011f40 comp rx=0 tx=0).stop 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.360+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1333141049 shutdown_connections 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.360+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564492c048b0 0x564492c06cd0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.360+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564492b7f820 0x564492b8d870 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.360+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564492a00e40 0x564492b91fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.360+0000 7f9c10c00700 0 client.27967 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.360+0000 7f9c3c620b80 1 --2- 172.21.15.43:0/1333141049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564492b7f410 0x564492b8d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.360+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1333141049 >> 172.21.15.43:0/1333141049 conn(0x564492a60fc0 msgr2=0x564492b94a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.360+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1333141049 shutdown_connections 2024-05-27T07:39:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.360+0000 7f9c3c620b80 1 -- 172.21.15.43:0/1333141049 wait complete. 2024-05-27T07:39:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.361+0000 7f9c3c620b80 20 client.27967 trim_cache size 0 max 16384 2024-05-27T07:39:35.359 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:39:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.363+0000 7f9c27e00700 1 -- 172.21.15.43:0/4216551577 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c0c0b2a70 msgr2=0x7f9c0c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.363+0000 7f9c27e00700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c0c0b2a70 0x7f9c0c0b4f80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9c281027b0 tx=0x7f9c14009260 comp rx=0 tx=0).stop 2024-05-27T07:39:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.363+0000 7f9c27e00700 1 -- 172.21.15.43:0/4216551577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c28103870 msgr2=0x7f9c2810f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.363+0000 7f9c27e00700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c28103870 0x7f9c2810f140 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9c180098e0 tx=0x7f9c180047b0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.364+0000 7f9c27e00700 1 -- 172.21.15.43:0/4216551577 shutdown_connections 2024-05-27T07:39:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.364+0000 7f9c27e00700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f9c0c0b2a70 0x7f9c0c0b4f80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.364+0000 7f9c27e00700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c281a6da0 0x7f9c281a71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.364+0000 7f9c27e00700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c28103870 0x7f9c2810f140 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.364+0000 7f9c27e00700 1 --2- 172.21.15.43:0/4216551577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c28110ad0 0x7f9c2810f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.364+0000 7f9c27e00700 1 -- 172.21.15.43:0/4216551577 >> 172.21.15.43:0/4216551577 conn(0x7f9c28078ab0 msgr2=0x7f9c28112120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.364+0000 7f9c27e00700 1 -- 172.21.15.43:0/4216551577 shutdown_connections 2024-05-27T07:39:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:35.364+0000 7f9c27e00700 1 -- 172.21.15.43:0/4216551577 wait complete. 2024-05-27T07:39:35.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 75852 5.81 MB/sec execute 576 sec latency 90.289 ms 2024-05-27T07:39:36.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:39:36.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 76007 5.82 MB/sec execute 577 sec latency 95.788 ms 2024-05-27T07:39:36.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:36 smithi043 ceph-mon[31531]: pgmap v861: 129 pgs: 129 active+clean; 1.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 3.0 MiB/s wr, 150 op/s 2024-05-27T07:39:36.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:36 smithi179 ceph-mon[35888]: pgmap v861: 129 pgs: 129 active+clean; 1.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 3.0 MiB/s wr, 150 op/s 2024-05-27T07:39:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:37.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:36 smithi132 ceph-mon[35283]: pgmap v861: 129 pgs: 129 active+clean; 1.5 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 3.0 MiB/s wr, 150 op/s 2024-05-27T07:39:37.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 76110 5.83 MB/sec execute 578 sec latency 96.400 ms 2024-05-27T07:39:38.663 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:38 smithi043 ceph-mon[31531]: pgmap v862: 129 pgs: 129 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-05-27T07:39:38.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 76252 5.83 MB/sec execute 579 sec latency 169.326 ms 2024-05-27T07:39:38.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:38 smithi179 ceph-mon[35888]: pgmap v862: 129 pgs: 129 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-05-27T07:39:39.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:38 smithi132 ceph-mon[35283]: pgmap v862: 129 pgs: 129 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 5.2 MiB/s wr, 219 op/s 2024-05-27T07:39:39.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 76349 5.83 MB/sec execute 580 sec latency 90.879 ms 2024-05-27T07:39:40.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 76460 5.83 MB/sec execute 581 sec latency 102.180 ms 2024-05-27T07:39:40.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:40 smithi043 ceph-mon[31531]: pgmap v863: 129 pgs: 129 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.8 MiB/s wr, 139 op/s 2024-05-27T07:39:40.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:40 smithi179 ceph-mon[35888]: pgmap v863: 129 pgs: 129 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.8 MiB/s wr, 139 op/s 2024-05-27T07:39:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.091+0000 7fab41a00700 1 -- 172.21.15.43:0/2911915611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab3c109860 msgr2=0x7fab3c1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.091+0000 7fab41a00700 1 --2- 172.21.15.43:0/2911915611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab3c109860 0x7fab3c1111c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fab34009990 tx=0x7fab34009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.091+0000 7fab41a00700 1 -- 172.21.15.43:0/2911915611 shutdown_connections 2024-05-27T07:39:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.091+0000 7fab41a00700 1 --2- 172.21.15.43:0/2911915611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab3c109860 0x7fab3c1111c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.091+0000 7fab41a00700 1 --2- 172.21.15.43:0/2911915611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab3c1045e0 0x7fab3c1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.091+0000 7fab41a00700 1 --2- 172.21.15.43:0/2911915611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab3c108f10 0x7fab3c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.091+0000 7fab41a00700 1 -- 172.21.15.43:0/2911915611 >> 172.21.15.43:0/2911915611 conn(0x7fab3c100030 msgr2=0x7fab3c102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.091+0000 7fab41a00700 1 -- 172.21.15.43:0/2911915611 shutdown_connections 2024-05-27T07:39:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.092+0000 7fab41a00700 1 -- 172.21.15.43:0/2911915611 wait complete. 2024-05-27T07:39:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.093+0000 7fab41a00700 1 Processor -- start 2024-05-27T07:39:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.094+0000 7fab41a00700 1 -- start start 2024-05-27T07:39:41.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.094+0000 7fab41a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab3c1045e0 0x7fab3c19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.094+0000 7fab41a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab3c108f10 0x7fab3c19d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.094+0000 7fab41a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab3c109860 0x7fab3c1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.094+0000 7fab41a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c114390 con 0x7fab3c1045e0 2024-05-27T07:39:41.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.094+0000 7fab41a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c114210 con 0x7fab3c108f10 2024-05-27T07:39:41.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.094+0000 7fab41a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c114510 con 0x7fab3c109860 2024-05-27T07:39:41.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.095+0000 7fab3aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab3c1045e0 0x7fab3c19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.095+0000 7fab3aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab3c1045e0 0x7fab3c19cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54782/0 (socket says 172.21.15.43:54782) 2024-05-27T07:39:41.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.095+0000 7fab3aa00700 1 -- 172.21.15.43:0/832581340 learned_addr learned my addr 172.21.15.43:0/832581340 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:41.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.096+0000 7fab32000700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab3c108f10 0x7fab3c19d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.097+0000 7fab3aa00700 1 -- 172.21.15.43:0/832581340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab3c109860 msgr2=0x7fab3c1a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.097+0000 7fab3b400700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab3c109860 0x7fab3c1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.097+0000 7fab3aa00700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab3c109860 0x7fab3c1a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.097+0000 7fab3aa00700 1 -- 172.21.15.43:0/832581340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab3c108f10 msgr2=0x7fab3c19d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.097+0000 7fab3aa00700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab3c108f10 0x7fab3c19d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.097+0000 7fab3aa00700 1 -- 172.21.15.43:0/832581340 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3c1a1e50 con 0x7fab3c1045e0 2024-05-27T07:39:41.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.097+0000 7fab32000700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab3c108f10 0x7fab3c19d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:41.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.098+0000 7fab3aa00700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab3c1045e0 0x7fab3c19cf80 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00c370 tx=0x7fab2c00c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:41.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.098+0000 7fab33e00700 1 -- 172.21.15.43:0/832581340 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c0090d0 con 0x7fab3c1045e0 2024-05-27T07:39:41.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.099+0000 7fab41a00700 1 -- 172.21.15.43:0/832581340 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3c1a20e0 con 0x7fab3c1045e0 2024-05-27T07:39:41.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.099+0000 7fab41a00700 1 -- 172.21.15.43:0/832581340 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3c1a9a60 con 0x7fab3c1045e0 2024-05-27T07:39:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.100+0000 7fab33e00700 1 -- 172.21.15.43:0/832581340 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab2c009270 con 0x7fab3c1045e0 2024-05-27T07:39:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.100+0000 7fab33e00700 1 -- 172.21.15.43:0/832581340 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c004760 con 0x7fab3c1045e0 2024-05-27T07:39:41.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:40 smithi132 ceph-mon[35283]: pgmap v863: 129 pgs: 129 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 3.8 MiB/s wr, 139 op/s 2024-05-27T07:39:41.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.106+0000 7fab33e00700 1 -- 172.21.15.43:0/832581340 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fab2c01a020 con 0x7fab3c1045e0 2024-05-27T07:39:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.110+0000 7fab33e00700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fab200b2af0 0x7fab200b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.110+0000 7fab50479b80 1 Processor -- start 2024-05-27T07:39:41.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.110+0000 7fab33e00700 1 -- 172.21.15.43:0/832581340 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fab2c0e23d0 con 0x7fab3c1045e0 2024-05-27T07:39:41.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.111+0000 7fab50479b80 1 -- start start 2024-05-27T07:39:41.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.111+0000 7fab32000700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fab200b2af0 0x7fab200b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.113+0000 7fab32000700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fab200b2af0 0x7fab200b5000 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fab3c110540 tx=0x7fab24009550 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:41.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.117+0000 7fab50479b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.118+0000 7fab50479b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b15080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.118+0000 7fab50479b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b159d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.118+0000 7fab50479b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e970a3ae40 con 0x55e970b155c0 2024-05-27T07:39:41.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.118+0000 7fab50479b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e970a2a9e0 con 0x55e970b1b540 2024-05-27T07:39:41.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.118+0000 7fab50479b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e970a2ea20 con 0x55e97098de40 2024-05-27T07:39:41.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.118+0000 7fab32000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b15080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.118+0000 7fab32000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b15080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48536/0 (socket says 172.21.15.43:48536) 2024-05-27T07:39:41.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.118+0000 7fab32000700 1 -- 172.21.15.43:0/2148926556 learned_addr learned my addr 172.21.15.43:0/2148926556 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:41.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.119+0000 7fab3aa00700 1 --2- 172.21.15.43:0/2148926556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.119+0000 7fab32000700 1 -- 172.21.15.43:0/2148926556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 msgr2=0x55e970b1b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:41.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.119+0000 7fab32000700 1 --2- 172.21.15.43:0/2148926556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.119+0000 7fab32000700 1 -- 172.21.15.43:0/2148926556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 msgr2=0x55e970b159d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:39:41.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.119+0000 7fab32000700 1 --2- 172.21.15.43:0/2148926556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.119+0000 7fab32000700 1 -- 172.21.15.43:0/2148926556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9709f3420 con 0x55e970b1b540 2024-05-27T07:39:41.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.120+0000 7fab3aa00700 1 --2- 172.21.15.43:0/2148926556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:39:41.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.120+0000 7fab32000700 1 --2- 172.21.15.43:0/2148926556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b15080 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fab2400ed10 tx=0x7fab24009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=999eea8e39f921d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:41.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.120+0000 7fab2be00700 1 -- 172.21.15.43:0/2148926556 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24011bb0 con 0x55e970b1b540 2024-05-27T07:39:41.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.121+0000 7fab2be00700 1 -- 172.21.15.43:0/2148926556 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab24023070 con 0x55e970b1b540 2024-05-27T07:39:41.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.121+0000 7fab2be00700 1 -- 172.21.15.43:0/2148926556 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24010630 con 0x55e970b1b540 2024-05-27T07:39:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.122+0000 7fab50479b80 1 -- 172.21.15.43:0/2148926556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 msgr2=0x55e970b15080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.122+0000 7fab50479b80 1 --2- 172.21.15.43:0/2148926556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b15080 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fab2400ed10 tx=0x7fab24009f90 comp rx=0 tx=0).stop 2024-05-27T07:39:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.122+0000 7fab50479b80 1 -- 172.21.15.43:0/2148926556 shutdown_connections 2024-05-27T07:39:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.122+0000 7fab50479b80 1 --2- 172.21.15.43:0/2148926556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b159d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.122+0000 7fab50479b80 1 --2- 172.21.15.43:0/2148926556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.122+0000 7fab50479b80 1 --2- 172.21.15.43:0/2148926556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b15080 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.122+0000 7fab50479b80 1 -- 172.21.15.43:0/2148926556 >> 172.21.15.43:0/2148926556 conn(0x55e9709edfc0 msgr2=0x55e970b078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.122+0000 7fab50479b80 1 -- 172.21.15.43:0/2148926556 shutdown_connections 2024-05-27T07:39:41.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.123+0000 7fab50479b80 1 -- 172.21.15.43:0/2148926556 wait complete. 2024-05-27T07:39:41.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.123+0000 7fab50479b80 1 -- start start 2024-05-27T07:39:41.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.123+0000 7fab50479b80 1 Processor -- start 2024-05-27T07:39:41.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.124+0000 7fab50479b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.124+0000 7fab50479b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b1f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.124+0000 7fab50479b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b1f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.124+0000 7fab50479b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e970930fa0 con 0x55e970b155c0 2024-05-27T07:39:41.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.124+0000 7fab50479b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e9709cb760 con 0x55e970b1b540 2024-05-27T07:39:41.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.124+0000 7fab50479b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e970b22010 con 0x55e97098de40 2024-05-27T07:39:41.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.124+0000 7fab50479b80 10 client.0 _ll_register_callbacks cb 0x55e9709d4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:41.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.125+0000 7fab3aa00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.125+0000 7fab3aa00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:58054/0 (socket says 172.21.15.43:58054) 2024-05-27T07:39:41.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.125+0000 7fab3aa00700 1 -- 172.21.15.43:0/2540528630 learned_addr learned my addr 172.21.15.43:0/2540528630 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:41.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.125+0000 7fab32000700 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b1f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.125+0000 7fab3b400700 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b1f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.126+0000 7fab32000700 1 -- 172.21.15.43:0/2540528630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 msgr2=0x55e970b1ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:41.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.126+0000 7fab32000700 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.126+0000 7fab32000700 1 -- 172.21.15.43:0/2540528630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 msgr2=0x55e970b1f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.126+0000 7fab32000700 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b1f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.126+0000 7fab32000700 1 -- 172.21.15.43:0/2540528630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e97097a8d0 con 0x55e970b155c0 2024-05-27T07:39:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.126+0000 7fab3aa00700 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.126+0000 7fab3b400700 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b1f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.126+0000 7fab32000700 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b1f210 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fab24009930 tx=0x7fab2401fee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f6936054b3be1b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.126+0000 7fab28c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:39:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.127+0000 7fab28c00700 1 -- 172.21.15.43:0/2540528630 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24004220 con 0x55e970b155c0 2024-05-27T07:39:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.127+0000 7fab28c00700 1 -- 172.21.15.43:0/2540528630 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fab240043c0 con 0x55e970b155c0 2024-05-27T07:39:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.127+0000 7fab50479b80 1 -- 172.21.15.43:0/2540528630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab24023070 con 0x55e970b155c0 2024-05-27T07:39:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.127+0000 7fab28c00700 1 -- 172.21.15.43:0/2540528630 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fab2401e4c0 con 0x55e970b155c0 2024-05-27T07:39:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.127+0000 7fab50479b80 10 client.18582 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.127+0000 7fab50479b80 1 -- 172.21.15.43:0/2540528630 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55e9709fb060 con 0x55e970b155c0 2024-05-27T07:39:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.128+0000 7fab28c00700 1 -- 172.21.15.43:0/2540528630 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab24023070 con 0x55e970b155c0 2024-05-27T07:39:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.128+0000 7fab28c00700 1 -- 172.21.15.43:0/2540528630 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fab2400f040 con 0x55e970b155c0 2024-05-27T07:39:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.128+0000 7fab50479b80 10 client.18582 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.128+0000 7fab50479b80 10 client.18582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.128+0000 7fab50479b80 20 client.18582 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.128+0000 7fab50479b80 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e970b2f220 0x55e970b31670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.128+0000 7fab50479b80 4 client.18582 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.128+0000 7fab50479b80 1 -- 172.21.15.43:0/2540528630 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e970a3cd90 con 0x55e970b2f220 2024-05-27T07:39:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.129+0000 7fab3aa00700 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e970b2f220 0x55e970b31670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.130+0000 7fab28c00700 10 client.18582 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.130+0000 7fab3aa00700 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e970b2f220 0x55e970b31670 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43a0733ea75b9788 server_cookie=d7ee4f535d7e169e in_seq=0 out_seq=0 2024-05-27T07:39:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab28c00700 1 -- 172.21.15.43:0/2540528630 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e970a3cd90 con 0x55e970b2f220 2024-05-27T07:39:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab28c00700 10 client.18582 handle_command_reply: tid=0 2024-05-27T07:39:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab50479b80 1 client.18582 shutdown 2024-05-27T07:39:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab50479b80 1 -- 172.21.15.43:0/2540528630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 msgr2=0x55e970b1f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab50479b80 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b1f210 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fab24009930 tx=0x7fab2401fee0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab50479b80 1 -- 172.21.15.43:0/2540528630 shutdown_connections 2024-05-27T07:39:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab50479b80 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b1f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab50479b80 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b1ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab50479b80 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e970b2f220 0x55e970b31670 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab50479b80 1 --2- 172.21.15.43:0/2540528630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b1f210 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.131+0000 7fab50479b80 1 -- 172.21.15.43:0/2540528630 >> 172.21.15.43:0/2540528630 conn(0x55e9709edfc0 msgr2=0x55e970b07580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.132+0000 7fab28c00700 0 client.18582 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.132+0000 7fab50479b80 1 -- 172.21.15.43:0/2540528630 shutdown_connections 2024-05-27T07:39:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.132+0000 7fab50479b80 1 -- 172.21.15.43:0/2540528630 wait complete. 2024-05-27T07:39:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.132+0000 7fab50479b80 20 client.18582 trim_cache size 0 max 16384 2024-05-27T07:39:41.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.150+0000 7fab50479b80 1 Processor -- start 2024-05-27T07:39:41.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.151+0000 7fab50479b80 1 -- start start 2024-05-27T07:39:41.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.151+0000 7fab50479b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e97098de40 0x55e970b15080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.151+0000 7fab50479b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b155c0 0x55e970b1dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.151+0000 7fab50479b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e970b1b540 0x55e970b1a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.151+0000 7fab50479b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fab1800f250 con 0x55e97098de40 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.151+0000 7fab50479b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab1800d710 con 0x55e970b155c0 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.151+0000 7fab50479b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fab1800eb80 con 0x55e970b1b540 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.152+0000 7fab3aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e97098de40 0x55e970b15080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.152+0000 7fab3aa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e97098de40 0x55e970b15080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54810/0 (socket says 172.21.15.43:54810) 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.152+0000 7fab3aa00700 1 -- 172.21.15.43:0/3233349558 learned_addr learned my addr 172.21.15.43:0/3233349558 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.152+0000 7fab32000700 1 --2- 172.21.15.43:0/3233349558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b155c0 0x55e970b1dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.152+0000 7fab3aa00700 1 -- 172.21.15.43:0/3233349558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e970b1b540 msgr2=0x55e970b1a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.152+0000 7fab3aa00700 1 --2- 172.21.15.43:0/3233349558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e970b1b540 0x55e970b1a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.152+0000 7fab3aa00700 1 -- 172.21.15.43:0/3233349558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b155c0 msgr2=0x55e970b1dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.152+0000 7fab3aa00700 1 --2- 172.21.15.43:0/3233349558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b155c0 0x55e970b1dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.152+0000 7fab3aa00700 1 -- 172.21.15.43:0/3233349558 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e970b1fac0 con 0x55e97098de40 2024-05-27T07:39:41.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.153+0000 7fab3aa00700 1 --2- 172.21.15.43:0/3233349558 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e97098de40 0x55e970b15080 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fab2c019ff0 tx=0x7fab2c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d0afb20c3d364f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:41.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.153+0000 7fab2b400700 1 -- 172.21.15.43:0/3233349558 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c0a9980 con 0x55e97098de40 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.153+0000 7fab2b400700 1 -- 172.21.15.43:0/3233349558 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab2c0a9280 con 0x55e97098de40 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.153+0000 7fab2b400700 1 -- 172.21.15.43:0/3233349558 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c0b1c50 con 0x55e97098de40 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.154+0000 7fab50479b80 1 -- 172.21.15.43:0/3233349558 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e97098de40 msgr2=0x55e970b15080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.154+0000 7fab50479b80 1 --2- 172.21.15.43:0/3233349558 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e97098de40 0x55e970b15080 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fab2c019ff0 tx=0x7fab2c00cf50 comp rx=0 tx=0).stop 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.154+0000 7fab50479b80 1 -- 172.21.15.43:0/3233349558 shutdown_connections 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.154+0000 7fab50479b80 1 --2- 172.21.15.43:0/3233349558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e970b1b540 0x55e970b1a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.154+0000 7fab50479b80 1 --2- 172.21.15.43:0/3233349558 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e97098de40 0x55e970b15080 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.154+0000 7fab50479b80 1 --2- 172.21.15.43:0/3233349558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b155c0 0x55e970b1dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.154+0000 7fab50479b80 1 -- 172.21.15.43:0/3233349558 >> 172.21.15.43:0/3233349558 conn(0x55e9709edfc0 msgr2=0x55e970b18950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.154+0000 7fab50479b80 1 -- 172.21.15.43:0/3233349558 shutdown_connections 2024-05-27T07:39:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.155+0000 7fab50479b80 1 -- 172.21.15.43:0/3233349558 wait complete. 2024-05-27T07:39:41.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.155+0000 7fab50479b80 1 -- start start 2024-05-27T07:39:41.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.155+0000 7fab50479b80 1 Processor -- start 2024-05-27T07:39:41.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.156+0000 7fab50479b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b374c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.156+0000 7fab50479b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b272d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.156+0000 7fab50479b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b27810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.156+0000 7fab50479b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fab1800e4b0 con 0x55e970b155c0 2024-05-27T07:39:41.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.156+0000 7fab50479b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab1800dde0 con 0x55e970b1b540 2024-05-27T07:39:41.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.156+0000 7fab50479b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e970b27e00 con 0x55e97098de40 2024-05-27T07:39:41.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.156+0000 7fab50479b80 10 client.0 _ll_register_callbacks cb 0x55e9709d4c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:41.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.157+0000 7fab3b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b27810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.157+0000 7fab3b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b27810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:48566/0 (socket says 172.21.15.43:48566) 2024-05-27T07:39:41.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.157+0000 7fab3b400700 1 -- 172.21.15.43:0/2641619060 learned_addr learned my addr 172.21.15.43:0/2641619060 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:41.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.157+0000 7fab3aa00700 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.157+0000 7fab32000700 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b272d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.158+0000 7fab3b400700 1 -- 172.21.15.43:0/2641619060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 msgr2=0x55e970b374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:41.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.158+0000 7fab3b400700 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b374c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.158+0000 7fab3b400700 1 -- 172.21.15.43:0/2641619060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 msgr2=0x55e970b272d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:41.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.158+0000 7fab3b400700 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b272d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.158+0000 7fab3b400700 1 -- 172.21.15.43:0/2641619060 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e970b35de0 con 0x55e970b1b540 2024-05-27T07:39:41.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.158+0000 7fab3aa00700 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b374c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:41.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.158+0000 7fab32000700 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b272d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:39:41.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.160+0000 7fab3b400700 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b27810 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fab34009990 tx=0x7fab3400bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68c671cd559a7ff9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:41.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.160+0000 7fab28c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:39:41.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.161+0000 7fab28c00700 1 -- 172.21.15.43:0/2641619060 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab340052e0 con 0x55e970b1b540 2024-05-27T07:39:41.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.161+0000 7fab28c00700 1 -- 172.21.15.43:0/2641619060 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fab34005480 con 0x55e970b1b540 2024-05-27T07:39:41.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.161+0000 7fab50479b80 1 -- 172.21.15.43:0/2641619060 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab2c0a9280 con 0x55e970b1b540 2024-05-27T07:39:41.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.162+0000 7fab28c00700 1 -- 172.21.15.43:0/2641619060 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fab34005680 con 0x55e970b1b540 2024-05-27T07:39:41.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.162+0000 7fab50479b80 10 client.37970 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:41.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.162+0000 7fab50479b80 1 -- 172.21.15.43:0/2641619060 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55e970b98eb0 con 0x55e970b1b540 2024-05-27T07:39:41.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.162+0000 7fab28c00700 1 -- 172.21.15.43:0/2641619060 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fab2c0a9280 con 0x55e970b1b540 2024-05-27T07:39:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.163+0000 7fab28c00700 1 -- 172.21.15.43:0/2641619060 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fab340039f0 con 0x55e970b1b540 2024-05-27T07:39:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.163+0000 7fab50479b80 10 client.37970 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.163+0000 7fab50479b80 10 client.37970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.163+0000 7fab50479b80 20 client.37970 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.163+0000 7fab50479b80 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e970b995c0 0x55e970b9b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.163+0000 7fab50479b80 4 client.37970 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:39:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.163+0000 7fab50479b80 1 -- 172.21.15.43:0/2641619060 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e970a36570 con 0x55e970b995c0 2024-05-27T07:39:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.163+0000 7fab3aa00700 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e970b995c0 0x55e970b9b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:41.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.165+0000 7fab3aa00700 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e970b995c0 0x55e970b9b9e0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26a7c12684553394 server_cookie=a6b787601b55f9f3 in_seq=0 out_seq=0 2024-05-27T07:39:41.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.165+0000 7fab28c00700 10 client.37970 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:41.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.165+0000 7fab28c00700 1 -- 172.21.15.43:0/2641619060 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e970a36570 con 0x55e970b995c0 2024-05-27T07:39:41.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.165+0000 7fab28c00700 10 client.37970 handle_command_reply: tid=0 2024-05-27T07:39:41.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.165+0000 7fab50479b80 1 client.37970 shutdown 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.165+0000 7fab50479b80 1 -- 172.21.15.43:0/2641619060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 msgr2=0x55e970b27810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.165+0000 7fab50479b80 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b27810 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fab34009990 tx=0x7fab3400bf50 comp rx=0 tx=0).stop 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.166+0000 7fab50479b80 1 -- 172.21.15.43:0/2641619060 shutdown_connections 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.166+0000 7fab50479b80 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e970b1b540 0x55e970b27810 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.166+0000 7fab50479b80 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e97098de40 0x55e970b374c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.166+0000 7fab50479b80 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e970b995c0 0x55e970b9b9e0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.166+0000 7fab50479b80 1 --2- 172.21.15.43:0/2641619060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e970b155c0 0x55e970b272d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.166+0000 7fab50479b80 1 -- 172.21.15.43:0/2641619060 >> 172.21.15.43:0/2641619060 conn(0x55e9709edfc0 msgr2=0x55e970b1dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.166+0000 7fab28c00700 0 client.37970 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.166+0000 7fab50479b80 1 -- 172.21.15.43:0/2641619060 shutdown_connections 2024-05-27T07:39:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.166+0000 7fab50479b80 1 -- 172.21.15.43:0/2641619060 wait complete. 2024-05-27T07:39:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.166+0000 7fab50479b80 20 client.37970 trim_cache size 0 max 16384 2024-05-27T07:39:41.165 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:39:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.167+0000 7fab41a00700 1 -- 172.21.15.43:0/832581340 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fab200b2af0 msgr2=0x7fab200b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.167+0000 7fab41a00700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fab200b2af0 0x7fab200b5000 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fab3c110540 tx=0x7fab24009550 comp rx=0 tx=0).stop 2024-05-27T07:39:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.167+0000 7fab41a00700 1 -- 172.21.15.43:0/832581340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab3c1045e0 msgr2=0x7fab3c19cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.167+0000 7fab41a00700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab3c1045e0 0x7fab3c19cf80 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00c370 tx=0x7fab2c00c730 comp rx=0 tx=0).stop 2024-05-27T07:39:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.167+0000 7fab41a00700 1 -- 172.21.15.43:0/832581340 shutdown_connections 2024-05-27T07:39:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.167+0000 7fab41a00700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fab200b2af0 0x7fab200b5000 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.167+0000 7fab41a00700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab3c109860 0x7fab3c1a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.167+0000 7fab41a00700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fab3c1045e0 0x7fab3c19cf80 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.167+0000 7fab41a00700 1 --2- 172.21.15.43:0/832581340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab3c108f10 0x7fab3c19d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.167+0000 7fab41a00700 1 -- 172.21.15.43:0/832581340 >> 172.21.15.43:0/832581340 conn(0x7fab3c100030 msgr2=0x7fab3c110e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.168+0000 7fab41a00700 1 -- 172.21.15.43:0/832581340 shutdown_connections 2024-05-27T07:39:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:41.168+0000 7fab41a00700 1 -- 172.21.15.43:0/832581340 wait complete. 2024-05-27T07:39:41.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 76535 5.83 MB/sec execute 582 sec latency 105.462 ms 2024-05-27T07:39:42.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 76650 5.83 MB/sec execute 583 sec latency 87.100 ms 2024-05-27T07:39:42.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:42 smithi043 ceph-mon[31531]: pgmap v864: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 205 op/s 2024-05-27T07:39:42.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:42 smithi179 ceph-mon[35888]: pgmap v864: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 205 op/s 2024-05-27T07:39:43.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:42 smithi132 ceph-mon[35283]: pgmap v864: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 205 op/s 2024-05-27T07:39:43.208 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-27T07:39:43.208 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-05-27T07:39:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:43.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 76767 5.83 MB/sec execute 584 sec latency 104.506 ms 2024-05-27T07:39:44.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 76877 5.83 MB/sec execute 585 sec latency 91.027 ms 2024-05-27T07:39:44.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:44 smithi043 ceph-mon[31531]: pgmap v865: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 134 op/s 2024-05-27T07:39:44.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:44 smithi179 ceph-mon[35888]: pgmap v865: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 134 op/s 2024-05-27T07:39:45.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:44 smithi132 ceph-mon[35283]: pgmap v865: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 134 op/s 2024-05-27T07:39:45.843 INFO:tasks.workunit.client.0.smithi043.stdout: 10 76993 5.83 MB/sec execute 586 sec latency 121.016 ms 2024-05-27T07:39:46.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.325+0000 7fdd4ba00700 1 -- 172.21.15.43:0/642775125 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd441045c0 msgr2=0x7fdd441049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.325+0000 7fdd4ba00700 1 --2- 172.21.15.43:0/642775125 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd441045c0 0x7fdd441049d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009990 tx=0x7fdd38009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.326+0000 7fdd4ba00700 1 -- 172.21.15.43:0/642775125 shutdown_connections 2024-05-27T07:39:46.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.326+0000 7fdd4ba00700 1 --2- 172.21.15.43:0/642775125 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd44109840 0x7fdd441111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.326+0000 7fdd4ba00700 1 --2- 172.21.15.43:0/642775125 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd441045c0 0x7fdd441049d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.327+0000 7fdd4ba00700 1 --2- 172.21.15.43:0/642775125 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd44108ef0 0x7fdd44109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.327+0000 7fdd4ba00700 1 -- 172.21.15.43:0/642775125 >> 172.21.15.43:0/642775125 conn(0x7fdd440ffe70 msgr2=0x7fdd441022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:46.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.327+0000 7fdd4ba00700 1 -- 172.21.15.43:0/642775125 shutdown_connections 2024-05-27T07:39:46.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.327+0000 7fdd4ba00700 1 -- 172.21.15.43:0/642775125 wait complete. 2024-05-27T07:39:46.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.329+0000 7fdd4ba00700 1 Processor -- start 2024-05-27T07:39:46.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.329+0000 7fdd4ba00700 1 -- start start 2024-05-27T07:39:46.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.330+0000 7fdd4ba00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd441045c0 0x7fdd4419ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.330+0000 7fdd4ba00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44108ef0 0x7fdd4419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.330+0000 7fdd4ba00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd44109840 0x7fdd4419f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.330+0000 7fdd4ba00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdd44114370 con 0x7fdd44109840 2024-05-27T07:39:46.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.330+0000 7fdd4ba00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdd441141f0 con 0x7fdd44108ef0 2024-05-27T07:39:46.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.330+0000 7fdd4ba00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdd441144f0 con 0x7fdd441045c0 2024-05-27T07:39:46.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.330+0000 7fdd43e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44108ef0 0x7fdd4419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.330+0000 7fdd43e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44108ef0 0x7fdd4419b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52832/0 (socket says 172.21.15.43:52832) 2024-05-27T07:39:46.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.330+0000 7fdd43e00700 1 -- 172.21.15.43:0/979818062 learned_addr learned my addr 172.21.15.43:0/979818062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:46.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.331+0000 7fdd49000700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd441045c0 0x7fdd4419ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.331+0000 7fdd49a00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd44109840 0x7fdd4419f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.331+0000 7fdd43e00700 1 -- 172.21.15.43:0/979818062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd441045c0 msgr2=0x7fdd4419ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.331+0000 7fdd43e00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd441045c0 0x7fdd4419ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.331+0000 7fdd43e00700 1 -- 172.21.15.43:0/979818062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd44109840 msgr2=0x7fdd4419f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.331+0000 7fdd43e00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd44109840 0x7fdd4419f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.331+0000 7fdd43e00700 1 -- 172.21.15.43:0/979818062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4419fc10 con 0x7fdd44108ef0 2024-05-27T07:39:46.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.332+0000 7fdd43e00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44108ef0 0x7fdd4419b2e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000e970 tx=0x7fdd3000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:46.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.333+0000 7fdd41600700 1 -- 172.21.15.43:0/979818062 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30013be0 con 0x7fdd44108ef0 2024-05-27T07:39:46.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.333+0000 7fdd4ba00700 1 -- 172.21.15.43:0/979818062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4419ff00 con 0x7fdd44108ef0 2024-05-27T07:39:46.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.333+0000 7fdd4ba00700 1 -- 172.21.15.43:0/979818062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd44076af0 con 0x7fdd44108ef0 2024-05-27T07:39:46.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.335+0000 7fdd41600700 1 -- 172.21.15.43:0/979818062 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd30013d80 con 0x7fdd44108ef0 2024-05-27T07:39:46.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.335+0000 7fdd41600700 1 -- 172.21.15.43:0/979818062 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30005020 con 0x7fdd44108ef0 2024-05-27T07:39:46.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.341+0000 7fdd41600700 1 -- 172.21.15.43:0/979818062 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fdd300051c0 con 0x7fdd44108ef0 2024-05-27T07:39:46.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.345+0000 7fdd5a524b80 1 Processor -- start 2024-05-27T07:39:46.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.345+0000 7fdd41600700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd2c0bb200 0x7fdd2c0bd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.345+0000 7fdd5a524b80 1 -- start start 2024-05-27T07:39:46.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.346+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8aff80a70 0x55d8b0013430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.346+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8afe94ec0 0x55d8b0013970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.346+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8b002f190 0x55d8b00315b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.346+0000 7fdd5a524b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d8aff41ec0 con 0x55d8b002f190 2024-05-27T07:39:46.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.346+0000 7fdd5a524b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d8aff31a60 con 0x55d8afe94ec0 2024-05-27T07:39:46.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.346+0000 7fdd5a524b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d8aff35aa0 con 0x55d8aff80a70 2024-05-27T07:39:46.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.346+0000 7fdd49a00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd2c0bb200 0x7fdd2c0bd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.346+0000 7fdd41600700 1 -- 172.21.15.43:0/979818062 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fdd300e2730 con 0x7fdd44108ef0 2024-05-27T07:39:46.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.347+0000 7fdd43e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8afe94ec0 0x55d8b0013970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.347+0000 7fdd43e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8afe94ec0 0x55d8b0013970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52834/0 (socket says 172.21.15.43:52834) 2024-05-27T07:39:46.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.347+0000 7fdd43e00700 1 -- 172.21.15.43:0/2012088081 learned_addr learned my addr 172.21.15.43:0/2012088081 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:46.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.348+0000 7fdd49a00700 1 --2- 172.21.15.43:0/2012088081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8b002f190 0x55d8b00315b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.348+0000 7fdd43e00700 1 -- 172.21.15.43:0/2012088081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8aff80a70 msgr2=0x55d8b0013430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:46.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.348+0000 7fdd43e00700 1 --2- 172.21.15.43:0/2012088081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8aff80a70 0x55d8b0013430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.348+0000 7fdd43e00700 1 -- 172.21.15.43:0/2012088081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8b002f190 msgr2=0x55d8b00315b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:46.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.348+0000 7fdd43e00700 1 --2- 172.21.15.43:0/2012088081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8b002f190 0x55d8b00315b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.348+0000 7fdd43e00700 1 -- 172.21.15.43:0/2012088081 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8afefa4a0 con 0x55d8afe94ec0 2024-05-27T07:39:46.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.349+0000 7fdd43e00700 1 --2- 172.21.15.43:0/2012088081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8afe94ec0 0x55d8b0013970 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000c5b0 tx=0x7fdd30013f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87d16c276ce3d0c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:46.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.349+0000 7fdd36000700 1 -- 172.21.15.43:0/2012088081 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd300a96f0 con 0x55d8afe94ec0 2024-05-27T07:39:46.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.349+0000 7fdd36000700 1 -- 172.21.15.43:0/2012088081 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd300adaa0 con 0x55d8afe94ec0 2024-05-27T07:39:46.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.349+0000 7fdd36000700 1 -- 172.21.15.43:0/2012088081 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd300054b0 con 0x55d8afe94ec0 2024-05-27T07:39:46.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.350+0000 7fdd5a524b80 1 -- 172.21.15.43:0/2012088081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8afe94ec0 msgr2=0x55d8b0013970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.350+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/2012088081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8afe94ec0 0x55d8b0013970 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000c5b0 tx=0x7fdd30013f30 comp rx=0 tx=0).stop 2024-05-27T07:39:46.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.353+0000 7fdd49a00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd2c0bb200 0x7fdd2c0bd710 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009510 tx=0x7fdd3c009920 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:46.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.354+0000 7fdd5a524b80 1 -- 172.21.15.43:0/2012088081 shutdown_connections 2024-05-27T07:39:46.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.354+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/2012088081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8b002f190 0x55d8b00315b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.354+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/2012088081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8aff80a70 0x55d8b0013430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.354+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/2012088081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8afe94ec0 0x55d8b0013970 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.354+0000 7fdd5a524b80 1 -- 172.21.15.43:0/2012088081 >> 172.21.15.43:0/2012088081 conn(0x55d8aff2e410 msgr2=0x55d8b000f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:46.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.354+0000 7fdd5a524b80 1 -- 172.21.15.43:0/2012088081 shutdown_connections 2024-05-27T07:39:46.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.354+0000 7fdd5a524b80 1 -- 172.21.15.43:0/2012088081 wait complete. 2024-05-27T07:39:46.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.355+0000 7fdd5a524b80 1 -- start start 2024-05-27T07:39:46.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.355+0000 7fdd5a524b80 1 Processor -- start 2024-05-27T07:39:46.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.357+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0022770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.357+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8aff80a70 0x55d8b0022d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.357+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 0x55d8b001db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.357+0000 7fdd5a524b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d8afe38020 con 0x55d8aff80a70 2024-05-27T07:39:46.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.357+0000 7fdd49000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0022770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.357+0000 7fdd49000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0022770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:57134/0 (socket says 172.21.15.43:57134) 2024-05-27T07:39:46.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.357+0000 7fdd49000700 1 -- 172.21.15.43:0/895785374 learned_addr learned my addr 172.21.15.43:0/895785374 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:46.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.357+0000 7fdd5a524b80 1 -- 172.21.15.43:0/895785374 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d8afed27e0 con 0x55d8b002f190 2024-05-27T07:39:46.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.357+0000 7fdd5a524b80 1 -- 172.21.15.43:0/895785374 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d8b0016240 con 0x55d8afe94ec0 2024-05-27T07:39:46.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.358+0000 7fdd49a00700 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 0x55d8b001db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.358+0000 7fdd5a524b80 10 client.0 _ll_register_callbacks cb 0x55d8afedbcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:46.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.358+0000 7fdd49000700 1 -- 172.21.15.43:0/895785374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 msgr2=0x55d8b001db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:46.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.358+0000 7fdd49000700 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 0x55d8b001db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.358+0000 7fdd49000700 1 -- 172.21.15.43:0/895785374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8aff80a70 msgr2=0x55d8b0022d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.358+0000 7fdd49000700 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8aff80a70 0x55d8b0022d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.358+0000 7fdd49000700 1 -- 172.21.15.43:0/895785374 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8afe81950 con 0x55d8afe94ec0 2024-05-27T07:39:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.359+0000 7fdd49000700 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0022770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009990 tx=0x7fdd38005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e6ec70c7be14ffb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.360+0000 7fdd2aa00700 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-05-27T07:39:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.360+0000 7fdd2aa00700 1 -- 172.21.15.43:0/895785374 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38027060 con 0x55d8afe94ec0 2024-05-27T07:39:46.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.361+0000 7fdd2aa00700 1 -- 172.21.15.43:0/895785374 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fdd38004b80 con 0x55d8afe94ec0 2024-05-27T07:39:46.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.361+0000 7fdd5a524b80 1 -- 172.21.15.43:0/895785374 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd300adaa0 con 0x55d8afe94ec0 2024-05-27T07:39:46.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.363+0000 7fdd2aa00700 1 -- 172.21.15.43:0/895785374 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fdd3802e050 con 0x55d8afe94ec0 2024-05-27T07:39:46.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.363+0000 7fdd5a524b80 10 client.27985 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:46.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.363+0000 7fdd5a524b80 1 -- 172.21.15.43:0/895785374 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55d8aff020e0 con 0x55d8afe94ec0 2024-05-27T07:39:46.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.363+0000 7fdd2aa00700 1 -- 172.21.15.43:0/895785374 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd300adaa0 con 0x55d8afe94ec0 2024-05-27T07:39:46.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.363+0000 7fdd2aa00700 1 -- 172.21.15.43:0/895785374 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fdd38023530 con 0x55d8afe94ec0 2024-05-27T07:39:46.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.364+0000 7fdd5a524b80 10 client.27985 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:46.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.364+0000 7fdd5a524b80 10 client.27985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:46.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.364+0000 7fdd5a524b80 20 client.27985 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:46.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.364+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d8b002dc70 0x55d8b002e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.364+0000 7fdd5a524b80 4 client.27985 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:46.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.364+0000 7fdd5a524b80 1 -- 172.21.15.43:0/895785374 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8aff43e10 con 0x55d8b002dc70 2024-05-27T07:39:46.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.366+0000 7fdd43e00700 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d8b002dc70 0x55d8b002e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.370+0000 7fdd43e00700 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d8b002dc70 0x55d8b002e130 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5542ad2d43e5b5b5 server_cookie=26fdd4d3a9486aee in_seq=0 out_seq=0 2024-05-27T07:39:46.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.370+0000 7fdd2aa00700 10 client.27985 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:46.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd2aa00700 1 -- 172.21.15.43:0/895785374 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55d8aff43e10 con 0x55d8b002dc70 2024-05-27T07:39:46.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd2aa00700 10 client.27985 handle_command_reply: tid=0 2024-05-27T07:39:46.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd5a524b80 1 client.27985 shutdown 2024-05-27T07:39:46.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd5a524b80 1 -- 172.21.15.43:0/895785374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 msgr2=0x55d8b0022770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0022770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009990 tx=0x7fdd38005750 comp rx=0 tx=0).stop 2024-05-27T07:39:46.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd5a524b80 1 -- 172.21.15.43:0/895785374 shutdown_connections 2024-05-27T07:39:46.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 0x55d8b001db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0022770 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d8b002dc70 0x55d8b002e130 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/895785374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8aff80a70 0x55d8b0022d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.373+0000 7fdd5a524b80 1 -- 172.21.15.43:0/895785374 >> 172.21.15.43:0/895785374 conn(0x55d8aff2e410 msgr2=0x55d8b0013430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.374+0000 7fdd2aa00700 0 client.27985 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.375+0000 7fdd5a524b80 1 -- 172.21.15.43:0/895785374 shutdown_connections 2024-05-27T07:39:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.375+0000 7fdd5a524b80 1 -- 172.21.15.43:0/895785374 wait complete. 2024-05-27T07:39:46.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.376+0000 7fdd5a524b80 20 client.27985 trim_cache size 0 max 16384 2024-05-27T07:39:46.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.429+0000 7fdd5a524b80 1 Processor -- start 2024-05-27T07:39:46.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.429+0000 7fdd5a524b80 1 -- start start 2024-05-27T07:39:46.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.430+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0013a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.430+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8aff80a70 0x55d8b0019350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.430+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8b002f190 0x55d8b00156a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.430+0000 7fdd5a524b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2400f330 con 0x55d8b002f190 2024-05-27T07:39:46.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.430+0000 7fdd5a524b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2400d7f0 con 0x55d8aff80a70 2024-05-27T07:39:46.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.430+0000 7fdd5a524b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2400ec60 con 0x55d8afe94ec0 2024-05-27T07:39:46.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.430+0000 7fdd43e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8aff80a70 0x55d8b0019350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.430+0000 7fdd43e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8aff80a70 0x55d8b0019350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:52852/0 (socket says 172.21.15.43:52852) 2024-05-27T07:39:46.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.430+0000 7fdd43e00700 1 -- 172.21.15.43:0/2050143211 learned_addr learned my addr 172.21.15.43:0/2050143211 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:46.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.431+0000 7fdd49000700 1 --2- 172.21.15.43:0/2050143211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0013a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.431+0000 7fdd49a00700 1 --2- 172.21.15.43:0/2050143211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8b002f190 0x55d8b00156a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.431+0000 7fdd49000700 1 -- 172.21.15.43:0/2050143211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8aff80a70 msgr2=0x55d8b0019350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.431+0000 7fdd49000700 1 --2- 172.21.15.43:0/2050143211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8aff80a70 0x55d8b0019350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.431+0000 7fdd49000700 1 -- 172.21.15.43:0/2050143211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8b002f190 msgr2=0x55d8b00156a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.431+0000 7fdd49000700 1 --2- 172.21.15.43:0/2050143211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8b002f190 0x55d8b00156a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.431+0000 7fdd49000700 1 -- 172.21.15.43:0/2050143211 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8b001d940 con 0x55d8afe94ec0 2024-05-27T07:39:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.432+0000 7fdd49000700 1 --2- 172.21.15.43:0/2050143211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0013a80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd380169d0 tx=0x7fdd38009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5832d630824e1c96 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.433+0000 7fdd35600700 1 -- 172.21.15.43:0/2050143211 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38027070 con 0x55d8afe94ec0 2024-05-27T07:39:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.434+0000 7fdd35600700 1 -- 172.21.15.43:0/2050143211 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd380095d0 con 0x55d8afe94ec0 2024-05-27T07:39:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.434+0000 7fdd35600700 1 -- 172.21.15.43:0/2050143211 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38016620 con 0x55d8afe94ec0 2024-05-27T07:39:46.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.436+0000 7fdd5a524b80 1 -- 172.21.15.43:0/2050143211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 msgr2=0x55d8b0013a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.436+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/2050143211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0013a80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd380169d0 tx=0x7fdd38009ca0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.437+0000 7fdd5a524b80 1 -- 172.21.15.43:0/2050143211 shutdown_connections 2024-05-27T07:39:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.437+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/2050143211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8b002f190 0x55d8b00156a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.437+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/2050143211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0013a80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.437+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/2050143211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8aff80a70 0x55d8b0019350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.437+0000 7fdd5a524b80 1 -- 172.21.15.43:0/2050143211 >> 172.21.15.43:0/2050143211 conn(0x55d8aff2e410 msgr2=0x55d8b0014f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:46.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.438+0000 7fdd5a524b80 1 -- 172.21.15.43:0/2050143211 shutdown_connections 2024-05-27T07:39:46.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.438+0000 7fdd5a524b80 1 -- 172.21.15.43:0/2050143211 wait complete. 2024-05-27T07:39:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.438+0000 7fdd5a524b80 1 -- start start 2024-05-27T07:39:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.439+0000 7fdd5a524b80 1 Processor -- start 2024-05-27T07:39:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.439+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0023f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.439+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8aff80a70 0x55d8b00244a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.439+0000 7fdd5a524b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 0x55d8b0028520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.439+0000 7fdd5a524b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2400e590 con 0x55d8aff80a70 2024-05-27T07:39:46.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.439+0000 7fdd5a524b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2400dec0 con 0x55d8b002f190 2024-05-27T07:39:46.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.439+0000 7fdd5a524b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55d8b0028a60 con 0x55d8afe94ec0 2024-05-27T07:39:46.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.440+0000 7fdd5a524b80 10 client.0 _ll_register_callbacks cb 0x55d8afedbcf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:46.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.440+0000 7fdd49000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0023f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.440+0000 7fdd49000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0023f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:57162/0 (socket says 172.21.15.43:57162) 2024-05-27T07:39:46.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.440+0000 7fdd49000700 1 -- 172.21.15.43:0/4160882225 learned_addr learned my addr 172.21.15.43:0/4160882225 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:46.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.440+0000 7fdd49a00700 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 0x55d8b0028520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.440+0000 7fdd43e00700 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8aff80a70 0x55d8b00244a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.441+0000 7fdd49a00700 1 -- 172.21.15.43:0/4160882225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 msgr2=0x55d8b0023f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:46.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.441+0000 7fdd49a00700 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0023f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.441+0000 7fdd49a00700 1 -- 172.21.15.43:0/4160882225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8aff80a70 msgr2=0x55d8b00244a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:46.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.441+0000 7fdd49a00700 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8aff80a70 0x55d8b00244a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.441+0000 7fdd49a00700 1 -- 172.21.15.43:0/4160882225 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8b0028c60 con 0x55d8b002f190 2024-05-27T07:39:46.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.441+0000 7fdd49000700 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0023f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:46.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.442+0000 7fdd49a00700 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 0x55d8b0028520 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009fd0 tx=0x7fdd3c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3469e07dd47c177c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:46.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.442+0000 7fdd2aa00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:39:46.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.443+0000 7fdd2aa00700 1 -- 172.21.15.43:0/4160882225 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c02b060 con 0x55d8b002f190 2024-05-27T07:39:46.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.443+0000 7fdd2aa00700 1 -- 172.21.15.43:0/4160882225 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fdd3c0185f0 con 0x55d8b002f190 2024-05-27T07:39:46.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.444+0000 7fdd5a524b80 1 -- 172.21.15.43:0/4160882225 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd380095d0 con 0x55d8b002f190 2024-05-27T07:39:46.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.444+0000 7fdd2aa00700 1 -- 172.21.15.43:0/4160882225 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fdd3c0187f0 con 0x55d8b002f190 2024-05-27T07:39:46.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.444+0000 7fdd5a524b80 10 client.37988 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:46.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.444+0000 7fdd5a524b80 1 -- 172.21.15.43:0/4160882225 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x55d8b0099e50 con 0x55d8b002f190 2024-05-27T07:39:46.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.445+0000 7fdd2aa00700 1 -- 172.21.15.43:0/4160882225 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdd380095d0 con 0x55d8b002f190 2024-05-27T07:39:46.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.445+0000 7fdd2aa00700 1 -- 172.21.15.43:0/4160882225 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fdd3c026b00 con 0x55d8b002f190 2024-05-27T07:39:46.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.445+0000 7fdd5a524b80 10 client.37988 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:46.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.445+0000 7fdd5a524b80 10 client.37988 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:46.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.445+0000 7fdd5a524b80 20 client.37988 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:46.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.445+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d8b009a520 0x55d8b009c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:46.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.445+0000 7fdd5a524b80 4 client.37988 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-27T07:39:46.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.445+0000 7fdd5a524b80 1 -- 172.21.15.43:0/4160882225 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d8aff3d5f0 con 0x55d8b009a520 2024-05-27T07:39:46.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.447+0000 7fdd49000700 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d8b009a520 0x55d8b009c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:46.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.447+0000 7fdd49000700 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d8b009a520 0x55d8b009c940 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bbfaf395c14da4d server_cookie=c9860147536712b9 in_seq=0 out_seq=0 2024-05-27T07:39:46.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.447+0000 7fdd2aa00700 10 client.37988 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:46.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.449+0000 7fdd2aa00700 1 -- 172.21.15.43:0/4160882225 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d8aff3d5f0 con 0x55d8b009a520 2024-05-27T07:39:46.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.449+0000 7fdd2aa00700 10 client.37988 handle_command_reply: tid=0 2024-05-27T07:39:46.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.449+0000 7fdd5a524b80 1 client.37988 shutdown 2024-05-27T07:39:46.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.450+0000 7fdd5a524b80 1 -- 172.21.15.43:0/4160882225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 msgr2=0x55d8b0028520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.450+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 0x55d8b0028520 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009fd0 tx=0x7fdd3c004ad0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.450+0000 7fdd5a524b80 1 -- 172.21.15.43:0/4160882225 shutdown_connections 2024-05-27T07:39:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.450+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d8b002f190 0x55d8b0028520 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.450+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d8afe94ec0 0x55d8b0023f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.450+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55d8b009a520 0x55d8b009c940 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.450+0000 7fdd5a524b80 1 --2- 172.21.15.43:0/4160882225 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d8aff80a70 0x55d8b00244a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.450+0000 7fdd2aa00700 0 client.37988 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:46.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.450+0000 7fdd5a524b80 1 -- 172.21.15.43:0/4160882225 >> 172.21.15.43:0/4160882225 conn(0x55d8aff2e410 msgr2=0x55d8b00159c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:46.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.451+0000 7fdd5a524b80 1 -- 172.21.15.43:0/4160882225 shutdown_connections 2024-05-27T07:39:46.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.451+0000 7fdd5a524b80 1 -- 172.21.15.43:0/4160882225 wait complete. 2024-05-27T07:39:46.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.451+0000 7fdd5a524b80 20 client.37988 trim_cache size 0 max 16384 2024-05-27T07:39:46.452 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:39:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-05-27T07:39:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "564173bd-1d1d-4299-9f55-3fc5f6126006", 2024-05-27T07:39:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-05-27T07:39:46.452 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:39:46.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.452+0000 7fdd4ba00700 1 -- 172.21.15.43:0/979818062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd2c0bb200 msgr2=0x7fdd2c0bd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.452+0000 7fdd4ba00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd2c0bb200 0x7fdd2c0bd710 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009510 tx=0x7fdd3c009920 comp rx=0 tx=0).stop 2024-05-27T07:39:46.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.452+0000 7fdd4ba00700 1 -- 172.21.15.43:0/979818062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44108ef0 msgr2=0x7fdd4419b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:46.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.452+0000 7fdd4ba00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44108ef0 0x7fdd4419b2e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000e970 tx=0x7fdd3000ed30 comp rx=0 tx=0).stop 2024-05-27T07:39:46.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.452+0000 7fdd4ba00700 1 -- 172.21.15.43:0/979818062 shutdown_connections 2024-05-27T07:39:46.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.452+0000 7fdd4ba00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fdd2c0bb200 0x7fdd2c0bd710 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.452+0000 7fdd4ba00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd44109840 0x7fdd4419f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.452+0000 7fdd4ba00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd441045c0 0x7fdd4419ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.452+0000 7fdd4ba00700 1 --2- 172.21.15.43:0/979818062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44108ef0 0x7fdd4419b2e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:46.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.452+0000 7fdd4ba00700 1 -- 172.21.15.43:0/979818062 >> 172.21.15.43:0/979818062 conn(0x7fdd440ffe70 msgr2=0x7fdd44102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:46.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.453+0000 7fdd4ba00700 1 -- 172.21.15.43:0/979818062 shutdown_connections 2024-05-27T07:39:46.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:46.453+0000 7fdd4ba00700 1 -- 172.21.15.43:0/979818062 wait complete. 2024-05-27T07:39:46.683 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:46 smithi043 ceph-mon[31531]: pgmap v866: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 134 op/s 2024-05-27T07:39:46.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:46 smithi043 ceph-mon[31531]: scrub queued for path: / 2024-05-27T07:39:46.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:46 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:46.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:46 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:39:46.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:46 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:39:46.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:46 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:39:46.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:46 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:46.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 77074 5.83 MB/sec execute 587 sec latency 123.675 ms 2024-05-27T07:39:46.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:46 smithi179 ceph-mon[35888]: pgmap v866: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 134 op/s 2024-05-27T07:39:46.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:46 smithi179 ceph-mon[35888]: scrub queued for path: / 2024-05-27T07:39:46.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:46 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:46.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:46 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:39:46.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:46 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:39:46.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:46 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:39:46.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:46 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:46 smithi132 ceph-mon[35283]: pgmap v866: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 134 op/s 2024-05-27T07:39:47.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:46 smithi132 ceph-mon[35283]: scrub queued for path: / 2024-05-27T07:39:47.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:46 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:46 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:39:47.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:46 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:39:47.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:46 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:39:47.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:46 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 77158 5.82 MB/sec execute 588 sec latency 395.890 ms 2024-05-27T07:39:47.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:39:47.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:39:47.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:39:47.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:39:47.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:39:47.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: active paths [/] 2024-05-27T07:39:47.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:47 smithi043 ceph-mon[31531]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:39:47.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:39:47.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:39:47.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:39:47.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:39:47.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: active paths [/] 2024-05-27T07:39:47.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:47.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:47 smithi179 ceph-mon[35888]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:48.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: active paths [/] 2024-05-27T07:39:48.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:48.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:48.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:47 smithi132 ceph-mon[35283]: scrub summary: idle+waiting paths [/] 2024-05-27T07:39:48.698 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:48 smithi043 ceph-mon[31531]: pgmap v867: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 207 op/s 2024-05-27T07:39:48.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 77257 5.82 MB/sec execute 589 sec latency 116.837 ms 2024-05-27T07:39:48.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:39:48.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:48 smithi179 ceph-mon[35888]: pgmap v867: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 207 op/s 2024-05-27T07:39:49.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:48 smithi132 ceph-mon[35283]: pgmap v867: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 207 op/s 2024-05-27T07:39:49.589 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:39:49.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 77364 5.82 MB/sec execute 590 sec latency 137.772 ms 2024-05-27T07:39:49.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:49 smithi179 ceph-mon[35888]: pgmap v868: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.4 MiB/s wr, 138 op/s 2024-05-27T07:39:50.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:49 smithi132 ceph-mon[35283]: pgmap v868: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.4 MiB/s wr, 138 op/s 2024-05-27T07:39:50.193 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:49 smithi043 ceph-mon[31531]: pgmap v868: 129 pgs: 129 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.4 MiB/s wr, 138 op/s 2024-05-27T07:39:50.841 INFO:tasks.workunit.client.0.smithi043.stdout: 10 77487 5.83 MB/sec execute 591 sec latency 136.090 ms 2024-05-27T07:39:51.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 77588 5.82 MB/sec execute 592 sec latency 124.709 ms 2024-05-27T07:39:52.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 77711 5.82 MB/sec execute 593 sec latency 105.311 ms 2024-05-27T07:39:52.904 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:52 smithi043 ceph-mon[31531]: pgmap v869: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.4 MiB/s wr, 214 op/s 2024-05-27T07:39:52.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:52 smithi179 ceph-mon[35888]: pgmap v869: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.4 MiB/s wr, 214 op/s 2024-05-27T07:39:53.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:52 smithi132 ceph-mon[35283]: pgmap v869: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.4 MiB/s wr, 214 op/s 2024-05-27T07:39:53.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 77824 5.82 MB/sec execute 594 sec latency 85.060 ms 2024-05-27T07:39:54.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 77918 5.81 MB/sec execute 595 sec latency 93.325 ms 2024-05-27T07:39:54.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:54 smithi043 ceph-mon[31531]: pgmap v870: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 MiB/s wr, 148 op/s 2024-05-27T07:39:54.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:54 smithi179 ceph-mon[35888]: pgmap v870: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 MiB/s wr, 148 op/s 2024-05-27T07:39:55.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.021+0000 7fb8a8a00700 1 -- 172.21.15.43:0/2541313722 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4012650 con 0x7fb8ac109860 2024-05-27T07:39:55.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.022+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2541313722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8ac109860 msgr2=0x7fb8ac1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.022+0000 7fb8b0e00700 1 --2- 172.21.15.43:0/2541313722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8ac109860 0x7fb8ac1111c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400b0d0 tx=0x7fb8a400b3e0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.022+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2541313722 shutdown_connections 2024-05-27T07:39:55.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.022+0000 7fb8b0e00700 1 --2- 172.21.15.43:0/2541313722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8ac109860 0x7fb8ac1111c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.022+0000 7fb8b0e00700 1 --2- 172.21.15.43:0/2541313722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8ac1045e0 0x7fb8ac1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.022+0000 7fb8b0e00700 1 --2- 172.21.15.43:0/2541313722 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8ac108f10 0x7fb8ac109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.023+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2541313722 >> 172.21.15.43:0/2541313722 conn(0x7fb8ac100030 msgr2=0x7fb8ac102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.023+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2541313722 shutdown_connections 2024-05-27T07:39:55.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.023+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2541313722 wait complete. 2024-05-27T07:39:55.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.025+0000 7fb8b0e00700 1 Processor -- start 2024-05-27T07:39:55.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.025+0000 7fb8b0e00700 1 -- start start 2024-05-27T07:39:55.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.026+0000 7fb8b0e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8ac1045e0 0x7fb8ac10e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.026+0000 7fb8b0e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8ac108f10 0x7fb8ac10e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.026+0000 7fb8b0e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8ac109860 0x7fb8ac10eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.026+0000 7fb8b0e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac0772f0 con 0x7fb8ac109860 2024-05-27T07:39:55.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.027+0000 7fb8b0e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac077170 con 0x7fb8ac108f10 2024-05-27T07:39:55.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.027+0000 7fb8b0e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac077470 con 0x7fb8ac1045e0 2024-05-27T07:39:55.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.027+0000 7fb8a9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8ac1045e0 0x7fb8ac10e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.027+0000 7fb8a9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8ac1045e0 0x7fb8ac10e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:36362/0 (socket says 172.21.15.43:36362) 2024-05-27T07:39:55.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.027+0000 7fb8a9e00700 1 -- 172.21.15.43:0/2873777122 learned_addr learned my addr 172.21.15.43:0/2873777122 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:55.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.027+0000 7fb8a9400700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8ac108f10 0x7fb8ac10e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.028+0000 7fb8a9e00700 1 -- 172.21.15.43:0/2873777122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8ac108f10 msgr2=0x7fb8ac10e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.028+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8ac108f10 0x7fb8ac10e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.028+0000 7fb8a9e00700 1 -- 172.21.15.43:0/2873777122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8ac109860 msgr2=0x7fb8ac10eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.028+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8ac109860 0x7fb8ac10eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.028+0000 7fb8a9e00700 1 -- 172.21.15.43:0/2873777122 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ac1ad3c0 con 0x7fb8ac1045e0 2024-05-27T07:39:55.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.029+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8ac1045e0 0x7fb8ac10e450 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00d950 tx=0x7fb89c00dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.029+0000 7fb8a2a00700 1 -- 172.21.15.43:0/2873777122 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c016070 con 0x7fb8ac1045e0 2024-05-27T07:39:55.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.030+0000 7fb8a2a00700 1 -- 172.21.15.43:0/2873777122 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb89c010460 con 0x7fb8ac1045e0 2024-05-27T07:39:55.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.030+0000 7fb8a2a00700 1 -- 172.21.15.43:0/2873777122 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c00f620 con 0x7fb8ac1045e0 2024-05-27T07:39:55.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.030+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2873777122 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ac1ad6b0 con 0x7fb8ac1045e0 2024-05-27T07:39:55.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.030+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2873777122 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ac1adba0 con 0x7fb8ac1045e0 2024-05-27T07:39:55.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.039+0000 7fb8a2a00700 1 -- 172.21.15.43:0/2873777122 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb89c019070 con 0x7fb8ac1045e0 2024-05-27T07:39:55.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.042+0000 7fb8a2a00700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8900b2bc0 0x7fb8900b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.042+0000 7fb8a2a00700 1 -- 172.21.15.43:0/2873777122 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fb89c0e2570 con 0x7fb8ac1045e0 2024-05-27T07:39:55.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.043+0000 7fb8a9400700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8900b2bc0 0x7fb8900b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.044+0000 7fb8a9400700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8900b2bc0 0x7fb8900b50d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb894005950 tx=0x7fb8940058e0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.063+0000 7fb8bf9beb80 1 Processor -- start 2024-05-27T07:39:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.064+0000 7fb8bf9beb80 1 -- start start 2024-05-27T07:39:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.064+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.064+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182b5d30 0x5654182af870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.064+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 0x5654182b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.064+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5654181dde40 con 0x565418130e40 2024-05-27T07:39:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.064+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5654181cd9e0 con 0x5654182afdb0 2024-05-27T07:39:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.064+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5654181d1a20 con 0x5654182b5d30 2024-05-27T07:39:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.065+0000 7fb8a9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.065+0000 7fb8a9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57374/0 (socket says 172.21.15.43:57374) 2024-05-27T07:39:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.065+0000 7fb8a9e00700 1 -- 172.21.15.43:0/3306948902 learned_addr learned my addr 172.21.15.43:0/3306948902 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.065+0000 7fb8aa800700 1 --2- 172.21.15.43:0/3306948902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 0x5654182b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.065+0000 7fb8a9400700 1 --2- 172.21.15.43:0/3306948902 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182b5d30 0x5654182af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.065+0000 7fb8a9e00700 1 -- 172.21.15.43:0/3306948902 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182b5d30 msgr2=0x5654182af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.065+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/3306948902 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182b5d30 0x5654182af870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.065+0000 7fb8a9e00700 1 -- 172.21.15.43:0/3306948902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 msgr2=0x5654182b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.065+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/3306948902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 0x5654182b01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.066+0000 7fb8a9e00700 1 -- 172.21.15.43:0/3306948902 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565418196420 con 0x565418130e40 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.066+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/3306948902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b57f0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0a9120 tx=0x7fb89c0a8fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f9b92b31f3ee46 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.066+0000 7fb89b400700 1 -- 172.21.15.43:0/3306948902 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c020040 con 0x565418130e40 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.066+0000 7fb89b400700 1 -- 172.21.15.43:0/3306948902 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb89c0ad7d0 con 0x565418130e40 2024-05-27T07:39:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.066+0000 7fb89b400700 1 -- 172.21.15.43:0/3306948902 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c0a9680 con 0x565418130e40 2024-05-27T07:39:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.067+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/3306948902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 msgr2=0x5654182b57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.067+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/3306948902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b57f0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0a9120 tx=0x7fb89c0a8fb0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.069+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/3306948902 shutdown_connections 2024-05-27T07:39:55.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.069+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/3306948902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 0x5654182b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.069+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/3306948902 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b57f0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.069+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/3306948902 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182b5d30 0x5654182af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.069+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/3306948902 >> 172.21.15.43:0/3306948902 conn(0x565418190fc0 msgr2=0x5654182aa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.069+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/3306948902 shutdown_connections 2024-05-27T07:39:55.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.069+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/3306948902 wait complete. 2024-05-27T07:39:55.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.070+0000 7fb8bf9beb80 1 -- start start 2024-05-27T07:39:55.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.070+0000 7fb8bf9beb80 1 Processor -- start 2024-05-27T07:39:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.071+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.071+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182afdb0 0x5654182b9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.071+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 0x5654182ba040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.071+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5654180d3fa0 con 0x565418130e40 2024-05-27T07:39:55.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.071+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56541816e760 con 0x5654182b5d30 2024-05-27T07:39:55.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.071+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5654182cd7b0 con 0x5654182afdb0 2024-05-27T07:39:55.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.072+0000 7fb8bf9beb80 10 client.0 _ll_register_callbacks cb 0x565418177c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:55.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.072+0000 7fb8a9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.072+0000 7fb8a9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57378/0 (socket says 172.21.15.43:57378) 2024-05-27T07:39:55.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.072+0000 7fb8a9e00700 1 -- 172.21.15.43:0/2671773595 learned_addr learned my addr 172.21.15.43:0/2671773595 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:55.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.072+0000 7fb8aa800700 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 0x5654182ba040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.072+0000 7fb8a9400700 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182afdb0 0x5654182b9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.073+0000 7fb8a9e00700 1 -- 172.21.15.43:0/2671773595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182afdb0 msgr2=0x5654182b9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.073+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182afdb0 0x5654182b9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.073+0000 7fb8a9e00700 1 -- 172.21.15.43:0/2671773595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 msgr2=0x5654182ba040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.073+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 0x5654182ba040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.073+0000 7fb8a9e00700 1 -- 172.21.15.43:0/2671773595 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56541811d8d0 con 0x565418130e40 2024-05-27T07:39:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.073+0000 7fb8aa800700 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 0x5654182ba040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:39:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.074+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b95c0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0a92f0 tx=0x7fb89c0e2430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b516251d2d6799f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.074+0000 7fb88fe00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:39:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.074+0000 7fb88fe00700 1 -- 172.21.15.43:0/2671773595 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c0c8060 con 0x565418130e40 2024-05-27T07:39:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.074+0000 7fb88fe00700 1 -- 172.21.15.43:0/2671773595 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fb89c0a5a10 con 0x565418130e40 2024-05-27T07:39:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.074+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/2671773595 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb89c0ad7d0 con 0x565418130e40 2024-05-27T07:39:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb88fe00700 1 -- 172.21.15.43:0/2671773595 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fb89c0c2030 con 0x565418130e40 2024-05-27T07:39:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb8bf9beb80 10 client.18609 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/2671773595 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x56541819e060 con 0x565418130e40 2024-05-27T07:39:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb88fe00700 1 -- 172.21.15.43:0/2671773595 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb89c0ad7d0 con 0x565418130e40 2024-05-27T07:39:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb88fe00700 1 -- 172.21.15.43:0/2671773595 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fb89c0b1df0 con 0x565418130e40 2024-05-27T07:39:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb8bf9beb80 10 client.18609 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb8bf9beb80 10 client.18609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb8bf9beb80 20 client.18609 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5654182da770 0x5654182dcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb8bf9beb80 4 client.18609 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:39:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.075+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/2671773595 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654181dfd90 con 0x5654182da770 2024-05-27T07:39:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.076+0000 7fb8a9400700 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5654182da770 0x5654182dcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.077+0000 7fb88fe00700 10 client.18609 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.077+0000 7fb8a9400700 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5654182da770 0x5654182dcbc0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35a73f8917dc32ef server_cookie=210fd81caa2bb2f3 in_seq=0 out_seq=0 2024-05-27T07:39:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.079+0000 7fb88fe00700 1 -- 172.21.15.43:0/2671773595 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5654181dfd90 con 0x5654182da770 2024-05-27T07:39:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.079+0000 7fb88fe00700 10 client.18609 handle_command_reply: tid=0 2024-05-27T07:39:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.079+0000 7fb8bf9beb80 1 client.18609 shutdown 2024-05-27T07:39:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.079+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/2671773595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 msgr2=0x5654182b95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.079+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b95c0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0a92f0 tx=0x7fb89c0e2430 comp rx=0 tx=0).stop 2024-05-27T07:39:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.079+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/2671773595 shutdown_connections 2024-05-27T07:39:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.079+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 0x5654182ba040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.080+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x5654182da770 0x5654182dcbc0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.080+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182b95c0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.080+0000 7fb88fe00700 0 client.18609 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.080+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/2671773595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182afdb0 0x5654182b9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.080+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/2671773595 >> 172.21.15.43:0/2671773595 conn(0x565418190fc0 msgr2=0x5654182aa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.080+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/2671773595 shutdown_connections 2024-05-27T07:39:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.080+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/2671773595 wait complete. 2024-05-27T07:39:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.080+0000 7fb8bf9beb80 20 client.18609 trim_cache size 0 max 16384 2024-05-27T07:39:55.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:54 smithi132 ceph-mon[35283]: pgmap v870: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 MiB/s wr, 148 op/s 2024-05-27T07:39:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.122+0000 7fb8bf9beb80 1 Processor -- start 2024-05-27T07:39:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.123+0000 7fb8bf9beb80 1 -- start start 2024-05-27T07:39:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.123+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565418130e40 0x5654182af870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.123+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5654182afdb0 0x5654182b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.123+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 0x5654182d5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.123+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac1ae8e0 con 0x5654182afdb0 2024-05-27T07:39:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.123+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac0748c0 con 0x5654182b5d30 2024-05-27T07:39:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.123+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac1ae210 con 0x565418130e40 2024-05-27T07:39:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.123+0000 7fb8a9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5654182afdb0 0x5654182b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.124+0000 7fb8a9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565418130e40 0x5654182af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.124+0000 7fb8a9e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565418130e40 0x5654182af870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.43:36374/0 (socket says 172.21.15.43:36374) 2024-05-27T07:39:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.124+0000 7fb8a9e00700 1 -- 172.21.15.43:0/3277283383 learned_addr learned my addr 172.21.15.43:0/3277283383 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:55.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.124+0000 7fb8aa800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 0x5654182d5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.124+0000 7fb8aa800700 1 -- 172.21.15.43:0/3277283383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565418130e40 msgr2=0x5654182af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.124+0000 7fb8aa800700 1 --2- 172.21.15.43:0/3277283383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565418130e40 0x5654182af870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.125+0000 7fb8aa800700 1 -- 172.21.15.43:0/3277283383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5654182afdb0 msgr2=0x5654182b4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.125+0000 7fb8aa800700 1 --2- 172.21.15.43:0/3277283383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5654182afdb0 0x5654182b4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.125+0000 7fb8aa800700 1 -- 172.21.15.43:0/3277283383 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654182ba3b0 con 0x5654182b5d30 2024-05-27T07:39:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.125+0000 7fb8aa800700 1 --2- 172.21.15.43:0/3277283383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 0x5654182d5800 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400b0d0 tx=0x7fb8a4009d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d735cf073e821be9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.126+0000 7fb89aa00700 1 -- 172.21.15.43:0/3277283383 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4020070 con 0x5654182b5d30 2024-05-27T07:39:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.126+0000 7fb89aa00700 1 -- 172.21.15.43:0/3277283383 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8a4009780 con 0x5654182b5d30 2024-05-27T07:39:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.126+0000 7fb89aa00700 1 -- 172.21.15.43:0/3277283383 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4012650 con 0x5654182b5d30 2024-05-27T07:39:55.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.127+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/3277283383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 msgr2=0x5654182d5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.127+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/3277283383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 0x5654182d5800 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400b0d0 tx=0x7fb8a4009d10 comp rx=0 tx=0).stop 2024-05-27T07:39:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.127+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/3277283383 shutdown_connections 2024-05-27T07:39:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.127+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/3277283383 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182b5d30 0x5654182d5800 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.127+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/3277283383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x565418130e40 0x5654182af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.127+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/3277283383 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5654182afdb0 0x5654182b4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.127+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/3277283383 >> 172.21.15.43:0/3277283383 conn(0x565418190fc0 msgr2=0x5654182b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.127+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/3277283383 shutdown_connections 2024-05-27T07:39:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.128+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/3277283383 wait complete. 2024-05-27T07:39:55.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.128+0000 7fb8bf9beb80 1 -- start start 2024-05-27T07:39:55.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.128+0000 7fb8bf9beb80 1 Processor -- start 2024-05-27T07:39:55.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.129+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182aea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.129+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 0x5654182aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.129+0000 7fb8bf9beb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182b5d30 0x5654182af500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.129+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac075660 con 0x565418130e40 2024-05-27T07:39:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.129+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac074f90 con 0x5654182afdb0 2024-05-27T07:39:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.129+0000 7fb8bf9beb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5654182dd1a0 con 0x5654182b5d30 2024-05-27T07:39:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.129+0000 7fb8bf9beb80 10 client.0 _ll_register_callbacks cb 0x565418177c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:39:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.130+0000 7fb8a9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 0x5654182aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.130+0000 7fb8a9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 0x5654182aefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.43:45252/0 (socket says 172.21.15.43:45252) 2024-05-27T07:39:55.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.130+0000 7fb8a9400700 1 -- 172.21.15.43:0/482077743 learned_addr learned my addr 172.21.15.43:0/482077743 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:39:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.130+0000 7fb8aa800700 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182b5d30 0x5654182af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.130+0000 7fb8a9400700 1 -- 172.21.15.43:0/482077743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182b5d30 msgr2=0x5654182af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.130+0000 7fb8a9400700 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182b5d30 0x5654182af500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.130+0000 7fb8a9400700 1 -- 172.21.15.43:0/482077743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 msgr2=0x5654182aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:39:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.130+0000 7fb8a9400700 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.130+0000 7fb8a9400700 1 -- 172.21.15.43:0/482077743 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565418344290 con 0x5654182afdb0 2024-05-27T07:39:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.131+0000 7fb8a9400700 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 0x5654182aefc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac103ab0 tx=0x7fb8940047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d7728f57e68ddef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:39:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.131+0000 7fb88fe00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-05-27T07:39:55.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.132+0000 7fb88fe00700 1 -- 172.21.15.43:0/482077743 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894027c20 con 0x5654182afdb0 2024-05-27T07:39:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.132+0000 7fb88fe00700 1 -- 172.21.15.43:0/482077743 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(161..161 src has 1..161) ==== 9496+0+0 (secure 0 0 0) 0x7fb894011440 con 0x5654182afdb0 2024-05-27T07:39:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.132+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/482077743 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8a4009780 con 0x5654182afdb0 2024-05-27T07:39:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.133+0000 7fb88fe00700 1 -- 172.21.15.43:0/482077743 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7fb894011640 con 0x5654182afdb0 2024-05-27T07:39:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.133+0000 7fb8bf9beb80 10 client.38003 fetch_fsmap learned FSMap version 147 2024-05-27T07:39:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.133+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/482077743 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=4+}) -- 0x5654183453e0 con 0x5654182afdb0 2024-05-27T07:39:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.134+0000 7fb88fe00700 1 -- 172.21.15.43:0/482077743 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8a4009780 con 0x5654182afdb0 2024-05-27T07:39:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.134+0000 7fb88fe00700 1 -- 172.21.15.43:0/482077743 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 147) ==== 4187+0+0 (secure 0 0 0) 0x7fb89402e070 con 0x5654182afdb0 2024-05-27T07:39:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.134+0000 7fb8bf9beb80 10 client.38003 fetch_fsmap finished waiting for FSMap version 147 2024-05-27T07:39:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.134+0000 7fb8bf9beb80 10 client.38003 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:39:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.134+0000 7fb8bf9beb80 20 client.38003 populate_metadata read hostname 'smithi043' 2024-05-27T07:39:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.134+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x565418345b50 0x565418347f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:39:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.134+0000 7fb8bf9beb80 4 client.38003 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:39:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.134+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/482077743 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5654181d9570 con 0x565418345b50 2024-05-27T07:39:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.135+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x565418345b50 0x565418347f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:39:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.136+0000 7fb88fe00700 10 client.38003 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.136+0000 7fb8a9e00700 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x565418345b50 0x565418347f70 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a62d6d6cc822de82 server_cookie=9a60638472cbfbb2 in_seq=0 out_seq=0 2024-05-27T07:39:55.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.137+0000 7fb88fe00700 1 -- 172.21.15.43:0/482077743 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5654181d9570 con 0x565418345b50 2024-05-27T07:39:55.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.137+0000 7fb88fe00700 10 client.38003 handle_command_reply: tid=0 2024-05-27T07:39:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.137+0000 7fb8bf9beb80 1 client.38003 shutdown 2024-05-27T07:39:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.138+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/482077743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 msgr2=0x5654182aefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.138+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 0x5654182aefc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac103ab0 tx=0x7fb8940047c0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.138+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/482077743 shutdown_connections 2024-05-27T07:39:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.138+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654182b5d30 0x5654182af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.138+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x565418345b50 0x565418347f70 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.138+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565418130e40 0x5654182aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.138+0000 7fb8bf9beb80 1 --2- 172.21.15.43:0/482077743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654182afdb0 0x5654182aefc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.138+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/482077743 >> 172.21.15.43:0/482077743 conn(0x565418190fc0 msgr2=0x5654182b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.138+0000 7fb88fe00700 0 client.38003 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:39:55.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.139+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/482077743 shutdown_connections 2024-05-27T07:39:55.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.139+0000 7fb8bf9beb80 1 -- 172.21.15.43:0/482077743 wait complete. 2024-05-27T07:39:55.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.139+0000 7fb8bf9beb80 20 client.38003 trim_cache size 0 max 16384 2024-05-27T07:39:55.137 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:39:55.137 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-27T07:39:55.137 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-05-27T07:39:55.137 INFO:teuthology.orchestra.run.smithi043.stdout: "564173bd-1d1d-4299-9f55-3fc5f6126006": { 2024-05-27T07:39:55.137 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-05-27T07:39:55.137 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "564173bd-1d1d-4299-9f55-3fc5f6126006", 2024-05-27T07:39:55.137 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-05-27T07:39:55.137 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:39:55.137 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:39:55.138 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:39:55.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.140+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2873777122 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8900b2bc0 msgr2=0x7fb8900b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.140+0000 7fb8b0e00700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8900b2bc0 0x7fb8900b50d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb894005950 tx=0x7fb8940058e0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.140+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2873777122 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8ac1045e0 msgr2=0x7fb8ac10e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:39:55.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.140+0000 7fb8b0e00700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8ac1045e0 0x7fb8ac10e450 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00d950 tx=0x7fb89c00dd10 comp rx=0 tx=0).stop 2024-05-27T07:39:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.140+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2873777122 shutdown_connections 2024-05-27T07:39:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.140+0000 7fb8b0e00700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb8900b2bc0 0x7fb8900b50d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.140+0000 7fb8b0e00700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb8ac109860 0x7fb8ac10eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.140+0000 7fb8b0e00700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8ac1045e0 0x7fb8ac10e450 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.140+0000 7fb8b0e00700 1 --2- 172.21.15.43:0/2873777122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8ac108f10 0x7fb8ac10e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:39:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.140+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2873777122 >> 172.21.15.43:0/2873777122 conn(0x7fb8ac100030 msgr2=0x7fb8ac108590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:39:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.141+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2873777122 shutdown_connections 2024-05-27T07:39:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:39:55.141+0000 7fb8b0e00700 1 -- 172.21.15.43:0/2873777122 wait complete. 2024-05-27T07:39:55.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 78007 5.81 MB/sec execute 596 sec latency 108.812 ms 2024-05-27T07:39:56.213 INFO:tasks.cephfs.filesystem:scrub status for tag:564173bd-1d1d-4299-9f55-3fc5f6126006 - {'path': '/', 'tag': '564173bd-1d1d-4299-9f55-3fc5f6126006', 'options': 'recursive,force'} 2024-05-27T07:39:56.663 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:56 smithi043 ceph-mon[31531]: pgmap v871: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 MiB/s wr, 148 op/s 2024-05-27T07:39:56.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 78146 5.81 MB/sec execute 597 sec latency 103.822 ms 2024-05-27T07:39:56.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:56 smithi179 ceph-mon[35888]: pgmap v871: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 MiB/s wr, 148 op/s 2024-05-27T07:39:57.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:56 smithi132 ceph-mon[35283]: pgmap v871: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 MiB/s wr, 148 op/s 2024-05-27T07:39:57.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 78257 5.81 MB/sec execute 598 sec latency 101.129 ms 2024-05-27T07:39:57.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:57 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:39:57.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:57 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:39:57.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:57 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:39:57.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:57 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:39:57.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:57 smithi043 ceph-mon[31531]: scrub summary: idle 2024-05-27T07:39:57.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:57 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:39:57.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:57 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:39:57.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:57 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:39:57.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:57 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:39:57.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:57 smithi179 ceph-mon[35888]: scrub summary: idle 2024-05-27T07:39:58.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:57 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:39:58.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:57 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:39:58.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:57 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:39:58.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:57 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:39:58.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:57 smithi132 ceph-mon[35283]: scrub summary: idle 2024-05-27T07:39:58.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 78387 5.82 MB/sec execute 599 sec latency 132.562 ms 2024-05-27T07:39:58.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:39:58 smithi043 ceph-mon[31531]: pgmap v872: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.8 MiB/s wr, 232 op/s 2024-05-27T07:39:58.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:39:58 smithi179 ceph-mon[35888]: pgmap v872: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.8 MiB/s wr, 232 op/s 2024-05-27T07:39:59.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:39:58 smithi132 ceph-mon[35283]: pgmap v872: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.8 MiB/s wr, 232 op/s 2024-05-27T07:39:59.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 cleanup 600 sec 2024-05-27T07:40:00.842 INFO:tasks.workunit.client.0.smithi043.stdout: 10 cleanup 601 sec 2024-05-27T07:40:00.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:00 smithi043 ceph-mon[31531]: pgmap v873: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.6 MiB/s wr, 159 op/s 2024-05-27T07:40:00.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:00 smithi043 ceph-mon[31531]: Health detail: HEALTH_WARN 7 failed cephadm daemon(s) 2024-05-27T07:40:00.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:00 smithi043 ceph-mon[31531]: [WRN] CEPHADM_FAILED_DAEMON: 7 failed cephadm daemon(s) 2024-05-27T07:40:00.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:00 smithi043 ceph-mon[31531]: daemon mds.j on smithi043 is in error state 2024-05-27T07:40:00.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:00 smithi043 ceph-mon[31531]: daemon mds.d on smithi043 is in error state 2024-05-27T07:40:00.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:00 smithi043 ceph-mon[31531]: daemon mds.g on smithi043 is in error state 2024-05-27T07:40:00.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:00 smithi043 ceph-mon[31531]: daemon mds.a on smithi043 is in error state 2024-05-27T07:40:00.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:00 smithi043 ceph-mon[31531]: daemon mds.l on smithi179 is in error state 2024-05-27T07:40:00.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:00 smithi043 ceph-mon[31531]: daemon mds.f on smithi179 is in error state 2024-05-27T07:40:00.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:00 smithi043 ceph-mon[31531]: daemon mds.c on smithi179 is in error state 2024-05-27T07:40:00.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:00 smithi179 ceph-mon[35888]: pgmap v873: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.6 MiB/s wr, 159 op/s 2024-05-27T07:40:00.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:00 smithi179 ceph-mon[35888]: Health detail: HEALTH_WARN 7 failed cephadm daemon(s) 2024-05-27T07:40:00.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:00 smithi179 ceph-mon[35888]: [WRN] CEPHADM_FAILED_DAEMON: 7 failed cephadm daemon(s) 2024-05-27T07:40:00.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:00 smithi179 ceph-mon[35888]: daemon mds.j on smithi043 is in error state 2024-05-27T07:40:00.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:00 smithi179 ceph-mon[35888]: daemon mds.d on smithi043 is in error state 2024-05-27T07:40:00.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:00 smithi179 ceph-mon[35888]: daemon mds.g on smithi043 is in error state 2024-05-27T07:40:00.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:00 smithi179 ceph-mon[35888]: daemon mds.a on smithi043 is in error state 2024-05-27T07:40:00.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:00 smithi179 ceph-mon[35888]: daemon mds.l on smithi179 is in error state 2024-05-27T07:40:00.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:00 smithi179 ceph-mon[35888]: daemon mds.f on smithi179 is in error state 2024-05-27T07:40:00.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:00 smithi179 ceph-mon[35888]: daemon mds.c on smithi179 is in error state 2024-05-27T07:40:01.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:00 smithi132 ceph-mon[35283]: pgmap v873: 129 pgs: 129 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 3.6 MiB/s wr, 159 op/s 2024-05-27T07:40:01.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:00 smithi132 ceph-mon[35283]: Health detail: HEALTH_WARN 7 failed cephadm daemon(s) 2024-05-27T07:40:01.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:00 smithi132 ceph-mon[35283]: [WRN] CEPHADM_FAILED_DAEMON: 7 failed cephadm daemon(s) 2024-05-27T07:40:01.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:00 smithi132 ceph-mon[35283]: daemon mds.j on smithi043 is in error state 2024-05-27T07:40:01.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:00 smithi132 ceph-mon[35283]: daemon mds.d on smithi043 is in error state 2024-05-27T07:40:01.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:00 smithi132 ceph-mon[35283]: daemon mds.g on smithi043 is in error state 2024-05-27T07:40:01.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:00 smithi132 ceph-mon[35283]: daemon mds.a on smithi043 is in error state 2024-05-27T07:40:01.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:00 smithi132 ceph-mon[35283]: daemon mds.l on smithi179 is in error state 2024-05-27T07:40:01.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:00 smithi132 ceph-mon[35283]: daemon mds.f on smithi179 is in error state 2024-05-27T07:40:01.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:00 smithi132 ceph-mon[35283]: daemon mds.c on smithi179 is in error state 2024-05-27T07:40:01.842 INFO:tasks.workunit.client.0.smithi043.stdout: 5 cleanup 602 sec 2024-05-27T07:40:02.843 INFO:tasks.workunit.client.0.smithi043.stdout: 2 cleanup 603 sec 2024-05-27T07:40:02.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:02 smithi043 ceph-mon[31531]: pgmap v874: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.0 MiB/s wr, 260 op/s 2024-05-27T07:40:02.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:02 smithi043 ceph-mon[31531]: osdmap e162: 12 total, 12 up, 12 in 2024-05-27T07:40:02.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:02 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:40:02.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:02 smithi179 ceph-mon[35888]: pgmap v874: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.0 MiB/s wr, 260 op/s 2024-05-27T07:40:02.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:02 smithi179 ceph-mon[35888]: osdmap e162: 12 total, 12 up, 12 in 2024-05-27T07:40:02.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:02 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:40:03.070 INFO:tasks.workunit.client.0.smithi043.stdout: 0 cleanup 603 sec 2024-05-27T07:40:03.070 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-05-27T07:40:03.070 INFO:tasks.workunit.client.0.smithi043.stdout: Operation Count AvgLat MaxLat 2024-05-27T07:40:03.070 INFO:tasks.workunit.client.0.smithi043.stdout: ---------------------------------------- 2024-05-27T07:40:03.071 INFO:tasks.workunit.client.0.smithi043.stdout: NTCreateX 112368 16.044 5222.358 2024-05-27T07:40:03.071 INFO:tasks.workunit.client.0.smithi043.stdout: Close 82497 0.303 247.177 2024-05-27T07:40:03.071 INFO:tasks.workunit.client.0.smithi043.stdout: Rename 4755 37.498 550.880 2024-05-27T07:40:03.071 INFO:tasks.workunit.client.0.smithi043.stdout: Unlink 22766 14.370 2483.619 2024-05-27T07:40:03.071 INFO:tasks.workunit.client.0.smithi043.stdout: Qpathinfo 101752 10.334 4806.601 2024-05-27T07:40:03.071 INFO:tasks.workunit.client.0.smithi043.stdout: Qfileinfo 17689 0.609 38.821 2024-05-27T07:40:03.071 INFO:tasks.workunit.client.0.smithi043.stdout: Qfsinfo 18661 2.129 356.659 2024-05-27T07:40:03.071 INFO:tasks.workunit.client.0.smithi043.stdout: Sfileinfo 9190 5.069 278.221 2024-05-27T07:40:03.071 INFO:tasks.workunit.client.0.smithi043.stdout: Find 39315 59.459 4864.919 2024-05-27T07:40:03.072 INFO:tasks.workunit.client.0.smithi043.stdout: WriteX 55397 1.146 310.222 2024-05-27T07:40:03.072 INFO:tasks.workunit.client.0.smithi043.stdout: ReadX 175553 0.019 15.470 2024-05-27T07:40:03.072 INFO:tasks.workunit.client.0.smithi043.stdout: LockX 362 2.949 42.891 2024-05-27T07:40:03.072 INFO:tasks.workunit.client.0.smithi043.stdout: UnlockX 362 2.632 9.032 2024-05-27T07:40:03.072 INFO:tasks.workunit.client.0.smithi043.stdout: Flush 7897 13.694 672.170 2024-05-27T07:40:03.072 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-05-27T07:40:03.072 INFO:tasks.workunit.client.0.smithi043.stdout:Throughput 5.82009 MB/sec 10 clients 10 procs max_latency=5222.368 ms 2024-05-27T07:40:03.083 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T07:40:03.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T07:40:03.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:02 smithi132 ceph-mon[35283]: pgmap v874: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.0 MiB/s wr, 260 op/s 2024-05-27T07:40:03.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:02 smithi132 ceph-mon[35283]: osdmap e162: 12 total, 12 up, 12 in 2024-05-27T07:40:03.097 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:02 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:40:04.348 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:04 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.348 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:04 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.348 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:04 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.349 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:04 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.349 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:04 smithi132 ceph-mon[35283]: pgmap v876: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 4.9 MiB/s wr, 221 op/s 2024-05-27T07:40:04.349 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:04 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:04.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:04 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:04 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:04 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:04 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:04 smithi043 ceph-mon[31531]: pgmap v876: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 4.9 MiB/s wr, 221 op/s 2024-05-27T07:40:04.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:04 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:04.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:04 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:04 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:04 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:04 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:04.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:04 smithi179 ceph-mon[35888]: pgmap v876: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 4.9 MiB/s wr, 221 op/s 2024-05-27T07:40:04.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:04 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:05.348 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:40:05.348 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:40:05.348 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:05 smithi132 ceph-mon[35283]: pgmap v877: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 5.5 MiB/s wr, 253 op/s 2024-05-27T07:40:05.348 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:05 smithi132 ceph-mon[35283]: pgmap v878: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 4.4 MiB/s wr, 178 op/s 2024-05-27T07:40:05.348 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:05.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:40:05.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:40:05.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:05 smithi043 ceph-mon[31531]: pgmap v877: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 5.5 MiB/s wr, 253 op/s 2024-05-27T07:40:05.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:05 smithi043 ceph-mon[31531]: pgmap v878: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 4.4 MiB/s wr, 178 op/s 2024-05-27T07:40:05.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:05.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:40:05.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:40:05.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:05 smithi179 ceph-mon[35888]: pgmap v877: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 5.5 MiB/s wr, 253 op/s 2024-05-27T07:40:05.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:05 smithi179 ceph-mon[35888]: pgmap v878: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 4.4 MiB/s wr, 178 op/s 2024-05-27T07:40:05.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:40:06.348 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:06 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:06.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:06 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:06.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:06 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:07.199 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-05-27T07:40:07.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-27T07:40:07.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:07 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:07.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:07 smithi043 ceph-mon[31531]: pgmap v879: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 6.1 MiB/s wr, 353 op/s 2024-05-27T07:40:07.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:07 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:07.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:07 smithi179 ceph-mon[35888]: pgmap v879: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 6.1 MiB/s wr, 353 op/s 2024-05-27T07:40:07.598 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:07 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:07.598 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:07 smithi132 ceph-mon[35283]: pgmap v879: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 6.1 MiB/s wr, 353 op/s 2024-05-27T07:40:08.353 DEBUG:teuthology.parallel:result is None 2024-05-27T07:40:08.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-27T07:40:08.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:08 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:08.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:08 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:08.477 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-27T07:40:08.477 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-27T07:40:08.488 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-27T07:40:08.488 INFO:teuthology.task.exec:Executing custom commands... 2024-05-27T07:40:08.489 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-05-27T07:40:08.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-27T07:40:08.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-27T07:40:08.597 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:08 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:08.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-27T07:40:09.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.169+0000 7fb79fe00640 1 -- 172.21.15.43:0/2621019839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7a010e010 msgr2=0x7fb7a0110500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.169+0000 7fb79fe00640 1 --2- 172.21.15.43:0/2621019839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7a010e010 0x7fb7a0110500 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fb798009900 tx=0x7fb79802a480 comp rx=0 tx=0).stop 2024-05-27T07:40:09.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.170+0000 7fb79fe00640 1 -- 172.21.15.43:0/2621019839 shutdown_connections 2024-05-27T07:40:09.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.170+0000 7fb79fe00640 1 --2- 172.21.15.43:0/2621019839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7a010e010 0x7fb7a0110500 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.170+0000 7fb79fe00640 1 --2- 172.21.15.43:0/2621019839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7a010b6b0 0x7fb7a010dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.170+0000 7fb79fe00640 1 --2- 172.21.15.43:0/2621019839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7a0104c50 0x7fb7a0105060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.170+0000 7fb79fe00640 1 -- 172.21.15.43:0/2621019839 >> 172.21.15.43:0/2621019839 conn(0x7fb7a00fe210 msgr2=0x7fb7a0100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.170+0000 7fb79fe00640 1 -- 172.21.15.43:0/2621019839 shutdown_connections 2024-05-27T07:40:09.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.170+0000 7fb79fe00640 1 -- 172.21.15.43:0/2621019839 wait complete. 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.171+0000 7fb79fe00640 1 Processor -- start 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.171+0000 7fb79fe00640 1 -- start start 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79fe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7a0104c50 0x7fb7a0102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7a010b6b0 0x7fb7a0102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7a010e010 0x7fb7a0102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a0113b70 con 0x7fb7a010b6b0 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79fe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a01139f0 con 0x7fb7a010e010 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79fe00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a0113cf0 con 0x7fb7a0104c50 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7a010b6b0 0x7fb7a0102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7a010b6b0 0x7fb7a0102a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43832/0 (socket says 172.21.15.43:43832) 2024-05-27T07:40:09.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79e000640 1 -- 172.21.15.43:0/3421133191 learned_addr learned my addr 172.21.15.43:0/3421133191 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:09.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79ea00640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7a0104c50 0x7fb7a0102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.172+0000 7fb79f400640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7a010e010 0x7fb7a0102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb79e000640 1 -- 172.21.15.43:0/3421133191 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7a0104c50 msgr2=0x7fb7a0102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb79e000640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7a0104c50 0x7fb7a0102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb79e000640 1 -- 172.21.15.43:0/3421133191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7a010e010 msgr2=0x7fb7a0102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb79e000640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7a010e010 0x7fb7a0102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb79e000640 1 -- 172.21.15.43:0/3421133191 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a01a8910 con 0x7fb7a010b6b0 2024-05-27T07:40:09.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb79ea00640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7a0104c50 0x7fb7a0102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:09.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb79e000640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7a010b6b0 0x7fb7a0102a50 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fb78800eb90 tx=0x7fb78800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb797400640 1 -- 172.21.15.43:0/3421133191 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb788004d20 con 0x7fb7a010b6b0 2024-05-27T07:40:09.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb79fe00640 1 -- 172.21.15.43:0/3421133191 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a01a8c00 con 0x7fb7a010b6b0 2024-05-27T07:40:09.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb797400640 1 -- 172.21.15.43:0/3421133191 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb78800ee90 con 0x7fb7a010b6b0 2024-05-27T07:40:09.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.173+0000 7fb797400640 1 -- 172.21.15.43:0/3421133191 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb788010690 con 0x7fb7a010b6b0 2024-05-27T07:40:09.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.174+0000 7fb79fe00640 1 -- 172.21.15.43:0/3421133191 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a01a9150 con 0x7fb7a010b6b0 2024-05-27T07:40:09.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.175+0000 7fb79fe00640 1 -- 172.21.15.43:0/3421133191 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a006a230 con 0x7fb7a010b6b0 2024-05-27T07:40:09.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.177+0000 7fb797400640 1 -- 172.21.15.43:0/3421133191 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fb788010830 con 0x7fb7a010b6b0 2024-05-27T07:40:09.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.179+0000 7fb797400640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb7800b29d0 0x7fb7800b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.180+0000 7fb797400640 1 -- 172.21.15.43:0/3421133191 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 9584+0+0 (secure 0 0 0) 0x7fb7880e3320 con 0x7fb7a010b6b0 2024-05-27T07:40:09.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.180+0000 7fb79ea00640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb7800b29d0 0x7fb7800b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.181+0000 7fb79ea00640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb7800b29d0 0x7fb7800b4ee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb790005fd0 tx=0x7fb79000b5f0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.181+0000 7fb797400640 1 -- 172.21.15.43:0/3421133191 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb788054df0 con 0x7fb7a010b6b0 2024-05-27T07:40:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.420+0000 7fb79fe00640 1 -- 172.21.15.43:0/3421133191 --> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fb7a010eb30 con 0x7fb7800b29d0 2024-05-27T07:40:09.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.445+0000 7fb797400640 1 -- 172.21.15.43:0/3421133191 <== mgr.35033 v2:172.21.15.43:6800/2368318638 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7fb7a010eb30 con 0x7fb7800b29d0 2024-05-27T07:40:09.443 INFO:teuthology.orchestra.run.smithi043.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-27T00:00:00", "created": "2024-05-27T07:14:13", "first": "2024-05-27T07:15:00", "last": "2024-05-27T07:40:00", "last_pruned": "2024-05-27T07:40:00", "created_count": 26, "pruned_count": 20, "active": true} 2024-05-27T07:40:09.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.451+0000 7fb79fe00640 1 -- 172.21.15.43:0/3421133191 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb7800b29d0 msgr2=0x7fb7800b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.451+0000 7fb79fe00640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb7800b29d0 0x7fb7800b4ee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb790005fd0 tx=0x7fb79000b5f0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.451+0000 7fb79fe00640 1 -- 172.21.15.43:0/3421133191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7a010b6b0 msgr2=0x7fb7a0102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.451+0000 7fb79fe00640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7a010b6b0 0x7fb7a0102a50 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fb78800eb90 tx=0x7fb78800c6a0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.451+0000 7fb79fe00640 1 -- 172.21.15.43:0/3421133191 shutdown_connections 2024-05-27T07:40:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.451+0000 7fb79fe00640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fb7800b29d0 0x7fb7800b4ee0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.451+0000 7fb79fe00640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7a010e010 0x7fb7a0102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.451+0000 7fb79fe00640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7a010b6b0 0x7fb7a0102a50 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.451+0000 7fb79fe00640 1 --2- 172.21.15.43:0/3421133191 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7a0104c50 0x7fb7a0102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.451+0000 7fb79fe00640 1 -- 172.21.15.43:0/3421133191 >> 172.21.15.43:0/3421133191 conn(0x7fb7a00fe210 msgr2=0x7fb7a010e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.452+0000 7fb79fe00640 1 -- 172.21.15.43:0/3421133191 shutdown_connections 2024-05-27T07:40:09.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.453+0000 7fb79fe00640 1 -- 172.21.15.43:0/3421133191 wait complete. 2024-05-27T07:40:09.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:09 smithi043 ceph-mon[31531]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:09.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:09 smithi043 ceph-mon[31531]: pgmap v880: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-05-27T07:40:09.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:09 smithi179 ceph-mon[35888]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:09.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:09 smithi179 ceph-mon[35888]: pgmap v880: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-05-27T07:40:09.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-27T07:40:09.598 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:09 smithi132 ceph-mon[35283]: fsmap cephfs:5 5 up:active 2024-05-27T07:40:09.598 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:09 smithi132 ceph-mon[35283]: pgmap v880: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-05-27T07:40:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.777+0000 7f6b0be00640 1 -- 172.21.15.43:0/4094683545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b04108bb0 msgr2=0x7f6b04102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.777+0000 7f6b0be00640 1 --2- 172.21.15.43:0/4094683545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b04108bb0 0x7f6b04102750 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009900 tx=0x7f6af802a410 comp rx=0 tx=0).stop 2024-05-27T07:40:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.777+0000 7f6b0be00640 1 -- 172.21.15.43:0/4094683545 shutdown_connections 2024-05-27T07:40:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.777+0000 7f6b0be00640 1 --2- 172.21.15.43:0/4094683545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b04109050 0x7f6b04102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.777+0000 7f6b0be00640 1 --2- 172.21.15.43:0/4094683545 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b04103300 0x7f6b0410f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.777+0000 7f6b0be00640 1 --2- 172.21.15.43:0/4094683545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b04108bb0 0x7f6b04102750 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.778+0000 7f6b0be00640 1 -- 172.21.15.43:0/4094683545 >> 172.21.15.43:0/4094683545 conn(0x7f6b040fe380 msgr2=0x7f6b041007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.778+0000 7f6b0be00640 1 -- 172.21.15.43:0/4094683545 shutdown_connections 2024-05-27T07:40:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.778+0000 7f6b0be00640 1 -- 172.21.15.43:0/4094683545 wait complete. 2024-05-27T07:40:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.779+0000 7f6b0be00640 1 Processor -- start 2024-05-27T07:40:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.779+0000 7f6b0be00640 1 -- start start 2024-05-27T07:40:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.779+0000 7f6b0be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b04103300 0x7f6b041a25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b0be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b04108bb0 0x7f6b041a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b0be00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b04109050 0x7f6b0419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b0be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6b04112c50 con 0x7f6b04103300 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b0be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6b04112ad0 con 0x7f6b04108bb0 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b0be00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6b04112dd0 con 0x7f6b04109050 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b09400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b04103300 0x7f6b041a25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b09400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b04103300 0x7f6b041a25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43856/0 (socket says 172.21.15.43:43856) 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b09400640 1 -- 172.21.15.43:0/2709229428 learned_addr learned my addr 172.21.15.43:0/2709229428 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b08a00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b04108bb0 0x7f6b041a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b09e00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b04109050 0x7f6b0419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b09400640 1 -- 172.21.15.43:0/2709229428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b04109050 msgr2=0x7f6b0419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b09400640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b04109050 0x7f6b0419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b09400640 1 -- 172.21.15.43:0/2709229428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b04108bb0 msgr2=0x7f6b041a2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b09400640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b04108bb0 0x7f6b041a2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b09400640 1 -- 172.21.15.43:0/2709229428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0419cf20 con 0x7f6b04103300 2024-05-27T07:40:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.780+0000 7f6b09e00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b04109050 0x7f6b0419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.781+0000 7f6b09400640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b04103300 0x7f6b041a25c0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f6af8005940 tx=0x7f6af80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.781+0000 7f6b02000640 1 -- 172.21.15.43:0/2709229428 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af803f030 con 0x7f6b04103300 2024-05-27T07:40:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.781+0000 7f6b0be00640 1 -- 172.21.15.43:0/2709229428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b0419d1b0 con 0x7f6b04103300 2024-05-27T07:40:09.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.781+0000 7f6b02000640 1 -- 172.21.15.43:0/2709229428 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6af8003c90 con 0x7f6b04103300 2024-05-27T07:40:09.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.781+0000 7f6b0be00640 1 -- 172.21.15.43:0/2709229428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b0419d6a0 con 0x7f6b04103300 2024-05-27T07:40:09.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.781+0000 7f6b02000640 1 -- 172.21.15.43:0/2709229428 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af8049060 con 0x7f6b04103300 2024-05-27T07:40:09.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.783+0000 7f6b0be00640 1 -- 172.21.15.43:0/2709229428 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b04103e90 con 0x7f6b04103300 2024-05-27T07:40:09.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.786+0000 7f6b02000640 1 -- 172.21.15.43:0/2709229428 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6af8042030 con 0x7f6b04103300 2024-05-27T07:40:09.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.787+0000 7f6b02000640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6aec0b2950 0x7f6aec0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:09.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.787+0000 7f6b02000640 1 -- 172.21.15.43:0/2709229428 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 9584+0+0 (secure 0 0 0) 0x7f6af8103500 con 0x7f6b04103300 2024-05-27T07:40:09.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.788+0000 7f6b08a00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6aec0b2950 0x7f6aec0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:09.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.788+0000 7f6b08a00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6aec0b2950 0x7f6aec0b4e60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6af0005fd0 tx=0x7f6af000b510 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:09.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:09.789+0000 7f6b02000640 1 -- 172.21.15.43:0/2709229428 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6b0419d6a0 con 0x7f6b04103300 2024-05-27T07:40:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.023+0000 7f6b0be00640 1 -- 172.21.15.43:0/2709229428 --> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f6b04075600 con 0x7f6aec0b2950 2024-05-27T07:40:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.043+0000 7f6b02000640 1 -- 172.21.15.43:0/2709229428 <== mgr.35033 v2:172.21.15.43:6800/2368318638 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f6b04075600 con 0x7f6aec0b2950 2024-05-27T07:40:10.041 INFO:teuthology.orchestra.run.smithi043.stdout:/ 1m 6m3h 2024-05-27T07:40:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.046+0000 7f6b0be00640 1 -- 172.21.15.43:0/2709229428 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6aec0b2950 msgr2=0x7f6aec0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.046+0000 7f6b0be00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6aec0b2950 0x7f6aec0b4e60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6af0005fd0 tx=0x7f6af000b510 comp rx=0 tx=0).stop 2024-05-27T07:40:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.046+0000 7f6b0be00640 1 -- 172.21.15.43:0/2709229428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b04103300 msgr2=0x7f6b041a25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.046+0000 7f6b0be00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b04103300 0x7f6b041a25c0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f6af8005940 tx=0x7f6af80042f0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.046+0000 7f6b0be00640 1 -- 172.21.15.43:0/2709229428 shutdown_connections 2024-05-27T07:40:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.046+0000 7f6b0be00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6aec0b2950 0x7f6aec0b4e60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.046+0000 7f6b0be00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6b04108bb0 0x7f6b041a2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.046+0000 7f6b0be00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b04109050 0x7f6b0419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.046+0000 7f6b0be00640 1 --2- 172.21.15.43:0/2709229428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b04103300 0x7f6b041a25c0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.046+0000 7f6b0be00640 1 -- 172.21.15.43:0/2709229428 >> 172.21.15.43:0/2709229428 conn(0x7f6b040fe380 msgr2=0x7f6b04074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.047+0000 7f6b0be00640 1 -- 172.21.15.43:0/2709229428 shutdown_connections 2024-05-27T07:40:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:10.047+0000 7f6b0be00640 1 -- 172.21.15.43:0/2709229428 wait complete. 2024-05-27T07:40:10.073 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-27T07:40:10.083 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-27T07:40:10.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:10 smithi043 ceph-mon[31531]: from='client.18618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:40:10.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:10 smithi179 ceph-mon[35888]: from='client.18618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:40:10.598 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:10 smithi132 ceph-mon[35283]: from='client.18618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:40:11.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:11 smithi043 ceph-mon[31531]: from='client.18624 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:40:11.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:11 smithi043 ceph-mon[31531]: pgmap v881: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:40:11.465 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:11 smithi179 ceph-mon[35888]: from='client.18624 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:40:11.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:11 smithi179 ceph-mon[35888]: pgmap v881: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:40:11.598 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:11 smithi132 ceph-mon[35283]: from='client.18624 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-27T07:40:11.598 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:11 smithi132 ceph-mon[35283]: pgmap v881: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-05-27T07:40:12.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:12 smithi043 ceph-mon[31531]: osdmap e163: 12 total, 12 up, 12 in 2024-05-27T07:40:12.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:12 smithi179 ceph-mon[35888]: osdmap e163: 12 total, 12 up, 12 in 2024-05-27T07:40:13.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:12 smithi132 ceph-mon[35283]: osdmap e163: 12 total, 12 up, 12 in 2024-05-27T07:40:13.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:13 smithi043 ceph-mon[31531]: pgmap v883: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-05-27T07:40:13.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:13 smithi179 ceph-mon[35888]: pgmap v883: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-05-27T07:40:14.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:13 smithi132 ceph-mon[35283]: pgmap v883: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-05-27T07:40:15.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:15 smithi043 ceph-mon[31531]: pgmap v884: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-05-27T07:40:15.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:15 smithi179 ceph-mon[35888]: pgmap v884: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-05-27T07:40:16.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:15 smithi132 ceph-mon[35283]: pgmap v884: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.4 MiB/s wr, 138 op/s 2024-05-27T07:40:17.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:17 smithi043 ceph-mon[31531]: pgmap v885: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 338 KiB/s wr, 29 op/s 2024-05-27T07:40:17.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:17 smithi179 ceph-mon[35888]: pgmap v885: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 338 KiB/s wr, 29 op/s 2024-05-27T07:40:18.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:17 smithi132 ceph-mon[35283]: pgmap v885: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 338 KiB/s wr, 29 op/s 2024-05-27T07:40:19.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:19 smithi043 ceph-mon[31531]: pgmap v886: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 338 KiB/s wr, 29 op/s 2024-05-27T07:40:19.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:19 smithi179 ceph-mon[35888]: pgmap v886: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 338 KiB/s wr, 29 op/s 2024-05-27T07:40:20.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:19 smithi132 ceph-mon[35283]: pgmap v886: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 338 KiB/s wr, 29 op/s 2024-05-27T07:40:21.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:21 smithi043 ceph-mon[31531]: pgmap v887: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 338 KiB/s wr, 29 op/s 2024-05-27T07:40:21.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:21 smithi179 ceph-mon[35888]: pgmap v887: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 338 KiB/s wr, 29 op/s 2024-05-27T07:40:22.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:21 smithi132 ceph-mon[35283]: pgmap v887: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 338 KiB/s wr, 29 op/s 2024-05-27T07:40:23.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:23 smithi043 ceph-mon[31531]: pgmap v888: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 571 B/s rd, 74 KiB/s wr, 8 op/s 2024-05-27T07:40:23.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:23 smithi179 ceph-mon[35888]: pgmap v888: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 571 B/s rd, 74 KiB/s wr, 8 op/s 2024-05-27T07:40:24.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:23 smithi132 ceph-mon[35283]: pgmap v888: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 571 B/s rd, 74 KiB/s wr, 8 op/s 2024-05-27T07:40:25.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:25 smithi043 ceph-mon[31531]: pgmap v889: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 66 KiB/s wr, 7 op/s 2024-05-27T07:40:25.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:25 smithi179 ceph-mon[35888]: pgmap v889: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 66 KiB/s wr, 7 op/s 2024-05-27T07:40:26.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:25 smithi132 ceph-mon[35283]: pgmap v889: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 66 KiB/s wr, 7 op/s 2024-05-27T07:40:26.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-05-27T07:40:26.659 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:27.838 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:27 smithi043 ceph-mon[31531]: pgmap v890: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 75 KiB/s wr, 8 op/s 2024-05-27T07:40:27.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:27 smithi179 ceph-mon[35888]: pgmap v890: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 75 KiB/s wr, 8 op/s 2024-05-27T07:40:28.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:27 smithi132 ceph-mon[35283]: pgmap v890: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 75 KiB/s wr, 8 op/s 2024-05-27T07:40:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.522+0000 7ff2c1c00700 1 -- 172.21.15.43:0/4273095469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2bc1045c0 msgr2=0x7ff2bc1049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.522+0000 7ff2c1c00700 1 --2- 172.21.15.43:0/4273095469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2bc1045c0 0x7ff2bc1049d0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0099c0 tx=0x7ff2ac009cd0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.522+0000 7ff2c1c00700 1 -- 172.21.15.43:0/4273095469 shutdown_connections 2024-05-27T07:40:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.522+0000 7ff2c1c00700 1 --2- 172.21.15.43:0/4273095469 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2bc109840 0x7ff2bc1111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.522+0000 7ff2c1c00700 1 --2- 172.21.15.43:0/4273095469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2bc1045c0 0x7ff2bc1049d0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.522+0000 7ff2c1c00700 1 --2- 172.21.15.43:0/4273095469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2bc108ef0 0x7ff2bc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.522+0000 7ff2c1c00700 1 -- 172.21.15.43:0/4273095469 >> 172.21.15.43:0/4273095469 conn(0x7ff2bc0ffe90 msgr2=0x7ff2bc1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.522+0000 7ff2c1c00700 1 -- 172.21.15.43:0/4273095469 shutdown_connections 2024-05-27T07:40:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.522+0000 7ff2c1c00700 1 -- 172.21.15.43:0/4273095469 wait complete. 2024-05-27T07:40:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.523+0000 7ff2c1c00700 1 Processor -- start 2024-05-27T07:40:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.524+0000 7ff2c1c00700 1 -- start start 2024-05-27T07:40:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.524+0000 7ff2c1c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2bc1045c0 0x7ff2bc19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.524+0000 7ff2c1c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2bc108ef0 0x7ff2bc19d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.524+0000 7ff2c1c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2bc109840 0x7ff2bc1a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.524+0000 7ff2c1c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc114320 con 0x7ff2bc108ef0 2024-05-27T07:40:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.524+0000 7ff2c1c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1141a0 con 0x7ff2bc109840 2024-05-27T07:40:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.524+0000 7ff2c1c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1144a0 con 0x7ff2bc1045c0 2024-05-27T07:40:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2ba000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2bc108ef0 0x7ff2bc19d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2ba000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2bc108ef0 0x7ff2bc19d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60028/0 (socket says 172.21.15.43:60028) 2024-05-27T07:40:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2ba000700 1 -- 172.21.15.43:0/2769828251 learned_addr learned my addr 172.21.15.43:0/2769828251 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2baa00700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2bc1045c0 0x7ff2bc19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2bb400700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2bc109840 0x7ff2bc1a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2ba000700 1 -- 172.21.15.43:0/2769828251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2bc1045c0 msgr2=0x7ff2bc19cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2ba000700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2bc1045c0 0x7ff2bc19cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2ba000700 1 -- 172.21.15.43:0/2769828251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2bc109840 msgr2=0x7ff2bc1a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2ba000700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2bc109840 0x7ff2bc1a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2ba000700 1 -- 172.21.15.43:0/2769828251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2bc1a1e60 con 0x7ff2bc108ef0 2024-05-27T07:40:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.525+0000 7ff2bb400700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2bc109840 0x7ff2bc1a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.526+0000 7ff2ba000700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2bc108ef0 0x7ff2bc19d530 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7ff2a800d980 tx=0x7ff2a800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.526+0000 7ff2b3400700 1 -- 172.21.15.43:0/2769828251 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8017070 con 0x7ff2bc108ef0 2024-05-27T07:40:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.526+0000 7ff2b3400700 1 -- 172.21.15.43:0/2769828251 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2a8009dc0 con 0x7ff2bc108ef0 2024-05-27T07:40:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.526+0000 7ff2b3400700 1 -- 172.21.15.43:0/2769828251 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a800f3c0 con 0x7ff2bc108ef0 2024-05-27T07:40:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.526+0000 7ff2c1c00700 1 -- 172.21.15.43:0/2769828251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2bc1a2150 con 0x7ff2bc108ef0 2024-05-27T07:40:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.526+0000 7ff2c1c00700 1 -- 172.21.15.43:0/2769828251 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2bc1a9a70 con 0x7ff2bc108ef0 2024-05-27T07:40:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.531+0000 7ff2b3400700 1 -- 172.21.15.43:0/2769828251 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ff2a8009930 con 0x7ff2bc108ef0 2024-05-27T07:40:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.531+0000 7ff2d0836b80 1 Processor -- start 2024-05-27T07:40:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.532+0000 7ff2d0836b80 1 -- start start 2024-05-27T07:40:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.532+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e35497be40 0x55e354b007f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.532+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354b00d30 0x55e354afa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.532+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e354afadb0 0x55e354afb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.532+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e354a28e40 con 0x55e35497be40 2024-05-27T07:40:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.532+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e354a189e0 con 0x55e354b00d30 2024-05-27T07:40:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.532+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e354a1ca20 con 0x55e354afadb0 2024-05-27T07:40:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.532+0000 7ff2bb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e35497be40 0x55e354b007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.533+0000 7ff2bb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e35497be40 0x55e354b007f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60038/0 (socket says 172.21.15.43:60038) 2024-05-27T07:40:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.533+0000 7ff2bb400700 1 -- 172.21.15.43:0/2893718843 learned_addr learned my addr 172.21.15.43:0/2893718843 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.533+0000 7ff2baa00700 1 --2- 172.21.15.43:0/2893718843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354b00d30 0x55e354afa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.533+0000 7ff2ba000700 1 --2- 172.21.15.43:0/2893718843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e354afadb0 0x55e354afb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.533+0000 7ff2bb400700 1 -- 172.21.15.43:0/2893718843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e354afadb0 msgr2=0x55e354afb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.533+0000 7ff2b3400700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff2a00b29f0 0x7ff2a00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.533+0000 7ff2bb400700 1 --2- 172.21.15.43:0/2893718843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e354afadb0 0x55e354afb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.533+0000 7ff2bb400700 1 -- 172.21.15.43:0/2893718843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354b00d30 msgr2=0x55e354afa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.533+0000 7ff2bb400700 1 --2- 172.21.15.43:0/2893718843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354b00d30 0x55e354afa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.533+0000 7ff2bb400700 1 -- 172.21.15.43:0/2893718843 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3549e1420 con 0x55e35497be40 2024-05-27T07:40:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.534+0000 7ff2b3400700 1 -- 172.21.15.43:0/2769828251 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7ff2a80e2580 con 0x7ff2bc108ef0 2024-05-27T07:40:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.534+0000 7ff2bb400700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff2a00b29f0 0x7ff2a00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.534+0000 7ff2bb400700 1 --2- 172.21.15.43:0/2893718843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e35497be40 0x55e354b007f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009dd0 tx=0x7ff2b40077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc37e0e7805a6fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.534+0000 7ff2a7e00700 1 -- 172.21.15.43:0/2893718843 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40149d0 con 0x55e35497be40 2024-05-27T07:40:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.534+0000 7ff2a7e00700 1 -- 172.21.15.43:0/2893718843 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2b4014bf0 con 0x55e35497be40 2024-05-27T07:40:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.534+0000 7ff2a7e00700 1 -- 172.21.15.43:0/2893718843 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b401f5a0 con 0x55e35497be40 2024-05-27T07:40:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.535+0000 7ff2d0836b80 1 -- 172.21.15.43:0/2893718843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e35497be40 msgr2=0x55e354b007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.535+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/2893718843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e35497be40 0x55e354b007f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009dd0 tx=0x7ff2b40077e0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.535+0000 7ff2bb400700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff2a00b29f0 0x7ff2a00b4f00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff2b401e000 tx=0x7ff2b401fcd0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.536+0000 7ff2d0836b80 1 -- 172.21.15.43:0/2893718843 shutdown_connections 2024-05-27T07:40:29.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.536+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/2893718843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e354afadb0 0x55e354afb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.536+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/2893718843 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e35497be40 0x55e354b007f0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.536+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/2893718843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354b00d30 0x55e354afa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.536+0000 7ff2d0836b80 1 -- 172.21.15.43:0/2893718843 >> 172.21.15.43:0/2893718843 conn(0x55e3549dbfc0 msgr2=0x55e354af58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:29.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.536+0000 7ff2d0836b80 1 -- 172.21.15.43:0/2893718843 shutdown_connections 2024-05-27T07:40:29.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.536+0000 7ff2d0836b80 1 -- 172.21.15.43:0/2893718843 wait complete. 2024-05-27T07:40:29.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.537+0000 7ff2d0836b80 1 -- start start 2024-05-27T07:40:29.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.537+0000 7ff2d0836b80 1 Processor -- start 2024-05-27T07:40:29.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.537+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e35497be40 0x55e354b04440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.537+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354afadb0 0x55e354b04980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e354b00d30 0x55e354b04ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e35491efa0 con 0x55e354afadb0 2024-05-27T07:40:29.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55e3549b9760 con 0x55e35497be40 2024-05-27T07:40:29.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e354b07800 con 0x55e354b00d30 2024-05-27T07:40:29.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2d0836b80 10 client.0 _ll_register_callbacks cb 0x55e3549c2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2ba000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354afadb0 0x55e354b04980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2ba000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354afadb0 0x55e354b04980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60040/0 (socket says 172.21.15.43:60040) 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2ba000700 1 -- 172.21.15.43:0/3513380342 learned_addr learned my addr 172.21.15.43:0/3513380342 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2baa00700 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e35497be40 0x55e354b04440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2bb400700 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e354b00d30 0x55e354b04ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2ba000700 1 -- 172.21.15.43:0/3513380342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e354b00d30 msgr2=0x55e354b04ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2ba000700 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e354b00d30 0x55e354b04ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2ba000700 1 -- 172.21.15.43:0/3513380342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e35497be40 msgr2=0x55e354b04440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2ba000700 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e35497be40 0x55e354b04440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.538+0000 7ff2ba000700 1 -- 172.21.15.43:0/3513380342 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3549688d0 con 0x55e354afadb0 2024-05-27T07:40:29.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.539+0000 7ff2ba000700 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354afadb0 0x55e354b04980 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80a93c0 tx=0x7ff2a80a92b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831865db612ed451 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:29.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.539+0000 7ff2a4c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:29.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.539+0000 7ff2a4c00700 1 -- 172.21.15.43:0/3513380342 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8020050 con 0x55e354afadb0 2024-05-27T07:40:29.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.539+0000 7ff2a4c00700 1 -- 172.21.15.43:0/3513380342 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7ff2a80a5a20 con 0x55e354afadb0 2024-05-27T07:40:29.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.539+0000 7ff2d0836b80 1 -- 172.21.15.43:0/3513380342 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2b4014bf0 con 0x55e354afadb0 2024-05-27T07:40:29.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.539+0000 7ff2a4c00700 1 -- 172.21.15.43:0/3513380342 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7ff2a80a8ba0 con 0x55e354afadb0 2024-05-27T07:40:29.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.539+0000 7ff2d0836b80 10 client.18636 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:29.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.539+0000 7ff2d0836b80 1 -- 172.21.15.43:0/3513380342 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x55e3549e9060 con 0x55e354afadb0 2024-05-27T07:40:29.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.540+0000 7ff2a4c00700 1 -- 172.21.15.43:0/3513380342 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2b4014bf0 con 0x55e354afadb0 2024-05-27T07:40:29.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.540+0000 7ff2a4c00700 1 -- 172.21.15.43:0/3513380342 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7ff2a80e20f0 con 0x55e354afadb0 2024-05-27T07:40:29.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.540+0000 7ff2d0836b80 10 client.18636 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:29.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.540+0000 7ff2d0836b80 10 client.18636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:29.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.540+0000 7ff2d0836b80 20 client.18636 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:29.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.540+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e354b149e0 0x55e354b16e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.540+0000 7ff2d0836b80 4 client.18636 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:29.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.540+0000 7ff2d0836b80 1 -- 172.21.15.43:0/3513380342 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e354a2ad90 con 0x55e354b149e0 2024-05-27T07:40:29.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.541+0000 7ff2baa00700 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e354b149e0 0x55e354b16e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.542+0000 7ff2baa00700 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e354b149e0 0x55e354b16e30 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2819b0ff3d005324 server_cookie=4b5fee26e09a2030 in_seq=0 out_seq=0 2024-05-27T07:40:29.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.542+0000 7ff2a4c00700 10 client.18636 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:40:29.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2a4c00700 1 -- 172.21.15.43:0/3513380342 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e354a2ad90 con 0x55e354b149e0 2024-05-27T07:40:29.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2a4c00700 10 client.18636 handle_command_reply: tid=0 2024-05-27T07:40:29.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2d0836b80 1 client.18636 shutdown 2024-05-27T07:40:29.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2d0836b80 1 -- 172.21.15.43:0/3513380342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354afadb0 msgr2=0x55e354b04980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:29.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354afadb0 0x55e354b04980 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80a93c0 tx=0x7ff2a80a92b0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2d0836b80 1 -- 172.21.15.43:0/3513380342 shutdown_connections 2024-05-27T07:40:29.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e354b00d30 0x55e354b04ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e35497be40 0x55e354b04440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e354b149e0 0x55e354b16e30 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/3513380342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354afadb0 0x55e354b04980 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2d0836b80 1 -- 172.21.15.43:0/3513380342 >> 172.21.15.43:0/3513380342 conn(0x55e3549dbfc0 msgr2=0x55e354af5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:29.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2a4c00700 0 client.18636 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:40:29.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.544+0000 7ff2d0836b80 1 -- 172.21.15.43:0/3513380342 shutdown_connections 2024-05-27T07:40:29.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.545+0000 7ff2d0836b80 1 -- 172.21.15.43:0/3513380342 wait complete. 2024-05-27T07:40:29.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.545+0000 7ff2d0836b80 20 client.18636 trim_cache size 0 max 16384 2024-05-27T07:40:29.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.567+0000 7ff2d0836b80 1 Processor -- start 2024-05-27T07:40:29.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.567+0000 7ff2d0836b80 1 -- start start 2024-05-27T07:40:29.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.568+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e35497be40 0x55e354afa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.568+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354afadb0 0x55e354b068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.568+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b10860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.568+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1b2c80 con 0x55e354b00d30 2024-05-27T07:40:29.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.568+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1b1080 con 0x55e354afadb0 2024-05-27T07:40:29.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.568+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1b2580 con 0x55e35497be40 2024-05-27T07:40:29.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.568+0000 7ff2bb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b10860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.568+0000 7ff2ba000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354afadb0 0x55e354b068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.568+0000 7ff2bb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b10860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60048/0 (socket says 172.21.15.43:60048) 2024-05-27T07:40:29.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.568+0000 7ff2bb400700 1 -- 172.21.15.43:0/4244557698 learned_addr learned my addr 172.21.15.43:0/4244557698 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:29.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.569+0000 7ff2baa00700 1 --2- 172.21.15.43:0/4244557698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e35497be40 0x55e354afa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.569+0000 7ff2bb400700 1 -- 172.21.15.43:0/4244557698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e35497be40 msgr2=0x55e354afa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:29.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.569+0000 7ff2bb400700 1 --2- 172.21.15.43:0/4244557698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e35497be40 0x55e354afa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.569+0000 7ff2bb400700 1 -- 172.21.15.43:0/4244557698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354afadb0 msgr2=0x55e354b068e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:29.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.569+0000 7ff2bb400700 1 --2- 172.21.15.43:0/4244557698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354afadb0 0x55e354b068e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.569+0000 7ff2bb400700 1 -- 172.21.15.43:0/4244557698 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e354b05230 con 0x55e354b00d30 2024-05-27T07:40:29.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.569+0000 7ff2bb400700 1 --2- 172.21.15.43:0/4244557698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b10860 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4007b30 tx=0x7ff2b4023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36749264e49cbb5a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:29.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.570+0000 7ff2a7400700 1 -- 172.21.15.43:0/4244557698 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4031070 con 0x55e354b00d30 2024-05-27T07:40:29.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.570+0000 7ff2a7400700 1 -- 172.21.15.43:0/4244557698 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2b401f540 con 0x55e354b00d30 2024-05-27T07:40:29.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.570+0000 7ff2a7400700 1 -- 172.21.15.43:0/4244557698 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4014e30 con 0x55e354b00d30 2024-05-27T07:40:29.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.571+0000 7ff2d0836b80 1 -- 172.21.15.43:0/4244557698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 msgr2=0x55e354b10860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:29.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.571+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/4244557698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b10860 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4007b30 tx=0x7ff2b4023000 comp rx=0 tx=0).stop 2024-05-27T07:40:29.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.571+0000 7ff2d0836b80 1 -- 172.21.15.43:0/4244557698 shutdown_connections 2024-05-27T07:40:29.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.571+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/4244557698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b10860 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.571+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/4244557698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e35497be40 0x55e354afa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.571+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/4244557698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354afadb0 0x55e354b068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.571+0000 7ff2d0836b80 1 -- 172.21.15.43:0/4244557698 >> 172.21.15.43:0/4244557698 conn(0x55e3549dbfc0 msgr2=0x55e354b01430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:29.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.571+0000 7ff2d0836b80 1 -- 172.21.15.43:0/4244557698 shutdown_connections 2024-05-27T07:40:29.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.571+0000 7ff2d0836b80 1 -- 172.21.15.43:0/4244557698 wait complete. 2024-05-27T07:40:29.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.571+0000 7ff2d0836b80 1 -- start start 2024-05-27T07:40:29.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.572+0000 7ff2d0836b80 1 Processor -- start 2024-05-27T07:40:29.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.572+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e35497be40 0x55e354b1c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.572+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354afadb0 0x55e354b1c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.572+0000 7ff2d0836b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b1cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.572+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1b1eb0 con 0x55e354b00d30 2024-05-27T07:40:29.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.572+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1b17b0 con 0x55e354afadb0 2024-05-27T07:40:29.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.572+0000 7ff2d0836b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55e354afb970 con 0x55e35497be40 2024-05-27T07:40:29.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.572+0000 7ff2d0836b80 10 client.0 _ll_register_callbacks cb 0x55e3549c2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:29.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.573+0000 7ff2bb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b1cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.573+0000 7ff2bb400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b1cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60060/0 (socket says 172.21.15.43:60060) 2024-05-27T07:40:29.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.573+0000 7ff2bb400700 1 -- 172.21.15.43:0/1419572596 learned_addr learned my addr 172.21.15.43:0/1419572596 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:29.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.573+0000 7ff2baa00700 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e35497be40 0x55e354b1c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.573+0000 7ff2ba000700 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354afadb0 0x55e354b1c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.573+0000 7ff2bb400700 1 -- 172.21.15.43:0/1419572596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e35497be40 msgr2=0x55e354b1c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.573+0000 7ff2bb400700 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e35497be40 0x55e354b1c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.573+0000 7ff2bb400700 1 -- 172.21.15.43:0/1419572596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354afadb0 msgr2=0x55e354b1c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.573+0000 7ff2bb400700 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354afadb0 0x55e354b1c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.573+0000 7ff2bb400700 1 -- 172.21.15.43:0/1419572596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e354b7e4e0 con 0x55e354b00d30 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.574+0000 7ff2bb400700 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b1cb20 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009e20 tx=0x7ff2b4009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf049a5912533b80 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.574+0000 7ff2a4c00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.574+0000 7ff2a4c00700 1 -- 172.21.15.43:0/1419572596 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b403b020 con 0x55e354b00d30 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.574+0000 7ff2a4c00700 1 -- 172.21.15.43:0/1419572596 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7ff2b4020930 con 0x55e354b00d30 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.574+0000 7ff2d0836b80 1 -- 172.21.15.43:0/1419572596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2b401f540 con 0x55e354b00d30 2024-05-27T07:40:29.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.574+0000 7ff2a4c00700 1 -- 172.21.15.43:0/1419572596 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7ff2b4046050 con 0x55e354b00d30 2024-05-27T07:40:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.574+0000 7ff2d0836b80 10 client.18642 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.574+0000 7ff2d0836b80 1 -- 172.21.15.43:0/1419572596 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x55e354b7f620 con 0x55e354b00d30 2024-05-27T07:40:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.575+0000 7ff2a4c00700 1 -- 172.21.15.43:0/1419572596 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff2b401f540 con 0x55e354b00d30 2024-05-27T07:40:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.575+0000 7ff2a4c00700 1 -- 172.21.15.43:0/1419572596 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7ff2b40219d0 con 0x55e354b00d30 2024-05-27T07:40:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.575+0000 7ff2d0836b80 10 client.18642 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.575+0000 7ff2d0836b80 10 client.18642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.575+0000 7ff2d0836b80 20 client.18642 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.575+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e354b7fa70 0x55e354b81e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.575+0000 7ff2d0836b80 4 client.18642 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-27T07:40:29.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.575+0000 7ff2d0836b80 1 -- 172.21.15.43:0/1419572596 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e354a24570 con 0x55e354b7fa70 2024-05-27T07:40:29.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.576+0000 7ff2baa00700 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e354b7fa70 0x55e354b81e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:29.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.577+0000 7ff2baa00700 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e354b7fa70 0x55e354b81e90 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3940c6f8b2c624e server_cookie=b097884a168e05b8 in_seq=0 out_seq=0 2024-05-27T07:40:29.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.577+0000 7ff2a4c00700 10 client.18642 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.577+0000 7ff2a4c00700 1 -- 172.21.15.43:0/1419572596 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e354a24570 con 0x55e354b7fa70 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.577+0000 7ff2a4c00700 10 client.18642 handle_command_reply: tid=0 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.577+0000 7ff2d0836b80 1 client.18642 shutdown 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.577+0000 7ff2d0836b80 1 -- 172.21.15.43:0/1419572596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 msgr2=0x55e354b1cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.577+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b1cb20 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009e20 tx=0x7ff2b4009230 comp rx=0 tx=0).stop 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.578+0000 7ff2d0836b80 1 -- 172.21.15.43:0/1419572596 shutdown_connections 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.578+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55e354b7fa70 0x55e354b81e90 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.578+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e354b00d30 0x55e354b1cb20 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.578+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e35497be40 0x55e354b1c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.578+0000 7ff2d0836b80 1 --2- 172.21.15.43:0/1419572596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e354afadb0 0x55e354b1c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.578+0000 7ff2d0836b80 1 -- 172.21.15.43:0/1419572596 >> 172.21.15.43:0/1419572596 conn(0x55e3549dbfc0 msgr2=0x55e354afe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:29.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.578+0000 7ff2a4c00700 0 client.18642 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:40:29.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.578+0000 7ff2d0836b80 1 -- 172.21.15.43:0/1419572596 shutdown_connections 2024-05-27T07:40:29.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.578+0000 7ff2d0836b80 1 -- 172.21.15.43:0/1419572596 wait complete. 2024-05-27T07:40:29.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.578+0000 7ff2d0836b80 20 client.18642 trim_cache size 0 max 16384 2024-05-27T07:40:29.578 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:40:29.578 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-05-27T07:40:29.578 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-05-27T07:40:29.578 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:40:29.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 -- 172.21.15.43:0/2769828251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff2a00b29f0 msgr2=0x7ff2a00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:29.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff2a00b29f0 0x7ff2a00b4f00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff2b401e000 tx=0x7ff2b401fcd0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 -- 172.21.15.43:0/2769828251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2bc108ef0 msgr2=0x7ff2bc19d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:29.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2bc108ef0 0x7ff2bc19d530 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7ff2a800d980 tx=0x7ff2a800dc90 comp rx=0 tx=0).stop 2024-05-27T07:40:29.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 -- 172.21.15.43:0/2769828251 shutdown_connections 2024-05-27T07:40:29.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff2a00b29f0 0x7ff2a00b4f00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2bc109840 0x7ff2bc1a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2bc1045c0 0x7ff2bc19cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 --2- 172.21.15.43:0/2769828251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff2bc108ef0 0x7ff2bc19d530 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:29.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 -- 172.21.15.43:0/2769828251 >> 172.21.15.43:0/2769828251 conn(0x7ff2bc0ffe90 msgr2=0x7ff2bc107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:29.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 -- 172.21.15.43:0/2769828251 shutdown_connections 2024-05-27T07:40:29.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:29.579+0000 7ff2c1c00700 1 -- 172.21.15.43:0/2769828251 wait complete. 2024-05-27T07:40:29.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:29 smithi043 ceph-mon[31531]: pgmap v891: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s wr, 1 op/s 2024-05-27T07:40:29.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:29 smithi179 ceph-mon[35888]: pgmap v891: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s wr, 1 op/s 2024-05-27T07:40:30.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:29 smithi132 ceph-mon[35283]: pgmap v891: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s wr, 1 op/s 2024-05-27T07:40:30.462 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-27T07:40:30.463 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-27T07:40:30.914 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:31.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:31 smithi043 ceph-mon[31531]: pgmap v892: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s wr, 1 op/s 2024-05-27T07:40:31.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:31 smithi179 ceph-mon[35888]: pgmap v892: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s wr, 1 op/s 2024-05-27T07:40:32.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:31 smithi132 ceph-mon[35283]: pgmap v892: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s wr, 1 op/s 2024-05-27T07:40:32.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:32 smithi043 ceph-mon[31531]: pgmap v893: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s wr, 1 op/s 2024-05-27T07:40:33.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:32 smithi132 ceph-mon[35283]: pgmap v893: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s wr, 1 op/s 2024-05-27T07:40:33.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:32 smithi179 ceph-mon[35888]: pgmap v893: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s wr, 1 op/s 2024-05-27T07:40:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.269+0000 7f9078200700 1 -- 172.21.15.43:0/2372941837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9070115360 msgr2=0x7f9070117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.269+0000 7f9078200700 1 --2- 172.21.15.43:0/2372941837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9070115360 0x7f9070117780 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f906800b0a0 tx=0x7f906800b3b0 comp rx=0 tx=0).stop 2024-05-27T07:40:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.269+0000 7f9078200700 1 -- 172.21.15.43:0/2372941837 shutdown_connections 2024-05-27T07:40:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.269+0000 7f9078200700 1 --2- 172.21.15.43:0/2372941837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9070115360 0x7f9070117780 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.269+0000 7f9078200700 1 --2- 172.21.15.43:0/2372941837 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9070075b50 0x7f9070075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.269+0000 7f9078200700 1 --2- 172.21.15.43:0/2372941837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90700764a0 0x7f9070114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.269+0000 7f9078200700 1 -- 172.21.15.43:0/2372941837 >> 172.21.15.43:0/2372941837 conn(0x7f9070100030 msgr2=0x7f9070102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.269+0000 7f9078200700 1 -- 172.21.15.43:0/2372941837 shutdown_connections 2024-05-27T07:40:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.269+0000 7f9078200700 1 -- 172.21.15.43:0/2372941837 wait complete. 2024-05-27T07:40:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.271+0000 7f9078200700 1 Processor -- start 2024-05-27T07:40:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.271+0000 7f9078200700 1 -- start start 2024-05-27T07:40:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.271+0000 7f9078200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9070075b50 0x7f90701a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.271+0000 7f9078200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90700764a0 0x7f90701a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.271+0000 7f9078200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9070115360 0x7f90701a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.271+0000 7f9078200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f907011a920 con 0x7f9070115360 2024-05-27T07:40:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.271+0000 7f9078200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f907011a7a0 con 0x7f9070075b50 2024-05-27T07:40:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.271+0000 7f9078200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f907011aaa0 con 0x7f90700764a0 2024-05-27T07:40:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.272+0000 7f9076200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9070115360 0x7f90701a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.272+0000 7f9076200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9070115360 0x7f90701a5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60088/0 (socket says 172.21.15.43:60088) 2024-05-27T07:40:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.272+0000 7f9076200700 1 -- 172.21.15.43:0/2114842921 learned_addr learned my addr 172.21.15.43:0/2114842921 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.272+0000 7f9075800700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9070075b50 0x7f90701a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.272+0000 7f9076200700 1 -- 172.21.15.43:0/2114842921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90700764a0 msgr2=0x7f90701a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.272+0000 7f9076200700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90700764a0 0x7f90701a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.272+0000 7f9076200700 1 -- 172.21.15.43:0/2114842921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9070075b50 msgr2=0x7f90701a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.272+0000 7f9076200700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9070075b50 0x7f90701a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.272+0000 7f9076200700 1 -- 172.21.15.43:0/2114842921 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90701a61c0 con 0x7f9070115360 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.273+0000 7f9076200700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9070115360 0x7f90701a5b20 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f906800b830 tx=0x7f9068004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.273+0000 7f906e000700 1 -- 172.21.15.43:0/2114842921 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9068021070 con 0x7f9070115360 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.273+0000 7f906e000700 1 -- 172.21.15.43:0/2114842921 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9068007b80 con 0x7f9070115360 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.273+0000 7f9078200700 1 -- 172.21.15.43:0/2114842921 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90701a6450 con 0x7f9070115360 2024-05-27T07:40:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.273+0000 7f906e000700 1 -- 172.21.15.43:0/2114842921 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90680052a0 con 0x7f9070115360 2024-05-27T07:40:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.273+0000 7f9078200700 1 -- 172.21.15.43:0/2114842921 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90701a6930 con 0x7f9070115360 2024-05-27T07:40:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.274+0000 7f9078200700 1 -- 172.21.15.43:0/2114842921 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90700512b0 con 0x7f9070115360 2024-05-27T07:40:33.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.277+0000 7f906e000700 1 -- 172.21.15.43:0/2114842921 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f9068024070 con 0x7f9070115360 2024-05-27T07:40:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.279+0000 7f906e000700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f90580b29f0 0x7f90580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.279+0000 7f906e000700 1 -- 172.21.15.43:0/2114842921 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f906800f020 con 0x7f9070115360 2024-05-27T07:40:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.279+0000 7f9075800700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f90580b29f0 0x7f90580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.280+0000 7f9075800700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f90580b29f0 0x7f90580b4f00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9064005950 tx=0x7f906400b410 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.281+0000 7f906e000700 1 -- 172.21.15.43:0/2114842921 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f906802b020 con 0x7f9070115360 2024-05-27T07:40:33.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.571+0000 7f9078200700 1 -- 172.21.15.43:0/2114842921 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f907002e5e0 con 0x7f9070115360 2024-05-27T07:40:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.574+0000 7f906e000700 1 -- 172.21.15.43:0/2114842921 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6132 (secure 0 0 0) 0x7f90680abb10 con 0x7f9070115360 2024-05-27T07:40:33.572 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:40:33.573 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":152,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-27T07:08:25.128097+0000","modified":"2024-05-27T07:40:07.879740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34484,"mds_1":34448,"mds_2":34424,"mds_3":34403,"mds_4":24484},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24484":{"gid":24484,"name":"i","rank":4,"incarnation":26,"state":"up:active","state_seq":33,"addr":"172.21.15.179:6839/93338133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":93338133},{"type":"v1","addr":"172.21.15.179:6839","nonce":93338133}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34403":{"gid":34403,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":40,"addr":"172.21.15.132:6833/512644396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":512644396},{"type":"v1","addr":"172.21.15.132:6833","nonce":512644396}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"e","rank":2,"incarnation":22,"state":"up:active","state_seq":36,"addr":"172.21.15.132:6835/1074601310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1074601310},{"type":"v1","addr":"172.21.15.132:6835","nonce":1074601310}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34448":{"gid":34448,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":33,"addr":"172.21.15.132:6837/2777087908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":2777087908},{"type":"v1","addr":"172.21.15.132:6837","nonce":2777087908}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34484":{"gid":34484,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.132:6839/4104367678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":4104367678},{"type":"v1","addr":"172.21.15.132:6839","nonce":4104367678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34484,"qdb_cluster":[34448,34484,34424,34403,24484]},"id":1}]} 2024-05-27T07:40:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 -- 172.21.15.43:0/2114842921 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f90580b29f0 msgr2=0x7f90580b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f90580b29f0 0x7f90580b4f00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9064005950 tx=0x7f906400b410 comp rx=0 tx=0).stop 2024-05-27T07:40:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 -- 172.21.15.43:0/2114842921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9070115360 msgr2=0x7f90701a5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9070115360 0x7f90701a5b20 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f906800b830 tx=0x7f9068004d00 comp rx=0 tx=0).stop 2024-05-27T07:40:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 -- 172.21.15.43:0/2114842921 shutdown_connections 2024-05-27T07:40:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f90580b29f0 0x7f90580b4f00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9070115360 0x7f90701a5b20 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9070075b50 0x7f90701a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 --2- 172.21.15.43:0/2114842921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90700764a0 0x7f90701a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 -- 172.21.15.43:0/2114842921 >> 172.21.15.43:0/2114842921 conn(0x7f9070100030 msgr2=0x7f90701006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:33.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.578+0000 7f9078200700 1 -- 172.21.15.43:0/2114842921 shutdown_connections 2024-05-27T07:40:33.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:33.579+0000 7f9078200700 1 -- 172.21.15.43:0/2114842921 wait complete. 2024-05-27T07:40:33.578 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 152 2024-05-27T07:40:33.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:33 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2114842921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:34.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:33 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2114842921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:34.215 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:33 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2114842921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-27T07:40:34.400 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-05-27T07:40:34.877 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:35.030 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:34 smithi043 ceph-mon[31531]: pgmap v894: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s wr, 0 op/s 2024-05-27T07:40:35.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:34 smithi132 ceph-mon[35283]: pgmap v894: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s wr, 0 op/s 2024-05-27T07:40:35.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:34 smithi179 ceph-mon[35888]: pgmap v894: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s wr, 0 op/s 2024-05-27T07:40:37.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.329+0000 7fd09ea00700 1 -- 172.21.15.43:0/1841810609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0981045e0 msgr2=0x7fd0981049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:37.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.329+0000 7fd09ea00700 1 --2- 172.21.15.43:0/1841810609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0981045e0 0x7fd0981049f0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd0880099c0 tx=0x7fd088009cd0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.329+0000 7fd09ea00700 1 -- 172.21.15.43:0/1841810609 shutdown_connections 2024-05-27T07:40:37.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.329+0000 7fd09ea00700 1 --2- 172.21.15.43:0/1841810609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd098109860 0x7fd0981111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.329+0000 7fd09ea00700 1 --2- 172.21.15.43:0/1841810609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0981045e0 0x7fd0981049f0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.329+0000 7fd09ea00700 1 --2- 172.21.15.43:0/1841810609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd098108f10 0x7fd098109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.330+0000 7fd09ea00700 1 -- 172.21.15.43:0/1841810609 >> 172.21.15.43:0/1841810609 conn(0x7fd098100030 msgr2=0x7fd098102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:37.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.330+0000 7fd09ea00700 1 -- 172.21.15.43:0/1841810609 shutdown_connections 2024-05-27T07:40:37.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.330+0000 7fd09ea00700 1 -- 172.21.15.43:0/1841810609 wait complete. 2024-05-27T07:40:37.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.331+0000 7fd09ea00700 1 Processor -- start 2024-05-27T07:40:37.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.332+0000 7fd09ea00700 1 -- start start 2024-05-27T07:40:37.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.332+0000 7fd09ea00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0981045e0 0x7fd09819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.332+0000 7fd09ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd098108f10 0x7fd09819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.332+0000 7fd09ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd098109860 0x7fd09819f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.332+0000 7fd09ea00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd098114390 con 0x7fd098109860 2024-05-27T07:40:37.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.332+0000 7fd09ea00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd098114210 con 0x7fd098108f10 2024-05-27T07:40:37.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.332+0000 7fd09ea00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd098114510 con 0x7fd0981045e0 2024-05-27T07:40:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.333+0000 7fd09ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd098109860 0x7fd09819f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.333+0000 7fd09ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd098109860 0x7fd09819f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58864/0 (socket says 172.21.15.43:58864) 2024-05-27T07:40:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.333+0000 7fd09ca00700 1 -- 172.21.15.43:0/2151448795 learned_addr learned my addr 172.21.15.43:0/2151448795 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.333+0000 7fd097e00700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0981045e0 0x7fd09819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.333+0000 7fd08f400700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd098108f10 0x7fd09819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.333+0000 7fd09ca00700 1 -- 172.21.15.43:0/2151448795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0981045e0 msgr2=0x7fd09819ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.333+0000 7fd09ca00700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0981045e0 0x7fd09819ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.333+0000 7fd09ca00700 1 -- 172.21.15.43:0/2151448795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd098108f10 msgr2=0x7fd09819b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.333+0000 7fd09ca00700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd098108f10 0x7fd09819b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.333+0000 7fd09ca00700 1 -- 172.21.15.43:0/2151448795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09819fc70 con 0x7fd098109860 2024-05-27T07:40:37.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.334+0000 7fd09ca00700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd098109860 0x7fd09819f5d0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd090007ee0 tx=0x7fd09000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:37.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.334+0000 7fd095600700 1 -- 172.21.15.43:0/2151448795 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd090004bb0 con 0x7fd098109860 2024-05-27T07:40:37.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.334+0000 7fd095600700 1 -- 172.21.15.43:0/2151448795 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd090004d50 con 0x7fd098109860 2024-05-27T07:40:37.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.334+0000 7fd095600700 1 -- 172.21.15.43:0/2151448795 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900058b0 con 0x7fd098109860 2024-05-27T07:40:37.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.335+0000 7fd09ea00700 1 -- 172.21.15.43:0/2151448795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09819ff60 con 0x7fd098109860 2024-05-27T07:40:37.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.335+0000 7fd09ea00700 1 -- 172.21.15.43:0/2151448795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0980759e0 con 0x7fd098109860 2024-05-27T07:40:37.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.339+0000 7fd095600700 1 -- 172.21.15.43:0/2151448795 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd09001a070 con 0x7fd098109860 2024-05-27T07:40:37.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.340+0000 7fd0ad52eb80 1 Processor -- start 2024-05-27T07:40:37.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.341+0000 7fd0ad52eb80 1 -- start start 2024-05-27T07:40:37.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.341+0000 7fd095600700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd07c0b29f0 0x7fd07c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.341+0000 7fd095600700 1 -- 172.21.15.43:0/2151448795 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7fd0900e17c0 con 0x7fd098109860 2024-05-27T07:40:37.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.341+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136c47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.341+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136be870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.341+0000 7fd08f400700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd07c0b29f0 0x7fd07c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.341+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136bf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.341+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b4135ece40 con 0x55b4136bedb0 2024-05-27T07:40:37.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.341+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b4135dc9e0 con 0x55b41353fe40 2024-05-27T07:40:37.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.341+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b4135e0a20 con 0x55b4136c4d30 2024-05-27T07:40:37.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.342+0000 7fd09ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136bf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.342+0000 7fd09ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136bf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58870/0 (socket says 172.21.15.43:58870) 2024-05-27T07:40:37.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.342+0000 7fd09ca00700 1 -- 172.21.15.43:0/1236744884 learned_addr learned my addr 172.21.15.43:0/1236744884 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:37.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.342+0000 7fd08f400700 1 --2- 172.21.15.43:0/1236744884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136be870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.342+0000 7fd097e00700 1 --2- 172.21.15.43:0/1236744884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136c47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.342+0000 7fd09ca00700 1 -- 172.21.15.43:0/1236744884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 msgr2=0x55b4136be870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:37.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.342+0000 7fd09ca00700 1 --2- 172.21.15.43:0/1236744884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136be870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.342+0000 7fd09ca00700 1 -- 172.21.15.43:0/1236744884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 msgr2=0x55b4136c47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:37.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.342+0000 7fd09ca00700 1 --2- 172.21.15.43:0/1236744884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136c47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.342+0000 7fd09ca00700 1 -- 172.21.15.43:0/1236744884 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4135a5420 con 0x55b4136bedb0 2024-05-27T07:40:37.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.343+0000 7fd08f400700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd07c0b29f0 0x7fd07c0b4f00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd08000b6c0 tx=0x7fd080012890 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:37.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.343+0000 7fd09ca00700 1 --2- 172.21.15.43:0/1236744884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136bf1c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd0900a46f0 tx=0x7fd090007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b89a4c3252ce22 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:37.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.343+0000 7fd08d600700 1 -- 172.21.15.43:0/1236744884 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900aca20 con 0x55b4136bedb0 2024-05-27T07:40:37.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.343+0000 7fd08d600700 1 -- 172.21.15.43:0/1236744884 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0900b14d0 con 0x55b4136bedb0 2024-05-27T07:40:37.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.343+0000 7fd08d600700 1 -- 172.21.15.43:0/1236744884 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900e15a0 con 0x55b4136bedb0 2024-05-27T07:40:37.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.344+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/1236744884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 msgr2=0x55b4136bf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:37.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.344+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/1236744884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136bf1c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd0900a46f0 tx=0x7fd090007f90 comp rx=0 tx=0).stop 2024-05-27T07:40:37.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.344+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/1236744884 shutdown_connections 2024-05-27T07:40:37.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.344+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/1236744884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136bf1c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.344+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/1236744884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136c47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.344+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/1236744884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136be870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.344+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/1236744884 >> 172.21.15.43:0/1236744884 conn(0x55b41359ffc0 msgr2=0x55b4136b98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:37.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.344+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/1236744884 shutdown_connections 2024-05-27T07:40:37.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.344+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/1236744884 wait complete. 2024-05-27T07:40:37.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.345+0000 7fd0ad52eb80 1 -- start start 2024-05-27T07:40:37.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.345+0000 7fd0ad52eb80 1 Processor -- start 2024-05-27T07:40:37.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136c8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b4134e2fa0 con 0x55b4136bedb0 2024-05-27T07:40:37.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b41357d760 con 0x55b41353fe40 2024-05-27T07:40:37.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b4136cb7a0 con 0x55b4136c4d30 2024-05-27T07:40:37.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd0ad52eb80 10 client.0 _ll_register_callbacks cb 0x55b413586c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:37.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd08f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd08f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58876/0 (socket says 172.21.15.43:58876) 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd08f400700 1 -- 172.21.15.43:0/3848878417 learned_addr learned my addr 172.21.15.43:0/3848878417 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd097e00700 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd09ca00700 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd08f400700 1 -- 172.21.15.43:0/3848878417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 msgr2=0x55b4136c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd08f400700 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd08f400700 1 -- 172.21.15.43:0/3848878417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 msgr2=0x55b4136c8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd08f400700 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136c8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.346+0000 7fd08f400700 1 -- 172.21.15.43:0/3848878417 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b41352c8d0 con 0x55b4136bedb0 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.347+0000 7fd08f400700 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c8aa0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fd08000f890 tx=0x7fd08001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd39ee91f7ed1d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.347+0000 7fd086000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.347+0000 7fd086000700 1 -- 172.21.15.43:0/3848878417 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080004a30 con 0x55b4136bedb0 2024-05-27T07:40:37.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.347+0000 7fd086000700 1 -- 172.21.15.43:0/3848878417 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7fd080004bd0 con 0x55b4136bedb0 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.347+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/3848878417 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0900b14d0 con 0x55b4136bedb0 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd086000700 1 -- 172.21.15.43:0/3848878417 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7fd08002f050 con 0x55b4136bedb0 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd0ad52eb80 10 client.18660 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/3848878417 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x55b4135ad060 con 0x55b4136bedb0 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd086000700 1 -- 172.21.15.43:0/3848878417 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd0900b14d0 con 0x55b4136bedb0 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd086000700 1 -- 172.21.15.43:0/3848878417 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7fd080005290 con 0x55b4136bedb0 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd0ad52eb80 10 client.18660 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd0ad52eb80 10 client.18660 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd0ad52eb80 20 client.18660 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b4136d8af0 0x55b4136dafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd0ad52eb80 4 client.18660 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:37.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.348+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/3848878417 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4135eed90 con 0x55b4136d8af0 2024-05-27T07:40:37.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.349+0000 7fd097e00700 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b4136d8af0 0x55b4136dafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.350+0000 7fd097e00700 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b4136d8af0 0x55b4136dafa0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb193a059dfef6e8 server_cookie=b5277c461204853e in_seq=0 out_seq=0 2024-05-27T07:40:37.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.350+0000 7fd086000700 10 client.18660 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:40:37.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.352+0000 7fd086000700 1 -- 172.21.15.43:0/3848878417 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55b4135eed90 con 0x55b4136d8af0 2024-05-27T07:40:37.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.352+0000 7fd086000700 10 client.18660 handle_command_reply: tid=0 2024-05-27T07:40:37.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 client.18660 shutdown 2024-05-27T07:40:37.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/3848878417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 msgr2=0x55b4136c8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:37.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c8aa0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fd08000f890 tx=0x7fd08001a000 comp rx=0 tx=0).stop 2024-05-27T07:40:37.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/3848878417 shutdown_connections 2024-05-27T07:40:37.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b4136d8af0 0x55b4136dafa0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136c8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/3848878417 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c8aa0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd086000700 0 client.18660 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:40:37.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/3848878417 >> 172.21.15.43:0/3848878417 conn(0x55b41359ffc0 msgr2=0x55b4136b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:37.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/3848878417 shutdown_connections 2024-05-27T07:40:37.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/3848878417 wait complete. 2024-05-27T07:40:37.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.353+0000 7fd0ad52eb80 20 client.18660 trim_cache size 0 max 16384 2024-05-27T07:40:37.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.370+0000 7fd0ad52eb80 1 Processor -- start 2024-05-27T07:40:37.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.370+0000 7fd0ad52eb80 1 -- start start 2024-05-27T07:40:37.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.371+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136be870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.371+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.371+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.371+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981b2d80 con 0x55b4136bedb0 2024-05-27T07:40:37.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.371+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981b1180 con 0x55b41353fe40 2024-05-27T07:40:37.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.371+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981b2680 con 0x55b4136c4d30 2024-05-27T07:40:37.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.371+0000 7fd08f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.371+0000 7fd08f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58892/0 (socket says 172.21.15.43:58892) 2024-05-27T07:40:37.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.371+0000 7fd08f400700 1 -- 172.21.15.43:0/2742571339 learned_addr learned my addr 172.21.15.43:0/2742571339 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:37.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.371+0000 7fd097e00700 1 --2- 172.21.15.43:0/2742571339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136be870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.372+0000 7fd09ca00700 1 --2- 172.21.15.43:0/2742571339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.372+0000 7fd08f400700 1 -- 172.21.15.43:0/2742571339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 msgr2=0x55b4136c3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:37.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.372+0000 7fd08f400700 1 --2- 172.21.15.43:0/2742571339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136c3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.372+0000 7fd08f400700 1 -- 172.21.15.43:0/2742571339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 msgr2=0x55b4136be870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:37.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.372+0000 7fd08f400700 1 --2- 172.21.15.43:0/2742571339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136be870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.372+0000 7fd08f400700 1 -- 172.21.15.43:0/2742571339 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4136c6ad0 con 0x55b4136bedb0 2024-05-27T07:40:37.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.372+0000 7fd08f400700 1 --2- 172.21.15.43:0/2742571339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c2c10 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fd08000f860 tx=0x7fd08000bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a80f65207547cb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:37.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.372+0000 7fd08cc00700 1 -- 172.21.15.43:0/2742571339 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0800051b0 con 0x55b4136bedb0 2024-05-27T07:40:37.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.372+0000 7fd08cc00700 1 -- 172.21.15.43:0/2742571339 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd080038070 con 0x55b4136bedb0 2024-05-27T07:40:37.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.372+0000 7fd08cc00700 1 -- 172.21.15.43:0/2742571339 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080004980 con 0x55b4136bedb0 2024-05-27T07:40:37.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.373+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2742571339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 msgr2=0x55b4136c2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:37.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.373+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/2742571339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c2c10 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fd08000f860 tx=0x7fd08000bd90 comp rx=0 tx=0).stop 2024-05-27T07:40:37.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.373+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2742571339 shutdown_connections 2024-05-27T07:40:37.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.373+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/2742571339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b4136c4d30 0x55b4136c3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.373+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/2742571339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b41353fe40 0x55b4136be870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.373+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/2742571339 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136c2c10 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.373+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2742571339 >> 172.21.15.43:0/2742571339 conn(0x55b41359ffc0 msgr2=0x55b4136c8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:37.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.373+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2742571339 shutdown_connections 2024-05-27T07:40:37.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.374+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2742571339 wait complete. 2024-05-27T07:40:37.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.374+0000 7fd0ad52eb80 1 -- start start 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.374+0000 7fd0ad52eb80 1 Processor -- start 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.375+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b41353fe40 0x55b4136d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.375+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136dfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.375+0000 7fd0ad52eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b4136c4d30 0x55b4136e03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.375+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981b1fb0 con 0x55b4136bedb0 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.375+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981b18b0 con 0x55b4136c4d30 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.375+0000 7fd0ad52eb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55b4136c8340 con 0x55b41353fe40 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.375+0000 7fd0ad52eb80 10 client.0 _ll_register_callbacks cb 0x55b413586c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.375+0000 7fd08f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136dfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.375+0000 7fd08f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136dfeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58904/0 (socket says 172.21.15.43:58904) 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.375+0000 7fd08f400700 1 -- 172.21.15.43:0/2058838632 learned_addr learned my addr 172.21.15.43:0/2058838632 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:37.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.376+0000 7fd097e00700 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b41353fe40 0x55b4136d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.376+0000 7fd09ca00700 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b4136c4d30 0x55b4136e03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.376+0000 7fd08f400700 1 -- 172.21.15.43:0/2058838632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b41353fe40 msgr2=0x55b4136d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:37.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.376+0000 7fd08f400700 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b41353fe40 0x55b4136d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.376+0000 7fd08f400700 1 -- 172.21.15.43:0/2058838632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b4136c4d30 msgr2=0x55b4136e03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:37.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.376+0000 7fd08f400700 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b4136c4d30 0x55b4136e03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.376+0000 7fd08f400700 1 -- 172.21.15.43:0/2058838632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4136e0a00 con 0x55b4136bedb0 2024-05-27T07:40:37.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.376+0000 7fd09ca00700 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b4136c4d30 0x55b4136e03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:37.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.376+0000 7fd08f400700 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136dfeb0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fd08000bb90 tx=0x7fd08001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ccc2ce1adfeb9a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:37.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.376+0000 7fd086000700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:37.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.377+0000 7fd086000700 1 -- 172.21.15.43:0/2058838632 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080033840 con 0x55b4136bedb0 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.377+0000 7fd086000700 1 -- 172.21.15.43:0/2058838632 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7fd0800339e0 con 0x55b4136bedb0 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.377+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2058838632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd080038070 con 0x55b4136bedb0 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.377+0000 7fd086000700 1 -- 172.21.15.43:0/2058838632 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7fd080033c20 con 0x55b4136bedb0 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.377+0000 7fd0ad52eb80 10 client.18666 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.377+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2058838632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x55b413743410 con 0x55b4136bedb0 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.377+0000 7fd086000700 1 -- 172.21.15.43:0/2058838632 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd080038070 con 0x55b4136bedb0 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.377+0000 7fd086000700 1 -- 172.21.15.43:0/2058838632 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7fd08003c420 con 0x55b4136bedb0 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.378+0000 7fd0ad52eb80 10 client.18666 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.378+0000 7fd0ad52eb80 10 client.18666 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.i 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.378+0000 7fd0ad52eb80 20 client.18666 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.378+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b413743a60 0x55b413745e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.378+0000 7fd0ad52eb80 4 client.18666 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:40:37.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.378+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2058838632 --> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b4135e8570 con 0x55b413743a60 2024-05-27T07:40:37.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.378+0000 7fd097e00700 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b413743a60 0x55b413745e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:37.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.379+0000 7fd097e00700 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b413743a60 0x55b413745e80 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e2591cbb8779fb03 server_cookie=b7c1a300b4656af8 in_seq=0 out_seq=0 2024-05-27T07:40:37.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.379+0000 7fd086000700 10 client.18666 ms_handle_connect on v2:172.21.15.179:6838/93338133 2024-05-27T07:40:37.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd086000700 1 -- 172.21.15.43:0/2058838632 <== mds.4 v2:172.21.15.179:6838/93338133 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b4135e8570 con 0x55b413743a60 2024-05-27T07:40:37.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd086000700 10 client.18666 handle_command_reply: tid=0 2024-05-27T07:40:37.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 client.18666 shutdown 2024-05-27T07:40:37.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2058838632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 msgr2=0x55b4136dfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:37.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136dfeb0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fd08000bb90 tx=0x7fd08001cbe0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2058838632 shutdown_connections 2024-05-27T07:40:37.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.179:6838/93338133,v1:172.21.15.179:6839/93338133] conn(0x55b413743a60 0x55b413745e80 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b4136c4d30 0x55b4136e03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b41353fe40 0x55b4136d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 --2- 172.21.15.43:0/2058838632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b4136bedb0 0x55b4136dfeb0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2058838632 >> 172.21.15.43:0/2058838632 conn(0x55b41359ffc0 msgr2=0x55b4136c2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:37.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd086000700 0 client.18666 ms_handle_reset on v2:172.21.15.179:6838/93338133 2024-05-27T07:40:37.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2058838632 shutdown_connections 2024-05-27T07:40:37.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 1 -- 172.21.15.43:0/2058838632 wait complete. 2024-05-27T07:40:37.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.380+0000 7fd0ad52eb80 20 client.18666 trim_cache size 0 max 16384 2024-05-27T07:40:37.381 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:40:37.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.381+0000 7fd09ea00700 1 -- 172.21.15.43:0/2151448795 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd07c0b29f0 msgr2=0x7fd07c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:37.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.381+0000 7fd09ea00700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd07c0b29f0 0x7fd07c0b4f00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd08000b6c0 tx=0x7fd080012890 comp rx=0 tx=0).stop 2024-05-27T07:40:37.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.381+0000 7fd09ea00700 1 -- 172.21.15.43:0/2151448795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd098109860 msgr2=0x7fd09819f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:37.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.381+0000 7fd09ea00700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd098109860 0x7fd09819f5d0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd090007ee0 tx=0x7fd09000d3b0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.381+0000 7fd09ea00700 1 -- 172.21.15.43:0/2151448795 shutdown_connections 2024-05-27T07:40:37.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.381+0000 7fd09ea00700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd07c0b29f0 0x7fd07c0b4f00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.381+0000 7fd09ea00700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd098109860 0x7fd09819f5d0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.381+0000 7fd09ea00700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0981045e0 0x7fd09819ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.381+0000 7fd09ea00700 1 --2- 172.21.15.43:0/2151448795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd098108f10 0x7fd09819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:37.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.381+0000 7fd09ea00700 1 -- 172.21.15.43:0/2151448795 >> 172.21.15.43:0/2151448795 conn(0x7fd098100030 msgr2=0x7fd098102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:37.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.382+0000 7fd09ea00700 1 -- 172.21.15.43:0/2151448795 shutdown_connections 2024-05-27T07:40:37.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:37.382+0000 7fd09ea00700 1 -- 172.21.15.43:0/2151448795 wait complete. 2024-05-27T07:40:37.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:37 smithi043 ceph-mon[31531]: pgmap v895: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s wr, 0 op/s 2024-05-27T07:40:37.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:37 smithi179 ceph-mon[35888]: pgmap v895: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s wr, 0 op/s 2024-05-27T07:40:37.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:37 smithi132 ceph-mon[35283]: pgmap v895: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s wr, 0 op/s 2024-05-27T07:40:38.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-05-27T07:40:38.871 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:39.697 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:39 smithi043 ceph-mon[31531]: pgmap v896: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 0 op/s 2024-05-27T07:40:39.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:39 smithi179 ceph-mon[35888]: pgmap v896: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 0 op/s 2024-05-27T07:40:39.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:39 smithi132 ceph-mon[35283]: pgmap v896: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 0 op/s 2024-05-27T07:40:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.381+0000 7f6f55a00700 1 -- 172.21.15.43:0/3606599474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f5010c5b0 msgr2=0x7f6f5010c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.381+0000 7f6f55a00700 1 --2- 172.21.15.43:0/3606599474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f5010c5b0 0x7f6f5010c9c0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009990 tx=0x7f6f40009ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.381+0000 7f6f55a00700 1 -- 172.21.15.43:0/3606599474 shutdown_connections 2024-05-27T07:40:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.381+0000 7f6f55a00700 1 --2- 172.21.15.43:0/3606599474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f50106ec0 0x7f6f50113350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.381+0000 7f6f55a00700 1 --2- 172.21.15.43:0/3606599474 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f5010c5b0 0x7f6f5010c9c0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.382+0000 7f6f55a00700 1 --2- 172.21.15.43:0/3606599474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f50106570 0x7f6f50106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.382+0000 7f6f55a00700 1 -- 172.21.15.43:0/3606599474 >> 172.21.15.43:0/3606599474 conn(0x7f6f50102070 msgr2=0x7f6f501044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.382+0000 7f6f55a00700 1 -- 172.21.15.43:0/3606599474 shutdown_connections 2024-05-27T07:40:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.382+0000 7f6f55a00700 1 -- 172.21.15.43:0/3606599474 wait complete. 2024-05-27T07:40:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.384+0000 7f6f55a00700 1 Processor -- start 2024-05-27T07:40:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.384+0000 7f6f55a00700 1 -- start start 2024-05-27T07:40:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.385+0000 7f6f55a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f50106570 0x7f6f5019d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.385+0000 7f6f55a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f50106ec0 0x7f6f5019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.385+0000 7f6f55a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f501a18b0 0x7f6f501a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.385+0000 7f6f55a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50116570 con 0x7f6f50106ec0 2024-05-27T07:40:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.385+0000 7f6f55a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501163f0 con 0x7f6f501a18b0 2024-05-27T07:40:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.385+0000 7f6f55a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501166f0 con 0x7f6f50106570 2024-05-27T07:40:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.385+0000 7f6f46000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f50106ec0 0x7f6f5019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.385+0000 7f6f46000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f50106ec0 0x7f6f5019d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58922/0 (socket says 172.21.15.43:58922) 2024-05-27T07:40:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.385+0000 7f6f46000700 1 -- 172.21.15.43:0/2820486553 learned_addr learned my addr 172.21.15.43:0/2820486553 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.385+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f50106570 0x7f6f5019d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.386+0000 7f6f46000700 1 -- 172.21.15.43:0/2820486553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f50106570 msgr2=0x7f6f5019d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.386+0000 7f6f46000700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f50106570 0x7f6f5019d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.386+0000 7f6f46000700 1 -- 172.21.15.43:0/2820486553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f501a18b0 msgr2=0x7f6f501a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.386+0000 7f6f46000700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f501a18b0 0x7f6f501a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.386+0000 7f6f46000700 1 -- 172.21.15.43:0/2820486553 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f501a2400 con 0x7f6f50106ec0 2024-05-27T07:40:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.386+0000 7f6f46000700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f50106ec0 0x7f6f5019d6e0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800e940 tx=0x7f6f3800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.386+0000 7f6f47e00700 1 -- 172.21.15.43:0/2820486553 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3800cbd0 con 0x7f6f50106ec0 2024-05-27T07:40:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.386+0000 7f6f47e00700 1 -- 172.21.15.43:0/2820486553 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f3800cd70 con 0x7f6f50106ec0 2024-05-27T07:40:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.386+0000 7f6f55a00700 1 -- 172.21.15.43:0/2820486553 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f501a26f0 con 0x7f6f50106ec0 2024-05-27T07:40:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.387+0000 7f6f47e00700 1 -- 172.21.15.43:0/2820486553 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f38018680 con 0x7f6f50106ec0 2024-05-27T07:40:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.387+0000 7f6f55a00700 1 -- 172.21.15.43:0/2820486553 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f501a9ed0 con 0x7f6f50106ec0 2024-05-27T07:40:41.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.391+0000 7f6f47e00700 1 -- 172.21.15.43:0/2820486553 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6f38010430 con 0x7f6f50106ec0 2024-05-27T07:40:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.392+0000 7f6f64638b80 1 Processor -- start 2024-05-27T07:40:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.392+0000 7f6f64638b80 1 -- start start 2024-05-27T07:40:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.393+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be528e9f0 0x560be5321330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.393+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be51a2e40 0x560be5321870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.393+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 0x560be532e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.393+0000 7f6f64638b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560be524fe40 con 0x560be51a2e40 2024-05-27T07:40:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.393+0000 7f6f64638b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560be523f9e0 con 0x560be528e9f0 2024-05-27T07:40:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.393+0000 7f6f64638b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560be5243a20 con 0x560be5321e70 2024-05-27T07:40:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.393+0000 7f6f4ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be51a2e40 0x560be5321870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.393+0000 7f6f4ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be51a2e40 0x560be5321870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58928/0 (socket says 172.21.15.43:58928) 2024-05-27T07:40:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.393+0000 7f6f4ea00700 1 -- 172.21.15.43:0/3039472799 learned_addr learned my addr 172.21.15.43:0/3039472799 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f47e00700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f340b29f0 0x7f6f340b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f4f400700 1 --2- 172.21.15.43:0/3039472799 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be528e9f0 0x560be5321330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f46000700 1 --2- 172.21.15.43:0/3039472799 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 0x560be532e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f4ea00700 1 -- 172.21.15.43:0/3039472799 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 msgr2=0x560be532e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/3039472799 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 0x560be532e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f4ea00700 1 -- 172.21.15.43:0/3039472799 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be528e9f0 msgr2=0x560be5321330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/3039472799 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be528e9f0 0x560be5321330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f4ea00700 1 -- 172.21.15.43:0/3039472799 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560be5208420 con 0x560be51a2e40 2024-05-27T07:40:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f46000700 1 --2- 172.21.15.43:0/3039472799 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 0x560be532e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f47e00700 1 -- 172.21.15.43:0/2820486553 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f6f380e2ce0 con 0x7f6f50106ec0 2024-05-27T07:40:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f4f400700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f340b29f0 0x7f6f340b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.394+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/3039472799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be51a2e40 0x560be5321870 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b650 tx=0x7f6f400096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c158d3e8d8209185 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.395+0000 7f6f3fe00700 1 -- 172.21.15.43:0/3039472799 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f40005550 con 0x560be51a2e40 2024-05-27T07:40:41.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.395+0000 7f6f3fe00700 1 -- 172.21.15.43:0/3039472799 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f40003b10 con 0x560be51a2e40 2024-05-27T07:40:41.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.395+0000 7f6f3fe00700 1 -- 172.21.15.43:0/3039472799 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4000bb40 con 0x560be51a2e40 2024-05-27T07:40:41.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.396+0000 7f6f64638b80 1 -- 172.21.15.43:0/3039472799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be51a2e40 msgr2=0x560be5321870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.396+0000 7f6f64638b80 1 --2- 172.21.15.43:0/3039472799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be51a2e40 0x560be5321870 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000b650 tx=0x7f6f400096a0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.396+0000 7f6f4f400700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f340b29f0 0x7f6f340b4f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x560be5323ff0 tx=0x7f6f4800b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.396+0000 7f6f64638b80 1 -- 172.21.15.43:0/3039472799 shutdown_connections 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.396+0000 7f6f64638b80 1 --2- 172.21.15.43:0/3039472799 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 0x560be532e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.396+0000 7f6f64638b80 1 --2- 172.21.15.43:0/3039472799 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be528e9f0 0x560be5321330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.396+0000 7f6f64638b80 1 --2- 172.21.15.43:0/3039472799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be51a2e40 0x560be5321870 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.396+0000 7f6f64638b80 1 -- 172.21.15.43:0/3039472799 >> 172.21.15.43:0/3039472799 conn(0x560be523c390 msgr2=0x560be531d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.396+0000 7f6f64638b80 1 -- 172.21.15.43:0/3039472799 shutdown_connections 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.396+0000 7f6f64638b80 1 -- 172.21.15.43:0/3039472799 wait complete. 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.397+0000 7f6f64638b80 1 -- start start 2024-05-27T07:40:41.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.397+0000 7f6f64638b80 1 Processor -- start 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be51a2e40 0x560be532b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be532baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be5321e70 0x560be532c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f64638b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560be5145fa0 con 0x560be528e9f0 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f64638b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560be51e0760 con 0x560be5321e70 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f64638b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560be532eb20 con 0x560be51a2e40 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f64638b80 10 client.0 _ll_register_callbacks cb 0x560be51e9c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f46000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be532baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f46000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be532baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58940/0 (socket says 172.21.15.43:58940) 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f46000700 1 -- 172.21.15.43:0/88169748 learned_addr learned my addr 172.21.15.43:0/88169748 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be51a2e40 0x560be532b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.398+0000 7f6f4f400700 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be5321e70 0x560be532c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.399+0000 7f6f46000700 1 -- 172.21.15.43:0/88169748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be51a2e40 msgr2=0x560be532b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.399+0000 7f6f46000700 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be51a2e40 0x560be532b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.399+0000 7f6f46000700 1 -- 172.21.15.43:0/88169748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be5321e70 msgr2=0x560be532c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.399+0000 7f6f46000700 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be5321e70 0x560be532c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.399+0000 7f6f46000700 1 -- 172.21.15.43:0/88169748 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560be518f8d0 con 0x560be528e9f0 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.399+0000 7f6f46000700 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be532baf0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f6f380a5c10 tx=0x7f6f380a93b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e4172a89cb259f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.399+0000 7f6f3cc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.399+0000 7f6f3cc00700 1 -- 172.21.15.43:0/88169748 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f380aa290 con 0x560be528e9f0 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.399+0000 7f6f3cc00700 1 -- 172.21.15.43:0/88169748 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f6f380aa000 con 0x560be528e9f0 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.399+0000 7f6f64638b80 1 -- 172.21.15.43:0/88169748 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f40003b10 con 0x560be528e9f0 2024-05-27T07:40:41.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f3cc00700 1 -- 172.21.15.43:0/88169748 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f6f380a9b90 con 0x560be528e9f0 2024-05-27T07:40:41.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f64638b80 10 client.18678 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:41.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f64638b80 1 -- 172.21.15.43:0/88169748 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x560be5210060 con 0x560be528e9f0 2024-05-27T07:40:41.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f3cc00700 1 -- 172.21.15.43:0/88169748 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f40003b10 con 0x560be528e9f0 2024-05-27T07:40:41.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f3cc00700 1 -- 172.21.15.43:0/88169748 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7f6f380b2660 con 0x560be528e9f0 2024-05-27T07:40:41.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f64638b80 10 client.18678 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:41.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f64638b80 10 client.18678 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:40:41.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f64638b80 20 client.18678 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:41.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f64638b80 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560be533bde0 0x560be533e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f64638b80 4 client.18678 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:41.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.400+0000 7f6f64638b80 1 -- 172.21.15.43:0/88169748 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560be5251d90 con 0x560be533bde0 2024-05-27T07:40:41.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.401+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560be533bde0 0x560be533e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.402+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560be533bde0 0x560be533e230 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bc122edeab407791 server_cookie=450806680f9a5f78 in_seq=0 out_seq=0 2024-05-27T07:40:41.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.402+0000 7f6f3cc00700 10 client.18678 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:40:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f3cc00700 1 -- 172.21.15.43:0/88169748 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x560be5251d90 con 0x560be533bde0 2024-05-27T07:40:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f3cc00700 10 client.18678 handle_command_reply: tid=0 2024-05-27T07:40:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 client.18678 shutdown 2024-05-27T07:40:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 -- 172.21.15.43:0/88169748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 msgr2=0x560be532baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be532baf0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f6f380a5c10 tx=0x7f6f380a93b0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 -- 172.21.15.43:0/88169748 shutdown_connections 2024-05-27T07:40:41.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560be533bde0 0x560be533e230 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be5321e70 0x560be532c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be51a2e40 0x560be532b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 --2- 172.21.15.43:0/88169748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be532baf0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f3cc00700 0 client.18678 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:40:41.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 -- 172.21.15.43:0/88169748 >> 172.21.15.43:0/88169748 conn(0x560be523c390 msgr2=0x560be5321310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:41.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 -- 172.21.15.43:0/88169748 shutdown_connections 2024-05-27T07:40:41.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 1 -- 172.21.15.43:0/88169748 wait complete. 2024-05-27T07:40:41.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.405+0000 7f6f64638b80 20 client.18678 trim_cache size 0 max 16384 2024-05-27T07:40:41.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.422+0000 7f6f64638b80 1 Processor -- start 2024-05-27T07:40:41.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.422+0000 7f6f64638b80 1 -- start start 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.422+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be51a2e40 0x560be5326e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.422+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be528e9f0 0x560be532a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.422+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be5321e70 0x560be531d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.422+0000 7f6f64638b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b3200 con 0x560be5321e70 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.422+0000 7f6f64638b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b1600 con 0x560be528e9f0 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.422+0000 7f6f64638b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b2b00 con 0x560be51a2e40 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.423+0000 7f6f4f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be5321e70 0x560be531d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.423+0000 7f6f4f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be5321e70 0x560be531d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58956/0 (socket says 172.21.15.43:58956) 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.423+0000 7f6f4f400700 1 -- 172.21.15.43:0/2195857229 learned_addr learned my addr 172.21.15.43:0/2195857229 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.423+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/2195857229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be51a2e40 0x560be5326e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.423+0000 7f6f46000700 1 --2- 172.21.15.43:0/2195857229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be528e9f0 0x560be532a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.423+0000 7f6f4f400700 1 -- 172.21.15.43:0/2195857229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be51a2e40 msgr2=0x560be5326e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.423+0000 7f6f4f400700 1 --2- 172.21.15.43:0/2195857229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be51a2e40 0x560be5326e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.423+0000 7f6f4f400700 1 -- 172.21.15.43:0/2195857229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be528e9f0 msgr2=0x560be532a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.423+0000 7f6f4f400700 1 --2- 172.21.15.43:0/2195857229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be528e9f0 0x560be532a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.423+0000 7f6f4f400700 1 -- 172.21.15.43:0/2195857229 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560be5319240 con 0x560be5321e70 2024-05-27T07:40:41.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.424+0000 7f6f4f400700 1 --2- 172.21.15.43:0/2195857229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be5321e70 0x560be531d5c0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6f480144f0 tx=0x7f6f48012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef255b843ea4c798 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:41.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.424+0000 7f6f3f400700 1 -- 172.21.15.43:0/2195857229 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48026070 con 0x560be5321e70 2024-05-27T07:40:41.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.424+0000 7f6f3f400700 1 -- 172.21.15.43:0/2195857229 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f4801b030 con 0x560be5321e70 2024-05-27T07:40:41.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.424+0000 7f6f3f400700 1 -- 172.21.15.43:0/2195857229 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48013810 con 0x560be5321e70 2024-05-27T07:40:41.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.425+0000 7f6f64638b80 1 -- 172.21.15.43:0/2195857229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be5321e70 msgr2=0x560be531d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:41.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.425+0000 7f6f64638b80 1 --2- 172.21.15.43:0/2195857229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be5321e70 0x560be531d5c0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6f480144f0 tx=0x7f6f48012000 comp rx=0 tx=0).stop 2024-05-27T07:40:41.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.425+0000 7f6f64638b80 1 -- 172.21.15.43:0/2195857229 shutdown_connections 2024-05-27T07:40:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.425+0000 7f6f64638b80 1 --2- 172.21.15.43:0/2195857229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be5321e70 0x560be531d5c0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.425+0000 7f6f64638b80 1 --2- 172.21.15.43:0/2195857229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be51a2e40 0x560be5326e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.425+0000 7f6f64638b80 1 --2- 172.21.15.43:0/2195857229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be528e9f0 0x560be532a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.425+0000 7f6f64638b80 1 -- 172.21.15.43:0/2195857229 >> 172.21.15.43:0/2195857229 conn(0x560be523c390 msgr2=0x560be532cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.425+0000 7f6f64638b80 1 -- 172.21.15.43:0/2195857229 shutdown_connections 2024-05-27T07:40:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.425+0000 7f6f64638b80 1 -- 172.21.15.43:0/2195857229 wait complete. 2024-05-27T07:40:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.426+0000 7f6f64638b80 1 -- start start 2024-05-27T07:40:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.426+0000 7f6f64638b80 1 Processor -- start 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.426+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be51a2e40 0x560be53259e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.426+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be5325f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.426+0000 7f6f64638b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 0x560be5332e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.426+0000 7f6f64638b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b2430 con 0x560be528e9f0 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.426+0000 7f6f64638b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b1d30 con 0x560be51a2e40 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.426+0000 7f6f64638b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x560be5319950 con 0x560be5321e70 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.426+0000 7f6f64638b80 10 client.0 _ll_register_callbacks cb 0x560be531c220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.427+0000 7f6f46000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be5325f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.427+0000 7f6f46000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be5325f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58960/0 (socket says 172.21.15.43:58960) 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.427+0000 7f6f46000700 1 -- 172.21.15.43:0/932788195 learned_addr learned my addr 172.21.15.43:0/932788195 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:41.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.427+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be51a2e40 0x560be53259e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.427+0000 7f6f4f400700 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 0x560be5332e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.427+0000 7f6f46000700 1 -- 172.21.15.43:0/932788195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 msgr2=0x560be5332e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.427+0000 7f6f46000700 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 0x560be5332e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.427+0000 7f6f46000700 1 -- 172.21.15.43:0/932788195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be51a2e40 msgr2=0x560be53259e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.427+0000 7f6f46000700 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be51a2e40 0x560be53259e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.427+0000 7f6f46000700 1 -- 172.21.15.43:0/932788195 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560be5333350 con 0x560be528e9f0 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.428+0000 7f6f46000700 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be5325f20 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f6f380a9650 tx=0x7f6f380097c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf3f02ba86dc54c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.428+0000 7f6f3cc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.428+0000 7f6f3cc00700 1 -- 172.21.15.43:0/932788195 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f380adf40 con 0x560be528e9f0 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.428+0000 7f6f64638b80 1 -- 172.21.15.43:0/932788195 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f4801b030 con 0x560be528e9f0 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.428+0000 7f6f3cc00700 1 -- 172.21.15.43:0/932788195 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f6f380b2560 con 0x560be528e9f0 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.428+0000 7f6f3cc00700 1 -- 172.21.15.43:0/932788195 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f6f380b29f0 con 0x560be528e9f0 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.428+0000 7f6f64638b80 10 client.18684 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.428+0000 7f6f64638b80 1 -- 172.21.15.43:0/932788195 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x560be5334530 con 0x560be528e9f0 2024-05-27T07:40:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.428+0000 7f6f3cc00700 1 -- 172.21.15.43:0/932788195 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f4801b030 con 0x560be528e9f0 2024-05-27T07:40:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.429+0000 7f6f3cc00700 1 -- 172.21.15.43:0/932788195 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7f6f380a8a60 con 0x560be528e9f0 2024-05-27T07:40:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.429+0000 7f6f64638b80 10 client.18684 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.429+0000 7f6f64638b80 10 client.18684 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-05-27T07:40:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.429+0000 7f6f64638b80 20 client.18684 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.429+0000 7f6f64638b80 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560be53a6a20 0x560be53a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.429+0000 7f6f64638b80 4 client.18684 mds_command: new command op to 34403 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:40:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.429+0000 7f6f64638b80 1 -- 172.21.15.43:0/932788195 --> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560be524b570 con 0x560be53a6a20 2024-05-27T07:40:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.429+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560be53a6a20 0x560be53a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.430+0000 7f6f4ea00700 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560be53a6a20 0x560be53a8e40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=954ac587eab415fe server_cookie=720bbb577843ab05 in_seq=0 out_seq=0 2024-05-27T07:40:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.430+0000 7f6f3cc00700 10 client.18684 ms_handle_connect on v2:172.21.15.132:6832/512644396 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f3cc00700 1 -- 172.21.15.43:0/932788195 <== mds.3 v2:172.21.15.132:6832/512644396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560be524b570 con 0x560be53a6a20 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f3cc00700 10 client.18684 handle_command_reply: tid=0 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 client.18684 shutdown 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 -- 172.21.15.43:0/932788195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 msgr2=0x560be5325f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be5325f20 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f6f380a9650 tx=0x7f6f380097c0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 -- 172.21.15.43:0/932788195 shutdown_connections 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560be5321e70 0x560be5332e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.132:6832/512644396,v1:172.21.15.132:6833/512644396] conn(0x560be53a6a20 0x560be53a8e40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560be51a2e40 0x560be53259e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 --2- 172.21.15.43:0/932788195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560be528e9f0 0x560be5325f20 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 -- 172.21.15.43:0/932788195 >> 172.21.15.43:0/932788195 conn(0x560be523c390 msgr2=0x560be532a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f3cc00700 0 client.18684 ms_handle_reset on v2:172.21.15.132:6832/512644396 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 -- 172.21.15.43:0/932788195 shutdown_connections 2024-05-27T07:40:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.431+0000 7f6f64638b80 1 -- 172.21.15.43:0/932788195 wait complete. 2024-05-27T07:40:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f64638b80 20 client.18684 trim_cache size 0 max 16384 2024-05-27T07:40:41.433 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:40:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f55a00700 1 -- 172.21.15.43:0/2820486553 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f340b29f0 msgr2=0x7f6f340b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f55a00700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f340b29f0 0x7f6f340b4f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x560be5323ff0 tx=0x7f6f4800b040 comp rx=0 tx=0).stop 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f55a00700 1 -- 172.21.15.43:0/2820486553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f50106ec0 msgr2=0x7f6f5019d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f55a00700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f50106ec0 0x7f6f5019d6e0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800e940 tx=0x7f6f3800ed00 comp rx=0 tx=0).stop 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f55a00700 1 -- 172.21.15.43:0/2820486553 shutdown_connections 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f55a00700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6f340b29f0 0x7f6f340b4f00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f55a00700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f501a18b0 0x7f6f501a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f55a00700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f50106570 0x7f6f5019d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f55a00700 1 --2- 172.21.15.43:0/2820486553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f50106ec0 0x7f6f5019d6e0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.432+0000 7f6f55a00700 1 -- 172.21.15.43:0/2820486553 >> 172.21.15.43:0/2820486553 conn(0x7f6f50102070 msgr2=0x7f6f50104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.433+0000 7f6f55a00700 1 -- 172.21.15.43:0/2820486553 shutdown_connections 2024-05-27T07:40:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:41.433+0000 7f6f55a00700 1 -- 172.21.15.43:0/2820486553 wait complete. 2024-05-27T07:40:41.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:41 smithi043 ceph-mon[31531]: pgmap v897: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 0 op/s 2024-05-27T07:40:41.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:41 smithi179 ceph-mon[35888]: pgmap v897: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 0 op/s 2024-05-27T07:40:41.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:41 smithi132 ceph-mon[35283]: pgmap v897: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 0 op/s 2024-05-27T07:40:42.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-05-27T07:40:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:43.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:43 smithi043 ceph-mon[31531]: pgmap v898: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 0 op/s 2024-05-27T07:40:43.715 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:43 smithi179 ceph-mon[35888]: pgmap v898: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 0 op/s 2024-05-27T07:40:43.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:43 smithi132 ceph-mon[35283]: pgmap v898: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s wr, 0 op/s 2024-05-27T07:40:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.105+0000 7f6e6e000700 1 -- 172.21.15.43:0/3735634315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e6810f6b0 msgr2=0x7f6e6810fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.105+0000 7f6e6e000700 1 --2- 172.21.15.43:0/3735634315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e6810f6b0 0x7f6e6810fac0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009990 tx=0x7f6e5c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.105+0000 7f6e6e000700 1 -- 172.21.15.43:0/3735634315 shutdown_connections 2024-05-27T07:40:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.105+0000 7f6e6e000700 1 --2- 172.21.15.43:0/3735634315 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e6810b100 0x7f6e68117610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.105+0000 7f6e6e000700 1 --2- 172.21.15.43:0/3735634315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e6810f6b0 0x7f6e6810fac0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.105+0000 7f6e6e000700 1 --2- 172.21.15.43:0/3735634315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e6810a7b0 0x7f6e6810abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.105+0000 7f6e6e000700 1 -- 172.21.15.43:0/3735634315 >> 172.21.15.43:0/3735634315 conn(0x7f6e68076af0 msgr2=0x7f6e68076f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.105+0000 7f6e6e000700 1 -- 172.21.15.43:0/3735634315 shutdown_connections 2024-05-27T07:40:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.105+0000 7f6e6e000700 1 -- 172.21.15.43:0/3735634315 wait complete. 2024-05-27T07:40:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.106+0000 7f6e6e000700 1 Processor -- start 2024-05-27T07:40:45.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.106+0000 7f6e6e000700 1 -- start start 2024-05-27T07:40:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.107+0000 7f6e6e000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e6810a7b0 0x7f6e681a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.107+0000 7f6e6e000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e6810b100 0x7f6e681a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.107+0000 7f6e6e000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e6810f6b0 0x7f6e681a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.107+0000 7f6e6e000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6811a740 con 0x7f6e6810a7b0 2024-05-27T07:40:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.107+0000 7f6e6e000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6811a5c0 con 0x7f6e6810b100 2024-05-27T07:40:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.107+0000 7f6e6e000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6811a8c0 con 0x7f6e6810f6b0 2024-05-27T07:40:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.107+0000 7f6e6cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e6810a7b0 0x7f6e681a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.107+0000 7f6e6cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e6810a7b0 0x7f6e681a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58766/0 (socket says 172.21.15.43:58766) 2024-05-27T07:40:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.107+0000 7f6e6cc00700 1 -- 172.21.15.43:0/2407615613 learned_addr learned my addr 172.21.15.43:0/2407615613 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.108+0000 7f6e67e00700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e6810b100 0x7f6e681a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.108+0000 7f6e6d600700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e6810f6b0 0x7f6e681a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.108+0000 7f6e6cc00700 1 -- 172.21.15.43:0/2407615613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e6810f6b0 msgr2=0x7f6e681a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.108+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e6810f6b0 0x7f6e681a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.108+0000 7f6e6cc00700 1 -- 172.21.15.43:0/2407615613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e6810b100 msgr2=0x7f6e681a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.108+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e6810b100 0x7f6e681a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.108+0000 7f6e6cc00700 1 -- 172.21.15.43:0/2407615613 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e681a5f90 con 0x7f6e6810a7b0 2024-05-27T07:40:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.108+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e6810a7b0 0x7f6e681a1170 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0057f0 tx=0x7f6e5c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.109+0000 7f6e65600700 1 -- 172.21.15.43:0/2407615613 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c027070 con 0x7f6e6810a7b0 2024-05-27T07:40:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.109+0000 7f6e65600700 1 -- 172.21.15.43:0/2407615613 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e5c00fd60 con 0x7f6e6810a7b0 2024-05-27T07:40:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.109+0000 7f6e65600700 1 -- 172.21.15.43:0/2407615613 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c022de0 con 0x7f6e6810a7b0 2024-05-27T07:40:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.109+0000 7f6e6e000700 1 -- 172.21.15.43:0/2407615613 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e681a6220 con 0x7f6e6810a7b0 2024-05-27T07:40:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.109+0000 7f6e6e000700 1 -- 172.21.15.43:0/2407615613 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e681a66b0 con 0x7f6e6810a7b0 2024-05-27T07:40:45.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.113+0000 7f6e65600700 1 -- 172.21.15.43:0/2407615613 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6e5c02a070 con 0x7f6e6810a7b0 2024-05-27T07:40:45.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.114+0000 7f6e7e05bb80 1 Processor -- start 2024-05-27T07:40:45.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.114+0000 7f6e7e05bb80 1 -- start start 2024-05-27T07:40:45.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.114+0000 7f6e65600700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6e500b2a70 0x7f6e500b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de46b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e65600700 1 -- 172.21.15.43:0/2407615613 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f6e5c0e3a50 con 0x7f6e6810a7b0 2024-05-27T07:40:45.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de46bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de4776e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c6de398e40 con 0x55c6de2ebe40 2024-05-27T07:40:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c6de3889e0 con 0x55c6de3d79f0 2024-05-27T07:40:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c6de38ca20 con 0x55c6de4752c0 2024-05-27T07:40:45.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e6d600700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6e500b2a70 0x7f6e500b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e67e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de46bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e67e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de46bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58770/0 (socket says 172.21.15.43:58770) 2024-05-27T07:40:45.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e67e00700 1 -- 172.21.15.43:0/4218559897 learned_addr learned my addr 172.21.15.43:0/4218559897 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:45.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.115+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/4218559897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de46b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.116+0000 7f6e6d600700 1 --2- 172.21.15.43:0/4218559897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de4776e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.116+0000 7f6e67e00700 1 -- 172.21.15.43:0/4218559897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 msgr2=0x55c6de4776e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:45.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.116+0000 7f6e67e00700 1 --2- 172.21.15.43:0/4218559897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de4776e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.116+0000 7f6e67e00700 1 -- 172.21.15.43:0/4218559897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 msgr2=0x55c6de46b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:45.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.116+0000 7f6e67e00700 1 --2- 172.21.15.43:0/4218559897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de46b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.116+0000 7f6e67e00700 1 -- 172.21.15.43:0/4218559897 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6de351420 con 0x55c6de2ebe40 2024-05-27T07:40:45.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.116+0000 7f6e6d600700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6e500b2a70 0x7f6e500b4f80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6e60006100 tx=0x7f6e60008780 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:45.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.116+0000 7f6e67e00700 1 --2- 172.21.15.43:0/4218559897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de46bc90 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f6e54008c30 tx=0x7f6e54008f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a319b1b94b69707 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:45.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.117+0000 7f6e5a000700 1 -- 172.21.15.43:0/4218559897 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e54014070 con 0x55c6de2ebe40 2024-05-27T07:40:45.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.117+0000 7f6e5a000700 1 -- 172.21.15.43:0/4218559897 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e5400ed70 con 0x55c6de2ebe40 2024-05-27T07:40:45.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.117+0000 7f6e5a000700 1 -- 172.21.15.43:0/4218559897 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e540056d0 con 0x55c6de2ebe40 2024-05-27T07:40:45.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.118+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/4218559897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 msgr2=0x55c6de46bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:45.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.118+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/4218559897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de46bc90 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f6e54008c30 tx=0x7f6e54008f40 comp rx=0 tx=0).stop 2024-05-27T07:40:45.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.118+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/4218559897 shutdown_connections 2024-05-27T07:40:45.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.118+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/4218559897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de4776e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.118+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/4218559897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de46b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.118+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/4218559897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de46bc90 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.118+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/4218559897 >> 172.21.15.43:0/4218559897 conn(0x55c6de385390 msgr2=0x55c6de466710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:45.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.118+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/4218559897 shutdown_connections 2024-05-27T07:40:45.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.118+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/4218559897 wait complete. 2024-05-27T07:40:45.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.119+0000 7f6e7e05bb80 1 -- start start 2024-05-27T07:40:45.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.119+0000 7f6e7e05bb80 1 Processor -- start 2024-05-27T07:40:45.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de4721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de472710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de472c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c6de28efa0 con 0x55c6de2ebe40 2024-05-27T07:40:45.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c6de329760 con 0x55c6de3d79f0 2024-05-27T07:40:45.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c6de477e00 con 0x55c6de4752c0 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e7e05bb80 10 client.0 _ll_register_callbacks cb 0x55c6de332c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e6cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de4721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e6cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de4721d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58780/0 (socket says 172.21.15.43:58780) 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e6cc00700 1 -- 172.21.15.43:0/1895141289 learned_addr learned my addr 172.21.15.43:0/1895141289 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e6d600700 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de472c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.120+0000 7f6e67e00700 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de472710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e6cc00700 1 -- 172.21.15.43:0/1895141289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 msgr2=0x55c6de472c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de472c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e6cc00700 1 -- 172.21.15.43:0/1895141289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 msgr2=0x55c6de472710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de472710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e6cc00700 1 -- 172.21.15.43:0/1895141289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6de2d88d0 con 0x55c6de2ebe40 2024-05-27T07:40:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e67e00700 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de472710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:45.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de4721d0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0057c0 tx=0x7f6e5c0b1390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fb0ca61a0b4858 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e4ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e4ea00700 1 -- 172.21.15.43:0/1895141289 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c0c4060 con 0x55c6de2ebe40 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e4ea00700 1 -- 172.21.15.43:0/1895141289 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f6e5c0aae00 con 0x55c6de2ebe40 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.121+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/1895141289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e5400ed70 con 0x55c6de2ebe40 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e4ea00700 1 -- 172.21.15.43:0/1895141289 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f6e5c0aa4a0 con 0x55c6de2ebe40 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e7e05bb80 10 client.18696 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/1895141289 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x55c6de359060 con 0x55c6de2ebe40 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e4ea00700 1 -- 172.21.15.43:0/1895141289 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e5400ed70 con 0x55c6de2ebe40 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e4ea00700 1 -- 172.21.15.43:0/1895141289 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7f6e5c0b32d0 con 0x55c6de2ebe40 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e7e05bb80 10 client.18696 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e7e05bb80 10 client.18696 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e7e05bb80 20 client.18696 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c6de484d70 0x55c6de487220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e7e05bb80 4 client.18696 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:45.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.122+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/1895141289 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6de39ad90 con 0x55c6de484d70 2024-05-27T07:40:45.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.123+0000 7f6e67e00700 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c6de484d70 0x55c6de487220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.124+0000 7f6e67e00700 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c6de484d70 0x55c6de487220 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c984e15dbfbf5cb9 server_cookie=6b5af33892c1ee32 in_seq=0 out_seq=0 2024-05-27T07:40:45.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.124+0000 7f6e4ea00700 10 client.18696 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:40:45.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e4ea00700 1 -- 172.21.15.43:0/1895141289 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c6de39ad90 con 0x55c6de484d70 2024-05-27T07:40:45.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e4ea00700 10 client.18696 handle_command_reply: tid=0 2024-05-27T07:40:45.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e7e05bb80 1 client.18696 shutdown 2024-05-27T07:40:45.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/1895141289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 msgr2=0x55c6de4721d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:45.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de4721d0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0057c0 tx=0x7f6e5c0b1390 comp rx=0 tx=0).stop 2024-05-27T07:40:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/1895141289 shutdown_connections 2024-05-27T07:40:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c6de484d70 0x55c6de487220 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de472c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de4721d0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/1895141289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de472710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.127+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/1895141289 >> 172.21.15.43:0/1895141289 conn(0x55c6de385390 msgr2=0x55c6de466260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.126+0000 7f6e4ea00700 0 client.18696 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:40:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.127+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/1895141289 shutdown_connections 2024-05-27T07:40:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.127+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/1895141289 wait complete. 2024-05-27T07:40:45.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.127+0000 7f6e7e05bb80 20 client.18696 trim_cache size 0 max 16384 2024-05-27T07:40:45.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.144+0000 7f6e7e05bb80 1 Processor -- start 2024-05-27T07:40:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.144+0000 7f6e7e05bb80 1 -- start start 2024-05-27T07:40:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de2ebe40 0x55c6de480ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de3d79f0 0x55c6de46b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de4752c0 0x55c6de46bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6e681b6dd0 con 0x55c6de4752c0 2024-05-27T07:40:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6e681b51d0 con 0x55c6de2ebe40 2024-05-27T07:40:45.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6e681b66d0 con 0x55c6de3d79f0 2024-05-27T07:40:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e6d600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de4752c0 0x55c6de46bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e6d600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de4752c0 0x55c6de46bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58796/0 (socket says 172.21.15.43:58796) 2024-05-27T07:40:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e6d600700 1 -- 172.21.15.43:0/4275035054 learned_addr learned my addr 172.21.15.43:0/4275035054 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e67e00700 1 --2- 172.21.15.43:0/4275035054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de3d79f0 0x55c6de46b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/4275035054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de2ebe40 0x55c6de480ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e6d600700 1 -- 172.21.15.43:0/4275035054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de3d79f0 msgr2=0x55c6de46b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e6d600700 1 --2- 172.21.15.43:0/4275035054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de3d79f0 0x55c6de46b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e6d600700 1 -- 172.21.15.43:0/4275035054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de2ebe40 msgr2=0x55c6de480ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.145+0000 7f6e6d600700 1 --2- 172.21.15.43:0/4275035054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de2ebe40 0x55c6de480ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.146+0000 7f6e6d600700 1 -- 172.21.15.43:0/4275035054 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6de472fc0 con 0x55c6de4752c0 2024-05-27T07:40:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.146+0000 7f6e6d600700 1 --2- 172.21.15.43:0/4275035054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de4752c0 0x55c6de46bc90 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f6e600196c0 tx=0x7f6e60013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce90dcc5d6dc879c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.146+0000 7f6e59600700 1 -- 172.21.15.43:0/4275035054 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e60028070 con 0x55c6de4752c0 2024-05-27T07:40:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.146+0000 7f6e59600700 1 -- 172.21.15.43:0/4275035054 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e600238b0 con 0x55c6de4752c0 2024-05-27T07:40:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.146+0000 7f6e59600700 1 -- 172.21.15.43:0/4275035054 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6001f040 con 0x55c6de4752c0 2024-05-27T07:40:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.147+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/4275035054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de4752c0 msgr2=0x55c6de46bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.147+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/4275035054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de4752c0 0x55c6de46bc90 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f6e600196c0 tx=0x7f6e60013000 comp rx=0 tx=0).stop 2024-05-27T07:40:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.147+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/4275035054 shutdown_connections 2024-05-27T07:40:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.147+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/4275035054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de4752c0 0x55c6de46bc90 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.147+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/4275035054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de2ebe40 0x55c6de480ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.147+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/4275035054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de3d79f0 0x55c6de46b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.147+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/4275035054 >> 172.21.15.43:0/4275035054 conn(0x55c6de385390 msgr2=0x55c6de470ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.148+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/4275035054 shutdown_connections 2024-05-27T07:40:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.148+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/4275035054 wait complete. 2024-05-27T07:40:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.148+0000 7f6e7e05bb80 1 -- start start 2024-05-27T07:40:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.148+0000 7f6e7e05bb80 1 Processor -- start 2024-05-27T07:40:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.149+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de478880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.149+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de478e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.149+0000 7f6e7e05bb80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de4e7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.149+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6e681b6000 con 0x55c6de2ebe40 2024-05-27T07:40:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.149+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6e681b5900 con 0x55c6de3d79f0 2024-05-27T07:40:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.149+0000 7f6e7e05bb80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c6de4e7de0 con 0x55c6de4752c0 2024-05-27T07:40:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.149+0000 7f6e7e05bb80 10 client.0 _ll_register_callbacks cb 0x55c6de332c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e6cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de478880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e6cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de478880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58808/0 (socket says 172.21.15.43:58808) 2024-05-27T07:40:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e6d600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de4e7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e6cc00700 1 -- 172.21.15.43:0/179302034 learned_addr learned my addr 172.21.15.43:0/179302034 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e67e00700 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de478e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e6cc00700 1 -- 172.21.15.43:0/179302034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 msgr2=0x55c6de4e7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de4e7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e6cc00700 1 -- 172.21.15.43:0/179302034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 msgr2=0x55c6de478e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de478e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e6cc00700 1 -- 172.21.15.43:0/179302034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6de4e7fe0 con 0x55c6de2ebe40 2024-05-27T07:40:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e67e00700 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de478e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e6cc00700 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de478880 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0aa640 tx=0x7f6e5c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0342bcf81ac151 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.150+0000 7f6e4ea00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.151+0000 7f6e4ea00700 1 -- 172.21.15.43:0/179302034 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c0a6850 con 0x55c6de2ebe40 2024-05-27T07:40:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.151+0000 7f6e4ea00700 1 -- 172.21.15.43:0/179302034 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f6e5c00fae0 con 0x55c6de2ebe40 2024-05-27T07:40:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.151+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/179302034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e600238b0 con 0x55c6de2ebe40 2024-05-27T07:40:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.151+0000 7f6e4ea00700 1 -- 172.21.15.43:0/179302034 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f6e5c0cbc50 con 0x55c6de2ebe40 2024-05-27T07:40:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.151+0000 7f6e7e05bb80 10 client.18702 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.151+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/179302034 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x55c6de4ef300 con 0x55c6de2ebe40 2024-05-27T07:40:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.151+0000 7f6e4ea00700 1 -- 172.21.15.43:0/179302034 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e600238b0 con 0x55c6de2ebe40 2024-05-27T07:40:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.151+0000 7f6e4ea00700 1 -- 172.21.15.43:0/179302034 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7f6e5c0a9bf0 con 0x55c6de2ebe40 2024-05-27T07:40:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.152+0000 7f6e7e05bb80 10 client.18702 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.152+0000 7f6e7e05bb80 10 client.18702 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-05-27T07:40:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.152+0000 7f6e7e05bb80 20 client.18702 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.152+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c6de4ef8d0 0x55c6de4f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.152+0000 7f6e7e05bb80 4 client.18702 mds_command: new command op to 34424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:40:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.152+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/179302034 --> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c6de394570 con 0x55c6de4ef8d0 2024-05-27T07:40:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.152+0000 7f6e67e00700 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c6de4ef8d0 0x55c6de4f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.153+0000 7f6e67e00700 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c6de4ef8d0 0x55c6de4f1cf0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4bcf3b06751cb7bd server_cookie=fb8bb4e033f4fe8e in_seq=0 out_seq=0 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.153+0000 7f6e4ea00700 10 client.18702 ms_handle_connect on v2:172.21.15.132:6834/1074601310 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e4ea00700 1 -- 172.21.15.43:0/179302034 <== mds.2 v2:172.21.15.132:6834/1074601310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c6de394570 con 0x55c6de4ef8d0 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e4ea00700 10 client.18702 handle_command_reply: tid=0 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 client.18702 shutdown 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/179302034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 msgr2=0x55c6de478880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de478880 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0aa640 tx=0x7f6e5c009ca0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/179302034 shutdown_connections 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.132:6834/1074601310,v1:172.21.15.132:6835/1074601310] conn(0x55c6de4ef8d0 0x55c6de4f1cf0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c6de4752c0 0x55c6de4e7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6de2ebe40 0x55c6de478880 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e4ea00700 0 client.18702 ms_handle_reset on v2:172.21.15.132:6834/1074601310 2024-05-27T07:40:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 --2- 172.21.15.43:0/179302034 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6de3d79f0 0x55c6de478e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/179302034 >> 172.21.15.43:0/179302034 conn(0x55c6de385390 msgr2=0x55c6de46b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/179302034 shutdown_connections 2024-05-27T07:40:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.154+0000 7f6e7e05bb80 1 -- 172.21.15.43:0/179302034 wait complete. 2024-05-27T07:40:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e7e05bb80 20 client.18702 trim_cache size 0 max 16384 2024-05-27T07:40:45.153 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:40:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e6e000700 1 -- 172.21.15.43:0/2407615613 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6e500b2a70 msgr2=0x7f6e500b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e6e000700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6e500b2a70 0x7f6e500b4f80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6e60006100 tx=0x7f6e60008780 comp rx=0 tx=0).stop 2024-05-27T07:40:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e6e000700 1 -- 172.21.15.43:0/2407615613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e6810a7b0 msgr2=0x7f6e681a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e6e000700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e6810a7b0 0x7f6e681a1170 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0057f0 tx=0x7f6e5c009f90 comp rx=0 tx=0).stop 2024-05-27T07:40:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e6e000700 1 -- 172.21.15.43:0/2407615613 shutdown_connections 2024-05-27T07:40:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e6e000700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f6e500b2a70 0x7f6e500b4f80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e6e000700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e6810f6b0 0x7f6e681a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e6e000700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e6810a7b0 0x7f6e681a1170 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e6e000700 1 --2- 172.21.15.43:0/2407615613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e6810b100 0x7f6e681a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.155+0000 7f6e6e000700 1 -- 172.21.15.43:0/2407615613 >> 172.21.15.43:0/2407615613 conn(0x7f6e68076af0 msgr2=0x7f6e68108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.156+0000 7f6e6e000700 1 -- 172.21.15.43:0/2407615613 shutdown_connections 2024-05-27T07:40:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:45.156+0000 7f6e6e000700 1 -- 172.21.15.43:0/2407615613 wait complete. 2024-05-27T07:40:45.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:45 smithi132 ceph-mon[35283]: pgmap v899: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 0 op/s 2024-05-27T07:40:45.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:45 smithi043 ceph-mon[31531]: pgmap v899: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 0 op/s 2024-05-27T07:40:45.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:45 smithi179 ceph-mon[35888]: pgmap v899: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 0 op/s 2024-05-27T07:40:46.004 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-05-27T07:40:46.478 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:47.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:47 smithi043 ceph-mon[31531]: pgmap v900: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 0 op/s 2024-05-27T07:40:47.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:47 smithi179 ceph-mon[35888]: pgmap v900: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 0 op/s 2024-05-27T07:40:48.099 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:47 smithi132 ceph-mon[35283]: pgmap v900: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 0 op/s 2024-05-27T07:40:48.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.912+0000 7f6069c00700 1 -- 172.21.15.43:0/4037132084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6064108ef0 msgr2=0x7f6064109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.912+0000 7f6069c00700 1 --2- 172.21.15.43:0/4037132084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6064108ef0 0x7f6064109300 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f60500099c0 tx=0x7f6050009cd0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.912+0000 7f6069c00700 1 -- 172.21.15.43:0/4037132084 shutdown_connections 2024-05-27T07:40:48.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.912+0000 7f6069c00700 1 --2- 172.21.15.43:0/4037132084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6064109840 0x7f60641111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.912+0000 7f6069c00700 1 --2- 172.21.15.43:0/4037132084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60641045c0 0x7f60641049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.912+0000 7f6069c00700 1 --2- 172.21.15.43:0/4037132084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6064108ef0 0x7f6064109300 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.912+0000 7f6069c00700 1 -- 172.21.15.43:0/4037132084 >> 172.21.15.43:0/4037132084 conn(0x7f60640ffe90 msgr2=0x7f60641022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.912+0000 7f6069c00700 1 -- 172.21.15.43:0/4037132084 shutdown_connections 2024-05-27T07:40:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.912+0000 7f6069c00700 1 -- 172.21.15.43:0/4037132084 wait complete. 2024-05-27T07:40:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.913+0000 7f6069c00700 1 Processor -- start 2024-05-27T07:40:48.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.914+0000 7f6069c00700 1 -- start start 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.914+0000 7f6069c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60641045c0 0x7f606419ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.914+0000 7f6069c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6064108ef0 0x7f606419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.914+0000 7f6069c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6064109840 0x7f606419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.915+0000 7f6069c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6064114350 con 0x7f60641045c0 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.915+0000 7f6069c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f60641141d0 con 0x7f6064109840 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.915+0000 7f6069c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f60641144d0 con 0x7f6064108ef0 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.915+0000 7f6062a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60641045c0 0x7f606419ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.915+0000 7f6062a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60641045c0 0x7f606419ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58832/0 (socket says 172.21.15.43:58832) 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.915+0000 7f6062a00700 1 -- 172.21.15.43:0/3738009077 learned_addr learned my addr 172.21.15.43:0/3738009077 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.915+0000 7f6062000700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6064108ef0 0x7f606419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.915+0000 7f6063400700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6064109840 0x7f606419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.916+0000 7f6062a00700 1 -- 172.21.15.43:0/3738009077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6064108ef0 msgr2=0x7f606419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.916+0000 7f6062a00700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6064108ef0 0x7f606419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.916+0000 7f6062a00700 1 -- 172.21.15.43:0/3738009077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6064109840 msgr2=0x7f606419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.916+0000 7f6062a00700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6064109840 0x7f606419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.916+0000 7f6062a00700 1 -- 172.21.15.43:0/3738009077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606419fc50 con 0x7f60641045c0 2024-05-27T07:40:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.916+0000 7f6062a00700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60641045c0 0x7f606419ad80 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f605400d950 tx=0x7f605400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.916+0000 7f605b400700 1 -- 172.21.15.43:0/3738009077 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6054016070 con 0x7f60641045c0 2024-05-27T07:40:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.916+0000 7f605b400700 1 -- 172.21.15.43:0/3738009077 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6054009dd0 con 0x7f60641045c0 2024-05-27T07:40:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.916+0000 7f6069c00700 1 -- 172.21.15.43:0/3738009077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f606419ff40 con 0x7f60641045c0 2024-05-27T07:40:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.917+0000 7f605b400700 1 -- 172.21.15.43:0/3738009077 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605400f3c0 con 0x7f60641045c0 2024-05-27T07:40:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.917+0000 7f6069c00700 1 -- 172.21.15.43:0/3738009077 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60641a7860 con 0x7f60641045c0 2024-05-27T07:40:48.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.921+0000 7f605b400700 1 -- 172.21.15.43:0/3738009077 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f6054019070 con 0x7f60641045c0 2024-05-27T07:40:48.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.922+0000 7f60787c7b80 1 Processor -- start 2024-05-27T07:40:48.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.922+0000 7f60787c7b80 1 -- start start 2024-05-27T07:40:48.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.922+0000 7f605b400700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f60480b29f0 0x7f60480b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.922+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd15937f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.922+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd158d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.922+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd158e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.922+0000 7f60787c7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564fd14bbe40 con 0x564fd158ddb0 2024-05-27T07:40:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.922+0000 7f60787c7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564fd14ab9e0 con 0x564fd140ee40 2024-05-27T07:40:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.922+0000 7f60787c7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564fd14afa20 con 0x564fd1593d30 2024-05-27T07:40:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.922+0000 7f605b400700 1 -- 172.21.15.43:0/3738009077 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f60540e26c0 con 0x7f60641045c0 2024-05-27T07:40:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6063400700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f60480b29f0 0x7f60480b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6063400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd158e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6063400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd158e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58840/0 (socket says 172.21.15.43:58840) 2024-05-27T07:40:48.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6062a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd15937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6063400700 1 -- 172.21.15.43:0/708080252 learned_addr learned my addr 172.21.15.43:0/708080252 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6062000700 1 --2- 172.21.15.43:0/708080252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd158d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6063400700 1 -- 172.21.15.43:0/708080252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 msgr2=0x564fd158d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6063400700 1 --2- 172.21.15.43:0/708080252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd158d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6063400700 1 -- 172.21.15.43:0/708080252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 msgr2=0x564fd15937f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6063400700 1 --2- 172.21.15.43:0/708080252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd15937f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.923+0000 7f6063400700 1 -- 172.21.15.43:0/708080252 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fd1474420 con 0x564fd158ddb0 2024-05-27T07:40:48.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.924+0000 7f6063400700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f60480b29f0 0x7f60480b4f00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f605c012e70 tx=0x7f605c0124f0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.924+0000 7f6063400700 1 --2- 172.21.15.43:0/708080252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd158e1c0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f605c019dc0 tx=0x7f605c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad14c235a779084 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.924+0000 7f604fe00700 1 -- 172.21.15.43:0/708080252 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c029070 con 0x564fd158ddb0 2024-05-27T07:40:48.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.924+0000 7f604fe00700 1 -- 172.21.15.43:0/708080252 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f605c0122a0 con 0x564fd158ddb0 2024-05-27T07:40:48.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.924+0000 7f604fe00700 1 -- 172.21.15.43:0/708080252 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c007a90 con 0x564fd158ddb0 2024-05-27T07:40:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.925+0000 7f60787c7b80 1 -- 172.21.15.43:0/708080252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 msgr2=0x564fd158e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.925+0000 7f60787c7b80 1 --2- 172.21.15.43:0/708080252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd158e1c0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f605c019dc0 tx=0x7f605c01c000 comp rx=0 tx=0).stop 2024-05-27T07:40:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.925+0000 7f60787c7b80 1 -- 172.21.15.43:0/708080252 shutdown_connections 2024-05-27T07:40:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.925+0000 7f60787c7b80 1 --2- 172.21.15.43:0/708080252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd158e1c0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.925+0000 7f60787c7b80 1 --2- 172.21.15.43:0/708080252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd15937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.925+0000 7f60787c7b80 1 --2- 172.21.15.43:0/708080252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd158d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.925+0000 7f60787c7b80 1 -- 172.21.15.43:0/708080252 >> 172.21.15.43:0/708080252 conn(0x564fd146efc0 msgr2=0x564fd15888d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.926+0000 7f60787c7b80 1 -- 172.21.15.43:0/708080252 shutdown_connections 2024-05-27T07:40:48.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.926+0000 7f60787c7b80 1 -- 172.21.15.43:0/708080252 wait complete. 2024-05-27T07:40:48.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.926+0000 7f60787c7b80 1 -- start start 2024-05-27T07:40:48.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.926+0000 7f60787c7b80 1 Processor -- start 2024-05-27T07:40:48.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.927+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd140ee40 0x564fd15974e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.927+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd158ddb0 0x564fd1597a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.927+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd1597f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.927+0000 7f60787c7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564fd13b1fa0 con 0x564fd140ee40 2024-05-27T07:40:48.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.927+0000 7f60787c7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564fd144c760 con 0x564fd158ddb0 2024-05-27T07:40:48.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.927+0000 7f60787c7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564fd159ebb0 con 0x564fd1593d30 2024-05-27T07:40:48.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.927+0000 7f60787c7b80 10 client.0 _ll_register_callbacks cb 0x564fd1455c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:48.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.927+0000 7f6062a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd140ee40 0x564fd15974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.927+0000 7f6062a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd140ee40 0x564fd15974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58846/0 (socket says 172.21.15.43:58846) 2024-05-27T07:40:48.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.927+0000 7f6062a00700 1 -- 172.21.15.43:0/1472281190 learned_addr learned my addr 172.21.15.43:0/1472281190 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:48.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.928+0000 7f6062000700 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd158ddb0 0x564fd1597a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.928+0000 7f6063400700 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd1597f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.928+0000 7f6062a00700 1 -- 172.21.15.43:0/1472281190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 msgr2=0x564fd1597f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.928+0000 7f6062a00700 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd1597f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.928+0000 7f6062a00700 1 -- 172.21.15.43:0/1472281190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd158ddb0 msgr2=0x564fd1597a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.928+0000 7f6062a00700 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd158ddb0 0x564fd1597a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.928+0000 7f6062a00700 1 -- 172.21.15.43:0/1472281190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fd13fb8d0 con 0x564fd140ee40 2024-05-27T07:40:48.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.928+0000 7f6062a00700 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd140ee40 0x564fd15974e0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f605400b2f0 tx=0x7f60540108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489a9f6c1c48b0f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.928+0000 7f604cc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:48.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.928+0000 7f604cc00700 1 -- 172.21.15.43:0/1472281190 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6054020070 con 0x564fd140ee40 2024-05-27T07:40:48.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f604cc00700 1 -- 172.21.15.43:0/1472281190 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f60540a5b60 con 0x564fd140ee40 2024-05-27T07:40:48.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f60787c7b80 1 -- 172.21.15.43:0/1472281190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564fd14bdd90 con 0x564fd140ee40 2024-05-27T07:40:48.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f604cc00700 1 -- 172.21.15.43:0/1472281190 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f60540c3050 con 0x564fd140ee40 2024-05-27T07:40:48.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f60787c7b80 10 client.18714 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:48.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f60787c7b80 1 -- 172.21.15.43:0/1472281190 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x564fd147c060 con 0x564fd140ee40 2024-05-27T07:40:48.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f604cc00700 1 -- 172.21.15.43:0/1472281190 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x564fd14bdd90 con 0x564fd140ee40 2024-05-27T07:40:48.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f604cc00700 1 -- 172.21.15.43:0/1472281190 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7f60540e2270 con 0x564fd140ee40 2024-05-27T07:40:48.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f60787c7b80 10 client.18714 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:48.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f60787c7b80 10 client.18714 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:40:48.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f60787c7b80 20 client.18714 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:48.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f60787c7b80 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564fd15abe70 0x564fd15ae320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f60787c7b80 4 client.18714 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:48.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.929+0000 7f60787c7b80 1 -- 172.21.15.43:0/1472281190 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fd14b7570 con 0x564fd15abe70 2024-05-27T07:40:48.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.930+0000 7f6062000700 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564fd15abe70 0x564fd15ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.931+0000 7f6062000700 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564fd15abe70 0x564fd15ae320 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=58318489d5244ef4 server_cookie=89068d52f638ab17 in_seq=0 out_seq=0 2024-05-27T07:40:48.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.931+0000 7f604cc00700 10 client.18714 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:40:48.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.933+0000 7f604cc00700 1 -- 172.21.15.43:0/1472281190 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x564fd14b7570 con 0x564fd15abe70 2024-05-27T07:40:48.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.933+0000 7f604cc00700 10 client.18714 handle_command_reply: tid=0 2024-05-27T07:40:48.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.933+0000 7f60787c7b80 1 client.18714 shutdown 2024-05-27T07:40:48.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.933+0000 7f60787c7b80 1 -- 172.21.15.43:0/1472281190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd140ee40 msgr2=0x564fd15974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.933+0000 7f60787c7b80 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd140ee40 0x564fd15974e0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f605400b2f0 tx=0x7f60540108d0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.934+0000 7f60787c7b80 1 -- 172.21.15.43:0/1472281190 shutdown_connections 2024-05-27T07:40:48.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.934+0000 7f60787c7b80 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564fd15abe70 0x564fd15ae320 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.934+0000 7f60787c7b80 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd1597f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.934+0000 7f60787c7b80 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd140ee40 0x564fd15974e0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.934+0000 7f60787c7b80 1 --2- 172.21.15.43:0/1472281190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd158ddb0 0x564fd1597a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.934+0000 7f60787c7b80 1 -- 172.21.15.43:0/1472281190 >> 172.21.15.43:0/1472281190 conn(0x564fd146efc0 msgr2=0x564fd1588580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.934+0000 7f604cc00700 0 client.18714 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:40:48.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.934+0000 7f60787c7b80 1 -- 172.21.15.43:0/1472281190 shutdown_connections 2024-05-27T07:40:48.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.934+0000 7f60787c7b80 1 -- 172.21.15.43:0/1472281190 wait complete. 2024-05-27T07:40:48.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.934+0000 7f60787c7b80 20 client.18714 trim_cache size 0 max 16384 2024-05-27T07:40:48.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.951+0000 7f60787c7b80 1 Processor -- start 2024-05-27T07:40:48.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.951+0000 7f60787c7b80 1 -- start start 2024-05-27T07:40:48.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd158d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd1584980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd1599980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f60787c7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564fd15ae900 con 0x564fd158ddb0 2024-05-27T07:40:48.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f60787c7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f60641ae7f0 con 0x564fd140ee40 2024-05-27T07:40:48.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f60787c7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f60641acbf0 con 0x564fd1593d30 2024-05-27T07:40:48.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f6062000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd1584980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f6062000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd1584980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58856/0 (socket says 172.21.15.43:58856) 2024-05-27T07:40:48.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f6062000700 1 -- 172.21.15.43:0/378025016 learned_addr learned my addr 172.21.15.43:0/378025016 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:48.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f6062000700 1 -- 172.21.15.43:0/378025016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 msgr2=0x564fd1599980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f6062000700 1 --2- 172.21.15.43:0/378025016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd1599980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f6062000700 1 -- 172.21.15.43:0/378025016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 msgr2=0x564fd158d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f6062000700 1 --2- 172.21.15.43:0/378025016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd158d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.952+0000 7f6062000700 1 -- 172.21.15.43:0/378025016 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fd15982d0 con 0x564fd158ddb0 2024-05-27T07:40:48.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.953+0000 7f6062000700 1 --2- 172.21.15.43:0/378025016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd1584980 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f6050003760 tx=0x7f6050005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d026536847c5fbb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.953+0000 7f604f400700 1 -- 172.21.15.43:0/378025016 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60500054f0 con 0x564fd158ddb0 2024-05-27T07:40:48.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.953+0000 7f604f400700 1 -- 172.21.15.43:0/378025016 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6050005690 con 0x564fd158ddb0 2024-05-27T07:40:48.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.953+0000 7f604f400700 1 -- 172.21.15.43:0/378025016 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605000bc40 con 0x564fd158ddb0 2024-05-27T07:40:48.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.954+0000 7f60787c7b80 1 -- 172.21.15.43:0/378025016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 msgr2=0x564fd1584980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.954+0000 7f60787c7b80 1 --2- 172.21.15.43:0/378025016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd1584980 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f6050003760 tx=0x7f6050005040 comp rx=0 tx=0).stop 2024-05-27T07:40:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.954+0000 7f60787c7b80 1 -- 172.21.15.43:0/378025016 shutdown_connections 2024-05-27T07:40:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.954+0000 7f60787c7b80 1 --2- 172.21.15.43:0/378025016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd1599980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.954+0000 7f60787c7b80 1 --2- 172.21.15.43:0/378025016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd158d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.954+0000 7f60787c7b80 1 --2- 172.21.15.43:0/378025016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd1584980 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.954+0000 7f60787c7b80 1 -- 172.21.15.43:0/378025016 >> 172.21.15.43:0/378025016 conn(0x564fd146efc0 msgr2=0x564fd15925f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.954+0000 7f60787c7b80 1 -- 172.21.15.43:0/378025016 shutdown_connections 2024-05-27T07:40:48.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.954+0000 7f60787c7b80 1 -- 172.21.15.43:0/378025016 wait complete. 2024-05-27T07:40:48.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.955+0000 7f60787c7b80 1 -- start start 2024-05-27T07:40:48.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.955+0000 7f60787c7b80 1 Processor -- start 2024-05-27T07:40:48.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd15b4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd15a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f60787c7b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd15a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f60787c7b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f60641ad320 con 0x564fd158ddb0 2024-05-27T07:40:48.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f60787c7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f60641ae0f0 con 0x564fd140ee40 2024-05-27T07:40:48.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f60787c7b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f60641ada20 con 0x564fd1593d30 2024-05-27T07:40:48.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f60787c7b80 10 client.0 _ll_register_callbacks cb 0x564fd1455c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:48.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f6062000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd15a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f6062000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd15a3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58868/0 (socket says 172.21.15.43:58868) 2024-05-27T07:40:48.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f6062000700 1 -- 172.21.15.43:0/2554632384 learned_addr learned my addr 172.21.15.43:0/2554632384 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:48.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f6062a00700 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd15b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f6062000700 1 -- 172.21.15.43:0/2554632384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 msgr2=0x564fd15a43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f6062000700 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd15a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f6062000700 1 -- 172.21.15.43:0/2554632384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 msgr2=0x564fd15b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:48.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f6062000700 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd15b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.956+0000 7f6062000700 1 -- 172.21.15.43:0/2554632384 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fd15b2ad0 con 0x564fd158ddb0 2024-05-27T07:40:48.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.957+0000 7f6062000700 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd15a3e40 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f6050005920 tx=0x7f6050003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d20f45bc8931cb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:48.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.957+0000 7f604cc00700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:48.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.957+0000 7f604cc00700 1 -- 172.21.15.43:0/2554632384 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605000b6b0 con 0x564fd158ddb0 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.957+0000 7f60787c7b80 1 -- 172.21.15.43:0/2554632384 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6050005690 con 0x564fd158ddb0 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.957+0000 7f604cc00700 1 -- 172.21.15.43:0/2554632384 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f605000b850 con 0x564fd158ddb0 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.957+0000 7f604cc00700 1 -- 172.21.15.43:0/2554632384 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f6050042050 con 0x564fd158ddb0 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.957+0000 7f60787c7b80 10 client.18720 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.958+0000 7f60787c7b80 1 -- 172.21.15.43:0/2554632384 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x564fd1615b90 con 0x564fd158ddb0 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.958+0000 7f604cc00700 1 -- 172.21.15.43:0/2554632384 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6050005690 con 0x564fd158ddb0 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.958+0000 7f604cc00700 1 -- 172.21.15.43:0/2554632384 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7f605000f970 con 0x564fd158ddb0 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.958+0000 7f60787c7b80 10 client.18720 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.958+0000 7f60787c7b80 10 client.18720 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.958+0000 7f60787c7b80 20 client.18720 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.958+0000 7f60787c7b80 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564fd1616110 0x564fd1618530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.958+0000 7f60787c7b80 4 client.18720 mds_command: new command op to 34448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.958+0000 7f60787c7b80 1 -- 172.21.15.43:0/2554632384 --> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564fd132e250 con 0x564fd1616110 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.959+0000 7f6062a00700 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564fd1616110 0x564fd1618530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.960+0000 7f6062a00700 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564fd1616110 0x564fd1618530 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d19c5ea99bbdd07 server_cookie=ade2f6b5e7dd322a in_seq=0 out_seq=0 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.960+0000 7f604cc00700 10 client.18720 ms_handle_connect on v2:172.21.15.132:6836/2777087908 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.960+0000 7f604cc00700 1 -- 172.21.15.43:0/2554632384 <== mds.1 v2:172.21.15.132:6836/2777087908 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564fd132e250 con 0x564fd1616110 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.960+0000 7f604cc00700 10 client.18720 handle_command_reply: tid=0 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.960+0000 7f60787c7b80 1 client.18720 shutdown 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 1 -- 172.21.15.43:0/2554632384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 msgr2=0x564fd15a3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd15a3e40 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f6050005920 tx=0x7f6050003aa0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 1 -- 172.21.15.43:0/2554632384 shutdown_connections 2024-05-27T07:40:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.132:6836/2777087908,v1:172.21.15.132:6837/2777087908] conn(0x564fd1616110 0x564fd1618530 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fd1593d30 0x564fd15a43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564fd140ee40 0x564fd15b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 1 --2- 172.21.15.43:0/2554632384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564fd158ddb0 0x564fd15a3e40 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 1 -- 172.21.15.43:0/2554632384 >> 172.21.15.43:0/2554632384 conn(0x564fd146efc0 msgr2=0x564fd15919a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f604cc00700 0 client.18720 ms_handle_reset on v2:172.21.15.132:6836/2777087908 2024-05-27T07:40:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 1 -- 172.21.15.43:0/2554632384 shutdown_connections 2024-05-27T07:40:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 1 -- 172.21.15.43:0/2554632384 wait complete. 2024-05-27T07:40:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.961+0000 7f60787c7b80 20 client.18720 trim_cache size 0 max 16384 2024-05-27T07:40:48.969 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:40:48.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.962+0000 7f6069c00700 1 -- 172.21.15.43:0/3738009077 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f60480b29f0 msgr2=0x7f60480b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.962+0000 7f6069c00700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f60480b29f0 0x7f60480b4f00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f605c012e70 tx=0x7f605c0124f0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.962+0000 7f6069c00700 1 -- 172.21.15.43:0/3738009077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60641045c0 msgr2=0x7f606419ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.962+0000 7f6069c00700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60641045c0 0x7f606419ad80 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f605400d950 tx=0x7f605400dc60 comp rx=0 tx=0).stop 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.962+0000 7f6069c00700 1 -- 172.21.15.43:0/3738009077 shutdown_connections 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.962+0000 7f6069c00700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f60480b29f0 0x7f60480b4f00 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.962+0000 7f6069c00700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6064109840 0x7f606419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.962+0000 7f6069c00700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60641045c0 0x7f606419ad80 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.962+0000 7f6069c00700 1 --2- 172.21.15.43:0/3738009077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6064108ef0 0x7f606419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.962+0000 7f6069c00700 1 -- 172.21.15.43:0/3738009077 >> 172.21.15.43:0/3738009077 conn(0x7f60640ffe90 msgr2=0x7f60641022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.963+0000 7f6069c00700 1 -- 172.21.15.43:0/3738009077 shutdown_connections 2024-05-27T07:40:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:48.963+0000 7f6069c00700 1 -- 172.21.15.43:0/3738009077 wait complete. 2024-05-27T07:40:49.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:49 smithi043 ceph-mon[31531]: pgmap v901: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:49.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-05-27T07:40:49.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:49 smithi179 ceph-mon[35888]: pgmap v901: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:50.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:49 smithi132 ceph-mon[35283]: pgmap v901: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:50.288 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:51.959 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:51 smithi043 ceph-mon[31531]: pgmap v902: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:51.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:51 smithi179 ceph-mon[35888]: pgmap v902: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:52.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:51 smithi132 ceph-mon[35283]: pgmap v902: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:52.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.692+0000 7f06a6000700 1 -- 172.21.15.43:0/2609336122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06a0108f10 msgr2=0x7f06a0109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:52.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.692+0000 7f06a6000700 1 --2- 172.21.15.43:0/2609336122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06a0108f10 0x7f06a0109320 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f06880099c0 tx=0x7f0688009cd0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.692+0000 7f06a6000700 1 -- 172.21.15.43:0/2609336122 shutdown_connections 2024-05-27T07:40:52.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.692+0000 7f06a6000700 1 --2- 172.21.15.43:0/2609336122 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06a0109860 0x7f06a01111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.692+0000 7f06a6000700 1 --2- 172.21.15.43:0/2609336122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06a01045e0 0x7f06a01049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.692+0000 7f06a6000700 1 --2- 172.21.15.43:0/2609336122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06a0108f10 0x7f06a0109320 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.692+0000 7f06a6000700 1 -- 172.21.15.43:0/2609336122 >> 172.21.15.43:0/2609336122 conn(0x7f06a0100030 msgr2=0x7f06a0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:52.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.692+0000 7f06a6000700 1 -- 172.21.15.43:0/2609336122 shutdown_connections 2024-05-27T07:40:52.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.692+0000 7f06a6000700 1 -- 172.21.15.43:0/2609336122 wait complete. 2024-05-27T07:40:52.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.693+0000 7f06a6000700 1 Processor -- start 2024-05-27T07:40:52.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.694+0000 7f06a6000700 1 -- start start 2024-05-27T07:40:52.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.694+0000 7f06a6000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06a01045e0 0x7f06a019cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.694+0000 7f06a6000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06a0108f10 0x7f06a019d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.694+0000 7f06a6000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06a0109860 0x7f06a01a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.694+0000 7f06a6000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f06a0114320 con 0x7f06a0108f10 2024-05-27T07:40:52.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.694+0000 7f06a6000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f06a01141a0 con 0x7f06a0109860 2024-05-27T07:40:52.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.694+0000 7f06a6000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f06a01144a0 con 0x7f06a01045e0 2024-05-27T07:40:52.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f0696a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06a0108f10 0x7f06a019d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f0696a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06a0108f10 0x7f06a019d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58884/0 (socket says 172.21.15.43:58884) 2024-05-27T07:40:52.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f0696a00700 1 -- 172.21.15.43:0/3581439846 learned_addr learned my addr 172.21.15.43:0/3581439846 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:52.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f069f400700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06a01045e0 0x7f06a019cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f069fe00700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06a0109860 0x7f06a01a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f0696a00700 1 -- 172.21.15.43:0/3581439846 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06a01045e0 msgr2=0x7f06a019cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:52.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f0696a00700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06a01045e0 0x7f06a019cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f0696a00700 1 -- 172.21.15.43:0/3581439846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06a0109860 msgr2=0x7f06a01a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f0696a00700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06a0109860 0x7f06a01a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f0696a00700 1 -- 172.21.15.43:0/3581439846 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a01a1e00 con 0x7f06a0108f10 2024-05-27T07:40:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f069f400700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06a01045e0 0x7f06a019cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.695+0000 7f069fe00700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06a0109860 0x7f06a01a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.696+0000 7f0696a00700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06a0108f10 0x7f06a019d4d0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f068800b580 tx=0x7f068800f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.696+0000 7f069cc00700 1 -- 172.21.15.43:0/3581439846 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0688026070 con 0x7f06a0108f10 2024-05-27T07:40:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.696+0000 7f069cc00700 1 -- 172.21.15.43:0/3581439846 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0688005030 con 0x7f06a0108f10 2024-05-27T07:40:52.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.696+0000 7f06a6000700 1 -- 172.21.15.43:0/3581439846 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a01a2090 con 0x7f06a0108f10 2024-05-27T07:40:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.696+0000 7f069cc00700 1 -- 172.21.15.43:0/3581439846 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06880049f0 con 0x7f06a0108f10 2024-05-27T07:40:52.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.696+0000 7f06a6000700 1 -- 172.21.15.43:0/3581439846 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a01a9a10 con 0x7f06a0108f10 2024-05-27T07:40:52.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.700+0000 7f069cc00700 1 -- 172.21.15.43:0/3581439846 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f0688029070 con 0x7f06a0108f10 2024-05-27T07:40:52.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.701+0000 7f06b4b32b80 1 Processor -- start 2024-05-27T07:40:52.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.701+0000 7f06b4b32b80 1 -- start start 2024-05-27T07:40:52.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.702+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 0x55c585b5e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.702+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b5e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.702+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b6b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.702+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c585a8ce40 con 0x55c5859dfe40 2024-05-27T07:40:52.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.702+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c585a7c9e0 con 0x55c585acb9f0 2024-05-27T07:40:52.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.702+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c585a80a20 con 0x55c585b5ee70 2024-05-27T07:40:52.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.702+0000 7f069f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b5e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.702+0000 7f069f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b5e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58894/0 (socket says 172.21.15.43:58894) 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.702+0000 7f069f400700 1 -- 172.21.15.43:0/1536156226 learned_addr learned my addr 172.21.15.43:0/1536156226 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f069cc00700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f06840b2a70 0x7f06840b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f0696a00700 1 --2- 172.21.15.43:0/1536156226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b6b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f069cc00700 1 -- 172.21.15.43:0/3581439846 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f06880e33e0 con 0x7f06a0108f10 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f069f400700 1 -- 172.21.15.43:0/1536156226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 msgr2=0x55c585b6b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f069f400700 1 --2- 172.21.15.43:0/1536156226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b6b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f069fe00700 1 --2- 172.21.15.43:0/1536156226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 0x55c585b5e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f069f400700 1 -- 172.21.15.43:0/1536156226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 msgr2=0x55c585b5e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f069f400700 1 --2- 172.21.15.43:0/1536156226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 0x55c585b5e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f069f400700 1 -- 172.21.15.43:0/1536156226 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c585a45420 con 0x55c5859dfe40 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f069fe00700 1 --2- 172.21.15.43:0/1536156226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 0x55c585b5e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:52.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.703+0000 7f069fe00700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f06840b2a70 0x7f06840b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.704+0000 7f069f400700 1 --2- 172.21.15.43:0/1536156226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b5e870 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f069000bd90 tx=0x7f069000f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07a4d5bd8f3d6ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.704+0000 7f0694c00700 1 -- 172.21.15.43:0/1536156226 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069000fe50 con 0x55c5859dfe40 2024-05-27T07:40:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.704+0000 7f0694c00700 1 -- 172.21.15.43:0/1536156226 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f06900118d0 con 0x55c5859dfe40 2024-05-27T07:40:52.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.704+0000 7f0694c00700 1 -- 172.21.15.43:0/1536156226 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0690010750 con 0x55c5859dfe40 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.705+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1536156226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 msgr2=0x55c585b5e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.705+0000 7f069fe00700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f06840b2a70 0x7f06840b4f80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0698009710 tx=0x7f069800b040 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.705+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/1536156226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b5e870 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f069000bd90 tx=0x7f069000f640 comp rx=0 tx=0).stop 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.705+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1536156226 shutdown_connections 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.705+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/1536156226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b6b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.705+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/1536156226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 0x55c585b5e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.705+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/1536156226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b5e870 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.705+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1536156226 >> 172.21.15.43:0/1536156226 conn(0x55c585a79390 msgr2=0x55c585b5a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.705+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1536156226 shutdown_connections 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.705+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1536156226 wait complete. 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.706+0000 7f06b4b32b80 1 -- start start 2024-05-27T07:40:52.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.706+0000 7f06b4b32b80 1 Processor -- start 2024-05-27T07:40:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.707+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b68680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.707+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585acb9f0 0x55c585b68bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.707+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585b5ee70 0x55c585b69100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.707+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c585982fa0 con 0x55c5859dfe40 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.707+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c585a1d760 con 0x55c585b5ee70 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.707+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55c585b6bb80 con 0x55c585acb9f0 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.707+0000 7f06b4b32b80 10 client.0 _ll_register_callbacks cb 0x55c585a26c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.707+0000 7f069f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b68680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.707+0000 7f069f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b68680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58896/0 (socket says 172.21.15.43:58896) 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.707+0000 7f069f400700 1 -- 172.21.15.43:0/1490330905 learned_addr learned my addr 172.21.15.43:0/1490330905 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.708+0000 7f0696a00700 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585acb9f0 0x55c585b68bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.708+0000 7f069f400700 1 -- 172.21.15.43:0/1490330905 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585acb9f0 msgr2=0x55c585b68bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.708+0000 7f069f400700 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585acb9f0 0x55c585b68bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.708+0000 7f069fe00700 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585b5ee70 0x55c585b69100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.708+0000 7f069f400700 1 -- 172.21.15.43:0/1490330905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585b5ee70 msgr2=0x55c585b69100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.708+0000 7f069f400700 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585b5ee70 0x55c585b69100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.708+0000 7f069f400700 1 -- 172.21.15.43:0/1490330905 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5859cc8d0 con 0x55c5859dfe40 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.708+0000 7f069f400700 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b68680 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f069000ff50 tx=0x7f0690011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5cf8c4242ddeb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.708+0000 7f068d600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.709+0000 7f068d600700 1 -- 172.21.15.43:0/1490330905 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0690011c90 con 0x55c5859dfe40 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.709+0000 7f068d600700 1 -- 172.21.15.43:0/1490330905 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f0690005040 con 0x55c5859dfe40 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.709+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1490330905 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06900118d0 con 0x55c5859dfe40 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.709+0000 7f068d600700 1 -- 172.21.15.43:0/1490330905 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f0690031050 con 0x55c5859dfe40 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.709+0000 7f06b4b32b80 10 client.18732 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.709+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1490330905 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x55c585a4d060 con 0x55c5859dfe40 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.709+0000 7f068d600700 1 -- 172.21.15.43:0/1490330905 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f06900118d0 con 0x55c5859dfe40 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.710+0000 7f068d600700 1 -- 172.21.15.43:0/1490330905 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7f069002c030 con 0x55c5859dfe40 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.710+0000 7f06b4b32b80 10 client.18732 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.710+0000 7f06b4b32b80 10 client.18732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.710+0000 7f06b4b32b80 20 client.18732 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:52.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.710+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c585b78ce0 0x55c585b7b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.710+0000 7f06b4b32b80 4 client.18732 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T07:40:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.710+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1490330905 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c585a8ed90 con 0x55c585b78ce0 2024-05-27T07:40:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.710+0000 7f0696a00700 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c585b78ce0 0x55c585b7b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.712+0000 7f0696a00700 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c585b78ce0 0x55c585b7b1d0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=933cb60aa0b6611b server_cookie=ec66112d104ec2a1 in_seq=0 out_seq=0 2024-05-27T07:40:52.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.712+0000 7f068d600700 10 client.18732 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f068d600700 1 -- 172.21.15.43:0/1490330905 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55c585a8ed90 con 0x55c585b78ce0 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f068d600700 10 client.18732 handle_command_reply: tid=0 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 client.18732 shutdown 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1490330905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 msgr2=0x55c585b68680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b68680 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f069000ff50 tx=0x7f0690011440 comp rx=0 tx=0).stop 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1490330905 shutdown_connections 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585b5ee70 0x55c585b69100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c585b78ce0 0x55c585b7b1d0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b68680 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/1490330905 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585acb9f0 0x55c585b68bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1490330905 >> 172.21.15.43:0/1490330905 conn(0x55c585a79390 msgr2=0x55c585b5e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:52.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f068d600700 0 client.18732 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:40:52.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1490330905 shutdown_connections 2024-05-27T07:40:52.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.714+0000 7f06b4b32b80 1 -- 172.21.15.43:0/1490330905 wait complete. 2024-05-27T07:40:52.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.715+0000 7f06b4b32b80 20 client.18732 trim_cache size 0 max 16384 2024-05-27T07:40:52.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.732+0000 7f06b4b32b80 1 Processor -- start 2024-05-27T07:40:52.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.732+0000 7f06b4b32b80 1 -- start start 2024-05-27T07:40:52.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.732+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b63e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.732+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 0x55c585b675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.732+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b6a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.732+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c585b5a680 con 0x55c5859dfe40 2024-05-27T07:40:52.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.732+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f06a01b2c30 con 0x55c585acb9f0 2024-05-27T07:40:52.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.732+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f06a01b1030 con 0x55c585b5ee70 2024-05-27T07:40:52.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f069f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b63e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f069f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b63e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58902/0 (socket says 172.21.15.43:58902) 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f069f400700 1 -- 172.21.15.43:0/3496289555 learned_addr learned my addr 172.21.15.43:0/3496289555 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f0696a00700 1 --2- 172.21.15.43:0/3496289555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 0x55c585b675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f069fe00700 1 --2- 172.21.15.43:0/3496289555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b6a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f069f400700 1 -- 172.21.15.43:0/3496289555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 msgr2=0x55c585b6a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f069f400700 1 --2- 172.21.15.43:0/3496289555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b6a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f069f400700 1 -- 172.21.15.43:0/3496289555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 msgr2=0x55c585b675e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f069f400700 1 --2- 172.21.15.43:0/3496289555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 0x55c585b675e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f069f400700 1 -- 172.21.15.43:0/3496289555 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c585b5b000 con 0x55c5859dfe40 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.733+0000 7f069fe00700 1 --2- 172.21.15.43:0/3496289555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b6a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.734+0000 7f069f400700 1 --2- 172.21.15.43:0/3496289555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b63e90 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f0690005a00 tx=0x7f069000f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc03d5d3ef8aad0a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.734+0000 7f068fe00700 1 -- 172.21.15.43:0/3496289555 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069003a070 con 0x55c5859dfe40 2024-05-27T07:40:52.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.734+0000 7f068fe00700 1 -- 172.21.15.43:0/3496289555 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f069000b600 con 0x55c5859dfe40 2024-05-27T07:40:52.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.734+0000 7f068fe00700 1 -- 172.21.15.43:0/3496289555 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069001bb70 con 0x55c5859dfe40 2024-05-27T07:40:52.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.735+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3496289555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 msgr2=0x55c585b63e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:52.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.735+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/3496289555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b63e90 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f0690005a00 tx=0x7f069000f640 comp rx=0 tx=0).stop 2024-05-27T07:40:52.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.735+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3496289555 shutdown_connections 2024-05-27T07:40:52.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.735+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/3496289555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b6a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.735+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/3496289555 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c5859dfe40 0x55c585b63e90 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.735+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/3496289555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c585acb9f0 0x55c585b675e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.735+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3496289555 >> 172.21.15.43:0/3496289555 conn(0x55c585a79390 msgr2=0x55c585b69d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:52.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.735+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3496289555 shutdown_connections 2024-05-27T07:40:52.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.735+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3496289555 wait complete. 2024-05-27T07:40:52.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.736+0000 7f06b4b32b80 1 -- start start 2024-05-27T07:40:52.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.736+0000 7f06b4b32b80 1 Processor -- start 2024-05-27T07:40:52.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5859dfe40 0x55c585b625b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c585acb9f0 0x55c585b62af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f06b4b32b80 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b63050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f06a01b1760 con 0x55c585acb9f0 2024-05-27T07:40:52.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f06a01b2530 con 0x55c5859dfe40 2024-05-27T07:40:52.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f06b4b32b80 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f06a01b1e60 con 0x55c585b5ee70 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f06b4b32b80 10 client.0 _ll_register_callbacks cb 0x55c585a26c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f0696a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c585acb9f0 0x55c585b62af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f0696a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c585acb9f0 0x55c585b62af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58914/0 (socket says 172.21.15.43:58914) 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f0696a00700 1 -- 172.21.15.43:0/3046987068 learned_addr learned my addr 172.21.15.43:0/3046987068 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f069f400700 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5859dfe40 0x55c585b625b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.737+0000 7f069fe00700 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b63050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.738+0000 7f0696a00700 1 -- 172.21.15.43:0/3046987068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 msgr2=0x55c585b63050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.738+0000 7f0696a00700 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b63050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.738+0000 7f0696a00700 1 -- 172.21.15.43:0/3046987068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5859dfe40 msgr2=0x55c585b625b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.738+0000 7f0696a00700 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5859dfe40 0x55c585b625b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.738+0000 7f0696a00700 1 -- 172.21.15.43:0/3046987068 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c585b70050 con 0x55c585acb9f0 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.738+0000 7f0696a00700 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c585acb9f0 0x55c585b62af0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f068800b600 tx=0x7f068809eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2707514406b3f4fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.738+0000 7f068d600700 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.738+0000 7f068d600700 1 -- 172.21.15.43:0/3046987068 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06880a9e30 con 0x55c585acb9f0 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.738+0000 7f068d600700 1 -- 172.21.15.43:0/3046987068 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f06880e0980 con 0x55c585acb9f0 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.738+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3046987068 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f069000b600 con 0x55c585acb9f0 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f068d600700 1 -- 172.21.15.43:0/3046987068 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f06880e0b80 con 0x55c585acb9f0 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f06b4b32b80 10 client.18738 fetch_fsmap learned FSMap version 152 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3046987068 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=4+}) -- 0x55c585b71280 con 0x55c585acb9f0 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f068d600700 1 -- 172.21.15.43:0/3046987068 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f069000b600 con 0x55c585acb9f0 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f068d600700 1 -- 172.21.15.43:0/3046987068 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 152) ==== 4219+0+0 (secure 0 0 0) 0x7f068806f900 con 0x55c585acb9f0 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f06b4b32b80 10 client.18738 fetch_fsmap finished waiting for FSMap version 152 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f06b4b32b80 10 client.18738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-27T07:40:52.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f06b4b32b80 20 client.18738 populate_metadata read hostname 'smithi043' 2024-05-27T07:40:52.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c585be38a0 0x55c585be5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:52.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f06b4b32b80 4 client.18738 mds_command: new command op to 34484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-27T07:40:52.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.739+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3046987068 --> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c585a88570 con 0x55c585be38a0 2024-05-27T07:40:52.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.740+0000 7f069f400700 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c585be38a0 0x55c585be5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.741+0000 7f069f400700 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c585be38a0 0x55c585be5cc0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52db9f3055f3311c server_cookie=63cbc7d9202d7e56 in_seq=0 out_seq=0 2024-05-27T07:40:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.741+0000 7f068d600700 10 client.18738 ms_handle_connect on v2:172.21.15.132:6838/4104367678 2024-05-27T07:40:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.741+0000 7f068d600700 1 -- 172.21.15.43:0/3046987068 <== mds.0 v2:172.21.15.132:6838/4104367678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c585a88570 con 0x55c585be38a0 2024-05-27T07:40:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.741+0000 7f068d600700 10 client.18738 handle_command_reply: tid=0 2024-05-27T07:40:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.741+0000 7f06b4b32b80 1 client.18738 shutdown 2024-05-27T07:40:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3046987068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c585acb9f0 msgr2=0x55c585b62af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c585acb9f0 0x55c585b62af0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f068800b600 tx=0x7f068809eba0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3046987068 shutdown_connections 2024-05-27T07:40:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c585b5ee70 0x55c585b63050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c5859dfe40 0x55c585b625b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.132:6838/4104367678,v1:172.21.15.132:6839/4104367678] conn(0x55c585be38a0 0x55c585be5cc0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 1 --2- 172.21.15.43:0/3046987068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c585acb9f0 0x55c585b62af0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3046987068 >> 172.21.15.43:0/3046987068 conn(0x55c585a79390 msgr2=0x55c585b69be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f068d600700 0 client.18738 ms_handle_reset on v2:172.21.15.132:6838/4104367678 2024-05-27T07:40:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3046987068 shutdown_connections 2024-05-27T07:40:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 1 -- 172.21.15.43:0/3046987068 wait complete. 2024-05-27T07:40:52.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.742+0000 7f06b4b32b80 20 client.18738 trim_cache size 0 max 16384 2024-05-27T07:40:52.744 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-05-27T07:40:52.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.743+0000 7f06a6000700 1 -- 172.21.15.43:0/3581439846 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f06840b2a70 msgr2=0x7f06840b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.743+0000 7f06a6000700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f06840b2a70 0x7f06840b4f80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0698009710 tx=0x7f069800b040 comp rx=0 tx=0).stop 2024-05-27T07:40:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.743+0000 7f06a6000700 1 -- 172.21.15.43:0/3581439846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06a0108f10 msgr2=0x7f06a019d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.743+0000 7f06a6000700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06a0108f10 0x7f06a019d4d0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f068800b580 tx=0x7f068800f510 comp rx=0 tx=0).stop 2024-05-27T07:40:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.743+0000 7f06a6000700 1 -- 172.21.15.43:0/3581439846 shutdown_connections 2024-05-27T07:40:52.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.743+0000 7f06a6000700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f06840b2a70 0x7f06840b4f80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.743+0000 7f06a6000700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06a0109860 0x7f06a01a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.743+0000 7f06a6000700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06a01045e0 0x7f06a019cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.743+0000 7f06a6000700 1 --2- 172.21.15.43:0/3581439846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06a0108f10 0x7f06a019d4d0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:52.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.743+0000 7f06a6000700 1 -- 172.21.15.43:0/3581439846 >> 172.21.15.43:0/3581439846 conn(0x7f06a0100030 msgr2=0x7f06a010a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:52.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.744+0000 7f06a6000700 1 -- 172.21.15.43:0/3581439846 shutdown_connections 2024-05-27T07:40:52.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:52.744+0000 7f06a6000700 1 -- 172.21.15.43:0/3581439846 wait complete. 2024-05-27T07:40:53.696 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-05-27T07:40:53.697 INFO:tasks.fwd_scrub:done joining 2024-05-27T07:40:53.698 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-27T07:40:53.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T07:40:53.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:53 smithi043 ceph-mon[31531]: pgmap v903: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:53.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:53 smithi179 ceph-mon[35888]: pgmap v903: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:54.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.026+0000 7fcebd400640 1 -- 172.21.15.43:0/4193442664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fceb8104370 msgr2=0x7fceb8110740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.026+0000 7fcebd400640 1 --2- 172.21.15.43:0/4193442664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fceb8104370 0x7fceb8110740 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009900 tx=0x7fceb002a410 comp rx=0 tx=0).stop 2024-05-27T07:40:54.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.027+0000 7fcebd400640 1 -- 172.21.15.43:0/4193442664 shutdown_connections 2024-05-27T07:40:54.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.027+0000 7fcebd400640 1 --2- 172.21.15.43:0/4193442664 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fceb810a0c0 0x7fceb8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.027+0000 7fcebd400640 1 --2- 172.21.15.43:0/4193442664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fceb8104370 0x7fceb8110740 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.027+0000 7fcebd400640 1 --2- 172.21.15.43:0/4193442664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb8109c20 0x7fceb81037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.027+0000 7fcebd400640 1 -- 172.21.15.43:0/4193442664 >> 172.21.15.43:0/4193442664 conn(0x7fceb8079360 msgr2=0x7fceb80797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:54.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.027+0000 7fcebd400640 1 -- 172.21.15.43:0/4193442664 shutdown_connections 2024-05-27T07:40:54.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.027+0000 7fcebd400640 1 -- 172.21.15.43:0/4193442664 wait complete. 2024-05-27T07:40:54.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.028+0000 7fcebd400640 1 Processor -- start 2024-05-27T07:40:54.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.028+0000 7fcebd400640 1 -- start start 2024-05-27T07:40:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.029+0000 7fcebd400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb8104370 0x7fceb8074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.029+0000 7fcebd400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fceb8109c20 0x7fceb8075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.029+0000 7fcebd400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fceb810a0c0 0x7fceb8075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.029+0000 7fcebd400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fceb8113ce0 con 0x7fceb810a0c0 2024-05-27T07:40:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.029+0000 7fcebd400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fceb8113b60 con 0x7fceb8109c20 2024-05-27T07:40:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.029+0000 7fcebd400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fceb8113e60 con 0x7fceb8104370 2024-05-27T07:40:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.029+0000 7fceb7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fceb810a0c0 0x7fceb8075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.030+0000 7fceb7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fceb810a0c0 0x7fceb8075a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58942/0 (socket says 172.21.15.43:58942) 2024-05-27T07:40:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.030+0000 7fceb7200640 1 -- 172.21.15.43:0/366907468 learned_addr learned my addr 172.21.15.43:0/366907468 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.030+0000 7fceb6800640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb8104370 0x7fceb8074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.030+0000 7fceb7200640 1 -- 172.21.15.43:0/366907468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb8104370 msgr2=0x7fceb8074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.030+0000 7fceb7200640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb8104370 0x7fceb8074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.030+0000 7fceb7200640 1 -- 172.21.15.43:0/366907468 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fceb8109c20 msgr2=0x7fceb8075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.030+0000 7fceb7200640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fceb8109c20 0x7fceb8075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.030+0000 7fceb7200640 1 -- 172.21.15.43:0/366907468 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb81a8a80 con 0x7fceb810a0c0 2024-05-27T07:40:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.030+0000 7fceb6800640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb8104370 0x7fceb8074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:40:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.031+0000 7fceb7200640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fceb810a0c0 0x7fceb8075a40 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fceb0005ab0 tx=0x7fceb00040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.031+0000 7fceaf400640 1 -- 172.21.15.43:0/366907468 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb003f030 con 0x7fceb810a0c0 2024-05-27T07:40:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.031+0000 7fceaf400640 1 -- 172.21.15.43:0/366907468 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fceb0003a80 con 0x7fceb810a0c0 2024-05-27T07:40:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.031+0000 7fceaf400640 1 -- 172.21.15.43:0/366907468 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0035070 con 0x7fceb810a0c0 2024-05-27T07:40:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.031+0000 7fcebd400640 1 -- 172.21.15.43:0/366907468 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb81a8d10 con 0x7fceb810a0c0 2024-05-27T07:40:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.031+0000 7fcebd400640 1 -- 172.21.15.43:0/366907468 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb81a91b0 con 0x7fceb810a0c0 2024-05-27T07:40:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.032+0000 7fcebd400640 1 -- 172.21.15.43:0/366907468 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceb806a230 con 0x7fceb810a0c0 2024-05-27T07:40:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.035+0000 7fceaf400640 1 -- 172.21.15.43:0/366907468 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fceb0042030 con 0x7fceb810a0c0 2024-05-27T07:40:54.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.037+0000 7fceaf400640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce9c0b2960 0x7fce9c0b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.037+0000 7fceaf400640 1 -- 172.21.15.43:0/366907468 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7fceb0103430 con 0x7fceb810a0c0 2024-05-27T07:40:54.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.037+0000 7fceb6800640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce9c0b2960 0x7fce9c0b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.038+0000 7fceb6800640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce9c0b2960 0x7fce9c0b4e70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fceb8104900 tx=0x7fcea8009520 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:54.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.040+0000 7fceaf400640 1 -- 172.21.15.43:0/366907468 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fceb00ca9e0 con 0x7fceb810a0c0 2024-05-27T07:40:54.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:53 smithi132 ceph-mon[35283]: pgmap v903: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:54.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.260+0000 7fcebd400640 1 -- 172.21.15.43:0/366907468 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fceb80764f0 con 0x7fceb810a0c0 2024-05-27T07:40:54.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.265+0000 7fceaf400640 1 -- 172.21.15.43:0/366907468 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) ==== 74+0+25314 (secure 0 0 0) 0x7fceb80764f0 con 0x7fceb810a0c0 2024-05-27T07:40:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:40:54.264 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":163,"fsid":"1bb64d42-1bf6-11ef-bc9b-c7b262605968","created":"2024-05-27T06:56:43.552142+0000","modified":"2024-05-27T07:40:11.609763+0000","last_up_change":"2024-05-27T07:06:53.842598+0000","last_in_change":"2024-05-27T07:06:26.540195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T07:01:14.696118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T07:08:16.033058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T07:08:20.929248+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":162,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-27T07:08:34.823559+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":162,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"92381bc8-d8af-4c8d-b55c-2311e7de0211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6803","nonce":3368788577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6805","nonce":3368788577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6809","nonce":3368788577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":3368788577},{"type":"v1","addr":"172.21.15.43:6807","nonce":3368788577}]},"public_addr":"172.21.15.43:6803/3368788577","cluster_addr":"172.21.15.43:6805/3368788577","heartbeat_back_addr":"172.21.15.43:6809/3368788577","heartbeat_front_addr":"172.21.15.43:6807/3368788577","state":["exists","up"]},{"osd":1,"uuid":"8749b216-d75b-4f50-be38-9d9c7c67fe57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6801","nonce":3434174667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6803","nonce":3434174667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6807","nonce":3434174667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3434174667},{"type":"v1","addr":"172.21.15.132:6805","nonce":3434174667}]},"public_addr":"172.21.15.132:6801/3434174667","cluster_addr":"172.21.15.132:6803/3434174667","heartbeat_back_addr":"172.21.15.132:6807/3434174667","heartbeat_front_addr":"172.21.15.132:6805/3434174667","state":["exists","up"]},{"osd":2,"uuid":"97582b6e-2b42-48b6-bf39-511a1121e069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6801","nonce":2313942243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6803","nonce":2313942243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6807","nonce":2313942243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2313942243},{"type":"v1","addr":"172.21.15.179:6805","nonce":2313942243}]},"public_addr":"172.21.15.179:6801/2313942243","cluster_addr":"172.21.15.179:6803/2313942243","heartbeat_back_addr":"172.21.15.179:6807/2313942243","heartbeat_front_addr":"172.21.15.179:6805/2313942243","state":["exists","up"]},{"osd":3,"uuid":"8247fd20-bc21-430a-a3b9-bd5221468995","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6811","nonce":902648995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6813","nonce":902648995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6817","nonce":902648995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":902648995},{"type":"v1","addr":"172.21.15.43:6815","nonce":902648995}]},"public_addr":"172.21.15.43:6811/902648995","cluster_addr":"172.21.15.43:6813/902648995","heartbeat_back_addr":"172.21.15.43:6817/902648995","heartbeat_front_addr":"172.21.15.43:6815/902648995","state":["exists","up"]},{"osd":4,"uuid":"8f6a847d-4753-4061-bf71-27d740189512","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6809","nonce":4132674108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6811","nonce":4132674108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6815","nonce":4132674108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":4132674108},{"type":"v1","addr":"172.21.15.132:6813","nonce":4132674108}]},"public_addr":"172.21.15.132:6809/4132674108","cluster_addr":"172.21.15.132:6811/4132674108","heartbeat_back_addr":"172.21.15.132:6815/4132674108","heartbeat_front_addr":"172.21.15.132:6813/4132674108","state":["exists","up"]},{"osd":5,"uuid":"dbd4f8b4-32c6-46e5-bfcf-0c77099e2c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6809","nonce":1714122228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6811","nonce":1714122228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6815","nonce":1714122228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":1714122228},{"type":"v1","addr":"172.21.15.179:6813","nonce":1714122228}]},"public_addr":"172.21.15.179:6809/1714122228","cluster_addr":"172.21.15.179:6811/1714122228","heartbeat_back_addr":"172.21.15.179:6815/1714122228","heartbeat_front_addr":"172.21.15.179:6813/1714122228","state":["exists","up"]},{"osd":6,"uuid":"a722fe71-7939-43d7-9c9e-72f2c7e559be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6819","nonce":2712807127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6821","nonce":2712807127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6825","nonce":2712807127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":2712807127},{"type":"v1","addr":"172.21.15.43:6823","nonce":2712807127}]},"public_addr":"172.21.15.43:6819/2712807127","cluster_addr":"172.21.15.43:6821/2712807127","heartbeat_back_addr":"172.21.15.43:6825/2712807127","heartbeat_front_addr":"172.21.15.43:6823/2712807127","state":["exists","up"]},{"osd":7,"uuid":"3b6b6895-34fe-4afc-90c1-a0d30b365713","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6817","nonce":2427045470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6819","nonce":2427045470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6823","nonce":2427045470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2427045470},{"type":"v1","addr":"172.21.15.132:6821","nonce":2427045470}]},"public_addr":"172.21.15.132:6817/2427045470","cluster_addr":"172.21.15.132:6819/2427045470","heartbeat_back_addr":"172.21.15.132:6823/2427045470","heartbeat_front_addr":"172.21.15.132:6821/2427045470","state":["exists","up"]},{"osd":8,"uuid":"10a2bd5e-5ac8-441f-b2e5-deb1f9239b17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6817","nonce":2617404629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6819","nonce":2617404629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6823","nonce":2617404629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":2617404629},{"type":"v1","addr":"172.21.15.179:6821","nonce":2617404629}]},"public_addr":"172.21.15.179:6817/2617404629","cluster_addr":"172.21.15.179:6819/2617404629","heartbeat_back_addr":"172.21.15.179:6823/2617404629","heartbeat_front_addr":"172.21.15.179:6821/2617404629","state":["exists","up"]},{"osd":9,"uuid":"19e77076-5052-4d5f-a35b-00bb595cf435","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6827","nonce":3342932073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6829","nonce":3342932073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6833","nonce":3342932073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3342932073},{"type":"v1","addr":"172.21.15.43:6831","nonce":3342932073}]},"public_addr":"172.21.15.43:6827/3342932073","cluster_addr":"172.21.15.43:6829/3342932073","heartbeat_back_addr":"172.21.15.43:6833/3342932073","heartbeat_front_addr":"172.21.15.43:6831/3342932073","state":["exists","up"]},{"osd":10,"uuid":"78b7db29-142a-4f61-a360-8139d14f676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6825","nonce":3139130496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6827","nonce":3139130496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6831","nonce":3139130496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3139130496},{"type":"v1","addr":"172.21.15.132:6829","nonce":3139130496}]},"public_addr":"172.21.15.132:6825/3139130496","cluster_addr":"172.21.15.132:6827/3139130496","heartbeat_back_addr":"172.21.15.132:6831/3139130496","heartbeat_front_addr":"172.21.15.132:6829/3139130496","state":["exists","up"]},{"osd":11,"uuid":"dfe7c464-567c-4836-8042-4bb9daedb76b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6825","nonce":55249493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6827","nonce":55249493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6831","nonce":55249493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":55249493},{"type":"v1","addr":"172.21.15.179:6829","nonce":55249493}]},"public_addr":"172.21.15.179:6825/55249493","cluster_addr":"172.21.15.179:6827/55249493","heartbeat_back_addr":"172.21.15.179:6831/55249493","heartbeat_front_addr":"172.21.15.179:6829/55249493","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:59:58.559806+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:00:34.812891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:10.614775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:01:47.467456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:02:25.243761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:02.706162+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:03:40.711501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:19.129607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:04:57.004929+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:05:34.981379+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:13.123685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T07:06:51.119952+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:6800/1809594794":"2024-05-28T07:13:54.391569+0000","172.21.15.43:0/2266081183":"2024-05-28T07:13:54.391569+0000","172.21.15.43:0/3331041602":"2024-05-28T07:13:54.391569+0000","172.21.15.43:6801/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/3399450175":"2024-05-28T07:13:54.391569+0000","172.21.15.43:6800/1651567761":"2024-05-28T06:57:52.506320+0000","172.21.15.43:0/1332970913":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/1752945868":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2426463283":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/2081880966":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6800/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:6801/1809594794":"2024-05-28T07:13:54.391569+0000","172.21.15.43:0/3253329315":"2024-05-28T07:13:54.391569+0000","172.21.15.43:0/830904719":"2024-05-28T06:57:52.506320+0000","172.21.15.43:6801/3521718752":"2024-05-28T06:57:18.380825+0000","172.21.15.43:0/338672925":"2024-05-28T06:57:52.506320+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]},{"pool":4,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T07:40:54.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.268+0000 7fcebd400640 1 -- 172.21.15.43:0/366907468 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce9c0b2960 msgr2=0x7fce9c0b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.268+0000 7fcebd400640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce9c0b2960 0x7fce9c0b4e70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fceb8104900 tx=0x7fcea8009520 comp rx=0 tx=0).stop 2024-05-27T07:40:54.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.268+0000 7fcebd400640 1 -- 172.21.15.43:0/366907468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fceb810a0c0 msgr2=0x7fceb8075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.268+0000 7fcebd400640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fceb810a0c0 0x7fceb8075a40 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fceb0005ab0 tx=0x7fceb00040e0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.268+0000 7fcebd400640 1 -- 172.21.15.43:0/366907468 shutdown_connections 2024-05-27T07:40:54.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.268+0000 7fcebd400640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fce9c0b2960 0x7fce9c0b4e70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.268+0000 7fcebd400640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fceb8109c20 0x7fceb8075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.268+0000 7fcebd400640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fceb810a0c0 0x7fceb8075a40 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.268+0000 7fcebd400640 1 --2- 172.21.15.43:0/366907468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb8104370 0x7fceb8074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.268+0000 7fcebd400640 1 -- 172.21.15.43:0/366907468 >> 172.21.15.43:0/366907468 conn(0x7fceb8079360 msgr2=0x7fceb8101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:54.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.269+0000 7fcebd400640 1 -- 172.21.15.43:0/366907468 shutdown_connections 2024-05-27T07:40:54.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.269+0000 7fcebd400640 1 -- 172.21.15.43:0/366907468 wait complete. 2024-05-27T07:40:54.297 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T07:01:14.696118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T07:08:16.033058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T07:08:20.929248+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '162', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 162, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-27T07:08:34.823559+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '162', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 162, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-27T07:40:54.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T07:40:54.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.588+0000 7ffa13e00640 1 -- 172.21.15.43:0/1934588135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa1410e870 msgr2=0x7ffa1410eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.588+0000 7ffa13e00640 1 --2- 172.21.15.43:0/1934588135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa1410e870 0x7ffa1410eca0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009900 tx=0x7ff9fc02a410 comp rx=0 tx=0).stop 2024-05-27T07:40:54.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.588+0000 7ffa13e00640 1 -- 172.21.15.43:0/1934588135 shutdown_connections 2024-05-27T07:40:54.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.588+0000 7ffa13e00640 1 --2- 172.21.15.43:0/1934588135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa1410f330 0x7ffa14116ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.588+0000 7ffa13e00640 1 --2- 172.21.15.43:0/1934588135 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa1410e870 0x7ffa1410eca0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.588+0000 7ffa13e00640 1 --2- 172.21.15.43:0/1934588135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa1410a2e0 0x7ffa1410a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.588+0000 7ffa13e00640 1 -- 172.21.15.43:0/1934588135 >> 172.21.15.43:0/1934588135 conn(0x7ffa14076c80 msgr2=0x7ffa140770c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:54.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.588+0000 7ffa13e00640 1 -- 172.21.15.43:0/1934588135 shutdown_connections 2024-05-27T07:40:54.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.588+0000 7ffa13e00640 1 -- 172.21.15.43:0/1934588135 wait complete. 2024-05-27T07:40:54.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.590+0000 7ffa13e00640 1 Processor -- start 2024-05-27T07:40:54.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.590+0000 7ffa13e00640 1 -- start start 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.590+0000 7ffa13e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa1410a2e0 0x7ffa141a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.590+0000 7ffa13e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa1410e870 0x7ffa141a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.590+0000 7ffa13e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa1410f330 0x7ffa141a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.590+0000 7ffa13e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1411a0e0 con 0x7ffa1410a2e0 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.590+0000 7ffa13e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffa14119f60 con 0x7ffa1410f330 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.590+0000 7ffa13e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1411a260 con 0x7ffa1410e870 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa12a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa1410a2e0 0x7ffa141a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa12a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa1410a2e0 0x7ffa141a6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57566/0 (socket says 172.21.15.43:57566) 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa12a00640 1 -- 172.21.15.43:0/1036996080 learned_addr learned my addr 172.21.15.43:0/1036996080 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa13400640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa1410f330 0x7ffa141a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa12a00640 1 -- 172.21.15.43:0/1036996080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa1410e870 msgr2=0x7ffa141a6d90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa12a00640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa1410e870 0x7ffa141a6d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa12a00640 1 -- 172.21.15.43:0/1036996080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa1410f330 msgr2=0x7ffa141a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa12a00640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa1410f330 0x7ffa141a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa12a00640 1 -- 172.21.15.43:0/1036996080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa141a1100 con 0x7ffa1410a2e0 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa13400640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa1410f330 0x7ffa141a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.591+0000 7ffa12a00640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa1410a2e0 0x7ffa141a6850 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400d950 tx=0x7ffa0400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.592+0000 7ffa0be00640 1 -- 172.21.15.43:0/1036996080 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa04017070 con 0x7ffa1410a2e0 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.592+0000 7ffa0be00640 1 -- 172.21.15.43:0/1036996080 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffa04009c30 con 0x7ffa1410a2e0 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.592+0000 7ffa13e00640 1 -- 172.21.15.43:0/1036996080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa141a13f0 con 0x7ffa1410a2e0 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.592+0000 7ffa0be00640 1 -- 172.21.15.43:0/1036996080 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0400f620 con 0x7ffa1410a2e0 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.592+0000 7ffa13e00640 1 -- 172.21.15.43:0/1036996080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa141ad6d0 con 0x7ffa1410a2e0 2024-05-27T07:40:54.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.593+0000 7ffa13e00640 1 -- 172.21.15.43:0/1036996080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa1406a150 con 0x7ffa1410a2e0 2024-05-27T07:40:54.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.597+0000 7ffa0be00640 1 -- 172.21.15.43:0/1036996080 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7ffa04010460 con 0x7ffa1410a2e0 2024-05-27T07:40:54.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.599+0000 7ffa0be00640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff9f40b2950 0x7ff9f40b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:54.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.599+0000 7ffa0be00640 1 -- 172.21.15.43:0/1036996080 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7ffa040e26d0 con 0x7ffa1410a2e0 2024-05-27T07:40:54.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.599+0000 7ffa0a000640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff9f40b2950 0x7ff9f40b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:54.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.600+0000 7ffa0a000640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff9f40b2950 0x7ff9f40b4e60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc02a930 tx=0x7ff9fc005fd0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:54.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.602+0000 7ffa0be00640 1 -- 172.21.15.43:0/1036996080 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ffa040a9ce0 con 0x7ffa1410a2e0 2024-05-27T07:40:54.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.815+0000 7ffa13e00640 1 -- 172.21.15.43:0/1036996080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffa141ad8a0 con 0x7ffa1410a2e0 2024-05-27T07:40:54.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.816+0000 7ffa0be00640 1 -- 172.21.15.43:0/1036996080 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v163) ==== 93+0+10 (secure 0 0 0) 0x7ffa04009de0 con 0x7ffa1410a2e0 2024-05-27T07:40:54.814 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-05-27T07:40:54.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.819+0000 7ffa13e00640 1 -- 172.21.15.43:0/1036996080 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff9f40b2950 msgr2=0x7ff9f40b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.819+0000 7ffa13e00640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff9f40b2950 0x7ff9f40b4e60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc02a930 tx=0x7ff9fc005fd0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.819+0000 7ffa13e00640 1 -- 172.21.15.43:0/1036996080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa1410a2e0 msgr2=0x7ffa141a6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.819+0000 7ffa13e00640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa1410a2e0 0x7ffa141a6850 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400d950 tx=0x7ffa0400de10 comp rx=0 tx=0).stop 2024-05-27T07:40:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.819+0000 7ffa13e00640 1 -- 172.21.15.43:0/1036996080 shutdown_connections 2024-05-27T07:40:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.819+0000 7ffa13e00640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7ff9f40b2950 0x7ff9f40b4e60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.819+0000 7ffa13e00640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa1410f330 0x7ffa141a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.819+0000 7ffa13e00640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa1410e870 0x7ffa141a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.819+0000 7ffa13e00640 1 --2- 172.21.15.43:0/1036996080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa1410a2e0 0x7ffa141a6850 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.819+0000 7ffa13e00640 1 -- 172.21.15.43:0/1036996080 >> 172.21.15.43:0/1036996080 conn(0x7ffa14076c80 msgr2=0x7ffa14107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:54.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.820+0000 7ffa13e00640 1 -- 172.21.15.43:0/1036996080 shutdown_connections 2024-05-27T07:40:54.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:54.820+0000 7ffa13e00640 1 -- 172.21.15.43:0/1036996080 wait complete. 2024-05-27T07:40:54.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T07:40:54.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:54 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/366907468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:40:54.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:54 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/366907468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:40:55.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:54 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/366907468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-27T07:40:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.132+0000 7fd1efe00640 1 -- 172.21.15.43:0/3077067561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1f010f330 msgr2=0x7fd1f0116ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:55.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.132+0000 7fd1efe00640 1 --2- 172.21.15.43:0/3077067561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1f010f330 0x7fd1f0116ad0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8009ca0 tx=0x7fd1e802a8c0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.132+0000 7fd1efe00640 1 -- 172.21.15.43:0/3077067561 shutdown_connections 2024-05-27T07:40:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.132+0000 7fd1efe00640 1 --2- 172.21.15.43:0/3077067561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1f010f330 0x7fd1f0116ad0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.132+0000 7fd1efe00640 1 --2- 172.21.15.43:0/3077067561 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd1f010e870 0x7fd1f010eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.132+0000 7fd1efe00640 1 --2- 172.21.15.43:0/3077067561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd1f010a2e0 0x7fd1f010a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.132+0000 7fd1efe00640 1 -- 172.21.15.43:0/3077067561 >> 172.21.15.43:0/3077067561 conn(0x7fd1f0076c80 msgr2=0x7fd1f00770c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.133+0000 7fd1efe00640 1 -- 172.21.15.43:0/3077067561 shutdown_connections 2024-05-27T07:40:55.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.133+0000 7fd1efe00640 1 -- 172.21.15.43:0/3077067561 wait complete. 2024-05-27T07:40:55.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.134+0000 7fd1efe00640 1 Processor -- start 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.134+0000 7fd1efe00640 1 -- start start 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.134+0000 7fd1efe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1f010a2e0 0x7fd1f01a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.134+0000 7fd1efe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd1f010e870 0x7fd1f01a6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.134+0000 7fd1efe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd1f010f330 0x7fd1f01a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.134+0000 7fd1efe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f011a140 con 0x7fd1f010a2e0 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.134+0000 7fd1efe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f0119fc0 con 0x7fd1f010e870 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.134+0000 7fd1efe00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f011a2c0 con 0x7fd1f010f330 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.134+0000 7fd1eea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1f010a2e0 0x7fd1f01a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.135+0000 7fd1eea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1f010a2e0 0x7fd1f01a6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57596/0 (socket says 172.21.15.43:57596) 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.135+0000 7fd1eea00640 1 -- 172.21.15.43:0/1670798540 learned_addr learned my addr 172.21.15.43:0/1670798540 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.135+0000 7fd1eea00640 1 -- 172.21.15.43:0/1670798540 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd1f010f330 msgr2=0x7fd1f01a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.135+0000 7fd1eea00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd1f010f330 0x7fd1f01a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.135+0000 7fd1eea00640 1 -- 172.21.15.43:0/1670798540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd1f010e870 msgr2=0x7fd1f01a6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:55.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.135+0000 7fd1eea00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd1f010e870 0x7fd1f01a6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.135+0000 7fd1eea00640 1 -- 172.21.15.43:0/1670798540 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f01a0f90 con 0x7fd1f010a2e0 2024-05-27T07:40:55.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.135+0000 7fd1eea00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1f010a2e0 0x7fd1f01a6790 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000c920 tx=0x7fd1e000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:55.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.136+0000 7fd1e7400640 1 -- 172.21.15.43:0/1670798540 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0007c70 con 0x7fd1f010a2e0 2024-05-27T07:40:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.136+0000 7fd1e7400640 1 -- 172.21.15.43:0/1670798540 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd1e0007e10 con 0x7fd1f010a2e0 2024-05-27T07:40:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.136+0000 7fd1efe00640 1 -- 172.21.15.43:0/1670798540 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f01a1280 con 0x7fd1f010a2e0 2024-05-27T07:40:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.136+0000 7fd1e7400640 1 -- 172.21.15.43:0/1670798540 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0018710 con 0x7fd1f010a2e0 2024-05-27T07:40:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.136+0000 7fd1efe00640 1 -- 172.21.15.43:0/1670798540 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f01ad610 con 0x7fd1f010a2e0 2024-05-27T07:40:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.137+0000 7fd1efe00640 1 -- 172.21.15.43:0/1670798540 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f006a150 con 0x7fd1f010a2e0 2024-05-27T07:40:55.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.139+0000 7fd1e7400640 1 -- 172.21.15.43:0/1670798540 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7fd1e000f450 con 0x7fd1f010a2e0 2024-05-27T07:40:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.141+0000 7fd1e7400640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd1d00b29d0 0x7fd1d00b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.141+0000 7fd1e7400640 1 -- 172.21.15.43:0/1670798540 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7fd1e00e2c30 con 0x7fd1f010a2e0 2024-05-27T07:40:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.141+0000 7fd1e7e00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd1d00b29d0 0x7fd1d00b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:55.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.142+0000 7fd1e7e00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd1d00b29d0 0x7fd1d00b4ee0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8005d80 tx=0x7fd1d8005c90 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:55.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.145+0000 7fd1e7400640 1 -- 172.21.15.43:0/1670798540 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd1e0014030 con 0x7fd1f010a2e0 2024-05-27T07:40:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.357+0000 7fd1efe00640 1 -- 172.21.15.43:0/1670798540 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd1f002dc20 con 0x7fd1f010a2e0 2024-05-27T07:40:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.358+0000 7fd1e7400640 1 -- 172.21.15.43:0/1670798540 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v163) ==== 104+0+11 (secure 0 0 0) 0x7fd1e00aa1e0 con 0x7fd1f010a2e0 2024-05-27T07:40:55.356 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-05-27T07:40:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 -- 172.21.15.43:0/1670798540 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd1d00b29d0 msgr2=0x7fd1d00b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd1d00b29d0 0x7fd1d00b4ee0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8005d80 tx=0x7fd1d8005c90 comp rx=0 tx=0).stop 2024-05-27T07:40:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 -- 172.21.15.43:0/1670798540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1f010a2e0 msgr2=0x7fd1f01a6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1f010a2e0 0x7fd1f01a6790 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000c920 tx=0x7fd1e000cde0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 -- 172.21.15.43:0/1670798540 shutdown_connections 2024-05-27T07:40:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7fd1d00b29d0 0x7fd1d00b4ee0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd1f010e870 0x7fd1f01a6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd1f010f330 0x7fd1f01a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 --2- 172.21.15.43:0/1670798540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1f010a2e0 0x7fd1f01a6790 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 -- 172.21.15.43:0/1670798540 >> 172.21.15.43:0/1670798540 conn(0x7fd1f0076c80 msgr2=0x7fd1f0107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 -- 172.21.15.43:0/1670798540 shutdown_connections 2024-05-27T07:40:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.361+0000 7fd1efe00640 1 -- 172.21.15.43:0/1670798540 wait complete. 2024-05-27T07:40:55.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T07:40:55.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.690+0000 7f881fe00640 1 -- 172.21.15.43:0/3690375967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8820102a70 msgr2=0x7f8820102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:55.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.690+0000 7f881fe00640 1 --2- 172.21.15.43:0/3690375967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8820102a70 0x7f8820102e80 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f880c009900 tx=0x7f880c02a410 comp rx=0 tx=0).stop 2024-05-27T07:40:55.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.690+0000 7f881fe00640 1 -- 172.21.15.43:0/3690375967 shutdown_connections 2024-05-27T07:40:55.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.690+0000 7f881fe00640 1 --2- 172.21.15.43:0/3690375967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f882010d300 0x7f882010f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.690+0000 7f881fe00640 1 --2- 172.21.15.43:0/3690375967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8820102a70 0x7f8820102e80 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.690+0000 7f881fe00640 1 --2- 172.21.15.43:0/3690375967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88201089a0 0x7f8820102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.690+0000 7f881fe00640 1 -- 172.21.15.43:0/3690375967 >> 172.21.15.43:0/3690375967 conn(0x7f8820077fe0 msgr2=0x7f8820078420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:55.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.691+0000 7f881fe00640 1 -- 172.21.15.43:0/3690375967 shutdown_connections 2024-05-27T07:40:55.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.691+0000 7f881fe00640 1 -- 172.21.15.43:0/3690375967 wait complete. 2024-05-27T07:40:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.692+0000 7f881fe00640 1 Processor -- start 2024-05-27T07:40:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.692+0000 7f881fe00640 1 -- start start 2024-05-27T07:40:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.692+0000 7f881fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8820102a70 0x7f882019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.692+0000 7f881fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88201089a0 0x7f882019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.692+0000 7f881fe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f882010d300 0x7f88201a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.692+0000 7f881fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8820112a10 con 0x7f88201089a0 2024-05-27T07:40:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.692+0000 7f881fe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8820112890 con 0x7f8820102a70 2024-05-27T07:40:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.692+0000 7f881fe00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8820112b90 con 0x7f882010d300 2024-05-27T07:40:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88201089a0 0x7f882019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88201089a0 0x7f882019cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57614/0 (socket says 172.21.15.43:57614) 2024-05-27T07:40:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881e000640 1 -- 172.21.15.43:0/2741488482 learned_addr learned my addr 172.21.15.43:0/2741488482 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881ea00640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8820102a70 0x7f882019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881f400640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f882010d300 0x7f88201a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881e000640 1 -- 172.21.15.43:0/2741488482 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f882010d300 msgr2=0x7f88201a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881e000640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f882010d300 0x7f88201a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881e000640 1 -- 172.21.15.43:0/2741488482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8820102a70 msgr2=0x7f882019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881e000640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8820102a70 0x7f882019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881e000640 1 -- 172.21.15.43:0/2741488482 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88201a1610 con 0x7f88201089a0 2024-05-27T07:40:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.693+0000 7f881f400640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f882010d300 0x7f88201a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.694+0000 7f881e000640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88201089a0 0x7f882019cd10 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f880c005940 tx=0x7f880c0040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.694+0000 7f8817400640 1 -- 172.21.15.43:0/2741488482 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c03f030 con 0x7f88201089a0 2024-05-27T07:40:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.694+0000 7f8817400640 1 -- 172.21.15.43:0/2741488482 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f880c003a80 con 0x7f88201089a0 2024-05-27T07:40:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.694+0000 7f881fe00640 1 -- 172.21.15.43:0/2741488482 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88201a18a0 con 0x7f88201089a0 2024-05-27T07:40:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.694+0000 7f8817400640 1 -- 172.21.15.43:0/2741488482 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c035070 con 0x7f88201089a0 2024-05-27T07:40:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.694+0000 7f881fe00640 1 -- 172.21.15.43:0/2741488482 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88201a1d90 con 0x7f88201089a0 2024-05-27T07:40:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.695+0000 7f881fe00640 1 -- 172.21.15.43:0/2741488482 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f882006a150 con 0x7f88201089a0 2024-05-27T07:40:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.699+0000 7f8817400640 1 -- 172.21.15.43:0/2741488482 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f880c042030 con 0x7f88201089a0 2024-05-27T07:40:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.700+0000 7f8817400640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f88040b2950 0x7f88040b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.701+0000 7f8817400640 1 -- 172.21.15.43:0/2741488482 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f880c103470 con 0x7f88201089a0 2024-05-27T07:40:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.701+0000 7f881ea00640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f88040b2950 0x7f88040b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.702+0000 7f881ea00640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f88040b2950 0x7f88040b4e60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8810005fd0 tx=0x7f8810005850 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.702+0000 7f8817400640 1 -- 172.21.15.43:0/2741488482 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f880c0caa20 con 0x7f88201089a0 2024-05-27T07:40:55.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.916+0000 7f881fe00640 1 -- 172.21.15.43:0/2741488482 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f88201a92e0 con 0x7f88201089a0 2024-05-27T07:40:55.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.917+0000 7f8817400640 1 -- 172.21.15.43:0/2741488482 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v163) ==== 100+0+11 (secure 0 0 0) 0x7f880c0ca320 con 0x7f88201089a0 2024-05-27T07:40:55.915 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-05-27T07:40:55.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 -- 172.21.15.43:0/2741488482 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f88040b2950 msgr2=0x7f88040b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:55.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f88040b2950 0x7f88040b4e60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8810005fd0 tx=0x7f8810005850 comp rx=0 tx=0).stop 2024-05-27T07:40:55.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 -- 172.21.15.43:0/2741488482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88201089a0 msgr2=0x7f882019cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:55.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88201089a0 0x7f882019cd10 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f880c005940 tx=0x7f880c0040e0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 -- 172.21.15.43:0/2741488482 shutdown_connections 2024-05-27T07:40:55.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f88040b2950 0x7f88040b4e60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f882010d300 0x7f88201a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88201089a0 0x7f882019cd10 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 --2- 172.21.15.43:0/2741488482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8820102a70 0x7f882019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:55.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 -- 172.21.15.43:0/2741488482 >> 172.21.15.43:0/2741488482 conn(0x7f8820077fe0 msgr2=0x7f8820101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:55.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 -- 172.21.15.43:0/2741488482 shutdown_connections 2024-05-27T07:40:55.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:55.920+0000 7f881fe00640 1 -- 172.21.15.43:0/2741488482 wait complete. 2024-05-27T07:40:55.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:55 smithi043 ceph-mon[31531]: pgmap v904: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:55.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:55 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1036996080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:40:55.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:55 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1670798540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:40:55.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-27T07:40:55.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:55 smithi179 ceph-mon[35888]: pgmap v904: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:55.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:55 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1036996080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:40:55.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:55 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1670798540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:40:56.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:55 smithi132 ceph-mon[35283]: pgmap v904: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:56.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:55 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1036996080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-27T07:40:56.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:55 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1670798540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-27T07:40:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.226+0000 7f2942400640 1 -- 172.21.15.43:0/991245501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f293c10da00 msgr2=0x7f293c10fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.226+0000 7f2942400640 1 --2- 172.21.15.43:0/991245501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f293c10da00 0x7f293c10fe20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f2924009900 tx=0x7f292402a410 comp rx=0 tx=0).stop 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.227+0000 7f2942400640 1 -- 172.21.15.43:0/991245501 shutdown_connections 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.227+0000 7f2942400640 1 --2- 172.21.15.43:0/991245501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f293c110360 0x7f293c112780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.227+0000 7f2942400640 1 --2- 172.21.15.43:0/991245501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f293c10da00 0x7f293c10fe20 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.227+0000 7f2942400640 1 --2- 172.21.15.43:0/991245501 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c1027c0 0x7f293c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.227+0000 7f2942400640 1 -- 172.21.15.43:0/991245501 >> 172.21.15.43:0/991245501 conn(0x7f293c0fe030 msgr2=0x7f293c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.227+0000 7f2942400640 1 -- 172.21.15.43:0/991245501 shutdown_connections 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.227+0000 7f2942400640 1 -- 172.21.15.43:0/991245501 wait complete. 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.228+0000 7f2942400640 1 Processor -- start 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.228+0000 7f2942400640 1 -- start start 2024-05-27T07:40:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.228+0000 7f2942400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c1027c0 0x7f293c1a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.228+0000 7f2942400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f293c10da00 0x7f293c1a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.228+0000 7f2942400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f293c110360 0x7f293c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.228+0000 7f2942400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f293c115d20 con 0x7f293c10da00 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.228+0000 7f2942400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f293c115ba0 con 0x7f293c110360 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.228+0000 7f2942400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f293c115ea0 con 0x7f293c1027c0 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.229+0000 7f293be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f293c10da00 0x7f293c1a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.229+0000 7f293be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f293c10da00 0x7f293c1a2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57624/0 (socket says 172.21.15.43:57624) 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.229+0000 7f293be00640 1 -- 172.21.15.43:0/1339708109 learned_addr learned my addr 172.21.15.43:0/1339708109 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.229+0000 7f2941000640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c1027c0 0x7f293c1a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.229+0000 7f2941a00640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f293c110360 0x7f293c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.229+0000 7f293be00640 1 -- 172.21.15.43:0/1339708109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c1027c0 msgr2=0x7f293c1a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.229+0000 7f293be00640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c1027c0 0x7f293c1a24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.229+0000 7f293be00640 1 -- 172.21.15.43:0/1339708109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f293c110360 msgr2=0x7f293c19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.229+0000 7f293be00640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f293c110360 0x7f293c19c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.229+0000 7f293be00640 1 -- 172.21.15.43:0/1339708109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293c19cd90 con 0x7f293c10da00 2024-05-27T07:40:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.230+0000 7f293be00640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f293c10da00 0x7f293c1a2a20 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f292402a930 tx=0x7f2924003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.230+0000 7f2939600640 1 -- 172.21.15.43:0/1339708109 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292403f070 con 0x7f293c10da00 2024-05-27T07:40:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.230+0000 7f2939600640 1 -- 172.21.15.43:0/1339708109 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2924003e10 con 0x7f293c10da00 2024-05-27T07:40:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.230+0000 7f2939600640 1 -- 172.21.15.43:0/1339708109 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2924035030 con 0x7f293c10da00 2024-05-27T07:40:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.230+0000 7f2942400640 1 -- 172.21.15.43:0/1339708109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f293c19cfc0 con 0x7f293c10da00 2024-05-27T07:40:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.230+0000 7f2942400640 1 -- 172.21.15.43:0/1339708109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f293c19d4b0 con 0x7f293c10da00 2024-05-27T07:40:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.231+0000 7f2942400640 1 -- 172.21.15.43:0/1339708109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f293c06a830 con 0x7f293c10da00 2024-05-27T07:40:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.234+0000 7f2939600640 1 -- 172.21.15.43:0/1339708109 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f2924042070 con 0x7f293c10da00 2024-05-27T07:40:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.236+0000 7f2939600640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29200b2950 0x7f29200b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.236+0000 7f2939600640 1 -- 172.21.15.43:0/1339708109 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f2924103710 con 0x7f293c10da00 2024-05-27T07:40:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.236+0000 7f2941000640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29200b2950 0x7f29200b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.237+0000 7f2941000640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29200b2950 0x7f29200b4e60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2934005fd0 tx=0x7f2934009870 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:56.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.243+0000 7f2939600640 1 -- 172.21.15.43:0/1339708109 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f29240cacc0 con 0x7f293c10da00 2024-05-27T07:40:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.458+0000 7f2942400640 1 -- 172.21.15.43:0/1339708109 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f293c069d00 con 0x7f293c10da00 2024-05-27T07:40:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.458+0000 7f2939600640 1 -- 172.21.15.43:0/1339708109 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v163) ==== 103+0+11 (secure 0 0 0) 0x7f293c069d00 con 0x7f293c10da00 2024-05-27T07:40:56.456 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-05-27T07:40:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 -- 172.21.15.43:0/1339708109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29200b2950 msgr2=0x7f29200b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29200b2950 0x7f29200b4e60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2934005fd0 tx=0x7f2934009870 comp rx=0 tx=0).stop 2024-05-27T07:40:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 -- 172.21.15.43:0/1339708109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f293c10da00 msgr2=0x7f293c1a2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f293c10da00 0x7f293c1a2a20 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f292402a930 tx=0x7f2924003c20 comp rx=0 tx=0).stop 2024-05-27T07:40:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 -- 172.21.15.43:0/1339708109 shutdown_connections 2024-05-27T07:40:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f29200b2950 0x7f29200b4e60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f293c110360 0x7f293c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f293c10da00 0x7f293c1a2a20 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 --2- 172.21.15.43:0/1339708109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c1027c0 0x7f293c1a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 -- 172.21.15.43:0/1339708109 >> 172.21.15.43:0/1339708109 conn(0x7f293c0fe030 msgr2=0x7f293c100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.461+0000 7f2942400640 1 -- 172.21.15.43:0/1339708109 shutdown_connections 2024-05-27T07:40:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.462+0000 7f2942400640 1 -- 172.21.15.43:0/1339708109 wait complete. 2024-05-27T07:40:56.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-27T07:40:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.754+0000 7f227b200640 1 -- 172.21.15.43:0/2178525631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2274104390 msgr2=0x7f2274110760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.754+0000 7f227b200640 1 --2- 172.21.15.43:0/2178525631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2274104390 0x7f2274110760 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f226c009ca0 tx=0x7f226c02a8b0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.754+0000 7f227b200640 1 -- 172.21.15.43:0/2178525631 shutdown_connections 2024-05-27T07:40:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.754+0000 7f227b200640 1 --2- 172.21.15.43:0/2178525631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f227410a0e0 0x7f2274103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.754+0000 7f227b200640 1 --2- 172.21.15.43:0/2178525631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2274104390 0x7f2274110760 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.754+0000 7f227b200640 1 --2- 172.21.15.43:0/2178525631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2274109c40 0x7f22741037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.754+0000 7f227b200640 1 -- 172.21.15.43:0/2178525631 >> 172.21.15.43:0/2178525631 conn(0x7f22740ff380 msgr2=0x7f22741017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.754+0000 7f227b200640 1 -- 172.21.15.43:0/2178525631 shutdown_connections 2024-05-27T07:40:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.754+0000 7f227b200640 1 -- 172.21.15.43:0/2178525631 wait complete. 2024-05-27T07:40:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.755+0000 7f227b200640 1 Processor -- start 2024-05-27T07:40:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.755+0000 7f227b200640 1 -- start start 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.756+0000 7f227b200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2274104390 0x7f22741a2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.756+0000 7f227b200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2274109c40 0x7f22741a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.756+0000 7f227b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f227410a0e0 0x7f227419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.756+0000 7f227b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2274113ce0 con 0x7f227410a0e0 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.756+0000 7f227b200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2274113b60 con 0x7f2274109c40 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.756+0000 7f227b200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2274113e60 con 0x7f2274104390 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.756+0000 7f2279200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f227410a0e0 0x7f227419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.756+0000 7f2279200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f227410a0e0 0x7f227419c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57650/0 (socket says 172.21.15.43:57650) 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.756+0000 7f2279200640 1 -- 172.21.15.43:0/118277173 learned_addr learned my addr 172.21.15.43:0/118277173 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.756+0000 7f2273e00640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2274104390 0x7f22741a2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f226be00640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2274109c40 0x7f22741a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f2279200640 1 -- 172.21.15.43:0/118277173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2274104390 msgr2=0x7f22741a2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f2279200640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2274104390 0x7f22741a2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f2279200640 1 -- 172.21.15.43:0/118277173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2274109c40 msgr2=0x7f22741a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f2279200640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2274109c40 0x7f22741a2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f2279200640 1 -- 172.21.15.43:0/118277173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227419cd80 con 0x7f227410a0e0 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f2273e00640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2274104390 0x7f22741a2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f2279200640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f227410a0e0 0x7f227419c740 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f226c005cc0 tx=0x7f226c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f2271600640 1 -- 172.21.15.43:0/118277173 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c004450 con 0x7f227410a0e0 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f2271600640 1 -- 172.21.15.43:0/118277173 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f226c040070 con 0x7f227410a0e0 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f2271600640 1 -- 172.21.15.43:0/118277173 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c045c80 con 0x7f227410a0e0 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.757+0000 7f227b200640 1 -- 172.21.15.43:0/118277173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227419d010 con 0x7f227410a0e0 2024-05-27T07:40:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.758+0000 7f227b200640 1 -- 172.21.15.43:0/118277173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f227419d560 con 0x7f227410a0e0 2024-05-27T07:40:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.759+0000 7f227b200640 1 -- 172.21.15.43:0/118277173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f227406a2b0 con 0x7f227410a0e0 2024-05-27T07:40:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.761+0000 7f2271600640 1 -- 172.21.15.43:0/118277173 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f226c0045f0 con 0x7f227410a0e0 2024-05-27T07:40:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.763+0000 7f2271600640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f22580b2960 0x7f22580b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:40:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.763+0000 7f2271600640 1 -- 172.21.15.43:0/118277173 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f226c1040e0 con 0x7f227410a0e0 2024-05-27T07:40:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.763+0000 7f2273e00640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f22580b2960 0x7f22580b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:40:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.764+0000 7f2273e00640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f22580b2960 0x7f22580b4e70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f227419dbb0 tx=0x7f2264005db0 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:40:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:56.765+0000 7f2271600640 1 -- 172.21.15.43:0/118277173 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f226c0cb7a0 con 0x7f227410a0e0 2024-05-27T07:40:56.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:56 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2741488482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:40:56.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:56 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/1339708109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:40:56.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:56 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2741488482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:40:56.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:56 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/1339708109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:40:57.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.033+0000 7f227b200640 1 -- 172.21.15.43:0/118277173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f22741a93f0 con 0x7f227410a0e0 2024-05-27T07:40:57.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.068+0000 7f2271600640 1 -- 172.21.15.43:0/118277173 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+816172 (secure 0 0 0) 0x7f226c0cb0a0 con 0x7f227410a0e0 2024-05-27T07:40:57.073 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-27T07:40:57.074 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-27T07:40:57.074 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 26, 2024-05-27T07:40:57.074 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.074 INFO:teuthology.orchestra.run.smithi043.stdout: "active_gid": 35033, 2024-05-27T07:40:57.074 INFO:teuthology.orchestra.run.smithi043.stdout: "active_name": "x", 2024-05-27T07:40:57.074 INFO:teuthology.orchestra.run.smithi043.stdout: "active_addrs": { 2024-05-27T07:40:57.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-05-27T07:40:57.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6800", 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2368318638 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6801", 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2368318638 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: "active_addr": "172.21.15.43:6801/2368318638", 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: "active_change": "2024-05-27T07:13:54.391867+0000", 2024-05-27T07:40:57.075 INFO:teuthology.orchestra.run.smithi043.stdout: "active_mgr_features": 4540701547738038271, 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "standbys": [ 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 25069, 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "z", 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_features": 4540701547738038271, 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "available_modules": [ 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "alerts", 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.076 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.078 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.079 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.080 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.081 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.082 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_destination": { 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_destination", 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.083 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email address to send alerts to", 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_from_name": { 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_from_name", 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "Ceph", 2024-05-27T07:40:57.084 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email From: name", 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_host": { 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_host", 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.085 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP server", 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_password": { 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_password", 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.086 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Password to authenticate with", 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_port": { 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_port", 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.087 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "465", 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP port", 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_sender": { 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_sender", 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.088 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP envelope sender", 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_ssl": { 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_ssl", 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.089 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T07:40:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_user": { 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_user", 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "User to authenticate as", 2024-05-27T07:40:57.091 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.092 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "balancer", 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.093 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.094 INFO:teuthology.orchestra.run.smithi043.stdout: "active": { 2024-05-27T07:40:57.094 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "active", 2024-05-27T07:40:57.094 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.094 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.094 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.094 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.094 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.094 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.094 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.094 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_time": { 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_time", 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0000", 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.095 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.096 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T07:40:57.096 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:40:57.096 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.096 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.096 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_weekday": { 2024-05-27T07:40:57.096 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_weekday", 2024-05-27T07:40:57.096 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:57.096 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.096 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_max_iterations": { 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_max_iterations", 2024-05-27T07:40:57.097 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:57.098 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.098 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.098 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "25", 2024-05-27T07:40:57.098 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:57.098 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "250", 2024-05-27T07:40:57.098 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.098 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T07:40:57.098 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.098 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.098 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_metrics": { 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_metrics", 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.099 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_step": { 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_step", 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.5", 2024-05-27T07:40:57.100 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0.001", 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "0.999", 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "aggressiveness of optimization", 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: "end_time": { 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_time", 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.101 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.102 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.102 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2359", 2024-05-27T07:40:57.102 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.102 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.102 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.102 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T07:40:57.102 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:40:57.102 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.102 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.103 INFO:teuthology.orchestra.run.smithi043.stdout: "end_weekday": { 2024-05-27T07:40:57.103 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_weekday", 2024-05-27T07:40:57.103 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:57.103 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.103 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.103 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.103 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-05-27T07:40:57.103 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-05-27T07:40:57.103 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.103 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T07:40:57.104 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:40:57.104 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.104 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.104 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.104 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.104 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.104 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.104 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.104 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.105 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.106 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.107 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.108 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.109 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "min_score": { 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_score", 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.110 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "upmap", 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-compat", 2024-05-27T07:40:57.111 INFO:teuthology.orchestra.run.smithi043.stdout: "none", 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "read", 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap", 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap-read" 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Balancer mode", 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_ids": { 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_ids", 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.112 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.113 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.114 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_deviation": { 2024-05-27T07:40:57.115 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_deviation", 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_optimizations": { 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_optimizations", 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm", 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_down_multiplier": { 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_down_multiplier", 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_refresh_rate": { 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_refresh_rate", 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "20", 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_starting_port": { 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_starting_port", 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4721", 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_ptrace": { 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_ptrace", 2024-05-27T07:40:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_interval": { 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_interval", 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_memory_target_ratio": { 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.7", 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm_log_destination": { 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm_log_destination", 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "file", 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "file,syslog", 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "syslog" 2024-05-27T07:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "cgroups_split": { 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cgroups_split", 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T07:40:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "config_checks_enabled": { 2024-05-27T07:40:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_checks_enabled", 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "config_dashboard": { 2024-05-27T07:40:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_dashboard", 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_alertmanager": { 2024-05-27T07:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_alertmanager", 2024-05-27T07:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T07:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_base": { 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_base", 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T07:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Container image name, without the tag", 2024-05-27T07:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_elasticsearch": { 2024-05-27T07:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_elasticsearch", 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "elasticsearch container image", 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_grafana": { 2024-05-27T07:40:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_grafana", 2024-05-27T07:40:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T07:40:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_haproxy": { 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_haproxy", 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T07:40:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "HAproxy container image", 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_agent": { 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_agent", 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.136 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.137 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T07:40:57.137 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.137 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.137 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.137 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger agent container image", 2024-05-27T07:40:57.137 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.137 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.137 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.137 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_collector": { 2024-05-27T07:40:57.138 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_collector", 2024-05-27T07:40:57.138 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.138 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.138 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.138 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T07:40:57.138 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.138 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.138 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.138 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger collector container image", 2024-05-27T07:40:57.138 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_query": { 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_query", 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.139 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger query container image", 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_keepalived": { 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_keepalived", 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.140 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Keepalived container image", 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_loki": { 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_loki", 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T07:40:57.141 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Loki container image", 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_node_exporter": { 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_node_exporter", 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.142 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.143 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.143 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T07:40:57.143 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.143 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.143 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.143 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:57.143 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.143 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.143 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.144 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.144 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_nvmeof": { 2024-05-27T07:40:57.144 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_nvmeof", 2024-05-27T07:40:57.144 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.144 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.144 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.144 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T07:40:57.144 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.144 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Nvme-of container image", 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_prometheus": { 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_prometheus", 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T07:40:57.145 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_promtail": { 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_promtail", 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T07:40:57.146 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Promtail container image", 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_samba": { 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_samba", 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.147 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Samba/SMB container image", 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_snmp_gateway": { 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_snmp_gateway", 2024-05-27T07:40:57.148 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SNMP Gateway container image", 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.149 INFO:teuthology.orchestra.run.smithi043.stdout: "container_init": { 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_init", 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "daemon_cache_timeout": { 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "daemon_cache_timeout", 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.151 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "default_cephadm_command_timeout": { 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "900", 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.152 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "default_registry": { 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_registry", 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io", 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.153 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "device_cache_timeout": { 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_cache_timeout", 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1800", 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.154 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache device inventory", 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "device_enhanced_scan": { 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_enhanced_scan", 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.155 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "facts_cache_timeout": { 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "facts_cache_timeout", 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.156 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache host facts data", 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "grafana_dashboards_path": { 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "grafana_dashboards_path", 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.157 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "host_check_interval": { 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "host_check_interval", 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.158 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to perform a host check", 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "hw_monitoring": { 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hw_monitoring", 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.159 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "inventory_list_all": { 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "inventory_list_all", 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.160 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.161 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "log_refresh_metadata": { 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_refresh_metadata", 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.164 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.165 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.166 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "*", 2024-05-27T07:40:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T07:40:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "max_count_per_host": { 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_count_per_host", 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of daemons per service per host", 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd_draining_count": { 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_osd_draining_count", 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.171 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: "migration_current": { 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "migration_current", 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.173 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.173 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.173 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "internal - do not modify", 2024-05-27T07:40:57.173 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.173 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.173 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.173 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-05-27T07:40:57.173 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-05-27T07:40:57.173 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "root", 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm-package", 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: "root" 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T07:40:57.174 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.175 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.175 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.175 INFO:teuthology.orchestra.run.smithi043.stdout: "oob_default_addr": { 2024-05-27T07:40:57.175 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "oob_default_addr", 2024-05-27T07:40:57.175 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.175 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.175 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.175 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "169.254.1.1", 2024-05-27T07:40:57.175 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.176 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.176 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.176 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T07:40:57.176 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.176 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.176 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.176 INFO:teuthology.orchestra.run.smithi043.stdout: "prometheus_alerts_path": { 2024-05-27T07:40:57.176 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus_alerts_path", 2024-05-27T07:40:57.176 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.177 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.177 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.177 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T07:40:57.177 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.177 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.177 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.177 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T07:40:57.177 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.177 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.177 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.178 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_insecure": { 2024-05-27T07:40:57.178 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_insecure", 2024-05-27T07:40:57.178 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.178 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.178 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.178 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.178 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.178 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.178 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_password": { 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_password", 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T07:40:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_url": { 2024-05-27T07:40:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_url", 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_username": { 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_username", 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "secure_monitoring_stack": { 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "secure_monitoring_stack", 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "service_discovery_port": { 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "service_discovery_port", 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8765", 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "cephadm service discovery port", 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "ssh_config_file": { 2024-05-27T07:40:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssh_config_file", 2024-05-27T07:40:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T07:40:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "use_agent": { 2024-05-27T07:40:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_agent", 2024-05-27T07:40:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "use_repo_digest": { 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_repo_digest", 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_failed_host_check": { 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_failed_host_check", 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_daemons": { 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_daemons", 2024-05-27T07:40:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_hosts": { 2024-05-27T07:40:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_hosts", 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crash", 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "retain_interval": { 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retain_interval", 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "31536000", 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_recent_interval": { 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_recent_interval", 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1209600", 2024-05-27T07:40:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dashboard", 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T07:40:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T07:40:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_ENABLED": { 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.205 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.206 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.207 INFO:teuthology.orchestra.run.smithi043.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.208 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T07:40:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T07:40:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.209 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T07:40:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T07:40:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.210 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T07:40:57.211 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T07:40:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.212 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T07:40:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T07:40:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.213 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T07:40:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T07:40:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.214 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.215 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T07:40:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T07:40:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.216 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.217 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.218 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.219 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.220 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_URL": { 2024-05-27T07:40:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_URL", 2024-05-27T07:40:57.221 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.223 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.224 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.225 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.226 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.227 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "[]", 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.228 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "{}", 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.229 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.230 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.231 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.232 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.233 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.234 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.235 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.236 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.237 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.238 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.239 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T07:40:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T07:40:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T07:40:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.241 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.242 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.243 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.246 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8", 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.247 INFO:teuthology.orchestra.run.smithi043.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T07:40:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T07:40:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "45", 2024-05-27T07:40:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.248 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.249 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T07:40:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T07:40:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.250 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T07:40:57.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T07:40:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.252 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T07:40:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T07:40:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.253 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T07:40:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T07:40:57.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.255 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T07:40:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T07:40:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.256 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T07:40:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T07:40:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.257 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.258 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T07:40:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T07:40:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.259 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "cross_origin_url": { 2024-05-27T07:40:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cross_origin_url", 2024-05-27T07:40:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.260 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "crt_file": { 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crt_file", 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.262 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "debug": { 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "debug", 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable/disable debug options", 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.263 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "jwt_token_ttl": { 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "jwt_token_ttl", 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "28800", 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.264 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.265 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.266 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.267 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.268 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.269 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.270 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "motd": { 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "motd", 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.271 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "The message of the day", 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_resolve_ip_addr": { 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.272 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.273 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8080", 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.274 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.275 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl_server_port": { 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl_server_port", 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8443", 2024-05-27T07:40:57.277 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.278 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "redirect", 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect" 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.279 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "url_prefix": { 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url_prefix", 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.280 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "devicehealth", 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_monitoring": { 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_monitoring", 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "monitor device health metrics", 2024-05-27T07:40:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "mark_out_threshold": { 2024-05-27T07:40:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mark_out_threshold", 2024-05-27T07:40:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2419200", 2024-05-27T07:40:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": { 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_name", 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "device_health_metrics", 2024-05-27T07:40:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T07:40:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "retention_period": { 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retention_period", 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15552000", 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain device health metrics", 2024-05-27T07:40:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_frequency": { 2024-05-27T07:40:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_frequency", 2024-05-27T07:40:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-05-27T07:40:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T07:40:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "self_heal": { 2024-05-27T07:40:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "self_heal", 2024-05-27T07:40:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T07:40:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T07:40:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_threshold": { 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_threshold", 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7257600", 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "diskprediction_local", 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.304 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.305 INFO:teuthology.orchestra.run.smithi043.stdout: "predict_interval": { 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predict_interval", 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.306 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "predictor_model": { 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predictor_model", 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "prophetstor", 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.307 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.308 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.309 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "influx", 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": false, 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "influxdb python module not found", 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: "batch_size": { 2024-05-27T07:40:57.310 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "batch_size", 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5000", 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "database": { 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "database", 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "ceph", 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": { 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hostname", 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server hostname", 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.313 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "30", 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "5", 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.314 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.315 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.317 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.317 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.317 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.317 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.317 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.317 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.317 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.317 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.317 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.317 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.318 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.318 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.318 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.318 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.318 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.318 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.318 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.318 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.319 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.320 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.320 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.320 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.320 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.320 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.320 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.320 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.320 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.320 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.321 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.321 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.321 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.321 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.321 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.321 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.321 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.321 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.321 INFO:teuthology.orchestra.run.smithi043.stdout: "password": { 2024-05-27T07:40:57.322 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "password", 2024-05-27T07:40:57.322 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.322 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.322 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.322 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.322 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.322 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.322 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.322 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "password of InfluxDB server user", 2024-05-27T07:40:57.322 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: "port": { 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "port", 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8086", 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.323 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.324 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.324 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server port", 2024-05-27T07:40:57.324 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.324 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.324 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.324 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.324 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.324 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-05-27T07:40:57.325 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.326 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.326 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.326 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "false", 2024-05-27T07:40:57.326 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.326 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.326 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.326 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:40:57.326 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.326 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.326 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.327 INFO:teuthology.orchestra.run.smithi043.stdout: "threads": { 2024-05-27T07:40:57.327 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threads", 2024-05-27T07:40:57.327 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.327 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.327 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.327 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:57.327 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:57.327 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "32", 2024-05-27T07:40:57.327 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "username": { 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "username", 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.328 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.329 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.329 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "username of InfluxDB server user", 2024-05-27T07:40:57.329 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.329 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.329 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.329 INFO:teuthology.orchestra.run.smithi043.stdout: "verify_ssl": { 2024-05-27T07:40:57.329 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "verify_ssl", 2024-05-27T07:40:57.329 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.329 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "true", 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.331 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "insights", 2024-05-27T07:40:57.331 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.331 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.331 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.331 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.331 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.331 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.331 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.331 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.332 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.333 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.333 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.333 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.333 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.333 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.333 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.333 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.333 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.334 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.334 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.334 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.334 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.334 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.334 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.334 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.334 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.334 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.335 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.336 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.336 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.336 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.336 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.336 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.336 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.336 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.336 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.337 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.337 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.337 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.337 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.338 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.338 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.338 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.338 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.338 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.338 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.338 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.338 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.338 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.339 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.339 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.339 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.339 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.339 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.339 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.339 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.339 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.339 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "iostat", 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.340 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.341 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.342 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.343 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.344 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.345 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.346 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "k8sevents", 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_event_retention_days": { 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ceph_event_retention_days", 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.347 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7", 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "config_check_secs": { 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_check_secs", 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.348 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "10", 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.349 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.350 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.351 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.352 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.353 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.354 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.355 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "localpool", 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: "failure_domain": { 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "failure_domain", 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.356 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "host", 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "failure domain for any created local pool", 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.357 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.358 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.359 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.360 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.361 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.362 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.362 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.362 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.362 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.362 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.362 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.362 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.362 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": { 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_size", 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default min_size for any created local pool", 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.363 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "num_rep": { 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "num_rep", 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3", 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default replica count for any created local pool", 2024-05-27T07:40:57.364 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.365 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.365 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.365 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": { 2024-05-27T07:40:57.365 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_num", 2024-05-27T07:40:57.365 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.365 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.365 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.365 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "128", 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "prefix": { 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prefix", 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.366 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.367 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.367 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.367 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.367 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.367 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.367 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name prefix for any created local pool", 2024-05-27T07:40:57.367 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.367 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.367 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.368 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.369 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.369 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.369 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.369 INFO:teuthology.orchestra.run.smithi043.stdout: "subtree": { 2024-05-27T07:40:57.369 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subtree", 2024-05-27T07:40:57.369 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.369 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.369 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.369 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "rack", 2024-05-27T07:40:57.370 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.370 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.370 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.370 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T07:40:57.370 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T07:40:57.370 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.370 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.370 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.370 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mds_autoscaler", 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirroring", 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "nfs", 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-05-27T07:40:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "fail_fs": { 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "fail_fs", 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.394 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.395 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.396 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.396 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.396 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.396 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.396 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.396 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.396 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.396 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.396 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.397 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.398 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator": { 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.399 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm", 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "rook", 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "test_orchestrator" 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Orchestrator backend", 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.400 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.401 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.401 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.401 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.401 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.401 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.401 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.401 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.401 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_perf_query", 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.402 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.403 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.404 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.404 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.404 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.404 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.404 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.404 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.404 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.404 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.404 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.404 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.405 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.405 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.405 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.405 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.405 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.405 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.405 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.405 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.405 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.406 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.406 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.406 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.406 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.406 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.406 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.406 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.406 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.407 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.407 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.407 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.407 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.407 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.407 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.407 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.407 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.407 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.407 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.408 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.408 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.408 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.408 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.408 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.408 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.408 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.408 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.409 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.409 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.409 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.409 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.409 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.409 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.409 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.409 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.409 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.410 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.410 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.410 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.410 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.410 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.410 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.410 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.410 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.410 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_support", 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.411 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.412 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.413 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.413 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.413 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.413 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.413 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.413 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.413 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.413 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.413 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.414 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.415 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.416 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.417 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_autoscaler", 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.419 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.420 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.421 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.422 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.423 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.424 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.425 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "threshold": { 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threshold", 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1.0", 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.426 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "scaling threshold", 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "progress", 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_pg_recovery_event": { 2024-05-27T07:40:57.427 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_pg_recovery_event", 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.428 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.429 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.430 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.431 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.432 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.433 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.433 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.433 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.433 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.433 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.433 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.433 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.433 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.433 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.434 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "max_completed_events": { 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_completed_events", 2024-05-27T07:40:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "50", 2024-05-27T07:40:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "number of past completed events to remember", 2024-05-27T07:40:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long the module is going to sleep", 2024-05-27T07:40:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus", 2024-05-27T07:40:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "cache": { 2024-05-27T07:40:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cache", 2024-05-27T07:40:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "exclude_perf_counters": { 2024-05-27T07:40:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "exclude_perf_counters", 2024-05-27T07:40:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools": { 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools", 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "300", 2024-05-27T07:40:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_interval": { 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_interval", 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15.0", 2024-05-27T07:40:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "9283", 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "stale_cache_strategy": { 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stale_cache_strategy", 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "log", 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "default", 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "default", 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "error" 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-05-27T07:40:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_support", 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_snap_create": { 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_snap_create", 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "mirror_snapshot_schedule": { 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "trash_purge_schedule": { 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "trash_purge_schedule", 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "restful", 2024-05-27T07:40:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_auth": { 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_auth", 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.478 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.478 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.478 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.478 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.478 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.478 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.478 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.478 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-05-27T07:40:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-05-27T07:40:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.480 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.480 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.480 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.480 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.480 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.480 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.480 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.480 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.480 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.481 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-05-27T07:40:57.481 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-05-27T07:40:57.481 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.481 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.481 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.481 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.481 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.481 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.481 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.481 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.482 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.482 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.482 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.482 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.482 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.482 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.482 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.482 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.483 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rgw", 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.484 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.485 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.486 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.488 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.489 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rook", 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.490 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.491 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.492 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.493 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.496 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": { 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "storage_class", 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "local", 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.497 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "selftest", 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.498 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.499 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.500 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.501 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.502 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "roption1": { 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption1", 2024-05-27T07:40:57.503 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.504 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.504 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.504 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.504 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.504 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.504 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.504 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.504 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.504 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "roption2": { 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption2", 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "xyz", 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.505 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.506 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.506 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption1": { 2024-05-27T07:40:57.506 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption1", 2024-05-27T07:40:57.506 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.506 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.506 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.506 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.506 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption2": { 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption2", 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.507 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.508 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.508 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.508 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.508 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.508 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.508 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.508 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.508 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.508 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption3": { 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption3", 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.509 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption4": { 2024-05-27T07:40:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption4", 2024-05-27T07:40:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.510 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption5": { 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption5", 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.511 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption6": { 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption6", 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.512 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption7": { 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption7", 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.513 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "42", 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.514 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "testkey": { 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testkey", 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.515 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "testlkey": { 2024-05-27T07:40:57.516 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testlkey", 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.517 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.518 INFO:teuthology.orchestra.run.smithi043.stdout: "testnewline": { 2024-05-27T07:40:57.518 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testnewline", 2024-05-27T07:40:57.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.518 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.518 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.518 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.518 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.518 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.519 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.519 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.519 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.519 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.519 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.519 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smb", 2024-05-27T07:40:57.520 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.520 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.520 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.520 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.520 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.520 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.520 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.521 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.521 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.521 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.521 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.521 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.521 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.521 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.521 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.521 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.521 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.522 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.522 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.522 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.522 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.522 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.522 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.522 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.522 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.523 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.524 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.525 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.526 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "update_orchestration": { 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "update_orchestration", 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.527 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snap_schedule", 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.528 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_m_granularity": { 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_m_granularity", 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.529 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "dump_on_update": { 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dump_on_update", 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "dump database to debug log on update", 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.530 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.531 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.532 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.533 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.534 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.535 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stats", 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.537 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.538 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.538 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.538 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.538 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.538 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.538 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.538 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.538 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.538 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.539 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.540 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.541 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.541 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.541 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.541 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.541 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.541 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.541 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.541 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.541 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.542 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.542 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.542 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.542 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.542 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.542 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.542 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.542 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.542 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.543 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.543 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.543 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.543 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.543 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.543 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.543 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.543 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.543 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.543 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.544 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.544 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.544 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.544 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.544 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.544 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.544 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.544 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.544 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "status", 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.545 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.546 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.547 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.549 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.550 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.550 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.550 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.550 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.550 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.550 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.550 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.550 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.550 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.552 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.552 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.552 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.552 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.552 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.552 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.552 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.552 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.552 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.552 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.553 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.553 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.553 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.553 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.553 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.553 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telegraf", 2024-05-27T07:40:57.553 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.553 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.554 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.554 INFO:teuthology.orchestra.run.smithi043.stdout: "address": { 2024-05-27T07:40:57.554 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "address", 2024-05-27T07:40:57.554 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.554 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.554 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.554 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T07:40:57.554 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.554 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.554 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15", 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.555 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.556 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.558 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.559 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.560 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.562 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telemetry", 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_basic": { 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_basic", 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_crash": { 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_crash", 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.564 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_device": { 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_device", 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.565 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_ident": { 2024-05-27T07:40:57.566 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_ident", 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_perf": { 2024-05-27T07:40:57.567 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_perf", 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "contact": { 2024-05-27T07:40:57.568 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "contact", 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "description": { 2024-05-27T07:40:57.569 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "description", 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "device_url": { 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_url", 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.572 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "24", 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "8", 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_opt_revision": { 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "last_opt_revision", 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1", 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.574 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard": { 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard", 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.575 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard_description": { 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard_description", 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.583 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.583 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.583 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.583 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.583 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.583 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.583 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.583 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: "organization": { 2024-05-27T07:40:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "organization", 2024-05-27T07:40:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: "proxy": { 2024-05-27T07:40:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "proxy", 2024-05-27T07:40:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "url": { 2024-05-27T07:40:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url", 2024-05-27T07:40:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T07:40:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "test_orchestrator", 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "volumes", 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_clones": { 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_clones", 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4", 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "periodic_async_work": { 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "periodic_async_work", 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Periodically check for async work", 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_delay": { 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_delay", 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_no_wait": { 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix", 2024-05-27T07:40:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "discovery_interval": { 2024-05-27T07:40:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "discovery_interval", 2024-05-27T07:40:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "100", 2024-05-27T07:40:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "identifier": { 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "identifier", 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_host": { 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_host", 2024-05-27T07:40:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_port": { 2024-05-27T07:40:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_port", 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10051", 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_sender": { 2024-05-27T07:40:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_sender", 2024-05-27T07:40:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T07:40:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 35030, 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "y", 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_features": 4540701547738038271, 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "available_modules": [ 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "alerts", 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.629 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.630 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.631 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.632 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_destination": { 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_destination", 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.633 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email address to send alerts to", 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_from_name": { 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_from_name", 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "Ceph", 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.634 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email From: name", 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_host": { 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_host", 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.635 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP server", 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_password": { 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_password", 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.636 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Password to authenticate with", 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_port": { 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_port", 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "465", 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.637 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP port", 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_sender": { 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_sender", 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.638 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP envelope sender", 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_ssl": { 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_ssl", 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.639 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_user": { 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_user", 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.640 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "User to authenticate as", 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.641 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.642 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "balancer", 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "active": { 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "active", 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.643 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_time": { 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_time", 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0000", 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.644 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_weekday": { 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_weekday", 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.645 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_max_iterations": { 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_max_iterations", 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "25", 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "250", 2024-05-27T07:40:57.646 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_metrics": { 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_metrics", 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.647 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_step": { 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_step", 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.5", 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0.001", 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "0.999", 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "aggressiveness of optimization", 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: "end_time": { 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_time", 2024-05-27T07:40:57.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2359", 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "end_weekday": { 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_weekday", 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T07:40:57.651 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:40:57.652 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.652 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.652 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.652 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.652 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.652 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.652 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.652 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.652 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.653 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.653 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.653 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.653 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.653 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.653 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.653 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.653 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.653 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.654 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.654 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.654 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.654 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.654 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.654 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.654 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.654 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.655 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.655 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.655 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.655 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.655 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.655 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.655 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.655 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.655 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.656 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.657 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.657 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.657 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.657 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.657 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.657 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.657 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.657 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.657 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.657 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.658 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.658 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.658 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.658 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.658 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.658 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.658 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.658 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.658 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.659 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.659 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.659 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.659 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.659 INFO:teuthology.orchestra.run.smithi043.stdout: "min_score": { 2024-05-27T07:40:57.659 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_score", 2024-05-27T07:40:57.659 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.659 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.659 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.660 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.660 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.660 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.660 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.660 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T07:40:57.660 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.660 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.660 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.661 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-05-27T07:40:57.661 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-05-27T07:40:57.661 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.661 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.661 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.661 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "upmap", 2024-05-27T07:40:57.661 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.661 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.661 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.661 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-compat", 2024-05-27T07:40:57.662 INFO:teuthology.orchestra.run.smithi043.stdout: "none", 2024-05-27T07:40:57.662 INFO:teuthology.orchestra.run.smithi043.stdout: "read", 2024-05-27T07:40:57.662 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap", 2024-05-27T07:40:57.662 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap-read" 2024-05-27T07:40:57.662 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.662 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Balancer mode", 2024-05-27T07:40:57.662 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.662 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.662 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.663 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_ids": { 2024-05-27T07:40:57.663 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_ids", 2024-05-27T07:40:57.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.663 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.663 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.663 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.663 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.663 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.663 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.664 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T07:40:57.664 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.664 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.664 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.664 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:57.664 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:57.664 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.664 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.665 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.665 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:57.665 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.665 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.665 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.665 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T07:40:57.665 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.665 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.666 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.666 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.666 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.666 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.666 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.666 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.666 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.666 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_deviation": { 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_deviation", 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.667 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_optimizations": { 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_optimizations", 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.668 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm", 2024-05-27T07:40:57.669 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_down_multiplier": { 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_down_multiplier", 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.670 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_refresh_rate": { 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_refresh_rate", 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "20", 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.671 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_starting_port": { 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_starting_port", 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4721", 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.672 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_ptrace": { 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_ptrace", 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.673 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_interval": { 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_interval", 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.674 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_memory_target_ratio": { 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.7", 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.675 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm_log_destination": { 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm_log_destination", 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.676 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "file", 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "file,syslog", 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "syslog" 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "cgroups_split": { 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cgroups_split", 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.677 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "config_checks_enabled": { 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_checks_enabled", 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.678 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "config_dashboard": { 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_dashboard", 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.679 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_alertmanager": { 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_alertmanager", 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.680 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_base": { 2024-05-27T07:40:57.681 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_base", 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Container image name, without the tag", 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_elasticsearch": { 2024-05-27T07:40:57.682 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_elasticsearch", 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "elasticsearch container image", 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.683 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_grafana": { 2024-05-27T07:40:57.684 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_grafana", 2024-05-27T07:40:57.684 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.684 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.684 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.684 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T07:40:57.684 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.684 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.684 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.684 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_haproxy": { 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_haproxy", 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.685 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "HAproxy container image", 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_agent": { 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_agent", 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.686 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.687 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T07:40:57.687 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.687 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.687 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.687 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger agent container image", 2024-05-27T07:40:57.687 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.687 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.687 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.687 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_collector": { 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_collector", 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger collector container image", 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.688 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.689 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.689 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_query": { 2024-05-27T07:40:57.689 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_query", 2024-05-27T07:40:57.689 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.689 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.689 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.689 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T07:40:57.689 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.689 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.690 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.690 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger query container image", 2024-05-27T07:40:57.690 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.690 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.690 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.690 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_keepalived": { 2024-05-27T07:40:57.690 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_keepalived", 2024-05-27T07:40:57.690 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Keepalived container image", 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.692 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_loki": { 2024-05-27T07:40:57.692 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_loki", 2024-05-27T07:40:57.692 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.692 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.692 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.692 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T07:40:57.692 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.692 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.692 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.692 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Loki container image", 2024-05-27T07:40:57.693 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.693 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.693 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.693 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_node_exporter": { 2024-05-27T07:40:57.693 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_node_exporter", 2024-05-27T07:40:57.693 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.693 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.693 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.693 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T07:40:57.694 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.694 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.694 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.694 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:57.694 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.694 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.694 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.694 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_nvmeof": { 2024-05-27T07:40:57.694 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_nvmeof", 2024-05-27T07:40:57.695 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.695 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.695 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.695 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T07:40:57.695 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.695 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.695 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.695 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Nvme-of container image", 2024-05-27T07:40:57.695 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.695 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_prometheus": { 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_prometheus", 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.696 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_promtail": { 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_promtail", 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.697 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Promtail container image", 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_samba": { 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_samba", 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.698 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Samba/SMB container image", 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_snmp_gateway": { 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_snmp_gateway", 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T07:40:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.700 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.700 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.700 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SNMP Gateway container image", 2024-05-27T07:40:57.700 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.700 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.700 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.700 INFO:teuthology.orchestra.run.smithi043.stdout: "container_init": { 2024-05-27T07:40:57.700 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_init", 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "daemon_cache_timeout": { 2024-05-27T07:40:57.701 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "daemon_cache_timeout", 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.702 INFO:teuthology.orchestra.run.smithi043.stdout: "default_cephadm_command_timeout": { 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "900", 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.703 INFO:teuthology.orchestra.run.smithi043.stdout: "default_registry": { 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_registry", 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io", 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "device_cache_timeout": { 2024-05-27T07:40:57.704 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_cache_timeout", 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1800", 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache device inventory", 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "device_enhanced_scan": { 2024-05-27T07:40:57.705 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_enhanced_scan", 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.706 INFO:teuthology.orchestra.run.smithi043.stdout: "facts_cache_timeout": { 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "facts_cache_timeout", 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache host facts data", 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "grafana_dashboards_path": { 2024-05-27T07:40:57.707 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "grafana_dashboards_path", 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.708 INFO:teuthology.orchestra.run.smithi043.stdout: "host_check_interval": { 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "host_check_interval", 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to perform a host check", 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.709 INFO:teuthology.orchestra.run.smithi043.stdout: "hw_monitoring": { 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hw_monitoring", 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.710 INFO:teuthology.orchestra.run.smithi043.stdout: "inventory_list_all": { 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "inventory_list_all", 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.712 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "log_refresh_metadata": { 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_refresh_metadata", 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.713 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.714 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.715 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.716 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.717 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.718 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "*", 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.719 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "max_count_per_host": { 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_count_per_host", 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of daemons per service per host", 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.720 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd_draining_count": { 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_osd_draining_count", 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.721 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.723 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:56 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2741488482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-27T07:40:57.723 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:56 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/1339708109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-27T07:40:57.727 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.727 INFO:teuthology.orchestra.run.smithi043.stdout: "migration_current": { 2024-05-27T07:40:57.727 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "migration_current", 2024-05-27T07:40:57.727 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.727 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.727 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.727 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.728 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.728 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.728 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.728 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "internal - do not modify", 2024-05-27T07:40:57.728 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.728 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.728 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.728 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-05-27T07:40:57.728 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-05-27T07:40:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "root", 2024-05-27T07:40:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm-package", 2024-05-27T07:40:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "root" 2024-05-27T07:40:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T07:40:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "oob_default_addr": { 2024-05-27T07:40:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "oob_default_addr", 2024-05-27T07:40:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "169.254.1.1", 2024-05-27T07:40:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T07:40:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "prometheus_alerts_path": { 2024-05-27T07:40:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus_alerts_path", 2024-05-27T07:40:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T07:40:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T07:40:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_insecure": { 2024-05-27T07:40:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_insecure", 2024-05-27T07:40:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T07:40:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_password": { 2024-05-27T07:40:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_password", 2024-05-27T07:40:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T07:40:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_url": { 2024-05-27T07:40:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_url", 2024-05-27T07:40:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_username": { 2024-05-27T07:40:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_username", 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "secure_monitoring_stack": { 2024-05-27T07:40:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "secure_monitoring_stack", 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "service_discovery_port": { 2024-05-27T07:40:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "service_discovery_port", 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8765", 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "cephadm service discovery port", 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "ssh_config_file": { 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssh_config_file", 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "use_agent": { 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_agent", 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T07:40:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "use_repo_digest": { 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_repo_digest", 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_failed_host_check": { 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_failed_host_check", 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_daemons": { 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_daemons", 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_hosts": { 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_hosts", 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crash", 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.749 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.750 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.751 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.752 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.753 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "retain_interval": { 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retain_interval", 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "31536000", 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.754 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.755 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_recent_interval": { 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_recent_interval", 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.756 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1209600", 2024-05-27T07:40:57.757 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.757 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.757 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.757 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T07:40:57.757 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.757 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.757 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.757 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.757 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dashboard", 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.758 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.759 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.759 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.759 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.759 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.759 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.759 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.759 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.760 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.761 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.761 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T07:40:57.761 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T07:40:57.761 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.761 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.761 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.761 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.761 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.761 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.762 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.762 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.762 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.762 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.762 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.762 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_ENABLED": { 2024-05-27T07:40:57.762 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T07:40:57.762 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.764 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.765 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.765 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.765 INFO:teuthology.orchestra.run.smithi043.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T07:40:57.765 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T07:40:57.765 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.765 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.765 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.765 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.765 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.766 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.766 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.766 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.766 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.766 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.766 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.766 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T07:40:57.766 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.767 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.768 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T07:40:57.768 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T07:40:57.768 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.768 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.768 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.768 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.768 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.768 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.768 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.769 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T07:40:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T07:40:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.776 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.777 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_URL": { 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_URL", 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.778 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.779 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.780 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.781 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.782 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.783 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "[]", 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.784 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "{}", 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.789 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.790 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.791 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.792 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T07:40:57.793 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.794 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.795 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.796 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.796 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.796 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T07:40:57.796 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T07:40:57.796 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.796 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.796 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.796 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.797 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.798 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.798 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T07:40:57.798 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.798 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.798 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.798 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.798 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.798 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.798 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.799 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T07:40:57.799 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T07:40:57.799 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.799 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.799 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.799 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.799 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.799 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.799 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8", 2024-05-27T07:40:57.801 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.801 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.801 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.801 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.801 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.801 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.801 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.802 INFO:teuthology.orchestra.run.smithi043.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T07:40:57.802 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T07:40:57.802 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.802 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.802 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.802 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "45", 2024-05-27T07:40:57.802 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.802 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.802 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.802 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.803 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.803 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.803 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.803 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T07:40:57.803 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T07:40:57.803 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.803 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.803 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.803 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.804 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.804 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.804 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.804 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.804 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.804 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.804 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.805 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.806 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.806 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.806 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T07:40:57.806 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T07:40:57.806 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.806 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.806 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.806 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.806 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.807 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.807 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.807 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.807 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.807 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.807 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.807 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T07:40:57.807 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T07:40:57.807 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.808 INFO:teuthology.orchestra.run.smithi043.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.809 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.811 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.812 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "cross_origin_url": { 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cross_origin_url", 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.813 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "crt_file": { 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crt_file", 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.814 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "debug": { 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "debug", 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.815 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable/disable debug options", 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "jwt_token_ttl": { 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "jwt_token_ttl", 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "28800", 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.816 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.817 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.818 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.820 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.821 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.822 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "motd": { 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "motd", 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.823 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "The message of the day", 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_resolve_ip_addr": { 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.824 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.825 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-05-27T07:40:57.826 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8080", 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.827 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.828 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.829 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-05-27T07:40:57.829 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-05-27T07:40:57.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.829 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.829 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.829 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.829 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.829 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.829 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.830 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.830 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.830 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.830 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.830 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl_server_port": { 2024-05-27T07:40:57.830 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl_server_port", 2024-05-27T07:40:57.830 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.830 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.830 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8443", 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-05-27T07:40:57.831 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-05-27T07:40:57.832 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.832 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.832 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.832 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "redirect", 2024-05-27T07:40:57.832 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.832 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.832 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.832 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.832 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect" 2024-05-27T07:40:57.832 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.833 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.833 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.833 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.833 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.833 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-05-27T07:40:57.833 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-05-27T07:40:57.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.833 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.833 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.834 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-05-27T07:40:57.834 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-05-27T07:40:57.834 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-05-27T07:40:57.834 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.834 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.834 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.834 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.834 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.835 INFO:teuthology.orchestra.run.smithi043.stdout: "url_prefix": { 2024-05-27T07:40:57.835 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url_prefix", 2024-05-27T07:40:57.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.835 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.835 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.835 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.835 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.835 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.835 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.835 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.836 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.836 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.836 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.836 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.836 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.836 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "devicehealth", 2024-05-27T07:40:57.836 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.837 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.837 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.837 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_monitoring": { 2024-05-27T07:40:57.837 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_monitoring", 2024-05-27T07:40:57.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.837 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.837 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.837 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.837 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.837 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.838 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.838 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "monitor device health metrics", 2024-05-27T07:40:57.838 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.838 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.838 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.838 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.838 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.838 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.839 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.839 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.839 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.839 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.839 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.839 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.839 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.839 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.839 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.840 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.840 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.840 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.840 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.840 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.840 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.840 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.840 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.840 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.841 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.841 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.841 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.841 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.841 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.841 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.841 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.841 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.844 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "mark_out_threshold": { 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mark_out_threshold", 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2419200", 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.845 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": { 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_name", 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "device_health_metrics", 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.846 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "retention_period": { 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retention_period", 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15552000", 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.847 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain device health metrics", 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_frequency": { 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_frequency", 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.848 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "self_heal": { 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "self_heal", 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.849 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T07:40:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_threshold": { 2024-05-27T07:40:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_threshold", 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7257600", 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "diskprediction_local", 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.854 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.855 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.856 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.857 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.858 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "predict_interval": { 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predict_interval", 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.859 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "predictor_model": { 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predictor_model", 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.860 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "prophetstor", 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "influx", 2024-05-27T07:40:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": false, 2024-05-27T07:40:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "influxdb python module not found", 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "batch_size": { 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "batch_size", 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5000", 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "database": { 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "database", 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "ceph", 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": { 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hostname", 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.867 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server hostname", 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "30", 2024-05-27T07:40:57.868 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "5", 2024-05-27T07:40:57.869 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.869 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.869 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T07:40:57.869 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.869 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.869 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.869 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.869 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.869 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.870 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.870 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.870 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.870 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.870 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.870 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.870 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.870 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.870 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.870 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.871 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.871 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.871 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.871 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.871 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.871 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.871 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.871 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.871 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.872 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.872 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.872 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.872 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.872 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.872 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.872 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.872 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.872 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.873 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.873 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.873 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.873 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.873 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.873 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.873 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.873 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.873 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.874 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.874 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.874 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.874 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.874 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.874 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.874 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.874 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.874 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.876 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.876 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.876 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.876 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.876 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.876 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.876 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.876 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.877 INFO:teuthology.orchestra.run.smithi043.stdout: "password": { 2024-05-27T07:40:57.877 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "password", 2024-05-27T07:40:57.877 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.877 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.877 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.877 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.877 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.877 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.877 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.877 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "password of InfluxDB server user", 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "port": { 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "port", 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8086", 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.878 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server port", 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.879 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "false", 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.880 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "threads": { 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threads", 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:57.881 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "32", 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "username": { 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "username", 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.882 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "username of InfluxDB server user", 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "verify_ssl": { 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "verify_ssl", 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "true", 2024-05-27T07:40:57.883 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "insights", 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.884 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.885 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.886 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.887 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.888 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.889 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.890 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "iostat", 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.891 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.892 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.893 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.894 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.895 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.896 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.897 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "k8sevents", 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_event_retention_days": { 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ceph_event_retention_days", 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7", 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T07:40:57.898 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "config_check_secs": { 2024-05-27T07:40:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_check_secs", 2024-05-27T07:40:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:57.899 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "10", 2024-05-27T07:40:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T07:40:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.900 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.901 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.902 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.902 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.902 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.902 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.902 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.902 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.902 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.902 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.903 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.903 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.903 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.903 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.903 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.903 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.903 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.903 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.903 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.903 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.904 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.904 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.904 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.904 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.904 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.904 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.904 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.904 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.904 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.905 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.906 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.906 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.906 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.906 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.906 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.906 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.906 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.906 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.906 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.907 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "localpool", 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "failure_domain": { 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "failure_domain", 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "host", 2024-05-27T07:40:57.909 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.910 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.910 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.910 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "failure domain for any created local pool", 2024-05-27T07:40:57.910 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T07:40:57.910 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.910 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.910 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.910 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.911 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.912 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.912 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.912 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.912 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.912 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.912 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.912 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.912 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.913 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.913 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.913 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.913 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.913 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.913 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.913 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.914 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.915 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.916 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": { 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_size", 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.917 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default min_size for any created local pool", 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "num_rep": { 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "num_rep", 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.918 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3", 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default replica count for any created local pool", 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": { 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_num", 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.919 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "128", 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "prefix": { 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prefix", 2024-05-27T07:40:57.920 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.921 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.921 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.921 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.921 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.921 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.921 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.921 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name prefix for any created local pool", 2024-05-27T07:40:57.921 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.921 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.922 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.922 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.922 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.922 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.922 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.922 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.922 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "subtree": { 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subtree", 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.923 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "rack", 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.924 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mds_autoscaler", 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.925 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.926 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.926 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.926 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.926 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.926 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.926 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.926 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.926 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.926 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.927 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.928 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.929 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.930 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirroring", 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.931 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.932 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.933 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.934 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.935 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.936 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.936 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.936 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.936 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.936 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.936 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.936 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.936 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.936 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.937 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.938 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.938 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.938 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.938 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.938 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.938 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.938 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.938 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.938 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.938 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "nfs", 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.939 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.940 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.941 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.942 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.943 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.945 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.946 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-05-27T07:40:57.946 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.946 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.946 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.946 INFO:teuthology.orchestra.run.smithi043.stdout: "fail_fs": { 2024-05-27T07:40:57.947 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "fail_fs", 2024-05-27T07:40:57.947 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.950 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.950 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.950 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.950 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.950 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.950 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.950 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.950 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.950 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.951 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.951 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.951 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.951 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.951 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.951 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.951 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.951 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.951 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.951 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.952 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.952 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.952 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.952 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.952 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.952 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.952 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.955 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator": { 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.956 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm", 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "rook", 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "test_orchestrator" 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Orchestrator backend", 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.957 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.958 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_perf_query", 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.959 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.960 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.961 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.962 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.963 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_support", 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.966 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.967 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.968 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.969 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.970 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_autoscaler", 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.972 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.973 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.974 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.975 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.975 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.975 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.975 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.975 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.975 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.975 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.975 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.975 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.976 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.977 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.979 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.980 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.980 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.980 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.980 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.980 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.980 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.980 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.980 INFO:teuthology.orchestra.run.smithi043.stdout: "threshold": { 2024-05-27T07:40:57.980 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threshold", 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1.0", 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "scaling threshold", 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.982 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.982 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "progress", 2024-05-27T07:40:57.982 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.982 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.982 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.982 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_pg_recovery_event": { 2024-05-27T07:40:57.982 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_pg_recovery_event", 2024-05-27T07:40:57.982 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.985 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.986 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:57.987 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.990 INFO:teuthology.orchestra.run.smithi043.stdout: "max_completed_events": { 2024-05-27T07:40:57.990 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_completed_events", 2024-05-27T07:40:57.990 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.990 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.990 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.990 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "50", 2024-05-27T07:40:57.990 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.990 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.990 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.990 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "number of past completed events to remember", 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:57.991 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.992 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.992 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.992 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long the module is going to sleep", 2024-05-27T07:40:57.992 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.992 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.992 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.992 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:57.992 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:57.992 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:57.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:57.994 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus", 2024-05-27T07:40:57.994 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:57.994 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:57.994 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:57.994 INFO:teuthology.orchestra.run.smithi043.stdout: "cache": { 2024-05-27T07:40:57.994 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cache", 2024-05-27T07:40:57.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.995 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.995 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:57.995 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.995 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.995 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.995 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.995 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.995 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.995 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.995 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.996 INFO:teuthology.orchestra.run.smithi043.stdout: "exclude_perf_counters": { 2024-05-27T07:40:57.996 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "exclude_perf_counters", 2024-05-27T07:40:57.996 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.996 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.996 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.996 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:57.996 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.996 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.996 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.997 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:57.998 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:57.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.000 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.001 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools": { 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools", 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.002 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.003 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "300", 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_interval": { 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_interval", 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.004 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15.0", 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.005 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.006 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "9283", 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.007 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "stale_cache_strategy": { 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stale_cache_strategy", 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.008 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "log", 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.009 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "default", 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: "default", 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: "error" 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-05-27T07:40:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-05-27T07:40:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-05-27T07:40:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-05-27T07:40:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-05-27T07:40:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.011 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_support", 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.012 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.013 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.014 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.015 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.017 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.018 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.019 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_snap_create": { 2024-05-27T07:40:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_snap_create", 2024-05-27T07:40:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.020 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: "mirror_snapshot_schedule": { 2024-05-27T07:40:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T07:40:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.021 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.022 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.023 INFO:teuthology.orchestra.run.smithi043.stdout: "trash_purge_schedule": { 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "trash_purge_schedule", 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.024 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "restful", 2024-05-27T07:40:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.025 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_auth": { 2024-05-27T07:40:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_auth", 2024-05-27T07:40:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.026 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-05-27T07:40:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-05-27T07:40:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.027 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.028 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.029 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.030 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.031 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.032 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.034 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.035 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.036 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rgw", 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.037 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.038 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.039 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.040 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.041 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.042 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rook", 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.044 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.045 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.046 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.047 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.048 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.049 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": { 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "storage_class", 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.050 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.051 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "local", 2024-05-27T07:40:58.051 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.051 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.051 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.051 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T07:40:58.051 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.051 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.051 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.051 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.051 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "selftest", 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.052 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.053 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.053 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.053 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.053 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.053 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.053 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.053 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.053 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.053 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.053 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.054 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.054 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.054 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.054 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.054 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.054 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.054 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.054 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.054 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.055 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.055 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.055 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.055 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.055 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.055 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.055 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.055 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.056 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.056 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.056 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.056 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.056 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.056 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.056 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.056 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.057 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.057 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.057 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.057 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.057 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.057 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.057 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.057 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.057 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.057 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.058 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.058 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.058 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.058 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.058 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.058 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.058 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.058 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.058 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.059 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.059 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.059 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.059 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.059 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.059 INFO:teuthology.orchestra.run.smithi043.stdout: "roption1": { 2024-05-27T07:40:58.059 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption1", 2024-05-27T07:40:58.059 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.060 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.060 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.060 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.060 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.060 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.060 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.060 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.060 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.060 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.060 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.061 INFO:teuthology.orchestra.run.smithi043.stdout: "roption2": { 2024-05-27T07:40:58.061 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption2", 2024-05-27T07:40:58.061 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.061 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.061 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.061 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "xyz", 2024-05-27T07:40:58.061 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.062 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.062 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.062 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.062 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.066 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.066 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.066 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption1": { 2024-05-27T07:40:58.066 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption1", 2024-05-27T07:40:58.066 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.066 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption2": { 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption2", 2024-05-27T07:40:58.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption3": { 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption3", 2024-05-27T07:40:58.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption4": { 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption4", 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption5": { 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption5", 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption6": { 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption6", 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption7": { 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption7", 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "42", 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "testkey": { 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testkey", 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "testlkey": { 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testlkey", 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "testnewline": { 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testnewline", 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smb", 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.079 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.080 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.082 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.083 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.084 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "update_orchestration": { 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "update_orchestration", 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T07:40:58.085 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snap_schedule", 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_m_granularity": { 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_m_granularity", 2024-05-27T07:40:58.086 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "dump_on_update": { 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dump_on_update", 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "dump database to debug log on update", 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.088 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.089 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.089 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.089 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.089 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.089 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.089 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.090 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.091 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.092 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.092 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.092 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.092 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.092 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.092 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.092 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.093 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.093 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.093 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.093 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.093 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.093 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.093 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.093 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.093 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.093 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.094 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.094 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.094 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.094 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.094 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.094 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.094 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.094 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.095 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.095 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.095 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.095 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.095 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.095 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.095 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.095 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.095 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.098 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stats", 2024-05-27T07:40:58.098 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.098 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.098 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.098 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.098 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.098 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.105 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.105 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.105 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.105 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.105 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.105 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.105 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.105 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.105 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.106 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "status", 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.107 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.108 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.109 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.110 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.112 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telegraf", 2024-05-27T07:40:58.113 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "address": { 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "address", 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.114 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15", 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.115 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.116 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.117 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.118 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.119 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.120 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.121 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telemetry", 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_basic": { 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_basic", 2024-05-27T07:40:58.122 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_crash": { 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_crash", 2024-05-27T07:40:58.123 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_device": { 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_device", 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.124 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_ident": { 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_ident", 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.125 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_perf": { 2024-05-27T07:40:58.126 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_perf", 2024-05-27T07:40:58.127 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.127 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.127 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.127 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.127 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.127 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.127 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.127 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T07:40:58.127 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "contact": { 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "contact", 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.128 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: "description": { 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "description", 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.129 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: "device_url": { 2024-05-27T07:40:58.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_url", 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.131 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-05-27T07:40:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-05-27T07:40:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.132 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.133 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.133 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.133 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.133 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.133 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.133 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.133 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:58.133 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:58.134 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.134 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.134 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.134 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "24", 2024-05-27T07:40:58.134 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "8", 2024-05-27T07:40:58.134 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.134 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.134 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.135 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.135 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.135 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.135 INFO:teuthology.orchestra.run.smithi043.stdout: "last_opt_revision": { 2024-05-27T07:40:58.135 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "last_opt_revision", 2024-05-27T07:40:58.135 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.135 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.135 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.136 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1", 2024-05-27T07:40:58.136 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.136 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.136 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.136 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.136 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.136 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.137 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard": { 2024-05-27T07:40:58.137 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard", 2024-05-27T07:40:58.137 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.137 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.137 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.137 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.137 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.137 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard_description": { 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard_description", 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.138 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.139 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.140 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.141 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.142 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.143 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "organization": { 2024-05-27T07:40:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "organization", 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "proxy": { 2024-05-27T07:40:58.145 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "proxy", 2024-05-27T07:40:58.146 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.146 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.146 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.146 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.146 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.146 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.146 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.146 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.146 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.146 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.147 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.147 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.147 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.147 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.147 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.147 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.147 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.147 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.147 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "url": { 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url", 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.148 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "test_orchestrator", 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.149 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.150 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.151 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.152 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.153 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.154 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.155 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "volumes", 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.156 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.157 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.158 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.159 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.160 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.162 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_clones": { 2024-05-27T07:40:58.162 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_clones", 2024-05-27T07:40:58.162 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.162 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.162 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.162 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4", 2024-05-27T07:40:58.162 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.162 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.162 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.162 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T07:40:58.163 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.163 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.163 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.163 INFO:teuthology.orchestra.run.smithi043.stdout: "periodic_async_work": { 2024-05-27T07:40:58.163 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "periodic_async_work", 2024-05-27T07:40:58.163 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.163 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.163 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.163 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Periodically check for async work", 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_delay": { 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_delay", 2024-05-27T07:40:58.164 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.165 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.165 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.165 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.165 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.165 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.165 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.165 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T07:40:58.165 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.165 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.165 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.166 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_no_wait": { 2024-05-27T07:40:58.166 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T07:40:58.166 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.166 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.166 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.166 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.166 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.166 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.167 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.167 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T07:40:58.167 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.167 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.167 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.167 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.167 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.167 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.167 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.168 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.168 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.168 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.168 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.168 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.168 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.168 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.168 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.168 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.168 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.169 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.169 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix", 2024-05-27T07:40:58.169 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.169 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.169 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.169 INFO:teuthology.orchestra.run.smithi043.stdout: "discovery_interval": { 2024-05-27T07:40:58.169 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "discovery_interval", 2024-05-27T07:40:58.170 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:58.170 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.170 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.170 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "100", 2024-05-27T07:40:58.170 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.170 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.170 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.170 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.170 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.170 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.171 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.171 INFO:teuthology.orchestra.run.smithi043.stdout: "identifier": { 2024-05-27T07:40:58.171 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "identifier", 2024-05-27T07:40:58.171 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.171 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.171 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.171 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.171 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.171 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.172 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.172 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.172 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.172 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.172 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.172 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:58.172 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:58.172 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.172 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.173 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.173 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:58.173 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.173 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.173 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.173 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.173 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.173 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.173 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.174 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.174 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.174 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.174 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.174 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.174 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.174 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.174 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.174 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.174 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.175 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.175 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.175 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.175 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.175 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.175 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.175 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.175 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.175 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.175 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.176 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.177 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.177 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.177 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.177 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.177 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.177 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.177 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.177 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.177 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.178 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.179 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.180 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_host": { 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_host", 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.181 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_port": { 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_port", 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10051", 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.182 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_sender": { 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_sender", 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm", 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "dashboard", 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-05-27T07:40:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "restful", 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_schedule" 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "available_modules": [ 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "alerts", 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How frequently to reexamine health status", 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.190 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_destination": { 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_destination", 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.191 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email address to send alerts to", 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_from_name": { 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_from_name", 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "Ceph", 2024-05-27T07:40:58.192 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email From: name", 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_host": { 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_host", 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.193 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP server", 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_password": { 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_password", 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.194 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Password to authenticate with", 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_port": { 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_port", 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.195 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "465", 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP port", 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_sender": { 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_sender", 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.196 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP envelope sender", 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_ssl": { 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_ssl", 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.197 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_user": { 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_user", 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.198 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "User to authenticate as", 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.199 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.200 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "balancer", 2024-05-27T07:40:58.201 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.201 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.201 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.201 INFO:teuthology.orchestra.run.smithi043.stdout: "active": { 2024-05-27T07:40:58.201 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "active", 2024-05-27T07:40:58.201 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.201 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.201 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.201 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically balance PGs across cluster", 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_time": { 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_time", 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.202 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.203 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0000", 2024-05-27T07:40:58.203 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.203 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.203 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.203 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "beginning time of day to automatically balance", 2024-05-27T07:40:58.203 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:40:58.203 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.203 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.203 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_weekday": { 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_weekday", 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:40:58.204 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_max_iterations": { 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_max_iterations", 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "25", 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "250", 2024-05-27T07:40:58.205 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.206 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-27T07:40:58.206 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.206 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.206 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.206 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_metrics": { 2024-05-27T07:40:58.206 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_metrics", 2024-05-27T07:40:58.206 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.206 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.206 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.207 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "pgs,objects,bytes", 2024-05-27T07:40:58.207 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.207 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.207 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.207 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-27T07:40:58.207 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-27T07:40:58.207 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.207 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_step": { 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_step", 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.5", 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0.001", 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "0.999", 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "aggressiveness of optimization", 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-27T07:40:58.208 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.209 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.209 INFO:teuthology.orchestra.run.smithi043.stdout: "end_time": { 2024-05-27T07:40:58.209 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_time", 2024-05-27T07:40:58.209 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.209 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.209 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.209 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2359", 2024-05-27T07:40:58.209 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.209 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.210 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.210 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ending time of day to automatically balance", 2024-05-27T07:40:58.210 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-27T07:40:58.210 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.210 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.210 INFO:teuthology.orchestra.run.smithi043.stdout: "end_weekday": { 2024-05-27T07:40:58.210 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_weekday", 2024-05-27T07:40:58.210 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:58.211 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.211 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.211 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.211 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-05-27T07:40:58.211 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-05-27T07:40:58.211 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.211 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-27T07:40:58.211 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-27T07:40:58.211 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.212 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.212 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.212 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.212 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.212 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.212 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.212 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.212 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.212 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.214 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.214 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.214 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.214 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.214 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.214 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.214 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.214 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.215 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.216 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.217 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "min_score": { 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_score", 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-27T07:40:58.218 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "upmap", 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.219 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-compat", 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "none", 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "read", 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap", 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap-read" 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Balancer mode", 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_ids": { 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_ids", 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.220 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.221 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.222 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_deviation": { 2024-05-27T07:40:58.223 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_deviation", 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_optimizations": { 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_optimizations", 2024-05-27T07:40:58.224 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm", 2024-05-27T07:40:58.225 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_down_multiplier": { 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_down_multiplier", 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.226 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_refresh_rate": { 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_refresh_rate", 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "20", 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-27T07:40:58.227 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_starting_port": { 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_starting_port", 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4721", 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.228 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-27T07:40:58.229 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.229 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.229 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.229 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_ptrace": { 2024-05-27T07:40:58.229 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_ptrace", 2024-05-27T07:40:58.229 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.229 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.229 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.229 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_interval": { 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_interval", 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_memory_target_ratio": { 2024-05-27T07:40:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_memory_target_ratio", 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.7", 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm_log_destination": { 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm_log_destination", 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "file", 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "file,syslog", 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: "syslog" 2024-05-27T07:40:58.233 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "cgroups_split": { 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cgroups_split", 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.234 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "config_checks_enabled": { 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_checks_enabled", 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.235 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "config_dashboard": { 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_dashboard", 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.236 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.237 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.237 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.237 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.237 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-27T07:40:58.237 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.237 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.237 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.237 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_alertmanager": { 2024-05-27T07:40:58.237 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_alertmanager", 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.238 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_base": { 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_base", 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Container image name, without the tag", 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.239 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.240 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.240 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_elasticsearch": { 2024-05-27T07:40:58.240 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_elasticsearch", 2024-05-27T07:40:58.240 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.240 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.240 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.240 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-27T07:40:58.240 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "elasticsearch container image", 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_grafana": { 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_grafana", 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.241 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-27T07:40:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_haproxy": { 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_haproxy", 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "HAproxy container image", 2024-05-27T07:40:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_agent": { 2024-05-27T07:40:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_agent", 2024-05-27T07:40:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger agent container image", 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_collector": { 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_collector", 2024-05-27T07:40:58.245 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.246 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.246 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.246 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-27T07:40:58.246 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.246 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.246 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.246 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger collector container image", 2024-05-27T07:40:58.246 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.246 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.247 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.247 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_query": { 2024-05-27T07:40:58.247 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_query", 2024-05-27T07:40:58.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.247 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.247 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.247 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-27T07:40:58.247 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.247 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.248 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.248 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger query container image", 2024-05-27T07:40:58.248 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.248 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.248 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.248 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_keepalived": { 2024-05-27T07:40:58.248 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_keepalived", 2024-05-27T07:40:58.248 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.248 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Keepalived container image", 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.249 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_loki": { 2024-05-27T07:40:58.250 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_loki", 2024-05-27T07:40:58.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.250 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.250 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.250 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-05-27T07:40:58.250 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.250 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.250 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.250 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Loki container image", 2024-05-27T07:40:58.250 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.251 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.251 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.251 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_node_exporter": { 2024-05-27T07:40:58.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_node_exporter", 2024-05-27T07:40:58.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.251 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.251 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.251 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-27T07:40:58.251 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_nvmeof": { 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_nvmeof", 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.252 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Nvme-of container image", 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_prometheus": { 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_prometheus", 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.253 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_promtail": { 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_promtail", 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.254 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Promtail container image", 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_samba": { 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_samba", 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Samba/SMB container image", 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_snmp_gateway": { 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_snmp_gateway", 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SNMP Gateway container image", 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "container_init": { 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_init", 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Run podman/docker with `--init`", 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "daemon_cache_timeout": { 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "daemon_cache_timeout", 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "default_cephadm_command_timeout": { 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_cephadm_command_timeout", 2024-05-27T07:40:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "900", 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "default_registry": { 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_registry", 2024-05-27T07:40:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io", 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "device_cache_timeout": { 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_cache_timeout", 2024-05-27T07:40:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1800", 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache device inventory", 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: "device_enhanced_scan": { 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_enhanced_scan", 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "facts_cache_timeout": { 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "facts_cache_timeout", 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache host facts data", 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: "grafana_dashboards_path": { 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "grafana_dashboards_path", 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "host_check_interval": { 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "host_check_interval", 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to perform a host check", 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "hw_monitoring": { 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hw_monitoring", 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-27T07:40:58.267 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "inventory_list_all": { 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "inventory_list_all", 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-27T07:40:58.268 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.269 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "log_refresh_metadata": { 2024-05-27T07:40:58.270 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_refresh_metadata", 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.271 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.272 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.273 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.274 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.274 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.274 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.274 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.274 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.274 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.274 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.274 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.274 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-27T07:40:58.275 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-27T07:40:58.276 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-27T07:40:58.277 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.277 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.277 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.277 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "*", 2024-05-27T07:40:58.277 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.277 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.277 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.277 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-27T07:40:58.277 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.277 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "max_count_per_host": { 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_count_per_host", 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.278 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of daemons per service per host", 2024-05-27T07:40:58.279 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.279 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.279 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.279 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd_draining_count": { 2024-05-27T07:40:58.279 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_osd_draining_count", 2024-05-27T07:40:58.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.279 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.279 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.280 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:58.280 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.280 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.280 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-27T07:40:58.280 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.280 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.280 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.280 INFO:teuthology.orchestra.run.smithi043.stdout: "migration_current": { 2024-05-27T07:40:58.281 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "migration_current", 2024-05-27T07:40:58.281 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.281 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.281 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.281 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.281 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.281 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.281 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.281 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "internal - do not modify", 2024-05-27T07:40:58.281 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.282 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.282 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.282 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-05-27T07:40:58.282 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-05-27T07:40:58.282 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.282 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.282 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.282 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "root", 2024-05-27T07:40:58.282 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm-package", 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: "root" 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "mode for remote execution of cephadm", 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: "oob_default_addr": { 2024-05-27T07:40:58.284 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "oob_default_addr", 2024-05-27T07:40:58.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.284 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.284 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.284 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "169.254.1.1", 2024-05-27T07:40:58.284 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.284 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.284 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.284 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-27T07:40:58.285 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.285 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.285 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.285 INFO:teuthology.orchestra.run.smithi043.stdout: "prometheus_alerts_path": { 2024-05-27T07:40:58.285 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus_alerts_path", 2024-05-27T07:40:58.285 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.285 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.285 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.286 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-27T07:40:58.286 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.286 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.286 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.286 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-27T07:40:58.286 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.286 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.286 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.287 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_insecure": { 2024-05-27T07:40:58.287 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_insecure", 2024-05-27T07:40:58.287 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.287 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.287 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.287 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.287 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.287 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.287 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.288 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-27T07:40:58.288 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.288 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.288 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.288 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_password": { 2024-05-27T07:40:58.288 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_password", 2024-05-27T07:40:58.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.288 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.288 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_url": { 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_url", 2024-05-27T07:40:58.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_username": { 2024-05-27T07:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_username", 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "secure_monitoring_stack": { 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "secure_monitoring_stack", 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "service_discovery_port": { 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "service_discovery_port", 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8765", 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "cephadm service discovery port", 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.293 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.294 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "ssh_config_file": { 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssh_config_file", 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.295 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "use_agent": { 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_agent", 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.296 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "use_repo_digest": { 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_repo_digest", 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-27T07:40:58.297 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_failed_host_check": { 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_failed_host_check", 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.298 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if the host check fails", 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_daemons": { 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_daemons", 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.299 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_hosts": { 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_hosts", 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.300 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crash", 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.301 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.302 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.303 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.304 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.305 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.306 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "retain_interval": { 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retain_interval", 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "31536000", 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.307 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.308 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_recent_interval": { 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_recent_interval", 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1209600", 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-27T07:40:58.309 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dashboard", 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.310 INFO:teuthology.orchestra.run.smithi043.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-27T07:40:58.311 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-27T07:40:58.311 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.311 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.311 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.311 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:58.311 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.311 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.311 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.311 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.311 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_HOST": { 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.312 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.313 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.313 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.313 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.313 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-27T07:40:58.313 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-27T07:40:58.313 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.313 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.313 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.313 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_ENABLED": { 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_ENABLED", 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.314 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.315 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.315 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.315 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.315 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.315 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.315 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.315 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.315 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.315 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-27T07:40:58.316 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-27T07:40:58.316 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.316 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.316 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.316 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.316 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.316 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.316 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.316 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.316 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: "ENABLE_BROWSABLE_API": { 2024-05-27T07:40:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-27T07:40:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.317 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.318 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.318 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.318 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.318 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.318 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.318 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.318 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-27T07:40:58.318 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-27T07:40:58.319 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.319 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.319 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.319 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.319 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.319 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.319 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.319 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.319 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.320 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.320 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.320 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-27T07:40:58.320 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-27T07:40:58.320 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.320 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.320 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.320 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.321 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.321 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.321 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.321 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.321 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.321 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.321 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.321 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.322 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.323 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.324 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.324 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-27T07:40:58.324 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-27T07:40:58.324 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.324 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.324 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.324 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.324 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.325 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.326 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-27T07:40:58.327 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_PASSWORD": { 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-27T07:40:58.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.330 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_URL": { 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_URL", 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.331 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_USERNAME": { 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.333 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.334 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.335 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.336 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "MANAGED_BY_CLUSTERS": { 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "[]", 2024-05-27T07:40:58.337 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "MULTICLUSTER_CONFIG": { 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "MULTICLUSTER_CONFIG", 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.338 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "{}", 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_HOST": { 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.339 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.340 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.341 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-27T07:40:58.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-27T07:40:58.343 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-27T07:40:58.344 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.344 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.344 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.344 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.344 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.344 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.344 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.344 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.344 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.344 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.345 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-27T07:40:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-27T07:40:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_ENABLED": { 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-27T07:40:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-27T07:40:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.353 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.353 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.353 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.353 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.353 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.353 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-27T07:40:58.353 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-27T07:40:58.353 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.354 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.354 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.354 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8", 2024-05-27T07:40:58.354 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.354 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.354 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.354 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.354 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.354 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.354 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.355 INFO:teuthology.orchestra.run.smithi043.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-27T07:40:58.355 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-27T07:40:58.355 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.355 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.355 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.355 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "45", 2024-05-27T07:40:58.355 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.355 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.356 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.356 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.356 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.356 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.356 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.356 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ACCESS_KEY": { 2024-05-27T07:40:58.356 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-27T07:40:58.356 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.356 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.357 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.357 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.357 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.357 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.357 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.357 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.357 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.357 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.357 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.358 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.358 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-27T07:40:58.358 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-27T07:40:58.358 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.358 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.358 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.358 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-05-27T07:40:58.358 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.358 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.359 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.359 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.359 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.359 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.359 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.359 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SECRET_KEY": { 2024-05-27T07:40:58.359 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-27T07:40:58.359 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.359 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.360 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SSL_VERIFY": { 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "UNSAFE_TLS_v1_2": { 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "UNSAFE_TLS_v1_2", 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.363 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.364 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "cross_origin_url": { 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cross_origin_url", 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.366 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "crt_file": { 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crt_file", 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.367 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "debug": { 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "debug", 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable/disable debug options", 2024-05-27T07:40:58.368 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "jwt_token_ttl": { 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "jwt_token_ttl", 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "28800", 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.369 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.370 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.371 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.372 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.373 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.374 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.375 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "motd": { 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "motd", 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.376 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "The message of the day", 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_resolve_ip_addr": { 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "redirect_resolve_ip_addr", 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.377 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-05-27T07:40:58.378 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8080", 2024-05-27T07:40:58.379 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.380 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.381 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.381 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.381 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.381 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.381 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.381 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.381 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.382 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl_server_port": { 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl_server_port", 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8443", 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.383 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.384 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "redirect", 2024-05-27T07:40:58.385 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.385 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.385 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.385 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.385 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect" 2024-05-27T07:40:58.385 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.385 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.385 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.385 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.385 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-05-27T07:40:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-05-27T07:40:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-05-27T07:40:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-05-27T07:40:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-05-27T07:40:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.387 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.387 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.387 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.387 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.387 INFO:teuthology.orchestra.run.smithi043.stdout: "url_prefix": { 2024-05-27T07:40:58.387 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url_prefix", 2024-05-27T07:40:58.387 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.387 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.387 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.389 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "devicehealth", 2024-05-27T07:40:58.389 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.389 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.389 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.389 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_monitoring": { 2024-05-27T07:40:58.389 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_monitoring", 2024-05-27T07:40:58.389 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.389 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.389 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.390 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.390 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.390 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.390 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.390 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "monitor device health metrics", 2024-05-27T07:40:58.390 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.390 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.390 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.391 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.391 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.391 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.391 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.391 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.391 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.391 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.392 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.392 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.392 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.392 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.392 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.392 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.393 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.393 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.393 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.393 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.393 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.393 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.393 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 -- 172.21.15.43:0/118277173 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f22580b2960 msgr2=0x7f22580b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f22580b2960 0x7f22580b4e70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f227419dbb0 tx=0x7f2264005db0 comp rx=0 tx=0).stop 2024-05-27T07:40:58.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 -- 172.21.15.43:0/118277173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f227410a0e0 msgr2=0x7f227419c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:40:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f227410a0e0 0x7f227419c740 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f226c005cc0 tx=0x7f226c003c20 comp rx=0 tx=0).stop 2024-05-27T07:40:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 -- 172.21.15.43:0/118277173 shutdown_connections 2024-05-27T07:40:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f22580b2960 0x7f22580b4e70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2274109c40 0x7f22741a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f227410a0e0 0x7f227419c740 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 --2- 172.21.15.43:0/118277173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2274104390 0x7f22741a2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:40:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 -- 172.21.15.43:0/118277173 >> 172.21.15.43:0/118277173 conn(0x7f22740ff380 msgr2=0x7f22741068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:40:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 -- 172.21.15.43:0/118277173 shutdown_connections 2024-05-27T07:40:58.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:40:57.085+0000 7f227b200640 1 -- 172.21.15.43:0/118277173 wait complete. 2024-05-27T07:40:58.397 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:57 smithi179 ceph-mon[35888]: pgmap v905: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:58.397 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:57 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/118277173' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T07:40:58.397 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:57 smithi132 ceph-mon[35283]: pgmap v905: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:58.397 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:57 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/118277173' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T07:40:58.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.398 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.398 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.399 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.400 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.401 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "mark_out_threshold": { 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mark_out_threshold", 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2419200", 2024-05-27T07:40:58.402 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": { 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_name", 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "device_health_metrics", 2024-05-27T07:40:58.403 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "retention_period": { 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retention_period", 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15552000", 2024-05-27T07:40:58.404 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain device health metrics", 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_frequency": { 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_frequency", 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-05-27T07:40:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "self_heal": { 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "self_heal", 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and check device health", 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_threshold": { 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_threshold", 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7257600", 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "diskprediction_local", 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.411 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.412 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.413 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.414 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.415 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.415 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.415 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.415 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.415 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.415 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.415 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.415 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "predict_interval": { 2024-05-27T07:40:58.416 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predict_interval", 2024-05-27T07:40:58.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.417 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.417 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.417 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-05-27T07:40:58.417 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.417 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.417 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.417 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.417 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.417 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.418 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.418 INFO:teuthology.orchestra.run.smithi043.stdout: "predictor_model": { 2024-05-27T07:40:58.418 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predictor_model", 2024-05-27T07:40:58.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.418 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.418 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.418 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "prophetstor", 2024-05-27T07:40:58.418 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.418 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.419 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.420 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-05-27T07:40:58.420 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.420 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.420 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.420 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.420 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.420 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.420 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.420 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "influx", 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": false, 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "influxdb python module not found", 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "batch_size": { 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "batch_size", 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.422 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5000", 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "database": { 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "database", 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.423 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.424 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.424 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "ceph", 2024-05-27T07:40:58.424 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.424 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.424 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.424 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-27T07:40:58.424 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.424 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.424 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.425 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": { 2024-05-27T07:40:58.425 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hostname", 2024-05-27T07:40:58.425 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.425 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.425 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.425 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.425 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.425 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.425 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.426 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server hostname", 2024-05-27T07:40:58.426 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.426 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.426 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.426 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:58.426 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:58.426 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.427 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.427 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.427 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "30", 2024-05-27T07:40:58.427 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "5", 2024-05-27T07:40:58.427 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.427 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.427 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-27T07:40:58.427 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.427 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.428 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.428 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.428 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.428 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.428 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.428 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.428 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.428 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.428 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.429 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.429 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.429 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.429 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.429 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.429 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.429 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.429 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.429 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.429 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.430 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.430 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.430 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.430 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.430 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.430 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.430 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.430 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.430 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.431 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.431 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.431 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.431 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.431 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.431 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.431 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.434 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.435 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "password": { 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "password", 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.436 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "password of InfluxDB server user", 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "port": { 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "port", 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.437 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8086", 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server port", 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.438 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-05-27T07:40:58.439 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "false", 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "threads": { 2024-05-27T07:40:58.440 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threads", 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "32", 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: "username": { 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "username", 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "username of InfluxDB server user", 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "verify_ssl": { 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "verify_ssl", 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "true", 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.443 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "insights", 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.444 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.445 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.446 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.447 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.448 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.449 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.450 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "iostat", 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.451 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.453 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.454 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.455 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.456 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.456 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.456 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.456 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.456 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.456 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.456 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.456 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.457 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.458 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.458 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "k8sevents", 2024-05-27T07:40:58.458 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.458 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.458 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.458 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_event_retention_days": { 2024-05-27T07:40:58.458 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ceph_event_retention_days", 2024-05-27T07:40:58.458 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.459 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.459 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.459 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7", 2024-05-27T07:40:58.459 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.459 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.459 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.459 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-27T07:40:58.459 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.459 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.459 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.460 INFO:teuthology.orchestra.run.smithi043.stdout: "config_check_secs": { 2024-05-27T07:40:58.460 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_check_secs", 2024-05-27T07:40:58.460 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.460 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.460 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.460 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:58.460 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "10", 2024-05-27T07:40:58.460 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.460 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.461 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.462 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.462 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.462 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.462 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.462 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.462 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.462 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.462 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.462 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.462 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.463 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.463 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.463 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.463 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.463 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.463 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.463 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.463 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.463 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.464 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.464 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.464 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.464 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.464 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.464 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.464 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.464 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.464 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.465 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.465 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.465 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.465 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.465 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.465 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.465 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.465 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.465 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.465 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.466 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.466 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.466 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.466 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.466 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.466 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.466 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.466 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.467 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.468 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.468 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.468 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.468 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.468 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.468 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.468 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.468 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.468 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "localpool", 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "failure_domain": { 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "failure_domain", 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "host", 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "failure domain for any created local pool", 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": { 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_size", 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default min_size for any created local pool", 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "num_rep": { 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "num_rep", 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3", 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.477 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default replica count for any created local pool", 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": { 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_num", 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "128", 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.478 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default pg_num for any created local pool", 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "prefix": { 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prefix", 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.479 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name prefix for any created local pool", 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.480 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "subtree": { 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subtree", 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "rack", 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.481 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mds_autoscaler", 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.482 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.483 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.484 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.485 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.486 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.487 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.488 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.489 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirroring", 2024-05-27T07:40:58.489 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.489 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.489 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.489 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.489 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.490 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.491 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.492 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.492 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.492 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.492 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.492 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.492 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.492 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.492 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.493 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.494 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.494 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.494 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.494 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.494 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.494 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.494 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.495 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.496 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.496 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.496 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.496 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.496 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.496 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.496 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.496 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.497 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "nfs", 2024-05-27T07:40:58.498 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.498 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.498 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.498 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.498 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.498 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.498 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.498 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.498 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.499 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.500 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.501 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.502 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.502 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.502 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.502 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.502 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.502 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.502 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.502 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.502 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.503 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.503 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.503 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.503 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.503 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.503 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.503 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.503 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.504 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.504 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.504 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.504 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.504 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.504 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.504 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.504 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.504 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.505 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.505 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.505 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.505 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.505 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.505 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.505 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.505 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.505 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.505 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.506 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.506 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.506 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.506 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-05-27T07:40:58.506 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.506 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "fail_fs": { 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "fail_fs", 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.507 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.508 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.509 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.510 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.512 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator": { 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.513 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm", 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: "rook", 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: "test_orchestrator" 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Orchestrator backend", 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.514 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.515 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_perf_query", 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.516 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.517 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.519 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.520 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.521 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_support", 2024-05-27T07:40:58.522 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.523 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.524 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.525 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.526 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.527 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.527 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.527 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.527 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.527 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.527 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.527 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.527 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.528 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.530 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_autoscaler", 2024-05-27T07:40:58.530 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.530 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.530 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.530 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.530 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.530 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.530 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.531 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.531 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.531 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.531 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.531 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.531 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.531 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.531 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.531 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.531 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.532 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.532 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.532 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.532 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.532 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.532 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.532 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.532 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.532 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.533 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.533 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.533 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.533 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.533 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.533 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.533 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.533 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.533 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.533 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.534 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.534 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.534 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.534 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.534 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.534 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.534 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.534 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.534 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.535 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.535 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.535 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.535 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.535 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.535 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.535 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.535 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.535 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.535 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.536 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.536 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.536 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.536 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.536 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.536 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.536 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.536 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.537 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.537 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.537 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.537 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.537 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.537 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.537 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:58.537 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:58.537 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.538 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.538 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.538 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:58.538 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.538 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.538 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.538 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.538 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.538 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.538 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.539 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.540 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.540 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.540 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.540 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.540 INFO:teuthology.orchestra.run.smithi043.stdout: "threshold": { 2024-05-27T07:40:58.540 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threshold", 2024-05-27T07:40:58.540 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:58.540 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1.0", 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "scaling threshold", 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "progress", 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_pg_recovery_event": { 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_pg_recovery_event", 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.542 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.543 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.544 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.546 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.547 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.548 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "max_completed_events": { 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_completed_events", 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.549 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "50", 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "number of past completed events to remember", 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.550 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long the module is going to sleep", 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.551 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus", 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.552 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "cache": { 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cache", 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.553 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "exclude_perf_counters": { 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "exclude_perf_counters", 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.555 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.556 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.557 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.558 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.559 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools": { 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools", 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.560 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.561 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.562 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "300", 2024-05-27T07:40:58.562 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.562 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.562 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.562 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.562 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.562 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.562 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_interval": { 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_interval", 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15.0", 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.563 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.564 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.565 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.566 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "9283", 2024-05-27T07:40:58.566 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.566 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.566 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.566 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-27T07:40:58.566 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.566 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.566 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.566 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.567 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: "stale_cache_strategy": { 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stale_cache_strategy", 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "log", 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.568 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "default", 2024-05-27T07:40:58.569 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.570 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.570 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.570 INFO:teuthology.orchestra.run.smithi043.stdout: "default", 2024-05-27T07:40:58.570 INFO:teuthology.orchestra.run.smithi043.stdout: "error" 2024-05-27T07:40:58.570 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.570 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.570 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.570 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.570 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.571 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-05-27T07:40:58.571 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-05-27T07:40:58.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.571 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.571 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.571 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-05-27T07:40:58.571 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-05-27T07:40:58.571 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-05-27T07:40:58.571 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.571 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_support", 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.575 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.575 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.575 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.575 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.575 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.575 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.575 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.575 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.576 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.576 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.576 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.576 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.576 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.576 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.576 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.576 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.577 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.577 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.577 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.577 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.577 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.577 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.577 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.577 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.577 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.578 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_snap_create": { 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_snap_create", 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.579 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "mirror_snapshot_schedule": { 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirror_snapshot_schedule", 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.580 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.581 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "trash_purge_schedule": { 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "trash_purge_schedule", 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.582 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "restful", 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_auth": { 2024-05-27T07:40:58.583 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_auth", 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-05-27T07:40:58.584 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.585 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.586 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.587 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.588 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.590 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.592 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.593 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rgw", 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.594 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.595 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.596 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.597 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.599 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.599 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.599 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.599 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.599 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.599 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.599 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.599 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.599 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.600 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.601 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rook", 2024-05-27T07:40:58.602 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.602 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.602 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.602 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.602 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.602 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.602 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.602 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.602 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.603 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.604 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.604 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.604 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.604 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.604 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.604 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.604 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.604 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.604 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.605 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.606 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.606 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.606 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.606 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.606 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.606 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.606 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.606 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.606 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.607 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.607 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.607 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.607 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.607 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.607 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.607 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.607 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.607 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.608 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.609 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.610 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.610 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.610 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.610 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.610 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": { 2024-05-27T07:40:58.610 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "storage_class", 2024-05-27T07:40:58.610 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.610 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.610 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "local", 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "selftest", 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.612 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.613 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.614 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.615 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.616 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.616 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.616 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.616 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.616 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.616 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.616 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.616 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.616 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.616 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.617 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "roption1": { 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption1", 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.618 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "roption2": { 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption2", 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "xyz", 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.619 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption1": { 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption1", 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.620 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption2": { 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption2", 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.621 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption3": { 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption3", 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.622 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption4": { 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption4", 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.623 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption5": { 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption5", 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.624 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption6": { 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption6", 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.625 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption7": { 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption7", 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.626 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "42", 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "testkey": { 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testkey", 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "testlkey": { 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testlkey", 2024-05-27T07:40:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "testnewline": { 2024-05-27T07:40:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testnewline", 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smb", 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.634 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.634 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.634 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.634 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.634 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.634 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.634 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.634 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.634 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.635 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.636 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.636 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.636 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.636 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.636 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.636 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.636 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.636 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.636 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.637 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.638 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.638 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.638 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.638 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.638 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.638 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.638 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.638 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.638 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.639 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.639 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.639 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.639 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.639 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.639 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.639 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.639 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.639 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.639 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: "update_orchestration": { 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "update_orchestration", 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.640 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically update orchestration when smb resources are changed", 2024-05-27T07:40:58.641 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.641 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.641 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.641 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snap_schedule", 2024-05-27T07:40:58.641 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.642 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.642 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.642 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_m_granularity": { 2024-05-27T07:40:58.642 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_m_granularity", 2024-05-27T07:40:58.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.642 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.642 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.642 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.642 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.643 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.643 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.643 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow minute scheduled snapshots", 2024-05-27T07:40:58.643 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.643 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.643 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.643 INFO:teuthology.orchestra.run.smithi043.stdout: "dump_on_update": { 2024-05-27T07:40:58.643 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dump_on_update", 2024-05-27T07:40:58.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.644 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.644 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.644 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.644 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.644 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.644 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.644 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "dump database to debug log on update", 2024-05-27T07:40:58.644 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.645 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.645 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.645 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.645 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.645 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.645 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.645 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.645 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.646 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.646 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.646 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.646 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.646 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.646 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.646 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.646 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.646 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.647 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.648 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.649 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.650 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.651 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stats", 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.653 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.656 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.657 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.658 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.659 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "status", 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.660 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.661 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.662 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.663 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.664 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.665 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telegraf", 2024-05-27T07:40:58.666 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "address": { 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "address", 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.667 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15", 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.668 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.669 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.670 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.671 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.672 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.673 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.676 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telemetry", 2024-05-27T07:40:58.676 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.676 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.676 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.676 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_basic": { 2024-05-27T07:40:58.676 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_basic", 2024-05-27T07:40:58.676 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.676 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.676 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.676 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_crash": { 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_crash", 2024-05-27T07:40:58.677 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-27T07:40:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_device": { 2024-05-27T07:40:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_device", 2024-05-27T07:40:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-27T07:40:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_ident": { 2024-05-27T07:40:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_ident", 2024-05-27T07:40:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-27T07:40:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_perf": { 2024-05-27T07:40:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_perf", 2024-05-27T07:40:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-27T07:40:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "contact": { 2024-05-27T07:40:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "contact", 2024-05-27T07:40:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "description": { 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "description", 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "device_url": { 2024-05-27T07:40:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_url", 2024-05-27T07:40:58.687 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.687 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.687 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.687 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-27T07:40:58.687 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.687 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.687 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.687 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.687 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.687 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.688 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "24", 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "8", 2024-05-27T07:40:58.689 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "last_opt_revision": { 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "last_opt_revision", 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.690 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1", 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard": { 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard", 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.691 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard_description": { 2024-05-27T07:40:58.692 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard_description", 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.694 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.695 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.696 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.697 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.698 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "organization": { 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "organization", 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.699 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "proxy": { 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "proxy", 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.700 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.701 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "url": { 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url", 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.702 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "test_orchestrator", 2024-05-27T07:40:58.703 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.704 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.705 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.706 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.707 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.708 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.708 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.708 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.708 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.708 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.708 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.708 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.708 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.708 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.709 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.710 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.710 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.710 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.710 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.710 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.710 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.710 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.710 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.710 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "volumes", 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.711 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.712 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.712 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.712 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.712 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.712 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.712 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.712 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.712 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.712 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.713 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.714 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.714 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.714 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.714 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.714 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.714 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.714 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.714 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.715 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.716 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.716 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.716 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.716 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.716 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.716 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.716 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.716 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.716 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.717 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.717 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.717 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.717 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.717 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.717 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.717 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.718 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.718 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.718 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.718 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.718 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.718 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.718 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.718 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.719 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_clones": { 2024-05-27T07:40:58.719 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_clones", 2024-05-27T07:40:58.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.719 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.719 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.719 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4", 2024-05-27T07:40:58.719 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.719 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.719 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.720 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-27T07:40:58.720 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.720 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.720 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.720 INFO:teuthology.orchestra.run.smithi043.stdout: "periodic_async_work": { 2024-05-27T07:40:58.720 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "periodic_async_work", 2024-05-27T07:40:58.720 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.720 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.720 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Periodically check for async work", 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_delay": { 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_delay", 2024-05-27T07:40:58.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_no_wait": { 2024-05-27T07:40:58.722 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_no_wait", 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.723 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.724 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix", 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "discovery_interval": { 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "discovery_interval", 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "100", 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.725 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "identifier": { 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "identifier", 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.726 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.727 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.728 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.729 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.730 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-05-27T07:40:58.731 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.732 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "sqlite3_killpoint": { 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sqlite3_killpoint", 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "dev", 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-05-27T07:40:58.733 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_host": { 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_host", 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.734 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_port": { 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_port", 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.735 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10051", 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_sender": { 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_sender", 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-05-27T07:40:58.736 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: "services": { 2024-05-27T07:40:58.737 INFO:teuthology.orchestra.run.smithi043.stdout: "dashboard": "https://172.21.15.43:8443/" 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "always_on_modules": { 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus": [ 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "balancer", 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "crash", 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "devicehealth", 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator", 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscaler", 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "progress", 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_support", 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "status", 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "telemetry", 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: "volumes" 2024-05-27T07:40:58.738 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific": [ 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "balancer", 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "crash", 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "devicehealth", 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator", 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscaler", 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "progress", 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_support", 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "status", 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "telemetry", 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "volumes" 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy": [ 2024-05-27T07:40:58.739 INFO:teuthology.orchestra.run.smithi043.stdout: "balancer", 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "crash", 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "devicehealth", 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator", 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscaler", 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "progress", 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_support", 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "status", 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "telemetry", 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "volumes" 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "reef": [ 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "balancer", 2024-05-27T07:40:58.740 INFO:teuthology.orchestra.run.smithi043.stdout: "crash", 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "devicehealth", 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator", 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscaler", 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "progress", 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_support", 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "status", 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "telemetry", 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "volumes" 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "squid": [ 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "balancer", 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "crash", 2024-05-27T07:40:58.741 INFO:teuthology.orchestra.run.smithi043.stdout: "devicehealth", 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator", 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscaler", 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "progress", 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_support", 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "status", 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "telemetry", 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "volumes" 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "last_failure_osd_epoch": 91, 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "active_clients": [ 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "devicehealth", 2024-05-27T07:40:58.742 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 370973468 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "libcephsqlite", 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.743 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-05-27T07:40:58.744 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-05-27T07:40:58.744 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3293338206 2024-05-27T07:40:58.744 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.744 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:58.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.744 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_support", 2024-05-27T07:40:58.744 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-05-27T07:40:58.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3861121958 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snap_schedule", 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1512878046 2024-05-27T07:40:58.745 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snap_schedule", 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 54444325 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.746 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:58.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-27T07:40:58.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.747 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "volumes", 2024-05-27T07:40:58.747 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-05-27T07:40:58.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-27T07:40:58.747 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-05-27T07:40:58.747 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-05-27T07:40:58.747 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2008627542 2024-05-27T07:40:58.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.747 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:58.748 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-27T07:40:58.748 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-27T07:40:58.748 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-27T07:40:58.756 DEBUG:tasks.check_counter:Getting stats from a 2024-05-27T07:40:58.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-27T07:40:58.757 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:57 smithi043 ceph-mon[31531]: pgmap v905: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:58.758 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:57 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/118277173' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-27T07:40:59.236 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:40:59.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:40:59 smithi043 ceph-mon[31531]: pgmap v906: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:40:59.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:40:59 smithi179 ceph-mon[35888]: pgmap v906: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:41:00.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:40:59 smithi132 ceph-mon[35283]: pgmap v906: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:41:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-05-27T07:41:01.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:01 smithi043 ceph-mon[31531]: pgmap v907: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:41:01.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:01 smithi179 ceph-mon[35888]: pgmap v907: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:41:02.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:01 smithi132 ceph-mon[35283]: pgmap v907: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-05-27T07:41:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-27T07:41:02.742 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1893, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi043 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump' 2024-05-27T07:41:02.744 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-05-27T07:41:02.754 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-05-27T07:41:02.755 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T07:41:02.755 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:41:02.836 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T07:41:02.837 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:41:02.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:41:02.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:02 smithi043 ceph-mon[31531]: pgmap v908: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 34 KiB/s wr, 15 op/s 2024-05-27T07:41:03.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:02 smithi132 ceph-mon[35283]: pgmap v908: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 34 KiB/s wr, 15 op/s 2024-05-27T07:41:03.216 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:02 smithi179 ceph-mon[35888]: pgmap v908: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 34 KiB/s wr, 15 op/s 2024-05-27T07:41:03.357 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:05.662 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:05 smithi043 ceph-mon[31531]: pgmap v909: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 34 KiB/s wr, 15 op/s 2024-05-27T07:41:05.662 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:05 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:41:05.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.663+0000 7f2365000700 1 -- 172.21.15.43:0/691157570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23601045e0 msgr2=0x7f23601049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:05.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.663+0000 7f2365000700 1 --2- 172.21.15.43:0/691157570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23601045e0 0x7f23601049f0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f23500099c0 tx=0x7f2350009cd0 comp rx=0 tx=0).stop 2024-05-27T07:41:05.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.664+0000 7f2365000700 1 -- 172.21.15.43:0/691157570 shutdown_connections 2024-05-27T07:41:05.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.664+0000 7f2365000700 1 --2- 172.21.15.43:0/691157570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2360109860 0x7f23601111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:05.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.664+0000 7f2365000700 1 --2- 172.21.15.43:0/691157570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23601045e0 0x7f23601049f0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:05.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.664+0000 7f2365000700 1 --2- 172.21.15.43:0/691157570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2360108f10 0x7f2360109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:05.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.664+0000 7f2365000700 1 -- 172.21.15.43:0/691157570 >> 172.21.15.43:0/691157570 conn(0x7f2360100030 msgr2=0x7f2360102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:05.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.664+0000 7f2365000700 1 -- 172.21.15.43:0/691157570 shutdown_connections 2024-05-27T07:41:05.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.664+0000 7f2365000700 1 -- 172.21.15.43:0/691157570 wait complete. 2024-05-27T07:41:05.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.666+0000 7f2365000700 1 Processor -- start 2024-05-27T07:41:05.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.666+0000 7f2365000700 1 -- start start 2024-05-27T07:41:05.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.666+0000 7f2365000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23601045e0 0x7f236019cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:05.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.667+0000 7f2365000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2360108f10 0x7f236019d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:05.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.667+0000 7f2365000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2360109860 0x7f23601a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:05.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.667+0000 7f2365000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2360114340 con 0x7f2360108f10 2024-05-27T07:41:05.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.667+0000 7f2365000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f23601141c0 con 0x7f2360109860 2024-05-27T07:41:05.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.667+0000 7f2365000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f23601144c0 con 0x7f23601045e0 2024-05-27T07:41:05.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.667+0000 7f2355400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2360108f10 0x7f236019d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:05.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.667+0000 7f2355400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2360108f10 0x7f236019d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56172/0 (socket says 172.21.15.43:56172) 2024-05-27T07:41:05.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.667+0000 7f2355400700 1 -- 172.21.15.43:0/2986082100 learned_addr learned my addr 172.21.15.43:0/2986082100 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:41:05.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.667+0000 7f235de00700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23601045e0 0x7f236019cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:05.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.667+0000 7f235e800700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2360109860 0x7f23601a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:05.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f2355400700 1 -- 172.21.15.43:0/2986082100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23601045e0 msgr2=0x7f236019cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:05.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f2355400700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23601045e0 0x7f236019cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:05.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f2355400700 1 -- 172.21.15.43:0/2986082100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2360109860 msgr2=0x7f23601a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:05.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f2355400700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2360109860 0x7f23601a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:05.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f2355400700 1 -- 172.21.15.43:0/2986082100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23601a1e00 con 0x7f2360108f10 2024-05-27T07:41:05.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f235e800700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2360109860 0x7f23601a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:41:05.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f235de00700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23601045e0 0x7f236019cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:41:05.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f2355400700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2360108f10 0x7f236019d4d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f234800d980 tx=0x7f234800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:05.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f2357400700 1 -- 172.21.15.43:0/2986082100 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348017070 con 0x7f2360108f10 2024-05-27T07:41:05.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f2357400700 1 -- 172.21.15.43:0/2986082100 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2348010460 con 0x7f2360108f10 2024-05-27T07:41:05.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.668+0000 7f2365000700 1 -- 172.21.15.43:0/2986082100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23601a20f0 con 0x7f2360108f10 2024-05-27T07:41:05.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.669+0000 7f2365000700 1 -- 172.21.15.43:0/2986082100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23601a9a10 con 0x7f2360108f10 2024-05-27T07:41:05.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.669+0000 7f2357400700 1 -- 172.21.15.43:0/2986082100 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234800de60 con 0x7f2360108f10 2024-05-27T07:41:05.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.670+0000 7f2365000700 1 -- 172.21.15.43:0/2986082100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f233c005330 con 0x7f2360108f10 2024-05-27T07:41:05.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.673+0000 7f2357400700 1 -- 172.21.15.43:0/2986082100 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7f2348009b40 con 0x7f2360108f10 2024-05-27T07:41:05.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.675+0000 7f2357400700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f23440b29f0 0x7f23440b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:05.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.676+0000 7f2357400700 1 -- 172.21.15.43:0/2986082100 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 9496+0+0 (secure 0 0 0) 0x7f23480e2460 con 0x7f2360108f10 2024-05-27T07:41:05.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.676+0000 7f235de00700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f23440b29f0 0x7f23440b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:05.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.676+0000 7f235de00700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f23440b29f0 0x7f23440b4f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2350000c00 tx=0x7f2350005940 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:05.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.679+0000 7f2357400700 1 -- 172.21.15.43:0/2986082100 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f23480a9a10 con 0x7f2360108f10 2024-05-27T07:41:05.716 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:05 smithi179 ceph-mon[35888]: pgmap v909: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 34 KiB/s wr, 15 op/s 2024-05-27T07:41:05.716 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:05 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:41:05.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:05 smithi132 ceph-mon[35283]: pgmap v909: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 34 KiB/s wr, 15 op/s 2024-05-27T07:41:05.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:05 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-27T07:41:05.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.954+0000 7f2365000700 1 -- 172.21.15.43:0/2986082100 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f233c005620 con 0x7f2360108f10 2024-05-27T07:41:05.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.955+0000 7f2357400700 1 -- 172.21.15.43:0/2986082100 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v163) ==== 81+0+944 (secure 0 0 0) 0x7f23480a9310 con 0x7f2360108f10 2024-05-27T07:41:05.953 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/1809594794 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2266081183 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/3331041602 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/1651567761 2024-05-28T06:57:52.506320+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/3399450175 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/1651567761 2024-05-28T06:57:52.506320+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1332970913 2024-05-28T06:57:18.380825+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1752945868 2024-05-28T06:57:18.380825+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2426463283 2024-05-28T06:57:18.380825+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2081880966 2024-05-28T06:57:52.506320+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/3521718752 2024-05-28T06:57:18.380825+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/1809594794 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/3253329315 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:05.954 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/830904719 2024-05-28T06:57:52.506320+0000 2024-05-27T07:41:05.955 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/3521718752 2024-05-28T06:57:18.380825+0000 2024-05-27T07:41:05.955 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/338672925 2024-05-28T06:57:52.506320+0000 2024-05-27T07:41:05.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.959+0000 7f2365000700 1 -- 172.21.15.43:0/2986082100 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f23440b29f0 msgr2=0x7f23440b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:05.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.959+0000 7f2365000700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f23440b29f0 0x7f23440b4f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2350000c00 tx=0x7f2350005940 comp rx=0 tx=0).stop 2024-05-27T07:41:05.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.959+0000 7f2365000700 1 -- 172.21.15.43:0/2986082100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2360108f10 msgr2=0x7f236019d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:05.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.959+0000 7f2365000700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2360108f10 0x7f236019d4d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f234800d980 tx=0x7f234800dd40 comp rx=0 tx=0).stop 2024-05-27T07:41:05.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.959+0000 7f2365000700 1 -- 172.21.15.43:0/2986082100 shutdown_connections 2024-05-27T07:41:05.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.959+0000 7f2365000700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7f23440b29f0 0x7f23440b4f00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:05.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.959+0000 7f2365000700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2360109860 0x7f23601a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:05.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.959+0000 7f2365000700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23601045e0 0x7f236019cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:05.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.959+0000 7f2365000700 1 --2- 172.21.15.43:0/2986082100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2360108f10 0x7f236019d4d0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:05.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.959+0000 7f2365000700 1 -- 172.21.15.43:0/2986082100 >> 172.21.15.43:0/2986082100 conn(0x7f2360100030 msgr2=0x7f236010a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:05.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.960+0000 7f2365000700 1 -- 172.21.15.43:0/2986082100 shutdown_connections 2024-05-27T07:41:05.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:05.960+0000 7f2365000700 1 -- 172.21.15.43:0/2986082100 wait complete. 2024-05-27T07:41:05.959 INFO:teuthology.orchestra.run.smithi043.stderr:listed 16 entries 2024-05-27T07:41:06.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:06 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/2986082100' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:41:06.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:41:06.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:41:06.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:06 smithi043 ceph-mon[31531]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:41:06.716 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:06 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/2986082100' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:41:06.716 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:41:06.716 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:41:06.716 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:06 smithi179 ceph-mon[35888]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:41:06.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:06 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/2986082100' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:41:06.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-27T07:41:06.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-27T07:41:06.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:06 smithi132 ceph-mon[35283]: from='mgr.35033 172.21.15.43:0/1211716209' entity='mgr.x' 2024-05-27T07:41:06.913 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-27T07:41:06.914 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:41:06.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-05-27T07:41:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/mon.a/config 2024-05-27T07:41:07.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:07 smithi043 ceph-mon[31531]: pgmap v910: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 46 KiB/s wr, 17 op/s 2024-05-27T07:41:07.801 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:07 smithi043 ceph-mon[31531]: pgmap v911: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 21 op/s 2024-05-27T07:41:07.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:07 smithi132 ceph-mon[35283]: pgmap v910: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 46 KiB/s wr, 17 op/s 2024-05-27T07:41:07.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:07 smithi132 ceph-mon[35283]: pgmap v911: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 21 op/s 2024-05-27T07:41:07.965 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:07 smithi179 ceph-mon[35888]: pgmap v910: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 46 KiB/s wr, 17 op/s 2024-05-27T07:41:07.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:07 smithi179 ceph-mon[35888]: pgmap v911: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 55 KiB/s wr, 21 op/s 2024-05-27T07:41:08.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:08 smithi043 ceph-mon[31531]: osdmap e164: 12 total, 12 up, 12 in 2024-05-27T07:41:08.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:08 smithi132 ceph-mon[35283]: osdmap e164: 12 total, 12 up, 12 in 2024-05-27T07:41:08.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:08 smithi179 ceph-mon[35888]: osdmap e164: 12 total, 12 up, 12 in 2024-05-27T07:41:09.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:09 smithi132 ceph-mon[35283]: pgmap v913: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 69 KiB/s wr, 26 op/s 2024-05-27T07:41:09.928 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:09 smithi043 ceph-mon[31531]: pgmap v913: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 69 KiB/s wr, 26 op/s 2024-05-27T07:41:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.930+0000 7feedd600700 1 -- 172.21.15.43:0/723198531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feed8109860 msgr2=0x7feed81111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.930+0000 7feedd600700 1 --2- 172.21.15.43:0/723198531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feed8109860 0x7feed81111c0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7feed000b0d0 tx=0x7feed000b3e0 comp rx=0 tx=0).stop 2024-05-27T07:41:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.930+0000 7feedd600700 1 -- 172.21.15.43:0/723198531 shutdown_connections 2024-05-27T07:41:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.930+0000 7feedd600700 1 --2- 172.21.15.43:0/723198531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feed8109860 0x7feed81111c0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.930+0000 7feedd600700 1 --2- 172.21.15.43:0/723198531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feed81045e0 0x7feed81049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.930+0000 7feedd600700 1 --2- 172.21.15.43:0/723198531 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed8108f10 0x7feed8109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.930+0000 7feedd600700 1 -- 172.21.15.43:0/723198531 >> 172.21.15.43:0/723198531 conn(0x7feed8100030 msgr2=0x7feed8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.931+0000 7feedd600700 1 -- 172.21.15.43:0/723198531 shutdown_connections 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.931+0000 7feedd600700 1 -- 172.21.15.43:0/723198531 wait complete. 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.932+0000 7feedd600700 1 Processor -- start 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.932+0000 7feedd600700 1 -- start start 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feedd600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed81045e0 0x7feed8075ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feedd600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feed8108f10 0x7feed8075ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feedd600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feed8109860 0x7feed8076530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feedd600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feed8114340 con 0x7feed8108f10 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feedd600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7feed81141c0 con 0x7feed8109860 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feedd600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7feed81144c0 con 0x7feed81045e0 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feece000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feed8108f10 0x7feed8075ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feece000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feed8108f10 0x7feed8075ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56186/0 (socket says 172.21.15.43:56186) 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feece000700 1 -- 172.21.15.43:0/3732310419 learned_addr learned my addr 172.21.15.43:0/3732310419 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-05-27T07:41:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feed6a00700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed81045e0 0x7feed8075ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.933+0000 7feed7400700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feed8109860 0x7feed8076530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.934+0000 7feece000700 1 -- 172.21.15.43:0/3732310419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed81045e0 msgr2=0x7feed8075ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.934+0000 7feece000700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed81045e0 0x7feed8075ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.934+0000 7feece000700 1 -- 172.21.15.43:0/3732310419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feed8109860 msgr2=0x7feed8076530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.934+0000 7feece000700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feed8109860 0x7feed8076530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.934+0000 7feece000700 1 -- 172.21.15.43:0/3732310419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed81a90b0 con 0x7feed8108f10 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.934+0000 7feece000700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feed8108f10 0x7feed8075ff0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7feec000d980 tx=0x7feec000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.935+0000 7feecfe00700 1 -- 172.21.15.43:0/3732310419 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0017070 con 0x7feed8108f10 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.935+0000 7feecfe00700 1 -- 172.21.15.43:0/3732310419 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feec000fd20 con 0x7feed8108f10 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.935+0000 7feedd600700 1 -- 172.21.15.43:0/3732310419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed81a93a0 con 0x7feed8108f10 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.935+0000 7feecfe00700 1 -- 172.21.15.43:0/3732310419 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec000e3b0 con 0x7feed8108f10 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.935+0000 7feedd600700 1 -- 172.21.15.43:0/3732310419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feed81a98e0 con 0x7feed8108f10 2024-05-27T07:41:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.936+0000 7feedd600700 1 -- 172.21.15.43:0/3732310419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feed80512b0 con 0x7feed8108f10 2024-05-27T07:41:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.939+0000 7feecfe00700 1 -- 172.21.15.43:0/3732310419 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 149783+0+0 (secure 0 0 0) 0x7feec000f890 con 0x7feed8108f10 2024-05-27T07:41:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.940+0000 7feecfe00700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feebc0b29f0 0x7feebc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:41:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.941+0000 7feecfe00700 1 -- 172.21.15.43:0/3732310419 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 9584+0+0 (secure 0 0 0) 0x7feec00e2480 con 0x7feed8108f10 2024-05-27T07:41:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.941+0000 7feed6a00700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feebc0b29f0 0x7feebc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:41:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.942+0000 7feed6a00700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feebc0b29f0 0x7feebc0b4f00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7feec8005950 tx=0x7feec800b410 comp rx=0 tx=0).ready entity=mgr.35033 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:41:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:09.943+0000 7feecfe00700 1 -- 172.21.15.43:0/3732310419 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7feec00a9a00 con 0x7feed8108f10 2024-05-27T07:41:09.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:09 smithi179 ceph-mon[35888]: pgmap v913: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 69 KiB/s wr, 26 op/s 2024-05-27T07:41:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.204+0000 7feedd600700 1 -- 172.21.15.43:0/3732310419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7feed802e5e0 con 0x7feed8108f10 2024-05-27T07:41:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.205+0000 7feecfe00700 1 -- 172.21.15.43:0/3732310419 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v164) ==== 81+0+944 (secure 0 0 0) 0x7feec00a9300 con 0x7feed8108f10 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/1809594794 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2266081183 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/3331041602 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/1651567761 2024-05-28T06:57:52.506320+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/3399450175 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/1651567761 2024-05-28T06:57:52.506320+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1332970913 2024-05-28T06:57:18.380825+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1752945868 2024-05-28T06:57:18.380825+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2426463283 2024-05-28T06:57:18.380825+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2081880966 2024-05-28T06:57:52.506320+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/3521718752 2024-05-28T06:57:18.380825+0000 2024-05-27T07:41:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/1809594794 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:10.204 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/3253329315 2024-05-28T07:13:54.391569+0000 2024-05-27T07:41:10.204 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/830904719 2024-05-28T06:57:52.506320+0000 2024-05-27T07:41:10.204 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/3521718752 2024-05-28T06:57:18.380825+0000 2024-05-27T07:41:10.204 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/338672925 2024-05-28T06:57:52.506320+0000 2024-05-27T07:41:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.209+0000 7feedd600700 1 -- 172.21.15.43:0/3732310419 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feebc0b29f0 msgr2=0x7feebc0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.209+0000 7feedd600700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feebc0b29f0 0x7feebc0b4f00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7feec8005950 tx=0x7feec800b410 comp rx=0 tx=0).stop 2024-05-27T07:41:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.209+0000 7feedd600700 1 -- 172.21.15.43:0/3732310419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feed8108f10 msgr2=0x7feed8075ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:41:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.209+0000 7feedd600700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feed8108f10 0x7feed8075ff0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7feec000d980 tx=0x7feec000dc90 comp rx=0 tx=0).stop 2024-05-27T07:41:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.209+0000 7feedd600700 1 -- 172.21.15.43:0/3732310419 shutdown_connections 2024-05-27T07:41:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.209+0000 7feedd600700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.43:6800/2368318638,v1:172.21.15.43:6801/2368318638] conn(0x7feebc0b29f0 0x7feebc0b4f00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.209+0000 7feedd600700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feed8109860 0x7feed8076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.209+0000 7feedd600700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed81045e0 0x7feed8075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.209+0000 7feedd600700 1 --2- 172.21.15.43:0/3732310419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feed8108f10 0x7feed8075ff0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:41:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.209+0000 7feedd600700 1 -- 172.21.15.43:0/3732310419 >> 172.21.15.43:0/3732310419 conn(0x7feed8100030 msgr2=0x7feed810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:41:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.210+0000 7feedd600700 1 -- 172.21.15.43:0/3732310419 shutdown_connections 2024-05-27T07:41:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-05-27T07:41:10.210+0000 7feedd600700 1 -- 172.21.15.43:0/3732310419 wait complete. 2024-05-27T07:41:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:listed 16 entries 2024-05-27T07:41:10.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:10 smithi132 ceph-mon[35283]: from='client.? 172.21.15.43:0/3732310419' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:41:10.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:10 smithi043 ceph-mon[31531]: from='client.? 172.21.15.43:0/3732310419' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:41:10.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:10 smithi179 ceph-mon[35888]: from='client.? 172.21.15.43:0/3732310419' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-27T07:41:11.035 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-05-27T07:41:11.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:41:11.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-27T07:41:11.159 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:41:11.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi043.stderr:ceph-fuse[88755]: fuse finished with error 0 and tester_r 0 2024-05-27T07:41:11.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:11 smithi043 ceph-mon[31531]: pgmap v914: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 17 KiB/s wr, 2 op/s 2024-05-27T07:41:11.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:11 smithi132 ceph-mon[35283]: pgmap v914: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 17 KiB/s wr, 2 op/s 2024-05-27T07:41:11.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:11 smithi179 ceph-mon[35888]: pgmap v914: 129 pgs: 129 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 17 KiB/s wr, 2 op/s 2024-05-27T07:41:13.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:13 smithi132 ceph-mon[35283]: pgmap v915: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 17 KiB/s wr, 2 op/s 2024-05-27T07:41:13.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:13 smithi043 ceph-mon[31531]: pgmap v915: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 17 KiB/s wr, 2 op/s 2024-05-27T07:41:13.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:13 smithi179 ceph-mon[35888]: pgmap v915: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 17 KiB/s wr, 2 op/s 2024-05-27T07:41:15.848 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:15 smithi132 ceph-mon[35283]: pgmap v916: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-05-27T07:41:15.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:15 smithi043 ceph-mon[31531]: pgmap v916: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-05-27T07:41:15.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:15 smithi179 ceph-mon[35888]: pgmap v916: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-05-27T07:41:17.261 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-05-27T07:41:17.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:41:17.261 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-27T07:41:17.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:41:17.340 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-05-27T07:41:17.431 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-27T07:41:17.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:41:17.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:41:17.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:41:17.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link delete ceph-brx 2024-05-27T07:41:17.651 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-27T07:41:17.672 INFO:tasks.cephadm:Teardown begin 2024-05-27T07:41:17.672 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/cephadm.py", line 2270, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1893, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi043 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump' 2024-05-27T07:41:17.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:41:17.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:41:17.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:17 smithi043 ceph-mon[31531]: pgmap v917: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2024-05-27T07:41:17.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:17 smithi043 ceph-mon[31531]: osdmap e165: 12 total, 12 up, 12 in 2024-05-27T07:41:17.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:17 smithi179 ceph-mon[35888]: pgmap v917: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2024-05-27T07:41:17.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:17 smithi179 ceph-mon[35888]: osdmap e165: 12 total, 12 up, 12 in 2024-05-27T07:41:18.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:17 smithi132 ceph-mon[35283]: pgmap v917: 129 pgs: 129 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2024-05-27T07:41:18.098 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:17 smithi132 ceph-mon[35283]: osdmap e165: 12 total, 12 up, 12 in 2024-05-27T07:41:18.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:41:18.322 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-27T07:41:18.322 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:41:18.384 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:41:18.457 DEBUG:teuthology.orchestra.run.smithi179:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-27T07:41:18.515 INFO:tasks.cephadm:Stopping all daemons... 2024-05-27T07:41:18.515 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-27T07:41:18.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.a 2024-05-27T07:41:18.960 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:18 smithi043 systemd[1]: Stopping Ceph mon.a for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:41:19.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:19 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a[31527]: 2024-05-27T07:41:19.145+0000 7ff891200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:41:19.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:19 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a[31527]: 2024-05-27T07:41:19.145+0000 7ff891200700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-27T07:41:19.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:19 smithi043 podman[140659]: 2024-05-27 07:41:19.269670366 +0000 UTC m=+0.268127997 container died 83c5e162e73ac83426edc994a78b2a05b76872cd934b4bdcf1fd06e3b26c3ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-27T07:41:19.847 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:19 smithi043 podman[140659]: 2024-05-27 07:41:19.838946399 +0000 UTC m=+0.837403994 container cleanup 83c5e162e73ac83426edc994a78b2a05b76872cd934b4bdcf1fd06e3b26c3ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:41:19.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:19 smithi043 bash[140659]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a 2024-05-27T07:41:20.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 27 07:41:20 smithi043 podman[140671]: 2024-05-27 07:41:20.116072052 +0000 UTC m=+0.828603041 container remove 83c5e162e73ac83426edc994a78b2a05b76872cd934b4bdcf1fd06e3b26c3ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-a, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:41:20.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.a.service' 2024-05-27T07:41:20.774 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:41:20.775 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-27T07:41:20.775 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-27T07:41:20.775 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.b 2024-05-27T07:41:21.348 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:20 smithi132 systemd[1]: Stopping Ceph mon.b for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:41:21.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:21 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-b[35279]: 2024-05-27T07:41:21.439+0000 7f2c6a400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:41:21.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:21 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-b[35279]: 2024-05-27T07:41:21.439+0000 7f2c6a400700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-27T07:41:21.849 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:21 smithi132 podman[77305]: 2024-05-27 07:41:21.533325285 +0000 UTC m=+0.263112193 container died 0f4e4ee5f492472259d70a517cde08a69176b19804c7a4972bfbf58e81ba1e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-05-27T07:41:22.266 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:21 smithi132 podman[77305]: 2024-05-27 07:41:21.990748612 +0000 UTC m=+0.720535514 container cleanup 0f4e4ee5f492472259d70a517cde08a69176b19804c7a4972bfbf58e81ba1e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-05-27T07:41:22.267 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:21 smithi132 bash[77305]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-b 2024-05-27T07:41:22.598 INFO:journalctl@ceph.mon.b.smithi132.stdout:May 27 07:41:22 smithi132 podman[77319]: 2024-05-27 07:41:22.267677483 +0000 UTC m=+0.716099721 container remove 0f4e4ee5f492472259d70a517cde08a69176b19804c7a4972bfbf58e81ba1e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T07:41:22.799 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.b.service' 2024-05-27T07:41:22.975 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:41:22.975 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-27T07:41:22.975 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-27T07:41:22.975 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.c 2024-05-27T07:41:23.466 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:23 smithi179 systemd[1]: Stopping Ceph mon.c for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:41:23.968 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-c[35876]: 2024-05-27T07:41:23.558+0000 7f0706400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:41:24.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-c[35876]: 2024-05-27T07:41:23.558+0000 7f0706400700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-27T07:41:24.055 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:23 smithi179 podman[80007]: 2024-05-27 07:41:23.621452504 +0000 UTC m=+0.231955542 container died 554dd2bd2723005388a7e57562bae43876e1674f6257eb9150c3b07af13a7952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:41:24.385 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:24 smithi179 podman[80007]: 2024-05-27 07:41:24.110322291 +0000 UTC m=+0.720825370 container cleanup 554dd2bd2723005388a7e57562bae43876e1674f6257eb9150c3b07af13a7952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-27T07:41:24.386 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:24 smithi179 bash[80007]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-c 2024-05-27T07:41:24.716 INFO:journalctl@ceph.mon.c.smithi179.stdout:May 27 07:41:24 smithi179 podman[80020]: 2024-05-27 07:41:24.387095308 +0000 UTC m=+0.754066255 container remove 554dd2bd2723005388a7e57562bae43876e1674f6257eb9150c3b07af13a7952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mon-c, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-05-27T07:41:24.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mon.c.service' 2024-05-27T07:41:25.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:41:25.066 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-27T07:41:25.066 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-27T07:41:25.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.x 2024-05-27T07:41:25.460 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:41:25 smithi043 systemd[1]: Stopping Ceph mgr.x for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:41:25.771 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:41:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:41:25.675+0000 7fb9e7c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:41:25.771 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:41:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x[31760]: 2024-05-27T07:41:25.675+0000 7fb9e7c00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-27T07:41:26.088 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:41:25 smithi043 podman[140777]: 2024-05-27 07:41:25.778547244 +0000 UTC m=+0.229398817 container died 0ae7c08d9f686e3ad97fb8ca31c8075df36a3ffb0a09dba42c7136bc0405afbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:41:26.710 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:41:26 smithi043 podman[140777]: 2024-05-27 07:41:26.376980608 +0000 UTC m=+0.827832208 container cleanup 0ae7c08d9f686e3ad97fb8ca31c8075df36a3ffb0a09dba42c7136bc0405afbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:41:26.710 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:41:26 smithi043 bash[140777]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x 2024-05-27T07:41:26.710 INFO:journalctl@ceph.mgr.x.smithi043.stdout:May 27 07:41:26 smithi043 podman[140789]: 2024-05-27 07:41:26.604095283 +0000 UTC m=+0.809117296 container remove 0ae7c08d9f686e3ad97fb8ca31c8075df36a3ffb0a09dba42c7136bc0405afbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:41:27.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.x.service' 2024-05-27T07:41:27.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:41:27.205 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-27T07:41:27.205 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-27T07:41:27.205 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.y 2024-05-27T07:41:27.598 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:41:27 smithi132 systemd[1]: Stopping Ceph mgr.y for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:41:28.253 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:41:27 smithi132 podman[77438]: 2024-05-27 07:41:27.913705351 +0000 UTC m=+0.269770542 container died a66e6a3505753eaea9c0f4e3c7a712297c94290187fbfc82a87f469078423a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-05-27T07:41:28.598 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:41:28 smithi132 podman[77438]: 2024-05-27 07:41:28.430065327 +0000 UTC m=+0.786130558 container cleanup a66e6a3505753eaea9c0f4e3c7a712297c94290187fbfc82a87f469078423a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:41:28.598 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:41:28 smithi132 bash[77438]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y 2024-05-27T07:41:29.098 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:41:28 smithi132 podman[77453]: 2024-05-27 07:41:28.690202484 +0000 UTC m=+0.723915675 container remove a66e6a3505753eaea9c0f4e3c7a712297c94290187fbfc82a87f469078423a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-27T07:41:29.098 INFO:journalctl@ceph.mgr.y.smithi132.stdout:May 27 07:41:28 smithi132 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-05-27T07:41:29.175 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.y.service' 2024-05-27T07:41:29.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:41:29.286 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-27T07:41:29.287 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-27T07:41:29.287 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.z 2024-05-27T07:41:29.716 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:41:29 smithi179 systemd[1]: Stopping Ceph mgr.z for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:41:30.216 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:41:29 smithi179 podman[80125]: 2024-05-27 07:41:29.929104576 +0000 UTC m=+0.218654774 container died dc3f23bc322893644abf48800075e652f9a41551ce69bde6d125d086c7c1a3c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:41:30.493 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:41:30 smithi179 podman[80125]: 2024-05-27 07:41:30.466082406 +0000 UTC m=+0.755632641 container cleanup dc3f23bc322893644abf48800075e652f9a41551ce69bde6d125d086c7c1a3c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:41:30.493 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:41:30 smithi179 bash[80125]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z 2024-05-27T07:41:30.966 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:41:30 smithi179 podman[80138]: 2024-05-27 07:41:30.692809229 +0000 UTC m=+0.745525337 container remove dc3f23bc322893644abf48800075e652f9a41551ce69bde6d125d086c7c1a3c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:41:30.966 INFO:journalctl@ceph.mgr.z.smithi179.stdout:May 27 07:41:30 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-05-27T07:41:31.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mgr.z.service' 2024-05-27T07:41:31.344 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:41:31.344 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-27T07:41:31.344 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-27T07:41:31.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.0 2024-05-27T07:41:31.711 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:31 smithi043 systemd[1]: Stopping Ceph osd.0 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:41:32.460 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0[43769]: 2024-05-27T07:41:31.981+0000 7f4fa8600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:41:32.461 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0[43769]: 2024-05-27T07:41:31.981+0000 7f4fa8600700 -1 osd.0 165 *** Got signal Terminated *** 2024-05-27T07:41:32.461 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0[43769]: 2024-05-27T07:41:31.981+0000 7f4fa8600700 -1 osd.0 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:41:38.060 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:37 smithi043 podman[140905]: 2024-05-27 07:41:37.756801569 +0000 UTC m=+5.930218041 container died 2934ad7a01cfce3e7124f4485e6d0c699f8cd38357a3798cbbe4b032a99aec72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-27T07:41:38.461 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:38 smithi043 podman[140905]: 2024-05-27 07:41:38.282900553 +0000 UTC m=+6.456317051 container cleanup 2934ad7a01cfce3e7124f4485e6d0c699f8cd38357a3798cbbe4b032a99aec72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T07:41:38.461 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:38 smithi043 bash[140905]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0 2024-05-27T07:41:38.960 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:38 smithi043 podman[140928]: 2024-05-27 07:41:38.576564555 +0000 UTC m=+0.803443952 container remove 2934ad7a01cfce3e7124f4485e6d0c699f8cd38357a3798cbbe4b032a99aec72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-27T07:41:39.764 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:39 smithi043 podman[141019]: 2024-05-27 07:41:39.371939177 +0000 UTC m=+0.038891302 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:41:39.765 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:39 smithi043 podman[141019]: 2024-05-27 07:41:39.494143947 +0000 UTC m=+0.161096103 container create 600a1758be8f9f4b99ec93a2c532912101ad03aa2e0accf7dc1399a674badb21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-27T07:41:40.178 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:39 smithi043 podman[141019]: 2024-05-27 07:41:39.764659003 +0000 UTC m=+0.431611223 container init 600a1758be8f9f4b99ec93a2c532912101ad03aa2e0accf7dc1399a674badb21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:41:40.181 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:39 smithi043 podman[141019]: 2024-05-27 07:41:39.777182705 +0000 UTC m=+0.444134844 container start 600a1758be8f9f4b99ec93a2c532912101ad03aa2e0accf7dc1399a674badb21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-27T07:41:40.181 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:39 smithi043 podman[141019]: 2024-05-27 07:41:39.930379953 +0000 UTC m=+0.597332108 container attach 600a1758be8f9f4b99ec93a2c532912101ad03aa2e0accf7dc1399a674badb21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:41:40.960 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 27 07:41:40 smithi043 podman[141019]: 2024-05-27 07:41:40.693148727 +0000 UTC m=+1.360100890 container died 600a1758be8f9f4b99ec93a2c532912101ad03aa2e0accf7dc1399a674badb21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T07:41:41.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.0.service' 2024-05-27T07:41:41.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:41:41.681 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-27T07:41:41.681 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-27T07:41:41.682 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.1 2024-05-27T07:41:42.098 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:41 smithi132 systemd[1]: Stopping Ceph osd.1 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:41:42.849 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1[39843]: 2024-05-27T07:41:42.565+0000 7f2585600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:41:42.849 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1[39843]: 2024-05-27T07:41:42.565+0000 7f2585600700 -1 osd.1 165 *** Got signal Terminated *** 2024-05-27T07:41:42.849 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1[39843]: 2024-05-27T07:41:42.565+0000 7f2585600700 -1 osd.1 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:41:48.456 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:48 smithi132 podman[77559]: 2024-05-27 07:41:48.137651749 +0000 UTC m=+6.035330358 container died 4627b716a906318e6f543f25ecff1ceae08f08659e41a2c0d020cbab510b6188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:41:48.849 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:48 smithi132 podman[77559]: 2024-05-27 07:41:48.59374497 +0000 UTC m=+6.491423639 container cleanup 4627b716a906318e6f543f25ecff1ceae08f08659e41a2c0d020cbab510b6188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5) 2024-05-27T07:41:48.849 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:48 smithi132 bash[77559]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1 2024-05-27T07:41:49.348 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:48 smithi132 podman[77573]: 2024-05-27 07:41:48.88704457 +0000 UTC m=+0.733008197 container remove 4627b716a906318e6f543f25ecff1ceae08f08659e41a2c0d020cbab510b6188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-05-27T07:41:50.066 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:49 smithi132 podman[77658]: 2024-05-27 07:41:49.622502478 +0000 UTC m=+0.036156023 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:41:50.066 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:49 smithi132 podman[77658]: 2024-05-27 07:41:49.763406605 +0000 UTC m=+0.177060192 container create 053144dc819243794c3177d93a72ca76c5d9e4c353bb1a81385057488c161435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:41:50.066 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:50 smithi132 podman[77658]: 2024-05-27 07:41:50.067280168 +0000 UTC m=+0.480933774 container init 053144dc819243794c3177d93a72ca76c5d9e4c353bb1a81385057488c161435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-27T07:41:50.349 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:50 smithi132 podman[77658]: 2024-05-27 07:41:50.080337476 +0000 UTC m=+0.493991090 container start 053144dc819243794c3177d93a72ca76c5d9e4c353bb1a81385057488c161435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:41:50.349 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:50 smithi132 podman[77658]: 2024-05-27 07:41:50.233022889 +0000 UTC m=+0.646676532 container attach 053144dc819243794c3177d93a72ca76c5d9e4c353bb1a81385057488c161435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:41:51.513 INFO:journalctl@ceph.osd.1.smithi132.stdout:May 27 07:41:51 smithi132 podman[77658]: 2024-05-27 07:41:51.141553767 +0000 UTC m=+1.555207390 container died 053144dc819243794c3177d93a72ca76c5d9e4c353bb1a81385057488c161435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:41:51.943 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.1.service' 2024-05-27T07:41:52.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:41:52.051 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-27T07:41:52.051 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-27T07:41:52.051 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.2 2024-05-27T07:41:52.466 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:41:52 smithi179 systemd[1]: Stopping Ceph osd.2 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:41:52.966 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:41:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2[40928]: 2024-05-27T07:41:52.610+0000 7fceb6800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:41:52.966 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:41:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2[40928]: 2024-05-27T07:41:52.610+0000 7fceb6800700 -1 osd.2 165 *** Got signal Terminated *** 2024-05-27T07:41:52.967 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:41:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2[40928]: 2024-05-27T07:41:52.610+0000 7fceb6800700 -1 osd.2 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:41:58.716 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:41:58 smithi179 podman[80248]: 2024-05-27 07:41:58.432015767 +0000 UTC m=+5.978460920 container stop f9e6b6d1a1a13f2640253cf5ddba34e0155c5d53a77b046128a21c03286df37c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-27T07:41:58.716 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:41:58 smithi179 podman[80248]: 2024-05-27 07:41:58.462896159 +0000 UTC m=+6.009341279 container died f9e6b6d1a1a13f2640253cf5ddba34e0155c5d53a77b046128a21c03286df37c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:41:58.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:41:58 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:41:58.632+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:41:58.970 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:41:58 smithi179 podman[80248]: 2024-05-27 07:41:58.953162654 +0000 UTC m=+6.499607800 container cleanup f9e6b6d1a1a13f2640253cf5ddba34e0155c5d53a77b046128a21c03286df37c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-05-27T07:41:58.970 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:41:58 smithi179 bash[80248]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2 2024-05-27T07:41:59.462 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:41:58 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:41:58.965+0000 7f3392000700 -1 osd.3 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.403103+0000 front 2024-05-27T07:41:33.403253+0000 (oldest deadline 2024-05-27T07:41:58.703384+0000) 2024-05-27T07:41:59.462 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:41:58 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:41:58.995+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:41:59.623 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:41:59 smithi179 podman[80267]: 2024-05-27 07:41:59.246858216 +0000 UTC m=+0.798115654 container remove f9e6b6d1a1a13f2640253cf5ddba34e0155c5d53a77b046128a21c03286df37c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:41:59.966 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:41:59 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:41:59.624+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:00.235 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:41:59 smithi179 podman[80365]: 2024-05-27 07:41:59.888294686 +0000 UTC m=+0.037217155 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:42:00.235 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:42:00 smithi179 podman[80365]: 2024-05-27 07:42:00.015042513 +0000 UTC m=+0.163964994 container create 2d77f3b88cf00bc001df6ef3ffed864e7458bce0817f57d3479eabf5c25724b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:42:00.460 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:41:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:41:59.969+0000 7f3392000700 -1 osd.3 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.403103+0000 front 2024-05-27T07:41:33.403253+0000 (oldest deadline 2024-05-27T07:41:58.703384+0000) 2024-05-27T07:42:00.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:41:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:41:59.985+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:00.624 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:00.394+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:00.625 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:42:00 smithi179 podman[80365]: 2024-05-27 07:42:00.335274611 +0000 UTC m=+0.484197105 container init 2d77f3b88cf00bc001df6ef3ffed864e7458bce0817f57d3479eabf5c25724b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:42:00.625 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:42:00 smithi179 podman[80365]: 2024-05-27 07:42:00.345717753 +0000 UTC m=+0.494640211 container start 2d77f3b88cf00bc001df6ef3ffed864e7458bce0817f57d3479eabf5c25724b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:42:00.625 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:42:00 smithi179 podman[80365]: 2024-05-27 07:42:00.459526996 +0000 UTC m=+0.608449480 container attach 2d77f3b88cf00bc001df6ef3ffed864e7458bce0817f57d3479eabf5c25724b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T07:42:00.966 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:00.626+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:00.966 INFO:journalctl@ceph.osd.2.smithi179.stdout:May 27 07:42:00 smithi179 podman[80365]: 2024-05-27 07:42:00.804927849 +0000 UTC m=+0.953850343 container died 2d77f3b88cf00bc001df6ef3ffed864e7458bce0817f57d3479eabf5c25724b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-2-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:42:01.190 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:42:00.949+0000 7f3392000700 -1 osd.3 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.403103+0000 front 2024-05-27T07:41:33.403253+0000 (oldest deadline 2024-05-27T07:41:58.703384+0000) 2024-05-27T07:42:01.191 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:00.997+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:01.567 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:01.348+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:01.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:01 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:01.203+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:01.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.2.service' 2024-05-27T07:42:01.749 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:42:01.749 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-27T07:42:01.750 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-27T07:42:01.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.3 2024-05-27T07:42:01.827 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:01.477+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:01.966 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:01.579+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:01.967 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:01.569+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:02.132 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:02.025+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:02.132 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:01 smithi043 systemd[1]: Stopping Ceph osd.3 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:42:02.132 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:42:01.950+0000 7f3392000700 -1 osd.3 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.403103+0000 front 2024-05-27T07:41:33.403253+0000 (oldest deadline 2024-05-27T07:41:58.703384+0000) 2024-05-27T07:42:02.444 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:42:02.317+0000 7f3396800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:42:02.444 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:42:02.317+0000 7f3396800700 -1 osd.3 165 *** Got signal Terminated *** 2024-05-27T07:42:02.444 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:42:02.317+0000 7f3396800700 -1 osd.3 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:42:02.598 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:02 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:02.202+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:02.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:02.445+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:02.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:02 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:02.586+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:02.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:02 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:02.601+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:02.716 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:02 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:02.357+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:03.210 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:42:02.903+0000 7f3392000700 -1 osd.3 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.403103+0000 front 2024-05-27T07:41:33.403253+0000 (oldest deadline 2024-05-27T07:41:58.703384+0000) 2024-05-27T07:42:03.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:02.999+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:03.598 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:03 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:03.230+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:03.621 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:03 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:03.594+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:03.621 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:03 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:03.358+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:03.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:03.449+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:03.966 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:03 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:03.621+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:04.210 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:42:03.927+0000 7f3392000700 -1 osd.3 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.403103+0000 front 2024-05-27T07:41:33.403253+0000 (oldest deadline 2024-05-27T07:41:58.703384+0000) 2024-05-27T07:42:04.211 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:03.975+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:04.598 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:04 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:04.202+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:04.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:04 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:04.608+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:04.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:04 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:04.363+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:04.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:04 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:04.608+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:04.937 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:04.473+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:05.210 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:42:04.938+0000 7f3392000700 -1 osd.3 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.403103+0000 front 2024-05-27T07:41:33.403253+0000 (oldest deadline 2024-05-27T07:41:58.703384+0000) 2024-05-27T07:42:05.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:04.969+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:05.500 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:05 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:05.245+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:05.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:05.570+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:05.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:05.370+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:05.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:05.606+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:05.918 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:05 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:05.474+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:06.210 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:05 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:42:05.919+0000 7f3392000700 -1 osd.3 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.403103+0000 front 2024-05-27T07:41:33.403253+0000 (oldest deadline 2024-05-27T07:41:58.703384+0000) 2024-05-27T07:42:06.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:05 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:05.923+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:06.598 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:06 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:06.263+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:06.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:06 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:06.434+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:06.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:06.573+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:06.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:06.607+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:06.716 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:06.361+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:07.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:06 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:06.947+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:07.210 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:06 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3[50593]: 2024-05-27T07:42:06.920+0000 7f3392000700 -1 osd.3 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.403103+0000 front 2024-05-27T07:41:33.403253+0000 (oldest deadline 2024-05-27T07:41:58.703384+0000) 2024-05-27T07:42:07.598 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:07 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:07.293+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:07.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:07.461+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:07.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:07 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:07.604+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:07.716 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:07 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:07.385+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:07.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:07 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:07.566+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:08.332 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:08 smithi043 podman[141147]: 2024-05-27 07:42:08.001212245 +0000 UTC m=+5.828343614 container died f7e9ba69131c308042acc1219e1bac0ae07296a53319058548a2e90ee5b8c7b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-27T07:42:08.332 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:07 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:07.986+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:08.598 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:08 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:08.251+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:08.696 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:08 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:08.468+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:08.696 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:08 smithi043 podman[141147]: 2024-05-27 07:42:08.470619741 +0000 UTC m=+6.297751106 container cleanup f7e9ba69131c308042acc1219e1bac0ae07296a53319058548a2e90ee5b8c7b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-27T07:42:08.696 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:08 smithi043 bash[141147]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3 2024-05-27T07:42:08.716 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:08.398+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:08.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:08.627+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:08.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:08.627+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:08.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:08.535+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:08.956 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:08 smithi043 podman[141165]: 2024-05-27 07:42:08.696943461 +0000 UTC m=+0.684089772 container remove f7e9ba69131c308042acc1219e1bac0ae07296a53319058548a2e90ee5b8c7b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:42:09.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:08 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:08.958+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:09.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:08 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:08.958+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:09.598 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:09.241+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:09.599 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:09.166+0000 7fcf2dc00700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.740145+0000 front 2024-05-27T07:41:46.740209+0000 (oldest deadline 2024-05-27T07:42:09.040483+0000) 2024-05-27T07:42:09.716 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:09.400+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:09.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:09.507+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:09.717 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:09.586+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:09.717 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:09.586+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:09.744 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:09 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:09.462+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:09.744 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:09 smithi043 podman[141255]: 2024-05-27 07:42:09.456543356 +0000 UTC m=+0.035072846 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:42:09.744 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:09 smithi043 podman[141255]: 2024-05-27 07:42:09.572769543 +0000 UTC m=+0.151299071 container create 314e10cc77b5f5a26063ee0f7d39deb32565d71f2cc62c9b5ce3fd5225bbc9d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-27T07:42:10.210 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:09 smithi043 podman[141255]: 2024-05-27 07:42:09.837642813 +0000 UTC m=+0.416172355 container init 314e10cc77b5f5a26063ee0f7d39deb32565d71f2cc62c9b5ce3fd5225bbc9d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:42:10.210 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:09 smithi043 podman[141255]: 2024-05-27 07:42:09.849498602 +0000 UTC m=+0.428028140 container start 314e10cc77b5f5a26063ee0f7d39deb32565d71f2cc62c9b5ce3fd5225bbc9d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True) 2024-05-27T07:42:10.210 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:09 smithi043 podman[141255]: 2024-05-27 07:42:09.925264971 +0000 UTC m=+0.503794542 container attach 314e10cc77b5f5a26063ee0f7d39deb32565d71f2cc62c9b5ce3fd5225bbc9d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-27T07:42:10.211 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:09 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:09.910+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:10.211 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:09 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:09.910+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:10.598 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:10 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:10.186+0000 7fcf2dc00700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.740145+0000 front 2024-05-27T07:41:46.740209+0000 (oldest deadline 2024-05-27T07:42:09.040483+0000) 2024-05-27T07:42:10.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:10 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:10.239+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:10.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:10 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:10.239+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:10.673 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 27 07:42:10 smithi043 podman[141255]: 2024-05-27 07:42:10.326309469 +0000 UTC m=+0.904839047 container died 314e10cc77b5f5a26063ee0f7d39deb32565d71f2cc62c9b5ce3fd5225bbc9d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-27T07:42:10.673 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:10.468+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:10.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:10.599+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:10.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:10.599+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:10.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:10.449+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:10.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:10.489+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:10.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:10.489+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:11.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.3.service' 2024-05-27T07:42:11.203 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:42:11.203 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-27T07:42:11.203 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-27T07:42:11.203 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.4 2024-05-27T07:42:11.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:10.950+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:11.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:10 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:10.950+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:11.599 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:11.181+0000 7fcf2dc00700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.740145+0000 front 2024-05-27T07:41:46.740209+0000 (oldest deadline 2024-05-27T07:42:09.040483+0000) 2024-05-27T07:42:11.599 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:11 smithi132 systemd[1]: Stopping Ceph osd.4 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:42:11.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:11.212+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:11.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:11.212+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:11.716 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:11.483+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:11.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:11.584+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:11.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:11.584+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:11.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:11.454+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:11.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:11.454+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:11.956 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:11 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:11.509+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:11.956 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:11 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:11.509+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:12.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:11.797+0000 7fcf32400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:42:12.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:11.797+0000 7fcf32400700 -1 osd.4 165 *** Got signal Terminated *** 2024-05-27T07:42:12.099 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:11.797+0000 7fcf32400700 -1 osd.4 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:42:12.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:11 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:11.957+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:12.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:11 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:11.957+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:12.598 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:12 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:12.216+0000 7fcf2dc00700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.740145+0000 front 2024-05-27T07:41:46.740209+0000 (oldest deadline 2024-05-27T07:42:09.040483+0000) 2024-05-27T07:42:12.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:12 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:12.235+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:12.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:12 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:12.235+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:12.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:12.482+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:12.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:12.482+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:12.717 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:12.615+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:12.717 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:12.615+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:12.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:12.448+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:12.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:12.448+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:12.908 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:12.492+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:12.908 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:12.492+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:13.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:12.909+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:13.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:12 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:12.909+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:13.598 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:13.235+0000 7fcf2dc00700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.740145+0000 front 2024-05-27T07:41:46.740209+0000 (oldest deadline 2024-05-27T07:42:09.040483+0000) 2024-05-27T07:42:13.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:13.252+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:13.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:13.252+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:13.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:13.146+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:13.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:13.622+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:13.716 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:13.622+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:13.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:13.496+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:13.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:13.496+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:13.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:13.410+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:13.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:13.410+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:13.921 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:13 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:13.513+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:13.921 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:13 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:13.513+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:14.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:13 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:13.922+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:14.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:13 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:13.922+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:14.598 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:14.228+0000 7fcf2dc00700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.740145+0000 front 2024-05-27T07:41:46.740209+0000 (oldest deadline 2024-05-27T07:42:09.040483+0000) 2024-05-27T07:42:14.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:14.162+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:14.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:14.278+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:14.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:14.278+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:14.670 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:14.409+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:14.670 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:14.409+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:14.670 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:14.466+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:14.670 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:14.466+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:14.897 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:14 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:14.473+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:14.898 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:14 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:14.473+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:14.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:14.670+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:14.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:14.670+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:15.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:14 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:14.899+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:15.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:14 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:14.899+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:15.598 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:15 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:15.187+0000 7fcf2dc00700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.740145+0000 front 2024-05-27T07:41:46.740209+0000 (oldest deadline 2024-05-27T07:42:09.040483+0000) 2024-05-27T07:42:15.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:15 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:15.196+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:15.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:15 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:15.235+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:15.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:15 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:15.235+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:15.664 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:15.476+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:15.664 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:15.476+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:15.665 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:15.389+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:15.665 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:15.389+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:15.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:15 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:15.428+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:15.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:15 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:15.428+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:15.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:15.665+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:15.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:15.665+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:16.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:15 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:15.871+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:16.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:15 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:15.871+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:16.583 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:16 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:16.284+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:16.584 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:16 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:16.284+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:16.584 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:16 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4[46875]: 2024-05-27T07:42:16.162+0000 7fcf2dc00700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.740145+0000 front 2024-05-27T07:41:46.740209+0000 (oldest deadline 2024-05-27T07:42:09.040483+0000) 2024-05-27T07:42:16.584 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:16 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:16.210+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:16.624 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:16.454+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:16.624 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:16.454+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:16.625 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:16.346+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:16.625 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:16.346+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:16.904 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:16 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:16.467+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:16.904 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:16 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:16.467+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:16.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:16.626+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:16.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:16.626+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:17.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:16 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:16.905+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:17.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:16 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:16.905+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:17.512 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:17 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:17.263+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:17.512 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:17 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:17.263+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:17.512 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:17 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:17.207+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:17.610 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:17.434+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:17.610 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:17.434+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:17.610 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:17.306+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:17.611 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:17.306+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:17.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:17.425+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:17.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:17.425+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:17.830 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:17 smithi132 podman[77776]: 2024-05-27 07:42:17.513895393 +0000 UTC m=+5.862430210 container died b63730da29c2778894f7f27df44d5e3483b97e30eb8e3a8ed580db3ccbf6526f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:42:17.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:17.611+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:17.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:17.611+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:18.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:17 smithi132 podman[77776]: 2024-05-27 07:42:17.998983149 +0000 UTC m=+6.347518004 container cleanup b63730da29c2778894f7f27df44d5e3483b97e30eb8e3a8ed580db3ccbf6526f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-05-27T07:42:18.098 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:17 smithi132 bash[77776]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4 2024-05-27T07:42:18.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:17.913+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:18.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:17 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:17.913+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:18.349 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:18 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:18.171+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:18.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:18 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:18.285+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:18.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:18 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:18.285+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:18.631 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:18.390+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:18.631 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:18.390+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:18.631 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:18.309+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:18.631 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:18.309+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:18.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:18 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:18.459+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:18.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:18 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:18.459+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:18.848 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:18 smithi132 podman[77793]: 2024-05-27 07:42:18.351164021 +0000 UTC m=+0.820170753 container remove b63730da29c2778894f7f27df44d5e3483b97e30eb8e3a8ed580db3ccbf6526f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-05-27T07:42:18.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:18.632+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:18.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:18.632+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:19.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:18 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:18.953+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:19.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:18 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:18.953+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:19.348 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:19 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:19.236+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:19.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:19 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:19.236+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:19.349 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:19 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:19.158+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:19.349 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:19 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:19.158+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:19.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:19 smithi132 podman[77882]: 2024-05-27 07:42:18.929405521 +0000 UTC m=+0.035767509 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:42:19.349 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:19 smithi132 podman[77882]: 2024-05-27 07:42:19.068821636 +0000 UTC m=+0.175183661 container create 0f24f28d010078f2f1ac0d0d18ea15e386ecab9e5107e4a36adb32f9404213b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4-deactivate, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-05-27T07:42:19.655 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:19.312+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:19.655 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:19.312+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:19.655 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:19.350+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:19.656 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:19.350+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:19.656 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:19.350+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:19.849 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:19 smithi132 podman[77882]: 2024-05-27 07:42:19.38945622 +0000 UTC m=+0.495818315 container init 0f24f28d010078f2f1ac0d0d18ea15e386ecab9e5107e4a36adb32f9404213b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:42:19.849 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:19 smithi132 podman[77882]: 2024-05-27 07:42:19.401258197 +0000 UTC m=+0.507620256 container start 0f24f28d010078f2f1ac0d0d18ea15e386ecab9e5107e4a36adb32f9404213b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-27T07:42:19.849 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:19 smithi132 podman[77882]: 2024-05-27 07:42:19.513456451 +0000 UTC m=+0.619818500 container attach 0f24f28d010078f2f1ac0d0d18ea15e386ecab9e5107e4a36adb32f9404213b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:42:19.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:19 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:19.509+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:19.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:19 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:19.509+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:19.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:19 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:19.509+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:19.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:19.656+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:19.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:19.656+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:20.130 INFO:journalctl@ceph.osd.4.smithi132.stdout:May 27 07:42:19 smithi132 podman[77882]: 2024-05-27 07:42:19.856150624 +0000 UTC m=+0.962512679 container died 0f24f28d010078f2f1ac0d0d18ea15e386ecab9e5107e4a36adb32f9404213b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T07:42:20.417 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:20 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:20.003+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:20.417 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:20 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:20.003+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:20.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:20 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:20.130+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:20.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:20 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:20.130+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:20.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:20 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:20.226+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:20.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:20 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:20.226+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:20.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:20 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:20.226+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:20.665 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:20.308+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:20.666 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:20.309+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:20.666 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:20.347+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:20.666 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:20.347+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:20.666 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:20.347+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:20.699 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.4.service' 2024-05-27T07:42:20.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:20 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:20.532+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:20.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:20 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:20.532+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:20.710 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:20 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:20.532+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:20.806 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:42:20.806 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-27T07:42:20.806 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-27T07:42:20.806 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.5 2024-05-27T07:42:20.920 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:20.667+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:20.920 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:20.667+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:21.216 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:20 smithi179 systemd[1]: Stopping Ceph osd.5 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:42:21.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:20 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:20.984+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:21.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:20 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:20.984+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:21.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:21 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:21.142+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:21.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:21 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:21.142+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:21.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:21 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:21.271+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:21.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:21 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:21.271+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:21.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:21 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:21.271+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:21.697 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:21.381+0000 7f6cec400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:42:21.697 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:21.381+0000 7f6cec400700 -1 osd.5 165 *** Got signal Terminated *** 2024-05-27T07:42:21.697 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:21.381+0000 7f6cec400700 -1 osd.5 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:42:21.697 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:21.332+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:21.697 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:21.332+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:21.698 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:21.348+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:21.698 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:21.348+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:21.698 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:21.348+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:21.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:21.551+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:21.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:21.551+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:21.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:21.551+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:21.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:21.698+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:21.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:21.698+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:21.967 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:21.698+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.156732+0000 front 2024-05-27T07:41:55.156805+0000 (oldest deadline 2024-05-27T07:42:21.057018+0000) 2024-05-27T07:42:22.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:21.985+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:22.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:21 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:21.985+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:22.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:22 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:22.188+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:22.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:22 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:22.188+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:22.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:22 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:22.291+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:22.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:22 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:22.291+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:22.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:22 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:22.291+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:22.694 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:22.317+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:22.694 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:22.317+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:22.694 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:22.317+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:22.694 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:22.347+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:22.694 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:22.347+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:22.694 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:22.347+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:22.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:22.552+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:22.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:22.552+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:22.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:22 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:22.552+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:22.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:22.694+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:22.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:22.694+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:22.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:22.694+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.156732+0000 front 2024-05-27T07:41:55.156805+0000 (oldest deadline 2024-05-27T07:42:21.057018+0000) 2024-05-27T07:42:23.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:23 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:23.027+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:23.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:23 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:23.027+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:23.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:23 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:23.027+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:23.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:23 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:23.198+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:23.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:23 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:23.198+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:23.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:23 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:23.334+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:23.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:23 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:23.334+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:23.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:23 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:23.334+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:23.652 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:23.318+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:23.652 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:23.318+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:23.652 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:23.318+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:23.652 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:23.361+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:23.652 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:23.361+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:23.652 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:23.361+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:23.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:23 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:23.600+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:23.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:23 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:23.600+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:23.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:23 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:23.600+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:23.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:23.652+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:23.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:23.652+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:23.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:23.652+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.156732+0000 front 2024-05-27T07:41:55.156805+0000 (oldest deadline 2024-05-27T07:42:21.057018+0000) 2024-05-27T07:42:24.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:24.031+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:24.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:24.031+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:24.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:24.031+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:24.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:24 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:24.171+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:24.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:24 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:24.171+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:24.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:24 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:24.287+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:24.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:24 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:24.287+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:24.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:24 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:24.287+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:24.618 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:24 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:24.327+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:24.619 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:24 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:24.327+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:24.619 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:24 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:24.327+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:24.619 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:24 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:24.346+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:24.619 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:24 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:24.346+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:24.619 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:24 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:24.346+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:24.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:24.551+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:24.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:24.551+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:24.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:24 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:24.551+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:24.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:24 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:24.619+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:24.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:24 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:24.619+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:24.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:24 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:24.619+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.156732+0000 front 2024-05-27T07:41:55.156805+0000 (oldest deadline 2024-05-27T07:42:21.057018+0000) 2024-05-27T07:42:25.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:25.067+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:25.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:25.067+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:25.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:25.067+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:25.591 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:25 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:25.315+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:25.591 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:25 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:25.315+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:25.591 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:25 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:25.315+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:25.591 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:25 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:25.328+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:25.591 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:25 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:25.328+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:25.591 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:25 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:25.328+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:25.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:25 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:25.221+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:25.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:25 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:25.221+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:25.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:25 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:25.269+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:25.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:25 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:25.269+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:25.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:25 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:25.269+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:25.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:25.529+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:25.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:25.529+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:25.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:25 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:25.529+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:25.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:25 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:25.592+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.346120+0000 front 2024-05-27T07:41:35.346033+0000 (oldest deadline 2024-05-27T07:42:01.246287+0000) 2024-05-27T07:42:25.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:25 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:25.592+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:42.349442+0000 front 2024-05-27T07:41:42.349562+0000 (oldest deadline 2024-05-27T07:42:08.151681+0000) 2024-05-27T07:42:25.966 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:25 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5[47520]: 2024-05-27T07:42:25.592+0000 7f6ce7c00700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.156732+0000 front 2024-05-27T07:41:55.156805+0000 (oldest deadline 2024-05-27T07:42:21.057018+0000) 2024-05-27T07:42:26.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:26 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:26.046+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:26.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:26 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:26.046+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:26.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:26 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:26.046+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:26.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:26 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:26.185+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:26.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:26 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:26.185+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:26.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:26 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:26.272+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:26.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:26 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:26.272+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:26.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:26 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:26.272+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:26.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:26.301+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:26.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:26.301+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:26.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:26.301+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:26.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:26.348+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:26.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:26.348+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:26.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:26 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:26.348+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:26.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:26 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:26.558+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:26.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:26 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:26.558+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:26.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:26 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:26.558+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:27.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:27 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:27.036+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:27.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:27 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:27.036+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:27.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:27 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:27.036+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:27.466 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:27 smithi179 podman[80471]: 2024-05-27 07:42:27.163598377 +0000 UTC m=+5.936285034 container died 9da519f3c2d74e3eabb9db34de132f7ae97b5c04e0f15eed57e9a0597a199c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-27T07:42:27.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:27.347+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:27.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:27.347+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:27.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:27.347+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:27.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:27.307+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:27.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:27.307+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:27.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:27 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:27.307+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:27.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:27 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:27.207+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:27.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:27 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:27.207+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:27.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:27 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:27.306+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:27.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:27 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:27.306+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:27.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:27 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:27.306+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:27.941 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:27 smithi179 podman[80471]: 2024-05-27 07:42:27.658105437 +0000 UTC m=+6.430792094 container cleanup 9da519f3c2d74e3eabb9db34de132f7ae97b5c04e0f15eed57e9a0597a199c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-27T07:42:27.941 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:27 smithi179 bash[80471]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5 2024-05-27T07:42:27.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:27 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:27.542+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:27.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:27 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:27.542+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:27.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:27 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:27.542+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:28.216 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:27 smithi179 podman[80490]: 2024-05-27 07:42:27.942971226 +0000 UTC m=+0.764590627 container remove 9da519f3c2d74e3eabb9db34de132f7ae97b5c04e0f15eed57e9a0597a199c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:42:28.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:27 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:27.990+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:28.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:27 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:27.990+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:28.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:27 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:27.990+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:28.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:28 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:28.226+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:28.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:28 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:28.226+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:28.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:28 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:28.271+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:28.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:28 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:28.271+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:28.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:28 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:28.271+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:28.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:28 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:28.271+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:28.716 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:28 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:28.327+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:28.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:28 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:28.327+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:28.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:28 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:28.327+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:28.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:28 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:28.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:28.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:28 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:28.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:28.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:28 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:28.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:28.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:28.586+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:28.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:28.586+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:28.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:28.586+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:28.985 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:28 smithi179 podman[80578]: 2024-05-27 07:42:28.63781752 +0000 UTC m=+0.035020270 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:42:28.985 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:28 smithi179 podman[80578]: 2024-05-27 07:42:28.777305061 +0000 UTC m=+0.174507847 container create baa4cbbd7b1f8571f2c01472705d103ef3c66629487be59293c48698bf447448 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=) 2024-05-27T07:42:29.305 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:29 smithi179 podman[80578]: 2024-05-27 07:42:29.098253447 +0000 UTC m=+0.495456280 container init baa4cbbd7b1f8571f2c01472705d103ef3c66629487be59293c48698bf447448 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-27T07:42:29.305 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:29 smithi179 podman[80578]: 2024-05-27 07:42:29.109791025 +0000 UTC m=+0.506993818 container start baa4cbbd7b1f8571f2c01472705d103ef3c66629487be59293c48698bf447448 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:42:29.306 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:29 smithi179 podman[80578]: 2024-05-27 07:42:29.213562031 +0000 UTC m=+0.610764844 container attach baa4cbbd7b1f8571f2c01472705d103ef3c66629487be59293c48698bf447448 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-27T07:42:29.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:28.990+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:29.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:28.990+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:29.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:28 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:28.991+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:29.575 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:29.306+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:29.576 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:29.306+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:29.576 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:29.306+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:29.576 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:29.353+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:29.576 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:29.353+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:29.576 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:29 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:29.353+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:29.598 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:29 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:29.234+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:29.598 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:29 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:29.234+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:29.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:29 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:29.265+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:29.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:29 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:29.265+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:29.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:29 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:29.265+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:29.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:29 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:29.265+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:29.948 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:29 smithi179 podman[80578]: 2024-05-27 07:42:29.57862583 +0000 UTC m=+0.975828651 container died baa4cbbd7b1f8571f2c01472705d103ef3c66629487be59293c48698bf447448 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-27T07:42:29.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:29 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:29.545+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:29.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:29 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:29.545+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:29.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:29 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:29.545+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:29.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:29 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:29.545+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:30.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:29 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:29.964+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:30.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:29 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:29.964+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:30.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:29 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:29.964+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:30.468 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.5.service' 2024-05-27T07:42:30.527 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:30 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:30.288+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:30.527 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:30 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:30.288+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:30.527 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:30 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:30.288+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:30.527 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:30 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:30.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:30.528 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:30 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:30.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:30.528 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:30 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:30.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:30.528 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:30 smithi179 podman[80603]: 2024-05-27 07:42:30.430054331 +0000 UTC m=+0.833762792 container remove baa4cbbd7b1f8571f2c01472705d103ef3c66629487be59293c48698bf447448 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-27T07:42:30.528 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:30 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.5.service: Deactivated successfully. 2024-05-27T07:42:30.528 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:30 smithi179 systemd[1]: Stopped Ceph osd.5 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968. 2024-05-27T07:42:30.528 INFO:journalctl@ceph.osd.5.smithi179.stdout:May 27 07:42:30 smithi179 systemd[1]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.5.service: Consumed 8min 29.996s CPU time. 2024-05-27T07:42:30.558 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:42:30.559 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-27T07:42:30.559 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-27T07:42:30.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.6 2024-05-27T07:42:30.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:30.207+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:30.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:30.207+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:30.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:30.268+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:30.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:30.268+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:30.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:30.268+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:30.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:30 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:30.268+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:30.956 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:30 smithi043 systemd[1]: Stopping Ceph osd.6 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:42:30.994 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:30 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:30.574+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:30.994 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:30 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:30.574+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:30.995 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:30 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:30.574+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:30.995 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:30 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:30.574+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:31.212 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:30 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:30.957+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:31.213 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:30 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:30.957+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:31.213 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:30 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:30.957+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:31.213 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:30 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:30.957+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:02.310357+0000 front 2024-05-27T07:42:02.310439+0000 (oldest deadline 2024-05-27T07:42:30.512571+0000) 2024-05-27T07:42:31.213 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:31.120+0000 7f8a0aa00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:42:31.213 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:31.120+0000 7f8a0aa00700 -1 osd.6 165 *** Got signal Terminated *** 2024-05-27T07:42:31.213 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:31.120+0000 7f8a0aa00700 -1 osd.6 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:42:31.470 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:31 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:31.249+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:31.470 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:31 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:31.249+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:31.470 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:31 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:31.249+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:31.471 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:31 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:31.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:31.471 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:31 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:31.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:31.471 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:31 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:31.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:31.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:31 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:31.257+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:31.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:31 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:31.257+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:31.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:31 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:31.257+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:31.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:31 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:31.258+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:31.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:31 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:31.250+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:31.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:31 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:31.250+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:31.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:31 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:31.250+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:31.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:31.531+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:31.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:31.531+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:31.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:31.531+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:31.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:31 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:31.531+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:32.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:32 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:32.007+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:32.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:32 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:32.007+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:32.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:32 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:32.007+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:32.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:32 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:32.007+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:02.310357+0000 front 2024-05-27T07:42:02.310439+0000 (oldest deadline 2024-05-27T07:42:30.512571+0000) 2024-05-27T07:42:32.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:32 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:32.204+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:32.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:32 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:32.204+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:32.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:32 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:32.204+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:32.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:32 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:32.278+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:32.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:32 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:32.278+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:32.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:32 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:32.278+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:32.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:32 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:32.278+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:32.716 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:32 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:32.297+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:32.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:32 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:32.297+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:32.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:32 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:32.297+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:32.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:32 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:32.306+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:32.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:32 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:32.306+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:32.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:32 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:32.306+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:32.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:32 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:32.577+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:32.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:32 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:32.577+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:32.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:32 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:32.577+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:32.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:32 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:32.577+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:33.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:33.025+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:33.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:33.025+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:33.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:33.025+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:33.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:33.025+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:02.310357+0000 front 2024-05-27T07:42:02.310439+0000 (oldest deadline 2024-05-27T07:42:30.512571+0000) 2024-05-27T07:42:33.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:33.302+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:33.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:33.302+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:33.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:33.302+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:33.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:33.302+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:33.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:33.239+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:33.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:33.239+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:33.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:33 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:33.239+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:33.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:33.316+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:33.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:33.316+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:33.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:33.316+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:33.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:33.279+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:33.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:33.279+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:33.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:33 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:33.279+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:33.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:33.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:33.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:33.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:33.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:33.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:33.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:33.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:34.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:33.990+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:34.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:33.990+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:34.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:33.990+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:34.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:33 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:33.990+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:02.310357+0000 front 2024-05-27T07:42:02.310439+0000 (oldest deadline 2024-05-27T07:42:30.512571+0000) 2024-05-27T07:42:34.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:34 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:34.244+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:34.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:34 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:34.244+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:34.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:34 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:34.244+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:34.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:34 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:34.351+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:34.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:34 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:34.351+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:34.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:34 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:34.351+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:34.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:34 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:34.351+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:34.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:34 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:34.360+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:34.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:34 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:34.360+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:34.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:34 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:34.360+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:34.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:34 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:34.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:34.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:34 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:34.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:34.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:34 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:34.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:34.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:34 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:34.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:34.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:34.561+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:34.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:34.561+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:34.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:34.561+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:34.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:34.561+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:35.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:34.975+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:35.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:34.975+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:35.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:34.975+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:35.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:34 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:34.975+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:02.310357+0000 front 2024-05-27T07:42:02.310439+0000 (oldest deadline 2024-05-27T07:42:30.512571+0000) 2024-05-27T07:42:35.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:35 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:35.281+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:35.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:35 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:35.281+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:35.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:35 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:35.281+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:35.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:35 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:35.310+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:35.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:35 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:35.310+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:35.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:35 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:35.310+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:35.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:35 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:35.310+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:35.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:35 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:35.353+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:35.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:35 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:35.353+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:35.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:35 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:35.353+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:35.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:35 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:35.258+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:35.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:35 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:35.258+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:35.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:35 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:35.258+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:35.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:35 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:35.258+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:35.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:35 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:35.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:35.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:35 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:35.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:35.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:35 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:35.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:35.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:35 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:35.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:36.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:35 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:35.981+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.794658+0000 front 2024-05-27T07:41:33.794571+0000 (oldest deadline 2024-05-27T07:41:58.494874+0000) 2024-05-27T07:42:36.460 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:35 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:35.981+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:46.602651+0000 front 2024-05-27T07:41:46.602567+0000 (oldest deadline 2024-05-27T07:42:08.804523+0000) 2024-05-27T07:42:36.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:35 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:35.981+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.009037+0000 front 2024-05-27T07:41:57.008944+0000 (oldest deadline 2024-05-27T07:42:22.309134+0000) 2024-05-27T07:42:36.461 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:35 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6[57231]: 2024-05-27T07:42:35.981+0000 7f8a01200700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:02.310357+0000 front 2024-05-27T07:42:02.310439+0000 (oldest deadline 2024-05-27T07:42:30.512571+0000) 2024-05-27T07:42:36.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:36 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:36.317+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:36.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:36 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:36.317+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:36.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:36 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:36.317+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:36.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:36 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:36.334+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:36.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:36 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:36.334+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:36.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:36 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:36.334+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:36.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:36 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:36.334+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:36.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:36 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:36.365+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:36.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:36 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:36.365+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:36.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:36 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:36.365+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:36.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:36 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:36.287+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:36.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:36 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:36.287+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:36.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:36 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:36.287+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:36.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:36 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:36.287+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:36.960 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:36 smithi043 podman[141360]: 2024-05-27 07:42:36.766221568 +0000 UTC m=+5.782899617 container died 829e3cbd626c03796ee16dc3877135983e235f5d9b9a44236ae33bc4bd076300 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-27T07:42:36.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:36 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:36.568+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:36.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:36 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:36.568+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:36.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:36 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:36.569+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:36.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:36 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:36.569+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:37.339 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:37 smithi043 podman[141360]: 2024-05-27 07:42:37.314354423 +0000 UTC m=+6.331032466 container cleanup 829e3cbd626c03796ee16dc3877135983e235f5d9b9a44236ae33bc4bd076300 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:42:37.339 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:37 smithi043 bash[141360]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6 2024-05-27T07:42:37.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:37.339+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:37.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:37.339+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:37.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:37.339+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:37.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:37.381+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:37.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:37.381+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:37.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:37.381+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:37.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:37 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:37.381+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:37.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:37 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:37.356+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:37.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:37 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:37.356+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:37.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:37 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:37.356+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:37.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:37 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:37.245+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:37.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:37 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:37.245+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:37.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:37 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:37.245+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:37.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:37 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:37.245+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:37.960 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:37 smithi043 podman[141378]: 2024-05-27 07:42:37.574616724 +0000 UTC m=+0.800399360 container remove 829e3cbd626c03796ee16dc3877135983e235f5d9b9a44236ae33bc4bd076300 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:42:37.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:37.612+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:37.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:37.612+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:37.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:37.612+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:37.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:37 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:37.612+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:38.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:38 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:38.354+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:38.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:38 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:38.354+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:38.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:38 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:38.354+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:38.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:38 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:38.340+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:38.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:38 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:38.340+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:38.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:38 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:38.340+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:38.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:38 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:38.340+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:38.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:38 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:38.340+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:38.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:38 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:38.355+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:38.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:38 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:38.355+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:38.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:38 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:38.355+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:38.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:38 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:38.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:38.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:38 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:38.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:38.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:38 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:38.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:38.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:38 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:38.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:38.777 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:38 smithi043 podman[141466]: 2024-05-27 07:42:38.390862015 +0000 UTC m=+0.034334857 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:42:38.778 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:38 smithi043 podman[141466]: 2024-05-27 07:42:38.508938317 +0000 UTC m=+0.152411218 container create 6bb3eb772b9515a9b8b3cb8d87cf4f3324715db5dfe5c97344290f39107d0272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521) 2024-05-27T07:42:38.778 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:38 smithi043 podman[141466]: 2024-05-27 07:42:38.779257988 +0000 UTC m=+0.422730913 container init 6bb3eb772b9515a9b8b3cb8d87cf4f3324715db5dfe5c97344290f39107d0272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-27T07:42:38.778 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:38.580+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:38.778 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:38.580+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:38.778 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:38.580+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:38.778 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:38 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:38.580+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:39.210 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:38 smithi043 podman[141466]: 2024-05-27 07:42:38.790898345 +0000 UTC m=+0.434371199 container start 6bb3eb772b9515a9b8b3cb8d87cf4f3324715db5dfe5c97344290f39107d0272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:42:39.211 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:38 smithi043 podman[141466]: 2024-05-27 07:42:38.886169438 +0000 UTC m=+0.529642376 container attach 6bb3eb772b9515a9b8b3cb8d87cf4f3324715db5dfe5c97344290f39107d0272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux ) 2024-05-27T07:42:39.533 INFO:journalctl@ceph.osd.6.smithi043.stdout:May 27 07:42:39 smithi043 podman[141466]: 2024-05-27 07:42:39.262878459 +0000 UTC m=+0.906351370 container died 6bb3eb772b9515a9b8b3cb8d87cf4f3324715db5dfe5c97344290f39107d0272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-27T07:42:39.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:39.349+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:39.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:39.349+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:39.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:39.349+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:39.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:39.298+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:39.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:39.298+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:39.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:39.298+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:39.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:39.298+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:39.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:39.298+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:39.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:39 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:39.318+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:39.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:39 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:39.318+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:39.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:39 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:39.318+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:39.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:39 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:39.251+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:39.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:39 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:39.251+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:39.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:39 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:39.251+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:39.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:39 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:39.251+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:39.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:39 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:39.251+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:39.955 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:39 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:39.544+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:39.955 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:39 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:39.544+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:39.955 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:39 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:39.544+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:39.956 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:39 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:39.544+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:39.956 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:39 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:39.544+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:40.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.6.service' 2024-05-27T07:42:40.119 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:42:40.119 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-27T07:42:40.119 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-27T07:42:40.119 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.7 2024-05-27T07:42:40.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:40.254+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:40.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:40.254+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:40.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:40.254+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:40.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:40.254+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:40.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:40.254+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:40.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:40 smithi132 systemd[1]: Stopping Ceph osd.7 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:42:40.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:40.395+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:40.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:40.395+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:40.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:40.395+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:40.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:40.395+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.964840+0000 front 2024-05-27T07:42:13.964912+0000 (oldest deadline 2024-05-27T07:42:39.865007+0000) 2024-05-27T07:42:40.624 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:40.304+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:40.624 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:40.304+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:40.624 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:40.304+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:40.625 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:40.290+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:40.625 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:40.290+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:40.625 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:40.290+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:40.625 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:40.290+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:40.625 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:40 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:40.290+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:40.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:40 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:40.590+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:40.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:40 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:40.590+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:40.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:40 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:40.590+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:40.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:40 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:40.590+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:40.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:40 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:40.590+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:41.099 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:40.675+0000 7f2befc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:42:41.099 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:40.675+0000 7f2befc00700 -1 osd.7 165 *** Got signal Terminated *** 2024-05-27T07:42:41.099 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:40 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:40.675+0000 7f2befc00700 -1 osd.7 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:42:41.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:41 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:41.379+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:41.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:41 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:41.379+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:41.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:41 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:41.379+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:41.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:41 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:41.379+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.964840+0000 front 2024-05-27T07:42:13.964912+0000 (oldest deadline 2024-05-27T07:42:39.865007+0000) 2024-05-27T07:42:41.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:41 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:41.264+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:41.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:41 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:41.264+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:41.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:41 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:41.264+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:41.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:41 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:41.264+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:41.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:41 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:41.264+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:41.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:41 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:41.333+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:41.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:41 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:41.333+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:41.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:41 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:41.333+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:41.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:41 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:41.333+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:41.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:41 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:41.333+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:41.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:41 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:41.276+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:41.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:41 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:41.276+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:41.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:41 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:41.276+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:41.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:41 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:41.276+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:41.850 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:41 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:41.547+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:41.850 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:41 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:41.547+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:41.850 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:41 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:41.547+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:41.850 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:41 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:41.547+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:41.850 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:41 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:41.547+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:42.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:42.215+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:42.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:42.215+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:42.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:42.215+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:42.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:42.215+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:42.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:42.215+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:42.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:42.338+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:42.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:42.338+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:42.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:42.338+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:42.600 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:42 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:42.338+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.964840+0000 front 2024-05-27T07:42:13.964912+0000 (oldest deadline 2024-05-27T07:42:39.865007+0000) 2024-05-27T07:42:42.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:42 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:42.292+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:42.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:42 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:42.292+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:42.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:42 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:42.292+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:42.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:42 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:42.292+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:42.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:42 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:42.292+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:42.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:42 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:42.292+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:42.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:42 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:42.292+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:42.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:42 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:42.292+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:42.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:42 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:42.292+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:42.807 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:42.547+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:42.807 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:42.547+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:42.808 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:42.547+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:42.808 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:42.547+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:42.808 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:42 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:42.547+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:43.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:43 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:43.326+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:43.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:43 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:43.326+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:43.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:43 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:43.326+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:43.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:43 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:43.326+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.964840+0000 front 2024-05-27T07:42:13.964912+0000 (oldest deadline 2024-05-27T07:42:39.865007+0000) 2024-05-27T07:42:43.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:43 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:43.179+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:43.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:43 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:43.179+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:43.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:43 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:43.179+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:43.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:43 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:43.179+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:43.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:43 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:43.179+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:43.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:43.323+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:43.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:43.323+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:43.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:43.323+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:43.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:43.323+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:43.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:43.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:43.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:43.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:43.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:43.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:43.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:43.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:43.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:43 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:43.253+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:43.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:43 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:43.516+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:43.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:43 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:43.516+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:43.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:43 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:43.516+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:43.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:43 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:43.516+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:43.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:43 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:43.516+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:44.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:44.375+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:44.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:44.376+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:44.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:44.376+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:44.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:44.376+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.964840+0000 front 2024-05-27T07:42:13.964912+0000 (oldest deadline 2024-05-27T07:42:39.865007+0000) 2024-05-27T07:42:44.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:44.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:44.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:44.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:44.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:44.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:44.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:44.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:44.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:44 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:44.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:44.716 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:44.270+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:44.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:44.270+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:44.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:44.270+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:44.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:44.270+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:44.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:44.270+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:44.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:44.335+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:44.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:44.335+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:44.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:44.335+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:44.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:44 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:44.335+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:44.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:44.554+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:44.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:44.554+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:44.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:44.554+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:44.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:44.554+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:44.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:44 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:44.554+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:45.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:45.363+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:47.154682+0000 front 2024-05-27T07:41:47.154786+0000 (oldest deadline 2024-05-27T07:42:13.054954+0000) 2024-05-27T07:42:45.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:45.363+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.358183+0000 front 2024-05-27T07:41:55.358241+0000 (oldest deadline 2024-05-27T07:42:18.858500+0000) 2024-05-27T07:42:45.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:45.363+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.661551+0000 front 2024-05-27T07:42:04.661621+0000 (oldest deadline 2024-05-27T07:42:30.561810+0000) 2024-05-27T07:42:45.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7[53858]: 2024-05-27T07:42:45.363+0000 7f2be6400700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.964840+0000 front 2024-05-27T07:42:13.964912+0000 (oldest deadline 2024-05-27T07:42:39.865007+0000) 2024-05-27T07:42:45.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:45.133+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:45.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:45.133+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:45.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:45.133+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:45.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:45.133+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:45.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:45.133+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:45.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:45.385+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:45.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:45.385+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:45.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:45.385+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:45.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:45.385+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:45.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:45.227+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:45.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:45.227+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:45.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:45.227+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:45.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:45.227+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:45.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:45 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:45.227+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:45.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:45 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:45.563+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:45.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:45 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:45.563+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:45.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:45 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:45.563+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:45.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:45 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:45.563+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:45.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:45 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:45.563+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:46.598 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:46 smithi132 podman[78000]: 2024-05-27 07:42:46.38288288 +0000 UTC m=+5.848767411 container died c7835527819a26de98da5f355371a84d4b80deedd1031d560ff9e7538c85b19a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-27T07:42:46.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:46 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:46.170+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:46.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:46 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:46.170+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:46.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:46 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:46.170+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:46.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:46 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:46.170+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:46.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:46 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:46.171+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:46.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:46 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:46.393+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:46.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:46 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:46.393+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:46.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:46 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:46.393+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:46.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:46 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:46.393+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:46.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:46 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:46.225+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:46.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:46 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:46.225+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:46.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:46 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:46.225+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:46.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:46 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:46.225+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:46.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:46 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:46.225+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:46.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:46 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:46.522+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:46.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:46 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:46.522+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:46.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:46 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:46.522+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:46.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:46 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:46.522+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:46.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:46 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:46.522+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:47.099 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:46 smithi132 podman[78000]: 2024-05-27 07:42:46.920031709 +0000 UTC m=+6.385916236 container cleanup c7835527819a26de98da5f355371a84d4b80deedd1031d560ff9e7538c85b19a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:42:47.099 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:46 smithi132 bash[78000]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7 2024-05-27T07:42:47.546 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:47 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:47.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:47.546 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:47 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:47.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:47.546 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:47 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:47.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:47.546 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:47 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:47.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:47.546 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:47 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:47.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:47.546 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:47 smithi132 podman[78019]: 2024-05-27 07:42:47.197099696 +0000 UTC m=+0.801900981 container remove c7835527819a26de98da5f355371a84d4b80deedd1031d560ff9e7538c85b19a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-27T07:42:47.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:47.393+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:47.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:47.393+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:47.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:47.393+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:47.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:47.393+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:47.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:47.249+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:47.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:47.249+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:47.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:47.249+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:47.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:47.249+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:47.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:47 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:47.249+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:47.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:47 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:47.563+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:47.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:47 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:47.563+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:47.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:47 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:47.563+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:47.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:47 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:47.563+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:47.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:47 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:47.563+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:48.294 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:47 smithi132 podman[78108]: 2024-05-27 07:42:47.894553223 +0000 UTC m=+0.033512942 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:42:48.294 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:48 smithi132 podman[78108]: 2024-05-27 07:42:48.006281482 +0000 UTC m=+0.145241228 container create d03235897df15cd833a6be175b2147d03f4bf7d6b02cf7b1ec86472ff907537b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-27T07:42:48.294 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:48 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:48.093+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:48.294 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:48 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:48.093+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:48.294 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:48 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:48.094+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:48.294 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:48 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:48.094+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:48.294 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:48 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:48.094+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:48.295 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:48 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:48.094+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:48.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:48 smithi132 podman[78108]: 2024-05-27 07:42:48.293691749 +0000 UTC m=+0.432651564 container init d03235897df15cd833a6be175b2147d03f4bf7d6b02cf7b1ec86472ff907537b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:42:48.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:48 smithi132 podman[78108]: 2024-05-27 07:42:48.304380873 +0000 UTC m=+0.443340591 container start d03235897df15cd833a6be175b2147d03f4bf7d6b02cf7b1ec86472ff907537b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:42:48.599 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:48 smithi132 podman[78108]: 2024-05-27 07:42:48.392337107 +0000 UTC m=+0.531296844 container attach d03235897df15cd833a6be175b2147d03f4bf7d6b02cf7b1ec86472ff907537b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:42:48.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:48.287+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:48.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:48.287+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:48.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:48.287+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:48.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:48.287+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:48.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:48.287+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:48.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:48.385+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:48.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:48.385+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:48.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:48.385+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:48.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:48.385+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:48.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:48 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:48.385+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:48.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:48.600+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:48.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:48.600+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:48.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:48.600+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:48.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:48.600+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:48.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:48.600+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:48.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:48 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:48.600+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:49.098 INFO:journalctl@ceph.osd.7.smithi132.stdout:May 27 07:42:48 smithi132 podman[78108]: 2024-05-27 07:42:48.771130331 +0000 UTC m=+0.910090113 container died d03235897df15cd833a6be175b2147d03f4bf7d6b02cf7b1ec86472ff907537b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-27T07:42:49.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:49 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:49.142+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:49.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:49 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:49.142+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:49.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:49 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:49.142+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:49.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:49 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:49.142+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:49.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:49 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:49.142+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:49.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:49 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:49.143+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:49.561 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.7.service' 2024-05-27T07:42:49.667 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:42:49.667 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-27T07:42:49.667 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-27T07:42:49.667 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.8 2024-05-27T07:42:49.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:49.336+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:49.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:49.337+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:49.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:49.337+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:49.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:49.337+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:49.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:49.337+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:49.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:49.305+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:49.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:49.305+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:49.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:49.305+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:49.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:49.305+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:49.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:49.305+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:49.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:49 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:49.305+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.101973+0000 front 2024-05-27T07:42:23.102049+0000 (oldest deadline 2024-05-27T07:42:48.402188+0000) 2024-05-27T07:42:49.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:49 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:49.611+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:49.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:49 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:49.611+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:49.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:49 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:49.611+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:49.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:49 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:49.611+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:49.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:49 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:49.611+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:49.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:49 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:49.611+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:50.216 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:49 smithi179 systemd[1]: Stopping Ceph osd.8 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:42:50.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:50 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:50.174+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:50.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:50 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:50.174+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:50.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:50 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:50.174+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:50.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:50 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:50.174+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:50.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:50 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:50.174+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:50.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:50 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:50.174+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:50.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:50.259+0000 7f720d400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:42:50.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:50.259+0000 7f720d400700 -1 osd.8 165 *** Got signal Terminated *** 2024-05-27T07:42:50.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:50.259+0000 7f720d400700 -1 osd.8 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:42:50.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:50.265+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:50.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:50.265+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:50.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:50.265+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:50.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:50.265+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:50.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:50.265+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:50.718 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:50.265+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.101973+0000 front 2024-05-27T07:42:23.102049+0000 (oldest deadline 2024-05-27T07:42:48.402188+0000) 2024-05-27T07:42:50.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:50.346+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:50.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:50.346+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:50.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:50.346+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:50.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:50.346+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:50.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:50 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:50.346+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:50.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:50.627+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:50.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:50.627+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:50.962 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:50.627+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:50.962 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:50.627+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:50.962 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:50.627+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:50.962 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:50 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:50.627+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:51.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:51.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:51.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:51.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:51.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:51.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:51.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:51.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:51.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:51.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:51.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:51 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:51.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:51.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:51.332+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:51.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:51.332+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:51.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:51.332+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:51.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:51.332+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:51.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:51.332+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:51.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:51.242+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:51.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:51.242+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:51.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:51.242+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:51.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:51.242+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:51.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:51.242+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:51.717 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:51 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:51.242+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.101973+0000 front 2024-05-27T07:42:23.102049+0000 (oldest deadline 2024-05-27T07:42:48.402188+0000) 2024-05-27T07:42:51.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:51.654+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:51.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:51.654+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:51.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:51.654+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:51.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:51.654+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:51.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:51.654+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:51.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:51 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:51.654+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:52.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:52.212+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:52.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:52.212+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:52.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:52.212+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:52.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:52.212+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:52.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:52.212+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:52.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:52.212+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.101973+0000 front 2024-05-27T07:42:23.102049+0000 (oldest deadline 2024-05-27T07:42:48.402188+0000) 2024-05-27T07:42:52.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:52.308+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:52.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:52.308+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:52.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:52.308+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:52.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:52.308+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:52.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:52 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:52.308+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:52.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:52 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:52.147+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:52.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:52 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:52.147+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:52.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:52 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:52.148+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:52.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:52 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:52.148+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:52.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:52 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:52.148+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:52.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:52 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:52.148+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:52.945 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:52.605+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:52.945 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:52.605+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:52.945 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:52.605+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:52.945 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:52.605+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:52.945 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:52.605+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:52.945 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:52 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:52.605+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:53.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:53.168+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:53.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:53.168+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:53.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:53.168+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:53.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:53.168+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:53.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:53.168+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:53.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:53.168+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.101973+0000 front 2024-05-27T07:42:23.102049+0000 (oldest deadline 2024-05-27T07:42:48.402188+0000) 2024-05-27T07:42:53.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:53.289+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:53.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:53.289+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:53.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:53.289+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:53.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:53.289+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:53.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:53 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:53.289+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:53.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:53 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:53.156+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:53.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:53 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:53.156+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:53.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:53 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:53.156+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:53.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:53 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:53.156+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:53.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:53 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:53.156+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:53.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:53 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:53.156+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:53.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:53 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:53.560+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:53.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:53 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:53.560+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:53.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:53 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:53.560+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:53.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:53 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:53.560+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:53.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:53 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:53.560+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:53.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:53 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:53.560+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:54.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:54.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:54.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:54.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:54.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:54.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:54.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:54.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:54.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:54.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:54.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:54.149+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:54.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:54.149+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:54.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:54.149+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:54.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:54.149+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:54.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:54.149+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:54.468 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:54 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:54.149+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.101973+0000 front 2024-05-27T07:42:23.102049+0000 (oldest deadline 2024-05-27T07:42:48.402188+0000) 2024-05-27T07:42:54.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:54 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:54.204+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:54.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:54 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:54.204+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:54.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:54 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:54.204+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:54.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:54 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:54.204+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:54.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:54 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:54.204+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:54.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:54 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:54.204+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:54.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:54.571+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:54.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:54.571+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:54.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:54.571+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:54.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:54.571+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:54.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:54.571+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:54.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:54 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:54.571+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:55.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:55.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:55.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:55.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:55.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:55.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:55.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:55.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:55.467 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:55.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:55.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:55.133+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:34.976882+0000 front 2024-05-27T07:41:34.977078+0000 (oldest deadline 2024-05-27T07:42:00.277156+0000) 2024-05-27T07:42:55.467 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:55.133+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.287957+0000 front 2024-05-27T07:41:45.287762+0000 (oldest deadline 2024-05-27T07:42:12.289494+0000) 2024-05-27T07:42:55.468 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:55.133+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:57.492865+0000 front 2024-05-27T07:41:57.492966+0000 (oldest deadline 2024-05-27T07:42:21.592951+0000) 2024-05-27T07:42:55.468 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:55.133+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:01.593962+0000 front 2024-05-27T07:42:01.594049+0000 (oldest deadline 2024-05-27T07:42:33.394565+0000) 2024-05-27T07:42:55.468 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:55.133+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:13.395771+0000 front 2024-05-27T07:42:13.395948+0000 (oldest deadline 2024-05-27T07:42:38.596534+0000) 2024-05-27T07:42:55.468 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:55 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8[54462]: 2024-05-27T07:42:55.133+0000 7f7203c00700 -1 osd.8 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.101973+0000 front 2024-05-27T07:42:23.102049+0000 (oldest deadline 2024-05-27T07:42:48.402188+0000) 2024-05-27T07:42:55.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:55 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:55.171+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:55.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:55 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:55.171+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:55.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:55 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:55.171+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:55.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:55 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:55.171+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:55.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:55 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:55.171+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:55.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:55 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:55.171+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:55.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:55 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:55.595+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:55.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:55 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:55.595+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:55.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:55 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:55.595+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:55.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:55 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:55.595+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:55.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:55 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:55.595+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:55.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:55 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:55.595+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:56.283 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:55 smithi179 podman[80694]: 2024-05-27 07:42:55.988638301 +0000 UTC m=+5.863872052 container died eaeee19a212e3bfccb14c480aa77dfcb77e643c3be657cedc031554456becdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:42:56.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:56 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:56.147+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:56.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:56 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:56.147+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:56.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:56 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:56.147+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:56.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:56 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:56.147+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:56.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:56 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:56.147+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:56.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:56 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:56.147+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:56.629 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:56.283+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:56.629 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:56.283+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:56.629 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:56.283+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:56.629 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:56.283+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:56.629 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:56 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:56.283+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:56.629 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:56 smithi179 podman[80694]: 2024-05-27 07:42:56.629179387 +0000 UTC m=+6.504413135 container cleanup eaeee19a212e3bfccb14c480aa77dfcb77e643c3be657cedc031554456becdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-27T07:42:56.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:56.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:56.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:56.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:56.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:56.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:56.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:56.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:56.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:56.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:56.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:56 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:56.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:56.966 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:56 smithi179 bash[80694]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8 2024-05-27T07:42:56.967 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:56 smithi179 podman[80712]: 2024-05-27 07:42:56.838991948 +0000 UTC m=+0.832151446 container remove eaeee19a212e3bfccb14c480aa77dfcb77e643c3be657cedc031554456becdc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:42:57.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:57 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:57.194+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:57.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:57 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:57.194+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:57.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:57 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:57.194+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:57.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:57 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:57.194+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:57.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:57 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:57.194+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:57.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:57 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:57.194+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:57.618 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:57 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:57.246+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:57.618 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:57 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:57.246+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:57.618 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:57 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:57.246+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:57.618 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:57 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:57.246+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:57.618 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:57 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:57.246+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:57.952 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:57 smithi179 podman[80798]: 2024-05-27 07:42:57.523955779 +0000 UTC m=+0.032651538 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:42:57.952 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:57 smithi179 podman[80798]: 2024-05-27 07:42:57.681529789 +0000 UTC m=+0.190225584 container create 09364af518baefb6be73841b7d3b80b42c21b3e12d89fd8226faef76e5753da5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-27T07:42:57.952 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:57 smithi179 podman[80798]: 2024-05-27 07:42:57.952219953 +0000 UTC m=+0.460915758 container init 09364af518baefb6be73841b7d3b80b42c21b3e12d89fd8226faef76e5753da5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:42:57.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:57.657+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:57.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:57.657+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:57.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:57.657+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:57.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:57.657+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:57.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:57.657+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:57.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:57 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:57.657+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:58.217 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:57 smithi179 podman[80798]: 2024-05-27 07:42:57.964443953 +0000 UTC m=+0.473139762 container start 09364af518baefb6be73841b7d3b80b42c21b3e12d89fd8226faef76e5753da5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD) 2024-05-27T07:42:58.217 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:58 smithi179 podman[80798]: 2024-05-27 07:42:58.104258016 +0000 UTC m=+0.612953854 container attach 09364af518baefb6be73841b7d3b80b42c21b3e12d89fd8226faef76e5753da5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-27T07:42:58.591 INFO:journalctl@ceph.osd.8.smithi179.stdout:May 27 07:42:58 smithi179 podman[80798]: 2024-05-27 07:42:58.439161452 +0000 UTC m=+0.947857280 container died 09364af518baefb6be73841b7d3b80b42c21b3e12d89fd8226faef76e5753da5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-05-27T07:42:58.591 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:58 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:58.274+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:58.591 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:58 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:58.274+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:58.591 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:58 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:58.274+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:58.592 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:58 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:58.274+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:58.592 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:58 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:58.275+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:58.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:58 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:58.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:58.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:58 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:58.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:58.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:58 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:58.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:58.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:58 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:58.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:58.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:58 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:58.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:58.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:58 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:58.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:58.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:58 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:58.180+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:42:58.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:58 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:58.661+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:58.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:58 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:58.661+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:58.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:58 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:58.661+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:58.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:58 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:58.661+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:58.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:58 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:58.661+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:58.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:58 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:58.661+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:59.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.8.service' 2024-05-27T07:42:59.529 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:42:59.530 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-27T07:42:59.530 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-27T07:42:59.530 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.9 2024-05-27T07:42:59.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:59 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:59.210+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:42:59.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:59 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:59.210+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:42:59.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:59 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:59.210+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:42:59.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:59 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:59.210+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:42:59.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:59 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:59.210+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:42:59.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:59 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:59.210+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:42:59.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:42:59 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:42:59.210+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:42:59.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:59 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:59.292+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:42:59.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:59 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:59.292+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:42:59.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:59 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:59.292+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:42:59.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:59 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:59.292+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:42:59.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:42:59 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:42:59.292+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:42:59.864 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:59.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:42:59.864 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:59.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:42:59.864 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:59.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:42:59.864 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:59.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:42:59.864 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:59.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:42:59.864 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:59 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:42:59.616+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:42:59.864 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:42:59 smithi043 systemd[1]: Stopping Ceph osd.9 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:43:00.460 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:00.140+0000 7f2388000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:00.460 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:00.140+0000 7f2388000700 -1 osd.9 165 *** Got signal Terminated *** 2024-05-27T07:43:00.461 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:00.140+0000 7f2388000700 -1 osd.9 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:43:00.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:00 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:00.176+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:00.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:00 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:00.176+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:00.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:00 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:00.176+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:00.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:00 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:00.176+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:00.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:00 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:00.176+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:00.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:00 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:00.176+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:00.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:00 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:00.176+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:00.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:00.342+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:00.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:00.342+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:00.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:00.342+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:00.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:00.342+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:00.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:00.342+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:00.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:00 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:00.342+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:00.903 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:00.572+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:43:00.903 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:00.572+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:43:00.903 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:00.572+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:43:00.903 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:00.572+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:43:00.903 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:00.572+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:43:00.903 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:00 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:00.572+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:43:01.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:01 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:01.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:01.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:01 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:01.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:01.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:01 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:01.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:01.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:01 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:01.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:01.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:01 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:01.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:01.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:01 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:01.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:01.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:01 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:01.135+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:01.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:01.334+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:01.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:01.334+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:01.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:01.334+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:01.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:01.334+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:01.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:01.334+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:01.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:01 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:01.334+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:01.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:01.614+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:43:01.996 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:01.614+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:43:01.996 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:01.614+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:43:01.996 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:01.614+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:43:01.996 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:01.614+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:43:01.996 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:01 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:01.614+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:43:02.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:02 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:02.154+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:02.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:02 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:02.154+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:02.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:02 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:02.154+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:02.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:02 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:02.154+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:02.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:02 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:02.154+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:02.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:02 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:02.154+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:02.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:02 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:02.154+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:02.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:02 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:02.380+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:02.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:02 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:02.380+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:02.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:02 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:02.380+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:02.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:02 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:02.380+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:02.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:02 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:02.380+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:02.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:02 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:02.380+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:02.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:02.585+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:43:02.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:02.585+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:43:02.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:02.585+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:43:02.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:02.585+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:43:02.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:02.585+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:43:02.962 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:02 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:02.585+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:43:03.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:03 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:03.121+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:03.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:03 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:03.121+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:03.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:03 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:03.121+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:03.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:03 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:03.121+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:03.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:03 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:03.121+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:03.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:03 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:03.121+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:03.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:03 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:03.121+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:03.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:03 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:03.429+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:03.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:03 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:03.429+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:03.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:03 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:03.429+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:03.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:03 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:03.429+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:03.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:03 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:03.429+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:03.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:03 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:03.429+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:03.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:03.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:43:03.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:03.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:43:03.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:03.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:43:03.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:03.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:43:03.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:03.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:43:03.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:03 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:03.584+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:43:04.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:04 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:04.123+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:04.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:04 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:04.123+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:04.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:04 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:04.123+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:04.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:04 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:04.123+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:04.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:04 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:04.123+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:04.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:04 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:04.123+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:04.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:04 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:04.123+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:04.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:04 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:04.426+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:04.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:04 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:04.426+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:04.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:04 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:04.426+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:04.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:04 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:04.426+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:04.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:04 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:04.426+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:04.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:04 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:04.426+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:04.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:04.581+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.212369+0000 front 2024-05-27T07:41:33.212517+0000 (oldest deadline 2024-05-27T07:42:01.415162+0000) 2024-05-27T07:43:04.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:04.581+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:45.519227+0000 front 2024-05-27T07:41:45.519335+0000 (oldest deadline 2024-05-27T07:42:10.819487+0000) 2024-05-27T07:43:04.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:04.581+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.521548+0000 front 2024-05-27T07:41:55.521886+0000 (oldest deadline 2024-05-27T07:42:19.021661+0000) 2024-05-27T07:43:04.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:04.581+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.728569+0000 front 2024-05-27T07:42:04.728780+0000 (oldest deadline 2024-05-27T07:42:28.828888+0000) 2024-05-27T07:43:04.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:04.581+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:14.132114+0000 front 2024-05-27T07:42:14.132269+0000 (oldest deadline 2024-05-27T07:42:39.432399+0000) 2024-05-27T07:43:04.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:04 smithi043 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9[63789]: 2024-05-27T07:43:04.581+0000 7f237e800700 -1 osd.9 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.534926+0000 front 2024-05-27T07:42:23.535300+0000 (oldest deadline 2024-05-27T07:42:47.635155+0000) 2024-05-27T07:43:05.501 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:05 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:05.099+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:05.501 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:05 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:05.099+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:05.501 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:05 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:05.099+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:05.501 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:05 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:05.099+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:05.501 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:05 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:05.099+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:05.501 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:05 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:05.099+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:05.501 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:05 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:05.099+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:05.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:05.403+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:05.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:05.403+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:05.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:05.403+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:05.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:05.403+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:05.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:05.403+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:05.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:05 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:05.403+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:06.160 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:05 smithi043 podman[141583]: 2024-05-27 07:43:05.879276761 +0000 UTC m=+5.864413743 container died b32a02e585550b36806ebcefee0581e7f0163aeb8ec574d6350ee64c01ad764e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:43:06.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:06 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:06.069+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:06.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:06 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:06.069+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:06.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:06 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:06.069+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:06.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:06 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:06.069+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:06.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:06 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:06.069+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:06.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:06 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:06.069+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:06.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:06 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:06.069+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:06.460 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:06 smithi043 podman[141583]: 2024-05-27 07:43:06.357331929 +0000 UTC m=+6.342468908 container cleanup b32a02e585550b36806ebcefee0581e7f0163aeb8ec574d6350ee64c01ad764e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:43:06.460 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:06 smithi043 bash[141583]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9 2024-05-27T07:43:06.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:06.394+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:06.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:06.394+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:06.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:06.394+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:06.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:06.394+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:06.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:06.394+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:06.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:06 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:06.394+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:06.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:06 smithi043 podman[141614]: 2024-05-27 07:43:06.550283713 +0000 UTC m=+0.654100976 container remove b32a02e585550b36806ebcefee0581e7f0163aeb8ec574d6350ee64c01ad764e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240521) 2024-05-27T07:43:07.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:07 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:07.057+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:07.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:07 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:07.057+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:07.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:07 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:07.057+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:07.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:07 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:07.057+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:07.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:07 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:07.057+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:07.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:07 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:07.057+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:07.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:07 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:07.057+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:07.614 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:07 smithi043 podman[141702]: 2024-05-27 07:43:07.269469216 +0000 UTC m=+0.033236828 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:43:07.615 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:07 smithi043 podman[141702]: 2024-05-27 07:43:07.367664198 +0000 UTC m=+0.131431847 container create f51e9b6185eba0380d209091b69a9fe1f5e30161267b3db4982ac72fbf376d2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521) 2024-05-27T07:43:07.615 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:07 smithi043 podman[141702]: 2024-05-27 07:43:07.615641509 +0000 UTC m=+0.379409143 container init f51e9b6185eba0380d209091b69a9fe1f5e30161267b3db4982ac72fbf376d2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:43:07.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:07 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:07.347+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:07.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:07 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:07.347+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:07.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:07 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:07.347+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:07.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:07 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:07.347+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:07.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:07 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:07.347+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:07.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:07 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:07.347+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:07.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:07 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:07.347+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:07.960 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:07 smithi043 podman[141702]: 2024-05-27 07:43:07.626994758 +0000 UTC m=+0.390762417 container start f51e9b6185eba0380d209091b69a9fe1f5e30161267b3db4982ac72fbf376d2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240521, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-27T07:43:07.961 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:07 smithi043 podman[141702]: 2024-05-27 07:43:07.712074551 +0000 UTC m=+0.475842250 container attach f51e9b6185eba0380d209091b69a9fe1f5e30161267b3db4982ac72fbf376d2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:43:08.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:08 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:08.095+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:08.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:08 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:08.095+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:08.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:08 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:08.095+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:08.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:08 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:08.095+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:08.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:08 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:08.095+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:08.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:08 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:08.095+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:08.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:08 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:08.095+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:08.383 INFO:journalctl@ceph.osd.9.smithi043.stdout:May 27 07:43:08 smithi043 podman[141727]: 2024-05-27 07:43:08.174894666 +0000 UTC m=+0.042769060 container died f51e9b6185eba0380d209091b69a9fe1f5e30161267b3db4982ac72fbf376d2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-27T07:43:08.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:08.356+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:08.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:08.356+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:08.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:08.356+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:08.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:08.356+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:08.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:08.356+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:08.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:08.356+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:08.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:08 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:08.356+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:08.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.9.service' 2024-05-27T07:43:08.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:08.845 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-27T07:43:08.845 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-27T07:43:08.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.10 2024-05-27T07:43:09.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:08 smithi132 systemd[1]: Stopping Ceph osd.10 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:43:09.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:09.052+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:09.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:09.052+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:09.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:09.052+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:09.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:09.052+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:09.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:09.052+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:09.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:09.052+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:09.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:09.052+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:09.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:09.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:09.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:09.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:09.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:09.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:09.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:09.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:09.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:09.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:09.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:09.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:09.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:09 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:09.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:09.849 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:09.471+0000 7f3bf9e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:09.849 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:09.471+0000 7f3bf9e00700 -1 osd.10 165 *** Got signal Terminated *** 2024-05-27T07:43:09.849 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:09 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:09.471+0000 7f3bf9e00700 -1 osd.10 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:43:10.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:10 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:10.082+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:10.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:10 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:10.082+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:10.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:10 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:10.082+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:10.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:10 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:10.082+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:10.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:10 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:10.082+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:10.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:10 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:10.082+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:10.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:10 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:10.082+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:10.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:10.354+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:10.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:10.354+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:10.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:10.354+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:10.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:10.354+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:10.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:10.354+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:10.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:10.354+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:10.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:10 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:10.354+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:11.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:11.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:11.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:11.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:11.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:11.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:11.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:11.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:11.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:11.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:11.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:11.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:11.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:11 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:11.132+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:11.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:11.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:11.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:11.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:11.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:11.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:11.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:11.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:11.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:11.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:11.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:11.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:11.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:11 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:11.320+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:12.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:12 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:12.137+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:12.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:12 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:12.137+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:12.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:12 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:12.137+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:12.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:12 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:12.137+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:12.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:12 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:12.137+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:12.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:12 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:12.137+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:12.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:12 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:12.137+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:12.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:12.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:12.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:12.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:12.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:12.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:12.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:12.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:12.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:12.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:12.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:12.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:12.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:12 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:12.277+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:13.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:13.175+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:13.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:13.175+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:13.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:13.175+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:13.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:13.175+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:13.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:13.175+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:13.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:13.175+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:13.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:13 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:13.175+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:13.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:13.318+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:13.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:13.318+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:13.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:13.318+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:13.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:13.318+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:13.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:13.318+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:13.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:13.318+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:13.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:13 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:13.318+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:14.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:14.221+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:35.505039+0000 front 2024-05-27T07:41:35.505084+0000 (oldest deadline 2024-05-27T07:42:00.805174+0000) 2024-05-27T07:43:14.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:14.221+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:43.108180+0000 front 2024-05-27T07:41:43.108289+0000 (oldest deadline 2024-05-27T07:42:09.509627+0000) 2024-05-27T07:43:14.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:14.221+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:54.811819+0000 front 2024-05-27T07:41:54.811528+0000 (oldest deadline 2024-05-27T07:42:20.111283+0000) 2024-05-27T07:43:14.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:14.221+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-05-27T07:42:04.616571+0000 front 2024-05-27T07:42:04.616281+0000 (oldest deadline 2024-05-27T07:42:28.116461+0000) 2024-05-27T07:43:14.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:14.221+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.819322+0000 front 2024-05-27T07:42:12.819260+0000 (oldest deadline 2024-05-27T07:42:37.519435+0000) 2024-05-27T07:43:14.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:14.221+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:23.823747+0000 front 2024-05-27T07:42:23.824003+0000 (oldest deadline 2024-05-27T07:42:47.923756+0000) 2024-05-27T07:43:14.600 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:14 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10[60967]: 2024-05-27T07:43:14.221+0000 7f3bf5600700 -1 osd.10 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.727347+0000 front 2024-05-27T07:42:33.727047+0000 (oldest deadline 2024-05-27T07:42:57.827042+0000) 2024-05-27T07:43:14.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:14.297+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:14.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:14.297+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:14.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:14.297+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:14.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:14.297+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:14.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:14.297+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:14.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:14.297+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:14.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:14 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:14.297+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:15.494 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:15 smithi132 podman[78224]: 2024-05-27 07:43:15.185716292 +0000 UTC m=+5.867506185 container died 2cc3dcf0860be8099f0e011cdf984b43f231dd356c2a24e5d26a3e793b69b076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True) 2024-05-27T07:43:15.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:15.319+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:15.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:15.319+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:15.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:15.319+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:15.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:15.319+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:15.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:15.319+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:15.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:15.319+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:15.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:15 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:15.319+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:15.849 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:15 smithi132 podman[78224]: 2024-05-27 07:43:15.70652334 +0000 UTC m=+6.388313252 container cleanup 2cc3dcf0860be8099f0e011cdf984b43f231dd356c2a24e5d26a3e793b69b076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:43:15.849 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:15 smithi132 bash[78224]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10 2024-05-27T07:43:16.349 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:16 smithi132 podman[78242]: 2024-05-27 07:43:16.016402937 +0000 UTC m=+0.815799770 container remove 2cc3dcf0860be8099f0e011cdf984b43f231dd356c2a24e5d26a3e793b69b076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-27T07:43:16.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:16.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:16.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:16.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:16.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:16.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:16.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:16.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:16.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:16.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:16.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:16.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:16.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:16 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:16.340+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:17.157 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:16 smithi132 podman[78329]: 2024-05-27 07:43:16.766753969 +0000 UTC m=+0.033734837 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:43:17.158 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:16 smithi132 podman[78329]: 2024-05-27 07:43:16.884295849 +0000 UTC m=+0.151276774 container create f162a541a534ca8bad4d496530b429e60fa25f5f6ba4d6784525f2cbf08dcde0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:43:17.158 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:17 smithi132 podman[78329]: 2024-05-27 07:43:17.158575275 +0000 UTC m=+0.425556177 container init f162a541a534ca8bad4d496530b429e60fa25f5f6ba4d6784525f2cbf08dcde0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-27T07:43:17.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:17 smithi132 podman[78329]: 2024-05-27 07:43:17.169876387 +0000 UTC m=+0.436857258 container start f162a541a534ca8bad4d496530b429e60fa25f5f6ba4d6784525f2cbf08dcde0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-05-27T07:43:17.599 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:17 smithi132 podman[78329]: 2024-05-27 07:43:17.265065366 +0000 UTC m=+0.532046281 container attach f162a541a534ca8bad4d496530b429e60fa25f5f6ba4d6784525f2cbf08dcde0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, io.buildah.version=1.33.5) 2024-05-27T07:43:17.720 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:17.310+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:17.720 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:17.310+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:17.720 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:17.310+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:17.720 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:17.310+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:17.720 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:17.310+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:17.720 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:17.310+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:17.720 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:17.310+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:17.720 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:17 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:17.310+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6820 osd.8 since back 2024-05-27T07:42:52.867807+0000 front 2024-05-27T07:42:52.867913+0000 (oldest deadline 2024-05-27T07:43:16.967734+0000) 2024-05-27T07:43:17.979 INFO:journalctl@ceph.osd.10.smithi132.stdout:May 27 07:43:17 smithi132 podman[78329]: 2024-05-27 07:43:17.624631485 +0000 UTC m=+0.891612423 container died f162a541a534ca8bad4d496530b429e60fa25f5f6ba4d6784525f2cbf08dcde0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True) 2024-05-27T07:43:18.400 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.10.service' 2024-05-27T07:43:18.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:18.501 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-27T07:43:18.502 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-27T07:43:18.502 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.11 2024-05-27T07:43:18.577 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:18.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:18.577 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:18.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:18.577 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:18.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:18.578 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:18.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:18.578 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:18.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:18.578 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:18.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:18.578 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:18.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:18.578 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:18 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:18.303+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6820 osd.8 since back 2024-05-27T07:42:52.867807+0000 front 2024-05-27T07:42:52.867913+0000 (oldest deadline 2024-05-27T07:43:16.967734+0000) 2024-05-27T07:43:18.966 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:18 smithi179 systemd[1]: Stopping Ceph osd.11 for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:43:19.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.103+0000 7f86b3600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:19.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.103+0000 7f86b3600700 -1 osd.11 165 *** Got signal Terminated *** 2024-05-27T07:43:19.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.103+0000 7f86b3600700 -1 osd.11 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:43:19.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.257+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:19.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.257+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:19.468 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.257+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:19.469 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.257+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:19.469 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.257+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:19.469 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.257+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:19.469 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.257+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:19.469 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:19 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:19.257+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6820 osd.8 since back 2024-05-27T07:42:52.867807+0000 front 2024-05-27T07:42:52.867913+0000 (oldest deadline 2024-05-27T07:43:16.967734+0000) 2024-05-27T07:43:20.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:20.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:20.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:20.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:20.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:20.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:20.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:20.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:20.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:20.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:20.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:20.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:20.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:20.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:20.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:20 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:20.302+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6820 osd.8 since back 2024-05-27T07:42:52.867807+0000 front 2024-05-27T07:42:52.867913+0000 (oldest deadline 2024-05-27T07:43:16.967734+0000) 2024-05-27T07:43:21.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:21.343+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:21.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:21.343+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:21.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:21.343+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:21.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:21.343+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:21.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:21.343+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:21.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:21.343+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:21.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:21.343+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:21.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:21 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:21.343+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6820 osd.8 since back 2024-05-27T07:42:52.867807+0000 front 2024-05-27T07:42:52.867913+0000 (oldest deadline 2024-05-27T07:43:16.967734+0000) 2024-05-27T07:43:22.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:22.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:22.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:22.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:22.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:22.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:22.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:22.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:22.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:22.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:22.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:22.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:22.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:22.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:22.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:22 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:22.305+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6820 osd.8 since back 2024-05-27T07:42:52.867807+0000 front 2024-05-27T07:42:52.867913+0000 (oldest deadline 2024-05-27T07:43:16.967734+0000) 2024-05-27T07:43:23.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:23.309+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-27T07:41:33.240178+0000 front 2024-05-27T07:41:33.240301+0000 (oldest deadline 2024-05-27T07:41:57.940445+0000) 2024-05-27T07:43:23.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:23.309+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-05-27T07:41:44.944194+0000 front 2024-05-27T07:41:44.944272+0000 (oldest deadline 2024-05-27T07:42:10.244471+0000) 2024-05-27T07:43:23.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:23.309+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-05-27T07:41:55.949584+0000 front 2024-05-27T07:41:55.949622+0000 (oldest deadline 2024-05-27T07:42:18.849796+0000) 2024-05-27T07:43:23.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:23.309+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-05-27T07:42:12.156143+0000 front 2024-05-27T07:42:12.156193+0000 (oldest deadline 2024-05-27T07:42:40.956967+0000) 2024-05-27T07:43:23.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:23.309+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-05-27T07:42:22.658564+0000 front 2024-05-27T07:42:22.659019+0000 (oldest deadline 2024-05-27T07:42:47.959012+0000) 2024-05-27T07:43:23.717 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:23.309+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-05-27T07:42:33.662496+0000 front 2024-05-27T07:42:33.662457+0000 (oldest deadline 2024-05-27T07:42:59.561909+0000) 2024-05-27T07:43:23.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:23.309+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-05-27T07:42:44.864014+0000 front 2024-05-27T07:42:44.864071+0000 (oldest deadline 2024-05-27T07:43:07.163747+0000) 2024-05-27T07:43:23.718 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:23 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11[61276]: 2024-05-27T07:43:23.309+0000 7f86a9e00700 -1 osd.11 165 heartbeat_check: no reply from 172.21.15.179:6820 osd.8 since back 2024-05-27T07:42:52.867807+0000 front 2024-05-27T07:42:52.867913+0000 (oldest deadline 2024-05-27T07:43:16.967734+0000) 2024-05-27T07:43:24.966 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:24 smithi179 podman[80916]: 2024-05-27 07:43:24.698498057 +0000 UTC m=+5.761116798 container died 6ffa1e6f5c5fa4c029beef56f45b3b434ab8567f4603c8c41afe46d53fe93ca2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:43:25.440 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:25 smithi179 podman[80916]: 2024-05-27 07:43:25.123168648 +0000 UTC m=+6.185787386 container cleanup 6ffa1e6f5c5fa4c029beef56f45b3b434ab8567f4603c8c41afe46d53fe93ca2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-27T07:43:25.440 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:25 smithi179 bash[80916]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11 2024-05-27T07:43:25.716 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:25 smithi179 podman[80937]: 2024-05-27 07:43:25.440952185 +0000 UTC m=+0.721922114 container remove 6ffa1e6f5c5fa4c029beef56f45b3b434ab8567f4603c8c41afe46d53fe93ca2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-27T07:43:26.536 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:26 smithi179 podman[81025]: 2024-05-27 07:43:26.131198048 +0000 UTC m=+0.037379027 image pull e36ea4fc8a5807e8d2701370e350a0de3856538797f7fba64cfce7e344bd6799 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6 2024-05-27T07:43:26.536 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:26 smithi179 podman[81025]: 2024-05-27 07:43:26.233369393 +0000 UTC m=+0.139550400 container create 43cd819097426304dcd9ebb72ccf58f5bf2014ae0d02ca406e99e988224fa9e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD) 2024-05-27T07:43:26.536 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:26 smithi179 podman[81025]: 2024-05-27 07:43:26.536980646 +0000 UTC m=+0.443161657 container init 43cd819097426304dcd9ebb72ccf58f5bf2014ae0d02ca406e99e988224fa9e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux ) 2024-05-27T07:43:26.967 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:26 smithi179 podman[81025]: 2024-05-27 07:43:26.548716453 +0000 UTC m=+0.454897457 container start 43cd819097426304dcd9ebb72ccf58f5bf2014ae0d02ca406e99e988224fa9e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-27T07:43:26.967 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:26 smithi179 podman[81025]: 2024-05-27 07:43:26.627562527 +0000 UTC m=+0.533743581 container attach 43cd819097426304dcd9ebb72ccf58f5bf2014ae0d02ca406e99e988224fa9e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True) 2024-05-27T07:43:27.383 INFO:journalctl@ceph.osd.11.smithi179.stdout:May 27 07:43:27 smithi179 podman[81025]: 2024-05-27 07:43:27.004490429 +0000 UTC m=+0.910671400 container died 43cd819097426304dcd9ebb72ccf58f5bf2014ae0d02ca406e99e988224fa9e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-27T07:43:27.787 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@osd.11.service' 2024-05-27T07:43:27.899 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:27.899 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-27T07:43:27.899 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-27T07:43:27.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a 2024-05-27T07:43:28.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.a.service' 2024-05-27T07:43:28.122 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:28.122 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-27T07:43:28.122 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-27T07:43:28.122 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d 2024-05-27T07:43:28.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.d.service' 2024-05-27T07:43:28.379 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:28.379 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-27T07:43:28.379 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-27T07:43:28.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g 2024-05-27T07:43:28.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.g.service' 2024-05-27T07:43:28.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:28.630 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-27T07:43:28.630 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-27T07:43:28.630 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j 2024-05-27T07:43:28.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.j.service' 2024-05-27T07:43:28.873 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:28.873 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-27T07:43:28.873 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-27T07:43:28.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.b 2024-05-27T07:43:29.349 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:43:28 smithi132 systemd[1]: Stopping Ceph mds.b for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:43:29.849 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:43:29 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b[66148]: 2024-05-27T07:43:29.444+0000 7f0946000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:29.849 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:43:29 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b[66148]: 2024-05-27T07:43:29.444+0000 7f0946000700 -1 mds.b *** got signal Terminated *** 2024-05-27T07:43:37.725 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:43:37 smithi132 podman[78434]: 2024-05-27 07:43:37.422109569 +0000 UTC m=+8.133082227 container died 22b3908559b9711ad263d0860ccaa9332a5dcaf78db8d4e109de55e2ead12ab7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-27T07:43:38.349 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:43:38 smithi132 podman[78434]: 2024-05-27 07:43:38.020161006 +0000 UTC m=+8.731133660 container cleanup 22b3908559b9711ad263d0860ccaa9332a5dcaf78db8d4e109de55e2ead12ab7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD) 2024-05-27T07:43:38.349 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:43:38 smithi132 bash[78434]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b 2024-05-27T07:43:38.634 INFO:journalctl@ceph.mds.b.smithi132.stdout:May 27 07:43:38 smithi132 podman[78446]: 2024-05-27 07:43:38.369985784 +0000 UTC m=+0.931984913 container remove 22b3908559b9711ad263d0860ccaa9332a5dcaf78db8d4e109de55e2ead12ab7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-b, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0) 2024-05-27T07:43:38.666 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.b.service' 2024-05-27T07:43:38.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:38.769 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-27T07:43:38.769 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-27T07:43:38.769 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.e 2024-05-27T07:43:39.099 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:43:38 smithi132 systemd[1]: Stopping Ceph mds.e for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:43:39.599 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:43:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-e[65525]: 2024-05-27T07:43:39.338+0000 7f6b20c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:39.599 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:43:39 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-e[65525]: 2024-05-27T07:43:39.338+0000 7f6b20c00700 -1 mds.e *** got signal Terminated *** 2024-05-27T07:43:44.099 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:43:43 smithi132 podman[78554]: 2024-05-27 07:43:43.810550209 +0000 UTC m=+4.602264347 container died 983414c01141f7f9ff13d595c1054c4bfa6ac2de60cb2fb0f9f9a86c0afad972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True) 2024-05-27T07:43:44.532 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:43:44 smithi132 podman[78554]: 2024-05-27 07:43:44.256251923 +0000 UTC m=+5.047966057 container cleanup 983414c01141f7f9ff13d595c1054c4bfa6ac2de60cb2fb0f9f9a86c0afad972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-27T07:43:44.532 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:43:44 smithi132 bash[78554]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-e 2024-05-27T07:43:44.848 INFO:journalctl@ceph.mds.e.smithi132.stdout:May 27 07:43:44 smithi132 podman[78580]: 2024-05-27 07:43:44.53253295 +0000 UTC m=+0.698818637 container remove 983414c01141f7f9ff13d595c1054c4bfa6ac2de60cb2fb0f9f9a86c0afad972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-e, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:43:45.058 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.e.service' 2024-05-27T07:43:45.164 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:45.164 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-27T07:43:45.165 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-27T07:43:45.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.h 2024-05-27T07:43:45.598 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:43:45 smithi132 systemd[1]: Stopping Ceph mds.h for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:43:46.099 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:43:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-h[65826]: 2024-05-27T07:43:45.731+0000 7efcd4e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:46.099 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:43:45 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-h[65826]: 2024-05-27T07:43:45.731+0000 7efcd4e00700 -1 mds.h *** got signal Terminated *** 2024-05-27T07:43:51.196 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:43:50 smithi132 podman[78687]: 2024-05-27 07:43:50.927349815 +0000 UTC m=+5.326575924 container died b73a5f07d84bc1c662be33babdcbff5f74212cf9f6362f9f4a7fa31374662ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-h, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-27T07:43:51.510 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:43:51 smithi132 podman[78687]: 2024-05-27 07:43:51.318035016 +0000 UTC m=+5.717261125 container cleanup b73a5f07d84bc1c662be33babdcbff5f74212cf9f6362f9f4a7fa31374662ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-h, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-27T07:43:51.510 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:43:51 smithi132 bash[78687]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-h 2024-05-27T07:43:51.849 INFO:journalctl@ceph.mds.h.smithi132.stdout:May 27 07:43:51 smithi132 podman[78699]: 2024-05-27 07:43:51.510934218 +0000 UTC m=+0.566804134 container remove b73a5f07d84bc1c662be33babdcbff5f74212cf9f6362f9f4a7fa31374662ff7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:43:51.969 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.h.service' 2024-05-27T07:43:52.071 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:52.071 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-27T07:43:52.071 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-27T07:43:52.072 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.k 2024-05-27T07:43:52.599 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:43:52 smithi132 systemd[1]: Stopping Ceph mds.k for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:43:53.099 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:43:52 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-k[65226]: 2024-05-27T07:43:52.703+0000 7f8fd9400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:53.099 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:43:52 smithi132 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-k[65226]: 2024-05-27T07:43:52.703+0000 7f8fd9400700 -1 mds.k *** got signal Terminated *** 2024-05-27T07:43:57.209 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:43:56 smithi132 podman[78805]: 2024-05-27 07:43:56.915559831 +0000 UTC m=+4.361126756 container died 5e67a7a8e86d8a49b6d55b8c401658f3f2fab82f8af9f06cf2a9277e9da3055d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-k, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-27T07:43:57.530 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:43:57 smithi132 podman[78805]: 2024-05-27 07:43:57.330027132 +0000 UTC m=+4.775594046 container cleanup 5e67a7a8e86d8a49b6d55b8c401658f3f2fab82f8af9f06cf2a9277e9da3055d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-k, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90) 2024-05-27T07:43:57.530 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:43:57 smithi132 bash[78805]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-k 2024-05-27T07:43:57.849 INFO:journalctl@ceph.mds.k.smithi132.stdout:May 27 07:43:57 smithi132 podman[78819]: 2024-05-27 07:43:57.530878161 +0000 UTC m=+0.589400224 container remove 5e67a7a8e86d8a49b6d55b8c401658f3f2fab82f8af9f06cf2a9277e9da3055d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:43:58.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.k.service' 2024-05-27T07:43:58.124 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:58.124 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-27T07:43:58.124 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-27T07:43:58.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c 2024-05-27T07:43:58.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.c.service' 2024-05-27T07:43:58.320 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:58.320 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-27T07:43:58.321 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-27T07:43:58.321 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f 2024-05-27T07:43:58.464 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.f.service' 2024-05-27T07:43:58.564 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:43:58.565 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-27T07:43:58.565 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-27T07:43:58.565 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.i 2024-05-27T07:43:58.716 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:43:58 smithi179 systemd[1]: Stopping Ceph mds.i for 1bb64d42-1bf6-11ef-bc9b-c7b262605968... 2024-05-27T07:43:59.467 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:43:59 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-i[66439]: 2024-05-27T07:43:59.162+0000 7fc97ea00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-27T07:43:59.467 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:43:59 smithi179 ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-i[66439]: 2024-05-27T07:43:59.162+0000 7fc97ea00700 -1 mds.i *** got signal Terminated *** 2024-05-27T07:44:01.800 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:44:01 smithi179 podman[81213]: 2024-05-27 07:44:01.49114888 +0000 UTC m=+2.453666660 container died 0c675283f0dcfa73cd017db1ece474a9c5e42cdfe35d1bb574feadc8a0d2635a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.build-date=20240521) 2024-05-27T07:44:02.197 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:44:01 smithi179 podman[81213]: 2024-05-27 07:44:01.913588101 +0000 UTC m=+2.876105891 container cleanup 0c675283f0dcfa73cd017db1ece474a9c5e42cdfe35d1bb574feadc8a0d2635a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-27T07:44:02.197 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:44:01 smithi179 bash[81213]: ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-i 2024-05-27T07:44:02.467 INFO:journalctl@ceph.mds.i.smithi179.stdout:May 27 07:44:02 smithi179 podman[81227]: 2024-05-27 07:44:02.197789788 +0000 UTC m=+0.690403529 container remove 0c675283f0dcfa73cd017db1ece474a9c5e42cdfe35d1bb574feadc8a0d2635a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e6a098888d5cb5eb3200725617291c1104881c62c1d90236d9eab3f8efd213b6, name=ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-vshankar-testing-20240521.063558-debug-38f9d90, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-27T07:44:02.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.i.service' 2024-05-27T07:44:02.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:44:02.852 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-27T07:44:02.852 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-27T07:44:02.852 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l 2024-05-27T07:44:02.948 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1bb64d42-1bf6-11ef-bc9b-c7b262605968@mds.l.service' 2024-05-27T07:44:03.048 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-27T07:44:03.049 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-27T07:44:03.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T07:44:03.384 INFO:teuthology.orchestra.run.smithi043.stdout:Deleting cluster with fsid: 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T07:44:08.880 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T07:44:09.203 INFO:teuthology.orchestra.run.smithi132.stdout:Deleting cluster with fsid: 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T07:44:14.190 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-27T07:44:14.511 INFO:teuthology.orchestra.run.smithi179.stdout:Deleting cluster with fsid: 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T07:44:19.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:44:19.624 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:44:19.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-27T07:44:19.807 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-27T07:44:19.809 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630/remote/smithi043/crash 2024-05-27T07:44:19.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T07:44:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:tar: /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T07:44:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:tar: Error is not recoverable: exiting now 2024-05-27T07:44:19.916 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630/remote/smithi132/crash 2024-05-27T07:44:19.917 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T07:44:20.021 INFO:teuthology.orchestra.run.smithi132.stderr:tar: /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T07:44:20.021 INFO:teuthology.orchestra.run.smithi132.stderr:tar: Error is not recoverable: exiting now 2024-05-27T07:44:20.025 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630/remote/smithi179/crash 2024-05-27T07:44:20.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/crash -- . 2024-05-27T07:44:20.132 INFO:teuthology.orchestra.run.smithi179.stderr:tar: /var/lib/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-27T07:44:20.133 INFO:teuthology.orchestra.run.smithi179.stderr:tar: Error is not recoverable: exiting now 2024-05-27T07:44:20.136 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-27T07:44:20.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-27T07:44:20.299 INFO:teuthology.orchestra.run.smithi043.stdout:2024-05-27T07:24:40.419561+0000 mon.a (mon.0) 1571 : cluster [WRN] Health check failed: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-27T07:44:20.301 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-27T07:44:20.301 INFO:tasks.cephadm:Compressing logs... 2024-05-27T07:44:20.301 DEBUG:teuthology.orchestra.run.smithi043:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:44:20.306 DEBUG:teuthology.orchestra.run.smithi132:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:44:20.314 DEBUG:teuthology.orchestra.run.smithi179:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:44:20.382 INFO:teuthology.orchestra.run.smithi043.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-27T07:44:20.382 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-05-27T07:44:20.382 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T07:44:20.384 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-05-27T07:44:20.385 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T07:44:20.386 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T07:44:20.387 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.g.log 2024-05-27T07:44:20.389 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.j.log 2024-05-27T07:44:20.390 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-05-27T07:44:20.391 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-05-27T07:44:20.392 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mgr.x.log 2024-05-27T07:44:20.392 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mon.a.log 2024-05-27T07:44:20.393 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mgr.x.log: 87.6% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T07:44:20.394 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.6.log 2024-05-27T07:44:20.394 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T07:44:20.395 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.a.log 2024-05-27T07:44:20.396 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T07:44:20.397 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.d.log 2024-05-27T07:44:20.398 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.9.log 2024-05-27T07:44:20.399 INFO:teuthology.orchestra.run.smithi179.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-27T07:44:20.399 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T07:44:20.400 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T07:44:20.401 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-05-27T07:44:20.402 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.f.log 2024-05-27T07:44:20.402 INFO:teuthology.orchestra.run.smithi179.stderr: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T07:44:20.404 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T07:44:20.405 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-05-27T07:44:20.407 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.c.log 2024-05-27T07:44:20.408 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.5.log: 87.6% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T07:44:20.409 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mon.c.log 2024-05-27T07:44:20.410 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mgr.z.log 2024-05-27T07:44:20.411 INFO:teuthology.orchestra.run.smithi179.stderr: 90.5% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T07:44:20.411 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.i.log 2024-05-27T07:44:20.413 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T07:44:20.416 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.i.log: /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.11.log 2024-05-27T07:44:20.419 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.11.log: 83.9%gzip -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T07:44:20.420 INFO:teuthology.orchestra.run.smithi179.stderr: -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.8.log 2024-05-27T07:44:20.421 INFO:teuthology.orchestra.run.smithi132.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-27T07:44:20.421 INFO:teuthology.orchestra.run.smithi132.stderr:: No such file or directory 2024-05-27T07:44:20.421 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T07:44:20.422 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.log 2024-05-27T07:44:20.423 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-27T07:44:20.424 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.7.log 2024-05-27T07:44:20.425 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.b.log 2024-05-27T07:44:20.426 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-05-27T07:44:20.427 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-05-27T07:44:20.428 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mgr.y.log 2024-05-27T07:44:20.429 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.8.log: /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-volume.log: 92.8% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mgr.z.log.gz 2024-05-27T07:44:20.429 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.h.log 2024-05-27T07:44:20.430 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.10.log 2024-05-27T07:44:20.431 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.l.log 2024-05-27T07:44:20.431 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-27T07:44:20.432 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.e.log 2024-05-27T07:44:20.433 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-27T07:44:20.434 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mon.b.log 2024-05-27T07:44:20.435 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.k.log 2024-05-27T07:44:20.442 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mon.b.log: /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.k.log: 92.8% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mgr.y.log.gz 2024-05-27T07:44:20.480 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.l.log: 90.9% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T07:44:20.522 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.d.log: /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.9.log: 84.2% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T07:44:20.532 INFO:teuthology.orchestra.run.smithi132.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T07:44:20.535 INFO:teuthology.orchestra.run.smithi043.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-05-27T07:44:20.587 INFO:teuthology.orchestra.run.smithi043.stderr: 90.3% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T07:44:20.683 INFO:teuthology.orchestra.run.smithi179.stderr: 94.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.c.log.gz 2024-05-27T07:44:20.756 INFO:teuthology.orchestra.run.smithi132.stderr: 90.4% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-27T07:44:20.761 INFO:teuthology.orchestra.run.smithi132.stderr: 87.6% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-27T07:44:20.921 INFO:teuthology.orchestra.run.smithi132.stderr: 83.9% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-27T07:44:21.111 INFO:teuthology.orchestra.run.smithi179.stderr: 94.5% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.f.log.gz 2024-05-27T07:44:21.282 INFO:teuthology.orchestra.run.smithi043.stderr: 94.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-27T07:44:21.478 INFO:teuthology.orchestra.run.smithi132.stderr: 90.9% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T07:44:21.666 INFO:teuthology.orchestra.run.smithi043.stderr: 90.9% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-27T07:44:21.763 INFO:teuthology.orchestra.run.smithi179.stderr: 92.0% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mon.c.log.gz 2024-05-27T07:44:21.836 INFO:teuthology.orchestra.run.smithi043.stderr: 92.4% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.g.log.gz 2024-05-27T07:44:22.236 INFO:teuthology.orchestra.run.smithi179.stderr: 95.3% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.l.log.gz 2024-05-27T07:44:22.402 INFO:teuthology.orchestra.run.smithi043.stderr: 93.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.j.log.gz 2024-05-27T07:44:23.419 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mgr.x.log.gz 2024-05-27T07:44:27.436 INFO:teuthology.orchestra.run.smithi043.stderr: 94.0% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.d.log.gz 2024-05-27T07:44:28.863 INFO:teuthology.orchestra.run.smithi132.stderr: 91.7% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mon.b.log.gz 2024-05-27T07:44:29.801 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mon.a.log.gz 2024-05-27T07:44:31.267 INFO:teuthology.orchestra.run.smithi043.stderr: 94.5% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.a.log.gz 2024-05-27T07:44:47.663 INFO:teuthology.orchestra.run.smithi179.stderr: 91.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-05-27T07:44:47.722 INFO:teuthology.orchestra.run.smithi179.stderr: 91.2% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.11.log.gz 2024-05-27T07:44:52.774 INFO:teuthology.orchestra.run.smithi179.stderr: 90.8% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-05-27T07:44:53.731 INFO:teuthology.orchestra.run.smithi179.stderr: 91.0% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.8.log.gz 2024-05-27T07:45:39.855 INFO:teuthology.orchestra.run.smithi179.stderr: 94.2% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.i.log.gz 2024-05-27T07:45:39.860 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-05-27T07:45:39.861 INFO:teuthology.orchestra.run.smithi179.stderr:real 1m19.489s 2024-05-27T07:45:39.861 INFO:teuthology.orchestra.run.smithi179.stderr:user 3m9.195s 2024-05-27T07:45:39.861 INFO:teuthology.orchestra.run.smithi179.stderr:sys 0m14.296s 2024-05-27T07:46:00.843 INFO:teuthology.orchestra.run.smithi043.stderr: 91.3% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-05-27T07:46:09.121 INFO:teuthology.orchestra.run.smithi043.stderr: 91.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-05-27T07:46:15.417 INFO:teuthology.orchestra.run.smithi043.stderr: 91.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.6.log.gz 2024-05-27T07:46:19.447 INFO:teuthology.orchestra.run.smithi043.stderr: 91.0% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.9.log.gz 2024-05-27T07:47:59.021 INFO:teuthology.orchestra.run.smithi132.stderr: 91.4% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-05-27T07:48:09.704 INFO:teuthology.orchestra.run.smithi132.stderr: 91.0% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.10.log.gz 2024-05-27T07:49:52.879 INFO:teuthology.orchestra.run.smithi132.stderr: 91.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-05-27T07:49:55.111 INFO:teuthology.orchestra.run.smithi132.stderr: 91.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-osd.7.log.gz 2024-05-27T07:50:27.406 INFO:teuthology.orchestra.run.smithi043.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-05-27T07:50:27.454 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-27T07:50:27.454 INFO:teuthology.orchestra.run.smithi043.stderr:real 6m7.101s 2024-05-27T07:50:27.454 INFO:teuthology.orchestra.run.smithi043.stderr:user 7m1.532s 2024-05-27T07:50:27.455 INFO:teuthology.orchestra.run.smithi043.stderr:sys 0m43.260s 2024-05-27T07:59:12.676 INFO:teuthology.orchestra.run.smithi132.stderr: 94.3% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.e.log.gz 2024-05-27T07:59:51.499 INFO:teuthology.orchestra.run.smithi132.stderr: 94.2% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.k.log.gz 2024-05-27T08:01:43.103 INFO:teuthology.orchestra.run.smithi132.stderr: 94.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.h.log.gz 2024-05-27T08:02:36.009 INFO:teuthology.orchestra.run.smithi132.stderr: 95.1% -- replaced with /var/log/ceph/1bb64d42-1bf6-11ef-bc9b-c7b262605968/ceph-mds.b.log.gz 2024-05-27T08:02:36.013 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-05-27T08:02:36.013 INFO:teuthology.orchestra.run.smithi132.stderr:real 18m15.650s 2024-05-27T08:02:36.013 INFO:teuthology.orchestra.run.smithi132.stderr:user 18m2.872s 2024-05-27T08:02:36.013 INFO:teuthology.orchestra.run.smithi132.stderr:sys 2m30.205s 2024-05-27T08:02:36.015 INFO:tasks.cephadm:Archiving logs... 2024-05-27T08:02:36.015 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630/remote/smithi043/log 2024-05-27T08:02:36.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T08:03:30.910 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630/remote/smithi132/log 2024-05-27T08:03:30.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T08:04:34.978 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630/remote/smithi179/log 2024-05-27T08:04:34.979 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T08:05:01.843 INFO:tasks.cephadm:Removing cluster... 2024-05-27T08:05:01.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 --force 2024-05-27T08:05:02.276 INFO:teuthology.orchestra.run.smithi043.stdout:Deleting cluster with fsid: 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T08:05:03.344 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 --force 2024-05-27T08:05:04.271 INFO:teuthology.orchestra.run.smithi132.stdout:Deleting cluster with fsid: 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T08:05:06.031 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 --force 2024-05-27T08:05:07.909 INFO:teuthology.orchestra.run.smithi179.stdout:Deleting cluster with fsid: 1bb64d42-1bf6-11ef-bc9b-c7b262605968 2024-05-27T08:05:09.148 INFO:tasks.cephadm:Removing cephadm ... 2024-05-27T08:05:09.149 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T08:05:09.212 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T08:05:09.272 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-27T08:05:09.333 INFO:tasks.cephadm:Teardown complete 2024-05-27T08:05:09.333 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-27T08:05:09.344 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/install/__init__.py", line 632, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1893, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi043 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump' 2024-05-27T08:05:09.345 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-27T08:05:09.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T08:05:09.353 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T08:05:09.361 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T08:05:09.519 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-27T08:05:09.530 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-27T08:05:09.530 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T08:05:09.538 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T08:05:09.545 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T08:05:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-05-27T08:05:09.592 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-05-27T08:05:09.599 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-05-27T08:05:09.610 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T08:05:09.610 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-05-27T08:05:09.610 INFO:teuthology.orchestra.run.smithi179.stdout:^* hv01.front.sepia.ceph.com 3 6 377 39 +800us[ +782us] +/- 59ms 2024-05-27T08:05:09.610 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 -625us[ -643us] +/- 82ms 2024-05-27T08:05:09.610 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 -368us[ -386us] +/- 73ms 2024-05-27T08:05:09.611 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T08:05:09.683 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T08:05:09.683 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-05-27T08:05:09.683 INFO:teuthology.orchestra.run.smithi043.stdout:^* hv01.front.sepia.ceph.com 3 7 377 47 +821us[ +857us] +/- 59ms 2024-05-27T08:05:09.683 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 -638us[ -638us] +/- 82ms 2024-05-27T08:05:09.684 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -389us[ -389us] +/- 73ms 2024-05-27T08:05:09.684 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T08:05:09.707 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-27T08:05:09.707 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-05-27T08:05:09.707 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv01.front.sepia.ceph.com 3 6 377 42 +753us[ +757us] +/- 59ms 2024-05-27T08:05:09.708 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 45 -686us[ -682us] +/- 82ms 2024-05-27T08:05:09.708 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 -420us[ -415us] +/- 73ms 2024-05-27T08:05:09.708 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-27T08:05:09.711 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-27T08:05:09.723 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-27T08:05:09.724 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-27T08:05:09.752 DEBUG:teuthology.orchestra.run.smithi043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T08:05:10.094 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T08:05:10.351 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-27T08:05:10.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T08:05:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:05:10.635 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T08:05:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:05:10.730 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-27T08:05:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:05:10.823 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-27T08:05:10.835 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-27T08:05:10.875 INFO:teuthology.task.internal:Duration was 4702.647132 seconds 2024-05-27T08:05:10.875 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-27T08:05:10.911 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-27T08:05:10.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T08:05:10.919 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T08:05:10.927 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T08:05:11.060 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T08:05:11.069 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T08:05:11.097 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-27T08:05:11.588 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-27T08:05:11.588 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2024-05-27T08:05:11.588 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T08:05:11.654 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2024-05-27T08:05:11.654 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T08:05:11.733 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2024-05-27T08:05:11.734 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T08:05:11.809 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-27T08:05:11.810 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T08:05:11.818 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T08:05:11.825 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T08:05:11.910 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-27T08:05:11.910 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T08:05:11.916 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T08:05:11.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T08:05:13.743 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-27T08:05:13.756 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-27T08:05:13.756 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T08:05:13.841 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T08:05:13.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T08:05:14.027 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-27T08:05:14.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:05:14.047 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:05:14.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:05:14.151 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2024-05-27T08:05:14.164 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2024-05-27T08:05:14.167 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2024-05-27T08:05:14.220 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:05:14.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:05:14.277 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:05:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:05:14.330 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T08:05:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T08:05:14.387 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-27T08:05:14.399 INFO:teuthology.task.internal:Transferring archived files... 2024-05-27T08:05:14.400 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630/remote/smithi043 2024-05-27T08:05:14.400 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T08:05:14.551 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630/remote/smithi132 2024-05-27T08:05:14.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T08:05:14.664 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727630/remote/smithi179 2024-05-27T08:05:14.664 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T08:05:14.766 INFO:teuthology.task.internal:Removing archive directory... 2024-05-27T08:05:14.766 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T08:05:14.774 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T08:05:14.781 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T08:05:14.833 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-27T08:05:14.847 INFO:teuthology.task.internal:Not uploading archives. 2024-05-27T08:05:14.848 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-27T08:05:14.877 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-27T08:05:14.877 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T08:05:14.885 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T08:05:14.891 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T08:05:14.939 INFO:teuthology.orchestra.run.smithi043.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 08:05 /home/ubuntu/cephtest 2024-05-27T08:05:14.940 INFO:teuthology.orchestra.run.smithi132.stdout: 393505 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 08:05 /home/ubuntu/cephtest 2024-05-27T08:05:14.949 INFO:teuthology.orchestra.run.smithi179.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 08:05 /home/ubuntu/cephtest 2024-05-27T08:05:14.952 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-27T08:05:14.964 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-27T08:05:14.998 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi043 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump' 2024-05-27T08:05:15.003 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/dbench}} duration: 4702.647131681442 failure_reason: 'Command failed on smithi043 with status 22: ''sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 1bb64d42-1bf6-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump''' flavor: default mon.a-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 mon.b-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 mon.c-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 owner: scheduled_vshankar@teuthology status: fail success: false 2024-05-27T08:05:15.003 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T08:05:15.084 INFO:teuthology.run:FAIL